Narrativity and the Senses in Virtual Reality Abstract Irene Kalesi MediaLAB Amsterdam irenekalesi@hotmail.com Jorrit Groot MediaLAB Amsterdam jorritgroot@hotmail.com Peter McLaughlin MediaLAB Amsterdam petermirrorimageuk@gmail.com
This paper will explore the possible ways that storytelling experience can be more immersive with the implementation of multiple sensual stimuli. The visual and auditory elements in Virtual Reality (VR) and their implications on the story and human perception (through psychological and brain studies) will be analyzed. The human perception and the brain's best function in VR will be centralized since a multisensory experience provides more input for the user to feel familiar and immersed in the story. The importance of the correlation of the narrative and the senses will be highlighted since in the virtual world the senses are the ones that can make the user immerse by sending feedback to his/her brain. The qualitative approach to VR storytelling aims to combine existent studies with the Project Orpheus' production documentation and theoretical hypotheses.
Author Keywords Virtual, reality; narratives; soundscape; perception, multisensory, experience; brain, studies; GPS, cells.
Introduction Virtual Reality is a medium that tried to emerge several times but unsuccessfully until now. At the present moment, VR passes its more fruitful period because the entertainment industry (games, VR films) is interested in it. VR has been struggling for decades to surface, the present moment, with the contemporary technological leaps, seems to be the perfect timing to investigate through its infinite possibilities regarding
viewer’s engagement through senses’ stimulation and structured narratives.
Methods The paper will follow:
Virtual Reality has been viewed by many as an extension of other media like cinema or just a paratext to video game and filmic/serial franchises. However, things are not that simple. VR is much more than people think. Virtual reality shows how different it is from its predecessors since it requires a new language and a different artistic approach. VR exercises its remediating power on theatre, cinema and video games. It is a medium that has incorporated others (theatre, cinema, games) but at the same time the created amalgam has different possibilities, implications on the user and the overall experience. Bolter and Grusin (2010) stated that “mediation is the remediation of reality because media themselves are real and because the experience of media is the subject of remediation” (59). Therefore, it is important to view Virtual Reality as an experience-oriented medium that focuses on empathy and improvement of the user’s position and involvement in each storyworld. The VR world is an experimental ground on which developers focus on visual gratification and exploration of the visual aspect of VR relegating the rest of the senses to second place. Virtual bedazzlement has been the easy way out for the VR developers but not taking into consideration the users' limitations, needs and preferences can only produce bad content which can potentially lead to VR’s detriment. As James Milward, creator of the Sleepy Hollow VR Experience said "Bad Content will destroy VR” (BBC news, 2016). Thus, the concentration on visual representation should be accompanied by narrative depth.
• a theoretical qualitative approach Virtual Reality viewing it as a new storytelling medium following McLuhan's theory on the power of the medium, the concept of Remediation formulated by Bolter and Grusin as well as Elsaesser’s theory on sound. • Brain studies and perception researches will assemble the ways that the user behaves and accepts Virtual Reality. • The case study of Project Orpheus will offer the quantitative side of the paper since the visual and auditory examples will be taken and analysed from it. • The theoretical research will be tested with the Project Orpheus Experience as its main case study.
1. Visualizing a Narrative Narratives are the spinal cords of experiences. They represent ways of approaching the real and fantasy world. Paul Cobley (2001) explained the term as “a particular form of representation implementing signs” (6). In most of the media, narratives can be verbal, gesticular, personal or abstract with the incorporation of colors and concepts. Therefore, because of their importance, the narration of stories in VR is even more important since the medium offers different possibilities for the user’s experience. The biggest difference is that the position of the viewer is annulled and the user becomes an organic part of the virtual reality environment and story.
Since Virtual Reality puts the user in a subjective but simultaneously ubiquitous position, he can be paralleled to the omniscient narrator in stories. Acquiring the ability to have peripheral view in the story’s environment is pivotal in the way that the story should revolve around the user or involve her/him in an integral way. According to Sandy Louchart and Ruth Aylett (2001), “the consideration of the user and his or her behaviour as a primary resource for the storytelling system, brings a different perspective to the role of the user within the story, a character based interactive storytelling system (18). The narrative should be structured as if the user is a “visitor” and thus should be guided through visual/auditory guides or an “investigator” who has to familiarize with the environment and follow the story on his/her own understanding. Therefore, when the narration has an abstract nature, it implies an increase of the flexibility of the user’s position as well as of the interest levels which make the people pay more attention to the actions happening around. Thus, the way the story is presented can help the person assume the role and experience the formation of his/her presence around his/her virtual character. If Marshall McLuhan (1964) considered technology as the extension of the human central nervous system, then Virtual Reality could be thought as the best example of a living organism. In VR, the spherical immersive aspect of the experience can be linked to self-revitalization and not autoamputation since it is a puzzle where human perception along with the senses and the environment are essential to the successful experience of VR. The person is not a spectator or a mere critic but s/he is part of the story and above all the organic part of the whole technological venture. However, stories in VR cannot be considered complete without the careful implementation of sensual stimuli. Sight has the primary role in the construction of a virtual world
since every VR experience focuses mostly on the visual part. The narratives that are visualized in VR should be intrinsically intertwined with visually interesting cues and interactions with the user. If the user is unconvincingly or without the proper technological tools introduced to different places then, the level of immersion decreases and thus the loss of attention is next to follow. In Virtual Reality, the human brain, especially the region of the hippocampus, is trying to map out the reality that it’s encountering. Space mapping is the process during which, depth and distances between objects and people are calculated, so that the brain can move on to memorizing details and immersing in the environment. According to a study published in Journal Nature Neuroscience (2014), experiments on rats have shown that virtual and physical environments trigger different cognitive processes. In VR, the place cells or else called GPS cells of the brain function irregularly in VR in comparison to real environments, so the brain needs more diverse stimuli in order to space map and thus be able to immerse in the environment and consequently the story. The human brains are in need of all the senses to make sense of the environment around them and thus when Virtual Reality simply offers visual stimuli, it is not enough for the cognitive processes to substantiate the virtual surroundings Thus, the role of the sound is the one that attains the improvement of the space mapping and the whole experience.
2. The Power of Sound A visual narrative can seem interesting to our eyes but the isolation of the sense of sight is not a right move and most importantly a challenging process to our brain. "Spatial perception depends on a synthesis of information from multiple sources, including all the senses” Loren Frank told
Cynthia Fox from Bioscience Technology. Especially in Virtual Reality, the auditory experience is the one to be explored deeply. According to Elsaesser (2010), “It is the ear that renders the image visible” (145). Thus, if sound makes our vision better and our sense of presence stronger then the developers should direct themselves towards exploring the infinite possibilities of sound. The implementation of sound and most importantly binaural sound can improve the brain function of space mapping and perception that lead to immersion. As Jason Dorrier from singularityhub.com (2014) explained "positional 3D audio tricks the ears that it originates in specific locations". This illusionary recreation of a reality is the key element to immersion, memory creation and storing. The hippocampus which is responsible for these processes should be the object of research because the way this part of the brain works will provide solutions beyond entertainment purposes. The most recent advancement in Virtual Sound is the soundscape which is an organic way of producing and localizing sound in the space around the user. The sound distance, the echoing, the sudden changes of the source of the sounds create a whole new field of possibilities for developers to measure the levels of immersion and perception of virtual reality. The sound design progress offers the users a stimulating new way of interacting with the environment in VR since the user, the sound and the surroundings form a living sphere. This experiential sphere should be orchestrated properly because timing is of the essence when sound and vision are mingled. In cybertherapy.info, it is stated that "the feeling of presence cannot be reached by one sense it is a straightforward approach to couple different senses to reach an optimal feeling of presence within a virtual reality".
3. The Curious Case of Project Orpheus Creating a Virtual Reality experience connected to a television series as a transmedia paratext but at the same time as an autonomous product is an interesting and challenging process. Throughout all the phases of a production, there are some hypotheses that really make sense on how the film’s narration would be successful in VR. The premise of the Project Orpheus’ virtual experience is to give the audience an imaginary glimpse of how an after death experience can be visually represented and sensed. Since the VR experience is linked to the titular series, the final film should help the users immerse in an ambiguous after death environment with eerie and vaguely threatening elements but also create direct and implicit references to the Project Orpheus series. More specifically, the concept of Death and the afterlife is a non-existent, extremely ambiguous formulated by cultural/religious beliefs and prejudice but also offers a certain freedom to create a virtual environment. The visual stimuli of Project Orpheus vary from direct interaction with the viewer to the formation of a stimulating environment ofr the user to immerse into it. For example, the main character, the young boy functions as a visual guide but at the same time as a multi-positional stimulus. Even though the script of the experience does not contain speech, the movements of the boys can be seen as a substitution of dialogue that and a form of interaction. Moreover, the multiple visual cues in the corridor and office scene challenge the user’s ability to receive and analyze different stimuli simultaneously in order to create suspense and provoke an unsettling atmosphere. The successful combination of these two senses will be the gateway for the users to feel immersed and ready to be part of the virtual world. For instance, the sound in Project Orpheus is multiple accompanying the correspondent visual
cues but also "misleading" the user in order to create suspense and awake the sense of exploration in the unknown territory of VR. During the editing phase of the film, the lack of sound decreased the impact of the experience. Thus, the addition of the heartbeat of the user, the heavy breathing coming from the entities in the film as well as the footsteps creates an ominous environment. They transmit emotion since natural noise helps the audience immerse in the environment and atmosphere. The implementation of the subtle violin and string instruments fading in the air adds an eerie layer as transitions in each of the scenes of the experience. On the whole, the power of sound is evident when with primal instincts and connected with fear. Even in the realm of film, as Elsaesser (2010) indicated “One of the features of the so-called “new sound picture” is that it puts the spectator into “free-fall”, in time as well as in space.” (144). Therefore if you transfer the feeling of free-falling experienced by spectators in a movie theatre to a virtual reality context, the sense is magnified and the impact on the user is gravely amplified. The user doesn’t have to empathize with what the camera is showing because s/he controls the narrative he/she is being exposed to. So in the case of the Project Orpheus is not just an immersion tool but also is linked to fear and the uneasy feeling of danger.
Conclusion The combination of multiple senses will improve the users' experience. Storytelling is an organism that feeds from the senses and thus the creation of a stimulating narrative can benefit its meaning. The brain function and the role of psychology should be centralized during the formation of narratives.
The spherical storyworld of VR will bring new dimensions to world building in order to visually create a reality but also acoustically since the sound localization will be the key element to the engagement of the users. Narratives are interesting and possible in VR, so storytellers should invent new techniques to tell stories and implement different sensual input. If the presence of a user is stronger, the story becomes more interesting and vice versa. Virtual Reality is a multimedium and should be treated as such, thus the narratives should be viewed from multiple perspectives as if in a spherical environment.
Further Discussion The different forms of narratives that can succeed in VR should be explored and all the possible combinations of sensorial stimuli should be tested in order to have a more complete and immersive experience. Even if the senses of taste or smell are not to be implemented in an organic way that does not, the proper manipulation of sight hearing and touch can stimulate the other senses too by association or through digging in the user’s memory storage.
Acknowledgements Thanks to Avrotros, NL Films, for providing everything needed for our case study “The Project Orpheus Experience” a reality. Special Thanks to We Make VR for introducing us to the technical side of Virtual Reality which helped us understand and appreciate the medium even more. Last but not least, thanks to MediaLAB for funding and support in our project since the beginning
References 1. Aylett, R., and S. Louchart. "Towards a Narrative Theory of Virtual Reality." Virtual Reality 7.1 (2003): n. pag. Web.
11. Wlassoff, Viatcheslav. "Brains Aren’t Fooled By Virtual Reality." Brain Blogger Brains Arent Fooled By Virtual Reality Comments. N.p., 24 Feb. 2015. Web. 28 Jan. 2016. .
2. BBC News. BBC, 29 Sept. 2015. Web. 18 Jan. 2016 3. Bolter, J D, and Richard A. Grusin. Remediation: Understanding New Media. Cambridge, Mass: MIT Press, 1999. Print. 4. Cobley, Paul. Narrative. London: Routledge, 2001. Print. 5. Dorrier, Jason. "What's Missing from Virtual Reality? Immersive 3D Soundscapes - Singularity HUB." Singularity HUB. N.p., 06 July 2014. Web. 28 Jan. 2016. 6. Elsaesser, Thomas, and Malte Hagener. "Chapter 6: Cinema as Ear." Film Theory: An Introduction through the Senses. New York: Routledge, 2010. 7. Fox, Cynthia. "Virtual Reality vs. Real Life: How Brain Neurons Light Up." Bioscience Technology. N.p., 01 Dec. 2015. Web. 28 Jan. 2016. 8. McLuhan, Marshall. Understanding Media: The Extensions of Man. 1964. Print. 9. "SOUND in VR." Sound in VR. N.p., n.d. Web. 28 Jan. 2016. 10. Wolpert, Stuart. "Brain's Reaction to Virtual Reality Should Prompt Further Study, Suggests New Research by UCLA Neuroscientists." UCLA Newsroom. N.p., 24 Nov. 2014. Web. 28 Jan. 2016.