Inês Salselas
ABSTRACT
Sound design has been a fundamental component of audiovisual storytelling. However, with technological developments things are rapidly changing. More sensory information is available and, at the same time, the user is gaining agency upon the narrative, being offered the possibility of navigating or making other decisions. These new characteristics of immersive environments bring new challenges to storytelling in interactive narratives and require new strategies and techniques for audiovisual narrative progression. Can technology offer an immersive environment where the user has the sensation of agency, of choice, where her actions are not mediated by evident controls but subliminally induced in a way that it is ensured that a narrative is being followed? Can sound be a subliminal element that induces attentional focus on the most relevant elements for the narrative, inducing storytelling and biasing search in an audiovisual immersive environment? Herein, we present a literature review that has been guided by this prospect. With these questions in view, we present our exploration process in finding possible answers and potential solution paths. We point out that consistency, in terms coherency across sensory modalities and emotional matching may be a critical aspect.
- E. Adams. 2014. Fundamentals of game design, third edition. New Riders, Pearson Education.Google Scholar
- Giacomo Albert. 2012. Immersion as category of audiovisual experience: From Long Beach to Hollywood. Worlds of Audiovision (2012).Google Scholar
- V Alves and L Roque. 2009. A proposal of soundscape design guidelines for user experience enrichment. Audio Mostly 2009 (2009), 27--32.Google Scholar
- Richard A Andersen, Lawrence H Snyder, David C Bradley, and Jing Xing. 1997. Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annual Review of Neuroscience 20, 1 (1997), 303--330.Google ScholarCross Ref
- D. Arsenault. 2005. Dark waters: Spotlight on immersion.. In Game-On North America 2005 Conference: Eurosis. Elsevier, Ghent, Belgium, 50--52.Google Scholar
- Erkin Asutay, Daniel Västfjäll, Ana Tajadura-Jimenez, Anders Genell, Penny Bergman, and Mendel Kleiner. 2012. Emoacoustics: A study of the psychoacoustical and psychological dimensions of emotional sound design. Journal of the Audio Engineering Society 60, 1/2 (2012), 21--28.Google Scholar
- Dominik R Bach, John G Neuhoff, Walter Perrig, and Erich Seifritz. 2009. Looming sounds as warning signals: The function of motion cues. International Journal of Psychophysiology 74, 1 (2009), 28--33.Google ScholarCross Ref
- Durand R. Begault. 1992. Perceptual Effects of Synthetic Reverberation on Three-Dimensional Audio Systems. J. Audio Eng. Soc 40, 11 (1992), 895--904.Google Scholar
- Durand R Begault and Leonard J Trejo. 2000. 3-D sound for virtual reality and multimedia. (2000).Google Scholar
- Anne J Blood and Robert J Zatorre. 2001. Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. Proceedings of the National Academy of Sciences 98, 20 (2001), 11818--11823.Google ScholarCross Ref
- Margaret M. Bradley. 2008. Natural selective attention: Orienting and emotion. Psychophysiology 46 (2008), 1--11.Google ScholarCross Ref
- Emily Brown and Paul Cairns. 2004. A grounded investigation of game immersion. In CHI'04 extended abstracts on Human factors in computing systems. ACM, 1297--1300.Google ScholarDigital Library
- Alicia Callejas, Juan Lupianez, María Jesús Funes, and Pío Tudela. 2005. Modulations among the alerting, orienting and executive control networks. Experimental Brain Research 167, 1 (2005), 27--37.Google ScholarCross Ref
- Alicia Callejas, Juan Lupiáñez, and P10 Tudela. 2004. The three attentional networks: On their independence and interactions. Brain and cognition 54, 3 (2004), 225--227.Google Scholar
- Lihan Chen and Jean Vroomen. 2013. Intersensory binding across space and time: a tutorial review. Attention, Perception, & Psychophysics 75, 5 (2013), 790--811.Google ScholarCross Ref
- Jayoung Cho, Eunjou Yi, and Gilsoo Cho. 2001. Physiological responses evoked by fabric sounds and related mechanical and acoustical properties. Textile Research Journal 71, 12 (2001), 1068--1073.Google ScholarCross Ref
- Priscilla Chueng and Phil Marsden. 2003. Designing auditory spaces: the role of expectation. In Proceedings of 10th International Conference on Human Computer Interaction. Citeseer, 616--620.Google Scholar
- Carol L Colby and Michael E Goldberg. 1999. Space and attention in parietal cortex. Annual Review of Neuroscience 22, 1 (1999), 319--349.Google ScholarCross Ref
- Karen Collins. 2011. Making gamers cry: mirror neurons and embodied interaction with game sound. In Proceedings of the 6th audio mostly conference: a conference on interaction with sound. ACM, 39--46.Google ScholarDigital Library
- Rudolph P Darken, David Bernatovich, John P Lawson, and Barry Peterson. 1999. Quantitative measures of presence in virtual environments: the roles of attention and spatial comprehension. CyberPsychology & Behavior 2, 4 (1999), 337--347.Google ScholarCross Ref
- Thainá Cristina Demarque and ES Lima. 2013. Auditory hallucination: Audiological perspective for horror games. SBC-Proceedings of SBGames 2013 (2013).Google Scholar
- Jon Driver and Charles Spence. 1998. Cross-modal links in spatial attention. Philosophical Transactions of the Royal Society of London B: Biological Sciences 353, 1373 (1998), 1319--1331.Google ScholarCross Ref
- Jon Driver and Charles Spence. 1998. Crossmodal attention. Current opinion in neurobiology 8, 2 (1998), 245--253.Google Scholar
- Marian T Dura. 2006. The phenomenology of the music-listening experience. Arts Education Policy Review 107, 3 (2006), 25--32.Google ScholarCross Ref
- Inger Ekman. 2008. Psychologically motivated techniques for emotional sound in computer games. In Proc. AudioMostly 2008, 3rd Conference on Interaction with Sound, Piteå, Sweden. 20--26.Google Scholar
- Laura Ermi and Frans Mäyrä. 2005. Fundamental components of the gameplay experience: Analysing immersion. Worlds in play: International perspectives on digital games research 37, 2 (2005), 37--53.Google Scholar
- Jonathan Freeman and Jane Lessiter. 2001. Here, there and everywhere: the effects of multichannel audio on presence. In Proceedings of The Seventh International Conference on Audio Display, Helsinki, Finland. GIT, 231--234.Google Scholar
- Tom Garner, Mark Grimshaw, and Debbie Abdel Nabi. 2010. A preliminary experiment to assess the fear value of preselected sound parameters in a survival horror game. In Proceedings of the 5th Audio Mostly Conference: A Conference on Interaction with Sound. ACM, 10.Google ScholarDigital Library
- Tom A Garner and Mark Grimshaw. 2013. Psychophysiological assessment of fear experience in response to sound during computer video gameplay. In IADIS HCCI Conference.Google Scholar
- William W Gaver. 1993. How do we hear in the world? Explorations in ecological acoustics. Ecological psychology 5, 4 (1993), 285--313.Google Scholar
- Oliver Grau. 2003. Virtual Art: from illusion to immersion. MIT press.Google ScholarDigital Library
- Mark Grimshaw. 2007. The Resonating spaces of first-person shooter games. In Proceedings of the 5th International Conference on Game Design and Technology.Google Scholar
- Mark Grimshaw, John Charlton, and Richard Jagger. 2011. First-person shooters: Immersion and attention. Eludamos. Journal for Computer Game Culture 5, 1 (2011), 29--44.Google Scholar
- Stephen Handel. 2006. Perceptual coherence: Hearing and seeing. Oxford University Press.Google Scholar
- Claudia Hendrix and Woodrow Barfield. 1996. Presence within virtual environments as a function of visual display parameters. Presence: Teleoperators & Virtual Environments 5, 3 (1996), 274--289.Google ScholarDigital Library
- Don Ihde. 2007. Listening and voice: Phenomenologies of sound, Second Edition. Suny Press.Google Scholar
- Lucica Iordanescu, Emmanuel Guzman-Martinez, Marcia Grabowecky, and Satoru Suzuki. 2008. Characteristic sounds facilitate visual search. Psychonomic Bulletin & Review 15, 3 (2008), 548--554.Google ScholarCross Ref
- W.James. 1890. The Principles of Psychology. Vol. Volume 1. New York: Henry Holt.Google Scholar
- Charlene Jennett, Anna L Cox, Paul Cairns, Samira Dhoparee, Andrew Epps, Tim Tijs, and Alison Walton. 2008. Measuring and defining the experience of immersion in games. International Journal of Human-Computer Studies 66, 9 (2008), 641--661.Google ScholarDigital Library
- Charlene Ianthe Jennett. 2010. Is game immersion just another form of selective attention? An empirical investigation of real world dissociation in computer game immersion. Ph.D. Dissertation. UCL (University College London).Google Scholar
- Miho S Kitamura, Katsumi Watanabe, and Norimichi Kitagawa. 2016. Positive emotion facilitates audiovisual binding. Frontiers in Integrative Neuroscience 9 (2016), 66.Google ScholarCross Ref
- Pontus Larsson, Daniel Västfjäll, and Mendel Kleiner. 2004. Perception of self-motion and presence in auditory virtual environments. In Proceedings of seventh annual workshop presence. 252--258.Google Scholar
- Pontus Larsson, Daniel Västfjäll, Pierre Olsson, and Mendel Kleiner. 2007. When what you hear is what you see: Presence and auditory-visual integration in virtual environments. In Proceedings of the 10th annual international workshop on presence. 11--18.Google Scholar
- Scott D Lipscomb. 1999. Cross-modal integration: Synchronization of auditory and visual components in simple and complex media. The Journal of the Acoustical Society of America 105, 2 (1999), 1274.Google ScholarCross Ref
- Alison McMahan. 2003. Immersion, engagement and presence. The video game theory reader 67 (2003), 86.Google Scholar
- Simon Moncrieff, Chitra Dorai, and Svetha Venkatesh. 2001. Affect computing in film through sound energy dynamics. In Proceedings of the ninth ACM international conference on Multimedia. ACM, 525--527.Google ScholarDigital Library
- Janet Horowitz Murray. 2017. Hamlet on the holodeck: The future of narrative in cyberspace. MIT press.Google ScholarDigital Library
- Niels Christian Nilsson, Rolf Nordahl, and Stefania Serafin. 2016. Immersion revisited: A review of existing definitions of immersion and their relation to different theories of presence. Human Technology 12 (2016).Google Scholar
- Toemme Noesselt, Daniel Bergmann, Maria Hake, Hans-Jochen Heinze, and Robert Fendrich. 2008. Sound increases the saliency of visual events. Brain research 1220 (2008), 157--163.Google Scholar
- Kenji Ozawa, Satoshi Ohtake, Yôiti Suzuki, and Toshio Sone. 2003. Effects of visual information on auditory presence. Acoustical Science and Technology 24, 2 (2003), 97--99.Google ScholarCross Ref
- Jim R Parker and John Heerema. 2008. Audio interaction in computer mediated games. International Journal of Computer Games Technology 2008 (2008), 1.Google ScholarDigital Library
- Renato S Pellegrini. 2001. Quality assessment of auditory virtual environments. In Hiipakka, J., Zacharov, N., Takala, T. (eds.): Proceedings of the 7th International Conference on Auditory Display. Laboratory of Acoustics and Audio Signal Processing and the Telecommunications Software and Multimedia Laboratory. 161--168.Google Scholar
- Steven E Petersen and Michael I Posner. 2012. The attention system of the human brain: 20 years after. Annual review of neuroscience 35 (2012), 73--89.Google Scholar
- Michael I Posner (Ed.). 2012. Cognitive Neuroscience of Attention (second edition ed.). Guilford Press, New York.Google Scholar
- Michael I Posner and Stephen J Boies. 1971. Components of attention. Psychological review 78, 5 (1971), 391--408.Google Scholar
- Michael I Posner and Steven E Petersen. 1990. The attention system of the human brain. Annual review of neuroscience 13, 1 (1990), 25--42.Google Scholar
- Jane Raymond. 2009. Interactions of attention, emotion and motivation. Progress in brain research 176 (2009), 293--308.Google Scholar
- Amir Raz and Jason Buhle. 2006. Typologies of Attentional Networks. Nature Reviews Neuroscience 7, 5 (2006), 367--379.Google ScholarCross Ref
- Brendan Rooney, Ciarán Benson, and Eilis Hennessy. 2012. The apparent reality of movies and emotional arousal: A study using physiological and self-report measures. Poetics 40, 5 (2012), 405--422.Google ScholarCross Ref
- Marie-Laure Ryan. 2003. Narrative as virtual reality: Immersion and interactivity in literature and electronic media. Johns Hopkins University Press.Google ScholarDigital Library
- Michael Schutz and Scott Lipscomb. 2007. Hearing gestures, seeing music: Vision influences perceived tone duration. Perception 36, 6 (2007), 888--897.Google ScholarCross Ref
- Mel Slater. 2009. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society of London: Biological Sciences 364, 1535 (2009), 3549--3557.Google ScholarCross Ref
- Charles Spence and Jon Driver. 1997. Audiovisual links in exogenous covert spatial orienting. Perception & psychophysics 59, 1 (1997), 1--22.Google Scholar
- C. Spence and Driver J. 1997. Engineering Psychology and Cognitive Ergonomics: Volume Two - Job Design and Product Design. Routledge, Chapter Audiovisual Links in Attention: Implications for Interface Design.Google Scholar
- Ana Tajadura-Jiménez, Pontus Larsson, Aleksander Väljamäe, Daniel Västfjäll, and Mendel Kleiner. 2010. When room size matters: acoustic influences on emotional responses to sounds. Emotion 10, 3 (2010), 416.Google ScholarCross Ref
- Erik Van der Burg, Christian NL Olivers, Adelbert W Bronkhorst, and Jan Theeuwes. 2008. Pip and pop: nonspatial auditory signals improve spatial visual search. Journal of Experimental Psychology: Human Perception and Performance 34, 5 (2008), 1053.Google ScholarCross Ref
- Daniel Västfjäll. 2003. The subjective sense of presence, emotion recognition, and experienced emotions in auditory virtual environments. CyberPsychology & Behavior 6, 2 (2003), 181--188.Google ScholarCross Ref
- Daniel Västfjäll, Pontus Larsson, and Mendel Kleiner. 2002. Emotion and auditory virtual environments: affect-based judgments of music reproduced with virtual reverberation times. CyberPsychology & Behavior 5, 1 (2002), 19--32.Google ScholarCross Ref
- Valentijn T Visch, Ed S Tan, and Dylan Molenaar. 2010. The emotional and cognitive effect of immersion in film viewing. Cognition and Emotion 24, 8 (2010), 1439--1445.Google ScholarCross Ref
- Elizabeth Wenzel, Frederic Wightman, Doris Kistler, and Scott Foster. 1988. Acoustic origins of individual differences in sound localization behavior. The Journal of the Acoustical Society of America 84, S1 (1988), S79--S79.Google ScholarCross Ref
- Bob G Witmer and Michael J Singer. 1998. Measuring presence in virtual environments: A presence questionnaire. Presence 7, 3 (1998), 225--240.Google ScholarDigital Library
- Jenny Yiend. 2010. The effects of emotion on attention: A review of attentional processing of emotional information. Cognition and Emotion 24, 1 (2010), 3--47.Google ScholarCross Ref
Index Terms
- The role of sound in inducing storytelling in immersive environments
Recommendations
Exploring the usability of immersive interactive storytelling
VRST '10: Proceedings of the 17th ACM Symposium on Virtual Reality Software and TechnologyThe Entertainment potential of Virtual Reality is yet to be fully realised. In recent years, this potential has been described through the Holodeck™ metaphor, without however addressing the issue of content creation and gameplay. Recent progress in ...
Sound design inducing attention in the context of audiovisual immersive environments
AbstractSound design has been a fundamental component of audiovisual storytelling in linear media. However, with recent technological developments and the shift towards non-linear and immersive media, things are rapidly changing. More sensory information ...
Reality Tales: Facilitating User-Character Interaction with Immersive Storytelling
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing SystemsReality Tales is a platform to facilitate interaction with fictional story characters for readers (13-20 years). The platform leverages binaural audio and multi-voice narration for an immersive story experience. Several paradigms for interactive ...
Comments