Third Wave or Winter? The Past and Future of Smell in HCI

1 INTRODUCTION

In the past decades, there has been a monumental and multi-disciplinary effort from all corners of Human-Computer Interaction to bring more human senses into interactive systems [5]. This trend is most pronounced in the recent wave of virtual reality systems, the first to finally hit the mainstream, that aim to take advantage of as many senses as possible.

Integrating as many senses into interactive systems as possible is significant because we use all our senses in critical, everyday interactions, yet, currently, only an extremely narrow subset when interacting with computers. Many sub-fields of HCI note this discrepancy, one of these is olfactory interface research. Odors play critical roles in our lives: from the pleasures of food [23] to detecting potential hazards [27] to an essential role in memories and emotions [13,24]. HCI researchers have been leveraging these opportunities [20] in interactive experiences, such as to increase virtual presence [2], provide notifications [17], promote wellbeing [1], and enhance safety [18], just to cite a few.

In more than half a century of HCI research and decades of olfactory research in HCI, there have been only a few workshops [5,21], and there have been no panels that have taken a wide and radical look at the field. Historically, the integration of scent in technology could roughly be seen as having two public waves (analogous to virtual reality's three to four “waves” [12:3]).

The first smell-wave occurred in the early 20^th century (1900-1960s), specifically in the cinematic arts, and culminated in 1959 and 1960 with the premiere of Behind the Great Wall in AromaRama [8,10] Scent of Mystery in Smell-O-Vision [15]. Most notably, this first wave of smell was the birthplace of the of the most groundbreaking steps in HCI research: the Sensorama [11] by Morton Heilig (also considered by most the “first VR device”, a predecessor even to Sutherland's work [28]).

The second wave (1990-2000s) of scent in technology – bringing with more advances in HCI – can be roughly traced to the adoption of the World Wide Web. Researchers and technologists grew fascinated with what could be transmitted over the Internet. Attempts at remotely delivering the very smell of the coffee then began to emerge [25]. Around 1999 (and thereby the dot-com era), start-ups rose, promising to enhance our desktop experience with scents over the web (e.g., DigiScents, TriSenx, SenseIT). Separately, and arguably far more ubiquitous and successful, was the rise of home fragrance technologies, such as automatic air fresheners, which led to makers, researchers, and hobbyists hacking them for their own purposes [9,31]. The 90s and 00s saw a sharp rise in scent in HCI. Our co-organizer Jofish Kaye was among the first to explore scent's potential for communicating information remotely [14]. Yasuyuki Yanagida experimented with vortex generators to deliver smells at a distance [29,30]. Brewster et al. experimented with designing smell-based interactions for photobooks [3], amongst others. Despite these exciting developments, scent interfaces went back to sleep in the mid-2000s, though seeing a steady – albeit small – rise in HCI publications until the next wave [5].

Are we seeing a new wave or a winter? More recently, there have been exciting developments framing multisensory interfaces as the future of HCI [21], research on the influence of smell on body image [4] and collaboration [19], work increasing VR realism via the secondary system contributing to smell (the trigeminal nerve) [6], efforts towards digitizing smell sensations [7,26], and new work demonstrating accurate prediction of perceived odor from molecular structure [16]. Additionally, we are in an unprecedented societal shift, now aware of the importance of smell due to the coverage of COVID-19 related smell loss [22,32]. This panel asks if we are on the cusp of a third wave and whether the field is here to stay (academically and commercially) or should prepare for another winter.

The panel will explore three related questions have riddled the field for 60 years: (1) when will society be ready for these technologies, (2) what is the “killer” application for scent (and what is scent good for), and (3) will we ever sense and reproduce smells faithfully?

2 PANEL FORMAT

The panel will include three to five panelists, one of whom will act as a moderator. It will be hybrid: in other words, the panel will be in person, and attendees can also participate virtually.

The panel will begin with the moderator introducing themselves, the panelists, and the topic of discussion. Panelists will briefly (each five minutes) summarize their most recent research and their take on the future of smell in HCI. Each panelist will be invited to additionally provide illustrative scents to the audience, which the organizers (Brooks and Lopes) will produce using scratch-and-sniff cards created only for this panel. As such, during these provocative presentations, when a panelist refers to an odor, a prompt will appear on screen so that the audience member can scratch their “CHI23 scratch-and-sniff card” and smell the associated odor, fully immersing their sense of smell in the discussion.

The moderator will guide the discussion using the following prompts, but also open the conversation to the audience:

After the introductions (15-25 minutes) and initial discussion (40 minutes), we will open the floor for a discussion with the audience. We encourage an interactive debate and deep discussion around these questions, especially as they have become timely after COVID-19’s reported effects on olfaction (e.g., anosmia) and a growing interest in this field. The discussion will be streamed on Zoom to allow for remote participation. Remote attendees can ask questions through either the chat function or a question-submission portal, accessible to the moderator and panelists with the help of a student volunteer.

Attendance: We are confident that the panel will be of interest to the CHI community and have good attendance. First, we would be advertising the panel within the STT23 workshop at CHI, which is not a workshop exclusively on smell but will also be a gathering place for some olfactory researchers from HCI. Moreover, in 2020, we self-organized an online symposium that served as the first pilot for our proposal. This symposium included many papers that were not shown due to the cancellation of CHI 2020. We had 255 people registered from over 33 countries worldwide. At CHI 2021, we organized a virtual workshop that was the first-time researchers had a forum to present their work.

Logistical needs and considerations: We mostly need typical A/V support for a hybrid event for the panel. Given the opportunity for providing smell scratch-and-sniff cards, we could use one student volunteer to distribute samples to attendees as they enter. Mailed scratch-and-sniff cards will be available to remote attendees ahead of CHI.

3 PANELISTS

The panel is organized by Jas Brooks, a PhD student at the University of Chicago, and their advisor Pedro Lopes, an Assistant Professor at the University of Chicago. Jas's doctoral research focuses on the development of chemical interfaces, including olfactory interfaces. The panel itself will be moderated by HCI veteran Jofish Kaye, currently Senior Director of Interaction Design & Artificial Intelligence at anthem.ai. Jofish's Master's Thesis “Symbolic Olfactory Display” from 2001 is an early and groundbreaking example of scent-use in HCI.

The panelists below were selected to provide complimentary perspectives on the history and future of this topic.

Marianna Obrist is Professor of Multisensory Interfaces at University College London., she was head of the Sussex Computer Human Interaction (SCHI 'sky') Lab at the School of Engineering and Informatics at the University of Sussex. Her research ambition is to establish touch, taste, and smell as interaction modalities in human-computer interaction (HCI). Her research is mainly supported by an ERC starting grant. As part of her research, she developed a novel scent-delivery technology that was exhibited at the World Economic Forum (WEF) 2019 and 2020 in Davos. Supported by an ERC proof-of-concept, this technology is now commercialized through OWidgets Ltd, a university start-up she co-founded in 2019. Before joining Sussex, Marianna was a Marie Curie Fellow at Newcastle University. She was selected Young Scientist 2017 and 2018 to attend the WEF in China and became an inaugural member of the ACM Future of Computing Academy (ACM-FCA) in 2017. More recently, Marianna was appointed as a Visiting Professor at the Burberry Material Futures Research Group at RCA and spent the summer 2019 as a Visiting Professor at the HCI Engineering Group at MIT CSAIL. Website: multi-sensory.info

Jofish Kaye runs research teams to produce thoughtful and ethical HCI & AI products and is currently working with anthem.ai to improve healthcare. His research explores the social, cultural, and technological effects of technology on people, and how people's decisions, needs, and behaviors can change and improve those technologies. He recently ran a research group at Mozilla, chaired CHI 2016, co-chaired CHI 2021 workshops track, and has a long running interest in improving diversity, inclusion, and accessibility. In 2001, Jofish received his master's in Media Arts & Sciences from the MIT Media Lab for work on olfactory information displays, largely regarded as a foundational text in the subfield. Website: jofish.com

Judith Amores is a Research Fellow at the MGH/Harvard Medical School and a Research Affiliate at the MIT Media Lab, where she did her Ph.D. and master's and helped run VR/AR at MIT as a co-president. She holds a multimedia engineering degree and has worked at Microsoft Research, URL Barcelona, and the Google Creative Lab. Her awards and publications include over 27 peer-reviewed research papers, two patents, a Facebook Graduate Fellowship, LEGO Foundation sponsored research, and was a finalist of the Innovation by Design Awards. She also received the Scent Innovator Award by CEW and IFF. Website: judithamores.com

Jas Brooks is a Ph.D. student in Computer at the University of Chicago, advised by Professor Pedro Lopes. Their research expands the computer interface to engage with the chemical interactions of our bodies and investigates what this paradigm enables. These chemical interfaces have given rise to new I/O methods and are uniquely situated to help with smell and taste disorders. Jas has published work at top Human-Computer Interaction (HCI) venues, including ACM CHI and UIST, of which 2 received Best Paper Awards. Their work has also been awarded an Honorable Mention in the Fast Company Innovation by Design Awards for Experimental Design and covered in media publications like WIRED, Fast Company, Digital Trends, and IEEE Spectrum. Jas is a National Science Foundation Graduate Research Fellow. Independently, Jas documents the history of scent technologies and media. They are currently revisiting the story of Behind the Great Wall in AromaRama, conserving the last known Smell-O-Vision system and Scent of Mystery fragrance from 1960, and interviewing employees from scent tech companies at the turn-of-the-21^st century (e.g., DigiScents, Inc.). The latter is in collaboration with Professor Simon Niedenthal (Univ. of Malmö) and received a Scientific Contribution Award at the Digital Olfaction Society's 2022 World Congress. Website: jasbrooks.net

Acknowledgements

This work was supported by NSF grant 2047189. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any funding agencies.