Keywords

1 Introduction

The aesthetics of an interface design plays an important role in the quality of a user’s experience. Research has shown that the aesthetics of an interactive, interface design can impact user engagement, completion time, and error rate [1,2,3]. Additional research indicated that the visual appeal of the design encouraged users to stay engaged and complete tasks, even when the organization and functionality of the interface design did not yield the fastest time in task completion [4].

Audiovisual design elements contribute to the aesthetics of an object or experience. The relationships between line, form, color, texture, contrast, rhythm, and audio elements such as timbre, volume, and velocity combine in different ways to communicate emotions, harmony, resolution, tension, and movement. Design characteristics such as bright colors, contrast, position, movement, and sound can define audiovisual hierarchies that prioritize information by highlighting specific audio or visual elements. The Gestalt laws of perception simplify the communication process by helping us group audiovisual information with similar design characteristics (e.g., size, shape, color, texture, rhythm, timbre, pitch, volume, velocity, etc.). Discord and tension, as well as harmony, also impact the aesthetics of an experience. Both define the affective experience, and they can highlight differences and draw attention to specific relationships, as well as capture the attention of the user.

However, the aesthetics of a design is not only defined by the audiovisual characteristics of an object or experience. Aesthetics is not just an affective experience based on emotional reactions. There isn’t one definition of aesthetics or specific set of criteria that defines a “good” aesthetic experience. Kant [5] proposed that aesthetics involves logic, as well as the sensory properties of an object or experience. Baumgarten [6] emphasized the importance of sensory perception and cognition in aesthetics. Tuan [7] also highlighted the importance of the senses and cognition in defining the meaning of objects and physical environments, and in turn, the aesthetic experience: “An object or place achieves concrete reality when our experience of it is total, that is, through all the senses as well as with the active and reflective mind” (p. 18).

Artists, designers, philosophers, and psychologists continue to redefine and expand the definition of aesthetics. New electronic technologies that involve interactive, multimedia designs challenge us to define a new aesthetic discourse. The physical interactions of the user in environments where information continually changes, and in hybrid spaces that blend physical and virtual spaces, define the aesthetics of the experience. In these electronic environments, the aesthetics of a design is not static. It continually evolves as relationships change and new experiences are added to our knowledge base.

Research has shown that semantic relationships, sensory perception, cognition, schemas, memory, physical interaction and embodiment, and social discourse impact how we interpret information and the aesthetics of a design. We can gain a better understanding of the aesthetics of interaction design by looking at research in semiotics, cross-modal perception, cognitive psychology, and philosophy. This paper highlights some of the research in these fields that applies to the aesthetics of interaction design.

2 Interactive Semiotics

The semiotics of an interactive, multimedia design is an essential part of the aesthetics of the design. Design can symbolize specific functions. Affordances in design refer to the perceived and actual design elements of an object that suggest how the object should be used [8]. Some affordances that contribute to the aesthetics of the design include:

  • Physical Affordance: Design property that makes it possible to physically do something with the object

  • Sensory Affordance: Design property that enables the use of the senses

  • Cognitive Affordance: Design property that makes it possible to think or know something

  • Functional Affordance: Design property that defines the purpose of the design [9, p. 8]

Affordances define layers of signification that become part of the semantic structure of the design, and in turn, part of the design aesthetics. Through experience and interaction with an object, users learn what to expect from a particular design. They form conclusions that can be applied to the interpretation of other designs.

In multimedia computing, the integration of sound and visual design elements expands the semantic structure of a design and creates an interactive dialogue with multiple layers of associations. Research [10] has shown that individuals actively engage in the process of deciphering the relationships between visuals and audio, and derive pleasure from this process, which becomes part of the aesthetic experience. There is “a neurological pleasure in complex processing” because “the brain is pleasurably occupied with the task of simultaneously processing (and perhaps matching) two different visual and auditory codes” [10, p. 53].

In audiovisual design, each sensory stimulus can impact the perception of other stimuli. Research has shown that the perception of visual information is altered when sound is added to the visuals [11,12,13,14]. Sound can enhance the detection of specific individual visual elements, as well as improve the detection of motion [11, 15]. The intensity of sound can highlight the perceived contrast and intensity of a visual stimulus [11, 16], and the perception of repetition in visuals is enhanced with the addition of repetitive sounds [17].

O’Leary and Rhodes [18] also discovered that segmenting sensory information in visual stimuli can result in the perception of segmentation in the audio stimuli. For example, their research showed that visual elements that are presented as two separate elements or movements result in the perception of two separate audio tones when concurrent sounds are introduced. Vroomen and de Gelder [11] discovered that if a tone separates from an auditory stream, visual elements also separate from synchronized visual stimuli. However, when continuous visual elements are present in an audiovisual design, the sounds will also be perceived as continuous [18].

A multimedia design is a synthesis of different semiotic structures that enables users to transcend the limited perspective of specific media and actions. Layers of spatial, temporal, and sensory networks continually change. There are parallel, synchronous formats and linear progressions through the content. A new audiovisual semantic structure integrates the syntax of the media into complex affective and cognitive models. The integration of different media results in an overarching metasyntax that is transmodal [19, 20]. The metasyntax integrates the semantic, spatial, and temporal modalities of words, images, sound, and movements into a holistic, multisensory experience. This pluralism results in a polysemiotic semantic structure that defines a discursive communication experience. The different media and movements create an enactive, iconic, and symbolic space in which semiotic structures overlap [21]. The metasyntax creates a fluid semiosis and design aesthetic by defining sensory and cognitive relationships that transcend the meaning of individual elements.

With interaction design, causality also contributes to the semiotics and aesthetics of the design. Actions lead to specific audiovisual responses, and audiovisual events can trigger other events. There is a lack of closure that keeps the semantic structure and aesthetics of the design in flux and continually evolving.

In interactive computing, the physical motions of the user also contribute to the semiotics and aesthetics of the design through the spatial grammar of interaction [22]. The movements define rhythm and tempo. The physical interaction leads to additional information and events that define new sensory and cognitive relationships. Djajadiningrat, Matthews, and Stienstra [23] referred to the “semantics of motion” which shifts action from a purely “non-functional” role to an aesthetic role that is “necessary” for an engaging experience (pp. 10–11). In designing interactive environments, there is usually a focus on the specific action or movement that generates a response. However, the movements in between these actions, which symbolize the potential for new ideas and creative exploration, also contribute to the aesthetic interpretation of the interactive experience.

3 Space and Time

Space and time also define the aesthetics of the design. Space and time are flexible entities that describe relationships between events. The process of interaction creates patterns and rhythms that encode space and time into tangible representations of dynamic relationships and the transformation of ideas [22].

Visual elements and sound create sensory responses and impact the perception of space and time. Visuals can define spatial relationships through perspective, color, transparency, and transitions that take place over time. Sound can be both spatial and linear. Melodic sequences are linear, but sound also penetrates space and creates additional layers of depth that expand our awareness of spatial and temporal relationships. Research has demonstrated that audio and visual stimuli can impact the perception of spatial location [24,25,26,27,28,29]. The velocities [30,31,32,33], relative intensities [34, 35], and duration [35] of auditory and visual stimuli also impact the perception of time and whether or not sounds and visuals appear to be synchronized and occur simultaneously.

Silence is a design element that also contributes to the aesthetics of an experience. It is a space that signifies open-ended possibilities. Silence can intensity the experience as the user waits for something to happen. It can also provide an opportunity for the user to reflect on the information and relationships that have been presented.

In interactive programs, the space between events is as important as the space where events actually happen. Space is where sensory and cognitive relationships are formed which lead to new aesthetic and learning experiences. Space defines moments in the continual process of change, demonstrating that “space and time serve as the contexts in which all communication entities exist and unfold” [36, p. 74].

3.1 Hybrid Spaces

In some interactive applications, such as virtual reality and augmented reality applications, there is also the integration of different types of spaces that impacts the semiotics and aesthetics of the design. These hybrid environments create opportunities to explore sensory and cognitive relationships in the physical and virtual spaces from different perspectives. Ross [37] pointed out that augmented reality’s “potential innovativeness lies in its ability to generate new ways of perceiving for the spectator or to disclose what was previously unperceived—unseen, unheard, unfelt, unsmelt” (Introduction section, para. 4).

In hybrid environments, the audiovisual information in physical and virtual spaces becomes part of the syntax and aesthetics of the interactive experience. The interface designs integrate the aesthetics of the physical environment with the aesthetics of the virtual world and program functions. Virtual and physical spaces overlap and combine different information and spatial relationships defined by images, sound, and text. The integration of these diverse layers of sensory and cognitive information impacts the semiotics and aesthetics of the interface design. In augmented reality applications, for example, virtual information is displayed on top of images and sounds in the physical environment. The user experience includes the cognitive processing of facts and analytical relationships, as well as the sensory perception of audiovisual stimuli in the virtual and physical spaces [38].

The integration of these layers of cognitive and sensory information presents designers with new challenges in semiotics and aesthetics. In addition, virtual and physical information may appear within a frame formed by the mobile device or other hardware. These frames form boundaries that define the space and how users interpret the information in that space:

Only select elements from the physical space are visible within the frame, creating a focus on that information rather than the information outside the frame. The frame itself suggests a finite limit because the experience is defined by discrete groups of data and spatial relationships… The technology emphasizes specific units of information at defined moments in time, rather than highlighting the connections between memories and experiences [38, p. 242].

3.2 Temporal Dynamics

The temporal dynamics of an interactive design also define the semiotics and aesthetics of the work. Information and semantic relationships change over time. There is an emphasis on “now” and “immediacy” [39]. Users anticipate immediate feedback and changes. These temporal dynamics drive the interaction as users feel compelled to update and share information on a regular basis [39]. The “present” takes on significance as it represents the current state of affairs in an environment that is continually changing.

There is also an emphasis on non-linear, pluralistic concepts of temporality that emphasize simultaneity. With multimedia design, it is possible to perceive sounds and visuals simultaneously as well as sequentially, resulting in the juxtaposition of different rhythms and sensory experiences. Because we can hear sounds while we look at visuals, we are able to simultaneously explore different temporal connections between the visuals and sound [40].

Rhythm is a design element that exemplifies the temporal dynamics of interaction design and plays an important role in the aesthetics of interactive, multimedia design. Layers of rhythms in the visual designs, sound, animations, and transitions between elements highlight the temporal relationships. Combinations of linear, sequential rhythms, as well as cyclical rhythms, create audiovisual counterpoints and syncopation. Pauses, empty spaces, and silence alter the rhythm and temporal dynamics of the interactive aesthetics [39]. The repetition of specific rhythms can weave individual elements into a coherent whole. Rhythm can create a unifying, overarching structure for the flexible, semiotic codes that characterize the dynamic aesthetics of multimedia design [39].

The physical interaction of the user also creates layers of rhythm. Djajadiningrat, Matthews, and Stienstra [23] referred to this rhythm as the “choreography motion” and “semantics of motion” which is necessary to envision an experience (p. 31, pp. 10–11). Physical movements define patterns in space and time that create a rhythmic counterpoint to the audiovisual patterns [41]. These patterns may repeat or complement the rhythms in the audiovisual elements, or they may be different and create layers of contrast.

4 Perception and Cognition

Sensory perception and cognition contribute to the aesthetic interpretation of an object or experience. Norman [42] noted that there is beauty that “is associated with the object itself” which is processed perceptually at the visceral, subconscious level, and there is beauty that is related to consciousness (p. 314). Referring to this conscious level of aesthetics interpretation, Norman [42] stated:

It is only at the reflective level that full-fledged emotions reside. This level is intellectually driven. It is conscious and aware of emotional feelings. Moreover, it uses the rich history of prior experiences, one’s own self-image, and personal meanings to evaluate any experience (p. 315).

Brunel, Carvalho, and Goldstone [43] cited established research that demonstrated the connection between perception and cognition, noting that the senses generally make it easier to identify [44], detect [45], categorize [46], and recognize [47] information [43, para. 3].

Parallel connections between sensory experiences also contribute to our interpretation of the aesthetics of an object or experience. When sensory experiences are not “congruent,” it is possible to assign connections between sensory modalities that the user will later associate with these modalities [48,49,50]. We learn to integrate these sensory experiences and group them into “multimodal units” through an associative learning process called “unitization” [51, para. 1]. We also integrate current perceptual experiences with past experiences [52, 43]. As the learning process continues, new cognitive and aesthetic interpretations of an action or experience evolve.

Cross-modal perception also plays an important role in cognition and aesthetics. As previously discussed in the section on Interactive Semiotics, sensory stimuli can impact the perception of other sensory information. Cross-modal perception adds multiple layers of complex sensory relationships and interpretations to the cognitive and aesthetic models.

5 Interaction and Embodied Aesthetics

Embodiment is another dimension of interactive aesthetics. Physical movements bridge the physical and virtual spaces and help users create tangible connections to the visual and cognitive relationships defined in the digital world [53].

Merleau-Ponty [54] noted that “we perceive the world with our body” (p. 239), and he highlighted the interrelationship between perception and embodiment in the interpretation of objects and space: “The identity of the thing through perceptual experience is only another aspect of the identity of one’s own body throughout exploratory movements; thus they are the same in kind as each other” (p. 215). Klemmer, Hartmann, and Takayma [55] pointed out that “our bodies play a central role in shaping human experience in the world, understanding of the world, and interactions in the world” (p. 140).

Physical movements help us interpret spatial and temporal relationships and shape our cognitive understanding of objects and experiences [55,56,57]. Movement through space leads to different viewpoints, resulting in new interpretations that define the aesthetics of an object or experience.

In interaction design, interfaces that incorporate tangible connections to the physical world engage the senses and augment the learning experience [55, 58]. Palmerius [59] pointed out that “our sense of touch and kinesthetics is capable of supplying large amounts of intuitive information about the location, structure, stiffness and other material properties of objects” (p. 154). Dourish [60] noted that interaction with physical objects enhances cognition because tangible computing “is a physical realization of a symbolic reality, and the symbolic reality is, often, the world being manipulated” (p. 207).

The cognitive semantics theory of conceptual metaphor states that logic and reasoning are founded on image schemas formed by “patterns of our bodily orientations, movements, and interaction” that we develop into abstract references [56, p. 90]. Physical movement through space and interaction with tangible objects lead to symbolic representations [57]. Penny [60] pointed out that “the persuasiveness of interactivity is not in the images per se, but in the fact that bodily behavior is intertwined with the formation of representations… This interaction renders conventional critiques of representation inadequate, and calls for the theoretical and aesthetic study of embodied interaction.” (p. 83).

Physical interaction leads to new semantic models and new aesthetic interpretations. We gain new perspectives and see additional relationships based on our physical interaction with the objects. Abrahamson and Lindgren [59] noted that “we develop the skill of controlling and interpreting the world through the mediating artifact” (p. 4). For Piaget [61], logic and the cognitive processing of information are derived from physical and mental interaction, and it is the coordination of action that leads to reflective abstraction. With physical interaction, we use reflective practice to work through ideas rather than just think about them [55].

The significance of embodiment and physical actions in interaction design is summed up by Djajadiningrat, Matthews, and Stienstra [23] who felt “the philosophy of embodiment dissolves the mind-body distinction, rather than replacing the Cartesian priority of ‘mind over body’ with a similarly dualist priority of ‘body over mind’… Instead of a belief in mental models to successfully steer our actions, we may need to design for products that support the view that our understanding of the world springs from our bodily engagement with it” (p. 27). Interaction with the physical environment leads to the synthesis of sensory and cognitive information into abstract models that we use to interpret information and define the aesthetics of a design.

6 Cognitive Collages, Schemas, Memory

Sensory stimuli, interaction and embodiment, and experience are the building blocks for semantic and cognitive models. Research has also shown that the creation of semantic models is a dynamic process that changes over time. Hopfield [62] explored dynamic connectionist models where semantic representations change through interaction with other cognitive processes. These models evolve as an emergent process through learning [63, 64], and lead to complex cognitive models and personal interpretations of information called cognitive collages [65]. Turkle and Papert [66] referred to this type of cognitive mapping of diverse perceptual responses as bricolage.

In a non-linear, multisensory information space, layers of events and time, along with affective domains based on sensory experiences, provide a fluid, multisensory information space that supports reflection and the building of personal networks of associations and cognitive collages. These evolving sensory and cognitive models lead to the formation of complex schemas that help us interpret information [67] and define the aesthetics of an object or experience. Schemas play an important role in knowledge construction [68, 69]. As we learn how something works, we build new associations that draw on these experiences and create new schemas that are defined by the assimilation of information from other schemas [70]. For Moriarty, the “process begins with observation and then proceeds in a back-and-forth process of developing hypotheses and comparing the observations with information known and filed in memory” [71, p. 181].

Memory is an important dimension in aesthetics. Kinesthetic memory, derived from physical actions, contributes to the creation of sensory and cognitive models that define the aesthetics of an interactive design. Physical actions and movements create muscle memory or implicit memory that helps us learn and remember how to perform actions [72]. Costello [73] noted, “Therefore, we not only move our body in certain ways when we feel emotion, but also, in moving our body, we can, through this engrained kinetic intelligence, provoke memories of emotions and bodily sensation” [p. 258].

Memory also contributes to the abstraction and synthesis of information that enables us to draw on past experiences and intuition to interpret the meaning and aesthetics of an object or experience [74]. Kant [5] believed that imagination is also a part of cognition that helps us understand objects. He noted that we gain this understanding “from the influence of the senses, from the play of imagination, the laws of memory, the power of habit, inclination, etc.” [5, p. 194]. Imagination, like memory, synthesizes information so we can apply it. However, imagination adds another dimension to this process because it leads to the exploration of relationships and contributes to flexible models of interpretation and aesthetics.

7 Social Discourse

Online discussions and collaborations foster the development of social discourse that also defines the aesthetics of an interactive design by adding layers of mediated socio-cultural interpretation to the design syntax. This online interaction redefines the social and cultural context for information and leads to new interpretations and aesthetics based on diverse perspectives.

Ingold [75] highlighted the importance of multiple perspectives and the impact the “creative interweaving of experience in discourse” has on the perception of society (p. 285). Jenkins [76] pointed out that social discourse leads to designs that are “shaped by cultural and social protocols” (p. 133). Users expect and define new ways of interacting with each other, which leads to new approaches to interface design. Benkler [77] noted that decentralized methods of collaboration, along with the focus on sharing information in social media networks, lead to the creation of new “patterns of production” that are defined by social interaction protocols (p. 3).

French art critic Bourriaud [78] recognized the significance of social context in defining aesthetics in his book Esthétique Relationnelle (Relational Aesthetics). Bourriaud [78] considered relational aesthetics “a set of artistic practices which take as their theoretical and practical point of departure the whole of human relations and their social context, rather than an independent and private space” (p. 113). He emphasized the significance of the relationship the object or experience creates with participants which defines the “criteria of co-existence” [78, p. 109].

With online discussions and collaborations, multiple perspectives lead to a mediated discourse that transforms the meaning of signs and creates a fluid ontology and dynamic aesthetic experience. This type of flexible ontology reflects the following description of the dynamic nature of contemporary society as noted by O’Neill and Hubbard [79]: “If one accepts that mobility, flux and change are normal conditions of our contemporary world, then issues of becoming rather than being appear more in tune with the manifold process by which differences are materialised, embodied and experienced” (p. 47).

8 Conclusion

With interactive technology, the aesthetics of an interface design is defined by numerous factors including semiotics, perception, cognition, schemas, memory, interaction, embodiment, and social discourse (Fig. 1). In interactive programs, where information and events continually change, a discursive, multimedia experience leads to fluid sensory, semantic, and aesthetic relationships. Cross-modal perception and a metasyntax impact the interpretation of these dynamic relationships. Interaction and embodiment define the user’s physical relationship to the technology, causal relationships, and the functionality of the design. Each user brings a unique interpretation to the design because of past experiences which create cognitive collages and schemas. Memory and imagination synthesize these experiences into abstract models that define the aesthetic interpretation of the object or experience.

Fig. 1.
figure 1

Numerous elements, in addition to the audiovisual design, define the dynamic aesthetics of an interactive, multimedia program. Layers of sensory and semantic relationships continually change and contribute to the formation of complex cognitive collages. Copyright 2019 Patricia Search. All rights reserved.

Full size image

New technologies also impact the aesthetics of an interactive experience. Social media and networking software support social discourse and the creation of collective memory and pluralistic interpretations of interactive designs. Some technologies, such as artificial intelligence agents and cognitive filters, attempt to interpret and anticipate the user’s interests by preselecting specific information. These technologies may expand the user’s interpretation of the information or limit it by discouraging exploration.

Designers need to be aware of the sensory and cognitive restrictions in technologies and the multiple dimensions of interactive aesthetics, so they can determine the most effective ways to use different media and embodiment in interaction design. The audiovisual design is important, but it is not the only factor to consider. Usability testing generally focuses on evaluating the stylistic, affective domain of an audiovisual design and how the design impacts user engagement and performance. However, there are other factors in the aesthetics of the user experience that should be evaluated. We need to consider how the different dimensions of aesthetics discussed in this paper impact the user’s interpretation of the experience and develop research and usability tests to measure those factors. This research can help designers understand how to incorporate the different dimensions of aesthetics into engaging and intuitive interactive experiences.