Design Principles of a Mixed-Reality Shopping Assistant System in Omnichannel Retail

Featured Application

Mixed-reality shopping assistance system for omnichannel retail environments.

Abstract

New digital technologies furnish retail managers with new means to enhance consumer experiences in omnichannel retailing. Conceptual academic literature and industry emphasize the promising use of immersive digital displays and their potential benefits for retailers. In this research, we present the design of a personal shopping assistance system that is based on optical see-through mixed-reality technology. Microsoft HoloLens 2 was leveraged as the archetype to realize this novel system, facilitating consumer information search and decision making. The design incorporates various shopping assistance elements (i.e., product information, reviews, recommendations, product availability, videos, a virtual cart, and an option to buy). Users can interact with these elements with gesture-based inputs to navigate through the interface. A qualitative study with 35 participants was conducted to collect users’ feedback and perceptions about the mixed-reality shopping assistant system. Derived from the qualitative feedback, we propose seven design principles that aim to support future designs and developments of mixed-reality shopping applications for head-mounted displays in omnichannel retail: rigor, informativeness, tangibility, summary, comparability, flexibility and holism.

1. Introduction

The retail sector has been undergoing constant changes, including the transition towards omnichannel retailing, where retail firms are challenged to offer customers a holistic and seamless customer experience [1]. Omnichannel retailing aims to seamlessly integrate various retail channels, such as physical stores and online and mobile commerce platforms, into a single, seamless, and personalized shopping experience [2]. Traditional retailers have shown limited competence in providing these shopping experiences to contemporary customers, which poses the problem of deploying the omnichannel retail model [3,4,5]. One of the various solutions to push the omnichannel transition is implementing digital technologies such as immersive displays, wearables, and smartphones at different stages along the customer journey, i.e., the search, decision-making, purchase, and post-purchase phases, to provide a better shopping experience [6].

In the pre-purchase phase, customers seek information about the product, its functionalities, and the brand through online shopping assistant systems such as Google or Amazon [7]. Shopping assistant systems that leverage the functionalities of a digital technology to combine the functionalities of both physical and digital retail can deliver a holistic shopping experience along the pre-purchase phase and help them to facilitate the purchase. Even though smartphones are currently the most popular customer-owned device, they require an unnatural posture when interacting with digital or physical objects in the environment and require the engagement of hands at all times. This leads to poor focus and attention towards the surrounding environment, raises safety issues, and creates a poor experience [8,9]. Immersive display technologies such as mixed-reality (MR)-based head-mounted displays (HMDs) have the potential to overcome these interaction limitations through hands-free, heads-up interactions, and free movement while interacting with digital and physical objects [10]. These devices have been suggested as a technology megatrend that can create seamless consumer experiences across channels; several researchers have addressed this application of MR devices at the physical point-of-sale (PoS) or in the form of smart mirrors, for example [2,11,12,13].

In addition to the motivation in academia, the retail industry has started to embrace immersive technologies, with applications such as in IKEA stores [14]. Furthermore, immersive HMDs are expected to diffuse into the market for commercial use; the market revenue for immersive technology hardware and software will accelerate to USD 5 billion in 2023 [15]. Consequently, the increasing maturity and development of MR-based HMDs such as the Microsoft (MS) HoloLens have great potential to be affordable and adopted by consumers in the near future [16].

Hence, the first aim of this study was to demonstrate how MR technology can be leveraged in retail by proposing a novel MR-enabled shopping assistant system, using MS HoloLens as the archetype. The system is designed as a potential global shopping platform (such as Amazon) offering all types of products. Customers can benefit from the MR shopping assistant in a typical shopping situation (e.g., at the physical point-of-sale), on the go, or at home. The present literature lacks a clear understanding of customer perceptions towards MR-enabled shopping applications and specific design principles based on these perceptions. Hence, we posed the following research question: Which principles need to be considered in the design of an MR shopping assistant for omnichannel retail?

To answer this research question, we investigated users’ perceptions through 35 interviews after technology interactions in a laboratory experiment. Based on the user’s feedback and the collected literature to develop the MR shopping assistant, we provide a set of preliminary design principles for MR shopping applications that aim to offer designers a set of reusable solutions for future developments of such systems. The development of the design principles constituted the second aim of this research. This study aims to extend the literature around designing and deploying MR-based systems in the domain of human–computer interactions (HCIs) within information systems (ISs) and digital retail.

This research aims to address the lack of attention given to the application of mixed-reality (MR) systems in an omnichannel retail environment. This is a case of application and neglect spotting. Hence, the novelty of this research is the attempt to contribute towards these spotted gaps. The article communicates the design and the design process of an artifact, i.e., the MR shopping assistant system, which can help researchers and developers to design similar systems. The system is a prototype example of how MR can be deployed in an omnichannel retail context by using findings in the literature of web-based shopping assistant systems and mixed-reality systems in other contexts. The system builds over theories of research such as in [17,18,19,20] where this concept was discussed in brief, but a reproducible design process was missing.

Furthermore, the research proposes a set of concrete design principles that are built on the foundations of present literature as well findings of the research. These design principles can act as an important part of system building [20,21]. No such principles were found in the literature in the context of MR-based shopping assistant systems.

2. Background and Previous Work

2.1. Customer Experience and Shopping Assistant Systems

Omnichannel retail can be understood as the next phase of digital retail, where customers conduct their shopping journey in several channels simultaneously [22,23]. Holistic customer experiences have attracted attention in academia and industry in this regard, and have been described as a key concept of omnichannel retailing [2]. Customer experiences occur with indirect and personal responses at any retailer’s touchpoint, which customers may encounter at any shopping journey stage [24,25]. These stages address the customer’s search, evaluation, purchase, and post-purchase phases [26]. In addition to its holistic nature [26], the prevalent literature suggests that customer experiences should be regarded as “non-deliberate, spontaneous responses and reactions to particular stimuli within a specific context” (p. 637) [6]. Recent studies indicate that the context refers to the customer’s ability to process various stimuli at a given point in time, which supports the subjective and unique evaluation of any customer’s personal experience [27].

Among various solutions that aim to support retailers in providing seamless omnichannel experiences, shopping assistant systems have been suggested as promising tools that can be leveraged in online and offline retail environments [28,29]. These systems act as decision support by retrieving the customer’s shopping history, deploying detailed product information, suggesting real-time recommendations, and enabling the easy comparison of alternative products [30,31]. Moreover, shopping assistant systems have been developed for various customer-facing devices. For instance, mobile shopping assistance applications can be downloaded to customers’ smartphones and used at the PoS to locate specific products in the store and extend social content into the physical world [28]. In contrast, web-based shopping assistant applications refer to Amazon, Groupon, and Google Shopper, providing product information and reviews from customers who have already purchased and experienced the item of interest [7]. Immersive technologies have also been deployed in retail environments as shopping assistant systems [32]. Previous examples include systems that enable customers to virtually try on clothes or accessories (e.g., sunglasses) in online shops [31,33] or in physical stores where customers can utilize smart mirrors or virtual fitting rooms [29,34]. These studies have shown that immersive technologies enhance the decision-making process by enabling trial and imagination of the products in the form of a digital twin and enhance the customer’s overall shopping experience [29,31,33,34].

2.2. Mixed-Reality Technology

The ‘reality–virtuality continuum’ developed by Milgram and Kishino [35] embodies display environments, ranging from a real to a completely virtual environment, as shown in Figure 1. According to the continuum, MR lies between the two extremes, encompassing augmented reality and virtuality [36]. The authors define MR (also known as hybrid reality) as merging the real world around us with an entirely 3D-envisioned virtual environment by introducing computer-generated elements into the real world [35]. This can be achieved using devices that overlay digital content on top of the physical world, such as Microsoft’s HoloLens.

Milgram and Kishino [35] classify MR into six different display environments, which can be used to distinguish different types of immersive contemporary environments. Virtual reality (VR) can be understood as a deployment of the “virtual environment” extrema of the RV continuum where the user is fully immersed in digital space without any awareness of the physical world. VR devices include HMDs such as Oculus. Augmented reality (AR) refers to the overlay of digital content on top of the physical world. This can be achieved using devices such as smartphones or tablets, which display digital content on top of a camera view of the real world. This technology is a manifestation of class 4 displays, where a “video-see-through” is created to make the user aware of the physical world around them. Different terminology has been used to describe immersive environments: extended reality (XR), which is an umbrella term that refers to the broad spectrum of immersive technologies, including AR, VR, and MR. This includes any present or future device that lies on the RV continuum.

In this study, we focused on optical-see-through MR HMDs. These devices are equipped with see-through capabilities where computer-generated graphics can be optically superimposed onto the real world [35]. Flavian et al. [37] described this technology as pure mixed reality (PMR), characterized by the integration of digital objects into the real world so that they are indistinguishable from real objects, as shown in Figure 1. The authors also introduced an updated reality–virtuality continuum in which all realities appear independently, including PMR. Among immersive displays, recent studies have stated that MR-enabled HMDs have great potential to be used for retail purposes and will change the face of retail in the next ten years [2].

2.3. Mixed-Reality Shopping Assistant Systems and Design Principles

A shopping assistant system which leverages the power of data and analytics to tackle customers’ needs is an optimal tool for retailers to compete in omnichannel retail [38]. Moreover, MR-based shopping assistant systems have been suggested for use in omnichannel retail environments to enhance the customer experience [39,40]. Several studies specifically used optical-see-through MR devices such as MS HoloLens for retail applications to provide utilitarian and hedonic value to the customers. Earlier studies used MS HoloLens to design an in-store recommender system that recognizes the products in stores and displays relevant recommendations [17,18]. Along with the digital recommendations, the authors highlighted features such as image recognition and immersive digital user interfaces to be used in MR-based shopping assistant systems. Similarly, Fuchs et al. [39] used the MS HoloLens to develop a system that can identify packaged products in retail environments. Cheng et al. [19] developed a shopping assistant application using continuous context awareness and natural interaction techniques (NUI) and integrating digital information in the user interface. These MR applications are developed by extracting requirements from the literature and matching them appropriately by leveraging the capabilities of the device and advancements in the field [18]. Hence, we propose that the general concept of an MR shopping assistant system can be understood as a system where the MR device is used as a medium between the user and retail ecosystem.

The MR device is used to collect data while also producing output for the user. The back-end computational system device synthesizes real-time data in addition to the data from the retailer to enhance the shopping journey of the user in the form of different shopping assistance elements that can be accessed by the user through the MR device. This concept is depicted in Figure 2.

These developments further use design theory as the basis to develop prototypes as proofs of concept. In design theory, design principles are considered the foundation that aims to provide a “systemization of knowledge” to support industry practitioners and researchers to develop optimal solutions [41,42]. These principles can also serve as guidelines for developers to support the design of artifacts by describing the standards of a good interactive prototype or system [21]. As supported by [20], standardized tools in the field of MR-based omnichannel retail systems are lacking, and no specific design principles have been found by the authors towards this.

Hence, the proposed shopping system in this study was built on requirements extracted from prior studies. These findings were combined with feedback from users to propose design principles for future iterations. These principles do not aim to give an exhaustive account of all the required components in the development of an MR shopping assistant system, but present a study that extracts components from the literature and combines them with comments from the users to help future research and development of similar systems. At the core of the system’s purpose is to assist customers along their omnichannel shopping journey by providing utilitarian and hedonic value, consequently enhancing their shopping experience.

3. Research Methodology

To answer to the aforementioned research question, the design science research (DSR) methodology was utilized.

Design science research is a methodology for conducting research in the field of information systems, first proposed by Hevner [43]. DSR aims to create new and useful artifacts, such as software systems, in order to solve problems and improve the functioning of organizations. DSR involves the creation and evaluation of artifacts such as constructs (terms and symbols), models (abstractions and representations), methods (algorithms and practices), and instantiations (implemented or prototype systems) [44]. These artifacts are designed to meet identified business needs and enhance problem-solving and organizational capabilities using intellectual and computational tools. Theories about the application and impact of these artifacts are developed as they are created and used [7,41,42,43,45,46]. In 2007, Peffers [46] proposed the design science research methodology (DSRM) process model, which is a specific model for conducting DSR. The DSRM process model is a six-phase model that includes the steps shown in Figure 3 that are used as the methodological framework for our research. These steps are followed through the research, while the article reports majorly the design and development of the intended artifact.

DSR is a suitable methodology for the current intended research because the DSR literature talks about the role of instantiations and the specification of design theories for methodologies and interventions as well as for products and applications. The important aspect addressed by the authors is the critique of the word ‘theory’ in design-based research, and consequently bridging the gap between engineering, design, and knowledge creation in an academic context.

Section 1 and Section 2 briefly report the results of the first two steps of the process i.e., identifying the problem, and defining the objectives of a solution. Here, we developed an understanding that an MR-based shopping assistant can help contribute to the research gap. The design, development, and demonstration of the artifact are the core of the research article and are described in the following Section 4. Here, a requirement extraction-matching process was followed to design the system in a systematic manner. The evaluation of the prototype was performed using qualitative semi-structured interviews, and the results are communicated by defining a set of design principles that can be used to design future systems. Since DSR is a methodology that is based on iterations, these design principles can be used to design the next iteration of the system.

4. System Design

The proposed MR shopping assistant was designed for MS HoloLens, one of the most advanced headsets on the commercial market [33]. Other commercial optical see-through headsets such as Magic Leap [47] and Meta 2 [48] need to be connected to a processing system such as a personal computer; however, MS HoloLens [49] works as a portable standalone computer. In mid-2019, Microsoft launched the second generation of its HMD, MS HoloLens 2, which is equipped with eye- and hand-tracking technology, among other new features [50,51,52]. The new generation also provides a wider field of view, less weight, and promises a more natural user interaction than the first generation. MS HoloLens 2 uses holographic display technology, which projects computer-generated digital objects in real time into the user’s physical environment. Users can interact with these digital objects using specific hand gestures to simulate a natural interaction with physical objects [40]. Thus, the device creates an environment where digital and physical objects coexist and interact [51]. With these elements, the proposed MR shopping assistant aims to provide utilitarian as well as hedonic value to the customer’s shopping experience [53] throughout the product search, evaluation, and purchase phases.

To design the MR shopping application, we adapted elements typically used in electronic shopping platforms [54] and combined them with the advances of MR-enabled HMDs [55]. As stated in the previous section, we first defined requirements that are translated from an understanding of the problem space for a solution artifact [56].

4.1. Requirement Extraction

The first requirement (R1) refers to the need for the automatic recognition of physical products enabled by the MR device, MS HoloLens. This would enable the system to detect the product of interest and provide assistance accordingly, thus offering a personalized and tailored experience [17]. Cheng et al.’s [19] work supports this requirement by identifying continuous context awareness as an advantage of MR HMDs that can be leveraged in immersive shopping experiences to enhance the shopping experience. Mora et al. [18] further stated that MR-enabled shopping assistants should be able to detect products in the user’s field of view. This functionality aims to provide an alternative to customers inputting the name or model number into the system. In comparison, automatic recognition aims to reduce query failures that occur if ambiguous terms are used, or the input is incorrect and cannot search for the product. This further prevents users from time-intensive search processes such as scrolling through long lists of product keywords [57].

Followed by the recognition of the product of interest, the MR shopping assistant is required to provide information that is not currently available in physical stores, but is in online resources, bridging the gap between digital and physical retail [19]. We extracted seven major design features in the proposed MR shopping assistant system, which define the second requirement (R2): product information, reviews from previous customers, recommendations for alternative or additional items, a commercial video, the shopping cart, the buy button, and the product’s availability. These features should be integrated into the system in the form of a user interface that can be used by the customers. These features are derived from e-commerce platforms [50] and are described in

Table 1. Design features for an MR shopping assistant.

Design Feature	Purpose	Description
Product information	Provides relevant information about the product’s features and characteristics.	Shoppers seek information about their purchases through online and offline channels, which encourages digital applications to provide this information [59]. Product information is one of the critical components for customers’ intentions to use shopping assistants [7].
Reviews (textual and star rating)	Supports decisions by anecdotal comments from fellow shoppers.	Reviews “are viewed as valuable because peers who have no interest in positively biasing the content presented on application, provide the product reviews” [7]. Reviews enable an opportunity for the shopper to evaluate the product based on prior shopper experiences. Consequently, this helps customers in the decision-making phase.
Recommendations	Helps to find similar, supplementary, or additional products.	Recommendations provide an opportunity to discover new products [18]. These recommendations can be tailored according to the user’s preferences, providing a personalized shopping experience, and finding preferential products.
Videos	Enriches product information with embodied clips about using the product, user instructions, reviews, unboxing, etc.	Video-sharing platforms play a significant role in shaping customer perception and decision-making processes [60]. Previous studies also used videos as metadata for products in retail environments and even for AR retail applications [61]. These have been used in various forms, such as product advertisements and reviews from other customers environments [62,63].
Availability	Provides information about online or on-shelf stock in the vicinity of the user.	Prior research advocates integrating offline inventory in online media to provide customers a better shopping. experience [64].
The buy button	Enables the completion of the purchase journey on the spot or home delivery.	Conducting the actual purchase is an integral part of digital retail and e-commerce platforms [65,66]. This is performed due to crises of immediacy [67], where the customer needs to be addressed in real time. Hence, this feature would allow the shopper to add a product to a virtual shopping cart and continue shopping towards the final purchase either digitally or at the POS.
Virtual shopping cart	Allows users to add and delete products and review the shopping list before payment.	The cart gives the user a sense of security by introducing a two-step process before payment to make the final purchase decision with more confidence. The shopping cart secures online price promotions, obtaining more information on specific products, organizing shopping items, and entertainment [68].

[58].

The third requirement (R3) refers to the need for natural user interactions (NUIs) within the system’s user interface and the design features defined in R2. MS HoloLens 2 facilitates NUIs by providing instinctual multimodal interactions. Facilitated by hand recognition technology, customers can apply hand gestures to interact with 3D buttons. An NUI interface aims to provide users with experience in a hedonic sense by making it fun, enjoyable, easy, and intuitive to control and interact with the system [48]. The desired NUI interface provides a hedonic experience without compromising the utilitarian aspect of performing the shopping task (i.e., goals). The three requirements are summarized in

Table 2. Extracted requirements for an MR shopping assistant.

Problem	Requirements		Description	References
Product (information) needs to be found in other sources (e.g., online search engines).	R1	Recognizing the product	The system is required to identify the product of interest.	[18,19]
Receiving product information and conducting the purchase on various platforms or the physical POS. Possibility of being out of stock without receiving prior notice.	R2	Content design elements	The system should integrate elements that provide meta-information about the product of interest: product information, recommendations, reviews, videos; and assist the shoppers in the purchase process: availability, virtual shopping cart, and a button to buy.	[50]
Increasing the hedonic and utilitarian value of the shopping experience with immersive devices.	R3	Natural user interaction interface	The system should be able to provide an immersive user interface that integrates the content design elements and can be navigated using NUI techniques.	[69]

.

4.2. Requirement Matching

In the next step, the defined requirements are matched by leveraging the capabilities of the MR HMD. The following paragraphs outline information about the corresponding technical development.

4.2.1. R1: Recognizing the Product

The system is programmed with Unity [70] and MR Toolkit (MRTK) [71]. It uses image recognition technology and the Vuforia software development kit [72] to identify the product of interest; Vuforia-based image recognition systems “are able to detect objects with maximum accuracy” (p. 479) [73]. As shown in Figure 4 (left), the product of interest is recognized once it enters the user’s field of view. More specifically, the system identifies the product’s image, which had previously been added to its database.

4.2.2. R2: Design Features

The design features are projected as 3D buttons, as shown in Figure 4. The corresponding digital objects are anchored onto the user’s field of view once the product of interest is recognized. These digital objects can be controlled using hand gestures. The interface includes four 3D buttons of relevant product meta information, as presented in Figure 4.

• Information—Four information windows are integrated into the user interface. The ‘next’ and ‘previous’ buttons can be used to switch between the windows. The information is divided among four elements to decrease information overload in the interface and give the user the feeling of control over the interface.
• Review—The review interface shows text-based content as well as a typical five-point star rating. The text displays first-hand comments about the product from fellow customers. Reviews give an impression of the product’s quality and assist customers in the decision process [59,74].
• Recommendation—The recommendation interface presents 2D images of similar, supplementary, or additional items. The developed MR shopping assistant system displays two recommendations per product to minimize the user’s head movement and enhance the readability of the text.
• Video—A video screen is displayed in the center of the user’s field of view and scaled to the product’s width (as indicated with the white rectangle in Figure 4). When users touch the video element with the respective touch gesture, the video toggles between play and pause mode. The integrated videos are sound-equipped and provide users with a multimedia experience.
• Availability—The availability element includes a slider element and is named ‘find it near me’. The MRTK facilitates this feature by continuously changing a certain value when the slider is moved along a line. This allows customers to increase or decrease the radius around them to find and collect a product. It also shows the distance to various stores and the number of available items at the location.
• Buy button—The buy button can be pressed to conduct the purchase. When the user presses the buy button, a pop-up message with “added to cart” appears as visual feedback that affirms the user’s action, as shown in Figure 5.
• Virtual shopping cart—The virtual cart feature is designed towards achieving gamification, which can help users engage and improve their overall shopping experience [75]. The hand gestures used in this interaction are explained in the next section: R3. The shopping cart is permanently accessible with the hand menu gesture. When the user opens their hand in the FOV, a rotating sphere appears in the middle of the user’s open palm. When the sphere is touched with the other hand, using the touch gesture, the cart window renders in front of the user (see Figure 6, right). With the drag gesture, users can grab, move, rotate, and resize the cart window. Customers can scroll through the list of items that have been added to the cart. The items on the list are displayed with their respective quantity and price. The window also shows the total price of the purchase. As shown in Figure 6 (right), the button with the pin symbol allows users to change between a “static” or “follow me” mode. Pinning the virtual shopping cart means selecting the static mode, which forces the cart window to stay in a specific cartesian position in relevance to the environment. In the “follow me mode”, the shopping cart window follows users as they move around the shopping environment. The system tracks the position of the user and adjusts the position of the cart window accordingly. Consequently, the cart window stays in the user’s field of view and can be reached by near-touch interaction.

4.2.3. R3: Natural User Interaction Interface

The interface was developed with a basic color scheme, size references, and fonts from the standardized toolkit (i.e., MRTK). The hand gestures that facilitate the NUI in the application are depicted in Figure 7.

• Touch—The deployed digital buttons can be pressed using the touch gesture. A touch curser appears as a white ring on the tip of the user’s index finger. The user can then select an element by tapping on it. Scrolling is performed by swiping on the element’s surface, comparable with control gestures on touchscreen monitors. The touch cursor is hidden when the user’s hand is not close to a digital object. Instead, hand rays appear from the palm of the user’s hands.
• Air Tap—Users can select distant objects with an air tap gesture [76]. As soon as the user selects one of the content design elements, the application opens a window displaying its content, as shown in Figure 7. This interaction technique is based on the instinctive action of pressing a physical button, hence providing the users with an easy-to-learn interface navigation.
• Drag—A pinch slider can be moved by grabbing the slider, either directly or at a distance, and using the drag gesture. In the developed MR shopping assistant, the slider is used for the content design element availability to select the maximum distance from the user’s location.
• Hand Menu—The hand menu gesture is used to open the virtual shopping cart element. As soon as users put their right palm flat in the HMD’s field of view, a rotating sphere will appear in the palm of their hand. The Require Flat Hand feature is deployed to prevent false activation of the system. The system tracks the palm, and the sphere moves along with the user’s palm.

Now, the overall flow of the system can be understood, as shown in Figure 8. The real-time data from HoloLens help the device to carry product and gesture detection. Once a product or a gesture is recognized by the device in its field of view, the corresponding function is triggered, as mentioned in Section 4.2.1, Section 4.2.2, Section 4.2.3. The data from the HoloLens also get fed to the data processing unit, which along with the retail data, facilitate personalization for the user. The ultimate aim of the system is to display interactable shopping assistance elements. The user interaction with the system is shown in Video S1.

5. Experimentation Methodology

The customer perceptions of the designed system were elaborated by an empirical study performed in a laboratory environment in Germany for five consecutive days in August 2020. Respondents were recruited through random sampling using broadcast messages and open participation calls on social media. Random sampling has several advantages in this context as it helps to ensure that the sample is representative of the larger population being studied. By randomly selecting participants from a population, researchers can increase the likelihood that the sample will accurately reflect the characteristics of the whole population. It further helps to reduce bias in the selection of participants. If the participants are selected using a systematic or nonrandom method, there is a risk that the sample will be biased in some way. For example, if researchers only select participants who are available at a certain time of day, their sample may not be representative of the population as a whole. Random sampling helps to increase the precision of the results by reducing the likelihood that the results will be influenced by extraneous factors. This can provide confidence in the validity of the results and increase the generalizability of the findings.

The sample consisted of 35 participants (18 male, 17 female) from 11 different nationalities (16 participants from Germany). Ages ranged between 17 and 38 years old (Mage = 26.68, SDage = 4.25). When asking respondents about their general level of experience with MR devices, approximately half of the sample group (52%) had used an MR device multiple times before or used them regularly. Figure 9 shows the distribution of MR experience among the sample group.

Informed consent was obtained prior to the experiment. Due to the COVID-19 pandemic, the data collection procedure strictly followed the general safety precautions. The experiment started with approximately 20 min of interaction with the MR shopping assistant and was followed by a 15 min interview. After finishing, each participant received financial compensation of EUR 15. Participants were assigned to one of two shopping scenarios—”in-store” or “hotel lobby”—maintaining a gender balance. Except for the short scenario description, the experimental setup, the task, and the selected products were equal for all participants. Users were allowed to ask questions while interacting with the device.

The sample group was asked to browse through the experimental setup, which consisted of 13 different products presented on a table in the middle of the laboratory. The products represented a variety of items from three different categories: search products (i.e., pack of coffee beans, pair of sneakers, pack of milk, LED screen, pack of soft drinks), experience products (i.e., pack of beer, video game disk, a bottle of rum, pack of chocolates, a tech magazine), and services (i.e., insurance, law service, business consultancy). Search products referred to tangible items whose quality could be determined before consuming the product, while experience products can only be evaluated after using or consuming the item [77,78]. Service products were defined as intangible items, such as one hour of consultancy service, represented by company logos printed on physical paper in the experimental setup.

The shopping task was based on general browsing, as adopted from the study by Venkatesh et al. [7], where shoppers were not given a predetermined goal to shop for any specific products [79]. The general browsing task created a context where customers utilized the proposed technology to gather information independent of a current need within a specific category [79]. The subsequent interview aimed to gain feedback on the system’s design, usage perceptions, and customer experience. The semi-structured interviews lasted between 30 and 40 min after the participants’ interactions with the MR shopping assistant.

Table 3. Questions for the semi-structured interviews.

Question Number	Semi-Structured Interview Question
Q1	How was your overall experience with the MR shopping assistant?
Q2	Were there any pain points when using the MR shopping assistant?
Q3	If you could change something about the system, what would it be?
Q4	How would the MR shopping assistant benefit your overall experience?

lists the four questions addressed to the participants.

6. Results

The transcriptions of our interviews with 35 participants were each approximately five pages long. Following the concept-driven coding approach of thematic coding analysis [80], the two first authors of this study separately categorized the interviewees’ semi-structured responses into future development suggestions, as well as positive and negative feedback. Next, the data were structured according to the defined requirements (R1–R3). This coding scheme was applied in MAXQDA Analytics Pro 2018 (Release 18.2.5, Build 2019) [81] with a respective folder organization and a color identification system.

The consistency and accuracy of the application with the codes were evaluated based on four rounds of code cross-checking [80] in online meetings. The applied coding was compared between the researcher’s individual results. One main development of the discussions was to separate the negative feedback from suggestions for improving existing content design elements. After using the improved coding scheme, we compared the interpretation and calculated the level of concordance, which reached 80% over the course of the meetings. Finally, the authors reconciled differences and agreed on one final version of the coding results. There were no observable differences in the present analysis between age, gender, and nationality.

In their responses to the opening question (Q1), 29 (83%) participants gained a positive impression about their experience with the MR shopping assistant. It was described as “great”, “very interesting”, “cool”, “impressive”, and “exciting”. First-time users were particularly fascinated by the integration of digital objects into the actual world. One said: “It feels so unreal to see, it’s really cool. And it’s also fun” (#8, female, 25 years). In total, four of the six participants (17%) who had a negative impression from interacting with the MR shopping assistant mentioned hardware factors such as the weight of the device as contributing towards the negative perception. Two participants reported having simulator sickness in the form of a headache after wearing the HMD for 10–15 min. The system’s design and usability were mentioned by two (6%) participants as a negative effect on their overall MR shopping assistant evaluation.

6.1. Recognizing the Product (R1)

In total, 16 (46%) participants identified problems with the sensitivity of product recognition. Participants criticized that they had to stand at the correct distance in front of the product and turn their heads in a specific position to see the digital objects. Otherwise, the content design elements would not be shown at all or quickly disappeared. One respondent said: “All the information was right there. But when I moved my head a little bit, they were not always moving with me”. (#20, female, 25 years). In contrast, 19 participants (54%) did not mention any problems concerning the system’s recognition of the products.

6.2. Design Features (R2)

Participants gave feedback on the design features and suggested low-level improvements about them for future iterations of the system. Most participants (24/69%) appreciated the product information displayed in the MR shopping assistant prototype, including statements such as “I liked the additional information offered so I could get a better understanding of the product” (#31, male, 31 years). Moreover, participants emphasized several significant suggestions for future development cycles in the context of product information. First, eight (23%) customers stated that the system required product pictures from various angles or 360-degree views in addition to existing textual information. One criticized: “There are no photos of the products. So usually, when you buy something, you would check the photo, what it looks like, zoom in to see the design and stuff like that. So that was not there” (#5, male, 38 years). Second, fifteen participants (43%) were interested in viewing the product as a 3D object (such as to test the fit of new furniture in their apartment or virtually try on new clothes). As one subject said: “The best would be to have a virtual clone of the product, that you can turn, see from all angles; like a 3D model of it” (#40, male, 36 years). More color variations of the products (e.g., for sneakers) in the form of 3D digital objects or pictures could be an extension of such a feature, as requested by 5 of the 35 participants (14%). Third, some subjects (17%) would have preferred more product-specific details, such as calorie information and the expiration date for food or warranty conditions for electronic devices.

Four (11%) participants missed an average star rating of the reviews given byim previous customers. One participant stated: “I usually click once to get the reviews of the product. So, there was no overall average of the reviews” (#8, male, 30 years). Furthermore, product pictures or videos from other customers, in addition to text-based reviews, would be of interest to see if the product looks like the retailer presented it (#24, male, 34 years).

In total, 15 participants (43%) raised the relevance of recommendations, especially in combination with comparing products with each other. The integration of similar products, respective prices, and reviews would represent an additional feature of the presented MR shopping assistant prototype. One participant added: “If you could provide the price range of the recommendations, that would give customers a better idea of alternative products” (#19, male, 27 years). Two participants (6%) communicated their concerns about the authenticity of recommendations and whether those would be controlled/pushed by the retailer who offered the application or specific data algorithms.

Half of the sample group (17/49%) highlighted the video embedded in the middle of the user’s field of view. Of those, 11 participants (31%) found the video content useful and entertaining. One participant said that “The video information is the best one because people tend to watch videos for information generally. Like unboxing videos are very important to me” (#3, male, 32 years). Six users (17%) raised the prospect of offering a variety of videos. Two (6%) participants mentioned that they would not use the video feature for everyday products (search products), but only for experience or service products.

The product availability element convinced 10 (29%) users to mention the advantages of easily finding a product nearby. One participant highlighted: “So it was good to find different stores near you […] the information about it and the price” (#6, male, 30 years). One participant suggested adding a time estimation for product delivery. Another participant recommended an integrated navigation system that would guide customers to the nearest store.

According to three (9%) customers, the system should provide a functionality to shop offline and deliver later. In total, 40% (14) of the sample group found it easy to put items into the virtual shopping cart by pressing the Buy button within the MR shopping assistant. As one subject said: “Add-to-Cart was really easy. I think it happened flawlessly almost every time” (#32, female, 25 years). The virtual shopping cart could be permanently fixed into the user’s field of view; thus, five users (14%) highlighted the advantage of having the shopping overview permanently available. Three (9%) participants said that they liked to open the cart with the ball. One of them said: “I felt like a superhero” (#38, male, 25 years).

When asked how the MR shopping assistant would benefit their overall shopping experience (Q4), most participants (31/89%) approved that having all necessary information in a single field of view helps make a purchase decision (11/32%). Moreover, six (17%) respondents said that the MR shopping assistant would quickly provide additional information at the POS. One said: “And even if there is not any person to help, the system gives sufficient knowledge to the user” (#30, female, 30 years). According to five users, the search process and purchase process would be easier and faster (5/14%), although two (6%) participants stated that it would be rather useful for nonshopping situations, such as advertising.

6.3. Natural Interaction Techniques (R3)

Twelve (34%) participants found the system particularly easy to use, although six (17%) participants criticized that the overall interaction was not easy and intuitive. An additional nine (26%) users had difficulties pressing the buttons or using the slider, which added items to the shopping cart (3/9%). Nine (26%) users would have liked to have a wider field of view or be able to minimize content. The same feedback applied to the size of videos (5/14%). In total, 12 (34%) participants reported unfamiliar interaction techniques with the MR shopping assistant, and that it required some time to become accustomed to it. One said: “I think, in the beginning, you need some time. It works if you’ve got a feeling for it. And at the end, It’s simple” (#22, female, 26). Five users (5/14%) criticized the required head movement to see all the information displayed within the field of view. One participant stated: “When I’m moving the head, the physical object is not always moving with me. It would be nice to zoom in or out” (#20, female, 25 years). Other than this, 10 participants (28%) would have liked to redesign/customize the tool, mentioning features such as a selection of languages.

As a result of the qualitative study,

Table 4. Results from the qualitative interviews.

	I/A/O	Design Features for Future Iterations	N	%	Subcategory
Recognizing the Product (R1)
Product Recognition	O	Recognizing the product using image recognition	-	-	PR
Design Features (R2)
Product Information	O	Text-based information	-	-	PI
	A	3D models of the products to project in the real environment	15	43	PIA
	A	Product pictures with 360° view	8	23	PIA
	I	More detailed product information for specific products (e.g., calories and expiration date for food)	6	17	PII
	I	Product variations (e.g., colors)	5	14	PII
Reviews	O	Product reviews with star ratings	-	-	R
	I	Average star rating of reviews	4	11	RI
Recommendations	O	Recommendations based on the product of interest	-	-	RC
	A	Comparison of similar products (incl. description, price, reviews)	15	43	RCA
Videos	O	Video based on the product	-	-	V
	I	Adjustable size of videos	5	14	VI
	A	Choice among several videos	6	17	VA
	I	Controlling volume of videos within the application	2	6	VI
Product Availability	O	Showing availability of the product	-	-	PA
	A	Option to shop offline and receive delivery later	3	9	PAA
	A	Starting navigation to the next location near me	2	6	PAA
Buy	O	Option to buy the product	-	-	B
Virtual Shopping Cart	O	A virtual shopping cart for the user	-	-	VC
Natural Interaction Techniques (R3)
NUI	O	Implementing natural interaction techniques
Field of View	I	Zoom option (i.e., minimize, maximize)	5	14	NI
Color	A	Customizable color scheme	10	28	NA
Language	A	Choice of multiple languages	1	3	NA
Total			35	100

provides recommendations for future iterations of the MR shopping assistant for MS HoloLens. Suggestions are categorized according to the content design elements and indicate whether it represents the original design feature of the prototype implemented in the research (O), improvement for an existing design feature (I), or an additional feature (A). The number and percentage of mentions in the case of improvements and additional features are also depicted.

The extracted features from the user suggestions are comparable to the present literature, as several authors have mentioned similar features and concepts in the case of shopping assistant systems. Considering the user suggestions, most of the suggestions are in line with earlier research towards deploying shopping assistant systems, but the present research validates these for MR systems.

Three-dimensional models and 360° view pictures of retail products have been adopted previously in web-based shopping applications. They have also been a topic of research interest to deploy such practices in mobile-based AR [82,83]. As the users also suggested more detailed and product-specific information along with product variations, it validates the idea of including a large amount of information in shopping assistance applications that can communicate different aspects of the product, including more tangible aspects such as the shape, size, and color [83]. The impact of user reviews along with average ratings by fellow shoppers have been studied by authors [84]. The “binary bias” pointed out by the authors [80] is in line with how the users in the study expressed their desired shopping journey. Product recommendations and comparison as a useful shopping assistance element has been proposed in mixed-reality systems [17,18], where the authors also outlined the importance of personalization and flexibility in an omnichannel shopping journey. This is reflected in the results on a user interface level, as users suggested control over features such as the adjustable size of videos, volume, zoom, color scheme, and languages. A customizable user interface has been suggested in the literature by authors such as [85]. The user suggestions such as the option to buy offline and receive the delivery later, and the navigation to the next store aligns with the concept of omnichannel retail, since it embodies the principle of integrating online and offline channels of retail where a customer can design their own shopping journey based on different touchpoints [22,23]. These features collectively contribute to the list of features proposed by previous authors [7,17,18,19] conceptualizing shopping assistant systems with novel technologies such as MR.

Using the grounded theory approach by Strauss and Corbin [86], as shown in Figure 10, the codes were utilized to form subcategories of the features. Table 4 depicts how the subcategories were formed based on the original design feature, an improvement or an addition. For example, in the case of product information, the original feature was put in the category “PI”, an improvement (I) in the feature was put in the category “PII”, and an additional feature was put in the category “PAA”.

The subcategories were further categorized based on what principle of design they contribute towards. The authors identified seven principles that capture the concept of the features. The subcategories often contributed towards more than one identified concept, as shown in

Table 5. Categorization of the design features.

	Rigor	Informativeness	Tangibility	Summary	Comparability	Flexibility	Holism
PR	x						x
PI		x		x	x		x
PIA		x	x		x		x
PII		x	x		x		x
R		x			x		x
RI		x		x	x		x
RC					x	x	x
RCA					x	x	x
V		x		x	x		x
VI		x		x	x	x	x
VA		x		x	x	x	x
PA		x			x		x
PAA		x			x	x	x
B						x	x
VC						x	x
N						x	x
NI		x	x			x	x
NA						x	x

.

7. Description of the Design Principles

We analyzed users’ suggestions and feedback reported in the previous section. Consequently, these were combined with the original requirements and design features of the MR shopping assistant. As a result, seven design principles are proposed that can help develop MR-based shopping assistant systems in the future.

• Rigor: As mentioned in Section 4.1, product recognition offers users a certain amount of accuracy. This makes the system rigorous by detecting the correct product of interest as compared to the manual entry by the user [57]. This feature was received well by the users, and no major suggestions were identified. Although users mentioned the sensitivity of interaction, this can be optimized in the future with advancements in image/product recognition technology. The current state-of-the-art tool, Vuforia SDK, is used by developers to deploy product recognition in the system and provide a rigorous application that can help enhance a user’s customer experience. Hence, MR-based shopping assistants should provide a rigorous system that minimizes error and consequently enhances the users’ shopping experience.
• Informativeness: Prior studies in e-commerce suggest that customers seek information in shopping assistant platforms to make buying decisions [59]. This is one of the most important features that has been used in many different channels of e-commerce. With the advancement of the Internet and other information sources, customers seek more diverse information. In addition to traditional text-based information in physical and web-based retail platforms that communicate the price, brand, etc., customers consider information such as reviews from fellow customers, online videos, off-shelf availability of the product, and information that is specific to the category of products, such as dietary information for food products. These findings suggest that an MR-based shopping assistant system should integrate a diverse set of meta-information elements to help customers make better buying decisions.
• Tangibility: An MR-based shopping assistant system aims to integrate different touchpoints in retail. Tangibility is one of the features that MR technology can leverage in the retail sector. Digital platforms based on the web or smartphones lack in conveying tangible information. An MR system can overcome this limitation by deploying 3D objects in physical environments that are rendered according to their physical properties. The results of this study reflect that customers seek features such as a 360° view, 3D models of the product, etc., to better understand the product.
• Summary: This principle correlates with the crisis of immediacy [67], where customers seek assistance in real time and as quickly as possible. Many customers have hinted towards faster options to seek all the necessary information in a concise manner, e.g., a summarized rating of stars, a comparison between different products, etc. This enables making faster buying decisions for products that are more utilitarian in nature and do not need a large amount of information. This can also apply to products that do not need to be used to be judged on their quality [7].
• Comparability: The retail market offers a huge stock of products; therefore, customers seek to compare products to make better buying decisions. Hence, a shopping assistant system should be able to provide these options to help make informed decisions. Examples can be comparisons between items in terms of color, price, and reviews, or offer similar products in terms of recommendations.
• Flexibility: The results show that users seek flexibility or customization options in their shopping experience and the shopping assistant interface. This includes changing the language, customizable color schemes, and the option to resize windows. Customers also expressed the importance of having options to design their shopping journey, such as shopping offline or choosing different delivery options. A flexible and customizable shopping journey puts the customer in charge and delivers a personalized shopping experience, which is what customers seek in an omnichannel environment [3].
• Holism: Omnichannel experiences comprise both hedonic and utilitarian values [23]; therefore, it is important to ensure that MR applications provide users with an enjoyable and useful experience. The developed tool comprises a seamless experience that deploys features such as ease of use through NUI, an opportunity to discover new products through recommendations, gamified elements such as the virtual shopping cart used, multimedia information such as videos, and shopping with the minimal use of limbs in navigating the interface. These examples could be deployed in future applications to create holistic, seamless, and integrated shopping journeys.

The proposed design principles aim to offer designers a set of reusable solutions for future developments of such systems. The proposed design principles represent a combination of features used in typical e-commerce and features that are specific to MR. These principles define a set of high-level explanations of design features extracted from the literature and later reviewed by potential customers. The principles are summarized in

Table 6. MR shopping assistant system design principles.

Design Principle	Description
Rigor	The system should be able to provide a seamless shopping experience by minimizing query failures and optimizing search processes.
Informativeness	The system should integrate a diverse set of meta-information elements about the products.
Tangibility	The systems should be able to communicate the tangible properties of a product.
Summary	The system should provide the information in a concise form that can help customers to make faster shopping decisions.
Comparability	The system should facilitate the comparison of similar products.
Flexibility	The system should facilitate customizing the application.
Holism	The system should provide a holistic experience by integrating hedonic elements and introducing natural interaction techniques.

.

8. Conclusions and Limitations

MR devices have high potential for retail customers to enhance their shopping experience in terms of hedonic and utilitarian value. Prior studies suggest that MR can be deployed in shopping assistant systems, although there are not yet any design principles to develop such artifacts. As mentioned in the Introduction, the first aim of this study was to propose a novel MR-enabled shopping assistant system, using MS HoloLens as the archetype, by combining content elements from e-commerce with the particular characteristics of the MR device. We developed a system that leveraged the attributes of the MR technology. The system recognized the product of interest and imposed 3D digital objects into the user’s field of view.

Seven design features were incorporated (product information, reviews, recommendations, videos, availability, buy, and virtual shopping cart) which could be navigated using NUI techniques. We explored which design principles need to be considered in the design of an MR shopping assistant for omnichannel retail. A qualitative study was conducted to capture the user perceptions towards the use of the proposed system, where 35 participants experienced the MR shopping assistant in a simulated omnichannel retail environment. Based on this study, we derived seven design principles for the future development of MR shopping assistant systems which combine retailing aspects with MR-specific characteristics: rigor, informativeness, tangibility, summary, comparability, flexibility, and holism. These design principles conclude the aim of the study, although there are some limitations to the implications of the results. Novel technologies such as the MR shopping assistant can have different effects on individuals based on their attributes. Even though this study tried to minimize the effect by random sampling and recruiting diverse participants, it cannot be eliminated. Moreover, the study did not focus on specific elements, such as the effect of prior user experience or different product categories. This study did not include quantitative measurements such as the System Usability Scale (SUS). Additionally, because the current study was only conducted with Hololens2, the effectiveness of the current design principles with other hardware could be a subject of future research. This study provides the design principles for an early stage of MR shopping assistant systems; therefore, future studies might address the following topics:

• The identification of design patterns of MR-enabled shopping applications that enhance the omnichannel experience;
• Further exploration of the proposed design principles with other MR shopping applications;
• The extension of the proposed design principles with further design features for MR-enabled HMDs in the retail context;
• Adding more assisting elements in MR shopping systems, and quantitative evaluation of those systems;
• Conducting similar studies with a different demographic.