Keywords

1 Introduction

More than three decades of augmented cognition research have undoubtedly enhanced our understanding of human-computer interactions and produced numerous advancements in technological systems capable of extending user (or operator) abilities [1,2,3]. Much of the research in this area has focused on identifying and overcoming cognitive bottlenecks stemming from human information-processing limitations [4, 5]. In recent years, however, there have been growing calls to expand augmented cognition research beyond the information-processing perspective and consider the role of broader mental states and processes in human-computer interactions [6]. Among those, self-regulatory processes have been identified as critical to human adaptive performance [7]. Self-regulation refers to an individual’s ability to compare his/her current and desired states and take necessary action to resolve any discrepancy between the two [8, 9]. Some recent work has begun to answer these calls by specifically addressing the role self-regulation plays in adaptive performance [see 10, 11].

Building on this initial work exploring self-regulation in the context of augmented cognition [11], we highlight several considerations that suggest expanding further in this direction may be useful. First, existing research on self-regulation in augmented cognition has focused almost exclusively on the regulation of physiological states [12]. However, research has shown that individuals develop a variety of adaptive strategies to address discrepancies and promote behaviors leading to goal accomplishment [18]. These self-control strategies go beyond regulating one’s physiological states and include regulating one’s external circumstances (i.e., situational strategies) and controlling one’s mental representation of the circumstances [i.e., cognitive strategies; 13]. We propose that considering the full array of self-control strategies in the context of augmented cognition can enhance human-computer interaction by (a) developing systems that more effectively support the user in initiating and exercising self-control and (b) expanding on the current mitigation strategies used by the system to address human cognitive bottlenecks. Although there is some overlap between self-control strategies and mitigation strategies currently used in HCI systems, there are self-control strategies that have not been considered in this context but could be beneficial (discussed in more detail later). Thus, expanding the view of self-regulation in this context could result in enhanced HCI systems that support and extend the individual’s self-regulatory capabilities.

Second, for optimal functioning, it is necessary for the two systems–human and computer–to cooperate in a highly dynamic way [14]. This means recognizing the intrinsic need of humans to self-regulate and then designing technology to consider the self-control actions that humans inherently implement to address discrepancies between their current and desired states. In other words, successful human-computer integration requires congruent goals and alignment of control strategies initiated by either system.

Finally, adopting a broader view of self-regulation–and in particular the consideration of self-control strategies–further highlights the potential for these expanded approaches to be applied in mobile and online environments, as has recently been suggested [e.g., 15,16,17]. For example, mobile and online applications suggest self-regulation might be conceptualized as involving not only how individuals regulate responses in the moment, but also how they select and modify their environments to support goal pursuit.

Thus, the purpose of this paper is to begin addressing some of the suggested benefits of an expanded self-regulatory framework in human-computer interactions [7, 16]. We do so by adopting an emerging view of self-control strategies [18, 19] and exploring implications of this view for conceptualizing and developing augmented cognition efforts. More specifically, we (a) discuss how this self-regulatory perspective might inform our understanding of augmented cognition, (b) explore how this self-regulatory perspective might be used to enhance augmented cognition efforts, and (c) present ideas for future research stemming from the integration of these two areas.

1.1 Self-regulation and Self-control

Broadly, self-regulation is defined as “processes involved in attaining and maintaining (i.e., keeping regular) goals, where goals are internally represented (i.e., within the self) desired states” [20, p. 158]. Control theory, a prominent theoretical perspective on self-regulation, highlights three major components: standards, monitoring, and operating [21, 22]. Standards refer to an individual’s ideals or goals. Monitoring refers to an individual’s comparison between his/her current state and the desired state. Operating refers to an individual’s response to any discrepancy between the current state and the desired state. From this control theory perspective, individuals try to reduce discrepancies between their standards and current state [9].

As a simple illustration, this control system functions similarly to a thermostat. For example, the thermostat has a goal level (e.g., 74°), measures the environmental temperature to determine if there is a discrepancy between the current environment and the goal level, and adjusts accordingly [e.g., turns on air conditioning to reduce the temperature; 23]. Control theory involves a negative feedback loop, which can cause individuals to increase effort in order to decrease the discrepancy between the current and desired state [22, 24]. Thus, self-regulation is generally seen as a controlled dynamic process in which individuals regulate their behavior in the face of external factors that may influence their current state.

Within this broad self-regulation perspective, the actions an individual takes in order to resolve any discrepancies have been conceptualized as self-control [13, 18, 25]. Thus, self-control takes place during the operating phase of self-regulation and can entail stopping undesirable responses or starting desirable responses related to a goal [26,27,28]. Researchers have found that higher self-control is related to a wide range of positive outcomes. For example, individuals with greater self-control have healthier habits, better academic success, more personal accomplishments, and fewer maladaptive behaviors [29]. On the other hand, weaker self-control has been shown to relate to negative outcomes such as criminal behaviors, obesity, drug and alcohol abuse, financial debt, impulsive spending, and procrastination [21, 30, 31]. Thus, the hallmark of a successful and healthy individual is demonstration of high self-control [32].

1.2 Self-control Strategies

Recent views of self-control have begun to take a broad perspective that considers multiple ways in which an individual can regulate his/her thoughts, feelings, and actions [18]. Building on the process model of emotion regulation [33, 34], Duckworth and colleagues [13, 18] have recently proposed that self-control strategies can be organized into a process model according to their underlying mechanisms, the stage at which they are used, and the amount of effort that is required. The resulting process model of self-control includes five families of strategies that can be divided into those that are concerned with changing external circumstances (i.e., situational control strategies) and those that are concerned with changing the mental representation of the circumstances [i.e., cognitive control strategies; 18].

Two of these self-control strategies focus on the situation: situation selection and situation modification. Situation selection involves intentionally selecting situations that facilitate accomplishment of valued goals. Situation selection is the most forward-thinking strategy, where individuals engaging in this strategy avoid situations that undermine goal pursuit and approach situations that facilitate goal pursuit. For example, to complete a task, an individual might choose to work in a quiet place (e.g., library) rather than in a noisy environment (e.g., bar). Situation modification involves changing aspects of the situation to facilitate goal pursuit. This strategy entails changing physical aspects of the environment. For instance, to complete a task, an individual might place his/her phone out of sight to avoid getting distracted by notifications.

Three of these self-control strategies focus on cognitive control: attentional deployment, cognitive change, and response modulation. Attentional deployment involves selectively attending to features of the situation to promote goal pursuit. In cases where individuals cannot select or modify the situation, they can attend to particular aspects of the environment to increase the chances of successful goal pursuit. In addition, individuals may purposely divert attention away from distracting stimuli. For example, students taking a test may attend to the test questions at hand, rather than the sound of the clock hands ticking as time is passing.

Cognitive change involves altering the way we think about the situation or task to facilitate goal pursuit. This can entail increasing the value of a task-related goal or decreasing the value of a potential distraction or impulse. For example, individuals completing a work task might construe the task as on opportunity to demonstrate their skills or view a potential distraction as a waste of time.

Finally, response modulation refers to the suppression of undesirable impulses or the amplification of desirable impulses in the moment by sheer will. This strategy thus involves just saying “no” to a disruptive impulse or forcing oneself to focus on the task or goal at hand. Response modulation can be considered a last-ditch effort to manage impulses that often fails [e.g., 36]. Based on this, Duckworth and colleagues [13, 18, 35] suggest that the other self-control strategies may be more effective than response modulation.

2 How Can a Self-regulation View Inform Our Understanding of Augmented Cognition?

The expanded view of self-regulation entailing five self-control strategies may inform our understanding of current augmented cognition approaches in three ways. First, this perspective provides an expanded framework from which to consider some of the existing approaches to augmented cognition. For example, major mitigation strategies include scheduling (including task pacing and task sequencing), modality augmentation and switching, cueing, decluttering, mixed initiative, context-sensitive help, and transposition [12]. Scheduling involves manipulating the time or the order of tasks for the user. Within this, task pacing refers to the scheduling of tasks from high-priority to low-priority, and task sequencing refers to changing simultaneous events into sequential events or dividing each task into smaller chunks and rearranging segments. Similarly, modality augmentation and switching involves two approaches: modality switching and modality redundancy. Modality switching refers to changing the sensory modality (e.g., switching from visual to auditory modality) to distribute processing load, and modality redundancy involves providing information in multiple modalities. Cueing involves capturing the user’s attention by manipulating displayed information. Decluttering involves reducing the amount or complexity of information provided. Mixed-initiative systems involve the combination of user and system adjusting the level of system autonomy. Context-sensitive help involves providing necessary information to the user at the time assistance is needed. Transposition refers to changing information displayed to the user from verbal to spatial or vice-versa. Finally, some recent work in the context of augmented cognition has also focused on assisting in the regulation of physiological states [e.g., through breathing; 10, 15].

The self-control strategies framework provides a new vantage point for interpreting these mitigation approaches. For example, mitigation strategies such as scheduling (including task pacing and task sequencing), modality augmentation and switching, mixed initiative, context-sensitive help, and transposition appear to represent modifications of the situation (i.e., the context in which the user operates). The purpose of these strategies is to change various aspects of the situation in order to promote user performance. These are thus consistent with the notion of situation modification. In addition, cueing and decluttering attempt to influence the user’s attention by manipulating the salience of information. Therefore, these are consistent with the notion of attentional deployment. Additionally, approaches focusing on the regulation of physiological states appear to be consistent with response modulation [e.g., deep breathing is a suggested application of response modulation; 18]. Note also that some mitigation strategies may overlap with more than one of the self-control strategies. For instance, as noted above, task sequencing appears to fit with the concept of situation modification but it may also overlap with cognitive change and attentional deployment. Similar to cognitive change, task sequencing may help with the reframing of larger tasks into smaller chunks. Similar to attentional deployment, task sequencing may also assist in directing attention by dividing each task into smaller segments. In addition, mixed initiative seems to have some overlap with situation modification but this appears to be a somewhat broader approach; thus, it may have connections with other self-control strategies depending on the particular form it takes. Overall, this analysis suggests that (a) current mitigation strategies fit largely within situation modification and attentional deployment, (b) there is limited overlap with cognitive change and response modulation, and (c) there appears to be little to no connection with situation selection (see Table 1 for a summary).

Table 1. Connections between self-control strategies and mitigation strategies.
Full size table

Second, this expanded view of self-regulation further emphasizes the role of the human user. Although it has been proposed in the context of augmented cognition that there are two adaptive systems–the “adaptive operator” (i.e., human user) and the adaptive technical system–the primary focus has tended to be on the technical system [16, 37]. As with the information-processing approach, the main expectation is that improvements in the human-machine system will result from the technical system enhancing or controlling the user’s adaptive cognitive processes [14]. In many ways, this approach overlooks the role that the human operator plays in the adaptive process. Self-regulation is so inherent in human beings that it is reasonable to expect that self-regulatory processes occur regardless of the technical system’s mitigation efforts. As has been proposed, leveraging this inherent need and ability of the human operator might be useful when developing HCI systems [38]. In particular, we suggest that HCI systems could be developed to support the human user more effectively in their self-regulatory efforts. Recently, Schwartz and Fuchs (2017) argued for the use of a multidimensional approach when assessing user states [38]. They introduced the Real-time Assessment of Multidimensional User States (RASMUS) as a broad diagnostic system that is capable not only of detecting performance deteriorations but also inferring the causes of such declines by assessing six user states (workload, fatigue, motivational aspects of engagement, attention, situation awareness, and emotional states). Expanding on this multidimensional approach in assessing user states, we propose that the next logical step would be to design systems that employ mitigation strategies that are best suited to address the root causes of performance declines. For example, motivational and engagement issues are probably not effectively addressed by task sequencing or task pacing but would rather need to be addressed by increasing user engagement and motivation to allocate extra resources to the task at hand. When it comes to self-regulatory failures, we propose that self-control strategies could be useful in considering the various ways that human users can be supported in their regulatory efforts.

Finally, for human-computer adaptive functioning to be productive, it is necessary for the two systems–human and technical–to have congruent adaptive goals and strategies [14]. One way to accomplish the effective integration of the two systems is to enable the human operator to communicate his/her initial goals and preferred strategies at the onset of interaction. In addition, conceptualizing the human operator as a self-regulating individual highlights that his/her goals and strategies are likely to shift over time when facing a discrepancy between current and desired state. If the computer fails to adjust accordingly under these circumstances, mismatches in goals and strategies between the two systems may significantly undermine performance over time and even result in counterproductive interaction of the two systems [16, 37, 39]. Beyond adjusting goals and strategies as needed, keeping the human operator actively involved may require building in opportunities to initiate, modify, or override assistance from the technical system. Leveraging self-regulatory processes to promote the human user’s active involvement can be especially beneficial when human-computer interaction takes place in less structured environments, such as mobile and online platforms. Thus, this more active view of the user further emphasizes the importance of developing systems that are capable of adapting over time and specifically highlights the potential need to adapt to shifting user goals and strategies.

3 How Can a Self-regulation View Enhance Augmented Cognition Efforts?

The self-control strategies perspective may also help inform the development of approaches to augmented cognition. In particular, this perspective might be leveraged to develop systems that are more capable of mimicking human self-regulatory processes, which in turn may better support users in achieving task-related goals. That is, each self-control strategy might be used to identify new or revised mitigation strategies to support user performance.

As previously discussed, situation selection refers to strategies that involve purposefully choosing to be in an environment that facilitates achievement of tasks or goals. In some cases, individuals cannot choose the situation in which they complete a task (e.g., pilots). If situation selection cannot be used, augmented cognition strategies need to focus on the remaining self-control strategies. In cases where individuals can choose the situation (e.g., educational, mobile, or online training applications), situation selection could be a highly effective strategy that might be incorporated into augmented cognition approaches. In these cases, the system could be designed to prompt or support effective selection of the situation for task completion. For instance, a variety of measurements could assist in identifying when situation selection may be particularly important. The system could use measures of the individual (e.g., performance metrics, self-ratings, physiological measures), the surrounding environment (e.g., noise, lighting), and the task (e.g., complexity, importance) to assist with situation selection. As one example, online training often involves some level of learner control (e.g., choosing when and where to initiate training). Learner control can result in increased interest and motivation for the user. However, in hypermedia environments this control can also lead to problems such as disorientation, distraction, and cognitive overload [40]. To address this, learner control could be combined with a system-assisted situation selection approach, where the system monitors the person’s state, the surrounding environment, and key task characteristics, and then prompts the person to change situations if the current context is not conducive to the training at hand.

Situation modification refers to changing aspects of the situation to facilitate goal pursuit. As noted previously, several mitigation strategies (e.g., scheduling, task sequencing, task pacing, modality augmentation and switching, decluttering, context-sensitive help, transposition, mixed initiative) can be viewed as forms of situation modification. Beyond these existing mitigation strategies, the concept of situation modification suggests other approaches. For instance, although task characteristics can cause overload, so can features of the work environment (e.g., noise levels, temperature). For example, open office environments can be stressful and demotivating due to noise [41]. Additionally, past research has shown that increased office noise is associated with lower levels of job satisfaction [42]. To alleviate the effects of negative features of work environments, augmented cognition systems could monitor ambient physical conditions and the user’s physiological state and then suggest environmental modifications as appropriate. For example, if the system recognized an increase in the decibel level of the room and user metrics revealed a reduction in task performance, the system could suggest the individual use noise cancelling headphones or earplugs.

Attentional deployment involves selectively attending to features of the situation to promote goal pursuit. This type of strategy is similar to situation modification but involves a different mechanism. Situation modification is aimed at facilitating overall self-control by changing aspects of the physical or social environment. On the other hand, attentional deployment does not entail changing aspects of the environment; instead, it involves focusing attention on particular features of that environment. As noted above, current mitigation strategies that map onto attentional deployment include cueing, decluttering, task sequencing, context-sensitive help, and transposition. A related approach within attentional deployment could be to have the system detect signs that the user is distracted and then manipulate the salience of displayed information to draw the user’s attention back to the task. For example, if the system detects that an individual’s eyes are not focusing on the task, the system could highlight particular portions of the screen. Note, however, that implementing these strategies at the wrong moment could cause a disruption in the user’s work, thus preventing successful mitigation. As suggested by Afergan, Hineks, Shibata, and Jacob (2015), physiological sensors could be used to modulate these attention-related approaches in real-time as a way to combat user disruption [43].

Cognitive change refers to thinking about a situation or task differently to facilitate goal pursuit. In terms of current mitigation strategies, task sequencing has some overlap with cognitive change. Beyond this, several other strategies involving influencing the user’s thinking about the situation or task might be possible. For instance, one approach to individual motivation emphasizes the concepts of valence (expected value of an outcome), instrumentality (belief regarding the link between performance and an outcome), and expectancy [belief regarding the link between effort and performance; see 44]. Based on this, the system could attempt to influence the individual’s level of one or more of these variables at key points in time. For example, by monitoring the individual’s engagement and performance (e.g., attention levels across task components) and task characteristics (e.g., importance, complexity), the system might be able to issue reminders of the importance of the task and how it fits in with broader objectives. Another form of cognitive change that could act to enhance performance in appropriate situations (e.g., training, education) is gamification–the application of game playing elements to encourage engagement [see 45, 46]. If the task is designed to include a gamification component (e.g., leaderboards), the system could intervene when users have low task engagement. For example, the system could use current physiological or performance indicators to determine when the user has low task engagement. Once the system has detected a need for a mitigation strategy, it could implement a gamification element by reminding the user about an underlying dynamic game (e.g., revealing the user’s current ranking within a training environment). This game element could then prompt users to think about the situation differently, facilitating goal pursuit.

Finally, response modulation involves the suppression of undesirable impulses or the amplification of desirable impulses in the moment. Human control over goal-incongruent impulses is an imperfect process, but augmented cognition might assist with this. As noted, interventions focusing on regulation of physiological states [e.g., 10] appear to overlap with this self-control strategy. Building on this, if physiological indicators of disruptive impulses could be identified, they might be used in mitigation strategies, where these indicators trigger an intervention in which the user is guided through steps to reduce the strength of these impulses (e.g., mindfulness).

4 Future Directions

Integrating the self-control strategies perspective and augmented cognition work also highlights several research directions that might benefit both areas. One potential direction involves the development of measures of the self-control strategies to assess the user’s initial preferences and capabilities in implementing these strategies. These measures could help provide the system with a starting point for understanding the user’s potential behavior in response to the many distractions or temptations that might be experienced during task performance. This information might then be used in system customization efforts in accordance with the expected user reactions to any interruptions.

A second direction involves developing a better understanding of the dynamic physiological changes that take place as impulses are generated and different self-control strategies are implemented. By using measures included in augmented cognition-based systems, indicators related to physiological changes resulting from impulse generation and strategy implementation might be observed (e.g., cortisol level, heart rate). Understanding these underlying relationships may provide insights regarding how to objectively and dynamically assess users’ reactions to various distractions, which would enable the system to implement customizations as needed. The increasing prevalence and technological advancement of wearable devices can also support the measurement of physiological changes in an efficient [e.g., 47] and accurate way [e.g., 48, 49].

A third potential direction involves exploring different ways in which mixed reality technologies [50]–such as virtual reality, augmented virtuality, and augmented reality–can directly support and enhance human self-regulatory capabilities. The virtuality continuum [50] of these technologies allows for different levels of real and virtual environments to be experienced. Future research efforts can examine how visual display devices that incorporate these technologies (e.g., virtual reality head-mounted displays, augmented reality glasses) may strengthen the implementation of self-control strategies by enhancing or limiting sensory (e.g., visual) information from the environment. For instance, virtual reality head-mounted displays can support situation selection by immersing the user in a complete virtual environment. Other self-control strategies might be enhanced by using augmented reality glasses: situation modification and attention deployment could be supported by limiting or masking disruptive elements in the environment, and cognitive change and response modulation could be supported by adding virtual layers that may act as motivational or inhibitory reminders.

Finally, a fourth direction is to capture each individual’s unique pattern of self-control strategy use with the aim of integrating it later with different systems that the user may wish to use. This can be applied in two contexts: (a) relatively-similar situations and (b) relatively-novel situations. In relatively-similar situations, the user experiences a situation that is similar to previous situations in which the user’s pattern of self-control strategy use was observed. In this case, standalone data about a user’s preferences or capabilities in implementing self-control strategies could act as an add-on with other software programs to facilitate the customization process of systems to maximize individual performance. By capitalizing on similar aspects of these experiences, users may adapt more quickly to other systems and these systems could become more effective and versatile. In relatively-novel situations, the user experiences a situation that can be considered new in comparison to the characteristics of situations for which the user’s pattern of self-control strategy use was recorded. In this case, learning about user patterns of self-control strategy use for achieving various goals in previous situations would involve building a dynamic dataset optimized by machine learning through many user-system interactions. This proposed machine learning software could then analyze novel situations or experiences and the system would be able to suggest specific self-control strategy interventions that are expected to be most effective based on each user’s capacities and limitations. In this way, individual self-control strategy habits could be automated to help users adapt to and thrive in new situations where no habits have yet been formed, allowing higher levels of adaptation to be reached in an accelerated way.

5 Conclusion

Recent work has begun to highlight the potential importance of self-regulatory processes in the context of augmented cognition. The current paper attempts to expand further in this direction by discussing an emerging view of self-regulation focusing on self-control strategies. We discuss how a self-control strategies perspective may inform our understanding of current augmented cognition approaches in several ways, suggest new approaches for developing mitigation strategies, and highlight future research directions. We hope that this preliminary discussion of this integration leads to additional work that may be beneficial to both areas.