Metrics for Robot Proficiency Self-assessment and Communication of Proficiency in Human-robot Teams

2.1 Concepts Related to Proficiency Self-assessment

Proficiency assessment is closely related to explainability in artificial intelligence (explainable AI, or XAI). Hoffman et al. [70] identify three purposes of XAI: “How does [the AI] work?”, “What mistakes can [the AI] make?”, and “Why did [the AI] just do that?” XAI emphasizes causal factors that a human can use to calibrate trust in and reliance on decisions made by an AI algorithm. The explanation of the causal factors might include bounds on the algorithm’s confidence in its performance or reliability. The overlap between proficiency self-assessment and XAI would include intersecting sets, but neither XAI nor proficiency self-assessment is a subset of the other. For example, proficiency self-assessment might include an explanation, but it might also be a clear statement about how proficient the agent is without any explanation. The complement to this is when XAI includes a discussion of what bounds influence the competency of the algorithm without yielding a clear assertion about whether the algorithm will be useful in the present context. Furthermore, explanations of “Why did the AI just do that?” emphasize post hoc evaluation and may de-emphasize a priori and in situ assertions about likely success. Finally, proficiency self-assessment can be used by an agent to autonomously initiate a change in goals or behaviors, thus supporting mixed initiative interactions that tend to be outside the scope of XAI.

Communicating proficiency is also closely related to transparency in human-machine interaction. Chen et al. define transparency as “the descriptive quality of an interface pertaining to its abilities to afford an operator’s comprehension about an intelligent agent’s intent, performance, future plans, and reasoning process” [21]. These elements of transparency suggest that the concepts of transparency and proficiency self-assessment overlap, but, as with proficiency self-assessment and XAI, neither is subsumed by the other. Transparency emphasizes in situ assessments of the causal factors contributing to agent behavior including the beliefs and motivations of the agent, and can include projections of the likely success of the agent in its task. Transparency is a property of an interface, and the interface may not explicitly report real-time performance metrics [134] or other assessments of proficiency.

Endsley’s three levels of situation awareness (SA)—perception, comprehension, and projection [40]—are widely used throughout HRI research and are particularly relevant to metrics definition for robot self-assessment of proficiency. The robot’s proficiency measures could identify shortfalls or alignments between required capability to perform a task and robot capability (perception), explaining or reasoning as to why and the degree to which success or failure is likely (comprehension), or predicting the robot’s ability to accomplish a task (projection). Communication of proficiency can also be categorized using these levels of SA, similar to Chen et al.’s situation-awareness-based agent transparency (SAT) model [21]. In the SAT model, an autonomous agent’s goals and actions (level 1 SAT), reasoning process (level 2 SAT), and projections/predictions (level 3 SAT) optionally with associated uncertainty measures (U) can be communicated transparently to improve team effectiveness. The SAT model has been used to develop interfaces that adhere to its principles at level 1, 1 + 2, 1 + 2 + 3, and 1 + 2 + 3 + U [136], the latter of which is most related to communicating proficiency self-assessment due to its inclusion of uncertainty measures. Given that Endsley’s three levels of SA are primarily used for categorizing human understanding, they are used as such for metrics of human understanding later in this article.

Throughout the remainder of the article, relevant metrics from each of these related concepts are referenced if similar measures exist. The intent of this article is to leverage metrics from these areas and recontextualize them as appropriate as well as propose new metrics that are particular to each stage of the proficiency-based interaction flow.

2.2 Human-robot Roles

For the purposes of this article, the human is the problem holder, meaning they act as a partner who is teamed with the robot and who requested the robot to perform a task. The human’s role is therefore most akin to that of a supervisor, operator, or teammate [133], in that they have some experience with the robot and are intended to work with or alongside it. More specifically, the human’s role as the problem holder is somewhere between that of an information consumer [56] (receives and uses information from the robot to make decisions) and an abstract supervisor [71] (uses the information received from the robot to modify its objectives and goals). The experience level of the human may vary, though, ranging from novice to expert, which will impact the human’s understanding of the robot’s proficiency and therefore how the robot should communicate its proficiency. As part of the human-robot team, the robot can serve multiple roles including individual support, team support, or as a team member [153]. The robot may consider the implications of the human-robot roles during interactions to influence how it chooses to communicate its proficiency. These human-robot roles set the use case for the metrics reviewed in this article; i.e., other roles such as the human as a bystander or the robot serving a social role are not considered.

2.3 Proficiency-based Interaction Flow

To frame this article’s discussion of metrics, a proficiency-based interaction flow that consists of four stages is proposed:

• RSA: Robot performs Self-Assessment of proficiency.

• RCP: Robot Communicates its Proficiency to the human.

• HUP: Human processes the communication and conveys their Understanding of the robot’s Proficiency.

• RPH: Robot Perceives the Human’s intentions, values, and assessments.

This four-stage interaction flow is shown in Figure 1. For a given proficiency-based interaction, all stages may not occur (e.g., interaction with a robot that does not possess the required perception capabilities for RPH will skip this stage), but the flow is intentionally abstract to encompass all proficiency-based interactions. The metrics at each stage can be used to evaluate human-robot interactions that include robot proficiency self-assessment and communication of proficiency, or can be used by the robot system to evaluate its own performance. The use of the metrics by robot systems may be particularly important for robots that attempt to improve their communications of proficiency to humans over time. Metrics are reviewed for each stage of the proficiency-based interaction flow in the following sections. The metrics reviewed at the RSA and RPH stages are categorized and defined similarly (both deal with evaluations of the task and teaming), as are those at the RCP and HUP stages (both deal with communications). Connections between the metrics within and across stages are detailed in each section as appropriate.

Fig. 1.

View Figure

Each stage impacts subsequent stages in the flow. The robot’s self-assessed measure of proficiency at the RSA stage will be converted into a communicable form at the RCP stage. When that proficiency is communicated, the human will attempt to understand it at the HUP stage. When the human conveys that understanding back to the robot (e.g., by pointing and asking about an object the robot referenced in its communication), the robot may perceive the human’s behavior in the RPH stage to infer information about the human, such as the human’s intention. Based on how the robot uses this information in the RPH stage, the RSA stage may be repeated (e.g., if the human made any physical updates to the task to improve the likelihood of success), which may include updating assessments about its own proficiency (e.g., if the human suggests different strategies to assist in accomplishing the task). Alternatively, the interaction may progress from the RPH stage to the RCP stage wherein the robot communicates its proficiency again using a different method (e.g., if it perceived the human did not understand the previous communication) and/or updating the information being communicated (e.g., if the human asked a clarifying question in response to the previous communication).

A summary of the metrics presented throughout the rest of this article can be seen in Table 1, organized by the stage of the proficiency-based interaction flow they are associated with and grouped into categories.

2.4 The Fetch Scenario

To ground the discussion of metrics to a common reference point, a human-robot interaction scenario is defined that will be referenced throughout the article: a mobile manipulator robot is tasked with finding bolts, screws, and other components to place into a specific type of bin and then deliver the bin to another station for the human to use to build a gearbox (Figure 2). The process will be performed continuously until several bins are filled by the robot and used by the human to build several gearboxes, aiming to produce a specified number of gearboxes per hour. The human problem holder in this scenario has commanded the robot to perform the task, will monitor the robot’s performance, can ask questions about progress made, can intervene to augment the scenario as needed, and will inspect the final state of the bin before building the gearbox. This scenario is similar to that described in Frasca et al. [50]. Throughout the rest of this article, this scenario is referred to as “the Fetch scenario.” Robot proficiency self-assessment techniques and communication methods are not specified here but will be as needed when the scenario is referenced. To illustrate the proficiency-based interaction flow in Figure 1, below is an example of the events at each stage when occurring in situ for the Fetch scenario:

Fig. 2.

View Figure

• RSA: The robot conducts proficiency self-assessment at picking up a bin filled with screws, producing a measure of low confidence at succeeding.

• RCP: The robot speaks to the human, “I am unlikely to pick up the bin without dropping it.”

• HUP: The human heard the robot but does not fully understand why the robot has low confidence, so they follow up and ask, “Why is your confidence low?”

• RPH: The robot perceives this communication and decides that it will communicate its proficiency again with more detail (i.e., advancing to the RCP stage next).

2.5 Limitations

It should be noted that the metrics presented in this article are each at varying levels of development, maturity, and prominence. Some metrics have not yet been robustly validated through experimentation, while others have been utilized substantially throughout research albeit in different contexts. Readers are encouraged to refer to the research articles cited for each metric (if available) for further information regarding implementation and nuances. The authors acknowledge that the newly proposed metrics in this article may be less substantially defined than others. Continued definition, development, and comparison of these metrics toward effective implementation will be required as proficiency self-assessment research progresses, so the metrics presented in this article should be considered a starting point for the field.

3.1 RSA: Uncertainty

This category of metrics relates to the robot’s confidence in its ability to complete the task to a particular performance standard. Uncertainty is different from performance prediction; an agent can have high confidence that a particular sequence of actions will result in task success or failure, whereas a robot that is uncertain in its task model may produce a range of possible outputs without a means of distinguishing higher-performing outputs from lower-performing ones. While the following metrics are largely used for in situ self-assessment, metrics of uncertainty can also be used a priori and post hoc. For example, Johnson et al. [75] evaluated model uncertainty according to an acceptable bound of uncertainty, considering the maximum uncertainty to be that of the model before any training data has been obtained (i.e., a priori self-assessment). The level of “acceptable” uncertainty (i.e., the level of uncertainty at which the model’s output is accepted) could then serve as a useful post hoc metric.

Alignment of uncertainty and performance. This metric is measured through correlations between the variance of the model’s output and the actual task performance achieved using that output. Fleming and Daw [46] used a similar metric for evaluating uncertainty-based approaches, correlating the confidence in the model to the actual error incurred by its output; they also used the correlation between the model confidence and the strength of the input stimulus as a metric. When a robot may represent a task according to multiple models, the uncertainty of the candidate models may indicate which is best suited for a particular learning problem. For example, Fitzgerald et al. [45] evaluated the correlation between (1) the uncertainty of two models trained over a set of noisy training data and (2) their respective performance in performing a novel task. The resulting comparison is an evaluation of whether model uncertainty serves as a proxy for expected task performance.

Risk-averse reward. The total reward over a sequence of actions by the robot, based on the cost of correct, incorrect, and undecided actions, is used to calculate this metric. When used as a reward function in self-assessment problems, risk-averse metrics consist typically of three rewards: the reward associated with (1) making a correct decision, (2) making an incorrect decision, and (3) making no decision (and thus rejecting the task). An agent trained to optimize this reward function via reinforcement learning will (ideally) learn to gauge its confidence in a decision such that it rejects the task when it is uncertain in its model’s output. When used as a metric for robot self-assessment (e.g., [27, 73, 171]), the cumulative reward over a series of tasks reflects the robot’s ability to assess risk. This cumulative reward is increased by rejecting tasks in lieu of achieving poor performance on them, while also attempting tasks that the robot expects to complete successfully.

Uncertainty reduction. Calculating the accuracy in predicting what training data will reduce the robot’s uncertainty is another means of evaluating a robot’s ability to assess which training data points will improve its task model. In this context, it is assumed that the robot has access to a human teacher who may provide additional training data to the robot. If the robot’s assessment of its uncertainty is correct, it will be able to correctly identify training samples that, once labeled by a teacher, will lead to reduced uncertainty in the overall task model. Rakicevic and Kormushev [119] presented an online active learning approach for a task learning and transfer application. The exploration component was decoupled from the task model and performed an informed search in the trial-parameter space to generate the subsequent most informative trials, by simultaneously exploiting information from previous trials and reducing the task model’s overall uncertainty.

3.2 RSA: Performance

This category of metrics relates to the extent to which the robot was able to complete its task. While performance and proficiency are not the same thing, measures of performance can be used to provide information about the robot’s proficiency. A robot that is performing a task poorly when previously believing it was proficient at the task should consider whether the failure is a result of a lack of proficiency or simply circumstances outside its control. When a robot has performed the same or similar tasks multiple times in the past, then it can use its past performance in these tasks as an a priori assessment of its proficiency [73, 171]; this assumes that the robot is capable of determining how past performance relates to the current task, particularly the current context.

Performance metrics primarily relate only to in situ and post hoc applications. Cumulative scores, rewards, and penalties can be assessed after a mission to determine how well a task was performed. Additionally, performance summaries [135] can be used to evaluate proficiency. Estimates of performance and proficiency can be further evaluated via subjective scoring by the human at the HUP stage (see Section 5). Additionally, when evaluating a robot’s self-assessment ability, a set of experts may use appropriate behavioral coding standards to define a “ground truth” label for proficient and inproficient robot behavior.

Mission progress. In sequential tasks, mission progress refers to the number or percentage of sub-tasks that have been successfully completed. Real-time estimates of mission progress or partial summaries of mission state can include reaching mileposts, satisfying preconditions or postconditions, and deviations from scripts [134, 135]. When the robot achieves a milestone or satisfies necessary preconditions or postconditions, it can potentially increase its confidence in its proficiency in that task. On the other hand, failures to achieve milestones or satisfy preconditions or postconditions might reflect negatively on the robot’s proficiency in the absence of a more detailed understanding to explain away such failures.

Replanning triggers. The metric evaluates the rate or frequency of replanning a mission. Replanning can be triggered by changes in the environment [51, 167], by lack of progress [161], at regular time intervals [26], when a plan drifts away from the problem holder’s intention [137], or when a goal-reasoning system indicates that the goal currently being pursued cannot be accomplished [4]. Changes to the environment or robot that lead to state spaces in which the robot is no longer proficient should be detected in order to enable replanning.

Accuracy vs. rejection curves. This is the relationship between the agent’s task rejection rate and its performance on accepted tasks. An accuracy vs. rejection curve reflects (1) the range of risk assessments (rejection rates) that are plausible for a particular task and (2) the acceptable level of risk for a particular task according to the corresponding performance expectation. In practice, a single rejection rate may be selected such that the robot rejects tasks that exceed that risk threshold. This metric has been demonstrated in the context of assessing the reliability of a robot’s perception [27, 171] and to evaluate the tradeoff between the accuracy and autonomy afforded to a robot as a function of how much assistance it receives from a human teacher [44]. Overall, accuracy vs. rejection curves provide a post hoc evaluation of the robot’s performance across a range of risk tolerances.

3.3 RSA: Time

This category of metrics relates to the robot’s uncertainty and expected performance over time. The extent to which a robot is proficient at a task is highly governed by the probability that the robot will accomplish the task within a time bound. In a human-robot interaction context, time-based thresholds are often used as indicators to predict and evaluate failures and to aid the robot in making decisions as to when to ask for human assistance to recover from a failure.

Predicted vs. actual completion time. This metric measures the accuracy in predicting the robot.s task completion time according to changes in system and environment. Assessment of predicted task completion time against a time threshold is a another metric for how proficiently the robot will be able to complete a task. In the Fetch scenario, the robot can use the planner’s output of “predicted time to execute a particular trajectory” to decide a priori, based on a predefined time threshold, whether to replan or to execute the recommended trajectory. Additionally, the combination of uncertainty and time (e.g., the ratio of maximum uncertainty, as discussed in Section 3.1, to expected task completion time) can be a useful metric to choose among goals relevant to the task when there are conflicting goals. This metric may also serve to measure the difference between the predicted failure time and actual time when a failure occurred [132]. A comparison of predicted completion time to actual completion time as a post hoc self-assessment is a simple indicator of how well a task was performed.

Productive time. The duration of time in which the robot operates autonomously is its productive time. This metric, which is primarily applicable in supervisory control tasks, measures the continuous amount of time that the agent can remain autonomous before requiring assistance. It may serve as an in situ metric toward measuring a robot’s proficiency toward a task or a subgoal. For example, Olsen and Goodrich [113] defined productive time as the ratio of the time spent in autonomous operations to the total time spent across autonomous, manual, and unscheduled manual operations.

Reliability. This is the duration of time in which the robot meets performance standards under defined working conditions. Some of the commonly used time parameters for this metric include mean time between failures (MTBF) and mean time to failure (MTTF) [78]. In the Fetch scenario, the system time between perception failures is used as a decision-making parameter to reset the vision system.

Forecasting time. This is the time elapsed between the prediction of a failure and the occurrence of that failure. This metric may be measured at multiple levels of confidence in the form of a prediction time curve, measuring the time elapsed between initial fault detection and the confirmation of that fault [131, 132]. Alternatively, the volume of data required to detect a failure may be tracked as a proxy for time [132].

3.4 RSA: Events

This category of metrics relates to changes in the system status that affect the robot’s performance. At the most basic level, counting events can provide insight into how well the robot is performing a task. Examples include, but are not limited to, counts of failures, human interventions, requests by the robot for assistance, corrections, and the amount of required training. These types of measurements are largely applicable to in situ and post hoc interactions. Additionally, in a human-robot team context the numerical thresholds on these measures for acceptable performance might directly impact other measures of proficiency. For example, in a time-critical mission, a lower threshold on the allowed number of replans or reattempts for the robot would result in a higher number of human interventions.

Interventions. External interventions that can affect (positively or negatively) the robot’s performance are counted and can be compared to another set of countable instances (e.g., successfully performed tasks without interventions). In a human-robot teaming context, interventions may be initiated by the robot (e.g., in response to expected need for data [44]) as well as by the human (e.g., in response to an impending robot failure or collision [27]). Depending on their nature, these interventions can provide useful insight into the proficiency of the robot. Proportions of autonomously completed and human-assisted steps within a task can also be an indicator of productivity. For example, Fitzgerald et al. [44] measure the ratio of supervised and unsupervised steps in an assembly task, where supervision consisted of indicating the object the robot should use in the next step of the task. In a safety-critical setting, a higher ratio of human-initiated autonomy switches to total autonomy switches or interventions suggests an overall poorer task performance, as compared to a scenario with a lower ratio.

Repeated attempts. This metric can be measured when an action is performed in response to system or environment anomalies that affects the robot’s performance, resulting in repeat attempts of that action. When a robot fails in an operation during a task, it might reattempt the operation or ask a human for assistance. Monitoring the number of reattempts (or the number of failures) can characterize a robot’s progress on a task. In the Fetch scenario, the robot failing to pick up a screw after a pre-set number of reattempts could initiate a request for human assistance. For scenarios where direct human assistance is not possible, a robot might use the number of failures as an indicator to abort or reinitialize the operation. Often counts of such failure events may be used in combination with a time metric. Returning to the Fetch scenario, a certain number of detection failures by the perception system, combined with time between failures, could be used as a decision-making parameter to reset the camera and vision system. Monitoring the ratio of number of successful task executions vs. total task executions [123] can serve as an indicator of a robot’s reliability, which can be used a priori to update the robot’s self-confidence prior to the next task repetition.

Violation of performance envelopes. The number of times when the robot’s performance is not within acceptable bounds in a mission is measured by this metric. Performance envelopes [69] can be established based on prior experience. In a human-robot teaming context, frequent violation of performance envelopes can indicate that the robot is not proficient according to the performance standard set by the human for a particular environment. Examples of ways to measure whether performance is within acceptable bounds include (1) comparing in situ performance estimates to predefined numerical thresholds [9], (2) testing whether performance exceeds history-based aspiration levels from Simon’s theory of satisficing behavior [140], (3) using barrier-function-based measures to evaluate proximity to constraint violations [37], and (4) detecting the violation of verification and validation assumptions in real time with a human-in-the-loop [129].

4.1 Modalities for Communicating Proficiency

The robot will use one or more modalities to communicate its proficiency to the human. The inherent limitations of the communication modality will affect what kind of proficiency measures can be communicated and what aspects of the communication the human is expected to understand (i.e., inputs to the HUP stage).

A visualization display consisting of graphics, charts, or images displayed on a monitor (e.g., visualizations of input-output relationships of a neural network [172] or heat maps to convey image saliency [22, 65]) can be rich in information and can be transmitted instantaneously, but may require some additional interpretation from the human. These could include text as part of a visualization (e.g., highlighting text that indicates what component of the robot system is being used to perform a task [165]), a historical log, or sentences via natural language generation (e.g., narrating robot experiences during task execution [126], summarizing explanations based on situation criteria [172]). Natural language generated by the robot can also be conveyed visually and could appear instantaneously while still requiring interpretation from the human (i.e., reading), albeit the format may be more immediately recognizable and can afford increased user satisfaction and confidence with a system [38]. Audio can also be used for conveying natural language to form statements (e.g., communicating faults such as low battery [62]), explanations (e.g., explaining the values, tradeoffs, and competing objectives of planned behaviors [152]), or questions (e.g., the robot asking if the human understood what it communicated [41]). Non-speech audio, such as “auditory icons,” or sounds that represent concepts being conveyed [104] may also be used for alerts to get a user’s attention. Auditory communication is most appropriate for short, concise messages that require immediate response, whereas visual displays might be preferred in cases where the message is complex, must be referred to later, or has a spatial component [29].

Lights positioned on the robot may also be used for alerts (e.g., when encountering a person or obstacle [144]) or for communicating directional information (e.g., which direction the robot is planning to turn [139]). Robot motion and augmented reality (AR) communication methods can utilize a common reference frame of the physical scene wherein parts of the environment can be explicitly referenced [18]. For example, motions performed by the robot may reference itself (e.g., arm movement to express incapability at lifting an object [90], torso movement to correlate with confidence level [154]) and/or the environment (e.g., adjusting motion trajectories to more clearly communicate robot intent for object selection [34, 98], pointing to objects for the human to manipulate [60], gazing toward points of interest [2]). AR projections into the physical environment can also reference the robot (e.g., light projection onto the floor to depict the planned path of travel [15, 138]) or parts of the environment (e.g., projecting a hologram icon onto a tool the robot is planning to use [18]). Virtual reality (VR) methods can display animations of simulated robot movement, such as showing alternatives that could have been performed as the result of a post hoc assessment after a failed task. These allow for adjustable perspective that may not be feasible using other modalities (e.g., an immersed operator judging planned robot movements prior to their execution [23]).

Multi-modal interfaces can maximize the richness of communications and optimize human workload. In the context of a swarm of robots, transparent communications of the collective have utilized color-coded visualizations of status combined with robot speech and vibrations for haptic feedback to alleviate workload and increase situation awareness [59]. For human-robot handovers, combinations of robot motion and haptic feedback have been used when a robot holds onto an object longer than expected to convey information to the human partner, which can influence the human’s behavior [1]. In a multi-modal interface, the information being communicated may be redundant across modalities (e.g., text-captioned speech [80]) to ensure that the information is received by the human (which has been shown to be effective at resolving human uncertainty in complex tasks [3]), whereas non-redundant information in two separate modalities may contain two different pieces of information (e.g., visual display showing low battery, speech describing possibility of future task success) or a modulation of the information [114] (e.g., speech describing task failure with robot motion to explain which part of the robot failed). An advanced robot system may also choose one modality over another to communicate an aspect of proficiency based on contextual factors and the inherent limitations of single modalities described here.

4.2 RCP: Attributes

The proficiency measures being communicated by the robot will possess one or more possible attributes. For the proficiency self-assessment context, attributes from the explanation research literature are leveraged including those that may be predictive of explanation quality [169], failure types for explaining impossible robot behaviors [120], truthfulness of agents (e.g., transparency vs. deception [33]), and others to distill a common set. These attributes can be used to characterize the contents of a robot communication and will factor into complexity metrics (see next section). See Table 2 for examples of each of the attributes as applied to the Fetch scenario.

Information communicated. The following attributes characterize the type of proficiency information included in the communication from the robot:

• Alternatives. These are task strategies that could enable higher-proficiency self-assessments to be made by either increasing the likelihood of success, optimizing performance thresholds (e.g., a robot presenting possible grasps it could execute for an operator to select [94]), or enabling otherwise impossible tasks to be performed. Each of the alternatives may also have associated probabilities for success or failure and their communication may possess some of the other attributes in this list. Alternatives presented a priori would be predictions of expected failure (e.g., predicting and mitigating potential trust-induced failures by proposing alternatives [160]), in situ alternatives could be caused by detecting a failure in progress, and evaluations of proficiency post hoc may lead to communicating alternatives for next time.

• Contextual reference. Elements in the scene that contribute to the self-assessed measures of proficiency may be referenced within the communication. Reference to elements in the scene may be made relatively (e.g., the object to the robot’s left) or absolutely (e.g., the red object); this attribute is referred to as locality in [126], where a robot narrates its experiences. Contextual reference can include characteristics of the environment, objects interacted with, and reference to either agent.

• Governance. The robot’s self-assessment may be governed via an internal set of rules or policies used to drive the resulting proficiency measures and then referenced in the communication. Explainable AI systems often have these types of mechanisms (e.g., [117]) to provide reasoning for their decision-making to a user. Inclusion of this attribute in a communication may also inform how the human tasks the robot in the future while those same rules and policies are active.

• Impact. This attribute notes the effect of anticipated success or failure on subsequent actions or future tasks. Inclusion of this type of information may be based on predicting performance on part of an upcoming task and describing how the robot’s proficiency may propagate failure and success throughout the entire task (e.g., if the robot predicts issues with lifting a type of object and the task specification includes lifting several of that object). The robot may have a governing set of rules or policies as described previously that, if adhered to for a given scenario, may violate each other (e.g., the robot cannot follow a person while also avoiding the kitchen if the person walks into the kitchen [121]).

• Novelty. The robot may communicate its proficiency based on prior experience with the same task or similar tasks or that the task is perceived to be novel; this metric attribute captures the amount of novelty. Similarity-based self-assessment techniques (e.g., [7, 58, 74]) utilize these types of comparisons as part of measuring proficiency and could therefore output them as part of the communication.

• Probability. Measures of probability will be included in all communications of proficiency, albeit likely at different abstraction levels. These types of measures are akin to expressions of uncertainty (e.g., [20, 163]) or self-confidence (e.g., [5, 89]), or strictly as probabilities when used in explanations (e.g., [91]). Probability measures can be conveyed explicitly (e.g., displayed text or spoken words) or implicitly (e.g., visual display characteristics to correlate with uncertainty communication [81, 83, 97]). Conveying measures of uncertainty may impact the credibility of the robot and therefore the human’s trust of the robot’s proficiency.

Nature of communication. The nature of the proficiency information being communicated (i.e., how it is being communicated) can be characterized using the following attributes:

• Framing. The proficiency measures will be framed as measures of either success or failure (i.e., attribute framing as positive or negative proportions [95], or critical or complimentary recommendations [163]). Raman and Kress-Gazit [120] describe three types of failure conditions that can be communicated: unsynthesizable when the robot attempted the task but failed (applicable to post hoc communication only), unsatisfiable when an unchangeable condition of the scenario physically prevents the task from being attempted, or unrealizable when the task could be attempted if something was changed. These same types of failures can be inverted for types of success measures when the robot was able to attempt the task and succeeded (synthesizable; applicable to post hoc communication only) or the conditions of the scenario enable the task to be attempted (satisfiable and realizable; applicable to a priori communication only).

• Truthfulness. The robot may be truthful in its communications or intentionally acting subversively or deceptively rather than transparently in order to elicit a particular response from the human at the HUP stage of the interaction flow (e.g., giving the human the perception of having control over the robot or that the robot is more capable than it actually is [150]). Deceptive robot motion may also be used to exaggerate, quickly switch, or communicate ambiguous goals [33], which could be the case if the robot is in an adversarial role.

• Redundancy. The communication of proficiency may be redundant across modalities when a robot utilizes multi-modal communication. The entire communication may be redundant across modalities (e.g., text-captioned speech), or only certain attributes of the communication may be redundant (e.g., contextual reference to an object in the space may be described via speech while the robot motions toward it). Redundancy in a communication may be used to ensure that the communication is received by the human but also may lead to cognitive overload [114].

• Scope. Scope relates to how much of the task or to what property of the task the proficiency measure refers to. The scope could be the entire task, a subtask, or a requirement of the task/subtask.

The proficiency measures derived from the RSA stage will inform about the possible attributes the communication could possess at the RCP stage (e.g., communicating alternatives if a counterfactual self-assessment method is used). These attributes can also be used to inform metrics at the HUP stage whereby evaluations can be performed to determine if the human perceives and comprehends these attributes (which may or may not be required for effective task performance).

Communicability. Some aspects of the proficiency measures derived at the RSA stage may not be communicable by the robot. A lack of communicability could be due to the inherent limitations of its available communication modalities or a lack of functionality to convert proficiency measures into a communicable form. Communicability concerns a robot’s overall proficiency-based communication capabilities, comparing the number of unique attributes the robot is able to communicate to a master list of possible attributes. This characterizes the overall robot capability rather than individual instances of communication. For example, if the robot in the Fetch scenario lacks the ability to specify which object is obstructing its path, then proficiency measures that make contextual reference to the object are not communicable. A system may also be able to reason about communicability, conducting feasibility analyses of the limitations of its available modalities [13]. This is an important characterization to make in order to set expectations for measures at the HUP stage and as a guideline when designing a robot’s proficiency self-assessment and communication methods.

4.3 RCP: Complexity

Several metrics cover the complexity of communicated proficiency and are influenced by the attributes the communication possesses (see prior section) with some modality-specific measures. These qualitative and quantitative metrics are most useful for relative comparisons between communications of proficiency (e.g., one communication is more or less abstract than another) rather than absolute scales.

Abstraction. The level of abstraction refers to the amount of detail in the description that is given about a piece of information, which can be represented in coarse or fine resolution [147]. A measure of abstraction can be applied to each of the previously reviewed communication attributes. For example, in the Fetch scenario, contextual reference to a bolt could be communicated by the robot pointing to it (coarse) or it could simultaneously state via speech that it cannot pick up the bolt by its head (fine). For communicating probability, the robot’s confidence in task success could be stated as high/low (coarse) or as a percentage (fine). While no absolute scale exists for information abstraction, some efforts specify a mapping of task characteristics into abstraction levels. For example, Rosenthal et al. [126] describe a mobile robot communicating information about its navigation performance, defining levels of increasing abstraction starting with coordinates of movements (level 1), then to traversal times and distances (level 2), then to right/left turns and straight segments (level 3). Similar scales could be developed based on different task types.

Clutter. Any extraneous information that is included as part of the communication of proficiency that may distract from or obfuscate the core of the communication is considered clutter. Objective metrics to evaluate clutter are specific to the modalities used for the communication. Visual clutter metrics include feature congestion and edge density [125]; Roundtree et al. [127] specify a set of metrics (including visual clutter) for swarm visualizations that were shown to be predictive of human perceived transparency and performance. Visual clutter metrics are also useful when considering the entire scene where activities occur (e.g., for everyday driving [86]). The use of extra words or description used in text and speech modalities can be evaluated by comparing the number of unique words used to the total number of words in the communication [130]. Readability analyses of text- or speech-based communications can be performed by utilizing readability indices such as the Flesch-Kincaid readability test [47] or the FOG index [57]. These analyses are devoid of context, however, so additional subjective ratings should also be conducted at the HUP stage.

Comprehensiveness. The number of attributes possessed by a communication is indicative of its comprehensiveness. Comprehensiveness can be expressed as a ratio of the number of unique attributes the communication possesses to the number the robot is capable of communicating (i.e., its communicability). At the HUP stage, assessments of comprehensiveness can be conducted to evaluate human understanding of completeness (the quantity of information received compared to the quantity expected [87, 93]).

Simplicity. For the purposes of this article, simplicity refers to the degree to which there is a “simple” explanation for the proficiency measurement being communicated. This concept is borrowed from evaluation of explanations that are measured by generating a causal model for the explanation and counting the number of root causes in the model [169]. Such a model may be generated by identifying causal language provided in the explanation and then counting the connections between causes and effects [142]. Evaluating the number of root causes has been shown to be predictive of explanation quality [169], but determining the causal pathways for an explanation is somewhat subjective; more formal methods may be needed to validate this metric.

Size. The size of a communication refers to the number of features it contains, which are modality specific. Note that size is not the same as length, which is time based (see Section 4.4). The size of a text or speech display can be measured by counting the number of words or sentences [126]. Similar quantitative metrics can be applied to robot motions used for communication (either physically or simulated in VR) including the distance traveled by the robot’s limbs, the number of repetitions of communicative movements, and the speed of the movements [90]. In Kwon et al. [90], repetition of expressive robot motion at a fast or moderate speed was shown to increase a human’s understanding of the robot’s goal and made the cause of robot incapability clearer. Static visual displays may lack a discernible set of features to produce a metric for size.

4.4 RCP: Efficiency

The robot will communicate its proficiency to the human over a period of time. The amount of time used to communicate is a function of the length of the message and the speed at which it is communicated.

Communication time. The amount of time required for a message to be (1) generated, (2) transmitted, and (3) understood by the recipient [102] is the communication time. For the purposes of this article, these phases are adapted to fit the proficiency-based interaction flow and are delineated into three separate metrics: conversion time, transmission time, and perception/comprehension time. The latter is out of scope for RCP metrics but in scope for HUP metrics (see Section 5.1). Communication time will be most impacted by the overall complexity of the communication. A highly transparent robot will communicate more information that may be less efficient than others, producing a transparency-efficiency tradeoff [72]. It is intended that the complexity and efficiency measures provided in this article could be used to conduct similar analyses for evaluating HRI within the proficiency-based interaction flow.

Conversion time. The first phase is the time required by the robot to convert the measures of proficiency it intends to convey to the human into a communicable form (see Section 3), a common step in facilitating information exchange in human-robot teams [14]. Depending on how the robot’s self-assessment functions, measurement of conversion time may be difficult. Concretization is a process used to produce a communicable version of an explanation generated by a system [146]. If an explicit process like this is used by the robot, then conversion time could be measured and reported by the robot.

Communication latency. Latency is the amount of time between when a message is sent and when it is received; it is a common metric for evaluating HRI communications [151]. As an efficiency metric at the RCP stage, communication latency can be measured either as the amount of time between the RSA stage and the transmission of the communicated proficiency (see next metric) or as the amount of time between conversion and transmission. Unless the robot explicitly communicates when it is in each phase, latency measures between conversion and transmission may be difficult to measure. Latency between transmission and perception/comprehension time is covered in Section 5.2. High communication latency times may also impact HUP metrics, particularly if the human is expected to react in a timely manner to the communication, such as to intervene to improve the robot’s proficiency measures.

Transmission time. The second phase refers to the length of time from the moment the robot starts communicating its proficiency until it stops. This can be measured according to the observable start and end time of communication (e.g., the length of sentences spoken by or movements performed by the robot). If the robot produces speech, natural language measures including the number of words [126], conciseness [19, 146, 168], and talking speed will impact this metric. A concise communication can be further qualified as one that is considered complete but minimally represented [84]. Transmission time for communication via physical or simulated robot motion in VR will be impacted by movement speed, distance traveled by the robot’s limbs, and the number of repetitions of movements [90]. Kwon et al. [90] used those measures to derive a cost metric when determining effective methods for communicating incapability with robot motions. Robot movements may have to be communicated at a particular speed if they pertain to the proficiency measure. For static visual displays on a screen or via AR, transmission time may be (near) instantaneous.

5.1 HUP: Perception

The first level of human understanding involves perceiving the communication. This is influenced by factors under the robot’s control, such as the volume used by the robot or the visibility of its movement to the human receiver. It is also influenced by factors outside of the robot’s control, such as ambient noise or environmental occlusions. Finally, it can be influenced by cognitive factors of the receiver, such as distractedness or a lack of shared context with the robot. In the Fetch scenario, the robot might communicate its proficiency using speech, such as “I am unlikely to succeed at retrieving the large screw.” Accurate perception of this statement would mean hearing all of the words correctly. If a visual communication method was used, accurate perception would require seeing the communication.

Perception clarity. The message should be perceived clearly by the human partner. Perception clarity can be measured by evaluating the accuracy of the received message; less accuracy means less clarity. For text- or speech-based communication methods, accuracy can be measured by using a read-back method in which the human repeats back the communication provided by the robot, measuring the deviation between the human’s read-back and the robot’s original message. This method is similar to the read-back/hear-back method employed by airline pilots and air traffic controllers [141]. For nonverbal communication such as robot motion, the human could repeat the motion instead (e.g., mimicking a hand signal). Another method for measuring perception clarity is to count the number of follow-up queries made by the human that ask the robot to repeat the communication or clarify what attributes were included in the communication (called “explainee return questions” in Madumal et al. [99]). For example, using the Fetch scenario, the human may inquire, “Did you say ‘small’ or ‘large’ screws?” This follow-up method is also used in subsequent HUP levels to evaluate comprehension clarity (Section 5.2) and projection clarity (Section 5.3). Characterizing the type of follow-up (whether for perception, comprehension, or projection clarity) may be difficult to objectively discern and is subject to interpretation. Follow-up communications from the human to the robot can be tracked at the RPH stage as a count of the number of communication events that occur (see Section 6.4).

Perception completeness. Measures of perception completeness compare the quantity of the information that is received compared to the quantity that is expected [87, 93]). At the perception level, this is a measure of how much signal is received by the human. For example, in the Fetch scenario, the human might only receive the first five words of the robot’s utterance (“I am unlikely to succeed”) and therefore wouldn’t have enough information to accurately perceive the communication. A human may or may not be aware that they have incompletely received a message. The amount of completeness could be assessed through explicit questions.

Perception time. This metric refers to the amount of time required for a human to receive the proficiency communication. Measures of communication complexity including clutter, comprehensiveness, and size from RCP will impact the perception time. For example, larger messages may take longer to receive due to longer transmission and reading times. Perception time can be measured using an end-of-message response, such as the start of a read-back or a simple acknowledgment of receipt. Perception time is not exclusively a feature of the communication, as it can be affected by the partner’s current cognitive processing capacity.

5.2 HUP: Comprehension

After perception, the robot’s proficiency communication must be processed and interpreted to extract task-relevant meaning. This may involve combining the current communication with prior ones to provide complete information. It may also involve resolving ambiguity in the communication, perhaps by using context, prior knowledge, or prior communications. For example, in the Fetch scenario, the robot can communicate its proficiency by saying, “I am unlikely to succeed at retrieving the large screw.” While there might be several screws on the table, the adjective “large” enables a human listener to resolve the potentially ambiguous reference, increasing the likelihood of successful comprehension. The robot might also point to the screw, multimodally communicating redundant information to increase comprehension.

Given that the levels of human understanding at the HUP stage are sequential, all metrics for comprehension will be impacted by those for perception (i.e., the human cannot comprehend that which they do not perceive). Even with perfect perception, comprehension can be affected by a variety of factors. Comprehension issues stem from a mismatch between how the robot has formulated its communication and how the human interprets it. This mismatch can arise from vocabulary-based ambiguity in the communication, such as using unfamiliar words, words with multiple meanings, or unclear referents like “that one” (i.e., in these examples, varying abstraction of the contextual reference attribute of the communication). Errors in comprehension can also arise when the robot incorrectly assumes the human has contextual information or prior knowledge that, when missing, affects the comprehension of the communication. In the example above, if the human only knew about small screws, they might incorrectly comprehend the robot’s statement to be about the small screws.

Comprehension clarity. The clarity of the proficiency communication at this level of HUP refers to the accuracy of human understanding. Assessments at this level largely consist of subjective ratings from the human and can be evaluated along several parameters that will impact overall comprehension clarity. The parameters deal with several factors of the interaction including comprehension of the properties of the communication, human expectations, awareness of goals, and the internal machinations of how proficiency measures were derived. Parameters of comprehension clarity include:

• Coherence. External coherence refers to whether the communication overlaps or fits with the human’s existing mental models, and internal coherence refers to how well the parts of a communication fit together with each other [169]. Each of the attributes possessed by a proficiency communication should be internally coherent with each other to increase comprehension.

• Comprehensibility. A comprehensible communication is one that is presented in a human understandable fashion, meaning in terms, structure, and semantics similar to that which a human expert would produce if provided the same input (adapted from [42, 105]).

• Explicability. Evaluations of explicability compare what the robot communicated regarding its proficiency to what the human had expected it to. This is adapted from explainable systems’ research that typically refers to an agent communicating its plan [17], so is most relevant for a priori and in situ communications. The use of explanatory actions during planning have been used to increase explicability of a robot’s actions [145].

• Fluency. The linguistic quality of a communication—sometimes referred to as naturalness—typically assessed automatically generated text or speech communications [52]. Evaluating this metric (and others related to natural language generation) as a relative comparison of communications rather than absolutely has been found to greatly improve consistency of the measure [112]. It should be noted that for the HUP stage of the proficiency-based interaction flow, fluency does not refer to the evaluation of collaborative activities between a human and a robot (as in [67]).

• Informativeness. This parameter is a measurement of how much informational content a communication possesses [52]. This can be evaluated in reference to the attributes the communication possesses, i.e., if some attributes are informative or if they are extraneous.

• Intelligibility. Evaluations of intelligibility measure the human’s understanding of how the robot’s model functions [8]. This is a similar goal of some transparent robot systems, such as those that aim to communicate the cause of incapability to the human [82]. This metric could also be evaluated objectively in comparison to how the robot actually functions.

• Legibility. The human’s comprehension of a proficiency communication is considered legible when the goal of the communication is not known (adapted from [17, 32]). This is sometimes also called readability or transparency. In the context of this article, the goal of a communication refers to what attributes of the robot’s proficiency it is attempting to communicate (e.g., contextual reference; see Section 4.3 for other attributes). This metric should only be evaluated in scenarios where the human is not aware of the robot’s communication goal, such as a priori to a task being performed or post hoc if the human did not observe any of the robot’s previous activities (i.e., scenarios wherein the human is less aware of what the robot might communicate). The legibility rating scale introduced in Dragan et al. [32] can be used for robot motion communication modalities and adapted for others.

• Predictability. If the human knows the goal the robot is attempting to communicate, predictability can be evaluated as to how the proficiency communication aligns with the human’s expectations (adapted from [17, 34]). The human may be able to predict what the robot’s proficiency communication will be post hoc to a task if they observed the robot’s previous activities. If the human has prior, repeated experience with the robot, predictability of the proficiency communication could be evaluated at any temporal level. A predictability scale is provided by Dragan and Srinivasa [35] that—similar to the legibility rating scale—is designed for subjective rating of robot motion but can be adapted for other communication modalities.

See Table 3 for examples of each parameter as applied to the Fetch scenario. Comprehension clarity can be objectively evaluated by counting the number of follow-up queries made by the human to the robot in order to clarify the meaning of a proficiency communication. These queries can be characterized according to the proficiency attributes they refer to (see Section 4.3). For example, using the Fetch scenario, a post hoc inquiry relevant to comprehension would be asking, “Is this the type of screw you had an issue with?” in order to clarify the contextual reference attribute of the proficiency communication. Comprehension clarity can also be identified through backchanneling actions, such as head nods, that indicate that the human understands what the robot is trying to communicate [64].

Comprehension completeness. At the comprehension level, completeness refers to the quantity of information understood by the human compared to the quantity that is expected (adapted from [87, 93]). For example, the human may query the robot in the Fetch scenario in situ and ask, “Which object are you trying to grab?” and will expect the robot’s responding communication to include contextual reference to the object. This is similar to explicability, which measures how close a robot’s plan aligns with the human’s expectations [17]. This definition can be adapted to refer to proficiency communications regarding future events that occur a priori or in situ to a task.

Communication consistency. Consistent communication refers to information presentation over time that has repeatable formatting and is compatible with prior information [84]. In the context of proficiency communication, consistency can be subjectively rated by the human or objectively evaluated along several axes according to the properties of the communication produced at the RCP stage (e.g., the communication modality used, transmission time, and the attributes it possesses). Consistency is similar to predictability in that evaluating it assumes the human is expecting the robot to communicate its proficiency in a certain way. In some explainability research, consistency is referred to as precision; a precise communication is one that the human believes to be sufficient and not surprising in relation to prior explanations [168].

Content quality. The quality of the content provided in the communication can be evaluated in terms of goodness or satisfaction, two terms that are used frequently throughout explainability research. Subjective evaluation scales can be used to assess these aspects of comprehension, such as the checklists provided by Hoffman et al. [70] where the human rates their level of agreement with various statements related to understanding, detail sufficiency, completeness, and trust. Several of the previously mentioned metrics have been shown to be predictive of subjective ratings of explanation quality including the possession of attributes like alternatives, completeness, and internal coherence [169]. Evaluations of this metric are impacted by the clarity metrics previously mentioned in that evaluations of poor content quality may be influenced by poor clarity.

Processing difficulty. Depending on factors like the communication’s clarity, quality, and timing, the human may face challenges in processing the conveyed information while attempting to comprehend it. Broadly used measures for evaluating cognitive workload in human-robot interaction can be utilized, including subjective ratings like the NASA-TLX scale [61] or objective evaluations such as the introduction of a secondary task [24] and biometric measures like the use of functional near-infrared spectroscopy (fNIRS) to classify workload of auditory or visual processing [118]. Difficulty in processing may also be evidenced by additional follow-up inquiries made by the human. These measures may also be used at the RPH stage as inputs to the robot’s perception of the human.

Comprehension time. This is the amount of time required for a human to make sense of a communication. Perception time (Section 5.1) plus comprehension time is the total amount of time that a human needs to understand a communication from a robot. In our conception, comprehension time starts when the human has fully received the message and ends when they fully understand the message’s content. In reality, perception and comprehension time often overlap, because people make sense of a communication as it arrives, rather than waiting for a complete communication to arrive before beginning to interpret it. The use of anticipatory motions prior to robot actions has been shown to increase reaction times when participants were tasked with labeling robot actions [54]. More complex communications will require more time to interpret, because they will require more cognitive processing. Verbal messages may need to be repeated back, textual statements may be read more than once, and visual displays may be inspected repeatedly by the human before the message is fully interpreted.

5.3 HUP: Projection

When interacting with a robot communicating its proficiency, projection refers to the human’s ability to use the proficiency communication to plan future actions and interactions between the robot and the human. In the Fetch scenario, if the robot has informed the human that it is unlikely to succeed at picking up the large screw, successful projection would involve adapting future actions to incorporate this new information. For example, the human might decide to pick up and deliver the screw themselves, rather than depending on the robot to do it. Alternately, the human might adapt the robot’s environment to make task completion more likely, such as by reorienting the screw to a position that is easier for the robot to grasp.

Even if the prior levels of HUP (perception and comprehension) are successful, a human partner can still make errors in projection if they have an inaccurate understanding of the task or the robot’s abilities. For example, the human might think that setting the screw upright would improve the likelihood of the robot completing the screw retrieval task, when in fact it has no bearing on the robot’s proficiency because the screw is just too big to fit in its gripper. Projection depends on the human’s model of why the robot’s self-assessed proficiency is low, and if this model is wrong, then the human’s projection will be ultimately incorrect. Having a robot provide its proficiency assessments at higher levels like explanation and prediction, described in Section 2.1, can help mitigate some of these errors.

As with perception and comprehension, projection can be measured explicitly and implicitly. Explicit measures might involve questionnaires for the human that examine their expectations of future robot behavior. Implicit measures include observations of what intervention a human chooses after receiving the robot’s communication (e.g., moving a problematic object, instructing the robot to perform the task differently) and whether their plans appear to change based on that communication. Implicit measures assume a model of the human’s reasoning (e.g., if they predict X, they will do Y). Mistakes in this human reasoning model will lead to inaccurate measurements of projection.

Expectations. This metric addresses how accurately a person can identify future outcomes. A robot’s communication of proficiency can influence this metric by helping improve (or harm) the accuracy of a person’s expectations. To measure expectations, people can be directly queried about their projection in the task (e.g., by asking, “What do you think the robot will do next?”). For example, in a study of multi-robot operation, researchers had participants operate several virtual UAVs simultaneously [124]. At two points during the task, they froze the simulation and asked participants to report which direction the robots were going to go next. This is an example of the situation awareness global assessment technique (SAGAT) [39], initially proposed for aviation.

Projection clarity. This metric refers to how easily a human is able to project into the future given what the robot has told them about its proficiency. It does not necessarily require the robot’s proficiency communication to be accurate, just that the human can use it to identify future outcomes. As such, an inaccurate robot proficiency assessment may lead to an overall inaccurate expectation (the previous metric) but high projection clarity if the human is able to make a confident (though ultimately incorrect) projection. In one study that measured projection clarity, researchers designed a game in which a robot took actions that conveyed its (hidden) skills, and a human then selected additional actions for the robot based on their understanding of the robot’s skills [128]. Projection clarity was reported as the number of participants who had greater than “neutral” ratings of confidence in their projections of the robot’s skills after the game. Projection clarity can also be measured similarly to perception and comprehension clarity by objectively evaluating the number of follow-up queries made by the human to the robot asking to clarify information about the future state of the task. In the Fetch scenario, the human may ask the robot, “If I adjust the orientation of the screw, will that help?”

Congruity. Human-robot congruity is a measure of how aligned human and robot mental models are. It is most effectively measured with respect to particular features of the agents’ mental models, such as their beliefs, goals, or expectations of rewards. For projection, congruity measures the similarity between human and robot expectations about future outcomes. For example, researchers have used techniques like cross-training to ensure that humans and robots have similar mental models of tasks, and then have measured the congruity of those mental models by calculating the decreasing uncertainty the robot has about observed human actions during training [111]. Note that this does not rely on either agent being correct in their mental model; congruity measures alignment, not accuracy.

Command changes. Command changes are human actions taken to augment the robot’s behavior based on the human’s projection level of understanding. The goal of initiating command changes is to remediate the situation and increase the robot’s future proficiency. This metric is reminiscent of the metric for human interventions defined for semi-autonomous operation in other HRI scenarios [151]. Command changes to robot behavior can initiate changes in robot action or the robot’s goal. Using the Fetch scenario, a change in action would be the human adjusting the order in which the robot fills in the bin (maybe starting with screws first if the screw placement obstructs the bolts), where a change in goal would be for the robot to deliver empty bins to the workstation and the human decides to gather the screws and bolts on their own. If changes occur, they can be tracked at the RPH stage as a number of correction or intervention events (see Section 6.4).

Environment changes. Changes to the environment are human actions in response to a proficiency communication that attempt to correct the situation by modifying conditions of the environment. For example, the robot in the Fetch scenario may not be able to deliver a bin to the workstation if the workstation already has a bin on it, so the human may remove that bin to modify the scene. Similar to command changes, if the robot communicates high proficiency, then a lack of environment changes may be indicative of proper projection. These changes can be tracked at the RPH stage as a count of the number of modification events (see Section 6.4).

6.1 RPH: Uncertainty and Coherence

Metrics for uncertainty and coherence evaluate how well model outputs, behaviors, and model representations are “working” in the context of a human-robot interaction.

Model consistency. Model consistency means that the predictions or estimates do not change too quickly or too slowly relative to the expectations associated with the task. Explicit models can output not only estimates of human goals, plans, and intentions [25, 106, 109, 143] but also predictions about human activities or behaviors [63, 96]. Model consistency metrics evaluate the rates at which these estimates and predictions change and compare those rates to expectations. Explicit models often include internal states such as beliefs about processes, representations about likely objects in the world, and so forth. If internal states change too quickly or too slowly relative to expectations, then model misalignment might be occurring.

Behavior consistency. Behavior consistency means that the robot’s actions (e.g., effector activity, speech acts) induced by either explicit models or implicit assumptions do not change too quickly or too slowly relative to expectations. Both overly consistent behaviors (never changing behaviors even when the world changes) or rapidly changing behaviors (thrashing through a set of actions) can indicate that the robot’s behaviors do not align with the human’s intent, because the robot is not responsive enough to human activity or not consistent enough to coordinate with intended interactions. For example, (1) reactive or rapidly changing responses by a robot in a shared workspace decrease fluency in human-robot teaming [68], (2) unpredictable or illegible drone behavior can induce fear or stress in human-drone interaction [156], and (3) compliance with social or proxemic norms has a strong impact on acceptance of robots in social interactions [155].

The human also generates action or communication behaviors. The consistency of human communication can indicate how well the robot’s models align with the human’s intention. For example, (1) misalignment can correlate with changes in sentiment across speech acts or chats [79], (2) disfluence [110] can indicate misalignment under natural language collaboration, (3) changing spatio-temporal references in shared workspaces can indicate misalignment [48], and (4) inconsistent deictic gestures can indicate misalignment in gesture-based collaboration.

Similarly, the consistency of human actions can also indicate the quality of alignment. For example, (1) operator-induced oscillations in shared or supervisory control can indicate failures by the robot to support intended human action [28, 115], (2) spatial distancing can indicate that a human is compensating for proxemic misalignment [107], (3) extraneous commands can indicate misalignment between human intent and perceived robot in learning by demonstration [85], and (4) violations of the neglect benevolence principle can indicate misalignment between human intent and perceived swarm behavior in human-swarm interaction [108].

Model uncertainty. Uncertainty means that there is variability of estimates, predictions, or internal representations of a robot. High levels of uncertainty can indicate misalignment between the robot’s models and the human’s intent either because the intent is unclear to the robot or the robot does not have internal representations that allow it to properly model the intent. Uncertainty is a well-known metric category, and therefore, a short list of relevant metrics suffices for this article. Uncertainty can be quantified using conventional methods like variance, confidence intervals, correlation quality, and the probability of outliers. Uncertainty can also be quantified using aggregation techniques such as behavioral entropy [10, 55, 166].

Model prediction accuracy. Prediction accuracy is the straightforward evaluation of how well a robot’s model corresponds to actual human activity when explicit behavioral predictions are made. Like uncertainty, accuracy is a well-known metric category, so a short list of metrics suffices. Prediction accuracy can be measured using conventional techniques such as the percentage of correct predictions, precision and recall, F1 scores, or ROC curves. Evaluation can be performed and data gathered post hoc (either after a mission is complete or between repeated tasks) to assess the alignment of the robot’s models of human intentions and human intentions. Post hoc analysis might identify overconfidence of in situ estimates or inconsistent performance across repeated tasks. When combined with data-driven approaches for refining robot behavior, indirect measures like convergence of inverse reinforcement learning models or persistence of learned reward structures can indicate alignment.

6.2 RPH: Performance

Proficiency should correlate with alignment of robot behaviors and human intent. Measures of performance can be used to provide in situ or post hoc information about the alignment of the robot’s models and the human’s intent when prior experience can be used to establish expectations about the values of the performance criteria. Simply put, low performance probably indicates that the robot did not contribute to solving the problem held by the human. Performance measures are not applicable a priori since performance measures depend on activity and outcomes.

For example, as mentioned in the RSA: Performance metrics (see Section 3.2), cumulative scores, rewards, and penalties can be assessed post hoc by the robot to determine how well a task was performed. Poor post hoc scores can indicate that the robot did not successfully accomplish or honor the human’s intention. To this end, the RSA: Performance and Events metrics are applicable at the RPH stage and so are not reiterated here. Similar metrics of mission progress, replanning triggers, and violation of performance envelopes provide information about the alignment between the robot’s models and the human’s intent if the human has explicitly stated a set of expectations about progress. Additionally, subjective scoring by the problem holder at the HUP stage (see Section 5) or by a set of experts using appropriate behavioral coding standards can be used to compute performance. Techniques like automatically generated performance summaries [135] facilitate post hoc subjective scoring. Low subjective scoring can indicate misalignment between robot behavior and human intent.

6.3 RPH: Time

Metrics related to time deal with the duration of phases of autonomy, time elapsed between signals of interest, or the perseveration of an activity. Time-based metrics are simple in the sense that they do not need access to predictions from the robot’s model nor access to the states in the robot’s internal model.

Persistence. Persistence-based metrics reflect the amount of time spent in a particular activity. Robot-oriented metrics include the amount of time the robot spends in a mission phase [134], at a particular location, or in a particular sub-task. Long periods of perseveration or very brief periods in an activity can indicate misalignment if the human assumes regular progress toward a goal [134]. Human-oriented metrics can include the amount of time a human fixates on a location in space [16, 148, 149] or menu of an interface [66] or works on the same sub-task. These metrics can indicate that the human is trying to figure out what the robot is doing, which indicates misalignment.

Expected execution time. The deviation between the robot’s expected task completion time and the human’s is a straightforward indication of how well robot behaviors align with human intent.

Coordination time. When tasks or missions can be repeated, the time required before or after a mission is executed can indicate the quality of alignment. A long debriefing time can indicate that there is misalignment between what the robot did at various phases of the mission and what the human thought was needed. Similarly, a long prebriefing time can indicate a priori a mismatch between the needs of the problem holder and the likely ability of the robot to be able to satisfy those needs without significant human input.

Event timing. The time elapsed between expected or planned events can indicate how well the robot’s behaviors align with human intent. Relevant events include communication acts, activities by either human or robot, subtask completion by either human or robot, or changes in performance indicators. Example metrics include (1) the frequency of interactions, (2) the pacing of communications such as mean time between connection events [122], (3) the time at which mileposts are reached or the order with which mileposts are reached [134], (4) the rate of change of instantaneous performance indicators [24], and (5) dwell time of eye gaze in interface-mediated or social interactions [2].

6.4 RPH: Events

Section 3 on RSA metrics notes that events can be defined in various ways and that counting events can be useful in understanding how well a robot is performing. In the context of a robot perceiving the human, relevant events can include actions initiated by the human to affect the robot’s behavior or actions initiated by the robot to gather information from the human. The metric types and metrics in this subsection are closely related to the definition of clarity given in Section 5.2; they are based on the assumption that a lack of clarity to a human manifests itself to the robot in various forms, with each form providing information about the alignment of robot behavior and human intention. Proper alignment may also be evidenced by a lack of interaction from the human if the robot communicates a high proficiency measure. For post hoc evaluations, events may be counted when debriefing and may include counting instances of overuse or disuse [92], as well as measures of transparency [21] or awareness violations [36].

Corrections. Corrections are human activities that repair or modify results from the robot’s behavior. Counting the number of corrections directly measures misalignment between robot models and human intent. When the relationship between the robot and the human follows a supervisory control mode, the number of undo commands also indicates misalignment. When the relationship between human and robot is more collaborative, the number of times the human undoes the robot’s work or redoes it in a different way indicates misalignment.

Interventions. Interventions are human activities that seek to alter the future behaviors of the robot, especially if those future behaviors are likely to produce poor results (i.e., command changes that occur at the HUP stage; see Section 5.3). Interventions can include changing the robot’s actions, altering the robot’s goals, or changing autonomy modes, such as the human stopping a robot to do the task without the robot or changing to a mode where the human has more control. Interventions can also include demonstrating tasks or giving negative or positive feedback [159]. Counting interventions is a metric that can indicate misalignment.

Modifications. Modifications are human activities that shape the environment or task setup (i.e., environment changes that occur at the HUP stage; see Section 5.3). Modifications can include task shaping such as cleaning up a room before a Roomba vacuums [49], placing objects within the reach of the robot’s gripper, adding RFID tags to important objects in the world [88], or changing lighting conditions to improve object recognition or mapping. Modifications can also include explicit adaptations that a human makes to a robot’s model, like labeling objects or tuning parameters to alter performance. Modifications are representative of a person’s understanding of the robot’s abilities. As such, when the dyad is in alignment, the human is making modifications that assist the robot—but when there is misalignment, modifications may not help the task completion by the robot. Due to this, both types of modifications should be reported.

Communications. Communication includes the number of speech acts, instant messages, interface requests, or queries initiated by either the robot or the human. This number depends on context because in some applications a lot of communication indicates that the robot understands the human’s intent (e.g., the team is rapidly communicating needed information in a timely way to enable task progress) and in others a lot of communication indicates poor understanding of the human’s intent (e.g., repeated requests for information or repeated instructions from a human). Communication can be as simple as clicks on an interface or gestures with the hand or mouse, or as sophisticated as natural language communication, eye gaze [2], deictic gesturing [103, 116], or social cueing [11]. Any follow-up inquiries made by the human at the HUP stage (see Section 5) can be counted toward this metric.

6.5 RPH: Human Factors

The alignment between the robot’s models and the human’s intent will impact the problem holder. Thus, there are a number of human factor measures that can provide information about the human’s experience throughout the phases of interaction. Information about the human’s experience reveals information about alignment. Since human factor metrics are well documented, relevant measures are simply listed. These measures include human workload (such as NASA TLX [61]), physiological indicators of workload or stress [158], real-time estimates of trust [31], real-time estimates of situation awareness [101, 170], and violations of human performance limitations [67]. High levels of workload, mistrust, stress, or frustration can indicate misalignment of robot behavior with human intentions.