Off-line simulation inspires insight: A neurodynamics approach to efficient robot task learning

Abstract

There is currently an increasing demand for robots able to acquire the sequential organization of tasks from social learning interactions with ordinary people. Interactive learning-by-demonstration and communication is a promising research topic in current robotics research. However, the efficient acquisition of generalized task representations that allow the robot to adapt to different users and contexts is a major challenge. In this paper, we present a dynamic neural field (DNF) model that is inspired by the hypothesis that the nervous system uses the off-line re-activation of initial memory traces to incrementally incorporate new information into structured knowledge. To achieve this, the model combines fast activation-based learning to robustly represent sequential information from single task demonstrations with slower, weight-based learning during internal simulations to establish longer-term associations between neural populations representing individual subtasks. The efficiency of the learning process is tested in an assembly paradigm in which the humanoid robot ARoS learns to construct a toy vehicle from its parts. User demonstrations with different serial orders together with the correction of initial prediction errors allow the robot to acquire generalized task knowledge about possible serial orders and the longer term dependencies between subgoals in very few social learning interactions. This success is shown in a joint action scenario in which ARoS uses the newly acquired assembly plan to construct the toy together with a human partner.

Introduction

Currently, new generations of robots are built that are supposed to interact closely with ordinary people in their working and living environments. These robots have to master a wide variety of everyday tasks that cannot be completely designed in advance by experts as in traditional industrial applications (Schaal, 2007). A major challenge of current robotics research is thus to endow robots with an adaptive, efficient and user-friendly instruction method that would allow ordinary people to teach the robot new tasks in an open-ended manner. Ideally, naïve users may bring their own learning experiences in social interactions with other people to the robotics domain. Learning by observing and imitating others behaviors and their consequences is a powerful social learning mechanism for human-to-human knowledge transfer (Bandura, 1971). It is attractive for the robotics domain as well since learning by observation significantly speeds up skill acquisition compared to individual discovery in potentially dangerous trial-and-error learning. While a full-blown, human-like social learning capacity for robots still remains a distant goal, major progress has been made over the last decade in various research directions of the programming by demonstration approach (for review papers and collections see e.g., (Billard et al., 2008, Dautenhahn and Nehaniv, 2002)). Most robotics experiments thus far have focused on the level of transferring motor skills for object manipulation from human to robot. Some other research started at the more abstract level of learning serial tasks defined by a sequence of subgoals or events in domains like assembly work (Ikeuchi & Suehiro, 1994), navigation (Nicolescu & Matarić, 2003) or household manipulation (Pardowitz, Knoop, Dillmann, & Zöllner, 2007). Recent developments stress the importance of social learning cues such as verbal feedback and communicative gestures to guide the real-time learning process in an incremental manner (Otero et al., 2008, Thomaz and Breazeal, 2008). However, little attention has been paid thus far on the generality of the acquired task knowledge, and the efficiency of the learning process in terms of the number of demonstrations needed (Pardowitz et al., 2007, Wu and Demiris, 2010). To act as an intelligent and flexible co-worker it is not sufficient that the robot memorizes a demonstrated task, such as, for instance assembling a furniture or laying the dinner table, as a simple linear sequence of events. The robot should be able to adapt the serial order of task execution to the preferences of different users, and at the same time should be able to understand and represent the task structure where the achievement of multiple independent subgoals may enable a final outcome. This in turn requires an ability to connect temporally nonadjacent subtasks. Very importantly, since multiple demonstrations of the same task would be time-consuming and annoying for users, the acquisition of generalized task knowledge in very few demonstrations is crucial for user acceptance.

In this paper we present a neurodynamics approach to robot task learning that takes inspiration from a hypothesis about how the nervous system might efficiently consolidate initial memory traces of sequential events into structured knowledge. Converging lines of evidence from computational theories (e.g., connectionist networks  McClelland, O’reilly, & McNaughton, 1995, for review see  O’Reilly & Norman, 2002) and neurophysiological studies (Euston, Tatsuno, & McNaughton, 2007, for review see Sutherland & McNaughton, 2000) support the notion of two complementary learning systems that allow distributed neural structures to gradually integrate new input patterns without disturbing previously stored information. A fast system is responsible for the rapid storage of newly demonstrated sequential events. The spontaneous re-activation of these memory traces during “off-line states”, in which the neural system is not processing external inputs, facilitates the gradual adjustment of synaptic weights in associative networks of the slow system encoding generalized knowledge of the sequential structure. To model the two complementary learning systems, we apply the theoretical framework of dynamic neural fields (DNFs) that has been proven in the past to provide key processing mechanisms for applications in cognitive modeling (Schöner, 2008) and in cognitive robotics (Erlhagen & Bicho, 2006). Most importantly, DNFs explain the existence of self-sustained activity in neural populations as the result of reciprocal positive feedback between neighboring neurons (Amari, 1977). Persistent activity has been reported in many areas of higher association cortices and is commonly believed to support a multitude of relevant cognitive functions such as working memory, decision making, and the learning of associations between events separated in time (Curtis and Lee, 2010, Miller, 2000). The intrinsically stable dynamics of population activity modeled by DNFs allows us not only to implement the short-term maintenance of task-relevant sequential information but also the active rehearsal of this information during off-line learning periods not constrained by the time course of observed events.

To test the learning model in real-world robotics experiments, we adopt a construction task that we have used in previous work to test a DNF architecture for natural and fluent human–robot interactions (HRI) (Bicho et al., 2011, Bicho et al., 2010). The main goal of the present study is to acquire in a social learning situation with human tutors the knowledge about possible serial orders of executing the assembly steps. This shared task knowledge was predefined by the designer in our earlier HRI studies. To this end, one or more human tutors first show the humanoid robot ARoS the assembly work consisting of a series of assembly steps necessary to construct a toy object from its parts. The tutor then provides immediate verbal feedback about predicted next steps when the robot tries to reproduce the serial order from memory. Demonstrations with different serial orders together with the integration of prediction errors in the associative learning process allow the robot to acquire generalized task knowledge in very few social learning interactions. This success is shown in a joint action scenario in which ARoS uses the newly acquired assembly plan to construct the toy together with a human partner.

The rest of the paper is structured as follows: Section  2 describes the construction task, the learning paradigm and the robotic platform ARoS. Section  3 presents an overview about basic processing principles of the dynamic neural field framework. Section  4 contains the description of the DNF based learning model. Experimental results are presented in Section  5. The paper finishes with a discussion of results and future work in Section  6.