Vision and RFID data fusion for tracking people in crowds by a mobile robot
Introduction
Giving a mobile robot the ability of automatically following a person appears to be a key issue to make it efficiently interact with humans. Numerous applications would benefit from such a capability. Service robotics is obviously one of these applications, as it requires interactive robots [16] able to follow a person to provide continual assistance in office buildings, museums, hospital environments, or even in shopping centers. Service robots clearly need to move in ways that are socially suitable for people. Such a robot have to localize its user, to discriminate him/her from others passers-by and to be able to follow him/her across complex human-centered environment. In this context, tracking a given person in crowds from a mobile platform appears to be fundamental. However, numerous difficulties arise: moving cameras with limited view field, cluttered background, illumination variations, hard real-time constraints, and so on.
The literature offers many tools to go beyond these difficulties. Our paper focuses on particle filtering framework as it easily enables to fuse heterogeneous data from embedded sensors. Despite their sporadicity, these dedicated person detectors and their hardware counterpart are very discriminant when present.
The paper is organized as follows. Section 2 depicts an overview of the corresponding works done within our robotic context and introduces our contributions. Section 3 describes our omnidirectional RFID prototype. This sensor is very discriminant when present in order to detect the user wearing an RFID tag. Section 4 recalls some PF basics and details our new importance function for multimodal person tracking. The developed control strategy to achieve a person following task in a crowded environment is detailed in Section 5, while Section 6 presents the mobile robot which has been used for our tests and the obtained results. Finally, Section 7 summarizes our contributions and discusses future extensions.
Section snippets
Overview and related work
Particle filters (PF) [5] through different schemes are currently investigated for person tracking in both robotics and vision communities. Besides the well-known CONDENSATION scheme, the fairly seldom exploited ICONDENSATION [26] variant steers sampling towards state space regions of high likelihood by incorporating both the dynamics and the measurements in the importance function. PF represent the posterior distribution by a set of samples, or particles, with associated importance weights.
Device description
The device consists of: (i) A CAENRFID3 A941 multiprotocol off-the-shelf reader which works at 870 MHz, with a programmable emitting RF power from 100 to 1200 mW. (ii) Eight directive antennas to detect the passive tags worn on the customer’s clothes. (iii) A prototype circuit in order to sequentially initialize each antenna (Fig. 1). With a single antenna, only a tag angle relative to the antenna plane can be estimated. With our eight antennas, the tag can be
Basics on particle filters and data fusion
Particle filters (PF) aim at recursively approximating the posterior probability density function (pdf) of the state vector at time k conditioned on the set of measurements . A linear point-mass combinationis determined where is the Dirac distribution. It expresses the selection of a value – or “particle” – with probability – or “weight” – . An approximation of the conditional expectation of any
A sensor-based control law for person following task
Now, we address the problem of making the robot follow the tagged person. To this aim, we use the data provided by both the tracker and the RFID system. We first briefly present the considered robotic system and the chosen control strategy, before detailing the different designed control laws.
Rackham description and software architecture
Rackham is an iRobot B21r mobile platform. Its standard equipment has been extended with one digital camera mounted on a Directed Perception pan-tilt unit, one ELO touch-screen, a pair of loudspeakers, an optical fiber gyroscope, wireless Ethernet and the RFID system previously described in Section 3 (Fig. 7). All these devices enable Rackham to act as a service robot in utilitarian public areas. It embeds robust Human Robot interaction abilities and efficient basic navigation skills.
We have
Conclusion
Tracking provides important capabilities for human robot interaction and assistance of humans in utilitarian populated spaces. The paper exhibits three contributions. A first contribution concerns the customization of an off-the-shelf RFID system to detect tags within a 360° view field and the coarse distance estimation thanks to the multiplexing of eight antennas. A second contribution concerns the development of a multimodal person tracker which combines the accuracy advantages of monocular
Acknowledgments
The authors are very grateful to Léo Bernard and Antoine Roguez for their involvements in this work which was partially conducted within the EU STREP Project Commrob funded by the European Commission Division FP6 under Contract .
References (47)
- et al.
An approach to multi-modal human–machine interaction for intelligent service robot
Robotics and Autonomous Systems
(2003) - et al.
Multimodal tracking of people using laser scanners and video camera
Image and Vision Computing (IVC’08)
(2008) - et al.
A survey of socially interactive robots
Robotics and Autonomous Systems (RAS’03)
(2003) - et al.
Advances in vision algorithms and systems beyond the visible spectrum
Computer Vision and Image Understanding (CVIU’07)
(2007) - et al.
An architecture for autonomy
International Journal of Robotic Research (IJRR’98)
(1998) - M. Andriluka, S. Roth, B. Schiele, People-tracking by detection and people detection by tracking, in: International...
- M. Anne, J. Crowley, V. Devin, G. Privat, Localisation intra-bâtiment multi-technologies: RFID, wifi et vision, in:...
- K. Arras, O. Mozos, W. Burgard, Using boosted features for detection of people in 2D range scans, in: International...
- et al.
A tutorial on particle filters for on-line non-linear/non-gaussian bayesian tracking
Transaction on Signal Processing
(2002) - N. Bellotto, H. Hu, Vision and laser data fusion for tracking people with a mobile robot, in: International Conference...
Visual Control of Robots: High Performance Visual Servoing
A new approach to visual servoing in robotics
IEEE Transactions on Robotics and Automation
Multi-cue pedestrian detection and tracking from a moving vehicle
International Journal of Computer Vision (IJCV’07)
Video-based face recognition and tracking from a robot companion
International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI’09)
Novel approach to nonlinear/non-gaussian bayesian state estimation
Radar and Signal Processing IEE Proceedings F
High-performance rotation invariant multi-view face detection
Transaction on Pattern Analysis Machine Intelligence (PAMI’07)
Cited by (79)
Efficient 6-DoF camera pose tracking with circular edges
2023, Computer Vision and Image UnderstandingRobot Systems for Rail Transit Applications
2020, Robot Systems for Rail Transit ApplicationsHumans as path-finders for mobile robots using teach-by-showing navigation
2023, Autonomous RobotsLocalizing RFIDs in Pixel Dimensions
2022, ACM Transactions on Sensor NetworksFusIon Data Tracking System (FITS)
2022, IEEE Sensors JournalA Monocular Vision-Based Human-Following Approach for Mobile Robots
2022, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)