Vision and RFID data fusion for tracking people in crowds by a mobile robot

https://doi.org/10.1016/j.cviu.2010.01.008Get rights and content

Abstract

In this paper, we address the problem of realizing a human following task in a crowded environment. We consider an active perception system, consisting of a camera mounted on a pan-tilt unit and a 360° RFID detection system, both embedded on a mobile robot. To perform such a task, it is necessary to efficiently track humans in crowds. In a first step, we have dealt with this problem using the particle filtering framework because it enables the fusion of heterogeneous data, which improves the tracking robustness. In a second step, we have considered the problem of controlling the robot motion to make the robot follow the person of interest. To this aim, we have designed a multi-sensor-based control strategy based on the tracker outputs and on the RFID data. Finally, we have implemented the tracker and the control strategy on our robot. The obtained experimental results highlight the relevance of the developed perceptual functions. Possible extensions of this work are discussed at the end of the article.

Introduction

Giving a mobile robot the ability of automatically following a person appears to be a key issue to make it efficiently interact with humans. Numerous applications would benefit from such a capability. Service robotics is obviously one of these applications, as it requires interactive robots [16] able to follow a person to provide continual assistance in office buildings, museums, hospital environments, or even in shopping centers. Service robots clearly need to move in ways that are socially suitable for people. Such a robot have to localize its user, to discriminate him/her from others passers-by and to be able to follow him/her across complex human-centered environment. In this context, tracking a given person in crowds from a mobile platform appears to be fundamental. However, numerous difficulties arise: moving cameras with limited view field, cluttered background, illumination variations, hard real-time constraints, and so on.

The literature offers many tools to go beyond these difficulties. Our paper focuses on particle filtering framework as it easily enables to fuse heterogeneous data from embedded sensors. Despite their sporadicity, these dedicated person detectors and their hardware counterpart are very discriminant when present.

The paper is organized as follows. Section 2 depicts an overview of the corresponding works done within our robotic context and introduces our contributions. Section 3 describes our omnidirectional RFID prototype. This sensor is very discriminant when present in order to detect the user wearing an RFID tag. Section 4 recalls some PF basics and details our new importance function for multimodal person tracking. The developed control strategy to achieve a person following task in a crowded environment is detailed in Section 5, while Section 6 presents the mobile robot which has been used for our tests and the obtained results. Finally, Section 7 summarizes our contributions and discusses future extensions.

Section snippets

Overview and related work

Particle filters (PF) [5] through different schemes are currently investigated for person tracking in both robotics and vision communities. Besides the well-known CONDENSATION scheme, the fairly seldom exploited ICONDENSATION [26] variant steers sampling towards state space regions of high likelihood by incorporating both the dynamics and the measurements in the importance function. PF represent the posterior distribution by a set of samples, or particles, with associated importance weights.

Device description

The device consists of: (i) A CAENRFID3 A941 multiprotocol off-the-shelf reader which works at 870 MHz, with a programmable emitting RF power from 100 to 1200 mW. (ii) Eight directive antennas to detect the passive tags worn on the customer’s clothes. (iii) A prototype circuit in order to sequentially initialize each antenna (Fig. 1). With a single antenna, only a tag angle relative to the antenna plane can be estimated. With our eight antennas, the tag can be

Basics on particle filters and data fusion

Particle filters (PF) aim at recursively approximating the posterior probability density function (pdf) p(xk|z1:k) of the state vector xk at time k conditioned on the set of measurements z1:k=z1,,zk. A linear point-mass combinationp(xk|z1:k)i=1Nwk(i)δxk-xk(i),i=1Nwk(i)=1,is determined where δ(·) is the Dirac distribution. It expresses the selection of a value – or “particle” – xk(i) with probability – or “weight” – wk(i),i=1,,N. An approximation of the conditional expectation of any

A sensor-based control law for person following task

Now, we address the problem of making the robot follow the tagged person. To this aim, we use the data provided by both the tracker and the RFID system. We first briefly present the considered robotic system and the chosen control strategy, before detailing the different designed control laws.

Rackham description and software architecture

Rackham is an iRobot B21r mobile platform. Its standard equipment has been extended with one digital camera mounted on a Directed Perception pan-tilt unit, one ELO touch-screen, a pair of loudspeakers, an optical fiber gyroscope, wireless Ethernet and the RFID system previously described in Section 3 (Fig. 7). All these devices enable Rackham to act as a service robot in utilitarian public areas. It embeds robust Human Robot interaction abilities and efficient basic navigation skills.

We have

Conclusion

Tracking provides important capabilities for human robot interaction and assistance of humans in utilitarian populated spaces. The paper exhibits three contributions. A first contribution concerns the customization of an off-the-shelf RFID system to detect tags within a 360° view field and the coarse distance estimation thanks to the multiplexing of eight antennas. A second contribution concerns the development of a multimodal person tracker which combines the accuracy advantages of monocular

Acknowledgments

The authors are very grateful to Léo Bernard and Antoine Roguez for their involvements in this work which was partially conducted within the EU STREP Project Commrob funded by the European Commission Division FP6 under Contract FP6-045441.

References (47)

  • M. Bregonzio, M. Taj, A. Cavallaro, Multimodal particle filtering tracking using appearance, motion and audio...
  • L. Brèthes, F. Lerasle, P. Danès, M. Fontmarty, Particle filtering strategies for data fusion dedicated to visual...
  • D. Calisi, L. Iocchi, R. Leone, Person following through appearance models and stereo vision using a mobile robot, in:...
  • B. Castano, M. Rodriguez, An artificial intelligence and RFID system for people detection and orientation in big...
  • G. Cielniak, A. Lilienthal, T. Duckett, Improved data association and occlusion handling for vision-based people...
  • P.I. Corke

    Visual Control of Robots: High Performance Visual Servoing

    (1996)
  • B. Espiau et al.

    A new approach to visual servoing in robotics

    IEEE Transactions on Robotics and Automation

    (1992)
  • D.M. Gavrila

    Multi-cue pedestrian detection and tracking from a moving vehicle

    International Journal of Computer Vision (IJCV’07)

    (2008)
  • T. Germa et al.

    Video-based face recognition and tracking from a robot companion

    International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI’09)

    (2009)
  • R. Gockley, J. Forlizzi, R. Simmons, Natural person-following behavior for social robots, in: International Conference...
  • N.J. Gordon et al.

    Novel approach to nonlinear/non-gaussian bayesian state estimation

    Radar and Signal Processing IEE Proceedings F

    (1993)
  • D. Hahnel, W. Burgard, D. Fox, K. Fishkin, M. Philipose, Mapping and localization with RFID technology, in:...
  • C. Huang et al.

    High-performance rotation invariant multi-view face detection

    Transaction on Pattern Analysis Machine Intelligence (PAMI’07)

    (2007)
  • Cited by (79)

    • Efficient 6-DoF camera pose tracking with circular edges

      2023, Computer Vision and Image Understanding
    • Robot Systems for Rail Transit Applications

      2020, Robot Systems for Rail Transit Applications
    • Localizing RFIDs in Pixel Dimensions

      2022, ACM Transactions on Sensor Networks
    • FusIon Data Tracking System (FITS)

      2022, IEEE Sensors Journal
    • A Monocular Vision-Based Human-Following Approach for Mobile Robots

      2022, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    View all citing articles on Scopus
    View full text