Skip to main content
SpringerLink
Log in

Structure-based color learning on a mobile robot under changing illumination

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

A central goal of robotics and AI is to be able to deploy an agent to act autonomously in the real world over an extended period of time. To operate in the real world, autonomous robots rely on sensory information. Despite the potential richness of visual information from on-board cameras, many mobile robots continue to rely on non-visual sensors such as tactile sensors, sonar, and laser. This preference for relatively low-fidelity sensors can be attributed to, among other things, the characteristic requirement of real-time operation under limited computational resources. Illumination changes pose another big challenge. For true extended autonomy, an agent must be able to recognize for itself when to abandon its current model in favor of learning a new one; and how to learn in its current situation. We describe a self-contained vision system that works on-board a vision-based autonomous robot under varying illumination conditions. First, we present a baseline system capable of color segmentation and object recognition within the computational and memory constraints of the robot. This relies on manually labeled data and operates under constant and reasonably uniform illumination conditions. We then relax these limitations by introducing algorithms for (i) Autonomous planned color learning, where the robot uses the knowledge of its environment (position, size and shape of objects) to automatically generate a suitable motion sequence and learn the desired colors, and (ii) Illumination change detection and adaptation, where the robot recognizes for itself when the illumination conditions have changed sufficiently to warrant revising its knowledge of colors. Our algorithms are fully implemented and tested on the Sony ERS-7 Aibo robots.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Anzani, F., Bosisio, D., Matteucci, M., & Sorrenti, D. G. (2005). On-line color calibration in non-stationary environments. In The ninth international robocup symposium, Osaka, Japan, 18–19 July 2005.

  • Belongie, S., Malik, J., & Puzicha, J. (2002). Shape matching and object recognition using shape contexts. In Pattern analysis and machine intelligence.

  • Brainard, D. H., & Freeman, W. T. (1997). Bayesian color constancy. Journal of Optical Society of America A, 14(7), 1393–1411.

    Article  Google Scholar 

  • Brainard, D. H., & Wandell, B. A. (1986). Analysis of the retinex theory of color vision. Journal of Optical Society of America A, 3(10), 1651–1661.

    Article  Google Scholar 

  • Buchsbaum, G. (1980). A spatial processor model for object color perception. Journal of Franklin Institute, 310, 1–26.

    Article  Google Scholar 

  • Cameron, D., & Barnes, N. (2003), Knowledge-based autonomous dynamic color calibration, In The seventh international robocup symposium.

  • Chen, S., Siu, M., Vogelgesang, T., Yik, T. F., Hengst, B., Pham, S. B., & Sammut, C. (2002). RoboCup-2001: the fifth robocup competitions and conferences. Berlin: Springer.

    Google Scholar 

  • Cohen, D., Ooi, Y. H., Vernaza, P., & Lee, D. D. (2004). RoboCup-2003: the seventh robocup competitions and conferences. Berlin: Springer.

    Google Scholar 

  • Comaniciu, D., & Meer, P. (2002). Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5), 603–619.

    Article  Google Scholar 

  • Cormen, T. H., Leiserson, C. E., Rivest, R. L., & Stein, C. (2001). Introduction to algorithms (2nd ed.). Cambridge: MIT.

    MATH  Google Scholar 

  • Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley.

    MATH  Google Scholar 

  • Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern classification. New York: Wiley.

    Google Scholar 

  • Efron, B., & Tibshirani, R. J. (1993). An introduction to bootstrap. London: Chapman and Hall.

    MATH  Google Scholar 

  • Farshidi, F., Sirouspour, S., & Kirubarajan, T. (2005). Active multi-camera object recognition in presence of occlusion, In The IEEE international conference on intelligent robots and systems (IROS).

  • Finlayson, G. (1996). Color in perspective. IEEE Transactions of Pattern Analysis and Machine Intelligence, 18(10), 1034–1038.

    Article  Google Scholar 

  • Finlayson, G., & Hordley, S. (2000). Improving gamut mapping color constancy. IEEE Transactions on Image Processing, 9(10), 1174–1183.

    Article  Google Scholar 

  • Finlayson, G., Hordley, S., & Hubel, P. (2001). Color by correlation: A simple, unifying framework for color constancy, IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11).

  • Forsyth, D. (1990). A novel algorithm for color constancy. International Journal of Computer Vision, 5(1), 5–36.

    Article  Google Scholar 

  • Fred, A. L. N., & Jain, A. K. (2003). Robust data clustering, In The international conference of computer vision and pattern recognition (pp. 128–136), June 2003.

  • Gevers, T., & Smeulders, A. W. M. (1999). Color based object recognition. Pattern Recognition, 32(3), 453–464.

    Article  Google Scholar 

  • Gonzalez, R. C., & Woods, R. E. (2002). Digital image processing. New York: Prentice Hall.

    Google Scholar 

  • Hyams, J., Powell, M. W., & Murphy, R. R. (2000). Cooperative navigation of micro-rovers using color segmentation. Journal of Autonomous Robots, 9(1), 7–16.

    Article  Google Scholar 

  • Johnson, D. H., Gruner, C. M., Baggerly, K., & Seshagiri, C. (2001). Information-theoretic analysis of neural coding. Journal of Computational Neuroscience, 10, 47–69.

    Article  Google Scholar 

  • Jungel, M. (2004). Using layered color precision for a self-calibrating vision system. In The eighth international RoboCup symposium, Lisbon, Portugal.

  • Kitano, H., Asada, M., Noda, I., & Matsubara, H. (1998). Robot world cup. Robotics and Automation, 5(3), 30–36.

    Article  Google Scholar 

  • Land, E. H. (1977). The retinex theory of color constancy. Scientific American, 237, 108–129.

    Article  MathSciNet  Google Scholar 

  • Lauziere, Y. B., Gingras, D., & Ferrie, F. P. (1999). Color camera characterization with an application to detection under daylight. In Vision Interface (VI) (pp. 280–287), May 1999.

  • Lauziere, Y. B., Gingras, D., & Ferrie, F. P. (1999). Autonomous physics-based color learning under daylight. In The EUROPTO conference on polarization and color techniques in industrial inspection (vol. 3826, pp. 86–100).

  • Least Squares Line Fitting. http://mathworld.wolfram.com/LeastSquaresFitting.html.

  • Lenser, S., & Veloso, M. (2003). Automatic detection and response to environmental change. In The international conference of robotics and automation (ICRA), May 2003.

  • Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision (IJCV), 60(2), 91–110.

    Article  Google Scholar 

  • Minten, B. W., Murphy, R. R., Hyams, J., & Micire, M. (2001). Low-order-complexity vision-based docking. IEEE Transactions on Robotics and Automation, 17(6), 922–930.

    Article  Google Scholar 

  • Montgomery, D. C., & Runger, G. C. (2002). Applies statistics and probability for engineers (3rd ed.). New York: Wiley.

    Google Scholar 

  • Pantofaru, C., & Hebert, M. (2005). A comparison of image segmentation algorithms, cmu-ri-tr-05-40. Robotics Institute, Carnegie Mellon University, Tech. Rep., September 2005.

  • Pineau, J., Montemerlo, M., Pollack, M., Roy, N., & Thrun, S. (2003). Towards robotic assistants in nursing homes: challenges and results. In RAS special issue on socially interactive robots.

  • Rofer, T., & Jungel, M. (2003). Vision-based fast and reactive Monte-Carlo localization. In The IEEE international conference on robotics and automation (pp. 856–861), Taipei, Taiwan.

  • Rosenberg, C., Hebert, M., & Thrun, S. (2001). Color constancy using kl-divergence, In IEEE international conference on computer vision.

  • Rubner, Y., Tomasi, C., & Guibas, L. J. (2000). The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision, 40(2), 99–121.

    Article  MATH  Google Scholar 

  • Schulz, D., & Fox, D. (2004). Bayesian color estimation for adaptive vision-based robot localization. In The IEEE international conference on intelligent robots and systems (IROS).

  • Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation, In IEEE transactions on pattern analysis and machine intelligence (PAMI).

  • Sridharan, M., & Stone, P. (2004). Towards illumination invariance in the legged league. In The RoboCup symposium.

  • Sridharan, M., & Stone, P. (2005a). Autonomous color learning on a mobile robot. in The twentieth national conference on artificial intelligence (AAAI).

  • Sridharan, M., & Stone, P. (2005b). Real-time vision on a mobile robot platform. In The IEEE international conference on intelligent robots and systems (IROS).

  • Sridharan, M., & Stone, P. (2006). Autonomous planned color learning on a mobile robot without labeled data, In The ninth IEEE international conference on control, automation, robotics and vision (ICARCV), December 2006.

  • Sridharan, M., Kuhlmann, G., & Stone, P. (2005). Practical vision-based Monte Carlo localization on a legged robot. In The international conference on robotics and automation (ICRA), April 2005.

  • Stone, P., Dresner, K., Fidelman, P., Jong, N. K., Kohl, N., Kuhlmann, G., Lin, E., Sridharan, M., & Stronger, D. (2004). UT Austin Villa 2004: coming of age, TR 04-313. Department of Computer Sciences, UT-Austin, Tech. Rep., October 2004

  • Sumengen, B., Manjunath, B. S., & Kenney, C. (2003). Image segmentation using multi-region stability and edge strength, In The IEEE international conference on image processing (ICIP), September 2003.

  • Swain, M., & Ballard, D. H. (1991). Color indexing. International Journal of Computer Vision, 7(1), 11–32.

    Article  Google Scholar 

  • The Sony Aibo robots. (2004). http://www.sonystyle.com.

  • Thrun, S. (2006). Stanley: the robot that won the DARPA grand challenge. Journal of Field Robotics, 23(9), 661–692.

    Article  Google Scholar 

  • Thrun, S., Fox, D., Burgard, W., & Dellaert, F. (2001). Robust Monte Carlo localization for mobile robots. Journal of Artificial Intelligence, 128(1), 99–141.

    Article  MATH  Google Scholar 

  • Torralba, A., Murphy, K. P., & Freeman, W. T. (2004). Sharing visual features for multiclass and multiview object detection, In The IEEE Conference on computer vision and pattern recognition (CVPR), Washington, DC.

  • Trichevsky, R. E., & Trofimov, V. K. (1981). The performance of universal coding. IEEE Transactions of Information Theory, 27, 199–207.

    Article  Google Scholar 

  • Tsin, Y., Collins, R. T., Ramesh, V., & Kanade, T. (2001). Bayesian color constancy for outdoor object recognition. In IEEE pattern recognition and computer vision, December 2001.

  • Uther, W., Lenser, S., Bruce, J., Hock, M., & Veloso, M. (2001). Cm-pack’01: fast legged robot walking, robust localization, and team behaviors, In The fifth international robocup symposium, Seattle, USA.

Download references

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering, The University of Texas at Austin, 1, University Station, C0803, Austin, TX, 78712, USA

    Mohan Sridharan & Peter Stone

Authors
  1. Mohan Sridharan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Stone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohan Sridharan.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sridharan, M., Stone, P. Structure-based color learning on a mobile robot under changing illumination. Auton Robot 23, 161–182 (2007). https://doi.org/10.1007/s10514-007-9038-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-007-9038-7

Keywords

Search

Navigation