Skip to main content
SpringerLink
Log in

A Generative Learning Approach to Sensor Fusion and Change Detection

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

We present a system for performing multi-sensor fusion that learns from experience, i.e., from training data and propose that learning methods are the most appropriate approaches to real-world fusion problems, since they are largely model-free and therefore suited for a variety of tasks, even where the underlying processes are not known with sufficient precision, or are too complex to treat analytically. In order to back our claim, we apply the system to simulated fusion tasks which are representative of real-world problems and which exhibit a variety of underlying probabilistic models and noise distributions. To perform a fair comparison, we study two additional ways of performing optimal fusion for these problems: empirical estimation of joint probability distributions and direct analytical calculation using Bayesian inference. We demonstrate that near-optimal fusion can indeed be learned and that learning is by far the most generic and resource-efficient alternative. In addition, we show that the generative learning approach we use is capable of improving its performance far beyond the Bayesian optimum by detecting and rejecting outliers and that it is capable to detect systematic changes in the input statistics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Ernst MO, Banks MS. Humans integrate visual and haptic information in a statistically optimal fashion. Nature. 2002;415(6870):429–33.

    Article  CAS  PubMed  Google Scholar 

  2. Angelaki DE, Gu Y, DeAngelis GC. Multisensory integration: psychophysics, neurophysiology, and computation. Curr Opin Neurobiol. 2009;19(4):452–8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Ernst MO, Blthoff HH. Merging the senses into a robust percept. Trends Cognit Sci. 2004;8(4):162–9.

    Article  Google Scholar 

  4. Beauchamp MS. See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr Opin Neurobiol. 2005;15(2):145–53.

    Article  CAS  PubMed  Google Scholar 

  5. Stein BE, Stanford TR. Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci. 2008;9(4):255–66.

    Article  CAS  PubMed  Google Scholar 

  6. Driver J, Noesselt T. Multisensory interplay reveals crossmodal influences on sensory-specific brain regions, neural responses, and judgments. Neuron. 2008;57(1):11–23.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Wallace MT. the development of multisensory processes. Cognit Process. 2004;5(2):69–83.

    Article  Google Scholar 

  8. Calvert GA, Thesen T. Multisensory integration: methodological approaches and emerging principles in the human brain. J Physiol Paris. 2004;98(1):191–205.

    Article  PubMed  Google Scholar 

  9. Ghazanfar AA, Schroeder CE. Is neocortex essentially multisensory? Trends Cognit Sci. 2006;10(6):278–85.

    Article  Google Scholar 

  10. Stratton GM. Vision without inversion of the retinal image. Psychol Rev. 1987;4(4):341.

    Article  Google Scholar 

  11. Howard IP, Templeton WB. Human spatial orientation. John Wiley & Sons; 1966.

  12. McGurk H, MacDonald J. Hearing lips and seeing voices. Nature. 1976;264:746–48.

    Google Scholar 

  13. Botvinick M, Cohen J. Rubber hands ’feel’ touch that eyes see. Nature. 1998;391(6669):756.

    Article  CAS  PubMed  Google Scholar 

  14. Shams L, Kamitani Y, Shimojo S. What you see is what you hear. Nature. 2000;408(6814):788.

    Article  CAS  PubMed  Google Scholar 

  15. King A. Development of multisensory spatial integration. Oxford University Press; 2004. p. 1–24.

  16. Gori M, Del Viva M, Sandini G, Burr DC. Young children do not integrate visual and haptic form information. Curr Biol. 2008;18(9):694–8.

    Article  CAS  PubMed  Google Scholar 

  17. Hecht T, Gepperth A. A generative-discriminative learning model for noisy information fusion. In: IEEE international conference on development and learning (ICDL); 2015.

  18. Gepperth A, Lefort M. Biologically inspired incremental learning for high-dimensional spaces. In: IEEE international conference on development and learning (ICDL); 2015.

  19. Gepperth A, Karaoguz C. A bio-inspired incremental learning architecture for applied perceptual problems. Cognit Comput.  2016 (in press).

  20. Gepperth A, Lefort M, Hecht T, Körner U. Resource-efficient incremental learning in high dimensions. In: European symposium on artificial neural networks (ESANN); 2015.

  21. Lefort M, Gepperth A. Active learning of local predictable representations with artificial curiosity. In: IEEE international conference on development and learning (ICDL); 2015.

  22. Gepperth A. Efficient online bootstrapping of representations. Neural Netw. 2012;41:39–50.

    Article  PubMed  Google Scholar 

  23. Kohonen T. Essentials of the self-organizing map. Neural Netw. 2013;37:52–65.

    Article  PubMed  Google Scholar 

  24. Martin JG, Meredith MA, Ahmad K. Modeling multisensory enhancement with self-organizing maps. Front Comput Neurosci. 2009;3:8.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Anastasio TJ, Patton PE. A two-stage unsupervised learning algorithm reproduces multisensory enhancement in a neural network model of the corticotectal system. J Neurosci. 2003;23(17):6713–27.

    CAS  PubMed  Google Scholar 

  26. Pavlou A, Casey M. Simulating the effects of cortical feedback in the superior colliculus with topographic maps. In: Neural Networks (IJCNN), the 2010 international joint conference on, IEEE; 2010. p. 1–8.

  27. Mayor J, Plunkett K. A neurocomputational account of taxonomic responding and fast mapping in early word learning. Psychol Rev. 2010;117(1):1.

    Article  PubMed  Google Scholar 

  28. Johannes B, Cornelius W, Stefan W. A som-based model for multi-sensory integration in the superior colliculus. In: Neural networks (IJCNN), the 2012 international joint conference on, IEEE; 2012. p. 1–8.

  29. Apostolos G, Haibo L, Mihaela G. An ensemble of SOM networks for document organization and retrieval. In: International conference on adaptive knowledge representation and reasoning (AKRR05); 2005. p. 6.

  30. Bruno B, Emilio C. A bio-inspired fusion method for data visualization. In: Hybrid artificial intelligence systems. Springer; 2010. p. 501–509.

  31. Yin H. ViSOM-a novel method for multivariate data projection and structure visualization. IEEE Trans Neural Netw. 2002;13(1):237–43.

    Article  PubMed  Google Scholar 

  32. Jantvik T, Gustafsson L, Papliski AP. A self-organized artificial neural network architecture for sensory integration with applications to letter-phoneme integration. Neural Comput. 2011;23(8):2101–39.

    Article  PubMed  Google Scholar 

  33. Valentina G, Julien M, Jon-Fan H, Plunkett K. The impact of labels on visual categorisation: a neural network model. In: Proceedings of the 30th Annual Conference of the Cognitive Science Society. Cognitive Science society; 2008.

  34. Landy MS, Banks MS, Knill DC. Ideal-observer models of cue integration. In: Sensory cue integration. Oxford University Press; 2011. p. 5–29.

  35. Knill DC, Pouget A. The bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 2004;27(12):712–9.

    Article  CAS  PubMed  Google Scholar 

  36. Jacobs RA. Optimal integration of texture and motion cues to depth. Vis Res. 1999;39(21):3621–9.

    Article  CAS  PubMed  Google Scholar 

  37. Battaglia PW, Jacobs RA, Aslin RN. Bayesian integration of visual and auditory signals for spatial localization. JOSA A. 2003;20(7):1391–7.

    Article  PubMed  Google Scholar 

  38. Ernst MO. A Bayesian view on multimodal cue integration. Knoblich G, Thornton IM, Grosjean M, Shiffrar M, editors. Human body perception from the inside out. New York: Oxford University Press; 2006, p. 105–131.

  39. Helbig HB, Ernst MO. Optimal integration of shape information from vision and touch. Exp Brain Res. 2007;179(4):595–606.

    Article  PubMed  Google Scholar 

  40. Makkook M, Basir O, Karray F. A reliability guided sensor fusion model for optimal weighting in multimodal systems. In: Acoustics, speech and signal processing, 2008. ICASSP 2008. IEEE international Conference on, IEEE; 2008. p. 2453–2456.

  41. Song X, Cui J, Zhao H, Zha H. Bayesian fusion of laser and vision for multiple people detection and tracking. In: SICE annual conference, IEEE 2008; 2008. p. 3014–3019.

  42. Klingbeil L, Reiner R, Romanovas M, Traechtler M, Manoli Y. Multi-modal sensor data and information fusion for localization in indoor environments. In: Positioning navigation and communication (WPNC), 2010 7th workshop on, IEEE; 2010. p. 187–192.

  43. Gepperth A, Dittes B, Garcia OM. The contribution of context information: a case study of object recognition in an intelligent car. Neurocomputing. 2012;94:77–86.

    Article  Google Scholar 

Download references

Acknowledgments

Thomas Hecht has received a research grant from the Direction Générale de l’Armement (DGA), France.

Author information

Authors and Affiliations

  1. U2IS, ENSTA ParisTech, INRIA, Université Paris-Saclay, 828 Boulevard des Marechaux, 91762, Palaiseau, France

    Alexander R. T. Gepperth & Thomas Hecht

  2. BITS Pilani - K K Birla Goa Campus, NH 17B, Zuarinagar, Goa, 403726, India

    Mandar Gogate

Authors
  1. Alexander R. T. Gepperth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Hecht
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mandar Gogate
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander R. T. Gepperth.

Ethics declarations

Conflict of interest

Alexander Gepperth, Thomas Hecht and Mandar Gogate declare that they have no conflict of interest.

Informed Consent

All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2008 (5). Additional informed consent was obtained from all patients for which identifying information is included in this article.

Human and Animal Rights

This article does not contain any studies with human participants or animals performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gepperth, A.R.T., Hecht, T. & Gogate, M. A Generative Learning Approach to Sensor Fusion and Change Detection. Cogn Comput 8, 806–817 (2016). https://doi.org/10.1007/s12559-016-9390-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-016-9390-z

Keywords

Search

Navigation