Skip to main content
SpringerLink
Log in

Brain-computer interface for human-multirobot strategic consensus with a differential world model

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In a distributed multi-robot system, the world model maintained by each robot is inconsistent due to measurement errors from onboard sensors, which will produce different and even incorrect strategies. In this paper, we propose an advanced interaction approach for human-multirobot strategic consensus. First, an opinion dynamics model is used to find the consistent multi-robot strategy, which is not necessarily the best choice due to the inaccurate world model. When the human receives the strategy from the robots, he/she can accept or reject it and reselect the strategy via a brain-computer interface (BCI). Of course, human judgment may be incorrect, and the BCI has false detections. Thus, the robots do not directly accept the human strategy but add it to the opinion dynamics model as a new node and recalculate the final consistent strategy. In addition, we developed a custom-designed simulation system based on the Robot Operating System and Gazebo to realize and evaluate the human-multirobot interaction. The extensive simulation results show that the proposed approach can significantly improve the correct rate of strategy selection compared with robot-only or human-only control, as well as the traditional human-robot interaction methods and other strategic consensus models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. The simulation system has been open source on the GitHub: https://github.com/nubot-nudt/BCI_Multi_Robot

References

  1. Asensio-Cubero J, Gan JQ, Palaniappan R (2016) Multiresolution analysis over graphs for a motor imagery based online bci game. Comput Biol Med 68:21–26

    Article  Google Scholar 

  2. Belda-Lois JM, Mena-del Horno S, Bermejo-Bosch I, Moreno J C, Pons J L, Farina D, Iosa M, Molinari M, Tamburella F, Ramos A et al (2011) Rehabilitation of gait after stroke: a review towards a top-down approach. J Neuroeng Rehabil 8(1):66

    Article  Google Scholar 

  3. Beverina F, Palmas G, Silvoni S, Piccione F, Giove S et al (2003) User adaptive bcis: Ssvep and p300 based interfaces. PsychNology J 1(4):331–354

    Google Scholar 

  4. Bi L, Fan X A, Liu Y (2013) Eeg-based brain-controlled mobile robots: a survey. IEEE Trans Hum-Mach Syst 43(2):161–176

    Article  Google Scholar 

  5. Bouldin D, Davies D L (1979) A cluster separation measure. IEEE Trans Pattern Anal Mach Intell 1(2):224–227

    Google Scholar 

  6. Cappo E A, Desai A, Collins M, Michael N (2018) Online planning for human–multi-robot interactive theatrical performance. Autonomous Robots

  7. Dai W, Lu H, Xiao J, Zeng Z, Zheng Z (2019) Multi-robot dynamic task allocation for exploration and destruction. J Intell Robot Syst: 1–25

  8. Dai W, Lu H, Xiao J, Zheng Z (2019) Task allocation without communication based on incomplete information game theory for multi-robot systems. J Intell Robot Syst 94(3–4):841–856

    Article  Google Scholar 

  9. Dragan A D, Srinivasa SS (2013) A policy-blending formalism for shared control. Int J Robot Res 32(7):790–805

    Article  Google Scholar 

  10. Erp J V, Lotte F, Tangermann M (2012) Brain-computer interfaces: beyond medical applications. IEEE Computer Society Press

  11. French JR Jr (1956) A formal theory of social power. Psychol Rev 63(3):181

    Article  Google Scholar 

  12. Friedkin N E (2015) The problem of social control and coordination of complex systems in sociology: a look at the community cleavage problem. IEEE Control Syst Mag 35(3):40–51

    Article  MathSciNet  Google Scholar 

  13. Gao X, Xu D, Cheng M, Gao S (2003) A bci-based environmental controller for the motion-disabled. IEEE Trans Neural Syst Rehabil Eng 11(2):137–140

    Article  Google Scholar 

  14. Han X, Lin K, Gao S, Gao X (2019) A novel system of ssvep-based human-robot coordination. J Neural Eng 16(1)

  15. Harary F (1959) A criterion for unanimity in French’s theory of social power

  16. Hegselmann R, Krause U et al (2002) Opinion dynamics and bounded confidence models, analysis, and simulation. J Artif Soc Social Simul 5(3)

  17. Hochberg L R, Bacher D, Jarosiewicz B, Masse N Y, Simeral J D, Vogel J, Haddadin S, Liu J, Cash S S, Van Der Smagt P et al (2012) Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485(7398):372

    Article  Google Scholar 

  18. Jennings N R (1995) Controlling cooperative problem solving in industrial multi-agent systems using joint intentions. Artif Intell 75(2):195–240

    Article  Google Scholar 

  19. Kirchner E A, Kim S K, Tabie M, Wöhrle H, Maurus M, Kirchner F (2016) An intelligent man-machine interface—multi-robot control adapted for task engagement based on single-trial detectability of p300. Front Hum Neurosci 10:291

    Article  Google Scholar 

  20. Kitano H, Asada M, Kuniyoshi Y, Noda I, Osawa E (1997) Robocup: the robot world cup initiative. In: Proceedings of the first international conference on autonomous agents. ACM, pp 340–347

  21. Kolling A, Walker P, Chakraborty N, Sycara K, Lewis M (2015) Human interaction with robot swarms: a survey. IEEE Trans Hum-Mach Syst 46(1):9–26

    Article  Google Scholar 

  22. Leeb R, Tonin L, Rohm M, Desideri L, Carlson T, Millan JDR (2015) Towards independence: a bci telepresence robot for people with severe motor disabilities. Proc IEEE 103(6):969– 982

    Article  Google Scholar 

  23. Liang S, Choi K S, Qin J, Pang W M, Heng P A (2016) Enhancing training performance for brain–computer interface with object-directed 3d visual guidance. Int J Comput Assist Radiol Surg 11 (11):2129–2137

    Article  Google Scholar 

  24. Lu H, Yang S, Zhang H, Zheng Z (2011) A robust omnidirectional vision sensor for soccer robots. Mechatronics 21(2):373–389

    Article  Google Scholar 

  25. Lu H, Li X, Zhang H, Hu M, Zheng Z (2013) Robust and real-time self-localization based on omnidirectional vision for soccer robots. Adv Robot 27(10):799–811

    Article  Google Scholar 

  26. McFarland D J, Wolpaw J R (2008) Brain-computer interface operation of robotic and prosthetic devices. Computer 41(10):52–56

    Article  Google Scholar 

  27. Mondada L, Karim M E, Mondada F (2016) Electroencephalography as implicit communication channel for proximal interaction between humans and robot swarms. Swarm Intell 10(4):247–265

    Article  Google Scholar 

  28. Onose G, Grozea C, Anghelescu A, Daia C, Sinescu C, Ciurea A, Spircu T, Mirea A, Andone I, Spânu A et al (2012) On the feasibility of using motor imagery eeg-based brain–computer interface in chronic tetraplegics for assistive robotic arm control: a clinical test and long-term post-trial follow-up. Spinal Cord 50(8):599

    Article  Google Scholar 

  29. Paggiaro A, Birbaumer N, Cavinato M, Turco C, Formaggio E, Del Felice A, Masiero S, Piccione F (2016) Magnetoencephalography in stroke recovery and rehabilitation. Front Neurol 7:35

    Article  Google Scholar 

  30. Peterson M (2017) An introduction to decision theory. Cambridge University Press, Cambridge

    Book  Google Scholar 

  31. Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner P, Muller-Putz G R (2010) Self-paced operation of an ssvep-based orthosis with and without an imagery-based “brain switch:” a feasibility study towards a hybrid bci. IEEE Trans Neural Syst Rehabil Eng 18(4):409–414

    Article  Google Scholar 

  32. Proskurnikov A V, Tempo R (2017) A tutorial on modeling and analysis of dynamic social networks. Part i. Ann Rev Control 43:65–79

    Article  Google Scholar 

  33. Rosenfeld A, Agmon N, Maksimov O, Kraus S (2017) Intelligent agent supporting human-multi-robot team collaboration. Artif Intell S0004370217301029

  34. Rzepecki J, Delcourt J, Da Silva M P, Le Callet P (2012) Virtual interactions: can eeg help make the difference with real interaction?. In: 2012 IEEE international conference on multimedia and expo workshops. IEEE, pp 151–156

  35. Saeedi S, Chavarriaga R, Leeb R, Millan JDR (2016) Adaptive assistance for brain-computer interfaces by online prediction of command reliability. IEEE Comput Intell Mag 11(1):32–39

    Article  Google Scholar 

  36. Sengupta P, Stalin John MR, Sridhar S S (2019) Classification of conscious, semi-conscious and minimally conscious state for medical assisting system using brain computer interface and deep neural network. J Med Robot ResS0004370217301029. https://doi.org/10.1142/S2424905X19420042

  37. Shang C, Fang H, Chen J, Zhang J (2018) Interacting with multi-agent systems through intention field based shared control methods. In: 2017 Asian control conference, ASCC 2017 2018-Janua, pp 150–155

  38. Van Erp J, Lotte F, Tangermann M (2012) Brain-computer interfaces: beyond medical applications. Computer 45(4):26–34

    Article  Google Scholar 

  39. Vidal J J (1973) Toward direct brain-computer communication. Ann Rev Biophys Bioeng 2 (1):157–180

    Article  Google Scholar 

  40. Wolpaw J R, Birbaumer N, Heetderks W J, McFarland D J, Peckham P H, Schalk G, Donchin E, Quatrano L A, Robinson C J, Vaughan T M (2000) Brain-computer interface technology: a review of the first international meeting. IEEE Trans Rehabil Eng 8(2):164–173

    Article  Google Scholar 

  41. Wolpaw J R, Birbaumer N, McFarland D J, Pfurtscheller G, Vaughan T M (2002) Brain–computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791

    Article  Google Scholar 

  42. Yao W, Dai W, Xiao J, Lu H, Zheng Z (2015) A simulation system based on ros and gazebo for robocup middle size league. In: 2015 IEEE international conference on robotics and biomimetics (ROBIO). IEEE, pp 54–59

  43. Ye D, Zhang M, Sutanto D (2013) Self-adaptation-based dynamic coalition formation in a distributed agent network: a mechanism and a brief survey. IEEE Trans Parallel Distrib Syst 24(5):1042–1051

    Article  Google Scholar 

  44. Ye M, Qin Y, Govaert A, Anderson D O B, Cao M (2019) An influence network model to study discrepancies in expressed and private opinions. Automatica

  45. Young B M, Nigogosyan Z, Nair V A, Walton L M, Song J, Tyler M E, Edwards D F, Caldera K, Sattin J A, Williams J C et al (2014) Case report: post-stroke interventional bci rehabilitation in an individual with preexisting sensorineural disability. Front Neuroeng 7:18

    Google Scholar 

  46. Yu Y, Zhou Z, Liu Y, Jiang J, Yin E, Zhang N, Wang Z, Liu Y, Wu X, Hu D (2017) Self-paced operation of a wheelchair based on a hybrid brain-computer interface combining motor imagery and p300 potential. IEEE Trans Neural Syst Rehabil Eng: 2516– 2526

  47. Yuan Y, Su W, Li Z, Shi G (2018) Brain–computer interface-based stochastic navigation and control of a semiautonomous mobile robot in indoor environments. IEEE Trans Cogn Dev Syst 11(1):129–141

    Article  Google Scholar 

  48. Zhang Y, Zhou G, Jin J, Wang M, Wang X, Cichocki A (2013) L1-regularized multiway canonical correlation analysis for ssvep-based bci. IEEE Trans Neural Syst Rehabil Eng 21(6):887– 896

    Article  Google Scholar 

  49. Zhu D, Bieger J, Molina G G, Aarts R M (2010) A survey of stimulation methods used in ssvep-based bcis. Comput Intell Neurosci 2010:1

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China [grant numbers U19A2083, U1813205, U1913202]; the National Key Research and Development Program [grant number 2017YFB1002505]; and the Hunan Provincial Innovation Foundation For Postgraduate [grant number CX2018B010]. The authors gratefully acknowledge this support.

Author information

Authors and Affiliations

  1. College of Intelligence Science and Technology, National University of Defense Technology, Changsha, Hunan, China

    Yaru Liu, Wei Dai, Huimin Lu, Yadong Liu & Zongtan Zhou

Authors
  1. Yaru Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huimin Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yadong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zongtan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zongtan Zhou.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Yaru Liu and Wei Dai contribute equally to this work.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(MP4 40.8 MB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Dai, W., Lu, H. et al. Brain-computer interface for human-multirobot strategic consensus with a differential world model. Appl Intell 51, 3645–3663 (2021). https://doi.org/10.1007/s10489-020-01963-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01963-2

Keywords

Search

Navigation