Abstract
This paper describes a model of the multimodal speech and visual gesture (mSVG) control for aerobots operating at higher nCA autonomy levels, within the context of a patrol, search, and rescue application. The developed mSVG control architecture, its mathematical navigation model, and some high level command operation models were discussed. This was successfully tested using both MATLAB simulation and python based ROS Gazebo UAV simulations. Some limitations were identified, which formed the basis for the further works presented.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Green, S., Chen, X., Billinnghurst, M., Chase, J.G.: Human robot collaboration: an augmented reality approach a literature review and analysis. Mechatronics 5(1), 1–10 (2007)
Abioye, A.O., Prior, S.D., Thomas, G.T., Saddington, P.: The multimodal edge of human aerobotic interaction. In: Blashki, K., Xiao, Y. (eds.) International Conferences Interfaces and Human Computer Interaction, pp. 243–248. IADIS Press, Madeira (2016)
Abioye, A.O., Prior, S.D., Thomas, G.T., Saddington, P., Ramchurn, S.D.: Multimodal human aerobotic interaction. In: Isaías, P. (ed.) Smart Technology Applications in Business Environments, pp. 39–62. IGI Global (2017)
Root, S., Air Zermatt: The Matterhorn 101 - This is all you need to know about the Matterhorn (2016). https://www.redbull.com/int-en/the-horn-air-zermatt-matterhorn-rescue-team. Available 2016–10-17; Accessed 2017–06-07
Aeryon Labs Inc.: Whitepaper - intuitive control of a micro UAV (2011). https://aeryon.com/whitepaper/ituitivecontrol. First Available 2011–02-07; Accessed 2016–01-22
Fong, T., Nourbakhsh, I.: Interaction challenges in human-robot space exploration. In: Proceedings of the Fourth International Conference and Exposition on Robotics for Challenging Situations and Environments. Number January 2004, pp. 340–346 (2000)
Oviatt, S.: Multimodal interfaces. In: Jacko, J.A., Sears, A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, 1st edn, pp. 286–304. Lawrence Erlbaum Associates, Incorporated, London (2003)
Preece, J., Sharp, H., Rogers, Y.: Interaction Design: Beyond Human-Computer Interaction, 4th edn. Wiley, Glasgow (2015)
Turk, M.: Multimodal interaction: a review. Pattern Recognit. Lett. 36(1), 189–195 (2014)
Shah, J., Breazeal, C.: An empirical analysis of team coordination behaviors and action planning with application to human-robot teaming. Hum. Factors: J. Hum. Factors Ergon. Soc. 52(2), 234–245 (2010)
Bischoff, R., Graefe, V.: Dependable multimodal communication and interaction with robotic assistants. In: Proceedings - IEEE International Workshop on Robot and Human Interactive Communication, pp. 300–305 (2002)
Harris, J., Barber, D.: Speech and gesture interfaces for squad level human robot teaming. In: Karlsen, R.E., Gage, D.W., Shoemaker, C.M., Gerhart, G.R. (eds.) Unmanned Systems Technology Xvi, vol. 9084. SPIE (2014)
Redden, E.S., Carstens, C.B., Pettitt, R.A.: Intuitive Speech-based Robotic Control. U.S. Army Research Laboratory (Technical Report ARL-TR-5175) (2010)
Cacace, J., Finzi, A., Lippiello, V.: Multimodal Interaction with Multiple Co-located Drones in Search and Rescue Missions. CoRR abs/1605.0, pp. 1–6 (2016)
Lee, A., Kawahara, T., Shikano, K.: Julius an open source real-time large vocabulary recognition engine. In: Eurospeech, pp. 1691–1694 (2001)
Fernandez, R.A.S., Sanchez-lopez, J.L., Sampedro, C., Bavle, H., Molina, M., Campoy, P.: Natural user interfaces for human-drone multi-modal interaction. In: 2016 International Conference on Unmanned Aircraft Systems (ICUAS), Arlington, VA, USA, pp. 1013–1022. IEEE (2016)
Barber, D.J., Howard, T.M., Walter, M.R.: A multimodal interface for real-time soldier-robot teaming. 9837, 98370M (2016)
Borkowski, A., Siemiatkowska, B., Szklarski, J.: Towards semantic navigation in mobile robotics. In: Engels, G., Lewerentz, C., Schäfer, W., Schürr, A., Westfechtel, B. (eds.) Graph Transformations and Model-Driven Engineering. LNCS, vol. 5765, pp. 719–748. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-17322-6_30
Hill, S.G., Barber, D., Evans, A.W.: Achieving the vision of effective soldier-robot teaming : recent work in multimodal communication. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts, pp. 177–178 (2015)
Kattoju, R.K., Barber, D.J., Abich, J., Harris, J.: Technological evaluation of gesture and speech interfaces for enabling dismounted soldier-robot dialogue. 9837, 98370N (2016)
Ng, W.S., Sharlin, E.: Collocated interaction with flying robots. In: Proceedings - IEEE International Workshop on Robot and Human Interactive Communication, pp. 143–149 (2011)
Cauchard, J.R., Jane, L.E., Zhai, K.Y., Landay, J.A.: Drone & me: an exploration into natural human-drone interaction. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 361–365 (2015)
Obaid, M., Kistler, F., Kasparaviciute, G., Yantaç, A.E., Fjeld, M.: HowWould you gesture navigate a drone? A user-centered approach to control a drone. In: Proceedings of the 20th International Academic Mindtrek Conference, Tampere, Finland, pp. 113–121. ACM, New York (2016)
Nagi, J., Giusti, A., Gambardella, L.M., Di Caro, G.A.: Human-swarm interaction using spatial gestures. In: IEEE International Conference on Intelligent Robots and Systems (Iros), pp. 3834–3841 (2014)
Acknowledgement
This research was financially supported by the Petroleum Technology Development Fund (PTDF) of the Federal Government of Nigeria. Accessible via the following PTDF Reference Number: 16PHD052 and PTDF File Number: 862.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Abioye, A.O., Prior, S.D., Thomas, G.T., Saddington, P., Ramchurn, S.D. (2018). The Multimodal Speech and Visual Gesture (mSVG) Control Model for a Practical Patrol, Search, and Rescue Aerobot. In: Giuliani, M., Assaf, T., Giannaccini, M. (eds) Towards Autonomous Robotic Systems. TAROS 2018. Lecture Notes in Computer Science(), vol 10965. Springer, Cham. https://doi.org/10.1007/978-3-319-96728-8_36
Download citation
DOI: https://doi.org/10.1007/978-3-319-96728-8_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-96727-1
Online ISBN: 978-3-319-96728-8
eBook Packages: Computer ScienceComputer Science (R0)