Abstract
Virtual Reality (VR) provides immersive user experience which makes them a cost effective solution to employ for various training purposes. However, a major shortcoming of VR systems is their limitation when it comes to interacting with the environment. Typically, when users wear a head mounted display their vision will be limited to virtual world and their external vision will be blocked. They will not be able to see useful objects in their environment such as controllers, buttons or even their hands. In this paper, we describe design of a training system for aerospace industry where real and virtual images blended, creating an augmented virtuality. The real world images are obtained from a camera mounted on the head-mounted-display. Some of the predefined objects, such as game controllers and user’s hands, are detected via deep learning algorithms and blended into the virtual reality images providing a more comfortable and immersive user experience. Furthermore, camera and object detection algorithms are employed to interact with VR headset making it more convenient tool for training simulators.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Kading, W.: The advanced Daimler-Benz driving simulator. SAE 9530012 (1995)
Soma, H., Hiramatsu, K.: Driving simulator experiment on drivers’ behaviour and effectiveness of danger warning against emergency braking of leading vehicle. In: Proceedings of 16th ESV, Canada (1998)
University of Valencia Forklift Simulator. http://www.uv.es/uvweb/university-research-institute-robotics-information-communication-technologies/en/research-groups/lsym/projects/forklift-truck-simulator-forklift-1285895484292/ProjecteInves.html?id=1285898571581. Accessed 20 Nov 2017
Smart Mover Ultimate Car Simulator. http://thefutureofthings.com/5741-smart-mover-ultimate-car-simulator/. Accessed 15 Sept 2017
University of Queensland Driving Simulator. http://www.carrsq.qut.edu.au/simulator/. Accessed 25 June 2017
Virtual Driver. https://www.driverinteractive.com/. Accessed 12 Feb 2018
Chen, L., Dayz, T.W., Tangx, W., John, N.W.: Recent developments and future challenges in medical mixed reality. In: Proceedings of 16th IEEE International Symposium on Mixed and Augmented Reality (2017)
Wyk, E.V., Villiers, R.: Virtual reality training applications for the mining industry. In: Proceedings of the 6th International Conference on Computer Graphics, Virtual Reality, Visualization and Interaction in Africa, Pretoria, South Africa, pp. 53–63 (2009)
L3 Aircraft Pushback. http://www.l-3training.com/applications/aviation/aircraft-pushback-training-simulator. Accessed 25 July 2017
Milgram, P., Kishino, F.: A Taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. E77-D(12), 1321–1329 (1994)
Leap Motion. https://www.leapmotion.com/. Accessed 01 Sept 2017
Wang, C., Liu, Z., Chan, S.C.: Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans. Multimed. 17(1), 29–39 (2015)
Viola, P.A., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), San Diego, USA, pp. 886–893 (2005)
Lowe, D.G.: Distinctive image features from scale-invariant key points. Int. J. Comput. Vis. 60(2), 91–110 (2004)
Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.J.: SURF: Speeded Up Robust Features. Comput. Vis. Image Underst. 110(3), 346–359 (2008)
Calonder, M., Lepetit, V., Özuysal, M., Trzcinski, T., Strecha, C., Fua, P.: BRIEF: computing a local binary descriptor very fast. IEEE Trans. Pattern Anal. Mach. Intell. 34(7), 1281–1298 (2012)
Redmon, J., Farhadi, A.: YOLO9000: Better, Faster, Stronger. arXiv preprint arXiv:1612.08242 (2016). https://arxiv.org/pdf/1612.08242.pdf. Accessed 20 Nov 2017
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: SSD: Single Shot Multibox Detector. arXiv preprint arXiv:1512:02325 (2015). https://arxiv.org/pdf/1512.02325.pdf. Accessed 20 Nov 2017
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, https://arxiv.org/pdf/1704.04861.pdf. Accessed 20 Nov 2017
Unity. https://unity3d.com/. Accessed 02 Dec 2017
Object Detection Video Demo. https://youtu.be/GoR8qaWhzSM. Accessed 8 Mar 2018
Project Video Demo. https://youtu.be/GnKbR5Z2bnI. Accessed 8 Mar 2018
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Ercan, M.F., Liu, Q., Amari, Y., Miyazaki, T. (2018). Object Detection with Deep Learning for a Virtual Reality Based Training Simulator. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2018. ICCSA 2018. Lecture Notes in Computer Science(), vol 10963. Springer, Cham. https://doi.org/10.1007/978-3-319-95171-3_57
Download citation
DOI: https://doi.org/10.1007/978-3-319-95171-3_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-95170-6
Online ISBN: 978-3-319-95171-3
eBook Packages: Computer ScienceComputer Science (R0)