Skip to main content

Object Detection with Deep Learning for a Virtual Reality Based Training Simulator

  • Conference paper
  • First Online:
Computational Science and Its Applications – ICCSA 2018 (ICCSA 2018)

Abstract

Virtual Reality (VR) provides immersive user experience which makes them a cost effective solution to employ for various training purposes. However, a major shortcoming of VR systems is their limitation when it comes to interacting with the environment. Typically, when users wear a head mounted display their vision will be limited to virtual world and their external vision will be blocked. They will not be able to see useful objects in their environment such as controllers, buttons or even their hands. In this paper, we describe design of a training system for aerospace industry where real and virtual images blended, creating an augmented virtuality. The real world images are obtained from a camera mounted on the head-mounted-display. Some of the predefined objects, such as game controllers and user’s hands, are detected via deep learning algorithms and blended into the virtual reality images providing a more comfortable and immersive user experience. Furthermore, camera and object detection algorithms are employed to interact with VR headset making it more convenient tool for training simulators.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Kading, W.: The advanced Daimler-Benz driving simulator. SAE 9530012 (1995)

    Google Scholar 

  2. Soma, H., Hiramatsu, K.: Driving simulator experiment on drivers’ behaviour and effectiveness of danger warning against emergency braking of leading vehicle. In: Proceedings of 16th ESV, Canada (1998)

    Google Scholar 

  3. University of Valencia Forklift Simulator. http://www.uv.es/uvweb/university-research-institute-robotics-information-communication-technologies/en/research-groups/lsym/projects/forklift-truck-simulator-forklift-1285895484292/ProjecteInves.html?id=1285898571581. Accessed 20 Nov 2017

  4. Smart Mover Ultimate Car Simulator. http://thefutureofthings.com/5741-smart-mover-ultimate-car-simulator/. Accessed 15 Sept 2017

  5. University of Queensland Driving Simulator. http://www.carrsq.qut.edu.au/simulator/. Accessed 25 June 2017

  6. Virtual Driver. https://www.driverinteractive.com/. Accessed 12 Feb 2018

  7. Chen, L., Dayz, T.W., Tangx, W., John, N.W.: Recent developments and future challenges in medical mixed reality. In: Proceedings of 16th IEEE International Symposium on Mixed and Augmented Reality (2017)

    Google Scholar 

  8. Wyk, E.V., Villiers, R.: Virtual reality training applications for the mining industry. In: Proceedings of the 6th International Conference on Computer Graphics, Virtual Reality, Visualization and Interaction in Africa, Pretoria, South Africa, pp. 53–63 (2009)

    Google Scholar 

  9. L3 Aircraft Pushback. http://www.l-3training.com/applications/aviation/aircraft-pushback-training-simulator. Accessed 25 July 2017

  10. Milgram, P., Kishino, F.: A Taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. E77-D(12), 1321–1329 (1994)

    Google Scholar 

  11. Leap Motion. https://www.leapmotion.com/. Accessed 01 Sept 2017

  12. Wang, C., Liu, Z., Chan, S.C.: Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans. Multimed. 17(1), 29–39 (2015)

    Article  Google Scholar 

  13. Viola, P.A., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  14. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), San Diego, USA, pp. 886–893 (2005)

    Google Scholar 

  15. Lowe, D.G.: Distinctive image features from scale-invariant key points. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  16. Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.J.: SURF: Speeded Up Robust Features. Comput. Vis. Image Underst. 110(3), 346–359 (2008)

    Article  Google Scholar 

  17. Calonder, M., Lepetit, V., Özuysal, M., Trzcinski, T., Strecha, C., Fua, P.: BRIEF: computing a local binary descriptor very fast. IEEE Trans. Pattern Anal. Mach. Intell. 34(7), 1281–1298 (2012)

    Article  Google Scholar 

  18. Redmon, J., Farhadi, A.: YOLO9000: Better, Faster, Stronger. arXiv preprint arXiv:1612.08242 (2016). https://arxiv.org/pdf/1612.08242.pdf. Accessed 20 Nov 2017

  19. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: SSD: Single Shot Multibox Detector. arXiv preprint arXiv:1512:02325 (2015). https://arxiv.org/pdf/1512.02325.pdf. Accessed 20 Nov 2017

  20. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, https://arxiv.org/pdf/1704.04861.pdf. Accessed 20 Nov 2017

  21. Unity. https://unity3d.com/. Accessed 02 Dec 2017

  22. Object Detection Video Demo. https://youtu.be/GoR8qaWhzSM. Accessed 8 Mar 2018

  23. Project Video Demo. https://youtu.be/GnKbR5Z2bnI. Accessed 8 Mar 2018

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to M. Fikret Ercan .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ercan, M.F., Liu, Q., Amari, Y., Miyazaki, T. (2018). Object Detection with Deep Learning for a Virtual Reality Based Training Simulator. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2018. ICCSA 2018. Lecture Notes in Computer Science(), vol 10963. Springer, Cham. https://doi.org/10.1007/978-3-319-95171-3_57

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-95171-3_57

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-95170-6

  • Online ISBN: 978-3-319-95171-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics