Abstract
In the last few years, deep learning has attracted wide interest and achieved great success in many computer vision related applications, such as image classification, object detection, object tracking, pose estimation and action recognition. One specific application that can greatly benefit from the recent advance of deep learning is robot vision-based obstacle avoidance. Vision-based obstacle avoidance systems are mostly based on classification algorithms. Most of these algorithms use either color images or depth images as the main source of information. In this paper, the aim is to investigate whether using information extracted from both types of images simultaneously would give better performance than using each one separately. To do this, we chose the convolutional neural network (CNN) as the classifier and HSV-based method to achieve the fusion. We tested this approach using two widely used pre-trained CNN architectures, namely Resnet-50 and GoogLeNet using a dataset locally collected. The results indicate that the image fusion-based classification algorithm achieve a higher accuracy (91.3%) than the one based on depth images (80.4%) but lower than the one based on color images (93.7%). These results can be partly explained by the fact that the used classifiers were pre-trained using color image datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cornacchia M, Kakillioglu B, Zheng Y, Velipasalar S (2018) Deep learning-based obstacle detection and classification with portable uncalibrated patterned light. IEEE Sens J 18(20):8416–8425
Khan S, Rahmani H, Shah SAA, Bennamoun M (2018) A guide to convolutional neural networks for computer vision. Morgan & Claypool Publishers, San Rafael
Pujar K, Chickerur S, Patil MS (2017) Combining RGB and depth images for indoor scene classification using deep learning. In: IEEE international conference on computational intelligence and computing research, Coimbatore, India
Eitel A, Springenberg JT, Spinello L, Riedmiller M, Burgard W (2015) Multimodal deep learning for robust RGB-D object recognition. In: IEEE/RSJ international conference on intelligent robots and systems, Hamburg, Germany
Zeng H, Yang B, Wang X, Liu J, Fu D (2019) RGB-D object recognition using multi-modal deep neural network and DS evidence theory. Sensors 19:529. https://doi.org/10.3390/s19030529
Ophoff T, Van Beeck K, Goedemé T (2019) Exploring RGB + Depth fusion for real-time object detection. Sensors 19:866. https://doi.org/10.3390/s19040866
Al Jabri K (2018) Vision-based mobile robot obstacle-avoidance using deep learning algorithm, MSc Thesis, Department of Electrical and Computer Engineering, Sultan Qaboos University
He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. arXiv:1512.03385v1[cs.CV] 10 Dec 2015
Charreyron S, Jackson S, Miranda-Moreno L (2013) Toward a flexible system for pedestrian data collection with a microsoft Kinect motion-sensing device. Transp Res Record J Transp Res Board 2339(1):80–89
Koubaa A, Sriti M-F, Javed Y, Alajlan MM, Qureshi B, Qureshi B, Mahmoud A (2016) Turtlebot at office: a service-oriented software architecture for personal assistant robots using ROS. In: International conference on autonomous robot systems and competitions (ICARSC)
Ademovic A (2016) An introduction to robot operating system: the ultimate robot application framework. Developers
Iwaszczuk D, Koppanyi Z, Gard N, Zha B, Toth C, Yilmaz A (2018) Semantic labeling of structural elements in buildings by fusing RGB and depth images inn an encoder-decoder CNN framework. Int Arch Photogramm Remote Sens Spat Inf Sci 42(1):225–232
Iwaszczuk D, Koppanyi Z, Gard NA, Zha B, Toth C, Yilmaz A (2018) Semantic labeling of structural elements in buildings by fusing RGB and depth images in an encoder-decoder CNN framework. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol XLII-1
Powers DMW (2011) Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation. J Mach Learn Technol 2(1):37–63
Yang K, Qinami K, Fei-Fei L, Deng J, Russakovsky O (2019) Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy. IMAGNET, 17 Sept 2019
Acknowledgements
The authors would thank the Department of Electrical and Computer Engineering of Sultan Qaboos University for hosting the first author during the work on the project that let to this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Mechal, C.E., El Idrissi, N.E.A., Mesbah, M. (2022). CNN-Based Obstacle Avoidance Using RGB-Depth Image Fusion. In: Bennani, S., Lakhrissi, Y., Khaissidi, G., Mansouri, A., Khamlichi, Y. (eds) WITS 2020. Lecture Notes in Electrical Engineering, vol 745. Springer, Singapore. https://doi.org/10.1007/978-981-33-6893-4_78
Download citation
DOI: https://doi.org/10.1007/978-981-33-6893-4_78
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-33-6892-7
Online ISBN: 978-981-33-6893-4
eBook Packages: EngineeringEngineering (R0)