Skip to main content
SpringerLink
Log in

Autonomous Perception and Grasp Generation Based on Multiple 3D Sensors and Deep Learning

  • Conference paper
  • First Online:
Intelligent and Safe Computer Systems in Control and Diagnostics (DPS 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 545))

Included in the following conference series:

  • 252 Accesses

Abstract

Grasping objects and manipulating them is the main way the robot interacts with its environment. However, for robots to operate in a dynamic environment, a system for determining the gripping position for objects in the scene is also required. For this purpose, neural networks segmenting the point cloud are usually applied. However, training such networks is very complex and their results are unsatisfactory. Therefore, we propose an innovative and end-to-end approach to generating the grip position that replaces (3D) point cloud segmentation with 2D image segmentation. For this purpose, we create an OrthoView module that acts as an adapter between 3D space (point cloud) and 2D space (image). The 2D object mask created in it serves as basic information in the process of selecting the final grip from among the grips generated for the entire scene using Contact GraspNet. An unquestionable conceptual advantage of our solution (OrthoView) is the fact that only one resulting 2D image is created from the point cloud, which can be the result of merging (integrating) 3D images from many cameras. Therefore, it allows for the fusion of information from any number of cameras, without the need to implement solutions for identifying the same objects seen from different perspectives (cameras). In order to test our solution stream, we created 6 scenes of different complexity, on the basis of which we present the effectiveness of our solution. Thus, eliminating the need for 3D point cloud segmentation and reducing the image segmentation problem to infer from only one 2D photo can significantly improve the performance of the position estimation system for grasping objects.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kocić, J., Jovičić, N., Drndarević, V.: Sensors and sensor fusion in autonomous vehicles. In: 2018 26th Telecommunications Forum (℡FOR), pp. 420–425 (2018). https://doi.org/10.1109/TELFOR.2018.8612054

  2. Zhu, P., Wen, L., Bian, X., Ling, H., Hu, Q.: Vision Meets Drones: A Challenge. arXiv:1804.07437 [cs] (2018)

  3. Ebert, D.M., Henrich, D.D.: Safe human-robot-cooperation: image-based collision detection for industrial robots. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 2, pp. 1826–1831 (2002). https://doi.org/10.1109/IRDS.2002.1044021

  4. Caltagirone, L., Bellone, M., Svensson, L., Wahde, M.: LIDAR–camera fusion for road detection using fully convolutional neural networks. Robot. Auton. Syst. 111, 125–131 (2019). https://doi.org/10.1016/j.robot.2018.11.002

    Article  Google Scholar 

  5. Banerjee, K., Notz, D., Windelen, J., Gavarraju, S., He, M.: Online camera LiDAR fusion and object detection on hybrid data for autonomous driving. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1632–1638 (2018). https://doi.org/10.1109/IVS.2018.8500699

  6. Qi, C.R., Su, H., Niessner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and Multi-View CNNs for Object Classification on 3D Data. arXiv:1604.03265 [cs] (2016)

  7. Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.: Multi-view Convolutional Neural Networks for 3D Shape Recognition. arXiv:1505.00880 [cs] (2015)

  8. Maturana, D., Scherer, S.: VoxNet: A 3D Convolutional Neural Network for real-time object recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922–928 (2015). https://doi.org/10.1109/IROS.2015.7353481

  9. Zhou, Y., Tuzel, O.: VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection. arXiv:1711.06396 [cs] (2017)

  10. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. arXiv:1612.00593 [cs] (2017)

  11. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. arXiv:1706.02413 [cs] (2017)

  12. Fooladgar, F., Kasaei, S.: Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images. arXiv:1912.11691 [cs] (2019)

  13. Chen, X., et al..: Bi-directional cross-modality feature propagation with separation-and-aggregation gate for RGB-D semantic segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12356, pp. 561–577. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_33

  14. Hazirbas, C., Ma, L., Domokos, C., Cremers, D.: FuseNet: incorporating depth into semantic segmentation via fusion-based cnn architecture. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10111, pp. 213–228. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54181-5_14

    Chapter  Google Scholar 

  15. Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-View 3D Object Detection Network for Autonomous Driving. arXiv:1611.07759 [cs] (2017)

  16. Li, B., Zhang, T., Xia, T.: Vehicle Detection from 3D Lidar Using Fully Convolutional Network. arXiv:1608.07916 [cs] (2016)

  17. Beltran, J., Guindel, C., Moreno, F.M., Cruzado, D., Garcia, F., de la Escalera, A.: BirdNet: a 3D Object Detection Framework from LiDAR information. arXiv:1805.01195 [cs] (2018)

  18. Zhou, Y., et al.: End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds. arXiv:1910.06528 [cs] (2019)

  19. Yang, B., Liang, M., Urtasun, R.: HDNET: Exploiting HD Maps for 3D Object Detection. arXiv:2012.11704 [cs] (2020)

  20. Wirges, S., Fischer, T., Frias, J.B., Stiller, C.: Object Detection and Classification in Occupancy Grid Maps using Deep Convolutional Networks. arXiv:1805.08689 [cs] (2018)

  21. Kowalczuk, Z., Szymański, K.: Classification of objects in the LIDAR point clouds using Deep Neural Networks based on the PointNet model. IFAC-PapersOnLine. 52, 416–421 (2019). https://doi.org/10.1016/j.ifacol.2019.08.099

    Article  Google Scholar 

  22. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. arXiv:1703.06870 [cs] (2018)

  23. He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs] (2015)

  24. Deng, X., Xiang, Y., Mousavian, A., Eppner, C., Bretl, T., Fox, D.: Self-supervised 6D object pose estimation for robot manipulation. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3665–3671 (2020). https://doi.org/10.1109/ICRA40945.2020.9196714

  25. Pas, A. ten, Gualtieri, M., Saenko, K., Platt, R.: Grasp Pose Detection in Point Clouds. arXiv:1706.09911 [cs] (2017)

  26. Mousavian, A., Eppner, C., Fox, D.: 6-DOF GraspNet: Variational Grasp Generation for Object Manipulation. arXiv:1905.10520 [cs] (2019)

  27. Eppner, C., Mousavian, A., Fox, D.: ACRONYM: A Large-Scale Grasp Dataset Based on Simulation. arXiv:2011.09584 [cs] (2020)

  28. Zhou, Q.-Y., Park, J., Koltun, V.: Open3D: A Modern Library for 3D Data Processing. arXiv:1801.09847 [cs] (2018)

  29. Kowalczuk, Z., Glinko, J.: Training of deep learning models using synthetic datasets. In: 2022 15th International Conference on Diagnostics of Processes and Systems (DPS). (POLSPAR,PAN), pp. 1–12 (2022)

    Google Scholar 

  30. Sundermeyer, M., Mousavian, A., Triebel, R., Fox, D.: Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes. arXiv:2103.14127 [cs] (2021)

Download references

Acknowledgements

This research was founded by The National Centre for Research and Development, grant no.: POIR.01.01.01-00-0833/18.

Author information

Authors and Affiliations

  1. Gdansk University of Technology, Gdansk, Poland

    Zdzisław Kowalczuk & Jan Glinko

  2. Avena Technologie, Gdansk, Poland

    Jan Glinko

Authors
  1. Zdzisław Kowalczuk
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jan Glinko
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zdzisław Kowalczuk .

Editor information

Editors and Affiliations

  1. Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Gdańsk, Poland

    Zdzislaw Kowalczuk

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kowalczuk, Z., Glinko, J. (2023). Autonomous Perception and Grasp Generation Based on Multiple 3D Sensors and Deep Learning. In: Kowalczuk, Z. (eds) Intelligent and Safe Computer Systems in Control and Diagnostics. DPS 2022. Lecture Notes in Networks and Systems, vol 545. Springer, Cham. https://doi.org/10.1007/978-3-031-16159-9_13

Download citation

Publish with us

Policies and ethics

Search

Navigation