A Tutorial to Use the MediaPipe Framework with ROS2

Abstract

This chapter presents a tutorial on the use of the MediaPipe package for ROS2, developed by the authors. Through the MediaPipe framework it is possible to collect data from the positions of the hand, face and body of an operator, using a webcam, and publish them in ROS2 topics, which can be used for various applications in robotics. A complete example application using this technology is also covered in this chapter. MediaPipe is used to collect data from the image of an operator’s pose (hands, fingers, face and body positions), and this information is then used to control the navigation of a robot inside a simulated environment. The programming language used is Python and the only hardware needed for execution is a computer with Linux Ubuntu 20.04, ROS2 Galactic and a webcam. At the end of the chapter, the reader will be able to develop control strategies or other functionalities that use information about the movements of the operator, captured directly by the developed ROS2 package, through a webcam. All files used in this chapter can be accessed from the link https://github.com/dmartinelli1997/media_pipe_ros2, as well as the project videos. All code used in this project can be found in the following repository: https://github.com/dmartinelli1997/Script_ROSBOOK2021.git. (This project was partially funded by National Counsil of Technological and Scientific Development of Brazil (CNPq), and by Coordination for the Improvement of Higher Level People (CAPES).) (This study was financed by the Human Resource Program of The Brazilian National Agency for Petroleum, Natural Gas, and Biofuels—PRH-ANP—FINEP Management, contractual instrument n$^{\circ }$ 0.1.19.0240.00, Ref 0431/19 - PRH21-UTFPR.)