Coordinated control of a dual-arm robot for surgical instrument sorting tasks
Introduction
Surgical instrument sorting tasks, which are usually manually completed by the medical staff, are a crucial part of the overall management of surgical instruments. Because of the large number and special structure of surgical instruments, the manual processing method has certain drawbacks: it is time-consuming, poses an infection risk and has the potential for errors [1]. Moreover, most surgical instruments with special structures, such as joints, alveolus and crevices, increase the difficulty and intensity of manual sorting and pose a threat to the health of the medical staff. To address these challenges, a feasible solution is to develop a surgical instrument sorting system to accurately perform the sorting tasks, replacing the highly mechanized, repetitive and replaceable medical staff tasks [2].
The surgical scissors shown in Fig. 1, which are widely used surgical instruments for sorting tasks and challenging rigid objects under joint limit constraints [3], are chosen as the research objects in this paper. The surgical scissors to be sorted must be selected from a pile of surgical instruments [4], and unlocked and stretched by grasping on both sides for subsequent operations. Because surgical instruments are generally randomly placed, vision-based recognition and pose information extraction are needed to prepare for subsequent operations. Proposed sorting information extraction methods mainly include 3D reconstruction [5], feature construction and matching [6], and pose detection [7]. For example, Li et al. presented a recognition and location method based on stereo vision for surgical instrument sorting, increasing the sorting accuracy [8]. Xu et al. designed a recognition algorithm for a pile of surgical instruments using the QR core and used an affine transform method to estimate the poses [4], [9], [10]. The method proposed in this paper is more general and practical in the identification of surgical instruments and the detection of pose information.
Currently, most studies on the surgical instrument sorting task mainly focus on identification and classification, and few works have discussed the implementation of the sorting task. A dual-arm robot is more flexible and reliable than a single arm robot and satisfies the requirements of multiple operations [11]. A dual-arm robot can implement specific tasks, such as rescue activity [12], aerial work [13], [14], rehabilitation training [15], robot assembly [16] and so on. Some scholars have performed much research on dual-arm robots. For example, García et al. used the similarity index of cooperative tasks to carry out motion planning for a dual-arm robot system, improving the performance of cooperative tasks [17], [18]. Moreover, dual-arm robot research topics have led to increased research on humanoid robots [19], [20] and teleoperation systems [21], [22]. An adaptive neural control approach was proposed to reduce the influence of the output hysteresis of dual-arm coordination [20]. Talasaz et al. used two 7-DOF haptic wands as feedback units combined with visual force feedback to provide haptic feedback for a dual-arm teleoperation system [21]. Many studies have considered the adaptability of various factors in a robot’s operating environment. Dual-arm coordination can be even extended to a multi-arm robot, but the control complexity increases significantly [23], [24]. Hence, a dual-arm robotic system is proposed for surgical instrument sorting tasks. Compared with the previously mentioned works which rarely address implementation, the dual-arm robotic system introduced in this paper can implement the instrument sorting tasks more efficiently.
It is challenging for robots to implement the specific sorting task operations because the requirements of specific operations vary in different sorting environments. Generally, mechanical arms are subject to both structural (parameter) uncertainties and unstructured (model-free) uncertainties [25]. According to the general approximation theory [26], fuzzy logic systems can converge rapidly and approximately with nonlinear systems and adaptively adjust to changes in the operating environment [27], [28], [29], [30], [31]. Bhattacharyya et al. proposed a multi-class distinction combined with interval type-2 fuzzy logic and ANFIS, which dealt with the uncertainty of an electroencephalography (EEG) signal effectively [28]. Jiang et al. presented a robust fuzzy adaptive control method to lower the effect of dead-zone nonlinearity for a dual-arm robot. The hybrid fuzzy control strategy proposed in this paper takes position, velocity and force into account and establishes bilayer fuzzy controller, which can adaptively adjust the contact force during dual-arm coordinated operations.
This paper aims to address the implementation problem of surgical instrument sorting tasks. To design a feasible coordinated control strategy for dual-arm cooperative operations during instrument sorting tasks, three constraint relations of dual-arm coordinated motion tasks are defined. A hybrid force/position control strategy is then designed to enable the robot arms to coordinate their movement subject to constraints. Moreover, to adapt to the changing environment, a fuzzy control algorithm is integrated into the hybrid force/position control strategy to dynamically adjust the motion parameters.
The remainder of this paper is organized as follows. Section 2 introduces the visual extraction method for obtaining sorting information before the sorting operation. Section 3 presents the constraint analysis of the dual-arm coordinated operations. Three different constraint models and a passive compliance structure are introduced. Section 4 proposes the coordinated control strategy for the dual-arm robot, which combines a bilayer fuzzy hybrid force/position control method with a fuzzy control algorithm. Experiments and results are presented in Section 5. Section 6 presents the conclusion and discusses future works.
Section snippets
Sorting information extraction
In the sorting process, the position and posture of surgical instruments are random. Dual-arm robots should extract sorting information before attempting operations. In this section, instrument recognition and location are used to identify the types of surgical instruments and their locations. The grasping poses are obtained by detecting the grasping centre and the instruments’ principal axes.
Constraint analysis for coordinated operations
In this section, a constraint analysis is performed during the dual-arm coordinated operation and a passive compliant structure is presented to simplify the constrained motion of surgical scissor opening.
Coordinated control for dual-arm coordinate operations
Based on the results of the constraint analysis, a coordinated control strategy is proposed for the dual-arm robot in this section. The proposed control strategy is carried out in a master–slave control mode, in which the master arm adopts bilayer fuzzy force/position hybrid control, and the slave arm adopts position control. Moreover, a fuzzy control algorithm is integrated into the proposed control strategy to adapt to changing environments.
Experiments
This section presents the experiments to evaluate the proposed coordinate control strategy and discusses the experimental results.
Conclusion
The process of surgical scissor sorting involves the following essential tasks: unlocking the alveolar structure of the surgical scissors, stretching the surgical scissors to detect cleanliness and placing the surgical scissors. Dual-arm coordinated operation can easily cause damage to the surgical scissors, so it must be adaptively adjusted. Therefore, in this paper, the spatial position and best surgical instrument grasping pose are obtained through image recognition and detection, providing
Acknowledgements
This work was financially supported by the National Natural Science Foundation of China (Grant Nos. U1613224, U1713218), in part by the Shenzhen Key Laboratory Project, China (Grant No. ZDSYS201707271637577), Shenzhen Peacock Plan, China (No. KQTD2016113010571019) and Shenzhen Fundamental Research Funds, China (Grant Nos. JCYJ20170307170252420 and JCYJ20170413104438332).
Qihan Wu received the B.S. degree from China University of Mining and Technology, XuZhou, China, in 2016. And he is the M.S. student in Harbin Institute of Technology (Shenzhen), China. He currently does the research as a guest M.S. student in Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences from 2017. His research interests include dual-arm robot and motion planning and image processing algorithm.
References (33)
- et al.
Needle in a haystack: Interactive surgical instrument recognition through perception and manipulation
Robot. Auton. Syst.
(2017) - et al.
Dual-arm cooperative manipulation under joint limit constraints
Robot. Auton. Syst.
(2018) - et al.
Automatic registration of laser reflectance and colour intensity images for 3D reconstruction
Robot. Auton. Syst.
(2002) - et al.
How do ICP variants perform when used for scan matching terrain point clouds?
Robot. Auton. Syst.
(2017) - et al.
Dual arm manipulation—A survey
Robot. Auton. Syst.
(2012) - et al.
Interval type-2 fuzzy logic based multiclass ANFIS algorithm for real-time EEG based movement control of a robot arm
Robot. Auton. Syst.
(2015) - et al.
Fuzzy force control and state detection in vertebral lamina milling
Mechatronics
(2016) - et al.
Optimal path planning and execution for mobile robots using genetic algorithm and adaptive fuzzy-logic control
Robot. Auton. Syst.
(2017) - et al.
Validation of cleaning evaluation of surgical instruments with RFID tags attached based on cleaning appraisal judgment guidelines
- et al.
Robotic handling of surgical instruments in a cluttered tray
IEEE Trans. Autom. Sci. Eng.
(2015)
Invariant descriptors for 3 d object recognition and pose
IEEE Trans. Pattern Anal. Mach. Intell.
A surgical instruments sorting system based on stereo vision and impedance control
A vision-guided robot manipulator for surgical instrument singulation in a cluttered environment
An integrated vision-based robotic manipulation system for sorting surgical tools
Design and analysis of the dual arm manipulator for rescue robot
Image-based visual-impedance control of a dual-arm aerial manipulator
IEEE Robot. Autom. Lett.
Cited by (27)
Petri nets-based digital twin drives dual-arm cooperative manipulation
2023, Computers in IndustryActor–critic learning based coordinated control for a dual-arm robot with prescribed performance and unknown backlash-like hysteresis
2022, ISA TransactionsCitation Excerpt :In [5], the authors propose a finite-time adaptive fuzzy controller for an uncertain DAR. Moreover, the authors in [16] concentrate on designing a coordinated control method for a DAR to complete the specific surgical instrument sorting tasks. In [20], adaptive backstepping control with fuzzy approximation is investigated to handle the tracking problems for the dual arms of a humanoid robot.
A flexible and collaborative approach to robotic box-filling and item sorting
2021, Robotics and Autonomous SystemsCitation Excerpt :In such a way, the true potential of cobots cannot be demonstrated, which is fast adaptation to the variabilities arising from the environment and human co-workers. Several research works have aimed to improve the manipulation flexibility of cobots in performing repetitive tasks such as pick-and-place [3,4], sorting [5,6], and boxing [7], through adaptive control systems [8–10] that exploit sensory information such as vision [11–13], and force [14,15]. These operations are particularly important from a flexible automation perspective, since they are of a repetitive nature, and involve large body movements that can lead to musculoskeletal disorders in long term [16,17].
Research on robot sewing method based on process modeling
2024, International Journal of Intelligent Robotics and ApplicationsResearch Progress and Forward Hotspots of Vitreo-retinal Microsurgical Robot
2023, Jixie Gongcheng Xuebao/Journal of Mechanical Engineering
Qihan Wu received the B.S. degree from China University of Mining and Technology, XuZhou, China, in 2016. And he is the M.S. student in Harbin Institute of Technology (Shenzhen), China. He currently does the research as a guest M.S. student in Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences from 2017. His research interests include dual-arm robot and motion planning and image processing algorithm.
Meng Li received the B.S. degree from Shenyang Jianzhu University, Shenyang, China, in 2015, and the M.S. degrees from Harbin Institute of Technology (Shenzhen), China, in 2018. He is currently a research assistant in the Center for Cognitive Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China. His current research interests include the combination of machine learning and medical assistant robots.
Xiaozhi Qi received the B.S. degree from Yanshan University, in 2009, and the M.S. and Ph.D. degrees from Harbin Institute of Technology (Shenzhen), China, in 2011 and 2017, respectively. He is currently a postdoctor in the Center for Cognitive Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China. His current research interests include medical assistant robots and space deployable mechanisms.
Ying Hu (M’11) received the B.S. degree from Shanghai Jiaotong University, Shanghai, China, in 1991, and the M.S. and Ph.D. degrees in mechanical engineering from Harbin Institute of Technology, Shenzhen, China, in 1998 and 2007, respectively. She is currently a Professor in the Center for Cognitive Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China. She is the author or coauthor of more than 60 scientific papers published in refereed journals and conference proceedings. Her research interests include parallel robots, medical assistant robots, and mobile robots.
Bing Li (SM’16) received the Ph.D. degree from Hong Kong Polytechnic University, Hong Kong, in 2001. He was a Professor of Mechatronics in 2006. He is currently the Head of the School of Mechanical Engineering and Automation with the Harbin Institute of Technology (Shenzhen), China. His research interests include parallel manipulators and control, mechanical vibration and control. Prof. Li is serving as an Associate Editor of the International Journal of Mechanisms and Robotic Systems.
Jianwei Zhang (M’91) received the B.S. and M.S. degrees from Department of Computer Science, Tsinghua University, Beijing, China, in 1986 and 1989 respectively, and the Ph.D. degree from the Department of Computer Science, Institute of Real-Time Computer Systems and Robotics, University of Karlsruhe, Karlsruhe, Germany, in 1994. He is currently a Professor and Head of the TAMS Group, University of Hamburg, Hamburg, Germany. He has published more than 200 journal and conference papers, technical reports, four book chapters, and two research monographs. His research interests include multimodal perception, robot learning, and mobile service robots. Dr. Zhang has received several awards, including the IEEE ROMAN and IEEE AIM Best Paper Awards.