Improving teleoperation system performance in the presence of estimated external force

doi:10.1016/j.rcim.2016.12.004

Robotics and Computer-Integrated Manufacturing

Volume 46, August 2017, Pages 86-93

https://doi.org/10.1016/j.rcim.2016.12.004 Get rights and content

Highlights

•
Design a control scheme for bilateral teleoperation via estimated external force.
•
Transparency improvement of the system in contact condition.
•
Considering the effect of time delays in communication channels between master and slave robot.
•
Demonstrating stability of closed loop system theoretically and experimentally.

Abstract

Teleoperation systems have been developed in order to manipulate objects in environments where the presence of humans is impossible, dangerous or less effective. The presence of force signals plus position signals in the control scheme could effectively improve transparency. However, the main restriction is force measurement in some applications. A new modified strategy for estimating the external forces acting on the master and slave robots is developed. The main advantage of this strategy is that the necessity for force sensors is eliminated, leading to lower cost and further applicability. A novel control algorithm with estimated force signals is proposed for a general nonlinear bilateral teleoperation system with time delay. The stability condition in the teleoperation system with the new control algorithm is verified by means of Lyapunov stability analysis. The designed control algorithm guarantees stability of the teleoperation system in the presence of an estimated operator and environmental force. Experimental results confirm the efficiency of the novel control algorithm in position tracking and force reflection.

Introduction

Teleoperation systems have become an extensive and interesting field for researchers in the last decade. The main function of teleoperation systems is to operate from a remote location [1], [2]. A useful application of teleoperation systems is to control a robotic vehicle in hazardous situations. Other applications include telesurgery, space technology and underwater exploration [3], [4], [5]. In bilateral teleoperation systems, the main purpose is to control the remote manipulator and sense the forces exerted on the robot in a remote environment. Therefore, stability and transparency are two prominent objectives that should be satisfied. Transparency in a teleoperation system is investigated by measuring the impedance, which is transmitted to the operator through the environment.

Conventional teleoperation system controllers use position and velocity signals of master and slave robots [6], [7], [8], [9]. In these approaches, position tracking is disturbed when the slave robot is in contact with the environment. In fact, position error between the master and slave is inevitable in these control schemes. Such error deteriorates the transparency of the closed-loop teleoperation system.

Transparency can be enhanced if force signals are transmitted in addition to position and velocity signals. Therefore, a four-channel architecture teleoperation system is ideal to achieve both position tracking and force reflection simultaneously, since the operator would have a better sense of the environment.

To improve system transparency, force signals have been applied in the control structure in addition to position and velocity signals [10], [11], [12]. In these schemes, bilateral teleoperation system stability is considered. Stability analysis is only achieved for the linear bilateral teleoperation system; but in many robotic tasks the dynamic model is nonlinear. However, measuring external force signals in various robotic tasks is also not feasible due to the high cost of force sensors. The associated noises are also high and force sensor installation may be difficult.

To overcome these drawbacks, several works have been done in the case of robotic manipulators to estimate external forces [13], [14], [15], [16], [17]. In these researches external force exerted on the robotic manipulators was estimated and imported into the control scheme; however, for some major reasons the employed algorithms are not applicable for teleoperation systems. In robotic manipulators, external force was estimated based on the mentioned algorithms, then it was employed on the control scheme, and subsequently, system stability was taken into account with the proposed controller. In contrast with robotic manipulators, teleoperation systems include three sections: master robot, slave robot and communication channels. All three sections affect system stability, therefore the effects must be considered in control scheme design to achieve proper stability and transparency in closed-loop systems. Thus, due to this significant difference between robotic manipulators and teleoperation systems, it is not possible to apply every force estimation algorithm or control scheme for teleoperation systems. For example, time delay is an undesirable problem associated only with communication channels in teleoperation systems. This issue can deteriorate system stability and transparency, therefore it must be considered in control scheme and force estimation algorithm to attain appropriate performance.

Several researchers have proposed force estimation algorithms to eliminate force signal measurement in teleoperation systems [18], [19], [20], [21], [22], [23], [24]. The estimation of external forces is presented on the master and slave in combination with a modified version of the sliding mode bilateral teleoperation algorithm [18], [19]. The main drawback is that the force reflecting from the closed-loop system is not considered, whereas force reflection is one of the most important purposes of bilateral teleoperation systems. A synchronization scheme of bilateral teleoperation by using a state observer has also been proposed for improvement [20]. However, reliable force synchronization has not been achieved. In [21] it was experimentally proven that external force is estimated but it is not analytically demonstrated. The other drawback is that transparency in the control system is not considered. Another problem concerning these force estimation algorithms is that employing them may not guarantee the stability of closed-loop systems.

In [22], [23] a sliding-mode based control scheme with force estimation algorithm for linear bilateral teleoperation system was proposed. A force estimation algorithm was designed for time variant human and environment forces. It was proved that the force estimation error is bounded. The absolute stability concept was also employed to analyze the stability and transparency of the teleoperation system. Consequently, it was proved that linear teleoperation is stable and system transparency has been improved during contact condition. However, the proposed controller is only practical for linear teleoperation systems.

In [24] a control scheme for nonlinear bilateral micro-macro teleoperation was proposed. The control scheme includes a PD controller along with operator and environmental forces. Subsequently, a force estimation algorithm for time-varying operator and environmental forces to eliminate force measurement was developed. It was demonstrated that position error has been reduced more in comparison with the lack of estimated external forces in the control scheme during collision between the slave robot and with environment. However, there is always a position error between master and slave robot during contact state.

In [25], [26], force estimation algorithm was proposed based on the improved extended active observer (IEAOB) and extended active observer (EAOB) for teleoperation systems, respectively. In these researches, it was demonstrated that system stability was satisfied in the presence of external uncertainties, but system transparency was not taken into account.

In the present paper a novel control scheme is proposed for a nonlinear bilateral teleoperation system with the presence of constant time delays. The difference between this proposed control scheme and common controllers used for nonlinear teleoperation systems is the existence of force signals in the control scheme. To remove force measurement, a new modified force estimation algorithm is proposed. Stability and transparency of the closed-loop teleoperation system with the existence of estimated forces is proven by the Lyapunov stability criteria. Experimental results validate the performance of nonlinear bilateral teleoperation. Moreover, position error converges to zero in free motion and when the slave is in contact with the environment and force reflection is also appropriately satisfied. The results confirm that the estimated external forces suitably converge to real external forces. In addition, this proposed control scheme is experimentally compared with the common controller. It is demonstrated that the transparency of the nonlinear teleoperation system has improved.

Section snippets

Model definition

The dynamic model of a teleoperation system is considered as: $M_{m} (q_{m}) {\ddot{q}}_{m} + C_{m} (q_{m}, {\dot{q}}_{m}) {\dot{q}}_{m} + g_{m} (q_{m}) = T_{m} - F_{h}$ $M_{s} (q_{s}) {\ddot{q}}_{s} + C_{s} (q_{s}, {\dot{q}}_{s}) {\dot{q}}_{s} + g_{s} (q_{s}) = F_{e} - T_{s}$ where ${{\ddot{q}}_{m}, {\dot{q}}_{m}, q}_{m}, {\ddot{q}}_{s}, {\dot{q}}_{s}, q_{s} \in R^{n}$ are acceleration, velocity and joint position of the master and slave robots respectively; $F_{h}, F_{e} \in R^{n}$ are operator and environment forces, respectively; $T_{m}, T_{s} \in R^{n}$ represent the control inputs; $M_{m} (q_{m}), M_{s} (q_{s}) \in R^{n \times n}$ are the symmetric and positive-definite inertia matrices; and $C_{m} (q_{m}, {\dot{q}}_{m}), C_{s} (q_{s}, {\dot{q}}_{s}) \in R^{n \times n}$ represent the coriolis matrices of the

Nonlinear bilateral teleoperation control design

In this section a control scheme is proposed for nonlinear bilateral teleoperation systems. The control scheme consists of estimated external forces in addition to the PD + d controller.

The control scheme is given by: $T_{m} = K_{Pm} [q_{s} (t - T) - q_{m}] + K_{D} [{\dot{q}}_{s} (t - T) - {\dot{q}}_{m}] - B_{m} {\dot{q}}_{m} + {\hat{F}}_{e} + g_{m}$ $T_{s} = K_{Ps} [q_{s} - q_{m} (t - T)] + K_{D} [{\dot{q}}_{m} - \dot{q}_{s} (t - T)] + B_{s} {\dot{q}}_{s} + {\hat{F}}_{h} - g_{s}$ where $K_{Pm}$ , $K_{Ps}$ , $K_{D}$ , $B_{m}$ and $B_{s}$ are positive definite matrices in $R^{n \times n}$ ; ${\hat{F}}_{h}$ and ${\hat{F}}_{e}$ are the estimated human and environmental force $s$ s respectively.

Free motion analysis

Theorem 2

With regard to the proposed control scheme, position tracking would occur in free motion and the position error would converge to zero. In such case, it is assumed that the human and environmental forces equal zero. ( $F_{h} = F_{e} = 0$ ).

Proof

By satisfying $B_{m} B_{s} > T^{2} K_{Pm} K_{P s}$ with regard to Eq. (33) and with the non-negative of V, it is proven that ${\dot{q}}_{m}$ , ${\dot{q}}_{s}$ , ${∆ F}_{1}$ , ${∆ F}_{2}$ , ${∆ F}_{h}$ and ${∆ F}_{e} \in L_{2}$ . Thus, it could be concluded that ${\dot{q}}_{m}$ , ${\dot{q}}_{s}$ , ${∆ F}_{1}$ , ${∆ F}_{2}$ , ${∆ F}_{h}$ , ${∆ F}_{e}$ and $q_{m} - q_{s} \in L_{\infty}$ with regards to Eq. (27), property 2 and being bounded by

Experimental results

The master and slave employed is a 3DOF Haptic International Phantom Omni. A 6DOF force sensor was installed on the master and slave robot to validate the estimated human force algorithm. This force sensor is capable of measuring force in the x, y, z directions and torque in the x, y, z directions as well. The master, slave and force sensors are shown in Fig. 2.

Prior to the experiments, the master and slave robots’ dynamics were identified. The dynamic models of the master and slave robots are

Conclusion

A PD+d controller with a force estimation algorithm for a nonlinear bilateral teleoperation system was designed. First, the PD+d controller was analyzed, and then a solution for improving the transparency of the teleoperation system when the slave robot collides with the environment was proposed. A force estimation algorithm was developed for time-variant human and environmental forces. Thus, by using an appropriate candidate Lyapunov function, it was proven that the closed-loop bilateral

Acknowledgment

This research was funded by the University of Malaya Research Grant (UMRG) Program no. RP001C-13AET.

References (33)

K. Yoshida et al.
Predictive PD control for teleoperation with communication time delay
Int. Fed. Autom. Control
(2008)
K. Huang et al.
Hybrid PD-based control framework for passive bilateral teleoperation over the internet
Int. Fed. Autom. Control
(2011)
F. Coutinho et al.
Stiffness estimation for robotic tasks with force observers
Control Eng. Pract.
(2014)
A. Gutiérrez-Giles et al.
GPI based velocity/force observer design for robot manipulators
ISA Trans.
(2014)
L. Chan et al.
Extended active observer for force estimation and disturbance rejection of robotic manipulators
Robot. Auton. Syst.
(2013)
A. Mohammadi et al.
Nonlinear disturbance observer design for robotic manipulators
Control Eng. Pract.
(2013)
Y. Nakajima et al.
Heartbeat synchronization with haptic feedback for telesurgical robot
IEEE Trans. Ind. Electron.
(2014)
F. Kanehiro et al.
Efficient reaching motion planning method for low-level autonomy of teleoperated humanoid robots
Adv. Robot.
(2014)
S.-J. Lee et al.
Design and control of tele-matched surgery robot
Mechatronics
(2014)
M. Wilde et al.
Effects of multivantage point systems on the teleoperation of spacecraft docking
IEEE Trans. Hum.-Mach. Syst.
(2014)

S. Cobos-Guzman et al.

Design of an underwater robot manipulator for a telerobotic system

Robotica

(2013)

D. Lee et al.

Passive bilateral teleoperation with constant time delay

IEEE Trans. Robot.

(2006)

E. Nuno et al.

A globally stable PD controller For bilateral teleoperators

IEEE Trans. Robot.

(2008)

T. Ishii et al.

Bilateral control with local force feedback for delay-free teleoperation

Int. Workshop Adv. Motion Control

(2012)

J.H. Park et al.

Sliding mode control of bilateral teleoperation systems with force reflection on the internet

IEEE Int. Conf. Intell. Robots Syst.

(2000)

J. Ueda et al.

Force-reflecting bilateral teleoperation with time delay by signal filtering

IEEE Trans. Robot. Autom.

(2004)

Cited by (22)

An adaptive sparse general regression neural network-based force observer for teleoperation system
2023, Engineering Applications of Artificial Intelligence
Citation Excerpt :
For the above reasons, researchers have extensively investigated various sensorless force estimation methods, and force estimation methods based on inverse dynamics (Azimifar et al., 2017; Zhou and Haraguchi, 2022) and state space model (Cao et al., 2021; Katsura et al., 2007; Liu et al., 2021) have been proposed. Zhou and Haraguchi (2022) achieved the gripping force estimation of robotic tweezers by inverse dynamics, Azimifar et al. (2017) considered the effect of teleoperation communication channel time delay on force estimation based on the inverse dynamic model. Katsura et al. (2007) designed an effective disturbance observer to observe the reaction torque to achieve force control of an industrial robot, Cao et al. (2021) proposed an adaptive Kalman filter with variable time periods to perform force/torque estimation, Liu et al. (2021) developed a Kalman filter-based method combining momentum and interference observer to estimate the contact force and used neural networks to compensate for model uncertainty noise.
Restricted by factors such as small robot size and harsh operating environment, the inability to obtain the interaction force between the robot arm and the environment through force sensors has become problematic in promoting the application of teleoperation systems in minimally invasive surgery, nuclear waste cleanup, and other fields. To accurately obtain the interaction force without the force sensors, a force observer based on an adaptive sparse general regression neural network (ASGRNN) is proposed in this paper. The proposed force observer uses a machine learning-based approach to obtain estimated force, thus eliminating the need for the dynamic parameters of the robot arm. Also, an innovative feature selection method incorporating the wrapper method and sparse regularization is proposed to select the input features of the force observer. Secondly, two new criteria are defined to eliminate the useless support vectors in the model. In addition, an improved antlion optimization algorithm (IALO) is proposed to optimize the bandwidth parameters of the model. To verify the performance of the proposed force observer, a 6-degree-of-freedom teleoperation robot experimental platform is built and compared with three existing force estimation models. The results show that the proposed force observer outperforms the existing model in terms of estimation accuracy, and the mean square error (MSE) is at least 35.79% lower than the existing model. In conclusion, this paper provides a feasible and effective force observer for a teleoperation system where force sensors are not applicable and dynamic parameters are non-knowable.
AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop
2022, Robotics and Computer-Integrated Manufacturing
Citation Excerpt :
One major challenge is that the existing industrial robot program still requires an experienced operator to edit or programming [8]. It is still a great obstacle for users who are experts in the manufacturing field but lacking experience in robot operation [9]. As one basis for the collaborative manufacturing system, teleoperation of industrial robots has been emphasized and investigated much recently [10,11].
The teleoperation and coordination of multiple industrial robots play an important role in today’s industrial internet-based collaborative manufacturing systems. The user-friendly teleoperation approach allows operators from different manufacturing domains to reduce redundant learning costs and intuitively control the robot in advance. Nevertheless, only a few preliminary works have been introduced very recently, let alone its effective implementation in the manufacturing scenarios. To address the gap, this research proposes a novel multi-robot collaborative manufacturing system with human-in-the-loop control by leveraging the cutting-edge augmented reality (AR) and digital twin (DT) techniques. In the proposed system, the DTs of industrial robots are firstly mapped to physical robots and visualize them in the AR glasses. Meanwhile, a multi-robot communication mechanism is designed and implemented, to synchronize the state of robots in the twin. Moreover, a reinforcement learning algorithm is integrated into the robot motion planning to replace the conventional kinematics-based robot movement with corresponding target positions. Finally, three interactive AR-assisted DT modes, including real-time motion control, planned motion control, and robot monitoring mode are generated, which can be readily switched by human operators. Two experimental studies are conducted on (1) a single robot with a commonly used peg-in-hole experiment, and (2) the motion planning of multi-robot collaborative tasks, respectively. From the experimental results, it can be found that the proposed system can well handle the multi-robot teleoperation tasks with high efficiency and owns great potentials to be adopted in other complicated manufacturing scenarios in the near future.
Robust observer-based adaptive synchronization control of uncertain nonlinear bilateral teleoperation systems under time-varying delay
2021, Measurement: Journal of the International Measurement Confederation
Citation Excerpt :
In addition, the stability of the controller together with the nonlinear disturbance observer was not proven in a unified framework. Recently, to consider the aforementioned challenges, a control technique based on the non-linear disturbance observer was developed for the teleoperation system subject to constant time delay [21]. To enhance the results of previous studies, a non-linear disturbance observer-based adaptive controller was designed for the system [22].
Due to the practical restrictions such as uncertain dynamics, external disturbances and time-varying delays and lack of enough information about the remote environment, the performance of teleoperation systems cannot be guaranteed in most cases. However, some practical tasks such as tele-surgery conducted by the nonlinear teleoperation system require high performance. Recent research on teleoperation systems motivates us to propose a new robust adaptive control law for nonlinear bilateral teleoperation systems with time-varying delays, external disturbances and uncertain dynamics in a unified framework. To this end, a novel adaptive torque observer is developed to relax the system from force sensors. The main advantage of this paper is to design properly a mechanism, which is free from joint accelerations due to the acceleration measurement difficulty in robotic systems. Moreover, the key point of the proposed algorithm is to enhance the robust behavior of the system in the presence of various uncertainties by adding an auxiliary term in the control loop. Furthermore, the stability and the convergence of synchronization error to zero are also proven with the aid of both Lyapunov-Krasovskii function and linear matrix inequality. Simulation results emphasize on the effectiveness of the proposed Observed-based control approach in comparison with the related research.
Observer-based adaptive force–position control for nonlinear bilateral teleoperation with time delay
2021, Control Engineering Practice
In this paper, a novel observer-based force control scheme is proposed to guarantee the position and force tracking in nonlinear teleoperation systems, subject to constant communication time delay. Stability of the teleoperation system is analytically proved using the Lyapunov stability theorem. Enough condition for asymptotic convergence of the force and position are derived. The transparency is also improved by force tracking capability of the system. The teleoperation controller is then applied to a pair of planar 2-DOF robots, connected via a communication channel with constant time delay. The experimental and simulation results demonstrate the effectiveness of the proposed control algorithm, in force estimation and dealing with time delay.
Flexible telemanipulation based handy robot teaching on tape masking with complex geometry
2020, Robotics and Computer-Integrated Manufacturing
Citation Excerpt :
Based on this, the feature of user-friendly and ease-of-use to enable complex robot manipulations are thereby proposed by [1–3]. Meanwhile, the design of perception and feedback system to ensure users’ operating awareness and safety are also very important [4–9]. Traditional solutions, such as offline programming or teach pendant based teaching methods, to cope with robotic manipulation tasks tend to have high demands for operators.
Traditional robot teaching methods are cumbersome, tedious and difficult to scale for high-mix low-volume applications. The tape masking, a common process for surface protection before plasma spraying, spray painting and shot peening, is one of those domains where robotic automation lacks flexibility and reliability due to the complexity in task. Fortunately, it is still within the grasps of human-robot collaborative systems. This work presents a telemanipulation-based robot teaching framework that is able to let the robot manipulator cope with the taping tasks with complex workpiece geometries. The proposed framework allows quick calibration, variable motion mapping, and indexing so that the operators can easily set up and guide the robotic taping system to cover the tapes onto the layers and grooves of different workpieces. This framework enables the operators to change the motion mapping scale for both large-scale guidance and fine motion dexterous manipulation. Meanwhile, an indexing function makes it possible for the operators to re-map their poses from the edges of their comfortable regions. A portable VR system is applied in the telemanipulation system. With its six DoF motion precisely measured in real-time, the proposed motion remapping algorithms enable the operators to directly guide the robot in their selected scales. Experimental results show that the proposed framework facilitates robot programming on the manipulation of the complex workpieces that have multi-layer surfaces and grooves in between. It also reduces the teaching time comparing to other methods. This system and method improve teaching efficiency and convenience, which has potential value to be deployed in manufacturing.
A novel bilateral impedance controls for underwater tele-operation systems
2020, Applied Soft Computing Journal
Citation Excerpt :
In the absence of force sensors, for the purpose of increasing the transparency, some researchers attempt to estimate the environmental force. In [12], a novel control method is proposed on the basis of the force estimation on both the master and slave robots for the purpose of substituting the force sensors. Concerning the time delay, position tracking experiments are performed to verify the proposed method.
Owing to characteristics of the flow and the variability, it is extremely difficult to achieve the stability and the transparency of the underwater tele-operation system. In practice, the accurate force may not easily be acquired due to model uncertainties, the time delay and external disturbances. In order to enhance the stability and the transparency of the underwater tele-operation, an adaptive neural fuzzy inference system disturbance observer-based impedance control is proposed to both the master side and slave side. The learning algorithm of the adaptive neural fuzzy inference system network and the disturbance observer may simultaneously suppress model uncertainties of the nonlinear system and disturbances of external underwater environment. Concerning the time delay, the stability is analyzed by Lyapunov theorem. Numerical simulations are performed and results demonstrate the effective performance of the proposed method.

View all citing articles on Scopus

View full text

Improving teleoperation system performance in the presence of estimated external force

Highlights

Abstract

Introduction

Section snippets

Model definition

Nonlinear bilateral teleoperation control design

Free motion analysis

Experimental results

Conclusion

Acknowledgment

Int. Fed. Autom. Control

Int. Fed. Autom. Control

Control Eng. Pract.

ISA Trans.

Robot. Auton. Syst.

Control Eng. Pract.

Heartbeat synchronization with haptic feedback for telesurgical robot

IEEE Trans. Ind. Electron.

Efficient reaching motion planning method for low-level autonomy of teleoperated humanoid robots

Adv. Robot.

Design and control of tele-matched surgery robot

Mechatronics

Effects of multivantage point systems on the teleoperation of spacecraft docking

IEEE Trans. Hum.-Mach. Syst.

Design of an underwater robot manipulator for a telerobotic system

Robotica

Passive bilateral teleoperation with constant time delay

IEEE Trans. Robot.

A globally stable PD controller For bilateral teleoperators

IEEE Trans. Robot.

Bilateral control with local force feedback for delay-free teleoperation

Int. Workshop Adv. Motion Control

Sliding mode control of bilateral teleoperation systems with force reflection on the internet

IEEE Int. Conf. Intell. Robots Syst.

Force-reflecting bilateral teleoperation with time delay by signal filtering

IEEE Trans. Robot. Autom.