Skip to main content
SpringerLink
Log in

Robust Kalman filtering with long short-term memory for image-based visual servo control

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Visual servo control systems based on Kalman filter (KF) is susceptible to noise interference, the initialization of the Jacobi matrix is difficult, and the observation value of the Jacobian matrix is not accurate. In order to address these problems, we proposed a robust KF algorithm with long short-term memory (LSTM) for an image-based visual servo control system and applied the system to an uncalibrated image-based visual servo (IBVS) control system to estimate the filtering gain error, state estimation error, and the observation error, which were then used for online training in LSTM. The visual servo control system controls the motion of the manipulator, and simultaneously updates the LSTM network. Therefore, the Jacobian matrix obtained using LSTM was employed to estimate the state volume of the robust KF, which constitutes a circulatory system, and the complementary effect was realized. The method was applied to a six-degrees-of-freedom manipulator of the eye-in-hand model to conduct experiments. The simulation results indicate that the proposed visual servo control system has strong anti-noise interference capability. Furthermore, it facilitates Jacobian matrix initialization and has high observation accuracy for the Jacobian matrix.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Agand P, Shoorehdeli MA, Khaki-Sedigh A (2017) Adaptive recurrent neural network with Lyapunov stability learning rules for robot dynamic terms identification. Eng Appl Artif Intell 65:1–11

    Article  Google Scholar 

  2. Chaumette F, Hutchinson S (2006) Visual servo control - Part I: Basic approaches. IEEE Robotics & Automation Magazine 13(4):p82–p90

    Article  Google Scholar 

  3. Cheng ZY, Chang XJ, Zhu L et al (2019) MMALFM: Explainable recommendation by leveraging reviews and images. ACM Trans Inf Syst 37(2):16. https://doi.org/10.1145/3291060

    Article  Google Scholar 

  4. Cheng ZY, Ding Y, He XN et al (2018) A3NCF: An adaptive aspect attention model for rating prediction. IJCAI International Joint Conference on Artificial Intelligence:3748–3754

  5. Chherawala Y, Roy PP, Cheriet M (2017) Combination of context-dependent bidirectional long short-term memory classifiers for robust offline handwriting recognition. Pattern Recogn Lett 90:58–64

    Article  Google Scholar 

  6. Cortez B, Carrera B, Kim Y-J et al (2018) An architecture for emergency event prediction using LSTM recurrent neural networks. Expert Syst Appl 97:315–324

    Article  Google Scholar 

  7. Elman JL (1990) Finding Structure in Time. Cogn Sci 14:179–211

    Article  Google Scholar 

  8. Fausett LV, Elwasif W (1994) Predicting performance from test scores using backpropagation and counter propagation. IEEE International Conference on Neural Networks - Conference Proceedings 5:3398–3402

    Google Scholar 

  9. Gao J, An X, Proctor A et al (2017) Sliding mode adaptive neural network control for hybrid visual servoing of underwater vehicles. Ocean Eng 142:666–675

    Article  Google Scholar 

  10. Gu J, Wang H, Pan Y et al (2015) Neural network based visual servo control for CNC load/unload manipulator. OPTIK 126(23):4489–4492

    Article  Google Scholar 

  11. Guo YY, Cheng ZY, Nie LQ et al (2019) Attentive long short-term preference modeling for personalized product search. ACM Transactions on Information Systems (TOIS) 37(2):19. https://doi.org/10.1145/3295822

    Article  Google Scholar 

  12. He F, Guo Y, Gao C (2018) Human segmentation of infrared image for mobile robot search. Multimed Tools Appl 77(9):10701–10714

    Article  Google Scholar 

  13. He W, Huang B, Dong Y et al (2018) Adaptive neural network control for robotic manipulators with unknown deadzone. IEEE Transactions on Cybernetics 48(9):2670–2682

    Article  Google Scholar 

  14. He W, Huang H, Ge SS (2017) Adaptive neural network control of a robotic manipulator with time-varying output constraints. IEEE Transactions on Cybernetics 47(10):3136–3147

    Article  Google Scholar 

  15. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735

    Article  Google Scholar 

  16. How D, N T LCK, Sahari KSM (2016) Behavior recognition for humanoid robots using long short-term memory. Int J Adv Robot Syst 13(6):1–14

    Article  Google Scholar 

  17. Huang D, Jiang Z, Zou L et al (2017) Drug-drug interaction extraction from biomedical literature using support vector machine and long short term memory networks. Inf Sci 415-416:100–109

    Article  Google Scholar 

  18. Jin L, Li S, Luo X et al (2018) Neural dynamics for cooperative control of redundant robot manipulators. IEEE Transactions on Industrial Informatics 14(9):p3812–p3821

    Article  Google Scholar 

  19. Li S, Ghasemi A, Xie W et al (2018) An enhanced IBVS controller of a 6DOF manipulator using hybrid PD-SMC method. Int J Control Autom Syst 16(2):p844–p855

    Article  Google Scholar 

  20. Li Y, Li Y, Kim H et al (2018) Active contour model-based segmentation algorithm for medical robots recognition. Multimed Tools Appl 77(9):10485–10500

    Article  Google Scholar 

  21. Li S, Zhou M-C, Luo X (2018) Modified primal-dual neural networks for motion control of redundant manipulators with dynamic rejection of harmonic noises. IEEE Transactions on Neural Networks and Learning Systems 29(10):4791–4801

    Article  MathSciNet  Google Scholar 

  22. Liu H, Mi XW, Li YF (2018) Wind speed forecasting method based on deep learning strategy using empirical wavelet transform, long short term memory neural network and Elman neural network. Energy Convers Manag 156:498–514

    Article  Google Scholar 

  23. Miljković Z, Mitić M, Lazarević M et al (2013) Neural network reinforcement learning for visual control of robot manipulators. Expert Syst Appl 40(5):1721–1736

    Article  Google Scholar 

  24. Mitic M, Miljkovic Z (2014) Neural network learning from demonstration and epipolar geometry for visual control of a nonholonomic mobile robot. Soft Comput 18(5):1011–1025

    Article  Google Scholar 

  25. Nadi F, Derhami V, Rezaeian M (2014) Visual servoing control of robot manipulator with Jacobian matrix estimation. In: Proceedings of the 2014 International Conference on Robotics and Mechatronics. IEEE, Tehran, p 405–409

  26. Peng Y, Lu B (2017) Discriminative extreme learning machine with supervised sparsity preserving for image classification. Neurocomputing 261:242–252

    Article  Google Scholar 

  27. Peng Y, Lu B (2017) Robust structured sparse representation via half-quadratic optimization for face recognition. Multimed Tools Appl 76(6):p8859–p8880

    Article  Google Scholar 

  28. Qian J, Su JB (2003) On-line estimation of image Jacobin matrix based on Kalman filter. Control and Decision 18(1):77–80

    Google Scholar 

  29. Qin F, Gao N, Peng Y et al (2018) Fine-grained leukocyte classification with deep residual learning for microscopic images. Comput Methods Prog Biomed 162:243–252

    Article  Google Scholar 

  30. Qu J, Zhang F, Fu Y et al (2017) Adaptive neural network visual servoing of dual-arm robot for cyclic motion. Ind Robot 44(2):210–221

    Article  Google Scholar 

  31. Sadeghzadeh M, Calvert D, Hussein A, Self A (2015) Learning visual servoing of robot manipulator using explanation-based fuzzy neural networks and Q-Learning. Journal of Intelligent and Robotic Systems: Theory and Applications 78(1):83–104

    Article  Google Scholar 

  32. Salgado I, Chairez I (2018) Adaptive unknown input estimation by sliding modes and differential neural network observer. IEEE Transactions on Neural Networks and Learning Systems 29(8):3499–3509

    Article  MathSciNet  Google Scholar 

  33. Wang F, Chao Z-Q, Huang L-B et al (2017) Trajectory tracking control of robot manipulator based on RBF neural network and fuzzy sliding mode. Clust Comput. https://doi.org/10.1007/s10586-017-1538-4

  34. Wang M, Qu W, Chen W (2018) Hybrid sensing and encoding using pad phone for home robot control. Multimed Tools Appl 77(9):10773–10786

    Article  Google Scholar 

  35. Zhang Y, Muhammad K, Tang C (2018) Twelve-layer deep convolutional neural network with stochastic pooling for tea category classification on GPU platform. Multimed Tools Appl 77(17):22821–22839

    Article  Google Scholar 

  36. Zhang Q, Wang H, Dong J et al (2017) Prediction of sea surface temperature using long short-term memory. IEEE Geosci Remote Sens Lett 14(10):1745–1749

    Article  Google Scholar 

  37. Zhang Y, Zhang Y, Hou X et al (2018) Seven-layer deep neural network based on sparse autoencoder for voxelwise detection of cerebral microbleed. Multimed Tools Appl 77(9):10521–10538

    Article  Google Scholar 

  38. Zhao W, Li H, Zou L et al (2016) Singular value decomposition aided cubature Kalman filter with neural network in uncalibrated binocular stereo visual servoing system. International Journal of Control and Automation 9(12):357–372

    Article  Google Scholar 

  39. Zhao Y, Xie W, Liu S et al (2015) Neural network-based image moments for robotic visual servoing. Journal of Intelligent and Robotic Systems: Theory and Applications 78(2):239–256

    Article  Google Scholar 

  40. Zhong XG, Zhong XY, Peng XF (2013) Robust Kalman filtering cooperated Elman neural network learning for vision-sensing-based robotic manipulation with global stability. Sensors 13:13464–13486

    Article  Google Scholar 

  41. Zhong X, Zhong X, Peng X (2015) Robots visual servo control with features constraint employing Kalman-neural-network filtering scheme. Neurocomputing 151(P1):268–277

    Article  Google Scholar 

  42. Zhou Z, Zhang R, Zhu Z (2019) Uncalibrated dynamic visual servoing via multivariate adaptive regression splines and improved incremental extreme learning machine. ISA Trans. https://doi.org/10.1016/j.isatra.2019.02.029

Download references

Funding

This work is supported by Zhejiang Provincial Natural Science Foundation of China (No. LY18F030018), Science Foundation of Zhejiang Sci-Tech University (No. 18032232-Y), and Natural Science Foundation of China (No. 51376055).

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Zhejiang Sci-Tech University, Hangzhou, 310018, China

    Zhiyu Zhou & Ruoxi Zhang

  2. School of Mechanical Engineering, Hangzhou Dianzi University, Hangzhou, 310018, China

    Zefei Zhu

Authors
  1. Zhiyu Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruoxi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zefei Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiyu Zhou.

Ethics declarations

Conflict of interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Z., Zhang, R. & Zhu, Z. Robust Kalman filtering with long short-term memory for image-based visual servo control. Multimed Tools Appl 78, 26341–26371 (2019). https://doi.org/10.1007/s11042-019-07773-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-07773-0

Keywords

Search

Navigation