Skip to main content
SpringerLink
Log in

Renal surface reconstruction and segmentation for image-guided surgical navigation of laparoscopic partial nephrectomy

  • Original Article
  • Published:
Biomedical Engineering Letters Aims and scope Submit manuscript

Abstract

An unpredictable dynamic surgical environment makes it necessary to measure morphological information of target tissue real-time for laparoscopic image-guided navigation. The stereo vision method for intraoperative tissue 3D reconstruction has the most potential for clinical development benefiting from its high reconstruction accuracy and laparoscopy compatibility. However, existing stereo vision methods have difficulty in achieving high reconstruction accuracy in real time. Also, intraoperative tissue reconstruction results often contain complex background and instrument information that prevents clinical development for image-guided systems. Taking laparoscopic partial nephrectomy (LPN) as the research object, this paper realizes a real-time dense reconstruction and extraction of the kidney tissue surface. The central symmetrical Census based semi-global block stereo matching algorithm is proposed to generate a dense disparity map. A GPU-based pixel-by-pixel connectivity segmentation mechanism is designed to segment the renal tissue area. An in-vitro porcine heart, in-vivo porcine kidney and offline clinical LPN data were performed to evaluate the accuracy and effectiveness of our approach. The algorithm achieved a reconstruction accuracy of ± 2 mm with a real-time update rate of 21 fps for an HD image size of 960 × 540, and 91.0% target tissue segmentation accuracy even with surgical instrument occlusions. Experimental results have demonstrated that the proposed method could accurately reconstruct and extract renal surface in real-time in LPN. The measurement results can be used directly for image-guided systems. Our method provides a new way to measure geometric information of target tissue intraoperatively in laparoscopy surgery.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Himal HS. Minimally invasive (laparoscopic) surgery. Surg Endosc Interv Tech. 2002;16:1647–52. https://doi.org/10.1007/s00464-001-8275-7.

    Article  Google Scholar 

  2. Peters TM, Linte CA, Yaniv Z, Williams J. Mixed and augmented reality in medicine. Boca Raton, FL, USA: CRC Press; 2018. p. 1–13.

    Book  Google Scholar 

  3. Joeres F, Mielke T, Hansen C. Laparoscopic augmented reality registration for oncological resection site repair. Int J Comput Assist Radiol Surg. 2021;16:1577–86. https://doi.org/10.1007/s11548-021-02336-x.

    Article  Google Scholar 

  4. Tang R, Ma L-F, Rong Z-X, Li M-D, Zeng J-P, Wang X-D, et al. Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: a review of current methods. Hepatobiliary Pancreat Dis Int. 2018;17:101–12. https://doi.org/10.1016/j.hbpd.2018.02.002.

    Article  Google Scholar 

  5. Schwabenland I, Sunderbrink D, Nollert G, Dickmann C, Weingarten M, Meyer A, et al. Flat-Panel CT and the Future of OR Imaging and Navigation. In: Fong Y, Giulianotti PC, Lewis J, Groot Koerkamp B, Reiner T, (eds.) Imaging Vis. Mod. Oper. Room Compr. Guide Physicians, New York, NY: Springer; 2015, 89–106. https://doi.org/10.1007/978-1-4939-2326-7_7.

  6. Song Y, Totz J, Thompson S, Johnsen S, Barratt D, Schneider C, et al. Locally rigid, vessel-based registration for laparoscopic liver surgery. Int J Comput Assist Radiol Surg. 2015;10:1951–61. https://doi.org/10.1007/s11548-015-1236-8.

    Article  Google Scholar 

  7. Herrera SEM, Malti A, Morel O, Bartoli A. Shape-from-Polarization in laparoscopy. 2013 IEEE 10th international symposium on biomedical imaging, IEEE, 2013, 1412–5.

  8. Fusaglia M, Hess H, Schwalbe M, Peterhans M, Tinguely P, Weber S, et al. A clinically applicable laser-based image-guided system for laparoscopic liver procedures. Int J Comput Assist Radiol Surg. 2016;11:1499–513.

    Article  Google Scholar 

  9. Maier-Hein L, Groch A, Bartoli A, Bodenstedt S, Boissonnat G, Chang P-L, et al. Comparative validation of single-shot optical techniques for laparoscopic 3-D surface reconstruction. IEEE Trans Med Imaging. 2014;33:1913–30.

    Article  Google Scholar 

  10. Stoyanov D, Scarzanella MV, Pratt P, Yang G-Z. Real-time stereo reconstruction in robotically assisted minimally invasive surgery. International conference medical image computing and computer-assisted intervention, Springer, 2010, 275–82.

  11. Grasa OG, Bernal E, Casado S, Gil I, Montiel JMM. Visual slam for handheld monocular endoscope. IEEE Trans Med Imaging. 2014;33:135–46. https://doi.org/10.1109/TMI.2013.2282997.

    Article  Google Scholar 

  12. Hu M, Penney G, Figl M, Edwards P, Bello F, Casula R, et al. Reconstruction of a 3D surface from video that is robust to missing data and outliers: application to minimally invasive surgery using stereo and mono endoscopes. Med Image Anal. 2012;16:597–611. https://doi.org/10.1016/j.media.2010.11.002.

    Article  Google Scholar 

  13. Takeshita T, Nakajima Y, Kim MK, Onogi S, Mitsuishi M, Matsumoto Y. 3D shape reconstruction endoscope using shape from focus. 2009 4th international conference on computer vision theory and applications, 2009, 411–6.

  14. Groch A, Seitel A, Hempel S, Speidel S, Engelbrecht R, Penne J, et al. 3D surface reconstruction for laparoscopic computer-assisted interventions: comparison of state-of-the-art methods. Medical imaging 2011 visualization, image-guided procedures, and modeling, vol. 7964, SPIE, 2011, 351–9.

  15. Stoyanov D, Mylonas GP, Deligianni F, Darzi A, Yang GZ. Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures. Int. Conf. Med. Image Comput. Comput.-Assist. Interv., Springer, 2005, 139–46.

  16. Stoyanov D. Stereoscopic scene flow for robotic assisted minimally invasive surgery. International conference medical image computing and computer-assisted intervention, Springer, 2012, 479–86.

  17. Bernhardt S, Abi-Nahed J, Abugharbieh R. Robust dense endoscopic stereo reconstruction for minimally invasive surgery. Int. MICCAI Workshop Medical Computer Vision, Springer, 2012, 254–62.

  18. Totz J, Thompson S, Stoyanov D, Gurusamy K, Davidson BR, Hawkes DJ, et al. Fast semi-dense surface reconstruction from stereoscopic video in laparoscopic surgery. International conference information processing in computer-assisted interventions, Springer, 2014, 206–15.

  19. Marques B, Plantefève R, Roy F, Haouchine N, Jeanvoine E, Peterlik I, et al. Framework for augmented reality in Minimally Invasive laparoscopic surgery. 2015 17th International conference on E-health networking, application & services (HealthCom), 2015, 22–7. https://doi.org/10.1109/HealthCom.2015.7454467.

  20. Luo H, Hu Q, Jia F. Details preserved unsupervised depth estimation by fusing traditional stereo knowledge from laparoscopic images. Healthc Technol Lett. 2019;6:154–8. https://doi.org/10.1049/htl.2019.0063.

    Article  Google Scholar 

  21. Wei R, Li B, Mo H, Lu B, Long Y, Yang B, et al. Stereo dense scene reconstruction and accurate laparoscope localization for learning-based navigation in robot-assisted surgery. arXiv preprint arXiv:2110.03912, 2021.

  22. Li Z, Liu X, Drenkow N, Ding A, Creighton FX, Taylor RH, et al. Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers. Proc. IEEECVF international conference on computer vision, 2021, 6197–206.

  23. Allan M, Mcleod J, Wang C, Rosenthal JC, Hu Z, Gard N, et al. Stereo correspondence and reconstruction of endoscopic data challenge. arXiv preprint arXiv:2101.01133, 2021.

  24. Luo H, Yin D, Zhang S, Xiao D, He B, Meng F, et al. Augmented reality navigation for liver resection with a stereoscopic laparoscope. Comput Methods Programs Biomed. 2020;187:105099. https://doi.org/10.1016/j.cmpb.2019.105099.

    Article  Google Scholar 

  25. Luo H, Wang C, Duan X, Liu H, Wang P, Hu Q, et al. Unsupervised learning of depth estimation from imperfect rectified stereo laparoscopic images. Comput Biol Med. 2022;140:105109. https://doi.org/10.1016/j.compbiomed.2021.105109.

    Article  Google Scholar 

  26. Su L-M, Vagvolgyi BP, Agarwal R, Reiley CE, Taylor RH, Hager GD. Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology. 2009;73:896–900. https://doi.org/10.1016/j.urology.2008.11.040.

    Article  Google Scholar 

  27. Zhang X, Wang J, Wang T, Ji X, Shen Y, Sun Z, et al. A markerless automatic deformable registration framework for augmented reality navigation of laparoscopy partial nephrectomy. Int J Comput Assist Radiol Surg. 2019;14:1285–94. https://doi.org/10.1007/s11548-019-01974-6.

    Article  Google Scholar 

  28. Wang C, Cheikh FA, Kaaniche M, Elle OJ. Liver surface reconstruction for image guided surgery. Medical Imaging 2018 image-guided procedures, robotic interventions, and modeling, vol. 10576, 576–83.

  29. Devernay F, Mourgues F, Coste-Maniere E. Towards endoscopic augmented reality for robotically assisted minimally invasive cardiac surgery. Proceedings international workshop on medical imaging and augmented reality, 2001, 16–20. https://doi.org/10.1109/MIAR.2001.930258.

  30. Vagvolgyi B, Su L-M, Taylor R, Hager GD. Video to CT registration for image overlay on solid organs. Proc Augment Real Med Imaging Augment Real Comput-Aided Surg AMIARCS, 2008, 78–86.

  31. Chang P-L, Stoyanov D, Davison AJ, Edwards P. Real-time dense stereo reconstruction using convex optimisation with a cost-volume for image-guided robotic surgery. International conference medical image computing and computer-assisted intervention, Springer, 2013, 42–9.

  32. Zhou H, Jagadeesan J. Real-time dense reconstruction of tissue surface from stereo optical video. IEEE Trans Med Imaging. 2020;39:400–12. https://doi.org/10.1109/TMI.2019.2927436.

    Article  Google Scholar 

  33. Hirschmuller H, Scharstein D. Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans Pattern Anal Mach Intell. 2009;31:1582–99. https://doi.org/10.1109/TPAMI.2008.221.

    Article  Google Scholar 

  34. Penza V, Ortiz J, Mattos LS, Forgione A, De Momi E. Dense soft tissue 3D reconstruction refined with super-pixel segmentation for robotic abdominal surgery. Int J Comput Assist Radiol Surg. 2016;11:197–206. https://doi.org/10.1007/s11548-015-1276-0.

    Article  Google Scholar 

  35. Zhang P, Luo H, Zhu W, Yang J, Zeng N, Fan Y, et al. Real-time navigation for laparoscopic hepatectomy using image fusion of preoperative 3D surgical plan and intraoperative indocyanine green fluorescence imaging. Surg Endosc. 2020;34:3449–59. https://doi.org/10.1007/s00464-019-07121-1.

    Article  Google Scholar 

  36. Spangenberg R, Langner T, Rojas R. Weighted Semi-Global Matching and Center-Symmetric Census Transform for Robust Driver Assistance. In: Wilson R, Hancock E, Bors A, Smith W (eds.) Comput. Anal. Images Patterns, vol. 8048, Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, 34–41. https://doi.org/10.1007/978-3-642-40246-3_5.

  37. Hirschmuller H. Stereo processing by semiglobal matching and mutual information. IEEE Trans Pattern Anal Mach Intell. 2008;30:328–41. https://doi.org/10.1109/TPAMI.2007.1166.

    Article  Google Scholar 

  38. Hirschmuller H. Accurate and efficient stereo processing by semi-global matching and mutual information. 2005 IEEE Computer society conference on computer vision and pattern recognition CVPR05, vol. 2, 2005, 807–14 vol. 2. https://doi.org/10.1109/CVPR.2005.56.

  39. Hernandez-Juarez D, Chacón A, Espinosa A, Vázquez D, Moure JC, López AM. Embedded real-time stereo estimation via semi-global matching on the GPU. Procedia Comput Sci. 2016;80:143–53. https://doi.org/10.1016/j.procs.2016.05.305.

    Article  Google Scholar 

  40. Canny J. A Computational Approach to Edge Detection. IEEE Trans Pattern Anal Mach Intell 1986;PAMI-8:679–98. https://doi.org/10.1109/TPAMI.1986.4767851.

  41. Št´ava O, Beneš B. Chapter 35—Connected Component Labeling in CUDA. In: Hwu WW (ed). GPU Comput. Gems Emerald Ed., Boston: Morgan Kaufmann, 2011, 569–81. https://doi.org/10.1016/B978-0-12-384988-5.00035-8.

  42. Besl PJ, McKay ND. Method for registration of 3-D shapes. Sens. Fusion IV Control Paradig. Data Struct., vol. 1611, International Society for Optics and Photonics, 1992, 586–606.

  43. Mountney P, Stoyanov D, Yang G-Z. Three-dimensional tissue deformation recovery and tracking. IEEE Signal Process Mag. 2010;27:14–24. https://doi.org/10.1109/MSP.2010.936728.

    Article  Google Scholar 

Download references

Funding

This work was supported by Natural Science Foundation of China (Grant No. 62173014) and Natural Science Foundation of Beijing Municipality (Grant No. L192057).

Author information

Authors and Affiliations

  1. School of Engineering Medicine, Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University, No. 37 Xueyuan Road, Haidian District, Beijing, 100083, China

    Xiaohui Zhang, Yubo Fan & Chunjing Tao

  2. School of Biomedical Engineering, Beihang University, No. 37 Xueyuan Road, Haidian District, Beijing, 100083, China

    Xuquan Ji & Yubo Fan

  3. School of Mechanical Engineering and Automation, Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang Unviersity, No. 37 Xueyuan Road, Haidian District, Beijing, 100083, China

    Junchen Wang

Authors
  1. Xiaohui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuquan Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junchen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yubo Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chunjing Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. XZ: Performed the research, Designed the reconstruction method, Analyzed the results. XJ: Designed the segmentation method, Coding the parallel computing. JW: Reviewed the manuscript, Acquired image data and project funding. YF: Supervised the project, Reviewed the final manuscript, Worked on the manuscript with support. CT: Supervised the project, Reviewed the final manuscript, Worked on the manuscript with support.

Corresponding authors

Correspondence to Xuquan Ji, Junchen Wang or Chunjing Tao.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent to publish

The authors affirm that human research participants provided informed consent for publication of the images in Figs. 2a, 8a, b and Online Resource video.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (MP4 6687 kb)

Supplementary file2 (MP4 44490 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Ji, X., Wang, J. et al. Renal surface reconstruction and segmentation for image-guided surgical navigation of laparoscopic partial nephrectomy. Biomed. Eng. Lett. 13, 165–174 (2023). https://doi.org/10.1007/s13534-023-00263-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13534-023-00263-1

Keywords

Search

Navigation