Skip to main content

Advertisement

SpringerLink
Log in

Automatic needle tracking using Mask R-CNN for MRI-guided percutaneous interventions

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Accurate needle tracking provides essential information for MRI-guided percutaneous interventions. Passive needle tracking using MR images is challenged by variations of the needle-induced signal void feature in different situations. This work aimed to develop an automatic needle tracking algorithm for MRI-guided interventions based on the Mask Region Proposal-Based Convolutional Neural Network (R-CNN).

Methods

Mask R-CNN was adapted and trained to segment the needle feature using 250 intra-procedural images from 85 MRI-guided prostate biopsy cases and 180 real-time images from MRI-guided needle insertion in ex vivo tissue. The segmentation masks were passed into the needle feature localization algorithm to extract the needle feature tip location and axis orientation. The proposed algorithm was tested using 208 intra-procedural images from 40 MRI-guided prostate biopsy cases, and 3 real-time MRI datasets in ex vivo tissue. The algorithm results were compared with human-annotated references.

Results

In prostate datasets, the proposed algorithm achieved needle feature tip localization error with median Euclidean distance (dxy) of 0.71 mm and median difference in axis orientation angle (dθ) of 1.28°, respectively. In 3 real-time MRI datasets, the proposed algorithm achieved consistent dynamic needle feature tracking performance with processing time of 75 ms/image: (a) median dxy = 0.90 mm, median dθ = 1.53°; (b) median dxy = 1.31 mm, median dθ = 1.9°; (c) median dxy = 1.09 mm, median dθ = 0.91°.

Conclusions

The proposed algorithm using Mask R-CNN can accurately track the needle feature tip and axis on MR images from in vivo intra-procedural prostate biopsy cases and ex vivo real-time MRI experiments with a range of different conditions. The algorithm achieved pixel-level tracking accuracy in real time and has potential to assist MRI-guided percutaneous interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Gupta S, Madoff DC (2007) Image-guided percutaneous needle biopsy in cancer diagnosis and staging. Tech Vasc Interv Radiol 10:88–101. https://doi.org/10.1053/j.tvir.2007.09.005

    Article  PubMed  Google Scholar 

  2. McWilliams JP, Lee EW, Yamamoto S, Loh CT, Kee ST (2010) Image-guided tumor ablation: emerging technologies and future directions. Semin Interv Radiol 27:302–313. https://doi.org/10.1055/s-0030-1261789

    Article  Google Scholar 

  3. Campbell-Washburn AE, Faranesh AZ, Lederman RJ, Hansen MS (2015) Magnetic resonance sequences and rapid acquisition for MR-guided interventions. Magn Reson Imaging Clin 23:669–679. https://doi.org/10.1016/j.mric.2015.05.006

    Article  Google Scholar 

  4. Campbell-Washburn AE, Tavallaei MA, Pop M, Grant EK, Chubb H, Rhode K, Wright GA (2017) Real-time MRI guidance of cardiac interventions. J Magn Reson Imaging 46:935–950. https://doi.org/10.1002/jmri.25749

    Article  PubMed  PubMed Central  Google Scholar 

  5. DiMaio SP, Kacher D, Ellis R, Fichtinger G, Hata N, Zientara G, Panych L, Kikinis R, Jolesz F (2006) Needle artifact localization in 3T MR images. Stud Health Technol Inform 119:120–125

    CAS  PubMed  Google Scholar 

  6. DiMaio SP, Samset E, Fischer G, Iordachita I, Fichtinger G, Jolesz F, Tempany CM (2007) Dynamic MRI scan plane control for passive tracking of instruments and devices. In: Medical image computing and computer-assisted intervention (MICCAI), pp 50–58. https://doi.org/10.1007/978-3-540-75759-7_7

  7. Görlitz RA, Tokuda J, Hoge SW, Chu R, Panych LP, Tempany C, Hata N (2010) Development and validation of a real-time reduced field of view imaging driven by automated needle detection for MRI-guided interventions. In: SPIE medical imaging, pp 762515–762519. https://doi.org/10.1117/12.840837

  8. Zijlstra F, Bouwman JG, Braškutė I, Viergever MA, Seevinck PR (2016) Fast Fourier-based simulation of off-resonance artifacts in steady-state gradient echo MRI applied to metal object localization. Magn Reson Med 78:2035–2041. https://doi.org/10.1002/mrm.26556

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Moore CM, Robertson NL, Arsanious N, Middleton T, Villers A, Klotz L, Taneja SS, Emberton M (2013) Image-guided prostate biopsy using magnetic resonance imaging–derived targets: a systematic review. Eur Urol 63:125–140. https://doi.org/10.1016/j.eururo.2012.06.004

    Article  PubMed  Google Scholar 

  10. Tan N, Lin W-C, Khoshnoodi P, Asvadi NH, Yoshida J, Margolis DJA, Lu DSK, Wu H, Sung KH, Lu DY, Huang J, Raman SS (2016) In-bore 3-T MR-guided transrectal targeted prostate biopsy: prostate imaging reporting and data system version 2–based diagnostic performance for detection of prostate cancer. Radiology 283:130–139. https://doi.org/10.1148/radiol.2016152827

    Article  PubMed  PubMed Central  Google Scholar 

  11. Verma S, Choyke PL, Eberhardt SC, Oto A, Tempany CM, Turkbey B, Rosenkrantz AB (2017) The current state of MR imaging–targeted biopsy techniques for detection of prostate cancer. Radiology 285:343–356. https://doi.org/10.1148/radiol.2017161684

    Article  PubMed  PubMed Central  Google Scholar 

  12. Ladd ME, Erhart P, Debatin JF, Romanowski BJ, Boesiger P, McKinnon GC (1996) Biopsy needle susceptibility artifacts. Magn Reson Med 36:646–651. https://doi.org/10.1002/mrm.1910360423

    Article  CAS  PubMed  Google Scholar 

  13. Cole GA, Harrington K, Su H, Camilo A, Pilitsis JG, Fischer GS (2014) Closed-loop actuated surgical system utilizing real-time in situ MRI guidance. In: International symposium on experimental robotics (ISER), pp 785–798. https://doi.org/10.1007/978-3-642-28572-1_54

  14. Mikaiel S, Simonelli J, Li X, Lee Y, Lee YS, Sung K, Lu D, Tsao TC, Wu HH (2020) MRI-guided targeted needle placement during motion using hydrostatic actuators. Int J Med Robot Comput Assist Surg 16(2):e2041. https://doi.org/10.1002/rcs.2041

    Article  Google Scholar 

  15. Mehrtash A, Ghafoorian M, Pernelle G, Ziaei A, Heslinga FG, Tuncali K, Fedorov A, Kikinis R, Tempany CM, Wells WM, Abolmaesumi P, Kapur T (2019) Automatic needle segmentation and localization in MRI with 3D convolutional neural networks: application to MRI-targeted prostate biopsy. IEEE Trans Med Imaging 38:1026–1036. https://doi.org/10.1109/TMI.2018.2876796

    Article  PubMed  Google Scholar 

  16. Zijlstra F, Viergever MA, Seevinck PR (2019) SMART tracking: simultaneous anatomical imaging and real-time passive device tracking for MR-guided interventions. Phys Med 64:252–260. https://doi.org/10.1016/j.ejmp.2019.07.019

    Article  CAS  PubMed  Google Scholar 

  17. Ho M, Kim Y, Cheng SS, Gullapalli R, Desai JP (2015) Design, development, and evaluation of an MRI-guided SMA spring-actuated neurosurgical robot. Int J Rob Res 34:1147–1163. https://doi.org/10.1177/0278364915579069

    Article  PubMed  PubMed Central  Google Scholar 

  18. Bergeles C, Vartholomeos P, Qin L, Dupont PE (2013) Closed-loop commutation control of an MRI-powered robot actuator. In: 2013 IEEE international conference on robotics and automation, pp 698–703. https://doi.org/10.1109/ICRA.2013.6630649

  19. Moreira P, Patel N, Wartenberg M, Li G, Tuncali K, Heffter T, Burdette EC, Iordachita I, Fischer GS, Hata N, Tempany CM, Tokuda J (2018) Evaluation of robot-assisted MRI-guided prostate biopsy: needle path analysis during clinical trials. Phys Med Biol 63:20NT02. https://doi.org/10.1088/1361-6560/aae214

    Article  PubMed  PubMed Central  Google Scholar 

  20. Mwikirize C, Nosher JL, Hacihaliloglu I (2018) Signal attenuation maps for needle enhancement and localization in 2D ultrasound. Int J Comput Assist Radiol Surg 13:363–374. https://doi.org/10.1007/s11548-017-1698-y

    Article  PubMed  Google Scholar 

  21. Uherčík M, Kybic J, Liebgott H, Cachard C (2010) Model fitting using RANSAC for surgical tool localization in 3-D ultrasound images. IEEE Trans Biomed Eng 57:1907–1916. https://doi.org/10.1109/TBME.2010.2046416

    Article  PubMed  Google Scholar 

  22. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    Google Scholar 

  23. Pourtaherian A, Ghazvinian Zanjani F, Zinger S, Mihajlovic N, Ng G, Korsten H, de With P (2017) Improving needle detection in 3D ultrasound using orthogonal-plane convolutional networks. In: Medical image computing and computer-assisted intervention (MICCAI), pp 610–618. https://doi.org/10.1007/978-3-319-66185-8_69

  24. Weine J, Breton E, Garnon J, Gangi A, Maier F (2019) Deep learning based needle localization on real-time MR images of patients acquired during MR-guided percutaneous interventions. In: Proceedings of the ISMRM 27th annual meeting, p 973

  25. He K, Gkioxari G, Dollár P, Girshick RB (2017) Mask R-CNN. In: International conference on computer vision (ICCV), pp 2980–2988. https://doi.org/10.1109/ICCV.2017.322

  26. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: International conference on neural information processing systems (NIPS), pp 91–99. https://doi.org/10.1109/TPAMI.2016.2577031

  27. Vuola AO, Akram SU, Kannala J (2019) Mask-RCNN and U-Net ensembled for nuclei segmentation. In: IEEE international conference on biomedical imaging (ISBI), pp 208–212. https://doi.org/10.1109/isbi.2019.8759574

  28. Li X, Raman SS, Lu D, Lee Y, Tsao T, Wu HH (2019) Real-time needle detection and segmentation using Mask R-CNN for MRI-guided interventions. In: Proceedings of the ISMRM 27th annual meeting, p 972

  29. Abdulla W (2017) Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow. GitHub repository

  30. Li X, Lee Y-H, Mikaiel S, Simonelli J, Tsao TC, Wu HH (2020) Respiratory motion prediction using fusion-based multi-rate Kalman filtering and real-time golden-angle radial MRI. IEEE Trans Biomed Eng 67(6):1727–1738. https://doi.org/10.1109/TBME.2019.2944803

    Article  PubMed  Google Scholar 

  31. Hansen MS, Sørensen TS (2013) Gadgetron: an open source framework for medical image reconstruction. Magn Reson Med 69:1768–1776. https://doi.org/10.1002/mrm.24389

    Article  PubMed  Google Scholar 

  32. Sørensen TS, Atkinson D, Schaeffter T, Hansen MS (2009) Real-time reconstruction of sensitivity encoded radial magnetic resonance imaging using a graphics processing unit. IEEE Trans Med Imaging 28:1974–1985. https://doi.org/10.1109/TMI.2009.2027118

    Article  PubMed  Google Scholar 

  33. Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: common objects in context. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) European conference on computer vision (ECCV). Springer, Cham, pp 740–755. https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  34. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: International conference on neural information processing systems (NIPS), pp 3320–3328

  35. Boggs PT, Rogers JE (1990) Orthogonal distance regression. Contemp Math 112:183–194. https://doi.org/10.6028/nist.ir.89-4197

    Article  Google Scholar 

  36. Patel NA, van Katwijk T, Gang Li, Moreira P, Weijian Shang, Misra S, Fischer GS (2015) Closed-loop asymmetric-tip needle steering under continuous intraoperative MRI guidance. In: IEEE engineering in medicine and biology society (EMBC), pp 4869–4874. https://doi.org/10.1109/EMBC.2015.7319484

  37. Kim YK, Kim YK, Park HJ, Park MJ, Lee WJ, Choi D (2014) Noncontrast MRI with diffusion-weighted imaging as the sole imaging modality for detecting liver malignancy in patients with high risk for hepatocellular carcinoma. Magn Reson Imaging 32:610–618. https://doi.org/10.1016/j.mri.2013.12.021

    Article  PubMed  Google Scholar 

  38. Stamey TA, Freiha FS, McNeal JE, Redwine EA, Whittemore AS, Schmid HP (1993) Localized prostate cancer. Relationship of tumor volume to clinical significance for treatment of prostate cancer. Cancer 71:933–938. https://doi.org/10.1002/1097-0142(19930201)71:3+%3c933:aid-cncr2820711408%3e3.0.co;2-l

    Article  CAS  PubMed  Google Scholar 

  39. Renfrew M, Griswold M, Çavusoglu MC (2018) Active localization and tracking of needle and target in robotic image-guided intervention systems. Auton Robots 42:83–97. https://doi.org/10.1007/s10514-017-9640-2

    Article  PubMed  Google Scholar 

  40. Song S-E, Cho NB, Iordachita II, Guion P, Fichtinger G, Whitcomb LL (2011) A study of needle image artifact localization in confirmation imaging of MRI-guided robotic prostate biopsy. In: IEEE international conference on robotics and automation (ICRA), pp 4834–4839. https://doi.org/10.1109/ICRA.2011.5980309

Download references

Acknowledgements

This research received funding support from Siemens Healthineers and the Department of Radiological Sciences at UCLA. The authors appreciate helpful discussions about the manuscript writing with the UCLA Graduate Writing Center and Dr. Le Zhang.

Author information

Authors and Affiliations

  1. Department of Radiological Sciences, University of California Los Angeles, 300 UCLA Medical Plaza, Suite B119, Los Angeles, CA, 90095, USA

    Xinzhou Li, Adam S. Young, Steven S. Raman, David S. Lu & Holden H. Wu

  2. Department of Bioengineering, University of California Los Angeles, Los Angeles, CA, USA

    Xinzhou Li & Holden H. Wu

  3. Department of Mechanical and Aerospace Engineering, University of California Los Angeles, Los Angeles, CA, USA

    Yu-Hsiu Lee & Tsu-Chin Tsao

Authors
  1. Xinzhou Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adam S. Young
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Steven S. Raman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David S. Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yu-Hsiu Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tsu-Chin Tsao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Holden H. Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Holden H. Wu.

Ethics declarations

Conflict of interest

This study was supported in part by Siemens Healthineers. Siemens Healthineers had no role in study design, data collection and analysis, or preparation of this manuscript.

Research involving human participants and informed consent

Prostate datasets were analyzed under an institutional review board approved and HIPAA-compliant retrospective study protocol with a waiver of consent.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Needle feature segmentation (red) and tip tracking (green) videos corresponding to the results reported in Table 4a. The videos are played back with the actual image acquisition rate of 15 frames/s (temporal resolution of 67 ms/frame) (MP4 22219 kb).

Needle feature segmentation (red) and tip (green) tracking videos corresponding to the results reported in Table 4b. The videos are played back with the actual image acquisition rate of 12 frames/s (temporal resolution of 84 ms/frame) (MP4 51805 kb).

Needle feature segmentation (red) and tip (green) tracking videos corresponding to the results reported in Table 4c. The videos are played back with the actual image acquisition rate of 12 frames/s (temporal resolution of 84 ms/frame) (MP4 35742 kb).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Young, A.S., Raman, S.S. et al. Automatic needle tracking using Mask R-CNN for MRI-guided percutaneous interventions. Int J CARS 15, 1673–1684 (2020). https://doi.org/10.1007/s11548-020-02226-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02226-8

Keywords

Search

Navigation