Skip to main content
SpringerLink
Log in

Multi-task Video Enhancement for Dental Interventions

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2022 (MICCAI 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13437))

Abstract

A microcamera firmly attached to a dental handpiece allows dentists to continuously monitor the progress of conservative dental procedures. Video enhancement in video-assisted dental interventions alleviates low-light, noise, blur, and camera handshakes that collectively degrade visual comfort. To this end, we introduce a novel deep network for multi-task video enhancement that enables macro-visualization of dental scenes. In particular, the proposed network jointly leverages video restoration and temporal alignment in a multi-scale manner for effective video enhancement. Our experiments on videos of natural teeth in phantom scenes demonstrate that the proposed network achieves state-of-the-art results in multiple tasks with near real-time processing. We release Vident-lab at https://doi.org/10.34808/1jby-ay90, the first dataset of dental videos with multi-task labels to facilitate further research in relevant video processing applications.

This work was supported in part by The National Centre for Research and Development, Poland, under grant agreement POIR.01.01.01–00–0076/19.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bradley, A., Klivington, J., Triscari, J., van der Merwe, R.: Cinematic-L1 video stabilization with a log-homography model. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1041–1049 (2021)

    Google Scholar 

  2. Charbonnier, P., Blanc-Feraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of 1st International Conference on Image Processing, vol. 2, pp. 168–172. IEEE (1994)

    Google Scholar 

  3. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 801–818 (2018)

    Google Scholar 

  4. Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4641–4650 (2021)

    Google Scholar 

  5. DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation. arXiv preprint arXiv:1606.03798 (2016)

  6. Ehret, T., Davy, A., Morel, J.M., Facciolo, G., Arias, P.: Model-blind video denoising via frame-to-frame training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11369–11378 (2019)

    Google Scholar 

  7. Jung, H., Kim, Y., Jang, H., Ha, N., Sohn, K.: Multi-task learning framework for motion estimation and dynamic scene deblurring. IEEE Trans. Image Process. 30, 8170–8183 (2021)

    Article  Google Scholar 

  8. Katsaros, E., Ostrowski, P.K., Wȩsierski, D., Jezierska, A.: Concurrent video denoising and deblurring for dynamic scenes. IEEE Access 9, 157437–157446 (2021)

    Article  Google Scholar 

  9. Kokkinos, I.: UberNet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In: Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, pp. 6129–6138 (2017)

    Google Scholar 

  10. Kühnisch, J., Meyer, O., Hesenius, M., Hickel, R., Gruhn, V.: Caries detection on intraoral images using artificial intelligence. J. Dent. Res. 101(2), 158–165 (2021)

    Google Scholar 

  11. Lai, W.S., Huang, J.B., Wang, O., Shechtman, E., Yumer, E., Yang, M.H.: Learning blind video temporal consistency. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 170–185 (2018)

    Google Scholar 

  12. Le, H., Liu, F., Zhang, S., Agarwala, A.: Deep homography estimation for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7652–7661 (2020)

    Google Scholar 

  13. Low, J.F., Dom, T.N.M., Baharin, S.A.: Magnification in endodontics: a review of its application and acceptance among dental practitioners. Eur. J. Dent. 12(04), 610–616 (2018)

    Article  Google Scholar 

  14. Mao, X., Liu, Y., Shen, W., Li, Q., Wang, Y.: Deep residual fourier transformation for single image deblurring. arXiv preprint arXiv:2111.11745 (2021)

  15. Marstal, K., Berendsen, F., Staring, M., Klein, S.: SimpleElastix: a user-friendly, multi-lingual library for medical image registration. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 134–142 (2016)

    Google Scholar 

  16. Misra, I., Shrivastava, A., Gupta, A., Hebert, M.: Cross-stitch networks for multi-task learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994–4003 (2016)

    Google Scholar 

  17. Murphy, K.P.: Machine learning: a probabilistic perspective. MIT press (2012)

    Google Scholar 

  18. Nekrasov, V., Dharmasiri, T., Spek, A., Drummond, T., Shen, C., Reid, I.: Real-time joint semantic segmentation and depth estimation using asymmetric annotations. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 7101–7107. IEEE (2019)

    Google Scholar 

  19. Niklaus, S., Mai, L., Liu, F.: Video frame interpolation via adaptive separable convolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 261–270 (2017)

    Google Scholar 

  20. Rashid, U., et al.: A hybrid mask RCNN-based tool to localize dental cavities from real-time mixed photographic images. PeerJ Comput. Sci. 8, e888 (2022)

    Google Scholar 

  21. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  22. Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019)

    Google Scholar 

  23. Teed, Z., Deng, J.: RAFT: recurrent all-pairs field transforms for optical flow. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 402–419. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_24

    Chapter  Google Scholar 

  24. Vandenhende, S., Georgoulis, S., Van Gool, L.: MTI-Net: multi-scale task interaction networks for multi-task learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 527–543. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_31

    Chapter  Google Scholar 

  25. Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D., Van Gool, L.: Multi-task learning for dense prediction tasks: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43 (2021)

    Google Scholar 

  26. Wang, X., Chan, K.C., Yu, K., Dong, C., Change Loy, C.: EDVR: video restoration with enhanced deformable convolutional networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1954–1963 (2019)

    Google Scholar 

  27. Xu, D., Ouyang, W., Wang, X., Sebe, N.: PAD-Net: multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 675–684 (2018)

    Google Scholar 

  28. Zhang, M., et al.: RT-VENet: a convolutional network for real-time video enhancement. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 4088–4097 (2020)

    Google Scholar 

  29. Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., Yang, J.: Pattern-affinitive propagation across depth, surface normal and semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4106–4115 (2019)

    Google Scholar 

  30. Zhong, Z., Gao, Y., Zheng, Y., Zheng, B.: Efficient spatio-temporal recurrent neural network for video deblurring. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 191–207. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_12

    Chapter  Google Scholar 

  31. Zhou, S., Zhang, J., Pan, J., Xie, H., Zuo, W., Ren, J.: Spatio-temporal filter adaptive network for video deblurring. In: Proceedings of the IEEE International Conference on Computer Vision (2019)

    Google Scholar 

  32. Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J.: UNet++: a nested u-net architecture for medical image segmentation. In: Stoyanov, D., et al. (eds.) DLMIA/ML-CDS -2018. LNCS, vol. 11045, pp. 3–11. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00889-5_1

    Chapter  Google Scholar 

  33. Zhu, G., Piao, Z., Kim, S.C.: Tooth detection and segmentation with mask R-CNN. In: 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pp. 070–072. IEEE (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Gdańsk University of Technology, Gdansk, Poland

    Efklidis Katsaros, Piotr K. Ostrowski, Krzysztof Włódarczak, Emilia Lewandowska, Jacek Ruminski, Anna Jezierska & Daniel Węsierski

  2. Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland

    Anna Jezierska

  3. Dental Sense Medicover, Gdansk, Poland

    Łukasz Lassmann

  4. Master Level Technologies, Gdansk, Poland

    Damian Siupka-Mróz & Łukasz Lassmann

Authors
  1. Efklidis Katsaros
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Piotr K. Ostrowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Krzysztof Włódarczak
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emilia Lewandowska
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jacek Ruminski
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Damian Siupka-Mróz
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Łukasz Lassmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Anna Jezierska
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Daniel Węsierski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Węsierski .

Editor information

Editors and Affiliations

  1. Rochester Institute of Technology, Rochester, NY, USA

    Linwei Wang

  2. Chinese University of Hong Kong, Hong Kong, Hong Kong

    Qi Dou

  3. University of Virginia, Charlottesville, VA, USA

    P. Thomas Fletcher

  4. National Center for Tumor Diseases (NCT/UCC), Dresden, Germany

    Stefanie Speidel

  5. Case Western Reserve University, Cleveland, OH, USA

    Shuo Li

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 14286 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Katsaros, E. et al. (2022). Multi-task Video Enhancement for Dental Interventions. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13437. Springer, Cham. https://doi.org/10.1007/978-3-031-16449-1_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16449-1_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16448-4

  • Online ISBN: 978-3-031-16449-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation