Skip to main content

Advertisement

Springer Nature Link
Log in

Semantic segmentation in medical images through transfused convolution and transformer networks

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Recent decades have witnessed rapid development in the field of medical image segmentation. Deep learning-based fully convolution neural networks have played a significant role in the development of automated medical image segmentation models. Though immensely effective, such networks only take into account localized features and are unable to capitalize on the global context of medical image. In this paper, two deep learning based models have been proposed namely USegTransformer-P and USegTransformer-S. The proposed models capitalize upon local features and global features by amalgamating the transformer-based encoders and convolution-based encoders to segment medical images with high precision. Both the proposed models deliver promising results, performing better than the previous state of the art models in various segmentation tasks such as Brain tumor, Lung nodules, Skin lesion and Nuclei segmentation. The authors believe that the ability of USegTransformer-P and USegTransformer-S to perform segmentation with high precision could remarkably benefit medical practitioners and radiologists around the world.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  1. Ganguly D, Chakraborty S, Balitanas M, Kim TH (2010) Medical imaging: a review. Commun Comput Inf Sci 78(CCIS):504–516. https://doi.org/10.1007/978-3-642-16444-6_63

    Article  Google Scholar 

  2. Cao H, Liu H, Song E, Hung CC, Ma G, Xu X, Jin R, Lu J (2020) Dual-branch residual network for lung nodule segmentation. Appl Soft Comput J 86:105934. https://doi.org/10.1016/j.asoc.2019.105934

    Article  Google Scholar 

  3. Hashemzehi R, Mahdavi SJS, Kheirabadi M, Kamel SR (2020) Detection of brain tumors from MRI images base on deep learning using hybrid model CNN and NADE. Biocybern Biomed Eng 40:1225–1232

    Article  Google Scholar 

  4. Garcia-Arroyo JL, Garcia-Zapirain B (2019) Segmentation of skin lesions in dermoscopy images using fuzzy classification of pixels and histogram thresholding. Comput Methods Prog Biomed 168:11–19. https://doi.org/10.1016/j.cmpb.2018.11.001

    Article  Google Scholar 

  5. Nogueira-Rodríguez A, Domínguez-Carbajales R, López-Fernández H, Iglesias Á, Cubiella J, Fdez-Riverola F, Reboiro-Jato M, Glez-Peña D (2021) Deep neural networks approaches for detecting and classifying colorectal polyps. Neurocomputing 423:721–734. https://doi.org/10.1016/j.neucom.2020.02.123

    Article  Google Scholar 

  6. Chlebus G, Schenk A, Moltz JH, van Ginneken B, Hahn HK, Meine H (2018) Automatic liver tumor segmentation in CT with fully convolutional neural networks and object-based postprocessing. Sci Rep 8:15497

    Article  Google Scholar 

  7. Lal S, Das D, Alabhya K, Kanfade A, Kumar A, Kini J (2021) NucleiSegNet: robust deep learning architecture for the nuclei segmentation of liver cancer histopathology images. Comput Biol Med 128:104075. https://doi.org/10.1016/j.compbiomed.2020.104075

    Article  Google Scholar 

  8. Sharma N, Aggarwal LM (2010) Automated medical image segmentation techniques. J Med Phys 35(1):3–14. https://doi.org/10.4103/0971-6203.58777

    Article  Google Scholar 

  9. Ramesh N, Yoo JH, Sethi IK (1995) Thresholding based on histogram approximation. IEE Proc Vision, Image Signal Process 142:271. https://doi.org/10.1049/ip-vis:19952007

    Article  Google Scholar 

  10. Sharma N, Ray A, Sharma S, Shukla KK, Pradhan S, Aggarwal LM (2008) Segmentation and classification of medical images using texture-primitive features: application of BAM-type artificial neural network. J Med Phys 33:119–126. https://doi.org/10.4103/0971-6203.42763

    Article  Google Scholar 

  11. Gletsos M, Mougiakakou SG, Matsopoulos GK, et al (2003) A computer-aided diagnostic system to characterize CT focal liver Lesions: Design and Optimization of a Neural Network Classifier. IEEE Trans Inf Technol Biomed. https://doi.org/10.1109/TITB.2003.813793

  12. Zheng X, Lei Q, Yao R, Gong Y, Yin Q (2018) Image segmentation based on adaptive K-means algorithm. Eurasip J Image Video Process 2018. https://doi.org/10.1186/s13640-018-0309-3

  13. Ahmadi N (2020) A hybrid intelligent approach for image segmentation and feature extraction using fuzzy clustering, lattice boltzmann and GLDM techniques. J Soft Comput Decis Support Syst 7:1–5. https://doi.org/10.5815/ijigsp.2012.06.01

  14. Bezdek JC, Ehrlich R, Full W (1984) FCM: the fuzzy c-means clustering algorithm. Comput Geosci 10:191–203. https://doi.org/10.1016/0098-3004(84)90020-7

    Article  Google Scholar 

  15. Krizhevsky A, Sutskever I, Hinton GE (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60:84–90. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  16. Ciresan DC, Giusti A, Gambardella LM, Schmidhuber J (2012) Deep neural networks segment neuronal membranes in electron microscopy images. In: NIPS, pp 2852–2860

  17. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. MICCAI. https://doi.org/10.48550/arXiv.1505.04597

  18. Ibtehaz N, Rahman MS (2020) MultiResUNet: rethinking the U-net architecture for multimodal biomedical image segmentation. Neural Netw 121:74–87. https://doi.org/10.1016/j.neunet.2019.08.025

    Article  Google Scholar 

  19. Lou A, Guan S, Loew M (2020) DC-UNet: rethinking the U-net architecture with dual channel efficient CNN for medical images segmentation. arXiv. https://doi.org/10.48550/arXiv.2006.00414

  20. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2020) UNet++: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging 39:1856–1867. https://doi.org/10.1109/TMI.2019.2959609

    Article  Google Scholar 

  21. Azad R, Asadi-Aghbolaghi M, Fathy M, Escalera S (2019) Bi-directional ConvLSTM U-net with densley connected convolutions. In: Proceedings - 2019 international conference on computer vision workshop, ICCVW 2019. https://doi.org/10.48550/arXiv.1909.00166

  22. Jha D, Riegler MA, Johansen D et al (2020) DoubleU-net: a deep convolutional neural network for medical image segmentation. In: Proceedings - IEEE Symposium on Computer-Based Medical Systems. https://doi.org/10.48550/arXiv.2006.04868

  23. Chen L, Bentley P, Mori K, Misawa K, Fujiwara M, Rueckert D (2018) DRINet for medical image segmentation. IEEE Trans Med Imaging 37:2453–2462. https://doi.org/10.1109/TMI.2018.2835303

    Article  Google Scholar 

  24. Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Zhang T, Gao S, Liu J (2019) CE-net: context encoder network for 2D medical image segmentation. IEEE Trans Med Imaging 38:2281–2292. https://doi.org/10.1109/TMI.2019.2903562

    Article  Google Scholar 

  25. Dosovitskiy A, Beyer L, Kolesnikov A et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. ICLR 2021. https://doi.org/10.48550/arXiv.2010.11929

  26. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017). Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). Curran Associates Inc., Red Hook, NY, USA, pp 6000–6010

  27. Nishio M, Nagashima C, Hirabayashi S, Ohnishi A, Sasaki K, Sagawa T, Hamada M, Yamashita T (2017) Convolutional auto-encoders for image denoising of ultra-low-dose CT. Heliyon 3:e00393. https://doi.org/10.1016/j.heliyon.2017.e00393

    Article  Google Scholar 

  28. Brain MRI segmentation | Kaggle (n.d.) https://www.kaggle.com/mateuszbuda/lgg-mri-segmentation

  29. Finding and Measuring Lungs in CT Data | Kaggle (n.d.) https://www.kaggle.com/kmader/finding-lungs-in-ct-data/data/

  30. Caicedo JC, Goodman A, Karhohs KW, Cimini BA, Ackerman J, Haghighi M, Heng CK, Becker T, Doan M, McQuin C, Rohban M, Singh S, Carpenter AE (2019) Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nat Methods 16:1247–1253. https://doi.org/10.1038/s41592-019-0612-7

    Article  Google Scholar 

  31. Codella NC, Rotemberg VM, Tschandl P, Celebi ME, Dusza SW, Gutman D, Helba B, Kalloo A, Liopyris K, Marchetti MA, Kittler H, Halpern AC (2019) Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (ISIC). ArXiv, abs/1902.03368

  32. Tschandl P, Rosendahl C, Kittler H (2018) Data descriptor: the HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci Data 5:180161. https://doi.org/10.1038/sdata.2018.161

    Article  Google Scholar 

  33. COVID-19 - Medical segmentation (n.d.) http://medicalsegmentation.com/covid19/

  34. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison S, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) PyTorch: An imperative style, high-performance deep learning library. NeurIPS. https://doi.org/10.48550/arXiv.1912.01703

  35. Reddi SJ, Kale S, Kumar S (2018) On the convergence of Adam and beyond. In: 6th international conference on learning representations, ICLR 2018 - conference track proceedings. https://doi.org/10.48550/arXiv.1904.09237

  36. Rizwan I, Haque I, Neubert J (2020) Deep learning approaches to biomedical image segmentation. Informatics Med Unlocked 18:100297/1–100297/10029711. https://doi.org/10.1016/j.imu.2020.100297

  37. Billot B, Greve DN, van Leemput K et al (2020) A learning strategy for contrast-agnostic MRI segmentation.Proceedings of the Third Conference on Medical Imaging with Deep Learning 121:75-93. https://doi.org/10.48550/arXiv.2003.01995

  38. Alom MZ, Yakopcic C, Hasan M, Taha TM, Asari VK (2019) Recurrent residual U-net for medical image segmentation. J Med Imaging 6:1. https://doi.org/10.1117/1.jmi.6.1.014006

    Article  Google Scholar 

  39. Zhang Z, Fu H, Dai H, Shen J, Pang Y, Shao L (2019) ET-Net: A generic edge-aTtention guidance network for medical image segmentation. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. Springer-Verlag, Berlin, Heidelberg, pp 442–450. https://doi.org/10.1007/978-3-030-32239-7_49

  40. Oktay O, Schlemper J, Folgoc LL, Lee MJ, Heinrich MP, Misawa K, Mori K, McDonagh SG, Hammerla NY, Kainz B, Glocker B, Rueckert D (2018) Attention U-Net: learning where to look for the pancreas. ArXiv, abs/1804.03999

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, Delhi Technological University, New Delhi, India

    Tashvik Dhamija

  2. Department of Electrical Engineering, Delhi Technological University, New Delhi, India

    Anunay Gupta

  3. Department of Civil Engineering, Delhi Technological University, New Delhi, India

    Shreyansh Gupta

  4. Department of Computer Science Engineering, Delhi Technological University, New Delhi, India

    Anjum & Rahul Katarya

  5. Department of Electrical and Electronic Engineering Science, University of Johannesburg, Johannesburg, South Africa

    Ghanshyam Singh

Authors
  1. Tashvik Dhamija
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Anunay Gupta
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Shreyansh Gupta
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Anjum
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Rahul Katarya
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Ghanshyam Singh
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rahul Katarya.

Ethics declarations

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dhamija, T., Gupta, A., Gupta, S. et al. Semantic segmentation in medical images through transfused convolution and transformer networks. Appl Intell 53, 1132–1148 (2023). https://doi.org/10.1007/s10489-022-03642-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03642-w

Keywords

Profiles

  1. Anjum View author profile
  2. Ghanshyam Singh View author profile