Abstract
Multi-view medical image analysis often depends on the combination of information from multiple views. However, differences in perspective or other forms of misalignment can make it difficult to combine views effectively, as registration is not always possible. Without registration, views can only be combined at a global feature level, by joining feature vectors after global pooling. We present a novel cross-view transformer method to transfer information between unregistered views at the level of spatial feature maps. We demonstrate this method on multi-view mammography and chest X-ray datasets. On both datasets, we find that a cross-view transformer that links spatial feature maps can outperform a baseline model that joins feature vectors after global pooling.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The code for the experiments is available at https://vantulder.net/code/2021/miccai-transformers/.
References
Bekker, A.J., Shalhon, M., Greenspan, H., Goldberger, J.: Multi-view probabilistic classification of breast microcalcifications. IEEE Trans. Med. Imaging 35(2), 645–653 (2015). https://doi.org/10.1109/TMI.2015.2488019
Carneiro, G., Nascimento, J., Bradley, A.P.: Deep learning models for classifying mammogram exams containing unregistered multi-view images and segmentation maps of lesions. In: Zhou, S.K., Greenspan, H., Shen, D. (eds.) Deep Learning for Medical Image Analysis, pp. 321–339. Academic Press (January 2017). https://doi.org/10.1016/B978-0-12-810408-8.00019-5
Hashir, M., Bertrand, H., Cohen, J.P.: Quantifying the value of lateral views in deep learning for chest X-rays. In: Medical Imaging with Deep Learning, pp. 288–303. PMLR (September 2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–780 (2016). https://doi.org/10.3389/fpsyg.2013.00124
Heath, M., Bowyer, K., Kopans, D., Moore, R., Kegelmeyer Jr, P.: The Digital Database for Screening Mammography (2001)
Irvin, J., et al.: CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. arXiv:1901.07031 [cs, eess] (January 2019)
Lee, R.S., Gimenez, F., Hoogi, A., Miyake, K.K., Gorovoy, M., Rubin, D.L.: A curated mammography data set for use in computer-aided detection and diagnosis research. Sci. Data 4, 1–9 (2017). https://doi.org/10.1038/sdata.2017.177
Liu, Y., Zhang, F., Zhang, Q., Wang, S., Wang, Y., Yu, Y.: Cross-view correspondence reasoning based on bipartite graph convolutional network for mammogram mass detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3811–3821. IEEE, Seattle, WA, USA (June 2020). https://doi.org/10.1109/CVPR42600.2020.00387
Ma, J., et al.: Cross-view Relation Networks for Mammogram Mass Detection. arXiv:1907.00528 [cs] (June 2019)
Nasir Khan, H., Shahid, A.R., Raza, B., Dar, A.H., Alquhayz, H.: Multi-view feature fusion based four views model for mammogram classification using convolutional neural network. IEEE Access 7, 165724–165733 (2019). https://doi.org/10.1109/ACCESS.2019.2953318
Rubin, J., Sanghavi, D., Zhao, C., Lee, K., Qadir, A., Xu-Wilson, M.: Large Scale Automated Reading of Frontal and Lateral Chest X-Rays using Dual Convolutional Neural Networks. arXiv:1804.07839 [cs, stat] (April 2018)
Sun, L., Wang, J., Hu, Z., Xu, Y., Cui, Z.: Multi-view convolutional neural networks for mammographic image classification. IEEE Access 7, 126273–126282 (2019). https://doi.org/10.1109/ACCESS.2019.2939167
Vaswani, A., et al.: Attention is all you need. Adv. Neural. Inf. Process. Syst. 30, 5998–6008 (2017)
Wang, H., et al.: Breast mass classification via deeply integrating the contextual information from multi-view data. Pattern Recogn. 80, 42–52 (2018). https://doi.org/10.1016/j.patcog.2018.02.026
Wu, B., et al.: Visual Transformers: Token-based Image Representation and Processing for Computer Vision. arXiv:2006.03677 [cs, eess] (November 2020)
Wu, N., et al.: Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Trans. Med. Imaging 39(4), 1184–1194 (2019). https://doi.org/10.1109/tmi.2019.2945514
Zhao, X., Yu, L., Wang, X.: Cross-view attention network for breast cancer screening from multi-view mammograms. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1050–1054 (May 2020). https://doi.org/10.1109/ICASSP40776.2020.9054612
Acknowledgments
The research leading to these results is part of the project “MARBLE”, funded from the EFRO/OP-Oost under grant number PROJ-00887. Some of the experiments were carried out on the Dutch national e-infrastructure with the support of SURF Cooperative.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
van Tulder, G., Tong, Y., Marchiori, E. (2021). Multi-view Analysis of Unregistered Medical Images Using Cross-View Transformers. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12903. Springer, Cham. https://doi.org/10.1007/978-3-030-87199-4_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-87199-4_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87198-7
Online ISBN: 978-3-030-87199-4
eBook Packages: Computer ScienceComputer Science (R0)
