Lei Cai
ABSTRACT
Multi-modality data are widely used in clinical applications, such as tumor detection and brain disease diagnosis. Different modalities can usually provide complementary information, which commonly leads to improved performance. However, some modalities are commonly missing for some subjects due to various technical and practical reasons. As a result, multi-modality data are usually incomplete, raising the multi-modality missing data completion problem. In this work, we formulate the problem as a conditional image generation task and propose an encoder-decoder deep neural network to tackle this problem. Specifically, the model takes the existing modality as input and generates the missing modality. By employing an auxiliary adversarial loss, our model is able to generate high-quality missing modality images. At the same time, we propose to incorporate the available category information of subjects in training to enable the model to generate more informative images. We evaluate our method on the Alzheimer's Disease Neuroimaging Initiative~(ADNI) database, where positron emission tomography~(PET) modalities are missing. Experimental results show that the trained network can generate high-quality PET modalities based on existing magnetic resonance imaging~(MRI) modalities, and provide complementary information to improve the detection and tracking of the Alzheimer's disease. Our results also show that the proposed methods generate higher quality images than baseline methods as measured by various image quality statistics.
- Le An, Pei Zhang, Ehsan Adeli, Yan Wang, Guangkai Ma, Feng Shi, David S Lalush, Weili Lin, and Dinggang Shen . 2016. Multi-level canonical correlation analysis for standard-dose PET image estimation. IEEE Transactions on Image Processing Vol. 25, 7 (2016), 3303--3315.Google Scholar
Digital Library
- Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla . 2017. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence Vol. 39, 12 (2017), 2481--2495.Google Scholar
- Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel . 2016. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems. 2172--2180. Google ScholarDigital Library
- Özgün cCiccek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger . 2016. 3D U-Net: learning dense volumetric segmentation from sparse annotation International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 424--432.Google Scholar
- Emily L Denton, Soumith Chintala, Rob Fergus, et almbox. . 2015. Deep generative image models using a Laplacian pyramid of adversarial networks Advances in neural information processing systems. 1486--1494. Google ScholarDigital Library
- Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang . 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 38, 2 (2016), 295--307. Google ScholarDigital Library
- Hao Dong, Paarth Neekhara, Chao Wu, and Yike Guo . 2017. Unsupervised image-to-image translation with generative adversarial networks. arXiv preprint arXiv:1701.02676 (2017).Google Scholar
- Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox . 2015. Learning to generate chairs with convolutional neural networks Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 1538--1546.Google Scholar
- Hongyang Gao, Hao Yuan, Zhengyang Wang, and Shuiwang Ji . 2017. Pixel Deconvolutional Networks. arXiv preprint arXiv:1705.06820 (2017).Google Scholar
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio . 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680. Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun . 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
- Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten . 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, Vol. Vol. 1. 3.Google Scholar
- Sergey Ioffe and Christian Szegedy . 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015).Google ScholarDigital Library
- Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros . 2017. Image-to-image translation with conditional adversarial networks. arXiv preprint (2017).Google Scholar
- Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu . 2013. 3D convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence Vol. 35, 1 (2013), 221--231. Google ScholarDigital Library
- Diederik P Kingma and Jimmy Ba . 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton . 2012. Imagenet classification with deep convolutional neural networks Advances in neural information processing systems. 1097--1105. Google ScholarDigital Library
- Yann LeCun, Yoshua Bengio, and Geoffrey Hinton . 2015. Deep learning. nature Vol. 521, 7553 (2015), 436.Google Scholar
- Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner . 1998. Gradient-based learning applied to document recognition. Proc. IEEE Vol. 86, 11 (1998), 2278--2324.Google ScholarCross Ref
- Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et almbox. . 2016. Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint (2016).Google Scholar
- Rongjian Li, Wenlu Zhang, Heung-Il Suk, Li Wang, Jiang Li, Dinggang Shen, and Shuiwang Ji . 2014. Deep learning based imaging data completion for improved brain disease diagnosis International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 305--312.Google Scholar
- Jonathan Long, Evan Shelhamer, and Trevor Darrell . 2015. Fully convolutional networks for semantic segmentation Proceedings of the IEEE conference on computer vision and pattern recognition. 3431--3440.Google Scholar
- Michael Mathieu, Camille Couprie, and Yann LeCun . 2015. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440 (2015).Google Scholar
- Mehdi Mirza and Simon Osindero . 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).Google Scholar
- Augustus Odena, Christopher Olah, and Jonathon Shlens . 2016. Conditional image synthesis with auxiliary classifier gans. arXiv preprint arXiv:1610.09585 (2016).Google ScholarDigital Library
- Alec Radford, Luke Metz, and Soumith Chintala . 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).Google Scholar
- Olaf Ronneberger, Philipp Fischer, and Thomas Brox . 2015. U-net: Convolutional networks for biomedical image segmentation International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.Google Scholar
- SC Strother, ME Casey, and EJ Hoffman . 1990. Measuring PET scanner sensitivity: relating countrates to image signal-to-noise ratios using noise equivalents counts. Ieee transactions on nuclear science Vol. 37, 2 (1990), 783--788.Google Scholar
- Qi Wang, Mengying Sun, Liang Zhan, Paul Thompson, Shuiwang Ji, and Jiayu Zhou . 2017 b. Multi-Modality Disease Modeling via Collective Deep Matrix Factorization Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1155--1164. Google ScholarDigital Library
- Yan Wang, Guangkai Ma, Le An, Feng Shi, Pei Zhang, David S Lalush, Xi Wu, Yifei Pu, Jiliu Zhou, and Dinggang Shen . 2017 a. Semisupervised tripled dictionary learning for standard-dose PET image prediction using low-dose PET and multimodal MRI. IEEE Transactions on Biomedical Engineering Vol. 64, 3 (2017), 569--579.Google ScholarCross Ref
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli . 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing Vol. 13, 4 (2004), 600--612. Google ScholarDigital Library
- Michael W Weiner, Dallas P Veitch, Paul S Aisen, Laurel A Beckett, Nigel J Cairns, Robert C Green, Danielle Harvey, Clifford R Jack, William Jagust, Enchi Liu, et almbox. . 2013. The Alzheimer's Disease Neuroimaging Initiative: a review of papers published since its inception. Alzheimer's & dementia: the journal of the Alzheimer's Association Vol. 9, 5 (2013), e111--e194.Google Scholar
- Shuo Xiang, Lei Yuan, Wei Fan, Yalin Wang, Paul M Thompson, and Jieping Ye . 2013. Multi-source learning with block-wise missing data for Alzheimer's disease prediction. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 185--193. Google ScholarDigital Library
- Shuo Xiang, Lei Yuan, Wei Fan, Yalin Wang, Paul M Thompson, and Jieping Ye . 2014. Bi-level multi-source learning for heterogeneous block-wise missing data. NeuroImage Vol. 102, 0 (2014), 192--206.Google ScholarCross Ref
- Tao Xu, Han Zhang, Xiaolei Huang, Shaoting Zhang, and Dimitris N Metaxas . 2016. Multimodal deep learning for cervical dysplasia diagnosis International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 115--123.Google Scholar
- Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, and Dimitris Metaxas . 2017. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In IEEE Int. Conf. Comput. Vision. 5907--5915.Google ScholarCross Ref
- Shaoting Zhang and Dimitris Metaxas . 2016. Large-Scale medical image analytics: Recent methodologies, applications and Future directions. Medical Image Analysis Vol. 33 (2016), 98--101.Google ScholarCross Ref
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros . 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593 (2017).Google Scholar
Index Terms
- Deep Adversarial Learning for Multi-Modality Missing Data Completion
Recommendations
Deep Learning Based Inter-modality Image Registration Supervised by Intra-modality Similarity
Machine Learning in Medical ImagingAbstractNon-rigid inter-modality registration can facilitate accurate information fusion from different modalities, but it is challenging due to the very different image appearances across modalities. In this paper, we propose to train a non-rigid inter-...
Improving PET-CT Image Segmentation via Deep Multi-modality Data Augmentation
Machine Learning for Medical Image ReconstructionAbstractPositron emission tomography (PET) - computed tomography (CT) is a widely-accepted imaging modality for staging, diagnosis and treatment response monitoring of cancers. Deep learning based computer aided diagnosis systems have achieved high ...
Comments