Skip to main content
SpringerLink
Log in

Mathematical Foundations of AIM

  • Reference work entry
  • First Online:
Artificial Intelligence in Medicine

  • 656 Accesses

Abstract

With the tremendous success of deep learning in recent years, the field of medical imaging has undergone unprecedented changes. Despite the great success of deep learning in medical imaging, these recent developments are largely empirical. Our goal in this chapter is to provide an overview of some of the key mathematical foundations of deep learning to the medical imaging community. In particular, we will consider ties with traditional machine learning methods, unrolling techniques which connect deep learning to iterative algorithms, and geometric interpretations of modern deep networks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 699.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 1,199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inform Process Syst. 2012;25:1097–1105.

    Google Scholar 

  2. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations, 2015.

    Google Scholar 

  3. Riesenhuber M, Poggio T. Hierarchical models of object recognition in cortex. Nat Neurosci. 1999;2(11):1019–25.

    Article  CAS  Google Scholar 

  4. Jin KH, McCann MT, Froustey E, Unser M. Deep convolutional neural network for inverse problems in imaging. IEEE Trans Image Process. 2017;26(9):4509–22.

    Article  Google Scholar 

  5. Ye JC, Han Y, Cha E. Deep convolutional framelets: a general deep learning framework for inverse problems. SIAM J Imag Sci. 2018;11(2):991–1048.

    Article  Google Scholar 

  6. Ye JC, Sung WK. Understanding geometry of encoder-decoder CNNs. Int Conf Mach Learn, 2019;97:7064–7073.

    Google Scholar 

  7. Gregor K, LeCun Y. Learning fast approximations of sparse coding. Int Conf Mach Learn, 2010, p. 399–406.

    Google Scholar 

  8. Monga V, Li Y, Eldar YC. Algorithm unrolling: interpretable, efficient deep learning for signal and image processing. IEEE Signal Process Mag. 2021;38(2):18–44.

    Article  Google Scholar 

  9. Hammernik K, Klatzer T, Kobler E, Recht MP, Sodickson DK, Pock T, Knoll F. Learning a variational network for reconstruction of accelerated MRI data. Magn Reson Med. 2018;79(6):3055–71.

    Article  Google Scholar 

  10. Sun J, Li H, Xu Z et al. Deep ADMM-Net for compressive sensing MRI. Adv Neural Inf Proces Syst, 2016;29:10–18.

    Google Scholar 

  11. Eldar YC, Kutyniok G. Compressed sensing: theory and applications. Cambridge: Cambridge University Press; 2012.

    Google Scholar 

  12. Unser M. A representer theorem for deep neural networks. J Mach Learn Res. 2019;20(110):1–30.

    Google Scholar 

  13. Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the Brain. Psychol Rev. 1958;65(6):386.

    Article  CAS  Google Scholar 

  14. Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature. 1986;323(6088):533–6.

    Article  Google Scholar 

  15. Cybenko G. Approximation by superpositions of a sigmoidal function. Math Control Signals Syst. 1989;2(4):303–14.

    Article  Google Scholar 

  16. Telgarsky M. Benefits of depth in neural networks. In: Conference on learning theory. PMLR; 2016, pp. 1517–1539.

    Google Scholar 

  17. Eldan R, Shamir O. The power of depth for feedforward neural networks. In: Conference on learning theory. PMLR; 2016. pp. 907–940.

    Google Scholar 

  18. Raghu M, Poole B, Kleinberg J, Ganguli S, Sohl-Dickstein J. On the expressive power of deep neural networks. In: International conference on machine learning. PMLR; 2017. pp. 2847–2854.

    Google Scholar 

  19. Yarotsky D. Error bounds for approximations with deep ReLU networks. Neural Netw. 2017;94:103–14.

    Article  Google Scholar 

  20. Bishop CM. Pattern recognition and machine learning. New York: Springer; 2006.

    Google Scholar 

  21. Schölkopf B, Smola AJ, Bach F, et al. Learning with kernels: support vector machines, regularization, optimization, and beyond. London: MIT Press; 2002.

    Google Scholar 

  22. Vapnik V. The nature of statistical learning theory. New York: Springer Science & Business Media; 2013.

    Google Scholar 

  23. Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings; 2011, pp. 315–323.

    Google Scholar 

  24. LeCun YA, Bottou L, Orr GB, Müller K. Efficient BackProp. In: Neural networks: tricks of the trade, Lecture notes in computer science. Berlin, Heidelberg: Springer; 2012. p. 9–48.

    Chapter  Google Scholar 

  25. Fukushima K. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern. 1980;36(4):193–202.

    Article  CAS  Google Scholar 

  26. Hubel DH, Wiesel TN. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol. 1962;160(1):106–54.

    Article  CAS  Google Scholar 

  27. Qayyum A, Anwar SM, Awais M, Majid M. Medical image retrieval using deep convolutional neural network. Neurocomputing. 2017;266:8–20.

    Article  Google Scholar 

  28. Ibtehaz N, Rahman MS. MultiResUNet: rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Netw. 2019;121:74–87.

    Google Scholar 

  29. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hub-bard W, Jackel LD. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1989;1(4):541–51.

    Article  Google Scholar 

  30. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: IEEE conference on computer vision and pattern recognition, 2015, p. 3431–3440.

    Google Scholar 

  31. Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics; 2015. p. 315–323.

    Google Scholar 

  32. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15(1):1929–58.

    Google Scholar 

  33. Hosseini SAH, Yaman B, Moeller S, Hong M, Akçakaya M. Dense recurrent neural networks for accelerated MRI: history-cognizant unrolling of optimization algorithms. IEEE J Select Topics Signal Process. 2020;14(6):1280–91.

    Article  Google Scholar 

  34. Goodfellow I, Bengio Y, Courville A. Deep learning. Cambridge: MIT Press; 2016.

    Google Scholar 

  35. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–80.

    Article  CAS  Google Scholar 

  36. Chung J, Gulcehre C, Cho K, Bengio Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 Workshop on Deep Learning, December 2014.

    Google Scholar 

  37. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Adv Neural Inform Process Syst. 2017, pp. 5998–6008.

    Google Scholar 

  38. Cha E, Oh G, Ye JC. Geometric approaches to increase the expressivity of deep neural networks for MR reconstruction. IEEE J Select Topic Signal Process. 2020;14(6):1292–305.

    Article  Google Scholar 

  39. Zhou D-X. Universality of deep convolutional neural networks. Appl Comput Harmon Anal. 2020;48(2):787–94.

    Article  Google Scholar 

  40. Belkin M, Hsu D, Ma S, Mandal S. Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proc Natl Acad Sci. 2019;116(32):15849–54.

    Article  CAS  Google Scholar 

  41. Belkin M, Hsu D, Xu J. Two models of double descent for weak features. SIAM J Math Data Sci. 2020;2(4):1167–80.

    Article  Google Scholar 

  42. Chen SS, Donoho DL, Saunders MA. Atomic decomposition by basis pursuit. SIAM Rev. 2001;43(1):129–59.

    Article  Google Scholar 

  43. Daubechies I, Defrise M, De Mol C. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun Pure Appl Math. 2004;57(11):1413–57.

    Article  Google Scholar 

  44. Xin B, Wang Y, Gao W, Wipf D, Wang B. Maximal sparsity with deep networks? Adv Neural Inf Process Syst, 2016;29:4340–4348.

    Google Scholar 

  45. Chen X, Liu J, Wang Z, Yin W. Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS’18). Red Hook, NY, USA: Curran Associates Inc.; 2018, pp. 9079–9089.

    Google Scholar 

  46. Liu J, Chen X, Wang Z, Yin W ALISTA: analytic weights are as good as learned weights in LISTA. In: International conference on learning representations. 2019.

    Google Scholar 

  47. Solomon O, Eldar YC, Mutzafi M, Segev M. SPARCOM: sparsity based super-resolution correlation microscopy. SIAM J Imag Sci. 2019;12(1):392–419.

    Article  Google Scholar 

  48. Wang Z, Liu D, Yang J, Han W, Huang T. Deep networks for image super-resolution with sparse prior. In: Proceedings of the IEEE international conference on computer vision. 2015, pp. 370–378.

    Google Scholar 

  49. Hauptmann A, Lucka F, Betcke M, Huynh N, Adler J, Cox B, Beard P, Ourselin S, Arridge S. Model-based learning for accelerated, limited-view 3-D photoacoustic tomography. IEEE Trans Med Imaging. 2018;37(6):1382–93.

    Article  Google Scholar 

  50. Adler J, Öktem O. Learned primal-dual reconstruction. IEEE Trans Med Imaging. 2018;37(6):1322–32.

    Article  Google Scholar 

  51. Solomon O, Cohen R, Zhang Y, Yang Y, He Q, Luo J, van Sloun RJG, Eldar YC. Deep unfolded robust PCA with application to clutter suppression in ultrasound. IEEE Trans Med Imaging. 2020;39(4):1051–63.

    Article  Google Scholar 

  52. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770–778.

    Google Scholar 

  53. Dardikman-Yoffe G, Eldar YC. Learned SPARCOM: unfolded deep super-resolution microscopy. Opt Express. 2020;28(19):27736–63.

    Article  Google Scholar 

  54. Hubel DH, Wiesel TN. Receptive fields of single neurones in the cat’s striate cortex. J Physiol. 1959;148(3):574–91.

    Article  CAS  Google Scholar 

  55. Quiroga RQ, Reddy L, Kreiman G, Koch C, Fried I. Invariant visual representation by single neurons in the human brain. Nature. 2005;435(7045):1102–7.

    Article  CAS  Google Scholar 

  56. Roweis ST, Saul LK. Nonlinear dimensionality reduction by locally linear embedding. Science. 2000;290(5500):2323–6.

    Article  CAS  Google Scholar 

  57. Tenenbaum JB, De Silva V, Langford JC. A global geometric framework for nonlinear dimensionality reduction. Science. 2000;290(5500):2319–23.

    Article  CAS  Google Scholar 

  58. Kingma DP, Welling M. Auto-encoding variational Bayes. In: International conference on learning representations. 2014.

    Google Scholar 

  59. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. Adv Neural Inform Process Syst. 2014, 27.

    Google Scholar 

  60. Rezende D, Mohamed S. Variational inference with normalizing flows. In: International conference on machine learning. PMLR; 2015, pp. 1530–1538.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Math and Computer Science, Weizmann Institute of Science, Rehovot, Israel

    Yonina C. Eldar

  2. Amazon, San Jose, CA, USA

    Yuelong Li

  3. Department Bio and Brain Engineering & Department Mathematical Sciences, Korea Advanced Institute of Science & Technology (KAIST), Daejeon, Republic of Korea

    Jong Chul Ye

Authors
  1. Yonina C. Eldar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuelong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jong Chul Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yonina C. Eldar .

Editor information

Editors and Affiliations

  1. Department of Women’s and Children’s Health, Karolinska Institutet, Stockholm, Sweden

    Niklas Lidströmer

  2. Imperial College London, London, UK

    Hutan Ashrafian

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Eldar, Y.C., Li, Y., Ye, J.C. (2022). Mathematical Foundations of AIM. In: Lidströmer, N., Ashrafian, H. (eds) Artificial Intelligence in Medicine. Springer, Cham. https://doi.org/10.1007/978-3-030-64573-1_333

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-64573-1_333

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-64572-4

  • Online ISBN: 978-3-030-64573-1

  • eBook Packages: MedicineReference Module Medicine

Publish with us

Policies and ethics

Search

Navigation