Skip to main content
SpringerLink
Log in

Deep Image Prior

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, super-resolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs. Apart from its diverse applications, our approach highlights the inductive bias captured by standard generator network architectures. It also bridges the gap between two very popular families of image restoration methods: learning-based methods using deep convolutional networks and learning-free methods based on handcrafted image priors such as self-similarity (Code and supplementary material are available at https://dmitryulyanov.github.io/deep_image_prior).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. Equation (2) can also be thought of as a regularizer R(x) in the style of (1), where \(R(x)=0\) for all images that can be generated by a deep ConvNet of a certain architecture with the weights being not too far from random initialization, and \(R(x)=+\infty \) for all other signals.

  2. http://www.cs.tut.fi/~foi/GCF-BM3D/index.html#ref_results.

References

  • Athar, S., Burnaev, E., & Lempitsky, V. S. (2018). Latent convolutional models. In CoRR.

  • Bahat, Y., Efrat, N., & Irani, M. (2017). Non-uniform blind deblurring by reblurring. In Proceedings of CVPR (pp. 3286–3294). IEEE Computer Society.

  • Bevilacqua, M., Roumy, A., Guillemot, C., & Alberi-Morel, M. (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC (pp. 1–10).

  • Bojanowski, P., Joulin, A., Lopez-Paz, D., & Szlam, A. (2017). Optimizing the latent space of generative networks. In CoRR.

  • Boominathan, L., Maniparambil, M., Gupta, H., Baburajan, R., & Mitra, K. (2018). Phase retrieval for fourier ptychography under varying amount of measurements. In CoRR.

  • Bristow, H., Eriksson, A. P., & Lucey, S. (2013). Fast convolutional sparse coding. In CVPR (pp. 391–398). IEEE Computer Society.

  • Buades, A. (2005). NLM demo. Retrieved December 2017 from http://demo.ipol.im/demo/bcm_non_local_means_denoising/.

  • Buades, A., Coll, B., & Morel, J. M. (2005). A non-local algorithm for image denoising. In Proceedings of CVPR (Vol. 2, pp. 60–65). IEEE Computer Society.

  • Burger, M., Osher, S. J., Xu, J., & Gilboa, G. (2005). Nonlinear inverse scale space methods for image restoration. In Variational, geometric, and level set methods in computer vision, third international workshop, VLSM (pp. 25–36).

  • Burger, H. C., Schuler, C. J., & Harmeling, S. (2012). Image denoising: Can plain neural networks compete with bm3d? In CVPR (pp. 2392–2399).

  • Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8), 2080–2095.

    Article  MathSciNet  Google Scholar 

  • Dong, C., Loy, C.C., He, K., & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In Proceedings of ECCV (pp. 184–199).

  • Dosovitskiy, A., & Brox, T. (2016a). Generating images with perceptual similarity metrics based on deep networks. In NIPS (pp. 658–666).

  • Dosovitskiy, A., & Brox, T. (2016b). Inverting convolutional networks with convolutional networks. In CVPR. IEEE Computer Society.

  • Dosovitskiy, A., Tobias Springenberg, J., & Brox, T. (2015). Learning to generate chairs with convolutional neural networks. In Proceedings of CVPR (pp. 1538–1546).

  • Erhan, D., Bengio, Y., Courville, A., & Vincent, P. (2009). Visualizing higher-layer features of a deep network. Tech. Rep. Technical Report 1341, University of Montreal.

  • Field, D. J. (1987). Relations between the statistics of natural images and the response properties of cortical cells. Josa A, 4(12), 2379–2394.

    Article  Google Scholar 

  • Glasner, D., Bagon, S., & Irani, M. (2009). Super-resolution from a single image. In Proceedings of ICCV (pp. 349–356).

  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Proceedings of NIPS (pp. 2672–2680).

  • Grosse, R. B., Raina, R., Kwong, H., & Ng, A. Y. (2007). Shift-invariance sparse coding for audio classification. In UAI (pp. 149–158). AUAI Press.

  • Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., & Zhang, L. (2015). Convolutional sparse coding for image super-resolution. In ICCV (pp. 1823–1831). IEEE Computer Society.

  • He, K., Sun, J., & Tang, X. (2013). Guided image filtering. T-PAMI, 35(6), 1397–1409.

    Article  Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In CVPR (pp. 1026–1034). IEEE Computer Society.

  • Heide, F., Heidrich, W., & Wetzstein, G. (2015). Fast and flexible convolutional sparse coding. In CVPR (pp. 5135–5143). IEEE Computer Society.

  • Huang, J., & Mumford, D. (1999). Statistics of natural images and models. In CVPR (pp. 1541–1547). IEEE Computer Society.

  • Huang, J.B., Singh, A., & Ahuja, N. (2015). Single image super-resolution from transformed self-exemplars. In CVPR (pp. 5197–5206). IEEE Computer Society.

  • Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017). Globally and locally consistent image completion. ACM Transactions on Graphics (Proceedings of SIGGRAPH)36(4), 107:1–107:14 (2017)

  • Ilyas, A., Jalal, A., Asteri, E., Daskalakis, C., & Dimakis, A. G. (2017). The robust manifold defense: Adversarial training using generative models. In CoRR.

  • Kim, J., Lee, J. K., & Lee, K. M. (2016). Accurate image super-resolution using very deep convolutional networks. In CVPR (pp. 1646–1654). IEEE Computer Society.

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. In CoRR.

  • Kingma, D. P., & Welling, M. (2014). Auto-encoding variational bayes. In Proceedings of ICLR.

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, K. Q. Weinberger (Eds.) Advances in neural information processing systems (Vol. 25, pp. 1097–1105). New York:Curran Associates, Inc.

  • Lai, W. S., Huang, J. B., Ahuja, N., & Yang, M. H. (2017). Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR. IEEE Computer Society.

  • Lebrun, M. (2011). BM3D code. Retrieved December 2017 from https://github.com/gfacciol/bm3d.

  • Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In CVPR. IEEE Computer Society.

  • Lefkimmiatis, S. (2016). Non-local color image denoising with convolutional neural networks. In CVPR. IEEE Computer Society.

  • Mahendran, A., & Vedaldi, A. (2015). Understanding deep image representations by inverting them. In CVPR. IEEE Computer Society.

  • Mahendran, A., & Vedaldi, A. (2016). Visualizing deep convolutional neural networks using natural pre-images. In IJCV.

  • Mairal, J., Bach, F., Ponce, J., & Sapiro, G. (2010). Online learning for matrix factorization and sparse coding. Journal of Machine Learning Research, 11(Jan), 19–60.

    MathSciNet  MATH  Google Scholar 

  • Mallat, S. G. (1989). A theory for multiresolution signal decomposition: The wavelet representation. PAMI, 11(7), 674–693.

    Article  Google Scholar 

  • Marquina, A. (2009). Nonlinear inverse scale space methods for total variation blind deconvolution. SIAM Journal on Imaging Sciences, 2(1), 64–83.

    Article  MathSciNet  Google Scholar 

  • Papyan, V., Romano, Y., & Elad, M. (2017). Convolutional neural networks analyzed via convolutional sparse coding. Journal of Machine Learning Research, 18(83), 1–52.

    MathSciNet  MATH  Google Scholar 

  • Papyan, V., Romano, Y., Sulam, J., & Elad, M. (2017). Convolutional dictionary learning via local processing. In ICCV. IEEE Computer Society.

  • Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M. F., Hoppe, H., & Toyama, K. (2004). Digital photography with flash and no-flash image pairs. ACM Transactions on Graphics, 23(3), 664–672.

    Article  Google Scholar 

  • Plotz, T., & Roth, S. (2017). Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1586–1595).

  • Ren, J. S. J., Xu, L., Yan, Q., & Sun, W. (2015). Shepard convolutional neural networks. In NIPS (pp. 901–909).

  • Roth, S., & Black, M. J. (2009). Fields of experts. CVPR, 82(2), 205–229.

    Google Scholar 

  • Ruderman, D. L., & Bialek, W. (1993). Statistics of natural images: Scaling in the woods. In NIPS (pp. 551–558). Morgan Kaufmann.

  • Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. In Proceedings of the eleventh annual international conference of the center for nonlinear studies on experimental mathematics : Computational issues in nonlinear science: Computational issues in nonlinear science (pp. 259–268). New York, NY, USA: Elsevier North-Holland, Inc.

  • Sajjadi, M. S. M., Scholkopf, B., & Hirsch, M. (2017). Enhancenet: Single image super-resolution through automated texture synthesis. In The IEEE international conference on computer vision (ICCV).

  • Scherzer, O., & Groetsch, C. W. (2001). Inverse scale space theory for inverse problems. In Scale-space and morphology in computer vision, third international conference (pp. 317–325).

  • Shedligeri, P. A., Shah, K., Kumar, D., & Mitra, K. (2018). Photorealistic image reconstruction from hybrid intensity and event based sensor. In CoRR.

  • Shocher, A., Cohen, N., & Irani, M. (2018). “Zero-shot” super-resolution using deep internal learning. In CVPR: IEEE Computer Society.

    Book  Google Scholar 

  • Simoncelli, E. P., & Adelson, E. H. (1996). Noise removal via bayesian wavelet coring. In ICIP (1) (pp. 379–382). IEEE Computer Society.

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. In CoRR.

  • Tai, Y., Yang, J., & Liu, X. (2017). Image super-resolution via deep recursive residual network. In CVPR. IEEE Computer Society.

  • Turiel, A., Mato, G., Parga, N., & Nadal, J. (1997). Self-similarity properties of natural images. In NIPS (pp. 836–842). The MIT Press.

  • Turkowski, K. (1990). Filters for common resampling-tasks. In A. S. Glassner (Ed.), Graphics gems (pp. 147–165). Cambridge: Academic Press.

    Chapter  Google Scholar 

  • Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2018). Deep image prior. In CVPR. IEEE Computer Society.

  • Veen, D. V., Jalal, A., Price, E., Vishwanath, S., & Dimakis, A. G. (2018). Compressed sensing with deep image prior and learned regularization. In CoRR.

  • Zeiler, M. D., Krishnan, D., Taylor, G. W., & Fergus, R. (2010). Deconvolutional networks. In Proceedings of CVPR (pp. 2528–2535). IEEE Computer Society.

  • Zeyde, R., Elad, M., & Protter, M. (2010). On single image scale-up using sparse-representations. In J. D. Boissonnat, A. Chenin, P. Cohen, C. Gout, T. Lyche, M.-L. Mazure, & L. L. Schumaker (Eds.), Curves and Surfaces (Vol. 6920, pp. 711–730)., Lecture Notes in Computer Science Berlin: Springer.

    Chapter  Google Scholar 

  • Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In ICLR.

  • Zhu, S. C., & Mumford, D. (1997). Prior learning and gibbs reaction–diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(11), 1236–1250.

    Article  Google Scholar 

Download references

Acknowledgements

DU and VL are supported by the Ministry of Education and Science of the Russian Federation (Grant 14.756.31.0001) and AV is supported by ERC 638009-IDIU.

Author information

Authors and Affiliations

  1. Skolkovo Institute of Science and Technology, Moscow, Russia

    Dmitry Ulyanov & Victor Lempitsky

  2. Oxford University, Oxford, England

    Andrea Vedaldi

Authors
  1. Dmitry Ulyanov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrea Vedaldi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Victor Lempitsky
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dmitry Ulyanov.

Additional information

Communicated by Chen Change Loy.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ulyanov, D., Vedaldi, A. & Lempitsky, V. Deep Image Prior. Int J Comput Vis 128, 1867–1888 (2020). https://doi.org/10.1007/s11263-020-01303-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-020-01303-4

Keywords

Search

Navigation