Skip to main content
SpringerLink
Log in

Survey of Learning Based Single Image Super-Resolution Reconstruction Technology

  • MATHEMATICAL THEORY OF IMAGES AND SIGNALS REPRESENTING, PROCESSING, ANALYSIS, RECOGNITION, AND UNDERSTANDING
  • Published:
Pattern Recognition and Image Analysis Aims and scope Submit manuscript

Abstract

With the development of information technology, there is a high demand for high-resolution images. Image super-resolution reconstruction technology is to estimate a high-resolution image with better quality from one or a sequence of low-resolution images, with the help of signal processing technology. The core idea is to integrate useful information with strong correlations and complementarities from single image or multiple images as desired. Learning based single image super-resolution reconstruction technology is the current research hotspot. The paper systematically overviews this technology and discuss some main categories of it, such as super-resolution reconstruction based on neighbors, based on sparse representation, based on deep learning. At the end of the paper, challenge issues and future research directions for super-resolution image reconstruction are put forward.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.

Similar content being viewed by others

REFERENCES

  1. S. C. Park, K. P. Min, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag. 20 (3), 21–36 (2003).

    Article  Google Scholar 

  2. J. L. Harris, “Diffraction and resolving power,” J. Opt. Soc. Am. 57 (7), 931–936 (1964).

    Article  Google Scholar 

  3. S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag. 20 (3), 21–36 (2003).

    Article  Google Scholar 

  4. J. D. Ouwerkerk, “Image super-resolution survey,” Image Vision Comput. 24 (10), 1039–1052 (2006).

    Article  Google Scholar 

  5. S. Xiao, G. Han, and Y. Wo, “Survey of digital image super resolution reconstruction technology,” Comput. Sci. 36 (12), 8–13 (2009).

    Google Scholar 

  6. H. Shen, P. Li, L. Zhang, and Y. Wang, “Overview on super resolution image reconstruction,” Opt. Tech. 35 (2), 194–203 (2009).

    Google Scholar 

  7. K. Joshi, “SR image reconstruction methods—a survey,” J. Emerging Technol. Innovative Res. 2 (4), 997–1005 (2015).

    Google Scholar 

  8. G. Pandey and U. Ghanekar, “A compendious study of super-resolution techniques by single image,” Optik 166, 47–160 (2018).

    Article  Google Scholar 

  9. Z. Wang, J. Chen, and C. H. Hoi Steven, “Deep learning for image super-resolution: A survey,” IEEE Trans. Pattern Anal. Mach. Intell. (2020). arXiv:1902.06068 [cs.CV]

  10. S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Trans. Pattern Anal. Mach. Intell. 24 (9), 1167–1183 (2002).

    Article  Google Scholar 

  11. W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, “Learning low-level vision,” Int. J. Comput. Vision 40 (1), 25–47 (2000).

    Article  Google Scholar 

  12. W. T. Freeman, T. R. Jones, and E. C. Pasztor, “Example-based super-resolution,” IEEE Comput. Graph. Appl. 22 (2), 56–65 (2002).

    Article  Google Scholar 

  13. J. Sun, N. Zheng, H. Tao, and H. Shum, “Image hallucination with primal sketch priors,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Madison, 2003), pp. 729–736.

  14. H. Chang, D. Y. Yeung, and Y. Xiong, “Super-resolution through neighbor embedding,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Washington DC, 2004), pp. 275–282.

  15. S. T. Roweis and L. K. Saul, “Nonlinear dimensionality reduction by locally linear embedding,” Science 290 (5500), 2323–2326 (2000).

    Article  Google Scholar 

  16. B. Li, H. Chang, S. Shan, and X. Chen, “Locality preserving constraints for super-resolution with neighbor embedding,” in Proceedings of the IEEE International Conference on Image Processing (Cairo, 2009), pp. 1189–1192.

  17. K. Su, Q. Tian, Q. Xue, N. Sebe, and J. Ma, “Neighborhood issue in single-frame image super-resolution,” in Proceedings of the IEEE International Conference on Multimedia and Expo (Amsterdam, 2005).

  18. X. Liao, G. Han, Y. Wo, and X. Chen, “Super-resolution image reconstruction on stepwise magnification of neighbor embedding,” J. South China Univ. Technol. (Nat. Sci. Ed.) 41 (5) (2013).

  19. T. M. Chan and J. Zhang, “An improved super-resolution with manifold learning and histogram matching,” in Proceedings of the IAPR International Conference on Biometrics (Hong Kong, 2006), pp. 756–762.

  20. W. Fan, and D. Y. Yeung, “Image hallucination using neighbor embedding over visual primitive manifolds,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Minneapolis, 2007), pp. 1–7.

  21. K. Zhang, X. Gao, X. Li, and D. Tao, “Partially supervised neighbor embedding for example-based image super-resolution,” IEEE J. Sel. Top. Signal Process. 5 (2), 230–239 (2011).

    Article  Google Scholar 

  22. X. Gao, K. Zhang, D. Tao, and X. Li, “Image super-resolution with sparse neighbor embedding,” IEEE Trans. Image Process. 21 (7), 3194–3205 (2012).

    Article  MathSciNet  Google Scholar 

  23. Z. Zhang, C. Qi, and Y. Hao, “Locality preserving partial least squares for neighbor embedding-based face hallucination,” in Proceedings of the IEEE Conference on Image Processing (Phoenix, 2016), pp. 409–413.

  24. V. A. Rahiman and S.N. George, “Single image super-resolution using neighbor embedding and statistical prediction model,” Comput. Electr. Eng. 62, 281–292 (2017).

    Article  Google Scholar 

  25. T. Chan, J. Zhang, J. Pu, and H. Huang, “Neighbor embedding based super-resolution algorithm through edge detection and feature selection,” Pattern Recognit. Lett. 30 (5), 494–502 (2009).

    Article  Google Scholar 

  26. X. Liao, G. Han, Y. Wo, H. Huang, and Z. Li, “Super-resolution image reconstruction based on manifold learning and gradient constraint,” J. South China Univ. Technol. (Nat. Sci. Ed.) 40 (4), 8–15 (2012).

  27. J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution as sparse representation of raw image patches,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Anchorage, 2008), pp. 1–8.

  28. E. J. Candes and M. Wakin. “An introduction to compressive sampling,” IEEE Signal Process. Mag. 25 (2), 21–30 (2008).

    Article  Google Scholar 

  29. J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process. 19 (11), 2861–2873 (2010).

    Article  MathSciNet  Google Scholar 

  30. J. Yang, Z. Wang, Z. Lin, S. Cohen, and T. Huang, “Couple dictionary training for image super-resolution,” IEEE Trans. Image Process. 21 (8), 3467– 3478 (2012).

    Article  MathSciNet  Google Scholar 

  31. R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in Proceedings of the International Conferences on Curves and Surfaces (Avignon, 2010), pp. 711–730.

  32. S. Yang, M. Wang, Y. Sun, F. Sun, and L. Jiao, “Compressive sampling based single-image super-resolution reconstruction by dual-sparsity and non-local similarity regularizer,” Pattern Recognit. Lett. 33 (9), 1049–1059 (2012)

    Article  Google Scholar 

  33. X. Lu, H. Yuan, P. Yan, Y. Yuan, and X. Li, “Geometry constrained sparse coding for single image super-resolution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Providence, 2012), pp. 1648–1655.

  34. R. Timofte, V. D. Smet, and L. V. Gool, “Anchored neighborhood regression for fast example-based super-resolution,” in Proceedings of the IEEE International Conference on Computer Vision (Sydney, 2013), pp. 1920–1927.

  35. R. Timofte, V. D. Smet, and L. V. Gool, “A+: Adjusted anchored neighborhood regression for fast super-resolution,” in Proceedings of the IEEE Asian Conference on Computer Vision (Singapore, 2014), pp. 111–126.

  36. D. Dai, R. Timofte, and L. V. Gool, “Jointly optimized regressors for image super-resolution,” Comput. Graphics Forum 34 (2), 95–104 (2015).

    Article  Google Scholar 

  37. E. Agustsson, R. Timofte, and L. V. Gool, “Regressor basis learning for anchored super-resolution,” in Proceedings of the International Conference on Pattern Recognition (Cancun, 2016), pp. 3850–3855.

  38. Z. Pan, J. Yu, C. Xiao, and W. Sun, “Dictionary learning and structural self-similarity-based codebook mapping for single image super resolution,” J. Comput.-Aided Des. Comput. Graphics 27 (6), 1032–1038 (2015).

    Google Scholar 

  39. S. Yang and R. Zhao, “Image super-resolution based on low-rank matrix and dictionary learning,” J. Comput. Res. Dev. 53 (4), 884–891 (2016).

    Google Scholar 

  40. M. Protter, I. Yavneh, and M. Elad, “Closed-form MMSE estimation for signal denoising under sparse representation modeling over a unitary dictionary,” IEEE Trans. Signal Process. 58 (7), 3471–3484 (2010).

    Article  MathSciNet  Google Scholar 

  41. S. Mallat and G. Yu, “Super-resolution with sparse mixing estimators,” IEEE Trans. Image Process. 19 (11), 2889–2900 (2010).

    Article  MathSciNet  Google Scholar 

  42. S. Yang, Z. Liu, M. Wang, F. Sun, and L. Jiao, “Multitask dictionary learning and sparse representation based single-image super-resolution reconstruction,” Neurocomputing 74 (17), 3193–3203 (2011).

    Article  Google Scholar 

  43. S. Yang, M. Wang, Y. Chen Y, and Y. Sun, “Single-image super-resolution reconstruction via learned geometric dictionaries and clustered sparse coding,” IEEE Trans. Image Process. 21 (9), 4016–4028 (2012).

    Article  MathSciNet  Google Scholar 

  44. Z. Pan, J. Yu, C. Xiao, and W. Sun, “Single image super resolution based on adaptive multi-dictionary learning,” Acta Electron. Sin. 43 (2), 209–216 (2015).

    Google Scholar 

  45. W. Dong, L. Zhang, G. Shi, and X. Wu, “Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization,” IEEE Trans. Image Process. 20 (7), 1838–1857 (2011).

    Article  MathSciNet  Google Scholar 

  46. W. Dong, L. Zhang, R. Lukac, and G. Shi, “Sparse representation based image interpolation with nonlocal autoregressive modeling,” IEEE Trans. Image Process. 22 (4), 1382–1394 (2013).

    Article  MathSciNet  Google Scholar 

  47. S. Zhan, Q. Fang, F. Yang, L. Chang, and T. Yan, “Image super-resolution reconstruction via improved dictionary learning based on coupled feature space,” Acta Electron. Sin. 44 (5), 1189–1195 (2016).

    Google Scholar 

  48. X. Liao, K. Bai, Q. Zhang, X. Jia, S. Liu, and J. Zhan, “Image super-resolution based on sparse coding with multi-class dictionaries,” Comput. Inf. 38 (6), 1301–1319 (2019).

    Google Scholar 

  49. C. Dong, C. C. Loy, K. He, and Tang, X., “Learning a deep convolutional network for image super-resolution,” in Proceedings of the European Conference on Computer Vision (Zurich, 2014), pp. 184–199.

  50. C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38 (2), 295–307 (2016).

    Article  Google Scholar 

  51. C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in Proceedings of the European Conference on Computer Vision (Amsterdam, 2016), pp. 391–407.

  52. J. Bruna, P. Sprechmann, and Y. LeCun, “Super-resolution with deep convolutional sufficient statistics,” in Proceedings of the International Conference on Learning Representations (Caribe, 2016), pp. 1–17.

  53. A. Dosovitskiy and T. Brox, “Generating images with perceptual similarity metrics based on deep networks,” in Proceedings of the Advances in Neural Information Processing Systems (Barcelona, 2016), pp. 658–666.

  54. J. Kim, J. K. Lee, and K. M. Lee, “Deeply-recursive convolutional network for image super-resolution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Las Vegas, 2016), pp. 1637–1645.

  55. W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Las Vegas, 2016), pp. 1874–1883.

  56. C. Ledig, L. Theis, F. Huszar, et al. “Photo-realistic single image super-resolution using a generative adversarial network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Hawaii, 2017), pp. 4681–4690.

  57. A. Bulat, J. Yang, and G. Tzimiropoulos, “To learn image super-resolution, use a GAN to learn how to do image degradation first,” in Proceedings of the European Conference on Computer Vision (ECCV) (Munich, 2018), pp. 185–200.

  58. S. Bell-Kligler, A. Schocher, and M. Irani, “Blind super-resolution kernel estimation using an internal-GAN,” in Proceedings of Advances in Neural Information Processing Systems (Vancouver, 2019), pp. 284–293.

Download references

Funding

This work was supported by Young Creative Talents Project of Department of Education of Guangdong Province (Natural Science) (2016KQNCX092, 2017KQNCX117, 2016KQNCX089), National Natural Science Foundation of China (61772144, 61702119), Foreign Science and Technology Cooperation Plan Project of Guangzhou Science Technology and Innovation Commission (201807010059), Natural Science Foundation of Guangdong (2016A030313472, 2018A030313994, 2018A0303130187), Science and Technology Program of Guangzhou (201607010152).

Author information

Authors and Affiliations

  1. School of Computer Science, Guangdong Polytechnic Normal University, Guangzhou, China

    K. Bai, X. Liao, Q. Zhang, X. Jia & S. Liu

Authors
  1. K. Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. X. Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Q. Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. X. Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. S. Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to K. Bai, X. Liao, Q. Zhang, X. Jia or S. Liu.

Ethics declarations

The authors declare no conflict of interest neither in financial nor in any other area.

Additional information

Kejia Bai. Kejia Bai was born in Guiyang, HuNan, China, in 1974. He is an associate professor in the School of Computer Science, Guangdong Polytechnic Normal University, Guangzhou, China. He received PhD degree in System Engineering from South China University of Technology in 2013. His research interests include image processing, pattern recognition, artificial intelligence, super-resolution, and target tracking.

Xiuxiu Liao. Xiuxiu Liao was born in Longhui, HuNan, China, in 1983. She is a lecturer in the School of Computer Science, Guangdong Polytechnic Normal University, Guangzhou, China. She received her PhD degree in Computer Application Technology from South China University of Technology in 2013. Her research interests include image processing, compressive sensing, and machine learning.

Qian Zhang. Qian Zhang was born in Zibo, ShanDong, China, in 1982. She is a lecturer in the School of Computer Science, Guangdong Polytechnic Normal University, Guangzhou, China. She received her PhD degree in Computer Application Technology from South China University of Technology in 2013. Her research interests include image processing and deep learning.

Xiping Jia. Xiping Jia was born in Lantian, Shanxi, China, in 1976. He is an associate professor in the School of Computer Science, Guangdong Polytechnic Normal University, Guangzhou, China. He received his PhD degree in Computer Application Technology from South China University of Technology in 2008. His research interests include image processing, data mining, and machine learning.

Shaopeng Liu. Shaopeng Liu was born in 1984. He received the Ph.D. degree from Sun Yat-Sen University, Guangdong, China, in 2013. Currently, he is a lecturer in the School of Computer Science, Guangdong Polytechnic Normal University, China. His research interests include machine learning and medical image analysis.

X. Liao is corresponding author

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bai, K., Liao, X., Zhang, Q. et al. Survey of Learning Based Single Image Super-Resolution Reconstruction Technology. Pattern Recognit. Image Anal. 30, 567–577 (2020). https://doi.org/10.1134/S1054661820040045

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S1054661820040045

Keywords:

Search

Navigation