Wei Zhou
ABSTRACT
There has been a growing interest in developing image super-resolution (SR) algorithms that convert low-resolution (LR) to higher resolution images, but automatically evaluating the visual quality of super-resolved images remains a challenging problem. Here we look at the problem of SR image quality assessment (SR IQA) in a two-dimensional (2D) space of deterministic fidelity (DF) versus statistical fidelity (SF). This allows us to better understand the advantages and disadvantages of existing SR algorithms, which produce images at different clusters in the 2D space of (DF, SF). Specifically, we observe an interesting trend from more traditional SR algorithms that are typically inclined to optimize for DF while losing SF, to more recent generative adversarial network (GAN) based approaches that by contrast exhibit strong advantages in achieving high SF but sometimes appear weak at maintaining DF. Furthermore, we propose an uncertainty weighting scheme based on content-dependent sharpness and texture assessment that merges the two fidelity measures into an overall quality prediction named the Super Resolution Image Fidelity (SRIF) index, which demonstrates superior performance against state-of-the-art IQA models when tested on subject-rated datasets.
Supplemental Material
Available for Download
- Peter Burt and Edward Adelson. 1983. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications 31, 4 (1983), 532--540.Google Scholar
Cross Ref
- Hong Chang, Dit-Yan Yeung, and Yimin Xiong. 2004. Super-resolution through neighbor embedding. In CVPR. I--I.Google Scholar
- William T Freeman, Thouis R Jones, and Egon C Pasztor. 2002. Example-based super-resolution. IEEE Computer Graphics and Applications 22, 2 (2002), 56--65.Google ScholarDigital Library
- Jingcai Guo, Shiheng Ma, Jie Zhang, Qihua Zhou, and Song Guo. 2020. Dual-view attention networks for single image super-resolution. In Proceedings of the 28th ACM International Conference on Multimedia. 2728--2736.Google ScholarDigital Library
- Rania Hassen, Zhou Wang, and Magdy MA Salama. 2013. Image sharpness assessment based on local phase coherence. IEEE Transactions on Image Processing 22, 7 (2013), 2798--2810.Google ScholarCross Ref
- Mahdi S Hosseini, Yueyang Zhang, and Konstantinos N Plataniotis. 2019. Encoding visual sensitivity by maxpol convolution filters for image sharpness assessment. IEEE Transactions on Image Processing 28, 9 (2019), 4510--4525.Google ScholarCross Ref
- Robert Keys. 1981. Cubic convolution interpolation for digital image processing. IEEE Transactions on Acoustics, Speech, and Signal Processing 29, 6 (1981), 1153--1160.Google ScholarCross Ref
- Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In CVPR. 1646--1654.Google Scholar
- Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR. 624--632.Google Scholar
- Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR. 4681--4690.Google Scholar
- Anmin Liu, Weisi Lin, and Manish Narwaria. 2011. Image quality assessment based on gradient similarity. IEEE Transactions on Image Processing 21, 4 (2011), 1500--1512.Google Scholar
- Chao Ma, Chih-Yuan Yang, Xiaokang Yang, and Ming-Hsuan Yang. 2017. Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding 158 (2017), 1--16.Google ScholarDigital Library
- Kede Ma, Wentao Liu, Tongliang Liu, Zhou Wang, and Dacheng Tao. 2017. dipIQ: Blind image quality assessment by learning-to-rank discriminable image pairs. IEEE Transactions on Image Processing 26, 8 (2017), 3951--3964.Google ScholarDigital Library
- Kede Ma, Wentao Liu, Kai Zhang, Zhengfang Duanmu, Zhou Wang, and Wangmeng Zuo. 2017. End-to-end blind image quality assessment using deep neural networks. IEEE Transactions on Image Processing 27, 3 (2017), 1202--1213.Google ScholarCross Ref
- Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing 21, 12 (2012), 4695--4708.Google ScholarDigital Library
- Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. 2012. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters 20, 3 (2012), 209--212.Google ScholarCross Ref
- Anush Krishna Moorthy and Alan Conrad Bovik. 2011. Blind image quality assessment: From natural scene statistics to perceptual quality. IEEE Transactions on Image Processing 20, 12 (2011), 3350--3364.Google ScholarDigital Library
- Sung Cheol Park, Min Kyu Park, and Moon Gi Kang. 2003. Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 20, 3 (2003), 21--36.Google ScholarCross Ref
- Amy R Reibman, Robert M Bell, and Sharon Gray. 2006. Quality assessment for super-resolution image enhancement. In IEEE International Conference on Image Processing. 2017--2020.Google ScholarCross Ref
- Ann Marie Rohaly, Philip J Corriveau, John M Libert, Arthur A Webster, Vittorio Baroncini, John Beerends, Jean-Louis Blin, Laura Contin, Takahiro Hamada, David Harrison, et al . 2000. Video quality experts group: Current results and future directions. In Visual Communications and Image Processing, Vol. 4067. SPIE, 742--753.Google Scholar
- Michele A Saad, Alan C Bovik, and Christophe Charrier. 2012. Blind image quality assessment: A natural scene statistics approach in the DCT domain. IEEE Transactions on Image Processing 21, 8 (2012), 3339--3352.Google ScholarDigital Library
- Mehul P Sampat, Zhou Wang, Shalini Gupta, Alan Conrad Bovik, and Mia K Markey. 2009. Complex wavelet structural similarity: A new image similarity index. IEEE Transactions on Image Processing 18, 11 (2009), 2385--2401.Google ScholarDigital Library
- Hamid R Sheikh and Alan C Bovik. 2006. Image information and visual quality. IEEE Transactions on Image Processing 15, 2 (2006), 430--444.Google ScholarDigital Library
- Hamid R Sheikh, Alan C Bovik, and Gustavo De Veciana. 2005. An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Transactions on Image Processing 14, 12 (2005), 2117--2128.Google ScholarDigital Library
- Guangming Shi, Wenfei Wan, Jinjian Wu, Xuemei Xie, Weisheng Dong, and Hong Ren Wu. 2019. SISRSet: Single image super-resolution subjective evaluation test and objective quality assessment. Neurocomputing 360 (2019), 37--51.Google ScholarDigital Library
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- Wen Sun, Qingmin Liao, Jing-Hao Xue, and Fei Zhou. 2018. SPSIM: A superpixel-based similarity index for full-reference image quality assessment. IEEE Transactions on Image Processing 27, 9 (2018), 4232--4244.Google ScholarCross Ref
- Ying Tai, Jian Yang, and Xiaoming Liu. 2017. Image super-resolution via deep recursive residual network. In CVPR. 3147--3155.Google Scholar
- Cuong T Vu, Thien D Phan, and Damon M Chandler. 2011. S3: A spectral and spatial measure of local perceived sharpness in natural images. IEEE Transactions on Image Processing 21, 3 (2011), 934--945.Google ScholarDigital Library
- Guangcheng Wang, Leida Li, Qiaohong Li, Ke Gu, Zhaolin Lu, and Jiansheng Qian. 2017. Perceptual evaluation of single-image super-resolution reconstruction. In IEEE International Conference on Image Processing. 3145--3149.Google ScholarDigital Library
- Qing Wang and Rabab Kreidieh Ward. 2007. A new orientation-adaptive interpolation method. IEEE Transactions on Image Processing 16, 4 (2007), 889--900.Google ScholarDigital Library
- Shenlong Wang, Lei Zhang, Yan Liang, and Quan Pan. 2012. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In CVPR. 2216--2223.Google Scholar
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.Google ScholarDigital Library
- Zhihao Wang, Jian Chen, and Steven CH Hoi. 2020. Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 10 (2020), 3365--3387.Google ScholarDigital Library
- Zhou Wang and Qiang Li. 2010. Information content weighting for perceptual image quality assessment. IEEE Transactions on Image Processing 20, 5 (2010), 1185--1198.Google ScholarDigital Library
- Zhou Wang, Eero P Simoncelli, and Alan C Bovik. 2003. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, Vol. 2. IEEE, 1398--1402.Google ScholarCross Ref
- Jinjian Wu, Weisi Lin, Guangming Shi, and Anmin Liu. 2012. Perceptual quality metric with internal generative mechanism. IEEE Transactions on Image Processing 22, 1 (2012), 43--54.Google Scholar
- Qingbo Wu, Zhou Wang, and Hongliang Li. 2015. A highly efficient method for blind image quality assessment. In IEEE International Conference on Image Processing. 339--343.Google ScholarDigital Library
- Jun Xiao, Qian Ye, Rui Zhao, Kin-Man Lam, and Kao Wan. 2021. Self-feature Learning: An Efficient Deep Lightweight Network for Image Super-resolution. In Proceedings of the 29th ACM International Conference on Multimedia. 4408--4416.Google ScholarDigital Library
- Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alan C Bovik. 2013. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Transactions on Image Processing 23, 2 (2013), 684--695.Google ScholarDigital Library
- Jin Yamanaka, Shigesumi Kuwashima, and Takio Kurita. 2017. Fast and accurate image super resolution by deep CNN with skip connection and network in network. In International Conference on Neural Information Processing. Springer, 217--225.Google ScholarCross Ref
- Chih-Yuan Yang, Chao Ma, and Ming-Hsuan Yang. 2014. Single-image super-resolution: A benchmark. In ECCV. 372--386.Google Scholar
- Chih-Yuan Yang and Ming-Hsuan Yang. 2013. Fast direct super-resolution by simple functions. In ICCV. 561--568.Google Scholar
- Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. 2010. Image super-resolution via sparse representation. IEEE Transactions on Image Processing 19, 11 (2010), 2861--2873.Google ScholarDigital Library
- Wenming Yang, Yapeng Tian, Fei Zhou, Qingmin Liao, Hai Chen, and Chenglin Zheng. 2016. Consistent coding scheme for single-image super-resolution via independent dictionaries. IEEE Transactions on Multimedia 18, 3 (2016), 313--325.Google ScholarDigital Library
- Hojatollah Yeganeh, Mohammad Rostami, and Zhou Wang. 2015. Objective quality assessment of interpolated natural images. IEEE Transactions on Image Processing 24, 11 (2015), 4651--4663.Google ScholarDigital Library
- Zhenqiang Ying, Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, and Alan Bovik. 2020. From patches to pictures (PaQ-2-PiQ): Mapping the perceptual space of picture quality. In CVPR. 3575--3585.Google Scholar
- Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. 2011. FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing 20, 8 (2011), 2378--2386.Google ScholarDigital Library
- Fei Zhou, Rongguo Yao, Bozhi Liu, and Guoping Qiu. 2019. Visual quality assessment for super-resolved images: database and method. IEEE Transactions on Image Processing 28, 7 (2019), 3528--3541.Google ScholarDigital Library
- Wei Zhou, Zhou Wang, and Zhibo Chen. 2021. Image super-resolution quality assessment: Structural fidelity versus statistical naturalness. In IEEE International Conference on Quality of Multimedia Experience. 61--64.Google ScholarCross Ref
Index Terms
- Quality Assessment of Image Super-Resolution: Balancing Deterministic and Statistical Fidelity
Recommendations
Reduced-reference quality assessment of image super-resolution by energy change and texture variation
Highlights- In this paper, we propose a novel reduced-reference quality assessment metric for image super-resolution (RRIQA-SR) by two components: the energy change in ...
AbstractIn this paper, we propose a novel reduced-reference quality assessment metric for image super-resolution (RRIQA-SR) based on the low-resolution (LR) image information. With the pixel correspondence, we predict the perceptual similarity ...
Curvature preserving image super-resolution with gradient-consistency-anisotropic-regularization prior
Single image super-resolution (SR) often suffers from annoying interpolation artifacts such as blur, jagged edges, and ringing. In this paper, we aim to achieve artifact-free SR reconstruction from an input low resolution (LR) image using adaptive de-...
Image super-resolution by estimating the enhancement weight of self example and external missing patches
Image super-resolution (SR) is the process of generating a high-resolution (HR) image using one or more low-resolution (LR) inputs. Many SR methods have been proposed, but generating the small-scale structure of an SR image remains a challenging task. ...
Comments