Image retargeting quality assessment based on content deformation measurement

https://doi.org/10.1016/j.image.2018.06.010Get rights and content

Highlights

  • First time, introduce the flexible registration measurement into the field of retargeting image quality evaluation.

  • Propose the foreground retention ratio against the preservation degree of main area.

  • Propose the concentration deletion ratio against extreme cases of content loss.

  • Optimize the ARS standard parametric equation by visual characteristics.

Abstract

Content-aware based image retargeting methods aim at visualizing source images on diverse display devices with different aspect ratios while preventing the incurrence of visual distortions as far as possible. An objective retargeting assessment method, which has a high consistency with subjective perception, can measure the retargeting effects of different retargeting methods, thus promoting the development of retargeting technology. However, the correlation between the existing objective evaluation methods and subjective evaluation is not strong enough. In this paper, we propose an objective retargeting assessment method by fusing multiple features to quantify the content loss and structural deformation caused by retargeting. In order to measure the global deformation, there are two features are proposed which are elastic registration based on B-spline function and foreground retention ratio. Moreover, improved Aspect Ratio Similarity (IARS) and concentrated deletion ratio make for better detection of local deformation. These features are fused to obtain the objective retargeting assessment method. Compared with the state-of-art objective assessment methods on three public datasets, the proposed method has a better correlation with subjective perception.

Introduction

Nowadays, with the rapid development of the Internet and mobile terminals, most visual media are transmitted through electronic networks, making it necessary for images to be scaled to different resolutions to meet the requirements of diverse devices. Under these circumstances, it is important to maintain image retargeting quality. Subjective quality assessment is the most accurate method of determining the retargeted image quality, but it is time-consuming and impractical in most cases. Therefore, the quality assessments on retargeted images are mainly based on objective evaluation [1].

In practical applications, images with non-uniform retargeting result in a loss of image content and structural distortion. Different retargeting methods [2] may cause different types of content loss and structural deformation [3]. The discrete retargeting methods treat images as pixels [4], so the structural deformation is mainly due to the retention or discarding of content, which produces the phenomenon of line breaks, saw-tooth edges, and changes in aspect ratio (e.g., Seam Carving (SC) [5], Cropping (CR) and Shift-Map (SM)) [6]. The continuous retargeting methods usually add a deformed mesh and then resize the mesh under constraint equations, which leads to the image being scaled or stretched, and to geometrical deformation (e.g., Scaling (SCL), Scale-and-Stretch (SNS) [7], Streaming Video (SV) [8] and non-homogeneous Warping (WARP) [9]). To evaluate the retargeted image quality, we usually use objective or subjective methods to measure the artifacts caused by retargeting. Considering that there are many kinds of artificial traces, objective evaluations mainly take the subjective perception of images as the final goal. Designing Image Retargeted Quality Assessment (IRQA) standards for retargeted images has always been a challenge in this field.

In recent years, there has been substantial progress in image quality assessment methods [10], however IRQA is still in its infancy. The reason is that traditional quality assessment criteria are only effective for operations on the images of the same size, but not for the retargeted images [11]. Moreover, the traditional image quality assessment criteria cannot measure the structural deformation or content loss in retargeted images [12].

The research on objective IRQA algorithms is no more than ten years old. Earth Mover’s Distance (EMD) [13] is a classic IRQA method, which uses the histograms of image features to represent the main contents of the image, and calculates the distance between two feature matrices before and after retargeting. However, the structural deformation of the retargeted image cannot be effectively identified, because the distribution of histograms cannot describe the local details of the image.

Currently, most IRQA methods use the image matching algorithm to find the correspondence between the original and retargeted images. Liu et al. [14] used a SIFT flow algorithm to find the matching features between the images and then calculated the local structural similarity for matching SIFT features. Note that the SIFT matching algorithm also has some shortcomings, such as fewer matching points and less uniform distribution, which will affect the final assessment to a certain extent. Hsu et al. [15] suggested that the evaluation criteria should take into account both the global structural changes and the local content loss. They defined the local variance between two SIFT flow vector fields to measure the geometric distortion of the retargeted image, but the SIFT flow vector field is insensitive to the smooth part of the image, and so this greatly affects the subsequent measurement.

To better describe the visual differences between the original image and the retargeted image, researchers introduced image descriptors to measure the image distortion degree. For the first time, Karimi et al. [16] used an optical flow descriptor to measure the difference between two images by directly resizing the retargeted image to the original image size and then calculating the optical flow difference of image blocks between the two images. Aspect Ratio Similarity (ARS) criteria [17] offer innovation by extracting the image information through backward registration to predict the image changes and then adjusts the corresponding aspect ratio of the matching image blocks before and after retargeting. ARS does not detect the overall structural change of the image, thus its evaluation is not comprehensive.

To make the predicted results consistent with the human visual aesthetic, researchers added some physiological and psychological characteristics of Human Visual System (HVS) to IRQA [18]. Based on a Structural Similarity Index Measure (SSIM), Image Retargeting SSIM (IR-SSIM) [19] introduces the visual saliency map and face detection but it did not include other artifact detections, so it still gives a low correlation between objective evaluation results and subjective perception. Liang et al. [20] proposed an image quality assessment method by fusing symmetry detection and image aesthetics criterion. In this way, the correlation of objective assessment with subjective perception can be improved. Nevertheless, the definition and accuracy of symmetry detection remains a challenge that limits the general application of this criterion. By the analysis above, the state-of-the-art objective assessment methods still have a low correlation with the subjective perception.

Recently researchers have developed effective features for retargeting image correspondence estimation. In order to improve the alignment between the original image and the retargeted image, Zhang [21] proposed a three-level representation of the retargeting process and combined both fidelity measures and inconsistency detection to evaluate the retargeting quality. But the same amount of shape distortion or discontinuities in three-level representation may have disparate impact on the retargeting quality. Furthermore, Zhang [22] proposed a multiple-level feature (MLF) which includes a low-level aspect ratio similarity, a mid-level edge group similarity feature, and a high-level face block similarity feature to address the shape/structure related to distortion, to predict the retargeted image quality. These three-level features are complementary in being able to measure different qualities of degradation. For the high-level IRQA task in MLF, the face block is only used, and there still needs to be high-level representations closer to subjective perception. Jiang et al. [23] proposed a sparse representation by applying two overcomplete dictionaries for objective image retargeting quality assessment, which used the intrinsic discriminative power of sparse representation for similarity measurement. This representation does reduce the time complexity of the evaluation, but in the process of training the dictionary, the accuracy of the complete dictionary has a decisive influence on the subsequent evaluation. Fu et al. [24] measured texture and semantic similarity from the deep-learned features extracted by the CNN model, and simultaneously used hand-crafted features and deep-learned features to estimate the perceptual quality degradation.

In order to improve the objective assessment for IRQA further, this paper estimates the distortion of retargeted images from the structural deformation of the image, such as linear curve as well as the excess loss of significant content, which are the main factors that influence subjective perception. The IRQA method proposed in this paper can measure both the global structural deformation and local content loss. The main contributions are as follows:

(1) A new global deformation method is introduced by using elastic registration based on B-spline function, which can overcome the limitations of image sizes, detect the deformation mapping through elastic registration, and measure the global image deformation by calculating the amount of deformation in the whole deformation space;

(2) For an image with distinct foreground content and background, the overall foreground retention ratio is calculated from the global image using Robust Background Detection (RBD) saliency algorithm.

(3) The improved Aspect Ratio Similarity (IARS) is proposed, which can better measure the change in local aspect ratio and have better relevance to subjective evaluation;

(4) The excessive concentrated deletion ratio is defined to measure the amount of lost content of the large area in the local image, which mostly occurs in images retargeted by discrete retargeting methods;

The rest of this paper is organized as follows: Section 2 introduces the theoretical method of measuring structural deformation and content loss. In Section 3, the objective image retargeting quality assessment method proposed in this paper is verified and compared with other representative algorithms on three public datasets. Section 4 sets out conclusions together with suggestions on some future work.

Section snippets

Content-based image retargeting deformation measurementmethod

The objective quality assessment of retargeted images is a high-level semantic task [17]. Image retargeting aims at preserving the relative important image content as much as possible, while avoiding the introduction of visual distortion. The correlation measurement between the original image and the retargeted image is extremely complex due to different kind of artificial deformation introduced in the retargeting process. In this paper, we reveal that the structural deformation and content

Experiments and discussions

To demonstrate the effectiveness of the proposed objective metric, experiments are performed on three public databases: MIT RetargetMe [29] and NRID [30] and CUHK [31]. The weights in Eq. (14) are training by MIT RetargetMe database because this database includes more kind of deformation than NRID and CUHK, which are ωi=0.523, ωb=0.314, ωc=0.225, ω=0.688, and ωf=0.106. When an image has no distinct foreground and background, ωf is set to 0. The Kendall Rank Coefficient (KRCC) is taken as the

Conclusion and future work

This paper proposes an effective evaluation method for the objective quality of retargeted images. This method fuses the global deformation measurement, foreground retention ratio, IARS and the concentrated deletion ratio to measure the deformation of content losses and structure deformation. As an objective evaluation metric of retargeted image quality, the proposed method has good correlation with subjective scores and it is effective and accurate. The hierarchical matching mechanism for

Acknowledgments

This work was granted by Tianjin Sci-tech Planning Projects (Grant No. 14RCGFGX00846), the Natural Science Foundation of Hebei Province, China (Grant No. F2015202239), Tianjin Sci-tech Planning Projects (Grant No. 15ZCZDNC00130) and Tianjin Sci-tech Planning Projects (Grant No. 17ZLZDZF00040).

References (37)

  • LiL. et al.

    Color image quality assessment based on sparse representation and reconstruction residual

    J. Vis. Commun. Image Represent.

    (2016)
  • KarimiM. et al.

    Quality assessment of retargeted images by salient region deformity analysis

    J. Vis. Commun. Image Represent.

    (2017)
  • MaL. et al.

    Retargeted Image Quality Assessment: Current Progress and Future Trends, Visual Signal Quality Assessment

    (2015)
  • ZhuL. et al.

    Fast genetic multi-operator image retargeting

  • LiangY. et al.

    Objective quality prediction of image retargeting algorithms

    IEEE Trans. Visual Comput. Graphics

    (2017)
  • FangY. et al.

    Optimized multioperator image retargeting based on perceptual similarity measure

    IEEE Trans. Syst. Man Cybern. Syst.

    (2017)
  • AvidanS. et al.

    Seam carving for content-aware image resizing

    ACM Trans. Graph.

    (2007)
  • PritchY. et al.

    Shift-map image editing

  • WangY.S. et al.

    Optimized scale-and-stretch for image resizing

    ACM Trans. Graph.

    (2008)
  • LangM. et al.

    A system for retargeting of streaming video

  • WolfL. et al.

    Non-homogeneous Content-driven Video-retargeting

  • GaoH. et al.

    A novel unsupervised segmentation quality evaluation method for remote sensing images

    Sensors

    (2017)
  • FuY. et al.

    No reference image quality assessment metric based on visual perception

    Algorithms

    (2016)
  • PeleO. et al.

    Fast and robust Earth Mover’s Distances

  • C. Liu, J. Yuen, A. Torralba, et al., SIFT Flow: Dense Correspondence across Different Scenes, Vol. 5304, 2008, pp....
  • HsuC.C. et al.

    Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss

    IEEE J. Sel. Top. Sign. Proces.

    (2014)
  • ZhangY. et al.

    Backward registration based aspect ratio similarity (ARS) for image retargeting quality assessment

    IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc.

    (2016)
  • LiuY.J. et al.

    Image retargeting quality assessment

    Comput. Graph. Forum

    (2011)
  • Cited by (16)

    • Image retargeting quality assessment based on saliency-driven classification

      2022, Signal Processing: Image Communication
      Citation Excerpt :

      We employ RetargetMe [31] and CUHK [39] databases to evaluate the performance of the proposed methods. These two databases contain enough testing images and provide a reliable benchmark for assessing the performance of IRQA algorithms [13–15,20,40–43]. The RetargetMe database provides 37 original images, which has been resized by eight typical retargeting algorithms, including CR, SV [11], MOP [12], SC [4], SCL, SM [29], SNS [30], and WARP [6].

    • Conditional Random Field based salient proposal set generation and its application in content aware seam carving

      2020, Signal Processing: Image Communication
      Citation Excerpt :

      In this paper, we evaluate a generic object proposal generation by utilizing CRF based edge classification learning framework per se, and have abstained from any form of additional supervision such as off-the-shelf saliency measures (regional saliency map computation), negative data, and pretrained features. We rank the proposals in their relative order of salience thus making the scheme suitable for applications such as adaptive compression, seam carving [13–15], scene understanding, video summarization etc. Most of the object proposal generation methods either resort to objectness cues or similarity based aggregation resulting in coherent regions.

    • NetNCSP: Nonoverlapping closed sequential pattern mining

      2020, Knowledge-Based Systems
      Citation Excerpt :

      It has been widely applied in various fields, such as big data mining [4,5], big data intelligence [6], e-commerce shopping analysis [7], biological sequence analysis [8], and event analysis [9]. To handle some specific issues, many methods have been proposed, such as negative SPM [10,11], maximal frequent pattern mining [12,13], three-way pattern mining [14,15], closed SPM [16,17], gap constraint SPM [18,19]. Our previous work proposed an effective algorithm NOSEP and reported that the nonoverlapping SPM has better performance than other state-of-the-art gap constraint SPM methods in finding useful patterns in biology sequences and avoiding under-expression and over-expression in time series [34].

    • Image retargeting quality assessment: A survey

      2023, Journal of Intelligent and Fuzzy Systems
    View all citing articles on Scopus
    View full text