Elsevier

Image and Vision Computing

Volume 30, Issue 9, September 2012, Pages 655-667
Image and Vision Computing

Multi-operator image retargeting with automatic integration of direct and indirect seam carving

https://doi.org/10.1016/j.imavis.2012.06.008Get rights and content

Abstract

Multi-operator image resizing can preserve important objects and structure in an image by combining multiple image resizing operators. However, traditional multi-operator methods do not take both horizontal and vertical content-aware resizing potential into consideration, which essentially leads to squeeze/stretch effect in the resultant images. In this paper, we propose a new multi-operator scheme that addresses aforementioned issue by integrating direct and indirect seam carving. Compared with previous methods, the proposed scheme remarkably reduces the cost of deciding when to change operators, by employing a newly defined image artifact measure. Furthermore, we propose a novel seam carving enhancement, named ACcumulated Energy Seam Carving (ACESC), as a basic operator to improve global structure preservation. By combining horizontal and vertical seam carving, our scheme preserves the shapes of important objects well. We present typical results to demonstrate the effectiveness of our method. User study shows that our method has high user preference.

Highlights

► We propose a new multi-operator scheme by integrating direct and indirect seam carving. ► The proposed scheme remarkably reduces the cost of evaluating the similarity measures. ► We propose a novel seam carving enhancement, ACESC, to improve global structure preservation. ► With horizontal and vertical seam carving, we preserve the shapes of important objects well.

Introduction

In recent years, content-aware image resizing (a. k. a., retargeting) techniques, which can preserve visually important contents and maintain good perceptual invariance in an image when the size and aspect ratio are changed, have evoked a great deal of interests. Such techniques are especially meaningful for transforming images or video clips between multimedia devices with different resolutions.

The techniques can be coarsely divided into five major categories [7]: cropping methods, warping methods, seam carving methods, patch based methods and multi-operator retargeting methods. In cropping methods [8], [9], [10], [11], an optimal sub-window of the target size, which contains visually important regions, is searched from the input image. However, a disadvantage is that such methods may discard a large part of essential regions when the important objects are near the image periphery. Warping methods resize images non-homogeneously [3], [12], [13], [14], [15], [16]. By fixing a mesh on the image, they warp the designed mesh non-uniformly to reach the desired size based on a global optimization function. Patch based methods [17] achieve resizing by optimizing a patch-based similarity measure between the input and target image. One limitation of this method is its high computational cost, as pointed out in [6]. Seam carving (SC) methods [4], [18] perform image retargeting by iteratively adding or removing the most unimportant curves (i.e., seams) from the image. For images with large homogeneous region going across from its left to right (e.g., the sky region in Fig. 1 going across the image from left to right), a better seam carving scheme, which is called indirect seam carving, first resizes the height by inserting/deleting the horizontal seams in order to achieve the target aspect ratio, and then scales the intermediate image to the target size [14], [19]. The advantage of indirect seam carving is that the resizing is mostly done in the homogeneous region, resulting in less visual artifacts. Seam carving methods are especially suitable for images with sparsely distributed objects. It is prone to produce undesirable artifacts when resizing images with many prominent objects in them.

Multi-operator retargeting techniques [1], [19] adaptively utilize multiple operators including scaling, seam carving and cropping in turn to achieve content-aware image resizing. Those techniques employ an objective similarity measure to select the best combination from all possible combinations of different operators. A recent survey [6] indicates that compared with most state-of-the-art algorithms, multi-operator frameworks can produce retargeted images highly preferred by people. However, a major quality limitation is that when the resizing task is mono-directional, such techniques neglect the warping potential in the other dimension and are prone to distort the object shape (see Fig. 1 MultiOp, the people are “narrowed”). Another problem is that the computational cost of evaluating the image similarity measures is high and exponentially depends on the number of operators used [20].

Considering the aforementioned pros and cons, we propose a simple yet effective multi-operator resizing scheme to exploit both horizontal and vertical content-aware resizing potential, which has not been focused in previous multi-operator techniques [1], [19]. The scheme automatically and effectively combines four simple resizing operators. In particular, direct and indirect seam carvings are first performed, and then a similarity transformation is employed to scale the image to a suitable size, followed by cropping the image to the target size. Although we believe that taking the global warping as a basic operator will enhance the ability to distribute distortions in multiple directions, we choose seam carving for its simplicity. Furthermore, we propose a seam artifacts measure to reduce the computational cost of selecting effective changing points among performing different operators. Moreover, we design a novel seam energy scheme for seam carving, named ACcumulated Energy Seam Carving (ACESC), to relieve the seam distortion significantly and eliminate the expansion ratio limit of seam carving for enlarging. Compared with several state-of-the-art image retargeting methods, experiments in a benchmark image retargeting dataset indicate that the proposed multi-operator scheme has comparable performance and user preference in preserving the global structure and the shapes of important objects. Its computational cost is remarkably lower than that of the previous multi-operator methods.

The remainder of the paper is organized as follows. In Section 2, we survey related works. In Section 3, we introduce some preliminary knowledge on image retargeting. Section 4 details the proposed multi-operator scheme. Experiment results are presented and discussed in Section 5. We conclude the paper in Section 6.

Section snippets

Related work

In this section, we give a brief survey on the content-aware image and video retargeting algorithms.

Among cropping methods, [21], [22], [23], [24] constructed an optimal sub-window to preserve the prominent regions of the original image, which usually involves attention model extraction and object detection. With four psychological attributes (region of interest, attention value, minimal perceptible size, and minimal perceptible time), Liu et al. [8] employed an image attention model to extract

Preliminary

This section briefly summarizes the concepts and some details of original seam carving [18], and an image similarity measure, Bi-Directional Warping (BDW) [1], which are referred in the development of the proposed framework in the next section.

Our multi-operator framework

The frequently used notations in later discussions are shown in Table 1. We will re-clarify the notations when they are used.

In this section, we first give an overview of the framework in Section 1. Then we discuss the automatic procedure and efficiency issue in Section 2. Finally, we propose an improved seam carving in Section 3.

Results and discussions

In this section, we compare our proposed multi-operator scheme with several recently published algorithms including: 1) multi-operator (MultiOp) [1]; 2) streaming video (SV) [2]; 3) shift-maps (SM) [28]; 4) scale-and-stretch (SNS) [14]; 5) seam carving (SC) [4]; 6) nonhomogeneous warping (Warp) [3]; and 7) energy-based deformation (LG) [25]. We also include the results of 8) direct scaling operator (SCL), and 9) manually chosen cropping windows (CR) [6]. The comparison is performed in a

Conclusion

In this paper, we propose a new multi-operator framework that automatically combines direct seam carving, indirect seam carving, similarity transformation and cropping. Our contributions are three-fold: 1) we put forward a multi-operator retargeting framework towards automatic combination of direct and indirect seam carving; 2) we propose a measure to estimate the artifacts caused by seam carving and select the candidate set to speed up the existing multi-operator retargeting framework; 3)

Acknowledgment

We would like to thank Dr. Michael Rubinstein for valuable suggestions on the multi-operator retargeting algorithm. We would also like to thank the reviewers for their various comments, which helped improve this paper. This work was partially supported by the National Natural Science Foundation of China (NNSFC no. 60975044) and the National Fundamental Research Program of China (no. 2010CB327900).

References (34)

  • M. Rubinstein, A. Shamir, S. Avidan, Multi-operator media retargeting, ACM Trans. Graphics. 28...
  • P. Krähenbühl et al.

    A system for retargeting of streaming video

    ACM Trans. Graph.

    (2009)
  • L. Wolf et al.

    Non-homogeneous content-driven video-retargeting

  • M. Rubinstein, A. Shamir, S. Avidan, Improved seam carving for video retargeting, ACM Trans. Graphics. 27...
  • F. Liu et al.

    Automatic image retargeting with fisheye-view warping

  • M. Rubinstein, D. Gutierrez, O. Sorkine, A. Shamir, A comparative study of image retargeting, ACM Trans. Graphics. 29...
  • V. Daniel et al.

    A survey of image retargeting techniques

  • H. Liu et al.

    Automatic browsing of large pictures on mobile devices

  • B. Suh et al.

    Automatic thumbnail cropping and its effectiveness

  • A. Santella et al.

    Gaze-based interaction for semi-automatic photo cropping

  • G. Ciocca et al.

    Self-adaptive image cropping for small displays

    IEEE Trans. Consum. Electron.

    (2007)
  • R. Gal et al.

    Feature-aware texturing

  • Y.F. Zhang et al.

    Shrinkability maps for content-aware video resizing

    Comput. Graphics Forum

    (2008)
  • Y.-S. Wang et al.

    Optimized scale-and-stretch for image resizing

    ACM Trans. Graphics.

    (2008)
  • G.-X. Zhang et al.

    A shape-preserving approach to image resizing

    Comput. Graphics Forum

    (2009)
  • Y. Guo et al.

    Image retargeting using mesh parameterization

    IEEE Trans. Multimedia

    (2009)
  • D. Simakov et al.

    Summarizing visual data using bidirectional similarity

  • Cited by (23)

    • Image retargeting using depth assisted saliency map

      2017, Signal Processing: Image Communication
      Citation Excerpt :

      Their algorithm uses edge detector, face detector and line detector to improve the energy function. To improve image resizing, some methods combine seam carving with other operators ‎[9,10]. Rubinstein et al. ‎[10] combined seam carving with cropping and scaling.

    • Shrink image by feature matrix decomposition

      2014, Neurocomputing
      Citation Excerpt :

      Besides the redefined energy function, there are other ways to directly extend seam carving. For example, utilizing multi-operator [17] instead of single operator is a typical improvement. Rubinstein et al. [18] combined scaling, cropping, and seam carving together to resize images.

    View all citing articles on Scopus

    This paper has been recommended for acceptance by Michael Goesele.

    View full text