Multi-operator image retargeting with automatic integration of direct and indirect seam carving☆
Highlights
► We propose a new multi-operator scheme by integrating direct and indirect seam carving. ► The proposed scheme remarkably reduces the cost of evaluating the similarity measures. ► We propose a novel seam carving enhancement, ACESC, to improve global structure preservation. ► With horizontal and vertical seam carving, we preserve the shapes of important objects well.
Introduction
In recent years, content-aware image resizing (a. k. a., retargeting) techniques, which can preserve visually important contents and maintain good perceptual invariance in an image when the size and aspect ratio are changed, have evoked a great deal of interests. Such techniques are especially meaningful for transforming images or video clips between multimedia devices with different resolutions.
The techniques can be coarsely divided into five major categories [7]: cropping methods, warping methods, seam carving methods, patch based methods and multi-operator retargeting methods. In cropping methods [8], [9], [10], [11], an optimal sub-window of the target size, which contains visually important regions, is searched from the input image. However, a disadvantage is that such methods may discard a large part of essential regions when the important objects are near the image periphery. Warping methods resize images non-homogeneously [3], [12], [13], [14], [15], [16]. By fixing a mesh on the image, they warp the designed mesh non-uniformly to reach the desired size based on a global optimization function. Patch based methods [17] achieve resizing by optimizing a patch-based similarity measure between the input and target image. One limitation of this method is its high computational cost, as pointed out in [6]. Seam carving (SC) methods [4], [18] perform image retargeting by iteratively adding or removing the most unimportant curves (i.e., seams) from the image. For images with large homogeneous region going across from its left to right (e.g., the sky region in Fig. 1 going across the image from left to right), a better seam carving scheme, which is called indirect seam carving, first resizes the height by inserting/deleting the horizontal seams in order to achieve the target aspect ratio, and then scales the intermediate image to the target size [14], [19]. The advantage of indirect seam carving is that the resizing is mostly done in the homogeneous region, resulting in less visual artifacts. Seam carving methods are especially suitable for images with sparsely distributed objects. It is prone to produce undesirable artifacts when resizing images with many prominent objects in them.
Multi-operator retargeting techniques [1], [19] adaptively utilize multiple operators including scaling, seam carving and cropping in turn to achieve content-aware image resizing. Those techniques employ an objective similarity measure to select the best combination from all possible combinations of different operators. A recent survey [6] indicates that compared with most state-of-the-art algorithms, multi-operator frameworks can produce retargeted images highly preferred by people. However, a major quality limitation is that when the resizing task is mono-directional, such techniques neglect the warping potential in the other dimension and are prone to distort the object shape (see Fig. 1 MultiOp, the people are “narrowed”). Another problem is that the computational cost of evaluating the image similarity measures is high and exponentially depends on the number of operators used [20].
Considering the aforementioned pros and cons, we propose a simple yet effective multi-operator resizing scheme to exploit both horizontal and vertical content-aware resizing potential, which has not been focused in previous multi-operator techniques [1], [19]. The scheme automatically and effectively combines four simple resizing operators. In particular, direct and indirect seam carvings are first performed, and then a similarity transformation is employed to scale the image to a suitable size, followed by cropping the image to the target size. Although we believe that taking the global warping as a basic operator will enhance the ability to distribute distortions in multiple directions, we choose seam carving for its simplicity. Furthermore, we propose a seam artifacts measure to reduce the computational cost of selecting effective changing points among performing different operators. Moreover, we design a novel seam energy scheme for seam carving, named ACcumulated Energy Seam Carving (ACESC), to relieve the seam distortion significantly and eliminate the expansion ratio limit of seam carving for enlarging. Compared with several state-of-the-art image retargeting methods, experiments in a benchmark image retargeting dataset indicate that the proposed multi-operator scheme has comparable performance and user preference in preserving the global structure and the shapes of important objects. Its computational cost is remarkably lower than that of the previous multi-operator methods.
The remainder of the paper is organized as follows. In Section 2, we survey related works. In Section 3, we introduce some preliminary knowledge on image retargeting. Section 4 details the proposed multi-operator scheme. Experiment results are presented and discussed in Section 5. We conclude the paper in Section 6.
Section snippets
Related work
In this section, we give a brief survey on the content-aware image and video retargeting algorithms.
Among cropping methods, [21], [22], [23], [24] constructed an optimal sub-window to preserve the prominent regions of the original image, which usually involves attention model extraction and object detection. With four psychological attributes (region of interest, attention value, minimal perceptible size, and minimal perceptible time), Liu et al. [8] employed an image attention model to extract
Preliminary
This section briefly summarizes the concepts and some details of original seam carving [18], and an image similarity measure, Bi-Directional Warping (BDW) [1], which are referred in the development of the proposed framework in the next section.
Our multi-operator framework
The frequently used notations in later discussions are shown in Table 1. We will re-clarify the notations when they are used.
In this section, we first give an overview of the framework in Section 1. Then we discuss the automatic procedure and efficiency issue in Section 2. Finally, we propose an improved seam carving in Section 3.
Results and discussions
In this section, we compare our proposed multi-operator scheme with several recently published algorithms including: 1) multi-operator (MultiOp) [1]; 2) streaming video (SV) [2]; 3) shift-maps (SM) [28]; 4) scale-and-stretch (SNS) [14]; 5) seam carving (SC) [4]; 6) nonhomogeneous warping (Warp) [3]; and 7) energy-based deformation (LG) [25]. We also include the results of 8) direct scaling operator (SCL), and 9) manually chosen cropping windows (CR) [6]. The comparison is performed in a
Conclusion
In this paper, we propose a new multi-operator framework that automatically combines direct seam carving, indirect seam carving, similarity transformation and cropping. Our contributions are three-fold: 1) we put forward a multi-operator retargeting framework towards automatic combination of direct and indirect seam carving; 2) we propose a measure to estimate the artifacts caused by seam carving and select the candidate set to speed up the existing multi-operator retargeting framework; 3)
Acknowledgment
We would like to thank Dr. Michael Rubinstein for valuable suggestions on the multi-operator retargeting algorithm. We would also like to thank the reviewers for their various comments, which helped improve this paper. This work was partially supported by the National Natural Science Foundation of China (NNSFC no. 60975044) and the National Fundamental Research Program of China (no. 2010CB327900).
References (34)
- M. Rubinstein, A. Shamir, S. Avidan, Multi-operator media retargeting, ACM Trans. Graphics. 28...
- et al.
A system for retargeting of streaming video
ACM Trans. Graph.
(2009) - et al.
Non-homogeneous content-driven video-retargeting
- M. Rubinstein, A. Shamir, S. Avidan, Improved seam carving for video retargeting, ACM Trans. Graphics. 27...
- et al.
Automatic image retargeting with fisheye-view warping
- M. Rubinstein, D. Gutierrez, O. Sorkine, A. Shamir, A comparative study of image retargeting, ACM Trans. Graphics. 29...
- et al.
A survey of image retargeting techniques
- et al.
Automatic browsing of large pictures on mobile devices
- et al.
Automatic thumbnail cropping and its effectiveness
- et al.
Gaze-based interaction for semi-automatic photo cropping
Self-adaptive image cropping for small displays
IEEE Trans. Consum. Electron.
Feature-aware texturing
Shrinkability maps for content-aware video resizing
Comput. Graphics Forum
Optimized scale-and-stretch for image resizing
ACM Trans. Graphics.
A shape-preserving approach to image resizing
Comput. Graphics Forum
Image retargeting using mesh parameterization
IEEE Trans. Multimedia
Summarizing visual data using bidirectional similarity
Cited by (23)
Image retargeting using depth assisted saliency map
2017, Signal Processing: Image CommunicationCitation Excerpt :Their algorithm uses edge detector, face detector and line detector to improve the energy function. To improve image resizing, some methods combine seam carving with other operators [9,10]. Rubinstein et al. [10] combined seam carving with cropping and scaling.
Shrink image by feature matrix decomposition
2014, NeurocomputingCitation Excerpt :Besides the redefined energy function, there are other ways to directly extend seam carving. For example, utilizing multi-operator [17] instead of single operator is a typical improvement. Rubinstein et al. [18] combined scaling, cropping, and seam carving together to resize images.
Content-Aware Image Resizing Technology Based on Composition Detection and Composition Rules
2023, Electronics (Switzerland)Multi-operator image retargeting with visual quality preservation of salient regions
2023, Multimedia SystemsMulti-operator image retargeting in compressed domain by preserving aspect ratio of important contents
2022, Multimedia Tools and Applications
- ☆
This paper has been recommended for acceptance by Michael Goesele.