Visual saliency guided video compression algorithm

Highlights

• A two-stage novel video compression architecture is proposed for video compression.

• Machine learning scheme over three dimensional features is used for saliency computation.

• Saliency computation at macroblock level saves computation.

• Thresholding of mutual information between successive frames indicates the frames requiring re-computation of saliency.

• The motion vectors propagate the saliency values for macroblocks in P frames.

Abstract

Recently Saliency maps from input images are used to detect interesting regions in images/videos and focus on processing these salient regions. This paper introduces a novel, macroblock level visual saliency guided video compression algorithm. This is modelled as a 2 step process viz. salient region detection and frame foveation. Visual saliency is modelled as a combination of low level, as well as high level features which become important at the higher-level visual cortex. A relevance vector machine is trained over 3 dimensional feature vectors pertaining to global, local and rarity measures of conspicuity, to yield probabilistic values which form the saliency map. These saliency values are used for non-uniform bit-allocation over video frames. To achieve these goals, we also propose a novel video compression architecture, incorporating saliency, to save tremendous amount of computation. This architecture is based on thresholding of mutual information between successive frames for flagging frames requiring re-computation of saliency, and use of motion vectors for propagation of saliency values.

Introduction

The H.264 video compression standard has a wide range of applications from low bit-rate internet streaming applications to HDTV broadcast. The incipient HEVC seeks to further improve upon H.264, thereby achieving an even higher coding efficiency. This has been possible through improved exploitation of spatio-temporal prediction and entropy coding. The focus has however remained on minimization of the quantifiable ‘objective’ redundancy. In this paper we propose a video compression scheme which exploits features of human perception of video content for enhancing compression efficiency of H.264 based coding scheme.

It is now well established that the acuity of the human eye is limited to only 1–2° of visual angle [1]. This implies that when viewed from a recommended distance of 1.2 m, the eye can crisply perceive only a 2 cm radial region (computed as 1.2×tan(2°/2)) on a standard definition 32” LCD. Also, a recent eye-tracking study [2] on inter-observer saliency variations in task-free viewing of natural images has concluded that images known to have salient regions generate highly correlated eye-fixation maps for different viewers. These have reinforced the application of visual saliency to video compression among numerous other applications, like subjective visual quality assessment, progressive transmission, re-targeting for hand-held devices, effective website designs, and thumb nailing. Saliency based compression is achieved by encoding visually salient regions with high priority, while treating less interesting regions with low priority to save bits. In essence, it allows us to meet our goal of compression without significant degradation of viewing experience or ‘subjective’ quality.

Since 1998 a vast amount of research has gone into modelling of the human visual system (HVS) for the purpose of video processing. The earlier approaches were gaze contingent, dependant on eye trackers to record points of fixation. These were narrowly restricted to the case of a single viewer with eye-tracking apparatus, and hence not very useful. Later approaches exploited the computational neurobiological models to automatically predict the regions likely to attract human attention. However, each model came with its own merits and shortcomings, leaving salient region detection still a challenging and exciting area of research.

Itti et. al. [3] modelled visual attention as a combination of low level features pertaining to the degree of dissimilarity between a region and its surroundings. Novel center-surround approaches like [4] model saliency as the fraction of dissimilar pixels in concentric annular regions around each pixel. Hou and Zhang [5] take a completely different approach, suppressing the response to frequently occurring features while capturing deviances. Other transform domain approaches like [6], [7] follow a similar line of thought. Although these approaches work on psychological patterns with high accuracy, they often fail to detect salient objects in real life images. Some failure cases of these approaches are shown in Fig. 1. It is evident that these saliency maps are not quite close to ground truth.

The failure of these approaches can be attributed to Gestalt's grouping principle [8] which concerns the effect produced when the collective presence of a set of elements becomes more meaningful than their presence as separate elements. Thus, in this work we model saliency as a combination of low level, as well as high level features which become important at the higher-level visual cortex. Many authors like [9] resort to a linear combination of features such as contrast and skin color, but do not provide any explanation for the weights chosen. Hence, we propose a learning based feature integration algorithm where we train a Relevance Vector Machine (RVM) [10], [11] with 3 dimensional feature vectors to output probabilistic saliency values.

One of the earliest automated (as opposed to gaze contingent), visual saliency based video compression model was proposed by Itti [12] in 2004. In [12] a small number of virtual foveas attempt to track salient objects over video frames; and non-salient regions are Gaussian blurred to achieve compression. Guo and Zhang [6] use their PQFT approach for proto-object detection, and apply a multi-resolution wavelet domain foveation filter suppressing coefficients corresponding to background. The OPTOPOIHSH project which aimed at region of interest (ROI) based video compression to allow acceptable quality video transmission through low bandwidth channels, also employs some form of blurring [13]. Selective blurring can however lead to unpleasant artifacts and generally scores low on subjective evaluation. A novel bit allocation strategy through quantization parameter (QP) tuning, achieving compression while preserving visual quality, is presented in [14] which we adopt here.

A saliency preserving video compression scheme has been presented in [15], which reduces coding artifacts so that saliency of the region of interest is retained. In [16] a bit allocation strategy has been proposed which is based upon evaluation of perceptual distortion sensitivity of each macroblock. In [17] a video coding technique has been proposed which use visual saliency for adjusting image fidelity for compression. They use a saliency computation scheme different from the approach presented in this paper.

A simplified flow diagram of our compression model is shown in Fig. 2. In all the existing compression approaches, the saliency map is computed for each frame which can prove to be computationally very expensive. This is avoidable considering the temporal redundancy inherent in videos. We propose here a video coding architecture, incorporating visual saliency propagation, to save on a large amount of saliency computation, and hence time. This scheme uses thresholding of mutual information (MI) between successive frames for flagging frames which require re-computation of saliency; and use of motion vectors for carrying forward the saliency values.

The contribution of this paper to this field of study is thus twofold. First, a supervised procedure to compute saliency of an image using RVM over 3 dimensional feature vectors, pertaining to global, local and rarity measures of conspicuity is proposed. Second, a video coding architecture aimed at significant decrease in computation, and therefore time, is proposed. To arrive at this architecture, a novel saliency propagation and segmentation scheme based upon MI is implemented.

In this we have used H.264 encoder and decoder for establishing effectiveness of the strategy. However, the same scheme can be used with HEVC encoder and decoder. In HEVC, as in H.264, uniform reconstruction quantization (URQ) is used, with quantization scaling metrices supported for the various block sizes [18]. Accordingly, our QP tuning algorithm can be used with HEVC as well. However, flexibility in block size can permit grouping of MBs based upon saliency values.

The remainder of this paper is organized as follows. In Section 2 we discuss, in detail, the steps involved in our learning based saliency algorithm. Since all video coding operations are MB based, we learn saliency at MB level to save on unnecessary computation. This section also contains some results and comparison of our algorithm with other leading approaches. In Section 3 we describe our complete video coding architecture in which various issues relating to saliency propagation/re-calculation and bit allocation are addressed. Compression result on some varied video sequences and gain over standard H.264 with RDO is presented in Section 4 and conclusions are drawn in Section 5.