Subjective quality evaluation of compressed digital compound images

Highlights

• We build a new Compound Image Quality Assessment Database (CIQAD).

• Subjective test is conducted to get the MOS values of images in CIQAD.

• Five compression methods are compared with respect to the obtained MOS values.

• We study the applicability of existing IQA metrics to evaluate quality of DCIs.

• Objective quality evaluation metric tailored for distorted DCIs is much desired.

Abstract

Visual quality evaluation of compressed Digital Compound Images (DCIs) becomes important in many multi-device communication systems. In this paper, we study subjective quality evaluation for compressed DCIs and investigate whether existing Image Quality Assessment (IQA) metrics are effective to evaluate the visual quality of compressed DCIs. A new Compound Image Quality Assessment Database (CIQAD) is therefore constructed, including 24 reference and 576 compressed DCIs. The subjective scores of these DCIs are obtained via visual judgement of 62 subjects using Paired Comparison (PC) in which the Hodgerank decomposition is adopted to generate uncompleted but near balanced pairs. Fourteen state-of-the-art IQA metrics are adopted to assess quality of images in CIQAD, and experimental results indicate that the existing IQA methods are limited in evaluating visual quality of DCIs. Compression results of five coding methods are thus compared with respect to different quality metrics to illustrate the limitation.

Introduction

Inspired by various Internet related applications, such as remote computing platforms [1], virtualized screen sharing systems [2], multi-client communication systems [3], and increasing visual contents are shared between different digital devices (i.e., computers, mobile phones, tablets, etc.). In these applications, visual contents (e.g., web pages, PowerPoint and PDF files, and computer screen images) are represented in the form of Digital Compound Images (DCIs). For efficient transmission among different devices, transmission systems should be deployed in a platform-free manners (without consideration of different protocols of different systems) and DCIs have to be highly compressed. For this requirement, cloud computing systems provide a strategy to share DCIs from the cloud to thin-clients; such systems render visual contents as DCIs in the cloud and then transmit them after compression to clients [4]. Many other computer generated compound images, such as promotional posters and microblogs, are also represented and stored in the form of DCIs.

Existing studies have demonstrated that traditional compression algorithms (such as JPEG, JPEG2000 and H.264 intra coding) cannot encode DCIs efficiently due to the high frequency components in textual regions [5], [6]. To address this problem, many studies [4], [5], [6], [7], [8] propose segmentation techniques to decompose DCIs into textual and pictorial parts, and encode these two different parts using different coding techniques. In these methods, the compression performance is generally evaluated by Peak Signal-to-Noise Ratio (PSNR), which is proved to be inconsistent with human visual perception [9]. The associated allocation of coding bits to textual and pictorial parts does not align with characteristics of Human Visual System (HVS). Generally, higher clarity of textual contents is required for better reading experience, with less degradation in pictorial regions.

Quality of Experience (QoE) has been investigated to evaluate users’ visual experience on web pages, so-called Web QoE [10]. Unfortunately, current Web QoE studies mainly focus on objective Quality of Service (QoS) metrics, e.g., loss ratio, round-trip times, full rendering time, etc. They rarely take into account the human perception for the webpage contents. For example, if texts and pictures are encoded and transmitted independently, given a certain overall loss ratio, different allocations to pictorial and textual parts may lead to quite different visual experiences.

Therefore, it is much desired to investigate the quality evaluation of compressed DCIs from the perspective of human visual perception. According to the results of visual quality evaluation, we can compare the compression performance of different algorithms for encoding DCIs. Moreover, visual quality evaluation for DCIs can be used to guide the processing (e.g., encoding, transmitting, rendering and enhancement) of DCIs. Currently, the visual quality evaluation of compressed DCIs has not yet been studied. Although various Image Quality Assessment (IQA) approaches have been proposed to evaluate quality of distorted natural images [11], [12], whether these IQA methods can be applied to evaluate the quality of compressed DCIs is still an open problem.

In this paper, we present an in-depth study on subjective quality assessment of DCIs with compression artifacts. A Compound Image Quality Assessment Database (CIQAD) is built, consisting of 24 reference images and 576 compressed versions. The reference images are obtained from Internet with various layouts of content. The distorted images are derived by five classical compression methods: JPEG, JPEG2000, H.264 intra coding, DjVu [13] and Layer Segmentation based Compression (LSC) [4]. We adopt the Pairwise Comparison (PC) method [14] to calculate Mean Opinion Score (MOS) values of these images towards better evaluation accuracy [15]. In particular, we employ the Hodgerank framework [16] to generate incomplete pairs in subjective test. This framework induces less effort and labor to human observers without significantly sacrificing evaluation accuracy and reliability. Based on the computed MOS values, 14 commonly used IQA approaches are employed to evaluate the quality of DCIs in CIQAD. To the best of our knowledge, this is the first attempt to study the quality assessment of DCIs. The CIQAD as well as the subjective scores are availabe to the research community for further quality assessment investigation of DCIs at https://sites.google.com/site/ciqadatabase/.

In the remaining of this paper, we first introduce the related work in Section 2. In Section 3, we present the detailed configuration of CIQAD. The subjective assessment methodology is described in Section 4. We report the experimental results in Section 5, and conclude the paper in the final section.