Collaborative Personalization of Image Enhancement

Abstract

This paper presents methods for personalization of image enhancement, which could be deployed in photo editing software and also in cloud-based image sharing services. We observe that users do have different preferences for enhancing images and that there are groups of people that share similarities in preferences. Our goal is to predict enhancements for novel images belonging to a particular user based on her specific taste, to facilitate the retouching process on large image collections. To that end, we describe an enhancement framework that can learn user preferences in an individual or collaborative way. The proposed system is based on a novel interactive application that allows to collect user’s enhancement preferences. We propose algorithms to predict personalized enhancements by learning a preference model from the provided information. Furthermore, the algorithm improves prediction performance as more enhancement examples are progressively added. We conducted experiments via Amazon Mechanical Turk to collect preferences from a large group of people. Results show that the proposed framework can suggest image enhancements more targeted to individual users than commercial tools with global auto-enhancement functionalities.

Appendix: Detailed Enhancement Operations

We approximate the processes of linearization and reversion to original nonlinear space with gamma curves with parameter $\gamma = 2.2$ and $\gamma ^{-1} = 0.455$, respectively. RGB is linearized using $c_\mathrm{linear} = c^\gamma$, where $c = R, G, B$ (normalized). The linearized RGB values are then color corrected and contrast enhanced, and finally “unlinearized” by applying the inverse operation.

1.1 Contrast Curve Specification

To manipulate the contrast, we use the power and S-curves.

Power curve: $\tau$. This is equivalent to the gamma curve (but note that is kept separate from the gamma curve, as seen in Fig.  2): $y = x^\tau$, with $x$ and $y$ being the normalized input and output intensities.

S-curve: $\lambda , a$. The S-curve is also commonly used to globally modify contrast. The formula we used to specify the S-curve is

$$\begin{aligned} y = \left\{ \begin{array}{ll} a - a \left( 1 - \frac{x}{a} \right) ^\lambda &{} \text{ if } x \le a \\ a + (1-a) \left( \frac{x-a}{1-a} \right) ^\lambda &{} \text{ otherwise } \end{array} \right. , \end{aligned}$$

(5)

with $x$ and $y$ being the normalized input and output intensities (see Fig.  15a).

Fig. 15

Two examples of contrast curves. a S-curve to specify global contrast change. b Parameterized curve to specify shadow, mid-tone, and highlight regions. In our enhancement pipeline, we use (a)

1.2 Color Correction: Temperature $T$ and Tint $h$

We color correct based on color temperature $T$ and tint $h$, rather than applying a $3 \times 3$ diagonal matrix⁶. The notion of temperature and tint is a more natural parameterization from the photographer’s perspective. Also, we deal with only two instead of three parameters. (One can “normalize” the matrix so that the resulting luminance is unchanged, yielding two independent parameters, but these numbers are perceptually less meaningful.)

The color temperature is determined by comparing its chromaticity with that of an ideal black-body radiator. In practice, however, color temperature is relative to a standard, usually D65 standard, which is equivalent to 6,500 K. The temperature curve is along the blue–yellow line. Unintuitive as it may seem, higher color temperatures (5,000 K or more) are “cool” colors (green–blue), while lower are “warm” colors (yellow–red). Tint, however, is orthogonal to color temperature, and controls change along the green-magenta axis.

1.3 Auto-Correction as Preprocess Step

Our version of auto-correction consists of auto-white balance followed by auto-contrast stretch.

Auto-white balance. We make the gray-world assumption for the brightest 5 % of the pixels (Buchsbaum 1980). The gray-world assumption is that the average color of all pixels in the image, i.e., the white point, is gray (R=G=B) with equal luminance as the average color. A matrix M is computed such that it transforms the average color to gray. This matrix M is then applied to all the pixels in the image. We found that empirically, it is more robust to use a certain percentage of the brightest pixels that are not saturated to compute the white point. More sophisticated techniques (such as Hsu et al. (2008)) may be used, but even then, limitations exist. This portion generates 3 parameters, each a color band multiplier.

Auto-contrast stretch. We find the intensity $I_0$ such that a maximum of 0.4 % of the pixels are darker or as dark as $I_0$, and intensity $I_1$ such that a maximum of 1 % of the pixels are brighter or as bright as $I_1$. We then linearly stretch the brightness so that $I_0$ is mapped to $0$ and $I_1$ is mapped to $255$ (with appropriate clamping at $0$ and $255$). This portion generates 2 parameters, a shift and a scale, similarly applied to all color bands.