Video Enhancement with Task-Oriented Flow

Abstract

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.

Appendix

Additional Qualitative Results We show additional results on the following benchmarks: Vimeo interpolation benchmark (Fig. 14), Vimeo denoising benchmark (Fig. 15 for RGB videos, and Fig. 16 for grayscale videos), Vimeo deblocking benchmark (Fig. 17), and Vimeo super-resolution benchmark (Fig. 18). We randomly select testing images from test datasets. Differences between different algorithms are more clearer when zoomed in.

Flow Estimation Module We used SpyNet (Ranjan and Black 2017) as our flow estimation module. It consists of four sub-networks with the same network structure, but each sub-network has an independent set of parameters. Each sub-network consists of five sets of 7$\times$7 convolutional (with zero padding), batch normalization and ReLU layers. The number of channels after each convolutional layer is 32, 64, 32, 16, and 2. The input motion to the first network is a zero motion field.

Image Processing Module We use slight different structures in the image processing module for different tasks. For temporal frame interpolation both with and without masks, we build a residual network that consists of an averaging network and a residual network. The averaging network simply averages the two transformed frames (from frames 1 and 3). The residual network also takes the two transformed frames as input, but calculates the difference between the actual second frame and the average of two transformed frames through a convolutional network consists of three convolutional layers, each of which is followed by a ReLU layer. The kernel sizes of three layers are 9$\times$9, 1$\times$1, and 1$\times$1 (with zero padding), and the numbers of output channels are 64, 64, and 3. The final output is the summation of the output of the averaging network and the residual network.

Fig. 14

Qualitative results on video interpolation. Samples are randomly selected from the Vimeo interpolation benchmark. The differences between different algorithms are clear only when zoomed in

Fig. 15

Qualitative results on RGB video denoising. Samples are randomly selected from the Vimeo denoising benchmark. The differences between different algorithms are clear only when zoomed in

Fig. 16

Qualitative results on grayscale video denoising. Samples are randomly selected from the Vimeo denoising benchmark. The differences between different algorithms are clear only when zoomed in

Fig. 17

Qualitative results on video deblocking. Samples are randomly selected from the Vimeo deblocking benchmark. The differences between different algorithms are clear only when zoomed in

Fig. 18

Qualitative results on video super-resolution. Samples are randomly selected from the Vimeo super-resolution benchmark. The differences between different algorithms are clear only when zoomed in. DeepSR was originally trained on 30–50 images, but evaluated on 7 frames in this experiment, so there are some artifacts

For video denoising/deblocking, the image processing module uses the same six-layer convolutional structure (three convolutional layers and three ReLU layers) as interpolation, but without the residual structure. We have also tried the residual structure for denoising/deblocking, but there is no significant improvement.

For video super-resolution, the image processing module consists of four pairs of convolutional layers and ReLU layers. The kernel sizes for these four layers are 9$\times$9, 9$\times$9, 1$\times$1, and 1$\times$1 (with zero padding), and the numbers of output channels are 64, 64, 64, and 3.

Mask Network Similar to our flow estimation module, our mask estimation network is also a four-level convolutional neural network pyramid as in Fig. 4. Each level consists of the same sub-network structure with five sets of 7$\times$7 convolutional (with zero padding), batch normalization and ReLU layers, but an independent set of parameters (output channels are 32, 64, 32, 16, and 2). For the first level, the input to the network is a concatenation of two estimated optical flow fields (four channels after concatenation), and the output is a concatenation of two estimated masks (one channel per mask). From the second level, the input to the network switch to a concatenation of, first, two estimated optical flow fields at that resolution, and second, bilinear-upsampled masks from the previous level (the resolution is twice of the previous level). In this way, the first level mask network estimates a rough mask, and the rest refines high frequency details of the mask.

We use cycle consistencies to obtain the ground truth occlusion mask for pre-training the mask network. For two consecutive frames $I_1$ and $I_2$, we calculate the forward flow $v_{12}$ and the backward flow $v_{21}$ using the pre-trained flow network. Then, for each pixel p in image $I_1$, we first map it to $I_2$ using $v_{12}$ and then map it back to $I_1$ using $v_{21}$. If it maps to a different point rather to p (up to an error threshold of two pixels), then this point is considered to be occluded.