Three-stream fusion network for first-person interaction recognition

Highlights

• We propose a novel three-stream framework for first-person interaction recognition.

• The three-stream correlation fusion can consider correlations between a target and a camera wearer.

• Superior performance by using the proposed three-stream architecture with the fusion method.

Abstract

First-person interaction recognition is a challenging task because of unstable video conditions resulting from the camera wearer’s movement. For human interaction recognition from a first-person viewpoint, this paper proposes a three-stream fusion network with two main parts: three-stream architecture and three-stream correlation fusion. The three-stream architecture captures the characteristics of the target appearance, target motion, and camera ego-motion. Meanwhile the three-stream correlation fusion combines the feature map of each of the three streams to consider the correlations among the target appearance, target motion, and camera ego-motion. The fused feature vector is robust to the camera movement and compensates for the noise of the camera ego-motion. Short-term intervals are modeled using the fused feature vector, and a long short-term memory (LSTM) model considers the temporal dynamics of the video. We evaluated the proposed method on two public benchmark datasets to validate the effectiveness of our approach. The experimental results show that the proposed fusion method successfully generated a discriminative feature vector, and our network outperformed all competing activity recognition methods in first-person videos where considerable camera ego-motion occurs.

Introduction

Despite the increasing research on computer vision, the task of understanding human activity in videos remains a challenging task. Recent approaches based on deep learning techniques have achieved significant progress in third-person activity recognition [1], [2], [3], [4], [5]. In third-person videos, the camera is fixed and at a large distance from people and objects. Many researchers have also studied human activity recognition in first-person video [6], [7], [8], [9], [10]. First-person video is captured by a camera mounted on a person or object. The video has a unique characteristics called camera ego-motion, which is not usually seen in third-person video. When the video includes camera ego-motion, the appearance of the target subjects is easily distorted and the motion vectors are disordered. Therefore, the recognition of activities in first-person video requires an appropriate approach tailored to these particular characteristics. In this paper, we analyze first-person video frames and focus especially on the interaction between a camera wearer and a human subject.

In Fig. 1, the optical flow extracted from two consecutive RGB frames shows various motion vectors in each third-person and first-person video. As shown in Fig. 1(a), motion vectors in the third-person video appear around the region where people are positioned because the camera is fixed and stationary. However, Fig. 1(b) shows complex motion vectors for the target and the camera wearer. The motion vectors appear close to the region where the target is positioned and the entire region of optical flow. Camera ego-motion renders analyzing the appearance and motion characteristics of the target in first-person video more challenging for three reasons: (1) it is difficult to build discriminative motion features from the complex motion vector in first-person video; (2) target appearance, which is an important feature for analyzing the target’s behavior, is severely distorted because of the camera wearer’s movement; (3) camera ego-motion can occur when the camera wearer moves, regardless of the target’s actions, and its data can be noisy when first-person interactions are analyzed.

This paper proposes a three-stream fusion network to recognize interactions in first-person video where large amounts of camera ego-motion occur. The proposed method is composed of two main parts: three-stream architecture and three-stream correlation fusion (TSCF). The three-stream architecture consists of target appearance stream, target motion stream, and ego-motion stream. The target appearance stream and the motion stream capture the appearance features and motion features of the target, respectively, and the ego-motion stream captures features of the camera wearer’s movement. The camera ego-motion is an important clue to the camera wearer’s movement because the wearer’s pose in first-person video is unknown. To generate robust features for the camera ego-motion, the TSCF considers two types of correlations. It considers the correlation between the target appearance and motion to utilize the spatiotemporal relationship. This relationship complements the target’s appearance and motion features, which are distorted by the camera wearer’s movement. To consider the problem of the camera wearer’s movement being noisy, the TSCF also uses the correlation between the target and camera ego-motion to determine whether the camera wearer’s movement is caused by the target’s action. In other words, we can determine whether the camera ego-motion is an important clue for analyzing first-person interactions between the target and the camera wearer.

The main contributions of this paper are summarized as follows: (1) we propose a novel deep learning framework called the three-stream fusion network. The proposed network is specialized in extracting discriminative features and considers camera ego-motion an important feature for analyzing the camera wearer’s movement; (2) we also introduce a fusion method called TSCF, which considers the correlations between the target’s appearance, motion and the camera ego-motion. The proposed fusion method creates robust features mitigate the effects of the camera ego-motion; (3)wWe show that our proposed method outperforms state-of-the-art activity recognition methods using the JPL First-Person Interaction dataset and the UTKinect-FirstPerson dataset.