Soccer player recognition using spatial constellation features and jersey number recognition
Graphical abstract
Introduction
Soccer is one of the most popular sports in the world. Interest in automatic soccer analysis tools grew significantly in recent years. Soccer analysis results can be used for new ways of storytelling on TV, for match preparations or statistical evaluation. One fundamental analysis is the identification of players to individually associate actions and statistics. But due to the fact that no reliable automatic identification technologies exist at present, this task is typically carried out by human annotators, using respective tools and standards. The identification of players in broadcast sports videos is of utmost interest and accordingly, a number of researchers addressed this problem in the recent past as reviewed in Section 2.
However, identifying players in broadcast soccer videos automatically (and even manually) is challenging. Especially for the overview camera this task is difficult due to the low resolution per player, which makes face recognition impossible. And even jersey numbers are often hard to recognize, especially in standard definition (SD) resolutions. Only with the rise of widely available high definition (HD) content in recent years, jersey number recognition became feasible. The spatial constellation of players supports manual annotators in the identification process, as players do not move randomly over the pitch. However, this feature is left unexploited in most automatic identification approaches.
Therefore, the main contributions of this work are spatial team constellation features and models that are suitable for soccer player identification. Furthermore, a combination of these spatial constellation models with jersey number recognition is contributed.
This paper is organized as follows. First, related work in the area of player identification in sports video is presented in Section 2. Within this work, player identification is posed as an assignment problem. The exact formulation and how to solve an assignment problem is described in Section 3. A way to model spatial constellation of player positions is explained in Section 4. The jersey number recognition using convolutional neural networks (CNN) is presented in Section 5. Then, both modalities are combined, as described in Section 6. In Section 7, the conducted experiments are described in detail. Both modalities are evaluated separately and then compared to the combined method. Finally, a conclusion is drawn in Section 8 including possible further research directions.
Section snippets
Related work
Existing work on identification of players in team sport broadcasts mostly rely on visual features only and two subcategories stand out in particular: One performing face recognition, while the other investigates jersey number recognition.
As an example for the first group Ballan et al. (2007) performed face recognition for soccer close-up shots. Specific problems that may occur in sport video close-ups such as high variation in pose, illumination, scale and occlusion are addressed by employing
Player identification as an assignment problem
Within this work, player observations are given for certain time periods. Consecutive observations of the same player are aggregated into a trajectory. We further assume that the team assignment of each trajectory is already known or easily ascertained due to distinguishable jersey colors. Player trajectories are separated according to their team and the following conditions apply for identifying soccer players only within the same team.
Player constellation
One method to identify players in a soccer match is based on the spatial constellation of players within their team. In this section, we describe how these constellations can be used for the representation of observed players and for the trained models of known players. Therefore the feature representation xo of observations o ∈ Ovis, the trained models Mp of known players p ∈ P, and a distance (or similarity) function d(xo, Mp) between feature representations and models are described. Player
Jersey number recognition using convolutional neural networks
For player identification, recognizing jersey numbers is the most straightforward approach. In professional soccer leagues, every player has a unique jersey number throughout a complete season, allowing an unambiguous mapping from jersey number to player and vice versa.
A convolutional neural network is trained to recognize these numbers in a grayscale image region of 40 × 40 pixel. Grayscale images are used to avoid that the neural network erroneously learns color as a feature for certain
Classifier fusion
In order to combine recognition based on positional features with jersey number recognition, both modalities are fused. There are mainly three types of fusion possible: Feature fusion, cost matrix fusion and classifier (i.e. assignment) fusion. In our approach, cost matrix fusion is used. This allows for different kinds of cost functions for each modality, which would be more difficult to implement when performing feature fusion. And it has the advantage that the combined solution represents an
Experimental results
In order to assess the presented player identification approach independently of previous steps in a video processing pipeline (e.g. automatic player tracking), it is based on manually created player trajectories. This avoids that errors in an automatic tracking process are propagated to the player identification. For this manual labeling, automatic player detection on regularly sampled video frames (every second) was performed. Then, human annotators were asked to associate player bounding
Conclusion
In this paper, a method for player identification in broadcast soccer videos is presented. For this, a combination of jersey number recognition and spatial constellation of players on the pitch is described. The general problem of identifying players in overview shots of TV soccer broadcast is posed as a piece-wise assignment problem. Within this assignment problem, both modalities are integrated by combining cost matrices for jersey number recognition and constellation features respectively.
References (28)
- et al.
Player identification in interactive sport scenes using region space analysis prior information and number recognition
International Conference on Visual Information Engineering (VIE)
(2003) - et al.
MasterCam FVV: robust registration of multiview sports video to a static high-resolution master camera for free viewpoint video
IEEE International Conference on Image Processing, ICIP
(2014) - et al.
Soccer players identification based on visual local features
6th ACM International Conference on Image and Video Retrieval
(2007) - et al.
Computerized real-time analysis of football games
IEEE Pervasive Comput.
(2005) - et al.
Person re-identification using group information
International Conference on Digital Image Computing: Techniques and Applications (DICTA)
(2013) - Chollet, F., 2015. Keras....
- et al.
The harsh rule of the goals: data-driven performance indicators for football teams
IEEE International Conference on Data Science and Advanced Analytics (DSAA)
(2015) - et al.
Dynamical stability and predictability of football players: the study of one match
Entropy
(2014) - et al.
Detection and recognition of sports(wo)men from multiple views
ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC)
(2009) - et al.
Quantifying the performance of individual players in a team activity
PLoS ONE
(2010)