Soccer player recognition using spatial constellation features and jersey number recognition

https://doi.org/10.1016/j.cviu.2017.04.010Get rights and content

Highlights

  • An approach for player identification in broadcast soccer videos is proposed.

  • Player Identification is performed on wide-angle shots, causing low resolution per player.

  • Spatial constellation features describing a player’s position can aid identification.

  • A Combination of spatial features and jersey number recognition is presented.

  • Identification accuracy is improved from 0.69 (jersey number recognition) to 0.82 (additional spatial features).

Abstract

Identifying players in soccer videos is a challenging task, especially in overview shots. Face recognition is not feasible due to low resolution, and jersey number recognition suffers from low resolution, motion blur and unsuitable player pose. Therefore, a method to improve visual identification using spatial constellations is proposed here. This method models a spatial constellation as a histogram over relative positions among all players of the team. Using constellation features might increase identification performance but is not expected to work well as a single mean of identification. Thus, this constellation-based recognition is combined with jersey number recognition using convolutional neural networks. Recognizing the numbers on a player’s shirt is the most straight-forward approach, as there is a direct mapping between numbers and players.

Using spatial constellation as a feature for identification is based on the assumption that players do not move randomly over the pitch. Players rather follow a tactical role such as central defender, winger, forward, etc. However in soccer, players do not strictly adhere to these roles, variations occur more or less frequently. By learning constellation models for each player, we avoid a categorical assignment of a player to one single tactical role and therefore incorporate each player’s typical behaviour in terms of switching positions.

The presented player identification process is expressed as an assignment problem. Here, an optimal assignment of manually labeled trajectories to known player identities is calculated. Using an assignment problem allows for a flexible fusion of constellation features and jersey numbers by combining the respective cost matrices. Evaluation is performed on 14 different shots of six different Bundesliga matches. By combining both modalities, the accuracy is improved from 0.69 to 0.82 when compared with jersey number recognition only.

Introduction

Soccer is one of the most popular sports in the world. Interest in automatic soccer analysis tools grew significantly in recent years. Soccer analysis results can be used for new ways of storytelling on TV, for match preparations or statistical evaluation. One fundamental analysis is the identification of players to individually associate actions and statistics. But due to the fact that no reliable automatic identification technologies exist at present, this task is typically carried out by human annotators, using respective tools and standards. The identification of players in broadcast sports videos is of utmost interest and accordingly, a number of researchers addressed this problem in the recent past as reviewed in Section 2.

However, identifying players in broadcast soccer videos automatically (and even manually) is challenging. Especially for the overview camera this task is difficult due to the low resolution per player, which makes face recognition impossible. And even jersey numbers are often hard to recognize, especially in standard definition (SD) resolutions. Only with the rise of widely available high definition (HD) content in recent years, jersey number recognition became feasible. The spatial constellation of players supports manual annotators in the identification process, as players do not move randomly over the pitch. However, this feature is left unexploited in most automatic identification approaches.

Therefore, the main contributions of this work are spatial team constellation features and models that are suitable for soccer player identification. Furthermore, a combination of these spatial constellation models with jersey number recognition is contributed.

This paper is organized as follows. First, related work in the area of player identification in sports video is presented in Section 2. Within this work, player identification is posed as an assignment problem. The exact formulation and how to solve an assignment problem is described in Section 3. A way to model spatial constellation of player positions is explained in Section 4. The jersey number recognition using convolutional neural networks (CNN) is presented in Section 5. Then, both modalities are combined, as described in Section 6. In Section 7, the conducted experiments are described in detail. Both modalities are evaluated separately and then compared to the combined method. Finally, a conclusion is drawn in Section 8 including possible further research directions.

Section snippets

Related work

Existing work on identification of players in team sport broadcasts mostly rely on visual features only and two subcategories stand out in particular: One performing face recognition, while the other investigates jersey number recognition.

As an example for the first group Ballan et al. (2007) performed face recognition for soccer close-up shots. Specific problems that may occur in sport video close-ups such as high variation in pose, illumination, scale and occlusion are addressed by employing

Player identification as an assignment problem

Within this work, player observations are given for certain time periods. Consecutive observations of the same player are aggregated into a trajectory. We further assume that the team assignment of each trajectory is already known or easily ascertained due to distinguishable jersey colors. Player trajectories are separated according to their team and the following conditions apply for identifying soccer players only within the same team.

Player constellation

One method to identify players in a soccer match is based on the spatial constellation of players within their team. In this section, we describe how these constellations can be used for the representation of observed players and for the trained models of known players. Therefore the feature representation xo of observations oOvis, the trained models Mp of known players pP, and a distance (or similarity) function d(xo, Mp) between feature representations and models are described. Player

Jersey number recognition using convolutional neural networks

For player identification, recognizing jersey numbers is the most straightforward approach. In professional soccer leagues, every player has a unique jersey number throughout a complete season, allowing an unambiguous mapping from jersey number to player and vice versa.

A convolutional neural network is trained to recognize these numbers in a grayscale image region of 40 × 40 pixel. Grayscale images are used to avoid that the neural network erroneously learns color as a feature for certain

Classifier fusion

In order to combine recognition based on positional features with jersey number recognition, both modalities are fused. There are mainly three types of fusion possible: Feature fusion, cost matrix fusion and classifier (i.e. assignment) fusion. In our approach, cost matrix fusion is used. This allows for different kinds of cost functions for each modality, which would be more difficult to implement when performing feature fusion. And it has the advantage that the combined solution represents an

Experimental results

In order to assess the presented player identification approach independently of previous steps in a video processing pipeline (e.g. automatic player tracking), it is based on manually created player trajectories. This avoids that errors in an automatic tracking process are propagated to the player identification. For this manual labeling, automatic player detection on regularly sampled video frames (every second) was performed. Then, human annotators were asked to associate player bounding

Conclusion

In this paper, a method for player identification in broadcast soccer videos is presented. For this, a combination of jersey number recognition and spatial constellation of players on the pitch is described. The general problem of identifying players in overview shots of TV soccer broadcast is posed as a piece-wise assignment problem. Within this assignment problem, both modalities are integrated by combining cost matrices for jersey number recognition and constellation features respectively.

References (28)

  • E. Andrade et al.

    Player identification in interactive sport scenes using region space analysis prior information and number recognition

    International Conference on Visual Information Engineering (VIE)

    (2003)
  • F. Angehrn et al.

    MasterCam FVV: robust registration of multiview sports video to a static high-resolution master camera for free viewpoint video

    IEEE International Conference on Image Processing, ICIP

    (2014)
  • L. Ballan et al.

    Soccer players identification based on visual local features

    6th ACM International Conference on Image and Video Retrieval

    (2007)
  • M. Beetz et al.

    Computerized real-time analysis of football games

    IEEE Pervasive Comput.

    (2005)
  • A. Bialkowski et al.

    Person re-identification using group information

    International Conference on Digital Image Computing: Techniques and Applications (DICTA)

    (2013)
  • Chollet, F., 2015. Keras....
  • P. Cintia et al.

    The harsh rule of the goals: data-driven performance indicators for football teams

    IEEE International Conference on Data Science and Advanced Analytics (DSAA)

    (2015)
  • M. Couceiro et al.

    Dynamical stability and predictability of football players: the study of one match

    Entropy

    (2014)
  • D. Delannay et al.

    Detection and recognition of sports(wo)men from multiple views

    ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC)

    (2009)
  • J. Duch et al.

    Quantifying the performance of individual players in a team activity

    PLoS ONE

    (2010)
  • D. Farin et al.

    Fast camera calibration for the analysis of sport sequences

    IEEE International Conference on Multimedia and Expo, ICME

    (2005)
  • W. Frencken et al.

    Oscillations of centroid position and surface area of soccer teams in small-sided games

    Eur. J. Sport Sci.

    (2011)
  • S. Gerke et al.

    Identifying soccer players using spatial constellation features

    KDD Workshop on Large-Scale Sports Analytics

    (2015)
  • S. Gerke et al.

    Soccer jersey number recognition using convolutional neural networks

    IEEE International Conference on Computer Vision Workshops

    (2015)
  • Cited by (0)

    View full text