Skip to main content
SpringerLink
Log in

Multi-Camera Multi-Target Tracking with Space-Time-View Hyper-graph

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Incorporating multiple cameras is an effective solution to improve the performance and robustness of multi-target tracking to occlusion and appearance ambiguities. In this paper, we propose a new multi-camera multi-target tracking method based on a space-time-view hyper-graph that encodes higher-order constraints (i.e., beyond pairwise relations) on 3D geometry, appearance, motion continuity, and trajectory smoothness among 2D tracklets within and across different camera views. We solve tracking in each single view and reconstruction of tracked trajectories in 3D environment simultaneously by formulating the problem as an efficient search of dense sub-hypergraphs on the space-time-view hyper-graph using a sampling based approach. Experimental results on the PETS 2009 dataset and MOTChallenge 2015 3D benchmark demonstrate that our method performs favorably against the state-of-the-art methods in both single-camera and multi-camera multi-target tracking, while achieving close to real-time running efficiency. We also provide experimental analysis of the influence of various aspects of our method to the final tracking performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. Many methods do not form tracklets but perform association directly on detections in each frame. In this work, we unify these methods by treating individual frame detections as tracklets of length one.

  2. The last frame index of \({\mathcal {T}}\) and the first frame index of \({\mathcal {T}}'\) may correspond to multiple detections from different camera views.

  3. Note that this is different from the degree of the nodes, which specifies how many hyper-edges can associate with one node.

  4. The \(\beta \)-subhypergraph indicates the sub-hypergraph of STV hyper-graph, which includes \(\beta \) nodes.

  5. The calculation of the number of hyper-edges, including nodes \(\nu \), \(\nu '\) and \(\nu _j\) is a combinational problem, that is to choose \(k-3\) nodes from the reliable node set \(\varOmega _i-\{\nu , \nu ', \nu _j\}\). Specifically, we set \(\rho _i = 0\) for \(|\varOmega _i| < 3\), since there does not exist enough nodes to construct a hyper-edge in that case.

  6. We will make our method and our implementation of Hofmann et al. (2013) along with the tracking results available after the paper decision.

  7. Since different input detections and ground truth are used, it is unfair to directly compare the tracking results of the proposed method with the results presented in Hofmann et al. (2013).

References

  • Andriyenko, A., & Schindler, K. (2011). Multi-target tracking by continuous energy minimization. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1265–1272).

  • Andriyenko, A., Schindler, K., & Roth, S. (2012). Discrete-continuous optimization for multi-target tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1926–1933).

  • Attanasi, A., Cavagna, A., Castello, L. D., Giardina, I., Jelic, A., Melillo, S., et al. (2015). GReTA—a novel global and recursive tracking algorithm in three dimensions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(1), 1.

    Article  Google Scholar 

  • Berclaz, J., Fleuret, F., & Fua, P. (2009). Multiple object tracking using flow linear programming. In Winter-PETS (pp. 1–8). Snowbird: IEEE.

  • Berclaz, J., Fleuret, F., Türetken, E., & Fua, P. (2011). Multiple object tracking using k-shortest paths optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(9), 1806–1819.

    Article  Google Scholar 

  • Breitenstein, M. D., Reichlin, F., Leibe, B., Koller-Meier, E., & Gool, L. J. V. (2011). Online multi-person tracking-by-detection from a single, uncalibrated camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(9), 1820–1833.

    Article  Google Scholar 

  • Brendel, W., Amer, M. R., & Todorovic, S. (2011). Multiobject tracking as maximum weight independent set. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1273–1280).

  • Chari, V., Lacoste-Julien, S., Laptev, I., & Sivic, J. (2015). On pairwise costs for network flow multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 5537–5545).

  • Dehghan, A., Tian, Y., Torr, P. H. S., & Shah, M. (2015). Target identity-aware network flow for online multiple target tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1146–1154).

  • Dollár, P., Wojek, C., Schiele, B., & Perona, P. (2012). Pedestrian detection: An evaluation of the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4), 743–761.

    Article  Google Scholar 

  • Felzenszwalb, P. F., McAllester, D. A., & Ramanan, D. (2008). A discriminatively trained, multiscale, deformable part model. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).

  • Ferryman, J. M., & Shahrokni, A. (2009). PETS2009: Dataset and challenge. In Winter-PETS (pp. 1–6).

  • Fleuret, F., Berclaz, J., Lengagne, R., & Fua, P. (2008). Multicamera people tracking with a probabilistic occupancy map. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 267–282.

    Article  Google Scholar 

  • Hofmann, M., Wolf, D., & Rigoll, G. (2013). Hypergraphs for joint multi-view reconstruction and multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 3650–3657).

  • Hong, L., & Cui, N. (2000). An interacting multipattern joint probabilistic data association (imp-jpda) algorithm for multitarget tracking. Signal Processing, 80(8), 1561–1575.

    Article  Google Scholar 

  • Huang, C., Li, Y., & Nevatia, R. (2013). Multiple target tracking by learning-based hierarchical association of detection responses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4), 898–910.

    Article  Google Scholar 

  • Isard, M., & Blake, A. (1998). Condensation—conditional density propagation for visual tracking. International Journal of Computer Vision, 29(1), 5–28.

    Article  Google Scholar 

  • Izadinia, H., Saleemi, I., Li, W., & Shah, M. (2012) (MP)\(^2\)T: Multiple people multiple parts tracker. In Proceedings of European Conference on Computer Vision (pp. 100–114).

  • Jiang, H., Fels, S., & Little, J. J. (2007). A linear programming approach for multiple object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).

  • Khan, Z., Balch, T. R., & Dellaert, F. (2005). MCMC-based particle filtering for tracking a variable number of interacting targets. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(11), 1805–1918.

    Article  Google Scholar 

  • Kim, J., Dai, Y., Li, H., Du, X., & Kim, J. (2013). Multi-view 3D reconstruction from uncalibrated radially-symmetric cameras. In Proceedings of IEEE International Conference on Computer Vision (pp. 1896–1903).

  • Klinger, T., Rottensteiner, F., & Heipke, C. (2015). Probabilistic multi-person tracking using dynamic bayes networks. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, II–3/W5, 435–442.

    Article  Google Scholar 

  • Kostrikov, I., Horbert, E., & Leibe, B. (2014). Probabilistic labeling cost for high-accuracy multi-view reconstruction. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1534–1541).

  • Kuhn, W., & Tucker, A. (1951) Nonlinear programming. In Proceedings of 2nd Berkeley Symposium (pp. 481–492).

  • Kuo, C. H., & Nevatia, R. (2011). How does person identity recognition help multi-person tracking? In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1217–1224).

  • Leal-Taixé, L., Milan, A., Reid, I.D., Roth, S., & Schindler, K. (2015). Motchallenge 2015: towards a benchmark for multi-target tracking. CoRR abs/1504.01942.

  • Leal-Taixé, L., Pons-Moll, G., & Rosenhahn, B. (2011). Everybody needs somebody: modeling social and grouping behavior on a linear programming multiple people tracker. In Workshops in Conjunction with IEEE International Conference on Computer Vision (pp. 120–127).

  • Leal-Taixé, L., Pons-Moll, G., & Rosenhahn, B. (2012) Branch-and-price global optimization for multi-view multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1987–1994).

  • Leven, W. F., & Lanterman, A. D. (2009). Unscented kalman filters for multiple target tracking with symmetric measurement equations. IEEE Transaction on Automatic Control, 54(2), 370–375.

    Article  MathSciNet  Google Scholar 

  • Liu, H., & Yan, S. (2012). Efficient structure detection via random consensus graph. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 574–581).

  • Liu, H., Yang, X., Latecki, L. J., & Yan, S. (2012). Dense neighborhoods on affinity graph. IJCV, 98(1), 65–82.

    Article  MathSciNet  MATH  Google Scholar 

  • Liu, Y., Li, H., & Chen, Y. Q. (2012). Automatic tracking of a large number of moving targets in 3d. In Proceedings of European Conference on Computer Vision (pp. 730–742).

  • Marchesotti, L., Marcenaro, L., Ferrari, G., & Regazzoni, C. S. (2002) Multiple object tracking under heavy occlusions by using kalman filters based on shape matching. In Proceedings of IEEE International Conference on Image Processing (pp. 341–344).

  • Milan, A. (2011) Continuous energy minimization tracker. http://www.milanton.de/contracking/index.html.

  • Milan, A., Leal-Taixé, L., Schindler, K., Roth, S., & Reid, I.D. (2015). Multiple object tracking benchmark: 3d mot. https://motchallenge.net/results/3D_MOT_2015/.

  • Milan, A., Roth, S., & Schindler, K. (2014). Continuous energy minimization for multitarget tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(1), 58–72.

    Article  Google Scholar 

  • Ojala, T., Pietikäinen, M., & Mäenpää, T. (2000). Gray scale and rotation invariant texture classification with local binary patterns. In Proceedings of European Conference on Computer Vision (pp. 404–420).

  • Pellegrini, S., Ess, A., Schindler, K., & Gool, L. J. V. (2009). You’ll never walk alone: modeling social behavior for multi-target tracking. In Proceedings of IEEE International Conference on Computer Vision (pp. 261–268).

  • Pirsiavash, H., Ramanan, D., & Fowlkes, C. C. (2011). Globally-optimal greedy algorithms for tracking a variable number of objects. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1201–1208).

  • Reid, D. B. (1979). An algorithm for tracking multiple targets. IEEE Transactions on Automatic Control, 24, 843–854.

    Article  Google Scholar 

  • Shi, X., Ling, H., Hu, W., Yuan, C., & Xing, J. (2014). Multi-target tracking with motion context in tensor power iteration. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 3518–3525).

  • Shu, G., Dehghan, A., Oreifej, O., Hand, E., & Shah, M. (2012). Part-based multiple-person tracking with partial occlusion handling. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1815–1821).

  • Smith, K., Gatica-Perez, D., & Odobez, J. M. (2005). Using particles to track varying numbers of interacting people. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 962–969).

  • Stiefelhagen, R., Bernardin, K., Bowers, R., Garofolo, J. S., Mostefa, D., & Soundararajan, P. (2006). The CLEAR 2006 evaluation. CLEAR (pp. 1–44). Berlin: Springer.

    Google Scholar 

  • Wen, L., Li, W., Yan, J., Lei, Z., Yi, D., & Li, S. Z. (2014) Multiple target tracking based on undirected hierarchical relation hypergraph. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (pp. 3457–3464).

  • Wu, Z., Hristov, N.I., Kunz, T. H., & Betke, M. (2009). Tracking-reconstruction or reconstruction-tracking? Comparison of two multiple hypothesis tracking approaches to interpret 3D object motion from several camera views. In Proceedings of the IEEE Workshop on Motion and Video Computing (pp. 1–8).

  • Wu, Z., Kunz, T. H., & Betke, M. (2011). Efficient track linking methods for track graphs using network-flow and set-cover techniques. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1185–1192).

  • Yang, B., & Nevatia, R. (2012). Multi-target tracking by online learning of non-linear motion patterns and robust appearance models. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1918–1925).

  • Yang, B., & Nevatia, R. (2012). An online learned CRF model for multi-target tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 2034–2041).

  • Yang, M., Liu, Y., Wen, L., You, Z., & Li, S. Z. (2014). A probabilistic framework for multitarget tracking with mutual occlusions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).

  • Yu, Q., & Medioni, G. G. (2009). Multiple-target tracking by spatiotemporal monte carlo markov chain data association. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(12), 2196–2210.

    Article  Google Scholar 

  • Zhang, L., Li, Y., & Nevatia, R. (2008). Global data association for multi-object tracking using network flows. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).

  • Zhou, D., Huang, J., & Schölkopf, B. (2006). Learning with hypergraphs: Clustering, classification, and embedding. Advances in Neural Information Processing Systems (pp. 1601–1608). Cambridge: MIT Press.

    Google Scholar 

Download references

Acknowledgments

We would like to thank Dawei Du for a number of suggestions that considerably improved the quality of this paper. Longyin Wen and Siwei Lyu were supported by US National Science Foundation Research Grant (CCF-1319800). Zhen Lei was supported by the National Key Research and Development Plan (Grant No. 2016 YFC0801002), the Chinese National Natural Science Foundation Projects #61375037, #61473291. Honggang Qi was supported by National Nature Science Foundation of China #61472388.

Author information

Authors and Affiliations

  1. Computer Science Department, University at Albany, State University of New York, Albany, USA

    Longyin Wen & Siwei Lyu

  2. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Zhen Lei

  3. Computer Engineering Department, University at Albany, State University of New York, Albany, USA

    Ming-Ching Chang

  4. School of Computer and Control Engineering, University of the Chinese Academy of Sciences, Beijing, China

    Honggang Qi

Authors
  1. Longyin Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhen Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ming-Ching Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Honggang Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Siwei Lyu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Honggang Qi.

Additional information

Communicated by Hiroshi Ishikawa, Takeshi Masuda, Yasuyo Kita and Katsushi Ikeuchi.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (avi 11348 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, L., Lei, Z., Chang, MC. et al. Multi-Camera Multi-Target Tracking with Space-Time-View Hyper-graph. Int J Comput Vis 122, 313–333 (2017). https://doi.org/10.1007/s11263-016-0943-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-016-0943-0

Keywords

Search

Navigation