Skip to main content
SpringerLink
Log in

Using In-frame Shear Constraints for Monocular Motion Segmentation of Rigid Bodies

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

It is a well known result in the vision literature that the motion of independently moving objects viewed by an affine camera lie on affine subspaces of dimension four or less. As a result a large number of the recently proposed motion segmentation algorithms model the problem as one of clustering the trajectory data to its corresponding affine subspace. While these algorithms are elegant in formulation and achieve near perfect results on benchmark datasets, they fail to address certain very key real-world challenges, including perspective effects and motion degeneracies. Within a robotics and autonomous vehicle setting, the relative configuration of the robot and moving object will frequently be degenerate leading to a failure of subspace clustering algorithms. On the other hand, while gestalt-inspired motion similarity algorithms have been used for motion segmentation, in the moving camera case, they tend to over-segment or under-segment the scene based on their parameter values. In this paper we present a principled approach that incorporates the strengths of both approaches into a cohesive motion segmentation algorithm capable of dealing with the degenerate cases, where camera motion follows that of the moving object. We first generate a set of prospective motion models for the various moving and stationary objects in the video sequence by a RANSAC-like procedure. Then, we incorporate affine and long-term gestalt-inspired motion similarity constraints, into a multi-label Markov Random Field (MRF). Its inference leads to an over-segmentation, where each label belongs to a particular moving object or the background. This is followed by a model selection step where we merge clusters based on a novel motion coherence constraint, we call in-frame shear, that tracks the in-frame change in orientation and distance between the clusters, leading to the final segmentation. This oversegmentation is deliberate and necessary, allowing us to assess the relative motion between the motion models which we believe to be essential in dealing with degenerate motion scenarios.We present results on the Hopkins-155 benchmark motion segmentation dataset [27], as well as several on-road scenes where camera and object motion are near identical. We show that our algorithm is competitive with the state-of-the-art algorithms on [27] and exceeds them substantially on the more realistic on-road sequences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Namdev, R.K., Kundu, A., Krishna, K.M., Jawahar, C.V.: Motion segmentation of multiple objects from a freely moving monocular camera. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 4092–4099, IEEE (2012)

  2. Lezama, J., Alahari, K., Sivic, J., Laptev, I.: Track to the future: Spatio-temporal video segmentation with long-range motion cues. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2011)

  3. Elhamifar, E., Vidal, R.: Sparse subspace clustering. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2009, pp. 2790–2797. IEEE (2009)

  4. Sugaya, Y., Kanatani, K.: Geometric structure of degeneracy for multi-body motion segmentation. In: Statistical Methods in Video Processing, pp. 13–25. Springer, Berlin Heidelberg (2004)

  5. Flores-Mangas, F., Jepson, A.D.: Fast rigid motion segmentation via incrementally-complex local models. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2259–2266. IEEE (2013)

  6. Yan, J., Pollefeys, M.: A general framework for motion segmentation: Independent, articulated, rigid, non-rigid, degenerate and non-degenerate. In: Computer Vision ECCV 2006, pp. 94–106, Springer, Berlin Heidelberg (2006)

  7. Kanatani, K.: Motion segmentation by subspace separation and model selection. Image 1, 1 (2001)

    Google Scholar 

  8. Rao, S.R., Tron, R., Vidal, R., Ma, Y.: Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008. pp. 1–8. IEEE (2008)

  9. Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(9), 1627–1645 (2010)

    Article  Google Scholar 

  10. Fragkiadaki, K., Shi, J.: Figure-ground image segmentation helps weakly-supervised learning of objects. In: Computer Vision ECCV 2010, pp. 561–574. Springer, Berlin Heidelberg (2010)

  11. Isack, H., Boykov, Y.: Energy-based geometric multi-model fitting. Int. J. Comput. Vis. 97(2), 123–147 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  12. Delong, A., Osokin, A., Isack, H.N., Boykov, Y.: Fast approximate energy minimization with label costs. Int. J. Comput. Vis. 96(1), 1–27 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  13. Lee, Y.J., Kim, J., Grauman, K.: Key-segments for video object segmentation. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1995–2002. IEEE (2011)

  14. Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 888–905 (2011)

    Google Scholar 

  15. Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)

  16. Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(10), 1568–1583 (2006)

    Article  Google Scholar 

  17. Comport, A.I., Malis, E., Rives, P.: Real-time quadrifocal visual odometry. Int. J. Robot. Res. 29(2-3), 245–266 (2010)

    Article  Google Scholar 

  18. Wertheimer, M.M.: Laws of organization in perceptual forms (1938)

  19. Wong, H.S., Chin, T.J., Yu, J., Suter, D.: Efficient multi-structure robust fitting with incremental top-k lists comparison. In: Computer VisionACCV 2010, pp. 553–564. Springer, Berlin Heidelberg (2011)

  20. Romero-Cano, V., Nieto, J.I.: Stereo-based motion detection and tracking from a moving platform. In: Intelligent Vehicles Symposium (IV), 2013 IEEE pp. 499–504, IEEE (2013)

  21. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354–3361. IEEE (2012)

  22. Brox, T., Malik, J.: Object segmentation by long term analysis of point trajectories. In: Computer Vision ECCV 2010, pp. 282–295. Springer, Berlin Heidelberg (2010)

  23. Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vis. 9(2), 137–154 (1992)

    Article  Google Scholar 

  24. Kundu, A., Krishna, K.M., Sivaswamy, J.: Moving object detection by multi-view geometric techniques from a single camera mounted robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009. IROS 2009, pp. 4306–4312. IEEE (2009)

  25. Tsai, D., Flagg, M., Nakazawa, A., Rehg, J.M.: Motion coherent tracking using multi-label mrf optimization. Int. J. Comput. Vis. 100(2), 190–202 (2012)

    Article  MathSciNet  Google Scholar 

  26. Jain, S., Govindu, V.M.: Efficient Higher-Order Clustering on the Grassmann Manifold, ICCV, 2013 (2013)

  27. Tron, R., Vidal, R.: A benchmark for the comparison of 3-d motion segmentation algorithms. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007, CVPR’07. pp. 1–8. IEEE (2007)

  28. Vidal, R., Ma, Y.: A unified algebraic approach to 2-D and 3-D motion segmentation. In: Computer Vision ECCV 2004, pp. 1–15. Springer, Berlin Heidelberg (2004)

  29. Fragkiadaki, K., Zhang, G., Shi, J.: Video segmentation by tracing discontinuities in a trajectory embedding. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1846-1853. IEEE (2012)

  30. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  31. Li, Z., Guo, J., Cheong, L.-F., Zhou, S.Z.: Perspective Motion Segmentation via Collaborative Clustering, ICCV (2013)

  32. Lenz, P., Ziegler, J., Geiger, A., Roser, M.: Sparse scene flow segmentation for moving object detection in urban environments. In: 2011 IEEE Intelligent Vehicles Symposium (IV), pp. 926–932. IEEE (2011)

  33. Zografos, V., Nordberg, K.: Fast and accurate motion segmentation using Linear Combination of Views. In: BMVC, pp. 1–11 (2011)

  34. Weiss, Y., Adelson, E.H.: A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models. In: 1996 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1996. Proceedings CVPR’96, pp. 321–326. IEEE (1996)

  35. Black, M.J., Anandan, P.: The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comp. Vis. Image Underst. 63(1), 75–104 (1996)

    Article  Google Scholar 

  36. Li, T., Kallem, V., Singaraju, D., Vidal, R.: Projective factorization of multiple rigid-body motions. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007, CVPR’07. pp. 1–6. IEEE (2007)

  37. Weiss, Y.: Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In: 1997 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1997, Proceedings. pp. 520–526. IEEE (1997)

  38. Weiss, Y.: Segmentation using eigenvectors: a unifying view. In: The proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, vol. 2, pp. 975–982. IEEE (1999)

  39. Torr, P.H.S.: Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Phys. Eng. Sci. 356(1740), 1321–1340 (1998)

    MathSciNet  MATH  Google Scholar 

  40. Vidal, R., Soatto, S., Sastry, S.S.: A factorization method for 3D multi-body motion estimation and segmentation. In: Proceesings of the Annual Allerton Conference on Communication Control and Computing, vol. 140, no. 13, pp. 11626-1635. The University; 1998 (2002)

  41. Vidal, R., Sastry, S.: Optimal segmentation of dynamic scenes from two perspective views. In: Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2003, vol. 2, pp. II–281. IEEE (2003)

  42. Shakernia, O., Vidal, R., Sastry, S.: Multibody motion estimation and segmentation from multiple central panoramic views. In: IEEE International Conference on Robotics and Automation, 2003, Proceedings, ICRA’03, vol. 1, pp. 571–576. IEEE (2003)

  43. Chen, G., Lerman, G.: Spectral curvature clustering (SCC). Int. J. Comput. Vis. 81(3), 317–330 (2009)

    Article  Google Scholar 

  44. Chin, T.-J., Wang, H., Suter, D.: The ordered residual kernel for robust motion subspace clustering. In: NIPS, vol. 9, pp. 333–341 (2009)

  45. Shi, J., Tomasi, C.: Good features to track. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1994, Proceedings CVPR’94, pp. 593–600. IEEE (1994)

  46. Zappella, L., Provenzi, E., Llad, X., Salvi, J.: Adaptive motion segmentation algorithm based on the principal angles configuration. In: Computer Vision-ACCV 2010, pp. 15–26. Springer (2011)

  47. Vidal R., Hartley, R.: Motion segmentation with missing data by PowerFactorization and Generalized PCA. In: CVPR (2004)

  48. Zappella, L. et al.: Enhanced local subspace affinity for feature-based motion segmentation. Pattern Recogn. 44.2, 454–470 (2011)

    Article  Google Scholar 

  49. Ochs, P., Brox, T.: Higher order motion models and spectral clustering. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2012)

  50. Yuan, C., Medioni, G., Kang, J., Cohen, I.: Detecting motion regions in the presence of a strong parallax from a moving camera by multiview geometric constraints. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(9), 1627–1641 (2007)

    Article  Google Scholar 

  51. Kang, J., Cohen, I., Medioni, G., Yuan, C.: Detection and tracking of moving objects from a moving platform in presence of strong parallax. In: Tenth IEEE International Conference on Computer Vision, 2005, ICCV 2005, vol. 1, pp. 10–17 (2005)

  52. Lourakis, M.I., Argyros, A.A., Orphanoudakis, S.C.: Independent 3D motion detection using residual parallax normal flow fields? In: Proceedings of the International Conference on Computer Vision, pp. 1012–1017 (1998)

  53. Shashua, A., Wolf, L.: Homography tensors: on algebraic entities that represent three views of static or moving planar points. In: Proceedings of the European Conference on Computer Vision, vol. 1, pp. 507–521 (2000)

Download references

Author information

Authors and Affiliations

  1. Robotics Research Center, IIIT Hyderabad, Gachibowli, Hyderabad, 500032, Andhra Pradesh, India

    Siddharth Tourani & K Madhava Krishna

Authors
  1. Siddharth Tourani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. K Madhava Krishna
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Siddharth Tourani.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tourani, S., Krishna, K.M. Using In-frame Shear Constraints for Monocular Motion Segmentation of Rigid Bodies. J Intell Robot Syst 82, 237–255 (2016). https://doi.org/10.1007/s10846-015-0195-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-015-0195-1

Keywords

Search

Navigation