Skip to main content
Log in

Abstract

In this paper we address the problem of recovering 3D non-rigid structure from a sequence of images taken with a stereo pair. We have extended existing non-rigid factorization algorithms to the stereo camera case and presented an algorithm to decompose the measurement matrix into the motion of the left and right cameras and the 3D shape, represented as a linear combination of basis-shapes. The added constraints in the stereo camera case are that both cameras are viewing the same structure and that the relative orientation between both cameras is fixed. Our focus in this paper is on the recovery of flexible 3D shape rather than on the correspondence problem. We propose a method to compute reliable 3D models of deformable structure from stereo images. Our experiments with real data show that improved reconstructions can be achieved using this method. The algorithm includes a non-linear optimization step that minimizes image reprojection error and imposes the correct structure to the motion matrix by choosing an appropriate parameterization. We show that 3D shape and motion estimates can be successfully disambiguated after bundle adjustment and demonstrate this on synthetic and real image sequences. While this optimization step is proposed for the stereo camera case, it can be readily applied to the case of non-rigid structure recovery using a monocular video sequence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aanaes, H. and Kahl, F. 2002, Estimation of deformable structure and motion. In Workshop on Vision and Modelling of Dynamic Scenes, ECCV'02, Copenhagen, Denmark.

  • Bar-Itzhack, I.Y. 2000. New method for extracting the quaternion from a rotation matrix. Journal of Guidance, Control and Dynamics, 23(3):1085–1087.

    MathSciNet  Google Scholar 

  • Brand, M. 2001. Morphable models from video. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii.

  • Brand, M. and Bhotika, R. 2001. Flexible flow for 3D nonrigid tracking and shape recovery. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii, pp. 315–322.

  • Bregler, C., Hertzmann, A., and Biermann, H. 2000. Recovering non-rigid 3D shape from image streams. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Hilton Head, South Carolina, pp. 690–696.

  • Del Bue, A. and Agapito, L. 2004. Non-rigid 3D shape recovery using stereo factorization. In Asian Conference on Computer Vision (ACCV2004), vol. 1. Jeju, South Korea.

  • Del Bue, A., Smeraldi, F., and Agapito, L. 2004, Non-rigid structure from motion using nonparametric tracking and non-linear optimization. In Workshop in Articulated and Nonrigid Motion ANM04, held in Conjunction with CVPR2004. Washington.

  • Essa, I. and Basu, S. 1996. Modeling, tracking and interactive animation of facial expressions and head movements using input from video. In Proceedings of Computer Animation Conference. Geneva, Switzerland.

  • Horn, B. 1987, Closed form solutions of absolute orientation using unit quaternions. J. Optical Soc. of America A. 4(4): 629–642.

    MathSciNet  Google Scholar 

  • Irani, M. 1999, Multi-frame optical flow estimation using subspace constraints. In Proc. 7th International Conference on Computer Vision, Kerkyra, Greece.

  • Parke, F.I. and Waters, K. 1996. Computer Facial Animation. A.K. Peters, Ltd.

  • Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D.H. 1998. Synthesising realistic facial expressions from photographs. In Proceedings of the ACM SIGGRAPH Conference on Computer Graphics.

  • Sugaya, Y. and Kanatani, K. 2004. Extending interrupted feature point tracking for 3-D affine reconstruction. IEICE Transactions on Information and System, E87-D(4):1031–1039.

    Google Scholar 

  • Tan, J. and Ishikawa, S. 2001. Deformable shape recovery by factorization based on a spatiotemporal measurement matrix. Computer Vision and Image Understanding, 82:101–109.

    Article  Google Scholar 

  • Tomasi, C. and Kanade, T. 1991. Shape and motion from image streams: A factorization method. International Journal in Computer Vision, 9(2):137–154.

    Google Scholar 

  • Torresani, L., Yang, D., Alexander, E., and Bregler, C. 2001. Tracking and modeling non-rigid objects with rank constraints. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii.

  • Tresadern, P. and Reid, I. 2003. Synchronizing image sequences of non-rigid objects. In Proc. British Machine Vision Conference, Norwich.

  • Triggs, B., McLauchlan, P., Hartley, R., and Fitzgibbon., A. 2000. Bundle adjustment---A modern synthesis. In W. Triggs, A. Zisserman, and R. Szeliski (Eds.), Vision Algorithms: Theory and Practice, LNCS. Springer Verlag, pp. 298–375.

  • Vetter, T. and Blanz, V. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the ACM SIGGRAPH Conference on Computer Graphics, pp. 187–194.

  • Xiao, J., Chai, J., and Kanade, T. 2004. A closed-form solution to non-rigid shape and motion recovery. In The 8th European Conference on Computer Vision (ECCV 2004).

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alessio Del Bue.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Cite this article

Del Bue, A., Agapito, L. Non-Rigid Stereo Factorization. Int J Comput Vision 66, 193–207 (2006). https://doi.org/10.1007/s11263-005-3958-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-005-3958-5

Keywords

Navigation