Skip to main content
SpringerLink
Log in

Video retargeting combining warping and summarizing optimization

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

We construct a unified interactive video retargeting system for video summarization, completion, and reshuffling. Our system combines the advantages of both video warping and summarizing processing. We first warp the video to present initial editing results, then refine the results using patch-based summarizing optimization, which mainly eliminates possible distortion produced in the warping step. We develop a Mean Value Coordinate (MVC) warping method due to its simplicity and efficiency used in the initialization. For refining processing, the summarization optimization is built on a 3D bidirectional similarity measure between the original and edited video, to preserve the coherence and completeness of the final editing result. We further improve the quality of summarization by applying a color histogram matching during the optimization, and accelerate the summarization optimization by using a constrained 3D Patch-Match algorithm. Experiment results show that the proposed video retargeting system effectively supports video summarization, completion, and reshuffling while avoiding issues like texture broken, video jittering, and detail losing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM Trans. Graph. 26(3), 10 (2007)

    Article  Google Scholar 

  2. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: Patch-Match: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 1–11 (2009)

    Article  Google Scholar 

  3. Dahan, M.J., Chen, N., Shamir, A., Cohen-Or, D.: Combining color and depth for enhanced image segmentation and retargeting. Vis. Comput. 28(12), 1181–1193 (2012)

    Article  Google Scholar 

  4. Deselaers, T., Dreuw, P., Ney, H.: Pan, zoom, scan—time-coherent, trained automatic video cropping. In: CVPR 2008, pp. 1–8. IEEE Press, New York (2008)

    Google Scholar 

  5. Floater, M., Kós, G., Reimers, M.: Mean value coordinates in 3D. Comput. Aided Geom. Des. 22(7), 623–631 (2005)

    Article  MATH  Google Scholar 

  6. Gal, R., Cohen-Or, D.: Feature-aware texturing. In: Rendering Techniques 2006 (2006)

    Google Scholar 

  7. Hormann, K., Floater, M.: Mean value coordinates for arbitrary planar polygons. ACM Trans. Graph. 25(4), 1424–1441 (2006)

    Article  Google Scholar 

  8. Ju, T., Schaefer, S., Warren, J.: Mean value coordinates for closed triangular meshes. In: ACM SIGGRAPH 2005 Papers, pp. 561–566. ACM, New York (2005)

    Chapter  Google Scholar 

  9. Kopf, J., Fu, C., Cohen-Or, D., Deussen, O., Lischinski, D., Wong, T.: Solid texture synthesis from 2D exemplars. ACM Trans. Graph. 26(3), 2 (2007)

    Article  Google Scholar 

  10. Krahenbuhl, P., Lang, M., Hornung, A., Gross, M.: A system for retargeting of streaming video. ACM Trans. Graph. 28(5), 1–10 (2009)

    Article  Google Scholar 

  11. Liu, F., Gleicher, M.: Video retargeting: automating pan and scan. In: Proceedings of the 14th Annual ACM International Conference on Multimedia, pp. 241–250. ACM, New York (2006)

    Chapter  Google Scholar 

  12. Pritch, Y., Kav-Venaki, E., Peleg, S.: Shift-map image editing. In: ICCV, pp. 151–158. IEEE Press, New York (2010)

    Google Scholar 

  13. Rubinstein, M., Shamir, A., Avidan, S.: Improved seam carving for video retargeting. ACM Trans. Graph. 27(3), 16 (2008)

    Article  Google Scholar 

  14. Rubinstein, M., Shamir, A., Avidan, S.: Multi-operator media retargeting. ACM Trans. Graph. 28(3), 1–11 (2009)

    Article  Google Scholar 

  15. Rubinstein, M., Gutierrez, D., Sorkine, O., Shamir, A.: A comparative study of image retargeting. ACM Trans. Graph. 29, 160 (2010)

    Article  Google Scholar 

  16. Saad, Y., Saad, Y.: Iterative Methods for Sparse Linear Systems vol. 620. PWS Publishing Company, Boston (1996)

    MATH  Google Scholar 

  17. Santella, A., Agrawala, M., DeCarlo, D., Salesin, D., Cohen, M.: Gaze-based interaction for semi-automatic photo cropping. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 771–780. ACM, New York (2006)

    Chapter  Google Scholar 

  18. Si, H., TetGen, A.: A Quality Tetrahedral Mesh Generator and Three-Dimensional Delaunay Triangulator. Weierstrass Institute for Applied Analysis and Stochastic, Berlin (2006)

    Google Scholar 

  19. Simakov, D., Caspi, Y., Shechtman, E., Irani, M.: Summarizing visual data using bidirectional similarity. In: CVPR 2008, pp. 1–8. IEEE Press, New York (2008)

    Google Scholar 

  20. Suh, B., Ling, H., Bederson, B., Jacobs, D.: Automatic thumbnail cropping and its effectiveness. In: ACM Symposium on User Interface Software and Technology, pp. 95–104. ACM, New York (2003)

    Google Scholar 

  21. Wang, D., Li, G., Jia, W., Luo, X.: Saliency-driven scaling optimization for image retargeting. Vis. Comput. 27(9), 853–860 (2011)

    Article  Google Scholar 

  22. Wang, Y., Fu, H., Sorkine, O., Lee, T., Seidel, H.: Motion-aware temporal coherence for video resizing. ACM Trans. Graph. 28(5), 1–10 (2009)

    Google Scholar 

  23. Wang, Y., Lin, H., Sorkine, O., Lee, T.: Motion-based video retargeting with optimized crop-and-warp. In: ACM SIGGRAPH 2010 Papers, pp. 1–9. ACM, New York (2010)

    Chapter  Google Scholar 

  24. Wang, Y., Hsiao, J., Sorkine, O., Lee, T.: Scalable and coherent video resizing with per-frame optimization. In: ACM Transactions on Graphics (TOG), vol. 30, p. 88. ACM, New York (2011)

    Google Scholar 

  25. Wei, L., Zhou, K., Han, J., Guo, B., Shum, H.: Inverse texture synthesis (2008)

  26. Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 463–476 (2007)

    Article  Google Scholar 

  27. Wolf, L., Guttmann, M., Cohen-Or, D.: Non-homogeneous content-driven video-retargeting. In: ACM Transactions on Graphics, pp. 1–6 (2007)

    Google Scholar 

  28. Wu, H., Feng K, W.Y., Wong, T., Lee, T., Heng, P.: Resizing by symmetry-summarization. In: ACM Transactions on Graphics (TOG), vol. 29, p. 159. ACM, New York (2010)

    Google Scholar 

  29. Xiao, C., Liu, S., Fu, H., Lin, C., Song, C., Huang, Z., He, F., Peng, Q.: Video completion and synthesis. Comput. Animat. Virtual Worlds 19(3–4), 341–353 (2008)

    Article  Google Scholar 

  30. Xiao, C., Liu, M., Yongwei, N., Dong, Z.: Fast exact nearest patch matching for patch-based image editing and processing. IEEE Trans. Vis. Comput. Graph. 17(8), 1122–1134 (2011)

    Article  Google Scholar 

  31. Zhang, G.X., Cheng, M.M., Hu, S.M., Martin, R.R.: A shape-preserving approach to image resizing. In: Computer Graphics Forum, Wiley Online Library, vol. 28, pp. 1897–1906 (2009)

    Google Scholar 

  32. Zhang, Y., Hu, S., Martin, R.: Shrinkability maps for content-aware video resizing. In: Computer Graphics Forum, Wiley Online Library, vol. 27, pp. 1797–1804 (2008)

    Google Scholar 

Download references

Acknowledgements

This work was partly supported by the National Basic Research Program of China (No. 2012CB725303), the NSFC (Nos. 61070081 and 41271431), the Open Project Program of the State Key Lab of CAD&CG (Grant No. A1208), the Luojia Outstanding Young Scholar Program of Wuhan University, the Project of the Science and Technology Plan for Zhejiang Province (Grant No. 2012C21004), the Fundamental Research Funds for the Central Universities, and the Academic Award for Excellent Ph.D. Candidates funded by Ministry of Education of China (No. 5052012211001).

Author information

Authors and Affiliations

  1. Computer School, Wuhan University, Wuhan, China

    Yongwei Nie, Qing Zhang & Chunxia Xiao

  2. College of Computer Science and Information Technology, Zhejiang Wanli University, Zhejiang, China

    Renfang Wang

Authors
  1. Yongwei Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Renfang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chunxia Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chunxia Xiao.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nie, Y., Zhang, Q., Wang, R. et al. Video retargeting combining warping and summarizing optimization. Vis Comput 29, 785–794 (2013). https://doi.org/10.1007/s00371-013-0830-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-013-0830-4

Keywords

Search

Navigation