Skip to main content
SpringerLink
Log in

Video style transfer by consistent adaptive patch sampling

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

This paper addresses the example-based stylization of videos. Style transfer aims at editing an image so that it matches the style of an example. This topic has been recently investigated by several researchers, both in the industry and in academia. The difficulty lies in how to capture the style of an image and correctly transferring it to a video. In this paper, we build on our previous work “Split and Match” for still pictures, based on adaptive patch synthesis. We address the issue of extending that particular technique to video, ensuring that the solution is spatially and temporally consistent. Results show that our video style transfer is visually plausible, while being very competitive regarding computation time and memory when compared to neural network approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://join.lovingvincent.com/.

  2. https://deepart.io/.

  3. http://prisma-ai.com/.

References

  1. Durand, F.: An invitation to discuss computer depiction. In: NPAR, New York, NY, USA, pp. 111–124 (2002). https://doi.org/10.1145/508530.508550

  2. Kyprianidis, J., Collomosse, J., Wang, T., Isenberg, T.: State of the art: a taxonomy of artistic stylization techniques for images and video. IEEE Trans. Vis. Comput. Graph. 19(5), 866–885 (2013)

    Article  Google Scholar 

  3. Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. CoRR. arXiv:1508.06576

  4. Li, C., Wand, M.: Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: European Conference on Computer Vision, pp. 702–716. Springer (2016)

  5. Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. In: ICLR (2017)

  6. Frigo, O., Sabater, N., Delon, J., Hellier, P.: Split and match: example-based adaptive patch sampling for unsupervised style transfer. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June (2016)

  7. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: SIGGRAPH, New York, NY, USA, pp. 341–346 (2001). https://doi.org/10.1145/383259.383296

  8. Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  9. Freeman, W., Jones, T., Pasztor, E.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)

    Article  Google Scholar 

  10. Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: SIGGRAPH ’01 Proceedings of the 28th annual conference on Computer graphics and interactive techniques, New York, NY, USA, pp. 327–340 (2001). https://doi.org/10.1145/383259.383295

  11. Reinhard, E., Ashikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001). https://doi.org/10.1109/38.946629

    Article  Google Scholar 

  12. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: ICCV, Washington, DC, USA, pp. 1033 (1999). http://dl.acm.org/citation.cfm?id=850924.851569

  13. Bénard, P., Cole, F., Kass, M., Mordatch, I., Hegarty, J., Senn, M.S., Fleischer, K., Pesare, D., Breeden, K.: Stylizing animation by example. ACM Trans. Graph. 32(4), 1191–11912 (2013). https://doi.org/10.1145/2461912.2461929

    Article  MATH  Google Scholar 

  14. Barnes, C., Shechtman, E., Goldman, D., Finkelstein, A.: The generalized patchmatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) Computer Vision ECCV 2010, Series Lecture Notes in Computer Science, vol. 6313, pp. 29–43. Springer, Berlin (2010). https://doi.org/10.1007/978-3-642-15558-1_3

    Chapter  Google Scholar 

  15. Liang, L., Liu, C., Xu, Y.-Q., Guo, B., Shum, H.-Y.: Real-time texture synthesis by patch-based sampling. ACM Trans. Graph. 20(3), 127–150 (2001). https://doi.org/10.1145/501786.501787

    Article  Google Scholar 

  16. Shih, Y., Paris, S., Barnes, C., Freeman, W.T., Durand, F.: Style transfer for headshot portraits. ACM Trans. Graph. 33(4), 148 (2014)

    Article  Google Scholar 

  17. Yi, Z., Li, Y., Ji, S., Gong, M.: Artistic stylization of face photos based on a single exemplar. Vis. Comput. 33(11), 1443–1452 (2017). https://doi.org/10.1007/s00371-016-1290-4

    Article  Google Scholar 

  18. Freeman, W., Pasztor, E., Carmichael, O.: Learning low-level vision. Int. J. Comput. Vis. 40(1), 25–47 (2000)

    Article  MATH  Google Scholar 

  19. Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009). https://doi.org/10.1109/TPAMI.2008.222

    Article  Google Scholar 

  20. Weiss, Y.: Belief propagation and revision in networks with loops. Technical Report, Cambridge, MA, USA (1997)

  21. Elad, M., Milanfar, P.: Style-transfer via texture-synthesis. CoRR (2016). arXiv:1609.03057

  22. Ulyanov, D., Vedaldi, A., Lempitsky, V.S.: Instance normalization: The missing ingredient for fast stylization. CoRR (2016). arXiv:1607.08022

  23. Puy, G., Kitic, S., Pérez, P.: Unifying local and non-local signal processing with graph CNNS. CoRR (2017). arXiv:1702.07759

  24. Joshi, B.J., Stewart, K., Shapiro, D.: Bringing impressionism to life with neural style transfer in come swim. CoRR (2017). arXiv:1701.04928

  25. Farbman, Z., Lischinski, D.: Tonal stabilization of video. In: ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2011), vol. 30, no. 4, pp. 89:1–89:9 (2011)

  26. Frigo, O., Sabater, N., Delon, J., Hellier, P.: Motion driven tonal stabilization. IEEE Trans. Image Process. 25(11), 5455–5468 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  27. Bonneel, N., Tompkin, J., Sunkavalli, K., Sun, D., Paris, S., Pfister, H.: Blind video temporal consistency. In: ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2015), vol. 34, no. 6 (2015)

  28. Ruder, M., Dosovitskiy, A., Brox, T.: Artistic Style Transfer for Videos. In: Rosenhahn, B., Andres, B. (eds) Pattern Recognition. GCPR 2016. Lecture Notes in Computer Science, vol 9796. Springer, Cham (2016)

  29. Fišer, J., Jamriška, O., Lukáč, M., Shechtman, E., Asente, P., Lu, J., Sýkora, D.: Stylit: illumination-guided example-based stylization of 3D renderings. ACM Trans. Graph. 35(4), 92 (2016)

    Google Scholar 

  30. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann Publishers Inc., San Francisco (1988)

    MATH  Google Scholar 

  31. Frigo, O., Sabater, N., Demoulin, V., Hellier, P.: Optimal transportation for example-guided color transfer. Computer Vision – ACCV 2014, 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1–5, 2014, Revised Selected Papers, Part III (2014)

  32. Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: IEEE International Conference on Computer Vision (ICCV), Sydney, Australia, December (2013). http://hal.inria.fr/hal-00873592

  33. Sundaram, N., Brox, T., Keutzer, K.: Dense point trajectories by GPU-accelerated large displacement optical flow. In: European Conference on Computer Vision (ECCV), Series Lecture Notes in Computer Science. Springer, September (2010). http://lmb.informatik.uni-freiburg.de//Publications/2010/Bro10e

  34. Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., et al. (eds.) European Conference on Computer Vision (ECCV), Series Part IV, LNCS 7577, pp. 611–625. Springer, Berlin (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Technicolor R&I, 975 Avenue des Champs Blancs, 35576, Cesson-Sévigné, France

    Oriel Frigo, Neus Sabater & Pierre Hellier

  2. Université Paris Descartes, 45 Rue des Saints-Pères, Paris, France

    Julie Delon

Authors
  1. Oriel Frigo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Neus Sabater
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Julie Delon
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pierre Hellier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Oriel Frigo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Frigo, O., Sabater, N., Delon, J. et al. Video style transfer by consistent adaptive patch sampling. Vis Comput 35, 429–443 (2019). https://doi.org/10.1007/s00371-018-1474-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-018-1474-1

Keywords

Search

Navigation