Skip to main content
SpringerLink
Log in

NEO-3DF: Novel Editing-Oriented 3D Face Creation and Reconstruction

  • Conference paper
  • First Online:
Computer Vision – ACCV 2022 (ACCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13841))

Included in the following conference series:

Abstract

Unlike 2D face images, obtaining a 3D face is not easy. Existing methods, therefore, create a 3D face from a 2D face image (3D face reconstruction). A user might wish to edit the reconstructed 3D face, but 3D face editing has seldom been studied. This paper presents such method and shows that reconstruction and editing can help each other. In the presented framework named NEO-3DF, the 3D face model we propose has independent sub-models corresponding to semantic face parts. It allows us to achieve both local intuitive editing and better 3D-to-2D alignment. Each face part in our model has a set of controllers designed to allow users to edit the corresponding features (e.g., nose height). In addition, we propose a differentiable module for blending the face parts and making it possible to automatically adjust the face parts (both the shapes and the locations) so that they are better aligned with the original 2D image. Experiments show that the results of NEO-3DF outperform existing methods in intuitive face editing and have better 3D-to-2D alignment accuracy (14% higher IoU) than global face model-based reconstruction. Code available at https://github.com/ubc-3d-vision-lab/NEO-3DF.

J. Gregson—This work was done when James Gregson was at Huawei Technologies Canada.

S. Du—This work was supported by the University of British Columbia (Okanagan) [GR017752].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Allen, B., Curless, B., Popović, Z.: The space of human body shapes: reconstruction and parameterization from range scans. ACM Trans. Graph. (TOG) 22(3), 587–594 (2003)

    Article  Google Scholar 

  2. Bai, Z., Cui, Z., Liu, X., Tan, P.: Riggable 3D face reconstruction via in-network optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6216–6225 (2021)

    Google Scholar 

  3. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194 (1999)

    Google Scholar 

  4. Bouritsas, G., Bokhnyak, S., Ploumpis, S., Bronstein, M., Zafeiriou, S.: Neural 3D morphable models: spiral convolutional networks for 3D shape representation learning and generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7213–7222 (2019)

    Google Scholar 

  5. Chang, F.J., Tran, A.T., Hassner, T., Masi, I., Nevatia, R., Medioni, G.: Expnet: landmark-free, deep, 3D facial expressions. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 122–129. IEEE (2018)

    Google Scholar 

  6. Chen, A., Chen, Z., Zhang, G., Mitchell, K., Yu, J.: Photo-realistic facial details synthesis from single image. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9429–9439 (2019)

    Google Scholar 

  7. Deng, Y., Yang, J., Chen, D., Wen, F., Tong, X.: Disentangled and controllable face image generation via 3D imitative-contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5154–5163 (2020)

    Google Scholar 

  8. Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., Tong, X.: Accurate 3D face reconstruction with weakly-supervised learning: from single image to image set. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)

    Google Scholar 

  9. Egger, B., et al.: 3D morphable face models-past, present, and future. ACM Trans. Graph. (TOG) 39(5), 1–38 (2020)

    Article  Google Scholar 

  10. Farkas, L.G., Kolar, J.C., Munro, I.R.: Geography of the nose: a morphometric study. Aesthetic Plastic Surg. 10(1), 191–223 (1986)

    Article  Google Scholar 

  11. Feng, Y., Feng, H., Black, M.J., Bolkart, T.: Learning an animatable detailed 3D face model from in-the-wild images. ACM Trans. Graph. (ToG) 40(4), 88:1–88:13 (2021)

    Google Scholar 

  12. Feng, Y., Wu, F., Shao, X., Wang, Y., Zhou, X.: Joint 3D face reconstruction and dense alignment with position map regression network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 534–551 (2018)

    Google Scholar 

  13. Foti, S., Koo, B., Stoyanov, D., Clarkson, M.J.: 3D shape variational autoencoder latent disentanglement via mini-batch feature swapping for bodies and faces. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18730–18739 (2022)

    Google Scholar 

  14. Ghafourzadeh, D., et al.: Local control editing paradigms for part-based 3D face morphable models. Comput. Anim. Virt. Worlds 32(6), e2028 (2021)

    Google Scholar 

  15. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)

    Google Scholar 

  16. Kesterke, M.J., et al.: Using the 3D facial norms database to investigate craniofacial sexual dimorphism in healthy children, adolescents, and adults. Biol. Sex Differ. 7(1), 1–14 (2016)

    Article  Google Scholar 

  17. Koizumi, T., Smith, W.A.: Shape from semantic segmentation via the geometric rényi divergence. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2312–2321 (2021)

    Google Scholar 

  18. Lattas, A., et al.: Avatarme: realistically renderable 3D facial reconstruction “in-the-wild". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 760–769 (2020)

    Google Scholar 

  19. Le, B.H., Deng, Z.: Interactive cage generation for mesh deformation. In: Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, pp. 1–9 (2017)

    Google Scholar 

  20. Lee, C.H., Liu, Z., Wu, L., Luo, P.: Maskgan: towards diverse and interactive facial image manipulation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  21. Lee, G.H., Lee, S.W.: Uncertainty-aware mesh decoder for high fidelity 3D face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6100–6109 (2020)

    Google Scholar 

  22. Lewis, J.P., Anjyo, K., Rhee, T., Zhang, M., Pighin, F.H., Deng, Z.: Practice and theory of blendshape facial models. Eurograph. (State Art Rep.) 1(8), 2 (2014)

    Google Scholar 

  23. Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4D scans. ACM Trans. Graph. (TOG) 36(6), 1–17 (2017)

    Google Scholar 

  24. Lin, J., Yuan, Y., Shao, T., Zhou, K.: Towards high-fidelity 3D face reconstruction from in-the-wild images using graph convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5891–5900 (2020)

    Google Scholar 

  25. Martyniuk, T., Kupyn, O., Kurlyak, Y., Krashenyi, I., Matas, J., Sharmanska, V.: Dad-3dheads: a large-scale dense, accurate and diverse dataset for 3D head alignment from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20942–20952 (2022)

    Google Scholar 

  26. Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296–301. IEEE (2009)

    Google Scholar 

  27. Piao, J., Sun, K., Wang, Q., Lin, K.Y., Li, H.: Inverting generative adversarial renderer for face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619–15628 (2021)

    Google Scholar 

  28. Ramanathan, N., Chellappa, R.: Modeling age progression in young faces. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 1, pp. 387–394. IEEE (2006)

    Google Scholar 

  29. Ranjan, A., Bolkart, T., Sanyal, S., Black, M.J.: Generating 3D faces using convolutional mesh autoencoders. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 704–720 (2018)

    Google Scholar 

  30. Rhee, S.C., Woo, K.S., Kwon, B.: Biometric study of eyelid shape and dimensions of different races with references to beauty. Aesthetic Plastic Surg. 36(5), 1236–1245 (2012)

    Article  Google Scholar 

  31. Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)

    Google Scholar 

  32. Shi, T., Yuan, Y., Fan, C., Zou, Z., Shi, Z., Liu, Y.: Face-to-parameter translation for game character auto-creation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 161–170 (2019)

    Google Scholar 

  33. Sorkine, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Symposium on Geometry processing, vol. 4, pp. 109–116 (2007)

    Google Scholar 

  34. Tena, J.R., De la Torre, F., Matthews, I.: Interactive region-based linear 3D face models. In: ACM SIGGRAPH 2011 Papers, pp. 1–10. ACM (2011)

    Google Scholar 

  35. Tran, L., Liu, X.: Nonlinear 3D face morphable model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7346–7355 (2018)

    Google Scholar 

  36. Vetter, T., Blanz, V.: Estimating coloured 3D face models from single images: an example based approach. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 499–513. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0054761

    Chapter  Google Scholar 

  37. Wood, E., Baltrusaitis, T., Hewitt, C., Dziadzio, S., Cashman, T.J., Shotton, J.: Fake it till you make it: face analysis in the wild using synthetic data alone. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3681–3691 (2021)

    Google Scholar 

  38. Zhu, W., Wu, H., Chen, Z., Vesdapunt, N., Wang, B.: Reda: reinforced differentiable attribute for 3D face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4958–4967 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The University of British Columbia, Vancouver, BC, Canada

    Peizhi Yan & Rabab Ward

  2. Huawei Technologies Canada, Burnaby, BC, Canada

    James Gregson, Qiang Tang & Zhan Xu

  3. The University of British Columbia (Okanagan), Kelowna, BC, Canada

    Shan Du

Authors
  1. Peizhi Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James Gregson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiang Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rabab Ward
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shan Du
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shan Du .

Editor information

Editors and Affiliations

  1. University of Wollongong, Wollongong, NSW, Australia

    Lei Wang

  2. University of Bonn, Bonn, Germany

    Juergen Gall

  3. University of Adelaide, Adelaide, SA, Australia

    Tat-Jun Chin

  4. National Institute of Informatics, Tokyo, Japan

    Imari Sato

  5. Johns Hopkins University, Baltimore, MD, USA

    Rama Chellappa

1 Electronic supplementary material

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yan, P., Gregson, J., Tang, Q., Ward, R., Xu, Z., Du, S. (2023). NEO-3DF: Novel Editing-Oriented 3D Face Creation and Reconstruction. In: Wang, L., Gall, J., Chin, TJ., Sato, I., Chellappa, R. (eds) Computer Vision – ACCV 2022. ACCV 2022. Lecture Notes in Computer Science, vol 13841. Springer, Cham. https://doi.org/10.1007/978-3-031-26319-4_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-26319-4_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-26318-7

  • Online ISBN: 978-3-031-26319-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation