Skip to main content
SpringerLink
Log in
Book cover

European Conference on Computer Vision

ECCV 2022: Computer Vision – ECCV 2022 Workshops pp 298–313Cite as

ArtFacePoints: High-Resolution Facial Landmark Detection in Paintings and Prints

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13801))

Abstract

Facial landmark detection plays an important role for the similarity analysis in artworks to compare portraits of the same or similar artists. With facial landmarks, portraits of different genres, such as paintings and prints, can be automatically aligned using control-point-based image registration. We propose a deep-learning-based method for facial landmark detection in high-resolution images of paintings and prints. It divides the task into a global network for coarse landmark prediction and multiple region networks for precise landmark refinement in regions of the eyes, nose, and mouth that are automatically determined based on the predicted global landmark coordinates. We created a synthetically augmented facial landmark art dataset including artistic style transfer and geometric landmark shifts. Our method demonstrates an accurate detection of the inner facial landmarks for our high-resolution dataset of artworks while being comparable for a public low-resolution artwork dataset in comparison to competing methods.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: YOLOv4: optimal speed and accuracy of object detection (2020). https://doi.org/10.48550/ARXIV.2004.10934

  2. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1021–1030 (2017). https://doi.org/10.1109/ICCV.2017.116

  3. Chandran, P., Bradley, D., Gross, M., Beeler, T.: Attention-driven cropping for very high resolution facial landmark detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5860–5869 (2020). https://doi.org/10.1109/CVPR42600.2020.00590

  4. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981). https://doi.org/10.1145/358669.358692

    Article  MathSciNet  Google Scholar 

  5. Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style (2015). https://doi.org/10.48550/ARXIV.1508.06576

  6. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  7. He, K., Xue, X.: Facial landmark localization by part-aware deep convolutional network. Adv. Multimedia Inf. Process. - PCM 2016, 22–31 (2016). https://doi.org/10.1007/978-3-319-48890-5_3

    Article  Google Scholar 

  8. Honari, S., Molchanov, P., Tyree, S., Vincent, P., Pal, C., Kautz, J.: Improving landmark localization with semi-supervised learning. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1546–1555 (2018). https://doi.org/10.1109/CVPR.2018.00167

  9. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1501–1510 (2017). https://doi.org/10.1109/ICCV.2017.167

  10. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  11. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1867–1874 (2014). https://doi.org/10.1109/CVPR.2014.241

  12. Kordon, F., Maier, A., Kunze, H.: Latent shape constraint for anatomical landmark detection on spine radiographs. Bildverarbeitung für die Medizin 2021, 350–355 (2021). https://doi.org/10.1007/978-3-658-33198-6_85

    Article  Google Scholar 

  13. Kowalski, M., Naruniec, J., Trzcinski, T.: Deep alignment network: a convolutional neural network for robust face alignment. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2034–2043 (2017). https://doi.org/10.1109/CVPRW.2017.254

  14. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  15. Madhu, P., et al.: Enhancing Human Pose Estimation in Ancient Vase Paintings via Perceptually-grounded Style Transfer Learning. arXiv:2012.05616 (2020)

  16. Nibali, A., He, Z., Morgan, S., Prendergast, L.: Numerical Coordinate Regression with Convolutional Neural Networks. arXiv:1801.07372 (2018)

  17. Nichol, K.: Painter by Numbers, WikiArt (2016). https://www.kaggle.com/c/painter-by-numbers

  18. Robinson, J.P., Li, Y., Zhang, N., Fu, Y., Tulyakov, S.: Laplace landmark localization. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10102–10111 (2019). https://doi.org/10.1109/ICCV.2019.01020

  19. Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: 2013 IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 397–403 (2013). https://doi.org/10.1109/ICCVW.2013.59

  20. Sindel, A., Maier, A., Christlein, V.: Art2Contour: salient contour detection in artworks using generative adversarial networks. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 788–792 (2020). https://doi.org/10.1109/ICIP40778.2020.9191117

  21. Stricker, M., Augereau, O., Kise, K., Iwata, M.: Facial Landmark Detection for Manga Images (2018). https://doi.org/10.48550/ARXIV.1811.03214

  22. Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5686–5696 (2019). https://doi.org/10.1109/CVPR.2019.00584

  23. Sun, K., et al.: High-resolution representations for labeling pixels and regions (2019). https://doi.org/10.48550/ARXIV.1904.04514

  24. Sun, Y., Wang, X., Tang, X.: Deep convolutional network cascade for facial point detection. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3476–3483 (2013). https://doi.org/10.1109/CVPR.2013.446

  25. Wang, X., Bo, L., Fuxin, L.: Adaptive wing loss for robust face alignment via heatmap regression. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6970–6980 (2019). https://doi.org/10.1109/ICCV.2019.00707

  26. Wu, Y., Ji, Q.: Facial landmark detection: a literature survey. Int. J. Comput. Vision 127(2), 115–142 (2018). https://doi.org/10.1007/s11263-018-1097-z

    Article  Google Scholar 

  27. Yaniv, J., Newman, Y., Shamir, A.: The face of art: landmark detection and geometric style in portraits. ACM Trans. Graph. 38(4) (2019). https://doi.org/10.1145/3306346.3322984

  28. Zhang, Z., Luo, P., Loy, C.C., Tang, X.: Facial landmark detection by deep multi-task learning. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 94–108. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_7

    Chapter  Google Scholar 

  29. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017). https://doi.org/10.1109/ICCV.2017.244

Download references

Acknowledgement

Thanks to Daniel Hess, Oliver Mack, Daniel Görres, Wibke Ottweiler, GNM, and Gunnar Heydenreich, CDA, and Thomas Klinke, TH Köln, and Amalie Hänsch, FAU Erlangen-Nürnberg for providing image data, and to Leibniz Society for funding the research project “Critical Catalogue of Luther portraits (1519–1530)” with grant agreement No. SAW-2018-GNM-3-KKLB, to the European Union’s Horizon 2020 research and innovation programme within the Odeuropa project under grant agreement No. 101004469 for funding this publication, and to NVIDIA for their GPU hardware donation.

Author information

Authors and Affiliations

  1. Pattern Recognition Lab, FAU Erlangen-Nürnberg, Erlangen, Germany

    Aline Sindel, Andreas Maier & Vincent Christlein

Authors
  1. Aline Sindel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andreas Maier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vincent Christlein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aline Sindel .

Editor information

Editors and Affiliations

  1. IBM Research AI and MIT-IBM Watson AI Lab, Haifa, Israel

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sindel, A., Maier, A., Christlein, V. (2023). ArtFacePoints: High-Resolution Facial Landmark Detection in Paintings and Prints. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13801. Springer, Cham. https://doi.org/10.1007/978-3-031-25056-9_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25056-9_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25055-2

  • Online ISBN: 978-3-031-25056-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation