Abstract
Large datasets that were made publicly available to the research community over the last 20 years have been a key enabling factor for the advances in deep learning algorithms for NLP or computer vision. These datasets are generally pairs of aligned image/manually annotated metadata, where images are photographs of everyday life. Scholarly and historical content, on the other hand, treat subjects that are not necessarily popular to a general audience, they may not always contain a large number of data points, and new data may be difficult or impossible to collect. Some exceptions do exist, for instance, scientific or health data, but this is not the case for cultural heritage (CH). The poor performance of the best models in computer vision - when tested over artworks - coupled with the lack of extensively annotated datasets for CH, and the fact that artwork images depict objects and actions not captured by photographs, indicate that a CH-specific dataset would be highly valuable for this community. We propose DEArt, at this point primarily an object detection and pose classification dataset meant to be a reference for paintings between the XIIth and the XVIIIth centuries. It contains more than 15000 images, about 80% non-iconic, aligned with manual annotations for the bounding boxes identifying all instances of 69 classes as well as 12 possible poses for boxes identifying human-like objects. Of these, more than 50 classes are CH-specific and thus do not appear in other datasets; these reflect imaginary beings, symbolic entities and other categories related to art. Additionally, existing datasets do not include pose annotations. Our results show that object detectors for the cultural heritage domain can achieve a level of precision comparable to state-of-art models for generic images via transfer learning.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Other filters may be implemented, e.g. testing the number of artworks in a category rather than the number of subcategories.
- 2.
References
Aranganayagi, S., Thangavel, K.: Clustering categorical data using silhouette coefficient as a relocating measure. In: International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007). vol. 2, pp. 13–17. IEEE (2007)
Berg, T.L., Berg, A.C.: Finding iconic images. 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8 (2009)
Carneiro, G., da Silva, N.P., Del Bue, A., Costeira, J.P.: Artistic image classification: an analysis on the PRINTART database. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 143–157. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33765-9_11
Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)
Clark, K., Manning, C.D.: Deep reinforcement learning for mention-ranking coreference models. arXiv preprint arXiv:1609.08667 (2016)
Dictionary.PDF, W.: The merriam webster dictionary. In: The Merriam Webster Dictionary (2016)
Everingham, M., Gool, L.V., Williams, C.K.I., Winn, J.M., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vision 88, 303–338 (2009)
Fiorucci, M., Khoroshiltseva, M., Pontil, M., Traviglia, A., Bue, A.D., James, S.: Machine learning for cultural heritage: a survey. Pattern Recognit. Lett. 133, 102–108 (2020)
Garcia, N., Vogiatzis, G.: How to read paintings: semantic art understanding with multi-modal retrieval. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11130, pp. 676–691. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11012-3_52
Ginosar, S., Haas, D., Brown, T., Malik, J.: Detecting people in cubist art. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8925, pp. 101–116. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16178-5_7
Gonthier, N., Gousseau, Y., Ladjal, S., Bonfait, O.: Weakly supervised object detection in artworks. ArXiv abs/1810.02569 (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Honnibal, M., Montani, I.: spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing (2017, to appear)
Huang, X., Belongie, S.J.: Arbitrary style transfer in real-time with adaptive instance normalization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1510–1519 (2017)
Kadish, D., Risi, S., Løvlie, A.S.: Improving object detection in art images using only style transfer. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2021)
Khosla, A., Yao, B., Fei-Fei, L.: Integrating randomization and discrimination for classifying human-object interaction activities. In: Fu, Y. (ed.) Human-Centered Social Media Analytics, pp. 95–114. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-05491-9_5
Kuznetsova, A., et al.: The open images dataset v4. Int. J. Comput. Vision 128, 1956–1981 (2020)
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Maji, S., Bourdev, L., Malik, J.: Action recognition from a distributed representation of pose and appearance. In: CVPR 2011, pp. 3177–3184. IEEE (2011)
Milani, F., Fraternali, P.: A dataset and a convolutional model for iconography classification in paintings. J. Comput. Cultural Herit. 14, 1–18 (2021)
Palmer, S., Rosch, E., Chase, P.: Canonical perspective and the perception of objects. Attention and performance IX, pp. 135–151 (1981)
Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP), pp. 1532–1543 (2014)
Ren, S., He, K., Girshick, R.B., Sun, J.: Faster r-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39, 1137–1149 (2015)
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115, 211–252 (2015)
Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: Labelme: a database and web-based tool for image annotation. Int. J. Comput. Vision 77, 157–173 (2007)
Strezoski, G., Worring, M.: Omniart: a large-scale artistic benchmark. ACM Trans. Multim. Comput. Commun. Appl. 14, 88:1–88:21 (2018)
Westlake, N., Cai, H., Hall, P.: Detecting people in artwork with cnns. ArXiv abs/1610.08871 (2016)
Yarlagadda, P., Monroy, A., Carqué, B., Ommer, B.: Recognition and analysis of objects in medieval images. In: ACCV Workshops (2010)
Ypsilantis, N.A., García, N., Han, G., Ibrahimi, S., van Noord, N., Tolias, G.: The met dataset: Instance-level recognition for artworks. ArXiv abs/2202.01747 (2021)
Acknowledgement
This research has been supported by the Saint George on a Bike project 2018-EU-IA-0104, co-financed by the Connecting Europe Facility of the European Union.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Reshetnikov, A., Marinescu, MC., Lopez, J.M. (2023). DEArt: Dataset of European Art. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13801. Springer, Cham. https://doi.org/10.1007/978-3-031-25056-9_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-25056-9_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25055-2
Online ISBN: 978-3-031-25056-9
eBook Packages: Computer ScienceComputer Science (R0)