Matthias Springstein
ABSTRACT
In this paper, we introduce iART: an open Web platform for art-historical research that facilitates the process of comparative vision. The system integrates various machine learning techniques for keyword- and content-based image retrieval as well as category formation via clustering. An intuitive GUI supports users to define queries and explore results. By using a state-of-the-art cross-modal deep learning approach, it is possible to search for concepts that were not previously detected by trained classification models. Art-historical objects from large, openly licensed collections such as Amsterdam Rijksmuseum and Wikidata are made available to users.
Supplemental Material
- Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. CoRR abs/1603.04467 (2016). arXiv:1603.04467 http://arxiv.org/abs/1603.04467Google Scholar
- Matthias Becker, Martin Bogner, Fabian Bross, François Bry, Caterina Campanella, Laura Commare, Silvia Cramerotti, Katharina Jakob, Martin Josko, Fabian Kneißl, Hubertus Kohle, Thomas Krefeld, Elena Levushkina, Stephan Lücke, Alessandra Puglisi, Anke Regner, Christian Riepl, Clemens Schefels, Corina Schemainda, Eva Schmidt, Stefanie Schneider, Gerhard Schön, Klaus Schulz, Franz Siglmüller, Bartholomäus Steinmayr, Florian Störkle, Iris Teske, and Christoph Wieser. 2018. ARTigo -- Social Image Tagging [Dataset and Images]. https://doi.org/10.5282/ubm/data.136..Google Scholar
- Reiner Diedrichs. 2021. Kenom Digitaler Münzkatalog. Retrieved June 15, 2021 from https://www.kenom.deGoogle Scholar
- Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/f3ada80d5c4ee70142b17b8192b2958e-Abstract.htmlGoogle Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016. 770--778. https://doi.org/10.1109/CVPR.2016.90Google Scholar
- Iconclass. 2021. Iconclass. Retrieved June 15, 2021 from http://www.iconclass.orgGoogle Scholar
- Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-Scale Similarity Search with GPUs. CoRR (2017). http://arxiv.org/abs/1702.08734Google Scholar
- Sabine Lang and Björn Ommer. 2018. Attesting Similarity: Supporting the Organization and Study of Art Image Collections with Computer Vision. Digital Schol- arship in the Humanities 33, 4 (2018), 845--856. https://doi.org/10.1093/llc/fqy006Google Scholar
- Leland McInnes and John Healy. 2018. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. CoRR (2018). http://arxiv.org/abs/1802. 03426Google Scholar
- Kiri Nichol. 2016. Painter by Numbers. Retrieved June 15, 2021 from https://www.kaggle.com/c/painter-by-numbersGoogle Scholar
- The Metropolitan Museum of Art. 2020. iMet Collection 2020 - FGVC7. Retrieved June 15, 2021 from https://www.kaggle.com/c/imet-2020-fgvc7Google Scholar
- Fabian Offert, Peter Bell, and Oleg Harlamov. 2020. imgs.ai. https://imgs.ai/. Accessed: 2021-06--15.Google Scholar
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 8024--8035. https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html Google ScholarDigital Library
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. CoRR (2021). https://arxiv.org/abs/2103.00020Google Scholar
- Rijksmuseum. 2021. Rijksmuseum Amsterdam, Home of the Dutch masters. Retrieved June 15, 2021 from https://www.rijksmuseum.nl/enGoogle Scholar
- Luca Rossetto, Ivan Giangreco, Claudiu Tanase, and Heiko Schuldt. 2016. vitrivr: A Flexible Retrieval Stack Supporting Multiple Query Modes for Searching in Multimedia Collections. In 24th ACM International Conference on Multimedia. 1183--1186. https://dl.acm.org/doi/10.1145/2964284.2973797 Google ScholarDigital Library
- Christoph Wieser, François Bry, Alexandre Bérard, and Richard Lagrange. 2013. ARTigo: Building an Artwork Search Engine with Games and Higher-Order Latent Semantic Analysis. In Disco 2013, Workshop on Human Computation and Machine Learning in Games at the International Conference on Human Computation (HComp). https://www.en.pms.ifi.lmu.de/publications/PMS-FB/PMS-FB-2013-3/PMS-FB-2013-3-paper.pdf[18] Wikimedia. 2019. Wikidata. Retrieved June 15, 2021 from https://www.wikidata.org/wiki/Wikidata:Main_PageGoogle Scholar
- Heinrich Wölfflin. 1915. Kunstgeschichtliche Grundbegriffe. Bruckmann, Munich.Google Scholar
Index Terms
- iART: A Search Engine for Art-Historical Images to Support Research in the Humanities
Recommendations
Relevance-based Margin for Contrastively-trained Video Retrieval Models
ICMR '22: Proceedings of the 2022 International Conference on Multimedia RetrievalVideo retrieval using natural language queries has attracted increasing interest due to its relevance in real-world applications, from intelligent access in private media galleries to web-scale video search. Learning the cross-similarity of video and ...
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders
Despite the evolution of deep-learning-based visual-textual processing systems, precise multi-modal matching remains a challenging task. In this work, we tackle the task of cross-modal retrieval through image-sentence matching based on word-region ...
IR Questioner: QA-based Interactive Retrieval System
ICMR '21: Proceedings of the 2021 International Conference on Multimedia RetrievalImage retrieval from a given text query (text-to-image retrieval) is one of the most essential systems, and it is effectively utilized for databases (DBs) on the Web. To make them more versatile and familiar, a retrieval system that is adaptive even for ...
Comments