Amar Viswanathan Kannan
Mohammad Abdullah Al Faruque
ABSTRACT
Keeping up with the rapid growth of Deep Learning (DL) research is a daunting task. While existing scientific literature search systems provide text search capabilities and can identify similar papers, gaining an in-depth understanding of a new approach or an application is much more complicated. Many publications leverage multiple modalities to convey their findings and spread their ideas - they include pseudocode, tables, images and diagrams in addition to text, and often make publicly accessible their implementations. It is important to be able to represent and query them as well. We utilize RDF Knowledge graphs (KGs) to represent multimodal information and enable expressive querying over modalities. In our demo we present an approach for extracting KGs from different modalities, namely text, architecture images and source code. We show how graph queries can be used to get insights into different facets (modalities) of a paper, and its associated code implementation. Our innovation lies in the multimodal nature of the KG we create. While our work is of direct interest to DL researchers and practitioners, our approaches can also be leveraged in other scientific domains.
Supplemental Material
- Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2017. Learning to Represent Programs with Graphs. arxiv: cs.LG/1711.00740Google Scholar
- Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, et almbox. 2018. Construction of the literature graph in semantic scholar. arXiv preprint arXiv:1805.02262 (2018).Google Scholar
- Davide Buscaldi, Danilo Dess`i, Enrico Motta, Francesco Osborne, and Diego Reforgiato Recupero. 2019. Mining scholarly publications for scientific knowledge graph construction. In European Semantic Web Conference. Springer, 8--12.Google Scholar
- Natthawut Kertkeidkachorn and Ryutaro Ichise. 2017. T2KG: An End-to-End System for Creating Knowledge Graph from Unstructured Text. In AAAI Workshops (AAAI Workshops), Vol. WS-17. AAAI Press. http://dblp.uni-trier.de/db/conf/aaai/aaai2017w.html#Kertkeidkachorn17Google Scholar
- Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. arXiv preprint arXiv:1808.09602 (2018).Google Scholar
- Makoto Miwa and Mohit Bansal. 2016. End-to-End Relation Extraction using LS™s on Sequences and Tree Structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 1105--1116. https://doi.org/10.18653/v1/P16-1105Google ScholarCross Ref
- Anderson Rossanez and Julio Cesar dos Reis. 2019. Generating Knowledge Graphs from Scientific Literature of Degenerative Diseases. (2019).Google Scholar
- Aditi Roy, Ioannis Akrotirianakis, Amar V. Kannan, Dmitriy Fradkin, Arquimedes Canedo, Kaushik Koneripalli, and Tugba Kulahcioglu. 2020. Diag2Graph: Representing Deep Learning Diagrams in Research Papers as Knowledge Graphs. In IEEE International Conference on Image Processing.Google ScholarCross Ref
- Guus Schreiber and Yves Raimond. 2014. RDF 1.1 Primer. W3C Working Group Note, Vol. 25 (2014).Google Scholar
- Shih-Yuan Yu, Ahmet Salih Aksakal, Sujit Rokka Chhetri, and Mohammad Abdullah Al Faruque. 2020. Deep Code Curator -- code2graph Part-II. Technical Report TR-20-01. Center for Embedded and Cyber-Physical Systems University of California, Irvine, Irvine, CA 92697--2620, USA. http://cecs.uci.edu/files/2019/05/TR-19-01.pdfGoogle Scholar
Index Terms
- Multimodal Knowledge Graph for Deep Learning Papers and Code
Recommendations
Benchmark and Neural Architecture for Conversational Entity Retrieval from a Knowledge Graph
WWW '24: Proceedings of the ACM on Web Conference 2024This paper introduces a novel information retrieval (IR) task of Conversational Entity Retrieval from a Knowledge Graph (CER-KG), which extends non-conversational entity retrieval from a knowledge graph (KG) to the conversational scenario. The user ...
Knowledge Graph Completion by Context-Aware Convolutional Learning with Multi-Hop Neighborhoods
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge ManagementThe main focus of relational learning for knowledge graph completion (KGC) lies in exploiting rich contextual information for facts. Many state-of-the-art models incorporate fact sequences, entity types, and even textual information. Unfortunately, most ...
Bidirectional Joint Representation Learning with Symmetrical Deep Neural Networks for Multimodal and Crossmodal Applications
ICMR '16: Proceedings of the 2016 ACM on International Conference on Multimedia RetrievalCommon approaches to problems involving multiple modalities (classification, retrieval, hyperlinking, etc.) are early fusion of the initial modalities and crossmodal translation from one modality to the other. Recently, deep neural networks, especially ...
Comments