Improving University Faculty Evaluations via multi-view Knowledge Graph

Highlights

• A novel university faculty evaluation system based on knowledge graph is proposed.

• We conduct entity representations through knowledge graph embedding methods.

• This paper proposes a factor (ADF) for academic development prediction.

• In our case, ADF is superior to the traditional h-index, g-index, and RG score.

Abstract

University faculties generate a large amount of heterogeneous data in e-learning environments that online systems and toolkits have made widely available in all aspects of teaching and scientific researching activities. How to use the data efficiently and scientifically for faculty evaluations has recently become an important issue in university performance systems. However, it is still a challenge to comprehensively assess faculty members using multi-source and multi-modal data due to the lack of uniform representations and evaluation processes. To this end, this paper proposes a novel University Faculty Evaluation System based on a multi-view Knowledge Graph (UFES-KG) that integrates heterogeneous faculty data. Relevant data, collected both on the Internet and through university-administered internal systems, includes faculty information such as scientific research papers, patents, funds, monographs, awards, professional activities and teaching performance. Furthermore, we construct entity representations through knowledge graph embedding methods to retain their semantic information. In addition, by integrating the academic development status of scholars in the previous three years as well as student evaluation data, this paper proposes an academic development factor (ADF) for making predictions about faculty academic development. The experimental results show that this factor is closely related to the features of the knowledge graph and student evaluations. In a certain case study, this factor is superior to the traditional h-index, g-index, and RG score. Intuitively and scientifically, this multi-view approach can improve evaluations of university faculties.

Introduction

The evaluation of academic development has been a long-term focus of both educational and industrial circles. How to comprehensively assess a scholar via a systemic evaluation has become an essential question to advance intelligent educational services, such as intelligent tutor systems (ITS), scientific resource retrieval, and faculty career evaluations [1], [2], [3]. To address this question, several previous efforts have been carried out that establish qualitative and quantitative measurements to evaluate a scholar or professor not only in terms of publications but also in aspects including research projects [4], patents [5], teaching quality [6], [7], etc. Given the full integration of the Internet into the academic domain, information about scholarly activities is now preserved and disseminated on the Internet.

However, a comprehensive evaluation system must be established in which all the required information is based on multi-source and multi-modal data without a uniform representation. Especially in evaluating university faculties, the assessment criteria cover multiple fields, including research ability and teaching skills, which require evaluators to collect heterogeneous data from the Internet and internal systems [8], [9], [10].

A comprehensive indicator considering as many aspects as possible for predicting faculty development is also required. Predicting the professional potential of university faculty based on comprehensive aspects through modeling or profiling using both empirical and projected indicators provides a novel approach for assessing faculty [11], [12].

For these reasons, in this paper, we propose an innovative university faculty evaluation system based on a knowledge graph that integrates heterogeneous data for profiling university faculties. First, the relevant data is collected from the Internet and internal systems, including faculty information and related scientific research papers, patents, funds, monographs, awards, professional activities and teaching performance. The graph structure is defined according to the collected data and preprocessing. Second, we construct entity representations through knowledge graph embedding methods to retain semantic information. By applying the embedded representations on the constructed knowledge graph, the effectiveness of the proposed graph structure among faculties and other entities is vividly displayed. Third, this paper also proposes an ADF for the prediction of faculty academic development, which integrates faculty member’s academic development status and student evaluation data from previous years. A case experiment is conducted to examine the effectiveness of this factor in two aspects: the relevance between the indicator and features of the knowledge graph as well as student evaluations, and the predictive accuracy and correlation of the ADF and other widely used academic evaluation indicators.

The rest of this paper is organized as follows. We present a brief literature review in Section 2. The definition, hypothesis and initial construction of the knowledge graph of university faculties are introduced in Section 3. Section 4 proposes embedding representations of entities stored in the previously constructed graph and conducts experiments on the ADF to examine the effectiveness and practicability of our proposed system. The whole architecture of the system (UFES-KG) is presented in Section 5, and concluding remarks are given in Section 6.