An incremental graph-partitioning algorithm for entity resolution

Highlights

• A novel incremental data association algorithm is proposed for entity resolution.

• Order of magnitude faster than batch algorithms, with little/no loss in accuracy.

• Shows 30–40% better F-Score on John Smith dataset, compared to leading heuristics.

• Proposed algorithm leverages the Clique Partition Problem.

• Quickly updates solution for new references as well as changes in similarity scores.

Abstract

Entity resolution is an important data association task when fusing information from multiple sources. Oftentimes the information arrives continuously and the entity resolution algorithm needs to efficiently update its solution upon receiving new information. In this work, we introduce an incremental entity resolution algorithm based on a graph partitioning formulation. The developed algorithm is able to handle both incrementally arriving entity references, as well as incrementally arriving information which changes the pairwise similarity scores between the references. New information is handled in a way that allows the algorithm to reconsider past decisions when contradicting information arrives. Because the graph partitioning formulation used is NP-Hard, a heuristic algorithm is developed to produce good solutions, which is also compatible with a blocking technique to limit the number of required comparisons. The algorithm is tested on a variety of datasets (randomly generated and real) and it is shown that allowing the algorithm to consider revised scores and revisit prior decisions offers a substantial improvement to accuracy (approximately 30–40% better F-Score on a natural language dataset), compared to other greedy heuristics on the same set of coefficients. It is also shown that, on a test set with 100 references, the incremental algorithm is up to an order of magnitude faster than a batch algorithm approach that re-solves the entire problem.

Introduction

In today’s era of data deluge, information fusion has increasing applications in myriad of domains like disaster relief [28], [38], consumer marketing [39], etc. In most of these applications, data are typically gathered by heterogeneous sources such as human observers and/or human operated or automated physical sensors like cameras, LIDAR, acoustic sensors, etc. These data are gathered over time and may contain duplicated references of the same set of objects, which may cause inconsistencies and inefficiencies in the data analysis efforts. For this reason, data association becomes an essential step in multi-source information fusion, which identifies when two sensors are observing the same object and, if so, those observations are fused together, creating “cumulative” evidence. This cumulative evidence can be used as a surrogate to the real world, for building and testing various hypotheses and models.

In settings where the observed data is in the form of named entities such as people, places, organizations, etc., this problem is often referred to as entity resolution. In these settings, the data is often in the form of structured or unstructured natural language, like databases, field reports, news articles, websites, citation lists, etc. Observations of entities are called entity references, and the goal is to identify when multiple entity references refer to the same entity, in which case they are said to be co-referent. In this paper, the term entity resolution is used, rather than data association, since the test datasets used later are primarily drawn from that community.

In many applications, the data on which entity resolution is performed is constantly changing over time. This newly arriving information takes two forms: (1) addition of new entity references into the current dataset; and (2) New information that may revise beliefs about the current entity resolution solution. The challenge of incremental entity resolution is to update the current solution to reflect these changes without needing to resolve the entire problem. If done correctly, incremental entity resolution can provide tremendous gains in computation time, with very little to no loss in the solution accuracy.

The main contribution of this paper is an incremental heuristic algorithm for the Clique Partition Problem (CPP), a well studied graph partitioning problem [13], [21], [37], [45]. The algorithm to be presented updates a current partitioning solution when a new entity reference arrives or if new information about an existing reference arrives. The incremental algorithm developed in this paper is tested on a variety of datasets to establish its performance under a range of conditions.

The remainder of this paper is organized as follows. Section 2 reviews prior work in incremental entity resolution with an emphasis on incremental clustering. The new incremental algorithm is presented in Section 3. The testing methodology is discussed in Section 4, along with the experimental results for both random and real datasets. Finally, we conclude with a summary and some future research directions.