Nested Named Entity Recognition: A Survey

2.1 What is Flat NER?

The traditional NER task aims to identify named entities within the text that satisfy the following two assumptions:

Named entities that satisfy these assumptions are generally referred to as flat name entities. Correspondingly, the traditional NER task is essentially the flat NER task. The most common method for the flat NER task is to use sequence tagging techniques with a sequence tag scheme, which allows the model to classify individual tokens (i.e., words) and have the tokens combined together to identify named entities. Each token will be assigned with a tag consisting of a position indicator and an entity type, where the position indicator is used to represent the token’s role in a named entity.

A common sequence tag scheme is the BIO (Beginning-Inside-Outside) tag scheme, while other schemes such as IO, BIOES, BIES, BMEWO, and BILOU are also widely used. Figure 1(a) illustrates a sentence using the BIO tag scheme for flat NER task, where two named entities are contained in the given sentence. “Erin” labeled “B-PER” and “Harrison” labeled “I-PER” are combined as a unit to identify a person entity “Erin Harrison”, while the non-entity words (e.g., visited) are labeled as “O”. Formally, the traditional flat NER system learns to produce the corresponding token-level annotation sequence Y = {\( y_1, y_2, \ldots , y_K \)} for the given sentence X = {\( x_1, x_2, \ldots , x_K \)} following a sequence tag scheme, where K is the number of the tokens in the sentence. Alternatively, named entities in the sentence can be indicated by a list of tuples. In this example, the tuple \( \lt \)1, 2, PER\( \gt \) points to the person entity “Erin Harrison”. In practice, the sequence tag scheme is more commonly used for flat NER than the tuple tag scheme.

Fig. 1.

View Figure

2.2 What is Nested NER?

In many practical applications, it is common that the named entities have a nested structure [13, 48]. Specifically, an entity could contain other entities or be a part of other entities, which breaks the second assumption mentioned above. As the example shown in Figure 1(b), the outer entity “the Oakland Zoo” contains an inner entity, i.e., “Oakland”. In the AnCora corpus of Spanish and Catalan newspaper text, nearly half of entities are embedded within another entity. Named entities with the nested structure are also prevalent in specific domains. For example, approximately 17% of entities in the GENIA corpus, a biomedical domain corpus labeled with entity categories such as protein and DNA, are embedded. Consequently, the NER task is required for further recognizing named entities with nested structures (i.e., both outer entities and inner entities), rather than the longest outer entity only. The inherent complexity of nested entities makes the nested NER a more challenging task than traditional flat NER.

Consistent with flat NER, sequence tag schemes can also be used for nested NER. Figure 1(b) shows an example of annotated nested named entities using two different tag schemes, namely the sequence tag scheme (e.g., BIO) and the tuple tag scheme. The choice of tag scheme is often closely related to the adopted solution. For instance, the layered-based approach generally applies a sequence tag scheme, while the region-based approach utilizes the tuple tag scheme. Due to the limitations in labeling the nested structures, sequence tag schemes always require multiple corresponding token-level annotation sequences to fully identify all the nested entities in one sentence (three corresponding annotation sequences are illustrated in this example). In contrast, the triples without redundant information are more concise (three corresponding annotation sequences are replaced by three triples in this example). Formally, given a sentence with K tokens, X = {\( x_1, x_2, \ldots , x_K \)}, the output of a nested NER system is a list of tuples Y = {\( \lt I^{head}_1, I^{tail}_1, t_1\gt \), \( \lt I^{head}_2, I^{tail}_2, t_2\gt , \ldots , \lt I^{head}_m, I^{tail}_m, t_m\gt \)}, each of which specifies a entity mentioned in the sentence. Here, \( I^{head}_i \) and \( I^{tail}_i \) are the head index and the tail index of the ith named entity mention, respectively; \( t_i \) is the entity category from a predefined category set; m is the number of named entities in the given sentence.

2.3 Nested NER vs. Flat NER

Although the goal of both the nested NER task and the flat NER task is to identify name entities in sentences, the ideas for solving these two tasks are essentially distinct. We further discuss and compare these two tasks in this section.

Generalization. The traditional flat NER task aims to identify named entities from text that satisfy the two assumptions introduced above. In practice, numerous complex named entities break the two strict assumptions, mainly including nested, multi-category, overlapping, and discontinuous named entities. A brief introduction is as follows:

Among them, the nested named entity is most common in various domains. The nested NER task will further recognize nested named entities at all levels in addition to the flat named entity at the top level. In all nested NER datasets (more details in Sections 2.4 and 2.5), a proportion of entities do not contain other entities. In other words, flat named entities are included in the nested NER datasets. Furthermore, all nested NER approaches theoretically have the ability to simultaneously recognize flat named entities, while most flat NER approaches cannot address the nested problem. Therefore, we consider that the nested NER is more generalized, and the flat NER can be regarded as a simplified task of nested NER.

Independence. Recent studies have demonstrated that the performance of traditional flat NER approaches will dramatically suffer when recognizing nested entities [23, 29]. Agreeing with Finkel and Manning [2009] [13], the problem of ignoring nested structures in most flat NER approaches is due to technological rather than ideological reasons. As shown in Figure 1(a), the flat named entity information contained in a sentence can be easily expressed by an annotation sequence. For this reason, considerable studies treat the flat NER problem as a sequence tagging problem; accordingly, the sequence tagging model is applied to assign one label (e.g., B-Location, I-Person) to each token in the input sentence. In contrast, the nested named entity information contained in a sentence is difficult to represent through a simple annotation sequence (the example in Figure 1(b) contains three annotation sequences), making standard tagging techniques unsuitable for recognizing nested entities. More importantly, we consider that nested entities contain two key properties, namely partly shared entity boundary and token with multiple entity categories. In order to address this issue, the researchers attempt to reduce the dependence on the sequence tagging model and gradually propose various innovative architectures. Extensive experiments have verified that approaches sensitive to the properties of nested entities are obviously superior to the traditional NER models. This reflects the independence of the nested NER task from the traditional NER task. As the most common entity type (besides flat entities), it has practical importance to handle and explore nested NER task independently. Following the independence of nested NER, this article concentrates on the nested NER task and reviews numerous nested NER approaches to enlighten and guide researchers and practitioners in this area.

2.4 Flat NER Datasets

In the following, we summarize major benchmark datasets that are being widely used in flat NER task. Each tagged corpus is a collection of documents that contain annotations of one or more entity categories.

MUC-6, MUC-7 are part of the MUC corpus, which is focused on tasks related to information extraction, starting in 1987. There are in total seven conferences in this series from MUC-1 to MUC-7. In particular, MUC-6 and MUC-7 extend previous versions by means of adding tasks of NER.

CoNLL 2003 is a classic dataset and contains annotations for Reuters news in English and Frankfurter Rundschau in German. The English dataset involves 22,117 sentences corresponding to 302,811 tokens in four entity categories, including Locations, Organizations, Persons, and MISC (Miscellaneous).

JNLPBA dataset is originally from the GENIA corpus (more details in Section 2.5). The entire GENIA dataset is used for training, and new data is annotated for testing. Only the flat top-most entities are present in this dataset.

OntoNotes 5.0 is a corpus of large-scale, accurate, and integrated annotation of multiple levels of the shallow semantic structure in text provided by the OntoNotes project [42]. The dataset comprises various genres in three languages: roughly 1.5 million words of English, 800K of Chinese, and 300K of Arabic. Especially, the English NER data is tagged with a set of 18 proper name entity categories, consisting of 89 sub-categories.

The above-mentioned datasets with their data sources and number of entity categories (i.e., #Tags) are reported in Table 1. The datasets before 2005 were mainly constructed by annotating news articles with a small number of entity categories. Since then, more datasets have been constructed on diverse kinds of text sources, including Wikipedia articles, conversations, and user-generated text (e.g., tweets), while the number of entity categories has increased significantly (e.g., 89 in OntoNotes).

2.5 Nested NER Datasets

In this section, we describe major benchmark datasets that are being widely used in literature for training and eventual evaluation of proposed nested NER techniques.

ACE 2004, ACE 2005 belong to ACE Corpus, which contains entities, relations, and events for English, Chinese and Arabic. In ACE, the entities are limited to the following seven categories: Person, Organization, Facility, Location, GPE (geo-political entity), Weapon, and Vehicle. The entire English dataset of ACE 2004 contains 8,507 sentences with 27,749 entities, and about 40% of the sentences contain nested entities. In terms of the English dataset of ACE 2005, it contains 9,341 sentences with 30,931 entities, and about 35% of the sentences contain nested entities (the max nesting depth is 6).

GENIA corpus is a richly-annotated corpus for bio-text mining that contains 2,000 MEDLINE abstracts. The corpus contains 18,546 sentences corresponding to 55,740 tokens, annotated with five kinds of biological entities (DNA, RNA, Protein, Cell line, and Cell category), 36 fine-grained subcategories, and with parts of speech.

AnCora is annotated with parts of speech, parse trees, semantic roles, and word senses, including Spanish and Catalan portions. The entity categories present are Person, Location, Organization, Date, Number, and Other. The corpus annotators made a distinction between strong and weak entities. They define strong named entities as a word, a number, a date, or a string of words that refer to a single individual entity in the real world. Weak named entities consist of a noun phrase and must contain a strong entity. Weak entities are very prevalent; 47.1% of entities are embedded.

GermEval 2014 is a new German dataset sampled from German Wikipedia and News Corpora. The dataset covers over 31,000 sentences corresponding to over 590,000 tokens and contains 12 classes of named entities: four main classes (Person, Location, Organisation, and Other) with their subclasses, annotated at two levels (inner and outer chunks). The entire dataset contains over 41,000 named entities, about 7.8% of them embedded in other named entities, about 11.8% are derivations (deriv) and about 5.6% are parts of named entities concatenated with other words (part).

CNEC 2.0 is a corpus of 8,993 Czech sentences with manually annotated 35,220 Czech named entities, classified according to a two-level hierarchy of 46 named entities.

TAC KBP 2017 is a part of the TAC Knowledge Base Population (KBP), which is a set of evaluation tracks for populating knowledge bases from unstructured text. All KBP 2017 tasks are in trilingual (English, Chinese, and Spanish). Approximately 500 core documents are annotated with entities, relations, and events (ERE) according to the guidelines for Rich ERE. Five major coarse-grained entity categories are labeled in Rich ERE: Person, GPE, Location, Organization, and Facility. There are no entity subcategories.

NNE is a nested named entity dataset in English that is built on top of the full Wall Street Journal portion of the Penn Treebank (PTB). The entire dataset comprises 279,795 mentions of 114 entity categories with up to 6 layers of nesting. About 60% top-level entity mentions contain averaging 2.25 structural mentions. In addition, 19,144 out of 260,386 total spans are assigned multiple categories.

Table 1 presents a brief overview of various nested NER datasets over the course of time, along with the data source, languages involved, entity categories, and so on. As shown in Table 1, all nested NER datasets are constructed based on formal documents (e.g., news or science articles) rather than user-generated text. In addition, some researchers have paid attention to the nested NER problem in minority languages (e.g., CNEC 2.0 in Czech). Among these datasets, ACE 2005 and GENIA datasets are the most widely used in many recent nested NER studies (more details in Table 2).

2.6 Evaluation Metrics

NER models, both flat NER and nested NER, are generally evaluated by comparing their predictions with manual annotations. This section presents a brief introduction to the widely used quantified comparisons, including exact match and relaxed match.

Exact Match Evaluation. The NER task involves two essential steps, namely correctly detecting the entity boundaries and correctly determining their entity categories. Regarding the exact match evaluation, a named entity is considered correctly recognized only if the detected boundaries and categories are both consistent with the manual annotations. Precision (also called positive predictive value) is the fraction of correct entities among the recognized entities, while Recall (known as sensitivity) is the fraction of correct entities that were recognized. Usually, Precision and Recall scores are not discussed in isolation. F-score or F-measure provides a single score that balances both the concerns of Precision and Recall in one number. To be specific, Precision, Recall, and F-score are calculated based on the number of true positives (TP), false positives (FP), and false negatives (FN), which are defined as follows:

Based on this, Precision and Recall are calculated as (1) \( \begin{align} \text{Precision} &= \frac{TP}{TP+FP}, \end{align} \) (2) \( \begin{align} \text{Recall} &= \frac{TP}{TP+FN}. \end{align} \) And the traditional F-measure or balanced F-score (F1 score) is the harmonic mean of Precision and Recall, (3) \( \begin{align} \text{F1} &= \frac{2}{ \text{Precision}^{-1}+\text{Recall}^{-1}}. \end{align} \) The highest possible value of an F1 score is 1.0, indicating perfect Precision and Recall, and the lowest possible value is 0, if either the Precision or the Recall is zero. The widespread used F1 score gives equal importance to Precision and Recall.

Considering that most NER systems need to distinguish multiple entity categories, it is required to evaluate the performance with more than two entity classes (multiclass classification). In this setup, the final score is obtained by macro-averaging (taking all classes as equally important) or micro-averaging (biased by class frequency). Macro-averaged F-score is suitable for accessing a NER system performance overall across the sets of data. On the other hand, micro-averaged F-score can be a useful measure when the dataset varies in size.

Relaxed Match Evaluation. The relaxed match evaluation, also known as MUC evaluation, scores NER systems from two perspectives, namely the ability to predict the correct entity category and the ability to detect entity boundary information. In terms of an entity, a correct category will be recorded if its predicted entity category is the reference (true) category regardless of its detected boundary (as long as there is an overlap); a correct boundary will be recorded regardless of its category assignment. ACE further designs a more complex entity evaluation procedure to consider a few issues like partial match and errors in entity category, subcategory, and class. However, this complex evaluation method is not widely used in recent NER studies due to its high complexity and low versatility.

2.7 Related Surveys

Although we present the first comprehensive survey for the nested NER task, there are several survey papers related to the NER task (mainly for flat NER task) so far. Specifically, Nadeau and Sekine [2007] [38] presented the first survey of the NER task. They investigated early research in the NER field (from 1991 to 2006), including handcrafted rule-based algorithms and early machine learning techniques. Ling and Weld [2012] [31] introduced fine-grained entity recognition and formulated the tagging problem as multiclass, multilabel classification. Then Marrero et al. [2013] [36] analyzed the NER task from a theoretical and practical point of view. Saju and Shaja [2017] [44] provided a short survey on extracting named entities from new domains using big data analytics. Moreover, Dai [2018] [7] also provided a short survey on complex named entities, including nested entities, overlapping entities, discontinuous entities, and so on. Yadav and Bethard [2018] [64] presented a short survey of recent advances in NER, especially for the input representation. Goyal et al. [2018] [16] surveyed the development and progress made in the NER task. Li et al. [2020] [25] provided a comprehensive review on existing deep learning techniques for the NER task.

4.1 Early Rule-Based Approach

Early approaches for addressing nested NER rely on hand-craft rules and rule-based post-processing. Shen et al. [2003] [46] presented the first rule-based solution to recognize biomedical nested entities (referred to as cascaded entities in their article). Their approach contains four basic patterns corresponding to types of nested entities, namely (1) \( \langle \)entity1\( \rangle \) head noun \( \rightarrow \) \( \langle \)entity2\( \rangle \), (2) \( \langle \)entity1\( \rangle \) \( \langle \)entity2\( \rangle \) \( \rightarrow \) \( \langle \)entity3\( \rangle \), (3) modifier \( \langle \)entity1\( \rangle \) \( \rightarrow \) \( \langle \)entity2\( \rangle \), and (4) \( \langle \)entity1\( \rangle \) word \( \langle \)entity2\( \rangle \) \( \rightarrow \) \( \langle \)entity3\( \rangle \). They also leveraged the Hidden Markov Models (HMMs) [10] and integrated various features, including simple deterministic features, morphological features, POS features, and semantic trigger features, to recognize flat entities. HMM is a generative statistical model that assigns a probable target sequence to each input sentence following the Viterbi algorithm [55]. HMM is able to capture the locality of phenomena, thereby improving its evaluation performance. Subsequently, Zhou et al. [2004] [70] and Zhou [2006] [69] proposed similar approaches based on the above rules. To be specific, Zhou et al. [2004] [70] also presented a HMM-based NER model, while enhancing the rule-based post-processing to automatically extract rules from training data for nested NER task. From the GENIA corpus, they observed six useful patterns (two extend patterns were added except the above four patterns) of nested entity name constructions. Zhou [2006] [69] further proposed a NER approach called Mutual Information Independence Model (MIIM) in the biomedical domain. In MIIM, he also presented the rule-based post-processing for addressing nested entities, while leveraging the same patterns mentioned above.

4.2 Layered-Based Approach

The layered-based approach is an intuitive solution for nested NER. These models generally contain multiple layers (or levels) according to the hierarchical nature of the structure in nested named entities. Each layer is used to identify a group of named entities, which can be entities at a certain level or with a certain length. Since the layer module is the basic unit in layered-based approaches, we divide them into two subcategories according to the composition of the layer module, including (1) encoder-decoder module and (2) decoder-only module. Specifically, the encoder-decoder module refers to the basic unit containing a feature encoder and a tag decoder. The decoder-only module refers to the basic unit only containing a tag decoder, which means that these approaches only encode the sentence once before tag decoding. We present these two representative layered-based architectures in Figures 3 and 4, respectively.

Fig. 3.

View Figure

Fig. 4.

View Figure

4.3 Region-Based Approach

The region-based approaches generally treat the nested NER task as a multiclass classification problem, and various strategies have been designed to obtain the representation of potential regions (i.e., subsequences) before classification. We mainly divide existing region-based approaches into two categories based on the progressing strategy, including (1) enumeration-based strategy and (2) boundary-based strategy. To be specific, the enumeration-based strategy refers to the region-based approach that learns representations of all enumerated regions from the input sentence, and then classifies them into corresponding entity categories. The boundary-based strategy refers to the region-based approach that establishes representations of candidate regions (possibly entities) by leveraging boundary information, and subsequently accomplishes the entity classification. We respectively present the architectures of two progressing strategies in Figures 5 and 6.

Fig. 5.

View Figure

Fig. 6.

View Figure

4.4 Hypergraph-Based Approach

The hypergraph-based approaches leverage the hypergraph structure to express the nested structure, and the hyperarc in a hypergraph is able to precisely tag the token in one sentence that belongs to different named entities. Lu and Roth [2015] [33] firstly presented a novel directed hypergraph (referred to as mention hypergraph) for both boundary detection and category prediction. The proposed hypergraph structure consists of five types of nodes, which are used to compactly represent many entities of different semantic categories and boundaries. Moreover, the hyperarc will naturally address the nested structure problem as it can connect two or more nodes. These paths altogether form a unique sub-hypergraph of the original hypergraph to express all nested entities in a sentence. In order to address the structural ambiguity issue of Lu and Roth [2015] [33], Wang and Lu [2018] [56] proposed a neural segmental hypergraph model using neural networks to obtain distributed feature representations. We present their hypergraph structure in Figure 7. Their model has a \( O(cmn) \) time complexity (m is the number of entity categories, n is the number of words in a sentence, and c is the maximal number of words for each entity). Particularly, the segmental feature is introduced into hypergraph for addressing the structural ambiguity issue [33]. Different from the strategy of constructing the hypergraph in the above two approaches, Katiyar and Cardie [2018] [23] introduced another hypergraph structure based on the BILOU tag scheme (an extended version of BIO tag scheme), and proposed a standard LSTM-based sequence tagging model to learn a hypergraph representation for nested entities in the input sentence. In their hypergraph structure, each node is the BILOU tag corresponding to the input token. There are two types of directed edges in this hypergraph, including simple edges for which there is only one head node for every tail node and hyperarcs that connect more than one head node to a tail node. They treated the hypergraph construction procedure as a multilabel assignment process, and greedily decoded the hypergraph from left to right to find the most likely sub-hypergraph.

Fig. 7.

View Figure

4.5 Transition-Based Approach

Transition-based nested NER approaches are mainly inspired by transition-based dependency parsers [39]. As shown in Figure 8, such approaches parse a sentence from left to right, building a tree based on greedy decoding one action at a time. Finkel and Manning [2009] [13] proposed a discriminative constituent parser for recognizing nested named entities. The parser aims to extract a constituency-based parsing tree from a sentence that represents its nested structure. In addition, the parser is similar to the chart-based PCFG parser [12], except that instead of putting probabilities over rules, it puts clique potentials over local subtrees. Recently, Wang et al. [2018] [57] introduced a neural transition-based approach to model the nested structure of entities. They first mapped the sentence with nested entities to the forest structure, where each entity corresponds to a constituent of the forest. Their system then learns to construct the forest structure in a bottom-up manner through an action sequence. To be specific, the system consists of three types of transition action (i.e., SHIFT, REDUCE, UNARY) and employs a stack to partially store processed nested elements. The system’s state is defined as [S, i, A] that denotes stack, buffer front index, and action history, respectively. In each step, one of three types of actions will be applied to change the system’s state. In addition, Marinho et al. [2019] [35] introduced another neural transition-based approach, the Hierarchical and Nested Named Entity Recognition (HNNER) model, to handle different levels of nested entities. Specifically, they designed the transition system based on a word stack, a word buffer, a mention stack, and an output buffer. Moreover, they considered four types of possible system actions, including OUT, SHIFT, TRANSITION, and REDUCE. They further proposed a set of modifier classes that introduce certain concepts that change the meaning of an entity, such as absence, or uncertainty about a given entity.

Fig. 8.

View Figure

4.6 Other Approaches

Apart from the above-mentioned approaches, there are a few distinctive and particular studies and explorations on the nested NER, and it is difficult to classify them into the five types described above. In earlier studies, Gu [2006] [18] treated the nested NER task as a binary classification problem and solved it using Support Vector Machine (SVM) [5]. The SVM is a widely used supervised model that can perform well for handling high dimensional input space problem in many text mining tasks. In this solution, he used two tag schemes to set the class label for each token, including outmost labeling and inner labeling. Muis and Lu [2017] [37] developed a gap-based tag schema to capture nested structures of entities. They defined the mention separators to represent nested named entities. At each gap between two words, they considered eight possible types of mention separators based on the combination of three cases, including (1) the entity is starting at the next word (S), (2) the entity is ending at the previous word (E), and (3) the entity is continuing to the next word (C). Since each mention separators sequence can only present the entity information in the same entity category, they supported multiple entity categories by using multiple sequences, one for each entity category. Strakova et al. [2019] [50] proposed two neural network architectures for nested NER. For their models, they need to concatenate the nested entity multiple labels into one multilabel. The first model predicts the label of each token with a standard LSTM-CRF model. In the second model, the nested NER task will be viewed as a sequence-to-sequence task, in which the input sequence is the tokens and the output sequence is the labels. The decoder predicts labels for each token, until a special label “\( \left\langle eow \right\rangle \)” (end of the word) is predicted and the decoder moves to the next token. Lin et al. [2019] [29] proposed Anchor-Region Networks (ARNs), a sequence-to-nuggets architecture for recognizing nested entities. The ARNs first identifies anchor words (i.e., possible head words) of all named entities, and then recognizes the entity boundaries for each anchor word by exploiting regular phrase structures. Furthermore, they also designed Bag Loss, an objective function that can train ARNs in an end-to-end manner without using any anchor word annotation. Li et al. [2020] [27] proposed to formulate the NER task as a Machine Reading Comprehension (MRC) task. This formulation naturally tackles the nested structure problem. Specifically, they transformed the tagging-style annotated NER dataset to a set of {QUESTION, ANSWER, CONTEXT} tuples. For the question generation, they used a template-based procedure, which can take annotation guideline notes as references. Furthermore, they designed and adopted one region selection strategy for the answer representation. They first used two binary classifiers, one to predict whether each token is the start index or not, the other to predict whether each token is the end index or not. They further used another binary classification model to match each predicted start index with its corresponding end index since there could be multiple entities of the same category. Luo and Zhao [2020] [34] proposed a novel bipartite flat-graph (BiFlaG) network for nested NER, which contains a flat NER module and a graph module. The BiFlaG can jointly learn flat entities and effectively captures the bidirectional interaction (inside-out and outside-in directions) between them. Their model first uses the flat NER module to recognize the outermost entities and construct entity graphs. And then, each entity graph is fed to the subsequent graph module, which aims to effectively learn the dependencies of inner entities and improve outermost entity prediction. From these approaches, we can observe that most of them (e.g., Gu [2006] [18], Muis and Lu [2017] [37], Strakova et al. [2019] [50], Li et al. [2020] [27]) need to design unique tag schemes to facilitate the model to capture the nested structure of entities. Alternatively, some models additionally consider some auxiliary knowledge (e.g., anchor words, graph structure) to solve the nesting problem of entities.

4.7 Summary and Discussion

As shown in Table 2, we summarize all nested NER approaches according to the model architecture, along with their respective input representation, encoder-decoder, and performance. Apart from early rule-based approaches, the other four main model architectures have achieved great success in nested NER task. We sum up these four types of approaches in general, as follows. The layered-based approach is inspired by the sequence tagging approaches and identifies entities in the sentence layer by layer. The region-based approach also decomposes the nested NER, that is, produces all possible regions in the sentence and classifies them independently. The hypergraph-based approach presents a novel data structure to express the nested structure from the input sentence, which regards identifying the entities in the sentence as finding the best sub-hypergraph in the complete mention hypergraph. The transition-based approach is mainly inspired by the transition-based parsing and treats the nested NER task as a sequential action decision process. Furthermore, we infer the following three observations based on Table 2.

First, deep neural models have already been widely used by the existing approaches to address the nested NER task. Earlier approaches from all four model architectures are normally based on feature engineering and machine learning algorithms. Recently, deep neural models have become dominant and have various usages depending on different architectures. For example, as we can be seen from Table 2, many layered-based approaches only leverage the neural model as the sequential feature extractor (i.e., encoder). In the case of region-based, it is also feasible to build a neural-based classifier (i.e., decoder) to predict which entity category the region belongs to.

Second, we can observe that approaches from all four model architectures can achieve state-of-the-art performance by comparing all existing approaches. Thus, it is hard to determine which architecture is the optimal solution and most appropriate for nested NER task. Furthermore, we observe that a few of the existing approaches incorporate the characteristics from more than one model architecture. For example, Wang et al. [2020] [58] presented a layered model, and each model layer enumerates all regions of a certain length and predicts the entity category for them. Long et al. [2020] [32] proposed a region-based model, while the hierarchical region generation is from bottom to up recursively (i.e., similar to the layered-based approach). In our opinion, future efforts can be spared to explore how to fuse the existing model architectures.

Third, recent nested NER studies mainly introduce pre-trained language models as (or as a supplement to) the input representation layer of their model. The overall experimental results of various nested NER approaches shown in Table 2 demonstrate that the pre-trained language model embedding can extremely improve the performance of the proposed approaches, especially on the ACE 2005 dataset. Pre-trained language model representation has become one of the most effective auxiliary means to the nested NER task. With the tremendous growth of representation learning, pre-trained language models have been used in a wide range of NLP fields with huge success [9, 28, 62].

5.1 Entity Dependency

Entity dependency refers to the dependency between different entities in a sentence. In particular, the entity may be included in another entity in the nested entity setting. Take “A spokesman for the Oakland Zoo said ...” as an example, the outer entity “the Oakland Zoo” contains an inner entity “Oakland”. It is effective for identifying entities to leverage such dependence between different entities, especially true for entities with nested structure. The targeted learning of the model to the entity dependency is consistent with the process of human entity recognition. From Table 3, we can observe that many existing approaches have the ability to capture the entity dependency. We categorize the existing approaches depending on the varying degrees of captured entity dependency, as follows.

Partial Entity Dependency. For nested entities, the dependencies between entities are bidirectional, that is, from the inner entity to the outer entity (i.e., inside-out direction) and from the outer entity to the inner entity (i.e., outside-in direction). Most current approaches can only capture and transfer knowledge about entity dependency in one direction. To be specific, in terms of layered-based approaches, most of them can transfer the learned entity dependency in a certain direction. For example, the proposed models of Ju et al. [2018] [22] and Fisher and Vlachos [2019] [14] can learn entity dependency in inside-out direction, while the proposed models of Shibuya and Hovy [2020] [47] and Luo and Zhao [2020] [34] can learn entity dependency in out-inside direction. Besides the layered-based approaches, the transition-based approaches [13, 35, 57] can utilize knowledge of the inner entity in predicting the outer entity (i.e., inside-out direction), because they parse the sentence (i.e., the sequential decision of transitions) from left to right.

Full Entity Dependency. Only a few of the existing approaches can fully consider the dependence between entities in sentences. The representative approaches are hypergraph-based approaches [33, 56] that can naturally consider all the entities in a sentence by using a global optimization strategy. For example, Lu and Roth [2015] [33] can represent each of model outputs with a single fully-observed structure, which can be globally optimized with standard gradient-based methods. In addition, the region-based approach proposed by Sun et al. [2020] [52] incorporates a span relation network to capture the relationships between all possible spans in the sentence.

None Entity Dependency. Many approaches recognize entities without considering the dependence between entities, mainly based on regions. It is because existing region-based approaches mostly require to independently classify each region or boundary. In addition, there are some approaches, such as Lin et al. [2019] [29] and Li et al. [2020] [27], also do not consider the entity dependency.

5.2 Stage Framework

Generally speaking, the mainstream models to solve the flat NER problem are single-stage models (e.g., LSTM-CRF, CNN-LSTM-CRF), which complete entity detection and category prediction simultaneously. Extending to the nested NER task, many solutions including more than one stage have been proposed, in addition to continuing with the single-stage framework. For such approaches, the recognition process is artificially decomposed into several sequential operating stages or processes to improve the overall performance. Accordingly, we divide the existing approaches into single-stage and multi-stage frameworks.

Single-Stage Framework. The nested NER approaches with the single-stage framework have the ability to jointly detect entity boundaries and predict entity categories in only a single stage. Among the existing approaches, all hypergraph-based approaches [23, 33, 56] and transition-based approaches [13, 35, 57] belong to single-stage framework. In addition, some region-based approaches that first enumerate regions [29, 40, 49, 63] are classified as single-stage frameworks. These approaches only need to classify all possible regions (optionally considering its context) from the sentence. The single-stage nested NER approaches share the common characteristic of high demand for computing power to enhance the relatively slow training speed.

Multi-Stage Framework. The nested NER approaches with the multi-stage framework have two or more successive operating stages. Normally, such solutions artificially decompose the recognition process into more than one stage to improve the overall performance. It is worth noting that the number of stages of these approaches can be fixed or varying.

From Table 3, we can observe that employing a multi-stage framework is currently a prevalent manner to solve nested NER task, which also indicates the complexity of the nested NER task. Combined with Table 2, we further observe that the model performance is not directly related to the stage framework property. In other words, there is no clear optimal one among single-stage, fixed multi-stage, and unfixed multi-stage frameworks for the nested NER task.

5.3 Error Propagation

Through the above categorizations, we observe that most of the existing models have more than one stage or consider entity dependency. However, these models are likely to suffer from the error propagation problem, i.e., errors in the previous decisions are propagated to the next decisions. In a nutshell, entity recognition consists of multiple consecutive decisions, which is prone to error propagation. Therefore, we consider error propagation as an important property of nested NER models and categorize existing approaches accordingly.

With Error Propagation. The nested NER models that are divided into two or more stages and there is an order relationship between the stages will raise the error propagation problem. The approaches that may cause error propagation for this reason include part of region-based approaches [14, 32, 53, 59, 61, 68] and most layered-based approaches [1, 22, 34, 47]. We take the layered-based approach as an example to explain in detail the causes of error propagation. Layered-based models commonly first extract the inner entities and feed them into the next layer to extract outer entities. Intuitively, extracting wrong entities from the previous layer will affect the performance of the next layer. Moreover, when an outer entity is extracted first, the inner one will be ignored by the model. Thus, most layered-based approaches suffer from error propagation. In addition, all transition-based approaches [13, 35, 57], which use a greedy training strategy, will raise error propagation (although they belong to the single-stage framework). For transition-based models, during training, the current action depends on the golden previous actions, while during testing, the entire action sequence is generated by the model. For this reason, an erroneous action prediction will further deviate from the predictions of follow-up actions. We also consider such accumulated discrepancy in transition-based approaches as error propagation.

Without Error Propagation. In all existing approaches, there are mainly two types of approaches that can avoid error propagation. The first type is hypergraph-based approaches [33, 56] that leverage graph-based global optimization strategies to complete boundary detection and category prediction at the same time. The second type is single-stage region-based approaches [4, 40, 63], which independently determine the entity category of each region. In particular, Wang et al. [2020] [58] also successfully avoided error propagation since each layer in their model predicts whether each region of a certain length is a complete entity mention.

After the above analysis, we further observe that considering entity dependence does not necessarily lead to error propagation, such as hypergraph-based approaches [33, 56]. Error propagation does not necessarily exist in multi-stage approaches, such as Muis and Lu [2017] [37] (independent between stages). More importantly, the current research results cannot verify that the model with error propagation will perform worse than the model without error propagation. We believe that the error propagation in the model will definitely bring some negative effects. So far, the performance level of the nested NER approach generally has room for improvement, thus many studies choose to improve performance first after weighing error propagation and performance improvement.

5.4 Tag Scheme

In the Background section, we introduce two types of tag schemes for the NER task, including the sequence tag scheme and the tuple tag scheme. The sequence tag schemes produce multiple sequences for expressing entities with different levels in sentences, and the tuple scheme uses a list of tuples to indicate all entities in sentences. In the existing nested NER approaches, both types of tag schemes are extremely pervasive. The approaches applying the sequence tag scheme mainly treat nested NER task as sequence tagging or sequence generation tasks, such as multiple sequence tagging tasks (i.e., layered-based approaches), and sequential decision tasks (i.e., transition-based approaches). The approaches using the tuple tag scheme mainly regard the nested NER task as a classification task, such as all region-based approaches. In addition, a small number of approaches require both the sequence tag scheme and the tuple tag scheme, neither of which is indispensable. For example, in the multi-stage region-based approaches [59, 68], multiple stages may require different tag schemes for their different targets. This indicates that different types of tag schemes have individuality and complement each other.

6.1 Challenges

Learning the nested structure. Benefiting from the powerful word-level representation, the nested NER system significantly improves the overall performance. The pre-trained language model representation has become one of the most effective auxiliary means to the nested NER task. This fact also conveys an optimistic message that without pre-trained language model embeddings, there is still remaining a large space worth exploring in future nested NER research. We consider that in addition to adopting or integrating powerful pre-trained language models as a complement, researchers also need to propose more effective and innovative methods for the nested nature of entities. This puts forward a requirement for researchers to face this peculiar nature of the NER task without hesitation.

Some researchers explored and treated the entire nested NER problem as other learning problems (e.g., multi-layer sequence tagging problem and syntactic structure parsing problem) to handle the nested structure in our task. Therefore, it is the most challenging to propose a nested NER model that can precisely express the nested structure, including the boundary information of entities and the dependency information between entities.

Alternatively, some studies complete nested NER in two steps, namely entity boundary detection and entity category prediction. In the case where different named entities partly share the same boundary word (e.g., a person entity “a spokesman for the Oakland Zoo” and a facility entity “the Oakland Zoo” share the same boundary word “Zoo”), this makes the nested NER model challenging to cope with the step of boundary detection. Furthermore, detecting all boundaries independently ignores the entity dependency even if different entities partly share a boundary, where the dependency between entities is one of the most important factors in nested structure. Thus, learning the nested structure is a consistent challenge in this task, even if the nested NER task is decomposed.

Data annotation. Data annotation remains time-consuming and expensive in the NER task. As shown in Table 1, there are relatively few nested NER datasets, and there are not many nested instances (that is, sentences containing nested structures) in each dataset. Specifically, 17% of the entities in the GENIA corpus are embedded within another entity, and 35% of sentences contain nested named entities in ACE 2005 corpora. More importantly, such datasets generally come from formal documents (e.g., news, biological domain), which makes the nested NER approaches difficult to apply to real-world scenarios. Compared to flat NER datasets, the quality of data annotation is more concerned in nested NER datasets. Another challenge for data annotation is that the current data formats, including the BIO tag scheme and the triple scheme, cannot well present the nested entity structure. We believe that the nested NER model will learn the nested structure more efficiently and effectively, if we label such type of information when entities are embedded within another entity.

6.2 Future Directions

A United architecture for nested NER. Based on the above discussion, we have at least four major architectures for nested NER task, including layered-based, region-based, hypergraph-based, and transition-based approaches. Meanwhile, we also categorize existing approaches based on entity dependency, stage framework, error propagation, and tag scheme. According to this, we can find that each architecture or category approach has its own advantages and shortcomings. It is worth noting that there are several explorations to propose a combination model that integrates the advantages from multiple types of approaches. We consider that the Pyramid model proposed by Wang et al. [2020] [58] is a successful instance, where the Pyramid model fuses the characteristics of region-based approaches though it is a layered-based approach. Therefore, it is worth exploring one united architecture for nested NER, which will have advantages in both theoretical and experimental results.

Deep transfer learning for nested NER. While we focus on developing more advanced architecture to address the nested NER problem, we also expect more other technologies to gain significant improvements for this task. Deep transfer learning has achieved great success in the NLP field in recent years. For addressing the flat NER problem, some studies [60] proposed the multitask models to jointly train on the flat NER task and its related tasks (e.g., the Chunking task and the Part-of-Speech task). These multitask models leverage the related task to extract and transfer shared knowledge, so that improves the overall performance both on flat NER task and its related tasks. Therefore, we believe that such a deep transfer manner can also perform well in nested NER task. Besides, one interesting transfer direction might be the granularity knowledge transfer, which means transferring knowledge from flat NER task to nested NER task. Different from the joint training manner, we desire the granularity knowledge transfer model can leverage the coarse-grained knowledge from a flat NER model (could be a sequence tagging architecture) to improve the overall performance of the nested NER model.

Semi-supervised Learning for nested NER. As we mentioned before, the data annotation cost for the nested NER task is much higher than the flat NER task. As data annotation is one important challenge for our task, one interesting research direction is developing nested NER approaches (e.g., semi-supervised learning architectures and even unsupervised learning architectures) to adapt to the current environment. In addition, all approaches we present in this survey are strong supervised models. In flat NER task, there are many semi-supervised approaches, but they are hardly applied in nested NER task since such approaches are mainly based on sequence tagging architecture. For example, He and Sun [2017] [20] proposed a unified model which can learn from out-of-domain corpora and in-domain unannotated texts, where the self-training learning function is based on the sentence confidence and decision boundary. We consider that exploring the semi-supervised models for nested NER can further enhance the possibility of applying nested NER systems into real-world downstream natural language applications.

A united tag scheme for nested NER. As discussed in Section 5.4, different types of nested NER approaches adopt unique tag schemes. For example, region-based approaches generally leverage the tuple tag scheme as the entity label information, while layered-based approaches retain the sequence tag scheme, which is more commonly used in the flat NER task. The original intention of the tag scheme is to present the entity label information in the sentence. In order to accurately and comprehensively present the label information of more complex nested entities, the choice of tag scheme is particularly critical. In practice, neither of the above tag schemes can fully express the nested structure, because the dependency relationship between inner and outer entities in the same nesting structure has to be abandoned in the tagging process. To the best of our knowledge, no existing work separately focuses on the design of precise tag schemes for nested NER. We expect a breakout in this research direction in the future.