The latent words language model

Abstract

We present a new generative model of natural language, the latent words language model. This model uses a latent variable for every word in a text that represents synonyms or related words in the given context. We develop novel methods to train this model and to find the expected value of these latent variables for a given unseen text. The learned word similarities help to reduce the sparseness problems of traditional n-gram language models. We show that the model significantly outperforms interpolated Kneser–Ney smoothing and class-based language models on three different corpora. Furthermore the latent variables are useful features for information extraction. We show that both for semantic role labeling and word sense disambiguation, the performance of a supervised classifier increases when incorporating these variables as extra features. This improvement is especially large when using only a small annotated corpus for training.

Introduction

Since the very beginning of the field of natural language processing, researchers have tried to create models of natural language that offer a computer program a mean of generating, manipulating and classifying a given text. Early models were constructed by hand and were limited to a small domain (Winograd, 1972). It was soon found that these models were too rigid for natural language that contains a lot of variation that is not easily captured in fixed rules (Winston, 1976). With the introduction of large, hand-annotated corpora (e.g. the Penn treebank (Marcus et al., 1994)), most researchers turned to shallow methods for the classification of natural language. Examples of these include support vector machines (Cortes and Vapnik, 1995), hidden Markov models (Baum et al., 1970), maximum entropy classifiers (Berger et al., 1996) and others, which have been successfully used for many tasks, including named entity recognition (Sang and De Meulder, 2003), part-of-speech labeling (Brants, 2000) and phrase chunking (Kudo and Matsumoto, 2001). Limitations of this approach are however that for every new task a new corpus needs to be manually annotated, which is very costly both in man-hours, and the required crafting of features by hand that capture correlations between the input data and the output data.

Ideally we would like to develop models that have some deeper understanding of natural language. Large improvements in computational speed and storage size, together with the availability of enormous unlabeled corpora, have sparked an interest in unsupervised models that learn structures or statistics from unlabeled texts. Examples of these are topic models (Blei et al., 2003, Griffiths et al., 2005), unsupervised part-of-speech taggers (Goldwater and Griffiths, 2007) and deep neural networks (Collobert and Weston, 2008). Research has shown that the statistics or structures learned by these models can be successfully used in many supervised tasks, such as document classification (Blei et al., 2003), part-of-speech tagging, word segmentation in Chinese (Li and McCallum, 2005), dependency parsing (Koo et al., 2008) and named entity recognition (Miller et al., 2004). However, till now no unsupervised, generative model was proposed that learns accurate, fine-grained semantic similarities between words.

In this article we develop the latent words language model (LWLM). This model is a novel statistical model that learns context dependent hidden (or latent) words from unlabeled texts. A hidden word of a certain word in a specific textual context is a variable in a Bayesian network that generates the word. This variable models the probability distribution of every word in the vocabulary of the corpus on which the model is trained, indicating the probability that the concerned vocabulary word occurs in the considered context. Words that have a high probability of replacing the original word in the given context are often synonyms or have at least a strong semantic relationship with the original word. Table 1 shows the actual output of the model on an example sentence. This model has the following attractive properties: (1) it is trained on unlabeled texts, which are abundantly available in electronic format for most languages and do not require any manual annotation, (2) it does not require any knowledge of the syntactic or semantic structure of the text, and as such is completely language independent¹ and (3) it learns fine-grained semantic classes (i.e. the hidden words) that capture semantic and syntactic similarities.

The learned hidden words can be used to overcome the sparseness problem fundamental to natural language processing. We show this in three applications. In a first application we incorporate the hidden words in a language model and show that it outperforms Kneser–Ney and cluster language models on three different corpora. We then use the hidden words into a semantic role labeling system and word sense disambiguation system. For both systems this extra knowledge improves the accuracy significantly. The resulting word sense disambiguation system outperforms all other published systems on the corpus of the Senseval3 workshop. Furthermore the hidden words are included using a simple method (i.e. as extra features), which generalizes to virtually all feature-based natural language processing systems.

We fully describe the LWLM in Section 2. The LWLM is heavily influenced by the existing work on n-gram language models, and we first introduce these models in Section 2.1. Then we develop methods for training and inference the LWLM, and compare it to existing language models. We will see in Section 3 that the latent variables learned in this model can be successfully employed in supervised information extraction tasks, which is demonstrated by a semantic role label and a word sense disambiguation classifier. Finally we discuss related work in Section 4 and conclude this article in Section 5.