On integrating a language model into neural machine translation

Abstract

Recent advances in end-to-end neural machine translation models have achieved promising results on high-resource language pairs such as En→ Fr and En→ De. One of the major factor behind these successes is the availability of high quality parallel corpora. We explore two strategies on leveraging abundant amount of monolingual data for neural machine translation. We observe improvements by both combining scores from neural language model trained only on target monolingual data with neural machine translation model and fusing hidden-states of these two models. We obtain up to 2 BLEU improvement over hierarchical and phrase-based baseline on low-resource language pair, Turkish→ English. Our method was initially motivated towards tasks with less parallel data, but we also show that it extends to high resource languages such as Cs→ En and De→ En translation tasks, where we obtain 0.39 and 0.47 BLEU improvements over the neural machine translation baselines, respectively.

Introduction

Neural machine translation (NMT) is an end-to-end neural network based approach to statistical machine translation (Kalchbrenner, Blunsom, 2013, Sutskever, Vinyals, Le, 2014, Cho, van Merrienboer, Gulcehre, Bougares, Schwenk, Bengio, 2014, Bahdanau, Cho, Bengio) which has recently been able to achieve state-of-the-art translation quality on many language pairs (Sutskever, Vinyals, Le, 2014, Jean, Cho, Memisevic, Bengio, Chung, Cho, Bengio, Luong, Manning). For the recent advances made since the publication of the initial technical report version of this manuscript Gulcehre et al. (2015), we refer readers to Section 2.

A large part of the recent successes of NMT has been due to the availability of large amounts of high quality, sentence aligned corpora. In the case of low resource language pairs however, or in a task with severe domain restrictions, large parallel corpora may not be available. In contrast, monolingual corpora is almost always abundant and universally available. Despite being “unlabeled”, monolingual corpora still exhibit rich linguistic structure that can be useful for machine translation.

In this work, we explore the question of how to leverage monolingual corpora. Specifically, we explore two ways of integrating a language model (LM) trained only on monolingual data (target language) into an NMT system. First, by combining the LM and NMT system at the output level (which we term shallow fusion). Second, by fusing the LM and NMT at the level of their hidden states, enabling us to combine them nonlinearly (which we term deep fusion).

We evaluate the proposed fusion strategies under two distinct settings. The first setting involves low-resource translation tasks (Turkish → English and Chinese → English SMS Chat), where training an NMT system with a small parallel corpus alone often leads to severe overfitting. Second, we further evaluate our methods on higher-resource tasks (German→ English and Czech→ English). In addition to observing an improvement in translation quality on the small scale tasks, we observe an improvement in the large scale setting. In addition, our experiments show that our deep fusion method of fusing the LM and NMT systems more significantly improves translation quality than the shallow fusion method.

In the following section (Section 2), we review recent work in neural machine translation. We present our basic model architecture in Section 3  and describe our shallow and deep fusion approaches in Section 4. Next, we describe our datasets and experiments in Sections 5–6.