A knowledge graph based speech interface for question answering systems

Abstract

Speech interfaces to conversational systems have been a focus in academia and industry for over a decade due to its applicability as a natural interface. Speech recognition and speech synthesis constitute the important input and output modules respectively for such spoken interface systems. In this paper, the speech recognition interface for question answering applications is reviewed, and existing limitations are discussed. The existing spoken question answering (QA) systems use an automatic speech recogniser by adapting acoustic and language models for the speech interface and off-the-shelf language processing systems for question interpretation. In the process, the impact of recognition errors and language processing inaccuracies is neglected. It is illustrated in the paper how a semantically rich knowledge graph can be used to solve automatic speech recognition and language processing specific problems. A simple concatenation of a speech recogniser and a natural language processing system is a shallow method for a speech interface. An effort beyond merely concatenating these two units is required to develop a successful spoken question answering system. It is illustrated in this paper how a knowledge graph based structured data can be used to build a unified system combining speech recognition and language understanding. This facilitates the use of a semantically rich data model for speech interface.

Introduction

A question answering system can be defined as a system which searches for a suitable answer in a knowledge base for a given question by the user. The answer may be one word, a sentence snippet, a well constructed and meaningful sentence or a collection of sentences with a logical coherence. The answer type depends on the application for which a question answering system is developed. A question answering (QA) system can be developed on different paradigms: database querying, information retrieval and knowledge graph based. The database querying method typically involves developing a question-answer pair database for a specific domain and then fetching the answer depending on the user question. Information retrieval is usually finding unstructured data, i.e. text, that satisfies the user query from large collections of data, usually on the Web. The knowledge graph based approach involves a semantic analysis of the query and then accessing a structured database (Jurafsky and Martin, 2008). The database can be a fully relational database or simple structured databases like sets of Resource Description Framework (RDF)¹ triples. A speech interface to the above discussed QA systems is generally called spoken question answering. A general pipeline of spoken question answering is illustrated in Fig. 1.

A speech interface to a question answering system makes the interaction more natural. To achieve this, the most common approach is to combine an automatic speech recognition and a speech synthesis unit with a question answering system. The development of an automatic speech recognition (ASR) and a speech synthesis (SS) modules are straight forward given a database of question-answer pairs. The complexity increases if a QA system is based on information retrieval or on a knowledge graph, since it involves a semantic analysis of the question. The challenges involved in developing a spoken QA is reviewed in Rosso et al. (2012). The ASR and SS systems have their own dependencies and unsolved research problems, adding to the complexities of QA system development. In this paper, we are addressing only the speech recognition part of the interface, which is the first step of a spoken QA process and also more complicated than speech synthesis because of the necessity to recognise multiple speakers. In contrast, a speech synthesiser can be developed based on a single speaker. The core research challenges (e.g. multiple speakers, ambient noise, different dialects, etc.) in ASR are not the focus of this paper. The main focus is on the interface between a speech recogniser and a QA system. It has to be noted that many question answering systems which integrate speech recognition without speech synthesis in the overall architecture are also called “speech interfaced question answering” or “spoken question answering”. We are following the same terminology, as our focus is only on the speech recognition interface. It should be mentioned that spoken question answering can also be developed for spoken documents, but the focus of this paper is only on text documents.

A review of existing spoken question answering systems is presented in this paper, and the limitations are discussed. The existing spoken QA systems use ASR by adapting the acoustic and the language model to a pre-determined set of QA data. After the recognition step, language processing is carried out to interpret the recognised questions. In most of the methods discussed in the review section, the focus is on developing a spoken QA system rather than to address the issues of recognition errors or language processing faults. Language processing involves parts of speech tagging, parsing, named entity extraction, relation extraction, sequence labeling etc. It is integrated with the ASR output depending on the QA application. If the input to QA is an entity from a recognised sentence, then an off-the-shelf entity recogniser is integrated. The recognition accuracy and the efficiency of a language processing unit play an important role in the overall success of a spoken QA. To address problems associated with the speech interface to QA, we will use a knowledge graph and demonstrate with examples how it could be used to solve problems related to automatic speech recognition and spoken language understanding (We refer to language understanding as spoken language understanding (SLU) here, since it is processing of the recognised utterance). ASR specific problems include the use in rescoring to reduce the word error rate, for the out-of-vocabulary problem and rarely seen words, the creation of language models and multi-domain speech recognition, while SLU specific problems include slot filling and intent detection, entity relation extraction, unsupervised data collection for training/testing. In the literature, the knowledge graph concept is used for SLU (discussed in Section 7.2) but to our knowledge we are the first to demonstrate its use to solve ASR specific problems. We have also demonstrated how a knowledge graph can be used as a common data model to build a unified system combining speech recognition and language understanding. We use the term “Spoken Language Recognition and Understanding” (SLRU) for a unified system. There are few works in the literature where they have considered ASR and SLU as one unit (Yaman et al., 2008) but there is no method which describe the speech interface with this framework. In SLRU we are making use of the semantics in the knowledge graph to address the recognition errors and interpret the text.

The motivation for speech interfaced QA is presented in Section 2. A review of conventional spoken QA systems is conducted in Section 3. The motivation for a knowledge graph based speech interface is presented in Section 4. In Section 5, the definition of speech recognition, spoken language understanding and knowledge graph is presented. The SLRU design is described in Section 6. In Section 7, we illustrate how the knowledge graph helps to address specific problems in ASR and SLU. The conclusion is presented in Section 8.