Deep neural network ensemble for reducing artificial noise in bandwidth extension

Abstract

In this paper, we propose a deep neural network (DNN) ensemble for reducing artificial noise in speech bandwidth extension (BWE). The proposed DNN ensemble consists of three DNN models; one is a classification model, and the other two are regression models. When estimating sub-band energies of the high-frequency region using sequential DNNs in a frequency domain, the over-estimation of sub-band energies causes annoying artificial noise. To mitigate this artificial noise, we design a DNN classification model that can classify over-estimation frames against normal frames. Then, we separately develop two DNN regression models using half of the entire training set and a limited training set built with over-estimation frames and some normal frames to improve the performance at the over-estimation frames. Since the outputs of the classification model are probabilities of either a normal frame or an over-estimation frame, respectively, two regression models are adjustably combined by using the probabilistic weights; thus, the final output of the DNN ensemble is the weighted sum of two estimated sub-band energies. As a result, artificial noise is significantly reduced, yielding improved speech quality. The proposed method is objectively and subjectively evaluated by comparing it with conventional approaches.

Introduction

In traditional narrowband (NB) speech codec-based digital speech transmission systems [1], the frequency band is limited from 300 Hz to 3.4 kHz. Recently, many services use a wideband (WB) speech codec, which covers the frequency bandwidth from 50 Hz to 7 kHz [2]. However, network support for the WB codec and terminal devices replacements have been required to use the WB service, and bandwidth extension (BWE) algorithms with NB speech signals have been proposed for this transition period to provide speech communication services close to WB.

BWE algorithms can be broadly categorized as either using side information which is extracted on the encoder side or only using NB speech information. In the case of using side information, high-band (HB) feature vectors are extracted and embedded in additional bits, whereas the BWE algorithms with only NB speech information can be applied without modifying existing transmission systems. Thus, BWE with only NB speech information is more challenging than those that use side information [3]. BWE without side information is called blind BWE or stand-alone BWE and can be further divided into two categories. One is based on a speech production model using linear predictive coding (LPC) coefficients [4], and the other directly estimates the HB spectra in the frequency domain using extrapolation [5] or machine learning [6]. Especially, deep neural network (DNN) regression model-based BWE algorithms that can estimate the log power magnitudes of the high-frequency band were first proposed in [7].

Recently, many studies have been reported using DNN models. For example, DNN was used for estimating HB cepstrum with log magnitudes of the NB speech signal and autocorrelation function coefficients in [8], [9]. Furthermore, convolutional neural network (CNN), recurrent neural network (RNN), convolutional recurrent neural network (CRNN), and generative adversarial networks (GAN) models were employed for estimating HB spectra in [10], [11], [12], [13]. More recently, WaveNet, a dilated convolution layer-based waveform generative model which was firstly proposed in the speech synthesis domain, was adopted for speech bandwidth extension [14]. Deep learning-based methods turn out to improve the performance of BWE by effectively estimating the high-frequency band. However, they generally have two problems. One is over-smoothing [15], and the other is over-estimation. Especially in the BWE tasks, improving the over-estimation problem is more critical because over-estimation generates artificial noise, which sounds very annoying to the listener [16]. Although improving clarity and reducing artificial noise have a trade-off relationship, many studies have focused only on improving clarity and have not considered the occurrence of artificial noise.

In this paper, we propose a DNN ensemble for the BWE task that can significantly improve speech quality by reducing artificial noise caused by over-estimation of HB energies. To do this, two different DNN regression models yielding sub-band energies are trained; one is trained using the half of the entire training set, and the other is trained using a limited training set composed of the over-estimation frames and some normal frames. Additionally, the DNN classification model that determines the probability of each DNN regression model is also trained. Finally, we estimate sub-band energies of the HB by calculating the weighted sum of the outputs of the two sub-band energies with probabilities. The experimental results show that the proposed approach provides better speech quality than the conventional DNN-based speech bandwidth extension algorithm.

The remainder of this paper is organized as follows. In the next section, we review the conventional methods for speech bandwidth extension. In Section 3, we describe the design of the proposed DNN ensemble for reducing artificial noise. The experiments and results of the proposed method are discussed in Section 4, and conclusions are presented in Section 5.