A. Data generation

The goal was to create a deep learning based metric that is robust across a large variety of speakers, noise environments, signal-to-noise ratios (SNR) and methods of (post-) processing or denoising of the analyzed audio sample. Spanning such a large range of audio types required us to create a large, labelled dataset with sound samples for each of these categories. We divided this data into five subsets: training set (27694 samples), validation set (3078 samples), SNR sweep set (140 samples), Speech-Noise-Distortion set (370 samples), and test set (4500 samples).

The training set contains 14063 unprocessed samples (i.e. without any compression or denoising) created by mixing speech and noise samples, and 384 noisy speech samples from real-world recordings. The speech and noise samples are single channel snippets of 4 sec duration. Most of them are in English but a considerable amount are in other languages. These are then mixed and synthesized to create both single and multi-channel unprocessed samples. All mixed samples are between –20 and 30 dB SNR, with 1% set to be without noise (Fig 2A). We chose this distribution to resemble real-life noise environments [27] with a slight shift towards lower SNRs since at high SNRs scores would be too high to be reliably differentiated. Towards the lower SNR extreme ends, the speech is barely audible or not at all. -5 dB SNR could be a busy restaurant and 5 dB SNR could be a person talking next to you with a busy street at some distance in the background. Above 20 dB SNR the background noise is almost inaudible. We also included samples without noise (SNR is infinite) to include the corner cases. The samples are processed with one or multiple of 649 different models of 49 types, including a variety of deep learning based denoising methods, classical denoising methods (like beamformer), and explicitly added distortions. The resulting samples are always single-channel, since the multi-channel samples are converted to single-channel by their processing, e.g. beamforming. The validation set (not used for training) used for ensemble selection is generated in the same way as the training set.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Data set and human ratings overview.

A) SNR distribution of all unprocessed samples of the training set (n = 14847 samples). B) Human ratings for all sound samples (with and without processing) in the training set, SNR sweep set, Speech-Noise-Distortion set, and the validation set (n = 31282). C) Speech-Noise-Distortion data set ratings across categories (n = 126, n = 126, n = 118), sorted by sub-sets with clean speech, samples with large amounts of noise, and samples with strong distortions and artifacts. D) Ratings across the SNR range using the SNR sweep dataset (n = 20 samples per SNR, i.e. 140 samples in total).

https://doi.org/10.1371/journal.pone.0278170.g002

The SNR sweep set (not used for training) contains only mixed speech and noise samples without any further processing. For given pairs of speech and noise, they are mixed with a range of SNR values ranging from -20 to infinity (only speech). This is useful to control the sound quality dimension (see definition below) as this should ideally only be impacted from adding noise when the noise directly interferes with the speech, e.g. when large amounts of white noise degrades the speech quality. However, the sound quality should not be impacted if an additional separate signal is partially overlapping, e.g. a siren or alarm.

The Speech-Noise-Distortion set (not used for training) contains 370 samples split into 3 categories: clean speech audio, very noisy audio that is not processed, and clean speech audio with strong processing artifacts. The processing augmentations included clipping, amplitude modulations, non-linear distortions and a comb filter. The resulting metric scores should be at the extreme ends of the individual categories of noise and sound quality. Lower scores for either noise or sound quality should also negatively impact the overall score, since both are required to be good for a good overall impression. We use these samples to test how correlated results of the metric are across these categories.

The test set (not used for training) contains 4500 sound samples and is constructed from the WHAMVox18 easy, WHAMVox hard (both freely available, including a detailed description of their construction), and the commonly used Valentini [28] set. We used 250 sound samples for each of the three data sets and processed them with 5 different methods, resulting in six (five processed, one unprocessed) times 750 sound samples. The 5 methods used were: 1) demucs [29] and 2) MHanet [30] (both current deep learning denoising systems), 3) SEGAN [31] (an older deep learning denoising system), 4) the dialogue isolate feature of the iZotope audio editing tool [32] (aimed at isolating speech), and 5) a Wiener filter [33] without particular fine-tuning. Those processing methods were not used in any other data set, to ensure a fair comparison between our metric and existing metrics.

B. Human ratings

B.1. Human rating procedure.

To obtain a reliable ground truth for each audio sample we crowdsourced ratings via Amazon Mechanical Turk. To ensure good consistency between the raters we provided detailed descriptions for each of the categories (overall quality, noise, and sound quality; Table 1), which are aligned with the international standard by the International Telecommunication Union ITU P.835 [1]. All categories are rated on a scale from 1 (worst) to 5 (best). Overall quality includes all aspects of a sound sample and should include all aspects of the sound file, be it artifacts in the speech, background noise or microphone quality. The noise category is focused on background noise and only refers to the amount of non-speech signal that is present. If the noise is loud enough to fully mask the speech, the sample should have a 1 as the noise rating, and if there is no background noise present, the noise rating should be a 5. The sound quality rating refers to the presents of distortions in the speech, artifacts (e.g. words cut off), or bad microphone quality. For example, a sample where many (parts of) words are chopped of that has no background noise should have a low sound quality score, a high noise score and a low to medium overall score, depending on the severity of the missing speech.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Definition and rating instructions for human raters.

https://doi.org/10.1371/journal.pone.0278170.t001

Human raters rated samples by category, first the overall quality for 50 samples, then noise for 50 samples and finally the sound quality for 50 samples. Audio samples are presented one at a time and raters are asked to rate them such that the ratings most closely align with the descriptions given in Table 1. To improve rating consistency, we also provided 2 sound examples per category for each of the three categories, i.e. 30 examples in total: 2 samples for each of the 3 categories, for each of the 5 scales. For each batch of samples.

We collected ratings from 342 individual raters, each of whom underwent a screening test to ensure high data quality. All raters were sourced from countries where English is a native language. The screening test consisted of two parts, was attempted by 1014 individual raters, and passed by 419 of them. Of those, 77 of the raters who passed the test did not complete more ratings after that, leading to the 342 final raters. The first part was a test to ensure that people were wearing headphones and was adapted from prior-art [34]. In the second part, participants rated approximately 10 samples for each of the three categories overall, noise and sound quality. The samples were chosen such that their rating is towards the extreme end of the scale, e.g. they either are of very good or very bad sound quality. For every sample, an acceptable range of ratings was defined. If a participant failed the headphone test, or too many ratings were outside the acceptable range, that participant was not allowed to provide more ratings. We did not directly test for hearing loss but assumed sufficient hearing since the screening test was passed. Screening ratings were not included in the dataset. Each sample is rated approx. nine times by different raters for the three categories, i.e. 27 ratings per sample. This results in a total of 777,642 ratings for the training set and 225,639 ratings for the various other sets (see Fig 1).

Since the ratings collected for the dataset were crowdsourced via Amazon Mechanical Turk, there was no direct contact with human participants as part of this study. We did not receive or collect any personal information of the raters. The raters and the authors gave consent to the Amazon Mechanical Turk guidelines, available under https://www.mturk.com/participation-agreement and https://www.mturk.com/legal-licenses.

C. Deep neural network architecture and training

Our metric is composed of an ensemble of neural networks, trained on the labeled dataset described above. We trained 20 non-intrusive networks, using a mean-squared error loss on the difference of the human rating/label and the rating predicted by the network. Each network in the ensemble consists of an encoder from one of two possible architectures followed by several dense layers and a fully connected output layer predicting each of the three rated categories (overall quality, sound quality and noise). The first architecture uses a 2-dimensional convolutional encoder in the frequency domain, while the second uses a pretrained wav2vec 2.0 network to embed the raw waveform.

In the convolutional network, the waveform is first converted into the frequency domain using the short-time Fourier transform (STFT) with a Hann window with a size of 512 samples and a step size of 128 samples. The complex and imaginary parts of STFT are stacked along channel dimension. The resulting input tensor is of shape (B, 503, 257, 2), where B is the batch size. The input is then passed through a ResNet [35] consisting of 14 convolutional layers with 32 filters with kernel size 3x3, stride 1 and rectified linear unit (ReLU) activations. The convolution layers are followed by a global average pooling layer and 2 dense layers with ReLU activations and 256 and 128 units respectively followed by a dense output layer with 3 output units and a tanh nonlinearity.

In the wav2vec network we first pass the raw waveform through a pretrained wav2vec 2.0 network [26] with frozen weights, treating the non-quantized feature vectors produced by the last layer of transformer encoder of the wav2vec model as the embedding. The wav2vec embedding is then processed through 2 dense layers with ReLU activations and 256 and 128 units respectively followed by a global average pooling layer and finally a dense output layer with 3 output units and a tanh nonlinearity.

In total we trained 9 convolutional networks and 11 wav2vec networks. All networks from a given architecture were identical except for the random seeds used to initalize the weights at the start of training. The networks were trained for 350,000 (convolutional networks) and 90,000 (wav2vec) steps, with a batch size of 32, using an Adam optimizer with a learning rate of 0.0003. Throughout training, the mean-squared error between the human ratings and metric predictions declines until training reaches a plateau (Fig A1 in S1 Appendix).

D. Deep neural network ensemble

A. Metric performance evaluation

The central requirement for a good computational speech metric is approximating human psychoacoustics well. The first columns for overall, noise, and sound quality in Fig 3 show Pearson correlations of our metric with human ratings. Correlations are measured on a hold-out test set that was rated by human raters, too and consists of different speakers, different environments and different noises to what our metric was trained on. Correlations up to 0.4 are too poor to be useful for most applications. Between 0.4 and 0.7 they could be use to make out a coarse trend in the quality but for correlations above 0.7, the corresponding metric can be considered useful for guiding algorithmic development. However, higher correlations are better, as some applications rely on precise feedback, e.g. guiding a neural network architecture search or even training a neural network directly on the rating predictions provided by the metric. Most of the correlations of our metric are between 0.8 and 0.9, which is more than sufficiently high to use the presented metric as a substitute for human ratings in many scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Pearson correlations of intrusive and non-intrusive metrics with human ratings on overall, noise, sound quality.

Ratings are subsets of the WHAMVox_easy set (n = 250), WHAMVox_hard (n = 250) and of the Valentini set (n = 250). Red shades indicate low correlations and blue shades high correlations. The numbers in bold are the highest correlations in each category and the numbers with an asterisk * have no statistically significant difference in correlation [36] to the highest based on the t-test.

https://doi.org/10.1371/journal.pone.0278170.g003

To compare the established intrusive metrics (CSIIm, COVL, ESTOI, CBAK, CSIG) and new non-intrusive deep learning based metrics (DNSMOSP835 [14] and NISQA [15]) to ours, we use a test set of samples with a variety of speech, noise and processing methods (Fig 3). We neither trained our metric on the speech or noise samples, nor on the processing methods used in the test set. In 51 out of 56 tests, our metric outperformed or had no statistically significant difference (using the t-test) to the best performing metric in each benchmark. In 54 out of 56 tests, our metric either had the highest correlation or was not significantly lower than the metric with the highest correlation compared to the other two deep learning metrics. The established metrics ESTOI [37] and COVL [38] are mostly capable of capturing the overall human ratings of the sound samples, while CBAK [38] and CSIG [38] mostly capture the noise and sound quality categories well, respectively. Note that our metric can estimate ratings better than those traditional metrics, without requiring a clean reference sample. In applications where a clean reference sample is available and the computing budget is highly limited, intrusive metrics such as CSIIm, CBAK, CSIG can provide a fast and reasonably accurate solution. The five out of 56 categories where our metric performed worse were tests using the SEGAN model for processing which creates an unusual type of artifact that non-intrusive metrics could not judge well. In all of the five tests higher correlations were achieved by traditional intrusive metrics (COVL and ESTOI) since they rely on a clean speech sample to better assess the actual speech quality. The two deep learning based metrics DNSMOSP835 and NISQA show mixed results and for overall and sound quality correlation scores they were significantly worse than the best performing metrics, on all 36 tests. In the noise category they were able to achieve 2 and 3 out of 18 correlations with human ratings that were not significantly different from our metric, respectively for DNSMOSP835 and NISQA. When comparing mean-squared errors (MSE) between the metric predictions and human ratings all show the same results for the best prediction and our metric outperformed or matched all other metrics on 52 out of 56 benchmarks (Fig A3 in S1 Appendix). Similar to the Pearson correlations, the other deep learning metrics NISQA and DNSMOSP835 showed significantly higher MSE in all 36 overall and sound quality benchmarks, and in 4 of the 18 noise benchmark scenes their MSEs were not significantly different from the MSE of our metric. Another way of comparing the metric results with human ratings are residual error distributions [36], here with a threshold of 0.4 (Fig A4 in S1 Appendix). Also, this analysis confirms the robust results of our metric, since it outperformed or matched all other metrics on 54 out of 56 benchmarks. Both NISQA and DNSMOSP835 showed particularly low correlations for the Valentini data set and Wiener filter processing. Possible explanations could be overfitting to a particular type of processing or dataset, or insufficient training samples on those types of audio. Note that our training set contains almost 15,000 unique noisy speech samples while e.g. DNSMOSP835 only uses 600 different noisy speech samples that are processed in different ways (see Fig A2 in S1 Appendix for the dependence of our metric on the fraction of samples used).

Note that prior to evaluation, each metric is normalized with a 3rd order mapping [36]. This is done since the score distribution of each MOS estimator tends to match that of its training MOS dataset, and those can have varying distributions based on how the ratings are collected. For example, if a MOS experiment contains mostly high quality samples, then raters will tend to score them lower than they would in a more balanced experiment (thinking they should use the whole range). Therefore, we follow the recommendation in ITU-T P.1401 [36] and apply a 3rd order mapping between predicted scores of an estimator and the MOS ratings, before evaluating the performance of the estimator. The mapping has the form f(y) = a + by + cy^2 + dy^3, where y are the raw predictions, and parameters a-d are set to minimize the RMSE between the MOS ratings and the mapped predictions, while being monotonously increasing over the range of raw predictions. One can think of this as an adaptation of an estimator to an MOS dataset that differs from its training MOS dataset, and this results in a fairer comparison between estimators.

To better understand metric differences and failure cases, we show scatterplots of the overall category in selected datasets and processing methods (Fig 4A with normalization, Fig A5 in the S1 Appendix without normalization). In the first row of samples without any processing (Fig 4A, baseline), all shown metrics, ours, ESTOI and DNSMOSP835, show the expected error pattern, estimate the trend correctly and mainly differ by their precision in this. When processing the samples with the deep learning based denoising system demucs (Fig 4A, second row), both our metric and ESTOI predict the ratings fairly well. DNSMOSP835, also work well for the WHAMVox sets but not for Valentini, where it degenerates to a mostly uniform rating. In our experience, this can happen when the metrics are trained either with too few examples with high overall quality, or when trained with too few medium-quality samples that are then processed by a strong denoising system. The third row of samples that are processed using a Wiener filter clearly illustrates the advantages of using deep learning based metrics. The Wiener filter was intended to simulate a failed experiment where the creators are trying to optimize a denoising system but did not fine-tune the filter well, resulting in an underwater-like noise. Both deep learning based metrics were able to differentiate samples of different qualities, while ESTOI degenerated to a uniform rating. Note that the difference in average estimates between the different dataset with Wiener filter on ESTOI sets is due to the normalization we applied. Overall, the metrics seem to handle failure cases gracefully in that they degenerate to a small range of values, which can be detected in real-world scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison of our non-intrusive metric with ESTOI (intrusive) and DNSMOS P.835 (non-intrusive) on human ratings on overall, noise, sound quality and comparison of human ratings with our metric.

A) The first row shows audio without processing, the second row shows audio processed with the deep learning-based denoising system demucs, and the third row shows audio processed with a Wiener filter. The plots are normalized to account for the individual differences [36] in the datasets (a comparison of the graphs without corrections is included in the appendix). B) Comparison of the average human ratings with our metric for the SNR sweep data set (n = 140) over an SNR range from -20 dB to clean audio, across the categories overall, noise, sound quality.

https://doi.org/10.1371/journal.pone.0278170.g004

Next, we want to confirm the broad applicability of our presented metric across a wide range of SNRs. A comparison of human ratings and ratings predicted by our metric on the SNR sweep set (Fig 4B) shows that the ratings are well correlated with correlations of 0.975, 0.978 and 0.929 for the overall, noise, and sound quality categories, respectively. The main inconsistencies occur for extreme sounds–clean or almost purely noise, but even there they stay within a score difference of 0.2. Compared to human ratings, our metrics network shows a slight bias towards the average ratings and tends to avoid more extreme ratings that are close to 1 or 5.

Additionally, to scaling well with the large dataset size (see S1 Appendix) we can further improve correlations with human ratings by basing our metric on an ensemble of deep learning models. Creating ensembles instead of using single models is expected to lead to better performance at the cost of computational complexity/processing time. To explore the tradeoff between accuracy and computational demand, we compared single neural networks and a range of ensembles that consist of two to ten neural networks (Fig 5). The best single model has an overall correlation of 0.928 with human ratings on the validation set, while the best ensemble has a correlation of 0.943, i.e. using an ensemble improves correlations by 1% - 2% compared to the best individual models. Our target for the metric is to be useful for evaluating systems that modify speech (e.g. speech denoising) and while it has to be executed fast enough to evaluate thousands of speech enhancement models, e.g. for an evolutionary search [7], the computational budgets for this use-case are sufficient to allow for ensembles instead of a single model. Therefore, we chose an ensemble that simultaneously exhibits high correlations at 0.941 and low 90^th percentile absolute error, with the 90th percentile absolute error being the 90th percentile value of the per file absolute errors. It contains 5 networks, with 3 using STFT inputs and 2 using wav2vec inputs. The ensemble captures the performance of our model on the more challenging examples of the dataset, similar to a lower bound on performance. While optimizing both correlations and absolute error, we are also choosing the ensemble to have low a computing complexity with 0.8 seconds of execution time per 4 second sample on an AMD Ryzen Threadripper 1920X 12-Core CPU which was limited to using 6 cores for processing to enable more tasks in parallel (70ms on an NVIDIA 2080 RTX). The computational load renders it sufficient to be used for tightly spaced algorithm evaluation on small to medium sized data sets, e.g. as evaluation during neural network training.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Ensemble size performance vs. computational complexity.

Computation times are based on the validation set (n = 3078 samples) calculated on an AMD Ryzen Threadripper 1920X 12-Core CPU but artificially limited to using 6 of those cores.

https://doi.org/10.1371/journal.pone.0278170.g005

B. Web API

To further speed up research in speech-quality sensitive areas such as speech enhancement or compression, we are making our metric freely available online via a web-based drag & drop interface and via a web API, both under metric.audatic.ai.

Using the web-based metric does not require any computational resources from the user since the computation is performed by the server. Multiple files (up to 15) can be uploaded at once and all files will be added to the queue of files to be evaluated. After the evaluation finished, their corresponding overall, noise and sound quality scores can be queried by the user. To prevent single users from monopolizing the API, there is a limit of 1 hour of audio files that can be submitted per user per 24 hours. A detailed explanation of the API usage can be found in the appendix and under metric.audatic.ai.

We encourage users to experiment with a variety of samples and report cases where ratings differ from the user’s opinion to the authors. We will use this feedback to continue to improve the metrics and continually increase its accuracy.