Fig. 1. DET plots for all primary systems, for the NFI-TNO evaluation. Experimental conditions were Dutch language, 60-s model training segments obtained from one session, 15-s test segments. The boxes and circles indicate the actual and minimum detection cost operating points, where the boxes represent the 95% confidence intervals.

Fig. 2. DET plots for all primary systems, for the NIST 2004 evaluation. Experimental conditions were English language, 1 conversation side model training duration, and an equal test segment duration. The boxes and circles indicate the actual and minimum detection cost operating points, where the boxes represent the 95% confidence intervals.

Fig. 3. Effect of training duration (30, 60, 120 s) on EER, for system T3 in NFI-TNO.

Fig. 4. DET curves for all conditions, formed from the test duration levels 10 s, 30 s, and 1 side, and the training duration levels 10 s, 30 s, 1 side and 8 sides. These are the results for system S9 in the NIST 2004 evaluation.

Fig. 5. DET performance as a function of duration condition for system S2.

Fig. 6. The effect of number of sessions in model training on the EER for all systems. Data points within a system are connected for visibility, using the same line type scheme as in Fig. 1.

Fig. 7. The effect of handset variation for a system in NIST 2004, which shows typical behaviour.

Fig. 8. The effect of handset variation for another system, which shows more robustness to handset variation.

Fig. 9. EER in the several conditions for all systems in NFI-TNO. From left to right are two conditions of Dutch, English, and two cross language conditions. The condition ‘xl-dt’ are trials with Dutch test segments and models trained on other languages, the condition ‘xl-dm’ has Dutch models and test segments in other languages. Data points within a system are connected for visibility, using the same line type scheme as in Fig. 1.

Fig. 10. The effect of language mix in NIST 2004 for system S8. Restricting to same language trials generally aids performance, while restricting to English-only trials is advantageous to systems utilizing lexical information and the English ASR transcripts such as this system.

Fig. 11. Restricting trials to English-only is not advantageous to systems relying only on acoustic information, such as here in system S2.

Fig. 12. Separation of target trials by English/other language, for a contrastive submission of site S11.

Fig. 13. Separation of non-target trials by English/other language, for the system shown in Fig. 12.

Fig. 14. Separation of target trials by other spoken language (“accent”), for system S1. All trials contained only English speech.

Fig. 15. Separation of target trials by transmission type, for the system shown in Fig. 12.

Fig. 16. Separation of non-target trials by transmission type, for the same system as in Fig. 15.

Fig. 17. NIST 2004 DET performance for various summed channel conditions, for system S9.

Fig. 18. DET performance for summed channel conditions for system S11, separated for impostor trial speakers as to whether they occurred as conversation partner in one of the three training conversations.

Fig. 19. DET performance of summed channel data for different gender mixes, for system S11. The curves separate the gender mix in test segments.

Fig. 20. DET performance of summed channel data for different gender mixes, for system S11. The curves separate the gender mix in training segments.

Fig. 21. DET performance of summed channel data for different gender mixes, for system S11. The curves separate the gender mix in both training and test segments.

Fig. 22. Comparison of the EER for the four systems that participated in both NIST 2003 and NFI-TNO evaluations. The NFI-TNO EERs have been corrected for the difference in speech duration from NIST.

The cost parameters for the NIST and NFI-TNO evaluations

The normalization factor has been included in the cost parameters C.

A comparison of the conditions of the various NIST and the NFI-TNO evaluation

ASR WER, automatic speech recognition transcripts, word error rate.

Design of duration and summed channel condition

Numbers indicate how many sites participated in the condition. The primary condition is indicated in bold type. The bottom and right margin are 2-speaker detection task conditions.

Actual and minimum detection costs for the NFI-TNO evaluation, as well as the equal error rate (EER)

Standard errors (SE) are indicated. The number of trials in this condition were N_tar = 521 and N_non = 9676. Here, .

Actual and minimum detection costs for the 24 systems in the NIST 2004 evaluation, as well as the EER

Standard errors (SE) are indicated for the EER. The number of trials in this condition were N_tar = 568 and N_non = 4634. Here, .

Analysis of deviance for the effects of system and language

The numbers are taken from the analysis program, not all decimal places are relevant. ‘Df’ means ‘degrees of freedom’.

Pairwise comparison of the language effect on EER