BioSecure Multimodal Evaluation Campaign 2007 (BMEC’2007)

Abstract

This chapter presents the experimental results from the mobile scenario of the BioSecure Multimodal Evaluation Campaign 2007 (BMEC’2007). This competition was organized by the BioSecure Network of Excellence (NoE) and aimed at testing the robustness of monomodal and multimodal biometric verification systems to degraded acquisition conditions. The database used for the evaluation is the large-scale multimodal database acquired in the framework of the BioSecure NoE in mobility conditions. During this evaluation, the BioSecure benchmarking methodology was followed to enable a fair comparison of the submitted algorithms. In this way, we believe that the BMEC’2007 database and results will be useful both to the participants and, more generally, to all practitioners in the field as a benchmark for improving methods and for enabling evaluation of algorithms.

Appendices

11.8 Equal Error Rate

The equal error rate is computed as the point where FAR(t) = FRR(t) (Fig. 11.16a). In practice, the score distributions are not continuous and a crossover point might not exist. In this case (Fig. 11.16b,c), the EER value is computed as follows

Fig. 11.16

FAR vs FRR curve: (a) example where EER point exists, (b) and (c) examples where EER point does not exist

$$EER = \left \{\begin{array}{lc} \frac{FAR({t}_{1})+FRR({t}_{1})} {2} & \textrm{if }FAR({t}_{1}) - FRR({t}_{1}) \leq FRR({t}_{2}) - FAR({t}_{2}) \\ \frac{FAR({t}_{2})+FRR({t}_{2})} {2} & \textrm{otherwise} \end{array} \right .$$

where

$$t1 {=\max }_{t\in S}\{t\vert FRR(t) \leq FAR(t)\},\ t2 {=\min }_{t\in S}\{t\vert FRR(t) \geq FAR(t)\}$$

and S is the set of thresholds used to calculate the score distributions.

11.9 Parametric Confidence Intervals

In this section, we present the parametric method used to estimate the confidence intervals on the FRR and FAR values. This method has already been explained by R.M. Bolle et al. [8].

Suppose we have M client scores and N impostor scores. We denote these sets of scores by $<Emphasis Type="Bold">\text{X}</Emphasis> =\{ {X}_{1},\ldots ,{X}_{M}\}$ and $<Emphasis Type="Bold">\text{Y}</Emphasis> =\{ {Y }_{1},\ldots ,{Y }_{N}\}$ respectively. In the following, we suppose that available scores are similarity measures.

Let \text{S} be the set of thresholds used to calculate the score distributions. For the set of client scores, \text{X}, assume that this is a sample of M numbers drawn from a population with distribution F, that is, $F(x) = Prob(X \leq x),x \in <Emphasis Type="Bold">\text{S}</Emphasis>$.

Let the impostor scores \text{Y} be a sample of N numbers drawn from a population with distribution $G(y) = Prob(Y \leq y),y \in <Emphasis Type="Bold">\text{S}</Emphasis>$. In this way, FRR(x) = F(x) and $FAR(y) = 1 - G(y)$, x and y ∈\text{S}.

From now, we have to find an estimate of these distributions at some threshold t ₀ ∈\text{S} and then, we have to estimate the confidence interval for these estimations.

• The estimate of F(t ₀) using data \text{X} is the unbiased statistic:

  $$ \begin{array}{rcl} {\hat{\rm F}}({t}_{0}) = \frac{1} {M}\sum_{i=1}^{M}{\bf 1}({X}_{ i} \leq {t}_{0})& & 11.1 \\ \end{array} $$

  F̂(t ₀) is so obtained by simply counting the X _i ∈\text{X} that are smaller than t ₀ and dividing by M.

  In the same way, the estimate G(t ₀) using data Y is the unbiased statistic:

  $$\begin{array}{rcl} {\hat{\rm G}}({t}_{0}) = \frac{1} {N}\sum_{i=1}^{N}{\bf 1}({Y }_{ i} \leq {t}_{0})& & 11.2 \\ \end{array}$$

• In the following, let us concentrate on the distribution F. For the moment, let us keep x = t ₀ and let us determine the confidence interval for F̂(t ₀).First define Z as a binomial random variable, the number of successes, where success means (X ≤ t ₀) is true, in M trials with probability of success $F({t}_{0}) = Prob(X \leq {t}_{0})$. This random variable Z has binomial probability mass distribution:

  $$P(Z = z) = {M}\choose{z}F{({t}_{0})}^{z}{(1 - F({t}_{0}))}^{M-z},\textrm{ }z = 0,\ldots ,M$$

  The expectation of Z is E(Z) = MF(t ₀) and the variance is ${\sigma }^{2}(Z) = MF({t}_{0})\ (1 - F({t}_{0}))$.From this, it follows that the random variable Z∕M has expectation F(t ₀) and variance $F({t}_{0})(1 - F({t}_{0}))/M$. When M is large enough, using the law of large numbers, Z∕M is distributed according to a normal distribution—i.e., $Z/M \sim N(F({t}_{0}),F({t}_{0})(1 - F({t}_{0}))/M)$.

  It now can be seen that Ẑ∕M=F̂(t ₀). Hence, for large M, F̂(t ₀) is normally distributed, with an estimate of the variance given by:

  $$\begin{array}{rcl} \hat{\sigma }({t}_{0}) = \sqrt{\frac{{\hat{\rm F}} ({t}_{0 } )(1 - {\hat{\rm F}} ({t}_{0 } ))}{M}} & & 11.4 \\ \end{array}$$

  So, confidence intervals can be determined. For example, a 90% interval of confidence is

  $$\begin{array}{rcl} F({t}_{0}) \in [{\hat{\rm F}}({t}_{0}) - 1.645\hat{\sigma }({t}_{0}),{\hat{\rm F}}({t}_{0}) + 1.645\hat{\sigma }({t}_{0})]& & 11.5 \\ \end{array}$$

  Estimates Ĝ(t ₀) for the probability distribution G(t ₀) using a set of impostor scores \text{Y} can be obtained in a similar fashion. Parametric confidence intervals are computed by replacing F̂(t ₀) with Ĝ(t ₀) in Eqs. 11.4 and 11.5.

11.10 Participants

AMSL Otto-von-Guericke-Universitaet MagdeburgAdvanced Multimedia and Security Lab, Biometrics GroupPO Box 4120, Universitaetsplatz 239106 Magdeburg - Deutschland

Balamand University of BalamandDeir El-Balamand, El-Koura, North Lebanon

EPFL Ecole Polytechnique Fédérale de LausanneSpeech Processing and Biometrics GroupEcublens, 1015 Lausanne - Switzerland

Swansea University of Wales Swansea, Speech and Image Lab.Singleton Park - SA2 8PP, Swansea - United Kingdom

TELECOM-ParisTech (formerly GET-ENST), Département TSI46 rue Barrault, 75013 Paris - France

TELECOM SudParis (formerly GET-INT), Département Electronique et Physique 9 rue Charles Fourier, 91011 Evry Cedex 11 - France

UAM (formerly UPM), Universidad Autonoma de MadridBiometrics Research Lab., Ctra. Colmenar, km. 15, E-28049 Madrid - Spain

UNIFRI University of Fribourg, Informatics Department, DIVA groupChemin du Musée 3, 1700 Fribourg - Switzerland

UNIS University of Surrey, Centre for Vision, Speech and Signal ProcessingGU2 7XH Guildford - United Kingdom

UniTOURS Université Fran\c{c}ois Rabelais de Tours, Laboratoire d’informatique64 avenue Jean Portalis, 37200 Tours - France

UVIGO Universidad de Vigo, Teoria de la Senal y Comunicaciones ETSI Telecomunicacion, Campus Universitario, 36310 Vigo - Spain