Statistical evaluation of data from multi-laboratory testing of a measurement method intended to indicate the presence of dust resulting from the collapse of the World Trade Center

Abstract

In this paper we describe a statistical analysis of the inter-laboratory data summarized in Rosati et al. (2008) to assess the performance of an analytical method to detect the presence of dust from the collapse of the World Trade Center (WTC) on September 11, 2001. The focus of the inter-lab study was the measurement of the concentration of slag wool fibers in dust which was considered to be an indicator of WTC dust. Eight labs were provided with two blinded samples each of three batches of dust that varied in slag wool concentration. Analysis of the data revealed that three of labs, which did not meet measurement quality objectives set forth prior to the experimental work, were statistically distinguishable from the five labs that did meet the quality objectives. The five labs, as a group, demonstrated better measurement capability although their ability to distinguish between the batches was somewhat mixed. This work provides important insights for the planning and implementation of future studies involving examination of dust samples for physical contaminants. This work demonstrates (a) the importance of controlling the amount of dust analyzed, (b) the need to take additional replicates to improve count estimates, and (c) the need to address issues related to the execution of the analytical methodology to ensure all labs meet the measurement quality objectives.

Appendices

Appendix 1: Scheffe contrasts for the slag wool inter-lab data

The method of Scheffe contrasts provides a basis for comparison of combinations of means in the analysis of variance that is quite general and conservative. The method can be applied to any linear combination of factor means in the analysis of variance where the sum of the coefficients in the linear combination equal zero. (Scheffe, H. [1959] The analysis of variance). This procedure is well suited to the evaluation of the data from the group of five labs (A, B, C, D, and H) versus the data from the group of three labs (E, F, and G). Normal probability plots of the residuals for the analysis of variance model for each batch that provide support for the assumption of approximate normality of the error term are shown in Fig. 3.

Scheffe contrast to compare the mean of group 1 labs A, B, C, D, and E to the mean of group 2 labs E, F, and G:

Define the contrast as

$$ L = \sum\limits_{{i = 1}}^8 {{c_i}{\mu_i}} $$

where

μ _i =:

True mean for ith lab

c _i =:

1/5 for i = 1, 2, 3, 4, and 8 (labs A, B, C, D, and H)

c _i =:

−1/3 for i = 5, 6, and 7 (labs E, F, and G)

The estimated contrast is then

$$ \hat{L} = \sum\limits_{{i - 1}}^8 {{c_i}{{\bar{Y}}_i}} $$

where ${\bar{Y}_i} =$ observed mean for the ith lab

The confidence interval for $L$ is

$$ CI = \hat{L}\pm fs\sqrt {{(\sum {c_i^2/{n_i})} }} $$

where

f=:

$\sqrt {{{f_{{k - 1,n - k,1 - \alpha }}}/\left( {k - 1} \right)}}$

s=:

residual standard error from the analysis of variance

k=:

the number of labs

n=:

total number of determinations at all labs

n _i =:

number of determinations at the ith lab

${f_{{k - 1,n - k,1 - \alpha }}}$=:

(1-α) quantile of a $F(k - 1,n - k)$ distribution

In this set-up for the slag wool inter-lab data, if the calculated value of the confidence interval CI does not include zero then the test provides evidence to reject the hypothesis at the (1-α) level that the group means are equal. For each of the slag wool batches, the indication is that the mean of the group 1 labs is greater than the mean of group 2 labs at the 0.95 significance level. This is consistent with the graphical evidence that shows (1) the group 2 labs report generally lower values for all batch levels and (2) report values that do not discriminate well among the different batches.

Appendix 2: Confidence intervals for the batch means

The results of the one-way ANOVAs by batch using the data from the group 1 labs were used to calculate confidence intervals for the batch means following the procedure described by Caulcutt and Boddy, Statistics for Analytical Chemistry, page 137. The results are shown in Table 3, above. The procedure is summarized below.

Confidence interval for the true mean of the ith batch:

$$ \mathop{{\bar{X}}}\nolimits_i { }\pm { }t{ }\sqrt {{{ }\frac{{\mathop{{{{\hat{\sigma }}_b}}}\nolimits^2 }}{\text{k}}{ } + { }\frac{{\mathop{{{{\hat{\sigma }}_w}}}\nolimits^2 }}{\text{ n}}}} $$

where

$\mathop{{\bar{X}}}\nolimits_i$ =:

overall mean for the ith batch

$\hat{\sigma }_b^2$ =:

estimated between-lab variance

$$ \frac{{\left( {{\text{between lab mean square }}--{\text{ within lab mean square}}} \right)}}{\text{number of determinations per lab}} $$

$\hat{\sigma }_w^2$ =:

estimated within-lab variance (residual mean square)

t =:

t statistic with α/2 = 0.025 and appropriate degrees of freedom

k =:

number of labs

n =:

total number of determinations

Degrees of freedom for t:

$$ DF \approx \left( {k - 1} \right)\mathop{{\left[ {1 + \left( {{\text{no}}{\text{. of determinations per lab}} - 1} \right){\text{WLMS/BLMS}}} \right]}}\nolimits^2 $$

where

≈:

approximately equal

k =:

number of labs

WLMS =:

with lab mean square

BLMS =:

between-lab mean square

Appendix 3: Reproducibility computation

The computational formula for reproducibility R, given by Caulcutt and Boddy (1983), page 143 is

$$ R{ } = { }t{ }\sqrt {{{ }2(\hat{\sigma }_w^2 + \hat{\sigma }_b^2)}} $$

where

$\hat{\sigma }_w^2$ :

estimated within-lab variance

$\hat{\sigma }_b^2$ :

estimated between-lab variance

t =:

t statistic with α/2 = 0.025 and appropriate degrees of freedom

The values for $\hat{\sigma }_w^2$, $\hat{\sigma }_b^2$, and t are computed as described in Appendix 2, above. From the 10% batch ANOVA using the data from group 1, the values are

$$ \hat{\sigma }_w^2 = 48,408,382 $$

$$ \hat{\sigma }_b^2 = 86,774,290 $$

$$ {t_{{0.025,6}}} = 2.45 $$

$$ R{ } = { }2.45{ }\sqrt {{{ }2(135,182,672)}} $$

$R{ } = { }40,285$ fibers/gram.