IUPAC/CITAC Guide: Classification, modeling and quantification of human errors in a chemical analytical laboratory (IUPAC Technical Report)

1.1 Scope and field of application

This Guide is developed for the study of human errors in chemical analysis and the reduction of the risk of such errors.

The document is intended for quality control, measurement, and testing (chemical analytical) laboratories, for accreditation bodies, laboratory customers, regulators, quality managers, metrologists, and analytical chemists–analysts.

1.2 Terms and definitions

Terms and definitions used in this Guide correspond to JCGM 200 (VIM) [20], ISO 3534 [21], ISO 9000 [22] and ISO Guide 73 [23]. The most relevant definitions relating to human errors in chemical analysis are given here.

1.3 Symbols

Δ j j ′ ( i )

synergy of two quality system components j and j′ (j′≠j) in blocking human error by scenario i

E*

score of the effectiveness of the quality system in decreasing likelihood and severity of human errors

f _HE

fraction of the quality of the measurement/test results which may be lost due to residual risk of human errors

i

index of a human error scenario

I

number of human error scenarios possible in a chemical analytical process

j

index of a component (layer) of a laboratory quality system

J

number of components of a laboratory quality system

k

index of a kind of human error

K

number of kinds of human errors possible in a chemical analytical process

l _i

expert judgment on severity (loss of quality of test results) of human error scenario i

L*

severity score

p _i

expert judgment on likelihood of human error scenario i

P*

likelihood score

q _j

effectiveness of quality system component j in decreasing likelihood and severity of human errors

q j ∗

score of the effectiveness of component j

q ˜ m ∗

score of the quality system effectiveness at step m of a chemical analytical process

Q

quality

Q _res

resulting quality

m

index of a step of a chemical analytical process

M

number of steps of a chemical analytical process

n _MC

number of Monte Carlo trials

r _ij

reduction of likelihood and severity of error scenario i as a result of error blocking by quality system layer j, degree of their interaction

r ˜ i j

reduction value taking into account the synergy factor

r*

score of the risk reduction

R*

score of the residual risk

s _ij

synergy factor of quality system component j with the other system components in blocking error scenario i

t

time

u

standard measurement uncertainty [20]

u _HE

standard uncertainty caused by residual risk of human error

u _HE-r

relative standard uncertainty caused by residual risk of human error

u _r

relative standard measurement uncertainty [20]

u _res

resulting standard measurement uncertainty, including the human error contribution

u _res-r

resulting relative standard uncertainty, including the human error contribution

1.4 Abbreviations

CAS number

unique numerical identifier assigned by Chemical Abstract Service (CAS) to every chemical substance

CITAC

Cooperation on International Traceability in Analytical Chemistry

CODEX

Alimentarius Commission – international organization developing food standards, guidelines and related documents, named Food Book (Codex Alimentarius in Latin)

CRM

certified reference material

FPD

flame photometric detector

GC

gas chromatography

ICH

International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use

ICP-MS

inductively coupled plasma mass spectrometry

ISO/TS

International Organization for Standardization Technical Specification

JCGM

Joint Committee for Guides in Metrology

Lab

laboratory

LC

liquid chromatography

MRL

maximum residue limit

MS

mass spectrometer

MW

molar mass

NIST

National Institute of Standards and Technology, US

pmf

probability mass function

QuEChERS

Quick, Easy, Cheap, Effective, Rugged and Safe method for sample preparation in pesticide residue analysis

RSD

relative standard deviation

SD

standard deviation

SOP

standard operation procedure

VIM

International vocabulary of metrology – Basic and general concepts and associated terms [20]

XSD

halogen selective detector

2.1 Commission errors

Errors of commission (mistakes and violations) are inappropriate actions resulting in something other than what was intended [5]. An example is the choice of a chemical analytical method for a reference material homogeneity study having a reproducibility standard deviation larger than the standard deviation of the method for which the reference material is intended.

2.2 Omission errors

Errors of omission (lapses and slips) are inactions contributing to a deviation from the intended path or outcome [5]. For example, such an error occurs when a column from a previous chemical analysis is forgotten in the chromatograph and not replaced by the column required in SOP.

2.3 Mapping human errors

Mapping human errors, as potential hazards which may occur at every step of chemical analysis, is necessary for quality risk management. Such a map is shown in Fig. 1. Kinds of commission errors – mistakes and violations – are listed and marked by brackets in the left part of Fig. 1. Omission errors are in the right part of the figure. Pointers show links between human errors and steps of the chemical analysis. There are, for example, the following main steps: 1) choice of the chemical analytical method and corresponding SOP, 2) sampling, 3) analysis of a test portion, and 4) calculation of test results and reporting. However, after sampling in many chemical analytical methods, sample preparation is required, including freezing, milling, and/or decomposition. The chemical analysis may start from an analyte extraction from a test portion, and separation of the analyte from other components of the extract. The analyte identification and confirmation are important in some cases. Then only calibration of the measuring system and quantification of the analyte concentration are relevant. On the other hand, choosing an analytical method and SOP may not occur in a laboratory where only one method and corresponding SOP are applied for a specific task. Many chemical analytical laboratories are not responsible for sampling, etc.

Fig. 1:

A map of human errors in chemical analytical process. Kinds of mistakes and violations are marked by brackets. Pointers show links of errors to steps of the analytical process. Nets indicate error scenarios. Adapted from Ref. [4].

In general, the kind of human error k=1, 2, …, K, and the step of the chemical analytical measurement/testing process m=1, 2, …, M, in which the error may happen (location of the error), form the event scenario i=1, 2, …, I. Different scenarios at the same location are shown in Fig. 1 as a net. There are a maximum I=K×M scenarios of human errors on the map. As K=9 in this Guide, I=9M.

Examples of mapping human errors in pH measurement of groundwater, multi-residue pesticide analysis of fruits and vegetables, and ICP-MS analysis of geological samples are provided in Annex A. A part of the error scenarios adduced in this annex may seem trivial for professionals in pH-metry, food analysis, and/or chemical analysis in geology. However, the fact is that people make trivial errors in their day-to-day life.

3.1 Approach

Historically, the first approach to human error modeling was the person approach. It was focused on unsafe acts of an analyst (a “bad apple” in the laboratory staff) arising from his/her forgetfulness, inattention, poor motivation, carelessness, negligence, and recklessness. Reducing unwanted variability in the analyst’s behavior includes disciplinary measures; naming, blaming and shaming; more rules; and more automation. The bad apple model is simple and easy to implement. However, it leads to conflicts in a laboratory and, finally, is not effective [3, 31]. The person approach is not used in this Guide.

The assumption of the later system approach is that analysts do not come to their laboratory to make errors. As no one can change human nature, error countermeasures in this approach are based on the laboratory quality system defense. The model, named “Swiss cheese”, is widely applied in the system approach [1, 32] and is implemented in this Guide.

3.2 Swiss cheese model

A quality system in a chemical analytical laboratory should prevent, block, or impede errors of analysts. The system includes the following main defensive components/layers: 1) validation of the chemical analytical method and SOP formulation; 2) training analysts and proficiency testing; 3) quality control with monitoring Shewhart charts of results of reference material analysis, internal and external inspections, etc.; and 4) supervision. Other components are also possible. In general, there are j=1, 2, …, J such quality system components/layers as shown in Fig. 2.

Fig. 2:

A laboratory quality system against human errors in the house of security. Human error scenarios are indicated by pointers i=1, 2, …, I. Quality system components/layers are shown as the Swiss cheese slices j=1, 2, …, J. Estimates of reduction r_ij of likelihood and severity of error scenario i, as the result of interaction between the error and layer j, form here the interrelationship matrix. Adapted from Ref. [33].

None of these layers alone can prevent all human errors. Each system layer is imagined in this model as a slice of Swiss cheese in Fig. 2. The holes in the layers, like in the cheese slices, are the layer’s weak points, not blocking human errors. Unlike in the cheese, these holes can be opened, shut and shifted depending on analyte, matrix, analyst, and other conditions. The presence of holes in a layer will not lead to system failure, as a rule, since other layers are able to prevent a bad outcome. That is shown in Fig. 2 as the pointers blocked by the layers. In order for an incident to occur and an atypical test result to appear, the holes in the layers must line up at the same time to permit a trajectory of incident opportunity to pass the system (through its defect), as depicted in Fig. 2 by the longest pointer.

Examples of modeling human errors are available in Annex A.

4.1 Swiss cheese in house-of-security

The house-of-security with the Swiss cheese for quantification of human errors in chemical analysis consists of the following elements illustrated in Fig. 2:

1. list of human error scenarios i=1, 2, …, I – at the left wall of the house;

2. expert judgments on likelihood p_i of scenarios i=1, 2, …, I and their severity (loss of quality of chemical analytical results) l_i – at the right wall of the house;

3. list of quality system components j=1, 2, …, J considered as protective layers of the system – at the ceiling of the house;

4. interrelationship matrix of expert judgments about reduction r_ij of likelihood and severity of error scenario i as the result of interaction between the error and protective layer j – the main content of the house;

5. synergy Δ j j ′ ( i ) of two quality system components j and j′ (j′≠j) in blocking human error by scenario i – at the house roof;

6. scores of effectiveness q j ∗ of quality system component j, scores of effectiveness q ˜ m ∗ of quality system at step m of the chemical analysis, and scores of effectiveness E* of the quality system in whole in decreasing likelihood and severity of human errors – at the house floor.

4.2 Elicitation scale

The elicitation process of expert judgments [35, 36] on the likelihood of human error scenarios and other topics requires a scale able to transform the semi-intuitive personal knowledge and experience of the expert into discrete quantities. For example, the geometrical scale with three values (1, 3, 9) was used originally in the house-of-security approach [34] in order to emphasize the dramatic character of expert judgments for the security system of an organization. However, this scale may not be informative enough. On the other hand, when the scale from 1 to 9 is more detailed, the choice on the scale may be hampered. For example, at the arithmetic scale of nine values (1, 2, …, 9), the probability that an expert will choose the first value of the scale is about 30%, while the probability of the choice of the last scale value is <5%, according to Bernford’s law, applied in forensic fraud detection [37].

As such a scale is intended in the house-of-security approach for the separation of important events from the less important, the scale division should be clearly visible, i.e. the number of scale values should be limited. Therefore, the mentioned geometrical scale (1, 3, 9) was extended by 0 only for an unfeasible scenario, and the known scale of four values (0, 1, 3, 9) [38] was considered to be the optimal solution for a chemical analytical task.

4.3 Likelihood and severity

An expert in the chemical analytical method may estimate the likelihood p_i of error scenario i by the scale (0, 1, 3, 9): likelihood of an unfeasible scenario as p_i=0, weak likelihood as p_i=1, medium as p_i=3, and strong (maximal) likelihood as p_i=9.

For the characterization of human errors in an analytical method as a whole, the likelihood score P* (expressed in %), equal to the averaged and normalized likelihood value, is used:

The P* value can be interpreted as a kind of “intuitive” or “subjective” (mean) error probability [19] of human error by any scenario in measurement or testing by the chemical analytical method. For example, P*=10% means that human error may happen (on average) in one of 10 measurements by this method.

Severity of scenario i is estimated as expected loss l_i of quality of the test results, when the error by this scenario is not blocked. The same expert may estimate severity using the same scale: no severity as l_i=0, light severity as l_i=1, medium as l_i=3, and high (maximal) severity as l_i=9.

For the characterization of human error severity in the chemical analytical method overall, the following score L* (%), equal to the averaged and normalized severity value is used:

For example, L*=50% can be interpreted as the severity occurring when half of measurement results burdened with human errors cannot be corrected and the measurements should be repeated.

4.4 Interrelationship matrix

To characterize the quality system in detail, one should estimate the possible reduction r_ij of likelihood and severity of scenario i as a result of the error blocking by quality system layer j (degree of their interaction). Such estimation is again the task of the expert in the chemical analytical method. The judgments about the interaction between an error by scenario i and layer j can be formulated using four ensuing degrees: no interaction as r_ij=0, low as r_ij=1, medium as r_ij=3, and high (maximal) interaction as r_ij=9. The r_ij values form the interrelationship matrix with i=1, 2, …, I rows, and j=1, 2, …, J columns. An empty row i of the matrix (r_ij=0 for any j) means that the quality system is unable to prevent scenario i; an empty column j (r_ij=0 for any i) indicates the uselessness of layer j.

Three dimensions are used in Fig. 2 to show how an error scenario may fall into a weak point of a quality system component (a hole of a cheese slice): two dimensions for the slices and another dimension for the error scenario pointers, perpendicular to the slices. However, the error scenario numbers have only one dimension i, and the slice numbers have also one dimension j. Therefore, the interrelationship matrix of r_ij values is two-dimensional. The total number of the matrix entries is I×J.

4.5 Synergy of the quality system components

Blocking human error according to scenario i by a quality system component j can be more effective in the presence of another component j′ (j′≠j) because of their synergy Δ j j ′ ( i ) . For example, such component j is the training of analysts for the correct performance of a measurement and for avoiding the routine method violation in scenario i (with the purpose of shortening the measurement process and decreasing the necessary time). The training is more effective when the analytical method is validated and the SOP is already formulated, i.e. in the presence of the component j′. In this case the synergy is Δ j j ′ ( i )   =   + 1. When the synergy is absent, Δ j j ′ ( i )   =   0. Theoretically, the synergy can also be negative and Δ j j ′ ( i )   =   − 1 is possible. However, quality systems in chemical analytical laboratories do not contain, as a rule, antagonistic components.

The synergy of component j with the rest of the components of the quality system in blocking human error scenario i in whole can be characterized by the following factor:

The synergy factor value is 1≤s_ij≤2 when antagonistic components are absent and J≥2.

Taking into account the synergy factor, the interrelationship matrix is to be transformed, replacing r_ij by r ˜ i j = r i j   s i j in every cell ij of the matrix.

4.6 Effectiveness

Effectiveness of quality system component j in decreasing likelihood and severity of human errors is evaluated as

A score q j ∗ of effectiveness of component j relative to other components of the quality system, i.e. priority or importance of component j, expressed in %, is:

Calculation of the score values q j ∗ allows the evaluation of the quality system components for all steps of the chemical analysis together. However, an analyst may be interested to know which step m is less protected from errors, with the intent to improve it. A score q ˜ m ∗ (%) of the quality system effectiveness at step m of the chemical analysis was developed for this purpose [39]:

and i′=m+M(k–1) are the scenario numbers related to the same error location (step m) for all kinds of error k=1, 2, …, K. For example, for m=1, M=6 and K=9, there are i′=1, 7, …, 49.

A score E* of effectiveness of the quality system in whole can be calculated as relative to an ideal (virtual/imagined) quality system tending to zero defects. This system has the maximal degree of interaction, r_ij=9, of every human error by scenario i and every system component/layer or slice of the Swiss cheese, j. Thus, the effectiveness score, expressed in %, is:

Examples of the quantification are available in Annex A.

5.1 Risk reduction

As the risk of human error is a combination of the likelihood and severity of that error, their reduction r ˜ i j is the risk reduction. The r ˜ i j value can be normalized by dividing its multipliers r_ij and s_ij by their maximal values, 9 and 2, respectively. Averaging the normalized risk reduction values for the interrelationship matrix over all the error scenarios and quality system components leads to score r* characterizing the (mean) risk reduction by the laboratory quality system, expressed in %:

5.2 Residual risk and its consequences

A score of residual risk of human errors R* (%), which are not prevented/blocked or reduced/mitigated by the quality system, is:

The fraction of the quality f_HE of the analytical results which may be lost due to residual risk of human errors, expressed in %, is:

When an ideal laboratory quality system is able to prevent or block human errors completely, one has R*=0% and f_HE=0%: there is no loss of quality, the quality system effectiveness score is E*=100%. If a quality system is not effective at all or absent, E*=0%, as r ˜ i j = 0 for all i and j. Then R*=100% and f_HE=(P*/100%)L*. The extreme case of a complete loss of quality is theoretically possible when, in absence of a quality system (R*=100%), scores P* and L* reach also 100%, i.e. human errors are inevitable and destructive. Thus, f_HE=100% as well.

In practice, 0 % < f_HE < 100 %, and the residual risk of human errors can be interpreted as a source of measurement uncertainty when a human being is involved in the measurement process and this human interaction with the measuring system is taken into account. Such interpretation is discussed in Annex B. Examples of the calculation of the risks, their consequences for quality of analytical results, and corresponding contributions to uncertainty budget are available in Annex A.

6.1 Variability

Any expert is also a human being and the elicitation process (by which the expert is prompted to provide error likelihood, severity, and other estimates) is influenced by aleatory and epistemic uncertainty [40], intrapersonal conflicts [41], etc. Therefore, the evaluation of variability of the error quantification scores and relative risks due to the inherent expert’s hesitancy is important also.

The expert may feel a natural doubt choosing one of close values from the proposed scale: 0 or 1? 1 or 3? 3 or 9? One change of an expert judgment, for example, on the likelihood of scenario i from p_i=0 to p_i=1 and vice versa (p_i=0↔1) leads to a 0.2% change (shift) of the likelihood score P* according to formula (1) at I=54 scenarios, as in Annex A, Example 2. The change of the expert judgment p_i=1↔3 results in a 0.4% shift of the P* value. The maximum correction of the likelihood score P* for I=54 scenarios caused by one change p_i=3↔9 is 1.2%.

The influence of a judgment change on the score value increases with decreasing I [39]. Thus, for I=36 scenarios, as in Annex A, Examples 1 and 3, changing p_i=3↔9 leads to a 1.9% shift in the P* value. The same is true for severity score L*. However, the evaluation of the variability of other scores, depending simultaneously on more than one expert judgment for the same scenario i, is more complicated.

A detailed analysis of the score variability, as well as the variability of the corresponding loss of quality f_HE, based on Monte Carlo simulations, is presented in Annex C.

6.2 Specificity

The score values and the residual risk of human errors are specific for the chemical analytical measurement/test method and for the laboratory conditions, i.e. they may be different in different laboratories active in the same field and using the same method. On the other hand, it is impossible to expect an equal risk of human errors in measurement or testing using different methods, even in the same laboratory.

Changes in the laboratory environment, as well as in any quality system component or staff, require a re-evaluation of the quality of the analytical results, which may be lost due to residual risk of human errors f_HE.

The re-evaluation may indicate either an f_HE increase (e.g. due to the retirement of an experienced supervisor) or its decrease (e.g. due to the acquisition of a new, more accurate and more automated measuring system) [42].

6.3 Latent errors

Errors due to poor laboratory design, a defect in the equipment, or an unsuccessful management decision, not depending on the sampling inspector and/or the analyst/operator, are defined in ISO/TS 22367 [15], clause 3.5, as “latent errors”. Latent errors are not considered in this Guide.

6.4 Positive human factors

In non-routine chemical analysis (scientific research, development of new chemical analytical methods and instruments, etc.) human errors are also possible. At the same time, the most successful way of solving problems arising in such analysis is human as well. The knowledge and experience of analytical chemists, their activity, creativity, and other abilities, are the positive human factors that can help to overcome the problems [43].

Neither human errors in non-routine chemical analysis, nor positive human factors are discussed in this Guide.

A-1-1 Introduction and main steps of the pH measurements

There is a wide spectrum of methods available using different measuring equipment, from pH indicator strips and routine pH meters with thermo-compensation to multifunctional instruments (allowing measurements of pH together with some other water properties without probe changes) and pH opto-sensing flow injection analysis. In any pH measurement, human error may influence the quality of the measurement results. However, first of all, this is important for such an object as groundwater [33]. The problem is that pH of groundwater is influenced by the partial pressure of dissolved carbon dioxide, which is much larger than the atmospheric one. CO₂ degassing continues during the measurement process due to the water stirring. Therefore, time is necessary (about 10 min in a case study [44]) for obtaining the stable response of a measuring instrument when the drift of the response does not exceed 0.02 pH unit in 1 min.

Thus, the step m=1 in pH-metry is the choice of the method with corresponding equipment and SOP. The step m=2 is the sampling of groundwater. Then there is the proper pH measurement, step m=3. The last step, m=M=4, is the calculation of the test results and reporting.

A-1-2 The map of human error scenarios

A total of I=9M=36 scenarios of human error in pH measurement of groundwater are discussed below, based on the optimistic assumption that an analyst knows what the quantity pH is.

A-1-3 Elicited data and error quantification scores

The elicited data for the mapped 36 human error scenarios and the four components of quality system listed in clause 3.2 of this Guide, including the interrelationship matrix with 36×4=144 entries, are presented in Table 1.

The synergy of the validation of the measurement method and SOP formulation as component j=1 of the quality system, and training of analysts as component j′=2 of this system, is Δ 12 ( i )   =   + 1 for any i=1–24. The same is for validation and quality control, j′=3: Δ 13 ( i )   =   + 1 for any i=1–24. A synergy in the cases of malicious violations and omission errors (lapses and slips) is unreliable, i.e. Δ j j ′ ( i )   =   0 for any i=25–36 and any j′≠j, both are from 1 to 4. The supervision (j=4) synergy with other quality system components is also absent here, Δ 4 j ′ ( i )   =   0 for any i=1–36 and any j′=1–3. The synergy factor s_ij values calculated by formula (3) are presented in Table 1 as well.

The likelihood score calculated by formula (1) is P*=26%. The human error severity in this method, evaluated by formula (2), is L*=67%. Note, here the obtained values of P* and L* differ slightly from the 27% and 65%, respectively, published for the same pH measurements in Ref. [33]. A similar difference is also seen in other score values. The reason is that the unfeasible scenarios (i=29 and i=33) were not taken into account in Ref. [33] and the total number of scenarios was limited by I=34.

The values of score q j ∗ of effectiveness of the quality system components j=1–4 calculated by formulas (4) and (5) are q 1 ∗   =   14   % , q 2 ∗   =   37   % , q 3 ∗   =   15   % and q 4 ∗   =   33   % . Thus, the most effective here is training of analysts, while the least effective is validation of the measurement method and SOP formulation.

Effectiveness scores q ˜ m ∗ of the quality system at different steps m=1–4 of the pH measurements, calculated by formulas (6) are q ˜ 1 ∗   =   31   % , q ˜ 2 ∗   =   12   % , q ˜ 3 ∗   =   50   % and q ˜ 4 ∗   =   7   % . From these score values one can understand that the quality system is the most effective at the step of proper measurements and the least effective at the step of calculation and reporting of the measurement results. The score of effectiveness of the quality system in whole, calculated by formula (7), is E*=59%.

A-1-4 Residual risk and measurement uncertainty

The score of residual risk of human errors by formulas (8) and (9) is R*=62%. The percentage of the quality of the measurement/test results which may be lost due to residual risk of human errors is f_HE=10.8% by formula (10).

The standard measurement uncertainty reported for test item preparation in proficiency testing of pH measurement of groundwater [44] was u=0.10 (pH units). The contribution to the uncertainty budget caused by residual risk of human errors by formula (14), Annex B, is u_HE=0.05. Thus, u_HE is not negligible and not the dominant contribution in the uncertainty budget. The resulting uncertainty value by formula (13), Annex B, is u_res=0.11.

A-2-1 Introduction and main steps of the analysis

Investigation of atypical results of pesticide residue analysis in fruits and vegetables, in particular out-of-specification results exceeding maximum residue limits (MRL), based on metrological concepts showed that only a minor part of these results can be defined as metrologically-related, i.e. caused by measurement problems. The newest applications of chromatography and mass spectrometry using detailed libraries of mass spectra of pesticides, their metabolites and derivatives, cannot exclude human errors in the analysis. Moreover, human errors are the greatest source of failures in pesticide identification and confirmation, even if performed by the most diligent and intelligent analysts. Theoretical calculations of probabilities of false positive and false negative results of pesticide identification in real samples, based on the chemical structure of analytes and measurement uncertainty, are practically unacceptable, as they do not take into account mislabeling, contamination of a sample, inappropriate spikes, etc.

There are M=6 main steps m=1, 2, …, 6 of the analysis: 1) sampling, 2) sample processing, 3) sample preparation, further named “extraction”, 4) identification and confirmation of pesticides, further “identification”, 5) measurement of their amount in the extract, further “quantification”, and 6) calculation of the pesticide concentrations in the analyzed sample and reporting, further “reporting”.

Sampling is conducted by certified inspectors for official control according to the CODEX guidelines directly from the field, packing houses, and logistics centers before sending the product to the market. Laboratory (Lab) samples of different fruits and vegetables (1 kg usually) undergo the same general procedure at the Lab, starting from processing, i.e. homogenization/blending.

Sample preparation for gas chromatography (GC) is performed by the Mini-Luke method based on extraction of analytes with acetone from a test portion of 15 g taken from the homogenized laboratory sample. For liquid chromatography (LC) the QuEChERS method of sample preparation is used, employing extraction with acetonitrile from 10 g test portions.

The extracts are analyzed by GC system with mass spectrometer (MS). Electron ionization is applied in the MS in full scan mode. Two other GC systems, equipped with flame photometric (FPD) and halogen selective (XSD) detectors, respectively, are used to verify the results obtained by GC/MS screening, and also to detect those analytes that exhibit a better response to either XSD or FPD than can be achieved with MS in full scan mode. LC-amenable pesticides are determined by combining an ultra-performance LC system with an advanced tandem quadrupole MS system, operated in multiple reaction monitoring mode with electrospray ionization [39].

A-2-2 The map of human error scenarios

A total of I=9M=54 scenarios of human errors in multi-residue pesticide analysis of fruits and vegetables are discussed below.

A-2-3 Elicited data and error quantification scores

The elicited data for the mapped 54 human error scenarios, and the same four components of quality system as in Example 1, are presented in Table 2. The interrelationship matrix in Table 2 has 54×4=216 entries. The synergy between quality system components was taken into account similar to that in Example 1. The only exceptions are the cases of reckless violations, malicious violations, and omission errors, for which any synergy was considered unreliable in this analytical task. In other words, Δ j j ′ ( i )   =   0 for any i=31–54 and any j′≠j.

The likelihood score calculated by formula (1) here is P*=19%. The human error severity, evaluated by formula (2), is L*=84%. The following effectiveness score values q j ∗ of the quality system components are calculated by formulas (4) and (5): for validation – q 1 ∗   =   22   % , for training – q 2 ∗   =   26   % , for quality control – q 3 ∗   =   25   % , and for supervision – q 4 ∗   =   27   % . Thus, the most effective/important component of the quality system is supervision, followed by training, quality control, and validation.

The q ˜ m ∗ values calculated by formula (6) are: for sampling – q ˜ 1 ∗   =   7   % , for sample processing – q ˜ 2 ∗   =   9   % , for extraction – q ˜ 3 ∗   =   23   % , for identification – q ˜ 4 ∗   =   22   % , for quantification – q ˜ 5 ∗   =   29   % , and for reporting – q ˜ 6 ∗   =   11   % . These values show that the ability of the quality system to prevent human errors at sampling is minimal. This situation is caused by the fact that sampling is performed in the field, i.e. inspectors work mostly out of the Lab. The sample processing and reporting steps also require a more attention for the improvement of the quality system. Effectiveness of the whole quality system for all steps of the analysis by formula (7) is E*=71%.

A-2-4 Residual risk and measurement uncertainty

The score of residual risk of human errors by formulas (8) and (9) is R*=65%. The percentage of the quality of the measurement/test results which may be lost due to residual risk of human errors is f_HE=9.9% by formula (10).

The relative standard measurement uncertainty reported for tomatoes, for example, averaged for all analytes and expressed in % of an analytical result, was u_r=20% [45]. The contribution of the uncertainty budget caused by residual risk of human errors by formula (14), Annex B, is u_HE-r=10% relative. Thus, u_HE-r is not negligible and not the dominant contribution in the uncertainty budget, similar to the case of pH measurements of groundwater in Example 1. The resulting uncertainty value by formula (13), Annex B, is u_res-r=22% relative.

A-3-1 Introduction and main steps of the analysis

ICP-MS is used widely in many laboratories for the chemical analysis of geological and other samples. There are typically four main steps m of the analysis with ICP-MS: 1) sample preparation, 2) calibration of the ICP-MS measuring system, 3) measurement of analyte concentrations in the prepared solutions, and 4) calculation of elemental mass fractions in analyzed samples and reporting (M=4).

The sample preparation of rocks and sediments is based on the fusion of the sample with lithium metaborate or sodium peroxide flux. Then the obtained bead is dissolved in nitric acid in an ultrasonic bath. The solution should be filtered and diluted with water to a sample/solution weight ratio in the range from 1:1000 to 1:4000. Samples of peridotites and a number of types of magma can be prepared by digestion of a sample with an HF–HNO₃ mixture in an ultrasonic bath. When samples contain resistant phases, e.g. zircon, the applied temperature and pressure are increased using microwaves or digestion bombs. Then samples are evaporated to incipient dryness, refluxed in nitric acid, evaporated and dissolved again, filtered, and diluted with water. For analysis of trace and rare earth elements the sample digestion with an HF–HClO₄ mixture under pressure can be applied. In any case, an analytical blank is prepared identically to the samples.

Synthetic and natural certified reference materials (CRMs) are used for the preparation of matrix matched calibrators of ICP-MS. The concentration of the acids and flux quantity in such calibrators should be the same as in the samples prepared for analysis. This is in addition to the known requirement of CRMs to have a composition close to the composition of the analyzed samples in order to minimize matrix effects. The CRMs (not the same as for calibration) are used also as internal standards and quality control samples.

A-3-2 The map of human error scenarios

In spite of achievements in instrument development there are still a number of human error scenarios which should be taken into account in a routine laboratory for quality risk management. A total of I=9M=36 scenarios of human errors in ICP-MS analysis of geological samples are discussed below [18].

A-3-3 Elicited data and error quantification scores

Elicited expert judgments on human errors by the described 36 scenarios and the same quality system components and their synergy, as in Example 1, are presented in Table 3. The interrelationship matrix here is also of 36×4=144 entries.

The likelihood score, summarizing the elicited judgments, is P*=22%. The human error severity score is L*=56%. The most effective/important component of the quality system by formulas (4) and (5) in this analysis is quality control ( q 3 ∗   =   27   % ) , followed by training ( q 2 ∗   =   26   % ) , validation ( q 1 ∗   =   24   % ) , and supervision ( q 4 ∗   =   23   % ) .

The q ˜ m ∗ calculation by formulas (6) for different steps of the analysis shows that the ability of the quality system to prevent human errors at the ICP-MS calibration ( q ˜ 2 ∗ = 14   % ) is minimal. At the measurement step q 3 ∗   =   25   % , at sample preparation q 1 ∗   =   30   % , and at calculation and reporting q 4 ∗   =   32   % . The effectiveness of the entire quality system for all steps of the analysis is characterized by E*=55%, calculated using formula (7).

A-3-4 Residual risk and measurement uncertainty

The score of residual risk of human errors by formulas (8) and (9) here is R*=65%. The percentage of the quality of the measurement/test results which may be lost due to residual risk of human errors is f_HE=8.1% by formula (10).

The standard measurement uncertainty reported, for example, for determination of 10 ng·g^-1 of ⁶⁰Ni in aqueous samples by ICP-MS [47] was u=0.75ng·g^-1. The contribution to the uncertainty budget caused by residual risk of human errors by formula (14), Annex B, is u_HE=0.32ng·g^-1. The resulting uncertainty value by formula (13), Annex B, is u_res=0.82 ng·g^-1.

As in Examples 1 and 2, u_HE here is also not negligible and not the dominant contribution in the uncertainty budget.

C-1 An expert judgment as a discrete quantity

An expert judgment for human error quantification is a discrete quantity that can take any scale value among (0, 1, 3, 9) according to the judgment probability mass function (pmf). When a value is chosen on the scale, the expert may still feel a doubt concerning neighboring scale values as pointed out in clause 6.1 of this Guide. Choosing 0, this expert thinks about 1 as a value which is also possible with equal or lower pmf. Choosing 1, the expert necessarily takes into account 0 and 3, but with equally lower pmf, etc. However, more distant scale values are not relevant. Otherwise, the expert is not experienced in the field and should not participate in the elicitation process.

On the other hand, the score values calculated directly from the elicited data can be interpreted as obtained when the expert judgments are completely confident, i.e. when a Dirac delta function, centered at a specific expert estimate on the scale, is applied as the pmf.

The following distributions modeling an expert behavior are studied below: 1) of completely confident expert judgments: the pmf at a chosen value is 1.00, whereas the remaining values on the scale have a total pmf equal to zero; 2) of confident expert judgments: the pmf at a chosen scale value is 0.90, whereas close values on the right and/or on the left on the scale have a total pmf equal to 0.10; 3) of reasonably doubting expert judgments: the pmf at a chosen value is 0.70, whereas close values on the scale have a total pmf equal to 0.30; and 4) of irresolute expert judgments: the pmf at a chosen value is 0.50, and the close values on the scale have a total pmf equal to 0.50 also. More pmf details are shown in Table 4. These four pmfs represent properly the whole range of cases from the most to the least confident expert judgments in the framework of the proposed modeling.

Sampling from the distributions for random generation of expert judgments as discrete values was performed using a code developed in R [18].

C-2 Algorithm of simulations

Since human error quantification scores P*, L*, q j ∗ , q ˜ m ∗ and E*, are calculated as algebraic combinations of the elicited expert judgments p_i, l_i, r_ij, and synergy factors s_ij (which, in the present context, are considered as entirely known), the probability distributions for these scores depend on the distributions of the expert judgments. Monte Carlo simulations of the score distributions were performed based on the following algorithm inspired by JCGM 101 [52]:

1. input of the elicited estimates p_i, l_i and r_ij, synergy factors s_ij, numbers K of kinds of human error, M of steps of the chemical analysis, I of human error scenarios, J of the laboratory quality system components, and the number of the Monte Carlo trials n_MC=100 000;

2. assignment of pmfs to the expert judgments p_i, l_i, r_ij;

3. simulation of possible values of expert judgments on human error by scenario i according to the chosen pmf on the scale values (0, 1, 3, 9) for i=1 to I: the matrix of simulated values is of dimension I×n_MC;

4. determination of the numerical distributions for scores P*, L*, q j ∗ , q ˜ m ∗ and E* by propagating the simulated distributions of the expert judgments into the relevant equations discussed in this Guide and evaluation of the score mean, median, and standard deviation (mean and median can be different because of a possible asymmetry in the simulated distributions);

5. plotting histograms for the distributions of the scores.

Note that for completely confident judgments with pmf by the Dirac delta function (for the scores calculated directly from the elicited data) the mean and median values obviously coincide, with the standard deviation of the simulated values being zero.

C-3 Examples

The elicited expert judgments on human errors in ICP-MS analysis of geological samples, Annex A, Example 3, Table 3, are used for the illustration of distributions of score values for error quantification with the Monte Carlo method [18].

Distributions of quality loss values due to residual risk of human errors are also studied with the Monte Carlo simulations [42]. All three sets of expert judgments on human errors from Annex A, Examples 1–3, are used here for the illustration of this application.

C-3-1 Distribution of score values for quantification of human errors

Results of the direct score calculations in comparison to the mean, median, and standard deviation of relevant score distributions simulated with the Monte Carlo method for ICP-MS analysis of geological samples are presented in Table 5.

Table 5:

The score values (%) calculated directly from the elicited data for ICP-MS analysis in comparison to those obtained by Monte Carlo simulations.

Score	Calculated directly	Monte Carlo simulations
		Confident expert			Reasonably doubting			Irresolute expert
		Mean	Median	SD	Mean	Median	SD	Mean	Median	SD
P*	22	24	23	2	26	26	3	29	29	4
L*	56	55	55	3	53	53	5	51	51	5
q 1 ∗	24	25	24	2	26	25	4	27	26	5
q 2 ∗	26	26	26	2	26	26	3	26	26	4
q 3 ∗	27	27	27	2	26	26	3	26	26	4
q 4 ∗	23	23	23	2	22	22	3	22	21	4
q ˜ 1 ∗	30	29	28	6	26	25	9	25	23	10
q ˜ 2 ∗	14	15	14	4	16	15	6	18	17	8
q ˜ 3 ∗	25	25	25	5	26	25	9	27	26	11
q ˜ 4 ∗	32	32	31	6	31	30	9	30	29	11
E*	55	54	54	3	53	53	4	51	51	5

SD is standard deviation of a simulated score value from its mean.

One can see from Table 5 that the mean score values of a confident expert are very close to the score values calculated directly (of a completely confident expert). The mean values, as well as the median values, change systematically depending on the confidence of the expert judgments, i.e. depending on the relevant pmfs. Accordingly, it makes sense that the corresponding standard deviations increase as the expert judgments become less confident. However, the fact is that all the mean (and the median) values of the simulated scores remain consistent with the score values calculated directly from the data within two such standard deviations.

It appears from Table 5 that a less confident expert may lead to larger estimates for likelihood score P*, from 22% to 29%, on average. Hence, the less confident the expert is, the more underestimated the P* value directly calculated from the data. When P* is increasing, the standard deviation is also increasing, from zero to 4%.

In spite of the equivalence of P* mean and median (rounded) values for a reasonably doubting expert, a minor asymmetry of the histogram in Fig. 4 is visible, probably due to the non-equidistant scale of the expert estimates/judgments through which the input pmfs were propagated. However, there are also other possible reasons for the asymmetry, e.g. when an expert unconsciously avoids one of the extreme choices on the scale (0 and 9).

Fig. 4:

A histogram of the likelihood score P* (%) in ICP-MS analysis, corresponding to judgments of a reasonably doubting expert, simulated by Monte Carlo method. Reproduced from Ref. [18] with permission from Elsevier.

A less confident expert leads to a reduction in the estimates of the severity from mean L*=56% to 51%, but again with an increasing standard deviation, from zero to 5%. There is no difference between mean and median for a reasonably doubting expert, as in the likelihood score.

From q j ∗ values in Table 5 one can find that the most effective/important component of the quality system (quality control) is characterized by the q 3 ∗ score values from 27% to 26% with a standard deviation from zero to 4%. There is no difference between the mean and the median of the simulated values for this score.

The q ˜ m ∗ simulated values for different steps of the ICP-MS analysis support the conclusion in Annex A, Example 3, that the ability of the quality system to prevent human errors at instrument calibration (m=2) is minimal. The q ˜ 2 ∗ score values are from 14% to 18% with a standard deviation up to 8%, depending on the expert’s confidence. The variability of q ˜ m ∗ scores is the largest in comparison to other scores in Table 5. A difference of (1–2)% between the mean and the median of the q ˜ m ∗ score values and an evident histogram asymmetry, as in Fig. 5 for a reasonably doubting expert, are observed.

Fig. 5:

A histogram of the score q ˜ 2 ∗ (%) of effectiveness of the quality system at second step of the ICP-MS analysis (the instrument calibration). This histogram corresponds to judgments of a reasonably doubting expert, simulated by Monte Carlo method. Reproduced from Ref. [18] with permission from Elsevier.

Effectiveness E* of the whole quality system for all steps of the ICP-MS analysis is from 55% to 51% with a standard deviation varying from zero to 5%. In general, E* tends to be overestimated in direct calculation from the elicited data, similar to L* (as opposed to P* tending to be underestimated) when the confidence of expert judgments is decreasing.

C-3-2 Distribution of possible loss of quality

The results of f_HE direct calculations and simulations for the model of reasonably doubting expert judgments are shown in Table 6.

Table 6:

Quality loss f_HE values (%) calculated directly from the elicited data in comparison to those obtained by Monte Carlo simulations for reasonably doubting expert judgments.

Chemical analytical method	Calculated directly	Monte Carlo simulations
		Mean	Median	SD
pH metry of groundwater	10.8	11.2	11.1	1.6
Pesticide residues analysis	9.9	10.4	10.4	1.3
ICP-MS of geo-samples	8.1	9.5	9.4	1.4

The f_HE values, calculated directly by formula (10) and interpreted as obtained from completely confident expert judgments, are close for the three examples. For all examples, the mean of the simulated f_HE values for the model of reasonably doubting expert judgments is a little larger than the f_HE calculated directly (not more than for one standard deviation of the simulated values). In other words, the estimated possible loss of quality due to residual risk of human errors is larger when an expert’s doubt is taken into account. A similar effect was noted in clause C-3-1 concerning scores of likelihood, severity, and quality system effectiveness: less confident expert judgments lead to less optimistic score values.

A histogram, for example, of f_HE simulated values for reasonably doubted expert judgments on human errors in elemental analysis of geological samples with ICP-MS is shown in Fig. 6. This histogram is practically symmetric, its mean and median values differing insignificantly.

Fig. 6:

A histogram of simulated f_HE values (%) in ICP-MS analysis, based on reasonably doubting expert judgments. ©Bureau International des Poids et Mesures. Reproduced from Ref. [42] by permission of IOP Publishing. All rights reserved.

C-4 Robustness

The score robustness for the quality risk management and improvement of a laboratory quality system can be considered satisfactory when a score’s relative variability, expressed as relative standard deviation RSD (a ratio of standard deviation to mean), does not exceed 0.4, rounded up from 1/3. In other words, a score is robust when the standard deviation of the expert judgments can be defined as insignificant in comparison to the score mean. Such a rule of 1/3 is used in metrology, e.g. for verification of weights and preparation of test items for proficiency testing. Practically the same rule is applied in spectroscopy for determination of limit of detection, as an analyte concentration equal to three standard deviations of the measuring system response for a blank (noise).

It is important also that the score’s relative range, i.e. the difference between the maximal and the minimal score values (calculated directly from elicited data and simulated values) related to their average, does not exceed the same 0.4 [18].