External quality assessment programs in the context of ISO 15189 accreditation

Introduction

In laboratory medicine, external quality assessment (EQA) plays a crucial role in the harmonization and standardization processes by ensuring the evaluation and monitoring of the comparability of test results across different laboratories and over time [1], [2], [3], [4]. The purpose of EQA programs includes (a) the evaluation of laboratory performance for specific tests and its continuous monitoring, (b) the identification of interlaboratory differences, (c) the evaluation of method/diagnostic system performances, (d) the degree of comparability between methods/diagnostic systems and (e) the monitoring of the success of harmonization/standardization efforts for improving results comparability [5]. Moreover, if recognized reference laboratories identify the assigned value through an approved reference measurement procedure (RMP), EQA information can highlight the metrological traceability of methods/diagnostic systems. Many providers of external quality assessment schemes (EQAS) also supply information to the laboratory on postanalytical aspects (e.g. comparability between measurement units or between reference ranges/decisional limits) when such data are required, reporting them using the same procedure that is used for issuing patient results (same measurement units included in the medical report). EQA providers can evaluate pre-analytical aspects by means of dedicated surveys because control materials call for a preanalytical treatment that is not the one used for patient samples [6], [7], [8], [9], [10], [11], [12], [13], [14], [15]. The reliability of the information provided in EQA reports is closely related to a proven commutability of the control materials adopted with a consequent avoidance that differences due to matrix effects are attributed to differences between methods/diagnostic systems.

In recent years, several publications dealing with quality assurance in laboratory medicine have stressed the importance of EQA programs and their correct management in order to provide objective evidence of the acceptability and reliability of examination results [16], [17], [18], [19]. Nevertheless, the management of EQAS still remains an open question for laboratory professionals for reasons reported in Table 1 [20], [21].

A process of harmonization is needed in order to make participation in EQAS effective and to guarantee the congruity of EQAS information provided by different providers.

The aim of the present study was to describe the critical aspects that EQA providers and laboratory professionals should control in order to assure the effective management of EQAS and compliance with ISO 15189 accreditation.

International Standard ISO 15189 and EQA

In section 5.6 (Ensuring Quality of Examination Results), the International Standard for the Accreditation of Medical Laboratories, ISO 15189:2012 [22], reports the requirements to be fulfilled in order to guarantee the correct management of procedures performed for internal quality control (IQC), the interlaboratory comparison programs (e.g. EQAS and proficiency testing) and the comparability of examination results. Moreover, it focuses on the need to implement appropriate assurance procedures to monitor pre- and postanalytical processes, such as the establishment of quality indicators (QIs) (section 4.14.7). Currently, IQC, EQAS and QIs are considered essential tools for medical laboratories in evaluating and monitoring their performance.

Specific EQA requirements, which are specified in ISO 15189 and can be considered the roadmap for implementing reliable procedures [22], are as follows:

• “the participation in the interlaboratory comparison programmes appropriate to the examination and interpretation of examination results” and, when they are not commercially available, the establishment of alternative approaches using appropriate materials (“such materials may include: certified reference materials; samples previously examined; material from cell or tissue repositories; exchange of samples with other laboratories; control materials that are tested daily in interlaboratory comparison programmes”);

• participation in interlaboratory comparison programmes that: (a) “substantially fulfils the relevant requirements of ISO/IEC 17043 [23]; (b) provides clinically relevant challenges that mimic patient samples and have the effect of checking the entire examination process, including pre-examination procedures, and post-examination procedures, where possible”;

• establishment of “a documented procedure..... that includes defined responsibilities and instructions for participation, and any performance criteria that differ from the criteria used in the interlaboratory comparison programme”;

• reviewing and discussing performance in EQAS with relevant staff and, when predetermined performance criteria are not fulfilled, implementing and recording corrective actions, and monitoring their effectiveness. The returned results are to be evaluated for trends that indicate potential nonconformities and preventive action to undertake;

• ethical conduct “using the same procedures as those used for patient samples. ...not communicate with other participants in the interlaboratory comparison programme about sample data until after the date for submission of the data...... and using confirmatory examinations before submission of the data” and not overdoing samples evaluated in different days.

Although the standard calls for all tests provided by the laboratory to undergo EQAS, this is difficult to achieve in some cases as mentioned above. The implementation of alternative procedures to “provide objective evidence for determining the acceptability of examination results” requires a professional competence with in-depth knowledge of measurand peculiarities, examination procedures and the purpose of the test and the impact of results on patient outcome.

External quality assessment schemes: weaknesses

Currently, EQA providers worldwide provide a report for each survey proposed, the aim being to inform each participating laboratory about its analytical performance, the performance of methods/diagnostics systems used and the comparability of results through the analysis of the statistical data of different methods/diagnostic systems. Moreover, some authors call to evaluate the analytical performance in the EQA report, not only in relation to the total error (TE) but also in terms of bias and imprecision to better address the investigation of error causes [24].

In order to guarantee the provision of reliable information, EQA providers must design schemes complying with approved guidelines and recommendations as well as with international standards [18], [25], [26], [27], [28], [29]. Awareness must be raised of the commutability of control samples, an important characteristic, in order to avoid any misinterpretation. In particular, commutability can affect the procedure for data processing and identification of the assigned value, the choice of analytical performance specifications (APS) and the evaluation of metrological traceability degree of analytical examination procedures to RMP.

However, it can be difficult for EQA providers to find commutable control samples for all tests included in the schemes, in addition to the increased workload incurred, and the costs. Any failure to achieve or define commutability for one or more measurands, or for a specific examination procedure, must be kept in mind when choosing results processing, identifying an assigned value and performance specifications. If the degree of commutability differs from one diagnostic system to another (same reagent, calibrator and instrument) and all are based on the same method (same methodological/technological principle, e.g. nephelometric, turbidimetric), the results should be processed as diagnostic system-related rather than method-related, or independently by any group (all results), in order to prevent erroneous evaluations being made on comparing statistical data from different groups [consensus value, interlaboratory variability, CV%, standard deviation (SD)]. Moreover, the evaluation of laboratory performances based on a diagnostic system-related consensus value, rather than a method-related consensus value, ensures that unsatisfactory performances, due to commutability problems, are not inappropriately assigned to laboratory procedures. When the assigned value is obtained with an RMP (reference value), the score between laboratory result and reference value can be incorrectly attributed to a different degree of metrological traceability of the examination procedure used rather than to the RMP. Whenever the reference value is not utilized, no considerations can be made concerning the metrological traceability order of the laboratory method/diagnostic system.

Further important items are statistical approach (parametric or non-parametric) for data processing, the use of mean or median as a consensus value and the estimation of assigned value uncertainty, which is an ISO 17043:2010 requirement.

The uncertainty of assigned value is a requirement of the ISO 17043:2010, and EQA providers “should have criteria for the acceptability of an assigned value in terms of its uncertainty... that have to be based on a goal to limit the effect that uncertainty in the assigned value has on evaluation, i.e. the criteria limit, the probability of having a participant receive an unacceptable evaluation because of the uncertainty in the assigned value” [23].

As yet, the “management” of the uncertainty of assigned values (UAV) in relation to performance evaluation has received little attention: if uncertainty is taken into consideration, how can the score between the laboratory result and the assigned value be calculated, and how can the APS be applied? However, the UAV in EQAS reports provides laboratories with useful information for the estimating measurement uncertainty (UM) of examination procedures (requirement of ISO 15189). The UAV, in fact, represents the uncertainty of the bias that is included in the formula for calculating the UM [30].

The choice of APS depends on the scope of the schemes and the aspect of analytical quality to be evaluated (TE, bias, imprecision); consequently, they depend on the target-setting procedure used. Schemes with an educational scope usually have more stringent limits than those with a regulatory scope, and because the consensus value is the most frequently used assigned value and laboratories make one determination, the aspect assessed is generally TE [29]. Figure 1 shows an example of how the laboratory can identify the causes of unsatisfactory performance if it is evaluated on the basis of both reference (target) value and the diagnostic system-related consensus value. Moreover, the example given highlights the way in which laboratory performance can change if the uncertainty value is added to, or subtracted from, an assigned value. The data reported in Figure 1 are results from three laboratories participating in the EQAS for glycated hemoglobin (HbA_1c) managed by the Centre of Biomedical Research (Padova, Italy) where control materials are fresh whole blood samples (commutable samples) and the APS used was chosen following clinical recommendations. Three performance limits are calculated on the basis of Fraser’s formula: optimum (Opt), desirable (Des) and acceptable (Accep) [31], [32]. The target value is assigned in fourfold by five approved IFCC network laboratories with IFCC RMP, and expanded uncertainty is reported at k=2. The consensus value is the median after the elimination of values outside ±3SD (SD=75th–25th percentile/1.349), and the expanded uncertainty at k=2 related to consensus value is calculated according to the ISO 13528:2015 [33]. The performance is obtained by comparing the score percentage of each result with the APS. The score percentage is calculated according to the formula: ([laboratory result−AV]/AV×APS) 100×100, where AV is the assigned value. When the score percentage is from 0 to ±50, the performance is Optimum (Opt), from +50 to +100 or from −50 to −100, desirable (Des), from +100 to +150 or from −100 to −150 , acceptable (Accep) and superior to +150, or inferior to −150 Unacceptable (Unac).

Figure 1:

Example of information reported in the EQA report of EQAS for glycated hemoglobin managed by the Centre of Biomedical Research (Padova, Italy).

The target value has been assigned in fourfold by five approved IFCC network laboratories with IFCC Reference Measurement Procedure (IFCC RMP). Consensus value is the median after elimination of values outside±SDrobust (SDrobust=75th–25th percentile/1.349). Expanded uncertainty of target and consensus value has been calculated at k=2, according to ISO 13528:2015.

The criteria for defining APS, which are now well established, were first recognized in the Conference of the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) on definition of APS, held in Milan in 2014 [34], but they should be chosen on the basis of the impact that the performance in question has on patient management. A different criterion should therefore be selected for each measurand. Experiences have shown that the most practical specifications to use are those based on biological variation and state-of-the-art criteria because clinical data are not always available [35], [36], [37]. However, in the interests of encouraging laboratories to achieve quality improvements, it should be borne in mind that, when the limits based on biological variation are too restrictive (high percentage of laboratories with poor performances) or too wide (high percentage of laboratories with good performance), they should be replaced with other limits that reflect the state of the art (e.g. multiples or under-multiples of those derived from Fraser’s formula, so that no more than 25% of the results of laboratories considered have an unacceptable level) [20]. This also applies to situations in which current technology allows the use of more restrictive goals, but the analysis of advantages to patient management does not justify the effort to maintain such a high level of quality.

Although the criterion suggested by ISO 15189 for the choice of a reliable EQAS is the accreditation of the EQAS provider in compliance with the ISO 17043 [23], this standard reports the requirements, but it does not specify which criteria and procedures the EQA providers are to follow. Consequently, several providers, both nationally and internationally, design their schemes on the basis of different choices, producing different information in each EQAS report.

The freedom to choose different criteria and procedures has given rise to differences between schemes managed by different providers. Calling for EQAS harmonization irrespective of location and choice of provider is the key to assuring reliability of information given in EQAS reports.

Medical laboratories: competence is needed

Due to lack of harmonization between EQAS, a laboratory participating in EQAS managed by two different providers for the same measurand may receive a non-comparable assessment of its performance. Laboratory professionals are therefore obliged to analyze all characteristics of available EQAS other than evidence of ISO 17043 accreditation (as required by ISO 15189). Different guidelines and recommendations, available in literature from several years, define the appropriate quality specifications (e.g. scheme characteristics, information to be provided in the participant report, provision of advice and education, independency and ethical conduct) to assure the suitability of an EQAS [38], [39]. The competence of laboratory professionals, which is crucial to attaining these ends, should include not only the knowledge of the characteristics affecting the reliability of EQAS and APS stated in approved documents but also the pragmatism needed to identify an alternative whenever the available EQAS are inadequate. The choice of the “right” EQAS does not, per se, guarantee its effectiveness [18]. In fact, both the fundamental steps of the analysis of information provided in EQA reports and the identification of improvement actions to be undertaken call for the involvement of all laboratory staff with a role in the specific activity. The individual working in a specific process can identify critical activities, all possible causes of errors, decide upon any adequate corrective actions required and implement the containment factors to obviate any recurrence of the error identified [40].

Another task required by ISO 15189 is the above-reported identification of alternative approaches when EQAS is unavailable. The possible approaches suggested in ISO 15189 are often difficult to follow due to the stability of sample matrix, the shortage of volume or the low frequency of the test request (“sample previously examined”), the difficulty in obtaining certified reference material for some tests in addition to related high costs (“certified reference materials, material from cell or tissue repositories”), the distance between laboratories that perform the rare/esoteric tests and the identification of bias acceptability between results determined on similar or different instrumentations (“exchange of samples with other laboratories”) and the lack of IQC managed in interlaboratory programs for these types of tests (“control materials that are tested daily in interlaboratory programmes”). A pragmatic approach should be used in order to guarantee the confidence of results, and where no approach suggested by ISO 15189 is possible, a clinical audit could be undertaken to investigate the satisfaction of physicians concerning the compliance of results with their clinical purpose.

The competence evaluation of operators on the basis of EQA performance is a practice sometimes used, but it can be questionable when it is not clearly defined how the human factor affects the performance, especially for automated analytical procedures. Otherwise, the evaluation of professional competence through EQA performances is highly effective when the laboratory result is expressed as an interpretative comment (e.g. morphological evaluation of peripheral blood cells). The participation in EQAS should involve each and every professional who issues interpretative comments, as occurs for patient samples, in order to evaluate not only the degree of professional competence but also the congruity between comments formulated by different professionals [41], [42].

Pathway towards harmonization

ISO 15189 accreditation emphasizes the importance of laboratory professionals’ aware participation in EQAS, calling for their competence and the achievement of performance improvement by means of a structured process planned and implemented on the basis of an appropriate root cause analysis and the implementation of corrective actions by all staff involved in the specific activities. In fact, although criteria and procedures on the implementation of suitable EQAS are available, it is recognized that it is not always possible to comply with them for all laboratory tests. Moreover, numerous EQAS providers are present on the market, proposing schemes based on designs, criteria and procedures that vary greatly.

Efforts for harmonization must be planned and must involve all stakeholders: EQAS providers and laboratory professionals, accreditation auditors and in vitro diagnostic (IVD) manufacturers. Moreover, the European Organization for External Quality Assurance providers in Laboratory Medicine (EQALM) has an important role in highlighting relevant issues for discussion in scientific meeting or working groups [www.eqalm.org].

The process of harmonization should start in defining if the quality specifications for schemes with educational scope have to be different or equal to those with regulatory scope. However, for educational schemes harmonization is supposedly easier to achieve, and for regulatory schemes the different peculiarities of national institutions make this difficult. As it is widely recognized that laboratory performances depend on all phases of the total testing process, not only on the intra-analytical phase, it should be considered whether the opportunity “to judge” the accuracy of laboratory results on the basis of the PT is best, or if it would be better to judge on the basis of an accreditation process such as the ISO 15189 [22].

EQA providers, working together to guarantee the harmonization of information provided and facilitate the choice of laboratories, should at least

• define the more suitable quality specifications to be implemented and universally adopted, in relation to each measurand;

• support laboratories in the identification of alternative approaches to follow when EQAS are unavailable;

• identify the information to describe in the reports that guarantee the comprehensibility of the causes of unacceptable performance, in particular when they are due to laboratory problems or inadequate performance of the examination procedure used.

Moreover, the collaboration between EQA providers allows the use of high-quality control samples and the verification of their commutability, and attains a more statistically significant level of data processing, thus reducing inherent costs. A further advantage is the potential to implement schemes for analytes for which an individual EQA organizer does not have enough participants to justify setting up a scheme, promoting quicker coverage of more analytes.

The choice of the providers concerning the more suitable quality specifications to be adopted has to be based on considerations of the test purpose in order to guarantee that the performance assessment best reflects the impact on test interpretation.

Laboratory professionals should strive to develop a well-structured system for the EQAS management that includes, for example, a laboratory operating procedure (SOP), checklists, and specific QIs. An SOP helps to identify the more appropriate EQAS on the basis of scheme characteristics, to maintain congruent behavior between different operators, to carry out all needed steps to achieve effective findings and to avoid the risk of the error underestimation. All operators follow the same criteria and procedures and pursue the quality objectives seeking the same goals. Moreover, supervision by a designated senior professional to validate the improvement actions and define the intervention priorities guarantees that this activity is integrated in the entire quality assurance system.

Specific checklists, useful tools for remembering all possible causes of error for analysis, guarantee the collection of all evidence necessary to demonstrate, and obviate, the causes of error. Figure 2 gives an example of a checklist used for identifying unsatisfactory performance in our laboratory since 2016. It reports some items proposed by other authors, demonstrating the validity of this approach for the root cause analysis [19]. The checklists should also be used when the performance, satisfactory and unsatisfactory, demonstrates a worsening in at least the last two surveys in order to prevent the occurrence of unacceptable performance. For all EQA results, the laboratory should evaluate the distribution of the scores. For example, if all results are above or below the assigned value of the acceptable range, a calibration problem may be present.

Figure 2:

Example of checklist for identification of causes of unsatisfactory performance in EQAS.

The suitability of the SOPs used for the internal management of EQAS should, furthermore, be monitored by specific QIs that, periodically evaluated during the year, allow the identification of the improvement needs of the procedures in place. Table 2 reports QIs proposed by the Working Group “Laboratory Errors and Patient Safety” (WG-LEPS) of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC), together with the results collected in the last few years [43]. A well-constructed EQA management plan will allow the laboratory to meet quality and accreditation requirements and to verify the measurement processes associated with the purpose of tests at a minimum of expense.

ISO 15189 accreditation auditors should be able to understand the choices made by laboratories and evaluate the competence of professionals in the management of EQA procedures. High competence is also required to the ISO 17043 accreditation auditors to evaluate the characteristics, performance specifications and information of schemes designed by EQA providers. The IVD manufacturers must support laboratories and EQA providers by providing high-quality products and reliable technical information (e.g. metrological traceability).

The activities of all stakeholders must be coordinated by the scientific societies and international federations that have the task of promoting the preparation of guidelines and recommendations that help, on the one hand, laboratories to carry out suitable procedures to manage the EQAS and, on the other hand, the EQA providers to design reliability schemes on the basis of the same characteristics and performance specifications. Moreover, they should support the divulgation of the approved documents and the training of all the stakeholders.

Conclusions

Medical laboratories have long used EQA procedures, but continuous progress made in laboratory medicine calls for constant development and change in EQA design. As a logical evolution in quality management, EQA organizers must arrange their schemes according to quality specifications in order to constantly encourage improvement to higher standards, thus striving to meet laboratory and clinical needs.

There is an urgent need to harmonize the criteria and procedures used by different EQAS providers in order to guarantee the suitability of information provided to laboratories and facilitate them in report interpretation; in particular, guidelines that describe a structured approach to carry out EQA procedures should be issued when it is not possible to comply with the quality specifications suggested by approved documents [21], [44]. The process of harmonization is not easy and requires considerable efforts to share and continuously update choices on the basis of increasingly better quality criteria. Initiatives designed to coordinate harmonization activities at an international level are welcome, but only a closer cooperation between scientific and professional organizations, laboratory professionals and IVD manufacturers, will allow us to achieve a greater interchangeability and comparability of laboratory information. This is a duty and an ethical mandate for laboratory professionals and their scientific organizations [2].