Using analytical performance specifications in a medical laboratory

Introduction

Our primary goal in the routine clinical pathology laboratory is to provide results which support medical decision making for the benefit of the patient. The issuing of results which do not reflect the true value of a substance in the patient (e.g. concentration of sodium in the circulation) has the potential to harm patients by supporting incorrect decisions. There are many decisions made in the routine laboratory to provide results which are fit for their clinical need. Analytical Performance Specifications (APS) can provide guidance on the amount of error that is acceptable. There are a range of factors, mainly analytical and pre-analytical, where a quantitative assessment of the required quality is needed (Table 1), and a number procedures undertaken in a laboratory where these factors are assessed (Table 2). This paper aims to describe the use of APS in the routine laboratory. Please note that this is a personal viewpoint and represents actions in the routine clinical chemistry laboratory under my supervision. I am certain there will be many alternate approaches to the use of APS in other laboratories and also many areas for improvement in our laboratory.

There is now a long history of the processes to establish APS, with the Milan consensus document providing the higher level concepts in current use [1]. The three “Milan models” are 1. Effect on clinical decision making, 2. Based on biological variation and 3. State of the art. These concepts will be used in the discussion below.

By way of background, pathology laboratories in Australia have been subject to compulsory accreditation for many years. This is carried out by the National Association of Testing Authorities (NATA) using ISO 15189 [2] as the overarching standard, as well as Australian standards from the Australian Commission on Safety and Quality in Healthcare [3]. Laboratories in other countries may have different regulatory systems and other influences on their routine practice.

It is my experience that there is a wide variation in how different laboratories carry out any activities. These differences may be the result of careful local thought with good intentions. It may also be that local decisions are made without awareness of wider issues or recommendations. I follow up this statement with another opinion, that in the absence of evidence to the contrary, assume that laboratories act in different ways. The evidence for similar actions and outcomes comes from External Quality Assurance (EQA) activities. Following on from this, if we want laboratories to act in similar ways, this will not happen without common decision-making processes, for example in the preparation of laboratory standards, and assessment against those standards. Given these statements of belief (spoken only half in jest), we can assume that laboratories used APS in different ways in their processes decision-making. As indicated above, this paper discusses the ways APS are used in decision making in our laboratory.

Before proceeding further, it is necessary to explore what are possible sources for APS for use in the routine laboratory. APS are commonly established externally by a professional body. In practice for our laboratory, and for most Australian laboratories, this usually means APS from the Royal College of Pathologists of Australasia Quality Assurance Program (RCPAQAP). These have been in development over many years [4] and are freely available at the RCPAQAP website [5]. APS from other EQA organisations may be used, for example if enrolled on other programs, or from other professional bodies such as the CDC lipid program [6] or CDC steroid program [7]. Additionally, we make decisions using “in-house” APS. These are usually based on either within-subject biological variation (CV_I) for assay imprecision (CV_A) (Milan level 2) or state of the art (Milan level 3). I describe these as “in-house” as the quality standard (e.g. optimal, desirable, minimal, other), the data set and clinical decision for use are not prescribed externally. Before proceeding further, when considering the use of biological variation as a standard, it is important to recognize the tremendous work of the EFLM working group on biological variation in ensuring the availability of high quality data in this area [8, 9]. I also note in this paper using the concept of “state of the art” in a looser manner than the specific published definitions or opinions proposed for establishment of formal APS [10], referring to any comparison of laboratory analytical performance with that obtained in other routine laboratories.

Laboratory activities utilising (APS)

External quality assurance

External Quality Assurance (EQA) is a required laboratory activity under ISO 15189 and for numerical pathology results a quantitative assessment is made comparing a laboratory’s result against a target. The difference between these values is assessed and an APS provided by the EQA program is the usual tool for this process. The most commonly issued EQA program in Australia is the RCPAQAP where APS have been determined based on a combination of biological variation and state of the art [4]. The APS are applied in multiple locations in the EQA reports, including graphical reporting of recent results, monitoring trends over time and across the concentration range, and in summary reports highlighting findings (Figure 1). The use of APS in this setting can be seen as somewhat analogous to the use population reference intervals to quickly scan a pathology report. In the same way that reference intervals do not define the presence or absence of disease, APS usually do not define the presence of absence of a clinically important deviation in assay performance, but they do allow a rapid assessment of the report for results that should be given more consideration. When using an APS it is important to understand the basis for their development and how they should be used. An EFLM Task and Finish group following after the Milan consensus conference identified six elements needed to describe APS for use in EQA and to allow appropriate use and comparison with APS between different programs [11]. The APS provided in most RCPAQAP programs are used for total error (i.e. are applied to individual results), are “aspirational” (i.e. are aiming to improve laboratory testing, and are set on a combination of biological variation and state of the art. The approximate definition of state of the art used is that about 80 % of laboratories can achieve the standard [4, 10]. The effect of using APS set in this way should be to improve the less well-performing laboratories.

Figure 1:

Example EQA report from the RCPAQAP showing the role of APS in graphical displays (highlighted in image by added dark arrows). (A) Frequency histogram, (B) Youden plot, (C) time sequence, (D) concentration plot. Reproduced with permission from the RCPAQAP.

The choice of APS to use in any setting is a key component of their use. While it is a dated reference, the paper from Ricos et al. in 1996 showed extreme differences between the APS used by different EQA providers in Europe with the widest APS typically 3 to 6 times wider than the narrowest [12]. As a wise statistician once said: never use a statistical test you do not understand. A relevant take on this would be advice to laboratories to “never use an APS you do not understand”.

One recent use of specific APS based on single components of the Milan models is analysis of the RCPAQAP Liquid Serum Chemistry program. This program uses freshly prepared, unaltered serum which is rapidly distributed to over 200 laboratories in Australia, which can be assumed to be commutable [13]. Using imprecision criteria relative to biological variation, there are two measurands (triglycerides and iron) for which the total country CV (CV_C, the CV of results from all participating laboratories) meet the optimal level (CV_C<0.25 × CV_I), 8 measurands which meet the desirable level (CV_C<0.5 × CV_I) and another 13 which meet the minimal level (CV_C<0.75 × CV_I). With this knowledge, laboratories can advise that patients can be monitored across the country with these tests using results from any laboratories. For tests not meeting these standards, variation in results in a patient is likely to reflect between-laboratory differences rather than variation in the patient. This information, the combination of data with appropriate APS, allows us to answer clinical queries regarding questions about interpreting results derived from different laboratories. Australia has recommended using common reference intervals for a number of measurands [14]. The RCPAQAP Liquid serum chemistry program allows assessment of acceptable between-method bias using criteria relative to the reference interval width, which is a combination of within and between person biological variation combined with analytical variation. This data has supported the recommended reference intervals for the majority of method in routine use and identified methods which should be reviewed [13]. This information is vital in supporting decisions by laboratories when they consider adoption of these recommended common reference intervals.

It is worth noting that EQA programs, by design, include a state-of-the-art component. Any graphical output of results or of calculated performance (e.g. CV) allows visual assessment of results to answer the question “am I performing similarly to other laboratories”. Similar to most other EQA programs, the RCPAQAP provides z-scores for individual results, providing a quantitative state of the art assessment for results, consistent with a Milan 3 approach.

Method selection

Assays can be developed in-house (so-called lab developed tests or in-house in vitro diagnostic devices) or they can be purchased from manufacturers (kit methods), with the latter being much more common in the routine laboratory. Analytical performance, e.g. bias, imprecision, reportable range, acceptable sample types, are factors that are included in the purchasing decision. Consideration of these performance factors can also be seen as a way of demonstrating to manufacturers that we value good analytical performance. For better or worse, such analytical considerations are only one factor in a purchase of analytical equipment, with menu, reliability, turn-around time, footprint, software, costs, maintenance requirements and other factors generally pushing down the analytical quality as a decision making factor. Indeed at our last purchase of automated biochemistry equipment, the analytical factors were scored at only 10 % of the total decision-making process. Assigning a higher value to this aspect may provide a better reflection of the importance of this issue.

When assessing equipment with multiple assays, it is important to consider how the analytical performance may affect clinical decision making. For example, with multi-parameter auto analysers, if one manufacturer has all assays with imprecision at the optimal level with respect to CV_I, and another is all at the desirable level, the clinical benefit of the better analyser may be minimal. But if the number of poorly-performing assays on one analyser (e.g. CV_A significantly greater than the minimal standard) is more than another, more clinical decisions may be affected, making this factor potentially a better selection criteria. The use of APS is important in these considerations, helping identify poorly performance assays based on EQA data or manufacturer’s supplied information.

Method verification/validation

Following purchase of an analytical system, there is a need to verify its performance in your laboratory. The key principle that should be applied here is whether the system is performing in your laboratory in the way that the manufacturer intended. For this purpose, the manufacturer’s claims in the IFU become the de facto APS.

In practice in our laboratory we use a range of APS in addition to those indicated by the manufacturer to assess a range of properties during assay verification.

Imprecision data produced during verification is generally compared with multiple APS. These include the manufacturers expected performance (from the IFU), within-subject biological variation, state of the art (e.g. using imprecision data from the RCPAQAP), and comparison with the previous method in use. While performing as expected for the manufacturer’s method is the pass/fail criteria, understanding how the assay performs against the other criteria can be useful to understand how it will be expected to perform in routine use. Bias relative to the expected performance of the method is generally assessed using QC and EQA samples, using the supplied APS for the QC material, even though these are often very wide, the EQA APS for that material. Bias is also assessed relative to the previous method with a method comparison study along with assessing the effect on flagging rates with reference intervals and clinical decision points. The finding of a significant bias relative to the previous method will lead to a need to change reference intervals, alert clinicians of the difference, and report results on different lines in paper and electronic systems.

Other aspects of performance that can be assessed include interferences (especially the measurable visible interferences e.g. haemolysis, lipaemia, icterus), sample container type (e.g. serum, heparin, gel tubes, EDTA) and analyte stability. When doing experiments to confirm or extend manufacturers claims and make decisions about implementation, APS are needed. In general, in our laboratory we use the RCPAQAP APS as these provide a robust starting point to use for any measurand. An example of sample type assessment in this manner is shown in Figure 2. Two important variations need to be considered. These are the APS used by manufacturers, which are often unstated (“hidden APS”), the need to consider the “accuracy/utility balance”, and the error budget.

Figure 2:

Sample type comparison using RCPAQAP APS. (A) Urea (mmol/L), (B) urate (umol/L). X – lithium heparin, ◊ – EDTA, dashed lines – APS. For urea, heparin and EDTA considered equivalent to serum, for urate, only heparin acceptable.

When a manufacturer’s IFU makes a claim about a factor that may affect the value of results, there must be an APS behind this claim. Examples of such claims are that serum and heparin are both acceptable sample types, that there is no interference up to an haemolysis level of 300 mg/dL (3 g/L) released haemoglobin, or that a sample is stable for up to three days at room temperature. In each case the manufacturer will be relying on experimental data and a change, defined by an APS, is not exceeded by the claimed conditions. I refer to these as “hidden” APS as they are often not specifically referenced by the manufacturer, or that they are not recognized by the reader of the IFU.

Examples from two manufacturers from current IFUs for haemolysis interference in serum potassium are as follows: haemoglobin of 20 mg/dL (0.2 g/L) with a criteria of a change in potassium of ≤0.1 mmol/L; haemoglobin of 50 mg/dL (0.5 g/L) with a criteria of change in potassium of +/− 0.14 mmol/L to 3.5 mmol/L then +/− 4 %. The units of mg/dL are those used by these manufacturers. If a laboratory accepts the stated haemolysis limit, they are accepting the manufacturer’s APS. Consideration is required as to the appropriateness of the APS for use. On many occasions the APS for an IFU claim is not supplied and the laboratory is unaware of the APS used to provide the limitation. On other occasions claims are made with references to external sources. A current claim from one manufacturer for both serum and heparin plasma being acceptable sample types for serum AST references a paper from 1989 where the average difference between these tube types is 28 % at a serum value of 23 U/L [19]. Therefor the unstated APS from this manufacturer for sample type could be seen as this value. There are several risks to good laboratory practice here. A laboratorian might accept the recommendations based on the supplied APS from the manufacturer without considering its relevance, or the laboratorian may accept the limitations without even knowing what performance they are accepting. Additionally the evidence for the limitation is commonly not available.

The next issue is the “accuracy/utility balance”. As shown in Figure 3, accepting tighter APS for these types of decisions is not a pure benefit. For example, if a tight limit for the effect of haemolysis is taken, and potassium results with haemolysis greater than the limit are with-held and a recollection requested. While only more accurate potassium results are released, the need for recollections will subject patients to more venepunctures, take the time of the collector and slow down the delivery of care. A study from the Australian Institute of Health Innovation has shown that every haemolysed sample adds an average of 18 min to every Emergency Department stay [20]. As an example of this trade-off, based on local data for serum potassium, an allowable haemolysis of 20 mg/dL (0.2 g/L, see above) would withhold 25 % of samples from the Emergency Department, a limit of 0.25 mmol/L (based on the RCPAQAP APS) would withhold 15 % of samples, however the limit we have selected of 100 mg/dL (1 g/L, with an effect of approximately 0.4 mmol/L) flags about 5 % of samples. The use of a wider APS in certain settings is a balance between the benefits of accuracy and the costs of not releasing results [21].

Figure 3:

Illustration of the accuracy-utility balance.

There are of course alternatives to with-holding results [21, 22] and in this case, it is our practice to correct the potassium results for the effect of haemolysis for the Emergency Department, taking into account the uncertainty of this correction process [23].

There is another aspect to take into account as these various source of error are considered. If each source of error, e.g. unidentified bias, interference, analyte stability, tube type etc., is allowed to have an effect up to the selected APS, then in if all of these should occur in the same sample, with an effect in the same direction, the error could be multiples of the APS. The concept here is known as the error budget, where each of the error types must be small relative to the APS to allow for co-exiting errors. The concept of error budget has typically been applied only to the analytical phase [24], however clearly all factors that can affect the numerical value of results should be taken into account. Again there is a cost to this approach and consideration must be given to the likelihood of this occurring, against the cost of adopting very tight APS.

Measurement uncertainty

It is a requirement of ISO 15189 that laboratories calculate the Uncertainty of Measurement (MU) for all tests performed in the laboratory, and in Australia there is an NPAAC standard for this activity [25] in addition to an ISO standard (ISO/TC 20914) [26]. As well as calculating a value, it is necessary to assess whether this is satisfactory for routine use. Again for this purpose we use laboratory-selected APS. If the long term uncertainty of a test is small relative to CV_I, then the test is accepted. If this standard is not met, then the performance of the test relative to other routine assays is assessed using EQA data. The performance is accepted if it is similar to that seen in comparison laboratories. These represent the concepts in Milan levels 2 and 3 respectively. Assays which do not meet either of these criteria are the ones that need to be addressed to try and improve performance. On some occasions where biological variation data is not available, for example therapeutic drug testing, we have determined in-house values for CV_I against which to assess imprecision requirements [27, 28, 29].

Result change protocol

While there are many steps taken to avoid errors in the results released from the laboratory [30], on some occasions errors are found after a result has been released. For example after a QC failure with correction of the error and re-analysis of the samples, there may need to be correction of previously issued results. There needs to be a consideration of the magnitude of the change in deciding to alert the clinician and change the report. In recognizing that contacting physicians for result changes of a likely low clinical impact has a cost in time and interruption, we have established change limits for most tests to ensure consistent actions at all times. In general these change limits are based on the APS from the RCPAQAP, however many have different changes within the reference interval, on the understanding that a change wholly within the interval is, for many tests, unlikely to change management. For example, we do not reissue changes entirely within the reference interval for serum amylase, lipase, AST, ALT and chloride amongst other tests. This is an example of the accuracy-utility balance where a set of general APS have been modified for a specific purpose.

Conclusions

APS can be used to guide many decisions affecting laboratory performance and are used in many different procedures including method selection, verification, EQA, QC, MU, interferences, sample type and sample stability. In using APS for these decisions, what they mean, how they should be applied, their strengths and limitations and appropriateness for the issue in question. Additionally, laboratories can use the basic components which are used to derive APS (e.g. BV, State of the art) to determine acceptable laboratory performance. Laboratories need to be aware of assumptions and “hidden” APS in manufacturers’ information and consider the accuracy/utility balance and error budget when implementing performance assessment. Additionally, given the high likelihood of variation in laboratory practice, a survey for how laboratories use APS in their activities may allow us to learn from other laboratories about best practice in this field, with such a recent survey indicating that awareness and use of APS in the routine laboratory remains a work in progress [31].

Note: This paper is based on a presentation by the author at the 5th Symposium Cutting Edge of Laboratory Medicine in Europe CELME 2023, Prague, Czech Republic, October 12–13, 2023 Analytical Performance Specifications.