Introduction

The Global Initiative for Asthma estimates that 300 million people worldwide have asthma.1 Prevalence rates as high as 32% have been recorded in the United Kingdom and Australia,2 and the prevalence is increasing in many parts of the world.3–5 Despite evidence-based guidelines,1,6–9 there is consistent evidence that asthma is suboptimally controlled, resulting in unnecessary morbidity, loss of school and workdays, and high costs for countries.9–11 There are 250,000 asthma-related deaths each year.1

There are many reasons why guidelines are poorly implemented, including physician’s lack of knowledge or inertia of practice.12,13 As electronic health records are now the norm in many parts of the world,14,15 it is feasible to provide professionals with computer decision support systems (CDSSs) to prompt evidence-based actions within routine consultations, potentially improving professional adherence to guidelines.

Our systematic review aimed to synthesise the evidence for the use of CDSSs by professionals managing people with asthma. We were primarily interested in the effectiveness of CDSSs in improving patient outcomes, but also sought to investigate process measures of guideline adherence and practical usage of the system.

Materials and Methods

Our protocol is registered with the PROSPERO international prospective register of systematic reviews (CRD 42012002412). We followed the methodology described in the Cochrane Handbook for Systematic Reviews of Interventions.16

Inclusion criteria

We used the PICOS (Participants, Intervention, Comparator, Outcomes, Study design) strategy for describing trials in which we were interested:

Participants

As this study is a review of the evidence, the study participants were de facto the health professionals using CDSSs who were caring for people with asthma—i.e., doctors, nurses and others (e.g., physiotherapists).

Intervention

We adopted Wyatt et al.’s definition of CDSs as ’active knowledge systems which use two or more items of patient data to generate case-specific advice.’17 Haynes and Wilczynski similarly described such systems as ‘information technology which matches characteristics of individual patients to a computerised knowledge base’, with software algorithms generating patient-specific information in the form of recommendations.18 There are various levels of sophistication for CDSSs, from reminders to enter specific data, prescribe certain drugs/vaccines or provide an asthma action plan, to a system retrieving patient asthma information from an electronic health-care record and providing a critique on the intended clinical action. Systems were included if they used patient data to generate case-specific asthma advice. Systems relating only to the task of asthma diagnosis or those exclusively providing patients with support for self-management were excluded.

Comparator

The comparator was ‘usual care’, specifically without the use of a CDSS.

Outcomes

Our primary interest was in the impact of CDSSs on clinical asthma control. In line with recommended guidelines,19 we included outcomes that reflected current control (including asthma-related quality of life) and frequency of asthma exacerbations (including frequency of the general practitioner’s asthma visits, emergency department asthma visits and asthma hospitalisations).

We were also interested in the process by which CDSSs might impact asthma control, both practical usage issues (e.g. the proportion of professionals who actually used the CDSS, the numbers of alerts issued and the impact on time within the consultation) as well as process measures reflecting enhanced guideline adherence (e.g. changes in treatment, in tests ordered and in the proportion of patients with asthma action plans).

Study design

All reports of randomised controlled trials of CDSSs used by health-care professionals for patients with asthma, in any language, published and unpublished, were eligible for inclusion. No other study designs were included.

Information sources and search strategy

We searched Medline, Embase, Cochrane Central Register of Controlled Trials, Health Technology Assessment and Inspec (engineering) databases from 1990 to April 2012 with the terms listed in Supplementary Appendix 1. We wrote to experts and authors of all included studies requesting additional relevant studies. We searched for ongoing and unpublished trials on the following websites: https://portal.nihr.ac.uk/Pages/NRRArchive.aspx, www.clinicaltrials.gov, www.controlled-trials.com and www.anzctr.org.au.

Study selection

Two authors (PM and SM) independently screened titles and abstracts, assessing them against the inclusion criteria. The full text of each potentially eligible paper was reviewed by both authors to decide whether the study should be included. Disagreements were resolved by discussion and, if necessary, arbitration of a third researcher (HP, AS or JCW).

Data collection and abstraction

Using a piloted data extraction form, PM and SM independently extracted the following data from included trials: country, setting, funding, study design, health-care professionals, patient population, features of the CDSS intervention, description of the control group, outcome measures and any adverse effects. Extraction tables were compared, and discussed with a third researcher (HP, AS or JW) arbitrating in the event of unresolved disagreement.

Quality of reporting of trials

We assessed the risk of bias in each trial using the seven-criteria approach described in the Cochrane Handbook for Systematic Reviews of Interventions.16 Overall, each study was rated as follows: A: low risk of bias—no bias found; B: moderate risk of bias—one criterion for risk of bias; C: high risk of bias—more than one criterion for risk of bias.

Synthesis of results

We anticipated considerable heterogeneity in the populations studied, and in the interventions and the outcomes reported in the trials precluding meta-analysis of data. Instead, we planned to undertake a narrative synthesis based on our theoretical model of how such computer systems are expected to exert their effects (see Figure 1). The expectation is that, in a linked causal chain, CDSSs will impact process outcomes, which, in turn, will impact clinical outcomes. The theory underpinning their effectiveness is that relevant reminders and recommendations during a consultation will influence clinicians’ behaviour and thereby improve guideline adherence as measured by process outcomes (e.g. more rational ordering of investigations, prescribing of treatment and use of asthma action plans). Implementation of evidence-based practices will consequently be measureable in clinical outcomes for asthma patients, such as fewer exacerbations, emergency department attendances and hospitalisations.

Figure 1
figure 1

Theoretical model showing how a computer decision support system can improve asthma outcomes.

Full size image

Results

Study selection

Figure 2 is the PRISMA flow diagram. From 5,787 titles, eight studies were selected,20–27 seven in English and one in Spanish.26 One study had two reports.24,28 None of the experts we contacted identified any additional eligible studies. We found nine ongoing and eight unpublished trials (Supplementary Appendix 2).

Figure 2
figure 2

PRISMA flow diagram.

Full size image

We excluded a small group of studies from the early 1990s of computerised theophylline dose calculators because they addressed a specific problem in emergency care and have already been evaluated in a Cochrane review.29

Study characteristics

See Table 1 for details of study characteristics. Most studies were cluster randomised controlled trials20–26 in primary care in the UK21,25 or the Netherlands.23,24 Two studies randomised practices to receive a CDSS for asthma prescribing or a system for angina or cholesterol prescribing.21,24 The practices providing data on angina and cholesterol prescribing were unaware that their (usual care) asthma prescribing data were control data for the parallel asthma study.

Table 1 Characteristics of studies
Full size table

Six of the systems were integrated into an electronic health record:20,22–24,26,27 one was partly integrated21 and one was a stand-alone system.25 Five of the studies20,21,23,26,27 explicitly reported that the system gave prescribing advice and reminders. One system concentrated solely on the prescribing of influenza vaccine for ‘at-risk’ children.27 Four studies were based on asthma management guidelines.21,25–27 One system included a complex risk prediction algorithm,25 and one system ‘critiqued’ the doctor’s intended management plan and made recommendations.23

Risk of bias within studies

Table 2 lists the quality assessment: most studies were rated at high risk of bias. The study by Eccles et al.21 was rated at low risk of bias and that by Martens et al.24 at moderate risk of bias.

Table 2 Risk of bias summary table
Full size table

Effectiveness of CDSSs

The impact of CDSS on process, usage and clinical outcomes is detailed in Table 3. It was anticipated that usage and process outcomes would influence clinical outcomes as reflected in our model (Figure 1).

Table 3 Effectiveness of CDSS: process outcomes—guideline adherence
Full size table

Practical aspects of CDSS use

In the study by Eccles et al.,21 the median number of activations of the system per practice was zero. In that by Kuilboer et al.,23 10,863 visits generated 10,532 decision support comments, but the doctor waited for the critique only 22% of the time, and then read only a third of them. In Tierney et al.’s study,27 doctors complied with a third of the systems’ suggestions. Bell et al.20 reported that the CDSS was used 70% of the time. In the study by Fiks et al.,22 the vaccine alerts were only active during 27% of visits.

Process outcomes

Changes in tests ordered

Eccles et al.,21 McCowan et al.25 and Plaza et al.26 all reported that the systems made no difference in the rates of ordering spirometry, X-rays, allergy tests or blood tests. Bell et al.20 reported an increase in spirometry requests at intervention practices from 15 to 24%, whereas there was a decrease at control practices from 8 to 1%. In Kuilboer et al.,23 peak expiratory flow rate and spirometry tests were ordered more often in the intervention group, in patients over 11 years of age. In a four-arm trial, Tierney et al.27 reported that between 39 and 50% of patients received the suggestion to obtain pulmonary function tests.

Changes in treatments

Eccles et al.,21 the only trial at low risk of bias, found no difference in asthma-related prescribing as a result of the intervention. Martens et al.24 demonstrated an increase in the prescribing of inhaled corticosteroids to 44% of asthma patients (95% confidence interval (CI), 30–56%) in the intervention group, compared with 27% (95% CI, 14–47%) in the control group.

In the trial by Bell et al.,20 there was a highly significant (P=0.006) difference between the rate of prescribing inhaled corticosteroids in the subgroup of urban intervention practices compared with urban control practices. Urban and suburban practices were analysed separately in the cluster controlled trial because of marked baseline differences in patient population: the urban practices had more severe asthma.

Kuilboer et al.23 demonstrated a significant reduction in the prescribing of cromoglyate in a post hoc analysis. Plaza et al.26 demonstrated a doubling of treatment conforming to guidelines, from 18 to 34% (P=0.02). Vaccination rates increased in both arms of the Fiks trial with no significant differences.22 McCowan et al.25 found no difference in asthma-related prescribing between the trial arms due to the intervention. Tierney et al.27 reported on treatment suggestions for both asthma and chronic obstructive pulmonary disease. For example, across the four arms of the Tierney trial, between 5 and 9% of patients received the suggestion to ‘start inhaled corticosteroids.’ However, only 11–30% of the physicians or pharmacists complied with this suggestion.

Clinical outcomes

Asthma symptoms

Three studies reported asthma symptoms.21,25,26 Eccles and coworkers30 reported that the CDSS had no effect on the validated Newcastle Asthma Symptoms Questionnaire (mean difference −0.6 (95% CI, −2.1 to 0.9)).21

Plaza et al.26 reported that asthma daytime symptoms, but not night-time symptoms, were significantly reduced in the intervention group compared with the control group (Wilcoxon P<0.02). McCowan et al.25 reported no significant differences in asthma symptoms between the intervention and control groups (odds ratio 0.3, 95% CI, 0.03–3.3), although this study was underpowered.

Asthma-related quality of life

Three studies reported asthma-related quality of life.21,26,27 The study by Eccles et al.,21 a trial at low risk of bias, reported no effect on the validated Asthma Quality of Life Questionnaire.31 Plaza and coworkers32 reported quality of life using the Spanish version of the St George’s Respiratory Questionnaire and found significant improvement in all domains (activity P=0.002, symptoms P=0.003, impact P=0.001).26 Tierney et al.27used two different quality-of-life scales,33,34 but found a significant result only in one subdomain, possibly due to multiple testing.

Frequency of asthma exacerbations

Two studies reported exacerbation rates. In the study by Plaza et al.,26 exacerbation rates were not significantly different between the control and intervention groups: mean exacerbations, 1.3 (s.e.)=1.2) in the control group and 0.5 (s.e.=0.3) in the intervention group (Wilcoxon P=0.2). McCowan et al.25 reported that in the intervention group 12/147 patients had exacerbations compared with 57/330 in the control group: control patients were approximately twice as likely to experience an exacerbation as were intervention patients (odds ratio 0.4, 95% CI, 0.2–0.9, after adjustment for clustering). The denominators were different because of study dropouts.

Unscheduled health-care utilisation

McCowan et al.25 reported significantly fewer unscheduled general practitioner consultations in the intervention group in comparison with the control group (odds ratio 0.6, 95% CI, 0.4–0.95). Four studies reported no differences in the frequency of asthma-related visits to the general practitioner.20,22,23,26

Two studies reported no significant difference between the intervention and control groups in emergency department visits or hospitalisations.25,27 The absolute numbers were close to zero.

Discussion

Main findings

We found eight relevant trials, four of which reported clinical measures of asthma control.21,25–27 The key finding was that CDSSs for health-care professionals were ineffective in improving patient outcomes because the systems were rarely used,21–23 and there was low compliance with the advice when it was issued.23,27 However, when systems are used, clinical outcomes can improve.20,25

Strengths and limitations of this study

A strength of this review is its robust search strategy. We used the Cochrane-suggested terminology for asthma and randomised controlled trials, and drew on our eHealth research group’s inclusive search terms for CDSS.35 Nevertheless, we may have missed some relevant studies, and the list of ongoing trials suggests that more evidence may be available in due course.

In contrast to the methodology used by the recent McMaster group series of reviews in which improvement was considered to have occurred if >50% of the selected outcomes showed benefit,18,36–41 we report specific clinical, usage and process outcomes from each trial to explain why the systems were having an effect or not.

We did not perform a meta-analysis as populations and outcomes across trials were too heterogeneous. Descriptions of interventions were often poorly described, which may have limited our interpretation of the findings.

Interpretation of findings in relation to previously published work

Our review focuses on asthma as a clinically important area for CDSSs. A crucial observation was that the systems were rarely used.13,21,42 Usage was not considered in the recent McMaster group’s meta-regression,43 although this is clearly fundamental to understanding the reasons for lack of effect, and should be a crucial focus for development if systems are to improve patient outcomes above the 15–31% impact on outcomes reported by the McMaster group in a series of reviews.36–41 Usage rate of the systems should be a core standard for reporting trials of CDSSs.

The McMaster group’s meta-regression explored the features of CDSSs associated with system ‘effectiveness’. They found (1) stand-alone programs, (2) advice directed at both health-care practitioners and patients, (3) requiring users to enter an explanation for any overrides of system recommendations and (4) developers’ involvement in trials to be associated with better patient outcomes. Poor integration (as in a stand-alone program), however, risks clinicians avoiding using the system as in Eccles et al.21 The issue, however, is complex as advice presented at the time of care does not always predict success, possibly because practitioners become overwhelmed by such integrated alerts that interrupt their workflow.43

Our recent analysis of recordings of general practice consultations emphasised the importance of the timing of alerts in the context of prescribing safety CDSSs.44 The practitioner, negotiating with the patient, makes decisions regarding drugs and management throughout the consultation when information about allergies, sensitivities, interactions and guideline recommendations might be useful. Provision of information during the final computer-based task of generating the prescription can frustrate clinicians, who then override the alerts. Integration with workflow requires a detailed study of the consultation process.

Implications for future research, policy and practice

A detailed description of the CDSS intervention under investigation is essential to providing insight into what promotes a well-used and effective system that can inform future development. Taxonomies and frameworks such as those described by Kawamoto et al.,45 Garg et al.46 or Berlin et al.47 may provide a suitable basis for a full description. Future research should substantiate our theoretical model (Figure 1), which we suggest as a possible useful framework. In terms of the logical chain from usage to process outcomes to clinical outcomes, Bell et al.20 demonstrated that usage rates have an impact on process outcomes, and Plaza et al.26 demonstrated the impact of process outcomes on health outcomes. However, we feel that further research is required to evidence this model more thoroughly.

Conclusions

Our review suggests that current CDSSs are unlikely to result in improved outcomes in asthma because they are rarely used and the advice not followed. A key challenge in the future design of decision support systems lies in the better integration and alignment with professional workflows such that they are adopted into routine practice.