Development of symptom assessments utilising item response theory and computer-adaptive testing—A practical method based on a systematic review

Abstract

Assessment of individual patients’ distress is a cornerstone of clinical care for advanced cancer. Patients’ ability to fill out lengthy questionnaires is compromised by many factors. Computer-adaptive tests (CAT) offer a promising approach to developing tailored instruments, that administer only items relevant to the individual patient. A systematic review of the literature about CATs in medical databases was conducted. Based on the results, a method for developing a CAT was designed that requires nine steps: (1) build an item pool; (2) administer the items to a predefined sample in a calibration study; (3) eliminate inappropriate items; (4) examine whether all items are influenced by a single dominant trait; (5) calibrate the items to the best-fitting item response theory (IRT) model; (6) evaluate items’ parameter equivalence across subgroups; (7) build an item bank with the calibrated items; (8) develop the CAT; and (9) pilot test the developed CAT. CAT offers the chance to extend the usefulness of patient-reported outcome (PRO) measurements from clinical studies to daily clinical practice.

Introduction

Caring for patients who have advanced cancer and for their family members challenges clinicians with complex and fluctuating multidimensional issues [1]. As important navigation aids, both for clinical care decision making and for clinical research, assessments of outcomes reported by patients and their families serve to screen for, identify, characterise, and monitor the condition and status of patients and their families. Because most patients with advanced cancer suffer from concurrent and interrelated problems, first-level screening assessments may not provide sufficient information, and multiple second-level assessments for specific symptoms – for example, the Brief Fatigue Inventory (BFI) [2] or the Hospital Anxiety And Depression scale (HADS) [3] – must be applied. The concurrent use of several second-level assessments is poorly standardised and is challenging in daily clinical practice. Although there are differences among patients in their perceived burden in responding to questionnaire-based studies of symptoms and in the number of items they may be able to respond to reliably [4], it is generally agreed that most patients with advanced cancer can cope with responding to only limited numbers of questions. As new assessment instruments evaluating important issues are being developed in all aspects of advanced cancer and palliative care, it is worth exploring the use of tailored interventions for improving care and outcomes [5], [6].

Most symptom-assessment questionnaires currently used in medicine are developed by using classical test theory (CTT). CTT requires that every item or question in an assessment instrument be administered to every patient tested in order to obtain valid scores. In developing a fixed-length questionnaire for patients with advanced cancer, it is necessary to find a balance between keeping the response burden for patients low while still obtaining sufficient measurement precision. When instruments are adapted by simply reducing the number of questions – a recent example is the EORTC QLQ-C15-PAL [7] – the result may be an instrument that might be reliable enough for assessing differences between study populations – but not for assessing individual patients to guide patient care in daily clinical practice [8].

Forty years ago, item response theory (IRT) was introduced as an alternative to CTT for test development in education and psychology. Now IRT is used in conjunction with powerful computer capabilities and offers an exciting new research area in patient-reported outcome (PRO) measurement. Important groundwork has been the development of item banks containing questions that have been calibrated with a set of psychometric properties covering the whole range of a latent trait, such as pain or fatigue [9], [10]. Based on these item banks, a simple reduction of the items at the risk of losing important information is no longer necessary. In a process resembling that used intuitively in structuring a clinical interview, computer-adaptive testing (CAT) selects from the item bank the most informative question for each individual respondent based on the respondent's answers to previously administered questions. CAT-based individualised item administration results in shorter assessments without the trade-off of losing measurement precision [11], [12]. The use of item banks in combination with CAT affords the possibility of developing questionnaires feasible for use in daily clinical practice. The development of IRT based assessment tools is also receiving progressively more attention. Recently a group of outcomes scientists and the National Institutes of Health (NIH) formed a cooperative network funded under the NIH Roadmap for Medical Research Initiative to re-engineer the clinical research enterprise. This initiative – the Patient-Reported Outcomes Measurement Information System (PROMIS) – aims to revolutionise the way patient-reported outcome tools are selected and used in clinical research and practice evaluation. One main goal of the PROMIS initiative is to develop a set of publicly available CATs for the clinical research community (http://www.nihpromis.org) [13], [14].

Developing new CAT instruments based on IRT poses specific challenges. This systematic review summarises the methods used in IRT-based CAT development for PROs and, informed by those methods, provides a stepwise approach to the development of PRO assessment instruments that can be used by clinical researchers in oncology to evaluate and interpret articles on IRT- and CAT-based PROs.