https://orcid.org/0000-0002-1074-2599
Figures
Abstract
Background
Bronchiectasis is a chronic respiratory disease characterised by airways widening and recurrent infections, resulting in episodes of chronic cough, sputum expectoration, and dyspnoea. This leads to deterioration in daily function, repeated hospital admissions and poor quality of life. The prevalence and mortality related to bronchiectasis is increasing worldwide with growing economic burden on healthcare systems. Physiotherapy for bronchiectasis aims to decrease accumulation of sputum, dyspnoea, and improve exercise capacity and daily function. A robust evidence base to support physiotherapy in bronchiectasis is currently lacking. This is partly because of inconsistency and poor reporting of outcomes in available studies.
A core outcome set is the minimum acceptable group of outcomes that should be used in clinical trials for a specific condition. This decreases research waste by improving consistency and reporting of key outcomes and facilitates the synthesis of study outcomes in systematic reviews and guidelines.
The aim of the study is therefore to develop a core outcome set and outcome measurement set for physiotherapy research in adults with bronchiectasis. This will ensure outcomes important to key stakeholders are consistently used and reported in future research.
Methods and analysis
This project will use the COMET Initiative and COSMIN guidelines of core outcome set development and will include three phases. In the first phase, a comprehensive list of outcomes will be developed using systematic review of reported outcomes and qualitative interviews with patients and physiotherapists. Then consensus on key outcomes will be established in phase two using a Delphi survey and a consensus meeting. Finally, in phase three, we will identify appropriate instruments to measure the core outcomes by evaluating the psychometric properties of available instruments and a stakeholders’ meeting to establish consensus.
Ethics
The study was reviewed and has received ethical approval from the health-related Research Ethics Committee- Edge Hill University (ETH2021-0217).
Registration
This study is registered with the COMET database. https://www.comet-initiative.org/Studies/Details/1931.
The full systematic review protocol is registered in PROSPERO under the number CRD42021266247.
Citation: Hamzeh H, Spencer S, Kelly C (2022) Development of a core outcome set and outcome measurement set for physiotherapy trials in adults with Bronchiectasis (COS-PHyBE study): A protocol. PLoS ONE 17(2): e0263695. https://doi.org/10.1371/journal.pone.0263695
Editor: Nagendra Kumar Kaushik, Kwangwoon University, REPUBLIC OF KOREA
Received: November 17, 2021; Accepted: January 17, 2022; Published: February 8, 2022
Copyright: © 2022 Hamzeh et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: HH has a Graduate Teaching Assistant studentship from Edge Hill University for this doctoral research. https://www.edgehill.ac.uk/ The funders had and will not have a role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Background
Bronchiectasis is a chronic respiratory disease characterised by widening and thickening of the airways, leading to accumulation of secretions and recurrent infections [1,2]. Bronchiectasis has a negative impact on quality of life, including difficulty completing daily activities, social embarrassment, anxiety, and sleep deprivation [3,4]. The prevalence of bronchiectasis has increased globally in recent years [5], causing a substantial economic burden of approximately $36,000 annually per person as calculated in US and Spanish cohorts [6]. Cost is attributed to hospitalizations, outpatient service use, physiotherapy and rehabilitation, and long-term medications [7,8]. UK bronchiectasis-related mortality is more than twice that of the general population [9], while 5 years mortality is 12.4% in European population [10].
Physiotherapy for bronchiectasis encompasses a group of interventions aimed at improving symptoms and functionality, including airway clearance, pulmonary rehabilitation, exercise and breathing training [11]. These are usually delivered via customised interventions involving teaching self-administered airway clearance techniques and a home exercise program [12].
While physiotherapy is recognised as a core element of bronchiectasis care [12–14], it currently lacks high quality evidence of its effectiveness [15,16]. This is partly attributable to difficulties in aggregating data from clinical trials in systematic reviews, due to inconsistent outcome reporting and variation of measurement instruments, with some important outcomes, such as exacerbation, hospitalisation and side effects, commonly missing [17–22]. This is particularly relevant for bronchiectasis research where there are very few large-scale trials and therefore the synthesis of results from smaller trials is important for assessing evidence of effectiveness. Consequently, using COMET (Core Outcome Measurement in Effectiveness Trials) methodology to standardise outcome reporting is important for improving the design of future trials [23]. Spargo and colleagues (2019) [24] has developed a core outcome set for bronchiectasis, it included all types of intervention and the expert group of the Delphi study included mainly physicians, which limits its validity for use in physiotherapy specific trials. The COS included 18 outcomes, which is a large number to be used in trials and systematic reviews. Defining OMIs for bronchiectasis studies were not included in the study, which limits the usability of its results by researchers and clinicians.
Aims and objectives
A core outcome set (COS) comprises the minimum agreed outcomes that should be measured and reported in trials for a given health condition [25]. Thus, establishing not only ‘what’ should be measured in this area of research, but also ‘how’ to measure it. The main aim of this study is therefore to work with patients and healthcare professionals to develop a COS for use in trials examining the benefits of physiotherapy for adults with bronchiectasis. This core outcome set focuses on adults as children may have different problems and needs that require a separate set of outcomes. The main objectives are:
- To identify a list of outcomes currently reported in physiotherapy trials for bronchiectasis
- To evaluate consistency in outcome reporting in published trials and trial protocols.
- To explore the important outcomes for stakeholders, including patients and physiotherapists
- To establish consensus among researchers and stakeholders on the most important outcomes to be included in the COS.
- To develop a Core Measurement Set (CMS) by identifying outcome measurement instruments (OMIs) for reporting the COS.
Methods
The study design is based on recommendations developed by COMET and COSMIN (Consensus-based Standards for the selection of health Measurement Instruments) standard setting initiatives. COMET has supported the development of hundreds of COS’s with over 800 currently registered on their online database [26]. Protocol methods are based on the Core Outcome Set—Standardised Protocol Items (COS-STAP) [27] and the 11 minimum standards for COS development (COS-STAD) [28] shown in Table 1. The three-phase study design is shown in Fig 1 and the protocol is registered on the COMET database.
Stakeholder involvement
The COS-PHyBE study aims to produces a COS that is meaningful and inclusive for all stakeholders. Therefore, patient representatives and respiratory physiotherapists were invited to be research advisors, who will be involved in designing, piloting, and recruitment for the study. The research team also includes researchers and systematic reviewers with interest in bronchiectasis research.
Phase one: Generating long-list of potential outcomes
We will create a long-list of potential outcomes using a systematic review of academic literature and semi-structured interviews of patients and physiotherapists.
Systematic review of academic literature.
We did not identify similar or ongoing systematic reviews on the international prospective register of systematic reviews (PROSPERO). We will undertake a systematic review of randomised control trials (RCTs) and protocols for RCTs investigating the benefits of physiotherapy interventions for bronchiectasis. We will limit the search to RCTs as the purpose of the COS-PHyBE study is to influence the design of future RCTs. The protocol for this systematic review is registered on PROSPERO (CRD42021266247).
We will search PubMed, EMBASE, CINAHL, Pedro, and the Cochrane Central Register of Controlled Trials (CENTRAL) from their inception to current date. We will also search protocol registries, reference lists, dissertations, and relevant conference proceedings for relevant trials. The search will be limited to trials published in the English language due to limited translation resources for publications in other languages.
Randomised and quasi randomised clinical trials of any physiotherapy intervention for bronchiectasis will be included. Trials will be excluded if the study population included children (<18 years) or mixed respiratory populations (e.g., participants from multiple respiratory disease cohorts).
Two reviewers will independently perform study selection and data extraction, with disagreements resolved by consensus. We will screen titles and abstracts for eligibility and confirm using the full text. Study selection will be reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline. We will extract study population characteristics, all reported outcomes, their definitions and domains, the OMIs used to measure them, and the measurement timepoints.
The COS taxonomy will be used to classify outcomes. The 38 outcome domains cover the core areas of death, physiological outcomes, life impact, resource use, and adverse effects [29]. Results will include a narrative description of the outcomes and their categories, variation in outcome definitions, different interventions, and differences in outcomes across different interventions.
Semi-structured stakeholder interviews.
The aim of this phase is to identify outcomes important to patients who receive the treatment and physiotherapists who assess these outcomes in daily practice. This will ensure that the long-list will include, in addition to outcomes identified in the literature, the most important outcomes to stakeholders, to enhance utility and uptake of the final COS [23].
We will interview 6–12 patients and the same number of physiotherapists, considered sufficient to achieve data saturation, whereby no new outcomes are identified [30]. sample will include adult patients diagnosed with bronchiectasis and previously treated by physiotherapy, as well as physiotherapists clinically involved in bronchiectasis care for at least one year. To ensure a broad range of views are represented we will recruit participants through online channels. This will assist recruiting participants from multiple countries and settings, and with different backgrounds and experiences.
Patients will be recruited through the European Lung Foundation (ELF) and via patient support groups on social media. Physiotherapists will be recruited through the Association of Chartered Physiotherapists in Respiratory Care (ACPRC), and other professional networks. Participants unable to complete an interview in English will be excluded.
Interviews will be done remotely via video conferencing or telephone. Participant information and informed consent forms will be sent to participants before the interview date and consent will be recorded during the interview. A semi structured interview format will be followed, and the main theme of the interviews will be exploring outcomes important to participants. Interview guides will be prepared considering similar qualitative studies and results of the systematic review. Interview guides were prepared for each group and validated by patients and physiotherapist representatives (S1 Appendix).
The interviews will be digitally recorded then transcribed verbatim. Transcripts will be analysed using thematic analysis methods [31], supported by NVivo qualitative data analysis software version 12 [32]. The analysis will aim to identify and explain important outcomes and describe them using participants’ language. Outcomes which are mentioned directly or indirectly will be extracted from the qualitative data. Thematic analysis will be guided by the COMET outcomes classification as an analytical framework. Frequency that each outcome is mentioned and justifications for its value to participants will be used to determine the importance of outcomes. A list of all outcomes identified from interviews will be prepared by the end of analysis. The Consolidated criteria for Reporting Qualitative research (COREQ) will be used to report the qualitative study [33].
Phase two: Prioritise outcomes and finalise COS
Consensus on core outcomes will be achieved using a Delphi survey, followed by a stakeholder meeting to establish consensus.
Delphi survey.
COS studies commonly use Delphi surveys to achieve consensus on the most important outcomes to include [23,34]. The technique ensures all participants make an equal contribution and involves multiple anonymised prioritisation rounds [35]. A modified Delphi design is recommended in COS development, where the list of outcomes in phase 1 comprises the first round [23]. This modified Delphi study will involve two rounds conducted using the onlinesurveys.ac.uk interface (JISC, Bristol, UK) to maximise participation and enhance credibility [34]. Each round will take approximately 3 weeks and participants will be promoted with weekly reminders.
Formulation of the survey items. The two lists of outcomes produced from the systematic review and the interviews will be merged to produce the long outcomes list. This list will then be discussed with the research advisors to merge similar outcomes and decide the list of outcomes to be taken into the Delphi survey. Using the COMET taxonomy the list will be grouped into outcome domains [23]. A definition for each outcome in simple language will be prepared and verified by patient representatives. The final survey will then be pilot tested by research advisors to ensure clarity and feasibility before data collection.
Sample and recruitment. Adults aged 18+ with bronchiectasis, physiotherapists, and researchers able to complete a survey in the English language will be asked to participate. Consent to participate will be implied by completion of the survey.
There is no required sample size for a Delphi study, but good representation of each group is essential [23,34]. Therefore, a pragmatic approach will be taken, and all interested participants will be included in the study. Having different group sizes may downgrade the voice of smallest group. Hence, data analysis will be performed separately for the three groups, this will allow for intra and inter group variability to be explored and for equal group’s representation in the process.
Recruitment will be facilitated through multiple channels to focus on maximising international participation and representation of various demographic populations. We will recruit through patient networks (such as the European Lung Foundation), professional organisations (such as The International Confederation of Cardiorespiratory Physical Therapists (ICCrPT)), research networks (such as the European Multicentre Bronchiectasis Audit and Research Collaboration (EMBARC)), other professional organisations and societies, patients’ groups on social media, and contacting key researchers identified in phase 1. A Lay language summary explaining the study objectives will be prepared using documents from the COMET Patient Participation, Involvement and Engagement group.
First round. A nine-points Likert scale will be used to score items according to their importance and an option of ‘unable to score’ will be added for each item. 1 to 3 signifies an outcome is of limited importance, 4 to 6 important but not critical, and 7 to 9 critical. This scoring system is widely used in COS studies and is recommended by COMET and GRADE [25]. Participants will be asked to rate each outcome’s importance according to their own experiences and opinions. One open ended question will be added at the end of the first-round questionnaire to capture any new suggested outcomes.
Second round. All outcomes will be carried through to second round in addition to new suggested outcomes from round one. New suggested outcomes will go through a review process; they will be carried out to next stage if they are genuinely new and can’t be included under any other outcomes. For each outcome, collective scores obtained from each group in round one will be displayed alongside the participant’s personal previous score, scores will be accompanied by graphs to enhance visual presentation. Participants will be asked to consider previous scores and rate each item using the same Likert scale. All participants in round two will be asked if they are interested in attending the subsequent consensus meeting.
Consensus levels will be defined as following:
- Included: a score of 7 to 9 from more than 70% of participants and a score of 1 to 3 from less than 15% of participants in all groups. These items will be included in the final COS.
- Excluded: a score of 7 to 9 from less than 50% of participants in all groups. These items will be discarded from the final COS.
- No consensus: items which does not achieve the inclusion or exclusion criteria. These will be taken into the consensus meeting for discussion and final voting.
Missing data. The rate of missing and incomplete responses will be reported with the results of the Delphi study. If a participant did not complete all rounds, available responses will be included in analysis. As analysis will be performed separately for each item, incomplete responses will not be discarded, and available items will be included in the analysis. To test if missing data affect representativeness, the first-round participant profiles will be compared to participants who completed both rounds.
Consensus meeting.
Following the Delphi survey, a consensus meeting will be held to discuss the results and recommend the final COS outcome domains. A pragmatic representative sample will be invited from participants who expressed interest in the second Delphi round. Outcomes which achieved no consensus in the Delphi survey will be subjected to voting. A modified nominal group technique will be used, which is a structured group discussion that involves generating, defining, and ranking items to reach consensus while limiting individual dominance [36]. This method will involve a series of discussing outcomes, nominating most and least important outcomes by each participant, anonymous voting on outcomes, discussion of voting results and agreeing final COS. Nine point Likert scale will be used for voting, and the same consensus criteria of the Delphi survey will be applied. Voting results will be presented by the end of meeting for final COS agreement.
Phase three: Development of a core measurement instrument set
The COSMIN/COMET guidelines for establishing a set of measurement instruments for the COS [37] will be followed (Table 2). The process includes a systematic review followed by a stakeholder’s meeting.
Step 1. Conceptual considerations.
The outcomes for which CMS will be identified in phase two of the study. Scope of the CMS is detailed in Table 2.
Step 2. Identify existing outcome measurement instruments.
Existing instruments will be identified as part of the systematic review of available trials in phase one. Information regarding instruments used in trials will be extracted, they will appropriately be tabulated and classified according to relevant outcomes. Another search will be run to identify any available systematic reviews which evaluated psychometric properties of these instruments. The COSMIN Database of systematic reviews of outcome measurement instruments and PubMed Central will be searched for this purpose.
Systematic review of outcome measurement instruments. If no systematic reviews can be identified for a certain instrument, or if the available reviews are outdated or of poor quality, a separate systematic review of outcome measurement properties will be conducted. The review will be performed according to the COSMIN/COMET guidelines [35]. We will search PubMed, EMBASE, and CINAHL from inception to current date. The search will be limited to trials published in the English language. Studies which investigated psychometric properties of the instruments in bronchiectasis will be included.
Two reviewers will independently perform study selection and data extraction, with disagreements resolved by consensus. We will extract study population, instrument definition, results of the measurement properties assessed, evidence on interpretability and feasibility (e.g., scores description, use of devices, floor and ceiling effects, minimal important change or difference, ease of standardization and calculation, completion time). For PROMs, mode of administration, original language and available translations will additionally be extracted.
Step 3. Quality assessment of outcome measurement instruments.
Evaluation of the quality of the measurement properties for each OMI will be assessed using the COS-MIN tool which was developed by COSMIN and COMET [37]. This tool is recommended to evaluate the psychometric properties of patient reported, clinician-reported, and performance-based outcome measurement instrument, in addition to laboratory values. Rated measurement properties include different forms of validity (content, structural, criterion, cross cultural), internal consistency, reliability, measurement error, hypotheses testing, and responsiveness. Each measurement property will be rated as positive, intermediate, or negative. Then overall quality of OMI will be rated from high to unknown according to the recommended criteria.
Step 4. Selection of outcome measurement instruments for the COS.
A table of the psychometric properties and quality for each OMI will be prepared. For each outcome, OMIs will be recommended based on their quality assessment. An OMI will be recommended only if it meets the following minimum COS inclusion requirements: at least high-quality evidence of good content validity, high quality evidence of internal consistency, test-retest or inter-rater reliability (if applicable), and if it seems feasible.
Where there is missing data regarding psychometric properties for an OMI, we will recommend further validation work. These will be added to the final list as provisional outcome measurement instruments.
Stakeholder’s meeting. The aim of the final stakeholders meeting will be to ensure transparency of the process and approve the final CMS. We will invite key researchers, physiotherapists, and patient representatives to attend. Participants will discuss results from the systematic review and recommendations for the final list of instruments and appropriate time points for measurement. A single OMI will be chosen for each outcome, this will be based on discussion of feasibility aspects in case two instruments has similar quality criteria.
The status and timeline of the study
The first phase of the study; systematic review of outcome reporting is currently ongoing. The expected end date of the study is January 2024
Limitations of the study design
Although all efforts will be made to encourage international participation in the study, this will be limited by the inability to conduct the study in languages other than English. Using stakeholders meeting only to determine the core measurement instruments may risk bias towards opinions of individuals present at the meeting. We anticipate that our dissemination strategy will lead to satisfactory uptake of the COS with relevant researchers, but it is not guaranteed that it will be implemented in future research.
Dissemination plan
The study will be published in peer reviewed journals. The results from this research will be presented at meetings, seminars and symposiums and at relevant conferences. They will also be shared with all researchers who participated in the project and experts in the field. The results will also be presented to interest groups and organisations, such as EMBARC, ELF, ERS, ICCRPT, ACPRC, BTS. We will additionally publicise the COS updates and results via the project’s Twitter page. Lay language summaries will be shared with relevant societies and patient groups for dissemination through their websites and social media.
Supporting information
S1 Appendix. Interview guides for the semi-structured stakeholder interviews.
https://doi.org/10.1371/journal.pone.0263695.s001
(DOCX)
References
- 1. Hill AT, Sullivan AL, Chalmers JD, De Soyza A, Stuart Elborn J, Andres Floto R, et al. British thoracic society guideline for bronchiectasis in adults. Thorax. 2019;74. PubMed Central PMCID: PMC30545985. pmid:30545985
- 2. Aliberti S, Lonni S, Dore S, McDonnell MJ, Goeminne PC, Dimakou K, et al. Clinical phenotypes in adult patients with bronchiectasis. The European respiratory journal. 2016;47(4):1113–22. Epub 2016/02/06. pmid:26846833.
- 3. Pasteur MC, Bilton D, Hill AT. British thoracic society guideline for non-CF bronchiectasis. Thorax. 2010;65(SUPPL. 1):i1–i58. PubMed Central PMCID: PMC20627931. pmid:20627931
- 4. Dudgeon EK, Crichton M, Chalmers JD. “The missing ingredient”: the patient perspective of health related quality of life in bronchiectasis: a qualitative study. BMC Pulmonary Medicine. 2018;18(1):81. pmid:29788953
- 5. Chandrasekaran R, Mac Aogáin M, Chalmers JD, Elborn SJ, Chotirmall SH. Geographic variation in the aetiology, epidemiology and microbiology of bronchiectasis. BMC Pulmonary Medicine. 2018;18(1):83. pmid:29788932
- 6. Goeminne PC, Hernandez F, Diel R, Filonenko A, Hughes R, Juelich F, et al. The economic burden of bronchiectasis–known and unknown: a systematic review. BMC Pulmonary Medicine. 2019;19(1):54. pmid:30819166
- 7. Blanchette C, Noone J, Stone G, Zacherle E, Patel R, Howden R, et al. Healthcare Cost and Utilization before and after Diagnosis of Pseudomonas aeruginosa among Patients with Non-Cystic Fibrosis Bronchiectasis in the U.S. Medical Sciences. 2017;5(4):20. pmid:29099036
- 8. Diel R, Chalmers JD, Rabe KF, Nienhaus A, Loddenkemper R, Ringshausen FC. Economic burden of bronchiectasis in Germany. European Respiratory Journal. 2019;53(2):1802033.
- 9. Quint JK, Millett ERC, Joshi M, Navaratnam V, Thomas SL, Hurst JR, et al. Changes in the incidence, prevalence and mortality of bronchiectasis in the UK from 2004 to 2013: A population-based cohort study. European Respiratory Journal. 2016;47(1):186–93. PubMed Central PMCID: PMC26541539. pmid:26541539
- 10. McDonnell MJ, Aliberti S, Goeminne PC, Restrepo MI, Finch S, Pesci A, et al. Comorbidities and the risk of mortality in patients with bronchiectasis: an international multicentre cohort study. the lancet respiratory medicine. 2016;4(12):969–79. pmid:27864036
- 11. Bott J, Blumenthal S, Buxton M, Ellum S, Falconer C, Garrod R, et al. Guidelines for the physiotherapy management of the adult, medical, spontaneously breathing patient. Thorax. 2009;64 Suppl 1:i1–51. Epub 2009/05/07. pmid:19406863.
- 12. Hill AT. on behalf ofthe British Thoracic Society. BTS National Audit Report: Adult and Paediatric Bronchiectasis Audits 2017. British Thoracic Society Reports. 2018;9(2).
- 13. Phillips J, Lee A, Pope R, Hing W. Physiotherapists’ use of airway clearance techniques during an acute exacerbation of bronchiectasis: a survey study. Archives of Physiotherapy. 2021;11(1):3. pmid:33517917
- 14. O’Neill B, Bradley JM, McArdle N, MacMahon J, O’Neill B, Bradley JM, et al. The current physiotherapy management of patients with bronchiectasis: a UK survey. International Journal of Clinical Practice. 2002;56(1):34–5. Language: English. Entry Date: 20021101. Revision Date: 20170414. Publication Type: journal article. pmid:11831830
- 15. Aliberti S, Masefield S, Polverino E, De Soyza A, Loebinger MR, Menendez R, et al. Research priorities in bronchiectasis: a consensus statement from the EMBARC Clinical Research Collaboration. European Respiratory Journal. 2016:ERJ-01888-2015. pmid:27288031
- 16. Spinou A, Chalmers JD. Respiratory physiotherapy in the bronchiectasis guidelines: is there a loud voice we are yet to hear? The European respiratory journal. 2019;54(3). PubMed Central PMCID: PMC31563873.
- 17. Welsh EJ, Evans DJ, Fowler SJ, Spencer S. Interventions for bronchiectasis: an overview of Cochrane systematic reviews. Cochrane Database of Systematic Reviews. 2015;(7). pmid:26171905.
- 18. Kelly C, Grundy S, Lynes D, Evans DJ, Gudur S, Milan SJ, et al. Self-management for bronchiectasis. Cochrane Database Syst Rev. 2018;2(2):Cd012528. Epub 2018/02/08. pmid:29411860; PubMed Central PMCID: PMC6491127 Grundy–none known David Evans–none known Stephen Milan–none known Sharada Gudur‐ none know.
- 19. Lee AL, Burge AT, Holland AE. Airway clearance techniques for bronchiectasis. Cochrane Database of Systematic Reviews. 2015;2015(11). PubMed Central PMCID: PMC26591003. pmid:26591003
- 20. Phillips J, Lee A, Pope R, Hing W. Effect of airway clearance techniques in patients experiencing an acute exacerbation of bronchiectasis: a systematic review. Physiotherapy Theory & Practice. 2020;36(12):1300–15. pmid:30776932. Language: English. Entry Date: 20201118. Revision Date: 20201118. Publication Type: Article.
- 21. Lee AL, Williamson HC, Lorensini S, Spencer LM. The effects of oscillating positive expiratory pressure therapy in adults with stable non-cystic fibrosis bronchiectasis: A systematic review. Chronic Respir Dis. 2015;12(1):36–46. PubMed Central PMCID: PMC25518845. pmid:25518845
- 22. Lee AL, Hill CJ, McDonald CF, Holland AE. Pulmonary rehabilitation in individuals with non-cystic fibrosis bronchiectasis—a systematic review. Archives of Physical Medicine and Rehabilitation 2017 Apr;98(4):774–782. 2017. pmid:27320420
- 23. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(3):280. pmid:28681707
- 24. Spargo M, Ryan C, Downey D, Hughes C. Development of a core outcome set for trials investigating the long-term management of bronchiectasis. Chronic Respir Dis. 2018;16. PubMed Central PMCID: PMC30278785. pmid:30278785
- 25. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(1):132. pmid:22867278
- 26. Gargon E, Gorst SL, Williamson PR. Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research. PLOS ONE. 2019;14(12):e0225980. pmid:31830081
- 27. Kirkham JJ, Gorst S, Altman DG, Blazeby JM, Clarke M, Tunis S, et al. Core Outcome Set-STAndardised Protocol Items: the COS-STAP Statement. Trials. 2019;20(1):116. pmid:30744706
- 28. Kirkham JJ, Davis K, Altman DG, Blazeby JM, Clarke M, Tunis S, et al. Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLOS Medicine. 2017;14(11):e1002447. pmid:29145404
- 29. Dodd S, Clarke M, Becker L, Mavergames C, Fish R, Williamson PR. A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery. Journal of clinical epidemiology. 2018;96:84–92. Epub 2017/12/31. pmid:29288712; PubMed Central PMCID: PMC5854263.
- 30. Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Quality and Quantity. 2018;52(4):1893–907. http://dx.doi.org/10.1007/s11135-017-0574-8. pmid:2993758550.
- 31. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006;3(2):77–101. pmid:32100154
- 32. QSR International Pty L. NVivo (released in March 2020) 2020.
- 33. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care. 2007;19(6):349–57. pmid:17872937
- 34. Sinha IP, Smyth RL, Williamson PR. Using the Delphi Technique to Determine Which Outcomes to Measure in Clinical Trials: Recommendations for the Future Based on a Systematic Review of Existing Studies. PLOS Medicine. 2011;8(1):e1000393. pmid:21283604
- 35. Hsu C-C, Sandford B. The Delphi Technique: Making Sense of Consensus. Practical Assessment, Research and Evaluation. 2007;12:10.
- 36. Van de Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. Am J Public Health. 1972;62(3):337–42. pmid:5011164.
- 37. Prinsen CAC, Vohra S, Rose MR, Boers M, Tugwell P, Clarke M, et al. How to select outcome measurement instruments for outcomes included in a “Core Outcome Set”–a practical guideline. trials. 2016;17(1):449–. pmid:27618914