Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Towards reporting guidelines of research using whole-body vibration as training or treatment regimen in human subjects—A Delphi consensus study

  • Anika Wuestefeld,

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Visualization

    Affiliation Department of Clinical and Developmental Neuropsychology, University of Groningen, Groningen, Netherlands

  • Anselm B. M. Fuermaier ,

    Roles Conceptualization, Methodology, Supervision, Writing – review & editing

    a.b.m.fuermaier@rug.nl

    Affiliation Department of Clinical and Developmental Neuropsychology, University of Groningen, Groningen, Netherlands

  • Mario Bernardo-Filho,

    Roles Conceptualization, Writing – review & editing

    Affiliation Laboratory of Mechanical Vibration and Integrative Practices, Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brazil

  • Danúbia da Cunha de Sá-Caputo,

    Roles Conceptualization, Writing – review & editing

    Affiliations Laboratory of Mechanical Vibration and Integrative Practices, Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brazil, Faculty Bezerra de Araújo, Physiotherapy Course, Universidade do Estado do Rio de Janeiro, Rio de Janeirom, Brazil

  • Jörn Rittweger,

    Roles Conceptualization, Writing – review & editing

    Affiliation German Aerospace Center, University of Cologne, Cologne, Germany

  • Eckhard Schoenau,

    Roles Conceptualization, Writing – review & editing

    Affiliations Department of Pediatrics, Medical Faculty and University Hospital, University of Cologne, Cologne, Germany, Center for Prevention and Rehabilitation, UniReha GmbH, Medical Faculty and University Hospital, University of Cologne, Cologne, Germany

  • Christina Stark,

    Roles Conceptualization, Writing – review & editing

    Affiliation Department of Pediatrics, Medical Faculty and University Hospital, University of Cologne, Cologne, Germany

  • Pedro J. Marin,

    Roles Conceptualization, Writing – review & editing

    Affiliation CyMO Research Institute, Valladolid, Spain

  • Adérito Seixas,

    Roles Conceptualization, Writing – review & editing

    Affiliation Escola Superior de Saúde, Universidade Fernando Pessoa, Porto, Portugal

  • Stefan Judex,

    Roles Conceptualization, Writing – review & editing

    Affiliation Department of Biomedical Engineering, Stony Brook University, Stony Brook, NY, United States of America

  • Redha Taiar,

    Roles Conceptualization, Writing – review & editing

    Affiliation University of Reims, Reims, France

  • Csaba Nyakas,

    Roles Conceptualization, Writing – review & editing

    Affiliation Department of Morphology and Physiology, Faculty of Health Sciences, Semmelweis University, Budapest, Hungary

  • Eddy A. van der Zee,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Department of Molecular Neurobiology, University of Groningen, Groningen, Netherlands

  • Marieke J. G. van Heuvelen,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Center for Human Movement Sciences, University of Groningen, University Medical Center Groningen, Groningen, Netherlands

  • Oliver Tucha

    Roles Conceptualization, Methodology, Supervision, Writing – review & editing

    Affiliation Department of Clinical and Developmental Neuropsychology, University of Groningen, Groningen, Netherlands

Abstract

Background

Whole-body vibration (WBV) is a method utilizing vibrating platforms to expose individuals to mechanical vibration. In its various applications, it has been linked to improved muscular, skeletal, metabolic, or cognitive functioning, quality of life, and physiological parameters such as blood pressure. Most evidence concerning WBV is inconclusive and meta-analytical reviews may not readily produce insights since the research has a risk of misunderstandings of vibration parameters and incomplete reporting occurs. This study aims at laying an empirical foundation for reporting guidelines for human WBV studies to improve the quality of reporting and the currently limited comparability between studies.

Method

The Delphi methodology is employed to exploit the integrated knowledge of WBV experts to distil the specific aspects of WBV methodology that should be included in such guidelines. Over three rounds of completing online questionnaires, the expert panel (round 1/2/3: 51/40/37 experts respectively from 17 countries with an average of 19.4 years of WBV research experience) rated candidate items.

Results

A 40-item list was established based on the ratings of the individual items from the expert panel with a large final consensus (94.6%).

Conclusion

The final consensus indicates comprehensiveness and valuableness of the list. The results are in line with previous guidelines but expand these extensively. The present results may therefore serve as a foundation for updated guidelines for reporting human WBV studies in order to improve the quality of reporting of WBV studies, improve comparability of studies and facilitate the development of WBV study designs.

Introduction

Whole-Body Vibration (WBV) has received increasing attention in the field of physical, physiological, and psychological rehabilitation [13]. WBV is a method utilizing vibrating platforms (i.e., training devices) to expose the individuals’ whole body to vibration (usually with a frequency between 5–60 Hz, depending on the device). This technique was originally aimed at improving physical, muscular, and skeletal functioning [46], but has been employed for more widespread applications since. So, the focus lies on WBV as a training or treatment method, which must be distinguished from detrimental vibrations that people may be exposed to during transportation or in occupational settings [7].

In its original application, the short- and long-term effects on physical functioning within different scientific fields including medicine and movement sciences were examined [810]. More recently, WBV was also evaluated for its effects on, among others, blood pressure [11], body mass [12,13], pulmonary function [14], quality of life [15] and cognitive functioning [1618]. WBV research in clinical settings is complemented by evidence from animal studies [1922]. With these new applications, the field of WBV research became more extensive, leading to (1) an increasing number of studies, (2) new and additional outcome measures, (3) further methodological differences regarding devices, vibration parameters and research designs used [23], and ultimately, (4) growing interdisciplinarity.

Across all applications of WBV in the various research fields, the evidence concerning its effects on various functions has been inconsistent (e.g., see these reviews [6,8,10,2429]). Based on the inconclusive evidence within the different fields, no firm conclusions about the effects of WBV training machines can be drawn as of now. More high-quality studies with improved methodology [30] and complete and correct reporting with sufficient detail may help to receive greater insight into the effects of WBV. Such studies can form the basis for future meta-analytic studies [31] in order to identify modulating factors of WBV effects and, ultimately, to identify optimal settings.

WBV regimes are defined by several technical components. Essential vibration parameters include, but are not limited to, vibration frequency, amplitude or acceleration, and duration [1,6,23,32]. These parameters are commonly reported inaccurately or completely omitted from study reports [32] and can be subject to misunderstandings because of inconsistent terminology (e.g. amplitude vs. peak-to-peak amplitude vs. peak-to-peak displacement) and lack of information how these parameters were assessed [1,32]. Thus, misunderstandings or omissions may not be readily apparent and may be included in published work. Based on the interdisciplinarity of WBV research, coherent terminology may be difficult to attain [32]. Yet, omitting important information or using vibration-specific terms interchangeably may lead to inaccurate conclusions and will preclude high-quality study designs, their replicability, the reporting of results, the comparability between studies, and the ability to explain inconsistencies [1]. Therefore, complete and correct use of terminology is essential to ensure high-quality reporting, as well as increasing the possibility and validity of comparing findings (meta-analyses or systematic reviews). The possibility that these consistencies lead to the incomparability of studies urges the consideration of consistent reporting.

As a consequence of these issues and to achieve a deeper understanding of the possible effects of WBV, complete, consistent, and correct reporting is vital. A common manner to ensure consistent reporting is the adherence to standardized reporting of research. The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network [33] provides reporting guidelines such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) or the Consolidated Standards of Reporting Trials (CONSORT) statement [34,35]. These statements identify key characteristics of any high-quality publication.

Adherence to already existing general reporting guidelines, like CONSORT, is not sufficient to ensure high-quality reporting in the field of WBV since the guidelines are not specific enough to ensure complete, consistent, and correct reporting. The inherent technical components of WBV and their susceptibility to misunderstandings, the large variation in devices, parameter settings, and its applications require tailored guidelines. Therefore, instead of following only the CONSORT statement, new guidelines applicable to WBV research should be adopted. Ideally, these guidelines will function as an addition to the CONSORT statement and apply to all research utilizing WBV.

Previous efforts towards developing reporting guidelines specific to WBV research specified reporting recommendations for WBV studies and defined and explained the physical principles of WBV (Rauch and colleagues [1], representing the International Society of Musculoskeletal and Neuronal Interactions). The recommendations were based on suggestions by experts, but suggestion collection and the decision process behind selecting suggestions for the recommendations are unclear. In addition, recent developments in the field of WBV research (e.g., new outcome measures, changes of devices, animal or cell culture studies) are not incorporated.

The EQUATOR Network published recommendations on establishing new reporting guidelines [36] and provides a platform for guidelines to increase their impact. In order to encourage the development of guidelines when needed, the Network recommends that reporting guidelines should (1) include a range of stakeholders with expertise, (2) generate a list of items through, for example, a Delphi consensus study, followed by (3) meetings of the executive group to discuss items or additional information which should be included [36].

Unlike prior publications initiating reporting guidelines or explanations of terminology [1,6,32], systematically obtained consensus between WBV experts about the crucial aspects of reporting human WBV research can lay the foundation for reporting guidelines. These guidelines should be widely accepted by the scientific community, can be used in all disciplines using WBV, and follow the recommendations of the EQUATOR Network. Thus, the aim of the present study is to commence the development of such reporting guidelines. The aim is not to collect aspects of studies relevant to reporting research which are outlined in general reporting guidelines (e.g., the CONSORT statement) not specific for WBV research (e.g., sample size, age of participants, type of study). The final reporting guidelines, which will be predominantly based on the present study and finalized by the executive group, will act as an addition to these general reporting recommendations. An overview of the background of the 14 members of the executive group can be found in the supporting information (S1 File).

The Delphi consensus methodology is utilized to inquire and collect the opinion and knowledge of experts in the field from various backgrounds in the present study. This complies with the recommendations of the EQUATOR Network, is a first step in the process of developing reporting guidelines for WBV research on humans, and uses the prior publication by Rauch et al. [1] as a starting point.

Method

Delphi consensus methodology

The present study utilized the Delphi method. This method is a structured process to gather information about a certain topic from a relevant group of people or experts [37]. It is based on the assumption that the combination of opinions from various individuals lies closer to the truth than the opinion of one person alone [38]. The Delphi method generally includes several rounds during which the opinion of experts is inquired in a structured, yet interactive, manner [39]. Throughout these rounds, controlled feedback is given to gain the most reliable consensus of a panel of experts. This allows re-evaluation of prior responses and is thought to produce trustworthy and reliable information. Building on the knowledge of the participants, the result reflects their insights and opinions [40].

For the purposes of this study, a modified Delphi method of three rounds was employed to gain an understanding about which aspects should be included in reporting guidelines for human WBV studies. An overview of the rounds is given in Table 1 and the full instructions and questionnaires can be found in the supporting information (S2 File). If necessary, more rounds would be added during the progress of the study. Instead of the traditional Delphi approach of starting the first round with open-ended questions [41], the first round in the present study was modified to begin with a pre-selected list of items. It has been deemed acceptable to modify the first round if it is based on literature and prior knowledge; is a common approach to the first round [41,42] and may increase response rate [43]. The This list of items in the present study was constructed by means of a literature review, the previous recommendations by Rauch et al [1] and related publications [6,23,32], experience of the authors, and a discussion during the second conference of the World Association of Vibration Exercise Experts (WAVEX; August 2018, Groningen, The Netherlands).

thumbnail
Table 1. Overview of the Delphi rounds for the establishment of reporting guidelines in human WBV studies.

https://doi.org/10.1371/journal.pone.0235905.t001

Based on the differences between animal and human research [44], the present study will focus on WBV studies with human subjects only. A separate Delphi study will be conducted to initiate reporting guidelines for animal and cell culture studies.

The rounds were conducted consecutively starting in February 2019 and the conclusion of the last round in May 2019. The panel members were asked to complete each questionnaire within three weeks. A weekly reminder to participate in the current round was sent to the experts who did not complete the questionnaire yet. In the first round, potential panel experts who are members of the WAVEX received an additional reminder to participate from the board of the association after two weeks. All questionnaires were created with the online survey software Qualtrics (Version 02/2019-05/2019 of Qualtrics; Copyright © 2019 Qualtrics; Provo, UT, USA [45]) and distributed via Qualtrics by e-mail. This software additionally ensured anonymous data collection throughout the whole study process.

The responses to the individual questionnaires were treated confidentially and the study was approved by the Ethical Committee Psychology (ECP) affiliated to the University of Groningen, the Netherlands on 20/12/2018 (ref. no.; 18211-O).

Criteria regarding consensus achievement

Consensus about the aspects which should be included in reporting guidelines was is defined as an agreement (yes, of importance to report) of at least 70% [46,47] while the disagreement rate [48,49] (no, not important to report) is below 20% prior to the start of the study (see Table 2). An agreement of 20% or below and a disagreement of 20% or above by the expert panel is regarded as judging the item irrelevant. In case items were neither agreed nor disagreed upon (below 70% of agreement and below 20% of disagreement or agreement above 70% and disagreement above 20%) or fall directly on the set criteria (e.g., an agreement of exactly 20%), the items are named as ‘optional items’ in the final list of aspects of the present study, which may be valuable in certain studies, as automatic exclusion may not be justified [50]. These criteria were set by the executive group after consulting the literature of Delphi studies and were judged to reasonably indicate consensus. Yet, based on the lack of agreement about the definition of consensus [41,46], these values are to be seen as indicative of agreement and not as an absolute judgement.

thumbnail
Table 2. Overview of consensus criteria specifying the agreement-rates and corresponding disagreement-rates.

https://doi.org/10.1371/journal.pone.0235905.t002

The results are analyzed quantitatively with: (1) Krippendorff’s alpha to estimate overall agreement per round [5153]; (2) the McNemar χ2-test to estimate stability of ratings between round 1 and 2 [46,54]; (3) and Cohen’s Kappa to estimate the agreement between round 1 and 2 per item [55,56]. These estimates can be found in the supporting information (S3 File) since violations of the assumptions of these estimates may reduce their validity.

Participants and recruitment

Suitable panel members with expertise of WBV from any scientific discipline making use of WBV in human studies are essential to the present study. The inclusion criteria were designed to ensure the high expertise of possible panel members. Thus, the inclusion criteria were the following: (1) potential panel members must be willing to take part in the study and (2) have at least two English, scientific (peer-reviewed) publications utilizing WBV with human participants. Exclusion criteria were: (1) one English scientific (peer-reviewed) publication utilizing WBV with human participants and (2) if two or more scientific (peer-reviewed) publications were published but in a language other than English (i.e., only no or one English peer-reviewed publications) possible experts were excluded. Experts were identified through membership in the WAVEX (https://internationalwavexmeeting.wordpress.com/), publications determined via the scientific databases PubMed, PsycInfo, and Web of Science, and by recommendations of members of WAVEX. Based on this a list of potential experts was compiled and invitations to participate in the study were sent to them. Additionally, the experts who received an invitation had the opportunity to recommend further experts which could take part in the study.

In total 136 experts were contacted. Fifty-six of these opened and filled in parts of the first questionnaire (41.2% response rate). Fifty-one of these 56 participants completed the questionnaire (51/56 = 91.1% completion rate). The expert panel consisted of the 51 participants who completed the first questionnaire and only these experts were contacted in the subsequent round. In the second round, 40 panel members completed the survey (78% response rate). Only the experts who completed round 2 were invited to the third round. Thirty-seven of these 40 experts completed round 3 (92.5% response rate).

Materials and procedure

An overview of the procedure of the present study is given in Fig 1 and the full instructions and questionnaires of all three rounds can be found in the supporting information (S2 File).

thumbnail
Fig 1. Flowchart of the Delphi study building towards reporting guidelines in human WBV studies.

‘n’ refers to the number of items in each category.

https://doi.org/10.1371/journal.pone.0235905.g001

Round 1.

The first round had two aims: (1) to receive an initial rating whether the pre-determined items should be included in guidelines concerning WBV studies with humans and (2) to collect suggestions of additional aspects the experts see as relevant. In the beginning of this round, the expert panel members received information about the aim of the study and, after giving informed consent, were asked to answer several general demographic questions. In addition to age and gender, participants also indicated their highest completed education, academic background and position, years of experience with research and clinical work, the current distribution of their time (research/clinical/other), years of experience with WBV in research and clinical settings, and number of published studies utilizing WBV.

After completing the short demographic questionnaire, the experts were presented with the pre-determined items of importance. These 48 pre-determined items were separated into four categories: information about the (1) vibration, (2) device, (3) administration, and (4) participants. The experts were asked to indicate whether they agree (‘yes’), do not agree (‘no’), or are neutral (‘neutral/ don’t know’) about the importance of each specific item when reporting human WBV studies. After each block, the experts had the opportunity to suggest additional items and add questions or comments.

Two open questions were included (item 6 and 36). If experts agreed with item 6 (reporting how vibration parameters are measured), they were asked to suggest how the measurements should be completed. Similarly, if experts agreed with item 36 (reporting of participants subjective experiences during WBV), they were asked to list the, in their opinion, relevant experiences.

Round 2.

The aim of the second round was twofold. First, the experts were asked to indicate their judgement about the items of the first round, while taking the first rounds ratings of the panel into account. Hence, the experts re-evaluate their initial responses to the items in light of their fellow experts’ opinion. This process is thought to be crucial to establish consensus, as convergence of the individual opinions occurs [57]. Second, the additionally suggested items and the results of the two open questions of the first round were presented to the experts. Similar to the pre-determined items, the experts’ judgement on whether to include these items in reports of human WBV studies was inquired.

Round 3.

Similar to round 2, the items with consensus (agreement of at least 70% while the disagreement rate is below 20%) regarding their importance to report, along with their ratings of the second round, were presented.

The aim of the third round was to seek the experts’ agreement on the final list of items. The panel members were asked to indicate agreement (yes/no) with the whole agreed-upon list based on the previous rounds (see Fig 1). It was pointed out to the experts that not all items may generally apply to each WBV study. However, items may still be relevant for particular studies. In case of disagreement, the experts were asked to specify the reason for disagreement, i.e., which aspect they disagree with.

Results

Round 1

Sample characteristics.

Table 3 gives an overview of the demographic characteristics of the 51 expert panel members. 41.2% of the experts were female and the experts’ age was distributed roughly equally across age groups with a mean of M = 46 years (SD = 10.4 years) ranging from 27 to 68 years. Most experts obtained a PhD (n = 30) and fewer a master’s degree (n = 4) or a professorship (n = 17) and had an average of 18 years of research experience and 11 years of clinical experience (Table 3). The panel experts were located in 17 different countries. Various academic backgrounds were reported, including medicine, engineering, physiotherapy, psychology, and movement sciences. The expert panel reported an average of 11 publications utilizing WBV with humans ranging from 2 to 40. Additionally, 37 experts reported clinical experience with WBV.

thumbnail
Table 3. Demographic characteristics of the expert panel.

https://doi.org/10.1371/journal.pone.0235905.t003

Agreement on pre-determined items.

The ratings of the pre-determined list can be found in Table 4. For the first category information about the vibration 14 out of 16 items had an agreement rating of at least 70% (items 1–14). For item 15 (report the accuracy of vibration parameters), 68.6% of the experts were in favor of reporting the item. The remaining experts responded ‘neutral/do not know’ (23.5%) rather than ‘no’ (7.8%). For item 16 (which acceleration (in RMS) occurs), 37.3% of the experts were in favor of reporting. More than 20% of the experts voted against reporting this item, while 39.2% did not make a definite decision (‘neutral’).

thumbnail
Table 4. Pre-determined items of importance when reporting human WBV studies and their importance rating of round 1 and 2.

https://doi.org/10.1371/journal.pone.0235905.t004

For the items referring to the information about the device, four of six items were agreed upon (items 17–20). The two items of no-consensus; item 21 (how the energy is generated (e.g. direct mechanical transmission or electromagnetic transmission) as this can influence the performance of the device) and item 22 (how the energy is generated the size of the vibration platform) had a higher rating of ‘neutral/do not know’ than favoring against reporting it.

Information about the administration items were generally agreed upon for 14 of the 16 items (item 23–36 with an agreement of 70% or higher). For item 37 (reporting the location of the intervention) and 38 (reporting the conditions of the test room) most experts not in favor voted against reporting these items.

Five of the ten items concerning information about the participants were agreed upon (items 39–43). For item 44 (how it is assured that neck and head are not negatively affected by the vibration) and 47 (whether possible glasses were disturbing during the vibration), most experts not in favor of reporting responded with ‘neutral/do not know’. Most of the remaining experts voted against reporting item 45 (the participants' clothing during the WBV), 46 (skin and muscle perfusion during the WBV), and 48 (whether the participants wore glasses during WBV).

All answers regarding the two open questions can be found in the available dataset. For item 6 (how should vibration parameters be measured), the use of an accelerometer was suggested 24 times. Other suggestions were brought forward three times or less. These answers were rated for their importance in the second round of the study (see Table 5).

thumbnail
Table 5. Round 2 ratings of the additionally suggested items of round 1.

https://doi.org/10.1371/journal.pone.0235905.t005

Six responses to item 39 (which subjective experiences of WBV should be reported) were suggested at least four times. These items are (1) level of pain, (2) dizziness, (3) fatigue, exhaustion, and/ or tiredness, (4) tingling, itching, and/or burning sensations, (5) perceived effort or exertion, (6) experiences of (dis)comfort. Other suggestions were brought forward three times or less and are included in the available dataset.

Suggested items.

A list of additionally suggested items and comments for each of the four categories are included in the dataset. The first part of the questionnaire regarding information about the vibration elicited several comments and additional items.

First, it was suggested twice that surface electromyography (sEMG) should be included to evaluate and record the electrical signals produced by muscle activity. Second, it was suggested twice that the way the participants’ body mass might influence the vibration parameters should be measured. It has further been suggested that this could be done by comparing the vibration parameters of the empty vibration device and while a participant is using it. Third, it was suggested once that the acceleration in ms-2 RMS should be reported. Fourth, it was suggested that accelerations should be measured vertically and horizontally.

For information about the vibration, it was suggested twice that regardless of using own or manufacturer settings (item 11), the vibration parameters should always be measured, as they can be faulty. Therefore, several questions were added to the questionnaire of round 2. First, whether it should be reported if manufacturer settings are used but additionally report own vibration parameter measurement; second, whether it is sufficient to report manufacturer settings without additional measurements; and third, whether vibration parameters should always be measured regardless whether manufacturer settings are used or not.

For the second category information about the device some additional items were suggested. Yet, they were judged to be unrelated to this category (see dataset for the comments) and thus no additional questions from these comments were added to the questionnaire of round 2.

For information about the administration, two additional items were suggested. First, reporting whether parts of the subjects are exposed to vibration in contrast to whole body vibration (e.g., only the feet) was suggested twice. Second, it was suggested once to report the decision which parts of the subjects are exposed to vibration and why.

For information about the participants two additional items were suggested. First, it was commented three times that the training history of the subjects should be reported. Second, it was suggested once to report the subjects’ history of injuries.

All comments and additional suggestions are included in the dataset.

Round 2

Sample characteristics.

See Table 3 for an overview of the demographic characteristics of the 40 panel members that completed round 2.

Integration of comments from round 1.

In round 2, three of the items of the first round were rephrased based on comments from the experts. Item 8 (report which acceleration occurs) was rephrased to which acceleration occurs based on pilot study or relevant literature. Item 14 (where on the platform the vibration parameters are measured, if own settings are used) was rephrased to where on the platform the vibration parameters are measured, since it was commented by the expert panel that the vibration parameters have to be assessed since the information of the manufacturers may be inaccurate. Item 10 (the position on the platform where peak-to-peak displacement (PPD) or amplitude is measured) was rephrased due to comments of the expert that the phrasing was not optimal. It was incorporated into item 14 since PDD and amplitude are vibration parameters.

Agreement ratings.

The results of the judgement on the pre-determined items which were presented along with the ratings of the first round can be found in Table 4.

For items previously judged as important, agreement rates mostly increased in all four categories (see Table 4). For the first category information about the vibration the same items as in the first round had an agreement of at least 70% and no disagreement over 20% (items 1–14 excluding item 10). For all, except four items (1, 2, 6, 7), the agreement rate increased. Similarly to the first category, the second category information about the device also showed similar ratings in round 2 as in round 1. Again, the items previously judged as important to report, received higher agreement rates in the second round (items 17–20). Information about the administration items were agreed upon for the same items (items 23–35 with an agreement of 70% or higher) of the first round. As depicted in Table 4, ratings of item 36 fell below the 70% threshold and the ratings of item 37 increased. The same items regarding information about the participants were agreed upon (items 39–43) in round 2 and the agreement-rate increased for all.

The ratings of the additional suggested items and the responses of the open questions are presented in Table 5. For the additional items concerning information about vibration, no item was agreed-upon in terms of necessity of reporting. In the first round, the question how should vibration parameters be measured? was answered by the experts. These answers were rated in the second round (see Table 5). An agreement was established for the use of accelerometers (77.5%), with frequency and amplitude (92.5%), and frequency, amplitude, and peak-to-peak displacement (70%). Since various usages of accelerometers were suggested in the first round, the experts were additionally asked to rate how the accelerometer should be used, if agreed to it. The majority voted for the use of 3D-accelerometers on both participant and platform and horizontally and vertically (see Table 5).

Two additional items were suggested for the information about the administration (see Table 5). Both reached an agreement-rate of 70% or higher (82.5% and 70%). However, one of these items also reached a disagreement-rate of 20% (see Table 5).

For the category information about the participants two suggestions were made in the first round. The first suggestion was agreed-upon, while the second was not (see Table 5). Supplementary to the additional suggestions, the answers of the open question of which subjective experience should be reported were rated. Ten items were agreed upon and two were disagreed upon (see Table 6).

thumbnail
Table 6. Final list of aspects of human WBV studies after round 3 of this Delphi study.

https://doi.org/10.1371/journal.pone.0235905.t006

Round 3

Sample characteristics.

See Table 3 for an overview of the demographic characteristics of the 37 panel members that completed round 3.

Final agreements.

The final, agreed-upon, list can be found in Table 6. Thirty-five of the 37 experts comprising the expert panel in the last round, agreed with this final list (agreement of 94.6%). One of the experts disagreed with the subjective experiences of the list, without giving further explanations. The second disagreement concerned the statement of ‘how vibration parameters should be measured’ with the explanations that these recommendations are mutually exclusive, and it needs to be clarified whether 1D- or 3D-accelerometers should be utilized. The statements of the two disagreeing experts can be found in the available dataset.

Discussion

The aim of the present study was to lay an empirical foundation for the first steps in the development of reporting guidelines for all scientific disciplines applying WBV as a training or treatment method to human subjects. This was based on the recommendations by the EQUATOR Network [36] with a three-round Delphi method. As elaborated in the introduction, a need for updating and extending the recommendations by Rauch et al. [1] was identified on the grounds of creating the guidelines in a systematic fashion and the still persisting incomplete reporting and misunderstandings of terminology. The combination of these aspects may hamper the comparability of studies, interferes with their replicability, and ultimately prevents the achievement of pivotal insights into the effects of WBV. Therefore, the knowledge and opinions of international experts working with WBV was collected and integrated.

The Delphi method is widely used due to its advantages. First, it can be applied to groups of varying size [58]. Second, it is thought to minimize issues of collective decision making, such as reciprocal influence and lack of anonymity [37] and, thus, prevents the imposition of opinions from dominant individuals [59]. Third, with the Delphi method, the opinion of a large number of individuals with diverse geographical and professional backgrounds is collectible. Fourth, anonymity allows the participants to opinionate unpopular aspects and change their responses [60]. Therefore, the utilization of the Delphi method appears appropriate to lay the foundation for the reporting guidelines and as the methods of the present study.

Throughout this Delphi study, a “yes”/“no”/“neutral/don’t know” rating system is used. The experts are asked to consider whether one specific item should be reported in a publication of a WBV study or not. Additionally, the experts are instructed to consider the whole field of WBV, as opposed to personal studies, since the guidelines should be relevant for all applications of WVB. Thus, another rating system was judged to reduce the explanatory power of the results due to a missing indication about the reasoning why an item was judged, for example as “very important” compared to “important”.

In the next step, the executive group will use the generated list of the Delphi study to develop the reporting guidelines, according to the EQUATOR Network [36] recommendations. Issues that will be considered are (1) modifications of items (e.g., rephrasing), (2) adding further items deemed as important (e.g., define the boundaries of low and high frequency), (3) discussing the comments of disagreeing experts and potential consequences for the guidelines, (4) decide about the inclusion/exclusion of the optional items, and (5) the publication and dissemination strategy (e.g., register at EQUATOR Network).

The three rounds of the present study ensured that the expert panel was able to refine their judgements about the pre-determined items, suggest additional items, and rate all items repeatedly. A large consensus on the final list of aspects was established according to this Delphi study (94.6%). The results show high levels of consensus for various items, as judged by an agreement rate of 70% while the disagreement rate was below 20%. The items which attained consensus were included in the final list (see Table 6).

Even though most items reached either an agreement or disagreement, some items were left without consensus after the three rounds (i.e., no agreement of 70% or higher and no disagreement of 20% or higher). We propose that these optional items could be individually considered for reporting as they can be valuable for particular studies. There are 19 optional items (see Tables 3 and 4 for ratings). These items were disproportionally often rated as neutral compared to items of consensus (see Tables 4 and 5). It can be speculated that either unfamiliarity with the item or ambiguously posed questioning of the item were reason of reaching no consensus. Since items were rephrased throughout the three rounds, ensuring proper and unambiguous phrasing, possible unfamiliarity with items is conceivable. Thus, considering the optional items for future WBV studies and reporting is sensible depending on the study. The final decision about these items will be completed by the executive group and reported on in the new guidelines.

The identified items of the final list largely correspond to the advice given by previous publications concerning the terminology and vibration parameters [6,32] as well as the reporting recommendations of WBV studies [1]. The aspects discussed by Lorenzen et al. [32] (i.e., magnitude of vibration), which are commonly misused or misunderstood, are in line with the final list of the present study. Similarly, the technical parameters discussed by Rittweger [6] correspond to the first category (information about the vibration) of the final list. Deviating from the final list, Rittweger [6] emphasized the physiological effects of WBV (e.g., EMG, skin and muscle perfusion, safety concerns). Some of these aspects were included in the first round of the present study (e.g., report how it is assured that neck and head are not negatively affected) or were suggested by experts. However, these aspects did not reach consensus and were only included in the optional list (see Tables 4 and 5).

While safety concerns have been given extensive consideration before [6,23], according to the experts’ ratings, this information may not be generally applicable for WBV reporting guidelines. Consideration of safety is, nevertheless, relevant for WBV study design and the mentioned references may be consulted. However, this is a separate issue requiring more attention and investigation. Thus, recommendations concerning safe parameter settings will not be a part of the upcoming reporting guidelines.

One of the starting points for creating the initial included aspects for the Delphi study were the recommendations by Rauch et al. [1]. Their suggestions are, with few exceptions, consistent with the final list of the present study. Instead of the four categories of the present study, Rauch and colleagues [1] recommend reporting aspects of two categories: (1) items related to the WBV device and (2) items related to study participants.

All aspects discussed in their recommendations concerning the first category [1], except skidding of feet (optional item), assessing skidding (not included), and changes in vibration settings (not included), reached consensus in the present study. The aspect of changes in vibration settings may not be relevant for WBV research per se as (1) changes in the protocol of studies should always be reported [34] and (2) reporting a-priori planned changes of vibration parameters is an inherent part of item 4 (which frequency (Hz) is used; Table 6) and other items specifying vibration parameters. Aspects of Rauch and colleagues’ second category [1], which are not directly included in the present study are the explained difficulties related to assessing vibration parameters on the body of participants. Yet, this is incorporated in item 14 of the final list (how vibration parameters should be measured; Table 6).

While the results of the present study are in accordance with the 13 recommendations by the team around Rauch [1], the results add further 27 WBV-specific aspects to the recommendations. For example, the information concerning participants and administrations, as well as more specifications concerning information about vibration and the device, such as where and how the vibration parameters are measured. Therefore, the final list (1) is an update on previous recommendations, (2) extends beyond the category information about vibration, (3) is approved by experts in the field, and (4) may, thus, serve as the foundation on which reporting guidelines can be established upon.

Limitations

Based on its advantages and the aim of the study, the utilization of the Delphi method was considered appropriate in order to empirically identify items of importance for reporting guidelines of human WBV studies. Yet, some possible limitations must be acknowledged.

Regarding the response rate, of the 136 originally invited experts, 56 opened and started to fill out the first questionnaire and 51 of these completed the questionnaire (41.2% of the invited experts). However, it remains unclear why some invited experts participated in the study while others did not. Thus, the possibility of non-response bias [61] is conceivable, which may hamper the significance of our results. Plausible explanations for non-responses include, but are not limited to, the reason that (1) invitation e-mails were filtered out by the recipient e-mail providers, (2) the expert forgot to fill in the questionnaire despite reminders, (3) the time requirement of participation was too high, (4) the experts were too occupied, (5) the expert may not render the study as relevant, (6) the potential panel member felt they were lacking expertise, and/or (7) one expert participated on behalf of their research team. Since the reasons for non-response have not been investigated, the possible bias on the results cannot be determined. These considerations also apply to the drop-out over the three rounds since it remains unclear why some experts did not continue to be part of the expert panel. The demographic data of the participants over the three rounds indicates that experts from various geographical and academic backgrounds did not participate in subsequent surveys. There is, however, an indication that more physical therapists from Brazil dropped out (42% of overall drop-out) and dropped out panel members who are physical therapists had either a M.Sc. (2 of 7) or PhD (5 of 7). These observations could indicate a potential bias. Possible explanations could be, but are not limited to, that physical therapists in Brazil (1) have a higher work-load and, thus, less time to respond to the surveys in comparison to the other experts; (2) deem the commencement of reporting guidelines unnecessary; (3) respond for a research team more frequently; (4) felt they have less expertise to answer the questions; or (5) judge the topic irrelevant for their field. The explanations of drop-out cannot be (dis)proven without contacting these experts. However, the drop-out led to only very slight changes in age, experience rate, number of published studies and the other demographic information. Notwithstanding, 72.5% of the 51 experts of the first round participated in the study until the end and continued to be part of the panel and measures were taken to reduce the drop-out rate (i.e., reminder e-mails). This rate is sufficient to achieve a valid Delphi process [62,63].

The second limitation of the Delphi method and the present study is the arbitrary determination of a consensus rate. There is no agreed-upon consensus definition in the scientific literature and is defined differently in various studies utilizing this method ranging from an agreement rate of 51% to 100% [37,39,46]. The rational of the boundary conditions for the present study was based on the probable unattainability of an extremely high agreement rate (e.g., 100%), yet the agreement should be considerably higher than 50%. The disagreement condition was introduced to ensure that an item is not accepted in case of diverging opinions, i.e., if there are substantial numbers of experts agreeing but also disagreeing or a high number of experts refrain from giving a definite rating (i.e., choose “neutral/don’t know”). Items without consensus were not disregarded but suggested as optional items. Thus, some items that may be important are not omitted entirely but the reader has the chance to make his or her own evaluation.

Based on the evolving field of WBV research and the limitations of the present study, future research may potentially re-evaluate and update the presented list once the reporting guidelines have been established.

Strengths

The expertise and knowledge of the panel is one of the main determinants of the validity of the study [39]. If expertise is limited, the results may not reflect the opinion of the scientific community and important aspects may be neglected. One indication of expertise are the characteristics of the panel. For the present study, the characteristics of the experts indicate that they indeed created a panel with expertise and valuable insights in the utilization of WBV, even when considering drop-out. With most panel members having a doctor title (57%) or being a professor (41%), the academic level of the experts was high. Also, the number of publications (M = 11.6 in round 3) and years of experience with WBV in research (M = 19.4 in round 3) and clinical settings (M = 11.6 in round 3) supports the presence of expertise. The inclusion of experts from various geographic locations and academic backgrounds further points to a heterogeneous, experienced expert panel, which increases the quality of responses, as more alternatives are likely to be considered [39,64]. Even though the panel appears to consist of experts of WBV research, their level of expertise of all aspects of WBV may vary. This may explain the high neutrality ratings of the items of no consensus (Tables 4 and 5).

Small panel sizes and response rates may distort validity of the results of a Delphi study. There is little data on panel size and its effect on validity of reaching consensus [64]. Yet, the reliability is thought to increase with panel size, while differences may be small with a panel of more than 12 experts [64]. The present study included 37 to 51 experts, exceeding this advised minimum group size. Additionally, the experts’ characteristics remained diverse regarding age, gender, and geographic location throughout the rounds. One may therefore conclude that the panel size was adequate to collect information and base conclusions on the panel’s expertise [37].

Conclusion

The present study aimed to commence the establishment of reporting guidelines for human WBV studies based on an empirical inquiry. A list of 40 aspects which were judged to be important in reporting WBV studies (94.6% agreement of the expert panel) was created. This study represents and includes a wide range of experts from various academic and geographic backgrounds and extends the literature to aspects of other important categories, such as information about the administration. Thus, the study adds to the existing literature by creating a list of approved-upon items. Furthermore, the guidelines that will be based on the study will extend the general reporting guidelines (e.g., CONSORT statement [34]) for clinical trials and acts as an addition to it. The need of the reporting guidelines, the high agreement on the final list, generally praising comments from the experts, and a low drop-out rate indicate the relevance and importance of the present study.

Supporting information

S1 File. Overview of the executive group.

https://doi.org/10.1371/journal.pone.0235905.s001

(DOCX)

S2 File. All questionnaires of the three rounds.

https://doi.org/10.1371/journal.pone.0235905.s002

(DOCX)

Acknowledgments

We would like to express our gratitude to the experts who participated in the Delphi panel and contributed their time and knowledge in order to enable the presented commencement of WBV reporting guidelines.

References

  1. 1. Rauch F, Sievanen H, Boonen S, Cardinale M, Degens H, Felsenberg D, et al. Reporting whole-body vibration intervention studies: recommendations of the International Society of Musculoskeletal and Neuronal Interactions. J Musculoskelet Neuronal Interact. 2010;10: 193–198. pmid:20811143
  2. 2. van der Zee EA, Heesterbeek M, Tucha O, Fuermaier ABM, van Heuvelen MJG. Whole Body Vibration, Cognition, and the Brain. In: Taiar R, Machado CB, Chiementin X, Bernardo-Filho M, editors. Whole Body Vibrations–Physical and Biological Effects on the Human Body. Boca Raton: CRC press; 2019. pp. 151–170.
  3. 3. Alam MM, Khan AA, Farooq M. Effect of whole-body vibration on neuromuscular performance: A literature review. Work (Reading, Mass.). 2018;59: 571–583.
  4. 4. Whedon GD, Deitrick JE, Shorr E. Modification of the effects of immobilization upon metabolic and physiologic functions of normal men by the use of an oscillating bed. Am J Med. 1949;6: 684–711. pmid:18134065
  5. 5. Sanders CE. Cardiovascular and peripheral vascular diseases: treatment by a motorized oscillating bed. J Am Med Assoc. 1936;106: 916.
  6. 6. Rittweger J. Vibration as an exercise modality: how it may work, and what its potential might be. Eur J Appl Physiol. 2010;108: 877–904. pmid:20012646
  7. 7. International Organization for Standardization. ISO 2631–1:1997—Mechanical vibration and shock—Evaluation of human exposure to whole-body vibration. 1997.
  8. 8. Leite HR, Camargos ACR, Mendonça VA, Lacerda ACR, Soares BA, Oliveira VC. Current evidence does not support whole body vibration in clinical practice in children and adolescents with disabilities: a systematic review of randomized controlled trial. Braz J Phys Ther. 2018.
  9. 9. Bogaerts A, Verschueren S, Delecluse C, Claessens AL, Boonen S. Effects of whole body vibration training on postural control in older individuals: A 1 year randomized controlled trial. Gait & Posture. 2007;26: 309–316. https://doi-org.proxy-ub.rug.nl/10.1016/j.gaitpost.2006.09.078.
  10. 10. Lam FM, Lau RW, Chung RC, Pang MY. The effect of whole body vibration on balance, mobility and falls in older adults: a systematic review and meta-analysis. Maturitas. 2012;72: 206–13. pmid:22609157
  11. 11. Figueroa A, Gil R, Wong A, Hooshmand S, Park SY, Vicil F, et al. Whole-body vibration training reduces arterial stiffness, blood pressure and sympathovagal balance in young overweight/obese women. Hypertension research: official journal of the Japanese Society of Hypertension. 2012;35: 667–72.
  12. 12. Milanese C, Piscitelli F, Zenti MG, Moghetti P, Sandri M, Zancanaro C. Ten-week whole-body vibration training improves body composition and muscle strength in obese women. Int J Med Sci. 2013;10: 307–311. pmid:23423629
  13. 13. Song GE, Kim K, Lee DJ, Joo NS. Whole body vibration effects on body composition in the postmenopausal korean obese women: pilot study. Korean J Fam Med. 2011;32: 399–405. pmid:22745878
  14. 14. Cardim AB, Marinho PE, Nascimento JF Jr, Fuzari HK, Dornelas de Andrade A. Does Whole-Body Vibration Improve the Functional Exercise Capacity of Subjects With COPD? A Meta-Analysis. Respir Care. 2016;61: 1552–1559. pmid:27651524
  15. 15. Santin-Medeiros F, Santos-Lozano A, Cristi-Montero C, Garatachea Vallejo N. Effect of 8 months of whole-body vibration training on quality of life in elderly women. Research in Sports Medicine. 2017;25: 101–107. pmid:27885859
  16. 16. Müller SV, von Schweder AJ, Frank B, Dengler R, Münte TF, Johannes S. The effects of proprioceptive stimulation on cognitive processes in patients after traumatic brain injury. Arch Phys Med Rehabil. 2002;83: 115–121. pmid:11782841
  17. 17. Regterschot GRH, Van Heuvelen MJG, Zeinstra EB, Fuermaier ABM, Tucha L, Koerts J, et al. Whole Body Vibration Improves Cognition in Healthy Young Adults. 2014: In press. pmid:24949870
  18. 18. Fuermaier ABM, Tucha L, Koerts J, van Heuvelen MJG, van der Zee EA, Lange KW, et al. Good vibrations- effects of whole body vibration on attention in healthy individuals and individuals with ADHD. PLoS One. 2014;9: e90747–e90747. pmid:24587412
  19. 19. Téglás T, Dörnyei G, Bretz K, Nyakas C. Whole-body pulsed EMF stimulation improves cognitive and psychomotor activity in senescent rats. Behav Brain Res. 2018;349: 163–168. pmid:29702177
  20. 20. Van der Zee EA, Riedel G, Rutgers EH, De Vries C, Postema F, et al. Enhanced neuronal activity in selective brain regions of mice induced by whole body stimulation. Federation of European Neuroscience Societies Abstract. 2010;5: 24–49.
  21. 21. Heesterbeek M, Jentsch M, Roemers P, Keijser J, Tóth K, Nyakas C, et al. Whole Body Vibration Enhances Choline Acetyltransferase-Immunoreactivity in Cortex and Amygdala. Journal of Neurology & Translational Neurocience. 2017;5: 1079.
  22. 22. Boerema AS, Heesterbeek M, Boersma SA, Schoemaker R, de Vries E, van Heuvelen M, et al. Beneficial Effects of Whole Body Vibration on Brain Functions in Mice and Humans. Dose-response: a publication of International Hormesis Society. 2018;16: 1559325818811756.
  23. 23. Kiiski J, Heinonen A, Järvinen TL, Kannus P, Sievänen H. Transmission of vertical whole body vibration to the human body. Journal of bone and mineral research: the official journal of the American Society for Bone and Mineral Research. 2008;23: 1318–25.
  24. 24. Pozo-Cruz B, Adsuar JC, Parraca JA, Pozo-Cruz J, Olivares PR, Gusi N. Using Whole-Body Vibration Training in Patients Affected with Common Neurological Diseases: A Systematic Literature Review. Journal of Alternative & Complementary Medicine. 2012;18: 29–41.
  25. 25. Fischer M, Vialleron T, Laffaye G, Fourcade P, Hussein T, Cheze L, et al. Long-Term Effects of Whole-Body Vibration on Human Gait: A Systematic Review and Meta-Analysis. FRONTIERS IN NEUROLOGY. 2019;10.
  26. 26. Savage R, Billing D, Furnell A, Netto K, Aisbett B. Whole-body vibration and occupational physical performance: a review. INTERNATIONAL ARCHIVES OF OCCUPATIONAL AND ENVIRONMENTAL HEALTH}. 2016;89: 181–197.
  27. 27. Verhulst ALJ, Savelberg HHCM, Vreugdenhil G, Mischi M, Schep G.s Whole-body vibration as a modality for the rehabilitation of peripheral neuropathies: implications for cancer survivors suffering from chemotherapy-induced peripheral neuropathy. ONCOLOGY REVIEWS. 2015;9.
  28. 28. Slatkovska L, Alibhai SMH, Beyene J, Cheung AM. Effect of whole-body vibration on BMD: a systematic review and meta-analysis. OSTEOPOROSIS INTERNATIONAL. 2010;21: 1969–1980. pmid:20407890
  29. 29. Nordlund MM, Thorstensson A. Strength training effects of whole-body vibration? Scandinavian Journal Of Medicine \& Science In Sports. 2007;17: 12–17.
  30. 30. American Psychological Association. Revision of Ethical Standard 304 of the 'Ethical Principles of Psychologists and Code of Conduct' (2002, as amended 2010). Am Psychol. 2016;71: 900–900. pmid:28032786
  31. 31. Cooper H, Hedges LV. Research Synthesis as a Scientific Process. In: Cooper H, Hedges LV, Valentine JC, editors. Handbook of Research Synthesis and Meta-Analysis.: Russell Sage Foundation; 2009. pp. 4–16.
  32. 32. Lorenzen C, Maschette W, Koh M, Wilson C. Inconsistent use of terminology in whole body vibration exercise research. J Sci Med Sport. 2009;12: 676–678. pmid:18762453
  33. 33. EQUATOR Network. 2019; http://www.equator-network.org.
  34. 34. Schulz KF, Altman DG, Moher D, the CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Trials. 2010;11.
  35. 35. Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PloS Medicine. 2009;6: e1000097. pmid:19621072
  36. 36. Moher D, Schulz KF, Simera I, Altman DG. Guidance for Developers of Health Research Reporting Guidelines. PLOS Medicine. 2010;7: e1000217. pmid:20169112
  37. 37. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6: e20476–e20476. pmid:21694759
  38. 38. Habibi A, Sarafrazi A, Izadyar S. Delphi Technique Theoretical Framework in Qualitative Research. The International Journal of Engineering And Science. 2014;3: 8–13.
  39. 39. Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41: 376–382. pmid:12581103
  40. 40. Rowe G, Wright G. Expert opinions in forecasting. role of the delphi technique. In: Armstrong J, editor. Principles of Forecasting: A Handbook of Researchers and Practitioners. Boston: Kluwer Academic Publishers; 2001. pp. 125–144.
  41. 41. Hsu C, Sandford BA. The Delphi Technique: Making Sense of Consensus. practical assessment, research & evaluation. 2007;12: 1–8.
  42. 42. Kerlinger FN, editor. Foundations of behavioral research. New York: Holt, Rinehart, and Winston, Inc; 1973.
  43. 43. Lin Y, Chen C, Lee W, Lin T, Kuo L, Lin C, et al. Development and pilot testing of an informed consent video for patients with limb trauma prior to debridement surgery using a modified Delphi technique. BMC Medical Ethics. 2017;18.
  44. 44. Pound P, Ebrahim S, Sandercock P, Bracken MB, Roberts I. Where is the evidence that animal research benefits humans? BMJ: British Medical Journal. 2004;328: 514–517. pmid:14988196
  45. 45. Qualtrics. Copyright Year 2019. Qualtrics. Provo, Utah, USA. 2005: Version: 02/2019–05/2019; https://www.qualtrics.com.
  46. 46. Rayens MK, Hahn EJ. Building consensus using the policy Delphi method. Policy, Politics, & Nursing Practice. 2000;1: 308–318.
  47. 47. Gevers DW, Kremers SP, de Vries NK, van Assema P. Clarifying concepts of food parenting practices. A Delphi study with an application to snacking behavior. Appetite. 2014;79: 51–57. pmid:24732407
  48. 48. Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. Journal of clinical epidemiology. 2011;64: 395–400. pmid:21194891
  49. 49. Sepulveda KA, Dinglas VD, Chessare CM, Friedman LA, Needham DM, Sepulveda KA, et al. Core Outcome Measures for Clinical Research in Acute Respiratory Failure Survivors. An International Modified Delphi Consensus Study. American Journal of Respiratory and Critical Care Medicine: an Official Journal of the American Thoracic Society. 2017;196: 1122–1130.
  50. 50. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31: 684–706. pmid:28190381
  51. 51. Krippendorff K. Estimating the Reliability, Systematic Error and Random Error of Interval Data. Educational and Psychological Measurement. 1970;30: 61–70.
  52. 52. Hayes AF, Krippendorff K. Answering the Call for a Standard Reliability Measure for Coding Data. Communication Methods and Measures. 2007;1: 77–89.
  53. 53. Zapf A, Castell S, Morawietz L, Karch A. Measuring inter-rater reliability for nominal data—which coefficients and confidence intervals are appropriate? BMC Medical Research Methodology. 2016;16: 93. pmid:27495131
  54. 54. Hahn EJ, Rayens MK. Consensus for tobacco policy among former state legislators using the policy Delphi method. Tob Control. 1999;8: 137. pmid:10478396
  55. 55. Molnar FJ, Man-Son-Hing M, Dalziel WB, Mitchell SL, Power BE, Byszewski AM, et al. Assessing the quality of newspaper medical advice columns for elderly readers. CMAJ: Canadian Medical Association journal = journal de l'Association medicale canadienne. 1999;161: 393–395. pmid:10478163
  56. 56. Weir CR, Hicken BL, Rappaport H, Nebeker JR. Crossing the Quality Chasm: The Role of Information Technology Departments. Am J Med Qual. 2006;21: 382–393. pmid:17077420
  57. 57. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus Methods: Characteristics and Guidelines for Use. Am J Public Health. 1984;74: 979–983. pmid:6380323
  58. 58. Mbakwe AC, Saka AA, Choi K, Lee Y. Alternative method of highway traffic safety analysis for developing countries using delphi technique and Bayesian network. Accid Anal Prev. 2016;93: 135–146. pmid:27183516
  59. 59. Brooks KW. Delphi technique: expanding applications. North Central Assocation Quarterly. 1979;53: 377–387.
  60. 60. Green KC, Armstrong JS, Graefe A. Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared. Foresight. 2007;8.
  61. 61. Sackett DL. Bias in analytic research. Journal of Chronic Diseases. 1979;32: 51–63. pmid:447779
  62. 62. Sumsion T. The Delphi Technique: An Adaptive Research Tool. British Journal of Occupational Therapy. 1998;61: 153–156.
  63. 63. Walker AM, Selfe J. The Delphi method: a useful tool for the allied health researcher. British Journal of Therapy and Rehabilitation. 1996;3: 677–681.
  64. 64. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998;2: 1–88.