Study design methodology in neurosurgical research

Abstract

Background

Medical research has evolved from individual expert-described opinions and techniques to scientifically designed methodology-based studies. There are different study designs, and selecting an appropriate study design is critical to appropriately answer the research question being investigated.

Main body

A research question may be addressed using different approaches that can be descriptive, analytical, or experimental. The choice of study design is influenced by features as that related to exposure (intervention) and disease (outcome); considerations related to time, resources, ethics and gaps in scientific knowledge that remain to be filled. The purpose of this review is to provide an overview of the basic study designs as it is the foundation of neurosurgical research to provide valid scientific evidence.

Conclusion

Quality in scientific research begins with a clear hypothesis and a well-formulated design. This entails a thorough understanding of the different study designs to choose the best suited to answer the investigated research question.

Introduction

Medical research has evolved from individual expert described opinions and techniques to scientifically designed methodology-based studies [1].

A scientific study requires formulating a research question, refining the research question (literature review), formulating research objectives, identifying the study population and setting, deciding on the study design and methodology, writing the research protocol, ethical approval, data collection, data analysis and interpretation of data analysis results. In neurosurgical research as in all other medical research drawing meaningful conclusions can only be based on data collected from a valid scientific design and using appropriate statistical methods [2]. There are different study designs and selecting an appropriate study design is critical to appropriately answer the research question and hence add to the literature valid scientific data that can help in clinical decision-making and evidence-based practice.

There are different study designs, and selecting an appropriate study design is critical to appropriately answer the research question. The following is an overview of the different study designs common in neurosurgery. Where applicable illustrative examples of published neurosurgical studies will be outlined.

Types of study design

Medical research is classified into primary and secondary studies. Primary studies are those that are actually performed by the investigators, while secondary studies summarize the results of different primary studies.

A-Primary research includes clinical or experimental studies. There are three main areas in primary research including basic medical research, clinical research, and epidemiological research. Basic medical research includes animal experiments, cell studies, anatomical studies, biochemical, genetic, physiologic investigations and studies on the properties of drugs and materials. Epidemiological studies investigate the distribution and historical changes in the frequency of diseases and the causes for these diseases. Clinical studies involve research on human subjects. However, it may be difficult to classify individual studies into one of these three main categories. A more practical way to classify the types of research studies based on their function is to group them into observational and interventional (experimental) studies. The former can be further subclassified into descriptive and analytical studies [3].

In observational studies, the investigator observes the participants in the study and gathers data by simply recording events as they happen, without playing an active part in what takes place. These studies can be either clinical or epidemiologic. In interventional (experimental) studies, the investigator tries to find a relation between an intervention and the outcome and exposes participants to some kind of intervention, such as a surgical procedure, or device or new drug, in order to evaluate it. Interventional studies can be done in basic sciences, where they are called “experimental studies;” in the community they are called “epidemiological trials,” or in clinical settings, they are called “clinical trials” [4].

B-Secondary research consolidates available studies and includes: reviews (systematic reviews or narrative review) and meta-analyses.

Figure 1 summarizes the different study designs common in the neurosurgical literature.

Fig. 1

Common clinical research designs in neurosurgery

Observational study designs

Observational studies do not conduct any experiment or intervention methods. Investigated factors are not controlled, repetition of events are not generally possible and randomization facilities are limited in these studies. However, their results are largely consistent with real life. They can be classified as descriptive or analytical.

Descriptive observational studies

Includes case reports, case series, descriptive cohort and cross-sectional studies (prevalence studies).

Case reports, case series and descriptive cohorts

Patient and disease characteristics related to some rare or remarkable finding in a patient are called a “case report.” When the number of patients are more than one, this is called a “case series.” Though controversy exists in the literature to the exact number of cases beyond which would comprise a case series. Esene et al. [5] published a statistical justification outlining that 5 is the maximum sample size for case reports. This is the most simple form of research and does not contain a control group. Case series are usually the starting points of the examined hypothesis in the case-control, cross-sectional or cohort studies [6]. A case report can be published as a brief report, adverse effects, “How I do it”, clinical note or case illustration. The “CARE” (CAse Reports) [7] and “SCARE” (Surgical CAse REport) [8] guidelines have been developed by an international group of experts to support an increase in the accuracy, transparency, completeness, and usefulness of case reports. The PROCESS (Preferred Reporting of Case Series in Surgery) guidelines [9] have been developed to improve on the transparency of reporting of case series. It is worth noting that there is a lot of confusion in the neurosurgical literature on what is defined as a case series and what in fact is a descriptive cohort. Esene et al. [10] discussed extensively this mislabeling in the neurosurgical literature. Case series are descriptive with an outcome-based sampling, while “descriptive cohorts” have an exposure-based sampling of patients (e.g., surgical intervention), followed over time to assess outcome (complication, recurrence, etc.). A comparative group is not a defining feature of a cohort study and distinguishes descriptive from analytic cohorts [11]. Descriptive cohorts are primarily descriptive compared to classical cohort studies which are analytic. They typically describe events occurring in a defined population over time rather than draw conclusions about association or causation. However, like classical cohort studies, they can be retrospective, prospective or both [12].

A hypothetical illustrative scenario is as follows:

A neurosurgery resident is formulating a study to assess the recurrence rate among patients with ruptured cerebral aneurysms treated by endovascular coil embolization in his institution. Two possible study designs are as follows:

• Retrospective descriptive cohort study if the resident retrospectively samples all patients with ruptured cerebral aneurysms treated with endovascular coil embolization (exposure) and evaluate to see if they had a recurrence requiring re-treatment (outcome).

• Case series if the resident assembles all cases of recurrent cerebral aneurysms requiring re-treatment (outcome) and then looks back to review what technique (simple coiling, stent assisted coiling, etc.) was used (exposure).

Cross-sectional (descriptive or prevalence) studies

Cross-sectional studies are study designs used to evaluate an association between an exposure and outcome at the same time. It can be classified under either descriptive or analytic, and therefore depends on the question being answered by the investigator. Since cross-sectional studies are designed to collect information at the same point in time, this provides an opportunity to measure prevalence, epidemiology or survey a disease or clinical outcome. They reflect the situation of a disease or clinical outcome at a particular moment in a particular population [6]. Cross-sectional studies should be cautiously interpreted as there is a potential risk for bias (as response/participation bias, for example the sicker patients are the more likely to be included).

An example of an analytical cross-sectional study is assessment of surgery safety checklist as “Time-out and its role in Neurosurgery” [13]. In this study, a 15-item survey was used to evaluate how time-outs were performed across 5 hospitals affiliated with a single neurosurgery training program to identify consistencies and variations. Another example of a descriptive cross-sectional study is a study on the “Anatomical variations in circle of Willis in patients undergoing CT cerebral angiography in a tertiary hospital in Nepal” [14], which provides a snapshot on the prevalence of anatomic variations in this group of patients.

Population (ecological) studies

An ecological study is an observational study defined by the level at which data are analyzed, namely at the population (for example city, county or country) or community level rather than individual level. Ecological studies are often used to measure prevalence and incidence of a disease. Often the information about the disease and exposure is abstracted from published databases and therefore does not require expensive or time-consuming data collection. An example is a published study on the “trends in the incidence of subarachnoid hemorrhage in South Korea from 2006 to 2009: an ecological study [15]. The investigators used a national data base (database managed by Health Insurance Review and Assessment Service), which contains all hospital records of every Korean citizen to answer their research question.

Analytical observational studies

Analytical observational studies are the more common category of studies encountered in the neurosurgical literature. They involve comparisons between 2 or more groups. They are based on a research question that is etiologic, diagnostic, prognostic or therapeutic [16].

They include analytical cohort studies, case control studies and analytical cross-sectional studies. Often these studies are the only practical method of studying various problems, for example studies of etiology or in instances where a randomized control trial maybe unethical, or if the condition to be studied is rare. Cohort studies are used to study incidence, causes and prognosis. As these studies measure events in chronological order they can be used to distinguish between cause and effect. Cross-sectional studies as used to determine prevalence. They are relatively quick and easy but do not permit distinction between cause and effect. Case control studies compare groups retrospectively. They seek to identify possible predictors of outcome and are useful for studying outcomes or rare diseases. They are often used to generate hypotheses that can then be studied via prospective cohort or other studies [17].

Cohort studies (analytical)

In a cohort study, the investigator defines a study population (cohort) that is selected on the basis of exposure (e.g., surgical procedure or risk factor), who are then followed up for a specified period of time to assess outcome (as surgical outcome) or disease (as recurrence) in each exposure group. They are useful for studying common diseases and rare exposures. Depending on whether the outcome of interest has occurred at the time, a cohort study can be classified as retrospective or prospective. A cohort study in which exposure and outcome have already occurred before the start of the study is termed retrospective cohort study. Conversely, studies in which the outcome has yet to occur are called prospective cohort studies. Prospective cohort studies follow patients from the time of surgery to assess outcome or recurrent disease. This can take years of follow-up and hence is more time-consuming. An ambidirectional cohort study has both retrospective and prospective phases of the study [18].

The use of cohorts is often mandatory as a randomized controlled trial may be unethical. For example, you cannot deliberately expose patients to cigarette smoke to answer research questions regarding influence of smoking on spinal fusion or aneurysm rupture. Thus, research on risk factors relies heavily on cohort studies. As cohort studies measure potential causes before outcome has occurred, hence the study can demonstrate that these “causes” preceded the outcome, thereby avoiding the debate as which is the cause, and which is the effect [17].

An illustrative observational analytical cohort study is a publication by Kotter et al. [19] comparing “surgical outcomes following laminectomy with fusion versus laminectomy alone in patients with degenerative cervical myelopathy.” This study fulfills the criteria for a prospective observational analytical cohort study: the aim of the study was to assess surgical outcome in a group of patients diagnosed with the same disease (cervical myelopathy); it is a prospective study as the outcome was not yet known when the study was initiated; there are 2 comparative surgical groups hence this is an analytical and not a descriptive cohort study.

Case-control studies

A case-control study compares a group with a known disease (or other well-defined outcome) to a control group without disease and looks back in time to identify whether the presence or absence of a disease attribute(s) differs between these 2 groups. Case-control studies are most useful in instances where prospective studies are too inefficient or costly, such as in rare diseases and diseases with a long latency period. They are cheap and efficient but susceptible to recall bias, selection bias and therefore, rely heavily on sound study designs [20]. A review of case-control studies in the surgery literature by Milhailovic et al. [21] shows that many articles reported as “case-control” are in fact not case-control studies but instead atypical or misreported case series and cohort studies. This had led some Journals to implement a checklist to standardize the reporting of research studies such as the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline that was developed by a group of biostatisticians, epidemiologists, and other methodology experts [22]. Figure 2 summarizes the key differences between case-control studies and analytical cohort studies.

Fig. 2

Key differences between case control studies and cohort studies (analytical)

An illustrative example of a case-control study is a study by Zhang et al. [23] who evaluated “geometric features associated with middle cerebral artery bifurcation aneurysm formation: a matched case-control study.” This study fulfills the criteria for a case control study: the study aimed at identifying geometric (risk) factors for middle cerebral artery bifurcation aneurysm development; the group of patients with the disease were retrospectively identified (patients with unilateral middle cerebral artery bifurcation aneurysms); comparison was made to a control group (the contralateral middle cerebral artery bifurcation was chosen to compensate for genetic and environmental risk factors).

Experimental (interventional) studies

An experimental study is an investigation where the researcher assigns or allocates the exposure (treatment) to both study (exposed) and control (non-exposed groups). They can be randomized or not and can have a control group (controlled trial), or none (uncontrolled intervention). The structure of the experimental study is similar to that of the prospective cohort study except for the fact that the investigator does not allocate the exposure in the latter. The exposed and unexposed are then followed forward to ascertain the frequency of outcomes [4].

Randomized controlled trials are the scientific gold standard for evaluating the effectiveness of novel interventions relative to the standard of care. By design patients are randomly allocated to an experimental group or control group [24]. The power of randomized control trials is largely due to the unique element of “randomization”, a technique that ideally allows 2 or more interventions to be compared based on outcomes, free from selection bias (even when influenced by unknown factors, as these are randomly distributed [25]. The advantages and disadvantages of randomized control trials as compared with observational cohort studies are outlined in Table 1.

Table 1 Advantages and disadvantages of randomized control trials and observational cohort studies

There are a number of challenges involved when conducting randomized control trials in neurosurgery including quality of binding and reporting, bias, funding, recruitment and learning curve. Takroni et al. [26] discussed these challenges in a detailed publication.

Equipoise refers to the situation in which there is no clear evidence that one intervention is superior or inferior to another, which constitutes the rationale behind conducting randomized control trials. However, some trialists fail to make the distinction between two related but completely different concepts: clinical equipoise and individual equipoise. Individual equipoise exists when the clinician involved in the research has no preference and is truly uncertain about the overall benefit or harm offered by treatment [27]. Clinical equipoise, on the other hand, is a term that relates to the collective opinion of a body of clinicians whose consensus opinion is that none of the various interventions in a clinical trial is clearly superior [28]. Individual equipoise, however, may introduce inherent bias, and as such may not equate to clinical equipoise, which is typically based on the best available literature at the time. However, when clinical equipoise is utilized as a base for designing a clinical trial, the participating neurosurgeons will have to set aside their individual biases and agree on the study design and methodology, especially the inclusion and exclusion criteria for the research study.

Randomized control trials also utilize allocation concealment (researchers do not have foreknowledge of upcoming patient-group assignments during randomization), reducing information bias as well as selection bias [25]. Studies can furthermore be single or double blinded, the former indicating that subjects or outcome assessors do not know the assignment status during the study, and the latter indicating that neither subjects nor researchers know assignment status. Although double blinding may not always be achievable (as when a surgical procedure is performed), allocation concealment and blinding mitigate bias both from the subject (which may otherwise impact adherence) and researcher (i.e., observer-expectancy effect and confirmation bias) [24].

Outcomes or end points of a trial are variable that are monitored during the study to document the impact of a given intervention or exposure on the health of a specific population. The primary outcome is the variable that is most relevant to answering the research question. Ideally it should be patient centered (i.e., outcome that matters to patients, such as quality of life and survival). Secondary outcomes are additional outcomes monitored to help interpret the results of the primary outcome. Another method of reporting outcomes is in the form of composite outcomes. These are a set of different outcomes combined into one outcome variable to show a statistical effect and increase the power of the study, instead of having one outcome [26]. A well-known multicenter, non-blinded randomized trial is the ARUBA trial (Medical management with or without interventional therapy for unruptured brain arteriovenous malformations) [29]. The trial showed that medical management alone is superior to medical management with interventional therapy for the prevention of death or stroke in patients with unruptured brain arteriovenous malformations followed up for 33 months. There was however a lot of criticism regarding the study design, progression and analysis/conclusion. Namely, the increased use of stand-alone embolization relative to microsurgery in a cohort with predominantly low-grade lesions combined with a short follow-up period amplified the treatment risk. This example shows how critical it is to design a randomized control trial to ensure its external validity (generalizability or the extent to which the study results can be applied to situations outside the study conditions).

Beyond clinical trials: approach to secondary research

Quite often, several clinical trials attempt to answer similar questions about clinical effectiveness. Since these trials are conducted on different groups and under different settings using different selection criteria, the results are not identical. However, when the results from individual studies are combined using appropriate techniques, significant benefits of treatment may show.

Types of reviews

There are 2 types of reviews: narrative and systematic, the latter is subclassified into systematic and meta-analysis.

Narrative reviews are the typical review articles of scientific journals written after a literature review on the subject with subjective interpretation of the retrieved information [30]. It is useful to provide an overview of a particular topic.

Systematic review presents an objective and balanced summary of the existing literature, based on all the relevant studies, taking into account the methodologic quality of the study design and the execution of each study reviewed. A systematic review should include a clearly defined research question; criteria for selection of the primary studies; criteria used to identify and assess the methodologic quality of the selected primary studies; and the methods used to extract and summarize the results of the primary trials on which the conclusions are based [31].

Meta-analysis is commonly defined as “as statistical analysis which combines or integrates the results of several independent clinical trials considered by the analyst to be “combinable” [32]. It is basically a review in which the results (data) from several primary trials are combined statistically to produce a single estimate of the effects of a particular intervention, with the aim of increasing the precision of the summary of results that it produces from these primary trials [33]. Meta-analysis is basically the quantitative estimate of the net benefit of multiple trials that have been included in a systematic review and hence the quality of a meta-analysis depends on the quality of the systematic review to produce the desired outcome. Among the various checklists available for assessment of the quality of systematic reviews is the QUOROM (Quality of Reporting of Meta-analysis) [34]. A well-known example is the number of trials assessing efficacy of endovascular mechanical thrombectomy after cerebral large-vessel occlusion, which prompted the following meta-analysis “Endovascular thrombectomy after large-vessel ischemic stroke: a meta-analysis of individual patient data from five randomized trials” [35], which showed significant benefit of mechanical thrombectomy in patients with large vessel anterior circulation occlusion.

Evidence level of medical studies

The evidence pyramid shows the evidence level of a scientific study in clinical practice. The evidence pyramid is illustrated in Fig. 3. According to the evidence pyramid, the “meta-analysis or systematic review” produces the most reliable evidence.

Fig. 3

Hierarchy of clinical evidence

Conclusions

Quality in scientific research begins with a clear hypothesis and a well-formulated design. This entails a thorough understanding of the different study designs to choose the best suited to answer the investigated research question.

A research question may be addressed using different approaches that can be descriptive, analytical, or experimental. The choice of study design is influenced by features of exposure (intervention) and disease (outcome); considerations related to time, resources, ethics and gaps in scientific knowledge that remain to be filled. Observational studies including case reports, case series, cross-sectional studies, case-control studies and cohort studies permit investigation of prevalence, incidence, associations, causes and outcomes. Randomized controlled trial, is a theoretical gold standard in research because of its ability to demonstrate causal inferences, is not always feasible and can limit the autonomy of a clinician in medical decision-making due to stringency of design consideration. Integrative studies usually the combination of many primary studies include systematic reviews and meta-analysis represent the most rigorous and extensive review of evidence on a specific research topic and therefore reside at the top of the evidence hierarchy.