Introduction

Systematic reviews comparing key treatments for both Hirschsprung’s Disease and gastroschisis have failed to identify gold standard treatments for either condition1,2,3,4. This is likely to be due to a combination of reasons including the small size and retrospective nature of many of the primary studies, but also inadequate data reporting, heterogeneity in outcome definition, and use of surrogate markers of success instead of clinically relevant outcomes1. In a recent study reviewing 283 randomly selected Cochrane systematic reviews covering a range of conditions and interventions, Kirkham et al. demonstrated that over 30% of the randomised controlled trials included in these systematic reviews either did not publish at all, or did not fully publish the results for their identified primary outcomes5. This suggests that the potential for reporting bias is widespread within the medical literature.

Potential reporting bias and potential lack of patient relevance, combined with the fact that chance and confounding are likely to be impacting the results of many Hirschsprung’s Disease and gastroschisis studies6 limits the ability of the existing evidence base to help guide clinical management. It has been proposed that the development of core outcome sets for key conditions, combined with increased collaboration can help address these problems1,7,8.

Core outcome sets are groups of outcomes that have been identified through a systematic review and Delphi process, and ratified by key stakeholders as the outcomes that should at a minimum be reported in every study of that condition8. They are a tool that has been championed by the COMET (Core Outcome Measures in Effectiveness Trials) initiative as a method for reducing reporting bias, increasing patient relevance of research, and improving meta-analysis. Their use in other conditions has been shown to significantly improve the quality of research being conducted9, thereby enabling more evidence-based clinical practice.

Prior to developing a core outcome set it is essential to know which outcomes are currently reported in the published literature, and the quality of their reporting. The aims of this work were therefore to conduct two systematic reviews, in order to identify which outcomes are currently reported in studies comparing surgical interventions for Hirschsprung’s Disease and gastroschisis, determine the completeness of data reporting, and make an empirical assessment of the likelihood of reporting bias in included studies.

Methods

Two systematic reviews were conducted according to pre-specified protocols that were prospectively registered on PROSPERO (CRD42015024996 and CRD42015025026). Multiple search strategies were used to identify relevant articles from Medline and EMBASE. Search terms were identified from database thesauri and free text relating to either Hirschsprung’s Disease and operative interventions for Hirschsprung’s Disease, or gastroschisis and interventions for gastroschisis, and combined using Boolean operators. Searches were limited to papers published in or after 2010 in order to ensure that identified outcomes were contemporaneous. Full search strategies are provided in supplementary files one and two.

Study selection and data extraction

Identified titles were assessed for inclusion by three investigators acting independently (BA and AI for Hirschsprung’s Disease, and BA and NP for gastroschisis). Any conflicts were resolved by discussion, with recourse to a fourth investigator (MK) where necessary. Data from included articles were extracted independently for each review by the same two investigators, with conflicts resolved in the same manner. The following data were extracted from all studies: study design, year of study, interventions investigated, population investigated, size of study population, inclusion and exclusion criteria, outcomes reported, whether outcomes were primary or secondary, time-points at which outcomes were measured, which of the criteria for complete reporting the study met, and the ORBIT grade5 for each investigated outcome. ORBIT grades range from A-I and are used to denote the completeness of reporting for a particular outcome.

Inclusion and exclusion criteria

All study designs except case studies and expert opinion were considered for inclusion in the review. Studies where only the abstract was available (e.g. conference proceedings) were not included in the assessment of completeness of data reporting, as this could only be analysed from the full study report. The search was limited to papers published after 2010 in order to ensure that the outcomes identified by the review remained contemporaneous. Only studies including more than 10 infants were eligible for inclusion.

Additionally, Hirschsprung’s Disease studies were eligible for inclusion if they:

  1. 1

    Compared two or more interventions for infants with biopsy confirmed Hirschsprung’s Disease and

  2. 2

    Reported outcomes following the primary definitive procedure.

Additionally, Hirschsprung’s Disease studies were excluded if:

  1. 1

    They only reported outcomes from one intervention without a comparator.

  2. 2

    They only reported outcomes for infants undergoing a re-do procedure or

  3. 3

    They only reported outcomes from the non-definitive procedure e.g. stoma formation in infants planned for a staged procedure.

Additionally, gastroschisis studies were eligible for inclusion if they compared two or more methods of visceral reduction and defect closure in infants with gastroschisis, and were excluded if they only reported outcomes from one intervention without a comparator.

Both searches were undertaken in August 2015.

Outcome description

The primary aim of this study was to generate a list of all outcomes investigated by eligible studies. The median number of outcomes reported per study was calculated as a secondary outcome measure. Similar outcomes were merged to one common term prior to analysis.

Outcome terms were then assigned to one of the five core areas from the OMERACT filter 2.0. The OMERACT filter 2.0 represents five core areas that should be covered by outcomes in order to ensure a full breadth of reporting10. These areas are (1) adverse events, (2) life impact, (3) resource use, (4) pathophysiological manifestations, and (5) death.

Completeness of reporting

A secondary aim of this review was to determine the completeness of outcome reporting in Hirschsprung’s Disease and gastroschisis studies. Harman et al. proposed five core questions that could be used to assess how transparently researchers had identified and reported their choice of outcomes11, whilst the ORBIT criteria are used to determine whether it is likely that data is missing from the studies report5. The percentage of studies meeting all five of Harman et al.’s core criteria for complete, transparent outcome reporting, and the percentage of studies reporting full data, as described by the ORBIT criteria were calculated for all outcomes investigated. Harman et al.’s five core criteria are:

  1. 1

    Is the primary outcome clearly stated?

  2. 2

    Is the primary outcome clearly defined so that another researcher would be able to reproduce its measurement?

  3. 3

    Are the secondary outcomes clearly stated?

  4. 4

    Are the secondary outcomes clearly defined?

  5. 5

    Do the authors explain the use of the outcomes which they have selected?

Data Synthesis

The number of outcomes reported in each eligible study and the number of times each outcome was reported were counted, and the medians and interquartile ranges calculated. The number and percentage of studies answering yes to all five core questions proposed by Harman et al., and the number and percentage with full reporting according to ORBIT criteria for all investigated outcomes were also calculated.

Results

Hirschsprung’s Disease included studies

The search retrieved 696 unique titles related to Hirschsprung’s Disease, 640 of which were excluded following title and abstract review. Following full paper review, a further 21 papers were excluded with reasons (Supplementary File 3), leaving 35 studies as eligible for inclusion in the review (Fig. 1). Eighteen manuscripts (51%) were retrospective case series, nine (26%) were prospective cohort studies, three (9%) were case control studies, three (9%) were systematic reviews, one (3%) was a registry based study, and one (3%) was a randomised controlled trial. The median number of study participants was 62 (IQR44-156).

Figure 1
figure 1

Hirschsprung’s Disease PRISMA flow chart.

Full size image

Hirschsprung’s Disease reported outcomes

In the 35 included studies, 95 outcomes were investigated a total of 337 times. Thirty-five outcomes were considered to be too similar to at least one other outcome to be meaningfully differentiated, and these outcomes were therefore mapped to one common term (e.g. continence/incontinence, or frequency of stool/bowel movement frequency). Following this exercise, 74 unique outcomes were identified as having been reported (Table 1).

Table 1 Hirschsprung’s Disease outcomes and number of times reported, grouped according to OMERACT filter 2.0.
Full size table

Outcomes were mapped to the OMERACT filter 2.0. Overall, 33 ‘adverse event’ outcomes, 28 ‘life impact’ outcomes, seven ‘resource use’ outcomes, five ‘pathophysiological manifestation’ outcomes, and one ‘death’ outcome were reported. Adverse event outcomes accounted for 171 of the 338 outcomes that studies investigated (51%), whilst life impact outcomes accounted for 109 (32%).

Overall, faecal incontinence was the most commonly reported outcome and, appearing in 32 studies (91%), the only one to be investigated in more than 75% of studies. Outcomes investigated in more than 50% of studies were enterocolitis (23 studies, 66%), constipation (20 studies, 57%) and length of stay (18 studies, 51%). Thirty-three (45%) of the 74 unique outcomes were only investigated in one study. The median number of outcomes investigated per study was 11 (IQR6-13). Due to the retrospective nature of many of the included studies, and the lack of clearly defined end-points within them, it was not possible to make any meaningful assessment of the ages at which key outcomes were measured.

Hirschsprung’s Disease Completeness of reporting

Of the 35 included studies, seven (20%) were available only as abstracts and were therefore excluded from assessment of overall quality and completeness of reporting. Of the remaining 28 studies, only 7 (25%) met all five of the core criteria that Harman et al. describe for achieving complete, high quality reporting of a study’s results. When assessed against the ORBIT criteria, of the 28 full papers, only seven (25%) fully reported each of the outcomes they set out to investigate (Table 2).

Table 2 Number of outcomes reported and completeness of outcome reporting in Hirschsprung’s Disease studies.
Full size table

Gastroschisis included Studies

The search strategy returned 234 titles related to gastroschisis, reduced to 211 after exclusion of duplicates. Following review of titles and abstracts, 167 records were excluded. A further fourteen records were then excluded with reasons following full paper analysis (Supplementary File 4), resulting in 30 studies that were deemed eligible for inclusion in the review (Fig. 2). Twenty-two papers (73%) were retrospective case series, four were prospective cohort studies (13%), two were systematic reviews (7%), one was a case-control study (3%), and one was a registry study (3%). There were no eligible randomised controlled trials. The median number of study participants was 122 (IQR 53-285).

Figure 2
figure 2

Gastroschisis PRISMA flow chart.

Full size image

Gastroschisis outcomes

Within the included studies, 102 outcomes were investigated a total of 247 times. Within these 102 outcomes there were 63 that were felt to be too similar to at least one other outcome to be meaningfully differentiated, and these were therefore mapped to one common term. Following this mapping process, there remained 62 unique outcomes (Table 3).

Table 3 Gastroschisis outcomes and number of times reported, grouped according to OMERACT filter 2.0.
Full size table

Outcomes were mapped to the OMERACT filter 2.0. Overall, 22 ‘adverse event’ outcomes (35%), 18 ‘pathophysiological manifestation’ outcomes (29%), 12 ‘life impact’ outcomes (20%), nine ‘resource use’ outcomes (15%), and one ‘death’ outcome (1%) were reported. ‘Adverse event’ outcomes were reported 97 times (39% of all reported outcomes), whilst ‘life impact’ outcomes were reported 58 times (23% of all reported outcomes).

Eight of the 62 identified outcomes related to feeding (13%), and five related to mechanical ventilation (8%). These two areas therefore appear to be of interest to researchers. However, the greatest number of studies in which any individual outcome from either of these areas was investigated was 13 (43%), suggesting there is little agreement on what the most important outcome in each area is. Overall, the most commonly investigated outcome was total length of stay, appearing in 24 studies (80%). Mortality (19 studies, 63%), and development of necrotising enterocolitis (16 studies, 53%) were the only other outcomes to be reported in more than 50% of studies. The median number of times an outcome was reported was once (IQR1-2). Thirty-one outcomes (50%) were only reported in one study. The median number of outcomes reported per study was 9 (IQR5-11).

Gastroschisis completeness of reporting

Of the 30 included studies, three were only available as abstracts, and therefore excluded from analysis of completeness and quality of reporting. Of the remaining 27 studies, none met all five of the core criteria defined by Harman et al. for complete, high quality outcome reporting. When assessed against the ORBIT criteria, 12 studies (44%) fully reported each of the outcomes they set out to investigate (Table 4).

Table 4 Number of outcomes reported and completeness of outcome reporting in gastroschisis studies.
Full size table

Discussion

This review shows substantial heterogeneity of outcome reporting in Hirschsprung’s Disease and gastroschisis studies. For each condition, over 60 unique outcomes were investigated, only 5% of which were investigated in more than half of the included studies, and over 40% of which were only investigated in one study.

Within Hirschsprung’s Disease research, 32% of investigated outcomes were in the ‘life impact’ core area of the OMERACT filter 2.0, suggesting that there is potential for current research to be investigating outcomes of relevance to patients. The lack of meaningful information on when outcomes are measured however, meant that it was not possible to determine whether these outcomes were investigated at time points where their results would be considered valid (e.g. whether faecal continence was measured in infants who were old enough to be expected to be continent). With 64% of gastroschisis outcomes fitting into either the ‘pathophysiological manifestations’ or ‘adverse events’ core areas, it would suggest that gastroschisis research tends to have a shorter-term focus. With no gastroschisis studies, and only 25% of Hirschsprung’s Disease studies meeting all of the criteria for high quality, complete outcome reporting, and only 25% of Hirschsprung’s Disease and 44% of gastroschisis studies fully reporting all outcome data, there also appears to be a high risk of reporting bias within both research fields.

The search strategy and inclusion criteria for this study were designed to be sensitive whilst maintaining contemporaneity of the identified outcomes. This should provide the most robust basis for development of a core outcome set. Excluding studies conducted prior to 2010 does however introduce the potential for these reviews to miss important outcomes that were fully investigated in high quality studies prior to this point. One example of this is the only randomised controlled trial carried out to date which compares operative primary fascial closure with silo placement for treatment of gastroschisis12. Such exclusions have the potential to alter the data on completeness of reporting. However, we were unable to identify any plausible reasons why the study designs used, or quality of reporting should be significantly different pre and post 2010. We therefore do not believe that these exclusions will have significantly altered the conclusions of this review. By including in the current analysis outcomes investigated by systematic reviews, outcomes reported by significant primary studies prior to 2010 (including that of Pastor et al.) should also have been captured, and will therefore be included in the list of outcomes used for the development of the core outcome sets.

In order to assess the quality and completeness of outcome reporting, we used tools that were initially designed for assessing randomised controlled trials and systematic reviews. However, it is not unreasonable to expect that any study making active comparison of two interventions meets all five of Harman et al.’s core criteria for high quality, transparent reporting, and that they fully report data for all stated a priori outcomes.

It has previously been suggested that variability in outcome definition and reporting is limiting the development of a high quality evidence in Hirschsprung’s Disease and gastroschisis1,2,4. Our review reinforces that message, and adds to it by suggesting that there is also a high risk of reporting bias in research into both conditions, thereby further reducing the reliability of conclusions drawn from the current literature. This high risk of reporting bias echoes what was shown in a large selection of randomised controlled trials included in Cochrane systematic reviews5.

The focus of the included studies on pathophysiological manifestations and adverse events, and lack of clarity of time-points for measurement of life impact outcomes raises the possibility that patients and their parents were not involved in determining which outcomes should be investigated by researchers and clinicians. This apparent lack of knowledge as to which outcomes are important to patients and parents in determining the success of gastroschisis and Hirschsprung’s Disease treatment may account in part for the heterogeneity of outcome reporting that our reviews have demonstrated. The difficulties this heterogeneity creates for supporting clinical practice are compounded by the high risk of reporting bias that this review has also demonstrated. Both of these factors suggest that at present, caution should be exercised when using the existing literature to argue in favour of a particular treatment for either condition. There are two concrete steps that can be taken to remedy this situation, firstly, development of core outcome sets for use in gastroschisis and Hirschsprung’s Disease, and secondly, enhanced national and international collaborative research studies and trials. These measures would reduce reporting bias, ease meta-analysis, increase statistical power to answer clinically relevant questions, and improve the patient relevance of research7,8,9.

These systematic reviews provide the evidence that development and implementation of core outcome sets are required for Hirschsprung’s Disease and gastroschisis. The outcomes identified by these reviews could be used as the starting point for a robust Delphi processes involving patients, parents and multi-disciplinary clinical groups to develop such core outcome sets. These will be the first core outcome sets to be developed for any paediatric surgical condition, and we do not wish them to be developed simply as an academic exercise. Hirschsprung’s Disease and gastroschisis were specifically chosen as the first conditions in which to develop paediatric surgical core outcome sets in order to limit the possibility of this occurring. We believe they limit this risk for two reasons. Firstly, both conditions are actively studied at present, and therefore, have a community of researchers who are anticipating the use of the developed core outcome sets. Secondly, there have been contemporaneous, UK-wide cohorts of infants established for both conditions, in whom long-term follow-up studies could be conducted utilising the developed core outcome sets. We have elected to present the results of both systematic reviews as one manuscript, as there are several common themes which lend themselves to being demonstrated best through reporting in a single article. Doing this illustrates to the paediatric surgical community the need for development and implementation of core outcome sets in both of these conditions, and potentially, in others as well.

Additional Information

How to cite this article: Allin, B. S. R. et al. Variability of outcome reporting in Hirschsprung’s Disease and gastroschisis: a systematic review. Sci. Rep. 6, 38969; doi: 10.1038/srep38969 (2016).

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.