Lecture

Systematic Overviews and Meta-Analyses in Cardiology. Evidence-Based Cardiology VIII

Alvaro Avezum

São Paulo, SP - Brazil

One of the most difficult challenges for physicians involved in clinical practice is to efficiently organize the vast amount of data concerning medical therapeutics. Most of these data come from observational research (i.e. patient medical charts), which are usually useful to standardize clinical practice and therapeutic decisions, however, due to intrinsic biases, are less helpful when used to compare reliability of a particular treatment. Randomized controlled clinical trials have thus been proposed as an effective method to compare treatment results, in an attempt to reduce biases ^1-4. Conversely, even randomized clinical trials may not be reliable enough to present specific answers, due to weak methodological design, or rather, they may be too small to detect modest differences, which can, on the other hand, be clinically important and realistically expected.

Overviews of clinical trials have been essential for physicians to successfully manage he huge amount of written or electronic clinical information, as well as to identify potential areas for scientific research. Although clinical overviews can be very useful, they display a great diversity, ranging from anedoctal reviews to systematic overviews of well conducted trials with appropriate statistic design.

An alternative method to analyze the wide range of differences among clinical reviews is to divide in reliable and unreliable, weak and robust, clinical practice modifiers and confounding factors.

Performing systematic overviews implies in a formal training in this specific field, not only regarding overview conduction, but mainly analysis of correct clinical information and understanding intrinsic and/or potential limitations.

The topic will be discussed thus in a practical way, based on firm methodological grounds which can influence clinical practice. We believe that epidemiological and methodological understanding are essential for medical formation, and are only useful when properly applied to clinical practice, influencing clinical decision-making on daily patient care.

Background - In the seventies decade, psychologists drew attention to the fact that there was a need to standardize the necessary steps to reduce systematic bias and random errors (chance) in clinical trials overviews. In 1987, Yusuf and Peto outlined the importance of systematic overviews of randomized studies to obtain clinically significant results ^5,6. It was also in 1987 that Mulrow remarked the poor quality of clinical trials ⁷. In 1988, Oxman and Guyatt published guidelines to help physicians to critically evaluate systematic overviews, concerning reliability, results and applicability ⁸.

Concept - Although the words meta-analysis and overview are indiscriminately used as similar strategies, there are critical differences between them, which allow a better understanding, and accordingly, an appropriate utilization.

Systematic overview - A quantitative and qualitative method to combine similar studies in order to improve statistical power, precision of estimates of risks and benefits, reduce biases and increase reliability and accuracy. It has to be based on a critical evaluation of the methodology used in the analyzed studies.

Meta-analyses - A quantitative review which uses statistical methods to summarize the results of several similar studies.

Non-systematic overview - A process where studies are selected without inclusion criteria, and methodological quality is not well known. The following characteristics are usually present: poor methodological quality, obvious source of bias, equivocal conclusions. It is not recommended to be used as a basis for clinical decision making.

Rationale - Clinicians, health professionals and those responsible for public health policies have been constantly faced with a vast amount of clinical literature which is frequently difficult to manage. As long as they are properly performed, systematic overviews promote an efficient summary of existing literature and provide robust data for clinical decision making.

Due to the amount of relevant literature available, it is important to note that, in order to understand a systematic overview, some premises are required: 1) a large amount of data must be summarized in "small portions which can be palatable for scientific digestion"; 2) approximately 2,000,000 medical articles are currently published in roughly 20,000 journals.

In 1992, The New England Journal of Medicine and the British Medical Journal alone comprehended 1,100 articles which consisted of about 4,400 pages.

Why should systematic overviews be thus planned? 1) To obtain data from inadequately designed studies, with poor statistical power and conflicting results; 2) to try to obtain effective estimates of treatment; 3) to allow adequate analyses of subgroups of interest; 4) to verify information about drug efficacy for regulatory purposes; 5) to provide information that will initiate larger and more reliable studies; 6) to analyze and compare similar trials; 7) to effectively summarize all available data; 8) to refine clinical information which is difficult to interpret; 9) to generate, justify and refine hypotheses; 10) to recognize and avoid limitations in previous studies; 11) to estimate sample size; 12) to disclose possible adverse effects; 13) potential actions on health policies: to create clinical guidelines and legislation concerning the use of diagnostic tests and therapeutic interventions (as long as robust, statistically adequate studies are used) ^9-11.

How to execute systematic overviews - In order to perform systematic overviews, the following steps are necessary, so that the process can be efficient and methodologically correct. 1) To generate the question; 2) to detail the protocol to be conducted; 3) to identify, selected and validate trials; 4) to collect and analyze data; 5) to report results; 6) to conclude and make inference; 7) to publish the paper.

Limitations - As any strategy in epidemiological research, systematic overviews must be analyzed not only regarding their rationale and applicability, but also the aspects related to their potential and the intrinsic limitations, Understanding such limitations allows us to obtain all necessary information from a systematic overview ¹². That is crucial when interpreting results, so that questionable extrapolations, subject to legitimacy criticisms, can be avoided. 1) Retrospective research; 2) quality of trials; 3) clinical heterogeneity (patients and treatment from distinct trials are comparable) as well as methodological heterogeneity (inappropriate definition of study objectives, inaccurate collection of data, etc) among combined studies: principle: "adding apples and oranges"; 4) inappropriate combination of studies (including inadequate statistical analysis); publishing bias (generally, results of negative clinical trials are less likely to be published than results of positive trials) and observer bias (which trials should be included?).

Methodological considerations - Two distinct approaches can be used in order to establish an adequate overview methodological quality:

a) To decrease bias and random errors: in order to obtain all the eligible data it is important to extensively analyze published and unpublished trials, use only randomized trials when reviewing therapeutic interventions, obtain complete unbiased information about all the subgroups studied, and to include trials which have used in their analyses intention-to-treat (patients are analyzed in their originally randomized groups) and complete follow-up.

b) Source of heterogeneity: combined studies may show methodological differences (design, patient selection, objectives, randomization, analyses and follow-up), known as statistical heterogeneity, or clinical differences, known as clinical heterogeneity. The amount of heterogeneity (quantification) is more important as a limitation for a systematic overview than its presence. It is important to realize that non significant heterogeneity (when not identified by statistical tests) does not imply in homogeneity or sound consistency.

c) Questions that must be elucidated in order to classify a systematic overview as relevant: 1) was the research for evidence extensive enough? 2) were there biases when selecting the articles? 3) have all the articles had their reliability tested? 4) were results of the relevant trials properly combined? 5) are the conclusions based on data analyses related to the overview?

Critical evaluation of evidence originated from systematic overviews - In a systematic overview, a few items must be evaluated to critically assess reliability, clinical importance and generalization of the results ¹³.

A) Are the results from the systemic overview (therapy) reliable? 1) is this overview based on randomized therapeutic interventions trials? 2) in the description of methods: a) is there proper search and inclusion of all relevant studies? b) is there a specific evaluation of each study reliability? 3) are the study results concordant?

B) Are the results of the overview relevant? (evaluation of number needed to treat).

C) Should qualitative differences of therapeutic effects in some subgroups of patients be taken into account? 1) do results in these subgroups present convincing clinical or biological plausibility? 2) are the differences statistically significant? 3) where the differences found hypothesized before the study was started and were them confirmed by other independent studies? 4) was this particular subgroup the only one to be analyzed during the overview?

Current aspects - A novel and distinct aspect, which is analyzing data robustness, has recently been added to systematic overview to improve methodological analysis.

Using individual data from patients taking part in the studies - Instead of using data published in the article, data used in the systematic overview is obtained from the original database (raw data). Reliability of this information is preferable to the former method.

Cumulative execution - Some authors recommend that cumulative overviews should be performed, that is, as soon as new trials are released, data is automatically incorporated to the overall summation. That helps to identify if the risk or benefit presented has become statistically significant, or, according to others, whether the evidence is definitive enough to be used for recommendations.

Prospective execution - Presently, similar trials with the same endpoints are almost simultaneously conducted. Due to that simultaneity, it is feasible to elaborate a systematic overview protocol to combine the results of these studies later. In such way, it is possible to improve comparison among the studies in view of the similarity of the questions, clinical results assessed and statistical analysis used.

Overviews using large and well powered trials - It is considered to be scientific evidence of the best quality. Randomized clinical trials with adequate statistical power, when correctly conducted and assessing relevant outcomes, are the best scientific evidence to generate therapeutic recommendations in cardiology. Systematic overviews of these studies constitute an exceptional evidence since they allow a more precise estimate of risk or benefit, with shorter confidence intervals, corroborating the estimate of precision. It is important to remark that systematic overviews of small trials with inadequate statistical power are not the best scientific evidence for therapeutic recommendations in cardiology. These overviews are indeed designed to identify promising areas for scientific research.

Optimal information size method - Originally developed by Yusuf and Pogue ¹⁴, it assesses reliability of the available evidence from systematic overviews. It is assumed that a systematic overview should have at least the same amount of information (events) as a randomized clinical trial with adequate statistical methodology. The designation "optimal information size" is characterized as the minimum amount of information required to obtain reliable conclusions regarding the efficacy of therapeutic interventions. The approach used in this model is similar to that of randomized prospective trials, which can be interrupted in advance (before the established time for analysis of results) if significant risks or benefits are observed, For example, the application of this strategy in systematic overviews of magnesium sulfate and thrombolytic therapy has shown distinct results, that is, when reviewing the use of magnesium sulfate, there was not enough reliable and robust information to establish a clear beneficial effect of the drug (few trials, few events and inadequate statistical power to identify relative risk reduction). Conversely, with respect to thrombolytic therapy, there was a larger amount of information, due a larger number of trials, a greater number of events and appropriate statistical power. Finally, this method assesses the reliability and robustness of overview, and it should be undertaken previous to early and inadvertent conclusions of preliminary benefits of therapeutic interventions ^14,15.

Final considerations

In systematic overviews, the generated question regarding patient treatment should be biological sensible and practical. Consistent clinical answers should be obtained based on the summation of all selected randomized trials, with conclusions based on the appropriate selection of the studies and adequate methodology for collecting, analyzing and correctly interpreting the data obtained. Some systematic overviews have been recognized as the best scientific evidence for a beneficial therapeutic intervention, however, others are simply used to indicate the direction of a potential benefit, and are neither robust not reliable enough to permit recommendations in cardiology. Once again, cardiologists must be prepared to critically evaluate all available literature, including systematic overviews.

There are some recommendations concerning systematic overviews: 1) all relevant clinical trials included in the overview (publishing and unpublished, not restricted to English) should be obtained through careful research to reduce publication biases; 2) all systematic overviews should follow a detailed protocol, with objectives, methods and a well designed analysis. Limitations such as the fact that it is a retrospective research can be overcome if overviews are prospectively designed before the results of the trials are reported; 3) numerous attempts should be carried out to obtain completed data for the available relevant clinical trials; 4) agreement among researchers (trialists) about common definitions frequently used in the results of trials could reduce problems associated to heterogeneity; 5) using data from individual patients can improve reliability and clinical usefulness of a systematic overview; 6) extensive systematic overviews, when well conducted, can be more useful than small reviews. Overviews of large randomized trials with adequate statistical methodology, using data from individual patients, are the best strategy to evaluate the magnitude of global therapeutic effects and effects on clinically important subgroups ^9-11.

Overall, systematic overviews are a useful strategy to evaluate and describe effects of a therapeutic intervention, however, understanding their advantages and limitations is essential for their appropriate use.

Instituto Dante Pazzanese de Cardiologia - São Paulo

Mailing adress: Alvaro Avezum – Instituto Dante Pazzanese de Cardiologia - Av. Dr. Dante Pazzanese, 500 – 04012-180 – São Paulo, SP – Brazil

Publication Dates