Introduction

Assessing the effects of treatment for a chronic disease and differentiating them from the consequences of the natural progression of the disease poses a range of methodological challenges for an observational researcher, and these challenges become even more pronounced when a feedback loop exists between disease progression and treatment progression. Studying the cardiovascular effects of glucose-lowering medications presents exactly these challenges since diabetes is a chronic disease that progresses over time with escalating cardiovascular consequences, and the medications involved in its treatment change in response to the progression of the underlying disease. Further, the very cardiovascular effects of the diabetes treatment under study may represent complications of diabetes itself so that disentangling drug from disease effects requires careful application of study design principles coupled with appropriate interpretation.

A growing body of observational literature assessing the effect of different glucose-lowering regimens on cardiovascular disease has been recently produced with inconsistent results (see Electronic supplementary material [ESM] Table 1). Increasing numbers of large databases, often based on healthcare utilisation data, have been assembled throughout the world and used to study the effect of therapeutics on health outcomes [1, 2]. These databases have attractive aspects: they reflect routine practice, which allows for the evaluation of real-world effectiveness and safety in large populations that include patients who are often under-represented in, or completely excluded from, randomised controlled trials (RCTs); are sufficiently large to allow the study of rare events or newly marketed products; and are readily available for analysis by researchers, without the time and monetary costs common to large RCTs.

However, if not rigorously conducted, studies based on observational data can encounter methodological problems that can compromise their validity and bias their results. In some instances, the medical and scientific community may be overwhelmed by the sheer numbers of people included in these studies and may overlook these methodological issues; similarly, these biases can be self serving for the custodian of the data or the sponsor of the analyses (e.g. pharmaceutical industries, private insurers, governments, etc.).

In light of these considerations, observational research can offer a valuable opportunity to evaluate the cardiovascular effects of glucose-lowering therapy, but only if adequate attention is paid to the methodological issues involved so that the risk of spurious and misleading findings is reduced.

The objectives of this review article were to assess the design and analysis methods used in studies of the association between glucose-lowering medications and cardiovascular outcomes, with a focus on specific methodological challenges; to illustrate these methodological challenges and how they might affect a study, providing specific examples from the reviewed literature; and to suggest approaches that may be employed to manage these methodological challenges.

Identification of relevant literature, data abstraction and data review

We conducted a systematic search to identify observational studies assessing the effects of glucose-lowering medications on cardiovascular outcomes. We searched PubMed for articles published from January 2000 to December 2012, combining search terms for glucose-lowering medications, cardiovascular disease and methodological issues likely to be encountered in observational research (ESM Search strategy). We restricted the search to observational studies on glucose-lowering medications with cardiovascular disease as a primary or secondary outcome. We excluded studies in a language other than English, studies not conducted in humans, and those that focused on secondary diabetes or gestational diabetes (ESM Search strategy).

The titles and abstracts of identified articles (N = 2,875) were further screened for eligibility by three team members (E. M. Garry, V. G. Gillet, E. Patorno), and in cases where the abstract was unavailable or provided insufficient detail, the full text was evaluated. Any discordance among reviewers was resolved by consensus. This step led to the inclusion of 68 articles. We supplemented this search by reviewing bibliographies of original articles, reviews and meta-analyses to identify additional articles, and selectively searching other data sources including the Cochrane Database of Systematic Reviews, Web of Science, and relevant websites of medical societies, for the period January 2000 to December 2012. This step led to the inclusion of 13 additional articles, for a total of 81 articles (Fig. 1), including those evaluating the effect of glucose-lowering medications on cardiovascular outcomes and all-cause mortality among patients with cardiovascular disease (where mortality is a proxy for a cardiovascular outcome).

Fig. 1
figure 1

Flow chart of included studies

Full size image

Data on type of study, exposure, outcome, effect on cardiovascular outcomes, and relevant methodological aspects were abstracted from the 81 selected articles and summarised in a table format (ESM Table 1) by three members of the team (E. M. Garry, A. R. Patrick, V. G. Gillet). Subsequently, articles were independently reviewed by three methodologically trained epidemiologists (E. Patorno, A. R. Patrick, J. D. Seeger) for the evaluation of the methodology and then discussed at meetings for consensus. At these meetings, results of the review were discussed and a list of methodological issues and their possible management, compiled up to that point in time, was reassessed and updated based on potential new elements arising from the ongoing critical review of the literature. Any difference in opinion was discussed among a broader set of authors (E. Patorno, J. D. Seeger, S. Schneeweiss, A. R. Patrick, E. M. Garry, V. G. Gillet) to achieve consensus.

Major methodological issues

We identified the following recurrent methodological issues among the reviewed studies: 1) handling of temporality in administrative databases; 2) handling of risks that vary with time and treatment duration; 3) unclear definitions of the exposure risk window; 4) handling of exposures that change over time; and 5) handling of confounding by indication.

Temporality considerations in administrative databases

A temporal sequence that properly places exposure relative to when covariates are assessed and when outcomes occur is crucial to the assessment of exposure effects (Fig. 2) [3]. Patient characteristics (covariates) should be assessed during the time preceding the first exposure to the drug of interest, and follow-up for outcome occurrence should start after criteria for cohort entry and exposure status have been established.

Fig. 2
figure 2

Study temporality: adapted from [3] with permission. Original copyright © 2010 John Wiley & Sons, Ltd

Full size image

Appropriate sequencing of these factors reduces opportunities for immortal time bias to become a factor in the study. Immortal time refers to a period of observation time during which the event cannot occur, i.e. person-time that is event-free by definition [4]. This happens, for example, when an exposure definition requires that everyone meeting the definition must have survived for a specified period. This event-free person-time may be incorrectly included in the denominator for the exposed group and lead to a biased reduction in the estimated incidence rate of cardiovascular events, by diluting the treated person-time with some other person-time that has no risk for study outcomes. The bias from this source can lead to a spurious finding of a protective effect of treatment. Immortal time bias can occur when exposure status depends on information that is not yet known at the time of cohort entry (i.e. it becomes known during study follow-up) (Fig. 3). In many studies, exposure status is defined on the basis of future use of a specific diabetes treatment [5–15], combination of treatments [5, 6, 16–18], or treatment dose [11, 13, 19]. This conditioning of exposure status on the basis of future information invites immortal time bias, particularly when coupled with a follow-up that begins before diabetes treatment status, i.e. exposure status, is defined. Examples from the reviewed studies used a specific calendar date [8, 9, 20] or a diagnostic definition [5, 7, 14, 18, 21–25] to identify cohort entry and follow-up initiation, but defined diabetes treatment status on the basis of future drug use ‘at any time during follow-up’ (Fig. 4) [16, 17, 21–23, 25]. Defining cohort entry by exposure to a drug of interest (particularly a new-user cohort, see next section) substantially reduces the opportunity for immortal time bias to affect a study, but does not remove it entirely. For example, if a study follows a cohort of drug initiators, but then compares monotherapy to combination therapy, where these subgroups are identified ‘at any time during follow-up’ [5, 13, 16–18, 21, 26], then immortal person-time is created by this exposure definition. Alternatively, if cohort entry is defined by more than one drug prescription/dispensing or more than a certain duration on treatment, but follow-up initiates at the first prescription/dispensing [10, 12, 17, 27–32], then immortal person-time may be a source of bias.

Fig. 3
figure 3

Biased exposure status based on future information. Immortal time bias can occur when exposure status depends on information that is not yet known at the time of cohort entry (i.e. it becomes known during study follow-up). Rx indicates the receipt of a medication

Full size image
Fig. 4
figure 4

Immortal time bias. Rx indicates the receipt of a medication. Orange line, immortal time; black cross, cardiovascular outcome

Full size image

On a similar note, future exposures and events that occur during follow-up should not be used as criteria for cohort entry. Several studies included or excluded patients on the basis of information collected during follow-up, such as use of insulin after cohort entry [6, 10, 12, 17, 33], use of glucose-lowering combinations [29, 34–36], switching to other glucose-lowering therapy [10, 12, 20, 37] or drug adherence [6, 27, 28, 38]. By conditioning cohort entry on the basis of future information these studies have the potential for bias arising from immortal time.

Appropriate sequencing of covariate assessment, cohort entry and exposure status, and start of follow-up will also reduce the potential for bias that might arise through adjustment for covariates that have been affected by exposure to treatment. These covariates are termed ‘intermediate variables’ since they are intermediate between exposure and outcome, and their value has been influenced by the drug exposure so that adjustment for the covariate removes some of the drug effect, typically resulting in a bias toward the null [39] (overadjustment bias [40], Fig. 5). However, this bias is unpredictable and can lead to larger effects in particular situations [41]. Case–control studies, where patient characteristics that are used for covariate adjustment are measured during the time leading up to the case or control date, can be particularly subject to bias from this source. Many case–control investigations on the effects of diabetes therapy and cardiovascular outcomes are potentially affected by this form of bias [34, 42–49]. However, cohort studies are not exempt from this methodological concern [5, 7, 8, 12, 14, 17, 20, 22, 25, 27, 28, 30, 37, 50–53]. Examples of variables that are likely to be intermediate between glucose-lowering therapy and cardiovascular outcome include post-treatment medical conditions such as hypertension and hyperlipidaemia [34, 43, 45, 48, 49], medication use during follow-up [5, 8, 12, 14, 17, 20, 28, 37, 45, 47, 48], and metabolic information such as BMI and HbA1c ascertained after cohort entry and treatment initiation [25, 26, 34, 49–52].

Fig. 5
figure 5

Overadjustment bias. Overadjustment bias derives from adjusting for an intermediate variable between exposure and outcome, i.e. an outcome risk factor that has been influenced by exposure to the drug of interest. In the example HbA1c is intermediate between diabetes therapy and cardiovascular outcome

Full size image

A cohort study that defines entry on the basis of the first use of a drug, i.e. an exposure-defined cohort, and has a well-defined temporal sequence of covariate assessment, exposure definition and start of follow-up, improves the assessment of drug effects and reduces the potential for immortal time bias.

Time-varying hazards and treatment duration effects

The risks associated with specific medical conditions tend to vary over time, mostly increasing with increases in age. Diabetes is associated with the development of cardiovascular, renal and neurological outcomes, and with an increased risk of cancer development [54, 55]. The occurrence of these disease-specific outcomes develops on different time scales, including both age and duration of diabetes.

Similarly, risks and benefits that result from use of medications may vary over time, sometimes decreasing and sometimes increasing the longer a medication is used. Early susceptibility or intolerance to drugs may lead to rapid occurrence of medication-related adverse effects and discontinuation of the medication, potentially resulting in a ‘survivor cohort’ of prevalent drug users that is composed of people who are less susceptible to a range of outcomes by virtue of having passed through this early high-risk period. These individuals may spuriously appear to receive more benefit from the use of a drug when compared with patients who have newly initiated its use. Further, those patients who continue to take the medication may be demonstrating adherence and this makes them more likely to do well on any therapy so that they appear to receive more benefit than new users of a drug. An existing data source contains glucose-lowering medication users at different stages of therapy, some of whom represent such a survivor cohort. A study comparing a survivor cohort of glucose-lowering medication users to new users might be expected to show that the new users experience more outcomes, including cardiovascular outcomes. Further, when a new medication is initiated in response to diabetes progression, comparing new users with prevalent drug users may lead to findings of spurious drug harm, as patients escalating therapy may have higher baseline cardiovascular risk than a survivor cohort of prevalent users with adequate glucose control.

A new-user design, which identifies cohorts of patients at the time they initiate a new drug, is particularly well suited to evaluate drug effects that vary over time [3, 56, 57]. The well-defined start of follow-up in these new-user cohorts has the effect of ‘synchronising’ patients on a same time scale that is relevant to the drug effect, making it possible to assess whether and by how much the risk of an outcome changes concomitantly with duration of use. Although a new-user design is better suited to detect or assess time-dependent drug effects, only a limited number of the reviewed investigations employed a new-user design [6, 12, 17, 19, 28, 30–32, 37, 58–65].

A new-user design should be applied to cohorts of both the medication being studied and that to which it is compared, with the design being enhanced by a comparison medication (or class of medications) that tends to be used for patients at a similar stage of diabetes, as this better distinguishes drug effects from disease effects. For example, if the study drug is typically used at more advanced diabetes stages, i.e. second- or third-line therapies, then a comparator medication that is similarly used is likely to be most appropriate. Conversely, a comparison with a more generally recognised first-line glucose-lowering medication such as metformin leads to extensive methodological challenges even when an incident user design is employed. A variant of the new-user design, which can address research involving patients characterised by more advanced diabetes stages, compares patients switching or augmenting from a first-line diabetes treatment to the study drug of interest with new switchers/augmenters to the comparison drug [66].

The occurrence of cardiovascular outcomes may also vary over time according to duration or cumulative dose of diabetes therapy. To account for these time-varying hazards, an investigation should include analyses that estimate the incidence of cardiovascular outcomes according to duration or cumulative exposure to glucose-lowering agents. For this purpose, the number of drug dispensings and the dispensed dose (e.g. in mg or IU) during the follow-up can be used to provide an estimate of the cumulative exposure. Specifically, the person-time with similar exposure categories can be pooled together and the observed outcomes can be assigned to categories of cumulative exposure defined by the accumulated dispensed dose or treatment duration, e.g. subjects with a 1 year exposure to a glucose-lowering medication will be compared with subjects with 1 year exposure with a comparator agent [67]. Subjects will be balanced with regard to all baseline covariates and follow-up will start after the accumulation of pre-specified levels of dispensed dose or treatment duration. Few studies included analyses assessing the effect of cumulative exposure to glucose-lowering agents [11, 12, 19, 37, 45, 47, 49, 61, 68].

In summary, a new-user design, which includes drug initiators, increases the chances of identifying more comparable patients with respect to the underlying risk of cardiovascular disease, and is particularly suited to detect and evaluate medication effects that vary over time. To account for the time-varying cardiovascular effect possibly associated with duration or cumulative dose of diabetes therapy, an investigation should include analyses assessing the occurrence of cardiovascular outcomes according to cumulative exposure to specific glucose-lowering agents.

Exposure risk window definition

The exposure risk window is the time period during which any harmful or beneficial effect with regard to a specific outcome can be attributed to a drug of interest. Glucose-lowering agents may contribute to cardiovascular outcomes through different biological mechanisms, which can lead to increased or decreased risk of specific comorbidities. This mechanism should be reflected in the time window chosen for outcome identification. If a plausible biological mechanism requires some minimum time before a specific cardiovascular outcome could manifest, then it is reasonable to consider a lag time between the exposure and the start of follow-up. This approach is highly dependent on background knowledge and the specific research hypothesis, so it must be cautiously applied and carefully tailored to a particular study. A limited number of reviewed studies have considered lag periods in their study design [12, 69]. The use of a lag period reduces the chances of protopathic bias (sometimes termed reverse causation), which may occur when conditions with some preclinical phase, e.g. atherosclerosis or mild stages of heart failure, influence medication selection. If a glucose-lowering medication is selectively chosen for a patient with early symptoms of underlying cardiovascular disease, then the medication may appear to cause cardiovascular outcomes because it was initiated closer to the occurrence of a cardiovascular event than would a comparator medication. Maru et al found that any pharmacological therapy for diabetes was associated with an increased risk of heart failure during the first year of treatment but not in the subsequent years, suggesting that some preclinical conditions that precipitated the therapy, and not the therapy itself, could explain this early increased risk in patients with diabetes [21].

Another aspect to consider is the appropriate duration of the exposure risk window, which may be quite relevant in the assessment of the effects of glucose-lowering medications on cardiovascular disease. It is plausible for example that, shortly before the occurrence of a cardiovascular event, patients may stop or change treatment. Thus, the exposure risk window is often extended for some time both to reflect an exposure latency period and to address the potential for treatment discontinuation or change close to an outcome (Fig. 6). Few studies assessing the effect of glucose-lowering medications on cardiovascular disease have considered latency periods after drug discontinuation [59–61, 63, 65, 70–72] or have conducted sensitivity analyses to test different exposure risk window definitions [46, 49, 73].

Fig. 6
figure 6

Informative censoring. Rx indicates the receipt of a medication. Black cross, cardiovascular outcome

Full size image

In the assessment of the clinical effects of glucose-lowering agents, it is advisable to clearly identify the biological hypothesis to test and to choose the appropriate exposure risk window in line with this, considering lag or latency periods and sensitivity analyses to address uncertainty.

Time-varying exposures

Treatment for a progressive chronic condition such as diabetes tends to exhibit high levels of discontinuation, switching or augmentation.

Diabetes treatment might change in response to advancing diabetes, or owing to adverse effects associated with specific agents. Both scenarios have the potential of leading to the observation of an increased risk of cardiovascular outcomes shortly after therapy change or discontinuation, and may cause bias. Although an as-treated (AT) analysis, which terminates exposure to a medication upon discontinuation, is often the approach of choice for the assessment of drug safety in observational studies, it may be prone to bias if the discontinuation predicts future cardiovascular outcomes (informative censoring, Fig. 6). In this setting, censoring upon discontinuation removes outcomes from their appropriate exposure category. The assessment of the temporal distribution of cardiovascular outcome occurrence, for example shortly after drug discontinuation, can assist in assessing the presence of informative censoring. Sensitivity analyses based on the AT approach with varying latency periods after drug discontinuation can also be used to identify potential informative censoring [58–61, 63, 65, 70–72]. An intention-to-treat (ITT) approach, which carries forward the initial exposure status and disregards changes in treatment status over time, is not affected by informative censoring bias in the same way. However, it might be biased through exposure misclassification that increases with longer follow-up periods and shorter time on treatment before discontinuation, and remains open to potential differential loss to follow-up [3, 74]. In most cases, such misclassification tends to reduce the effect of a medication and will produce conservative results [75, 76]. Although ITT analysis is the most commonly used approach in this literature review, it is worth considering results arising from both AT and ITT analyses in evaluating the clinical effects of glucose-lowering agents, in light of the strengths and limitations inherent in each approach [58, 60, 65, 70].

A few studies accounted for time-varying exposures by using Cox models analysing exposure to drugs as a time-dependent variable [19, 64, 71, 72, 77, 78]. These models make the assumption that treatment changes are independent of cardiovascular outcomes and may lead to biased results in the presence of patient characteristics that vary over time, affecting both diabetes treatment choice and cardiovascular risk [79], a likely scenario in the context of a chronic disease requiring therapy adjustments commensurate to its natural progression. Few studies have accounted for time-dependent confounders [21, 69]. Suitable strategies exist to address time-dependent confounding such as marginal structural models [80, 81] or g computation [82]. However, these methodologies require extensive programming and (as with any observational method) are based on the assumption that all important predictors for treatment change can be identified and are available in the data source, i.e. assumption of no unmeasured confounding.

In summary, both AT and ITT analyses should be considered in evaluating the effects of glucose-lowering agents on cardiovascular outcomes. Sensitivity analyses assessing when most cardiovascular outcomes occur can assist in identifying informative censoring. When selecting the strategy to account for time-varying exposures in the analysis, the researcher may be faced with a trade-off between methodological transparency and ability to address confounding.

Confounding

One of the principal challenges of observational studies is confounding, a form of bias that derives from an uneven distribution of baseline risks across comparison groups and results in confounded estimates of the differences between them [83]. In pharmacoepidemiological studies, confounding by indication [84], or drug channelling bias [85] is one of the most important threats to validity. Physicians prescribe drug treatments in light of the diagnostic and prognostic information available at the time of prescribing. If predictors of patient outcomes are unevenly distributed among treatment groups, then failing to control for such factors will lead to confounding [2].

Diabetes is an established risk factor for cardiovascular disease [86], and increasing diabetes duration or severity is associated with higher risk for cardiovascular morbidity and mortality. Thus, patients treated with medications used later in the course of diabetes, such as second- or third-line agents, might experience higher rates of cardiovascular outcomes compared with patients managed with diet alone or with first-line therapy users, and this can lead to confounding (Fig. 7), if not taken into account in the study design. Confounding can be reduced by the choice of a proper comparator group. Comparison groups that have been used in the recent literature include non-diabetic individuals [43, 87, 88], untreated diabetic patients [8, 33, 42, 48, 73, 89–92], diabetic patients receiving a comparator drug [5, 6, 9–13, 16–18, 20, 26–31, 34, 35, 37, 38, 44, 46, 47, 49, 51, 53, 58, 60–63, 65, 70–72, 77, 78, 93–95], and combinations of the above often in the form of any identifiable patient that did not use a specific agent of interest [7, 14, 15, 21–25, 45, 47, 50, 52, 64, 68, 69, 96–104].

Fig. 7
figure 7

Confounding by duration or severity of diabetes

Full size image

Non-diabetic individuals are not likely to be an appropriate comparator group because unmeasured lifestyle factors such as diet, exercise, socioeconomic status and BMI, as well as unmeasured severity of underlying comorbidities, are more likely to be unbalanced between patients with and without diabetes. Untreated diabetic patients are not optimal comparators either. Patients with diet-controlled diabetes will typically have less severe disease than users of glucose-lowering drugs, and thus be at lower risk of cardiovascular outcomes, or conversely diabetic individuals not receiving medications might have differential barriers to treatment and surveillance for comorbidities compared with medication users, and thus be at higher risk for cardiovascular outcomes.

Comparisons restricted to treated patients with diabetes offer improved confounding control, particularly when both treated and comparison represent new users of medications that in clinical practice are prescribed to patients with overlapping baseline characteristics [105]. Patients initiating second- or third-line therapies or combinations of two or more glucose-lowering agents might be characterised by higher diabetes duration or severity compared with a generally recognised first-line agent such as metformin or an agent used as monotherapy [5, 10, 13, 16–18, 28, 51, 72, 89, 92]. Thus, when second- or third-line diabetes therapies or specific combinations of agents represent the treatments of interest, choosing initiators of other second- or third-line agents or other combinations as comparators [20, 29–32, 35, 46, 47, 49, 58–63, 70], considering comparisons among cohorts of new switchers/augmenters (in addition to new users), and controlling for diabetes duration and underlying diabetes-related comorbidities (e.g. cardiovascular comorbidities and diabetes complications such as neuropathy, nephropathy and retinopathy), can reduce confounding.

Once the choice of a proper comparator has been made, several strategies at the study design or analysis level can be implemented to further reduce confounding, including restrictions in the study population, matching, and stratification through regression models. Many adjustment methods are constrained by the number of covariates that can be accounted for per outcome [106]. In the setting of a large number of potential confounders such as found in large healthcare utilisation databases, propensity score methodology is an effective strategy for confounding adjustment, especially when the study outcomes are relatively uncommon [107]. A propensity score is the estimated probability of receiving one treatment exposure vs another [108], and can be used to reduce confounding via matching, stratification, regression adjustment or some combination of these strategies [109]. Propensity score matching in particular offers investigators the ability to balance treatment groups across all potential confounders and to inspect the achieved balance across covariates, by comparing these variables before and after matching in a similar manner to the comparison of randomised treatment groups in an RCT [110, 111]. Demographics, clinical characteristics and healthcare history are all likely to be associated with the choice of glucose-lowering agent, and can be considered in balancing treatment groups through propensity score matching. Similarly, cardiovascular-related indications or contra-indications that might lead to differential prescribing should be taken into account in evaluating the cardiovascular risk associated with glucose-lowering agents. For example, underlying heart failure or coronary disease may be associated with decreased likelihood of receiving a prescription for metformin and thiazolidinediones, since both are contra-indicated in patients with heart failure [112, 113] and rosiglitazone has been associated with an increased risk of myocardial infarction [114, 115]. Older patients, who are at increased risk for hypoglycaemia, which may exacerbate myocardial ischaemia and cause dysrhythmias, may be less likely to receive agents that predispose to this adverse effect, such as sulfonylureas, meglitinides or insulin [112, 113]. Likewise, overweight or obese patients might be less likely to be prescribed with agents that promote weight gain, such as thiazolidinediones and sulfonylureas, in favour of agents not influencing weight or promoting weight loss, such as metformin, DPP-4 inhibitors or GLP-1 receptor agonists [112, 113].

Finally, in the context of large administrative databases with many potential confounders, the high-dimensional propensity score algorithm, an automated extension of propensity score methodology that empirically selects covariates across thousands of diagnostic, procedural and drug treatment codes, may help the researcher in identifying proxies for confounders, which are unmeasured in healthcare claims data, and ultimately address aspects of unmeasured confounding [116–118].

In summary, the choice of a proper comparator group, i.e. one that is characterised by similar healthcare utilisation, medical conditions, and diabetes duration or severity, combined with propensity score methodology, is an effective strategy to balance important baseline risk factors across treatment groups and, therefore, reduce confounding.

Other issues

As in any observational study, there are other aspects that a researcher should consider in evaluating the clinical effects of diabetes therapies.

Misclassification of exposure, outcome, and covariates

In administrative claims data, dispensed medications are recorded accurately with respect to date and quantity dispensed, and data quality is generally considered better than self-reports and physician notes [119–122]. Nonetheless, many studies used self-reported information [8, 44] and inpatient or outpatient medical records [16, 21, 23, 25, 26, 28, 31, 33, 48–50, 53, 73, 95, 97, 100] to investigate the effect of diabetes therapy on cardiovascular outcomes, which may lead to exposure misclassification. However, in claims data, chronic therapies with multiple refills, such as glucose-lowering medications, can undergo some form of misclassification depending on how the day’s supply is calculated and how long a latency period is considered after drug discontinuation. These elements, combined with an understanding of the pharmacokinetics and pharmacodynamics of the drug of interest, should be considered to limit the chances of exposure misclassification.

Misclassification of the outcome is also possible when claims data are used. In this regard, high specificity of the outcome is preferable to high sensitivity, as the relative risk estimates are unbiased when specificity approaches 100% [123]. It has been shown that claims data can identify cardiovascular endpoints with very high specificity, resulting in relatively small potential for bias associated with misclassification of the outcome [124–128]. Among the evaluated literature, few explorative analyses assessed the potential effect of outcome misclassification on the study estimates [68, 70].

Finally, misclassification of confounding variables can lead to incomplete control of these variables and ultimately residual confounding. Under this scenario, prioritising sensitivity in covariate definition may minimise the chances of confounder misclassification.

Special issues in case–control studies

Several observational studies have used a case–control design to assess the effects of diabetes therapy on cardiovascular outcomes [33, 34, 42–49, 73, 98, 100, 102, 104]. The time-varying nature of diabetes therapy and the long duration of follow-up required to assess cardiovascular risk are frequent arguments for the choice of this study approach [48, 49, 98]. However, in the context of administrative data, a few considerations need to be made regarding the choice of a case–control vs a cohort study design: (1) case–control studies are always nested in an underlying cohort of diabetic patients or glucose-lowering therapy users, which is always identifiable in claims data, and cannot estimate absolute incidence rates and rate differences unless the sampling fractions of cases and controls are known, i.e. the underlying cohort needs to be enumerated [3]; (2) case–control studies can be more prone to specific issues than cohort studies: as previously mentioned, the correct chronology of confounder assessment can be more challenging with a case–control design; (3) case–control studies tend to focus on the period immediately prior to an event. This period might not always be the most relevant for cardiovascular diseases and could be particularly prone to exposure misclassification if, as previously mentioned, patients are at increased risk of treatment change or discontinuation in proximity to an event, and no latency period is considered.

Thus, unless additional information such as laboratory tests, diagnostic results, or survey data are collected at additional cost and integrated into a healthcare claims dataset, case–control studies offer no advantage relative to cohort studies for the evaluation of the association between diabetes therapy and cardiovascular risk in claims data.

Identified major methodological challenges, suggested strategies to reduce bias, and specific rationale

Methodological challenge

Strategy to reduce bias

Specific rationale

Temporality considerations in administrative databases

• Use a cohort study with entry defined by the first use of the drug(s) of interest (exposure-defined cohort) and that has a well-defined temporal sequence of covariate assessment, exposure definition and start of follow-up

• To facilitate the understanding of the effects of glucose-lowering medications and reduce the potential for immortal time bias and overadjustment by intermediate variables

Time-varying hazards and treatment duration effects

• Employ a new-user design based on drug initiators

• Improves comparability of treatment groups with respect to the underlying risk of outcomes

• Useful for medication effects that vary over time

• Consider analyses that estimate cardiovascular effects according to duration or cumulative exposure to specific glucose-lowering agents

• To account for the time-varying hazards associated with duration or cumulative dose of diabetes therapy

Exposure risk window definition

• Clearly identify the biological hypothesis

• To choose an appropriate exposure risk window

• Apply sensitivity analyses that vary the exposure risk window definition

• To address uncertainty regarding the optimal time from exposure to start of follow-up (lag time) or continuation of effects after drug discontinuation (latency period)

Time-varying exposures

• Consider both AT and ITT analyses and vary assumptions regarding duration of effect

• To address more fully potential informative censoring and exposure misclassification

• Recognise trade-off between methodological transparency and ability to address confounding

• Improve selection of methods to account for time-varying exposures

Confounding

• Choose an appropriate comparator group (i.e. characterised by similar healthcare utilisation, medical conditions, and diabetes duration or severity)

• Apply appropriate methods to account for confounding, such as propensity score methods

• To reduce potential confounding by indication

Conclusions

The review of recent observational studies evaluating the association between glucose-lowering medications and cardiovascular outcomes illustrates several methodological issues that appear to be incompletely addressed. Future studies can clarify the effects of diabetes treatment by appropriately addressing recurrent methodological challenges such as temporality in healthcare databases, time-varying hazards and treatment duration effects, exposure risk window definition, time-varying exposure, and confounding (Text boxes: Identified major methodological challenges, suggested strategies to reduce bias, and specific rationale and Glossary). We provided some suggestions on strategies to minimise the impact of these methodological issues, which can be summarised as follows: (1) Use a cohort study with entry defined by the first use of the drug(s) of interest and that has a well-defined temporal sequence of covariate assessment, exposure definition and start of follow-up, to facilitate understanding of the effects of glucose-lowering medications and reduce the chances of immortal time bias. (2) Employ a new-user design based on drug initiators, which increases the chances of identifying more comparable treatment groups with respect to the underlying cardiovascular risk, and is particularly suited to detect and evaluate medication effects that vary over time; consider analyses that estimate cardiovascular effects according to cumulative exposure to specific agents. (3) In the assessment of the association between glucose-lowering agents and cardiovascular outcomes, clearly identify the biological hypothesis to test and choose the appropriate exposure risk window. Use sensitivity analyses to identify the optimal lag time before start of follow-up, and latency period after drug discontinuation. (4) Consider both AT and ITT analyses and use sensitivity analyses to assess chances of informative censoring. Remain cognisant of the implications and trade-offs in selecting the strategy for dealing with time-varying exposures. (5) Choose an appropriate comparator group (i.e. characterised by similar healthcare utilisation, medical conditions and diabetes duration or severity) and use appropriate methods to account for confounding, such as propensity score methods.

In spite of the methodological issues identified in this review, and illustrated in this text, improper practices with regard to study design and analysis persist in observational studies that attempt to assess the effect of glucose-lowering medications on other clinical outcomes as well [129–132]. Greater attention to the principles described in this paper would serve to address certain research practices that are particularly susceptible to spurious findings and may mislead the medical community. Careful consideration by the researcher of these specific issues and of the potential strategies to address them is warranted.