FormalPara Key Points

These two case studies demonstrate that the selected comparator groups may have significant impacts on observational study findings.

Due to potential channeling bias resulting from contraindication language in the product label, phosphodiesterase-5 inhibitor users are not the appropriate comparator group when studying the association of testosterone replacement therapy and myocardial infarction risk, and triptan users cannot be appropriately compared with any group undergoing acute anti-migraine prescription treatment when evaluating adverse cardiovascular outcomes in migraine patients.

The appropriateness of study design must be considered in comparative drug safety studies. Statistical methods can mitigate but do not necessarily eliminate confounding bias caused by flawed study design.

1 Introduction

When evaluating the presence or absence of an association between a drug therapy and the risk of specific adverse events, comparator choice can directly affect the validity of observational study results, which in turn can impact clinical interpretations, regulatory decision making, and patient access to medications. This is problematic when there is population heterogeneity and there is concern with unmeasured confounding (i.e., an inability to adjust for differences). For example, several studies investigated the association between testosterone replacement therapy (TRT) and the risk of acute myocardial infarction (AMI) and reported conflicting results [1]. Two observational studies found a statistically significant association between TRT and cardiovascular (CV) harm [2, 3]; two reported TRT had a statistically significant mortality benefit [4, 5]; and one was considered inconclusive but found that TRT offered a protective effect against CV outcomes in patients in the highest quartile of myocardial infarction (MI) risk [6]. In September 2014, a review of these studies led the United States Food and Drug Administration to issue a randomized clinical trial request to determine whether TRT causes cardiovascular harm [1].

This study explores the potential impact of comparator selection bias on the findings of two real-world case studies. Both case studies collected and analyzed retrospective data from an electronic medical claims database (Truven Health Analytics MarketScan®) and utilized multiple comparator cohorts to demonstrate the impact of study design. The first case study investigated the association between TRT and the risk of AMI in a population of adult men treated for hypogonadism. This case study demonstrates how it is possible to arrive at different results by comparing men treated with TRT and a group of untreated men versus comparison with a group of men treated with phosphodiesterase-5 inhibitors (PDE5is), which must be used with caution in men with preexisting heart problems [7,8,9]. These findings demonstrate the impact of differences in comparator choice on information about the association between TRT and risk of AMI.

The second case study evaluated the risk of AMI and stroke among treated and untreated adult patients with migraine and non-migraine controls. Although analgesics containing opioids or barbiturates are not generally recommended for routine acute treatment of migraine due to the associated risks of addiction, they are widely prescribed to migraine patients [10]. Approved acute treatments for migraine include acetaminophen, nonsteroidal anti-inflammatory drugs (NSAIDs), and triptans. However, NSAIDs have been associated with adverse events including heart attack or stroke [11], and triptans are contraindicated in patients with cardiovascular disease (CVD), ischemic bowel disease, or other cardiovascular conditions [12,13,14,15,16,17,18].

As a result of this contraindication, migraine patients treated with triptan in real-world settings may have a lower baseline CVD risk compared with migraine patients who are not exposed to triptans [19]. Studies on the association between acute treatments for migraine and ischemic events must consider the CV risk associated with migraine occurrence and severity, as well as the extent to which migraine and migraine severity affect the choice/recommendation of migraine treatments. To this end, data from the MarketScan databases were used to conduct a real-world, retrospective, comparative safety study of triptans versus NSAIDs. The purpose of the present report is to highlight the importance of channeling bias, a type of selection bias, as illustrated by these two representative studies.

2 Methods

2.1 Study Design

Both studies utilized a similar retrospective observational cohort new-user study design. All patients were enrolled continuously in a health plan (defined as no enrollment gap > 31 consecutive days) for ≥ 1 year before the index date. The baseline period was defined as 365 days prior to the index date.

The TRT (2004–2013) and triptan cohort (2004–2016) case studies used data from the United States (US)-based MarketScan database, including individual-level healthcare claims information from health plans and Medicare Part D Supplemental/Medicaid programs. Exposure was defined by prescription and the outcome of interest was defined as the first occurrence of AMI, defined by an inpatient International Classification of Diseases, 9th edition (ICD-9) diagnosis code (ICD-9-CM codes 410.X0, 410.X1) or record of death within 24 hours of an emergency department visit for ischemic heart disease (ICD-9-CM codes 410.X0, 410.X1, 411.1, 411.8X, and 413.X).

All analyses were performed using SAS® version 9.2 (Copyright © 2002–2008 by SAS Institute Inc., Cary, NC, USA) or SAS® Enterprise Guide version 7.1 (Copyright © 2017 by SAS Institute Inc., Cary, NC, USA). Due to the lack of ideal comparator cohorts, both studies used multiple cohorts including an active comparator and an inactive comparator cohort (two comparator cohorts in the TRT study, and three in the triptan study). Propensity score matching was performed using a 1:1 greedy matching algorithm [20]. Both case studies collected baseline patient characteristics (i.e., demographics, hospital utilization, CV risk factors, comorbidities, and concomitant medications) during the baseline period and compared across cohorts before and after propensity matching to assess the presence of channeling bias. These covariates were selected a priori based on their availability in the database and the plausibility that they had an association with an exposure status or risk for CV outcomes (Online Resource 1 and 2, see electronic supplementary material [ESM]).

2.2 Testosterone Replacement Therapy (TRT) Case Study

2.2.1 Study Population

All subjects eligible for study inclusion were males aged ≥ 18 years with one or more of the following: one or more new prescription for TRT after the baseline period, one or more new prescription for PDE5i after the baseline period, or a diagnosis of hypogonadism (ICD 9 codes 257.2 [other testicular hypofunction]; 257.8 [other testicular dysfunction]; 257.9 [unspecified testicular dysfunction]; 758.7 [Klinefelter’s syndrome]). Patients were excluded if they were female or of dual gender, received their first prescriptions of TRT and a PDE5i concomitantly (± 3 days), or had a diagnosis of pulmonary arterial hypertension. Patients with a history of MI were not excluded from the study.

The index date was identified as the first dispensing of TRT or a PDE5i for treated patients, and, for untreated patients, the date of the first hypogonadism diagnosis or a randomly chosen physician visit within the subsequent cohort accrual block during which they are matched for non-initiators without a match during their initial accrual block.

2.2.2 Study Cohorts

Study cohorts and matching were as follows:

  • The active comparator (PDE5i-treated) cohort included patients who received one or more prescription for PDE5i and were propensity score-matched to a new TRT patient.

  • The inactive comparator (untreated) cohort included patients who had a diagnostic code for hypogonadism and did not receive any testosterone prescriptions.

  • Patients included in the primary TRT-treated cohort received one or more prescription for a testosterone product and were matched to an untreated patient.

  • In the secondary comparison, patients in the TRT-treated cohort were matched to PDE5i-treated patients who were censored at the time a PDE5i prescription was initiated during the follow-up period. To be differentiated from the cohort included in the primary comparison (i.e., TRT-treated patients vs untreated), this TRT-treated cohort was then termed the secondary TRT-treated cohort.

2.2.3 Statistical Analysis

Two comparator and two control cohorts were defined using calendar time-specific propensity score (CTPS) matching [21]. Thus, patients were matched on the propensity score (including all measured baseline covariates) within discrete 6-month periods of calendar time. The baseline of the general population at the initial 6-month block was assessed by randomly choosing a physician visit during that block and using that baseline in the propensity score calculation. If, after propensity score matching, this patient from the general population could not be matched to any of the cohorts, a second random physician visit from a subsequent 6-month block was chosen and propensity score estimation and matching was repeated until the patient from the general population was matched to a patient in one of the other cohorts. If any baseline covariates remained imbalanced after CTPS, they were to be included in the multivariate regression model.

Baseline characteristics before and after CTPS matching were presented using descriptive statistics [22]. Balance was assessed with absolute standardized differences (absolute standardized difference ≤ 0.1 indicated a good balance) [22]. Incidence rates of AMI were calculated per 1000 person-years (PY) with 95% confidence intervals (CIs) [22]. A Cox proportional hazard model was used to assess the association between AMI and TRT use [22]. The assumption of proportional hazard was checked and no severe violation was observed; baseline covariates that remained imbalanced after CTPS were adjusted in the multivariate regression model [22]. Subgroup analyses also were performed: (i) prior CVD and (ii) age (18–65 vs > 65 years) [22].

2.3 Triptan Cohort Study

2.3.1 Study Population

All subjects eligible for study inclusion were adult patients aged ≥ 18 years during the study period. Patients with migraine had at least one inpatient or outpatient diagnosis of migraine (ICD-9 code 346.XX; ICD-10 code G43.XX) or one prescription of a triptan during the study period. A random sample of patients without migraine diagnosis was also included. Subjects were excluded if they had been diagnosed with cluster headache (ICD-9 codes 339.00–339.02; ICD-10 codes G44.00X–G44.02X) before the index date. Regardless of migraine diagnosis, patients were excluded if they had been exposed to triptans, NSAIDs, or opiate medications during the baseline period. Patients who were prescribed medication for other acute/chronic pain indications (patients with osteoarthritis [ICD-9 code 715.XX, ICD-10 code M15-19.XX], rheumatoid arthritis [ICD-9 code 714.XX, ICD-10 code M05-06.XX], chronic lower back pain [ICD-9 code 720–724.XX, ICD-10 code M54.XX], other pain due to injury or trauma [ICD-9 code 338.XX, ICD-10 code G89.XX]), and patients hospitalized for > 3 days within 30 days before the date they were prescribed medication for acute treatment of migraine were also excluded. Patients with a history of MI were not excluded from the study.

2.3.2 Study Cohorts

The case study chose three comparator cohorts (i.e., the prescription NSAID cohort, opiate cohort, and non-prescription treated [untreated] cohort) to compare with the triptan cohort. In addition, a positive control analysis was conducted to compare the untreated migraine cohort with the non-migraine general patient cohort. Patient cohorts were defined as follows:

  • Triptan cohort: migraine patients who received one or more triptan prescription (note migraine is the only indication for drugs of the triptan class; therefore, a diagnosis of migraine was not required for the triptan cohort).

  • Prescription NSAID cohort: migraine patients who received one or more NSAID prescription. Diagnosis of migraine was required within 30 days before the first NSAID prescription. Exposure to over-the-counter NSAIDs was not measured in the database.

  • Opiate cohort: migraine patients who received one or more opiate prescription to treat migraine attacks. Diagnosis of migraine was required within 30 days before the first opiate prescription.

  • Untreated cohort: patients who had a migraine diagnosis but lacked any prior prescription for any acute treatment for migraine. Untreated migraine patients who initiated a new acute treatment for migraine at a later time were eligible for the untreated cohort until the new treatment was initiated. Given that over-the-counter NSAIDs were not recorded in the database, it is likely that some patients in the non-treated cohort used over-the-counter NSAIDs to treat migraine.

  • Non-migraine general patient cohort: patients who did not have a diagnosis of migraine and did not receive a prescription for any acute treatment for migraine. Patients who later received a diagnosis of migraine or any prescription for acute treatment for migraine were eligible for the non-migraine general patient cohort until the diagnosis and/or prescription occurred. Non-migraine patients were age- and sex-matched to untreated migraine patients at a 1:4 ratio. The index date was defined as a randomly selected clinical visit within a 3-month window when a match was found.

2.3.3 Statistical Analysis

The primary analysis compared AMI/ischemic stroke risk between propensity score-matched triptan-treated patients and opiate-treated patients, using a Cox proportional hazard model. The secondary analysis evaluated AMI/ischemic stroke risk between triptan-treated patients and NSAID-treated and non-prescription-treated migraine patients versus general patients.

The exposure window for episodic migraine medication was defined using an ‘as-treated’ approach in which each prescription fill began a new exposure interval. For the index fill of the prescription, the exposure window was the estimated average duration between the index prescription and the first refill among the respective treated cohorts (e.g., triptans, NSAIDs, opiates). For the first prescription refill, the exposure window was the observed duration between the index fill and the first refill. From the second prescription refill onward, the exposure window was the average of the last two observed prior durations. Hence, exposure windows for prescription refills were specific to each individual patient. The exposure cycles were repeated until no refill was observed. A washout period was added at the end of each exposure interval (e.g., 30 days).

Traditional propensity score analysis and Cox proportional hazard regression were used to estimate adjusted hazard ratios (aHRs) with two-sided 95% confidence intervals (CIs). To minimize confounding bias, propensity score methods were used to ensure cohorts were comparable with respect to measured confounders. The balance for all covariates before and after propensity scoring was assessed through the analysis of standardized differences (standardized differences > 0.1 indicated a meaningful imbalance between comparison cohorts). Active cohorts were compared using 1:1 propensity score matching. The following patient groups were compared: triptan-treated versus opioid-treated, triptan-treated versus prescription NSAID-treated, triptan-treated versus untreated migraine, and untreated migraine versus non-migraine. Cox proportional hazards analyses were adjusted for age, sex, and other comorbid conditions that remained unbalanced after propensity score matching. Multivariable Cox regression analysis was used to compare MI risk in untreated migraine patients versus non-migraine patients. Sensitivity analyses were performed on the intent-to-treat (ITT) data and after 30-, 60-, and 90-day washout windows.

3 Results

3.1 Case Study: TRT and Myocardial Infarction (MI) Risk in Adult Men Treated for Hypogonadism

3.1.1 Baseline and Patient Demographics

After CTPS matching, each of the TRT-treated versus PDE5i-treated cohorts included 198,528 men, and each of the TRT-treated versus untreated cohorts included 207,196 men (Table 1). Baseline demographic and clinical characteristics, AMI risk factors, and CVD history for the TRT- and PDE5i-matched cohorts are summarized in Table 1. Before CTPS matching, the PDE5i cohort had a lower prevalence of all CV risk factors and had a lower prevalence of prior CVD, all of which contributed to a lower baseline risk of CVD regardless of prescribed use (Table 1). After CTPS matching, both cohorts were comparable with respect to risk factors for AMI and prior CVD. The most commonly reported prior CVD events were unspecified ischemic heart disease and other heart disease; prevalence of prior AMI was ≤ 0.7%. Baseline characteristics for TRT and untreated cohorts were published previously [22] and are summarized in Table 1.

Table 1 Baseline characteristics: Case study vs Li et al. [22]
Full size table

3.1.2 Outcome Analysis

The crude incidence rates of MI among TRT-treated and PDE5i-treated patients were 4.63 per 1000 PY (95% CI 4.30–4.96) versus 3.93 per 1000 PY (95% CI 3.62–4.23), respectively. These rates increased significantly among patients > 65 years old and among those with prior CVD conditions (Online Resource 3, see ESM). The Cox regression model revealed no significant association between TRT use and AMI when comparing TRT-treated (i.e., all administration routes) versus PDE5i-treated patients overall (aHR 0.99; 95% CI 0.93–1.06) (Table 2). An increased risk of AMI observed among TRT-treated patients aged > 65 years versus PDE5i-treated patients aged > 65 years was not significant (aHR 1.12; 95% CI 0.99–1.26); however, a significantly increased risk of AMI was observed among TRT-treated patients with prior CVD versus PDE5i-treated patients with prior CVD (aHR 1.13; 95% CI 1.03–1.25) (Table 2). Results of the Cox regression model comparing TRT-treated versus untreated hypogonadal men were published previously and are included for comparison; the model revealed no significant association between TRT use and AMI when comparing TRT-treated with untreated cohorts overall or when stratified by age or prior CVD (Table 2) [22].

Table 2 Treatment differences in the risk of acute MI: Cox regression model based on time-to-first-event analysis (TRT case study)
Full size table

3.2 Cohort Study: Triptans and MI/Ischemic Stroke Risk in Adults with Migraine

3.2.1 Baseline and Patient Demographics

The selection of the study population and reasons for exclusion are summarized in Fig. 1. Patients were categorized into five cohorts according to migraine treatment initiated: triptans (N = 436,642), prescription NSAIDs (N = 334,152), opiates (N = 55,234), untreated migraine (N = 1,168,212), and general non-migraine patients (N = 11,735,009).

Fig. 1
figure 1

Disposition of patients identified from the Truven Health Analytics MarketScan® research database between January 2005 and December 2016. N number of patients; NSAIDs nonsteroidal anti-inflammatory drugs

Full size image

Baseline demographic data for each of the cohorts are presented in Table 3. Detailed information about the baseline characteristics of the study population have been published elsewhere [19]. Overall, patients initiating triptan treatment were younger than untreated migraine patients and patients initiating prescription NSAIDs or opiates. Patients initiating triptans also had fewer comorbidities, CV risk factors, and hospitalizations and emergency room visits compared with the prescription NSAID, opiate, or untreated cohorts. They also had fewer prescriptions for several concomitant medications at baseline than patients treated with prescription NSAIDs or opiates [19].

Table 3 Baseline demographics and patient characteristics of inpatients identified from the Truven Health Analytics MarketScan® research database between January 2005 and December 2016a
Full size table

3.2.2 Risk of AMI in Migraine Patients by Treatment Cohort

Cox proportional hazards analyses for AMI showed patients with migraine who were prescribed triptans had significantly lower risk of MI than patients with untreated migraine using 30-, 60-, and 90-day windows or ITT methods (Table 4). However, the risk of MI in patients with migraine who were ≥ 65 years old and were treated with triptans was not significantly different than the risk in similarly aged patients with untreated migraine. Additionally, there was not a significant difference in MI risk between patients with migraine and a history of ischemic CVD who were prescribed triptans versus patients with untreated migraine.

Table 4 Hazard ratios of acute MI risk in migraine patients by type of acute treatment
Full size table

Cox proportional hazards analyses of the risk of MI in migraine patients prescribed triptans compared with migraine patients prescribed NSAIDs had mixed results and, as expected, these estimates pointed towards a protective effect of triptans on the risk of MI when compared with patients who were prescribed NSAIDs. The point estimate ranged from 0.33 to 0.84 depending on the chosen study window (Table 4). However, there was no significant difference in MI risk in patients prescribed triptans compared with patients prescribed NSAIDs or opiates when looking at only patients ≥ 65 years old or patients with a history of ischemic CVD.

3.2.3 Positive Control Analysis: Ischemic Stroke Risk in Migraine Patients by Treatment Cohort

Results of the Cox proportional hazards analyses for ischemic stroke are also presented in Table 5. The positive control analysis showed patients with untreated migraine had a significantly higher ischemic stroke risk than non-migraine patients (HR 1.87; 95% CI 1.77–1.97; p < 0.0001). Risk of ischemic stroke remained significantly higher in patients with untreated migraine than in non-migraine patients when analyses were restricted to patients ≥ 65 years old (HR 1.49; 95% CI 1.36–1.63; p < 0.0001) or patients with a history of CVD (HR 1.60; 95% CI 1.44–1.77; p < 0.0001). The increased risk of ischemic stroke in patients with untreated migraine remained when ITT sensitivity analyses were performed on the overall patient population (HR 1.65; 95% CI 1.60–1.72; p < 0.0001), patients ≥ 65 years old (HR 1.44; 95% CI 1.35–1.54; p < 0.0001), or patients with a history of CVD (HR 1.48; 95% CI 1.37–1.61; p < 0.0001). Hazard ratios were unchanged when analyses were restricted to patients with no contraceptive use, indicating that ischemic stroke risk was not impacted by contraceptive use (Table 5). In contrast, such positive association was not observed regarding MI risk when patients with untreated migraine were compared with the non-migraine patient population (HR 0.90; 95% CI 0.83–0.97; p = 0.0044) (Table 5).

Table 5 Hazard ratios for acute MI risk and ischemic stroke risk in untreated migraine patients versus non-migraine patients
Full size table

4 Discussion

These case studies demonstrate that comparator group selection can influence observational study findings. Because precautions related to the use of these medications in patients with CV diseases or risk factors have likely resulted in a channeling bias, PDE5i users are not an appropriate comparator group when studying the association of TRT and MI risk, and triptan users are not an appropriate comparator group when evaluating adverse CV outcomes in migraine patients. The appropriateness of study design must always be considered in drug safety studies, and although statistical methods may alleviate some bias caused by flawed study design, they cannot be depended upon to eliminate the effects of confounding bias, especially when label information may be driving selective prescribing (e.g., patients with high CV risk or elderly high-risk patients channeled away from PDE5is or triptans due to labeling restrictions).

Comparison of TRT-treated versus untreated men in our previous publication revealed no association between TRT use and AMI in patients > 65 years old or for patients with prior CVD (Table 2) [22]. The contrasting findings in the present TRT case study demonstrate the imposition of potential channeling bias as a result of using PDE5i-treated patients as a comparator group; although results using Cox regression analysis showed no difference in MI risk between TRT-treated and PDE5i-treated patients, a significantly increased risk of MI was erroneously observed among TRT-treated patients > 65 years and TRT-treated patients with baseline CVD likely due to channeling bias. In order to address confounding by severity, a treated active comparator with the same underlying disease is better than an untreated or inactive comparator. It is also important to consider a drug/drug class at the same stage of disease severity and with the same or similar contraindications. These are all important considerations when designing a real-world evidence study.

The TRT case study reinforces methodological concerns; for example, that channeling bias may have existed in previously published research. Selecting PDE5i users as a comparator group likely introduced channeling bias into the study by Finkle et al. [2], which found TRT use to be associated with an overall two-fold increased risk of MI compared with PDE5i use among TRT-treated elderly patients and reported an approximately three-fold increase in incidence rate among TRT-treated younger men with pre-existing CV conditions [2]. Prescribers are aware that PDE5is are contraindicated in patients taking organic nitrates and should not be prescribed to men for whom sexual activity is inadvisable due to their CV status [7,8,9]; similar information was not provided to prescribers when considering treatment with TRT. Considering results of a meta-analysis suggesting low endogenous testosterone increased CV risk and risk of CVD death [23], it is reasonably possible that patients with worse CV conditions were treated with TRT. Because TRT has been associated with improved sexual function among men with erectile dysfunction and low testosterone [24, 25], it is also possible that healthcare practitioners prescribe TRT to hypogonadal patients with high CVD risk and erectile dysfunction symptoms and prescribe PDE5is to patients without known heart conditions. This was evident in the TRT case study, wherein baseline characteristics of pre-matched TRT-treated patients suggested worse health (i.e., significantly increased hyperlipidemia) compared with PDE5i-treated patients. Differences in study designs, patient populations, comparator groups, availability of laboratory tests, and statistical analyses may explain the inconsistencies in study findings.

Migraine has been associated with increased CV risk including increased risk of ischemic stroke, MI, angina or coronary revascularization, and CV mortality [26,27,28]. This retrospective cohort study of acute treatment for migraine revealed a lack of association between triptan use and CV risk. Although multiple comparators were chosen, potential channeling bias resulting from contraindications stated in the triptan product labels may have been strong enough to select the heathiest patients from among all migraine patients on acute medications. These findings are in line with those of a previous analysis in which we used a prediction model to assess baseline CV risk among migraine patients prescribed triptans in comparison with those prescribed opiates or NSAIDs, as well as untreated migraine patients and general non-migraine patients [19]. In that analysis, patients who were prescribed triptans had lower CV risk than those prescribed opiates or NSAIDs (in fact, they were the healthiest among all treatment groups) [19]. We conclude it is likely that patients with migraine who also have high CV risk are being prescribed triptans at lower rates than they are prescribed opiate or NSAID alternatives.

Similar to the findings of the TRT case study regarding patients with low testosterone, the findings of the retrospective triptan cohort study support the hypothesis that channeling bias may occur in choosing whether or not to prescribe triptans for patients with migraine. Given the CV warnings included in current triptan labeling, the findings of so-called protective effects of triptans for MI and stroke compared with alternative treatments suggest channeling of migraine patients at high CV risk away from triptans in favor of alternative therapies.

Each of these observational studies is susceptible to the potential for confounding bias. In real clinical practice, prescriptions are not administered at random, and although the potential for confounding bias can be adjusted through careful selection or using statistical methods, bias might remain because no method can alleviate all potential for selection bias in a study. For example, as a risk factor for many comorbidities, diabetes severity can cause residual confounding or selection bias in real-world comparative studies of anti-diabetes medication. In a cohort study evaluating non-fatal major adverse cardiovascular events (MACE), adjustment of baseline characteristics did not remove baseline differences between metformin and insulin, as illustrated by increased HR for pre-exposure MACE (HR 1.85) [29]. Similarly, in the TRT/hypogonadism case study, confounding by indication/severity could have been introduced if the PDE5i or untreated cohort had higher serum testosterone levels versus the treated cohort, because lower testosterone levels have been linked to higher rates of CV events [30,31,32] and those getting treatment likely have lower levels than untreated patients. In the triptan/migraine cohort study, multiple active comparators, new user design, propensity score techniques, and sensitivity analysis were used in an effort to minimize confounding bias. Typically, having multiple comparators can provide different population characteristics and is a good sensitivity analysis to assess for potential confounding (as with the PDE5i example, which showed no increased risk in untreated patients and an increased risk with PDE5i use). However, this was not the case for triptans because healthier patients were selected for prescription of triptans because of the potential vasoconstrictive effect of triptans. The fact that they had fewer comorbidities, CV risk factors, and hospitalizations and emergency room visits compared with the prescription NSAID, opiate, or untreated cohorts might have contributed to the false impression of a triptan protective effect. In addition, both prescription NSAIDs and opiates have demonstrated adverse effects on CVD previously [11, 33]; hence, all comparisons showed a false protective effect.

Both studies were informed by several database limitations. First, it is difficult to verify the validity of diagnosis codes and to refine statistical analyses owing to the limited clinical details in the MarketScan database. Second, data regarding important confounding variables (e.g., smoking, alcohol use, body weight, body height, and social economic status) are not available in the claims database. Such variables are likely to be equally distributed among the study cohorts; in addition, propensity score matching was used to reduce any differences in distribution.

Measurement bias is a third important limitation. Because claims data are generally collected for the purpose of payment rather than research, the presence or absence of disease may not be accurate; similarly, diagnostic codes may be incorrectly coded or included as rule-out criteria rather than actual disease. Furthermore, diagnoses, medical procedures, and medicine dispensing were only included in the database if corresponding billing codes were generated.

Fourth, MarketScan claims databases are based on a large convenience sample, rather than a random sample. The included data generally originate from large employers, and medium and small firms are not well represented. Because the sample is not random, it may contain biases and may not be highly generalizable to other populations. Further, migraine diagnoses are not all recorded in claim databases, including MarketScan. According to Kolodner et al., medical and pharmacy claims were highly specific and moderately sensitive when used to identify patients with migraine [33]. This may lead to misclassification of exposure status in the comparison with non-migraine patients and likely draw the association towards a null finding.

Lastly, regarding the retrospective cohort study on acute treatment for migraine, only prescribed medications are recorded in the MarketScan database; it does not contain any over-the-counter NSAID medications, which can be used widely to treat migraine, pain or inflammation. The lack of over-the-counter drug information may result in unmeasured confounding, which could potentially impact the study findings. Furthermore, due to lack of information on over-the-counter NSAIDs, there may also be concerns about misclassification with respect to exposure status and exposure window. It should also be noted that it is difficult to estimate the exposure window for medications prescribed on an as-needed basis. Claims data do not include information about which exact days medication was consumed, whether it was taken as prescribed, or whether it was consumed at all. To avoid potential misclassifications when assessing drug exposure, exposure windows were defined using an ‘as-treated’ approach based on prescription fills, and 30-day, 60-day, 90-day, and ITT analyses were assessed. To further confirm the above negative findings, a positive control analysis (untreated migraine versus general non-migraine patients) was constructed in which some statistically significant increased risk for stroke was observed.

5 Conclusions

These case studies demonstrate that the choice of the comparator group can influence observational study findings. The TRT study findings demonstrated how, due to labeling restrictions, PDE5is may not be a good comparator to select in studies assessing CV outcomes. Similarly, due to vasoconstriction and labeling restrictions, triptan users were selected to have less baseline risk, which prevented researchers from identifying any appropriate comparator groups when evaluating adverse CV outcomes in migraine patients. In future real-world CV safety studies, such channeling may pose substantial challenges when comparing triptans to novel migraine treatments.

These observations contribute to the evolving understanding of methodological approaches used in observational research and how the appropriateness of study design must be considered in future drug safety studies.