Commentary: New York City: measuring progress as it’s made

Evaluation is foundational to public health practice. As a health department, we frequently monitor process outcomes (i.e. did the public recall our media campaigns?) and less often use longer-term outcomes (i.e. disease prevalence, hospitalization rates). Reduced mortality–still less used – is rightly viewed as a critical metric.

Muennig and colleagues report an ambitious attempt to quantify the impact of New York City initiatives launched under Mayor Michael Bloomberg on overall mortality and specific causes of death.¹ They offer a complex approach to isolate the specific impact of the City’s initiatives, but ultimately conclude that their analysis can neither prove nor disprove cause and effect. What should we take away from this?

First is the immense value of researchers contributing to evaluation science. Here are posed at least two questions: Are officials’ claims to reducing the City’s mortality rate valid? And were health advances experienced by the same people who became healthier, or rather by newcomer, healthier residents? The authors used complementary approaches, coarsened exact matching (CEM) based on survey data, and age-period-cohort (APC) models based on administrative death records, to determine mortality change between 2002 and 2010. They compare this with a baseline period of 1992–2000 and controlled for differences in demographics, including immigration. These approaches allow the researchers to estimate the magnitude of change while controlling for factors that affect mortality rates.

These models are useful, but our analysts were unable to replicate several of the reported findings. Making the data publicly available would be of great service. Moreover, the design’s pre- and post-periods implied a meaningful difference between the two periods. Control of non-communicable diseases (NCDs) intensified under Bloomberg, but these efforts were not a rupture from the preceding years.²

As practitioners at the interface of politics and science, we welcome prideful statements by elected officials about public health, especially in the neglected area of chronic disease. For any NCD analysis, the main challenge is time: these diseases have a long natural history. As the authors note, impact might be expected to take more time. And implementation – never instant – was often challenged in the legislature and the courts. We question whether mortality is the most useful measure of impact on this scale. Two examples, traffic-related policies and tobacco control initiatives, help articulate this point.

Traffic-related policies pursued in the study period targeted a vast array of activities: promoting cycling, reducing emissions, calming traffic and reducing injury. To use only the lens of mortality is to mischaracterize their aim. Indeed, some of the policies were not fully realized by 2010 – bike-sharing had not yet launched and bike lanes continued to expand in subsequent years. Even then, transport-related unintentional injury fatalities are not a major contributor to overall mortality.³

Likewise, New York City’s aggressive tobacco control agenda would have the most immediate impact on hospitalizations for acute myocardial infarction (AMI), not all circulatory disease mortality as assessed. AMI contributed to less than 20 percent of that total between 2002 and 2010.^4,5 Further, as the authors acknowledged, reductions in numbers of deaths due to lung cancer, particularly in the 65–84 age group, would not be immediate.

To determine whether a change in demographic characteristics contributed to lower mortality, the APC analysis used comparison cities. These were selected for comparable foreign-born populations that tend to be healthier than native-born populations – but other factors may introduce bias. Other cities adopted similarly ambitious public health initiatives. New York City (NYC)’s accelerated tobacco control agenda began in 2001, but California passed a statewide ban on smoking in all enclosed workplaces in 1995. Similarly, teenage deaths due to motor vehicle crashes declined sharply in much of the country in the early 2000s – but in NYC, where young adults drive less, we saw less decline.

Last, substantial variation in health outcomes by race and income are ‘controlled’ when these are rightfully a target of public health action. Our own analyses suggest that tobacco control efforts have had whole-population benefit; but in public housing developments, which concentrate intergenerational disadvantage, diabetes and HIV mortality have remained disproportionately high.^6,7

Public health in New York City today

Since 2010, we have continued a robust commitment to public health while expanding into new focus areas and launching innovative programmes and policies, many reflecting a broadened lens beyond the ‘lives saved’ mantra of the Bloomberg era. Whereas we want to reduce premature mortality, it is also important to help our population live healthier lives less burdened by preventable illness and disability.

To this end, Mayor Bill de Blasio and First Lady Chirlane McCray committed over $800 million across five years to ThriveNYC, an initiative totalling 54 initiatives by some 20 agencies to improve mental health, the leading contributor to disability. ‘Ending the Epidemic’ aims to bring new HIV infections in New York City down by more than half to 600 a year. And recognition of the substantial variation in health by race and income is now centre stage. This lens applies to the entire agency and is strengthened by the newly established Center for Health Equity.

Muennig and colleagues ask a good question that we should always ask of an intervention: did it work? But it is a question they could not answer. The question we should not ask in this case is whether we should stop. Well-founded policy should not be halted by demands for evidence.⁸ New York City policy initiatives that addressed tobacco, traffic safety, air quality and healthy food came at great political costs. Even as we write, the rule for simple informational sodium warning icons awaits the outcome of its third court challenge. Since 2014, there has been little policy action on tobacco control and no action on sugary beverages. Our data show plateaus in adult tobacco use and sugary beverage consumption. Although this analysis yielded uncertain conclusions, these data are clear – we cannot take our foot off the gas. In light of our unfolding political climate, it is all the more critical that we work with researchers and choose appropriate evaluation strategies to test impact. There is yet more work to be done.

Acknowledgment

The authors would like to acknowledge the contributions of several health department staff to this commentary, particularly Gretchen Van Wye, Catherine Stayton, Jennifer Norton, Mary Huynh, Sungwoo Lim, Wenhui Li, Shadi Chamany, Tsu-Yu Tsao, Corinne Schiff and Veronica Lewin.

Conflict of interest: None declared.

References