Keywords

1 Introduction

Following the analysis of the global architecture of indicators for the SDG framework, this chapter will discuss the ways that data production, use and harmonisation have been central to the construction of the materialities underpinning the epistemic infrastructure of the SDGs. For the infrastructure to work smoothly, it is not enough to simply produce the underpinning data and indicators: they also have to be harmonised to create a common ground of global knowledge in otherwise fragmented governing spaces. In the language of statistics, harmonisation indicates the ‘creation of a desired degree of comparability between statistics of different countries’ (Ehling, 2003, p. 17). This construction of the SDGs’ epistemic infrastructure has relied on the rise of the use of ‘non-traditional sources’ of data alongside ‘official statistics’ to monitor development agendas, and the harmonisation work done by International Organisations and National Statistics Offices to produce these monitoring frameworks. The concept of harmonisation emerges as one of the central ways that International Organisations govern the multiplicity of country-level measures to create universal, global-level metrics. Therefore, it represents a core material element of the epistemic infrastructure: this is the process via which flows of data are produced and the way the infrastructure materially unifies previously disparate parts.

By coordinating the diverse actors (including country governments, civil society, and various experts), IOs aim to achieve the development of universal measures, allowing for comparison between countries and monitoring of progress across the goals whilst sustaining the country ownership of data and indicators and their active participation in shaping the agenda. Therefore, harmonisation emerges as the central—yet often overlooked—process through which global knowledge and the epistemic infrastructures of global public policy are constructed. Harmonisation is a process through which a variety and diversity of national statistics become translated into one global number.

These universal metrics smooth out profound political, economic and cultural differences between different countries, promote and produce universal policy agendas, and ‘create comparability’ (Ehling, 2003, p. 17) by promoting competition between countries in various policy arenas. In order to achieve these goals, the process of harmonisation requires constant navigation between country-level measures, grounded in specific political and historical contexts, and global standards, striving for universality and the internationalisation of measurement. In this way, IOs produce the conditions by which countries construct knowledge about their own political, social and economic realities through harmonisation; the latter, in turn, produces and maintains the infrastructure that upholds global public policy, facilitated by commensurability and the production of common goals. We argue that there is no understanding of global governance without an understanding of the processes of harmonisation.

Harmonisation is indeed much more central to the global governance space than standardisation: the two concepts are closely linked but also quite distinct. Global spaces are often assumed to be governed through standards (Ponte et al., 2011)—and even though this is undeniably reflected in the history and practices of the international statistical community, as we will argue in this chapter, it is harmonisation that allows the SDGs to play their central role in creating the global governing space. A concept with a long legacy and multiple meanings, standardisation for the global statistics community encompasses a diverse array of efforts to produce statistical standards in methodologies, interpretation, estimation, dissemination and use of data and statistics. Despite the often-blurred distinction between the two concepts in meetings and official documents, scholars of quantification argue that standardisation is distinct from harmonisation, as the former is the ‘complete eradication of difference’ while the latter creates a unified field that is not necessarily uniform (Barry, 2001, p. 73).

In the following section, we outline the ways that standardisation and harmonisation have played a key role in producing governable realities, and how the harmonisation of statistics always entwines the scientific and the social (Desrosières, 2000). We then turn to the specifics of how data governance has become particularly complicated with the fragmentation of the global statistics community, as well as how the tensions that arise with data harmonisation are central to tensions in the 2030 Agenda as a whole. The following section discusses the example of data harmonisation for one SDG indicator—tuberculosis incidence (3.3.2)—to highlight how technical decisions about how to collect, estimate, collate and impute data in order to make global numbers about health are also decisions about how to delineate global public policy. Finally, we discuss how statistical capacity development has become a central stage for the harmonisation of data and the creation of global public policy.

2 Governing Realities: Harmonising and Standardising Data

In the social science and history of quantification and policy, harmonisation and standardisation are specialised modes of creating bound realities: this is particularly important for spatialising and cohering governable entities on national, regional and global scales. James Scott (1998, p. 13) argued that the use of standard spatial measurements and the mapping of land to be governed allowed sovereigns to ‘see like a state’, which, for example, for the creation of the modern French state hinged on the creation and use of the universal meter. Following Bowker and Star (2000), Timmermans and Epstein (2010, p. 71) define standardisation ‘as a process of constructing uniformities across time and space, through the generation of agreed-upon rules’. These standards then ‘tend to span more than one community of practice or activity site; they make things work together over distance or heterogeneous metrics; and they are usually backed up by external bodies of some sort, such as professional organizations, manufacturers’ associations, or the state’ (Timmermans & Epstein, 2010, p. 71). In the realm of producing globally governable entities, Vincanne Adams (2016, p. 24) shows how the disability adjusted life year (DALY) was specifically situated by Bill Gates to become the Lord Chancellor—or the ‘one metric to rule them all’—in the context of global health. Adams (2016, p. 29) argues that the DALY, by abstracting ‘quality of life and [turning] it into a fiscally meaningful form’, standardises ill health globally and makes health universally governable—here, by distinction, converting ill health into an economic concept and an entity to be governed by markets rather than a sovereign. Emerging from a ‘crisis of data’ in the Global South in the early 1990s, the DALY was meant to provide a universal yardstick for measuring successful or failing health interventions as well as the effectiveness of entire health systems, creating comparison between countries that might track health very differently from each other on the national scale.

Harmonisation, however, is distinct from standardisation. Andrew Barry (2001, p. 74) argues that, for the European Union, harmonisation allows for the ‘“mutual recognition” of national standards’ that facilitates unification and a common foundation for international activity in the production of a common ‘technological zone’ rather than the elimination of difference across borders. The harmonisation that International Organisations take on in the context of the monitoring of the SDGs includes elements of these different formations of governability. Harmonisation, in the context of the European Union, is the process of setting ‘the conditions within which a limited degree of standardisation […] is expected to occur’ (Barry, 2001, p. 64). Barry shows how creating a unified technological zone was crucial for creating an integrated Europe, and that technological regulation was a key component of this process.

Similar to harmonisation in the context of the European Union, the central United Nations secretariat delegates the creation of standards for harmonising sustainable development across the world. As we discussed in the previous chapter, this is achieved through the production and monitoring of indicators for progression on the SDGs. In using the term ‘harmonisation’ to describe dealing with difference over a vast array of economic, social, and political institutions, Barry distinguishes it from standardisation by asserting that the former allows for unification while the latter is the ‘complete eradication of difference’ (2001, p. 73). Rhetorically, at least, this emphasis on harmonisation in the context of the SDGs highlights the fact that actors in the UN space are still trying to unify very different national contexts under the global banner of sustainability, without claiming that they strive to remove all differences.

The technical work of harmonisation is, of course, always political and social. Manfred Ehling (2003, p. 29) refers to ‘conflicts of interest’ that must be addressed in the process of harmonisation, as ‘an abstraction from the different national institutions is needed for the definition of [the] international concept’ that is at the centre of harmonisation work. This international concept allows for both input and output harmonisation of data. Ehling breaks down strategies for harmonisation across time and geography into three ideal types, which are useful for distinguishing between different modes of governing data in the context of the monitoring of the SDGs. First, input harmonisation—also known as method harmonisation—requires harmonising the tools for data production. This can include, for example, requiring that ‘all participating countries use precisely the same survey procedures in an ideal case’, like standardising the questions on survey questionnaires (Ehling, 2003, p. 22). Output harmonisation, on the other hand, requires the establishment of an ‘international concept’ and involves statistical procedures to convert the ‘product of data collection to match that international concept. The second of Ehling’s ideal types is ex-ante output harmonisation, which—like input harmonisation—uses the design of data production tools as the space for producing harmonised data to capture that international concept but leaves the determination of methods for producing data to individual countries. Finally, in the context of ex-post output harmonisation, ‘national statistics are subsequently adapted by means of a conversion procedure in such a way that comparable statistics can be created’ (Ehling, 2003, p. 22, our emphasis). This abstraction and the production of an international concept creates a fundamental tension ‘between the quality criteria “international comparability” and “relevance of the (national) statistical concepts”’ (Ehling, 2003, p. 22). Because of the differences in national institutions, harmonisation will always create a gap between the nationally relevant concept and the international concept. This gap is wider or narrower depending on how much countries’ data production is shaped by the international concept. Alain Desrosières (2000, p. 173) argues that in the context of social statistics—for example, for education, health and poverty—this process of harmonisation is by definition both scientific—‘directed at the production of knowledge—and social—directed at the production of a common language as a foundation for debate on social issues’ (his emphasis).

Ideals of standards and harmony do not exist on their own. Like algorithms, techniques of harmonisation for SDG indicators are ‘sociomaterial tangles’ that are ‘composed of collective human practices’ (Seaver, 2017, pp. 3, 5). In the context of the SDGs, these techniques of harmonisation—including the production of ‘international concepts’ to structure data collection and synthesis—require deliberation, the providing of material and evidentiary support, and compromise in mandated spaces for such deliberation, support and compromise within UN agencies, affiliated International Organisations and working groups of the UN Statistical Commission. Most important of all, of course, is the fact that a ‘standard or a regulation does not have any natural force or intrinsic momentum. It is an authority which may be obeyed, ignored or opposed’ (Barry, 2001, p. 75). The harmonisation and standardisation of data require authority to be taken seriously, and they require country buy-in in order to be implemented on the national or local levels. In the rest of this chapter, we will delve carefully into the ‘sociomaterial tangles’ of the harmonisation of data for global SDG monitoring, and how these tangles serve work to unify heterogeneous actors under the banner of a global movement for sustainable development.

3 Centrality of Data Production and Harmonisation for the SDGs’ Global Public Policy

In order to compare social, economic, political or environmental conditions in two different geographical locations, statisticians harmonise data—by either creating the conditions for producing comparable data or adjusting data after they have been produced—that may have been produced with even slightly different methodologies, including sampling techniques or differently worded answers to questions on household surveys. As many members of the global statistics community made clear in interviews, there is also the importance of harmonising data temporally—being sure to be able to compare contemporary statistics to those that were produced at a different moment in time, when different techniques might have been available—a process that makes statistics ‘sustainable’. In the production of the SDGs, harmonisation across space and time happens on many levels, and International Organisations (IOs) and National Statistics Offices (NSOs) engage all (and combinations of all) three of Ehling’s ideal modes of harmonisation in the production of comparable data, outlined in the section above. Harmonisation occurs on the global level: IOs ‘create comparability’ between countries in order to rank performance and identify progress on the SDGs, using both official national data and data produced by donors, civil society and academia. It also happens on the national level, where NSOs are responsible for harmonising data production across different governmental agencies, non-traditional sources of data like geospatial data, and non-governmental and donor-produced data, in order to create a national view of policy problems coherent across different data sources and across time.

3.1 Types of Harmonisation

Outlining the ideal types of harmonisation, an example of (near) input harmonisation is UNICEF’s Multiple Indicator Cluster Survey (MICS), which is a standardised survey questionnaire that has been used since 1995 to produce comparable data about women and children in 118 countries in the policy arenas of health, education, poverty and more. Data produced by MICS are used both by UN agencies—particularly UNICEF—and countries to monitor progress on poverty, health and education goals. Ex-post output harmonisation, however, is of particular use to UN agencies in harmonising data for monitoring the SDGs. This process involves taking nationally produced data—which may not use the same exact survey questions or identical age ranges in survey sampling, for example—and using statistical tools to adjust situated data that can be compared to others in other contexts. More recently, International Organisations (e.g., the World Bank in their activities on poverty data, e.g., Povcalnet database and the global poverty numbers) started to highlight the value of ex-ante output harmonisation. At the level of country dialogue, the IO experts shape the design and collection of the household surveys in ways that then fit the ‘global level’ requirements.

Statisticians and development data specialists explicitly link the production, harmonisation and use of data to both the production of global agendas and the success of such agendas. From their perspective, there is a danger if data practices do not link up closely enough to the global agenda. Representing Statistics Sweden in the first consultative process in 2015 for the SDG indicators on the part of the Inter-Agency and Expert Group on SDG Indicators (IAEG-SDGs), Viveka Palm highlighted many difficulties in creating harmonised development data, including that:

The targets are formulated in words—the data needs to be much more specific. This is the only way to gather data in a harmonised fashion. So, between the policy makers [sic] wish and the possible measurement there will often be a gap. The statistics strive to be objective and so they are sometimes hard to interpret in the indicator sense (Measuring the activities in the economy is not equivalent to measure “Sustainable economy” for example.). (IAEG-SDGs, 2015b, p. 382)

Converting political and policy goals—and targets—into statistical measures that can then be ‘harmonised’ is emphasised by Palm, among other statisticians, as being a key problem faced by members of the global statistical communityFootnote 1 in bringing the ambitious 2030 Agenda into fruition. There are often ‘gaps’ between the intention of the targets and what is measurable—both in the sense of what is quantifiable and what data is actually available. Many of the statisticians consulted expressed dismay that they had not been a part of the goal-setting process themselves, in order to help formulate language that would more easily lend goals and targets measurable. However, some scholars have argued that, no matter how participatory the process, the focus on creating and measuring indicators has—like the MDGs before them—simplified a very ambitious agenda into a practice of ‘treasuring what we [already] measure’, rather than a genuine revolution in what matters in global development (Fukuda-Parr & McNeill, 2019; Yap & Watene, 2019).

Within the UN’s 2017 Resolution to adopt the SDG framework, the following roles are set out for IOs and member states around the production, harmonisation and estimation of national data used to monitor the SDGs, where the UN secretariat:

Urges international organizations to base the global review on data produced by national statistical systems and, if specific country data are not available for reliable estimation, to consult with concerned countries to produce and validate modelled estimates before publication, urges that communication and coordination among international organizations be enhanced in order to avoid duplicate reports, ensure consistency of data and reduce response burdens on countries,Footnote 2 and urges international organizations to provide the methodologies used to harmonize country data for international comparability and produce estimates through transparent mechanisms. (UNGA, 2017, p. 3)

Built into the SDG monitoring framework, as recognised by the UN, is the responsibility of IOs to harmonise nationally produced data for the purposes of ‘international comparability’, to make available the means by which they ‘produce and validate modelled estimates’, and to coordinate with other IOs in order to verify such internationally comparable and sometimes imputed data. In this way, despite the explicit attempts of different UN institutions to guarantee the leadership role of member states in the 2030 Agenda, it is in fact the IOs who are the enforcers of harmonisation. Thus, it is IOs who have the final responsibility to validate the contours of this global public policy—and the processes of harmonisation are of key importance to the International Organisations as central to the production of their ‘flagship’ metrics (such as the International Poverty Line by the World Bank, or the Human Development Index). Consequently, this is the area in which the conflict between IOs and countries and contestation over the global numbers emerge, since the numbers produced via harmonisation might not be the same as the national numbers.

3.2 The ‘Data Revolution’ and Its Effects on Harmonisation

For many members of the global statistical community, the centrality of data production, harmonisation and use has been sped up by the SDGs. However, it is also the result of the ‘data revolution’ of proliferated and proliferating digital technologies, whose unequal distribution has exacerbated information inequalities that already existed between the Global South and the Global North. In 2014, UN Secretary-General Ban-Ki Moon commissioned a report on these uneven effects of the data revolution on global development, which was published as A World That Counts. The goal of this report was to set out a path towards ‘mobilising the data revolution for sustainable development’ (IEAG, 2014, p. 2). The authors argue, beyond a common lack of capacity and resources, that too ‘often, existing data remain unused because they are released too late or not at all, not well-documented and harmonized, or not available at the level of detail needed for decision-making’ (2014, p. 3). As part of a larger movement for evidence-based policymaking, this report reiterated that ‘improving data is a development agenda in its own right’ and called for a UN-led global public-private partnership—which would become the Global Partnership for Sustainable Development Data (GPSDD)—and an annual global conference on development data—which would become the UN World Data Forum. These communities include both those supporting and harmonising ‘official statistics’ and those working in parallel streams of ‘non-traditional’ sources of data—Big Data, private industry, geospatial, academic and so on—with which National Statistics Offices and International Organisations must increasingly contend. Mobilising the data revolution, in this context, means demanding that NSOs and statistical offices of IOs know how to ‘filter the wheat from the chaff’ on the part of what is useable and what is not (MacFeely, 2019, p. 130).

For statisticians in the UN space, then, the goal is to produce standards for official statistics across both geographic difference and over time. ‘Statistics’ for this community means both producing data through standardised methodologies and converting these data—as well as alternate streams of data—into official statistics through standardised methodologies. These are standard methodologies that would allow statisticians to compare poverty rates in two different countries as well as compare poverty rates at two different points in time. With the rise of non-traditional sources of data in SDG monitoring, statisticians see the potential for capturing phenomena that NSOs do not currently capture, which has the potential to provide evidence to vulnerable populations who want to make their perspectives count. However, the rise of non-traditional sources of data creates new problems that statisticians must contend with, as well as new forms of governance both on the part of NSOs and IOs in harmonising data. For some, non-traditional sources of data also raise the question of what has been called the ‘sustainability’ of statistics, as one statistician put it:

[One] of the things that I’m also very concerned is the sustainability of statistics itself. Because I mean there are sometimes people that bring data to the table that are from one particular moment in time, because somebody had time, had resources and opportunity to collect the information. And maybe it is a good snapshot of that moment in time, but ultimately that is not what we were interested in. Because what we are measuring is development. That means change over time, and my concern is that we have ad hoc collection of information at one point in time and somebody else will decide years later. Inevitably people will divide the newer number by the old number and say something has grown by 10% or discreet increased by X%. But of course, that statement is only correct if the two methodologies have been identical. And that is very often not the case with Big Data and ad hoc data collection. So, we are arguing strongly that the national statistical system is the only one you need to institutionalise data collection and perhaps sometimes be a little bit more modest. Collect less but collect it consistently so that after five or 10 years you can really make meaningful assessments of whether you have made progress towards any policy agenda. (UN Statistician, 3)

In order to track change over time, the two data sets must be harmonised—here, in particular, through input harmonisation, that is, the two methodologies for collecting the data at different points in time are ‘identical’. This statistician argues that a benefit of harmonisation is that it grants statistics the characteristic of ‘sustainability’: statistics that are produced now will then continue to have meaning in the future. He contrasts these sustainable statistics with ‘Big Data and ad hoc data collection’, which utilise data production techniques that might not be standardised over time and thus will not be guaranteed to have meaning in the future. From this perspective, measuring development—linked implicitly here to achieving development—requires harmonisation and the taking on board of standards.

As non-traditional sources of data and their use have increased in the context of international development since the late 1990s, the development data world has quickly grown and become more and more fragmented—and the work required to harmonise development data has become more complex in the process. The growing of the field has also led to conflicts over authority of ‘official statistics’, particularly on the national level, as NSOs have had to engage with data scientists and producers of non-traditional sources of data more and more. One member of the global statistics community argued that the difference between data and statistics is a thorny issue in SDG monitoring, which still requires some effort to overcome, as it is still not clear ‘about whose issue’ the production of SDG data is:

I think statisticians feel like this is all their thing, whereas I think some of the people who’ve been involved, for example, in data science in the private sector see data as something which has a role in industry and in decision-making and in government to some extent that is completely separate from the process of producing official statistics and so I think that debate is to some extent still, I think the different sides, insofar as there are sides, have come to trust each other better, which is good and this is obviously an ongoing debate and it’s part of the shift I think within countries about thinking more systemically about their use of data across government and having statistics as part of that, but not the only part of that. (Civil Society, 1)

As an attempt to address these complexities, the 48th UN Statistical Commission, in 2017, called upon the IAEG-SDGs to ‘develop guidelines on how custodian agencies and countries can work together to contribute to the data flows necessary to have harmonized statistics’ (UNSC, 2017, p. 48). The working group developed guidelines and best practices for harmonising between the national, regional and international levels (IAEG-SDGs, 2018; IAEG-SDGs, 2019). It also produced a series of ‘data flow’ case studies to understand the ‘how an indicator is adjusted, estimated or modelled, and validated by the national statistical system for global reporting’, following one indicator from one country to one international agency in each case (IAEG-SDGs, 2017, p. i). These case studies make it clear how the data reporting and harmonisation processes look very different depending on the indicator and the International Organisation responsible for it—and the transparency about how data are converted to match the ‘international concept’ also varies tremendously. They also make clear the varying role of country-level ministries (and which ministries are) involved in the production of harmonised data. The WHO, for example, states that it has the same relationship with each country for gathering morbidity and mortality data about tuberculosis (through the Ministries of Health and National TB Programs), who reports directly to the WHO at regular points during the year which helps them produce data estimates every year for the country. On the other hand, in the case of ‘Indicator 2.1.2 Prevalence of moderate and severe food insecurity’, FAO engages with some countries a bit differently, liaising with NSOs in some cases or with the private company Gallup in others in order to harmonise national data about the indicator.

Conceptualising the harmonisation of data in the context of the SDGs requires dipping into the technical language of statisticians and data scientists, in order to understand the contours of how data is governed for harmonisation. For example, one technique for ‘filtering the wheat from the chaff’, as MacFeely put it (2019, p. 130), and facilitating input harmonisation for SDG monitoring, is by producing data in such a way that they can be used for multiple purposes. This requires ‘interoperability’, which is:

the ability to join-up and merge data without losing meaning (JUDS, 2016). In practice, data is said to be interoperable when it can be easily re-used and processed in different applications, allowing different information systems to work together. Interoperability is a key enabler for the development sector to become more data-driven. (Morales & Orrell, 2018, p. 9)

In this way, interoperability is a technical tool for making data inherently harmonisable. From the perspective of the UN, this requires ‘being modest’ with data collection (UN Statistician, 3)—setting manageable expectations about what is collectable and programmable to be sustainable.

A particularly contentious space in the production of harmonised data are statistical models and the practice of imputation, the latter of which UNECE defines as a ‘procedure for entering a value for a specific data item where the response is missing or unusable’ (UNECE, 2000, p. 8). As Brazil’s NSO (the Instituto Brasileiro de Geografia e Estatística) asserted in its official assessment of the SDG indicators suggested on 11 August 2015, ‘the use of mathematical and/or statistical models to calculate indicators must also be disregarded, since any model is developed according to a given set of assumptions and relatively arbitrary parameters’ (IAEG-SDGs, 2015b, p. 4). According to many statisticians, there is some degree of statistical modelling and estimation that is required in producing robust statistical information on countries and their economic and social statistics. In response to IBGE’s assertion about the use of statistical models, a representative of FAO argued that:

Contrary to the implied preoccupation that informs the statement, it is the absence of a proper statistical model in informing an indicator that creates arbitrariness, variability and the impossibility to harmonize measures across countries. The presumption that meaningful indicators could be produced by simple arithmetic computation from primary data collected through censuses or surveys without any statistical treatment is actually a very dangerous one. Models based on sound statistical inference theory are essential, and their use should be broadly promoted, as they are the only instrument to ensure a sufficient degree of reliability and comparability of indicators, which should always be seen as estimates of the likely true value of the variable of interest. (IAEG-SDGs, 2015b, p. 39)

Here, this member of the global statistical community is arguing that techniques for adjusting, modelling and imputing data are central not only to the goal of producing harmonisable and meaningful data to monitor global progress towards the SDGs, but also to the production of national statistics in general. The ‘dangerous’ idea is in fact that data can be made meaningful without ‘a proper statistical model’—it is its absence that would make the processing of data ‘arbitrary’.

3.3 Neocolonialism? The Creation of ‘Parallel Systems’

However, due to a few key actors in the global space—in particular, the Bill & Melinda Gates Foundation (BMGF)—many NSO representatives now understand statistical modelling and imputation as a ‘neo-colonial’ practice. One member of the global statistics community argued that BMGF has ‘largely focused its funding on setting up parallel systems’ to national statistics systems, and as a result, in some countries in the Global South, there are ‘tens and hundreds of millions of dollars going into parallel systems while the civil registration system is starved of resources, and it’s because of this obsession with metrics’ (UN Statistician, 13). This creation of parallel systems has real material effects on NSOs and national ministries, and it understandably produces uneasiness with national statisticians on certain technical aspects of producing harmonised data:

[There’s] this tension here which then comes back to haunt us, because I think we’ve seen a lot of National Statistics Offices in the Global South being disempowered because they lack technically trained staff, so they see the enterprise of estimation and modelling as extremely threatening, see it now as something that’s done in Seattle[, Washington] with super computers and people who look like the people who work at Google and Twitter, so it’s like that’s some kind of rocket science and so they want to cast that as a form of neo-colonialism as opposed to saying, well, hang on a minute, we do need to do modelling, we do need to do estimation. It’s even done in statistical agencies in countries like New Zealand and the United States and Canada. No data system is complete, we’re going to miss people. That’s just the story of population data and we model, we adjust, there’s a whole science around moving from raw data to meaningful and consistent estimates and that’s not just some monopoly which [IHME’s Chris] Murray and Gates have, that’s something which should be imbued across the entire statistical system of the globe and those skillsets need to be developed in places like Ouagadougou. (UN Statistician, 13, our emphasis)

As discussed in the previous chapter, the capacity to produce, process and use national and sub-national level data is a fulcrum point for the tension between nationally driven development agendas and global agendas like the SDGs. Harmonising data for global monitoring similarly invokes this tension, as IOs enact global public policy through data production, use and harmonisation, processes which we will now outline in the policy areas of health and poverty. This is also an area where the often-contested data practices take place, in order to fill in the missing data (in the cases where the household surveys are missing for a couple of years)—such as nowcasting data or taking country averages.

3.4 Spotlight on Health: The Case of Data Collection on Tuberculosis

Harmonising and monitoring the 27 indicators listed under Goal 3—the health goal—are the responsibility of many UN agencies, including UNICEF, UNAIDS, UNODC, DESA-PopDiv and OECD. Of course, the WHO is at least partly responsible for the large proportion of these (20 of the 27), as well as many health-related indicators situated outside of Goal 3. For each of these indicators, the WHO works with both nationally produced data and its own global estimates of disease burden, which are also modelled on the basis of nationally produced data. Their global estimates are currently produced by the BMGF-funded Institute for Health Metrics and Evaluation (IHME), due to a memorandum of understanding signed between the WHO and IHME in 2018 to partner to produce global estimates (Tichenor & Sridhar, 2020). Since WHO’s own team for producing global health estimates, which was led by Colin Mathers, resigned in 2019, IHME ‘is now really the only modelling game in town’ when it comes to estimates of global disease burden (UN Statistician, 1). Because these estimates can deviate quite widely from numbers reported by ministries of health, and because the WHO is organisationally responsible to its member states, the organisation consults extensively with countries before publishing their global, harmonised health data. To get a sense of what this process of harmonisation looks like for health data, we need to turn to the process of harmonisation for one specific indicator: SDG 3.3.2, ‘Tuberculosis incidence per 100,000’.

In the report on case studies of data flows discussed above, the WHO described its method of collecting—as well as estimating—tuberculosis (TB) morbidity and mortality data for all countries in the following way:

Estimates of the burden of disease caused by TB and measured in terms of incidence (SDG indicator 3.3.2, expressed per 100,000 population per year) and mortality are produced annually by the World Health Organization (WHO), using case notification and death registration information gathered from every country through surveillance systems, special studies (including surveys of the prevalence of disease), mortality surveys, surveys of underreporting of detected TB, in-depth analysis of surveillance and other data, expert opinion and consultation with countries. (IAEG-SDGs, 2017, pp. 22–23)

The authors of the data flows report asked each custodian agency, in a separate category, to describe the ‘process by which national data is converted to SDG indicator’ (2017, p. 23), where the WHO has inserted its annually updated methodology appendix for estimating TB incidence. In the 2020 version of this methodology paper, Glaziou and his colleagues explain that TB incidenceFootnote 3 (rather than prevalence or mortalityFootnote 4) for the global indicator was determined in four ways, depending on the country:

(​i​) results from TB prevalence surveys (29 countries, 66% of global incidence); (​ii)​ notifications in high-income countries adjusted by a standard factor to account for underreporting and underdiagnosis (139 countries, 6% of global incidence) and (​iii)​ national inventory studies (8 countries, 17% of global incidence); (​iv)​ case notification data combined with expert opinion about case detection gaps (39 countries representing 11% of global incidence in 2019). (Glaziou et al., 2020, p. 1)

The WHO in this way employs both input harmonisation and output harmonisation in its creation of global numbers. For many countries, the organisation also uses multiple methods to create the complete timeline of TB incidence within the years 2000–2019. The ideal mode of producing the SDG indicator for TB, as expressed by the WHO, was through method (ii), where countries were recording notifications of new cases of TB through a highly functional health surveillance system.

However, the focus is on TB incidence, rather than mortality or prevalence. This is because the target that the indicator 3.3.2 is meant to measure sets out to ‘end the epidemics of AIDS, tuberculosis, malaria and neglected tropical diseases’, among other things—a goal acknowledged to be over-ambitious. Tracking incidence ‘was estimated with considerable uncertainty in most countries in 2014’, but the WHO argues that ‘notifications of cases to national authorities provide a good proxy if there is limited under-reporting of detected cases and limited under or over-diagnosis of cases’ (WHO, 2021, p. 3). The process by which numbers obtained through these four pathways are ‘adjusted by a standard factor to account for underreporting and underdiagnosis’ is determined by ‘expert opinion’, which the WHO has determined in regional workshops ‘where expert opinion was systematically elicited following an in-depth analysis of surveillance data’ (Glaziou et al., 2020, p. 5). Glaziou and his colleagues set out the limitations of the WHO’s mode of estimating TB incidence, including ‘a generally small number of interviewed experts; lack of recognition of over-reporting; and others’ (Glaziou et al., 2020, p. 6). Further, IHME has also developed an alternative mode of measuring TB burden estimation (Murray et al., 2014), which is ‘generally consistent’ with WHO’s estimates on the global level but varies widely in certain countries. The authors argue that the solution to this problem of ‘considerable uncertainty’ is statistical capacity development: ‘Discrepancies in estimates from different agencies reflect the questionable quality and completeness of the underlying data. Further convergence in estimates will result from improvements in measurements at country level’ (Glaziou et al., 2020, pp. 22–23)

Clear in this document on data flows for reporting on TB, too, are complications involved with creating an ‘international concept’ for monitoring global progress on TB, as well as the ways that scientific and social—to use Desrosières (2000) language—are always intertwined in the creation of quantified governance. Most important is the problem of the co-morbidity of TB and HIV, since TB has become the leading cause of AIDS-related deaths (Pawlowski et al., 2012). Because of the ways that TB and HIV ‘act in synergy’ to accelerate ‘the decline of immunological functions’ (Pawlowski et al., 2012, p. 1), indicators to measure TB/HIV have been high on the priority list for organisations invested in combatting TB and HIV, including WHO, UNAIDS, PEPFAR and the Global Fund. Measuring TB prevalence among HIV-positive populations (with the goal of creating a global number for Indicator 3.3.2) in many countriesFootnote 5 where TB is most prevalent is highly uncertain: it is based on assumptions about HIV prevalence in certain countries and assumptions about the prevalence of TB among HIV-positive populations. The international concept for Indicator 3.3.2 must both encompass and gloss over the syndemic of HIV and TB.

Once the WHO’s TB programme has its own estimates of TB incidence, annually by August, the organisation then communicates these numbers with countries to verify, and revises its own estimates based on the feedback it receives, before publishing their global numbers every October (WHO, 2021, p. 2). This estimation and verification process predates the SDGs and mirrors other data production and verification procedures for other health programmes within the WHO. The WHO, then, has maintained ministries of health as its main points of contact on the country level rather than national statistics offices, which are the main points of contact for a large part of the global statistics community. As one member of the global statistics community put it:

WHO is in a position which is slightly different from the other agencies because our constituencies are not the national statistical offices because our constituencies are the ministries of health. So, our interlocutors in country are ministries of health. So, even if we do approach the national statistical office, this usually has to be through the ministries of health. And our governing bodies are ministries of health; they are not the national statistical offices. (UN Statistician, 5)

He went on to assert the fact that collecting and verifying health data is in fact in the 1945 mandate of the WHO, and the SDGs are merely the most recent global agenda to sit upon their already existing data production system. He also asserted that although the WHO encourages countries to create relationships between the ministry of health and the statistical office, they can do nothing to enforce those relationships.

However, from the perspective of UNSD, the harmonisation of data on the country level—and its enforcement—should also be in the remit of custodian agencies. In the context of health, as with other policy sectors, there are multiple ministries on the national level interacting and producing data and statistics that may or may not be comparable within different organisations on the country level. According to one representative of UNSD, it is the responsibility of the NSOs and UN agencies to make sure that this data is harmonised on the country level. This requires ‘incentivising’ different ministers on the country level, by saying, in the case of health, to these ministry officials:

‘OK guys, you’re doing all those wonderful numbers for health and the Ministry of Health, but have you double checked with the National Statistical Office, are your numbers in sync with the overall population numbers that the Statistical Office manages from the census’. Even stupid little things like age groupings. We have had situations where one ministry has age groups from zero to five, from six to 10 and 11 to 15 and then the next, the other ministry has it grouped from zero to three, from four to seven and so in three-year intervals and so if you then want to conduct any kind of study, like how is health and education for instance related to each other, is there any discernible effort if you run an education concern on health behaviour. You can’t do it because the two ministries, and that’s precisely I think where the National Statistical Office has a role to play to harmonise the frames and, yes, when you work together with others, with the other children then sometimes you have to choose something that is slightly suboptimal for you. (UN Statistician, 3)

Harmonising these frames on the national level allows for different policy arenas to be in communication with each other.

4 Statistical Capacity Development: The Material Production of Development Data

The incomplete, unequal or inconsistent production of official statistics and development data has been framed, particularly since the late 1990s, as its own development problem that requires careful strategy and planning to address. Even before the creation of the Millennium Development Goals and their indicators, certain development plans like the World Bank’s Poverty Reduction Strategy Papers, introduced in 1999, and UNICEF’s 1990 World Summit for Children put quantified indicators at their centre as a means to measure progress towards development goals and promote adherence to them. The underperformance of many National Statistical Offices (NSO) in the Global South to produce the evidence that these development agendas demanded of them, as well as the ‘conflicting donor agendas’ (Marrakech Roundtable, 2004, p. 2) that NSOs had to negotiate and that also shifted attention from domestic agendas to global ones, led to a rise in institutional and financial attention to sustainable statistical capacity development—at the time called ‘statistical capacity building’.

Support for statistical systems has been categorised—by Rolando Avendano et al. (2021), Shaida Badiee et al. (2017) and others—into two forms: support that takes a ‘demand-driven’ approach and that which takes a ‘supply-driven’ approach to development data and official statistics. There is variability amongst and within International Organisations as to which approach they take. The Partnership in Statistics for development in the twenty-first century (PARIS21), for example, firmly supports a ‘demand-driven’ approach to capacity development, as an organisation whose objective is to advocate for better statistical systems in countries in the Global South for these countries’ own development objectives. This is an economic metaphor for indicating that the production of statistics and data should be driven by the demand of countries for evidence to inform their national policymaking, rather than the ‘supply-driven’ model of data production that has dominated global governance. Supply-driven statistical capacity development refers to an emphasis on the mere existence of data as evidence of success and not on whether national statistical systems were actually strengthened in the process. Although many UN agencies might not admit to adhering to this philosophy, in practice this latter approach is the dominant one. In the context of the SDGs, the statistical capacity development that UN agencies must facilitate, as custodian agencies of individual indicators, is to provide support for countries to produce data to populate these 231 unique indicators. Some of this funding goes not to statistical systems but instead ‘data intermediaries’, which include international consultants and global data producers like the Institute for Health Metrics and Evaluation (IHME). In this way, most statistical capacity development within the SDGs is supply-driven, as donors provide funding to help produce the data they need for global monitoring. As the UN Statistical Commission (UNSC) set out roles for the post-2015 development agenda, it:

stressed the urgent need for investments to enhance national statistical capacity, especially in developing and least developed countries, to measure progress towards the post-2015 development agenda at national, regional and global levels, and enable national statistical offices to play a leading and co-ordinating role in this process. (UNSC, 2016, p. 12)

In this way, the UNSC recognised that one of the key goals of enhancing national statistical capacity was to enable national statistics offices to take leadership over the SDG monitoring framework, and the governing framework by extension. Fundamentally, however, the bulk of the statistical capacity development work that IOs take on facilitates their ability to harmonise data for their purposes of consolidating policy arenas and creating comparability between countries. In other words, much of the work of statistical capacity development in practice in the context of SDG monitoring—despite rhetoric that describes it otherwise—is about producing a terrain for IOs to govern.

In 2016, the United Nations General Assembly commissioned a study by the Joint Inspection Unit to evaluate the work of the UN development system on strengthening national capacities for statistical analysis and data collection, particularly in the support of achieving globally agreed goals, including the MDGs. The authors of the report on the study emphasised how the SDGs would put even more pressure on national statistical offices than the MDGs did, and that although statistics were the means to the end of achieving development goals, and not the goal itself, strengthening national statistical systems was critical to making progress on development goals. Thus, achieving the goals would require unprecedented support from the UN system in order to strengthen the production, dissemination and use of statistics. The authors of the study also identified the great challenges to the success of this support being ‘the coordination of activities, the sustainability of the results and the relevance of activities to the priorities of all national stakeholders’ (UNJIU, 2016, p. 8).

The landscape for producing statistics and data for national, regional and global development objectives and policymaking has changed in fundamental ways in the twenty-first century. Although attempts to universalise official statistics have been a part of the UN’s programme since its inception, there has been a broadening of the statistical community with the ‘emergence of quote/unquote, “data science”’ in the last decade and a half (UN Statistician, 13). This ‘emergence’ and rapid rise of data science in international development has produced new epistemic communities, fostered new partnerships and initiatives, and further fragmented the global public policy space. This shifting statistical terrain ‘poses serious questions in terms of “what’s the role of official statistics” and “what’s the particular position of an official statistical agency”’, whether that be an NSO or an UN agency or other multilateral or bilateral organisation (UN Statistician, 13). In the evaluation of the World Bank’s investment in data for development by its Independent Evaluation Group (IEG), the authors argue that a ‘coherent architecture existed for the older generation of partnerships for statistical capacity building, but coherence is missing for the new partnerships involving data innovation’ (World Bank, 2018, p. x).

Accompanying the conceptual work of PARIS21 with financial support of statistical capacity development, the World Bank established the Global Trust Fund for Statistical Capacity Building (TFSCB) in 2000 to help countries strengthen the production of their official statistics. At the centre of both of these efforts was the goal of building ‘a culture of evidence-based policy making’ (Marrakech Roundtable, 2004, p. 2). PARIS21’s work varies widely, but in its 2017 mission statement on ‘Capacity Development 4.0’, Keijzer and Klingebiel (2017, p. 15) argue that ‘country ownership’ of the development of what they call ‘National Statistics Systems’ (NSS) is most critical for creating sustainable official statistics production. As a key mode to support this country ownership, PARIS21 helps countries in the Global South develop National Strategies for the Development of Statistics (NSDS) to help create domestic plans for ‘evidence-based policy’. This has echoed in other corners of this community, as the authors of that UN the Joint Inspection Unit mentioned above also argued that, with the goal of promoting evidence-based policy, the production and use of statistics must be understood as inseparable: ‘It is […] not a case of supporting either production or use of statistics, as the two are intertwined and have a logical linkage’ (UNJIU, 2016, p. 9).

Through the institutional, deliberative and financial work of supporting the production of development statistics, International Organisations have actively worked towards harmonising global public policy through the production and use of quantified data. While PARIS21 and TFSCB have had a more holistic conception of developing NSOs and their larger NSSs, many UN agencies have supported statistical capacity development activities that are largely focused on their own policy arenas, like the International Labour Organization’s (ILO) activities for bolstering the production of labour statistics or UNESCO’s work on bolstering education statistics. The MDGs were fundamental to placing the measurement of indicators at the centre of global public policy, as they explicitly put the responsibility of monitoring and reporting on progress on the shoulders of member states and UN ‘custodian agencies’ of each indicator. This obligation to monitor and report on poverty, health, education and other indicators also made clear just how many gaps there were in the regular production of social statistics in many countries in the Global South.

The MDGs also brought to the fore a systemic problem in the production of development data and official statistics for global development, as one member of the global statistics community described it:

in a very aid dependent country, [there are] actually two data systems. There’s the data system that the government is painstakingly trying to build [with] inadequate money and not enough people and not enough technology and so on and just to provide that continual feed of information for government decision-making. And then, there’s the data system which donors have and fund and is for their own monitoring and evaluation and in line with their own programmes of their preferred surveys for international comparisons and those sort of things, and [often] in a very aid dependent country it’s not hard to see how the latter can undermine the former in all sorts of ways, just by diversion of resources, by diversion of people, by warped political incentives, and so on. (Civil Society, 1)

In fact, in their 2005 summary of recommendations, the Inter-Agency and Expert Group on Millennium Development Goals Indicators (IAEG-MDGs) emphasised that ‘international agencies should rely more heavily on official statistics produced by national statistical offices for their data needs [and] coordination by donors, bilateral and United Nations agencies in countries should be improved’ (IAEG-MDGs, 2005, p. 8). In this way, statistical capacity development represents two processes at once: on the one hand, it is the work of multilateral, bilateral and philanthropic organisations to provide technical and financial support for the statistical system that a country needs for its own programming, and on the other, it is also the support for the statistical system these International Organisations need for monitoring their own priorities. These two goals for statistical capacity development can—theoretically—align (and sometimes they do), but there are also ample examples of when they do not. In light of these issues, PARIS21 and the UN Statistical Division (UNSD) have attempted to address the problems that these parallel and sometimes contentious double streams of data can produce.

It is a widely held view that the SDGs were a large step towards producing a global agenda that is participatory for all member states. UNSD played a key role in trying to produce a space—both at the annual UN Statistical Commission but also by creating working groups that mandated country participation from various geographic regions. According to one member of the global statistical community:

It was curious for instance, the role as go-between of the UN Statistical Division, which was somehow overreacting to try to overcome political tensions in terms of having the countries with the perception that it was driven by the UN or by the international system and they didn’t want to. So, they went to the other extreme to let all the countries with the national statistical offices […] decide on what to use and which kind of information or indicators should be included. Which in many cases was really an impossible task for the national statistical offices because they were not acquainted with all the domains of development, especially with environmental issues, which were I think supposedly the core of the SDGs. So, we started a process that was painful at the very beginning, but then it went very well in the sense [of starting to work] together to interact as never before. Even from my national experience and then afterwards with the regional and the global experience, I have never witnessed a process that was so really participatory in many aspects. (UN Statistician, 11)

UNSD’s introduction of institutional modes for participation included mandating that member states from two different geographic regions co-chair the Inter-Agency and Expert Group on SDG Indicators (IAEG-SDGs) and that member states have an active role in refining the indicators for each SDG target and goal. As this UN agency representative makes clear, this participatory process was much more involved than those of earlier development agendas and required the active involvement of NSOs—some of whom ‘were not acquainted with all the domains of development’—to decide on methodologies and data sources for all aspects of the SDGs.

As the representative for the Samoan and Fijian statistical offices expressed to the IAEG-SDGs in the first SDG indicator consultative process in 2015:

We believe it is HIGH TIME for all National Statisticians, to see statistical development appear as a development objective in its own right. I hope, madam Chair, as a fellow national statistician, you share in our delight. Having said this, we strongly believe that we require a better indicator, something that builds on the World Bank’s [Statistical Capacity Index], but which would allow the measurement of 3 core components of national statistical capacity: a. Human capacity (trained, experienced staff to do their job); b. Financial capacity (with Governments providing more than just ‘shoe-string’ budgets for their NSOs that extend beyond payment of salaries, and actually enables NSOs to do their jobs; and c. Political-institutional capacity, that embraces a culture of evidence informed policy development, planning, monitoring of progress and accounting for results—which requires access to quality and timely statistics. (IAEG-SDGs, 2015a, p. 2)

The SDGs converted statistical capacity development into this ‘development objective in its own right’. Perhaps expectedly, consensus about what defines statistical capacity in global governance does not exist. According to one member of the global statistical community (UN Statistician, 7), ‘individual organisations are defining capacity and statistical capacity the way they think or their institution mandates to do that. [Consensus] would mean that there are parties who argue about it, [that] there would be a specific discussion, oh, let’s define statistical capacity once for all, and then we all concur to that one, which it’s like with many other terms, [but] it’s not per se happening in this way’ for statistical capacity. Because of this lack of consensus, and because of the political power of the SDGs, for many countries the ability to monitor the SDGs themselves becomes a crucial measure of ‘statistical capacity’. This has in fact been proposed by some agencies as the way to measure SDG Target 17.19: ‘By 2030, build on existing initiatives to develop measurements of progress on sustainable development that complement gross domestic product, and support statistical capacity-building in developing countries’. In this way, International Organisations also direct the statistical capacity through technical and financial assistance that they provide towards supporting the SDG monitoring system itself, with the goal of harmonising and standardising official statistics across the world to make progress universally comparable. For example, the Joint Development Account programme on statistics and data—launched in 2016 with the support of the World Bank, the United Nations Population Fund, the United Nations Development Programme and the European Union, as well as all ten UN agencies of the Account—has the aim of ‘strengthening the statistical capacity of developing countries to measure, monitor and report on the implementation of the Sustainable Development Goals, and on progress with regard to their targets and indicators’ (UNGA, 2019, p. 15). This recursive quality of measuring statistical capacity development then demands practices of harmonisation on the level of the entire SDG monitoring framework.

To conclude this section, as International Organisations have tried to create it as a development priority in its own right, statistical capacity development has quickly become a ‘chaotic’ and highly fragmented field: global goals and their monitoring become more and more central to global progress, in addition to the rise of the use of non-traditional sources of data for monitoring policy goals. As an instrument for influencing national-level policy, as well as a global instrument informed by national priorities, the SDGs’ monitoring framework requires increased and sustained financial and technical support that is currently lacking in some countries and geographic regions. For example, according to a representative from the UN Economic Commission for Africa at the 11th Meeting of the IAEG-SDGs in November 2020, there are 52 of the SDGs’ 231 unique indicators on which no African country is currently reporting (Ilboudo, 2020). Therefore, like the MDGs before them, although on a different scale and with an explicit attempt at the participatory co-production of global public policy, the SDGs are shaping what is important to measure, and what policies national governments prioritise in the process.

5 Conclusion

This chapter explored the process of harmonisation as central in establishing the building blocks for the epistemic infrastructure of the SDGs. As we argued in the Introduction, data and indicators are the key material manifestations of these structures—and yet, due to the high fragmentation of global governance, IOs continue to be central actors that have to coordinate processes of harmonising these data, as well as support the development of the capacity of national statistical systems so that they are able to continue to produce them.

Indeed, at the start of the twenty-first century, the concept of ‘harmonisation’ was introduced as a means to address the problem of the outsized power of donor entities—both bilateral funding organisations from the Global North and UN agencies—in directing global public policy. ‘Harmonisation’ in this context was used to highlight the problem to dual streams of policymaking—those of the ‘global’, driven by these powerful entities, and those of the ‘national’ in countries that receive development aid. Statistical capacity development was meant to assert the importance of country-driven (rather than donor-led) development, in the same way that the World Bank’s Heavily Indebted Poor Country (HIPC) Initiative was meant to put countries in the driver’s seat of their own development plans.

Although the focus in this chapter was specifically on the practice of harmonisation and the development of statistical capacity, in interviews with members of the global statistics community as well as in official documents of key IOs, both ‘harmonisation’ and statistical capacity development were not used exclusively to describe how data could be made comparable. They were also used to construct powerful and often persuasive narratives of the promotion of country-led development agendas, reaffirming the need for the alignment of donor priorities to countries’ priorities, and facilitating an agenda that would be more palatable to the Global South, the participation and approval of which was now more necessary than ever before. It is to the production of these narratives that the next chapter will turn.