Abstract
The two cross-national indices with gender-differentiated data introduced by the UNDP in 1995, as well as several other such indices developed subsequently, are an important resource for researchers and policy makers interested in gender disparities. Yet questions remain regarding how these indices should be interpreted and how valid they are. Relying on a framework that synthesizes key guidelines concerning the methodology of measurement, this article offers an assessment of indices currently used to study gender disparities on a global scale and sheds light on these unresolved questions. We answer two questions—what do these indices with gender-differentiated data actually measure? and, how valid are these indices?—and discuss the implications of our assessment for users and producers of gender indices.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
For an overview of resources on gender-differentiated data, including the production of data, the actors involved in the dissemination of data, and statistical databases, see UNECLAC (2002).
This discussion has led to the proposal of multiple gender index, in addition to the two UNDP indices: the Gender Equality Index (White 1997), the Gender Inequality Index (Forsythe et al. 1998), the Relative Status of Women index (Dijkstra and Hanmer 2000), the Standardized Index of Gender Equality (Dijkstra 2002), the European Union Gender Equality Index (Plantenga et al. 2003), the African Gender and Development Index (UNECA 2004), the Gender Equity Index (Social Watch 2004), the Social Institutions and Gender Index (Jütting et al. 2006, 2008; Branisa et al. 2009), the Gender Equality in Education Index (Unterhalter 2006), the Global Gender Gap Index (Hausmann et al. 2007b), and the Multidimensional Gender Equality Index (Permanyer 2008). This discussion also includes various assessments of gender indices (Bardhan and Klasen 1999; Dijkstra 2002, 2006; Charmes and Wieringa 2003; Chant 2006; Klasen 2006a, b, c; Shuller 2006; Jütting et al. 2008; Permanyer 2010).
Though many other indices have been proposed, not all are equally relevant to the current assessment. Some of these indices are presented as purely theoretical exercises. Others constitute merely a recalculation of the data used in computing the UNDP’s two pioneering indices, the Gender-related Development Index and Gender Empowerment Measure. And yet others present data that have become dated or cover only a small number of countries. All these exercises are interesting and useful, and they form part of the broad debate about gender measures. But a core challenge in the development of suitable gender indices concerns the practical challenge of developing data about sometimes-hard–to-measure concepts across a large number of countries in a sustainable manner.
In this assessment, we rely on texts about the methodology underlying each index as well as the data from which indicators and, thence, indices are constructed. On the methodology used to construct each index, we consulted Anand and Sen (2003a, b), Branisa et al. (2009), Hausmann et al. (2007b), Jütting et al. (2006, 2008), Porter and Schwab (2008), Social Watch (2005, 2008a: 68–79, b) and UNDP (1995, 2007: 225–227, 358–360). The data on each index are generally available in reports or the websites of each organization.
In making choices about indicators, it is important to avoid a mismatch between the theoretical and operational definition of a concept. The use of multiple indicators to measure the same conceptual dimension, that is, indicators that are not mutually exclusive, can be justified as a way to produce a more-robust measure. But if the goal is to produce an index that measures many conceptual dimensions, and an assumption is made that each indicator measures a distinct part of a conceptual dimension, the use of multiple indicators leads to the problem of redundancy and double counting in the aggregation process. The concern about extraneous indicators is that they distort the meaning of the measure and cloud its interpretation.
The difference between the GDI and the HDI is that the GDI differentiates between genders and uses gender inequalities to adjust the HDI scores downwards (UNDP 2007).
The one problem of the HDI that the GDI partially skirts, through its focus on gender, is the critique that the HDI does not address distributional issues.
The main conceptual difference between these two indices being that the Social Watch index dropped the dimension focused on health while the World Economic Forum index retained it.
Indeed, though it is probably safe to say that all five indices avoid the introduction of extraneous elements that are not part of the concept’s meaning, it is not possible to assess with certainty whether the conceptual dimensions the GEM, GEI and GGGI do include are jointly exhaustive of the meaning of their overarching concepts.
Capitalization is used to distinguish references to the conceptual dimensions and indicators of the indices under review from other uses of the same words.
The simultaneous use of the Legislators, Senior Officials and Managers and the Parliamentary Seats indicators in the GEM and GGGI, and the GEI, also leads to a problem of partial redundancy.
This problem of “indicator creep” is not unique to indicators that are selected in part because of the ready availability of data. For example, the GGGI’s indicator Heads of State uses information about the past 50 years to assess the present, counter to the approach taken for all the other GGGI indicators, and hence introduces an extraneous element into the index.
In addition, the World Economic Forum’s GGGI includes a scaling choice with regard to the Heads of State indicator, which is measured with a simple count of the number of years in the past 50 during which a country has had a male or female president or prime minister.
This problem also affects the GGGI’s indicator Wage Equality for Similar Work, which is based on a question about women’s earnings relative to men (Porter and Schwab 2008: question 9.13), and thus excludes a priori the possibility of collecting data that might show that women earn more than men.
Some concerns about the reliability and comparability of many of the data sets used in these indices have been expressed by Srinivasan (1994). Moreover, some blatant errors have been introduced by those responsible for generating gender indices. For example, with regard to the indicator Positions as Legislators, Senior Officials and Managers (female/male ratio), the World Economic Forum data point to sudden changes, from 0.27 to 0.48 in Italy between the 2007 and 2008 report, from 0.09 to 0.30 for Bangladesh from 2006 to 2007, and from 0.08 to 0.58 for France from 2006 to 2007 reports! (Hausmann et al. 2006: 37, 64; 2007b: 42, 72, 87; 2008: 94). These changes are, on the face of things, incredible and indeed the data for the earlier year differ from those reported by the World Economic Forum’s own data source, the International Labor Organization’s LABORISTA data site. In other words, concerns about data reliability cannot be put to rest.
The only exceptions concern the Life Expectancy indicator in the GDI, the Healthy Life Expectancy indicator in the GGGI, and the Number of Births indicator in the GGGI. The two UNDP indices and the GEI also make an adjustment for the share of women and men in the population.
In the 2005 version of the GGGI, deviations from the standard of parity favoring women were given the same weight as deviations favoring men, but interpreted not as an indicator of inequality but rather as a measure of women’s empowerment or of “women’s supremacy over men” (Hausmann et al. 2007a: 22).
Because the UNDP gender indices are concerned with absolute levels, and not all the indicators use the same metrics, the two UNDP gender indices rely on the rescaling of some indicators. Specifically, the GDI’s Life Expectancy and Earned Income indicators and the GEM’s Earned Income indicator are normalized using the same reasonable procedures as in the HDI, that is, the notion of goalposts with minimum and maximum levels. In addition, so as to capture the idea of diminishing returns, the GDI’s Earned Income indicator is log transformed.
In the case of the GEM, however, only the Earned Income indicator actually contains information about absolute levels of attainment.
For a critique of this weighting scheme, see Bardhan and Klasen (1999: 987–90, 994).
Logically speaking, given Social Watch’s commitment to developing a relative measure, they would in effect have to say that a country in which men and women hold the same number of seats in parliament is indistinguishable from, to give a real example, a country in which the military has taken power and decided to rule without a parliament.
In contrast, the use of standards of parity that diverge from a 50/50 ratio for the Healthy Life Expectancy and Number of Births indicators draws on considerable research (Hausmann et al. 2007a: 5).
Another problem, as indicated above, is that the SIGI’s indicator scales are mainly ordinal scales; yet the normalization and aggregation of these indicators is treated as entirely trouble-free. Some of the problems with the SIGI may be due to its creation in two stages, by two different teams. Specifically, Jütting et al. (2006, 2008) initially created a data set on the indicators and later Branisa et al. (2009) worked on an aggregation scheme.
Even though this weighting choice is shown to have a large potential impact (Anand and Sen 2003b: 216), no results of an empirical test gauging the impact of picking different weighting are reported.
As an example, the UNDP’s GEM is the combination of the values of the Professional and Technical Positions indicator and the Positions as Legislators, Senior Officials and Managers indicator; and the combination of the values of the index’s three conceptual dimensions: Power over Economic Resources, Economic Participation and Decision-making, and Political Participation and Decision-making. When a conceptual dimension is measured with only one indicator, as with the GEM’s Political Participation and Decision-making dimension, no aggregation of indicators is necessary and the value of the relevant indicators—in this case the Parliamentary Seats indicator—is the value of the conceptual dimension.
The Literacy Rate indicator is assigned double the weight of the combined three enrollment indicators.
As a result of this procedure, the maximum difference in weight between indicators is 2.4. For the weights assigned to each indicator, see Hausmann et al. (2007a: 6).
At this stage, each subindex is rescaled by assigning the value 0 to the country with the best performance and 1 to the country with the worst performance.
These index developers essentially follow Babbie’s (1995: 171) advice that “items [should] be weighted equally unless there are compelling reasons for differential weighting. That is, the burden of proof should be on differential weighting; equal weighting should be the norm.” Indeed, in the one exception to the equal-weighting option among theory-driven weighting—for the GDI’s indicators of the Knowledge conceptual dimension—the choice of weights is explicitly justified on the basis of the distinction between stock and flow indicators (Jahan 2003: 156). But this is questionable methodological advice. Moreover, there are well-thought-out arguments against the default option of assigning equal weights. See, for example, Bardhan and Klasen (1999: 987–90, 994) on the GDI’s weighting scheme.
A related choice that affects the weights of indicators is the selection of extraneous and redundant indicators (on these indicators, see Sect. 2.2).
To be consistent, given the justification offered by the creators of the GGGI, the empirically derived weights would be recalculated each year, using the data on all the years that were being compared.
In addition to the problems mentioned in the text, several indices do not use the same weights for some indicators in all countries. Social Watch includes a country in the GEI even though data are missing on 4 of its 10 indicators, while the World Economic Forum includes a country in the GGGI even if 2 of the 14 indicators are missing. In those cases, the indicators that are used are reweighted—the weights of the missing indicators are redistributed across the available indicators—thus introducing a problem of comparability into the data.
We found two partial exceptions. One concerns the GDI, in that the UNDP states that tests were conducted to assess the impact of assigning different weights to the HDI’s indicators (Ul Haq 2003: 128). However, no results are reported and similar tests appear not to have been conducted with the GDI indicators proper. Another exception concerns SIGI (Branisa et al. 2009: 6); but here again no results are reported.
The replicability of the aggregation process is ensured because the aggregation procedures are explicitly articulated and the underlying data on each indicator are made public by the index producers or are otherwise publicly available. However, when comparing the data used by the index producers to the data from the sources cited by the index producers, we have found numerous discrepancies.
Given this result, one wonders whether it would be wiser if the data driven weightings were simply dropped by these two indices and a simpler and easier to communicate choice were adopted.
The GDI’s general conception of gender inequality could be seen as making it hard to understand, because it is not directly apparent whether inequalities favor men or women. But this problem of interpretation is resolved through an analysis of disaggregate-level data.
Though Social Watch (2005: 77) suggests that the GEI be used to draw “conclusions about critical deficiencies in what women are able or allowed to do,” the name of their index—the Gender Equity Index—suggests that it is a measure of gender equity. Likewise, the name of the World Economic Forum’s index, the Global Gender Gap Index, is only somewhat accurate—gender gaps can go in either direction—and the creators of this index confusingly suggest that they are measuring gender equality broadly speaking when they prominently state that the GGGI is “a tool for … tracking global gender-based inequalities” (Hausmann et al. 2007a: 3).
Going beyond the methodological issues we addressed in this article, we have serious reservations about using data on these indices as time series. Yearly data on all indicators are simply not available and in some cases the data on indicators used to calculate the index for a certain year can span up to 10 years. Nonetheless, the producers of these indices sometimes use their data to discuss address “evolution and trends,” to “track progress over time” and to identify “countries progressing and regressing” (Hausmann et al. 2007a: 3, Social Watch 2005: 73, 2008a: 68–85). We would suggest that the data we currently have do not allow us to talk with confidence about trends over time.
References
Abdullah, R. (2000). A framework on indicators for action on women’s health needs and rights after Beijing. Kuala Lumpur: Asian-Pacific Resource and Research Centre for Women Malaysia.
Anand, S., & Sen, A. (2003a). Human development index: Methodology and measurement. In S. Fukuda-Parr & A. K. S. Kumar (Eds.), Readings in human development: Concepts, measures, and policies for a development (pp. 138–51). Oxford: Oxford University Press. [Originally published as Human Development Report Office Occasional Paper No 12 (New York: UNDP, 1994)].
Anand, S., & Sen, A. (2003b). Gender inequality in human development: Theories and measurement. In S. Fukuda-Parr & A. K. S. Kumar (Eds.), Readings in human development: Concepts, measures, and policies for a development (pp. 210–227) Oxford: Oxford University Press. [Originally published as Human Development Report Office Occasional Paper No 19. New York: UNDP, 1995].
Babbie, E. (1995). The practice of social research. Belmont, CA: Wadsworth Publishing Company.
Bardhan, K., & Klasen, S. (1999). UNDP’s gender-related indices: A critical review. World Development, 27(6), 985–1010.
Booysen, F. (2002). An overview and evaluation of composite indices of development. Social Indicators Research, 59(2), 115–151.
Branisa, B., Klasen, S., & Ziegler, M. (2009). Background paper: The construction of the social institutions and gender index (SIGI). University of Goettingen, Germany, March 6, 2009. Available at: http://www.oecd.org/dataoecd/49/19/42295804.pdf.
Breitenbach, E., & Galligan, Y. (2006). Measuring gender equality: Reflecting on experiences and challenges in the UK and Ireland. Policy and Politics, 34(4), 597–614.
Chant, S. (2006). Re-thinking the ‘Feminization of Poverty’ in relation to aggregate gender indices. Journal of Human Development, 7(2), 201–220.
Charmes, J., & Wieringa, S. (2003). Measuring women’s empowerment: An assessment of the gender-related development index and the gender empowerment measure. Journal of Human Development, 4(3), 419–435.
Dijkstra, A. G. (2002). Revisiting UNDP’s GDI and GEM: Towards an alternative. Social Indicators Research, 57(3), 301–338.
Dijkstra, A. G. (2006). Towards a fresh start in measuring gender equality: A contribution to the debate. Journal of Human Development, 7(2), 275–283.
Dijkstra, A. G., & Hanmer, L. (2000). Measuring socio-economic inequality: Towards an alternative to the UNDP gender-related development index. Feminist Economics, 6(2), 41–75.
Forsythe, N., Korzeniewick, R. P., & Durrant, V. (1998). Gender inequalities, economic growth and structural adjustment: A longitudinal evaluation. In Paper presented to XXI conference of the Latin American studies association (LASA), Washington, September 24–26, 1998.
Fukuda-Parr, S. (2003). Rescuing the human development concept from the HDI: Reflections on a New Agenda. In S. Fukuda-Parr & A. K. S. Kumar (Eds.), Readings in human development: Concepts, measures, and policies for a development (pp. 117–124). Oxford: Oxford University Press.
Hausmann, R., Tyson, L. D., & Zahidi, S. (2006). The global gender gap report 2006. Switzerland: World Economic Forum.
Hausmann, R., Tyson, L. D., & Zahidi, S. (2007a). The global gender gap index 2007. In R. Hausmann, L. D. Tyson, & S. Zahidi (Eds.), The global gender gap report 2007 (pp. 3–27). Switzerland: World Economic Forum.
Hausmann, R., Tyson, L. D., & Zahidi, S. (2007b). The global gender gap report 2007. Switzerland: World Economic Forum.
Hausmann, R., Tyson, L. D., & Zahidi, S. (2008). The global gender gap report 2008. Switzerland: World Economic Forum.
Jahan, S. (2003). Evolution of the human development index. In S. Fukuda-Parr & A. K. S. Kumar (Eds.), Readings in human development: Concepts, measures, and policies for a development (pp. 152–163). Oxford: Oxford University Press.
Jütting, J. P., Morrisson, C., Dayton-Johnson, J., & Drechsler, D. (2006). Measuring gender (In)Equality: Introducing the gender, institutions and development data base (GID) (Paris: OECD Development Centre Working Papers 247, OECD Development Centre).
Jütting, J. P., Morrisson, C., Dayton-Johnson, J., & Drechsler, D. (2008). Measuring gender (In)Equality: The OECD gender, institutions and development data base. Journal of Human Development, 9(1), 65–86.
Kelly, L., Kennedy, L., & Horvath, M. (2008). The next step: Developing transnational indicators on violence against women, addendum to the Report of the special rapporteur on violence against women, its causes and consequences, Yakin Ertürk, UN General Assembly, A/HRC/7/6/Add.5, February 25, 2008. New York: UN.
Klasen, S. (2006a). UNDP’s gender-related measures: Some conceptual problems and possible solutions. Journal of Human Development, 7(2), 243–274.
Klasen, S. (2006b). Gender-related indicators of well-being. In M. McGillivray & A. Shorrocks (Eds.), Measuring human well-being (pp. 167–192). London: Palgrave-Macmillan.
Klasen, S. (Ed.). (2006c). Special issue: Revisiting the gender-related development index (GDI) and gender empowerment measure (GEM). Journal of Human Development 7(2).
Morrisson, C., & Jütting, J. P. (2005). Women’s discrimination in developing countries: A new data set for better policies. World Development, 33(7), 1065–1081.
Nussbaum, M. C. (2000). Women and human development. Cambridge: Cambridge University Press.
OECD Development Centre. (2009a). Country profiles: Gender equality and social indicators. Available at: http://genderindex.org/countries.
OECD Development Centre. (2009b). Coding of the social institutions variables. Available at: http://genderindex.org/sites/default/files/GID_variables.pdf.
Permanyer, I. (2008). On the measurement of gender equality and gender-related development levels. Journal of Human Development, 9(1), 87–108.
Permanyer, I. (2010). The measurement of multidimensional gender inequality: Continuing the debate. Social Indicators Research, 95(2), 181–198.
Plantenga, J., Figueiredo, H., Remery, C., & Smith, M. (2003). Towards an EU gender equality index. A feasibility study commissioned by and presented to the European Commission, Utrecht School of Economics, Utrecht. December 2003.
Porter, M., & Schwab, K. (2008). Executive opinion survey 2008. Available at: http://www.kadin-indonesia.or.id/enm/images/dokumen/KADIN-117-2573-29022008.pdf.
Sen, A. (1999). Development as freedom. Oxford: Oxford University Press.
Shuller, D. (2006). The uses and misuses of the gender-related development index and gender empowerment measure: A review of the literature. Journal of Human Development, 7(2), 161–181.
Social Watch. (2004). Social Watch annual report 2004. Montevideo: Instituto del Tercer Mundo.
Social Watch. (2005). No country treats its women the same as its men. The gender equity index: A new perspective. In Social Watch, Social Watch Annual Report 2005 (pp. 73–77). Montevideo: Instituto del Tercer Mundo.
Social Watch. (2008a). Social Watch annual report 2008. Montevideo: Instituto del Tercer Mundo.
Social Watch. (2008b). Technical notes: The construction of the GEI. Available at: http://www.socialwatch.org/en/avancesyRetrocesos/IEG_2008/tablas/technicalNotes.html.
Srinivasan, T. N. (1994). Data bases for development analysis: An overview. Journal of Development Economics, 44(1), 3–27.
Ul Haq, M. (2003). The birth of the human development index. In S. Fukuda-Parr, & A. K. S. Kumar (Eds.), Readings in human development: Concepts, measures, and policies for a development (pp. 127–37). Oxford: Oxford University Press.
UNDP. (1990). Defining and measuring human development. n UNDP, Human Development Report 1990 (pp. 9–16). New York: Oxford University Press.
UNDP. (1995). Measuring gender inequality. In UNDP, Human development report 1995. Gender and human development (pp. 72–86). New York: UNDP and Oxford University Press.
UNDP. (2006). Measuring democratic governance: A framework for selecting pro-poor and gender sensitive indicators. Oslo: UNDP Oslo Governance Centre.
UNDP. (2007). Calculating the human development indices. In UNDP, Human Development Report 2007/2008. Fighting climate change: human solidarity in a divided world (pp. 355–61). New York: UNDP.
UNECA (United Nations Economic Commission for Africa). (2004). The African gender and development index. Addis Ababa: Economic Commission for Africa.
UNECE (United Nations Economic Commission for Europe). (2001). Final Report. ECE/UNDP task force meeting on a regional gender web-site. Geneva: UNECE, Statistical Division.
UNECLAC (Economic Commission for Latin America and the Caribbean). (1999). Indicadores de Género para el Seguimiento y la Evaluación del Programa de Acción Regional para las Mujeres de América Latina y el Caribe, 1995–2001, y la Plataforma de Acción de Beijing (Santiago, Chile: LC/L.1186).
UNECLAC (Economic Commission for Latin America and the Caribbean). (2002). Tools and indicators for gender impact analysis, monitoring and evaluation. New York, Inter-Agency Network on Women and Gender Equality, IANWGE/2002/6, February 2002.
UNESCAP (Economic and Social Commission for Asia and the Pacific). (2003). Gender indicators for monitoring the implementation of the Beijing platform for action in the Asia-Pacific Region. New York: United Nations.
UNIFEM (United Nations Development Fund for Women). (2002). Innovations in measuring and monitoring. In UNIFEM, Progress of the world’s women 2002 Volume 2: Gender equality and the millennium development goals (pp. 55–61). New York: UNIFEM.
UNRISD (United Nations Research Institute for Social Development). (2005). Gender equality: Striving for justice in an unequal world. Geneva: UNRISD.
Unterhalter, E. (2006). Measuring gender inequality in education in South Asia. Kathmandu: The United Nations Children’s Fund [UNICEF], Regional Office for South Asia; and United Nations Girls’ Education Initiative [UNGEI].
Walby, S. (2007). Indicators to measure violence against women. Working paper no 1, prepared for the Expert group meeting on indicators to measure violence against women, organized by the United Nations Division for the Advancement of Women, UNECE and UNSD, in collaboration with ECLAC, ESCWA, ESCAP, and ECA, Geneva, Switzerland, October 8–10, 2007.
White, H. (1997). Patterns of gender discrimination: an examination of the UNDP’s gender development index. The Hague: Institute of Social Studies.
Yinger, N., et al. (2002). A framework to identify gender indicators for reproductive health and nutrition programming. Washington, DC: USAID, Interagency Gender Working Group Subcommittee on Research and Indicators.
Acknowledgments
We thank Robert Harris for research assistance, and Jonathan Kulick and Svend-Erik Skaaning for comments.
Author information
Authors and Affiliations
Corresponding author
Appendix
Rights and permissions
About this article
Cite this article
Hawken, A., Munck, G.L. Cross-National Indices with Gender-Differentiated Data: What Do They Measure? How Valid Are They?. Soc Indic Res 111, 801–838 (2013). https://doi.org/10.1007/s11205-012-0035-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11205-012-0035-7