Skip to main content
SpringerLink
Log in

Aggregating Political Dimensions: Of the Feasibility of Political Indicators

  • Published:
Social Indicators Research Aims and scope Submit manuscript

Abstract

Political indicators are widely used in academic writing and decision making, but remain controversial. This paper discusses the problems related to the aggregation functions they use. Almost always, political indicators are aggregated by weighted averages or summations. The use of such functions is based on untenable assumptions (existence of homogeneous substitution rates, total compensation, and strict monotonicity). We show through concrete examples how these hidden assumptions are likely to produce results that are basically an artifact of ad hoc decisions, which additionally contradict very fundamental notions common to all credible political theories. We suggest, also through example, that some—necessarily partial—solutions are possible.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. See for example http://www.carleton.ca/cifp/gdp_indicator_descriptions.ht. From now on, we will use somewhat inexactly expressions as “indicator”, “index”, and “measure” as synonymous.

  2. http://www.gaportal.org/global-indicators/corruption-perceptions-index.

  3. This includes also indexes that apparently constitute exceptions. For example, the Worldwide Governance Indicator uses an aggregation function which is more sophisticated than others. By the Unobserved Components Method, the WGI tries to isolate the signal from the noise sent by each variable. However, the outcome of this operation, as the coordinators of the index explicitly acknowledge, “is simply a weighted average of the rescaled scores for each country” (Kaufmann et al. 2010, p. 10). Something similar can be said about Polity IV (2010), see below.

  4. It subtracts autocracy from democracy, and adds the variables of democracy and autocracy to obtain the respective scores. However, below this operation, there is a non-linear transformation of the scales, sending to zero the score of categories that are below a cut-off point. These transformations have their own issues, but we will concentrate here on the “high level” operation over the scores, not in the way the scores themselves were produced.

  5. Do not confuse with BTI Management Index, at http://www.bertelsmann-transformation-index.de/en/bti/ranking/management-index/.

  6. http://www.transparency.org.

  7. http://www.freedomhouse.org.

  8. Adding numerical tags from ordinal scales is far from uncontroversial, but we will drop the issue here.

  9. In economy typically individuals are taking decisions on their behalf. In politics they are habitually discussing rules that will apply for everybody, or for large chunks of the population.

  10. For analogous exercises, see the Bouyssou et al. (2000) discussion of the Human Development Index, and other examples, p. 57. Anybody who is familiar in the way PIs are used in regressions will see that this kind of transformation is everything but extraordinary.

  11. In that they use basically the same method, averages or summations, but plug in different parameters.

  12. Rather atypically, and strangely, the FSI does not deal explicitly with variables. Each box has a mark, based on a soft assessment of criteria that “are neither exclusive nor exhaustive”. http://www.fundforpeace.org/web/index.php?option=com_content&task=view&id=452&Itemid=900.

  13. We took subsets of size two because there are more of them than subsets of size four, three or one. In each of these choices the exercise we present here is much easier to perform.

  14. Additionally, we found in the process of correcting the paper that there were many forms of obtaining the same results. That is, several other arrays of weights would produce a similar outcome.

  15. The domain of PIs is bounded above and below.

  16. This definition can easily be generalized for g with different domains in each of its variables.

  17. Once the scores of the variables have been fixed. See note 4 relative to Polity.

  18. In Beliakov et al. (2007) weights are attributed ad hoc. For a more general example in this vein, see Gutierrez et al. (2010).

  19. Note besides that, since scales are sometimes different and the aggregation is generally produced through a two-step procedure, in practice the majority of indicators allot different weights to their variables. The FSI at least gives the same weight to all its boxes.

References

  • Ball, P. (1996). Who did what to whom? Planning and implementing a large scale human. American Association for the Advancement of Science. http://shr.aaas.org/Ball/contents.html, Accessed June 1, 2010.

  • Becker, G. (1983). A theory of competition among pressure groups for political influence. The Quarterly Journal of Economics, 98, 371–400.

    Article  Google Scholar 

  • Beliakov, G., Pradera, A., & Calvo, T. (2007). Aggregation functions. A guide for practitioners. New York: Springer.

    Google Scholar 

  • Bertocchi, G., Guerzoni, A. (2010). Growth, history or institutions? What explains state fragility in Sub-Saharan Africa. IZA DP No. 4817.

  • Bollen, K., & Paxton, P. (2000). Subjective measures of liberal democracy. Comparative Political Studies, 33(1), 58–86.

    Article  Google Scholar 

  • Bouyssou, D., Marchant, T., Pirlot, M., Perny, P., Tsoukias, A., & Vincke, P.H. (2000). Evaluation and desicion models: A critial perspective. Massachusetts: Kluwer Academic Publisher.

    Google Scholar 

  • Bouyssou, D., & Perny, P. (1992). Ranking methods for valued preference relations: A characterization of a method based on leaving and entering flows. European Journal of Operational Research, 61, 186–194.

    Article  Google Scholar 

  • Carment, D. , Prest, S., & Samy, Y. (2008). Determinants of state fragility and implications for aid allocation: An assessment based on the country indicators for foreign policy project. Conflict Management and Peace Science, 25, 349–373.

    Article  Google Scholar 

  • Chauvet, L., & Collier, P. (2004). Development effectiveness in fragile states: Spillovers and turnarounds. Oxford: Centre for the Study of African Economies, Department of Economics, Oxford University.

  • Cohen, W., Schapire, R., & Singer, Y. (1999). Learning to order things. In Advances in Neural Information Processing Systems 11: Proceedings of NIPS’98. MIT Press.

  • Collier, D., Sekhon, J., & Stark, P. (2010). Inference and shoe leather. In: F. David (Eds.), Statistical models and causal inference. A dialogue with the social sciences. Cambridge: Cambridge University Press.

    Google Scholar 

  • Di John, J. (2008). Conceptualising the causes and consequences of failed states: A critial review of the literature. Crisis States Working Papers Series, 2, 1–51.

    Google Scholar 

  • Elster, J. (1993). Political psychology. Cambridge: Cambridge University Press.

    Google Scholar 

  • Fabra, J., & Ziaja, S. (2009). Users’ guide on measuring fragility. Germany: German Development Institute (DIE), United Nation Development Programme (UNDP).

  • Fearon, J., & Laitin, D. (2003). Ethnicity, insurgency, and civil war. American Political Science Review, 97(1), 75–90.

    Article  Google Scholar 

  • Freedman, D. (2005). Statistical models: Theory and practice. New York: Cambridge University Press.

    Book  Google Scholar 

  • Freedman, D. (2010). Statistical models and causal inference. A dialogue with the social sciences. New York: Cambridge University Press.

    Google Scholar 

  • Fund for Peace. (2007). Failed States Index. http://www.fundforpeace.org/web/index.php?option=com_content&task=view&id=229&Itemid=366. Accessed November 19, 2010.

  • Grabisch, M., Marichal, J. L., Mesiar, R., & Pap, E. (2009). Aggregation functions. Cambridge: Cambridge University Press.

    Google Scholar 

  • Green, D., & Shapiro, I. (1994). Pathologies of rational choice theory: A critique of applications in political science. New Haven: Yale U. Press.

    Google Scholar 

  • Gutierrez, F. (2010). Evaluating state performance. A critical view of state failure and fragility indexes. European Journal of Development Research, 3, 1–23.

    Google Scholar 

  • Gutierrez, F., & Argoty, C. (2010). Order preserving functions from finite posets to \({\mathbb{R}}\): How many are there? In F. Gutierrez, D. Buitrago, A. Gonzales & C. Lozano (Eds.), Measuring poor state performance: Problems, perspectives and paths ahead. London: London School of Economics.

  • Gutierrez, F., Buitrago, D., Gonzales, A., & Lozano, C. (2010). Measuring poor state performance: problems, perspectives and paths ahead. London: London School of Economics.

    Google Scholar 

  • Gutierrez, F., & Gonzalez, A. (2009). Force and Ambiguity: Evaluating sources for cross-national research—the case of military interventions. Crisis States Working Papers Series, 50(2), 1–32.

    Google Scholar 

  • Howard, T. O. (2008). Revisiting state failure: Developing a causal model of state failure based upon theoretical insight. Civil Wars, 10(2), 125–146.

    Article  Google Scholar 

  • IDA. (2008). Additions to IDA Resources: Fifteenth Replenishment. http://go.worldbank.org/U94WRYNMQ. Accessed February 3, 2010.

  • Kaufmann, D., Kraay, A., & Mastruzzi, M. (2010). The worldwide governance indicators. Methodology and analytical issues. Development Research Group. Macroeconomics and Growth Team, The World Bank. Policy Research Working Paper 5430.

  • King, G., Keohane, R. O., & Verba, S. (1994). Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Labreuche, C., & Grabisch, M. (2003). The Choquet integral for the aggregation of interval scales in multicriteria decision making. Fuzzy Sets and Systems, 137, 11–26.

    Article  Google Scholar 

  • Lavrač, N., Flach, P., & Zupan, B. (1999). Rule evaluation measures: A unifying view. In Proceedings of the 9th International Workshop on Inductive Logic Programming, Springer-Verlag, 174–185.

  • Lootsma, F. (1997). Fuzzy logic for planning and decision making. London: Kluwer academic publishers.

    Google Scholar 

  • McMahon, E., & Kornheiser, E. (2010). Assessing the assessors: Correlating democracy methodologies. Social Indicators Research, 97(2), 269–277.

    Article  Google Scholar 

  • Munck, G., & Verkuilen, J. (2002). Conceptualizing and measuring democracy evaluating alternative indice. Comparative political studies, 1(35), 5–34.

    Google Scholar 

  • Munda, G. (2005). “Measuring sustainability”: A multi-criterion framework. Environment, Development and Sustainability, 7, 117–134.

    Article  Google Scholar 

  • Munda, G., & Nardo M. (2005). Non-compensatory composite indicators for ranking countries: A defensible setting. Technical Report EUR 21833 EN, European Commission.

  • Nardo, M., Sasaina, M., Saltelli, A., & Tarantola, S. (2005). Handbook on constructing composite indicators: Methodology and user guide. OECD Statistics Working Paper, 13, 1–55.

  • Noda, E., & Freitas, A. A. (2006). Data mining and knowledge discovery approaches based on rule induction techniques. In E. Triantaphyllou & G. Felici (Eds.), Massive computing series (pp. 395–432). Heidelberg, Germany: Springer.

  • Pǎun, G. (1983). An impossibility theorem for indicators aggregation. Fuzzy Sets and Systems, 9, 205–210.

    Article  Google Scholar 

  • Pele, D. T. (2009). About the impossibility theorem for indicators aggregation. Journal of Applied Quantitative Methods, 4(1), 82–87.

    Google Scholar 

  • Polity IV. (2010). Annual Time-Series 1800–2009. http://www.systemicpeace.org/inscr/inscr.ht. Accessed August 10, 2010.

  • Przeworski, A. (2004). States and markets. A primer in political economy. New York: Cambridge University Press.

    Google Scholar 

  • Torra, V., & Narukawa, Y. (2004). On the interpretation of some fuzzy integrals. In Proceedings of the 1st International Conference on Modeling Decisions for Artificial Intelligence, Lecture Notes in Artificial Intelligence, vol. 3131, pp. 316–326.

  • Torra, V., & Narukawa, Y. (2006). The interpretation of fuzzy integrals and their application to fuzzy systems. International Journal of Approximate Reasoning, 41, 43–58.

    Article  Google Scholar 

  • UNDP. (2006). Indicators for human rights based approaches to development in UNDP programming: A users’ guide. United Nations Development Programme, Bureau for Development Policy, Democratic Governance Group. http://www.undp.org/oslocentre/docs06/HRBA%20indicators%20guide.pdf. Accessed February 13, 2011.

  • von Neumann, J., & Morgenstern, O. (1944). The theory of games and economic behavior. Princeton: Princeton University Press.

    Google Scholar 

  • Vreeland, J. R. (2008). The effect of political regime on civil war: Unpacking anocracy. Journal of Conflict Resolution, 52(3), 401–425.

    Article  Google Scholar 

  • Ziaja, S., & Fabra, J. (2010). State fragility indices: Potentials, messages and limitations. Briefing Paper, German Development Institute. http://www.die-gdi.de/CMS-Homepage/openwebcms3.nsf/(ynDK_contentByKey/ANES-86AFKY/$FILE/BP%2010.2010.pdf Accessed February 3, 2011.

  • Zintl, R. (2008). On the rationality and stability of a minimal consensus. In: J. Kühnelt (Ed.), Political legitimization without morality? (pp. 73–90). Netherlands: Springer.

    Chapter  Google Scholar 

  • Zupan, B., Bohanec, M., Bratko, I., Demsar, J. (1997). Machine learning by function decomposition. In: D. H. Fisher Jr. (Ed.), Proceedings Fourteenth International Conference on Machine Learning (ICML-97), pages 421–429, San Mateo, CA. Morgan Kaufmann.

Download references

Acknowledgments

The discussion and results reported in this article were obtained within the framework of the Crisis States Programme. We also acknowledge the support of Colciencias through the program “Jovenes Investigadores e Innovadores” and of the Instituto de Estudios Politicos y Relaciones Internacionales (IEPRI) at the Universidad Nacional de Colombia. We benefited from the great contributions of Camila Lozano and Carolina Acosta, and from the observations and criticisms of the journal’s anonymous referees.

Author information

Authors and Affiliations

  1. Instituto de Estudios Políticos y Relaciones Internacionales, Universidad Nacional de Colombia, Santa Fé de Bogotá, Colombia

    Francisco Gutiérrez Sanín

  2. Universidad Nacional de Colombia, Santa Fé de Bogotá, Colombia

    Diana Buitrago & Andrea González

Authors
  1. Francisco Gutiérrez Sanín
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diana Buitrago
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea González
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diana Buitrago.

Appendix: Building the WYWIWYG Exercise

Appendix: Building the WYWIWYG Exercise

The following is the list of indicators of the Failed States Index (Fund for Peace 2007):

I-1:

Mounting Demographic Pressures

I-2:

Massive Movement of Refugees or Internally Displaced Persons creating Complex Humanitarian Emergencies

I-3:

Legacy of Vengeance-Seeking Group Grievance or Group Paranoia

I-4:

Chronic and Sustained Human Flight

I-5:

Uneven Economic Development along Group Lines

I-6:

Sharp and/or Severe Economic Decline

I-7:

Criminalization and/or Delegitimization of the State

I-8:

Progressive Deterioration of Public Services

I-9:

Suspension or Arbitrary Application of the Rule of Law and Widespread Violation of Human Rights

I-10:

Security Apparatus Operates as a “State Within a State”

I-11:

Rise of Factionalized Elites

I-12:

Intervention of Other States or External Political Actors

We utilized the same aggregation function as the FSI:

$$ \sum_{i=1}^{12}w_i x_i $$

where the w i are the weights of Table 4 and the x i are the indicators. To find the perturbed weights for the successive versions of the FSI we used the method of minimal variance explained in Beliakov et al. (2007, p. 79).

Having as dependent variable the version of the index with different weights, we ran a multiple linear regression with the independent variables shown in Table 7.

Table 7 The independent variables of the WYWIWIG models
Full size table

The sample was taken for 75 countries that had information for all the variables for year 2007. Descriptive statistics of the independent variables are shown in Table 8.

Table 8 Descriptive statistics of the independent variables
Full size table

We checked the models for:

  • Normality, using the Shapiro–Wilk test, where a p value bigger than 0.05 indicates no violation.

  • Heteroscedaskicity by the Breusch–Pagan test, where a p value bigger than 0.05 indicates no violation.

  • Multicollinearity by the Variance Inflation Factor; the VIF values must remain less than 10

As seen in Table 4, all models pass all the tests. The p values of the variables in each model are shown in Table 9.

Table 9 p values of the main variables in each model
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gutiérrez Sanín, F., Buitrago, D. & González, A. Aggregating Political Dimensions: Of the Feasibility of Political Indicators. Soc Indic Res 110, 305–326 (2013). https://doi.org/10.1007/s11205-011-9932-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11205-011-9932-4

Keywords

Search

Navigation