Abstract
In 2011, the World Bank’s Independent Evaluation Group made its project ratings database public. With broad geographic and sectoral coverage, this database is a valuable resource for research on development effectiveness, what works and what does not. This chapter first provides an overview for scholars interested in evaluation but unfamiliar with the World Bank evaluation system. Next we examine whether geopolitical or institutional factors influence ratings. We focus on how projects are selected for performance assessments and what factors influence project ratings. We find evidence that bureaucratic factors influence selection and that one geopolitical variable—nonpermanent United Nations Security Council membership—impacts ratings. Nonetheless if researchers control for potential biases, World Bank project ratings can provide a valuable way to measure project quality in many applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In response NGO and donor pressure, the World Bank has gradually increased the amount of information it makes public. The “Access to Information” policy adopted in 2010 under World Bank President Robert Zoellick marked a major step forward, providing access to new information on projects and putting in place procedures to request information and to appeal when requests are initially denied.
- 2.
Outsider papers: Buntaine and Parks (2013), Dreher et al. (2013), Girod and Tobin (2016), Kilby (2015), Malik and Stone (2018), Michaelowa and Borrmann (2006), Sud and Olmstead-Rumsey (2012), and Winters (2014). Insider papers: Blum (2014), Bulman et al. (2017), Chauvet et al. (2010), Chauvet et al. (2017), Cruz and Keefer (2013), Deininger et al. (1998), Denizer et al. (2013), Dollar and Levin (2005), Dollar and Svensson (2000), Geli et al. (2014), Guillaumont and Laajaj (2006), Isham and Kaufmann (1999), Isham et al. (1997), Kaufmann and Wang (1995), Limodio (2011), Malesa and Silarszky (2005), Moll et al. (2015), Pohl and Mihaljek (1992), Ralston (2014) and Smets et al. (2013).
- 3.
The influence of geopolitics has been demonstrated at every stage of the project cycle (preparation: Kilby (2013b); approval (number of loans): Dreher et al. (2009), Kaja and Werker (2010); disbursement (speed): Kilby (2013a); disbursement (enforcement of conditionality): Kilby (2009); disbursement (electioneering): Kersting and Kilby (2016)) except at the evaluation stage. Most of the literature focuses on US informal influence over the World Bank, which is headquartered just two blocks from the White House and depends on the US for funding and leadership.
- 4.
For a detailed history of OED, see Grasso et al. (2003). In the text below we use IEG to refer to the evaluation department even before 2005.
- 5.
This reduction was driven by an expanding World Bank project portfolio but also increasing demands on IEG to generate other products at the country and sector levels (Grasso et al. 2003). More than 70% of IEG’s budget was devoted to PPARs in 1976 (Grasso et al. 2003, 169); the figure was less than 10% of IEG’s USD 34 million budget in 2011 when the time devoted to a PPAR averaged six staff weeks (IEG 2011, 38). PPAR coverage differs across activities, depending on “novelty” and importance. For example, IEG kept PPAR coverage of structural adjustment programs at 100% in the 1980s while it reduced coverage of investment projects.
- 6.
Calculated from 1590 IBRD/IDA-funded projects with overall outcome ratings measured as “Satisfactory” or “Unsatisfactory.” For our estimation sample, it is 17.6%.
- 7.
PPAR is the report (also called a Project Performance Audit Report or just Project Audit Report in early IEG documents); PAR is the database name for its associated ratings.
- 8.
In a few cases, a second PPAR rating is available. To allow such variation, we take the project’s last PCR/EVM/ES entry as its ICR rating and the project’s first PPAR entry as its PPAR rating.
- 9.
Despite some selection based on ICR ratings, the overall share of projects initially rated satisfactory does not vary much between those projects that get reevaluated and others: ICR ratings average 72% satisfactory for projects with no subsequent PPAR versus 78% for projects with a subsequent PPAR. Starting in the mid-1980s, IEG began “group audits” for sequences of projects in the same sector in a given country and “cluster audits” of similar projects in several neighboring countries (Grasso et al. 2003, 48–49). Selection of projects for a PPAR is ultimately the decision of IEG division chiefs but with input from staff (Grasso et al. 2003, 49).
- 10.
This pattern is apparent in IEG (2010); Appendix B reports that only 12% of projects with an economic rate of return above 10% were rating “moderately unsatisfactory” or lower. Denizer et al. (2013) argue that World Bank procedures promote applying relatively uniform standards to goal setting and evaluation.
- 11.
The Vocational Training and Technological Development Project in Uruguay was approved in 1978 and completed in 1986. Its PCR (equivalent to an ICR) was issued in 1988 and the project was included with two subsequent Uruguayan education projects in a 2006 PPAR. The longest interval in our estimation sample is 12.75 years.
- 12.
Countries with ICRs (# ICRs in parentheses) but no PPARs as of 9/30/2013 are: Angola (15), Cape Verde (22), Sao Tome and Principe (13), Tonga (5), the Bahamas (5), Grenada (9), St. Kitts and Nevis (5), St. Vincent and the Grenadines (6), Turkmenistan (3), West Bank and Gaza (40), Kosovo (20), Namibia (2), Montenegro (5), South Sudan (1).
- 13.
The hazard model explores how the time until PPAR depends on institutional, country, and project characteristics. This is a form of a duration model that allows for censoring. Specifically, the model treats cases without a PPAR as not yet having a PPAR, since in principle a project could be audited at any point in the future. Estimation results are reported in terms of a hazard ratio relative to the baseline so that a ratio greater than one indicates a higher likelihood (“greater risk”) of a PPAR/shorter time until a PPAR; a ratio less than one indicates a lower likelihood of a PPAR/longer time until a PPAR. Hazard ratios have the advantage of not depending on values of the other variables (just as slope terms in a linear model do not depend on where you evaluate them).
- 14.
The hazard model sample includes all years with available data whereas the PPAR rating sample below excludes FY2002 (which proves to be an outlier in that setting perhaps due to the 9/11 attacks). Hazard model results are largely unchanged if we also exclude FY2002 here.
- 15.
Tourism data are sparse so we average across available years for each country, resulting in a purely cross-sectional variable. The specification also includes log population so that the tourism coefficient would be the same if we measured tourist arrivals per capita.
- 16.
We add one before taking logs; we disregard the few negative values reported by the OECD since this is defined as a gross measure.
- 17.
Results are similar if we use current year membership only. However, the three-year period has a few advantages. It allows for the delay between the decision to carry out a PPAR and the release of the PPAR. It also simplifies treatment of cases where the country held the WBEB seat for only part of the calendar year (since WBEB membership follows the World Bank’s fiscal year).
- 18.
The Weibull distribution is a generalization of the exponential distribution and is commonly used in hazard models because it is more flexible than the exponential and results in constant hazard ratios. We select logit (which estimates the probability of a PPAR by the end date) rather than probit because logit odds ratios can easily be compared to hazard ratios.
- 19.
Results comparable for Gompertz, loglogistic, and lognormal distributions. A likelihood ratio test rejects the shape assumption of the exponential model in favor of Weibull though results are again similar. Estimates are similar with gamma distribution but standard errors are larger and a few variables (notably June ICR) fail to reach statistical significance.
- 20.
However, if we omit GDP per capita in this multivariate model, the IDA hazard ratio becomes significantly greater than one. The two variables are strongly correlated as only low-income countries qualify for IDA’s concessional credits. (This may also explain why, in the specifications of Table 6.5 below, IDA is significant while log GDP PC is not).
- 21.
The impact of a single year is trivial but years in office range from 1 to 45 so there is a sizeable effect some cases. Because years in office might be related to conflict, we explored the impact of a conflict dummy derived from PRIO data. The magnitude of the Years in office hazard rate is unchanged and so is not driven by omitted information about conflict.
- 22.
Again, controlling for conflict does not change results.
- 23.
IEG states: “Each PPAR is subject to internal IEG peer review, Panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible Bank department. The PPAR is also sent to the borrower for review. IEG incorporates both Bank and borrower comments as appropriate, and the borrowers’ comments are attached to the document that is sent to the Bank’s Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public.” (IEG 2015a, vi)
- 24.
We do not interact UNSC with Downgrade or Upgrade because, in combination, these are rare events, for example, only seven cases where a UNSC member has a downgrade. Looking at UNSC membership, timing is critical (since nonpermanent membership lasts only two years) whereas it is less critical when considering other geopolitical variables (bilateral aid, UN voting alignment, etc.) that vary less over time. This explains why results differ between UNSC@ICR and UNSC@PPAR while results for the other geopolitical variables do not depend on whether measured at the time of the ICR or the PPAR.
- 25.
We do not present ordered probit results because of the challenges involved in presenting useful marginal effects in this setting. However, the sign and significance of estimated coefficients for the ordered probit latent variable model match the linear model.
- 26.
Project characteristics include an IDA dummy and a Program Loan dummy. Country characteristics include GDP per capita, GDP growth rate, inflation, openness, the ratio of short-term debt to total debt, the average number of years of schooling, the combined Freedom House rating, and government years in office, all at approval. Sample size falls from 1500 observations (on 120 countries) in the parsimonious specifications to 1329 observations (on 93 countries) in the full specification. This is less than the 1371 observations in Table 6.5 because we exclude FY2002 PPARs. The full specification probit sample is slightly smaller still, at 1219 observations (on 69 countries) due to lack of variation in PPAR ratings (all Satisfactory or all Unsatisfactory) in some countries.
References
Azevedo, João Pedro. 2011. wbopendata: Stata module to access World Bank databases, Statistical Software Components S457234, Boston College Department of Economics. http://ideas.repec.org/c/boc/bocode/s457234.html
Beck, Thorsten, George Clarke, Alberto Groff, Philip Keefer, and Patrick Walsh. 2001. New tools in comparative political economy: The Database of Political Institutions. World Bank Economic Review 15:165–176.
Blum, Jürgen René. 2014. What factors predict how public sector projects perform? A review of the World Bank’s public sector management portfolio. World Bank Policy Research Working Paper 6798.
Bulman, David, Walter Kolkma, and Aart Kraay. 2017. Good countries or good projects? Comparing macro and micro correlates of World Bank and Asian Development Bank project performance. Review of International Organizations 12(3):335–363.
Buntaine, Mark, and Bradley C. Parks. 2013. When do environmentally-focused assistance projects achieve their objectives? Evidence from World Bank post-project evaluations. Global Environmental Politics 13(2):67–90.
Chauvet, Lisa, Paul Collier, and Marguerite Duponchel. 2010. What explains aid project success in post-conflict situations? World Bank Policy Research Working Paper 5418.
Chauvet, Lisa, Paul Collier, and Andreas Fuster. 2017. Supervision and Project Performance: A Principal-Agent Approach.〈hal-01516966〉.
Cruz, Cesi, and Philip Keefer. 2013. The organization of programmatic parties and the politics of bureaucratic reform. World Bank Policy Research Working Paper 6686.
Deininger, Klaus, Lyn Squire, and Swati Basu. 1998. Does economic analysis improve the quality of foreign assistance? World Bank Economic Review 12(3):385–418.
Denizer, Cevdet, Daniel Kaufmann, and Aart Kraay. 2013. Good countries or good projects? Macro and micro correlates of World Bank project performance. Journal of Development Economics 105:288–302.
Dollar, David, and Victoria Levin. 2005. Sowing and reaping: Institutional quality and project outcomes in developing countries. World Bank Policy Research Working Paper # 3524.
Dollar, David, and Jakob Svensson. 2000. What explains the success or failure of structural adjustment programmes? The Economic Journal 110(466):894–917.
Dreher, Axel, Jan-Egbert Sturm, and James Vreeland. 2009. Development aid and international politics: Does membership on the UN Security Council influence World Bank decisions? Journal of Development Economics 88: 1–18.
Dreher, Axel, Stephan Klasen, James Raymond Vreeland, and Eric Werker. 2013. The costs of favoritism: Is politically-driven aid less effective? Economic Development and Cultural Change 62(1):157–191.
Freedom House. 2012. Freedom in the World.
Geli, Patricia, Aart Kraay, and Hoveida Nobakht. 2014. Predicting World Bank project outcome ratings. World Bank Policy Research Working Paper 7001.
Girod, Desha M., and Jennifer Tobin. 2016. Take the money and run: The determinants of compliance with aid agreements. International Organization 70(1) 209–239.
Grasso, Patrick G., Sulaiman S. Wasty, Rachel V. Weaving (Eds). 2003. World Bank Operations Evaluation Department: The First 30 Years. Washington, DC: The World Bank.
Guillaumont, Patrick, and Rachid Laajaj. 2006. When instability increases the effectiveness of aid projects. World Bank Policy Research Working Paper # 4034.
IEG (World Bank Independent Evaluation Group). 2010. Cost-Benefit Analysis in World Bank Projects. Washington, DC: The World Bank.
IEG (World Bank Independent Evaluation Group). 2011. IEG Work Program and Budget (FY12) and Indicative Plan (FY13–14). Washington, DC: The World Bank.
IEG (World Bank Independent Evaluation Group). 2014. Independent Evaluation Group - World Bank Project Performance Ratings. http://ieg.worldbankgroup.org/ratings
IEG (World Bank Independent Evaluation Group). 2015a. Project Performance Assessment Report. Brazil: First Programmatical Reform Loan for Environmental Sustainability. Report No.: 92270-BR.
IEG (World Bank Independent Evaluation Group). 2015b. World Bank Project Performance Ratings - Codebook. https://ieg.worldbankgroup.org/Data/reports/ieg-wb-project-performance-ratings-codebook_092015.pdf
Isham, Jonathan, Daniel Kaufmann, and Lant Pritchett. 1997. Civil liberties, democracy, and the performance of government projects. World Bank Economic Review 11(2):219–242.
Isham, Jonathan, and Daniel Kaufmann. 1999. The forgotten rationale for policy reform: The productivity of investment projects. Quarterly Journal of Economics 114(1):149–184.
Kaufmann, Daniel, and Yan Wang. 1995. Macroeconomic policies and project performance in the social sectors: A model of human capital production and evidence from LDCs. World Development 23(5)751–765.
Kaja, Ashwin, and Eric Werker. 2010. Corporate governance at the World Bank and the dilemma of global governance. World Bank Economic Review 24(2):171–198.
Kersting, Erasmus, and Christopher Kilby. 2016. With a little help from my friends: Global electioneering and World Bank lending. Journal of Development Economics 121:153–165.
Kilby, Christopher. 2000. Supervision and performance: The case of World Bank projects. Journal of Development Economics 62(1):233–259.
Kilby, Christopher. 2009. The political economy of conditionality: an empirical analysis of World Bank loan disbursements. Journal of Development Economics 89(1):51–61.
Kilby, Christopher. 2013a. An empirical assessment of informal influence in the World Bank. Economic Development and Cultural Change 61(2):431–464.
Kilby, Christopher. 2013b. The political economy of project preparation: An empirical analysis of World Bank projects. Journal of Development Economics 105:211–225.
Kilby, Christopher. 2015. Assessing the impact of World Bank preparation on project outcomes. Journal of Development Economics 115:111–123.
Limodio, Nicola. 2011. The success of infrastructure projects in low income countries and the role of selectivity. World Bank Policy Research Working Paper # 5694.
Malesa, Thaddeus, and Peter Silarszky. 2005. Does World Bank effort matter for success of adjustment operations? In Stefan Koeberle, Harold Bedoya, Peter Silarsky, and Gero Verheyen (Eds.), Conditionality Revisited: Concepts, Experiences, and Lessons. (Washington: The World Bank). Chapter 13, pages 127–141.
Malik, Rabia, and Randall W. Stone. 2018. Corporate influence in World Bank lending. Journal of Politics 80(1):103–1118.
Michaelowa, Katharina, and Axel Borrmann. 2006. Evaluation bias and incentive structures in bi- and multilateral aid agencies. Review of Development Economics 10(2):313–329.
Moll, Peter, Patricia Geli, and Pablo Saavedra. 2015. Correlates of success in World Bank development policy lending. World Bank Policy Research Working Paper 7181.
Morrison, Kevin M. 2013. Membership no longer has its privileges: The declining informal influence of board members on IDA lending. Review of International Organizations 8(2):291–312.
OECD (Organization for Economic Cooperation and Development). 2015. Geographical distribution of financial flows: Flows to developing countries. OECD International Development Statistics (database). https://doi.org/10.1787/data-00566-en.
Pohl, Gerhard, and Dubravko Mihaljek. 1992. Project evaluation and uncertainty in practice: A statistical analysis of rate-of-return divergences of 1,015 World Bank projects. The World Bank Economic Review 6(2):255–277.
Ralston, Laura. 2014. Success in difficult environments: A portfolio analysis of fragile and conflict-affected states. World Bank Policy Research Working Paper 7098.
Smets, Lodewijk, Stephen Knack, and Nadia Molenaers. 2013. Political ideology, quality at entry and the success of economic reform programs. Review of International Organizations 8(4):447–476.
Stone, Randall. 2008. The scope of IMF conditionality. International Organization 62:589–620.
Strezhnev, Anton, and Erik Voeten. 2012. United Nations General Assembly Voting Data, https://dataverse.harvard.edu/dataset.xhtml?persistentId=hdl:1902.1/12379.
Sud, Inder, and Jane Olmstead-Rumsey. 2012. Development outcomes of World Bank projects: Real or illusory improvements. George Washington University Institute for Global and International Studies Working Paper #1.
United Nations. 2014. Countries Elected Members of the Security Council. http://www.un.org/en/sc/members/elected.asp Accessed 9/15/2014.
USAID (United States Agency for International Development). 2014. U.S. overseas loans and grants: Obligations and loan authorizations, July 1, 1945–September 30, 2012 (Greenbook). https://explorer.usaid.gov/reports-greenbook.html Accessed 3/20/2014.
U.S. State Department. 1984–2013. Voting Practices in the United Nations. GPO, Washington.
Winters, Matthew S. 2014. Targeting, accountability and capture in development projects. International Studies Quarterly 58(2):393–404.
World Bank. 2012a. Access to Information. Annual Report FY 2011. Moving Forward Transparency and Accountability. (Washington: The World Bank).
World Bank. 2012b. Projects database. http://go.worldbank.org/KPMUDAVVT0 Accessed 11/30/2012.
World Bank. 2014. World Development Indicators. http://databank.worldbank.org/data/reports.aspx?source=world-development-indicators Accessed 10/14/2014.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Appendix
Appendix
Rights and permissions
Copyright information
© 2019 The Author(s)
About this chapter
Cite this chapter
Kilby, C., Michaelowa, K. (2019). What Influences World Bank Project Evaluations?. In: Dutta, N., Williamson, C.R. (eds) Lessons on Foreign Aid and Economic Development. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-030-22121-8_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-22121-8_6
Published:
Publisher Name: Palgrave Macmillan, Cham
Print ISBN: 978-3-030-22120-1
Online ISBN: 978-3-030-22121-8
eBook Packages: Economics and FinanceEconomics and Finance (R0)