Skip to main content

Advertisement

SpringerLink
Log in

How much Confidence can we have in EU-SILC? Complex Sample Designs and the Standard Error of the Europe 2020 Poverty Indicators

  • Published:
Social Indicators Research Aims and scope Submit manuscript

Abstract

If estimates are based on samples, they should be accompanied by appropriate standard errors and confidence intervals. This is true for scientific research in general, and is even more important if estimates are used to inform and evaluate policy measures such as those aimed at attaining the Europe 2020 poverty reduction target. In this article I pay explicit attention to the calculation of standard errors and confidence intervals, with an application to the European Union Statistics on Income and Living Conditions (EU-SILC). The estimation of accurate standard errors requires among others good documentation and proper sample design variables in the dataset. However, this information is not always available. Therefore, I complement the existing documentation on the sample design of EU-SILC and test the effect on estimated standard errors of various simplifying assumptions with regard to the sample design. It is shown that accounting for clustering within households is of paramount importance. Although this results in many cases in a good approximation of the standard error, taking as much as possible account of the entire sample design generally leads to more accurate estimates, even if sample design variables are partially lacking. The effect is illustrated for the official Europe 2020 indicators of poverty and social exclusion and for all European countries included in the EU-SILC 2008 dataset. The findings are not only relevant for EU-SILC users, but also for users of other surveys on income and living conditions which lack accurate sample design variables.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. In some imputation methods, a missing value is imputed by taking the non-missing value of another (otherwise similar) observation; the latter is called the “donor”.

  2. http://132.203.59.36/DAD.

  3. http://132.203.59.36/DASP/index.html.

  4. For non-smooth indicators such as many of the Laeken poverty indicators the jackknife is not recommended (e.g. Shao and Chen 1998; del Mar Rueda and Muñoz 2011). In the case of the at-risk-of-poverty indicator (FGT0 and FGT1) the resulting standard errors using the bootstrap are very close to those obtained on the basis of linearisation using the DASP module for Stata (figures available from the author).

  5. If a PSU is self-representing, it is included with a probability equal to 1, which means that the PSU is rather a stratum than a PSU. For variance calculations this makes a difference with the case in which a stratum in the dataset contains only one PSU which has been selected (or contains respondents) among a larger amount of PSUs which populate the stratum in reality, but were not selected in the sample.

  6. For an in-depth discussion on the quality of the sample design variables in the ‘EU-SILC Eurostat’ dataset, see Goedemé (2010a). For a more precise description of the problems associated with the sample design variables in the UDB and the use of the region variable as a stratification variable, see Goedemé (2010b).

  7. In most countries many types of top–bottom coding would not make a big difference: neither for the estimated number of poor (cf. Van Kerm 2007), nor for the estimated standard errors (figures available from the author).

  8. Estimates by country can be found in Goedemé (2010b). The weak correlation between deprivation and income poverty has been extensively documented in the literature, e.g. Dewilde (2004, 2008) and Whelan and Maître (2007).

  9. Please note that in the case of Hungary also the ‘Eurostat data’ do not contain fully accurate sample design variables.

  10. http://doiop.com/SvysetEU-SILC2008.

  11. It should be noted that Eurostat is currently working on these issues and several projects are running to improve the sample design variables as well as to evaluate the feasibility of various approaches which should allow researchers to properly estimate standard errors for all EU-SILC countries.

  12. Figure applies to a two-tailed test. For a one-tailed test at least a difference of around 177,000 persons is needed. The estimation does not take account of the fact that the proportion is partially based on a random poverty line. Please note that for inter-temporal comparisons spanning a period of less than 5 years, one should take account of the covariance between the 2 years under evaluation as a result of the panel character of EU-SILC. The present estimate ignores this potential covariance. In addition, in many countries PSUs have been selected for the entire duration of EU-SILC, so also for differences between point estimates separated by more than 4 years, one should take account of the entire variance–covariance structure.

References

  • Alfons, A., Temple, M., & Filzmoser, P. (2009). On the influence of imputation methods on Laeken indicators: Simulations and recommendations. In Conference of European statisticians, Neuchatel, Switzerland, October 5–7, 2009 (pp. 9).

  • Araar, A., & Duclos, J.-Y. (2007). DASP: Distributive analysis stata package. PEP, CIRPÉE and World Bank, Université Laval.

  • Araar, A., & Duclos, J.-Y. (2009). DAD: A software for poverty and distributive analysis. Journal of Economic and Social Measurement, 34(2/3), 175–189.

    Google Scholar 

  • Atkinson, A. B., Cantillon, B., Marlier, E., & Nolan, B. (2002). Social indicators: The EU and social inclusion. Oxford: Oxford University Press.

    Google Scholar 

  • Berger, Y. G., & Skinner, C. J. (2003). Variance estimation for a low income proportion. Journal of the Royal Statistical Society. Series C (Applied Statistics), 52(4), 457–468.

    Article  Google Scholar 

  • Biewen, M. (2002). Bootstrap inference for inequality, mobility and poverty measurement. Journal of Econometrics, 108(2), 317–342.

    Article  Google Scholar 

  • Biewen, M., & Jenkins, S. P. (2006). Variance estimation for generalized entropy and Atkinson inequality indices: The complex survey data case. Oxford Bulletin of Economics and Statistics, 68(3), 371–383.

    Article  Google Scholar 

  • Cochran, W. G. (1977). Sampling techniques. New York: Wiley.

    Google Scholar 

  • Davidson, R., & Duclos, J.-Y. (2000). Statistical inference for stochastic dominance and for the measurement of poverty and inequality. Econometrica, 68(6), 1435–1464.

    Article  Google Scholar 

  • Davidson, R., & Flachaire, E. (2007). Asymptotic and bootstrap inference for inequality and poverty measures. Journal of Econometrics, 141(1), 141–166.

    Article  Google Scholar 

  • de Vos, K., & Zaidi, A. M. (1998). Poverty measurement in the European Union: Country-specific or union-wide poverty lines? Journal of Income Distribution, 8(1), 77–92.

    Article  Google Scholar 

  • del Mar Rueda, M., & Muñoz, J. F. (2011). Estimation of poverty measures with auxiliary information in sample surveys. Quality and Quantity, 45(3), 687–700.

    Article  Google Scholar 

  • Dewilde, C. (2004). The multidimensional measurement of poverty in Belgium and Britain: A categorical approach. Social Indicators Research, 68(3), 331–369.

    Article  Google Scholar 

  • Dewilde, C. (2008). Individual and institutional determinants of multidimensional poverty: A European comparison. Social Indicators Research, 86(2), 233–256.

    Article  Google Scholar 

  • Duclos, J.-Y., & Araar, A. (2006). Poverty and equity. Measurement, policy, and estimation with DAD. New York: Springer.

    Google Scholar 

  • Efron, B., & Tibshirani, R. J. (1998). An introduction to the bootstrap. Boca Raton: Chapman & Hall/CRC.

    Google Scholar 

  • European Commission. (2006). Portfolio of overarching indicators and streamlined social inclusion, pensions, and health portfolios. Brussels: European Commission.

    Google Scholar 

  • European Council. (2010). European Council 17 June 2010 conclusions. Brussels: European Council.

    Google Scholar 

  • Eurostat. (2002). Monographs of official statistics. Variance estimation methods in the European Union. Luxembourg: Office for Official Publications of the European Communities.

    Google Scholar 

  • Eurostat. (2010a). 2008 comparative EU intermediate quality report. Version 2—June 2010. Luxembourg: Eurostat.

  • Eurostat. (2010b). Combating poverty and social exclusion. A statistical portrait of the European Union 2010. Luxembourg: Publications Office of the European Union.

    Google Scholar 

  • Eurostat. (2010c). Description of target variables: Cross-sectional and longitudinal 2008 operation (Version January 2010). Brussels: European Commission.

    Google Scholar 

  • Foster, J., Greer, J., & Thorbecke, E. (1984). A class of decomposable poverty measures. Econometrica, 52(3), 761–766.

    Article  Google Scholar 

  • Frazer, H., Marlier, E., Natali, D., Van Dam, R., & Vanhercke, B. (2010). Europe 2020: Towards a more social EU? In E. Marlier, D. Natali, & R. Van Dam (Eds.), Europe 2020. Towards a more social EU? (pp. 15–44). Brussels: P.I.E. Peter Lang.

    Google Scholar 

  • Goedemé, T. (2010a). The construction and use of sample design variables in EU-SILC. A user’s perspective. Report prepared for Eurostat, Antwerp: Herman Deleeck Centre for Social Policy, University of Antwerp.

  • Goedemé, T. (2010b). The standard error of estimates based on EU-SILC. An exploration through the Europe 2020 poverty indicators. CSB Working Paper Series, WP 10/09 (p. 36). Antwerp: Herman Deleeck Centre for Social Policy, University of Antwerp.

  • Goedemé, T., & Rottiers, S. (2011). Poverty in the Enlarged European Union. A discussion about definitions and reference groups. Sociology Compass, 5(1), 77–91. doi:10.1111/j.1751-9020.2010.00350.x.

  • Groves, R. M., Fowler, F. J. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). New Jersey: Wiley.

    Google Scholar 

  • Guio, A.-C. (2009). What can be learned from deprivation indicators in Europe? In Paper presented at the indicator subgroup of the Social Protection Committee, February 10, 2009 (p. 33).

  • Heeringa, S. G., West, B. T., & Berglund, P. A. (2010). Applied survey data analysis. Boca Raton: Chapman & Hall/CRC.

    Book  Google Scholar 

  • Howes, S., & Lanjouw, J. O. (1998). Does sample design matter for poverty rate comparisons? Review of Income and Wealth, 44(1), 99–109.

    Article  Google Scholar 

  • Jolliffe, D., Datt, G., & Sharma, M. (2004). Robust poverty and inequality measurement in Egypt: Correcting for spatial-price variation and sample design effects. Review of Development Economics, 8(4), 557–572.

    Article  Google Scholar 

  • Jolliffe, D., & Semykina, A. (1999). sg117—robust standard errors for the Foster–Greer–Thorbecke class of poverty indices. Stata Technical Bulletin, STB, 51, 34–36.

    Google Scholar 

  • Kakwani, N. (1993). Statistical inference in the measurement of poverty. The Review of Economics and Statistics, 75(4), 632–639.

    Article  Google Scholar 

  • Kalton, G. (1983). Introduction to survey sampling (Vol. 35, quantitative applications in the social sciences). Beverly Hills: Sage Publications.

    Google Scholar 

  • Kangas, O., & Ritakallio, V.-M. (2007). Relative to what? Cross national pictures of European poverty measured by regional, national and European standards. European Societies, 9(2), 119–145.

    Article  Google Scholar 

  • Kish, L. (1965). Survey sampling. New York: Wiley.

    Google Scholar 

  • Kolenikov, S. (2010). Resampling variance estimation for complex survey data. Stata Journal, 10(2), 165–199.

    Google Scholar 

  • Lee, E. S., & Forthofer, R. N. (2006). Analyzing complex survey data. Second edition (Vol. 71, quantitative applications in the social sciences). Thousand Oaks: Sage Publications.

    Google Scholar 

  • Lohmann, H. (2011). Comparability of EU-SILC survey and register data: The relationship among employment, earnings and poverty. Journal of European Social Policy, 21(1), 37–54. doi:10.1177/0958928710385734.

    Google Scholar 

  • Marlier, E., Atkinson, A. B., Cantillon, B., & Nolan, B. (2007). The EU and social inclusion. Facing the challenges. Bristol: The Policy Press.

    Google Scholar 

  • Mooney, C. Z., & Duval, R. D. (1993). Bootstrapping: A nonparametric approach to statistical inference (Vol. 95, quantitative applications in the social sciences). Newbury Park: Sage Publications.

    Google Scholar 

  • OECD. (2008). Growing unequal? Income distribution and poverty in OECD countries. Paris: OECD.

    Google Scholar 

  • Osier, G. (2009). Variance estimation for complex indicators of poverty and inequality using linearization techniques. Survey Research Methods, 3(3), 167–195.

    Google Scholar 

  • Preston, I. (1995). Sampling distributions of relative poverty statistics. Journal of the Royal Statistical Society. Series C (Applied Statistics), 44(1), 91–99.

    Google Scholar 

  • Rodgers, J. R., & Rodgers, J. L. (1993). Chronic poverty in the United States. The Journal of Human Resources, 28(1), 25–54.

    Article  Google Scholar 

  • Shao, J. (1996). Invited discussion paper resampling methods in sample surveys. Statistics: A Journal of Theoretical and Applied Statistics, 27(3), 203–237.

    Google Scholar 

  • Shao, J., & Chen, Y. (1998). Bootstrapping sample quantiles based on complex survey data under hot deck imputation. Statistica Sinica, 8(4), 1071–1085.

    Google Scholar 

  • Sturgis, P. (2004). Analysing complex survey data: Clustering, stratification and weights. Social Research Update, 43, 1–6.

    Google Scholar 

  • Thuysbaert, B. (2008). Inference for the measurement of poverty in the presence of a stochastic weighting variable. Journal of Economic Inequality, 6(1), 33–55.

    Article  Google Scholar 

  • Trede, M. (2002). Bootstrapping inequality measures under the null hypothesis: Is it worth the effort? Journal of Economics, 77(Supplement 1), 261–282.

    Article  Google Scholar 

  • Van Kerm, P. (2002). Inference on inequality measures: A Monte Carlo experiment. Journal of Economics, 77(Supplement 1), 283–306.

    Article  Google Scholar 

  • Van Kerm, P. (2007). Extreme incomes and the estimation of poverty and inequality indicators from EU-SILC. IRISS Working Paper Series (p. 51). Luxembourg: CEPS-Instead.

  • Verma, V., Betti, G., & Gagliardi, F. (2010). An assessment of survey errors in EU-SILC. Eurostat Methodologies and Working Papers (p. 70). Luxembourg: Eurostat.

  • Whelan, C. T., & Maître, B. (2007). Income, deprivation and economic stress in the enlarged European Union. Social Indicators Research, 83(2), 309–329.

    Article  Google Scholar 

  • Wolff, P. (2010). 17% of EU citizens were at-risk-of-poverty in 2008. Statistics in focus (p. 8). Luxembourg: Eurostat.

  • Wolter, K. M. (2007). Introduction to variance estimation. New York: Springer.

    Google Scholar 

  • Zheng, B. (2001). Statistical inference for poverty measures with relative poverty lines. Journal of Econometrics, 101(2), 337–356.

    Article  Google Scholar 

Download references

Acknowledgments

This article has benefited from comments by Karel Van den Bosch, Guillaume Osier, Ulrich Kohler, Lina Salanauskaite, Vincent Corluy, Joris Ghysels, Rudi Van Dam, Koen Decancq and an anonymous referee. I am grateful to Fabienne Montaigne and Pascal Wolff for offering me the opportunity to run my Stata programmes on the EU-SILC data available to Eurostat. The paper has been presented at the Final Equalsoc Conference, June 2010 in Amsterdam, the EU-LFS and EU-SILC 2nd European User Conference, March 2011 in Mannheim as well as the 4th Conference of the European Survey Research Association, July 2011 in Lausanne. Comments and suggestions from the participants are gratefully acknowledged. This research has been funded by the Research Foundation—Flanders (FWO). All opinions expressed in this paper as well as any remaining errors and shortcomings are my own. Findings, opinions and suggestions presented in this article do not necessarily reflect those of Eurostat.

Author information

Authors and Affiliations

  1. Herman Deleeck Centre for Social Policy, University of Antwerp, St. Jacobstraat 2 (M479), 2000, Antwerp, Belgium

    Tim Goedemé

  2. Research Foundation—Flanders, Egmontstraat 5, 1000, Brussel, Belgium

    Tim Goedemé

Authors
  1. Tim Goedemé
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tim Goedemé.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Goedemé, T. How much Confidence can we have in EU-SILC? Complex Sample Designs and the Standard Error of the Europe 2020 Poverty Indicators. Soc Indic Res 110, 89–110 (2013). https://doi.org/10.1007/s11205-011-9918-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11205-011-9918-2

Keywords

Search

Navigation