Abstract
Recommendations for practice are routinely included in articles that report educational research. Robinson et al. suggest that reports of primary research should not routinely do so. They argue that single primary research studies seldom have sufficient external validity to support claims about practice policy. In this article, I draw on recent statistical research that has formalized subjective notions about generalizability from experiments. I show that even rather large experiments often do not support generalizations to policy-relevant inference populations. This suggests that single primary studies are unlikely to be sufficiently generalizable to support recommendations for practice.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Cochran, W. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24, 295–313.
Hedges, L. V., & O'Muircheartaigh, C. (2010). Improving inference for population level treatment effects in social experiments. Working paper, Northwestern University Institute for Policy Research.
Kalton, G. (1968). Standardization: a technique to control for extraneous variables. Applied Statistics, 17, 118–136.
Kitagawa, E. M. (1964). Standardized comparisons in population research. Demography, 1, 296–315.
Kruskal, W., & Mosteller, F. (1979). Representative sampling III: the current statistical literature. International Statistical Review, 47, 245–265.
Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data. New York: Wiley.
Mosteller, F., Light, R. J., & Sachs, J. A. (1996). Sustained inquiry in education: lessons learned from skill grouping and class size. Harvard Educational Review, 66, 797–842.
Nye, B., Hedges, L. V., & Konstantopoulos, S. (2000). The effects of small classes on achievement: the results of the Tennessee class size experiment. American Educational Research Journal, 37, 123–151.
O'Muircheartaigh, C., & Hedges, L. V. (2013). Generalizing from unrepresentative experiments: a stratified propensity score approach. Journal of the Royal Statistical Society, Series C (in press)
Oaxaca, R. (1973). Male–female wage differentials in urban labor markets. International Economic Review, 14, 693–709.
Robinson, D. H., Levin, J. R., Schraw, G., Patal, E. A., & Hunt, E. B. (2013). On going (way) beyond one's data: a proposal to restrict recommendations for practice in primary educational research journals. Educational Psychology Review (in press).
Roschelle, J., Shechtman, N., Tatar, D., & Hegedus, S. (2010). Integration of technology, curriculum, and professional development for advancing middle school mathematics. American Educational Research Journal, 47, 833–878.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
Rosenberg, P. (1962). Test factor standardization as a method of interpretation. Social Forces, 41, 53–61.
Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society, Series A, Part, 2, 369–386.
Tipton, E. (2013). Improving generalizations from experiments using propensity score subclassification: assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics. (in press)
Tipton, E., & Hedges, L. V. (2013). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness (in press)
Author information
Authors and Affiliations
Corresponding author
Additional information
This paper is based in part on work supported by the US National Science Foundation (NSF) under grants #0815295 and #1118978. Any opinions, findings, and conclusions or recommendations are those of the authors and do not necessarily represent the views of the NSF.
Rights and permissions
About this article
Cite this article
Hedges, L.V. Recommendations for Practice: Justifying Claims of Generalizability. Educ Psychol Rev 25, 331–337 (2013). https://doi.org/10.1007/s10648-013-9239-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10648-013-9239-x