Abstract
When the process of publication favors studies with smallp-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailedp-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Begg, C. B. (1994). Publication bias. In H. Cooper & L. V. Hedges,The handbook of research synthesis (pp. 399–409). New York: Russell Sage Foundation.
Begg, C. B., & Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data (with discussion).Journal of the Royal Statistical Society, Series A, 151, 419–463.
Bozarth, J. D., & Roberts, R. R. (1972). Signifying significant significance.American Psychologist, 27, 774–775.
Cooper, H., & Hedges, L. V. (1994).The handbook of research synthesis. New York: Russell Sage Foundation.
Coursol, A., & Wagner, E. E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias.Professional Psychology, 17, 136–137.
Dawes, R. M., Landman, J., & Williams, M. (1984). Discussion on meta-analysis and selective publication bias.American Psychologist, 39, 75–78.
Dear, K. B. G., & Begg, C. B. (1992). An approach for assessing publication bias prior to performing a meta-analysis.Statistical Science, 7, 237–245.
Dickersin, K., Min, Y-I, & Meinert, C. L. (1991). The fate of controlled trials funded by the NIH in 1979.Controlled Clinical Trials, 12, 634.
Dickersin, K., Min, Y-I, & Meinert, C. L. (1992). Factors influencing the publication of research results: Followup of applications submitted to two institutional review boards.Journal of the American Medical Association, 267, 374–378.
Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publication bias in clinical research.Lancet, 337, 867–872.
Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis.Psychological Bulletin, 82, 1–20.
Hedges, L. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences.Journal of Educational Statistics, 9, 61–85.
Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis.Statistical Science, 7, 246–255.
Hedges, L. V., & Olkin, I. (1985).Statistical methods for meta-analysis. New York: Academic Press.
Hedges, L. V., & Vevea, J. L. (1993).Estimating effect size under publication bias: Small sample properties and robustness of a selection model. Manuscript submitted for publication.
Iyengar, S., & Greenhouse, J. B. (1988). Selection models and the file drawer problem.Statistical Science, 3, 109–135.
Kendall, M., & Stuart, A. (1979).The advanced theory of statistics. Volume 2, Inference and relationship (4th ed.). London and High Wycombe: Charles Griffin and Company.
Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions.British Journal of Mathematical and Statistical Psychology, 31, 107–112.
Light, R. J., & Pillemer, D. B. (1984).Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.
Melton, A. W. (1962). Editorial.Journal of Experimental Psychology, 64, 553–557.
National Research Council (1992).Combining information: Statistical issues and research opportunities. Washington, DC: National Academy Press.
Nelson, N., Rosenthal, R., & Rosnow, R. L. (1986). Interpretation of significance levels by psychological researchers.American Psychologist, 41, 1299–1301.
Rosenthal, R., & Gaito, J. (1963). The interpretation of levels of significance by psychological researchers.Journal of Psychology, 55, 33–38.
Rosenthal, R., & Gaito, J. (1964). Further evidence for the cliff effect in the interpretation of levels of significance.Psychological Reports, 4, 570.
Smith, M. L. (1980). Publication bias in meta-analysis.Evaluation in Education, 4, 22–24.
Smith, M. L., Glass, G. V., & Miller, T. I. (1980).The benefits of psychotherapy. Baltimore: The Johns Hopkins University Press.
Sterling, T. C. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vice versa.Journal of the American Statistical Association, 54, 30–34.
Vevea, J. L., Clements, N. C., & Hedges, L. V. (1993). Assessing the effects of selection bias on validity data for the general aptitude test battery.Journal of Applied Psychology, 78, 981–987.
White, K. R. (1982). The relation between socioeconomic status and achievement.Psychological Bulletin, 31, 461–481.
Author information
Authors and Affiliations
Additional information
Authors' note: The contributions of the authors are considered equal, and the order of authorship was chosen to be reverse-alphabetical.
Rights and permissions
About this article
Cite this article
Vevea, J.L., Hedges, L.V. A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60, 419–435 (1995). https://doi.org/10.1007/BF02294384
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF02294384