Publication bias as a threat to the validity of meta-analytic results

This paper reviews the evidence in support of the contention that publication bias is a potential threat to the validity of meta-analytic results in criminology and similar fields. It then provides a critique of the traditional file drawer or failsafe N method for examining publication bias, and an overview of four newer methods that can be used to detect publication bias. These include two (trim and fill and cumulative meta-analysis) that enable the researcher to estimate the magnitude of the influence of publication bias on the overall mean effect size. Advantages and limitations of both traditional and newer methods are examined. The methods reviewed are illustrated through their application to a meta-analysis of the effects of drug courts on recidivism by Wilson et al. (Journal of Experimental Criminology, 2, 459–487, 2006).

Hannah R. RothsteinEmail:

---

Hannah R. Rothstein   is Professor of Management at Baruch College and the Graduate Center of the City University of New York. She is the author of various employment-related meta-analyses, as well as numerous articles and book chapters on methodological issues in meta-analysis. Dr. Rothstein is co-author of the Comprehensive Meta-Analysis software, and, with Alex Sutton and Michael Borenstein, is co-editor of Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments, (Wiley, 2005). She is currently writing two books on meta-analysis with Michael Borenstein, Larry Hedges and Julian Higgins. Dr. Rothstein is a fellow of the Society for Industrial and Organizational Psychology and of the American Psychological Association, and she serves on the Editorial Boards of the Psychological Bulletin, Organizational Research Methods, and the Journal of Experimental Criminology. Her Ph.D. in Industrial and Organizational Psychology is from the University of Maryland.