Analysis of Cluster-Randomized Experiments: A Comparison of Alternative Estimation Approaches

Lynn VavreckUCLA, Department of Political Science, 4289 Bunche Hall Box 951472, Los Angeles, CA 90095-1472 e-mail: lvavreck{at}ucla.edu (corresponding author) Analysts of cluster-randomized field experiments have an arrayof estimation techniques to choose from. Using Monte Carlo simulation,we evaluate the properties of point estimates and standard errors(SEs) generated by ordinary least squares (OLS) as applied toboth individual-level and cluster-level data. We also compareOLS to alternative random effects estimators, such as generalizedleast squares (GLS). Our simulations assess efficiency acrossa variety of scenarios involving varying sample sizes and numbersof clusters. Our results confirm that conventional OLS SEs areseverely biased downward and that, for all estimators, gainsin efficiency come mainly from increasing the number of clusters,not increasing the number of individuals within clusters. Wefind relatively minor differences across alternative estimationapproaches, but GLS seems to enjoy a slight edge in terms ofthe efficiency of its point estimates and the accuracy of itsSEs. We illustrate the application of alternative estimationapproaches using a clustered experiment in which Rock the VoteTV advertisements were used to encourage young voters in 85cable TV markets to vote in the 2004 presidential election. Authors' note: We thank Rock the Vote for permission to usetheir public service announcements in this field experiment.The authors are grateful to Alan Gerber for suggestions throughoutthe design phase of this project. We are also grateful to DanKotin and Margaret Coblentz, who worked with cable operators,distributed the advertisements, and assembled the data. We thankTerence Leong for his programming expertise. Replication materialsare available on the Political Analysis Web site.