Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment

Alan S. Gerber and Donald P. GreenYale University, Institution for Social and Policy Studies, P.O. Box 208209, 77 Prospect Street, New Haven, CT 06520 e-mail: kevin.arceneaux{at}temple.edu (corresponding author) e-mail: alan.gerber{at}yale.edu e-mail: donald.green{at}yale.edu In the social sciences, randomized experimentation is the optimalresearch design for establishing causation. However, for a numberof practical reasons, researchers are sometimes unable to conductexperiments and must rely on observational data. In an effortto develop estimators that can approximate experimental resultsusing observational data, scholars have given increasing attentionto matching. In this article, we test the performance of matchingby gauging the success with which matching approximates experimentalresults. The voter mobilization experiment presented here comprisesa large number of observations (60,000 randomly assigned tothe treatment group and nearly two million assigned to the controlgroup) and a rich set of covariates. This study is analyzedin two ways. The first method, instrumental variables estimation,takes advantage of random assignment in order to produce consistentestimates. The second method, matching estimation, ignores randomassignment and analyzes the data as though they were nonexperimental.Matching is found to produce biased results in this applicationbecause even a rich set of covariates is insufficient to controlfor preexisting differences between the treatment and controlgroup. Matching, in fact, produces estimates that are no moreaccurate than those generated by ordinary least squares regression.The experimental findings show that brief paid get-out-the-votephone calls do not increase turnout, while matching and regressionshow a large and significant effect.