THE INTERNAL AND EXTERNAL VALIDITY OF THE REGRESSION DISCONTINUITY DESIGN: A META‐ANALYSIS OF 15 WITHIN‐STUDY COMPARISONS

Theory predicts that regression discontinuity (RD) provides valid causal inference at the cutoff score that determines treatment assignment. One purpose of this paper is to test RD's internal validity across 15 studies. Each of them assesses the correspondence between causal estimates from an RD study and a randomized control trial (RCT) when the estimates are made at the same cutoff point where they should not differ asymptotically. However, statistical error, imperfect design implementation, and a plethora of different possible analysis options, mean that they might nonetheless differ. We test whether they do, assuming that the bias potential is greater with RDs than RCTs. A second purpose of this paper is to investigate the external validity of RD by exploring how the size of the bias estimates varies across the 15 studies, for they differ in their settings, interventions, analyses, and implementation details. Both Bayesian and frequentist meta‐analysis methods show that the RD bias is below 0.01 standard deviations on average, indicating RD's high internal validity. When the study‐specific estimates are shrunken to capitalize on the information the other studies provide, all the RD causal estimates fall within 0.07 standard deviations of their RCT counterparts, now indicating high external validity. With unshrunken estimates, the mean RD bias is still essentially zero, but the distribution of RD bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used.     

The Influence of Low‐Income Children's Participation in Head Start on Their Parents’ Education and Employment

Head Start is the oldest and largest federally funded preschool program in the United States. From its inception in 1965, Head Start not only provided early childhood education, care, and services for children, but also sought to promote parents’ success. However, almost all evaluation studies of Head Start have focused solely on children's cognitive and social outcomes rather than on parents’ outcomes. The present study examines whether children's participation in Head Start promotes parents’ educational advancement and employment. We use data from the Head Start Impact Study (HSIS), a randomized trial of over 4,000 newly entering three‐ and four‐year‐old children. We find that parents of children in the three‐year‐old cohort (but not the four‐year‐old cohort), who were randomly assigned to and participated in Head Start, had steeper increases in their own educational attainment by child age six years compared to parents of children in the control group. This pattern is especially strong for parents who had at least some college experience at baseline, as well as for African‐American parents. We do not find evidence that Head Start helped parents enter or return to the workforce over time. Results are discussed in the context of using high‐quality early childhood education as a platform for improving both child and parent outcomes.     

Nonpoor children in head start: Explanations and implications

According to the Head Start Act (1998), children are income‐eligible for the program if their “families' incomes are below the poverty line.” There are a number of statutory exceptions to this general rule and, according to the Head Start Bureau, the result is that about 6 percent of the children in the program are not poor. But the major national surveys of Head Start families report that 30 percent or more of Head Start children are not “poor.” This paper confirms and explains the high proportion of nonpoor children in Head Start: at enrollment, at least 28 percent are not poor; at midyear, at least 32 percent are not poor; and by the end of the program year, at least 34 percent and perhaps more than 50 percent are not poor. Although the presence of some of these nonpoor children seems to be an appropriate or at least understandable aspect of running a national program with Head Start's current organizational structure, the presence of others seems much less warranted and raises substantial questions of horizontal equity. Moreover, taken together, the large number of nonpoor children suggests that the program is not well targeted to fulfill its mission of providing compensatory services to developmentally disadvantaged children—and reveals the essential ambiguity of Head Start's role in the wider world of early care and education. The income and program dynamics that have led to so many nonpoor children being in Head Start are also at work in many other programs, and, thus, our findings demonstrate the need to understand better how income eligibility is determined across various means‐tested programs.     

Strengthening the Experimenter’s Toolbox: Statistical Estimation of Internal Validity

Experiments have become an increasingly common tool for political science researchers over the last decade, particularly laboratory experiments performed on small convenience samples. We argue that the standard normal theory statistical paradigm used in political science fails to meet the needs of these experimenters and outline an alternative approach to statistical inference based on randomization of the treatment. The randomization inference approach not only provides direct estimation of the experimenter’s quantity of interest—the certainty of the causal inference about the observed units—but also helps to deal with other challenges of small samples. We offer an introduction to the logic of randomization inference, a brief overview of its technical details, and guidance for political science experimenters about making analytic choices within the randomization inference framework. Finally, we reanalyze data from two political science experiments using randomization tests to illustrate the inferential differences that choosing a randomization inference approach can make.     

Designing and Analyzing Randomized Experiments: Application to a Japanese Election Survey Experiment

Randomized experiments are becoming increasingly common in political science. Despite their well-known advantages over observational studies, randomized experiments are not free from complications. In particular, researchers often cannot force subjects to comply with treatment assignment and to provide the requested information. Furthermore, simple randomization of treatments remains the most commonly used method in the discipline even though more efficient procedures are available. Building on the recent statistical literature, we address these methodological issues by offering general recommendations for designing and analyzing randomized experiments to improve the validity and efficiency of causal inference. We also develop a new statistical methodology to explore causal heterogeneity. The proposed methods are applied to a survey experiment conducted during Japan's 2004 Upper House election, where randomly selected voters were encouraged to obtain policy information from political parties' websites. An R package is publicly available for implementing various methods useful for designing and analyzing randomized experiments .     

Survey‐Based Measurement of Public Management and Policy Networks

Networks have become a central concept in the policy and public management literature; however, theoretical development is hindered by a lack of attention to the empirical properties of network measurement methods. This paper compares three survey‐based methods for measuring organizational networks: the roster, the free‐recall name generator, and a hybrid name generator that combines these two classic approaches. Results indicate that the roster and free‐recall name generator methods both suffer from important limitations. The roster method tends to measure many linkages among a limited set of network actors, whereas the name generator tends to measure fewer linkages among a larger set of network actors. Using survey data on policy networks within California regional planning processes (N = 752), we find that the hybrid method strikes an effective balance between these techniques. The hybrid approach performs well in terms of identifying a large number of network actors and interconnections between them. Although no survey‐based measurement technique is perfect, this study suggests that the hybrid name generator is an excellent alternative for the measurement of complex networks with large or shifting boundaries that encompass a diverse set of actors.     

Cost‐Effectiveness of Early Childhood Interventions to Enhance Preschool: Evidence from a Randomized Experiment in Head Start Centers Enrolling Historically Underserved Populations

We evaluate the cost‐effectiveness of two early childhood interventions that use instructional coaching and parent coaching as levers for improvement. The study design allows us to compare the individual effects of each intervention as well as their combined effect on student outcomes. We find that teachers receiving instructional coaching improve their use of evidence‐based instructional practices, while families receiving parent coaching show increases in numerous responsive parenting behaviors associated with positive child outcomes. Both interventions demonstrate positive impacts on students, but only parent coaching shows statistically significant effects across a range of student outcomes. Instructional coaching alone is substantially less costly and may therefore be the most cost‐effective of the three treatment conditions; however, small sample sizes limit our ability to reach definitive conclusions. Policy simulations suggest that implementing these interventions could raise the overall cost‐effectiveness of Head Start by at least 16 percent.     

Approval voting and Shapley ranking

Public Choice - Applying an array of quasi-experimental designs, proponents of causal inference approaches to studying American politics are setting their sights on the study of Congress. In many...     

Factors affecting property tax reliance: Additional evidence

This paper contributes to the understanding of the variation in property tax reliance. Factors thought to underlie the relative use of property taxes in financing local public services are examined. Because of the importance of external debt and tax limits in determining both tax policies and local government organization, particular attention is paid to local government structure and the impact on tax composition. An index from the industrial organization literature is used to study the effects of local government fragmentation. The empirical results are consistent with expectations and indicate that external tax limits may increase property tax reliance by providing incentives to change the structure of local government.     

Compulsory Voting and Parties’ Vote‐Seeking Strategies

I advance a theory about how compulsory voting affects the behavior of political parties. The theory suggests that parties will pivot toward programmatic vote‐seeking strategies and away from clientelistic tactics, such as vote buying, where voting is compulsory. I test my expectations in three separate studies, using several data sources and empirical approaches. In Study 1, cross‐national analyses show that parties behave more programmatically under compulsory voting and that vote buying is less common where voting is mandatory. In Study 2, synthetic control and difference‐in‐differences analyses show that a switch to compulsory voting in Thailand produced an increase in programmatic vote seeking. In Study 3, a list experiment conducted in tandem with a natural experiment shows that compulsory voting leads parties to rely less on vote‐buying tactics in Argentina. I conclude by discussing the implications of these findings, which together are broadly supportive of my theoretical expectations.     

Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons

This paper analyzes 12 recent within‐study comparisons contrasting causal estimates from a randomized experiment with those from an observational study sharing the same treatment group. The aim is to test whether different causal estimates result when a counterfactual group is formed, either with or without random assignment, and when statistical adjustments for selection are made in the group from which random assignment is absent. We identify three studies comparing experiments and regression‐discontinuity (RD) studies. They produce quite comparable causal estimates at points around the RD cutoff. We identify three other studies where the quasi‐experiment involves careful intact group matching on the pretest. Despite the logical possibility of hidden bias in this instance, all three cases also reproduce their experimental estimates, especially if the match is geographically local. We then identify two studies where the treatment and nonrandomized comparison groups manifestly differ at pretest but where the selection process into treatment is completely or very plausibly known. Here too, experimental results are recreated. Two of the remaining studies result in correspondent experimental and nonexperimental results under some circumstances but not others, while two others produce different experimental and nonexperimental estimates, though in each case the observational study was poorly designed and analyzed. Such evidence is more promising than what was achieved in past within‐study comparisons, most involving job training. Reasons for this difference are discussed. © 2008 by the Association for Public Policy Analysis and Management.     

A mixed blessing for the left? Early voting,turnout and election outcomes in Norway

Is there a relationship between turnout and election outcomes? Although this is a classic topic in political science, most studies on multiparty systems have important theoretical and empirical shortcomings. First, we argue that the proper implication of the theoretical argument that underpins research on the turnout-vote nexus is that high levels of turnout should typically benefit both traditional social democratic parties and parties of the radical right relative to other types of parties, including not only those of the traditional right, but also ‘left-libertarian’ parties. Second, few have studied the relationship between turnout and election outcomes with a research design that is appropriate for causal inference. In our empirical study, our identification strategy is to exploit a Norwegian reform of early voting rules as an exogenous source of variation in turnout. Our theoretical expectations are largely borne out in our empirical results.     

The Winner Takes It All: Revisiting the Effect of Direct Democracy on Citizens’ Political Support

Despite the strong theoretical expectations about the beneficial effect of direct democratic instruments on citizens’ political support, the empirical evidence is scarce and inconsistent. We add to this literature by studying the effect of the use of a direct democratic process on citizens’ political support and its underlying causal mechanism. Using a unique research design that combines a strong test of causality with a high level of ecological validity, we surveyed inhabitants of a Belgian neighborhood that held a local referendum and a comparison group (i.e. inhabitants of a comparable neighborhood without referendum) before and after the referendum (n = 1049). Using difference-in-differences analysis and first difference regression analysis, we show that in line with our expectations the increase in political support following the referendum is not driven by involvement or procedural fairness perceptions but by an increase in support levels among the winners of the decision. Moreover, despite the contested nature of the issue, losers’ level of political support did not decrease significantly after the result of the referendum was announced.     

STRENGTHENING THE REGRESSION DISCONTINUITY DESIGN USING ADDITIONAL DESIGN ELEMENTS: A WITHIN‐STUDY COMPARISON

The sharp regression discontinuity design (RDD) has three key weaknesses compared to the randomized clinical trial (RCT). It has lower statistical power, it is more dependent on statistical modeling assumptions, and its treatment effect estimates are limited to the narrow subpopulation of cases immediately around the cutoff, which is rarely of direct scientific or policy interest. This paper examines how adding an untreated comparison to the basic RDD structure can mitigate these three problems. In the example we present, pretest observations on the posttest outcome measure are used to form a comparison RDD function. To assess its performance as a supplement to the basic RDD, we designed a within‐study comparison that compares causal estimates and their standard errors for (1) the basic posttest‐only RDD, (2) a pretest‐supplemented RDD, and (3) an RCT chosen to serve as the causal benchmark. The two RDD designs are constructed from the RCT, and all analyses are replicated with three different assignment cutoffs in three American states. The results show that adding the pretest makes functional form assumptions more transparent. It also produces causal estimates that are more precise than in the posttest‐only RDD, but that are nonetheless larger than in the RCT. Neither RDD version shows much bias at the cutoff, and the pretest‐supplemented RDD produces causal effects in the region beyond the cutoff that are very similar to the RCT estimates for that same region. Thus, the pretest‐supplemented RDD improves on the standard RDD in multiple ways that bring causal estimates and their standard errors closer to those of an RCT, not just at the cutoff, but also away from it.     

The incidence of local government allocations in Tanzania

Since 1999, Tanzania has been actively pursuing reforms of the way in which the central government finances local government activities. This article looks at the potential problems with the mechanism that the central government currently uses to distribute budget resources among the 114 local government authorities and sets out to reveal the incidence of central government allocations to local authorities in Tanzania. Two strands of economics literature—the public choice literature and the political economy literature—consider the distribution of central government resources across local governments. Using regression analysis, this study specifies an incidence model that reveals the determinants of variations in local government allocations in Tanzania. The empirical analysis only finds weak evidence supporting the presumed pro‐poor allocation of some local government resources by the central government in Tanzania. Instead, the study finds more convincing support for substantial pro‐wealthy and pro‐urban tendencies in the way in which central government officials divide public resources across local government units. Copyright © 2003 John Wiley & Sons, Ltd.     

Can Cooperation Be Mandated? Implementing Intergovernmental Environmental Management in New South Wales and New Zealand

Various forces have led to new thinking about approaches tointergovernmental regulation, emphasizing a shift from the heavyhand of coercive mandates to more cooperative approaches. Thisresearch examines the implementation of cooperative policiesaddressing aspects of environmental management in New SouthWales, Australia, and in New Zealand. Analyses are undertakenof the cooperative policy designs, government agencies' implementationefforts and styles, and local government officials' perceptionsof intergovernmental cooperation. Bringing about intergovernmentalcooperation is not automatic, nor does it appear to be easy.An empirical modeling shows that high levels of agency commitmentand capacity are required in order to foster facilitative implementationstyles. The design of policy mandates is important in signalingcooperative desires and structuring implementation to fosterappropriate agency implementation styles.     

Living up to the normative ideal of the human right to adequate housing in urban Ghana

This article constructs a holistic narrative on the right to adequate housing in Ghana by piecing together the fragmented literature on the various dimensions of the right using the traditional “qualitative” approach to literature review. Results of the thematically synthesized literature reveal that housing rights in urban Ghana pale empirically in comparison to its normative tenets. The right is violated in all its dimensions, ranging from high insecurity of tenure to the cultural unsuitability of many high-rise dwellings. Major causes identified include the normative framing of slums as illegitimate, defunct and unenforced rental legislation, poor integration of modern and vernacular housing designs, and poor justiciability of the right within the domestic legal system. The situation in Ghana mirrors those in other countries across the globe—however, with local peculiarities—and points to why empirical work on the right within diverse sociopolitical contexts is necessary.     

A Framework for Dynamic Causal Inference in Political Science

Dynamic strategies are an essential part of politics. In the context of campaigns, for example, candidates continuously recalibrate their campaign strategy in response to polls and opponent actions. Traditional causal inference methods, however, assume that these dynamic decisions are made all at once, an assumption that forces a choice between omitted variable bias and posttreatment bias. Thus, these kinds of “single‐shot” causal inference methods are inappropriate for dynamic processes like campaigns. I resolve this dilemma by adapting methods from biostatistics, thereby presenting a holistic framework for dynamic causal inference. I then use this method to estimate the effectiveness of an inherently dynamic process: a candidate’s decision to “go negative.” Drawing on U.S. statewide elections (2000–2006), I find, in contrast to the previous literature and alternative methods, that negative advertising is an effective strategy for nonincumbents. I also describe a set of diagnostic tools and an approach to sensitivity analysis.     

Presidential address: From the Great Society to continuous improvement government: Shifting from “does it work?” to “what would make it better?”

In the 1960s, various social programs were started (like Head Start) or dramatically expanded (like AFDC). Loosely, this period of expansion is called the Great Society. Too many Great Society social programs, unfortunately, have been disappointments—at least when compared to the high hopes of the '60s. Even if they “work,” most of us wish that they worked much better. Some people take such statements to mean that the Great Society's social programs should be defunded. Most Great Society programs, however, are surely here to stay, for they serve important social functions. How many of us really think there could be an America without a social safety net? It is now time to do the difficult and unglamorous work of systematic program improvement. Instead of testing program efficacy over and over again, we should engage in ongoing and evidence‐based efforts to increase program effectiveness (in both large and small ways). © 2009 by the Association for Public Policy Analysis and Management.