When Should We Use Unit Fixed Effects Regression Models for Causal Inference with Longitudinal Data?

Many researchers use unit fixed effects regression models as their default methods for causal inference with longitudinal data. We show that the ability of these models to adjust for unobserved time‐invariant confounders comes at the expense of dynamic causal relationships, which are permitted under an alternative selection‐on‐observables approach. Using the nonparametric directed acyclic graph, we highlight two key causal identification assumptions of unit fixed effects models: Past treatments do not directly influence current outcome, and past outcomes do not affect current treatment. Furthermore, we introduce a new nonparametric matching framework that elucidates how various unit fixed effects models implicitly compare treated and control observations to draw causal inference. By establishing the equivalence between matching and weighted unit fixed effects estimators, this framework enables a diverse set of identification strategies to adjust for unobservables in the absence of dynamic causal relationships between treatment and outcome variables. We illustrate the proposed methodology through its application to the estimation of GATT membership effects on dyadic trade volume.     

On Measuring and Reducing Selection Bias With a Quasi‐Doubly Randomized Preference Trial

Randomized experiments provide unbiased estimates of treatment effects, but are costly and time consuming. We demonstrate how a randomized experiment can be leveraged to measure selection bias by conducting a subsequent observational study that is identical in every way except that subjects choose their treatment—a quasi‐doubly randomized preference trial (quasi‐DRPT). Researchers first strive to think of and measure all possible confounders and then determine how well these confounders as controls can reduce or eliminate selection bias. We use a quasi‐DRPT to study the effect of class time on student performance in an undergraduate introductory microeconomics course at a large public university, illustrating its required design elements: experimental and choice arms conducted in the same setting with identical interventions and measurements, and all confounders measured prospectively to treatment assignment or choice. Quasi‐DRPTs augment randomized experiments in real‐world settings where participants choose their treatments.     

Does WIC work? The effects of WIC on pregnancy and birth outcomes

Support for WIC, the Special Supplemental Nutrition Program for Women, Infants, and Children, is based on the belief that "WIC works." This consensus has lately been questioned by researchers who point out that most WIC research fails to properly control for selection into the program. This paper evaluates the selection problem using rich data from the national Pregnancy Risk Assessment Monitoring System. We show that relative to Medicaid mothers, all of whom are eligible for WIC, WIC participants are negatively selected on a wide array of observable dimensions, and yet WIC participation is associated with improved birth outcomes, even after controlling for observables and for a full set of state-year interactions intended to capture unobservables that vary at the state-year level. The positive impacts of WIC are larger among subsets of even more disadvantaged women, such as those who received public assistance last year, single high school dropouts, and teen mothers.     

Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons

This paper analyzes 12 recent within‐study comparisons contrasting causal estimates from a randomized experiment with those from an observational study sharing the same treatment group. The aim is to test whether different causal estimates result when a counterfactual group is formed, either with or without random assignment, and when statistical adjustments for selection are made in the group from which random assignment is absent. We identify three studies comparing experiments and regression‐discontinuity (RD) studies. They produce quite comparable causal estimates at points around the RD cutoff. We identify three other studies where the quasi‐experiment involves careful intact group matching on the pretest. Despite the logical possibility of hidden bias in this instance, all three cases also reproduce their experimental estimates, especially if the match is geographically local. We then identify two studies where the treatment and nonrandomized comparison groups manifestly differ at pretest but where the selection process into treatment is completely or very plausibly known. Here too, experimental results are recreated. Two of the remaining studies result in correspondent experimental and nonexperimental results under some circumstances but not others, while two others produce different experimental and nonexperimental estimates, though in each case the observational study was poorly designed and analyzed. Such evidence is more promising than what was achieved in past within‐study comparisons, most involving job training. Reasons for this difference are discussed. © 2008 by the Association for Public Policy Analysis and Management.     

Can Nonexperimental Estimates Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within‐Study Comparison

The ability of nonexperimental estimators to match impact estimates derived from random assignment is examined using data from the evaluation of two interdistrict magnet schools. As in previous within‐study comparisons, nonexperimental estimates differ from estimates based on random assignment when nonexperimental estimators are implemented without pretreatment measures of academic performance. With comparison groups consisting of students drawn from the same districts or districts with similar student body characteristics as the districts where treatment group students reside, using pretreatment test scores reduces the bias in nonexperimental methods between 64 and 96 percent. Adding pretreatment test scores does not achieve as much bias reduction when the comparison group consists of students drawn from districts with different student body characteristics than the treatment group students’ districts. The results suggest that using pretreatment outcome measures and comparison groups that are geographically aligned with the treatment group greatly improves the performance of nonexperimental estimators.     

Connections Matter: How Interactive Peers Affect Students in Online College Courses

Peers affect individual's productivity in the workforce, in education, and in other team‐based tasks. Using large‐scale language data from an online college course, we measure the impacts of peer interactions on student learning outcomes and persistence. In our setting, students are quasi‐randomly assigned to peers, and as such, we are able to overcome selection biases stemming from endogenous peer grouping. We also mitigate reflection bias by utilizing rich student interaction data. We find that females and older students are more likely to engage in student interactions. Students are also more likely to interact with peers of the same gender and with peers from roughly the same geographic region. For students who are relatively less likely to be engaged in online discussion, exposure to more interactive peers increases their probabilities of passing the course, improves their grade in the course, and increases their likelihood of enrolling in the following academic term. This study demonstrates how the use of large‐scale, text‐based data can provide insights into students’ learning processes.     

Does Regression Produce Representative Estimates of Causal Effects?

With an unrepresentative sample, the estimate of a causal effect may fail to characterize how effects operate in the population of interest. What is less well understood is that conventional estimation practices for observational studies may produce the same problem even with a representative sample. Causal effects estimated via multiple regression differentially weight each unit's contribution. The “effective sample” that regression uses to generate the estimate may bear little resemblance to the population of interest, and the results may be nonrepresentative in a manner similar to what quasi‐experimental methods or experiments with convenience samples produce. There is no general external validity basis for preferring multiple regression on representative samples over quasi‐experimental or experimental methods. We show how to estimate the “multiple regression weights” that allow one to study the effective sample. We discuss alternative approaches that, under certain conditions, recover representative average causal effects. The requisite conditions cannot always be met.     

THE INTERNAL AND EXTERNAL VALIDITY OF THE REGRESSION DISCONTINUITY DESIGN: A META‐ANALYSIS OF 15 WITHIN‐STUDY COMPARISONS

Theory predicts that regression discontinuity (RD) provides valid causal inference at the cutoff score that determines treatment assignment. One purpose of this paper is to test RD's internal validity across 15 studies. Each of them assesses the correspondence between causal estimates from an RD study and a randomized control trial (RCT) when the estimates are made at the same cutoff point where they should not differ asymptotically. However, statistical error, imperfect design implementation, and a plethora of different possible analysis options, mean that they might nonetheless differ. We test whether they do, assuming that the bias potential is greater with RDs than RCTs. A second purpose of this paper is to investigate the external validity of RD by exploring how the size of the bias estimates varies across the 15 studies, for they differ in their settings, interventions, analyses, and implementation details. Both Bayesian and frequentist meta‐analysis methods show that the RD bias is below 0.01 standard deviations on average, indicating RD's high internal validity. When the study‐specific estimates are shrunken to capitalize on the information the other studies provide, all the RD causal estimates fall within 0.07 standard deviations of their RCT counterparts, now indicating high external validity. With unshrunken estimates, the mean RD bias is still essentially zero, but the distribution of RD bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used.     

External Validity in Policy Evaluations That Choose Sites Purposively

Evaluations of the impact of social programs are often carried out in multiple sites, such as school districts, housing authorities, local TANF offices, or One‐Stop Career Centers. Most evaluations select sites purposively following a process that is nonrandom. Unfortunately, purposive site selection can produce a sample of sites that is not representative of the population of interest for the program. In this paper, we propose a conceptual model of purposive site selection. We begin with the proposition that a purposive sample of sites can usefully be conceptualized as a random sample of sites from some well‐defined population, for which the sampling probabilities are unknown and vary across sites. This proposition allows us to derive a formal, yet intuitive, mathematical expression for the bias in the pooled impact estimate when sites are selected purposively. This formula helps us to better understand the consequences of selecting sites purposively, and the factors that contribute to the bias. Additional research is needed to obtain evidence on how large the bias tends to be in actual studies that select sites purposively, and to develop methods to increase the external validity of these studies. © 2012 by the Association for Public Policy Analysis and Management.     

University Technology Transfer and Manufacturing Innovation: The Case of Italy

This article analyzes the influence of universities on Italian firms' probability to innovate. Using firm‐level data, we focus on institutionalized technology transfer (TT) activities in universities, namely spin‐offs, patents, and research contracts. Results show that TT activities play a significant role in the probability to innovate by Italian manufacturing firms located in the same province as the university. Nevertheless, the effect is not uniform: the contribution of university TT activities to the probability of firms' innovating is concentrated in certain territorial areas (North‐East and Center) and sectors (science based and scale intensive) and among firms that are large.     

A Meta‐Analysis of the Relationship between Public Service Motivation and Job Satisfaction

In recent years, much research has been conducted on the relationship between public service motivation (PSM) and various outcomes, including job satisfaction. This article presents a meta‐analysis aggregating the effects of PSM on job satisfaction. Meta‐regression analysis is used to assess the impact of numerous study characteristics and to identify potential issues of publication bias. The findings, based on 28 separate studies, show no evidence of publication bias and support the positive relationship between PSM and job satisfaction. Furthermore, the results support the importance of providing individuals with the opportunity to serve the public within this relationship. Given the organizational benefits that can be derived from improved job satisfaction and the focus of PSM research on its implications for job satisfaction, these findings are of interest to both academics and practitioners in the field of public administration.     

Executive Orders and Administrative Control

One of the key recommendations of the Winter Commission was the empowerment of governors over the executive branch. However, key institutions have not evolved in this direction; the long ballot still exists in most states, and the formal powers of governors have strengthened to their probable capacity. The authors suggest that a quasi‐formal power—the gubernatorial use of executive orders—may be a significant tool for empowering the governor in the state administrative realm. Analyzing all executive orders in 49 states for 2004 and 2005, they find variation in the aggregate use of and functions performed through these orders. Many executive orders do allow the government more direction and control of state bureaucracy. Finally, the authors suggest that the study of executive orders may be necessary to understand gubernatorial power in the executive arena and beyond.     

Identifying the Source of Incumbency Advantage through a Constitutional Reform

This study provides one of the first causal estimates of both the personal and partisan incumbency advantages. Using data on six local elections taking place during the last 20 years in 278 municipalities in Portugal, it relies on a reform introducing mayoral term limits as a natural experiment that creates exogenous variation on the incumbency status of officeholders while holding the incumbency status of the party constant. A new methodology combining two quasi‐experimental methods, the regression discontinuity and the difference‐in‐discontinuities designs, allows for a credible estimation of the independent personal and partisan returns to incumbency. Results causally identify the personal effect as the driver of the incumbency advantage.     

Does Collaboration Make Any Difference? Linking Collaborative Governance to Environmental Outcomes

This paper addresses two research questions: (1) Does collaborative environmental governance improve environmental outcomes? and (2) How do publicly supported collaborative groups with different levels of responsibility, formalization, and representativeness compare in this regard? Using a representative watershed quality data series, the EPA's National Rivers and Streams Assessment and Wadeable Streams Assessment, in conjunction with a watershed management regime database coded for this analysis, I test the relationship between collaborative governance and watershed quality for 357 watersheds. Since these are observational data, a multilevel propensity score matching method is used to control for selection bias. Using an augmented inverse propensity weighted estimator, I estimate the average treatment effect on the treated for six different water quality and habitat condition metrics. Collaborative watershed groups are found to improve water chemistry and in‐stream habitat conditions. I then use hierarchical linear regression modeling to examine how group responsibilities, membership diversity, and formalization affect the predicted impact of a collaborative group. Groups that engage in management activities (in comparison to coordination or planning) are found to achieve greater environmental gains. Limited differentiation is found with regards to the presence of a group coordinator, increased goal specificity, or greater stakeholder diversity.     

Behind the Curve: Clarifying the Best Approach to Calculating Predicted Probabilities and Marginal Effects from Limited Dependent Variable Models

Models designed for limited dependent variables are increasingly common in political science. Researchers estimating such models often give little attention to the coefficient estimates and instead focus on marginal effects, predicted probabilities, predicted counts, etc. Since the models are nonlinear, the estimated effects are sensitive to how one generates the predictions. The most common approach involves estimating the effect for the “average case.” But this approach creates a weaker connection between the results and the larger goals of the research enterprise and is thus less preferable than the observed‐value approach. That is, rather than seeking to understand the effect for the average case, the goal is to obtain an estimate of the average effect in the population. In addition to the theoretical argument in favor of the observed‐value approach, we illustrate via an empirical example and Monte Carlo simulations that the two approaches can produce substantively different results.     

Comparing Inference Approaches for RD Designs: A Reexamination of the Effect of Head Start on Child Mortality

The regression discontinuity (RD) design is a popular quasi‐experimental design for causal inference and policy evaluation. The most common inference approaches in RD designs employ “flexible” parametric and nonparametric local polynomial methods, which rely on extrapolation and large‐sample approximations of conditional expectations using observations somewhat near the cutoff that determines treatment assignment. An alternative inference approach employs the idea of local randomization, where the very few units closest to the cutoff are regarded as randomly assigned to treatment and finite‐sample exact inference methods are used. In this paper, we contrast these approaches empirically by re‐analyzing the influential findings of Ludwig and Miller ( 2007 ), who studied the effect of Head Start assistance on child mortality employing parametric RD methods. We first review methods based on approximations of conditional expectations, which are relatively well developed in the literature, and then present new methods based on randomization inference. In particular, we extend the local randomization framework to allow for parametric adjustments of the potential outcomes; our extended framework substantially relaxes strong assumptions in prior literature and better resembles other RD inference methods. We compare all these methods formally, focusing on both estimands and inference properties. In addition, we develop new approaches for randomization‐based sensitivity analysis specifically tailored to RD designs. Applying all these methods to the Head Start data, we find that the original RD treatment effect reported in the literature is quite stable and robust, an empirical finding that enhances the credibility of the original result. All the empirical methods we discuss are readily available in general purpose software in R and Stata; we also provide the dataset and software code needed to replicate all our results.     

Is Voting Habit Forming? New Evidence from Experiments and Regression Discontinuities

Field experiments and regression discontinuity designs test whether voting is habit forming by examining whether a random shock to turnout in one election affects participation in subsequent elections. We contribute to this literature by offering a vast amount of new statistical evidence on the long‐term consequences of random and quasi‐random inducements to vote. The behavior of millions of voters confirms the persistence of voter turnout and calls attention to theoretically meaningful nuances in the development and expression of voting habits. We suggest that individuals become habituated to voting in particular types of elections. The degree of persistence appears to vary by electoral context and by the attributes of those who comply with an initial inducement to vote.     

A Closer Look at Reporting Bias in Conflict Event Data

Recent data collections about political violence are frequently based on media reports, which can lead to reporting bias. This is an issue in particular for the emergent literature on communication technology and conflict, since this technology may not only affect violence, but also the reporting about it. Using the effect of cellphones on violence as an example, this article presents a quantitative assessment of reporting bias in a micro‐level analysis. Comparing media‐based event reports and those from military sources, the results show that the purported violence‐increasing effect of cellphone coverage is partly due to higher reporting rates of violence in cellphone‐covered areas. A simple diagnostic procedure for this problem is implemented. Applied to the analysis of cellphones and violence in Africa, it produces a pattern that is consistent with reporting bias driving much of the effect found in the Pierskalla and Hollenbach (2013) study about this topic.     

CORPORATE TAX CUTS AND FOREIGN DIRECT INVESTMENT

Accurate policy evaluation is central to optimal policymaking, but difficult to achieve. Most often, analysts have to work with observational data and cannot directly observe the counterfactual of a policy to assess its effect accurately. In this paper, we craft a quasi‐experimental design and apply two relatively new methods—the difference‐in‐differences estimation and the synthetic controls method—to the policy debate on whether corporate tax cuts increase foreign direct investment (FDI). The taxation–FDI relationship has attracted wide attention because of mixed findings. We exploit a quasi‐experimental design for Russian regions, which were granted autonomy to reduce corporate profit tax in 2003, enabling them to simultaneously experiment with different tax policies. We estimate both the average and local treatment effects of two types of tax cuts on FDI inflows. We find that, on average, relative to the absence of tax cuts, nondiscriminatory tax cuts on direct investment profit increase FDI, but discriminatory tax cuts on selected government‐sanctioned investment projects do not. Yet for both types of tax cuts, local treatment effects vary dramatically from region to region. Our research has important implications for the design of tax policy and fiscal incentive, and the assessment of fiscal policy reforms.