Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons

This paper analyzes 12 recent within‐study comparisons contrasting causal estimates from a randomized experiment with those from an observational study sharing the same treatment group. The aim is to test whether different causal estimates result when a counterfactual group is formed, either with or without random assignment, and when statistical adjustments for selection are made in the group from which random assignment is absent. We identify three studies comparing experiments and regression‐discontinuity (RD) studies. They produce quite comparable causal estimates at points around the RD cutoff. We identify three other studies where the quasi‐experiment involves careful intact group matching on the pretest. Despite the logical possibility of hidden bias in this instance, all three cases also reproduce their experimental estimates, especially if the match is geographically local. We then identify two studies where the treatment and nonrandomized comparison groups manifestly differ at pretest but where the selection process into treatment is completely or very plausibly known. Here too, experimental results are recreated. Two of the remaining studies result in correspondent experimental and nonexperimental results under some circumstances but not others, while two others produce different experimental and nonexperimental estimates, though in each case the observational study was poorly designed and analyzed. Such evidence is more promising than what was achieved in past within‐study comparisons, most involving job training. Reasons for this difference are discussed. © 2008 by the Association for Public Policy Analysis and Management.     

The use of client surveys to gauge the threat of contamination in welfare reform experiments

This paper examines the type of evidence policy analysts have used to identify the presence and magnitude of contamination in welfare reform experiments. Peter Rossi's critique of the New Jersey Family Development Program evaluation motivates the following discussion. In this critique Rossi and others contend that client misperception about experimental control‐group assignment resulted in contamination that negates reported treatment effectiveness. By applying the framework of the Rubin Causal Model (RCM) to isolate “pure” and “impure” experimental and control cases, the actual group assignment and not self‐reported membership is shown to be a more accurate gauge of treatment level and effect. The analysis reveals that the form of contamination Rossi detected leads to underestimates of treatment effects, not their evaporation. While contamination is a legitimate threat in any research design its identification must be based on empirical measures. © 2003 by the Association for Public Policy Analysis and Management.     

Cognitive and non‐cognitive predictors of success in adult education programs: Evidence from experimental data with low‐income welfare recipients

Using data on approximately 2,000 low‐income welfare recipients in a three‐site random‐assignment intervention conducted in the early 1990s (the NEWWS), we examine the role of cognitive and non‐cognitive factors in moderating experimental impacts of an adult education training program for women who lacked a high school degree or GED at the time of random assignment. Both cognitive and noncognitive skills (in particular, locus of control) moderate treatment impacts. For the sample as a whole, assignment to an education‐focused program had a statistically significant (albeit modest) 8 percentage point impact on the probability of degree receipt. For those with low cognitive skills, virtually all of these program impacts were eliminated. However, non‐cognitive skills play a substantively important role such that women with high cognitive skills but low non‐cognitive skills are only half as likely to earn a degree as their counterparts with high skills of both types. © 2008 by the Association for Public Policy Analysis and Management.     

The long‐term effects on children and adolescents of a policy providing work supports for low‐income parents

New Hope, an employment‐based poverty‐reduction intervention for adults evaluated in a random‐assignment experimental design, had positive impacts on children's achievement and social behavior two and five years after random assignment. The question addressed in this paper was the following: Did the positive effects of New Hope on younger children diminish or even reverse when children reached the challenges of adolescence (eight years after random assignment)? Small positive impacts on school progress, school motivation, positive social behavior, child well‐being, and parent control endured, but impacts on school achievement and problem behavior were no longer evident. The most likely reasons for lasting impacts were that New Hope families were slightly less likely to be poor, and children had spent more time in center‐based child care and structured activities. New Hope represents a model policy that could produce modest improvements in the lives of low‐income adults and children. © 2011 by the Association for Public Policy Analysis and Management.     

On Measuring and Reducing Selection Bias With a Quasi‐Doubly Randomized Preference Trial

Randomized experiments provide unbiased estimates of treatment effects, but are costly and time consuming. We demonstrate how a randomized experiment can be leveraged to measure selection bias by conducting a subsequent observational study that is identical in every way except that subjects choose their treatment—a quasi‐doubly randomized preference trial (quasi‐DRPT). Researchers first strive to think of and measure all possible confounders and then determine how well these confounders as controls can reduce or eliminate selection bias. We use a quasi‐DRPT to study the effect of class time on student performance in an undergraduate introductory microeconomics course at a large public university, illustrating its required design elements: experimental and choice arms conducted in the same setting with identical interventions and measurements, and all confounders measured prospectively to treatment assignment or choice. Quasi‐DRPTs augment randomized experiments in real‐world settings where participants choose their treatments.     

Corruption as a Self‐Fulfilling Prophecy: Evidence from a Survey Experiment in Costa Rica

An influential literature argues that corruption behaves as a self‐fulfilling prophecy. Its central claim is that the individual returns to corruption are a function of the perceived corruptibility of the other members of society. Empirically, this implies that if one were to exogenously increase beliefs about societal levels of corruption, willingness to engage in corruption should also increase. We evaluate this implication by utilizing an information experiment embedded in a large‐scale household survey recently conducted in the Gran Área Metropolitana of Costa Rica. Changes in beliefs about corruption were induced via the random assignment of an informational display depicting the increasing percentage of Costa Ricans who have personally witnessed an act of corruption. Consistent with the self‐fulfilling prophecy hypothesis, we find that internalizing the information from the display on average increased the probability that a respondent would be willing to bribe a police officer by approximately .05 to .10.     

The Role of Theory in Experimental Design: Experiments Without Randomization

Some who have written about the logic of experimentation arguethat random assignment of subjects to treatment conditions isan essential attribute of an experiment. Others disagree. Ratherthan treating this as a matter of dueling definitions, we considerexperiments without randomization from a theoretical perspective.Our central contention here is that, for some research questions,theory dictates systematic (not random) assignment of respondentsto experimental conditions. Two such areas of inquiry are researchon political tolerance and on institutional legitimacy. Thisarticle gives cursory attention to the former body of work anddetailed attention to the latter, based on an experiment conductedin a survey in 2001 on the consequences of the American presidentialelection for institutional legitimacy. Because in both instancestheory requires nonrandom assignment, the problem becomes oneof identifying the costs of nonrandomization (threats to internalvalidity) and specifying analytical techniques that might amelioratethose costs. Consequently, we present results from a statisticalapproach that addresses the problem of nonrandomization. Themost important claim of this article is that theory ought tospecify research design, including experimental designs, andthat dogmatic attachment to one definition of experiment willnot serve the discipline of political science.     

Do experimental and nonexperimental evaluations give different answers about the effectiveness of government‐funded training programs?

This paper uses meta‐analysis to investigate whether random assignment (or experimental) evaluations of voluntary government‐funded training programs for the disadvantaged have produced different conclusions than nonexperimental evaluations. Information includes several hundred estimates from 31 evaluations of 15 programs that operated between 1964 and 1998. The results suggest that experimental and nonexperimental evaluations yield similar conclusions about the effectiveness of training programs, but that estimates of average effects for youth and possibly men might have been larger in experimental studies. The results also suggest that variation among nonexprimental estimates of program effects is similar to variation among experimental estimates for men and youth, but not for women (for whom it seems to be larger), although small sample sizes make the estimated differences somewhat imprecise for all three groups. The policy implications of the findings are discussed. © 2006 by the Association for Public Policy Analysis and Management     

How to build a time machine for public affairs executives

Temporal factors such as time and timing are so self‐evident in public affairs that they are rarely mentioned and even less frequently researched. Time is seen as the independent variable—a scaffold that underpins a calendar of communication events, legislative and regulatory processes, or issue lifecycles. However, time is a more complex and essential variable to contemplate than public affairs (and most other) executives give it credit for. This paper explores 16 unique ways to think about time, applying those dimensions to the Amazon HQ2 (second headquarters) selection process to illustrate how public affairs executives could use them to illuminate aspects of strategy and behavior they might otherwise not consider. Strategically, understanding interconnected dimensions of time improves clarity in the choice‐making process, allowing public affairs executives to exploit ideal timing as they execute their strategies.     

Damned If You Do and Damned If You Don’t: Title VII and Public Employee Promotion Disparate Treatment and Disparate Impact Litigation

What has been the impact of the U.S. Supreme Court’s 2009 decision in Ricci v. Destefano on the selection and promotion practices of public employers?; Relying solely on circumstantial evidence, the Supreme Court held that the Civil Service Board of New Haven, Connecticut, had engaged in Title VII disparate treatment discrimination by refusing to certify the results of a promotion examination that led, in turn, to a disparate impact on African American firefighters. To limit the discretion of public employers to disregard such selection and promotion exam results, the Ricci majority held that a public employer must “have a strong basis in evidence to believe it will be subject to disparate‐impact liability if it fails to the take the race‐conscious discriminatory action.” This article argues that the decision effectively prohibits public employers from rejecting the results of selection and promotion instruments, even though there is evidence that screening instruments inequitably affect protected groups. It also forces public employers to become more careful in developing selection and promotion examinations or face the possibility of costly Title VII litigation.     

G.W.F. Hegel on Self‐Determination and Democratic Theory

This article claims that the major alternative models of contemporary democratic theory—the aggregative, deliberative, and agonistic models—are grounded on a norm of self‐determination, but each conceptualizes this self‐determination in a different, and one‐sidedly narrow, way. G.W.F. Hegel provides a conceptual scheme in which to understand the development and synthesize the insights of these three articulations of self‐determination. He also argues that the political embodiment of a complete self‐determination must be founded on economic self‐interest, though a self‐interest expanded to a concern for the common good through the experience of self‐government in one's economic and political associations. Thus, rather than separating economic and political spheres, as contemporary democratic theorists do, Hegel makes a case that modern self‐determination requires a structural harmony between these spheres.     

The Cost‐Effectiveness of Welfare‐to‐Work Programs: A Meta‐Analysis

This article uses the tools of meta‐analysis to assess cost–benefit studies of 50 mandatory welfare‐to‐work programs that were targeted at Aid for Families with Dependent Children recipients and evaluated by random assignment. The findings suggest that the costs of a typical evaluated welfare‐to‐work program probably exceeded its benefits from the perspective of the government, but those assigned to the program, and possibly society as a whole, may reap small positive net benefits. However, there are individual programs that are very cost‐beneficial. Further analyses of the determinants of the cost‐benefits of welfare‐to‐work programs to government and society as a whole show that some services are cost‐neutral, but others entail net costs, which are sometimes substantial. The findings suggest that less successful programs might be made more cost‐beneficial by dropping vocational training and basic education as program components, leaving mainly lower cost components, such as mandated job search and sanctions, but also possibly the more costly financial incentives.     

Democratic Accountability and Retrospective Voting: A Laboratory Experiment

Understanding the incentives of politicians requires understanding the nature of voting behavior. I conduct a laboratory experiment to investigate whether voters focus on the problem of electoral selection or if they instead focus on electoral sanctioning. If voters are forward‐looking but uncertain about politicians’ unobservable characteristics, then it is rational to focus on selection. But doing so undermines democratic accountability because selection renders sanctioning an empty threat. In contrast to rational choice predictions, the experimental results indicate a strong behavioral tendency to use a retrospective voting rule. Additional experiments support the interpretation that retrospective voting is a simple heuristic that voters use to cope with a cognitively difficult inference and decision problem and, in addition, suggest that voters have a preference for accountability. The results pose a challenge for theories of electoral selection and voter learning and suggest new interpretations of empirical studies of economic and retrospective voting.     

Five‐year effects of an anti‐poverty program on marriage among never‐married mothers

Using data from an experimental evaluation of the New Hope project, an anti‐poverty program that increased employment and income, this study examined the effects of New Hope on entry into marriage among never‐married mothers. Among never‐married mothers, New Hope significantly increased rates of marriage. Five years after random assignment, 21 percent of women assigned to the New Hope condition were married, compared to 12 percent of those assigned to the control group. The New Hope impact on marriage was robust to variations in model specification. The program also increased income, wage growth, and goal efficacy among never‐married mothers, and decreased depression. In non‐experimental analyses, income and earnings were associated with higher probability of marriage and material hardship was associated with lower probability of marriage. © 2006 by the Association for Public Policy Analysis and Management     

Can Nonexperimental Estimates Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within‐Study Comparison

The ability of nonexperimental estimators to match impact estimates derived from random assignment is examined using data from the evaluation of two interdistrict magnet schools. As in previous within‐study comparisons, nonexperimental estimates differ from estimates based on random assignment when nonexperimental estimators are implemented without pretreatment measures of academic performance. With comparison groups consisting of students drawn from the same districts or districts with similar student body characteristics as the districts where treatment group students reside, using pretreatment test scores reduces the bias in nonexperimental methods between 64 and 96 percent. Adding pretreatment test scores does not achieve as much bias reduction when the comparison group consists of students drawn from districts with different student body characteristics than the treatment group students’ districts. The results suggest that using pretreatment outcome measures and comparison groups that are geographically aligned with the treatment group greatly improves the performance of nonexperimental estimators.     

Using Preferred Applicant Random Assignment (PARA) to Reduce Randomization Bias in Randomized Trials of Discretionary Programs

Randomization bias occurs when the random assignment used to estimate program effects influences the types of individuals that participate in a program. This paper focuses on a form of randomization bias called “applicant inclusion bias,” which can occur in evaluations of discretionary programs that normally choose which of the eligible applicants to serve. If this nonrandom selection process is replaced by a process that randomly assigns eligible applicants to receive the intervention or not, the types of individuals served by the program—and thus its average impact on program participants—could be affected. To estimate the impact of discretionary programs for the individuals that they normally serve, we propose an experimental design called Preferred Applicant Random Assignment (PARA). Prior to random assignment, program staff would identify their “preferred applicants,” those that they would have chosen to serve. All eligible applicants are randomly assigned, but the probability of assignment to the program is set higher for preferred applicants than for the remaining applicants. This paper demonstrates the feasibility of the method, the cost in terms of increased sample size requirements, and the benefit in terms of improved generalizability to the population normally served by the program.     

Activating the Forces of Public Service Motivation: Evidence from a Low‐Intensity Randomized Survey Experiment

Employees with higher public service motivation (PSM) are likely to perform better in public service jobs. However, research on how practitioners may capitalize on this knowledge is sparse. This article expands the understanding of how to activate employee PSM, which is understood as a human resource that is present in the work environment. Using a randomized survey experiment with 528 law students, this article shows how low‐intensity treatments may activate PSM and how the effect of PSM activation efforts compares with efforts to activate another, less self‐determined type of motivation (relating to the need for feelings of self‐importance). The findings are robust and suggest that low‐intensity efforts to activate PSM have a positive effect on an individual's behavioral inclinations. However, efforts toward the activation of motivation relating to feelings of self‐importance appear to engender an effect of similar size.     

Are Some Citizens More Equal than Others? Evidence from a Field Experiment

The equal treatment of all citizens is one of the fundamental principles of good administrative practice. Nevertheless, there are growing numbers of media and scientific reports on unequal treatment by public administrations. This article examines the unequal treatment of citizens by gender and ethnic origin by means of a survey‐based field experiment in German local government. With the help of two vignettes and randomized assignment of names, responses to fake citizen requests by local governments are analyzed for speed, quality, and service orientation. The results show very limited discrimination effects. While there is no evidence for general ethnic discrimination, a more differentiated analysis indicates patterns of ethnic discrimination conditioned by gender.     

The importance of industry compliance in improving advertising self‐regulatory processes

As the most visible element of the marketing communications mix, advertising has had its critics and, given the choice, developed countries usually select a self‐regulatory approach to deal with unacceptable advertising. The recent breakdown of one of the world's longest established advertising self‐regulatory programmes in Australia has reopened the 20‐year old debate that has taken place in the leading academic and business journals concerned with enhancing understanding of such regulatory systems. This paper focuses on the activity of code enforcement in improving the effectiveness of advertising self‐regulatory frameworks. The key findings of an Australian study, where its scheme has recently failed, are presented and discussed in this context and a key concept is developed. Copyright © 2003 Henry Stewart Publications     

Self‐Provision of Public Services: Its Evolution and Impact

This article establishes a framework for explaining the ways in which citizens, as clients of public services, attempt to deal with situations of combined market and government failures. Under certain conditions, citizens are driven to create self‐production mechanisms that often are extralegal or illegal. When faced with such social initiatives, politicians often support them, either passively or actively, by institutionalizing the new mechanisms. The article explains the evolution of the self‐provision model and its implications. The analysis includes a theoretical framework and a practical intervention scheme.