On Measuring and Reducing Selection Bias With a Quasi‐Doubly Randomized Preference Trial

Randomized experiments provide unbiased estimates of treatment effects, but are costly and time consuming. We demonstrate how a randomized experiment can be leveraged to measure selection bias by conducting a subsequent observational study that is identical in every way except that subjects choose their treatment—a quasi‐doubly randomized preference trial (quasi‐DRPT). Researchers first strive to think of and measure all possible confounders and then determine how well these confounders as controls can reduce or eliminate selection bias. We use a quasi‐DRPT to study the effect of class time on student performance in an undergraduate introductory microeconomics course at a large public university, illustrating its required design elements: experimental and choice arms conducted in the same setting with identical interventions and measurements, and all confounders measured prospectively to treatment assignment or choice. Quasi‐DRPTs augment randomized experiments in real‐world settings where participants choose their treatments.     

How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It

In principle, experiments offer a straightforward method for social scientists to accurately estimate causal effects. However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation. Typical examples include controlling for posttreatment variables in statistical models, eliminating observations based on posttreatment criteria, or subsetting the data based on posttreatment variables. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects. Moreover, problems associated with conditioning on posttreatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals. We demonstrate the severity of experimental posttreatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real‐world data. We conclude by providing applied researchers with recommendations for best practice.     

The Intersection of Redistricting,Race, and Participation

The drawing of congressional district lines can significantly reduce political participation in U.S. House elections, according to recent work. But such studies have failed to explain which citizens’ voting rates are most susceptible to the dislocating effects of redistricting and whether the findings are generalizable to a variety of political contexts. Building on this nascent literature and work on black political participation, we show that redistricting's negative effects on participation—measured by voter roll‐off in U.S. House elections—are generally strongest among African Americans, but that black voters can be mobilized when they are redrawn into a black representative's congressional district. Our findings, based on data from 11 postredistricting elections in five states from 1992 through 2006, both expand the empirical scope of previous work and suggest that redistricting plays a previously hidden role in affecting black participation in congressional contests.     

The long‐term effects of housing assistance on work and welfare

This paper examines the long‐term effect of project‐based housing assistance—public housing and private assisted housing—on work, earnings, and welfare receipt. We use the Panel Study of Income Dynamics–Assisted Housing Database to identify women ever living in project‐based assisted housing and to create comparison groups using propensity scores. Analyses show no evidence that moving into this type of assisted housing is associated with sustained reductions in employment rates, work hours, or earnings. Although welfare rates decline, they remain higher for assisted housing recipients compared with nonrecipients. Overall, these findings align with those of recent experimental and nonexperimental studies that include other forms of housing assistance and cover different time periods. © 2009 by the Association for Public Policy Analysis and Management.     

Does the Motivation for Market‐Based Reform Matter? The Case of Responsibility‐Centered Management

Organizational theorists have long examined the implications of market‐oriented policies for public agencies. Current research often aims to understand the effects of policies imposed on organizations by external stakeholder groups, but few studies have attempted to gain a better understanding of what mechanisms cause agencies to select into these strategies. The purpose of this article is to understand, first, which factors make an organization more likely to adopt a decentralized, market‐based budgeting system—termed “responsibility‐centered management (RCM)—and, second, whether this type of system has implications for organizational performance. Using data on doctorate‐granting public and private nonprofit four‐year universities in the United States, the authors find that mission, resource dependence, and state party control influence the take‐up of RCM. In terms of effects, RCM creates winners for graduation rates (white students) and degree production (science, technology, engineering, and math departments), which raises questions of equity across groups.     

Only When We Agree! How Value Congruence Moderates the Impact of Goal‐Oriented Leadership on Public Service Motivation

Questions of how and when managers can motivate the workforce of public organizations are fundamental for scholars and practitioners alike. A dominant assertion is that goal‐oriented leadership strategies, such as transformational leadership, foster public service motivation (PSM). However, existing studies rely on designs that are vulnerable to endogeneity and rarely investigate the scope conditions of the leadership‐PSM relationship. Combining a field experiment with 364 managers and surveys of their 3,470 employees, the authors show that transformational leadership and transactional leadership, when induced experimentally, do not have the claimed positive effect on PSM. In fact, the results indicate that goal‐oriented leadership can have demotivating effects when employee and organizational values are incongruent. Public managers should therefore carefully assess existing levels of value (in)congruence before implementing goal‐oriented leadership strategies, and—in case of value conflicts—seek to align perceptions of the desirable among members of the organization.     

CORPORATE TAX CUTS AND FOREIGN DIRECT INVESTMENT

Accurate policy evaluation is central to optimal policymaking, but difficult to achieve. Most often, analysts have to work with observational data and cannot directly observe the counterfactual of a policy to assess its effect accurately. In this paper, we craft a quasi‐experimental design and apply two relatively new methods—the difference‐in‐differences estimation and the synthetic controls method—to the policy debate on whether corporate tax cuts increase foreign direct investment (FDI). The taxation–FDI relationship has attracted wide attention because of mixed findings. We exploit a quasi‐experimental design for Russian regions, which were granted autonomy to reduce corporate profit tax in 2003, enabling them to simultaneously experiment with different tax policies. We estimate both the average and local treatment effects of two types of tax cuts on FDI inflows. We find that, on average, relative to the absence of tax cuts, nondiscriminatory tax cuts on direct investment profit increase FDI, but discriminatory tax cuts on selected government‐sanctioned investment projects do not. Yet for both types of tax cuts, local treatment effects vary dramatically from region to region. Our research has important implications for the design of tax policy and fiscal incentive, and the assessment of fiscal policy reforms.     

Did Ontario's Zero Tolerance & Graduated Licensing Law reduce youth drunk driving?

On April 1, 1994, Ontario, Canada, instituted a new graduated driver license (GDL) system that effectively set the legal blood alcohol content (BAC) threshold at zero for the first few years of a youth's driving eligibility. I use data from the 1983–2001 Ontario Student Drug Use Surveys (OSDUS) to examine whether the Zero Tolerance (ZT) policy reduced self‐reported drinking and alcohol‐involved driving among youth. I find that rates of drunk driving reported by 16‐ to 17‐year‐olds—who faced new, lower legal limits after adoption of the ZT policy—were about 5 percentage points lower after the law was implemented. Visual inspection of the data, however, shows that the estimated reduction is an artifact of a pre‐existing trend: Drunk driving rates in this age group were falling steadily throughout the 1980s and into the 1990s. Estimates that account for this pre‐existing trend or that consider shorter windows around the 1994 implementation date return effects on alcohol‐involved driving that are either small and statistically insignificant or large and implausibly signed (positive). These null findings are robust to using the associated change in outcomes for slightly younger (14–15) or slightly older (19–20) youths as controls in a difference‐in‐differences framework. I similarly find no robust effect on drinking participation. This suggests that Ontario's age‐targeted drunk driving law—despite being harsher than similar policies in the United States—was not responsible for reductions in Canadian youth road fatalities over the past two decades. © 2006 by the Association for Public Policy Analysis and Management     

Learning More from Political Communication Experiments: Pretreatment and Its Effects

Research on political communication effects has enjoyed great progress over the past 25 years. A key ingredient underlying these advances is the increased usage of experiments that demonstrate how communications influence opinions and behaviors. Virtually all of these studies pay scant attention to events that occur prior to the experiment—that is, in “pretreatment events.” In this article, we explore how and when the pretreatment environment affects experimental outcomes. We present two studies—one where we control the pretreatment environment and one where it naturally occurred—to show how pretreatment effects can influence experimental outcomes. We argue that, under certain conditions, attending to pretreatment dynamics leads to novel insights, including a more accurate portrait of the pliability of the mass public and the identification of potentially two groups of citizens—what we call malleability reactive and dogmatic.     

The efficacy and effect of racial profiling: A mathematical simulation approach

Racial profiling—the use of race, ethnicity, or national origin by law enforcement officials to make judgments of criminal suspicion—is assessed in terms of its effect on targeted populations and on law enforcement efficiency. A mathematical simulation, comparing multiple profiling and non‐profiling scenarios, is employed. This analysis indicates that racial profiling exacerbates incarceration disparities between groups whether or not the groups differ in criminality rates, and that the long‐term effects of profiling in terms of criminal captures depend on the calibration of profiling rates to criminality rates. The highest long‐term criminal capture rates appear to occur when stop rate ratios match, or are slightly below, criminality rate ratios between groups. When the possibility of a deterrent effect is modeled, profiling appears to yield fewer criminal captures and have little or no crime reduction effect, and may even increase overall crime rates. © 2006 by the Association for Public Policy Analysis and Management     

Conducting Meta-Analyses of Evaluations of Government-Funded Training Programs

Government‐funded training programs in the United States have often been subject to rigorous evaluation. Indeed, many of these programs have been evaluated with random assignment, although sophisticated quasi‐experimental methods have also been used. Until very recently, however, there has been little systematic attempt to use the cumulative information vested in these evaluations to attempt determine which kinds of programs work best in which setting and with respect to which types of client. Meta‐analysis—a set of statistical procedures for systematically synthesizing findings from separate studies—can, in theory at least, address these and other topics that evaluation of individual programs cannot. This article discusses the steps in conducting such a synthesis, summarizes the results of three recently conducted meta‐analyses of training and welfare‐to‐work programs, identifies limitations to the meta‐analytic approach, and considers ways in which some of these limitations can be overcome.     

The Rise and Fall of Power Line EMFs: The Anatomy of a Magnetic Controversy

The suggestion that power line electromagnetic fields (EMFs) cause diseases like cancer has generated dozens of popular articles and television news segments, hundreds of scientific studies, and numerous consensus reports; it has attracted the attention of epidemiologists, biologists, physicists, policymakers and lawyers. This article will examine the evolution of this controversy through a detailed analysis of the arguments that have been used for and against the hypothesis that power line EMFs have adverse health effects. This article argues that the power line EMF issue provides a classic case study for exploring the challenges citizens, scientists, and policymakers face in sorting out a complex science‐based controversy. This story not only brings together many different perspectives and—from popular notions of cancer clusters to complex epidemiological arguments, from to state‐of‐the‐art animal studies to policy instruments such as the precautionary principle—but also reveals the manner in which a heated controversy can be effectively resolved over time.     

The politics of middle ageism

Abstract

Ageism directed at those in their middle years—middle ageism—is being given teeth by midlife downsizing, drops in real income and workforce participation, substandard employment, hiring discrimination, and court decisions that weaken age anti‐discrimination laws. This essay explores the high costs of this trend—not only for individuals currently at midlife, but for their families, the next cohorts aging into the middle years, the value of seniority, the power of employers over workers in the global workforce, and the meaning of the life course. In Europe premature superannuation became a problem decades ago. Middle ageism is becoming a problem in the US, ironically, at the same time that social justice movements have produced increases in peak wages and expectations of “seniority,” broadly defined, for women and African‐Americans. The market's attempt to drive wages to the bottom thus clashes with expanding cultural visions of age‐graded progress. This essay speculates that middle ageism may have negative effects on the age‐wage curve. Ending with a call to institute “full‐employment‐plus” (a set of programs that take life‐course issues into full consideration), the essay argues that academics and the public need to become more alert to the social construction of the midlife and the life course and the manipulation of age as a powerful category of difference.     

Partisan Elites as Culprits? How Party Cues Shape Partisan Perceptual Gaps

Partisanship often colors how citizens perceive real‐world conditions. For example, an oft‐documented finding is that citizens tend to view the state of the national economy more positively if their party holds office. These partisan perceptual gaps are usually taken as a result of citizens' own motivated reasoning to defend their party identity. However, little is known about the extent to which perceptual gaps are shaped by one of the most important forces in politics: partisan elites. With two studies focusing on perceptions of the economy—a quasi‐experimental panel study and a randomized experiment—we show how partisan perceptual differences are substantially affected by messages coming from party elites. These findings imply that partisan elites are more influential on, and more responsible for, partisan perceptual differences than previous studies have revealed.     

Are Parental Welfare Work Requirements Good for Disadvantaged Children? Evidence From Age‐of‐Youngest‐Child Exemptions

This paper assesses the impact of welfare reform's parental work requirements on low‐income children's cognitive and social‐emotional development. The identification strategy exploits an important feature of the work requirement rules—namely, age‐of‐youngest‐child exemptions—as a source of quasi‐experimental variation in first‐year maternal employment. The 1996 welfare reform law empowered states to exempt adult recipients from the work requirements until the youngest child reaches a certain age. This led to substantial variation in the amount of time that mothers can remain home with a newborn child. I use this variation to estimate the impact of work‐requirement‐induced increases in maternal employment. Using a sample of infants from the Birth cohort of the Early Childhood Longitudinal Study, the reduced form and instrumental variables estimates reveal sizable negative effects of maternal employment. An auxiliary analysis of mechanisms finds that working mothers experience an increase in depressive symptoms, and are less likely to breastfeed and read to their children. In addition, such children are exposed to nonparental child care arrangements at a younger age, and they spend more time in these settings throughout the first year of life.     

Differential Registration Bias in Voter File Data: A Sensitivity Analysis Approach

The widespread availability of voter files has improved the study of participation in American politics, but the lack of comprehensive data on nonregistrants creates difficult inferential issues. Most notably, observational studies that examine turnout rates among registrants often implicitly condition on registration, a posttreatment variable that can induce bias if the treatment of interest also affects the likelihood of registration. We introduce a sensitivity analysis to assess the potential bias induced by this problem, which we call differential registration bias. Our approach is most helpful for studies that estimate turnout among registrants using posttreatment registration data, but it is also valuable for studies that estimate turnout among the voting‐eligible population using secondary sources. We illustrate our approach with two studies of voting eligibility effects on subsequent turnout among young voters. In both cases, eligibility appears to decrease turnout, but these effects are found to be highly sensitive to differential registration bias.     

Legislative Victory, Electoral Uncertainty: Explaining Outcomes in the Battles over Lesbian and Gay Civil Rights

Previous studies indicate that in cases of relatively low issue salience, the interest group model best explains lesbian and gay antidiscrimination policy in the American states. The analysis of state and local public policy prohibiting discrimination on the basis of sexual orientation concludes that for cases of high issue salience, the morality politics model best describes outcomes. The interest group politics model is used here in a case study of Wisconsin's passage of a comprehensive antidiscrimination policy, while the morality politics model is used to investigate the electoral outcomes of anti‐gay ballot initiatives in several states. The results of this analysis conform with prior research—when lesbian and gay issues are not salient, the interest group politics model best explains resulting policy, however, under salient conditions, the morality politics model best describes outcomes. Finally, the implications of this research for social scientists and activists are discussed.     

Location‐Efficient Mortgages: Is the Rationale Sound?

Location efficient mortgage (LEM) programs are an increasingly popular approach to combating urban sprawl. LEMs allow families who want to live in densely populated, transit‐rich communities to obtain a larger mortgage with a smaller down payment than traditional underwriting guidelines allow. LEMs are premised on the proposition that homeowners in such “location‐efficient” areas can safely be allowed to breach underwriting guidelines designed to prevent mortgage default because they have lower than average automobile‐related transportation expenses and more income available for mortgage payments. This paper employs records of more than 8000 FHA‐insured mortgages matched with data on various measures of location efficiency to test this proposition. The results suggest that it does not hold and that LEMs—like other low‐down‐payment mortgage programs—will raise mortgage default rates. This cost must be weighed against any potential anti‐sprawl benefits LEMs may have. © 2001 by the Association for Public Policy Analysis and Management.