What Can We Learn From A Doubly Randomized Preference Trial?—An Instrumental Variables Perspective

The doubly randomized preference trial (DRPT) is a randomized experimental design with three arms: a treatment arm, a control arm, and a preference arm. The design has useful properties that have gone unnoticed in the applied and methodological literatures. This paper shows how to interpret the DRPT design using an instrumental variables (IV) framework. The IV framework reveals that the DRPT separately identifies three different treatment effect parameters: the Average Treatment Effect (ATE), the Average Treatment Effect on the Treated (ATT), and the Average Treatment Effect on the Untreated (ATU). The ATE, ATT, and ATU parameters are important for program evaluation research because in realistic settings many social programs are optional rather than mandatory and some people who are eligible for a program choose not to participate. Most of the paper is concerned with the interpretation of the research design. To make the ideas concrete, the final section provides an empirical example using data from an existing DRPT study.     

Using implementation research to solve implementation problems: The case of energy emergency assistance

Generalizations drawn from implementation research can offer useful guidance in formulating new implementation plans, as the development of a revenue-recycling program illustrates. According to the conventional wisdom drawn from implementation research, the program design should be simple and all the steps necessary to produce the desired results should be identified at the outset. The program that was developed in this case was an extensive one, intended to provide financial relief to all U.S. citizens in an energy emergency. The program would be under the control of one existing agency where equipment, personnel, space, and standard operating procedures were most suitable to the goals of the plan. Although the admonitions derived from past research are straightforward, it turns out that adhering to them in the practical process of resolving specific implementation problems requires imagination and tenacity. Because implementation takes place in settings that are extraordinarily diverse, new generalizations regarding implementation will not be easy. Besides, those that engage in implementation research must reconcile themselves to the fact that their generalizations face a demanding test, far more demanding then that encountered in most policy analysis research—“Does it really work?”     

ENVIRONMENTAL MONITORING; A CASE OF LEARNING THROUGH EVALUATION?

The aim of environmental monitoring programs is to provide information useful in enforcing existing environmental statutes and helping to detect and prevent future problems. Despite the considerable number of monitoring programs being conducted, information on the quality of the nation's environment remains poor and of little use in supporting environmental policymaking. This paper examines the role evaluation has played in developing an effective nationwide environmental monitoring system. Background information on environmental monitoring programs is provided, followed by a discussion of the evaluations of past programs and recommendations made for improvement. One program in particular, the "Status and Trends" program administered by the National Oceanic and ATmospheric Administration, is examined to explore whether environmental monitoring demonstrates a case of learning through evaluation. "Status and Trends" is based on the experience gained during several of the previous monitoring programs and incorporates many of the recommendations spawned from earlier environmental programs. Whether the program demonstrates a case of learning through evaluation will depend on the ability of the program to evolve as we learn more about environmental quality and out ability to measure it.     

Do Anti‐immigration Parties Matter? The Case of the Sweden Democrats and Local Refugee Policy

With the increased electoral success of anti‐immigration parties, questions regarding what impact the parties actually have naturally follow. Previous research has rarely explored this question. Furthermore, within this body of research, one is given different answers. While some argue that anti‐immigration parties have made an impact on public policy, others find no such evidence. This article shows that some of this inconsistency is a consequence of the methodological strategies that have been employed. Previous studies are either single case studies or comparisons of a small number of countries. Consequently, different parties in different institutional settings are compared, making it difficult to estimate the actual impact of the party of interest. In order to circumvent such methodological problems, this article explores the question of anti‐immigration party impact on a local level and asks if the Sweden Democrats (SD) have managed to influence decisions on refugee reception in Swedish municipalities. The analysis shows mixed results. First, while unable to find an independent effect of the size of the representation of the SD, it appears that the party's impact is conditioned by them holding the balance of power. Second, the SD's impact is not dependent upon whether there is right‐wing or left‐wing rule, although local migration policy is stricter when the main right‐wing party has strong support.     

The Effects of Building Strong Families: A Healthy Marriage and Relationship Skills Education Program for Unmarried Parents

This article examines the impacts of Building Strong Families, a healthy marriage and relationship skills education program serving unmarried parents who were expecting or had recently had a baby. Based on a random assignment research design, the analysis uses survey data from more than 4,700 couples across eight research sites to estimate program effects. Results varied across sites, with one site having a pattern of positive effects (but no effect on marriage) and another having numerous negative effects. However, when impacts are averaged across all research sites, the findings indicate that the program had no overall effects on couples’ relationship quality or the likelihood that they remained together or got married.     

Administrative (de)centralization,performance equity,and outcome achievement in rural contexts: An empirical study of U.S. child welfare systems

Whether public services are better delivered through centralized or decentralized administrative systems is one of the most enduring debates in public administration. This article contributes to the debate with a study of program outcome achievement in U.S. child welfare services. We examine whether centralized state‐administered or decentralized county‐administered systems are more successful at realizing desired outcomes, which exhibit greater performance equity, and which are more successful at securing program outcomes in rural settings. Using a coarsened exact matching approach to quasi‐experimental research, we find that centralized child welfare systems exhibit greater success at achieving desired outcomes—in both rural contexts and overall. No discernible difference is found in centralized and decentralized systems' performance equity; outcome disparities exist under both structures. The study thus lends credence to centralization arguments, but at the same time cautions that centralized administration is not a structural panacea.     

The low‐income housing tax credit program goes mainstream and moves to the suburbs

Abstract

The Low‐Income Housing Tax Credit (LIHTC) program is now 20 years old. With the maturing of the program, the use of tax credits has become commonplace in the development of rental housing across the nation. This article examines how the program has changed both financially and spatially. Specifically, the article asks whether it provides a mechanism that can help deconcentrate impoverished renters by providing access to low‐poverty neighborhoods.

This research finds that as the price for tax credits rises, the program becomes increasingly popular with developers who are helping it make inroads in low‐poverty suburbs. By entering the suburbs, the LIHTC program is meeting and even exceeding the performance of the Housing Choice Voucher Program in terms of offering opportunities to live in low‐poverty settings.     

大规模干部教育培训工作的实效性问题研究——以检察机关教育培训工作为视角

大规模培训干部、大幅度提高干部素质需要做到对症下药。当前,检验培训工作成效的衡量标准难于界定、现有干部教育培训制度规范相对滞后、干部多样化的培训需求得不到满足是制约干部教育培训工作科学发展的三大难点问题。解决这些问题,需要切实从干部教育培训的特殊矛盾入手,需要扭转传统培训体制单方面强调组织需求而忽视个人培训需求的局面,需要运用好计划与市场两种手段。     

ATTITUDE MEASURES IN EVALUATION RESEARCH: A RESEARCH NOTE

The paper focuses on the use of attitude measures in evaluation research. It is suggested that attitude measures can assist evaluators in surmounting the problems of assessing program effectiveness both in process and impact evaluations. Attitude change can be conceptualized as the intended output of programs, as intervening between program variables and behavior or as proxy measures of behavioral target variables that are not readily measurable. Attitude measures can play avital role in evaluation research if an adequate methodology is employed.     

Three Lenses of Evidence‐Based Policy

This article discusses recent trends to incorporate the results of systematic research (or ‘evidence’) into policy development, program evaluation and program improvement. This process is consistent with the New Public Management (NPM) emphasis on efficiency and effectiveness. Analysis of evidence helps to answer the questions ‘what works? and ‘what happens if we change these settings?’ Secondly, some of the well known challenges and limitations for ‘evidence‐based’ policy are outlined. Policy decisions emerge from politics, judgement and debate, rather than being deduced from empirical analysis. Policy debate and analysis involves an interplay between facts, norms and desired actions, in which ‘evidence’ is diverse and contestable. Thirdly, the article outlines a distinction between technical and negotiated approaches to problem‐solving. The latter is a prominent feature of policy domains rich in ‘network’ approaches, partnering and community engagement. Networks and partnerships bring to the negotiation table a diversity of stakeholder ‘evidence’, ie, relevant information, interpretations and priorities. Finally, it is suggested that three types of evidence/perspective are especially relevant in the modern era – systematic (‘scientific’) research, program management experience (‘practice’), and political judgement. What works for program clients is intrinsically connected to what works for managers and for political leaders. Thus, the practical craft of policy development and adjustment involves ‘weaving’ strands of information and values as seen through the lens of these three key stakeholder groups. There is not one evidence‐base but several bases. These disparate bodies of knowledge become multiple sets of evidence that inform and influence policy rather than determine it.     

Partnering and Parenting in Poverty: A Qualitative Analysis of a Relationship Skills Program for Low‐Income,Unmarried Families

Since the mid‐1990s, the federal government has funded numerous relationship skills programs, including some specifically targeting low‐income, unmarried parents, in an effort to strengthen couples’ relationships and increase family stability. The previous research on the effectiveness of these interventions has revealed mixed results about whether such programs can improve the relationships of lower income couples who tend to experience lower relationship quality, lower marriage rates, and higher rates of relationship dissolution. This article draws on in‐depth qualitative data collected during an 18‐month ethnographic study of one federally funded relationship skills program for unmarried, low‐income couples expecting a new baby. Overall, though parents found the financial management lessons included in the classes only minimally useful, if at all, they found other aspects of the program particularly useful for three main reasons: (1) classes allowed parents to focus exclusively on their couple relationships in ways they rarely did otherwise; (2) program incentives helped parents make financial ends meet that month; and (3) parents learned that the challenges they personally experienced were often endemic to the romantic and co‐parenting relationships of unmarried parents who have few resources and experience more challenges that tend to undermine relationship quality, such as financial stress and relational ambiguity. Engaging with other couples around shared challenges normalized couples’ relationship problems and lessened the resentment and animosity that typically characterized their partner interactions. These findings have important implications for healthy marriage and relationship policy. Program developers should avoid lessons that imply low‐income, unmarried parents’ spending habits and family‐formation decisions are deficient. Interventions should instead encourage couples to discuss their shared challenges and minimize their tendency to individualize relational and financial strain.     

The Politics of Policy Design and Implementation: A Comparative Study of Two Belgian Service Voucher Programs

Abstract

When two policy programs, identical in many important respects, are implemented in the same national context, similar program results would be expected. If this is not the case, an explanation is needed. In this article two Belgian voucher programs are compared: one for household services and one for childcare services. The first program was termed a success while the latter was abandoned after disappointing results. Ideological differences between policy makers at cabinet level are identified as the most crucial factor explaining the contrasting results of the two programs. The failure of the childcare voucher scheme was due to its deeply flawed policy program design, particularly with respect to the implementing structure as well as the needs and demands of the designated target group. This comparative study confirms the usefulness and validity of two well-known theoretical constructs: an integrated implementation model and related statutory coherence thesis. The necessity of more attention to the potential influence of higher-level political-institutional factors and the dynamic interaction between policy formulation/design and policy implementation is one important lesson for contemporary implementation research. Another is that policy instruments per se do not determine policy results. Their role can only be fruitfully analyzed and explained in the context where they are selected and applied.     

Society as emergent and more than rational: An essay on the inappropriateness of program evaluation

In considering the efficacy of program evaluation researchers have not put language in proper perspective. They have not acknowledged that information gained from applied research is partly determined by situational factors, that language is selective and goal-oriented, justifying activity as often as it describes it, or that some important action cannot be expressed. The importance of language distinguishes applied social assessment from the natural science models of research from which it was derived. Examples from a wide range of applied situations are used to illustrate that, as an anticedent-consequent model of objective quantified assessment, evaluation is also inconsistent with the complex, resourceful, emergent social processes it is intended to measure. Three alternatives to quantification are discussed to illustrate interventions that may be made when the disadvantages of program evaluation outweigh the advantages.Preparation of the paper was supported by National Institute of Mental Health post-doctoral training grant 5T222-MH00180. Comments and requests for reprints should be sent to Nancy Cochran, Department of Mental Health and Developmental Disabilities, State of Illinois, 160 N. LaSalle Street, Chicago. Illinois 6061.     

Stakeholder partnerships as collaborative policymaking: Evaluation criteria applied to watershed management in California and Washington

Public policymaking and implementation in the United States are increasingly handled through local, consensus‐seeking partnerships involving most affected stakeholders. This paper formalizes the concept of a stakeholder partnership, and proposes techniques for using interviews, surveys, and documents to measure each of six evaluation criteria. Then the criteria are applied to 44 watershed partnerships in California and Washington. The data suggest that each criterion makes a unique contribution to the overall evaluation, and together the criteria reflect a range of partnership goals—both short‐term and long‐term, substantive and instrumental. Success takes time—frequently about 48 months to achieve major milestones, such as formal agreements and implementation of restoration, education, or monitoring projects. Stakeholders perceive that their partnerships have been most effective at addressing local problems and at addressing serious problems—not just uncontroversial issues, as previously hypothesized. On the other hand, they perceive that partnerships have occasionally aggravated problems involving the economy, regulation, and threats to property rights. © 2002 by the Association for Public Policy Analysis and Management.     

A Systematic Review of Field Experiments in Public Administration

Field experiments have become popular in public administration. By allowing for the identification of causal effects in realistic settings, field experiments may become central in several research agendas of relevance to the field. Conducting field experiments is difficult and problems often occur along the way. However, researchers new to the method have few resources in public administration to consider the problems that arise when conducting field experiments. This systematic review identifies 42 field experiments in public administration and serves as an introduction to field experiments in public administration. The article discusses how field experiments developed over time and highlights trends in field experimentation in public administration. It then discusses issues to consider when designing field experiments. Among these are costs, practicality, ethics, and validity. Finally, the authors suggest a future research agenda for public administration field experiments.     

UNPACKING COMPLEX MEDIATION MECHANISMS AND THEIR HETEROGENEITY BETWEEN SITES IN A JOB CORPS EVALUATION

This study aims to test the theory underlying Job Corps, one of the largest education and training programs in the U.S. serving disadvantaged youth. Central to the program are vocational training and general education that serve as two concurrent mediators transmitting the program impact on earnings. To distinguish the relative contribution of each, we develop methods for decomposing the Job Corps impact on earnings into an indirect effect transmitted through vocational training, an indirect effect transmitted through general education, and a direct effect attributable to supplementary services. We further ask whether general education and vocational training reinforce each other and produce a joint impact greater than the sum of the two separate pathways. Moreover, we examine the heterogeneity of each causal effect across all the Job Corps centers. This article presents concepts and methods for defining, identifying, and estimating not only the population averages but also the between‐site variance of these causal effects. Our analytic procedure incorporates a series of weighting strategies to enhance the internal and external validity of the results and assesses the sensitivity to potential violations of the identification assumptions.     

Securing Quality Through Contracts? The Development of Quasi-markets for Social Care in Britain

This article examines the early development in Britain of quasi-markets for domiciliary care. Initially there were widespread doubts about the appropriateness of market mechanisms for social care because it was regarded as inherently different from other goods and services. We examine these differences and how they are most apparent in domiciliary care — which is provided to the most vulnerable people in the most vulnerable of settings. The study provides a demanding test and illuminating case study of the introduction of competition through quasi-markets. It highlights some potential threats to service quality and demonstrates that to secure such quality for vulnerable users, public sector purchasers need to develop long-term, high-trust relationships with a range of high quality care providers.     

The Synthetic Control Method for Comparative Case Studies: An Application Estimating the Effect of Managerial Discretion Under Performance Management

Public management researchers are often interested in estimating the effects of aggregate-level management reforms and policy changes, but they frequently rely on observational data that pose a number of threats to internal validity. The synthetic control method for comparative case studies (SCM) has great potential for advancing public management scholarship by addressing some of the methodological challenges associated with observational data and providing intuitive graphical results that help researchers communicate their findings to audiences unfamiliar with quantitative methods. SCM uses a transparent, data-driven algorithm for selecting a weighted combination of control units that act as a plausible counterfactual for estimating causal effects. This article demonstrates SCM by investigating the effect of enhancing managerial discretion under performance accountability systems in the context of public education. The article also provides a number of possible avenues for future public management research using SCM and practical guidance for applying the method.     

Political information,social environments,and deviants

This research explores whether an individual's information about politics is influenced by the political environment in which the person is located. It seeks to determine whether and how the political environment influences one's attention to and retention of political information.