USING LINEAR TREND MODELS TO ANALYZE POLICY IMPACT

Policy researchers have become increasingly familiar with a number of improved techniques for analyzing data obtained from interrupted time-series designs for evaluating public programs and policies. In this paper we contribute to this trend by presenting two groups of data analysis techniques which are not currently widely used by policy researchers, but are likely to be valuable adjuncts to traditional regression techniques for analyzing data obtained from interrupted time-series designs. First, aids for model specification are presented that enable the analyst to define an appropriate linear trend model—often one which will reduce the degree of multicollinearity and, therefore, produce more precise estimates of the impacts of a public program or policy. Next we consider approaches for point estimation and joint (simultaneous) interval estimation of a policy intervention's total effect at various points in time.     

PROGRAM IMPLEMENTATION VERSUS PROGRAM DESIGN: WHICH ACCOUNTS FOR POLICY "FAILURE"?

Many influential implementation scholars now argue that "street-level" bureaucrats, rather than legislators or high-level administrators, make public policy in the U.S. Such authors as Pressman and Wildavsky cite creaming in employment and training programs as an especially clear example of well-meaning programs that fail when implemented. This paper argues that two of the most significant and lasting of these programs, the U.S. Employment Service and the Manpower Development and Training Act, were designed to encourage creaming. The essay asserts implementation scholars overstate the disconnection between program design and program implementation because they assume there is little disconnection between program legitimation and program design. A better conception of design permits one to perceive that these programs were legitimated on the grounds they would serve a large number of constituents, but were designed to do so by serving employers. The combination of these premises made creaming an imperative of program operation, and the implementors who cream remain faith ful to original program strategy. This finding suggests a redirection of policy research toward a more rlgorous analysis of program design and a better understanding of the relationship between legitimation, design, and implementation.     

Implementing Performance-Based Program Budgeting: A System-Dynamics Perspective

The article proposes a model for evaluating budget reforms that combines insights from budgeting, policy implementation, and system-dynamics literatures. System-dynamics modeling combines both quantitative and qualitative research techniques to provide a new framework for applied research; its use is illustrated using performance budgeting as an example. Applied to the implementation of Florida's performance-based program budget, the model identifies actions in the short run that will increase the reform's likelihood of success: providing clear communications; facilitative budget and accounting routines; reliable performance information. The model also identifies critical legislative behaviors that influence executive implementation: how the legislature in the long-run uses performance information to inform resource allocation and how it applies incentives or sanctions to programs that achieve or fail to achieve their performance standards. The legislature has the opportunity to use program reviews prepared by legislative staff to invigorate the executive branch's resolve to continue implementing the reform.     

The deserving sick: Income-maintenance policy towards the ill and disabled

Illness and disability often result in a serious loss of income. Government response to this problem has been piecemeal, so that there is a patchwork of different programs, some state and some federal, covering particular types of medical disabilities and particular categories of people. This article briefly describes existing programs and then evaluates income-maintenance policy towards the ill and disabled using four broad criteria: clarity of objectives, adequacy of coverage, adequacy of benefits, and equity. The general findings are that extreme program fragmentation in this policy area have led to competing (and sometimes inconsistent) objectives, large gaps in population coverage, benefit levels that do not meet even minimal standards, and tremendous inequities across different programs. Recommendations for improvement include the development of a more workable definition of disability and development of a guaranteed minimum income plan in conjuction with supplementary disability insurance plans.The author would like to thank Randall Bovbjerg and James Vaupel for their thoughtful criticisms.     

How important is information format? An experimental study of home energy audit programs

Among alternative regulatory responses, information provision programs are receiving increasing attention despite the decidedly mixed evidence about their effectiveness. This paper provides detailed experimental evidence to demonstrate that for one important example of an information program, the use of energy audits to stimulate residential energy conservation, the effectiveness of the program is highly sensitive to the information processing behavior of the users of the information. Simple changes in the format of the information provided to homeowners produced marked improvements in the efficiency of consumer choices. The study illustrates Stern's conclusion in a recent issue of this journal that laboratory experimentation provides a useful alternative to model-based analysis of natural experiments, especially for policy design issues involving information acquisition and processing. The paper also offers conclusions about improving the effectiveness of home energy audits, as well as the entire class of information provision programs.     

Wildfire Management in the United States: The Evolution of a Policy Failure

Wildland fires constitute a major crisis in American environmental policy, a crisis created by a longstanding policy failure. This article explores the political processes that generated and reinforced this policy failure over time. The concepts of bounded rationality, punctuated equilibria, and self- reinforcing mechanisms are applied to study the evolution of American wildfire policy between 1905 and the present. This study finds that a self-defeating wildfire suppression policy was established in the period 1905 through 1911, and subsequently reinforced for more than five decades. This policy did not include a complementary program to counteract the gradual accumulation of flammable organic materials (fuels) that occurred in many ecosystems when fires were suppressed. The resulting fuel accumulations have greatly increased the risk of damaging, high-intensity wildfires in a range of American wildlands. A combination of fire suppression and fuel reduction programs will be needed to manage this risk in the future.     

Challenges confronting nonprofit housing organizations’ self‐sufficiency programs

Abstract

In recent years, interest has grown at the federal level in strategies to combine subsidized housing with programs promoting household self‐sufficiency. This article explores how nonprofit housing organizations conceptualize their self‐sufficiency programs for their residents. A broad definition of self‐sufficiency is presented—one that is not exclusively focused on the individual and, instead, also includes program strategies that are focused on changing the context in which individuals live and work.

The paper then analyzes the relationship between the self‐sufficiency strategies being implemented in the nonprofit housing world and how these organizations will be affected by welfare reform, the shrinking and restructuring of federally subsidized housing, the emergence of block grant job training and workforce development programs, and the general devolution of government programs into ever more fungible pots at state and local levels. These transformations in the domestic policy agenda will present challenges to nonprofit housing organizations and to the goal of promoting self‐sufficiency.     

International measurement of food security: Enhancing alignment between evidence and assistance programs

A definition of food security used in developing countries by United Nations Agencies, The United States Agency for International Development (USAID) and other international development agencies establish four dimensions of food security: access, availability, utilization, and stability. This definition provides a framework for a multidimensional, rigorous measure of food security across developing countries and enables an analysis of the magnitude and severity of food insecurity across time and space. This framework also permits an analysis of the drivers of food insecurity that foster appropriate, cost‐effective food assistance programs. In the developed world, food security is often defined more narrowly, resulting in a less rigorous measurement of food security. Food security experts in developed countries can exploit the framework applied in the developing world to improve the measurement of food insecurity, as well as evidence‐based food assistance programs that are informed by its measurement.     

Public Preferences for Program Tradeoffs: Community Values for Budget Priorities

A growing literature concerns techniques to improve community-based reforms and citizen-centered governance in order to reinforce the trust in democratic government. We analyze a contingent choice technique that systematically collects information about individual citizens' relative values of a set of state public programs. Individual citizens are asked to allocate a fixed increment of public funds. Individuals reveal their marginal willingness to trade off (MWTTO) additions in one program for additions in another. MWTTO values provide program rankings and information concerning the relative strength of citizen preferences. An example of a contingent choice survey is described.     

Taken for Granted? Managing for Social Equity in Grant Programs

Managing for social equity performance has long been a goal without much guidance for public managers. We examine social equity performance in the context of indirect governance through the administration of grant programs and, more specifically, the matching of policy responses (grant funding) to social needs. Grant program managers must allocate funding to match needs while also ensuring accountability, but common administrative models that rely on competition can undermine social equity performance. We develop a unique framework to analyze the relative social equity performance of four models of grant administration in general. These models are defined by whether competitions or formulas are used to select grantees and to allocate funding. We test the implications of the framework in an analysis of funding distributions from the nonentitlement Community Development Block Grant program in four states. Our findings suggest that social equity in grant programs is better served when grantors do not rely solely on competitive grant contracting in the selection and distribution of grant funds, which is typical in grant administration. However, policy makers and managers can design institutional arrangements that utilize competition, but in a manner that does not create a bias against more socially equitable funding decisions.     

The Politics of Policy Design and Implementation: A Comparative Study of Two Belgian Service Voucher Programs

Abstract

When two policy programs, identical in many important respects, are implemented in the same national context, similar program results would be expected. If this is not the case, an explanation is needed. In this article two Belgian voucher programs are compared: one for household services and one for childcare services. The first program was termed a success while the latter was abandoned after disappointing results. Ideological differences between policy makers at cabinet level are identified as the most crucial factor explaining the contrasting results of the two programs. The failure of the childcare voucher scheme was due to its deeply flawed policy program design, particularly with respect to the implementing structure as well as the needs and demands of the designated target group. This comparative study confirms the usefulness and validity of two well-known theoretical constructs: an integrated implementation model and related statutory coherence thesis. The necessity of more attention to the potential influence of higher-level political-institutional factors and the dynamic interaction between policy formulation/design and policy implementation is one important lesson for contemporary implementation research. Another is that policy instruments per se do not determine policy results. Their role can only be fruitfully analyzed and explained in the context where they are selected and applied.     

A Comparative Assessment of Elite Policy Recruits in Canada

Recent case studies and large-N survey evidence has confirmed long-suspected shortages of public sector “policy capacity”. Studies have found that government policy workers in various jurisdictions differ considerably with respect to types of policy work they undertake, and have identified uneven capacity for policy workers to access and apply technical and scientific knowledge to public issues. This suggests considerable difficulties for government’s ability to meet contemporary policy and governance challenges. Despite growing attention to these matters, studies have not examined the “elite” policy workers many governments recruit to address these capacity shortages. Using an established survey instrument, this study of two Canadian recruitment programs provides the first comparative analysis of elite policy recruits, as policy workers. Three research questions anchor the study: (1) What is the profile of these actors? (2) What types of policy work do “elite” policy analysts actually engage in? (3) How does their policy work compare by recruitment program? The article provides fresh comparative data on the nature of elite policy work and policy analytical capacity, but, more importantly, a crucial baseline for future comparative study of how elite recruitment may facilitate “supply-side” capacity gains expected from recruitment programs.     

Empirical Research and the Development and Assessment of Transitional Justice Mechanisms

In this paper, we demonstrate different methods of empiricalresearch available to transitional justice practitioners andscholars. Guidelines on how to conduct research in the fieldof transitional justice are outlined on the basis of the principlesof monitoring and evaluation for decision making, program developmentand policy development. We argue that such methods offer policymakers a systematic way to consult a population and constructa comprehensive view of what this population has experienced(e.g., violations), what they know (e.g., knowledge of varioustransitional justice mechanisms), what they believe in (e.g.,definition of justice) or what they need (e.g., accountability).Such knowledge is essential for developing effective evidence-basedtransitional justice programs.     

Allocations among programs having counteractive outcomes

Difficulties have been encountered in attempting to apply benefit-cost analysis to the problem of determining resource allocations among social programs, especially when there are multiple outcomes associated with each of the programs. Further complications result when program outcomes are characterized as counteractive: a specific allocation of resources to a program is likely to achieve a favorable impact regarding certain objectives while that allocation is also likely to have an unfavorable impact regarding other objectives. A model which promises to be useful in determining optimal resource allocations among programs having counteractive outcomes is presented. Two steps are required for the use of the model: (1) values, reflecting priorities, need to be assigned to the various objectives that programs are intended to accomplish and (2) an ensemble of probabilistic functions relating resources to program outcomes needs to be specified and combined to reflect the value of an ensemble of programs. In addition to determining optimal resource allocations, the model provides insights into the consequences of resource decisions based on noneconomic considerations. An example illustrates the major features of the model.     

Do experimental and nonexperimental evaluations give different answers about the effectiveness of government‐funded training programs?

This paper uses meta‐analysis to investigate whether random assignment (or experimental) evaluations of voluntary government‐funded training programs for the disadvantaged have produced different conclusions than nonexperimental evaluations. Information includes several hundred estimates from 31 evaluations of 15 programs that operated between 1964 and 1998. The results suggest that experimental and nonexperimental evaluations yield similar conclusions about the effectiveness of training programs, but that estimates of average effects for youth and possibly men might have been larger in experimental studies. The results also suggest that variation among nonexprimental estimates of program effects is similar to variation among experimental estimates for men and youth, but not for women (for whom it seems to be larger), although small sample sizes make the estimated differences somewhat imprecise for all three groups. The policy implications of the findings are discussed. © 2006 by the Association for Public Policy Analysis and Management     

Programmatic action in Chinese health policy—The making and design of “Healthy China 2030”

The Programmatic Action Framework (PAF) is a powerful analytical perspective to advance our understanding of policy change by stressing the role of programmatic groups and how they promote their preferred policy programs. While the PAF has been applied successfully to European countries and the US, a considerable research gap remains regarding its explanatory power in other regions and political settings. To step into this research gap, we apply the PAF to authoritarian China through a qualitative case study of the making of “Healthy China 2030”, a national health policy program. We explore two research questions: firstly, to what extent can the PAF be operationalized in a nondemocracy to identify programmatic groups and evaluate policy design, and second, what are the limitations and challenges facing the application of the PAF to nondemocracies? Our empirical analysis shows that the design of this policy program involves actors from different policy areas, despite the top-down political structure. With regards to the transfer of PAF hypotheses, we show that the framework is a helpful lens to identify programmatic groups and to evaluate policy programs' potential for success based on coherence, responsiveness, and program fit. Meanwhile, the research process of our study revealed the challenges of applying policy process theories to China, including hard-to-reach political actors for data collection, lack of transparency regarding collaboration processes between actors, and a lack of publicly available program-related information.     

Assessing the role of vocational rehabilitation in disability policy.

Since 1975, vocational rehabilitation has represented a small and declining component of federal disability policy. This trend is perhaps reflective of the relatively crude assessment techniques that have been applied to the program in the past. Using the Virginia Vocational Rehabilitation (VR) program as a prototype, we outline how the data and methods of assessment can be improved for purposes of directing public policy. The key issues include identifying an appropriate comparison group for VR, analysis of longitudinal earnings data, and methods for refining measures of program cost. The analysis provides "fixed-effects" estimates of net earnings impacts for each of three postprogram years stratified by disability classification and gender. These treatment impacts are compared to total and service-specific costs. In general, this analysis suggests that evaluation of VR can be substantially improved and that these improvements can be attained at relatively modest analytic cost.     

Welfare and child support program knowledge gaps reduce program effectiveness

There is little research on knowledge of the policy rules that could affect individuals, either in general or in evaluations of new programs. The lack of research is surprising, given that knowledge gaps could limit the effectiveness of reforms or lead to incorrect inferences regarding the effects of a policy change. In this article, we use survey data to examine the level and sources of welfare participants' policy knowledge in the context of a substantial change in child support and welfare policy in Wisconsin. We find very low levels of knowledge of child support policy rules. Multivariate analyses suggest that people tend to learn policy rules by experience; we find less consistent support for knowledge being primarily imparted through interactions with caseworkers. A difference‐in‐difference analysis suggests that if participants had been more knowledgeable, program impacts would have been larger. Implications of this research for policy implementation and policy evaluations are discussed. © 2007 by the Association for Public Policy Analysis and Management     

A Race to the Bottom? Exploring County Spending Shortfalls under Welfare Reform in North Carolina

North Carolina's experiment with local control of Temporary Assistance to Needy Families programs provides a unique example of the budget implications of state to local government policy devolution. County‐level expenditure data indicate that after state to local government devolution, only 45 of 100 counties met welfare maintenance of effort (MOE) budget requirements. Several important factors, including control over program eligibility and benefit levels, were significantly associated with MOE spending variation. Counties given more flexibility spent dramatically less than others. Many failed to meet MOE requirements without penalty, year after year. These issues raise serious questions about decentralization from both the budget and policy perspectives.