Reinventing Reforms: How to Improve Program Management Using Performance Measures. Really

This paper looks at the design and use of incentivized performance measures to motivate managerial efficiency and promote greater program effectiveness. It starts off by looking at recent reforms like the Government Performance and Results Act to understand why they were largely unsuccessful in altering the decision‐making process of government agencies. One problem was that performance measures have been both numerous and complicated, thereby making their role in management and oversight difficult. Equally important, no external incentives were attached to program accomplishments. The paper then examines what elements would be needed to build a management system that encourages both more efficient and more effective agency behavior. The goal of performance budgeting is to develop performance measures that display the progress of a program toward its stated objectives. Assessments based on these measures may then call for rewards or punishments. As such, it also may encourage program managers toward improved performance. The paper examines the pitfalls and complexities dealt with by Congress and Office of Management and Budget in the process. For example, a performance system must distinguish between funding program needs, as warranted by sectoral indicators, and management concerns. It must also unambiguously tie incentives to performance measures to motivate agencies, while building in commitment devices for the principals. Incentivized performance measures may not be appropriate in all conditions, but may be helpful for motivating managers and improving program performance in particular circumstances.     

When Do Regression‐Adjusted Performance Measures Track Longer‐Term Program Impacts? A Case Study for Job Corps

The use of performance management systems has increased since the Government Performance and Results Act of 1993. While these systems share the goal of trying to improve service delivery and participant outcomes, they do not necessarily provide information on the causal (value‐added) effects of a program, which requires a rigorous impact evaluation. One approach for potentially improving the association between program performance measures and impacts is to adjust performance measures for differences across performance units in participant characteristics and local economic conditions. This article develops a statistical model that describes the conditions under which regression adjustment improves the performance–impact correlation. We then use the model to examine the performance–impact association using extensive data from a large‐scale random assignment evaluation of Job Corps, the nation's largest training program for disadvantaged youths. We find that while regression adjustment changes the Job Corps center performance measures, the adjusted performance measures are not correlated with the impact estimates. The main reasons are the weak associations between the unadjusted Job Corps performance measures and participants’ longer‐term outcomes as measured by the evaluation, as well as the likely presence of unobserved factors across centers that are correlated with outcomes.     

Extending the Theory of Goal Ambiguity to Programs: Examining the Relationship between Goal Ambiguity and Performance

One of the main assumptions of empirical studies conducted on the influence of goal ambiguity in public management is that goal ambiguity relates negatively to performance. However, this relationship has rarely been tested at the program level because common goal ambiguity and performance measures for disparate government programs have been scant. The availability of Program Assessment Rating Tool (PART) results for a number of federal programs provides the opportunity for an analysis testing the foregoing assumption. Measures of program goal ambiguity—target, timeline, and program evaluation—are shown to have negative relationships with different program performance scores, taking into account alternative influences or biases on performance. This analysis extends the theory of goal ambiguity by providing the first analysis of large‐sample federal programs. The theoretical and practical implications are presented in the discussion and conclusion.     

How Credible Is the Evidence,and Does It Matter? An Analysis of the Program Assessment Rating Tool

This research empirically assesses the quality of evidence that agencies provided to the Office of Management and Budget in the application of the Program Assessment Rating Tool (PART), introduced in 2002 to more rigorously, systematically, and transparently assess public program effectiveness and hold agencies accountable for results by tying them to the executive budget formulation process and program funding. Evidence submitted by 95 programs administered by the U.S. Department of Health and Human Services for the PART assessment is analyzed using measures that capture the quality of evidence and methods used by programs and information on characteristics of agencies that might relate to program results and government funding decisions. The study finds that of those programs offering some evidence, most was internal and qualitative, and about half did not assess how their performance compared to other government or private programs with similar objectives. Programs were least likely to provide externally generated evidence of their performance relative to long‐term and annual performance goals. Importantly, overall PART and results scores were (statistically) significantly lower for programs that failed to provide quantitative evidence and did not use long‐term measures, baseline measures or targets, or independent evaluations. Although the PART program results ratings and overall PART scores had no discernible consequences for program funding over time, the PART assessments appeared to take seriously the evaluation of evidence quality, a positive step forward in recent efforts to base policy decisions on more rigorous evidence.     

Making Government Accountable: Lessons from a Federal Job Training Program

This article describes the evolution of a performance measurement system in a government job training program. In this program, a federal agency establishes performance measures and standards for substate agencies. The performance measurement system's evolution is at least partly explained as a process of trial and error characterized by a feedback loop: The federal agency establishes performance measures, the local managers learn how to game them, the federal agency learns about gaming and reformulates the performance measures, possibly leading to new gaming, and so on. The dynamics suggest that implementing a performance measurement system in government is not a one-time challenge but benefits from careful monitoring and perhaps frequent revision.     

The Professional Researcher as Change Agent in the Government–Performance Movement

Although academic and professional publications give the impression that performance measurement is a growing government practice, in actuality the use of this technology is not as deep or as widespread as it may appear. Even when performance measures are used, governments rarely integrate them into planning, budget, personnel, and other management processes. Most professional researchers located primarily in academic institutions, but also in research and government organizations, approach performance measurement as though governmental officials, elected or otherwise, are already sold on its usefulness. Instead, they need to function as "change agents," using a variety of strategies to gain acceptance and understanding of the strengths and limitations of performance measurement. This article draws on the authors' experiences with the Community Benchmarks Program of the Maxwell School in Onondaga County and a review of the current literature. It suggests guidelines for professional researchers who want to increase the use of performance measures by governments at all levels.     

Do Job Corps performance measures track program impacts?

Since the 1993 Government Performance and Results Act, performance measurement systems based on short‐term program outcomes have been increasingly used to assess the effectiveness of federal programs. This paper examines the association between program performance measures and long‐term program impacts, using nine‐year follow‐up data from a recent large‐scale, national experimental evaluation of Job Corps, the nation's largest federal job training program for disadvantaged youths. Job Corps is an important test case because it uses a comprehensive performance system that is widely emulated. We find that impacts on key outcomes are not associated with measured center performance levels. Participants in higherperforming centers had better outcomes; however, the same pattern holds for comparable controls. Thus, the performance measurement system is not achieving the goal of ranking and rewarding centers on the basis of their ability to improve participant outcomes relative to what these outcomes would have been otherwise. © 2008 by the Association for Public Policy Analysis and Management.     

Housing Voucher Take‐Up and Labor Market Impacts

Low participation rates in government assistance programs are a major policy concern in the United States. This paper studies take‐up of Section 8 housing vouchers, a program in which take‐up rates are quite low among interested and eligible households. We link 18,109 households in Chicago that were offered vouchers through a lottery to administrative data and study how baseline employment, earnings, public assistance, arrests, residential location, and children's academic performance predict take‐up. Our analysis finds mixed evidence of whether the most disadvantaged or distressed households face the largest barriers to program participation. We also study the causal impact of peer behavior on take‐up by exploiting idiosyncratic variation in the timing of voucher offers. We find that the probability of lease‐up increases with the number of neighbors who recently received voucher offers. Finally, we explore the policy implications of increasing housing voucher take‐up by applying reweighting methods to existing causal impact estimates of voucher receipt. This analysis suggests that greater utilization of vouchers may lead to larger reductions in labor market activity. Differences in take‐up rates across settings may be important to consider when assessing the external validity of studies identifying the effects of public assistance programs.     

The Fly Ball Effect: A Theoretical Framework for Understanding the Impacts of Short‐Term Seed Grants

The federal government has long used grants‐in‐aid to encourage state and local governments to carry out federal policies. Little research has been done that examines how short‐term seed grants affect program continuation. We propose the “fly ball effect” as a theoretical framework for understanding how seed money should impact program maintenance. Our theory suggests that short‐term seed grants by themselves should result in considerable funding uncertainty and program eliminations or stagnation once the initial grant money expires. We use data from drug court start‐ups in four states to provide empirical support for our theory. We argue that understanding the logic of the fly ball effect can help granting governments to improve the effectiveness of their grant funding systems, at least as measured by strong program continuation and expansion.     

The Design and Practice of Integrating Evidence: Connecting Performance Management with Program Evaluation

In recent decades, governments have invested in the creation of two forms of knowledge production about government performance: program evaluations and performance management. Prior research has noted tensions between these two approaches and the potential for complementarities when they are aligned. This article offers empirical evidence on how program evaluations connect with performance management in the U.S. federal government in 2000 and 2013. In the later time period, there is an interactive effect between the two approaches, which, the authors argue, reflects deliberate efforts by the George W. Bush and Barack Obama administrations to build closer connections between program evaluation and performance management. Drawing on the 2013 data, the authors offer evidence that how evaluations are implemented matters and that evaluations facilitate performance information use by reducing the causal uncertainty that managers face as they try to make sense of what performance data mean.     

Fiscal Decentralization and Natural Disaster Mitigation: Evidence from the United States

How decentralized government structure influences public service delivery has been a major focus of debate in the public finance literature. In this paper, we empirically examine the effect of fiscal decentralization on natural disaster damages across the U.S. states. We construct a unique measure of decentralization using state and local government expenditures on natural resources, which include investment in flood control and mitigation measures, among others. Using state‐level panel data from 1982 to 2011, we find that states that are more decentralized in natural resource expenditures have experienced more economic losses from floods and storms. This effect is only pronounced in states that are at higher risks of flooding. Our findings suggest that fiscal decentralization may lead to inefficient protection against natural disasters and provide implications for the assignment of disaster management responsibilities across different levels of government in the U.S. federal system.     

Cream‐Skimming,Parking and Other Intended and Unintended Effects of High‐Powered,Performance‐Based Contracts

As performance‐based contracting in social welfare services continues to expand, concerns about potential unintended effects are also growing. We analyze the incentive effects of high‐powered, performance‐based contracts and their implications for program outcomes using panel data on Dutch cohorts of unemployed and disabled workers that were assigned to private social welfare providers in 2002 to 2005. We employ a difference‐in‐differences design that takes advantage of the fact that contracts gradually moved from partial performance‐contingent pay to full (100 percent) performance‐contingent contracting schemes. We develop explicit measures of selection into the programs and find evidence of cream skimming and other gaming activities on the part of providers, but little impact of these activities on program outcomes. Moving to a system with contract payments fully contingent on performance appears to increase job placements, but not job duration, for more readily employable workers.     

Judicious Incentives: International Public Policy Responses to the Globalization of Environmental Management

Public policymakers and regulators worldwide are grappling with the desire to improve environmental quality through appropriate regulation of business, while also streamlining government. Concurrently, environmentally conscience consumers are calling for improved environmental performance by industry. As a result of these pressures, regulators and lawmakers worldwide are attempting to craft effective policies that create adequate incentives for environmental protection on the part of firms, in the face of decreasing budgets and an increased demand for the use of market‐based incentives. To aid decision makers as they struggle with these concerns, this study provides a detailed case examination of the dilemmas and responses of national‐level regulators as they try to develop appropriate responses to the rise of international and “voluntary” management regimes. To accomplish these goals, this article compares the public policy responses of governments around the world to one such voluntary international environmental regime: ISO 14001. ISO 14001 is a form of industry self‐regulation in response to market forces calling for harmonization in environmental management and as a result of consumer and trade‐partner demands. This study examines the relationships between regulators and the regulated in order to understand if ISO 14001 certified firms are receiving regulatory relief or other forms of public policy/regulatory benefits as a result of their certification. It will also examine the impact that government incentives (or their absence) are having on the certification decisions of firms around the world. This information helps us to begin to understand how the trends toward smaller government and voluntary environmental regimes are affecting one another.     

Joined‐Up for What? Response to Carey and Harris on Adaptive Collaboration

Carey and Harris present the concept of adaptive management as a practice for supporting effective collaboration, suggesting that performance information be used to modify actions. They observe that end‐outcome performance information is less useful because of long delays between actions and effects, and recommend instead that the performance information should concern the collaborative process itself. The New Zealand government has followed a similar path to the Australian journey described by Carey and Harris. First, New Zealand tried using end outcomes to drive collaboration. Then, New Zealand tried using process measures, but found that the resulting collaboration lacked purpose and urgency. More recently, New Zealand has found great success in using intermediate‐outcome measures to drive adaptive collaboration: measures with intrinsic value, but short delay between action and effect. We echo Carey and Harris’ call for adaptive collaboration, but write to suggest that intermediate outcomes, rather than process measures, may drive more purposive management.     

政府绩效视角下的行政管理体制改革战略反思——基于地方政府公务员的感知调查

改革开放以来,我国行政管理体制改革一直坚持以组织结构调整为主的模式.对于这一模式的影响,尤其对地方政府的影响,一直缺少科学层面的检验和理论层面的反思.本文选取政府绩效作为标杆,利用地方政府公务员的问卷调查数据,探索政府部门间关系和部门管理制度建设影响部门绩效的路径和程度,使用结构方程模型实证检验了行政管理体制改革战略在地方层面的有效性.结果表明,部门间关系作为政府结构的核心特征对于政府绩效并没有直接的影响作用,其影响作用是通过部门制度建设和部门文化间接发挥的;部门制度建设对政府绩效的整体影响要大于部门间关系.因此,在今后的改革中应该给予管理制度层面的改革更多的重视,通过局部的、具体制度的改革为整体的、基本制度的变革创造条件.同时,还应在改革过程中维护政府组织文化的凝聚力和激励性.     

The Relative Influence of Appropriation Subcommittees: Institutional Structure and Program Performance

Unlike previous analyses that evaluate the influence of the authorization committees on government agency management, we examine the US congressional appropriation subcommittees to see how their structural characteristics affect the performance of the programs that they fund. Specifically, we look at whether the competition for resources and member workload within subcommittees affects the effectiveness and efficiency of the programs under their purview. As part of the Congressional budget process, appropriation subcommittees are annually allocated new budgets from which they must fund a set of government programs. We find that the level of resources in a subcommittee affects program performance. Greater subcommittee resources are negatively correlated with program performance. We also find that workload matters—programs in subcommittees with fewer issue areas and programs per member are more effective and efficient. A subcommittee's capacity to focus on management problems and issue areas leads to better program performance.     

Organizing for Crisis Management: Building Governance Capacity and Legitimacy

What makes a well‐functioning governmental crisis management system, and how can this be studied using an organization theory–based approach? A core argument is that such a system needs both governance capacity and governance legitimacy. Organizational arrangements as well as the legitimacy of government authorities will affect crisis management performance. A central argument is that both structural features and cultural context matter, as does the nature of the crisis. Is it a transboundary crisis? How unique is it, and how much uncertainty is associated with it? The arguments are substantiated with empirical examples and supported by a literature synthesis, focusing on public administration research. A main conclusion is that there is no optimal formula for harmonizing competing interests and tensions or for overcoming uncertainty and ambiguous government structures. Flexibility and adaptation are key assets, which are constrained by the political, administrative, and situational context. Furthermore, a future research agenda is indicated.     

Outcomes–Based Performance Management in the Public Sector: Implications for Government Accountability and Effectiveness

Requirements for outcomes–based performance management are increasing performance–evaluation activities at all government levels. Research on public–sector performance management, however, points to problems in the design and management of these systems and questions their effectiveness as policy tools for increasing governmental accountability. In this article, I analyze experimental data and the performance–management experiences of federal job–training programs to estimate the influence of public management and system–design factors on program outcomes and impacts. I assess whether relying on administrative data to measure program (rather than impacts) produces information that might misdirect program managers in their performance–management activities. While the results of empirical analyses confirm that the use of administrative data in performance management is unlikely to produce accurate estimates of true program impacts, they also suggest these data can still generate useful information for public managers about policy levers that can be manipulated to improve organizational performance.     

Whole of Government: the Solution to Managing Crises?

The frequency and severity of natural disasters has placed a clear emphasis on the role of governments in responding to these crises. During the past decade, disaster events have had a significant impact on the relevant communities as well as raising questions regarding the role of government and the bureaucratic coordination of planning and response processes. These events have placed a renewed focus on the ability of governments to plan, prepare, and respond in an effective way to crises. They have also tended to indicate that there remain serious challenges to government coordination and that crises create a unique series of challenges for the public sector. At the heart of understanding how governments respond to crises are notions of bureaucratic coordination. It has been suggested that joined‐up or whole of government arrangements may provide an appropriate means in which to approach crisis management. As a result a number of key themes emerge including the nature of crisis management, role of leadership, understanding coordination, impact of organisational culture, and the interactions between individuals and institutions. This paper will consider these issues and provide a review of the relevant literature, to understand the synergies that exist in connected responses to crises.     

Pluralism,corporatism and perception of corruption

When people know who is influencing the elected politicians and they may ‘put the rascals out’ in case they feel that the incumbents are corrupt, ceteris paribus, their perception of the level of corruption should not be affected by lobbying. If on the other hand people are not sure which or how many actors are influencing public policy and they are not able to hold the government truly accountable as interest group influence is constant with different governments, people will be more likely to perceive the government as corrupted. The former system is a characteristic of corporatism and the latter of pluralism. This problem is exacerbated by the fact that interest groups with resources such as business groups or firms in pluralist systems are more influential than groups with few resources. Thus, people may perceive pluralist policy‐making system as more corrupt than corporatist policy‐making system where fewer visible actors have more or less equal weight in the policy‐making process. Copyright © 2014 John Wiley & Sons, Ltd.