Better than Raw: A Guide to Measuring Organizational Performance with Adjusted Performance Measures

Like oysters on the half shell, some things are better when they're raw. In evaluating the performance of organizations and providing guidance for improving performance, however, raw performance measures, such as test scores or success rates, are often inferior to performance measures adjusted for client and environmental characteristics, or adjusted performance measures (APMs). Using examples from a variety of public services and data on public schools in Georgia, we compare the performance data generated by raw scores and by APMs. We conclude with guidance for constructing and using adjusted performance measures.     

When Do Regression‐Adjusted Performance Measures Track Longer‐Term Program Impacts? A Case Study for Job Corps

The use of performance management systems has increased since the Government Performance and Results Act of 1993. While these systems share the goal of trying to improve service delivery and participant outcomes, they do not necessarily provide information on the causal (value‐added) effects of a program, which requires a rigorous impact evaluation. One approach for potentially improving the association between program performance measures and impacts is to adjust performance measures for differences across performance units in participant characteristics and local economic conditions. This article develops a statistical model that describes the conditions under which regression adjustment improves the performance–impact correlation. We then use the model to examine the performance–impact association using extensive data from a large‐scale random assignment evaluation of Job Corps, the nation's largest training program for disadvantaged youths. We find that while regression adjustment changes the Job Corps center performance measures, the adjusted performance measures are not correlated with the impact estimates. The main reasons are the weak associations between the unadjusted Job Corps performance measures and participants’ longer‐term outcomes as measured by the evaluation, as well as the likely presence of unobserved factors across centers that are correlated with outcomes.     

Factors Influencing the Use of Performance Data to Improve Municipal Services: Evidence from the North Carolina Benchmarking Project

Many local governments measure and report their performance, but the record of these governments in actually using performance measures to improve services is more modest. The authors of this study examine patterns of performance measurement use among a set of North Carolina cities and conclude that the types of measures on which officials rely, the willingness of officials to embrace comparison, and the degree to which measures are incorporated into key management systems distinguish cities that are more likely to use performance measures for service improvement from those less likely to do so.     

绩效管理改革的效果:来自美国联邦政府的证据

从1993年的《政府绩效与结果法案》(GPRA)以及布什政府的项目评估定级工具(PART)已经建立了一种新的政府管理惯例,两者的目的均在于促进美国联邦政府绩效管理实务的发展。但从绩效信息的使用方面来看,两者所做出的努力都极其有限,即一旦考虑其它因素,机构雇员不论是否参与PART评审,是否参与GPRA的实施进程,他们几乎都没有影响到绩效信息的使用。本文采用顺序Probit回归模型,通过对美国联邦审计总署基于1996、2000、2003和2007年的数据进行调研,最终得出依赖于难以观测的官僚行为来实现的政府改革是具有局限性的以及在政府绩效改善的过程中绩效信息使用的重要性。对数据的深层分析还揭示了一系列影响绩效信息使用的组织因素,这包括领导对结果的承诺、监管者引导学习惯例、工作动机的性质、将测量与行动链接起来的能力、管理的自由裁量权以及利益相关者之间的政治冲突。     

Does Involvement in Performance Management Routines Encourage Performance Information Use? Evaluating GPRA and PART

The Government Performance and Results Act (GPRA) of 1993 and the George W. Bush administration’s Program Assessment Rating Tool (PART) established new routines that were intended to foster performance management practices. Using data from two surveys, the authors find that the involvement of agency employees in GPRA processes and PART reviews generally had little direct effect on performance information use once other factors are accounted for. The main exception is that managerial involvement in GPRA processes and PART reviews is associated with the use of performance data to refine measures and goals. This reflects the limits of government‐wide reform efforts that depend on difficult‐to‐observe bureaucratic behavior. The authors also find that a series of organizational factors—leadership commitment to results, learning routines led by supervisors, the motivational nature of the task, and the ability to link measures to actions—are positive predictors of performance information use.     

The Problem with Programs: Multiple Perspectives on Program Structures in Program‐Based Performance‐Oriented Budgets

Often the transition to performance‐based budgeting requires the initial development of a program structure. While a substantial body of literature is available to assist governments in developing performance measures, program development has received little recent attention in the scholarly literature. This research draws on a case study of the State of Georgia's effort to develop a program structure. The problems encountered by Georgia provide the foundation for an analysis that explains how programs interact with competing budgetary purposes and agendas. The analysis in turn suggests ways that budget structures might be adjusted to better balance competing purposes.     

Performance measures in tax administration: Chile as a case study

The article proposes a set of tax administration performance measures and contrasts them with measures actually used by the Chilean tax administration agency. The goals assumed for the tax administration agency (TA) are to maximize tax revenue collection and provide quality services to taxpayers. Ideal performance measures (PMs) would measure the deviation of actual outcome from a best‐practice standard, given the value of all variables affecting organisation performance that are outside management control. The key challenge is to build and calculate these best‐practice outcomes. In Chile the PM in use, for the first goal, is the ratio of actual to potential tax revenue collection. This PM does adjust revenue collection for variations in the tax structure and rate, but it fails to control other variables that affect performance such as the TA budget and per capita income. The PM in use, for the second goal, is taxpayer satisfaction measured through sample surveys. This seems the appropriate PM, as quality of taxpayer services depends directly on the TA efforts to improve them. Copyright © 2005 John Wiley & Sons, Ltd.     

The Professional Researcher as Change Agent in the Government–Performance Movement

Although academic and professional publications give the impression that performance measurement is a growing government practice, in actuality the use of this technology is not as deep or as widespread as it may appear. Even when performance measures are used, governments rarely integrate them into planning, budget, personnel, and other management processes. Most professional researchers located primarily in academic institutions, but also in research and government organizations, approach performance measurement as though governmental officials, elected or otherwise, are already sold on its usefulness. Instead, they need to function as "change agents," using a variety of strategies to gain acceptance and understanding of the strengths and limitations of performance measurement. This article draws on the authors' experiences with the Community Benchmarks Program of the Maxwell School in Onondaga County and a review of the current literature. It suggests guidelines for professional researchers who want to increase the use of performance measures by governments at all levels.     

One Standard Fits All? The Pros and Cons of Performance Standard Adjustments

Despite the wide‐ranging use of performance measures within the public sector and the growing use of performance bonuses to recognize performance achievements, the use of formal performance standards adjustment procedures in public performance measurement systems is relatively rare. Burt S. Barnow of Johns Hopkins University and Carolyn J. Heinrich of the University of Wisconsin–Madison set forth the basic arguments in favor of and against the use of formal or informal procedures for adjusting performance expectations. They describe how performance standards adjustments processes are currently (or have been) in use, review the evidence of their effectiveness or problems encountered in their application, and explore the consequences of the inadequacy of performance standards adjustments. The authors offer recommendations for the improvement of public sector performance measurement systems and conclude why this area remains fruitful for future research experimentation. The nice thing about standards is that there are so many of them to choose from. —Andrew S. Tannenbaum     

Disparate Measures: Public Managers and Performance-Measurement Strategies

Over the past two decades, many studies have investigated the scope and significance of performance measurement in public organizations. Nonetheless, there is more to learn about the challenges facing public managers who want to measure organizational outputs and use the feedback to improve performance. Specifically, managers are often faced with redundant measures of the same output, each of which may be preferred by a different political principal or stakeholder. Furthermore, a manager's choice of measures can have serious consequences for both the estimation of agency problems and the success of programmatic solutions. We test these assertions in an analysis of educational organizations in Texas. We find that managers' assessments of organizational performance and decisions regarding solutions depend on the choice of performance measures.     

Reinventing Reforms: How to Improve Program Management Using Performance Measures. Really

This paper looks at the design and use of incentivized performance measures to motivate managerial efficiency and promote greater program effectiveness. It starts off by looking at recent reforms like the Government Performance and Results Act to understand why they were largely unsuccessful in altering the decision‐making process of government agencies. One problem was that performance measures have been both numerous and complicated, thereby making their role in management and oversight difficult. Equally important, no external incentives were attached to program accomplishments. The paper then examines what elements would be needed to build a management system that encourages both more efficient and more effective agency behavior. The goal of performance budgeting is to develop performance measures that display the progress of a program toward its stated objectives. Assessments based on these measures may then call for rewards or punishments. As such, it also may encourage program managers toward improved performance. The paper examines the pitfalls and complexities dealt with by Congress and Office of Management and Budget in the process. For example, a performance system must distinguish between funding program needs, as warranted by sectoral indicators, and management concerns. It must also unambiguously tie incentives to performance measures to motivate agencies, while building in commitment devices for the principals. Incentivized performance measures may not be appropriate in all conditions, but may be helpful for motivating managers and improving program performance in particular circumstances.     

Models of Performance-Measurement Use in Local Governments: Understanding Budgeting, Communication, and Lasting Effects

While attention has been paid to a few cities and counties exhibiting effective performance measurement systems, most U.S. local governments have been active in the development and use of performance measurement for several decades. This research examines the effects of performance-measurement information on budgetary decision making, communication, and other operations of U.S. local governments. Data are drawn from a national survey of city and county administrators and budgeters that included nearly 300 governments. Findings indicate the use of performance measurement by local departments is pervasive, although survey respondents are less enthusiastic about measurement effectiveness. Study results show subtle distinctions between city and county officials in their use of performance measurement for budgetary purposes and processes. Research findings indicate the consistent, active integration of measures throughout the budget process is important in determining real budget and communication effects in local governments.     

Using Performance Measures for Federal Budgeting: Proposals and Prospects

The political system, the popular press, and the public have recently been concerned about measuring government performance. This concern for measuring performance should imply a concern for measuring it correctly. With this in mind, the Congressional Budget Office (CBO) recently conducted an analysis of the use of performance measures in the budget process. The study attempts to review the issues raised by performance budgeting in the context of past and current efforts to link performance measures and budgeting. This article focuses on two portions of that study: The status of the current federal performance measurement efforts and specific observations designed to inject a note of caution into the current debate about performance measurement and budgeting.     

Do Legislatures Matter in Budgetary Reform?

A recurring problem in the implementation of budgetary reforms at the state level is whether and how to engage legislatures in the effort. This paper posits that legislatures are critical for effective implementation and that different legislative institutional arrangements may have differential effects on both legislative and agency implementation. Drawing on a multistate survey, the analysis shows that higher levels of legislative responsibility for budgeting as well as legislative engagement in oversight of performance information are significantly associated with increased use of performance measures in making budgetary decisions both at the legislative and agency levels.     

Why Measure Performance? Different Purposes Require Different Measures

Performance measurement is not an end in itself. So why should public managers measure performance? Because they may find such measures helpful in achieving eight specific managerial purposes. As part of their overall management strategy, public managers can use performance measures to evaluate, control, budget, motivate, promote, celebrate, learn, and improve. Unfortunately, no single performance measure is appropriate for all eight purposes. Consequently, public managers should not seek the one magic performance measure. Instead, they need to think seriously about the managerial purposes to which performance measurement might contribute and how they might deploy these measures. Only then can they select measures with the characteristics necessary to help achieve each purpose. Without at least a tentative theory about how performance measures can be employed to foster improvement (which is the core purpose behind the other seven), public managers will be unable to decide what should be measured.     

Economic performance and U.S. Senate elections: Reply

Major criticisms made by Chressanthis and Shaffer (C-S) are addressed: the authors' failure to include noneconomic variables in the regression model, the inclusion of open elections in the sample, and the use of variables other than the state unemployment rate as economic performance measures. Errors in C-S's interpretations are noted. Additional regression results are reported using modifications of the authors' original model. These suggest that the change (notrelative change as in C-S) in state unemployment rate is superior, and that the authors' model yields a stronger effect of state economic performance on election outcomes than does the C-S model.John Beck contributed helpful comments.     

Determinants of Performance Measurement: An Investigation into the Decision to Conduct Citizen Surveys

This article develops and tests a comprehensive framework explaining the decision to measure performance, specifically the decision of local governments to conduct citizen surveys. It is structured around a fundamental distinction between subjective performance measures obtained for use in decision making and those that are produced solely for their symbolic value. The author suggests that the field of public administration may be taking a simplified view concerning the promotion and adoption of citizen surveys, overlooking important aspects of the decision-making process of performance-oriented public managers and neglecting the impact of politics and symbols. Political rationality may undercut managerial rationality in the decision to adopt citizen surveys, and symbolic adoption may be the underlying cause of low levels of information use. This study identifies policies to increase adoption of citizen surveys but cautions that simply promoting the adoption of surveys as inherently good may be a naive and wasteful course of action. Practitioners who have already made the decision to measure subjective performance through citizen surveys, or are facing such a decision, can find in this analysis a structure to assess past decisions or guide future decisions.     

Legislative Influences on Performance Management Reform

The management literature argues that legislative involvement is important to the implementation of performance management reform, but it does not specify how legislatures should be engaged or how different legislative organizational arrangements affect reform. This article blends theories of management and legislative professionalism to better understand the influence of legislatures on the implementation of management reform. Drawing on data from several surveys, it examines the influence of legislative organization on the managerial use of performance measures. The findings suggest that citizen legislatures are associated with better administrative practices than professional legislatures and that the quality of legislative involvement may be more important than its quantity.     

Evaluating Capital Management: A New Approach

The objective of this article is to develop a methodology for evaluating capital management performance. We have employed a new methodology, fuzzy rule–based systems (FRBS), that allows evaluators to break down complex systems into manageable components. FRBS can also use a variety of data types by converting data into ordinal input measures that are not overly sensitive to small measurement errors. In developing an overall performance measure, the evaluator under an FRBS creates explicit rules for combining inputs that can reflect the complex contextual judgments commonly made by evaluators. Using survey data on 35 large American cities from the Government Performance Project, we develop an FRBS for city capital management.     

A Multiple–Indicator Approach to Municipal Service Evaluation: Correlating Performance Measurement and Citizen Satisfaction across Jurisdictions

Early work on municipal service–quality assessment recommended multiple measures of performance from both providers and users. Citizen satisfaction surveys have rivaled their more quantitative counterpart, administrative performance measures, in adoption, but the implication of survey results for action is not well understood by managers or scholars. To achieve meaningfully integrated multiple measures of service quality, we need to explore the dimensions of citizen satisfaction and review patterns of satisfaction across localities. We also need to understand the relationship between administrative performance measures and citizen perceptions. This cross–sectional analysis of municipal citizen satisfaction and performance benchmark data suggests that citizen satisfaction survey results are useful to managers in conjunction with performance–measurement programs as part of a multiple–indicator approach to evaluating municipal service quality. However, understanding citizen perceptions requires a different perspective than that applied to administrative service performance measurement.