Program evaluation and the commissioning entity

The present decade has witnessed a shift from planning to emphasis on program performance and accountability. However, program evaluation must contend with many different policy preferences articulated by groups and institutions involved with social programs. In particular, the entity commissioning an evaluation expects some policy dmileage from the undertaking. Evaluation intended as an aid to decisionmaking cannot afford to separate the program under study from its organizational and political context.Federal categorical social programs like Head Start fill many needs for many groups. Evaluation preoccupied with experimental rigor often underestimates the need to adapt to political and administrative realities. At the other extreme, excessive accommodation to program ideology and organizational self-interest undermines the credibility of the evaluative process.Its limitations notwithstanding, program evaluation does yield informed policy alternatives. It invites rational public discussion of social programs and adds substance to the process of political bargaining.     

What can we actually get from program evaluation?

This paper assesses the role program evaluation can play in assisting decisions on public programs. The author looks at evaluation from the standpoint of decisionmakers interested in finding out the right answers about their programs. The discussion focuses on the assistance that various types of evaluation can give to program managers and to policymakers concerned with legislative changes and budget levels. The paper includes recent examples of relevant evaluation work. The concluding section analyzes some of the problems decisionmakers face in trying to get reliable, useful evaluation.     

Fiscal Federalism in 1986: The spotlight Continues To Swing Toward the States and Local Governments

The year in Washington saw the president and the Congress atloggerheads on the budget but in agreement on important taxlegislation. A prominent casualty of the battle to slash thefederal deficit was Revenue Sharing. "Sequestration" and therhetoric of austerity that doomed Revenue Sharing notwithstanding,total federal grants (adjusted for inflation and populationgrowth) rose in 1986 for the second year in a row, continuinga recovery from the six-year decline that had begun in 1979.The net effects of the Tax Reform Act of 1986 will vary substantiallyamong the states, but the federal legislation guarantees thattax reform will be a central issue in many state legislaturesin 1987. Federal mandates continued to be promulgated in 1986with little regard for the costs to states and localities. Anotable exception was the outcome of the struggle over fundingfor grants for the treatment facilities needed to comply withthe standards prescribed by the Clean Water Act of 1972: muchmore generous funding than was sought by the president.     

ENVIRONMENTAL MONITORING; A CASE OF LEARNING THROUGH EVALUATION?

The aim of environmental monitoring programs is to provide information useful in enforcing existing environmental statutes and helping to detect and prevent future problems. Despite the considerable number of monitoring programs being conducted, information on the quality of the nation's environment remains poor and of little use in supporting environmental policymaking. This paper examines the role evaluation has played in developing an effective nationwide environmental monitoring system. Background information on environmental monitoring programs is provided, followed by a discussion of the evaluations of past programs and recommendations made for improvement. One program in particular, the "Status and Trends" program administered by the National Oceanic and ATmospheric Administration, is examined to explore whether environmental monitoring demonstrates a case of learning through evaluation. "Status and Trends" is based on the experience gained during several of the previous monitoring programs and incorporates many of the recommendations spawned from earlier environmental programs. Whether the program demonstrates a case of learning through evaluation will depend on the ability of the program to evolve as we learn more about environmental quality and out ability to measure it.     

The Farm Crisis and the Future of Rural Local Governments

America's farmers are facing difficult financial times, andtheir problems are spilling over to rural local governments.Declining farm incomes and farm property values erode the localtax base. At the same time, demand for publicly provided servicesmay increase, creating a squeeze between falling revenues andhigher costs. This article documents the impact the farm crisishas had on local government tax bases in eight multi-countyregions in the Midwest. The impacts found, while severe, wouldgenerally be manageable in the absence of other shocks. However,when changes in the intergovernmental aid system–suchas the elimination of General Revenue Sharing and other cutsin federal aid—are factored in, local governments in agriculturallydependent areas can encounter financial stress. Higher taxesand lower service levels may permanently change the qualityof life in rural America.     

ATTITUDE MEASURES IN EVALUATION RESEARCH: A RESEARCH NOTE

The paper focuses on the use of attitude measures in evaluation research. It is suggested that attitude measures can assist evaluators in surmounting the problems of assessing program effectiveness both in process and impact evaluations. Attitude change can be conceptualized as the intended output of programs, as intervening between program variables and behavior or as proxy measures of behavioral target variables that are not readily measurable. Attitude measures can play avital role in evaluation research if an adequate methodology is employed.     

Making decentralization work: The view from City Hall

The success of decentralization experiments depends ironically on central government—and, in particular, on the character of City Hall initiatives, designs, and administrative organization. Urban administrators were forced to experiment in the dark with decentralization, and their initiatives often traced an erratic, evolutionary course marked by false steps and by serendipity. In particular, programs that looked promising from a City Hall perspective often failed because they were insensitive to street-level behavior. City Hall initiatives have also been hampered by problems of cooperation between citizens and public employees and by coordination within government. Seen in this light, successful decentralization requires a learning process in which citizens and public employees develop strategies for working together on focused neighborhood problems.This article is based on Chapter 8 of my book, Neighborhood Democracy: The Politics and Impacts of Decentralization (Lexington, Mass.: D. C. Heath, 1973).     

American welfare strategies: Three programs under the social security act

In the United States, a single piece of legislation, the Social Security Act, is the major vehicle through which the cash assistance to citizens is provided. This act contains many subprograms, programs so different in concept, administration and programmatic implication that many people do not know that the same piece of legislation makes them all possible. In this paper three programs—social security, unemployment compensation, and public assistance—are examined in a sociohistorical, sociocultural context. The roots of these programs are analyzed, their current operations outlined, and the policy problems currently confronting them are detailed. The ways in which the programs relate to the political mythology of the society is seen as important. Because of the continual conflicts arising out of the administration of public assistance, three special cases involving that program are mentioned.Special thanks is due to Mrs. Shirley Roles, who made many useful conceptual and editorial suggestions.     

Comment on Kirk Mcclure's “The low‐income housing tax credit program goes mainstream and moves to the suburbs”

Abstract

As McClure's article notes, the Low‐Income Housing Tax Credit (LIHTC) program has indeed gone mainstream. Given the tarnished reputation of many other federal low‐income housing programs, this is good news. It is also surprising in some ways considering the many programmatic flaws inherent in the LIHTC program.

As a point of departure, I look at why McClure and others are able to describe the program in a positive light despite its many flaws. I attribute this to the unique political culture of the United States, for which the LIHTC program is well suited. In addition, it sidesteps one of the thorniest problems that have bedeviled low‐income housing programs—the spatial isolation of poor minorities. Until the LIHTC program explicitly addresses this issue, however, any praise must be tempered by a great deal of caution.     

City Officials and General Revenue Sharing: Attitudes, Perceptions and Implications for Future Intergovernmental Policy

This article describes the 1972 State and Local Fiscal AssistanceAct(General Revenue Sharing), its perceived impact on city expenditures,its impact on the attitudes of city officials, and GRS's implicationfor future intergovernmental policy. Using data from nine yearsof an annual survey to chief executives in cities with over50,000 population, we conclude the program's most importantcontribution was in the general attitudes it fostered amonglocal officials. Our analysis indicates that GRS remains extremelypopular with urban chief executives and that its impact variesamong expenditure categories and cities. We conclude that theReagan administration may be able to develop viable policy alternativesbecause of the lack of consensus among many city officials whoalso exhibit varying degrees of support for the president'sproposed policies. In addition, the consensus which characterizedthe public interest groups in prior years may be eroding, andthis could create greater possibilities for policy change. ^* We would like to thank Tracey Gladstone and Eila Pakkanenfor their coding assist-ance. Ilese Hoyniak provided data processingassistance. We would like to thank Purdue University for providingcomputer time and other assistance.     

Do conditional cash transfer programs shift votes? Evidence from the Honduran PRAF

How do national social programs influence local voting? This study utilizes the experimental set up of a conditional cash transfer program to show that small, targeted cash transfers can have large electoral effects. The Honduran PRAF program allocated an average of $18 per capita per year to poor households within municipalities that were randomly assigned to receive the program. Although the program was administered at the national level, the program increased an incumbent mayor's re-election probabilities by 39%, without significantly influencing voting behavior in presidential elections. Moreover, the evidence suggests that transferring cash to poor households were more effective at increasing political support than interventions providing public goods for poor villages.     

How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment

In recent years, propensity score matching (PSM) has gained attention as a potential method for estimating the impact of public policy programs in the absence of experimental evaluations. In this study, we evaluate the usefulness of PSM for estimating the impact of a program change in an educational context (Tennessee's Student Teacher Achievement Ratio Project [Project STAR]). Because Tennessee's Project STAR experiment involved an effective random assignment procedure, the experimental results from this policy intervention can be used as a benchmark, to which we compare the impact estimates produced using propensity score matching methods. We use several different methods to assess these nonexperimental estimates of the impact of the program. We try to determine “how close is close enough,” putting greatest emphasis on the question: Would the nonexperimental estimate have led to the wrong decision when compared to the experimental estimate of the program? We find that propensity score methods perform poorly with respect to measuring the impact of a reduction in class size on achievement test scores. We conclude that further research is needed before policymakers rely on PSM as an evaluation tool. © 2007 by the Association for Public Policy Analysis and Management     

The three faces of evaluation

Program evaluation can be used for three purposes: to learn about a program's operations and effects, to control the behavior of those responsible for program implementation, and to influence the responses of outsiders in the program's political environment. Most agencies fail to take full advantage of all three faces of evaluation. They begin their evaluation programs too late; they assign evaluation responsibilities to staffs which lack the requisite skills; or they yield to temptations to distort or suppress evaluation findings. The case of the evaluation of civil service reform demonstrates how these multiple purposes can be pursued together with constructive results. It also suggests several lessons of more general relevance.     

Strategies for homeless research in the 1990s

Abstract

Existing empirically based knowledge about homelessness is neither extensive nor well grounded. Proper policy formation and program design that address homelessness are badly served by knowledge deficiencies. This paper lays out a strategy that will develop appropriate research methodology and generate better empirically based knowledge. It proposes a program of technical research, as well as moditica‐tions to several national data collection programs, such as the Current Population Survey, to collect data on the literal homeless and the precariously housed. It also presents a rationale for starting longitudinal studies that focus on the processes that lead into and out of the homeless state. Finally, it proposes evaluation research to measure the effectiveness and efficiency of promising programs.     

How does government funding affect nonprofits' program spending? Evidence from international development organizations

Under the new governance models, the scope of government funding of nonprofit activities in service delivery and policy implementation is extensive. There is a long‐standing concern that government funding may compromise nonprofit operations and lead to unintended consequences. This research examines the concern by exploring the effect of government funding on nonprofits' spending on programs and services. Existing theories and empirical evidence propose competing arguments concerning the relationship between government funding and nonprofits' program spending. Using a 20‐year panel dataset of international development nonprofits registered with the United States Agency for International Development, we find that nonprofits receiving more government funding spend significantly higher proportions of their organizational resources on development programs. Government funding seems to increase nonprofits' program spending and to make them more focused on their mission‐related activities.     

Analysis and urban government: Experience of the New York City—Rand institute

For almost four years New York City and the Rand Corporation have been engaged in an enterprise distinctive if not unique: the subjection of a wide variety of the City's problems to the scrutiny of independent analysts. This paper* is the attempt of a participant in that enterprise to describe the background of that effort and the novel arrangements made to institutionalize it, to outline the nature and effect of the analyses produced, and then to reflect on some of the lessons this effort has taught some of its participants.     

Black-box models of candidate evaluation

All contemporary models of candidate evaluation are memory-based models in that they treat the direction and strength of evaluation as a function of the mix of positively and negatively valued (valenced) information retrieved from memory. Yet, oddly enough, despite the assumption that memory mediates judgment, none of the major models looks at the processes involved in what information voters recall and how that evidence was integrated into a summary evaluation. In this sense then, political science models of vote choice are black-box models: They are silent about how voters actually go about interpreting information and integrating the evidence into a summary evaluation of the candidates. In this article we critique the major political science models, call attention to the implicit assumptions they make about what evidence is assumed to be in memory, and conclude with an argument for introducing process into our explanations of vote choice.     

Presidential address: From the Great Society to continuous improvement government: Shifting from “does it work?” to “what would make it better?”

In the 1960s, various social programs were started (like Head Start) or dramatically expanded (like AFDC). Loosely, this period of expansion is called the Great Society. Too many Great Society social programs, unfortunately, have been disappointments—at least when compared to the high hopes of the '60s. Even if they “work,” most of us wish that they worked much better. Some people take such statements to mean that the Great Society's social programs should be defunded. Most Great Society programs, however, are surely here to stay, for they serve important social functions. How many of us really think there could be an America without a social safety net? It is now time to do the difficult and unglamorous work of systematic program improvement. Instead of testing program efficacy over and over again, we should engage in ongoing and evidence‐based efforts to increase program effectiveness (in both large and small ways). © 2009 by the Association for Public Policy Analysis and Management.