PROGRAM EVALUATION AND ADMINISTRATIVE CONTROL

What is the role of program evaluation research in making and implementing social policy and how do program evaluators view the social programs they evaluate? This article discusses the findings of two closely related studies that examine these questions from the perspective of organization theory. These studies found that program evaluations rely on methods of research that do not correspond to the extent of uncertainty in social programs over goals and causal knowledge, and that most evaluations define organizational effectiveness solely in terms of goal achievement. In addition, of the various groups with a stake in the evaluation process, only federal and top program administrators influenced the implementation and interpretation of the studies. These findings, when taken together, suggest that program evaluation research acts more to augment administrative control than to increase our understanding of the effects of social policy.     

EVALUATION AS SOCIAL POLICY

Policy research has become an integral part of the policymaking process but is changing as the goals and emphases o f policies change. The paper explores the shift in emphasis in evaluation research away from predominantly programmatic concerns and toward accountability issues in personnel and job evaluation. It depicts an inward turning in domestic policies from the provision of services to evaluation of government itself, and an acceleration in conflict over how the work of government should be structured and performed, and by whom. Using examples from three levels o f government, it describes what makes policies controversial, their evaluations political, and how the search for legitimacy and legitimizing technologies raises the stakes for all actors in the policy process–elected officials, administrators, consumers, and evaluators. Conclusions are drawn regarding the effect on research o f these changes in policy.     

PROGRAM COORDINATION IN THE EMPLOYMENT SERVICE: A STATE AND LOCAL POLICY PERSPECTIVE

This paper analyzes the role of coordination criteria in employment and training service delivery within the Job Training Partnership Act system. The evolution of coordination criteria is related to current initiatives in the Employment Service at the state and local levels. An operational set of program outcome measures for monitoring coordination efforts is discussed. This set of outcome measures is formulated in the broad context of stabilizing employment and averting the "structuralization" of unemployment impacts wrought by fundamental economic change in the labor market. The assessment and monitoring of such coordination efforts is discussed.     

PROGRAM IMPLEMENTATION VERSUS PROGRAM DESIGN: WHICH ACCOUNTS FOR POLICY "FAILURE"?

Many influential implementation scholars now argue that "street-level" bureaucrats, rather than legislators or high-level administrators, make public policy in the U.S. Such authors as Pressman and Wildavsky cite creaming in employment and training programs as an especially clear example of well-meaning programs that fail when implemented. This paper argues that two of the most significant and lasting of these programs, the U.S. Employment Service and the Manpower Development and Training Act, were designed to encourage creaming. The essay asserts implementation scholars overstate the disconnection between program design and program implementation because they assume there is little disconnection between program legitimation and program design. A better conception of design permits one to perceive that these programs were legitimated on the grounds they would serve a large number of constituents, but were designed to do so by serving employers. The combination of these premises made creaming an imperative of program operation, and the implementors who cream remain faith ful to original program strategy. This finding suggests a redirection of policy research toward a more rlgorous analysis of program design and a better understanding of the relationship between legitimation, design, and implementation.     

RESEARCH, EVALUATION, AND POLICY ANALYSIS: HEURISTICS FOR DISCIPLINED INQUIRY

Terminology like "evaluation research" and "policy analysis research" poses a fundamental conflict in talking about what constitutes legitimate inquiry activities. Using the two-part definition of disciplined inquiry developed by Cronbach and Suppes (1969). it is possible to characterize research, evaluation and policy analysis as three separate and distinct forms of disciplined inquiry. As distinct forms, each has its own unique definition, purpose(s), products, intended outcomes or effects, and audiences. Further, under the definitions proposed, emergent inquiry paradigms have as much utility as older, conventional models.     

POLICY EVALUATION. DEMOCRATIC THEORY. AND THE DIVISION OF SCHOLARLY LABOR

The implicit priorities of political science have been inhospitable to policy evaluation as a scholarly endeavor. Political science, at least as practiced in the West, draws from a philosophic tradition that views political life, and especially democratic political life, as a public objective, not principally as instrumental for other social ends. Comparative policy analysis deviated from that tradition by treating political conditions as independent variables. Policy evaluation, which assesses the social consequences of governmental actions, deviated even further, adopting an explicitly and totally instrumental approach. The argument ofthis essay is that the tactics of policy evaluation should be adopted for assessing the impact of policy patterns on democratic institutions and citizenship. This recommendation flows from a diagnosis of the division of labor in policy evaluation and an identification of the optimum entry points for political science.     

FIELD NETWORK RESEARCH IN POLICY EVALUATION

The field-network evaluation studies (FNES) approach to policy evaluation research seeks to overcome deficiencies of small-sample case study method and of the large-sample survey research approach. Methodologically, FNES uses many of the familiar techniques of both these approaches interviews, data collection and analysis, limited sampling, field observation, and document analysis. The crucial difference is sample size. The FNES approach using a middle-range sample of 50 to 60 research sites has the advantages of the case study method in being able to provide an in depth account of a program and how it operates, and is sufficiently flexible to permit a shift in the analytical framework as the research proceeds thus overcoming the inflexibility of the instruments of survey research.
Seven major studies employing the FNES approach have been carried out or are underway in the United States of America. The principal components of the methodology are a network of university-based field associates and a central management group. The field associates collect the information and the data using a uniform analytical framework and reporting form. The central staff aggregates these analyses into a single report that cuts across the sample. This approach and the use of a middle-range sample make it possible to draw generalizable conclusions based on the national experience, and also provide sufficient detail to differentiate policy impacts among the sample jurisdictions.     

A HANDS-ON APPROACH TO TEACHING PROGRAM EVALUATION

This article examines the changing nature of program evaluation and the way in which program evaluation is taught to MPA students. Drawing on her experience at George Washington University, the authors explains an approach to teaching program evaluation that has received a great deal of positive feedback. The approach has its foundation, real-world, hands-on evaluation by the students, which gives them invaluable experience. Furthermore, because there is extensive instructor involvement, the author asserts that the experience proves also to be of value to the professor.     

IMPLEMENTING A PROGRAM OF COOPERATIVE FEDERALISM IN SURFACE MINING POLICY

This article outlines the legislative history of the Surface Mining Control and Reclamation Act of 1977 and describes the forces that promoted a federalist arrangement of shared powers and responsibility for implementing the environmental reclamation provisions of the Act. The major components of the Act are reviewed and the Act is examined in terms of its general effectiveness in promoting reclamation as well as its contribution to the practice of American federalism and the capacity of state government to implement complex regulatory programs. The article concludes that SMCRA has had a positive impact in terms of improving state capabilities, expanding public involvement in decision making, and enhancing federal and state cooperation.     

AN IMPACT IDENTIFICATION METHOD FOR DEVELOPMENT PROGRAM EVALUATION

Evaluation is essential to planning and implementing effective develop- ment programs. The resource limitations, multidimensionality of develop- ment program impacts, cultural barriers between evaluators and those impacted, and the diversity of settings in which evaluations must be con- ducted arc serious constraints on the use of traditional research methods to evaluate development programs. Similar constraints are faced in con- ducting most social impact assessments. We suggest that a method of impact identification that makes use of experts, including the people ef- fected by a project, can be used in development project evaluation. The method, which is based on the noniinal group technique, is described, as are some of its applications in Latin America and the Caribbean. We conclude iiy considering directions for future development and the relationship be- tween planning, evaluation, and impact assessment.     

POLICY COMMUNITY,POLICY NETWORKS AND SCIENCE POLICY IN AUSTRALIA

Abstract: The policy community approach to policy analysis has been developed from ideas expressed by Richardson and Jordan in 1979. The approach is increasingly used to identify the key actors and groups of actors who make policy decisions, to analyse the interactions between the groups and to hypothesise about the types of outcomes achieved. The approach aids analysis in policy contexts which cut across the boundaries of individual organisations in the policy process, across the public and private sectors, and beyond national boundaries. This paper evaluates several expositions of the approach with reference to an analysis of science policy in Australia and suggests typological refinements which enhance its explanatory capacity.     

IMPROVING THE QUALITY OF RESEARCH SYNTHESIS IN PROGRAM EVALUATION

Because research synthesis enables one to determine either the overall effectiveness of different types of treatments, it is becoming increasingly popular as a tool in program evaluation. Numerous methodological problems arise, however, when research synthesis is applied to studies conducted in field settings. The present paper categorlzes and discusses these problems as being threats to either the (1) internal validity (whether one can draw conclusions about cause and effect), (2) statistical conclusion validity (whether one's inferential statistics are capable of detecting cause-and effect relationships, (3) construct validity (whether one's treatments and outcome measures are valid operationalizations of the independent and dependent variables of interest), or (4) external validity (whether one can generalize results to particular populations, settings, or time periods) of research synthesis (see Cook & Campbell, 1979). Specific recommendations are made for minimizing these threats to validity, In order to improve the quality of research synthesis in program evaluation. Finally, the validit framework is applied to an actual synthesis to identify strengths, weaknesses, and potential means of enhancing research quality.     

INTERNATIONAL FUTURES AND POLICY

Donna Meadows with John Richardson and Gerhart Bruckman . Groping in the Dark: The First Decade of Global Modelling .
Brita Schwarz with Uno Svedin and Björn Wittrock . Methods in Future Studies: Problems and Applications .
U.S. Congress, Office of Technology Assessment . Global Models, World Futures, and Public Policy: A Critique .
Council on Environmental Quality and Department of State . The Global 2000 Report to the President: Entering the Twenty-First Century
Organization for Economic Co-operation and Development , Interfutures: Facing the Future: Mastering the Probable and Managing the Unpredictable
Richard Barnet . The Lean Years: Politics in the Age of Scarcity
Lester Brown . Building a Sustainable Society
Orio Giarini . Dialogue on Wealth and Welfare: An Alternative View of World Capital Formation
Julian Simon . The Ultimate Resource
Michael Marien and Lane Jennings (Eds.) . Future Survey Annual