IMPROVING THE QUALITY OF RESEARCH SYNTHESIS IN PROGRAM EVALUATION

Because research synthesis enables one to determine either the overall effectiveness of different types of treatments, it is becoming increasingly popular as a tool in program evaluation. Numerous methodological problems arise, however, when research synthesis is applied to studies conducted in field settings. The present paper categorlzes and discusses these problems as being threats to either the (1) internal validity (whether one can draw conclusions about cause and effect), (2) statistical conclusion validity (whether one's inferential statistics are capable of detecting cause-and effect relationships, (3) construct validity (whether one's treatments and outcome measures are valid operationalizations of the independent and dependent variables of interest), or (4) external validity (whether one can generalize results to particular populations, settings, or time periods) of research synthesis (see Cook & Campbell, 1979). Specific recommendations are made for minimizing these threats to validity, In order to improve the quality of research synthesis in program evaluation. Finally, the validit framework is applied to an actual synthesis to identify strengths, weaknesses, and potential means of enhancing research quality.     

Certainty or Accessibility: Attitude Strength in Candidate Evaluations

Attitude strength is an important, but contested, subject in social psychology. Political scientists often rely on measures of attitude strength such as attitude importance, accessibility, or certainty in their work while ignoring the politically meaningful differences across types of strength. This omission is particularly relevant in the discussion of the formation of candidate evaluations. The research reported here indicates that accessibility is not the relevant type of attitude strength when describing how voters use issues in evaluating candidates. Instead, voters' reliance on issues when evaluating candidates depends on the voter's certainty about where the candidates stand. Given the different antecedents of certainty and accessibility, this result suggests that that citizens are able to more carefully process and use information available to them during an election campaign than would be expected by the prevailing theories of attitude formation .     

Response Latency Methodology for Survey Research: Measurement and Modeling Strategies

In public opinion research, response latency is a measure ofattitude accessibility, which is the ease or swiftness withwhich an attitude comes to mind when a respondent is presentedwith a survey question. Attitude accessibility represents thestrength of the association in memory between an attitude objectand an evaluation of the object. Recent research shows thatattitude accessibility, as measured by response latency, castslight on a wide range of phenomena of public opinion and politicalbehavior. We discuss response latency methodology for surveyresearch and advocate the use of latent response latency timers(which are invisible both to respondents and interviewers) asa low cost, low-maintenance alternative to traditional methodsof measuring response latency in public opinion surveys. Weshow that with appropriate model specification latent responselatency timers may provide a suitable alternative to the morecomplicated and expensive interviewer-activated timers.     

FIELD NETWORK RESEARCH IN POLICY EVALUATION

The field-network evaluation studies (FNES) approach to policy evaluation research seeks to overcome deficiencies of small-sample case study method and of the large-sample survey research approach. Methodologically, FNES uses many of the familiar techniques of both these approaches interviews, data collection and analysis, limited sampling, field observation, and document analysis. The crucial difference is sample size. The FNES approach using a middle-range sample of 50 to 60 research sites has the advantages of the case study method in being able to provide an in depth account of a program and how it operates, and is sufficiently flexible to permit a shift in the analytical framework as the research proceeds thus overcoming the inflexibility of the instruments of survey research.
Seven major studies employing the FNES approach have been carried out or are underway in the United States of America. The principal components of the methodology are a network of university-based field associates and a central management group. The field associates collect the information and the data using a uniform analytical framework and reporting form. The central staff aggregates these analyses into a single report that cuts across the sample. This approach and the use of a middle-range sample make it possible to draw generalizable conclusions based on the national experience, and also provide sufficient detail to differentiate policy impacts among the sample jurisdictions.     

State Capacity,Bureaucratic Politicization,and Corruption in the Brazilian State

Responding to recent articles in Governance highlighting the need for improved measurement of bureaucratic characteristics, this article describes efforts to map Brazil's federal agencies on three dimensions—capacity, autonomy, and partisan dominance—derived from data on more than 326,000 civil servants. The article provides a “proof of concept” about the utility of agency‐level measures of these variables, demonstrating how they relate to an output common to all agencies: corruption. The article provides a first step in the direction of building a comparative research program that offers objective evaluation of bureaucracies within nation‐states, with the intent of better disentangling their impact on governance outcomes.     

NEW DIRECTIONS IN MEDICAL OUTCOMES RESEARCH: THE VIEW FROM HARVARD

This study evaluates the emerging field of medical outcomes research based on interviews at Harvard Medical School. Medical outcomes research at Harvard is multifaceted, but may be summarized as involving certain key components. The emphasis at Harvard is on “quality of care,” viewed as a combination of outcomes evaluation and process evaluation, delivery of the service and its result. In a time of shrinking budgets, outcomes and process research can be justified economically because of the relatively low cost of patient self‐reports and because of their usefulness. Physicians may be motivated to measure quality of care by emphasizing the positive effects of research on the process of care and outcomes. Medical outcomes research is a field that is undergoing change and is becoming increasingly oriented toward the practical needs of patients and physicians.     

When Do Regression‐Adjusted Performance Measures Track Longer‐Term Program Impacts? A Case Study for Job Corps

The use of performance management systems has increased since the Government Performance and Results Act of 1993. While these systems share the goal of trying to improve service delivery and participant outcomes, they do not necessarily provide information on the causal (value‐added) effects of a program, which requires a rigorous impact evaluation. One approach for potentially improving the association between program performance measures and impacts is to adjust performance measures for differences across performance units in participant characteristics and local economic conditions. This article develops a statistical model that describes the conditions under which regression adjustment improves the performance–impact correlation. We then use the model to examine the performance–impact association using extensive data from a large‐scale random assignment evaluation of Job Corps, the nation's largest training program for disadvantaged youths. We find that while regression adjustment changes the Job Corps center performance measures, the adjusted performance measures are not correlated with the impact estimates. The main reasons are the weak associations between the unadjusted Job Corps performance measures and participants’ longer‐term outcomes as measured by the evaluation, as well as the likely presence of unobserved factors across centers that are correlated with outcomes.     

Assessing the role of vocational rehabilitation in disability policy.

Since 1975, vocational rehabilitation has represented a small and declining component of federal disability policy. This trend is perhaps reflective of the relatively crude assessment techniques that have been applied to the program in the past. Using the Virginia Vocational Rehabilitation (VR) program as a prototype, we outline how the data and methods of assessment can be improved for purposes of directing public policy. The key issues include identifying an appropriate comparison group for VR, analysis of longitudinal earnings data, and methods for refining measures of program cost. The analysis provides "fixed-effects" estimates of net earnings impacts for each of three postprogram years stratified by disability classification and gender. These treatment impacts are compared to total and service-specific costs. In general, this analysis suggests that evaluation of VR can be substantially improved and that these improvements can be attained at relatively modest analytic cost.     

PROGRAM EVALUATION AND ADMINISTRATIVE CONTROL

What is the role of program evaluation research in making and implementing social policy and how do program evaluators view the social programs they evaluate? This article discusses the findings of two closely related studies that examine these questions from the perspective of organization theory. These studies found that program evaluations rely on methods of research that do not correspond to the extent of uncertainty in social programs over goals and causal knowledge, and that most evaluations define organizational effectiveness solely in terms of goal achievement. In addition, of the various groups with a stake in the evaluation process, only federal and top program administrators influenced the implementation and interpretation of the studies. These findings, when taken together, suggest that program evaluation research acts more to augment administrative control than to increase our understanding of the effects of social policy.     

Explicit Evidence on the Import of Implicit Attitudes: The IAT and Immigration Policy Judgments

The implicit association test (IAT) is increasingly used to detect automatic attitudes. Yet a fundamental question remains about this measure: How well can it predict individual judgments? Though studies find that IAT scores shape individual evaluations, these inquiries do not account for an array of well-validated, theoretically relevant variables, thus raising the challenge of omitted variable bias. For scholars using the IAT, the risk here is one of misattributing to implicit attitudes what can be better explained by alternate and rigorous self-reports of explicit constructs. This paper examines the IAT’s performance in the context of U.S. immigration politics. Using a representative web survey of adults, I demonstrate the IAT effectively captures implicit attitude toward Latino immigrants. Critically, I then show these attitudes substantively mold individual preferences for illegal and legal immigration policy, net of political ideology, socio-economic concerns, and well-established measures of intolerance toward immigrants, such as authoritarianism and ethnocentrism. Combined, these results suggest the IAT measures attitudes that are non-redundant and potent predictors of individual political judgments.     

What Can We Learn From A Doubly Randomized Preference Trial?—An Instrumental Variables Perspective

The doubly randomized preference trial (DRPT) is a randomized experimental design with three arms: a treatment arm, a control arm, and a preference arm. The design has useful properties that have gone unnoticed in the applied and methodological literatures. This paper shows how to interpret the DRPT design using an instrumental variables (IV) framework. The IV framework reveals that the DRPT separately identifies three different treatment effect parameters: the Average Treatment Effect (ATE), the Average Treatment Effect on the Treated (ATT), and the Average Treatment Effect on the Untreated (ATU). The ATE, ATT, and ATU parameters are important for program evaluation research because in realistic settings many social programs are optional rather than mandatory and some people who are eligible for a program choose not to participate. Most of the paper is concerned with the interpretation of the research design. To make the ideas concrete, the final section provides an empirical example using data from an existing DRPT study.     

RESEARCH, EVALUATION, AND POLICY ANALYSIS: HEURISTICS FOR DISCIPLINED INQUIRY

Terminology like "evaluation research" and "policy analysis research" poses a fundamental conflict in talking about what constitutes legitimate inquiry activities. Using the two-part definition of disciplined inquiry developed by Cronbach and Suppes (1969). it is possible to characterize research, evaluation and policy analysis as three separate and distinct forms of disciplined inquiry. As distinct forms, each has its own unique definition, purpose(s), products, intended outcomes or effects, and audiences. Further, under the definitions proposed, emergent inquiry paradigms have as much utility as older, conventional models.     

Judging presidential character: The demise of Gary Hart

This article investigates the public reaction to the scandal which effectively ended Gary Hart's quest for the 1988 Democratic presidential nomination. Employing NES panel data covering the period in which the scandal surfaced, and integrating arguments drawn from research on attitude change, media priming, and candidate evaluation, this analysis of the Hart case illuminates more general questions about how citizens respond to media communications during the course of an election campaign and of the factors that facilitate or inhibit attitude change. The investigation lends support to contemporary theories of attitude change that emphasize citizens' levels of political involvement and prior predispositions; uncovers evidence of media priming as views about controversial standards of morality were newly engaged in defining citizens' post-scandal evaluations of Hart; and yields evidence that negative responses to Hart in the wake of the scandal were tempered among citizens who typically weigh policy criteria alongisde candidate characteristics when formulating their overall candidate evaluations.     

The Catastrophic Link Between the Importance and Extremity of Political Attitudes

The catastrophe theory of attitudes (Latané and Nowak, 1994) predicts that unimportant attitudes act as continuous dimensions, with normal distributions and gradual changes in evaluation, while important attitudes act as categories, with bipolar or unipolar extreme distributions and catastrophic (abrupt) changes in evaluation. A major derivation from this theory is that attitude importance and extremity should be correlated, with more important attitudes being more extreme. This prediction was confirmed for 14 specific political issues at both the group and the individual level, as well as for political involvement and general liberalism. However, general political involvement was not related to the extremity of evaluation for specific issues; similarly, partisanship predicted extremity of general liberalism but not extremity on specific issues. Results suggest that attitude importance and extremity must be measured at corresponding levels of specificity in order for a relationship between them to hold. These results have implications for attitude change in both individuals and societies.     

Society as emergent and more than rational: An essay on the inappropriateness of program evaluation

In considering the efficacy of program evaluation researchers have not put language in proper perspective. They have not acknowledged that information gained from applied research is partly determined by situational factors, that language is selective and goal-oriented, justifying activity as often as it describes it, or that some important action cannot be expressed. The importance of language distinguishes applied social assessment from the natural science models of research from which it was derived. Examples from a wide range of applied situations are used to illustrate that, as an anticedent-consequent model of objective quantified assessment, evaluation is also inconsistent with the complex, resourceful, emergent social processes it is intended to measure. Three alternatives to quantification are discussed to illustrate interventions that may be made when the disadvantages of program evaluation outweigh the advantages.Preparation of the paper was supported by National Institute of Mental Health post-doctoral training grant 5T222-MH00180. Comments and requests for reprints should be sent to Nancy Cochran, Department of Mental Health and Developmental Disabilities, State of Illinois, 160 N. LaSalle Street, Chicago. Illinois 6061.     

Multiple objectives in environmental protection programs

This research demonstrates the need for and the usefulness of a multi-dimensional approach to the management of environmental programs. While the thrust of the paper is toward environmental quality programs the conceptual development is appropriate for many public sector programs. The paper describes an experiment in the determination of multi-dimensional objectives within three programs in a state environmental quality program. In addition, measures of effectiveness, based on the multi-dimensional objectives, are proposed as relevant measures of program effectiveness.This research was sponsored in part by the Pennsylvania Department of Environmental Resources under contract SPC 412070 and in part by the Center for the Study of Environmental Policy, The Pennsylvania State University, University Park, Pennsylvania.     

Evaluating the Evaluation Strategy as an Instrument for Budgetary Control

This article reports recent research undertaken on the contribution of program evaluation to that part of the budget process for which the Australian public service has carriage. It examines the degree of success which the Labor government's evaluation strategy attained in serving budget agency information purposes. The research is not conclusive, only indicative. Based on two sets of evidence — interviews with officers in the Department of Finance¹ and an analysis of survey results on the use of evaluation in the budget — it contends the evaluation strategy has had only a marginal impact on central budgetary processes. By documenting changes in the approaches of budget officials the article reports the impact of program evaluation on the core function of Finance — the provision of policy advice relating to the expenditure priorities of government.     

Strategies for homeless research in the 1990s

Abstract

Existing empirically based knowledge about homelessness is neither extensive nor well grounded. Proper policy formation and program design that address homelessness are badly served by knowledge deficiencies. This paper lays out a strategy that will develop appropriate research methodology and generate better empirically based knowledge. It proposes a program of technical research, as well as moditica‐tions to several national data collection programs, such as the Current Population Survey, to collect data on the literal homeless and the precariously housed. It also presents a rationale for starting longitudinal studies that focus on the processes that lead into and out of the homeless state. Finally, it proposes evaluation research to measure the effectiveness and efficiency of promising programs.