Taking Nonprofit Intermediaries Seriously: A Middle‐Range Theory for Implementation Research

The federal government often works through nonprofit intermediaries to reach and empower communities in the United States. One increasingly popular policy strategy is to offer grant funding to intermediary organizations in an effort to strengthen communities. Funded intermediaries are tasked with building the capacity of faith‐based and community organizations at the local level, but the policy theory that informs these programs does not specify how these capacity‐building efforts will lead to stronger communities. Missing is a middle‐range implementation theory that links inputs to community‐level changes through the actions of an intermediary. Derived from empirical case study evidence using process‐tracing analysis, the theory of the community‐integrated intermediary posited in this article helps fill that gap.     

How Credible Is the Evidence,and Does It Matter? An Analysis of the Program Assessment Rating Tool

This research empirically assesses the quality of evidence that agencies provided to the Office of Management and Budget in the application of the Program Assessment Rating Tool (PART), introduced in 2002 to more rigorously, systematically, and transparently assess public program effectiveness and hold agencies accountable for results by tying them to the executive budget formulation process and program funding. Evidence submitted by 95 programs administered by the U.S. Department of Health and Human Services for the PART assessment is analyzed using measures that capture the quality of evidence and methods used by programs and information on characteristics of agencies that might relate to program results and government funding decisions. The study finds that of those programs offering some evidence, most was internal and qualitative, and about half did not assess how their performance compared to other government or private programs with similar objectives. Programs were least likely to provide externally generated evidence of their performance relative to long‐term and annual performance goals. Importantly, overall PART and results scores were (statistically) significantly lower for programs that failed to provide quantitative evidence and did not use long‐term measures, baseline measures or targets, or independent evaluations. Although the PART program results ratings and overall PART scores had no discernible consequences for program funding over time, the PART assessments appeared to take seriously the evaluation of evidence quality, a positive step forward in recent efforts to base policy decisions on more rigorous evidence.     

Museum Outreach Programs to Promote Community Engagement in Local Environmental Issues

Ideas of ‘community’ and ‘community voice’ have been mobilised in collaborative programs developed between the National Museum of Australia and the Murray‐Darling Basin Commission since 1992. This collaboration is set within a broader context of changing ideas and practice around governance, community and environmental issues. The recent Murray‐Darling Outreach Project (MDOP), a series of museum outreach projects with regional communities aiming to increase community engagement in local environmental issues in the Murray‐Darling Basin, is specifically examined. Evaluation research on the MDOP found that the approach to program development significantly shaped the types of voices, range of issues presented and the nature of the communicative forums established. The potential impacts on online audiences are discussed along with broader implications for government agencies involved in community partnerships.     

From Birth to School: Early Childhood Initiatives and Third‐Grade Outcomes in North Carolina

This study examines the community‐wide effects of two statewide early childhood policy initiatives in North Carolina. One initiative provides funding to improve the quality of child care services at the county level for all children between the ages of 0 to 5, and the other provides funding for preschool slots for disadvantaged four‐year‐olds. Differences across counties in the timing of the rollout and in the magnitude of the state financial investments per child provide the variation in programs needed to estimate their effects on schooling outcomes in third grade. We find robust positive effects of each program on third‐grade test scores in both reading and math. These effects can best be explained by a combination of direct benefits for participants and spillover benefits for others. Our preferred models suggest that the combined average effects on test scores of investments in both programs at 2009 funding levels are equivalent to two to four months of instruction in grade 3.     

The Impact of the Australian Government Job Network Contracting on Not‐for‐Profit Service Providers

This article examines the impact of Australian government contracting for the provision of Job Network services on not‐for‐profit organisations ¹ . The term not‐for‐profit is used to categorise organisations which are variously described as non‐government, non‐profit, community and third sector organisations ( Lyons 2001 ).     

The Impact of Elections on Cooperation: Evidence from a Lab‐in‐the‐Field Experiment in Uganda

Communities often rely on sanctioning to induce public goods contributions. Past studies focus on how external agencies or peer sanctioning induce cooperation. In this article, we focus instead on the role played by centralized authorities, internal to the community. Combining “lab‐in‐the‐field” experiments with observational data on 1,541 Ugandan farmers from 50 communities, we demonstrate the positive effect of internal centralized sanctioning authorities on cooperative behavior. We also show that the size of this effect depends on the political process by which authority is granted: subjects electing leaders contribute more to public goods than subjects who were assigned leaders through a lottery. To test the ecological validity of our findings, we relate farmers’ behavior in the experiment to their level of cooperation in their community organization. We show that deference to authority in the controlled setting predicts cooperative behavior in the farmers’ natural environment, in which they face a similar social dilemma.     

Preventing the Use of Deadly Force: The Relationship between Police Agency Policies and Rates of Officer‐Involved Gun Deaths

Killings of civilians by police officers have become a matter of intense public concern in the United States. High‐profile deaths, especially those of black citizens, have caused outrage and sparked the Black Lives Matter movement with calls for dramatic changes in how police agencies operate. However, little systematic research exists to answer questions about which policies should be ended or put in place to reduce these deaths. The authors leverage a large data set of gun deaths by police officers in the United States, combined with agency‐level policy data and community demographic data, to examine whether certain policies are associated with lower or higher rates of officer‐involved gun deaths. Findings show that one policy—the requirement that officers file a report when they point their guns at people but do not fire—is associated with significantly lower rates of gun deaths.     

An International Comparison of the Incidence of Public Sector Whistle‐Blowing and the Prediction of Retaliation: Australia,Norway, and the US

We compare results of several large‐scale survey studies of whistle‐blowing by public sector employees, from samples in Australia, Norway, and the United States. Specifically, we review incidence rates of wrongdoing, whistle‐blowing and retaliation, as perceived by thousands of respondents employed by many government agencies in these countries. Despite differences in measurement in the studies and variation in results, findings suggest that wrongdoing was quite frequently observed and was reported in many cases. Retaliation resulted for far fewer than half of the whistle‐blowers, in all three nations. We also compare findings from the various studies concerning variables that predicted retaliation against whistle‐blowers in the different samples. Differences in results across the three countries add to existing evidence that country and workplace cultures can affect the whistle‐blowing process.     

The Cost‐Effectiveness of Welfare‐to‐Work Programs: A Meta‐Analysis

This article uses the tools of meta‐analysis to assess cost–benefit studies of 50 mandatory welfare‐to‐work programs that were targeted at Aid for Families with Dependent Children recipients and evaluated by random assignment. The findings suggest that the costs of a typical evaluated welfare‐to‐work program probably exceeded its benefits from the perspective of the government, but those assigned to the program, and possibly society as a whole, may reap small positive net benefits. However, there are individual programs that are very cost‐beneficial. Further analyses of the determinants of the cost‐benefits of welfare‐to‐work programs to government and society as a whole show that some services are cost‐neutral, but others entail net costs, which are sometimes substantial. The findings suggest that less successful programs might be made more cost‐beneficial by dropping vocational training and basic education as program components, leaving mainly lower cost components, such as mandated job search and sanctions, but also possibly the more costly financial incentives.     

Energy demand‐side management: New perspectives for a new era

Over the past decade and a half, state governments have assumed greater responsibility over demand‐side management (DSM) operations. Whereas DSM programs formerly were initiated primarily by utilities or state public utility commissions, they are now becoming increasingly state‐initiated and incentivized through funding mechanisms or efficiency‐level mandates. The supporting literature, however, has yet to respond to these changes and to verify that DSM funding or mandates are effective policy mechanisms. Furthermore, the supporting literature has yet to resolve some of the research design and methodological challenges that traditionally plague DSM evaluations. As states continue to expand their energy and climate policy efforts, and the federal government considers the possibility of national decarbonization policies, of which DSM is a key strategy, the need for empirical research on the effectiveness of DSM programs will grow. This essay describes the current status of DSM efforts in the U.S. and explores how these programs affect electricity operations. The relationship between DSM policy and program efforts and the amount of saved electricity is tested with a methodological approach aimed at minimizing the selection bias that is inherent in the nonexperimental research design. Results confirm that state‐run DSM efforts contribute to electricity savings across the country. Public benefit funds coupled with performance incentives are found to encourage utility participation in DSM programs. Energy efficiency portfolio standards and performance incentives effectively promote electricity savings, but public benefit funds without the support of other DSM policies are not significant drivers of either DSM program participation or total DSM electricity savings. © 2011 by the Association for Public Policy Analysis and Management.     

The dissemination and utilization of welfare‐to‐work experiments in state policymaking

Abstract This paper reports the results of a telephone survey of state‐level officials as to the influence of evaluations of three state welfare innovations: California's GAIN, New York's CAP, and Florida's Project Independence. The three experiments were known to those interviewed, yet they did not have dramatic, decisive effects on policymaking. However, GAIN and CAP appear to have influenced policymaking in less dramatic and more subtle respects. Much more important than empirical findings about the effects of tested programs was information about how these programs actually operated in the field along with evidence that the policies tested in welfare‐to‐work experiments were logically consistent (that is, there was no obvious reason to think that they would be unsuccessful), could clear federal waivers, and would not encounter major political resistance. © 2000 by the Association for Public Policy Analysis and Management.     

THE OPERATIONAL DIMENSIONS OF RESULTS‐BASED FINANCING

As part of a growing focus on the effectiveness of development assistance from the World Bank and other agencies, new efforts are being made to relate development finance more closely to outcomes achieved rather than to inputs used, through the results‐based financing approach. We provide a framework for analyzing the operational dimensions of results‐based financing, including the conditions that suit this approach, and how best to define, measure, and report results. We review some of the early World Bank experience with this approach. Noting that this approach is as yet not fully tested, we suggest evaluative issues for future research while highlighting strengths and challenges in the range of techniques adopted so far. Copyright © 2014 John Wiley & Sons, Ltd.     

Investing in Not‐for‐Profit Sector Capacity: The Australian Capital Territory's Community Sector Development Program (CSDP)

In 2013 the Australian Capital Territory (ACT) government's Community Services Directorate (CSD) initiated a suite of projects designed to address the not‐for‐profit (NFP) community services sector's capacity to adapt to a rapidly changing policy and operational environment. In common with other Australian governments, the ACT relies upon NFP organisations to deliver a wide range of community‐based human/social services. The procurement of services rendered to, or on behalf of, government by a third party provider under contract accounts for about 94% of the CSD's investment in the NFP community services sector. The ACT government, therefore, has a vested interest in ensuring that NFP providers are operationally capable, financially viable, and economically sustainable. Accordingly, the government launched a Community Sector Reform Program (CSRP) focussing on red tape reduction, sector development, and the establishment of a community sector outcomes framework. What lessons might the CSRP offer for other jurisdictions?     

Government Reform,Political Ideology,and Administrative Burden: The Case of Performance Management in the Bush Administration

This article examines how ideological differences between political officials and agencies may have affected the implementation of an ostensibly nonpartisan, government‐wide administrative initiative: the George W. Bush administration's Program Assessment Rating Tool (PART) review of federal programs. The analysis reveals that managers in agencies associated with liberal programs and employees (“liberal agencies”) agreed to a greater extent than those in agencies associated with conservative programs and employees (“conservative agencies”) that PART required significant agency time and effort and that it imposed a burden on management resources. Further analysis reveals that differences in reported agency effort can be explained partly by objective differences in the demands that PART placed on agencies—liberal agencies were required to evaluate more programs and implement more improvement plans relative to their organizational capacity—and partly by the ideological beliefs of employees—on average, liberal managers reported more agency effort, even after accounting for objective measures of administrative burden.     

Linking program implementation and effectiveness: Lessons from a pooled sample of welfare‐to‐work experiments

This paper addresses the question: How does implementation influence the effectiveness of mandatory welfare‐to‐work programs? Data from three large‐scale, multi‐site random assignment experiments were pooled; quantitative measures of program implementation were constructed; and multilevel statistical modeling was used to examine the relationship between program implementation and effects on short‐term client earnings. Individual‐level data were analyzed for 69,399 sample members and group‐level implementation data were analyzed for 59 local programs. Findings indicate that, other things being equal, earnings effects are increased by: an emphasis on quick client employment, an emphasis on personalized client attention, staff caseloads that do not get too large, and limited use of basic education. Findings also show that mandatory welfare‐to‐work programs can be effective for many types of people, and that focusing on clients who are especially job‐ready (or not) does not have a consistent influence on a program's effectiveness. © 2003 by the Association for Public Policy Analysis and Management     

Autonomy and Control in Australian Agencies: Data and Preliminary Findings from a Cross‐National Empirical Study

This article is developed from data gathered through the analysis of a survey of ‘agencies’ at Commonwealth and Australian Capital Territory (ACT) levels of government, undertaken as the Australian component of a cross‐national empirical study of autonomy and control in non‐departmental public sector organisations. It presents preliminary findings from one part of what is potentially a very important enterprise in comparative research, along with discussion of methodological issues which need to be confronted in many such comparative exercises. The data reveal that Australia agencies have been granted more autonomy than agencies in other countries contributing to this survey, though that autonomy varies markedly across functions such as personnel management and financial management. While the article represents just a snapshot in time in agency autonomy, we believe it provides a robust baseline for future changes in the way agencies are managed in the Australian public sector.     

Partnerships between local governments and community‐based organisations: exploring the scope for synergy

The utility of both local governments and community‐based organisations can be considerably enhanced when these agencies work in partnership with one another. Different roles will be played by local governments and community organisations in different types of partnership arrangements. Distinguishing among these roles helps allocate responsibilities better among the partner agencies, and it is also helpful for scheduling implementation, devising appropriate capacity building programmes and designing suitable accountability mechanisms. An analytical framework to help with these tasks is developed and presented in this article. Copyright © 2003 John Wiley & Sons, Ltd.     

An effectiveness‐based evaluation of five state pre‐kindergarten programs

Since 1980, the number of state pre‐kindergarten (pre‐K) programs has more than doubled, with 38 states enrolling more than one million children in 2006 alone. This study evaluates how five state pre‐K programs affected children's receptive vocabulary, math, and print awareness skills. Taking advantage of states' strict enrollment policies determined by a child's date of birth, a regression‐discontinuity design was used to estimate effects in Michigan, New Jersey, Oklahoma, South Carolina, and West Virginia. For receptive vocabulary, only New Jersey and Oklahoma yielded significant standardized impacts, though two of the three other coefficients were in a direction indicating positive effects. For math, all the coefficients were positive but only Michigan and New Jersey yielded reliable results. The largest impacts were for print awareness, where all five coefficients were positive and four were reliable in Michigan, New Jersey, South Carolina, and West Virginia. The five states were not randomly selected and, on average, have higher quality program standards than non‐studied states, precluding formal extrapolation to the nation at large. However, our sample of states differed in many other ways, permitting the conclusion that state pre‐K programs can have positive effects on children's cognitive skills, though the magnitude of these effects varies by state and outcome. © 2008 by the Association for Public Policy Analysis and Management.     

Monitoring across Sectors: Examining the Effect of Nonprofit and For‐Profit Contractor Ownership on Performance Monitoring in State and Local Contracts

What is the effect of contractors' nonprofit and for‐profit ownership on the scope and nature of performance measurement used by government agencies? Quantitative and qualitative data were generated through semistructured interviews administered to a sample of state and local public agencies and private contractors across five jurisdictions. The findings of this study suggest that monitoring officers working with nonprofit rather than for‐profit contractors are more likely to rely on qualitative performance data and examine equitable access to services, contractors' reputation, and compliance with industry rules and regulations. Although organizational ownership may not be well understood by practitioners, performance appears to be conceptualized differently across sectors. The author calls for a better understanding of the impact of the identified differences in performance measurement on the effectiveness of contract monitoring.