The Use and Efficacy of Capacity‐Building Assistance for Low‐Performing Districts: The Case of California's District Assistance and Intervention Teams

The theory of action upon which high‐stakes accountability policies are based calls for systemic reforms in educational systems that will emerge by pairing incentives for improvement with extensive and targeted technical assistance (TA) to build the capacity of low‐performing schools and districts. To this end, a little discussed and often overlooked aspect of the No Child Left Behind Act (NCLB) mandated that, in addition to sanctions, states were required to provide TA to build the capacity of struggling schools and Local Education Agencies (LEAs, or districts) to help them improve student achievement. Although every state in the country provides some form of TA to its lowest performing districts, we know little about the content of these programs or about their efficacy in improving student performance. In this paper, we use both quantitative and qualitative analyses to explore the actions taken by TA providers in one state—California—and examine whether the TA and support tied to California's NCLB sanctions succeeds in improving student achievement. Like many other states, California requires that districts labeled as persistently failing under NCLB (in Program Improvement year 3, PI3) work with external experts to help them build the capacity to make reforms that will improve student achievement. California's lowest performing PI3 districts are given substantial amounts of funding and are required to contract with state‐approved District Assistance and Intervention Teams (DAITs), whereas the remaining PI3 districts receive less funding and are asked to access less intensive TA from non‐DAIT providers. We use a five‐year panel difference‐in‐difference design to estimate the impacts of DAITs on student performance on the math and English language arts (ELA) standardized tests relative to non‐DAIT TA during the two years of the program intervention. We find that students in districts with DAITs perform significantly better on math California Standards Tests (CSTs) averaged over both treatment years and in each of the first and second years. We do not find evidence that students in districts with DAITs perform higher on ELA CSTs over the combined two years of treatment, although we find suggestive evidence that ELA performance increases in the second year of treatment relative to students in districts with non‐DAIT TA. Ordinary least squares (OLS) regressions that explore the association between specific activities fostered by DAITs and changes in districts’ gains in achievement over the two years of treatment show that DAIT districts that report increasing their focus on using data to guide instruction, shifting district culture to generate and maintain high expectations of students and staff, and increasing within‐district accountability for student performance, have higher math achievement gains over the course of the DAIT treatment. In addition, DAIT districts that increase their focus on ELA instruction and shift district culture to one of high expectations have higher ELA achievement gains than do DAIT districts that do not have a similar focus. © 2012 by the Association for Public Policy Analysis and Management.     

Supplying Disadvantaged Schools with Effective Teachers: Experimental Evidence on Secondary Math Teachers from Teach For America

Teach For America (TFA) is an important but controversial source of teachers for hard‐to‐staff subjects in high‐poverty U.S. schools. We present findings from the first large‐scale experimental study of secondary math teachers from TFA. We find that TFA teachers are more effective than other math teachers in the same schools, increasing student math achievement by 0.07 standard deviations over one school year. Addressing concerns about the fact that TFA requires only a two‐year commitment, we find that TFA teachers in their first two years of teaching are more effective than more experienced non‐TFA teachers in the same schools.     

Status versus growth: The distributional effects of school accountability policies

Although the federal No Child Left Behind program judges the effectiveness of schools based on their students' achievement status, many policy analysts argue that schools should be measured, instead, by their students' achievement growth. Using a 10‐year student‐level panel data set from North Carolina, we examine how school‐specific pressure associated with status and growth approaches to school accountability affect student achievement at different points in the prior‐year achievement distribution. Achievement gains for students below the proficiency cut point emerge in schools failing either type of accountability standard, with the effects clearer for math than for reading. In contrast to prior research highlighting the possibility of educational triage, we find little or no evidence that failing schools in North Carolina ignore the students far below proficiency under either approach. Importantly, we find that the status, but not the growth, approach reduces the reading achievement of higher performing students. Our analysis suggests that the distributional effects of accountability pressure depend not only on the type of pressure for which schools are held accountable (status or growth), but also the tested subject. © 2010 by the Association for Public Policy Analysis and Management.     

Impact of the Indiana Choice Scholarship Program: Achievement Effects for Students in Upper Elementary and Middle School

This paper examines the impact of the Indiana Choice Scholarship Program on student achievement for low‐income students in upper elementary and middle school who used a voucher to transfer from public to private schools during the first four years of the program. We analyzed student‐level longitudinal data from public and private schools taking the same statewide standardized assessment. Overall, voucher students experienced an average achievement loss of 0.15 SDs in mathematics during their first year of attending a private school compared with matched students who remained in a public school. This loss persisted regardless of the length of time spent in a private school. In English/Language Arts, we did not observe statistically meaningful effects. Although school vouchers aim to provide greater educational opportunities for students, the goal of improving the academic performance of low‐income students who use a voucher to move to a private school has not yet been realized in Indiana.     

The Effects of Tulsa's Pre‐K Program on Middle School Student Performance

As states have upgraded their commitment to pre‐K education over the past two decades, questions have arisen. Critics argue that program effects are likely to fade out or disappear over time, while supporters contend that program effects are likely to persist under certain conditions. Using data from Tulsa Public Schools, three neighboring school districts, and the state of Oklahoma, and propensity score weighting, we estimate the effects of Tulsa's universal, school‐based pre‐K program on multiple measures of academic progress for middle school students. We find enduring effects on math achievement test scores, enrollment in honors courses, and grade retention for students as a whole, and similar effects for certain subgroups. We conclude that some positive effects of a high‐quality pre‐K program are discernible as late as middle school.     

Competition in the sandbox: A test of the effects of preschool competition on educational outcomes

The emergence of publicly subsidized preschool raises important policy questions about the role of market forces and, in places where competition to provide these services exists, presents a setting in which the effects of competition on educational outcomes can be tested. We test neo‐institutional hypotheses concerning the effects of competition to provide publicly‐funded prekindergarten (pre‐k) services on the performance of public schools and private organizations. We use student‐level data collected over a five‐year period on a large sample of children who attended publicly subsidized prekindergarten in Georgia. Overall, we find that more competition improves third grade reading and math test scores but does not significantly affect retention or school readiness ratings during elementary school. Not all children are equally affected by competition; for example, greater competition significantly decreases the likelihood of retention for children of the working poor. Contrary to the expectations of some theorists, both public schools and private organizations respond to increased competition in ways that improve test scores but not retention. However, children attending private prekindergarten have higher language arts scores and lower retention across the range of competition when compared with children who attended public school pre‐k. © 2006 by the Association for Public Policy Analysis and Management     

The Effect of Providing Breakfast in Class on Student Performance

Many schools have recently experimented with moving breakfast from the cafeteria to the classroom. We examine whether such a program increases achievement, grades, and attendance rates. We exploit quasi‐random timing of program implementation that allows for a difference‐in‐differences identification strategy. We find that providing breakfast in class relative to the cafeteria raises math and reading achievement by 0.09 and 0.06 standard deviations, respectively. These effects are most pronounced for low‐performing, free lunch‐eligible, Hispanic, and low body mass index students. A lack of differences by exposure time and effects on grades suggest that these impacts are on test‐taking performance rather than learning. At the same time, the results highlight the possibility that measured achievement may be biased downwards, and accountability penalties may be inappropriately applied, in schools where many students do not consume breakfast.     

The Impact of Incentives to Recruit and Retain Teachers in “Hard‐to‐Staff” Subjects

We investigate the effects of a statewide program designed to increase the supply of teachers in designated “hard‐to‐staff” areas, such as special education, math, and science. Employing a difference‐in‐difference estimator we find that the loan forgiveness component of the program was effective, reducing mean attrition rates for middle and high school math and science teachers by 10.4 percent and 8.9 percent, respectively. We also find that the impact of loan forgiveness varied with the generosity of payments; when fully funded, the program reduced attrition of special education teachers by 12.3 percent, but did not have a statistically significant impact when funding was substantially reduced. A triple‐difference estimate indicates that a one‐time bonus program also had large effects, reducing the likelihood of teachers’ exit by as much as 32 percent in the short run. A back‐of‐the‐envelope cost‐benefit analysis suggests that both the loan forgiveness and the bonus program were cost effective.     

THE IMPACTS OF DIFFERENT TYPES OF COLLEGE INSTRUCTORS ON STUDENTS’ ACADEMIC AND LABOR MARKET OUTCOMES

Based on a novel data set that links college administrative information with earnings records from a state college system for both public two‐year and four‐year colleges, this study quantifies the impacts of exposure to different types of instructors during students’ initial semester in college on their subsequent academic and labor market outcomes. To minimize bias from student sorting by type of instructor, we combine course‐set fixed effects with an instrumental variables approach that exploits term‐by‐term fluctuations in faculty composition in each department, therefore controlling for both between‐ and within‐course sorting. The findings suggest that two‐year students, particularly racial minority students, have substantially higher levels of exposure to adjuncts with temporary appointments than four‐year students. Two‐year students taking a heavy course schedule with temporary adjuncts are adversely affected in college persistence and subsequent credit accumulation, and the penalty is particularly pronounced among males and racial minority students with stronger academic potential. Such negative impacts on academic outcomes do not translate into poorer short‐ to medium‐term labor market performance. In the four‐year setting, no significant distinction is identified between different types of instructors on either student academic or labor market outcomes.     

The impact of no Child Left Behind on student achievement

The No Child Left Behind (NCLB) Act compelled states to design school accountability systems based on annual student assessments. The effect of this federal legislation on the distribution of student achievement is a highly controversial but centrally important question. This study presents evidence on whether NCLB has influenced student achievement based on an analysis of state‐level panel data on student test scores from the National Assessment of Educational Progress (NAEP). The impact of NCLB is identified using a comparative interrupted time series analysis that relies on comparisons of the test‐score changes across states that already had school accountability policies in place prior to NCLB and those that did not. Our results indicate that NCLB generated statistically significant increases in the average math performance of fourth graders (effect size 5 0.23 by 2007) as well as improvements at the lower and top percentiles. There is also evidence of improvements in eighth‐grade math achievement, particularly among traditionally low‐achieving groups and at the lower percentiles. However, we find no evidence that NCLB increased fourth‐grade reading achievement. © 2011 by the Association for Public Policy Analysis and Management.     

From Birth to School: Early Childhood Initiatives and Third‐Grade Outcomes in North Carolina

This study examines the community‐wide effects of two statewide early childhood policy initiatives in North Carolina. One initiative provides funding to improve the quality of child care services at the county level for all children between the ages of 0 to 5, and the other provides funding for preschool slots for disadvantaged four‐year‐olds. Differences across counties in the timing of the rollout and in the magnitude of the state financial investments per child provide the variation in programs needed to estimate their effects on schooling outcomes in third grade. We find robust positive effects of each program on third‐grade test scores in both reading and math. These effects can best be explained by a combination of direct benefits for participants and spillover benefits for others. Our preferred models suggest that the combined average effects on test scores of investments in both programs at 2009 funding levels are equivalent to two to four months of instruction in grade 3.     

College Guidance for All: A Randomized Experiment in Pre‐College Advising

Pre‐college advising programs exist in most disadvantaged high schools throughout the United States. These programs supplement traditional advising by high school guidance counselors and attempt to help underrepresented and disadvantaged students overcome the complexities of the postsecondary admission and financial aid processes. Existing evidence on these programs often uses within‐school randomization where spillovers and alternative supports may confound estimates. We provide the first evidence on a whole school intervention resulting from a school‐level randomized controlled trial in the United States. The college access program we study uses a near‐peer model where a recent college graduate works at the school assisting students in the application and enrollment process. Pooled results across the first three years of program implementation find no significant impacts on overall college enrollment. However, subgroup analyses reveal positive, significant effects among the groups most targeted by the intervention: Hispanic and low‐income students. Most of the impact comes through increasing two‐year college enrollment, but this appears to be new entrants rather than inducing students to move from four‐year to two‐year colleges. The observed positive effects for these subgroups attenuate over time. We attribute this drop in the estimated impact to departures in fidelity of the experiment. Even among the cohorts for which we find positive enrollment impacts, we find no significant impacts on college persistence.     

Cream‐Skimming,Parking and Other Intended and Unintended Effects of High‐Powered,Performance‐Based Contracts

As performance‐based contracting in social welfare services continues to expand, concerns about potential unintended effects are also growing. We analyze the incentive effects of high‐powered, performance‐based contracts and their implications for program outcomes using panel data on Dutch cohorts of unemployed and disabled workers that were assigned to private social welfare providers in 2002 to 2005. We employ a difference‐in‐differences design that takes advantage of the fact that contracts gradually moved from partial performance‐contingent pay to full (100 percent) performance‐contingent contracting schemes. We develop explicit measures of selection into the programs and find evidence of cream skimming and other gaming activities on the part of providers, but little impact of these activities on program outcomes. Moving to a system with contract payments fully contingent on performance appears to increase job placements, but not job duration, for more readily employable workers.     

Improving the Implementation and Effectiveness of Out‐of‐School‐Time Tutoring

School districts are spending millions on tutoring outside regular school day hours for economically and academically disadvantaged students in need of extra academic assistance. Under No Child Left Behind (NCLB), parents of children in persistently low‐performing schools were allowed to choose their child's tutoring provider, and together with school districts, they were also primarily responsible for holding providers in the private market accountable for performance. We present results from a multisite, mixed‐method longitudinal study of the impact of out‐of‐school time (OST) tutoring on student reading and mathematics achievement that link provider attributes and policy and program administration variables to tutoring program effectiveness. We find that many students are not getting enough hours of high‐quality, differentiated instruction to produce significant gains in their learning, in part because of high hourly rates charged by providers for tutoring. We identify strategies and policy levers that school districts can use to improve OST tutoring policy design and launch improved programs as waivers from NCLB are granted.     

Do Job Corps performance measures track program impacts?

Since the 1993 Government Performance and Results Act, performance measurement systems based on short‐term program outcomes have been increasingly used to assess the effectiveness of federal programs. This paper examines the association between program performance measures and long‐term program impacts, using nine‐year follow‐up data from a recent large‐scale, national experimental evaluation of Job Corps, the nation's largest federal job training program for disadvantaged youths. Job Corps is an important test case because it uses a comprehensive performance system that is widely emulated. We find that impacts on key outcomes are not associated with measured center performance levels. Participants in higherperforming centers had better outcomes; however, the same pattern holds for comparable controls. Thus, the performance measurement system is not achieving the goal of ranking and rewarding centers on the basis of their ability to improve participant outcomes relative to what these outcomes would have been otherwise. © 2008 by the Association for Public Policy Analysis and Management.     

Let Them Eat Lunch: The Impact of Universal Free Meals on Student Performance

This paper investigates the impact of extending free school lunch to all students, regardless of income, on academic performance in New York City middle schools. Using a difference-in-differences design and unique longitudinal, student-level data, we derive credibly causal estimates of the impacts of “Universal Free Meals” (UFM) on test scores in English Language Arts (ELA) and mathematics, and participation in school lunch. We find UFM increases academic performance by as much as 0.083 standard deviations in math and 0.059 in ELA for non-poor students, with smaller, statistically significant effects of 0.032 and 0.027 standard deviations in math and ELA for poor students. Further, UFM increases participation in school lunch by roughly 11.0 percentage points for non-poor students and 5.4 percentage points for poor students. We then investigate the academic effects of school lunch participation per se, using UFM as an instrumental variable. Results indicate that increases in school lunch participation improve academic performance for both poor and non-poor students; an additional lunch every two weeks increases test scores by roughly 0.08 standard deviations in math and 0.07 standard deviations in ELA. Finally, we explore potential unintended consequences for student weight outcomes, finding no evidence that UFM increases the probability that students are obese or overweight. We also find no evidence of increases in average body mass index (BMI). Instead, we find some evidence that participation in school lunch improves weight outcomes for non-poor students. Results are robust to an array of alternative specifications and assumptions about the sample.     

School Vouchers and Student Outcomes: Experimental Evidence from Washington,DC

School vouchers are the most contentious form of parental school choice. Vouchers provide government funds that parents can use to send their children to private schools of their choice. Here we examine the empirical question of whether or not a school voucher program in Washington, DC, affected achievement or the rate of high school graduation for participating students. The District of Columbia Opportunity Scholarship Program (OSP) has operated in the nation's capital since 2004, funded by a federal government appropriation. Because the program was oversubscribed in its early years of operation, and vouchers were awarded by lottery, we were able to use the “gold standard” evaluation method of a randomized experiment to determine what impacts the OSP had on student outcomes. Our analysis revealed compelling evidence that the DC voucher program had a positive impact on high school graduation rates, suggestive evidence that the program increased reading achievement, and no evidence that it affected math achievement. We discuss the implications of these findings in light of recent policy developments including the reauthorization of the OSP and the enactment or expansion of more than a dozen school voucher or voucher‐type programs throughout the United States in 2011 and 2012.     

Strategic Involuntary Teacher Transfers and Teacher Performance: Examining Equity and Efficiency

Despite claims that school districts need flexibility in teacher assignment to allocate teachers more equitably across schools and improve district performance, the power to involuntarily transfer teachers across schools remains hotly contested. Little research has examined involuntary teacher transfer policies or their effects on schools, teachers, or students. This article uses administrative data from Miami‐Dade County Public Schools to investigate the implementation and effects of the district's involuntary transfer policy, including which schools transferred and received teachers, which teachers were transferred, what kinds of teachers replaced them in their former schools, and how their performance—as measured by their work absences and value‐added in math and reading—compared before and after the transfer. We find that, under the policy, principals in the lowest performing schools identified relatively low‐performing teachers for transfer who, based on observable characteristics, would have been unlikely to leave on their own. Consistent with an equity improvement, we find that involuntarily transferred teachers were systematically moved to higher performing schools and generally were outperformed by the teachers who replaced them. Efficiency impacts are mixed; although transferred teachers had nearly two fewer absences per year in their new positions, transferred teachers continued to have low value‐added in their new schools.     

The Productivity Costs of Inefficient Hiring Practices: Evidence From Late Teacher Hiring

We use matched employee–employer records from the teacher labor market to explore the effects of late teacher hiring on student achievement. Hiring teachers after the school year starts reduces student achievement by 0.042 SD in mathematics and 0.026 SD in reading. This reflects, in part, a temporary disruption effect in the first year. In mathematics, but not in reading, late‐hired teachers remain persistently less effective, evidence of negative selection in the teacher labor market. Late hiring concentrates in schools that disproportionately serve disadvantaged student populations, contributing to challenges in ensuring an equitable distribution of educational resources for all students.     

Financial aid and for‐profit colleges: Does aid encourage entry?

Concerns over rising college tuition and slow economic growth have brought renewed attention to the role of federal and state financial aid programs in opening access to education. Despite a large body of literature examining the effects of grant aid on four‐year and public two‐year college enrollment, for‐profit colleges—particularly the vast majority that offer two‐year degrees and certificates—have largely been ignored. Using panel data methods and a new administrative data set of for‐profit colleges operating in California between 1989 and 2003, I assess the impact of the federal Pell Grant program, the G.I. Bill, and California's Cal Grant program on the net number of for‐profit colleges per county. The results suggest that for both Pell and Cal Grants, increases in the per‐student maximum award encourage for‐profit entry. This relationship is particularly strong in counties with high adult poverty levels, where more students are eligible for aid. Further, these gains in the private sector do not appear to come at the expense of the public sector. Rather, public community colleges also experience enrollment gains as the generosity of Pell and Cal Grants increases, although this reaction appears to be weaker than the reaction of for‐profits. © 2010 by the Association for Public Policy Analysis and Management.