The Use and Efficacy of Capacity‐Building Assistance for Low‐Performing Districts: The Case of California's District Assistance and Intervention Teams

The theory of action upon which high‐stakes accountability policies are based calls for systemic reforms in educational systems that will emerge by pairing incentives for improvement with extensive and targeted technical assistance (TA) to build the capacity of low‐performing schools and districts. To this end, a little discussed and often overlooked aspect of the No Child Left Behind Act (NCLB) mandated that, in addition to sanctions, states were required to provide TA to build the capacity of struggling schools and Local Education Agencies (LEAs, or districts) to help them improve student achievement. Although every state in the country provides some form of TA to its lowest performing districts, we know little about the content of these programs or about their efficacy in improving student performance. In this paper, we use both quantitative and qualitative analyses to explore the actions taken by TA providers in one state—California—and examine whether the TA and support tied to California's NCLB sanctions succeeds in improving student achievement. Like many other states, California requires that districts labeled as persistently failing under NCLB (in Program Improvement year 3, PI3) work with external experts to help them build the capacity to make reforms that will improve student achievement. California's lowest performing PI3 districts are given substantial amounts of funding and are required to contract with state‐approved District Assistance and Intervention Teams (DAITs), whereas the remaining PI3 districts receive less funding and are asked to access less intensive TA from non‐DAIT providers. We use a five‐year panel difference‐in‐difference design to estimate the impacts of DAITs on student performance on the math and English language arts (ELA) standardized tests relative to non‐DAIT TA during the two years of the program intervention. We find that students in districts with DAITs perform significantly better on math California Standards Tests (CSTs) averaged over both treatment years and in each of the first and second years. We do not find evidence that students in districts with DAITs perform higher on ELA CSTs over the combined two years of treatment, although we find suggestive evidence that ELA performance increases in the second year of treatment relative to students in districts with non‐DAIT TA. Ordinary least squares (OLS) regressions that explore the association between specific activities fostered by DAITs and changes in districts’ gains in achievement over the two years of treatment show that DAIT districts that report increasing their focus on using data to guide instruction, shifting district culture to generate and maintain high expectations of students and staff, and increasing within‐district accountability for student performance, have higher math achievement gains over the course of the DAIT treatment. In addition, DAIT districts that increase their focus on ELA instruction and shift district culture to one of high expectations have higher ELA achievement gains than do DAIT districts that do not have a similar focus. © 2012 by the Association for Public Policy Analysis and Management.     

Status versus growth: The distributional effects of school accountability policies

Although the federal No Child Left Behind program judges the effectiveness of schools based on their students' achievement status, many policy analysts argue that schools should be measured, instead, by their students' achievement growth. Using a 10‐year student‐level panel data set from North Carolina, we examine how school‐specific pressure associated with status and growth approaches to school accountability affect student achievement at different points in the prior‐year achievement distribution. Achievement gains for students below the proficiency cut point emerge in schools failing either type of accountability standard, with the effects clearer for math than for reading. In contrast to prior research highlighting the possibility of educational triage, we find little or no evidence that failing schools in North Carolina ignore the students far below proficiency under either approach. Importantly, we find that the status, but not the growth, approach reduces the reading achievement of higher performing students. Our analysis suggests that the distributional effects of accountability pressure depend not only on the type of pressure for which schools are held accountable (status or growth), but also the tested subject. © 2010 by the Association for Public Policy Analysis and Management.     

Does school accountability lead to improved student performance?

The leading school reform policy in the United States revolves around strong accountability of schools with consequences for performance. The federal government's involvement through the No Child Left Behind Act of 2001 reinforces the prior movement of many states toward policies based on measured student achievement. Analysis of state achievement growth as measured by the National Assessment of Educational progress shows that accountability systems introduced during the 1990s had a clear positive impact on student achievement. This single policy instrument did not, however, also lead to any narrowing in the Black‐White achievement gap (though it did narrow the Hispanic‐White achievement gap). Moreover, the Black‐White gap appears to have been adversely impacted over the decade by increasing minority concentrations in the schools. An additional issue surrounding stronger accountability has been a concern about unintended outcomes related to such things as higher exclusion rates from testing, increased dropout rates, and the like. Our analysis of special education placement rates, a frequently identified area of concern, does not show any responsiveness to the introduction of accountability systems.© 2005 by the Association for Public Policy Analysis and Management     

Equity and Accountability: The Impact of State Accountability Systems on School Finance

Using an 11‐year panel data set containing information on revenues, expenditures, and demographics for every school district in the United States, we examine the effects of state‐adopted school accountability systems on the adequacy and equity of school resources. We find little relationship between state implementation of accountability systems and changes in school finance equity, though we do find evidence that states in which courts overturned the school finance system during the decade exhibited significant equity improvements. Additionally, while implementation of accountability per se does not appear linked to changes in resource adequacy, states that implemented strong accountability systems did experience improvements.     

Do School Report Cards Produce Accountability Through the Ballot Box?

Public education has been transformed by the widespread adoption of accountability systems that involve the dissemination of school district performance information. Using data from Ohio, we examine if elections serve as one channel through which these accountability systems might lead to improvements in educational quality. We find little evidence that poor performance on widely disseminated state and federal indicators has an impact on school board turnover, the vote share of sitting school board members, or superintendent tenure, suggesting that the dissemination of district performance information puts little (if any) electoral pressure on elected officials to improve student achievement.     

Performance Standards and Employee Effort: Evidence From Teacher Absences

The 2001 No Child Left Behind Act (NCLB) increased accountability pressure in U.S. public schools by threatening to impose sanctions on Title‐1 schools that failed to make adequate yearly progress (AYP) in consecutive years. Difference‐in‐difference estimates of the effect of failing AYP in the first year of NCLB on teacher effort in the subsequent year suggest that on average, teacher absences in North Carolina fell by about 10 percent. The probability of being frequently absent similarly decreased. These reductions in teacher absences were driven by within‐teacher increases in effort and by teachers in the bottom half of the effectiveness distribution. On average, only a modest amount of the achievement gains attributable to the increased accountability pressure are explained by the corresponding decline in teacher absences.     

The Effects of Tulsa's Pre‐K Program on Middle School Student Performance

As states have upgraded their commitment to pre‐K education over the past two decades, questions have arisen. Critics argue that program effects are likely to fade out or disappear over time, while supporters contend that program effects are likely to persist under certain conditions. Using data from Tulsa Public Schools, three neighboring school districts, and the state of Oklahoma, and propensity score weighting, we estimate the effects of Tulsa's universal, school‐based pre‐K program on multiple measures of academic progress for middle school students. We find enduring effects on math achievement test scores, enrollment in honors courses, and grade retention for students as a whole, and similar effects for certain subgroups. We conclude that some positive effects of a high‐quality pre‐K program are discernible as late as middle school.     

Improving the Implementation and Effectiveness of Out‐of‐School‐Time Tutoring

School districts are spending millions on tutoring outside regular school day hours for economically and academically disadvantaged students in need of extra academic assistance. Under No Child Left Behind (NCLB), parents of children in persistently low‐performing schools were allowed to choose their child's tutoring provider, and together with school districts, they were also primarily responsible for holding providers in the private market accountable for performance. We present results from a multisite, mixed‐method longitudinal study of the impact of out‐of‐school time (OST) tutoring on student reading and mathematics achievement that link provider attributes and policy and program administration variables to tutoring program effectiveness. We find that many students are not getting enough hours of high‐quality, differentiated instruction to produce significant gains in their learning, in part because of high hourly rates charged by providers for tutoring. We identify strategies and policy levers that school districts can use to improve OST tutoring policy design and launch improved programs as waivers from NCLB are granted.     

The Impact of Providing Vision Screening and Free Eyeglasses on Academic Outcomes: Evidence from a Randomized Trial in Title I Elementary Schools in Florida

More than 20 percent of all school‐aged children in the United States have vision problems, and low‐income and minority children are disproportionately likely to have unmet vision care needs. Vision screening is common in U.S. schools, but it remains an open question whether screening alone is sufficient to improve student outcomes. We implemented a multi‐armed randomized controlled trial (RCT) to evaluate the impact of vision screening, and of vision screening accompanied by eye exams and eyeglasses, provided by a non‐profit organization to Title I elementary schools in three large central Florida school districts. We find that providing additional/enhanced screening alone is generally insufficient to improve student achievement in math and reading. In contrast, providing screening along with free eye exams and free eyeglasses to students with vision problems improved student achievement as measured by standardized test scores. We find, averaging over all students (including those without vision problems), that this more comprehensive intervention increased the probability of passing the Florida Comprehensive Achievement Tests (FCATs) in reading and math by approximately 2.0 percentage points. We also present evidence that indicates that this impact fades out over time, indicating that follow‐up actions after the intervention may be necessary to sustain these estimated achievement gains.     

Teacher Incentives and Student Achievement: Evidence from an Advancement Program

This paper examines the effects of a comprehensive performance pay program implemented in high‐need schools using administrative data from Louisiana. Exploiting the within‐student variation in the timing of implementation, we find a large and significant effect beginning with the second year of the program on math achievement. Similar but generally insignificant point estimates are observed in social studies. As for English Language Arts and science, there are no effects of the program. We provide evidence against changes in the composition of teacher workforce and postadoption student sorting as potential explanations for our results. We then show aggregate findings from a web‐survey that teachers may have altered their teaching practices following performance‐based compensation. Finally, under certain assumptions, we calculate the net present value of earnings impact to be close to a million dollars per school‐year for math.     

Curriculum Reform in The Common Core Era: Evaluating Elementary Math Textbooks Across Six U.S. States

Can a school or district improve student achievement simply by switching to a higher-quality textbook or curriculum? We conducted the first multi-textbook, multi-state effort to estimate textbook efficacy following widespread adoption of the Common Core State Standards (CCSS) and associated changes in the textbook market. Pooling textbook adoption and student test score data across six geographically and demographically diverse U.S. states, we found little evidence of differences in average achievement gains for schools using different math textbooks. We found some evidence of greater variation in achievement gains among schools using pre-CCSS editions, which may have been more varied in their content than post-CCSS editions because they were written for a broader set of standards. We also found greater variation among schools that had more exposure to a given text. However, these differences were small. Despite considerable interest and attention to textbooks as a low-cost, “silver bullet” intervention for improving student outcomes, we conclude that the adoption of a new textbook or set of curriculum materials, on its own, is unlikely to achieve this goal.     

Dual-Credit Courses and the Road to College: Experimental Evidence from Tennessee

Dual-credit courses expose high school students to college-level content and provide the opportunity to earn college credits, in part to smooth the transition to college. With the Tennessee Department of Education, we conduct the first randomized controlled trial of the effects of dual-credit math coursework on a range of high school and college outcomes. We find that the dual-credit advanced algebra course alters students’ subsequent high school math course-taking, reducing enrollment in remedial math and boosting enrollment in precalculus and Advanced Placement math courses. We fail to detect an effect of the dual-credit math course on overall rates of college enrollment. However, the course induces some students to choose four-year universities instead of two-year colleges, particularly for those in the middle of the math achievement distribution and those first exposed to the opportunity to take the course in eleventh rather than twelfth grade. We see limited evidence of improvements in early math performance during college.     

Accountability,Test Prep Incentives,and the Design of Math and English Exams

We examine how incentives for test prep varied between math and English language arts (ELA) on U.S. state accountability exams. We collected data on exam structure for grade 3 to 8 tests in six states that are the setting for most U.S. research in literatures where accountability matters. We show that math exams typically measured ability more precisely for students on the margin of achieving proficiency. This gave educators an incentive to spend more time preparing students for math tests than for ELA tests, consistent with the common finding of larger math effects in the literature.     

Sinking Swann: Public School Choice and the Resegregation of Charlotte's Public Schools

Public school choice is a widely used tool for education reform and may be a way to improve school accountability and efficiency. This article examines what happened to student outcomes when Charlotte‐Mecklenburg Schools, a large and diverse urban school district located in North Carolina, changed its assignment policy to one of open enrollment with mandatory choice. The previous policy used a broad array of magnet schools and a limited amount of mandatory busing to achieve desegregation. The new policy required that all students choose a school, and it specifically avoided using race or ethnicity considerations in assigning students. The article examines the impacts of the new policy on the end‐of‐grade standardized tests in reading and math. The article uses regression analysis to discover whether the scores of various groups of students increased or decreased after the policy change. The analysis suggests that the “race‐neutral” assignment policy was neither neutral in the opportunity it provided students to attend their school of choice nor in its academic outcomes. Anglo students were more likely to receive their first choice of schools and to improve their scores. African American students were less likely to receive their first choice school and their scores declined.     

School Vouchers and Student Outcomes: Experimental Evidence from Washington,DC

School vouchers are the most contentious form of parental school choice. Vouchers provide government funds that parents can use to send their children to private schools of their choice. Here we examine the empirical question of whether or not a school voucher program in Washington, DC, affected achievement or the rate of high school graduation for participating students. The District of Columbia Opportunity Scholarship Program (OSP) has operated in the nation's capital since 2004, funded by a federal government appropriation. Because the program was oversubscribed in its early years of operation, and vouchers were awarded by lottery, we were able to use the “gold standard” evaluation method of a randomized experiment to determine what impacts the OSP had on student outcomes. Our analysis revealed compelling evidence that the DC voucher program had a positive impact on high school graduation rates, suggestive evidence that the program increased reading achievement, and no evidence that it affected math achievement. We discuss the implications of these findings in light of recent policy developments including the reauthorization of the OSP and the enactment or expansion of more than a dozen school voucher or voucher‐type programs throughout the United States in 2011 and 2012.     

High School Exit Exams and Dropout in an Era of Increased Accountability

A key form of student‐level accountability is the requirement for students to pass high school exit exams (HSEEs) in order to receive a diploma. In this paper, we examine the impact of HSEEs on dropout during a period when these exams became more common and rigorous. Further, we study whether offering alternate pathways to graduation for students who cannot pass HSEEs moderates any dropout effects. Using a district‐grade‐level panel assembled from the Common Core of Data, we exploit the fact that new exit exam policies first affect a particular graduating class, so we can isolate the impact of exposure to HSEEs for students in one grade in a district separate from other unaffected grades in the same district. We estimate dropout effects by grade for all students, and by race, sex, and urbanicity. We find that HSEEs increase dropout rates for students in the 12th grade, with especially large effects for African‐American students. Dropout effects are uniformly larger in states that do not provide alternate pathways to receive a diploma or alternative credentials for students that cannot pass exit exams. We estimate that 1.25 percent of 12th graders in these states do not graduate with their high school class, likely due to having a diploma withheld because of inability to pass the requisite HSEE. © 2013 by the Association for Public Policy Analysis and Management.     

No Child Left Behind: flowers don’t grow in the desert

The No Child Left Behind legislation purports to effectively eliminate the long standing “achievement gap” between poor and minority students and their white [sic.] peers. We employ a multi-method approach to investigate (1) the discursive dominance and construction of NCLB, (2) the quantitative validity of the law's implicit causal model of educational achievement and reform, and (3) the experiences of teachers forced to negotiate the demands of NCLB in “failing” schools. Using data drawn from federal and state policy documents, U.S. Census, the State of Connecticut Department of Education, and interviews with teachers from urban schools, we find that: (1) Through the advocacy of state regulated systems of accountability and the imposition of “scientifically proven” pedagogical methods, NCLB constructs a model that removes the effects of structural inequalities from consideration. (2) Quantitative analysis of data drawn from Connecticut Academic Performance Test (CAPT) displays the inadequacy of this model. (3) Interviews with urban teachers further validate the inadequacy of this model and the importance of social structural variables in understanding and/or addressing the “achievement gap.”     

Competition in the sandbox: A test of the effects of preschool competition on educational outcomes

The emergence of publicly subsidized preschool raises important policy questions about the role of market forces and, in places where competition to provide these services exists, presents a setting in which the effects of competition on educational outcomes can be tested. We test neo‐institutional hypotheses concerning the effects of competition to provide publicly‐funded prekindergarten (pre‐k) services on the performance of public schools and private organizations. We use student‐level data collected over a five‐year period on a large sample of children who attended publicly subsidized prekindergarten in Georgia. Overall, we find that more competition improves third grade reading and math test scores but does not significantly affect retention or school readiness ratings during elementary school. Not all children are equally affected by competition; for example, greater competition significantly decreases the likelihood of retention for children of the working poor. Contrary to the expectations of some theorists, both public schools and private organizations respond to increased competition in ways that improve test scores but not retention. However, children attending private prekindergarten have higher language arts scores and lower retention across the range of competition when compared with children who attended public school pre‐k. © 2006 by the Association for Public Policy Analysis and Management     

Supplying Disadvantaged Schools with Effective Teachers: Experimental Evidence on Secondary Math Teachers from Teach For America

Teach For America (TFA) is an important but controversial source of teachers for hard‐to‐staff subjects in high‐poverty U.S. schools. We present findings from the first large‐scale experimental study of secondary math teachers from TFA. We find that TFA teachers are more effective than other math teachers in the same schools, increasing student math achievement by 0.07 standard deviations over one school year. Addressing concerns about the fact that TFA requires only a two‐year commitment, we find that TFA teachers in their first two years of teaching are more effective than more experienced non‐TFA teachers in the same schools.