Successful Program Development Using Implementation Evaluation

SUMMARY

Four methods used to enhance implementation of a resiliency-based early childhood substance abuse and violence prevention initiative are described. Project staff drew up formal agreements with participating programs to ensure administrative support for the intervention, provided high quality training for program implemcnt-ers, used a participatory evaluation approach, and monitored implementation both directly and through practitioners' self-report. The contribution of each of these methods to fidelity of implementation is discussed, and implementation challenges are identified. The growth and success of the project demonstrates the utility of implementation evaluation data for ongoing program development and improvement.     

Project LINK

Abstract

This article describes the outcomes of Project LINK, a Safe Schools/Healthy Students grantee in Larimer County, Colorado. The study analyzed the influence of policies and administrative support on the implementation of programs as well as on risk and protective factors predictive of violent behaviors and drug and alcohol use. Results showed increased feelings of safety in comparison to the rest of the state and changes in risk and protective factors by school level. Results also indicate that leadership support and enforcement of policies are associated with successful adoption, implementation, and positive behavioral outcomes.     

A Community Service for a Rural High School

Abstract

This case study illustrates a preventive consultation using the ecological paradigm to guide entry, assessment, problem solving, and collaborative intervention design and implementation in a rural high school which initially had serious reservations about working with urban academicians. The ecological paradigm's emphasis on the identification of resources, on collaborative work with community members, and on developing tailored solutions based on the strengths of the setting are discussed. A four year follow-up describing the long-term outcome and commentary from a key setting member are included.     

The Organizational,Community and Programmatic Characteristics that Predict the Effective Implementation of After-School Programs

Abstract

To identify characteristics predicting the effective implementation of after-school programs, in-depth interviews were conducted at five sites randomly selected from a subset of 16 ACE after-school sites serving high risk youth in a southwestern city. Qualitative data from structured in-depth interviews, follow-up telephone conversations with personnel as well as researcher observations during site visits were synthesized. Data identified three constellations of characteristics associated with effective implementation: staffing, community and programmatic. Staffing characteristics included limited staff turnover and sufficient training. Community characteristics included cultural sensitivity and community integration. Programmatic characteristics included clearly defined program goals and specific program content. Researcher observations found outcomes assessment would also facilitate program implementation.     

The School-Based Violence Prevention Planning Program

Abstract

Despite a growing research literature identifying evidence-based prevention strategies, schools often adopt programs lacking evidentiary support (Gottfredson and Gottfredson, 2002; Ringwalt et al., 2002). Further, when evidence-based programs are adopted they often suffer from poor implementation (Gottfredson and Gottfredson, 2002). This gap between research and practice defines a need for efforts to further the dissemination and effective implementation of evidence-based prevention strategies. The School-Based Violence Prevention Planning Program (SBV3P) is designed to improve the dissemination of evidence-based prevention strategies by building the capacity of school personnel to identify and effectively implement violence prevention strategies that have substantial empirical support. In this program, prevention planning teams from schools create violence prevention plans by progressing through a structured planning framework emphasizing data-driven decision making and the use of empirically validated violence prevention strategies. To build capacity, participating teams are provided with training, technical assistance, and online resources. Here we present the SBV3P and review a pilot test of this program.     

Implementing and Evaluating School-Based Primary Prevention Programs and the Importance of Differential Effects on Outcomes

Abstract

In 2004, Second Step (Committee for Children, 2002), a violence prevention program, was implemented in the Cleveland Heights-University Heights school district for 1,416 K through second grade students. Both process and outcome measures were used to evaluate program impact and examine issues related to the implementation and evaluation of evidence-based practices. Process measures indicated a high level of implementation fidelity, while pre/post outcome testing on a 20% random sample of students demonstrated significant reductions in reactive aggression scores, t= 2.221, df= 238, p< 0.05, and evidence of declines in proactive aggression. Authors discuss challenges and opportunities related to implementing and evaluating evidence-based programming in Safe Schools/Health Students sites, particularly regarding strategies for the assessment of differential intervention effects to evaluate program outcomes.     

E-Sites For Safe Schools

ABSTRACT

A substantial body of research indicates that positive school culture benchmarks are integrally tied to the success of school reform and change in general. Additionally, an emerging body of research suggests a similar role for school culture in effective implementation of school violence prevention and intervention efforts. However, little research is available that specifically focuses on the elements of school culture that promote the successful implementation of bullying intervention programs aimed at reducing the most prevalent form of school violence. Therefore, this case study attempted to identify the school culture characteristics that supported or interfered with implementation of the Olweus Bullying Prevention Program from the viewpoint of school staff key informants in a junior high school in its second year of program implementation. Data were collected primarily from open-ended, semi-structured interviews along with informal observations and analysis of school documents. Themes emerging from the interviews were examined in relation to the findings from the observations and documents. School culture characteristics that supported implementation included a sense of family, warmth, collaboration, and connections among staff and between staff and students, combined with a central focus on learning as the primary mission of the school. Identified barriers to implementation included the local community served by the district that was slow to change and accept differences. Outcomes suggest that, beyond ensuring fidelity of implementing the core components of the Olweus Bullying Prevention Program, examining the characteristics of a school's culture that promote or impede implementation efforts might further ensure its success and integration.     

Design and Implementation of Parent Programs for a Community-Wide Adolescent Alcohol Use Prevention Program

SUMMARY

Project Northland is a comprehensive community trial designed to evaluate programs for the primary prevention of alcohol-related problems during adolescence. It uses a community-wide model, that includes both demand and supply reduction techniques. The first phase of Project Northland (grades 6-8) was completed with the effective implementation of multiple, complementary programs including a multi-year social behavioral curriculum, intensive parental involvement components, peer leadership opportunities, and community-level changes through the formation of local task forces. This article describes the process used for developing and implementing the comprehensive parent interventions used during the first phase of the research, highlighting the elements that led to the successful implementation of the programs with very high participation rates by parents, their young adolescents, teachers, and community members.     

The Impact of Increased State Regulation of Campus Sexual Assault Practices: Perspectives of Campus Personnel

ABSTRACT

Seeking to improve university accountability and compliance with federal mandates, states are considering new reforms including mandating reporting of campus sexual assault allegations. These new policies remain an empirical “black box.” To address these gaps, the current exploratory study draws upon a recent survey that examines the extent of awareness, perceptions, concerns, and policy practices of victim advocates and other personnel concerning new legislation. Virginia is chosen as the state for this case study given its recent implementation of a controversial mandated reporting law, as well as other initiatives, across institutions of higher education. Findings suggest that campus personnel are aware and largely supportive of the new reforms, but they also share concerns. Implications for research and policy development are discussed.     

School-Based Social Competence Promotion as a Primary Prevention Strategy

Abstract

School-based promotion of social competence continues to be a promising strategy for the primary prevention of psy-chosocial difficulties in children. Drawing on more than a decade of experience using this strategy, the authors identify major challenges they have encountered in the process of developing, implementing, evaluating, and maintaining social competence promotion programs in the schools. They emphasize the importance of using a dynamic, social-ecological conceptualization of both social competence and the organizational change process. Particular attention is given to overcoming obstacles to implementation and to long-term adoption of programs. Throughout, examples are drawn from the author's specific experiences with two longitudinal, multi-site projects.     

Characteristics of Homeless Women with Dependent Children

Abstract

Large community initiatives are a growing phenomenon both in the U.S. and worldwide. These initiatives address a wide variety of issues, including early childhood development, by integrating institutions such as schools, health agencies, and faith-based institutions that focus on separate but related aspects of community concern. A major challenge facing these initiatives is the competing demands of developing organizational capacity to promote effective programming while simultaneously delivering demonstrable results and accountability. Empowerment evaluation (Fetterman, Kaftarian, &; Wandersman, 1996) is an approach to evaluation and organizational capacity building that equips participants at all levels of an organization to pursue programming quality and results. This article describes and presents lessons learned from the development and implementation of a system of tools and processes, grounded in the principles of empowerment evaluation, designed to promote quality in the planning, implementation, and evaluation of a statewide school readiness initiative. While these lessons are specifically applicable to community-based early childhood development initiatives, they are broadly applicable to initiatives fostering systems change through community development.     

A Systematic Evaluation of Restorative Justice Practices: School Violence Prevention and Response

ABSTRACT

Despite its increasing recognition and use in U.S. schools, a limited amount of research has evaluated the effect of restorative justice (RJ) for school violence prevention and response. To date, there is no standardized method for RJ implementation. Therefore, this systematic literature review investigates peer-reviewed studies on the application of RJ practices in K-12 school settings. Ten articles were included in the review. Results of the review indicate a high degree of variability regarding the implementation and evaluation of RJ practices in schools. However, the majority of studies reported positive outcomes, including improved social relationships and reductions in office discipline referrals. The utility of RJ as a school violence prevention and intervention approach are discussed, along with future research directions.     

An Elementary School Violence Prevention Program

Abstract

This article details the implementation and empirical evaluation of an elementary school violence program. The problem of school violence and the significance and focus of the study are discussed. The general research question in this study was to examine the potential for a computer-mediated anger management program to enhance or improve the conflict resolution skills of youth with aggressive behavior problems in an elementary school setting. This study employed an 8-week computer-mediated anger management program, called SMART Talk (Students Managing Anger Resolution Together), for teaching conflict resolutions skills. This program was derived from a meta-analysis (Scheckner et al., 2002) due to its large effect size, statistical significance, and computer-mediated execution in order to verify past results with a more specifically defined population of at-risk (i.e., aggressive) students. Overall, the results suggest that participants in the intervention group significantly reported more intentions to use nonviolent strategies in a future conflict than students in the control group. Further research is recommended in order to assess the replicability of this study to other populations. Recommendations include longer intervention duration, larger sample size, multiple school settings, and multi-systemic implementation (i.e., teacher and parent workshops) as well as longitudinal studies.     

A Review of the Use of Social Support in Anti-Bullying Programs

Abstract

Bullying is a significant problem in schools across America. Educators are dealing with the problem of bullying through the implementation of various anti-bullying programs. Additionally, researchers are studying the problem and have begun to focus on the importance of contextual factors surrounding bullying such as social support (Beran & Tutty, 2002; Demaray & Malecki, 2003; Furlong, Chung, Bates, & Morrison, 1995; Malecki & Demaray, 2004a; Natvig, Albrektsen, & Qvarnstrom, 2001; Rigby, 2000; Rigby & Slee, 1999). Social support is an important contextual factor to consider in the bullying cycle. However, the use of social support in anti-bullying programs and interventions is often not explicit or is lacking. This paper provides an overview of the research on social support as a contextual variable in bullying behaviors and reviews six existing anti-bullying programs with a specific focus on how they incorporate social support elements into their interventions.     

The Tennis Ball Bomb Incident or the Safety Plan as “Shelf Document”

Abstract

This article presents a case study of a single incident involving a high school student who brought a “tennis ball” bomb to school. Although the district had recently created a safety plan, with a specific protocol for the discovery of a bomb in the building, the principal and the crises team ignored it. The principal did not refer to the safety plan because the incident did not take on the dimensions of a crisis until relatively late in its course. The crisis response team did not use the safety plan because they felt that they already knew from experience how best to deal with the situation. Both felt a lack of ownership in the safety plan because they had not been involved in drafting it. The article examines these and other problems of safety plan implementation and suggests remedies.     

Implementing a Violence Intervention for Inner-City Adolescents

SUMMARY

Violence is a growing public health problem that disproportionately affects African American adolescents. There is a pressing need for more attention to be directed to the development of theoretically-based, research-validated, culturally sensitive preventive interventions and programs. This article focuses on the steps involved in the development, implementation, and evaluation of a culturally-specific program designed to reduce the prevalence of violent behaviors and related injuries and deaths among urban, African American teens. The project's success was heavily dependent upon the commitment and cooperation of the community organizations that participated in the field research. Although numerous unexpected challenges were encountered, the preliminary findings indicate that it is possible to effectively implement preventive interventions in urban communities with modest funding.     

Evaluation of an Incentive System at a Summer Camp for Youth Experiencing Homelessness

SUMMARY

Children who are homeless often experience emotional and behavioral problems. They also may have difficulty interacting with peers. Programs designed to reward positive behaviors may be successful in improving behavioral functioning for children experiencing emotional distress due to homelessness and related risk factors. Families experiencing homelessness may have a difficult time accessing mental health services. School settings may be optimal environments for implementing programs to improve behavioral and social development for these children. In this paper, we describe the implementation and outcomes of an incentive system, developed to improve school behaviors and interactions for children experiencing homelessness. This system was implemented during a summer camp designed to enrich reading skills. Teachers and aides administered bracelets as secondary reinforcers for positive and prosocial behaviors. Children could purchase primary reinforcers, such as toys and art supplies, with the bracelets. Results were positive, supporting project activities. Future programs and evaluation projects should focus on delivering and evaluating prevention and therapy services for youth experiencing homelessness.     

Monitoring School Violence

Abstract

National monitoring of school violence is essential for needs assessments, policymaking, and evaluation at the national level. As informative and important as national monitoring is, the data generated at the national level is often not useful at the district or school site levels. There is therefore a need for a feasible method of monitoring school violence on the district and school level; furthermore, we need to find ways to effectively apply national-level information to schools and school districts. over time should be the foundation for the design, implementation, and evaluation of interventions in this area. In this article, we propose an expanded concept of monitoring that links comparable data on school violence at the grade, school site, district, and national levels. The paper presents our conceptual framework and methodology and illustrates its implementation in a district in Israel. We present examples of reports generated to monitor school violence for the district as a whole and for each of the school sites. Finally, we conclude that this is a feasible and useful model that social services could adopt to monitor practice in many other areas.     

In the Eye of the Reformer: Higher Education Personnel Perspectives on Campus Sexual Assault Policy Implementation

ABSTRACT

Despite recent attention to campus sexual assault (CSA), we know little about how institutions of higher education (IHE) have approached policy reform. Using anonymous web-based survey responses of 190 staff, faculty, and administrators from IHEs, we examined priorities that guided CSA policy implementation, as well as characteristics that explain attitudes toward CSA policy reforms. Respondents were significantly more likely to think policies had improved than to think that CSA is exaggerated or feel frustrated by the process of implementing CSA reforms. Respondents were also more likely to name due process and fairness as a high priority on their campus than creating a victim-centered response or managing the campus’ public image, suggesting that IHEs have carefully considered due process protections as they have implemented policies.     

Prevention of Child Abuse

Abstract

The application of the ecological paradigm to the potential prevention of child abuse is discussed, using two examples of consultation services designed to increase social support. The examples illustrate the principle of designing different interventions according to the ecologies of different communities. Principles of the ecological paradigm that guided the interventions are discussed. A follow-up assessment of the interventions, based on interview and questionnaire results obtained from participants in the projects, is presented.