Lessons from a field experiment involving involuntary subjects 3,000 miles away

Objectives

Describe the challenges involved in conducting field experiments that entail a long distance between the research team and the research site.

Methods

A summary of the lessons learned from the field experiment of Hawaii??s Opportunity Probation with Enforcement (HOPE).

Results

Pre-trial planning is especially important when the research team is a long distance from the research site. A good communication strategy helps educate practitioners on the merits of conservative design choices, such as intent-to-treat, and helps to signal the importance of the study and therefore of maintaining the condition assignments and delivering the intervention with fidelity.

Conclusions

Distance creates additional challenges for the research team. These challenges make it even more essential to exploit assets at the research site. Distance creates more uncertainty, which makes pre-planning even more important, but it is expensive. Criminal-justice funding agencies?? support for exploratory studies as precursors to full-blown trials would improve the quality of experimental criminal-justice research.     

Challenges of conducting field experiments in correctional settings: boot camp prison study as an example

Objectives

The challenges of conducting a field experiment in a correctional setting.

Methods

This paper discusses the issues related to planning, design, and completion of a randomized control study of a correctional boot camp including the difficulties confronted and how these were or were not overcome.

Results

At the beginning, correctional administrators were interested in and supportive of the study and this greatly facilitated our initial work. However, as described in this paper, during the course of the research obstacles and difficulties arose. Using the boot camp experiment as an example, we review the trials and tribulations of trying to do a field experiment in corrections. The importance of collecting quantitative and qualitative in order to understand the experiences of both the control and experimental groups is emphasized, as is the need to examine the theoretical mechanisms hypothesized to lead to changes in outcomes.

Conclusion

Some of the challenges confronted in this randomized control trial are those facing anyone attempting to conduct a field experiment, others relate to the particular issues faced by those who are conducting experiments in correctional settings. The paper concludes with a discussion of the issues confronting researchers who are doing field experiments in corrections.     

Lessons from a partially controlled field trial

Objectives

Using the case of an on-going work-oriented prisoner-reentry experiment in Milwaukee, describe the challenges of organizing and sustaining a high-quality trial in the field in which only the randomization and data analysis are directly ??controlled?? by the evaluation team.

Methods

The case study is of a randomized experiment involving youthful male prisoners with a history of violence and gang membership, scheduled for release into Milwaukee. The intervention included six months of pre-release services with a work-release opportunity, and intensive services and supervision following release. The case study describes the initial experimental plan and how much of that plan could be salvaged in the face of delays, administrative errors, and other problems.

Results

The initial plan, when compared with the actual experiment, specified a larger and more homogeneous sample, more resources devoted to various aspects of the treatment, and more intensive supervision following release. These problems arose despite the best efforts of public officials. Randomization was preserved, and for that reason the results will still be of interest, although perhaps under-powered.

Conclusions

The ??gold standard?? may become a bit tarnished in the field. It was crucial in this experiment to have a member of the experimental team engaged with the relevant state agencies at every step of the process to sustain this effort and to ensure that the treatment was delivered and relevant data generated. A newsletter and regular meetings with agents proved useful. The outcomes will have high internal validity.     

Bullying prevention programs: the importance of peer intervention, disciplinary methods and age variations

Objectives

The aim of this paper is to respond to the commentary of Peter K. Smith, Christina Salmivalli, and Helen Cowie (Journal of Experimental Criminology, 2012), who raise concerns regarding some of the findings of our systematic review and meta-analyses on the effectiveness of bullying prevention programs. They target three findings in particular: (1) the significant association of ??Work with Peers?? with greater victimization; (2) the significant association of ??Disciplinary Methods?? with less bullying perpetration and victimization; and (3) the age variations in effectiveness, suggesting larger effect sizes for older age students.

Methods

We provide explicit information and further detailed analyses on the relationship between these features and effect sizes, including heterogeneity tests and results from weighted regression analyses. For one element in particular (work with peers), we present further research findings from evaluations conducted by Smith, Salmivalli, and Cowie (and also findings from other independent researchers) which support our previous findings. New within-program analyses to examine variations in effect sizes with the age of the students are also presented.

Results

Evaluations conducted by Smith, Salmivalli and Cowie (and by other independent researchers) indicate the same research conclusions: although peer support schemes appear effective based on attitudinal surveys, these schemes are not related to actual levels of bullying or victimization and, in fact, are quite often related to an increase in bullying and victimization. Our definition of ??disciplinary methods?? did not include the zero-tolerance approach or any type of harsh discipline as suggested in the commentary. In all relevant cases, ??disciplinary methods?? included sanctions within a warm and loving framework, following the Olweus bullying prevention guidelines. While most programs that utilized firm disciplinary methods were inspired by Olweus, the relationship between disciplinary methods and less victimization was not driven by the Olweus program (which was not related to the victimization effect size). Larger effect sizes (i.e. reductions in bullying and victimization) for programs implemented with older students is a robust result also found in a more recent systematic review regarding the effects of anti-bullying programs on bystander intervention. In within-program analyses, most results suggested that effect sizes were greater for younger students, but these results were driven by the less controlled evaluations. The most controlled evaluation (randomized experiment) provided the opposite result.

Conclusions

More research is clearly needed on the effectiveness of bullying prevention programs with students of different ages, and we also recommend randomized experiments to assess the importance of different intervention components.     

Procedural justice, routine encounters and citizen perceptions of police: main findings from the Queensland Community Engagement Trial (QCET)

Objectives

To test, under randomized field trial conditions, the impact of police using the principles of procedural justice during routine encounters with citizens on attitudes towards drink-driving, perceptions of compliance, and their satisfaction with the police.

Methods

We conducted the first randomized field trial??the ??Queensland Community Engagement Trial?? (QCET)??to test the impact of police engaging with citizens by operationalizing the key ingredients of procedural justice (neutrality, citizen participation, respect, and trustworthy motives) in a short, high-volume police?Ccitizen encounter. We randomly allocated 60 roadside Random Breath Testing (RBT) operations to control (business-as-usual) and experimental (procedural justice) conditions. Driver surveys were used to measure the key outcomes: attitudes towards drinking and driving, satisfaction with police and perceptions of compliance.

Results

Citizen perceptions of the encounter revealed that the experimental treatment was delivered as planned. We also found significant differences between the experimental and control groups on all key outcome measures: drivers who received the experimental RBT encounter were 1.24 times more likely to report that their views on drinking and driving had changed than the control group; experimental respondents reported small but higher levels of compliance (d?=?.07) and satisfaction (d?=?.18) with police during the encounter than did their control group counterparts.

Conclusions

Our results show that the way citizens perceive the police can be influenced by the way in which police interact with citizens during routine encounters, and demonstrate the positive benefits of police using the principles of procedural justice. Our study was limited by the use of paper-only surveys and low response rate. We also recognize that the experiment setting (RBT road blocks) is limiting and non-reflective of the wider set of routine police?Ccitizen encounters. Future research should be undertaken, using experimental methods, to replicate our field operationalization of procedural justice in different types of police?Ccitizen encounters.     

Defending and managing the pipeline: lessons for running a randomized experiment in a correctional institution

Objectives

To discuss the challenges faced in an experimental prisoner reentry evaluation with regard to managing the pipeline of eligible cases.

Methods

This paper uses a case study approach, coupled with a review of the relevant literature on issues of case flow in experimental studies in criminal justice settings. Included are recommendations for researchers on the management of case flow, reflections on the major research design issues encountered, and a listing of dilemmas that are likely to plague experimental evaluations of prisoner reentry programs.

Results

Particularly in a jail setting, anticipating the timing of release of a prisoner to the community is probably impossible given the large number of issues that impact release, many of which will be unanticipated. A detailed pipeline study is critical to the success of an experimental study targeting returning prisoners. Pipeline studies should be conducted under what will be the true conditions and context for enrollment, given all eligibility criteria.

Conclusions

With continued and systematic documentation of enrollment challenges in future experimental evaluations of reentry programs, as well as other experimental evaluations that involve individuals, academics can build a deep literature that would help facilitate future successful randomized experiments in the criminal justice field.     

Propensity score analysis: promise, reality and irrational exuberance

Objectives

The aim of this work is to examine the promise that propensity scores can yield accurate effect estimates in nonrandomized experiments, review research on the realities of the conditions needed to meet this promise, and caution against irrational exuberance about their capacity to meet this promise.

Methods

A review of selected experimental work that illustrates both the promise and realities of propensity score analysis.

Results

Propensity score analysis of nonrandomized experiments can yield the same results as randomized experiments. Those estimates depend on meeting the strong ignorability assumption that the available covariates well describe selection processes and on use of comparison groups that are from the same location with very similar focal characteristics. When those assumptions are not met, propensity scores may not yield accurate estimates.

Conclusions

The use of propensity score analysis has proliferated exponentially, especially in the last decade, but careful attention to its assumptions seems to be very rare in practice. Researchers and policymakers who rely on these extensive propensity score applications may be using evidence of largely unknown validity. All stakeholders should devote far more empirical attention to justifying that each study has met these assumptions.     

Can “disciplined passion” overcome the cynical view? An empirical inquiry of evaluator influence on police crime prevention program outcomes

Objectives

Investigate the degree and nature of influence that researchers have in police crime prevention programs and whether a high degree of influence is associated with biased reporting of results.

Methods

Meta-analytic inquiry of experimental and quasi-experimental studies (n?=?42), drawn from four Campbell Collaboration systematic reviews of leading police crime prevention strategies: problem-oriented policing, ??hot spots?? policing, ??pulling levers?? policing, and street-level drug enforcement.

Results

Larger program effects are not associated with studies with higher involvement on the part of the evaluator (e.g., assisting in strategy design, monitoring implementation, overcoming implementation problems).

Conclusions

This study does not find support for the cynical view, which holds that researchers have a personal stake in the program or are pressured to report positive results. Importantly, the evaluator??s involvement in the implementation of the program may be a necessary condition of successfully executed police experiments in complex field settings.     

A randomized controlled trial of the effectiveness of multisystemic therapy in the Netherlands: post-treatment changes and moderator effects

Objective

In the present randomized controlled trial, the effectiveness of multisystemic therapy (MST) in The Netherlands was examined. Moderator tests were conducted for ethnicity, age and gender.

Methods

The sample consisted of N?=?256 adolescents, referred because of conduct problems, and randomized to MST or treatment as usual (TAU). Assessments (questionnaires and observational ratings) took place before and immediately after the treatment.

Results

MST was more effective than TAU in decreasing externalizing behavior, ODD, CD and property offences, but not for violence. Findings were mixed for adolescents’ and parental cognitions: the MST group, compared to TAU, showed an improvement in parental sense of competence, and a decrease in adolescents’ hostility, but no change in self-esteem and an increase in personal failure. MST was effective for positive dimensions of parenting and associations with prosocial peers, but not for relationships with deviant peers. MST was equally effective for adolescents of different ages and with different ethnicities. However, MST showed larger (and more positive) effects for adolescent cognitions for boys than for girls.

Conclusions

Effects of MST in The Netherlands are generally comparable to the positive findings reported in American and Norwegian trials. MST seems equally effective across age and ethnic minority groups, but some gender moderator effects were found for adolescent cognitions.     

Early prevention of life-course personal and property violence: a 19-year follow-up of the Montreal Longitudinal-Experimental Study (MLES)

Objectives

This study assessed the impact of a multi-component prevention program on personal and property violence across three developmental periods (early adolescence, mid-adolescence and late adolescence/early adulthood).

Methods

The preventive intervention targeted disruptive kindergarten boys from low socioeconomic status families when they were 7 through 9 years of age. A randomized control trial was conducted to assess the impact of the preventive intervention relative to a control group.

Results

Two different approaches to data analysis were adopted: an intention-to-treat (ITT) approach and an instrumental variable (IV) approach. Results from the ITT analysis showed that the level of property violence for the intervention group was persistently lower across the three developmental periods compared to the control group. However, the intervention group did not differ from the control group on personal violence throughout adolescence and early adulthood. Results from the IV analysis generally confirmed these findings.

Conclusions

The discussion focuses on the differential effects of the prevention program on personal versus property violence.     

Hyperbolic Time Discounting, Offender Time Preferences and Deterrence

Objectives

This study examines the phenomena of intertemporal decision making??decisions involving costs and benefits that occur at different points in time. Two models of intertemporal time discounting are the exponential and hyperbolic models. Previous work in behavioral economics and psychology is relied on to make the case that the discounting of delayed outcomes (both gains and losses) may be hyperbolic rather than exponential.

Methods

Data were collected from 478 university undergraduate students who responded to a hypothetical scenario involving drunk driving. The potential gains and losses of drunk driving were delayed at five different intervals from ??tonight?? to ??10?years from now??, respondents were also asked to provide estimates of both the risk they would get caught if they did drink and drive and the probability that they would drink and drive under the conditions described in the scenario.

Results

Our results imply that individuals have hyperbolic time preferences for both rewards and gains, and that??unlike severity, the effect of which may be muted by risk??these discount functions appear to be operating independently of changes in the risk certainty of detection. Consistent with hyperbolic discounting, for example, when the benefit of drinking and driving was delayed by 1?week the self-reported intention to drink and drive increased by nearly 10%, however, when the gain was delayed by one month, intentions to drink and drive increased by only 4%. A smaller effect was found for delayed costs.

Conclusions

Avenues for additional research include the possibility of negative discount rates and the implications of persons?? awareness of their discount rates.     

Coalitions for a common purpose: managing relationships in experiments

Objectives

To discuss the character of relationships between operational and research staff and the necessary conditions for successful experiments.

Methods

A review of research and experience in the conduct of experiments, especially randomized controlled trials, examining the foundations for success, issues in maintaining cooperation with operational staff, implementation and leadership issues.

Results

The fundamental issue in successful experiments is the relationship between the operational and research entities, which most often resemble a coalition of temporary common interests rather than a partnership between parties with long-term common goals.

Conclusion

Experiments require close cooperation between the parties because of the need for maintenance and monitoring. Researchers who use field trials have solved many of the common problems faced by those embarking on experiments and those who do so will be rewarded by the quality of their findings. Relationships which may be characterised as temporary coalitions for a common purpose may, under the right conditions, ultimately mature into true research partnerships.     

Race, Space, and Violence: Exploring Spatial Dependence in Structural Covariates of White and Black Violent Crime in US Counties

Objectives

To join the literature on spatial analysis with research testing the racial invariance hypothesis by examining the extent to which claims of racial invariance are sensitive to the spatial dynamics of community structure and crime.

Methods

Using 1999?C2001 county-level arrest data, we employ seemingly unrelated regression models, spatial lag models, and geographically weighted regression analyses to (1) compare the extent of racial similarity/difference across these different modeling procedures, (2) evaluate the impact of spatial dependence on violent crime across racial groups, and (3) explore spatial heterogeneity in associations between macro-structural characteristics and violent crime.

Results

Results indicate that spatial processes matter, that they are more strongly associated with white than black violent crime, and that accounting for space does not significantly attenuate race-group differences in the relationship between structural characteristics (e.g., structural disadvantage) and violent crime. Additionally, we find evidence of significant variation across space in the relationships between county characteristics and white and black violent crime, suggesting that conclusions of racial invariance/variation are sensitive to where one is looking. These results are robust to different specifications of the dependent variable as well as different units of analysis.

Conclusions

Our study suggests the racial invariance debate is not yet settled. More importantly, our study has revealed an additional level of complexity??race specific patterns of spatially heterogeneous effects??that future research on social structure and racial differences in violence should incorporate both empirically and theoretically.     

Managing an on-the-spot lottery in reentry services

Objectives

Describe the authors?? experiences in designing and conducting a randomized field experiment of a community-based, reentry program for ex-offenders.

Methods

Two surveys: one with reentry clients not involved in our outcome evaluation, and a follow-up survey of participants who underwent randomization in order to participate in the outcome study. Qualitative input from program staff and clients were also recorded, supplemented with observations of the authors.

Results

Having a research staff member located at the program site proved to be a key advantage for monitoring frustrations voiced by program staff and prospective clients, thereby allowing for the modification of the selection procedures over time to minimize resistance. Ultimately, the simplest approach proved to be the most acceptable. The importance of certain procedural justice themes were suggested by the survey results and the observed acceptability of our on-the-spot lottery approach to randomization.

Conclusions

The survey results (and our onsite experiences) provided unequivocal evidence that randomization was unpopular, but that resistance can be partially mitigated by adhering to basic principles of procedural justice.     

Examining What Makes Violent Crime Victims Unique: Extending Statistical Methods for Studying Specialization to the Analysis of Crime Victims

Objectives

Much victimization research focuses on specific types of crime victims, which implies that the factors responsible for some victimization outcomes are distinct from others. Recent developments in victimization theory, however, take a more general approach, postulating that victimization regardless of type will share a similar basic etiology. This research examines how and whether the risk factors that are associated with violent victimization significantly differ from those that predict nonviolent victimization.

Methods

Using data from 3,682 Kentucky youth, we employ Osgood and Schreck??s (2007) Item Response Theory-based statistical approach for detecting specialization to determine the properties and predictors of tendencies for individuals to fall victim to specific types of crime.

Results

Findings show that victims typically experience varied outcomes, but some victims have a clear tendency toward violent victimization and that it is possible to predict this tendency.

Conclusions

The findings indicate that a more nuanced general approach, one that accounts for tendencies toward specific victimization outcomes, might add insight about the causes of victimization. This research also shows how statistical methods designed to examine offense specialization can add value for research on victimization.     

Does Self-Control Influence Maternal Attachment? A Reciprocal Effects Analysis from Early Childhood Through Middle Adolescence

Objectives

The purpose of this study is twofold. First, this study assesses the extent to which self-control and maternal attachment mutually influence one another. Second, it investigates whether this process continues to occur during adolescence. To date, studies of the etiology of self-control have yet to adequately address these issues, despite the fact that a number of theoretical perspectives emphasize the reciprocal nature of the parent-child relationship.

Methods

The current study seeks to shed light on these issues by examining the relationship between self-control and maternal attachment using structural equation modeling for eight waves of data spanning a period of time that encompasses early childhood through middle adolescence.

Results

The results yield two findings bearing on the adequacy of Gottfredson and Hirschi??s model of self-control development. First, measures of self-control and maternal attachment were found to mutually influence one another during childhood. Second, these effects were reduced to nonsignificance during adolescence.

Conclusions

This study finds that self-control emerges during childhood in a complex manner in which it both shapes and is shaped by parental attachment.     

The Moving Home Effect: A Quasi Experiment Assessing Effect of Home Location on the Offence Location

Objectives

This study aims to test whether the home location has a causal effect on the crime location. To accomplish this the study capitalizes on the natural experiment that occurs when offender??s move, and uses a unique metric, the distance between sequential offenses, to determine if when an offender moves the offense location changes.

Methods

Using a sample of over 40,000 custodial arrests from Syracuse, NY between 2003 and 2008, this quasi-experimental design uses t test??s of mean differences, and fixed effects regression modeling to determine if moving has a significant effect on the distance between sequential offenses.

Results

This study finds that when offenders move they tend to commit crimes in locations farther away from past offences than would be expected without moving. The effect is rather small though, both in absolute terms (an elasticity coefficient of 0.02), and in relation to the effect of other independent variables (such as the time in between offenses).

Conclusions

This finding suggests that the home has an impact on where an offender will choose to commit a crime, independent of offence, neighborhood, or offender characteristics. The effect is small though, suggesting other factors may play a larger role in influencing where offenders choose to commit crime.     

Block Randomized Trials at Places: Rethinking the Limitations of Small N Experiments

Objectives

Place-based policing experiments have led to encouraging findings regarding the ability of the police to prevent crime, but sample sizes in many of the key studies in this area are small. Farrington and colleagues argue that experiments with fewer than 50 cases per group are not likely to achieve realistic pre-test balance and have excluded such studies from their influential systematic reviews of experimental research. A related criticism of such studies is that their statistical power under traditional assumptions is also likely to be low. In this paper, we show that block randomization can overcome these design limitations.

Methods

Using data from the Jersey City Drug Market Analysis Experiment (N = 28 per group) we conduct simulations on three key outcome measures. Simulations of simple randomization with 28 and 50 cases per group are compared to simulations of block randomization with 28 cases. We illustrate the statistical modeling benefits of the block randomization approach through examination of sums of squares in GLM models and by estimating minimum detectable effects in a power analysis.

Results

The block randomization simulation is found to produce many fewer significantly unbalanced samples than the naïve randomization approaches both with 28 and 50 cases per group. Block randomization also produced similar or smaller absolute mean differences across the simulations. Illustrations using sums of squares show that error variance in the block randomization model is reduced for each of the three outcomes. Power estimates are comparable or higher using block randomization with 28 cases per group as opposed to naïve randomization with 50 cases per group.

Conclusions

Block randomization provides a solution to the small N problem in place-based experiments that addresses concerns about both equivalence and statistical power. The authors also argue that a 50 case rule should not be applied to block randomized place-based trials for inclusion in key reviews.