Can “disciplined passion” overcome the cynical view? An empirical inquiry of evaluator influence on police crime prevention program outcomes

Objectives

Investigate the degree and nature of influence that researchers have in police crime prevention programs and whether a high degree of influence is associated with biased reporting of results.

Methods

Meta-analytic inquiry of experimental and quasi-experimental studies (n?=?42), drawn from four Campbell Collaboration systematic reviews of leading police crime prevention strategies: problem-oriented policing, ??hot spots?? policing, ??pulling levers?? policing, and street-level drug enforcement.

Results

Larger program effects are not associated with studies with higher involvement on the part of the evaluator (e.g., assisting in strategy design, monitoring implementation, overcoming implementation problems).

Conclusions

This study does not find support for the cynical view, which holds that researchers have a personal stake in the program or are pressured to report positive results. Importantly, the evaluator??s involvement in the implementation of the program may be a necessary condition of successfully executed police experiments in complex field settings.     

Block Randomized Trials at Places: Rethinking the Limitations of Small N Experiments

Objectives

Place-based policing experiments have led to encouraging findings regarding the ability of the police to prevent crime, but sample sizes in many of the key studies in this area are small. Farrington and colleagues argue that experiments with fewer than 50 cases per group are not likely to achieve realistic pre-test balance and have excluded such studies from their influential systematic reviews of experimental research. A related criticism of such studies is that their statistical power under traditional assumptions is also likely to be low. In this paper, we show that block randomization can overcome these design limitations.

Methods

Using data from the Jersey City Drug Market Analysis Experiment (N = 28 per group) we conduct simulations on three key outcome measures. Simulations of simple randomization with 28 and 50 cases per group are compared to simulations of block randomization with 28 cases. We illustrate the statistical modeling benefits of the block randomization approach through examination of sums of squares in GLM models and by estimating minimum detectable effects in a power analysis.

Results

The block randomization simulation is found to produce many fewer significantly unbalanced samples than the naïve randomization approaches both with 28 and 50 cases per group. Block randomization also produced similar or smaller absolute mean differences across the simulations. Illustrations using sums of squares show that error variance in the block randomization model is reduced for each of the three outcomes. Power estimates are comparable or higher using block randomization with 28 cases per group as opposed to naïve randomization with 50 cases per group.

Conclusions

Block randomization provides a solution to the small N problem in place-based experiments that addresses concerns about both equivalence and statistical power. The authors also argue that a 50 case rule should not be applied to block randomized place-based trials for inclusion in key reviews.     

Propensity score analysis: promise, reality and irrational exuberance

Objectives

The aim of this work is to examine the promise that propensity scores can yield accurate effect estimates in nonrandomized experiments, review research on the realities of the conditions needed to meet this promise, and caution against irrational exuberance about their capacity to meet this promise.

Methods

A review of selected experimental work that illustrates both the promise and realities of propensity score analysis.

Results

Propensity score analysis of nonrandomized experiments can yield the same results as randomized experiments. Those estimates depend on meeting the strong ignorability assumption that the available covariates well describe selection processes and on use of comparison groups that are from the same location with very similar focal characteristics. When those assumptions are not met, propensity scores may not yield accurate estimates.

Conclusions

The use of propensity score analysis has proliferated exponentially, especially in the last decade, but careful attention to its assumptions seems to be very rare in practice. Researchers and policymakers who rely on these extensive propensity score applications may be using evidence of largely unknown validity. All stakeholders should devote far more empirical attention to justifying that each study has met these assumptions.     

Building group capacity for problem solving and police–community partnerships through survey feedback and training: a randomized control trial within Chicago’s community policing program

Objectives

To examine whether group capacity for problem solving and partnership building could be enhanced at police–community meetings by providing the results from community surveys and training for group facilitators.

Methods

A randomized control trial was conducted in 51 police beats in Chicago’s community policing program, CAPS. Unlike control beats, results from web-based community surveys were provided at beat meetings in the feedback and training beats, with facilitators in training beats also receiving training and exercises to guide problem solving about survey results. Analysis included OLS and logistic regression of data from questionnaires administered to police and resident participants, as well as observations at beat meetings, which measured resident capacity, attitudes about the police–community partnership, and problem-solving activities.

Results

Support for hypothesized effects was found with greater resident confidence in their ability to achieve outcomes and solve local problems, as well as officers viewing their relationships with residents at beat meetings more favorably. Effects, however, were inconsistent and limited to the feedback group. While additional training and support provided in training beats indicated fuller engagement in problem solving, possible negative effects on attitudes were observed.

Conclusions

Failure to find more effects is discussed in terms of implementation and resistance. Officer resistance to and a shift in organizational priorities away from community policing worked against achieving full program implementation. The beat meeting context provided a traditional framework for police–resident interactions that precluded more comprehensive use of community data and possibly heightened dissatisfaction with the level of problem solving that occurred.     

Procedural justice and police legitimacy: a systematic review of the research evidence

Objectives

We undertook a systematic review and meta-analysis to synthesize the published and unpublished empirical evidence on the impact of police-led interventions that use procedurally just dialogue focused on improving citizen perceptions of police legitimacy.

Methods

The systematic search included any public police intervention where there was a statement that the intervention involved police dialogue with citizens that either was aimed explicitly at improving police legitimacy, or used at least one core ingredient of procedural justice dialogue: police encouraging citizen participation, remaining neutral in their decision making, conveying trustworthy motives, or demonstrating dignity and respect throughout interactions. The studies included in our meta-analyses also had to include at least one direct outcome that measured legitimacy or procedural justice, or one outcome that is common in the legitimacy extant literature: citizen compliance, cooperation, confidence or satisfaction with police. We conducted separate meta-analyses, using random effects models, for each outcome.

Results

For every single one of our outcome measures, the effect of legitimacy policing was in a positive direction, and, for all but the legitimacy outcome, statistically significant. Notwithstanding the variability in the mode in which legitimacy policing is delivered (i.e., the study intervention) and the complexities around measurement of legitimacy outcomes, our review shows that the dialogue component of front-line police-led interventions is an important vehicle for promoting citizen satisfaction, confidence, compliance and cooperation with the police, and for enhancing perceptions of procedural justice.

Conclusions

In practical terms, our research shows the benefits of police using dialogue that adopts at least one of the principles of procedural justice as a component part of any type of police intervention, whether as part of routine police activity or as part of a defined police crime control program. Our review provides evidence that legitimacy policing is an important precursor for improving the capacity of policing to prevent and control crime.     

Scaling Criminal Offending

Objective:

This paper reviews a century of research on creating theoretically meaningful and empirically useful scales of criminal offending and illustrates their strengths and weaknesses.

Methods:

The history of scaling criminal offending is traced in a detailed literature review focusing on the issues of seriousness, unidimensionality, frequency, and additivity of offending. Modern practice in scaling criminal offending is measured using a survey of 130 articles published in five leading criminology journals over a two-year period that included a scale of individual offending as either an independent or dependent variable. Six scaling methods commonly used in contemporary criminological research are demonstrated and assessed using the National Longitudinal Survey of Youth 1979: dichotomous, frequency, weighted frequency, variety, summed category, and item response theory ??theta??.

Results:

The discipline of criminology has seen numerous scaling techniques introduced and forgotten. While no clearly superior method dominates the field today, the most commonly used scaling techniques are dichotomous and frequency scales, both of which are fraught with methodological pitfalls including sensitivity to the least serious offenses.

Conclusions:

Variety scales are the preferred criminal offending scale because they are relatively easy to construct, possess high reliability and validity, and are not compromised by high frequency non-serious crime types.     

Coalitions for a common purpose: managing relationships in experiments

Objectives

To discuss the character of relationships between operational and research staff and the necessary conditions for successful experiments.

Methods

A review of research and experience in the conduct of experiments, especially randomized controlled trials, examining the foundations for success, issues in maintaining cooperation with operational staff, implementation and leadership issues.

Results

The fundamental issue in successful experiments is the relationship between the operational and research entities, which most often resemble a coalition of temporary common interests rather than a partnership between parties with long-term common goals.

Conclusion

Experiments require close cooperation between the parties because of the need for maintenance and monitoring. Researchers who use field trials have solved many of the common problems faced by those embarking on experiments and those who do so will be rewarded by the quality of their findings. Relationships which may be characterised as temporary coalitions for a common purpose may, under the right conditions, ultimately mature into true research partnerships.     

A randomized test of initial and residual deterrence from directed patrols and use of license plate readers at crime hot spots

Objectives

To test the effects of short-term police patrol operations using license plate readers (LPRs) on crime and disorder at crime hot spots in Mesa, Arizona.

Methods

The study employed a randomized experimental design. For 15 successive 2-week periods, a four-officer squad conducted short daily operations to detect stolen and other vehicles of interest at randomly selected hot spot road segments at varying times of day. Based on random assignment, the unit operated with LPRs on some routes and conducted extensive manual checks of license plates on others. Using random effects panel models, we examined the impact of these operations on violent, property, drug, disorder, and auto theft offenses as measured by calls for service.

Results

Compared to control conditions with standard patrol strategies, the LPR locations had reductions in calls for drug offenses that lasted for at least several weeks beyond the intervention, while the non-LPR, manual check locations exhibited briefer reductions in calls regarding person offenses and auto theft. There were also indications of crime displacement associated with some offenses, particularly drug offenses.

Conclusions

The findings suggest that use of LPRs can reduce certain types of offenses at hot spots and that rotation of short-term LPR operations across hot spots may be an effective way for police agencies to employ small numbers of LPR devices. More generally, the results also provide some support for Sherman’s (1990) crackdown theory, which suggests that police can improve their effectiveness in preventing crime through frequent rotation of short-term crackdowns across targets, as it applies to hot spot policing.     

Pitfalls in the Use of Time Series Methods to Study Deterrence and Capital Punishment

Objectives

Evaluate the use of various time series methods to measure the deterrence effect of capital punishment.

Methods

The analysis of the time series approach to deterrence is conducted at two levels. First, the mathematical foundations of time series methods are described and the link between the time series properties of aggregate homicide and execution series and individual decision making is developed. Second, individual studies are examined for logical consistency.

Results

The analysis concludes that time series methods used to study the deterrence effects of capital punishment suffer from fundamental limitations and fail to provide credible evidence. The common limitation of these studies is their lack of attention to identification problems. Suggestions are made as to directions for future work that may be able to mitigate the weaknesses of the current literature.

Conclusions

Time series studies of capital punishment suffer from sufficiently serious identification problems that existing empirical findings are compatible with either the presence or the absence of a deterrent effect.     

Challenges of conducting field experiments in correctional settings: boot camp prison study as an example

Objectives

The challenges of conducting a field experiment in a correctional setting.

Methods

This paper discusses the issues related to planning, design, and completion of a randomized control study of a correctional boot camp including the difficulties confronted and how these were or were not overcome.

Results

At the beginning, correctional administrators were interested in and supportive of the study and this greatly facilitated our initial work. However, as described in this paper, during the course of the research obstacles and difficulties arose. Using the boot camp experiment as an example, we review the trials and tribulations of trying to do a field experiment in corrections. The importance of collecting quantitative and qualitative in order to understand the experiences of both the control and experimental groups is emphasized, as is the need to examine the theoretical mechanisms hypothesized to lead to changes in outcomes.

Conclusion

Some of the challenges confronted in this randomized control trial are those facing anyone attempting to conduct a field experiment, others relate to the particular issues faced by those who are conducting experiments in correctional settings. The paper concludes with a discussion of the issues confronting researchers who are doing field experiments in corrections.     

Mentoring programs to affect delinquency and associated outcomes of youth at risk: A comprehensive meta-analytic review

Objectives

To conduct a meta-analytic review of selective and indicated mentoring interventions for effects for youth at risk on delinquency and key associated outcomes (aggression, drug use, academic functioning). We also undertook the first systematic evaluation of intervention implementation features and organization and tested for effects of theorized key processes of mentor program effects.

Methods

Campbell Collaboration review inclusion criteria and procedures were used to search and evaluate the literature. Criteria included a sample defined as at risk for delinquency due to individual behavior such as aggression or conduct problems or environmental characteristics such as residence in a high-crime community. Studies were required to be random assignment or strong quasi-experimental design. Of 163 identified studies published from 1970–2011, 46 met criteria for inclusion.

Results

Mean effects sizes were significant and positive for each outcome category (ranging from d?=?0.11 for academic achievement to d?=?0.29 for aggression). Heterogeneity in effect sizes was noted for all four outcomes. Stronger effects resulted when mentor motivation was professional development but not by other implementation features. Significant improvements in effects were found when advocacy and emotional support mentoring processes were emphasized.

Conclusions

This popular approach has significant impact on delinquency and associated outcomes for youth at risk for delinquency. While evidencing some features may relate to effects, the body of literature is remarkably lacking in details about specific program features and procedures. This persistent state of limited reporting seriously impedes understanding about how mentoring is beneficial and ability to maximize its utility.     

The power of policing partnerships: sustaining the gains

Objectives

Third Party Policing (TPP) involves partnerships between police and third parties where the legal powers of third parties are harnessed to prevent or control crime problems. This paper explores the characteristics and mechanisms of TPP as a crime control strategy, focusing on how the partnership approach in policing can help sustain crime control gains over the long run. Using the ABILITY Truancy Trial as an example, I examine how policing can contribute to long-term social change for high-risk young people living in poor-performing school districts and high-risk communities.

Methods

The ABILITY Trial includes 102 young truants randomly allocated to a control (business-as-usual) or an experimental condition. The experimental condition activates the key theoretical components of Third Party Policing (TPP): a partnership between police and participating schools that activates and escalates (where needed) jurisdictional truanting laws (the legal lever).

Results

The paper presents a theoretical discussion of TPP and uses the ABILITY Trial to highlight the way TPP works in practice. Baseline data are presented for the ABILITY Trial. Outcome results are not presented.

Conclusions

Third Party Policing partnerships rest on the capacity of police to build relationships with third parties who have a stake in the crime problem, who possess responsive regulation legal levers, and who have a clear mandate to offer long-term solutions and help sustain the crime control gains. Partnerships, I argue, offer long-term solutions for police because they activate latent mechanisms, building the capacity for third parties to both maintain short-term gains and sustain the crime control gains beyond the lifespan of the initial police intervention.     

Assessing the Validity of the Law of Crime Concentration Across Different Temporal Scales

Objectives

The present study examined if Weisburd’s (Criminology 53(2):133–157, 2015) law of crime concentration held across different theoretically relevant temporal scales.

Methods

The cumulative percentages of Philadelphia, PA USA street blocks and intersections experiencing 25 and 50 % of street robberies by hour of the day, days of the week, and seasons of the year were compared to the bandwidth percentages established by Weisburd (2015). Different analyses were used to determine the stability of the micro-places’ street robbery levels within the three temporal scales.

Results

We found that the cumulative percentages of street blocks and intersections experiencing 25 and 50 % of street robberies at each of the three temporal scales closely matched the bandwidth percentages expected from Weisburd (2015) and some micro-places experienced street robberies across all temporal periods while others had more isolated temporal concentrations.

Conclusion

Weisburd’s (2015) law of crime concentration holds across different theoretically relevant temporal scales, and future criminology of place studies should not ignore temporal crime patterns. Further, it may be possible to refine hot spots policing approaches by incorporating spatial–temporal crime concentrations.

     

Predictable Policing: Measuring the Crime Control Benefits of Hotspots Policing at Bus Stops

Objectives

A fairly robust body of evidence suggests that hotspots policing is an effective crime prevention strategy. In this paper, we present contradictory evidence of a backfiring effect.

Methods

In a randomized controlled trial, aimed at reducing crime and disorder, London’s ‘hottest’ 102 bus-stops were targeted. Double patrol teams of Metropolitan Police Service uniformed officers visited the stops three times per shift (12:00–20:00), 5-times per week, for a duration of 15 min, over a 6 month period. Crucially, officers arrived and departed the bus stop on a bus, with significantly less time spent outside the bus stop setting. Outcomes were measured in terms of victim-generated crimes reported to the police and bus driver incident reports (DIRs), within targeted and catchment areas. We used adjusted Poisson-regression models to compare differences in pre- and post-treatment measures of outcomes and estimated-marginal-means to illustrate the treatment effect.

Results

DIRs went down significantly by 37 % (p = 0.07) in the near vicinity of the bus stops (50 m), by 40 % in the 100 m catchment area (p = 0.04) and marginally and non-significantly in the farthest catchment (10 %; p = 0.66), compared to control conditions. However, victim-generated crimes—the primary outcome measured in previous experiments—increased by 25 % (p = 0.10) in the near vicinity, by 23 % (p = 0.08) and 11 % (p ≤ 0.001) within the 100–150 m catchment areas, respectively.

Conclusions

These findings illustrate the role of bounded-rationality in everyday policing: reductions in crime are predicated on an elevated perceived risk-of-apprehension. Previous studies focused on clusters of addresses or public facilities, with police moving freely and unpredictably within the boundaries of the hotspot, but the patrol areas of officers in this experiment were limited to bus stops so offenders could anticipate their movements. Hotspots policing therefore backfires when offenders can systematically and accurately predict the temporal and spatial pattern of long-term targeting at a single location.

     

For whom does prison-based drug treatment work? Results from a randomized experiment

Objectives

Prison-based therapeutic community (TC) drug treatment followed by community aftercare is widely recognized as the most effective treatment paradigm for drug-dependent offenders. However, few randomized experiments have addressed this question and fewer studies have examined how interactions between treatment modality and individual characteristics may explain variations in outcomes.

Methods

Using a randomized experimental design, this study examined the effects of treatment modality [TC vs. Outpatient (OP) group counseling], individual psychosocial characteristics (e.g., risk, negative affect), and interactions on reincarceration over a 3-year follow-up period. Survival analysis using Cox regression with covariates was used to analyze data obtained from 604 subjects at a specialized drug treatment prison.

Results

The expected advantage of TC failed to emerge. Critical and heretofore unexamined interactions between treatment modality (TC vs. OP), inmate levels of risk, and negative effect help explain these unexpected findings.

Conclusion

The superiority of prison TC to less intensive OP counseling was not supported. The effects of TC appear to be conditioned by critical responsivity factors that have received little empirical attention.     

Where did it all go wrong? Implementation failure—and more—in a field experiment of procedural justice policing

Objectives

This paper presents the findings from a retrospective qualitative process evaluation to the Scottish Community Engagement Trial (ScotCET). The study explores the unanticipated results of a randomized field trial testing the effect of ‘procedurally just’ modes of road policing on public perceptions of police. The ScotCET intervention failed to produce the hypothesized results, producing instead significant, and unexplained, negative effects on key aspects of public perception. The present study seeks to examine, from the perspectives of officers implementing the experiment, what the impacts (intended or otherwise) of participation were.

Methods

Group interviews were held within the ScotCET experiment ‘units’ to explore how officers had collectively interpreted and framed ScotCET, and responded as a group to its requirements/demands. Nine groups were held over a 5-month period post experiment completion.

Results

Findings indicate that communication breakdowns during the ScotCET implementation led to misunderstandings of its aims and objectives, and of the requirements placed on officers. Within the context of organizational reform and perceived organizational ‘injustice’, commonly cited aspects of police culture were invoked to facilitate non-compliance with aspects of the experimental intervention, leading to implementation failures, and, possibly, a diffuse negative effect on the attitudes and behaviors of experiment officers.

Conclusions

Organizational structures and processes, and coercive top-down direction, are insufficient to ensure successful implementation of policing research, and, by implication, policing reforms, particularly those that demand alternative ways of ‘doing’ policing and ‘seeing’ citizens. Greater investment in organizational justice and encouraging openness to evidence-led knowledge is needed to promote change.

     

An experimental evaluation of a nationally recognized employment-focused offender reentry program

Objectives

The need for re-entry assistance is widely acknowledged, but specifics about what services actually lead to reduced recidivism are hard to find—at least among rigorous studies. This is a critical issue: at a time when there appears to be unprecedented support for expanding rehabilitative programs for offenders, there is a dearth of rigorously vetted program options from which to choose.

Methods

Collaborating with a nationally known employment-focused reentry program in Southern California, the authors compared employment, housing, and recidivism outcomes of reentering offenders (n?=?217) who were either randomized into the program or simply provided with a list of community resources. This approach was possible because the number of applicants at the time exceeded program capacity. Outcomes were based on self-report interviews conducted 1-year post-randomization and arrest records reflecting a 2-year follow-up period. The follow-up rate for interviews was 87 %.

Results

No significant differences were found for any of the between-group comparisons on any of the major intervention outcomes.

Conclusions

Findings from the study suggest a greater need to apply randomized designs to more carefully evaluate current reentry efforts. Methodological challenges of field experiments are also discussed.     

Defending and managing the pipeline: lessons for running a randomized experiment in a correctional institution

Objectives

To discuss the challenges faced in an experimental prisoner reentry evaluation with regard to managing the pipeline of eligible cases.

Methods

This paper uses a case study approach, coupled with a review of the relevant literature on issues of case flow in experimental studies in criminal justice settings. Included are recommendations for researchers on the management of case flow, reflections on the major research design issues encountered, and a listing of dilemmas that are likely to plague experimental evaluations of prisoner reentry programs.

Results

Particularly in a jail setting, anticipating the timing of release of a prisoner to the community is probably impossible given the large number of issues that impact release, many of which will be unanticipated. A detailed pipeline study is critical to the success of an experimental study targeting returning prisoners. Pipeline studies should be conducted under what will be the true conditions and context for enrollment, given all eligibility criteria.

Conclusions

With continued and systematic documentation of enrollment challenges in future experimental evaluations of reentry programs, as well as other experimental evaluations that involve individuals, academics can build a deep literature that would help facilitate future successful randomized experiments in the criminal justice field.     

Capital Punishment and Deterrence: Understanding Disparate Results

Objectives

Investigate how different model assumptions have driven the conflicting findings in the literature on the deterrence effect of capital punishment.

Methods

The deterrence effect of capital punishment is estimated across different models that reflect the following sources of model uncertainty: (1) the uncertainty about the probability model generating the aggregate murder rate equation, (2) the uncertainty about the determinants of an individual’s choice of committing a murder or not, (3) the uncertainty about state level heterogeneity, and (4) the uncertainty about the exchangeability between observations with zero murder case and those with positive murder cases.

Results

First, the estimated deterrence effects exhibit great dispersion across models. Second, a particular subset of models—linear models with constant coefficients—always predict a positive deterrence effect. All other models predict negative deterrence effects. Third, the magnitudes of the point estimates of deterrence effects differ mainly because of the choice of linear versus logistic specifications.

Conclusions

The question about the deterrence effect of capital punishment cannot be answered independently from substantive assumptions on what determines individual behavior. The need for judgment cannot be escaped in empirical work.