Quasi-experimental designs for community-level public health violence reduction interventions: a case study in the challenges of selecting the counterfactual

Objectives

We highlight the importance of documenting the step-by-step processes used for the selection of comparison areas when evaluating a community-level intervention that targets a large-scale community.

Methods

We demonstrate the proposed method using a propensity score matching framework for an impact analysis of the Cure Violence Public Health Model in Philadelphia. To select comparison communities, propensity score models are run using different levels of aggregation to define the intervention site. We discuss the trade-offs made.

Results

We find wide variation in documentation and explanation in the extant literature of the methods used to select comparison communities. The size of the unit of analysis at which a community is measured complicates the decision processes, and in turn, can affect the validity of the counterfactual.

Conclusions

It is important to carefully consider the unit of analysis for measurement of comparison communities. Assessing the geographic clustering of matched communities to mirror that of the treated community holds conceptual appeal and represents a strategy to consider when evaluating community-level interventions taking place at a large scale. Regardless of the final decisions made in the selection of the counterfactual, the field could benefit from more systematic diagnostic tools that document and guide the steps and decisions along the way, and ask: “could there have been another way of doing each step, and what difference would this have made?” Overall, across community-level evaluations that utilize quasi-experimental designs, documentation of the counterfactual selection process will provide a more fine-grained understanding of causal inference.