American Terrorism and Extremist Crime Data Sources and Selectivity Bias: An Investigation Focusing on Homicide Events Committed by Far-Right Extremists

This paper examines the reliability of the methods used to capture homicide events committed by far-right extremists in a number of open source terrorism data sources. Although the number of research studies that use open source data to examine terrorism has grown dramatically in the last 10 years, there has yet to be a study that examines issues related to selectivity bias. After reviewing limitations of existing terrorism studies and the major sources of data on terrorism and violent extremist criminal activity, we compare the estimates of these homicide events from 10 sources used to create the United States Extremist Crime Database (ECDB). We document incidents that sources either incorrectly exclude or include based upon their inclusion criteria. We use a “catchment-re-catchment” analysis and find that the inclusion of additional sources result in decreasing numbers of target events not identified in previous sources and a steadily increasing number of events that were identified in any of the previous data sources. This finding indicates that collectively the sources are approaching capturing the universe of eligible events. Next, we assess the effects of procedural differences on these estimates. We find considerable variation in the number of events captured by sources. Sources include some events that are contrary to their inclusion criteria and exclude others that meet their criteria. Importantly, though, the attributes of victim, suspect, and incident characteristics are generally similar across data source. This finding supports the notion that scholars using open-source data are using data that is representative of the larger universe they are interested in. The implications for terrorism and open source research are discussed.