Meta-analytically quantifying the reliability and biasability of forensic experts

Abstract:  In this paper we employ meta-analytic procedures and estimate effect sizes indexing the degree of reliability and biasability of forensic experts. The data are based on within-expert comparisons, whereby the same expert unknowingly makes judgments on the same data at different times. This allows us to take robust measurements and conduct analyses that compare variances within the same experts, and thus to carefully quantify the degree of consistency and objectivity that underlie expert performance and decision making. To achieve consistency, experts must be reliable, at least in the very basic sense that an expert makes the same decision when the same data are presented in the same circumstances, and thus be consistent with themselves. To achieve objectivity, experts must focus only on the data and ignore irrelevant information, and thus be unbiasable by extraneous context. The analyses show that experts are not totally reliable nor are they unbiasable. These findings are based on fingerprint experts decision making, but because this domain is so well established, they apply equally well (if not more) to all other less established forensic domains.