Proficiency Testing as a Basis for Estimating Uncertainty of Measurement: Application to Forensic Alcohol and Toxicology Quantitations

Abstract: While forensic laboratories will soon be required to estimate uncertainties of measurement for those quantitations reported to the end users of the information, the procedures for estimating this have been little discussed in the forensic literature. This article illustrates how proficiency test results provide the basis for estimating uncertainties in three instances: (i) For breath alcohol analyzers the interlaboratory precision is taken as a direct measure of uncertainty. This approach applies when the number of proficiency tests is small. (ii) For blood alcohol, the uncertainty is calculated from the differences between the laboratory’s proficiency testing results and the mean quantitations determined by the participants; this approach applies when the laboratory has participated in a large number of tests. (iii) For toxicology, either of these approaches is useful for estimating comparability between laboratories, but not for estimating absolute accuracy. It is seen that data from proficiency tests enable estimates of uncertainty that are empirical, simple, thorough, and applicable to a wide range of concentrations.     

Estimating the uncertainty associated with Widmark's equation as commonly applied in forensic toxicology

No computation is performed more frequently by forensic toxicologists than that involving Widmark's equation. The equation is employed to estimate either the number of drinks consumed or the corresponding blood or breath alcohol concentration. Despite the wide use of Widmark's equation, rarely is an uncertainty estimate also provided. Estimates from Widmark's equation involve at least seven uncertain random variables. Uncertainty estimates are presented that rely on methods of general error propagation compared to a method developed by Widmark. Assuming reasonable variable and uncertainty estimates, the error propagation method yielded for N=10.4 drinks, a combined uncertainty (standard deviation) of 1.3 drinks (CV=12.3%). Similarly, estimating the blood alcohol concentration yielded for 0.120 g/100 ml, an uncertainty of 0.0255 g/100 ml (CV=21.2%). Widmark's uncertainty method yielded 1.6 drinks (CV=15.4%). The derivation of Widmark's uncertainty estimate is also presented, showing that he considered only rho and beta to be uncertain. Widmark estimates for the number of drinks should include a 2CV estimate of approximately 25% while the blood alcohol concentration estimate should include a 2CV estimate of approximately 42%. Including valid estimates of uncertainty should enhance the legal admissibility and confidence for Widmark estimations.     

Lessons learned from inter-laboratory studies of carbon isotope analysis of honey

Forensic application of carbon isotope ratio measurements of honey and honey protein to investigate the degree of adulteration with high fructose corn syrup or other C₄ plant sugars is well established. These measurements must use methods that exhibit suitable performance criteria, particularly with regard to measurement uncertainty and traceability – low levels of adulteration can only be detected by methods that result in suitably small measurement uncertainties such that differences of 1‰ or less can be reliably detected. Inter-laboratory exercises are invaluable to assess the state-of-the art of measurement capabilities of laboratories necessary to achieve such performance criteria. National and designated metrology institutes from a number of countries recently participated in an inter-laboratory assessment (CCQM-K140) of stable carbon isotope ratio determination of bulk honey. The same sample material was distributed to a number of forensic isotope analysis laboratories that could not participate directly in the metrological comparison. The results from these studies have demonstrated that the majority of participants provided isotope delta values with acceptable performance metrics; that all participants ensured traceability of their results; and that where measurement uncertainties were reported; these were fit-for-purpose. A number of the forensic laboratories only reported precision rather than full estimates of measurement uncertainty and this was the major cause of the few instances of questionable performance metrics. Reporting of standard deviations in place of measurement uncertainties is common practice outside metrology institutes and the implications for interpretations of small differences in isotopic compositions are discussed. The results have also highlighted a number of considerations that are useful for organisers of similar inter-laboratory studies in the future.     

The influence of alcohol content variation in UK packaged beers on the uncertainty of calculations using the Widmark equation

It is common for forensic practitioners to calculate an individual's likely blood alcohol concentration following the consumption of alcoholic beverage(s) for legal purposes, such as in driving under the influence (DUI) cases. It is important in these cases to be able to give the uncertainty of measurement on any calculated result, for this reason uncertainty data for the variables used for any calculation are required. In order to determine the uncertainty associated with the alcohol concentration of beer in the UK the alcohol concentration (%v/v) of 218 packaged beers (112 with an alcohol concentration of ≤5.5%v/v and 106 with an alcohol concentration of >5.5%v/v) were tested using an industry standard near infra-red (NIR) analyser. The range of labelled beer alcohol by volume (ABV's) tested was 3.4%v/v – 14%v/v. The beers were obtained from a range of outlets throughout the UK over a period of 12?months. The root mean square error (RMSE) was found to be ±0.43%v/v (beers with declared %ABV of ≤5.5%v/v) and ±0.53%v/v (beers with declared %ABV of >5.5%v/v) the RMSE for all beers was ±0.48%v/v. The standard deviation from the declared %ABV is larger than those previously utilised for uncertainty calculations and illustrates the importance of appropriate experimental data for use in the determination of uncertainty in forensic calculations.     

Error and its Meaning in Forensic Science

The discussion of “error” has gained momentum in forensic science in the wake of the Daubert guidelines and has intensified with the National Academy of Sciences' Report. Error has many different meanings, and too often, forensic practitioners themselves as well as the courts misunderstand scientific error and statistical error rates, often confusing them with practitioner error (or mistakes). Here, we present an overview of these concepts as they pertain to forensic science applications, discussing the difference between practitioner error (including mistakes), instrument error, statistical error, and method error. We urge forensic practitioners to ensure that potential sources of error and method limitations are understood and clearly communicated and advocate that the legal community be informed regarding the differences between interobserver errors, uncertainty, variation, and mistakes.     

Forensic Analysis of Explosives Using Isotope Ratio Mass Spectrometry (IRMS)—Part 1: Instrument Validation of the DELTAplusXP IRMS for Bulk Nitrogen Isotope Ratio Measurements

Abstract: A significant amount of research has been conducted into the use of stable isotopes to assist in determining the origin of various materials. The research conducted in the forensic field shows the potential of isotope ratio mass spectrometry (IRMS) to provide a level of discrimination not achievable utilizing traditional forensic techniques. Despite the research there have been few, if any, publications addressing the validation and measurement uncertainty of the technique for forensic applications. This study, the first in a planned series, presents validation data for the measurement of bulk nitrogen isotope ratios in ammonium nitrate (AN) using the DELTA^plusXP (Thermo Finnigan) IRMS instrument equipped with a ConFlo III interface and FlashEA? 1112 elemental analyzer (EA). Appropriate laboratory standards, analytical methods and correction calculations were developed and evaluated. A validation protocol was developed in line with the guidelines provided by the National Association of Testing Authorities, Australia (NATA). Performance characteristics including: accuracy, precision/repeatability, reproducibility/ruggedness, robustness, linear range, and measurement uncertainty were evaluated for the measurement of nitrogen isotope ratios in AN. AN (99.5%) and ammonium thiocyanate (99.99+%) were determined to be the most suitable laboratory standards and were calibrated against international standards (certified reference materials). All performance characteristics were within an acceptable range when potential uncertainties, including the manufacturer’s uncertainty of the technique and standards, were taken into account. The experiments described in this article could be used as a model for validation of other instruments for similar purposes. Later studies in this series will address the more general issue of demonstrating that the IRMS technique is scientifically sound and fit‐for‐purpose in the forensic explosives analysis field.     

Determination of illicit drugs in seized materials: role of sampling and analysis in estimation of measurement uncertainty

The determination of illicit active ingredients in seized materials, in order to assess penal or administrative offences, is routinely carried out in many forensic toxicology laboratories. This paper presents main features of the protocol adopted in the Authors' laboratory for the above investigations. In particular, sampling and analysis are considered as the same measurement process quantifying their combined contribution to overall measurement uncertainty. Aspects concerning representative sampling in the case of single and multiple items are discussed. The effects of material heterogeneity are considered by analyzing separately distinct primary samples taken from different parts of the sampling target. Possible errors due to particles dimension that could arise when sub-sampling are also considered. Analytical precision, bias and other matrix effects are studied in order to quantify the component of the overall measurement uncertainty associated to the analysis of prepared test samples. Typical scenarios arising when measurement results are used to assess compliance with specification limits are also discussed revealing the crucial role of measurement uncertainty.     

Drugs WorkBook (DWB): A tool for the analysis of illicit drugs in seized materials

Accurate and reliable analytical measurements are essential when data are to be used to assist the Court in deciding whether or not a drug offence has been committed and therefore about either the innocence or guilt of the accused. The Italian law on drugs demands that compliance with specification limits be assessed on the basis of the actual content of controlled substance contained in seized materials. As a consequence, the role of measurement uncertainty, significant figures and rounding errors becomes critical. In order to assist analysts of forensic toxicology laboratories with illicit drug-related cases, a software tool named Drugs WorkBook (DWB) has been developed. The tool is useful for the quantification of illicit drugs in seized materials along with their measurement uncertainties, the assessment of compliance to specification limits, the printing of comprehensive laboratory reports and the organization of case archives. Other quality control topics, such as control charts, are included. The tool's databases can be edited by the user and maintained up to date. The tool is made freely available to the scientific community.     

Estimating Biological Characteristics With Virtual Laser Data

Laser scanning technology is increasingly being used in forensic anthropological research to obtain virtual data for archival purposes and post hoc measurement collection. This research compared the measurement accuracy of two laser scanners—the FARO Focus^3D 330X and the FARO Freestyle^3D—against traditionally obtained (i.e., by hand) control data (N = 454). Skeletal data were collected to address a novel question: the ability of laser scanning technology to produce measurements useful for biological characteristic estimation, such as sex and stature. Results indicate that both devices produced measurements very similar to control (c. 3‐mm average absolute error), but also illuminate a tendency to under‐measure. Despite these findings, the virtual data produced sex and stature estimates that varied little from control‐produced estimates, signifying the usefulness of virtual data for preliminary biological identification when the skeletal elements are no longer available for physical analysis.     

Uncertainty in Widmark calculations: ABV variation in packaged versions of the most popular beers in the UK

Forensic practitioners regularly use the Widmark equation to determine theoretical blood alcohol concentrations for use in cases involving alcohol. It is important with these calculations to determine the uncertainty associated with any result. Previous work has investigated the uncertainty in percent alcohol by volume (%ABV) from beers produced by small independent breweries in the UK but did not study the top selling beers in the UK. The top selling lagers and ales/bitters in the UK were identified by sales volume and the %ABV determined. These data was then used to determine the percent coefficient of variation (%CV) that should be used by forensic practitioners when constructing alcohol technical defence reports for use in forensic cases. These samples, from what may be described as ‘big’ brewers, were determined to have a smaller root mean square error (RMSE) (±0.1%v/v, n?=?35), and %CV than those previously reported for beers produced by small, independent breweries in the UK. The results from this study shows that different RMSE's should be used for %ABV when determining the uncertainty of results from Widmark calculations depending if the drinks consumed have been from either ‘big’ brewers or small, independent breweries.     

Metrological framework for selecting morphological characters to identify species and estimate developmental maturity of forensically significant insect specimens

Accurate age estimates of immature necrophagous insects associated with a human or animal body can provide evidence of how long the body has been dead. These estimates are based on species-specific details of the insects’ aging processes, and therefore require accurate species identification and developmental stage estimation. Many professionals who produce or use identified organisms as forensic evidence have little training in taxonomy or metrology, and appreciate the availability of formalized principles and standards for biological identification. Taxonomic identifications are usually most readily and economically made using categorical and qualitative morphological characters, but it may be necessary to use less convenient and potentially more ambiguous characters that are continuous and quantitative if two candidate species are closely related, or if identifying developmental stages within a species. Characters should be selected by criteria such as taxonomic specificity and metrological repeatability and relative error. We propose such a hierarchical framework, critique various measurements of immature insects, and suggest some standard approaches to determine the reliability of organismal identifications and measurements in estimating postmortem intervals. Relevant criteria for good characters include high repeatability (including low scope for ambiguity or parallax effects), pronounced discreteness, and small relative error in measurements. These same principles apply to individuation of unique objects in general.

Key points

• Metrological rigour can increase in forensic entomology by selecting measurements based on their metrological qualities.
• Selection of high-quality features for morphological identification of organisms should consider these criteria: (1) pronounced discreteness of features (minimising group overlap or maximizing interval); (2) high repeatability of assessment (such as symmetrical width rather than asymmetrical length); (3) small relative error in measurement (selecting the physically largest continuous rigid feature for measurement).
• These metrological principles also apply to individuation of unique objects in general.

     

Veldt fires in South Africa: Implications on osteometry and the biological profile

Standard operating procedures for forensic anthropological analyses dictate that thermally altered remains should not be measured, hindering the creation of a biological profile. Few studies have addressed estimating biological parameters from burned remains, with the greatest focus of this research area being on cremated remains. However, veldt fires are more common than cremation in the South African forensic context. The aim of this study was to explore the degree of structural changes observed in domestic pig (Sus scrofa) bones associated with thermal destruction and the potential impact on the estimation of a biological profile using standard osteometric methods. A total of 96 pig femora were divided equally into two categories: fresh and dry. Within each category, equal samples were exposed to different durations of burning, namely, 5, 10, and 20 min. Ten standard femoral anthropological measurements were collected before and after burning. Technical error of measurement and Wilcoxon signed-rank tests were used to assess changes in the femoral dimensions before and after burning. Most measurements were significantly different after burning, with the fresh bones decreasing in size by up to 7.8% and the dry bones decreasing in size by up to 4.0%. The magnitude of post-burning measurement changes for both burn conditions was similar to, or smaller than has previously been reported for observer measurement errors of commonly used variables investigated for standard osteometric studies. Veldt fires are less intense than cremation, thus causing less shrinkage.     

The Difficult Task of Assessing Perimortem and Postmortem Fractures on the Skeleton: A Blind Text on 210 Fractures of Known Origin

The distinction between perimortem and postmortem fractures is an important challenge for forensic anthropology. Such a crucial task is presently based on macro‐morphological criteria widely accepted in the scientific community. However, several limits affect these parameters which have not yet been investigated thoroughly. This study aims at highlighting the pitfalls and errors in evaluating perimortem or postmortem fractures. Two trained forensic anthropologists were asked to classify 210 fractures of known origin in four skeletons (three victims of blunt force trauma and one natural death) as perimortem, postmortem, or dubious, twice in 6 months in order to assess intraobserver error also. Results show large errors, ranging from 14.8 to 37% for perimortem fractures and from 5.5 to 14.8% for postmortem ones; more than 80% of errors concerned trabecular bone. This supports the need for more objective and reliable criteria for a correct assessment of peri‐ and postmortem bone fractures.     

Breath alcohol measurement variability associated with different instrumentation and protocols

Breath alcohol measurement has variability resulting from instrumental, procedural and biological components. Reliable estimates of the standard deviation (S.D.) are necessary for calculating uncertainty in the form of confidence intervals. These estimates are concentration dependant and can be obtained from models derived from duplicate breath test data. Duplicate data from four state jurisdictions (Alabama using the Drager 7110, Minnesota using the Intoxilyzer 5000, Washington using the BAC Datamaster and Wisconsin using the Intoximeter EC/IR) were analyzed to derive predictive models for standard deviation as a function of concentration. All jurisdiction/instrument combinations yielded forensically acceptable variation while showing a general linear increase in standard deviation with concentration. This is consistent with a multiplicative error model. Jurisdictions using the same instruments but not performing duplicate analyses could also employ the straightforward models derived here to estimate standard deviations for their measurement results.     

Rib Variation at Multiple Locations and Implications for Histological Age Estimation*

Existing histological age estimation methods using the rib were developed mainly from the midshaft; however, in forensic practice, uncertainty of sampling location often arises due to fragmented or previously sampled ribs. The potential for error increases when sampling location is uncertain and utilizing a section beyond the midshaft (either anterior or posterior) may result in erroneous age estimates. Additionally, there is debate within the field regarding the minimum number of sections needed for accurate age estimation. The aim of this research is to determine the importance of the midshaft distinction for age-at-death assessment and the necessity of analyzing serial sections by evaluating histological variables at sampling locations along the length of the rib. Three seriated histological sections at three sampling locations (anterior, midshaft, and posterior) were obtained from sixth ribs of ten postmortem human subjects. Cortical area (Ct.Ar) and osteon population density (OPD) were collected from each section (n = 90). Significant differences were determined in Ct.Ar between sampling locations, demonstrating the variation present along the length of the rib. A comparison of OPD at sampling locations revealed significant differences, suggesting that sampling site is critical to accurate age estimates. When sampling location is uncertain, a more anterior section should be taken. Analysis of serial sections within locations revealed no significant differences in OPD or Ct.Ar, supporting the practice of collecting data from one section for age estimation. While an age estimate can be achieved through the analysis of one section, best practice suggests reading two sections to capture intraindividual variation.     

Accuracy of 3D Scanners in Tooth Mark Analysis

The objective of this study was to compare the accuracy of contact and laser 3D scanners in tooth mark analysis. Ten dental casts were scanned with both 3D scanners. Seven linear measurements were made from the 3D images of dental casts and biting edges generated with DentalPrint© software (University of Granada, Granada, Spain). The uncertainty value for contact 3D scanning was 0.833 for the upper dental cast and 0.660 mm for the lower cast; similar uncertainty values were found for 3D-laser scanning. Slightly higher uncertainty values were obtained for the 3D biting edges generated. The uncertainty values for single measurements ranged from 0.1 to 0.3 mm with the exception of the intercanine distance, in which higher values were obtained. Knowledge of the error rate in the 3D scanning of dental casts and biting edges is especially relevant to be applied in practical forensic cases.     

Failings of Trauma‐Specific and Related Psychological Tests in Detecting Post‐Traumatic Stress Disorder in Forensic Settings

Judges and juries tend to be particularly impressed by test data, especially quantitative test data. Psychometric tests specific for assessing the presence of post‐traumatic stress disorder (PTSD) are commonly employed by forensic mental health evaluators. Most of these instruments, however, have been designed to detect PTSD in treatment or research, and not forensic, settings. Those who rely on these measures without adequate awareness of their often significant limits in correctly identifying malingering may induce finders of fact to inordinately confidently accept the presence of PTSD. This article reviews problematic structural and content components of trauma‐specific and related instruments used to evaluate PTSD and discusses the utility of specific techniques liable to be used in forensic settings to “fool” these measures.     

Error Rates in Forensic Child Sexual Abuse Evaluations

When mental health, medical, and social work professionals and paraprofessionals make false positive or false negative errors in their judgments about the validity of allegations of child sexual abuse, the consequences can be catastrophic for the affected children and adults. Because of the high stakes, practitioners, legal decision makers, and policy makers should have some idea of the magnitude and variability of error rates in this domain. A novel approach was used to estimate individual error rates for 110 professionals (psychologists, physicians, social workers, and others) who conduct or participate in forensic child sexual abuse evaluations. The median estimated false positive and false negative error rates were 0.18 and 0.36, respectively. Estimated error rates varied markedly from one participant to the next. For example, the false positive error rate estimates ranged from 0.00 to 0.83. These estimates are based on participants’ self-reported substantiation rates and on their subjective frequency distributions for the probability of truth for the abuse allegations they evaluate.     

Accuracy of comparison decisions by forensic firearms examiners

This black box study assessed the performance of forensic firearms examiners in the United States. It involved three different types of firearms and 173 volunteers who performed a total of 8640 comparisons of both bullets and cartridge cases. The overall false-positive error rate was estimated as 0.656% and 0.933% for bullets and cartridge cases, respectively, while the rate of false negatives was estimated as 2.87% and 1.87% for bullets and cartridge cases, respectively. The majority of errors were made by a limited number of examiners. Because chi-square tests of independence strongly suggest that error probabilities are not the same for each examiner, these are maximum-likelihood estimates based on the beta-binomial probability model and do not depend on an assumption of equal examiner-specific error rates. Corresponding 95% confidence intervals are (0.305%, 1.42%) and (0.548%, 1.57%) for false positives for bullets and cartridge cases, respectively, and (1.89%, 4.26%) and (1.16%, 2.99%) for false negatives for bullets and cartridge cases, respectively. The results of this study are consistent with prior studies, despite its comprehensive design and challenging specimens.