The feasibility of external blind DNA proficiency testing. I. Background and findings

We describe the origins, purposes, and findings of a national study to determine whether a large-scale program of blind proficiency testing in U.S. DNA laboratories is feasible and/or practical. Proficiency testing in clinical laboratories is reviewed, particularly as mandated by the Clinical Laboratory Improvement Acts and its role in the regulation of those laboratories. Proficiency testing in forensic urine drug testing labs is also briefly reviewed. Studies involving comparisons between open and blind proficiency testing are discussed. The clinical laboratory proficiency testing and regulation landscape provides the background for the DNA Act of 1994, and the congressional mandate to investigate blind proficiency testing in forensic DNA laboratories. Four models of blind proficiency testing are defined and discussed, along with the advantages and disadvantages of each and estimates of the costs of a large-scale program. The purposes of proficiency testing in a quality-assurance context are likewise discussed and related to the models and the arguments generally proffered for and against blind vs. open proficiency testing.     

Organizing a Proficiency Testing Program on Stamp Impressions Examination in Accordance with ISO/IEC 17043 Requirements

Stamps and stamp impressions examinations are based on matching defects and design details on the questioned and control samples. These examinations are routinely carried out by document examiners around the world. International proficiency tests for questioned documents examination have been available for decades while similar programs specifically focusing on stamp impressions examination are rare. This study reported a recent proficiency testing program on stamp impressions examination organized by an accredited provider in accordance with ISO/IEC 17043 requirements. Twenty‐four forensic laboratories registered for the program. Apart from giving details on the design and operation of the program, this study also aimed to provide the limitations and difficulties encountered in sample preparations, homogeneity, and stability tests of the testing materials. Various comments and feedbacks received from the participants, particularly in respect of examination approaches, challenges faced by the participants in forming conclusions and their suggestions for further improvement would be evaluated.     

The feasibility of external blind DNA proficiency testing. II. Experience with actual blind tests

The background and goals of a national study to determine the feasibility of blind proficiency testing in U.S. forensic DNA laboratories are discussed. Part of the project involved designing and executing a series of fifteen blind proficiency tests. Execution included biological specimen donor recruitment and case evidence manufacturing. Simulated cases were submitted to DNA laboratories by law enforcement agencies and in some cases by other forensic-science laboratories. Replicate-manufactured evidence was submitted to reference laboratories to simulate the workings of a larger-scale program. Ten tests were straightforward, and essentially tested analytical ability. Five tests involved selecting on the basis of case facts appropriate bloodstains for typing from a bloodstain pattern. We describe in detail our experience in designing and conducting these blind proficiency test trials, and relate those experiences to the overall issue of blind proficiency testing as a quality-assurance tool in forensic DNA laboratories. In this feasibility test series, one blind test was detected by a laboratory, a second one was shown to the lab by law enforcement, and a third was never completed because of lapses in communication. Turnaround times were relatively fast in the independent/commercial labs and relatively slow in the larger public laboratories. Two cross-state case-to-case CODIS "hits" were "planted" among the first series of ten blind tests. One pair was detected. One member of the second pair went to a lab that was not CODIS-ready.     

Perceptions of blind proficiency testing among latent print examiners

In recent years, scholars have levied multiple criticisms against traditional proficiency testing procedures in forensic laboratories. Consequently, on several occasions, authorities have formally recommended that laboratories implement blind proficiency testing procedures. Implementation has been slow, but laboratory management has increasingly expressed interest in initiating blind testing in at least some forensic disciplines, with some laboratories conducting blind testing in almost all disciplines. However, little is known about how a key population perceives blind proficiency testing, i.e., forensic examiners. We surveyed active latent print examiners (N = 338) to explore perceptions of blind proficiency testing and determine whether beliefs varied between examiners who work for laboratories with and without blind proficiency testing. Results suggest that examiners do not hold particularly strong beliefs about such procedures, but that examiners who work in laboratories with blind proficiency testing procedures view them significantly more positively than those who do not. Further, examiner responses provide insight into potential obstacles to continued implementation.     

Proficiency Testing as a Basis for Estimating Uncertainty of Measurement: Application to Forensic Alcohol and Toxicology Quantitations

Abstract: While forensic laboratories will soon be required to estimate uncertainties of measurement for those quantitations reported to the end users of the information, the procedures for estimating this have been little discussed in the forensic literature. This article illustrates how proficiency test results provide the basis for estimating uncertainties in three instances: (i) For breath alcohol analyzers the interlaboratory precision is taken as a direct measure of uncertainty. This approach applies when the number of proficiency tests is small. (ii) For blood alcohol, the uncertainty is calculated from the differences between the laboratory’s proficiency testing results and the mean quantitations determined by the participants; this approach applies when the laboratory has participated in a large number of tests. (iii) For toxicology, either of these approaches is useful for estimating comparability between laboratories, but not for estimating absolute accuracy. It is seen that data from proficiency tests enable estimates of uncertainty that are empirical, simple, thorough, and applicable to a wide range of concentrations.     

HAIRVEQ 2006: evolution of laboratories' performance after different educational actions

HAIRVEQ is a proficiency testing program for hair analysis of illicit drugs organized by the Istituto Superiore di Sanità (Rome, Italy) and the Institut Municipal d'Investigació Mèdica (Barcelona, Spain). The aim of the three exercises performed in 2006 was the evaluation of 32 laboratories' performance when analyzing the same hair sample containing opiates, cocaine and methadone, after carrying out some specific educational interventions. In the first round, the sample was sent to be analyzed following laboratory routine methodology. In the second round, standard operating procedures (SOP) for hair testing including sample preparation, method validation and qualitative and quantitative data evaluation, and an open hair sample for SOP training were also sent together with other hair samples including the one used for performance evaluation. After the second round, a workshop was held with participant laboratories to discuss methodological issues and interpretation of obtained results. An additional amount of open samples was distributed to the laboratories for implementing the SOPs. In the third round, the same unknown sample containing opiates, cocaine and methadone was resent for the final evaluation of laboratory performance. In the first round, 11 incorrect qualitative results (10 false negative and 1 false positive) were reported by seven laboratories (22%), in the second round, a reduction in the number of incorrect results was observed (4 false negatives and 1 false positive were reported by four laboratories, 13%) and in the third round, 5 false positives and 5 false negatives were reported by seven laboratories (22%). Concerning quantitative results, the scatter was similar between the three rounds and similar to the ones reported by other proficiency tests in hair analysis. More educational actions should be addressed to a group of laboratories, which did not yet show satisfying qualitative and quantitative results.     

Latent Print Proficiency Testing: An Examination of Test Respondents,Test-Taking Procedures,and Test Characteristics

Proficiency testing is a key component of quality assurance programs within crime laboratories and can help improve laboratory practices. However, current proficiency testing procedures contain significant limitations and can be misinterpreted by examiners and court personnel (Garrett & Mitchell, 2018). To evaluate some of these limitations, we surveyed latent print examiners (n = 198) after they completed a Collaborative Testing Services, Inc. proficiency test. Additionally, we evaluated test performance and used a quality metric algorithm to evaluate the quality of test prints. Results do not suggest that respondents are dissimilar to the broader examiner population, although they may engage in different behaviors when completing tests versus casework. Findings show that proficiency testing contains prints of high quality and is perceived as both relatively easy and representative of casework. The test discriminated between inexperienced and experienced respondents, and verification procedures were largely ineffective in reducing errors. Objective quality metrics may provide a path forward to improving proficiency testing in a measurable manner.     

Results of a proposed breath alcohol proficiency test program

Although proficiency test programs have long been used in both clinical and forensic laboratories, they have not found uniform application in forensic breath alcohol programs. An initial effort to develop a proficiency test program appropriate to forensic breath alcohol analysis is described herein. A total of 11 jurisdictions participated in which 27 modern instruments were evaluated. Five wet bath simulator solutions with ethanol vapor concentrations ranging from 0.0254 to 0.2659 g/210 L were sent to participating programs, instructing them to perform n = 10 measurements on each solution using the same instrument. Four of the solutions contained ethanol only and one contained ethanol mixed with acetone. The systematic errors for all instruments ranged from -11.3% to +11.4% while the coefficient of variations ranged from zero to 6.1%. A components-of-variance analysis revealed at least 79% of the total variance as being due to the between-instrument component for all concentrations. Improving proficiency test program development should consider: (1) clear protocol instructions, (2) frequency of proficiency testing, (3) use lower concentrations for determining limits-of-detection and -quantitation, etc. Despite the lack of a biological component, proficiency test participation should enhance the credibility of forensic breath test programs.     

The hair analysis proficiency testing program of the French Society of Analytical Toxicology

In an effort to improve laboratories performing hair analysis in forensic cases, the French Society of Analytical Toxicology (S.F.T.A.) has implemented a proficiency testing program since 1992. Actually about 10 laboratories are participating. Each survey is dedicated to one analyte or one pharmacological class: opiates (6-monoacetylmorphine, morphine and codeine), cocaine and benzoylecgonine, tetrahydrocannabinol, buprenorphine and norbuprenorphine, beta-blockers (metoprolol, atenolol), beta 2-agonists (salbutamol, clenbuterol). Animal hair was tested for clenbuterol. Prior to sending, hair samples were reduced to a powdered form, well mixed to ensure homogeneity, and then tested by GC/MS or HPLC/MS. Results confirm those obtained in a preliminary study on opiates and cocaine analysis in hair: a common analytical procedure has to be used by all the participants, including hydrolysis of hair. It is essential to work on authentic drug-positive hair samples and not on spiked samples. Participation at this program is free of charge and considered as an educational tool. Comparison of the results with those of other laboratories in Europe and USA shows that the analytical methods used during this program are in accordance with all the other procedures.     

How do latent print examiners perceive proficiency testing? An analysis of examiner perceptions,performance, and print quality

Proficiency testing has the potential to serve several important purposes for crime laboratories and forensic science disciplines. Scholars and other stakeholders, however, have criticized standard proficiency testing procedures since their implementation in laboratories across the United States. Specifically, many experts label current proficiency tests as non-representative of actual casework, at least in part because they are not sufficiently challenging (e.g., [1], [2], [3], [4]. In the current study, we surveyed latent print examiners (n = 322) after they completed a Collaborative Testing Services proficiency test about their perceptions of test items. We also evaluated respondents’ test performance and used a quality metric algorithm (LQMetrics) to obtain objective indicators of print quality on the test. Results were generally consistent with experts’ concerns about proficiency testing. The low observed error rate, examiner perceptions of relative ease, and high objective print quality metrics together suggest that latent print proficiency testing is not especially challenging. Further, examiners indicated that the test items that most closely resembled real-world casework were also the most difficult and contained prints of the lowest quality. Study findings suggest that including prints of lower quality may increase both the difficulty and representativeness of proficiency testing in latent print examination.     

Forensic Analysis of Explosives Using Isotope Ratio Mass Spectrometry (IRMS)—Part 2: Forensic Inter-Laboratory Trial: Bulk Carbon and Nitrogen Stable Isotopes in a Range of Chemical Compounds (Australia and New Zealand)

Abstract:  Comparability of data over time and between laboratories is a key issue for consideration in the development of global databases, and more broadly for quality assurance in general. One mechanism that can be utilized for evaluating traceability is an inter-laboratory trial. This paper addresses an inter-laboratory trial conducted across a number of Australian and New Zealand isotope ratio mass spectrometry (IRMS) laboratories. The main objective of this trial was to determine whether IRMS laboratories in these countries would record comparable values for the distributed samples. Four carbon containing and four nitrogen containing compounds were distributed to seven laboratories in Australia and one in New Zealand. The laboratories were requested to analyze the samples using their standard procedures. The data from each laboratory was evaluated collectively using International Standard ISO 13528 ( Statistical methods for use in proficiency testing by inter-laboratory comparisons ). "Warning signals" were raised against one participant in this trial. "Action signals" requiring corrective action were raised against four participants. These participants reviewed the data and possible sources for the discrepancies. This inter-laboratory trial was successful in providing an initial snapshot of the potential for traceability between the participating laboratories. The statistical methods described in this article could be used as a model for others needing to evaluate stable isotope results derived from multiple laboratories, e.g., inter-laboratory trials/proficiency testing. Ongoing trials will be conducted to improve traceability across the Australian and New Zealand IRMS community.     

HAIRVEQ: an external quality control scheme for drugs of abuse analysis in hair

The Istituto Superiore di Sanità of Rome, Italy, in cooperation with Institut Municipal d'Investigaciò Mèdica of Barcelona, Spain, set up an external quality control program (HAIRVEQ) to evaluate reliability in hair testing for drug abuse by laboratories from the Italian National Health Service. Samples included in the program were real hair samples from drugs consumers. Prior to sending, hair samples were reduced to powdered form, mixed to ensure homogeneity and tested with GC/MS by four Reference Laboratories. Up to now, four different exercises have been concluded and 23 laboratories participated. Samples containing high and low concentrations of opiates, cocaine and metabolites, low concentrations of MDMA and two blank samples, were included in the intercomparison exercises performed in the first year of HAIRVEQ activities. Results show an insufficient performance of participating laboratories. About 82% of laboratories reported incorrect results on a qualitative basis (false positive and false negative results) for some of the submitted samples. More than one-half of laboratories reported quantitative results (60%). On the basis of the calculated z scores, only between 35 and 55% of results reported should be considered as satisfying. Guidelines have to be provided by Italian authorities for method validation as well as set of recommended cut-off concentrations to orientate laboratories in their quality objectives when developing analytical methodologies as tools to improve reliability and consequently performance of hair analysis.     

The 1998-1999 collaborative exercises and proficiency testing program on DNA typing of the Spanish and Portuguese Working Group of the International Society for Forensic Genetics (GEP-ISFG)

A total of 28 laboratories (labs) submitted results for the 1998 collaborative exercise and the proficiency testing program of the Spanish and Portuguese Working Group of the International Society for Forensic Genetics (GEP-ISFG) group. This number increased to 46 labs in 1999. Six bloodstains were submitted, each one with 200 microl soaked in cotton except the sample no. 6 submitted for DNA quantification which had 2 microl. One of the samples was a mixed stain. A paternity testing case and a criminal case in the 1998 trial (GEP'98) and two paternity testing cases in 1999 (GEP'99) were included and the statistical evaluation of the evidence was requested in both cases. In the GEP'99 trial, a theoretical paternity testing case was included. A total of 52 DNA genetic markers were used by the participants in the GEP'98 trial, which increased to 101 in GEP'99. Despite this increasing number of participating labs, results remained quite satisfactory. All the labs used PCR-based DNA polymorphisms with an increasing number of markers, obtaining good results. SLPs were used by a decreasing number of labs but the results indicated a good level of expertise despite the different protocols used.Good results were also obtained for mtDNA despite the difficulties presented by the samples due to the presence of length heteroplasmy in some samples in both trials. The detection of heteroplasmy should, however, be improved.Similar conclusions were reached for both, the paternity and the criminal case by all the labs. Common methodologies for the statistical evaluation of the paternity case were used and the paternity index and the probability of paternity (with an a priori value of 0.5) reported by most of the labs. Also, a great uniformity was found in the evaluation of the criminal case despite the lack of a specific hypothesis in the design of the exercise. Some errors in statistical programs or in calculations were detected in a theoretical paternity case included in the GEP'99 trial for statistical analysis.     

法医精神病学能力验证相关问题探讨

能力验证是利用实验室间比对,按照预先制定的准则评价参加者的能力,是世界各国用以评价和保障实验室/机构能力所采用的重要手段.在我国司法鉴定领域,能力验证已逐渐成为执业管理、监督管理的重要手段.根据管理部门的要求,司法鉴定机构应通过参加能力验证活动证明其具有相应的技术能力,并作为内部质量控制、发现问题和持续改进的重要途径.作为公认的主观性较强、风险较高的司法鉴定行业,在法医精神病专业开展能力验证活动更具必要性与迫切性.近几年的运作尝试也表明,在本专业开展能力验证活动是可行的,评价结果亦是行业技术水平的真实反映.但在项目设计与运作上还有完善空间,如样本与物品制备可真实、完备些,可争取更大的技术指导与示范引导作用,应合理控制考核难度,并增强专家公议结果的可靠程度、可接受性,且运作方式应更灵活、多样.参加机构应提高对能力验证的认识,加强业务培训,科学对待;作业时严格遵守作业要求,通读懂材料,寻找关键点,并注重要点与逻辑性,反馈结论要尽量明确、唯一,争取较满意评价结果.     

Implementation of a Blind Quality Control Program in a Forensic Laboratory

A blind quality control (QC) program was successfully developed and implemented in the Toxicology, Seized Drugs, Firearms, Latent Prints (Processing and Comparison), Forensic Biology, and Multimedia (Digital and Audio/Video) sections at the Houston Forensic Science Center (HFSC). The program was put into practice based on recommendations set forth in the 2009 National Academy of Sciences report and is conducted in addition to accreditation required annual proficiency tests. The blind QC program allows HFSC to test its entire quality management system and provides a real-time assessment of the laboratory’s proficiency. To ensure the blind QC cases mimicked real casework, the workflow for each forensic discipline and their evidence submission processes were assessed prior to implementation. Samples are created and submitted by the HFSC Quality Division to whom the expected answer is known. Results from 2015 to 2018 show that of the 973 blind samples submitted, 901 were completed, and only 51 were discovered by analysts as being blind QC cases. Implementation data suggests that this type of program can be employed at other forensic laboratories.     

CLIA program; simplifying CLIA regulations relating to accreditation, exemption of laboratories under a state licensure program, proficiency testing, and inspection--HCFA. Final rule

This final rule responds to selected comments received on a final rule with a comment period implementing the Clinical Laboratory Improvement Amendments of 1988, which was published in the Federal Register on February 28, 1992, in the areas of proficiency testing and inspections for clinical laboratories. In responding to these comments, we accommodate, when possible, the Administration's regulatory reform initiative by reducing duplicative material, emphasizing outcome-oriented results, and simplifying regulations. In that regard, we also are streamlining our regulations in the areas of State exemption, and granting deemed status to laboratories accredited by an approved accreditation organization.     

Development and Implementation of an Effective Blind Proficiency Testing Program

Blind proficiency testing is ideal for testing crime laboratory personnel because the elements of analyst bias and anticipation are removed. However, sending proficiency tests through the laboratory system as real casework is difficult. The substantial challenges with preparing and administering blind tests may prevent laboratory managers from initiating blind testing. In 2015, the Harris County Institute of Forensic Sciences committed to improving its crime laboratory’s proficiency testing program by adding blind tests. The goal was to test the whole system, from evidence receipt to report release. With careful planning, trial-and-error, and ongoing assessment of available resources, not only was the program proven to be feasible, but there was also clear understanding of how to optimize our program. In this article, we share our experiences, lessons learned, and program details to assist other forensic service providers with developing their own blind testing programs, which would ultimately lead to improved quality assurance.     

Medicare, Medicaid and CLIA programs; revision of the laboratory regulations for the Medicare, Medicaid, and Clinical Laboratories Improvement Act of 1967 programs--HCFA. Final rule with comment period

This rule revises regulations for laboratories regulated under the Medicare, Medicaid and Clinical Laboratories Improvement Act of 1967 (CLIA '67) programs. The revisions recodify the regulations for these programs into a new part 493 in order to simplify administration and unify the health and safety requirements for all programs as much as possible. We will now have a single set of regulations for the three programs, with an additional subpart for the licensure procedures unique to the CLIA program. We are revising the regulations to remove outdated, obsolete and redundant requirements, make provision for new technologies and place increased reliance on outcome measures of performance. We provide for new uniform proficiency testing standards. We have also added requirements for additional specialties, such as clinical cytogenetics. We also implement the now and self-implementing provisions of the Clinical Laboratory Improvement Amendments of 1988 (CLIA '88).     

GSR2005—Continuity of the ENFSI Proficiency Test on Identification of GSR by SEM/EDX

Abstract:  Within the framework of the ENFSI Expert Working Group "Firearms" a proficiency test on the detection and identification of GSR by energy-dispersive scanning electron microanalysis (SEM/EDX) is carried out in a 2 years term. The latest test was performed in 2005/2006 and was denoted as GSR2005 . Seventy-five laboratories from 28 countries participated in this proficiency test and submitted in total 83 independent data-sets. The participating laboratories were requested to determine the total number of PbSbBa containing particles on a synthetic test sample following their own laboratory specific methods of automated GSR particle search and detection by SEM/EDX. Furthermore size and position of the detected particles had to be reported by the laboratories and were evaluated statistically. The results were compiled by means of z -scores according to the IUPAC and EURACHEM guidelines—assessing individual laboratory achievements (inter-laboratory) as well as intra-laboratory performance—and were compared to the results of the previous proficiency test run GSR2003 ( 1 ). The comparison shows that there is a noticeable improvement in the method's detection capability.     

GSR2005--continuity of the ENFSI proficiency test on identification of GSR by SEM/EDX.

Within the framework of the ENFSI Expert Working Group "Firearms" a proficiency test on the detection and identification of GSR by energy-dispersive scanning electron microanalysis (SEM/EDX) is carried out in a 2 years term. The latest test was performed in 2005/2006 and was denoted as GSR2005. Seventy-five laboratories from 28 countries participated in this proficiency test and submitted in total 83 independent data-sets. The participating laboratories were requested to determine the total number of PbSbBa containing particles on a synthetic test sample following their own laboratory specific methods of automated GSR particle search and detection by SEM/EDX. Furthermore size and position of the detected particles had to be reported by the laboratories and were evaluated statistically. The results were compiled by means of z-scores according to the IUPAC and EURACHEM guidelines-assessing individual laboratory achievements (inter-laboratory) as well as intra-laboratory performance-and were compared to the results of the previous proficiency test run GSR2003 (1). The comparison shows that there is a noticeable improvement in the method's detection capability.