Forensic science plays a critical role in the American criminal justice system. For decades, many feature-based fields of forensic science, such as firearm and toolmark identification, developed outside of the purview of the scientific community. Currently, black-box studies are used to assess the scientific validity of feature-based methods. The results of these studies are widely relied on by judges across the country. However, this reliance is misplaced. Black-box studies to date suffer from inappropriate sampling methods and high rates of missingness. Current black-box studies ignore both problems in arriving at the error rate estimates presented to courts. We explore the impact of each type of limitation using available data from black-box studies and court materials. We show that black-box studies rely on non-representative samples of examiners. Using a case study of a popular ballistics study, we find evidence that these non-representative samples may commit fewer errors than the wider population from which they came. We also find evidence that the missingness in black-box studies is non-ignorable. Using data from a recent latent print study, we show that ignoring this missingness likely results in systematic underestimates of error rates. Finally, we offer concrete steps to overcome these limitations.

翻译：法证科学在美国刑事司法系统中发挥着关键作用。几十年来,许多法证科学领域,如火器和工具标记识别,都是在科学界范围以外开发的。目前,黑盒研究被用来评估基于特征的方法的科学有效性。这些研究的结果被全国的法官广泛依赖。然而,这种依赖性是错误的。黑盒研究迄今受到不适当的取样方法和高失踪率的影响。目前的黑盒研究忽视了在得出提交法院的错误率估计方面的两个问题。我们利用黑盒研究和法院材料的现有数据来探索每一种限制的影响。我们显示黑盒研究依靠非代表性的检验者样本。我们利用流行弹道研究的案例研究发现,这些非代表性的样品的错误可能少于他们所来自的广大人口。我们还发现有证据表明,黑盒研究中的缺失是不可忽视的。我们利用最近一项潜伏印刷研究的数据,表明忽视这种缺失可能会导致系统低估错误率。最后,我们提出了克服这些限制的具体步骤。