Question Posed
On February 2, 2016, Forensic Science Standards Board (FSSB) member José Almirall asked the Legal Resource Committee the following question:
I know that a basic tenet in the American legal system is that we would rather let free a guilty person than incarcerate an innocent one, if we had a choice between the two evils. Is there a mandate that we (scientific expert witnesses) behave in a certain way to ensure this?
The reason I ask is that my scientific colleagues . . . are proposing that we set up our statistical tests to favor making one error (allowing a guilty defendant go free) over the possibility of making the error that we will wrongly convict.
My argument is that scientists don’t get this far in the decision process and that guilt and innocence is the purview of the jury [and] judge . . . . Our job as scientists is to present the evidence, and either error is equally bad from the scientific viewpoint (this is my opinion). . . . Does my logic make legal sense?
In response to a request for an example, he added on February 21, that:
The Forensic Science Standards Board is considering a recommendation of the Chemistry Scientific Area Committee to place ASTM E2926 on NIST’s Registry of Approved Standards. A member of the Chemistry SAC wrote the following:
The statistical matching process currently in the standard needs further examination to determine if there are more appropriate statistical criteria that could be used. As implemented now, the matching process takes a default position that every comparison will result in a match unless there is sufficient evidence in the data to prove otherwise. A consequence of this approach is that as the uncertainty of the measurement method used for comparison increases, the false match rate increases. This means that the effect of measurement uncertainty in this procedure does not work in favor of the defendant, a principle that is central to our justice system. Two potential alternative statistical approaches to the current matching process include statistical equivalence testing, as approved by the FDA for establishing a match between the biological activity of a generic drug and an existing approved drug, and likelihood ratios (or Bayes factors), as used in forensic DNA analysis. Both of these types of procedures give the benefit of the doubt caused by measurement uncertainty to the defendant and effectively operate so that the chance of a false match decreases or stays constant as measurement uncertainty increases.
Introduction1
ASTM 2926-13 (Standard Test Method for Forensic Comparison of Glass Using Micro X-ray Fluorescence (µ-XRF) Spectrometry) includes statistical procedures for determining whether two specimens of glass are “distinguishable” (and hence “did not originate from the same source of broken glass”) or instead are “indistinguishable” (and hence possibly “originated from the same source of glass”).2 The part of the Standard Test Method on “interpretation of comparisons” (§ 10.7) is reproduced as an appendix to this memorandum. We have previously urged that the Standard be revised in several ways, including offering more explicit guidance on how analysts should describe the analytical results and their interpretations in their reports and testimony. This memorandum does not change our conclusion that important revisions should be made before adding this Standard to the OSAC Registry, and it is not intended to depart from our previous comments.
As we understand the question, you would like to know whether the procedures in § 10.7 are consistent with legal principles for allocating the risk of error in criminal cases. We believe that the principle that false convictions are more serious than false acquittals does not preclude the use of these procedures, but neither does it prevent statistical equivalence testing or the use of a likelihood ratio or Bayes factor to inform the judge or jury of the probative value of the spectroscopic results. None of these methods is foreclosed solely by the need to guard against false convictions at the expense of an increased proportion of false acquittals.
We do not address other questions, such as ascertaining (1) which approach would best meet the needs and goals of the legal system, (2) the extent to which the scientific literature establishes that the process for classifying samples is reliable and has known conditional error rates, and (3) the suitability of the match/no-match criteria of the test method for the purpose of investigation as opposed to proof at trial.
† Memorandum from the Legal Res. Comm., to the Org. of Sci. Area Comms. for Forensic Sci., Nat’l Inst. of Standards & Tech. (rev. ed. Oct. 7, 2016). The Harvard Law Review Forum is reprinting this memorandum to accompany the Forensic Commentary Series. The memorandum has been only lightly edited for typographic formatting, and citations may not necessarily conform to the Bluebook. Original pagination is indicated symbolically.