How Good a Match is It? Putting Statistics into Forensic Firearms Identification

On February 14, 1929, gunmen working for Al Capone disguised themselves as police officers, entered the warehouse of a competing gang, and shot seven of their rivals dead. The St. Valentine’s Day Massacre is famous not only in the annals of gangland history, but also the history of forensic science. Capone denied involvement, but an early forensic scientist named Calvin Goddard linked bullets from the crime scene to Tommy guns found at the home of one of Capone’s men. Although the case never made it to trial—and Capone’s involvement was never proved in a court of law—media coverage introduced millions of readers to Goddard and his strange-looking microscope.

That microscope had a split screen that allowed Goddard to compare bullets or cartridge cases, the metal cases a gun ejects after firing a bullet, side by side. If markings on the bullets or cases matched, that indicated that they were fired from the same gun. Firearms examiners still use that same method today, but it has an important limitation: After visually comparing two bullets or cartridge cases, the examiner can offer an expert opinion as to whether they match. But they cannot express the strength of the evidence numerically, the way a DNA expert can when testifying about genetic evidence.

Now, a team of researchers at the National Institute of Standards and Technology (NIST) has developed a statistical approach for ballistic comparisons that may enable numerical testimony. While other research groups are also working on this problem, the advantages of the NIST approach include a low error rate in initial tests and that it is relatively easy to explain to a jury. The researchers described their approach in Forensic Science International.

When comparing two cartridge cases, the NIST method produces a numerical score that describes how similar they are. It also estimates the probability that random effects might cause a false positive match—a concept similar to match probabilities for DNA evidence.

“No scientific method has a zero error rate,” said John Song, a NIST mechanical engineer and the lead author of the study. “Our goal is to give the examiner a way to estimate the probability of this type of error so the jury can take that into account when deciding guilt or innocence.”

The new approach also seeks to transform firearm identification from a subjective method that depends on an examiner’s experience and judgment to one that is based on objective measurements. A landmark 2009 report from the National Academy of Sciences and a 2016 report from the President’s Council of Advisors on Science and Technology both called for research that would bring about this transformation.

The Theory Behind Forensic Ballistics

When a gun is fired, and the bullet blasts down the barrel, it encounters ridges and grooves that cause it to spin, increasing the accuracy of the shot. Those ridges dig into the soft metal of the bullet, leaving striations. At the same time that the bullet explodes forward, the cartridge case explodes backward with equal force against the mechanism that absorbs the recoil, called the breech face. This stamps an impression of the breech face into the soft metal at the base of the cartridge case, which is then ejected from the gun.

The theory behind firearm identification is that microscopic striations and impressions left on bullets and cartridge cases are unique, reproducible, and therefore, like “ballistic fingerprints” that can be used to identify a gun. If investigators recover bullets or cartridge cases from a crime scene, forensic examiners can test-fire a suspect’s gun to see if it produces ballistic fingerprints that match the evidence.

But bullets and cartridge cases that are fired from different guns might have similar markings, especially if the guns were consecutively manufactured. This raises the possibility of a false positive match, which can have serious consequences for the accused.

A Statistical Approach

In 2013, Song and his NIST colleagues developed an algorithm that compares three-dimensional surface scans of the breech face impressions on cartridge cases. Their method, called Congruent Matching Cells, or CMC, divides one of the scanned surfaces into a grid of cells, then searches the other surface for matching cells. The greater the number of matching cells, the more similar the two surfaces, and the more likely they are to have come from the same gun.

In their recent study, the researchers scanned 135 cartridge cases that were fired from 21 different 9-millimeter pistols. This produced 433 matching image pairs and 4,812 non-matching pairs. To make the test even more difficult, most of the pistols were consecutively manufactured.

The CMC algorithm classified all the pairs correctly. Furthermore, almost all the non-matching pairs had zero matching cells, with a handful having one or two due to random effects. All the matching pairs, on the other hand, had at least 18 matching cells. In other words, the matching and non-matching pairs fell into highly separated distributions based on the number of matching cells.

“That separation indicates that the probability of random effects causing a false positive match using the CMC method is very low,” said co-author and physicist Ted Vorburger.

A Better Way to Testify

Using well-established statistical methods, the authors built a model for estimating the likelihood that random effects would cause a false positive match. Using this method, a firearms expert would be able to testify about how closely the two cartridges match based on the number of matching cells, and also the probability of a random match, similar to the way forensic experts testify about DNA.

Although this study did not include enough test-fires to calculate realistic error rates for actual casework, the study has demonstrated the concept. “The next step is to scale up with much larger and more diverse datasets,” said Johannes Soons, a NIST mechanical engineer and co-author of the study.

With more diverse datasets, researchers will be able to create separate models for different types of guns and ammunition. That would make it possible to estimate random match rates for the various combinations that might be used in a crime.

Other groups of researchers are working on ways to express the strength of evidence numerically, not only for firearms but also fingerprints and other types of pattern evidence. Many of those efforts use machine learning and artificial intelligence-based algorithms to compare patterns in the evidence. But it can be difficult to explain how machine-learning algorithms work.

“The CMC method can be easily explained to a jury,” Song said. “It also appears to produce very low false positive error rates.”

---

Paper: J. Song, T.V. Vorburger, W. Chu, J. Yen, J.A. Soons, D.B. Ott, and N.F. Zhang. Estimating error rates for firearm evidence identifications in forensic science. Forensic Science International. Published online 13 December 2017. DOI: 10.1016/j.forsciint.2017.12.013