The National Institute of Standards and Technology (NIST) has launched the first large-scale “blackbox” study to test the accuracy of computer and mobile forensics. The stated official purpose of the study is “to identify and evaluate the body of scientific evidence that underlies the methods and practices used when conducting digital forensic examinations.” The NIST is seeking at least 150 individuals who conduct digital examinations on computer hard drives or mobile phones for the purposes of law enforcement, criminal defense, intelligence, corporate security, incident response, and other reasons.
The blackbox study is part of a larger NIST focus on evaluating the science underpinning other digital forensics methods, with other studies being initiated to examine DNA sourcing, firearms identification, and bitemark analysis. Such studies were strongly pushed by a National Academy of Sciences (NAS) report—“Strengthening Forensic Science in the United States: A Path Forward”—as a means of measuring the reliability of forensics methods that are subject to some degree on human judgement. Among other things, the report noted that the results of such a study could be used by courts and juries to weigh the results of digital forensics evidence.
Given that ediscovery, cybersecurity, and computer forensics practitioners have been cracking into computers and mobile devices for well over two decades, one would think that the government would have already taken a closer examination into whether they are doing it correctly. But, other than in the collection of tax revenues, U.S. government entities are not known for speedy action, as suggested by the 11-year delay between the NAS report recommendations and today’s NIST blackbox study.
In announcing the study, NIST digital forensics program head Barbara Guttman said, “We want to understand the state of the practice—can experts produce accurate and reliable information when examining data from a digital device?”
Really, you’re only getting around to asking that question now?
According to the NIST blackbox study announcement, the study isn’t designed to test “the proficiency of individual experts,” but aims to “measure the performance of the digital forensics community overall.” Conducted online, study participants will be given two hours to examine simulated digital evidence in the form of one virtual mobile phone and one virtual computer. Commonly used in digital forensics, these virtual devices are called “forensic images” and study participants will be able to “connect to them using the same software tools they use when working on real cases.” After their virtual examination, participants will be asked to answer relevant questions that might arise during the course of a real investigation. Enrollment is now open, and the study will be recruiting participants for about three months.
“In any forensics discipline, experts can encounter difficult cases,” the announcement states. “Fingerprints can be smudged and distorted. DNA can be degraded. One challenge with digital evidence is that it can often be difficult to find key bits of evidence among large volumes of data. Also, technology changes so quickly that it can be difficult to keep up.”
Guttman added that “Forensics experts can’t extract data perfectly in every possible scenario. Phones change. Apps change. The world just moves too fast.”
It certainly does, so let’s just hope that the NIST releases the results of this study while today’s digital forensics methods are still in practice.