In eDiscovery class today, Bruce Ellis Fein, the co-founder and legal director of Backstop LLP, a discovery software and services firm, visited our class and discussed the role of predictive coding in modern discovery.
A former associate at Sullivan & Cromwell, Mr. Fein observed that traditional doc review was not only a waste of paper, but also time and effort. Enter Backstop, which uses predictive coding methods to allow the user of the Backstop software to take information entered by human reviewers and generalize it to a larger group of documents, making the sorting process dramatically less taxing.
Among several interesting points in Mr. Fein’s presentation, Mr. Fein’s thoughts on recall and precision were especially interesting:
- Recall: Recall is the fraction of relevant documents that are identified as relevant by a search or review effort. According to Mr. Fein, recall is the most critical measure of accuracy. If you err in the measure of recall, you can get hammered with sanctions. That said, most documents that are responsive to a subpoena end up not being relevant to the case. In fact, one person could have a different opinion of a document’s relevancy on two different days. Human recall has been measured at around 50%. So if we can beat that even marginally using predictive coding, we will be better off. Computers can do this. Practice tip: That said, if you’re going to use predictive coding to analyze a corpus, get a written agreement regarding the recall level signed by opposing counsel. This will prevent any issues arising later in the discovery process. Further, predictive software is able to more accurately measure recall and gives an advantage to those who correctly utilize it.
- Precision: Precision is the fraction of documents identified as relevant by a search or review effort, that are in fact relevant. If a collection has low precision, this means there are a number of non-responsive documents categorized as responsive, demonstrating that the computer’s decisions are not very accurate yet. Mr. Fein suggests that if you are the producing party, avoid any discussion of precision because it will require you to to do more work in the form of review. If you are on the receiving end of production, you should insist on a specific level of precision.
Thanks to Mr. Fein for coming in and speaking to our class. It was an informative presentation and it sounds like Backstop has an exciting product and service; one of the few that truly uses predictive coding to make the ediscovery process more efficient.