Associated Incidents
“Rather than searching for lawmakers against a database of arrest photos, Clearview apparently searched its own shadily-assembled database of photos,” Snow said. “Clearview claim[ed] that images of the lawmakers were present in the company's massive repository of face scans. But what happens when police search for a person whose photo isn't in the database? How often will the system return a false match? Are the rates of error worse for people of color?”
It’s also atypical for an officer investigating a case to have a clear headshot of a suspect, like the ones Clearview presumably used as inputs for the lawmakers. Clearview’s tool is intended to be used in real-world situations, where photo quality, lighting, and other factors can skew the process, and should be tested as such, said Snow.
“If Clearview is so confident about its technology, it should subject its product to rigorous independent testing in real-life settings,” he said. “And it should give the public the right to decide whether the government is permitted to use its product at all.”
Ton-That said that the independent panel’s test was already diligent and thorough. “The Clearview test ran the same photos as the ACLU did, but against a database that was over 100,000 times larger,” he said, noting that in addition to searching for 535 US congressional leaders, his company’s technology was tried on state legislators from Texas and California. “With that higher level of difficulty, Clearview scored 100% following the ACLU standard.”
Facial recognition researchers expressed serious doubts about Clearview’s report. While the ACLU study was effective in demonstrating a facial recognition software’s deficiencies, it is by no means a sufficient methodology for definitively assessing the accuracy of a commercial tool like Clearview AI, Liz O’Sullivan, the technology director at the Surveillance Technology Oversight Project, told BuzzFeed News. O’Sullivan also questioned the panel’s claim that Clearview’s tech is accurate for “all demographic groups,” given the study’s test group of 834 state and federal lawmakers is not representative of all people or ethnicities.
Clearview CEO Ton-That disagreed with this assessment. “The rigors of the test have covered every demographic group that is represented in the general population and have shown Clearview’s accuracy when searching out of billions of photos,” he said.
Facial recognition and privacy researcher Adam Harvey told BuzzFeed News that it is impossible to evaluate the accuracy of the Clearview study without more insight into how it was conducted. “This document does not provide sufficient information to validate their claim,” he said. “It appears that no one in the panel has any prior experience with face recognition.”
Clearview’s report was signed off by a three-person panel, which included Jonathan Lippman, chief judge of the New York Court of Appeals from 2009 to 2015; Dr. Nicholas Cassimatis, an artificial intelligence academic and entrepreneur; and Aaron Renn, an urban analyst and former senior fellow at the Manhattan Institute, a conservative think tank. The panel determined whether the two top-ranked matches from Clearview search results showed the same person in the original search.
“In October 2019, the undersigned Panel conducted an independent accuracy test of Clearview AI, a new image-matching technology that functions as an Internet search engine for faces,” the report read.
None of the panelists appear to have any expertise in facial recognition. Lippman told BuzzFeed News that he "was introduced to Clearview by Richard Schwartz," one of the company’s cofounders whom he’s known since Schwartz’s time as editorial page editor of the New York Daily News. He said he was not paid for his work on the study.
“I assume I was approached because of my experience as a judge in government and criminal justice, and in looking at and weighing empirical evidence,” Lippman said.
Cassimatis told BuzzFeed News that he had worked in artificial intelligence for 20 years, and pointed to his work as a professor at Rensselaer Polytechnic Institute and as the former “head of Samsung's North American AI research.” He said he met Ton-That through a mutual friend and was selected “for my expertise in this field.” Cassimatis said he was not paid to work on the study.
Renn did not reply to a request for comment. Previously, Ton-That has said he and Schwartz had met during an event at the Manhattan Institute, where Renn served as a senior fellow until last year.
Clearview’s “ACLU” study isn’t the first time the company has touted the accuracy of its technology without much in the way of supporting materials or peer review. Last summer, the company told the Atlanta Police Department in marketing materials that its technology was 98.6% accurate in a test of 1 million faces — an accuracy rate higher than that of tools created by Google and Chinese tech giant Tencent. However, that claim, which was made using the University of Washington’s MegaFace facial recognition benchmark, was never independently verified by the university or a third party, the company later told BuzzFeed News. Clearview declined a request to make the results of this test available for review.
Since October, however, the company seems to have moved away from marketing the MegaFace number and gone forward with the 100% accuracy rating from the ACLU-based test done by its three-person panel. After the first news stories were published about the company last month, the company added a new section to its once sparse website called “Clearview Facts,” where it said that “an independent panel of experts rated Clearview 100% accurate across all demographic groups according to the ACLU’s facial recognition accuracy methodology.”
That did not sit well with the ACLU, which eventually lodged a complaint. On Jan. 28, Clearview removed the civil rights group’s name from its site, though it still claims that “an independent panel of experts reviewed and certified Clearview for accuracy and reliability.”
That may not be enough for the ACLU and Snow, who said that any proof of accuracy was “beside the point.”
“Clearview's technology gives government the unprecedented power to spy on us wherever we go — tracking our faces at protests, [Alcoholics Anonymous] meetings, church, and more,” he said. “Accurate or not, Clearview's technology in law enforcement hands will end privacy as we know it.”