Incident 131: Proctoring Algorithm in Online California Bar Exam Flagged an Unusually High Number of Alleged Cheaters

Description: The proctoring algorithm used in a California bar exam cited a third of thousands of applicants as cheaters, resulting in allegations where exam takers were instructed to prove otherwise without seeing their incriminating video evidence.

Suggested citation format

Hall, Patrick. (2020-12-04) Incident Number 131. in McGregor, S. (ed.) Artificial Intelligence Incident Database. Responsible AI Collaborative.

Incident Stats

Incident ID
131
Report Count
2
Incident Date
2020-12-04
Editors
Sean McGregor, Khoa Lam

Tools

New ReportNew ReportDiscoverDiscover

Incidents Reports

Remember when we said that the online bar exam’s “cheating” algorithm was going to be a problem? It already flagged Black and Brown folks for merely existing, but it didn’t stop there and managed to key in on all sorts of people as likely cheaters.

And this was kind of the point. The algorithm is designed to flag people for “suspicious” activity and then leave it to the humans to parse through the video to make sure it was a false positive. But maybe that’s a Herculean task….

Indeed, this video shows the California Committee of Bar Examiners revealing that of the 9,301 people who took the exam, 3,190 of them were flagged by the software. That’s 1 in 3!

Put aside whether or not software that flags a third of applicants is really doing its job — which is a good question! — when bar examiners are told that “it looks like 33 percent of the applicants cheated” the reaction should be “that’s obviously poppycock.” Instead California seems to be rolling with the “order to show cause” approach of asking a bunch, if not the bulk, of people flagged by the software to prove they weren’t cheating.

This is lunacy. The “Chapter 6” notices that California is sending around say stuff like “Facial view of your eyes was not within view of the camera for a prolonged period of time” which should be an invitation for the examiners to watch the goddamned video to see if it’s valid before bringing the applicant into it. This description is so generic that it’s clear the examiners never even bothered to look into these flags. Someone who looked at it and detected something suspicious would say “hey, the camera shows you repeatedly looking at your lap” not this vague garbage.

One notice sent around was “no audible sound was detected” which is a real trick because PEOPLE WEREN’T SUPPOSED TO MAKE NOISE. I get that they’re alleging that someone turned off their microphone or something, but that’s something that could be verified with further investigation. Why add unnecessary stress?

If California wasn’t ready to actually deal with the flags then they shouldn’t have gone with an online exam. It’s part of the deal. You make the exam online and that means more post-hoc proctoring. Farming it out to “prove us wrong” is utterly unacceptable.

Worse, reports are emerging on social media that applicants are being told that they have to respond to these vague allegations without seeing the video for themselves which sounds like an issue spotter. Hopefully these are just rumors because if the bar examiners are really trying to tag applicants based on a hastily composed algorithm that we already know creates false positives without access to the only relevant evidence… well that would be a new low.

Unfortunately, this exam cycle has taught me to not expect much.

California Bar Exam Flagged A THIRD Of Applicants As Cheating

More than one-third of those who took California’s first online bar exam in October were flagged for possible cheating, based on alerts sent by the test’s software, the state bar said.

Of the 9,301 people who took the entire exam, “we are currently reviewing 3,190 applicants that were flagged,” state bar official Tammy Campbell said during a Dec. 4 meeting of the California Bar’s Committee of Bar Examiners, according to a video recording. Test takers were flagged based on a number of rules infractions, including having food or electronic equipment like a cell phone, as well as behavior like gazing off-screen.

The inquiry, first reported by the ABA Journal, puts test takers flagged by the system at risk of being required to retake the exam, which is typically required before a law school graduate is allowed to work as an attorney in the state. Any review that substantiates widespread cheating also could give critics of bar exams in California and other states further ammunition to promote alternatives like diploma privilege, which allows law school grads to get licensed without taking an exam.

“We believe there were multiple factors that contributed to the number of flagged videos, including the unprecedented nature of this first-ever online remote bar exam and the large and diverse population who took it in California,” state bar spokeswoman Teresa Ruano said. “We will continue to refine and improve this process based on learnings from this first online exam.”

When asked about the large portion of those flagged, Ruano noted that some applicants brought prohibited items like food, drinks or digital clocks into their exam rooms—actions that would have typically been prohibited by a live proctor before test takers enter an in-person exam.

Lawyers for test takers who have been flagged say the widespread cheating allegations are preposterous and an overreaction to behaviors that exam administrators have determined are signs of potential violations of test rules.

Megan Zavieh, a Georgia-based lawyer with offices in California, represents more than two dozen exam takers contacted by the California Bar. She said some clients have received letters accusing them of moving their eyes away from camera range at the wrong moment.

“It’s not only, ‘no, I didn’t do that.’ It’s, ‘I have no idea what you’re talking about,’” Zavieh said.

The bar initially planned to review each of the possible cheating instances by Dec. 18, but Ruano said the review process has not yet been completed.

‘Frightened’ and ‘Angry’

California, like most larger states, shifted from an in-person test in July to an online exam in October because of the coronavirus pandemic. About 96% of California test takers took the test online.

Most of the states that held online tests contracted with Dallas-based ExamSoft to provide software for the remote exam. Artificial intelligence-enabled proctors gained access to test takers’ computers, including their web cams, to monitor behavior as they took the test.

Critics warned that remote proctoring could invade test takers’ privacy and could prove to be overly sensitive to signs of possible cheating.

Ruano said she couldn’t compare allegations of cheating in October to previous such allegations during in-person exams, given that “this information typically has been held confidential in the past.”

So-called “Chapter 6 Notices” of potential violations were sent to numerous test takers after ExamSoft video-file software flagged the conduct to the Bar, according to ABA Journal.

That spurred students to hire lawyers like Zavieh and Pasadena, Calif.-based ethics attorney Erin Joyce.

Joyce said her clients are concerned because of the impact the notices they’ve received might have on their ability to gain permanent or even provisional law licenses—even if they’re ultimately exonerated. “They’re frightened and they’re angry,” she said. “They put a lot of effort and expense into the test.”

Chapter 6 Notices contain allegations that are either “disputable” or “indisputable,” according to a list of frequently-asked questions the Bar posted on its website. A dozen types of alleged infractions are disputable and thus eligible for a hearing, according to a state bar Chapter 6 “Decisional Matrix.”

Indisputable allegations include those in which the test taker possessed notes or other study aids, or electronic devices like cell phones or digital watches. Disputable allegations, which can be contested by a hearing, include having radios or stereos, or “food or beverages, including but not limited to coffee and water,” in exam rooms during a remote-proctored exam.

Any action “taken or not taken based on potential irregularities is solely decided by the client,” said ExamSoft spokeswoman Nici Sandberg in a statement. “ExamSoft never plays a role in adjudicating instances of academic dishonesty.”

Tech Issues

California’s next bar exam, slated for Feb. 23-24, is also scheduled to take place primarily online.

In its Dec. 4 meeting, the Bar Examiners Committee also released data showing that 15% of the roughly 5,300 test takers who responded to a post-exam survey said they encountered technical issues on test day.

By contrast, the Bar panel said that ExamSoft reported that 8% of exam takers contacted the company about tech issues. The company successfully resolved 80% of the cases, the panel said.

Third of California Online Bar Exams Cited for Possible Cheating