Report 2583

The News

Since becoming a viral sensation and triggering a shift in the public discourse around artificial intelligence, ChatGPT has been put to the test by researchers across the U.S., who have tried to see how the OpenAI-powered chatbot would perform on high-level academic exams.

So far, ChatGPT has passed mock exams in fields like medicine and business. Here's a running list.

The Exams

Medical

A study found that ChatGPT performed "at or near the passing threshold" for all three parts of the United States Medical Licensing Exam, which is taken by doctors to receive their licensure. The bot was given dense multiple choice and open-ended questions, and achieved 60% accuracy across most of the exams, which is within the passing range.

"As AI becomes increasingly proficient, it will soon become ubiquitous, transforming clinical medicine across all healthcare sectors," the study stated.

Law

A team of University of Minnesota Law School professors found that ChatGPT would underperform the average law school student, but could skirt by with a passing grade on final exams in four courses. On its own, the bot would be a pretty mediocre law student, but it could assist students with their assignments, the researchers found.

Meanwhile, Bloomberg journalist Matthew S. Schwartz fed ChatGPT a law school take-home essay prompt, and it produced a "solid response."

In another study, ChatGPT didn't pass a multi-state bar exam practice test, scoring a 50.3%. But it did pass the Evidence and Torts sections, with researchers noting that it outperformed past AI models that have taken the bar.

Business

In a study titled "Would Chat GPT3 Get a Wharton MBA?", experts at the University of Pennsylvania's business school found that ChatGPT would score a B to B- grade on a final exam for an MBA core course, Operations
Management.

The study found it did an "amazing job" at the basics of the subject, but would sometimes make surprising mistakes in sixth grade-level math calculations.

Room for Disagreement

Despite concerns that ChatGPT could make testing obsolete, Danny Oppenheimer wrote in Times Higher Education that cheating has always existed in academia and has always been hard to police.

ChatGPT could make it easier for some students to cheat, but "but it doesn't fundamentally change the integrity dynamics in higher education," Oppenheimer argued.

He urged professors to consider changing rules and assessments so that using ChatGPT is allowed, forcing students to think about how they would solve a problem, rather than simply cheating to find the answer.