The viral AI chatbot ChatGPT recently took four law exams at the University of Minnesota and another exam at the University of Pennsylvania, Wharton School of Business, and the results did not disappoint.
Law professors at the University of Minnesota used the popular artificial intelligence (AI) software to have it answer exams in four courses: torts, employee benefits, taxation and aspects of constitutional law. Each exam had 95 multiple-choice and 12 essay questions.
The professors blindly graded the test answered by ChatGPT and university students' tests. Upon review, the AI bot garnered an average score of C+, which was lower than the students' average grade of B+.
The study also revealed that ChatGPT had performed better in the multiple-choice portion compared to the essay portion, where it became inconsistent.
“In writing essays, ChatGPT displayed a strong grasp of basic legal rules and had consistently solid organization and composition,” the professors explained. “However, it struggled to identify relevant issues and often only superficially applied rules to facts compared to real law students.”
With a C+ score, the chatbot is eligible for a law degree. However, this score would also mean academic probation based on the university’s standards.
Lead study author Jonathan Choi said that ChatGPT “would be a pretty mediocre law student”.
"The bigger potential for the profession here is that a lawyer could use ChatGPT to produce a rough first draft and make their practice that much more effective," he explained.
When it comes to using the AI bot in class, Choi shared that he and other professors have banned internet use during exams to prevent students from using the chatbot to cheat.
ChatGPT Acing Wharton’s MBA Final Exam
Meanwhile, a professor from the University of Pennsylvania also gave a grade of B to B- after taking an exam, but this time, it was for an MBA final test.
Operations Management Professor Christian Terwiesch surmised on his ChatGPT research that the bot provides correct and excellent explanations for "basic operations management and process analysis questions" but seems to struggle with basic arithmetic.
Terwiesch also noted that the chatbot needs help to perform simple calculations that human 6th graders can do and is still struggling with advanced process analysis questions. He elaborates that ChatGPT can improve its answer if a user hints about the correct answer.
One instance of ChatGPT learning the correct answer is when it initially failed to answer a question about queuing analysis but was able to solve it after Terwiesch added a hint to his prompt. When Terwiesch asked the bot the next day, ChatGPT could answer correctly.
Terwiesch gave a disclaimer explaining that this result cannot guarantee whether ChatGPT is "capable of learning from past feedback" or "just got lucky." He also shared that the quality of ChatGPT's answers seems to have a degree of randomness.
To his delight, Terwiesch found that ChatGPT can formulate humorous questions. The Wharton professor is even considering including the questions on future exams. However, he also shared that, even in questions, ChatGPT could be more foolproof as it still displayed subtle errors that made questions unanswerable.
Smart Scrutiny From the Academe
Despite initially strong support for the chatbot, Terwiesch warns others to be aware that the AI breakthrough still has limitations. He cautioned that ChatGPT gave incorrect answers to his questions and that it "made major mistakes in some fairly simple situations."
Terwiesch assures the public that students will still have to attend school and learn mathematics. They will also still need to know basic information, like the capital of a city, as that is foundational knowledge. "We are still far from an A+ for complex problems, and we still need a human in the loop," he added.
He recognizes the concern of fellow professors on the possibility of widespread cheating that K-12 students can do. Various educational institutions are currently on red alert over ChatGPT, with the New York City Department of Education at the forefront by outright banning it. Terwiesch even stated that ChatGPT is like a hack where students can push their exams to a friend with "average academic competence."
However, he argues that ChatGPT can become a trusty and "smart consultant." While it may generate incorrect but profound-seeming responses, it is precisely the "perfect training ground" for MBA students regarding discernment and critical thinking.
ChatGPT made its debut to the public last November. With its ability to generate coherent human-like answers, it has raised multiple ethical concerns in the educational sector since its release, being banned in certain universities and prompting companies to create AI detection software.