ChatGPT passes interventional cardiology certification exam
ChatGPT, OpenAI’s massively popular artificial intelligence (AI) chatbot, passed an interventional cardiology certification exam provided by the American College of Cardiology (ACC) and Society of Cardiovascular Angiography and Interventions (SCAI), according to a new research letter in JACC: Cardiovascular Interventions.[1]
However, fellows taking the same exam consistently achieved better scores.
“This study demonstrates AI’s capacity to answer multiple-choice questions from the field of interventional cardiology. However, it cautions against relying on ChatGPT-4 for decision-making within the field, as it often provides incorrect answers,” wrote senior author Emmanouil S. Brilakis, MD, PhD, director of the Center for Complex Coronary Interventions at the Minneapolis Heart Institute, and colleagues.
Brilakis et al. asked ChatGPT-4 to answer the simulated exam questions in two formats: multiple choice with forced justification (MC-J), which required answers to be explained, and multiple choice without forced justification (MC-NJ), which did not. The large language model’s score was better when answering in the MC-J format (76.7%) than when answering in the MC-NJ format (61.7%). It answered five out of six image-based questions correctly, but was unable to handle video-based questions.
When faced with more challenging questions that fewer than 75% of fellows answered correctly, ChatGPT did quite well, providing the right answer 75% of the time. On the questions that were easier for fellows, leading to a correct answer more than 75% of the time, ChatGPT got 77.3% correct. All in all, the average number of fellows answering questions correctly was 82.2%, higher than ChatGPT’s MC-J performance of 76.7%.
Brilakis and colleagues also emphasized the fact that the AI chatbot appeared to do worse when asked to answer the ACC/SCAI questions again. Its MC-J performance dropped from 76.7% to 65%.
“Specifically, three previously incorrect answers were corrected, whereas 10 previously correct answers were answered incorrectly upon retesting,” the authors wrote.
ChatGPT update promises key improvements
In April, OpenAI announced the launch of a new GPT-4 Turbo update with “improved capabilities in writing, math, logical reasoning and coding.” Would this latest version have performed even better than ChatGPT? Researchers are likely already working toward an answer.
Paid ChatGPT users now have access to GPT-4 Turbo.
Click here to read the full research letter.
Michael has more than 18 years of experience as a professional writer and editor. He has written at length about cardiology, radiology, artificial intelligence and other key healthcare topics.