ChatGPT answers straightforward cardiology questions, but struggles with complex cases
ChatGPT, the advanced artificial intelligence (AI) model capable of writing text and engaging in detailed conversations, can accurately answer simple questions related to cardiology and cardiovascular health, according to new research out of the Netherlands. When asked about more complex cases, however, ChatGPT’s cardiovascular knowledge still falls short.
The study, published in medRxiv and is still awaiting a full peer review, involved asking ChatGPT a series of “cardiovascular trivia questions” and “clinical case vignettes.”[1] Determining how well the AI model can answer these questions at this early stage can help preview its potential impact in the years ahead.
“The true potential of AI in healthcare lies in its ability to transform clinical workflows by automating routine and time-consuming tasks,” wrote co-authors Ralf E. Harskamp, MD, PhD, and Lukas De Clercq, two researchers from the University of Amsterdam. “This not only improves efficiency and productivity, but also frees up healthcare providers to focus on more critical and complex tasks, where AI can serve as a valuable support tool. By combining human expertise and computational power, AI has the potential to greatly enhance the delivery of healthcare services and improve patient outcomes.”
Overall, ChatGPT provided correct answers to 74% of the 50 straightforward trivia questions it was asked. Its accuracy varied from one topic to the next, achieving scores of 80% for coronary artery disease questions, 70% for atrial fibrillation questions and 60% for risk management questions.
For the longer case vignettes, ChatGPT was able to provide advice to a theoretical patient that was comparable to the advice a physician may give in that same scenario.
“We noted one major inconsistency, in which ChatGPT advised to use of thrombolytic agents as secondary preventative medications in a patients with a prior myocardial infarction (MI),” the authors wrote. “While there is a place for thrombolytics in the acute phase (in settings where primary percutaneous coronary intervention is not available), these medications have no place in chronic management of patients post MI.”
However, ChatGPT did struggle when asked to provide an expert-level consultation. In these instances, the AI model did still provide effective answers 50% of the time, but its other answers were either incomplete, inconclusive or just flat-out incorrect.
Reviewing these findings, the two authors noted that ChatGPT has shown significant potential as a tool for processing large amounts of healthcare data “quickly and accurately,” identifying illnesses based on patient symptoms, answering patient questions, advising physicians and more. Additional research is still required, however, as clinicians continue to learn more about this rapidly evolving technology.
“Multidisciplinary research is needed to figure out what the right place for ChatGPT could be in a community that is quickly starting to embrace this new form of technology,” the duo concluded. “It is important to consider the risks associated with the non-determinism, non-transparency, and increased centralization of large language models like ChatGPT and strive for greater accountability in their development and use, particularly in the critical and sensitive context of healthcare."
Read the full study—which, again, has not been peer reviewed—here.
What is ChatGPT? A quick refresher on one of the hottest topics in healthcare
OpenAI, a San Francisco-based startup founded by Sam Altman, Elon Musk, Peter Thiel and others, developed ChatGPT using a mix of supervised learning and reinforcement learning. While it was designed primarily to mimic a human speaker engaged in a conversation, ChatGPT can also create original art and music, perform parodies of existing art and write long essays.
OpenAI first shared ChatGPT with the general public in November 2022, and it became incredibly popular right away. Users experimented with the AI model in almost every way imaginable, and one report from January 2023 estimated this had boosted OpenAI’s value to approximately $29 billion.
The chat-based AI model has continued to grow in popularity, with specialists in every corner of the healthcare sector working to explore its full potential. Prior coverage of how ChatGPT may impact cardiology is available here and here.
Michael has more than 18 years of experience as a professional writer and editor. He has written at length about cardiology, radiology, artificial intelligence and other key healthcare topics.