ChatGPT and cardiology: A close look at the strengths and weaknesses of AI chatbots
A team of cardiologists from Cleveland Clinic and Stanford University recently tested ChatGPT, the popular artificial intelligence (AI) model, to see if it could accurately answer questions about preventive cardiology and cardiovascular disease. The model performed well, only missing a handful of questions, and the researchers concluded that ChatGPT showed considerable potential.
Cleveland Clinic cardiologist Ashish Sarraju, MD, was the lead author of that study. Sarraju spoke to Cardiovascular Business about his team’s findings and how he thinks AI-powered chatbots may impact patient care.
“Our expectation going in was that it would perform well with factual questions that it could have obtained from its training data across the internet from reputable websites … but what was interesting to us was that it also provided reasonable answers to more nuanced questions, such as what an individual should do if they're on a statin and their cholesterol is not controlled,” Sarraju said. “That’s not a piece of information that we typically find very readily on the internet. Its responses to those questions were interesting to us and a little surprising.”
Of course, ChatGPT also got some things wrong. The AI model struggled with questions about exercise, for instance, and it missed another question because it was referencing outdated information on the availability of a medication. The availability error was especially troubling to researchers because it highlights that AI is only going to be as effective as its training data. If the model was fed information that is now out of date, there is a clear risk of it sharing outdated information with a patient.
“In a field like medicine, even a small limitation can be critical,” Sarraju said. “Even if the limitation seems small, on an individual patient level, one mistake could be life threatening.”
In this VIDEO clip, Cleveland Clinic cardiologist Ashish Sarraju, MD, gives an overview of ChatGPT's performance when asked questions about cardiology and patient care. He said some answers were very surprising, but others were incorrect.
Sarraju was quick to emphasize that ChatGPT was not specifically designed to be a healthcare resource. However, he expects people to keep exploring its potential in this field, and it will likely just get more and more effective as time goes on. As a clinician, Sarraju said he is more curious about how these models address potential mistakes and biases as they evolve than how well they will be able to answer straightforward questions.
Exploring ChatGPT’s potential in healthcare
Will cardiology practices be using ChatGPT in the future? Sarraju said there are still too many hurdles to get over before he can answer that question. He did note, however, that it makes sense to look for bottlenecks in the way care is delivered right now. If you can identify obvious pain points that physicians and their patients are experiencing on a regular basis, you can start to brainstorm about different ways that ChatGPT—and other AI chatbots—can provide real value.
“Involving clinicians and patients early on in that process could be very intriguing,” he said. “I look forward to seeing if that turns into reality.”
In this VIDEO clip, Cleveland Clinic cardiologist Ashish Sarraju, MD, says he looks forward to seeing how AI chatbots will improve as time goes on.
Of course, there is always potential that implementing AI models in the wrong way could have a negative effect on patient care. If an AI model can be trained to answer basic questions about healthcare, for example, does that mean patients may skip going to the doctor altogether and just rely on guidance from chatbots such as ChatGPT? Sarraju is not too concerned about that possibility quite yet.
“Patients look up information on the internet even now,” he said. “Many of our patients, especially when it comes to preventive cardiology, read a lot—they read studies, they read books. Some of them have even done deep dives on topics beyond what I’ve done. Maybe chatbots will change that, but I ultimately still think patients look for the ability to interact with a human being. I find it hard to imagine a chatbot is going to offer enough for patients to replace clinicians altogether.”
Patients will always appreciate shaking their doctor’s hand, Sarraju said, and looking into their eyes. Patients want to feel heard, feel understood and feel that their health is being taken seriously by a trained professional.
Michael has more than 18 years of experience as a professional writer and editor. He has written at length about cardiology, radiology, artificial intelligence and other key healthcare topics.