Cardiologists ask popular AI model ChatGPT to answer questions about cardiology
ChatGPT, the popular artificial intelligence (AI) model capable of writing text and engaging in detailed conversations, is starting to be used more and more in healthcare research. Can ChatGPT write academic studies? Can it help screen patients for diagnostic tests? Researchers are working around the clock, it seems, to see just how much value this AI-powered chatbot can provide to modern health systems.
A team of cardiologists from Cleveland Clinic and Stanford University, for example, aimed to see if ChatGPT could answer questions about preventive cardiology and cardiovascular disease (CVD). The team shared its findings in JAMA.[1]
“Using a chat-based interface, this AI model responds to complex queries interactively,” wrote first author Ashish Sarraju, MD, a specialist with Cleveland Clinic’s department of cardiovascular medicine, and colleagues. “This study qualitatively evaluated the appropriateness of AI model responses to simple, fundamental CVD prevention questions.”
In December 2022, Sarraju et al. created 25 cardiology-related questions that addressed “fundamental preventive concepts.” Each question was asked three times, and a team of preventive cardiology specialists reviewed the answers. Each set of three answers was graded as being “appropriate,” “inappropriate” or “unreliable”—the latter category was used when the answers were inconsistent.
Overall, the group found, ChatGPT provided appropriate responses to 21 of the 25 questions. The other four responses were all inappropriate.
What questions caused ChatGPT to struggle?
“The AI model responded to questions about exercise by firmly recommending both cardiovascular activity and lifting weights, which may be incorrect and potentially harmful for certain patients,” the authors wrote. “Responses about interpreting a low-density lipoprotein cholesterol level of 200 mg/dL lacked relevant details, including familial hypercholesterolemia and genetic considerations. Responses about inclisiran suggested that it is commercially unavailable.”
Reviewing their findings, the authors noted that ChatGPT showed potential as a way to “assist clinical workflows by augmenting patient education and patient-clinician communication around CVD prevention queries.” They also detailed the limitations of their analysis, including the fact that CVD is too big of a topic to accurately summarize in just 25 questions.
In addition, the team concluded, “the AI tool’s responses did not include references to evidence to support any statements.”
What is ChatGPT?
OpenAI, a San Francisco-based startup founded by Sam Altman, Elon Musk, Peter Thiel and others, developed the chat-based AI model using a mix of supervised learning and reinforcement learning.
OpenAI first shared ChatGPT with the general public in November 2022, and it became incredibly popular right away. Users experimented with the AI model in almost every way imaginable, and one report from January 2023 estimated this had boosted OpenAI’s value to approximately $29 billion.
While ChatGPT was designed primarily to mimic a human speaker engaged in a conversation, it can also create original art and music, perform parodies of existing art and write long essays. To say the world’s tech giants are interested in the potential of ChatGPT would be an understatement. Microsoft is believed to have invested as much as $10 billion in OpenAI, and Google is working on a similar AI model of its own, Bard.
Michael has more than 18 years of experience as a professional writer and editor. He has written at length about cardiology, radiology, artificial intelligence and other key healthcare topics.