ChatGPT-4, the latest version of Open AI’s massively popular dialogue-based artificial intelligence (AI) model, can provide appropriate answers to questions patients may have about hypertension, according to new research published in Hypertension.[1]
“Previous research has highlighted that ChatGPT provided largely appropriate responses to queries related to the prevention of cardiovascular diseases,” wrote first author Yuichiro Yano, MD, PhD, a researcher with the department of family medicine and community health at Duke University and the department of advanced epidemiology at Shiga University of Medical Science in Japan, and colleagues. “However, these questions were formulated by experts in preventive cardiology and grounded in established guidelines. The evaluators assessing GPT’s responses were cognizant of the fact that they originated from ChatGPT, introducing a possible bias.”
Yano et al. took a different approach with this analysis, instructing ChatGPT to come up with the questions it would be answering—“Please provide a list of frequently asked questions from patients with hypertension”—and then asking it to provide its information in both English and Japanese.
After interacting with ChatGPT several times in both English and Japanese, the researchers agreed on 20 final questions to include in their analysis. Three hypertension or nephrology specialists were then brought in to grade the AI model’s answers as either appropriate or inappropriate. The reviewers were also asked to decide if the English or Japanese answer was “superior in terms of accuracy, comprehensiveness, professionalism and level of detail.”
Overall, the three reviewers determined that ChatGPT-4’s answers were appropriate for 17 of the 20 questions. Also, they all agreed that the English answers were superior to the Japanese answers to 11 of the 20 questions.
“Notably, ChatGPT-4 demonstrated superior performance in English compared with Japanese, which is likely because OpenAI’s models, including GPT-4, predominantly train on English data sets, thereby resulting in more precise and nuanced English responses,” the authors wrote. “Furthermore, the intricacies and nuances of Japanese grammar are distinctively complex. Capturing these subtleties may pose challenges for natural language processing models.”
The study did have certain limitations. Twenty questions may not be enough to truly evaluate ChatGPT’s abilities, for instance, and the three reviewers did not have an established criteria for what makes an answer appropriate or inappropriate. The group also emphasized that ChatGPT was not specifically designed to be a healthcare resource and “should not supplant professional medical consultation, diagnosis or therapy.”
With all of those things in mind, however, ChatGPT’s performance was still impressive.
“The present study indicates that a prominent online dialogue-based AI model, when evaluated by certified hypertension and nephrology specialists, offers largely suitable answers to hypertension-related inquiries,” the authors concluded.
Click here to read the full study in Hypertension, a journal published by the American Heart Association.
Michael has more than 16 years of experience as a professional writer and editor. He has written at length about cardiology, radiology, artificial intelligence and other key healthcare topics.