Internists, surgeons both overestimate risk; researcher calls for curriculum revamp

Residents in general surgery and internal medicine both vastly overestimated patients’ risk of postsurgical complications compared to a validated algorithm, a new study found.

In total, the residents overestimated risk by an average of 26 to 33 percent for all complications when compared to the American College of Surgeons’ National Surgical Quality Improvement Project (NSQIP) risk-adjustment model. For 91 percent of clinical estimates, the groups similarly overestimated risk, but internists had higher overestimates in 9 percent of cases.

All residents—76 from each specialty—answered an online assessment of seven real-life, complex clinical scenarios representing a diverse general surgery practice. They estimated the chance of any morbidity, mortality, surgical site infection, pneumonia and cardiac complications.

Their responses where then compared to the NSQIP model, an online calculator that processes 20 patient characteristics and can determine 30-day risks of 12 specific types of complications.

“Although the risk estimates for straightforward operations in otherwise healthy patients are available, patients with complex disease processes and high-risk premorbid states are conceptually more difficult to manage,” co-author Kevin Y. Pei, MD, from the department of surgery at Yale School of Medicine, and colleagues wrote in JAMA Surgery. “Providing these patients and their families with a clear, individualized, accurate estimate of their surgical risk that accounts for preoperative comorbid characteristics is necessary for them to make shared, informed treatment decisions.”

The authors pointed out reviewers have questioned the validity and accuracy of the NSQIP calculator, but it remains “one of the few available risk-adjusted models for surgical outcomes available for clinicians.” Even if the tool isn’t considered to be completely accurate, the wide margin between its predictions and those of the residents demonstrates the difficulty in estimating postsurgical risk in complex cases, Pei said on a JAMA podcast.

Pei said the motivation for the study came from his own observations at a quality improvement (QI) conference.

“One of our residents was pursuing a very complex case and there were very seasoned senior professors of surgery sitting in the audience, very confidently suggesting that the patient had a 99 percent mortality (risk) and therefore surgery was not offered to the patient, and so on and so forth,” he said. “It dawned on me that in this day and age where we have risk-adjusted models, maybe what we’re seeing at the QI conferences and effectively what we’re teaching our trainees may be off point and it’s not accurate.”

The study’s findings suggested Pei’s hypothesis could be correct. It also highlighted some different attitudes among residents from the two specialties. For instance, general surgery residents were significantly more confident with their responses and with not offering operations but less likely to discuss code status or consult risk-adjusted models.

“The surgery residents must feel more confident in their predictions because they have more familiarity with the procedure, the actual operation itself, whereas the medical residents, they know a lot about the comorbidities and how sick the patient and how complex the patient is but they lack the confidence of knowledge of the actual procedure,” Pei said. “That’s why I personally feel like the internal medicine residents were more likely to query a risk-adjusted calculator.”

Pei called for more collaboration in the academic setting between internal medicine and surgery. He also said instructors should rethink their risk-estimation methods.

“I think there’s a missed opportunity for us to design curriculum to bridge internal medicine and surgery, and bringing (internists) into the fold when we have a discussion about perioperative complications, because clearly our internal medicine colleagues add a lot of value in their understanding of the complex comorbidities and those that are chronic in nature,” Pei said.

“As faculty members, we should question our ability to predict risks. We shouldn’t be anecdotal about it when there are risk-adjusted models out there that not only is a great opportunity for education but I think it has direct impact on our patient care.”