Some hospital report cards don’t make the grade

Half of 10 popular reports that rank hospitals on quality earned poor grades in an evaluation of their scoring methods.

The Health Association of New York State (HANYS) assessed the quality of published report cards from organizations such as the Joint Commission, Truven Analytics, Leapfrog Group and Consumer Reports that the organizations claim are designed to help consumers make informed decisions.

“[T]o provide the greatest value to the public and providers, the information included in public report cards should be based on a standard set of measures that have proven to be valid, reliable, and evidence-based,” HANYS authors wrote. “A standardized method of evaluating hospital quality becomes even more critical as publicly available reports garner renewed attention from patients and providers, as well as payers and purchasers of health care.”

The “Report on Report Cards” applied nine criteria to assess the report cards. HANYS based the criteria on academic research and recommendations from a group convened by the National Quality Forum (NQF). The criteria were transparent methodology; evidence-based measures; measure alignment; appropriate data sources; most current data; risk-adjusted data; data quality; consistent data; and hospitals given an opportunity to preview the results.

Some organizations used administrative data rather than clinical data, for instance, which the HANYS group considered limited. Others sometimes culled data that was two years behind the reporting date, which may not have captured more recent improvements achieved through quality efforts.

The authors pointed out that some report generators potentially profit from the endeavors: Truven Analytics sells products to hospitals; Leapfrog charges licensing fees to use its logo; and Consumer Reports and U.S. News and World Report count on subscriptions and sales.

Report cards from government and accrediting organizations tended to best meet the criteria. “In general, the report cards receiving lower scores relied heavily on administrative claims data and/or unvalidated survey data; gathered comparative data points from different sources and time frames to generate a composite score or ranking; and/or did not use measures aligned with NQF, CMS [Centers for Medicare & Medicaid Services], or national accrediting organizations.”

The report assigned stars to each report card, with the highest possible score of three stars to those that met all or almost all of the criteria. The lowest score was a half star, which meant meeting one, partially some or none of the criteria.

Four report cards scored three stars: the Joint Commission Quality Check; Department of Health (DOH) Hospital-Acquired Infection Report; CMS Hospital Compare; and DOH Hospital Profile Quality Section. Niagara Health Quality Coalition New York State Hospital Report Card was the sole report card to get two stars.

Leapfrog Hospital Safety Score, Truven Health Analytics 100 Top Hospitals, Healthgrades America’s Best Hospitals and Consumer Reports Hospital Safety Ratings each earned one star. U.S. New and World Report received a half star.