Concordance of hospital ranks and category ratings when using the current technical specification of US Hospital Star Ratings and reasonable alternative specifications

Published in

Citation

Barclay ME, Dixon-Woods M, Lyratzopoulos G. Concordance of Hospital Ranks and Category Ratings Using the Current Technical Specification of US Hospital Star Ratings and Reasonable Alternative Specifications. JAMA Health Forum. 2022;3(5):e221006. doi:10.1001/jamahealthforum.2022.1006

  • 13 May 2022

Contributors

Why it matters

In the United States (US), a high-profile scheme – CMS Star Ratings – publicly rates the quality of care at individual hospitals. The Centers for Medicare and Medicaid Services (CMS) assigns star ratings to hospitals, where one star corresponds to the worst care and five stars to the best. The scheme is intended to help patient choice, and ratings are used by hospitals when marketing their services.

Despite the widespread use of Star Ratings, concerns have been expressed about the design of the scheme. Those concerns extend to the choice of the measures used to determine the ratings, how ratings are calculated, and whether different types of hospitals can be compared. It may well be that using different methods for deriving the Star Ratings could produce different results – and those methods might be very reasonable alternatives to those currently used.

In this study, we looked at how hospital-level performance on individual measures is judged, and the approach taken to combining measures into an overall summary of quality.

Approach

We examined what the impact would be on hospital rankings if different choices were made in the calculation of star ratings.

Using publicly available data for 3,339 US hospitals from October 2020, we used plausible alternatives to the current methods used for calculating star ratings and re-calculated performance under these specifications. We focused on three technical decisions involved in calculating CMS star ratings: how to standardise each individual quality measure; how to group the individual quality measures; and how to weight different quality domains.

We looked at the three example changes in detail. We also used a simulation method to look at changes in hospital performance when the three aspects of the technical specification were varied simultaneously. We analysed the proportion of hospitals that received a different star rating under the alternative specification compared to the 2021 CMS specification.

What we found

On average, half (52%) of the hospitals would be assigned a different star rating when using alternatives to the current approach – most of these would move into an adjacent star rating, eg from four stars to five stars, or vice versa. Even minor changes to the methods led to substantial reclassification of performance.

Our study has shown that the CMS star ratings are very sensitive to how performance ratings are calculated. Changes in methods can result in substantial shifts in hospital ratings, particularly between adjacent categories. Many top-ranked hospitals under the 2021 CMS specification could lose their coveted five-star status under reasonable alternative approaches.

This questions the extent to which these ratings – or other similar composite summaries of hospital quality – should be relied on for patient choice or organisational reputation, and highlights the need for transparent justification and explanation of the reasons for choosing particular approaches to calculating ratings.

Read the full article

Licensed under Creative Commons

These symbols show that the contents of this page are published under a Creative Commons licence called CC-BY-NC-ND 4.0.

It means that you’re free to reuse this work. In fact, we encourage it. We want our research to reach people who can help improve quality and safety in healthcare. But we do have a few rules:

  • Make sure you acknowledge The Healthcare Improvement Studies Institute (THIS Institute) as the creator and link back to this webpage.
  • You can’t sell this work for a fee, or use it for any activity that generates revenue without our permission.
  • Please don’t distribute a modified version to others without our permission.

You can read the fine print about the licence on the Creative Commons website. It’s meant to help us keep the integrity of our work and stay true to our values.

But ultimately we want our work to have impact. So if you’ve got a use in mind but you’re not sure it’s allowed, just ask us at enquiries@thisinstitute.cam.ac.uk