Why it matters
In the United States (US), a high-profile scheme – CMS Star Ratings – publicly rates the quality of care at individual hospitals. The Centers for Medicare and Medicaid Services (CMS) assigns star ratings to hospitals, where one star corresponds to the worst care and five stars to the best. The scheme is intended to help patient choice, and ratings are used by hospitals when marketing their services.
Despite the widespread use of Star Ratings, concerns have been expressed about the design of the scheme. Those concerns extend to the choice of the measures used to determine the ratings, how ratings are calculated, and whether different types of hospitals can be compared. It may well be that using different methods for deriving the Star Ratings could produce different results – and those methods might be very reasonable alternatives to those currently used.
In this study, we looked at how hospital-level performance on individual measures is judged, and the approach taken to combining measures into an overall summary of quality.
Approach
We examined what the impact would be on hospital rankings if different choices were made in the calculation of star ratings.
Using publicly available data for 3,339 US hospitals from October 2020, we used plausible alternatives to the current methods used for calculating star ratings and re-calculated performance under these specifications. We focused on three technical decisions involved in calculating CMS star ratings: how to standardise each individual quality measure; how to group the individual quality measures; and how to weight different quality domains.
We looked at the three example changes in detail. We also used a simulation method to look at changes in hospital performance when the three aspects of the technical specification were varied simultaneously. We analysed the proportion of hospitals that received a different star rating under the alternative specification compared to the 2021 CMS specification.
What we found
On average, half (52%) of the hospitals would be assigned a different star rating when using alternatives to the current approach – most of these would move into an adjacent star rating, eg from four stars to five stars, or vice versa. Even minor changes to the methods led to substantial reclassification of performance.
Our study has shown that the CMS star ratings are very sensitive to how performance ratings are calculated. Changes in methods can result in substantial shifts in hospital ratings, particularly between adjacent categories. Many top-ranked hospitals under the 2021 CMS specification could lose their coveted five-star status under reasonable alternative approaches.
This questions the extent to which these ratings – or other similar composite summaries of hospital quality – should be relied on for patient choice or organisational reputation, and highlights the need for transparent justification and explanation of the reasons for choosing particular approaches to calculating ratings.
