Special Edition from the QAS Committee: Best Hospitals by Specialty National Rankings

Introduction

With the rapid growth of online users there are only a few resources that have maintained a reputation as honest brokers in the evaluation of quality. In the domain of healthcare delivery the US News and World Report (USNWR) rankings occupy a unique space and their annual evaluation of hospitals is widely recognized and coveted. Quantifying the quality of care delivered by a hospital, however, is no small feat. The goal of this brief communication is to provide a succinct summary of the approach used by the USNWR to generate its evaluations.

Eligibility

A hospital’s eligibility for inclusion in the USNWR rankings is based on size, academic affiliation, availability of certain key technologies, and participation in the American Hospital Association (AHA) Annual Survey of Hospitals. For specialty-specific rankings, eligibility depends on meeting a threshold number of Medicare discharges with complex care* over a three-year period (for the 2019 report this period will be from 2015-2017). Specialty-specific rankings that are reported are shown below, as well as the number of discharges (total vs. surgical) that are required for eligibility.

The determination that a hospitalization involved “complex care” is made based on the
diagnosis and procedure codes included in the hospitalization record. Relatively straightforward procedures (e.g., inguinal hernia repair) are not considered for analysis.

USNWR Approach

The approach taken by the USNWR is based loosely on a framework originally proposed by Donabedian et al in 1966, whereby measurements of quality of health care are considered in categories of structure, process, and outcome. A brief description of each of these categories is below.

Structure

Often termed the “invisible hand” of quality, structural components of health care pertain to characteristics of delivery systems such as staffing ratios, certification/credentialing, and physical facility. One of the most important of these is facility volume, based on the prevailing belief that (for some types of services) higher volume is correlated with higher quality. In assessing structural components of hospital quality, the USNWR relies on several data sources. The American Hospital Association (AHA) annual survey has a wealth of information regarding the profile of services provided by every domestic hospital. This survey is the basis for determining hospital technology (e.g. transplant services, robotic surgery, intensity-modulated radiation therapy). Volume of services provided is determined based on data received through the Centers for Medicare and Medicaid Services (CMS). Other determinants of structural quality of care include nurse:patient staffing ratios, trauma center status, and Nurse Magnet status.

Process

Processes are the discrete activities/choices that are made by clinicians in their provision of care. The relationship between each process (e.g. which perioperative antibiotic is used, and for what duration) is clearly related to outcomes. A process-oriented approach to measuring quality of care focuses on identifying the most important elements of care to track. Measurements of process are probably the most challenging to establish in a way that is meaningful. For this reason, the USNWR does not measure/analyze hospitals according to compliance with any traditional process measures. As a substitute for process measures the USNWR approach relies on a hospital’s reputation within its community. Reputation is assessed through a survey which is partially conducted through a third party (Doximity) and results from this survey over a three year period are used (e.g. for the 2018 report, data from 2016/2017/2018 were used).

Outcomes

Outcomes are the most highly intuitive mechanism by which to monitor and report quality of care. Fundamentally, anything that affected a patient’s health or well-being (in a way that is experienced by the patient) can be considered an outcome. Commonly measured outcomes include mortality, complications, readmissions, etc. Within the USNWR ranking, hospital outcomes reports are based primarily on risk-adjusted mortality and the data for these mortality analyses come entirely from CMS. Importantly, mortality is based on death within 30 days of admission, and therefore identifies post-discharge occurrences. Transfers into a hospital are excluded from mortality calculations. Patients who carry a diagnosis related to hospice/palliative care are included in these analyses. Starting with the 2019 report a new outcome has been added – “Discharging Patients to Home”. This outcome examines the extent to which hospitals maximize the likelihood that patients transition directly from the hospital to home.

Patient Safety (Process/Outcomes Hybrid)

Historically, the USNWR approach to measuring patient safety was modeled on the Agency for Healthcare Research and Quality’s (AHRQ) Patient Safety Indicators (PSI).

Four specific PSIs were included in the 2018 report:

• PSI 4: Death among surgical inpatients with serious treatable complications (AKA “Failure to Rescue”)
• PSI 9: Postoperative hemorrhage or hematoma
• PSI 11: Postoperative respiratory failure
• PSI 15: Unrecognized abdominopelvic accidental puncture/laceration rate

A description of the criteria used to assign the occurrence of a PSI based on CMS data can be found here. As of the 2019 report, PSIs are no longer a part of the USNWR methodology.

Patient Experience

Starting with the 2019 report, USNWR will use the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) as a quantitative representation of patient experience. The HCAHPS survey data are federally mandated for hospitals that provide inpatient care to Medicare patients and are publicly reported through CMS’s hospital compare website. In the 2019 USNWR rankings patient experience will have a 5% total weight.

Data Source, Time Lag

The USNWR reports rely on hospitalization data for many aspects of its report, especially outcomes and hospital volume. Data from CMS are used for this purpose, and it is therefore notable that the data predominantly reflect the care provided to older fee-for-service Medicare beneficiaries. Also, Medicare data are not released immediately (usually a two year delay), resulting in a notable gap between the period of analysis and the publication of the USNWR rankings (e.g. the 2018 report reflects data from 2014-2016). Therefore, any institution that implements immediate/effective quality improvement upon receiving the 2019 report would not see the full impact of these changes until 3-5 years later.

Weighting

With some minor exceptions, the weighting of the various components of the USNWR score is shown in the table below:

Two exceptions to this weighting system are worth noting. First, several specialties (ophthalmology, psychiatry, rehabilitation, rheumatology) provide services almost completely on an outpatient basis, and for these specialties reputation is the only driver of national rankings. Second, ratings for the specialty “Cardiology and Heart Surgery” include a 3% weight for public transparency. Performance in this domain (for this specialty) is evaluated based on a hospital’s voluntary reporting of quality data through the American College of Cardiology and the Society of Thoracic Surgery websites.

How Do I Improve My Hospital’s Ranking?

Patients, providers, hospitals, and payers are – to varying degrees – keenly aware of the results of the USNWR hospital rankings. Given the stakes involved, it is only reasonable for hospitals to focus on steps that can be taken to improve their measured quality. In addition to efforts to reduce preventable adverse events, some hospitals have focused on nursing magnet status, or targeted efforts to improve their reputation score through campaigns to increase their physicians’ participation in the Doximity or mailed surveys.

While no system of measurement is perfect, the strength of the USNWR approach is that it evaluates quality in different ways and across a wide range of clinical activities. Focused improvements in specific domains – even if clinically meaningful – may not directly influence these rankings. Also, given the quantity and duration of data that form the basis for these assessments, it is best to take a long view on quality. Any quality improvement process that is intended to influence an evaluation like the USNWR rankings should be broad-based and built to last.

Selected References

• USNWR Best Hospitals by Specialty
• Methodology U.S. News & World Report 2018-19 Best Hospitals: Specialty Rankings
• Patient Safety Indicators Technical Specifications
• American Hospital Association Hospital Statistics, 2018 Edition
• Donabedian A. Evaluating the quality of medical care. Milbank Memorial Fund Quarterly 1966; 44:166-206.
• Medicare.gov - Hospital Compare
• HCAHPS: Patients' Perspectives of Care Survey