Science in Social Media
Debating the Effects of Readmission Penalties
The Hospital Readmissions Reduction Program (HRRP), which levies financial penalties for excess readmissions after select medical and surgical admissions, has been as widely critiqued as it has been celebrated. On one hand, there have been clinically and statistically significant declines in readmission rates since the announcement of the HRRP in 2012, with the largest improvements occurring in hospitals receiving the largest penalties.1,2 On the other hand, penalties in the program have been disproportionately allocated to academic medical centers and hospitals serving vulnerable populations, leading to concerns about widening disparities in care.3 There has also been mixed evidence that the HRRP has increased the use of emergency departments and observation stays.1,4
But of the myriad effects from the HRRP, none have been as controversial as the effect on patient mortality. There has long been concern that penalties for readmissions might create a perverse incentive whereby patients would be denied appropriate and necessary hospital care, which would increase a patient’s risk of death. Initial evidence exploring this relationship exhibited a weak but statistically significant inverse association between hospital 30-day mortality and readmission rates for patients with congestive heart failure.5 Detecting a meaningful association between reduced readmissions and increased mortality would represent the worst-case scenario for the HRRP.
What Is the Debate?
Two recent studies exemplify this controversial topic. In July 2017, a study led by Dr Kumar Dharmarajan and Dr Harlan Krumholz demonstrated that although 30-day mortality rates for congestive heart failure increased overall between 2008 and 2014, the trends in mortality between 2010 and 2012—when readmission declines were the greatest—were comparable with other periods.6 Moreover, the authors found a direct but weak correlation between a hospital’s change in 30-day readmission rate from 2008 to 2014 to its change in 30-day mortality rate. Through traditional publication channels, Dr Gregg Fonarow and Dr Clyde Yancy raised concerns about the focus on hospital-level analyses and suggested that assessments should be focused on all patients and hospitals given the potential for harm.7
A subsequent study led by Dr Ankur Gupta and Dr Fonarow published in January 2018 resurrected this debate with evidence suggesting the HRRP may have, in fact, led to increases in mortality.8 In a sample of Medicare beneficiaries linked to the Get With The Guidelines–Heart Failure registry data, the authors demonstrated significant declining trends in 30-day congestive heart failure readmissions after HRRP penalties were implemented, compared with trends before the HRRP. They also found that trends in 30-day congestive heart failure mortality significantly increased after HRRP, compared with trends before the HRRP. In response to this research, Krumholz et al9 highlighted potential limitations regarding the use of a sample of Medicare beneficiaries with heart failure and the decision to focus on patient-level analyses as opposed to the hospital level. Gupta et al10 rebutted that focusing the analysis on the patient level, rather than hospital level, limits the potential for ecological fallacy when comparing hospital-level results. Suffice it to say, the matter of readmissions and mortality has not been resolved.
Science in Social Media
Traditionally, discussing and debating scientific findings, such as the impact of HRRP, are limited to written responses in journals and discussions at academic conferences. Although both of these settings play an integral part in the scientific process, the timescale of these forums can span months and often engage a select few individuals chosen by editorial staff or conference planning committees. Conversely, social media platforms, like Twitter, provide a forum for all members of the scientific community to engage in real-time exchanges of ideas, questions, and critiques. However, these exchanges are often fleeting, and those not engaged in discussion or not participating in social media are left out.
Therefore, the purpose of this piece was to capture the rich debate that occurred after Circulation: Cardiovascular Quality and Outcomes Editor-in-Chief Dr Brahmajee Nallamothu summarized the debate surrounding the HRRP on February 15, 2018, and solicited feedback on this issue from the scientific community. Tweets are being presented in their original, unedited form, which may contain spelling and grammatical errors. Obviously, additional discussion has also occurred around this topic; the point of the selected tweets below is to be illustrative, and I recognize that I cannot be exhaustive.
@bnallamo (Dr. Brahmajee Nallamothu): “In @JAMACardio Letters to the Editor this week giants @hmkyale & @gcfmd reignite key debate about HRRP impact on mortality. Best piece to me involves discussion of ecological fallacy & importance of aggregate- vs unit-level effects in policy decisions…”
“So what matters more – that population trends show worse overall mortality after HRRP started or that same hospitals with lower readmission also had lower mortality? Would love policy gurus to weigh in...”
“e.g., if a policy to impose speed limits on all roads improves speeds & lowers crash risks on safer roads but this pushes more drivers to riskier roads where they drive faster & have worse crashes (with higher population mortality) – do we end the policy or fix riskier roads?”
Select responses to these tweets can be seen below, which I have categorized into 2 themes: (1) evaluating policy at the patient or group level and (2) assigning causality in policy. I believe both of these themes reflect the challenges in understanding the effects of health policy, including the HRRP.
Evaluating Policy at the Patient or Group Level
The first string of tweets is reflection on the methods of evaluating policy and at which level we should make causal inferences when evaluating health policy: the patient level or the group level. Underpinning this discussion are the epidemiological concepts of ecological fallacy and atomistic (or individualistic) fallacy. Ecological fallacy occurs when group-level or aggregate results are attributed to the individual level. Conversely, atomistic fallacy occurs when individual-level results are generalized to the group level.
@wbborden (Dr. William Borden): “The debate is a great one. Possible that lower readmits could be linked to lower mortality in one hospital and higher in another, depending on the hospital interventions and context. Clearly need monitoring together, possibly as a composite, and I think including length of stay”
@jasonwasfy (Dr. Jason Wasfy): “Agree with Bill. Also 30 day CHF mortality was increasing 2008–2010 (7.9% -> 8.1% - 8.4%) even before HRRP was created, much less implemented. Raises possibility that HRRP was not causally related to increased mortality. Trend could have been confounded by something else.”
@bnallamo: 2 great points. Underscore complexity of studying real-world policies in dynamic environments & endorse @hmkyale’s concerns. If pts are changing & some are succeeding, meaning of overall trends is unclear. Shouldn’t we figure out what successful hospitals are doing & share?”
@hmkyale (Dr. Harlan Krumholz): “The issue about learning about positive deviators is distinct from whether policy worked. There is heterogeneity in the success and we need to learn from it. But the core of this is that readmission dropped w/o evidence of harm in nationally representative data. @KDharmarajanMD”
@KDharmarajanMD (Dr. Kumar Dharmarajan): “@bnallamo risk of ecologic fallacy particularly concerning here when aggregate trends across hospitals not corroborated by within-hospital trends, which show the opposite (i.e. lower readmits track with lower mortality)”
@bnallamo: “Totally agree. I believe this is @hmkyale’s related point: if we see no uptick in mortality at those specific hospitals that decreased readmissions then lower readmissions by themselves are not leading to more deaths...”
In the case of the HRRP, the article by Dharmarajan et al suggests that the HRRP is not associated with increased mortality because hospitals with reduced readmissions also had reduced mortality. In this case, ecological fallacy would occur only when individual-level results differ from hospital-level results. The article by Gupta et al suggests that the HRRP is associated with increased likelihood of mortality at the patient level. However, these results might be subject to atomistic fallacy if findings are attributed to the group, that is, the HRRP increases mortality. Ultimately, in the face of contradictory evidence, we should be careful to avoid both ecological and atomistic fallacies when conducting evaluations at the patient and group level.
Assigning Causality to Health Policy
But even more important than the level at which we assign causality is whether or not we can assign causality at all. This next theme contains reflections on how we interpret findings from empirical evidence and assign causality to health policy. At the root of the debate are 2 fundamental concerns: (1) to what extent can unintended effects (eg, mortality) of a policy be attributed to the policy itself and (2) what is the burden of proof to demonstrate unintended effects and who should bear the burden of proof.
The first concern can be answered empirically. Dr Krumholz specifically highlights that mortality for patients with heart failure had been on the rise before both the HRRP and public reporting of readmissions and suggests other mechanisms by which increased heart failure mortality could be explained.
@hmkyale: “Mortality increase for patient’s w/heart failure predated the national hospital readmission reduction program or public reporting of readmission. And our look at hospitals was about mechanism…did hospitals that reduced readmission have bump in mortality. Ans: no. @KDharmarajan”
“The hospital readmission reduction program reduced readmissions w/o evidence of harm. 3 conditions. Readmits dropped for all. Only HF has increase in mortality but it predated the program; likely related to changing patterns of HF and where it is treated (inpatient v outpatient).”
The second concern cannot be answered empirically. Dr Fonarow suggested that caution is warranted and likely favored, given the stakes of the policy—the lives of patients.
@gcfmd (Dr. Gregg Fonarow): “If new highway policy designed to save money was resulting in 5 to 10K extra deaths of innocent drivers each year, should policymakers and congress act to address? Or just leave it to drivers, their families, and automakers to deal with the consequences? @NMHheartdoc @DLBHATTMD”
This position is commonly known as the precautionary principle, which suggests that the onus of proof should reside with researchers to prove that readmission reduction incentives do not cause unintended negative effects, which would ideally be demonstrated before implementation of the policy. This is antithetical to current practice, whereby health policies are implemented, and researchers must provide evidence that the policy causes harm for it to be reconsidered.
The debate on the effects of HRRP, both intended and unintended, is likely to continue as subsequent studies are conducted and published. Evaluating health policy is difficult because it often relies on retrospective, observational analysis of data at multiple levels, unlike experimental therapies, which rely on experimental study designs. Assigning causality to policy is even more challenging because the effects are often heterogeneous and underlying drivers are multicausal. Understanding policy evaluations and applying findings back into practice is even more challenging, which was evidenced in the following exchange:
@mikejohansen2 (Dr. Mike Johansen): “I have no idea what the effects of this intervention have been. I don’t have confidence in any of the evaluations. We simply need to study these complex interventions with much more rigor (especially when you expect meaningful, but small effect sizes).”
@MikeTPhD (Dr. Michael Thompson): “You’re absolutely right, it’s incredibly hard to tease out the effects of large scale policies. But the devil is always in the details. The real contributions will be evaluating local readmission reduction programs, which would hopefully inform the larger policy debate.”
As healthcare reform continues to shift toward value-based care, it will be critical that we learn from the difficulties presented by the HRRP, which I highlighted after the previous exchange.
@MikeTPhD (Dr. Michael Thompson): “Alternatively, we build the policy in such a way to enable rigorous prospective evaluation, rather than relying on retrospective observational studies.”
The complexity and constantly shifting landscape of health policy make Twitter an ideal setting for discussion. There is no doubt that traditional academic routes of peer review, editorial responses, and academic conference sessions will continue to be critical aspects of the scientific process. However, as this article demonstrates, social media platforms present a new and increasingly utilized forum to discuss and debate science in real time. In the case of Twitter, although individual tweets are limited in length, the depth and richness that can emerge from these discussions are unlimited.
Sources of Funding
Dr Thompson receives funding from Blue Cross Blue Shield of Michigan as Director of Analytics of the Michigan Value Collaborative. This funding source had no role in the preparation, review, or approval of the manuscript.
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.
- © 2018 American Heart Association, Inc.
- Desai NR,
- Ross JS,
- Kwon JY,
- Herrin J,
- Dharmarajan K,
- Bernheim SM,
- Krumholz HM,
- Horwitz LI
- Thompson MP,
- Waters TM,
- Kaplan CM,
- Cao Y,
- Bazzoli GJ
- Krumholz HM,
- Lin Z,
- Keenan PS,
- Chen J,
- Ross JS,
- Drye EE,
- Bernheim SM,
- Wang Y,
- Bradley EH,
- Han LF,
- Normand SL
- Fonarow GC,
- Yancy CW
- Gupta A,
- Allen LA,
- Bhatt DL,
- Cox M,
- DeVore AD,
- Heidenreich PA,
- Hernandez AF,
- Peterson ED,
- Matsouaka RA,
- Yancy CW,
- Fonarow GC
- Krumholz HM,
- Dharmarajan K,
- Normand ST
- Gupta A,
- Yancy CW,
- Fonarow GC