A Validated Risk Score for In-Hospital Mortality in Patients With Heart Failure From the American Heart Association Get With the Guidelines Program
Background— Effective risk stratification can inform clinical decision-making. Our objective was to derive and validate a risk score for in-hospital mortality in patients hospitalized with heart failure using American Heart Association Get With the Guidelines–Heart Failure (GWTG-HF) program data.
Methods and Results— A cohort of 39 783 patients admitted January 1, 2005, to June 26, 2007, to 198 hospitals participating in GWTG-HF was divided into derivation (70%, n=27 850) and validation (30%, n=11 933) samples. Multivariable logistic regression identified predictors of in-hospital mortality in the derivation sample from candidate demographic, medical history, and laboratory variables collected at admission. In-hospital mortality rate was 2.86% (n=1139). Age, systolic blood pressure, blood urea nitrogen, heart rate, sodium, chronic obstructive pulmonary disease, and nonblack race were predictive of in-hospital mortality. The model had good discrimination in the derivation and validation datasets (c-index, 0.75 in each). Effect estimates from the entire sample were used to generate a mortality risk score. The predicted probability of in-hospital mortality varied more than 24-fold across deciles (range, 0.4% to 9.7%) and corresponded with observed mortality rates. The model had the same operating characteristics among those with preserved and impaired left ventricular systolic function. The morality risk score can be calculated on the Web-based calculator available with the GWTG-HF data entry tool.
Conclusions— The GWTG-HF risk score uses commonly available clinical variables to predict in-hospital mortality and provides clinicians with a validated tool for risk stratification that is applicable to a broad spectrum of patients with heart failure, including those with preserved left ventricular systolic function.
Received January 29, 2009; accepted October 22, 2009.
In clinical practice, risk models may be useful to inform patient triage and treatment decisions. Patients hospitalized with heart failure (HF) provide a unique setting for such prognostic tools. HF is prevalent and an increasingly common reason for hospitalization in the United States and thus has a substantial public health and economic impact. Physicians often do not calibrate HF therapy to a patient’s risk for adverse outcomes,1 failing to deliver effective therapies to the highest risk patients, for whom the benefits of therapy are likely to be greatest. The ability to predict mortality risk could inform clinical decision-making, as a wide range of HF therapies are available, some of which are invasive and/or expensive. Objective prognostic information could guide the appropriate application of monitoring and treatment, potentially resulting in improvements in the quality of care delivered to and outcomes of patients hospitalized with HF.
Several predictive models have been reported for long-term outcomes in HF; however, these models have not been widely implemented in clinical practice. Although the cause of this phenomenon probably is multifactorial, the limitations of existing studies may play a role. Existing studies focus on clinical trial populations2–4 or single centers,5,6 which may not represent the general HF population, or the use of noncontemporary cohorts,7 which may have limited applicability to current practice. Other models have not been validated8,9; excluded patients with preserved left ventricular systolic function2; used complex clinical data that may not be available at the time of initial presentation to the hospital7,10; or place patients into categories of risk and do not permit a continuous characterization of individual mortality risk.9 The objective of this study was to derive and validate a predictive model for in-hospital mortality using readily available clinical data in a large contemporary and diverse population-based cohort of patients hospitalized with HF in almost 200 US hospitals. This information was then used to guide the development of a user-friendly and accessible risk score for in-hospital mortality for HF.
WHAT IS KNOWN
Although several predictive models have been reported for long-term outcomes in heart failure, these models are limited and have not been widely implemented in clinical practice.
The Get With the Guidelines–Heart Failure (GWTG-HF) risk score uses commonly available clinical variables to predict in-hospital mortality and provides clinicians with a validated tool for in-hospital mortality risk stratification that is applicable to a broad spectrum of heart failure patients, including those with preserved left ventricular systolic function.
WHAT THE STUDY ADDS
The GWTG-HF risk score can easily be calculated at the bedside and for hospitals participating in the GWTG-HF program; individual prediction of risk of in-hospital death is automatically calculated when admission data is entered into the GWTG data entry tool.
Application of the risk score could influence the quality of care provided to patients hospitalized with heart failure by informing clinical decision-making. Given that physicians’ failure to calibrate therapy to risk (the “risk-treatment paradox”) may be related to a failure to understand patient risk, this model may be useful in improving care for heart failure.
Data from the American Heart Association’s (AHA) Get With The Guidelines–Heart Failure module (GWTG-HF) was used for this analysis. GWTG is a voluntary quality initiative that uses a Web-based Patient Management Tool (PMT, Outcomes Sciences Inc, Cambridge, Mass) to collect clinical data, provide decision support, and provide real-time online reporting features. Participating hospitals are instructed to submit information on consecutive patients admitted to the hospital with new or worsening heart failure or patients who have significant HF symptoms during hospitalization to the program database. The diagnosis of HF is based on the treating clinician’s diagnosis. The GWTG-HF program and its component data elements have been described previously.11–13
The Patient Management Tool is intended for point-of-care and/or chart-review data collection by trained personnel and uses standardized data elements and definitions for information regarding presentation, in-hospital care, and outcomes. Data collected from the medical record include demographic and clinical characteristics, medical history, previous therapies, admission physical examination findings and laboratory values, in-hospital outcomes, and discharge disposition. All participating institutions were required to comply with local regulatory and privacy guidelines and to submit the GWTG protocol for review and approval by their institutional review board. Because data were used primarily at the local site for quality improvement, sites were granted a waiver of informed consent under the common rule.
Between January 2005 and June 2007, data were collected from 71 284 patients from 287 hospitals voluntarily participating in AHA GWTG-HF. Because not all participating hospitals submit complete data, hospitals that did not reliably report medical history or admission medications were excluded (85 hospitals with 24 752 patients), leaving 202 hospitals and 46 532 patients. Patients were excluded from analysis if they did not have a diagnosis of HF (n=1252), if they were transferred to a different acute care facility (n=1169), if the discharge date was invalid (n=10), or if data were missing for their discharge status (n=73) or left ventricular ejection fraction (LVEF) (n=4245). This resulted in a final study cohort of 39 783 patients from 198 hospitals. The study population was randomly divided into derivation (70%, n=27 850) and validation (30%, n=11 933) cohorts.
Candidate Predictor Variables
Potential predictor variables were selected based on prior literature, clinical relevance, and general availability at time of presentation. The following variable domains were considered: demographics (age, sex, race); comorbidities (history of atrial fibrillation or flutter, chronic obstructive pulmonary disease [COPD], diabetes, peripheral vascular disease, coronary artery disease [CAD], or myocardial infarction [MI], cerebrovascular disease, depression, and ischemic etiology of heart failure); admission laboratory information (blood urea nitrogen [BUN], serum sodium, serum hemoglobin, and serum creatinine); vital signs at presentation (systolic blood pressure and heart rate), and LV systolic dysfunction (LVEF ≤40%).
Patient characteristics were compared between the derivation and validation samples and between those who died and those who survived, using the χ2 test for categorical variables and Wilcoxon rank-sum test for continuous variables. Percentages were reported to describe categorical variables, mean±SDs were reported to describe the distribution of the normally distributed continuous variables, and medians with interquartile ranges were reported for continuous variables with skewed distributions. Multivariable logistic regression analysis using the generalized estimating equations (GEE) method was performed to assess candidate predictors of in-hospital mortality from a priori selected demographic, clinical, laboratory, and examination variables, based on existing literature and a statistically significant univariate relationship with mortality in the derivation sample.14 In the analytic cohort, sex was missing in 1.7%, medical history was missing in 2.1%, heart rate was missing in 3.2%, systolic blood pressure was missing in 3.2%, sodium was missing in 6.1%, BUN was missing in 4.4%, serum creatinine was missing in 4.6%, and hemoglobin was missing in 11.5%. Missing sex and medical history were imputed to the dominant category (male and no/absent, respectively). Missing laboratory and vital sign data were imputed to the corresponding median values. Continuous covariates were evaluated for the appropriateness of the linearity assumption using plots displaying the relationship of each variable with the log odds of mortality. When appropriate, knots were determined from the plots to create splines—variables that were modeled equivalent to piecewise linear continuous variables. Age, systolic blood pressure, and BUN were included as continuous variables because of their linear relationship with log odds of mortality, and the remaining continuous variables were modeled as splines with nonsignificant splints removed from the reduced model. LV systolic function was categorized as preserved (≥40%) or impaired (<40%). In addition, to assess for possible differences in the effect of individual risk factors, depending on LV function, we tested the significance of interactions of LVEF with atrial arrhythmias, peripheral vascular disease, COPD, CAD, or MI, ischemic etiology of HF, and the serum sodium splines.
The GEE approach with exchangeable working correlation matrix was used to account for within-hospital clustering.14 The reduced model included only the variables with probability values that were <0.05 in the multivariable analysis. The discriminatory ability of the model was assessed with a C-statistic, which represents the probability that the model would correctly predict that the risk was higher for a patient who had died versus those who survived. Calibration of the model was assessed by Hosmer-Lemeshow statistics and by comparing predicted versus observed probability plots using 25 groups. The reduced model was then validated by assessing model performance—discrimination and calibration in the validation cohort. The cohort was also stratified by LV function, and model performance was assessed in patients with preserved (EF ≥40%) and impaired (EF <40%) LV function.
The reduced model was refit using the entire study cohort to develop a risk score for predicting in-hospital mortality risk. A weighted integer was assigned to each independent predictor, based on the predictor’s coefficient in the reduced regression model. For each patient, the weighted integers were summed to obtain a total risk score with a range 0 to 100 points. The discrimination of the risk score was assessed with a C-statistic and the calibration by comparing predicted versus observed mortality rates.
All probability values are 2-sided, with P<0.05 considered statistically significant. All analyses were performed using SAS software (version 9.1, SAS Institute, Cary, NC). Outcome Sciences, Inc, serves as the data collection and coordination center for GWTG. The Duke Clinical Research Institute served as the data analysis center and has an agreement to analyze the aggregate deidentified data for research purposes.
The characteristics of patients enrolled in GWTG-HF and included in this study are shown in the Table. No significant differences in patient characteristics were present between the derivation and validation samples. In-hospital death occurred in 1139 (2.86%) patients. Those who died were more likely to have a prior heart failure diagnosis and an LVEF <40%. They were also older and more likely to have atrial fibrillation or flutter, CAD, COPD, anemia, and renal insufficiency but less likely to have hypertension, hyperlipidemia, or diabetes. (Table 1) Patients who died had a higher heart rate and lower blood pressure on admission. Additionally, those who died had higher serum creatinine and BUN and lower serum sodium and hemoglobin levels at admission.
Predictors of Mortality
In multivariable analysis, age, admission systolic blood pressure, admission BUN, admission serum sodium, admission heart rate, nonblack race, and the presence of COPD were independent predictors of in-hospital death. In both the derivation and validation data sets, the model had good discrimination (c-index=0.75 in each). Model calibration was also good in both as demonstrated by plots of predicted versus observed mortality (Figure 1) and the Hosmer-Lemeshow test (P=0.189 and P=0.604, respectively).
No significant interactions between LV systolic function and coexisting illnesses were present, indicating that the impact of individual risk factors on outcome was consistent regardless of LV function. When evaluated in subgroups of those with impaired and preserved LV systolic function, the model had good discrimination and calibration in both (c-index=0.75 and 0.74; Hosmer-Lemeshow test, P=0.888 and P=0.852, respectively).
A risk score was established using the 7 predictor variables identified in the multivariable model. The probability of in-hospital mortality can be estimated for an individual patient by summing points assigned to the value of each predictor for a total point score with the range of 0 to 100. (Figure 2) The risk score demonstrated good discrimination (c-statistic of 0.75) and calibration as demonstrated by the Hosmer-Lemeshow statistic (P=0.242) and plot of predicted versus observed mortality using 25 groups and the loess-smoothed calibration plot (Figure 3). Stratification of the entire cohort by deciles of predicted in-hospital mortality demonstrated the predicted risk of in-hospital death varied by >24-fold across deciles. The predicted in-hospital mortality rate in each decile was as follows: 0.4%, 0.7%, 1.0%, 1.3%, 1.6%, 2.1%, 2.5%, 3.3%, 4.6%, and 9.7%. The predicted mortality rates corresponded with the observed mortality rates in each decile (Figure 4).
Information on the GWTG-HF Risk Score is available in the Toolbox on the AHA GWTG website (http://www.americanheart.org/presenter.jhtml?identifier=3027533) and can easily be calculated at the bedside. For hospitals participating in the AHA GWTG-HF program, individual prediction of risk of in-hospital death is automatically calculated when admission data are entered into the GWTG data entry tool.
The American Heart Association GWTG-HF risk score reliably predicts in-hospital mortality in patients with preserved or impaired LV systolic function using 7 clinical factors routinely collected at the time of admission. Older age, low systolic blood pressure, elevated heart rate, low serum sodium, elevated BUN, presence of COPD, and nonblack race predicted an increased risk of death. Age, systolic blood pressure, and BUN contributed most substantially to the overall point score, whereas heart rate, presence of COPD, serum sodium, and nonblack race contributed relatively few points to the overall score. Additional factors known to be associated with mortality, including LV systolic function, depression, hemoglobin, and serum creatinine were considered but did not contribute to model discrimination beyond those variables included in the model.
Whereas serum sodium, heart rate, the presence of COPD, and nonblack race were independently associated with increased risk of mortality, they were less important with regard to the points contributed to the total GWTG-HF risk score. This highlights the fact that the presence of a significant independent association does not necessarily mean that a given factor will contribute substantially to a predictive model. For example, prior studies reported lower serum sodium as a strong predictor of mortality, yet the contribution of sodium to the overall risk score was relatively modest. Of note is the absence of LV systolic function in the model, which predicts outcomes in some HF risk models of long-term mortality.3,10 Some have argued that a prediction model for short-term mortality in acutely hospitalized patients should be independent of LV function because it may not be available at the time of initial clinical assessment.7 Because LV function did not contribute significantly to the overall risk model, the absence of LVEF data at the time of the initial patient assessment does not limit prognostication when using this model. Furthermore, the model had the same operating characteristics among patients with preserved and impaired LV systolic function, and therefore the risk score is applicable to a broad spectrum of HF patients.
Available risk stratification models for in-hospital mortality in patients hospitalized with HF have limitations. The Acute Decompensated Heart Failure National Registry (ADHERE) study identified BUN, serum creatinine, and systolic blood pressure as the best predictors of in-hospital mortality using classification and regression tree analysis.9 This model is appealing because it uses only 3 variables to classify patients as low, intermediate, or high risk. However, it does not allow more precise characterization of individual risk, and it does not include all variables that significantly inform outcomes. Specifically, multivariable logistic regression in the ADHERE cohort also identified heart rate and age as significant independent predictors of risk, but these variables did not inform the categorization of patients in the final risk tree.9 A prognostic model for in-hospital mortality from OPTIMIZE-HF was recently published with some overlapping variables but did not have a separate derivation and validation cohort and did not include data on admission BUN.8 Another study reported that the presence of cancer, systolic blood pressure ≤124 mm Hg, serum creatinine >1.4 mg/dL, BUN >37 mg/dL, serum sodium <136 mEq/L, and age >70 years predicted in-hospital mortality.5 However, this study only included patients from a single center, has not been validated in other cohorts, and incorporated only dichotomous variables, which, based on our data, is likely to obscure some of the information provided by these variables.
Other prognostic models for short-term mortality include the Enhanced Feedback for Effective Cardiac Treatment (EFFECT) and Outcomes of a Prospective Trial of Intravenous Milrinone for Exacerbations of Chronic Heart Failure (OPTIME-CHF) models.2,7 However, both were established for mortality 30 and 60 days after hospitalization, and the OPTIME-CHF model was derived from a clinical trial cohort limited to patients with LV systolic dysfunction, limiting its applicability to a wide range of community-based patients with HF. Additional predictive models for long-term mortality have been reported but are limited by derivation in clinical trial populations, the use of large numbers of variables, and inclusion of variables that are not typically assessed in clinical care, such as uric acid, peak oxygen consumption, or timing of diagnosis of HF.3,10,15
The GWTG-HF risk score for in-hospital mortality has several strengths and differs from other HF mortality risk models in many respects. It was derived using a contemporary cohort of population-based patients with diverse demographic characteristics and a wide range of comorbidities and included patients regardless of LV systolic function. Thus, this model is widely applicable. Additionally, it includes a relatively small number of variables routinely collected at the time of admission. Use of available clinical information makes the GWTG-HF score less susceptible to missing data. Several existing predictive models for long-term mortality in HF include more than 20 variables,3,10 some of which are not frequently obtained in the clinical care of HF. This renders the estimation of risk complex and difficult to incorporate into routine clinical practice. The GWTG-HF Risk Score can easily be calculated at the bedside and can be automatically calculated when admission data are entered into the GWTG data entry tool.
An accurate understanding of prognosis is fundamental to many clinical decisions in patients with HF. However, Hauptman et al16 found that fewer than one fifth of a cohort of clinicians caring for patients with HF believed they could accurately predict death, and, in fact, clinicians frequently incorrectly estimate risk in patients with HF.17,18 Generally, clinicians substantially over estimate risk, resulting in overutilization of critical care resources.18 Conversely, evidence demonstrates that medical therapy for HF is poorly calibrated to risk, with high-risk patients paradoxically receiving evidence-based medical therapy less often.1 The GWTG-HF risk score can be used at the point of care to quantify patient risk, thus facilitating patient triage and encouraging the use of evidence-based therapy in the highest-risk patients. The score could be used to increase the use of recommended medical therapy in high-risk patients and reduce resource utilization in those at low risk. Determining whether prospective application of this risk prediction score will favorably affect patient care and clinical outcomes should be the topic of future studies.
Certain factors should be considered in the interpretation of this study. First, because only information on in-hospital mortality was available, the overall mortality was relatively low compared with longer-term studies of populations with HF. However, the highest and lowest deciles of the risk score identified groups whose mortality varied by more than 24-fold. Second, the GWTG-HF risk score was not validated in a separate population. However, because of the large sample size, we were able to perform a robust validation in a randomly selected subpopulation, and actual event rates were closely correlated with predicted rates. Third, data were collected from the medical record and are thus dependent on the accuracy of documentation in the record. Finally, GWTG-HF is a voluntary registry, and participating hospitals may differ from nonparticipating hospitals. However, the registry includes a large number of hospitals and patients with diverse patient and hospital characteristics.
In summary, the validated GWTG-HF risk score uses routinely collected clinical data to predict the risk of in-hospital mortality for patients hospitalized with HF. Age, systolic blood pressure, and BUN are the admission variables most predictive of in-hospital mortality, with admission heart rate, serum sodium, presence of COPD, and nonblack race contributing modestly. Application of the risk score could influence the quality of care provided to patients hospitalized with HF by informing clinical decision-making.
Sources of Funding
GWTG-HF was supported in part by an unrestricted educational grant from GlaxoSmithKline. Dr Peterson is supported by an American Heart Association Pharmaceutical Roundtable Grant (067001N) and an American Heart Association Young Investigator Database Research Seed Grant. Dr Hernandez is supported by an American Heart Association Pharmaceutical Roundtable grant (0675060N).
Dr Hernandez has received research funding from Johnson & Johnson/Scios, Medtronic, and Merck and honoraria from Medtronic, Novartis, and AstraZeneca. Dr Fonarow has received research funding from the National Heart, Lung, and Blood Institute, honoraria from GlaxoSmithKline, Medtronic, and Novartis, and serves as a consultant for GlaxoSmithKline, Medtronic, and Novartis. Dr Masoudi has received research funding from Amgen, the National Heart, Lung, and Blood Institute, and the Agency for Healthcare Research and Quality; has served on advisory boards for Amgen and United Healthcare; and has contracts with the American College of Cardiology and the Oklahoma Foundation for Medical Quality.
Guest Editor for this article was Jack V. Tu, MD, PhD.
An abstract of this study was presented at the American Heart Association Quality of Care and Outcomes Research in Cardiovascular Disease and Stroke Conference 2008, Baltimore, Md.
Pocock SJ, Wang D, Pfeffer MA, Yusuf S, McMurray JJV, Swedberg KB, Ostergren J, Michelson EL, Pieper KS, Granger CB, on behalf of the CHARM Investigators. Predictors of mortality and morbidity in patients with chronic heart failure. Eur Heart J. 2006; 27: 65–75.
Senni M, Santilli G, Parrella P, De Maria R, Alari G, Berzuini C, Scuri M, Filippi A, Migliori M, Minetti B, Ferrazzi P, Gavazzi A. A novel prognostic index to determine the impact of cardiac conditions and co-morbidities on one-year outcome in patients with heart failure. Am J Cardiol. 2006; 98: 1076–1082.
Abraham WT, Fonarow GC, Albert NM, Stough WG, Gheorghiade M, Greenberg BH, O'Connor CM, Sun JL, Yancy CW, Young JB. Predictors of in-hospital mortality in patients hospitalized for heart failure: insights from the Organized Program to Initiate Lifesaving Treatment in Hospitalized Patients With Heart Failure (OPTIMIZE-HF). J Am Coll Cardiol. 2008; 52: 347–356.
Levy WC, Mozaffarian D, Linker DT, Sutradhar SC, Anker SD, Cropp AB, Anand I, Maggioni A, Burton P, Sullivan MD, Pitt B, Poole-Wilson PA, Mann DL, Packer M. The Seattle Heart Failure Model: prediction of survival in heart failure. Circulation. 2006; 113: 1424–1433.
LaBresh KA, Gliklich R, Liljestrand J, Peto R, Ellrodt AG. Using “Get With The Guidelines” to improve cardiovascular secondary prevention. Joint Commission Journal on Quality & Patient Safety. 2003; 29: 539–550.
Zugck C, Krnger C, Kell R, Korber S, Schellberg D, Kubler W, Haass M. Risk stratification in middle-aged patients with congestive heart failure: prospective comparison of the Heart Failure Survival Score (HFSS) and a simplified two-variable model. Eur J Heart Fail. 2001; 3: 577–585.