Registries and Selection Bias
The Need for Accountability
Outcomes researchers share a strong interest in determining what is being achieved in practice. In the quest to reveal what is actually being done in medicine, we are often in a position to use existing data about clinical practice and patient experience to answer relevant questions, rather than having to generate data de novo. Patient registries, with their size, scope, and focus on patients who represent those seen in clinical practice, have become an increasingly common source of data for outcomes researchers.
There is much to like about patient registries. These organized collections of data about clinical practice are a vehicle by which we can assemble massive amounts of information about patients and use it to assess quality of care, identify patterns of care, facilitate real-world research, and improve performance. In cardiovascular medicine, registries capture information about patients with acute coronary syndromes and heart failure as well as those who undergo procedures related to cardiac catheterization, percutaneous coronary intervention, carotid stents, and internal cardioverter defibrillators. Federal agencies are turning to registries to provide information about quality and produce insights about how to best practice medicine through an emphasis on comparative effectiveness research.
Outcomes research articles based on registry data are numerous, including those from NRMI, CRUSADE, GWTG-HF, GWTG-Stroke, NCDR ACTION, NCDR Cath PCI, NCDR ICD, and others. In our first year, Circulation: Cardiovascular Quality and Outcomes published several high-profile articles based on registries.1–4
Amid the enthusiasm for registries, Ferreira-Gonzalez et al5 publish an article in this issue of the journal that sends up a warning flare about the quality of registry data. These investigators used MASCARA, a Spanish registry of patients with an acute coronary syndrome, to investigate how well individual hospitals were able to comply with the instructions to enroll consecutive patients. Of the 50 hospitals participating in MASCARA, only 17 agreed to undergo scrutiny about how well they followed the enrollment directions of the registry. These 17 hospitals had enrolled 3265 patients in the registry. The quality audit revealed that, in the same time period, another 1439 eligible patients were not enrolled. Compared with enrolled patients, these patients had higher risk and received poorer quality of care, and had a 3-fold higher in-hospital mortality.
This finding is notable for many reasons. A number of registries delegate responsibility for patient selection and data collection to sites and individuals without experience and training in research methods. They are commonly operated as part of the clinical quality activities of an organization, rather than as observational studies. As a result, many individuals with registry-related responsibilities may also have a considerable amount of unrelated responsibility that impacts their time and ability to enroll all eligible patients. Oversight of such data collection efforts may also be minimal because of cost considerations. Consequently, there has been some concern about data quality and some registries have implemented robust quality assurance activities. These activities, however, have focused predominately on data accuracy and reliability.
Although data accuracy and reliability are important to the registry, proper patient selection is at least as important. Ferreira-Gonzalez et al demonstrate that a substantial proportion of eligible patients for MASCARA were not enrolled and that those patients had a markedly different profile, biasing the cohort toward a healthier group that received higher quality care. Our concern about the generalizability of clinical trial populations that often do not enroll typical patients and seem to systematically exclude those with important age, sex, race, and comorbidity characteristics is not considered to be a problem with registries. In fact, a frequently touted strength of registries is their ability to portray the experience of typical patients—individuals who were not screened out because of trial-specific inclusion or exclusion criteria. The patient selection bias that is illuminated in this study raises a troubling concern that registries—even those designed to enroll consecutive patients—are not immune to generalizability issues.
The consequences of selection bias on the results of such registry-based research are profound, including risk prediction scores that misrepresent risk in patients who are excluded, or selection of risk factors that do not apply to the population broadly. Virtually any research question might be affected in ways that are hard to predict.
The findings also have important implications for quality assessment. The disparity in quality of care between those selected and not selected indicates the potential for misrepresentation of hospital performance. Ironically, if there is variability in adherence to the registry design, institutions that adhere more closely to the rules may appear to perform worse.
There may even be inherent issues in the methodology used by Ferreira-Gonzalez et al. Although for many conditions there is a degree of uncertainty, leading to disagreement, about whether some patients meet criteria, the pertinent question that remains is whether the instructions are being followed in a way that is reproducible. The magnitude of the discrepancy between the registry enrollment at the hospitals and the quality audit conducted by Ferreira-Gonzalez and colleagues suggests that this is more than a minor disagreement.
How widespread is the problem? Although it is conceivable that the issues with MASCARA were peculiar to that registry, the mere possibility that selection bias is widespread among registries should initiate immediate efforts to assess the extent of the problem. Because every registry is unique in its implementation, variability in the selection bias among registries is likely. As we embark on a research era in which “real-world” registries are assuming an increasingly prominent role, understanding the prevalence and magnitude of such potential selection biases is critically important.
Registries have made the leap from second-tier status in American medicine, used locally by institutions that collected data for the purpose of improving care, to their current place at center stage. Registry data are poised to serve as an instrumental means of producing knowledge that will guide practice, as well as a source of information about relative performance, with potential revenue consequences. However, with the increasing prominence and utility comes increasing responsibility. In the future, registries will need to demonstrate their performance with respect to producing high-quality data. Success in this effort is critical to the achievement of their immense promise.
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.
Halim SA, Mulgund J, Chen AY, Roe MT, Peterson ED, Gibler WB, Ohman EM, Newby LK. Use of guidelines-recommended management and outcomes among women and men with low-level troponin elevation: insights from CRUSADE. Circ Cardiovasc Qual Outcomes. 2009; 2: 199–206.
Mahoney EM, Wang K, Cohen DJ, Hirsch AT, Alberts MJ, Eagle K, Mosse F, Jackson JD, Steg PG, Bhatt DL, on behalf of the REACH Registry Investigators. One-year costs in patients with a history of or at risk for atherothrombosis in the United States. Circ Cardiovasc Qual Outcomes. 2008; 1: 38–45.
Peterson ED, Roe MT, Rumsfeld JS, Shaw RE, Brindis RG, Fonarow GC, Cannon CP. A call to ACTION (Acute Coronary Treatment and Intervention Outcomes Network): a national effort to promote timely clinical feedback and support continuous quality improvement for acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2009; 2: 491–99.
Roe MT, Chen AY, Cannon CP, Rao S, Rumsfeld JS, Magid DJ, Brindis RG, Klein LW, Gibler WB, Ohman EM, Peterson ED, on behalf of the CRUSADE and ACTION–GWTG Registry Participants. Temporal changes in the use of drug-eluting stents for patients with non–ST-segment–elevation myocardial infarction undergoing percutaneous coronary intervention from 2006 to 2008: results from the Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA guidelines (CRUSADE) and Acute Coronary Treatment and Intervention Outcomes Network–Get With The Guidelines (ACTION–GWTG) Registries. Circ Cardiovasc Qual Outcomes. 2009; 2: 414–20.
Ferreira-Gonzalez I, Marsal JR, Mitjavila F, Parada A, Ribera A, Cascant P, Soriano N, Sánchez PL, Arós F, Heras M, Bueno H, Marrugat J, Cuñat J, Civeira E, Permanyer-Miralda G. Patient registries of acute coronary syndrome: assessing or biasing the clinical real world data? Circ Cardiovasc Qual Outcomes. 2009; 2: 540–547.