National Survey of UK Consultant Surgeons’ Opinions on Surgeon-Specific Mortality Data in Cardiothoracic Surgery
Background—In the United Kingdom, cardiothoracic surgeons have led the outcome reporting revolution seen over the last 20 years. The objective of this survey was to assess cardiothoracic surgeons’ opinions on the topic, with the aim of guiding future debate and policy making for all subspecialties.
Methods and Results—A questionnaire was developed using interviews with experts in the field. In January 2015, the survey was sent out to all consultant cardiothoracic surgeons in the United Kingdom (n=361). Logistic regression, bivariate correlation, and the χ2 test were used to assess whether there was a relationship between answers and demographic variables. Free-text responses were analyzed using the grounded theory approach. The response rate was 73% (n=264). The majority of respondents (58.1% oppose, 34.1% favor, and 7.8% neither) oppose the public release of surgeon-specific mortality data and associate it with several adverse consequences. These include risk-averse behavior, gaming of data, and misinterpretation of data by the public. Despite this, the majority overwhelmingly supports publication of team-based measures of outcome. The free-text responses suggest that this is because most believe that quality of care is multifactorial and not represented by an individual’s mortality rate.
Conclusions—There is evident opposition to surgeon-specific mortality data among UK cardiothoracic surgeons who associate this with several unintended consequences. Policy makers should refine their strategy behind publication of surgeon-specific mortality data and possibly consider shift toward team-based results for which there will be the required support. Stakeholder feedback and inclusive strategy should be completed before introducing major initiatives to avoid unforeseen consequences and disagreements.
WHAT IS KNOWN
Over the last 20 years, there have been 6 published surveys assessing cardiovascular specialists’ opinions on surgeon-specific mortality data, the majority originating from North America.
Primary issues around surgeon-specific mortality data raised in these surveys included the risk of misinterpretation by patients, the potential to induce risk-averse behavior by surgeons, preference for team-based (as opposed to individual-level) reporting, and concerns about the validity of the outcome metrics used.
WHAT THE STUDY ADDS
This study builds on the methodology of these previous surveys and uses a more advanced instrument.
There is increasing evidence in the United Kingdom and United States that cardiothoracic surgeons do not support publication of surgeon-specific mortality data in isolation. It is often not used by cardiologists in their decision-making, and there is accumulating evidence that it is associated with unintended consequences, such as risk-averse patient selection.
The richness of the free-text responses in this study confirms that policy makers need to improve their grasp of behavioral psychology and obtain stakeholder support before initiating major reporting initiatives.
Since its inception in the 1960s, British cardiothoracic surgery has been at the forefront of media attention.1 In an already challenging job, this has put strain on a profession that is barely half a century old.
One of the specialties’ greatest achievements is its publication of mortality rates after surgery. The Society for Cardiothoracic Surgeons in Great Britain and Ireland (SCTS) has been reporting outcomes for named surgeons since 2005. Using the example of SCTS, National Health Service (NHS) England published outcome data from 10 other specialties on its MyNHS website in 2013.2
There has been significant debate within the surgical community as to whether these initiatives have been of benefit. Although proponents assert that it is associated with an improvement in transparency and quality of care, opponents claim a range of negative side effects, including risk-aversion, disruption of innovation, and gaming of data.3–6 Outcome reporting of isolated mortality in the United Kingdom counters movements in the United States, where the focus has shifted toward supplementing mortality rate with multidimensional composite measures of outcome based on morbidity and process measures.7 The rationale behind these shifts is that providing an increase in the number of functional outcome end points allows quality to be better statistically discriminated than if mortality (which is infrequent in elective surgery) rates were used alone.
Editorial, see p 345
The objective of this study was to survey UK consultant cardiothoracic surgeons’ opinions on surgeon-specific mortality data (SSMD) and related areas, with the aim of guiding future debate and policy making. Surprisingly, no major stakeholder feedback took place before NHS England rolling out its outcome-reporting program in 2013. Therefore, the secondary objective of the survey is to provide guidance to other specialties which are developing outcome-reporting programs.
An online survey was designed based on 2 previous questionnaires used in New York3 and Chicago.8 These were developed further using literature search to discover themes not explored. Themes were developed and turned into questions through semistructured interviews with 10 experts in the field. The questionnaire was piloted, with feedback used to refine language used. Five-point Likert scales and free-text boxes were used to gather answers.
The final questionnaire was uploaded to an online platform (Qualtrics, UT). Anonymity was guaranteed to respondents, but they were given the option of providing their contact details. The online version of the survey can be viewed at http://imperial.eu.qualtrics.com/SE/?SID=SV_ewFsPdbUUJMTY6V
Study Design and Participants
In January 2015, a link to the online survey was e-mailed out to all practicing consultant cardiothoracic surgeons in the United Kingdom (n=361). A reminder e-mail was sent at 2 weeks, and the survey was closed after a month.
Statistical analysis was performed (SPSS v.22, IBM, Armonk, NY) to assess whether there was a relationship between answers and the demographic variables listed in Table 1. Four tests were used: Pearson’s χ2 test, bivariate correlation (Spearman’s with 2-tailed significance), and binomial logistic regression (both uni- and multivariable). For logistic regression, to convert the 5-point Likert Scale answers into a binomial coefficient, points 1 and 2 on the scale (eg, strongly oppose and somewhat oppose) were recoded into 1 and points 4 and 5 (eg, somewhat favor and strongly favor) were recoded into 0. The midpoint of scale was also recoded into 0 because this was felt to be the most conservative treatment of the neutral category. For the multivariable analysis, all 6 demographic variables were entered into the model because they were all felt to be relevant factors.
To graphically represent the factors influencing respondent’s answers, a heat map was created. Each demographic variable was allocated a color ranging from green to red depending on how many of the 4 tests performed were statistically significant for a given variable.
Analysis of free-text responses was performed using NVivo v.10 for Macintosh (QSR International, Melbourne, Australia). Three of the authors individually explored the textual data inductively (grounded theory approach) using content analysis to generate categories and explanations.9 Resulting themes were then checked and refined by assessing for inter-rater reliability. All responses were assigned at least one theme and more than this if required (ie, making many points in the same paragraph).
The study was authorized at an institutional level (Imperial College London, reference JRCO/15IC2539).
The overall response rate was 73% (n=264). Of these, 87% finished the survey, and mean time taken to completion was 10 minutes (±10.4).
Eighty-five respondents voluntarily gave identifiable contact information. This was used to extract individual’s risk-adjusted mortality rate (RAM) from the SCTS directory: http://scts.org/modules/surgeons/ (accessed in February 2015 and encompassed operations between 2010 and 2013).
The demographic characteristics of responders are displayed in Table 1, demonstrating a representative sample in terms of subspecialization, years of experience, and geographical location. The full breakdown of questions and answers are listed in Tables 2 and 3. The breakdown of the statistical analysis of factors influencing answers to all questions is displayed in Supplement S1 in the Data Supplement.
Public Release of Surgeon and Hospital-Specific Mortality Data (Q1–2)
The majority of respondents either strongly or somewhat oppose the public release of SSMD (58.1% oppose, 34.1% favor, and 7.8% neither). The experience of the surgeon responding was strongly predictive of opposition to SSMD. The 0 to 5 years of experience category was significantly more opposed to SSMD (P=0.049) than the 20+ years group.
Opinion was different with regards to the public release of hospital-specific mortality data, with respondents overwhelmingly favoring it (84.8% favor, 8.9% oppose, and 6.2% neither). No demographic variable was particularly predictive of respondent’s opinion.
Importance of SSMD in Assessing Ability of Surgeons and in Reflecting Quality of Care Given to Patients (Q3–4)
The majority of respondents feel that SSMD is important in assessing the overall ability of a surgeon (40.3% important, 33.4% not important, and 26.4% neither). Thoracic surgeons (P=0.040 versus adult cardiac surgeons) and those with higher RAM (P=0.023) were significantly more likely to oppose this view.
Opinion was split with regards to the importance of SSMD in assessing quality of care (38% important, 41.5% unimportant, and 20.5% neither). Respondents with a higher RAM were significantly more likely to feel that it was unimportant (P=0.005).
Impact of SSMD on Improvement in Outcomes in the United Kingdom and on Transparency and Accountability (Q5–6)
The majority of surgeons feel that SSMD has not contributed to improvements in outcomes seen over the last 10 years (44.5% not contributed, 35.0% contributed to improvements, and 20.5% neither). Those with higher RAM were significantly more likely to hold this opinion (P=0.030).
Most respondents feel that SSMD has improved transparency and accountability (43.7% improved, 30.3% made worse, and 26.0% about the same). Younger surgeons (P=0.038), thoracic surgeons (P=0.039), and those with higher RAM were significantly more likely to oppose this view (P=0.007).
Risk Aversion (Q7)
An overwhelming proportion of surgeons feel that SSMD has led to risk-averse behavior (86.6% yes, 5.5% no, and 7.9% maybe). Those in larger units were significantly more likely to hold this opinion (P=0.008).
Misinterpretation of SSMD by the Public and Referring Clinicians (Q8–9)
The majority of respondents feel that misinterpretation of SSMD by both patients (83.9% likely, 10.6% unlikely, and 5.5% undecided) and referring clinicians/cardiologists (65.8% likely, 21.6% unlikely, and 12.6% undecided) is likely. This opinion was significantly more likely in less experienced surgeons and significantly less likely in those involved in governance structures related to SSMD.
An overwhelming majority feels that training of juniors has worsened because of SSMD (74.7% worse, 5.9% better, and 19.4% same). Those with a higher RAM were more likely to feel that training had suffered (P=0.018).
Accuracy of Data Submitted to National Institute for Cardiovascular Outcomes Research and Definition of Chronic Obstructive Pulmonary Disease Question (Q11–12)
Most surgeons feel that information submitted to National Institute for Cardiovascular Outcomes Research is accurate (48.6% accurate, 33.2% inaccurate, and 18.2% neither). Those not involved in governance structures associated with SSMD were significantly more likely to feel that this information is inaccurate compared with those involved (P=0.020).
The correct answer to the definition of chronic obstructive pulmonary disease (as used in EuroSCORE) question is long-term bronchodilator use, which 64.3% of respondents answered correctly. Pediatric/thoracic surgeons (P=0.0000) and those not involved in governance structures related to SSMD were more likely to get this incorrect (P=0.041).
Gaming of Data (Q13)
The majority of respondents are of the opinion that gaming of patient disease status and comorbidities occur during data collection (51.2% yes, 15.3% no, and 33.5% maybe). No demographic variable was predictive of respondent’s opinion.
Opinion on Alternative Outcome Measures and Adequacy of Resources to Collect Data (Q14–15)
The overwhelming majority of respondents see benefit in public reporting of the outcome measures listed in Table 3. The significant majority, however, did not feel that their adult departments were adequately resourced to do this (60.2% adequately resourced, 33.2% not adequately resourced, and 6.6% maybe).
The demographic factors influencing answers to questions are represented as a heat map in Figure 1. The statistical analysis behind this is available in Supplement S1 in the Data Supplement. This heat map suggests that the 3 most powerful factors influencing surgeons’ opinion are their RAM (more negative opinions), whether they are involved in governance structures (more positive opinions), and how experienced they are (junior surgeons have more negative opinions).
One hundred and thirty-seven respondents (52%) made at least one comment, resulting in a total of 14 826 words of free-text responses that were thematically analyzed. A total of 31 different themes emerged from the texts, which were mentioned a total of 464 times (mean of 3.4 themes mentioned per person in their free-text response). For ease of interpretation, these themes were categorized into positive reflections, negative reflections, specific recommendations, and other (Figure 2).
The 4 commonest themes mentioned by respondents were risk-averse behavior, criticism of data accuracy and risk adjustment models, opinion that quality of care is multifactorial, and expression of a preference for team-based outcome measures. Example quotes are given in Table 4. The unedited free-text answers and their assigned themes are available in Supplement S2 in the Data Supplement.
This study is the first in-depth analysis of UK consultant cardiothoracic surgeon’s opinions on SSMD. The response rate was overwhelming, with 73% of practicing surgeons spending an average of 10 minutes on the survey. The collection of almost 15 000 words of free-text responses indicates the importance of the topic.
Fifty-eight percent of surgeons either oppose or strongly oppose public reporting of SSMD. This is more likely in young consultants and those with higher risk-adjusted mortality. Thematic analysis of free-text responses suggests 3 reasons for this:
First, respondents feel that quality of care is multifactorial and not represented by individual mortality rate. This opinion may have been driven by evidence associating perioperative mortality with failure to rescue rather than individual surgeon error.10 This has been recognized by the Society of Thoracic Surgeons in the United States who have focused on team and hospital results in recognition of the increasingly team-based nature of modern healthcare.11 Their own measurement of individual surgeon performance involves a composite measurement of 5 different morbidity outcomes in addition to mortality.
Second, the majority of respondents do not feel that SSMD has contributed to improvements in outcomes. This finding is supported by a recent Cochrane review, which was unable to find evidence of sufficient quality to associate SSMD with improved outcomes.12 The often quoted 40% reduction in mortality in New York after publication of results in 199113 is difficult to interpret because it is challenged by accusations of gaming of data, risk aversion, and outmigration of high-risk patients to other states. Similar reductions in mortality were seen in neighboring states and in other subspecialties without publication of results.14,15 Such observations may have influenced opinion of UK surgeons who seem to question the overall efficacy of SSMD.
Third, respondents associate SSMD with several negative side effects, the most important being risk-averse behavior and gaming of data. Eighty-seven percent of the respondents feel that SSMD has led to risk-averse behavior. This is consistent with opinion in Pennsylvania, where 59% of cardiologists had difficulty referring high-risk patients after public reporting of outcome data.16
Risk-averse or loss-averse behavior is concerning because it suggests that a surgeon may neglect the principle of beneficence and not offer an intervention that is in the best interest of a patient. Studies from diverse fields including psychology and behavioral science give insights into why surgeons in the United Kingdom may hold this opinion. Human behavior often departs from what is rational and what might be expected to get one the most utility. One of the reasons surgeons may be loss averse is that their marginal utility for reducing losses is greater than the marginal utility of increasing gains.17 In the current structure of the NHS, this issue of marginal utility is likely to be true because a surgeon will not receive significant reward for taking on a high-risk patient, but risks media attention and loss of job should a patient die. Risk aversion is also influenced by environmental factors such as peer-group behavior. When one’s peers are risk-averse, everyone in a unit is more likely to become so.18 Because cardiothoracic surgery is a tertiary service, units havig peer-group–induced risk-averseness may theoretically offer suboptimal treatment to a whole geographical region of patients because of the interaction of decision makers.
Prospect theory suggests individuals prefer a risky option to a certain one when choices are framed positively.19 When choices are framed negatively, the opposite is more likely. Thus, a surgeon with little likely reward (and significant punishment) for taking on a high-risk operation may see the decision as a negatively framed one. The theory also suggests that individuals overweigh the probability of rare events and underweigh common events.19 Death after elective cardiac surgery is uncommon, but given this behavioral phenomenon, surgeons may overweigh its impact and are more likely to see high-risk decisions as negatively framed ones. These observed behavioral phenomenon have been confirmed in neurobiological correlates of risk aversion, providing further evidence that assessment of risk is encoded in the brain.20 Prolonged stress elevates cortisol levels that have been associated with anxiety, impaired learning, risk-averse behavior, and burnout.21,22 Policy makers need to understand these complex processes to avoid unintended consequences when introducing quality improvement measures.23
The majority of respondents to the survey were of the opinion that gaming of patient disease status and comorbidities occur during data collection. Accusations of gaming (to overpredict risk and thus reduce adjusted mortality) were aimed at New York surgeons in the 1990s.13 These findings may be indicative of surgeons under immense pressure with fear of the consequences of bad results. A recent cross-sectional survey of over 10 000 physicians in the United Kingdom suggested that those who had undergone investigation were at higher risk of suicidal ideation and 77% more likely to be depressed.24 It is thus not unfeasible that surgeons may demonstrate abnormal behaviors in stressful environments as a method of coping. Although there is regional variation, the current process for data entry in the United Kingdom involves submission of case data by an individual surgeon (or his/her junior) to an institutional database. At this point, there is a basic check for incomplete/missing data before submission to National Institute for Cardiovascular Outcomes Research on a yearly basis. National Institute for Cardiovascular Outcomes Research does not currently implement any method of checking for inaccurate patient disease status and comorbidity. Concerns about gaming give support to the fact that independent nonconflicted parties should perform data entry that is crosschecked with a random validation mechanism. Statistically unexpected patterns in disease status could also be assessed as a further way of picking up purposefully inaccurate entries.
Despite the overall negative opinions expressed by respondents over SSMD, there is overwhelming (80.2%) support for the release of hospital- or unit-specific mortality data. An almost equal proportion supports publication of further morbidity data similar to that released by the Society of Thoracic Surgeons. This type of data collection may be a challenge because the majority of respondents felt that their audit departments were inadequately resourced. The support for unit-based mortality reporting is likely to be caused by the belief that it is a better measure of team performance. It is more likely to take into account the multiple factors that respondents feel are responsible for the overall quality of care, a theme mentioned 47 times in the free-text responses.
The findings from this survey have several significant policy implications. Lee and Cosgrove recently commented that for healthcare improvements: “Doctors, of course, must be central players in the transformation: Any ambitious strategy that they do not embrace is doomed.”25 Based on its unpopularity, the publication of individual surgeon mortality rates should be reconsidered. The survey suggests that its introduction may have had several far-reaching consequences. Its introduction has introduced a significant layer of additional decision and risk making for surgeons, who under stress may be incapable of dealing with this.26 The alternative to SSMD would be a shift toward team-based mortality reporting for which there is significant support. There is also clear agreement from surgeons that they would accept publication of further complex multidimensional outcome measures (similar to that published by the Society of Thoracic Surgeons) at both an individual and institutional level. Although accountability is critical in improving performance and ensuring good medical practice, an increase in institutional accountability may help offset some of the unintended consequences described by respondents in this survey. More important than specific policy changes is the engagement of policy makers with behavioral theory. This will allow them to better understand stakeholder’s autonomous motivations. They will then be able to identify the environmental contextual factors which make individuals more cooperative and more likely to engage with quality improvement measures.27
When interpreting the findings of this survey, its limitations must be taken into account. Although there was a significant response rate, there may also be respondent bias, where individuals with more negative opinions respond in greater proportion and with greater ferocity. There has been a significant amount of media attention around SSMD, and this may have altered individual’s opinions transiently. Opinions are not proof of causation, and this survey is not proof that negative behaviors such as risk-averseness or gaming of data actually occur. This survey is not representative of the situation outside of the United Kingdom, and we would urge international surgical societies to complete similar initiatives.
In summary, this survey has demonstrated a reasonable amount of opposition to SSMD by UK cardiothoracic surgeons. It associates it with a range of negative consequences, such as risk-averse behavior and gaming of data. This is a significant finding because the UK model is an example of an early adopter of publishing outcome data. Lessons that can be learnt should be applied to other medical and surgical subspecialties that will be facing similar challenges in the near future. Many of the concerns mentioned in this survey are surprising, original, and controversial. Most have not been raised in public before, and this is possibly because of individual’s fear of retribution. We recommend that other specialties create surveys to gain stakeholder feedback before quality initiative programs. This will give stakeholders an opportunity to speak out and give policy makers information with which to increase the chance of success.
Specific Author Contributions: O.A. Jarral, K. Baig, C. Pettengell, S. Westaby, and T. Athanasiou conceived the idea together. O.A. Jarral, K. Baig, C. Pettengell, S. Westaby, and T. Athanasiou designed the survey instrument together. O.A. Jarral, K. Baig, C. Pettengell, and T. Athanasiou completed the data collection and analysis. O.A. Jarral, R. Uppal, D.P. Taggart, A. Darzi, and S. Westaby edited and wrote the final article. All authors made a significant contribution and meet the criteria for authorship as per the ICMJE recommendations. All authors viewed and gave final approval of this article. All authors had full access to all of the data (including statistical reports and tables) in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Transparency Declaration: T. Athanasiou is senior author of this article and guarantor. He affirms that this article is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained. Data Sharing: The full original data set (downloaded from the Qualtrics survey software) is available from the corresponding author (firstname.lastname@example.org). This file contains no responder-identifiable information.
All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.
The Data Supplement is available at http://circoutcomes.ahajournals.org/lookup/suppl/doi:10.1161/CIRCOUTCOMES.116.002749/-/DC1.
- Received February 17, 2016.
- Accepted May 18, 2016.
- © 2016 American Heart Association, Inc.
- Tansey EME,
- Reynolds LAE
- Bridgewater B,
- Hickey GL,
- Cooper G,
- Deanfield J,
- Roxburgh J
- Pope C,
- Ziebland S,
- Mays N
- Westaby S,
- De Silva R,
- Petrou M,
- Bond S,
- Taggart D
- Shahian DM,
- He X,
- Jacobs JP,
- Kurlansky PA,
- Badhwar V,
- Cleveland JC Jr.,
- Fazzalari FL,
- Filardo G,
- Normand SL,
- Furnary AP,
- Magee MJ,
- Rankin JS,
- Welke KF,
- Han J,
- O’Brien SM
- Brown KL,
- Crowe S,
- Franklin R,
- McLean A,
- Cunningham D,
- Barron D,
- Tsang V,
- Pagel C,
- Utley M
- Tom SM,
- Fox CR,
- Trepel C,
- Poldrack RA
- Tversky A,
- Kahneman D
- Christopoulos GI,
- Tobler PN,
- Bossaerts P,
- Dolan RJ,
- Schultz W
- Kandasamy N,
- Hardy B,
- Page L,
- Schaffner M,
- Graggaber J,
- Powlson AS,
- Fletcher PC,
- Gurnell M,
- Coates J
- Halpern D,
- King D,
- Vlaev I,
- Hallsworth M
- Bourne T,
- Wynants L,
- Peters M,
- Van Audenhove C,
- Timmerman D,
- Van Calster B,
- Jalmbrant M
- Lee TH,
- Cosgrove T