Communication of Treatment Rankings Obtained From Network Meta-Analysis Using Data Visualization
- cardiovascular agents
- cardiovascular diseases
- coronary disease
- decision making, shared
- evidence-based medicine
- hydroxymethylglutaryl-CoA reductase inhibitors
The number of published network meta-analysis (NMA) reports has increased substantially in recent years. NMAs combine direct and indirect evidence and enable comparisons between all relevant treatment options for a given disease, even when some treatments have not been directly compared with each other. In the absence of randomized trials comparing all treatment options to each other, NMAs address important information needs of patients and clinicians about the comparative effectiveness of treatment alternatives.
NMA results may be difficult to communicate and interpret effectively given the large volume of complex information generated on multiple alternative treatments with multiple benefit and harm outcomes. For example, NMA comparing 5 treatments result in 10 pair-wise comparisons; if results are available for 3 benefit and 3 harm outcomes, decision makers are faced with 60 sets of results. Identifying the best treatment option to initiate therapy is, thus, not straightforward. Although several graphical and tabular displays exist to report the pertinent results of NMAs, existing reporting guidelines differ in their recommendations. Consequently, there is significant variation in the current way NMA findings are reported and presented.
A key strength of NMAs is the ability to rank treatments. However, such rankings are specific to individual outcomes and often change significantly across different benefit and harm end points. For example, a treatment that performs well in prolonging survival may fare unfavorably in terms of increasing the likelihood of side effects. Combining the relative performance of different treatments on multiple outcomes remains a challenge.
One option for generating a single coherent ranking of treatments is to quantitatively combine NMA findings with patient preferences.1 Preference information captures the relative importance of attributes that differ among alternative treatments. For example, a patient near the end of life may prefer a therapeutic strategy that minimizes drug-related side effects (however minor), even if the therapy has less potential for prolonging survival than its alternatives. In such a scenario, survival end points would carry less weight than adverse outcomes, favoring the treatments that have superior side effect profiles. Combination of NMA findings with preference information would allow for generating and communicating a coherent ranking of all treatment alternatives.
Statin Ranking Tool
A recent series of NMAs evaluated the comparative benefits and harms of widely popular cholesterol-lowering medications, statins, on the basis of a systematically identified set of randomized trials published since 1990.2,3 While some statins performed well in terms of benefit outcomes (measured in terms of reducing the risk of all-cause mortality, coronary, and cerebrovascular events),2 others had relatively favorable side effect profiles (measured in terms of increasing the risk of experiencing creatine kinase elevations, transaminase elevations, myalgia, and treatment discontinuations because of other adverse events).3 These analyses did not identify a single statin that outperformed its comparators in terms of both benefit and harm outcomes.
Data visualization offers an opportunity to communicate the top-level findings of these detailed analyses. The Statin Ranking Tool (available at http://lse.live.kiln.digital/statins/) uses a visually appealing, intuitive, and easy-to-navigate interactive interface to summarize the key findings of several published NMAs comparing the effectiveness of individual statins on 3 benefit and 4 harm outcomes. Focusing on five of the most commonly used statins (atorvastatin, lovastatin, pravastatin, rosuvastatin, and simvastatin), it excludes fluvastatin (rarely used in clinical practice) and pitavastatin (previous NMAs did not consider its effect on mortality, coronary, and cerebrovascular events).2,3 This tool was developed as part of the London School of Economics and Political Science Institute of Public Affairs data visualization project, which aims to make academic research visible and accessible to the wider public.
The Statin Ranking Tool has 2 data visualization features. The first feature presents the previously published rankings of individual statins on 7 outcomes, both separately and in combination with each other (Figure 1). Treatment rankings are expressed as probabilities (adding up to one for each statin and for each rank) and reflect the estimated effect sizes obtained from NMAs. As described previously,4 the surface under the cumulative ranking line (also known as SUCRA) for each treatment provides a numeric summary of the overall performance of each statin on each outcome (1.0 when a statin is certain to the best and 0.0 when it is certain to be the worst), taking into account not only the magnitude of the effect but also the uncertainty around it.
The novel aspect of the tool is its interactive web-based interface, which allows the users to focus on individual benefit and harm outcomes and see how different statins perform on each end point. In addition, the tool allows the users to specify the relative importance of multiple outcomes. The visualization is dynamic and reinforces the most critical gist message of the NMA findings: how the rankings of individual statins change as users modify the weights applied to different benefit and harm outcomes (from not important to very important). The tool quantitatively incorporates individual preferences by multiplying weights applied to different outcomes by the performance of different statins on different outcomes (as obtained from the NMA). The computation of overall rankings is, therefore, straightforward: SUCRA scores are multiplied by the normalized weights specified by the user (ie, weights applied to different outcomes add to 1).
The second key feature of the Statin Ranking Tool is an interactive network diagram, which presents the evolution of the evidence base on statins from 1990 to 2010 (Figure 2). The dynamic network diagram maps the existing comparisons of statins to each other and depicts the entirety of the randomized trial data on this extensively researched, popular, and widely used classes of drugs.
Given its web-based interface, both data visualization features of the Statin Ranking Tool have the potential to be continually updated and improved as more information becomes available. Accordingly, this approach can serve as a reporting companion to living systematic reviews and NMAs.
Limitations and Future Directions
Treatment rankings presented in the Statin Ranking Tool share the limitations of the evidence base. Randomized trials included in the NMAs differ in terms of their size, risk-of-bias, and external validity. Despite their limitations, these analyses offer the most comprehensive picture of how different statins compare to each other.
Treatment rankings in terms of rankograms and SUCRAs should not be interpreted in isolation from the relative treatment effect estimates on which they are based (ie, odds ratios, relative risks, and their 95% uncertainty intervals). Recent empirical evaluations have suggested that treatment rankings may not be statistically precise. In addition, rankograms and SUCRAs pose challenges for interpretation: they are not intuitive measures of effect and may not be clinically meaningful. Future applications of data visualization should consider incorporating additional information to help users effectively interpret the ranking information. This could include easy-to-understand visual summaries of absolute treatment effects using either bar charts or pictographs.
In the current Statin Ranking Tool, the overall ranking for each statin is a simple weighted average of its performance on different outcomes with the weight specified directly by the user (by moving the cursor from not important to very important). There are several opportunities to develop a more sophisticated ranking machinery. For example, the tool does not specify what very important and not important ends of the spectrum mean for each outcome. Accordingly, weighting the relative importance of different outcomes may be a challenging task: the tool neither considers nor demonstrates how the application of different weights would change the number of side effect episodes traded-off to prevent one major clinical event, or vice versa.
One opportunity is to formally combine NMA and multicriteria decision analysis.2 Multicriteria decision analysis is an established framework for making complex decisions, such as choosing among multiple options by trading off several positive and negative outcomes. Quantitative implementation of multicriteria decision analysis with NMA could serve as a basis for a treatment selection platform that could incorporate future NMAs as they become available. For example, an automated, freely available web-based tool could enable application of this combined method to any given treatment selection problem.
The Statin Ranking Tool offers a proof-of-concept novel approach to data visualization and summarizes an extensive body of literature previously synthesized in a series of NMAs. Methodological advances will further improve the ranking machinery behind the tool and enable its applicability to a wide range of treatment selection decisions informed by NMAs.
Sources of Funding
The data visualization presented in this article was developed by KILN studio in London, United Kingdom, with funding received as part of the London School of Economics and Political Science Institute of Public Affairs Data Visualization Project Competition.
- © 2016 American Heart Association, Inc.
- Naci H,
- van Valkenhoef G,
- Higgins JP,
- Fleurence R,
- Ades AE
- Naci H,
- Brugts J,
- Ades T