Comparison-Adjusted Funnel Plots for Network Meta-Analysis

A funnel plot is a scatterplot of the study effect size versus some measure of its precision, often its inverted standard error. It is the most common tool used to assess the presence of small-study effects in a meta-analysis [34] (link). A funnel plot which is asymmetrical with respect to the line of the summary effect implies that there are differences between the estimates derived from small and large studies.
Extending the use of funnel plots into network meta-analysis needs to account for the fact that studies estimate effects for different comparisons. As a result, there is not a single reference line against which symmetry can be judged. To account for the fact that each set of studies estimates a different summary effect we suggest the ‘comparison-adjusted’ funnel plot. Before using this plot, investigators should order the treatments in a meaningful way and make assumptions about how small studies differ from large ones. For example, if they anticipate that newer treatments are favored in small trials, then they could name the treatments from oldest to newest so that all comparisons refer to ‘old versus new intervention’. Other possibilities include defining the comparisons so that all refer to an active treatment versus placebo or sponsored versus non-sponsored intervention.
In the ‘comparison-adjusted’ funnel plot the horizontal axis presents the difference between the study-specific effect sizes from the corresponding comparison-specific summary effect [35] . For example, in a triangle , we get the three direct summary estimates from simple pairwise meta-analyses. The treatments have been named, say, from the oldest to newest. Then, for studies that compare treatments and (providing and observed effect ) the horizontal axis represents the difference . Similarly, it represents and for studies comparing XZ and YZ respectively. In the absence of small study effects the ‘comparison-adjusted’ funnel plot should be symmetric around the zero line.
To produce a comparison-adjusted funnel plot in STATA our command netfunnel can be used:
. netfunnel lnOR selnOR t1 t2, bycomparison(assuming that effect size lnOR has been estimated as t1 vs. t2)
The option bycomparison adds comparison-specific colors to the studies.
The routine netfunnel plots the comparisons as ‘treatment alphabetically or numerically earlier versus later treatment' (e.g. A vs. B or 1 vs. 2) for string or numerical treatment identifiers. Therefore, missing (small) studies lying on the right side of zero line suggest that small studies tend to exaggerate the effectiveness of treatments named earlier in alphabet compared to those later for a harmful outcome. If the outcome is beneficial such asymmetry would indicate that small-study effects favor treatments later in the alphabetical or numerical order. A ‘comparison-adjusted’ funnel plot is meaningless unless the treatments are named in an order that represents a characteristic potentially associated with small study effects. Consequently, we recommend its use only when specific assumptions about the directions of small study effects can be made.
Figure 5 shows the funnel plot for the rheumatoid arthritis network which provides an indication for the presence of small-study effects. The plot indicates that small studies tend to show that the active treatments are more effective than their respective comparison-specific weighted average effect.
The options fixed and random in netfunnel command specify whether the summaries will be derived from a fixed- or random-effects model. A linear regression line of the comparison-specific differences on the standard error of can be fitted to the plot using the addplot option, e.g. addplot(lfit selnOR _ES_CEN) (see the green line in Figure 5).
After running netfunnel a new variable is added to the dataset named _ES_CEN that includes the differences between study-specific effect sizes and comparison-specific summary estimates.
As with the conventional funnel plot, caution is needed in interpretation. Asymmetry should not be interpreted as evidence of publication bias. If the funnel plot suggests the presence of small-study effects, investigators can explore this further by employing appropriate network meta-regression or selection models [36] , [37] (link).