Estimating the contribution of studies in network meta-analysis: paths, flows and streams

Theodoros Papakonstantinou; Adriani Nikolakopoulou; Gerta Rücker; Anna Chaimani; Guido Schwarzer; Matthias Egger; Georgia Salanti

doi:10.12688/f1000research.14770.3

Home Browse Estimating the contribution of studies in network meta-analysis: paths,...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Method Article

Revised

Estimating the contribution of studies in network meta-analysis: paths, flows and streams

[version 3; peer review: 2 approved, 1 approved with reservations]

Theodoros Papakonstantinou¹^*, Adriani Nikolakopoulou ¹^*, Gerta Rücker², [...] Anna Chaimani³, Guido Schwarzer², Matthias Egger¹, Georgia Salanti¹

Theodoros Papakonstantinou¹^*, Adriani Nikolakopoulou ¹^*, [...] Gerta Rücker², Anna Chaimani³, Guido Schwarzer², Matthias Egger¹, Georgia Salanti¹

^* Equal contributors

PUBLISHED 13 Dec 2018

Author details Author details

¹ Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland
² Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center, University of Freiburg, Freiburg, Germany
³ Centre of Research in Epidemiology and Statistics Sorbonne Paris Cité (CRESS), Paris, France

Theodoros Papakonstantinou
Roles: Conceptualization, Formal Analysis, Methodology, Software, Writing – Original Draft Preparation, Writing – Review & Editing

Adriani Nikolakopoulou
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Gerta Rücker
Roles: Investigation, Methodology, Writing – Review & Editing

Anna Chaimani
Roles: Software, Writing – Review & Editing

Guido Schwarzer
Roles: Software, Writing – Review & Editing

Matthias Egger
Roles: Supervision, Writing – Review & Editing

Georgia Salanti
Roles: Conceptualization, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

In network meta-analysis, it is important to assess the influence of the limitations or other characteristics of individual studies on the estimates obtained from the network. The proportion contribution matrix, which shows how much each direct treatment effect contributes to each treatment effect estimate from network meta-analysis, is crucial in this context. We use ideas from graph theory to derive the proportion that is contributed by each direct treatment effect. We start with the ‘projection’ matrix in a two-step network meta-analysis model, called the H matrix, which is analogous to the hat matrix in a linear regression model. We develop a method to translate H entries to proportion contributions based on the observation that the rows of H can be interpreted as flow networks, where a stream is defined as the composition of a path and its associated flow. We present an algorithm that identifies the flow of evidence in each path and decomposes it into direct comparisons. To illustrate the methodology, we use two published networks of interventions. The first compares no treatment, quinolone antibiotics, non-quinolone antibiotics and antiseptics for underlying eardrum perforations and the second compares 14 antimanic drugs. We believe that this approach is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the proportion contributions of direct evidence from individual studies to network treatment effects.

Keywords

indirect evidence, proportion contributions, projection matrix, flow networks

Corresponding author: Adriani Nikolakopoulou

Competing interests: No competing interests were disclosed.

Grant information: TP received funding from a Campbell Collaboration Method Grant [grant no. CMG2.04].
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2018 Papakonstantinou T et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

How to cite: Papakonstantinou T, Nikolakopoulou A, Rücker G et al. Estimating the contribution of studies in network meta-analysis: paths, flows and streams [version 3; peer review: 2 approved, 1 approved with reservations]. F1000Research 2018, 7:610 (https://doi.org/10.12688/f1000research.14770.3) First published: 18 May 2018, 7:610 (https://doi.org/10.12688/f1000research.14770.1) Latest published: 13 Dec 2018, 7:610 (https://doi.org/10.12688/f1000research.14770.3)

Revised Amendments from Version 2

The revised version of this manuscript includes suggestions by Drs. Annette M. O’Connor and John R. Thomson. In particular:

We changed ‘percentages’ to ‘proportions’ throughout. Thus, Figure 1 has also been revised.
We clarify in which situations our algorithm can be used in sections 'Proportion contributions of direct comparisons' and 'Algorithm to decompose flows into proportion contributions'.
We have revised the Discussion according to reviewers' comments

See the authors' detailed response to the review by John R. Thompson
See the authors' detailed response to the review by Annette M. O'Connor
See the authors' detailed response to the review by Jochem König

Introduction

Decision making around multiple alternative healthcare interventions is increasingly based on meta-analyses of a network of relevant studies, which contribute direct and indirect evidence to different treatment comparisons^1,2. Limitations in the design and flaws in the conduct of studies synthesized in network meta-analysis (NMA) reduce the confidence in the results: a treatment comparison in the network may be directly or indirectly informed by studies at high risk of bias. A relative treatment effect from NMA (hereafter the NMA effect estimate) is estimated as a linear combination of the available direct estimates of the treatment effect (i.e. the results from pairwise meta-analyses) and the indirect evidence on the treatment effect.

Salanti et al. suggested that in order to assess the impact of study deficiencies on an NMA effect estimate, the limitations of studies contributing to direct estimates should be considered jointly, taking into account their relative contribution to the overall NMA effect estimate³. The proportion contribution matrix plays a key role in this approach: a matrix that shows how much each direct effect contributes to the estimation of the NMA effect.

The proportion contribution matrix is derived from the absolute contribution matrix. The absolute contributions of direct effects to an NMA effect is the projection matrix from a two-step NMA model^4,5. In the first stage, all direct effects are derived from pairwise meta-analyses. In the second stage, the NMA effect estimates are produced as a linear combination of the derived direct effects. The respective projection matrix is called the H matrix and it is analogous to the hat matrix in a linear regression model. The elements in the H matrix can be viewed as generalized weights from pairwise meta-analysis, but they do not add up to 1 and depend on the precision of the available studies, the degree of between-study heterogeneity and the network structure.

To translate the entries of the H matrix into proportion contributions, Salanti et al. suggested normalizing the absolute entries of each row of H and interpret them as proportions³. However, H represents the flow of evidence in different paths; the weight of each path is assigned to each direct effect involved. Thus, ignoring the multiple occurrences of the same values by taking standardized absolute values is incorrect. In particular, such a process overestimates the contribution of comparisons involved in long paths and underestimates the weights of the shortest paths. In this paper, we address this issue and present a method that properly translates the entries of the H matrix into proportions. The methodology is based on the observation that the rows of the H matrix can be interpreted as flow networks^4,6.

Motivating example

To illustrate the ideas presented in this paper, we will use a network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations⁷. This network was used in Salanti et al.³ and compares no treatment (x), quinolone antibiotic treatment (y), non-quinolone antibiotic treatment (u) and antiseptic treatment (v)⁷. The study outcome was the proportion of patients with persistent discharge from the ear after 1 week, measured using the odds ratio (OR). The network plot shown in Figure 1a shows that direct evidence exists for all comparisons except u versus x (non-quinolone antibiotic versus no treatment).

Figure 1. Network plot for the network of topical antibiotics without steroids for chronically discharging ears (a), comparison graph corresponding to the h^xy row of H matrix (b), flows f_uv with respect to the ‘x versus y’ network meta-analysis treatment effect are indicated along the edges), streams (c) and proportion contributions of each direct comparison (d).

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

In order to assess the confidence that should be placed in an NMA effect estimate, Salanti et al. suggested considering the quality of all pieces of evidence that contributed to it³. For example, the studies directly comparing ‘u versus v’ were judged to be at high risk of bias; however, in order to judge the quality of the NMA effect estimate of ‘u versus v’, we need to consider the amount of data that these studies contributed to its estimation.

Methods

We first present the random-effects two-stage NMA model first described by Lu et al.⁵. We will employ a simplified version of the H matrix described by König et al.⁴ that does not take into account the correlation induced by multi-arm trials. We ignore this correlation for the sake of ease of interpretation of the entries in the H matrix; we discuss implications of multi-arm trials at the end of the Methods section. Taking advantage of previous findings on how the flow of evidence can be considered in NMA^4,6, we present an algorithm to decompose the flow in a network and subsequently approximate the proportion contributions of direct effect estimates for each NMA effect estimate.

Two-stage network meta-analysis model

Consider a network of T competing treatments. The set of treatments is denoted by V = {x, y, u, v, ...} and let x denote the reference treatment. The number of NMA effects to be estimated is $(\begin{array}{l} T \\ 2 \end{array})$ but the estimation of T – 1 effects allows the derivation of the remaining effects via linear combination. We collect the T – 1 effects against the reference treatment x in a vector of basic parameters θ = (θ_xy,θ_xu,θ_xv, ...)′. In the case of a dichotomous outcome, θ is the parameter vector of all log-ORs compared to the common reference treatment x.

We assume that the distribution of effect modifiers is similar across comparisons and thus the transitivity assumption is plausible. The consistency assumption refers to the statistical manifestation of transitivity and implies that all sources of evidence are in agreement; this is expressed via the consistency equations

θ_uv = θ_xv – θ_xu, for all u, v ∈ V

Let us denote the number of comparisons with direct data (that is, at least one direct study) with D. For simplification, consider that there are no multi-arm studies. At the first stage of the NMA model, direct effects are estimated, using random-effects pairwise meta-analyses. The estimates of the direct effects are collected in a column vector ${\hat{θ}}^{D}$ of length D; their estimated variances are collected in a diagonal D × D matrix V^D. At the second stage, the NMA effects are estimated as

{\hat{θ}}^{N} = H {\hat{θ}}^{D} Equation 1

where H is

H = Y(X′(V^D)^–1X)^–1X′(V^D)^–1 Equation 2

Matrix X is a D × (T – 1) design matrix expressing the linear relationships between the available direct effects and the basic parameters and Y is a $(\begin{array}{l} T \\ 2 \end{array})$ × (T – 1) design matrix that links the NMA estimates with the basic parameters. Note that X is identical to Y only when there are direct studies for all treatment comparisons in the network.

Matrix H is of dimensions $(\begin{array}{l} T \\ 2 \end{array})$ × D and describes the influence of each direct effect (specified in the column) to an NMA effect (specified in the row). As Equation 2 implies, H is derived as a function of the variances of the direct effects ${\hat{θ}}^{D}$ and the network structure; therefore, the exact (absolute) contribution of each direct comparison depends on the precision of the available direct data and the comparison’s connectivity to the rest of the network. Note that it resembles the hat matrix in a linear regression model.

Let us focus on a single row of the H matrix which, say, corresponds to the NMA effect estimate of ‘x versus y’ and is denoted by h^xy. Elements of h^xy are denoted by $h_{u v}^{x y}$ and show the absolute contribution of the direct effect ${\hat{θ}}_{u v}^{D}$ indicated in the subscript (‘u versus v’) to the ‘x versus y’ NMA effect ${\hat{θ}}_{x y}^{N}$ . Consider our motivating example which examines the set of treatments V = {x, y, u, v}. Equation 1 implies that the NMA treatment effect for the ‘x versus y’ comparison is derived as a linear combination of the direct meta-analyses

{\hat{θ}}_{x y}^{N} = h_{x y}^{x y} {\hat{θ}}_{x y}^{D} + h_{y v}^{x y} {\hat{θ}}_{y v}^{D} + h_{x v}^{x y} {\hat{θ}}_{x v}^{D} + h_{y u}^{x y} {\hat{θ}}_{y u}^{D} + h_{u v}^{x y} {\hat{θ}}_{u v}^{D} Equation 3

The element $h_{x y}^{x y}$ represents the absolute but also the proportion contribution $p_{x y}^{x y}$ of the direct evidence for the particular NMA effect. Assuming that the comparison ‘x versus y’ is not part of any multi-arm study, the evidence to derive the NMA effect estimate can be portioned into direct and indirect estimates

{\hat{θ}}_{x y}^{N} = h_{x y}^{x y} {\hat{θ}}_{x y}^{D} + (1 - h_{x y}^{x y}) {\hat{θ}}_{x y}^{I} Equation 4

with ${\hat{θ}}_{x y}^{I}$ denoting the indirect effect for the ‘x versus y’ comparison; hence $p_{x y}^{x y} = h_{x y}^{x y}$ . While the proportion contribution of each direct effect to its NMA effect can be obtained as the diagonal of the H matrix, the proportion contributions of other direct relative effects via indirect evidence (e.g. the $p_{u v}^{x y}$ proportion contribution of ${\hat{θ}}_{u v}^{D}$ to ${\hat{θ}}_{x y}^{N}$ ) cannot be easily derived from the absolute contributions (that is, from $h_{u v}^{x y}$ ). In the next section we will present how the absolute contributions $h_{u v}^{x y}$ could be translated to proportion contributions $p_{u v}^{x y}$ .

To explain the method, we will continue focusing on one row of the H matrix, say h^xy, corresponding to the ‘x versus y’ comparison. Box 1 includes the definitions, along with the notation, of some of the notions used in this paper.

Box 1. Definitions

Set of vertices

The set of vertices is defined as the set of treatments examined in the network, V = {x, y, u, v ...}.

Set of directed edges

The set of directed edges E is defined as the set of direct comparisons respecting the signs of the entries of h^xy, the row of the H matrix corresponding to the ‘x versus y’ comparison. Edges are given a direction upon the definition of flows; the network itself corresponds to an undirected graph.

Set of flows

The set of flows is defined as F = {f_uv, ∀ uv ∈ E} where f_uv is equal to |h_uv|.

Comparison graph

A comparison graph is defined as a graph G_xy = (V, E, F) constructed from a row of the H matrix, h^xy; its definition derives from a set of vertices V, a set of edges E, and a set of flows, F.

Source

Source is defined as a vertex with no incoming edges.

Sink

Sink is defined as a vertex with no outgoing edges.

Path

A path π_i is defined as a sequence of connected directed edges belonging to E.

Stream

A stream S_i is defined as the composition of a path and its associated flow, S_i = (φ_i, π_i). with i = 1, ... , I where I is the total number of streams.

Comparison graph

König et al. showed that every row of the H matrix, h^xy, can be interpreted as a flow network with source x and sink y, and visualised in a directed acyclic graph (DAG)⁴. Thus, we create a graph G_xy = (V, E, F) from h^xy; its definition derives from a set of vertices V, a set of edges E, and a set of flows, F. The set of vertices is defined as the set of treatments examined in the network, V = {x, y, u, v ...}. Set E is defined as a set of directed edges that correspond to observed direct comparisons respecting the signs of the entries of h^xy. To simplify the notation, we drop from now on all superscripts assuming they all refer to xy. Then, the set E contains uv if h_uv > 0 or contains vu if h_uv < 0. The set of flows is defined as F = {f_uv, ∀ uv ∈ E} where f_uv is equal to |h_uv|.

The following conditions hold for the elements of set F (see Supplementary File 1 for proof):

a. The sum of outflows of node x (source) is 1

\sum_{x u \in E} f_{x u} = 1

b. The sum of inflows of node y (sink) is 1

\sum_{u y \in E} f_{u y} = 1

c. The flow passing through each internal node (any node except x or y) is conserved

\forall z \in V \ {x, y}, \sum_{v \in V} f_{v z} = \sum_{u \in V} f_{z u}

d. G_xy is acyclic; there is no path (sequence of edges) that visits the same vertex twice.

Consider, for example, the graph G_xy in Figure 1b, which corresponds to the xy comparison of the network of four treatments of Figure 1a. The set of vertices is V = {x, y, u, v} and the set of directed edges is E = {xy, xv, vy, vu, uy}. Flows f_uv are given along the edges; their numerical values are equal to the respective absolute entries of h^xy and the direction of their corresponding edge is indicated in the subscript. As properties (a) to (d) imply, the arrows in Figure 1b indicate that the outflows of x, as well as the inflows of y, equal 1, and that the inflows equal the outflows in the intermediate nodes u and v.

Streams

In Figure 1b there are three different paths from x to y, one based on direct evidence, {xy}, and two based on indirect evidence, {xv, vy} and {xv, vu, uy}. A path is a sequence of connected directed edges belonging to E, and we denote it as π_i. As property (d) implies, each node occurs at most once in π_i. Then, given the above properties of f_uv, we can assign a flow φ_i to each path π_i. Flow φ_i is equal to the smallest f_uv in the path π_i. Figure 1c shows the three paths from x to y; π₁, π₂ and π₃, and their corresponding flows. Path π₁ corresponds to xy and its flow, φ₁, equals the flow of the single edge in path, f_xy = 0.635. Path π₂ is constituted from two edges, xv and vy; thus, flow φ₂ = min(f_xv, f_vy) = 0.251. The flow corresponding to the third path π₃ is φ₃ = min(f_xv, f_vu, f_uy) = 0.114.

We define a stream, S_i, as the composition of a path and its associated flow, S_i = (φ_i, π_i) with i = 1, ..., I where I is the total number of streams; here I = 3. Note that it holds $\sum_{i = 1}^{I} φ_{i} = 1$ .

Proportion contributions of direct comparisons

In order to assign proportion contributions to each direct comparison, we need to split each stream’s flow to the involved edges in the stream’s path. Equation 3 can be re-written as

{\hat{θ}}_{x y}^{N} = φ_{1} {\hat{θ}}_{x y}^{D} + φ_{2} ({\hat{θ}}_{x v}^{D} - {\hat{θ}}_{y v}^{D}) + φ_{3} ({\hat{θ}}_{x v}^{D} - {\hat{θ}}_{u v}^{D} - {\hat{θ}}_{y u}^{D}) Equation 5

with $\sum_{i = 1}^{3} φ_{i} = 1$ . It follows from the properties of the elements of set F, that each NMA effect can be written as a linear combination of direct and indirect effects, in the form of Equation 5. The effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis.

To approximate the proportion contributions per comparison, we suggest dividing φ_i by the length of the respective path π_i, #π_i. This will leave the proportion contribution of the direct evidence of the same treatment comparison equal to the diagonal of the H matrix and assign to each comparison involved in an indirect route a portion of the respective stream’s flow. Note that directed edges might be involved in more than one path; we thus define the proportion contribution of an edge uv as

p_{u v} = \sum_{\forall i where u v \in π_{i}} φ_{i} / # π_{i} Equation 6

Figure 1d shows the derivation of the proportion contributions of each direct comparison in the network of topical antibiotics. Hence, from the row of the H matrix (0.635, 0.365, –0.114, –0.251, –0.114), which shows the absolute contributions of the direct effects ${\hat{θ}}_{x y}^{D}, {\hat{θ}}_{x v}^{D}, {\hat{θ}}_{y v}^{D}, {\hat{θ}}_{y u}^{D}, {\hat{θ}}_{u v}^{D}$ to ${\hat{θ}}_{x y}^{N}$ , we approximated their proportion contribution as 63.5%, 16.4%, 3.8%, 12.6% and 3.8%, respectively.

Algorithm to decompose flows into proportion contributions

In this section, we present an iterative algorithm that generalizes the process outlined above to derive proportion contributions of each direct effect to the estimation of a ‘x versus y’ NMA effect. We start by defining a graph G_xy from h^xy.

The algorithm is described as follows:

0. Set initial graph G₀ = (V, E₀, F₀) = G_xy. E₀ contains uv if h_uv > 0 or contains vu if h_uv < 0. The set of flows is F₀ = {f_0,uv | uv ∈ E₀}; numerical values of f_0,uv are equal to h_uv.

Then, repeat the process below I times, equal to the number of streams in G_i, until E_i = {Ø}.

1. In G_i–1, find the shortest path from x to y, π_i, and define its flow as φ_i = min{f_i–1,uv, uv ∈ π_i}. Then, use π_i and φ_i to define the stream S_i = (φ_i, π_i).
2. Recalculate the flow of edges uv ∈ π_i by subtracting φ_i from the flow of the edges of the stream found: f_i,uv = f_i–1,uv – φ_i ∀ uv ∈ π_i. The flow of the rest of the edges that do not belong to π_i remain unchanged: f_i,uv = f_i–1,uv ∀ uv ∉ π_i.
3. Define E_i as the set of edges uv for which f_i,uv > 0; this is E_i–1 after removing the edges with zero flow, E_i = E_i–1\{uv | f_i,uv = 0}. Collect f_i,uv to form the set F_i = {f_i,uv | uv ∈ E_i}.
4. If E_i ≠ {Ø} define G_i = (V, E_i, F_i) and go to step 1.

When the algorithm terminates, all streams S_i = (φ_i, π_i) have been identified and Equation 6 is used to derive the proportion contributions p_uv.

Repeating the same process for all NMA effects, we derive all $p_{u v}^{x y}$ and collect them in a matrix P of the same dimensions as H. The presented algorithm could be described as a reverse maximum flow Edmonds Karp algorithm⁸, but instead of adding we remove augmenting paths.

It is possible that multiple shortest paths exist; in this case, the order in which one chooses such a path could in principle result in different proportion contributions per comparison. We can, thus, use the following modification in the algorithm to impose consistency. Instead of selecting the shortest path, we assign cost values c_i,uv to each edge uv as follows: c_i,uv = 2 – f_i–1,uv. Then, we select the path from x to y with the minimum cost across comparisons included in π_i. The definition of the cost values c_i,uv assures that paths are selected from shortest to longest and removes any ambiguity regarding the selection of paths.

The starting point for the developed algorithm was a simplified version of the H matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the H matrix as described by König et al.⁴ that extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm. The estimator of heterogeneity as well as the assumption of a different or common heterogeneity across comparisons in the network does not modify any aspect of the method.

Calculations in this paper were performed using R.

Application

We apply the algorithm described above to the network of topical antibiotics⁷.

Proportion contributions of direct relative treatment effects to the estimation of the NMA effect between non-quinolone antibiotic and no treatment

Direct effects are obtained using the random effects model and the H matrix of dimension 6×5 is calculated using Equation 2. The H matrix, along with NMA effects, is given in Table 1.

Table 1. H matrix in the network of topical antibiotics without steroids for chronically discharging ears.

Columns correspond to direct comparisons and rows correspond to network meta-analysis (NMA) effects. Direct effects along with their variances and NMA effects with 95% confidence intervals (CIs) are given in the last column. Direct and NMA effects are measured as log odds ratios. Positive values favour the first treatment.

	xy	xv	yu	yv	uv	Direct effect	Variance of direct effect	NMA effect (95% CIs)
xy	0.635	0.365	–0.114	–0.251	–0.114	–2.29	0.42	–1.86 (–3.05,–0.67)
xu	0.603	0.397	0.632	–0.029	–0.368	–	–	–1.32 (–2.61,–0.02)
xv	0.545	0.455	0.170	0.375	0.170	0.35	0.63	–0.72 (–1.95,0.52)
yu	–0.032	0.032	0.745	0.223	–0.255	0.39	0.12	0.54 (–0.07,1.16)
yv	–0.090	0.090	0.284	0.627	0.284	1.24	0.15	1.14 (0.47,1.81)
uv	–0.058	0.058	–0.462	0.404	0.538	0.53	0.22	0.60 (–0.12,1.33)

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

We begin by applying step 0 of the algorithm. We construct the network G₀ = G_xy = (V, E₀, F₀) with source x and sink y corresponding to row h^xy. The set of vertices is V = {x, y, u, v} and the set of directed edges, taking into account the signs of the elements of h^xy, is E₀ = {xy, xv, vy, vu, uy}. The set of flows is F₀ = {f_0,uv | uv ∈ E₀}, where f_0,uv equal the respective absolute values of Table 1 and are given along the edges of Figure 1b.

Then, we apply the developed iterative algorithm until E_i = {Ø}. The iterations of the algorithm equal the number of existing streams from x to y and are illustrated in Figure 2.

Figure 2. Illustration of the steps of the algorithm for approximating proportion contributions per comparison in the network of topical antibiotics without steroids for chronically discharging ears focusing on the comparison ‘x versus y’.

Treatment labels: x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

First iteration

1. In G₀, find the shortest path from x to y, π₁ = {xy}. Define its flow as φ₁ = min{f_0,uv, uv ∈ π₁} = f_0,xy = 0.635. Define stream S₁ = (φ₁, π₁) (Figure 2a).
2. Recalculate the flow of edge xy ∈ π₁ as f_1,xy = f_0,xy – φ₁ = 0.635 – 0.635 = 0. The flow of the rest of the comparisons remains unchanged: f_1,uv = f_0,uv ∀ uv ∉ π₁ (Figure 2b).
3. Define E₁ as the set of edges uv for which f_1,uv > 0; edge xy is removed since its flow is zero, E₁ = E₀\{xy}. Collect f_1,uv to form the set F₁ = {f_1,uv | uv ∈ E₁} (Figure 2c).
4. Define G₁ = (V, E₁, F₁). As E₁ = {xv, vy, vu, uy} ≠ {Ø}, go to step 1 (Figure 2c).

Second iteration

1. In G₁, find the shortest path from x to y, π₂ = {xv,vy}. Define its flow as φ₂ = min{f_1,uv, uv ∈ π₂} = f_1,vy = 0.251. Define stream S₂ = (φ₂, π₂) (Figure 2d).
2. Recalculate the flow of edges xv and vy as f_2,xv = f_1,xv – φ₂ = 0.365 – 0.251 = 0.114 and f_2,vy = f_1,vy – φ₂ = 0.251 – 0.251 = 0. The flow of the rest of the comparisons remains unchanged: f_2,uv = f_1,uv ∀ uv ∉ π₂ (Figure 2e).
3. Define E₂ as the set of edges uv for which f_2,uv > 0; edge vy is removed since its flow is zero and thus E₂ = E₁\{vy}. Collect f_2,uv to form the set F₂ = {f_2,uv | uv ∈ E₂} (Figure 2f).
4. Define G₂ = (V, E₂, F₂). As E₂ = {xv, vu, uy} ≠ {Ø}, go to step 1 (Figure 2f).

Third iteration

1. In G₂, find the shortest path from x to y, π₃ = {xv, vu, uy}. Define its flow as φ₃ = min{f_2,uv, uv ∈ π₃} = 0.114. Define stream S₃ = (φ₃, π₃) (Figure 2g).
2. Recalculate the flow of edges xv, vu and uy as f_3,xv = f_3,vu = f_3,uy = f_2,xv – φ₃ = 0.114 – 0.114 = 0 (Figure 2h).
3. Define E₃ as the set of edges uv for which f_3,uv > 0; edges xv, vu and uy are removed since their flow is zero, E₃ = E₂\{xv, vu, uy}. Collect f_3,uv to form the set F₃ = {f_3,uv | uv ∈ E₃} (Figure 2i).
4. Define G₃ = (V, E₃, F₃). The set of direct edges is E₃ = {Ø} and the algorithm is terminated at this point (Figure 2i).

Figure 1c shows the flows of the three streams identified when applying the above algorithm. We then calculate the proportion contributions of each comparison to the ‘x versus y’ NMA treatment effect estimate using Equation 6 (Figure 1d). For instance, to calculate p_xv we first have to identify the relevant paths; these were π₂ and π₃. Consequently,

p_{x v} = \frac{φ_{2}}{| π_{2} |} + \frac{φ_{3}}{| π_{3} |} = \frac{0.251}{2} + \frac{0.114}{3} = 0.164 = 16.4 %

The calculations for deriving the proportion contributions of the other comparisons are shown in Table 2. Applying the algorithm to all NMA treatment effect estimates we get the entire proportion contribution matrix P.

Table 2. Proportion contributions of direct comparisons to the ‘x versus y’ network meta-analysis treatment effect in the network of topical antibiotics without steroids for chronically discharging ears.

	xy	xv	yu	yv	uv
xy	$p_{x y} = \frac{φ_{1}}{\| π_{1} \|}$ = 63.5%	$p_{x v} = \frac{φ_{2}}{\| π_{2} \|} + \frac{φ_{3}}{\| π_{3} \|}$ = 16.4%	$p_{y u} = \frac{φ_{3}}{\| π_{3} \|}$ = 3.8%	$p_{y v} = \frac{φ_{2}}{\| π_{2} \|}$ = 12.6%	$p_{u v} = \frac{φ_{3}}{\| π_{3} \|}$ = 3.8%

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Proportion study contributions

Matrix P (Table 3) shows the proportion contributions of each direct comparison to each NMA treatment effect estimate. These proportions can be distributed to individual studies within each comparison according to their weights from direct meta-analyses. For example, p_xy = 63.5% and there are two studies examining the xy comparison. The individual study weights for the two studies are 0.69 and 1.54 resulting in study proportion contributions of $\frac{0.69}{0.69 + 1.54} 63.5 % = 19.6 %$ and $\frac{1.54}{0.69 + 1.54} 63.5 % = 43.8 %$ to the xy NMA treatment effect estimate. The application of this process to the entire matrix P leads to the matrix P* shown in Table 4. Adjusted weights as proposed by Rücker & Schwarzer⁹ are used for multi-arm studies.

Table 3. Proportion contribution matrix P for the network of topical antibiotics without steroids for chronically discharging ears.

Cells show the proportion contribution of direct comparisons indicated in the column to the network meta-analysis treatment effects indicated in the rows.

	xy	xv	yu	yv	uv
xy	63.5%	16.4%	3.8%	12.6%	3.8%
xu	30.1%	19.4%	31.1%	1%	18.4%
xv	24.4%	45.5%	5.7%	18.8%	5.7%
yu	1.1%	1.1%	74.5%	11.1%	12.2%
yv	4.5%	4.5%	14.2%	62.7%	14.2%
uv	1.9%	1.9%	22.1%	20.2%	53.8%

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Table 4. Study proportion contribution matrix P* for the network of topical antibiotics without steroids for chronically discharging ears.

Cells show the proportion contribution of individual studies indicated in the column to the network meta-analysis treatment effects indicated in the rows.

	Study 1	Study 2	Study 3	Study 4	Study 5	Study 6	Study 7	Study 8	Study 9	Study 10	Study 11	Study 12	Study 13
xy	19.7	63.4	2.8	0.5	0.6	0.4	0.6	0.7	2.7	1.3	0.8	2.3	4.2
xu	9.3	40.4	8.8	4.4	5.3	3.5	5.1	5.4	6.9	6.5	3.9	0.2	0.3
xv	7.6	67.1	4.2	0.8	1	0.6	0.9	1	4.1	2	1.2	3.4	6.2
yu	0.3	4.63	13.6	10.5	12.7	8.4	12.2	12.9	12.2	4.3	2.6	2	3.7
yv	1.4	23.5	12.1	2	2.4	1.6	2.3	2.4	12.2	5	3	11.3	20.7
uv	0.6	8.4	19	3.1	3.8	2.5	3.6	3.8	14.4	19.1	11.5	3.6	6.7

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Using proportion study contributions to quantify the impact of a characteristic in a direct comparison

The algorithm translating the H matrix into study proportion contributions can be applied to quantify the influence that a study-level characteristic has in the estimation of the NMA effects. For instance, if risk of bias judgements for individual studies are available, we can obtain an approximation of the proportion of each NMA treatment effect estimate that is coming from studies with a ‘high’, ‘moderate’, or ‘low’ risk of bias. Salanti et al. suggested the visualisation of this information using a bar plot, in which direct comparisons of the same risk of bias level have been grouped³. Figure 3 shows such a bar plot using the algorithm described in this paper and distributing comparison proportion contributions to study proportion contributions; inspecting Figure 3 can support judgements regarding the importance of study limitations for different NMA treatment effect estimates. For instance, studies with high risk of bias contribute more than 50% in the estimation of the ‘u versus v’ comparison, potentially reducing the confidence that we can place in this particular NMA treatment effect estimate.

Figure 3. Bar plot showing the study proportion contributions of direct comparisons with low (green), moderate (yellow) and high (red) risk of bias.

The bar plot has been produced in CINeMA (Confidence In Network Meta-Analysis) software¹¹. Studies are synthesized using the random effects model. x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Proportion contributions of direct comparisons in a large complex network of interventions

So far, we have illustrated how to derive proportion contributions for a network with four treatments. However, the algorithm can be straightforwardly applied to large networks of any structure, as soon as the involved treatments are connected. Consider for example a large network examining antimanic drugs (Figure 4)¹⁰. Let us concentrate on the comparison PLA versus OLA (‘placebo versus olanzapine’); the algorithm starts by applying step 0 and constructing network G₀. Then, we continue by finding the shortest path in the first iteration, which corresponds to the direct comparison, and define its flow and stream S₁ = (φ₁, π₁). The number of algorithm’s iterations is equal to the number of streams from placebo to olanzapine, which turns out to be 16. The resulting entire proportion matrix is given in Supplementary File 2.

Figure 4. Network plot for the network of antimanic drugs.

ASE, asenapine; ARI, aripiprazole; PLA, placebo; HAL, haloperidol; QUE, quetiapine; LITH, lithium; ZIP, ziprasidone; OLA, olanzapine; DIV, divalproex; RIS, risperidone; CARB, carbamazepine; LAM, lamotrigine; PAL, paliperidone; TOP, topiramate; ASE, asenapine.

study;id;t;r;n;rob
Kasemsuwan 1997;1;2;3;19;2
Kasemsuwan 1997;1;1;14;16;2
van Hasselt 2002;2;2;32;79;2
van Hasselt 2002;2;1;66;83;2
van Hasselt 2002;2;4;77;91;2
van Hasselt 1997;3;2;7;14;3
van Hasselt 1997;3;3;24;40;3
van Hasselt 1997;3;4;34;39;3
Kaygusuz 2002;4;2;10;20;2
Kaygusuz 2002;4;3;10;20;2
Lorente 1995;5;2;8;159;2
Lorente 1995;5;3;9;149;2
Tutkun 1995;6;2;3;24;2
Tutkun 1995;6;3;14;20;2
VH 1998 daily;7;2;9;32;2
VH 1998 daily;7;3;12;36;2
VH 1998 weekly;8;2;16;39;2
VH 1998 weekly;8;3;13;31;2
Fradis 1997;9;2;10;19;1
Fradis 1997;9;3;8;18;1
Fradis 1997;9;4;13;17;1
Clayton 1990;10;3;19;60;3
Clayton 1990;10;4;14;42;3
Browning 1983a;11;3;15;18;2
Browning 1983a;11;4;13;20;2
Jaya 2003;12;2;6;21;1
Jaya 2003;12;4;6;19;1
Macfadyen 2005;13;2;66;196;1

Dataset 1.Outcome data from the example network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations7.

Data labels: study, name of individual studies; id, id of the individual studies; t; treatment; r, number of events; n, sample size; rob, risk of bias per study.

Discussion

In this paper, we present a new approach to derive proportion contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects. The derivation of the contribution of sources of evidence in a Bayesian NMA is not straightforward, as no analogue to the H matrix exists. The application of the method described in this paper, however, can be used to derive useful approximations of the contributions of studies.

Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates. Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the proportion contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its proportion contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.

Alternative methods to derive the relative contribution of all sources of evidence have been developed^12,13. Krahn et al. define influence functions to describe the extent to which changes in study effects would be translated into NMA treatment effects¹³. An alternative approach, based on the decomposition of Fisher’s information matrix, has been proposed to derive proportion study weights in a variety of meta-analysis models, including meta-regression, network meta-analysis and individual patient data meta-analysis¹². Further investigation of the degree of agreement between our algorithm and that of Riley et al.¹² would be of interest.

In the example implemented in the Application, there is no other possible set of paths, and associated streams, that could be selected from x to y in order to partition the inflow of x: π₁, π₂ and π₃ is the only possible set of streams (Figure 1c). Thus, even if we were taking paths using different criteria, i.e. from longest to shortest, according to values from the H matrix or even randomly, the proportion contributions given in Table 2 would be identical. However, cases exist where the selection of paths does influence the derivation of the P matrix. In Supplementary File 3, we elaborate on the selection of direct paths in the algorithm and discuss some alternative modifications of the algorithm. We are planning to examine the properties of the different approaches in greater detail in a follow up project.

We offer an R package¹⁴, which we also use in the software application CINeMA (Confidence In Network Meta-Analysis)¹¹, that aims to simplify the evaluation of confidence in the findings from NMA. While CINeMA largely follows the framework previously developed by Salanti et al.³, the refinement of several methodological aspects is currently under development. Core aspects of the approach include the consideration of the relative contributions of each direct comparison to each NMA treatment effect estimate. To this end, CINeMA uses the proportion contribution matrix as described in this paper. The command netweight in Stata has also been updated to use the described approach.

We believe that the approach described in this paper is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the proportion contributions of direct evidence from individual studies to network treatment effects.

Data availability

Dataset 1: Outcome data from the example network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations⁷. Data labels: study, name of individual studies; id, id of the individual studies; t; treatment; r, number of events; n, sample size; rob, risk of bias per study. DOI: 10.5256/f1000research.14770.d203174¹⁵.

Grant information

TP received funding from a Campbell Collaboration Method Grant [grant no. CMG2.04].

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Supplementary material

Supplementary File 1. Proof that the sum of all outflows from the source is equal to the sum of all inflows to the sink and both are 1.

Click here to access the data.

Supplementary File 2. Proportion contribution matrix P for the network of antimanic drugs. Cells show the proportion contribution of direct comparisons indicated in the column to the network meta-analysis treatment effects indicated in the rows. ASE, asenapine; ARI, aripiprazole; PLA, placebo; HAL, haloperidol; QUE, quetiapine; LITH, lithium; ZIP, ziprasidone; OLA, olanzapine; DIV, divalproex; RIS, risperidone; CARB, carbamazepine; LAM, lamotrigine; PAL, paliperidone; TOP, topiramate; ASE, asenapine.

Click here to access the data.

Supplementary File 3. Considerations on the selection of streams

Click here to access the data.

Faculty Opinions recommended

References

1. Eichler HG, Thomson A, Eichler I, et al.: Assessing the relative efficacy of new drugs: an emerging opportunity. Nat Rev Drug Discov. 2015; 14(7): 443–4. PubMed Abstract | Publisher Full Text
2. Jansen JP, Trikalinos T, Cappelleri JC, et al.: Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014; 17(2): 157–73. PubMed Abstract | Publisher Full Text
3. Salanti G, Del Giovane C, Chaimani A, et al.: Evaluating the quality of evidence from a network meta-analysis. PLoS One. 2014; 9(7): e99682. PubMed Abstract | Publisher Full Text | Free Full Text
4. König J, Krahn U, Binder H: Visualizing the flow of evidence in network meta-analysis and characterizing mixed treatment comparisons. Stat Med. 2013; 32(30): 5414–29. PubMed Abstract | Publisher Full Text
5. Lu G, Welton NJ, Higgins JP, et al.: Linear inference for mixed treatment comparison meta-analysis: A two-stage approach. Res Synth Methods. 2011; 2(1): 43–60. PubMed Abstract | Publisher Full Text
6. Rücker G: Network meta-analysis, electrical networks and graph theory. Res Synth Methods. 2012; 3(4): 312–24. PubMed Abstract | Publisher Full Text
7. Macfadyen CA, Acuin JM, Gamble C: Topical antibiotics without steroids for chronically discharging ears with underlying eardrum perforations. Cochrane Database Syst Rev. 2005; (4): CD004618. PubMed Abstract | Publisher Full Text
8. Cormen TH, editor: Introduction to algorithms. 2. ed., 8. printing, Cambridge, Mass.: MIT Press [u.a.]; 2007; 1180. Reference Source
9. Rücker G, Schwarzer G: Reduce dimension or reduce weights? Comparing two approaches to multi-arm studies in network meta-analysis. Stat Med. 2014; 33(25): 4353–69. PubMed Abstract | Publisher Full Text
10. Cipriani A, Barbui C, Salanti G, et al.: Comparative efficacy and acceptability of antimanic drugs in acute mania: a multiple-treatments meta-analysis. Lancet. 2011; 378(9799): 1306–15. PubMed Abstract | Publisher Full Text
11. Confidence In Network Meta-Analysis [Internet]. Reference Source
12. Riley RD, Ensor J, Jackson D, et al.: Deriving percentage study weights in multi-parameter meta-analysis models: with application to meta-regression, network meta-analysis and one-stage individual participant data models. Stat Methods Med Res. 2018; 27(10): 2885–2905. PubMed Abstract | Publisher Full Text | Free Full Text
13. Krahn U, Binder H, König J: Visualizing inconsistency in network meta-analysis by independent path decomposition. BMC Med Res Methodol. 2014; 14: 131. PubMed Abstract | Publisher Full Text | Free Full Text
14. Flow contribution R package [Internet]. Reference Source
15. Papakonstantinou T, Nikolakopoulou A, Rücker G, et al.: Dataset 1 in: Estimating the contribution of studies in network meta-analysis: paths, flows and streams. F1000Research. 2018. http://www.doi.org/10.5256/f1000research.14770.d203174

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 18 May 2018

Author details Author details

Theodoros Papakonstantinou
Roles: Conceptualization, Formal Analysis, Methodology, Software, Writing – Original Draft Preparation, Writing – Review & Editing

Adriani Nikolakopoulou
Roles: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Gerta Rücker
Roles: Investigation, Methodology, Writing – Review & Editing

Anna Chaimani
Roles: Software, Writing – Review & Editing

Guido Schwarzer
Roles: Software, Writing – Review & Editing

Matthias Egger
Roles: Supervision, Writing – Review & Editing

Georgia Salanti
Roles: Conceptualization, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

TP received funding from a Campbell Collaboration Method Grant [grant no. CMG2.04].
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (3)

version 3

Revised

Published: 13 Dec 2018, 7:610

https://doi.org/10.12688/f1000research.14770.3

version 2

Revised

Published: 03 Sep 2018, 7:610

https://doi.org/10.12688/f1000research.14770.2

version 1

Published: 18 May 2018, 7:610

https://doi.org/10.12688/f1000research.14770.1

© 2018 Papakonstantinou T et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Papakonstantinou T, Nikolakopoulou A, Rücker G et al. Estimating the contribution of studies in network meta-analysis: paths, flows and streams [version 3; peer review: 2 approved, 1 approved with reservations] F1000Research 2018, 7:610 (https://doi.org/10.12688/f1000research.14770.3)

NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 1

VERSION 1

PUBLISHED 18 May 2018

Views

Reviewer Report 16 Aug 2018

John R. Thompson, Department of Health Sciences, University of Leicester, Leicester, UK

Approved

https://doi.org/10.5256/f1000research.16071.r37273

This report is also available as a separate PDF.

I have no special knowledge of the measurement of information flow in a network meta-analysis and so I read this paper as a biostatistician with a general interest in meta-analysis.

I like the idea of associating a descriptive measure of importance to each edge in a network and the authors of this paper present an interesting step in that direction, though my feeling is that this proposal will not turn out to be the final solution. I have two areas of concern, one is to do with terminology and the other with the scope of applicability of the proposal solution.

In criticising the terminology, I have to accept that the authors are largely following common practice and so my criticisms are partly aimed at the meta-analysis community. None the less there were three things that irritated me. Firstly, throughout the paper, proportions are referred to as percentages. Next, the title of the paper says it is about "the contribution of studies in network meta-analysis" while actually it is about the contribution of different effect estimates. Finally, the measure adopted is the weight given to each effect estimate when calculating the pooled estimate. In this paper and elsewhere in the meta-analysis literature, this weight is called a contribution. Perhaps I am being too pedantic but the weight and contribution are different ideas and it does not help with the terminology confuses them.

Now the proposed method. Taking the example from the paper, the authors note that the pooled estimate of the xy effect can be calculated using equation 1, where θ̂^I_xvyis the indirect estimate of xy along the path xy. Further it is true, at least for this example, that ΣL_iφi= 1 where L_i is the length of the path. So we can attach the weights φ to the edges of the network, sum them when an edge contributes to more than one estimate and the resulting weights will sum to one over the whole network.

This argument works for the example presented in the paper but it is not clear to me what conditions have to hold for it to work generally. The authors note in the paper that including a multi-arm trial in the meta-analysis would cause a problem, presumably because some of the direct or indirect estimates would not be independent. Are there any other conditions that have to hold? For example, can we have any structure of random effects in the meta-analysis model? What about the Bayesian models that are often used for network meta-analysis?

The authors take the matrix, H, which projects individual estimates such as xy, xy, uy, etc. into their predicted values under the meta-analysis model and they present an algorithm for converting those values into weights that are equivalent to the φ's.The algorithm is sensible and works for the simple example in the paper but one is again left wondering whether or not it works under all circumstances. After all, the algorithm is presented without any proof that it works.

My own feeling is that a contribution is best measured by the sledgehammer approach of analysing the network with and without a particular edge, but the authors' suggested approach is much less computationally demanding and I think that it would be appreciated by many applied researchers provided they were certain that it could be safely used with their particular network and their particular meta-analysis model.

Is the rationale for developing the new method (or application) clearly explained?

Yes
Is the description of the method technically sound?

Yes
Are sufficient details provided to allow replication of the method development and its use by others?

Yes
If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

Competing Interests: No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Author Response 13 Dec 2018

Adriani Nikolakopoulou, Institute of Social and Preventative Medicine (ISPM), University of Bern, Bern, Switzerland

13 Dec 2018

Author Response

We are grateful to Dr. John R. Thompson for his time and valuable comments and for recommending approval of our paper. We believe that we have been able to address ... Continue reading We are grateful to Dr. John R. Thompson for his time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report 16 Aug 2018
John R. Thompson, Department of Health Sciences, University of Leicester, Leicester, UK

Approved

I have no special knowledge of the measurement of information flow in a network meta-analysis and so I read this paper as a biostatistician with a general interest in meta-analysis.

I like the idea of associating a descriptive measure of importance to each edge in a network and the authors of this paper present an interesting step in that direction, though my feeling is that this proposal will not turn out to be the final solution. I have two areas of concern, one is to do with terminology and the other with the scope of applicability of the proposal solution.

In criticising the terminology, I have to accept that the authors are largely following common practice and so my criticisms are partly aimed at the meta-analysis community. None the less there were three things that irritated me. Firstly, throughout the paper, proportions are referred to as percentages. Next, the title of the paper says it is about "the contribution of studies in network meta-analysis" while actually it is about the contribution of different effect estimates. Finally, the measure adopted is the weight given to each effect estimate when calculating the pooled estimate. In this paper and elsewhere in the meta-analysis literature, this weight is called a contribution. Perhaps I am being too pedantic but the weight and contribution are different ideas and it does not help with the terminology confuses them.

Thank you for these interesting notes on meta-analysis terminology. At first, we agree that referring to “percentage contributions” might not be fitting and calling them “proportions” is more appropriate. We, thus, replaced throughout percentages with proportions.

Regarding the reporting of contribution of studies in the title of our paper, we have extended the method to quantify the contribution of studies as suggested by the first reviewer, Dr. Jochem König (comment 6). Thus, we left the title of the paper unmodified.

Weights and contributions are indeed sometimes used interchangeably. In this paper, however, we believe that the definition of matrix and the set of flows render the distinction clear. We only talk about “weights” of paths in the Introduction, where we have not yet explained the terminology to be used and thus it is convenient describing the elements of matrix as generalized weights from pairwise meta-analysis. We apologise in advance if we have misunderstood your comment.

Now the proposed method. Taking the example from the paper, the authors note that the pooled estimate of the xy effect can be calculated using equation 1, where θ̂^I_xvyis the indirect estimate of xy along the path xy. Further it is true, at least for this example, that ΣL_iφi = 1 where L_i is the length of the path. So we can attach the weights φ to the edges of the network, sum them when an edge contributes to more than one estimate and the resulting weights will sum to one over the whole network.

This argument works for the example presented in the paper but it is not clear to me what conditions have to hold for it to work generally. The authors note in the paper that including a multi-arm trial in the meta-analysis would cause a problem, presumably because some of the direct or indirect estimates would not be independent. Are there any other conditions that have to hold? For example, can we have any structure of random effects in the meta-analysis model? What about the Bayesian models that are often used for network meta-analysis?

Thank you for this insightful comment. We have clarified at the beginning of the Methods section that we consider “a simplified version of the matrix described by König et al. [1] that does not take into account the correlation induced by multi-arm trials.”

In this case, and considering the properties of the elements of set it always holds that Equation 3 can be re-written as Equation 5. We now clarify in the subsection ‘Proportion contributions of direct comparisons’:

“It follows from the properties of the elements of set that each NMA effect can be written as a linear combination of direct and indirect effects, in the form of Equation 5”.

We clarify at the end of ‘Algorithm to decompose flows into percentage contributions’ section that multi-arm studies can be handled if a modified matrix is used. We also clarify that the structure of random effects does not play a role in the calculations. In particular, we write:
“The starting point for the developed algorithm was a simplified version of the matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. [1] that extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm. The estimator of heterogeneity as well as the assumption of a different or common heterogeneity across comparisons in the network does not modify any aspect of the method.”

Within a Bayesian framework, the matrix is not defined. We added in the Discussion:
“The derivation of the contribution of sources of evidence in a Bayesian NMA is not straightforward, as no analogue to the matrix exists. The application of the method described in this paper, however, can be used to derive useful approximations of the contributions of studies.”

The authors take the matrix, H, which projects individual estimates such as xy, xy, uy, etc. into their predicted values under the meta-analysis model and they present an algorithm for converting those values into weights that are equivalent to the φ's.The algorithm is sensible and works for the simple example in the paper but one is again left wondering whether or not it works under all circumstances. After all, the algorithm is presented without any proof that it works.

Thank you for pointing this out. The algorithm we present in the ‘Algorithm to decompose flows into proportion contributions’ is described for the general case of a network of interventions containing any number of studies, treatments and loops. Indeed, we do not present a proof that it works under all circumstances. We have, however, applied the algorithm in numerous NMA applications apart from the simple example used in the results of this paper. Moreover, as mentioned in the Discussion, the web application CINeMA (Confidence In Network Meta-Analysis) [2], that aims to simplify the evaluation of confidence in the findings from NMA, uses the proportion contribution matrix as described in this paper. Users of CINeMA have not reported any case that the algorithm does not work. We are planning to examine the properties of the different approaches on the selection of direct paths in the algorithm in a follow up project (preliminary work is described in Supplementary File 3), which could be combined with a proof that the algorithm (and its modifications) works.

My own feeling is that a contribution is best measured by the sledgehammer approach of analysing the network with and without a particular edge, but the authors' suggested approach is much less computationally demanding and I think that it would be appreciated by many applied researchers provided they were certain that it could be safely used with their particular network and their particular meta-analysis model.

We totally agree that such ‘leave one out’ approaches would be also of interest. Relevant methods have been proposed by Krahn et al. [3] for contribution to the effect and by Riley et al. [4] for contribution to the variance. The former is useful in measuring how much a direct effect could bias the NMA effect and thus is not relevant in this case where we opted for a representation of contribution of pieces of evidence according to a study characteristic. The latter is based on the decomposition of Fisher’s information matrix and is associated with some issues that need further investigation.

We had already mentioned and discussed the Riley et al. approach in the Discussion. We now added:
“Krahn et al. define influence functions to describe the extent to which changes in study effects would be translated into NMA treatment effects [3].”
We are grateful to Dr. John R. Thompson for his time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report 16 Aug 2018
John R. Thompson, Department of Health Sciences, University of Leicester, Leicester, UK

Approved

I have no special knowledge of the measurement of information flow in a network meta-analysis and so I read this paper as a biostatistician with a general interest in meta-analysis.

I like the idea of associating a descriptive measure of importance to each edge in a network and the authors of this paper present an interesting step in that direction, though my feeling is that this proposal will not turn out to be the final solution. I have two areas of concern, one is to do with terminology and the other with the scope of applicability of the proposal solution.

In criticising the terminology, I have to accept that the authors are largely following common practice and so my criticisms are partly aimed at the meta-analysis community. None the less there were three things that irritated me. Firstly, throughout the paper, proportions are referred to as percentages. Next, the title of the paper says it is about "the contribution of studies in network meta-analysis" while actually it is about the contribution of different effect estimates. Finally, the measure adopted is the weight given to each effect estimate when calculating the pooled estimate. In this paper and elsewhere in the meta-analysis literature, this weight is called a contribution. Perhaps I am being too pedantic but the weight and contribution are different ideas and it does not help with the terminology confuses them.

Thank you for these interesting notes on meta-analysis terminology. At first, we agree that referring to “percentage contributions” might not be fitting and calling them “proportions” is more appropriate. We, thus, replaced throughout percentages with proportions.

Regarding the reporting of contribution of studies in the title of our paper, we have extended the method to quantify the contribution of studies as suggested by the first reviewer, Dr. Jochem König (comment 6). Thus, we left the title of the paper unmodified.

Weights and contributions are indeed sometimes used interchangeably. In this paper, however, we believe that the definition of matrix and the set of flows render the distinction clear. We only talk about “weights” of paths in the Introduction, where we have not yet explained the terminology to be used and thus it is convenient describing the elements of matrix as generalized weights from pairwise meta-analysis. We apologise in advance if we have misunderstood your comment.

Now the proposed method. Taking the example from the paper, the authors note that the pooled estimate of the xy effect can be calculated using equation 1, where θ̂^I_xvyis the indirect estimate of xy along the path xy. Further it is true, at least for this example, that ΣL_iφi = 1 where L_i is the length of the path. So we can attach the weights φ to the edges of the network, sum them when an edge contributes to more than one estimate and the resulting weights will sum to one over the whole network.

This argument works for the example presented in the paper but it is not clear to me what conditions have to hold for it to work generally. The authors note in the paper that including a multi-arm trial in the meta-analysis would cause a problem, presumably because some of the direct or indirect estimates would not be independent. Are there any other conditions that have to hold? For example, can we have any structure of random effects in the meta-analysis model? What about the Bayesian models that are often used for network meta-analysis?

Thank you for this insightful comment. We have clarified at the beginning of the Methods section that we consider “a simplified version of the matrix described by König et al. [1] that does not take into account the correlation induced by multi-arm trials.”

In this case, and considering the properties of the elements of set it always holds that Equation 3 can be re-written as Equation 5. We now clarify in the subsection ‘Proportion contributions of direct comparisons’:

“It follows from the properties of the elements of set that each NMA effect can be written as a linear combination of direct and indirect effects, in the form of Equation 5”.

We clarify at the end of ‘Algorithm to decompose flows into percentage contributions’ section that multi-arm studies can be handled if a modified matrix is used. We also clarify that the structure of random effects does not play a role in the calculations. In particular, we write:
“The starting point for the developed algorithm was a simplified version of the matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. [1] that extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm. The estimator of heterogeneity as well as the assumption of a different or common heterogeneity across comparisons in the network does not modify any aspect of the method.”

Within a Bayesian framework, the matrix is not defined. We added in the Discussion:
“The derivation of the contribution of sources of evidence in a Bayesian NMA is not straightforward, as no analogue to the matrix exists. The application of the method described in this paper, however, can be used to derive useful approximations of the contributions of studies.”

The authors take the matrix, H, which projects individual estimates such as xy, xy, uy, etc. into their predicted values under the meta-analysis model and they present an algorithm for converting those values into weights that are equivalent to the φ's.The algorithm is sensible and works for the simple example in the paper but one is again left wondering whether or not it works under all circumstances. After all, the algorithm is presented without any proof that it works.

Thank you for pointing this out. The algorithm we present in the ‘Algorithm to decompose flows into proportion contributions’ is described for the general case of a network of interventions containing any number of studies, treatments and loops. Indeed, we do not present a proof that it works under all circumstances. We have, however, applied the algorithm in numerous NMA applications apart from the simple example used in the results of this paper. Moreover, as mentioned in the Discussion, the web application CINeMA (Confidence In Network Meta-Analysis) [2], that aims to simplify the evaluation of confidence in the findings from NMA, uses the proportion contribution matrix as described in this paper. Users of CINeMA have not reported any case that the algorithm does not work. We are planning to examine the properties of the different approaches on the selection of direct paths in the algorithm in a follow up project (preliminary work is described in Supplementary File 3), which could be combined with a proof that the algorithm (and its modifications) works.

My own feeling is that a contribution is best measured by the sledgehammer approach of analysing the network with and without a particular edge, but the authors' suggested approach is much less computationally demanding and I think that it would be appreciated by many applied researchers provided they were certain that it could be safely used with their particular network and their particular meta-analysis model.

We totally agree that such ‘leave one out’ approaches would be also of interest. Relevant methods have been proposed by Krahn et al. [3] for contribution to the effect and by Riley et al. [4] for contribution to the variance. The former is useful in measuring how much a direct effect could bias the NMA effect and thus is not relevant in this case where we opted for a representation of contribution of pieces of evidence according to a study characteristic. The latter is based on the decomposition of Fisher’s information matrix and is associated with some issues that need further investigation.

We had already mentioned and discussed the Riley et al. approach in the Discussion. We now added:
“Krahn et al. define influence functions to describe the extent to which changes in study effects would be translated into NMA treatment effects [3].”
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 13 Dec 2018

Adriani Nikolakopoulou, Institute of Social and Preventative Medicine (ISPM), University of Bern, Bern, Switzerland

13 Dec 2018

Author Response

We are grateful to Dr. John R. Thompson for his time and valuable comments and for recommending approval of our paper. We believe that we have been able to address ... Continue reading We are grateful to Dr. John R. Thompson for his time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report 16 Aug 2018
John R. Thompson, Department of Health Sciences, University of Leicester, Leicester, UK

Approved

I have no special knowledge of the measurement of information flow in a network meta-analysis and so I read this paper as a biostatistician with a general interest in meta-analysis.

I like the idea of associating a descriptive measure of importance to each edge in a network and the authors of this paper present an interesting step in that direction, though my feeling is that this proposal will not turn out to be the final solution. I have two areas of concern, one is to do with terminology and the other with the scope of applicability of the proposal solution.

In criticising the terminology, I have to accept that the authors are largely following common practice and so my criticisms are partly aimed at the meta-analysis community. None the less there were three things that irritated me. Firstly, throughout the paper, proportions are referred to as percentages. Next, the title of the paper says it is about "the contribution of studies in network meta-analysis" while actually it is about the contribution of different effect estimates. Finally, the measure adopted is the weight given to each effect estimate when calculating the pooled estimate. In this paper and elsewhere in the meta-analysis literature, this weight is called a contribution. Perhaps I am being too pedantic but the weight and contribution are different ideas and it does not help with the terminology confuses them.

Thank you for these interesting notes on meta-analysis terminology. At first, we agree that referring to “percentage contributions” might not be fitting and calling them “proportions” is more appropriate. We, thus, replaced throughout percentages with proportions.

Regarding the reporting of contribution of studies in the title of our paper, we have extended the method to quantify the contribution of studies as suggested by the first reviewer, Dr. Jochem König (comment 6). Thus, we left the title of the paper unmodified.

Weights and contributions are indeed sometimes used interchangeably. In this paper, however, we believe that the definition of matrix and the set of flows render the distinction clear. We only talk about “weights” of paths in the Introduction, where we have not yet explained the terminology to be used and thus it is convenient describing the elements of matrix as generalized weights from pairwise meta-analysis. We apologise in advance if we have misunderstood your comment.

Now the proposed method. Taking the example from the paper, the authors note that the pooled estimate of the xy effect can be calculated using equation 1, where θ̂^I_xvyis the indirect estimate of xy along the path xy. Further it is true, at least for this example, that ΣL_iφi = 1 where L_i is the length of the path. So we can attach the weights φ to the edges of the network, sum them when an edge contributes to more than one estimate and the resulting weights will sum to one over the whole network.

This argument works for the example presented in the paper but it is not clear to me what conditions have to hold for it to work generally. The authors note in the paper that including a multi-arm trial in the meta-analysis would cause a problem, presumably because some of the direct or indirect estimates would not be independent. Are there any other conditions that have to hold? For example, can we have any structure of random effects in the meta-analysis model? What about the Bayesian models that are often used for network meta-analysis?

Thank you for this insightful comment. We have clarified at the beginning of the Methods section that we consider “a simplified version of the matrix described by König et al. [1] that does not take into account the correlation induced by multi-arm trials.”

In this case, and considering the properties of the elements of set it always holds that Equation 3 can be re-written as Equation 5. We now clarify in the subsection ‘Proportion contributions of direct comparisons’:

“It follows from the properties of the elements of set that each NMA effect can be written as a linear combination of direct and indirect effects, in the form of Equation 5”.

We clarify at the end of ‘Algorithm to decompose flows into percentage contributions’ section that multi-arm studies can be handled if a modified matrix is used. We also clarify that the structure of random effects does not play a role in the calculations. In particular, we write:
“The starting point for the developed algorithm was a simplified version of the matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. [1] that extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm. The estimator of heterogeneity as well as the assumption of a different or common heterogeneity across comparisons in the network does not modify any aspect of the method.”

Within a Bayesian framework, the matrix is not defined. We added in the Discussion:
“The derivation of the contribution of sources of evidence in a Bayesian NMA is not straightforward, as no analogue to the matrix exists. The application of the method described in this paper, however, can be used to derive useful approximations of the contributions of studies.”

The authors take the matrix, H, which projects individual estimates such as xy, xy, uy, etc. into their predicted values under the meta-analysis model and they present an algorithm for converting those values into weights that are equivalent to the φ's.The algorithm is sensible and works for the simple example in the paper but one is again left wondering whether or not it works under all circumstances. After all, the algorithm is presented without any proof that it works.

Thank you for pointing this out. The algorithm we present in the ‘Algorithm to decompose flows into proportion contributions’ is described for the general case of a network of interventions containing any number of studies, treatments and loops. Indeed, we do not present a proof that it works under all circumstances. We have, however, applied the algorithm in numerous NMA applications apart from the simple example used in the results of this paper. Moreover, as mentioned in the Discussion, the web application CINeMA (Confidence In Network Meta-Analysis) [2], that aims to simplify the evaluation of confidence in the findings from NMA, uses the proportion contribution matrix as described in this paper. Users of CINeMA have not reported any case that the algorithm does not work. We are planning to examine the properties of the different approaches on the selection of direct paths in the algorithm in a follow up project (preliminary work is described in Supplementary File 3), which could be combined with a proof that the algorithm (and its modifications) works.

My own feeling is that a contribution is best measured by the sledgehammer approach of analysing the network with and without a particular edge, but the authors' suggested approach is much less computationally demanding and I think that it would be appreciated by many applied researchers provided they were certain that it could be safely used with their particular network and their particular meta-analysis model.

We totally agree that such ‘leave one out’ approaches would be also of interest. Relevant methods have been proposed by Krahn et al. [3] for contribution to the effect and by Riley et al. [4] for contribution to the variance. The former is useful in measuring how much a direct effect could bias the NMA effect and thus is not relevant in this case where we opted for a representation of contribution of pieces of evidence according to a study characteristic. The latter is based on the decomposition of Fisher’s information matrix and is associated with some issues that need further investigation.

We had already mentioned and discussed the Riley et al. approach in the Discussion. We now added:
“Krahn et al. define influence functions to describe the extent to which changes in study effects would be translated into NMA treatment effects [3].”
We are grateful to Dr. John R. Thompson for his time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report 16 Aug 2018
John R. Thompson, Department of Health Sciences, University of Leicester, Leicester, UK

Approved

I have no special knowledge of the measurement of information flow in a network meta-analysis and so I read this paper as a biostatistician with a general interest in meta-analysis.

I like the idea of associating a descriptive measure of importance to each edge in a network and the authors of this paper present an interesting step in that direction, though my feeling is that this proposal will not turn out to be the final solution. I have two areas of concern, one is to do with terminology and the other with the scope of applicability of the proposal solution.

In criticising the terminology, I have to accept that the authors are largely following common practice and so my criticisms are partly aimed at the meta-analysis community. None the less there were three things that irritated me. Firstly, throughout the paper, proportions are referred to as percentages. Next, the title of the paper says it is about "the contribution of studies in network meta-analysis" while actually it is about the contribution of different effect estimates. Finally, the measure adopted is the weight given to each effect estimate when calculating the pooled estimate. In this paper and elsewhere in the meta-analysis literature, this weight is called a contribution. Perhaps I am being too pedantic but the weight and contribution are different ideas and it does not help with the terminology confuses them.

Thank you for these interesting notes on meta-analysis terminology. At first, we agree that referring to “percentage contributions” might not be fitting and calling them “proportions” is more appropriate. We, thus, replaced throughout percentages with proportions.

Regarding the reporting of contribution of studies in the title of our paper, we have extended the method to quantify the contribution of studies as suggested by the first reviewer, Dr. Jochem König (comment 6). Thus, we left the title of the paper unmodified.

Weights and contributions are indeed sometimes used interchangeably. In this paper, however, we believe that the definition of matrix and the set of flows render the distinction clear. We only talk about “weights” of paths in the Introduction, where we have not yet explained the terminology to be used and thus it is convenient describing the elements of matrix as generalized weights from pairwise meta-analysis. We apologise in advance if we have misunderstood your comment.

Now the proposed method. Taking the example from the paper, the authors note that the pooled estimate of the xy effect can be calculated using equation 1, where θ̂^I_xvyis the indirect estimate of xy along the path xy. Further it is true, at least for this example, that ΣL_iφi = 1 where L_i is the length of the path. So we can attach the weights φ to the edges of the network, sum them when an edge contributes to more than one estimate and the resulting weights will sum to one over the whole network.

This argument works for the example presented in the paper but it is not clear to me what conditions have to hold for it to work generally. The authors note in the paper that including a multi-arm trial in the meta-analysis would cause a problem, presumably because some of the direct or indirect estimates would not be independent. Are there any other conditions that have to hold? For example, can we have any structure of random effects in the meta-analysis model? What about the Bayesian models that are often used for network meta-analysis?

Thank you for this insightful comment. We have clarified at the beginning of the Methods section that we consider “a simplified version of the matrix described by König et al. [1] that does not take into account the correlation induced by multi-arm trials.”

In this case, and considering the properties of the elements of set it always holds that Equation 3 can be re-written as Equation 5. We now clarify in the subsection ‘Proportion contributions of direct comparisons’:

“It follows from the properties of the elements of set that each NMA effect can be written as a linear combination of direct and indirect effects, in the form of Equation 5”.

We clarify at the end of ‘Algorithm to decompose flows into percentage contributions’ section that multi-arm studies can be handled if a modified matrix is used. We also clarify that the structure of random effects does not play a role in the calculations. In particular, we write:
“The starting point for the developed algorithm was a simplified version of the matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. [1] that extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm. The estimator of heterogeneity as well as the assumption of a different or common heterogeneity across comparisons in the network does not modify any aspect of the method.”

Within a Bayesian framework, the matrix is not defined. We added in the Discussion:
“The derivation of the contribution of sources of evidence in a Bayesian NMA is not straightforward, as no analogue to the matrix exists. The application of the method described in this paper, however, can be used to derive useful approximations of the contributions of studies.”

The authors take the matrix, H, which projects individual estimates such as xy, xy, uy, etc. into their predicted values under the meta-analysis model and they present an algorithm for converting those values into weights that are equivalent to the φ's.The algorithm is sensible and works for the simple example in the paper but one is again left wondering whether or not it works under all circumstances. After all, the algorithm is presented without any proof that it works.

Thank you for pointing this out. The algorithm we present in the ‘Algorithm to decompose flows into proportion contributions’ is described for the general case of a network of interventions containing any number of studies, treatments and loops. Indeed, we do not present a proof that it works under all circumstances. We have, however, applied the algorithm in numerous NMA applications apart from the simple example used in the results of this paper. Moreover, as mentioned in the Discussion, the web application CINeMA (Confidence In Network Meta-Analysis) [2], that aims to simplify the evaluation of confidence in the findings from NMA, uses the proportion contribution matrix as described in this paper. Users of CINeMA have not reported any case that the algorithm does not work. We are planning to examine the properties of the different approaches on the selection of direct paths in the algorithm in a follow up project (preliminary work is described in Supplementary File 3), which could be combined with a proof that the algorithm (and its modifications) works.

My own feeling is that a contribution is best measured by the sledgehammer approach of analysing the network with and without a particular edge, but the authors' suggested approach is much less computationally demanding and I think that it would be appreciated by many applied researchers provided they were certain that it could be safely used with their particular network and their particular meta-analysis model.

We totally agree that such ‘leave one out’ approaches would be also of interest. Relevant methods have been proposed by Krahn et al. [3] for contribution to the effect and by Riley et al. [4] for contribution to the variance. The former is useful in measuring how much a direct effect could bias the NMA effect and thus is not relevant in this case where we opted for a representation of contribution of pieces of evidence according to a study characteristic. The latter is based on the decomposition of Fisher’s information matrix and is associated with some issues that need further investigation.

We had already mentioned and discussed the Riley et al. approach in the Discussion. We now added:
“Krahn et al. define influence functions to describe the extent to which changes in study effects would be translated into NMA treatment effects [3].”
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 14 Aug 2018

Annette M. O'Connor, Department of Veterinary Diagnostic and Production Animal Medicine, College of Veterinary Medicine, Iowa State University, Ames, IA, USA

Approved

https://doi.org/10.5256/f1000research.16071.r36470

This paper described an updated approach to deriving the percentage contributions of the direct comparison used in treatment estimates. In the interests of full disclosure, I am not a statistician and was reviewing from the view point of an applied user of NMA. There are several papers on this topic in recent years, and it is an active body of work in this area. The article is straightforward to read, although made considerably more readable if one is aware of the content in references, 3,4 and 5.

I have to admit that this is an example of why I appreciate the open review system. I enjoyed the paper and found it easy to read and follow, but the others reviewer's comments were a significant contribution also. I don't have any additional critique of the proposed approach, but I look forward to seeing how the investigators address those or provided some discussion, as this would help the less statistically inclined reader like me. Again, the reason I wanted this, is that I am an applied user and such discussions are very helpful. In particular, I would like to have seen some discussion of the 1st comment. I would also like to see if the comment about Figure 3 could be incorporated - perhaps not feasible.

Concerning the approach to calculating the % contribution and how to weight the flows, I came away with the impression that the decision of equal split of the stream flow as arbitrary (perhaps not precisely the correct term) or perhaps pragmatic is better. Therefore, it is not surprising this is a debatable approach. Again, this was another reason why I am looking forward to see the investigators responses to the 1st reviewer's comments. I agree that the comparison of the methods of deriving the percentage contribution would be of interest but was not expecting that to be included. I look forward to this comparison being published.

Is the rationale for developing the new method (or application) clearly explained?

Yes
Is the description of the method technically sound?

Yes
Are sufficient details provided to allow replication of the method development and its use by others?

Yes
If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Epidemiology, infectious diseases, research synthesis

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Author Response 13 Dec 2018

Adriani Nikolakopoulou, Institute of Social and Preventative Medicine (ISPM), University of Bern, Bern, Switzerland

13 Dec 2018

Author Response

We are grateful to Dr. Annette M. O’Connor for her time and valuable comments and for recommending approval of our paper. We believe that we have been able to address ... Continue reading We are grateful to Dr. Annette M. O’Connor for her time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report: Annette M. O'Connor, Department of Veterinary Diagnostic and Production Animal Medicine, College of Veterinary Medicine, Iowa State University, Ames, IA, USA

Approved

This paper described an updated approach to deriving the percentage contributions of the direct comparison used in treatment estimates. In the interests of full disclosure, I am not a statistician and was reviewing from the view point of an applied user of NMA. There are several papers on this topic in recent years, and it is an active body of work in this area. The article is straightforward to read, although made considerably more readable if one is aware of the content in references, 3,4 and 5.

I have to admit that this is an example of why I appreciate the open review system. I enjoyed the paper and found it easy to read and follow, but the others reviewer's comments were a significant contribution also. I don't have any additional critique of the proposed approach, but I look forward to seeing how the investigators address those or provided some discussion, as this would help the less statistically inclined reader like me. Again, the reason I wanted this, is that I am an applied user and such discussions are very helpful. In particular, I would like to have seen some discussion of the 1st comment. I would also like to see if the comment about Figure 3 could be incorporated - perhaps not feasible.

Thank you very much for your positive feedback. Our answers to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2. In summary, regarding his first comment, we distinguish between representation of contribution of pieces of evidence according to a study characteristic (addressed in this paper) and quantification of the impact of within-study bias in NMA treatment effects (subject of future research). A new paragraph in the Discussion highlighting these concerns reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the proportion contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its proportion contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

As we explain in detail in our response to Dr. Jochem König (comment 4), we did not amend Figure 3 to incorporate directionality.

Concerning the approach to calculating the % contribution and how to weight the flows, I came away with the impression that the decision of equal split of the stream flow as arbitrary (perhaps not precisely the correct term) or perhaps pragmatic is better. Therefore, it is not surprising this is a debatable approach. Again, this was another reason why I am looking forward to see the investigators responses to the 1st reviewer's comments. I agree that the comparison of the methods of deriving the percentage contribution would be of interest but was not expecting that to be included. I look forward to this comparison being published.

We agree that equal splitting of stream flow is to some extent arbitrary but also pragmatic. We say in the Discussion:

“An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”
We are grateful to Dr. Annette M. O’Connor for her time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report: Annette M. O'Connor, Department of Veterinary Diagnostic and Production Animal Medicine, College of Veterinary Medicine, Iowa State University, Ames, IA, USA

Approved

This paper described an updated approach to deriving the percentage contributions of the direct comparison used in treatment estimates. In the interests of full disclosure, I am not a statistician and was reviewing from the view point of an applied user of NMA. There are several papers on this topic in recent years, and it is an active body of work in this area. The article is straightforward to read, although made considerably more readable if one is aware of the content in references, 3,4 and 5.

I have to admit that this is an example of why I appreciate the open review system. I enjoyed the paper and found it easy to read and follow, but the others reviewer's comments were a significant contribution also. I don't have any additional critique of the proposed approach, but I look forward to seeing how the investigators address those or provided some discussion, as this would help the less statistically inclined reader like me. Again, the reason I wanted this, is that I am an applied user and such discussions are very helpful. In particular, I would like to have seen some discussion of the 1st comment. I would also like to see if the comment about Figure 3 could be incorporated - perhaps not feasible.

Thank you very much for your positive feedback. Our answers to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2. In summary, regarding his first comment, we distinguish between representation of contribution of pieces of evidence according to a study characteristic (addressed in this paper) and quantification of the impact of within-study bias in NMA treatment effects (subject of future research). A new paragraph in the Discussion highlighting these concerns reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the proportion contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its proportion contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

As we explain in detail in our response to Dr. Jochem König (comment 4), we did not amend Figure 3 to incorporate directionality.

Concerning the approach to calculating the % contribution and how to weight the flows, I came away with the impression that the decision of equal split of the stream flow as arbitrary (perhaps not precisely the correct term) or perhaps pragmatic is better. Therefore, it is not surprising this is a debatable approach. Again, this was another reason why I am looking forward to see the investigators responses to the 1st reviewer's comments. I agree that the comparison of the methods of deriving the percentage contribution would be of interest but was not expecting that to be included. I look forward to this comparison being published.

We agree that equal splitting of stream flow is to some extent arbitrary but also pragmatic. We say in the Discussion:

“An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 13 Dec 2018

Adriani Nikolakopoulou, Institute of Social and Preventative Medicine (ISPM), University of Bern, Bern, Switzerland

13 Dec 2018

Author Response

We are grateful to Dr. Annette M. O’Connor for her time and valuable comments and for recommending approval of our paper. We believe that we have been able to address ... Continue reading We are grateful to Dr. Annette M. O’Connor for her time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report: Annette M. O'Connor, Department of Veterinary Diagnostic and Production Animal Medicine, College of Veterinary Medicine, Iowa State University, Ames, IA, USA

Approved

This paper described an updated approach to deriving the percentage contributions of the direct comparison used in treatment estimates. In the interests of full disclosure, I am not a statistician and was reviewing from the view point of an applied user of NMA. There are several papers on this topic in recent years, and it is an active body of work in this area. The article is straightforward to read, although made considerably more readable if one is aware of the content in references, 3,4 and 5.

I have to admit that this is an example of why I appreciate the open review system. I enjoyed the paper and found it easy to read and follow, but the others reviewer's comments were a significant contribution also. I don't have any additional critique of the proposed approach, but I look forward to seeing how the investigators address those or provided some discussion, as this would help the less statistically inclined reader like me. Again, the reason I wanted this, is that I am an applied user and such discussions are very helpful. In particular, I would like to have seen some discussion of the 1st comment. I would also like to see if the comment about Figure 3 could be incorporated - perhaps not feasible.

Thank you very much for your positive feedback. Our answers to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2. In summary, regarding his first comment, we distinguish between representation of contribution of pieces of evidence according to a study characteristic (addressed in this paper) and quantification of the impact of within-study bias in NMA treatment effects (subject of future research). A new paragraph in the Discussion highlighting these concerns reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the proportion contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its proportion contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

As we explain in detail in our response to Dr. Jochem König (comment 4), we did not amend Figure 3 to incorporate directionality.

Concerning the approach to calculating the % contribution and how to weight the flows, I came away with the impression that the decision of equal split of the stream flow as arbitrary (perhaps not precisely the correct term) or perhaps pragmatic is better. Therefore, it is not surprising this is a debatable approach. Again, this was another reason why I am looking forward to see the investigators responses to the 1st reviewer's comments. I agree that the comparison of the methods of deriving the percentage contribution would be of interest but was not expecting that to be included. I look forward to this comparison being published.

We agree that equal splitting of stream flow is to some extent arbitrary but also pragmatic. We say in the Discussion:

“An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”
We are grateful to Dr. Annette M. O’Connor for her time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report: Annette M. O'Connor, Department of Veterinary Diagnostic and Production Animal Medicine, College of Veterinary Medicine, Iowa State University, Ames, IA, USA

Approved

This paper described an updated approach to deriving the percentage contributions of the direct comparison used in treatment estimates. In the interests of full disclosure, I am not a statistician and was reviewing from the view point of an applied user of NMA. There are several papers on this topic in recent years, and it is an active body of work in this area. The article is straightforward to read, although made considerably more readable if one is aware of the content in references, 3,4 and 5.

I have to admit that this is an example of why I appreciate the open review system. I enjoyed the paper and found it easy to read and follow, but the others reviewer's comments were a significant contribution also. I don't have any additional critique of the proposed approach, but I look forward to seeing how the investigators address those or provided some discussion, as this would help the less statistically inclined reader like me. Again, the reason I wanted this, is that I am an applied user and such discussions are very helpful. In particular, I would like to have seen some discussion of the 1st comment. I would also like to see if the comment about Figure 3 could be incorporated - perhaps not feasible.

Thank you very much for your positive feedback. Our answers to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2. In summary, regarding his first comment, we distinguish between representation of contribution of pieces of evidence according to a study characteristic (addressed in this paper) and quantification of the impact of within-study bias in NMA treatment effects (subject of future research). A new paragraph in the Discussion highlighting these concerns reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the proportion contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its proportion contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

As we explain in detail in our response to Dr. Jochem König (comment 4), we did not amend Figure 3 to incorporate directionality.

Concerning the approach to calculating the % contribution and how to weight the flows, I came away with the impression that the decision of equal split of the stream flow as arbitrary (perhaps not precisely the correct term) or perhaps pragmatic is better. Therefore, it is not surprising this is a debatable approach. Again, this was another reason why I am looking forward to see the investigators responses to the 1st reviewer's comments. I agree that the comparison of the methods of deriving the percentage contribution would be of interest but was not expecting that to be included. I look forward to this comparison being published.

We agree that equal splitting of stream flow is to some extent arbitrary but also pragmatic. We say in the Discussion:

“An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”
Competing Interests: No competing interests were disclosed. Close
Report a concern

Views

Reviewer Report 13 Jun 2018

Jochem König, Division of Paediatric Epidemiology, Institute of Medical Biostatistics, Epidemiology, and Informatics (IMBEI), Johannes Gutenberg University Medical Center, Mainz, Germany

Approved with Reservations

https://doi.org/10.5256/f1000research.16071.r34177

The authors undertake a redefinition of the term ‘percentage contribution’ in network meta-analysis. In ordinary meta-analysis of randomized clinical trials, pooled estimates of a treatment effect can be represented as a weighted mean of treatment effects from

The authors undertake a redefinition of the term ‘percentage contribution’ in network meta-analysis. In ordinary meta-analysis of randomized clinical trials, pooled estimates of a treatment effect can be represented as a weighted mean of treatment effects from all contributing trials. Weights are positive and sum to one and naturally constitute the ‘percentage contribution’ of each study. In network meta-analysis treatment effect estimates are defined for all pairs of treatments. Each can be represented as a two-stage estimate, firstly pooling all trials comparing the same set of treatments, respectively, which gives direct effect estimates, and then pooling direct estimates into a network based mixed treatment comparison. The linear coefficients, used in the second stage constitute the ‘contribution of direct evidence to a network based comparison’. They can be read off, from the H matrix, as described by the authors. They share some but not all aspects with weights of a two-armed meta-analysis. Aspects shared in some sense are the flow properties described by the authors and in citation [4]. The aspect not shared, is that summation to one.

Indeed, in a purely linear graph connecting two extreme treatments through a chain of, say 10, pairwise intermediary comparisons, the network estimate is the sum of all direct effects. Any bias present in one direct effect estimate translates 1:1 into the network based estimate. Accordingly, the entries of the H matrix are all equal to 1 in the relevant row.

Both, Salanti [3] and the present paper aim to rescale or transform the rows of the H matrix in order to achieve a ‘sum to one’ condition. In the case of the linear graph, the percentage contribution is 10% for each direct comparison. I am not convinced, that this number adequately captures the role of each direct effect estimate in this example.

Consider now, an example where two treatments are connected through three paths with one, two, and three edges, respectively, each with flow 1/3. Then each direct comparison has the same influence on the network based treatment effect estimate: A bias of 1 unit in one direct estimate translates into a bias of 1/3 unit in the network based estimate. These proportions are conserved in Salanti’s notion of percentage contribution, but not so in the newly introduced concept. Salanti [3] attaches a percentage contribution of 1/6 to each comparison just as if we had pooled six equally precise trials comparing the same pair of treatments. But the influence of direct comparisons is different in this network and should be characterized by the number 1/3.

That is why I have objections against Salanti’s [3] concept of percentage contribution and even more so against the newly introduced concept, which attaches unequal contribution quantities of 1/9, 1/6 and 1/3, respectively.

The new method should not be introduced into routine of network meta-analysis. If this paper is going to be indexed, its draw backs and caveats should be very clearly set out.
The concept of streams is a nice one. It allows to represent a network based treatment effect as a weighted sum of a direct and indirect effect estimates each corresponding to one path from source to sink. Note however that these effects are stochastically interdependent and hence, the aggregation of streams is different from the usual process of pooling evidence.
To be somewhat less destructive, I propose to arrange the network graph with the source to the left and the sink to the right and all other vertices placed, such that all directed edges point from left to right. Then, due to the flow properties, any vertical cross-section gives a set of flows that sum to one. In that sense, the untransformed rows of the H matrix already contain percentage contributions. Displays used to present voter hiking can be used to represent these flows (see e.g. www.zeit.de/politik/deutschland/2017-05/waehlerwanderung-nrw-landtagswahl-cdu-spd-fdp).
I propose to modify Table 3 and Figure 3 in a way that the direction of influence becomes clear. Does a positive bias in one contributing direct effect translate into a positive or negative bias of the network based effect estimate? The colored fields in Figure 3 could be labelled with meaningfully ordered pairs of treatment letters.
Discussion “The calculation of the percentages uses the estimated variances of direct effects and thus incorporate associated uncertainty in their estimation.”
Note however, that in a linear network each direct effect estimate contributes equally, irrespective of the size and precision of the study.
Discussion “Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates.”
The two stage approach discussed in this paper is not essential. Both the flows and the percentage contribution defined in the paper can be distributed to single trials according to the weights used when pooling them into direct effect estimates. Then the risk of bias of single trials can be analyzed. A display similar to Figure 3 is possible that contains separately colored fields for individual studies.
Discussion, 2^nd paragraph: Concerning multi-armed designs, König et al.[4] have defined the flow into and out of the vertices in a way that does not depend on the choice of reference treatment.

Supplementary File 3:
1st paragraph: What is the vector space of G_xy over R?
2nd paragraph: ‘The number of directed paths’ obviously refers to the number of directed paths „which suffice to ‘spend’ the flow from source to sink”.

The authors state that, if no bridges exist, this number equals to df+1. However, it depends on the flows, too. Consider a network composed of two squares connected at one vertex. It has df=2 inconsistency degrees of freedom and generally 3 paths are needed to spend the flow between the two most distant vertices. However, if all edges have equal flows, only two paths are needed. Moreover, the connecting vertex has the same effect as the bridging edges. Hence the argument has to be extended to the case of graphs that can be split by cutting through a vertex.

Note also, that a network estimate that compares treatments from different subgroups of treatments connected through a bridge or a ‘breaking’ vertex can be written as the sum of three respective two subnetwork effect estimates: one comparing the source treatment to the bridge post, one for the bridge, if present, and one comparing the second bridge post to the sink. Because it is an unweighted sum, the contribution of the three (or two) summands should be equal.

Is the rationale for developing the new method (or application) clearly explained?

Yes
Is the description of the method technically sound?

Yes
Are sufficient details provided to allow replication of the method development and its use by others?

Yes
If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

Competing Interests: No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 03 Sep 2018

Adriani Nikolakopoulou, Institute of Social and Preventative Medicine (ISPM), University of Bern, Bern, Switzerland

03 Sep 2018

Author Response
We are grateful to Dr. Jochem König for the time and effort he spent to review our paper. We believe that his valuable comments and suggestions have substantially improved the ... Continue reading
We are grateful to Dr. Jochem König for the time and effort he spent to review our paper. We believe that his valuable comments and suggestions have substantially improved the manuscript. We have addressed all his comments and revised the paper accordingly. Below you can find detailed replies to the reviewer’s suggestions and comments.

Referee Report 13 Jun 2018
Jochem König, Division of Paediatric Epidemiology, Institute of Medical Biostatistics, Epidemiology, and Informatics (IMBEI), Johannes Gutenberg University Medical Center, Mainz, Germany

Approved with Reservations

1. The authors undertake a redefinition of the term ‘percentage contribution’ in network meta-analysis. In ordinary meta-analysis of randomized clinical trials, pooled estimates of a treatment effect can be represented as a weighted mean of treatment effects from all contributing trials. Weights are positive and sum to one and naturally constitute the ‘percentage contribution’ of each study. In network meta-analysis treatment effect estimates are defined for all pairs of treatments. Each can be represented as a two-stage estimate, firstly pooling all trials comparing the same set of treatments, respectively, which gives direct effect estimates, and then pooling direct estimates into a network based mixed treatment comparison. The linear coefficients, used in the second stage constitute the ‘contribution of direct evidence to a network based comparison’. They can be read off, from the H matrix, as described by the authors. They share some but not all aspects with weights of a two-armed meta-analysis. Aspects shared in some sense are the flow properties described by the authors and in citation [4]. The aspect not shared, is that summation to one.

Indeed, in a purely linear graph connecting two extreme treatments through a chain of, say 10, pairwise intermediary comparisons, the network estimate is the sum of all direct effects. Any bias present in one direct effect estimate translates 1:1 into the network based estimate. Accordingly, the entries of the H matrix are all equal to 1 in the relevant row.

Both, Salanti [3] and the present paper aim to rescale or transform the rows of the H matrix in order to achieve a ‘sum to one’ condition. In the case of the linear graph, the percentage contribution is 10% for each direct comparison. I am not convinced, that this number adequately captures the role of each direct effect estimate in this example.

Consider now, an example where two treatments are connected through three paths with one, two, and three edges, respectively, each with flow 1/3. Then each direct comparison has the same influence on the network based treatment effect estimate: A bias of 1 unit in one direct estimate translates into a bias of 1/3 unit in the network based estimate. These proportions are conserved in Salanti’s notion of percentage contribution, but not so in the newly introduced concept. Salanti [3] attaches a percentage contribution of 1/6 to each comparison just as if we had pooled six equally precise trials comparing the same pair of treatments. But the influence of direct comparisons is different in this network and should be characterized by the number 1/3.

That is why I have objections against Salanti’s [3] concept of percentage contribution and even more so against the newly introduced concept, which attaches unequal contribution quantities of 1/9, 1/6 and 1/3, respectively.

The new method should not be introduced into routine of network meta-analysis. If this paper is going to be indexed, its draw backs and caveats should be very clearly set out.

Thank you for this insightful comment. We agree with the point you make which we find that enlightens potential applications or extensions of the presented method to account for bias in network meta-analysis.
We would like to make a distinction between 'representation of the contribution of pieces of evidence in a 0 to 1 scale' and 'using this percentage contributions to represent and measure the impact and directionality of potential bias'. In this paper, we focus on the first aspect of translating the matrix into percentages and we subsequently apply it to depict the contribution of pieces of evidence according to a study characteristic. Our aim is to give a representation of the evidence that each comparison –or study, see our response to comment 6 below- contributes to the estimation of each NMA treatment effect without considering how this evidence could bias the results. Note that the bar plot of figure 3 gives a picture of the contribution of high, moderate or low risk of bias evidence without giving insight on the amount or direction of bias.

Quantification of bias in the direct evidence and its impact on the NMA estimates is subject of further research but not addressed within this paper. We have now added a paragraph in the Discussion to highlight the limitations of the presented method as you recommend. The paragraph reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the percentage contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its percentage contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

2. The concept of streams is a nice one. It allows to represent a network based treatment effect as a weighted sum of a direct and indirect effect estimates each corresponding to one path from source to sink. Note however that these effects are stochastically interdependent and hence, the aggregation of streams is different from the usual process of pooling evidence.

Thank you for this comment; indeed the aggregation of streams is not equivalent to the aggregation of studies in a pairwise meta-analysis. We write in ‘Percentage contributions of direct comparisons’:

“It can be shown that an NMA effect is a linear combination of the direct effects combined within paths.”

And we now added in the same section:

“Equation 5 represents θ_xy^N as a weighted sum of direct and two indirect effect estimates; the effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis.”

3. To be somewhat less destructive, I propose to arrange the network graph with the source to the left and the sink to the right and all other vertices placed, such that all directed edges point from left to right. Then, due to the flow properties, any vertical cross-section gives a set of flows that sum to one. In that sense, the untransformed rows of the H matrix already contain percentage contributions. Displays used to present voter hiking can be used to represent these flows (see e.g. www.zeit.de/politik/deutschland/2017-05/waehlerwanderung-nrw-landtagswahl-cdu-spd-fdp).

Thank you, we agree that horizontal representation is more intuitive. We re-arranged figures 1 and 2 accordingly.

4. I propose to modify Table 3 and Figure 3 in a way that the direction of influence becomes clear. Does a positive bias in one contributing direct effect translate into a positive or negative bias of the network based effect estimate? The colored fields in Figure 3 could be labelled with meaningfully ordered pairs of treatment letters.

Thank you very much for this comment which we believe is related to your comment 1. We agree that such a representation would be useful but we believe it could confuse readers with respect to the scope of this paper which is not to quantify the impact of within-study bias. As said in comment 1, the scope of this paper is to represent the percentage contributions of each piece of evidence; combining this information with information of study characteristics in a figure such as figure 3 may be a useful representation of the contribution of these characteristics to the estimation of NMA treatment effects. The influence of these characteristics, though, is not straightforwardly derived and it is subject to further research. Thus, we would prefer to leave Table 3 and Figure 3 unmodified and construct such representations of the direction of bias in future work where we aim to develop models to account for study-level bias using the presented methodology.

Please also see the reply to comment 1 and the addition to the Discussion.

5. Discussion “The calculation of the percentages uses the estimated variances of direct effects and thus incorporate associated uncertainty in their estimation.”
Note however, that in a linear network each direct effect estimate contributes equally, irrespective of the size and precision of the study.

We agree this statement is confusing and hence deleted this phrase from the Discussion. The paragraph now reads:

“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”

6. Discussion “Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates.”
The two stage approach discussed in this paper is not essential. Both the flows and the percentage contribution defined in the paper can be distributed to single trials according to the weights used when pooling them into direct effect estimates. Then the risk of bias of single trials can be analyzed. A display similar to Figure 3 is possible that contains separately colored fields for individual studies.

Thank you for this comment; we agree that such a distribution of percentage contributions to single trials is possible and relevant and we have updated the paper accordingly.

1.We added an extra paragraph/subsection ‘Percentage study contributions’ where we elaborate on the process you describe. The new paragraph reads:
“Matrix P (Table 3) shows the percentage contributions of each direct comparison to each NMA treatment effect estimate. These percentages can be distributed to individual studies within each comparison according to their weights from direct meta-analyses. For example, p_xy=63.5% and there are two studies examining the xy comparison. The individual study weights for the two studies are 0.69 and 1.54 resulting to study percentage contributions of (0.69 /(0.69+1.54))*63.5%=19.6% and (1.54/(0.69+1.54))*63.5%=43.8% to the xy NMA treatment effect estimate. The application of this process to the entire matrix P leads to the matrix P^* shown in Table 4. Adjusted weights as proposed by Rücker & Schwarzer (9) are used for multi-arm studies.”

2.Section ‘Using percentage contributions to quantify the impact of a characteristic in a direct comparison’ has been renamed to ‘Using percentage study contributions to quantify the impact of a characteristic in a direct comparison’ and follows the new ‘Percentage study contributions’ section. Text has been updated to refer to studies instead of direct comparisons and Figure 3 (and the new Table 4) now displays percentage study contributions to NMA treatment effect estimates.

3.The CINeMA (Confidence In Network Meta-Analysis) http://cinema.ispm.ch/ software has also been updated to display the bar chart using study contributions.

4. The first sentence of the Discussion now reads:
“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA.”

7. Discussion, 2^nd paragraph: Concerning multi-armed designs, König et al.[4] have defined the flow into and out of the vertices in a way that does not depend on the choice of reference treatment.

Thank you for this comment, we could indeed use the H matrix as described in section 3.3.1 of König et al.[4] (and implemented in netmeta as H.tilde) for multi-arm designs; some concerns about the interpretability of such a matrix have previously precluded us of doing so. In fact, the algorithm could be applied to any H matrix as long as its rows have the flow properties described in König et al.[4] and in our section ‘Comparison graph’. We have removed the discussion on multi-arm trials from the Discussion and added the following paragraph at the end of the Methods section:

“The starting point for the developed algorithm was a simplified version of the H matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. (4) that properly accommodates multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm.”

Supplementary File 3:
1st paragraph: What is the vector space of G_xy over R?

Thank you for this comment; as it was not clear and in order to simplify this part, we eliminated the notions of ‘vector space’ and ‘basis’ in the document. It now reads:
“However, in the general case, the number of paths used in the algorithm is smaller than the number of elements of Π. The number of directed paths which suffice to ‘spend’ the flow from source to sink is less or equal to df+1 where df is the inconsistency degrees of freedom in the Lu and Ades inconsistency model (1), df=D-T+1.”

2nd paragraph: ‘The number of directed paths’ obviously refers to the number of directed paths „which suffice to ‘spend’ the flow from source to sink”.

Thank you, we corrected it.

The authors state that, if no bridges exist, this number equals to df+1. However, it depends on the flows, too. Consider a network composed of two squares connected at one vertex. It has df=2 inconsistency degrees of freedom and generally 3 paths are needed to spend the flow between the two most distant vertices. However, if all edges have equal flows, only two paths are needed. Moreover, the connecting vertex has the same effect as the bridging edges. Hence the argument has to be extended to the case of graphs that can be split by cutting through a vertex.
Note also, that a network estimate that compares treatments from different subgroups of treatments connected through a bridge or a ‘breaking’ vertex can be written as the sum of three respective two subnetwork effect estimates: one comparing the source treatment to the bridge post, one for the bridge, if present, and one comparing the second bridge post to the sink. Because it is an unweighted sum, the contribution of the three (or two) summands should be equal.

Thank you very much for these interesting notes with which we totally agree. We extended the ‘Number of directed paths’ section in Supplementary File 3 to include discussion on the ‘breaking’ vertex situation and the consequences you describe. In particular we wrote:

“Another situation where the number of directed paths which suffice to ‘spend’ the flow from source to sink is less than df+1 elements occurs when a vertex has at least two inflows and two outflows, all equal between them. In such a situation the flow from source to sink may be spent in less than df+1 directed paths. Such a vertex will be called breaking vertex. Note that an NMA treatment effect estimate for treatments separated by a bridge or a breaking vertex (say treatments x and y) can be seen as an “indirect” comparison through either the intermediate bridge (constituting by treatments b₁ and b2) or the intermediate breaking vertex (b0).
In particular, in the case of a bridge, the NMA treatment effect estimate is derived as
θ_xy^N=θ_xb1^N+θ_b1b0^N+θ_b2y^N
and in the case of a breaking vertex the NMA treatment effect estimate is
θ_xy^N=θ_xb0^N+θ_b0y^N
It is, thus, implied that in such a case the separate subnetworks contribute equally to the estimation of θ_xy^N.”

Is the rationale for developing the new method (or application) clearly explained?

Yes

Is the description of the method technically sound?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly
Competing Interests: No competing interests were disclosed.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
We are grateful to Dr. Jochem König for the time and effort he spent to review our paper. We believe that his valuable comments and suggestions have substantially improved the manuscript. We have addressed all his comments and revised the paper accordingly. Below you can find detailed replies to the reviewer’s suggestions and comments.

Referee Report 13 Jun 2018
Jochem König, Division of Paediatric Epidemiology, Institute of Medical Biostatistics, Epidemiology, and Informatics (IMBEI), Johannes Gutenberg University Medical Center, Mainz, Germany

Approved with Reservations

1. The authors undertake a redefinition of the term ‘percentage contribution’ in network meta-analysis. In ordinary meta-analysis of randomized clinical trials, pooled estimates of a treatment effect can be represented as a weighted mean of treatment effects from all contributing trials. Weights are positive and sum to one and naturally constitute the ‘percentage contribution’ of each study. In network meta-analysis treatment effect estimates are defined for all pairs of treatments. Each can be represented as a two-stage estimate, firstly pooling all trials comparing the same set of treatments, respectively, which gives direct effect estimates, and then pooling direct estimates into a network based mixed treatment comparison. The linear coefficients, used in the second stage constitute the ‘contribution of direct evidence to a network based comparison’. They can be read off, from the H matrix, as described by the authors. They share some but not all aspects with weights of a two-armed meta-analysis. Aspects shared in some sense are the flow properties described by the authors and in citation [4]. The aspect not shared, is that summation to one.

Indeed, in a purely linear graph connecting two extreme treatments through a chain of, say 10, pairwise intermediary comparisons, the network estimate is the sum of all direct effects. Any bias present in one direct effect estimate translates 1:1 into the network based estimate. Accordingly, the entries of the H matrix are all equal to 1 in the relevant row.

Both, Salanti [3] and the present paper aim to rescale or transform the rows of the H matrix in order to achieve a ‘sum to one’ condition. In the case of the linear graph, the percentage contribution is 10% for each direct comparison. I am not convinced, that this number adequately captures the role of each direct effect estimate in this example.

Consider now, an example where two treatments are connected through three paths with one, two, and three edges, respectively, each with flow 1/3. Then each direct comparison has the same influence on the network based treatment effect estimate: A bias of 1 unit in one direct estimate translates into a bias of 1/3 unit in the network based estimate. These proportions are conserved in Salanti’s notion of percentage contribution, but not so in the newly introduced concept. Salanti [3] attaches a percentage contribution of 1/6 to each comparison just as if we had pooled six equally precise trials comparing the same pair of treatments. But the influence of direct comparisons is different in this network and should be characterized by the number 1/3.

That is why I have objections against Salanti’s [3] concept of percentage contribution and even more so against the newly introduced concept, which attaches unequal contribution quantities of 1/9, 1/6 and 1/3, respectively.

The new method should not be introduced into routine of network meta-analysis. If this paper is going to be indexed, its draw backs and caveats should be very clearly set out.

Thank you for this insightful comment. We agree with the point you make which we find that enlightens potential applications or extensions of the presented method to account for bias in network meta-analysis.
We would like to make a distinction between 'representation of the contribution of pieces of evidence in a 0 to 1 scale' and 'using this percentage contributions to represent and measure the impact and directionality of potential bias'. In this paper, we focus on the first aspect of translating the matrix into percentages and we subsequently apply it to depict the contribution of pieces of evidence according to a study characteristic. Our aim is to give a representation of the evidence that each comparison –or study, see our response to comment 6 below- contributes to the estimation of each NMA treatment effect without considering how this evidence could bias the results. Note that the bar plot of figure 3 gives a picture of the contribution of high, moderate or low risk of bias evidence without giving insight on the amount or direction of bias.

Quantification of bias in the direct evidence and its impact on the NMA estimates is subject of further research but not addressed within this paper. We have now added a paragraph in the Discussion to highlight the limitations of the presented method as you recommend. The paragraph reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the percentage contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its percentage contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

2. The concept of streams is a nice one. It allows to represent a network based treatment effect as a weighted sum of a direct and indirect effect estimates each corresponding to one path from source to sink. Note however that these effects are stochastically interdependent and hence, the aggregation of streams is different from the usual process of pooling evidence.

Thank you for this comment; indeed the aggregation of streams is not equivalent to the aggregation of studies in a pairwise meta-analysis. We write in ‘Percentage contributions of direct comparisons’:

“It can be shown that an NMA effect is a linear combination of the direct effects combined within paths.”

And we now added in the same section:

“Equation 5 represents θ_xy^N as a weighted sum of direct and two indirect effect estimates; the effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis.”

3. To be somewhat less destructive, I propose to arrange the network graph with the source to the left and the sink to the right and all other vertices placed, such that all directed edges point from left to right. Then, due to the flow properties, any vertical cross-section gives a set of flows that sum to one. In that sense, the untransformed rows of the H matrix already contain percentage contributions. Displays used to present voter hiking can be used to represent these flows (see e.g. www.zeit.de/politik/deutschland/2017-05/waehlerwanderung-nrw-landtagswahl-cdu-spd-fdp).

Thank you, we agree that horizontal representation is more intuitive. We re-arranged figures 1 and 2 accordingly.

4. I propose to modify Table 3 and Figure 3 in a way that the direction of influence becomes clear. Does a positive bias in one contributing direct effect translate into a positive or negative bias of the network based effect estimate? The colored fields in Figure 3 could be labelled with meaningfully ordered pairs of treatment letters.

Thank you very much for this comment which we believe is related to your comment 1. We agree that such a representation would be useful but we believe it could confuse readers with respect to the scope of this paper which is not to quantify the impact of within-study bias. As said in comment 1, the scope of this paper is to represent the percentage contributions of each piece of evidence; combining this information with information of study characteristics in a figure such as figure 3 may be a useful representation of the contribution of these characteristics to the estimation of NMA treatment effects. The influence of these characteristics, though, is not straightforwardly derived and it is subject to further research. Thus, we would prefer to leave Table 3 and Figure 3 unmodified and construct such representations of the direction of bias in future work where we aim to develop models to account for study-level bias using the presented methodology.

Please also see the reply to comment 1 and the addition to the Discussion.

5. Discussion “The calculation of the percentages uses the estimated variances of direct effects and thus incorporate associated uncertainty in their estimation.”
Note however, that in a linear network each direct effect estimate contributes equally, irrespective of the size and precision of the study.

We agree this statement is confusing and hence deleted this phrase from the Discussion. The paragraph now reads:

“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”

6. Discussion “Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates.”
The two stage approach discussed in this paper is not essential. Both the flows and the percentage contribution defined in the paper can be distributed to single trials according to the weights used when pooling them into direct effect estimates. Then the risk of bias of single trials can be analyzed. A display similar to Figure 3 is possible that contains separately colored fields for individual studies.

Thank you for this comment; we agree that such a distribution of percentage contributions to single trials is possible and relevant and we have updated the paper accordingly.

1.We added an extra paragraph/subsection ‘Percentage study contributions’ where we elaborate on the process you describe. The new paragraph reads:
“Matrix P (Table 3) shows the percentage contributions of each direct comparison to each NMA treatment effect estimate. These percentages can be distributed to individual studies within each comparison according to their weights from direct meta-analyses. For example, p_xy=63.5% and there are two studies examining the xy comparison. The individual study weights for the two studies are 0.69 and 1.54 resulting to study percentage contributions of (0.69 /(0.69+1.54))*63.5%=19.6% and (1.54/(0.69+1.54))*63.5%=43.8% to the xy NMA treatment effect estimate. The application of this process to the entire matrix P leads to the matrix P^* shown in Table 4. Adjusted weights as proposed by Rücker & Schwarzer (9) are used for multi-arm studies.”

2.Section ‘Using percentage contributions to quantify the impact of a characteristic in a direct comparison’ has been renamed to ‘Using percentage study contributions to quantify the impact of a characteristic in a direct comparison’ and follows the new ‘Percentage study contributions’ section. Text has been updated to refer to studies instead of direct comparisons and Figure 3 (and the new Table 4) now displays percentage study contributions to NMA treatment effect estimates.

3.The CINeMA (Confidence In Network Meta-Analysis) http://cinema.ispm.ch/ software has also been updated to display the bar chart using study contributions.

4. The first sentence of the Discussion now reads:
“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA.”

7. Discussion, 2^nd paragraph: Concerning multi-armed designs, König et al.[4] have defined the flow into and out of the vertices in a way that does not depend on the choice of reference treatment.

Thank you for this comment, we could indeed use the H matrix as described in section 3.3.1 of König et al.[4] (and implemented in netmeta as H.tilde) for multi-arm designs; some concerns about the interpretability of such a matrix have previously precluded us of doing so. In fact, the algorithm could be applied to any H matrix as long as its rows have the flow properties described in König et al.[4] and in our section ‘Comparison graph’. We have removed the discussion on multi-arm trials from the Discussion and added the following paragraph at the end of the Methods section:

“The starting point for the developed algorithm was a simplified version of the H matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. (4) that properly accommodates multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm.”

Supplementary File 3:
1st paragraph: What is the vector space of G_xy over R?

Thank you for this comment; as it was not clear and in order to simplify this part, we eliminated the notions of ‘vector space’ and ‘basis’ in the document. It now reads:
“However, in the general case, the number of paths used in the algorithm is smaller than the number of elements of Π. The number of directed paths which suffice to ‘spend’ the flow from source to sink is less or equal to df+1 where df is the inconsistency degrees of freedom in the Lu and Ades inconsistency model (1), df=D-T+1.”

2nd paragraph: ‘The number of directed paths’ obviously refers to the number of directed paths „which suffice to ‘spend’ the flow from source to sink”.

Thank you, we corrected it.

The authors state that, if no bridges exist, this number equals to df+1. However, it depends on the flows, too. Consider a network composed of two squares connected at one vertex. It has df=2 inconsistency degrees of freedom and generally 3 paths are needed to spend the flow between the two most distant vertices. However, if all edges have equal flows, only two paths are needed. Moreover, the connecting vertex has the same effect as the bridging edges. Hence the argument has to be extended to the case of graphs that can be split by cutting through a vertex.
Note also, that a network estimate that compares treatments from different subgroups of treatments connected through a bridge or a ‘breaking’ vertex can be written as the sum of three respective two subnetwork effect estimates: one comparing the source treatment to the bridge post, one for the bridge, if present, and one comparing the second bridge post to the sink. Because it is an unweighted sum, the contribution of the three (or two) summands should be equal.

Thank you very much for these interesting notes with which we totally agree. We extended the ‘Number of directed paths’ section in Supplementary File 3 to include discussion on the ‘breaking’ vertex situation and the consequences you describe. In particular we wrote:

“Another situation where the number of directed paths which suffice to ‘spend’ the flow from source to sink is less than df+1 elements occurs when a vertex has at least two inflows and two outflows, all equal between them. In such a situation the flow from source to sink may be spent in less than df+1 directed paths. Such a vertex will be called breaking vertex. Note that an NMA treatment effect estimate for treatments separated by a bridge or a breaking vertex (say treatments x and y) can be seen as an “indirect” comparison through either the intermediate bridge (constituting by treatments b₁ and b2) or the intermediate breaking vertex (b0).
In particular, in the case of a bridge, the NMA treatment effect estimate is derived as
θ_xy^N=θ_xb1^N+θ_b1b0^N+θ_b2y^N
and in the case of a breaking vertex the NMA treatment effect estimate is
θ_xy^N=θ_xb0^N+θ_b0y^N
It is, thus, implied that in such a case the separate subnetworks contribute equally to the estimation of θ_xy^N.”

Is the rationale for developing the new method (or application) clearly explained?

Yes

Is the description of the method technically sound?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly
Competing Interests: No competing interests were disclosed.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 03 Sep 2018

Adriani Nikolakopoulou, Institute of Social and Preventative Medicine (ISPM), University of Bern, Bern, Switzerland

03 Sep 2018

Author Response
We are grateful to Dr. Jochem König for the time and effort he spent to review our paper. We believe that his valuable comments and suggestions have substantially improved the ... Continue reading
We are grateful to Dr. Jochem König for the time and effort he spent to review our paper. We believe that his valuable comments and suggestions have substantially improved the manuscript. We have addressed all his comments and revised the paper accordingly. Below you can find detailed replies to the reviewer’s suggestions and comments.

Referee Report 13 Jun 2018
Jochem König, Division of Paediatric Epidemiology, Institute of Medical Biostatistics, Epidemiology, and Informatics (IMBEI), Johannes Gutenberg University Medical Center, Mainz, Germany

Approved with Reservations

1. The authors undertake a redefinition of the term ‘percentage contribution’ in network meta-analysis. In ordinary meta-analysis of randomized clinical trials, pooled estimates of a treatment effect can be represented as a weighted mean of treatment effects from all contributing trials. Weights are positive and sum to one and naturally constitute the ‘percentage contribution’ of each study. In network meta-analysis treatment effect estimates are defined for all pairs of treatments. Each can be represented as a two-stage estimate, firstly pooling all trials comparing the same set of treatments, respectively, which gives direct effect estimates, and then pooling direct estimates into a network based mixed treatment comparison. The linear coefficients, used in the second stage constitute the ‘contribution of direct evidence to a network based comparison’. They can be read off, from the H matrix, as described by the authors. They share some but not all aspects with weights of a two-armed meta-analysis. Aspects shared in some sense are the flow properties described by the authors and in citation [4]. The aspect not shared, is that summation to one.

Indeed, in a purely linear graph connecting two extreme treatments through a chain of, say 10, pairwise intermediary comparisons, the network estimate is the sum of all direct effects. Any bias present in one direct effect estimate translates 1:1 into the network based estimate. Accordingly, the entries of the H matrix are all equal to 1 in the relevant row.

Both, Salanti [3] and the present paper aim to rescale or transform the rows of the H matrix in order to achieve a ‘sum to one’ condition. In the case of the linear graph, the percentage contribution is 10% for each direct comparison. I am not convinced, that this number adequately captures the role of each direct effect estimate in this example.

Consider now, an example where two treatments are connected through three paths with one, two, and three edges, respectively, each with flow 1/3. Then each direct comparison has the same influence on the network based treatment effect estimate: A bias of 1 unit in one direct estimate translates into a bias of 1/3 unit in the network based estimate. These proportions are conserved in Salanti’s notion of percentage contribution, but not so in the newly introduced concept. Salanti [3] attaches a percentage contribution of 1/6 to each comparison just as if we had pooled six equally precise trials comparing the same pair of treatments. But the influence of direct comparisons is different in this network and should be characterized by the number 1/3.

That is why I have objections against Salanti’s [3] concept of percentage contribution and even more so against the newly introduced concept, which attaches unequal contribution quantities of 1/9, 1/6 and 1/3, respectively.

The new method should not be introduced into routine of network meta-analysis. If this paper is going to be indexed, its draw backs and caveats should be very clearly set out.

Thank you for this insightful comment. We agree with the point you make which we find that enlightens potential applications or extensions of the presented method to account for bias in network meta-analysis.
We would like to make a distinction between 'representation of the contribution of pieces of evidence in a 0 to 1 scale' and 'using this percentage contributions to represent and measure the impact and directionality of potential bias'. In this paper, we focus on the first aspect of translating the matrix into percentages and we subsequently apply it to depict the contribution of pieces of evidence according to a study characteristic. Our aim is to give a representation of the evidence that each comparison –or study, see our response to comment 6 below- contributes to the estimation of each NMA treatment effect without considering how this evidence could bias the results. Note that the bar plot of figure 3 gives a picture of the contribution of high, moderate or low risk of bias evidence without giving insight on the amount or direction of bias.

Quantification of bias in the direct evidence and its impact on the NMA estimates is subject of further research but not addressed within this paper. We have now added a paragraph in the Discussion to highlight the limitations of the presented method as you recommend. The paragraph reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the percentage contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its percentage contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

2. The concept of streams is a nice one. It allows to represent a network based treatment effect as a weighted sum of a direct and indirect effect estimates each corresponding to one path from source to sink. Note however that these effects are stochastically interdependent and hence, the aggregation of streams is different from the usual process of pooling evidence.

Thank you for this comment; indeed the aggregation of streams is not equivalent to the aggregation of studies in a pairwise meta-analysis. We write in ‘Percentage contributions of direct comparisons’:

“It can be shown that an NMA effect is a linear combination of the direct effects combined within paths.”

And we now added in the same section:

“Equation 5 represents θ_xy^N as a weighted sum of direct and two indirect effect estimates; the effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis.”

3. To be somewhat less destructive, I propose to arrange the network graph with the source to the left and the sink to the right and all other vertices placed, such that all directed edges point from left to right. Then, due to the flow properties, any vertical cross-section gives a set of flows that sum to one. In that sense, the untransformed rows of the H matrix already contain percentage contributions. Displays used to present voter hiking can be used to represent these flows (see e.g. www.zeit.de/politik/deutschland/2017-05/waehlerwanderung-nrw-landtagswahl-cdu-spd-fdp).

Thank you, we agree that horizontal representation is more intuitive. We re-arranged figures 1 and 2 accordingly.

4. I propose to modify Table 3 and Figure 3 in a way that the direction of influence becomes clear. Does a positive bias in one contributing direct effect translate into a positive or negative bias of the network based effect estimate? The colored fields in Figure 3 could be labelled with meaningfully ordered pairs of treatment letters.

Thank you very much for this comment which we believe is related to your comment 1. We agree that such a representation would be useful but we believe it could confuse readers with respect to the scope of this paper which is not to quantify the impact of within-study bias. As said in comment 1, the scope of this paper is to represent the percentage contributions of each piece of evidence; combining this information with information of study characteristics in a figure such as figure 3 may be a useful representation of the contribution of these characteristics to the estimation of NMA treatment effects. The influence of these characteristics, though, is not straightforwardly derived and it is subject to further research. Thus, we would prefer to leave Table 3 and Figure 3 unmodified and construct such representations of the direction of bias in future work where we aim to develop models to account for study-level bias using the presented methodology.

Please also see the reply to comment 1 and the addition to the Discussion.

5. Discussion “The calculation of the percentages uses the estimated variances of direct effects and thus incorporate associated uncertainty in their estimation.”
Note however, that in a linear network each direct effect estimate contributes equally, irrespective of the size and precision of the study.

We agree this statement is confusing and hence deleted this phrase from the Discussion. The paragraph now reads:

“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”

6. Discussion “Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates.”
The two stage approach discussed in this paper is not essential. Both the flows and the percentage contribution defined in the paper can be distributed to single trials according to the weights used when pooling them into direct effect estimates. Then the risk of bias of single trials can be analyzed. A display similar to Figure 3 is possible that contains separately colored fields for individual studies.

Thank you for this comment; we agree that such a distribution of percentage contributions to single trials is possible and relevant and we have updated the paper accordingly.

1.We added an extra paragraph/subsection ‘Percentage study contributions’ where we elaborate on the process you describe. The new paragraph reads:
“Matrix P (Table 3) shows the percentage contributions of each direct comparison to each NMA treatment effect estimate. These percentages can be distributed to individual studies within each comparison according to their weights from direct meta-analyses. For example, p_xy=63.5% and there are two studies examining the xy comparison. The individual study weights for the two studies are 0.69 and 1.54 resulting to study percentage contributions of (0.69 /(0.69+1.54))*63.5%=19.6% and (1.54/(0.69+1.54))*63.5%=43.8% to the xy NMA treatment effect estimate. The application of this process to the entire matrix P leads to the matrix P^* shown in Table 4. Adjusted weights as proposed by Rücker & Schwarzer (9) are used for multi-arm studies.”

2.Section ‘Using percentage contributions to quantify the impact of a characteristic in a direct comparison’ has been renamed to ‘Using percentage study contributions to quantify the impact of a characteristic in a direct comparison’ and follows the new ‘Percentage study contributions’ section. Text has been updated to refer to studies instead of direct comparisons and Figure 3 (and the new Table 4) now displays percentage study contributions to NMA treatment effect estimates.

3.The CINeMA (Confidence In Network Meta-Analysis) http://cinema.ispm.ch/ software has also been updated to display the bar chart using study contributions.

4. The first sentence of the Discussion now reads:
“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA.”

7. Discussion, 2^nd paragraph: Concerning multi-armed designs, König et al.[4] have defined the flow into and out of the vertices in a way that does not depend on the choice of reference treatment.

Thank you for this comment, we could indeed use the H matrix as described in section 3.3.1 of König et al.[4] (and implemented in netmeta as H.tilde) for multi-arm designs; some concerns about the interpretability of such a matrix have previously precluded us of doing so. In fact, the algorithm could be applied to any H matrix as long as its rows have the flow properties described in König et al.[4] and in our section ‘Comparison graph’. We have removed the discussion on multi-arm trials from the Discussion and added the following paragraph at the end of the Methods section:

“The starting point for the developed algorithm was a simplified version of the H matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. (4) that properly accommodates multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm.”

Supplementary File 3:
1st paragraph: What is the vector space of G_xy over R?

Thank you for this comment; as it was not clear and in order to simplify this part, we eliminated the notions of ‘vector space’ and ‘basis’ in the document. It now reads:
“However, in the general case, the number of paths used in the algorithm is smaller than the number of elements of Π. The number of directed paths which suffice to ‘spend’ the flow from source to sink is less or equal to df+1 where df is the inconsistency degrees of freedom in the Lu and Ades inconsistency model (1), df=D-T+1.”

2nd paragraph: ‘The number of directed paths’ obviously refers to the number of directed paths „which suffice to ‘spend’ the flow from source to sink”.

Thank you, we corrected it.

The authors state that, if no bridges exist, this number equals to df+1. However, it depends on the flows, too. Consider a network composed of two squares connected at one vertex. It has df=2 inconsistency degrees of freedom and generally 3 paths are needed to spend the flow between the two most distant vertices. However, if all edges have equal flows, only two paths are needed. Moreover, the connecting vertex has the same effect as the bridging edges. Hence the argument has to be extended to the case of graphs that can be split by cutting through a vertex.
Note also, that a network estimate that compares treatments from different subgroups of treatments connected through a bridge or a ‘breaking’ vertex can be written as the sum of three respective two subnetwork effect estimates: one comparing the source treatment to the bridge post, one for the bridge, if present, and one comparing the second bridge post to the sink. Because it is an unweighted sum, the contribution of the three (or two) summands should be equal.

Thank you very much for these interesting notes with which we totally agree. We extended the ‘Number of directed paths’ section in Supplementary File 3 to include discussion on the ‘breaking’ vertex situation and the consequences you describe. In particular we wrote:

“Another situation where the number of directed paths which suffice to ‘spend’ the flow from source to sink is less than df+1 elements occurs when a vertex has at least two inflows and two outflows, all equal between them. In such a situation the flow from source to sink may be spent in less than df+1 directed paths. Such a vertex will be called breaking vertex. Note that an NMA treatment effect estimate for treatments separated by a bridge or a breaking vertex (say treatments x and y) can be seen as an “indirect” comparison through either the intermediate bridge (constituting by treatments b₁ and b2) or the intermediate breaking vertex (b0).
In particular, in the case of a bridge, the NMA treatment effect estimate is derived as
θ_xy^N=θ_xb1^N+θ_b1b0^N+θ_b2y^N
and in the case of a breaking vertex the NMA treatment effect estimate is
θ_xy^N=θ_xb0^N+θ_b0y^N
It is, thus, implied that in such a case the separate subnetworks contribute equally to the estimation of θ_xy^N.”

Is the rationale for developing the new method (or application) clearly explained?

Yes

Is the description of the method technically sound?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly
Competing Interests: No competing interests were disclosed.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
We are grateful to Dr. Jochem König for the time and effort he spent to review our paper. We believe that his valuable comments and suggestions have substantially improved the manuscript. We have addressed all his comments and revised the paper accordingly. Below you can find detailed replies to the reviewer’s suggestions and comments.

Referee Report 13 Jun 2018
Jochem König, Division of Paediatric Epidemiology, Institute of Medical Biostatistics, Epidemiology, and Informatics (IMBEI), Johannes Gutenberg University Medical Center, Mainz, Germany

Approved with Reservations

1. The authors undertake a redefinition of the term ‘percentage contribution’ in network meta-analysis. In ordinary meta-analysis of randomized clinical trials, pooled estimates of a treatment effect can be represented as a weighted mean of treatment effects from all contributing trials. Weights are positive and sum to one and naturally constitute the ‘percentage contribution’ of each study. In network meta-analysis treatment effect estimates are defined for all pairs of treatments. Each can be represented as a two-stage estimate, firstly pooling all trials comparing the same set of treatments, respectively, which gives direct effect estimates, and then pooling direct estimates into a network based mixed treatment comparison. The linear coefficients, used in the second stage constitute the ‘contribution of direct evidence to a network based comparison’. They can be read off, from the H matrix, as described by the authors. They share some but not all aspects with weights of a two-armed meta-analysis. Aspects shared in some sense are the flow properties described by the authors and in citation [4]. The aspect not shared, is that summation to one.

Indeed, in a purely linear graph connecting two extreme treatments through a chain of, say 10, pairwise intermediary comparisons, the network estimate is the sum of all direct effects. Any bias present in one direct effect estimate translates 1:1 into the network based estimate. Accordingly, the entries of the H matrix are all equal to 1 in the relevant row.

Both, Salanti [3] and the present paper aim to rescale or transform the rows of the H matrix in order to achieve a ‘sum to one’ condition. In the case of the linear graph, the percentage contribution is 10% for each direct comparison. I am not convinced, that this number adequately captures the role of each direct effect estimate in this example.

Consider now, an example where two treatments are connected through three paths with one, two, and three edges, respectively, each with flow 1/3. Then each direct comparison has the same influence on the network based treatment effect estimate: A bias of 1 unit in one direct estimate translates into a bias of 1/3 unit in the network based estimate. These proportions are conserved in Salanti’s notion of percentage contribution, but not so in the newly introduced concept. Salanti [3] attaches a percentage contribution of 1/6 to each comparison just as if we had pooled six equally precise trials comparing the same pair of treatments. But the influence of direct comparisons is different in this network and should be characterized by the number 1/3.

That is why I have objections against Salanti’s [3] concept of percentage contribution and even more so against the newly introduced concept, which attaches unequal contribution quantities of 1/9, 1/6 and 1/3, respectively.

The new method should not be introduced into routine of network meta-analysis. If this paper is going to be indexed, its draw backs and caveats should be very clearly set out.

Thank you for this insightful comment. We agree with the point you make which we find that enlightens potential applications or extensions of the presented method to account for bias in network meta-analysis.
We would like to make a distinction between 'representation of the contribution of pieces of evidence in a 0 to 1 scale' and 'using this percentage contributions to represent and measure the impact and directionality of potential bias'. In this paper, we focus on the first aspect of translating the matrix into percentages and we subsequently apply it to depict the contribution of pieces of evidence according to a study characteristic. Our aim is to give a representation of the evidence that each comparison –or study, see our response to comment 6 below- contributes to the estimation of each NMA treatment effect without considering how this evidence could bias the results. Note that the bar plot of figure 3 gives a picture of the contribution of high, moderate or low risk of bias evidence without giving insight on the amount or direction of bias.

Quantification of bias in the direct evidence and its impact on the NMA estimates is subject of further research but not addressed within this paper. We have now added a paragraph in the Discussion to highlight the limitations of the presented method as you recommend. The paragraph reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the percentage contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its percentage contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

2. The concept of streams is a nice one. It allows to represent a network based treatment effect as a weighted sum of a direct and indirect effect estimates each corresponding to one path from source to sink. Note however that these effects are stochastically interdependent and hence, the aggregation of streams is different from the usual process of pooling evidence.

Thank you for this comment; indeed the aggregation of streams is not equivalent to the aggregation of studies in a pairwise meta-analysis. We write in ‘Percentage contributions of direct comparisons’:

“It can be shown that an NMA effect is a linear combination of the direct effects combined within paths.”

And we now added in the same section:

“Equation 5 represents θ_xy^N as a weighted sum of direct and two indirect effect estimates; the effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis.”

3. To be somewhat less destructive, I propose to arrange the network graph with the source to the left and the sink to the right and all other vertices placed, such that all directed edges point from left to right. Then, due to the flow properties, any vertical cross-section gives a set of flows that sum to one. In that sense, the untransformed rows of the H matrix already contain percentage contributions. Displays used to present voter hiking can be used to represent these flows (see e.g. www.zeit.de/politik/deutschland/2017-05/waehlerwanderung-nrw-landtagswahl-cdu-spd-fdp).

Thank you, we agree that horizontal representation is more intuitive. We re-arranged figures 1 and 2 accordingly.

4. I propose to modify Table 3 and Figure 3 in a way that the direction of influence becomes clear. Does a positive bias in one contributing direct effect translate into a positive or negative bias of the network based effect estimate? The colored fields in Figure 3 could be labelled with meaningfully ordered pairs of treatment letters.

Thank you very much for this comment which we believe is related to your comment 1. We agree that such a representation would be useful but we believe it could confuse readers with respect to the scope of this paper which is not to quantify the impact of within-study bias. As said in comment 1, the scope of this paper is to represent the percentage contributions of each piece of evidence; combining this information with information of study characteristics in a figure such as figure 3 may be a useful representation of the contribution of these characteristics to the estimation of NMA treatment effects. The influence of these characteristics, though, is not straightforwardly derived and it is subject to further research. Thus, we would prefer to leave Table 3 and Figure 3 unmodified and construct such representations of the direction of bias in future work where we aim to develop models to account for study-level bias using the presented methodology.

Please also see the reply to comment 1 and the addition to the Discussion.

5. Discussion “The calculation of the percentages uses the estimated variances of direct effects and thus incorporate associated uncertainty in their estimation.”
Note however, that in a linear network each direct effect estimate contributes equally, irrespective of the size and precision of the study.

We agree this statement is confusing and hence deleted this phrase from the Discussion. The paragraph now reads:

“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”

6. Discussion “Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates.”
The two stage approach discussed in this paper is not essential. Both the flows and the percentage contribution defined in the paper can be distributed to single trials according to the weights used when pooling them into direct effect estimates. Then the risk of bias of single trials can be analyzed. A display similar to Figure 3 is possible that contains separately colored fields for individual studies.

Thank you for this comment; we agree that such a distribution of percentage contributions to single trials is possible and relevant and we have updated the paper accordingly.

1.We added an extra paragraph/subsection ‘Percentage study contributions’ where we elaborate on the process you describe. The new paragraph reads:
“Matrix P (Table 3) shows the percentage contributions of each direct comparison to each NMA treatment effect estimate. These percentages can be distributed to individual studies within each comparison according to their weights from direct meta-analyses. For example, p_xy=63.5% and there are two studies examining the xy comparison. The individual study weights for the two studies are 0.69 and 1.54 resulting to study percentage contributions of (0.69 /(0.69+1.54))*63.5%=19.6% and (1.54/(0.69+1.54))*63.5%=43.8% to the xy NMA treatment effect estimate. The application of this process to the entire matrix P leads to the matrix P^* shown in Table 4. Adjusted weights as proposed by Rücker & Schwarzer (9) are used for multi-arm studies.”

2.Section ‘Using percentage contributions to quantify the impact of a characteristic in a direct comparison’ has been renamed to ‘Using percentage study contributions to quantify the impact of a characteristic in a direct comparison’ and follows the new ‘Percentage study contributions’ section. Text has been updated to refer to studies instead of direct comparisons and Figure 3 (and the new Table 4) now displays percentage study contributions to NMA treatment effect estimates.

3.The CINeMA (Confidence In Network Meta-Analysis) http://cinema.ispm.ch/ software has also been updated to display the bar chart using study contributions.

4. The first sentence of the Discussion now reads:
“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA.”

7. Discussion, 2^nd paragraph: Concerning multi-armed designs, König et al.[4] have defined the flow into and out of the vertices in a way that does not depend on the choice of reference treatment.

Thank you for this comment, we could indeed use the H matrix as described in section 3.3.1 of König et al.[4] (and implemented in netmeta as H.tilde) for multi-arm designs; some concerns about the interpretability of such a matrix have previously precluded us of doing so. In fact, the algorithm could be applied to any H matrix as long as its rows have the flow properties described in König et al.[4] and in our section ‘Comparison graph’. We have removed the discussion on multi-arm trials from the Discussion and added the following paragraph at the end of the Methods section:

“The starting point for the developed algorithm was a simplified version of the H matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. (4) that properly accommodates multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm.”

Supplementary File 3:
1st paragraph: What is the vector space of G_xy over R?

Thank you for this comment; as it was not clear and in order to simplify this part, we eliminated the notions of ‘vector space’ and ‘basis’ in the document. It now reads:
“However, in the general case, the number of paths used in the algorithm is smaller than the number of elements of Π. The number of directed paths which suffice to ‘spend’ the flow from source to sink is less or equal to df+1 where df is the inconsistency degrees of freedom in the Lu and Ades inconsistency model (1), df=D-T+1.”

2nd paragraph: ‘The number of directed paths’ obviously refers to the number of directed paths „which suffice to ‘spend’ the flow from source to sink”.

Thank you, we corrected it.

The authors state that, if no bridges exist, this number equals to df+1. However, it depends on the flows, too. Consider a network composed of two squares connected at one vertex. It has df=2 inconsistency degrees of freedom and generally 3 paths are needed to spend the flow between the two most distant vertices. However, if all edges have equal flows, only two paths are needed. Moreover, the connecting vertex has the same effect as the bridging edges. Hence the argument has to be extended to the case of graphs that can be split by cutting through a vertex.
Note also, that a network estimate that compares treatments from different subgroups of treatments connected through a bridge or a ‘breaking’ vertex can be written as the sum of three respective two subnetwork effect estimates: one comparing the source treatment to the bridge post, one for the bridge, if present, and one comparing the second bridge post to the sink. Because it is an unweighted sum, the contribution of the three (or two) summands should be equal.

Thank you very much for these interesting notes with which we totally agree. We extended the ‘Number of directed paths’ section in Supplementary File 3 to include discussion on the ‘breaking’ vertex situation and the consequences you describe. In particular we wrote:

“Another situation where the number of directed paths which suffice to ‘spend’ the flow from source to sink is less than df+1 elements occurs when a vertex has at least two inflows and two outflows, all equal between them. In such a situation the flow from source to sink may be spent in less than df+1 directed paths. Such a vertex will be called breaking vertex. Note that an NMA treatment effect estimate for treatments separated by a bridge or a breaking vertex (say treatments x and y) can be seen as an “indirect” comparison through either the intermediate bridge (constituting by treatments b₁ and b2) or the intermediate breaking vertex (b0).
In particular, in the case of a bridge, the NMA treatment effect estimate is derived as
θ_xy^N=θ_xb1^N+θ_b1b0^N+θ_b2y^N
and in the case of a breaking vertex the NMA treatment effect estimate is
θ_xy^N=θ_xb0^N+θ_b0y^N
It is, thus, implied that in such a case the separate subnetworks contribute equally to the estimation of θ_xy^N.”

Is the rationale for developing the new method (or application) clearly explained?

Yes

Is the description of the method technically sound?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly
Competing Interests: No competing interests were disclosed.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 18 May 2018

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3
Version 3 (revision) 13 Dec 18
Version 2 (revision) 03 Sep 18
Version 1 18 May 18	read	read	read

Jochem König, Johannes Gutenberg University Medical Center, Mainz, Germany
Annette M. O'Connor, Iowa State University, Ames, USA
John R. Thompson, University of Leicester, Leicester, UK

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

20 Views

16 Aug 2018 | for Version 1

John R. Thompson, Department of Health Sciences, University of Leicester, Leicester, UK

20 Views Cite this report Responses(1)

Approved

Is the rationale for developing the new method (or application) clearly explained?

Yes
Is the description of the method technically sound?

Yes
Are sufficient details provided to allow replication of the method development and its use by others?

Yes
If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

Competing Interests

No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

13 Dec 2018

Adriani Nikolakopoulou, Institute of Social and Preventative Medicine (ISPM), University of Bern, Bern, Switzerland

We are grateful to Dr. John R. Thompson for his time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report 16 Aug 2018
John R. Thompson, Department of Health Sciences, University of Leicester, Leicester, UK

Approved

I have no special knowledge of the measurement of information flow in a network meta-analysis and so I read this paper as a biostatistician with a general interest in meta-analysis.

I like the idea of associating a descriptive measure of importance to each edge in a network and the authors of this paper present an interesting step in that direction, though my feeling is that this proposal will not turn out to be the final solution. I have two areas of concern, one is to do with terminology and the other with the scope of applicability of the proposal solution.

In criticising the terminology, I have to accept that the authors are largely following common practice and so my criticisms are partly aimed at the meta-analysis community. None the less there were three things that irritated me. Firstly, throughout the paper, proportions are referred to as percentages. Next, the title of the paper says it is about "the contribution of studies in network meta-analysis" while actually it is about the contribution of different effect estimates. Finally, the measure adopted is the weight given to each effect estimate when calculating the pooled estimate. In this paper and elsewhere in the meta-analysis literature, this weight is called a contribution. Perhaps I am being too pedantic but the weight and contribution are different ideas and it does not help with the terminology confuses them.

Thank you for these interesting notes on meta-analysis terminology. At first, we agree that referring to “percentage contributions” might not be fitting and calling them “proportions” is more appropriate. We, thus, replaced throughout percentages with proportions.

Regarding the reporting of contribution of studies in the title of our paper, we have extended the method to quantify the contribution of studies as suggested by the first reviewer, Dr. Jochem König (comment 6). Thus, we left the title of the paper unmodified.

Weights and contributions are indeed sometimes used interchangeably. In this paper, however, we believe that the definition of matrix and the set of flows render the distinction clear. We only talk about “weights” of paths in the Introduction, where we have not yet explained the terminology to be used and thus it is convenient describing the elements of matrix as generalized weights from pairwise meta-analysis. We apologise in advance if we have misunderstood your comment.

Now the proposed method. Taking the example from the paper, the authors note that the pooled estimate of the xy effect can be calculated using equation 1, where θ̂^I_xvyis the indirect estimate of xy along the path xy. Further it is true, at least for this example, that ΣL_iφi = 1 where L_i is the length of the path. So we can attach the weights φ to the edges of the network, sum them when an edge contributes to more than one estimate and the resulting weights will sum to one over the whole network.

This argument works for the example presented in the paper but it is not clear to me what conditions have to hold for it to work generally. The authors note in the paper that including a multi-arm trial in the meta-analysis would cause a problem, presumably because some of the direct or indirect estimates would not be independent. Are there any other conditions that have to hold? For example, can we have any structure of random effects in the meta-analysis model? What about the Bayesian models that are often used for network meta-analysis?

Thank you for this insightful comment. We have clarified at the beginning of the Methods section that we consider “a simplified version of the matrix described by König et al. [1] that does not take into account the correlation induced by multi-arm trials.”

In this case, and considering the properties of the elements of set it always holds that Equation 3 can be re-written as Equation 5. We now clarify in the subsection ‘Proportion contributions of direct comparisons’:

“It follows from the properties of the elements of set that each NMA effect can be written as a linear combination of direct and indirect effects, in the form of Equation 5”.

We clarify at the end of ‘Algorithm to decompose flows into percentage contributions’ section that multi-arm studies can be handled if a modified matrix is used. We also clarify that the structure of random effects does not play a role in the calculations. In particular, we write:
“The starting point for the developed algorithm was a simplified version of the matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. [1] that extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm. The estimator of heterogeneity as well as the assumption of a different or common heterogeneity across comparisons in the network does not modify any aspect of the method.”

Within a Bayesian framework, the matrix is not defined. We added in the Discussion:
“The derivation of the contribution of sources of evidence in a Bayesian NMA is not straightforward, as no analogue to the matrix exists. The application of the method described in this paper, however, can be used to derive useful approximations of the contributions of studies.”

The authors take the matrix, H, which projects individual estimates such as xy, xy, uy, etc. into their predicted values under the meta-analysis model and they present an algorithm for converting those values into weights that are equivalent to the φ's.The algorithm is sensible and works for the simple example in the paper but one is again left wondering whether or not it works under all circumstances. After all, the algorithm is presented without any proof that it works.

Thank you for pointing this out. The algorithm we present in the ‘Algorithm to decompose flows into proportion contributions’ is described for the general case of a network of interventions containing any number of studies, treatments and loops. Indeed, we do not present a proof that it works under all circumstances. We have, however, applied the algorithm in numerous NMA applications apart from the simple example used in the results of this paper. Moreover, as mentioned in the Discussion, the web application CINeMA (Confidence In Network Meta-Analysis) [2], that aims to simplify the evaluation of confidence in the findings from NMA, uses the proportion contribution matrix as described in this paper. Users of CINeMA have not reported any case that the algorithm does not work. We are planning to examine the properties of the different approaches on the selection of direct paths in the algorithm in a follow up project (preliminary work is described in Supplementary File 3), which could be combined with a proof that the algorithm (and its modifications) works.

My own feeling is that a contribution is best measured by the sledgehammer approach of analysing the network with and without a particular edge, but the authors' suggested approach is much less computationally demanding and I think that it would be appreciated by many applied researchers provided they were certain that it could be safely used with their particular network and their particular meta-analysis model.

We totally agree that such ‘leave one out’ approaches would be also of interest. Relevant methods have been proposed by Krahn et al. [3] for contribution to the effect and by Riley et al. [4] for contribution to the variance. The former is useful in measuring how much a direct effect could bias the NMA effect and thus is not relevant in this case where we opted for a representation of contribution of pieces of evidence according to a study characteristic. The latter is based on the decomposition of Fisher’s information matrix and is associated with some issues that need further investigation.

We had already mentioned and discussed the Riley et al. approach in the Discussion. We now added:
“Krahn et al. define influence functions to describe the extent to which changes in study effects would be translated into NMA treatment effects [3].”

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

22 Views

14 Aug 2018 | for Version 1

Annette M. O'Connor, Department of Veterinary Diagnostic and Production Animal Medicine, College of Veterinary Medicine, Iowa State University, Ames, IA, USA

22 Views Cite this report Responses(1)

Approved

Is the rationale for developing the new method (or application) clearly explained?

Yes
Is the description of the method technically sound?

Yes
Are sufficient details provided to allow replication of the method development and its use by others?

Yes
If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Epidemiology, infectious diseases, research synthesis

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Author Response

13 Dec 2018

Adriani Nikolakopoulou, Institute of Social and Preventative Medicine (ISPM), University of Bern, Bern, Switzerland

We are grateful to Dr. Annette M. O’Connor for her time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer’s comments; our responses to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2.

Referee Report: Annette M. O'Connor, Department of Veterinary Diagnostic and Production Animal Medicine, College of Veterinary Medicine, Iowa State University, Ames, IA, USA

Approved

This paper described an updated approach to deriving the percentage contributions of the direct comparison used in treatment estimates. In the interests of full disclosure, I am not a statistician and was reviewing from the view point of an applied user of NMA. There are several papers on this topic in recent years, and it is an active body of work in this area. The article is straightforward to read, although made considerably more readable if one is aware of the content in references, 3,4 and 5.

I have to admit that this is an example of why I appreciate the open review system. I enjoyed the paper and found it easy to read and follow, but the others reviewer's comments were a significant contribution also. I don't have any additional critique of the proposed approach, but I look forward to seeing how the investigators address those or provided some discussion, as this would help the less statistically inclined reader like me. Again, the reason I wanted this, is that I am an applied user and such discussions are very helpful. In particular, I would like to have seen some discussion of the 1st comment. I would also like to see if the comment about Figure 3 could be incorporated - perhaps not feasible.

Thank you very much for your positive feedback. Our answers to the first reviewer, Dr. Jochem König, can be found in https://f1000research.com/articles/7-610/v2. In summary, regarding his first comment, we distinguish between representation of contribution of pieces of evidence according to a study characteristic (addressed in this paper) and quantification of the impact of within-study bias in NMA treatment effects (subject of future research). A new paragraph in the Discussion highlighting these concerns reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the proportion contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its proportion contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

As we explain in detail in our response to Dr. Jochem König (comment 4), we did not amend Figure 3 to incorporate directionality.

Concerning the approach to calculating the % contribution and how to weight the flows, I came away with the impression that the decision of equal split of the stream flow as arbitrary (perhaps not precisely the correct term) or perhaps pragmatic is better. Therefore, it is not surprising this is a debatable approach. Again, this was another reason why I am looking forward to see the investigators responses to the 1st reviewer's comments. I agree that the comparison of the methods of deriving the percentage contribution would be of interest but was not expecting that to be included. I look forward to this comparison being published.

We agree that equal splitting of stream flow is to some extent arbitrary but also pragmatic. We say in the Discussion:

“An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”

View more View less

Competing Interests

No competing interests were disclosed.

Back to all reports

Reviewer Report

44 Views

13 Jun 2018 | for Version 1

Jochem König, Division of Paediatric Epidemiology, Institute of Medical Biostatistics, Epidemiology, and Informatics (IMBEI), Johannes Gutenberg University Medical Center, Mainz, Germany

44 Views Cite this report Responses(1)

Approved With Reservations

The authors undertake a redefinition of the term ‘percentage contribution’ in network meta-analysis. In ordinary meta-analysis of randomized clinical trials, pooled estimates of a treatment effect can be represented as a weighted mean of treatment effects from all contributing trials. Weights are positive and sum to one and naturally constitute the ‘percentage contribution’ of each study. In network meta-analysis treatment effect estimates are defined for all pairs of treatments. Each can be represented as a two-stage estimate, firstly pooling all trials comparing the same set of treatments, respectively, which gives direct effect estimates, and then pooling direct estimates into a network based mixed treatment comparison. The linear coefficients, used in the second stage constitute the ‘contribution of direct evidence to a network based comparison’. They can be read off, from the H matrix, as described by the authors. They share some but not all aspects with weights of a two-armed meta-analysis. Aspects shared in some sense are the flow properties described by the authors and in citation [4]. The aspect not shared, is that summation to one.

Indeed, in a purely linear graph connecting two extreme treatments through a chain of, say 10, pairwise intermediary comparisons, the network estimate is the sum of all direct effects. Any bias present in one direct effect estimate translates 1:1 into the network based estimate. Accordingly, the entries of the H matrix are all equal to 1 in the relevant row.

Both, Salanti [3] and the present paper aim to rescale or transform the rows of the H matrix in order to achieve a ‘sum to one’ condition. In the case of the linear graph, the percentage contribution is 10% for each direct comparison. I am not convinced, that this number adequately captures the role of each direct effect estimate in this example.

Consider now, an example where two treatments are connected through three paths with one, two, and three edges, respectively, each with flow 1/3. Then each direct comparison has the same influence on the network based treatment effect estimate: A bias of 1 unit in one direct estimate translates into a bias of 1/3 unit in the network based estimate. These proportions are conserved in Salanti’s notion of percentage contribution, but not so in the newly introduced concept. Salanti [3] attaches a percentage contribution of 1/6 to each comparison just as if we had pooled six equally precise trials comparing the same pair of treatments. But the influence of direct comparisons is different in this network and should be characterized by the number 1/3.

That is why I have objections against Salanti’s [3] concept of percentage contribution and even more so against the newly introduced concept, which attaches unequal contribution quantities of 1/9, 1/6 and 1/3, respectively.

The new method should not be introduced into routine of network meta-analysis. If this paper is going to be indexed, its draw backs and caveats should be very clearly set out.
The concept of streams is a nice one. It allows to represent a network based treatment effect as a weighted sum of a direct and indirect effect estimates each corresponding to one path from source to sink. Note however that these effects are stochastically interdependent and hence, the aggregation of streams is different from the usual process of pooling evidence.
To be somewhat less destructive, I propose to arrange the network graph with the source to the left and the sink to the right and all other vertices placed, such that all directed edges point from left to right. Then, due to the flow properties, any vertical cross-section gives a set of flows that sum to one. In that sense, the untransformed rows of the H matrix already contain percentage contributions. Displays used to present voter hiking can be used to represent these flows (see e.g. www.zeit.de/politik/deutschland/2017-05/waehlerwanderung-nrw-landtagswahl-cdu-spd-fdp).
I propose to modify Table 3 and Figure 3 in a way that the direction of influence becomes clear. Does a positive bias in one contributing direct effect translate into a positive or negative bias of the network based effect estimate? The colored fields in Figure 3 could be labelled with meaningfully ordered pairs of treatment letters.
Discussion “The calculation of the percentages uses the estimated variances of direct effects and thus incorporate associated uncertainty in their estimation.”
Note however, that in a linear network each direct effect estimate contributes equally, irrespective of the size and precision of the study.
Discussion “Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates.”
The two stage approach discussed in this paper is not essential. Both the flows and the percentage contribution defined in the paper can be distributed to single trials according to the weights used when pooling them into direct effect estimates. Then the risk of bias of single trials can be analyzed. A display similar to Figure 3 is possible that contains separately colored fields for individual studies.
Discussion, 2^nd paragraph: Concerning multi-armed designs, König et al.[4] have defined the flow into and out of the vertices in a way that does not depend on the choice of reference treatment.

Supplementary File 3:
1st paragraph: What is the vector space of G_xy over R?
2nd paragraph: ‘The number of directed paths’ obviously refers to the number of directed paths „which suffice to ‘spend’ the flow from source to sink”.

The authors state that, if no bridges exist, this number equals to df+1. However, it depends on the flows, too. Consider a network composed of two squares connected at one vertex. It has df=2 inconsistency degrees of freedom and generally 3 paths are needed to spend the flow between the two most distant vertices. However, if all edges have equal flows, only two paths are needed. Moreover, the connecting vertex has the same effect as the bridging edges. Hence the argument has to be extended to the case of graphs that can be split by cutting through a vertex.

Note also, that a network estimate that compares treatments from different subgroups of treatments connected through a bridge or a ‘breaking’ vertex can be written as the sum of three respective two subnetwork effect estimates: one comparing the source treatment to the bridge post, one for the bridge, if present, and one comparing the second bridge post to the sink. Because it is an unweighted sum, the contribution of the three (or two) summands should be equal.

Is the rationale for developing the new method (or application) clearly explained?

Yes
Is the description of the method technically sound?

Yes
Are sufficient details provided to allow replication of the method development and its use by others?

Yes
If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

Competing Interests

No competing interests were disclosed.

Respond to this report

Responses (1)

Author Response

03 Sep 2018

Adriani Nikolakopoulou, Institute of Social and Preventative Medicine (ISPM), University of Bern, Bern, Switzerland

We are grateful to Dr. Jochem König for the time and effort he spent to review our paper. We believe that his valuable comments and suggestions have substantially improved the manuscript. We have addressed all his comments and revised the paper accordingly. Below you can find detailed replies to the reviewer’s suggestions and comments.

Referee Report 13 Jun 2018
Jochem König, Division of Paediatric Epidemiology, Institute of Medical Biostatistics, Epidemiology, and Informatics (IMBEI), Johannes Gutenberg University Medical Center, Mainz, Germany

Approved with Reservations

1. The authors undertake a redefinition of the term ‘percentage contribution’ in network meta-analysis. In ordinary meta-analysis of randomized clinical trials, pooled estimates of a treatment effect can be represented as a weighted mean of treatment effects from all contributing trials. Weights are positive and sum to one and naturally constitute the ‘percentage contribution’ of each study. In network meta-analysis treatment effect estimates are defined for all pairs of treatments. Each can be represented as a two-stage estimate, firstly pooling all trials comparing the same set of treatments, respectively, which gives direct effect estimates, and then pooling direct estimates into a network based mixed treatment comparison. The linear coefficients, used in the second stage constitute the ‘contribution of direct evidence to a network based comparison’. They can be read off, from the H matrix, as described by the authors. They share some but not all aspects with weights of a two-armed meta-analysis. Aspects shared in some sense are the flow properties described by the authors and in citation [4]. The aspect not shared, is that summation to one.

Indeed, in a purely linear graph connecting two extreme treatments through a chain of, say 10, pairwise intermediary comparisons, the network estimate is the sum of all direct effects. Any bias present in one direct effect estimate translates 1:1 into the network based estimate. Accordingly, the entries of the H matrix are all equal to 1 in the relevant row.

Both, Salanti [3] and the present paper aim to rescale or transform the rows of the H matrix in order to achieve a ‘sum to one’ condition. In the case of the linear graph, the percentage contribution is 10% for each direct comparison. I am not convinced, that this number adequately captures the role of each direct effect estimate in this example.

Consider now, an example where two treatments are connected through three paths with one, two, and three edges, respectively, each with flow 1/3. Then each direct comparison has the same influence on the network based treatment effect estimate: A bias of 1 unit in one direct estimate translates into a bias of 1/3 unit in the network based estimate. These proportions are conserved in Salanti’s notion of percentage contribution, but not so in the newly introduced concept. Salanti [3] attaches a percentage contribution of 1/6 to each comparison just as if we had pooled six equally precise trials comparing the same pair of treatments. But the influence of direct comparisons is different in this network and should be characterized by the number 1/3.

That is why I have objections against Salanti’s [3] concept of percentage contribution and even more so against the newly introduced concept, which attaches unequal contribution quantities of 1/9, 1/6 and 1/3, respectively.

The new method should not be introduced into routine of network meta-analysis. If this paper is going to be indexed, its draw backs and caveats should be very clearly set out.

Thank you for this insightful comment. We agree with the point you make which we find that enlightens potential applications or extensions of the presented method to account for bias in network meta-analysis.
We would like to make a distinction between 'representation of the contribution of pieces of evidence in a 0 to 1 scale' and 'using this percentage contributions to represent and measure the impact and directionality of potential bias'. In this paper, we focus on the first aspect of translating the matrix into percentages and we subsequently apply it to depict the contribution of pieces of evidence according to a study characteristic. Our aim is to give a representation of the evidence that each comparison –or study, see our response to comment 6 below- contributes to the estimation of each NMA treatment effect without considering how this evidence could bias the results. Note that the bar plot of figure 3 gives a picture of the contribution of high, moderate or low risk of bias evidence without giving insight on the amount or direction of bias.

Quantification of bias in the direct evidence and its impact on the NMA estimates is subject of further research but not addressed within this paper. We have now added a paragraph in the Discussion to highlight the limitations of the presented method as you recommend. The paragraph reads:

“Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the percentage contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its percentage contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.”

2. The concept of streams is a nice one. It allows to represent a network based treatment effect as a weighted sum of a direct and indirect effect estimates each corresponding to one path from source to sink. Note however that these effects are stochastically interdependent and hence, the aggregation of streams is different from the usual process of pooling evidence.

Thank you for this comment; indeed the aggregation of streams is not equivalent to the aggregation of studies in a pairwise meta-analysis. We write in ‘Percentage contributions of direct comparisons’:

“It can be shown that an NMA effect is a linear combination of the direct effects combined within paths.”

And we now added in the same section:

“Equation 5 represents θ_xy^N as a weighted sum of direct and two indirect effect estimates; the effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis.”

3. To be somewhat less destructive, I propose to arrange the network graph with the source to the left and the sink to the right and all other vertices placed, such that all directed edges point from left to right. Then, due to the flow properties, any vertical cross-section gives a set of flows that sum to one. In that sense, the untransformed rows of the H matrix already contain percentage contributions. Displays used to present voter hiking can be used to represent these flows (see e.g. www.zeit.de/politik/deutschland/2017-05/waehlerwanderung-nrw-landtagswahl-cdu-spd-fdp).

Thank you, we agree that horizontal representation is more intuitive. We re-arranged figures 1 and 2 accordingly.

4. I propose to modify Table 3 and Figure 3 in a way that the direction of influence becomes clear. Does a positive bias in one contributing direct effect translate into a positive or negative bias of the network based effect estimate? The colored fields in Figure 3 could be labelled with meaningfully ordered pairs of treatment letters.

Thank you very much for this comment which we believe is related to your comment 1. We agree that such a representation would be useful but we believe it could confuse readers with respect to the scope of this paper which is not to quantify the impact of within-study bias. As said in comment 1, the scope of this paper is to represent the percentage contributions of each piece of evidence; combining this information with information of study characteristics in a figure such as figure 3 may be a useful representation of the contribution of these characteristics to the estimation of NMA treatment effects. The influence of these characteristics, though, is not straightforwardly derived and it is subject to further research. Thus, we would prefer to leave Table 3 and Figure 3 unmodified and construct such representations of the direction of bias in future work where we aim to develop models to account for study-level bias using the presented methodology.

Please also see the reply to comment 1 and the addition to the Discussion.

5. Discussion “The calculation of the percentages uses the estimated variances of direct effects and thus incorporate associated uncertainty in their estimation.”
Note however, that in a linear network each direct effect estimate contributes equally, irrespective of the size and precision of the study.

We agree this statement is confusing and hence deleted this phrase from the Discussion. The paragraph now reads:

“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.”

6. Discussion “Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates.”
The two stage approach discussed in this paper is not essential. Both the flows and the percentage contribution defined in the paper can be distributed to single trials according to the weights used when pooling them into direct effect estimates. Then the risk of bias of single trials can be analyzed. A display similar to Figure 3 is possible that contains separately colored fields for individual studies.

Thank you for this comment; we agree that such a distribution of percentage contributions to single trials is possible and relevant and we have updated the paper accordingly.

1.We added an extra paragraph/subsection ‘Percentage study contributions’ where we elaborate on the process you describe. The new paragraph reads:
“Matrix P (Table 3) shows the percentage contributions of each direct comparison to each NMA treatment effect estimate. These percentages can be distributed to individual studies within each comparison according to their weights from direct meta-analyses. For example, p_xy=63.5% and there are two studies examining the xy comparison. The individual study weights for the two studies are 0.69 and 1.54 resulting to study percentage contributions of (0.69 /(0.69+1.54))*63.5%=19.6% and (1.54/(0.69+1.54))*63.5%=43.8% to the xy NMA treatment effect estimate. The application of this process to the entire matrix P leads to the matrix P^* shown in Table 4. Adjusted weights as proposed by Rücker & Schwarzer (9) are used for multi-arm studies.”

2.Section ‘Using percentage contributions to quantify the impact of a characteristic in a direct comparison’ has been renamed to ‘Using percentage study contributions to quantify the impact of a characteristic in a direct comparison’ and follows the new ‘Percentage study contributions’ section. Text has been updated to refer to studies instead of direct comparisons and Figure 3 (and the new Table 4) now displays percentage study contributions to NMA treatment effect estimates.

3.The CINeMA (Confidence In Network Meta-Analysis) http://cinema.ispm.ch/ software has also been updated to display the bar chart using study contributions.

4. The first sentence of the Discussion now reads:
“In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA.”

7. Discussion, 2^nd paragraph: Concerning multi-armed designs, König et al.[4] have defined the flow into and out of the vertices in a way that does not depend on the choice of reference treatment.

Thank you for this comment, we could indeed use the H matrix as described in section 3.3.1 of König et al.[4] (and implemented in netmeta as H.tilde) for multi-arm designs; some concerns about the interpretability of such a matrix have previously precluded us of doing so. In fact, the algorithm could be applied to any H matrix as long as its rows have the flow properties described in König et al.[4] and in our section ‘Comparison graph’. We have removed the discussion on multi-arm trials from the Discussion and added the following paragraph at the end of the Methods section:

“The starting point for the developed algorithm was a simplified version of the H matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as described by König et al. (4) that properly accommodates multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm.”

Supplementary File 3:
1st paragraph: What is the vector space of G_xy over R?

Thank you for this comment; as it was not clear and in order to simplify this part, we eliminated the notions of ‘vector space’ and ‘basis’ in the document. It now reads:
“However, in the general case, the number of paths used in the algorithm is smaller than the number of elements of Π. The number of directed paths which suffice to ‘spend’ the flow from source to sink is less or equal to df+1 where df is the inconsistency degrees of freedom in the Lu and Ades inconsistency model (1), df=D-T+1.”

2nd paragraph: ‘The number of directed paths’ obviously refers to the number of directed paths „which suffice to ‘spend’ the flow from source to sink”.

Thank you, we corrected it.

The authors state that, if no bridges exist, this number equals to df+1. However, it depends on the flows, too. Consider a network composed of two squares connected at one vertex. It has df=2 inconsistency degrees of freedom and generally 3 paths are needed to spend the flow between the two most distant vertices. However, if all edges have equal flows, only two paths are needed. Moreover, the connecting vertex has the same effect as the bridging edges. Hence the argument has to be extended to the case of graphs that can be split by cutting through a vertex.
Note also, that a network estimate that compares treatments from different subgroups of treatments connected through a bridge or a ‘breaking’ vertex can be written as the sum of three respective two subnetwork effect estimates: one comparing the source treatment to the bridge post, one for the bridge, if present, and one comparing the second bridge post to the sink. Because it is an unweighted sum, the contribution of the three (or two) summands should be equal.

Thank you very much for these interesting notes with which we totally agree. We extended the ‘Number of directed paths’ section in Supplementary File 3 to include discussion on the ‘breaking’ vertex situation and the consequences you describe. In particular we wrote:

“Another situation where the number of directed paths which suffice to ‘spend’ the flow from source to sink is less than df+1 elements occurs when a vertex has at least two inflows and two outflows, all equal between them. In such a situation the flow from source to sink may be spent in less than df+1 directed paths. Such a vertex will be called breaking vertex. Note that an NMA treatment effect estimate for treatments separated by a bridge or a breaking vertex (say treatments x and y) can be seen as an “indirect” comparison through either the intermediate bridge (constituting by treatments b₁ and b2) or the intermediate breaking vertex (b0).
In particular, in the case of a bridge, the NMA treatment effect estimate is derived as
θ_xy^N=θ_xb1^N+θ_b1b0^N+θ_b2y^N
and in the case of a breaking vertex the NMA treatment effect estimate is
θ_xy^N=θ_xb0^N+θ_b0y^N
It is, thus, implied that in such a case the separate subnetworks contribute equally to the estimation of θ_xy^N.”

Is the rationale for developing the new method (or application) clearly explained?

Yes

Is the description of the method technically sound?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly
Competing Interests: No competing interests were disclosed.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

Click here to access the data.

Downloaded data do not display as expected? Download the data (0.79KB)

[1] 1. Eichler HG, Thomson A, Eichler I, et al.: Assessing the relative efficacy of new drugs: an emerging opportunity. Nat Rev Drug Discov. 2015; 14(7): 443–4. PubMed Abstract | Publisher Full Text

[2] 2. Jansen JP, Trikalinos T, Cappelleri JC, et al.: Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014; 17(2): 157–73. PubMed Abstract | Publisher Full Text

[3] 3. Salanti G, Del Giovane C, Chaimani A, et al.: Evaluating the quality of evidence from a network meta-analysis. PLoS One. 2014; 9(7): e99682. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. König J, Krahn U, Binder H: Visualizing the flow of evidence in network meta-analysis and characterizing mixed treatment comparisons. Stat Med. 2013; 32(30): 5414–29. PubMed Abstract | Publisher Full Text

[5] 5. Lu G, Welton NJ, Higgins JP, et al.: Linear inference for mixed treatment comparison meta-analysis: A two-stage approach. Res Synth Methods. 2011; 2(1): 43–60. PubMed Abstract | Publisher Full Text

[6] 6. Rücker G: Network meta-analysis, electrical networks and graph theory. Res Synth Methods. 2012; 3(4): 312–24. PubMed Abstract | Publisher Full Text

[7] 7. Macfadyen CA, Acuin JM, Gamble C: Topical antibiotics without steroids for chronically discharging ears with underlying eardrum perforations. Cochrane Database Syst Rev. 2005; (4): CD004618. PubMed Abstract | Publisher Full Text

[8] 8. Cormen TH, editor: Introduction to algorithms. 2. ed., 8. printing, Cambridge, Mass.: MIT Press [u.a.]; 2007; 1180. Reference Source

[9] 9. Rücker G, Schwarzer G: Reduce dimension or reduce weights? Comparing two approaches to multi-arm studies in network meta-analysis. Stat Med. 2014; 33(25): 4353–69. PubMed Abstract | Publisher Full Text

[10] 10. Cipriani A, Barbui C, Salanti G, et al.: Comparative efficacy and acceptability of antimanic drugs in acute mania: a multiple-treatments meta-analysis. Lancet. 2011; 378(9799): 1306–15. PubMed Abstract | Publisher Full Text

[11] 11. Confidence In Network Meta-Analysis [Internet]. Reference Source

[12] 12. Riley RD, Ensor J, Jackson D, et al.: Deriving percentage study weights in multi-parameter meta-analysis models: with application to meta-regression, network meta-analysis and one-stage individual participant data models. Stat Methods Med Res. 2018; 27(10): 2885–2905. PubMed Abstract | Publisher Full Text | Free Full Text

[13] 13. Krahn U, Binder H, König J: Visualizing inconsistency in network meta-analysis by independent path decomposition. BMC Med Res Methodol. 2014; 14: 131. PubMed Abstract | Publisher Full Text | Free Full Text

[14] 14. Flow contribution R package [Internet]. Reference Source

[15] 15. Papakonstantinou T, Nikolakopoulou A, Rücker G, et al.: Dataset 1 in: Estimating the contribution of studies in network meta-analysis: paths, flows and streams. F1000Research. 2018. http://www.doi.org/10.5256/f1000research.14770.d203174

Estimating the contribution of studies in network meta-analysis: paths, flows and streams

Abstract

Keywords

Revised Amendments from Version 2

Introduction

Motivating example

Methods

Two-stage network meta-analysis model

Box 1. Definitions

Comparison graph

Streams

Proportion contributions of direct comparisons

Algorithm to decompose flows into proportion contributions

Application

Proportion contributions of direct relative treatment effects to the estimation of the NMA effect between non-quinolone antibiotic and no treatment

Table 1. H matrix in the network of topical antibiotics without steroids for chronically discharging ears.

Figure 2. Illustration of the steps of the algorithm for approximating proportion contributions per comparison in the network of topical antibiotics without steroids for chronically discharging ears focusing on the comparison ‘x versus y’.

First iteration

Second iteration

Third iteration

Table 2. Proportion contributions of direct comparisons to the ‘x versus y’ network meta-analysis treatment effect in the network of topical antibiotics without steroids for chronically discharging ears.

Proportion study contributions

Table 3. Proportion contribution matrix P for the network of topical antibiotics without steroids for chronically discharging ears.

Table 4. Study proportion contribution matrix P* for the network of topical antibiotics without steroids for chronically discharging ears.

Using proportion study contributions to quantify the impact of a characteristic in a direct comparison

Figure 3. Bar plot showing the study proportion contributions of direct comparisons with low (green), moderate (yellow) and high (red) risk of bias.

Proportion contributions of direct comparisons in a large complex network of interventions

Figure 4. Network plot for the network of antimanic drugs.

Discussion

Data availability

Grant information

Supplementary material

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

The problem

How to fix it

Competing Interests Policy

Stay Updated