Network meta-analyses in national

clinical guidelines: a description and

quality assessment

Heather Stegenga, Gabriel Rogers, Prashanth

Kandaswamy, Dylan Jones

Internal clinical guidelines programme

Centre for clinical practice

National Institute for Health and Clinical Excellence (NICE)

United Kingdom

Background

• A number of treatment options

• How to decide?

• Often gaps in the literature between treatments

of interest

• Rarely one trial comparing all possible

treatments for a decision problem

• Decisions made implicitly?

Value of network meta-analysis is

making decisions more explicit that are

otherwise made implicitly

Indirect treatment comparison (ITC)

C

A B

No head-to-head trials of A vs B

Each line represents

1 or more trials

Treatment effect A vs B (indirect) = treatment effect C vs A (direct) – treatment effect C vs B (direct)

Plus, add the variances of treatment effect.

Indirect treatment comparison (ITC)

F

B

A

C

B

A

C

E

D

C

A B

Key feature: disconnected network

Mixed/multiple treatment comparison

(MTC) or multiple-treatment meta-

analysis (MTM)

C

A B

Indirect estimate of A vs B

Head-to-head trial(s) between A vs B (direct estimate)

Pool indirect and direct estimates to get more precise estimate of treatment

effect

Mixed/multiple treatment comparison

(MTC) or multiple-treatment meta-

analysis (MTM)

Important!

Need to check difference (or

consistency) between indirect and

direct estimates

Inconsistency = imbalance in effect

modifiers in indirect and direct

studies

Why combine direct and indirect?

• Helpful if there is some reason that we might not be confident in the results of the head-to-head study/studies: – Results are not precise

– Low study quality

– Manufacturer supported

– Study sizes

– Small number of trials (ie. only 1 head-to-head trial)

• Level of consistency/inconsistency across the network (between direct and indirect treatment effects) can be informative and provide coherence throughout the whole network of treatments

A developing methodology = different

terminology

Indirect

treatment

comparison

(ITC)

Network

meta-

analysis

Multiple

treatment meta-

analysis (MTM)

Multiple

treatment

comparison

(MTC)

Mixed

treatment

comparison

(MTC)

A developing methodology = different

terminology

Indirect

treatment

comparison

(ITC)

Network

meta-

analysis

Multiple

treatment meta-

analysis (MTM)

Multiple

treatment

comparison

(MTC)

For the purposes of this project,

mixed/multiple treatment

comparisons AND indirect

treatment comparisons are both

considered network meta-analyses

(NMA)

Mixed

treatment

comparison

(MTC)

NICE Decision support unit checklist –

4 main areas

A. Definition of decision problem

B. Method of analysis

C. Issues specific to the network synthesis

D. Embedding the synthesis into a probabilistic cost-

effectiveness model

Our objective

• To describe and quality assess the NMAs used in NICE

clinical guidelines

• Adapted version of checklist – added some descriptive

variables to provide some points of comparison

Inclusion criteria: published up to January 2012

Exclusion criteria: in development (including those in

the public domain which have been out for public

consultation)

NICE clinical guidelines

Total 145 published guidelines

8% (11) with NMA

-7 de novo

-3 from published study

-1 with both

Number of networks per guideline

5 3

1

15

1 3

2

6

2 1

3

02468

10121416

Type of interventions

55% 27%

18% pharmacological

various (includingpharmcological)

endoscopic (includingradiotherapy, etc) orsurgical procedures

Use in economic model

0

1

2

3

4

5

6

7

8

9

10

Used in economic model

unclear

Methodology

0% 10% 20% 30% 40% 50% 60% 70% 80%

Bayesian

Bucher's adjusted indirect comparison

"Adjusted indirect comparison"

Comparison of point estimates

unknown

Re: methods

Complexity of networks

Average (range)

Interventions in decision

space

8.4 (2-19)

Additional interventions in

network (for indirect

comparisons)

2.2 (1-5)

Number of connected loops

(MTC)

6.2 (1-14)

Interventions with only one

link

29% (0-55%)

Number of trials 41 (2-128)

Trials in head-to-head

comparison (MTC) (vs

indirect)

78% (50-100%)

Links with only one trial 47% (0-100)

Presentation of NMA

• Diagram/picture of the network:

– All but 1 guidelines with de novo NMA

(1 without was an ITC)

Schizophrenia (CG82)

Re: presentation

– 3 of the 8 guidelines with de

novo networks, including the

2 most recent

– Use with caution: doesn’t take

into account uncertainty

around treatment effect

Colorectal cancer (CG131)

Re: presentation

– Only one guideline used a rankogram

– Rankograms which take into account the probability

of being best, second best, etc are ideal

Depression (update) (CG90)

Inconsistency (MTC)

• Recommended methods:

– Repeat indirect methods for each ‘loop’

– Develop a model of inconsistency and compare this

with the standard network inconsistency

• Some guidelines said they checked for inconsistency

but it was often not clear how

• Of the 122 possible inconsistencies across the

networks, there were only 2 reported inconsistencies (2

different guidelines).

• There was very little discussion about inconsistencies in

many of the guidelines. Does that mean that few were

found?

Overall difficulties encountered

• Considerable variation of where details found

• Some variation expected ie. different developers,

different topic areas and, importantly, developing

methodology

• Methods were not reported or were unclear

• Checklist is new (available in early 2012) and is an

attempt to improve methods and reporting

Discussion

• Value of network meta-analysis is being explicit about

decisions otherwise made implicitly

• Value of being explicit is lost when description is lost

• Inability to assess the adequacy of decisions made from

the results of the NMA

Recommendations:

Fully describe methods used

Present networks in diagram

Use rankograms

Report and discuss inconsistency

across networks

Further information

• All guidelines available from www.nice.org.uk

• NICE Decision support unit: http://www.nicedsu.org.uk/

My contact details:

Heather.stegenga@nice.org.uk

Disclosure of Interests (last 3 years)

Heather Stegenga

I certify that, to the best of my knowledge, no aspect of my current personal or

professional situation might reasonably be expected to affect significantly my

views on the subject on which I am presenting.