Download - The Evaluation of RCTs & Observational studies in HTAs on Selected Medical Device Therapies
The Evaluation of RCTs & Observational studies in HTAs on Selected Medical Device Therapies:
WHAT IS THE REAL STATUS OF THIS RELATIONSHIP AND WHAT IMPLICATIONS MAY IT HAVE?
Liesl Birinyi-Strachan: Medtronic Australia
Kathy Cargill: Medtronic International
Gillian Barnett: G Barnett & Associates
Background to medical devices
• Increasing focus of HTA on medical devices internationally
• There are some important differences between devices and drugs
Device development is shorter & incremental in nature (~2yrs)
Sometimes difficulty in conducting RCTs of medical devices
- Issues with blinding (sham often unethical); learning curve for implantable devices (outcomes improve over time); small sample sizes (most severe patients); appropriate comparators; short length (rapid technology turnover)
Research Questions
1.Is all available clinical evidence - both non-randomised /observational AND RCTs studies - being looked for & included in HTAs on selected medical devices?
2.Is the quality of non-randomised /observational studies & RCTs in HTAs on selected medical devices being assessed?
Objectives
To collate, compare and contrast published international HTAs on selected medical device therapies relevant to Medtronic
• Literature searches conducted through to Nov 2011 Medical databases (Medline, Embase, Cochrane and CRD York HTA databases)
HTA agency websites searched Inclusion criteria: • Publically available HTAs from selected medical device therapies (relevant to Medtronic) that treat a range of diseases
• Full HTAs (excluded mini-reviews or HZ scanning reports)
• No language limitations (internal translation assistance)
Methods
• Characteristics extracted according to a pre-defined protocol
• Internal quality checks of extracted data
Included HTAs by therapy
5
5
5
5
8
9
9
10
0 1 2 3 4 5 6 7 8 9 10
Thoracic Endovascular Abdominal Aortic Repair(TEVAR)
Trancatheter aortic valve implant (TAVI)
Intrathecal Baclofen (ITB)
Pacemakers (IPGs)
Cardiac Resynchronisation Therapy (CRT)
Deep Brain Stimulation (DBS)
Sacral Nerve Stimulation (SNS)
Implantable Cardiac Defibrllators (ICD)
56 HTAs
in total
8 therapies
Included HTAs by country
11
9
8
7
7
5
3
2
2
1
1
0 2 4 6 8 10 12
CanadaUSASpain
AustraliaUK
FranceSwedenAustria
ArgentinaBelgiumPoland 11
countries24 agencies
MSAC
AHRQ, BCBS, ICSI, ECRI, WSHCA
CADTH, OHTAC, MUHC
CA HTA, SESCS, AETSA, ANAES, AVALIA, UETS
NICE, BTAG
HAS
SBU, VG
KCE
ICES
IALLC
LBI-HTA
Research Question 1:
Were both non-randomised /observational AND RCTs studies - being looked for & included in HTAs on selected medical devices?
How many HTAs included observational studies/non-RCTs compared to RCTs?
0
5
10
15
20
25
30
35
40
45
INCLUDED NOT INCLUDED
73%
27%
RCTsObs/Non-RCTs
RCTs
32%
Obs/ NonRCT
41/56
15/56
38/56
18/56
No RCTsavailable
68%
73% vs
68%
Were observational studies & non-RCTs explicitly excluded?
0%
20%
40%
60%
80%
Observationalstudies explicitly
excluded
Search strategynot provided
Observationalstudies includedbut none found
72%
YES
17% 11%
13/18Why?
3/182/18
Did the HTAs that explicitly excluded observational studies seek to evaluate efficacy?
NoIn fact, 54% (7/13) actually sought to
examine “effectiveness” of the therapy under evaluation No pragmatic RCTs were searched for &/or
identified in the 7 HTAs
Only 1/13 HTAs sought to evaluate efficacy
In the HTAs that sought to examine “effectiveness” but excluded observational data…what did they conclude?
6 out of 7 of the HTAs had favourable
conclusions on therapy “effectiveness” ….. based on
RCT evidence only large numbers of
Is the quality/ risk of bias of observational studies & RCTs in HTAs on selected medical devices being assessed?
Research question 2
Quality Evaluation
0%
10%
20%
30%
40%
50%
60%
No qualityassessment
Assigned levelsor a score ONLY
Detailed qualityassessment
RCTs Observational studies
24% vs 26%
52% vs 45%
24% vs 29%
Did the type of QA performed affect the conclusions of the HTA?
No QA
RCTs Observational
Levels of evidence only
RCTs
Comprehensive QA
RCTs
FavourableUnfavourable
More studies required
Observational
Observational
In Summary
Of the HTAs that explicitly excluded observational data the majority aimed to evaluate therapy effectiveness
Observational/ non-randomised studies ARE being included in HTAs of selected medical devices
68%
“Distinguishing efficacy from effectiveness and emphasizing its importance to decision making dates back to at least 1978 (Office of Technology Assessment 1978), so it is far from a new concept. ………..But it is somewhat disconcerting that confusion still exists.”
Luce et al 2010: EBM, HTA, and CER: Clearing the Confusion. The Milbank Quarterly, 88(2);256–276
“efficacy trials are all too commonly misspecified as leading to evidence of effectiveness, which can result in over-interpreting their value and relevance to routine practice settings and the resulting decisions.”
In Summary
25% of the HTAs studied failed to assess quality of the studies altogether
……..and a further 45-50% only assigned a simple level as a proxy quality measure
When a comprehensive quality assessment was performed on selected HTAs of medical device therapies, those HTAs that included non-randomised & observational studies were less likely to result in a favourable conclusion (36%) as compared to HTAs that included assessment of RCTs (71%)…
Main Limitations of this analysis
Generalisability: Represents an analysis of published HTAs on selected medical device therapies of relevance to Medtronic
Only HTAs that were publically available in full-text were included
This analysis
Thank you