the methodology and challenges of the caffeine...
TRANSCRIPT
Innovative Solutions
Sound Science
The Methodology and Challenges of the Caffeine Systematic Review: An Expert Perspective
Dr. Daniele Wikoff - ToxStrategies
Experimental Biology 2017
Disclosure
2
•Member of the ILSI North America Caffeine Systematic Review
Team
• ILSI North America provided support to attend this meeting
Multidisciplinary Team
3
Daniele Wikoff, Brian T. Welsh, Rayetta Henderson, Gregory P.
Brorby, Janice Britt, Esther Myers, Jeffrey Goldberger, Harris R.
Lieberman, Charles O’Brien, Jennifer Peck, Milton Tennebein,
Connie Weaver, Seneca Harvey, Jonathan Urban, Candace
Doepker
Outline – “Methods” and Challenges
4
•Initiating the review
•Finding and assessing individual studies
•Integration and synthesis
•Challenges and solutions
Initiating the Review
5
IOM: Initiating the Review
6
Topic Formulation: Nawrot et al. 2003
7
• Highly cited reference
• Authors from Health Canada
• Review of multiple endpoints
• Guidance on “safe” level
• 400 mg/day in healthy adults
• Format is easy to follow for the
reader
Topic Formulation: Overall Analytical Framework
8
Populations
Pharmacokinetics/pharmacodynamics included as a contextual topic
Healthy Adolescents (12-19)
Healthy Children (3-11)
Healthy Adults
Healthy Pregnant Women
< 2.5 mg/kg-day
<400 mg/day
<300 mg/day
Acute Toxicity
Bone and Calcium
Cardiovascular
Behavior
Reproduction &
Development
Comparator/Exposure Outcome
PECO Questions
9
Population Healthy
Adults
Pregnant
Women
Adolescents Children
Exposure > 400
mg/day
> 300
mg/day
> 2.5
mg/kg-day
> 2.5
mg/kg-day
Comparator ≤ 400
mg/day
≤ 300
mg/day
> 2.5
mg/kg-day
≤ 2.5
mg/kg-day
Outcome Acute, Reproductive and Developmental,
Cardiovascular, Behavior, Bone and Calcium
For [population], is caffeine intake above [dose/exposure], compared to intakes
[dose/comparator] or less, associated with adverse effects on [outcome]?
Outcomes and Endpoints: Terminology Clarification
10
Endpoint: Specific E.g.:
Mortality Heart Rate Cholesterol
Morbidity Heart Rate VariabilityAortic stiffness/wave
velocity
Blood Pressure Cerebral blood flow Catecholamines
Endothelial function Electrical activityOther hemodynamic
measures
Homocysteine Ventricular function
Outcome: Broad (e.g., Cardiovascular)
Finding Individual Studies
11
IOM: Finding and Assessing Individual Studies
12
Overview of Literature Search Strategy
13
• Three databases searched January 1, 2001 to June 8, 2015
• Outcome- and database-
specific syntax developed
with librarian (concatenated;
in protocols)
• Search results imported into
DistillerSR
DistillerSR
14
Software designed to perform, track, audit, and
report systematic review projects
• Web-based platform (allows for multiple users, team review, coordination, etc.)
• Documentation of inclusion/exclusion (audit log tracks all changes)
• Custom forms and reporting
Screening Approach
15
PilotingConduct Search
Remove Duplicates
Initial Screen (Titles and Abstracts)
QA, Committee
Review
Subject Expert Review
Full-Text Screen
Subject Expert Review
Included Studies
Screening Form Example
Abstract with
keyword highlighting
Dynamic form
and questions
Article title,
journal, and
other info
16
Study Screening and Selection – Inclusion/Exclusion
17
• Quantitate estimate of caffeine exposure• e.g. Coffee, tea, cola, energy drink, supplements
• Must have reported an effect for an Outcome of interest
• Healthy population only• i.e. not having been hospitalized or diagnosed with disease and/or receiving medical treatment for a disease
at the time of the study
• Caffeine must have been studied alone, or in one of the approved forms• Mixtures with alcohol, nicotine, acetaminophen, or other drugs were excluded
• Available in English
• Case reports included for Acute only
• Must present original data; reviews excluded• Exception: Meta-analyses were included
Full Text Screening and Data Extraction
18
• Full text versions obtained
• Study first checked against inclusion/exclusion
criteria
• Generate evidence tables (generated in
DistillerSR; reported in AHRQ-SRDR)
1. Basic information as reported by author (direct
extraction of information)
2. Customized information (some interpretation)
– Exposure data need to be standardized?
– Selection of data used for comparison
» No observed effect level (NOEL)
» Lowest observed effect level (LOEL)
Standardized Caffeine Content (Supplemental Materials)
19
24 24.836.8
47.2
80
95
0
20
40
60
80
100
Soft Drink(cola)
Tea(green)
Soft Drink(citrus)
Tea(black)
EnergyDrink
Coffee
mg
Ca
ffe
ine
pe
r 8
oz
Caffeine Source
Did author conduct analysis or report based on caffeine? If no,
Mitchell et al. 2014; 2015
Assessing Individual Studies
20
IOM: Finding and Assessing Individual Studies
21
Individual study assessment
22
• Internal validity (risk of bias)
Aims to discern if anything in the study design
compromises the credibility between exposure
and outcome
• External validity
Evaluates generalizability; considered
directness
• Level of adversity
Categorizes the level of importance in decision-
making and assists in weighing endpoints
OHAT Risk of
Bias Rating
Tool
OHAT
Handbook
(GRADE)
GRADE (Guyatt
et al., 2011)
OHAT Risk of Bias Tool
23
• "OHAT Risk of Bias Rating Tool for
Human and Animal Studies” (2015)
• Set of 11 questions
• Grouped under 6 types of bias: selection,
confounding, performance,
attrition/exclusion, detections, and selective
reporting
• Applied based on study design
• Questions are rated by selecting
among 4 possible answers
RoB (Internal Validity – “Quality”)
24
+2: Definitely low
+1: Probably low
-1: Probably high
-2: Definitely high
2. For each study and each
question, apply responses based
on potential for risk of bias:
OHAT Handbook (January 9, 2015)
Table 9. Example of a Visual Summary of Risk of Bias Ratings for Animal Studies
Risk of Bias Question Stu
dy
1
Stu
dy
2
Stu
dy
3
Stu
dy
4
Stu
dy
5
Stu
dy
6
Stu
dy
7
Stu
dy
8
Stu
dy
9
Stu
dy
10
Stu
dy
11
Stu
dy
12
Stu
dy
13
Stu
dy
14
Stu
dy
15
Stu
dy
16
Stu
dy
17
Stu
dy
18
Stu
dy
19
Randomization + − ++ ++ − ++ + + ++ − − − + + + − − + ++
Allocation concealment − − − − − − + − − − − − − − − − − − −
Confounding (design/analysis) ++ + ++ ++ ++ + ++ ++ ++ ++ + ++ ++ + − − − − ++
Unintended exposure + + + + + + + + + + + + + + + + + + +
Identical experimental conditions ++ ++ + + ++ ++ ++ ++ ++ + ++ + ++ ++ ++ ++ ++ ++ ++
Adhere to protocol + + + + − + + + + + + + + + + + + + +
Blinding of researchers during study − − − − − − + − − − − − − − − − − − −
Missing outcome data − + ++ ++ −− − + − − + −− − − + ++ + ++ + ++
Assessment of confounding variables + + ++ ++ ++ − + + ++ ++ + + + ++ ++ − + + ++
Exposure characterization ++ − + + − − + + − − − + + + + + + − +
Outcome assessment + + + + + + ++ + + − ++ + + + + + + + +
Blinding of outcome assessors + + + + ++ + + + + + + + −− + ++ + + + +
Outcome reporting + + + ++ −− + + + + − + + −− + + + ++ − +
Key:
Definitely low risk of bias ++
Probably low risk of bias +
Probably high risk of bias −
Definitely high risk of bias −−
Studies are evaluated on all applicable risk of bias questions based on study design. The rating or answer to each risk of bias question is selected on an outcome basis prior to determining the tier from 4 options: definitely low risk of bias (++), probably low risk of bias (+), probably high risk of bias (-), or definitely high risk of bias (--).
52
4. Array the responses in tabular format to
observe trends in the body of evidence
Source: NTP OHAT (2015)
1. Evaluate RoB question based
on study type.
• (Q1) Randomization
• (Q2) Allocation Concealment
• (Q3) Identical experimental conditions
• (Q4) Confounding (design/analysis)
• (Q6) Blinding during study
• (Q7) Missing outcome data
• (Q8) Confidence in exposure
• (Q9) Outcome assessment
• (Q10) Outcome reporting
• (Q11) Other (Exposure/purity)
3. Evaluate responses for individual studies
when considering findings
Integration and Synthesis
25
IOM Framework for Synthesizing the Body of Evidence
26
Qualitative Endpoint and Outcome Syntheses (IOM Standard 4.2)
27
• General characterization of literature identified
• Endpoint characterizations
• Summary of individual studies
– Plots of findings
– Risk of bias figures
• Body of evidence assessment (including confidence) and conclusion (tabular/narrative)
• Outcome characterization
• Body of evidence assessment (including confidence) and conclusion (tabular/narrative)
• Strengths, limitations, uncertainties
• Data gaps
Integration of the OHAT Framework
28
• Handbook for conducting SRs
and evidence integration
• Process for qualitatively
evaluating data
• Accommodated our
anticipated dataset
• Can be tailored/modified by
endpoint as appropriate
Synthesize Evidence and Rate Confidence
29
Confidence in Body of Literature (Endpoint/Outcome) (from OHAT; consistent with IOM)
30
Endpoint
No. of
Studie
s
Initial
ConfidenceOverall RoB Indirectness Magnitude Confounding Consistency
Final Confidence
Rating
Rating
Based on study
type and study
features (OHAT,
2015)
Domain-based
evaluation of
risk of bias per
the OHAT RoB
tool (OHAT,
2015)
Was the study
designed to
evaluate the
PECO?
Strength of
effect
(when effect
observed below
the
comparator)
Were plausible
confounders that
would change
the observed
effect accounted
for?
Were findings
consistent in
demonstrating
effects or lack of
effects at or
below the
comparator?
What is the overall
rating when factors
that increase or
decrease confidence
were considered?
Endpoint #High/ Moderate/
Low/ Very Low↓ / - / ↑ ↓ / - / ↑ ↓ / - / ↑ ↓ / - / ↑ ↓ / - / ↑
High/ Moderate/
Low/ Very Low
↑ increased confidence - no change to confidence ↓ decreased confidence
Key Considerations in Evaluation of WoE and Developing Conclusions
31
• Findings relative to the comparator
• Individual study context (e.g., internal and external validity)
• Confidence in evidence base (e.g., consistency,
magnitude)
• Level of adversity
Reported in a common structure (tabular, narrative)
Endpoints
Outcomes
Overall
Challenges and Solutions
32
PECO for multiple outcomes and endpoints
33
Population Healthy
Adults
Healthy
Pregnant
Women
Healthy
Adolescents
Healthy
Children
Exposure > # mg/day > # mg/day > # mg/kg-
day
> # mg/kg-
day
Comparator ≤ # mg/day ≤ # mg/day ≤ # mg/kg-
day
≤ # mg/kg-
day
Outcome Adverse effect (five outcomes)
• For [population], is caffeine intake above [exposure], compared
to intakes [comparator] or less, associated with adverse effects
on [outcome]?
Population: “healthy”
34
• Issue: Determining inclusion/exclusion based on healthy populations• Compromised physical function; caffeine as
a therapeutic
• Physicians and SR expert played a key role in helping with this determination
1. Define healthy• Subjects who were not specifically described
as hospitalized, diagnosed with disease, and/or receiving medical treatment for a disease at the time of the study
2. Impact on SR• Many abstracts/papers excluded based on
definition (retained if included healthy control arm or similar)
– E.g., diabetes, Parkinson’s
Example:
Outcome: “adverse”
35
• Issue: Nawrot et al. 2003 update to adverse effects
1. Define adverse• Use effects in Nawrot?
• Subjective, difficult to draw a line
• Decision: be conservative and comprehensive
2. Impact on SRs• Pilot searches utilized “adverse” term (and
related) – too restrictive
• Extracted information based on author conclusion as well as analyst conclusion
• Characterization of effects in subgroups (e.g., physiological/clinical, order in progression of effect, etc.)
Outcome: “adverse” – an added layer of complexity
36
• Issue: Caffeine exposure has been associated with benefits and adverse effects
• Adverse effects in studies that are evaluating benefits? (e.g., RCTs)
• Controlled exposure, etc.
• OR lack of adverse effects in studies evaluating benefits?
• Null findings on potentially adverse outcomes (e.g., heart rate)
• Decision
• Include: studies reporting data associated with adverse effects within a benefit/therapy study.
• Exclude: Studies assessing only beneficial or therapeutic endpoints or outcomes following exposure to caffeine.
Outcome: multiple endpoints, multiple outcomes
37
• Issue: Outcome of each review associated with many endpoints
• Decision: Comprehensive and inclusive (consistent w/ characterization of hazards)
• Impact: Frameworks and implementation activities need to be accommodating
Cardiovascular: Endpoint Keywords Identified From Abstracts
acute stiffness (pulse
wave velocity)chest pain and palpitation
stroke volume and
cardiac contractility
arrhythmia cholesterolsupra ventricular
dysrhythmias
atrial fibrillationendothelial
function/performanceventricular function
blood pressure heart rate
cardiovascular disease hypertension
cerebral blood flow myocardial blood flow
Data integration challenges
38
• Accommodation of framework to our question (beyond classification of hazard,
but rather evaluation of a specific dose)
Challenges with reporting
39
1. Length of the paper
• Balance of rigor and transparency with “standard” reporting
• Utilized supplemental materials and AHRQ SRSR
2. Conclusions – developing and reporting
• Clarity within manuscript
• Stakeholders are diverse
• Transparency & breadth of topics made it difficult to be “simple”
Project Flow and Timeline
40
Project Flow
41
PHASE 1. Initiate Systematic Review
1
2
3
4
Define project team & SAB
Determine project objectives
Develop a systematic review protocol
Select appropriate tools for review implementation
PHASE 2. Find & Assess Individual Studies
PHASE 3. Synthesize the Body of Evidence
PHASE 4. Report Systematic Review
1
2
3
4
Conduct comprehensive systematic literature search
Screen & select studies; document data collection
Report findings of literature search
Refine outline (i.e., topic areas)
Develop specific systematic review process
Obtain literature
5
6
1
2
Systematically assess the body of evidence (by topic area)
Conduct a qualitative synthesis (by topic area)
1
2
Prepare draft manuscript
Prepare updated draft for submission (based on SAB comments)
3 Peer-review journal submission and publication
SAB
Meeting
SAB
Meeting
SAB
Meeting
SAB
Meeting
~4 months from Nov 1, 2014 to
March 1, 2015
~11 months from March 1, 2015
to Feb 1, 2016
~8 months from Feb 1, 2016 to
Oct 1, 2016
~5.5 months from Oct 1, 2016 to
March 20, 2017
Questions?
42