hunting side effects and explaining them: should we reverse evidence hierarchies upside down?...
TRANSCRIPT
Hunting side effects and explaining them: should we reverse evidence
hierarchies upside down?
Barbara Osimani
Catholic University of Milan
Evidence and Causality in the SciencesCanterbury, 5-7 September 2012
Hunting side effects and explaining them: should we reverse evidence hierarchies upside down?
In their comparative analysis of RCTs andobservational studies, Papanikolau et al. (2006)assert:
“it may be unfair to invoke bias and confounding to discredit observational studies as a source of evidence on harms” (p. 640, my emphasis).
Hunting side effects and explaining them: should we reverse evidence hierarchies upside down?
• Recent contributions by philosophers and health scientists have acknowledged the role of so called "lower level" evidence as a valid source of information contributory to assessing the risk profile of medications both on theoretical (Aronson and Hauben, 2006; Howick et al. 2009) and on empirical grounds (Benson and Hartz, 2000; Golder et al. 2011).
• Nevertheless current practices have difficulty in assigning a precise epistemic status to this kind of evidence and in amalgamating it with standard methods of hypothesis testing.
Topics
• Recent proposals to amend evidence hierarchies especially Vandenbroucke’s suggestion to reverse hierarchies when addressing the issue of risk discovery and assessment;
• The requirement of total evidence as applied to this specific context: how it is and how it should be interpreted (vs. lexicographic rule implicit in ranking);
• I will illustrate some of these points by reference to the recent debate on the causal association between paracetamol and increased asthma prevalence/exhacerbation.
Hierarchy reversal for risk assessment
Vandenbroucke J.P. (2008) Observational Research, Randomised Trials, and Two Views of Medical Science. Plos Medicine, 5 (3): 339-43
Hierarchy of study designs for intended effects of therapy
Hierarchy of study designs for discovery and explanation
i. Randomised controlled trials i. Anecdotal: case report and series, findings in data, literature
ii. Prospective follow-up studies ii. Case-control studies
iii. Retrospective follow-up studies iii. Retrospective follow-up studies
iv. Case-control studies iv. Prospective follow-up studies
v. Anecdotal: case report and series v. Randomised controlled trials
Vandenbroucke’s defence of hierarchy reversal (I)
1. Methodological point:Observational studies concerning adverse reactionswill not suffer from confounding in the same way asobservational studies for intended effects do. selection bias is less likely to affect observational studies with respect to adverse reactions.This because unintended effects, qua unintended, are not knownin advance, and thus also not known by the drug prescriber, whocannot take them into consideration and thus bias treatmentallocation.
Ignorance of possible effect = “natural masking”
Vandenbroucke’s defence of hierarchy reversal (II)
2. Epistemological point:
Context of discovery vs. context of evaluation:
Discovery is focused on explanation and hypothesisgeneration;Evaluation instead on hypothesis testing/confirmation.
And research methods differ in the opportunities they offerwith respect to either of these goals.
Vandenbroucke’s defence of hierarchy reversal (III)
• Vandenbroucke (2008) formalizes the contrast between the context of evaluation and the context of discovery in terms of different priors assigned to hypotheses of benefits and of adverse reactions.
• High priors for intended effects• Low priors for unintended ones
Vandenbroucke’s defence of hierarchy reversal (III)
1. It is the higher priors which make the results more robust, not the method (Vandenbroucke, 2008: 16-17).
2. The reason why we accept uncertain results for risks rather than for benefits is that evaluation and discovery studies are associated with different loss functions:
1. evaluation is related to the approval of health technologies and is required to assure stakeholders about their efficacy and safety,
2. whereas discovery is more related to the context of research for its own sake, which might explain why certain study designs are preferred to others in different circumstances.
Vandenbroucke’s defence of hierarchy reversal (III)
1. Priors are quickly swamped by data2. Stakes are not lower for detecting risks than
for testing the drug benefit: adverse drug reactions might be so severe as to reverse the safety profile of the drug and determine its withdrawal.
Prior knowledge about drug’s general capacity to produce unintended adverse reactions
• The acceptability of anecdotal evidence or of uncontrolled studies for assessing risk has to do with a high prior about the general capacity of the drug to bring about side-effects.
• Whereas there is total ignorance as to some specific side effects which might be possibly caused by the drug, still there is almost certainty about the fact that the drug will indeed cause side-effects beyond the ones already detected in the pre-marketing phase.
• This high prior derives from historical knowledge and past experience with pharmaceutical products and is also strongly reflected in the regulation which introduced the notion of “development (or potential) risk”, the pharmacosurveillance system, and the precautionary principle.
Vandenbroucke’s defence of hierarchy reversal on abductive grounds
• “For discoveries, the original case reports, lab observations, data analysis, or juxtaposition in literature may be so convincing that they stand by themselves, either because of the magnitude of the effect or because the new explanation suddenly and convincingly makes the new finding fall into place with previous unexplained data or previous ideas”. (Vandenbroucke, 2008: 6).
Case Study: hypothesis of causal connection between paracetamol and asthma
Asthma increase in the United States and inWestern countries in the last 3 decades: up to a 75% increase among adultsand to a 160% among children in the sameperiod.(Burr et al., 1989; Eneli et al., 2005, Ninan and
Russel, 1992; Mannino et al., 1998, 2002, Seaton et al. 1994).
Explanatory hypotheses for asthma epidemic1) increased exposure to outdoor and indoor pollutants;
2) decreased exposure to bacteria and childhood illnesses during infancy (the “hygiene hypothesis”);
3) increased obesity incidence and prevalence;
4) changes in diet and oxidant intake;
5) cytokine imbalance as a reaction to environmental allergens in early childhood leading to lifelong T-helper type 2 (allergic) dominance over T-helper type 1 (nonallergic) reactions, thus increasing the risk for atopic disease
Eneli et al., 2005; Seaton et al. 1994, Shaheen et al. 2000.
How suspicion fell upon paracetamol• Varner and colleagues (1998) detected a precise correspondence between increase of
asthma incidence and increased paracetamol use as a substitute for aspirin (following the recognition of an association between aspirin and Reye’s syndrome).
• The trend levelled off in the 1990s, i.e. at a time when paracetamol had already become one of the most widespread analgesics.
• Varner and colleagues tentative explanation was however that asthma increase was due to aspirin avoidance, for the reason that aspirin may protect from asthma through inhibition of prostaglandins.
• However, this hypothesis was soon discounted on grounds that, if this had been the case, then one should have observed a decrease of asthma incidence when aspirin was first introduced (Shaheen et al. 2000).
• Thus the suspicion finally fell upon paracetamol itself and subsequent investigations explicitly aimed to examine the hypothesis of causal connection between paracetamol and asthma.
Evidence for causal association between paracetamol and asthma
“Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative:
1) consistency of the association across geography, culture and age;2) strength of the association (comparative studies); 3) the dose-response relationship between paracetamol exposure and asthma; 4) the coincidence of the timing of increasing asthma prevalence and increasing
paracetamol use;5) the relationship between per-capita sales of paracetamol and asthma
morbidity across countries; 6) our inability to identify any other abrupt environmental change that could
explain this increase in asthma morbidity; 7) plausible mechanism: glutathione depletion in airway mucosa caused by
paracetamol”.
McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).
Consistency of the association across geography, culture and age (I) Source Year of
study Study objective Population Results
Beasley et al. Cross-cultural study
2008 Examine the risk of asthma rhynoconjunctivitis and eczema in children using paracetamol
122 centers in 54 countries 200,000 children 6-7 yr
Dose dependent increase in prevalence and severity of asthma > once per year: OR 1.61 (95% CI 1.46-1.77) ≥ once per month: OR 3.23 (95% CI 2.91-3.60) Association identified at almost all sites regardless of geography, culture, stage of development
Beasley et al. Cross-cultural study
2011 Examine the risk of asthma rhynoconjunctivitis and eczema in adolescents using paracetamol
122 centers in 54 countries 320,000 children 13-14 yr old
Dose dependent increase in prevalence and severity of asthma > once per year: OR 1.43 (95% CI 1.33-1.53) ≥ once per month: OR 2.51 (95% CI 2.33-2.70) Association identified at almost all sites regardless of geography, culture, stage of development
Systematic review and meta-analysis of epidemiol ogic studies Etminan et al.
2009 Quantify the association between acetaminophen use and the risk of asthma in children and adults.
Thirteen cross-sectional studies, four cohort studies, and two case-control studies comprising 425,140 subjects
Pooled odds ratio (OR) for asthma among subjects using acetaminophen was 1.63 (95% CI, 1.46 to 1.77). The risk of asthma in children among users of acetaminophen in the year prior to asthma diagnosis and within the first year of life was elevated (OR: 1.60 [95% CI, 1.48 to 1.74] and 1.47 [95% CI, 1.36 to 1.56], respectively). Only one study reported the association between high acetaminophen dose and asthma in children (OR, 3.23; 95% CI, 2.9 to 3.6). There was an increase in the risk of asthma and wheezing with prenatal use of acetaminophen (OR: 1.28 [95% CI, 1.16 to 41] and 1.50 [95% CI, 1.10 to 2.05], respectively).
Consistency of the association across geography, culture and age (II)
Longitudinal birth-cohort study Amberbir et al.
2011 Investigate the independent effects of paracetamol and geohelminth infection on the incidence of wheeze and eczema in a birth cohort.
population-based cohort of 1,065 pregnant women from Butajira, Ethiopia,
Paracetamol use was significantly associated with a dose-dependent increased risk of incident wheeze (adjusted odds ratio = 1.88 and 95% confidence interval 1.03-3.44 for one to three tablets and 7.25 and 2.02-25.95 for ≥ 4 tablets in the past month at age 1 vs. never), but not eczema.
Wickens et al. Birth cohort study
2011 investigate the associations between infant and childhood paracetamol use and atopy and allergic disease at 5-6 years.
New Zealand Paracetamol exposure between birth and 15 months in Christchurch (n=505) and between 5 and 6 years for all participants (Christchurch and Wellington) (n=914). Outcome data collected at 6 years for all participants. Logistic regression models were adjusted for potential confounders
Paracetamol exposure before the age of 15 months was associated with atopy at 6 years [adjusted odds ratio (OR)=3.61, 95% confidence interval (CI) 1.33-9.77]. Paracetamol exposure between 5 and 6 years showed dose-dependent associations with reported wheeze and current asthma but there was no association with atopy. Compared with use 0-2 times, the adjusted OR (95% CI) were wheeze 1.83 (1.04-3.23) for use 3-10 times, and 2.30 (1.28-4.16) for use >10 times: current asthma 1.63 (0.92-2.89) for use 3-10 times and 2.16 (1.19-3.92) for use >10 times: atopy 0.96 (0.59-1.56) for use 3-10 times, and 1.05 (0.62-1.77) for use >10 times.
Cross-sectional analysis McKeever et al.
2005 To investigate the associations between use of pain medication, particularly paracetamol, and asthma, COPD, and FEV1 in adults.
Data from the Third National Health and Nutrition Examination Survey (U.S.) Participants aged between 20 and 80 years, with complete data for relevant exposures, outcomes
Dose–response association of paracetamol use and asthma (adjusted odds ratio, 1.20; 95% CI, 1.12–1.28; p value for trend 0.001).
Shaheen et al. Case-control study
2000 To investigate whether frequent use in humans was associated with asthma.
Adults aged 16–49 years registered with 40 general practices in Greenwich, South London. Frequency of use of paracetamol and aspirin was compared in 664 individuals with asthma and in 910 without asthma.
After controlling for potential confounding factors OR for asthma, compared with never users, was 1.06 (95% CI 0.77 to 1.45) in infrequent users (<monthly), 1.22 (0.87 to 1.72) in monthly users, 1.79 (1.21 to 2.65) in weekly users, and 2.38 (1.22 to 4.64) in daily users (p (trend) = 0.0002). This association was present in users and nonusers of aspirin.
Shaheen et al. Multicentric case-control study
2008 To examine whether or not frequent paracetamol use is associated with adult asthma across Europe.
The network compared 521 cases with a diagnosis of asthma and reporting of asthma symptoms with 507 controls with no diagnosis of asthma and no asthmatic symptoms across 12 European centres.
Weekly use of paracetamol, compared with less frequent use, was strongly positively associated with asthma after controlling for confounders. OR 2.87 95% CI 1.49-5.37 No association was seen between use of other analgesics and asthma.
Consistency of the association across geography, culture and age (III)
Evidence for causal association between paracetamol and asthma
“Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative:
1) consistency of the association across geography, culture and age;2) strength of the association (comparative studies); 3) the dose-response relationship between paracetamol exposure and asthma; 4) the coincidence of the timing of increasing asthma prevalence and increasing
paracetamol use;5) our inability to identify any other abrupt environmental change that could
explain this increase in asthma morbidity; 6) the relationship between per-capita sales of paracetamol and asthma
morbidity cross countries; 7) plausible mechanism: glutathione depletion in airway mucosa caused by
paracetamol”.
McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).
Comparative studies
Source Year of
study Study objective Population Results
Case-control study Shaheen et al.
2000 Determine if frequent paracetamol use is a risk factor for asthma.
Adults aged 16-51 yr in South London Cases: n = 720 (51% response rate) Controls: n = 980 (49% response rate).
Never users: OR 1.06 (95% CI 0.77-1.45); Monthly users: OR 1.22 (95% CI 0.87-1.72); Weekly users: OR 1.79 (95% CI 1.21-2.65); Daily users: OR 2.38 (95% CI 1.22-4.64); P value for trend = 0.0002
Prospective cohort study Shaheen et al.
2002 Examine the relationship between prenatal paracetamol use and wheezing in offspring at 6 mo.
9400 women Increased risk of wheezing before 6 mo for offspring of frequent paracetamol users over 20-32 wk prenatally: OR 2.34 (95% CI 1.24-4.40).
Prospective cohort study Barr et al. Nurses’ Health Study
2004 Examine the relationship between paracetamol use and new onset of asthma
73,321 women (44-69 yr)
Increased risk of diagnosis of new-onset asthma with frequency of use Adjusted RR 1.63, 95% CI 1.11-2.39 Dose dependence: p value for trend = 0.006
Randomized double blind trial without placebo Boston University Fever Study
2002 Compare the incidence of adverse reactions among children administered paracetamol or ibuprofen
84,000 febrile children Age ≤ 12 yr Randomly assigned paracetamol or low –dose ibuprofen, or high dose ibuprofen
Among 1879 children with pre-existing asthma, outpatient visits for asthma were lower in the ibuprofen arm than the paracetamol arm (RR 0.56 95% CI 0.34-0.95); + dose-dependence Hospitalizations were nonsignificantly lower (RR 0.63 95% CI 0.25-1.60).
Evidence for causal association between paracetamol and asthma
“Many observations suggest that the epidemiologic association between acetaminophen and asthma is causative:
1) consistency of the association across geography, culture and age;2) strength of the association (comparative studies); 3) the dose-response relationship between paracetamol exposure and asthma; 4) the coincidence of the timing of increasing asthma prevalence and increasing
paracetamol use;5) the relationship between per-capita sales of paracetamol and asthma
morbidity and across countries; 6) our inability to identify any other abrupt environmental change that could
explain this increase in asthma morbidity; 7) plausible mechanism: glutathione depletion in airway mucosa caused by
paracetamol”.
McBride JT (2011) The Association of Acetaminophen and Asthma Prevalence and Severity, Pediatrics, 128 (6).
Varner et al. Systematic review
1998 Investigate relationship between substitution of aspirin with paracetamol and increased asthma prevalence in developed countries.
Systematic review of U.S. studies
Epidemiologic trends, known biologic effects of cytokines and PGE2 on allergic sensitization, and a potentially relevant pharmacologic effect of aspirin used to explain a component of the increasing prevalence of childhood asthma in the United States.
Coincidence of the timing of increasing asthma prevalence and increasing paracetamol use (4)
“Although other changes in the environment have been suggested that might explain an increase in childhood asthma, none so easily explains the rapid increase in asthma in the 1980s and the subsequent levelling off of asthma
prevalence over the last 15 years. Furthermore, the prevalence of childhood wheezing in 36 countries around the world is predicted by each country’s per-
capita sales of paracetamol”. McBride (2011) (5)
Ecologic Study Newson et al.
2000 Examine the rate of Asthma and aggregate consumption of acetaminophen in 1994-95.
English speaking countries in the ECHRIS study.
Prevalence of wheeze increased by 0.52% for 13-14 yr olds; By 0.26% for young adults, For each gram increase in per capita paracetamol sales. Prevalence of childhood wheezing in 36 countries around the world is predicted by each country’s per-capita sales of paracetamol.
Relationship between per-capita sales of paracetamol and asthma morbidity and across countries (5)
Possible mechanisms
Paracetamol
Low glutathione level
Lack of suppression of cyclooxigenase
pathway
**IgE mediated immune response
Inability to counteract oxydative
stress
Defective antigen
processing
**Toxic effects of n.acetyl-p-benzoquinone
mine
Lung Injury with Bronchoconstriction
Source: Eneli et al. 2005
Proof onus
• McBride (2011) explicitly warns against the use of Paracetamol in children with asthma or at risk for asthma and claims that if further evidence is required, then this is for documenting product safety rather than the contrary.
• This explicitly addresses the reluctance of sceptical commentators to accept such evidence as a sufficient basis for practice change and for establishing a causal relationship between paracetamol and asthma, on grounds that it does not result from randomized clinical trials (Eneli et al. 2005, Allmers et al. 2009, Johnson and Ownby, 2011; Karimi et al., 2006, Wickens et al. 2011, Chang 2011).
Hierarchy reversal = chronological1. Non-controlled observational studies (e.g.
ecological studies)
2. comparative studies (case-controlled retrospective)
3. prospective studies
+ evidence coming from cellular and molecular studies.
• However, the justification of this hypothesis is neither exclusively focused on the exclusion of confounders (Vandenbroucke’s point 1), nor based on utility functions(Vandenbroucke’s point 2).
• Instead, several independent pieces of evidence jointly support a given hypothesis and are considered to do at least as good a job as the one-shot proof presumably provided by an RCT (or a meta-analysis of RCTs).
1. Both according to Vandenbroucke’s point 1, as well as to the recent contribution by Aronson and Haube, (or Howick’s contribution to the topic) case reports and observational data are considered sufficient evidence for causal claims to the extent that possible confounders (i.e. alternative causes for the experimental result) can be confidently excluded.
• This kind of reasoning also guides the general framework of evidence hierarchies: The higher the likelihood that the study design rules out more confounders than others, the higher it is settled in the ranking.
• And it is a straightforward consequence of the method of hypothesis testing, which is an hypothetico/deductive mode of investigation, for which the evidence is supposed to refute the null hypothesis.
2. Vandenbroucke’s call on priors instead (point 2) invokes a Bayesian epistemology, where hypotheses are assigned probabilities and these are updated in the light of new data.
3. In McBride’s, the causal hypothesis is assessed abductively, by putting things together and inferring the implications of their joint occurrence. The hypothesis of causal connection provides a unified explanation of the different pieces of evidence, which would otherwise need a series of distinct explanatory facts.
Epistemological paradigmsEpistemology Method Assumptions Justification of “lower level”
evidence
Unificationist (qualitative abduction)
Connection of data in light of explanatory hypothesis
Connectedness (ontological) Explanatory power of hypothesis in light of data.(see also recent proposal of integration of Bradford-Hill criteria: Stegenga, 2011; Howick et al. 2009).
Inductive-Bayesian (quantitative abduction)
Bayes theorem Principle of total evidence - coherence
Probability of hypothesis given evidence
Hypothetico-deductive (statistical mode)
Hypothesis testing: likelihood of evidence if H0 = true (p-value)
Homogeneous populations with regard to all possible relevant causal factors(randomization)
Only if alternative hypotheses (confounders) can be safely excluded, or treatment effect swamps them by a statistically significant amount (Howick, 2011).
• The unificationist paradigm regards hypotheses as explanatory factors for the observed data. In order to be really explanatory they must accommodate as many data as possible. Data which fail to be taken into account are left unexplained, thus making the hypothesis less virtuous from a theoretical point of view.
• In the Bayesian paradigm hypotheses can be associated with any probability in the unit interval. The main requirement is coherence (in the mathematical sense of standard probability calculus) and that all available evidence is used: this because, bayesian epistemology tracks inductive uncertainty and all non-deductive logics are non-monotonic.
Epistemological paradigms
Non-monotonicity Principle of total evidence
Nonmonotonicity means that an addition to the premises may invalidate some previous conclusion.
The principle of total evidence responds to this issue and is an essential principle of uncertain inference (Carnap, 1947).
Linda is getting out of the bank It is 4 pm.Linda is getting out of the bank and she is going to an
antinuclear demonstration It is earlier than 4 pm.
Hypothetico-deductive method (statistical mode)
• Instead statistical hypothesis-testing is a kind of approach which admittedly follows a Popperian hypothetico-deductive method of scientific enquiry. And being this paradigm inherently deductive, it does not feel urged to address the issue of non-monotonicity.
• The very idea of hierarchies follows from the assumption that if you have a study which has the capacity to eliminates more confounders than another, than the former should trump the latter. Trumping means that higher level evidence discards any evidence of inferior ranking, and also makes it irrelevant.
Lexicographic rule for hierarchy implementation
Higher level studies trump lower level ones:1. when two studies of different levels deliver
contradictory findings, then the higher in the evidence study is considered more reliable and is allowed to discard the lower level one
2. lower level evidence adds nothing to higher level one and thus it can be neglected without loss of information.
• The strongest way to interpret hierarchies is to claim that, because it is assumed that randomization provides a guarantee for causality, than it should follow that there is no guarantee of causality without randomization.
• “No randomization in, no causes out”
Lexicographic rule for hierarchy implementation
Cause
• I think all instruments of investigation play an important role in scientific investigation.
• Analytical approaches work with truth conditions (RCTs set strict desiderata for considering sthg to be a cause for sthg else) but have strong methodological (only certain kind of evidence is at all meaningful) and epistemological limitations (failure to account for important phenomena such as causal interaction - see Muller).
• Inductive approaches work with imperfect indicators (P(E/H); dose-response relationship; specificity of the association); or theoretical virtues such as the explanatory power), thus not only they can, but they must take into account all evidence (non-monotonicity). Nevertheless they can give you no sure-fire guarantee that the delivered result demonstrates the truth of the hypothesis.
1. Probabilistic Hypothesis of Causal Connection
• From the time a risk is not known, to the moment in which it is incontrovertibly proven to be causally associated with the drug, there is a period of evidence accumulation which constitutes a state of partial and imperfect (but continuously increasing) knowledge.
• In this period it cannot be claimed that there is a causal link between the drug and the detected risk; but neither can we behave as if we knew nothing about it. Still, the latter attitude is precisely the only possible policy allowed by an epistemology grounded on hypothesis rejection.
2. Implementation of the precautionary principle
• Following the precautionary principle, you are not supposed to wait for the causal connection between harm and suspected drug to be certain, before you take adequate countermeasures, but instead, you should act as soon as the probability of causal connection is high enough to recommend countermeasures because of a negative risk/benefit balance.
• This probability might be also very low, in case the risk magnitude is considerably big with respect to the expected benefit.
• The frequentist mode of summarizing statistical data, following which hypotheses may only be accepted or rejected, cannot be of any use to this purpose.
• The problem of external validity and causal interaction is dramatic in the case of side-effects.
3. Problem of external validity (ontological
Conclusion• My aim is to raise awareness about the different
epistemological paradigms underlying the distinct evidence policies we may intuitively endorse.
• The take-home lesson is that different epistemologies grant different methodological actions and impose different, which in turn bring about relevant practical implications.
• Thus it is worthy to bear in mind the criteria underlying our evidence constraints whether we want to rank it or not, or else to reverse rankings.