CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONSPresented by:Bethany Harmon, Jenny Jackson, and Jill PawlowskiH 615November 22, 2013

Overview● Generalized Causal Inference: Methods for Multiple

Studies (Ch. 13)○ Multistudy programs of research○ Summaries of multiple studies

○ Narrative reviews ○ Quantitative reviews

● Critical Assessment of Assumptions (Ch. 14)● Reflection Activity (10 minutes)

Choosing a psychotherapist• Imagine for a moment you are experiencing a difficult emotional time in

your life…

• Because you are an informed, intelligent person, you do some reading on psychotherapy and discover that many different approaches are available.

• These assorted styles of psychotherapy, although they stem from different theories and employ different techniques, all share the same basic goal:

to help you change your life in ways that make you a happier, more productive, and effective person

http://www.edmondschools.net/Portals/3/docs/Terri_McGill/READ-TherapistChoice.pdf

Choosing a psychotherapist• Which type of therapy should you choose?

• What you would really like to know is: • Does psychotherapy really work? • If it does, which method works best?

• While many comparison studies have been done, most of them:• support the method used by the psychologists conducting the study• were rather small • are spread over a wide range of books and journals

• This makes a fully informed judgment extremely difficult.What should you do?

You have 2 options

Generalized causal inference: methods for multiple studies

• Single studies typically lack large, heterogeneous samples across persons, settings, times, treatments, and outcome measures; rarely use diverse methodologies

• Multiple studies usually have greater variation in all of the above = allows for better tests of generalized causal inference

http://library.downstate.edu/EBM2/2100.htm

Generalizing from multiple studies

• How is this accomplished?• Multistudy programs of research• Summaries of multiple studies

Multistudy programs of research

• Phased models• Same topic, same researcher/lab• Research proceeds in phases, from basic research discovery to

application in real world settings

• Directed programs of experiments• One researcher/one lab over time, or multiple researchers/multiple

labs at same time• Aim is to systematically investigate over many experiments the

explanatory variables (moderators, mediators, constructs) that may account for an effect, gradually refining generalizations

Multistudy programs of research

• Phased models• Example: phases of cancer research

• Pre-phase: basic research on the effects of a treatment on cancer cells in test tubes and on animals

• Phase I: review existing basic and applied research to determine testable hypothesis about new cancer agent

• Phase II: method development to ensure accurate and valid research procedures and technology are available to study the agent

• Phase III: intervention efficacy trials• Phase IV: intervention effectiveness studies• Phase V: field studies aimed at entire communities to determine public

health impact

Multistudy programs of research

• Phased models• Excel at exploring generalized causal inference• Rely on purposive sampling of persons, settings, times, outcomes, and

treatments• Address the 5 principles of causal explanation:

• Surface similarity: using human cancer cell lines to create comparable tumors on animals

• Ruling out irrelevancies: deliberately making patient characteristics diverse • Making discriminations: identifying which kind of cancers are most/least

affected• Interpolation/extrapolation: varying drug doses to see how toxicity and

clinical response may vary• Causal explanation: developing models of how a drug acts on human

cancer cells so that the potency of that action can be increased

Multistudy programs of research

Advantages

• Control over aspects of generalized causal inference from study to study

• Pursuit of precisely the questions that need answering at any given time

Disadvantages

Summaries of multiple studies• Same topic, different researchers• Allows more precise estimates of effects than could be had from

single studies• Better exploration of how causal relationships might change over

variation in study features• Helps clarify nature of relationship, boundaries, behavior within those

boundaries, and explanations• Wider knowledge base often results in more credibility than claims based on single studies

Narrative reviews of experiments• Describe existing literature using narrative descriptions without attempting

quantitative syntheses of study results• Reviewer relies on statistical significance levels to draw conclusions about

intervention effectiveness

• Summary of votes suggests whether treatment works • Also allows examination of potential moderators of the generalizability of

intervention effects

Narrative reviews combining experimental and nonexperimental research• Field experiments plus evidence from surveys, animal studies, and

basic laboratory studies

• Purpose is to match evidence across studies to a pattern that would be consistent with and explain an effect and clarify its generalizability• Useful when direct experimental manipulation of the treatment would be

unethical or impossible with humans

• Results can be controversial

Narrative reviews

Advantages

• Hypothesis generation• Thick description of a literature• Theory development with

qualitative categories and relationships among variables

Disadvantages

• Difficult to keep straight all of the relationships between effects and potential moderators

• Traditionally rely on box score summaries of results from significance tests of outcomes

• Imprecise descriptions of study results• Even more complex when trying to

examine relationships among outcomes and potential moderators

• Overwhelming!

Quantitative reviews of existing research• Glass (1976) coined the term meta-analysis

• AKA “Mega-Silliness”, “Blossoming Nose Blemish”, and “Classic of Social Science” • http://garfield.library.upenn.edu/classics1983/A1983QF87200001.pdf

• Meta-analysis converts each study’s outcomes to a common effect size metric, such that different outcomes have the same means and standard deviations.

• The common effect size metric can then be more readily averaged across studies

Essential steps of meta-analysis

• Problem identification and literature review• Aim: develop clear research question and framework of inclusion criteria for studies in the meta-analysis

• Studies included in review must address the question of interest and be relevant to the treatments, units, settings, measures, and times outlined in the guiding problem formation

• May also have methodological criteria

Essential steps of meta-analysis• Coding of studies

• Meta-analyses use a common coding scheme to quantify study characteristics and results

• Codes should reflect the researcher’s hypotheses• For example, in a review of interventions for childhood obesity, codes might be

developed based on whether the intervention included…• Behavioral, educational, environmental, diet, and/or physical activity features• Parental or family involvement• Outcome measures (e.g. BMI, change in diet or physical activity behaviors)

• Codes often include characteristics of the study report, participants, intervention, intervention process, and methodology

Essential steps of meta-analysis• Computing effect sizes

• Various study outcomes must be converted to a common metric before outcomes can be meaningfully compared over studies

• Two most appropriate effect size measures for meta-analyses of experiments:

• Standardized mean difference statistic (d) – for continuous outcome measuresdi = Xt

i – Xci where Xt

i = mean of treatment group in ith study

si Xci = mean of comparison group in ith study

si = pooled standard deviation of the two groups

• Odds ratio (o) – for dichotomous outcome measures oi = AD BC

Essential steps of meta-analysis• Analyzing meta-analytic data

• Theoretically, meta-analytic effect sizes are analyzed just like any other social or behavioral data, using both descriptive and inferential statistics with univariate and multivariate techniques

• Unusual statistical features• Desirability of weighting effect size estimates by a function of study sample size• Use of tests for homogeneity of effect sizes• Hierarchical nature of meta-analytic data• Dependency of effect sizes within studies• Presence of publication bias

Essential steps of meta-analysis• Interpreting and presenting results

• Generally, interpreting and presenting meta-analytic results poses few special problems

• Keep in mind: most meta-analytic data are correlational• We never randomly assign studies to the categories analyzed• Quasi-experimental design features to rule out threats to validity are rarely used

in meta-analysis• Meta-analysts record data observationally• Exception is for reviews of randomized experiments of the same intervention

Meta-analysis and the 5 principles of generalized causal inference

• Surface similarity: assess the match between research operations and the prototypical features of the targets of generalization

• Multiple studies typically represent many more constructs than any one study

• Greater chances of finding studies with operations that reflect constructs that may be of particular policy or research interest

• Reviews are superior to single studies in their potential to better represent prototypical attributes

Meta-analysis and the 5 principles of generalized causal inference

• Ruling out irrelevancies: consider deviations from the prototype that might not change the causal conclusion

• Must show that a given causal relationship holds over irrelevant features present in multiple studies

• Some meta-analyses aim to decrease, not increase heterogeneity of irrelevancies

• Others welcome heterogeneity to the extent that it resembles that found in the settings of desired application

Meta-analysis and the 5 principles of generalized causal inference

• Making discriminations: demonstrate that an inference holds only for the construct as specified, not for some alternative or respecified construct

• Aim is to make discriminations about the parts of the target persons, settings, measures, treatments, and time for which a cause and effect relationship will not hold

• Meta-analysis helps to clarify discourse about important constructs that have and have not been studied, and to clarify the range and boundaries of causal relationships

Relevance and irrelevance are always subject to revision over time

Meta-analysis and the 5 principles of generalized causal inference

• Interpolation and extrapolation: specify the range of persons, settings, treatments, outcomes, and times over which the cause and effect relationship holds more or less strongly or not at all

• Requires exploration of the existing range of instances to discover how effect sizes vary with position along the range

• The extremes are likely to be further apart and the intermediate values more numerous than in a single study

• Response surface modeling works better with meta-analysis than with single studies

Meta-analysis and the 5 principles of generalized causal inference

• Causal explanation: developing and testing explanatory theories about the target of generalization1. Easier to break down persons, settings, treatments,

and outcomes into their component parts in order to identify casually relevant components

2. Multiple regression can be used to identify redundancy among variables that moderate outcomes

3. Full explanation requires analysis of the micromediating causal processes that take place after a cause has been varied and before and effect has occurred

Conclusion• “Our enthusiasm for the methodology must be tempered by realistic

understandings of its limitations.” (Shadish et al, 2002; pg. 446)

• Flaws of meta-analysis are less problematic than those of narrative reviews• Be no less or more critical of meta-analysis than of any other scientific method

Quasi-experimentation: Ruling out Threats

• Relay heavily on researcher judgments especially on plausibility

• Improve inferences by adding design elements

Quasi-experiments: Pattern matching • Description

• Intended to rule out specific identified alternative causal pathways

• Problems• Specificity of theory• Chance

• More likely with more complex predictions• Statistical tests for fit over mean differences

• not as well developed

Excuses Against RCT• Problem: Quasi designs undermine likelihood of

conducting RCT

• RCT: Must be convinced of inferential-benefits over quasi before incurring costs

• Role of disciplinary culture

• “Each study calls for the strongest possible design, not the design of least resistance.” (pg. 487)

RCT Objection: Successful Implementation

• Comparison not between well don RCT and Quasi but between imperfect RCT and well done Quasi

• Counter:• RCT’s may have better counterfactual even with treatment

degradation• Even with attrition, RCT give better effect size estimates

• ** “Careful study and judgment to decide” (pg. 489)

RCT Objection: Strong Theory and Standardized Treatment Implementation

• Intervention-as-implemented

• Loss of statistical power • Large samples• Reduce influence of extraneous variables• Examine implementation quality

• Understanding of effectiveness in real world setttings

RCT Objection: Unfavorable Trade Offs?

• Critiques• Conservative Standards • Descriptive over explanatory • Belated program evaluation

• Counters• Based on artificial dichotomies, correctible problems, and

oversimplifications

RCT Objection: Invalid Model of Research Utilization

• Critique• Use of naïve rational choice model

• Counters• Causal relationships explained with information from multiple

sources• Contested rather than unanimous outcomes • Short-term instrumental use with minor variants on existing

practices • Research for conceptual changes

RCT Objection: Experiment and Policy Conditions Differ

• Experiment outcomes affected by:• Enthusiasm • Implementation differs form real-world setting• Random-assignment

• Counters• Lack of generalizability is an empirical question• Representation of real world conditions is a tradeoff • Planning can improve similarities in populations • True of any methodology• Disruptions in implementation is true in locally invented novel programs

• Major Problem: Effects of policy can only be seen by studding the entire event

RCT Objection: Imposing Treatment Vs. Local Solutions

• Critique:• Locally generated solutions produce better results

• Counters:• Experiments were needed to find out that imposed treatment did

not work • Null effects could be due to methodological weaknesses

• Distinguish between political-economic currency and intervention effects

Causal Generation Theory • Randomization allows for causation with few assumptions

• Follow 5-principles of generalized causal inferences when randomization is not an option• Researchers rely to heavily on weak purposive alternatives

• Causal generalization is more complicated than establishing causal relationships

Non-experimental Alternatives • Intensive causal studies

• Theory-based Evaluations

• Weaker quasi-experiments

• Statistical controls

Review of Key Terms• How do we know what is true?

• Correspondence theory• Coherence theory • Pragmatism

Review of Key Terms• Quine-Duhem thesis as a reaction to logical positivism• An epistemology is a philosophy of the justifications for

knowledge claims.• Naïve realism• Epistemological realism• Epistemological relativism

Shadish, Cook & Campbell on “Facts”

• Critical multiplism• “Stubbornly replicable” observations by independent critics• Building multiple theories into observations

• Dialectic process• Example of psychotherapy

• Use and misuse of causal knowledge

Reflection• What is the role of theory in the construction of

knowledge?• How will you choose your research question?• What do you do when you end up with non-significant

results?• If you have significant results, how can you be confident in

your findings?• How will you approach issues around use and misuse of

your findings?