conducting meta-analyses

Conducting Meta-Analyses

Marsha Sargeant, M.S.

Design And Statistical Analysis Laboratory

University of Maryland, College ParkDepartment of Psychology

2

Overview of Presentation

1. What is a meta-analysis and why is it important?

2. Overview of procedures involved in conducting a quantitative meta-analysis

3. Database structure

4. Interpretation of effect sizes

3

Meta-analysis Definition

A statistical analysis of the summary findings of many empirical studies

It’s quantitative!– Distinct from a meta-review

4

Background

Empirical findings grew exponentially in the middle 50 years of the 20th century

– Multiplied beyond our ability to comprehend and integrate it

– Hence a growing need to statistically and technically review, rather than through narrative

5

Background

Review of practices and methods of research reviewers and synthesizers in the social sciences (Jackson, 1978)

Failure to report methods of reviewing

6

Benefits of Meta-analyses

Increased statistical power Identification of sources of variability across

studies (e.g., inclusion of moderators) Detection of biases (e.g., Tower of Babel

bias) Detection of deficiencies in design, analysis,

or interpretation

Ioannidis & Lau, 1999

7

Limitations of Meta-analyses

Cannot improve the original studies Method is frequently misapplied Can never follow the rules of science

– Sources of bias are not controlled

Ioannidis & Lau, 1999

8

Rules of the Game

It is quantitative There is no arbitrary exclusion of data File drawer effect

– Dissertation research is research too!– Unpublished studies

Meta-analysis seeks general conclusions– It is contradictory to think that we can only compare

studies that are the same (if they were the same you wouldn’t need to compare them!)

Glass, 2000

9

Methodological Adequacy of Research Base

Findings must be interpreted within the bounds of the methodological quality of the research base synthesized.

Studies often cannot simply be grouped into “good” and “bad” studies.

Some methodological weaknesses may bias the overall findings, others may merely add “noise” to the distribution.

From “Practical Meta-analysis” by D.B. Wilson

10

Confounding of Study Features

Important study features are often confounding, obscuring the interpretive meaning of observed differences

If the confounding is not severe and you have a sufficient number of studies, you can model “out” the influence of method features to clarify substantive differences


11

Meta-analysis Overview

Descriptives– Effect sizes (e.g., correlation coefficients)– Distribution and central tendency summarized

Method section– Databases searched– Journals– What attempts were made to not have a biased search?– Criteria for inclusion– No effect studies

Rosenthal, 2005

12


Study quality– Use a weighting system– Use raters and non-dichotomous ratings to

avoid weighter bias – Optimally raters should be blind to the results of

the study– Ratings can be used as an adjustment on effect

size or as a moderator to determine whether quality is related to obtained effect size

Rosenthal, 2005

13


Consider independence of studies– Treat non-independent studies as a single study with

different dependent variables Recorded variables

– Number, Age, Sex, Education, etc– Volunteer status– Laboratory or field study?– Randomized?– Method of data collection (e.g., interview vs

questionnaire)– How constructs are operationalized– etc.

Rosenthal, 2005

14


Summarize recorded variables Study characteristics could all be potential

moderators of outcome aside from those with particular meaning for the specific area of research

Effect sizes (there are others)– R– Zr (Fisher’s r-Z transformation)– d family

Cohen’s d Hedge’s g Glass’s delta

Rosenthal, 2005

15

Examples of Different Types of Effect Sizes

Standardized mean difference– Group contrast research

Treatment groups Naturally occurring groups

– Inherently continuous construct Odds-ratio

– Group contrast research Treatment groups Naturally occurring groups

– Inherently dichotomous construct Correlation coefficient

– Association between variables research

From “Practical Meta-analysis - The Effect Size” by D.B. Wilson

16

Interpreting Effect Size Results

Cohen’s “Rules-of-Thumb”– standardized mean difference effect size

small = 0.20 medium = 0.50 large = 0.80

– correlation coefficient small = 0.10 medium = 0.25 large = 0.40

– odds-ratio small = 1.50 medium = 2.50 large = 4.30


17

Interpreting Effect Size Results

Rules-of-Thumb do not take into account the context of the intervention– a “small” effect may be highly meaningful for an

intervention that requires few resources and imposes little on the participants

– a small effect may be meaningful if the intervention is delivered to an entire population (prevention programs for school children)

– small effects may be more meaningful for serious and fairly intractable problems


18


Significance levels recorded – Recorded as the one-tailed standard normal

deviates associated with p’s E.g., p’s of .10, .01., .001 would be recorded as Z’s of

1.28, 2.33, and 3.09

19


Report central tendency– Unwieghted mean effect size– Weighted mean effect size (weighting by size of study –

can also use quality or other characteristic of interest)– Median– Proportion of studies showing effect sizes in the expected

direction– Report number of studies reported on– Optional: total number of participants on which the

weighted mean is based– Optional: median number of participants per obtained

effect size

20


Report variability– Standard deviation– Max and min effect size found at the 75th and

25th percentile– If normally distributed, the standard deviation is

estimated at .75(Q3-Q1)

21

Database Structure

Database structures– The hierarchical nature of meta-analytic data– The familiar flat data file– The relational data file– Advantages and disadvantages of each– What about the meta-analysis bibliography?

From “Practical Meta-analysis – Database Structure” by D.B. Wilson

22

Database Structure

Meta-analytic data is inherently hierarchical Any specific analysis can only include one

effect size per study (or one effect size per sub-sample within a study)

Analyses almost always are of a subset of coded effect sizes. Data structure needs to allow for the selection and creation of those subsets


23

Example of a Flat Data File

ID Paradigm ES1 DV1 ES2 DV2 ES3 DV3 ES4 DV422 2 0.77 323 2 0.77 331 1 -0.1 5 -0.05 5 -0.2 1136 2 0.94 340 1 0.96 1182 1 0.29 11

185 1 0.65 5 0.58 5 0.48 5 0.068 5186 1 0.83 5204 2 0.88 3229 2 0.97 3246 2 0.91 3274 2 0.86 3 -0.31 3 0.79 3 1.17 3295 2 7.03 3 6.46 3 . 3 0.57 .626 1 0.87 3 -0.04 3 0.1 3 0.9 3

1366 2 0.5 3

Note that there is only one record (row) per study

Multiple ESs handled by having multiplevariables, one for each potential ES.


24

Database Structure

Advantages and Disadvantages of a Single Flat File Structure Advantages

– All data is stored in a single location– Familiar and easy to work with– No manipulation of data files prior to analysis

Disadvantages– Only a limited number of ESs can be calculated per study– Any adjustments applied to ESs must be done repeatedly

When to use– Interested in a small predetermined set of ESs– Number of coded variables is modest– Comfort level with a multiple data file structure is low


25

Database StructureExample of Relational Data Structure(Multiple Related Flat Files)

ID PubYear MeanAge TxStyle100 92 15.5 2

7049 82 14.5 1

OutcomeID ESNum Type TxN CgN ES

100 1 1 24 24 -0.39100 2 1 24 24 0100 3 1 24 24 0.09100 4 1 24 24 -1.05100 5 1 24 24 -0.44

7049 1 2 30 30 0.347049 2 4 30 30 0.787049 3 1 30 30 0

Note that a single record in the file above is “related” to five records in the file to the right

Study Level Data File

Effect Size Level Data File


26

Database StructureExample of a More Complex MultipleFile Data Structure

ID PubYear MeanAge TxStyle100 92 15.5 2

7049 82 14.5 1

Study Level Data File Outcome Level Data FileID OutNum Constrct Scale

100 1 2 1100 2 6 1100 3 4 2

7049 1 2 47049 2 6 3

ID OutNum ESNum Months TxN CgN ES100 1 1 0 24 24 -0.39100 1 2 6 22 22 0100 2 3 0 24 24 0.09100 2 4 6 22 22 -1.05100 3 5 0 24 24 -0.44100 3 6 6 22 21 0.34

7049 1 2 0 30 30 0.787049 1 6 12 29 28 0.787049 2 2 0 30 30 0

Effect Size Level Data FileNote that study 100 has 3 records in the outcomes data file and 6 outcomes in the effect size data file, 2 for each outcome measured at different points in time (Months)


27

Database Structure

Advantages & Disadvantages of Multiple Flat Files Data Structure Advantages

– Can “grow” to any number of ESs– Reduces coding task (faster coding)– Simplifies data cleanup– Smaller data files to manipulate

Disadvantages– Complex to implement– Data must be manipulated prior to analysis (creation of “working”

analysis files)– Must be able to select a single ES per study for any given analysis

When to use– Large number of ESs per study are possible


28

What about Sub-Samples?

So far I have assumed that the only ESs that have been coded were based on the full study sample

What if you are interested in coding ESs separately for different sub-samples, such as, by gender or SES

– Just say “no”! Often not enough of such data for meaningful analysis Complicates coding and data structure

– Well, if you must, plan your data structure carefully Include a full sample effect size for each dependent measure

of interest Place sub-sample in a separate data file


29

Tips on Coding

Paper Coding– include data file variable names on coding form– all data along left or right margin eases data entry

Coding Directly into a Computer Database


30

Example Screen from a ComputerizedDatabase for Direct Coding

Figure 5.11: Example FileMaker Pro Screen for Data Entry from the ChallengeMeta-Analysis

31

Coding Directly into a Computer Database

Advantages– Avoids additional step of transferring data from paper to

computer– Easy access to data for data cleanup– Data base can perform calculations during coding process

(e.g., calculation of effect sizes)– Faster coding

Disadvantages– Can be time consuming to set up

the bigger the meta-analysis the bigger the payoff– Requires a higher level of computer skill


32

Final Comments

Meta-analysis – is a replicable and defensible method of

synthesizing findings across studies– often points out gaps in the research literature,

providing a solid foundation for the next generation of research on that topic

– illustrates the importance of replication– facilitates generalization of the knowledge gain

through individual evaluations


33

Thank You!

Email: [email protected]

Web: www.umd.academia.edu/MarshaSargeant

conducting meta-analyses

Documents