LECTURE 2EPSY 642 META ANALYSIS

FALL 2009

CONCEPTS AND OPERATIONSCONCEPTUAL DEFINITIONS: HOW ARE

VARIABLES DEFINED? Variables are operationally defined by assigning a

measurement to the variable; height is measured by a ruler or by a 5 item rating scale of low to high

CONSTRUCT: CONCEPT EMBEDDED IN A THEORETICAL STANCE OR PERSPECTIVEIntelligence may differ in theories as single or

multifactor constructs; measurement by the Stanford-Binet or Wechsler Intelligence Scale for Children will yield different score(s)

CONCEPTS AND OPERATIONSStudies need to be clear on the conceptual

definitionsEx. Willson (Willson, V. L. (1983). A meta-analysis of the relationship between science

achievement and

science attitude: Kindergarten through college. J. Research in Science Teaching, 20, 839-

850.) noted that attitude was conceived of as “interest” in the 1930s and 1940s, differed from attitude measured using the psychological development of the construct of attitude in the 1960s

Meta-analysis needs to include these definitions in the coding and analysis process

CONCEPTS AND OPERATIONSOperational definitions within studies need to be

examined and coded- are they consistent with the conceptual definitions, and if not, how and why do they differ?

As the selection and reading of studies progresses, make room for revision in all phases of this process

What studies to add- do you include studies that use the operationally defined measures, but were not focused on your concept?Ex. Pretest sensitization: do I use studies that

“studied” sensitization or studies that included pretests? If the former, 200 studies, if the latter 100,000- or is there a middle ground?

Defining and Delineating the Research Topic• Outcome construct definition

– Importance to the field to know what has been learned– How big is it? How many potential studies? – Conduct preliminary searches using various databases

• Refining the construct– How much resource is available? Eg. 1000 studies = 2-

3 years work – Are there specific sub-constructs more important than

others? Select them or one of them– Are there time-limitations (no studies before 19xx)– Are there too few studies for the given construct,

should it be broadened? Too few-> less than 10?

Defining and Delineating the Research TopicWhat is the typical research approach for the

topic area?All quantitativeAll qualitativeMixed quantitative and qualitative

Are there sufficient quantitative studies to provide evidence for findings?Can qualitative studies be included as a

separate part of the study? How?

Within-Unit vs. Between UnitWithin-unit vs. Between-unit processes: is or

can the focus be on unit growth or is it a “snapshot” that may be evaluated at different time points, but the units are different at each time point.

Ex. IQ growth differs if one assesses a group of people every 5 years for 50 years (or subsets that are measured over fairly long periods of time that overlap in age) vs. measuring 10 groups of people at age 20,25,…70 right nowThe conclusion about IQ and age is very

different in the two sets of studies

Clinical/case studies vs. Group Studies

Ideographic vs. Nomothetic conclusionsIs the intended result to be for groups or for

individuals?Are there possible interaction effects across

levels of inference (student vs. classroom vs. school) – what level will be the focus for the study?

What is a case study in one article will be part of a group study in another: a school may be a case study or it may be part of an experimental design

Determining Research Outlet• Does the proposed journal

– publish research on the construct?– Publish reviews or meta-analyses?

• Is there a journal devoted to reviews that your project would fit with?

• Has a recent similar meta-analysis been published? If so, will yours add anything new?– Ex. Allen, et al (under review) evaluated articles on

first grade retention after 1990 focusing on the quality of the research design in each study to determine if the effects were different from a fairly recent meta-analysis by Jimerson (2001)

Meta-Analysis as an interactive, developing process• View meta-analysis as evolutionary

– As studies are reviewed and included, purpose and scope may change

• Assume initial conceptualizations about both outcomes and potential predictors may change over time– Definitions, instruments, coding may all change

as studies are found and included

• Plan for revisions to all aspects of the meta-analysis

CONCEPTUAL RELEVANCE AND SELECTIONSTART BROAD, THEN REDUCEPROVIDE FOR INTER-SCREENER

RELIABILITYSEE TABLE 2.1 IN COOPER

CAUSAL INFERENCEMAKING CAUSAL INFERENCES ABOUT A

TREATMENT CAUSING AN OUTCOME MUST BE BASED ON STUDIES THAT TEST THAT HYPOTHESIS

IT IS IMPERMISSIBLE TO USE DIFFERENT STUDIES TO MAKE THE INFERENCE THAT DID NOT MAKE THE TREATMENT COMPARISON WITHIN THE STUDIES Study X reported a 10 point gain on test A for

Treatment 1 with a random sample of children in Smallville

Study Y reported a 1 point gain on test A for Treatment 2 with a random sample of children in Smallville

We cannot conclude Treatment 1 is better than 2

FINDING STUDIESSearchesSelection criteria

Searches• Traditional literature review methods:

– Current studies are cumulated Branching backward search uses the

– Reference Lists of current studies• Electronic searches

– Google, Google Scholar, PsyInfo, research library catalogs (for major research institution libraries)

– Searches of major journal article titles and abstracts (commonly available now through electronic libraries)

– Querying listserves, informal research networks, “in-group” researchers, prominent researchers

• Abstract vs. full content searches- electronic, pdf, hard copy

• Author requests: email or hard copy requests for newly published articles or other works not found in typical search outcomes

Selection Criteria• In or out:

– Any quantitative data available?• Descriptive data- means and SDs for all groups of interest?• Analysis summaries- F- or t-tests, ANOVA tables etc.

available that may be utilized?

• Iterative process: outs may come back in given broader definitions of a construct

• Duplicated articles/data reports? Decide on which to keep (earliest? Most complete?) why were multiple articles prepared? New groups included that can be used?

• Keep records of every study considered- excel or hard copy, for example

Selection Criteria• Useful procedure:• Create an index card for each study along

with notes of each to refer to• Organize studies into categories or clusters• Review periodically as new studies are added,

revise or regenerate categories and clusters• Consider why you organized the studies this

way- does it reflect the scope of research, construct organization, or other classes?