meta-analysis dr. ken kavale. meta-analysis quantitative research synthesis “what the research...

META-ANALYSIS

Dr. Ken Kavale

META-ANALYSIS

Quantitative Research Synthesis

“What the research says”

Knowledge Accumulation

Usable Knowledge

METHODS OF RESEARCH SYNTHESIS

Narrative ReviewSerial Presentation

No Accumulation

Findings remain independent and isolatedIf Accumulated

SUBJECTIVE

QUANTITATIVE REVIEWS

Voting MethodBox-Score Analysis

Studies classified by statistical significance or non-significance

What is statistical significance?Publication Bias

(“File Drawer Problem”)

META-ANALYSIS

Advantages• Uses quantitative (statistical) methods for

organizing and extracting information• Eliminates study selection bias – no

prejudgments about research quality are made• Makes use of all information – findings are

transformed to commensurate expressions of effect magnitude

META-ANALYSIS (con’t)

Detects interactions – study characteristics that may mediate findings are defined and measured, and their covariation is investigated

Seeks general conclusions – practical simplicity

OBJECTIVITY

VERIFIABILITY

REPRODUCABILITY

METHODS OF META-ANALYSIS

Scientific Method

Formulating ProblemsBroadly defined“Is Method x effective?”

SamplingLiterature search- Comprehensive- Published and unpublished

METHODS OF META-ANALYSIS (con’t)

Classifying and Coding StudiesStudy features

Data AnalysisEffect Size – z score (standard deviation units)

__ __ ES = XE – XC

SDC

METHODS OF META-ANALYSIS (con’t)

Effect Size Interpretation• Overlapping distributions and comparable

percentiles• Comparative context• Statistical power – small (.20), medium (.50),

large (.80>)• Binomial effect size display (BESD) – practical

significance• Common language effect size – research

significance

SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE

Intervention ES Number of Studies

Large

Mnemonic strategies

(Mastropieri & Scruggs, 1989)

1.62 24

Reading-comprehension strategies

(Talbot, Lloyd, & Tankersley, 1994)

(Mastropieri et al., 1996)

(Swanson, 1999)

1.13

0.98

0.72

48

68

58

Behavior modification

(Skiba & Casey, 1985) 0.93 41

Direct instruction

(White, 1988) 0.84 25

SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE (con’t)


Medium

Cognitive behavior modification

(Robinson, Smith, Miller & Brownell, 1999) 0.74 23

Psychotherapy

(Weisz & Weiss, 1993) 0.71 110

Formative evaluation

(Fuchs & Fuchs, 1986) 0.70 21

Early intervention

(Casto & Mastropieri, 1986) 0.68 74

Stimulant medication

(Crenshaw, Kavale, Forness, & Reeve, 1999)

(Kavale, 1982)

0.67

0.58

115

135



Medium

Computer-assisted instruction

(Schmidt, Weinstein, Niemie, & Walberg, 1985-86)

0.66 18

Peer tutoring

(Cook, Scruggs, Mastropieri, & Casto, 1985-86)

0.58 19

Word recognition strategies

(Swanson, 1999) 0.57 54



Small

Psycholinguistic training

(Kavale, 1981) 0.39 34

Reducing class size

(Glass & Smith, 1979) 0.31 77

Psychotropic medication

(Kavale & Nye, 1984) 0.30 70

Social-skills training

(Forness & Kavale, 1996)

(Quinn, Kavale, Mathur, Rutherford, & Forness, 1999)

0.21

0.20

53

35



Small

Modality instruction

(Kavale & Forness, 1987) 0.14 39

Diet restrictions

(Kavale & Forness, 1983) 0.12 23

Perceptual training

(Kavale & Mattson, 1983) 0.08 180

Special class training

(Carlberg & Kavale, 1980) -0.12 50

PERCEPTUAL-MOTOR TRAINING

EFFECT SIZE.08

Table 1

Average ES for Perceptual-Motor Outcome Classes

Outcome Class Mean ES Power Analysis

Perceptual/Sensory Motor

.166 Small

Academic Achievement

.013 Negligible

Cognitive .028 Negligible

PERCEPTUAL-MOTOR TRAINING (con’t)

Table 2

Average ES for Perceptual-Motor General Outcome Categories

General Outcome Categories Mean ES Power Analysis

Perceptual/Sensory Motor

Gross motor

Fine motor

Visual perception

Auditory perception

.214

.178

.149

.122

Small

Small

Small

Small


Table 2

Average ES for Perceptual-Motor General Outcome Categories


Academic Achievement

Readiness

Reading

Arithmetic

Language

Spelling

Handwriting

.076

-.039

.095

.031

.021

.053

Negligible

Negative

Negligible

Negligible

Negligible

Negligible

Cognitive/Aptitude

Verbal IQ

Performance IQ

-.007

.068

Negative

Negligible


Table 3

Average ES for Perceptual-Motor Specific Outcome Categories


Gross Motor Skills

Body awareness/image

Balance/posture

Locomotor skills

.256

.263

-.017

Small

Small

Negative

Visual Perceptual Skills

Visual discrimination

Figure-ground discrimination

Visual-motor ability

Visual integration

Visual spatial perception

Visual memory

.146

.173

.222

.086

.144

.062

Small

Small

Small

Negligible

Small

Negligible


Table 3

Average ES for Perceptual-Motor Specific Outcome Categories


Reading Achievement

Word recognition

Comprehension

Oral reading

Vocabulary

Speed/rate

-.016

-.055

-.037

-.012

-.038

Negative

Negative

Negative

Negative

Negative


Table 4

Average ES for Subject Groups

Subject Mean ES Power Analysis

Normal

Educable Mentally Retarded

(IQ = 50-75)

Trainable Mentally Retarded

(IQ = 25-50)

Slow Learner (IQ = 75-90)

Culturally Disadvantaged

Learning Disabled

Reading Disabled

Motor Disabled

.054

.132

.147

.098

.045

.018

-.007

.121

Negligible

Small

Small

Negligible

Negligible

Negligible

Negative

Small


Table 5

Average ES for Grade Level

Level Mean ES

Power Analysis

Preschool

Kindergarten

Primary (Grades 1-3)

Middle (Grades 4-6)

Junior high school

High school

.053

.099

.079

.066

.085

.088

Negligible

Negligible

Negligible

Negligible

Negligible

Negligible

MODALITY-MATCHED INSTRUCTION(Learning Styles)

EFFECT SIZE.14

Table 1

Effects of Modality Assessments

Modality Mean ES

(uncorrected)

Mean ES

(corrected)

% of Subjects Differentiated from Comparison Group

Auditory .925 .552 71%

Visual .899 .506 70%

Kinesthetic .970 .430 67%

MODALITY-MATCHED INSTRUCTION(Learning Styles) (con’t)

Table 2

Effects of Modality Matched Instruction

Method Number of ES

Mean ES

Standard Error of ES

% of Subjects Differentiated from Comparison Group

Auditory 80 .184 .028 57

Visual 81 .086 .037 54

Kinesthetic 44 .175 .045 57

MODALITY-MATCHED INSTRUCTION(Learning Styles) (con’t)

Table 2

Effects of Modality Matched Instruction

Modality

Total Auditory Visual Kinesthetic

ES % ES % ES % ES %

Word

Recognition .150 56 .203 58 .081 53 .197 58

Comprehension .046 52 .062 52 .034 51 .041 52

Vocabulary .174 57 .194 58 .141 56 .185 58

Spelling .184 57 .249 60 .088 54 .216 59

STIMULANT MEDICATION

EFFECT SIZE.58.67

STIMULANT MEDICATION (con’t)

Table 1

Average Effect Sizes for Stimulant Medication Research Outcome Classes

Mean ES Power Analysis

Behavioral

Global improvement ratings .886 Large

Rating scales and checklist .837 Large

Activity level .846 Large

Attention and concentration .782 Large

Behavior (social and classroom) .634 Medium-Large

Anxiety .118 Small

STIMULANT MEDICATION (con’t)

Table 1

Average Effect Sizes for Stimulant Medication Research Outcome Classes

Mean ES Power Analysis

Cognitive

Intelligence .391 Medium

Achievement .383 Medium

Drawing and copying .467 Medium

Perceptual, memory and motor .412 Medium

Learning characteristics .367 Small-Medium

FEINGOLD DIET

EFFECT SIZE.12

Table 1

Average Effect Sizes for Feingold Diet Research Outcome Categories

Category Mean ES Power Analysis

Conners Scale-Parents .156 Small

Conners Scale-Teachers .268 Small

Global improvement .128 Small

Hyperkinesis rating .293 Small

Attention .015 Negligible

Disruptive behavior .052 Negligible

Impulsivity .153 Small

Learning ability -.055 Negative

INTERVENTIONS FOR ADHD

Stimulant Medication Feingold Diet Effect Size Effect Size

.63 .12

Treatment DurationStimulant Medication Feingold Diet

10 weeks 39 weeks

ComparisonStimulant medication better than 5X more effective than the

Feingold diet in ¼ the time

SPECIAL CLASS PLACEMENT

EFFECT SIZE-.12

Table 1

Average ES by Special Education Classification

Diagnosis Mean ES

Mental retardation (IQ 50-75) -.14

Slow learning (IQ 75-90) -.34

Learning disability or

Emotional/behavioral disorder

.29

SOCIAL SKILLS TRAINING

EFFECT SIZE

.21

.20

SOCIAL SKILLS TRAINING (con’t)

Table 1

Average ES for Teacher Assessments of Social Skills Training for Students with Learning Disabilities

Training Program Mean ES % Equivalent

Adjustment .29 62

Dependency .25 60

Conduct disorder .22 59

Interaction .11 54

Hyperactivity .07 53

Academic competence .05 52

Teacher assessment composite mean .16 56


Table 2

Average ES for General Education Peer Assessments of Social Skills Training for Students with Learning Disabilities

Components Mean ES % Equivalent

Communication .25 60

Acceptance .23 59

Cooperation .22 59

Friendship .22 59

Rejection .20 58

Interaction .20 58

Social status .13 55

Peer assessment composite mean .21 58


Table 3

Average ES for Self-Assessments of Social Skills Training for Students with Learning Disabilities

Components Mean ES % Equivalent

Social status .38 65

Self-concept .28 61

Social problem solving .28 61

Social competence .27 61

Interaction .19 58

Locus of control .08 53

Self-assessment composite mean .24 59


Table 4

Average ES of Social Skills Training by Teacher, Peer, Self, Experimenter, and Parent Assessments of Students with

Emotional/Behavioral Disorders

Rater Mean ES % Equivalent

Teacher .22 59

Peer .22 59

Self .22 59

Experimenter .19 58

Parent .15 56

Rater composite mean .20 58


Table 5

Average ES of Training on Social Skill Dimensions for Students with Emotional/Behavioral Disorders

Dimension Mean ES % Equivalent

Social behavior .27 61

Social relations .27 61

Social problem solving .26 60

Social competence .22 59


Table 6

Average ES of Training on Social Skill Areas for Students with Emotional/Behavioral Disorders

Area Mean ES % Equivalent

Family relations .20 58

Communication .18 57

School behavior .18 57

Conduct disorder .13 55


Table 7

Average ES of Training on Social Skill Variables for Students with Emotional/Behavioral Disorders

Variable Mean ES % Equivalent

Anxiety .42 66

Adjustment .27 61

Cooperation .26 60

Interaction .24 59

Self-concept/esteem .16 56

Aggression .13 55

meta-analysis dr. ken kavale. meta-analysis quantitative research synthesis “what the research...

Documents