bringing educational innovation to scale: top-down, bottom-up, or a third way? 1 doug fuchs, kristen...
TRANSCRIPT
Bringing Educational Innovation to Scale: Top-Down,
Bottom-Up, or A Third Way?
1
Doug Fuchs, Kristen McMaster, Laura Saenz, Devin Kearns, Lynn Fuchs, Loulee Yen,
Don Compton, Chris Lemons, Wen Zhang Vanderbilt University
Chris SchatschneiderFlorida State University
R305G04104 Institute of Education Sciences
Big Q and Presentation Focus
If we can think of Grade 2-6 Peer-Assisted Learning Strategies (PALS) as a “best-evidence” reading practice, how do we scale it up? More importantly, how do we scale up any worthy educational innovation?
This was the general Q of an IES Goal 4 study conducted between 2004-2010. Years 1 & 2, at kindergarten; Years 3-5, at grades 2 thru 5.
Why this question is important practically and theoretically.
2
Presentation Overview
Purpose of PALS-Reading
Grade 2-6 PALS program
Challenges involved in scaling up
Re-thinking scaling up: “top-down” & “bottom-up”
Findings
3
Purpose of PALS
Supplements the general education core program
It is a peer-mediated program, and facilitates intensive practice of core academic skills
Implemented 3 times per wk in reading; twice in math
Creates a structure for teachers to differentiate instruction: many simultaneous peer-mediated lessons rather than one teacher-directed lesson
PALS-R: K, first grade, grades 2-6, high school
PALS-M: K, first grade, grades 2-6 4
PALS Research
Originally based on “Juniper Gardens Classwide Peer Tutoring” (e.g., Greenwood et al., 1989)
More than 15 yrs of school-based research Title I and Non-Title I schools; urban and
suburban High, average, and low achievers Students with LD “Validated Practice” status (USDE, WWC, BEE)
5
Grades 2-6 PALS
Partner Reading
Paragraph Shrinking
Prediction Relay
6
Partner Reading
Conducted for 11-12 minutes Stronger reader reads aloud for 5 minutes Weaker reader reads same text aloud for 5
minutes Weaker reader retells story for 1-2 minutes Readers read quickly, correctly, and with
expression Coaches listen, correct mistakes, and mark
points (1 point for each correctly read sentence and 10 points for story retell)
7
Paragraph Shrinking
Conducted for 10 minutes For 5 minutes:
Stronger reader reads new text aloud, summarizing paragraph by paragraph
– Name the most important who or what (1 point)– Name the most important thing about the who or
what (1 point)– Shrink it to 10 or fewer words (1 point)
For next 5 minutes: Weaker reader reads new text aloud, summarizing
paragraph by paragraph (as above) Coach listens, corrects mistakes, and marks points
8
Prediction Relay
5 minutes, stronger reader read new textMakes prediction (1)Reads half page (1) Checks prediction (1)States main idea (3)Makes new predictionContinues to read
5 minutes, weaker reader continues on in new text, with the same activities
Coach listens, corrects mistakes, and marks points 9
Support to Teachers in K-PALS & Grade 2-6 PALS R&D During program development and validation, research
assistants provided in-class help 1x or 2x per wk.
Support is costly, finite, and typically disappears during scale up. Absent support, quality of implementation often suffers.
Obvious Q: How to successfully de-couple an effective intervention from a support system and “brand name” (culture)? How to sustain and scale-up?
10
Challenges to Scaling Up
Just described is the well known problem of moving an innovation from A to B.
Less well known problem: sustaining an innovation across time, or dealing with “the changing counterfactual,”
Case in point: K-PALS.
11
K-PALS:Data from 1998-99 Study
Nashville
12
Phonological Awareness Effects by Achievement Level for 1990s Data
13
Phonemic Awareness: Segmenting
-0.2
0
0.2
0.4
0.6
0.8
1
PhonemicAw areness v.
Control
Decoding v.Control
PA + Decoding v.Control
Contrast
Effe
ct S
ize
Low -achieving
Average-achieving
High-achieving
Overall
Word Reading Effects by Achievement Level for 1990s Data
14
Word Reading: WRMT-R Word Identification
-0.2
0
0.2
0.4
0.6
0.8
1
PhonemicAw areness v.
Control
Decoding v.Control
PA + Decoding v.Control
Contrast
Effe
ct S
ize
Low -achieving
Average-achieving
High-achieving
Overall
Pseudo-word Reading Effects by Achievement Level for 1990s Data
15
Word Reading: WRMT-R Word Attack
-0.2
0
0.2
0.4
0.6
0.8
1
PhonemicAw areness v.
Control
Decoding v.Control
PA + Decoding v.Control
Contrast
Effe
ct S
ize
Low -achieving
Average-achieving
High-achieving
Overall
Fluency Effects by Achievement Level for 1990s Data
16
Fluency
-0.2
0
0.2
0.4
0.6
0.8
1
PhonemicAw areness v.
Control
Decoding v.Control
PA + Decoding v.Control
Contrast
Effe
ct S
ize
Low -achieving
Average-achieving
High-achieving
Overall
Elements of K-PALS
17
Teacher-led phonological awareness
Peer practice Sound-letter
correspondenceSight words readingWord readingSentence readingBook reading
Points earned for doing activities
Explaining Vanishing Effects Over Time: Segmenting
18
Explaining Vanishing Effects: Word Identification
19
Explaining Vanishing Effects: Pseudo-Word Reading
20
Explaining Vanishing Effects: Reading Fluency
21
Grade 2-6 PALS:Years 3, 4, & 5 of IES Study
22
PALS vs. Controls, 2006-2007
23
2006-2007 (Cohort 1) Results
-0.2
0
0.2
0.4
0.6
0.8
1
Control v.s. PALS
Pre-Post Contrast
Effe
ct S
ize TOWRE
ITBS
MAZE
PALS vs. Controls, 2007-2008
24
2007-2008 (Cohort 2) Results
-0.2
0
0.2
0.4
0.6
0.8
1
Control v.s. PALS
Pre-Post Contrast
Effe
ct S
ize
TOWRE
ITBS
MAZE
Moment of Truth (for the melodramatically-inclined)
25
Fear…depression…
Thinking differently: Top-down vs. bottom-up
“Now at starbucks: A rebound”
26
Method
Top Down PALS
27
4 Activities: Partner Reading (10 Minutes)Retell (2 Minutes)Paragraph Shrinking (10 Minutes)Prediction Relay (10 Minutes)
35 Minutes per Session
Reciprocal peer tutoring
A motivational peer reinforcement system
Emphasis on fidelity – Using PALS the way it was designed, without modifications.
Bottom Up PALS
28
Core Elements of PALS: 10 minutes of Partner Reading 10 minutes of Paragraph Shrinking 35 minutes per session minimum A motivational peer reinforcement system
Customized Elements of PALS Small “tweaks” Big changes Flexibility
29
Participating Sites across 5 Years
30
Participating Teachers in Years 3-5 (Grade 2-6 PALS)
Two cohorts of 3rd-, 4th-, and 5th-grade teachers:Cohort 1
Entered study in 2006-07Cohort 2
Entered study in 2007-08Two years of study participation:
Year 1 Assigned randomly to PALS or Control
Year 2 PALS Teachers selected Top Down or Bottom Up
PALS Control Teachers remained in Control group
31
Teachers by Year 2 Study Group
32
Students by Year 2 Study Group
Student Measures in Years 3, 4, & 5
Academic Measures Test of Word Reading Efficiency (TOWRE) Woodcock Reading Mastery Test-Revised
(WRMT-R) Word Identification Subtest
Wide Range Achievement Test (WRAT) Letter and Word Identification
Subtests Comprehensive Reading Assessment
Battery (CRAB; 2 passages) Oral reading (1 min & 3 min) Comprehension (10 open-ended
questions) CBM Maze Task (2 passages)
Correct maze choices made in 2.5 min Iowa Test of Basic Skills (ITBS)
Reading Comprehension Vocabulary
Student Characteristics Demographics SWAN
Teachers rated each student’s abilities to focus attention, control activity, and inhibit impulses
Teacher ratings Teachers rated each student’s
effort in reading and behavior in the classroom
33
Teacher Measures in Years 3, 4, & 5
Classroom Measures PALS Calendars PALS Fidelity Language Arts Observation Classroom Atmosphere Rating Scale
(Wehby) Survey of Enacted Curriculum (SEC):
English and Language Arts
Teacher Characteristics Demographics Berends teacher survey (assesses
school climate, teacher professional development, teacher efficacy, etc.)
34
Procedures
Pretesting (September-October)
PALS Workshops (September-October) Year 1: All teachers attend same workshop Year 2: Separate TD and BU workshops
PALS Implementation (~18 weeks) Teachers implemented 3 times per week for 35-40 min Weekly classroom visits from project staff Three “booster” sessions for TD and BU PALS teachers Two fidelity observations
Language arts observations in PALS and Control classrooms 45-60 min Momentary time sampling of a variety of reading instructional
components Supplementary field notes
Posttesting (March-May)
35
Analysis
Create latent pretest and posttest variables combining 5 reading measures into 1
Create a latent change score Produces an “error-free” change value
Run 2-level HLM analyses Outcome: Latent change score Variables: Treatment condition (TD, BU, Control); Site (TN,
MN, TX); latent pretest score Random effects: Level 2 teacher effects; ICC = .10
Test comparability of groups on variables plausibly related to selection of TD or BU
36
Results
37
PALS vs. Controls across cohorts (2007-08, 2008-09), sites, and TD/BU
38
Effect Sizes for PALS
-.20
.00
.20
.40
.60
.80
1.00
PALS vs. Control
Effe
ct S
ize
(Hed
ges'
g)
Overall
Low
Average
High
Results: Latent Change Score
39
40
Results by Student Rating:Latent Change Score
41
Did we successfully scale up?
Yes: PALS appears to be generally effective.
BU PALS classes appeared to outperform TD PALS and Controls under the following conditions:Core elements of PALS including a 35 minute
PALS sessionSupport provided by PALS staff
42
Potential Mediators:No Differences Between TD & BU
Fidelity year 1 (ALL Teachers TD) Were teachers who later became TD or BU
better PALS implementers in their 1st year? ALL PALS activities compared.
Fidelity year 2 (TD vs. BU year) When teachers chose between TD and BU
PALS, did their PALS fidelity change? Examined core elements of PALS.
Fidelity Across Years
43
44
Classroom-based Potential Mediators: No Differences
Latent Change-Year 1 Indirect method of looking at overall teacher
efficacy AND differential effect of PALS implementation in year 1
Number of PALS Sessions Did one group implement more PALS than the
other?
Measure of Classroom EnvironmentClimate, management, behavior, etc.
Classroom-based Potential Mediators
45
46
Other Potential Mediators
No differences were found for: Scale of attitude about student needsScale of attitude of about school programsScale of attitudes about self-efficacyScale of attitudes about principal leadership