what was learned from a second year of implementation
DESCRIPTION
What Was Learned from a Second Year of Implementation. William Corrin, Senior Research Associate MDRC. IES Research Conference Washington, DC June 8, 2009. Presentation Overview. Project Background Do teachers need more than a year to master new practices? - PowerPoint PPT PresentationTRANSCRIPT
What Was Learned from aSecond Year of Implementation
IES Research Conference
Washington, DCJune 8, 2009
William Corrin, Senior Research AssociateMDRC
Presentation Overview
Project Background Do teachers need more than a year to master new
practices?
- Year 2 Implementation Findings (and Year 1 Comparison)
- Impact findings
- Exploratory findings
o Two-Year Teachers and Replacement Teachers
o Two-Year Teachers: Year 1 vs. Year 2
Summary
Project Background
The ERO Study
The ERO study is an impact evaluation of two supplemental literacy programs targeted at 9th grade students with limited reading skills.
Key research question:
- What are the impacts of the two interventions, together and separately, on ninth-grade students’ reading achievement and reading behaviors?
The Interventions
Two literacy programs targeted at 9th grade students with limited literacy skills (2-5 years below grade level)- Reading Apprenticeship Academic Literacy (RAAL)
- Xtreme Reading (XR)
The programs are supplemental reading classes- Designed as full-year courses that replace a 9th grade
elective class (rather than a core academic class)
The two programs share common goals and instructional principles
Delivery of the Interventions Each high school implemented one of the two programs
for ninth graders for two school years
- 2005-06 (Cohort 1) and 2006-07 (Cohort 2)
One ERO teacher per high school
- Experienced full-time ELA or social studies teacher was trained to teach the ERO reading course
- Responsible for 4 sections of 10-15 students each
Training and technical assistance were provided to teachers by the developers
- Summer training institutes, school-year off-site booster training sessions and on-site coaching
Implementation Data Collection
To answer questions about how well the programs were put into place
Allows us to answer questions about how implementation compared across program years: Do teachers do better with more experience teaching these programs? That is, is implementation stronger in the second year compared to the first?
Allows us to investigate associations between implementation and impacts
Measuring Implementation Fidelity
Two site visits in Year 2 (one per semester; only one site visit in Year 1)
Implementation fidelity was assessed on two dimensions:- Classroom learning environment
- Comprehension instruction
Classroom observation ratings were obtained for 6 constructs common to both programs, and 7 program-specific constructs- Ratings are based on a 3-point scale
Measuring Implementation Fidelity (cont.)
Composite ratings for each of the two dimensions were calculated, and then used to classify sites as:
- “well aligned” (average rating > 2)
- “moderately aligned” (average rating = 1.5 – 1.9)
- “poorly aligned” (average rating < 1.5)
Implementation Findings
Implementation Findings
ERO teachers in Year 2- Of the 34 ERO teachers in Year 2:
o 25 had taught the entire first year of the study (13 RAAL, 12 XR)
o 2 had taught part of the first year (both XR)
o 7 were new to the programs (4 RAAL, 3 XR)
All Year 2 teachers taught the entire year
Implementation Findings (cont.)
Implementation fidelity in Year 2- In terms of classroom learning environment, 1 school
was rated as poorly aligned with program models (vs. 4 schools in Year 1)
- In terms of reading comprehension instruction, 1 school was rated as poorly aligned with program models (vs. 9 schools in Year 1)
- 23 schools were rated as well aligned on both dimensions (vs. 16 schools in Year 1)
- Similar pattern of implementation findings at RAAL and XR schools
Implementation Findings (cont.)
Implementation fidelity rated higher in Year 2- In terms of classroom learning environment, the
average rating at the spring site visit was 2.5 in Year 2 compared to 2.2 in Year 1
- In terms of reading comprehension instruction, the average rating at the spring site visit was 2.3 in Year 2 compared to 1.9
Learning EnvironmentBy Site Visit
Comprehension Instructionby Site Visit
Impact Findings
Random Assignment and Analysis
Random assignment:- School-Level
o 34 schools randomly assigned within 10 districts to RAAL or Xtreme Reading (17 RAAL sites and 17 Xtreme Reading sites)
- Student-Level o In each high school, eligible students were randomly
assigned to either: Enroll in an ERO class (ERO group) or Take one of the regular elective classes (non-ERO
group)
Analysis: Impacts are estimated by comparing the outcomes of students in the ERO and non-ERO group
Impact Findings: Reading Achievement
Reading achievement was measured using the GRADE, a standardized reading assessment.
Overall positive impact on reading comprehension- 0.09 SD for Cohort 1
- 0.08 SD for Cohort 2
No impact on vocabulary- Same result for both cohorts
Impact Findings: Reading Behaviors
Cohort 1- Estimated impacts on three reading behaviors were
not statistically significant (frequency of school-related reading, freq. of non-school-related reading, use of reading strategies)
Cohort 2- Estimated impact on students’ use of the reading
strategies taught by the programs is 0.09 SD and statistically significant
Exploratory Analyses
Implementation Fidelity by Teacher Experience and Year
Implementation ratings by year and by teacher experience:
Implementation Dimension Year
Schools with Two-Year Teachers (N=25)
Schools with Replacement
Teachers (N=9)
Learning Environment
1 2.4 1.7
2 2.5 2.4
Reading Comprehension
Instruction
1 2.0 1.8
2 2.3 2.3
Implementation Fidelity by Teacher Experience and Year
Overall implementation fidelity:
- Of the 25 sites with two-year teachers, 19 had higher overall implementation ratings in the second year compared to the first year
- Of the 9 sites with replacement teachers, 8 sites had higher overall implementation ratings with the replacement teacher compared to the original teacher
Composite Fidelity Scores by Site Visit
Variation in Impacts
Impacts on Student Outcomes by Teacher Experience Teaching ERO: Year 2
Student Outcome
Schools w/ Two-Year Teachers
(N=25)
Schools w/ Replacement
Teachers (N=9)
Reading Comprehension
0.09 SD*
0.06 SD
Use of Reading Strategies
0.10 SD*
0.07 SD
Variation in Impacts (cont.)
Impacts on Student Outcomes by Implementation Year: Schools w/ Two-Year Teachers (N=25) Student Outcome
Year 1
Year 2
Reading Comprehension
0.11 SD*
0.09 SD*
Use of Reading Strategies (diff. betw. yrs. is stat. sig. )
-0.05 SD
0.10 SD*
Summary Supplemental literacy courses for 9th grade students can have a
positive impact on students’ reading comprehension.
These programs can be implemented with reasonable fidelity in a diverse array of high schools, and implementation fidelity can improve over time.
On average, implementation fidelity was stronger in the second year.
There is suggestive, but not convincing, evidence that teachers gained greater mastery of the programs having a second year to teach it.
However, impacts on students reading comprehension remained the same. (Impacts on students’ use of reading strategies were greater.)
Further questions? William Corrin
- (212) 340-8840 Marie-Andree Somers
- (212) 340-8825 Paul Strasberg
- (202) 219-3400