Team Read Team Read Improving Literacy in the Seattle Improving Literacy in the Seattle

School DistrictSchool District

PA 590C – Program EvaluationPA 590C – Program Evaluation

Fall 2007Fall 2007

Anne DebuyserieAnne Debuyserie

“...a Seattle School district tutoring program dedicated to increasing the reading skills of elementary students through year-long coaching by high school students ... The program's goals for participating high-school tutors are to develop work experience, a sense of responsibility and accomplishment, and to learn the rewards of community service.”

(Team Read Mission Statement, 1998)

What is Team Read?What is Team Read?

Private-public partnership to Private-public partnership to improve literacy in Seattle Public improve literacy in Seattle Public SchoolsSchools

Cross-age tutoring program using Cross-age tutoring program using high school student as tutorshigh school student as tutors

Started in March 1998 with a three-Started in March 1998 with a three-phase implementationphase implementation

June 1999: 10 schools, 335 students, June 1999: 10 schools, 335 students, 300 coaches300 coaches

June 2000: addition of 7 schoolsJune 2000: addition of 7 schools

Team Read- Org ChartTeam Read- Org Chart

Team Read Stakeholders Team Read Stakeholders

Craig McCaw (and wife Susan): BusinessmanCraig McCaw (and wife Susan): Businessman John Stanford: Seattle Schools SuperintendentJohn Stanford: Seattle Schools Superintendent Joan Dore: Seattle Schools Reading SpecialistJoan Dore: Seattle Schools Reading Specialist Tricia McKay: Team Read Program MgrTricia McKay: Team Read Program Mgr Team Read Advising Board: Seattle School Team Read Advising Board: Seattle School

District Staff, Alliance for Education pgm District Staff, Alliance for Education pgm representatives, Mc Caws, community representatives, Mc Caws, community representatives.representatives.

Additional investors joined by year 2.Additional investors joined by year 2.

Who is Team Read?Who is Team Read? CoachesCoaches

– Strict selection guidelinesStrict selection guidelines– High expectations from Team Read High expectations from Team Read

leadersleaders– CompensationCompensation

ReadersReaders– School selection comes firstSchool selection comes first– Student selection based on reading levelStudent selection based on reading level– Eligibility based on needEligibility based on need– First-come, first-serve basisFirst-come, first-serve basis– Low socio-economic backgroundLow socio-economic background

Who is Team Read?Who is Team Read?

Site CoordinatorsSite Coordinators– Part-time, teachers or school staffPart-time, teachers or school staff– Meet informally with Tricia McKay for Meet informally with Tricia McKay for

feedbackfeedback VolunteersVolunteers

– Needed due to amount of workNeeded due to amount of work– Partnership with VISTA/Americorps Partnership with VISTA/Americorps

and U. of Washington Pipelineand U. of Washington Pipeline– Assist site coordinatorsAssist site coordinators

Team Read SessionTeam Read Session

Two 1 hr session per week during school Two 1 hr session per week during school yearyear

Standardized session format and structureStandardized session format and structure– 3.00 PM: coaches arrive3.00 PM: coaches arrive– 3.15 PM: snack with reader3.15 PM: snack with reader– 3.30 pm: reading session begins3.30 pm: reading session begins– 4.15 PM: reading session ends/ coaches fill 4.15 PM: reading session ends/ coaches fill

out paperworkout paperwork– 4.30 PM: reading coaches leave4.30 PM: reading coaches leave

Opportunity for site coordinators to be Opportunity for site coordinators to be creativecreative

Team Read Evaluation ResultsTeam Read Evaluation Results

11stst year evaluation: not as positive as year evaluation: not as positive as expectedexpected

22ndnd year: promising but Team Read’s year: promising but Team Read’s impact not as promising as hopedimpact not as promising as hoped

Tricia McKay disappointed by results Tricia McKay disappointed by results and wonders if findings provide her and wonders if findings provide her with the information needed for with the information needed for improvementsimprovements

Questions about the accuracy of Questions about the accuracy of results.results.

Team Read EvaluationsTeam Read Evaluations Evaluation conducted by Margo Jones, Evaluation conducted by Margo Jones,

statistician by trade.statistician by trade. 17 schools in the program by June 2000 17 schools in the program by June 2000

but only 10 used in the evaluation.but only 10 used in the evaluation. 3 main objectives to show Team Read’s 3 main objectives to show Team Read’s

Impact:Impact:– Goal#1: Do the reading skills of the student

readers improve significantly during their participation in the Team Read program?

– Goal#2: How does the program affect the reading coaches?

– Goal#3 What is working well, and what can be improved?

Evaluation Methods Goal #1Evaluation Methods Goal #1 Goal #1: Reading skills. Two-fold Goal #1: Reading skills. Two-fold

approachapproach

GradeGrade Pre-TestPre-Test Post-TestPost-Test Post-Test MetricPost-Test Metric

22 DRA Fall 1999DRA Fall 1999 DRA Spring DRA Spring 20002000 Test LevelTest Level

33DRA Spring DRA Spring

1999*1999*ITBS Spring ITBS Spring

2000*2000* NCENCE

44ITBS Spring ITBS Spring

19991999WASL Spring WASL Spring

20002000 Scale ScoreScale Score

55WASL Spring WASL Spring

19991999ITBS Spring ITBS Spring

20002000 NCENCE

Apples and orangesApples and oranges

GradGradee

Pre-TestPre-Test Post-TestPost-Test Post-Test Post-Test MetricMetric

22

33 NCENCE

44

55

Evaluation Methods Goal #1, Evaluation Methods Goal #1, Cont’dCont’d

22ndnd method: determine proportion of method: determine proportion of Team Read participants who moved Team Read participants who moved from below grade level test score to from below grade level test score to at-or above level at-or above level compared to all compared to all students from entire districtstudents from entire district

Note: one of the selection criteria to Note: one of the selection criteria to be in Team Read is to be within be in Team Read is to be within bottom bottom 25% of district reading 25% of district reading test scoretest score

Evaluation Methods Goal #1Evaluation Methods Goal #1

Both methods for goal 1 are flawed.Both methods for goal 1 are flawed. Note from Jones in report confirming problems Note from Jones in report confirming problems

with different tests.with different tests. “The question then becomes ‘Is it reasonable to

assume that the pre- and post-tests measure the same skills?’ The evaluator adopted a correlation criterion for answering this question. If the correlation between pre- and post-tests was near or above .8 (see footnote 5) (as was the case for the 2nd grade, where pre- and post-tests were essentially the same), the pre- to post-test change score was interpreted as a gain score. If the correlation coefficient was much less than .8, no analysis was performed.”

Evaluation Methods, Goals #1- Evaluation Methods, Goals #1- AssessmentAssessment

Visible problems with tests:Visible problems with tests:– Pre and post-tests do not measure the Pre and post-tests do not measure the

same skills except in 2same skills except in 2ndnd grade grade– All Team Read participants are within All Team Read participants are within

the bottom quartile of the reading the bottom quartile of the reading test scores and can’t be compared test scores and can’t be compared with students district-wide.with students district-wide.

– Analysis could not be carried out for Analysis could not be carried out for some grades (3some grades (3rdrd) due to ) due to incompatibility of reading testsincompatibility of reading tests

Evaluation Method, Goal #2Evaluation Method, Goal #2

Method: questionnaire given to coaches Method: questionnaire given to coaches during their last week of coaching. during their last week of coaching.

Goal: to assess the impact of Team Goal: to assess the impact of Team Read on coaches.Read on coaches.

Identify 3 main areas of job satisfaction:Identify 3 main areas of job satisfaction:– How positive is their experience with TR?How positive is their experience with TR?– Extent to which the coaches feel that TR Extent to which the coaches feel that TR

helps students readershelps students readers– How supportive is TR’s site coordinators How supportive is TR’s site coordinators

and staff?and staff?

Evaluation Method, Goal #2 - Evaluation Method, Goal #2 - AssessmentAssessment

Jones results showed that coaches Jones results showed that coaches were satisfied with their job and felt a were satisfied with their job and felt a sense of pride and accomplishment.sense of pride and accomplishment.

Test performed by Jones was Test performed by Jones was successful in meeting its goals, unlike successful in meeting its goals, unlike in Goal #1.in Goal #1.

Some questions provided answers as Some questions provided answers as to what to improve in the program.to what to improve in the program.

Test is useful to the program managerTest is useful to the program manager

Evaluation Method, Goal #3Evaluation Method, Goal #3

Method: interviews with major Method: interviews with major stakeholders, one site visit, review stakeholders, one site visit, review of last 5 years’ research literature of last 5 years’ research literature on cross-age tutoring and last 3 on cross-age tutoring and last 3 years research on best practices.years research on best practices.

Goal: assess the overall progress Goal: assess the overall progress of the program.of the program.

After the EvaluationAfter the Evaluation

Margo Jones makes her Margo Jones makes her recommendations based on her recommendations based on her findings to improve the efficacy of findings to improve the efficacy of Team Read.Team Read.

Tricia McKay reads the evaluation Tricia McKay reads the evaluation but fail to see what the results are but fail to see what the results are telling her about the programtelling her about the program

Evaluation Methods - SummaryEvaluation Methods - Summary

GoalsGoals MethodologyReading improvementamong elementarystudents

Analyzed standardized test scores for 10 of 17 Team Read schools. Rate of increase on pre- and post- tests. Percent of students moving from below reading level to at/above grade level. Collected data by grade level and by school. Compared Team Read results to district scores.

Coach development Surveyed coaches by questionnaire

Program implementation Interviewed stakeholdersObserved Team Read sessionReviewed other studies

Evaluation ProblemsEvaluation Problems

Academic v. Practitioner approach.Academic v. Practitioner approach. Result presentation.Result presentation. Research design flaws.Research design flaws.

– Not all the data was analyzedNot all the data was analyzed– Comparison with dissimilar itemsComparison with dissimilar items– Statistical analysis stretchedStatistical analysis stretched

Evaluation only covers 3 goals.Evaluation only covers 3 goals. Lack of communication between Lack of communication between

the evaluator and the program the evaluator and the program manager.manager.

What Could Have Been Done What Could Have Been Done Better?Better?

Impact: 2005-2006 Reading Results

From Team Read’s 2006 Annual Report.http://www.teamread.com/downloads/team_read_106903_report_proof.pdf

Team Read Update - 2006Team Read Update - 2006 Program Evaluation Results for 2005-2006

School Year   – 70% of 2nd graders and 52% of 3rd graders were

reading at/above or approaching grade level – 54% of 2nd graders and 35% of 3rd graders

gained greater than 1.5 grade levels in reading – 98% of the parents of 2nd & 3rd graders and 92%

of their teachers reported increased reading skills as a result of participation in Team Read

– 76% of 2nd & 3rd graders said that reading was more fun since joining Team Read