Washington State

Using Data to Drive Statewide Improvement

EffortsGreg Roberts, Evaluation Research Services (ERS)

Leslie Pyper, SPDG Director,

Learning Improvement Coordinator, Special Education, OSPI SPDG Director’s Webinar

November 17, 20101

Part 1 (Greg)Evaluation Approach

Application of Evaluation Approach

Early and Preliminary Findings

Part 2 (Leslie)

Thinking about SystemsLeveraging Evaluation EffortsCreating Systemic Change in WA

2

Evaluation Approach• Theory-driven evaluation• Program theory - what must be done to achieve desired

goals, what other important impacts may also be anticipated, and how these goals and impacts would be generated

• Normative theory and causative theory• Treatment refers to the services, materials, and activities

thought to be essential to generating desired changes.

3

Evaluation Approach• Program model as a vehicle for talking about exceedingly

complex social phenomena• Basis for initially capturing important information• Basis for clearly and usefully reporting evaluation activities

and findings • Means of identifying important research questions• Means of identifying program elements and outcomes for

measurement• Mechanism for Answers for identifying important features of

an implementation environment(s)

4

Evaluation Approach• Causative theory (see figure above) represents empirical

and substantive knowledge of the relationships that link a program’s treatments, its implementation processes, and its intended outcomes.

• Impact theory and impact evaluation are perhaps the most well advertised elements of this framework.

5

Washington State Application

• Program theory/model• Evaluation questions• Evaluation design• Measures, indicators• Procedures

– Survey– Site Visit– Extant Data

6

Program Model

7

Evaluation Questions• To what extent are the School-wide Activities in Figure 1

implemented in funded districts?• To what extent do Local Circumstances in Figure 1 inhibit or

facilitate implementation of School-wide Activities?• To what extent are School-wide Activities related to the

Change Mechanisms in Figure 1?• To what extent are School-wide Activities related to

Outcomes in Figure 1?• To what extent are Change Mechanisms related to

Outcomes in Figure 1?

8

Evaluation Questions• To what extent are the School-wide Activities in Figure 1

implemented in funded districts?• What Local Circumstances do districts and schools find

challenging?• What are the 2008-2009 levels of student outcomes in the

funded and matched districts?

9

Evaluation Design

10

Early and Preliminary Findings

• Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded

districts? • Level of Implementation

11

Early and Preliminary Findings

• Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded

districts? • Assessment Knowledge

12

Early and Preliminary Findings

• Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded

districts? • Frequency of Assessment

13

Early and Preliminary Findings

• Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded

districts? • Frequency of

Decision-making

14

Early and Preliminary Findings

• Question 2: What Local Circumstances do districts and schools find challenging?– Teachers and others tend to endorse practices that

are RTI-aligned, but mostly when they are not described in terms of RTI

– Relatively limited understanding on what RTI is and how it works, and a number of site visit participants had questions about the degree to which their “model” aligns with research or best practice

– Implementation theory

15

Early and Preliminary Findings• What are the 2008-2009 levels of student outcomes in the

funded and matched districts?– Percentage of students passing the 2008 WASL in Reading and Math, respectively– Baseline for ongoing comparison of changes in student outcomes

16

State Assessment - Reading

17

State Assessment - Math

18

Ongoing and Next Steps

• Small-scale study in one demo site (Walla Walla)

• Analysis of Year 2 data (2009-2010)• Collection of Year 3 data (2010-2011)• Refocusing program model and evaluation

questions

19

1. Are you working with an evaluator in developing and/or monitoring implementation of RTI in your state?

Feedback Pod

Yes/No

20

“Cheshire,” Alice began rather timidly, “Would you tell me please, which way I ought to go from here?”

“That depends a good deal on where you want to get to,” said the Cat.

“I don’t much care-” said Alice.

“Then it doesn’t matter which way you go,” said the Cat.

Alice’s Adventures in Wonderland by Lewis Carrol

Michigan Dept Ed. ppt 2-16-0921

295 School Districts

Educational Service DistrictsESDs

OSPI (SEA)

Local Control

Legislated as separate entities

Special Ed

SPDG

Demo Districts

State Leadership

Team

All special ed folks

22

Feedback Pod

A. directly by the SEA.

B. by other entities with direct guidance/training from the SEA to those entities. C. by other entities without coordinated influence from the SEA.

2. In my state, regional technical assistance and training is provided …

23

295 districts divided among 9 regional ESDs Leslie = ESD liaison for special ed CSA

Spokane*Spokane**Seattle*Seattle

Improvement planning - by region, using data profiles Root causes (linking activities to root causes)Quarterly updatesEvaluation of activities

Olympia*Olympia*

* May choose to focus some efforts on RTI and/or PBIS…

24

Federal

State

Local Education Agencies

TA System

Teachers/Staff

Effective Practices

System Alignment

Fixsen, 2008 25

Federal

State (OSPI)

LEAs

TA System (ESDs)

Teachers/Staff

Effective Practices

WA System Alignment

Adapted from Fixsen, 2008

IHEsTeacher Prep/

Tech Asst

Special Ed State Needs

Projects

26

Other TA efforts --- State Needs Projects

Needed a “system” to manage these projects and needed a system to provide TA to them.

to identify a “state need” to ensure consistency/alignment to provide training/support to the project staff

Projects are supposed to address a “state need” --

Realized that there wasn’t a process --

27

Contracted with UW-Tacoma to evaluate the State Needs Projects

Are they serving statewide? Are they using resources/materials aligned with State policies and current research? Are they providing on-going support? Are they collecting reliable data? Are they making adjustments based on evaluation of their efforts? Are they having an impact on practice? Are they having an impact on indicator performance? 28

Engage SEA departments/ESDs in strategic planning

Multiple depts + ESDs = Implementation Team

NCRTI – NWRCC – WRRC Need to develop common language Need to align messaging/PD/use of coaches… Need to develop review/vetting process for RTI Develop strategic plan for support

Build a sustainable State structure for RTI

Build capacity of SEA, ESDs to support districts

29

295 School Districts

Coordinated Services Agreement (CSA) w/ ESDs

(regional TA providers)

SEA/ESD Implementation Team

Special EdSPDG

30

OSPI Implementation Team- really should be called the Transformation Team

Teaching & Learning (Reading, Math)Classroom Assessment District & School ImprovementSpecial EducationMigrant/bilingual/ELLTitle I/LAPHighly CapableEquity & Civil RightsSecondary Ed & Dropout PreventionStudent Achievement (Achievement Gap)Ed TechnologyFinancial Resources & Governmental Relations (Data Governance)

Educational Service Districts (ESDs)

31

This would begin to align many efforts at the SEA – which would also impact the ESDs…

Dropout preventionDistrict & School Improvement & Accountability

K12 Reading Model

New Math framework

Comprehensive Assessment System

SPDG32

Implementation Team developed an assessment tool: (OSPI Efforts Inventory)

Assessing activities across SEA (pilot…11/12/10…two departments)

In order to develop a strategic plan for the state, we needed to understand WHAT everyone was doing, and HOW IT CONNECTED to the RTI structure.

33

Focus Area

Essential Components (RTI Framework) OSPI Efforts

Multi-Level System

ScreeningProgress

Monitoring

Data-Based

Decision Making Documents

Profes-sional

Develop-ment

Fund-ing

SourceData

Collected

Core Instruction/

Support

Interven-

tionsReading

Writing

Mathematics

Language

Behavior

Alcohol, Tobacco, or Other Drug UseMental/

Emotional Health

OSPI Efforts InventoryDepartment Contact Person(s) Date

34

Improved instructionfor students

35

3. We have developed common language across the SEA and our resources are aligned and collectively support the RTI framework.

Feedback Pod

True/False

36

engaged IHEs in IRIS Center training recruited IHE reps on State LT (gen & sped) engaged multiple IHEs in regional training meeting with several IHEs re: RTI efforts

Build capacity of IHEs

Build a sustainable State structure for RTI

UW-T 325T grant – participating on Advisory Board Project RTI: Restructuring, Transforming, Implementing a Dual-track RTI Teacher Preparation Program

37

4. We have developed common language across the state (IHEs, SEA, regional TA providers) and our resources are aligned and collectively support the RTI framework.

Feedback Pod True/False

38

Shewhart (1924); Deming & Juran (1948); Six-Sigma (1990)

• Plan – Decide what to do• Do – Do it (be sure)• Study – Look at the results• Act – Make adjustments • Cycle – Do over and over again until the

intended benefits are realized

PDSA Cycles: Trial & Learning

Dean L. Fixsen & Karen A. Blasé, 2009 39

“Knew”:

Lack of understanding of pilot site requirements Lack of consistent PD opportunities Lack of a state structure for PDLack of involvement of SEA beyond sped

o lack of understanding about connectionso lack of alignment o no vetting of materials

Lack of common understanding of RTI framework Lack of data system to manage RTI efforts Lack of preparation for leadership of RTI efforts Likely a lack of fidelity

40

Evaluation results –

1-2 PD opportunities (avg) More than half gave incorrect response or no response regarding defining screening & progress monitoring Collecting data but teachers struggling to use data Basic knowledge of RTI lacking Lack of buy-in Difficulty moving students through tiers Difficulty in using staff to provide interventions Is anyone monitoring fidelity?

41

So, need to develop a system to support demo sites and all others who want to implement RTI:

Develop a state structure for support of RTI develop common language develop a strategic (state-level) plan align efforts (messaging, resources, training,…)

Purchase/develop a data system for demo sitesProvide ‘hands on’ support to demo sites

42

Screening /progress monitoring Selecting EBPs Adapting on-site technical assistance Developing monthly TA calls (in addition to quarterly mtgs) Developing a Guide for Selection of EBPs “ “ Guide for Selection of Trainers/materials Regional training – IHEs, ESD, multiple SEA staff Using integrity rubric

Training for RTI demo sites

43

Build a data system

WA received $17.3 million for longitudinal data system -- requested meeting with data folks (SEA) -- have been meeting with Student Information Director and Data Governance Coordinator (SEA) -- also invited DSIA, Classroom Assessment

Spectrum K12Demo for Implementation teamReviewing training plans Meeting to draw up a “proof of concept” system

44

Supports for Implementation

RTI Coordinator mtgs (quarterly)Site visits (technical assistance)Will begin monthly contacts to provide additional supports

Developing a series of trainings for all(will consist of face-to-face, webinar, and recorded sessions)

45

How did you use your data this past year?

Student-level dataBuilding-level dataDistrict-level data

Protocols in place?

How is your data system driving implementation in your district???

Questions for RTI Coordinators

46

5. My SEA used data this past year to:

Feedback Pod

A) improve content, delivery, format, or target audience of PDB) revise guidance on RTI frameworkC) influence policy-makers, TA providers, SEA divisionsD) improve data collectionE)determine funding efforts F) revise evaluation effortsG)obtain technical assistance/support H) other: _____________________

Check all that apply

47

Intensive Technical Assistance w/ the National Center on RTI (MOU)

•Common language •Alignment across efforts •Build capacity at the SEA, IHEs, ESDs, & districts to support implementation of evidence-based practices (EBPs)•Data system for demonstration districts (expandable)•Information dissemination system to expand RTI efforts across universities, districts and professional organizations

48

Effective State Support Structures for RTI Include…

• Leadership, consensus, and aligned vision • Consistent messaging across departments• Capacity-building supports for systems that

support LEAs/schools (IHEs, ESDs), and for LEAs and schools directly

• Collection and use of data to inform decisions at all levels (student, classroom, school, LEA, SEA)

• Regulatory language, in general and/or special education, as appropriate, to support RTI

From Innovations conference, 201049

Invited to participate in the SISEP Community of Practice

Putting together a team of folks from the SEA & ESDs --- - from Implementation team - from CSA group

Our CSA group is now looking at “scaling up evidence-based practices”

50

Presenting to Cabinet 11/22/10 Putting the puzzle together for them:

info on efforts of multiple depts (including pilot data - OSPI Efforts Inventory) connecting RTI to the Superintendent’s Five Strategic Priorities # districts implementing across the state CSA work

>50% report implementing RTI

51

152 (51.5%) districts report implementing RTI

142 Reading – Elem (48.1%) 79 Reading – Middle (26.8%) 42 Reading – High School (14.2%) 98 Math – Elem (33.2%) 67 Math – Middle (22.7%) 37 Math – High School (12.5%) 69 Behavior – Elem (23.4%) 44 Behavior – Middle (14.9%) 26 Behavior – High School (8.8%)

2009-10 school year

52

At first, it felt like…

53

Each step we take has an impact on yet another area ----

Before long (that’s judged in “State” time ), we’ll have all of our state-level activities aligned…

54