from model demonstration to sustained implementation of evidence-based practices pat gonzalez, ph.d....

From Model Demonstration to Sustained Implementation of

Evidence-Based Practices

Pat Gonzalez, Ph.D. MDCC Project Officer, OSEP

Debbie Shaver, Ph.D. MDCC Co-Principal Investigator, SRI International

Ed Shapiro, Ph.D., Lehigh University

Suzanne Robinson, Ph.D., University of Kansas

Mary Wagner, Ph.D. MDCC Principal Investigator, SRI International

Model Demonstration Coordination Center (MDCC)

Debbie ShaverMDCC Co-Principal

InvestigatorSRI International

• Since 1970’s OSEP has funded model demonstrations projects (MDPs) to better understand how to improve outcomes for children and youth with disabilities.

• Since 2005, OSEP has funded MDCC to document and learn from the model development and implementation process and outcomes of eight cohorts of MDPs.

OSEP's model demonstrations and MDCC

3

MDCC works with cohorts of MDPs to:• Facilitate collaborative partnerships to learn and

share.• Help establish consistent design elements to deepen

evidence base for models.• Synthesize findings across MDPs to identify factors

that lead to high-quality implementation, sustainability, and wider adoption of evidence-based practices

• Identify key issues in translating research to practice to improve outcomes for children and youth

MDCC Roles

4

Cohort 4: Tiered writing approaches for

secondary students

Cohort 5: Tiered interventions for

English language learners

Cohort 6: Assistive technology for

young children

Cohort 7: Reentry of students from

juvenile justice facilities

Cohort 8: Stepping up technology implementation

Active Cohorts of MDP Grantees

5

Cohort 1: Progress monitoring in elementary reading

Cohort 2: Tertiary behavior interventions

Cohort 3: Early childhood language development

Previous Cohorts of MDP Grantees

6

7

Framework for understanding model implementation and outcomes

INTERVENTIONIMPLEMENTATION

INTERVENTIONOUTCOMES

SourceThe Model

Core intervention components―e.g.,▪ Methods to develop child-specific AT

plans for infants, toddlers, and preschoolchildren with disabilities

▪ Methods for developing effective localprograms for re-use of AT

▪ Methods to modify local policies andprocedures to facilitate the use of AT

FeedbackModel evaluationFidelity dataSocial validity data

InfluencesState/localOther external

factors

DestinationParticipating Organizations and Staff

Characteristics of participatingorganizations, programs, and staff

Implementation outcomes

Changes in:▪ Staff knowledge, attitudes, skills, and

actions

▪ Organizational structures, processes, andculture

▪ Community and other peripheralrelationships

Sustained implementation

PurveyorThe MDP Grantee

Core implementation components▪ Partnering with programs and services

in the community

▪ Providing professional development,training and support including coaching

▪ Ongoing assessment of implementa-tion quality

▪ Staff selection and staffing strategy

Systems Outcomes▪ Positive changes to

policies and procedures

▪ Positive changes toresource allocations (e.g.,increased access to AT)

Family Outcomes▪ Increased ability to help

their children develop andlearn

Child Outcomes▪ Increases in positive

social-emotional skills

▪ Increases in acquisitionand use of knowledge andskills

▪ Increases in use ofappropriate behavior tomeet needs

Family/childcharacteristics

Note: Adapted from Implementation Research: A Synthesis of the Literature (Fixsen et al., 2005).

Fixsen, D. L., Naoom, S. F., Blase, K. A. Friedman, R. M., and Wallace, F. (2005). Implementationresearch: A synthesis of the lite

Intervention Components

1. How do core intervention components change over time in response to implementation experiences, and how does this process differ across MDPs and cohorts?

2. How do variations in the core intervention components relate to fidelity of intervention, acceptance of the model, and establishment of conditions for sustaining the intervention?

Implementation Components

3. How do variations in core implementation components relate to fidelity of implementation, acceptance of the model, and establishment of conditions for sustained implementation?

MDCC evaluation questions [1 of 3]

8

Destination Organizations4. How do variations in the destination organizations (e.g.,

population served, organizational functioning, staff and leadership, climate/culture, resources, support for the model) relate to fidelity of implementation, acceptance of the model, and establishment of conditions supportive for sustained implementation?

Context5. How do variations in contextual factors relate to fidelity of

implementation, acceptance of the model, and establishment of conditions for sustained implementation?


9

Outcomes6. How do variations in core intervention and implementation

components, destination organizations, and contexts relate to:– Implementation outcomes (changes in practice/functioning

in the destination organizations, and changes in the skills, attitudes, and beliefs of staff)

– Child/youth outcomes– Systems-level outcomes?


10

MDCC:• Analyzes data provided by MDPs to address

evaluation questions throughout model implementation.

• Reports results of analyses and lessons learned to OSEP at implementation conclusion.

Then OSEP asks,• What happens to the models after the MDPs exit

their sites? • What promotes and hinders model sustainability

there and spread to other sites?

Following up on implementation experiences

11

• OSEP commissioned MDCC to facilitate early-cohort MDPs in returning to their sites to document:– The core intervention components of the

models that were still in place in their original schools and districts• as originally implemented,• adapted, or • abandoned.

– The extent to which the models, in whole or in part, had been implemented outside the original schools and districts.

Follow-up studies

12

• Develop qualitative templates for MDPs to use in reporting what they found.

• Document MDPs’ data sources and collection methods.

• Compile follow-up data collected by MDPs.• Facilitate conversations about the factors MDP

staff believed helped and hindered model sustainability and spread.

• Synthesize the findings and report to OSEP.

MDCC follow-up activities

13

• Follow-up reports from the first two cohorts can be found at:Model Demonstration Center Website's 'Products and Reports' page: http://mdcc.sri.com/prod_serv.html

• Also available:– Descriptions of cohorts and

projects– Cohort summary reports– “Briefs” on implementation-

related topics– Contact information

MDCC information and products

14

http://mdcc.sri.com/prod_serv.html

http://mdcc.sri.com/prod_serv.html

What Stays, What Goes, and Why?

Sustainability of Model Demonstration Project MP3

EDWARD S. SHAPIRO, PH.D.DIRECTOR, CENTER FOR PROMOTING

RESEARCH TO PRACTICEPROFESSOR, SCHOOL PSYCHOLOGY

LEHIGH UNIVERSITY

Agenda

Brief description of Project MP3What remained two years later.What changed (enhanced/diminished)Key factors identified by the site in

sustainability Technical factors (e.g., instructional

practices, financing systems), Human factors (e.g., skill levels,

communication patterns, staffing changes), Contextual factors (e.g., policy shifts,

demographics)

MP3 Objectives

Project MP3 documents the implementation of a school-wide progress monitoring model in five elementary schools across Pennsylvania.

The model provides general and special education teachers (K- 4) with the necessary support and expertise to use progress monitoring data to determine students’ needs for reading intervention and to develop a Response-to-Intervention framework for special education referral.

Objectives

Result in increased reading achievement for students in participating schools

Equip schools to prevent reading failure by “Catching” students falling behind Introducing intensive, differentiated, scientifically-

based interventions that alter students’ trajectories

Document school adoption of new practices

Project MP3

Large district (by PA standards), 11,750 students 13 elementary, 4 middle, 2 high schools

Funded in fall 2005, 9 month planning periodImplementation began fall 2006 in one schoolImplementation in all 3 schools in 2007All project support ended in June 2009Revisited for sustainability in April 2011Continued interaction with district currently

Participating Schools

School A

K-4 Enrollment

318

% Free/Reduced

32%

%Minority

30%

% 3rd Graders Passing

Reading PSSA

48%

K-4 Enrollment

302

% Free/Reduced

50%

%Minority

55%


Reading PSSA

77%

K-4 Enrollment

257

% Free/Reduced

32%

%Minority

45%


Reading PSSA

76%

TIER 1: All students in core program(Everyone taught reading from H-M)

Fall Benchmark

Student benchmarkscore = BENCHMARK

(90% will do fine)

TIER 1 Enrichment30 min 4-5x week

PM 3x/year

Student benchmarkscore = STRATEGIC

(Might be at risk)

TIER 2 Intervention30-45 min 4-5x weekPM every other week

Student benchmarkscore = INTENSIVE(Definitely at risk)

TIER 3 Intervention30-45 min 4-5x week

PM 1x week

TIER 1: All students in core program(Everyone taught reading from H-M)

Winter Benchmark

When we were finished….

Benchmarking & progress monitoring processes

Data based decision making team processes at school (administrative) and grade (teacher) level

Multi-tiered system of supports (MTSS)Tools for managing MTSSTools for analyzing MTSSFidelity of instruction and intervention

assessmentsParent engagement

Enhancements two years later: Technical factors

Fall benchmark for kindergarten push from September to October

Jump start program – interventions began before fall benchmark for students identified from previous spring in need of intensive intervention.

Grade level meetings in one school alternate between all grades meeting together at the same time one month, followed by focused and individual grade levels meeting the alternate month.

Data collection tools enhanced

Enhancements two years later: Technical factors

Evaluation of fidelity continued, mostly by principalAdded period of intervention for intensive studentsExpansion to all 13 elementary schoolsExpansion to 1 middle school (selected as pilot site)Two schools added PBIS modelSwitch from DIBELS to AIMSwebContinuation of parental engagement process

(letters)Desigated point of contact (teacher) for each grade

levelAll schools applied for and approved for using RTI

for SLD determination (state approval process)

Enhancements two years later: Human factors

Replication of model at all schoolsModel embraced as part of school cultureMinimal teacher turnoverSchools examined data outcomes and

focused on enhancing core instruction

Challenges: Contextual factors

Resource reductionsAdministrative staff changes, key leaders

retired Superintendent (twice) Director of C & I- elementary Director of Special Ed Director of Pupil Services Several supervisors Principals at two of three MP3

schools

General comments: Sustainability

Leadership and commitmentCulture shift, embedded perspectivesApproval at state level for SLD

determination useMaintenance of resources in light of budget

cutsPlanned from the start“What will be there when we leave?”“Can the staff continue without our

support?”Fade and scaffold

Two years later

Continuation despite staff changesMaintenance and routine part of school

culturesSupport at all levels of district from central

to localHiring includes knowledge and experience

with RTI

Thanks

Dr. Ed ShapiroEmail: [email protected]

RtI and Writing Instruction in High Schools

Suzanne RobinsonAssociate Professor of Special Education

University of Kansas

• Background for this “story”• Model demonstration project funded by OSEP 2010-2013• RtI focus on 9th grade, expanded to 10th grade• Focus on writing instruction

– Strategic writing program developed at KU-CRL for core instruction; Tier 2 adapted, focused instruction; Tier 3 intensive, personalized instruction

• Multiple baseline design• Year 1 in one urban district (ended relationship). Year 2 in

district with diverse population in different state. Year 3 added another high school. Year 4 focus on building sustained practices.

Secondary school RtI (tiered support) context

31Cohort 6 Model Demonstration Projects

Tier 1

1. Strategic Writing as 50% of the 9th English Language Arts curriculum (supported in the other

disciplines)

Tier 2

1. Small groups (some writing groups, some literature groups)

2. Co-teaching classes (2 teachers = 2+ groups)

Tier 3 (trials still underway on what works)

3. Writing class/lab taught by ELA teachers

4. Writing class/lab taught by Special Education teachers resurrected

5. SPED Resource classes with students meeting IEP goals/Tier 3 needs resurrected

6. Self-contained special education classes and ELL classes

7. Co-teaching special education teacher (during resource period) sometimes yes/sometimes

no

8. Peer tutors (yet to be implemented)

Impact of program on student performance (2011-2012)


Impact of program on student performance (2011-2012) [1 of 7]


Impact of program on student performance [ 5 of 7]


Common Implementation Challenges in Secondary School Tiered Support Models

• Staff capacity

• Fidelity

• Accountability

• Scheduling

• Resources

• Beliefs about roles and instructional responsibility

_____________ Negotiated contract constraints Other school reform initiatives

Sustaining … threats and strategies

1. District leadership, building leadership, and school staff must want the change and must recommit as often as needed as their understanding of what change entails grows and reconsider what systemic adoption entails (scaling up).

2. The students in the school have to need what the reform offers in ways that matter.

3. The district, school, and teachers must be able to assist in implementation and set up structures to practice, maintain, and sustain the reform (model components, roles on leadership teams, PD/coaching, accountability management).

4. Solutions need to be systemic and they need to be as simple as possible.

• Must come to consensus about what is absolutely necessary for accelerating achievement for all students in terms of:– Type of instruction needed– Necessity and possibility of mastery– How to facilitate mastery– Duration of instruction in the context of other demands– How to provide Tier 3 supports in the context of credit/scheduling

demands– Who will provide Tier 3 support and how to facilitate timely

communication between Tier 1 and 3 teachers

• How to engage all teachers (across departments) and support or specialized service personnel (ELL, SpEd, remedial, etc.)


Lessons learned in tiered support systems in academic arenas with secondary schools. . .

• Technical considerations in sustaining effective practices• Contextual considerations in sustaining effective practices• Human considerations in sustaining effective practices

Almost always, stakeholders in school-change projects have different, sometimes competing, as well as compatible interests in the outcomes of any reform effort. The interplay between compatible interests and different and/or personal interests can cause tension; these tensions compete for attention and can cause relational problems among partners in change.

In conclusion


What Promotes and Hinders Model Sustainability?

Mary WagnerMDCC Principal Investigator

SRI International

Model demonstration projects (MDPs) with strong implementation had:

– “Rolled” with the surprises, working with their sites where they found them.

– Focused on changing:• The knowledge, attitudes, skills, and behaviors of key

adults at all levels of the system.• Formal and informal organizational structures and cultures.• Relationships with stakeholders.

– Built capacity for change at all levels.– Sold their sites on the importance of collecting and using

data.– Produced evidence of impacts ASAP.– Actively learned from their experiences and adapted as

they went.

Implement well

46

• Multi-level leadership• Compatibility with ongoing systems; adapting to fit.• Improved practice and outcomes

– They also improve administrator and practitioner efficacy.

– Staff stayed committed over time.• Leaving behind useful, affordable tools (e.g.,

assessment materials, meeting protocols)• Increasing and relying on system-level supports

– District policies and resources drove sustainability. “It’s just the way we do things now.”

– State-level TA systems were critical for spread.

Supports for sustainability and spread

47

• Staff turnover; capacity doesn’t stay built – The antidote: a culture of good practice.

• Competing initiatives. – The antidote: use data to make the case; don’t forget

the power of parents.• Low salience of the problem being solved

– The antidote: use data to make the case; don’t forget the power of parents.

• Budget cuts. – The antidote: creative repurposing of resources;

“strong fingernails.”

Challenges to sustainability and spread

48

• Cohort 3 (early childhood language promotion models) final report coming soon.

• Cohort 3 submits data from its follow-up study to MDCC in August; report will follow.

• MDCC is preparing a “brief” on issues involved in MDPs replicating and disseminating promising models.

• Continued facilitation of cohorts 4 through 8.• Cohort 9 technology grantees starting up in the

fall.• Learn more at http://mdcc.sri.com !!

What’s next?

49

http://mdcc.sri.com/

Questions??

from model demonstration to sustained implementation of evidence-based practices pat gonzalez, ph.d....

Documents

model implementation

fidelity of implementation

sustained implementation

implementation of evidence

implementation componentshow

implementation process

highquality implementation

implementation experiences