Using Data to Problem SolveSusan Barrettwww.pbis.orgwww.pbismaryland.org

Thanks to…

Center on PBIS

Steve Goodman- Michigan

Tim Lewis

Rob Horner

George Sugai

Catherine Bradshaw

Don Kincaid

Adopt systems perspective at a Building Level

Systems Perspective

Organization do not “behave” …individuals behave

“Organization is group of individuals who behave together to achieve a common goal”

“Systems are needed to support collective use of best practices by individuals in an organization” (Horner, 2001)

Schools as Systems

Goal to create communities that for all its members have common

Vision

Language, &

Experience

Biglan, 1995; Horner, 2002

What a Leadership Team does…

Communicates common vision for schoolwide supports

Works collaboratively to establish building capacity to support all students

Commits resources to establish procedures for support

Develops methods for evaluating progress towards measureable outcomes

Action planning based on data

Should get easier for your school over time

HandbookDescribes core features

Expectations and teaching matrix (rules for settings)

Teaching plans and teaching schedule

Acknowledgement system

Continuum of consequences for problem behavior

Building Leadership TeamRegular meeting schedule and process

Regular schedule for annual planning and training

Annual Calendar of Activities

On-going support for staff

Use the PBIS Maryland website as reference

www.pbismaryland.org

Your Leadership TeamYour Leadership Team

Does your team understand the leadership Does your team understand the leadership function in managing and coordinating function in managing and coordinating implementation?implementation?

On a scale of 1 (low) to 5 (high), how well is On a scale of 1 (low) to 5 (high), how well is your team doing with this responsibility? your team doing with this responsibility?

Team TimeTeam Time

Standards and Protocols

Available on www.pbismaryland.org

What is a “PBIS trained” team?

What happens if a school does not meet criteria?

What is required to be a PBIS implementing school?

How does a school become inactive?

What is required for a school to be eligible for PBIS Maryland Recognition?

Purpose of Systems MeasuresBenchmarks of Quality Checklist

Evaluates status of Tier I Positive Behavior Supports

Completed Annually, Required for Recognition

Submitted to Jerry electronically on April 10

Submitted online at www.pbssurveys.org

Implementation Phase Inventory

Required for Recognition

Due November 10 and April 10 to Jerry

Self-Assessment Survey- Not required but…

Evaluates status of Schoolwide, Nonclassroom, Classroom and Individual Student Supports

Submitted online at www.pbssurveys.org

What is the BOQ?• Lists components of PBIS programs

that address the critical elements of PBIS implementation

• Completed by school teams on a yearly basis to assess how they score on a 100 point scale with regard to developing and implementing school-wide PBIS

• Useful in developing action plans for following year

• One of the measures used by MSDE-SPHS-JHU to determine schools achieving Exemplar Status

Three Components of Benchmarks of Quality

Team Member Rating FormCompleted by team members independently

Returned to coach

Scoring FormCompleted by coach using Scoring Guide

Used for reporting back to team

Scoring GuideDescribes procedure for completing BOQ

Includes a rubric for scoring each item

BOQ Will Provide:Summary of team members’ perceptions of PBIS implementation (scored: ++ in place, + needs improvement, and -

not in place)

• Objective assessment of school’s implementation based on criteria described in a rubric (100 point scale)

• Comparison between the above factors which will encourage discussion of strengths and weaknesses and provides ideas for action planning

What is the IPI?• Implementation Phases Inventory

• Two times/year – Due November 10, April 10

• Coach completes with Team• Four Phases

– Preparation– Initiation– Implementation– Maintenance

What is the SET?

School-wide Evaluation Tool

– One of several methods to evaluate Tier 1 – Required if school is seeking Recognition

Status– External certified SET Assessor will conduct

site visit – Should you be a SET Assessor?

Measures the level of implementation of SWPBIS(not intended to measure everything!)

• The Critical Features– Expectations Defined– Expectations Taught– System for Encouraging Expected

Behaviors– System for Discouraging Problem

Behaviors– Monitoring and Decision Making– Management– District Level Support

What does it measure?

Why use it?• The results help PBIS teams:

– Assess the features of PBIS in place– Determine annual goals for school-wide

effective behavior support – evaluate on-going efforts toward

school-wide behavior support – design and revise procedures as

needed – compare efforts toward school-wide

effective behavior support from year to year

What does it look like?• Permanent product review

– Office Discipline Referral (ODR) form– Current Action Plan– Discipline Handbook/Plan– School Improvement Plan– Lesson Plans & Schedule

• 2 to 3 hour school visit:– Observations

• Classroom and Non-classroom settings

– Interviews• Administrator, Staff, PBIS Team Members, and

Students

What is the Self-Assessment Survey?

• Self-assessment survey to assess the extent to which Positive Behavior Support practices and systems are in place within a school– School-wide (15 items)– Non-classroom (Specific Setting) (9 items)– Classroom (11 items)– Individual Student (8 items)

Who Completes the Self-Assessment Survey?

• Initially, the entire staff in a school completes the Survey.

• In subsequent years and as an on-going assessment and planning tool, the Survey can be completed in several ways:

– All staff at a staff meeting.

– Individuals from a representative group.

– Team member-led focus group.

Using the Self-Assessment Information for Decision Making

• Is a system in place?– “in place” > 66%

• Is there a need to focus on a system?– Current status of “in place” is < 66% and– Priority for improvement is “High” for > 50%

• Which system should receive focus first?– Always establish schoolwide as first priority

• Which features of the system need attention?

• Combine survey outcomes with information on office referrals, attendance, suspensions, vandalism, perceptions of staff/faculty

Individual Summary Charts

• Charts are provided for each system (school-wide, nonclassroom, classroom, and individual)

• Current status Charts

– Percentage of respondents who answered "In Place", "Partially In Place", and "Not In Place"

• Improvement Priority Charts

– Percentage of respondents who answered "High", "Medium", and "Low”

Example of PBS Self Assessment Survey Individual Summaries Chart

Analysis of Schoolwide System Chart

Shows a chart with bars for components of the schoolwide system

– Expectations defined (question 1)– Expectations taught (question 2)– Reward system (question 3)– Violations system (question 4-8) – Monitoring (question 10-12)– Management (question 9, 14-16)– District support (question 17-18)

Analysis of Schoolwide System Chart

Example of

PBS Self Assessment Survey

Individual Item Score

Schoolwide Component

White = In Place

Yellow = Partial In Place

Red = Not In Place

Why conduct Self-Assessment Survey in addition to Checklists?

• Checklists are conducted by team, all/most staff complete survey

• Look for areas of convergence across tools– Increases confidence of data

• Look for areas of divergence across tools– Decrease confidence of data?– Possible reasons for disparity…

• Lack of understanding of questions• Staff not fully aware of work of Building

Leadership Team• Support component not fully “In Place”

Differences between the BOQ action plan form

and the Self-Assessment SurveyBenchmarks of Quality

EBS Self-Assessment Survey

Purpose?

Evaluate on-going progress towards schoolwide PBS

Evaluate extent that all systems (schoolwide, nonclassroom, classroom, individual) are in place

When administered?

Monthly- progress monitor Tier 1

Annually

Who completes?School Leadership team, completed as a team

All school staff (or representative sample) completed individually

Time involved?10-15 minutes 30-45 minutes

To Do List• Review results from…

• BOQ Checklist

• Determine if you would like your school staff to complete Self-Assessment Survey

• Review School’s Action plan- What is the link to overall School Improvement Plan?

• Based on this information complete action plan for you!

• What celebrations can you share with your school community before this year is over?

• What is your plan to strengthen your schools’ behavior support for the next school Year?

Please take a moment to complete the appropriate section of the Please take a moment to complete the appropriate section of the Follow-Up Activity Worksheet to document the work yet to be doneFollow-Up Activity Worksheet to document the work yet to be donePlease take a moment to complete the appropriate section of the Please take a moment to complete the appropriate section of the Follow-Up Activity Worksheet to document the work yet to be doneFollow-Up Activity Worksheet to document the work yet to be done

Student Measures

• How do we know Implementation of Tier 1 PBIS is making an impact?

• What data should our team be reviewing?

• How do we build that into the agenda so we it is standard practice?

• Do we have a core group on our team that reviews that data prior to the monthly team meeting?

Data-Based Decision Making1. Determine what questions you want to answer

2. Determine what data will help to answer questions

3. Determine the simplest way to get data

4. Put system in place to collect data

5. Analyze data to answer questions

Focus on both Academic and Social Outcomes

1. Determine what questions you want to answer

Examples

• Can we predict problems/success?– When/where/who?

• Possible “function” of problem behavior?

• Who needs targeted or intensive academic supports?

• What environmental changes/supports are needed?

2. Determine what data will help to answer questions

• Existing data set(s)

• Current data collection

• Additional / new data

• Confidence in accuracy?

• Complete picture?

3. Determine the simplest way to get data

• Agreement on definitions• Standard forms / process• Frequency of collection• Target “Multi-purpose” data/use

Train ALL staff on use & provide on-going TA

4. Put system in place to collect data

• Build on existing systems

• Add components over time

• Central entry point– Electronic

5. Analyze data to answer questions

• Trends

• Instruction & supports in place/not in-place

• Pre/post “big outcomes”

• Comparisons (norm / local)– Relative growth– Absolute growth

By Location1998-99/#ODR by Location

301

76

2

7955

4

36 36 38

1 8 10

50

100

150

200

250

300

350

Location

#O

DR

By Behavior1998-99 #ODR/Behavior

0

20

40

60

80

100

120

140

160

180

200

Fighting Class Disruption Non-compliance InappropriateLanguage

Cut Class/Out ofArea

Property Smoking/Drugs

Behavior

By Student1998-99 By Student

0123456789

10111213141516171819202122232425

1 9

17 25

33 41

49

57 65

73 81

89

97

10

5

11

3

12

1

12

9

13

7

14

5

15

3

16

1

16

9

17

7

Student #

# o

f O

DR

By # of Referrals1988-99 #ODR per Student

80

32

16 15

8

4 4 3 24

2 3 30 1 0 1 1 0 0 1 0 1

0

10

20

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

#Referrals

0

5

10

15

20

Ave

Re

ferr

als

pe

r D

ay

Sept Oct Nov Dec Jan Feb Mar Apr May Jun

School Months

Office Referrals per Day per MonthLast year

0

5

10

15

20

Ave

Re

ferr

als

pe

r D

ay

Sept Oct Nov Dec Jan Feb Mar Apr May Jun

School Months

Office Referrals per Day per MonthLast year

0

5

10

15

20

Ave

Re

ferr

als

pe

r D

ay

Sept Oct Nov Dec Jan Feb Mar Apr May Jun

School Months

Office Referrals per Day per MonthLast Year and This Year

IF...

FOCUS ON...

More than 40% of students receive one or more office referrals

More than 2.5 office referrals per student

School Wide System

More than 35% of office referrals come from non-classroom settings

More than 15% of students referred from non-classroom settings

Non-Classroom System

More than 60% of office referrals come from the classroom

50% or more of office referrals come from less than 10% of classrooms

Classroom Systems

More than 10-15 students receive 5 or more office referrals

Targeted Group Interventions / Classroom Systems

Less than 10 students with 10 or more office referrals Less than 10 students continue rate of referrals after

receiving targeted group settings Small number of students destabilizing overall

functioning of school

Individual Student Systems

Final Thoughts

• Don’t collect data for collection sake – make sure informs the process

• Don’t “drown” in data – keep focused on the question

• Data without context are simply numbers

Reviewing Student MeasuresAnswer the “Big Five” questions1. How often are problem behavior events

occurring?2. Where are they happening?3. What types of problem behaviors?4. When are the problems occurring?5. Who is contributing?

Using SWIS “Big Five” reports1. Major Discipline Referrals per Day per Month2. Major Discipline Referrals by Location3. Major Discipline Referrals by Problem Behavior4. Major Discipline Referrals by Time5. Major Discipline Referrals by Student

Langley Elementary School: 478 Students, Grades K-5

Problem Identification (look at Major Discipline Referral per Day per Month on next slide)

1. Is there a problem with the absolute standard?

4784.78 .344.78 1.63

✔

PROBLEM- ODRs per day higher than national avg.

100

Langley Elementary SchoolReferrals per Day per Month

4784.78 .344.78 1.63

✔

Langley Elementary School: 478 Students, Grades K-5

Problem Identification (look at Major Discipline Referral per Day per Month on next slide)

1. Is there a problem with the absolute standard?2. Are there trends or patterns?

PROBLEM- ODRs per day higher than national avg.TREND- 4 consecutive mos. of increasing trend

Langley Elementary

Langley Elementary

Langley Elementary

Langley Elementary

PROBLEM- ODRs per day higher than national avg.TREND- 4 consecutive mos. of increasing trend

Happening mostly on the playground

Tardiness a problemDisrespect also a problem

Happening during morning and lunch recess periods

About 3% of students with 2 or more ODRs, 12 students with 5 or more ODRs, 5 students with >30 ODRs

Using Data to Build Solutions• Prevention: How can we avoid the problem context?

– Who, When, Where– Schedule change, curriculum change, etc

• Teaching: How can we define, teach, and monitor what we want?– Teach appropriate behavior– Use problem behavior as negative example

• Recognition: How can we build in systematic reward for desired behavior?

• Extinction: How can we prevent problem behavior from being rewarded?

• Consequences: What are efficient, consistent consequences for problem behavior?

• How will we collect and use data to evaluate (a) implementation fidelity, and (b) impact on student outcomes?

Next Steps

• Review Standards and Protocols

• Review Data Requirements to be an implementing school

• Review Requirements for Recognition

• Use handouts to build best practice routine with your school team