Download - Using Data to Problem Solve
Using Data to Problem SolveSusan Barrettwww.pbis.orgwww.pbismaryland.org
Thanks to…
Center on PBIS
Steve Goodman- Michigan
Tim Lewis
Rob Horner
George Sugai
Catherine Bradshaw
Don Kincaid
Adopt systems perspective at a Building Level
Systems Perspective
Organization do not “behave” …individuals behave
“Organization is group of individuals who behave together to achieve a common goal”
“Systems are needed to support collective use of best practices by individuals in an organization” (Horner, 2001)
Schools as Systems
Goal to create communities that for all its members have common
Vision
Language, &
Experience
Biglan, 1995; Horner, 2002
What a Leadership Team does…
Communicates common vision for schoolwide supports
Works collaboratively to establish building capacity to support all students
Commits resources to establish procedures for support
Develops methods for evaluating progress towards measureable outcomes
Action planning based on data
Should get easier for your school over time
HandbookDescribes core features
Expectations and teaching matrix (rules for settings)
Teaching plans and teaching schedule
Acknowledgement system
Continuum of consequences for problem behavior
Building Leadership TeamRegular meeting schedule and process
Regular schedule for annual planning and training
Annual Calendar of Activities
On-going support for staff
Your Leadership TeamYour Leadership Team
Does your team understand the leadership Does your team understand the leadership function in managing and coordinating function in managing and coordinating implementation?implementation?
On a scale of 1 (low) to 5 (high), how well is On a scale of 1 (low) to 5 (high), how well is your team doing with this responsibility? your team doing with this responsibility?
Team TimeTeam Time
Standards and Protocols
Available on www.pbismaryland.org
What is a “PBIS trained” team?
What happens if a school does not meet criteria?
What is required to be a PBIS implementing school?
How does a school become inactive?
What is required for a school to be eligible for PBIS Maryland Recognition?
Purpose of Systems MeasuresBenchmarks of Quality Checklist
Evaluates status of Tier I Positive Behavior Supports
Completed Annually, Required for Recognition
Submitted to Jerry electronically on April 10
Submitted online at www.pbssurveys.org
Implementation Phase Inventory
Required for Recognition
Due November 10 and April 10 to Jerry
Self-Assessment Survey- Not required but…
Evaluates status of Schoolwide, Nonclassroom, Classroom and Individual Student Supports
Submitted online at www.pbssurveys.org
What is the BOQ?• Lists components of PBIS programs
that address the critical elements of PBIS implementation
• Completed by school teams on a yearly basis to assess how they score on a 100 point scale with regard to developing and implementing school-wide PBIS
• Useful in developing action plans for following year
• One of the measures used by MSDE-SPHS-JHU to determine schools achieving Exemplar Status
Three Components of Benchmarks of Quality
Team Member Rating FormCompleted by team members independently
Returned to coach
Scoring FormCompleted by coach using Scoring Guide
Used for reporting back to team
Scoring GuideDescribes procedure for completing BOQ
Includes a rubric for scoring each item
BOQ Will Provide:Summary of team members’ perceptions of PBIS implementation (scored: ++ in place, + needs improvement, and -
not in place)
• Objective assessment of school’s implementation based on criteria described in a rubric (100 point scale)
• Comparison between the above factors which will encourage discussion of strengths and weaknesses and provides ideas for action planning
What is the IPI?• Implementation Phases Inventory
• Two times/year – Due November 10, April 10
• Coach completes with Team• Four Phases
– Preparation– Initiation– Implementation– Maintenance
What is the SET?
School-wide Evaluation Tool
– One of several methods to evaluate Tier 1 – Required if school is seeking Recognition
Status– External certified SET Assessor will conduct
site visit – Should you be a SET Assessor?
Measures the level of implementation of SWPBIS(not intended to measure everything!)
• The Critical Features– Expectations Defined– Expectations Taught– System for Encouraging Expected
Behaviors– System for Discouraging Problem
Behaviors– Monitoring and Decision Making– Management– District Level Support
What does it measure?
Why use it?• The results help PBIS teams:
– Assess the features of PBIS in place– Determine annual goals for school-wide
effective behavior support – evaluate on-going efforts toward
school-wide behavior support – design and revise procedures as
needed – compare efforts toward school-wide
effective behavior support from year to year
What does it look like?• Permanent product review
– Office Discipline Referral (ODR) form– Current Action Plan– Discipline Handbook/Plan– School Improvement Plan– Lesson Plans & Schedule
• 2 to 3 hour school visit:– Observations
• Classroom and Non-classroom settings
– Interviews• Administrator, Staff, PBIS Team Members, and
Students
What is the Self-Assessment Survey?
• Self-assessment survey to assess the extent to which Positive Behavior Support practices and systems are in place within a school– School-wide (15 items)– Non-classroom (Specific Setting) (9 items)– Classroom (11 items)– Individual Student (8 items)
Who Completes the Self-Assessment Survey?
• Initially, the entire staff in a school completes the Survey.
• In subsequent years and as an on-going assessment and planning tool, the Survey can be completed in several ways:
– All staff at a staff meeting.
– Individuals from a representative group.
– Team member-led focus group.
Using the Self-Assessment Information for Decision Making
• Is a system in place?– “in place” > 66%
• Is there a need to focus on a system?– Current status of “in place” is < 66% and– Priority for improvement is “High” for > 50%
• Which system should receive focus first?– Always establish schoolwide as first priority
• Which features of the system need attention?
• Combine survey outcomes with information on office referrals, attendance, suspensions, vandalism, perceptions of staff/faculty
Individual Summary Charts
• Charts are provided for each system (school-wide, nonclassroom, classroom, and individual)
• Current status Charts
– Percentage of respondents who answered "In Place", "Partially In Place", and "Not In Place"
• Improvement Priority Charts
– Percentage of respondents who answered "High", "Medium", and "Low”
Example of PBS Self Assessment Survey Individual Summaries Chart
Analysis of Schoolwide System Chart
Shows a chart with bars for components of the schoolwide system
– Expectations defined (question 1)– Expectations taught (question 2)– Reward system (question 3)– Violations system (question 4-8) – Monitoring (question 10-12)– Management (question 9, 14-16)– District support (question 17-18)
Analysis of Schoolwide System Chart
Example of
PBS Self Assessment Survey
Individual Item Score
Schoolwide Component
White = In Place
Yellow = Partial In Place
Red = Not In Place
Why conduct Self-Assessment Survey in addition to Checklists?
• Checklists are conducted by team, all/most staff complete survey
• Look for areas of convergence across tools– Increases confidence of data
• Look for areas of divergence across tools– Decrease confidence of data?– Possible reasons for disparity…
• Lack of understanding of questions• Staff not fully aware of work of Building
Leadership Team• Support component not fully “In Place”
Differences between the BOQ action plan form
and the Self-Assessment SurveyBenchmarks of Quality
EBS Self-Assessment Survey
Purpose?
Evaluate on-going progress towards schoolwide PBS
Evaluate extent that all systems (schoolwide, nonclassroom, classroom, individual) are in place
When administered?
Monthly- progress monitor Tier 1
Annually
Who completes?School Leadership team, completed as a team
All school staff (or representative sample) completed individually
Time involved?10-15 minutes 30-45 minutes
To Do List• Review results from…
• BOQ Checklist
• Determine if you would like your school staff to complete Self-Assessment Survey
• Review School’s Action plan- What is the link to overall School Improvement Plan?
• Based on this information complete action plan for you!
• What celebrations can you share with your school community before this year is over?
• What is your plan to strengthen your schools’ behavior support for the next school Year?
Please take a moment to complete the appropriate section of the Please take a moment to complete the appropriate section of the Follow-Up Activity Worksheet to document the work yet to be doneFollow-Up Activity Worksheet to document the work yet to be donePlease take a moment to complete the appropriate section of the Please take a moment to complete the appropriate section of the Follow-Up Activity Worksheet to document the work yet to be doneFollow-Up Activity Worksheet to document the work yet to be done
Student Measures
• How do we know Implementation of Tier 1 PBIS is making an impact?
• What data should our team be reviewing?
• How do we build that into the agenda so we it is standard practice?
• Do we have a core group on our team that reviews that data prior to the monthly team meeting?
Data-Based Decision Making1. Determine what questions you want to answer
2. Determine what data will help to answer questions
3. Determine the simplest way to get data
4. Put system in place to collect data
5. Analyze data to answer questions
Focus on both Academic and Social Outcomes
1. Determine what questions you want to answer
Examples
• Can we predict problems/success?– When/where/who?
• Possible “function” of problem behavior?
• Who needs targeted or intensive academic supports?
• What environmental changes/supports are needed?
2. Determine what data will help to answer questions
• Existing data set(s)
• Current data collection
• Additional / new data
• Confidence in accuracy?
• Complete picture?
3. Determine the simplest way to get data
• Agreement on definitions• Standard forms / process• Frequency of collection• Target “Multi-purpose” data/use
Train ALL staff on use & provide on-going TA
4. Put system in place to collect data
• Build on existing systems
• Add components over time
• Central entry point– Electronic
5. Analyze data to answer questions
• Trends
• Instruction & supports in place/not in-place
• Pre/post “big outcomes”
• Comparisons (norm / local)– Relative growth– Absolute growth
By Location1998-99/#ODR by Location
301
76
2
7955
4
36 36 38
1 8 10
50
100
150
200
250
300
350
Location
#O
DR
By Behavior1998-99 #ODR/Behavior
0
20
40
60
80
100
120
140
160
180
200
Fighting Class Disruption Non-compliance InappropriateLanguage
Cut Class/Out ofArea
Property Smoking/Drugs
Behavior
By Student1998-99 By Student
0123456789
10111213141516171819202122232425
1 9
17 25
33 41
49
57 65
73 81
89
97
10
5
11
3
12
1
12
9
13
7
14
5
15
3
16
1
16
9
17
7
Student #
# o
f O
DR
By # of Referrals1988-99 #ODR per Student
80
32
16 15
8
4 4 3 24
2 3 30 1 0 1 1 0 0 1 0 1
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
#Referrals
0
5
10
15
20
Ave
Re
ferr
als
pe
r D
ay
Sept Oct Nov Dec Jan Feb Mar Apr May Jun
School Months
Office Referrals per Day per MonthLast year
0
5
10
15
20
Ave
Re
ferr
als
pe
r D
ay
Sept Oct Nov Dec Jan Feb Mar Apr May Jun
School Months
Office Referrals per Day per MonthLast year
0
5
10
15
20
Ave
Re
ferr
als
pe
r D
ay
Sept Oct Nov Dec Jan Feb Mar Apr May Jun
School Months
Office Referrals per Day per MonthLast Year and This Year
IF...
FOCUS ON...
More than 40% of students receive one or more office referrals
More than 2.5 office referrals per student
School Wide System
More than 35% of office referrals come from non-classroom settings
More than 15% of students referred from non-classroom settings
Non-Classroom System
More than 60% of office referrals come from the classroom
50% or more of office referrals come from less than 10% of classrooms
Classroom Systems
More than 10-15 students receive 5 or more office referrals
Targeted Group Interventions / Classroom Systems
Less than 10 students with 10 or more office referrals Less than 10 students continue rate of referrals after
receiving targeted group settings Small number of students destabilizing overall
functioning of school
Individual Student Systems
Final Thoughts
• Don’t collect data for collection sake – make sure informs the process
• Don’t “drown” in data – keep focused on the question
• Data without context are simply numbers
Reviewing Student MeasuresAnswer the “Big Five” questions1. How often are problem behavior events
occurring?2. Where are they happening?3. What types of problem behaviors?4. When are the problems occurring?5. Who is contributing?
Using SWIS “Big Five” reports1. Major Discipline Referrals per Day per Month2. Major Discipline Referrals by Location3. Major Discipline Referrals by Problem Behavior4. Major Discipline Referrals by Time5. Major Discipline Referrals by Student
Langley Elementary School: 478 Students, Grades K-5
Problem Identification (look at Major Discipline Referral per Day per Month on next slide)
1. Is there a problem with the absolute standard?
4784.78 .344.78 1.63
✔
PROBLEM- ODRs per day higher than national avg.
100
Langley Elementary SchoolReferrals per Day per Month
4784.78 .344.78 1.63
✔
Langley Elementary School: 478 Students, Grades K-5
Problem Identification (look at Major Discipline Referral per Day per Month on next slide)
1. Is there a problem with the absolute standard?2. Are there trends or patterns?
PROBLEM- ODRs per day higher than national avg.TREND- 4 consecutive mos. of increasing trend
Langley Elementary
Langley Elementary
Langley Elementary
Langley Elementary
PROBLEM- ODRs per day higher than national avg.TREND- 4 consecutive mos. of increasing trend
Happening mostly on the playground
Tardiness a problemDisrespect also a problem
Happening during morning and lunch recess periods
About 3% of students with 2 or more ODRs, 12 students with 5 or more ODRs, 5 students with >30 ODRs
Using Data to Build Solutions• Prevention: How can we avoid the problem context?
– Who, When, Where– Schedule change, curriculum change, etc
• Teaching: How can we define, teach, and monitor what we want?– Teach appropriate behavior– Use problem behavior as negative example
• Recognition: How can we build in systematic reward for desired behavior?
• Extinction: How can we prevent problem behavior from being rewarded?
• Consequences: What are efficient, consistent consequences for problem behavior?
• How will we collect and use data to evaluate (a) implementation fidelity, and (b) impact on student outcomes?
Next Steps
• Review Standards and Protocols
• Review Data Requirements to be an implementing school
• Review Requirements for Recognition
• Use handouts to build best practice routine with your school team