Using Data for Decision-making

Rob Horner, Anne Todd, Steve Newton, Bob Algozzine, Kate Algozzine

Main Ideas Decisions are more likely to be effective and

efficient when they are based on data.

The quality of decision-making depends most on the first step (defining the problem to be solved)

Define problems with precision and clarity

Main Ideas Data help us ask the right questions…they do

not provide the answers: Use data to Identify problems Refine problems Define the questions that lead to solutions

Data help place the “problem” in the context rather than in the students.

Main Idea The process a team uses to problem solve is

important: Roles:

Facilitator; Recorder; Data analyst; Active member

Organization Agenda; Old business (did we do what we said we would

do); New business; Action plan for decisions. What happens BEFORE a meeting What happens DURING a meeting What happen AFTER a meeting

Main Ideas Build “decision systems” not “data systems” Use data in “decision layers”

Is there a problem? (overall rate of ODR) Localize the problem

(location, problem behavior, students, time of day)

Get specific Don’t drown in the data It’s “OK” to be doing well Be efficient

? Problem ?

Beh Location Time Studentof

Day

Setting A B C D E F G H I J K

Locations 1 2 3 4 5 6 7 8 9 10

Times A B C D E F G H I J K

Students 1 2 3 4 5 6 7 8 9 10 11

Using Data Do we have a problem? Refine the description of the problem?

What behavior, Who, Where, When, Why Test hypotheses

“I think the problem on the playground is due to Eric” “ We think the lunch period is too long” “We believe the end of ‘block schedule” is used poorly”

Define how to monitor if solution is effective

CollectCollect and Useand Use

DataData

Review Status and

Identify Problems

Develop andRefine

Hypotheses

Discuss andSelect

Solutions

Develop andImplementAction Plan

Evaluate andRevise

Action Plan

Problem Solving Foundations

Team Initiated Problem Solving (TIPS) Model

Identifying problems/issues What data to monitor

ODR per day per month OSS, ISS, Attendance, Teacher report Team Checklist/ SET (are we doing what we planned to do?)

What question to answer Do we have a problem?

What questions to ask of Level, Trend, Peaks How do our data compare with last year? How do our data compare with national/regional norms? How do our data compare with our preferred/expected status?

If a problem is identified, then ask What are the data we need to make a good decision?

Tota

l Offi

ce D

isci

plin

e R

efer

rals

Total Office Discipline Referrals as of January 10

Change Report OptionsChange Report Options1.41.82.72.52.753.4900.000

Using Data to Refine Problem Statement The statement of a problem is important for team-

based problem solving. Everyone must be working on the same problem with the same

assumptions.

Problems often are framed in a “Primary” form, that creates concern, but that is not useful for problem-solving.

Frame primary problems based on initial review of data Use more detailed review of data to build “Solvable Problem

Statements.”

Solvable Problem Statements(What are the data we need for a decision?) Solvable problem statements include

information about the five core “W” questions. What is problem, and how often is it happening Where is it happening Who is engaged in the behavior When the problem is most likely Why the problem is sustaining

Primary versus Precision Statements Primary Statements

Too many referrals September has more

suspensions than last year

Gang behavior is increasing

The cafeteria is out of control

Student disrespect is out of control

Precision Statements There are more ODRs

for aggression on the playground than last year. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

Primary versus Precision Statements Primary Statements

Too many referrals September has more

suspensions than last year

Gang behavior is increasing

The cafeteria is out of control

Student disrespect is out of control

Precision Statements There are more ODRs

for aggression on the playground than last year. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

Precise or Primary Statement? Children are using inappropriate language

with a high frequency in the presence of both adults and other children. This is creating a sense of disrespect and incivility in the school

James D. is hitting others in the cafeteria during lunch, and his hitting is maintained by peer attention.

Precise or Primary Statement? ODRs during December are higher than in any other

month.

Minor disrespect and disruption are increasing over time, and are most likely during the last 15 minutes of our block periods when students are engaged in independent seat work. This pattern is most common in 7th and 8th grades, involves many students, and appears to be maintained by escape from work (but may also be maintained by peer attention… we are not sure).

Precise or Primary Statement? Three 5th grade boys are name calling and

touching girls inappropriately during recess in an apparent attempt to obtain attention and possibly unsophisticated sexual expression.

Boys are engaging in sexual harassment

Organizing Data for Decision-making Compare data across time Moving from counts to count/month

SWIS summary 2008-2009 (Majors Only)3,410 schools; 1,737,432 students; 1,500,770 ODRs

Grade Range

Number of Schools

Avg. Enrollment per school

National Avg. for Major ODRs per 100 students, per school day

K-6 2,162 450 .34 = about 1 Major ODR every 3 school days, or about 34 every 100 days

6-9 602 657 .85 = a little less than 1 Major ODR per school day, or about 85 every 100 days

9-12 215 887 1.27 = more than 1 Major ODR per school day, or about 127 every 100 days

K-(8-12)

431 408 1.06 = about 1 Major ODR per school day, or about 106 every 100 days

Newton, J.S., Todd, A.W., Algozzine, K, Horner, R.H. & Algozzine, B. (2009). The Team Initiated Problem Solving (TIPS) Training Manual. Educational and Community Supports, University of Oregon unpublished training manual.

Start with the ODR/Day/Month Graph Use the information in the data to build a

narrative that draws the team into problem solving.

Be descriptive Link local data to national patterns Tie the data back to local conditions/events.

Elementary School 465 students (465/ 100 = 4.6 X .34= 1.56)

Our rate of problem behavior has been above the national average for schools our size for 9 of 10 months this year. There has been a

decreasing trend since Dec.

Elementary School 1000 Students (1000/100 =10 X .34= 3.4)

The rate of problem behavior has been at or below the national average schools our

size for 6 of 10 months. The past 4 months have been below the national

average

Middle School 765 students (765/100 = 7.6 X .85= 6.46)

The rate of problem behavior has been at or below the national average schools our

size for 9 of 10 months. The past 8 months have been below the national

average with a decreasing trend

Describe the narrative for this school

Describe the narrative for this school

Describe the narrative for this school

Describe the narrative for this school

Using Data to Build Precision Given that we know we have a problem

What problem behaviors Where are they occurring When are they occurring Who is involved Why do they keep happening

What questions to ask about the patterns of problem behaviors?1. Do we have one problem behavior situations

or more than one? 2. Do we have many problem behaviors or just

a few problem behaviors? 3. Do we have clusters of problem behaviors?4. What school wide expectations do we need

to re-teach?

Data should allow asking the right question… not supplying the answer. If many referrals in class

Which classes? Which students? When?

If many referrals in cafeteria Which students? What times? (beginning or end of lunch period?) What problem behaviors?

Disrespect is our most frequent problem behavior.We also have incidents of fighting and harassment

What are next questions?

Who, When, Why?

Our most frequent problem behavior is

disrespect, followed by inappropriate language,

disruption and tardy

What are next questions?

Who, When, Why?

We have many instances of disrespect,

aggression/fighting. technology violations, tardy, harassment, lying, skipping, and inappropriate language

What questions to ask about Referrals by Location

Where are the problems occurring?   Are there problems in many locations,

clusters of locations, or one location?

Many problem behaviors in class

Many problem behaviors in unstructured settings (hall, playground, parking lot,

bathroom)

Many problems in the cafeteria, hallway, common

area, class, bathroom. Where is the ‘unknown’

location?

What questions to ask about Referrals by Time When are the problem behaviors occurring?

How do those times match with the daily activities?

How does this information match up to Referrals by Location?

Most problems are occurring between 9:45-

10:45.

Other problematic times are 8-8:45 and 11:30.

Most problems are occurring at

noon

Many problems at 12:15, 7:45-8:30, 10:00-10:45

What questions to ask about Referrals by Student

What proportion of students has 0-1 ODR?

What proportion of students has 2-5 ODRs?

What proportion of students has 6+ ODRs?

Do we have systems of support that increase student success?

Student # 121 needs individualized support.

8 students are likely candidates for some type of Tier II support.

87% of our students have received 0-1 ODR

14 students are likely candidates for some type of Tier II support.

Student #119 needs individualized support

We have 11 students who are likely candidates for some

type of Tier II support93% of our students have received no more than one

ODR

What questions to ask about Referrals by Perceived Motivation

What is perceived as maintaining the problem behavior?

Are there one or more perceptions?

The problem behaviors are most likely maintained by task avoidance and peer

avoidance.We have many incidents with unknown

motivation

Problem behaviors appear to be maintained by peer and adult attention

Problem behaviors appear to be

maintained by escaping adult

attention

Using the TIPS Minutes to guideProblem Solving

1. Write the precision problem statement in the left hand column below

2. Define a goal and write it in the right hand column

Using Data to Build Solutions Prevention: How can we avoid the problem context?

Who, When, Where Schedule change, curriculum change, etc

Teaching: How can we define, teach, and monitor what we want? Teach appropriate behavior Use problem behavior as negative example

Recognition: How can we build in systematic reward for desired behavior?

Extinction: How can we prevent problem behavior from being rewarded?

Consequences: What are efficient, consistent consequences for problem behavior?

How will we collect and use data to evaluate (a) implementation fidelity, and (b) impact on student outcomes?

Solution DevelopmentPrevention

Teaching

Reward

Extinction

Corrective Consequence

Data Collection

Trevor Test Middle School

565 studentsGrades 6,7,8

Trevor Test Middle SchoolIs there a problem? If so, what is it?

0

5

10

15

20

Sep Oct Nov Dec Jan

School Months

Avg.

ODR

s Pe

r Sch

ool D

ay

School Avg. National Avg. = 4.8

Trevor Test Middle School 11/01/2007 through 01/31/2008 (last 3 mos.)Referrals by Problem Behavior

0102030405060708090

100110120130140

Min

orTa

rdy

Bom

bAr

son

Wea

pons

Oth

erU

nkno

wn

Dru

gsM

-Prp

ty M

isus

eM

-Oth

erM

-Dre

ssM

-Tec

hTe

chIn

app

affe

ctio

nO

ut b

ound

sM

-Unk

now

nG

ang

disp

lay

Skip

Trua

nLy

ing

M-D

isru

ptio

nD

ress

Toba

cco

Alco

hol

Com

bust

M-In

app

lan

Forg

e/Th

eft

Vand

alM

-Con

tact

M-D

isre

spt

Prop

dam

Agg/

Figh

tM

-Tar

dySk

ipH

aras

sD

isre

spt

Inap

p la

nD

isru

ptio

n

Num

ber o

f Ref

erra

ls

Referrals by Time

0102030405060708090

100110120130

7:00

AM

7:30

AM

8:00

AM

8:30

AM

9:00

AM

9:30

AM

10:0

0 A

M

10:3

0 A

M

11:0

0 A

M

11:3

0 A

M

12:0

0 P

M

12:3

0 P

M

1:00

PM

1:30

PM

2:00

PM

2:30

PM

3:00

PM

3:30

PM

4:00

PM

4:30

PM

5:00

PM

Num

ber o

f Ref

erra

ls

Referrals by Location

0

20

40

60

80

100

120

140

160

180

200

Ply

gd

Par

k lo

t

Unk

now

n

Offi

ce

Lock

er rm

Off-

Cam

pus

Sta

dium

Mus

ic rm

Libr

ary

Bat

hrm

Gym

Bus

Zn

Bus

Spe

cial

evt

Oth

er

Cla

ss

Com

mon Hal

l

Caf

é

Num

ber o

f Ref

erra

ls

Referrals by Student

0

10

20

30

40

50

60

70

80

1 13 16 18 2 20 24 28 30 33 38 4 9 17 21 37 43 23 31 39 40 41 5 8 11 29 12 22 25 35 42 6 14 34 15 26 36 7 3 19 32 27 10

Student No.

Num

ber o

f Ref

erra

ls

Precise Problem Statement &Hypothesis Development Many students from all grade levels are engaging in

disruption, inappropriate language and harassment in cafeteria and hallway during lunch, and the behavior is maintained by peer attention

A smaller number of students engage in skipping and noncompliance/defiance in classes, (mostly in rooms 13, 14 and 18), and these behaviors appear to be maintained by escape.

Solution DevelopmentPrevention

Teaching

Reward

Extinction

Corrective Consequence

Data Collection

Solution Development:For disruption in hall and cafeteriaPrevention *Teach behavioral expectations in

cafeteria*Maintain current lunch schedule, but shift classes to balance numbers.

Teaching

Reward Establish “Friday Five”: Extra 5 min of lunch on Friday for five good days.

Extinction Encourage all students to work for “Friday Five”… make reward for problem behavior less likely

Corrective Consequence Active supervision, and continued early consequence (ODR)

Data Collection Maintain ODR record and supervisor weekly report

Phoenix ElementaryUsing Data For Decision-Making

You are the Behavior Support team for Phoenix Elementary. 265 students k-5 Do you have a problem? Where? With Whom? What other information might you want? Given what you know, what considerations

would you have for possible action?

Examples Phoenix Elementary

What is national comparison? 265/100 = 2.65 2.65 X .34 = .90

Absolute level compared with last year, compared with teacher/staff impressions, compared with family impressions, compared with student impressions.

Where, what, when, who , why Hypotheses? Solutions

0

1

2

3

4

5

Mea

n S

tude

nt C

onta

cts

per D

ay

Sept Oct Nov Dec Jan Feb Mar Apr May June

School Months

Phoenix Student DisciplineContacts

Year 1Year 2

0

20

40

60

80

100

120

140

Num

ber o

f Stu

dent

Con

tact

s

Playgd ClassRestrm Caf OtherLocation

Phoenix ElementaryLocations

Year 1Year 2

Phoenix Elementary ODR per Student

0

2

4

6

8

10

12

14

161 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49

Students

Num

ber o

f Stu

dent

con

tact

s

Major ODRs Year 2 Only

Phoenix Elementary ODR per Time of Day

0

5

10

15

20

25

30

Time of Day

Num

ber o

f Ref

erra

ls

Problem Statement Do we have a problem? Build a precise problem statement

Solution DevelopmentPrevention

Teaching

Reward

Extinction

Corrective Consequence

Data Collection

Langley Elementary School478 StudentsK-5

Precision Statement/Hypothesis What Where When Who Why What other info needed?

Possible Solutions?

Solution DevelopmentPrevention

Teaching

Reward

Extinction

Corrective Consequence

Data Collection

Sandhill High school354 students

Sandhill High SchoolIs there a problem? If so, what is it?

0

1

2

3

4

5

Sep Oct Nov Dec Jan Feb Mar Apr May Jun

School Months

Avg.

ODR

s Pe

r Sch

ool D

ay

Previous Year This Year National Avg. = 3.72

Sandhill High School 02/01/2007 through 04/30/2007 (last 3 mos.)Referrals by Problem Behavior

05

1015202530354045505560

Arso

nBo

mb

Com

bust

Dre

ssG

ang

disp

lay

Iapp

affe

ctio

nM

-Con

tact

M-D

isre

spt

M-D

isru

ptio

nM

-Dre

ssM

-Inap

p la

nM

-Oth

erM

-Prp

ty M

isus

eM

-Tar

dyM

-Tec

hM

-Unk

now

nO

ther

beh

avO

ut b

ound

sTe

chTo

bacc

oU

nkno

wn

beha

vVa

ndal

Alco

hol

Wea

pons

M-W

arni

ngPr

op d

amTh

eft

Tard

yLy

ing

Dru

gsIn

app

lan

Har

ass

Agg/

Figh

tD

isru

ptio

nD

isre

spt

Skip

Num

ber o

f Ref

erra

ls

Referrals by Time

0

5

10

15

20

25

30

35

7:00

AM

7:30

AM

8:00

AM

8:30

AM

9:00

AM

9:30

AM

10:0

0 AM

10:3

0 AM

11:0

0 AM

11:3

0 AM

12:0

0 P

M

12:3

0 PM

1:00

PM

1:30

PM

2:00

PM

2:30

PM

3:00

PM

3:30

PM

4:00

PM

4:30

PM

5:00

PM

Num

ber o

f Ref

erra

ls

Referrals by Location

0

10

20

30

40

50

60

70

80

Bus

Bus

Zn

Lock

er rm

Off-

Cam

pus

Ply

gd

Spec

ial e

vt

Stad

ium

Gym

Bath

rm

Com

mon

Offi

ce

Park

lot

Libr

ary

Caf

é

Hal

l

Oth

er

Unk

now

n

Cla

ss

Num

ber o

f Ref

erra

ls

Referrals by Student

0

2

4

6

8

10

12

14

3 4 5 6 7 9 11 12 14 15 16 17 18 19 20 21 24 25 27 28 30 31 32 33 34 35 37 38 39 43 44 45 46 48 49 51 52 1 50 23 8 40 41 2 26 29 36 10 47 53 42 22 13

Num

ber o

f Ref

erra

ls

Precision Statement/Hypothesis What Where When Who Why What other info needed?

Possible Solutions?

Solution DevelopmentPrevention

Teaching

Reward

Extinction

Corrective Consequence

Data Collection

Summary Establish information systems for decision-

making… not data systems for reporting. Use data to identify problems Use data to clarify/refine problems Use knowledge of context and content to

build solutions. Use data to monitor impact of solutions.