Communicating through Data Displays

October 10, 2006

© 2006 Public Consulting Group, Inc.

Key Terms

Aggregated Data: Data that are presented in summary (as opposed to student-level data)

Alignment: The quality that allows you to compare one test to another test (A vertically aligned test represents real gains or losses from one year to the next)

Disaggregation: Summary data split into different subgroups (e.g. gender, race, ethnicity, lunch status, SPED)

Error: Impacts the validity of an assessment; Includes measurement error and sampling error

Inference: Conclusions that are drawn from a data set Sample: Group of students included in a data set Validity: The statistical term used to determine how much inference can be

made

The Framework

Measures Multiple measures allow for a more complete

picture of student performance Explorations

Explorations enable looking at the data through different lenses to answer essential questions

Disaggregators Disaggregators help reveal the various

factors that impact educational outcomes

Types of Measures

Measures are the “yardstick” that is used to measure student performance. The more measures that are used, the more robust and complete the

picture State Assessments (MCAS)

Usually taken in spring and reported the following fall Not vertically aligned Tests vary from year to year

National Assessments (Terra Nova, ITBS, etc.) Some districts choose to supplement the state assessment with a national

assessment Many of these are vertically aligned and are aligned from year to year National assessments are not aligned to the state curriculum framework

Diagnostic Assessments (DRA, DIBELS) Diagnostic assessments help identify students who need interventions and

supports Diagnostic assessments may not be vertically aligned

Types of Measures (cont’d) Subject Area and Course Grades

Grades are subjective and can tend to be inflated Grades can be compared to performance on state and other assessments to

identify disparities Disciplinary Records

Discipline data can be used to monitor high-risk students and explore the impact of behavior on performance

Discipline consequences provide important information for identifying inequities among groups of students (e.g., students with disabilities, ethnic groups)

Attendance Rates Attendance data can be used to identify students who are at risk Attendance data can be used to explore the relationship between attendance and

performance Graduation Rates

Graduation rates can be used to evaluate the effectiveness of curriculum and instruction

Graduation rates can be analyzed to identify inequities based on student characteristics

Types of Explorations

The type of exploration you choose depends on the question that you want to answer. Types of exploration include: Snapshot Cross-Sectional Longitudinal Gains Item-Level Student Listing Correlation

Snapshot

Shows how a group of students performed against a given measure at a certain point in time.

Limitations: This analysis only presents one point in time.

(Graph Type: Histogram / Bar)

How did students perform at a certain point in time?

State Assessment Performance5th Grade Math, 2005-2006

Washington Elementary School

26%

42%

21%

11%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Warning (n=90) Needs Improvement(n=144)

Proficient (n=72) Advanced (n=36)

Performance Level

Per

cent

of S

tude

nts

Historical

Looks at how students at a particular grade level performed on a given measure across multiple years.

This is what NCLB uses to calculate AYP.

Limitations: This analysis does not take into account differences in the group of students from year to year.

(Graph Type: Floating Column)

How did students at a certain grade-level perform historically?

Analysis of Proficiency over Time5th Grade State Math Assessment

Washington Elementary School

38% 36% 33% 30% 28% 31%

41%36%

30%24% 23%

26%

15%21%

27%

30% 30%28%

6% 6% 9%16% 19% 15%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2000 (n=34) 2001 (n=33) 2002 (n=33) 2003 (n=37) 2004 (n=43) 2005 (n=39)

Year

Pe

rce

nta

ge

of S

tud

en

ts

Warning Needs Improvement Proficient Advanced

Longitudinal

Looks at a cohort of students over time.

Shows “real gains” Limitations:

Comparisons of a group of students from one year to another are only valid using a vertically-aligned test.

(Graph Type: Line)

How did a cohort of students perform over time?

Terra Nova Math PerformanceClass of 2010, Disaggregated by Gender

Washington Elementary School

63 6568

71 6972 71

6259 61 63 65

69 6868 69 7175 77 77 76

0

10

20

30

40

50

60

70

80

90

2000(n=34)

2001(n=34)

2002(n=34)

2003(n=34)

2004(n=34)

2005(n=34)

2006(n=34)

Ave

rag

e N

CE

Sco

re

My District Male Female

Gains

Looks at the extent to which students are improving over time or losing ground based on a particular measure.

Limitations: Caution must be used when drawing conclusions about a given student based upon performance on two tests.

(Graph Type: Stacked Column)

How did students who performed at each level on a prior assessment perform on

subsequent assessments?

State Math Assessment Gains AnalysisGrade 5 (2006) Performance Grouped by Grade 3 (2004) Performance

Washington Elementary School

32

167

0

10 38

18

3

3

11

18

4

0

0

5

9

0

10

20

30

40

50

60

70

Warning (n=45) Needs Improvement(n=65)

Proficient (n=48) Advanced (n=16)

Grade 5 (2006) Performance Level

Num

ber

of S

tude

nts

3rd Grade Warning 3rd Grade NI 3rd Grade Proficient 3rd Grade Advanced

Student Listing

Allows the analysis of students in a group in relation to each other.

Conditional formatting can be added to highlight outliers.

Limitations: Student listings can be difficult to interpret when too many data elements are included.

Student Listing - 6th Grade, 2006-2007Performance on the 5th Grade and 3rd Grade Math Assessment

Washington Elementary School

Last Name First Name3rd Grd Perf Level

5th Grd Perf Level

Move-ment Gender Ethnicity Race

Lunch Status

SPED Status

LnameA FnameA Basic Basic 0 M Hispanic White Full YesLnameB FnameB Basic Basic 0 M Hispanic Black F/R NoLnameC FnameC Proficient Basic -1 M Hispanic Black F/R NoLnameD FnameD Below Basic Basic 1 F Not Hispanic White F/R NoLnameE FnameE Proficient Proficient 0 M Hispanic White Full YesLnameF FnameF Basic Below Basic -1 M Hispanic Black Full NoLnameG FnameG Basic Proficient 1 F Not Hispanic Black Full NoLnameH FnameH Basic Proficient 1 F Not Hispanic Black F/R YesLnameI FnameI Below Basic Basic 1 F Not Hispanic Asian F/R YesLnameJ FnameJ Proficient Basic -1 M Hispanic White Full NoLnameK FnameK Below Basic Basic 1 F Not Hispanic Asian F/R NoLnameL FnameL Below Basic Below Basic 0 M Not Hispanic Asian Full NoLnameM FnameM Advanced Proficient -1 M Hispanic White F/R No

Advanced one or more levelsStayed the sameDropped one or more levels

What are the characteristics of specific students?

Item Analysis

Displays how students did on each item or within a particular standard or strand.

Providing reference groups is important for tests that are not aligned from year to year because that is the only way to determine relative performance.

Limitations: Smaller sample sizes (e.g. classroom-level) limit the inferences that can be made

(Graph Type: Scatter)

How did a group of students perform on an item or on a set

of items on a specific assessment?

Percent Correct by Objective 5th Grade Math Assessment, 2005-2006

Washington Elementary School

40%

45%

50%

55%

60%

65%

70%

75%

80%

85%

90%

Ap

pro

x.

Me

as

ure

s

Wh

ole

Nm

brs

&D

ec

ima

ls

Co

mp

os

ing

/Re

vis

ing

Alg

eb

raic

Co

nc

ep

ts

Me

tric

Me

as

ure

s

Fra

cti

on

s

Inte

rpre

tati

on

Pe

rce

nts

Cla

ss

. &

Lo

g.

Re

as

on

ing

Ba

sic

Fa

cts

Ma

th.

Ap

pli

ca

tio

ns

Objective

Per

cent

Cor

rect

My School

District

State

Correlation

Looks at the relationship between performance in one measure to performance in another measure.

Correlation does not equal causation.

Limitations: Correlations should not be done with small groups of students.

How is performance in one measure related to performance

in another?

(Graph Type: Scatter)

State Assesment5th Grade Reading to 10th Grade Math

Anytown School District

200

300

400

500

600

700

800

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150

Grade 5 Reading Scale Score

Gra

de 1

0 M

ath

Sca

le S

core

Pearson Coefficient = 0.80

Advanced / Proficient

Needs Improvement /Warning

Cut Scores

Disaggregators

Disaggregators are used to reveal how performance between one group of students differs from another group.

Disaggregators include the following: Race Ethnicity Gender Special Education Status Lunch Status (Income Level) English Proficiency District Grade School

Limitations: Disaggregating small groups of students can lead to subgroups with only a few students. Caution must be used when making inferences from disaggregated data.

Teacher and Teacher Qualifications Program information Mobility Attendance Rates Discipline Infractions and

Consequences Course-taking Patterns Years in the School/District Retention

NC

LBS

ubgr

oups

How does performance differ from one group of students to

another?

Disaggregation (cont’d)State Assessment Performance

5th Grade Math, 2005-2006Washington Elementary School

16% 26% 35%11%

18%12%

-32% -31% -29%

-42%-26% -24%

-100%

-80%

-60%

-40%

-20%

0%

20%

40%

60%

80%

100%

Pe

rce

nt

of

Stu

de

nts

Proficient Advanced Needs Improvement Warning

My School(n=343)

My District(n=23,413)

My State(n=126,845)

(Graph Type: Floating Column)

Disaggregation (cont’d)

(Graph Type: Bar of Pie)

Proficiency by Years in District5th Grade Math Assessment, 2005-2006

Washington Elementary School

Advanced7%

Proficient32%

2 years16%

3 years13%

4 years7%

> 4 years7%

Warning orNeeds Improvement

78%

<= 1 year18%

Disaggregation (cont’d)

(Graph Type: Bar of Pie)

Terra Nova Math PerformanceClass of 2010, Disaggregated by Gender

Washington Elementary School

63 6568

71 6972 71

6259 61 63 65

69 6868 69 7175

77 77 76

0

10

20

30

40

50

60

70

80

90

2000(n=34)

2001(n=34)

2002(n=34)

2003(n=34)

2004(n=34)

2005(n=34)

2006(n=34)

Ave

rage

NC

E S

core

My District Male Female

Data Don’ts

Data can be dangerous! You should avoid:

Comparing performance on tests that have not been aligned; for example: Don’t compare 3rd grade scale scores to 5th grade scale scores Don’t compare 3rd grade Math scale scores to 3rd grade ELA scale scores

Making large inferences from a few data points; for example: Be wary of conclusions about a subject area based on one item on a test Be wary of conclusions about a student’s overall level based on

performance on one test Be wary of conclusions about a student’s strengths or weaknesses based

on performance on one item on one test

Questions?