1

Children First Intensive

2008 Grade 5 Social StudiesAnalyzing Outcomes for ESO Network 14

March 25, 2009 Social Studies Conference, PS/MS 3

Deena Abu-Lughod, Senior Achievement Facilitator

Network Leader: Bob Cohen

Brandon Alvarez, Joann Benoit, Deirdre Burke, Freddie Capshaw, Alan Godlewicz, Dr. Pamela McCarthy

2

Learning Intentions

Understand the construction of the Grade 5 Social Studies test

Understand Network 14 performance and improvement trends

How to use benchmark comparisons to reflect on curriculum

Understand how Social Studies outcomes relate to the ELA outcomes

3

Obtaining Information from the State

Copies of old state tests are available through NYSED:

http://www.nysedregents.org/testing/scostei/socstudies5.html

The Core Curriculum and Standards are available through these links.

http://www.emsc.nysed.gov/ciai/socst/pub/sscore1.pdf

http://www.emsc.nysed.gov/ciai/socst/socstands/socstand.html

The last page of the rating guide provides the “specifications grid” that shows the distribution of multiple choice questions by Unit and Standard.

The grid shows that questions appear sequentially by grade. In 2008, items 1-11 related to Grade 3 content, items 13-23 related to Grade 4 content, items 27-30 to Grade 4-5 content, and the last four items were cross-topical and skills-based.

4

Screen shot of a Specifications Grid

5

State Social Studies Toolkit: http://www.emsc.nysed.gov/ciai/socst/ssresources.html

6

Performance Highlights

Most schools have improved the percent of students passing the social studies exam by several points each year, notably x3, x20, x24, x86 and x360.

x20 has improved the most.

What are they doing that has made the difference?

7

Avg. Grade 5 Social Studies Scores, 2006-2008

50

55

60

65

70

75

80

85

Av

era

ge

Sc

ore

3 4 7 20 24 59 81 86 95 145 166 280 340 360 NetAvg.

Avg. 2006 Avg. 2007 Avg. 2008

8

Comparing 2007 and 2008 Results

The following two stacked bar graphs show the percent of students in each performance level on the Fall 2007 and Fall 2008 Social Studies exams.

Finally, there is a bar chart disaggregating the data by gender. Consistent with the tendency in literacy, girls outscored boys at all schools except x3, x81, and x360.

9

Gr 5 Social Studies 2007: Network 14 Dist. Of Students by Performance Level

1317

28

8

23

3

34

10

32

183

5

20

214

4

13

20

9

31

3

39

29

21

18

2

18

16

223

34

32

64

61

46

46

126 89

32

113

58

38

31

770

9

414

47

4

43

207 4

3722

52

218

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

3 4 20 24 59 81 86 95 145 279 280 340 360 Net.Avg.

School

Per

cen

t Lvl 4

Lvl 3

Lvl 2

Lvl 1

10

Grade 5 Social Studies 2008: Network 14 Dist. Of Students by Performance Level

9

17

32

10 7

39

627

19

31

71

27

4 410

313

3

6

21

13

3

17

9

31

20

13

16

20

6

8

15201

25

28

92

66

36

67

33

119

75

33

38

38

52

36

46

784

16

318

21

63

9

43

71

186

310

31

11

4

327

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

3 4 7 20 24 59 81 86 95 145 166 279 280 340 360 All

Lvl 4

Lvl 3

Lvl 2

Lvl 1

11

Grade 5 Fall 2008 Social Studies Scores, by Gender

50.0

55.0

60.0

65.0

70.0

75.0

80.0

85.0

09X004 09X145 09X166 10X003 10X007 10X020 10X024 10X059 10X081 10X086 10X095 10X280 10X340 10X360 Grand

Total

F

M

12

Comparing the Network to all Level 4 students

Your annotated copy of the Grade 5 Fall 2008 Social Studies exam indicates the Grade Level, Unit and Standard for each item and the percent of students in Networks 14 and 19 who answered correctly.

The percent of students who answered an item correctly is referred to as an “item analysis”.

However, there is a better way to use the item response data to evaluate curriculum. That is to compare the percent of students answering correctly to the percent of all level 4 students (or all level 3 students) who answered correctly. This helps you distinguish which kinds of questions separate the 2s from the 3s and the 3s from the 4s.

This is called a “benchmark” comparison.

13

Grade 5 Fall 2008 Social Studies Item Response: Networks 14+19 vs Level 4 Benchmark

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 2 21 2 2 2 25 2 27 2 2 3 31 3 3 3 35

Multiple Choice Item

Perc

ent A

nsw

erin

g Co

rrect

ly

Net 14+19 Avg. Level 4 Avg.

Small gap

Large gap

14

“Small” gap questions

15

A large gap question

16

Another large gap question

17

Using benchmarks to review curriculum

Examine the data comparing the percent of your students who answered each item correctly to the Level 3 or Level 4 benchmarks.

Items where the gap was the smallest indicate the types of skill and knowledge that your students were most successful at mastering.

Items where the gaps were large provide clues as to the types of skills and knowledge your students were less successful in mastering. For those skills and knowledge, review where in the curriculum they were taught and how they were assessed.

18

Are ELA and Social Studies scores related?

The correlation coefficient of the Grade 4 ELA with the Grade 5 social studies is .718. This is very high, and slightly higher than the correlation between these same students’ Grade 4 ELA and Math scores (.694). However, it is not as high as the correlation between the Grade 3 and Grade 4 ELA exams (.777).

This means that content-area instruction DOES matter. Note in particular how the correlation between the ELA and Social Studies scores are actually much higher at the middle schools that begin in Grade 5 (.766). That means content area instruction had less of an impact, or independent effect, on student outcomes.

19

Network 14: Correlation of 2008 Gr 4 ELA with Gr 5 Social Studies

1

1.5

2

2.5

3

3.5

4

4.5

0 10 20 30 40 50 60 70 80 90 100

Avg. Social Studies Score

Gr

4 E

LA

Pro

fici

ency

Rat

e

Students here did better in Social Studies than in ELA

Students here did better in ELA than in Social Studies

20

Is correlation a good thing?

Since the social studies exam requires strong literacy skills, it is not surprising to see a high correlation. The fact that the correlation is higher in middle schools with initial 5 th grades suggests a relative weakness in specific social studies skills.

A lower correlation is actually a measure of the greater independence of the social studies score, and therefore of the quality of social studies instruction.

What do the data tell us about what might be happening in different schools and classes?

21

Feedback and debrief; Evaluation

Please complete the Feedback Form now.

Did we achieve our intentions? Better understand the construction of the Grade 5 Social Studies

test Understand what the performance and improvement trends across

Network 14 schools have been How to use benchmark comparisons to revise curriculum Understand how the results of the social studies exam relate to

outcomes on the Grade 4 ELA How to use old tests to construct reliable measures of mastery in

earlier grades