a. effect size can tell a lot! b. future ready core nc accountability conference february 2009

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

1

A. Effect Size Can Tell a Lot!

B. Future Ready Core

NC Accountability ConferenceFebruary 2009

Mike Gallagher

Accountability Services Dept.

NCDPI

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


2

Purposes – Two Parts

• Part A. Show how effect size can give important information to compare test results across EOG, EOC and SAT and other tests.

• Part B. Review the background and give sources of information about the Future Ready Core.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


3

Part A: Effect Size Lens

• Usefulness: Enables one to compare across tests and test forms.

• Usefulness: Provides solid information about performance.

• Concept: What effect size means.

• Calculation: How to figure it.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


4

How does North Carolina improvement on SAT math compare with its improvement on NAEP math?

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


5

SAT Math Score Trend for N.C.

• Between 1990 and 2008, the Math gap between NC and the Nation narrowed from 31 pts. to 4 pts.

• The annual SAT reports on the DPI Accountability web site have lots more information:

www.ncpublicschools.org/accountability

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


6

SAT Math Score Trend

501 501 504508

512 514 516 518 518 515

470479 482

486492 496

505 507513 511

440450460470480490500510520530

1990 1992 1994 1996 1998 2000 2002 2004 2006 2008

Mea

n M

ath

SAT

Scor

e

Nation North Carolina

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


7

NC progress on NAEP mathNAEP results from 1992 to 2005

Compared with the U.S. average and with other states, North Carolina went from back of the pack to ahead of the pack. NC was somewhat sluggish 2003 to 2007.

For the NAEP data in the following slides, go to:

http://nces.ed.gov/nationsreportcard/mathematics/

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


8

NAEP Grade 4 Math

Trend in Average Scale Score

210

215

220

225

230

235

240

245

250

1990 1995 2000 2005 2010

NC

US

Cut for Basic = 214

Cut for Proficient = 249

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


9

NAEP Grade 8 MathTrend in Average Scale Score

250

255

260

265

270

275

280

285

290

1990 1992 1996 2000 2003 2005 2007

NC

USCut for Basic = 262

Cut for Proficient = 299

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


10

Finding the Effect Size aka, “Standardized Difference”

• Effect size compares two groups starting with a scale score.

• For calculation of effect size, the average scale score of each group is needed, as well as the standard deviation (SD) of the scores.

• Find the difference of the averages and divide by the SD.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


11

Comparing trends in tests: NAEP and SAT

• SAT math, comparing NC in 1990 and 2008.

…..Eff.Size = (511-470)/100 = 0.41• NAEP Grade 4 math, comparing NC in

1992 and 2007.

…..Eff.Size = (242-213)/30 = 0.97

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


12

Is that a “Big-un?”

General use Effect Size Indexes

• 0.20 … Small

• 0.50 … Medium

• 0.80 … Large

From A Power Primer, Jacob Cohen, Psychological Bulletin, 1992, Vol 112, No. 1, 155-159.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


13

While we’re talking SAT,

SAT puts an “official” question of the day on its web site.

The math questions are great thinking questions and they are mostly within our 8th grade territory. Why not have 8th graders use them as formative assessment for problem solving?

www.collegeboard.com

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


14

Effect Size and NC Tests

• Effect size can provide a comparison across test editions or subjects.

• See the attached table for some examples.

…. The 3rd Edition Math EOG had much more rigor than the 2nd.

…. Effect size enables continuous comparison.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


15

Obtaining Av. Scale Scores

• The “disag” report is on-line.

• Go to the Accountability web page.

www.ncpublicschools.org/accountability

then scroll way down to Data and Reports, chose “State, School System (LEA) and school performance disaggregated data report.”

• Now choose location, test, grade, subj.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


16

Green Book

Find out about getting data from the Green Book and its online brother:

• both in its standard form …

• and its interactive, electronic form.

http://www.ncpublicschools.org/accountability/testing/

Scroll down to “State Testing Results”

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


17

Questions to pose ..

• In your LEA (or school) which grades, courses, SAT results are stronger?

• In your LEA how do schools compare on average EOG, EOC, SAT?

• How does your LEA (school) compare with similar LEAs (schools)?

• What is the effect of what you’ve been trying to improve?

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


18

Current Research

The Problem With “Proficiency”: Limitations of Statistics and Policy Under No Child Left Behind. By Andrew Dean Ho, Educational Researcher, 2008; 37, 351

• “The limitations [of the Percentage of Proficient Students (PPS) statistic] are unpredictable, dramatic, and difficult to correct.” …. From abstract.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


19

Part BFuture Ready Core

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


20

ACT Study

… found that “whether planning to enter college or workforce training programs after graduation, high school students need to be educated in a comparable level in reading and mathematics.”

Ready for College and Ready for Work: Same or Different? 2006. Available on the web.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


21

Future Ready Core - Goals

On September 7, 2006, the NC Board of Ed approved goals: Future Ready Students for the 21st Century.

Note the first goal and the 1st item in that goal.

“Goal: NC public schools will produce globally competitive students.

• Every student excels in rigorous and relevant core curriculum that reflects what students need to know and demonstrate in a global 21st Century environment … ”

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


22

Future-Ready Core Course of Study

The State Board of Education in June 2007 approved the Future-Ready Core Course of Study with the goal that all students would graduate prepared for post-secondary education and careers.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


23

Future Ready Core Timeline

Beginning with the 2009-2010 ninth grade class, all students will be expected to meet the requirements outlined in the Future Ready Core Course of Study.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


24

FRC: Primary Differences.

• An additional math course for those students who previously did not pursue the College/University Prep Course of Study.

• An increase from 20 to 21 for the total number of credits required for graduation.

• A recommended 4 elective credits in a concentrated area.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


25

FRC: Mathematics

4 Credits(Algebra I, Geometry, Algebra II) OR (Integrated Math I, II, III) 4th Math Course to be aligned with the student’s post high school plans A student, in rare instances, may be able to take an alternative math course sequence as outlined under State Board of Education policy. Please see your school counselor for more details.

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


26

FRC - Information

• Information resources are linked from:

www.ncpublicschools.org/gradrequirements

click “Resources.”

PU

BLI

C S

CH

OO

LS O

F N

OR

TH

CA

RO

LIN

A


27

RESOURCES (on dpi web site)

• Course & Credit Requirements Chart

• Course and Credit Requirements Checklists for Parents

• Graduating Future Ready

• Making the Grade – Future Ready Graduates

• Math Sequence

• Questions and Answers

• Student Accountability Standards Brochure

a. effect size can tell a lot! b. future ready core nc accountability conference february 2009

Documents