1

State Special Education Outcomes: Results of a 2009 Survey of States

Jason Altman – NCEOMari Quenemoen – NAAC

TASH Annual Conference – Nov. 19, 2009

2

Introduction

• It is important that state large-scale assessment systems be both inclusive of and fair to all populations including students with disabilities

• A 2009 survey explored these issues as a whole and paid special attention to state practices in assessing students who may participate in an alternate assessment option

3

Survey

• All 50 states returned the 12th survey of state directors of special education and state directors of assessment by the National Center on Educational Outcomes (NCEO)

• The survey was disseminated to respondents electronically in the winter of 2009

• NCEO returned surveys for verification and, upon receipt, entered, reviewed, and analyzed the data

• Results of the eleventh survey were presented at the 2008 TASH annual conference

4

Presentation Purpose

• The purpose of this presentation is to provide a snapshot of new initiatives, trends, accomplishments, and emerging issues in the large scale assessment of students with disabilities

• Presenters will pay special attention to issues surrounding the testing of students with disabilities who may be assessed using an alternate assessment

Apples to Oranges or Apples to Apples

6

General Issues

• States reported that assessment validity and test design/content were areas of much success

• Issues related to English language learners with disabilities and the performance of urban schools were often selected as challenging

• States appear to have a very mixed viewpoint on the AA-MAS (7 respondents in each category)

• On the other hand, respondents appear to be strongly entrenched on the successful side of the ledger as related to reporting and monitoring

7

Regular Assessment IssuesAccommodations

• More than four in five states reported monitoring accommodations use in 2009

Directly observing test administrations

Interviewing students, teachers, and administrators

Conducting desk audits

Sending teams into schools on a scheduled basis

Sending teams into schools on a random basis

Sending teams into schools on a targeted basis

Completing online record reviews

Other

We do not monitor accommodations use

0 5 10 15 20 25

22

15

14

13

10

10

6

10

9

7

2

1

0

3

0

0

1

1

Unique States Regular States

Number of States

8

Regular Assessment IssuesAccommodations

• Most respondents also reported that accommodations cause some difficulty on test day in schools and districts within their state (n = 41)

Arranging for trained readers/interpreters

Ensuring proctors give correct accommodations

Scheduling substitute providers of accommodations

Training proctors in providing accommodations

Ensuring the ordering of special test editions

Ensuring that accommodations are recorded

Arranging for and checking equipment

Ensuring accommodations for makeup assessments

Other

We have ensured accommodations are carried out

0 2 4 6 8 10 12 14 16 18 20

19

17

14

12

11

6

5

1

9

9

3

4

3

3

3

2

4

1

3

0

Unique States Regular States

Number of States

9

Alternate Assessments Based on Modified Academic Achievement Standards

• In April 2007, new No Child Left Behind regulations on AA-MAS were finalized

• States have the option of developing alternate assessments based on modified achievement standards (AA-MAS)

• At the time of the 2007 survey of states we found that five states already had an assessment in place, 32 states were considering modifying an existing grade-level assessment and another twenty-five were considering developing a new assessment

• In the past two years, some states have further refined their motivations for moving in this assessment direction, while other states were instead making efforts to improve the assessments that they already offer

10

Alternate Assessments Based on Modified Academic Achievement Standards

• Since 2007, 14 states have decided not to develop of AA-MAS, while 12 states were still in the process of making their decision

• Of the 24 that have developed, or were developing, an AA-MAS, nine have already given the assessment and analyzed the data, one for the first time in 2008-09

8

1

15

12

14 We have given the as-sessment, and analyzed dataWe will give this as-sessment for the first time in 2008-09We are currently in the process of developing an AA-MASWe are currently research-ing the need for an AA-MASWe have decided not to develop at this time

11

Alternate Assessments Based on Modified Academic Achievement Standards

• States that were developing or have developed their AA-MAS were three times more likely to modify an existing grade-level test rather than design an entirely new test

5

15

2

26

2

Design an entirely new testModify an existing grade-level testOther, please explain belowResearching the Need or Not De-velopingNo Response

12

Alternate Assessments Based on Modified Academic Achievement Standards

• Respondents have been asked on the past two surveys about the changes they envisioned making when modifying existing tests

• Findings were similar to those found in 2007 save for a less frequent response to use of non-traditional items (keeping in mind that many more states have decided not to develop the AA-MAS at this time

Simplify vocabulary

Reduce the number of total test items or time

Use shortened or fewer reading passages

Use fewer answer choices

Include only multiple choice questions

Use non-traditional items or formats

0 5 10 15 20 25 30 35

16

14

13

12

6

3

26

26

29

23

11

14

2007 2009Number of States

13

Alternate Assessments Based on Modified Academic Achievement Standards

• States reported a variety of strategies and methods in determining content targets and blue prints for their AA-MAS

Same specifications as for regular assessment

Stakeholder panels

Review of literature on teaching at-risk learners

Special education curriculum literature

Teacher survey

Test company provided

Consultant developed

Other, please explain below

Researching the Need or Not Developing

No Response

0 5 10 15 20 25 30

17

11

7

6

3

3

1

2

26

2

Number of States

14

Alternate Assessments Based on Alternate Achievement Standards

• Seven states (HI, ID, MS, NE, NH, NV, and UT) and three unique states (Guam, PR, and RMI) are currently in the process of revising their AA-AAS

• Trend analysis from 2005 and 2003 data

15

AA-AAS Topics

• Test formats• Content alignment• Scoring methods• Rubric analysis• Methods for determining achievement levels• Scorers

16

AA-AAS Formats•40% of states use a portfolio or body of evidence (40% of these are standardized)•36% use a standardized set of performance events/tasks (44% of which require the submission of evidence)•16% use a multiple choice test

17

AA-AAS Content Alignment• Complete transition to alignment with academics

• Extended/expanded academic content standards (27 states) or grade level academic content standards (16)

• No states align AA-AAS to functional skills, and IEP teams no longer determine the content of the test for individual students

18

AA-AAS Scoring Procedures

Rubric

Points Assigned on a Rating Scale

Number of Items Correct

Reading Rate or Accuracy

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

80%

16%

10%

4%

74%

32%

24%

8%

66%

16%

38%

4%

2009 2005 2003

Percentage of States

19

Outcomes Measured by Rubrics• Primarily skill/competence, level of assistance, and alignment to academic content

• Trend away from scoring other factors

20

Outcomes Measured by Rubrics (2005 and 2009)

Level of Assistance

Skill/Competence

Alignment with Academic Content Standards

Ability to Generalize

Number/Variety of Settings

Degree of Progress

Appropriateness

Participation in General Education Settings

Support

Staff Support

Self Determination

Social Relationships

Parent Satisfaction

0 5 10 15 20 25 30

2009 2005

Number of States

21

Outcomes Measured by Rubrics ( States and Unique States 2009)

Level of Assistance

Skill/Competence

Alignment with Academic Content Standards

Ability to Generalize

Number/Variety of Settings

Degree of Progress

Appropriateness

Participation in General Education Settings

Support

Staff Support

Self Determination

Social Relationships

Parent Satisfaction

0 5 10 15 20 25 3024

25

15

10

10

6

8

5

6

5

4

3

1

3

5

4

2

1

2

2

1

Unique States States

Number of States or Unique States

22

Methods for Determining Achievement Levels

Contrasting groups

Body of work

Judgmental policy capturing

Reasoned judgment

Bookmarking

Modified Angoff

Other

0% 10% 20% 30% 40% 50% 60% 70%

12%

8%

4%

58%

23%

20%

37%

17%

29%

12%

10%

2009 2005

Percentage of States

23

Who Scores the AA-AAS?

test company contractor

student's special education teacher

teachers from other districts

state education agency

teachers in the same district as student

members of student's IEP team

university contractor

other

currently in revision

0 5 10 15 20 25

21

12

10

4

4

4

3

3

7

1

1

1

1

1

1

3

Unique States States

Number of States or Unique States

24

Thank you! For more information…

National Center on Educational OutcomesUniversity of Minnesota612-626-1530http://www.nceo.info

Jason Altman:altma014@umn.edu

National Alternate Assessment CenterUniversity of Kentucky859-257-7672 http://www.naacpartners.org

Mari Quenemoen: mari.quenemoen@uky.edu