1% + 2% = ______________: adding up what we know & don’t know about alternate assessments...

1% + 2% = ______________:ADDING UP WHAT WE KNOW & DON’T KNOW ABOUT ALTERNATE ASSESSMENTS

Stephen N. Elliott, PhD Gerald Tindal, PhDVanderbilt University University of Oregon

Alternate Assessments

Key

Defi

nitio

ns

AA-AAS or Alternate Assessments of Alternate Achievement Standards (capped at 1% of total population)

AA-MAAS or Alternate Assessments of Modified Alternate Achievement Standards (capped at 2% of total population)

Elliott & Tindal / CEC 2010

2

Elliott & Tindal / CEC 2010 3

Grade Level Content

Standards

Grade Level Content

StandardsModified

Performance Indicators

Grade Level Content

StandardsAlternate

Performance Indicators

Large-ScaleAchievement

Test With orWithout

Accommodations

ModifiedAlternate

Assessment

Alternate Assessment

Rating Scales, Portfolio, or

Performance Assessment

Alternate Proficiency Standards

Modified Proficiency Standards

Grade Level

Proficiency Standards

AYPReport

for All

Students

DIFFERENT ACHIEVEMENT STANDARDS / 1 ACCOUNTABILITY SYSTEM

(2% Capped)

(1% Capped)

Elliott & Tindal / CEC 2010 4

AchievementTest

ModifiedAchievement

Test

Performance Tasks,

ClassroomWork, &

ObservationsCollected by

Teachers

Grade Level Content Standards

Modified Grade Level Content Standards

Extended Content Standards

Standard 4:Geometry & Measurement1. Identify and describe the basic properties of figures (e.g., 2 or 3 dimensionality, symmetry, number of faces, types of angles).

Standard 4:Geometry & Measurement1. Identify and describe the basic properties of figures (e.g., 2 or 3 dimensionality, symmetry, number of faces, types of angles).

Standard 4:Geometry & Measurement1. Identify basic properties of 2 dimensional figures (e.g., number of sides, types of angles).

2. Identify basic properties of 3 dimensional figure.

Standard 4:Geometry & Measurement1. Identify basic properties of 2 dimensional figures (e.g., number of sides, types of angles).

2. Identify basic properties of 3 dimensional figure.

Standard 4:Geometry &Measurement

1. Count the number of sides of a square.

2. Count the number of sides of a triangle.

Standard 4:Geometry &Measurement

1. Count the number of sides of a square.

2. Count the number of sides of a triangle.

So

urces o

f Evid

ence

Definitions of Approaches for 1% Alternate Assessments

Elliott 2009 / SpEd 3825

5

Portfolio Assessment is an organized collection or documentation of student-generated or student-focused work typically depicting the range of individual student skills. [30 states]

Performance Assessment is a task or series of tasks requiring a student to provide a response or create a product to show mastery of a specific skill or content standard. [21 states]

Comprehensive Rating Scales of Achievement are rating scales anchored by descriptive rubrics for quantifying teacher judgments of students’ knowledge and skills based on repeated direct and indirect observations situated in a number of school settings. [13 states]

Multiple Choice/Constructed Response – [6 states]

Note. 2% AAs thus far have all used item formats like general achievement tests.

The students?

1%

+ 2

% =

-----

1% focus on “authentic” skills that are integrated across domains and have potential for use outside of school

2 % take a test that is sensitive to significant lack of basic skills yet is on grade level

In both groups, there is a need to distinguish access versus target skills (within students and over time)


6

The students?

1%

+ 2

% =

-----

Students with disabilities who have been determined unable to perform at or above the proficient level on a grade-level general achievement test. 1% eligible students have a severe cognitive

disability, require modified instruction and curriculum, & extensive support for skill generalization. (50 states)

2% eligible students are those “whose disability has prevented them from achieving grade-level proficiency and who likely will not reach grade-level achievement in the same timeframe as other students.” (DOE Regs. 34 CFR Part 200 Title 1 & NCLB) (14 states)


7

Assessment Challenges?

1%

+ 2

% =

-----

Standards and Standardization Alignment of items and grade level

standards Tension between the mandates of

standardization and validity evidence that is needed

How to compare assessment approaches: What kind of evidence?

How to avoid conflating independent variables with dependent measures?


8

Assessment Challenges?

1%

+ 2

% =

-----

Eliminating construct irrelevant variance so measurement is of targeted knowledge and skills rather than access skills. Teachers should ensure that for both 1% & 2%

assessments students have had an opportunity to learn the assessed curriculum.

For many 1% assessments, students require significant support to response to a question or item. The support must function as an acceptable accommodation or it is likely to undermine the validity of the test score inference.

For many 2% assessments, students are confronted by items with substantial extraneous information and thus create cognitive load problems.


9

Percent Proficient?

1%

+ 2

% =

-----

Fixing time and level of achievement while using status not growth

Moving from interval scores to nominal categories: Is there lost meaning?

Articulating different test types that tap into the same construct (Standard Test, AA-MAS, and AA-AAS)


10

Percent Proficient?

1%

+ 2

% =

-----

Nearly 75% of students taking 1% AAs across the country are deemed Proficient, while less than 30% of students with disabilities taking the general assessment are deemed Proficient. Clearly the meaning of proficient and the relative rigor of the associated cut-scores are different.

It is too early to report on comparable proficient rates for the 2% AA.


11

Consequences?

1%

+ 2

% =

-----

Advocacy and Adequacy: Balancing opportunity and appropriateness

Better measurement may be in the offing (e.g., accommodations and universal design)

Focus on post-secondary outcomes and the need for more vertical articulation of outcomes across grades


12

Consequences?

1%

+ 2

% =

-----

Most1% AAs require significant teacher involved in the assessment. This takes time, but in many cases seems to result in a more instructionally relevant assessment. Teachers also often receive valuable feedback about students long before their assessment is reported for accountability purposes.

The development of items for 2% AAs has provided valuable information about ways to improve items for the general achievement test.

Inclusive assessments have many positive consequences for students and teachers.


13

Technical Issues?

1%

+ 2

% =

-----

Measurement of learning with the same yardstick Different rules of engagement? Different types of evidence?

Conflation of process and outcome variables (how a test is taken with the outcome of the test)

Consideration of validity as a unitary construct


14

Technical Issues?

1%

+ 2

% =

-----

Alignment of knowledge and skills tested on both 1% and 2% AAs with grade-level content standards.

Reliability of scoring of 1% AAs by independent raters.

Developing meaningful Achievement Level Descriptors and associated cut scores for Proficient determination.

Difficulty measuring growth with 1% AAs.


15

The Future?

1%

+ 2

% =

-----

Informing policy with evidence about improved programs as well as improved performance

Moving to more formative assessments that integrate various initiatives (e.g., Response to Intervention) into large-scale assessments

Disseminating fugitive literature from what we know


16

The Future?

1%

+ 2

% =

-----

Revisions of both 1% and 2% AAs to ensure alignment with Common Content Standards.

Increased emphasis on characterizing students’ academic progress / growth.

Increased measurement of achievement using multiple-measures (e.g., interim assessments, formative assessments).

Limited number of states implementing 2% AAs due to costs.


17

Concluding Thoughts!

1%

+ 2

% =

-----

The same rules for collecting validity evidence are needed, irrespective of populations or assessment approaches

Growth should be the coin of the realm with models of accountability fostering not restricting this focus on measurement

Inclusive assessments such as 1% and 2% AAs have advanced accountability practices for students with disabilities.


18

Proficient Performance: What Does it Mean & How is it Achieved

Dianna Carrizales, Brad Lenhardt, & Nancy Latini

Good Scores are Hard to Get: Technically Speaking

Aran Felix, Kim Sherman, Gerald Tindal, & Naomi Zigmond

Alternate Assessments’ Contributions to Better Classroom Instruction and Testing

Stephen Elliott & Ryan Kettler

Remaining Sessions in the Alternate Assessment Strand

19


Stephen N. Elliott, PhD, Vanderbilt [email protected]

Gerald Tindal, PhD, University of Oregon

[email protected]

Thank you very much for joining us!

20

1% + 2% = ______________: adding up what we know & don’t know about alternate assessments...

Documents