center for applied linguistics standards, structure, & field testing of access for ells jessica...

Center for Applied Linguistics

Standards, Structure, & Field Testing of ACCESS for ELLs™Jessica Motz

Center for Applied Linguistics

Washington, DC

January 2005

Illinois 28th Annual Statewide Conference for Teachers Serving Linguistically and Culturally Diverse Students

IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 2

Outline

1. Background on the WIDA Project

2. WIDA Standards for English Language Proficiency – language vs. content

3. Structure of the ACCESS for ELLs™ Test

4. Sample Items

5. Pilot Test Results

6. Field Testing Update

7. Training for ACCESS for ELLs™

8. Other Issues/Questions/Discussion


Origins of the WIDA Consortium

Ten states, representing approx. 275,000 ELLs

2002: U.S. Dept. of Education Enhancement Grant Competition

3 Original States:

Wisconsin

Delaware

Arkansas

Early Additions:

District of Columbia

Rhode Island, Maine, New Hampshire, Vermont

Later Additions:

Illinois, Alabama


Multiple WIDA Components

Language Proficiency• English Language Proficiency Standards

• Large-Scale Assessment (ACCESS for ELLs)• Classroom Instruction & Assessment

Academic Content• Alternate Assessment (Alternate ACCESS)

Professional Development• Standards• Assessment• Instruction

Validation and Research


WIDA PartnersState Leadership

Tim Boals, Wisconsin DPI, Chair Steering Committee

Lead Developer (Standards and Project PI)

Margo Gottlieb, Illinois Resource Center

Item Specification Development

Fred Davidson, UIUC

Test Development

Language Testing Division, CAL

Professional Development

Lorraine Valdez-Pierce, George Mason University

Technical Applications (Database, Desire2Learn)

University of Wisconsin - Oshkosh


Centrality of the ELP Standards

ClassroomAssessmentFramework

Large-ScaleAssessmentFramework

English LanguageProficiencyStandards

& PerformanceDefinitions

ModelPerformance

Indicators:Classroom

ModelPerformance

Indicators:Large-Scale


Overall Organization of Standards

Frameworks for Classroom & Large-Scale Assessment (2)

English Language Proficiency Standards (5)

Language Domains (4)

Grade Level Clusters (4)

Language Proficiency Levels (5)

Model PIs are the lowest level ofexpression of the standards

Model Performance Indicators


The WIDA ELP StandardsStandard 1—SI

• English language learners communicate in English for social and instructional purposes in the school setting.

Standard 2— LA• English language learners communicate information, ideas

and concepts necessary for academic success in the content area of Language Arts.

Standard 3—MA• English language learners communicate information, ideas

and concepts necessary for academic success in the content area of Math.

Standard 4—SC• English language learners communicate information, ideas

and concepts necessary for academic success in the content area of Science.

Standard 5— SS• English language learners communicate information, ideas

and concepts necessary for academic success in the content area of Social Studies.


The Levels of English Language Proficiency

ENTERING

BEGINNING

DEVELOPING

EXPANDING

Never LEP

1

2

3

4

5

6

7

Formerly LEP

BRIDGING


Criteria for Proficiency Level Definitions

Comprehension and use of the technical language of the content areas

Extent of language (text or discourse) control

Development of phonological, syntactic, and semantic understanding or usage

ENTERING BEGINNING DEVELOPING EXPANDING BRIDGING

1 2 3 4 5


Large-Scale Standards: SC Reading



Classify living organisms (such as birds and mammals) by using pictures or

icons



Interpret data presented in text and

tables in scientific studies


Assessment Forms

Non-secure form for initial screening (July 1)• One for each grade level cluster - with items at all 5

proficiency levels

• Kindergarten form - individually administered

Secure forms for annual testing • Two (initially: 100 and 200) for each grade level

cluster• Tier A: Proficiency levels 1-3• Tier B: Proficiency levels 2-4• Tier C: Proficiency levels 3-5


Tier Alignment with Proficiency Levels

ENTERING BEGINNING DEVELOPINGEXPANDING BRIDGING

1 2 3 4 5

Annual ACCESS for ELLs

Tier A

Tier B

Tier C


WIDA Steering Committee Administration Decisions

Listening (15%): 20-25 minutes, machine scored

Reading (35%): 35-40 minutes, machine scored

Writing (35%): Up to 1 hour, rater scored

Speaking (15%): Up to 15 minutes, administrator scored


Structure of ACCESS for ELLs

Grade Leveland Tier

K

1-2

3-5

6-8

9-12

A (adaptive)

A B C

A B C

A B C

A B C

100 (roll-out Spring 2005)

999 (used to produce screener)

200 (roll-out Spring 2006)

Listening — group admin, machine scored

Reading — group admin, machine scored

Speaking — individual admin, adaptive, TA scored

Writing — group admin, rater scored

Domains

Forms


Item Creation Process (Fall 2003 – present)

Item specifications drafted

Item writers assembled from nominated ESL teachers in consortium states

Item writers trained using Blackboard distance-learning management software

Item writers submitted items electronically

Items reviewed by External Reviewers, also trained via Blackboard

Items reviewed and revised internally & organized into themes…


Thematic Folders of items arranged onto pilot test forms

Forms piloted in 5 WIDA districts

Pilot analysis & feedback incorporated; items created, revised by ESL teachers

Thematic Folders of items arranged onto field test forms

Forms field tested in 8 WIDA states

Field test analysis occurring presently

For more information on item development process: Jim Bauman’s session

Item Creation Process, continued… (Fall 2003 – present)


Pilot Testing Results


Pilot Test Participation

April – May 2004

5 Districts:

Kenosha & Milwaukee, WI; Chicago & Cicero, IL; Washington, DC

Approx. 1100 students in grades 1-12


Listening Test

Multiple choice

20-25 minutes


Sample Items: Listening, Science 1-2


Science Listening 1-2 P1

PI: “identify objects according to chemical or physical properties from pictures and oral statements”

SCRIPT: “A seed is small. Find the small seed.”



PI: “match objects with their chemical or physical properties from pictures and oral statements”

SCRIPT: “One day the seed will grow into something large, round, and heavy. Find what the seed grows into.”



PI: “identify and group objects according to chemical or physical properties from oral statements”

SCRIPT: “Seeds grow into plants. Find something else that grows.”


Percent Correct on Each Item (Grades 1-2: n = 173)

P1scored P2scored P3scored

0

0.2

0.4

0.6

0.8

1

Mean

0.940.89

0.79


Percent Correct on Each Item (by Tier)

A B C

Tier

0

0.2

0.4

0.6

0.8

1

Mea

n

0.839

0.983 1 0.98

0.589

0.897 0.882

P1scored

P2scored

P3scored


Percent Correct on Each Item (by Grade Level)

1 2

Grade

0

0.2

0.4

0.6

0.8

1

Mea

n

0.931 0.949

0.851

0.713

0.872

P1scored

P2scored

P3scored

IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 31PLEVEL

54321

Mean ITEMDIFF

1000

900

800

700

600

500

400

300

200

GRADELEV

g12

g35

g68

g912

Average Listening Item Difficulty Across Grade Clusters (by Proficiency Levels)


Average Reading Item Difficulty by Grade Level Cluster across P-Levels

1 2 3 4 5

Plevel

0

200

400

600

800

1,000

Mea

n It

emD

iffi

cult

y

GradeLevelg12

g35

g68

g912


Reading Test

Multiple choice

35-40 minutes


Examining Reading Items Across a Strand for the Different Tiers

Language Arts, Reading, Grades 3-5


Sample Item (Field Test): Reading, Lang. Arts, Grades 3-5, Tier C


#11: “To Jessica, he was the best dog in the world.” This sentence shows Jessica’s opinion. Which of the following also shows an opinion?PI (p3): Identify language associated with stating opinions found in fiction or non-fiction text.

#12: When Jessica saw the sign for the lost dog, why did she believe it was Blue?PI (p4): Differentiate between statements of fact and opinion found in various reading selections.

#13: Why does the woman say at the end of the story, “He’s your dog, all right!”?PI (p5): Identify author’s reasons or intent for selecting facts or opinions found in fiction or non-fiction from grade-level language arts text.

Items Tied to Performance Indicators: Tier C Items


Reading Items Adapted for Tier A Simpler text and more graphic support

Items at proficiency levels 1, 2, and 3• For example:

1. Which is Blue?PI (p1): Match labels or identify facts from pictures and phrases.

2. “I know he has white spots.” Which words in this sentence tell you it is a fact?PI (p2): Identify language associated with stating facts found in short fiction or non-fiction text supported by pictures or graphics

3. [Same item from Tier C] “To Jessica, he was the best dog in the world.” This sentence shows Jessica’s opinion. Which of the following also shows an opinion?PI (p3): Identify language associated with stating opinions found in fiction or non-fiction text.


Tier Alignment with Proficiency Levels on Test Forms (L, R, W)

ENTERING BEGINNING DEVELOPINGEXPANDING BRIDGING

1 2 3 4 5

Annual ACCESS for ELLs

Tier A

Tier B

Tier C


Writing Test

Up to 1 hour

4 tasks per tiered form: 1. SI

2. MA

3. SC

4. LA/SS

Model provided to give background and structure for the task


Lang. Arts Writing 6-8 P5 PI: “defend positions or stances using original ideas with supporting details”

Sample Item: Writing Grades 6-8


Standard 1 2 3 4 5

(1) SI 1 1 1 0 0

(2) LA/SS 1 1 1 1 1

(3) MA/SC 1 1 1 1 1

Speaking Test: Adaptive Format


Sample Item: Speaking


Task (Proficiency Level) 1 Example: Grades 3-5

SI Speaking 3-5:

P1 “Respond to WH- questions”

First let’s talk about things people do outside. This is a picture of people in a park. I’m going to ask you some questions about this picture.

Q1: (Point to TREE) What is this?

Q2: (Point to BALL) What is this?

Q3: (Point to DOG) What is this?

Q4: (If necessary) What else do you see in this picture (OR) What other things do you see in this picture?


Task (Proficiency Level) 2 Example:

SI Speaking 35:

P2 “Ask and respond to questions”

Now listen carefully. I’ve just asked you some questions about this picture. Now I want you to ask me some questions about it. (OR) Pretend you are the teacher and want to ask me some questions about this picture. For example, you could ask me, “Where are the people?” OK?

Q1: (Point to BOY ON BIKE) What do you want to know about him? (OR) Ask me a question about him.

Q2: (Point to PICNIC TABLE) What do you want to know about this? (OR) Ask me a question about this.

Q3: What other things do you want to know about his picture? (OR) What’s another question you can ask me about (anything in) this picture?(Answer student’s question.)


Task (Proficiency Level) 3 Example:

SI Speaking 35:

P3 “Exchange personal information”

Now let me tell you something about these children. (Point to CHILDREN PLAYING CATCH) Their names are Alex and Leticia. They like to play catch.

Q1: Do you like to play catch?

Q2: (If “Yes”) What else do you like to do?

Q3: (If “No”) What do you like to do?

Q4: What do you like about __________? (OR) Tell me something about ___________.

Q5: (If necessary) Tell me more.


Field Testing: Overview


Field Test Participation

Request:• Group-administered sections, 600 per form

(minimum 400 per form)

• Individually-administered sections, 225 per form

Includes 8 WIDA states

Proportional representation of states (approx. 5.5%)

Approx. 8700 students total (~3500 from Illinois)


Initial Field Test Analyses

Concurrent calibration of MC items (Rasch)

Separate scale construction for each domain

Raw score to scale score on the screener

NOTE: Steering Committee determined the following weights when a single level designation is needed:

Writing 35% (e.g 2)

Reading 35% (e.g. 2)

Listening 15% (e.g. 5)

Speaking 15% (e.g. 4)

Composite = 2.75 (not yet 3)


Sample Feedback received to date

Input from test administrators and coordinators:

• Test Length: The test is taking longer to administer than anticipated.

• Test Difficulty: The test is more difficult than anticipated.

• Grades 1-2 Test: Too challenging for fall first graders; these students should take K test.

• Grades 9-12 Test: There is not enough authentic literature on the Reading Test.

CAL and DPI responded to these concerns via e-mail and the D2L (online training) discussion board.


Timeline for Development and Implementation

Field Testing (incl. English speakers): September - January 2005

Analysis of Field Test Data: November 2004 - January 2005

Setting Cut Scores: January 2005

Spring Roll-Out of Form 100 (AL, ME, VT): February - May 2005

Practice Items for Fall Roll-out Available: March 15, 2005

Screener Available: July 2005

Fall Roll-Out of Form 100 (IL, DC): Fall 2005


Training for ACCESS for ELLs


Training for ACCESS

August 2004: SEA WebEx conference on logistics and data needed for Field Test

September 2004: LEA Coordinators trained online via Desire2Learn at UW – Oshkosh

September – November 2004: Field Test Administrators trained online via D2L

December 2004 – January 2005: Speaking Field Test Administrators trained in-person (DC, WI)

March – May 2005: Operational Test Administrators giving spring roll-out test trained


WIDA Field Test Administrator Workshop


Operational ACCESS Test Administrator Training: Possible Scenarios

Beginning March 2005

2-hour total online training for all sections of the test

Additional, as-needed training on scoring the Speaking test, with online or live (state/district) trained facilitators.

For more information and updates:

www.wida.us


Other Issues/Questions/Discussion

center for applied linguistics standards, structure, & field testing of access for ells jessica...

Documents

largescale standards

wida standards

irc conference

ells jessica motz center

field testing update

ells test

largescale slide

applied linguistics