1 utah academic language proficiency assessment ualpa assessment directors als directors it...

1

Utah Academic Language Proficiency Assessment

UALPA

Assessment DirectorsALS DirectorsIT Directors

Judy Park, Assessment & Accountability Director John Brandt, Information Technology DirectorDeborah Swensen Development Coordinator

Utah State Office of EducationSeptember 13, 2006

2

UALPA Update

Testing window begins Oct. 30.

Double testing with IPT & UALPA – only for students new to the U.S.

3

UALPA Update

Verify and correct IPT scores from 2005-2006

● Early Fall 2006, USOE data warehouse will produce a new IPT data table for release as part of its research file/disk collection. In August of 2007 this will become UALPA data.

● It will include a detailed LEP record for each LEP student in the 2005-06 school year.

● Data can be reviewed by the LEA to determine if any changes need to be made to a student’s LEP data.

● Changes must occur before the Fall clearinghouse reload (approx. November).

4

UALPA Update

● Testing all ELL

50% tested and answer documents returned to USOE by February 28

50% completed by April 30 (all testing is now complete)

● Scanning and scoring by MP in April and May

5

Linking IPT and UALPA

● Requirement Finding a way to measure “progress” between 2006 (IPT) &

2007 (UALPA)

● Problem Different tests

Multiple forms of IPT IPT and UALPA

Different students Different proficiency levels (A-E vs PEIAF) Different test constructs (social to academic language)

● Linking Options “Double-test” a sample of students this year

OR Statistically link test scores across years

6

Double-testing● Advantages

Able to have data for the same students on the two tests

If the sample is representative, can be a strong link

● Disadvantages Considerable fatigue potential Incredibly difficult to obtain a sample representative

enough for valid generalizations Multiple forms of IPT Additional expense of resources (time & money)

7

Statistical Linking

● Advantages Based on existing data sources

non-intrusive and lower cost

Includes all students - sampling not an issue Familiar methodology

● Disadvantages Not as strong as traditional equating methods Slightly harder to communicate

8

The Bottom Line

● There is no perfect approach● The essential question (i.e., how many students

in each school made progress?) can be answered with either method

● Statistical linking has fewer threats to validity unless the double-testing method is almost perfect

● Double-testing presents a considerable burden

TAC and PAC recommends Statistical Linking

9

Statistical Linking

● For 2006, determine the distribution of students in the proficiency levels Total % of students at each level For example, A=10%; B=30%; C=20%; D=15%; E=25%

● For 2007, UALPA scores will then be distributed into the same proficiency levels with the % of distributions as 2006

● Since the eventual UALPA and current IPT proficiency levels are not the same, the proficiency levels of A, B, C, D, E will be retained for 2007 (it is like converting between Celsius and Fahrenheit)

10

The 2007 UALPA scores with the same distribution as the 2006 IPT scores.

2006A B C D E10% 30% 20% 15% 25%

2007

A B C D E

10% 30% 20% 15% 25%

11

Statistical Linking

● Scores on 2007 UALPA will be translated and reported in terms of the IPT overall categories (A, B, C…)

● Progress will the based on the number/percent of students moving proficiency levels.

12

2007

Transition UALPA Score

A,B,C,D,E

2006

IPT Score

A,B,C,D,E

2008

UALPA Score

P,E,I,A,F

13

Validity of the Link● Will be evaluated by replicating the

statistical link for specific demographic categories: Gender Ethnicity Grade level Native Language

● This replication should be similar for all groups (using an error band) for 2006 and 2007

14

Reporting 2007 UALPA Scores● Overall scores will be reported using a

UALPA scale score and the IPT proficiency levels.

UALPA Scaled Score200-----------------------------------------------------------------------------------300

A B C D

15

Reporting 2007 UALPA Scores● Modalities will be reported using raw

scores

● Raw scores will include a band that will provide meaning and interpretation of the raw score

● e.g., Reading=13 of 20 pts; slightly above average for students scoring just at the B/C cut

16

Reporting 2008 UALPA Scores

● The 2008 UALPA will be an improved test● A standard setting will be held to establish the

UALPA proficiency levels of P,E, I, A, F.● 2008 UALPA scores will be reported using the

UALPA scale score and the UALPA proficiency levels

● Scale Score Results will be immediate● UALPA proficiency levels will be available after

standard setting● Progress can be determined using the 2007 and

2008 UALPA scale scores

17

2006 IPT Proficiency Levels

2007 IPT/UALPA Transition Proficiency Levels

Progress

2007 UALPA Scaled Scores

IPT to UALPA Plan

2008 UALPA Proficiency Levels 2008 UALPA Scaled Scores

Progress

2009 UALPA Proficiency Levels 2009 UALPA Scale Scores

Progress

18

2007 UALPA Progress● Progress will the based on the

number/percent of students moving in the proficiency levels from one year to the next

● AMAO Targets will be revised to appropriately show progress

Juan, 2006 at B who in 2007 becomes a C (has progressed)

Maria, 2006 B; 2007 B (has not progressed)

20

UALPA DocumentsPre-prints

● 2006-07 school year: A UALPA pre-print file is required from each LEA prior to UALPA testing.

● Bar coded labels will be returned to each LEA

● Answer Booklets and Test Documents acquired from USOE Assessment and Accountability

21

UALPA DocumentsPre-prints

● Labels must be applied to the UALPA multiple page test/answer documents

● Students who are new to the country will need to be given an SSID number before the UALPA is administered to them.

● For new students, test documents must be hand-bubbled as they are not identified as LEP students in the LEA SIS.

22

UALPA DocumentsPreprints

● The pre-print file definition will be the standard USOE specification also used for the CRT, DWA, Iowa and UBSCT.

Districts may use the same pre-print upload website and process.

Returned labels can be sorted in a variety of ways such as by school and/or grade upon district request.

The scheduling of the pre-print uploads and the students included in the file will differ from processes done for other testing.

23

Pre-print Model

24

UALPA DocumentsPre-print Request

Two possible procedures

25


● Option 1: Request and stockpile labels one month at a time

On a periodic basis (e.g., once a month) LEA submits a pre-print file to USOE containing records of returning LEA students who are to be tested. The pre-print file must be received by the USOE at least 10 working days previously. USOE will pre-print and FedEx the labels back to the LEA (two copies through February of 2007).

LEA receives labels, applies labels to the appropriate test documents and administers the tests.

First year ELL’s, not in the SIS system, will have to have answer docs hand bubbled.

Impact on LEA IT: The LEA’s SIS must be able to produce a pre-print file of just those LEP students that they want to assess. Likewise the SIS may need to be able to produce a list/report of those same students to aide in applying labels.

Option 1 is recommended as an ongoing practice.

26


● Option 2: Request and stockpile labels once a year

LEA submits pre-print file prior to school year’s testing containing records of ELL. The file must be received by the USOE at least 10 working days prior to the first testing. USOE filters out all students except those indicated as being LEP students. The USOE prints and FedEx or UPS the labels back to the LEA.

Upon receipt of labels, LEA’s designated staff applies labels to appropriate test documents and administers the tests.

First year ELL’s, not in the SIS system, will have to have answer docs hand bubbled.

This option easier technologically, but puts a bigger burden on LEAs where there is a large number of ELL. Not only will LEAs need to hand bubble a first year ELL students, but any student that moves into the district after labels are requested.

Impact on LEA IT: The LEA’s SIS will need to be able to produce a list/report of students having a LEP identifier on a monthly basis.

27

UALPA Documents Answer Document

● Answer sheets are booklets for all grade spans

K Teacher records for all 4 modalities

1-2 Teacher records speaking modality scores. Students complete other three modalities in consumable booklet.

3-6, 7-8, 9-12 Teacher records speaking modality scores. Students complete other three modalities on answer sheet.

28

Answer DocumentSpecial Codes

Marking the Special Codes Box For not participating mark:

● “Officially withdrawn from school” if the student is no longer enrolled in the school.

● “ELL – 1st year before 4/15” if the student is an ELL student and enrolled before April 15 of the current school year.

● “ELL – 1st year on or after 4/15” if the student is an ELL student and enrolled on or after April 15 of the current school year.

● “Home school” if the student is educated in a home school setting.

● “RT” if the student refused to take the test. This should be marked in the "Office Use Only" section for each

modality the student refused to respond to.

● “Unknown student” if the answer sheet was pre-printed for a student who cannot be identified as ever having been enrolled in the school.

29

Answer Document Special Codes

Marking the Special Codes Box

For non-standard participation mark:● Accommodated (ELL, IEP, 504)● Modified (IEP Team)● UAA, Utah Alternate Assessment (IEP Team)

OTHER● ELL First Year in U.S. Before April 15

If Accommodated is marked – please identify specific accommodations (up to 3) in the new Accommodations box.

30

RT FL

UALPA DocumentsAnswer Document

● Test administrator/proctor will mark the following

circles for each modality where applicable if: student/parent refuse testing (mark RT)

student reaches frustration level (mark FL)

31

Reporting New Students - UALPA

Each LEA will need to record new students being tested on the UALPA in a spreadsheet since not all their demographics (e.g. parent language) will be in an all-student file.

New students must be given a SSID for identification purposes, before the UALPA is administered.

Each LEA manually records demographic and IPT scores for new students who are placed as LEP students in the ’06-’07 school year: The spreadsheet format will be specified by Measured Progress.

The columns for the spreadsheet must include at least: student SSID, name, LEP code (A, B, C, D, E), grade, home language, and parents’ language, gender, ethnicity and instruction type.

32

Reporting New Students - IPT

● LEA submits IPT spreadsheet of new students to USOE

● LEA must maintain the spreadsheet on new LEP students as there is no USOE warehouse record for these students in April and there will not be until the end-of-year (July) load of the clearinghouse data.

● Separate LEA spreadsheets will be combined into a state file and student records matched to their UALPA test record.

● In June 2007 Measured Progress will provide student level UALPA test records, identified by SSID, for import into the USOE data warehouse where it can be joined to other education records for that student.

33

Reporting New Students

SSID Name IPT level Ethnicity Gender Primary Lang.

Parent’s lang.

Grade Instruction type

34

Scoring and Scanning2006-07

● For the 2006-07 UALPA administration, Measured Progress will Score and Scan the test.

35


● Beginning in 2007-08, USOE will be responsible for the scanning, scoring and reporting process.

LEAs will need to submit batches of test documents frequently to ensure that a student’s scores are received back at the LEA in a timely manner. If received before noon on Friday, USOE Computer Services will score and report test batches by the end of the day they are received.

USOE Computer Services will make scores available electronically, in printable report format, in Excel spreadsheets or as text files.

36


● For each scanning/scoring run a standard USOE all-students file is required File must include all the students enrolled in the LEA

during the time period tests to be scored were administered.

This file uses the same file definition as the pre-print file Due to the frequency of submissions, the LEA may just

include returning ELL students and students being tested for ELL placement in these all-students files.

37


● Impact on LEA IT: The LEA’s SIS must produce frequent all-student files Files may need to be limited to just ELL students Recording returned scores (depends on LEA needs

and operating procedures) SIS will also need to have an import program for scores and a

place in its database to store those scores

OR LEA staff will use a SIS data entry Window through a manual

process to enter the data one student at a time.

38

Scoring Model

39

Code Conversions

Conversion of codes from IPT to UALPA must be done by the beginning of the 2007-2008 school year. Codes being converted: IPT (A, B, C, D, E) to

UALPA (P, E, I, A, F). Each LEA SIS must have a conversion program run

to cross-walk at least all current LEP level codes from the old IPT code set (A,B,C,D,E,O) to the new UALPA set. The program will convert A to P, B to E, C to I, D to A, and E to F. More information will be provided to LEAs after the scoring of the ’06-’07 UALPA

40

Code Conversions

Options:

Convert LEP codes in all database fields to the new code set allowing all the B codes in the old set to become E codes in the new set.

Do not convert the historical codes, so one can explicitly tell what code was used in the years prior to 2006-07. Enter new codes in for students, upon receipt of UALPA results

41

Reporting

2006-2007 school year reports still need to be defined by USOE Assessment and Accountability section with input from the UALPA Advisory Committee, TAC, PAC and Contractor.

Future years: The UALPA score/results files will all be electronic for timeliness. This applies to reports whether they are printable or are to be imported into an SIS database.

42

Clearinghouse submissions (Districts to State)

● The clearinghouse requirements for LEP data will not change except: Oral, Reading and Writing Grades will be removed and

the LEP code will come from the P,E,I,A,F,O code set (beginning 2007-08) .

The new LEP proficiency level code submitted via the clearinghouse will only be used to indicate whether or not the student is an LEP student.

The LEP code determined by the UALPA test and stored in the USOE Assessment UALPA database is the authoritative LEP code not the code reported via the clearinghouse.

43


● By 2008 the UALPA database LEP code will also be used to validate the LEP Instruction Type code reported in the clearinghouse record.

● In the spring of 2007, the Assessment UALPA database may not have results of all UALPA tests in time for the clearinghouse Instruction Type validation in May and June. In this event the USOE will depend on the clearinghouse LEP level code for Instruction Type validation.

44


After the data populates the Assessment UALPA data, any student found by the system to have an Instruction Type without a legitimate LEP code (P,E, or I), the system will set that Instruction Type code field to blank.

If the LEP code of the student is P,E or I and there is no Instruction Type, a default Instruction Type will be inserted into that field.

45

Materials

Requests: Use the order form to request necessary testing materials.

Remember: The UALPA will be changed for improvements, so do not over order. Also, students will not be testing at the same time, so you will not need one for every student.

Flip charts are one per administrator (not one per student)

Materials are to be stored at the District level.

46

Materials

Packaging:As per district requirements. There are no predetermined distribution sets

After all tests are completed:Districts will sign off on administration and secure destroy testing materials.

47

Training

October 5th - 8:30 to 12 (Salt Lake City) October 19th - 2pm – 5pm (Nebo)

October 25 - 8.30 to 12 (Richfield)

At this time there has been no interest shown in this last training place or time. If there are no requests today, then this one will be canceled.

Training is for district teams of 3-4 people. Trained district members will be able to train

additional personnel within their district.

48

Assessment Directors Meeting

49

September State Board Meeting

● Secondary School Completion and Diplomas R277-705 Amendment

Adult Students not required to take UBSCT

No provision for “upgrading” diploma

Final approval October 6 Board Meeting

50


● Private School, Home School, Electronic High School (EHS), and Bureau of Indian Affairs (BIA) Student Participation in Public School Achievement Tests R277-604-1

● Approved

Private School Students MAY participate Payment in advance for assessment costs to USOE Payment in advance for administration costs (LEA) Participation under the supervision of LEA

51



● Approved

Home School Students SHALL be allowed to participate Must be enrolled in at least one course in public

school Costs the same as public students Participation under the supervision of LEA

52



● Approved

Electronic High School students MAY participate Must be enrolled in at least one course in public

school Costs the same as public students Participation under the supervision of LEA

53



● Approved

Bureau of Indian Affairs (BIA) students Costs the same as public students Participation under the supervision of LEA

54


● Assessment Participation and Accommodations Policy

Approved Final Copy available at www.usoe.org

http://www.usoe.org/

55

CRT Testing Consistency● Classroom sets of booklets

Test booklets may be compromised Decreased bulk and costs

Or● Test booklet for each student

Supported by best practices Students can practice in tests what they have learned

in the classroom (e.g. highlighting key words, etc.) Test administrator proctors must make sure that each

student receives his/her own booklet on subsequent test days

Greater cost

56

UBSCT

● Beginning October 2007 October Administration will be pre-equated Quick return of results

57

UBSCT● 2006 October Administration

Pre-printed answer sheets Data from February 2006 Results

● Test Administration Manual Not posted to website Distributed with all other materials

● Site Coordinator’s Preparation Guide Posted to website Distributed with all other materials

58

Item Development

59

Guiding Rules

● Pose a clear question or problem in the stem and contain as much of the item as possible.

● Write it as a question.

● Avoid use of tricky and ambiguous questions.

● Avoid use of negatives such as none or not. If they must be used they will be bolded.

60

Guiding Rules

● Use alternatives that are grammatically related to the stem.

● List options in ascending or descending order.

● Align questions to each core.● Preferably put the word in the stem

when testing for knowledge of a term and place alternative definitions in the response alternatives.

61

Guiding RulesExample:

Which definition identifies a cube? A. a shape with three sides.

B. a repeating pattern of circles.

C. a shape with 6 square faces.

D. a location on a map.

● Use appropriate vocabulary

Age & Core Italicized words

62

Item Writing Baird, H. (1997). Performance Assessment for Science Teachers. SLC, UT: USOE.

Rate the following questions:1. An acute angle is

A. greater than 90 degrees.B. exactly 90 degrees.C. less than 90 degrees.

2. What is the measure of an acute angle?A. greater than 90 degrees.

B. exactly 90 degrees.C. less than 90 degrees.

3. Bryan and four friends are going to the Jazz game. Each ticket cost $15.00 plus popcorn and treats are $4.00. How much will it cost for them to attend the game?

63

Item WritingRate quality of the following stems:4. Which of the following is not a right angle?

5. Which of the following is not a right angle?

Notice the difference in the distracters: 6. How many legs do spiders have?

A. 6 A. 3B. 4 or B . 4C. 3 C. 6D. 8 D. 8

64

Do’s and Don’ts of Item WritingAvoid use of wordy stems.

Avoid use of verbal clues such as a, an.

DO NOT use response alternatives such as "none of the above," "none of these," "both (a) and (c) above," or "all of the above.“

Avoid content that adversely or advantages a specific group of students.

65

Evaluate ItemsAn equation will always have an:

A. divide symbol.B. subtract sign.C. equal sign.D. multiplication fact.

What symbol is present in every equation?A. Addition signB. Equal signC. Subtraction signD. Multiplication sign

66

Creating Distractors● The distractors should be of equal

length or parallel.

● Each distractor should be independent so as not to give clues to answer another alternative.

● All distractors should be written so that they are all plausible to the less informed student.

67

Jill is baking cookies. The recipe calls for ¾ cups of white flour per dozen. How much flour is needed to make 3 dozen cookies? Simplify.

A. 14/24 of a square footB. 1 ½ cupsC. 4/7 of a square footD. 8/14 of a square foot

Key: Gradex.y.z / a.b

68

Which problem would best be solved using estimation?

A. 56 percent of 957 = B. 13 percent of 236 =C. 901 ÷ 32 =D. 47 ÷ 17 =

Key: Gradex.y.z

69

Focus Areas Rigor Alignment Item Writing Guidelines Do’s and Don’ts Distractor Guidelines

70

Computer Based Testing

71

CBT● Pearson Educational Measurement

Compensation Current Commitment Improved system

● Advisory Committee

● Intermediate Algebra 100% CBT? More than multiple choice

72

U-PASS High School Task Force

73

U-PASS Purpose

● To identify schools in need of assistance to meet state standards

● This is a “minimum competency” goal. It will be important to keep this in mind as we consider what values are reflected in the model.

74

2006 Milestones● April 2006

Meeting to review designs and make decisions● May – July

Draft business rules● September Assessment Directors Meeting

Review design and business rules

● September – October “Finalize” model USOE run all of the schools through the business rules Districts may replicate analyses in their districts USOE run multiple scenarios for status/progress

● November 14—Task Force Meeting! Meeting to review and clean up the business rules Review data and the multiple scenarios Make initial decisions for status & progress

75

2007 Milestones● January - February 2007

USOE Rerun all reports and fix all business rules Districts replicate the rules for their districts

● March 2007—Task Force Meeting! Meeting to review data and business rules Final planning meeting to approve business rules and

cutscores● April 2007

Develop initial school designations Meeting for districts to review all school designations Make any final adjustments to business rules and

calculations● July 15

Run all AYP/U-PASS/AMAO Reports● August 15

Release all U-PASS to districts for 30 day review● September 15

Release Accountability reports to the public

76

General Principles

● Multiple indicators Should not overwhelm schools’ data capacity

● Compensatory framework

● Every group should be required to meet status or progress

77

Subgroups

● The same approach for holding subgroups accountable (aggregate subgroup) in the 3-8 system will be used in the high school system as well

78

Participation

● Will be calculated using:

UBSCT/UAA - 10th graders attempting all 3 subtests (or UAA all 10th graders)

Math & Science CRTs—all CRT test takers/all students enrolled in CRT courses

Language Arts CRTs – All CRT test takers/all students enrolled in each grade

DWA—all students tested/all 9th graders

79

HS Status Components

● 10th grade UBSCT scores reading, writing, math

● CRTs—counting all those scores earned by : ELA--9, 10, 11 Math CRTs—Elementary Algebra, Geometry

(Intermediate Algebra beginning in 2008)

80

HS Status Components Three Science Options (Earth Science, Biology,

Chemistry, Physics) Option 1

All 9th & 10th grade Science CRTs would count Only the 11th grade CRT for students who did not take the science

CRT in both 9th and 10th (i.e., for the kids who have not received the 2 credits by 11th grade, their 11th grade science CRT will count)

Denominator= 11th graders and below with 2 CRT credits or fewer

Option 2 All 9th, 10th & 11th grade Science CRTs would count Denominator - all students enrolled in a science course

Option 3 All Science CRTs would count Denominator – all students enrolled in a science course

81

HS Status Components-2

● DWA—9th grade

● Graduation/completion rate

● Attendance (18 or more absences/year)

82

HS Status Components-2● Math Courses—full year credit for 9th and 10th grade

students enrolled in a non CRT course once the students have completed Algebra, Geometry, (Intermediate Algebra).

Students earning credit/(total number of students enrolled in 9th and 10th grade)-(number of students enrolled in CRT classes in 9th and 10th grade)

A non CRT course is defined in the current list of math courses, but USOE will have to list all the eligible courses beyond Intermediate Algebra

83

UBSCT Status

● The denominator will be all 10th graders in the school

● The numerator will be the count of students passing the first administration

84

Graduation rate

● How should it be weighted with other

U-PASS elements

● How should it be calculated?

85

Graduation rate – weightingOptions

● 10% of total score

● 10 – 50% of total score

● Similar to Participation – minimum requirement

● After status is calculated Can add points to a school’s status score Can add and subtract points to a school’s

status score

86

Graduation rate - calculation

● Same as AYP

● Develop a new calculation

87

Graduation rate

● Is there a way to more accurately determine the graduation rate for a school?

● Is there a more accurate way to track students dropping out or moving toward graduation?

88

Graduation Attribution Questions● Should a “sending” school be responsible, to any

extent, for whether a student graduates or drops out? If yes, for how long after leaving the sending school?

● Should a “receiving” school be responsible, to any extent, for whether a student graduates or drops out? If yes, after what period of enrollment does

responsibility begin?

● Who is responsible for whether or not a student graduates or drops out if a time gap results from the two answers above?

89

Proposals for Incorporating Graduation into UPASS

● The “difference method”

● The “minimum threshold” method

● ?

90

The Difference Method● Set an ambitious graduation rate target (e.g.,

85%) and be willing to raise it after a number of years (e.g., 90%)

● Compute the difference between the school’s graduation rate and the target (85%)

● Subtract (or add) this difference score to the overall status score Perhaps only add as an incentive Perhaps only add or subtract points if the school is x

points from the target (e.g., 1 SE)

91

The “minimum threshold” method

● Calculate the most accurate graduation rate possible given the current data system

● Graduation should count at least 10-15% for all schools

● A threshold will be established that will serve as a minimum graduation requirement that all schools must meet (and subgroups?) in order to pass

92

Graduation rate – weightingOptions

● 10% of total score

● 10 – 50% of total score

● Similar to Participation – minimum requirement

● After status is calculated Can add points to a school’s status score Can add and subtract points to a school’s

status score

93

Graduation rate - calculation

● Same as AYP

● Develop a new calculation

94

Status Computation Proposal

● Content and attendance subtotal Mathematics=30% ELA=30% Science=30% Attendance=10% UAA added into appropriate content areas

● Graduation calculationDepends on decision about calculation and

weighting

95

Mathematics

● UBSCT Math=25%

● CRT and course scores will be added together for a single score and will count 75% of the math score

96

ELA

● With a 9th grade UBSCT (reading and writing weighted

equally)=25% ELA CRTs=65% DWA=10%

● Without a 9th grade UBSCT (reading and writing weighted

equally)=25% ELA CRTs=75%

97

Science

● CRT scores=100%

98

Growth components● ELA

longitudinal growth from grades 8-11

● For Science and Math: Use previous grade scores as a pretest for the CRT

● For UBSCT: The difference between initial pass rate and ultimate pass

rate

99

FEEDBACK