elt testing and assessment session 3 today: 1. article discussion 2. practical steps for test...

ELT Testing and ELT Testing and AssessmentAssessment

Session 3

Today:1.Article Discussion2.Practical Steps for Test Construction3.Intro to Scoring, Grading and Feedback

1

Practical Steps to Practical Steps to Test ConstructionTest Construction

1.Assessing Clear, Unambiguous Objectives

2.Drawing Up Test Specifications3.Devising Test Tasks4.Designing Multiple-Choice Test

Items

2

1. Assessing Clear, 1. Assessing Clear, Unambiguous ObjectivesUnambiguous ObjectivesWhat is it you want to test?Ask yourself:

◦What should Ss know?◦What should Ss be able to do?

Answers should be stated as overt performance within a clear target linguistic domain

Ideally, these should be the objectives for the lesson/unit/course you are testing.

3

OBJECTIVES FOR LOW-INTERMEDIATE INTEGRATED COURSE

Form-focused objectives (listening & speaking) – Students will:

1. Recognize and produce tag questions, with the correct grammatical form and final intonation pattern, in simple social conversations

2. Recognize and produce wh-information questions with correct final intonation pattern and appropriate answers

Communication skills (speaking) – Students will:

3. State completed actions and events in a social conversation

4. Ask for confirmation in a social conversation

5. Give opinions about an event in a social conversation

6. Produce language with contextually appropriate intonation, stress and rhythm

Reading skills (simple essay or story) – Students will:

7. Recognize irregular past tense of selected verbs in a story or essay

Writing skills (simple essay or story) – Students will:

8. Write a one-paragraph story about a simple event in the past

9. Use conjunctions so and because in a statement of opinion4

2. Test Specs should 2. Test Specs should include:include:a. Outline of test (what, when/how

long, generally how)b. Skills to be included c. Item types and tasks

5

TASK: Create Test Specs for TASK: Create Test Specs for a mid-term test for coursea mid-term test for courseUsing objectives on slide 4You have 12 studentsYou have 30 minutes scheduled for

the testBut you also can incorporate some

form of assessment into the preceding class if you want

You need to test all skills

6

Ways to elicit responses Ways to elicit responses ……

Elicitation mode:

Oral (S listens) Written (S reads)

Word, pair of wordsSentence(s), question DirectionsMonologue, speechPre-recorded conversationInteractive (live) dialogue

Word, set of wordsSentence(2), questionDirectionsParagraphEssay, excerptShort story, book

Response mode:

Oral (for either oral or written elicitation)

Written (for either oral or written elicitation)

RepeatRead aloudYes/noShort responseDescribeRole playMonologue (speech)Interactive dialogue

Mark multiple-choice optionFill in the blankSpell a wordDefine a term (with a phrase)Short answer (2-3 sentences)essay 7

TASK: test specs outline…TASK: test specs outline…Speaking (time)

◦ Format:◦ Task:

Listening (time)◦ Format:◦ Task

Reading (time)◦ Format:◦ Task:

Writing (time)◦ Format:◦ Task:

Should cover:- Topics (objectives)- Implied elicitation and

response formats- Number of items in

each section- Time to be allocated for

each

8

What do you think of this What do you think of this outline?outline?

Speaking (5 minutes / person; previous day)◦ Format: oral interview, T and S◦ Task: T asks questions of S (Obj 3, 5; emphasis on

6) Listening (10 minutes)

◦ Format: T makes audiotape in advance, w one other voice on it

◦ Task: a. 5 minimal pair items; m-c (Obj 1) b. 5 interpretation items, m-c (Obj 2)

Reading (10 minutes)◦ Format: cloze test items (10 total) in storyline◦ Task: Fill in the blanks (Obj 7)

Writing (10 minutes)◦ Format: prompt for a topic about sitcom seen in

class◦ Task: writing a short opinion paragraph (Obj 9)

9

3. Devising Test Tasks 3. Devising Test Tasks Sample: test item – 1st draft – Listening, part bDirections: Listen to the question [on the tape].

Choose the sentence on your test page that is the best answer to the question:

Voice: Where did George go after the party last night?

S reads: a. Yes, he did.b. Because he was tired.c. To Elaine’s place for another party.d. He went home around eleven o’clock.

QUESTION: Any problem with this?10

As you revise your test draft As you revise your test draft – ask yourself:– ask yourself:

Are the directions to each section absolutely clear? Is there an example item for each section?Does each item measure a specified objective? Is each item stated in clear, simple language?Does each m-c item have appropriate distractors? Is the difficulty of each item appropriate for your

students? Is the language of each item sufficiently authentic?Do the sum of the items and the test as a whole

adequately reflect the learning objectives?

11

4. More on M-C Test Items4. More on M-C Test ItemsCons?

◦ Tests only recognition knowledge

◦ Guessing has effect on grades

◦ Restricts what can be tested

◦ Difficult to write good items

◦ Washback may be harmful

◦ Cheating easier

Pros?◦ Practicality◦ Reliability

12

M-C TerminologyM-C TerminologyM-C items are all receptive

(selective) response items (t-t chooses from set)

Stem = stimulusOptions/alternatives = answer

choices (typically 3-5)Key = correct answerDistractors = incorrect answers

13

What’s wrong with these What’s wrong with these tasks?tasks?

Excuse me, do you know ___?a.where is the post officeb.where the post office isc.where post office is

(Remember: task should be focused on specific learning objective)

14

How can we improve this?How can we improve this?

My eyesight has really been deteriorating lately. I wonder if I need glasses. I think I’d better go to the ____ to have my eyes checked.

a.pediatricianb.dermatologistc.optometrist

15

How can this task be How can this task be better?better?We went to visit the temples, ____

fascinating.a.which were beautifulb.which were especiallyc.which were holy

16

““Biased for best”Biased for best”Goal: to assure Ss have

opportunity to perform at highest personal level on test◦T. provides appropriate review◦T. suggests appropriate preparation

and test-taking strategies◦T. structures test so stronger Ss are

challenged and weaker Ss do not feel defeated/overwhelmed

17

Item Indices – Item Indices – used to accept, used to accept, discard or revise items (after trialing or discard or revise items (after trialing or using the test)using the test)1. Item Facility – extent to which an

item is easy or difficult for proposed group

# of Ss answering the item correctly___________________________________________________________________________________________________

Total # of Ss responding to that item

Example: IF = 13/20 = .65 or 65%Range of .15 to .85 acceptable

QUESTION: Why would you want to include questions at either end of the spectrum in your test? 18

2. Item Discrimination – the extent to which an item differentiates between high- and low-ability test-takers

Divide t-t into 3 groups (high, middle, low)

Example: 30 students – 3 groups of 10

high-group # correct – low group # correct_____________________________________________________________________________________

½ x total of your two comparison groups

ID= 7 -2 / ½ x 20 = 5/10 = .50

This is a moderate ID

(high ID would be close to 1.0)

19

For item indices, large-scale For item indices, large-scale tests use…tests use…

IRT – item response theory

20

For every-day classroom For every-day classroom testing, teachers use…testing, teachers use…

intuition

21

Distractor efficiency = extent to which a) distractors “lure” t-ts and

b) responses are evenly distributed among all distractors

Choices A B C* D EHigh Ss (10) 0 1 7 0 2Low Ss (10) 3 5 2 0 0

QUESTION: What do we learn about D and E?

22

Scoring – Scoring – Weight:Weight:

Oral interview 40%4 scores, 5-1 range x 2 = 40 points

Listening 20%10 items @ 2 pts/each = 20 points

Reading 20%10 items @ 2 pts/each = 20 points

Writing 20%2 scores, 5-1 range x 2 = 20 points

23

Grading – take into Grading – take into account:account:Country, culture, context of this

English classroomInstitutional expectations (often

unwritten)Explicit and implicit definitions of

grades that you have establishedRelationship you have established

with students/classStudent expectations engendered in

previous tests and quizzes

24

FeedbackFeedbackLetter gradeTotal scoreSubscoresRight/Wrong indicationMarginal commentsChecklist of areas needing workOral feedback after interviewSelf-assessmentPeer checkingWhole class discussion Individual post-test conference

25