Tests of receptive skills developing tests for Reading and Listening

KAYSERI, 29-31 JANUARY 2014

Aims of the session

! challenges in testing for comprehension ! what sub-skills are to be tested ! selecting texts ! writing items !  task types: pros and cons

What makes testing comprehension challenging?

What makes testing comprehension challenging?

Pros !  relatively easy to create if

texts are available

!  input can control learner output

!  objective marking is normally possible

Cons !  no observable learner

behaviour

!  familiarity and interest may affect success

!  if a learner fails, difficult to know why they failed

!  are we only testing what we intend to test?

receptive skills can only be tested indirectly

language performance ! inferences to comprehension abilities

set aims

select texts

write tasks

The process of writing a test

Sub-skills for reading

Sub-skills for listening

Selecting texts

!  a representative sample of text types learners are likely to encounter

!  appropriate extent

!  texts with high information content

!  clear structure

!  neutral but interesting themes

!  avoid relying on general knowledge

!  avoid culturally loaded texts

!  avoid texts learners have already read

Item writing checklist

"  text and item grammatically correct

"  English natural and appropriate

"  item achieves test objective

"  the intended sub-skills needed to answer item

"  one correct key (or a limited number of responses)

"  open answers within student abilities

"  open answers with valid and reliable marking

"  distractors all work

"  item economical and practical

"  key complete and correct

Common task types for testing receptive skills

Common task types for testing receptive skills

! multiple choice ! short answers ! gap fill !  information transfer ! multiple matching

! for both reading and listening

! note-taking, dictation or transcription ! for listening only

Common task types for testing receptive skills

! short answers ! multiple choice ! gap fill ! multiple matching !  information transfer

! for both reading and listening

! note-taking, dictation or transcription ! for listening only

What are the pros and cons of each task type?

Short answers

Pros !  each student can

complete an item according to their abilities

!  no distractors

!  links reception to controlled production

Cons !  need to decide if answer

length is specified !  unpredictable alternative

answers !  difficult to phrase stem

to determine answer !  encourages verbatim

quoting – comprehension?

Multiple choice

Pros !  very familiar format

!  quick to complete

!  easy to mark

!  very objective

!  high discrimination index

Cons !  difficult to write good

distractors

!  need to produce one correct answer only

!  temptation to use for every purpose

!  may encourage guessing

Tips for writing good distractors

" Make sure there is only ONE correct answer.

"  They should be the same form as the key.

"  They should be related logically to the key.

"  Key should not stand out – similar length and level.

"  Avoid synonyms so they can’t be eliminated at once.

"  Keep distractor extent and repetition to a minimum.

" Don’t pair up two distractors (these focus attention).

" Check for unintended clues (e.g. an … ! vowel).

"  Avoid illogical or obviously impossible distractors.

"  If nobody chooses a distractor, it isn’t a distractor!

Gap fill

Pros !  familiar format

!  easy to create

!  relatively easy to mark

Cons !  difficult to predict all

alternative answers

!  some items can be answered without reading the whole item

Multiple matching

Pros !  clear task, easy to do

!  easy to mark

Cons !  errors duplicate as

items aren’t independent of one another

!  can be solved by elimination (where there are no distractor items)

!  encourages guessing

Information transfer

Pros !  integrated ! real-life

skill

!  involving and challenging task

Cons !  two or more skills

tested at once

!  difficult to write tasks that limit the number of variables

!  difficult to mark

Note-taking, dictation, transcription

Pros Cons

Open comprehension questions

Pros !  requires detailed output

! better differentiation between students?

!  integrated: tests comprehension AND production

Cons !  answer extent is

unspecified

Thank you! zoltan.rezmuves@digimedia.hu

consonantvoiced.blogspot.com