The Dutch CEF Construct Project

J Charles Alderson, Neus Figueras, Henk Kuijper, Guenter Nold, Sauli Takala, Claire

Tardieu

Outline• Introduction• Issue to be addressed• Research Questions• Method• Results to date

– Problems with the CEF• Terminology: synonymy or not?• Gaps• Inconsistencies• Lack of definitions

– Problems with our Frames of Analysis• The Grid• Some results• Problems

Research Questions

• Do we have an instrument to help us construct tests based on the CEF?

• Are the CEF scales, together with the detailed description of language use contained in the document, sufficient to construct tests based on the CEF?

• If not, what is needed to develop such an instrument, and what should the document be like?

Definition of a reading or listening item, consistent with CEF

"A task (which has an intention/ goal/ purpose) for an individual or group, requiring an understanding of a text related to a theme in a domain which requires certain strategies, under certain conditions and limitations".

Method

• 6 experts• Close examination of CEF• Construction and revision of Frames of Analysis

for Reading and Listening• Apply Frames to sample tasks and Review• Devise Grids• Apply Grids to Specifications and Tasks/ Items• Revise Grids and Produce Guidelines• Final Report end July 2004

Results to date

• Two documents:

Extracts from CEF and from Manual relevant to Reading and Listening

Compilation of scales by level, for Reading and Listening

CEF Scales by Level: Reading A1

• Can understand familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type (p24)

• I can understand familiar names, words and very simple sentences, for example on notices and posters or in catalogues (p26)

• Can understand very short, simple texts a single phrase at a time, picking up familiar names, words and basic phrases and rereading as required (p69)

• Can understand short, simple messages on postcards (p69)• Can recognise familiar names, words and very basic phrases on

simple notices in the most common everyday situations (p70)• Can get an idea of the content of simpler informational material and

short simple descriptions, especially if there is visual support (p70)• Can follow short, simple written directions (e.g. to go from X to Y)

(p71)• No descriptor available for identifying cues and inferring (p72)

CEF Scales by Level: Listening A1

• Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type. (p24)

• I can recognise familiar words and very basic phrases concerning myself, my family and immediate concrete surroundings when people speak slowly and clearly. (p26)

• Can follow speech which is very slow and carefully articulated, with long pauses for him/her to assimilate meaning.(p66)

• Can understand instructions addressed carefully and slowly to him/her and follow short, simple directions. (p67)

• No descriptors available for:– UNDERSTANDING CONVERSATION BETWEEN NATIVE SPEAKERS– LISTENING AS A MEMBER OF A LIVE AUDIENCE– LISTENING TO AUDIO MEDIA AND RECORDINGS– WATCHING TV AND FILM– IDENTIFYING CUES AND INFERRING (Spoken)– NOTE-TAKING (LECTURES, SEMINARS, ETC.)

Results to date

(Using DIALANG Advisory Feedback, Appendix C of CEF)

• First draft Frame based on compilation of scales– Operations– What– Text– Conditions and Limitations

From Scales to Frames - A1 Reading

• Can get an idea of the content of simpler informational material and short simple descriptions, especially if there is visual support

Operation What Text Conditions & limitations

Get general idea

simpler informational

with visual support

short simple descriptions

From Scales to Frames - B2 Reading

• Can understand lengthy, complex instructions in his field, including details on conditions and warnings, provided he/she can reread difficult sections (p70)

Operation What Text Conditions & limitations

Understand relevant details

lengthy, complex instructions

in his field

conditions and warnings

rereading difficult sections

Conditions and Limitations

• in the most common everyday situations

• restricted mainly to common everyday language and language related to my job

• re-reading part of text

Conditions and Limitations

• text features, including– content– vocabulary and sentence structure– text structure– text length

• strategy

• real conditions (e.g using a dictionary)

• other

From Scales to Frames - B2 Reading

• Can understand lengthy, complex instructions in his field, including details on conditions and warnings, provided he/she can reread difficult sections (p70)

Operation What Text Conditions & limitations

Understand relevant details

lengthy, complex instructions

in his field

conditions and warnings

rereading difficult sections

Revised Frame

• in his field = text features

• re-reading difficult sections = strategy

Revised Frame for Reading

• Operation

• What: focus+topic/theme

• Text

• Text features

• Strategy

• Conditions and limitations

Re-Revised Frame for Listening

• Operation• What: focus+topic/theme• Text• Text features:

– Content– Vocabulary and sentence structure– Text structure– Pace and articulation– Text length– Strategy

• Conditions and limitations

Problems with the CEF

• Terminology problems: synonymy or not?

• Gaps

• Inconsistencies

• Lack of definition

Terminology problems: synonymy or not?

Operations at A2

• Understand• Take • Get• Follow• Identify • Infer

Operations at B2

• Understand• Scan• Monitor• Obtain• Select• Evaluate• Locate• Identify

Synonyms?

• I can understand familiar names, words and very simple sentences, for example on notices and posters or in catalogues” (page 26)

• “Can recognise familiar names, words and very basic phrases on simple notices in the most common everyday situations” (page 70)

Synonyms?

Can identify the main conclusions in clearly signalled argumentative texts.

Can recognise the line of argument in the treatment of the issue presented, though not necessarily in detail.”

Synonyms?

Can find specific, predictable information in simple everyday material such as advertisements, prospectuses, menus, reference lists and timetables.

Can locate specific information in lists and isolate the information required (e.g. use the ‘Yellow Pages’ to find a service or tradesman)?

Gaps in the CEF

• A description of the operations that comprehension consists of and a theory of how comprehension develops.

• A specification of micro-skills or subskills of comprehension.

• Concepts introduced in the text but not incorporated in the scales or related to the six levels in any way.

Gaps in the CEF

• competence, general competence, communicative language competence

• activities, processes, text, domain, strategy, task• context• ludic and aesthetic uses of language• texts• text to text activities • socio-cultural knowledge• study skills • tasks, including description, performance (conditions,

competences, linguistic factors), strategies, difficulty

Gaps in the CEF

• The Task: what is it that candidates have to do with text?

• Test methods and the processing demands they create

Inconsistencies

• Operation recognise only mentioned at the levels A1, B1 and C1 and not at the other levels

• The use of a dictionary only mentioned at B2 and C1

• Clear, slow and carefully articulated speech (A1)Clear, slow and articulated speech (A2)Clear, standard speech, familiar accent (B1)Normal speed, standard language (B2)

• “Simple notices” A1, “everyday notices” A2. No other references to “notices”, yet surely there are notices you need to be B2 to understand

Lack of definitions

• Simple

• the most common, everyday, familiar, concrete, predictable, straightforward, factual, complex, specialised, highly colloquial, short, long

• Number of words?

• Is a short text necessarily “easier” than a longer text?

Problems with our Frames

• Many of the problems with the CEF apply to our Frames

• Text? = 1) text source: letter, brochure, newspaper

2) text type: factual, argumentative, persuasive, instructive

• Strategy: hardly any. How to measure?

Conditions and Limitations

• Well enough to correspond regularly with a pen friend

• to a satisfactory level of comprehension • A1 and A2 do not have limitations and

conditions • More conditions and limitations at B2 than

any other level • “a large degree of independence”

Measurable?

From Frame to Grid

• Frame is a way of characterising each CEF Level, from the Can-Dos, level by level

• Grid goes beyond CEF in order to add dimensions thought to be necessary to characterise texts and tasks

• Grid is a means of describing an item or task at a given CEF level.

• Grid therefore has three columns: the dimension, the characteristic of the item by dimension, and the estimated level of that feature according to the CEF

Dimensions of Grid• Operation• What• Item type• Text source• Discourse type• Domain• Topic• Text length (N words or duration)• Vocabulary• Grammar• Text structure• Part of testlet? Number of items• Time to do total task • Readability (Reading)• Number of participants (Listening)• Text Speed (Listening)• Accent/ standard (Listening)• Pronunciation (Listening)• How often played (Listening)• Other (Listening)

Guides to the Grid - Operations

Processes of comprehension

• Focus on and retrieve explicitly stated information

• Make straightforward inferences

• Interpret and integrate ideas and information

• Examine and evaluate content, language and textual elements

Guides to the Grid – Item types• Selected response

– Multiple choice– True False– Multiple matching– Sequencing/ ordering– Citing

• Constructed response– Short answer question– Cloze (every nth)– Gap-filling (one word)– C-Test– Summary completion

• Extended response (creative, etc)– Essay– Summary– Report in own words– Justify– Other

Guides to the Grid – Discourse type

Descriptive impressionistic descriptions

technical descriptions

Narrative stories, jokes, anecdotes

reports

Expository definitions

explications

outlines

summaries

interpretations

Argumentative comments

formal argumentation

Instructive personal instructions

Guides to the Grid – Text Structure

• Collection / list

• Cause – effect

• Problem – solution

• Compare – contrast

• Description

Some results after analysing DIALANG Reading and Listening

items

• Agreement?

Table 1 Domain by Reading items Item CEF level Domain 1 A1 Personal; Personal; Personal or Occupational; Personal +

Occupational; Personal 2 A1 Personal; Occupational; Personal; Personal; Personal 3 A1 Personal; Public; Personal+Public; Personal+Occupational+

Public+Educational; Public 4 A2 Personal; Personal; Personal+Public; Personal; Personal+Public 5 A2 Public; Public; Public;

Personal+Occupational+Public+Educational; Occupational+Public+Educational

6 A2 Personal; Public; Personal+Occupational+Public+Educational; Personal;Public

7 A2 Public; Public; Personal+Public; Public; Occupational+Public 8 B1 Public; Public; Public+Educational; Public; Public etc

Table 2 Domain by Reading items summarised Item CEF level Domain % agreement 1 A1 Personal 60 2 A1 Personal 80 3 A1 Public 40 4 A2 Personal 60 5 A2 Public 60 6 A2 Personal/ Public 40/40 7 A2 Public 60 8 B1 Public 80 9 B1 Personal 40 10 B1 Public 100 11 B1 Public 100 12 B1 Occupational 60 13 B2 Public 60 14 B2 Public 40 15 B2 Public 60 16 B2 Public 40

Table 3 Text Source by Reading items summarised Item CEF level Text source % agreement 1 A1 Recipe 100 2 A1 Junk mail 40 3 A1 Advertising 100 4 A2 Labels and packaging 100 5 A2 Newspaper 60 6 A2 Brochure 60 7 A2 Advertisement 80 8 B1 Newspapers 80 9 B1 Magazine 60 10 B1 Newspaper 100 11 B1 Newspaper 100 12 B1 Business letter 80 13 B2 TV guide 80 14 B2 Magazine 80 15 B2 Magazine 80 16 B2 Brochure 100

Table 4 Topic by Listening items summarised Item CEF level Topic % agreement 1 A2 Train delay advisory 100 2 A2 Directions 83 3 A2 Food 50 4 A2 Family everyday life 60 5 A2 Death notice 80 6 A2 Competition for writers 100 7 B1 Food 80 8 B1 Radio and Music 75 9 B1 Illegal activity 80 10 B1 Travelling 80 11 B1 Appointment 67 12 B1 Food 100 13 B2 Social encounters 83 14 B2 National characteristics 100 15 C1 Art 80 16 C2 Shopping 100

Table 5 “What” by Reading items summarised Item CEF level What %

agreement 1 A1 Everyday expressions 60 2 A1 Probable meaning of unknown words 40 3 A1 Specific information 60 4 A2 Specific information 60 5 A2 Straightforward factual language 40 6 A2 Straightforward factual language 40 7 A2 Specific information 60 8 B1 Main points 60 9 B1 Specific predictable information 40 10 B1 Specific information 80 11 B1 Specific information 60 12 B1 Straightforward factual language. 40 13 B2 Straightforward factual language 40 14 B2 Main idea 60

Table 6 Operation by Reading items summarised Item CEF

level Operation % agreement DIALANG

subskill 1 A1 Main idea 100 Inferencing 2 A1 Main idea 40 Inferencing 3 A1 Important details 40 Specific detail 4 A2 Important details 60 Main idea 5 A2 Main idea 100 Main idea 6 A2 Main idea 100 Main idea 7 A2 Main idea 80 Inferencing 8 B1 Main idea 100 Main idea 9 B1 Main idea 60 Inferencing 10 B1 Important detail 60 Specific detail 11 B1 Important detail 60 Specific detail 12 B1 No agreement 0 Inferencing 13 B2 Main idea 80 Inferencing

Table 7 Operation by Listening Items summarised Item CEF level Operation %

agreement DIALANG Subskill

1 A2 Important detail 67 Main idea 2 A2 Main idea 67 Specific detail 3 A2 Inference 83 Inferencing 4 A2 Infer 60 Inferencing 5 A2 Main idea 80 Main idea 6 A2 Important detail 100 Specific detail 7 B1 Infer lexis 60 Inferencing 8 B1 Inference 75 Main idea 9 B1 Main idea 60 Main idea 10 B1 Main idea 100 Main idea 11 B1 Inferencing 83 Inferencing 12 B1 Inferences 100 Main idea 13 B2 Inference 50 Specific detail

Table 8 Mean agreement by dimension, Reading

Grid Dimension Mean Standard deviation

Text source 78.8 20.6

Topic 77.6 23.3

Operation 64.7 27.9

Discourse type 64.7 30.4

Text Structure 60.0 25.5

Domain 57.6 24.4

Grammar 52.9 24.4

Vocabulary 50.6 28.4

What 48.2 17.4

Table 9 Mean agreement by dimension, Listening

Grid Dimension Mean Standard deviation Accent/ standard 92.9 11.2 Topic 82.4 15.1 Operation 75.8 15.4 Text source 66.2 17.4 Pronunciation 64.6 20.5 Text speed 60.3 12.0 Domain 60.3 31.4 Grammar 55.0 20.9 Vocabulary 51.3 43.8 What 49.3 31.2 Discourse type 48.9 35.3 Text Structure 33.1 28.7

Problems

• Some dimensions are more objectively determined than others

• Some dimensions are very open at present

• Varying degrees of agreement among raters, for any given item and for any dimension across items

Proposed Guides to the Grid - Vocabulary

1. Only frequent words

2. Mostly frequent words

3. Rather extended

4. Extended

Proposed Guides to the Grid - Grammar

1. Only simple sentences

2. Mostly simple sentences

3. Frequent compound sentences

4. Many complex sentences

Problems?

• Some dimensions felt to be difficult to apply or irrelevant. Delete? Text structure; Discourse type; Readability

• Not yet inspected confidential specifications and guidelines to item writers

Problems?

• No relationship between any of our Grid dimensions, and CEF level, except for length of text in number of words

• But so far we have only analysed DIALANG, computer-based items

• Need to replicate with a range of paper and pencil tests, for English, French and German, at least

Problems

• Different aspects of a task or item may be at different CEF levels. How to combine?

• How to deal with testlets (multiple-item tasks)?

• What if no evidence for CEF level of items?

• What if no item/ task level empirical data?

Problems

• In order to know whether a given item is indeed at the level of difficulty intended, piloting on suitable samples of test-takers is crucial

• We need a suitable sample, i.e., we need to know the level of the test takers in order to judge the adequacy of the item – vicious circle!

Possible solutions• Describe the text and items using the dimensions of a

classification system (The Grids)• Make a guess at the level of an item (guided by the

classification system), leading to the estimated CEF-level• Pretest the items thus labelled• Calibrate the items• Do standard setting to set the boundaries of the levels

on the scale coming from the calibration• Assign a psychometric level to the items. • Assign a definitive level to the items. An item can only be

assigned to a definitive level if the psychometric level falls within the band of the estimated level (in other words if the estimation based on the content is comparable with the psychometric value)

End results

• Grid to characterise items at CEF levels

• Guidelines for item writing

• Means of estimating adequacy of coverage of test content

• Procedure to confirm CEF level empirically