incorporating the cefr into language test development ... · 1 jamie dunlea, british council...
TRANSCRIPT
www.britishcouncil.org 1
Jamie Dunlea, British Council
Incorporating the CEFR into language
test development: Using an international
framework in local contexts
E-merging Forum 4
March 13-14
Moscow
Language Assessment
Research
To summarize there is no gold standard, there is no
true cut-off score, there is no best standard setting
method, there is no perfect training, there is no
flawless implementation of any standard setting
method on any occasion and there is never
sufficiently strong validity evidence. In three words,
nothing is perfect. (Kaftandjieva, 2004)
www.britishcouncil.org 2
What is the CEFR?
What is the CEFR?
Common
European
Framework
of Reference for Language:
learning, teaching and assessment
www.britishcouncil.org 3
Published by the Council of Europe in 2001
“Formal origins of the CEFR date back to 1991” (Morrow,2004)
40 years of research in language education in Europe (Morrow,2004; Trim, 2010)
Waystage, Threshold, Vantage
www.britishcouncil.org 4
What is the CEFR?
What the blue book says…
Provides a common basis for the elaboration of language syllabuses, curriculum guidelines, examinations, textbooks, etc. across Europe.
defines levels of proficiency which allow learners’ progress to be measured at each stage of learning and on a life-long basis.
will facilitate the mutual recognition of qualifications gained in different learning contexts, and accordingly will aid European mobility.
www.britishcouncil.org 5
What is the CEFR?
“At the heart of the CEF are the
Common Reference levels.”
(Morrow, 2004)
www.britishcouncil.org 6
What is the CEFR?
www.britishcouncil.org 7
C2
C1 6
5
4
3
2
1
B2+
B2
B1+
B1
A2+
A2
C2
C1
B2
B1
C2
C1
B2
B1
A2
A1
Proficient
User
Independent
User
Basic
User
What is the CEFR?
The Global Scale summarizes “the
proposed Common Reference Levels
in single holistic paragraphs”
(Council of Europe, 2001, p. 24)
www.britishcouncil.org 8
What is the CEFR?
Can understand with ease virtually everything heard or read. Can summarise information from different spoken
and written sources, reconstructing arguments and accounts in a coherent presentation. Can express him/herself spontaneously, very fluently and precisely, differentiating finer shades of meaning even in more complex situations.
Can understand a wide range of demanding, longer texts, and recognise implicit meaning. Can express
him/herself fluently and spontaneously without much obvious searching for expressions. Can use language
flexibly and effectively for social, academic and professional purposes. Can produce clear, well-structured,
detailed text on complex subjects, showing controlled use of organisational patterns, connectors and cohesive
devices.
Can understand the main points of clear standard input on familiar matters regularly encountered in work, school,
leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is
spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe
experiences and events, dreams, hopes and ambitions and briefly give reasons and explanations for opinions and
plans.
Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g.
very basic personal and family information, shopping, local geography, employment). Can communicate in simple
and routine tasks requiring a simple and direct exchange of information on familiar and routine matters. Can
describe in simple terms aspects of his/her background, immediate environment and matters in areas of
immediate need.
Can understand the main ideas of complex text on both concrete and abstract topics, including technical
discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that
makes regular interaction with native speakers quite possible without strain for either party. Can produce clear,
detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and
disadvantages of various options.
Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs
of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal
details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way
provided the other person talks slowly and clearly and is prepared to help.
C2
C1
B2
B1
A2
A1
Can understand the main points of clear standard input on familiar matters regularly encountered in work, school,
leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is
spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe
experiences and events, dreams, hopes and ambitions and briefly give reasons and explanations for opinions and
plans.
Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g.
very basic personal and family information, shopping, local geography, employment). Can communicate in simple
and routine tasks requiring a simple and direct exchange of information on familiar and routine matters. Can
describe in simple terms aspects of his/her background, immediate environment and matters in areas of
immediate need.
Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs
of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal
details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way
provided the other person talks slowly and clearly and is prepared to help.
Can understand the main ideas of complex text on both concrete and abstract topics, including technical
discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that
makes regular interaction with native speakers quite possible without strain for either party. Can produce clear,
detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and
disadvantages of various options.
9
Can understand a wide range of demanding,
longer texts, and recognise implicit meaning.
Can express him/herself fluently and
spontaneously without much obvious searching
for expressions. Can use language flexibly and
effectively for social, academic and professional
purposes. Can produce clear, well-structured,
detailed text on complex subjects, showing
controlled use of organisational patterns,
connectors and cohesive devices.
C2
C1
B2
B1
A2
A1 10
C2
C1
B2
B1
A2
A1
Can understand the main points of clear
standard input on familiar matters regularly
encountered in work, school, leisure, etc. Can
deal with most situations likely to arise whilst
travelling in an area where the language is
spoken. Can produce simple connected text on
topics which are familiar or of personal interest.
Can describe experiences and events, dreams,
hopes and ambitions and briefly give reasons
and explanations for opinions and plans. 11
C2
C1
B2
B1
A2
A1
Can understand and use familiar everyday
expressions and very basic phrases aimed at
the satisfaction of needs of a concrete type. Can
introduce him/herself and others and can ask
and answer questions about personal details
such as where he/she lives, people he/she
knows and things he/she has. Can interact in a
simple way provided the other person talks
slowly and clearly and is prepared to help. 12
The global scale is “just the tip of the iceberg” (Morrow, 2004)
Illustrative descriptors in 54 scales
o Communicative activities
o Strategies
o Communicative language competences
www.britishcouncil.org 13
What is the CEFR?
Overall Listening Comprehension
Understanding Interaction between Native
Speakers.
Listening as a Member of a Live Audience
Listening to Announcements & Instructions
Overall Reading Comprehension
Reading Correspondence
Reading for Orientation
Reading for Information and Argument
Reading Instructions www.britishcouncil.org 14
What is the CEFR?
CEFR
LEVEL
C2
C1
B2
B1
A2
A1
Cambridge ESOL
IELTS TOEFL PBT TOEFL iBT
CPE 8
CAE 6.5 560 110-120
FCE 5 87-109
PET 4 457 57-86
KET
www.britishcouncil.org 15
What is the CEFR?
Based on information presented by individual exam boards
IIELTS scores are borderline band scores for that level (for more, see
www.ielts.org/researchers/common_european_framework)
Morrow (2004): notes ambiguity in terminology: “what are main points?”; “How many is most?”
Alderson et al (2006): problems for designing tests: Inconsistencies; Terminology problems; Lack of definition; Gaps.
O’Sullivan & Weir (2011): “lacks the theoretical rigor, coverage and explicitness necessary…to develop tests”
Davidson & Fulcher (2007): “does not detail particular contexts in which it is to be used, and so lacks the necessary detail on which to build test specifications.”
www.britishcouncil.org 16
Cautions, criticisms…
The CEFR is purely descriptive – not normative
The CEFR is language neutral – it needs to be applied with regard to each specific language.
The CEFR is context neutral – it needs to be applied and interpreted with regard to each specific educational context in accordance with the needs and priorities of that context.
The CEFR attempts to be comprehensive. It cannot, of course, claim to be exhaustive. Further elaboration and developments are welcomed.
www.britishcouncil.org 17
Principles for users of the CEFR
(North, Martyniuk, & Panthier, 2010)
City & Guilds Communicator IESOL Examination (O’sullivan, 2008)
Dutch state foreign language examinations (Berger, Kuiper, & Maris, 2009; Noijons & Kuipers, 2010)
TestDAF (Kecker & Eckes, 2010)
Trinity College Examinations (Papageorgio, 2007; Papageorgio, 2009)
The European language portfolio (Lenz, 2004)
The English Profile project (Trim, 2010)
www.britishcouncil.org 18
Applications to testing
TOEFL PBT (Tannenbaum & Wylie, 2005)
TOEFL iBT (Tannenbaum & Wylie, 2008)
GEPT, Taiwan (Wu & Wu, 2010; Wu ,
2012)
EIKEN, Japan (Dunlea & Figueras, 2012)
www.britishcouncil.org 19
Applications to testing: Outside Europe
It has problems, but…
It was not designed to be a completed, prescriptive document
It was designed to facilitate communication and collaboration amongst language educators
It is a work in progress
Attempts to link or relate exams to the CEFR should be seen as validation projects of the CEFR itself
Describing problems encountered in the “linking” process can lead to more extensive descriptions being added to the CEFR tool kit
www.britishcouncil.org 20
Is it useful?
A three-way classification of proficiency scales (Alderson, 1991) is often used :
user oriented assessor oriented constructor oriented
The CEFR is a user oriented scale (North, 2000)
It was not designed or intended for use as a rating scale or in test development without adaption and modification suitable for those purposes
www.britishcouncil.org 21
Is it useful?
Davidson & Fulcher (2007) encourage test developers to see the framework as a “series of guidelines from which tests (and teaching materials) can be built to suit local contextualized needs.”
www.britishcouncil.org 22
Is it useful?
The CEFR can be a
springboard to task and test
development
Socio-cognitive framework for language test development and validation (Weir, 2005; O’Sullivan & Weir, 2011)
Reading: Khalifa & Weir (2007)
Listening: Geranpayeh & Taylor (2013)
Speaking: Taylor, L. (2012).
Writing: Shaw & Weir (2009)
Test design & development: O’Sullivan (2012)
www.britishcouncil.org 23
Filling the gaps: validation
CONTEXT VALIDITY COGNITIVE VALIDITY
RESPONSE
SCORING VALIDITY
TEST-TAKER CHARACTERISTICS
CONSEQUENTIAL VALIDITY CRITERION –RELATED
VALIDITY
Filling the gaps: validation
www.britishcouncil.org 25
Are the characteristics of the test tasks and their
administration fair to the candidates who are taking them?
(Context validity)
Are the cognitive processes required to complete the
tasks appropriate? Are candidates likely to use the same
cognitive processes as they would if performing the task
in a ‘real world’ context? (Cognitive validity)
To what extent can we depend on the scores on the test?
What do the numbers or grades mean? (Scoring validity)
What effects does the test have on its various
stakeholders? (Consequential validity)
What external evidence is there outside of the test scores
themselves that the test is doing a good job? (Criterion-
related validity)
Filling the gaps: validation
www.britishcouncil.org 26
Filling the gaps: cognitive
(Khalifa & Weir, 2007)
Types of
reading
(goal
setting)
Expeditious reading: local Careful reading: local
Expeditious reading: global Careful reading: global
Levels
of
reading
www.britishcouncil.org 27
Filling the gaps: cognitive
(Khalifa & Weir, 2007)
Types of
reading
(goal
setting)
Expeditious reading: local Careful reading: local
Expeditious reading: global Careful reading: global
Levels
of
reading
Word recognition
Lexical access
Syntactic parsing
Establishing propositional meaning
Inferencing
Building a mental model
Creating a text level representation
Creating an intertextual representation
www.britishcouncil.org 28
Filling the gaps: cognitive OVERALL READING COMPREHENSION
B2 Can read with a large degree of independence, adapting style and
speed of reading to different texts and purposes, and using
appropriate reference sources selectively. Has a broad active reading
vocabulary, but may experience some difficulty with low-frequency
idioms. B1 Can read straightforward factual texts on subjects related to his/her
field and interest with a satisfactory level of comprehension. Can understand short, simple texts on familiar matters of a concrete
type which consist of high frequency everyday or job-related language
A2 Can understand short, simple texts containing the highest frequency
vocabulary, including a proportion of shared international vocabulary
items. A1 Can understand very short, simple texts a single phrase at a time,
picking up familiar names, words and basic phrases and rereading as
required.
OVERALL READING COMPREHENSION
B2 Can read with a large degree of independence, adapting style and
speed of reading to different texts and purposes, and using
appropriate reference sources selectively. Has a broad active reading
vocabulary, but may experience some difficulty with low-frequency
idioms. B1 Can read straightforward factual texts on subjects related to his/her
field and interest with a satisfactory level of comprehension. Can understand short, simple texts on familiar matters of a concrete
type which consist of high frequency everyday or job-related language
A2 Can understand short, simple texts containing the highest frequency
vocabulary, including a proportion of shared international vocabulary
items. A1 Can understand very short, simple texts a single phrase at a time,
picking up familiar names, words and basic phrases and rereading as
required.
www.britishcouncil.org 29
Filling the gaps: Test specification for
an A1 Reading task in the Aptis test
Types of
reading
(goal
setting)
Expeditious reading: local Careful reading: local
Expeditious reading: global Careful reading: global
Levels
of
reading
Word recognition
Lexical access
Syntactic parsing
Establishing propositional meaning
Inferencing
Building a mental model
Creating a text level representation
Creating an intertextual representation
www.britishcouncil.org 30
Filling the gaps: a test spec Test
Aptis
General Component Reading Task Multiple Choice Gap-Fill
Features of the Task
Skill focus Reading comprehension up to the sentence level
Task Level A1 A2 B1 B2 C1 C2
task
description
Multiple-choice gap fill. A short text of 6 sentences is presented. Each sentence
contains one gap. Test takers choose the best option from a pull-down menu for
each gap to complete the sentence. The first sentence is an example with the
gap completed. Each gap can be filled by reading within the sentence.
Cognitive
processing
Goal
setting
Expeditious reading: local
(scan/search for specifics)
Careful reading: local
(understanding sentence)
Expeditious reading: global
(skim for gist/search for key
ideas/detail)
Careful reading: global
(comprehend main idea(s)/overall
text(s))
Cognitive
processing
Levels of
reading
Word recognition
Lexical access
Syntactic parsing
Establishing propositional meaning (cl./sent. level)
Inferencing
Building a mental model
Creating a text level representation (disc. structure)
Creating an intertextual representation (multi-text)
www.britishcouncil.org 31
Features of the Input Text
Words 40-50 words (including target words for gaps)
Domain Public Occupational Educational Personal
Discourse
mode
Descriptive Narrative Expository Argument
ative
Instructive
Content
knowledge
General Specific
Cultural
specificity
Neutral Specific
Nature of
information Only concrete Mostly concrete Fairly abstract Mainly abstract
Lexical
Level
K1 K2 K3 K4 K5 K6 K7 K8 K9 K10
Text genre E-mails, letters, notes, postcards
Features of the Response
Target Length 1 word Lexical K1
Part of
Speech Noun, verb, adjective
Distractors Length 1 word Lexical K1
Part of
Speech Noun, verb, adjective
Key Within sentence Across sentences Across paragraphs
Filling the gaps: a test spec
www.britishcouncil.org 33
Skill focus Lvl Task description Types and levels of
reading
Sentence level
meaning
A1 A short text with 5 gaps. Filling each gap
only requires comprehension of the
sentence containing the gap. Text-level
comprehension is not required.
• Careful local reading
• Syntactic parsing
• Understanding
propositional meaning
Inter-sentence
cohesion
A2 Reorder jumbled sentences to form a
cohesive text
• Careful global reading
• Inferencing
• Building a mental
model
Text-level
comprehension
of short texts
B1 A short text with 7 gaps. Requires
comprehension of text across
sentences.
• Careful global reading
• Building a mental
model
Integrating
macro-
propositions and
understanding
important ideas
in longer texts
B2 Matching the most appropriate heading
to paragraphs. Requires integration of
micro- and macro-propositions within
and across paragraphs, and
comprehension of discourse structure of
more complex and abstract texts.
• Expeditious global
reading
• Creating a text level
representation
Filling the gaps: Test spec overview
for the Aptis reading test
www.britishcouncil.org 34
Filling the gap: a test spec
http://www.britishcouncil.org/aptis
More on the Aptis Test is
available online:
In three words, nothing is
perfect.
But that is part of the fun
Thank you!
www.britishcouncil.org 35
And remember…
References Alderson, J.C. (1991). Bands and scores. In Alderson, J.C. & North, B. (eds.), Language Testing in the 1990s. London. Macmillan.
Alderson, J., Figueras, N., Kuijper, H., Nold, G., Takala, S., Tardieu, C., (2006). Analysing tests of reading and listening in relation to the Common European Framework of Reference: the experience of the Dutch CEFR construct project. Language Assessment Quarterly, 3(1), 3-30
Bechger, T., Kujper, H., & Maris, G., (2009). Standard setting in relation to the Common European Framework of Reference for Languages: the case of the State Examination of Dutch as a Second Language. Language Assessment Quarterly. 6, 126-150.
Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge: Cambridge University Press.
Davidson, F., & Fulcher, G. (2007). The Common European Framework of Reference (CEFR) and the design of language testes: A matter of effect. Language Teaching, 40, 231-241.
Geranpayeh, A., & Taylor, L.(Eds.) (2013). Examining Listening: Research and practice in assessing second language listening. Studies in Language Testing 35. Cambridge: Cambridge University Press.
Dunlea, J., & Figueras, N. (2012). Replicating Results from a CEFR Test Comparison Project Across Continents. In D. Tsagari & I. Csepes (Eds.), Collaboration in Language Testing and Assessment. New York: Peter Lang.
Kaftandjieva, F. (2004). Standard Setting. In S. Takala (Ed.), Reference Supplement to the Preliminary Version of the Manual for Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEF). Strasbourg: Language Policy Division.
Khalifa, H., & Weir, C J. (2009). Examining Reading: Research and practice in assessing second language reading.
Studies in Language Testing 29. Cambridge: Cambridge University Press.
Lenz, P. (2004). The European language portfolio . In K. Morrow (Ed.). (2004). Insights from the Common European
Framework. Oxford, England: Oxford University Press.
www.britishcouncil.org 36
References Morrow, K. (Ed.). (2004). Insights from the Common European Framework. Oxford, England: Oxford University Press.
North, B. (2004). Relating assessments , examinations, and courses to the CEF. In K. Morrow (Ed.) Insights from the
Common European Framework. Oxford, England: Oxford University Press.
North, B., (2001) The development of a common framework scale of language proficiency. New York:Lang.
North, B., Martyniuk, W., & Panthier, J. (2010) Introduction: The manual for relating examinations to the Common
European Framework of Reference for Languages in the context of the Council of Europe’s work on language Education.
In Martyniuk, W. (Ed) Aligning Tests with the CEFR. Cambridge: Cambridge University Press.
O’Sullivan, B. (2012). Aptis Test Development Approach. Aptis Technical Report ATR-1. London: British Council.
O’Sullivan, B. (2008). City & Guilds Communicator IESOL Examination (B2) CEFR Linking Project: Case study Retrieved
from: reporthttp://www.cityandguilds.com/documents/ind_general_learning_esol/CG_Communicator_Report_BOS.pdf.
O’Sullivan, B., & Weir, C. (2011). Test development and validation. In B. O’Sullivan (Ed), Language Testing: Theories and
Practice. Oxford: Palgrave Macmillan.
Papageorgio, S. (2007). Relating the Trinity College London GESE and ISE exams to the Common European Framework of Reference: Piloting of the Council of Europe draft manual, final project report. London: Trinity College London. Retreived from http://www.trinitycollege.it/accreditamenti/cefr-report.pdf
Papageorgio, S. (2009). Linking international examinations to the CEFR: the Trinity College London Experience. In W. Martyniuk (Ed). Aligning Tests with the CEFR. Cambridge: Cambridge University Press.
Shaw, S., & Weir, C J. (2007). Examining writing: Research and practice in assessing second language writing, Studies in Language Testing 26. Cambridge: Cambridge University Press and Cambridge ESOL.
www.britishcouncil.org 37
References Tannenbaum, R. J., & Wylie, E. C. (2008). Linking English-Language Test Scores onto the Common European
Framework of Reference: An Application of Standard Setting Methodology (TOEFL iBT Research Report). Princeton, NJ:
ETS.
Tannenbaum, R. J., & Wylie, E. C. (2005). Mapping English language proficiency test scores onto the Common European
Framework (ETS Research Rep. No. RR-05-18; TOEFL Research Rep. No. RR–80). Princeton, NJ: ETS.
Taylor, L.(ed.) (2012). Examining Speaking: Research and Practice in Assessing Second Language Speaking. Studies in
Language Testing 35. Cambridge: Cambridge University Press.
Trim, J. (2010). The modern languages programme of the Council of Europe as a background to the English Profile
Project. English Profile Journal, 1(1), 1-12.
Weir, C. J. (2005). Limitations of the Common European Framework for developing comparable examinations and tests.
Language Testing, 22, 281–300.
Wu, J. R. W., & Wu, R.Y.F. (2010). Relating the GEPT reading comprehension tests to the CEFR. In W. Martyniuk (Ed).
Aligning Tests with the CEFR. Cambridge: Cambridge University Press.
Wu, R. Y. F. (2012). Establishing the validity of the General English Proficiency Test Reading Component through a
critical evaluation on alignment with the Common European Framework of Reference. Unpublished PhD thesis: University
of Bedfordshire, Bedfordshire.
Weir, C. J. (2005). Language Testing and Validation: an evidenced-based approach. Oxford: Palgrave.
www.britishcouncil.org 38