challenges in developing and delivering a valid test
DESCRIPTION
Challenges in Developing and Delivering a Valid Test. Michael King and Mabel Li NAFLE, July 2013. Contents. 1. Integrated Language Education System 2.Basic Concepts in Language Assessment 3. Considerations in Language Assessment – IELTS Approach 4.Q&A. - PowerPoint PPT PresentationTRANSCRIPT
www.britishcouncil.org 1
Challenges in Developing and Delivering a Valid Test
Michael King and Mabel LiNAFLE, July 2013
www.britishcouncil.org 2
Contents
1. Integrated Language Education System
2. Basic Concepts in Language Assessment
3. Considerations in Language Assessment – IELTS Approach
4. Q&A
www.britishcouncil.org 3
1. Integrated Language Education System
LearnersLearners
Curriculum Delivery
Curriculum Delivery
CurriculumCurriculum
Language Assessment
Language Assessment
www.britishcouncil.org 4
2.
Language Assessment: Basic Concepts
www.britishcouncil.org 5
What is assessment?
Noun
The evaluation or estimation of the nature, quality, or
ability of someone or something: "the assessment of
educational needs".
Synonyms
appraisal - rating - valuation - estimation - evaluation
www.britishcouncil.org 6
Why do we need to do assessment?
Evaluate learning
Measure achievement
Inform future teaching
Rank stu
den
ts
Benchmarking
Motivate students
Evaluate teaching
Sele
ction
Find gaps in learning
www.britishcouncil.org 7
What kinds of assessment are there?
Assignment
Conversation
PresentationSelf a
ssessm
ent
Observation
TestEssay
Portfo
lio
Peer assessment
Qu
estio
nn
air
e
Checklist
QuizGroup task
www.britishcouncil.org 8
3. Considerations in Language Assessment
What are the qualities of a good test?
www.britishcouncil.org 9
IELTS Approach to Test Qualities
• Validity • Reliability • Impact• Practicality
R
PI
V
Validity: test content
Test content matched to needs of candidates
Task-based approach to testing
Testing of integrated skills
Variety of Listening and Reading item types used
Face to face Speaking Test and direct marking
Accepting all standard varieties of native-speaker English
Validity: continual research based development
1980 ELTS introduced (English Language Testing Service)
Original 4 skills test
1989 tri-partite international partnership created, 1st delivery of IELTS
1995 major systems and content revision
2001 revised Speaking Test
2005 revised Writing assessment criteria
2007 introduction of half band for Speaking and Writing
Constants up to date: 9-band scale, skill profile & overall score, f2f
interview
Reliability of test materials
Reliability of results
Scoring system
Marker/Examiner selection and training
Double marking and EOR
Research and validation analysis
Monitoring: sample and targeted
Results benchmarked with international standards
How do you create effective test tasks and rating scales for Writing and Speaking?
Theoretical relevance
The test’s criteria and tasks should be defensible in terms of valid theories of oral performance
Discriminating power
The test criteria and tasks should enable relevant distinctions to be made between candidates
Assessability
The features selected for assessor attention should be salient enough for a single assessor to notice them
IELTS W/S Band Descriptors
Writing Speaking
Task Achievement/Response
Fluency and Coherence
Coherence and Cohesion Lexical Resource
Lexical Resource Grammatical Range and Accuracy
Grammatical Range and Accuracy
Pronunciation
www.britishcouncil.org 17
Results benchmarked with int’l standards
http://www.cambridgeenglish.org/about-us/what-we-do/international-language-standards/
Impact
Long-standing concern for the relationship between
IELTS and its contexts of use
Test development partly shaped by this concern
Early 1990s growing interest in test impact/washback
Systematic investigation of IELTS impact began in 1995
Consultation processes with stakeholders to ensure- IELTS is fit for purpose - cut scores are set appropriately
Use of GT module for immigration and employment
Standards setting events
General Medical Council, UK (June 04)
National Council for State Boards of Nursing, USA (Oct 04)
Canadian Immigration and Citizenship (May 05)
Main outcomes of IELTS Impact Studies
IELTS is a reasonable direct communicative performance test of the
four macro skills
IELTS is challenging but fair
IELTS uses authentic and appropriate content
IELTS is motivating for learners and teachers
IELTS engenders a positive effect on classroom activities and
learning processes
Practicality 可行性 48 test administrations per year
Global network of 900 test venues in over 130 countries/regions
Emphasis on standards, consistency and security
global standards
test centre selection, training, auditing
secure stock management
on-line results verification
Provision of information through a variety of channels
On-line registration
Results available 13 days after the test
Post test services
4. Q&A
www.britishcouncil.org 23
Acknowledgement
References:
Presentation for teachers event, Nick Charge
www.ielts.org