Download - Models of language proficiency (Part 1)
LING 906: Language Testing & Evaluation
Assessing language Assessing language proficiency: concepts proficiency: concepts and issuesand issues
A/P Mehdi RiaziA/P Mehdi Riazi
Macquarie UniversityMacquarie University
Review of last week’s Review of last week’s lecturelectureType, uses, purposes, and functions of Type, uses, purposes, and functions of
language testslanguage tests– Screening or selectionScreening or selection– DiagnosisDiagnosis– PlacementPlacement– Etc.Etc.
Two major Two major purposespurposes
Students’ mastery of specific objectives
Students’ overall (global) language ability
AchievementAchievement ProficiencyProficiency
Achievement TestsAchievement Tests
Teacher-made
Teacher-made
Standardised
Standardised
Achievement TestsAchievement Tests
Teacher-made
Teacher-made
Standardised
Standardised
The National Assessment Program – Literacy and Numeracy (NAPLAN) commenced in Australian schools. Every year, all students in Years 3, 5, 7 and 9 are assessed on the same days using national tests in Reading, Writing, Language Conventions (Spelling, Grammar and Punctuation) and Numeracy
Persuasive genreAudience The writer’s capacity to
orient, engage and persuade the reader
Text structure The organisation of the structural components of a persuasive text (introduction, body and conclusion) into an appropriate and effective text structure
Ideas The selection, relevance and elaboration of ideas for a persuasive argument
Persuasive devices The use of a range of persuasive devices to enhance the writer’s position and persuade the reader
Vocabulary The range and precision of contextually appropriate language choices
Descriptions of the 10 Writing criteria (standards)
Cohesion The control of multiple threads and relationships across the text, achieved through the use of grammatical elements (referring words, text connectives, conjunctions) and lexical elements (substitutions, repetitions, word associations)
Paragraphing The segmenting of text into paragraphs that assists the reader to follow the line of argument
Sentence structure The production of grammatically correct, structurally sound and meaningful sentences
Punctuation The use of correct and appropriate punctuation to aid the reading of the text
Spelling The accuracy of spelling and the difficulty of the words used
Our focus in these Our focus in these two sessionstwo sessions
Language proficiency Language proficiency tests tests –The conceptThe concept– ModelsModels– ExamplesExamples
Approaches to prof. Approaches to prof. test constructiontest construction
1.1. Target Language Use (TLU) Target Language Use (TLU) tasks tasks (what candidates are required to do (what candidates are required to do in the TLU and then creating the test)in the TLU and then creating the test)
2.2. Using proficiency frameworks Using proficiency frameworks (e.g., CEFR: hierarchically organised (e.g., CEFR: hierarchically organised description of language ability)description of language ability)
3.3. Construct-drivenConstruct-driven (defining or (defining or modeling language proficiency and modeling language proficiency and then creating the test accordingly)then creating the test accordingly)
Task-based Task-based assessmentassessment
TB test development TB test development procedureprocedure
• Define TLU domainDefine TLU domain• Select tasks from TLU domain to Select tasks from TLU domain to use as a basis for assessment use as a basis for assessment taskstasks
• Develop appropriate tasks for Develop appropriate tasks for assessmentassessment
The Occupational The Occupational English Test (OET)English Test (OET) OET is a language test for OET is a language test for
health practitioners who have health practitioners who have qualified to practise in one country qualified to practise in one country and wish to gain provisional and wish to gain provisional registration to practise their registration to practise their profession in another which has profession in another which has an English-speaking context. an English-speaking context.
The Test assesses English language The Test assesses English language proficiency proficiency as it is used in medical as it is used in medical and health professionsand health professions..
Approaches to prof. Approaches to prof. test constructiontest construction
1.1. Target Language Use (TLU) Target Language Use (TLU) tasks tasks (what candidates are required to do (what candidates are required to do in the TLU and then creating the test)in the TLU and then creating the test)
2.2. Using proficiency frameworks Using proficiency frameworks (e.g., CEFR: hierarchically organised (e.g., CEFR: hierarchically organised description of language ability)description of language ability)
3.3. Construct-drivenConstruct-driven (defining or (defining or modeling language proficiency and modeling language proficiency and then creating the test accordingly)then creating the test accordingly)
Common European Common European Framework (CEF)Framework (CEF)
Level Definition
C2Proficient userC1
B2Independent user
B1
A2Basic userA1
Construct-driven Construct-driven approachapproach Defining and modeling the Defining and modeling the
construct of language proficiencyconstruct of language proficiency
Operationalisation of the model Operationalisation of the model through test design, test through test design, test specifications, and scoring rubricsspecifications, and scoring rubrics
The Concept of The Concept of Language ProficiencyLanguage Proficiency
Proficiency is a “construct” or Proficiency is a “construct” or “trait” like “intelligence” or “trait” like “intelligence” or “motivation”“motivation”
Constructs or traits are invisible Constructs or traits are invisible
How to assess the constructs?How to assess the constructs?
The process The process involvedinvolved
Construct
Definition (modeling)
Operationalisation(Test/assessment)
Constructs or Constructs or TraitsTraits
Intelligence
Trait or construct
Wechsler Adult Intelligence Scale
Scores (Operational definition of intelligence)
Definition (Model)
Proficiency
Trait or construct
IELTS or TOEFL test
Scores (Operational definition of proficiency)
Definition (Model)
The construct and its operationalisation
The concept of The concept of construct validityconstruct validity Validity:Validity: a) to what extent a test a) to what extent a test
tests what it claims to test, b) to tests what it claims to test, b) to what extent our interpretation of what extent our interpretation of the test score reflects the test’s the test score reflects the test’s contentcontent
Construct validity: Construct validity: to what to what extent the test represents the extent the test represents the underlying constructunderlying construct
How is Proficiency How is Proficiency defined?defined? Different scholars and Different scholars and test specialists have test specialists have defined “defined “language language proficiencyproficiency” ” differentlydifferently
Alternative Alternative TermsTerms ProficiencyProficiency CompetenceCompetence Communicative CompetenceCommunicative Competence Knowledge of LanguageKnowledge of Language Language AbilityLanguage Ability Communicative Language Communicative Language
Ability (CLA)Ability (CLA)
Why do we need a Why do we need a model?model? Tests of proficiency need to Tests of proficiency need to
be based on a sound and be based on a sound and acceptable underlying acceptable underlying model or theory of language model or theory of language proficiency. Otherwise, proficiency. Otherwise, “construct validity” “construct validity” of the of the test would be under test would be under question.question.
Models of L2 Models of L2 ProficiencyProficiency1.1. Skills and components model Skills and components model (Lado, (Lado,
1960; Carroll, 1968)1960; Carroll, 1968)
2.2. Unitary competence hypothesisUnitary competence hypothesis (Oller, 1976)(Oller, 1976)
3.3. BICS/CALPBICS/CALP (Cummins, 1979; 1983)(Cummins, 1979; 1983)
4.4. Communicative competenceCommunicative competence (Canale (Canale and Swain, 1980; Canale, 1983)and Swain, 1980; Canale, 1983)
5.5. Communicative language ability Communicative language ability (Bachman, 1990; Bachman & Palmer, (Bachman, 1990; Bachman & Palmer, 1996)1996)
Skills & Skills & Component ModelComponent ModelUnderlying theories:Underlying theories:
1. Structural Linguistics1. Structural Linguistics
2. Behavioral Psychology2. Behavioral Psychology
Lado (1961)Lado (1961)ProficiencyProficiency
Skills Components
Listening
Speaking
Reading
Writing
Grammar
Vocabulary
Phonology
Features of tests in Features of tests in skills/component skills/component modelmodel
Discrete-point, Multiple-choice Discrete-point, Multiple-choice itemsitems
Different sections on different Different sections on different skills and components, but mostly skills and components, but mostly Listening, Reading, Gr., & Voc.Listening, Reading, Gr., & Voc.
Major emphasis on “linguistic Major emphasis on “linguistic competence” (knowledge of lang)competence” (knowledge of lang)
Features of tests in Features of tests in skills/component skills/component modelmodel
Recognition items & testsRecognition items & tests Usage rather than useUsage rather than use A good model for curriculum A good model for curriculum
and testingand testing Difficult to make; easy to scoreDifficult to make; easy to score Objective scoringObjective scoring
Proficiency
Trait or construct
First generation of TOEFL test
Scores (Operational definition of proficiency)
Definition (Model)
Skills/Components
First generation of First generation of TOEFLTOEFL
SkillsSkills– ListeningListening– ReadingReading
ComponentsComponents– Phonology (segmental & Phonology (segmental &
suprasegmental)suprasegmental)– Grammar (rules of language)Grammar (rules of language)– Vocabulary (knowledge of words)Vocabulary (knowledge of words)
Oller (1976)Oller (1976) Based on principal components Based on principal components
analysis of scores obtained from analysis of scores obtained from numerous tests (vocabulary, numerous tests (vocabulary, grammar, phonology, reading, grammar, phonology, reading, etc.), Oller (1976) defined etc.), Oller (1976) defined proficiency in terms of one proficiency in terms of one general factor (g-factor)general factor (g-factor)
Unitary Comp. Unitary Comp. HypothesisHypothesis
This representation This representation by Oller is known as by Oller is known as the unitary the unitary competence competence hypothesis (UCH)hypothesis (UCH)
Tests based on Tests based on UCHUCH Integrative tests Integrative tests
including:including:1.1. DictationDictation2.2. Cloze-testCloze-test3.3. CompositionComposition4.4. Oral interviewOral interview
Criticisms on Criticisms on UCHUCH
Conceptually:Conceptually: Derivation of one Derivation of one factor is the result of the nature of factor is the result of the nature of the data used (conscious & explicit the data used (conscious & explicit features of language & not features of language & not functional & sociolinguistics functional & sociolinguistics aspects)aspects)
Methodologically:Methodologically: PCA is PCA is inappropriate. It overestimates the inappropriate. It overestimates the sig. of the first derived factor.sig. of the first derived factor.
Criticisms on UCHCriticisms on UCH
Oller (1983) himself:Oller (1983) himself: One-factor representation of One-factor representation of
lang proficiency is implausible lang proficiency is implausible andand
that a general factor can be that a general factor can be divided into relatively more divided into relatively more analytic componentsanalytic components