LING 906: Language Testing & Evaluation

Assessing language Assessing language proficiency: concepts proficiency: concepts and issuesand issues

A/P Mehdi RiaziA/P Mehdi Riazi

Macquarie UniversityMacquarie University

Review of last week’s Review of last week’s lecturelectureType, uses, purposes, and functions of Type, uses, purposes, and functions of

language testslanguage tests– Screening or selectionScreening or selection– DiagnosisDiagnosis– PlacementPlacement– Etc.Etc.

Two major Two major purposespurposes

Students’ mastery of specific objectives

Students’ overall (global) language ability

AchievementAchievement ProficiencyProficiency

Achievement TestsAchievement Tests

Teacher-made

Teacher-made

Standardised

Standardised

Achievement TestsAchievement Tests

Teacher-made

Teacher-made

Standardised

Standardised

The National Assessment Program – Literacy and Numeracy (NAPLAN) commenced in Australian schools. Every year, all students in Years 3, 5, 7 and 9 are assessed on the same days using national tests in Reading, Writing, Language Conventions (Spelling, Grammar and Punctuation) and Numeracy

Persuasive genreAudience The writer’s capacity to

orient, engage and persuade the reader

Text structure The organisation of the structural components of a persuasive text (introduction, body and conclusion) into an appropriate and effective text structure

Ideas The selection, relevance and elaboration of ideas for a persuasive argument

Persuasive devices The use of a range of persuasive devices to enhance the writer’s position and persuade the reader

Vocabulary The range and precision of contextually appropriate language choices

Descriptions of the 10 Writing criteria (standards)

Cohesion The control of multiple threads and relationships across the text, achieved through the use of grammatical elements (referring words, text connectives, conjunctions) and lexical elements (substitutions, repetitions, word associations)

Paragraphing The segmenting of text into paragraphs that assists the reader to follow the line of argument

Sentence structure The production of grammatically correct, structurally sound and meaningful sentences

Punctuation The use of correct and appropriate punctuation to aid the reading of the text

Spelling The accuracy of spelling and the difficulty of the words used

Our focus in these Our focus in these two sessionstwo sessions

Language proficiency Language proficiency tests tests –The conceptThe concept– ModelsModels– ExamplesExamples

Approaches to prof. Approaches to prof. test constructiontest construction

1.1. Target Language Use (TLU) Target Language Use (TLU) tasks tasks (what candidates are required to do (what candidates are required to do in the TLU and then creating the test)in the TLU and then creating the test)

2.2. Using proficiency frameworks Using proficiency frameworks (e.g., CEFR: hierarchically organised (e.g., CEFR: hierarchically organised description of language ability)description of language ability)

3.3. Construct-drivenConstruct-driven (defining or (defining or modeling language proficiency and modeling language proficiency and then creating the test accordingly)then creating the test accordingly)

Task-based Task-based assessmentassessment

TB test development TB test development procedureprocedure

• Define TLU domainDefine TLU domain• Select tasks from TLU domain to Select tasks from TLU domain to use as a basis for assessment use as a basis for assessment taskstasks

• Develop appropriate tasks for Develop appropriate tasks for assessmentassessment

The Occupational The Occupational English Test (OET)English Test (OET) OET is a language test for OET is a language test for

health practitioners who have health practitioners who have qualified to practise in one country qualified to practise in one country and wish to gain provisional and wish to gain provisional registration to practise their registration to practise their profession in another which has profession in another which has an English-speaking context. an English-speaking context.

The Test assesses English language The Test assesses English language proficiency proficiency as it is used in medical as it is used in medical and health professionsand health professions..

Approaches to prof. Approaches to prof. test constructiontest construction

1.1. Target Language Use (TLU) Target Language Use (TLU) tasks tasks (what candidates are required to do (what candidates are required to do in the TLU and then creating the test)in the TLU and then creating the test)

2.2. Using proficiency frameworks Using proficiency frameworks (e.g., CEFR: hierarchically organised (e.g., CEFR: hierarchically organised description of language ability)description of language ability)

3.3. Construct-drivenConstruct-driven (defining or (defining or modeling language proficiency and modeling language proficiency and then creating the test accordingly)then creating the test accordingly)

Common European Common European Framework (CEF)Framework (CEF)

Level Definition

C2Proficient userC1

B2Independent user

B1

A2Basic userA1

Construct-driven Construct-driven approachapproach Defining and modeling the Defining and modeling the

construct of language proficiencyconstruct of language proficiency

Operationalisation of the model Operationalisation of the model through test design, test through test design, test specifications, and scoring rubricsspecifications, and scoring rubrics

The Concept of The Concept of Language ProficiencyLanguage Proficiency

Proficiency is a “construct” or Proficiency is a “construct” or “trait” like “intelligence” or “trait” like “intelligence” or “motivation”“motivation”

Constructs or traits are invisible Constructs or traits are invisible

How to assess the constructs?How to assess the constructs?

The process The process involvedinvolved

Construct

Definition (modeling)

Operationalisation(Test/assessment)

Constructs or Constructs or TraitsTraits

Intelligence

Trait or construct

Wechsler Adult Intelligence Scale

Scores (Operational definition of intelligence)

Definition (Model)

Proficiency

Trait or construct

IELTS or TOEFL test

Scores (Operational definition of proficiency)

Definition (Model)

The construct and its operationalisation

The concept of The concept of construct validityconstruct validity Validity:Validity: a) to what extent a test a) to what extent a test

tests what it claims to test, b) to tests what it claims to test, b) to what extent our interpretation of what extent our interpretation of the test score reflects the test’s the test score reflects the test’s contentcontent

Construct validity: Construct validity: to what to what extent the test represents the extent the test represents the underlying constructunderlying construct

How is Proficiency How is Proficiency defined?defined? Different scholars and Different scholars and test specialists have test specialists have defined “defined “language language proficiencyproficiency” ” differentlydifferently

Alternative Alternative TermsTerms ProficiencyProficiency CompetenceCompetence Communicative CompetenceCommunicative Competence Knowledge of LanguageKnowledge of Language Language AbilityLanguage Ability Communicative Language Communicative Language

Ability (CLA)Ability (CLA)

Why do we need a Why do we need a model?model? Tests of proficiency need to Tests of proficiency need to

be based on a sound and be based on a sound and acceptable underlying acceptable underlying model or theory of language model or theory of language proficiency. Otherwise, proficiency. Otherwise, “construct validity” “construct validity” of the of the test would be under test would be under question.question.

Models of L2 Models of L2 ProficiencyProficiency1.1. Skills and components model Skills and components model (Lado, (Lado,

1960; Carroll, 1968)1960; Carroll, 1968)

2.2. Unitary competence hypothesisUnitary competence hypothesis (Oller, 1976)(Oller, 1976)

3.3. BICS/CALPBICS/CALP (Cummins, 1979; 1983)(Cummins, 1979; 1983)

4.4. Communicative competenceCommunicative competence (Canale (Canale and Swain, 1980; Canale, 1983)and Swain, 1980; Canale, 1983)

5.5. Communicative language ability Communicative language ability (Bachman, 1990; Bachman & Palmer, (Bachman, 1990; Bachman & Palmer, 1996)1996)

Skills & Skills & Component ModelComponent ModelUnderlying theories:Underlying theories:

1. Structural Linguistics1. Structural Linguistics

2. Behavioral Psychology2. Behavioral Psychology

Lado (1961)Lado (1961)ProficiencyProficiency

Skills Components

Listening

Speaking

Reading

Writing

Grammar

Vocabulary

Phonology

Features of tests in Features of tests in skills/component skills/component modelmodel

Discrete-point, Multiple-choice Discrete-point, Multiple-choice itemsitems

Different sections on different Different sections on different skills and components, but mostly skills and components, but mostly Listening, Reading, Gr., & Voc.Listening, Reading, Gr., & Voc.

Major emphasis on “linguistic Major emphasis on “linguistic competence” (knowledge of lang)competence” (knowledge of lang)

Features of tests in Features of tests in skills/component skills/component modelmodel

Recognition items & testsRecognition items & tests Usage rather than useUsage rather than use A good model for curriculum A good model for curriculum

and testingand testing Difficult to make; easy to scoreDifficult to make; easy to score Objective scoringObjective scoring

Proficiency

Trait or construct

First generation of TOEFL test

Scores (Operational definition of proficiency)

Definition (Model)

Skills/Components

First generation of First generation of TOEFLTOEFL

SkillsSkills– ListeningListening– ReadingReading

ComponentsComponents– Phonology (segmental & Phonology (segmental &

suprasegmental)suprasegmental)– Grammar (rules of language)Grammar (rules of language)– Vocabulary (knowledge of words)Vocabulary (knowledge of words)

Oller (1976)Oller (1976) Based on principal components Based on principal components

analysis of scores obtained from analysis of scores obtained from numerous tests (vocabulary, numerous tests (vocabulary, grammar, phonology, reading, grammar, phonology, reading, etc.), Oller (1976) defined etc.), Oller (1976) defined proficiency in terms of one proficiency in terms of one general factor (g-factor)general factor (g-factor)

Unitary Comp. Unitary Comp. HypothesisHypothesis

This representation This representation by Oller is known as by Oller is known as the unitary the unitary competence competence hypothesis (UCH)hypothesis (UCH)

Tests based on Tests based on UCHUCH Integrative tests Integrative tests

including:including:1.1. DictationDictation2.2. Cloze-testCloze-test3.3. CompositionComposition4.4. Oral interviewOral interview

Criticisms on Criticisms on UCHUCH

Conceptually:Conceptually: Derivation of one Derivation of one factor is the result of the nature of factor is the result of the nature of the data used (conscious & explicit the data used (conscious & explicit features of language & not features of language & not functional & sociolinguistics functional & sociolinguistics aspects)aspects)

Methodologically:Methodologically: PCA is PCA is inappropriate. It overestimates the inappropriate. It overestimates the sig. of the first derived factor.sig. of the first derived factor.

Criticisms on UCHCriticisms on UCH

Oller (1983) himself:Oller (1983) himself: One-factor representation of One-factor representation of

lang proficiency is implausible lang proficiency is implausible andand

that a general factor can be that a general factor can be divided into relatively more divided into relatively more analytic componentsanalytic components