language testing (ppt)

8/10/2019 Language Testing (PPT)

1/45

Language Testing

Liu Jianda


2/45

Syllabus

Understand the general considerations that must be addressed in thedevelopment of new tests or the selection of existing language tests;Make their own judgements and decisions about either selecting anexisting language test or developing a new language test;Familiarise themselves with the fundamental issues, approaches, andmethods used in measurement and evaluation;Design, develop, evaluate and use language tests in ways that are

appropriate for a given purpose, context, and group of test takers;Understand the future development of language testing and theapplication of IT to computerized language testing.

It is expected that, by the end of this module, participants should beable to do the following :


3/45

In order to achieve these objectives, the module gives participants the opportunityto develop the following skills: writing test items collecting test data and conducting item analysis evaluating language tests with regard to validity and reliability

This is done by considering a wide range of issues and topics related to languagetesting. These include the following :General concepts in language testing and evaluationEvaluation of a language test: reliability and validityCommunicative approach to language testing

Design of a language testItem writing and item analysisInterpreting test resultsItem response theory and its applicationsComputerized language testing and its future development

Syllabus


4/45

Class Schedule1 Basic concepts in language testing2 Test validation: reliability and validity (1)3 Test validation: reliability and validity (2)4 Test construction (1)5 Test construction (2)6 Test construction (3)7 Test Construction (4)8 Test Construction (5)9 Test Construction (6)10 Rasch analysis (1)11 Rasch analysis (2)12 Language testing and modern technology


5/45

One 5000 6000 word paper on languagetesting

Collaborative work:You ll be divided into group of four to complete

the development of a test paper. Each of youwill be responsible for one part of the testpaper. But each part should contributeequally to the whole test paper. Therefore,besides developing your part, you need tocome together to discuss the whole testpaper in terms of reliability and validity.

Assessment


6/45

Bachman, L. F. & Palmer, A. (1996). Language Testing in

Practice . Oxford: Oxford University Press.

Brown, J. D. (1996). Testing in Language Programs . UpperSaddle River, NJ: Prentice Hall Regents.Li, X. (1997). The Science and Art of Language Testing .Changsha: Hunan Educational Press.McNamara, T. (1996). Measuring second language

performance . London ; New York: Longman.

Website:http://www.clal.org.cn/personal/testing/Leeds

Course books
http://www.clal.org.cn/personal/testing/Leedshttp://www.clal.org.cn/personal/testing/Leeds


7/45


8/45

Spolsky (1978) classified the developmentof language testing into three periods, ortrends: the prescientific period the psychometric/structuralist period the integrative/sociolinguistic period.

. A short history of language testing


9/45

grammar-translation approaches tolanguage teachingtranslation and free composition testsdifficult to score objectivelyno statistical techniques applied tovalidate the testssimple, but unfair to students

The prescientific period


10/45

The psychometric-structuralist period

audio-lingual and related teaching methodsobjectivity, reliability, and validity of tests

consideredmeasure discrete structure pointsmultiple-choice format (standardized tests)

follow scientific principles, have trainedlinguists and language testers


11/45

The integrative-sociolinguistic period

communicative competenceChomsky s (1965) distinction of competence and performance

Competence: an ideal speaker-listener s knowledge of the rules of thelanguage;

performance: the actual use of language in concrete situations Hymes s (1972) proposal of communicative competence

the ability of native speakers to use their language in ways that are not onlylinguistically accurate but also socially appropriate.

Canale & Swain s (1980) framework of communicative competence:Grammatical competence, mastery of the language code such as morphology,lexis, syntax, semantics, phonology;Sociolinguistic competence, mastery of appropriate language use in different

sociolinguistic contexts;Discourse competence, mastery of how to achieve coherence and cohesion inspoken and written communicationStrategic competence, mastery of communication strategies used tocompensate for breakdowns in communication and to enhance theeffectiveness of communication.


12/45


13/45

Ollers (1979) pragmatic proficiency test:Temporally and sequentially consistent with thereal world occurrences of language formsLinking to a meaningful extralinguistic contextfamiliar to the testees

Clark s (1978) direct assessment:approximating to the greatest extent thetesting context to the real worldCloze test and dictation (Yang, 2002b)Communicative testing or to testcommunicatively



14/45

Performance tests (Brown, Hudson, Norris, &Bonk, 2002; Norris, 1998)

Not discrete-point in natureIntegrating two or more of the language skills oflistening, speaking, reading, writing, and otheraspects like cohesion and coherence,

suprasegmentals, paralinguistics, kinesics,pragmatics, and cultureTask-based: essays, interviews, extensive readingtasks



15/45

Performance TestsThree characteristics:

The task should:be based on needs analysis (What criteria should be

used? What content and context? How should expertsbe used?)be as authentic as possible with the goal of measuringreal-world activitiessometimes have collaborative elements that stimulatecommunicative interactions

be contextualized and complexintegrate skills with contentbe appropriate in terms of number, timing, andfrequency of assessmentbe generally non-intrusive, that is, be aligned with thedaily actions in the language classroom


16/45


17/45

Development graph (Li, 1997: 5)


18/45


19/45


20/45

Development since 1990

Communicative language testing(Weir, 1990)Reliability and validitySocial functions of language testing


21/45

Ethical language testing

Washback (impact) (Qi, 2002; Wall, 1997)impact: effects of tests on individuals, policies or practices within the classroom,the school, the educational system or society as a wholewashback: effects of tests on language teaching and learningWays of investigating washback:

analyses of test resultsteachers and students accounts of what takes place in the classroom (questionnaires andinterviews)classroom observation

Ethics of test useuse with care (Spolsky, 1981: 20)codes of practice

Professionalization of the fieldtraining of professionalsdevelopment of standards of practice and mechanism for their implementation andenforcement

Critical language testingput language testing in the society


22/45

Factors affecting performance ofexaminees Communicativelanguage ability

TEST SCORE

Test methodfacets

Personalattributes

Randomfactors


23/45


Testing interlanguage pragmatic knowledgecurrently on research levelfocus on method validation

web-based test by RoeverComputerized language testing

Item bankingComputer-assisted language testingComputerized adaptive language testing

Test items adapted for individualsTest ends when examinee s ability is determinedTest time very shorter

Web-based testingPhonepass testing


24/45


Language testing and secondlanguage acquisition (Bachman &

Cohen, 1998)Help to define construct of languageabilityUse findings of language testing to provehypotheses in SLAProvide SLA researchers with testing andstandards of testing


25/45

Development of research methodology

Factor analysisThe main applications of factor analytic

techniques are:(1) to reduce the number of variables and(2) to detect structure in the relationshipsbetween variables, that is to classify variables .

Therefore, factor analysis is applied as adata reduction or structure detectionmethod


26/45

Generalizability theory (Bachman, 1997;Bachman, Lynch, & Mason, 1995)

Estimating the relative effects of differentfactors on test scores (facets)

The most generalizable indicator of anindividual s language ability is the universescore , however, in real world, we can onlyobtain scores from a limited sample ofmeasures, so we need to estimate thedependability of a given observed score as anestimate of the universe score.


27/45

Two stages are involved in applying G-theory totest development

G-studyThe purpose is to estimate the effects of the variousfacets in the measurement procedure (usually

conducted in pretesting).e.g. persons (differences in individuals speaking ability),raters (differences in severity among raters), tasks(differences in difficulty of tasks);

two-way interactions:task x rater different raters are rating the different tasksdifferentlyperson x task some tasks are differentially diffucult fordifferent groups of test takers (source of bias)person x rater some raters score the performance ofdifferent groups of test takers differently (indication of raterbias)


28/45

Two stages are involved in applying G-theory to test development

D-studyThe purpose is to design an optimal measure forthe interpretations or decisions that are to be made

on the basis of the test scores (estimation ofdependability).Generalizability coefficient (G coefficient) providesan estimate of the proportion of an individual sobserved score that can be attributed to his or heruniverse score, taking into consideration the effectsof the different conditions of measurementspecified in the universe of generalization. But it isappropriate for norm-referenced tests.For criterion-referenced tests, use phi coefficient.(GENOVA)


29/45

Item response theory (Rasch model)

It enables us to estimate the statisticalproperties of items and the abilities oftest takers so that these are notdependent upon a particular group of testtakers or a particular form of a test. It is

widely used in large-scale standardizedtest.


30/45


31/45

Basic procedures in SEM (Example from

Purpura, 1998) Examine the relationships between strategy use andsecond language test performance.Design two questionnaires for cognitive strategiesand metacognitive strategies (40 items)

Ask respondents to answer the questionnairesRespondents take a foreign language testCluster the 40 items to measure several variables

Compute the reliability of the variablesConduct factor analysis to identify factorsConduct SEM analysis (AMOS, EQS, LISREL)


32/45


33/45

Qualitative method

Verbal report (think-aloud,introspective)ObservationQuestionnaires and interviews

Discourse analysis


34/45

3. Classification of language tests

According to familiesNorm-referenced testsCriterion-referenced tests


35/45

Norm-referenced tests

Measure global language abilities (e.g.listening, reading speaking, writing)Score on a test is interpreted relative tothe scores of all other students whotook the test

Normal distribution


36/45


37/45

Norm-referenced tests

Students know the format of the testbut do not know what specific content

or skill will be tested A few relatively long subtests with avariety of question contents


38/45

Criterion-referenced testsMeasure well-defined and fairly specific objectivesInterpretation of scores is considered absolutewithout referring to other students scoresDistribution of scores need not to be normalStudents know in advance what types of questions,tasks, and content to expect for the test

A series of short, well-defined subtests with similar

question contents


39/45

According to decision purposes

Proficiency testsPlacement tests

Achievement testsDiagnostic tests


40/45

Proficiency tests

Test students general levels of languageproficiency

The test must provide scores that form awide distribution so that interpretations of thedifferences among students will be as fair aspossibleCan dramatically affect students lives, soslipshod decision making in this area wouldbe particularly unprofessional


41/45

Placement tests

Group students of similar ability levels(homogeneous ability levels)Help decide what each student sappropriate level will be within aspecific programRight tests for right purposes


42/45


43/45

Diagnostic tests Aimed at fostering achievement by promoting strengths andeliminating the weaknesses of individual studentsRequire more detailed information about the very specific areasin which students have strengths and weaknessesCriterion-referenced, conducted at the beginning or in themiddle of a language courseCan be diagnostic at the beginning or in the middle butachievement test at the endPerhaps the most effective use of a diagnostic test is to reportthe performance level on each objective (in a percentage) toeach student so that he or she can decide how and where toinvest time and energy most profitably


44/45


45/45

Public examinations vs.classroom tests

Purpose: proficiency vs. achievement(placement, diagnostic)

Format: standardized vs. open(objective vs. subjective)Scale: large-scale vs. small-scale (self-assessment)Scores: normality, backwash

language testing (ppt)

Documents