iatefl(16 apr 2016) - · pdf file4/16/2016 · 16/07/06 1 developing a...

16/07/06

1

Developingacomputer-basedspeakingtestofEnglishasaLinguaFranca

IATEFLAnnualConferenceBirmingham,April2016

YumiHato,KyotoIns0tuteofTechnology,JapanKatsunoriKanzawa,KyotoIns0tuteofTechnology,Japan

YasushiTsubota,KyotoIns0tuteofTechnology,Japan

HaruhikoMitsunaga,ShimaneUniversity,Japan

NicUnderhill,educa0onandassessmentconsultant,UK

2012 Oct. launchedprojectatKyotoIns0tuteofTechnology(KIT)

2013 Jan. heldsymposium“ExploringthePossibilityofIntroducingaSpeakingTestinUniversityEntranceExamina0ons”

Feb. startedtodrawuptestspecifica0on

Apr. startedtodevelopspeakingtestsystemwithKITresearchersininforma0ontechnology

May firstna0onalgrant,Grant-in-AidforScien0ficResearch,awarded

2014

Jul. startedtodevelopCBTadministra0onsystemandonlinera0ngsystemwithE-communica0ons,Inc.(CBTdeveloperandprovider)

2015 testadministra0on

Sep.

startedtoworkwithQQEnglish(onlineEnglishlessonproviderbasedinthePhilippines)inra0ng

2016 Apr. secondna0onalgrantawarded

Date NumberofExaminees Version

1stadministra0on

20-22Jan2015 596(551first-yearstudentsand45anchorexaminees) Ver.1,2,3

2ndadministra0on

20Dec2015 644(575first-yearstudentsand69anchorexaminees) Ver.4,5

OutlineofpresentaFon

-  Testspecifica0onPurposeTasktypesRa0ngscalesMarking

-  Preliminaryresults-  Futureplansandchallenges

Testspecifica0onhcps://kitspeakee.wordpress.com/testspec/Sampletesthcps://kitspeakee.wordpress.com/sampletest/

Testpurpose1

ToassesstowhatextenteachstudenthasacquiredthespeakingabilityinEnglishrequiredtoac0velypar0cipatein21stcenturyglobalsociety

Testpurpose2

Tosendposi0vewashbackmessages:-  Speakingisasimportantastheothercorelanguageskills.Itis

notjustlimitedtodailyconversa0onbutisaseriouscommunica0onmodeessen0altosucceedinthe21stcentury.

-  StudentsarelearningEnglishasalinguafrancaforcommunica0onbetweennon-na0vespeakersaswellaswithna0vespeakers.

-  Studentsmustlearntobeconfidentusersoftheirexis0nglanguageresourcesatanystageoftheirlanguagedevelopment,freefromanexcessiveconcerntoconformtoNSnorms.

-  Studentsmustbepreparedtospeakwithoutprepara0on,asspokeninterac0onisnormallyspontaneous.

Testpurpose3

Toevaluatethefeasibilityofincorpora0ngaspeakingtestintothecurrentEnglisheduca0onprogramme,anddevelopingaspeakingcomponentforanEnglishlanguageadmissionstesttoJapaneseuniversitypostgraduatecourses,withapossiblefutureapplica0ontoundergraduateentranceexamina0ons

16/07/06

2

Constructs

-  Theabilitytoachieveagiventasku0lisingspokenlanguageproficiencyand21stcenturyskills(crea0vityandinnova0on,cri0calthinking,problemsolving,globalawareness)

-  Confidentandfluentuseoflanguage

Tasktypes

KITtestSpec160402.docx page3

6. Tasktypes

Qno. Name Speech

sample Prompt 21stcenturyskills:Learnersshould…

Spokenlanguageproficiency:Learnersshould… Sampletask

Part11&2 Imagine 45

seconds Photo thinkcreativelyanddemonstrateoriginality speakcoherentlyandclearly Imaginewhotheowneroftheseshoesisand

whytheyarehere.3

Compare 45seconds

Twophotos

analysealternativesanddrawreasonableconclusion compare,decideandjustify

Whichofthesecarswouldyouprefertohave?Comparethecarsandexplainyourreasons.

Part24 Identify

differentvalues

45seconds

Audiodialogue&photos

understanddiversevaluesandperspectives

summariseandcontrastdifferentpointsofview

HowareSusan’sandKenji’sopinionsdifferent?[joinbigcompany/securityvsstartingownbusiness/excitinglife]

5 Takeposition

60seconds

Sameastask4

evaluateargumentsanddecideownposition stateandjustifyownposition Whichwayofthinkingdoyousupport?

Explainyourposition,givingexamples.6

Identifyproblem

45seconds

Audiodialogue&photos

interpretinformationtoIdentifyproblem describeproblem WhatistheproblemBillisfacing?[One

memberofthelabteamisn’tcollaborating]

7 Problemsolving

60seconds

Sameastask6 findsolutiontoproblem proposesolutiontoproblem IfyouwereBill,whatwouldyoudotosolve

theproblem?Part38

Planandorganise

60seconds,plusonerehearsal

identifyandorganizecomponentpartstomakeaplan

suggestaplanandseriesofstepstoachievegoal

Youwanttoorganizeavolunteergrouponcampustohelphomelesspeople.Identifythedifferentstepsyouwouldtakeandexplainhowyouwouldorganizethem.

9

Persuade

60seconds,plusonerehearsal

promoteandinfluence persuadebypresentingapositiveimageandmessage

Inaninterviewforascholarshipprogram,youareaskedtoexplainwhyyoushouldbeselected.Talkaboutyourpersonalachievementsandstrongpoints.

Part1

Ques0onno. 1&2 3

Name Imagine Compare

Speechsample 45seconds 45seconds

Prompt Photo Twophotos

21stcenturyskills:Learnersshould…

thinkcrea0velyanddemonstrateoriginality

analysealterna0vesanddrawreasonableconclusion

Spokenlanguageproficiency:Learnersshould…

speakcoherentlyandclearly compare,decideandjus0fy

Sampletask Imaginewhotheowneroftheseshoesisandwhytheyarehere.

Whichofthesecarswouldyouprefertohave?Comparethecarsandexplainyourreasons.

QuesFon1

Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.

QuesFon1

Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.

QuesFon1

Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.

16/07/06

3

QuesFon1

Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.

QuesFon1

Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.

QuesFon1

Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.

Recording

Part2Ques0onno. 4 5 6 7

Name Iden0fydifferentvalues Takeposi0on Iden0fyproblem Problemsolving

Speechsample 45seconds 60seconds 45seconds 60seconds

Prompt Audiodialogue&photos Sameastask4

Audiodialogue&photos Sameastask6

21stcenturyskills:Learnersshould…

understanddiversevaluesandperspec0ves

evaluateargumentsanddecideownposi0on

interpretinforma0ontoIden0fyproblem

findsolu0ontoproblem

Spokenlanguageproficiency:Learnersshould…

summariseandcontrastdifferentpointsofview

stateandjus0fyownposi0on describeproblem proposesolu0onto

problem

Sampletask

HowareSusan’sandKenji’sopinionsdifferent?[joinbigcompany/securityvsstarEngownbusiness/exciEnglife]

Whichwayofthinkingdoyousupport?ExplainyourposiEon,givingexamples.

WhatistheproblemBillisfacing?[Onememberofthelabteamisn’tcollaboraEng]

IfyouwereBill,whatwouldyoudotosolvetheproblem?

Part3

Ques0onno. 8 9

Name Planandorganise Persuade

Speechsample 60seconds,plusonerehearsal 60seconds,plusonerehearsal

Prompt - -

21stcenturyskills:Learnersshould…

iden0fyandorganizecomponentpartstomakeaplan promoteandinfluence

Spokenlanguageproficiency:Learnersshould…

suggestaplanandseriesofstepstoachievegoal

persuadebypresen0ngaposi0veimageandmessage

Sampletask

Youwanttoorganizeavolunteergrouponcampustohelphomelesspeople.IdenEfythedifferentstepsyouwouldtakeandexplainhowyouwouldorganizethem.

Inaninterviewforascholarshipprogram,youareaskedtoexplainwhyyoushouldbeselected.Talkaboutyourpersonalachievementsandstrongpoints.

16/07/06

4

Tasktypes

KITtestSpec160402.docx page3

6. Tasktypes

Qno. Name Speech

sample Prompt 21stcenturyskills:Learnersshould…

Spokenlanguageproficiency:Learnersshould… Sampletask

Part11&2 Imagine 45

seconds Photo thinkcreativelyanddemonstrateoriginality speakcoherentlyandclearly Imaginewhotheowneroftheseshoesisand

whytheyarehere.3

Compare 45seconds

Twophotos

analysealternativesanddrawreasonableconclusion compare,decideandjustify

Whichofthesecarswouldyouprefertohave?Comparethecarsandexplainyourreasons.

Part24 Identify

differentvalues

45seconds

Audiodialogue&photos

understanddiversevaluesandperspectives

summariseandcontrastdifferentpointsofview

HowareSusan’sandKenji’sopinionsdifferent?[joinbigcompany/securityvsstartingownbusiness/excitinglife]

5 Takeposition

60seconds

Sameastask4

evaluateargumentsanddecideownposition stateandjustifyownposition Whichwayofthinkingdoyousupport?

Explainyourposition,givingexamples.6

Identifyproblem

45seconds

Audiodialogue&photos

interpretinformationtoIdentifyproblem describeproblem WhatistheproblemBillisfacing?[One

memberofthelabteamisn’tcollaborating]

7 Problemsolving

60seconds

Sameastask6 findsolutiontoproblem proposesolutiontoproblem IfyouwereBill,whatwouldyoudotosolve

theproblem?Part38

Planandorganise

60seconds,plusonerehearsal

identifyandorganizecomponentpartstomakeaplan

suggestaplanandseriesofstepstoachievegoal

Youwanttoorganizeavolunteergrouponcampustohelphomelesspeople.Identifythedifferentstepsyouwouldtakeandexplainhowyouwouldorganizethem.

9

Persuade

60seconds,plusonerehearsal

promoteandinfluence persuadebypresentingapositiveimageandmessage

Inaninterviewforascholarshipprogram,youareaskedtoexplainwhyyoushouldbeselected.Talkaboutyourpersonalachievementsandstrongpoints.

IniFalraFngscales(Usedinthe1stadministra0on,Jan2015)

Scorera0ng

Taskachievement(45%weigh0ng)

Communica0velanguageproficiency(45%weigh0ng)

Antudeandeffort(10%weigh0ng)

5

-taskisachievedeffec0vely-contribu0onsarefullyrelevantbuttheremaybesomerepe00on

-communicateseffec0vely-producesextendeducerancesusingarangeofgrammarandvocabularydespiteapossiblelackoffluencyoraccuracy-demonstratesappropriateuseofcohesivedevices

-speaksconfidentlyandwillingly(generallycalmand/orforceful,ac0ve,posi0ve,engaged,enthusias0c,expressive,firmandasser0ve)-makesabigefforttousehis/herlanguageresources

4 Between3and5 Between3and5 Between3and5

3

-taskisachievedonlypar0allyorinadequately-contribu0onsaremostlyrelevant

-communicatessuccessfullybutmaybelimitedbylackoffluencyoraccuracy-constructsuceranceslongerthanshortphrasesusingbasicgrammarandvocabularyonly-demonstrateslimiteduseofcohesivedevices

-speakswithsomeconfidencebutissome0mesnervous,hesitantorunwilling-makessomeefforttousehis/herlanguageresources

2 Between1and3 Between1and3 Between1and3

1 -taskisnotachieved-contribu0onsaremostlynotrelevant.

-ooenfailstocommunicateduetolackoffluencyoraccuracy-producesshortuceranceswithwordsorphrasesonly-doesnotusecohesivedevices

-lacksconfidenceandunengaged-makeslicleefforttousehis/herlanguageresources

0 Nocontribu0onatall doesnotcommunicateatall Noeffortatall

RevisedraFngscales(Usedinthe2ndadministra0on,Dec2015)

Scorera0ng Taskachievement(80%weigh0ng) Taskdelivery(20%weigh0ng)

5 -taskisachievedwithsa0sfactorysuppor0ngdetail

-speaksfluentlyenoughtobecomprehensibleandwithsomeconfidence-given0meiswelluseddespitesomehesita0onorrepe00on

4 Between3and5 Between3and5

3 -taskispar0allyachieved,orisachievedwithminimalsuppor0ngdetail

-justfluentenoughtobecomprehensiblemostofthe0mebutmaylackconfidence-given0meisnoteffec0velyusedbecauseoffrequenthesita0onorrepe00on

2 Between1and3 Between1and3

1 -somerelevantwordsbuttaskisnotachieved

-isnotcomprehensiblemostofthe0me

0 -norelevantcontribu0on -nocomprehensiblecontribu0on

Stepsforensuringwashback

-  No0fiedstudentsofadministra0onofspeakingtestatfirst-yearstudentorienta0on

-  VisitedeverycompulsoryEnglishclasstoexplainpurposeofthetest,tasktypes,ra0ngscales,andhowtoprepareforthetest

-  DistributedJapanesetransla0onofinforma0onaboutthetestandmadeitavailableonwebsite

16/07/06

5

Marking

1stadministra0on 2ndadministra0on

Testdate 20-22Jan2015 19Dec2015

Numberofresponses 9ques0ons×670examinees=6,030responses

9ques0ons×713examinees=6,417responses

Raters7na0ve-speakers7non-na0vespeakers(Japanese-speakers)

9na0ve-speakers9non-na0vespeakers(inthePhillipines)

Ra0ngprocess*Doublemarking

Oneratermarkedallthenineresponsesgivenbyoneexaminee.

Eachratermarkedalltheresponsesforonetaskonly.

Dura0onofmarking 2weeks 2weeks

!Ver0calscaleforoveralldata

Exactmatches EsFmatedmatches

35.6% 33.3%

Exactmatches EsFmatedmatches

41.3% 39.7%

Inter-rateragreement(NSandJapanese)

Inter-rateragreement(PhilippinesandJapanese)

(Es0matedbyFacets(Linacre,1989))

11-1

Standardizedscoredistribu0onbasedonMany-facetItemResponseTheory(MIRT)

11-2

KeyfeaturesofKITSpeakingTest

-  ELF-  Focusonwashback-  Tailoredtothestudentcontext-  Sta0s0calvalida0on-  CBT(testadministra0onandra0ngsystems)

ELFfeatures

-  Purpose-  Constructs-  Tasktypes-  Dialoguerecordings-  Ra0ngscales-  Ra0ngprocess

Futurechallenges

-  Ra0ngscales-  Ratertraining-  Feedbackform,scorereport-  Linkteaching/learningandtes0ng-  Opera0onalissuesforentranceexams

16/07/06

6

Thankyou!

Gettheslideshere:hcps://goo.gl/6E5KsE＊

YumiHato:hato@kit.ac.jpHaruhikoMitsunaga:mitsunaga-h@soc.shimane-u.ac.jp

KatsunoriKanzawa:kanzawa@kit.ac.jpNicUnderhill:nic.underhill@b0nternet.com