dirgin, james. january 2014. please seek the author’s permission for distribution or commercial...

Dirgin, James. January 2014. Dirgin, James. January 2014. Please seek the author’s permission for Please seek the author’s permission for

distribution or commercial use.distribution or commercial use.

ILR Government Skill Level DescriptorsILR Government Skill Level Descriptorsforfor

Foreign Language Education & Faculty Training Foreign Language Education & Faculty Training Curriculum DesignCurriculum Design

Assessment DevelopmentAssessment Development

James DirginJames Dirgin

Washington DCWashington DCJanuary 2014January 2014



The Goal of the PresentationThe Goal of the Presentation

To familiarize with the ILR (Interagency Language Roundtable) To familiarize with the ILR (Interagency Language Roundtable) scale and the skill descriptors in generalscale and the skill descriptors in general

To understand the To understand the tasks, content, and accuracy (TCA) tasks, content, and accuracy (TCA) statements in each levelstatements in each level

To understand the transfer of TCA to test itemsTo understand the transfer of TCA to test items

To distinguish item difficulty based on the ILRTo distinguish item difficulty based on the ILR

Have fun… Have fun…



The ILRThe ILR(Interagency Language Roundtable)(Interagency Language Roundtable)

Created in the 60s to serve the USG for defining Created in the 60s to serve the USG for defining

foreign language proficiencyforeign language proficiency First used by the diplomatic and intelligence agenciesFirst used by the diplomatic and intelligence agencies

currently over 100 memberscurrently over 100 members Focuses on Focuses on what a person can dowhat a person can do

by using a “foreign language” by using a “foreign language”

regardless of how it is “learned”regardless of how it is “learned”

when interacting with “foreigners”when interacting with “foreigners”

in a “foreign setting”in a “foreign setting” Used for measuring Used for measuring non-nativenon-native (*) user abilities (*) user abilities




Six Base Levels (Levels 0 through 5)Six Base Levels (Levels 0 through 5) Five plus levels (Levels 0+ through 4+)Five plus levels (Levels 0+ through 4+) Increased accuracy within a level leads to plus levelsIncreased accuracy within a level leads to plus levels

Scale: Courtesy of the FBI Testing DivisionScale: Courtesy of the FBI Testing Division



UNDERLYING PRINCIPLES UNDERLYING PRINCIPLES OF THE ILR SYSTEMOF THE ILR SYSTEM

It is a It is a criterion-referenced proficiencycriterion-referenced proficiency scale scale Describes Describes functionalfunctional performances performances Focuses on Focuses on bothboth the abilities the abilities andand deficiencies deficiencies Approaches the functionality from three overlapping Approaches the functionality from three overlapping

planes: planes: globalglobal tasks, content and accuracy tasks, content and accuracy It is a It is a range*range* system system It is a It is a threshold*threshold* system system Each level consists of a Each level consists of a plus level plus level rangerange Each level Each level subsumessubsumes every level below it every level below it It is presented in It is presented in ever-expandingever-expanding intervals intervals



THE ILR SYSTEMTHE ILR SYSTEM The scale is an “output oriented” scale – The scale is an “output oriented” scale –

founded upon global proficiency functionalityfounded upon global proficiency functionality

(knowledge, cognitive skills, experience, education???)(knowledge, cognitive skills, experience, education???)

It approaches functionality from three overlapping It approaches functionality from three overlapping and interconnected planes: and interconnected planes:

globalglobal tasks tasks

content content

accuracyaccuracy

ALLALL THREE CRITERIA MUST APPLIED TO THREE CRITERIA MUST APPLIED TO EVERYTHINGEVERYTHING IN IN EACH AND EVERY INSTANCEEACH AND EVERY INSTANCE

Functionality is defined through “can do” and “cannot Functionality is defined through “can do” and “cannot do” statements in the ILR skill descriptorsdo” statements in the ILR skill descriptors



THE ILR SYSTEM THE ILR SYSTEM (Cont’d) (Cont’d)

Threshold (minimum requirement) systemThreshold (minimum requirement) system

Level definitions describe the lowest required Level definitions describe the lowest required performance (performance (thresholdthreshold)* for meeting the criteria…)* for meeting the criteria…

Any user at a given level level must sustain criteria for Any user at a given level level must sustain criteria for that level according to the ILR accuracy requirements…that level according to the ILR accuracy requirements…

Sustainment is defined as “showing evidence of Sustainment is defined as “showing evidence of performing performing most of the time most of the time at that level…” also known at that level…” also known as as the mastery principlethe mastery principle..




Thresholds for each levelThresholds for each level




RANGE (within the level)RANGE (within the level)

There may be varying degrees of performance within There may be varying degrees of performance within the same range…the same range…

However…They all have two things in common:However…They all have two things in common:




1- They 1- They all meet the criteriaall meet the criteria for that level… for that level…

However, some sustain “better” than others…However, some sustain “better” than others…

2- They 2- They all fail to sustainall fail to sustain at the next level at the next level

However, some fail “better” than the othersHowever, some fail “better” than the others




ACCURACYACCURACYDegree of accuracy in between “all or nothing” is Degree of accuracy in between “all or nothing” is defined through modifiers and qualifiers also known as defined through modifiers and qualifiers also known as the “weasel words”…the “weasel words”…

most of the time / frequently / mostly / usually /can…most of the time / frequently / mostly / usually /can…

indicate sustainment, strength, “above the threshold”indicate sustainment, strength, “above the threshold”

some of the time / rarely / infrequent / may…some of the time / rarely / infrequent / may…

indicate failed sustainment, weakness, deficiencyindicate failed sustainment, weakness, deficiency

There are very few definitions that are “none” or “all”There are very few definitions that are “none” or “all”



Authentic Texts Authentic Texts ORIENTATION MODEORIENTATION MODE(Basic and simple information – L1)(Basic and simple information – L1)

INSTRUCTIVE MODEINSTRUCTIVE MODE(Detailed concrete/factual information-L2)(Detailed concrete/factual information-L2)

EVALUATIVE MODEEVALUATIVE MODE(Analysis and evaluation of events – L3)(Analysis and evaluation of events – L3)

PROJECTIVE MODEPROJECTIVE MODE(Highly abstract and complex texts – L4)(Highly abstract and complex texts – L4)



The FrameworkThe FrameworkILR Proficiency Scale and Authentic TextsILR Proficiency Scale and Authentic Texts



ILR AND TEXT MODESILR AND TEXT MODES

• ILR READING DESCRIPTORS FOCUS ON…ILR READING DESCRIPTORS FOCUS ON…- NON-NATIVE READER’S FUNCTIONALITY- NON-NATIVE READER’S FUNCTIONALITY- THE CONTENT AREAS OF TEXTS- THE CONTENT AREAS OF TEXTS- HOW A NON-NATIVE READS THE TEXT- HOW A NON-NATIVE READS THE TEXT

• TEXT MODES FOCUS ON …TEXT MODES FOCUS ON …- WRITER’S COMMUNICATIVE PURPOSE - WRITER’S COMMUNICATIVE PURPOSE - TEXT RATED FROM IN-LANGUAGE PERSPECTIVE- TEXT RATED FROM IN-LANGUAGE PERSPECTIVE- HOW A NATIVE READS THE TEXT- HOW A NATIVE READS THE TEXT



PLUS LEVELSPLUS LEVELS

Not in the original ILR documentNot in the original ILR document

Supplementary (!) to the base levelsSupplementary (!) to the base levels

Indicate performance that isIndicate performance that is““substantially exceeding the base level and substantially exceeding the base level and

(but?) not consistently meeting (but?) not consistently meeting (sustaining?) the criteria for the next level”(sustaining?) the criteria for the next level”

Includes Includes propertiesproperties of both the base and the of both the base and the next levelnext level

There are NO plus level tasks!There are NO plus level tasks!



CHARACTERISTICS OF PLUS LEVELSCHARACTERISTICS OF PLUS LEVELS

1+ MAINLY L1 WITH L2 ELEMENTS ADDED1+ MAINLY L1 WITH L2 ELEMENTS ADDED(VOCABULARY AND GRAMMAR )(VOCABULARY AND GRAMMAR )

2+ 2+ a) L2 TEXT RAISED BY GRAMMAR/LEXICONa) L2 TEXT RAISED BY GRAMMAR/LEXICONb) L3 TEXT WITH EASY ATTRIBUTES b) L3 TEXT WITH EASY ATTRIBUTES (TOO MUCH FACTUAL INFORMATION)(TOO MUCH FACTUAL INFORMATION)

L3+ L3+ MAINLY L4, BUT L4 PROPERTIES NOT MAINLY L4, BUT L4 PROPERTIES NOT SUSTAINED THROUGHOUT SUSTAINED THROUGHOUT (NOT ENOUGH ABSTRACT FORMULATIONS)(NOT ENOUGH ABSTRACT FORMULATIONS)



ORIENTATION MODEORIENTATION MODEL1L1

Short, discrete, simple, and Short, discrete, simple, and compound sentencescompound sentences

Simple and generic vocabulary Simple and generic vocabulary Simple structure Simple structure Loosely ordered informationLoosely ordered information



CHARACTERISTICS OF CHARACTERISTICS OF LEVEL 1 TEXTSLEVEL 1 TEXTS

ORIENTATIONAL IN NATUREORIENTATIONAL IN NATURE PROVIDE SIMPLE GENERAL INFOPROVIDE SIMPLE GENERAL INFO MOSTLY ADS AND ANNOUNCEMENTSMOSTLY ADS AND ANNOUNCEMENTS AUTHOR ANONYMOUS/IRRELEVANT AUTHOR ANONYMOUS/IRRELEVANT SIMPLE DISCOURSESIMPLE DISCOURSE ONE MAIN IDEAONE MAIN IDEA GENERIC VOCABULARYGENERIC VOCABULARY PRESENT TIME FRAMEPRESENT TIME FRAME



LEVEL 1 GLOBAL TASKSLEVEL 1 GLOBAL TASKS

AT L1, READER CAN UNDERSTAND:AT L1, READER CAN UNDERSTAND:

THE GENERAL MAIN IDEA THE GENERAL MAIN IDEA PROMINENT DETAILS (WHO/WHEN/WHERE)PROMINENT DETAILS (WHO/WHEN/WHERE)



LEVEL 2 TEXTSLEVEL 2 TEXTSINSTRUCTIVE MODEINSTRUCTIVE MODE

INSTRUCTIVE/CONVEYS FACTUAL INFO NO INSTRUCTIVE/CONVEYS FACTUAL INFO NO AUTHOR PRESENTAUTHOR PRESENT

MOSTLY NEWS-MEDIA CONTENTMOSTLY NEWS-MEDIA CONTENT ANONYMOUS AUTHORANONYMOUS AUTHOR COHESIVE DISCOURSECOHESIVE DISCOURSE TOPIC-SPECIFIC VOCABULARYTOPIC-SPECIFIC VOCABULARY MULTIPLE TIMEFRAMESMULTIPLE TIMEFRAMES



LEVEL 2 GLOBAL TASKSLEVEL 2 GLOBAL TASKS

AT L2, READER CAN UNDERSTAND:AT L2, READER CAN UNDERSTAND:

THE MAIN IDEAS, MAIN PLAYERS (WHO/WHAT, THE MAIN IDEAS, MAIN PLAYERS (WHO/WHAT, WHERE, WHEN)WHERE, WHEN)

SEQUENCING OF EVENTS (MULTI-TIME FRAMES)SEQUENCING OF EVENTS (MULTI-TIME FRAMES) INTERACTION AMONG PLAYERS (DIRECTION)INTERACTION AMONG PLAYERS (DIRECTION) CAUSE/EFFECT RELATIONSHIPS (WHY)CAUSE/EFFECT RELATIONSHIPS (WHY) CONDITIONALS & NEGATIONS (IF/HOW)CONDITIONALS & NEGATIONS (IF/HOW) STATED FACTUAL OUTCOMES & RESULTSSTATED FACTUAL OUTCOMES & RESULTS



CHARACTERISTICS OF LEVEL 3 TEXTSCHARACTERISTICS OF LEVEL 3 TEXTSEVALUATIVE MODEEVALUATIVE MODE

EVALUATIVE MODEEVALUATIVE MODE OPINION OR EDITORIAL PIECESOPINION OR EDITORIAL PIECES SOCIAL/CULTURAL/POLITICAL ISSUESSOCIAL/CULTURAL/POLITICAL ISSUES AUTHOR IS PRESENT BY PERSONAL VIEWSAUTHOR IS PRESENT BY PERSONAL VIEWS ABSTRACT AND CONCRETE LEXICONABSTRACT AND CONCRETE LEXICON ABSTRACT LINGUISTIC FORMULATIONS IN ABSTRACT LINGUISTIC FORMULATIONS IN

COMPLEX SENTENCES COMPLEX SENTENCES EXTENDED DISCOURSEEXTENDED DISCOURSE DOMAIN SPECIFIC VOCABULARYDOMAIN SPECIFIC VOCABULARY SCIENTIFIC OR TECHNICAL DOCUMENTSSCIENTIFIC OR TECHNICAL DOCUMENTS



MAIN LEVEL 3 GLOBAL TASKSMAIN LEVEL 3 GLOBAL TASKS

AT L3, READER CAN UNDERSTAND:AT L3, READER CAN UNDERSTAND:ALL L2 ELEMENTS, PLUSALL L2 ELEMENTS, PLUS HIERARCHY OF ABSTRACT IDEASHIERARCHY OF ABSTRACT IDEAS SUPPORTING FACTS AND IDEASSUPPORTING FACTS AND IDEAS ABSTRACT HYPOTHETICAL CONDITIONSABSTRACT HYPOTHETICAL CONDITIONS(ASSUMPTIONS, SUPPOSITIONS)(ASSUMPTIONS, SUPPOSITIONS) PROS AND CONS OF AN ISSUEPROS AND CONS OF AN ISSUE INFERENCES & CONNOTATIONSINFERENCES & CONNOTATIONS COMPARISON AND CONTRAST OF IDEASCOMPARISON AND CONTRAST OF IDEAS CONCLUSIONS, SUGGESTIONS, SOLUTIONSCONCLUSIONS, SUGGESTIONS, SOLUTIONS TECHNICAL & DOMAIN SPECIFIC LEXICONTECHNICAL & DOMAIN SPECIFIC LEXICON



PROJECTIVE MODEPROJECTIVE MODE

Unpredictable turns of thoughtUnpredictable turns of thought Mixture of registerMixture of register Subtlety and nuanceSubtlety and nuance Highly organized, abstract, Highly organized, abstract,

sophisticatedsophisticated Tone, inferences; irony & humorTone, inferences; irony & humor

Dirgin, James. December 2012. Dirgin, James. December 2012. Please seek the author’s permission for Please seek the author’s permission for


Components of an ItemComponents of an Item

(1) (1) Orientation (Providing a brief context)Orientation (Providing a brief context)

(2) (2) Text (Passage)Text (Passage)

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

(3) (3) Stem (Question)Stem (Question)MULTIPLE CHOICEMULTIPLE CHOICE (4) (4) KEY (Correct answer) KEY (Correct answer) (5) (5) Distracter 1Distracter 1(6) (6) Distracter 2Distracter 2(7) (7) Distracter 3Distracter 3

Dirgin, James. December 2012. Dirgin, James. December 2012. Please seek the author’s permission for Please seek the author’s permission for


Components of an ItemComponents of an Item

Any single ONE of these 7 factors Any single ONE of these 7 factors may and will may and will

change the level of difficultychange the level of difficultyOf an item!Of an item!

dirgin, james. january 2014. please seek the author’s permission for distribution or commercial...

Documents