1

Educating the StakeholdersEducating the Stakeholders

ECOLT 2008

November 7, 2008

ILR Testing Committee

ECOLT 2008

November 7, 2008

ILR Testing Committee

2

Overview of PresentationOverview of Presentation

Intuitive Language Testing

Testing Misconceptions

Educating Stakeholders

Continuing Challenges

Intuitive Language Testing

Testing Misconceptions

Educating Stakeholders

Continuing Challenges

3

Why educate stakeholders?Why educate stakeholders?

Consequences of language tests reach far

beyond the testing sessions.

StakeholdersExaminees

Language teachers

Supervisors of linguists

USG agencies

Policy makers

Citizens

Consequences of language tests reach far

beyond the testing sessions.

StakeholdersExaminees

Language teachers

Supervisors of linguists

USG agencies

Policy makers

Citizens

4

Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01

What is limited

proficiency? ILR Level 2

Proficiency in what? Speaking

What type of Arabic? Modern Standard

Arabic

What is limited

proficiency? ILR Level 2

Proficiency in what? Speaking

What type of Arabic? Modern Standard

Arabic

“Five years after Arab

terrorists attacked the

United States, only 33

FBI agents have even

a limited proficiency in

Arabic …”

“Five years after Arab

terrorists attacked the

United States, only 33

FBI agents have even

a limited proficiency in

Arabic …”

5

Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01

Were all the agents tested? Most agents do not

have a language requirement.

What is meant by “any familiarity?” Speaking Modern

Standard Arabic

What does a score of zero mean?

Were all the agents tested? Most agents do not

have a language requirement.

What is meant by “any familiarity?” Speaking Modern

Standard Arabic

What does a score of zero mean?

“Counting agents who

know only a handful of

Arabic words -- including

those who scored zero on

the standard proficiency

test -- just 1 percent of the

FBI’s 12,000 agents have

any familiarity with the

language, the statistics

show.”

“Counting agents who

know only a handful of

Arabic words -- including

those who scored zero on

the standard proficiency

test -- just 1 percent of the

FBI’s 12,000 agents have

any familiarity with the

language, the statistics

show.”

6

Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01

What is ‘advanced professional proficiency?’ ILR Level 4

The tests were administered by the ILR? The ILR does not

administer tests.

What is ‘advanced professional proficiency?’ ILR Level 4

The tests were administered by the ILR? The ILR does not

administer tests.

“Youssef … is one of only

six FBI agents who scored

a 4 for ‘advanced

professional proficiency’ in

Arabic on standardized

speaking tests

administered by the

Interagency Language

Roundtable for federal

agencies.”

“Youssef … is one of only

six FBI agents who scored

a 4 for ‘advanced

professional proficiency’ in

Arabic on standardized

speaking tests

administered by the

Interagency Language

Roundtable for federal

agencies.”

7

Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01

Who is interpreting the language testing statistics? An attorney for a former

linguist, suing the FBI.

What language test was given? Speaking, not Translation

Who do agents rely on? Language Analysts and

Intelligence Analysts

Tested in Reading, Listening, Translation, Speaking

Who is interpreting the language testing statistics? An attorney for a former

linguist, suing the FBI.

What language test was given? Speaking, not Translation

Who do agents rely on? Language Analysts and

Intelligence Analysts

Tested in Reading, Listening, Translation, Speaking

“Youssef’s attorney, Stephen M. Kohn, said the statistics indicate that most FBI agents have no way to gauge the accuracy of translated materials and must rely on linguists or other third parties for their information.”

“Youssef’s attorney, Stephen M. Kohn, said the statistics indicate that most FBI agents have no way to gauge the accuracy of translated materials and must rely on linguists or other third parties for their information.”

8

Intuitive Test Theory Braun and Mislevy Intuitive Test Theory Braun and Mislevy

General testing P-primsA test measures what it says at the top

of the page.

A test is a test is a test.

A score is a score is a score.

Any two tests that measure the same thing can be made interchangeable with a little “equating” magic.

General testing P-primsA test measures what it says at the top

of the page.

A test is a test is a test.

A score is a score is a score.

Any two tests that measure the same thing can be made interchangeable with a little “equating” magic.

9

P-Prims, continuedP-Prims, continued

You score a test by adding up scores for items.

70% is passing – on any and every assessment.

You can tell if an item is good by looking at it.

Multiple-choice questions only measure recall.

Multiple-choice tests equal standardized tests equal high-stakes tests.

You score a test by adding up scores for items.

70% is passing – on any and every assessment.

You can tell if an item is good by looking at it.

Multiple-choice questions only measure recall.

Multiple-choice tests equal standardized tests equal high-stakes tests.

10

ILTA Code of EthicsILTA Code of Ethics

Principle 7

Language testers in their societal roles shall strive to improve the quality of language testing, assessment and teaching services, promote the just allocation of those services and contribute to the education of society regarding language learning and language proficiency.

Principle 7

Language testers in their societal roles shall strive to improve the quality of language testing, assessment and teaching services, promote the just allocation of those services and contribute to the education of society regarding language learning and language proficiency.

11

Principle 8

Language testers shall be mindful of their

obligations to the society within which they

work, while recognising that those

obligations may on occasion conflict with

their responsibilities to their test takers and

to other stakeholders.

Principle 8

Language testers shall be mindful of their

obligations to the society within which they

work, while recognising that those

obligations may on occasion conflict with

their responsibilities to their test takers and

to other stakeholders.

ILTA Code of EthicsILTA Code of Ethics

12

Overview of PresentationOverview of Presentation

Intuitive Language Testing

Testing Misconceptions

Educating Stakeholders

Continuing Challenges

Intuitive Language Testing

Testing Misconceptions

Educating Stakeholders

Continuing Challenges

13

Quote 1Quote 1

Hassan (Foreign Service Specialist): “I worked for an NGO in Mosul and helped several

US-based companies set up their computer systems. I constantly talked to vendors, made arrangements to access the appropriate routers, negotiated bandwidth for their Internet hubs, the works! In my spare time, I talked to my Iraqi buddies about all sorts of things. They were great at tipping me off whenever good stuff came at the local grocery store. I thought I was a 5 in speaking Iraqi Arabic. You scored me a 2. Need I tell you what I think about your test?”

Hassan (Foreign Service Specialist): “I worked for an NGO in Mosul and helped several

US-based companies set up their computer systems. I constantly talked to vendors, made arrangements to access the appropriate routers, negotiated bandwidth for their Internet hubs, the works! In my spare time, I talked to my Iraqi buddies about all sorts of things. They were great at tipping me off whenever good stuff came at the local grocery store. I thought I was a 5 in speaking Iraqi Arabic. You scored me a 2. Need I tell you what I think about your test?”

14

Testing Misconception Testing Misconception

A 3 is a 3 is a 3.

Proficiency testing does not equal achievement or performance testing.

Scores in one skill do not transfer across other skills.

A 3 given by a non-USG testing organization may not equal a 3 given by a USG agency.

A 3 is a 3 is a 3.

Proficiency testing does not equal achievement or performance testing.

Scores in one skill do not transfer across other skills.

A 3 given by a non-USG testing organization may not equal a 3 given by a USG agency.

15

Educating Stakeholders: Types of Tests

Educating Stakeholders: Types of Tests

Achievement = Memorized responses using the content of a specific textbook or curriculum.

Performance = Rehearsed ability to communicate in specific, familiar settings.

Proficiency = Unrehearsed general ability to accomplish communication tasks across a wide range of topics and settings.

The following self assessment used by permission, Dr. Ray Clifford, ACTFL 2007

Achievement = Memorized responses using the content of a specific textbook or curriculum.

Performance = Rehearsed ability to communicate in specific, familiar settings.

Proficiency = Unrehearsed general ability to accomplish communication tasks across a wide range of topics and settings.

The following self assessment used by permission, Dr. Ray Clifford, ACTFL 2007

16

A Self Assessment QuizA Self Assessment Quiz

You will be asked about 6 different testing

purposes.

For each of those test purposes, which type of

test would you choose?

a. Achievement

b. Performance

c. Proficiency

You will be asked about 6 different testing

purposes.

For each of those test purposes, which type of

test would you choose?

a. Achievement

b. Performance

c. Proficiency

17

Which type of test would you choose:Achievement, Performance, or Proficiency?

Which type of test would you choose:Achievement, Performance, or Proficiency?

1. To assess students’ language learning

after Chapter 3 of a beginning

language course?

1. To assess students’ language learning

after Chapter 3 of a beginning

language course?

18

Which type of test would you choose:Achievement, Performance, or Proficiency?

Which type of test would you choose:Achievement, Performance, or Proficiency?

1. To assess students’ language learning after Chapter 3 of a beginning language course?

2. To place students into a university’s sequence of courses?

1. To assess students’ language learning after Chapter 3 of a beginning language course?

2. To place students into a university’s sequence of courses?

19

Which type of test would you choose:Achievement, Performance, or Proficiency?

Which type of test would you choose:Achievement, Performance, or Proficiency?

1. To assess students’ language learning after Chapter 3 of a beginning

language course?

2. To place students into a university’s sequence of courses?

3. To test students completing a year-

long, intensive language course?

1. To assess students’ language learning after Chapter 3 of a beginning

language course?

2. To place students into a university’s sequence of courses?

3. To test students completing a year-

long, intensive language course?

20

Which type of test would you choose:Achievement, Performance, or Proficiency?

Which type of test would you choose:Achievement, Performance, or Proficiency?

1. To assess students’ language learning after Chapter 3 of a beginning

language course?

2. To place students into a university’s sequence of courses?

3. To test students completing a year-long, intensive language course?

4. To screen job applicants for a

specific job with well-defined,

repetitive tasks?

1. To assess students’ language learning after Chapter 3 of a beginning

language course?

2. To place students into a university’s sequence of courses?

3. To test students completing a year-long, intensive language course?

4. To screen job applicants for a

specific job with well-defined,

repetitive tasks?

21

Which test type would you choose:Achievement, Performance, or Proficiency?

Which test type would you choose:Achievement, Performance, or Proficiency?

1. To assess students’ language learning after Chapter 3 of a beginning

language course?

2. To place students into a university’s sequence of courses?

3. To test students completing a year-long, intensive language course?

4. To screen job applicants for a specific job with well-defined,

repetitive tasks?

5. To select someone to be the NATO

spokesperson on a news show with a

“hostile” moderator?

1. To assess students’ language learning after Chapter 3 of a beginning

language course?

2. To place students into a university’s sequence of courses?

3. To test students completing a year-long, intensive language course?

4. To screen job applicants for a specific job with well-defined,

repetitive tasks?

5. To select someone to be the NATO

spokesperson on a news show with a

“hostile” moderator?

22

Which test type would you choose:Achievement, Performance, or Proficiency?

Which test type would you choose:Achievement, Performance, or Proficiency?

1. To assess students’ language learning after Chapter 3 of a beginning

language course?

2. To place students into a university’s sequence of courses?

3. To test students completing a year-long, intensive language course?

4. To screen job applicants for a specific job with well-defined,

repetitive tasks?

5. To select someone to be the NATO spokesperson on a news show

with a “hostile” moderator?

6. To document a language ability rating

for a person’s military or civilian

personnel record?

1. To assess students’ language learning after Chapter 3 of a beginning

language course?

2. To place students into a university’s sequence of courses?

3. To test students completing a year-long, intensive language course?

4. To screen job applicants for a specific job with well-defined,

repetitive tasks?

5. To select someone to be the NATO spokesperson on a news show

with a “hostile” moderator?

6. To document a language ability rating

for a person’s military or civilian

personnel record?

23

Achievement, Performance, or Proficiency

Which test type did you choose?

Achievement, Performance, or Proficiency

Which test type did you choose?

1. To assess students’ language learning after Chapter 3 of a beginning language course?

2. To place students into a university’s sequence of courses?3. To test students completing a year-long, intensive language

course?4. To screen job applicants for a specific job with well-defined,

repetitive tasks?5. To select someone to be the NATO spokesperson on a

“hostile” news show?6. To document a language ability rating for a person’s military

or civilian personnel record?

1. To assess students’ language learning after Chapter 3 of a beginning language course?

2. To place students into a university’s sequence of courses?3. To test students completing a year-long, intensive language

course?4. To screen job applicants for a specific job with well-defined,

repetitive tasks?5. To select someone to be the NATO spokesperson on a

“hostile” news show?6. To document a language ability rating for a person’s military

or civilian personnel record?

24

Educating Stakeholders:FSI Examinees

Educating Stakeholders:FSI Examinees

Information on the Skill Level Descriptions

available online at www.govtilr.org

Self appraisals are suggested/required for

some language positions

Formal presentations on language testing

given to new entrants

Information on the Skill Level Descriptions

available online at www.govtilr.org

Self appraisals are suggested/required for

some language positions

Formal presentations on language testing

given to new entrants

25

FSI Stakeholder EducationFSI Stakeholder Education

Before Speaking Self-Assessment:

85% of applicants with self-assessed

proficiency tested at 42% passing rate.

After Speaking Self-Assessment:

61% of applicants with self-assessed

proficiency tested at 58% passing rate.

Before Speaking Self-Assessment:

85% of applicants with self-assessed

proficiency tested at 42% passing rate.

After Speaking Self-Assessment:

61% of applicants with self-assessed

proficiency tested at 58% passing rate.

26

Quote 2Quote 2

Nasra (Swahili language applicant): “Oh yeah, that test won’t be a problem at all. It’s my

mother tongue, so of course I’m a Level 5. I speak Swahili all the time. It’s not a big deal. You can test me right now, if you want. I’m ready.”

Nasra (Swahili language applicant): “Oh yeah, that test won’t be a problem at all. It’s my

mother tongue, so of course I’m a Level 5. I speak Swahili all the time. It’s not a big deal. You can test me right now, if you want. I’m ready.”

27

Testing MisconceptionTesting Misconception

All native speakers are Level 5.

Native speaker as a social variable.

Native speaker as the ideal.

All native speakers are Level 5.

Native speaker as a social variable.

Native speaker as the ideal.

28

Typical ScalesTypical Scales

Poor ExcellentGoodFair Satisfactory

1000 50

AABBCCDDFF

29

FBI OPI DataFBI OPI Data

Testing Native Speakers80% + of FBI Linguists are native or heritage

speakers of the language.

Native Speaker Self Appraisal ScoresNative Language

• Level 5: 70%

• Level 4/4+: 10%

• Level 3/3+: 20% (largely heritage speakers)

English• Level 5: 51%

Testing Native Speakers80% + of FBI Linguists are native or heritage

speakers of the language.

Native Speaker Self Appraisal ScoresNative Language

• Level 5: 70%

• Level 4/4+: 10%

• Level 3/3+: 20% (largely heritage speakers)

English• Level 5: 51%

30

Quote 3Quote 3

James (Manager of a new national security initiative):

“We need at least 10 linguists dedicated to this assignment in Arabic alone. Since this matter is of highest importance, we need them all to be Level 5. It’s important that we have the best people on the assignment.”

James (Manager of a new national security initiative):

“We need at least 10 linguists dedicated to this assignment in Arabic alone. Since this matter is of highest importance, we need them all to be Level 5. It’s important that we have the best people on the assignment.”

31

Who would you hire…As a Foreign Service Officer?

Who would you hire…As a Foreign Service Officer?

Aleem:

heritage speaker of Saudi Arabic

went to Arabic classes at a private language school

Language school-Listening: 4

Language school-Reading: 4

Language school-Speaking-MSA: 3+

Aleem:

heritage speaker of Saudi Arabic

went to Arabic classes at a private language school

Language school-Listening: 4

Language school-Reading: 4

Language school-Speaking-MSA: 3+

Richard:

native speaker of English

learned Arabic for 4 years

while in college, (1 year in

Qatar)

DLPT-Listening: 3

DLPT-Reading: 3

OPI-Speaking-MSA: 2

Richard:

native speaker of English

learned Arabic for 4 years

while in college, (1 year in

Qatar)

DLPT-Listening: 3

DLPT-Reading: 3

OPI-Speaking-MSA: 2

32

Who would you hire…As a Military Interpreter?Who would you hire…

As a Military Interpreter?

Adrienne:

worked at an Iraqi refugee

program for 1 year

DLPT-Listening: 2+

DLPT-Reading: 2

OPI-Speaking-MSA: 2+

Consecutive

Interpretation

Test-Iraqi/English: 2

Adrienne:

worked at an Iraqi refugee

program for 1 year

DLPT-Listening: 2+

DLPT-Reading: 2

OPI-Speaking-MSA: 2+

Consecutive

Interpretation

Test-Iraqi/English: 2

Catherine:

worked as a translator for

the United Nations for 2

years

DLPT-Listening: 2+

DLPT-Reading: 4

OPI-Speaking-MSA: 3

Translation Test: 3

Catherine:

worked as a translator for

the United Nations for 2

years

DLPT-Listening: 2+

DLPT-Reading: 4

OPI-Speaking-MSA: 3

Translation Test: 3

33

Who would you hire…As a Foreign Language Monitor?

Who would you hire…As a Foreign Language Monitor?

Jennifer:

native speaker of English

DLPT-Listening: 2

DLPT-Reading: 3

OPI-Speaking-MSA: 1+

Translation Test: 2

English Comp: 3+

Jennifer:

native speaker of English

DLPT-Listening: 2

DLPT-Reading: 3

OPI-Speaking-MSA: 1+

Translation Test: 2

English Comp: 3+

Mirna:

native speaker of

Arabic

DLPT-Listening: 2+

DLPT-Reading: 2

OPI-Speaking-

MSA: 2+

Translation Test: 2

English Comp: 2

Mirna:

native speaker of

Arabic

DLPT-Listening: 2+

DLPT-Reading: 2

OPI-Speaking-

MSA: 2+

Translation Test: 2

English Comp: 2

34

Who would you hire… As a MSA Oral Proficiency Interview Tester?

Who would you hire… As a MSA Oral Proficiency Interview Tester?

Mike:

heritage speaker of

Jordanian Arabic

OPI-Speaking-MSA: 3

OPI-Speaking-Jordanian:

3

OPI-Speaking-English: 3+

Mike:

heritage speaker of

Jordanian Arabic

OPI-Speaking-MSA: 3

OPI-Speaking-Jordanian:

3

OPI-Speaking-English: 3+

Imad:

native speaker of Jordanian

Arabic

OPI-Speaking-MSA: 5

OPI-Speaking-Jordanian: 4

OPI-Speaking-English: 1

Imad:

native speaker of Jordanian

Arabic

OPI-Speaking-MSA: 5

OPI-Speaking-Jordanian: 4

OPI-Speaking-English: 1

35

Who would you hire… As a Document Translator?

Who would you hire… As a Document Translator?

Julie:

native speaker of English

DLPT-Listening: 2

DLPT-Reading: 3

OPI-Speaking-MSA: 1+

Translation Test: 2+

Julie:

native speaker of English

DLPT-Listening: 2

DLPT-Reading: 3

OPI-Speaking-MSA: 1+

Translation Test: 2+

Hiam:

native speaker of Moroccan

Arabic

DLPT-Listening: 3

DLPT-Reading: 3

OPI-Speaking-MSA: 4

English Composition: 3

Hiam:

native speaker of Moroccan

Arabic

DLPT-Listening: 3

DLPT-Reading: 3

OPI-Speaking-MSA: 4

English Composition: 3

36

Testing MisconceptionTesting Misconception

Higher scores make better linguists.

Some job requirements do not necessitate high scores, or need high scores in only some skills.

Tests are usually not performance.

Other non-language skills may be required.

Higher scores make better linguists.

Some job requirements do not necessitate high scores, or need high scores in only some skills.

Tests are usually not performance.

Other non-language skills may be required.

37

Audio SampleAudio Sample

Native Speaker of English

College Graduate

Was in Peace Corps for 14 months in

Uzbekistan

Native Speaker of English

College Graduate

Was in Peace Corps for 14 months in

Uzbekistan

38

Educating ManagementEducating Management

Stressing proficiency and performance

Understanding job requirements and working

with HR to establish criteria

Stressing proficiency and performance

Understanding job requirements and working

with HR to establish criteria

39

Quote 4Quote 4

Debra, a newly hired Foreign Service Officer

signed up for a test in Swedish, hoping to

be assigned to Stockholm. The testing

team reports that Debra understands

Swedish quite well, but she speaks

Norwegian.

Debra, a newly hired Foreign Service Officer

signed up for a test in Swedish, hoping to

be assigned to Stockholm. The testing

team reports that Debra understands

Swedish quite well, but she speaks

Norwegian.

40

Testing MisconceptionTesting Misconception

I know what languages I can understand, speak, and write.

Mutually intelligible does not equal bilingual.

Languages are sometimes misidentified by examinees.

One skill ability does not equal all skills ability.

I know what languages I can understand, speak, and write.

Mutually intelligible does not equal bilingual.

Languages are sometimes misidentified by examinees.

One skill ability does not equal all skills ability.

41

Educating StakeholdersEducating Stakeholders

Ensure recruiters understand language

requirements

Define language for testing purposes

Resources such as the NVTC site,

Languages of the World web site

Ensure recruiters understand language

requirements

Define language for testing purposes

Resources such as the NVTC site,

Languages of the World web site

42

Quote 5Quote 5

Hanke (Polish OPI Tester Applicant, has an ILR Speaking 5 on record, 10 years old):

“Why would I want to lower my score on record? Who and why decided that a Polish linguist working for the Bureau cannot score higher than a 4? And, if this applies only to Polish, why are the requirements lower for the Polish language?”

Hanke (Polish OPI Tester Applicant, has an ILR Speaking 5 on record, 10 years old):

“Why would I want to lower my score on record? Who and why decided that a Polish linguist working for the Bureau cannot score higher than a 4? And, if this applies only to Polish, why are the requirements lower for the Polish language?”

43

Testing MisconceptionTesting Misconception

Test batteries and qualifying scores should

be the same across languages for the

same position.

All tests from a particular battery may not be

available in a language.

Some tests may not have the same maximum

possible score across languages.

Test batteries and qualifying scores should

be the same across languages for the

same position.

All tests from a particular battery may not be

available in a language.

Some tests may not have the same maximum

possible score across languages.

44

Educating StakeholdersEducating Stakeholders

Document testing standards

Back up decisions with research, where

available

Ensure policies are documented and

available to language professionals

Document testing standards

Back up decisions with research, where

available

Ensure policies are documented and

available to language professionals

45

Quote 6Quote 6

Silke (German linguist who has to retest because of expired scores):

“I don’t see why I have to retest. I’m a 5. Even assuming that my command of the German language somewhat diminished over the years (which it has not), it could not have gone lower than a 4.”

Silke (German linguist who has to retest because of expired scores):

“I don’t see why I have to retest. I’m a 5. Even assuming that my command of the German language somewhat diminished over the years (which it has not), it could not have gone lower than a 4.”

46

Testing MisconceptionTesting Misconception

Once a 4, always a 4.

Test scores are considered valid for varying

timeframes, depending on the agency and the

level scored

Regular retesting with new forms should be

encouraged.

Score may reflect the highest score given from

that exam, not the ‘true score’ of the examinee.

Once a 4, always a 4.

Test scores are considered valid for varying

timeframes, depending on the agency and the

level scored

Regular retesting with new forms should be

encouraged.

Score may reflect the highest score given from

that exam, not the ‘true score’ of the examinee.

47

Educating StakeholdersEducating Stakeholders

Research project underway at the Center

for the Advanced Study of Languages to

investigate testing periodicity.

Research project underway at the Center

for the Advanced Study of Languages to

investigate testing periodicity.

48

Quote 7Quote 7

Olga (Russian teacher): “I’ve had a lot of experience in test development.

I’ve been giving tests in my classroom for 20 years, and I give oral exams at the end of the class, so I could be an OPI tester. I don’t think that I even need to attend the training.”

Olga (Russian teacher): “I’ve had a lot of experience in test development.

I’ve been giving tests in my classroom for 20 years, and I give oral exams at the end of the class, so I could be an OPI tester. I don’t think that I even need to attend the training.”

49

Testing MisconceptionTesting Misconception

The best teachers make the best testers.

Teachers sometimes have difficulty swapping

hats from someone who fosters language learning

to someone who is taking an impartial

measurement.

True whether or not the examinee is the teacher’s

student.

The best teachers make the best testers.

Teachers sometimes have difficulty swapping

hats from someone who fosters language learning

to someone who is taking an impartial

measurement.

True whether or not the examinee is the teacher’s

student.

50

Educating Stakeholders:Tester Training

Educating Stakeholders:Tester Training

Extensive testing training:Up to 1 year (Tester or Examiner training)Discuss FSI’s Testing ManualRate taped testsObserve and discuss live testsConduct (unofficial) progress testsComplete online program & mastery

exerciseConduct certification tests

Extensive testing training:Up to 1 year (Tester or Examiner training)Discuss FSI’s Testing ManualRate taped testsObserve and discuss live testsConduct (unofficial) progress testsComplete online program & mastery

exerciseConduct certification tests

51

Tester AccountabilityTester Accountability

Display sufficient understanding of testing

principles!

Display sufficient understanding of testing

principles!

52

Quote 8Quote 8

Jennifer (Foreign Service applicant):

“I took 2 semesters of Arabic, was at the

top of my class, never missed an

assignment, always contributed in class,

got straight As and you guys tested me

S0+.

Your test sucks!”

Jennifer (Foreign Service applicant):

“I took 2 semesters of Arabic, was at the

top of my class, never missed an

assignment, always contributed in class,

got straight As and you guys tested me

S0+.

Your test sucks!”

53

Testing MisconceptionTesting Misconception

Consistent practice with material in the

foreign language will result in a good score

and regular improvement on ILR-based

tests.

Improvement depends on consistent practice with

increasingly complex language material.

ILR Level ranges are not equidistant.

Consistent practice with material in the

foreign language will result in a good score

and regular improvement on ILR-based

tests.

Improvement depends on consistent practice with

increasingly complex language material.

ILR Level ranges are not equidistant.

54

5

4

0

1

2

3

Level 5

Level 4

Level 3

Level 2

Level 1

Level 0

Level 3 Range 3

ILR Skill Level Descriptions

55

Quote 9Quote 9

Anna

“I just applied to work at XYZ and they

told me I had to pass their language

test. How do I know if I’m at the right

level to pass their test?”

Anna

“I just applied to work at XYZ and they

told me I had to pass their language

test. How do I know if I’m at the right

level to pass their test?”

56

Testing MisconceptionTesting Misconception

USG language tests will tell me all that I

need to know about the examinee.

Most ILR-based tests measure proficiency, not

achievement.

Most USG tests are not diagnostic in nature.

USG language tests will tell me all that I

need to know about the examinee.

Most ILR-based tests measure proficiency, not

achievement.

Most USG tests are not diagnostic in nature.

57

Educating Test TakersEducating Test Takers

DoD Familiarization Guides

Online language training (LangNet, LingNet)

New Self Assessments

www.govtilr.org

DoD Familiarization Guides

Online language training (LangNet, LingNet)

New Self Assessments

www.govtilr.org

58

Educating Stakeholders:What does Level X mean?Educating Stakeholders:

What does Level X mean?

Through tester training, illustrate Levels

Provide professional development training

throughout

Through tester training, illustrate Levels

Provide professional development training

throughout

59

The Next Big Thing is UsLev Grossman (Time March 2006)

The Next Big Thing is UsLev Grossman (Time March 2006)

It goes against everybody's inner cynic to read (or for that matter to write) a sentence like the following: We are on the verge of the greatest age of creativity and innovation the world has ever known. It smacks of treacly dotcomism. It smacks of I Love the '90s. My inner cynic is a tiny bit queasy right now. But lately it's a conclusion I've had a hard time avoiding. Consider the following idea. Things, broadly speaking, used to be invented by a small, shadowy élite. This mysterious group might be called the People Who Happened to Be in the Room at the Time. These people might have been engineers, or sitcom writers, or chefs. They were probably very nice and might have even been very, very smart. But however smart they were, they're almost certainly no match for a less élite but much, much larger group: All the People Outside the Room.

It goes against everybody's inner cynic to read (or for that matter to write) a sentence like the following: We are on the verge of the greatest age of creativity and innovation the world has ever known. It smacks of treacly dotcomism. It smacks of I Love the '90s. My inner cynic is a tiny bit queasy right now. But lately it's a conclusion I've had a hard time avoiding. Consider the following idea. Things, broadly speaking, used to be invented by a small, shadowy élite. This mysterious group might be called the People Who Happened to Be in the Room at the Time. These people might have been engineers, or sitcom writers, or chefs. They were probably very nice and might have even been very, very smart. But however smart they were, they're almost certainly no match for a less élite but much, much larger group: All the People Outside the Room.

60

Educating Stakeholders:The Public

Educating Stakeholders:The Public

ILR Website www.govtilr.org

National Security Language Initiative“They are also evaluating initial results and

outcomes to gain a more concrete understanding of program efficacy and best practices in foreign language instruction. Building on their program-specific work, the NSLI agencies are exploring other ways to advance and measure foreign language learning, such as enhanced assessments of language proficiency, and additional resources and materials for teachers” (Paper from August 2008)

ILR Website www.govtilr.org

National Security Language Initiative“They are also evaluating initial results and

outcomes to gain a more concrete understanding of program efficacy and best practices in foreign language instruction. Building on their program-specific work, the NSLI agencies are exploring other ways to advance and measure foreign language learning, such as enhanced assessments of language proficiency, and additional resources and materials for teachers” (Paper from August 2008)

61

Educating Stakeholders:Additional Online Resources

Educating Stakeholders:Additional Online Resources

Department of State Web site section on

testing:

Overview

FSI Test

Notes to Examinee

DVC/Field tests

Skill Level Descriptions

FAQs

MLAT

Department of State Web site section on

testing:

Overview

FSI Test

Notes to Examinee

DVC/Field tests

Skill Level Descriptions

FAQs

MLAT

62

Overview of PresentationOverview of Presentation

Intuitive Language Testing

Testing Misconceptions

Educating Stakeholders

Continuing Challenges

Intuitive Language Testing

Testing Misconceptions

Educating Stakeholders

Continuing Challenges

63

Continuing ChallengesContinuing Challenges

Finding time an resources to ‘spread the

word’

Continuing Interagency collaboration

Testing the LCTLs

Testing Listening

Testing Proficiency AND Performance

Testing across the ILR Scale

Finding time an resources to ‘spread the

word’

Continuing Interagency collaboration

Testing the LCTLs

Testing Listening

Testing Proficiency AND Performance

Testing across the ILR Scale