The training needsof different stakeholders

in language testingand assessment

Angela Hasselgreen, Ari Huhta, Jayanti Banerjee

Outline

• The design and implementation of the questionnaire

• Some overall themes• Comparing results by regions• Implications

The timetable

• First meeting:• Preliminary design:• Web-based

questionnaire (roll-out date):

• Data collection:• Data preparation (for

analysis):• Analysis + preparation

for conference:

24 – 25 January 2004

1 February 2004

1 March 2004

1 March – 15 April 2004

18 – 21 April 2004

23 April - 12 May 2004

Part 2: Questions for teacher trainers/educatorsPart 3: Questions for people who work, full time or part time, for language testing/examination boards or organisations that design school-external, regional or nationwide tests/examinations

Summary of responses

• 914 respondents to at least one part of the questionnaire

– Part 1: 741

– Part 2: 709

– Part 3: 689

• 501 respondents have provided contact details

Summary of responses

• Point to Note– Respondents did not necessarily answer every

item in each section for which they submitted data.

• Responses for parts 2 & 3 are particularly low– Part 1: 522 – 614 responses per item– Part 2: 181 – 212 responses per item– Part 3: 198 – 228 responses per item

Where do the respondents work?

Non-European countri

Other European count

United Kingdom

Turkey

Sw eden

Spain

Slovenia

Romania

Poland

Norw ay

Netherlands

Ireland

Hungary

Greece

Finland

Estonia

Bulgaria

Belgium

Missing

’Teachers only’: degree of formal training in assessment

Formal education on assessment

4321

Formal education on assessment

Fre

qu

en

cy

160

140

120

100

80

60

40

20

0

Std. Dev = 1,09

Mean = 3

N = 360,00

1= none2= short3= medium4= long

 

Correlation between assessment-related activities teachers are engaged in

and absence of professional training

1 Classroom-focussed assessment

05

1015202530354045

Prepa

ring

your

ow

...

Using

read

y m

ade.

..

Inte

rpre

ting

test

re...

Givi

ng fe

edba

ck

Using

self/

pee

r a...

Using

info

rmal

, co.

..

Using

portf

olio

Teachers’ perceived needs: do you need training in this area?

1 Classroom-focussed assessment

0102030405060

prep

arin

g yo

ur o

wn...

usin

g re

ady-

mad

e ...

inte

rpre

ting

test

re...

givi

ng fe

edba

ck

usin

g se

lf/ p

eer a

...

usin

g in

form

al, c

on...

usin

g po

rtfol

io

no

yes

Correlation between assessment-related activities teachers are engaged in

and absence of professional training

2 Purposes of assessment

0

5

10

15

20

25

30

35

To givegrades

To find outwhat needsto be taught

To placestudents

To awardfinal

certificates

Teachers’ perceived needs: do you need training in this area?

2 Purposes of assessment

05

1015202530354045

to givegrades

to find outwhat needs

to betaught

to placestudents

to awardfinal

certificates

no

yes

Correlation between assessment-related activities teachers are engaged in

and absence of professional training

3 Content and concepts

05

1015202530

T. receptive skills

T. productive skills

T. microlinguistic a...

T. integrated skills

T. culture

Est. reliability

Est. validity

Using statistics

Teachers’ perceived needs: do you need training in this area?

3 Content and concepts

0 10 20 30 40 50 60

testing receptive skills

testing productive skills

testing grammar/ vocabulary

testing integrated language skills

testing aspects of culture

establishing reliability

establishing validity

using statistics

yes

no

Correlation between assessment-related activities teachers are engaged in

and absence of professional training

4 External tests and exams

05

101520253035404550

Rating oral/ written perf.

Using statistics

Writing item

s

Reviewing items

Acting as an interviewer

Defining assessment ...

Teachers’ perceived needs : do you need training in this area?

4 External tests and exams

0

10

20

30

40

50

60

External:taking part in

rating

usingstatistics

w riting items review ingitems

acting as aninterview er

definingassessment

criteria

no

yes

Tentative conclusions on teachers’ needs

• Teachers generally perceive a need for training across the board

• Areas of particular need:– ELP/other portfolios, preparing own tests, peer/self

assessment, interpreting results and giving feedback– To award grades, to find out what needs to be taught,

placing students, awarding certificates– Aspects of culture, integrated skills, establishing reliability

and validity, statistics– Reviewing and writing items, statistics, defining criteria,

rating, interviewing

Teacher trainers: what they train

1 Classroom-based

0

20

40

60

80

100

120

Preparing yourown classroom

tests

Using ready-made tests

Givingfeedback

Using self /peer

assessment

Using informal,continuous,

non-test

Using portfolio

Not at all

Both

In-service

Pre-service

Correlation between assessment-related activities teacher trainers give training in, and absence of their own professional training

1 Classroom-based

0

10

20

30

40

50

60

Preparing own classroom te

sts

using ready m

ade tests

giving feedback

using self/peer a

ssessment

using inform

al, contin

uous tests

using portfolio

Teacher trainers: what they train

2 Purposes

0

20

40

60

80

100

120

To give grades To find out whatneeds to be

taught

To placestudents

To award finalcertificates

Not at all

Both

In-service

Pre-service

Correlation between assessment-related activities teacher trainers give training in

and absence of their own professional training

2 Purposes

0

10

20

30

40

50

To give gradesTo find out whatneeds to be

taught

To placestudents

To award finalcertificates

Teacher trainers: what they train

3 Contents and concepts

0

20

40

60

80

100

120

Test

ing re

cept

ive s

kills

Test

ing p

rodu

ctive

skil

ls

Test

ing g

ram

mar

/voc

abula

ry

Test

ing in

tegr

ated

lang

uage

skil

ls

Test

ing a

spec

ts o

f cult

ure

Estab

lishin

g re

liabil

ity

Estab

lishin

g va

lidity

Using

stat

istics

Not at all

Both

In-service

Pre-service

Correlation between assessment-related activities teacher trainers give training in

and absence of their own professional training

3 Contents and concepts

05

1015202530354045

Receptive skills

Productive skills

Microlinguistic aspects

Integrated language skills

Aspects of culture

Establishing reliability

Establishing validity

Using statistics

Tentative conclusions from teacher trainer data

• Teacher trainers perceive their own needs similarly to teachers• Much of what they train in is in areas in which they have no

professional training themselves (generally 30%-50%)• Areas of greatest need, where teacher training is not given, or

given by people with little professional training, largely coincide with the areas of greatest need in teachers:– ELP/other portfolios, preparing own tests, peer/self assessment,

interpreting results and giving feedback + informal continuous assessment

– To award grades, to find out what needs to be taught, placement, awarding certificates

– Aspects of culture, integrated skills, establishing reliability and validity, statistics + productive skills

Correlation between activities experts are engaged in and absence of professional training

  1 Item writing and rating

0

5

10

15

20

25

ratin

g sp

eakin

g or

wri.

..

mak

ing o

pen-

ende

d r..

.

writing

item

s/ta

sks

revie

wing it

ems/

test

s

actin

g as

inte

rview

er

mak

ing d

ecisi

ons

abo.

.

Experts’ perceived needs : do you need training in this area?

1 Item writing and rating

0

5

10

15

20

25R

atin

g sp

eaki

ngor

writ

ing

Mar

king

ope

n-en

ded

resp

onse

s

Writ

ing

item

s/de

sign

ing

test

task

s

Rev

iew

ing

item

s or

test

s

Act

ing

as a

nin

terv

iew

er

Mak

ing

deci

sion

s ab

out

com

posi

tion

ofte

sts

No

Yes

Correlation between which activities experts are engaged in and absence of professional training.

2 Developing tests and assessment systems

05

1015202530

planning new tests

designing te

st sp

eci...

defining ass

essmen...

pilotin

g items a

nd tests

training ra

ters etc.

statis

tical a

nalysis

o...

settin

g pass m

arks...

item banks

valid

ation re

search

Experts’ perceived needs : do you need training in this area?

2 Developing tests and assessment systems

0

5

10

15

20

25

Plannin

g ne

w test

s, s

yste

ms,

fram

ewor

ks

To fin

d ou

t wha

t nee

ds to

be

taug

ht

Design

ing te

st s

pecif

icatio

ns

Design

ing a

sses

smen

t crit

eria

Pilotin

g ite

ms

and

test

s

Traini

ng ra

ters

, mar

kers

, inte

rview

ers

Statis

tical

analy

sis o

f tes

ts a

nd ite

ms

Settin

g pa

ss m

arks

/cut

-off

scor

es

Creat

ing a

nd m

ainta

ining

item

ban

ks

Doing

valid

ation

rese

arch

No

Yes

Correlation between which activities experts are engaged in and absence of professional training

3 CEF related

23

24

25

26

27

28

29

30

using CEF as the basis fornew tests

relating existing tests tothe CEF

Experts’ perceived needs : do you need training in this area?

3 CEF related

0

5

10

15

20

25

30

Using CEF as basis for new testsystems

Relating existing tests to CEF

No

Yes

Tentative conclusions on experts’ needs

• Experts generally perceive need for training across the board

• Areas of particular need:– Item writing, reviewing items or tests, making decisions

about composition of tests– Statistical analysis, setting pass marks, creating/maintaining

item banks, doing validation research– Relating existing tests to the CEF, using CEF as basis for

new test systems

Regions compared

1. Southern Europe– Portugal, Spain, Andorra, Italy, Malta, Greece, Turkey2. Western Europe– Ireland, UK, France, Belgium, Netherlands, Germany,

Austria, Switzerland3. Northern Europe– Iceland, Norway, Sweden, Denmark, Finland4. Eastern Europe– Estonia, Latvia, Lithuania, Poland, Russia, Czech Rep.,

Slovakia, Hungary, Bulgaria, Romania, Ukraine, Slovenia, Croatia, Serbia and Montenegro, Macedonia, Azerbaijan

5. non-European countries

• Questionnaire 1 / Teachers

Teachers’ previous training

2 = A little (1-2 days)

1,8

1,9

2

2,1

2,2

South West North East non-Eur.

Teachers’ need for training- average number of activities, etc where need

for assessment training was reported (out of max 25)

02468101214161820

South West North East non-Eur.

Advanced training

Teachers’ reported need for advanced training 1

(activities out of max 25)

0 5 10 15 20

Belgium

Bulgaria

Estonia

Finland

Greece

Hungary

Ireland

Netherlands

Norway

Teachers’ reported need for advanced training 2

(activities out of max 25)

0 5 10 15 20

Norway

Poland

Romania

Slovenia

Spain

Sweden

Turkey

UK

Other Eur.

Statistically significant differences

Teachers’ previous training• no differences between regions

Teachers’ need for advanced training:• Western Europe < Eastern Europe• Western Europe < Southern EuropeTeachers’ need for basic training:• no statistically significant differences

• small n size made it impossible to calculate differences between individual countries

Top 5 needs (teachers / regions)

1st 2nd 3rd 4th 5th

South Portfolio

73%

Interpretation 65%

Statistics ext. 64%

Continuous 62%

Criteria

61%

West Reliability

55%

Validity

54%

Statistics

50%

Portfolio

45%

Self / peer

45%

North Portfolio

70%

Self / peer

63%

Reliability

62%

Validity

62%

Statistics

61%

East Portfolio

72%

Reliability

72%

Statistics

71%

Validity

71%

Interpretation 65%

Implications for training event(s)

• Common training events could be designed for both teachers and teacher trainers

• These should aim to cover the needs revealed here as pressing; however, it is likely that more ’core’ needs may exist for grass roots teachers (further investigation needed)

• A separate, specialist, training event would be necessary to cater for experts’ needs

• Regional events may be adapted to reflect different priorities

And finally..

Beneficial to belong to EALTA?

0

10

20

30

40

50

60

Teachers Teacher trainers Experts

Serie1

Serie2