what do the kids think ? a quantitative analysis of feedback questionnaires in standardised...

What do the kids think?

A quantitative analysis of feedback questionnaires in standardised reading tests

Eva Konrad & Annabell Marinell

Outline

• Research questions• Secondary school types in Austria• The research instrument• Results• Conclusions

A. Marinell & E. Konrad IATEFL TEASIG 2011

Research questions• Is there a connection between

– the test takers’ familiarity with the topics – their interest in the texts used – how difficult the test takers felt the texts to be– their familiarity with the test methodsand how well they felt it tested their reading ability?

• Is there a connection between the test takers’ total scores and how well they felt the standardised reading test tested their reading ability?

• Is there a difference in test takers’ responses from different school types?


Secondary school types in Austria


AHS BHS

1. HUM: comprise business studies, theory in the respective area as well as compulsory work placements

2. HAK: secondary schools for occupations in the business sector

3. HTL: secondary schools for engineering

4. BAKIP: nursery schools

5. BASOP: colleges for social education

6. HLFS: colleges for agriculture and forestry

1. Classical secondary academic school

2. Secondary academic school emphasizing mathematics and science

3. Secondary academic school emphasizing economics

4. Upper-secondary academic school specializing in instrumental music

5. Upper-secondary academic school specializing in fine arts

Research instrument: test booklet

• 4 independent reading tasks• Test formats:

• Multiple Choice (MCQ)• Multiple Matching (MM)• Sequencing (SEQ)• Note Form (NF)• True/False/Justification (TFJ)

(dropped: True/False/Not given)

• Answer sheet• Questionnaire


Research instrument: questionnaire


Questionnaire excerpt

Selected questions for research

1. How familiar were you with the topic of the texts?2. How interesting did you find the texts?3. How difficult did you find the texts?4. How familiar were you with the test methods?5. How well could you show your reading competency

in this test?


Face validity“The extent to which a test meets the expectations of those involved in its use, e.g. administrators, teachers, candidates and test score users; the acceptability of a test to its stakeholders.”

(McNamara 2000)

For our research:How well did the test takers feel the standardised reading test was able to measure their reading competency?


Test populationTrial 2008: AHS Trial 2011: AHS Trial 2011: BHS

(N=1665) (N=3041) (N=914) School type

Booklet 1 184 Booklet 1 303 Booklet 1 124 BAKIP

Booklet 2 192 Booklet 2 306 Booklet 2 128 HAKBooklet 3 186 Booklet 3 273 Booklet 3 160 HAK

Booklet 4 184 Booklet 4 297 Booklet 4 115 HTL

Booklet 5 183 Booklet 5 299 Booklet 5 112 HTL

Booklet 6 181 Booklet 6 388 Booklet 6 92 HUM



Booklet 9 117 Booklet 9 405

Booklet 10 88


Method

Programme

• MSB statistics programme• Developed by Gerhard Marinell and Gabriela Steckenberger


Method

MSB statistics interface

Data matrix


Significance level as quoted in Henning (1987): α=0.05

Question 1

Is there a connection between • the test takers’ familiarity with the topics • interest in the texts used• how difficult they felt the texts to be and how well they felt their reading ability was tested?

Is there a connection between these factors and the test takers’ total score?



face validity x familiarity with topic

face validity x interest in texts

face validity x difficulty of texts

0 20 40 60 80 100 120

25

40.6

90.6

38.9

61.1

100

22.5

57.5

27.5

2008 AHS2011 AHS2011 BHS

Values in %


total score x familiarity with topic

total score x interest in texts

total score x difficulty of texts

0 20 40 60 80 100 120

15.6

25

43.8

22

50

97

17.5

22.5

30

2008 AHS2011 AHS2011 BHS

Values in %

Values in %

face validity x familiarity with topic

total score x familiarity with topic

face validity x interest in texts

total score x interest in texts

face validity x difficulty of texts

total score x difficulty of texts

0 20 40 60 80 100 120

25

15.6

40.6

25

90.6

43.8

38.9

22

61.1

50

100

97

22.5

17.5

57.5

22.5

27.5

30

2008 AHS2011 AHS2011 BHS

Question 2

Is there a connection between the test takers’ familiarity with the test methods and how well they felt their reading ability was being tested?



face validity x multiple choice

face validity x note form

face validity x multiple matching

0 20 40 60 80 100 120

50.0

23.3

42.9

100

100

100

89

60

86

2008 AHS2011 AHS2011 BHS

Values in %

Question 3

Is there a connection between the test takers‘ perception of how well their reading competence was tested and their total scores?


Values in %

face validity x total score

0 20 40 60 80 100 120

75

100

80

2008 AHS 2011 AHS 2011 BHS

Conclusions• We could observe certain tendencies:

– Correlations in all of the factors were more frequent in 2011 than in 2008

– Correlations with face validity were more frequent than with total score

• Further analysis of feedback questionnaires is needed• We will be able to say more about BHS after more trials• We still have several questions:

– Are there differences between the different BHS school types?– What further information might we gain from the open

questions?


Bibliography

• Alderson, C. J., Clapham, C., & Wall, D. (2005). Language test construction and evaluation. Cambridge: CUP.

• Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge: CUP.

• Henning, G. (1987). A guide to language testing: development, evaluation, research. Cambridge, Mass: Newbury House Publishers.

• McNamara, T. (2000). Language testing. Oxford: OUP.

Thank you for your attention and enjoy the rest of the

conference!


what do the kids think ? a quantitative analysis of feedback questionnaires in standardised...

Documents

normal secondary schools

standardised school

reading ability

test takers familiarity

test development

test methodsand

test takers responses

test takers total scores