nwo / proo dutch programme council for educational research

Netherlands Organisation for Scientific Research

NWO / PROODutch Programme Council for Educational Research

Instruments for

measuring literacy

development of

at-risk adolescents

Ron Oostdam, Amos van Gelderen,

Roel van Steensel

University of Amsterdam


Literacy development of at-risk students in multilingual contexts: A tale of three cities

• International project in cooperation with the Ontario Institute for Studies in Education (OISE) and Service de la recherche en éducation (SRED)

• Main research questions:

What are the most important socio-cultural, educational and individual variables associated with literacy development of at-risk adolecents (grades 7 to 9) in multilingual contexts?

Do the different variables play similar roles in predicting at-risk and non-at-risk adolescents’ literacy?


Factors influencing students’ literacy

Home and

Community Environment

Individual Attributes

School Environment

Literacy Skills


Design of the research project

groups of variables

School environment (part- project 1)

1) nature of literacy tasks 2) embeddedness of literacy in curriculum3) Instruction selfregulatory literacy strategies4) connections with home-environment5) social context of teaching & learning

home and community environment(part-project 2)

1) nature of literacy tasks2) embedding of literacy3) use of self regulatory literacy strategies4) connections with school-environment5) social context 6) personal backgrounds

individual attributes(part-project 3)

1) knowledge of vocabulary and grammar2) metacognition reading and writing3) fluency of word and sentence processing4) non-verbal intelligence5) attitudes towards literacy


Integrative part project 4

Part project 4 is directed at the measurement of literacy The whole research project consists of a longitudinal part (3

years) and a large scale cross-sectional part Longitudinal approach consists of an in-depth study of a focused

sample of 40 at-risk adolescents The part-projects monitor the participants on the different groups

of variables The cross-sectional study takes place in part-project 4 only Results from the part-studies 1-3 are integrated In the cross-sectional study predictions about the relations

between the variables and literacy are tested


cross-sectional study

Allows testing with substantial statistical power for hypotheses derived from the multiple case-studies

Allows analyzing interactions between socio-cultural, educational and individual variables in a statistically reliable way

Allows comparison of literacy skills between at-risk and non-at-risk students


International Literacy Test

Literacy skills defined in a broad way: the degree in which students are able to read and write in order to participate in all kinds of functional and cultural exchange in the surrounding society

The PISA definition of reading is adopted (the ability to understand, use and reflect on written tasks in order to participate effectively in life) and extended to writing

Reading and writing tasks were constructed in close cooperation with Toronto and Geneva


Criteria for measuring literacy

Criterium of socio-cultural validity: measurement of literacy takes place with reading and writing tasks grounded in the lives of students

Test has to take into account differences in literacy skills For international comparison the tasks need to be designed in

such a way that they approach international, socio-cultural equivalence


Taxonomy literacy test

• Four types of text types: narrative, expository, argumentative, instructive (reading and writing)

• Four types of media: (school)books, newspapers and magazines, ‘official’ documents, internet (reading)

• Distinction made between continuous and discontinuous texts (reading)

• Three different types of questions: retrieving, interpreting, reflecting (reading)

• Two answer formats: multiple choice and open (reading)


Reading tasks pilot study(* = discontinuous)

Narrative(2 tasks)

Expository(7 tasks)

Argumentative(2 tasks)

Instructive(3 tasks)

Internet(2 tasks)

Wikipedia Website camping

(School)Books(3 tasks)

King clothes BasiliskVietnam

Newspapers & Magazines(8 tasks)

The board BokitoTV-guide*Newsletter*Aroma Jockey

Cell phone banBio-fuel

Recipe*

Official documents(1 task)

Regulations youth hostel*


Example reading task

HOUSE RULES1. Smoking and lighting fires are strictly prohibited on our premises!2. For safety and hygienic reasons hot dishes and drinks must not be prepared in

living rooms or bedrooms.3. During the night (from 10:00 p.m. until 6:00 a.m.) respect the rights of other

guests and residents to a good night’s sleep. Showers should not be used and noise should be avoided at these times. We expressly point out that in the event of disturbance caused by noise, measures can be taken against the person(s) who caused the noise, including the termination of the contract for accommodation.

4. Guests are requested to avoid waste and to be economical in using electricity, heating, and water. Kindly deposit any garbage that cannot be avoided in the appropriate containers for recycling or disposal.

5. No visitors from outside are allowed in the bedrooms. Visitors can be received in the cafeteria daily from 10:00 a.m. until 10:00 p.m. Persons who are not occupants of our hostel cannot stay overnight in the hostel.


Retrieving

5. No visitors from outside are allowed in the bedrooms. Visitors can be received in the cafeteria daily from 10:00 a.m. until 10:00 p.m. Persons who are not occupants of our hostel cannot stay overnight in the hostel.

What do the rules say about visitors from outside?

a) There are no restrictions with respect to receiving visitors from outside.

b) Visitors from outside are not allowed.c) Visitors from outside are only allowed at certain

times and in a certain place.d) Visitors from outside are only allowed after the hotel

management has given permission.


Interpreting

3. During the night (from 10:00 p.m. until 6:00 a.m.) respect the rights of other guests and residents to a good night’s sleep. Showers should not be used and noise should be avoided at these times. We expressly point out that in the event of disturbance caused by noise, measures can be taken against the person(s) who caused the noise, including the termination of the contract for accommodation.

Why can’t guests take showers during the night?

a) Because of safety and hygienic reasons.b) Because the hostel wants guests to be economical in the

use of water.c) Because the hostel does not provide hot water at night.d) Because taking a shower could disturb other

guests.


Reflecting

9. The hostel depends on a helping hand from our guests. We need your assistance with cleaning and tidying up the equipment, rooms and objects you have used during your stay as well as setting and clearing tables in the cafeteria.

What is the purpose of rule 9?

a) To offer a complaint.b) To give an order.c) To give advice.d) To make a request.


Writing tasks pilot study

Narrative(2 tasks)

Expository(1 task)

Argumentative(2 tasks)

Instructive(1 tasks)

King clothes(finish the story; last sentence given)

Bokito(letter to a friend)

Cell phone ban(article for schoolpaper)

Melissa(write a story; first and last sentence given)

Yummi Yummi Candybar(letter to producer chocolate bars)

Day out(exchange project between schools)


Example writing task

SAVE FOR TWO FREE CINEMA TICKETS!!!This is what you must do: On each Yummy Yummy candy bar wrapperthere is 1 point. Collect 10 points and send these in a sufficiently stampedenvelope to: Yummy Yummy Candy Bars Points Offer, PO Box 3333, 1273KB Etten-Leur, the Netherlands. Also include € 0.39 in stamps to coverpostage. Clearly write your name, address and postal code, and the free(FREE!) cinema tickets will be sent to your home as soon as possible. Thisoffer is open until April 15.

It is April 7. You have collected a total of 8 points, but you cannot findanymore bars with points. The bars in the shops have no points on thewrappers, but it is still not April 15. Thus, you cannot get your 10 pointstogether. Nevertheless, you wish to receive the two cinema tickets.Therefore, you send your 8 points along with two complete wrapperswithout points.

Write a letter to enclose with the points and wrappers. Explain why you are unable to send ten points. Convince Yummy YummyCandy Bars that you want to receive the two cinema tickets and that therewas nothing you could do to get ten points. Make sure that they send youthe cinema tickets! Then address the envelope.


Pilot study

Pilot of reading tasks 14 reading tasks (internet tasks not admistered in Switserland) 408 Dutch students (grade 7 and 9) 298 Swiss students (grade 7, 8 and 9) Interviews with 40 Dutch students

Pilot of writing tasks 6 writing tasks 80 Dutch students (grade 7 and 9) 24 Swiss students (grade 7, 8 and 9) Interviews with 4 Dutch students


Final selection tasks

Reading tasks Contribution of items to the overall reliability of the literacy test Difficulty level of the items Distribution of questions over formats (retrieving, interpreting,

reflecting; open or closed) Distribution of tasks over text types and media Differences between grades Opinion of students about the tasks

Writing tasks Global scoring on text quality and text length Global scoring of written texts on text type and content Opinion of students about the tasks

Practical constraints Maximum administration time (about 3 hours) Internet tasks (and writing tasks) demand computer facilities


Results reading tasks

Overall test reliability for reading tasks sufficient: .75 (Geneva, without internet tasks) and .77 (Amsterdam, including internet tasks)

Total number of items: 102: Retrieving (45), Interpreting (31), Reflecting(26)

One task showed a ceiling effect in Geneva as well as Amsterdam A relatively small number of items with a item rest correlation

below .10: Geneva (13); Amsterdam (19) All tasks, except for one, show a sufficient difference between

grades Most students evaluate the tasks as attractive and enjoyable to

read, with the exception of ‘House Rules’ and ‘TV guide’


Results writing tasks

The narrative assignments seem to work quite well• variability in content, quality and text length• In one task students ‘squeeze in’ the elements from the final sentence

right at the end of their texts

The expository assignment is problematic• Requested text type (a letter) often not delivered• Little variability in content

The argumentative assignments differ• Yummi Yummi candy bar evokes good writing products• Other task evokes texts without relevant content

The instructive assignment works well, although students find it too much work to process all the given information


Final remarks

– In the next 2-3 weeks a final selection made for 8 reading and 3-4 writing tasks

– Items with a low item rest correlation removed or adjusted

– Final data collection (first measurent) done in april-may


For further information:

Dr. R.J. Oostdam

Faculty of Social and Behavioral Sciences SCO Kohnstamm Institute for Educational ResearchUniversity of Amsterdam

P.O. Box 94208NL-1090 GE Amsterdam

phone: ** 31 (0)20 525 1330fax: ** 31 (0)20 525 1200E-mail: [email protected]


Effect sizes comparison 7th and 9th grade

Amsterdam Switzerland Difference

Bokito 0.30 0.30 .00

TV guide 0.41 0.32 .09

Recipe 0.56 0.26 .30

Aroma jockey 0.81 0.34 .47

Basilisk 0.10 0.04 .06

Youth hostel 0.37 0.21 .16

Rouler 0.51 0.46 .05

King Clothes 0.37 0.46 .09

Phone ban 0.34 0.40 .06

Newsletter 0.77 0.26 .51

Vietnam 0.37 0.46 .09

The board 0.27 0.71 .44

nwo / proo dutch programme council for educational research

Documents

risk adolescents literacy

atrisk adolescentsthe

different variables

individual variables

citiesinternational

written tasks

indepth study

research projectintegrative