evaluation of a family literacy program (mocep): the turkish example ayhan aksu-koÇ yeditepe...
TRANSCRIPT
EVALUATION OF A FAMILY LITERACY PROGRAM
(MOCEP): The Turkish Example
AYHAN AKSU-KOÇ Yeditepe University & Boğaziçi University
in collaboration withSEVİM CESUR & SABİHA ÖRÜNĞ
Istanbul University
5th Meeting of the “QualiFLY” ProjectHamburg, 19-22 February 2007
Overview of talk
• My experiences in evaluation:
• Two family literacy interventions launched by MOCEF
– Mother Child Education Program (MOCEP): home-based
– Evaluation study only
– Summer School Intervention Program: center based– Program development and evaluation study
• Focus on issues of sample selection, measures, analyses, and outcomes
FAMILY LITERACY: a multi-dimensional concept
• Beliefs about and practices of literacy among family members and its intergenerational transfer to children. (Wasik & Herrmann, 2004).
• Research in this domin covers (Morrow and Paratore, 1993): – Naturally occuring literacy and language
practices within families, – Family and parent influences on children’s
literacy, language and reading. – Intergenerational literacy intervention programs,– Home-school partnerships.
LITERACY• Literacy is a set of complex multi-dimensional skills
that improve over an individual’s lifetime such as – Reading, writing, use of analytical skills and
knowledge in a given subject matter (Anderson & Pearson,1984; Snow, Burns , & Griffin, 1998)
• Learning how to read and write calls for– grammatical and discourse competence achieved
at home • Literacy: a social practice that varies with
– the family’s socio-economic level, – social and political relations, – cultural beliefs and values,
(Berman & Slobin, 1994; Dickinson & Tabors, 2001; Gee, 1989; Heath, 1983; Wigglesworth & Stavans, 2001; Minami, 2002, Snow & Dickinson, 1990)
NATURALLY OCCURRING LITERACY PRACTICES in the FAMILY
• Importance of social and cultural contexts (Gadsden, 1995, 1996; Hart & Risley, 1995; Heath, 1983; Taylor, 1983)
– MC regard literacy as entertainment; promote play with print, more independent reading by child
– LC: regard literacy as work, emphasize reading practice
• Risk factors for low-income and migrant children: – low literate parents, – poorer educational opportunities, – home language other than school language
(Baker, Serpell, & Sonnenschein, 1995; Bus, van IJzendoorn, & Pellegrini, 1995; Lonigan, 1994 (Snow, 1994; Snow et al., 1998; Teale, 1986;
FAMILY LITERACY INTERVENTIONS
• Provide services to parents and children related to language and literacy skills.
• Include: (1) direct service to children in center-based
programs (2) adult education, (3) intergenerational programs including early
childhood, parenting and majority language education,
Mother Child Education Program(MOCEP)
an Intergenerational Literacy Intervention
• A home-based early childhood education and parent involvement program for parents with limited formal education (Kağıtçıbaşı, Bekman, Kuşçul, Özkök & Sucuka, 1995).
• Holistic approach focusing on mother-child interaction to stimulate healthy development by:– sensitizing the mother to developmental needs of
the child, – promoting the development of cognitive and
linguistic skills through the mediation of mother, – directing the mother to interact and communicate
with her child effectively
MOCEP
• Duration of program: 6 months• weekly meetings to train mothers to do educational
activities with their children • weekly group discussions for effective parenting• monthly home visits with feedback to ensure fidelity of
implementation
• Training aimed to develop: – cognitive skills: (through exercises in classification,
seriation, discrimination, etc.) – text-level abstract language skills: (through book-reading,
personal narratives, dinner-time conversations )– print-related skills: (through exercises in phoneme
recognition and segmentation, letter recognition and writing) (Kağıtçıbaşı, Sunar, & Bekman, 1988).
What were mothers taught and
what were children exposed to? • Mothers were trained to co-work with their children on:
– bookreading with questions-answers,– letter or word recognition by sound,– isolating first sound of words represented pictorially,– copying numbers, letters or shapes, – making one-to-one correspondences, – identifying similarities and differences,– classification and seriation.
• Mothers received handouts, books, crayons etc., to implement the activities at home.
• Mothers expected to work with child for 15-20 mins/day for 5 days.
AIMS OF THE EVALUATION STUDY
• To investigate the effects of home enrichment as an intergenerational intervention program,– immediately after its completion,– at the end of the first grade, and – at the end of the fourth grades.
• To explore the effect of different socio-economic backgrounds on children’s– cognitive and preliteracy skills, – literacy acquisition,– later academic achievements.
• Therefore, it is also a study of naturally occurring literacy practices.
• To evaluate the outcome, not the process.
DESIGN• A quasi-experimental design• One experimental and two control groups:
– Working class training (WCT)– Working class control (WCC)– Middle class (MC)
• Children were assessed at:– Baseline (pre-test) – End of the program (post-test 1) – Follow-up I (end of 1st grade)– Follow-up II (end of 4th grade)
SAMPLE• Program implemented in low-income areas of Istanbul.
• Participation on voluntary basis by mothers with a low-level of formal education and a 5-6- years-old
• WCT group -- selected among those attending the program.
• WCC group -- selected among families matched for child’s age and area of residence
• MC group -- selected among families matched for child’s age, living in high-income areas; child should NOT have attended preschool.
QUESTION of REPRESENTATIVENESS• Caution in drawing conclusions since random selection is
seldom possible:– Problem of self-selection:
• of families eligible for intervention, only those who chose to do so participate
• of participants selected for the evaluation, some may not agree to participate.
• To ensure representativeness: – recruit a group of volunteers representative of the
population and randomly assign into the treatment and control groups,
– determine how many families in a given area were eligible and how many participated,
– determine the level of attrition in the treatment and control groups and compare their pretest score, parent education, and marital status with those who remained to see if similar.
SAMPLEWorking Class Training
Working Class Control
Middle Class Control
Pretest (N=119)
5;0-6;0 year olds who never attended preschool
44 39 36
Post-test 1 (at end of 6 month program) (N=107)
41 35 31
Follow-up I (1st grade) (N=88) 32 33 26
Follow-up II (4th grade) (N=50) 21 14 15
Prefer larger sample size as caution for drop-outs in successive follow-ups.
VARIABLES OF ASSESMENT • For Home-Environment Comparisons:• Mother interview:
• Demographic characteristics of family – Parents’ age, education, occupation, income-level
• Literacy activities and resources in the home– Parental literacy practices
» Daily newspaper, mother book-reading, mother’s favourite book, frequency of book-reading
– Parent-child literacy activities» Length of time of bookreading to child, frequency of
bookreading– Availability of literacy materials
» Number of books of child, of parents, etc. . • Child-rearing practices
• Observations: • Availability of literacy materials• Parent-Child bookreading behaviours in the home
Content of Intervention and Outcomes
Content of intervention (basic concepts, oral language skills, phonological processing skills, print awareness)
Child Measures
(Pre-test & Post-test I)
School readiness
Classification, Seriation, Discrimination, Part-whole relations, basic concepts
Classification, Seriation, Discrimination, Part-whole relations
Logical thinking, Conceptual development
Mother-child bookreading, Explanatory question- answer talk with child, basic concepts
WISC-R Definitions,
PPVT Receptive vocabulary,
Listening comprehension
Text level abstract language use,
Vocabulary, Reading comprehension
Isolation of first sounds
Sound recognition
Letter Naming & Recognition
Phonological Awareness, Rhyming, Syllabification,
Letter Naming & Recognition, Book Reading Behaviors
Phonological processing, Print related skills for decoding and encoding,
Awareness of print conventions
1st grade and 4th grade outcome variables
Follow-up I (1st grade) Follow-up II (4th grade)Encoding : writing Real Words Encoding: Writing the description
of a children’s game told by E
Decoding : reading Real Words and Reading Pseudo Words
Reading Comprehension: read a
story-passage, answer questions
Reading comprehension: reading a
story and answering questions
Narrative production: sequencing a
set of story pictures, telling the story
Word definition: WISC-R
definitions task
Achievement in Math, Science,
Turkish, & Soc. Sci.
Measures: STANDARDIZED vs. NONSTANDARDIZED
• Prefer standardized tests if available: – for language /culture
• TOLD available for English but not Turkish,• PPVT for Turkish and English
– For age of child• PPVT can be used at age 5, age 9 or later
– For domain to be assessed• Mother-child bookreading behaviors
– Consider the time required for administration relative to information to be obtained
• Narrative production – coding is time consuming • Picture sequencing – easy to code but may not reflect the skill you
want
• Use informal measures otherwise: 2 examples
Measures: STANDARDIZED vs. NONSTANDARDIZED
• Nonstandardized measures:– are also effective predictors, may be more suitable
for some contents and contexts• Narrative retelling – for production• Story-picture elicitation – for production• Child bookreading behaviors,• Tests for classification, discrimination• Elicited Imitation – for grammar
An example: NARRATIVE COMPREHENSION & PRODUCTION
• Narratives (story-telling) entail units beyond the sentence, i.e., discourse level competence.
• Narratives require two types of mental schemes in addition to language schemes. – scripts (Nelson, 1996)
• Abstract knowledge about familiar events that occur in particular contexts (shopping, bath).
– story grammars (Stein & Glenn, 1979, 1982):
• Schemas used to structure the actual telling of the story – a beginning, a complication or a problem, and an
outcome – information about characters’ motives, plans and
feelings.
Importance of Narrative for Literacy Acquisition
• Scripts and story grammars also serve text comprehension in classroom discourse.
• Narratives:– serve as a developmental bridge to more advanced literary
forms such as expository texts because events are related temporal and causally rather than logically.
– provide the context for learning new syntactic structures and vocabulary items.
• Scripts and story grammars develop during the preschool years through:– adult-child conversations (e.g. dinner-time talk)– experience with oral stories, story-books and TV– telling and acting out stories (Nicolopoulou, 2004, use in
Head Start)
Mesures to assess Narrative Competence
• Elicitation of stories with pictures:– Single picture stories
– Sequenced picture stories
• Retelling after an orally presented story with or without pictures
• Stories produced using toys as props
• Spontaneously produced stories
CAT STORY:Picture description vs. Story telling
• MC (7;0. YB.1a.EY )….. Once there was, once there wasn’t. Once upon a time a bird had three ba… {there were} birds. Their mother had gone [MCL] to bring them worms [COMP]. They look and (see) a cat! {Good, a bit louder.} … Their mother was going. The cat, a cat was climbing the tree. The dog held its tail, the cat fell down. Their mother brought the worm. And the dog ran and went away.
• WC (7;2. EK.1a.SG) Cat, bird, dog {But tell nicely} Dog, bird, fish, sea, tree {What’s happening in this story?} There is cat, bird. And the tree also has grass. …And the tree has its grass, I said.{What else? Why not look at the pictures?} There is cat, there is dog, there is bird, there is the grass of the tree, the ground is yellow, there is dog, that is all.
• 49.8% of MC compared to 23.6% low-income children displayed ‘story telling’.
Aksu-Koç, Taylan & Bekman (2002).Need assessment in early childhood education and an evaluation of children’s level of linguistic competence in three provinces in Turkey. Project Report 00 R101. Report presented to MOCEF and Bogazici University Research Fund. June 2002.
Middle-income Low income
1st grade 2nd grade 1st grade 2nd grade
PPVT- Vocabulary
61.33 67.10 44.65 59.12
NARRATIVE
Plot-structure (max=6)
3.18 3.21 2.17 2.56
Mean # of Clauses (ns)
15.68 15.19 17.46 17.45
Mean # of Subordinate
Clauses (ns)
1.97 2.08 1.78 1.71
CAT STORYMean Vocabulary and Narrative scores for
Istanbul children by Home-context and Age
MC and Low-income children differ in terms of the organizing schemas (p<.05) rather than linguistic structures used for their expression.
Another example: Nonstandardized measure for SYNTACTIC COMPETENCE
• Elicited Imitation Test:– 16 sentences incorporating specific grammatical
structures of Turkish, – controlled for length in words (5-6), in
morphemes (10-14), and for memory constraints.• Each child is asked to repeat the model sentence right
after the experimenter reads it. • Coding in terms of three types of repetition:
– (i) correct (verbatim or with minor modification), – (ii) modified (resulting in structural and semantic
change), – (iii) no or ungrammatical repetition
Order Stimulus Sentence Experimental Control
1 8. Yemek yedikten sonra dişimizi fırçalamalıyız. 28.3 53.8
2 12. Ayşe yüzme bilmediği için denize giremedi. 12.0 32.3
3 10. Boyaları arkadaşıma verdiğimi unutmuşum. 15.2 21.5
4 14. Annemin pişirdiği börek çok sevilir. 28.3 48.4
5 13. Annemin bana dondurma almasını isterim. 25.0 46.3
6 9. Okulun bahçesinde ip atlayan kızlar var. 20.7 34.4
7 4. Ağaca çıkan kedi inmeye çalışıyor. 16.3 34.4
8 7. Annemle teyzem konuşurken biz oyun oynadık. 13.1 35.5
9 6. Makarnayı ıspanaktan daha çok severim. 20.7 28.0
10 3. Okul başlamadan önce saçımı kestireceğiz. 12.0 23.8
11 16. Büyüdüğüm zaman doktor olmak istiyorum. 8.7 31.2
12 1. Babam araba alıp bizi gezdirecek. 10.9 20.4
13 15. Ben okula yürüyerek gidebilirim.. 5.4 25.8
14 5. Yeni topumu kaybedersem çok üzülürüm. 7.6 22.6
15 2. Misafir gelince teyzemlere gidemedik. 8.7 19.4
16 11. Limon ekşidir ama acı değildir. 1.1 8.6
% of Ungrammatical Repetitions by Experimental and Control groups to Stimulus Sentences ordered in decreasing difficulty
ANALYSES and RESULTS
MOCEP EVALUATION STUDY
HOME ENVIRONMENT MEASURES on LITERACY ACTIVITIES
Mother Interview
Working Class Training
Working Class Control
Middle
Class
Frequency of buying newspaper
2.45 2.16 2.88 MC > WCT & WCC
WCT > WCC
Frequency of buying magazines
1.45 1.5 2.1 MC > WCT & WCC
Others read at home 1.66 1.69 2.05 MC > WCT & WCC
Frequency of mom bookreading to child
1.97 1.60 2.30 MC > WCT & WCC
WCT > WCC
Age look at pictures 40 mos 35 mos 28 mos MC > WCT & WCC
Age since first read books 45.41 43.25 37.25 MC > WCT & WCC
Frequency of others reading to child
1.62 1.9 2.3 MC > WCT & WCC
CHILD MEASURES: Pre-test Comparisons• WCT and WCC were matched for parental education, income level, mother’s age,
rate of mothers’ employment.
• WCT = 64 mos WCC = 68 mos, MC = 67 (p<.05); age effects were partialled out in the subsequent analyses.
• MC children scored higher than WCT & WCC on: (p>.001)– PPVT– Classification– Discrimination– Phonological awareness– Listening comprehension – Word definitions
• MC children scored higher than WCT on: (p>.001)– Part-whole relations– Rhyming– Letter naming – Letter recognition– Syllabification
• WCC children scored higher than WCT on (p>.001)– Phonemic awareness– Letter naming
IMMEDIATE EFFECTS:Posttest I- Pretest differences by Environment
0
4
8
Diff
ere
nce
S
core
s
ListeningComprehension
WISCR PhonemicAwareness
Classification Discrimination
W C T
W C C
M C
Figure1. Difference Scores for Postest I - Pretest Variables As a Function of Environment
WCTWCCMC
IMMEDIATE EFFECTS of INTERVENTIONSummary
• Not apparent for all variables when compared to control groups,
• Therefore should look at within group growth curves for positive change
• Some skills may need more time to benefit from intervention
0
0,5
1
1,5
2
Per
form
ance
Spelling Comprehension Decoding RealWords
DecodingPseudo Words
WCTWCCMC
FOLLOW-UP I: Impact on Literacy Skills at Grade 1
Decoding Real Words
1. Classification%17
2. WCT X Listening
Comprehension%5
3. Naming Letters
%74. Child Book
Reading%4
5. Other%67
FOLLOW-UP I: Posttraining Predictors of Literacy Skills at Grade 1
Reading Comprehension
2. Child Book Reading
%10
3. WISCR2%6
5. Other%62
4. Write Letters
%4
1. Superordinate
Categories%18
FOLLOW-UP I: Posttraining Predictors of Literacy Skills at Grade 1
FOLLOW-UP ISummary
• Significant effects on mechanical print skills, but not on reading comprehension, a higher-level conceptual skill.
• More powerful assessment if age-appropriate measures had been re-administered.
– e.g. PPVT, WISC-R, Listening Comprehension
• Should control for the quality of school children go.
• Should cooperate with teachers to capture other differences in performance.
FOLLOW–UP II Impact of Intervention at Grade 4
Measures:
– Encoding: writing the description of a children’s game told by experimenter
– Reading comprehension: reading a story and answering questions
– Narrative production: sequencing a set of story pictures, telling the story
– Word definition: WISC-R definitions task
– Achievement Tests in Math, Science, Turkish, & Soc. Sci.
FOLLOW–UP II Impact of Intervention at Grade 4
Academic Achievement at Grade 4 by Environment
0
0,5
1
1,5
2
2,5
3
3,5
4
Per
form
ance
Science Mathematic Social
WCTWCCMC
frequency of book reading to child Sequencing events in narrative
word-definition skills
frequency of mother's story-telling
Written expository text information
reading of familiar books, Organization of written text on paper
listening comprehension Spelling
knowledge of letter
listening comprehension
Reading comprehension predicted
the number of books possessed,
maternal discussions about writing,
child's ability to write his name,
ability to read familiar books and
skill in part-whole relations.
Word-definition skill
frequency of book reading to child Success in Social Sciences
knowledge of letters
understanding of part-whole relations
Success in Science
Predictors of Literacy Skills and Achievement at 4th Grade
Predictors Outcomes
Written Text Total Information
3. Other%72
2. Frequency of mom telling story to child
%11
1. Definitional skills (WISC-R)
%17
FOLLOW–UP II Posttraining Predictors of Literacy Skills at Grade 4
Word Definitions
1. Books possessed%26
2. Do child and mom talk about writing
%173. Child writing own
name%16
4. Read familiar books%11
5. Whole-part relations
%6
6. Other%24
Reading Comprehension
1. Letter recognition
%28
2. Story comprehension
%8
3. Other%64
FOLLOW–UP II Posttraining Predictors of Literacy Skills at Grade 4
FOLLOW-UP IISummary
• Effective in developing skills necessary for assimilation of verbal material.
• Exposing the child to narrative discourse by book-reading and story telling promotes long term competence for text-level language use.
• An educated mother who engages in literacy interactions with the child has most positive effects.
• Early competence in basic concepts and analytic skills contributes to science achievement.
FOLLOW-UP IISummary
• Need to use more powerful analyses.
– Use of statististical methods that allow us to see the contribution of different variables measured at different points in development.
• e.g. Path analysis, as in evaluation of the Summer School Intervention.
Path analysis for Summer School Intervention
Path analysis
Mother’s Education Correct R. 77 Correct R.
Pretest PROGRAM Posttest
.23 .94 .15 . 60
.18 .16 . 85.18 . 51
Modified R. .36 Modified R.
Narrative C. .10 Narrative C. .97
Father’s Education . 98 .21
Site 2 -.19 Vocabulary .52 Vocabulary . 85
Lit .46.25 .65.96 Lit
.56
Num .96 Num .48Site 3 .50
.97
.98
- .18 . 20
CONCLUSIONS, IMPLICATIONS and RECOMMENDATIONS
• Effects on mechanical skills (encoding and decoding) are observed earlier than effects on higher level cognitive functioning (listening comprehension). These develop in home contexts with high literacy activities, over longer periods of time. – Enrichment programmes should start earlier and be designed for longer
periods, for example, start at 4;0 years and spread over two years
• Effects of home enrichment programmes need time for the sedimentation of the skills gained, and the right context for their expression. – Long-term follow-ups are also necessary– Preplanning for follow-ups is important (sample size, measures, monitoring,
etc.)
• For a full evaluation of the intervention, need to assess the parent education component of the program.
CONCLUSIONS, IMPLICATIONS and RECOMMENDATIONS
• For a full evaluation of the intervention, need to assess the process of implementation of intervention.– records of rates of attendance and compliance– the degree to which mothers deliver the key parts of the
curriculum at home• Summer School Intervention
– a challenge for home based programs !
• For long term effects, try to ensure continuity between intervention, home and school.– teachers can be trained to support children’s development,
observe and do in progress-assessments. – Portfolios kept at home and in school.
• Assessment should involve multiple perspectives and sources of data.
CONCLUSIONS, IMPLICATIONS and RECOMMENDATIONS
• Meaningful to assess general intelligence for matching purposes (for both child and mother)
• Evaluation study done by an independent researcher not involved in program development
• Important to take into account values, beliefs, social practices and other learning experiences of the group reached.
THANK YOU