content

Promoting the wellbeing of Africans through policy-relevant research on population and health 1

Impact evaluation of the East African Quality in Early Learning (EAQEL)

Initiative

Presenter: Moses Ngware Contributors: ERP team

African Population and Health Research Center (APHRC)

Supported by Hewlett/QEDC May 2010


Content

Background

Research questions

Design & methods

Findings

Policy implications


Background

An intervention study being implemented by AKF- Reading to Learn (RtL/scaffolding).

Role of APHRC: Impact assessment of the proposed RtL

Promoting the wellbeing of Africans through policy-relevant research on population and health

Are children in lower primary grades (1, 2 and 3) able to read and do mathematics calculations more proficiently as a result of the RtL approach?

Are there differences in proficiency for children exposed to parental involvement in the RtL Approach (core model plus) compared:• to those exposed to the RtL Approach with no parental involvement

(core model), • to control schools?

What are the key contributing factors to these improvements in numeracy and literacy in grades 1, 2 and 3?

Research Questions


Design - EAQEL

Randomised Control Trial (RCT)

Four districts – 2 Kenya & 2 Uganda

Treatment and control schools in each district

120 & 109 primary schools in Kenya & Uganda, respectively

229 HT, over 1000 teachers, 14,000 pupils, 7000 parents are targeted.

Intervention: i. Core module in 1 district & ii. Core module plus in the other district.

Control Treatment

District No % No %

Kinango 33 45.

83 39 54.17

Kwale 30 62.50 18 37.50

Amolatar 25 51.

02 24 48.98

Dokolo 32 53.33 28 46.67

Total 120 52.40 109 47.60


Sample determination, effect size & power calculations

Assumptions (used OD software): Significance level = 0.05 Cell size (n) or # of pupils = 20 Effect size = 0.22 (informed by literature) Correlation coeff. = 0.10

For a power of 80%, 100 schools needed.

For a power of 90%, 128 schools needed.


Data Analysis Given that the causal effect for a single unit

u cannot be observed, we aim to identify the average causal effect for the entire sample

The ATE of t (relative to c) over U (or any sub-population) is given by:

ATE =E [Y1(u) – Y0(u)] = E [Y1(u)] – E [Y0(u)]


Data Analysis

TimeTreatment

Outcome

Treatment Group

Control Group

Average Treatment Effect on the Treated


Design Issues - Kenya

EAQEL

S I P

YES NO

YES G1 G2

NO G3 G4


Methods - instruments

Assessment tools:Single numeracy test for grades G1 – 3Single literacy test in Kiswahili or Lang’o G1-3Test items range from simple to difficultCurriculum & assessment experts, practitioners and researchers

were involved – for content validity.All instruments were pre-tested.

Survey tools:head teacher’s questionnaireteacher questionnairepupil questionnairehousehold questionnaire targeted parents/guardian


Sampling Process - Kenya

Kenya’s districts: Kwale & Kinango

Kinango

Treatment Control

Non-Kensip

Kensip

Non-Kensip

Kensip

Kwale

Treatment Control


Sampling process - Uganda

Uganda 2 Districts

District 1: Core Model District 2: Core Model Plus

Cluster of school zones

Control Treatment Treatment Control


BS Findings- Literacy written

No. of pupils Mean st. dev No. of pupils Mean st. dev

Overall (Both KE & UG) 4382 18.07 22.90 4365 9.08 6.61

Kenya 2223 30.69 24.50 2213 11.11 7.45

Uganda 2159 5.08 10.64 2152 6.99 4.80

Treatment 2206 16.72 21.78 2199 8.68 6.47

Control 2176 19.45 23.90 2166 9.48 6.73

Boys 2155 18.09 23.06 2158 9.17 6.66Girls 2227 18.06 22.74 2207 8.99 6.56

Grade 1 Grade 2

Literacy written (several aspects tested) - Administered orally to the pupils and were to write answers on a provided booklet (mean is out of 100%)


Literacy – 101 (Oral) Oral literacy (several aspects)- Pupils were required to

answer to a set of questions administered orally.

No. of pupils Mean st. dev No. of pupils Mean st. dev

Overall (Both KE & UG) 4318 55.27 16.56 4302 58.71 17.78

Kenya 2227 59.55 16.53 2199 65.09 18.81

Uganda 2091 50.70 15.33 2103 52.04 13.77

Treatment 2173 53.98 16.44 2162 57.40 17.36

Control 2145 56.57 16.58 2140 60.04 18.10

Boys 2114 55.37 16.48 2122 59.14 17.88Girls 2204 55.16 16.64 2180 58.30 17.68

Grade 1 Grade 2


Literacy written – Distribution of score - Kenya

050

100

150

200

250

300

350

400

Freq

uenc

y

-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.

0 20 40 60 80 100LIT WRITTEN

Source: EAQEL Baseline 1, 2009

Grade 1 Kenya

050

100

150

200

250

300

350

400

Freq

uenc

y

-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.

0 10 20 30LIT WRITTEN


Grade 2 Kenya

Poor performance in written literacy in Kenya- Both grades- Average % score <40%

Greater variability


Literacy written – Distribution of score -Uganda

0.0

5.1

.15

.2D

ensity

-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.

0 20 40 60 80LIT WRITTEN


Grade 1 Uganda

0.0

5.1

.15

Den

sity

-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.

0 10 20 30LIT WRITTEN


Grade 2 Uganda

Poor performance in written literacy in Kenya- Both grades- Average % score <10%


Literacy written – Distribution of score - Kenya

050

100

150

200

Freq

uenc

y

-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.

0 20 40 60 80 100Oral Literacy


Grade 1 Kenya

050

100

150

200

Freq

uenc

y

-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.



Grade 2 Kenya

Better performance on oral literacy than written in Kenya


Literacy written – Distribution of score -Uganda

Better performance on oral literacy than written in Kenya

050

100

150

200

Freq

uenc

y

-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.



Grade 1 Uganda

050

100

150

200

Freq

uenc

y

-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.



Grade 2 Uganda


Oral Literacy competencies: Listening

Grade 1: High rating on Listening skills by country

40

50

60

70

80

90

100

Identification of Objects Identifying set of instructions Sound discrimanation Listening comprehension

Perc

ent

GRADE 1: LISTENING BY COUNTRY

Kenya Uganda


Oral Literacy competencies: Listening

Grade 2: High rating on Listening skills by country

GRADE 2: LISTENING SKILLS BY COUNTRY

70

75

80

85

90

95

100

Identification ofObjects

Identifying set ofinstructions

Sound discrimanation Listeningcomprehension

Perc

enta

ge

Kenya Uganda


Oral Literacy competencies: speaking

Grade 1: High rating in Kenya than Uganda

GRADE 1: SPEAKING SKILLS BY COUNTRY

20

30

40

50

60

70

80

Describing objects Telling direction Story composition

Perc

enta

ges

Kenya Uganda


Oral Literacy competencies: speakingGRADE 2: SPEAKING SKILLS BY COUNTRY

20

30

40

50

60

70

80

90

Describing objects Telling direction Story composition describing process

Perc

enta

ges

Kenya Uganda



GRADE 1: READING SKILLS BY COUNTRY

0

10

20

30

40

50

60

70

80

90

Letters: vowels Letters: Consonants Words

Perc

enta

ges

Kenya Uganda

Grade 1: High rating in Kenya than Uganda Reading words scored lowly

Oral Literacy competencies: Reading


GRADE 2: READING SKILLS BY COUNTRY

0

10

20

30

40

50

60

70

80

90

100

Letters: vowels Letters: Consonants Words Simple sentences

Perc

enta

ges

Kenya Uganda


Oral Literacy competencies: Reading


Class size and literacy score- Grade 130

4050

6070

80(m

ean)

sco

re_p

erce

nt

0 50 100 150classize

bandwidth = .8

Class size Vs pupil score: Kenya Grade 1

2030

4050

6070

(mea

n) s

core

_per

cent

50 100 150 200 250 300classize

bandwidth = .8

Class size Vs pupil score: Uganda Grade 1

Correlation Kenya: -0.0256 Correlation Uganda: 0.2369


Correlation Kenya: -0.0918 Correlation Uganda: -0.0477

Class size and literacy score- Grade 230

4050

6070

80(m

ean)

sco

re_p

erce

nt

0 50 100 150classize

bandwidth = .8

Class size Vs pupil score: Kenya Grade 2

2040

6080

(mea

n) s

core

_per

cent

50 100 150 200 250classize

bandwidth = .8

Class size Vs pupil score: Uganda Grade 2


Availability of reading materials at school: - Story books

Literacy (Oral) – Availability of story books Grade 1

Story book: Available in class Coef P CI Available not in class -1.34 0.443 [-4.77 ; 2.09] Not available -3.95 0.027 [-7.44 ; -0.46] Uganda -24.42 0 [-27.3 ; -21.54] Constant 31.77 0 [29.3 ; 34.25]

Grade 2 Coef. P CI Story book: Available in class Available not in class -0.54 0.408 [-1.81 ; 0.74] Not available -0.65 0.329 [-1.96 ; 0.66] Uganda -3.93 0 [-5 ; -2.86] Constant 11.39 0 [10.44 ; 12.34]


Household characteristics 43.96 of the household heads have primary education,

with 34.81 with no education

46% of the pupils are not give homework from school.

Among those given homework, 29.23% are not assisted in doing the homework- this varies by country (Kenya- 55.10% and Uganda – 20.79%)

69% whose heads have no education, do not help their pupils in doing homework compared with 19% whose household heads have tertiary education


Literacy 101: Regression coefficientsRegression on Literacy (ORAL) performance by some selected HH characteristics

KENYA (n=2164) UGANDA (n=871) Coefficient CI Coefficient CI Hmwk: Always Helped (Ref) Hmwk: Sometimes Helped -2.03 [-04.66 ; 00.59] 0.22 [-02.64 ; 03.08] Homework: Not helped -2.20* [-04.59 ; 00.20] 0.94 [-02.21 ; 04.08] No Homework -3.85** [-06.22 ; -01.49] -1.01 [-02.89 ; 00.87] Grade 1 (Ref) Grade 2 -16.88** [-18.24 ; -15.53] -0.95 [-02.31 ; 00.42] HH edu: No education (Ref) HH edu: Primary 0.63 [-00.92 ; 02.17] 0.01 [-01.84 ; 01.86] HH edu: Secondary 5.49** [02.86 ; 08.12] 2.48** [00.24 ; 04.73] HH edu: Tertiary 4.44* [-00.16 ; 09.04] 1.44 [-02.42 ; 05.31] HH edu: Not known 1.53 [-02.09 ; 05.15] -0.38 [-02.61 ; 01.86] HH: Father (Ref) HH: Mother 1.31 [-00.92 ; 03.55] -1.13 [-03.38 ; 01.13] HH: Male Guardian 1.03 [-01.49 ; 03.55] -2.31 [-05.27 ;0 0.65] HH: Female Guardian 0.34 [-03.38 ; 04.06] -0.33 [-04.63 ; 03.97] Pup sex: Female (Ref) Pup sex: Male 0.57 [-00.78 ; 01.92] 0.98 [-00.38 ; 02.35] Constant 27.76 [25.13 ; 30.39] 7.31 [04.86 ; 09.75]


Regression results [2]

Pupils who come home with homework and helped within the household score better- in Kenya

HH education: Pupils from households headed by heads with secondary and tertiary education score high- Kenya

Uganda only secondary education matters

content

Documents

policyrelevant research

wellbeing of africans

research questionspromoting

health research center

erp team african population

control schools

primary schools

rtl approach core model