Implementing Quantitative Service Delivery Surveys:

Some lessons from schools surveys

Deon FilmerDevelopment Research GroupThe World Bank

Are You Being Served?November 3 2009

Why carry out a Quantitative Service Delivery Survey?

Outcomes are low …

Starting points: Learning outcomes are low

Inequality in TIMSS 2003 Mathematics test scores

200

300

400

500

600

700

Uni

ted

Sta

tes

Kor

ea

Hun

gary

Lith

uani

a

Sou

thA

frica

Chi

le

Mal

aysi

a

Bot

swan

a

Rus

sian

Fed

erat

ion

Iran

Phi

lippi

nes

Jord

an

Egy

pt

Indo

nesi

a

Quintile 5 Quintile 1 AverageSource: Analysis of TIMSS 2003 database

Why carry out a Quantitative Service Delivery Survey?

Outcomes are low …

… is it a lack of money?

Public spending is not enough to improve outcomesPattern across countries

* Difference in logs (x100) form rate predicted by GDP per capitaSource: WDR 2004

Public spending is not enough to improve outcomesPattern across primary schools in Mauritania

Similar changes in public spending can be associated with vastly different changes in outcomes…

Source: WDR 2004

…and vastly different changes in spending can be associated with similar changes in outcomes.

How to assess the lack of association between spending and outcomes?

Public spending benefits the rich more than the poor– Expenditure incidence analysis of public spending for

diagnosis Lack of demand by households

– Impact evaluation of programs to promote demand Money fails to reach frontline service providers

– Public expenditure tracking surveys (PETS) Poor quality services

– Quantitative Service Delivery Survey (QSDS) e.g., absenteeism, time on task

Often, public spending benefits the rich more than the poor Expenditure incidence of public spending on education

South A

frica 2

000

Bosnia

and H

erz.

2001

Albania 200

2

Nepal 2

003-2

004

Kenya

1992/9

3

Azerb

aijan

2001

Cambo

dia 20

02

Egypt

2004

-2005

Indone

sia 199

8

Brazil

1997

Malawi 1

994/95

Mexico

1999

Guinea

1994

Benin 200

3

Uganda

1992/9

3

Leso

tho 20

02

Cote d'Ivoir

e 1995

Mozambiq

ue 2003

Tanz

ania

1993

/940

5

10

15

20

25

30

35

40

Poorest quintileRichest quintile

Expenditure incidence can be changed through reallocation of public spendingExpenditure incidence of public spending

1993 1995 2000 20050.0

5.0

10.0

15.0

20.0

25.0

30.0

Poorest quintile Quintile 2 Quintile 3Quintile 4 Richest quintile

South Africa*

1990-1991 1994-1995 1997-19980

5

10

15

20

25

30

35

40

Poorest quintile Quintile 2 Quintile 3Quintile 4 Richest quintile

* Primary and Secondary only

Malawi

Lack of demand by households0

.2.4

.6.8

1P

roba

bilit

y of

enr

ollm

ent

1 2 3 4 5 6 7 8 9 10Decile

Non-recipient Recipient

Impact of a scholarship program on girls’ enrollment in Cambodia:Enrollment probability among recipient and non-recipient girls by economic status decile

Source: Filmer and Schady (2008)

Impact of a conditional cash transfer on girl’s and boy’s middle school enrollment

Impact of demand-side programs

Disbursed public spending on school grants that actually reach schools

Percent

GNI per capita (2000)

GNI per capita PPP

(2000)Ghana 1997/98 51 330 1880Kenya 2004 (secondary school

bursary funds)78 250 810

Madagascar 2002 88 2050 4610Peru 2001 (utilities) 70 / 97 670 2280PNG (2001/2002) 72 / 93 280 510Tanzania 2002-2003 62 270 1250Uganda 1991-1995/2001 <20 /

80Zambia 2001 (discretion/rule) 24 / 90 320 740Ye and Canagarajah (2002) for Ghana; Republic of Kenya (2005) for Kenya; Francken (2003) for Madagascar; Instituto Apoyo and World Bank (2002) for Peru; World (Bank 2004) for

PNG; MOF, Government of Tanzania (2005) for Tanzania; Reinikka and Svensson (2005) for Uganda; Das et al. (2002) for Zambia.

Percent of school grants that actually reach schools

Classic approach to analyzing education outcomes…

Inputs School Outputs / Outcomes

Money

$$

Quality of public services

… QSDS are a way to get inside the “black box” of service delivery at the facility level

What are Quantitative Service Delivery Surveys?

Take the facility (or staff) as the unit of analysis– Could be complemented with a household/users survey

Collect quantitative information about– Physical infrastructure– Staff characteristics– Income and expenditures– Governance and management– Characteristics/Quality of service provision– Outcomes

Two different sets of surveys

Indonesia:– December 1998: Early days of economic crisis …

were schools feeling any impact– April/May 2000: Longer-run school-level impacts

of the crisis, decentralization looming PNG

– April/May 2002: little knowledge about the status of services in PNG; particular interest in decentralization; explicit concern about expenditure tracking

Some lessons from experience, with a focus on two different sets of surveys

Indonesia:– 600 schools– 5 purposively selected provinces– 15 districts (40 schools per district)

PNG– 220 schools– 8 purposively selected provinces– 2 districts (10 schools per district)

Activity structure Indonesia:

– Close collaboration with research department of ministry of education.– Ministry staff served as full partners in pilot/questionnaire development;

served as regional survey supervisors.– Gave the survey some legs within the ministry, enabled substantially lower

costs … but cost in terms of capacity and experience.– Study conceived of as stand alone survey, with Ministry/policymakers as

primary audience. PNG:

– Partners with National Research Institute, an independent agency– Overseen by “working group” with various government, NGO, and other

representatives.– Little hands on input from Ministry of Education.– Study conceived of as a part of WB Poverty Assessment.

What worked well Indonesia

– Trends in enrollments at the school level Non-conventional wisdom result that enrollment impacts were

mainly urban and at the secondary level; and in non-private/non-secular schools.

But difficulty: enrollment levels/trends … not enrollment rates.– Perceptions of impact of crisis

Identified “general impact” and “school functioning” as two main impacts (exploratory PC analysis)

– Status of crisis-relief government programs (scholarship and grant programs)

Schools grants: Coverage; use; interesting substitution between grants and other sources of government (especially local government) sources of funding

Scholarships: Coverage (among students)– Trends in charging of fees

Indonesia 2000: Sources of school funding by grant receipt and public/private status

0

1000

2000

3000

4000

5000

6000

7000

8000

Public-Received

Grant

Public-NoGrant

Private-Received

Grant

Private-NoGrant

Grant Local Central

Primary schools

0

20000

40000

60000

80000

100000

120000

140000

Public-Received

Grant

Public-NoGrant

Private-Received

Grant

Private-NoGrant

Grant Local Central

Junior Secondary schoolsIn public schools, local government spending adjusted in response to grant

No adjustment in private schools

Substitution between grants and local government funding

Zambia 2001: Effect of a 100 Kwacha increase in expected and unexpected school grants on household expenditures on education

-50

-45

-40

-35

-30

-25

-20

-15

-10

-5

0Expected Unexpected

Kw

acha

Household spending falls by about 45 for each additional 100 Kwacha spent on anticipated grants

Substitution between grants and household spending

Source: Jishnu Das, Stefan Dercon, James Habyarimana, Pramilla Krishnan (2004)

What worked well PNG

– Descriptive status of schools (very little prior information)

– Good documentation of delays in subsidies / teacher pay

– Reasonable assessment of teacher absenteeism (pre-announced window for visit)

– Good data to construct “ghost teacher” estimate (with substantial effort in matching to government payroll records)

Delay in ability to use subsidy: PNG 2001

0

20

40

60

80

100

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

Accessible Remote

Percent who received any subsidy

0

5

10

15

20

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

Accessible Remote

Weeks delay

Note: Q1,Q3=National, Q2,Q4=Provincial

Absence rates among teachers and health workers

Note: Surveys were all fielded in 2002 or 2003. Sources: Chaudhury et al (2006) except for PNG, World Bank (2004) and Zambia, Das et al (2005).

0

10

20

30

40

50

Bangladesh Ecuador India Indonesia Papua NewGuinea

Peru Zambia Uganda

Primary schools Primary health facilities

PNG 2002: Depletion of the effective supply of teachers

Source: PESD 2002.

8572 68

100

0

20

40

60

80

100

Teachers onpayroll

Minus "ghost"teachers

Minus absentteachers

Minus schoolsclosed "for

lack ofteachers"

Results from QSDS:Effective supply of teachers

Percent of time officially allocated to schooling; when a teacher is present; and spent in teaching and learning activities

Beyond absenteeism: Effective supply of teaching

Egypt Lebanon Yemen Lao PDR Tunisia Cambodia Morocco Pernambuco (Brazil)

Ghana0

20

40

60

80

100

Official time Presence timeSources: Egypt, Yemen and Lebanon from Lane and Millot (2002); Tunisia, Pernambuco, Morocco and Ghana from Abhadzi, Millot and Prouty (2006); Cambodia from Benveniste, Marshall and Araujo (2008); and Laos from Benveniste, Marshall and Santibanez (2007).

Once there, and teaching … competency?

14 + 139 + 9

252 ÷ 7

864 ÷ 24

1 1/2 – 3/4

1/2 x 1/3

1/10 ÷ 1/5

75% of 36

1/4 x 1/6 ÷ 1/8

1/4 x 5

1/4 + 1/2 + 1/8

0 10 20 30 40 50 60 70 80 90 100

83.42

77.82

63.98

36.80

50.74

54.86

48.09

37.00

29.44

45.04Selected math Questions (Full sample)

% of Teachers who Answered Correctly

Que

stion

s

The Gambia: Percent of 4th and 6th grade teachers answering student-level math questions correctly

Source: The Gambia Impact Evaluation Team (2009). N=1049 Teachers.

Once there, and teaching … competency?

The Gambia: Percent of 4th and 6th grade teachers answering student-level literacy questions correctly

Source: The Gambia Impact Evaluation Team (2009). N=1049 Teachers.

MYSTERIOUS: Pleasant/Stange/Quiet/ Frightening

STARTLED : Began/Scattered/Frightened/Deafened

EVEN: Sandy/ Level/ Rocky/Hard

ENORMOUS: Heavy/Hard/Huge/Rotten

The children worked in ___ silence during the test. (Complete,

Common, Company, Count )

0 10 20 30 40 50 60 70 80 90 100

48.38

40.73

69.58

54.07

85.28

Selected Literacy Questions (Full Sample)

% of Teachers who Answered Correctly

Que

stion

s

Investigating accountability in education service deliveryPNG 2002: Teacher absence declines with parent and community involvement

0

5

10

15

20

25

30

0 1 2 3 4 5

Index of parent and community involvement

Per

cent

abs

ent

Source: PNG PESD 2002.

What was harder

Indonesia:– Trends in overall school incomes—never clear we had full

picture (what we did have was worrisome, especially for private schools)

– But, incredibly complicated system … is this worth doing when the system is so complex?

PNG:– Complex funding system … but able to track some specific

payments (school subsidies)– But … school financial data very spotty

only about half of the schools had documentation about spending, half about receipts

Only 30% of schools had both expenditures and receipts documentation

Funding education in PNG2001, million Kina

Leave fare

Payroll Salary(284)

Recurrent(153)

Development(91)

Salary Subsidy Q1&Q3

(40)

in kind

Leave fare(1) Subsidy Q1&Q3

Subsidy Q2&Q4 Subsidy Q2&Q4 (21)

Grants Grants (15)

Donors Grants & donationsNGOs (66)

(68)

Provincial Government and Administration

Dept. of Finance

and Treasury

National Dept. of

Education

Non-teaching staff

Parents

Bank account (school/ BOM/ Joint)

Project fees

School fees

School

Bank account

Teachers

Contractors

Source: Based on information collected during the PESD 2002 survey.

Q1,3

Q2,4

What I would think twice about doing again

Enrollment trends (unless have information on universe of schools and on population trends by area)– Hard (time consuming) to collect, hard to interpret

Too many instruments– PNG had 9 instruments, 7 at the school level.

Non-representative/random sample of parents

Survey instruments in PNG:– School (head teacher); – teacher roster; – select teachers; – data appendix; – grade 5 teacher; – board of management member; – parent; – District Education Advisor; – Provincial Education advisor.

I would think (very) hard about what financial data to collect

The more specific the better– But even there, school officials often don’t associate specific

transfers to “official” name Anything more than tracking a clearly defined transfer

is very hard. Even that is hard:– missing information at schools; – missing records at provincial level; – defining the “base”

Official declarations in Government Circulars Budget disbursements School level expectations

What I would never do again

Data entry using a package not designed for that purpose

(Data entry using a package not designed for that purpose)

Complex survey/tracking exercise in a country where policy environment not conducive to use information