Report Cards: The Impact of Providing School and Child Test-scores on Educational Markets

Jishnu Das (World Bank)

With:Tahir Andrabi (Pomona College)Asim Ijaz Khwaja (HKS, Harvard)

The Context

Private Schools have expanded dramatically since the 1990s in South Asia

61.6

34.9

2.7

.9

6334

.91.

21

66.2

28.3

3.1

2.4

87.5

8.1

3.9

.5

63.5

332.

31.

3

0

20

40

60

80

Pe

rce

nt

En

rolm

en

t

Punjab Sindh NWFP Balochistan Total

Govt

Privat

e

Other

Madra

ssa

Govt

Privat

e

Madra

ssa

Other

Govt

Privat

e

Other

Madra

ssa

Govt

Privat

e

Madra

ssa

Other

Govt

Privat

e

Other

Madra

ssa

Calculations by Nobuo Yoshida and Tomoyuki Sho (The World Bank)

PSLM (2005-06)

Enrolment in Pakistani Educational Institutions

Pakistan Rural India

The Context: Its not what you think!

Although much discussion about madrassas, this is not where the action is!

The number of new Madrassas added every year tapered off after 2000

0

2000

4000

6000

8000

1960 1970 1980 1990 2000 2010Year

Public Private Madrassa

By Year of Formation

New Educational Institutions in Pakistan

The Context: The “New” Village Environment

In our sample of 112 villages, there were 812 schools 50% of rural Pakistan’s most populous province—Punjab—

live in villages like one of these two

A village in Central Punjab A village in North Punjab

The Questions

Can better information about school performance take advantage of the market structure to improve educational outcomes?

What is the equilibrium impact of information on educational markets? Quality Price Quantity

Is there a strong case for the provision of better information?

Address these issues using an experimental design in Punjab province, Pakistan

Precursors

Information can lead to a number of different types of results! Positive: Information leads to greater accountability/verifiability, competition

(Bjorkman & Svenson (2008) – community-based health reporting/monitoring in Uganda; Jin & Leslie (2003) – Restaurant Hygiene cards in LA Rokoff and Turner (2008); Chiang (2008) – public school accountability in US; Hastings

(2007)

Nothing: Information may be known, not understood/credible/believed; Banerjee et al. (2007) - no learning improvements from information dissemination (Indian

state)

Negative: Information may lead to greater cream-skimming/sorting (winner takes all) Education : Chile (Urquiola and Mizala 2007) Health: Dranove et al. (2003) – hospital outcomes in NY Direct Manipulation: US – cheating teachers (Jacob & Levitt, 2003)

Gaps All market-level studies are observational (Dranove and others, Urquiola and Mizala,

Jin and Leslie. Experimental work thus far either in cases where markets are sparse or market

reactions not examined

First experimental equilibrium results on impact of information in education

Remainder of talk

A note on private schools The data The experiment The Results A note on the results

A note on private Schools

All unaided, very sparsely regulated, co-educational, mostly small “mom & pop” operations

Better learning than public schools

Learning

450

500

550

600

3 4 5 3 4 5 3 4 5

English

Math

Urdu

Kno

wle

dge

Sco

res

Government School Children in Classes 3, 4 & 5

Learning in government and private schools

Priv

ate

Sch

ools

in C

lass

3

Priv

ate

Sch

ools

in C

lass

3

Priv

ate

Sch

ools

in C

lass

3

450

500

550

600

3 4 5 3 4 5 3 4 5

English

Math

Urdu

Kno

wle

dge

Sco

res

Government School Children in Classes 3, 4 & 5

Learning in government and private schools

Priv

ate

Sch

ools

in C

lass

3

Priv

ate

Sch

ools

in C

lass

3

Priv

ate

Sch

ools

in C

lass

3

Probably causal differences

A note on Private Schools (II)

And cheaper, too!

260

184

259

75 6882

0

50

100

150

200

250

Co

st f

or

Eve

ry P

erc

en

t C

orr

ect

(in

Rs.

)

Government Private

Cost of Schooling

English Math Urdu

Data (I)

Sample: 112 villages from 3 districts in the Punjab, Pakistan Randomly selected from list of villages with at least one private

school in 2000 (3rd of villages & 50% of pop); somewhat bigger/richer than average village

Defining Schooling Markets: Goal – capture parents/children complete choice set & schools’

potential market Feasible:

92 % of children attend village school (HH census) Large distance effect – most primary-school children go within 15

minutes Create 15 minute (30m for RYK) boundaries around village HHs

(include some schools right outside village boundary) (Figure 3)

LEAPs Project - www.leapsproject.org : Baseline HH census (80,000) Four Rounds (2004-2007):

School (823) Questionnaires: General School Questionnaire Class Teacher Questionnaire Head Teacher Questionnaire

Educational Performance: Child-Tests (Follow 12,000 plus children over 4

years) in English, Urdu and Mathematics

Household-Level Information Detailed Household Interviews for randomly

selected HHs (1,800) Short school-based Child questionnaire (randomly

select 10 in each school)

Child

Administration Teacher

School

Family

Attainment

Data (II)

Data (III)

Survey Instruments & Timeline: HH census (80,000 hhs) – 2003 Round 1 (Baseline):

School-Based (Jan-Feb 04): (i) 823 primary schools + class 3 teachers; (ii) 800+ Class 3 teachers; (iii) 6,000 class 3 kids (brief info)

HH-Based (March-Apr 04): Detailed HH surveys (1,800); part matched on class 3 children

Child-Tests (Jan-Feb 04): 12,000+ Class 3 children – Norm-referenced test to maximize variation – Use Item Response Theory to get at underlying child knowledge; we administer (minimize cheating etc.)

Report Card Intervention – Sept/Oct Round 2 (2005): Report all of Round 1 Surveys/Tests

(96% children tracked)

The Experiment After baseline, villages within each of 3 districts divided with equal probability into

treatment and control

Report Card provided to each Class 3 kid parent in school-meeting – explain scores

Parent Card 1: Child Info

In all 3 subjects (Maths, Urdu & English): Child score and quintile Child’s School score &

quintile Child’s village score and

quintile

Quintile described as “needing a lot of work” to “very good”

The Report Card Intervention

Parent Card 2: Village Schools Info

For all Primary schools in villages give : School Name Tested Children School scores and

quintiles in all 3 subjects

“Bundled-Impact”: Information (child,

schools) Increase precision,

verifiability Meeting effect?

A note on the information

This is not value-added information Why?

Feasible intervention Theoretical considerations (who can back out

VA better?) Empirically doesn’t look too bad

Nevertheless, combination of selection and measurement error may lead to erroneous inference by parents

A further note on the information Reliability vs. Measurement

Error (Kane & Staiger) Information is fairly reliable

Low measurement error of test

Large variation across schools - see Figure

Selection (into schools) Value-added estimates? Selection Not as severe (see

learning gaps Figure) Need to have Information be

clear and understandable Policy feasible/relevant

Households may be better able to back out value-added

-4-2

02

sch

scor

e

0 5 10 15School ID

village w/ 15 schools; test-score & (2) standard-error bands (computed using IRT)

Family Rich-Poor

Father Literate-Illiterate

Mother Literate-Illiterate

Child Female-Male

Private-Public

0 50 100 150 0 50 100 150 0 50 100 150

English Math Urdu

Size of Adjusted Gap

Learning Gaps

What should we expect? 3 Broad Classes of models Symmetric information

Some unobservable components of quality for both schools and households

Asymmetric information: Price signals quality Asymmetric information: Price does not signal

quality

In Model 1 price declines for all schools; depending on structure of demand can get heterogeneous declines by initial quality; quality weakly improves

In Model 2 price declines more for initially higher performing schools; quality weakly improves

In Model 3 price/quality movements are ambiguous

Results: Quality

Village-level Average

Child-Level Average

Child-Level Average (No switchers)

Report Card Villages

0.114

(0.045)

0.095

(0.038)

0.102

(0.038)

Good Private School

0.039

(0.516)

Bad Private School

0.347

(0.019)

Government Schools

0.088

(0.053)

Notes

Learning:

Similar across subjects; holds 2 yrs after

Attrition:

Unlikely concern: no difference in baseline scores for attritors between treatment and control samples

Switching/Dropouts:

Results entirely driven by children who stayed in same school:

Few Switch Schools (5%); Gains similar if restrict to non-switching children

For gains to be attributable to switchers, need switchers to have gained 1.7sd, given numbers---highly unlikely!

Results: Quantity

Notes

Enrolment

Large increase in RC villages (almost 5 percent)

Entirely from Government schools, entry into Grade i

Switching

No evidence of increased churning

But evidence of differential churning

School Closures

Significant among initially low performing private schools

Enrolment Change

Probability that child switches

Probability that child drops-out

School Closure

Report Card Villages

33.78

(13.75)**

0.012

(0.009)

0.006

(0.004)

--

Good Private School

-2.079

(0.592)

0.039

(0.516)

0.037

(0.200)

Bad Private School

-4.738

(0.248)

0.347

(0.019)

0.117

(0.030)

Government Schools

6.536

(0.006)

0.088

(0.053)

NA

Results: Price (Private Schools Only)

Notes

Fees

Large Declines—24 percent across the board

Larger in initially higher performing private schools

These are reported by the school; we obtain identical results using reports from households

In Rs. ($1=Rs.60 in 2003)

In Log fees

Report Card Villages

-217.96

(65.090)***

-0.24

(0.087)***

Good Private School

-241.841

(0.000)

-0.257

(0.001)

Bad Private School

-139.171

(0.237)

-0.126

(0.286)

Results: Schools or Households?

Notes

Household

Little evidence of any big changes (consistent with Das and others 2009)

Children in bad private schools are now playing less Sleeping more

Spending more time in school

Schools

Private schools increase teacher eduation, textbooks

More time on task—fewer breaks

(5) (6) (7) (8)

Parental Time Spent on Education

with Kids

Kids' Time Spent on School Work Outside

of School

Parental Spending on Education Not

Including School FeesChild Time

Spent PlayingSUBGROUP POINT ESTIMATES, F-TEST p-VALUES IN PARENTHESESBad private school -1.490 6.530 -375.684 -85.07***

(0.234) (0.788) (0.174) (0.010)Good private school -0.890 27.012* -32.930 -11.430

(0.207) (0.052) (0.844) (0.519)Government school -0.149 1.563 -172.899* -11.520

(0.625) (0.873) (0.055) (0.108)

Table 8: School and Household Input Changes

Household Inputs (Household Level)

(1) (2) (3) (4)

Class Teachers Improve to Matric

Percent Change in Matric Teachers

Textbook Probit Break Time

SUBGROUP POINT ESTIMATES, F-TEST p-VALUES IN PARENTHESESBad private school 0.165 0.035 0.405 -22.789

(0.29) (0.56) (0.09) (0.01)Good private school 0.183 0.037 -0.009 1.609

(0.06) (0.08) (0.95) (0.65)Government school 0.064 0.025 0.046 1.57

(0.39) (0.87) (0.77) (0.47)

Table 8: School and Household Input Changes

School Inputs (School Level)

Conclusion

RC increase learning and/or fees drop – equity and efficiency both increase? Results depend on pre-existing market conditions (parental demand; eductaional production

function - school type vs. effort)

Cost of Intervention ~ fee drop RC exercise cost $1 per child (testing, grading & dissemination) Cost savings ~ $3/child in private schools (1/3rd of all children enrolled in private

schools)

Welfare calculations? Tricky: Typical Cost-Benefit calculations in LIC ignore welfare costs for providers

→ learning gains free of cost BUT: complete welfare analysis has to factor in provider welfare loss – transfer

from school to parents & decline in rents – need more structural approach

Policy Questions & Caveats: Public vs (socially cheaper) Private sector Intervention simultaneously improve private sector (equity, efficiency) and public

sector – State’s role as information provider (rather direct regulator)

Further Notes or how I began to worry that this may actually lead to policy

Theory: Outlined 3 classes of theory; there are others Question: Why changes in provider behavior, but no increased

churning? Alternative Question: (Hastings): Why switching but no change

in provider behavior? Answer: We don’t know the dynamic equilibrium process in

control villages (ratcheting?) Empirics: Is this a big effect?

CANNOT compare SD increases across tests Can simulate changes from 0.05sd to 1.3sd by changing the test! Have to link to some cardinal change (see Heckman)

Trying to calibrate to TIMMS using identical questions Can answer: how big is this change relative to the world distribution

Longer-term effects (up to 5 years later) How do we treat utility of providers in welfare computations?

Feasibility This is proof of concept; mainstreaming is a different issue