evaluating a bilingual education program in spain: …€¦ · evaluating a bilingual education...

EVALUATING A BILINGUAL EDUCATION PROGRAM IN SPAIN: THEIMPACT BEYOND FOREIGN LANGUAGE LEARNING

BRINDUSA ANGHEL, ANTONIO CABRALES and JESUS M. CARRO∗

Bilingual education programs, which consist of doing a substantial part of theinstruction in a language different from the native language of the students, exist inseveral countries like the United States, India, and Spain. While the economic benefits ofknowing a second language are well established, the potential effects over the learning ofother subjects have received much less attention. We evaluate a program that introducedbilingual education (in English and Spanish) in primary education in a group of publicschools of the Madrid region in 2004. Under this program, students not only studyEnglish as a foreign language but also some other subjects (at least Science, History,and Geography) are taught in English. In order to evaluate the program, a standardizedtest for all sixth grade students in Madrid on the skills considered “indispensable” atthat age is our measure of the outcome of primary education. Our results indicate thatthere is a clearly negative effect on the exam results for the subject taught in English,for children whose parents have less than upper secondary education. This negativeeffect is a composite of two phenomena: the effect of the program on the student’sknowledge of the subject and a reflection of the student ability to do the test in theirnative language when English is the medium of instruction. Although we are not ableto separate quantitatively these two effects, the composite effect has a relevant interest,because the results for exams taken in Spanish are the measures that determine academicprogression in the Spanish system. In contrast with the previous result, there is nosignificant effect for anyone on mathematical and reading skills, which were taught inSpanish. (JEL H40, I21, I28)

I. INTRODUCTION

Knowledge of a second language is widelybelieved to be essential for workers to succeed inan increasingly interconnected business world,and researchers tend to agree. Ginsburgh and

∗We gratefully acknowledge the help of José CarlosGibaja and Ismael Sanz to obtain the data, the comments fromGema Zamarro and seminar participants at various confer-ences, and the support from the Spanish Ministry of Scienceand Technology from grants ECO2009-07530 and ECO2012-31985 (B.A.), ECO2012-34581 (A.C.), ECO2009-11165and ECO2012-31358 (J.M.C.), CONSOLIDER-INGENIO2010—CSD2006-0016 (all), and Consejera de Educación dela Comunidad de Madrid under the Excelecon project.Anghel: Assistant Professor, Dpto. Análisis Económico, Uni-

versidad Autónoma de Madrid and FEDEA, Madrid,28049, Spain. Phone +34 91497 7040, Fax +34 915779575, E-mail [email protected]

Cabrales: Professor, Department of Economics, UniversityCollege London, London, WC1H 0AX, UK. Phone+44 203 108 5229, Fax +44 207 916 2775, [email protected]

Carro: Associate Professor, Department of Economics, Uni-versidad Carlos III de Madrid, Getafe, 28903, Spain.Phone +34 91624 9586, Fax +34 91624 9875, [email protected]

Prieto-Rodríguez (2011), for example, foundlarge estimates of the effects of foreign languageknowledge on wages in Mincerian regressions:the increases in wages ranged between 11% inAustria and 39% in Spain for knowledge of theEnglish language and even higher effects forknowledge of other languages.1,2 The returns tolearning English do not only flow to individuals,

1. An earlier analysis of the same data by Williams (2011)found a smaller impact: between 5% in Austria and Finland,to insignificant in Spain or France. But the reanalysis ofGinsburgh and Prieto-Rodríguez (2011) used more powerfultechniques to control for endogeneity.

2. The effects on U.S. workers are rather smaller, asone would expect from the lingua franca status of English.See for example Fry and Lowell (2003) who find no effect

ABBREVIATIONS

CDI: prueba de Conocimientos y DestrezasIndispensablesdiff-in-diff: Difference-In-DifferencesIV: Instrumental VariableOLS: Ordinary Least Squares

1202

Economic Inquiry(ISSN 0095-2583)Vol. 54, No. 2, April 2016, 1202–1223

doi:10.1111/ecin.12305Online Early publication November 27, 2015© 2015 Western Economic Association International

ANGHEL, CABRALES & CARRO: EVALUATING A BILINGUAL EDUCATION PROGRAM 1203

the country as a whole may also benefit: Fidr-muc and Fidrmuc (2009) show, for example,that widespread knowledge of languages is animportant determinant for foreign trade, withEnglish playing an especially important role.

The private initiative has taken notice of thesebenefits of second-language acquisition. Manyschools in Spanish-speaking countries, especiallythose that cater to the elites, offer bilingual edu-cation for their pupils; Banfi and Day (2004) doc-ument this for Argentina and Ordóñez (2004)for Colombia. The high returns for foreign lan-guage capabilities, and probably also the associ-ation with elite schools, have prompted severalSpanish administrations to offer bilingual educa-tion in schools across the country. The ministry ofeducation sponsors an agreement with the BritishCouncil that selects 80 schools all over Spainwhere instruction in English occupies a large per-centage of the curriculum. Much more ambitiousin scale is a program in the autonomous region ofMadrid which in the academic year 2013–2014has 406 public schools (316 primary schools and90 high schools, around 40% of the total) wherearound 40% of the instruction, including all thescience curriculum, is taught in English.3 Theseprograms have been so successful with voters thatboth major parties included in their 2011 generalelection platforms the promise of extending theprogram to the whole nation.4

This expansion of bilingual programs whereat least part of the instruction is in a foreignlanguage (i.e., different from the mother tongueof students) is certainly not a Spanish phe-nomenon. Other important examples are theEnglish schools in India (Munshi and Rosen-zweig 2006) and the one-way foreign languageimmersion programs for native English speak-ers in the United States (Center for AppliedLinguistics 2011).

on wages, or Saiz and Zoido (2005) who find an effect ofabout 5%.

3. Andalusia also has a bilingual program, but the per-centage of instruction in English is smaller, around 20% ofthe instruction time.

4. See, for example, in the program of the socialistparty PSOE, the statement “we will support the design oflinguistic projects to support the learning of English. Wewill also support the schools offering bilingual educationboth in vocational training and at the university” (http://www.psoe.es/saladeprensa/docs/608866/page/programa-electoral-para-las-elecciones-generales-2011.html) or theone of conservative party PP, which states “we will promoteSpanish–English bilingualism in the whole educationalsystem from pre-school to university” (http://www.pp.es/actualidad-noticia/programa-electoral-pp&uscore;5741.html).

It is thus clear, both to researchers and thegeneral public, that learning a foreign languageis important for economic reasons. But it alsohas some costs. The more obvious are the finan-cial ones: the teachers may need to be hired,trained, or retrained, and given the market valueof English knowledge they will be more costlythan other teachers; some extra conversationassistants may need to be hired; if successful,demand will grow and the program may need tobe expanded. But in addition to these costs timeis finite, and there is hardly ever a free lunch ineducational issues; so there may be other neg-ative effects from the policy that have receivedmuch less attention. The aim of this study isprecisely to test whether bilingual educationalprograms have a cost in terms of slower learningrates in other subjects.

To test this idea, we look at data from the bilin-gual education program in the region of Madrid.Although we will describe it in more detail later,the program (for primary schools) basically con-sists of using English to teach the subject called“Knowledge of the Environment,” that includesall teaching of Science, History, and Geography.English is also used as the educational mediumfor Art and sometimes Physical Education, andof course the English language classes. Overall,teaching in English comprises between 10 and 12of the 25 weekly hours of instruction.

To find out the effects of the program, we usea standardized exam that has been administeredeach year in all primary schools from the Spanishregion of Madrid to sixth grade students (12–13years of age), starting with the school year2004/2005. The exam tests for what are called“Indispensable Knowledge and Skills” (CDI inits Spanish acronym) in three areas: Spanish lan-guage, Mathematics, and General Knowledge;the latter basically corresponds to the materialtaught in “Knowledge of the Environment.” Theexam results are anonymous, but each studentanswers a questionnaire that includes a host ofsocioeconomic background variables, which wecan use as covariates. We use data from the firstgroup of schools that became bilingual in theregion of Madrid in 2004/2005, and we checkedthe results of the first- and second-treated studentcohorts, which took the exam in 2009/2010 andin 2010/2011, respectively. We then repeat theanalysis with the second group of schools thatbecame bilingual for their first-bilingual cohort,whose students took the exam in 2010/2011.

We have to face a double self-selection prob-lem. One is caused by schools who decide to

http://www.psoe.es/saladeprensa/docs/608866/page/programa-electoral-para-las-elecciones-generales-2011.htmlhttp://www.psoe.es/saladeprensa/docs/608866/page/programa-electoral-para-las-elecciones-generales-2011.htmlhttp://www.psoe.es/saladeprensa/docs/608866/page/programa-electoral-para-las-elecciones-generales-2011.htmlhttp://www.pp.es/actualidad-noticia/programa-electoral-pp&uscore;5741.htmlhttp://www.pp.es/actualidad-noticia/programa-electoral-pp&uscore;5741.htmlhttp://www.pp.es/actualidad-noticia/programa-electoral-pp&uscore;5741.html

1204 ECONOMIC INQUIRY

apply for the program, and a second one causedby students when choosing school. We takeseveral routes to control for these selection prob-lems. The main route to control for self-selectedschools is to take advantage of the test beingconducted in the same schools before and afterthe program was implemented in sixth grade. Tocontrol for student self-selection, we combinethe use of several observable characteristics (likeparents’ education and occupation) with thefact that most students were already enrolledat the different schools before the programwas announced. That is, in order to control forendogeneity problems, we use a difference-in-difference (diff-in-diff) approach with controls,comparing the exam results of children in thetreated schools before and after they becamebilingual with the group of nonbilingual schoolsbefore and after the treatment. Other ways ofcontrolling for endogeneity, like using as instru-ment being enrolled at treated schools beforethe announcement of the program which is aproxy for exogenous “assignment to treatment”of students, confirm the diff-in-diff estimates.

For the first-treated cohort, we find that theeffect of the program is not significantly differentfrom zero for either Mathematics or Spanish lan-guage, although it goes from positive to negative.For General Knowledge, the bilingual programhas a negative and significant effect on the examresults, for children of parents without a collegeeducation. The size of this effect is substantial, onthe order of 0.2 standard deviations.5 As GeneralKnowledge is the only subject taught in Englishfrom the three present in the exam, it seems clearthat the extra effort made to use English as themedium of instruction comes at the expense ofa worsening in the results of standard exami-nations of that subject in Spanish. In a sense,there is a confound because it is possible that thestudents do not know less, but simply they do notknow how to express it in Spanish.6 But, even ifthat is the case, this would also suggest that the

5. This is close in magnitude to the effects found byAngrist and Lavy (1999) in Israel for a class reduction of eightstudents, and by Krueger (1999) for the Tennessee STARexperiment, which reduced class size by seven students.

6. Hofstetter (2003) uses data of middle-school studentsin a low-income, predominately Latino area of southern Cal-ifornia to study the effect for English language learners ofdifferent linguistic accommodations of standardized tests inMathematics. What is relevant for our study is that part ofthe sample considered by Hofstetter (2003) was instructedin English. Of those students, some were tested in the lan-guage of instruction (English), while others were tested intheir native language (Spanish). Hofstetter (2003) finds a neg-ative effect of this particular mismatch between the language

level of linguistic competence in English is notenough to leap through that barrier. And, possi-bly more importantly, other standardized exami-nations which, unlike the CDI, do have academicconsequences (at the end of secondary and at theentry to university) are in Spanish, so a negativeresult in CDI is still an outcome of interest.

In the group of schools that started to partici-pate in 2004, the results for the second cohort ofstudents exposed to the program are very similar,even quantitatively, to those of the first cohort.However, for the group of schools that startedto participate in 2005, the effects are also nega-tive and significant only for General Knowledge,but they are smaller in size and only for childrenof parents with less than upper secondary educa-tion. We conjecture that this is because of a betterselection of those schools in terms of the Englishknowledge of the teachers, because for that groupof schools the conditions to be a part of the pro-gram were made stricter in that dimension.

There is a large body of research aimed atunderstanding the effects of bilingual educationprograms for immigrants in the United States.This literature finds mostly positive results ofthose programs. Willig (1985) concludes that thebetter the experimental design of the study, themore positive results were the effects of bilin-gual education, and Greene (1998) in anothermeta-study of the literature asserts that “an unbi-ased reading of the scholarly research suggeststhat bilingual education helps children who arelearning English.” Jepsen (2009), on the otherhand, finds that “students in bilingual educationhave substantially lower English proficiencythan other English Learners in first and secondgrades. In contrast, there is little differencebetween bilingual education and other programsfor students in grades three through five.” Butthose are typically programs for immigrants intoa foreign country, so the external validity to ourpopulation of those results is rather unclear.

There is much less evidence regarding theeffects of the foreign language programs aimedto immerse native English speakers in a foreignlanguage in the United States, or regarding bilin-gual education in English for countries whoseofficial language is not English. An exception

of instruction and the language of the test. Unfortunately, theinstitutional and the socioeconomic framework for these dataare very different from the ones in our study, so we think thatit would not be prudent to draw quantitative conclusions fromHofstetter (2003) about which part of the effect we find ingeneral knowledge is due to the mismatch between instructionand testing language.


is Admiraal, Westhoff, and de Bot (2006), whostudy the effect of the use of English as the lan-guage of instruction for secondary education inThe Netherlands. They state that “No effects havebeen found for receptive word knowledge and nonegative effects have been found with respect tothe results of their school leaving exams at theend of secondary education for Dutch and sub-ject matters taught through English.” It is hard toknow what to make of the differences betweenour two studies, because the educational sys-tems are very different, as are the societies wherethe programs are administered. But an intriguingquestion arises: could the costs of bilingual edu-cation be lowered if the program was started inhigh school? This is an important question forfurther research.

This study is organized as follows. Section IIdescribes the institutional setup and the programin some detail. Section III discusses the data andthe econometric model. Section IV contains themain results of this study and it has some addi-tional estimations and robustness checks. SectionV contains conclusions.

II. INSTITUTIONAL BACKGROUNDAND DESCRIPTION OF THE PROGRAM

The order from the regional ministry ofMadrid that initiated the bilingual school pro-gram argues that it is needed because “The fullintegration of Spain in the European contextimplies that students need to acquire more andbetter communication skills in different Euro-pean languages. Being able to develop theirdaily and professional activities using Englishas a second language opens new perspectivesand new relationship possibilities to studentsof bilingual schools in the Autonomous Regionof Madrid.” The integrated European labor andtrading market is thus the reason used by theadministration for fostering the program.

This is a good reasoning: in the currentrecession with a general unemployment rateabove 26% and a youth unemployment rate of57%, only 39,690 Spaniards emigrated in thefirst semester of 2013. This contrasts markedlywith the over 6 million unemployed, or withthe 40,000 yearly emigrants that Bergin et al.(2009) estimate for Ireland, a country ten timessmaller than Spain and with half its unemploy-ment rate. Of course, there are many reasons forthis, Bentolila and Ichino (2008) argue that thewelfare state and the family make it possible toaccommodate big unemployment shocks, but the

welfare state and the family are similar in Spainand Ireland, so it is indeed quite likely that thelack of proficiency of adult Spanish cohorts inEnglish is one problem hindering the emigrationthat the unemployment figures would suggestshould be a safety valve for the situation.

The Spanish educational system is composedof 6 years of primary school, 4 years of compul-sory secondary education (educacion secundariaobligatoria) and 2 years of noncompulsory edu-cation, which is divided into vocational training(ciclos formativos) and preparation for college(bachillerato). There are also 3 years of free pub-licly funded preschool, from ages 3 to 5. Morethan 96% of the students in the Madrid Regionattended preschool. The preschool children sharethe premises with those in primary school. Also,the preschoolers in one location have precedenceover other children applying to the same primaryschool. As a consequence of this precedencerule, most students at the primary level comefrom the preschool in the same location. In fact,if all the vacancies for 3 years old are filled andnone of them leaves the school at the primarylevel, there will not be any vacancies at that levelin that cohort. As a result, the school choice isalmost universally made when the student is 3years old. After that time, school changes are notfrequent, because it becomes extremely difficultto enter schools with high demand.

The facts mentioned about school choice andselection in the previous paragraph are importantfor our study. The bilingual program is applied atthe primary school level, not at preschool. Sinceat the time the bilingual program was designedand announced, there were students already in thepreschool level at the selected schools, their par-ents’ school choices were made 3 years prior tothat moment, when the program did not exist andwas not even planned. For this reason, the differ-ences between the first cohort of treated studentsand the previous cohorts cannot be related to theintroduction of the program.

The program started with children in the firstgrade of the selected primary schools in theschool year 2004/2005 and left others in thesame school, and all in the remaining schools,untreated. The program progressed with theirschool training for those treated students. Thestudents from the treated schools from cohortsstarting before the first year of the treatment donot participate in the program at all, and theyremain with the same course of study as studentsin untreated schools. Successive cohorts fromthe treated schools have also been treated, and


additional primary schools joined the programin successive years, always starting the treatmentwith first graders. Our data cover only the schoolsfrom the first cohort. Once the students from the2004/2005 cohort reached secondary education(in 2010/2011), a second phase kicked in, andsome high schools joined the program. As thatphase of the program is still in progress, we willnot be able to analyze it.

The program was initiated in 2004 with a callfor applications by schools, of which 25 wereselected in the first year,7 with initial plans forextension up to 110, which were later expandedto the present 316 schools because of the highdemand (out of a total of about 780 publicschools). A school wishing to be selected for theprogram had to submit an application. The threecriteria used to evaluate those applications are:

1. Degree of acceptance of the educa-tional community expressed through the supportreceived by the application by the school teachersand the School Board (a decision making bodycomposed of the principal and elected teachersand parents).

2. Feasibility of the application. This willtake into account the previous experience of theschool (some schools had started small pilotprograms on their own), teaching staff, particu-larly the teachers with an English specialization,the school resources, and the number of classesand students.

3. Balanced distribution of selected schoolsbetween the different geographical areas, takinginto account the school population between 3and 16.

The selected schools were not the 25 thatbest meet the first two criteria because of thecriterion for geographical equity. However, theselected schools had all close to top grades inthose criteria.

For the schools that were selected into theprogram in the following years, from 2005onwards, the criteria used in the evaluationchanged in one significant way. The former rule3 was replaced by:

3′. English level of the teachers in the school.This level is verified either with some officialcertificate (such as those awarded by the Univer-sity of Cambridge) that accredits a sufficient levelof command of the English language or by an

7. In fact, there were 26 schools that became bilingualin 2004/2005, out of which we have enough information on25 schools.

evaluation done directly by the education depart-ment of the regional government.

The balanced distribution is still mentioned asa desirable property of the allocation, but it is notgiven explicit points.

The order calls bilingual a school where thelanguage of instruction is English during at leastone third of the school time, and where Englishlanguage classes take five weekly periods (of45–60 minutes). It explicitly excludes the Span-ish language and Mathematics classes from beingtaught in English.

In Table 1, we describe the weekly curricu-lum from first to sixth grade in both bilingual andnonbilingual schools8 so that the margin of auton-omy in the number of teaching hours in bilin-gual schools becomes clear.

With Knowledge of the Environment (a sub-ject encompassing Science, Geography, and His-tory) plus five periods of English, the minimumis accomplished (remember that a school is con-sidered bilingual if instruction is carried out inEnglish at least one third of the school time).Different schools choose whether to increase theEnglish instruction by also teaching in that lan-guage Art, Physical Education, and Religion (orits alternative for those not wanting Religion,which is mostly a class in social norms and cul-ture). Expansion of English instruction from theminimum depends on the availability of teachers,but most schools end up having above 40% of theinstruction in English.

The program is certainly not costless. Theteachers involved in it receive a complement overtheir basic wage based on the “extra dedica-tion that results in a longer workday, because ofthe higher demands imposed by the activities ofclass preparation, processing, and adaptation ofmaterials into other languages, and regular atten-dance at coordination meetings outside schoolhours.” The extra work is estimated by the orderto be “on average of 3 hours per week for teach-ers, and 4 hours for coordinators.” The order doesnot say how the administration arrived at thisestimate. To compensate for the extra dedica-tion, the coordinators of the program in eachschool receive 1,980 euros a year; a teacherwho teaches more than 15 hours in English, forsubjects different than English language, 1,500euros; between 8 and 15 hours, 1,125 euros;and less than 8 hours, 750 euros. The program

8. The schedule in Table 1 applies to all schools in eachcategory (bilingual and nonbilingual). In Spain, schools haveno freedom in the number of hours the teachers dedicate toeach of the subjects.


TABLE 1Weekly Schedule by Area in Primary School, NonBilingual and Bilingual Schools

Number of Weekly Hours Number of Weekly Hours

NonBilingual Schools Bilingual Schools

First Cycle Second Cycle Third Cycle First Cycle Second Cycle Third Cycle

AreasFirst and

Second GradesThird and

Fourth GradesFifth and

Sixth GradesFirst and

Second GradesThird and

Fourth GradesFifth and

Sixth Grades

Knowledge of theenvironment

4 4 4 2.5 2.5 2.5

Art 3 2.5 2.5 1.5 1.5 1.5Physical education 3 3 2.5 2 1.5 1.5Spanish language 5 5 5 5 5 5Foreign language 2 2.5 3 5 5 5Mathematics 4 4 4 4 4 4Culture, religion 1.5 1.5 1.5 1.5 1.5 1.5Recess 2.5 2.5 2.5 2.5 2.5 2.5Extra hours 1 1.5 1.5Total 25 25 25 25 25 25

Note: Extra hours in bilingual schools can be assigned to any English-taught subject, usually Knowledge of the Environment.

provides “conversation assistants” to schools,typically college students from English-speakingcountries. Finally, the program provides trainingcourses in English for teachers, both in Spainand abroad. In the latter case, the program cov-ers transportation, living expenses, and fees forEnglish schools, mostly in the United Kingdomand Ireland.

In order to teach in English, the teachers haveto be either specialists in English or pass an exam.The exam is divided into two parts. The firstpart is a written exam, where they are tested onreading, writing, and listening comprehension,plus vocabulary and grammar. The second partis oral and it involves a 20-minute conversationwith the examiner.

III. DESCRIPTION OF DATAAND ECONOMETRIC MODEL

A. Description of Data

Our data come from a standardized exam thathas been administered each year in all primaryschools from the Spanish region of Madridto sixth grade students (12–13 years of age),starting with the school year 2004/2005.9 Theexam is called CDI (prueba de Conocimien-tos y Destrezas Indispensables), which means“Indispensable Knowledge and Skills Exam.”It is compulsory for all schools (public, pri-vate, or charter). Like the Organization for

9. Since the school year 2009/2010, the exam is alsoadministered to all students in the third grade of compulsorysecondary education (14–15 years old).

Economic Co-operation and Development’sPISA exam, the CDI exam does not have anyacademic consequences for the students, it isonly intended to give additional information toteachers, parents, and students.

The exam consists of two parts of 45 min-utes each: the first part includes tests of dictation,reading, language, and General Knowledge andthe second part is composed of mathematicsexercises. We use the exam scores as a mea-sure of student achievement, standardized to theyearly mean, in General Knowledge (whose con-tents are close to the subject “Knowledge of theEnvironment” which is taught in English) andin reading and mathematics (which are taught inSpanish). The exams are conducted in Spanishfor all students, whether or not they were in abilingual school.

Before taking the exam, a short question-naire is filled out by each student.10 In thequestionnaire, the students are asked a fewquestions about themselves, their parents, andthe environment in which they are living. Theanswers to these questions provide rich infor-mation on individual characteristics of students:from the questionnaire we obtain the age of thestudent; the country of birth, which we divideinto Spain, China, Latin America, Morocco,

10. The exam results are anonymized and only the stu-dents and their families know individual results. For this rea-son, we cannot use any measure of lagged achievement inthe analysis. Hence, for identification we need to rely on ourdiff-in-diff strategy and on the individual sociodemographiccontrols from the student questionnaire. If this identificationstrategy is valid, then the availability of information on laggedachievement would only serve to gain more precision.


Romania, and other, to have a sufficient numberof observations in each category; the level ofeducation of the parents; the occupation of theparents; the composition of the household inwhich the students lives; and the age at whichthe student started to go to school or preschool.From the exam, we have information at studentlevel on gender, whether the student has anyspecial educational needs and any disability.

Regarding the education of the parents, stu-dents were asked to provide this informationfor both the mother and the father. In order tofacilitate the interpretation, we choose the high-est level of education between the mother andthe father. We distinguish the following cate-gories: university education, higher secondaryeducation, vocational training, lower secondaryeducation, and no compulsory education. Thesame applies to the occupation of the parents:because we have the occupation of both themother and the father, we choose the highest levelbetween them. Thus, we differentiate betweenthe following categories: professional occupa-tions (e.g., teacher, researcher, doctor, engineer,lawyer, psychologist, artist, etc.); business andadministrative occupations (e.g., chief executiveofficer, civil servant, etc.); and blue-collar occu-pations (e.g., shop assistant, fireman, construc-tion worker, cleaning staff, etc.).11

The variable on the composition of the house-hold of the student comes from the answers tothe question: “With whom do you usually live?”We differentiate the following seven categories:lives only with the mother, lives with the motherand one sibling, lives with the mother and morethan one sibling, lives with the mother and thefather, lives with the mother and the father andone sibling, lives with the mother and the fatherand more than one sibling, and other situations.

First Group of Schools Implementingthe Program. The dataset with more informationavailable for our empirical analysis comes fromthe first cohort of treated students in bilingualschools in the region of Madrid. They startedfirst grade of primary school in 2004/2005, andtook the CDI exam in 2009/2010. This first-treated cohort is from the 25 schools that firstlyimplemented the bilingual program.12

11. Robustness checks using separately the education ofeach parent yield very similar results.

12. The schools first selected to implement the programin 2004/2005 were actually 26, but due to unknown reasonswe do not observe one of those schools in the year before thefirst-treated students finish. Therefore, we have to restrict ouranalysis to the 25 schools for which we have information.

In order to control for the endogeneity prob-lems caused by self-selection of students andschools which we will explain below, we usea diff-in-diff approach. We compare the perfor-mance of children in the treated schools beforeand after they became bilingual with the group ofnonbilingual schools before and after the treat-ment. Thus, we employ the data for 2008/2009and 2009/2010 cohorts. The four groups that weanalyze are the following: the group of bilin-gual schools in 2008/2009 (the treatment groupbefore the treatment), the group of nonbilingualschools in 2008/2009 (the control group beforethe treatment), the group of bilingual schools in2009/2010 (the treatment group after the treat-ment), and the group of nonbilingual schools in2009/2010 (the control group after the treatment).

Table 2 provides descriptive statistics of thesefour groups. If we compare the schools wherethe bilingual program was introduced, before andafter the treatment, we see an increase in theproportion of students with characteristics thatare positively correlated with academic perfor-mance. More concretely, the proportion of chil-dren whose parents have university educationincreases from 33% to 39%, the proportion ofchildren whose parents have lower secondaryeducation decreases from 26% to 22%, and theproportion of children whose parents did not fin-ish compulsory studies also decreases from 8%to 5%. There are also important changes withregards to the occupations of the parents of chil-dren from these two cohorts: the proportion ofchildren whose parents have professional occupa-tions increases from 24% to 29% and the propor-tion of children whose parents have blue-collaroccupations decreases from 58% to 51%.

Furthermore, in the treated school, there isan increase in the proportion of Spanish stu-dents from 81% in the year before treatment to87% in the first-treated cohort, which translatesinto a decrease in the proportion of immigrantstudents (the most important change is in thereduction of the proportion of Latin Americanstudents from 10% to 6%, whose performanceis generally worse than that of Spanish studentsor even of other immigrants, after conditioningon observables [Anghel and Cabrales 2014]).We also detect an increase in the percentage ofchildren who started school before 3 years from46% to 51%.

However, if we look at the control group,we do not see any important changes in thecomposition of cohorts from one year to another:these proportions remain almost constant in


TABLE 2Descriptive Statistics Benchmark

TreatmentBefore

ControlBefore

TreatmentAfter

ControlAfter

Variable Mean Mean Mean Mean Diff-in-DiffStandard

Error (Diff-in-Diff)

SubjectsMathematics 8.94 9.54 10.55 10.88 0.26 0.230Reading 2.87 2.93 3.53 3.59 0.01 0.062General knowledge 2.28 2.35 3.17 3.37 −0.13 0.057

Subjects—standardMathematics −0.11 0.00 −0.06 0.00 0.05 0.042Reading −0.04 0.00 −0.04 0.00 0.00 0.042General knowledge −0.05 0.00 −0.15 0.00 −0.11 0.042

Individual characteristicsFemale 0.50 0.49 0.51 0.49 0.01 0.022Student with special education 0.11 0.07 0.06 0.06 −0.04 0.011Student with disability 0.04 0.03 0.03 0.03 −0.01 0.007Student’s age 12.15 12.14 12.12 12.14 −0.04 0.017Student Spain 0.81 0.81 0.87 0.81 0.06 0.017Student Romania 0.03 0.02 0.02 0.02 −0.01 0.007Student Morocco 0.01 0.01 0.00 0.01 0.00 0.005Student Latin America 0.10 0.11 0.06 0.10 −0.03 0.013Student China 0.00 0.01 0.00 0.01 0.00 0.003Student other 0.05 0.04 0.04 0.05 −0.01 0.009

Parent educationUniversity 0.33 0.48 0.39 0.47 0.07 0.023Higher secondary 0.21 0.17 0.20 0.18 −0.02 0.018Vocational training 0.12 0.12 0.14 0.12 0.01 0.015Lower secondary 0.26 0.17 0.22 0.17 −0.04 0.017Did not finish compulsory 0.08 0.06 0.05 0.05 −0.02 0.011

Parent professionBusiness, civil servant 0.17 0.22 0.19 0.22 0.02 0.018Professional 0.24 0.33 0.29 0.33 0.05 0.020Blue collar 0.58 0.46 0.51 0.45 −0.06 0.022

Age starting schoolStart school before 3 0.46 0.51 0.51 0.54 0.02 0.022Preschool 3–5 0.49 0.44 0.47 0.43 0.00 0.021Start school at 6 0.03 0.03 0.02 0.02 −0.01 0.007Start school after 6 0.02 0.01 0.01 0.01 −0.01 0.005

Observation of schools 25 1,201 25 1,217Observation of students 1,135 55,793 1,145 53,150

both years (at most there is a difference ofone decimal).

The numbers presented above indicate thatthere has been a change (certainly not large andperhaps endogenous) in the characteristics of thestudents enrolled in the bilingual schools from theperiod before to the one after the treatment. Thischange involves an improvement in student char-acteristics, such as the level of education and theoccupation of parents, or their nationality, whichare known to be determinants of the academicperformance of children. The same change couldbe taking place in other unobservable determi-nants. In our analysis of this issue, we will use theadditional information that we describe in the fol-lowing paragraphs to account for these changes.

We obtained the list of children who attendedthe treated schools since they were 5 years old,the last year of preschool education. With that

list, first, we analyze the group of schools wherethe number of children who entered after theybecame bilingual (i.e., children who were notenrolled in that school when they were 5 yearsold) is less than 4 (i.e., about 16% in the aver-age class of 25). There are eight treated schoolsthat satisfy this condition. As before, we com-pare these schools before they became bilingual(the 2008/2009 cohort) and after they becamebilingual (the 2009/2010 cohort) and we use as acontrol group the group of nonbilingual schools(we drop from the descriptive statistics the other17 bilingual schools).

The descriptive analysis in Table 3 shows avery similar picture to the one in Table 2. We seethat the change in the characteristics of studentsfrom the year in which they became bilingual tothe next one goes in the same direction and isquantitatively similar as for the whole sample. We


TABLE 3Descriptive Statistics—Schools with Few

Movements

EightSchoolBefore

EightSchoolAfter

Variable Mean Mean

SubjectsMathematics 8.73 10.48Reading 2.92 3.61General knowledge 2.28 3.11

Subjects—standardMathematics −0.15 −0.07Reading −0.01 0.02General knowledge −0.05 −0.20

Individual characteristicsFemale 0.49 0.50Student with special education 0.08 0.07Student with disability 0.05 0.04Student’s age 12.17 12.12Student Spain 0.85 0.93Student Romania 0.02 0.01Student Morocco 0.01 0.00Student Latin America 0.10 0.05Student China 0.00 0.00Student other 0.03 0.01

Parent educationUniversity 0.27 0.36Higher secondary 0.20 0.22Vocational training 0.15 0.12Lower secondary 0.31 0.25Did not finish compulsory 0.08 0.05

Parent professionBusiness, civil servant 0.17 0.20Professional 0.23 0.26Blue collar 0.60 0.54

Age starting schoolStart school before 3 0.46 0.55Preschool 3–5 0.52 0.44Start school at 6 0.02 0.01Start school after 6 0.01 0.00

Observation of schools 8 8Observation of students 416 434

observe an important increase in the proportion ofstudents whose parents have university degrees,from 27% in the pretreatment cohort to 36% inthe posttreatment cohort, and a decrease in theproportion of students whose parents did not fin-ish compulsory education (from 8% to 5%). Wealso identify a small increase in the proportion ofstudents whose parents have professional occu-pations and a small drop in the proportion ofstudents whose parents have blue-collar occupa-tions. Furthermore, there is an increase in the pro-portion of Spanish students from one cohort tothe next one in the treated schools and there isa big drop in the proportion of Latin Americanstudents. Finally, the percentage of children whostarted to go to preschool before 3 years oldincreases by 6% points (from 44% to 50%). Alto-gether, the selection problem that we detected

with the full sample persists in the sample of eightschools with very few incoming students afterthey became bilingual.

Second, we restrict further the group of stu-dents we analyze, by studying only the character-istics of the group of children that were alreadyenrolled in the 25 treated schools since they were5 years of age. The introduction of the bilingualeducation program was not announced in advanceof enrolling those children in the treated schools.This analysis produces almost identical conclu-sions as in the previous cases (Table 4): we detectan increase in the proportion of students withcharacteristics that are positively correlated withtheir academic performance and this fact revealsonce again a selection problem.

Third, we analyze the group of new incomingchildren in the 25 schools that became bilingualin 2004/2005, in order to see whether their demo-graphic characteristics could be a partial sourceof endogeneity.

From Table 5, it is clear that these studentshave a socioeconomic background which is verysimilar to the one of the remaining students ofthe bilingual schools. There is only one excep-tion; it looks like the proportion of immigrantstudents among the new incoming students is sig-nificantly higher: about 29% of the new incomingstudents are immigrants (out of which 12% areLatin Americans) while only 13% of all studentsin the bilingual schools are immigrants (out ofwhich 6% are Latin American).

Finally, we examine the sample of schools thatapplied unsuccessfully to the call for the bilingualeducation program, and whose score was veryclose to the cut-off for being part of the program.There are 38 schools that satisfy these conditions.If these schools are similar to the schools thatbecame part of the program, they would representa better control group than the whole group ofschools. In addition, if we see for those schoolsa similar change in demographics from one yearto the next one as the change that we see for ourtreated group, this could indicate that the explana-tion for this change does not necessarily lie in theintroduction of the bilingual education program.

The descriptive statistics of these schools inTable 6 reveal that both hypotheses are partiallyvalid. First, these schools are more similar indemographics to the treated bilingual schoolsthan to the schools from the complete controlgroup (comparison with column 3 from Table 2).However, there are differences: the most impor-tant difference is that the proportion of LatinAmerican students in this new group of schools


TABLE 4Descriptive Statistics—Children Who Did Not Move

TreatmentBefore

ControlBefore

TreatmentAfter

ControlAfter

Variable Mean Mean Mean Mean Diff-in-DiffStandard

Error (Diff-in-Diff)

SubjectsMathematics 8.94 9.54 10.54 10.88 0.25 0.249Reading 2.87 2.93 3.57 3.59 0.05 0.067General knowledge 2.28 2.35 3.16 3.37 −0.14 0.062

Subjects—standardMathematics −0.11 0.00 −0.06 0.00 0.05 0.046Reading −0.04 0.00 −0.01 0.00 0.03 0.046General knowledge −0.05 0.00 −0.16 0.00 −0.12 0.046

Individual characteristicsFemale 0.50 0.49 0.51 0.51 −0.02 0.023Student with special education 0.11 0.07 0.05 0.07 −0.06 0.011Student with disability 0.04 0.03 0.04 0.03 −0.01 0.007Student’s age 12.15 12.14 12.09 12.15 −0.07 0.018Student Spain 0.81 0.81 0.93 0.81 0.11 0.018Student Romania 0.03 0.02 0.01 0.02 −0.02 0.007Student Morocco 0.01 0.01 0.01 0.01 0.00 0.005Student Latin America 0.10 0.11 0.04 0.10 −0.06 0.014Student China 0.00 0.01 0.00 0.01 0.00 0.003Student other 0.05 0.04 0.02 0.05 −0.03 0.010



Age starting schoolStart school before 3 0.46 0.51 0.52 0.54 0.03 0.023Preschool 3–5 0.49 0.44 0.47 0.43 0.00 0.023Start school at 6 0.03 0.03 0.01 0.02 −0.02 0.007Start school after 6 0.02 0.01 0.00 0.01 −0.01 0.005

Observation of schools 25 1,201 25 1,217Observation of students 1,135 55,793 849 53,150

is bigger than in the bilingual schools. Second,the characteristics of children change from thepretreatment cohort to the posttreatment cohortin the same direction as they change for thebilingual schools for those cohorts, even thoughthese changes are a bit smaller than in thebilingual schools.

There is one striking phenomenon regardingthis group of schools. The average scores oftheir students are significantly lower than thescores of the students of the bilingual schoolsin the year 2008/2009 before the treatment.However, in the posttreatment, the scores ofthe students in these schools improve consid-erably, reaching almost the same levels as thescores of the students in the bilingual schools inposttreatment period.

Nevertheless, given the similarities betweenthis group of schools and the treated schools, in

the next section, as a robustness check, we willuse this group of schools as a control group.

Second cohort of students in the first schoolsimplementing the program. We have data for thesecond cohort of students (class of 2010/2011)being treated in the first 25 schools implementingthe program. They started primary school in theyear 2005/2006. They are 1 year younger than thefirst cohort of treated students but they too werealready enrolled as preschool students when theprogram was announced. The descriptive statis-tics for them are very similar to those reportedin Table 2 for the treated cohort after the treat-ment and they are not reported here to save space.We will estimate the effect for this second-treatedcohort to see if there is any learning in theseschools from having implemented the program tothe first cohort of students.


TABLE 5Descriptive Statistics—Children Who Moved

Variable MeanStandardDeviation


Subjects—standardMathematics −0.05 1.07Reading −0.11 1.05General knowledge −0.08 0.98

Individual characteristicsFemale 0.49 0.50Student with special education 0.12 0.33Student with disability 0.03 0.16Student’s age 12.21 0.45Student Spain 0.71 0.46Student Romania 0.05 0.21Student Morocco 0.02 0.14Student Latin America 0.12 0.33Student China 0.01 0.09Student other 0.10 0.30

Parent educationUniversity 0.44 0.50Higher secondary 0.19 0.39Vocational training 0.13 0.34Lower secondary 0.18 0.38Did not finish compulsory 0.06 0.25



Observation of schools 26Observation of students 341

Second Group of Schools Implementingthe Program. A second group of 54 schoolswere selected to implement the program from2005 to 2006. These were added to the 25schools that started implementing the programin 2004/2005. We have data for the first cohortof treated students in these 54 schools. Theyfinished primary education and took the CDIexam in 2010/2011. We analyze the results forthese treated students separately from the stu-dents from the first 25 schools implementing theprogram for two reasons. First, there were somechanges in the criteria used to select schools,as explained in Section II. Second, the class of2010/2011 from the 25 schools is the secondcohort treated at those schools, whereas these arethe first cohort treated at the 54 schools.

Only 53 of the 54 schools are going to beused in our study. One school is considered tobe an outlier because at the same time it has a

TABLE 6Descriptive Statistics—Schools That Applied toBecome a Bilingual School and Scored High in

the Selection Criteria

Variable

Mean inPretreatment

Period

Mean inPosttreatment

Period


Subjects—standardMathematics −0.22 −0.10Reading −0.32 −0.06General knowledge −0.20 −0.02

Individual characteristicsFemale 0.47 0.47Student with special

education0.09 0.09

Student with disability 0.04 0.05Student’s age 12.20 12.18Student Spain 0.71 0.72Student Romania 0.04 0.04Student Morocco 0.01 0.02Student Latin America 0.17 0.16Student China 0.00 0.01Student other 0.06 0.06

Parent educationUniversity 0.38 0.39Higher secondary 0.20 0.21Vocational training 0.11 0.11Lower secondary 0.21 0.21Did not finish

compulsory0.10 0.07



Observation of schools 38 38Observation of students 1, 341 1, 292

very large (the tenth largest among 1,226 schools)increase on the average reading score from 2009to 2010, and a very large (the fourth largestamong 1,226 schools) reduction on the averagereading score the next year. Furthermore, suchlarge and contradictory changes only happen inReading; they do not happen in Mathematics orGeneral Knowledge. Given this, we decided toexclude this school from our analysis.13

The descriptive statistics for the first cohortof treated students in the 53 schools selected

13. Our estimates of the effect of the program were madeincluding and excluding this observation and there is almostno change.


TABLE 7Descriptive Statistics for the 2005/2006 Bilingual Schools

TreatmentBefore

ControlBefore

TreatmentAfter

ControlAfter

Variable Mean Mean Mean Mean Diff-in-DiffStandard Error

(Diff-in-Diff)

SubjectsMathematics 10.42 10.91 5.61 5.90 0.20 0.139Reading 3.53 3.59 3.80 3.87 −0.01 0.044General knowledge 3.34 3.37 5.39 5.53 −0.11 0.064

Subjects—standardMathematics −0.08 0.01 −0.09 0.00 0.00 0.032Reading −0.04 0.00 −0.04 0.00 −0.02 0.032General knowledge −0.02 0.00 −0.05 0.00 −0.032

Individual characteristicsFemale 0.47 0.49 0.46 0.49 −0.01 0.016Student with special education 0.07 0.06 0.08 0.06 0.01 0.008Student with disability 0.04 0.03 0.04 0.03 0.00 0.005Student’s age 12.17 12.14 12.13 12.15 −0.05 0.012Student Spain 0.76 0.82 0.81 0.82 0.05 0.012Student Romania 0.03 0.02 0.03 0.02 0.00 0.005Student Morocco 0.01 0.01 0.01 0.01 0.00 0.003Student Latin America 0.13 0.10 0.09 0.09 −0.03 0.009Student China 0.01 0.01 0.00 0.01 0.00 0.002Student other 0.06 0.05 0.05 0.05 −0.01 0.007



Age starting schoolStart school before 3 0.51 0.54 0.56 0.55 0.03 0.016Preschool 3–5 0.44 0.42 0.41 0.42 −0.03 0.016Start school at 6 0.03 0.02 0.02 0.02 −0.01 0.005Start school after 6 0.02 0.01 0.01 0.01 −0.02 0.003

Observations of schools 53 1,163 53 1,179Observations of students 2,057 51,076 2,056 54,807

to implement the program in 2005/2006 arereported in Table 7. The demographic character-istics of the last cohort of nontreated students atthese schools are closer to the general populationcharacteristics than those in the last nontreatedcohort of the 25 schools. This can be seen bylooking at the differences between the first twocolumns in Table 7 and comparing it with thosedifferences in Table 2. Moreover, the change indemographic characteristics observed when com-paring the last nontreated cohort with the first-treated cohort is smaller here than in the first 25schools selected to implement the program.

B. Econometric Model of Education Production

Model and Selection Problems. Here, we useas the outcome for primary education thestandardized scores of students in the CDI exam

described in Section III.A. For a given year, thescore in that test for student i in school j, yij, isdetermined by:

(1) yij = δbilj + βxi + vj + ui + ξij

where xi is the observable characteristics of stu-dents and their families are described in SectionIII.A, bilj indicates whether school j participatedin the bilingual program, ui is unobservable char-acteristics of the students, such as effort or ability,vj is characteristics of the school, like quality ofthe principal and teachers, and ξij is a randomshock. Our parameter of interest is the averageeffect of the bilingual program on yij, which inEquation (1) is captured by δ. The difficulty thatwe face when we run the regression of yij on biljand xi is that we could suffer from an endogeneitybias because of two self-selection problems:


1. Students are not randomly assigned toschools. Their parents choose the school. Ifthere is no excess of demand for the schoolthey have chosen, they are admitted. If there isexcess of demand, the admission is based oncriteria like proximity of the family hometo the school and family income, both ofwhich are not random and are correlated withschool outcomes.

2. Schools are not randomly selected toimplement the bilingual program. The programwas implemented only in (some of the) schoolsthat applied for it. An application could be apositive signal of quality of the principal andteachers, because of the significant amount ofextra work required by the program. It couldalso be a sign that the school had low demand(perhaps because of low quality) with teachersabout to be displaced.14

Estimation Strategy. To control for the endo-geneity problem caused by the self-selection ofschools and students explained, we use diff-in-diff estimation. This solves the self-selection ofschools into the program because we observe thesame school the first year the bilingual program isimplemented in sixth grade and the year before.Given the institutional framework, the only sig-nificant changes in resources and staff from oneyear to the next are those associated with thebilingual program.

With respect to the self-selection of students,the diff-in-diff strategy also helps to solve thisproblem. As mentioned in Section II since theadmission rules to primary school gives prece-dence to preschoolers in that same school, andgiven the timing of announcement of the pro-gram, the differences between the first cohortof treated students and the previous cohorts arenot expected to be related to the introductionof the program. Given this observation, if themovements of students in bilingual schools afterthe program was introduced were the same asin the absence of the program (i.e., the samechanges as in nontreated schools), a diff-in-diffstrategy would control for the students beingdifferently distributed between treated anduntreated schools. However, as one can see inTable 2 and we discussed in Section III.A, there

14. In Spain, a large majority of teachers are civilservants and cannot be fired. But they can be movedbetween schools within a region. Even in a small regionlike Madrid, this can entail substantial inconvenience andthey would be willing to make significant efforts to avoidschool closures.

is a change in the characteristics of the studentsin bilingual schools after the program was intro-duced. Fortunately, the diff-in-diff easily allowsus to incorporate observable characteristicsof students in the estimation to control forthese changes.

Given the diff-in-diff strategy, we are going toestimate the following regressions by OLS:

yij = α0 + α1bilj + α2 year2010(2)

+ δ(year 2010 · bilj

)+ εij

yij = α0 + α1bilj + α2 year 2010(3)

+ δ(year 2010 · bilj

)+ βxi + εij

where year 2010 is a dummy variable for theacademic year 2009/2010, the first year whenwe observe the children exposed to the bilingualeducation program in the CDI exam. Also, wewill study further whether the change in the stu-dent population in bilingual schools is affectingour estimates by checking the robustness of ourresults to other comparisons and ways of estimat-ing the effect of the program.

IV. RESULTS

A. Estimates of the Effect of the Programfor the First-Treated Cohort

In Table 8, we present estimates of models(2) and (3). The parameter associated with thevariable Bilingual school 2004/2005 in posttreat-ment period (y10 · bilj) gives the effect of theprogram we want to estimate. Without covari-ates, the effect of the program is not signifi-cant for the three subjects. However, as we men-tioned when presenting the descriptive statisticsof the data, the cohort of treated students hasdifferent characteristics than the previous cohortin those schools. Those characteristics affectpositively the outcome; that is why the effectof the program is smaller once this change inobservables is taken into account. This changein the estimated effect of the program whenintroducing covariates reflects the fact that thereis selection in students after introducing the pro-gram. For mathematics and reading, the effectis not significantly different from zero in eithercase, although it goes from positive to nega-tive. For General Knowledge, the bilingual pro-gram has a negative and significant effect overthe score. As General Knowledge is the onlysubject taught in English from the three presentin the exam, it seems clear that the extra effort


TABLE 8Diff-in-Diff with and without Covariates: All Students (Bilingual Schools 2004/2005)

Mathematics Reading General Knowledge

Constant 0.002 4.517*** 0.001 3.093*** 0.001 3.391***(0.015) (0.132) (0.014) (0.132) (0.014) (0.137)

Posttreatment period −0.001 −0.073*** 0.000 −0.084*** 0.002 −0.072***(0.012) (0.011) (0.013) (0.012) (0.015) (0.015)

Bilingual school 2004/2005 −0.110 −0.006 −0.043 0.053 −0.046 0.069(0.074) (0.058) (0.096) (0.091) (0.093) (0.094)

Bilingual school 2004/2005 0.053 −0.068 0.002 −0.110 −0.096 −0.229**In posttreatment period (0.075) (0.069) (0.096) (0.099) (0.102) (0.112)Female −0.157*** −0.035*** −0.176***

(0.007) (0.006) (0.007)Student with special educational needs −0.744*** −0.702*** −0.620***

(0.017) (0.019) (0.020)Student with disability −1.080*** −1.127*** −0.892***

(0.020) (0.026) (0.025)Student’s age −0.384*** −0.262*** −0.280***

(0.011) (0.010) (0.011)Student Romania 0.036 0.017 0.061*

(0.027) (0.025) (0.031)Student Morocco −0.053* −0.256*** −0.147***

(0.032) (0.038) (0.043)Student Latin America −0.249*** −0.073*** −0.193***

(0.015) (0.014) (0.016)Student China 0.600*** −0.282*** −0.319***

(0.051) (0.054) (0.052)Student other −0.129*** −0.031** −0.100***

(0.017) (0.016) (0.016)Parent education—university 0.340*** 0.273*** 0.249***

(0.016) (0.018) (0.018)Parent education—higher secondary 0.182*** 0.173*** 0.169***

(0.015) (0.018) (0.017)Parent education—vocational training 0.181*** 0.204*** 0.184***

(0.016) (0.019) (0.018)Parent education—lower secondary 0.100*** 0.105*** 0.102***

(0.015) (0.019) (0.017)Parent occupation—business, minister, city hall 0.167*** 0.139*** 0.102***

(0.010) (0.010) (0.011)Parent occupation—professional 0.251*** 0.205*** 0.151***

(0.009) (0.009) (0.010)Lives only with the mother −0.099*** −0.080*** −0.079***

(0.023) (0.024) (0.027)Lives with the mother and one sibling 0.071*** 0.034 0.030

(0.025) (0.025) (0.029)Lives with both parents 0.066*** 0.003 0.065**

(0.022) (0.023) (0.026)Lives with both parents and one sibling 0.174*** 0.068*** 0.100***

(0.022) (0.022) (0.025)Lives with both parents and more than one sibling 0.151*** 0.055** 0.063**

(0.022) (0.023) (0.026)Other situations 0.063*** 0.014 0.011

(0.022) (0.024) (0.026)Preschool between 3 and 5 −0.072*** −0.034*** −0.054***

(0.007) (0.006) (0.007)Start school at 6 −0.220*** −0.188*** −0.195***

(0.022) (0.022) (0.023)Start school at 7 or more −0.295*** −0.304*** −0.248***

(0.026) (0.032) (0.033)Observations 111,128 92,100 111,268 92,268 111,268 92,268

Notes: Dependent variables are the individual standardized grades in each of the three subjects. Base categories for dummies:male, student Spain, parent education—did not finish compulsory studies, parent occupation—blue-collar, lives with the motherand more than one sibling, and preschool before 3 years old. Standard errors clustered at school level in parentheses.

*Significant at 10%. **Significant at 5%. ***Significant at 1%.


made to use English as the medium of instruc-tion comes at the expense of a worsening in theresults of standard examinations of that subjectin Spanish.15

To make a more intensive and flexible use ofobservable characteristics, we estimate the diff-in-diff regression by groups of students that havesimilar observable characteristics. In this way,the performance of treated students is comparedwith the performance of students with the sameobservable characteristics in nontreated periodsand schools. Table 10 reports results by parentaleducation for those students that were born inSpain, do not have any special educational needs,and are not older than 12 years old.16 These rep-resent more than two-thirds of the populationof students. In estimates not reported here forbrevity, we use the parents’ profession to formgroups in addition to education variables, but thequalitative conclusion is the same. Other charac-teristics are included as covariates in the regres-sion, because it is not possible to construct totallyhomogeneous groups. The estimates in this tableare those of the parameter associated with thevariable Bilingual school 2004/2005 in posttreat-ment period, that is, the effect of the program wewant to estimate. As with estimates with covari-ates in Table 8, we only find significant effects forGeneral Knowledge. However, these estimates bygroups have the following features: for Math-ematics and General Knowledge the estimatedeffect is more negative for students whose par-ents have a lower level of education; for Mathe-matics all of them continue to be nonsignificant,but for General Knowledge there is not a sig-nificant effect for students whose parents haveuniversity education whereas it is significant forall the other students. Moreover, the differencebetween the effect for the university group andthe effect for the compulsory education group issignificantly different from zero at 10%. Surpris-ingly, for Reading, there is no clear pattern. In anycase, the effect over reading is not significant forany of the groups.

15. In a sense, there is a confound, because it is possiblethat the students do not know less but simply they do not knowhow to express it in Spanish. But, even if that is the case,this would also suggest that the level of linguistic competencein English is not enough to leap through that barrier. And,possibly more importantly, other standardized examinationswhich, unlike the CDI, do have academic consequences (at theend of secondary and at the entry to university) are in Spanish,so a negative result in CDI is still an outcome of interest.

16. 11–12 years is the theoretical age that corresponds tosixth grade, which is the grade at which the CDI exam is taken(see Section III.A).

B. Further Look to the Potential SelectionProblem

Estimates of δ in Tables 8 and/or 10 willcapture the effect of the program not only ifthere is only selection on observables but also ifthe selection on unobservable characteristics ishighly correlated with the observables that wehave, where by selection on unobservables inour model we mean that the diff-in-diff changesin unobservables are endogenous. In the lattercase, the β coefficients of the x variables (likeparent’s education) will be capturing the effectof the unobservables (like educational resourcesat home) leaving the estimate of the effect of theprogram (δ̂) approximately unbiased.17 However,to check the robustness of these estimates, in thissection we explore further the potential reasonsthat could lead to an endogenous change in thepopulation of treated students, with respect tonontreated students. Even though the beginningof the program was not anticipated, the treatmentlasted for 6 years until we observed our outcomevariable, and during that period the followingmovements of students may occur because ofthe program:

1. In the Spanish education system, the stu-dents who perform badly can be retained in agrade once during primary education. This hap-pens on average to around 15% of the students inany cohort of sixth-grade students.18 As a conse-quence of the learning challenges added by thebilingual program, there could be more studentsretained in a grade than in the previous cohorts inthe same school. We would not observe the out-come for these retained students in their cohortbecause they are not yet in the sixth grade, mak-ing the estimated effect more positive than whatactually is. If this were the case, in the second-treated graduating class we would observe ahigher proportion of retained students than inthe last nontreated group at bilingual schools. Inour data, the proportion of retained students inthe second-treated cohort (2005/2006) is 18.00%,and that proportion in the last nontreated cohort(2003/2004) is 16.53 %. The difference in these

17. Even if one expects the treatment indicator to becorrelated with unobservables, the fact that student’s schoolchoice was made prior to the announcement of the program,and that changing school is difficult afterwards, will likelymake the resulting correlation between (year2010 · bil) andthe changes in unobservables much smaller than the correla-tion between x and the unobservables.

18. In our dataset, we define retention as being in an oldergroup than the age that sixth grades should have according tothe compulsory schooling rules if not retained.


two proportions is very small and not statisticallydifferent from zero—the p value is .35 —even ifwe test it after controlling for observable changesin the composition of the two cohorts. Therefore,this does not seem to be a problem.

2. If a student starting primary education in2003/2004 was retained in a grade in a bilin-gual school, he would have gone from a non-treated cohort to a bilingual one. Most of theclassmates of that child would have started schoolin 2004/2005 and, therefore, they would havealready participated in the bilingual program forsome years. These retained students may havepreferred, or may have been recommended tomove to a school without the bilingual programin the grade they had to repeat. If this is the case,the treated cohort for which we observe our out-come variable may have a smaller proportion ofthese retained students from earlier, untreated,cohorts. Looking at our data, we find that for thefirst group of bilingual schools the proportion ofretained students taking the CDI exam falls from16.53% in 2009 (the last nontreated cohort) to11.98% in 2010 (the first-treated cohort). The dif-ference is significantly different from zero at 1%.One would expect this factor to improve the out-comes of the treated schools. However, this prob-lem can be solved by comparing the results in thediff-in-diff only for the nonretained students inboth the control and treated groups, as we do inTable 9.19

3. Some students that were in a bilingualschool when the program was implementedmight have disliked the program and they couldhave decided to change school at any pointbetween the year of introduction of the programand the outcome we observe. We conjecture thatthere will be a very small proportion of studentsin this group. The reason for our conjecture isthat if they had decided to move, they could nothave gone to a highly demanded school, sinceat this stage those schools would have all theirvacancies filled. Nevertheless, we do not haveindividual data to support our guess.

4. Finally, other endogenous movements canbe related to the fact that some of the treatedschools had vacancies. As mentioned in SectionIII.B, vacancies can be a reason for a school to

19. Another reason to exclude retained students from thecomparison is that even if they take the CDI exam with atreated cohort, they have not received full treatment becausethey entered the program only after being retained; andretention usually does not take place in the first years ofprimary education.

TABLE 9Separate Diff-in-Diff Regressions for

Observable Groups of Students: EstimatedTreatment Effect by Group

Groups byParents’Education Mathematics Reading

GeneralKnowledge Proportion

University −0.027 −0.117 −0.107 36.36%(0.096) (0.128) (0.134)

Postcompulsorysecondary

−0.083 −0.210 −0.259** 19.11%(0.121) (0.136) (0.120)

Compulsoryeducation orless

−0.115 −0.062 −0.338** 12.33%(0.081) (0.134) (0.154)

Notes: Dependent variables are the individual standardized gradesin each of the three subjects. The sample used for these estimates arestudents of Spanish origin (i.e., nonimmigrants), not older than 12years and that do not have special education needs. They are divided byparents’ education into three groups. Proportion is the percentage thateach group represents over the total sample of students (including thosegroups like students older than 12 years whose diff-in-diff estimatesare not presented here). The following covariates were included inthese regression though not reported: dummies for year of the examand bilingual schools, sex, occupation of the parents, composition ofthe household in which the student lives, and age at which the studentstarted to go to school, preschool, or daycare. Standard errors clusteredat school level in parentheses.


apply for the program. Having treated schoolswith vacancies gives the opportunity to studentswith a good level of English, that otherwise mightnot have attended these schools, to apply for oneof the vacancies once the program has started.Because the treatment we evaluate started 6 yearsbefore we measure the outcome, new studentscould have been coming for these reasons for5 years.20 Once controlled for retention as indi-cated, this seems to be a major source of thechanges in student population in the bilingualschools reported in Table 2.

Another way to control for the endogenousincoming students to bilingual schools is to useinformation on who was at those schools beforethe program was announced. That informationis equivalent to an assignment to treatment indi-cator in experimental programs. For those stu-dents in bilingual schools taking the exam in2009/2010 (i.e., the treated cohort), we know whowas already at this school when they were 5 yearsold. For these students, the implementation of theprogram was not known when deciding to enrollin this school. We use this information to performthe following two estimates.

20. This does not mean that all the newcomers will comebecause of this reason. Some movements of students wouldhave occurred regardless of the program (e.g., due to migra-tion) and we control for this by observing the same schoolbefore the program.


TABLE 10Diff-in-Diff with and without Covariates: Bilingual Schools with More Than 16% of the Students

Coming to the School After Age 5 Are Excluded


No x With x No x With x No x With x

Constant 0.002 4.536*** 0.001 3.098*** 0.001 3.421***(0.015) (0.133) (0.014) (0.132) (0.014) (0.137)

Posttreatment period −0.001 −0.073*** 0.000 −0.084*** 0.002 −0.072***(0.012) (0.011) (0.013) (0.012) (0.015) (0.015)


Bilingual school 2004/2005 in posttreatment period 0.077 −0.017 0.028 −0.092 −0.155 −0.273*(0.116) (0.104) (0.214) (0.213) (0.122) (0.142)

Observations 109,654 90,892 109,793 91,059 109,793 91,059

Notes: Dependent variables are the individual standardized grades in each of the three subjects. Standard errors clustered atschool level in parentheses. Although not reported, estimates with x include the same covariates as in Table 8.


First, we can use that information to detectbilingual schools with a very large proportion ofstudents in the treated cohort who were at theschool since they were 5 years old. This will avoidthe bias due to new students coming to the schoolwhen the program was already in place. We selectthe eight bilingual schools that have a propor-tion of students that were not in that school at5 years old smaller or equal than 16%. Table 10presents estimates of Equations (2) and (3) (i.e.,diff-in-diff estimates) using as treated group onlythose eight schools and excluding from the sam-ple the other 17 bilingual schools. The resultsare similar to the results in Table 8 using thewhole sample. The only difference is that the esti-mated effects are more imprecise as the higherstandard errors indicate. Furthermore, the sameresults are obtained when doing the diff-in-diffusing as treated students only those that were atthe treated schools before the announcement andintroduction of the program.

Second, a different approach to the diff-in-diffis to find a control group of schools, that is, asclose as possible to the treated schools. We haveinformation about the schools that applied to theprogram and the criteria announced to chooseschools, mentioned in Section II. In particular,among the 192 schools that applied, 64 schoolshad more than 60 points (out of 70) in thosecriteria. The 25 selected were all from this groupwith scores above 60. The other 38 schools thatwere not selected but are comparable in thesecriteria form a natural control group. By assum-ing that these are comparable schools, we do nothave to use the diff-in-diff strategy and we canrun a regression using only the 2009/2010 resultsof the exam. This controls for the selection of

schools into the program. To control for selectionof students, we include as covariates the charac-teristics of the students we observe, and we useas an instrumental variable (IV) the indicator ofhaving been at the same school when the studentwas 5 years old (i.e., having being assigned totreatment).21 Table 11 contains these two esti-mates. Both ordinary least squares (OLS) and IVestimates imply the same qualitative conclusionsas in the rest of the estimates presented: negativeand significant effect on General Knowledgeof being in the bilingual program and no effectsignificantly different from zero on mathematicsand reading.

In addition to the IV estimation, we have alsoused the diff-in-diff estimator in the subset ofschools that applied for the bilingual educationprogram mentioned in the previous paragraph.The results are in Table 12. Again, the qualitativeconclusions are the same as with the entire setof schools.

Falsification test. Finally, in our checks we per-form a falsification test with the 2009 and 2010data, using as (false) treated group the sec-ond group of schools implementing the pro-gram. Those schools will have their first classof treated students taking the exam in 2011, butin 2010 their students in sixth grade are not yettreated. The schools actually treated in 2010 areexcluded for this test. Since the sixth graders inboth schools will not be in bilingual programs,there should not be any treatment effect. If we

21. Krueger (1999) is an example in Economics of Edu-cation where a variable related to the assignment to treatmentis used as instrument to control for potentially endogenousstudent movements.


TABLE 11OLS and IV with Schools That Applied to Become a Bilingual School and Scored High in the

Selection Criteria


OLS IV OLS IV OLS IV

Constant 4.020*** 4.086*** 4.288*** 4.245*** 3.143*** 3.235***(0.739) (0.739) (0.857) (0.849) (0.826) (0.811)

Bilingual school 2004/2005 in posttreatment period −0.070 −0.123 −0.081 −0.046 −0.186* −0.261**(0.082) (0.093) (0.056) (0.060) (0.098) (0.110)

Female −0.249*** −0.247*** −0.115*** −0.116*** −0.182*** −0.179***(0.039) (0.038) (0.037) (0.037) (0.044) (0.044)

Student with special educational needs −0.876*** −0.875*** −0.783*** −0.784*** −0.718*** −0.717***(0.078) (0.077) (0.103) (0.101) (0.117) (0.116)

Student with disability −1.204*** −1.206*** −1.214*** −1.213*** −0.937*** −0.940***(0.083) (0.083) (0.129) (0.127) (0.119) (0.118)

Student’s age −0.340*** −0.344*** −0.345*** −0.343*** −0.267*** −0.271***(0.058) (0.058) (0.070) (0.069) (0.067) (0.065)

Student Latin America −0.251*** −0.264*** 0.061 0.069 0.012 −0.005(0.082) (0.081) (0.073) (0.072) (0.085) (0.085)

Student China 0.777** 0.774** −0.031 −0.028 0.032 0.028(0.372) (0.371) (0.263) (0.257) (0.220) (0.220)

Parent education—university 0.242*** 0.243*** 0.279*** 0.278*** 0.232** 0.233**(0.086) (0.085) (0.101) (0.100) (0.093) (0.093)

Parent education—higher secondary 0.080 0.081 0.210** 0.209** 0.143 0.145(0.075) (0.075) (0.099) (0.098) (0.093) (0.094)

Parent education—vocational training 0.055 0.057 0.243** 0.241** 0.142 0.145(0.102) (0.102) (0.116) (0.114) (0.107) (0.107)

Parent education—lower secondary −0.096 −0.095 0.128 0.127 −0.010 −0.007(0.086) (0.086) (0.094) (0.093) (0.100) (0.100)

Parent occupation—business, minister, city hall 0.189*** 0.190*** 0.063 0.062 0.117** 0.120**(0.049) (0.048) (0.052) (0.051) (0.052) (0.051)

Parent occupation—professional 0.268*** 0.268*** 0.133*** 0.133*** 0.088* 0.088**(0.051) (0.050) (0.050) (0.049) (0.045) (0.044)

Start school at 6 −0.463*** −0.454*** −0.196 −0.202 −0.162 −0.149(0.150) (0.152) (0.205) (0.202) (0.200) (0.202)

Start school at 7 or more −0.405*** −0.410*** −0.003 −0.000 0.012 0.006(0.125) (0.123) (0.219) (0.217) (0.167) (0.163)

Observations 2,177 2,177 2,192 2,192 2,192 2,192R-squared 0.288 0.287 0.194 0.194 0.165 0.163

Notes: Dependent variables are the individual standardized grades in 2009/2010 CDI exam in each of the three subjects.Reference categories for dummies and explanatory variables included in the estimates are as in equations with covariates inTable 8. However, explanatory variables with no significant coefficient in any equation or those variables related to compositionof the family living with the student are not reported here. Standard errors clustered at school level in parentheses.


find an effect, it could mean that the introduc-tion of the bilingual programs has spillovers tountreated cohorts. More problematic for our esti-mates, it could also mean that there are pretrends,or that there is some sort of selection of pro-gram schools by unobservables. The results ofthis falsification test are reported in Table 13. Theestimated effects for the three subjects are posi-tive but not significantly different from zero, notfinding any evidence supporting the aforemen-tioned problems.

C. Results for the Second Cohort of TreatedStudents in the Schools Selected in 2004/2005

The estimates from Sections IV.A and IV.Breport the effect of the program on the first cohortof students treated in the group of 25 schools

that first implemented the program. In 2010, thiscohort finished sixth grade, the last year of pri-mary education, and took the CDI exam. Like-wise, we can use the results of the sixth graders inthe CDI exam in 2011 as the output for the secondcohort of students treated at those 25 schools. Theavailability of this additional year of data allowsus to test whether there are any improvements inthe second cohort of treated students in the first25 schools.

Table 14 reports the estimated effect for thissecond treated cohort of students. The qualitativeconclusion is the same as with the first cohortof treated students, presented, and discussed inthe previous two subsections. Quantitatively, theestimates tend to be larger (including a less neg-ative effect on General Knowledge) than those


TABLE 12Diff-in-Diff Estimates Using as Control GroupSchools That Applied to Become a Bilingual

School and Scored High in the Selection Criteria

Mathematics ReadingGeneral

Knowledge

Constant 3.759*** 3.198*** 2.742***(0.642) (0.661) (0.628)

Year 2010 −0.044 0.013 −0.044(0.084) (0.069) (0.100)

Bilingual school2004/2005

0.052 0.201* 0.104(0.087) (0.110) (0.127)

Bilingual school2004/2005 inCDI exam2009/2010

−0.097 −0.203 −0.256*(0.109) (0.123) (0.152)

Notes: Dependent variables are the individual standard-ized grades in each of the three subjects. Standard errors clus-tered at school level in parentheses. Although not reported,estimates include the same covariates as in Table 8.

*Significant at 10%. **Significant at 5%. ***Significantat 1%.

TABLE 13Falsification Test: Diff-in-Diff Using as

False-Treated Group the Schools That WillImplement the Program 1 Year Later

Mathematics ReadingGeneral

Knowledge

Constant 4.535*** 3.116*** 3.414***(0.133) (0.123) (0.138)

Posttreatmentperiod

−0.074*** −0.088*** −0.075***(0.012) (0.013) (0.015)

Bilingual school2004/2005

−0.010 −0.076 −0.010(0.049) (0.049) (0.075)

Bilingual school2004/2005 inposttreatmentperiod

0.018 0.078 0.075(0.057) (0.051) (0.079)

Observations 90,178 90,345 90,345

Notes: Dependent variables are the individual standard-ized grades in each of the three subjects in 2009 and 2010exams. Standard errors clustered at school level in parenthe-ses. Although not reported, all estimates include the same xcovariates as in Table 8.

*Significant at 10%. **Significant at 5%. ***Significantat 1%.

reported in Table 8, but the differences are small.In any case, this small improvement in the secondcohort is not enough to make the negative averageeffect on General Knowledge insignificant.

D. Results for the First Cohort of TreatedStudents in the Schools Selected in 2005/2006

Next, we look at the estimated effects of thefirst-treated cohort of the 53 schools that becamebilingual in 2005/2006. Each new selected school

starts implementing the program in the first gradeand expands it to the other grades, year by year,until all the primary education classes in thoseschools follow the bilingual program. This allowsus to check if our results for the schools selectedin 2004 to participate are confirmed for theschools selected in 2005, as explained in SectionII, there were some significant changes in theselection criteria from one year to the next.

The estimates are reported in Table 15. Wesee that, as in the previous analysis for the first25 schools selected, the effect is not significantlydifferent from zero in mathematics and reading.However, for General Knowledge, the effect isnow nonsignificant. This change in the averageestimated effect could be because of a compo-sition effect, as the effect is heterogeneous. Asseen in Table 9, the effect is higher in absolutevalue the smaller the level of education of the par-ents. The students at these 53 schools have bet-ter sociodemographic characteristics than thoseat the first 25 bilingual schools for which wedetected a negative and significant effect in Gen-eral Knowledge. This is why we next look at theestimated effects by groups of observables.

We can see in Table 16 that here the effectsin mathematics and reading continue being notsignificant for any group. Also, as for the first25 bilingual schools, in General Knowledge, theeffect is heterogeneous, and it is clearly non-significant for those students whose parents havea college degree, and negative and significant forthose whose parents have only compulsory edu-cation or less. However, there is an importantdifference with respect to the estimated effect ofthe treatment in the first 25 schools presented inthe previous sections. The negative effect of theprogram is smaller (in absolute value) here. Thischange implies that for those students whose par-ents have postcompulsory secondary educationthe effect of the program in General Knowledgeis now not significantly different from zero. Theestimated effect is now −0.028 and in Table 9 itwas −0.259.22 Also, all the other estimates forthe effect in General Knowledge (column 3 inTable 16) and most of the other estimates in thistable are much smaller (in absolute value) thanthe estimated effects for the first 25 schools.

22. A test of equality of these two estimated effects for thegroup “postcompulsory secondary” rejects the null hypothesisof equality of effects at 10% (p value is .0588). However, thenull hypothesis that the effects for the other groups estimatedin Table 16 are equal to those in Table 9 cannot be rejectedat typical significant levels (p value is .3001 for “university”and .2320 for “compulsory education or less”).


TABLE 14Diff-in-Diff with and without Covariates: Second Class of Students Treated at the 25 Schools Selectedto Implement the Bilingual Program in 2004/2005. Comparing CDI 2010/2011 with CDI 2008/2009



Constant 0.006 4.451*** 0.007 2.859*** 0.004 3.548***(0.015) (0.140) (0.014) (0.124) (0.015) (0.132)

Posttreatment period −0.002 −0.034*** −0.004 −0.022* 0.001 −0.016(0.012) (0.012) (0.014) (0.013) (0.014) (0.014)


Bilingual school 2004/2005 in posttreatment period 0.079 −0.011 0.022 −0.082 −0.076 −0.210***(0.078) (0.086) (0.097) (0.096) (0.090) (0.091)

Observations 110,939 91,681 110,966 91,705 110,966 91,705

Notes: Dependent variables are the individual standardized grades in each of the three subjects in 2008/2009 and 2010/2011.Standard errors clustered at school level in parentheses. Although not reported, estimates with x include the same covariates asin Table 8.


TABLE 15Diff-in-Diff with and without Covariates: First Class of Students Treated at the 54 Schools Selected to

Implement the Bilingual Program in 2005/2006



Constant 0.005 5.177*** 0.002 3.265*** 0.004 3.720***(0.015) (0.139) (0.012) (0.136) (0.014) (0.138)

Posttreatment period −0.000 0.041*** 0.000 0.067*** 0.001 0.058***(0.012) (0.012) (0.011) (0.011) (0.014) (0.014)

Bilingual school 2005/2006 −0.088 0.003 −0.042 0.016 −0.024 0.073(0.066) (0.050) (0.056) (0.040) (0.075) (0.064)

Bilingual school 2005/2006 in posttreatment period −0.010 −0.058 −0.004 −0.022 −0.029 −0.086(0.064) (0.059) (0.049) (0.048) (0.069) (0.067)

Observations 109,885 95,861 109,996 96,004 109,996 96,004

Notes: Dependent variables are the individual standardized grades

evaluating a bilingual education program in spain: …€¦ · evaluating a bilingual education...

Documents