document of the world bank...currency equivalents (exchange rate effective september 17, 2012)...

Document of The World Bank

Report No: ICR2264

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-47670)

ON A

LOAN

IN THE AMOUNT OF EURO 80 MILLION

(US$ 104 MILLION EQUIVALENT)

TO THE

REPUBLIC OF TURKEY

FOR A

SECONDARY EDUCATION PROJECT

December 21, 2012

Human Development Sector Unit Turkey Country Unit Europe and Central Asia Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective September 17, 2012)

Currency Unit =

TL 1.00 = US$ 0.56 US$ 1.00 = TL 1.79

FISCAL YEAR

January 1 – December 31

ABBREVIATIONS AND ACRONYMS

BEP Basic Education Project CAS Country Assistance Strategy CPS Country Partnership Strategy EU European Union FM Financial Management GDP Gross Domestic Product ICR Implementation Completion and Results Report ICT Information and Communication Technology ISR Implementation Status and Results Report MoD Ministry of Development MoNE Ministry of National Education NGO Non-Governmental Organization OBBS Student Achievement Assessment Exam OECD Organization of Economic Cooperation and Development PAD Project Appraisal Document PCC Project Coordination Center PDO Project Development Objective PISA Program for International Student Assessment PTA Parent Teacher Association QAG Quality Assurance Group QALP Quality Assessment of the Lending Portfolio SBS Students’ Placement Test (Sosyal Bilgiler Sinavi) SEP Secondary Education Project TTL Task Team Leader VTI Vocational Training Initiatives

Vice President: Philippe H. Le Houerou Country Director: Martin Raiser Sector Manager: Alberto Rodriguez

Project Team Leader: Naveed Hassan Naqvi ICR Team Leader: Carla Pittalis

REPUBLIC OF TURKEY Secondary Education Project

CONTENTS

Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 8 3. Assessment of Outcomes .......................................................................................... 16 4. Assessment of Risk to Development Outcome ......................................................... 27 5. Assessment of Bank and Borrower Performance ..................................................... 27 6. Lessons Learned ....................................................................................................... 30 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 32 Annex 1. Project Costs and Financing ......................................................................... 33 Annex 2. Outputs by Component ................................................................................. 34 Annex 3. Economic and Financial Analysis ................................................................. 42 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 47 Annex 5. Beneficiary Survey Results ........................................................................... 50 Annex 6. Stakeholder Workshop Report and Results ................................................... 51 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 52 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders ...................... 56 Annex 9. List of Supporting Documents ...................................................................... 57 Annex 10. Original and Revised Components .............................................................. 59

MAP IBRD 33501 R2

i

A. Basic Information

Country: Turkey Project Name: Secondary Education Project

Project ID: P066149 L/C/TF Number(s): IBRD-47670 ICR Date: 12/21/2012 ICR Type: Core ICR

Lending Instrument: SIL Borrower: REPUBLIC OF TURKEY

Original Total Commitment:

USD 104.00M Disbursed Amount: USD 104.61M

Revised Amount: USD 100.07M Environmental Category: C Implementing Agencies: Ministry of National Education Cofinanciers and Other External Partners: B. Key Dates

Process Date Process Original Date Revised / Actual Date(s)

Concept Review: 09/05/2001 Effectiveness: 05/08/2006 05/08/2006

Appraisal: 11/11/2004 Restructuring(s):

03/25/2008 04/07/2010 07/06/2011 12/23/2011

Approval: 03/15/2005 Mid-term Review: 05/05/2008 Closing: 05/31/2010 04/30/2012 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Unsatisfactory Risk to Development Outcome: Substantial Bank Performance: Unsatisfactory Borrower Performance: Moderately Unsatisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Unsatisfactory Government: Moderately Unsatisfactory

Quality of Supervision: Unsatisfactory Implementing Agency/Agencies:

Moderately Unsatisfactory

Overall Bank Performance: Unsatisfactory Overall Borrower

Performance: Moderately Unsatisfactory

ii

C.3 Quality at Entry and Implementation Performance Indicators Implementation

Performance Indicators QAG Assessments (if any) Rating

Potential Problem Project at any time (Yes/No):

No Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

Yes Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:

Moderately Unsatisfactory

D. Sector and Theme Codes

Original Actual Sector Code (as % of total Bank financing) Central government administration 17 Primary education 30 Secondary education 50 67 Vocational training 33 3

Theme Code (as % of total Bank financing) Education for all 33 50 Education for the knowledge economy 67 50 E. Bank Staff

Positions At ICR At Approval Vice President: Philippe H. Le Houerou Shigeo Katsu Country Director: Martin Raiser Andrew N. Vorkink Sector Manager: Alberto Rodriguez Charles C. Griffin Project Team Leader: Naveed Hassan Naqvi Robin S. Horn ICR Team Leader: Carla Pittalis ICR Primary Author: Carla Pittalis F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) To improve the quality, economic relevance, and equity of secondary education to support lifelong learning. Revised Project Development Objectives (as approved by original approving authority) To support improvements of conditions for student learning in secondary education and in schools in sub-provinces with low enrollment rates.

iii

(a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Secondary education curricula more closely aligned to student competencies and international standards of learning.

Value quantitative or Qualitative)

Curricula status at the beginning of SEP. N/A

Secondary education curricula more closely aligned to student competencies and international standards of learning.

Early independent assessments found that curricula revised by MoNE (outside the scope of the Project) were generally in line with international standards.

Date achieved 04/07/2010 03/16/2005 04/07/2010 04/30/2012 Comments (incl. % achievement)

Achieved. However, as no assessment was conducted after 2008 it is difficult to assess the extent to which the totality of curricula revised under SEP were aligned to international standards. Baseline was taken from the April 2010 ISR

Indicator 2 : Beneficiaries, including students, parents, teachers and school administrators, are satisfied with the use of school grants received by schools in sub-provinces with low enrollment rates.

Value quantitative or Qualitative)

N/A

Beneficiaries, including students, parents, teachers and school administrators, are satisfied with the use of school grants received by schools in sub-provinces with low enrollment rates.

Stakeholders partially satisfied with the use of school grants.

Date achieved 04/07/2010 04/07/2010 04/30/2012 Comments (incl. % achievement)

Partially achieved. The SDP Beneficiary Assessment indicates mixed satisfaction with the use of school grants. The baseline was taken from the April 2012 ISR.

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 :

Curricula reform (including 1,000 modules for vocational education), textbooks, and other educational materials and equipment have been developed and implemented in general secondary schools and for at least 15 occupations in vocational education.

iv

Value (quantitative or Qualitative)

Already existing courses at the secondary education.

Instit. structure, procedures for curriculum devel. revised; Gen. Educ.: 5 programs rev. and impl. in 280 schools; Voc. Educ.: 10 programs revised and an avg of 2 impl. in 149 schools. In-service training for impl. of 1,400 gen., 298 voc. new programs.

Curricula reform (including 1,000 modules for vocational education), textbooks, and other educational materials and equipment have been developed and implemented in general secondary schools and for at least 15 occupations in vocational education.

66 secondary education curricula revised and 62 of them implemented; 1,151 modules for vocational education completed and 924 of them implemented.


Achieved. The target for secondary education curricula in the ISRs varies between 75, 65 and later to 66.

Indicator 2 : Teachers in at least 1,600 secondary education schools that previously had no ICT equipment use ICT in their classroom teaching.


0

Classrooms insrtuction use of ICT improved in 400 schools, and E-learning opportunities extended to 400 schools.

Teachers in at least 1,600 secondary education schools that previously had no ICT equipment use ICT in their classroom teaching.

ICT equipment was provided to 1,629 schools. The scope of work was significantly reduced during a restructuring. No information on extend to which teachers use ICT.


Partially achieved. The scope of equipment was significantly reduced during a restructuring, to include only projectors and photocopiers.

Indicator 3 : New career guidance and counseling system used by at least 300,000 students in secondary education.


0

Interagency collab. mechan. est; Career Info standards, materials devel; Resources to 6,200

New career guidance and counseling system used by at least 300,000

The Career Information System was launched online in April, 2010. At project closing, it counted

v

schools, 250 ISKUR offices, 145 Guidance & Counseling Centers w/ in-service training for 10,000 staff. Integration of career guid. in regular curriculum increased.

students in secondary education.

1.3 million registered users.


Achieved.

Indicator 4 : Student achievement results reported and shared with stakeholders and public.


None

Nat. Stud. Assmt Tool kit, feedback systems & rpt cards for 429 schools; Perf. Mgmt. System for Staff & Instit.; 3-4 Int'l. Assmt.; Univ. Entr. Exam, eval. alternatives; Annual indic. and proj. repts; Impact Assmt: ICT, quantit., soc. impact, dissem.

Student achievement results reported and shared with stakeholders and public.

Piloting of Student Achievement Assessment Exam at the secondary education level completed in May, 2009. Dissemination took place in 2011 for MoNE representatives and academicians.


Achieved.

Indicator 5 : School Development Management Teams, comprised of teachers and school administrators, are established in all secondary education schools and prepare school development plans in line with strategic management guidelines.


0

School planning toolkit devel. and sent to 6,200 schools; Training materials developed for schools on use of manual; Pilot planning process in schools Development Bureaus est. in all 81 provinces.

School Development Management Teams, comprised of teachers and school administrators, are established in all secondary education

School Development Management Teams formed in all secondary schools, and in 3,689 primary schools. 15,000 educational staff trained in school plan preparation.

vi

Training of staff in 5 schools in use of planning tools.

schools and prepare school development plans in line with strategic management guidelines.


Partially achieved. Educational staff was trained on how to prepare school plans, but the actual plans were not prepared.

Indicator 6 :

Teacher training skills in selected vocational schools are adapted to modernized education technologies, teachers start implementing the new training methods, and teachers and school administrators are satisfied with the use of competitive school grants.


Adaptation of teacher training skills to modernized technologies not in place, implementation of new training methods not in place, and no school grants available.



Teachers and administrators generally satisfied with the use of the grants.


Partially achieved. No independent assessment conducted to assess the extent to which teachers' skills were aligned to modernized education, and of implementation of new training methods.

Indicator 7 : Use of school grants is consistent with the needs identified in the school development proposals.


No school grants in place.

Use of school grants is consistent with the needs identified in the school development proposals.

Use of school grants is consistent with the needs identified in the school development

Use of school grants was determined at the school level, based on self-identified needs.

vii

proposals. Date achieved 04/07/2010 04/07/2010 04/30/2012 04/30/2012

Comments (incl. % achievement)

Achieved. However, although decisions on the use of grant funds occurred at the school level, the SDP Beneficiary Assessment indicates that participation of students, parents, and local community in decision-making was lower than expected.

Indicator 8 : Key project assessment studies carried out and results discussed.


None N/A

Key project assessment studies carried out and results discussed.

Beneficiary Assessments were carried out for Curricula Reform, School Development Program, and Vocational Training Initiatives. Dissemination/results discussion with relevant stakeholders was generally limited.

Date achieved 04/07/2010 03/16/2005 04/07/2010 04/30/2012

Comments (incl. % achievement)

Partially achieved. Target does not define what are considered key assessment studies. Several assessments and surveys were deleted from the Project over the course of implementation, resulting in limited information on project's overall outcomes.

G. Ratings of Project Performance in ISRs

No. Date ISR Archived DO IP

Actual Disbursements (USD millions)

1 06/29/2005 Satisfactory Satisfactory 0.00 2 06/28/2006 Satisfactory Satisfactory 0.00

3 12/22/2006 Moderately Unsatisfactory

Moderately Unsatisfactory 5.13

4 04/29/2007 Unsatisfactory Unsatisfactory 5.13











10 06/11/2010 Moderately Unsatisfactory Moderately Satisfactory 21.85

11 01/10/2011 Moderately Moderately Satisfactory 29.63

viii

Unsatisfactory 12 05/29/2011 Moderately Satisfactory Moderately Satisfactory 40.30

13 01/01/2012 Moderately Satisfactory Moderately Unsatisfactory 69.57

14 04/24/2012 Moderately Unsatisfactory Moderately Satisfactory 91.44

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made DO IP

03/25/2008 Y U U 1.87

This Level-I restructuring was based on the Ministry of National Education's request to revise the project design. Key changes included: (i)addition of two new components; (ii)revision of the PDO and key performance indicators; (iii) revision of the results framework; and (iv) extension of the loan closing date.

04/07/2010 Y MU MU 20.97

This Level-I restructuring was based on the Ministry of National Education's request to cancel one of the components included as part of the 2008 restructuring. Other key changes included: (i)inclusion of a new sub-component; (ii) revision of the PDO and key performance indicators; (iii) revision of the results framework; and (iv) extension of the loan closing date.

07/06/2011 N MS MS 46.14 This Level-II restructuring served the purpose of reallocating loan funds.

12/23/2011 N MS MS 66.87

This Level-II restructuring served the purpose of reallocating loan funds and extending the loan closing date.

If PDO and/or Key Outcome Targets were formally revised (approved by the original approving body) enter ratings below:

ix

Outcome Ratings Against Original PDO/Targets Highly Unsatisfactory Against Formally Revised PDO/Targets Moderately Unsatisfactory Overall (weighted) rating Unsatisfactory

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1. Turkey’s Secondary Education Project (SEP) was approved by the World Bank’s Board of Executive Directors on March 15, 2005. The Loan Agreement was signed on February 22, 2006. The loan became effective on May 8, 2006, and closed on April 30, 2012.

1.1 Context at Appraisal 2. Country and sector background. In 2001, less than four years before SEP was approved, the Turkish economy had suffered a serious financial crisis with devastating consequences, including a devaluation of the currency, a jump in nominal interest rates to 100 percent, the virtual collapse of the banking system, and the bankruptcy of scores of enterprises. In the aftermath of the 2001 crisis, authorities announced a New Economic Program, which included a series of stabilization measures. The combination of such interventions and the political stability that followed in Turkey since 2002 helped the economy rebound from the crisis. During the 2002-05 period Turkey enjoyed robust economic growth, with GDP growth averaging 7.3 percent annually. 3. According to the Project Appraisal Document (PAD), some of the challenges faced by Turkey from an education perspective included:

(i) Low educational attainment and learning achievement. Comparative international studies indicate that the educational attainment and educational performance of the Turkish population and labor force at the time of project appraisal were among the lowest of Organization of Economic Cooperation and Development (OECD) countries. In 2004, Turkey’s adult1 population had, on average, less than 10 years of formal education, compared to the OECD average of 12 years.2 Similarly, only 27 percent had completed upper secondary education, compared to an OECD average of 68 percent. 3 Low quality of secondary education was reflected in the 2003 results from OECD’s Program for International Student Assessment (PISA), which indicated that between 37 and 84 percent of Turkey’s 15-year-olds were not proficient in mathematics, reading, and problem solving beyond the lowest level of competency, including considerable proportions scoring below such level.4 5

(ii) A growing working-age population. Turkey’s working-age population grew by 10 percent from 2000 to 2005 alone. 6 Sustained economic growth following the 2001 crisis provided the right conditions for generating new jobs vis-à-vis this increasing trend. This, however, would require future workers to acquire the relevant competencies and knowledge to be competitive in a rapidly changing, increasingly knowledge-intensive and global economy.

1 25 to 64 years old 2 Source: OECD, Education at a Glance 2006 Report. 3 Source: OECD, Education at a Glance 2007 Report. 4 The following percentages of Turkish 15-year old students participating in the 2003 PISA scored below Level I, the lowest in the PISA scale: Mathematics, 27.7 percent; Reading: 12.5 percent; Problem-solving: 51.2 percent. 5 Source: OECD, Learning for Tomorrow’s World: First Results from PISA 2003 Report. 6 Working-age population (15-64 years old) increased from 41 million in 2000, corresponding to 63.8% of total population, to 45 million in 2005, corresponding to 65.7% of total population. Source: OECD (http://stats.oecd.org/index.aspx?queryid=254).

http://stats.oecd.org/index.aspx?queryid=254

2

(iii) Income inequality. Despite a steady decline over the previous four decades, income inequality remained a concern for Turkey. In 2004, Turkey’s poorest quintile of the population controlled 6 percent of total disposable household income, compared to 46.2 percent controlled by the richest quintile.7 Analytical work suggests a close link between income shares and education in Turkey. In 2005, for example, 56% of the lowest income quintile had only a primary school certificate compared to less than 35% for the richest quintile, while only 0.50% of households with a university degree belonged to the lowest quintile.8

4. Rationale for Bank involvement. The main reasons behind the World Bank’s involvement in Turkey’s Secondary Education Project can be summarized as follows:

(i) Desire of the Bank to remain engaged in Turkey’s education sector. The three decades preceding SEP approval had seen the development and consolidation of a collaborative relationship between the World Bank and Turkey’s Ministry of National Education (MoNE). Eight projects had been implemented since then. The earlier projects focused on vocational education and training, but over time the priority shifted to projects supporting the Government’s broader education reform agenda. At the time of appraisal two operations, the First Basic Education Project (BEP 1) and the Second Basic Education Project (BEP 2), were still ongoing. Both of them encountered implementation difficulties and, at closing, were rated Unsatisfactory and Moderately Unsatisfactory, respectively, by the Bank’s Independent Evaluation Group (IEG) on overall outcome. (ii) Bank experience in supporting reform programs similar to those being undertaken in Turkey. Turkey’s 8th National Development Plan, covering the period from 2001 to 2005, called for an extension of compulsory basic education from 8 to 12 years. Simultaneous with this expansion, the Government wished to align the structure and improve the quality of its secondary programs with those of the European Union (EU). According to the PAD, the Bank had supported similar reforms in nearby countries, working closely with the EU, and had international expertise in the areas of curriculum reform, integration of ICT in instruction and learning, career guidance and counseling, and assessments.

1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 5. The original objective of the Project was “to improve the quality, economic relevance, and equity of secondary education to support lifelong learning.” The key indicators were:

(i) Secondary school graduates from schools receiving new curriculum and related support via the Project will have improved performance, as measured by scores on national and international assessments and graduate tracer studies, than those from schools not receiving project support. (ii) Secondary schools receiving new curriculum and related support via the Project will have a reduced variance in student achievement between schools in different regions, between general and vocational schools, and an overall rise in achievement.

7 Turkstat, 2005 (http://www.turkstat.gov.tr/PreTablo.do?alt_id=24). 8 A. Duman: Education and Income Inequality in Turkey: Does Schooling Matter? Financial Theory and Practice 32 (3) 369-385 (2008)

http://www.turkstat.gov.tr/PreTablo.do?alt_id=24

3

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 6. The Project’s Development Objective was revised twice, through Level I restructurings approved by the World Bank’s Executive Board of Directors on March 25, 2008 and April 7, 2010, respectively (see Table 1).

Table 1: Evolution of Project Development Objectives and Key Performance Indicators

Approved March 15, 2005

Revised March 25, 2008

Revised April 7, 2010

Development Objective

To improve the quality, economic relevance, and equity of secondary education to support lifelong learning.

To support the Government’s reform of general secondary and vocational education by improving conditions for student learning, establishing a career guidance system and enhancing foreign language teaching.

To support improvements of conditions for student learning in secondary education and in schools in sub-provinces with low enrollment rates.

Key Performance Indicators

i.Secondary school graduates from schools receiving new curriculum and related support via the Project will have improved performance, as measured by scores on national and international assessments and graduate tracer studies, than those from schools not receiving project support.

ii.Secondary schools receiving new

curriculum and related support via the Project will have a reduced variance in student achievement between schools in different socio-economic regions, between general and vocational schools, and an overall rise in achievement.

i.Improved student performance as measured by national and international assessments.

ii.Improved curricula and teaching,

based on an independent review

iii.Improved foreign language capabilities of teachers to enhance future student foreign language capacity, as measured by assessments carried out following the training.

iv.Improved student decision-

making as a result of the new career guidance system, as measured by a survey of students.

i.Secondary education curricula more closely aligned to student competencies and international standards of learning.

ii.Beneficiaries, including

students, parents, teachers and school administrators, are satisfied with the use of school grants received by schools in sub-provinces with low enrollment rates.

7. The main reasons for the changes in the objective and design of the Project are summarized below. First Level I restructuring of March 25, 2008

Request to finance a new activity. Soon after approval, MoNE formalized a proposal to finance teachers’ foreign language training overseas under SEP. Although this idea had been put forward earlier in the preparation stage, the Project, subject to time pressures on the Bank’s side, was approved without the activity, but with the understanding that it could be included shortly after. Project implementation stalled for three years while both parties tried to reach an agreement about this activity. This phase was characterized by intense and at times difficult discussions between MoNE and the Bank as well as within the Government itself, about the appropriateness of funding overseas teachers’ language training under SEP. The Bank expressed concerns over MoNE’s proposal on technical and efficiency grounds. In 2008, with virtually no implementation, SEP was restructured to include a Training for

4

Foreign Language Teachers sub-component at an estimated cost of Euro 24 million, corresponding to 30 percent of the loan amount. The component aimed at financing training for foreign language teachers, mostly English, in Turkey and overseas.

Second Level I restructuring of April 7, 2010 Cancellation of newly added component. MoNE requested that the Training for Foreign Language Teachers sub-component be implemented with the support of a consulting firm, whose role would be to provide technical and logistical assistance in Turkey and overseas. A disagreement between MoNE and the Bank on the composition of the shortlist of consultancy firms and the inability to reach an agreement eventually resulted in a request by MoNE to drop the activity altogether. The Ministry requested that the funds previously allocated to this activity be reallocated towards a new sub-component in support of school grants.

1.4 Main Beneficiaries 8. The PAD puts great emphasis on social and gender dimensions, highlighting a need to focus on students from poor families with low levels of adults’ education. It specifically notes that the Project would target resources to schools with lower socio-economic status, and strengthen education and career counseling to address social inclusion, job information flows, and equity issues. Cited beneficiaries include students, teachers, parents, community leaders, policymakers, and employers. Social and gender dimensions, however, were only marginally reflected in the original project design through a reference to improved equity of secondary education in the original PDO. This objective, to be measured by variance in performance between schools in different socio-economic regions, was not supported by relevant project activities. In addition, it is not clear how an intermediate indicator intended to measure improved equity in the university selection process fit into the Project’s scope, though it was included in the original Results Framework. 9. As part of the March 2008 restructuring, teachers became a more prominent beneficiary group. A new component, Staff Development Capacity, was added to support teachers’ foreign language training, which was eventually dropped, while vocational teachers in selected vocational schools, and vocational schools themselves, would benefit from competitive grants under a new Vocational Training Initiatives sub-component.

10. As part of the April 2010 restructuring, primary and secondary schools in sub-provinces with low enrollment rates and in areas likely to have been more adversely affected by the global economic crisis were to benefit from the newly introduced School Development Program (SDP) sub-component.

1.5 Original Components 11. The Project, as described in the Project Appraisal Document (PAD) consisted of two components, each comprising three sub-components. Component 1: Secondary Education Reform (Estimated cost at appraisal: Euro 64.7 million; Actual disbursements: Euro 35.5 million):

1.1 Secondary Education Programs. The objective was to revise and implement new general education and vocational education curriculum programs to provide students with core skills for the knowledge economy and lifelong learning.

5

1.2 Information and Communications Technology (ICT). The objective was to provide ICT training as a core competency for youth to function in a modern knowledge economy, to improve instruction by using ICT to expand access to learning opportunities, and to support better management of learning and administrative processes.

1.3 Career Guidance and Counseling. The objective was to provide youth and adults with access to career information and guidance services to promote informed career choices and related education and training decisions and facilitate lifelong learning.

Component 2: Quality, Assessment and Management (Estimated cost at appraisal: Euro 5.5 million; Actual disbursements: Euro 3.8 million):

2.1 Student Assessment and Evaluation. The objective was to improve the quality and outcomes of secondary education by developing systems to collect and disseminate reliable information on student learning and outcomes, and on the performance of the education system staff and institutions, and to use information from these assessments to improve system performance and student outcomes. 2.2 School Development. The objective was to help secondary schools staff, parents, and students improve the performance and outcomes, educational achievement, and quality in education through school development activities to be designed and implemented within the school development process.

2.3 Project Management. The objective was to ensure effective administration and coordination of the overall Project.

1.6 Revised Components 12. SEP’s components were revised twice during the course of implementation. The main changes are summarized in Table 2 below (see Annex 10 for a detailed description). 13. The first revision took place as part of the Level I restructuring approved in March, 2008. A new component, Staff Capacity Development, which included teachers’ foreign language training, was added. In addition, the Project Management sub-component became a self-standing component and was expanded to include Monitoring and Evaluation. 14. The second revision took place as part of the Level I restructuring approved in April, 2010 (see Section 1.3). The main changes were the introduction of a new School Development Program sub-component, and the cancellation of the sub-component intended to finance training for foreign language teachers.

6

Table 2: Summary of original and revised components’ description (see Annex 10 for details)

Original Revised March 25, 2008 Revised April 7, 2010 Component 1: Secondary Education Reform

No change in title No change in title

(i) Revision and Implementation of new general education and vocational education curriculum programs to provide students with core skills to compete in the knowledge economy; (ii) Information and Communications Technology (ICT) training; and (iii) Career guidance and counseling to youth and adults.

(i) Development and implementation of new curricula for 71 courses in general secondary education and 19 programs in vocational education, and writing and revising textbooks and modules for the new curricula; (ii) ICT classrooms in 322 schools and training of teachers; and (iii) Career guidance and counseling, including facilitating intra-agency collaboration, providing career information and guidance to education and labor institutions, training staff, and improving integration of career guidance and counseling services with new curricula.

No change

Component 2: Quality, Assessment, and Management

Component 2: Quality Assessment No change in title

(i) Student Assessment and Evaluation, aimed at improving the quality and outcomes of secondary education by developing systems to collect and disseminate information on student learning and outcomes, and on the performance of the education system staff and institutions; (ii) School Development, aimed at helping secondary schools staff, parents, and students improve the performance and outcomes, educational achievement, and quality in education through school development activities; and (iii) Project Management, aimed at ensuring effective administration and coordination of the overall Project.

(i) Student Assessment, aimed at developing and piloting student assessments and questionnaires in line with the newly developed curricula; and (ii) School Improvement, through training of 40,000 teachers to implement the new school development model, aimed at helping secondary schools improve their performance and student educational achievement, and through the development of a performance management to improve the evaluation of teacher performance.

No changes in content. The name of the second sub-component “School Improvement” became “School Development”.

Component 3: Staff Capacity Development Component 3: School Grants

(i) Training for Foreign Language Teachers to improve the language skills and teaching methods of foreign language teachers through in-country and international training. The training was designed in two phases. The first phase included in-country training for 150 teachers and overseas training for 500 teachers. Phase two would be based on an evaluation of effectiveness of the first phase; (ii) Vocational Training Initiatives, consisting of competitive grants for vocational schools to help improve the quality and economic relevance of vocational technical education programs.

The Training for Foreign Language Teachers sub-component was cancelled. The revised component supported: (i) Vocational Training Initiatives (no change); and (ii) A School Development Program (SDP) to provide grants to schools in sub-provinces with low enrollment rates in order to provide additional assistance to schools in areas likely to have been affected adversely by the economic downturn.

Component 4: Project Management, Monitoring and Evaluation

No change in title

This component aimed to support: (i) Project administration and coordination; and (ii) Regular monitoring of the Project, evaluation of project sub-components, and evaluation of the overall impact of the Project on the quality and relevance of education

No change

7

1.7 Other significant changes 15. The Project underwent the following significant changes in addition to those outlined in Sections 1.3 and 1.6: School Development Program (SDP) expansion. As part of a Level II, July 2011 restructuring, following the availability of additional funds from the partial cancellation of an ICT contract (see Section 2) and the accelerating implementation of SDP, the program was scaled-up from 2,710 schools in 46 sub-provinces to include an additional 809 schools in 10 new sub-provinces. Subsequently, as part of the Level II, December 2011 restructuring, MoNE requested to further expand the program to include an additional 170 schools in four new sub-provinces, on the basis of further project savings and continued progress of SDP.9 In the original SDP design, schools were allowed to use up to 60 percent of total allocated amounts for the procurement of goods, with the rest allocated to services. In May 2011, this percentage was increased to 80 percent.10 The additional 10 sub-provinces included as part of the July 2011 restructuring, however, were given full flexibility.

Cancellation of pilot and control schools for curriculum reform. In the original project design, 210 pilot general secondary education schools, 149 pilot vocational and technical schools, and 259 control schools were identified for the implementation of curriculum programs. As part of the March 2008 restructuring, MoNE and the Bank agreed that the notion of pilot and control schools no longer made sense, given MoNE's plans to implement revised curricula in all secondary education schools over the following two years. As a result, it was decided that SEP would support the development and implementation of curriculum reform in all schools in Turkey while supplying materials and equipment to a selected number of schools. Changes in the number of schools receiving materials and equipment, including ICT. The introduction of a new component on Staff Capacity Development, as part of the March 2008 restructuring, resulted in a significant reallocation of funds. This led to a reduction in the number of general secondary education schools which would receive equipment and supplies (from 280 to 210) and to a reduction in the number of schools which would receive ICT equipment (from 400 to 322). Subsequently, as part of the April 2010 restructuring, provision of ICT equipment was scaled up from 322 schools to 1,629 schools, identified across Turkey on the basis of classrooms and students’ numbers, targeting students in grades 1 through 5. ICT expansion was made possible through a reallocation of funds, mostly from Career Guidance and Counseling, Student Assessment, and M&E.

Three loan closing date extensions, for a cumulative period of 23 months. The loan closing date was extended three times, within the scope of the March 2008, April 2010, and December 2011 restructurings. The first, a seven-month extension from May 31, 2010 to December 31, 2010, aimed to allow the implementation of a new component, Staff Capacity Development, and of the expanded Project Management, Monitoring & Evaluation component. The loan closing date was extended a second time by 12 months, from December 31, 2010 to December 31, 2011, to allow the implementation of the newly added School Development Program (SDP). The last extension, from December 31, 2011 to April 30, 2012, aimed to allow the completion of all activities under the SDP and under the Secondary

9 Of the 3,689 targeted schools, 18 were closed. As a result, SDP funds were used by 3,671 schools. 10 As reflected in the Project’s Operational Manual dated May, 2011

8

Education Reform component, in particular the delivery of goods and equipment needed for training associated with curriculum reform. Loan funds reallocations. Throughout implementation, SEP underwent several reallocations of funds to reflect changes in project design. All reallocations were approved as part of the restructurings. Results Framework revisions. SEP’s Results Framework underwent two significant revisions as part of the March 2008 and April 2010 restructurings, to reflect changes in the PDO and project design. Some ISRs reflected different key indicators and targets than those reported in the approved results frameworks.

Cancellation of most of the originally planned assessments and surveys. The PDO revision during the April 2010 restructuring was accompanied by a further scaling-down of the M&E activities, including the cancellation of most of the assessments and surveys planned in the original design. These included: (i) graduate tracer studies and qualitative parent and student surveys to assess the impact of curricula reform and ICT skills; (ii) an independent review of revised curricula and teaching; (iii) a student survey and assessment to measure the improvement of student decision-making as a result of the new career guidance system; (iv) an ICT impact assessment to measure the extent to which ICT skills improved in 322 selected schools; (v) annual assessment of teacher performance as part of the school performance management system; and (vi) an impact study on the overall Project. The cancellation of such studies is not discussed in the 2010 Restructuring Paper, but only reflected in the revised Results Framework.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry 16. Key strengths of project preparation are summarized below: Frequent interaction of the Bank both at the central ministry and beneficiary levels. Documentation from the preparation phase indicates regular communication between MoNE and the Bank. Treasury and the Ministry of Development were also closely involved in preparation, and took the opportunity to comment on Aide Memoires and other relevant documents. The Bank also had frequent interactions with targeted beneficiaries, such as informal focus group discussions with parents, school foundation members, teachers, and school staff and students in different regions, and multiple meetings with key stakeholders representatives, including labor and the enterprise sector, and Non-Governmental Organizations (NGO). Extensive analytical work conducted to provide inputs to project design. The PAD lists several pieces of analytical work as having contributed to project design. These include a Knowledge Economy Assessment, a Poverty Survey that included questions directly related to the functioning of the education system in Turkey, labor sector analysis and surveys, and “additional education sector work”. Complementarities with EU program. At the time of approval, the EU had two ongoing investment projects supporting the reform of secondary vocational education programs. The goals of the EU’s two ongoing projects, Strengthening the Vocational Education and

9

Training System Project and Modernization of Vocational and Technical Education Project, were to establish a more qualified vocational education and training system in line with EU standards, and to further modernize and adapt the system to make vocational education and training more responsive to the socio-economic needs of the country and to the key principles of lifelong learning. SEP was designed to complement these two operations.

17. However, several shortcomings undermined the quality of project preparation and design. Limited initial client ownership, attributed to several factors:

(i) A broadly-owned secondary education strategy was not in place. The PAD contains

references to government plans for reforming secondary education, as outlined in the 8th National Development Plan. The key elements of such reform were an increase in compulsory education from 8 to 12 years, with a corresponding extension of the secondary school cycle from three to four years, and an increase in secondary education enrollment. The PAD indicates that, simultaneous with this expansion, the Government wished to align the structure and improve the quality of its secondary programs with those of the European Union, broaden vocational secondary programs to better reflect the needs of the knowledge economy, and allow increased flexibility to move between programs and schools. The PAD, however, does not contain any references to a specific secondary education strategy, while documentation from the preparation phase indicates that such strategy was not in place and that basic policy reform agreements were reached in the course of project design.11 According to MoNE officials, the main implication of this was an overall lack of clarity within MoNE on how to best utilize Bank funds, resulting in a fragmented design approach, with activities being adapted to the Project’s financial envelope rather than built on a strategy-based needs assessment. As MoNE’s General Directorates each put forward their respective priorities, the original design came to include several sub-components addressing very different and complex areas - from curriculum reform to ICT, career guidance and counseling, student assessment and evaluation, and school development – lacking synergies within a common strategic framework.

(ii) The initial dialogue on project design appears to have been driven largely by the World

Bank, with the support of Treasury and the Ministry of Development (MoD). The Bank had an interest in maintaining education as an area of engagement in Turkey and to have the Project delivered under the 2004-07 Country Assistance Strategy (CAS), while Treasury and the MoD welcomed Bank financing and technical expertise. The implication was that SEP was perceived by the Ministry of National Education (MoNE) as a supply-driven Project without clear ownership, despite addressing key areas of reform.

(iii) MoNE’s lack of strong incentives to borrow from the Bank. The fact that Bank

financing would not translate into new budget resources for MoNE was a contributing factor to the Ministry’s low interest in the Project. This was compounded by MoNE’s then-recent experience with two problematic education projects.

11The May 2001 Identification Aide Memoire, for example, states that during the identification mission agreement was reached on priority areas for investment that could support the development of a comprehensive secondary education system for Turkey. Furthermore, the November 2001 minutes of the Concept Review Meeting point to a clear perception of the need for reform within and outside MoNE, but no consensus or discussion on how to bring that about.

10

(iv) The original project design differed from MoNE’s initial request. An Aide Memoire dated as early as April 2006, before loan effectiveness, indicates MoNE’s intention to include overseas teachers’ training into the Project. This idea, according to MoNE and Bank officials, had been brought forward prior to project approval. However, the Project was submitted for approval without the inclusion of the activity but with assurance by the Bank that it could be easily incorporated shortly after. In fact, this was not the case and following project approval the Bank voiced technical concerns over MoNE’s proposal. In retrospect, submitting the Project for approval leaving this issue unresolved is likely to have contributed to a stalling of implementation for three years, with a negative impact on project ownership and on the dialogue between MoNE and the Bank.

Overly ambitious and broad PDO. At the Project Concept Note stage, concerns were raised over the difficulty to achieve multiple reform objectives and suggestions were made to focus on one objective or implement them incrementally. However, the Project maintained an ambitious objective and went on to finance a broad set of activities. Inconsistencies between sector background analysis and diagnosis, as described in the PAD, and actual project design. The PAD emphasizes the Government’s goals to improve equity and reduce disparities in the quality of education, and to improve the participation of girls and children from low income families. The PAD further states that these issues are consistent with the Project’s focus on schools serving students from poorer families with low levels of education. However, as stated earlier, these dimensions were not adequately reflected in project design. The PAD also cites the issue of flexibility within secondary education and into tertiary education and other issues identified through analytical work, such as access to education and the link between education and employment. Although the latter would theoretically be addressed through revisions of curricula, in so far as they would become more relevant to employment needs, the Project did not incorporate activities to address the above aspects directly. Excessively long and costly preparation. SEP preparation, from Concept Note to approval, took 42 months, well above Turkey’s average of 18 months in Fiscal Year 2005. This was indicative of some of the underlying issues described above.

Important lessons from earlier operations not adequately incorporated into project design. These include: (i) the need to keep project design straightforward, simple and realistic, with a scope appropriate to existing implementation capacity; (ii) the importance of having a robust monitoring and evaluation system in place before project launching (see Section 2.3); and (iii) the need to pay close attention to the political, social and institutional aspects of education reforms (see Section 6).

Key risks not adequately identified and/or mitigated. The PAD rates MoNE’s Implementation Capacity risk as the highest, citing activity reporting issues, weak accountability and complex management systems, and a lack of follow-up and feedback mechanisms. Mitigating measures included putting in place a dedicated SEP team within MoNE, utilization of a Project Implementation Plan, and consolidation of technical assistance for each component to reduce the complexity of procurement. On the other hand, the PAD omitted to explicitly discuss fiduciary issues, and SEP was ultimately designed as a procurement-intensive operation despite concept-stage concerns within the Bank at the prospect of adding another project to MoNE, given the difficulties it experienced in implementing procurement according to the Bank's guidelines under BEP 1. The PAD is also silent on the political economy risks that ended up affecting implementation (see Section 6).

11

Failure to implement a project preparation grant. Project preparation was to be supported by a US$286,000 Policy and Human Resources Development (PHRD) grant12, of which only US$31,000 was disbursed. The grant was approved in June 1999, became effective in March 2001 and closed in June 2003, after two extensions. Funds were mostly allocated to develop a methodology for streamlining existing secondary general and vocational educational programs, develop new training programs for the new curricula, and provide technical assistance for the development of a career guidance system. The PAD states that delays due to multiple factors, including changes in Government leadership, Government policy efforts directed at the Basic Education Program, and several changes in the Bank’s task team caused the grant to be closed without any significant disbursement.

18. Quality Assurance Group (QAG)’s assessment of Quality at Design was Moderately Unsatisfactory. SEP was included in a Quality Assessment of the Lending Portfolio (QALP) in October 2008. The assessment rated Quality of Design as Moderately Unsatisfactory. It found that the March 2008 Project restructuring resulted in a more realistic PDO and positively acknowledged the inclusion of the Training for Foreign Language Teachers’ sub-component. Main weaknesses identified include: (i) original design not including training for foreign language teachers, despite it being thought to be desirable; (ii) inappropriateness of the outcome indicator related to increased scores on the PISA and National Assessment, judged as unrealistic and with attribution issues; (iii) implementation arrangements not sufficiently capturing the institutional and cultural complexities of MoNE’s bureaucracy; and (iv) lack of focus and follow-up on equity, despite it having been extensively discussed at the Decision Meeting and noted in the PAD. The QAG review argued that if equity was not a major issue to be addressed in the Project, such a shift should have been addressed during restructuring. However, it noted that the Project Paper submitted to the Board in 2008 states that the proposed changes were not expected to have an impact of the Project's original social aspects.

2.2 Implementation 19. Positive features of implementation allowed the Bank and MoNE to overcome challenging design and implementation issues, and ultimately agree on a design that was highly owned at all levels of MoNE. New teams and approach. After a long period of underperformance, implementation went through an impressive turnaround during the last two years. Two main factors contributed to this: (i) the introduction of the School Development Program (SDP), which represented a milestone in MoNE’s ownership of the Project. With exceptional efforts on the part of MoNE and the Bank, the SDP was designed and implemented quickly; (ii) the ability of new Bank and Project Coordination Center (PCC) teams to revamp a dialogue which had been hampered by persistent implementation issues and differing views on how to address them. The last two years of implementation were repeatedly referred to by MoNE as the most constructive phase since project inception, characterized by increased responsiveness and flexibility on the Bank’s side. These efforts translated in a disbursement of about Euro 62 million during this period, corresponding to 77 percent of loan funds; SDP accounted for half of this amount.

12 Grant No. TF26143

12

Bank’s desire to remain engaged in Turkey’s education sector. The decision to retain and extend SEP in spite of a history of non-performance provided an opportunity for the Bank to remain more deeply engaged in a sector of high importance to Turkey’s development. Decentralized Bank team. A Bank team consisting largely of country-based staff provided regular implementation support to the client. Country-based procurement and financial management staff was an integral part of the project team and facilitated the resolution of most fiduciary issues emerging throughout implementation, also thanks to a solid understanding of the country context. An Operations Officer, also country-based, functioned as a liaison between the Washington-based Task Team Leader (TTL) and the Project Coordination Center (PCC).

20. At the same time, implementation suffered from a number of shortcomings and delays, including some that were not entirely under the Bank’s or the PCC’s control.

Pressures to deliver and disburse. Following a long preparation period, the Bank found itself under pressure to finalize and approve SEP, which was the only education project included in the 2004-07 CAS. However, as already mentioned, unresolved design issues would ultimately lead to a lack of implementation for three years. Similarly, pressures to disburse funds following a long history of unsatisfactory performance later contributed to some SDP design shortcomings and an overall decrease in strategic relevance. Political economy factors. Political transitions affected project implementation and content. SEP’s seven-year implementation history saw the appointment of four Education ministers. Even though some level of continuity was maintained at MoNE’s under-secretariat level, implementation delays occurred as new consensus needed to be built during each transition, which often resulted in new agendas and priorities.

Staff turnover. On the Bank’s side, SEP task team leaders (TTL) changed three times, while two additional TTLs led the preparation phase. Similarly, MoNE’s PCC was led by three different directors, and its operations were somewhat affected by staff turnover at the technical level. Although these changes were not a key factor in project delays, they did cause some discontinuity, because of the time needed for new teams to become familiar with the Project and respective counterparts.

Technical and relationship challenges. For good part of implementation, technical disagreements between MoNE and the Bank over project content and procurement procedures delayed implementation, and also negatively impacted the dialogue between the two parties.

Request for evidence of progress from the ongoing Basic Education Project. Although SEP was approved in March 2005, Turkey’s Council of Ministers’ ratification and publication in the Official Gazette of SEP’s loan agreement, necessary to declare effectiveness, came only one year after Bank approval. The reason for this delay was largely linked to the Ministry of Development’s request to see some progress on the ongoing Second Basic Education Project (BEP 2) prior to moving forward with the implementation of a new operation.

13

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 21. The original M&E design was hindered by unrealistic and unattributable indicators. The original Results Framework was built around broad and longer-term outcome indicators that were unrealistic both in terms of (i) the time needed to see the results of major curriculum reform in student learning and employment outcomes; and (ii) the difficulty in attributing changes in such outcomes to project interventions, considering the many factors involved. During implementation, this should have been addressed through the use of more realistic, shorter-term intermediate indicators. Instead, unrealistic indicators were replaced by ones that were vague, subjective, and difficult to quantify, such as beneficiaries’ satisfaction with the use of school grants. In addition, the original Results Framework lacked meaningful baselines, while the instruments to be used for collecting information were vaguely defined, with no explanation on how some of the planned reports, assessments, and graduate tracer studies would be carried out. 22. Results indicators were progressively reduced in scope and in their outcome-focus, reflecting changes in the Project’s development objective as well as the shorter implementation period remaining after each restructuring. As part of the 2010 restructuring the Results Framework was significantly simplified, with limited emphasis on outcomes. Indicators became largely output-oriented, while most of the originally planned surveys and assessments were deleted. It should be noted that the increasingly limited time left for implementation after each restructuring made it inevitable to choose revised objectives and indicators that were more short-term and output-oriented. The SDP intermediate outcome: “Schools in districts with low enrollment rates benefit from school grants designed to improve quality and/or increase enrollment” was an exception, in that it was more outcome-oriented. This intermediate outcome was never monitored during implementation, because it was judged to be unrealistic given the short amount of time available for implementation. 23. M&E implementation was significantly scaled back over the life of the Project, limiting evidence on effectiveness of project activities. The original project design did not include any M&E component but extensive M&E activities were listed in the Results Framework to assess the outcomes of project activities. During the March 2008 restructuring a self-standing M&E sub-component was introduced, with an allocation of Euro 1.57 million. While this was a positive development, the sub-component was overly ambitious and vague, particularly with reference to an “evaluation of the overall impact of the Project on the quality and relevance of education”. Although the M&E sub-component’s description remained unchanged throughout implementation, its targets were significantly scaled-back after each restructuring. Initial plans to conduct project impact studies were substituted by a more generic reference to key project assessment studies. Accordingly, funding allocations were cut back by 67 percent, to Euro 0.52 million. While the cancellation of such impact studies was in line with project design changes and a shortening implementation period, efforts should have been made to maintain or introduce other means to measure the effectiveness of key activities financed under SEP. For example, the Project financed in-service training to some 59,000 teachers and administrators, but without any independent evaluation of the effectiveness of such training. Also, no assessment of the Career Information System was carried out, even though the website became effective two years before project closing. 24. Beneficiary assessments were a valuable instrument to better understand project contributions. The three Beneficiary Assessments conducted under SEP for Curricula Reform, Vocational Training Initiatives, and School Development Program, provided useful insights on beneficiaries’ views of project activities and generated useful lessons. However, as noted in some of the assessments themselves, the lack of baseline surveys and of ongoing monitoring and

14

evaluation activities from the outset limited the usefulness of those results in terms of assessing the Project’s achievements.

2.4 Safeguard and Fiduciary Compliance

Safeguards

25. SEP did not trigger any safeguard policies, and was rated as a Category C project. As part of the SDP design, both the Loan Agreement and the SDP Operational Manual were amended to specify the types of expenses eligible for small civil works. These were limited to: (i) improving the hygiene conditions in the lavatories and wash rooms of the schools by means of rehabilitation and/or renewal of sanitary facilities; (ii) improving lavatories and wash rooms of the schools by means of rehabilitation and/or renewal of sanitary facilities; and (iii) improving the energy efficiency in the schools by replacing the lighting facilities with "A class" energy saving bulbs and minor heat isolation works to improve heat losses through the walls, doors and windows. Given the small size of grants, the Bank did not require a separate safeguards review.

Procurement

26. MoNE’s procurement capacity improved over time, and procurement activities were mostly carried out in compliance with Bank’s policy and procedural requirements. The PCC’s procurement capacity was inadequate at the initial phase of implementation, but improved through an increase in the number of procurement specialists to four. During the last two years, two of them left but the effect of their departure was mitigated by civil servants taking over some of the procurement activities. 27. Nonetheless, procurement-related issues posed significant challenges to project implementation, generating tensions between MoNE and the Bank and taking time to find mutually satisfactory resolutions. Noteworthy examples include: (i) a disagreement between MoNE and the Bank on the shortlist composition of consultancy firms to carry out the teachers’ foreign language training led to the cancellation of this Euro 24 million component; (ii) another disagreement between MoNE and Bank on the technical specifications for the procurement of 322 ICT classrooms estimated at Euro 6 million also led to the cancellation of this activity. (iii) Bank's conclusion that MoNE did not carry out the procurement of ICT equipment for 1,629 secondary education schools in accordance with Bank's Procurement Guidelines. This resulted from differences between national and Bank procurement rules with regards to the participation of nationally debarred firms, for which the Turkish Procurement Law, differing from Bank rules, explicitly indicates that national regulations prevail over international financial agreements, such as those stipulated with the Bank itself. Part of this procurement was completed with contracts worth Euro 5.75 million for the provision of photocopiers and projectors while the remaining, amounting to about Euro 7 million, was cancelled by MoNE. Funds resulting from the above cancellations were eventually reallocated to other activities during project restructurings.

Financial Management

28. The Project was in compliance with the audit requirements of the loan agreement. Treasury Controllers audited the Project on an annual basis and submitted their reports and management letters on a timely basis. The opinions were unqualified (clean). Following the introduction and rapid implementation of the School Development Program (SDP), the Bank’s Financial Management Team requested an internal audit report of the Program, which was never received.

15

29. Nonetheless, financial management performance fluctuated throughout implementation, with issues related to quality and reliability of reporting. These were largely attributable to high turnover and lack of sufficiently qualified staff in the PCC. In addition, the Project maintained the accounting data in outdated financial management software that was procured for the previous Basic Education Project. The software did not enable the generation of adequate reports and the PCC eventually shifted to using spreadsheets to maintain project accounts. These issues resulted in a downgrade of financial management (FM) performance to Moderately Unsatisfactory in November 2009 in the annual FM Supervision Report. The rating was reversed to Moderately Satisfactory shortly after, following specific remedial actions taken or planned by the PCC. Overall, FM reporting was generally acceptable until the last semester of 2011. Issues of quality and reliability of FM reporting re-emerged in the months leading to the closing of the loan. The PCC did not submit acceptable reports in the third and fourth quarters of 2011 and for the year 2012.13 From the second half of 2011 onwards, in addition to recurrent staffing issues, some errors and delays in the interim un-audited financial reports indicated some level of internal controls deficiencies over financial reporting. These factors, together with the lack of an audit report for the SDP, resulted in a second downgrade of FM performance to Moderately Unsatisfactory in the last FM Supervision Report, dated April 2012.

2.5 Post-completion Operation/Next Phase 30. Curriculum reform was a priority of the Government at the time of SEP preparation, and is likely to be sustained in the long run through Government funding. MoNE’s implementation of curriculum reform at the national level and good implementation progress of this sub-component further point to strong commitment in this area. 31. Despite its popularity, the SDP ceased operation at project closing and the central management capacity has been disbanded. Furthermore, the future of school financing decentralization in Turkey is not yet certain. On the national level, the implementation of SDP coincided with a sector capacity building project which identified decentralization as an important aspect of the Turkish education system. By demonstrating the viability of decentralized financing, the SDP is seen to have contributed substantially to the ongoing dialogue and planning for decentralization. There is a risk, however, that the momentum created by SDP at local and national levels will not be sustained, considering that MoNE’s immediate priority is implementation of the compulsory 12-year education program and that the development of detailed plans for decentralization would be a complex and time consuming process likely to meet with some resistance. 32. Prospects of future operations are uncertain. While the World Bank and MoNE continue to develop an analytical program,14 and the ongoing 2012-15 Country Partnership Strategy (CPS) indicates a potential education project in FY15, the prospects of this materializing appear uncertain at this stage. It should be noted, as indicated by MoNE officials, that the Ministry’s inclination to borrow from the Bank is affected by a number of factors, including: (i) perception

13 Although the PCC did not submit acceptable financial reports for the third and fourth quarters of 2011, the auditors (Treasury Controllers) issued a clean audit opinion on the 2011 annual project financial statements. Therefore, the overdue reports for the third and fourth quarters of 2011 are considered as submitted. The 2012 reports are still outstanding at the time of the ICR. 14 World Bank Report No. 54131-TR Improving the Quality and Equity of Basic Education in Turkey: Challenges and Options – June 2011. The Early Childhood Education report is forthcoming at the time of the ICR.

16

of difficult implementation procedures; (ii) a lack of additionality to MoNE’s budget; (iii) MoNE’s preference to work with its own procurement guidelines and operational procedures; and (iv) availability of EU grants.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation 33. Relevance of the original PDO is rated High. Notwithstanding its overly ambitious design, the original PDO’s focus on improving the quality and economic relevance of secondary education was in line with Turkey’s desire to adapt to the knowledge economy, as noted in the 8th National Development Plan. The Project was listed under the 2004-07 CAS for Turkey, which pursued four major objectives: (i) supporting sound macroeconomic management and governance; (ii) promoting equitable human and social development; (iii) creating a more attractive business climate; and (iv) strengthening environmental management and disaster preparedness and mitigation capacity. SEP was designed to support the second objective and, indirectly, the third by enhancing the development of human capital. In the 2008-11 CPS, education is highlighted as an important element, in view of the importance of skills to improve competitiveness and employment, and its role in reducing inequalities. In the CPS, implementation of the Secondary Education Project was expected to contribute to the achievement of the Government’s strategic target of considerably increasing the net enrollment rate in secondary education, which, however, was not an objective of the Project, and improving the quality of secondary education by updating curriculum, providing effective use of ICT and raising qualification of teachers. 34. Education as a whole remains a priority in Turkey and was identified as the biggest area of concern for the Turkish people, according to a recent World Bank survey of 29 countries in the Europe and Central Asia Region. According to the survey, roughly 5 in 10 Turkish citizens believe that education should be the highest priority area for additional Government investment.15 In this context, the Bank’s engagement in Turkey’s education sector continues to be highly relevant.

35. While the Implementation Completion and Results Report (ICR) recognizes that the original Project development objective (PDO) was overly ambitious, the two revisions to it also meant that it lost its strategic relevance in terms of the Government's overall program and the CPS's objectives. As a result, relevance is rated Modest for the two revised PDOs.

SEP was approved in March 2005 as an ambitious reform project. In March 2008, three years later and with virtually no disbursements, the Project was restructured and its development objective narrowed down, becoming fragmented and output-oriented. The justification for the introduction of teachers’ foreign language training was not supported by evidence that this was an essential element in Turkey’s reform of the education system, and both the Bank and other Government agencies had raised concerns over the activity’s relevance and efficiency. Two years later, in April 2010, following limited progress, the Project was restructured again and its development objective further narrowed down to focus solely on improving

15 Improving the Quality and Equity of Basic Education in Turkey: Challenges and Options, World Bank Report No 54131-TR.

17

conditions for student learning, to be achieved largely through the provision of school grants and through continued revision of curricula. The basis for the School Development Program (SDP), as outlined in the 2010 Restructuring Paper, was that the economic crisis in Turkey had made it harder for parents and families to provide support to their local schools, adversely impacting learning conditions at schools in economically disadvantaged areas. The new SDP was designed to provide additional assistance to schools in areas likely to have been affected adversely by the economic downturn, and thus ameliorate the effects of the economic slowdown while building capacity at primary and secondary schools. However, no evidence is available that schools in selected sub-provinces had been adversely affected by the crisis, and that this had translated into worsening learning conditions. In fact, when the SDP was designed, data at the sub-provincial level were not available for economic variables but only for educational variables, most notably net enrollment rate for primary schooling, which was ultimately chosen as the driving targeting criteria.

3.2 Achievement of Project Development Objectives Original PDO: To improve the quality, economic relevance, and equity of secondary education to support lifelong learning. Key Performance Indicators Status 1. Secondary school graduates from schools receiving new

curriculum and related support via the Project will have improved performance, as measured by scores on national and international assessments and graduate tracer studies, than those from schools not receiving project support.

Achieved but no evidence meaningfully linkable to project.

2. Secondary schools receiving new curriculum and related support via the Project will have a reduced variance in student achievement between schools in different socio-economic regions, between general and vocational schools, and an overall rise in achievement.

Not achieved (lack of evidence)

36. No evidence on improved student performance and reduced variance in student achievement between schools. According to the PAD’s Results Framework, improvements in student performance and reduced variance across schools from different socio-economic regions were to be measured through (i) an annual administrative report on flow to higher education; (ii) student follow-up surveys; (iii) results from the Program for International Student Assessment (PISA); and (iv) annual national assessments. None of these planned reporting and assessments were conducted under the original PDO. Both PDO and key performance indicators were eventually judged to be overly ambitious and unrealistic, as many factors are known to contribute to improved student performance, with curriculum reform being just one of them. The indicator on reduced variance between schools was removed from the Results Framework as part of the 2008 restructuring. All remaining references to improved student performance were removed from the project’s Results Framework as part of the 2010 restructuring.

18

37. Achievement of the original PDO is rated Highly Unsatisfactory. Implementation under the original objective was very limited, with no measurable contributions to the PDO. In the three years since approval, disbursements amounted to Euro 1.85 million, corresponding to two percent of loan funds. None of the key performance and intermediate outcome indicators were met. Notwithstanding the good progress on curriculum reform throughout the remaining of project implementation, project activities cannot be meaningfully linked to improvements in quality, economic relevance, and equity of secondary education. Revised PDO (March 25, 2008 – April 7, 2010): To support the Government’s reform of general secondary and vocational education by improving conditions for student learning, establishing a career guidance system and enhancing foreign language teaching. Key Performance Indicators Status 1. Improved student performance as measured by

national and international assessments. Achieved but no evidence meaningfully linkable to Project.

2. Improved curricula and teaching, based on an independent review.

Largely achieved (but no follow-up independent assessment after 2008).

3. Improved foreign language capabilities of teachers to enhance future student foreign language capacity, as measured by assessments carried out following the training.

Not achieved

4. Improved student decision-making as a result of the new career guidance system, as measured by a survey of students.

Not achieved (lack of evidence)

38. Student performance improved between 2003 and 2009, but there is no evidence linking these achievements to Project interventions. From 2003 to 2009, the performance of Turkish students in PISA assessments improved significantly, with a sharp increase over the last three years (Figure 1). The rise in quality was accompanied by a decrease in inequality, evidenced by a narrower performance gap between the richest and poorest students (Figure 2). However, it would be difficult to link these improvements to the Project as in 2009, by the time the PISA was

Figure 1: Turkish students’ PISA performance 2003-09 Figure 2: Turkey PISA results by Quintile 2003 vs. 2009

Source: OECD

19

administered, only few curricula had been developed within the scope of SEP, and with a limited implementation period. 16 No study was carried out to evaluate the Project’s contribution to learning achievement and this indicator was removed from the Results Framework in 2010 with a consensus that such link was unrealistic. Given the lack of any sort of evaluation of the curriculum reform undertaken under SEP, it is not possible to determine how this activity contributed to better learning outcomes for students in Turkey. 39. Progress was made on curriculum revisions, but with no evidence of improved teaching methods. Between 2008 and 2010, implementation of the curricula reform component made considerable progress. According to MoNE’s Progress Report No. 4, as of mid-April 2010, 49 out of 66 curricula had been revised for most of the grades from 9 to 12. Of these, 14 had been published or were in the process of being published, and most of the corresponding textbooks had been published or were under review. Improvements in the quality of the new general and vocational-technical secondary education curricula were assessed through an Assessment of the Beneficiaries of the Curriculum Reform of the Secondary Education Project, undertaken in April 2012. The assessment highlights both strengths and weaknesses of the revised curricula. On the one hand, revised curricula were generally judged appropriate with respect to teaching, learning process, and measurement and evaluation dimensions. Beneficiaries also indicated an overall high level of satisfaction, including positive opinions on the possibility to integrate ICT into teaching and learning process, apply diverse methods, and link between teaching and assessment. On the other hand, aspects related to context, objectives/attainments, content measurement and, for some subjects, evaluation, were judged to be partly appropriate. One of the most important shortcomings identified by beneficiaries was a lack of a clearly defined philosophy and learning theory underlying the curricula reform.

40. No progress made towards improving teachers’ foreign language skills. The sub-component in support of this objective was fully cancelled at the request of MoNE before any implementation took place. 41. A career guidance system is in place, but it is too early to see its benefits. As part of SEP, a Memorandum of Understanding was signed between the Turkish Employment Agency (ISKUR), the Ministry of Development, the Council of Higher Education, and other relevant institutions. In close partnership with these agencies, a new web-based Career Information System (CIS) was developed and launched on April 1, 2010. The CIS (http://mbs.meb.gov.tr) is run by the Special Education General Directorate of MoNE. Designed to provide support to people of all ages based on the principle of lifelong learning, it includes three main sections: (i) a self-assessment of abilities, interests, and career values; (ii) an overview of available educational options; and (iii) labor market information, opportunities, and job application tools, including guidance on CV and preparation of job application letters. The website also includes several links to Turkish institutions, including ISKUR. By project closing the CIS website grew to 1.3 million registered users, against a target of 300,000. The scope of the CIS, however, remains limited in that it does not yet connect individuals and institutions to employers and existing job openings. Plans are underway to further evolve the system and to better serve the needs of various beneficiaries. A survey of CIS users, planned under the original project design, was deleted as

16 Four general secondary education curricula for Turkish literature, Mathematics, Geography, and Language and Expression were developed in 2004-05 by MoNE outside of the scope of the Project, and implemented in the 2005/06, 2006/07 and 2007/08 academic years, respectively, for grades 9, 10, and 11. Between end-2008 and mid-2009, the Project directly supported the revision of curricula for grade 11 for Physics, Chemistry, Biology, and History.

http://mbs.meb.gov.tr/

20

part of the 2010 review of the Results Framework, along with the intermediate outcome indicator measuring improved students’ decision-making on education and career choices as a result of strengthened career guidance services. At the time of the ICR no evidence is thus available to evaluate the effectiveness and benefits of the system. 42. Insufficient evidence to determine the contributions of Vocational Training Initiatives (VTI) to the PDO. Although the PDO makes a reference to vocational education reform, and a Vocational Training Initiative (VTI) sub-component was added as part of the 2008 restructuring, a result indicator and a target for this activity were only introduced during the 2010 restructuring, after VTI implementation had been completed. De facto, the target value reported in subsequent ISRs was the actual achieved result. The relevant intermediate outcome was vague and not easily measurable: “Vocational schools benefit from competitive school grants designed to modernize vocational training methods and education technologies”, and no directly relevant outcome indicators were introduced to measure the contributions of VTIs in terms of improved conditions of student learning in secondary education (PDO). The VTI sub-component financed grants to vocational technical schools on a competitive basis, with the aim of promoting innovation, and improving the quality of vocational secondary programs and their relevance to employment needs. A total of 433 applications were received, of which 46 proposals were accepted from 33 provinces and were implemented between 2008 and 2009. Evidence of outcomes is limited to the opinions of participating staff of the vocational schools, as reported in the Beneficiary Assessment. Participating teachers and administrators considered the impact of the initiatives to have been positive on all the levels examined: program development, professional development, school-industry cooperation, suitability to project goals, contribution to student-based learning, and sustainability. The highest impact was found to be on staff professional development. However, the Beneficiary Assessment report also flagged significant weaknesses with respect to the impact on school-industry relationships and sustainability. It was found that, while initiatives did tend to foster existing school-industry relationships, they did not facilitate the development of new ones. The benefits of both staff professional development and the provision of new equipment were thought not to be readily sustainable considering that: (i) foreign language skills regressed quickly without regular practice and follow-up training; (ii) the pace of technological change meant that new equipment quickly became out of date; and (iii) keeping up with relevant technology depended on close cooperation with industry which was often lacking. Overall, the design and implementation of the VTI sub-component did not take into account the constraints to promoting initiative at the school level and building effective school-industry cooperation in a centralized and hierarchical education system. 43. Achievement of the revised PDO is rated Unsatisfactory. Under the revised PDO, implementation progressed on several fronts, including the successful establishment of a Career Information System, an increased number of revised curricula, and implementation of several vocational training proposals. At the same time, PDO relevance weakened, while overall implementation remained slow, with only Euro 15 million disbursed five years after original project approval, corresponding to 19 percent of total loan funds. No evidence is available to assess PDO achievement, with the exception of the establishment of a career guidance system, whose benefits cannot however yet be measured. No progress was made on foreign language teaching enhancement, as the component was eventually cancelled with no implementation. While Turkish students’ performance improved significantly between 2003 and 2009, as reflected by PISA scores, there are clear attribution issues preventing from linking this improvement to project interventions. Further, no systematic evidence of efficiency gains is available (see Section 3.3).

21

Revised PDO (April 7, 2010 – April 30, 2012): To support improvements of conditions for student learning in secondary education and in schools in sub-provinces with low enrollment rates.

Key Performance Indicators Status 1. Secondary education curricula more closely aligned to student

competencies and international standards of learning. Achieved (but no follow-up independent assessment conducted after 2008)

2. Beneficiaries, including students, parents, teachers and school administrators, are satisfied with the use of school grants received by schools in sub-provinces with low enrollment rates.

Partially achieved. Beneficiaries partially satisfied with the use of school grants.

44. Early independent assessments indicate that curricula revised by MoNE were generally more aligned with international standards, but no follow-up assessment was conducted on Project-supported curricula revisions. By project closure, 66 curricula in general secondary education, representing the universe of all secondary education courses, were revised, and 62 of them were implemented. With respect to vocational secondary education, 1,151 modules were revised, and 924 of these were implemented, meeting the original target of 1,000 modules.17 Independent assessments financed under SEP in 2007 and 2008 indicate that four curricula developed by MoNE outside of the Project scope were generally in line with current international standards. Specifically, they were found to: (i) offer a clear set of learning objectives and content enabling the development of students’ knowledge and key competencies; and (ii) support students in structuring a firm set of values and attitudes necessary for further personal growth and social insertion. Textbooks designed and published in line with the revised curricula were also found to be pedagogically sound, to offer a large range of learning opportunities for students, and sufficient examples of teaching practices to be used in the classroom. At the same time, the assessments highlighted the fact that despite significant improvement of individual subject curricula, the country had not developed a national curriculum framework or underlying philosophy for secondary education that would provide the conceptual and theoretical basis of the new curricula, outline expected outcomes on students, and ensure cross-curricula coordination and a common set of cross-curricular objectives. The assessments also recommended increasing emphasis on monitoring the curriculum implementation process through teachers’ feedback and student assessments as well as increasing professional development support to teachers for curriculum implementation. Although extensive teachers’ in-service training was provided and individual, ad-hoc seminars were conducted to seek feedback on the implementation of the Geography, Mathematics, Turkish Literature, and Foreign Language curricula, respectively, there is no evidence that these recommendations were reflected more broadly and on an ongoing basis in the final years of the project implementation. Further, because no follow-up independent assessment was conducted after 2008 and because of MoNE’s decision to rely solely on national consultants, it is difficult to assess the extent to which the totality of curricula revised under the scope of SEP were aligned to international standards.

45. The School Development Program (SDP) introduced a paradigm shift in Turkey’s management of school funds, and was made possible by exceptional joint efforts of the Government and the Bank. With remarkable efforts on the part of MoNE and the Bank teams,

17 Outside of the Project’s scope, MoNE developed and implemented an additional 6,197 vocational secondary education modules.

22

SDP was designed and implemented quickly, and extended twice to expand coverage. By project closing, Euro 31.8 million were disbursed in school grants for 3,671 projects identified at the school level, exceeding the original targets. Although beneficiaries were satisfied with some aspects of the grants, pressure for rapid disbursement contributed to weaknesses in design and implementation. The SDP featured several positive aspects: Eighty percent or more of beneficiaries 18 agreed that SDP funds contributed to improving school learning environment. Beneficiaries also generally agreed that SDP contributed to several aspects of school life, including a general improvement in school environment and in students’ social behavior. Although not an explicit objective, an important outcome of the SDP was the capacity built at the school and sub-provincial levels in participatory development of school plans, and in budgeting, procurement and financial management. The SDP generated interaction between Parent Teacher Associations, Non-Governmental Organizations, and municipalities that did not exist before the Project. By demonstrating the viability of decentralized financing, the SDP is seen to have contributed substantially to the ongoing dialogue and planning for education decentralization. SDP planning and implementation provided for the first time an opportunity of close and regular communication between schools and sub-provincial MoNE Directorates. Furthermore, because the SDP grants provided the Directorates with a tool to proactively respond to some of the schools’ needs, schools’ confidence towards them was enhanced.

46. At the same time, several other factors limited the Program’s effectiveness:

Targeting did not always identify the neediest schools. Sub-provinces with less than 90 percent primary enrollment rates were selected for SDP support. Grants were targeted to all primary and secondary schools in these sub-provinces. Of the 3,671 schools that received school grants 3,513 were primary and pre-primary schools. This targeting approach was based on then existing data and had the advantage of simplifying criteria and administration, but not necessarily of reaching the schools with the greatest needs. The additional sub-provinces benefitting from the expanded SDP had enrollment rates above 90 percent. MoNE also found that the reported enrollment rates were not always accurate, due to unrecorded migration of families to other districts and, hence, in some cases the real enrollment rates in beneficiary sub-provinces were closer to 100 percent. Also, within sub-provinces, high variation was found in schools’ learning conditions and the severity of need. Participatory processes and local community decision-making, both innovative elements of the SDP, turned out to be more limited than expected. The Beneficiary Assessment results suggest that the participatory process and decision-making role of local stakeholders was weaker than expected. Participation of students, parents, and local community members appears to have been relatively low compared to that of school principals, Parent Teacher Associations (PTA), and teachers. More than 80 percent of respondents indicated that local community members had no saying in final decisions.

18 The SDP Beneficiary Assessment sample consisted of 150 beneficiaries in 11 districts and 50 schools; face-to-face interviews with 73 beneficiaries in 2 districts and 6 schools.

23

Although survey results indicated that PTAs had an active decision-making role, fieldwork evidence did not support this conclusion. Mixed beneficiary satisfaction with the use of grant funds. The Beneficiary Assessment indicates that 80 percent of total grant funds were spent for the purchase of equipment, namely computers, copying machines, and projectors and for minor school improvements. Schools visited as part of the assessment were found to have invested heavily (up to 20-50 percent of total grant funds) in seminars, special tutorials, publications and materials related to improvements of students’ scores in the Seviye Belirleme Sinavi (SBS), a central examination used as an entrance exam for secondary schools, administered to 8th grade students. While beneficiaries were generally satisfied with the use of grant funds for equipment and recreational facilities, study materials and tutoring services for the SBS are more controversial. On the one hand, these materials were viewed to create equal opportunities for particularly disadvantaged students, but on the other hand they were perceived to have little or no contribution towards enhanced student performance. This perception is supported by the findings of a study funded under SEP, where stakeholders (including administrators, teachers, parents and students) identified weaknesses with the SBS, including an excessive focus on test-taking skills, a perception that the test provides only partial feedback, and its low contribution to raising the quality of education (see Annex 2).

Lack of a strong M&E framework and structure from the outset reduced the value of the program as a pilot, limiting both the ability to learn from lessons about process and the availability of information on the outcomes of school projects. Although the level of beneficiaries’ satisfaction is an important indicator, it does not itself provide any measurement of the extent to which conditions for student learning were improved. The Ministry of Development indicated that at the time of SDP design it had requested that an impact assessment be included in the sub-component, to complement the findings of the Beneficiary Assessment with more objective data. However, such assessment was never incorporated. The Beneficiary Assessment itself points to monitoring deficiencies, such as lack of a regular and systematic reporting both at the sub-province and school level, and lack of well-functioning community monitoring and accountability mechanisms. The fact that SDP financed items such as volleyball and basketball courts and other smaller items that were not included in the Loan Agreement list of small civil works seems to suggest MoNE’s own difficulty to fully monitor the use of SDP funds.

47. Achievement of the last PDO is rated Moderately Unsatisfactory. Under the last PDO, the pace of implementation was positively reversed, resulting in the disbursement of about Euro 62 million in two years and in the achievement of most revised key performance and intermediate outcome targets. However, due to lack of evidence of improved conditions of student learning - neither one of the indicators being a sufficiently good measurement of this objective – modest relevance, and no systematic evidence of efficiency gains (see Section 3.3), this PDO is rated Moderately Unsatisfactory.

3.3 Efficiency 48. Efficiency is rated negligible, based on short evidence of the benefits attributable to the Project compounded by its long implementation period. At appraisal, the economic justification of the Project focused on improvements in productivity, competitiveness and equity of earnings through reform of secondary education. No Internal Rate of Return was estimated. With the evolution of objectives, components and targets through successive restructurings, project focus narrowed to improvements in learning conditions in both secondary and primary schools. The Project’s fragmented and evolving structure, combined with limited evaluation

24

studies, make it difficult to identify benefits directly attributable to the Project or assess impacts of the investment. 49. The components in initial project design were not well integrated across MoNE departments and they were implemented in relative isolation, without exploiting potential synergies and efficiencies across the program. Major sources of inefficiency during implementation included a lengthy implementation period – 5 years from approval, only 19 percent of allocated funds had been disbursed – and the time and resources allocated to components and procurements that were eventually dropped, in particular the large and complex components for foreign language training and ICT. Cost effectiveness in implementation is obscured by the frequent changes in components and targets and the lack of information tracking unit costs through project monitoring and reporting. For example, at a total cost of Euro 32 million, the SDP disbursed grants for some 3,700 school projects. Anecdotal evidence suggests that procurement under these grants was efficient because incentives were strong at the local level to make the most of the funds. However, no systematic monitoring or analysis of costs was conducted to support this. Based on comments in the SDP Beneficiary Assessment report, efficiency of the program may have been compromised to some extent by deficiencies in training local government and school staff in the new procedures. 50. From a fiscal perspective, SEP had negligible impact on sector recurrent costs. No new institutions or school enrollment capacity were created, with the exception of the Career Information System. 3.4 Justification of Overall Outcome Rating 51. Because of the large amounts disbursed under the last objective, the outcome ratings weighted in proportion of the share of loan disbursements under each PDO would result in an overall project rating of Moderately Unsatisfactory. The ICR, however, rates the overall outcome of the Project Unsatisfactory (see Table 3) for the reasons outlined below. SEP’s positive contributions include:

(i) Independent assessments indicating that curricula revised by MoNE were in closer alignment with international standards. In addition, the Beneficiary Assessment suggests an overall high level of satisfaction with the revised curricula, including positive opinions on the possibility to integrate ICT into teaching and learning processes, apply diverse methods, and link between teaching and assessment. (ii) SDP beneficiaries’ perception of improved school learning environment as a result of school grants. Most areas of activity funded through the SDP are perceived to have improved schools’ learning environment. More specifically, the Beneficiary Assessment suggests that SDP is largely perceived as a tool to overcome physical deficiencies of schools and, although to a lesser extent, as having increased the satisfaction of teachers towards their working environment and the autonomy of schools in identifying and fulfilling their needs. The majority of survey respondents also agreed that the SDP contributed to the improvement of students’ social behavior and to their satisfaction with their schools.

52. Notwithstanding these contributions, several factors undermined the overall outcome of the Project:

(i) No sufficient and measurable evidence of PDO achievement. While the very limited implementation under the first two PDOs resulted in no tangible outcomes, the absence of an adequate M&E framework makes it difficult to measure the extent to which the last PDO was

25

achieved. Though an increased alignment of revised curricula with international standards and beneficiaries’ views on SDP are important indicators, they do not themselves provide any objective measurement of the extent to which improved conditions for student learning were achieved. (ii) Decreasing relevance and outcome-focus of the PDO. As noted earlier, the two Level I restructurings resulted in the introduction of large sub-components that altered project design and resulted in output-oriented and overly narrow objectives whose relevance was significantly reduced (see Section 3.1). (iii) Frequent, substantial changes in design and objectives led to a lack of focus and effectiveness. SEP’s final design is substantially different from the original one. Over the course of implementation, the emphasis of the PDO shifted from improving “quality, economic relevance and equity of secondary education to support lifelong learning” to supporting “improvements of conditions for student learning in secondary education and in schools in sub-provinces with low enrollment rates”. Similarly, the focus of related outcome indicators changed from targeting improved student performance, as measured by national and international assessments, to curricula being more aligned to international standards.

(iv) Overall limited contributions to reform. Although SEP originated to support Turkey’s secondary education reform, it did not contribute to any significant reforms beyond the development of revised curricula. Similarly, although the SDP is seen to have contributed substantially to the ongoing dialogue on school decentralization, this remains a medium to long-term agenda, with prospects that are difficult to determine at this stage. (v) Low efficiency. As noted earlier, design and implementation factors resulted in negligible efficiency (see Section 3.3).

Table 3: Overall Outcome Rating19 20

Disbursement figures source: Client Connection (December 17, 2012).

19 Because project objectives were formally revised, project outcome was assessed against both the original and revised objectives. Separate outcome ratings were weighted in proportion of the share of loan disbursements made in the periods before and after the approval of the revisions. Outcome ratings comprise an assessment of PDOs relevance, extent of PDO achievement, and project efficiency. 20 Highly Satisfactory = 6; Satisfactory = 5; Moderately Satisfactory = 4; Moderately Unsatisfactory = 3; Unsatisfactory = 2; Highly Unsatisfactory = 1.

Original PDO (March 2005 - March 2008)

Revised PDO (March 2008-April 2010)

Revised PDO (April 2010-April 2012) Overall

Rating Value 1 2 3

Weighted value 0.02 0.33 2.32 2.68

UnsatisfactoryRating

Disbursed amount under each PDO (Euro) 1,585,125 13,385,102 61,981,649

Highly Unsatisfactory UnsatisfactoryModerately Unsatisfactory

76,951,875Weight (% disbursed before PDO change) 2% 17% 77% 96%

26

3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 53. A social assessment was planned during project preparation, but was never carried out. However, informal consultations and discussions with stakeholders and beneficiaries, including focus group discussions with parents, students, teachers, employers, NGOs, community leaders, and community members took place. The social analysis was also informed by a nearly simultaneous Labor Market study on Turkey, a Knowledge Economy Assessment, and a Poverty Assessment.21 54. SEP did not have any visible poverty, gender, or social development impact, though the PAD puts great emphasis on social and gender dimensions (see Section 1.4). Equity is also explicitly addressed in the PAD and was discussed extensively at the Decision Meeting. The 2008 Project Paper also points out that changes proposed for the Project were not expected to have an impact of the Project's original social aspects. However, project activities did not specifically target disadvantaged groups, nor were these issues followed through during implementation. (b) Institutional Change/Strengthening 55. SEP was implemented by a Project Coordination Center that was an integral part of MoNE and almost entirely staffed by civil servants. In this respect, institutional strengthening in the areas of project management, procurement, and financial management occurred within MoNE. At the same time, PCC dismantling at project closing resulted in the dispersion of such capacity. While experiences of former PCC staff remain within the Government, the cumulative knowledge and capacity of the PCC as an entity are no longer accessible. 56. Although not an explicit project objective, an important outcome of the SDP was expected to be the capacity built at the school and sub-provincial levels – in participatory development of school plans, budgeting, and procurement. However, SDP beneficiaries indicated that even though the SDP encouraged schools to determine their own needs and develop plans accordingly, control exercised by MoNE sub-provincial directorates over the tendering process limited school autonomy and contributed to an overall perception of low transparency. Moreover, the SDP Beneficiary Assessment found that participation of students and parents in the decision-making for grant allocation was low, and significantly more so for local community representatives.22

(c) Other Unintended Outcomes and Impacts (positive or negative) 57. No other unintended outcomes and impacts were observed.

21 World Bank Report No. 33254-TR Turkey Labor Market Study, April 2006; Turkey Knowledge Economy Assessment Study, March 2004; World Bank Report 29619-TU Turkey Joint Poverty Assessment Report, August 2005. 22 According to the SDP Beneficiary Assessment, PTAs had an active role within mechanisms of decision-making. According to findings from the fieldwork, however, PTAs were excluded from decision-making mechanisms.

http://siteresources.worldbank.org/INTTURKEY/Resources/361616-1144320150009/Labor_Study.pdf

http://www.bilgitoplumu.gov.tr/Documents/5/Documents/040300_TurkeyKnowledgeEconomyAssessmentStudy.pdf

http://www.bilgitoplumu.gov.tr/Documents/5/Documents/040300_TurkeyKnowledgeEconomyAssessmentStudy.pdf

http://issuu.com/worldbankturkeyoffice/docs/joint_poverty_assesment_2005_v1

http://issuu.com/worldbankturkeyoffice/docs/joint_poverty_assesment_2005_v1

27

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 58. No beneficiary survey or stakeholder workshops were conducted as part of the ICR. MoNE organized a closing conference for the Project in June 2012, and the ICR draws from the discussions held as part of it.

4. Assessment of Risk to Development Outcome 59. The risk to the development outcome is rated Substantial. Whether and to what extent curricula reform and school development grants, through school planning autonomy, contributed to improved learning conditions cannot be concluded with the evidence available from the SEP experience. While curricula revisions are likely to be sustained at regular intervals through government funding, and there are indications that school decentralization has become an increasingly important item in MoNE’s agenda, the latter remains a contentious topic that will require high-level political commitment and that will likely not be implemented in the short-run. As a result of limited evidence on PDO achievement and uncertainties over the future and timing of school decentralization and financial autonomy in Turkey, the risk to development outcome is rated Substantial.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry 60. Bank performance in ensuring quality at entry is rated Unsatisfactory. Although the original focus of the Project was relevant to Turkey’s context and consistent with the general priorities outlined in the 8th National Development Plan, several shortcomings at the design stage significantly limited the effectiveness of the operation: (i) lack of client ownership; (ii) lack of a well-defined and broadly owned secondary education strategy; (iii) overly ambitious PDO and results framework; (iv) excessively long and costly project preparation; (v) only partial reflection of lessons from earlier projects; and (vi) inadequate anticipation of some key risks and relevant mitigation measures. (b) Quality of Supervision 61. Bank performance at supervision is rated Unsatisfactory. 62. The ICR recognizes the Bank’s positive contributions to supporting project implementation, including:

(i) Contributing to a positive reversal in project performance during the last two years of implementation, resulting in the disbursement of 77 percent of loan funds and in the achievement of most of the Project’s revised targets. These include: (a) revision of 66 curricula for general secondary education, and implementation of 62 of them; (b) development of 1,151 modules for vocational education and implementation of 924 of them, against a target of 1,000; and (c) provision of school grants to 3,671 schools in 60 sub-provinces. Notwithstanding the design issues associated with it, the SDP, introduced as part of the 2010 restructuring, represented a turning point in establishing greater client ownership of the Project, and in improving the pace of disbursements through the use of school grants.

28

(ii) Regular supervision of fiduciary aspects by a fully decentralized team. As noted earlier, Procurement and Financial Management staff was entirely country-based. This allowed a more active involvement of the Bank in the resolution of complex procurement issues by assisting MoNE in identifying solutions that were in line with the Bank’s guidelines but that would minimize further delays and reputational risks for both the Client and the Bank. (iii) Quality and frequency of performance reporting increased over time. A total of 14 Implementation Status and Results reports (ISR) were prepared throughout the duration of the Project. Aside from the first two ISRs, which rated both PDO achievement and Implementation Progress as Satisfactory, overall performance ratings were generally realistic and agreed by Bank’s sector and country unit management. A mid-term review was planned in May 2008 but not formally carried out. Instead, intensive implementation support and review missions took place at the time of both Level I restructurings, in early 2008 and 2010.

63. At the same time, a number of aspects, which weighted heavily in the overall project performance, were not adequately addressed:

(i) The dialogue with the client focused, in large part, on short-term design adjustments, leaving little room to focus on the broader, national priorities for secondary education. The two restructurings were thus a missed opportunity for the Bank to recalibrate its support and remain relevant to Government policies on secondary education. Instead, the Bank focused on addressing short-term agendas that were not in line with national priorities, such as the teachers’ foreign language training, or on identifying fast-disbursing activities that heavily shifted the focus of the project, such as the School Development Program. In addition, reaching consensus on design changes and implementation issues involved long and difficult processes. Despite four restructurings and close follow-up in the field, SEP remained a Problem Project for 63 out of 82 months, corresponding to 77 percent of the entire implementation period. Although underlying issues were at times outside of its direct control, the Bank appears to have contributed to some of them. For example, SEP’s first restructuring proposal, stemming from MoNE’s request to reallocate a substantial amount of project funds to overseas teachers’ training, should have been resolved prior to submitting SEP for approval. Instead the Bank, under pressures to deliver the operation within a given fiscal year following a long preparation period, appears to have oversimplified the resolution of the issue by reassuring MoNE that the activity could have been easily added after approval. (ii) Repeated restructurings led to increasingly narrow results frameworks, which were much more short-term and output-oriented. Significant scaling-back of M&E throughout implementation was a deliberate choice but also one that was dictated by the decreasing time available for implementation. Nonetheless, the resulting indicators were output-oriented and difficult to measure, thus providing limited information on the Project’s overall results. The ability to measure project results was further impaired by the exclusion of virtually all key studies and assessments from the original project design, including failing to conduct a follow-up independent assessment on an important activity such as curriculum reform. Additional implications of this decision were a missed opportunity to support relevant capacity and M&E culture in MoNE, and to provide information on the effectiveness of different project activities and approaches that would have been of value to MoNE in planning future programs. (iii) The Bank was not well placed to resist internal and external pressures to proceed with project activities that were not grounded in best practice and linked to national priorities. For example, despite its technical concerns on the Training for Foreign Language Teachers sub-component, the Bank ultimately agreed to include the activity in the Project.

29

Apart from a compromise reached with MoNE to evaluate the first phase of the training to inform the design of the second phase, no new technical evidence or international experiences had been brought forward to address the underlying concerns.

(c) Justification of Rating for Overall Bank Performance 64. Overall Bank performance is rated Unsatisfactory based on the Unsatisfactory rating for both Quality at Entry and Quality of Supervision.

5.2 Borrower Performance (a) Government Performance 65. Government performance is rated Moderately Unsatisfactory. The rating reflects several factors, including: (i) the fact that despite a long preparation time, the original Project was submitted for approval with low ownership on MoNE’s side, and with a lack of a well-defined and broadly owned secondary education strategy; (ii) a lack of intervention over a progressive decline in M&E focus following the cancellation of key impact studies and surveys, including a weak M&E framework for the SDP; and (iii) positive contributions to the project dialogue throughout implementation on the one hand, but its inability to assist in resolving issues in a timely manner, resulting in very limited implementation over a five-year period. The rating does recognize the fact that Bank’s communication on initial design issues was not sufficiently clear and resulted in the Project being submitted for approval without a full resolution to MoNE’s requests. (b) Implementing Agency or Agencies Performance 66. The performance of MoNE and the Project Coordination Center (PCC) is rated Satisfactory during the last two years of the Project’s life, in recognition of fast implementation, particularly with respect to the School Development Program, and Moderately Unsatisfactory overall. The rating recognizes the fact that the Project was prepared and approved with limited ownership by MoNE, which was initially perceived as being supply-driven. As is the case for Government performance, the implementing agency rating also recognizes the fact that the Project was submitted for approval with unresolved design issues, and without sufficiently clear and transparent communication from the Bank on how to best address them. MoNE’s overall achievements in secondary education, reflected in increased enrollments, improved student performance, and reduced performance inequality are also noted (see Figures 1, 2, and 3). At the same time, with respect to the Project, MoNE was found to have contributed to delays, some of which were the result of weak procurement practices and frequently changing priorities at the political and decision-making levels. Although Fiduciary capacity fluctuated throughout implementation, the PCC’s capacity was generally adequate, reporting of project progress was done through comprehensive and annual progress reports, and the PCC played a pivotal role in the significant implementation turnaround towards the last stages of implementation. However, as noted by QAG, the effectiveness of PCC staff was constrained by weak accountability mechanisms and complex management systems within MoNE, particularly during the earlier stages. In general, the costly and lengthy design process, weak implementation reflected in less than 20 percent of loan funds disbursed during the first five years, and a progressive weakening of the Project Development Objective contribute to an overall Moderately Unsatisfactory rating.

30

(c) Justification of Rating for Overall Borrower Performance Overall Borrower performance is rated Moderately Unsatisfactory based on the Moderately Unsatisfactory ratings for both Government and Implementing Agency performance.

6. Lessons Learned General (i) Ownership of the implementing agency remains the single most important factor in determining the success of a project. Clearance and support from central Government agencies are a necessary but not sufficient condition for success. The initial dialogue on project design appears to have been largely driven by the World Bank, albeit with the support of Treasury and the Ministry of Development. As a result, SEP was long perceived by MoNE as a supply-driven project without clear ownership. (ii) A significantly flawed initial design is difficult and costly to correct during the course of implementation, despite instruments available to restructure operations. Such tools are best suited to adjust project designs which are based on solid premises and ownership, which was not the case for SEP. (iii) Better understanding and increased awareness of the political economy surrounding a specific program or operation is likely to positively affect its ultimate outcome. Specific factors that the Bank should be more attentive to include the differing roles and incentives driving institutions within the Government and their differing perceptions and rationale for working with the Bank, as well as a thorough understanding of sector developments and of the national policies driving them.

(iv) An investment project approach may not be the most effective to support a program’s roll-out at the national level. Such approach has value-added in piloting new programs or policies, as was the case in the original project design with respect to curriculum reform, or in cases where the Government sees value in the Bank’s technical and/or financial support of national programs. In the case of SEP, when the pilot approach to curriculum reform was no longer relevant because MoNE had decided to roll-out the program nationwide, the Bank was not able to support and add sufficient value to MoNE’s scaling-up of its secondary education programs, aside from the financing of curricula revisions which the Government would have likely carried out even in the absence of SEP.23 In such circumstances, the use of a results-based operation would seem more appropriate.

(v) From a Government perspective, carving out a relevant role for the Bank in a sophisticated country needs careful discussion of where the Bank’s value-added is, and how it can be maximized. In the case of SEP, this does not seem to have been the case, with Bank support being scattered across various and often changing activities, with no evidence that its technical and international knowledge, or availability of financing instrument options, was used to maximum effectiveness and efficiency.

23 Outside of the Project’s scope, MoNE carried out the revision and implementation of about 6,200 vocational secondary education modules, in addition to the 1,151 financed under SEP.

31

(vi) Assessing the outcomes of project activities should be at the core of each Bank-financed operation, because of the implications that such findings have on beneficiaries, but also on the implementing agency and government at large, by providing information on the effectiveness of different project activities and approaches to inform future policies and programs. In the case of SEP, attention towards the quality of M&E and the overall outcome-focus of the Project decreased overtime, hampering the ability to assess the ultimate benefits of most activities on the beneficiaries.

(vii) Remaining engaged in non-performing operations should be carefully evaluated against longer-term implications. Decisions to remain engaged in a given sector may provide an opportunity for the Bank (and the client) to broaden their engagement. However, while in the short term this approach may pay-off, the medium to long-term implications of extending a non-performing operation should be carefully evaluated. In the case of SEP, a negative perception towards Bank procedures has developed at some levels of MoNE, and was cited as one factor contributing to the Ministry’s decision to limit its engagement with the Bank on the lending front.

School Development Program (i) The specific needs for improving learning conditions vary greatly between schools and can best be identified at the local level. At the outset, the SDP Operational Manual specified the proportion of each grant to be allocated between teaching needs and goods and it precluded some types of expenditure. After finding in the course of implementation that real priorities varied greatly between schools, more discretion was given to the school in the revised Operational Manual for subsequent projects. (ii) The SDP experience supports the case for decentralization of school financing in Turkey, which should be high on MoNE’s agenda, but with some important adjustments. (iii) Better criteria are needed for identifying the schools with the greatest needs. Because of SDP design, resources were given indistinctly to all schools in a given sub-province, without the possibility to single-out those with greatest needs. Vocational Training Initiatives (i) Strong information and promotion activities are needed to ensure that calls for proposals reach beyond school administrations and existing project group members. In the case of VTI, the Beneficiary Assessment found that notifications did not reach all teachers, with the result that the same group of teachers tended to appear as beneficiaries in all projects and the opportunity for diversification through the VTI was hence limited. (ii) To achieve project ownership and capacity development at the school level, strong support and guidance is needed to schools in developing their own proposals. In the case of VTI, the Beneficiary Assessment observed that “participation in programs was generally guided by private companies with experience in project development. Because these companies generally act as service providers during the implementation phase of projects, they contact schools when a project call is published and undertake important responsibilities in determining the needs, as well as development and implementation of project.” This led to problems regarding ownership, unsuitable content and quality of projects, lack of sustainability, as well as lack of improvement in the project development capacity of the participating schools.

32

(iii) Stipulating partnerships with sector representatives or NGOs as a requirement in project proposals is ineffective in promoting such relationships unless the partner’s role and responsibilities are clarified and monitoring mechanisms are established. The Beneficiary Assessment found that, except where partnerships already existed, these tended to be treated as a formality for the sake of the proposal and that the role of partners was not significant.

(iv) Sustainability of project benefits depends on partnership between schools and industry. VTI beneficiaries highlighted that investments in equipment have great positive value initially, but this is lost as technology becomes outdated. In this context, they highlighted the importance of cooperation between school and industry, to provide vocational teachers and students access to the latest technology.

(v) Incentives and information dissemination are needed to spread the benefits of school projects beyond the individual institution. Formal evaluation of in-service trainings and broader recognition for professional development would increase the motivation for participation.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 67. The World Bank welcomes and appreciates the constructive and candid comments provided by MoNE, MoD, and the Undersecretariat of Treasury, as well as their support throughout the preparation of the ICR. In recognition of the remarkable efforts of MoNE and the Project Coordination Center during the last two years of implementation, and in response to comments received, the Project counterparts’ performance is explicitly recognized as being Satisfactory for that phase of the Project. (b) Cofinanciers 68. There was no co-financing with other international financial organizations. (c) Other partners and stakeholders

Not applicable.

33

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in Euro Million equivalent)

Components Appraisal Estimate (Euro millions)

Actual/Latest Estimate (Euro

millions)

Percentage of Appraisal

1. Secondary Education Reform 64.70 35.50 55% 2. Quality Assessment 5.47 3.76 69% 3. School Grants n.a. 34.62 n.a. 4. Project Management,

Monitoring and Evaluation n.a. 3.07 n.a.

Total Baseline Cost 70.17 76.95 110% Physical Contingencies 7.77 0.00 0.00 Price Contingencies 2.06 0.00 0.00

Total Project Costs 80.00 76.95 96% Front-end fee PPF 0.00 0.00 0.00 Front-end fee IBRD 0.80 0.80 100%

Total Financing Required 80.80 77.75 96%

(b) Financing

Source of Funds Type of Cofinancing

Appraisal Estimate

(Euro millions)

Actual/Latest Estimate

(Euro millions)

Percentage of Appraisal

Borrower 0.00 0.00 0.00 International Bank for Reconstruction and Development 80.00 76.95 96%

34

Annex 2. Outputs by Component24 Component 1: Secondary Education Reform Sub-component 1.1: Curricula Reform 1. Objective The objective of this sub-component was to revise and implement new general education and vocational education curriculum programs to provide students with core skills for the knowledge economy and lifelong learning. This objective remained unchanged throughout project implementation. At the time of project approval, primary education curricula and textbooks had already been revised and introduction of similar changes at the general secondary and vocational education levels was a high priority. 2. Achievement of the final Intermediate Outcome Indicators Intermediate Outcome Indicator: Curricula reform (including at least 1000 modules for vocational education), textbooks, and other educational materials and equipment have been developed and implemented in general secondary schools and for at least 15 occupations in vocational education. • 66 curricula in general secondary education were revised, and 62 of them were implemented;

1,151 modules were developed, and 924 of them were implemented under 15 areas for Vocational Secondary Education institutions, against a target of 1,000.

• About 50 textbooks were finalized by the end of the project, and others were under preparation. • A number of workshops were organized to coordinate curricula development activities in 10

pilot provinces. About 20,000 teachers were provided with in-service training for curriculum implementation through 186 seminars/workshops (see Table A below).

Table A: Seminars and Workshops Held under Subcomponent 1a No. of

Seminars No. of

Participants Teaching Program and Textbook Introduction Seminars 49 1,165 Education Management Seminars for School Principals 20 3,518 Meetings Held in the Textbook Writing Process 8 1,238 Seminars held for 81 Provincial National Education Directors 1 100 Seminars Organized by SEDG under SEP 108 14,478 Total 186 20,499 Source: MoNE’s SEP Closing Report

24 Component objectives, content, and intermediate outcome indicators reflect the latest/revised versions approved through the respective restructuring processes.

35

Sub-component 1.2: Information and Communications Technology (ICT) 1. Objective The objective of this sub-component was to provide ICT training as a core competency for youth to function in a modern knowledge economy, to improve the medium of instruction by using ICT, to expand access to learning opportunities and support better management of learning and administrative processes. 2. Achievement of the final Intermediate Outcome Indicators Intermediate Outcome Indicator: Teachers in at least 1,600 secondary education schools that previously had no ICT equipment use ICT in their classroom teaching. • ICT equipment worth Euro 5.3 million, namely projectors and photocopiers, was provided to

1,629 schools. The scope of work under this sub-component was significantly reduced during restructuring. Initially, the sub-component targeted installation of 322 ICT classrooms, procurement of multi-media software and materials for e-learning, installation of computer assisted science labs, monitoring integration of ICT technologies in the classrooms, extension of e-learning and in-service training for the use of ICT classrooms. Throughout implementation, a large share of the ICT contract envisaged under the Project was cancelled because of outdated specifications and differences between national procurement and World Bank procurement procedures.

Sub-component 1.3: Career Guidance and Counseling. 1. Objective The objective of this sub-component was to provide youth and adults with access to career information and guidance services to promote informed career choices and related education and training decisions and facilitate lifelong learning. This objective was also part of the original project design and remained unchanged during implementation. 2. Achievement of Intermediate Results Indicators: Intermediate Outcome Indicator: New career guidance and counseling system used by at least 300,000 students in secondary education. • As part of SEP, a Memorandum of Understanding was signed with Turkish Employment

Agency (ISKUR), Ministry of Development, Council of Higher Education and other respective institutions.

• In close partnership with these agencies, a new web-based Career Information System (CIS) was developed and became operational in April 2010.

• By project closure, the site (http://mbs.meb.gov.tr) had 1.3 million registered users.

36

Component 2: Quality Assessment Sub-component 2.1: Student Assessment 1. Objective The objective was to improve the quality and outcomes of secondary education by developing systems to collect and disseminate reliable information on student learning and outcomes, and on the performance of education system staff and institutions. 2. Achievement of the final Intermediate Outcome Indicators Intermediate Outcome Indicator: Student achievement results reported and shared with stakeholders and public. The activities under this the sub-component focused on two areas: (i) piloting of a national assessment of student outcomes; and (ii) carrying out research in the education sector. Piloting of Student Achievement Assessment Exam (OBBS) • SEP supported design of measurement tools for the OBBS, as part of which 13 academicians

were involved in the review of questions for Turkish Language and Literature, Mathematics, Science, Social Sciences and English courses. The new measurement tools were piloted in 20 schools in Ankara between May 26 and May 30, 2008.

• SEP also supported the piloting of OBBS at the secondary education level. OBBS was launched in 2002 under the Bank-financed Basic Education Project, covering primary education. Secondary education was included for the first time in 2009 under SEP. The test covered five areas: (i) Turkish Language and Literature; (ii) Mathematics; (iii) Natural Science; (iv) Social Sciences; and (v) English. The test was piloted on May 28, 2009. It was administered in 300 schools across Turkey, including 210 general secondary education schools and 90 vocational high schools in 30 provinces from 7 regions. A total of 140,000 students from grade 9 and 10 took the test.

• OBBS assessment results provided a picture of student knowledge and skills across the country. The results, however, were published and disseminated only in 2011, after significant delay. Dissemination took place through two workshops organized in Ankara in 2011 for Ministry representatives, academicians, provincial and sub-provincial staff of the Ministry. The following were gathered based on the assessment results: (i) one of the factors influencing learning difficulties for grade 9 and 10 students is the weak education received by these students in earlier grades; (ii) Females in grade 9 outperform males in Turkish, Social Sciences, Sciences and English; but no significant difference was found between male and female achievement rates in Math. In grade 10, absolute achievement of female students in Turkish and English tests is higher than that of male students.

• MoNE plans to conduct future assessments every three years through new computer-adapted tests. Nonetheless, at this stage, the ICR team could not find evidence on the application of assessment results for policy-making.

Education sector research The Project also supported implementation of nine research studies commissioned at the request of the Educational Research and Development Division (ERDD) of MONE looking at different aspects of secondary and vocational education. The results of these studies were widely disseminated and discussed with public in various workshops.

37

• The following studies were carried out: (i) 21st Century Student Profile of MONE. The purpose of the research was to obtain

information to what extent current student characteristics are compatible with the desired 21st century student profile. The results of the study were disseminated in a workshop organized on April 14-15, 2010.

(ii) Review of Class Applications for Common Grade 9 and Transfers between Schools at Secondary Education (vocational and technical education) level. Findings from this study suggest that students and parents are not adequately informed about the objective, scope and rules of the joint 9th class and school transition practices. The content and hours of intensive classroom course were not found suitable given the heavy curriculum load in grade 9; the promotion and referral services were also found as not sufficiently effective.

(iii) Expectations of the Sector from Vocational and Technical Education Schools for Girls. The objective of this study was to determine whether the education and training in the field of food technologies, offered by the schools affiliated with Directorate General of Technical Education for Girls, provide students with desired knowledge and skills. The findings suggest that professional practice, laboratories and lab equipment, and learning materials need to be improved. It was also suggested that teachers who work in the field of food technology should have university diploma in the area of food technology teaching.

(iv) Assessment Study on the Relations between Non-governmental Organizations and Provincial and Sub-provincial National Education Directorates. According to the study results, directors of provincial and district national education directorates have more positive views than the directors of NGOs. The main difference was about the use of physical facilities of schools and other educational institutions for educational activities.

(v) Assessment Study on Contribution of the Trade and Tourism Schools Serving the Needs. According to the study, teachers and administrators have theoretical knowledge but lack practical skills. Graduates and last grade students seem to be satisfied with their schools and changes in curricula affect education of students. Nonetheless, departments at schools do not meet the need for intermediate labor force of their areas: students are facing problems in using tools and equipment at enterprises since the schools are not equipped with the latest technological devices; most of the employed graduates are working in other fields rather than their own field of specification. Graduates do not meet the expectations of the enterprises.

(vi) Assessment Study on Contribution of Common Cultural Lessons into Vocational and Technical Education. The objective of this research was to identify the extent to which common courses contribute to vocational and technical education and to present suggestions for solutions to raise the level of contribution. According to the research, while common courses contribute to general culture and communication skills, they were not found to sufficiently support area/branch courses. The research suggested that common courses could be formulated more in line with the characteristics of the vocational area.

(vii) Assessment Study on Branch Preferences of Secondary Education Students. The objective of the research was to determine the factors that influence secondary schools students’ branch preferences. Results revealed that teachers out of branch should not be employed in the guide services of the schools and students and their parents should be provided with sufficient information about branches and professions. The results of this survey reveal the need for students to be encouraged to choose their branches according to their interests and capacities by means of an effective guidance.

(viii) Assessment Study on Level Determination Exam (SBS). This research was intended to evaluate the economic, sociological, psychological and physical impacts of SBS as well as its overall impact on academic achievement, school achievement and teaching

38

processes, and to analyze the contents and structure of SBS and the existing and potential models in transition to secondary education. The research found that SBS affects families financially by creating increased dependence on extra-school resources such as private tutoring schools and private lessons. While students were found more eager to study due to SBS, the test is viewed as the source of an exam-focused structure in the education system, including teaching methods. Furthermore, the research found that SBS results cannot be considered a sufficient indicator of academic achievement, and cannot be used for measuring the level of academic qualifications, due its focus on instantaneous performance on cognitive learning and basic skills. As a result, the majority of respondents suggested that SBS be abolished, and that arrangements be made to enable an open admission system in the long term.

(ix) Evaluation of Secondary Education in terms of Learning Period. The objective of this research was to evaluate the implications of raising the duration of secondary education to four years for schools, programs, students and parents. While teachers thought that increasing the duration of secondary education to four years made limited contributions to education, students and parents viewed the four-year secondary education as a change that improves the quality of the education process.

Sub component 2.2: School Development 1. Objective The objective of the School Development sub-component was to help secondary schools improve their own performance, their students’ educational attainment, and quality in education through school development activities to be designed and implemented within the school development. 2. Achievement of the final Intermediate Outcome Indicators Intermediate Outcome Indicator: School development management teams, comprised of teachers and school administrators, established in all secondary education schools and prepare school development plans in line with strategic management guidelines. • The Project supported creation of School Development Management Teams in all secondary

education schools, and 3,689 primary schools. As part of the process, 15,000 educational staff were trained in school plan preparation.

• The Project also supported design of a performance management system to evaluate the performance of teachers, school managers and civil servants of MONE. In this regard, the General Directorate of the Teacher Training and Education (GDTTE) set out the general field proficiencies for secondary-school teachers in general as well as in all special fields, with the exception of the occupational and technical fields. In addition, the general field proficiencies for managers, together with special field proficiencies of branch managers working in provincial organizations and the principals, head teachers and vice-principals of pre-school, primary, secondary and occupational/technical educational institutions have been set out by the General Directorate of Personnel. Finally, the Inception Board set standards for provincial organization branch directorates, preschool, primary, secondary and Occupational/Technical educational institutions. SEP supported the piloting of the performance management system in education in 14 provinces (Kayseri, Sincan, Kirikkale, Kirsehir, Hatay, Osmaniye, Nigde, Bartin, Karabuk, Aksaray, Nevsehir, Yozgat, Bilecik and Bursa) during September and October, 2011. At the time of this ICR, it is not yet possible to evaluate the outcomes of this sub-component. The pilot of the performance evaluation system is still ongoing and has not been finalized at the time of the ICR.

39

Component 3: School Grants Sub-component 3.225: Vocational Training Initiatives 1. Objective Introduced to the SEP in the 2008 restructuring, this sub-component financed grants to vocational technical schools on a competitive basis, with the aim of promoting innovation, improving the quality of vocational secondary programs and their relevance to employment needs. To address concerns in particular that vocational schools needed to be more responsive to the needs of the labor market, the program would help improve the quality and economic relevance of vocational technical education programs by supporting innovative proposals -- by vocational schools in cooperation with other partners -- to improve the skills and qualifications of vocational teachers and improve the quality of the vocational training institutions. 2. Achievement of the final Intermediate Outcome Indicators Intermediate Outcome Indicator: Teacher training skills in selected vocational schools are adapted to modernized education technologies, teachers start implementing the new training methods, and teachers and school administrators are satisfied with the use of competitive school grants. • Forty-six project proposals were approved and implemented in 33 provinces. Training and

consultancies accounted for the largest share of disbursements which amounted in total to Euro 2.7 million.

• Of the allocated budget of Euro 3 million for this sub-component, Euro 2.75 million was disbursed, of which 66 percent was for training and consultancy services, 22 percent for “per diems” and 12 percent for goods. Teacher training – in foreign language and technical subjects – featured prominently.

Sub-component 3.3: School Development Program 1. Objective The rationale for the School Development Program (SDP), from the March 2010 Restructuring Paper, was the need to address the adverse impact of the economic crisis on schools in Turkey’s poorer districts. The objective of the sub-component was to provide grants to schools in sub-provinces with low enrollment rates in order to provide additional assistance to schools in areas likely to have been affected adversely by the economic downturn. 2. Achievement of the final Intermediate Outcome Indicators Intermediate Outcome Indicator: Use of school grants is consistent with the needs identified in the school development proposals. • The original target was for SDP to support 2,710 schools in 46 sub-provinces. As a result of the

popularity of the program and demand from other provinces, the Government requested an

25 Sub-component 3.1, Training for Foreign Language Teachers, was dropped as part of the April, 2010 restructuring.

40

expansion of the program, which eventually benefitted 3,671 schools, of which 158 were secondary schools, in 60 sub-provinces.

• A total of Euro 31.8 million was spent under this component, against an original allocation of Euro 24 million.

Component 4: Project Management, Monitoring and Evaluation Sub-component 4.1: Project Implementation Support 1. Objective The Project Implementation Support sub-component had the objective to support effective administration and coordination of the Project, including Project execution, managing the resources of the Project, procuring goods and services under the Project, operating the financial management system, and ensuring timely and appropriate reporting. This sub-component was also intended to finance public awareness activities through dissemination of the key messages and accomplishments of the Project 2. Achievement of the final Intermediate Outcome Indicators Intermediate Outcome Indicator: No intermediate outcome indicators were formulated specifically for this sub-component. • The Project Coordination Center functioned as the administrative and coordinating unit

throughout the Project, and was instrumental, particularly during the last two years of implementation, in ensuring fast implementation of the School Development Program. Under its leadership during this phase of the Project, 77 percent of total loan funds were disbursed.

• At project closing, MoNE organized a closing conference for the Project in June 2012, in which main lessons were drawn and shared among MoNE central and provincial-level departments and directorates, with the participation of consulting companies and academicians involved in Project activities.

Sub-component 4.2: Monitoring and Evaluation (M&E) 1. Objective The revised objective of the M&E sub-component was to support regular monitoring of the Project, evaluation of Project sub-components, and an evaluation of the overall impact of the Project on the quality and relevance of education. 2. Achievement of the final Intermediate Outcome Indicators Intermediate Outcome Indicator: Key project assessment studies carried out and results discussed. • Under the Project, the following Beneficiary Assessments were conducted: (i) Vocational

Training Initiatives; (ii) Curriculum Reform; and (iii) School Development Program.

• The results of these assessments were discussed within MoNE and with other relevant stakeholders. The SDP Beneficiary Assessment was completed in August, 2012 and has, at the time of the ICR, only been discussed within MoNE.

41

Table B: Revised Intermediate Outcome Indicators (from April 2010 Restructuring)

Component `1: Secondary Education Reform

INTERMEDIATE OUTCOMES Target Actual

1.1 Curricula Reform 1.2 Information and Communications Technology (ICT) 1.3 Career Guidance, Counseling

1. Curricula reform (including at least 1,000 modules for vocational education), textbooks, and other educational materials and equipment have been developed and implemented in general secondary schools and for at least 15 occupations in vocational education. 1. Teachers in at least 1,600 secondary education

schools that previously had no ICT equipment use ICT in their classroom teaching.

1. New career guidance and counseling system

used by at least 300,000 students in secondary education.

66 secondary education curricula revised, of which 62 implemented. 1,151 modules for vocational education were developed, of which 924 were implemented. 50 Textbooks revised No information on the provision of educational materials or equipment as part of Component 1.1. Equipment delivered to 1,629 schools. The web-based Career Information System has 1.3 million registered users.

Component 2: Quality Assessment Target Actual 2.1 Student Assessment 2.2 School Development

1. Student achievement results reported and shared with stakeholders and public.

1. School development management teams,

comprised of teachers and school administrators, are established in all secondary education schools and prepare school development plans in line with strategic management guidelines.

Pilot test of the Student Success Assessment Exam (ÖBBS) completed in 210 general secondary and 90 vocational schools by 140,000 students. The exam covered five areas: (i) Turkish Language and Literature; (ii) Mathematics (Math, Geometry); (iii) Hard Sciences; (iv) Social Sciences; and (v) English. School Development Management Teams established in all secondary education schools and 3,689 primary schools. 15,000 educational staff trained in school development plan preparation.

Component 3: School Grants Target Actual 3.2 Vocational Training Initiatives 3.3 School Development Program

1. Teacher training skills in selected vocational schools are adapted to modernized education technologies

2. Teachers start implementing the new training methods

3. Teachers and school administrators are satisfied with the use of competitive school grants.

1. Use of school grants is consistent with the needs identified in the school development proposals.

Vocational Training Initiatives implemented in 46 targeted Vocational schools, including training of teachers, and associated grants to implement VTIs. New training methods in place in the 46 targeted schools.

3,671 schools in 60 sub-provinces received school grants, in line with school development proposals.

Component 4: Project Management, M&E

Target Actual

4.1. Project Implementation Support 4.2. Monitoring and Evaluation

Key project assessment studies carried out and results discussed.

Project Progress Reports prepared and submitted to the Bank on a yearly basis. VTI, Curriculum Reform, and SDP Beneficiary Assessments carried out. Results discussed with MoNE and other relevant stakeholders (SDP Assessment was completed in August, 2012 and was discussed within MoNE).

42

Annex 3. Economic and Financial Analysis

The limited information on outcomes, combined with the Project’s evolving and fragmented structure, and the large scale of Turkey’s education sector in relation to the Project, make it difficult to assess or quantify the benefits of the SEP in relation to costs, or to discern an overall impact. In the circumstances, an economic analysis is not possible but some relevant comments are provided here. Economic rationale

The economic rationale of the SEP as presented in the PAD was “to better prepare Turkey’s future workers to participate and compete in an economy that is experiencing rapid technological change, demanding of higher levels of productivity, and becoming increasingly knowledge-based.” The reform of secondary education reform was also justified by the need to narrow the wage gap between different segments of the youth population entering the labor market. The economic analysis in the PAD endorsed the government’s policy of expanding access to secondary education and increasing the proportion of secondary pupils in vocational schools, and assessed some of the policy options for achieving expansion. The original Project Development Objective was to improve the quality, economic relevance and equity of secondary education to support lifelong learning. The project design, however, addressed issues related only to the quality and content of secondary education, not to the access or equity issues. No expected rate of return was estimated for the project.

In subsequent restructurings, the objectives shifted first to improving conditions for student learning in general and vocational secondary schools, and then to improving the conditions for student learning in primary and secondary schools in sub-provinces with low enrollment rates. The broader economic rationale under the first Level 1 restructuring was not made explicit. Under the second major restructuring, at the time of the economic crisis, there was an explicit equity rationale of targeting schools with low enrollment rates, based on the assumption that these schools were disproportionately affected by the crisis, but substantiating data were not provided. Expected Benefits

According to the original Results Framework, achievement of the project’s development objective would be demonstrated through better student and graduate performance and reduced regional variance in student achievement in the schools benefiting from the project’s interventions as compared with other schools. While it was probably overly ambitious to expect the benefits of the new programs to be reflected in national assessments and graduate tracer studies within the proposed time frame of the project (considering the time needed to develop and fully implement new curricula and related in-service training, and for graduates to have benefited from the full four-year program), in the event this was not tested. Project objectives and indicators were revised through the two restructurings, the new curricula and related project-financed activities were not focused in selected schools, and no comparison could be made of results in beneficiary and non-beneficiary schools.

The expected benefits under the restructured operation – improvements in conditions for student learning -- were not quantified or related to economic outcomes.

43

Actual Project Benefits

As reported in section 3.2 of the ICR, the information available on project achievements is limited largely to the outputs that were delivered. The most significant were: revision of general education curricula; revision of vocational education modules; and support to implementation of the new curricula through textbook development and teacher training; support to establishment of a career guidance system; trial of student assessment; grants for sub-projects to improve teaching quality and relevance of vocational school programs; grants for sub-projects to improve learning conditions in schools in low-enrollment districts; and studies/research to inform policy. Beneficiary Assessments indicated broad satisfaction of the beneficiaries of the training and grants as well as approval of the revised curricula. No assessments were available to provide an indication of the expected educational, economic or social outcomes. SEP in the Sector Context

A sector study conducted by the World Bank in 2005 26 identified a number of weaknesses in Turkey’s secondary education system constraining its effectiveness in supporting national economic and social development. Only 27 percent of the adult population had a complete secondary school education, compared with 65 percent in the EU, 74 percent in Korea, 82 percent in Poland, and 87 percent in the US. Unemployment rates for recent graduates were quite high: more than half of eighth-grade graduates and nearly a quarter of secondary school graduates were still looking for work in their early twenties. The performance of Turkish students on international tests was far below the OECD average. Variance in student performance between regions and schools was particularly high and was reflected in high inequality in the incomes of school leavers. The Government of Turkey had recognized that an inefficient education sector was constraining economic growth, competitiveness and social cohesion and responded with its Eighth Five Year Plan outlining an ambitious collection of national education objectives. Yet the existing array of laws, programs and actions did not add up to a comprehensive strategy for achieving these objectives.

The key challenges for improving equity and efficiency in the sector as well as the quality and relevance of secondary education programs, as identified in this study and in subsequent analytical work, 27 included: changing governance and financing arrangements to increase efficiency and equity in the allocation of public resources; moving from a selection-based system (which favors students whose families can afford private tutoring for entry exams to secondary and higher education) to a results or outcome-based model to support lifelong learning; modernizing general education curricula in line with international standards; increasing the relevance of vocational training curricula to changing employment needs in the modern sector; restructuring to facilitate flexible transfer of students between general and vocational tracks; improving the quality and management of teachers.

At the time of preparing this ICR, most of the above reform issues remained to be addressed and a comprehensive national strategy for the sector was still needed. Nevertheless, Turkey had made significant progress in expanding enrollments (see Figure 3) and in

26 Turkey—Education Sector Study, Sustainable Pathways to an Effective, Equitable, and Efficient Education System for Preschool through Secondary School Education, December 31, 2005, The World Bank, Human Development Sector Unit, Europe and Central Asia Region, In Association with Education Reform Initiative/ Istanbul Policy Center. 27 In particular: Improving the Quality and Equity of Basic Education in Turkey, Challenges and Options, World Bank 2011.

44

modernizing basic and – through the SEP -- secondary education curricula. Some headway was made in strengthening learning outcomes: the 2006 and 2009 PISA results for Turkey pointed to improvements in student performance and reduction in inequalities (see Figures 1 and 2), although still well below the OECD average. It is unlikely that the improvements in PISA results could owe much to SEP, when revised secondary curricula were only very recently being implemented. Figure 3: Trends in Secondary Education enrollments

Source: OECD Value added by SEP

In practice, the SEP picked up several elements of MoNE’s agenda, based on the respective priorities of different departments, rather than constituting in itself an integrated approach to meeting a specific objective. SEP funding was not additional to MoNE’s budget and it can be speculated that, in the absence of Bank funding, most of the items financed by SEP would have proceeded in similar fashion through other sources of funding. The question arises as to how much if any value was added by using the World Bank to finance these activities. The views expressed by MONE officials indicated some instances when SEP-funded activities benefited from technical inputs and support from the Bank, although this was balanced to a considerable extent by the costs and difficulties they experienced in meeting the Bank’s reporting and procurement requirements. The Bank’s input was particularly appreciated in design of the school grants program (SDP) which was an entirely new approach for Turkey and stimulated new confidence in the potential viability of decentralized financing of schools. Efficiency and Cost Effectiveness in Implementation

The components in initial project design were not well integrated across departments of MoNE and they were implemented in relative isolation, without exploiting potential synergies and efficiencies across the program. Major sources of inefficiency during implementation included lengthy delays – 5 years from approval, only 19 percent of allocated funds had been disbursed – and the time and resources allocated to components and procurements that were eventually dropped, in particular the large and complex components for foreign language training and ICT.

30 28

4146

5356

7376

81 81 82 83

0

10

20

30

40

50

60

70

80

90

1995 2000 2005 2008 2009 2010

Trends in Enrollment Rates of 15-19 year-olds (as a % of population in age group)

Turkey

OECD Average

45

Cost effectiveness in implementation is obscured by the frequent changes in components

and targets and the lack of information tracking unit costs through project monitoring and reporting. For example, at a total cost of Euro 32 million, the School Development Program disbursed grants for some 3,700 school projects. Anecdotal evidence from staff responsible for implementation suggests that procurement under these grants was efficient because incentives were strong at the local level to make the most of the funds; however, no systematic monitoring or analysis of costs was conducted to support this. Based on comments in the assessment report, efficiency of the program may have been compromised to some extent by deficiencies in training local government and school staff in the new procedures. Similarly, training and consultancies accounted for Euro 20.3 million and 26 percent of disbursements (see Table C). According to MoNE’s Closing Report, about 58,900 teachers and administrators benefitted from in-service training under SEP, at a unit cost of Euro 283. The Project also utilized 196 consultants (see Table D), at a unit cost of Euro 18,900 and at a man/month cost of about Euro 2,500. No information, however, was available from project documents or M&E assessments concerning the effectiveness or cost effectiveness of these activities, for example compared to in-service training or consultancy employed by MoNE outside of the Project scope. Table C: Project Costs by Disbursement Category

Source: Client Connection, as of December 18, 2012 Table D: Consultancy services employed under SEP

Source: MoNE’s SEP Closing Report

Goods 21,819,277.23Consultants' Services 3,703,064.39Training and In-Service Training 16,650,906.25Incremental Operating Costs 258,876.31Unallocated 0.00Grants for VTI 2,750,683.24Grants for SDP 31,769,067.44Designated Account 0.00TOTAL 76,951,874.86

Category Description Disbursed (Euro)

Components/Sub-Components No. of Consultants Man/Month Used

Secondary Education Programs 95 756ICT 1 3Training of Foreign Language Teachers 1 2Career Guidance and Counseling 13 167Evaluation of Student Achievements 26 142School Development 41 392

Project Implementation Support 19TOTAL 196 1,462

46

Fiscal Impact

The project had negligible impact on sector recurrent costs. No new institutions were created by SEP, with the exception of the career guidance system, and the Project did not contribute to expanding enrollment capacity.

47

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/ Specialty

Lending Ibrahim Akcayoglu Project Officer ECSHD Meral Gokcek Program Assistant ECCU6 Michael Mertaugh Consultant MNSHD Elmas Arisoy Manager EASR2 Evelyn Villatoro Sr. Procurement Specialist EASR1 Hong Chen Operations Officer EASSD Nora Dudwick ECSSD Marit Ketilsson Granheim ECSHD Ferda Sahmali Sr Operations Officer ECSHD Virginia Jackson Consultant MNSHD Selma Karaman Program Assistant ECCU6 Tobias Linden Lead Education Specialist SASED Ayse Seda Aroymak Sr Financial Management Specialist ECSO3 Jennifer Manghinang Temporary ECSHD John A. Innes Lead Operations Officer ECSHD Robin S. Horn Adviser HDNED Pasi Sahlberg Senior Education Specialist ECSHD Salih Kemal Kalyoncu Senior Procurement Specialist ECSO2 Robert Schware Lead Informatics Specialist TWICT David Fretwell Lead Employment and Training Sp. ECSHD Ernesto Cuadra Lead Education Specialist MNSHE Rosita Maria Van Meel Senior Education Specialist ECSHD Helen Shahriari Senior Social Scientist AFTCS

Supervision/ICR Ayse Seda Aroymak Sr Financial Management Specialist ECSO3 Dilek Barlas Deputy Executive Secretary IPN Furuzan Bilir Operations Officer ECCU6 Alexandru Crisan Consultant MNSHD Maria E. Gracheva Senior Operations Officer SASHN Salih Kemal Kalyoncu Senior Procurement Specialist ECSO2

Hannah M. Koilpillai Senior Finance Officer CTRFC-His

Zeynep Lalik Sr Financial Management Specialist ECSO3 Jennifer Manghinang Temporary ECSHD Imelda Mueller Operations Analyst ECSH2 Ferda Sahmali Sr Operations Officer ECSHD Robert Schware Lead Informatics Specialist TWICT Ibrahim Sirer Senior Procurement Specialist ECSO2

48

Arzu Uraz Junior Professional Associate ECSH4 Kirill Vasiliev Education Specialist ECSH2 Elif Yonca Yukseker Program Assistant ECCU6 John A. Innes Lead Operations Officer ECSHD Robin S. Horn Adviser HDNED Pasi Sahlberg Senior Education Specialist ECSHD Maureen Anne McLaughlin Lead Education Specialist ECSHD Ross Pavis Senior Operations Officer AFTDE Majed El-Bayya Lead Procurement Specialist ECSO2 Kumar Rajeev Swami Senior Financial Management Sp. ECSO3 Juan Diego Alonso Education Economist LCSHE Emilio Ernesto Porta Pallais Senior Education Specialist HDNED Cristobal Ridao-Cano Country Sector Coordinator ECSH4 Mario Cristian Aedo Inostroza Senior Education Economist ECSH2

Ozgur Avcuoglu Junior Professional Associate ECSHD Hassan Naveed Naqvi Senior Education Economist ECSH2 Carla Pittalis Operations Officer ECCU6 Ahmet Levent Yener Sr. Human Development Specialist ECSH4 Virginia Jackson Consultant MNSHD Ireneusz Smolewski Senior Procurement Specialist ECSO2 Hassan Naveed Naqvi Senior Education Economist ECSH2 Salih Bugra Erdurmus Procurement Specialist ECSO2 Nino Kutateladze Operations Officer ECSH2

49

(b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)

No. of staff weeks USD Thousands (including travel and consultant costs)

Lending FY00 0.55 7.09 FY01 48.27 199.80 FY02 45.58 235.80 FY03 34.07 179.33 FY04 23.58 163.75 FY05 26.63 116.18

Total: 178.68 901.96 Supervision/ICR

FY00 0.00 0.00 FY01 0.00 0.00 FY02 0.00 0.00 FY03 0.00 0.00 FY04 0.00 0.00 FY05 7.07 35.26 FY06 35.20 203.48 FY07 33.60 147.39 FY08 46.78 243.29

FY09 31.68 131.19 FY10 64.14 239.31 FY11 59.30 165.72 FY12 30.53 94.56 FY13 6.18 28.28

Total: 314.48 1,288.48

50

Annex 5. Beneficiary Survey Results (if any)

No beneficiary surveys were conducted as part of the ICR.

51

Annex 6. Stakeholder Workshop Report and Results No stakeholder workshops were held as part of the ICR.

52

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR The following comments from the Ministry of National Education (MONE) and the Ministry of Development (MoD) on the draft Secondary Education Project ICR were sent by the Undersecretariat of Treasury via a letter dated December 11, 2012.

Ministry of National Education’s (MONE) Comments The Secondary Education Project ICR (Implementation, Completion and Results) Report (Report No: ICR00002264) prepared by the World Bank has been submitted to our Ministry for assessment. As stated in the Report, the Secondary Education Project, which came into effect on 22 February 2006, was restructured on various dates as required during the implementation phase. In line with the purpose and objectives identified as a result of these restructuring efforts, the Project outcomes, indicators, components and activities have also changed. These changes were also found appropriate by the Word Bank, who provided the Project funding. The studies carried out in line with the objectives and outcomes redefined after the restructuring are as follows:

• Within the scope of the restructuring of secondary education in a way that is sensitive to the experiences and interests of students and prioritizes problem-solving and coping in an environment of change, a total of 66 curricula, prepared with a structured approach and centred on the student, have been developed and put into implementation. Lessons and textbooks were prepared in accordance with these new programmes. Trainings were also delivered for effective and accurate utilization of the programmes.

• The classical programmes of the vocational high schools transferred to our Ministry from other Ministries (such as Justice Vocational High Schools, Title & Cadastre Vocational High Schools, Meteorology Vocational High Schools, Health Vocational High Schools and Agricultural Vocational High Schools) were replaced with new education programmes (curricula) prepared in a modular format based on qualifications with adequate flexibility and sensitivity towards the requirements of labour markets. In this scope, around 1150 education modules were prepared along with their corresponding booklets.

• Vocational education fields that were in conformity with, and oriented to endure the successful implementation of, the newly developed programmes and modules, and that had high socio-economic value in today’s world with good employment opportunities, such as aircraft, agriculture, health and justice, were provided with modern devices and equipment, hence upgrading the education environments to today’s conditions and rendering them capable of meeting the sector’s needs.

• IT equipment were purchased for 1629 secondary education schools in 50 provinces with development priority, in line with our Ministry’s policies to support usage of current technologies in education.

• For the first time in Turkey, a web-based National Vocational Information System (MBS) was established, which facilitates access to vocational guidance for individuals of all ages and which aims to spread vocational guidance works, support and coordinate the

53

vocational guidance efforts of the MoNE, İŞKUR (Turkish Employment Authority) and other agencies and organizations, and provide guidance to all individuals above the age of 13 on their life-long path of development and learning. Various activities were carried out to promote the MBS. As of September 2012, the system has 1.7 million registered users.

• With the Secondary Education Project, nine nationwide studies, aiming to contribute to identifying and developing the Ministry’s education policies, have been carried out – such as “MEB’in 21. Yüzyıl Öğrenci Profili” (MONE’s 21st Century Student Profile).

• In order to ensure a transparent, measurable, participatory and accountable management approach in our Ministry, the qualifications of all students, teachers and administrators of all levels were identified along with the standards for their educational institutions, and a web-based performance management system was created to enable transition to performance-based evaluation and assessment.

• Vocational Education Initiatives aimed to expedite the localization process of the system by including local vocational education institutions in the implementation of the secondary education reform process. In this scope, it was aimed to improve, through services procured domestically or from abroad, the competencies of vocational education and training institutions in the areas they provide education services, improve the vocational qualifications of technical vocational education branch teachers, and thereby raise the quality of both institutional and individual vocational education. In this scope, in order to improve the quality and economic feasibility of vocational education programmes, a competitive Project contest was designed with the purpose of improving the quality of vocational education institutions and developing the skills and qualifications of vocational courses teachers; from among the 433 projects applying for the competition, it was decided to support 46 project proposals that were innovative, sustainable and not for commercial profit.

• By implementing a grand programme at the national level (School Development Programme), it was aimed to increase schooling rates and education quality in disadvantaged schools; 3671 schools in 60 districts prepared projects to meet their own needs, through which they used grants equal to EUR 31,768 Million.

• In order to evaluate the Project, beneficiary evaluation studies were carried out for the Curriculum Reform, Vocational Education Initiatives and School Development Programme, and the results were shared with relevant agencies and organizations.

• In order to increase service quality at all levels in our Ministry, various training activities were carried out, workshops were organized and equipment support was provided.

When carrying out the Project activities: • The Project was executed in concordance with the Project objectives, aims and

performance indicators agreed upon in line with the final restructuring. • All activities were carried out according to the Bank rules, and all Bank approvals were

obtained. • Activities carried out under the Project were subjected to External Auditing every year,

and these external audits did not find anything violating the Bank rules. Also, our Ministry has always done the internal auditing, supervising and auditing the Project activities.

54

• The Project management, Project activities and procurement activities were reviewed during all mission visits paid on a regular basis by the Bank, and no noteworthy problems have ever been expressed in any of the mission reports.

• Regarding the expenses, constant communication was maintained with the Zagreb Office of the World Bank, and no significant problems were encountered in this area.

• Of the total Project funds of EUR80 million, EUR 76.95 million has been used. As stated in the ICR report, the fund utilization rate is 96%.

• The School Development Programme, introduced with the final restructuring of the Project, was successfully implemented in 3671 schools, and contributions were made to education and training in our country with the grand programme (please find below the comments of a students and parent regarding the SDP) Tuğba Ergün – Student, Diyadin, Ağrı: “The physical structure of the school was improved, enabling students to have their classes in a healthier environment. With the arrival of the electronic goods, the classes became more visual, making the courses easier-to-understand for students.” Kıyasettin Ekin – Parent, Hınıs, Erzurum: “We are very happy with the work done under the SDP; all the needs of the school were met and all lacking items were supplied. It has been a very significant success for our students. The new things provided to the schools increased our children’s desire to go to the school. We want such projects to continue.” In addition, in the evaluation made under F. Results Framework Analysis (a) PDO Indicator(s) in the ICR prepared by the Bank, the report includes the following statements:

for Indicator 1: “…generally in line with international standards” for indicator 2: “Stakeholders partially satisfied with the use of school grants.” for Indicator 3: “The career Information System was launched… At Project closing, it counted 1.3 million registered users.” for Indicator 4: “Piloting of Student Achievement Assessment Exam at the secondary education level completed in May 2009. Dissemination took place in 2011 for MONE representatives and academicians.”

Taking into consideration the abovementioned activities and explanations, we are of the opinion that the ICR and evaluations by the Bank are incorrect in the ratings:

1. “Moderately Unsatisfactory” as expressed under C. Rating Summary, C.1

Performance Rating by ICR Borrower Performance and C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)

2. “Moderately Unsatisfactory” as expressed under G.Ratings of Project performance in ISRs

Looking at the data and evaluations included under the heading H.Restructuring (if any), it is seen that there was a great progress in the last two years of the Project, and that 75% of the Project activities and Project spending were done in the last two years. The assessment for the last two years is MS. It is our opinion that the ratings fall short since the overall Project is rated as MU while 75% of the Project activates and financing are rated as MS.

55

Thus, we think it would be appropriate to review OOP’s assessments in the ICR report and revise them in consideration of Project outputs, loan usages and beneficiary evaluations.

Ministry of Development (MoD) Comments on the Draft ICR

As a result of the evaluations made, it is seen that the Project process starting in 2005 is described in a systematic and objective way in the draft completion report. As known, upon the request of the Ministry of National Education, various revisions were made within the scope of said Project. In this line, a new component was added to the Project, which allowed lending against projects prepared by schools, so as to increase access and education quality in districts where schooling rates remained under 90%. Then it was identified that some purchases of goods and educational activities would not be carried out until the Project end. Moreover, it was stated that problems continued in the purchase of information and communication technologies (ICT) equipment and that progress was slow in some provinces within the scope of the School Development Grants (SDG) components, while it was found appropriate to transfer the remaining 2 million Euros to the “Quality Assessment” components so as to ensure that the appropriation was used fully. In addition, it was evaluated that it would be appropriate to transfer all remaining funds to the “School Development Grants” component in case the allocated appropriations remained unspent due to cancelled activities or tenders.

In addition to the revisions, another point that should be mentioned is that carrying out an impact analysis to assess the grant outcomes is vital in terms of achieving the Project objectives. However, as stated in the report, no impact analyses were carried out in the Project. Hence, it has not been possible to measure the outcomes clearly. Moreover, there is no data showing the ways in which the ICT equipment provided to the teachers have been used. Additionally, the fact that the Project activities were adjusted according to the existence of a funding source has prevented the realization of strategy-based activities. The financial resource-based perception of the Project, the constant revision of Project components, and the changes in the Project team have created negative impacts on Project ownership. Despite these adversities, the Project has resulted in important gains for future projects as also described with meaningful expressions under “Lessons Learned” in the draft completion report.

Undersecretariat of Treasury Comments on the Draft ICR

We are of the opinion that the rating of the MoNE does not sufficiently take into account that ‘The Project was submitted for approval with unresolved design issues, and without sufficiently clear and transparent communication from the Bank on how to best address them.” The ICR rated MoNE’s performance as moderately unsatisfactory with the justifications mentioned on page 29, paragraph 66. However, MoNE’s limited ownership was mainly due to the project design which did not reflect MoNE’s priorities.

56

Annex 8. Comments of Co-financiers and Other Partners/Stakeholders Not applicable.

57

Annex 9. List of Supporting Documents World Bank: Project Appraisal Document, February 2005 (Document No.: 27983-TU) Aide Memoires Back to Office reports Correspondence with Government Project Information Document Decision Meeting Minutes Loan Agreement, February 2006 Amendments to the Loan Agreement SDP Operational Manual VTI Operational Manual 2004-07 Country Assistance Strategy 2008-11 Country Partnership Strategy 2012-15 Country Partnership Strategy Implementation Status and Results Reports Basic Education I Implementation Completion and Results Report Basic Education II Implementation Completion and Results Report Quality Assessment of Lending Portfolio

Ministry of National Education (MoNE): PCC Progress Reports No. 1through 6 SEP Closing Report, April 2012 MoNE’s Medium-term Strategy

Other: Turkey’s 2001-2005 8th National Development Plan

Consultations and Interviews: MoNE Salih Celik, Deputy Undersecretary of MoNE Unal Aykuz, PCC Director Tuncay Yelboga, PCC Deputy Director Kubra Akcam, Project Director, PCC Ufuk Yuzuncu, M&E Specialist, PCC Eda Ozbek, Student Assessment, PCC Nurdan Gezer, School Development, PCC Mustafa Yuruk, School Development Program and Vocational Training Initiatives, PCC Yurdagul Aydogan, General Director, Special Education & Guidance, MoNE

Ministry of Development Ahmet Alper Ege, Department Head Serdar Polat, Planning Expert Betul Ersin Uysal, Planning Expert Alper Yatmaz, Planning Expert

World Bank Maureen Mclaughlin, Sector Manager and Task Team Leader Naveed Naqvi, Task Team Leader

58

Keiko Sato, Lead Operations Officer and Portfolio Manager Florian Fichtl, Lead Operations Officer and Portfolio Manager Zeynep Lalik, Sr. Financial Management Specialist Salih Kalyoncu, Sr. Procurement Specialist Salih Bugra, Procurement Specialist Juan Diego Alonso, Education Economist

59

Annex 10. Original and Revised Components A. Original components (as approved on March 15, 2005) Component 1: Secondary Education Reform 1.1 Secondary Education Programs. The objective was to revise and implement new general education and vocational education curriculum programs to provide students with core skills for the knowledge economy and lifelong learning. The Project would finance technical assistance, equipment and materials to strengthen the institutional structures, develop technical procedures and new curricular content, implement revised general education and vocational education curriculum programs, and provide related in-service training for MONE managers and teachers. 1.2 Information and Communications Technology (ICT). The objective was to provide ICT training as a core competency for youth to function in a modern knowledge economy, to improve instruction by using ICT to expand access to learning opportunities, and to support better management of learning and administrative processes. The Project would finance technical assistance, goods and materials to utilize ICT to improve instruction, access on-line educational content and services on the Internet, and assess the educational impact of the ICT investments.

1.3 Career Guidance and Counseling: The objective was to provide youth and adults with access to career information and guidance services to promote informed career choices and related education and training decisions and facilitate lifelong learning. The Project would finance technical assistance, equipment, and materials to facilitate interagency cooperation, develop core career information and guidance resources and provide them to education and labor institutions, train existing staff for career guidance and improve the integration of career guidance and counseling into overall basic and secondary education programs. Component 2: Quality, Assessment and Management 2.1 Student Assessment and Evaluation. The objective was to improve the quality and outcomes of secondary education by developing systems to collect and disseminate reliable information on student learning and outcomes, and on the performance of the education system staff and institutions, and to use information from these assessments to improve system performance and student outcomes. The Project would finance technical assistance, equipment, test instrument development and printing, and training for MoNE staff at all levels of the system. 2.2 School Development. The objective was to help secondary schools staff, parents, and students improve the performance and outcomes, educational achievement, and quality in education through school development activities to be designed and implemented within the school development process. The Project would finance technical assistance, development and distribution of guide books and training materials, and training activities for school and provincial level MONE staff.

2.3 Project Management. The objective was to ensure effective administration and coordination of the overall project. The Project would finance related technical assistance, equipment, and materials.

60

B. Revised components (as approved on March 25, 2008)

Component 1: Secondary Education Reform 1.1 Curricula Reform. This sub-component would develop and implement new general education and vocational education curricula and courses to provide students with core skills for the knowledge economy and lifelong learning. The sub-component would finance consultants to develop and publish a curriculum framework, to develop new curricula for 71 courses in general secondary education and 19 programs in vocational education, and to write and revise textbooks and modules for the new curriculum. It would also support workshops and training for teachers, principals, ministry staff, and provincial education directorates to prepare and implement the new curricula. Lastly, the sub-component would finance equipment and learning materials in 210 general secondary schools and 149 vocational schools. 1.2 Information and Communications Technology (ICT). This sub-component would finance ICT classrooms in 322 schools as well as training of teachers in these schools in order to improve the competency of students to function in a modern knowledge economy. It would finance development of multi-media learning tools, consultant services, goods and materials to utilize ICT to improve instruction and access to on-line educational content. 1.3 Career Guidance and Counseling. This sub-component would finance activities to improve access to career guidance and counseling services in order to enable the individuals to select occupations in line with their interests and abilities, move ahead in their careers; and switch when deemed necessary. It would include (i) facilitating intra-agency collaboration to improve the development, dissemination, and use of career counseling services; (ii) developing core career information and guidance resources that will be available for use in education, training, labor, NGO, and private sector; (iii) providing career information and guidance resources to education and labor institutions and training existing staff in use of resources; and (iv) improving integration of career guidance and counseling services with new curricula.

Component 2: Quality Assessment 2.1 Student Assessment. The sub-component would finance consultant services and training to develop and pilot student assessments and questionnaires in line with the newly developed curricula. These assessments would help to improve the quality and outcomes of secondary education by developing systems to collect and disseminate reliable information on student learning and outcomes and on the performance of education system staff and institutions. 2.2 School Improvement. The sub-component would finance training for 40,000 teachers to implement the new school development model to help secondary schools improve their performance and student educational achievement. It would also support development of performance management system to improve the evaluation of teacher performance.

Component 3: Staff Capacity Development 3.1 Training for Foreign Language Teachers. The sub-component would support intensive training to improve the language skills and teaching methods of foreign language teachers through in-country and international training. The ultimate goal of this training would be to improve the foreign language skills of Turkish students, skills needed in the labor market and in

61

the EU. The training would be organized in two phases. The first phase would include in-country training for 150 teachers and overseas training for 500 teachers. Phase two of the training would be based on an evaluation of the effectiveness of the training in the first phase. 3.2 Vocational Training Initiatives. This sub-component would finance competitive grants for vocational schools to help improve the quality and economic relevance of vocational technical education programs. These grants would support innovative proposals by vocational schools in cooperation with other partners designed to improve the skills and qualifications of vocational teachers and improve the quality of the vocational training institutions. Selection of eligible beneficiaries, provision, and administration of grants would be governed by the Operational Manual for Vocational Training Initiatives approved by the Government on February 21, 2008.

Component 4: Project Management, Monitoring and Evaluation 4.1 Project Implementation Support. The sub-component would support effective administration and coordination of the Project. This would include coordinating project execution, managing the resources of the Project, procuring goods and services under the Project, operating the financial management system, and ensuring timely and appropriate reporting. This sub-component would also finance public awareness activities through dissemination of the key messages and accomplishments of the Project. 4.2 Monitoring & Evaluation. The sub-component would support regular monitoring of the Project, evaluation of project sub-components, and an evaluation of the overall impact of the Project on the quality and relevance of education. C. Revised components (as approved on April 7, 2010) In the Restructuring Paper dated March 24, 2010, the name of sub-component 2.2 changed from “School Improvement” to “School Development”. Component 3 was renamed “School Grants” and included a new School Development Program sub-component. The Training for Foreign Language Teachers was cancelled. Component 3: School Grants 3.1 Training for Foreign Language Teachers. This sub-component was cancelled. 3.2 Vocational Training Initiatives. No changes made. 3.3 School Development Program. This sub-component would provide grants to schools in sub-provinces with low enrollment rates in order to provide additional assistance to schools in areas likely to have been affected adversely by the economic downturn. Schools would be able to use the SDP grants to address their specific educational needs and to ameliorate the effects of the economic slowdown while building capacity at primary and secondary schools.

TuzTuzGölüGölü

HoyranHoyranGölüGölü

BaysehirBaysehirGölüGölü

AksehirAksehirGölüGölü

Çoruh

Murat

Kura

Firat

Sakarya

Devrez

Kizil

Cekerek

Kizil

Seyh

an

Göksu Ceyh

an

Kelkit

Kuzey Anadolu Daglari

Toros Daglari KaramanKaraman

KonyaKonya

AksarayAksaray

NigdeNigde

NevsehirNevsehir KayseriKayseri

MalatyaMalatya

AdiyamanAdiyaman

ElazigElazig

TunceliTunceli

ErzincanErzincan

BingölBingölMusMus

BitlisBitlis

DiyarbakirDiyarbakir BatmanBatman SiirtSiirt

SirnakSirnakMardinMardin

HakkariHakkari

VanVan

AgriAgriIgdirIgdir

ErzurumErzurum

KarsKars

ArtvinArtvin

RizeRize

GümüshaneGümüshaneBayburtBayburt

GiresunGiresun

TokatTokat

SivasSivas

AmasyaAmasya

KastamonouKastamonou

ÇankiriÇankiri

KarabükKarabük

BoluBoluSakaryaSakarya(Adapazari)(Adapazari)

KocaeliKocaeli(Izmit)(Izmit)

BilecikBilecik

EskisehirEskisehir

KütahyaKütahya

BursaBursa

YalovaYalova

IstanbulIstanbulTekirdagTekirdag

EdirneEdirneKirklareliKirklareli

ÇanakkaleÇanakkale

BalikesirBalikesir

ManisaManisaIzmirIzmir UsakUsak

AydinAydin

DenizliDenizliBurdurBurdur

IspartaIsparta

AfyonAfyon

AntalyaAntalya

MuglaMugla

KirikkaleKirikkale

ÇorumÇorum

YozgatYozgat

KirsehirKirsehir

AdanaAdana

OsmaniyeOsmaniye

KahramanKahraman Maras Maras

GaziantepGaziantep

KilisKilis

SanliurfaSanliurfa

ArdahanArdahan

ANKARAANKARA

Aras

DüzceDüzce

SYRIAN ARABSYRIAN ARABREPUBLICREPUBLIC

IRAQIRAQ

ARMENIAARMENIA

GEORGIAGEORGIA

BULGARIABULGARIARUSSIAN FEDERATIONRUSSIAN FEDERATION

AZER-AZER-BAIJANBAIJAN

AZER.AZER.

To To BatumiBatumi

To To KirovakanKirovakan

ToToMakuMaku

ToToSalmasSalmas

ToToOroumiehOroumieh

ToToDahukDahuk

To DamirTo DamirKabuKabu

ToToAl HasakahAl Hasakah

ToToAleppoAleppo

ToToBurgasBurgasToTo

KurdzhaliKurdzhali

Karaman

Konya

Aksaray

Nigde

Nevsehir Kayseri

Malatya

Adiyaman

Elazig

Tunceli

Erzincan

BingölMus

Bitlis

Diyarbakir Batman Siirt

SirnakMardin

Hakkari

Van

AgriIgdir

Erzurum

Kars

Artvin

RizeTrabzon

GümüshaneBayburt

Giresun

Ordu

Tokat

Sivas

Amasya

Samsun

Sinop

Kastamonou

Çankiri

Karabük

BartinZonguldak

Bolu

Düzce

Sakarya(Adapazari)

Kocaeli(Izmit)

Bilecik

Eskisehir

Kütahya

Bursa

Yalova

IstanbulTekirdag

EdirneKirklareli

Çanakkale

Balikesir

ManisaIzmir Usak

Aydin

DenizliBurdur

Isparta

Afyon

Antalya

Mugla

Kirikkale

Çorum

Yozgat

Kirsehir

Icel(Mersin)

Adana

Hatay (Antakya)

Osmaniye

Kahraman Maras

Gaziantep

Kilis

Sanliurfa

Ardahan

ANKARA

SYRIAN ARABREPUBLIC

IRAQ

ISLAMICREP. OFIRAN

ARMENIA

GEORGIA

BULGARIARUSSIAN FEDERATION

AZER-BAIJAN

AZER.

GRE

ECE

GREECE

TuzGölü

HoyranGölü

BaysehirGölü

AksehirGölü

Lake Van

Çoruh

Murat

Kura

Aras

Firat

Tigris

Euphrates

Sakarya

Devrez

Kizil

Cekerek

Kizil

Seyh

an

Göksu Ceyh

an

Kelkit

B lack Sea

Medi terranean Sea

Sea ofMarmara

Gulf ofAntalya

Istanbul Strait(Bosphorus)

ÇanakkaleStrait

(Dardanelles)

To Batumi

To Kirovakan

ToMaku

ToSalmas

ToOroumieh

ToDahuk

To DamirKabu

ToAl Hasakah

ToAleppo

To Ladhiqiyah

ToBurgasTo

Kurdzhali

ToKomatini

Kuzey Anadolu Daglari

Toros Daglari

Agri Dagi(5166 m)

26°E 28°E 30°E 32°E 34°E 36°E 38°E

42°E 44°E30°E28°E 32°E 34°E

36°N

38°N

40°N

42°N

40°N

42°N

TURKEY

This map was produced by the Map Design Unit of The World Bank. The boundaries, colors, denominations and any other informationshown on this map do not imply, on the part of The World BankGroup, any judgment on the legal status of any territory, or anyendorsement or acceptance of such boundaries.

0 50 150100

0 50 100 150 Miles

200 Kilometers

IBRD 33501R2

JULY 2008

TURKEYPROVINCE CAPITALS*

NATIONAL CAPITAL

RIVERS

MAIN ROADS

RAILROADS

PROVINCE BOUNDARIES*

INTERNATIONAL BOUNDARIES

*Province names are the same as their capitals.