the world bankdocuments1.worldbank.org/curated/en/...jun 27, 2017 · (qea): none problem project...

Document of The World Bank

Report No: ICR00003631

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H6330)

ON A

GRANT

IN THE AMOUNT OF SDR 3.2 MILLION (US$5 MILLION EQUIVALENT)

TO THE

DEMOCRATIC REPUBLIC OF TIMOR-LESTE

FOR A

SECOND CHANCE EDUCATION PROJECT

June 27, 2017

Education Global Practice East Asia Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

ii

CURRENCY EQUIVALENTS

(Exchange Rate Effective May 1, 2017)

Currency Unit = Timor-Leste US$ US$ 1.00 = Timor-Leste US$1

FISCAL YEAR

January 1 – December 31

ABBREVIATIONS AND ACRONYMS

CLC Community Learning Center CDP Capacity Development Plan CPS Country Partnership Strategy DSF Second Chance Education Project facilitators DNEAENF Directorate of Adult and Non-Formal Education EMIS Education Management Information System EP Equivalency Program ESSSP Education Sector Support Project FA Financing Agreement FM Financial Management GDP Gross Domestic Product GoTL Government of Timor-Leste ICR Implementation Completion and Results Report IDA International Development Association INFORDEPE National Institute for Professional Educators Training ISR Implementation Status Report IT Information Technology MDGs Millennium Development Goals M&E Monitoring and Evaluation MOE Ministry of Education MOF Ministry of Finance MTR Mid-term Review NDRE National Directorate for Recurrent Education NESP National Education Strategic Plan NSDP National Strategic Development Plan PCN Project Concept Note PDO Project Development Objective SCEP Second Chance Education Project TORs Terms of Reference

iii

TTL Task Team Leader TVET Technical and Vocational Education UNICEF United Nationals Children’s Education Fund WDR World Development Report

Senior Global Practice Director: Jaime Saavedra Chanduvi

Practice Manager: Harry Anthony Patrinos

Project Team Leader: Susiana Iskandar

ICR Team Leader: Susiana Iskandar

iv

DEMOCRATIC REPUBLIC OF TIMOR-LESTE Second Chance Education Project

CONTENTS

Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 6 3. Assessment of Outcomes .......................................................................................... 14 4. Assessment of Risk to Development Outcome ......................................................... 19 5. Assessment of Bank and Borrower Performance ..................................................... 20 6. Lessons Learned ....................................................................................................... 22 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 23 Annex 1. Project Costs and Financing .......................................................................... 24 Annex 2. Outputs by Component ................................................................................. 25 Annex 3. Economic and Financial Analysis ................................................................. 34 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 37 Annex 5. Beneficiary Survey Results ........................................................................... 39 Annex 6. Stakeholder Workshop Report and Results ................................................... 40 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 41 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 54 Annex 9. List of Supporting Documents ...................................................................... 55

MAP

v

A. Basic Information

Country: Timor-Leste Project Name: Timor-Leste Second Chance Education Project

Project ID: P116520 L/C/TF Number(s): IDA-H6330

ICR Date: 06/27/2017 ICR Type: Core ICR

Lending Instrument: SIL Borrower: THE DEMOCRATIC REPUBLIC OF TIMOR-LESTE

Original Total Commitment:

XDR 3.20M Disbursed Amount: XDR 3.15M

Revised Amount: XDR 3.20M1

Environmental Category: C

Implementing Agencies: MINISTRY OF EDUCATION

Cofinanciers and Other External Partners: B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 06/03/2009 Effectiveness: 04/06/2011

Appraisal: 05/05/2010 Restructuring(s): 12/17/2012 09/16/2015

Approval: 12/07/2010 Mid-term Review: 08/04/2014 07/15/2014

Closing: 12/31/2015 12/31/2016 C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Moderately Satisfactory

Risk to Development Outcome: Moderate

Bank Performance: Moderately Satisfactory

Borrower Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory

1 There is a glitch in the system; the data sheet indicates a revised amount of SDR 3.10 million, which is incorrect. The restructuring did not result in a reduction of the overall amount.

vi

Quality of Supervision: Satisfactory Implementing Agency/Agencies:

Moderately Satisfactory

Overall Bank Performance:

Moderately Satisfactory Overall Borrower Performance:


C.3 Quality at Entry and Implementation Performance Indicators

Implementation Performance

Indicators QAG Assessments

(if any) Rating

Potential Problem Project at any time (Yes/No):

Yes Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

Yes Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:


D. Sector and Theme Codes

Original Actual

Major Sector/Sector

Public Administration

Public administration - Education 42 42

Education

Adult, Basic and Continuing Education 58 58

Major Theme/Theme/Sub Theme2

Human Development and Gender

Education for All 50 100

Access to Education 50

Education Financing 50 E. Bank Staff

Positions At ICR At Approval

Regional Vice President: Victoria Kwakwa James W. Adams

Country Director: Michel Kerf Ferid Belhaj

Practice Manager: Harry Anthony Patrinos Eduardo Velez Bustillo

Task Team Leader(s): Susiana Iskandar Stephen David Close

2 The task team has requested OPCS to change the actual “major theme” codes to 100 percent “education for all”. At the time of ICR submission, OPCS has not yet made the modifications in the data sheet.

vii

ICR Team Leader: Susiana Iskandar

ICR Primary Author: Sandra F. Beemer F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document) The development objectives of this project are: (a) to increase the number of out-of-school youth and young adults who complete recognized equivalency programs; and (b) to decrease number and/or rate of leavers from each program and level, thus increasing internal efficiency of the program. Revised Project Development Objectives (as approved by original approving authority) The revised project development objective is to support the Ministry of Education in the establishment of a sustainable national equivalency program. (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1: Legal framework regarding recurrent education endorsed by Ministry of Education (MOE)

Value quantitative or qualitative)

None

Legal framework endorsed by MOE for the recurrent education law

N/A

Legal framework endorsed by MOE for the recurrent education law. Source: NDRE

Date achieved 12/31/2009 12/31/2016 12/31/2016 Comments (incl. % achievement)

Target achieved. The equivalency program (EP) Decree Law was approved by the Council of Ministers 4/2016 and promulgated by the president’s office in 7/2016 as Decreto-Lei n.o. 30/2016

Indicator 2: Curriculum endorsed by the MOE

Value quantitative or qualitative)

None

EP Level II curriculum revised and endorsed by MOE

EP Level II curriculum revised and endorsed by MOE. Source: NDRE


Target achieved. MOE endorsed the new curriculum in March 2016 prior to attaching it as an annex to the Decree Law.

Indicator 3: Number of out-of-school youth and young adults who completed NDRE equivalency programs

Value

viii

quantitative or qualitative) Date achieved 12/31/2009 12/31/2015 Comments (incl. % achievement)

Indicator was dropped during the 2015 restructuring.

Indicator 4: Rate of leavers from each program and level (% rate) Value quantitative or qualitative)

Date achieved 12/31/2009 12/31/2015 Comments (incl. % achievement)

Indicator was dropped during the 2015 restructuring

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1: Learning materials for equivalency program Level II endorsed by the MOE

Value (quantitative or qualitative)

None

Learning materials revised and endorsed by the MOE

Learning materials revised and endorsed by the MOE. Source: NDRE


Target met. Learning materials for EP Level II were endorsed by MOE 12/2016.

Indicator 2: NDRE staffed with teachers and pedagogic team


Partially staffed

Teachers and pedagogic team started working and retrained.

Teachers and pedagogic team recruited, trained and continue to work. Source: NDRE


Target met. There are 20 staff on the pedagogic team and 97 Level 1 teachers and 42 Level II teachers all on NDRE payroll.

Indicator 3: Teachers and pedagogic team trained to deliver the equivalency programs


Partial teacher training provided

Teacher and pedagogic team trained and deliver equivalency programs

Teachers and pedagogic team trained and deliver equivalency programs.

ix

Source: NDRE Date achieved 12/31/2009 12/31/2016 12/31/2016 Comments (incl. % achievement)

Target met. All 139 members of the pedagogic team and EP teachers were trained on the new curriculum and materials.

Indicator 4: Community Learning Centers established, equipped and functioning


None

At least 6 and maximum of 9 CLCs established, equipped and functioning

9 CLCs have been established, equipped and functioning.


Target met. Nine CLCs in 8 municipalities are equipped, functioning and implementing the EP.

Indicator 5: Semi-annual reports on recurrent education programs produced based on report template.


No Yes

Yes Source: NDRE/Bank Missions


Target met. The PMU and NDRE provided detailed progress reports for Bank mission use as well as other MOE departments.

Indicator 6: Direct project beneficiaries (% female) Value (quantitative or qualitative)

0.00 1,381 (55%) 1,328 Level 1(55%) 342 Level 2 (70%) Source: NDRE


Target was exceeded with a total of 1,670 EP students participating in the program. The target was exceeded by 21 percent

Indicator 7: Number of students who complete NDRE equivalency programs: Level I Value (quantitative or qualitative)

N/A


Indicator was dropped during the 2015 restructuring because the focus of restructuring was to establish a sustainable quality equivalency program.

Indicator 8: Number of students who complete NDRE equivalency programs: Level II Value (quantitative or qualitative)

N/A

Date achieved 12/31/2009 12/31/2015

x

Comments (incl. % achievement)

Indicator was dropped during the 2015 restructuring because creating CLCs and developing ‘semi-presential’ learning materials took more time than originally designed.

Indicator 9: Ratio of annual equivalency program completers compared to initial enrollments for equivalency programs.


N/A



Indicator 10: Ratio of students who pass equivalency Level II subject phases compared to total sitting for exams.


N/A



Indicator 11: Total number of students of NDRE programs studying community learning centers


N/A



G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual

Disbursements (USD millions)

1 02/11/2011 Satisfactory Satisfactory 0.00 2 04/18/2012 Satisfactory Moderately Satisfactory 0.38 3 01/04/2013 Moderately Satisfactory Moderately Satisfactory 0.64 4 08/19/2013 Moderately Satisfactory Moderately Satisfactory 1.00

5 05/21/2014 Moderately Satisfactory Moderately

Unsatisfactory 1.51

6 10/14/2014 Moderately

Unsatisfactory Moderately

Unsatisfactory 1.73


Unsatisfactory Moderately Satisfactory 2.20

xi


Unsatisfactory Moderately Satisfactory 2.39

9 11/09/2015 Moderately Satisfactory Moderately Satisfactory 2.75 10 05/25/2016 Satisfactory Moderately Satisfactory 3.40 11 11/17/2016 Moderately Satisfactory Moderately Satisfactory 4.10 12 12/21/2016 Moderately Satisfactory Satisfactory 4.27

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made

DO IP

12/15/2012 S MS 0.38

Level 2 restructuring to modify the percentage of expenditures to be financed under Category (2) salaries for contract teachers, teacher trainers, and field coordinators due to delays in hiring the teachers and field coordinators.

09/16/2015 Y MU MS 2.49

Level 1 restructuring revise PDO, results framework, extend closing date for 12 months, revise the title of component 4, include the draft of recurrent education legal framework, reduce the number of CLCs from 65 to a maximum of 9, reallocate funds to accommodate the project changes, change CLC grant agreement signatories, and clarify that small works and services were eligible expenditures.

If PDO and/or Key Outcome Targets were formally revised (approved by the original approving body) enter ratings below: Outcome Ratings Against Original PDO/Targets Moderately Unsatisfactory Against Formally Revised PDO/Targets Moderately Satisfactory Overall (weighted) rating Moderately Satisfactory

xii

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal 1. Country context. In May 2002, Timor-Leste had regained independence and, by and large, had to rebuild its institutions from zero. The challenges related to rebuilding institutions and building the needed capacity was confirmed by the 2011 World Development Report (WDR) on “Conflict, Security and Development” which stated that establishing a well-function state can take decades and newly-created countries are at a high risk of relapses into violence (as evidenced by the crisis in Timor-Leste in 2006). By the time of project appraisal, Timor-Leste was a country in transition from low-income to a medium-income status, graduation supported chiefly by oil and gas revenues. Although there had been significant progress in the creation of the architecture of governance and the associated legal and institutional frameworks, Timor-Leste still exhibited the characteristics of a fragile state. The country had made solid progress in establishing institutions to safeguard public sector accountability, but their effectiveness was constrained by limited capacity and evolving legal mandates. The functioning of both the private and the public sectors had improved during a period of development of government systems, but there were remaining constraints in staff qualifications and experience. The project’s alignment to the strategic objectives included developing the capacity to implement a coherent national development strategy and strengthening the human development outcomes and service delivery results. This was intended to more widely share the benefits of Timor-Leste’s oil and gas wealth which were consistent with the Bank’s strategic objectives as included in the Interim Strategy Note 2009-2010, and draft Country Partnership Strategy (CPS)3 organized by key themes of ‘Productivity’ and ‘Jobs’. 2. Sectoral context. At the country level, the Ministry of Education’s (MOE) National Education Strategic Plan (NESP) 2011-2030 emphasizes an infrastructure-led development strategy, but with an important role for human capital development. A Human Capital Fund had been established to help meet the human capital needs of NESP achievement. Priorities were set out for the education sector, to which the NESP is fully aligned. The Minister of Finance (MOF) had indicated that additional domestic resources could be available for the education sector, provided that progress was made on the competent management of government budget for education outcome and improved delivery on key initiatives supported by government. The MOE had identified the need to strengthen its management to deliver the results of these initiatives within the NESP framework.

3. On the overall public financing, education spending had remained constant at 2 percent of Gross Domestic Product (GDP). However, the total amount of education spending was increasing given the fast pace of GDP growth. The growing national resource envelope continued to support the expansion of the education sector. With the average

3 The draft CPS was approved by the World Bank Board of Directors February 19, 2013.

2

year-to-year enrollment growth at basic and secondary education at around 9-10 percent per annum during the same period, the increase of public expenditure in education had been at the faster pace than the enrollment expansion. It was estimated per student public spending was about US$120 in 2006/07, and rising to around US$190 in 2010. 4. Human development outcomes for Timorese in both health and education remained among the weakest in the East Asia and Pacific region, and population growth was still high (net total fertility rate of 5.7 in 2009/10). Nevertheless, change and growth in the education sector had been significant since independence in 2002. Net enrollment rate in primary education had increased from 68 percent in 2004/05 to 85 percent in 2007/08. However, there were still a large number of young and adult Timorese that lacked adequate basic education to support their participation in the Timorese economy and society. In 2009, only 33 percent of students at Grade 1 remained in school to Grade 6, and in 2010, 25 percent of children aged 6-14 years were not in school. The continued high dropout from education had led to a backlog of Timorese lacking basic education, which meant that approximately 36 percent of the population over fifteen years of age was functionally literate while another 27 percent are semi-literate and a full 37 percent of the adult population was totally illiterate. In the Education Strategy 2010-2030, the Government of Timor-Leste had established key goals including achievement of universal completion of basic education by 2025, gender parity in education access by 2015, and eradication of illiteracy among all age groups by 2015. The latter goal included the introduction of the National Equivalence Program (EP) that would allow accelerated completion of basic education by students who had not completed formal school programs, and possibly including students more than six years over-age in Grades 1-6. 5. At the time of appraisal, there was a dearth of opportunities for poor out-of-school and out-of-work youth, and a youth population lacking basic literacy, numeracy and generalist skills remained unequipped to help meet the emerging skills gaps of a growing economy. Second-chance education programs such as the national equivalency program (EP) were considered a central way of addressing the large stock of youth and adults that had not completed their basic education and/or secondary education. The Ministry of Education’s National Directorate of Recurrent Education (NDRE) (formerly the Directorate of Adult and Non-Formal Education (DNEAENF)) was the key government agency responsible for managing and delivering these second-chance programs. The Second Chance Education Program (SCEP) was developed to assist the government in improving the equivalency programs for the out-of-school and out-of-work youth. The broader objectives served by SCEP were to support three prioritized goals of the Education Strategic Plan 2010-2015 which were: (i) achieve universal completion of basic education by 2025; (ii) ensure gender parity by 2015 at all levels of education, among the teaching corps and in administrative and management positions; and (iii) eliminate illiteracy by 2015 among all age groups.

3

1.2. Original Project Development Objectives (PDO) and Key Indicators 6. The original development objectives of this project were: (a) to increase the number of out-of-school youth and young adults who complete recognized equivalency programs; and (b) to decrease number and/or rate of leavers from each program and level, thus increasing internal efficiency of the program. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 7. As a part of the September 16, 2015 project restructuring, the PDO was revised to: support the Ministry of Education in the establishment of a sustainable national equivalency program. The results framework was also modified to reflect the revised PDO, both outcome and intermediate indicators were dropped and a new set of indicators was developed as follows: Table 1: History of Project Indicators

Original PDO Indicator Adjustments at the time of September 2015 Restructuring

Number of out-of-school youth and young adults who complete NDRE equivalency programs

Dropped during restructuring

Rate of leavers from each program and level (%rate)


Legal framework regarding recurrent education endorsed by MOE (new indicator)

Curriculum endorsed by the MOE (new indicator)

Original Intermediate Indicators Adjustments at the time of September 2015 Restructuring

Number of students who complete NDRE equivalency programs: Level 1


Number of students who complete NDRE equivalency programs: Level 1I


Ratio of annual equivalency program completers compared to initial enrollments for equivalency programs


Ratio of students who pass EP Level II subject phases compared to total sitting for exams


Total number of students of NDRE programs studying at Community Learning Centers


Learning materials for equivalency program Level II endorsed by MOE

NDRE staffed with teachers and pedagogic team

Teachers and pedagogic team trained to deliver the equivalency program

Community Learning Centers (CLCs) established, equipped and functioning

4

Semi-annual reports on recurrent education programs produced based on a report template

Direct Project Beneficiaries (% female) Bank core indicator

1.4 Main Beneficiaries 8. The primary beneficiaries of the SCEP project were 1,328 (55 percent female) level 1 students and 342 level 2 (70 percent female) students who participated in the EP programs and the 110 EP teachers who benefitted from training. Additional beneficiaries of the project were: (i) the 20 pedagogical staff who have had increased capacity from their involvement in the development of the new curriculum; (ii) the nine district SCEP facilitators (DSFs) who provided support to communities in the development of their CLCs, (iii) community members within the nine communities where CLCs were created; (iv) district and community members that received training on EP data entry for the new Education Management Information System (EMIS); (v) district administrators involved in the development of the CLCs; and (vi) NDRE staff that participated in the implementation of the project and development of the new curriculum.

1.5 Original Components 9. Component 1: Curriculum Development and Production/Distribution of Learning and Instructional Materials (US$1,968,318 equivalent) aimed to support the: (i) development of new curricula; instructional materials, and teachers manuals for Equivalence Programs Level II, including distance learning; (ii) training of trainers to deliver Equivalence Level II programs; (iii) establishment of small school libraries at each community learning center established by DNEAENF; and (iv) printing and distribution of textbooks and instructional materials for Primary programs, EP Level I programs, and EP Level II programs, to support the curriculum revisions and introductions. 10. Component 2: Improved Quality of Service Delivery (US$1,347,780 equivalent) aimed at: (i) strengthening of NDRE management of services to out of school youths and adults through management and planning advice, training of managers and administrators as needed; (ii) development and implementation of a promotional campaign to support community engagement and improved participation in non-formal education; (iii) improvement of the monitoring and evaluation capacity of NDRE; and (iv) improvement of teaching capacity, including hiring and training of teachers, field coordinators and teacher trainers, and training of central community facilitators in non-formal education. 11. Component 3: Expansion of local capacity for service delivery through Community Learning Centers (US$1,456,000 equivalent). The aim of this component was to: (i) conduct workshops and training of communities and to introduce the concept of CLCs; and (ii) provide funding of grants to selected communities that agreed to establish and maintain a CLC (one in each of the 65 sub-districts of the recipient), to cover the cost of furniture, equipment, supplies and small works for the established CLCs.

5

12. Component 4: Information, Monitoring & Evaluation Technology - Pilots and Studies (US$227,903 equivalent). This component included: (i) upgrading of instructional, information and office technology to support the establishment and expansion of capacity and monitoring of programs within NDRE; (ii) carrying out of distance learning pilots; and (iii) conducting studies to build the knowledge base for stronger non-formal education programming. 1.6 Revised Components 13. During the February 2015 restructuring, several revisions were made to the project components to align project activities with the new PDO and PDO indicators. 14. Component 1: Curriculum Development and Production/Distribution of Learning and Instructional Materials. The original activities remained the same under this component. Additional activities were to: (i) trial a select number of EP Level I and Level II courses; (ii) provide support to the MOE task force to adjust the curriculum based on trial results and seek MOE endorsement for the national roll-out; and (iii) finance the printing of learning materials and manuals for the nationwide roll-out for EP Level II. 15. Component 2: Improved Quality of Service Delivery. The original activities remained the same for this component. Additional activities were: (i) the development of a regulatory framework to help regulate the recurrent education system and clarify roles and functions among the NDRE and district offices; management, accountability and ownership of CLCs; ways for NDRE to secure budget for recurrent education programs; and personnel status of tutors/teachers of EP; (ii) increased training for NDRE staff and include on-the-job training by the consultants hired by the project and overseas study visits; and (iii) to move support for national and international project management consultants/advisors from component 2 to component 4. 16. Component 3: Expansion of Local Capacity for Service Delivery through Community Learning Centers. This component continued to support the establishment of CLCs using a community participatory approach. However, the number of CLCs was reduced from 65 to the six that had already been established with the possibility of adding three more in 2016. This modification was made to allow the NDRE sufficient time to implement the program through a proper participatory approach that fully empowered the community. The reduced number allowed NDRE to create models of community-based CLCs to inform future CLC development and/or expansion. It also enabled the project to finance the additional activities included as part of the restructuring, and which proved highly beneficial to Timor-Leste. 17. Component 4: Information, Monitoring & Evaluation Technology - Pilots and Studies (original title of the component). The title of this component was changed to Project Management and Studies. The original project activities under this component remained the same. However, the project management activities under component 2 were moved to this component. The additional activities under this component were to support:

6

(i) additional NDRE positions for financial management, procurement, project coordination and an education strategic adviser; and (ii) an audit firm, which previously was funded by the Government. 1.7 Other significant changes 18. On December 17, 2012 the project was restructured to amend the percentage of expenditures to be financed under Category 2 for contract teachers, teacher trainers and field coordinators due to hiring delays. The modifications were: (i) 100 percent of expenditures incurred in 2013 instead of 2011; (ii) 75 percent of expenditures incurred in 2014 instead of 2012; and (iii) 50 percent of expenditures incurred in 2015 instead of 2013. 19. On December 1, 2014, the Ministry of Finance sent a letter requesting an extension of the project closing date, a restructuring of the project PDO and results framework and a modification to the financing agreement that would incorporate portions of the Government of Timor-Leste procurement guidelines. The Bank and Ministry of Finance had an extended dialog over the Bank procurement regulations that delayed the restructuring until an additional request was sent on June 26, 2015. 20. The Ministry of Finance sent a letter on June 26, 2015 to the Bank requesting a restructuring of the project based on findings and recommendations from the July 2014 project mid-term review (MTR). On September 16, 2015 there was a Level 1 restructuring of the project because: (i) the MOE had changed the direction of SCEP to one that focused on developing a sound legal framework for providing a sustainable recurrent education program, including the provision of officially endorsed, good quality curricula and learning materials; and (ii) the original PDO could not be accurately measured due to the MOE’s limited data collection capacity and data reliability. A summary of the project modifications were: (i) revise the Project Development Objectives (PDO); (ii) revise the results framework and associated indicators to reflect the modifications to the PDO; (iii) revise the title of component 4; (iv) provide additional support for the development of a draft recurrent education legal framework to support a quality and sustainable recurrent education system; (iv) reduce number of sub-districts receiving Community Learning Centers (CLCs) from 65 to the existing six with the possibility of adding three more CLCs in 2016; (v) reallocate funds to support the modifications to the project design; and (vi) extend the closing date by 12 months from December 31, 2015 to December 31, 2016. 2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry 21. Project preparation. The SCEP was prepared using a Sector Specific Investment Grant instrument given the targeted nature of program interventions. The preparation team included project activities based on the extensive review and analysis conducted to develop the NESP 2011-2015. The NESP findings were articulated based on a thorough situation analysis of progress since the first National Development Plan (2002-2007) and the challenges that affected the ministry’s ability to meet the country’s Millennium

7

Development Goals (MDGs) for 2015. The preparation team also took into consideration that the Bank was giving priority throughout the Pacific sub-region to programs aimed at out-of-school youth which was consistent with the World Bank’s Education Sector Strategy 2020, the Bank’s Interim Strategy Note 2009-2010 as well as the fragile state context of Timor-Leste. The preparation of the NESP and lessons learned from other projects all contributed to assisting the team in preparing and appraising the project in 11 months for Bank Board approval. The SCEP project would provide International Development Association (IDA) funding for the NDRE in the amount of SDR 3.2 million or US$5 million equivalent over five years (2011-2015), allowing NDRE to: (i) further develop the equivalency curriculum and its delivery; (ii) improve access to NDRE programs; (iii) improve student support services and assessment; and (iv) introduce new pedagogical approaches to delivery of courses. 22. Project preparation was collaborative and benefitted from a deliberative process between MOE and the Bank. The Bank preparation and appraisal teams consisted of technical experts that were appropriate for the development of the program, included peer reviewers with knowledge and experience in implementing second chance education who provided sound guidance on the project design. Lastly, MOE and the Bank developed the project based on their experience from implementing the Education Sector Support Program4 (2008-2013), which was supported by the Bank and Australian government, and performed satisfactorily. This provided the preparation team with assurance that the ministry had the capacity needed to move forward with its equivalency program agenda. 23. Project Design and Quality at Entry. The project design was fully aligned with the government’s NESP 2011-2030, which provided the platform for sectoral planning and financing, the 2015 MDG goals, and the Bank’s Interim Strategy Note 2009-2010, as well as the draft CPS. The project design took into consideration comments from the peer reviewers during the concept and appraisal review meetings which included: (i) being more specific about the profile of the project beneficiaries (e.g. adult learners or out-of-school youth); (ii) focusing the project on general skills (literacy and mathematics) through an equivalency program rather that technical and vocational education (TVET) because other donors were supporting TVET and SCEP would be complimentary to these programs; (iii) limiting the results indicators; and (iv) strengthening the design related to the creation of CLCs through the use of grants and related project indicators. 24. The project design supported capacity development for the NDRE that had been established under the MOE for non-formal education services but had not been fully integrated into the formal MOE structure. The project design included support for the EP, which had been developed with external technical support after independence, and was a modular program that targeted older youth and adult learners. This modular program

4 The project was supported through two IDA grants (Original: IDA-H3100 and Additional Financing: IDA-H5830) and two trust fund grants from the Australian government (TF90564) for an equivalent total of US$20 million approximately. It provided support to all levels of the education system, including post-secondary, as well as capacity strengthening to MOE.

8

intended to recognize the existing skills of students including livelihood needs and to be delivered through the CLCs. These were important design features because the project supported the existing system with the goal of improving it. The original project design included four components with nine sub-components. As indicated, these components and sub-components were focused on: (i) revisions of the EP curriculum and training trainers on the delivery of the curriculum; (ii) improving the quality of service delivery by strengthening NDRE management, increasing teacher capacity to teach the curriculum, hiring and training new teachers and developing an EP promotional campaign to support community engagement; (iii) expanding local capacity for service delivery through the CLCs; and (iv) conducting studies to build the knowledge base for stronger non-formal education programs. These original activities were necessary for improving the NDRE’s delivery system as well as the overall goal of the project which was to increase the number of out-of-school youth who completed the equivalency program. The design also benefited from the inclusion of national and international consultants to work with the established management structure so as to build capacity which was a high priority for Timor-Leste as a post-conflict fragile state. The design identified risks associated with: (i) capacity constraints of NDRE to make quality improvements and manage the implementation of expanded services; (ii) hiring EP teachers because of shortages and then retaining the ones that were hired because of limited demand for EP programs; (iii) limited ability to produce information in a timely manner to document program progress as well as for monitoring and evaluation of the project; (iv) limited experience and knowledge of Bank procurement procedures that would delay key spending; and (v) limited financial management experience that could delay project accounting and audits. The mitigation measures included use of a group of national and international consultants to support NDRE and frequent monitoring from the Bank Dili and Sydney offices particularly related to procurement and financial management thereby reducing the overall operational project risk to moderate. 25. The original PDO was chosen (in part) based on an assumption that project activities would begin quickly and produce early project results that could be measured. However, in retrospect, there was limited knowledge, on the part of the design team regarding the actual content of the NDRE database system as well as the difficulties associated with data collection. This led to implementation issues in measuring PDO indicators. Moreover, the creation of 65 CLCs was an ambitious aspect of the design particularly given the limited capacity of the NDRE and limited experience of communities in participatory approaches to develop CLCs. Exogenous factors also affected the project. The government was still emerging from a fragile state status and had experienced several presidential elections that led to changes in government and new cabinet appointments. This meant that within MOE and NDRE there were: (i) three ministers and one interim minister of education during the period of June 25, 2012 to July 31, 2015; (ii) three new vice ministers were appointed in 2013; (iii) replacement of one director general; and (iv) replacement/appointment of two new director generals and some national directors, including vacancies that lasted approximately 10 months for the directorate responsible for SCEP. These changes and vacancies contributed to delays in important decisions need related to implementation and more specifically the overall project restructuring

9

26. The 2015 Level I project restructuring, which was initiated in 2014, provided the opportunity to enhance the potential effectiveness of the project design by revising the PDO to ensure that the objectives could be measured and achieved based on a better understanding of the availability of data. The restructuring also provided the opportunity to reduce the number of CLCs from 65 to nine and focus on developing CLCs that would demonstrate their effectiveness for both communities, districts and the MOE (i.e., putting a stronger focus on community participation and ownership that could serve as a model for other CLCs). Moreover, as Timor-Leste continued to emerge from a fragile state, MOE and NDRE wanted to change the direction of the SCEP to focus on developing a sound legal framework that would provide for a sustainable recurrent education system. Second chance education was initially covered under the organic law, and the restructuring presented a good opportunity to revisit and strengthen it to better meet the needs of the target population. This was particularly important given the increased need for educating overaged children and adults that had been denied education services during the years of conflict. At this time, a concerted effort was made to ensure that: (i) the revised indicators could be measured for PDO achievement; (ii) there was a link between project activities and the results framework; (iii) relevant data existed or could be collected by NDRE to assess and inform project performance; (iv) the number of CLCs was reduced so that the project supported CLCs could be viewed as model CLCs thereby demonstrating to other communities, within the districts, how CLCs could contribute to the successful implementation of the EP; (v) the portion of the curriculum development was shifted to a semi-presential5 delivery mode as opposed to a face-to-face mode which made pedagogical and efficiency sense for overaged and adult learners; and (vi) project resources were reallocated for the completion of the curriculum and information management infrastructure. These modifications are judged highly appropriate, within the country context at the time, particularly the focus on a legal framework that would provide the needed underpinnings for sustainability. 2.2 Implementation 27. Implementation details. The project became effective in April 2011 and by April 2012 the project implementation was already experiencing delays. This was due in large part to the fact that the NDRE had undergone an internal restructuring and key staff left NDRE for employment in other MOE units. There were also issues with: (i) insufficient government resources being allocated for project monitoring costs which prevented NDRE staff from supervising activities at the local level; and (ii) the weakness of basic data collection on the equivalence program due to the reorganization and limited collaboration between NDRE and the EMIS on EP data requirements. Moreover, the evaluation and assessment of the three established CLCs, that was required to move forward with the establishment of additional CLCs, had not been conducted due to limited resources and staff. This delay meant that other sub-components could not proceed because other activities were sequenced and prioritize based on the CLC assessment. Despite these

5 Semi-presential delivery mode is a combination of 15 hours of presential classroom sessions plus 10 hours of independent learning and consultations.

10

challenges, NDRE had been able to recruit a planning and management advisor (September 2011) and a curriculum development advisor (February 2012) who were key for the development of terms of reference (TORs) for recruiting new teachers, a pedagogic team, field coordinators and developing the relevant learning materials. 28. By December 2012, the Bank task team leader (TTL) was based in Jakarta that allowed for more frequent Bank interaction with NDRE. At this point, a thorough evaluation of the existing CLCs had been conducted with support from UNESCO and a CLC operations manual had been developed based on the assessment. Moreover, the NDRE had developed a short- and long-term plan for recurrent education curriculum development and a framework for development of pedagogical materials. However, more delays in hiring EP teachers and the pedagogic team occurred due to the mid-2012 parliamentary and presidential elections. Since the teachers and pedagogic team were to be permanent positions no final decisions could be made until the election results were final. This delay had implications on the percentage paid on salaries for contract teachers, teacher trainers and field coordinators stipulated in the financing agreement (FA) which led to the first project restructuring indicated in Section 1.7 above. At this point, progress toward achievement of the PDO was downgraded to moderately satisfactory because the 65 CLCs and teachers needed for the delivery of the EP were not in place and the EP Level 2 curriculum and learning materials were delayed. Data collection for project indicators was also experiencing challenges and NDRE was not able to provide information on progress toward achieving the PDO. 29. The July 2014 MTR confirmed that the SCEP project remained relevant with its support for the development of an equivalency program for out-of-school youth, but that a restructuring was needed to ensure the project could reach the desired development objectives. As indicated above, there were exogenous factors that contributed to the delays in decision-making needed for the restructuring. In addition, and as indicated in Section 1.7 above, there were delays associated with the MOF and Bank discussion related to Bank and government procurement regulations. Despite these delays, during the MTR, the Bank team continued to work with NDRE on the restructuring and by June 26 2015, the procurement differences had been resolved and the government sent the official letter to the Bank requesting the agreed project restructuring. However, the restructuring could not be sent formally to Bank management and the Board of Executive Directors until August 2015 because of outstanding documents that were required such as the revised implementation plan and budget as well as final agreement on the PDO and results framework. The exogenous political realities created challenges and delays for the formal restructuring process. Despite these challenges, significant progress was being made in the project outputs. The outputs at the time of the restructuring included, but were not limited to: (i) approximately 1,100 students completing the level 1 program; (ii) hiring an international curriculum development expert to help develop the new EP curriculum; (iii) trialing of the eleven EP Level II subjects in the new curriculum in six trial locations; (iv) completion of the CLC grant manual with approval by the Bank; (v) completion of a needs and demand study which included the hiring of enumerators; (vi) recruitment and training of a national pedagogical team (20 members); (vii) hiring of an international legal adviser to work with national experts on the EP legal framework; (viii) hiring of an M&E specialist

11

who was working on designing a comprehensive but easy to operate M&E system; (ix) hiring of facilitators to support districts in establishing CLCs using a community-based approach; and (x) agreement with NDRE that the EP Level 1 Phase 3 and EP Level II Phase 4 and 5 curriculum materials would be developed using a semi-presential delivery mode. These were significant achievements given early project delays and capacity constraints. 30. At the time of project closing, all project activities had been completed which was a substantial achievement. The NDRE, with support from international experts, had developed: (i) the EP legal framework which became a Decree Law that was approved by the Council of Ministers and promulgated by the president, (ii) a new EP curriculum with materials in Portuguese and Tetum; and (iii) a monitoring and evaluation system that could capture needed data for reporting on the EP system. New teachers had been hired, nine model CLCs had been created and at total of 1,670 level 1 and 2 students had participated in the equivalency programs at the nine model CLCs in addition to the ten sub-districts where MOE is establishing new CLCs in 2017. (See Annex 2 for complete listing of outputs.). Again, it should be noted that all of these positive results occurred within the context of the exogenous factors indicated above. 31. There were 12 Bank review and implementation support missions from April 2012 through December 2016. The makeup of the Bank teams was consistently appropriate and well balanced with financial management, procurement, safeguards and implementation specialists as necessary. The task team leader during implementation, was based in Jakarta, which allowed for frequent visits to Timor-Leste and regular interaction between the Bank, NDRE and MOE. This also provided the needed continuity for implementation of the program and contributed significantly to the project’s completion. The total project disbursements were approximately US$4.483 million or approximately 89.65 percent of the total project funds of US$5.0 million equivalent. Approximately US$517,000 was cancelled and returned to the Bank. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 32. Design. The M&E design was simple and two-fold. The first aspect of M&E design was the monitoring and reporting of data related to the project results framework. This data was to be gather during field visits because there was no electronic data gathering mechanism in place at the CLC local level. The NDRE management was responsible for the coordination and reporting of data bi-annually to MOE and the Bank. Secondly, the project was to support the updating and upgrading of the monitoring and evaluation system, setting an information baseline, and integration of EP data into the MOE EMIS system. 33. Implementation and Utilization. Reporting of data for the results framework indicators was a challenge throughout the first three years of project implementation. This was due, in large part, to the fact that NDRE did not have adequate transportation to conduct the field surveys to collect the data. In addition, there were difficulties in paying field surveyors per diems because MOE and MOF were reluctant to release the project funds for this exercise. This reluctance stemmed from concerns related to their ability to account for the resources. However, by October 2012 NDRE had developed two M&E

12

tools and had been able to conduct one M&E visit. In 2013, NDRE did not conduct field visits, for reasons stated above, but by May 2014 information was collected for EP Level I classes. The 2014 data was still not adequate to report on the results framework data. These challenges contributed to the need to restructure the project indicators to ones that could be easily measured based on NDRE M&E capacity and a more realistic understanding of the data systems capabilities. After the 2015 restructuring, NDRE was able to provide the data for the project indicators and update the results framework on a regular basis. 34. The NDRE also experienced challenges with finding and hiring the needed consultants to update and upgrade the monitoring and evaluation system, setting an information baseline, and integration of EP data into the MOE EMIS system. This was due in large part to the limited availability of consultants with the needed information technology (IT) experience in Timor-Leste. By early 2015, the NDRE had contracted two IT specialists (national and international) to support the development of the NDRE IT system. Because of time limitation it was agreed that the consultants would focus on: (i) identifying the basic information related to learners and teachers for the equivalency program to better serve management decisions; (ii) developing forms to collect data; (iii) developing the process for collecting information and data; and (iv) assisting NDRE in processing data, including entering, cleaning and extracting reports. 35. By project closing, there had been considerable progress in developing the NDRE M&E system. The new system was developed using the Ruby programing language and the Rails framework web based application that captured key indicators related to EP teachers and learners that and allowed the users to search and filter information. The consultants had: (i) developed a user’s manual; and (ii) provided on the job training for users at the NDRE and municipal staff comprised of coordinators, relevant EP teachers, CLC councils and CLC committees on how to use the system as well as how to upload data and generate reports. The project CLCs had used CLC grant to purchase laptops and a dongle device that allowed the CLC focal point to log into the system and enter data. This was a major improvement from the use of field surveyors for gathering data for the EMIS system. A system maintenance plan had been developed, arrangements had been made to host the system in the cloud and the process to integrate the NDRE system into the MOE EMIS system was agreed. The MOE has also committed to continue to pay for the internet connection to maintain the system. By project closing, all agreed M&E activities were completed and systems were in use. These are all substantial M&E achievements that helped lay a strong foundation for M&E of the equivalency program going forward. 36. There were also studies completed under the project to monitor and evaluate components of the overall equivalency program which were: (i) an assessment of the country’s readiness for distance learning that concluded Timor-Leste’s infrastructure was not suitable to accommodate distance learning interventions and a rapid assessment/interview with learners after which the NDRE decided to use the semi-

13

presential delivery modality within the project, and (ii) a needs and demand study6 to gain a better understanding of the population that the equivalency program serves and those who deliver the courses. The study was used by NDRE to develop curricular materials and for organizing EP teacher training. Moreover, it was used to confirm the demand for EP Level II courses. (See Annex 2, Component 4 discussion for details on the studies.) 2.4 Safeguard and Fiduciary Compliance 37. Safeguards. The project was rated a Category “C” operation, not requiring a separate assessment. However, OP/BP 4.01 “Environment Assessment” was triggered due to the minor construction activities related to rehabilitation of existing schools and community facilities. The maximum amount eligible for small works would be US$5,000-10,000 and the grant manual included details on how to address the needs for environmental precautions when carrying out small works. The environmental guidelines were incorporated in the CLC Grant Manual and a construction engineer was hired to ensure the application of safeguards related to small works. Safeguards were monitored consistently during project implementation and ISMs and were consistently rated satisfactory. 38. Financial management. Financial management remained satisfactory throughout the life of the project. This was due to the fact that the unaudited interim financial reports (IFRs) were submitted on time and the NDRE was in compliance with the financial covenants of the grant agreement. The required audits were submitted timely and in accordance with the financing agreement. The final project audit is expected to be completed in June 2017 and will be sent to the Bank. 39. Procurement. Procurement was consistently rated moderately satisfactory. This was primarily due to the fact that the NDRE did not extend contracts within the effective period so the continuation of the consultants’ service had to be treated as ‘single source selection’. MOE higher management level turnover also contributed to the changing of activities plan caused the delays is in decision making which hampered the timely processing of the procurement activities. Despite the delays, progress was made for record keepings, most of the procurement contracts were filed properly (with project code, PP code and contract number) and contract and performance review was being adapted and consultant deliverables were reviewed/verified by the Directorate level officials. 2.5 Post-completion Operation/Next Phase 40. The NDRE and MOE are committed to continuing support of the equivalency program, sustaining the investments as well as expanding the program. As indicated above, at the time of closing, a strong foundation had been developed for the EP that included: (i) an important EP legal framework which became a Decree Law that was approved by the

6 The study administered 5,822 questionnaires; visited 13 municipalities and 60 administrative posts; interviewed 131 local authorities; 31 directors, section heads and RE coordinators; as well as 87 EP teachers and 165 literacy teachers; and observed 248 classrooms.

14

Council of Ministers and promulgated by the president; (ii) a new EP curriculum with materials in Portuguese and Tetum; (iii) trained NDRE staff that can continue with the expansion of the CLCs to other communities; (iv) new pedagogic advisors and trained teacher; and (v) a new M&E system. In addition to the foundation, the MOE allocated FY 2017 budget to sustain the nine newly established CLC and to create an additional ten CLCs. The MOE also contracted the pedagogic team and they will remain with the NDRE or be attached to the MOE in-service training agency, INFORDEPE. Finally, as indicated, the MOE will continue to support the internet connection for CLCs so that M&E data can easily be entered into the system. With all of the project achievements, NDRE has enhanced its relevance as an education provider. 3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation 41. Relevance of Objectives and Design. The original objective of increasing the out-of-school youth who completed an equivalency program as well as decreasing the dropouts was substantially relevant to the country’s sectoral needs when the project was developed. The objectives fit into the government’s National Strategic Development Plan (NSDP) 2011-2030, the NESP 2011-2030, and recognized the importance attributed to increased capacity, improved institutions and attention to out-of- school youth articulated in the 2011 WDR on Conflict, Security and Development. It also fit into the goals of the country’s Education Strategic Plan 2010-2015 that focused on achieving universal completion of basic education by 2025 and elimination of illiteracy by 2015 among all age groups. The project objectives continue to fit into the FY2013-17 Country Partnership Strategy for Timor-Leste, which indicates that education will be enhanced through improved quality of service provision in education and skills development. However, the original design was: (i) ambitious with the inclusion of 65 CLCs considering the time needed to create a CLC with strong community participation, and (ii) the design team underestimated the data that the NDRE system could produce for measuring the PDO indicators as well as the NDRE’s ability to collect the needed data. As indicated above, once the political environment was more stable, the MOE/NDRE wanted to refocus the SCEP on developing a sound legal framework which necessitate revising the PDO. 42. The 2015 restructuring that restated the project objective and modified the results framework is judged to be of high relevance. The restructuring accommodated the realities of the country’s emergence from that of a fragile state and the recognition that a legal framework for NDRE would be important for expansion of the equivalency program. The restructuring also recognized the importance of continuing the completion of the equivalency curriculum, adapting the equivalency curriculum to adult learners through the semi-presential delivery mode, and developing a robust M&E system. The original design features, along with the inclusion of the legal framework, laid a solid foundation for NDRE to build on going forward. The 2015 restructuring also modified the project design by reducing the number of CLCs from 65 to nine which was more realistic given limited community capacity to create CLCs and the focus on creating model CLCs to demonstrate

15

their effectiveness in delivering the EP. Overall, the PDO and design adjustments made at the time of the 2015 restructuring were highly relevant. 43. Relevance of Implementation. As indicated above, the project implementation had challenges, both exogenous and endogenous. Despite the challenges, the NDRE did complete the project activities with support from the advisors and the Bank supervision teams. Considering the challenges and the fact that this was the first project to be implemented by the NDRE, this was a substantial achievement. The difficulties in providing data for project indicators was consistently acknowledge by the Bank team in aides-mémoires and implementation status and results reports (ISRs) which led to the progress toward achieving the PDO being rated moderately unsatisfactory. The Bank team and the NDRE worked on the restructuring and by project closing the PDO was rated moderately satisfactory and NDRE had successfully implemented the project activities. The achievements came with strong support from technical assistance related development of legal frameworks, curriculum development, and M&E, as well as the Bank team during technical missions and ISMs. 3.2 Achievement of Project Development Objectives 44. This section evaluates the outcomes against the project development objective and the results framework that was adjusted during the September 16, 2015 formal restructuring. The development objective was to support the Ministry of Education in the establishment of a sustainable national equivalency program. More details on project outputs can be found in Annex 2. 45. Support in the establishment of a sustainable national equivalency program was to be measured by the PDO indicators: (i) legal framework regarding recurrent education endorsed by Ministry of Education; and (ii) curriculum endorsed by the MOE. The equivalency program (EP) Decree Law was approved by the Council of Ministers 4/2016 and promulgated by the president in 7/2016 as Decreto-Lei n.o. 30/2016, de 13 de Julho. In addition, a ministerial diploma7 regulating the implementation of the approved EP was also approved by the MOE along with one dispatch8 on program certification. This exceeded the target of having the framework just endorsed by the MOE. Moreover, the MOE, with support from the legal advisor, decided to include the licensing and accreditation of the EP into the ongoing elaboration of ministerial diplomas on formal education regulating licensing and accreditation rather than creating separate diplomas for the equivalency program. This ensures that the equivalency program is aligned with the formal education program. These two diplomas have been approved by the MOE. 46. The curriculum of the EP was developed and trialed in January 2015 and June 2015 based on the project supported needs and demand study with the support of the international curriculum development consultant. Based on the trialing, the curriculum was

7 A mistrial diploma is a minister’s official documents on a decision and only a minister has the mandated to sign this a diploma. 8 A dispatch is letter of instruction that comes from a minister or relevant officials.

16

revised with portions (EP I Level 3 and EP II Level 4 and 5) converted from the face-to-face delivery mode to a semi-presential delivery mode so as to provide flexibility for adult learners. This new curriculum was approved by the MOE minister and vice-minister thereby meeting the indicator target. In addition, the curriculum was attached as an annex to the Decree Law that was approved by the Council of Ministers. Once approved, the new curriculum was implemented immediately in three new CLCs beginning in 2016. The six established CLCs finished the 2016 term with the old curriculum and began teaching the new curriculum in 2017. With the implementation of these additional activities, the target for the second PDO indicator was exceeded.

47. The intermediate indicators that supported the establishment of a sustainable national equivalency program were: (i) learning materials for equivalency program Level II endorsed by the MOE; (ii) NDRE staffed with teachers and pedagogic team; (iii) teacher and pedagogic team trained to deliver the equivalency program; and (iv) CLCs established, equipped and functioning. During implementation, the newly established CLCs also were able to provide EP courses for a total of 1,328 level 1 students and 342 level 2 students. 48. The learning materials for equivalency program Level II were endorsed by the MOE and then attached to a decree law on the equivalency program that was promulgated in July 2016 thus exceeded the target. As originally designed, the project: (i) developed new learning materials for 11 Level II subjects (grades 7-9 equivalent) with the support of 12 national technical and pedagogical specialists and 10 international curriculum specialists hired for 3-4 months; (ii) supported the printing and distribution of Level I materials (grades 4-6 equivalent) using government funds; and (iii) developed seven teacher instructional manuals for the nine courses. In addition, NDRE has been staffed with 20 technical pedagogic staff that are coordinating their work with the National Directorate of Curriculum on their curriculum work as well as the INFORDEPE on teacher training and has recruited a total of 157 teachers (97 for Level 1 EP and 60 for Level II EP) to deliver the new EP program thereby meeting both targets. The pedagogic staff also supported the development of the curriculum and learning materials and trained teachers to deliver the new equivalency program. 49. As previously indicated, the 2015 restructuring reduced the total number of CLCs to six with the possibility of establishing three more prior to project closing. By the time of project closing, there were nine CLCs established and fully functioning. Nine communities in eight districts were able to meet the pre-requisites for establishing CLCs, which were: (i) conduct community socialization with the use of DSFs to ensure stakeholder participation in the development of CLCs; (ii) submit official application letter to NDRE requesting an official registry of CLC with the government; (iii) DSFs conduct workshops within communities to socialize the concept of CLCs, identify community needs, and help write proposals for accessing CLC project grants (total grant US$20,000); and (iv) provide training to CLC councils and committees on how to manage the CLC grants well as how to sustain the recurrent education program. The MOE views the nine CLCs as models for further expansion as indicated previously.

17

3.3 Efficiency 50. SCEP provides a foundation on which the government can expand and address the challenges of second chance education in Timor-Leste and promote human capital formation and economic development. International literature demonstrates the socio-political importance of second chance education in post-conflict contexts, and household survey data confirms the importance of education in Timor-Leste’s economy. Second chance education programs, however, are difficult to implement effectively not just in terms of service provision but also economic efficiency. SCEP’s investment helps the government addresses these economic challenges in its equivalency program in a manner that is both scalable and sustainable: this is demonstrated by the government’s commitment to continue to fund and expand the community learning centers. The economic and financial analysis also reveals a potentially positive rate of return and, albeit challenging based on international evidence, does seem to suggest that the project could have a positive impact on the lives of second chance education beneficiaries (see Annex 3). As a result, SCEP’s investment (especially through the post-restructuring design) has helped improve the effectiveness of Timor-Leste’s equivalency programs and benefits not just those who have obtained access during the project’s tenure but also those who will gain access as a result of the government’s scaling up. In terms of rating, the pre-restructured efficiency is rated modest while post-restructured efficiency is substantial: the restructured design was able to support additional activities within the existing budget envelope, while being highly relevant to Timor-Leste and critical for the establishment of a quality second chance education program that has the potential to deliver positive economic benefits. As a result, overall efficiency is rated modest. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 51. The overall outcome rating of the project is moderately satisfactory. The relevance of the objective pre-restructuring is rated substantial. The original project objective of increasing the numbers of out-of-school youth and young adults who completed a recognized equivalency program was and continues to be an important long-term goal for Timor-Leste. Although the overall project design did not include development of a legal foundation, the larger objective continues to be highly relevant for Timor-Leste. The pre-restructuring design is rated modest because of the ambitious nature of creating 65 CLCs and the limited understanding of the capabilities of the NDRE data system to provide needed PDO indicator data. Although activities related to the legal framework were not included, the project activities were all important for the development of a sustainable equivalency program in the context of Timor-Leste. By 2014 Timor-Leste was entering a more stable political environment and it was at that stage that the MOE, NDRE and Bank implementation team recognized the timing was appropriate for ensuring the fundamental legal underpinnings of the equivalency program. Based on this, the Bank worked with government to restructure the project with objectives that supported this redirection. Based on this, the 2015 restructuring objective is rated high and design is also considered highly relevant. The design, as indicated, was also fully aligned with the NSDP,

18

NESP, the CPS (2013-17) and fit into the WDR on Conflict, Security and Development. Although implementation faced many challenges, the project activities were implemented and a strong foundation for the equivalency program has been fully established. This is a considerable achievement for a country that became independent in 2002 and was designated as a fragile state. Based on this, overall efficacy is rated substantial. During the pre-restructuring period efficacy is rated modest. This is primarily because data for project indicators could not be collected systematically because the collection was dependent on field surveys which proved challenging as indicated above. However, once the project was restructured indicators could be monitored and, by project closing, all PDO and intermediate indicator targets had been met or exceeded. Efficacy for the post-restructuring period is rated substantial because of the important achievements of: (i) establishing the legal framework for the equivalency program that aligns it with the formal education sector; (ii) developing and approving the required EP materials; and (iii) establishing model CLCs as the delivery modality for the equivalency program all within the exogenous factors presented above. Finally, as indicated above, the overall project efficiency rating is modest. Table 2: Project Rating Original Project–12/7/2010–9/16/ 2015–56.00 percent disbursement-net project funds1

Project Relevance Achievement of PDO (Efficacy) Efficiency Overall Rating Substantial Modest Modest Moderately

Unsatisfactory

Restructured Project–9/17/2015–12/31/2016–44.00 percent disbursement-net project funds1

Project Relevance Achievement of PDO (Efficacy) Efficiency Overall Rating High Substantial Substantial Satisfactory

Overall Project Ratings – 89.6 percent disbursement of total grant as of 12/31/2016

Project Relevance Achievement of PDO (Efficacy) Efficiency Overall Rating2 Substantial Substantial Modest Moderately

Satisfactory2

1The net amount for the total project is US$4.483 million 2 Overall rating = (3)(.5600) +(5)(.4400) = 3.88 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 52. The current enrollment represents 57 percent female students. Although it is difficult to quantify the project’s impact on poverty reduction, gender and social development, it is believed that the new curriculum will greatly benefit future NDRE students who will have access to more relevant quality education and will have equal opportunities for further education and jobs as those studying in the formal school system. According to the Government’s 2014 information, the net enrollment rate of students in primary (grades 5-6 which are the grades that SCEP focused on) is 38.9 percent and the gross enrollment rate is 81.6 percent. The major issues contributing to this gap are that many students started school as older students due to the conflicts prior to 2002

19

independence and the 2006 conflict. The NDRE’s relevant and equivalent curriculum could therefore benefit the overaged students who missed the opportunity enroll. (b) Institutional Change/Strengthening 53. The project resulted in strengthening the capacity of the NDRE pedagogical team to the extent that they are now able to: (i) assist in the modification of the equivalency program going forward; (ii) train teachers on the semi-presential delivery modality; and (iii) monitor the training/teaching activities within the CLCs. These are significant achievements for a team that had never taken on such activities prior to the development of the equivalency program. (c) Other Unintended Outcomes and Impacts (positive or negative) 54. Anecdotal findings based on site visits by the Minister and Vice Minister of MOE as well as directors of the relevant directorates show that communities now feel a sense of ownership of the CLCs and the CLC programs because of their role in developing them. Moreover, some districts now view these CLCs as incubators for young entrepreneurs. Again, this is a strong indication that the project provided a solid foundation for the sustainability of the CLC activities. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not Applicable 4. Assessment of Risk to Development Outcome Rating: Moderate 55. The original project preparation identified the overall risk rating as substantial. The ratings can be summarized as: (i) high at the country level due to government budget management challenges thereby leaving NDRE with inadequate resources for expansion of the equivalency; (ii) substantial at the operational level based on limited capacity for making quality improvements for the equivalency program, monitoring and evaluation, and being able to hire and retain teachers for the equivalency program; and (iii) overall substantial for procurement and financial management because of limited capacity as well as experience with Bank-supported projects. As indicated previously, the risks were appropriately identified at the project level. However, the team did not identify the impact the exogenous political factors would have on slowing down project implementation. Going forward, the overall risk at the country level is likely to be substantial. The Bank has worked with the government on a Fragility Assessment that will be used to inform program design in the future. At the MOE level, the risk is likely to be moderate. This is based on the fact that the SCEP project has made a substantial contribution to capacity development within the NDRE and has been responsible for establishing a strong foundation (both legal and structural) for the NDRE and equivalency program. As indicated above, the MOE allocated FY 2017 budget to sustain the nine newly established CLC and

20

to create an additional ten CLCs, retain the EP level I and II teachers and have contracted the pedagogic team and they will remain with the NDRE. With all of the project achievements, NDRE has enhanced its relevance as an education provider. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 56. The project preparation team ensured that the project design supported all of the government strategic plans and the Bank’s CPS and 2011 WDR. The design was informed by the peer reviewers at both the project concept note and appraisal stage of project preparation. As mentioned previously, the design included support for the EP using the existing system with the goal of improving it. The original activities were necessary for improving the NDRE’s delivery system with the support of national and international so as to build capacity which was a high priority for Timor-Leste as a post-conflict fragile state. The preparation team also identified the appropriate project risks. However, the design team did not properly understand the limitations of the NDRE data system, which led to problems with gathering the needed data for reporting on project indicators. The project restructuring enhanced the design by: (i) supporting the development of a legal framework for the equivalency program thereby providing a strong foundation for future sustainability; (ii) improving the NDRE data system and data collection; and (iii) developing a semi-presential curriculum and materials to allow adult learners to work and attend school at the same time. (b) Quality of Supervision Rating: Satisfactory 57. The Bank conducted 12 implementation support missions, which included a mid-term review. The project team was actively engaged in supporting the government in its efforts to implement the project. Whenever implementation challenges arose, the Bank team worked with government to find solutions. The Bank team recognized at mid-term that a restructuring would be necessary and provided constant and consistent support to ensure that the restructuring was completed despite the various institutional processes and procedures associated with a restructuring. The 2015 restructuring modified the design to ensure that the project provided a strong foundation for future delivery of the equivalency program. Of particular importance was the Bank’s team active involvement in modifying the PDO to focus on the needed regulatory system. All of the 2015 modifications were highly appropriate. The Bank team also provided strong support for the curriculum development by encouraging the development of semi-presential materials for adult learns based on the finding of the demand and supply study. The Bank teams consistently reported on financial management and procurement progress and worked with the NDRE teams to build their capacity in these areas. The team also systematically documented project progress in aide-memoires, back-to-office reports, and Implementation Status Reports

21

(ISR), all of which kept Bank management informed of progress and provided the foundation for the ICR analysis. The Bank team’s performance and support to the NDRE was a critical component for the achievement of the project goals. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 58. Based on the analysis above, the overall Bank performance is rated moderately satisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 59. Overall government performance was uneven over the life of the project. During the project preparation, the government showed commitment to the project. However, with the presidential elections in 2012, there was frequent turn-over of ministers and staff in key positions throughout the project implementation. This led to early delays in implementation and hiring of the needed project advisors. The project restructuring was also delayed due to NDRE management vacancies and government procedures related to restructuring Bank support projects. However, the government did work to ensure the legal framework was finalized, a formal decree issues and promulgated by the president. This was substantial support by the government for the project goals. The government has also provided increased budget to NDRE to both sustain the project investments and to expand the CLCs that are used to teach the equivalency program. Based on this, government performance is rated moderately satisfactory (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 60. NDRE was responsible for managing implementation and the SCEP project was the first Bank-support project the directorate had managed. The capacity of NDRE was limited and needed to rely on technical assistance to help with design and development of the new equivalency program curriculum and materials. Exogenous factors and limited capacity slowed curriculum development, procurement, reporting and overall implementation. After the project was restructured in 2015, NDRE was able to successfully complete the curriculum development aspects of the project, get the curriculum approved by MOE and, in the process, build capacity. Moreover, they were able to successfully complete the legal framework for the equivalency program and ensure the MOE and Council of Minister approved the framework. These are all significant achievements. Based on these aspects, and the analysis in other section of the ICR, the implementation agency performance is rated moderately satisfactory. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory

22

61. Based on the analysis and ratings above, the overall Borrower performance is rated Moderately Satisfactory. 6. Lessons Learned 62. Lesson 1. A legal framework increases the likelihood of program sustainability. The government and Bank recognized that the NDRE needed a strong legal framework to ensure the continuation and expansion of the equivalency program once the SCEP project closed. The project supported the development of a recurrent education law which is now used as the basis for increased EP resources and program expansion. 63. Lesson 2. Needs and demand studies are important for supporting project design decisions. The SCEP project supported a needs and demand study that led to the decision to use a semi-presential delivery mode for level 2 and to develop learning materials for this approach at the time of restructuring. The study also provided the needed data and evidence to assist the NDRE in making the final decision to make the design change from face-to-face to semi-presential. 64. Lesson 3. Semi-presential approach provides opportunities for students to continue working while studying thereby improving the potential for increased economic benefit. There are limited studies that provide global evidence to the economic benefit of second chance education. However, for those studies that do exist, the rate of return to second chance education, increases as the opportunity costs are reduced. The SCEP project modified the teaching approach from face-to-face to semi-presential prior to the project restructuring in an effort to provide more flexibility for working students. 65. Lesson 4. Lack of sectoral knowledge and limited capacity can lead to implementation delays. When the SCEP project was designed, the NDRE had limited experience and capacity in: (i) the skills needed to ensure community participation in the creation of CLC; (ii) development of equivalency curricula; and (iii) monitoring and evaluation. This limited knowledge led to implementation challenges. The SCEP project provided strong technical assistance to the NDRE that helped build their capacity in these areas thereby ensuring sustainability of the equivalency program. 66. Lesson 5. Task team leaders based in close proximity to post-conflict countries is essential for ensuring the implementation of projects. The assignment of a task team leader based in Jakarta allowed for frequent visits to Timor-Leste so that problems could be resolved quickly. The presence of the TTL in Jakarta provided invaluable support and monitoring of project implementation. Experience with other Bank-supported project in post-conflict countries has shown that having the TTL nearby and/or in-country with quick accessibility is critical for ensuring timely project implementation.

23

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies See Annex 7 for borrower portion of the ICR. (b) Cofinanciers Not Applicable (c) Other partners and stakeholders Not Applicable

24

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent)

Components Appraisal Estimate

(USD millions)

Actual/Latest Estimate (USD

millions)

Percentage of Appraisal

Component 1: Curriculum Development and Production/Distribution of Instructional Materials

1.875 1.783 95.07

Component 2: Improve Quality of Service Delivery

1.283 1.8551 144.56

Component 3: Expansion of Local Capacity for Service Delivery through Community Learning Centers

1.300 0.304 2 23.35

Component 4: Project Management and Studies

0.217 0.5423 250.00

Total Baseline Cost 4.675

Physical Contingencies 0.091 Price Contingencies 0.234

Total Project Costs 5.000 4.483 89.65 Total Financing Required 5.000 4.483 89.65

1 When the project was restructured, there was a reallocation of resources from component 3 to component 2 so that training NDRE staff could be increased. 2 When the project was restructured, the number of CLCs was reduced from 65 to nine. As indicated, some of the resources were reallocated to component 2 and some to component 4. 3 When the project was restructured, project management activities under component 2 were moved to component 4 along with the requisite budget allocation. The additional activities under this component were to support: (i) additional NDRE positions for financial management, procurement, project coordination and an education strategic adviser; and (ii) an audit firm.

(b) Financing

Source of Funds Type of

Cofinancing

Appraisal Estimate

(USD millions)

Actual/Latest Estimate

(USD millions)

Percentage of Appraisal

Borrower 0.00 0.00 0.00 IDA Grant 5.00 4.83 89.65

25

Annex 2. Outputs by Component 67. As mentioned previously, there was a re-design of the components at the time of the September 2015 restructuring. This annex provides the output achievements based on the restructured component design. 68. Component 1: Curriculum Development and Production and Distribution of Instructional Material. (US$1,968,318 equivalent) The original activities under this component were not modified during the restructuring. These activities included: (i) development of new curricula; instructional materials, and teachers manuals for equivalence level II programs, including distance learning; (ii) training of trainers to deliver the equivalence Level II programs; (iii) establishment of small school libraries at each community learning center established by DNEAENF; and (iv) printing and distribution of textbooks and instructional materials for primary programs, equivalence level I programs and equivalence Level II programs, to support the curriculum revisions. The 2015 restructuring added: (i) trialing of a select number of EP level I and II courses; (ii) provision of support to the MOE task force to adjust the curriculum based on trial results and finalizing the curriculum; and (iii) MOE endorsement for the national roll-out of the EP level I and II courses. 69. The indicator used to measure component progress was: learning materials for equivalency program Level II endorsed by the MOE. The target was met for this intermediate indicators in that it was endorsed by the MOE and was attached to a decree law on the equivalency program that was promulgated in July 2016. 70. Subcomponent 1.1: Plan and Prepare NDRE curricula for Equivalence Level II-Phase 4 and 59. The activities related to planning and preparing the NDRE curricula for Phase 4 and 5 were successfully implemented. As originally designed, the project: (i) developed new learning materials, based on a face-to-face delivery method, for 11 Level II subjects (grades 7-9 equivalent) with the support of 12 national technical and pedagogical specialists and 10 international curriculum specialists hired for 3-4 months; (ii) supported the printing and distribution of Level I materials (grades 4-6 equivalent) using government funds; (iii) developed seven teachers instructional manuals for the nine courses; (iv) conducted three curriculum discussions workshops to validate the curriculum which was attended by INFORDEPE, SEFOPE, UNESCO, UNICEF, NGOs and religious institutions; (v) conducted a curriculum workshop that launched reform process with the 12 national pedagogical specialists that aimed to familiarize the district coordinators with the new curriculum. The 2015 restructuring included: (i) translation of materials for trialing of EP Level 1 and II materials; (ii) trialing of a total of 11 subjects for EP Level I (Phase 3) and EP Level II (Phase VI and VII) subjects that included two rounds of

9 Equivalency Program Level 1 originally consisted of Phases 1-3 and Level II of Phases 4-7. However, during the modification of the curriculum the government decided to collapse the Level II phases so that Phases 4 and 5 became Phase 4 and Phases 6 and 7 became Phase 5.

26

monitoring/classroom observations using materials translated into Tetum involving 262 students (see Tables 4 and 5 for details on EP levels and phases); (iii) production of an equivalence implementation manual10 for use by NDRE and teachers.

71. After the July 2014 mid-term review, it was learned that from the monitoring of the trial that students were dropping out11 of the EP program. Based on this, the project teams added an additional activity that introduced a “needs and demand” study related to the EP program. (See Component 4 discussion of the needs and demand study.) The findings of both the study and monitoring activities led the project to modify the curriculum to a semi-presential delivery modality (i.e. combination of 15 hours of presential classroom sessions plus 10 hours of independent learning and consultations). The semi-presential materials only applied to EP Level II Phases 4 and 5 and EP Level 1 Phase 312. The change in delivery method meant that all EP Level II subject materials were revised for semi-presential delivery and Level 1 Phase 3 materials were revised. These materials were printed in both Portuguese language and Tetum. In addition, the curriculum was further modified by collapsing the Level II phases, as indicated above, and creating three areas of learning; seven mandatory subjects (three units each of Portuguese, English, Tetum, mathematics, natural sciences, social sciences and vocational training) and three elective subjects (civics education, art and cultural education, and skills for life and work). (See Table 5 for EP Level II structure.) Table 3: Overall Structure of the new curriculum

Areas of Learning Components of Curricular

Development of Linguistics Tetum literacy Portuguese Literacy Oral Linguistics Support

Scientific Development Mathematics Natural Science Social Science l

Personal Development

Arts and Culture Health Physical Education Religion

10 The implementation manual includes: (i) goals and purposes of teaching and learning; (ii) texts and activities based on socio-cultural realities of the country; (iii) text style (language, words and idioms) easy to understand and reflecting the reality and context of the country; (iv) images, drawings and photographs representing the country; respect of individuals and country’s culture; and language that respects the diversity of color, race, gender and religion. 11 The number of Level I participants had dropped from 79 to 19 and Level II participants from 183 to 100 at the time of the second round of monitoring visits. 12 It was agreed that EP Level 1 Phases 1 and 2 would be only face-to-face because students entering the equivalency programs had mostly just graduated from literacy programs and needed more intensive interactions with teachers.

27

Table 4: Curriculum Structure for EP Level I Phases 1-3

Areas of Learning Subjects Phase 1 Phase 2 Phase 3 Linguistic Development Portuguese 3 3 3 Tetum 3 3 3 Scientific & Social Development

Mathematics 4 4 4

Natural Science 3 3 3 Social Science 3 3 3 Personal Development Vocational Course 4 4 4 Learning Load (Total Hours) 20 20 20

Table 5: Curriculum Structure for EP Level II Phases 4-5

Areas of Learning Subjects Phase 4 Phase 5 Linguistic Development Portuguese 3 3 Tetum 3 3 English 2 2 Scientific & Social Development Mathematics 4 4 Natural Science 4 4 Social Science 3 3 Personal Development Elective Subject 2 2 Vocational Course 4 4 Learning Load (Total Hours) 25 25

72. Subcomponent 1.2: Preparation and implementation of staff training for NDRE personnel and regional users. This activity was to complement activity 1.1 by training staff and teachers to support the new equivalence programs. During the project period, there were 20 national pedagogical advisers hired by the project which became the pedagogic team. These pedagogical advisers trained a total of 50 EP Level I teachers and 60 EP Level II teachers to deliver both the face-to-face and semi-presential courses as required by the new curriculum. Moreover, and as stipulated by the financing agreement, the pedagogic team and 110 teachers were all being funded by MOE as of January 2016. This is an important outcome of the project for sustainability. 73. This sub-component also hired and trained nine district SCEP facilitators (DSFs) who provided support to communities in the establishment of the CLC, helping organize meetings with EP teachers and village leaders as well as providing socialization of the EP program and supervision of EP teacher’s delivery of classes. In addition, the project trained and used the 13 district coordinators and the 9 DSFs in surveys that were used for the needs and demand study. 74. Subcomponent 1.3: Establishment of small libraries at each community learning center established by NDRE. The project also provided books for the nine CLCs to create small reference libraries. In addition, the government purchased learning kits for the nine CLCs. The books were intended to complement the EP learning materials/textbooks. The books purchased were: (i) seven titles for mathematics,

28

Portuguese, Tetum, natural and social sciences subjects; (ii) four titles for English, history, geography and life-skills; (iii) two titles for moral and civics, art and culture subjects; (iv) atlases; and (v) dictionaries and grammar books for Portuguese, Tetum and English.13 The titles for the books was identified by a technical team established by NDRE. 75. Subcomponent 1.4 Procure and distribute textbooks. The final learning materials were completed by November 2016. It was agreed with the Vice Minister of Education and Director General for Pre-School, Basic Education and Recurrent Education that the NDRE would print enough books for students entering EP Level 1 in early 2017 so as to be available for students entering the program. 76. Component 2: Improve Quality of Service Delivery. (US$1,347,780 equivalent). The aim of this component was to assist in improving the overall quality of services provided by NDRE by hiring consultants to work with NDRE in building capacity and transferring knowledge. The original activities under this component were not modified during the 2015 restructuring. However, the restructuring did add an important activity to this component which was to develop a regulatory framework for recurrent education that would clarify: (i) roles and functions among the NDRE and district offices; (ii) management, accountability and ownership of CLCs; (iii) ways for NDRE to secure budget for recurrent education programs; and (iv) personnel status of EP teachers. 77. The intermediate indicators used to measure sub-component progress were: (i) NDRE staffed with teachers and pedagogic team, and (ii) teachers and pedagogic team trained to deliver the equivalency program. NDRE has been staffed with a total of 20 technical staff who have become the pedagogic team for the equivalency program. These individuals supported the development of the curriculum and learning materials as well as training teachers to deliver the new equivalency program. They are now working for the NDRE and coordinate with National Directorate of Curriculum on their curriculum work and with the INFORDEPE on teacher training. In addition, NDRE recruited 97 teachers for the equivalency program Level 1 and 60 teachers for the Level II equivalency program. For both indicators the targets of teachers and pedagogic teams working, training and delivering the equivalency programs were met. 78. Subcomponent 2.1: Upgrading quality of services provided by NDRE. This sub-component was aimed at improving the capacity of NDRE (formerly the DNEAENF) both in terms of legal, technical and managerial. The project supported creation of a pedagogic team that was able, with the support of consultants, to: (i) develop the EP curriculum and learning materials; (ii) train EP teachers; and (iii) provide technical supervision of the CLCs. In addition, the project supported several consultants that were specifically hired to support capacity development and to transfer knowledge relevant to

13 It should be noted that the PAD proposed to procure 200 library books for each CLC that was established. However, once the CLCs were established they discovered that space was limited and that they might not be able to purchase that many library books. Therefore, the decision was made to give CLCs some flexibility in the number of titles they want to buy based on their needs and space available.

29

maintaining the equivalency program. Table 6 provides details on knowledge transfer activities. Finally, the project supported the development of a regulatory framework for the equivalency program. The related decree law was approved by the Council of Ministers on April 26, 2016 and promulgated by the president’s office. Table 6: Knowledge Transfer

SCEP Consultants Knowledge Transfer Activities Curriculum Development Specialist Helped in establishing NDRE capacity to

deliver EP through recruitment and training of 20 pedagogy team members. As a result, NDRE has a skilled and well-trained pedagogy team that is ready to perform the following tasks: plan, implement and supervise EP based on the officially approved curriculum; train EP teachers, including semi-presential modality; and prepare and administer examinations. In addition, they were also engaged in the development of EP learning materials (text books) with international authors. The Specialist, assisted by the pedagogy team recruited and trained 60 EP Level II teachers and trained of 50 EP Level I teachers.

Recurrent Education Specialist Consultant

Worked closely with the director of NDRE and staff with: (i) training of six District Support – facilitators (DSFs) on strategies and approaches to empower and engage communities in the establishment of CLCs, proper and accountable use of the CLC grants provided by the project, and implementation of the EP; (ii) the introduction of the concept of CLC to the district and sub-district administrators/coordinators, village authorities and figures and trained them on how to build awareness and consensus with the community in establishing CLCs; (iii) training of CLC committees and councils on the CLC grant management and how to maintain and sustain the EP and other recurrent education activities; and (iv) arranging a number of study visits for the MOE officials, including Director General of Pre-school and Basic Education, NDRE Director and staff, and

30

the Directors of District Education Offices to Indonesia and Thailand to learn about implementation of EP and other recurrent as well as options to manage CLCs.

Monitoring and Evaluation Consultant jointly with the Management Information System/Information Technology Consultant

The consultants facilitated the establishment of Monitoring and Evaluation system and manuals and provided training for NDRE staff, sub-district coordinators and CLC representatives. The training was aimed at building skills and knowledge on M&E tools, instruments and methods for data collection, analysis and reporting.

Legal Advisor The legal advisor led the process of drafting the Decree Law of Equivalency Program and ministerial diplomas and dispatches to operationalize the Decree Law. The advisor also briefed the NDRE officials and staff with relevant legal matters, including the Ministry’s organic law.

Information Technology Consultant The consultant trained 21 NDRE staff in practical skills on the use of internet and email to support their daily routine work in NDRE. In addition, the assigned staff of NDRE received practical training on data insertion, data analysis and system maintenance.

Project Coordinator The consultant organized management training. The training targeted NDRE Director, Department Heads, and staff, District Coordinators and CLC Presidents with knowledge to lead and manage the implementation of EP programs and to sustain CLCs.

79. Subcomponent 2.2: Program Outreach. The project outreach programs included: (i) establishment of an advocacy and outreach team that developed radio and TV announcements about the EP program that aired on radio and TV for the opening of the EP registration; (ii) TV announcements for hiring the technical and pedagogical specialist; (iii) redesign, printing and distribution of 1,300 brochures that provided an overview of NDRE’s programs (both Level I and II) and how the programs fit with the formal school system, technical and vocational streams as well as the literacy and numeracy programs; and (iv) preparation of a CLC advocacy program for Level 1 and II that was used at the community level to support community involvement in establishing CLCs. In addition to socialization activities and distribution of brochures, NDRE has also invited media (radio,

31

television and newspapers) to attend most workshops and training activities. In addition, the NDRE used direct visits to the communities and local authorities in targeted villages/sub-districts to gain support for the equivalency program. All substantial accomplishments. 80. Subcomponent 2.3: Upgrading Monitoring and Evaluation capacity of NDRE. There are two areas of capacity development related to this subcomponent: (i) information technology capacity development and (ii) monitoring and evaluation capacity development. In the area of information technology capacity development the project conducted an in-depth review of the current Education Management Information System (EMIS) that included existence of data validation function, security interface user friendliness. Based on the review, the local IT consultant, in close collaboration with the international consultant, developed a new, secure database, which uses the Ruby programming language and the Rails framework for web based applications. The system was developed to allow for the capture of EP data and reporting against key indicators, and allows the user to search and filter information by a number of criteria. They also developed a user manual and trained national and CLC staff on how to use the system as well as how to upload data and generate reports. A system maintenance plan was developed and it has been arranged to host the system up on the cloud. MOE EMIS staff were involved in all of these activities and are now responsible for maintaining the database at the national level. In addition to database completion and development of user manual and maintenance plan, an on-the-job training was provided to municipal staff comprised of coordinators, relevant EP teachers, CLC Councils and CLC Committees. 81. Monitoring and Evaluation Capacity Development. The international M&E advisor conducted a comprehensive review of NDRE’s M&E system that was aimed at helping NDRE make use of its EMIS to better monitor progress and manage information quality and use for non-formal education. As a result, the consultant prepared an M&E plan14 for NDRE which was based on the “program logic framework” as well as the “performance assessment framework”. Based on NDRE’s information needs, three education process areas were identified and for which data collection tools/modules were developed. They were as follows: (i) “learner record” to track registration, enrollment, transfer and results; (ii) “teacher record” covering initial teacher appointment and assignment, as well as performance score and attendance; and (iii) “institution record” to report on institutional details, teaching programs, inventories and grant use/implementation milestones. The M&E adviser also finalized the revised data collection forms covering all three process areas that are being used for the new curriculum rolled-out. It should also be noted that an M&E manual guiding data collection and use of forms was completed, translated from English into Tétum, and is currently being used. The team also designed a CLC Grants Management Information System that will allow NDRE to track all grants – from proposal stage to the close of grant.

14 The M&E plan document described the program goals, outcomes and outputs, and outlined the key evaluation questions, indicators and targets, and staff workflow.

32

82. Component 3: Expansion of Local Capacity for Services Delivery through Community Learning Centers. (US$1,456,000 equivalent) This component originally aimed to provide small community grants15 to communities for establishing community learning centers. The goal was to have a total of 65 CLCs, at least one in each sub-district in the nation. As previously indicated, the 2015 restructuring reduced the total number of CLCs to six with the possibility of establishing three more prior to project closing. By the time of project closing there were nine CLCs established and fully functioning. Nine communities in eight districts were able to meet the pre-requisites for establishing CLCs which were: (i) conduct community socialization with the use of DSFs to ensure stakeholder participation in the development of CLCs; (ii) submit official application letters to NDRE requesting an official registry of their CLC with the government; (iii) ensure DSFs conducted workshops within communities to socialize the concept of CLCs, identify community needs, and help write proposals for accessing CLC project grants (total grant US$20,000); and (iv) provide training on how to manage the CLC grants to CLC councils and communities as well as sustain the recurrent education programs. There were also at total of 1,670 level 1 and 2 students trained by the CLCs prior to project closing. (See Table 7 for details.) The project also developed a CLC grants manual. By project closing, the NDRE had allocated FY 2017 budget to sustain the nine CLCs and to establish an additional ten CLCs. Table 7: Equivalency Program Students Level 1 Level 2 Phase I, II, III Phase IV Phase V Table 1328 187 155 % Female 55 70

83. Component 4: Project Management and Studies. (US$227,903 equivalent). The original title of the component was Information, Monitoring & Evaluation Technology - Pilots and Studies with the aim of financing instructional, information and office related technology to support the establishment and expansion of capacity within NDRE as well as M&E activities. At the time of the 2015 restructuring, the title of this component was changed to Project Management and Studies because the information technology and M&E activities were being implemented under Component 2. In addition, there were some project management activities that were moved from component 2 to component 4. 84. The project provided NDRE offices with 21 desktop computers, one projector, five printers, 22 A/C units and routers as well as internet access. The project also supported the implementation of an asset management system with the ministry’s logistics staff monitoring assets and sharing the information with all concerned departments.

15 The grants were originally expected to provide a ‘small community grant’ of US$12,000 to US$20,000. There was an indicative amount of up to US$10,000 for minor construction and rehabilitation of infrastructure for existing facilities including water supply and then up to US$5,000 for equipment and initial supplies.

33

85. Subcomponent 4.1: Distance Learning Pilots. The project hired a consultant to assess the country’s readiness for distance learning. The consultant: (i) reviewed existing multi-media/audiovisual tools and infrastructure to support the tools; (ii) worked with legal advisor and NDRE officials to determine how the use of multi-media for distance education fit within the country’s basic education law, the ministry’s organic law and the NESP; and (iii) reviewed and collected experience from other countries and NGOs with experience in using multi-media for distance education. The conclusion of the report was that Timor-Leste’s infrastructure was not suitable to accommodate distance learning interventions that would be meaningful and effective for potential learners. It was at this point that NDRE decided to use the semi-presential delivery modality within the project. 86. Subcomponent 4.2: Studies to Build the Knowledge Base for Stronger Non-formal Education Programming. The project supported a needs and demand study16 to gain a better understanding of the population that the equivalency program serves and those who deliver the courses. The project hired a research specialist and 26 enumerators to conduct the study and the results were provided in a final report that was produced in October 2015. Some of the report findings were that: (i) classrooms were being shut due to teacher absenteeism resulting in students losing interest; (ii) students reported “increase knowledge as their main motivation to attend EP courses (97.4 percent) followed by certification (87.5 percent) and getting a job (73.9 percent); and (iii) teacher capacity is low (especially Level 1 teacher) and required a support structure to strengthen their teaching skills. The study was used by NDRE to develop curricular materials and for organizing EP teacher training. In addition, the study was used to confirm the demand for the EP level II.

87. Subcomponent 4.3: Project Management. At the time of restructuring several project management staff from the GPE-supported Management Strengthening Project were moved to the SCEP project to increase the overall capacity of the project management unit. The positions were: (i) project coordinator, (ii) procurement consultant, (iii) procurement assistant, (iv) senior finance officer, (v) finance officer/clerk, and (vi) education strategic advisory. Overall project management functioned well with routine updates and progress reports produced on time, coordination of activities was progressing and other departments within the ministry were being updated. All-in-all, the project activities were implemented per the agreed plan.

16 The study administered 5,822 questionnaires; visited 13 municipalities and 60 administrative posts; interviewed 131 local authorities; 31 directors, section heads and RE coordinators; as well as 87 EP teachers and 165 literacy teachers; and observed 248 classrooms.

34

Annex 3. Economic and Financial Analysis 88. Timor-Leste’s equivalency program is expected to be crucial not just for improving individual human capital but also building the institutions and social capital necessary for long-term economic development and poverty reduction. The role of education in a post-conflict context to strengthen institutions and promote peace building and conflict transformation is well researched (e.g.: see Smith 2010; Smith and Vaux 2003; Miall 2004; Davies 2010). Education enables individuals to participate effectively in the governance of their communities and institutions, and it provides “means of socialization and identity development through the transmission of knowledge, skills, values and attitudes across generations” (Smith 2010). Like many post-conflict countries, Timor-Leste is plagued with a significant proportion of its population having not completed primary education. The equivalency program aims to improve education outcomes for these individuals. 89. Individual earnings are positively associated with education in Timor-Leste reinforcing the importance of investing in education. Timor-Leste’s 2014/15 “Survey of Living Standards (TLSLS)” provides information on wage earners’ earnings in addition to their education level, age and gender. Estimates of a Mincerian earnings function reveal a positive association between education and earnings; individuals who have completed primary school earn 26 percent more than those with no education (Figure 1). Completing pre-secondary level education is associated with an additional 14 percent in earnings. This association between earnings and education demonstrates the importance for investing in education, and how crucial education is especially for those who have been excluded from the formal education system. 90. Given their costs, participating in Timor-Leste’s equivalency program can have economic rates of return in theory. The economic rate of return to participating in the equivalency program is modelled by comparing the effect of participation on an individual’s future earnings with the costs of participating which include private and public expenditure as well as foregone earnings (i.e.: the earnings an individual gives up by not working and participating in the equivalency program). To model the effect of participation on earnings under ideal circumstances, the effects of completing level 1 and level 2 equivalency program on earnings are assumed to be equal to the association of completing primary and pre-secondary education with earnings as described above. These ideal circumstances are for illustrative purposes only; the actual effect is not known in Timor-Leste, and international literature, as discussed, shows mixed success of these programs. To model costs, the original PAD estimated the cost per student per year in the level 1 equivalency program to be US$183 in 2009, which adjusted for inflation, is US$267 today. Community Learning Center receive a grant of US$17,000 USD per year which adds US$92 to the per student cost. According to the original PAD, 50 percent of equivalency level 1 students were between the ages of 11 and 25, 20 percent were older, and the age of the original 30 percent was unknown. Because the average age of individuals in the equivalency program is unknown, the costs and benefits for a 25 year-old are modelled (changes to this age actually have little effect on the results) and assumes that he or she will work until age 65. If an individual does not need to stop working and will not dropout,

35

the modeled rate of return for participating in the Level 1 and Level 2 equivalency program is 19 and 14 percent. These figures present an upper bound estimate of the programs’ rates of returns and rely on individuals being able to continue their jobs while participating and not dropping out. 91. However, achieving this high return for adults is challenging, as international evidence reveals. While there are many studies evaluating the impact of active labor market policies (see Betcherman, Olivas and Dar 2004 for a review), there are few studies examining the impact of second chance education programs that provide general education equivalency to dropouts later in life. Even fewer of these studies find positive effects on earnings, especially in the long-run (Table 1). For example, Nordlund, Stehlik and Strandh (2012) find that the association between second chance secondary education and earnings is nearly equivalent to that of ordinary secondary education. Jenkins et al. (2003) find positive effects on earnings from lifelong learning in the UK only for low-qualified males. Santiago Consultores (2009) in World Bank (2010) also find positive impacts on earnings especially for women where the diploma for the second chance education program is indistinguishable from that of the regular formal program. However, other studies find no effect on earnings or an economic return. Cave et al. (1993) evaluate JOBSTART in the United States which helps individual attain their high school equivalency certificate (GED) in addition to other training; their study finds no impact on earnings though they only have data up to two years after the program. Heckman and LaFontaine (2006) find no economic return to the United States GED program. Jacobson, LaLonde and Sullivan (2005) find no impact of providing general education to displaced workers. Nevertheless, available data and evidence do seem to suggest that there probably exists a positive impact on the lives of second chance education beneficiaries.

Table 8. International evaluations of second chance education programs and impacts on earnings

Program Impact Source Chile Chilecalifica + positive impact especially for women; diploma

indistinguishable from ordinary school diploma Santiago Consultores 2009 in World Bank 2010

US JOBSTART = no difference in earnings two years after program Cave et al. 1993

US GED x no economic return Heckman & LaFontaine 2006

US Community college retraining

x no positive effect of general education programs, only technical

Jacobson, LaLonde, and Sullivan 2005

UK Lifelong learning x no effect on earnings except males with low-level qualifications

Jenkins et al. 2003

Sweden 2nd chance education

= long term effect of 2nd chance secondary school on earnings similar to that of ordinary secondary school

Norland et al. 2012

36

Economic impact of the project 92. SCEP invests in the fundamental building blocks of a national equivalency program targeting key factors that promote the effectiveness of second chance education. The achievements of the project are discussed in detail in Annex 2 and address three challenges that international experience discussed above as well as the original PAD has identified as hindering the effectiveness of second chance education: (1) alignment of curriculum between second chance education and regular education, (2) reduction in dropouts, and (3) loss of income while participating. The project has made substantial progress at mitigating these three challenges. These include establishment of a comparable curriculum for the Level 1 and Level 2 equivalency programs, investments in improving the quality of these programs including design of instructional materials and teacher training and management capacity, the design and establishment of CLCs, and development of the semi-presential approach. These improvements in curriculum and the quality of service delivery help ensure a close alignment between the programs’ curricula and that of the formal education system helping to ensure that they are truly equivalents. The improvements in quality and systems to ensure availability of resources help address dropouts, as these factors were identified as one of the reasons for the high dropout rate of the program. The semi-presential approach also addresses dropouts and helps ensure students do not need to cease their livelihoods in order to achievement primary and pre-secondary equivalency. 93. Based on this foundation established by SCEP, the government has allocated financing to expand the equivalency program. In addition to allocating its own budget to finance the nine CLCs established during SCEP, the government has also allocated budget to create ten new CLCs. This demonstrates the success and sustainability of the equivalency program model developed during SCEP. While SCEP directly increased access to the equivalency level 1 program through the nine CLCs as well as to the equivalency level 2 program, the number of beneficiaries of SCEP will increase as the government expands the model. While it is not possible at this stage to forecast future expansions of the equivalency program by the government, SCEP provides the government with significant leverage to improve access and continue to address the challenges of providing education to those who were lost the chance to participate during their childhood. Moreover, SCEP project modified the teaching approach from face-to-face to semi-presential in an effort to provide more flexibility for working students and to reduce the potential opportunity costs thereby increasing the potential economic benefit.

37

Annex 4. Bank Lending and Implementation Support/Supervision Processes a) Task Team members

Names Title Unit Responsibility/

Specialty Lending Stephen David Close Human Development Specialist GEDDR Task Team Leader James A. Stevens Senior Operation Officer GEDDR Co-Task Team Leader Robert McGough Consultant GEDDR Education Specialist Luc-Charles Gacougnolle Operation Officer GEDDR Operation Stephen Hartung Financial Management Specialist GGO20 Financial Management Hanneke Van Tilburg Senior Counsel LEGEN Legal Michelle Lee Program Assistant EACNF Administration

Evelyn Villatoro Procurement Specialist EAPPR Procurement

Edward Daoud Senior Finance Officer GSUOA Disbursements

Sheila Braka Musiime Chief Counsel LEGES Lawyer

Cristiano Costa e Silva Nunes Senior Procurement Specialist GGODR Procurement

Sabrina Terry Program Assistant GHN02 Administration

Supervision/ICR Susiana Iskandar Senior Education Specialist GED02 Task Team Leader Stephen David Close Human Development Specialist GED02 Task Team Leader

Dandan Chen Lead Economist GED02 Education Specialist/Operation

Binh Thanh Vu Senior Education Specialist GED02 Education Team Leader for Pacific Countries

Franco Russo Senior Education Specialist GED02 Education Specialist/Operation

Isabella Micali Drossos Senior Counsel LEGLE Legal Marjorie Mpundu Senior Counsel LEGAM Legal Seble Berhanu Legal Analyst LEGES Legal Miriam Witana Sr. Procurement Specialist GGO08 Procurement Manuela Da Cruz Procurement Specialist GGO08 Procurement David Bruce Whitehead Financial Management Specialist GGO20 Financial Management

Robert J. Gilfoyle Senior Financial Management Specialist

GGO20 Financial Management

Unggul Suprayitno Senior Financial Management Specialist

GGO20 Financial Management

Jacinta Paula Bernardo Financial Management Analyst EACDF Financial Management Maria Isabel Alda Da Silva Operation Analyst EACDF Financial Management Rita Maria De Sousa Fernandes

Consultant GED02 Education Specialist

Jose Maria Tavares Mousaco Consultant GED02 Operation

38

Adelaide Goretti Neves De Camoes

Education Specialist GED02 Education Specialist

Kumala Sari Operation Analyst GSU08 Community Development

Titie Hadiyati Consultant GED02 Community Development/Operation

Agiel Bahri Consultant GED02 Architect/ Construction by Community

Maria Eulalia Belo Leite Senior Program Assistant EACDF Administration Kristen Andrew McDonall Program Assistant EACNF Administration Cornelio Quintao De Carvalho Team Assistant EACDF Administration

Rideca R. C. Duarte Team Assistant EACDF Administration Anna Coronado Program Assistant GED02 Administration Sandra F. Beemer Consultant GED02 ICR Primary Author Kevin MacDonald Consultant GED02 ICR Economic Analysis (b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)

No. of staff weeks USD Thousands (including travel and consultant costs)

Lending

FY 2009 3.28 31,196.27

FY 2010 6.85 84,964.09

Total 10.13 116,160.36

SPN/ICR

FY 2011 7.91 120,094.70

FY 2012 24.3 87,196.72

FY 2013 22.78 66,886.26

FY 2014 24.12 82,121.15

FY 2015 28.16 138,474.70

FY 2016 16.05 85,432.01

FY 2017 0.58 48,818.22

Total 123.9 532,437.03

39

Annex 5. Beneficiary Survey Results Not Applicable

40

Annex 6. Stakeholder Workshop Report and Results Not Applicable

41

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR

I. Summary 94. The National Directorate for Recurrent Education, within the Ministry of Education, is the Government of Timor-Leste’s designated provider of primary education equivalency programs for youth and adults who have either left formal education delivery system under various reasons or have never been to formal education system at all. Although there are some NGO programs available that offer non-formal programs, the DNER is the only public education authority serving Timor- Leste’s primary/secondary school leavers. It provides a number of related primary/secondary level equivalency programs delivered through both presential and semi-presential approach. 95. The National Directorate for Recurrent Education is mandated with three core programs including Literacy, Equivalency Program, and Community Skills Improvement Program, a program which is promoted through CLC established in all 9 Municipalities. In 2011, the Government of Timor-Leste signed an agreement with the International Development Association or the World Bank to support the Ministry of Education through its mandated National Directorate of Recurrent Education to implement the National Equivalency Program. This program was named as SCEP funded by IDA with Project Grant H633-TP. This project has been implemented over a period of five years from 2011 to 2015 but due to change of Government from IV Constitutional Government to V and VI Constitutional Governments including change of NDRE Directors, the project was restructured and extended from December 2015 to December 2016 to properly implement the objectives and ensure effective attainment of Indicators identified in the project development objective and its intermediate outcomes.

II. Project Description 96. Second Chance Education Project (SCEP) Development Objectives was originally to increase the number of out-of-school youth and young adults who complete recognized equivalency programs and (ii) to decrease number and/or rate of leavers from each program and level, thus increasing internal efficiency of the program. Through project restructuring in 2015 the objectives were changed to become to help Ministry of Education in the establishment of a sustainable national equivalency program with two main indicators; (i) Legal framework endorsed and submitted to the MOE, and (ii) curriculum endorsed by the MOE. Hence, the project activities were focused on activities to achieve the PDO to build a sustainable national equivalency program. Project Components were adjusted to meet the PDO and to fit with the time available including one year extension from December 2015 –December 2016:

42

III. What did work well in the Second Chance Education Project

3.1 Legal framework regarding Recurrent Education 97. As per project restructuring officially signed in October 7, 2015, two result indicators were agreed upon which included Legal Framework regarding Recurrent Education and National Curriculum for Equivalency Program endorsed by the Ministry of Education. NDRE recruited a Legal Advisor to identify and draft laws and Regulations required as basis for the Recurrent Education to operate and implement its National Equivalency Program, a programs which is categorized as another educational stream to absorb those who have either dropped out from Formal Education system or never been to school during schooling ages. 98. The draft Decree Law for Recurrent Education was presented intensively for internal discussion before being presented at a number of Workshops attended by relevant directorates of the Ministry of Education, relevant government institutions, non-government organizations including Equivalency Teachers. In late 2015, two workshops were held nationally with each was exposed to public through media coverage for public awareness. The decree law was thoroughly discussed at each of the workshop to cover inputs from participants at the workshop, findings of the needs and demand study and preferable learning approach survey. Moreover, it also covered lessons learned from the overseas visits by two teams who visited to Indonesia and Thailand from September 27 to October 6, 2015. 99. The above references were presented at the last workshop held on October 22-23, 2015 and the ultimate draft was then presented to H.E Minister of Education and H.E Vice Minister of Education before it was submitted to the Council of Ministers in April 2016 for approval. Finally, in June 2016, the Council of Ministers approved the proposed decree law and promulgated by the President of the Republic. The decree law entered into force since July 13, 2016 as Decreto-Lei nº 30/2016, de 13 de Julho. A Ministerial Diploma regulating the Implementation of the approved Equivalency Program was also approved by the Ministry of Education along with one dispatch for Equivalency Program certification. Initially it was discussed that, two additional Ministerial diplomas for licensing and accreditation would also be developed for NDRE to grant licenses and accreditation to those educational entities who wish to establish and open an Equivalency Program either as an additional program to their existing programs or as a core program of the Educational services provided. However, since NDRE is another directorate subordinates under the Ministry of Education, it was decided not to develop any separate diplomas exclusively just for NDRE alone apart from one that was being drafted at that time by the Legal Unit of the Ministry of Education for Formal Education. In late December 2016, it was confirmed that these two diplomas had been submitted for approval after minor alterations suggested by the Minister. During the discussion on the substance of the two diplomas, a team led by the Head of Department of the Literacy and Equivalency Program also participated and provided inputs related to the nature and scope of the Equivalency Program. The inputs were incorporated to ensure that these two diplomas also cover

43

Licensing and accreditation in part of Equivalency Program whose nature and approach is different to regular formal education. 100. In addition, in late 2015 and earlier 2016, there were discussions around which NDRE with the support of the Legal Adviser would develop a separate decree law or ministerial diploma governing the establishment of Community Learning Centers, however, after considering various aspects, the Ministry of Education decided not to keep Community Learning Centers as public institutions or public schools. It was anticipated that, having it subordinates under the Ministry of Education, it could put more burden in terms of budget on the Ministry of Education and also anticipated that CLCs may not always have students at all time as it is with formal schools. When there are no students enrolled, it would be a waste of resource and it could be difficult to convert those CLCs when they are already under the Ministry as public Institutions or public schools. Alternatively, the Ministry of Education through National Recurrent Education decided to only provide subsidies to the existing CLCs to help them continue implementing Equivalency program and at the same time encourage other CLCs to go through the legal process concerning the establishment of such organization with the Ministry of Justice so that they could be established legally. 3.2 Curriculum for National Equivalency Program 3.2.1 Curriculum Review 101. The review of the curriculum was primarily based on the fundamental sources which included the following: (i) the empirical analysis on the need and demand study carried out by Graziela Lunardi, the Senior Researcher, (ii) Quick survey on the flexible learning modality carried out in September 2015 by all district Support Facilitators, and (iii) lessons learned from the overseas studies to Indonesia and Thailand from September 27 to October 6, 2015. 102. These fundamental inputs were presented at the last workshop carried out in October 2015 whose output was then presented to H.E Minister of Education and Vice Minister of Education – 1 for endorsement before it was attached as an annex to the draft Decree Law to the Council of Ministers in April 2016. In addition to curriculum approval, the learning materials were also adjusted to accommodate new learning modality which is semi-presential. In addition, learning materials for level I was also quickly revised after suggestion made by the World Bank Mission team in April 2016 to adopt new learning modality of semi-presential at least at phase III of level I. Within such a very short time, content of the learning materials of Level I were quickly revised but it was not thoroughly done to accommodate the updated information and making it the same as the Formal education one. In late December 2016, H.E Vice Minister-I requested to carry out a further review but it was not possible due to time constraint as SCEP was going to be closed in a week time. Therefore, it was decided that Ministry would continue retain the curriculum reform team to review the level I learning materials accordingly. The proposed review has been carried out at the beginning of 2017 the team was on board.

44

3.2.2 Curriculum Structure 103. Following are tables illustrate Curricular structure of the Equivalency Program as per defined in the Decree Law No 30/2016, 13 July 2016.

Table – 9. New curriculum reform in the structure brought in the curricular grid.17

Areas of Learning Components of Curricular

Development of Linguistics Tetum literacy Portuguese Literacy Oral Linguistics Support

Scientific Development

Mathematics Natural Science Social Science l

Personal Development

Arts and Culture Health Physical Education Religion

Table – 10. Curricular Plan Of National Equivalency Program, Equivalent To Basic Education (Matrix Of Curricular Of Phase I, II and III)18

LEARNING PHASES

ARES OF LEARNING SUBJECTS I II III

Linguistic Development

Portuguese Language

3 3 3

Tétum Language 3 3 3

Scientific and Social Development

Mathematics 4 4 4

Natural Science 3 3 3

Social Science 3 3 3

Personal Development Vocational Course 4 4 4

LEARNING LOAD (TOTAL HOURS) 20 20 20

17 New Curriculum based on the recommendations of National Basic Education (cycle 1 and Cycle 2) 18 This curricular structure has been approved by MoE and subsequently by the Council of Ministers through Decree Law which provides basis for the Equivalency Programs.

45

Table - 11 Curricular Plan of National Equivalency Program, Equivalent To Basic Education (Matrix of Curriculum of Phases IV and V)19

LEARNING PHASES

AREAS OF LEARNING SUBJECTS IV V

Linguistic Development

Portuguese Language 3 3

Tétum Language 3 3

English Language 2 2

Scientific and Social Development

Mathematics 4 4

Natural Science 4 4

Social Science 3 3

Personal Development Elective Subject 2 2

Vocational Course 4 4

LEARNING LOAD (HOURS) 25 25

104. In addition to curricular structures illustrated in the tables above, the Equivalency Implementation Manual has also been developed and available for use by the Directorate. This manual is to support the Directorate implementing the Equivalency program in 2016 and beyond. This manual contains analysis and views based on the following parts:

a. Goals and purposes of teaching and learning (skills and competencies). b. Texts and activities based on socio-cultural reality of the country. c. Text style (language, words and idioms) which is easy to understand and reflects

reality and context of the country. d. Images, drawings and photographs representing reality of the country. e. Respect individual and country’s culture. f. Language that recognizes diversity of color, race, gender and religion.

105. In addition, the Directorate with the support of the International Development Consultant developed an Implementation Plan focuses on management of the Program. The plan contains the following parts:

19 This curicular structure was presented in the revised curriculum which was submitted along with the draft Decree Law and subsequently approved by the Council of Ministers on April 26, 2016.

46

a. Basis of Equivalency Programs; b. Phases of implementation (cycles of implementation including school calendar); c. Role of National Directorate particularly the department of Equivalency in

enrollment process, teachers training, class distribution, implementation, local exams and national exams including certification;

d. Role of Municipal Education Director and coordinators including CLCs in the enrollment process, classes distribution, implementation, local exams and national exams management;

e. Other stakeholders who involve in the implementation of the program; and f. Sustainability strategy including communication with internal and external

stakeholders particularly related to Life Skills implementation. 106. The implementation plan as stated above was developed with inputs from NDRE and it was finalized by the International Curriculum Development Consultant before end of December 2016. The implementation plan is available for NDRE to implement the New Equivalency Program Curriculum when there is no more support available. 3.2.3 Curriculum Approval & Implementation

107. Curriculum of the National Equivalency Program was endorsed after it had been revised based on the result of the trial carried out in January 2015 to June 2015 supported by the International Curriculum Development Consultant. Upon approval, it was firstly implemented with new learning modality of Semi-Presential in earlier April 2016 at the three new CLCs locations of Manatuto, Vera-Cruz of Dili and Aileu. Other six CLCs locations continued to use the previous Curriculum as students had been taught previously on the same curriculum used for trial and sifting it to new curriculum it would decrease learning momentum of the students and might cause students to drop-out. This was because at that time there were 108 participants ready to take part in the national exam scheduled on November 14, 2016.

108. In supporting the implementation of the new curriculum as described, the three mentioned CLC locations regularly received visits of the pedagogic team who provided clinical support to EP Teachers to teach effectively and ensure they teach based on the agreed methods of teaching and learning. Outcome of the support has culminated through indication of national exam result whereby 108 participants attended, 78 out of 29 male and 49 females passed with average score above 70. Besides it, 26 others consisted of 12 male, 14 female had not passed it, and they were given another chance to re-exam the subject failed. The re-exam was organized on December 5, 2016 at all six CLCs locations and all 26 participants passed the second round exam, except for 2 whom did not take part in any round of the exams provided. With both exams, it makes the overall passed figure 106.

47

3.3 Expansion of Local Capacity to deliver Equivalency Programs through Community Learning Centers 109. Another success which Ministry of Education through National Directorate for Recurrent Education with the support of Second Chance Education Project (SCEP) made was the establishment of 9 Community Learning Centers in the following municipalities; Iliomar at Losplaos, Baucau, Manatuto, two in Dili (Cristo Rei and Vera-Cruz), Aileu, Ermera, Maliana and the exclusive economic region of Oecusse. These nine CLCs were established after a series of community gathering and consultations at each of the respective municipality to socialize topics related to the essence of Community Learning Center and how it could help community to increase educational qualification and skills required to access labor markets. After the initial gatherings and socializations, some preparations were carried out with the support of 6 district facilitators recruited by SCEP to help community in the establishment of CLCs and implementation plan development of which it included EP programs and other life skill Programs. 110. Through such a participatory process, the participating communities of the respective locations were able to elect their own CLC council and committee and drafted the CLC legal status as criteria to access the grants. Within the period of less than one year, all nine CLCs were established with buildings renovated and furnished with some basic facilities such as chairs and tables for classrooms, working desks and chairs and cabinets for CLCs offices and computers and printers to facilitate the day-to-day operation. In addition, capacity building for CLC managers was also provided through trainings both in country and overseas to equip them the capacity and skills required to successfully manage the CLCs and carry out income generating activities for survival. CLCs presidents and secretaries were sent to Indonesia in December from 19-23, 2016 to visit some most successful CLCs in Jogjakarta to see how CLCs in Indonesia implement Equivalency Program and how they run income generating activities to sustain their CLCs. This was a strategy to re-orientate the vision of the CLCs Managers and help them to be more creative in terms of what should be considered in their operational in part of income making activities to ease the possible dependencies on the support coming from the Ministry of Education or other relevant government agencies. 111. Besides this, the National Directorate for Recurrent Education with the support of the Second Chance Education Project (SCEP) also identified 10 more CLCs locations from other municipalities which did not have a CLC. The identification process was carried out before end of December 2016 and some community meetings were carried out with communities from the respective municipalities. The communities of the 10 locations were facilitated to elect their CLC Committee and councils to be responsible for the establishment of the new CLCs and oversee its implementation. The establishment of the new 10 CLCs will be covered by the Ministry of Education under the state budget. Manual to support the implementation has been developed and finalized in accordance with current procedures applicable for such grants and subsidies. With the existing CLCs and expansion, NDRE will have the capacity in terms of facilities to implement the mandated program which is the National Equivalency program and giving an extra space to accommodate literacy program which has been delegated to Municipalities with nowhere to house it.

48

3.4 Capacity improvement to implement EP programs 3.4.1 Recruitment of Pedagogic Team and EP Teachers including capacity building. 112. Under the Second Chance Education Project, a lot of activities were carried out to increase the capacity of the National Directorate for Recurrent Education. These activities include recruitment of Pedagogic Team for Equivalency Program Level I and Level II. These two teams were given roles and responsibility to work wiht the International Curriculum Development consultant to develop EP Curriculum and train EP Teachers for EP trial and implementation at all CLC locations. The team carrid out these roles and responsibilites until last year where New organic Law of the Ministry came to effect that shifted the functions of Curriculum to the National Directorate for Curriculum and training to INFORDEPE. In April 2016, the World Bank Mission proposed the Ministry of Education to revise roles and responsibility of the team and sugested the Ministry to decide where to place them as there had been such a great investmet made to bring them on board and also trained them. It was discussed that these two teams would be put under the respective Directorate (Curriculum and INFORDEPE) unfortunately, it was informed that there was no working space available. Alternatively, it was decided to keep the teams in the National Directorate for Recurrent Education for the time being. The Director of the National Recurrent Education decided to merge the teams into one led by Coordinator of the Pedagogic Level II. 113. To support day-to-day functions, Terms of Reference of the Respective Pedagogic Teams have been revised with the support of the International Curriculum Development Consultant and Recurrent Education Specialist. The teams remain with the Recurrent Education and will continue to perform in the resepctive function to support the implementation of EP programs and help train EP teachers to effectively deliver the Programs. In additon, the National Directorate for Recurrent Education with the support of Second Chance Education Project also recruited 97 teachers for Equivalency program Level l. The recruitment to these 97 teachers were carried out in 2013 and paid on a decrease basis from 75% covered by Second Chance Education Project and 25% covered by Ministry of Education at first year, then it was down to 50% and in 2016 government took over the payment 100% covered by the state budget. In late 2014, the NDRE also recruited 42 new teachers assigned to Level II Equivalency Program. Those teachers were recruited with the support of Second Chance Education Project (SCEP) and paid out on the same scale as level I. With the expansion of CLCs to three new locations of Manatuto, Vera-Cruz and Dili, in late 2015, 18 more teachers were recruited and trained to help implement level II Equivalency Program in the newly established CLC in Manatuto, Dili (Verra-Cruz ) and Aileu. Training of teachers was provided numerous of times from when they were recruited (before trial), after trial and refreshment training again in late 2016. The two last trainings focused primarily on the new structure of the curriculum and its learning approaches.

49

3.4.2 Management Capacity Building 114. Management Capacity bulding activities were carried out both in-county and overseas. The in-country training was provided a number of times focusing on institutional Management which covered planning, directing, controling and evaluation, a holistic cycle which a manager needs to be aware with. This traning was provided over a period of five days in July 2016. The training focused also on communication strategies which NDER needs to build with key stakeholders such as CLCs and other relevant government organizations, such as SEFOPE, Social Solidarity and Minstry of Agriculture and fisheries. Besides this, from classroom training as stated, other capacity building activities were also carried out through coaching and monitoring provided by each of the consultant who were recruited and assigned at the National Directorate for Recurrent Education. 115. In additon to in-country capacity building program as specified above, other capacity buiilding program was also provided through overseas studies to Thailand in and Indonesia From September 27 to October 6, 2015 whose result was used as an input to the review of EP Curriculum and Legal Framework. Besides this, other visit was also carried out in October 2016 with participation of Municipal Educational Directors led by H.E Vice Minister-I and Director General who supervises the National Directorate for Recurrent Education. This team visited the Non-Formal Education in Jakarta and couple others in Bandung and Jogjakarta. The visit was to see how Non-Formal Education Directorate implement its national equivalency program inluding earlier chilhood program which they called PAUD through Community Learning Centers or so called “Pusat Kegiatan Belajar Masyarakat (PKBM)”. The visit had made a significant contribution in re-shaping the vision of Managers on the best practices to implement Equivalency program including earlier childhood program in Timor-Leste. It was confirmed that the managers had learned a lot from the visit and each made the commitment to contribute to the development and implementation of EP programs in the future. 116. Moreover, other last overseas study was carried out from December 19-23, 2016. There were two teams went on this study consisted of presidents and secretaries of all nine CLCs, two head of departments of Equivalency program and curriculum and other key staff of the directorate including EP teachers. One team went to Jogjakarta to visit Community Learning Centers to learn how CLC in Jogjakarta operate and manage their own programs including PAUD (earlier childhood education) and equivalency programs including community development programs. This study visit was to help CLCs and NDRE the capacity to manage CLCs sustainably and develop the most crative in-come generating activities for survival. This is another success which Project left as a legacy for NDRE and CLCs to be capable and creative.

3.4.3 M&E and Information Management System 117. Success of an organization will depend on how well an organization plans, directs, monitors and reallocates its resources to meet its deliverables. M&E and Information Management system is one of the most reliable instruments which could generate the most accurate and reliable information to benefit the need of the organization in terms of

50

planning and resources allocation. Being as an organization, DNER with the support of Second Chance Education Project established an M&E system equipped with tools and instruments required to collect data related to student’s enrollment, students achievement, student’s movement, teacher’s assignment and facilities. This is to help NDRE to track and evaluate its capacity and measure its achievement. The M&E system was developed and subsequent training was provided to M&E designated staff to do the Monitoring and evaluation over programs mandated to NDRE. 118. Besides this, the NDRE with the support of Second Chance Education Project also developed a Non-Formal Education Information Management System a system which was developed after an evaluation carried out by the M&E and IT/MIS Consultants on the existing EMIS Non-Formal system included in the EMIS formal. Finding of the evaluation appeared that, somehow, the non-formal education information management system had not been properly finalized, as a result, a lot of functions did not exist and it was not working properly. The evaluation revealed also that, improving it would take as much time as it would take to develop a new one. So it was decided to build a new one from the scratch rather than improving it. Based on that, a new system was developed in a way which it could be synchronized with the functionalities included in the M&E tools and instruments. Upon completion, the consultant trialed the system and carried out some training through on the-job-training to users both at national and sub-national to upload data relevant to current students, teachers and facilities. 119. Moreover, the system is a friendly used system and it can be accessed by phone at anytime and anywhere as long as there is an internet access to access data and find relevant information when needed. Ministry’s officials have been very pleased to see the completion of the system and have requested to integrate it with the Pre-School and Formal Basic Education system. Analysis on the possible integration was completed and report with recommendation has been submitted to NDRE. Besides this, the system has been upgraded to include a new function for NDRE to regularly send messages out to CLCs in case there is anything to inform. The new functionality enables the directorate to send invitations in case the letters cannot be delivered in time. This functionally is linked with the phones of the CLC management and the messages will appear on their phones automatically as soon as it is sent form the National Database. This is to increase communication between National and the Municipalities and it can also improve services delivery of the directorate. IV. What did not work well during the implementation of the project

4.1. Initial Implementation

120. At the begining when the project was developed and structured, it was not anticipated that there would be changes of Government and leadership at the Ministry of Education and the National Directorate to lead and provide instructions and guidance in terms of priorities that project could support. As a consequence, project had to wait during an intensive time for almost one and a half years. Due to that delays, the deliverables which had been defined at the earlier stage could not be delivered and those had to be revised in a Mid Term review carried out jointly between the Ministry of Education and the World

51

Bank in July 2014. Result of the review indicated that it might not be possible to implement all Project Intermediate Indicators and that it should nee a restructuring to align priorities with the time available. 121. Most deliverables remained undelivered and in 2014 it was almost three years already and keeping the Project going with the same outcomes indicators, the Project might not be successful as it had been expected. Therefore, after the review, the Ministry of Education and the World Bank agreed to restructure the project to make it more feasible. Along with Project restructuring, the Ministry of Education also requested an extension to the project extending the closing date from December 31, 2015 to December 31, 2016. 4.2 Project Management

122. Initially Second Chance Education Project benefited from the previous Project Management Unit who handled all Projects including Infrastructure. As a consequence, there was lack of attention given to Second Chance Education Project. The then Project Coordinator had to manage the ongoing projects which included ESSP and MSP. As a result, the Ministry of Education requested to hire a new Project Coordinator to ease the workload of the then Project Coordinator whom was then assigned as Coordinator only for Infrastructure Unit. 123. The New Project Coordinator on board on July 1, 2014 at the time when Project was going through the Mid Term Review. With the arrival of New Project Coordinator, the Ministry of Education was able to restructure the project and extended the life of the project for one more year. Within the year of 2014, all consultant required to implement the project were hired and led by Project Coordinator who managed the deliverables of the consultants regularly and coordinated meetings to dicuss issues and find their solutions. Since then, the project coordinator was able to keep communication going effectively both vertically and horizontally within the Ministry and the World Bank.

V. Project Sustainability 124. Objective of the Project was to help the Ministry of Education of Timor-Leste in the establishment of a sustainable Equivalency Program for those who have either dropped out from formal schooling system or never been to formal schools because of many reasons. During the life of the Project, Ministry of Education through its National Directorate for Recurrent Education with the support of Second Chance Education project was able to deliver the expected Project Development Objectives and its intermediate outcomes indicators. These have contributed to what are seen sustainable that include the following: 5.1. Legal Framework 125. Over a period of less than one and a half years, the National Directorate for Recurrent Education with the support of Second Chance Education Project was able to develop a legal basis for the National Equivalency Program. A Decree Law approving the Equivalency program was developed over an intense consultation both within and across

52

all relevant government and non-government organization. The decree law approved by the Council of Minister in June 2016 and latter promulgated by the President of the Republic to become as Decreto-Lei n.º 30/2016, de 13 de Julho. This decree law provides the basis for the National Equivalency Program to take shape in its form as one of the educational streams entitled under the National Directorate for Recurrent Education. 126. Besides this decree law, one ministerial diploma governing implementation of the Decree Law was developed and subsequently approved by the Minister of Education along with one dispatch approving the program certification. This is another fundamental basis for Equivalency program to sustain.

5.2. Programs mandated to NDRE 127. Another support which has been seen sustainable is the National Equivalency Program. Over the period of less than one and a half years from initial 2015, the national curriculum for equivalency program was developed, trialed and adjusted to fit with the adults learning preferences in terms of frequency. The curriculum was developed with inputs coming from the empirical analysis on the need and demands study carried out by Graziela Luanardi, the quick survey on the learning modalities, and lessons learned from the overseas visits to Indonesia and Thailand in October 2015. The National Curriculum for Equivalency Program was approved by the Minister of Education and was attached as an annex along with the Decree Law to the Council of Minister. 128. In addition to the national Equivalency program curriculum, all learning materials of both levels were revised and adjusted to fit the new curriculum with the new learning modalities. Implementation manual and implementation plan have been developed and finalized for future implementation. This will make certain that despite Project no longer operate, however, with these important tools, the National Directorate for Recurrent Education will be of capable to implement the program independently and sustainably. 5.3. Community Learning Centers 129. Initially, the National Directorate of Recurrent Education was not able to implement Equivalency Program at community level due to inexistence of facilities and teachers to implement it. Under the support of Second Chance Education Project, within one and a half years, 9 CLCs were established and are now functional at eight municipalities including Lospalos, Baucau, Manatuto, Dili, Aileu, Ermera, Maliana and the exclusive economy zone of Oecusse. These 9 CLCs have received support from the Ministry of Education through two grants instalments. First instalment was to help rehabilitate the buildings and second was to support the implementation of the core program of Equivalency program and Life-skills including purchasing of equipment to support program implementation. All nine CLCs are fully operational and as stated in the “what worked well part” that all received relevant trainings both in-country and overseas. This is to facilitate managers obtaining necessary skills and knowledge to properly manage CLCs and competently run EP program. In December 2016, the National Directorate also identified 10 more locations to establish new CLCs. These 10 new CLCs will be established

53

under the state budget and respective grant manual has been finalized and is available for NDRE to use. 130. With the expansion of 10 additional CLCs, it will provide more capacity for NDRE in terms of learning space and management to implement the National Equivalency program. The way how CLCs have been established is slightly different to other institution. These CLCs have been established with the support and initiative of the participating communities and life-skills programs have been designed to meet the need of the respective community. This is seen more community based orientation.

5.4. M&E and Non-Formal Education Information Management System 131. M&E and Non-Formal Education Information system will also be of help to the National Directorate for Recurrent Education. Training of users was carried out and manuals were developed and finalized to facilitate users operation. This is another important instrument to help NDRE continuously evaluate its performance and gradually plan for improvement, if the information in the system is properly analyzed and used for management purposes. Training on the use of data was also provided and hopefully, NDRE will continue to operate the system and use the tools to continue collect data and relevant information needed. In addition, procedures and process dealing with data maintenance and data warehouse were developed and staff was trained to handle it. DNRE was also well informed about the database management and upgrades.

54

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not Applicable

55

Annex 9. List of Supporting Documents Aides-memoire and Implementation Status Reports 2012 – 2016 Implementation Completion Report, Borrower Portion, Second Chance Education Project, 2017. Interim Strategy Note 2009-2010, Timor-Leste World Bank, Washington, D.C., 2009. National Education Strategic Plan 2011-2030, Ministry of Education, 2011. Project Appraisal Document, Second Chance Education, World Bank, 2010 World Development Report, “Conflict, Security and Development”, World Bank, 2011. World Bank Country Partnership Strategy, Timor-Leste, 2013-2017 Betcherman, Gordon, Karina Olivas, and Amit Dar. 2004. Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries. SP discussion paper No. 0402 January 2004, World Bank, Washington, DC. Cave, G., Bos, H., Doolittle, F. & Toussaint, C. (1993). Jobstart: A final Report on a Program for School dropouts. Manpower Demonstration Research Corporation. Davies, L. 2010. Education and Fragility. A Synthesis of Four Country Case Studies (Afghanistan, Bosnia and Herzegovina, Cambodia and Liberia), International Network for Education in Emergencies (INEE). Heckman, J. and LaFontaine, P. (2006). Bias-corrected Estimates of GED Returns. Journal of Labor Economics, Vol. 24, No. 3. Jacobson, L.S., R.J. LaLonde, and D.G. Sullivan. 2005. The returns to community college schooling for displaced workers. Journal of Econometrics 125: 271–403. Jenkins, A., A. Vognoles, A. Wolf, and F. Galindo-Rueda. 2003. The determinants and labour market effects of lifelong learning. Applied Economics 35: 1711–21 Miall, H. 2004 Conflict Transformation: A Multi-Dimensional Task, Berghof Handbook for Conflict Transformation. http://www.berghof-handbook.net/ Nordlund, M., T. Stehlik and M. Strandh 2012. Investment in Second-Chance Education for adults and income development in Sweden, Journal of Education and Work, DOI:10.1080/13639080.2012.664633 Shonkoff, J. and Phillips, D. (2000). From Neurons to Neighborhoods: The Science of Early Childhood Development. Washington, DC: National Academy Press.

56

Smith, A. 2010. The influence of education on conflict and peace building. Paris: UNESCO Smith, A., and Vaux, T. 2003 Education, conflict, and international development. London: Department of International Development World Bank 2010. Active labor market programs for youth. A framework to guide youth employment interventions. Employment Policy Primer No. 16. Washington, D.C.: The World Bank