leading schools digitally: school planning and budgeting
TRANSCRIPT
Leading Schools Digitally: Evaluation of the Electronic
School Planning and Budgeting System (e-RKAS) in Indonesia
THEWORLDBANK
JULY8,2020
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
1
This report is a product of the staff of the World Bank, supported by funding from the Australian Government. The findings, interpretations, and conclusions expressed in this report do not necessarily reflect the views of the Executive Directors of the World Bank or the governments they represent, or the Australian Government. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. © 2020 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202‐473‐1000 www.worldbank.org The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission maybe a violation of applicable law. The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of this work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978‐750‐8400, fax 978‐750‐4470, http://www.copyright.com/. All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202‐5222625; e‐mail: [email protected].
2
CONTENTS
CONTENTS ............................................................................................................................................................ 2
Acronyms ............................................................................................................................................................. 3
Executive Summary .............................................................................................................................................. 4
Background ...................................................................................................................................................... 4
Evaluating e‐RKAS ............................................................................................................................................ 5
Review Workshop Findings .......................................................................................................................... 5
Interview Findings ........................................................................................................................................ 6
Evaluating the impact of e‐RKAS on student achievement ......................................................................... 6
Recommendations ........................................................................................................................................... 7
Introduction ......................................................................................................................................................... 9
Background ...................................................................................................................................................... 9
The Theory of Change Behind e‐RKAS ........................................................................................................... 11
History of e‐RKAS Implementation ................................................................................................................ 13
Development of a MoHA System ............................................................................................................... 15
Deployment by the Ministry of Religious Affairs ....................................................................................... 15
Research Methodology ...................................................................................................................................... 15
Pilot and Evaluation ....................................................................................................................................... 16
Qualitative Evaluation Methodology ............................................................................................................. 20
Quasi‐experimental Impact Evaluation.......................................................................................................... 21
Key Findings ....................................................................................................................................................... 21
Feedback on Planning Documents ................................................................................................................. 21
Implementation of e‐RKAS ............................................................................................................................. 25
Impact of e‐RKAS on student achievement ................................................................................................... 27
Conclusion .......................................................................................................................................................... 33
Appendix 1: The National Education Standards ................................................................................................ 37
Appendix 2: Numbers of Participants in Pilot Activities .................................................................................... 39
Appendix 3: Pilot Schools ................................................................................................................................... 40
Appendix 4: Identification of the treatments .................................................................................................... 42
Appendix 5: Average UN scores of the eight sample school districts by subject: 2013‐2018 ........................... 51
Appendix 6: Portrait of BOS Spending in 8 Districts/Cities ................................................................................ 52
3
Acronyms
AKB Anggaran Kas Bulanan / Monthly Budget
ARKAS Aplikasi RKAS / School Activity and Budget Planning Application
BOS Bantuan Operasional Sekolah / School Operational Assistance
BOSDA Bantuan Operasional Sekolah Daerah / Local School Operational Assistance
BPKAD Badan Pengelolaan Keuangan dan Aset Daerah / Regional Financial and Asset Management Office a.k.a District/Province Financial Office
Dapodik Data Pokok Pendidikan / Education Basic Data
DEO District Education Office
DFAT Department of Foreign Affairs and Trade
DP2KAD Dinas Pendapatan Pengelolaan Keuangan dan Aset Daerah / Local Income and Finance and Asset Management Unit
EDM Evaluasi Diri Madrasah / Madrasah Self‐Evaluation
EDS Evaluasi Diri Sekolah / School Self‐Evaluation
e‐RKAM Elektronik Rencana Kegiatan Anggaran Madrasah / Electronic Based Madrasah Activity and Budget Plan
e‐RKAS Rencana Kegiatan dan Anggaran Sekolah Berbasis Elektronik / Electronic‐Based School Activity and Budget Plan
FY Fiscal Year
LPMP Lembaga Penjaminan Mutu Pendidikan / Provincial Education Quality Assurance Institution
MSS Minimum Service Standard
MoEC Ministry of Education and Culture
MoF Ministry of Finance
MoHA Ministry of Home Affairs
NES National Education Standard
NIP Nomor Induk Pegawai / Personnel Identification Number
PAUD Pendidikan Anak Usia Dini / Early Childhood Education
PEO Provincial Education Office
PER Public Expenditure Review
PISA Programme for International Student Assessment
PMP Penjaminan Mutu Pendidikan / Education Quality Assurance
RKA Rencana Kegiatan dan Anggaran/ Activity and Budget Plan
RKAS Rencana Kegiatan dan Anggaran Sekolah / School Activity and Budget Plan
RKS Rencana Kerja Sekolah / School Work Plan
RKT Rencana Kerja Tahunan / Yearly Work Plan
SD Sekolah Dasar / Primary School
Sidrap Sidenreng Rappang
SIMDA Sistem Informasi Manajemen Daerah / Regional Management Information System
SIPBOS Sistem Informasi Pengelolaan BOS / BOS Management Information System
SLB Sekolah Luar Biasa / Special School
SMA Sekolah Menengah Atas / Senior High School
SMK Sekolah Menengah Kejuruan / Senior Vocational School
SMP Sekolah Menengah Pertama / Junior High School
SSE School Self‐Evaluation
TA Technical Assistance
ToT Training of Trainers
TPA Tim Perencanaan dan Anggaran / Planning and Budgeting Team
UN Ujian Nasional / National Examination
4
Executive Summary
Background
The education system under the Ministry of Education and Culture (MoEC) is highly decentralized. Most school costs under MoEC are covered by fiscal transfers from the centre to provincial and district levels, some of which are earmarked for education use. One such type of transfer is school operational grants, known as Bantuan Operasional Sekolah (BOS) or school operational assistance. BOS funds are managed directly by schools, which have been delegated the autonomy to receive, plan and budget, spend, administer, and report their use. Experience has shown that many schools lack the capacity to use BOS funds effectively and efficiently to deliver better learning outcomes for students, while there have been no planning systems to manage use of BOS funds towards achieving the National Education Standards (NES) for individual schools. Under the BOS program, each school is required to conduct a School Self‐Evaluation (SSE) against the NES and use the results to develop its spending plans accordingly. However, most schools develop the SSE only when required to fulfil accreditation requirements. There has been no systematic review on the implementation and impacts of the SSE across schools. Thus, the only way to obtain data on school progress towards the eight standards is through the school accreditation process, which is only conducted in a sample of schools every five years. There is also no intensive monitoring and evaluation of how schools spend BOS funds. Every school has also been advised to develop a planning and budgeting system (Rencana Kegiatan dan Anggaran Sekolah or RKAS) to allocate and manage BOS funds. The concept of performance‐based planning and budgeting was introduced to ensure that schools use the RKAS to improve their performance. A World Bank review of BOS in 20141 revealed that only some schools develop an RKAS annually, while others simply copy the previous year’s data. Further, where they are developed, the RKAS is typically not based on the school’s SSE results, which means that it is not systematically aligned with the eight NESs. To support implementation of the RKAS, an application called the Rencana Kegiatan dan Anggaran Sekolah Berbasis Elektronik (e‐RKAS or electronic school plan) has been developed. This tool enables schools to input and manage the RKAS electronically, which is more efficient, better structured, and in line with a performance‐based planning approach. A first version of the e‐RKAS was created by the Education Office in Surabaya, East Java, to simplify the approval process of school plans and budgets. A more advanced version was then developed by the World Bank team and piloted in schools in DKI Jakarta province in 2015‐2016. In 2017, the World Bank and MoEC piloted the e‐RKAS program for schools in Bali province, Central Java province, Gorontalo district, Sidenreng Rappang (Sidrap) district, and Mojokerto city. In 2018, MoEC tried to replicate the World Bank e‐RKAS system using a system that requires internet connectivity for use but realized that many schools have no access to the internet. Consequently, referring to the e‐RKAS system developed by Gunung Kidul district in Yogyakarta province, MoEC developed a partly online RKAS tool called the MoEC Aplikasi RKAS (ARKAS). The application is an offline tool that can downloaded from the Education Office website and used by public and private schools, in areas with low or no internet connectivity. In addition, since 2015, other electronic school planning and budgeting applications have been developed by district or province education offices. Typically, applications developed by districts have the same basic functions: planning use of BOS and Bantuan Operasional Sekolah Daerah (BOSDA or local government BOS) funds; reporting use of funds against budgets; asset management; and financial administration (bookkeeping). In March 2019, the Ministry of Home Affairs (MoHA) introduced a fully online application called Sistem Informasi Pengelolaan BOS (SIPBOS or BOS management information system), which focuses on planning, monitoring, and expenditure reporting of school BOS funds. MoHA considers this system to be a refinement of MoEC’s ARKAS as it has more complete electronic management and administrative systems for BOS, from school level to district/province level.
1 Assessing the Role of the School Operational Grant Program (BOS) in Improving Education Outcomes in Indonesia, The World Bank, 2014.
5
In 2018, the World Bank Team also implemented a school planning and budgeting application for the Ministry of Religious Affairs (MoRA)’s religious schools. Having completed pilots successfully, MoRA has adopted the program and will implement it nationally to 50,000 madrasahs through the Realizing Education’s Promise project implemented in 2020 to 2024, which is funded by the World Bank.
Evaluating e‐RKAS
To assess the preliminary effects of introducing different e‐RKAS applications on the role and behaviour of stakeholders, as well as challenges experienced during implementation, the World Bank conducted an evaluation of the e‐RKAS program. The evaluation explored school perceptions of e‐RKAS, challenges faced, and differences in education outcomes and BOS spending experienced caused by use of the different applications. The initiatives evaluated include the pilot program implemented by MoEC and the World Bank and e‐RKAS applications implemented by local governments in eight districts. The evaluation comprised various methodologies: 1) A review workshop on the e‐RKAS pilot implementation. 2) Qualitative evaluation undertaken through in‐depth interviews guided by questionnaires, conducted in a
sample of 56 randomly selected schools, including primary and secondary schools in four districts and senior high schools in three provinces.
3) A quasi‐experimental research exercise undertaken to compare differences in education outcomes as the result of the different types of e‐RKAS implementation, using data collected in eight sample areas.
Review Workshop Findings The workshop determined that benefits of the e‐RKAS training noted by districts/cities, education offices, and schools included: 1) Assisting districts/cities, particularly in the planning sector (Bappeda) and Local Income and Finance and
Asset Management Unit (DP2KAD), to monitor planning, budgeting and use of education budgets. 2) Enabling the education office to monitor, evaluate, and supervise school planning, budgeting, and
reporting on use of school funds and achievements of the NES for each school. 3) Supporting schools to improve school planning and budgeting based on school quality scores, financial
administration, reporting, achieving the targets of the NES, and focusing on achieving student learning outcomes.
During the workshop, problems and challenges experienced during implementation of the pilot were identified. Challenges faced by Education Offices at districts and provinces included the following: (1) due to limited financial capacity, the Education Offices could not provide servers, conduct training at all schools, provide technical assistance during school‐level implementation, conduct monitoring and evaluation, or regularly edit local prices in the system within a single year; (2) it was difficult to coordinate with Badan Pengelolaan Keuangan dan Aset Daerah (BPKAD) or Local Finance Office to set up and regularly update local prices used for goods and services; and (3) there is no mechanism to analyse the school NES score provided by the MoEC team and provide recommendations on how schools might improve their performance based on the scores. Problems faced by schools included the following: (1) not all schools had the opportunity to participate in e‐RKAS training; (2) during the e‐RKAS training, not all schools had the NES score to use as the basis for school planning; (3) they struggled to differentiate between activities and sub activities; (4) they experienced problems in constructing outputs and outcomes for activities; (5) they found it difficult to estimate the number of goods and services needed for each activity/sub activity in the expenditure plan; and (6) they had limited understanding about cash inflow and cash outflow management, as well as bookkeeping concepts.
6
Interview Findings Interviews were conducted in schools in the five pilot areas as well as in DKI Jakarta province and Gunung Kidul district. They aimed to verify the availability of key planning documents, notably the Rencana Kerja Sekolah (RKS or school four‐year plan), Rencana Kerja Tahunan (RKT or annual school plan), Rencana Kegiatan dan Anggaran Sekolah (RKAS or school annual activity and budget plan), and the Evaluasi Diri Sekolah (EDS) or School Self‐Evaluation (SSE). They also reviewed the experience of schools in using the e‐RKAS application and the effect this had on school behaviors. With respect to planning documents, the evaluation found that: • Most schools (92%) had developed an RKS, though only 80% were able to show evaluators the document.
Where schools had developed an RKS, the principals felt that the document is important and should be produced and available. However, as schools perceived that there are no incentives to produce an RKS or consequences if there is not one, seven percent of the evaluated schools had not produced one.
• Most schools evaluated (89%) had developed an RKT. Only 50% of schools indicated that they send the RKT to the relevant Education Office, so it appears that these Offices do not require schools to submit an RKT. Where it was received, the Education Office filed the RKT to monitor and review school planning.
• All schools in the study indicated that they had developed an RKAS, but only 73% were able to produce the document when requested to do so. Thus, it appears that some schools struggle to administer school planning documents effectively. Only half of schools indicated that the school supervisor played a role in ensuring that RKAS activities would contribute to achieving the NES and improving learning quality. As with the RKS and RKT, it seems that school supervisor participation in the development of RKASs is not optimal.
• All sampled schools had completed the SSE, but only a small percentage use it as a reference when developing the RKAS. It seems that school see SSE as a separate document, not part of the whole system of planning and budgeting.
• Most schools (62%) felt that SSE contains too many indicators while 38% indicated that the number was adequate. Generally, though, most schools felt that SSE is very useful.
With respect to implementation of e‐RKAS, the evaluation found the following: • More than half of the schools in the study (52%) had used an e‐RKAS application. • Most schools noted that using e‐RKAS made them more disciplined in terms of time, which was important
because the system applies a structured schedule in the input process. However, some schools reported problems keeping to the schedule, either because of connectivity problems or due to revision of the application.
• A local regulation on RKAS is important as a legal framework to require schools to develop the documents. However, some schools did not know local government had developed such regulations, which could be because dissemination of the regulation was weak or not effective.
• Implementing e‐RKAS: improved school discipline in the process of developing the RKAS document (95.51% of sampled schools); made it easy to monitor progress (95.51%); and gave schools clear management guidance on using the budget (95.51%). Other improvements noted were that schools could define their priorities (93.26%) and determine the associated activities to implement (89.89%).
Evaluating the impact of e‐RKAS on student achievement Additional analysis was undertaken to identify the impact of e‐RKAS implementation on learning outcomes and to determine whether the varying types of e‐RKAS application impacted learning outcomes differently. The evaluation focused on availability and completeness of functions in the application. Because different e‐RKAS applications were used by local governments across different areas in Indonesia, the study’s hypothesis was that the more complete the application’s functionality, the more positive the effect would be on learning outcomes. Three groups were created to test this hypothesis, an ‘advanced e‐RKAS’ group, an ‘medium e‐RKAS’ group, and a control group. Analysis was conducted for the e‐RKAS applications implemented in eight
7
sampled areas, namely, DKI Jakarta province, Bogor city, and Bekasi city, Bogor district, Bekasi district, Tangerang city, Tangerang district, and Depok city. While e‐RKAS can make a positive contribution to improved student achievement in theory, it is not really possible to isolate use of e‐RKAS as the only variable that would drive changes in student achievement given numerous social, economic and educational factors that might also have an effect. These would include factors such as teacher competency, teaching and learning strategies, curriculum, learning assessment systems, equipment and facilities, and supervision from government. To eliminate potential confounding variables, the difference‐in‐differences method, or interrupted time series, was selected as the empirical identification strategy to disentangle changes in student achievements caused by the adoption of e‐RKAS. As all the data for intermediate outcomes were not available to be analysed, the exercise focused on student learning results only. To analyse the impact of differences that e‐RKAS implementation has had on learning results, an econometric model was applied. The result of this analysis was that neither the advanced e‐RKAS nor the medium e‐RKAS had a significant impact on student achievement. For both treatment groups, there was no significant variation in shift in either the level of student achievement or annual changes in the trend. There were no notable differences on learning results between schools that applied the advanced e‐RKAS or medium e‐RKAS, when compared with those not using e‐RKAS. Having noted the various limitations in this exercise, the results produced so far do not provide clear evidence that adoption of e‐RKAS does indeed improve student achievement as per our hypothesis. This should not, however, be taken imply that e‐RKAS definitively does not contribute to improving student achievement, but rather just that this first investigation does not provide supporting evidence for that hypothesis.
Recommendations
Schools clearly perceive that the SSE instrument is useful for them to evaluate their performance, but there are too many indicators Likewise, the e‐RKAS application, in any forms, improves school discipline in planning processes. However, the role of school committees and school supervisor in these planning processes seems weak. Most participants in the pilot believed that using e‐RKAS would lead to more timely school development, greater ease in determining activities and setting priorities, more accurate budgeting, and more accountable and transparent financial reports that are also easier to create. Based on the experiences of these pilots, the following recommendations are offered to guide further implementation: 1) Strengthen and simplify the SSE and e‐RKAS program and applications. SSE results are a key factor in
development of an RKAS as they are used as the basis for determining school activities to achieve the NES targets. Therefore, the SSE instrument needs to be revisited in terms of the number of the indicators and the measurement methods to ensure the validity of evaluation results. In further developing the SSE, indicators should be simpler reflecting the most important and relevant criteria. The MoEC ARKAS should incorporate functions to record SSE results and detailed activities to improve NES performance, as well functions to analyze school NES achievements and Minimum Service Standard (MSS) results annually and in four‐yearly cycles. The MoHA SIPBOS should include functions to capture SSE results, school NES achievement results per year and four years, teacher and school principal performance, and MSS results so that MoHA can see the school performance as a whole.
2) Eliminate unnecessary duplication between systems. To avoid confusion and additional administration,
District Education Offices (DEOs) and Provincial Education Offices (PEOs) should use the applications issued by MoEC and MoHA. It will also be beneficial to link the e‐RKAS to MoEC’s SSE application. Likewise, if MoEC and MoHA continue to implement both ARKAS and SIPBOS, these two applications should also be linked to prevent schools having to input the same data twice and to eliminate duplicate functionality.
8
3) Training on the applications should be widely conducted and cover all schools. Training can be done efficiently through a cascade training system. At the central level, there should be a national team that will train provincial teams, which could in turn become trainers of district teams. District team would then train school teams. Training for schools should be implemented by and under the coordination of the PEO/DEO. It could be financed by PEO/DEO or the school itself. Formative evaluations of training should be conducted by PEOs/DEOs, in collaboration with MoEC‐MoHA to assess school understanding of ARKAS and SIPBOS, challenges faced, and recommendations for improving both applications. Regular capacity building is also needed for PEOs/DEOs on how to conduct their roles and functions in the development of school planning and budgeting. This should be conducted by MoHA in collaboration with MoEC to ensure that PEO/DEO staff are able to supervise schools effectively. Cascade training is needed to ensure that: there is affordable, continuous training and capacity development for schools; quality control takes place in the system; the e‐RKAS program can become sustainable in the long term; and government structures (especially PEOs and DEOs) can monitor school progress and performance. The cascade approach can also be optimised by using cluster‐based approaches, such as principal and teacher working groups, to discussing implementation, challenges, and actions taken. To be successful, it should also be supported by provision of regular technical guidance to schools by the relevant education office, including school supervisors.
4) PEOs/DEOs should provide supporting facilities to implement ARKAS and SIPBOS. In line with current trends of conducting business affairs online and to support implementation of e‐government, PEOs and DEOs should provide servers or cloud‐based hosting to store all school planning and budgeting data and to facilitate the operation of ARKAS and SIPBOS. PEOs/DEOs should also assign a team to review the status of NES and MSS measured through the SSE, school activities and budget plans, NES targets to be achieved, and the achieved NES and MSS at the end of the fiscal year, as well as the involvement of school stakeholders in implementation of ARKAS/SIPBOS. Based on these reviews, they should provide schools recommendations and strategies to improve NES achievements and how this should translate into school plans and budgets. PEO/DEO reports analyzing school spending, school achievements, and recommended actions need to be shared with MoEC and MoHA annually as an attachment accompanying the proposed BOS allocations.
5) Develop regulations at all levels to smooth the implementation of e‐RKAS. MoEC should explicitly state in the BOS manual that ARKAS should be used as a tool to develop the plans for BOS use. The guideline should specify the school stakeholders that are required to be involved in the development of ARKAS. MoEC and MoHA need to reinforce the implementation of the circular letters in the field, evaluate implementation, and use evaluation results to improve the circular letters. Both Ministries should apply mechanisms to incentivize schools to apply e‐RKAS. These might include: i) making implementation of e‐RKAS and improved NES achievement indicators of success in a BOS performance scheme; ii) making implementation of e‐RKAS one of the requirements for receiving the BOS affirmative; and iii) not distributing BOS funds, except for the school fixed cost component, where e‐RKAS has not been implemented. Likewise, regulations at local government level are also needed. Finally, MoEC could make the ARKAS‐based BOS plans and reports produced by the application a requirement for BOS disbursement to ensure that all schools use the application. The planned school planning application will become an important mechanism in BOS policy, enabling direct transfer of funds from the centre to schools and providing a marketplace where schools spend BOS funding. The marketplace could usefully be integrated with the planning application electronically to become one integrated tool for school planning.
9
Introduction
Background
About 53 percent of total government spending in Indonesia—excluding subsidies and interest payments—is conducted by subnational governments, with district governments managing about 38 percent and provincial governments the remaining 15 percent.2 There are 34 provinces and 514 districts in the country. The government budget has more than doubled in real terms over the past 15 years (from 2003 to 2018) in tandem with economic growth, which has benefited all sectors and particularly education. Growth of the government budget, coupled with a constitutional mandate to allocate 20 percent of the budget to education, expanded education resources significantly. The overall education budget grew threefold in real terms during this 15‐year period, increasing to IDR 492.5 trillion (US$32.8 billion) in 2019.3 The education system under the Ministry of Education and Culture (MoEC) is highly decentralized. Most school costs under MoEC4 are covered by fiscal transfers from the centre to provincial and district levels, some of which are earmarked for education use. One type of transfer is the school operational grant, known as Bantuan Operasional Sekolah (BOS) or school operational assistance. These are provided to all formal public and private schools on a per‐student basis. BOS is transferred directly from the Ministry of Finance (MoF) to school accounts, public and private. In Law 23/2014, Special Needs Schools or Sekolah Luar Biasa (SLB), Senior Secondary Schools or Sekolah Menengah Atas (SMA), and Senior Secondary Vocational Schools or Sekolah Menengah Kejuruan (SMK) are the responsibility of provincial governments, while Early Childhood Education Centers or Pendidikan Anak Usia Dini (PAUD), Primary Schools or Sekolah Dasar (SD), and Junior Secondary Schools or Sekolah Menengah Pertama (SMP) are the responsibility of district governments. In Ministry of Home Affairs (MoHA)5 circular letter No. 910/106/SJ/2017, it is stated that BOS funds received by public schools must be recorded in the planning and budgeting document or Rencana Kegiatan dan Anggaran (RKA) of District Education Offices (DEOs) and Provincial Education Offices (PEOs), while BOS spending must be recorded in DEO and PEO financial reports. BOS funds also must be included in the district/provincial Annual Budget Plan or Anggaran Pendapatan dan Belanja Daerah (APBD) and financial report. According to the MoEC’s BOS Guidelines for Schools, 6 BOS aims to increase access and raise education outcomes through: (1) direct support for school operating costs; (2) exemption of tuition fees for public primary and junior secondary students; (3) easing the cost burden on public senior secondary students and private school students in achieving quality education; and (4) financial assistance for children from poor households. The BOS guidelines note that use of these funds should be aligned with school priorities and to accelerate achievement of the National Education Standard (NES). BOS funds are managed directly by schools, which have been delegated the autonomy to receive, plan and budget, spend, administer, and report their use. BOS is governed by its own operations manual, which has been updated and improved annually since 2005. It has become the main source of school revenue, especially for public schools where it is mandatory to receive BOS funds. Public SDs and SMPs are not allowed to receive tuition fees from parents, other than voluntary contributions. SMAs and private schools (at all levels),
2 Indonesia Economic Quarterly, World Bank. 2017. Link: http://pubdocs.worldbank.org/en/657051513163708686/IEQ‐Dec‐2017‐ENG.pdf 3 All conversions of US dollars are made using the 2019 national budget assumption of IDR 15,000 per US$1.00. 4 One of MoEC responsibilities is formulating and establishing policies in the field of education at all levels from ECE, Basic and Secondary Education to Higher Education. 5 One of the responsibilities of MoHA is formulating, establishing policies for the local government in delivering services and for the local government development. 6 Permendikbud No. 1 Tahun 2018, Attachment, pp 10‐11.
10
conversely, can charge tuition fee to parents, even where they receive BOS funds. This is because BOS for SMAs and SMKs covers fewer than 50 percent of total costs. The tuition fee at private schools is typically used to pay honoraria to teachers and education personnel, as well as to improve school facilities. An Education Public Expenditure Review (PER) conducted by the World Bank in 2016 showed that BOS spending has increased eightfold in nominal terms and nearly quadrupled in real terms over the last ten years (2015‐2016). It identified BOS overspending on honoraria (at 17 percent, exceeding the maximum threshold of 15 percent as set out in the guidelines) combined with under‐allocation of funds for teacher professional development (a low 5 percent) and support for poor students (a mere 2 percent). A World Bank review of the participation of school committees in overall school accountability7 conducted in 2016‐17 also showed that, because of the lack of clear guidance, school committees had virtually no opportunity to be involved in school planning, budgeting, and implementation of school activities nor in the evaluation of school achievement through the School Self‐Evaluation (SSE).8 From the perspective of school‐based management, involvement of parents and community representatives is crucial, both in general and in school committees, as it helps to ensure that school planning and implementation activities are transparent, accountable, and participative. It provides communities opportunities to provide inputs on and review the development of school self‐evaluation processes, review the results, contribute to decision about key priorities, develop school plans, support activities that require additional funding, monitor implementation progress, and engage more deeply in learning processes. In this way, school management, including quality management, becomes not only the responsibility of the school team (teacher, principals, and other personnel), but also of parents and the community. Experience has shown that many schools lack the capacity to use BOS funds effectively and efficiently to deliver better learning outcomes for students, while there have been no planning systems to manage use of BOS funds towards achieving the NES for individual schools. Under the BOS program, each school is required to conduct an SSE against the eight standards of NES and use the results to develop its spending plans accordingly. However, most schools develop the SSE only when required to fulfil accreditation requirements. There has been no systematic review on the implementation and impacts of the SSE across schools. Thus, the only way to obtain data on school progress towards the eight standards is through the school accreditation process, which is only conducted in a sample of schools every five years. There is also no intensive monitoring and evaluation of how schools spend BOS funds. Link between the SSE and NES were documented in a study conducted by the World Bank (2018),9 which used a randomized control trial to evaluate the introduction of an SSE system in DKI Jakarta intended to improve planning and use of school operations funds to support achievement of the NES. The intervention included a non‐monetary incentive performance contract between 50 randomly selected schools and district government. The study found that overall, on an aggregate measurement which averaged performance across the eight standards, no significant impact was measured from having a performance contract with the government of DKI Jakarta. However, when looking at the subcomponents that make up the NES, the study found some differences between schools that signed a performance contract and those that did not. As there was one full cycle of school budget planning and implementation in the evaluation period, this limited the validity of the study findings, but the paper provided some insights into NES achievements and the schools’ approaches to this. Having limited evidence on these issues, this report of e‐RKAS evaluation will significantly give input on the future development of performance‐based planning and budgeting in school level.
7 Review of Community Participation on School Autonomy and Accountability (World Bank, December 2016 – July 2017). 8 The MoEC SSE instruments can be found here: https://pmp.dikdasmen.kemdikbud.go.id/assets/files/unduhan/2019/02‐perangkat‐instrumen‐2019‐jenjang‐smp.pdf 9 Tazeen Fasih, Unika Shrestha, Santoso, Wisno Harto Adi Wijoyo, Performance Contracts for Schools in DKI Jakarta – early results from an experiment, The World Bank, 2018.
11
The Theory of Change Behind e‐RKAS
Although BOS constitutes only approximately 10% of total education budget, it is the main direct income for public school and small private schools. Thus, school‐level decisions about how to spend BOS and other school resources can make a major contribution to the quality of learning. Every school has also been advised to develop a planning and budgeting system (Rencana Kegiatan dan Anggaran Sekolah or RKAS) to allocate and manage BOS funds. The World Bank review of BOS in 2016 revealed that some schools develop an RKAS annually, while others simply copy the previous year’s data. Further, where they are developed, the RKAS is typically not based on the school’s SSE results, which means that it is not systematically aligned with the eight NESs. Finally, since government fiscal year (FY) 2017, MoHA, as the institution supervising local government expenditure, requires that BOS expenditure be reported in three categories: (i) investment; (ii) payments; and (iii) goods and services. Unfortunately, the RKAS and its reports had still not been classified into these three categories by FY 2019. BOS is provided in the form of a block grant, while schools report BOS expenditure based on 11 eligible expenditure components. The concept of performance‐based planning and budgeting was introduced to ensure that schools use the RKAS to improve their performance. The RKAS functions as a school strategic planning instrument. It lists activities that will be implemented in a fiscal year and is thus an important reference for a school to ensure that activities will support learning processes and lead to performance achievements that meet the NES. The idea is to link the SSE to the eight areas of the NES, identify NES targets at the school, and plan activities to achieve these targets. In 2016, the education quality assurance body of the Directorate General of Basic and Secondary Education, MoEC developed and issued an electronic SSE tool called Penjaminan Mutu Pendidikan (PMP or education quality assurance system). The output of the PMP is a school quality report that describes school achievements against each indicator of the NES. Using this tool, individual schools can set NES targets for the following year, based on the score of the existing SSE, and then translate the NES targets into activities and accompanying budget line items in the RKAS. At the end of the year, the SSE is conducted again to verify whether the targets were achieved. Most schools manage SSE and RKAS manually, while some do not develop it or simply copy RKAS data from previous years or from other schools. Further, manual systems make it difficult for education authorities to verify that performance‐based planning and budgeting has taken place and to review its quality. To solve these problems, the concept of an application called the Rencana Kegiatan dan Anggaran Sekolah Berbasis Elektronik (e‐RKAS or electronic school plan) was developed. This tool enables schools to input and manage the RKAS electronically, which is more efficient as it can report in real time on expenditure, while also providing a systematic structure to link activities and expenditure, in line with a performance‐based planning approach. The idea of e‐RKAS was to enable schools to develop, store, and manage school plans, as well as to monitor implementation of school activities. It would also increase transparency and accountability in financial management at the school level as information captured online can be monitored by the district or city education office, provincial education office, and the central government (Ministry of Education). Thus, in summary, the e‐RKAS application was expected to improve the quality and frequency of proper school budgeting that aligns BOS allocations with school‐level strategies to achieve the NES and thereby improve their SSE scores. In parallel, online access to information would increase the transparency of school BOS budgeting and enable local government both to verify and review schools’ achievements in this regard. The e‐RKAS theory of change then assumes that budget allocations that are better aligned with achieving the NES will improve the quality of the school across the eight standards, which should in turn lead to improvement of the performance, and especially educational results of the school over time. This theory of change assumes, therefore that the standards defined in the NES were selected with improvement in educational performance as a key criterion. It can be summarized as follows:
12
Table 1 E‐RKAS Theory of Change
Problem Intervention Output Intermediate outcomes
Outcomes
Schools are not allocating BOS funds purposefully to achieve NES, with negative effects on overall quality and performance
Introduce e‐RKAS and link its use with SSE Make e‐RKAS data available online to district and provincial governments to ensure its use
• Schools produce school plans on time every year
• Schools use e‐RKAS to determine spending priorities, aligned with NES and intended to improve SSE scores
• Education office and supervisor are able review and monitor school progress in use of e‐RKAS
• School spending is more efficient and effective
• School budgets and annual plans are geared towards attainment of NES
• Implementation of school plans leads to improved SSE scores over time
• School spending is more transparent and accountable
• School attainment NES improves, leading to improved educational quality and better learning results
Thus, it was anticipated that adoption of e‐RKAS could be expected to lead to changes in the intermediate outcomes and ultimate outcome through multiple mechanisms. First, the school planning function of e‐RKAS enables schools to use the SSE to identify and prioritize areas to focus on for continuous improvement, while engaging school leaders, school committee members, teachers, and parents in school planning by allowing them to propose their needs for the following year. Second, since schools are required to align spending with their planning and e‐RKAS would provide government officials at different levels real‐time access to the budget and the expenditure to monitor use of school resources, the system is expected to help schools to improve fiscal responsibility, budget discipline, transparency, and accountability. Third, the inputs of schools are synchronized in the system, improving the accuracy of national, provincial, and district‐level budgeting. Fourth, the system eliminates the burden of creating reports manually or inputting the same data in different systems for reporting purposes. Fifth, the prices and calculations are built into the system, which helps to minimize human errors and improve the accuracy of budgetary data. Given this, use of e‐RKAS has potential to improve the transparency and accuracy of budgetary planning, increase efficiency of resource allocation and enhance the collaboration of stakeholders in this process. All these immediate outcomes in school management are expected to lead to better student achievement. Analysis of reports on the use of BOS funds in eight districts/cities shows that the highest percentage of BOS funds is spent on library development (14%), school management (15.8%), payment of honoraria (14.5%), extracurricular activities (14.1%), and maintenance of facilities and infrastructure (11.7%). Meanwhile, spending on professional development of teachers and education personnel is very low (3.2%),10 which may help to explain the lack of impact of BOS Funds on student learning outcomes. Because schools are not using e‐RKAS, they do not make a link between school planning and school quality scores, increased focusing primarily on activities that do not have a direct correlation with improved student learning outcomes. By using the e‐RKAS application, which links school quality improvement with school planning and budgeting, schools should hopefully become more targeted in use of BOS funds, focusing on activities that have a direct effect on student learning outcomes (for example, increasing budget allocations for teacher and education personnel professional development).
10 MOEC Report on BOS Spending 2018 and 2019.
13
Figure 1 summarizes a more detailed theory of change on the impact of e‐RKAS on student achievement. Figure 1 Theory of change on the impact of e‐RKAS on student achievement
There is some evidence that implementation of site‐based budgeting programs led to, at the very least,
increased awareness of the budget‐making process on the part of teachers and parents.11 A quantitative analysis of data on the relationship of performance‐based budgeting and student achievement reveals that performance‐based budgeting has a positive correlation to student achievement, although this correlation is not considered statistically significant. In addition, a Center on Educational Policy report confirmed that student achievement has increased (and the achievement gap has decreased) since the enactment of the No Children Left Behind program and its many interconnected policies and programs, one of which appears to be more focused budgeting methods and practices that aim to increase achievement.12 The Performance Driven Budgeting (PDB) implementation variable had positive and statistically significant effects on grade four reading and math scores and grade five reading scores. Schools that implemented PDB in 1997‐1998 showed standardized test scores higher by approximately .06 than non‐implementing schools. Small positive impacts found when the effect of both planning and implementing for the PDB schools was tested jointly.13
History of e‐RKAS Implementation
A first version of the e‐RKAS was created by the Education Office in Surabaya, East Java, to simplify the approval process of school plans and budgets. Using this online application, the Head of Education Office at Surabaya
11 Michele Moser, "School‐based Budgeting: Increasing Influence and Information at the School Level in Rochester, New York," Journal of Education Finance 23, no. 4(1998): 507‐53 1. 12 School District Budgeting and Student Achievement, Scott Alan Burckbuchler, SCHOOL BUSINESS AFFAIRS, MAY 2009 . 13 School Budgeting and School Performance: The lmpact of New York City's Performance Driven Budgeting Initiative, Leanna Stiefel, Amy Ellen Schwartz, Carole Portas, and Dae Yeop Kim, Journal of Education Finance 28 (Winter 2003), 403‐424.
Function Working Mechanism Intermediate
Outcome Outcomes
School
planning
Performance‐based planning: Schools use school self‐
evaluation (SSE) to identify and prioritize area to improve
Stakeholder engagement: school leaders, school
committee, teachers and parents are engaged in school
planning by proposing their needs for the following year
Fiscal responsibility and budget discipline: Schools are
required to spend on their planning
Transparency and Monitoring: DEO, provinces and central
government have on‐time access to the budget and the
expenditure to monitor the use of school resources
Informed budgeting for BOS and BOP: Improve the
accuracy of national, provincial and district‐level budgeting
through synchronization
Reduction of repetitive labor: The system eliminates the
burden of creating reports manually or inputting the same
data in a separate software for reporting purpose
Accuracy of data: The prices and calculations are built in
the system in order to minimize human errorrs
Budgeting
Financing
Reporting
Assets
management
and
inventories
Transparency:
Audit findings
Efficiency of
resource
allocation:
School
performance
Collaboration:
Teacher
disposition/
satisfaction
Student
Achievement
14
could monitor, analyse, and approve the RKAS and reports directly through the application. Following this deployment, in 2015 to 2016, the e‐RKAS application was further developed and introduced to all schools in DKI Jakarta, with support from the World Bank. After observing implementation in Surabaya city and DKI province, MoEC showed interest in implementing the e‐RKAS program more widely. Consequently, in 2017, through the Pillar 2 ID‐TEMAN Trust Fund financed by the Australian Department of Foreign Affairs and Trade (DFAT), the World Bank and MoEC piloted the e‐RKAS program for schools in Bali province, Central Java province, Gorontalo district, Sidenreng Rappang (Sidrap) district, and Mojokerto city. The piloting aimed to test the adequacy of the tool to accommodate schools’ needs in planning and to identify challenges in use of the system. Follow up communication from MoEC indicated that the government plans to implement e‐RKAS nationwide because the pilot activities supported by the World Bank have provided evidence that use of e‐RKAS improves government planning and efforts to develop electronic tools and systems for school planning. Use of e‐RKAS by schools has gained political support in the House of Representatives and is recommended to be implemented immediately at schools and at all education levels in Indonesia, because the Government allocated funds for education are very large (comprising 20% of the total state budget). In this context, an evaluation of the e‐RKAS piloting and the system is strategically important to provide information and insights into how a nationwide e‐RKAS should be developed and applied. In 2018, MoEC tried to replicate the World Bank e‐RKAS system using a system that requires internet connectivity for use but realized that many schools have no access to the internet. Consequently, Gunung Kidul district in Yogyakarta province developed a partly online RKAS tool called the MoEC Aplikasi RKAS (ARKAS). The application is an offline tool that can be downloaded from the Education Office website and used by public and private schools, in areas with low or no internet connectivity. Each Education Office at district level has a server used to store school data. MoEC used the ARKAS application to track school expenditures against the eight NESs. In 2018, ARKAS was piloted in 105 schools in nine districts from eight provinces, providing training to the participating districts (which were in turn expected to be responsible for training other schools in 2019). In addition to the above, since 2015, other electronic school planning and budgeting applications have been developed by district or province education offices. Typically, applications developed by districts have the same basic functions: planning use of BOS and Bantuan Operasional Sekolah Daerah (BOSDA or local government BOS) funds; reporting use of funds against budgets; asset management; and financial administration (bookkeeping). There are two broad kinds of applications developed by districts: 1. A complete menu of options, including planning and budgeting, financial administration, budget
realization, and reporting (with ledger and sub‐ledger, bank, cash, tax, and settlement reports). This type of application has been implemented in three districts (DKI Jakarta, Kota Bogor, and Kota Bekasi).
2. An application with only planning and reporting functions. This application has been implemented in five districts (Bogor district, Bekasi district, Tangerang city, Tangerang district, and Depok city).
Although these districts have their own application which their schools use to report their RKAS to the district/province, these schools are also expected to develop an RKAS report using ARKAS and upload this to the MoEC BOS Website. ARKAS reporting is a requirement to receive the national BOS. Most districts/cities have separated the roles of the school and Education Office in using the application. In Bekasi City, for example, schools input activities into the application while the Education Office conducts online verification and approval. In Tangerang District and Tangerang City, schools develop a school plan using an offline system and submit this to the PEO/DEO for verification and approval. In Bekasi District, the school inputs data into the application, while the regional Tim Perencanaan dan Anggaran (TPA or Planning and Budget Team) approves school plans. The TPA team is formed by the Regent and includes personnel from the local Finance Office (Dinas Pendapatan Pengelolaan Keuangan dan Aset Daerah or DP2KAD), local Planning Office (Bappeda), and the Regional Secretary. The PEO/DEO does not have access to the application, but, if it
15
wants to see the recapitulation of the school planning and reporting of BOS funds, these can be accessed through the District Communication and Information Office.
Development of a MoHA System In August 2018, when conducting ARKAS training within the identified provinces, MoEC shared the ARKAS system with MoHA to introduce it and explain its functionality. However, MoHA observed that the ARKAS did not link to the asset management system, local account codes, district information system, or existing audit systems and requested review and improvement of ARKAS to rectify these issues. In 2019, MoEC improved the ARKAS by providing reporting facilities required by MoHA and scaled it up nationally through the issuance of a circular (SE No. 4313/D/PR/2019 dated 10 April 2019), with districts and provinces advising schools to implement ARKAS. MoEC trained 52 districts in 29 provinces in support of this rollout. The remaining districts used their own school planning and budgeting applications while awaiting further instruction from MoEC and MoHA on the application to be used. In February 2020, MoEC announced that, in FY 2020, all schools should implement ARKAS to manage school BOS funds. In March 2019, MoHA introduced a fully online application called Sistem Informasi Pengelolaan BOS (SIPBOS or BOS management information system), which focuses on planning, monitoring, and expenditure reporting of school BOS funds. MoHA considers this system to be a refinement of ARKAS as it has more complete electronic management and administrative systems for BOS, from school level to district/province level. It has more a comprehensive reporting system, including asset management reports and reports based on local account codes, while it also links with district information systems and audit systems. SIPBOS is managed centrally by MoHA and data is stored on the MoHA Cloud Server. On April 30, 2020, MoHA issued a Ministerial Regulation No. 24/2020 on BOS Management for Local Government, which became effective on May 14, 2020. Consequently, from June 2020, schools are required to do double data entry, for both ARKAS and SIPBOS.
Deployment by the Ministry of Religious Affairs In 2018, the World Bank Team also implemented a school planning and budgeting application for the Ministry of Religious Affairs (MoRA)’s religious schools. Through the Pillar 2 ID‐TEMAN Trust Fund financed by DFAT, the Bank team provided technical advice and support for implementation of MoRA’s Elektronik Rencana Kegiatan Anggaran Madrasah (e‐RKAM or performance‐based school budgeting) in 30 madrasahs, 20 public and 10 private, in Jombang District. Following this, the MoRA office in East Java used its own funds to train all 38 districts in the province on use of e‐RKAM involving 6,219 madrasahs in total. In 2019, another 460 madrasahs in Jombang district have used e‐RKAM to develop BOS plans and budgets for FY 2020. In 2019, MoRA, using its own funds and with technical support from a World Bank Team, trained 120 core teams at central office and 34 provinces on use of e‐RKAM. A provincial team from nine provinces (Aceh, South Sulawesi, Central Java, East Java, West Kalimantan, Bali, Papua, West Papua, and Banten) joined a central team as trainers in delivering training on e‐RKAM to 2,000 madrasahs within those nine provinces. Having completed these pilots successfully, MoRA has adopted the program and will implement it nationally to 50,000 madrasahs through the Realizing Education’s Promise project implemented in 2020 to 2024, which is funded by the World Bank. In phase one of this project in 2020, MoRA plans to train approximately 17,000 madrasahs on use of e‐RKAM. MoRA’s e‐RKAM also includes its own SSE, called Evaluasi Diri Madrasah (EDM), which also uses the NES as a basis for planning and budgeting, as well as for setting targets to be achieved. Phase one madrasahs will implement EDM and e‐RKAM in year 2020, while those in Phase 2 and 3 will be trained in 2021 and 2022 respectively.
Research Methodology
To assess the preliminary effects of introduction of different e‐RKAS applications on the role and behaviour of stakeholders, as well as challenges experienced during implementation, the World Bank conducted an evaluation of the e‐RKAS program, the results of which are presented in this report.
16
The first evaluation explored school perceptions of e‐RKAS, challenges faced, and differences in education outcomes and BOS spending experienced caused by use of the different applications. The initiatives evaluated include the pilot program implemented by MoEC and the World Bank in five areas. This evaluation consisted of two phases, an evaluation of e‐RKAS implementation and a qualitative evaluation. The second was evaluation of the e‐RKAS applications implemented by local governments in eight districts. Three types of applications were developed by districts: online, offline‐online, and offline. 1) Online applications are those that can be used directly by schools to input activities for the use of BOS and
BOSDA funds, while the Education Office conducts monitoring, verification, and approval for all schools. This type of this application was implemented in four Districts (DKI Jakarta province, Bogor City, Bogor District, and Bekasi City). The application names are e‐RKAS (DKI), SIRKAS SIREAL (Bogor City), SIRKAS SIBOS (Bogor District) and SIMBOS (Bekasi City).
2) Partly online (offline‐online) applications are those that can only be used by the Education Office, not by
schools. For this application, schools developed their activities manually (creating a digital file and a printed document) and send the documents to PEO/DEO, after which the PEO/DEO teams will input data from all schools. This application is connected to the Regional Management Information System or Sistem Informasi Manajemen Daerah (SIMDA), so that local government can monitor and evaluated use of BOS and BOSDA funds by schools. This type of application was implemented in three Districts. The application names are SIMDA (Bekasi District), SPEKTRA (Tangerang City), and SIMRAL (Tangerang District).
3) Offline applications are those that are developed as an MS Excel spreadsheet. In this type of application,
schools input their activities into the spreadsheet and submit it to the Education Office in both digital and printed formats, after which are aggregated and analyzed by the Education Office for all schools. This type of application was implemented in one district (Depok City).
Evaluation of the e‐RKAS Pilot
Evaluation of the e‐RKAS Pilot comprised two steps, an evaluation of the e‐RKAS implementation progress after schools attending the training and interviews conducted after school implementing the e‐RKAS in school. As has been described in the Introduction, the World Bank and MoEC conducted a pilot of e‐RKAS in five selected areas, comprising three districts (Mojokerto, Gorontalo, and Sidrap) and two provinces (Central Java and Bali). The objective of the pilot was to test implementation of e‐RKAS at the school level to determine its applicability, identify problems in implementation, and collect inputs from schools, districts, and provinces on how to improve the system. The pilot project team developed an e‐RKAS application to assist schools in preparing plans and budgets based on school quality report cards, set NES targets to be achieved for continued improvement, identify activities to be implemented next year, and develop a budget using local price standards. The e‐RKAS application also provides functions to manage the school monthly cash spending, to manage school assets, as well as to provide necessary financial reports. The e‐RKAS structure is shown in Figure 2 below.
17
Figure 2 e‐RKAS Structure: illustrative example
Input Structure: School inputs activities and budget in the system
Note: Included in the components are local prices for each component.
Planning and Reporting Output: School prints or saves outputs of e‐RKAM
Output: 8 National
Standards Output: 3 types of
expenditure
Output: BOS type of
expenditure
1. Content
2. Infrastructure
3. Competency of
graduates
4. Financing
5. Education &
educational
Personnel
6.Management
7. Education
evaluation
8.Process
1.Personnel
Expenditure
2.Good and
services
expenditure
3. Capital
expenditure
1. Library development
2. Stationaries
3.Power and services
4. Enrolment activities
5. Learning &
extracurricular
6. Evaluation
7. School management
8. Teacher and staff
development
9. School maintenance
10. Teacher fees
11. Learning tools
12. others
Output: RKAS & Monthly
Expenditure Plan
RKAS: List of
school activities
and budget in a
year.
Monthly
Expenditure
Plan: monthly
disbursement
plan based on
RKAS.
18
Asset Report. School report for all asset they own, updated annually
The pilot started with coordination meetings at central level, which included MoEC representatives from the Secretariat General of Basic Education, Directorate of Junior Education, Directorate of Primary School, Directorate of Senior Education, and Directorate of Vocational Education. The meetings, which took place between March and September 2017, introduced and reviewed the e‐RKAS program and application, explained the link between e‐RKAS and SSE results in the Rapor Mutu produced by the MoEC PMP team, and identified the provinces and districts that would participate in the pilot. Following this, the World Bank team held a workshop with stakeholders at central and local levels in October 2017, which included representatives from MoEC, the PMP team, the heads of relevant DEOs/PEOs, and heads of relevant District and Provincial Financial Offices. During the workshop, the World Bank and MoEC team introduced the e‐RKAS program, presented plans to conduct pilots in the field, and asked participants for technical and policy support for pilots. The local government team promised to complete necessary preparations for implementation of the pilot. The e‐RKAS pilot commenced with workshops involving district and province officials in the five targeted areas. During the workshops, the Bank‐MoEC team, including PMP representatives, met relevant officials at district and province level from the Education and Finance Offices, as well as five representatives from primary schools, junior secondary schools, primary madrasahs, and junior secondary madrasahs. The number of participants in each district/province ranged between 28 and 50. The central team introduced the e‐RKAS program, detailed pilot plan, outlined required assistance from the local team, and requested their involvement in the pilot. Workshops were held between December 2017 and February 2018. The participating district/province teams received the pilot well and provided necessary assistance. After the workshops with province and district teams, Training of Trainers (ToT) was undertaken from May to June 2018 to introduce the e‐RKAS program to district and province technical teams and train them in use of the application. Through the ToT, DEO, and PEO teams were expected to be able to assist schools in e‐RKAS deployment, provide necessary guidance, monitor implementation, and evaluate school achievement and performance against the NES. All trained schools were guided to develop school plans for BOS and other funding sources for the following fiscal year using the e‐RKAS program. The trained teams were expected to then be able to train other schools in their region in use of the e‐RKAS program. TOT participants comprised the PEO/DEO officials, Office Heads, Heads of Basic Education Divisions, Heads of Planning Divisions, Budgeting Operators, District/Provincial Financial officials, and the provincial quality assurance institutions. Also included were school teams, comprising the principal, treasurer, and school operator of 25 selected school per district/province, the school operator being the person responsible for inputting school data into the Data Pokok Pendidikan (Dapodik or Education Main Database). A breakdown of participants is presented in Table 2 below.
Input school Asset
1. Asset name
2. Specification
3. amount
4. date of
procurement
5. condition
School Asset Report Asset reconciliation
at district level
1. Asset name
2. Specification
3. amount
4. date of
procurement
5. condition
1. Asset name
2. Specification
3. amount
4. date of
procurement
5. condition
19
Table 2 List of ToT Activities implemented in the pilot areas
No Pilot Area Participants Workshop TOT
1 Sidrap district District team 15 primary school 10 junior (jr) secondary school
Dec 19‐21, 2017 May 30‐June 1,
2018
2 Mojokerto city District team 15 primary school 10 jr secondary school
Jan 23‐25, 2018 May 22‐24, 2018
3 Gorontalo district District team 15 primary school 10 jr secondary school
Jan 9‐11, 2018 May 14‐16, 2018
4 Bali Province Province team 10 senior secondary (sr) school 10 sr secondary vocational 5 special needs school
Feb 5‐7, 2018 May 3‐5, 2018
5 Central Java Province Province team 10 senior secondary school 10 sr secondary vocational 5 special needs school
Jan 16‐18, 2018 May 7‐9, 2018
ToT included detailed training materials as follows: A. Introduction of e‐RKAS program:
Explained the concept of performance‐based planning and reporting of school achievements against the eight national education standards, including targets categorized as outputs as well as outcomes.
Explained how to operate the e‐RKAS system and functions in the application – such as dashboard, references, incomes, expenditure, realization, reporting.
Introduced the e‐RKAS website.
Explained how to input activities and budget in the system, including programs under the School Unit and the eight National Education Standards, detailed activities categorized under Activity and Sub Activity, and budget items (goods and services) under component, account codes, and volume.
B. Practise use of e‐RKAS system and application:
Created login and password using the Nomor Induk Pegawai (NIP) for each school principal and operator.
Provided principal logins and password. School principals had responsibility to decide on activities in each national standard based on results of the SSE, while the school operator is required to input detailed activities, good and services needed to conduct the activities, and prices.
Linked and used quality report (the report of school achievement on eight national standards and indicators) to define school challenges and select priorities of school activities in coming year.
C. Developed e‐RKAS for each school:
Created budget plan: revenue, activities to be done for 1 year, and Anggaran Kas Bulanan (AKB) or monthly cash planning that will also show monthly expenditure.
Provided an exercise for schools to follow the bookkeeping process step by step. including to input cash inflow and outflow, create a balance, and note transactions, as well as to document tax.
Developed a document of actual expenditure in the system. After the ToT, at the request of DEO and PEO officials, the Bank Team provided technical assistance (TA) while the trained schools disseminated e‐RKAS to other schools within the region. Three districts requested support from central team assistants: Gorontalo district, Sidrap district, and Mojokerto municipality, which was
20
provided between May to July 2018. TA provided included guidance in filling out the application step by step, inputting activities into the application, and solve technical and other substantive problems.
Evaluation of e‐RKAS Pilot with stakeholders A review workshop on the e‐RKAS pilot implementation was held in Jakarta in August 2018. The objectives of the workshop were to review progress in using e‐RKAS to develop RKAS 2019, share constraints, problems, and challenges experienced, and discuss follow‐up plans at both school and education office levels. The workshop was organized by the Bank‐MoEC team and attended by representatives from DEOs, PEOs, and District Finance Offices related to planning and budgeting. During the workshop, problems and challenges experienced during the implementation of the pilot were identified. For example, the central government in encouraging provinces and districts to implement the e‐RKAS program, while financial constraints made it difficult to provide a large server to host e‐RKAS within the ministry, and prevented the delivery of training to more schools and education offices to ensure that all schools would use the system.
Phase 2: Interviews
Beyond the review workshop outlined above, a qualitative evaluation was undertaken, where data were collected through in‐depth interviews guided by questionnaires, between October 2018 and February 2019. A sample of 56 schools was randomly selected, including primary and secondary schools in four districts and senior high schools in three provinces. Eight schools were selected from each district/province, comprising four elementary and four junior secondary schools at district level and four senior secondary and four vocational senior secondary schools at provincial level. The sample was selected from the list of schools that participated in training during the pilot to ensure that sampled schools had experience in using the e‐RKAS application. For the non‐pilot areas, DKI Jakarta and Gunung Kidul, the sample was selected randomly from the school population. The detailed sample of schools was as follow: Table 3 Participating schools in e‐RKAS interviews
Study Areas School Total
No SD/SMP SMA/SMK
Small Large Small Large
1 Gorontalo 4 4 8
2 Sidrap 4 4 8
3 Mojokerto 4 4 8
4 Gunung Kidul 4 4 8
5 Bali 4 4 8
6 DKI Jakarta 4 4 8
7 Central Java 4 4 8
Total 56
Notes: SD = primary school; SMP = junior secondary school; SMA =senior secondary school; SMK = sr. secondary vocational school
In each school, there were three respondents: the school principal, a treasurer, and the operator. The evaluation aimed to verify the availability of key planning documents, notably the Rencana Kerja Sekolah (RKS or school four‐year plan), Rencana Kerja Tahunan (RKT or annual school plan), RKAS, and the SSE (also known as the Evaluasi Diri Sekolah or EDS). It also reviewed the experience of schools in using the e‐RKAS application and the effect this had on school behaviors. The evaluation also explored the successes and challenges of the process of scaling up the e‐RKAS system and explored what models for delivering assistance might have the best results in terms of uptake and use.
21
Quasi‐experimental Impact Evaluation
To deepen understanding of the effects of e‐RKAS implementation, the evaluation program used a quasi‐experimental method to compare differences in education outcomes as the result of the different types of e‐RKAS implementation. Data were collected in eight sample areas covering DKI Jakarta province, Bogor city, Bogor district, Bekasi municipality, Bekasi district, Tangerang municipality, Tangerang district, and Depok municipality from September to December 2019. Data were collected through interviews with planning and budgeting officials in the target districts and provinces, study of documentations, and analysing DAPODIK (MoEC main education data). Data was then analysed using inferential statistics and descriptive analyses. The exercise focused on junior secondary education because reliable student achievement (learning results) data are not available for primary schools, while senior secondary schools are managed at the province level rather than by the district. The objective of the analysis was to identify the impact of e‐RKAS implementation on learning results and attempt to determine whether the varying types of e‐RKAS application impacted learning outcomes differently. The evaluation focused on availability and completeness of functions in the application. Because different e‐RKAS applications were used across different areas in Indonesia, as outlined in the previous section, there were different tools with varying functions including school planning, budgeting, financing, reporting, and asset management and inventories. The study’s hypothesis was that the more complete the application’s functionality, the more positive the effect would be on learning results (as summarized in the Theory of Change presented in Figure 1).
Key Findings
Feedback from the review workshop with Stakeholders
Challenges faced by Education Offices at districts and provinces included the following: (1) due to limited financial capacity, the PEO and DEO could not provide servers, conduct training at all schools, provide TA during school‐level implementation, conduct monitoring and evaluation, or regularly edit local prices in the system within a single year; (2) it was difficult to coordinate with Badan Pengelolaan Keuangan dan Aset Daerah (BPKAD) to set up and regularly update local prices used for goods and services (components); and (3) there is no mechanism to analyse the school NES score (school quality report) provided by the MoEC PMP team and provide recommendations on how schools might improve their performance based on the scores. Problems faced by schools included the following: (1) not all schools had the opportunity to participate in e‐RKAS training; (2) during the e‐RKAS training, not all schools had the NES score to use as the basis for school planning; (3) they struggled to differentiate between activities and sub activities; (4) they experienced problems in constructing outputs and outcomes for activities; (5) they found it difficult to estimate the number of goods and services needed for each activity/sub activity in the expenditure plan; and (6) they had limited understanding about cash inflow and cash outflow management, as well as bookkeeping concepts such as creating balances, note transactions and tax documents.
Feedback on Planning Documents
Education Minister Decree (Permediknas) No. 19/2007 requires schools to develop several planning documents, namely: the RKS, which needs to be developed once every four years; the RKT, which lists activities for the year derived from the RKS; and the RKAS, which presents detailed activities and a budget for the period. Most schools (92.45%) had developed an RKS, though only 80% were able to show evaluators the document. Typically, the RKS is developed at the beginning of the four‐year period of as school principal and functions as a reference guide for school management to determine the direction of school development. Where schools had developed an RKS, the principals felt that the document is important and should be produced and
22
available. They felt that it provides school management (particularly principals) a vision for managing and developing the schools, guides development of schools’ yearly plans (RKAS), and ensures that school plans and budgets can be made available whenever requested by a supervisor or auditor. However, as schools observed that there are no financial incentives to produce an RKS or consequences if there is not one (in other words, that BOS funds are allocated regardless of whether or not the RKS is produced), seven percent of the evaluated schools had not produced one. Where an RKS had been developed, principals (100%), teachers (83.84%), treasurers (90.91%), the school committee (76.77%), and school supervisors (64.65%) made up the core team responsible for its development. However, not many schools (only 16%) involved the Education Office in developing the document. It seems that Education Office officials generally do not engage with school planning processes. The RKS was signed by the principal, while some were also signed by the school committee. In a small number of schools, supervisors and the education office also signed it. The process to develop an RKS is as follow: Figure 3 RKS development process
Establish the team to develop RKS
Review and analysis the previous RKS on the achievement and challenges, the result of SSE, and school environment
Each of member of the team propose ideas, then the team develop draft of each component of RKS.
Discuss the draft of RKS to school stakeholders (teachers, education staffs, school committee, supervisors
Agree on the contents and legalize the RKS document.
The RKT records activities that will be conducted during a given school year and is derived from the RKS. The RKT provides the primary reference for schools to develop their RKAS. Most schools evaluated (89%) had developed an RKT. This indicates that most, but not all, schools perceive the document to be important. The process to develop an RKT is as follow: Figure 4 RKT development process
Review the previous RKT and conduct the SSE.
Develop draft of RKT. Discuss the draft with school stakeholders.
Discuss the activities list with stakeholders and identify priorities.
Share the content by school supervisor for review and receive feedback.
Agree and legalize the RKT document.
The school team that develops the RKT includes the school principal (93.8%), teacher (93.9%), treasurer (80%), and school committee (85.4%). Only half of public schools (49.4%) indicated that the school supervisor was involved in its development, again suggesting that school supervisors are often not intensively involved in planning development, but rather in reviewing teaching‐learning processes. Only 50% of schools indicated that they send the RKT to the relevant Education Office, so it appears that these Offices do not require schools to submit an RKT. Where it was received, the Education Office filed the RKT to monitor and review school planning.
23
All schools in the study indicated that they had developed an RKAS, but only 73% were able to produce the document when requested to do so. Thus, it appears that some schools struggle to administer school planning documents effectively. For example, at some schools, the document is stored on a personal laptop and the person was not available when the interview took place. The process of developing the RKAS is as follow: Figure 5 RKAS development process
Constitute team and discuss the activities to be implemented in the year.
Discuss and draft the goods and services required and the budget needed.
Develop draft RKAS. Input into RKAS application if available.
Review the draft and discuss with the Education Office.
Approve and Legalize the RKAS.
Typically, the principal and school committee had signed the RKAS document, but only half of school indicated that the Education Office had signed it. A signature from the Education Office is not an obligation but helps to ensure that their attention is given to the school’s planning. Education Office should ideally monitor schools as they develop their RKAS and supervise how schools use their budget and manage activities. Only half of schools indicated that the school supervisor played a role in ensuring that RKAS activities would contribute to achieving the NES and improving learning quality. As with the RKS and RKT, it seems that school supervisor participation in the development of RKASs is not optimal. Around 95% of schools send the RKAS to the Education Office. Reasons given for submission were that: it is a requirement for BOS disbursement (16%); submitted as a monitoring and evaluation document (66%); for filing (9%); and to comply with the regulation (91%). Where schools did not send the RKAS to the Education Office, this was because they believed that there is not a regulation which requires this and they perceived that RKAS is a school document that anybody who wants to check can review at the school itself. According to schools, the role of Education Offices in the development of RKAS are to: 1) train schools on RKAS development (75%); 2) manage the schedule of RKAS input (67%); and 3) review RKAS document (75%). All sampled schools had completed the SSE, using either the instrument developed by PMP or instruments developed by local government. The main reasons schools identified for conducting the SSE were to: use it as a basis for planning and budgeting (32%); understand school conditions (18%); and to identify school achievement on the NES (27%). Thus, as can be seen, although all schools interviewed stated that they develop an SSE, only a small percentage use it as a reference when developing the RKAS. It seems that school see SSE as a separate document, not part of the whole system of planning and budgeting. Stakeholders participating in teams responsible for SSE development included: principals (100%), teachers (100%), treasurer (100%), school committee (86%), supervisor (54%), and students (61%). As with other planning documents, involvement of school supervisors in the SSE is minimal. In 2017, MoEC introduced PMP as a quality assurance instrument to replace the various SSE instruments applied by districts. All schools are required to complete the PMP electronically. The instrument consists of many indicators and needs to be completed by the school principal, teacher, student, and school committee. The instrument for the school principal, for example, contains 120 indicators and 163 sub‐indicators, while the instrument for teacher includes 104 indicators and 143 sub‐indicators. Half of the instrument is automatically completed using Dapodik data, while the other half is completed by a school stakeholder team. According to interview data, 90% of schools indicated that the quality assurance instruments reflect actual school achievement, while about 10% felt it did not, as can be seen from Figure 6 below.
24
Figure 6 Extent to which schools perceive that SSE represents school achievement
Most schools (62%) felt that SSE contains too many indicators while 38% indicated that the number was adequate. Generally, though, most schools felt that SSE is very useful. Figure 7 School perceptions on number of indicators of SSE instrument
Figure 8 School perceptions on the usefulness of SSE as an instrument
At a district level, only DKI has a link to the PMP Application, while the seven do not. Those districts that do not have a link to the PMP application conduct an assessment, through SSE, to measure NES achievement at the end of each year. The NES score is assessed by the school principal and supervisor and then verified by the Education Office.
32.14%
57.14%
9.52%
1.19%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Totally reflect Quite reflect Less reflect Not reflect
61.54%
38.46%
0%0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
too many enough Less
62.82%
34.62%
2.56%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
very useful useful not really
25
The Education Office and Quality Assurance Unit use the results of SSE collected from schools. However, the study showed that there are variations regarding the involvement of both institutions, as presented in Figure 9. In Bali, Gorontalo, West Java, Central Java, and South Sulawesi, the role of the Lembaga Penjaminan Mutu Pendidikan (LPMP or Education Quality Assurance Institution) based in the province is typically more dominant than that of the Education Office (with the exception of Yogyakarta, where the Education Office plays a greater role than the LPMP. Figure 9 Institution responsible for managing the result of SSE
Implementation of e‐RKAS
More than half of the schools in the study (52%) had used the e‐RKAS application. During the period when the study was conducted, various e‐RKAS applications were being tested and implemented in the field, including an application developed by the World Bank, one developed by MoEC, and tools developed by local Education Offices. In the pilot districts and provinces, only Bali and Sidrap were still using the World Bank version of e‐RKAS, while MoEC version was being used mostly in Sidrap (in addition to the World Bank tool) and parts of Gunung Kidul and Gorontalo. DKI Jakarta, Gunung Kidul, and Mojokerto were using a tool developed by the Education Office. The DKI Jakarta tool was based on a World Bank e‐RKAS application piloted in 2015‐2016, which it had improved by linking e‐RKAS reporting with an online reporting system used in the Province Finance Office. Only a limited number of schools were using an e‐RKAS application because there many e‐RKAS tools were introduced at the same time, leading to confusion at local education offices and schools regarding which one needed to be implemented. Consequently, some DEOs, PEOs and schools decided to wait for authorities to determination which application should be used. In some cases, schools did not engage with an e‐RKAS application due to their lack of understanding of how to use it. Table 4 e‐RKAS applications in use in sampled schools
As can be seen from the above table, most sampled schools used tools developed by the Education Office. DKI Jakarta and Gunung Kidul were not included in the five areas of the World Bank pilot, while Mojokerto
66.67% 66.67%
0%
37.50%
75%
16.67%
80%75%
41.67%
25%
75%
100%
25%
100%
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
Bali DIYogyakarta
DKI Jakarta Gorontalo JawaTengah
Jawa Timur Sulawesi
Education Office LPMP (Quality Assurance Unit)
Bali Gunung Kidul DKI Jakarta Gorontalo Mojokerto Sidrap Total
The World Bank 2 0 0 0 0 2 4
MoEC 0 1 0 1 0 6 8
Both 0 0 0 0 0 6 6
Edu Office 1 14 5 0 6 0 26
26
observed, during the training period, that it had developed its own tool. Some schools involved in the World Bank pilot training (12 out of 56) indicated that they no longer use the e‐RKAS application and have instead reverted to using manual tools (in MS Excel). All pilot areas indicated that they had attended the World Bank team’s e‐RKAS training, including schools in DKI Jakarta, while only one school in Bali and one in Sidrap stated that they had received training from MoEC. Gunung Kidul, DKI Jakarta, and Mojokerto education office also conduct e‐RKAS training to schools. Table 5 e‐RKAS training received by schools
Most schools noted that using e‐RKAS made them more disciplined in terms of time, which was important because the system applies a structured schedule in the input process. However, some schools reported problems keeping to the schedule, either because of connectivity problems or due to revision of the application. A local regulation on RKAS is important as a legal framework to require schools to develop the documents. DKI Jakarta Province has issued a regulation on e‐RKAS, while some schools in Bali and Gunung Kidul indicated that they believed the local government has such a regulation. Gorontalo, Central Java, Mojokerto, and Sidrap did not, however, produce such a regulation. In schools that did not know local government had developed a regulation where it did exist, this could be because dissemination of the regulation was weak or not effective. Table 6 Existence of regulations on e‐RKAS
It was anticipated that implementation of e‐RKAS would change school management behavior in various ways. The e‐RKAS application was developed to help schools to manage its activities and budgets and to enable them to improve accountability and quality. Interview findings suggested that implementing e‐RKAS: improved school discipline in the process of developing the RKAS document (95.6%); made it easy to monitor progress (95.51%); and gave schools clear management guidance on using the budget (95.6%). Other improvements noted were that schools could define their priorities (93.26%) and determine the associated activities to implement (89.9%). Table 7 Reported benefits of using e‐RKAS
No Changes in School Behavior %
1 Define priorities 93.3%
2 Determine associated activities to implement 89.9%
3 Improved school discipline in developing the RKAS document 95.5%
4 Made it easy to monitor progress 95.5%
5 Gave schools clear management guidance on using the budget 95.5%
6 Schools could record transactions easily 85.2%
7 Made it easy to develop reports 84.1%
Bali Gunung Kidul DKI Jakarta Gorontalo Central Java Mojokerto Sidrap Total
The World Bank 10 0 4 5 6 6 9 34
Both 2 0 0 0 0 0 6 6
MoEC 1 0 0 0 0 0 1 2
Edu Office 1 15 12 2 0 7 0 37
Bali Gunung Kidul DKI Jakarta Gorontalo Central Java Mojokerto Sidrap Total
Yes 4 4 11 0 0 1 0 20
No 10 11 4 6 6 14 16 67
27
Impact of e‐RKAS on student achievement
E‐RKAS applications have five possible functions, each with their own working mechanisms: school planning, budgeting, financing, reporting, and asset management. Together, these are expected to make schools more transparent, more efficient in resources allocation, and better in collaboration with school staff (the intermediate outcomes). School transparency can be assessed in the result of audits. Efficiency of resource allocation can be measured in school performance, one indicator of which is achievements in the NES. Collaboration can be measured through teacher disposition or satisfaction, which might be seen in the results of teacher competency tests, teacher performance assessments, or simply even levels of teacher absenteeism. The outcome, student achievement, can be measured in learning results. While e‐RKAS can make a positive contribution to improved student achievement in theory, it is not really possible to isolate use of e‐RKAS as the only variable that would drive changes in student achievement given numerous social, economic and educational factors that might also have an effect. These would include factors such as teacher competency, teaching and learning strategies, curriculum, learning assessment systems, equipment and facilities, and supervision from government. It can also be anticipated that use of e‐RKAS might not yield immediate improvements (say, within one year of use) as it is likely that there would be a delay between initial use and evidence of effects. For example, in terms of school resources, as is illustrated in Figure 10, the average student‐teacher ratio, the number of laboratories, libraries, and classrooms per school all reverse their trend in 2017, the same time period when districts gradually transferred to the e‐RKAS system. It is difficult to know how these might have influenced learning achievements as much as, if not more than, implementation of e‐RKAS (or the extent to which implementation of e‐RKAS might have contributed to these trends). Figure 10 Changes in school resources (2015‐2018)
28
To eliminate potential confounding variables, the difference‐in‐differences method, or interrupted time series, was selected as the empirical identification strategy to disentangle changes in student achievements caused by the adoption of e‐RKAS. The difference‐in‐differences method can be used to determine the causal effect of e‐RKAS, because e‐RKAS was implemented at a specific moment (to generate the before‐and‐after difference) and a group of schools was identified to serve as a control group. According to this method, student achievement trends before and after the e‐RKAS implementation timeframe for the control group represents the impact of all the other economic, social and educational factors on student achievement in the absence of e‐RKAS (as is illustrated in Figure 11). Therefore, deviations in the treatment group from that of the control group could be considered the impact of e‐RKAS. Figure 11 The logic of difference‐in‐differences
While the logic of the difference‐in‐differences method is intuitive and simple, there are several difficulties when applying it to measure the impact of e‐RKAS on learning achievements in the Indonesian context. First, as revealed in the qualitive interviews, the treatment schools to be evaluated are varied as many applications of the electronic school planning and budgeting system were adopted by school districts and provinces. All these applications may differ in terms of available functions, user‐friendliness, and technical support provided, and therefore may have had different impacts on student achievement. If the treatment group is assumed to be homogenous, impact might be obscured as the impact of effective applications is reduced by ineffective ones. This problem was resolved by specifying the characteristics of the different e‐RKAS applications and then grouping those systems with similar characteristics into different treatment groups. With the help of the district education officials, the e‐RKAS applications were classified according to four dimensions. Each dimension is characterized by the functions embedded in the application, input mechanisms, administration structure, and the role of the DEO. These dimensions are expected to be predictive of the impact of e‐RKAS. Based on these four dimensions, interviews were conducted with the district officials of eight districts in and around the capital area to collect information on the characteristics of their e‐RKAS systems between November 2019 and February 2020. These eight school districts were selected due to their proximity to the researchers given tight timelines for data collection. They were Bogor district, Bogor city, DKI Jakarta province, Bekasi city, Bekasi district, Tangerang city, Tangerang district, and Depok city. As has been outlined above, the planning and budgeting tools used in the eight districts in the sample reflect a considerable variation in e‐RKAS functions across districts, ranging from online applications to offline systems. Based on the self‐reported characteristics of different district e‐RKAS applications, the eight school districts were categorized into three groups.
29
1) Bogor district, Bogor city, and DKI Jakarta used e‐RKAS applications that allow schools to create a budgetary plan, record financial transitions, generate budgetary reports and conduct asset management in the system. All their applications are maintained online and can be integrated with other school management software or applications. Therefore, these school districts were classified as the first treatment group, namely, ‘advanced e‐RKAS’.
2) The e‐RKAS applications used by Bekasi city, Bekasi district, Tangerang city and Tangerang district do not provide all the functions as the ‘advanced e‐RKAS’, some are maintained offline and none can be integrated with other school management software or applications. Therefore, these four districts were classified as the second treatment group, called ‘medium e‐RKAS’.
3) Depok city is still using an Excel version file for budget planning and could thus serve as the control group. The characteristics of e‐RKAS of the eight school districts in the same are summarized in Appendix 4. In addition to difficulties in identifying the treatment groups, challenges were encountered collecting data related to the ultimate and intermediate outcomes. The ultimate outcome, namely student achievement, is measured by the Ujian Nasional (UN or National Examination). Students in junior secondary education are tested annually in Bahasa Indonesia, English, Mathematics, and Science. Figure 12 shows the average UN scores of students for these four subjects in the eight school districts from 2013 to 2018. The district average is a weighted average of average school scores, with weights proportional to the number of students served by the schools. The test scores were also weighted by the integrity index when it was available in the dataset (i.e. the 2018 test scores). Figure 12 Average UN scores in Bahasa Indonesia, English, Mathematics and Science) of the sample schools
per district (2013‐2018)
As can be seen, this indicates a declining trend in UN test scores from 2013 to 2018, a pattern which applies to each of the four subjects when disaggregated (see Appendix 5 for details). This is partially because the UN test moved from a paper‐based test to a computer‐based test in 2018. To check the extent to which the trend in UN test scores captures a general decline of student achievement given the change in test administration, the UN test scores of Bahasa Indonesia, Mathematics, and Science from 2013 to 2018 were compared with the Programme for International Student Assessment (PISA) test scores in Reading, Mathematics, and Science from 2012 to 2018. Figure 13 shows that trends in UN scores align quite well with PISA score trends for each of the three subjects, indicating a high criterion‐validity for the UN test despite the change in the method of test administration.
30
Figure 13 Comparison of PISA test scores (2012‐2018) and UN test scores (2013‐2018)
In addition to student achievement, this process would ideally aim to test whether e‐RKAS implementation has a positive impact on intermediate outcomes to understand the mechanisms through which adoption of e‐RKAS could affect the ultimate outcome. These intermediate outcomes include transparency of the budget planning process, measured by reduction of budgetary errors revealed during auditing; efficiency of school resource allocation, measured by school performance indicators; and the collaboration of stakeholders in budget planning processes, measured by teacher satisfaction or disposition. Unfortunately, the data on any of these intermediate outcomes could not be collected. Table 7 below outlines challenges that made this difficult. For example, schools are audited once every five years, it was not possible to create a time series of audit findings before and after the year in which e‐RKAS was implemented. The only available longitudinal measure of the intermediate outcomes were the school performance indicators collected through the SSE by the MoEC PMP team. However, these self‐reported data do not show much variation over time. The school performance indicators are also collected through the accreditation process. Although more objective than the self‐reported data, the accreditation results are only available when a school is accredited every five years, making it impossible to create a time series that spans the year in which e‐RKAS was implemented. Table 8 Sources of data for intermediate outcomes and challenges in collection
Outcome of interest Source of data Potential Challenges
Reduction in budgetary errors (e.g. number of errors found during auditing,
Audit Findings Only available when a school is audited, so it is not possible to create a time series of the audit
31
Outcome of interest Source of data Potential Challenges
discrepancy between the correct number and the submitted number
findings before and after the year in which e‐RKAS was implemented
School performance indicators that are supposed to be affected by e‐RKAS
School Self Evaluation
Available each year but self‐reported
Accreditation result More objective than school self‐evaluation but only available when a school is accredited (once every five year), so it is not possible to create a time series of accreditation result before and after the year in which e‐RKAS was implemented
Teacher satisfaction/disposition Not available NA
The third challenge is that student achievement of the three groups does not seem to be paralleled before the adoption of e‐RKAS, as indicated in Figure 12. Unlike most diff‐in‐diffs studies, we were not able to check the assumption of paralleled pre‐treatment trends until we completed data collection on the characteristics of e‐RKAS systems used by the eight districts in the sample, as identification of the treatment arms is built upon the characteristics. Given this situation, we chose to control the pre‐treatment trend in the regression by allowing each treatment group to have a different slope of change. While acknowledging the methodological challenges explained above, the following model was used to estimate the causal impact that e‐RKAS implementation has had on learning results:
where 𝑌 is the UN average test scores for district d in year t; 𝑌𝐸𝐴𝑅 is the trend variable that is defined as the year of the test minus 2012 so that it starts with a value of 1 in 2013. 𝑃𝑜𝑠𝑡_𝑒𝑅𝐾𝐴𝑆 is a dummy variable that equals to 1 for observations starting one year after the year that an e‐RKAS system was established in this district, assuming that e‐RKAS started to impact the student achievement in the second year; 0 otherwise. In our sample, three districts started to use e‐RKAS in 2016, three districts in 2017, and one district in 2015. We centered the starting point by creating the variable 𝑌𝑅_𝑆𝐼𝑁𝐶𝐸_𝑒𝑅𝐾𝐴𝑆 , which is defined as the year of the test minus the year in which e‐RKAS was used, so that this variable takes on a value of 1 for the second year when e‐RKAS was used, 2 for the third year when e‐RKAS was used; years before e‐RKAS was used is coded as 0. 𝜇 represents the district fixed effect that captures the time‐invariant characteristics of each school district. Ƹ is the stochastic error term that has a mean of 0 and a standard deviation of 1. 𝑇 represents two dummies to indicate whether a district adopts the “advanced e‐RKAS” or the “medium e‐RKAS” respectively. The reference group is the control group that does not use e‐RKAS. Acknowledging that the three groups do not have the same trend of change, we added a term of 𝑌𝐸𝐴𝑅 to capture the average time trend and an interaction term between 𝑌𝐸𝐴𝑅 and the treatment dummies (i.e., 𝑇 𝑌𝐸𝐴𝑅 ) to reflect the deviation from the average time trend for each treatment group. In this regression, 𝛽 is the change in the level of student achievement induced by a specific treatment (i.e., the change in the intercept), while 𝛽 is the annual change of the trend of student achievement induced by a specific treatment (i.e., the change in the slope). For example, the impact of “advanced e‐RKAS” on student achievement in the second year that e‐RKAS is used is 𝛽 𝛽 for the dummy of “advanced e‐RKAS”, and that in the fifth year that e‐RKAS is used is 𝛽 4𝛽 for the dummy of “advanced e‐RKAS”. We applied the econometric model to the longitudinal dataset of the eight districts from 2013 to 2018, as can be seen in Table 9 below.
𝑌 𝛽 𝛽 𝑌𝐸𝐴𝑅 𝛽 𝑃𝑜𝑠𝑡_𝑒𝑅𝐾𝐴𝑆 𝛽 𝑌𝑅_𝑆𝐼𝑁𝐶𝐸_𝑒𝑅𝐾𝐴𝑆 𝛽 𝑇 𝑌𝐸𝐴𝑅 )
𝛽 𝑇 𝑃𝑜𝑠𝑡_𝑒𝑅𝐾𝐴𝑆 ) 𝛽 𝑇 𝑌𝑅_𝑆𝐼𝑁𝐶𝐸_𝑒𝑅𝐾𝐴𝑆 ) 𝜇 Ƹ
32
As all the data for intermediate outcomes were not available to be analysed, the exercise focused on the outcome only. To analyse the impact of differences that e‐RKAS implementation has had on learning outcomes, an econometric model was applied. The dependent variable is the learning outcome as represented by examination scores (UN score). The independent variables were the type of e‐RKAS implemented: advanced e‐RKAS and medium e‐RKAS, with e‐RKAS not implemented as the control variable. The detailed econometric model is presented in Appendix 6. We applied the econometric model to the longitudinal dataset of the eight districts from 2013 to 2018, as can be seen in Table 8 below. Table 9 Estimated effects of e‐RKAS on UN average scores
Independent Implications Estimate of the coefficient (Dependent: UN average
scores)
𝑌𝐸𝐴𝑅 Linear year trend ‐1.15 (1.32)
𝑃𝑜𝑠𝑡_𝑒𝑅𝐾𝐴𝑆 Shift before and after the e‐RKAS independent of the treatments
‐4.92 (3.47)
𝑌𝑅_𝑆𝐼𝑁𝐶𝐸_𝑒𝑅𝐾𝐴𝑆 Linear year trend for post‐e‐RKAS years only ‐4.05** (1.35)
𝑇 𝑌𝐸𝐴𝑅 Linear year trend for the districts with advanced e‐RKAS
1.79 (1.77)
𝑇 𝑌𝐸𝐴𝑅 Linear year trend for the districts with medium e‐RKAS 0.24 (1.75)
𝑇 𝑃𝑜𝑠𝑡_𝑒𝑅𝐾𝐴𝑆 Change in the level of student achievement induced by advanced e‐RKAS
(Dropped due to multicollinearity)
𝑇 𝑃𝑜𝑠𝑡_𝑒𝑅𝐾𝐴𝑆 Change in the level of student achievement induced by medium e‐RKAS
5.06 (7.58)
𝑇𝑌𝑅_𝑆𝐼𝑁𝐶𝐸_𝑒𝑅𝐾𝐴𝑆
Annual change of the trend of student achievement induced by advanced e‐RKAS
(Dropped due to multicollinearity)
𝑇𝑌𝑅_𝑆𝐼𝑁𝐶𝐸_𝑒𝑅𝐾𝐴𝑆
Annual change of the trend of student achievement induced by medium e‐RKAS
0.00 (4.49)
School district fixed effect Yes
N 48 Note. The standard errors reported in the parenthesis are the Newey‐West standard errors to address the autocorrelation of time series data. *** p <‐ 0.01, ** p < 0.05, * p <0.01.
The result of this analysis was that neither the advanced e‐RKAS nor the medium e‐RKAS had a significant impact on student achievement. For both treatment groups, there was no significant variation in shift in either the level of student achievement or annual changes in the trend. There were no notable differences on learning outcomes between schools that applied the advanced e‐RKAS or medium e‐RKAS, when compared with those not using e‐RKAS. Having noted the various limitations in this exercise, the results produced so far do not provide clear evidence that adoption of e‐RKAS does indeed improve student achievement as per our hypothesis. This should not, however, be taken as an implication imply that e‐RKAS definitively does not contribute to improving student achievement, but rather just that this first investigation does not provide supporting evidence for that hypothesis. The absence of a noticeable difference in learning outcomes may be for various reasons. First, as stated above, the potential impact of e‐RKAS may be obscured by a general declining trend in UN test scores caused by the change of the test administration method occurring in the same period when e‐RKAS was being introduced. As was shown in Table 8, this trend is captured by the significantly negative coefficient of 𝑌𝑅_𝑆𝐼𝑁𝐶𝐸_𝑒𝑅𝐾𝐴𝑆 ,
which indicates the linear year trend in the post‐e‐RKAS period. Second, in terms of organizational change, it is very likely that the impact of e‐RKAS on student achievement would not be fully felt immediately after
33
implementation, as it can take some years for school districts/provinces and DEOs/PEOs to improve practices. Third, the results may also be a function of the small sample size of 48 schools. Fourth, the absence of measurable NES targets to be achieved in school plans may also be a contributing factor explaining the absence of any correlation between implementation of e‐RKAS and improved learning result. Fifth, the prioritization of school expenditure mostly on school maintenance and low prioritization school quality improvement could explain the absence of link between e‐RKAS and learning results.
Conclusion and Recommendations
Based on evaluation of the e‐RKAS pilot implemented by the World Bank, the ARKAS implemented by MoEC, and the online planning and budgeting systems developed by local governments, several conclusions can be drawn. Schools clearly perceive that the SSE instrument is useful for them to evaluate their performance, but there are too many indicators Likewise, the e‐RKAS application, in any forms, improves school discipline in planning processes. However, the role of school committees and school supervisor in these planning processes seems weak. The World Bank e‐RKAS, which was introduced one year before the evaluation, was intended to improve school capacity to develop activities and budget plans of school resources based on their existing NES achievements and to decide NES targets to be achieved annually and over four years using a fully online system, the objective being to enable schools to plan their use of funding efficiently and effectively to achieve targeted improvements. Most participants in the pilot believed that using e‐RKAS would lead to more timely school development, greater ease in determining activities and setting priorities, more accurate budgeting, and more accountable and transparent financial reports that are also easier to create. MoEC’s partly online ARKAS and MoHA’s online SIPBOS programs focused on how the school, district, and province teams plan, manage, administer, and report BOS funds correctly using an electronic system. The systems aimed to ensure good governance in managing BOS funds. Both applications emphasized that the BOS plan should support the acceleration of achieving NES by the school. The local governments school planning and budgeting systems were used to systematically link school BOS planning and budgeting with the Education Office and Finance Office at district/province level. The system aimed to feed the local government SIMDA with information related to school planning and budgeting, as well as reporting. Thus, the World Bank e‐RKAS intended to improve school capacity to plan and develop budgets, monitor implementation, and report results as a strategy to register improvements in the NES. MoEC, and MoHA applications aimed to improve good governance in managing BOS, while local government school planning and budgeting systems were intended to enable systematic information‐sharing of school plans with education offices and other offices at district/province level. The functions of these different applications also varied. The WB e‐RKAS consisted of very detailed functions providing information on SSE results, lists of outputs and outcomes related to NES to be achieved, lists of programs and activities to improve each standard, activities and budget plans based on the local price standards, monthly cash spending plans, and financial reports based on BOS components, MoHA’s three main categories, and the eight NES standards. The MoEC ARKAS incorporated functions on activity and budget planning, as well as financial reports based on the three core classifications, BOS components, and eight national standards. The MoHA SIPBOS contained the same functions as ARKAS, but with more comprehensive reporting functions. The Bogor district application included a cashless purchasing system and asset management system, while the DKI Jakarta province e‐RKAS application linked to the MoEC SSE application. Thus, different systems, had different strengths and limitations.
34
The only schools that have the human capacity to operate e‐RKAS, ARKAS, and/or partly and fully online school planning and budgeting systems were those involved in the World Bank e‐RKAS pilot, ARKAS training, and local government systems training. Most districts/provinces had limited financial capacity to train other schools within the areas to use the application. Likewise, districts/ provinces had limited financial capacity to provide the required server infrastructure to store e‐RKAS data and provide internet connectivity (except big provinces like DKI Jakarta). Financial constraints also limited the provision of technical assistance to schools. Lack of directions and guidelines meant that the district/province did not analyze school NES achievements or provide recommendations on how to improve school planning and budgeting for better school performance. Before MoEC and MoHA issued the circular letter on ARKAS/SIPBOS, the school planning documents as required in Permendiknas 19/2007 were not developed by all schools, while not all schools submitted a detailed school plan to the PEO/DEO. The PEO/DEO only signed 50 percent of submitted school plans, indicating that a school plan verification system at PEO/DEO did not exist. Given the two circular letters, schools, districts, and province now have a legal requirement to develop school plans accordingly and regularly. MoHA circular letters, in particular, are usually considered a regulation for local governments. The regulations issued by the Governor and the Head of DEO and PEO in DKI Jakarta province also contributed significantly to institutionalizing the use of e‐RKAS for school planning and budgeting development in the province. Based on the experiences of these pilots, the following recommendations are offered to guide further implementation: 1) Strengthen and simplify the SSE and e‐RKAS program and applications. SSE results are a key factor in
development of an RKAS as they are used as the basis for determining school activities to achieve the NES targets. Therefore, the SSE instrument needs to be revisited in terms of the number of the indicators and the measurement methods to ensure the validity of the evaluation results. In further developing the SSE, indicators should be simpler reflecting the most important and relevant criteria. For example, in the MoRA e‐RKAS application, there are only five main SSE indicators: 1) school actors are disciplined in delivering their tasks; 2) teachers and education personnel attend professional development; 3) learning assessment is applied correctly; 4) teachers and students have access to and use full sets of learning materials; and 5) the school is planning and budgeting for BOS in efficient and accountable ways. MoEC and MoHA have agreed to improve their applications so that they can be used by the school, district, and province effectively and both applications can produce data and information needed for education quality and resource management improvement. The MoEC ARKAS should incorporate functions to record SSE results and detailed activities to improve NES performance, as well functions to analyze school NES achievements and Minimum Service Standard (MSS) results annually and in four‐yearly cycles. The MoHA SIPBOS should include functions to capture SSE results, school NES achievement results per year and four years, teacher and school principal performance, and MSS results so that MoHA can see the school performance as a whole.
2) Eliminate unnecessary duplication between systems. To avoid confusion and additional administration,
DEOs and PEOs should use the applications issued by MoEC and MoHA. It will also be beneficial to link the e‐RKAS to MoEC’s SSE application. Likewise, if MoEC and MoHA continue to implement both ARKAS and SIPBOS, these two applications should also be linked to prevent schools having to input the same data twice and to eliminate duplicate functionality. For example, if ARKAS contains a detailed list of activities to be implemented by schools and a school plan/expenditure report broken down by the eight areas in NES, then SIPBOS should be able to use this data and reporting system. Since SIPBOS incorporates local price standard data and school audit results, ARKAS should have access to this data and reporting facility.
3) Training on the applications should be widely conducted and cover all schools. Training can be done efficiently through a cascade training system. At the central level, there should be a national team that
35
will train provincial teams, which could in turn become trainers of district teams. District team would then train school teams. Training for schools should be implemented by and under the coordination of the PEO/DEO. It could be financed by PEO/DEO or the school itself. For example, the MoRA East Java office trained MoRA district office representatives using its own funds. Likewise, schools in Jombang district attended e‐RKAS training in DEO office using their own funds. ARKAS/SIPBOS training could either be online, a combination of online and offline training, and face‐to‐face training as and when the situation permits. To overcome problems in implementation of ARKAS and SIPBOS, on‐the‐spot technical assistance should be provided regularly by the PEO/DEO team. Formative evaluations of training should be conducted by PEOs/DEOs, in collaboration with MoEC‐MoHA to assess school understanding of ARKAS and SIPBOS, challenges faced, and recommendations for improving both applications. Regular capacity building is also needed for PEOs/DEOs on how to conduct their roles and functions in the development of school planning and budgeting. This should be conducted by MoHA in collaboration with MoEC to ensure that PEO/DEO staff are able to supervise schools effectively. Cascade training is needed to ensure that: there is affordable, continuous training and capacity development for schools; quality control takes place in the system; the e‐RKAS program can become sustainable in the long term; and government structures (especially PEOs and DEOs) can monitor school progress and performance. The cascade approach can also be optimised by using cluster‐based approaches, such as principal and teacher working groups, to discussing implementation, challenges, and actions taken. To be successful, it should also be supported by provision of regular technical guidance to schools by the relevant education office, including school supervisors.
4) PEOs/DEOs should provide supporting facilities to implement ARKAS and SIPBOS. In line with current trends of conducting business affairs online and to support implementation of e‐government, PEOs and DEOs should provide servers or cloud‐based hosting to store all school planning and budgeting data and to facilitate the operation of ARKAS and SIPBOS. PEOs/DEOs should also assign a team comprising a representative of the LPMP with experience in analyzing the SES results and using it for NES improvement, university and education experts, and school supervisors. This team should review the status of NES and MSS measured through the SSE, school activities and budget plans, NES targets to be achieved, and the achieved NES and MSS at the end of the fiscal year, as well as the involvement of school stakeholders in implementation of ARKAS/SIPBOS. Based on these reviews, they should provide schools recommendations and strategies to improve NES achievements and how this should translate into school plans and budgets. PEO/DEO reports analyzing school spending, school achievements, and recommended actions need to be shared with MoEC and MoHA annually as an attachment accompanying the proposed BOS allocations. Data from these reports can be used as inputs for the intermediate outcome related to transparency and efficiency of audit data. School achievements on the NES can be used to measure school performance, which is the second intermediate outcome. The local Planning Office (Bappeda) should regularly monitor school NES achievements and compare these achievements with education targets in district/province five‐year strategic plans.
5) Develop regulations at all levels to smooth the implementation of e‐RKAS. MoEC should explicitly state in the BOS manual that ARKAS should be used as a tool to develop the plans for BOS use. The guideline should specify the school stakeholders that are required to be involved in the development of ARKAS. MoEC and MoHA need to reinforce the implementation of the circular letters in the field, evaluate implementation, and use evaluation results to improve the circular letters. Both Ministries should apply mechanisms to incentivize schools to apply e‐RKAS. These might include: i) making implementation of e‐RKAS and improved NES achievement indicators of success in a BOS performance scheme; ii) making implementation of e‐RKAS one of the requirements for receiving the BOS affirmative; and iii) not distributing BOS funds, except for the school fixed cost component, where e‐RKAS has not been implemented. Likewise, regulations at local government level are also needed. Finally, MoEC could make the ARKAS‐based BOS plans and reports produced by the application a requirement for BOS disbursement to ensure that all schools use the application. The planned school planning application will become an
36
important mechanism in BOS policy, enabling direct transfer of funds from the centre to schools and providing a marketplace where schools spend BOS funding. The marketplace could usefully be integrated with the planning application electronically to become one integrated tool for school planning.
The Governor or the Mayor and Head of province/district could provide a decree informing stakeholders about ARKAS and SIPBOS and the need for schools to use them in planning, managing, administering, and reporting BOS and other funding resources received. The Head of the PEO/DEO could issue decrees on when the schools should commence planning and budgeting, when to revise budgets in the system, and when to submit financial statements (expenditure report) to the PEO/DEO. Finally, MoEC could make the ARKAS‐based BOS plans and reports produced by the application a requirement for BOS disbursement to ensure that all schools use the application. MoEC could make the ARKAS‐based BOS plans and reports produced by the application a requirement for BOS disbursement to ensure that all schools use the application. The planned school planning application will become an important mechanism in BOS policy, enabling direct transfer of funds from the centre to schools and providing a marketplace where schools spend BOS funding. The marketplace could usefully be integrated with the planning application electronically to become one integrated tool for school planning.
37
Appendix 1: The National Education Standards
Law No. 20/2003 on the National Education System, which defines a number of key areas of education including the function and purposes of education, curriculum, education personnel, and others, also set the basic foundation for establishment of the National Education Standards (NES). It was followed up by Government Regulation No. 19/2005 on National Education Standards, which defines the national standard in the following eight areas: • Content • Process • Graduate competency • Teacher standards • School facilities • Education management • Funding, and • Assessment The regulation also mandates the establishment of National Education Standards Board (BNSP), which is tasked with preparing the detailed education standards and overseeing their implementation. There is also an accreditation process for schools, which is executed by the National Accreditation Board for Secondary School (BAN‐SM). The most recent change of accreditation standard was to include adjustment to the 2003 law and NES implementation in schools. The structure of the accreditation instrument follows the order of the eight areas of NES. Eight National Education Standards and Criteria Graduate competence 1.1. Graduates possess attitude dimension of performance 1.2. Graduates possess knowledge dimension of performance 1.3. Graduates possess skill dimension of performance Education Content 2.1. Learning materials are in line with graduate competence formulation/design 2.2. School‐based curriculum is developed according to the stipulated procedure 2.3. School is implementing the curriculum according to the regulation(s) Learning Process 3.1. Schools plan learning process according to the regulation(s) 3.2. Learning process is implemented accurately 3.3. Supervision and authentic assessment are conducted during learning process Education Assessment 4.1. Education assessment is implemented according to competence domain 4.2. Assessment technique(s) is/are objective and accountable 4.3. Education assessment is to be followed up 4.4. Assessment instrument(s) is/are to be in line with assessment aspects 4.5. Assessment needs to follow the procedure Teachers and Education Personnel 5.1. Teachers availability and competency are aligned with the regulation 5.2. Principals availability and competency are aligned with the regulation 5.3. Administration staff availability and competency are aligned with the regulation
38
5.4. Laboratory staff availability and competency are aligned 5.5. Librarians availability and competency are aligned with the regulation Facilities and infrastructure 6.1. Sufficient school student intake capacity 6.2. Schools possess proper and sufficient facilities and infrastructure 6.3. Schools possess complete and proper facilities and infrastructure Management 7.1. Schools conduct implementation planning 7.2. Program management is implemented according to the regulation 7.3. Principals are to show good performance in his/her school leadership 7.4. Schools manage MIS (Education) Funding 8.1. Schools provide cross‐subsidy service 8.2. School operational load is aligned with the regulation 8.3. School implement sound fund management
39
Appendix 2: Numbers of Participants in Pilot Activities
No Participants Gorontalo Sidrap Mojokerto Central Java
Bali
F M F M F M F M F M
District Level
1 MoEC 1 1 2 1 1 1 2 2 2 1
2 Education Office
9 3 4 1 2 3 10 2 12 7
3 MoRA 6 1 1 4 1 5 1 4
4 BPKD 3 2 2 1 3 1
BP2MK 5 1
Diskominfo 2
School Level
5 SD 4 4 2 1 2 1
6 SMP 4 4 2 2 2 4
7 MI 2 2 1 1 4 4
8 MTs 1 1 1 1 2 1 3
10 SMA 4 2
11 SMK 5 1
12 MA 2 31 11
13 SLB 4
Total 30 7 19 9 18 10 38 11 62 22
Total Per Area
37 28 28 49 84
40
Appendix 3: Pilot Schools
No Gorontalo Sidrap Mojokerto Central Java Bali
1 SDN 9 TOLANGOHULA
SD NEGERI 1 ALLAKUANG
SD NEGERI MENTIKAN 6
SLB NEGERI SUKOHARJO
SLB NEGERI 1 KARANGASEM
2 SDN 9 TILANGO SD NEGERI 1 ARAWA
SD NEGERI MENTIKAN 4 SLB N SRAGEN
SLB NEGERI 1 DENPASAR
3 SDN 1 BILUHU SD NEGERI 1 BILOKKA
SD NEGERI KRANGGAN 5
SLB NEGERI JEPARA
SLB NEGERI 1 BULELENG
4 SD NEGERI 9 TABONGO SD NEGERI 1 KULO
SD NEGERI KRANGGAN 2
SLB NEGERI SURAKARTA
SLB NEGERI 1 BANGLI
5 SD NEGERI 8 BATUDAA
SD NEGERI 1 RAPPANG
SD NEGERI KRANGGAN 1
SLB Negeri Semarang
SLB Negeri 1 Badung
6 SD NEGERI 7 LIMBOTO BARAT
SD NEGERI 1 WATANG SIDENRENG
SD NEGERI KEDUNDUNG 2 SMA NEGERI 1
BERGAS
SMAN 1 TABANAN
7 SD NEGERI 7 BOLIYOHUTO
SD NEGERI 11 PANGKAJENE
SD NEGERI KEDUNDUNG 1
SMA NEGERI 1 CANDIROTO
SMAN 1 MENGWI
8 SD NEGERI 3 TALAGA JAYA
SD NEGERI 2 BELAWAE
SD NEGERI GEDONGAN 1 SMAN 1 WELERI
SMAN 1 BLAHBATUH
9 SD NEGERI 24 PULUBALA SD NEGERI 2 KULO
SD NEGERI BALONGSARI 5 SMAN 2 KUDUS
SMAN 1 BANGLI
10 SD NEGERI 2 TELAGA
SD NEGERI 2 MACORAWALIE
SD NEGERI BALONGSARI 2
SMAN 1 BLORA SMAN 1 AMLAPURA
11 SD NEGERI 2 MOOTILANGO
SD NEGERI 2 MOJONG
SD NEGERI SURODINAWAN
SMAN 1 MUNTILAN
SMAN 2 BANJAR
12 SD NEGERI 2 BILATO SD NEGERI 2 WATANG SIDENRENG
SD NEGERI SENTANAN
SMAN 1 BANJARNEGARA SMAN 2 NEGARA
13 SD NEGERI 18 TELAGA BIRU
SD NEGERI 3 MOJONG
SD NEGERI MIJI 4 MOJOKERTO
SMAN 1 REMBANG
SMAN 2 SEMARAPURA
14 SD NEGERI 13 LIMBOTO
SD NEGERI 3 OTTING
SD NEGERI MIJI 3 SMA NEGERI 1 BATANG
SMAN 3 DENPASAR
15 SD NEGERI 11 BATUDAA PANTAI
SD NEGERI 3 WATANG SIDENRENG
SD NEGERI MERI 1
SMA NEGERI 05
SMAN BALI MANDARA
16 SMP NEGERI 4 SATAP MOOTILANGO
SD NEGERI 5 AMPARITA SD NEGERI WATES 6 SMA NEGERI 05
SMK NEGERI 3 SINGARAJA
17 SMP NEGERI 3 SATAP TABONGO
SD NEGERI 5 BARANTI
SD NEGERI WATES 4 SMA NEGERI 05
SMKN 1 DENPASAR
18 SMP NEGERI 3 SATAP ASPARAGA
SD NEGERI 5 LANCIRANG SMP NEGERI 1 SMA NEGERI 05
SMKN 1 GIANYAR
19 SMP NEGERI 1 TIBAWA
SD NEGERI 5 PASSENO
SMP NEGERI 2 SMA NEGERI 05
SMKN 1 KLUNGKUNG
20 SMP NEGERI 1 TELAGA
SD NEGERI 5 WATANG SIDENRENG
SMP NEGERI 3
SMK NEGERI 03
SMKN 1 KUTA SELATAN
21 SMP NEGERI 1 PULUBALA SD NEGERI 6 BATU
SMP NEGERI 4 SMK NEGERI 01
SMKN 1 SUKASADA
22 SMP NEGERI 1 LIMBOTO BARAT
SD NEGERI 7 BENTENG SMP NEGERI 5 SMK NEGERI 02
SMKN 1 NEGARA
41
No Gorontalo Sidrap Mojokerto Central Java Bali 23 SMP NEGERI 1
LIMBOTO SD NEGERI 8 WATANG SIDENRENG
SMP NEGERI 6
SMK NEGERI 06
SMKN 1 MANGGIS
24 SMP NEGERI 1 DUNGALIYO
SMP NEGERI 1 DUA PITUE SMP NEGERI 7 SMKN 1 DEMAK
SMKN 1 TEMBUKU
25 SMP NEGERI 1 BONGOMEME
SMP NEGERI 1 PANCA LAUTANG SMP NEGERI 8 SMKN 1 BAWEN
SMKN 3 TABANAN
26 SMP NEGERI 1 PANCA RIJANG SMP NEGERI 9 SMKN 1 KENDAL
27 SMP Negeri 1 Pangsid
SMKN 2 KENDAL
28 SMP NEGERI 1 TELLU LIMPOE
SMK NEGERI 7 SEMARANG
29 SMP NEGERI 1 WATANG PULU
SMK NEGERI SATU ATAP TUNTANG
30 SMP NEGERI 2 BARANTI
31 SMP NEGERI 2 PITU RIASE
32 SMP NEGERI 3 PANCA RIJANG
33 SMP NEGERI 4 DUA PITUE
42
Appendix 4: Identification of the treatments
Characteristics of e‐RKAS applications for districts categorized as ‘Advanced’
Question Bogor District Bogor City DKI Jakarta Province
A Application Profile
1 Name of application SIRKAS SIREAL SIRKAS SIBOS e‐RKAS
2 Type of connection Online Online online
3 Since 2017 2016 2015
4 Developer District Education office District Education office Individual Consultant
5 Integrated to other applications such as SIMDA (District Financial Management System), DAPODIK, asset and inventory system
Integrated to Dapodik, Sirkas/Sireal, Lapor BOS, and SIPLAH in 2019 through window menu SIMAKLAH. SIPLAH is application for online and cashless purchasing system
Integrated to SIMDA and SIPLAH, online purchasing and cashless system
Indirect integration through injection mechanism to: 1) SIAP BOS/BOP; 2) SIMKD; 3) e‐asset; and 4) SSE (LPMP DKI Jakarta)
6 Coverage of users: public, private, primary, and junior secondary
Public: primary and junior secondary Public: primary and junior secondary
Public: ECD, primary, junior, senior secondary, vocational school, and non‐formal
B Functions in applications
Functions for planning: Sub menu: school revenue (source of funds), expenditures, check balance to check balance between revenue and expenditures, final output is RKAS
revenue, source of fund and expenditures. Final output is RKAS
revenue, source of fund and expenditures. Final output is RKAS and AKB (monthly cash management)
Functions for budget realization and financial administration
• Bookkeeping menu to record all transactions whether cash and transfer through bank (General bookkeeping/ledger, subledger for cash, bank, and tax)
• Payment: combination between cash and bank
• Bookkeeping menu to record all transactions whether cash and transfer through bank (General bookkeeping/ ledger, subledger for cash, bank, and tax)
• Payment: cashless
• For budget realization and financial administration, it is process through SIAP BOS/BOP application
• Payment: cashless
Functions for reporting Reporting menu to print budget realization report referring format report in BOS Guidelines including sub menu reporting for asset,
Menu for budget realization of BOS/local BOS
9 reports generated through SIAP BOS/BOP and SIMKD (local government financial information system) SIMKD is connected DEO and DFO financial report.
43
Question Bogor District Bogor City DKI Jakarta Province
number of students, quarterly and six‐monthly report
Functions for asset management There is menu to record asset purchasing
Not link yet Link to e‐asset (application for asset management
Functions for SSE (School Self Evaluation) Application
Not link Not link There is SSE template in e‐RKAS application. Cooperation with LPMP Prov DKI (Local SSE team) SSE assessment is completed but it is not show real conditions of school, therefore the SSE still cannot be used for school planning and budgeting
Other functions N/A N/A N/A
Are all functions in one application? Yes, in SIRKAS SIREAL Yes, in SIRKAS In some applications, e‐RKAS, SIAP BOS/BOP, SIMKD, e‐asset, SSE template
C Input mechanism
1 Register First login using NPSN (school primary number) and then change password. After register completed, school should update its school profiles.
Using username and password
Register number is given by Communication and Information office
2 Input mechanism for planning and budgeting
a. Program, activity and sub activity should be (1) typed manually or (2) available in database
Available in database, school just chooses from database
Available in database, school just chooses from database
Available in database, school just chooses from database
b. Goods and services item, and its price: (1) typed manually, or (2) available in database
Available in database, school just chooses from database
Available in database, school just chooses from database. Link also to online shops (SIPLAH)
Available in database, school just chooses from database
3 Input mechanism for budget realization and financial administration:
a. Generating Receipt/Nota No Nota Nota is available in SIAP BOS/ BOP
b. Bookkeeping (ledger and sub ledger)
Sub menu of bookkeeping: bank and cash.
Sub menu of bookkeeping: bank and cash.
Bookkeeping is available in SIAP BOS/ BOP
c. others
4 Mechanism for generating report Report is produced from SIAP BOS/BOP dan SIMKD
44
Question Bogor District Bogor City DKI Jakarta Province
a. Automatic generated yes yes Yes. Generated automatically through SIAP BOS/BOP and SIMKD
b. Can be downloaded to excel or pdf
to excel to excel to excel
c. Report submission (from, to, type of report, how, and duration)
submission automatically from school to DEO, real time
submission automatically from school to DEO, real time
submission automatically from school to DEO and DFO, real time
5 Link mechanism to asset management and inventory
The application has asset management menu
not link Inject from SIAP BOS/BOP to e‐asset
6 Link SSE application not link not link not link
D Admin structure
1 Level of admin 1) School: principal and operator 2) District Education office
1) School: principal and operator
2) District Education office
1) School: principal and 8 staffs 2) District Education office 3) Communication and Information office
2 Role 1) School: preparing RKAS, spending, financial administration, and reporting
2) District Education office: maintenance and monitoring
1) School: preparing RKAS, spending, financial administration, and reporting
2) District Education office: maintenance and monitoring
1) School: preparing RKAS, spending, financial administration, and reporting
2) District Education office: open and close of input schedules
3) Communication and Information office: maintenance and update the application, providing register number of schools
E District Education office
1 Authority of District Education office maintenance dan monitor RKAS input
maintenance dan monitor RKAS input
open and close schedule for input, reset password, monitoring of planning, budgeting, financial administration, and reporting process
2 Type of report generated by system
a. Per school Yes Yes Per school and per chart of account
b. Recap/consolidated to all schools Yes Yes Yes, generated by DFO
Challenges
45
Characteristics of e‐RKAS applications for districts categorized as ‘Medium’
No Question Bekasi City Bekasi District Tangerang City Tangerang District
A Application Profile
1 Name of application • SIMBOS for BOS National
• SIMDA for Local BOS
SIMDA (BOS dan BOSDA) • 2016‐2017: SIPKD (Sistem Informasi Pengelolaan Keuangan Daerah / Local Government Financial Information System)
• Th 2019‐ SPEKTRA (Sistem Pengelolaan, Penatausahaan dan Pelaporan Keuangan Kota Tangerang / Financial Management, administration, and reporting System of Tangerang City)
SIMRAL (Sistem Informasi Managemen Perencanaan, Penganggaran dan Pelaporan).
2 Type of connection Online Offline (2016‐2019) (Th 2020 will be online)
School: Offline District Education Office: Online to District Finance Office
School: Offline District Education Office: Online to District Finance Office
3 Since 2017 2016 2016 2017
4 Developer District Education Office and District Finance Office
District Education Office and District Finance Office
District Finance Office District Finance Office
5 Integrated to other applications such as SIMDA (District Financial Management System), DAPODIK, asset and inventory system
Not integrated but referring to: 1. SIM PMP (score of 8
NES) 2. SIPLAH (for online
purchasing)
Not integrated Not integrated Not integrated
6 Coverage of user: public, private, primary, and junior secondary
Public: primary and junior secondary
Public: primary and junior secondary
Public: primary and junior secondary
Public: primary and junior secondary
46
No Question Bekasi City Bekasi District Tangerang City Tangerang District
B Functions in applications
Functions for planning: SIMBOS = planning and budgeting for BOS national SIMDA = planning and budgeting for local BOS
Only planning and budgeting menu for BOS national and local BOS
Planning and budgeting menu for BOS national
Planning and budgeting menu for BOS national
Functions for budget realization and financial administration
• Budget realization and financial administration (bookkeeping) is already included in this application (ledger and sub ledger, bank, cash, taxes, settlement report)
• Payment: cashless, except for electricity, internet, and cleaning service
Payment: combination between cash and bank
Payment: combination between cash and bank
Payment: combination between cash and bank
Functions for reporting Budget realization report and Investment Expenditure Report
N/A, generated manually N/A, generated manually N/A, generated manually
Functions for asset management N/A N/A N/A N/A
Functions for SSE (School Self Evolution) Application
Not link Not link Not link Not link
Other functions N/A N/A N/A N/A
Are all functions in one application?
in 3 applications (SIMBOS, SIMDA dan SIMPMP)
Only one application for planning and budgeting
Only one application for planning and budgeting
Only one application for planning and budgeting
C Input mechanism
1 Register username and password Using NPSN Password Password from DEO
2 Input mechanism for planning and budgeting
a. Program, activity and sub activity should be (1) typed manually or (2) available in database
Available in database, school just chooses from database.
Available in database, school just chooses from database
Available in database, school just chooses from database
typed manually
47
No Question Bekasi City Bekasi District Tangerang City Tangerang District
b. Goods and services item, and its price: (1) typed manually, or (2) available in database
Price is typed manually Price is typed manually Price is typed manually Price is typed manually
3 Input mechanism for budget realization and financial administration:
Masih manual menggunakan excel
a. Generating Receipt/Nota No receipt
b. Bookkeeping (ledger and sub ledger)
Bookkeeping is available in the system
is still generating manually Bookkeeping is available in the system
is still generating manually using excel
c. others
4 Mechanism for generating report
a. Automatic generated Yes Generated manually
b. Can be downloaded to excel or pdf
Print and should be stamped by Head of DEO and principal and filed by school.
Print and should be stamped by Head of DEO and principal and filed by school.
Print and should be stamped by Head of DEO and principal and filed by school.
c. Report submission (from, to, type of report, how, and duration)
Sent to DEO to get stamp and sign, and DFO to get initial. Sent in soft file and print out.
Sent to DEO to get stamp and sign, and DFO to get initial. Sent in soft file and print out.
5 Link mechanism to asset management and inventory
N/A N/A N/A N/A
6 Link SSE application not link not link not link not link
D Admin structure
1 Level of admin 1) School: principal and operator
2) School supervisor 3) District Education office
District Education office School: BOS treasurer District Finance office: Team BOS
DEO and DFO: Team BOS
2 Role 1) School: preparing RKAS, spending, financial administration, and reporting
N/A 1) School: preparing RKAS, spending, financial administration, and reporting
1) School: submit school planning
2) District Education office: Input data, Verification and Approval
48
No Question Bekasi City Bekasi District Tangerang City Tangerang District
2) School supervisor: verification and monitoring
3) District Education office: Verification, Monitoring and Controlling.
2) District Finance office: verification and approval
3) District Education: verification and approval
3) District Finance office: verification and approval
E District Education office
1 Authority of District Education office
Maintenance Monitoring Verification Controlling
Input Monitoring Verification Controlling
Planning and reporting in excel
Planning and reporting in excel
2 Type of report generated by system
Individual school report and recapitulation of all school reports
individual school plan and recapitulation of all school plans
a. Per school Yes Yes, but only for planning and budgeting document
Yes Yes
b. Recap/consolidated to all schools
Yes Yes, but only for planning and budgeting document
Yes Yes
Challenges School should input 3 times into BOSNAS, BOSDA and PMP MoHA will release new application for BOS. Current applications will not be used anymore and district should do capacity building for new application
Not online system DEO will be overload during planning and budgeting process
DEO will be overload during planning and budgeting process
DEO and school will be overload during planning and budgeting process. Some school must stay overnight to complete RKAS.
49
Characteristics of e‐RKAS applications for districts categorized as ‘Non e‐RKAS’
No Question Depok City
A Application Profile
1 Name of application 2019: Manual (Excel)
2020: RKAS application from MoEC ARKAS for school MARKAS for DEO
2 Type of connection Excel = offline Using upload, download, and synchronization mechanism
3 Since Excel 2019 and earlier
4 Developer NA
5 Integrated to other applications such as SIMDA (District Financial Management System), DAPODIK, asset and inventory system
Not integrated
6 Coverage of user: public, private, primary and junior secondary Public: primary and junior secondary
B Functions in applications
Functions for planning: RKAS format in excel
Functions for budget realization and financial administration Payment: combination between cash and bank transfer
Functions for reporting N/A, generated manually
Functions for asset management N/A
Functions for SSE (School Self Evaluation) Application Not link
Other functions N/A
Are all functions in one application? No application
C Input mechanism
1 Register Excel = no register
2 Input mechanism for planning and budgeting
a. Program, activity and sub activity should be (1) typed manually or (2) available in database
typed manually
b. Goods and services item, and its price: (1) typed manually, or (2) available in database
Price is typed manually
3 Input mechanism for budget realization and financial administration:
a. Generating Receipt/Nota
50
No Question Depok City
b. Bookkeeping (ledger and sub ledger) is still generating manually using excel
c. others
4 Mechanism for generating report
a. Automatic generated generated manually
b. Can be downloaded to excel or pdf Should be printed and stamped by Head of DEO and principal and filed by school
c. Report submission (from, to, type of report, how, and duration)
5 Link mechanism to asset management and inventory N/A
6 Link SSE application not link
D Admin structure
1 Level of admin No admin for Excel
2 Role School: input District Education office: receive planning document from school
E District Education office
1 Authority of District Education office Recap school planning to DEO planning
2 Type of report generated by system
a. Per school Yes
b. Recap/consolidated to all schools N/A
Challenges
51
Appendix 5: Average UN scores of the eight sample school districts by subject: 2013‐2018
52
Appendix 6: Portrait of BOS Spending in 8 Districts/Cities No Type of Activities DKI Kota Bekasi Kab Bekasi Kota
Tangerang Kab Tangerang Kota Bogor Kab Bogor Kota Depok Average
2018 2019 2018 2019 2018 2019 2018 2019 2018 2019 2018 2019 2018 2019 2018 2019 2018 2019
1 Library Development 14.6 15.3 18.9 18.2 18.9 13.5 11.1 10.4 13.1 15.1 13.5 14.2 17.6 16.6 11.5 8.9 14.9 14.0
2 Activity related to new student admissions
1.8 1.9 1.3 1.5 1.2 1.6 1.2 1.2 1.3 1.4 0.7 0.7 0.8 1.0 1.3 1.4 1.2 1.3
3 Learning and extracurricular activities
12.9 11.0 13.1 14.4 12.7 17.2 13.9 14.9 13.4 13.2 11.2 14.1 13.2 15.4 12.7 12.8 12.9 14.1
4 Learning evaluation activities
6.8 7.3 11.7 13.3 11.1 14.0 11.9 12.3 14.6 14.9 13.8 11.6 10.8 11.3 16.7 18.3 12.2 12.9
5 School management 9.3 11.5 17.8 17.9 18.9 19.4 17.2 16.8 15.0 14.7 12.2 12.0 15.7 17.5 13.6 16.8 15.0 15.8
6 Professional development for teacher and education personnel
1.8 1.7 3.0 1.7 3.8 3.9 2.4 2.8 3.4 3.4 3.2 2.4 4.6 5.2 4.6 4.6 3.4 3.2
7 subscriptions to power and services
4.3 4.4 7.8 9.2 4.3 5.1 7.4 9.3 5.8 5.7 5.2 5.7 4.1 4.2 7.6 8.7 5.8 6.5
8 Maintenance of facilities and infrastructure
16.7 16.1 9.7 13.8 5.7 6.8 10.0 12.1 10.0 10.5 10.4 15.9 9.0 9.4 7.5 9.1 9.9 11.7
9 Payment of honoraria 15.5 14.8 12.6 5.7 19.5 14.4 18.4 15.2 16.2 16.8 22.4 18.5 20.4 16.9 16.0 13.3 17.6 14.5
10 Purchase and maintenance of learning tools
12.4 16.0 2.8 4.3 3.0 4.0 4.2 5.1 3.5 4.3 3.8 4.9 2.2 2.6 3.9 6.1 4.5 5.9
11 others 3.8 1.3 0.8 2.4 3.9 3.7 1.4 4.5 2.7 0.0
Total 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100