final evaluation of the unicef supported component of the ... · urgent and complex issue of...
TRANSCRIPT
Evaluation of the UNICEF supported Component of the Project “Support for durable Solutions of the Revised
Strategy for Implementation of Annex VII of the Dayton Peace Agreement”
Bosnia and Herzegovina
REPORT
The content of this evaluation report is the sole responsibility of the Evaluator and should in no way be taken to reflect the views of UNICEF or the European Delegation to Bosnia and Herzegovina.
Zehra Kacapor-Dzihic
February 2017
2
Name of the project component under evaluation:
UNICEF supported component of the Project “Support for durable Solutions of the Revised Strategy for Implementation of Annex VII of the Dayton Peace Agreement”
Implementation period for the project component under evaluation:
15 June 2014 – 31 December 2016
Timeframe for the evaluation: November 2016 – January 2017
Country of project: Bosnia and Herzegovina
Evaluator: Zehra Kacapor-Dzihic
Name of the organisation commissioning the evaluation:
UNICEF Bosnia and Herzegovina
3
Table of Contents
Abbreviations .................................................................................................................................... 5
Executive Summary ....................................................................................................................... 6
1. Context and UNICEF Project Component Background ....................................... 10 1.1 UNICEF Project Component Context ........................................................................................... 10 1.2 UNICEF Project Component Description ................................................................................... 11 1.3 UNICEF Project Component - Theory of Change (ToC) ....................................................... 15
2. Evaluation Purpose, Objectives and Scope ............................................................. 17
3. Evaluation Methodology ................................................................................................... 19 3.1 Sampling Methodology ...................................................................................................................... 21 3.2 Safeguarding Ethics ............................................................................................................................ 22 3.3 Evaluation Matrix .................................................................................................................................. 24 3.4 Limitations, Assumptions, Risks and Mitigation Strategies .................................. 24
4. Key Findings .......................................................................................................................... 25 4.1 Relevance: To what extent is the project responding to the needs of stakeholders
and beneficiaries? ....................................................................................................................................... 25 4.2 Effectiveness: To what extent does the project meet the outcomes as defined by
the DoA? .......................................................................................................................................................... 28 4.3 Efficiency: To what extent did the management of the project ensure timelines and the efficient utilisation of resources? ................................................................................................. 38 4.4 Impact: To what extent has the project improved the social protection and inclusion systems in the target municipalities? ............................................................................ 41 4.5 Sustainability: To what extent are the achieved project outcomes sustainable? .... 42
5. Lessons Learned ................................................................................................................. 45
6. Conclusions ........................................................................................................................... 46 7. Recommendations ................................................................................................................................. 48
Annex 1. TERMS OF REFERENCE ........................................................................................ 51
Annex 2. Evaluation Matrix ...................................................................................................... 57
Annex 3. Logframe of the Overall Action ........................................................................... 67
Annex 4. Interview guides ........................................................................................................ 83
Annex 5. BENEFICIARY SATISFACTION SURVEY ......................................................... 86 Services provided................................................................................................................................................ 89
Annex 6. List of interviewed persons .................................................................................. 90
Annex 7. Survey data .................................................................................................................. 92
4
Table of Graphs and Tables
Graph 1: Intervention Logic of the UNICEF Project Component ............................... 13 Graph 2: UNICEF Project Component - Theory of Change Diagram ....................... 17 Graph 3. Did these or similar services exist in your community prior to the
establishment of the service you are currently using? ....................................... 35 Graph 4. How would you rate the services you have been using? ........................... 36 Graph 5. Do you believe that services will continue to be available for as long as you
and your child need them? ................................................................................ 43 Table 1: UNICEF Project Component Objectives and Indicators.............................. 13 Table 2. Evaluation Sample ...................................................................................... 22 Table 3. Limitations and Mitigation Approaches ....................................................... 24 Table 4. Achievement of Results per Indicator ......................................................... 28 Table 5. What services did you use? ........................................................................ 33 Table 6. How did your child and you become clients of service?.............................. 34 Table 7. What motivated you to become a client of the service for your child? ........ 34
5
Abbreviations
BD Brčko District
BiH Bosnia and Herzegovina
CSO Civil Society Organisation
CSW Centre for Social Work
CwD Children with Developmental Delays and Disabilities
DAC Development Assistance Committee (of OECD)
DoA Description of Action
ECD Early Childhood Development
ECI Early Childhood Detection and Intervention
EDUS NGO ‘Edukacija za sve’ (Education for all)
EQ Evaluation Question
EU European Union
FBiH Federation of Bosnia and Herzegovina
HRBA Human Rights-Based Approach
KAP Knowledge, Attitudes, Practices
M&E Monitoring and Evaluation
OECD Organization for Economic Cooperation and Development
OMJM One-minute junior movies
RS Republika Srpska
SPI Social Protection and Inclusion
ToR Terms of Reference
UN United Nations
UNEG United Nations Evaluation Group
UNHCR United Nations High Commissioner for Refugees
UNICEF United Nations Children’s Fund
WG Working Group
6
Executive Summary
This document represents the Final Evaluation Report of the UNICEF supported component of the project ‘Support for durable Solutions of the Revised Strategy for implementation of Annex VII of the Dayton Peace Agreement’ (hereinafter called ‘UNICEF Project Component’).
UNICEF Project Component Description
The project Support for durable Solutions of the Revised Strategy for Implementation of Annex VII of the Dayton Peace Agreement is aimed at addressing the issues of IDPs and returnees in Bosnia and Herzegovina and to ensure durable solutions for the most vulnerable families of IDPs, minority returnees and female victims of war. The EU and the UNHCR, which led the project, developed the project jointly in cooperation with a range of project partners. The UNICEF Project Component supported targeted local communities to develop and implement Local Action Plans for social protection and inclusion (SPI) and promote and strengthen SPI governance and inter-sector cooperation. It strengthened the capacities of professionals from the social, education and health sectors, the provision of SPI services for vulnerable children and families, particularly those coming from IDP and returnee populations, and raised awareness and increased positive attitudes and behavioural change regarding returnees and IDPs in the targeted localities.
The project was intended to achieve three results: 1) strengthened social protection and inclusion systems in the targeted localities, with particular focus on vulnerable returnees and IDPs; 2) enhanced capacities to deliver inclusive services to the most vulnerable returnees and IDPs identified in the targeted localities and 3) increased awareness and positive attitudes and behavioural change initiated regarding returnees and IDPs in the targeted localities. Project activities included capacity building and training initiatives, advocacy and technical assistance for the development of local policies, coordination and consultation, technical and financial assistance to strengthen services, campaigns and the production of ‘junior films’, etc.
The target groups for the UNICEF component were: 1) municipal authorities, service providers and CSOs in the prioritised municipalities and 2) vulnerable IDPs and minority returnee families residing in the municipalities and communities of priority concern who were unable to secure a sustainable solution and were therefore, together with their children, most in need of support.
The timeframe for the overall project was 36 months, with a duration of 30 months (15 June 2014 - 31 December 2016) for the UNICEF Project Component. Financial support for the overall 3-year project under the leadership of UNHCR amounted to EUR 8,107,500, while the UNICEF Project Component amounted to EUR 887,900. The UNICEF Project Component was based on lessons learned and applied the model for support to social protection and inclusion of children that was developed through the UNICEF SPIS projects.
Project Context
The end of the war in December 1995 signified the beginning of the process to resolve the urgent and complex issue of displacement across Bosnia and Herzegovina that affected more than one million refugees and internally displaced persons, including almost half a million so-called ‘minority’ returns. An integral part of the Dayton Peace Agreement, signed in December 1995, was Annex VII, which confirmed the commitment of the signatory parties to respect the rights of refugees and displaced persons and in particular the right of return as well as restitution of property and/or compensation. Substantial achievements were made in implementation of Annex VII during the first ten years after Dayton, particularly in terms of return and the restoration of property; however, progress then slowed until the adoption of the Revised Strategy for implementation of Annex VII in 2010. Today, there are approximately
7
98,324 IDPs 1 and over 470,000 minority returnees in Bosnia and Herzegovina (BiH). A considerable number of these persons continue to live in undignified conditions, with 8,600 especially vulnerable persons remaining in collective accommodation. Landmines continue to prohibit the reconstruction and/or use of property with 45,000 housing units of returnees awaiting reconstruction.
Multiple human rights violations against displaced persons and refugees continue to have a negative impact on their social situation and social networks. This has a correspondingly negative affect on their ability to fulfil their rights and causes them to remain dependent on others for basic needs such as shelter, food and water. Long-term displacement increases the risk that traditional family and social structures will break down, leaving IDPs dependent on outside aid and vulnerable to economic and other forms of exploitation. Similarly, returnees in BiH face significant challenges when attempting to integrate into their pre-conflict communities, mainly due to the fact that the very reason for their displacement was their belonging to a minority group. Division along ethnic or cultural lines affects return and restricts the ability of returnees to access basic services, including health, social protection and education.
Post-war transition creates general economic and social hardship for all segments of society in BiH, while continued challenges to the stability and prosperity of the country posed by political instability create an environment where social vulnerability is widespread and where IDPs and returnees simply ‘join’ this broader group of the vulnerable population.
Evaluation Methodology
The evaluation was carried out in three phases. A review of the project documentation and relevant literature as well as reference documents was conducted during the Inception Phase. This resulted in the Inception Report for the evaluation. Within this phase, the evaluation framework, primary data collection methods and evaluation tools were developed. The Field Phase had a limited timeframe and was devoted to the collection of data from key stakeholders at the municipal level. This was done through semi-structured interviews, focused discussion groups and site visits to a sample of seven municipalities (70% of the total number of municipalities targeted by the project). A beneficiary survey was conducted to capture the opinions of parents and children using ECD services in the local communities. During the Synthesis Phase, OECD-DAC criteria of relevance, efficiency, effectiveness, impact and sustainability were applied in order to assess the level of achievement of the planned project objectives; these were then used to draw informed conclusions and to make recommendations.
Summary of the Main Findings and Conclusions
The project is relevant and responds well to the recognised need for Bosnia and Herzegovina to improve the social protection systems for children at the municipal level. This is to be achieved by strengthening the social protection and inclusion strategic framework (through development of evidence-based Action Plans for social protection and inclusion) and service delivery for vulnerable children and their families, in particular for children coming from IDP and returnee groups. The project addresses important gaps and challenges within the current social protection and inclusion mechanisms at the local level. It provides an opportunity to devise systems to address the multidimensional needs of children by connecting social protection, health, education, law enforcement, civil society and other sectors in the targeted communities and to strengthen the response to the needs of children. The project was designed on the basis of an extensive evidence base and lessons learned through implementation of previous UNICEF SPIS projects and conducted assessments and studies
1 Official data from the Report on Implementation of the Revised Strategy for Implementation of Annex VII of Dayton peace Agreement, prepared by the Ministry of Human Rights and Refugees in October 2015. The figure needs to be corroborated.
8
focused on children.
The UNICEF Project Component corresponds to the strategic and legislative framework for social protection and inclusion in the country. The project is highly relevant to the needs of professionals and municipalities to implement quality and inclusive social services for vulnerable children.
As far as human rights and crosscutting issues are concerned, the UNICEF Project Component is highly relevant to BiH’s international commitments deriving from its ratification of the CRC and its status as an EU potential candidate country. It made a major contribution to the promotion and realisation of child rights by opening access to social services and early learning development for children left out of the system. The project has made a positive contribution to strengthening the capacities of a wide range of municipal level duty-bearers to protect and fulfil child rights. It has managed to ensure equity focus by orienting investment toward vulnerable children.
The project has achieved all of its envisaged outputs and contributed to its planned results, as confirmed through a review of the indicator set for the UNICEF component: all were achieved. The project achieved important results in the area of strengthening SPI systems at the local level through enhanced capacities of service delivery and the provision of quality inclusive services to children (particularly children coming from the IDP and returnee populations, most of which are located outside of urban centres in the targeted municipalities). Another strong achievement is the fact that most Action Plans for SPI developed by the SPI commissions are evidence-based and budgeted either through special budget lines for the action plan or through mayors’ decisions on funding activities and measures stemming from these documents.
UNICEF made successful efforts to ensure the use of available project resources strategically and efficiently. The project encountered significant delays at the onset of implementation, but UNICEF succeeded in overcoming them through intensive work with partners. Management efforts by the UNICEF project team were appropriate and contributed to effective and efficient implementation of the planned initiatives. No particular deviations from the projected budget were found, while the ratio of operational versus programme cost was balanced (16.3% for operational costs). The project built on lessons learned and the solid foundation of the SPIS Project, while the level of effort and related budgets have been utilised in such a way as to adequately invest in strengthening the social protection and inclusion system. These investments, particularly in relation to services, show direct benefit for children and families and should bring about tremendous returns for individuals, communities and society in general in the future .
Project efforts have contributed to moving the existing processes of change in the desired direction, whereby the action plans, mechanisms and services will have a positive impact on the lives of the targeted groups of children. The project supports the reform of the social protection and inclusion systems at the local level and in particular the promotion of a multi-sectorial approach to SPI through the establishment of services, local governance mechanisms, protocols on cooperation and referral mechanisms in the target municipalities. These mechanisms, when and if implemented, will have a positive effect on the lives of children. Yet it was not possible through this evaluation to establish the impact upon the lives of children coming from the IDP and returnee populations.
The effect and outcome of the UNICEF interventions are relatively sustainable. Action Plans
in most of the target communities are accompanied by budget allocations (except for Brčko,
Prijedor and Gradiska), which is an important factor of sustainability. However, the fact is that financial sustainability in the longer-term remains in question and therefore the financial constraints and scarce resources of service providers at the local level infers further dependence on external funding.
9
Recommendations
Operational Recommendations – relevant for the improvement of UNICEF’s future interventions based on the SPIS model.2
O1 (for UNICEF and partners) - Organise training based on the findings of a Training Needs Assessments in order to ensure that the training serves the purpose of education and other professionals acquiring practical skills.
O2 (for UNICEF) - Mainstream gender into programming of UNICEF support for SPI.
O3 (for UNICEF) – Further strengthen monitoring and reporting practices at the programme and impact evaluation levels.
Strategic and Programming Recommendations (SP) – This is aimed at informing policymaking, prioritising and programming of further support.
SP1 (for UNICEF and partners) – Scale-up the SPI model countrywide to ensure that at least 50% of municipalities apply the model to improve SPI, particularly for children.
SP2 (for UNICEF and government/donor partners) - Ensure continued support for IECD
and other services established/supported though the project in order to strengthen their sustainability mechanisms.
SP3 (for UNICEF and partners) - Advocate for the development of a framework for securing
financial sustainability in the longer-term to support and empower services developed through the project, particularly those targeting children from IDP/returnee families.
2 UNICEF’s model for SPI focus on supporting municipalities in establishing municipal commissions for SPI and the development and implementation of action plans for social protection and inclusion (SPI), based on detailed vulnerability assessments. The vulnerability assessments represent the situation of vulnerable groups in each municipality with a focus on returnees and IDPs and in the areas of education, health, social welfare, protection and justice. Action plans provide the framework for implementation of measures for SPI, including improved access to services, even in remote areas, that are often populated by returnee and IDP groups.
10
1. Context and UNICEF Project Component Background
The European Union and UNHCR developed the project ‘Support for durable Solutions of the Revised Strategy for Implementation of Annex VII of the Dayton Peace Agreement’ (hereinafter: the project), which is aimed at addressing the issues of IDPs and returnees in Bosnia and Herzegovina. As per the agreement between the UNHCR and UNICEF, this evaluation focuses solely on the UNICEF Project Component. UNICEF supported targeted local communities to develop and implement Local Action Plans for social protection and inclusion (SPI). The aim was to promote and strengthen SPI governance and inter-sector cooperation, provide SPI services to vulnerable children and families, and strengthen the capacities of professionals from the social, education and health sectors.
1.1 UNICEF Project Component Context
The Bosnian conflict induced massive internal and external displacement, resulting in more than 2.5 million people being forced from their homes. The international community brokered the Dayton Peace Agreement in order to end the war. An integral part of the Dayton Peace Agreement is Annex VII, which provides a set of measures for the resolution of the protracted issue of displacement resulting from the 1992-95 conflict. Annex VII includes the right of IDPs to return specifically to their “homes of origin”. As such, it confirms the commitment of the signatory parties to respect the rights of refugees and displaced persons and in particular the right of return as well as restitution of property and/or compensation. Substantial achievements in implementation of Annex VII were achieved during the first ten years after Dayton, particularly in terms of the return of persons and the restoration of their property. Progress then slowed with most of the remaining population of concern belonging to the group of vulnerable people with no property to be reconstructed or in a situation where they were either reluctant or too frail to return and unable to achieve solutions without special help.3 Recognition of this led to the Revised Strategy for Implementation of Annex VII, adopted by both houses of the Parliament of BiH in June 2010. It was in this strategy that for the first time national political stakeholders agreed that extremely vulnerable persons unable to return should be allowed to find a solution in their place of displacement. The Revised Strategy recognises the unmet needs of more than 100,000 internally displaced persons, refugees and other conflict-affected persons of concern, each one so far without a solution.
Today, there are approximately 98,324 IDPs and over 470,0004 minority returnees in Bosnia and Herzegovina (BiH). A considerable number of these persons continue to live in undignified conditions.5 This is particularly the case for the 8,600 particularly vulnerable persons who remain in collective accommodation. Many people are also unable to return because their pre-war property was destroyed: 45,000 housing units of returnees await reconstruction or their property remains inaccessible for use because of landmines that have yet to be cleared from their pre-war villages.
In general, after the end of the war IDPs and returnees in BiH initially faced numerous structural challenges that were different from the general population. Displacement works against IDPs by creating from the very beginning both physical and administrative obstacles to the realisation of their rights, something that does not apply to non-displaced citizens. The multiple violations of their human rights and the losses incurred by displaced persons had a negative impact on their social situation and social networks. This in turn had a negative effect on their ability to fulfil their social and economic rights, leading them to become dependent on others
3 Report on Implementation of the Revised Strategy for Implementation of Annex VII of the Dayton Peace Agreement, prepared by the Ministry of Human Rights and Refugees of BiH in October 2015. 4 Official data from the Report on Implementation of the Revised Strategy for Implementation of Annex VII of the Dayton Peace Agreement, prepared by the Ministry of Human Rights and Refugees of BiH in October 2015. The figure requires corroboration. 5 Ibid.
11
for their basic needs such as shelter, food and water. Moreover, the longer displacement continued the greater was the risk that traditional family and social structures would break down and leave IDPs both dependent on outside aid and vulnerable to economic and other forms of exploitation. Such dependency in turn reduced the chances of finding durable solutions for these people and in securing their sustainable reintegration into society, once changes to the political and security conditions enabled such solutions to take place. Similarly, returnees in BiH faced significant challenges when attempting to integrate into their pre-conflict communities. This was mainly due to the fact that the very reason for their displacement was belonging to a minority group and that return is affected by divisions along ethnic and cultural lines and that this affects their ability to access basic services, including health, social protection and education.
A peculiarity in post-Dayton BiH is the general economic and social hardship that effects all parts of the population combined with the continued challenges to the stability and prosperity of the country posed by political conflicts that create an environment where social vulnerability is widespread. Poverty and social vulnerability are high and multidimensional poverty (UNDP 2015) affects 33 per cent of the population, while youth unemployment is estimated to be between 40 and 60 per cent. UNICEF estimates that approximately 170,000 children in BiH are poor; the majority come from minority groups, such as Roma or internally displaced persons.6
1.2 UNICEF Project Component Description
The EU Delegation to BiH and UNHCR BiH, in consultation with project partners, together developed a project (Action) that addresses the issues of IDPs and returnees in Bosnia and Herzegovina. As per the Contribution Agreement signed by the parties, UNHCR assumed overall accountability and responsibility for communication with the donor and implementation and coordination of the Action.
The Terms of Reference (ToR) (see Annex 1), the Project Document (PD) and the Theory of Change (ToC) (see section 1.3 of this report) provide a comprehensive picture of the main features of the UNICEF Project Component to be evaluated, including its logic and the foreseen changes to be brought about through it. Graph 1 presents a visual overview of the UNICEF Project Component Theory of Change, as developed by the evaluator.
Objectives of the overall Project led by UNHCR
The main goal of the project is to ensure durable solutions for the most vulnerable families of IDPs, minority returnees and female victims of war. This is to be achieved through teamwork and cooperation among local authorities, local communities, the civil society sector and the project partners.
As per the overall project log frame, the following is the results framework of the overall Action (See Annex 3).
Overall objective: To enable national actors to address the remaining obstacles to solutions for displaced persons and returnees in accordance with the goals of the Revised Annex VII Implementation Strategy.
Specific Objective 1: Provide durable housing solutions for vulnerable returnee and IDP families.
6 https://www.unicef.org/bih/children_17672.html
12
Specific Objective 2: Provide sustainable economic opportunities for returnee and IDP families.
Specific Objective 3: Enable vulnerable returnees and IDPs to access their rights and inclusive social services.
UNICEF Project Component
UNICEF contributed directly to Specific Objective 3: Enable vulnerable returnees and IDPs to access their rights and inclusive social services. Contributing to this Objective, UNICEF aimed to strengthen the capacity and engagement of municipal level authorities in order to ensure fuller social inclusion in development plans and the provision of social services for vulnerable groups, including IDPs and minority returnees of concern to the Annex VII Revised Strategy.
With regard to the activities set out in the Project Description, the UNICEF Project Component was based on lessons learned and applied the model to support the social protection and inclusion of children that was developed through the UNICEF SPIS projects. UNICEF’s model for SPI offers a comprehensive and systematic framework for addressing the needs of most vulnerable groups. The model focuses on supporting municipalities in the development of Action Plans for social protection and inclusion (SPI), based on detailed vulnerability assessments. The vulnerability assessments presented the situation of vulnerable groups in each municipality, with the focus on returnees and IDPs, in the areas of education, health, social welfare, protection and justice. The Actions Plans provided the framework for implementation of measures for SPI, including improved access to services, even in remote areas often populated by returnee and IDPs groups. Successive evaluations of the UNICEF model have concluded that it offers a viable option for improving inclusive social services and overall strengthening of the social protection and inclusion system both vertically (across governments) and horizontally (across social service providers).7
As per UNICEF’s documentation, the following chart shows the intervention logic for the UNICEF Project Component.
7 UNICEF (2015), Final Evaluation of the SPIS Project Sarajevo.
13
Graph 1 Intervention Logic for the UNICEF Project Component
UNICEF ensures that the structures of the social protection and inclusion (SPI) commissions and other components implemented by UNICEF on the basis of its prior experience and expertise in BiH are formed and guided in a manner that ensures their full potential to contribute to the implementation of the Annex VII Revised Strategy and to the solutions to the remaining problems of wartime displacement. UNHCR coordinates the activities with those of other project partners to ensure that the different components of the Action form a coherent strategy and work effectively together to achieve this same objective.
Indicators relating to the UNICEF Project Component objectives/results are presented below in Table 1.
Table 1 UNICEF Project Component Objectives and Indicators
Objectives (results in the overall Action framework)
Indicators
R5: Social protection and inclusion systems strengthened in the targeted localities, with particular focus on vulnerable returnees and IDPs.
I5.1: Social Protection and Inclusion Commissions establishment in at least 10 municipalities.
I5.2: Evidence-based Action Plans on Social Protection and Inclusion developed in at least 10 municipalities.
R6: Enhanced capacities to deliver inclusive services to returnees and IDPs identified as most vulnerable in the targeted localities.
I6.1: At least 300 vulnerable children and caregivers provided with access to inclusive social services (education, health, social protection and child protection).
R8: Awareness raised and positive attitudes and behavioural change initiated in relation to returnees and IDPs in the targeted localities.
8.1: A 20% increase in positive attitudes toward returnees and IDPs in the targeted localities.
A5.1.EstablishanddevelopthecapacityofSocialProtectionandInclusionCommissionsinatleast10municipalities
A5.2.ConductanassessmenttoidentifyvulnerablereturneesandIDPsandtheirneedsinlinewiththeobstacles
tosolutionsasdefinedintheRevisedStrategyfortheImplementationofAnnexVIIinatleast10municipalities
A5.3.Developevidence-basedActionPlansonSocialProtectionandInclusioninatleast10municipalities
R5:Strengthenedsocialprotectionandinclusionsystemsintargeted
localities,withaparticularfocusonvulnerablereturneesandIDPs
A6.1.Developthecapacityofserviceprovidersfromtheeducation,healthandsocialwelfaresectorstoprovideinclusiveservicestovulnerablechildrenand
theirfamilies
A6.2.Supportserviceprovidersinatleast10municipalitiestoensureaccesstoinclusivesocialservicestovulnerablechildrenandfamilies(in
particularreturneeandIDPchildren)
R6:EnhancedcapacitiestodeliverinclusiveservicestoidentifiedmostvulnerablereturneesandIDPsintargetedlocalities
A8.1.ConductbaselineandevaluationsurveyonKnowledgeAttitudeandPracticestowardsIDPsandreturnees,amongrepresentativesampleofgeneral
population
A8.2.Changeperceptionsandpromotechildparticipationthroughschool-basedactivities
A8.3.RaiseawarenessonsocialinclusionofIDPsandreturneesthroughroundtablediscussionsandsocial
media
R8:Awarenessraisedandpositiveattitudeand
behaviour changeinitiatedregardingreturneesandIDPsintargetedlocalities
ToenablevulnerablereturneesandIDPstoaccesstheirrightsand
inclusivesocialservices
Toenable
nationalactorstoaddresstheremaining
obstaclestosolutionsfor
displacedpersonsandreturneesin
accordancewiththeRevised
AnnexVIIImplementationStrategygoals.
Activities ResultsSpecificObjective–UNICEFComponent
OverallObjectiveoftheAction
14
Project Activities
In order to achieve its aims in implementing its component of the Annex VII Action ‘Support for durable Solutions of the Revised Strategy for Implementation of Annex VII of the Dayton Peace Agreement Project’, UNICEF implemented a rich set of activities. This included capacity building initiatives for local authorities and related public institutions in charge of social protection and inclusion (municipal administrations, health centers, centers for social work, police, schools and CSOs, etc.) as well as the development of Local Action Plans for social protection and inclusion. It provided new working tools, instruments, material and guides, and provided grants to pedagogical institutions to support capacity building of education professionals in selected schools in the target municipalities. It supported service providers through the procurement of furniture and didactic material for work with children, direct work with children and youth on school based activities, campaigns, advocacy, exchanges of experience, etc. Graph 1 above presents a comprehensive list of activities grouped around each output.
Within this project, UNICEF did not only support activities for children from IDP and returnee families but also contributed to creating a favourable environment required to enable all vulnerable children and their families to enjoy the benefits of social inclusion. This was the only way to establish sustainable long-term support for all vulnerable children and to ensure better social inclusion and cohesion.
Target Groups and Beneficiaries
The main project activities were carried out in 10 municipalities in BiH: Bijeljina, Bosanski Petrovac, Derventa, Foča, Gradiška, Maglaj, Mostar, Prijedor, Živinice and Brčko District.
During the first four months of 2014, UNHCR, together with a BiH civil society partner the ‘Union for Sustainable Return and Integration’ (Unija za odrzivi povratak i integraciju – UZOPI), completed an initial assessment of the continuing presence of problems in relation to the Annex VII Revised Strategy in 50 municipalities in BiH. The assessment covered 847 local communities within these municipalities consulting a range of sources including municipal and local community leaders. The assessment described and ranked the severity of Annex VII related problems according to each category (housing, property, infrastructure, electrification, employment, health, education, social welfare and security) encountered in each community. UNHCR selected the target locations for the Annex 7 project in consultation with relevant government representatives, based on the vulnerability assessment.
The Description of the Action (DoA) presented below relates to the target groups and beneficiaries under the UNICEF Project Component.
Target Groups of the Project
1) Municipal authorities, service providers and CSOs in the prioritised municipalities: the project will continue to build the capacity of the SPI commissions and other service providers and CSOs.
2) Vulnerable IDP and minority returnee families residing in the target municipalities and communities of priority concern who have so far been unable to achieve a sustainable solution for themselves and their children and are therefore considered most in need of support.
15
Implementation Partners
UNICEF’s main partners are shown below.
Municipal authorities in the ten targeted communities 8 actively participated in the coordination and supervision of the implementation of project supported interventions in the targeted municipalities. They contributed to data collection and the preparation of action plans and played a leading role in the development of protocols required in order to ensure a more holistic approach toward tackling social protection and inclusion for children.
Public service providers participated through professionals from health centres, centres for social work, police and municipal administrations in the training activities and the implementation and coordination of the action plans, as per the agreed protocols, and in the mobilisation of families and the wider community.
NGO(s) participated through the delivery of trainings, participation in municipal level social protection and inclusion activities, and in the design and testing of new services for children in local communities (EDUS, local NGO service providers and local associations of parents, returnees, IDPs).
The role of the UNICEF country office in the UNICEF Project Component was twofold: 1) to ensure technical and financial management of the intervention and 2) to bring in the technical expertise and policy advocacy leverage needed to attain the envisaged outcomes. The UNICEF team also conducted ongoing monitoring of the progress of the commissions on the development, adoption and implementation of the Local Action Plans. Based on its inter-sectorial partnerships with key stakeholders at the state and entity level, UNICEF facilitated the dialogue on how legislation, practices and training programmes could best meet the needs of vulnerable children and families as defined in the relevant Municipal Action Plans.
Financing, Duration and Implementation Status
The overall project had a timeframe of 36 months, while the UNICEF Project Component started on 15 June 2014 and ended on 31 December 2016. Financial support for the overall 3-year project, under the leadership of UNHCR, derived from EU IPA funds (EUR 7,000,000) and from co-financing provided by UNHCR (EUR 1,107,500). The UNICEF Project Component amounted to EUR 887,900.
1.3 UNICEF Project Component - Theory of Change (ToC)
The UNICEF Project Component had a clear results framework, while the Theory of Change developed for the Social Protection and Inclusion System (SPIS) guided its Theory of Change. The following is a reconstruction of the Theory of Change developed by the consultant and based on a review of the UNICEF Project Component documentation.
Children coming from returnee and IDP groups lack adequate social protection and face barriers to exercising their rights. Creating an environment that fosters the healthy development of these vulnerable families and children requires targeted action at the local level, in communities where these populations have settled. The UNICEF Project Component focused on three main Domains of Change: 1) strengthened governance through a strategic framework surrounding the social protection and inclusion of children, 2) development of local capacities to deliver quality services to these children and 3) public awareness to reduce stigma and increase opportunities and social mobility. These elements contribute individually
8 Bijeljina, Bosanski Petrovac, Derventa, Foča, Gradiška, Maglaj, Mostar, Prijedor, Živinice and Brčko District.
16
to improving the social protection systems for children in general and for children from returnee and IDPs groups in particular and at the same time reinforce each other.
To order to make progress in any of the Domains of Change, the UNICEF Project Component had to enact Change Processes through targeted activities. Advocacy, technical assistance, coordination and consultation at the local level strengthened governance, while local capacities have been improved through training and advocacy for such trainings and technical and financial assistance for municipal authorities, public institutions and service providers. Public awareness was built around school-based activities, roundtable discussions, social networks, media campaigns and institutional links.
The Domains of Change are the key elements essential for children to enjoy quality services and social protection and inclusion. Strong local capacity creates the foundation for quality service provision for children in support of their healthy development. Responsible governance in the form of Commissions for Social Protection and Inclusion at the local level together with coordination mechanisms ensure that children received the resources and levels of protection they require. Increased public knowledgeable of social inclusion, child rights and inclusive education reduces discrimination and improves social mobility for these children.
All three elements work together to improve the services children receive. Quality service delivery is more sustainable and has a greater impact when supported by strong strategic frameworks at the local level and a public that is knowledgeable about the importance of social inclusion and child rights. As governance is strengthened so local services supported by action plans, curriculum development, documented best practice/lessons learned and social innovation are able to flourish. Similarly, the community is better able to support quality services for children and allow them to grow when the public is more aware of social inclusion, child rights and inclusive education. Through the strengthening of governance, local capacities and public awareness, the UNICEF Project Component was dedicated to ensuring that children enjoy access to their rights, quality service delivery and improved social protection and inclusion.
17
Graph 2 UNICEF Project Component - Theory of Change Diagram
2. Evaluation Purpose, Objectives and Scope
In November 2016, UNICEF contracted an independent consultant, Ms Zehra Kacapor Dzihic, to conduct the final evaluation of the UNICEF Project Component of the project ‘Support for durable Solutions of the Revised Strategy for Implementation of Annex VII of the Dayton Peace Agreement’ (hereinafter: the project). The aim was to evaluate the final (end) results and achievements of the UNICEF supported component of the project.
The objectives of the evaluation are outline in more detail below.
1. Provide feedback to the BiH UNICEF country office, UNHCR and national counterparts on the soundness (defined as relevance, effectiveness, efficiency and sustainability) and impact of the project approach in strengthening social protection and inclusion for vulnerable families and children, IDPs and returnees in particular. The purpose of this was threefold:
a. record good practice along with its implementation modalities, with particular focus on government at the local level;
b. identify systemic gaps in the application of SPI approaches at the local level;
c. evaluate the project impact by following the Description of the Action and Logical Framework.
2. Extract lessons learned and make recommendations aimed at the further enhancement of the social protection and inclusion system in BiH at the local level, based on the experiences gained through implementation of the project.
The scope of the evaluation was defined in accordance with a number of elements:
Local
Capacity
Enhanced systems and capacities to deliver quality
inclusive services at the local level
Governance
Strengthened Social Protection and Inclusion strategic framework and
coordination mechanisms
at local level
Public
Awareness
Raised awareness and
promoted social inclusion, child participation, social inclusion of IDPs and
returnees
Children from vulnerable returnee
and IDP population enjoy access to
rights and inclusive social services.
Quality Service
Delivery
Establish Action plans and protocols through advocacy,
technical assistance,
coordination and
consultation
Increase local capacity through training, technical
and financial assistance, and
advocacy
Increase public awareness through school-based
activities , round table
discussions. social networks,
media, campaigns, and
institutional linkages
18
- project implementation period to be considered - 15 June 2014 – 31 December 2016; - geographic coverage of 10 targeted municipalities9; - level of change/intervention - local level (municipal) system change (e.g. Local Action
Plans for SPI adoption, the institutionalisation of local services, the adoption of protocols and changes to administrative procedures sparked by the project).
The Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD-DAC) evaluation criteria applied to the evaluation.
a) Relevance: The assessment of the relevance of the project was based on an analysis of the national and local context, the challenges of social protection and inclusion in local communities and the needs and priorities of various stakeholders and beneficiaries (e.g. social service provision, access and quality, capacity building, etc.).
b) Effectiveness: Using the project’s log frame, the evaluator analysed the extent to which the results obtained through implementation of the project activities had contributed to the attainment of the planned objectives. Using evidence collected during the first two phases of the evaluation, the report attempted to explain the factors that had contributed to or hampered the achievement of results. This was assessed in terms of the mobilisation and capacity building of institutions for the improvement of the social protection and inclusion of children, the expansion of physical and human resource capacities for the provision of social services in the target municipalities, development of models of diversified social services and increased access for the most vulnerable children at the local level. An analysis of the coordination and synergy between activities at different levels of governance was also conducted. The report also discussed the additional/indirect effects (positive or negative) of the project.
c) Efficiency: The report analysed how well UNICEF organised itself in delivering its work with regard to managerial and budget efficiency. The analysis of efficiency was based on an assessment of outputs/activities in relation to project inputs, costs and planned timelines. The report also incorporated a discussion on issues related to the extent to which co-ordination with other similar interventions had been ensured in order to encourage synergy and avoid overlaps.
d) Impact: When it came to the impact assessment, the key task of the evaluation was to examine to what extent the project had increased local capacities to ensure that more children from the target groups of the project would benefit from social protection and inclusion policies and services. It also assessed how this would contribute to the overall improvement of the social protection and inclusion systems for children in BiH. The evaluator also assessed if the project had accelerated the pace of change and if it gave direction to reforms in the area of social protection and inclusion and social services.
e) Sustainability: The evaluation report reviewed sustainability factors in terms of project design, processes, implementation and the national context. Sustainability was analysed from a number of perspectives: legal, institutional, capacity building and financial. The report highlights those factors that facilitated or decreased the sustainability prospects of the results of the project (legislation, synergy with similar initiatives, engagement and ownership by local stakeholders, etc.).
Given the fact that the UNICEF Project Component operated in ten municipalities and considering the available resources and the time available for this evaluation, field consultations, interviews, focus groups and discussion groups were carried out in seven selected municipalities, as per UNICEF’s suggestion. The sample is described below in section 2.1.3.
9 Bijeljina, Bosanski Petrovac, Derventa, Foča, Gradiška, Maglaj, Mostar, Prijedor, Živinice and Brčko District.
19
3. Evaluation Methodology
According to the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD-DAC) Evaluation Quality Standards, the selection and application of adequate evaluation methodology is crucial for the production of reliable data that allows for valid evaluative judgments useful for learning and making decisions.
In the absence of realistic comparators or counterfactuals and in view of the available evaluation time and resources, the evaluation applied a non-experimental design.
The overall approach to the evaluation was utilisation-focused, gender and human rights responsive and followed a mixed method approach.
Utilisation-focused: The evaluation methodology was based on the evaluator’s findings from the inception phase on the actual interests and needs of the intended users of the evaluation, those of UNICEF in particular. As the evaluation advanced, the evaluator involved potential users in order to review progress in the evaluation and to draft deliverables and support the development of the evaluation recommendations.
The evaluation applied ‘mixed methods’ in order to optimise the potential of the analysis and to reach a sound evaluation. This included qualitative and quantitative methods and instruments, such as survey, focus groups and interviews, a document review and meetings with project partners and beneficiaries. In order to provide useful information to UNICEF and its partners the methodology was designed to allow for triangulation of the findings and thus maximise the quality of the recommendations.
The evaluation used three main sources of data: people, documents and files (including databases) and observations made during the site visits. It included both primary and secondary data types.
All individual and group interviews followed agreed interview protocols tailored to the respective stakeholder group and aligned with the overall evaluation framework. The interview protocols are included in Annex 3. The initial interview questions were open-ended and thus allowed the consulted stakeholders to focus on and highlight specific issues that they considered to be most relevant in relation to the broader question. As the interview progressed, the evaluator provided a number of thematic prompts to elicit additional information on specific topics addressed in the evaluation framework. UNICEF provided guidance during the establishment of the agenda for the assessment.
The second data source consisted of relevant documents, files and other types of written information (e.g. websites) that could inform the desk review part of the evaluation. Based on recommendations by the interviewed stakeholders and/or UNICEF, additional documents, such as research studies, were incorporated.
The third data source was observations made during visits to the local communities. The focus of these visits was to elicit information about contributions to results as well as on stakeholder perspectives concerning the usefulness of the interventions and their overall satisfaction with the project.
The fourth source of data was a survey conducted among parents of the children using the services established and/or supported by UNICEF.
The evaluation spanned a number of phases.
20
The desk research phase included a detailed outlining of the evaluation process, elaboration of the evaluation questions into an evaluation matrix and a desk review of the available project documentation. The documentation review comprised the following factors:
Project Documentation - Description of Action (narrative, log frame, budget), reports and monitoring reports;
UNICEF Gender Policy Guidelines;
situational analysis, studies, reports and other relevant material focused on social protection and inclusion in Bosnia and Herzegovina as well as other relevant processes in the country that influence developments in the sector.
The document review and the analysis of the project intervention were used to develop the Evaluation Matrix and in the design of the research methods to be applied during the main assessment stage. This process also clarified the approach and the sample of project partners and beneficiaries to be visited during the fieldwork as well as the sample of those to be included in the mini survey.
Field phase: The purpose of the field research phase for this evaluation was to collect the required information systematically in order to support the formulation of conclusions and answers to the evaluation questions. Therefore, the fieldwork included coordination with the UNICEF team, contact with the project partners and beneficiaries and the application of data collection methods. The main elements of the field research phase were the interviews, field visits and the mini survey. The evaluator visited the local communities and the related SPI commissions and social service providers in order to observe first-hand and learn about the expected impact of the project at a given stage.
Site visits: Site visits to local communities took place in Brčko, Gradiska, Prijedor,
Foča, Maglaj, Derventa and Zivinice in order to meet the SPI commissions, the
beneficiary children and their families. These communities were selected to ensure minimum balance between the entities and Brčko District. This required a minimum of five locations; two more were added due to their close proximity to the other locations. This allowed the evaluator to obtain a better insight into the examples of good practice, which were required for the development of briefs (See section 2.1.2 sampling methodology for an overview of the selection of the municipalities).
Interviews and discussion groups were conducted with the SPI commissions, including representatives of local self-governments targeted by the project in the seven sampled communities. Group discussions with teachers, participants of the trainings and courses were organised within the education component of the project
in Derventa, Gradiska and Foča. These communities were selected for discussion
groups with teachers because they were the target locations for education training. The discussion groups provided the evaluator with important findings concerning the impact of the implemented projects (see Annex 5 for a list of the persons interviewed).
The mini survey reached out to parents and children who used the social services provided in eight target communities. This was done with a view to understanding the range of perspectives that beneficiaries may have on the utility of the services and the results and challenges related to the support offered by the project. Two municipalities were not included in the survey because local services were not sustained at these locations.
UNICEF first developed the survey instrument in English for approval before translation into the local language. The survey questions had closed responses in most cases.
21
There was a limited timeframe to conduct the mini survey and this corresponded to the level of technical resources available in each of the local communities.
Adjustment was made to the data collection methodologies throughout the field research stage in accordance with the recognised need to respond to constraints and to maintain and where possible improve the quality of the evaluation. The Field Phase ended with a debriefing of UNICEF concerning the preliminary findings.
Analysis and Reporting: The review and analysis of information and the formulation of conclusions and recommendations were based on the collected documentation and a review of the responses to the survey, site visits and discussions and interviews conducted with a broad range of stakeholders. The ToR and the OECD-DAC principles formed the foundation and framework for the analysis. Wherever feasible, the target indicators, as defined in the log frame, were assessed and referred to in the final report.
Three methods of data analysis were employed:
descriptive analysis was used to understand the relevant contexts that influenced project implementation and to describe the project objectives and interventions. Descriptive analysis was used as a first step before moving on to more interpretative approaches.
Content analysis constituted the core of the qualitative analysis. Documents and stakeholder consultation notes were analysed to identify common trends, themes and patterns in relation to the evaluation questions. Content analysis was also used to flag divergent views and opposing trends. Emerging issues and trends constituted the raw material for crafting the preliminary observations that were subsequently refined and fed into the draft and final evaluation reports.
Quantitative/statistical analysis was used to interpret the quantitative data, e.g. the allocation and use of project funds. If and where feasible, it was used to quantify selected results of the stakeholder consultations.
To ensure the validity of data and as part of the process of synthesizing the information derived from different data sources and through different means of data collection, the following methods were used.
Triangulation – for example, comparing data generated from different data sources in order to identify trends and/or variations.
Complementarity – using, for example, data generated through one method of data collection to elaborate on information generated through another, such as using stakeholder consultations to explore the reasons for the strengths or shortcoming indicated in the existing documentation.
3.1 Sampling Methodology
In addition to a desk review of the relevant documentation and the results of semi-structured interviews with key informants and focus/discussion groups, the evaluation methodology included site visits to a sample of municipalities for in-depth review.
UNICEF proposed a sample of municipalities that the evaluator screened against several sampling criteria in order to check its representativeness in relation to the overall number of municipalities targeted by the project.
Sampling criteria:
1) geographic distribution of the municipalities, namely the municipalities from the two entities (FBiH and RS) and Brčko District (although the number of municipalities was
22
not proportional between the two entities it did reflect the figures relating to the IDP and returnee populations and their needs);
2) typology of supported service providers, i.e. supported health centres, centres for social work and social services (speech therapy, special education, resource rooms), kindergartens and primary schools that benefited from capacity building activities for inclusive education (this was important from the perspective of analysing the various service provision mechanisms for SPI and their effectiveness and efficiency).
As it was not required under the ToR for this evaluation, no control group of municipalities was constructed.
The sample below (Table 2) is based on a review of the key project documentation and analysis comprising seven municipalities made against the sampling criteria, representing 70 per cent of the total number of municipalities targeted by the project. The evaluator met a total of 55 key informants in the seven municipalities, as presented in Table 2 below.
Table 2 Evaluation Sample
Sample Site visits No. of KIIs Survey No. of respondents
Federation of BiH Zivinice
Maglaj
3
8
Bosanski Petrovac
Zivinice
Maglaj
34
63
36
Republika Srpska
Gradiska
Prijedor
Derventa
Foča
8
9
14
12
Gradiska
Prijedor
Derventa
Foča
10
9
7
10
Brčko District Brčko 1 Brčko 41
Total 55 210
UNICEF also proposed the survey sample to target children and families benefiting from all of the services supported by UNICEF in the targeted local communities. UNICEF established these services in 2015 and the local authorities have sustained them since 2016. Therefore, besides the aspect of beneficiary satisfaction, the survey also provided an insight into the sustainability aspect, which is one of the most important evidences for this evaluation. The survey was thus available to the entire universe of children and families benefiting from the supported services.
The survey was administered in eight target communities that sustained local services and resulted in 210 questionnaires completed by parents of children that utilised services supported by the project (Table 2 above presents an overview of respondents per community).
3.2 Safeguarding Ethics
The evaluator applied a Child-Rights Based Approach to the evaluation, guided by five core principles: normativity, participation, non-discrimination, accountability and transparency. The evaluator also applied the Common Understanding on Human Rights-Based Approaches to
23
Development Cooperation and Programming, approved by the United Nations Development Group in 2003:
All programmes of development cooperation, policies and technical assistance should further the realization of human rights as laid down in the Universal Declaration of Human Rights and other international human rights instruments.
Human rights standards contained in, and principles derived from, the Universal Declaration of Human Rights and other international human rights instruments guide all development cooperation and programming in all sectors and in all phases of the programming process.
Development cooperation contributes to the development of the capacities of “duty bearers” to meet their obligations and/or of “rights holders” to claim their rights.
The evaluation culminated in the draft and final evaluation report, which synthesizes evaluation findings and formulates recommendations.
Throughout the process, the evaluation was in compliance with the norms and standards of the United Nations Evaluation Group.
During data collection, attention was given to ensuring that the evaluation process was ethical and that participants involved in the process were able to express their opinions openly and in the knowledge that the confidentiality of their answers was protected (see the interview guide in Annex 4). The UNEG Code of Conduct for Evaluation in the United Nations System (March, 2008) was strictly respected: most notably, independence of judgement, impartiality, honesty, integrity, accountability, respect and protection of the rights and welfare of human subjects and communities, confidentiality, risk avoidance to prohibit harm to and/or burdens placed upon those participating in the evaluation, accuracy, completeness, reliability of reporting and transparency. The evaluator remained sensitive to the beliefs, mannerisms and customs of all stakeholders and acted with integrity and honesty in her relationships, ensuring that contact with individuals was characterised by respect for and the protection of anonymity and the confidentiality of individual information.
UNICEF took the lead in the recruitment of interviewees from different institutions, following the agreed methodology and list of stakeholders of relevance to the evaluation developed by the consultant. Participation in the evaluation was voluntary and opinions presented in the report were anonymous.
Parents and children for the survey were selected in cooperation with the project coordinators and service providers (ECD centres, resource rooms and services). Parents were informed about the scope of the survey and asked to confirm their consent.
The evaluation process engaged stakeholders at three levels: information, participation and consultation. Depending on the nature and engagement of each stakeholder in the UNICEF Project Component, a number of survey, interview and focus group discussion respondents answered qualitative questions on the elements that could be improved or that they otherwise considered satisfactory. Phrasing their responses as recommendations complied with the way in which the qualitative data was coded, analysed and interpreted for these particular questions. Thematic analysis involved viewing all of the available data several times, categorising the data into one or more (as relevant) of the evaluation questions, identifying emerging patterns and themes across all data within each evaluation question, reorganising the data according to thematic categories and synthesising the information contained in the thematic categories. The opinions, experiences and reflections shared during the interviews, focus group discussions and survey were valuable not only as insights and as inputs for evaluation findings but also for checking the perceived priorities for the continuation of the reform of social protection and inclusion for children in BiH by the key stakeholders.
24
This was very useful for informing the recommendations of the evaluation based on an open and participatory process initiated during the Field Phase. It will continue through the report with the presentation of the evaluation findings, conclusions and recommendations.
3.3 Evaluation Matrix
The Evaluation Matrix (see Annex 2) operationalized the evaluation questions by aligning specific questions to data sources and methods of data collection. The matrix guided the development of the data collection instruments and the data collection process. Each line of inquiry fed into the evaluation questions and when analysed, both individually and collectively, it allowed for an evaluation of the validity and the level of confidence in the reliability of the method.
3.4 Limitations, Assumptions, Risks and Mitigation Strategies
There were several constraints to carrying out the evaluation. Table 3 below presents the constraints together with the corresponding mitigation approaches.
Table 3 Limitations and Mitigation Approaches
Potential limitations/constraints Mitigation approaches
Some stakeholders were not available for interview during the site visits.
UNICEF informed stakeholders in advance of the visit and asked for their cooperation.
In those cases where stakeholders were unavailable for face-to-face consultations, the evaluator followed up with the respective individuals at a later date via telephone or Skype.
There was a lack of disaggregated data according to the returnee/IDP status and gender of beneficiary children.
Obtaining reliable gender/status disaggregated data was challenging due to the fact that both UNICEF and the municipal administrations in the target communities did not consistently collect gender/status disaggregated data.
UNICEF requested data that could be collected from different sources, as collected by the municipalities and the project team.
No Training Needs Assessments developed prior to delivery of the training.
The evaluator reconstructed the training needs retrospectively, based on the training reports, interviews with key informants and focus groups with professionals who took part in the training events.
The limited timeframe for this evaluation did not allow for the gathering of more extensive evidence on the functionality of the SPI commissions through field observations.
UNICEF and the evaluator ensured that all actors engaged in the commissions were contacted and offered interviews. Commission reports and UNICEF reports were also reviewed.
This evaluation could not conduct a cost analysis.
It was not possible to conduct a cost analysis because of the overall lack of consistent data on similar programmes, interventions and standards.
25
4. Key Findings
This Chapter examines the performance of the UNICEF Project Component of the overall project within the five OECD-DAC criteria (relevance, efficiency, effectiveness, impact and sustainability prospects). Consideration of these criteria was based on a set of evaluation questions and related evaluation criteria and indicators detailed in the Terms of Reference and the Evaluation Matrix (provided in Annex 1 and 2 respectively). To make referencing easier, the relevant evaluation question was highlighted together with the overall finding in order to guide the analysis.
4.1 Relevance: To what extent is the project responding to the needs of stakeholders and beneficiaries?
The following is an analysis of the relevance of the UNICEF Project Component as seen from the perspective of responding to the needs and priorities of the beneficiaries, particularly the IDP and returnee populations.
Evaluation Question 1. Extent to which the project addresses the needs of the IDP and returnee populations?
The overall project addressed the multitude of issues that the returnee and IDP populations face, including housing, economic and specifically the (UNICEF Project Component addressed) social protection and inclusion issues of children (including access to basic services such as health, social protection and education). The UNICEF Project Component incorporated previously gained lessons learned and best practice when addressing the complex issues of the most vulnerable groups. It strove to improve social protection and inclusion policies within local communities and simultaneously invested in the improvement or establishment of innovative social services through the provision of support to knowledge and skills building as well as infrastructure.
The UNICEF Project Component strategies are relevant to the needs of the most vulnerable children, particularly children and families belonging to the IDP and returnee populations. By context, these groups in particular experience the most challenges in terms of multiple deprivation, poverty, access to services and inclusion in their communities. The project was implemented in ten municipalities across Bosnia and Herzegovina; the selection was based on an analysis of the challenges and remaining needs of the IDP and returnee populations.
The UNICEF Project Component initiated its work in these municipalities by establishing/renewing contact with the commissions for social protection and inclusion. These commissions are municipal bodies responsible for developing local policies and action plans for social protection and inclusion. The first set of collaborative activities was to conduct a comprehensive assessment of the context of social exclusion and vulnerability as an evidence base for local policymaking. These studies proved to be extremely important for partner municipalities as they raised the issues of extremely vulnerable populations, including children, the elderly and others, coming from the IDP and returnee as well as the domicile population. These assessments showed that the main challenges that children and other vulnerable groups in local communities, particularly those hosting larger numbers of the IDP and returnee populations, face are those linked to equal access to social protection and inclusion services.
Through its efforts to strengthen the system for social protection and inclusion, the project addresses the challenges of vulnerable groups. This applies in particular to IDP and returnee families that have only limited access to adequate health, social protection and inclusion services and as a result experience vulnerability and social risks. The project also provided an opportunity to devise systems to address the multidimensional needs of these groups by
26
connecting social protection, health, education, law enforcement, civil society and other sectors in order to strengthen the response to their needs.
Taking into account the fact that by definition social inclusion includes a series of positive actions to achieve equality of access to services, the project provides a means through which the provision of equitable services for all citizens regardless of their status or background and including children coming from IDP and returnee populations can be instigated. Supporting the local commissions for social protection and inclusion to develop an evidence base on children and other groups within their communities has helped them generate more comprehensive data on their needs and status. This will assist local government to create informed policies on social protection and inclusion. Furthermore, local social services established and supported by the UNICEF Project Component have reached out to vulnerable children in the target communities with specific focus on the provision of services to IDP and returnee communities (as evidenced by field observations conducted within the evaluation). One example is the Local Action Plans for SPI and social services (e.g. speech therapy, special education for teachers and ECD resource rooms) organised in returnee communities in Maglaj and Foča, which have improved outreach to and the inclusion of these groups of children in services that were needed but previously did not exist in these settings.
Unfortunately, it was not possible for this evaluation to establish the exact number of IDPs and returnees reached through the UNICEF Project Component because of the fact that monitoring and reports do not disaggregate figures per these categories.
Evaluation question 2. Extent to which the project design is relevant in regard to the overall project goal and the achievement of its objectives in the given time period?
The overall project objective was to enable national actors to address the remaining obstacles to finding solutions for displaced persons and returnees in accordance with the Revised Annex VII Implementation Strategy goals. Within this overall framework, the UNICEF Project Component falls under Specific Objective 3: To enable vulnerable returnees and IDPs to access their rights and inclusive social services.
The project design is relevant to the overall project objective as it tackles the remaining obstacles to social protection and inclusion of IDPs and returnees through the institutionalisation of local mechanisms for social protection and inclusion. The established commissions and working groups within the auspices of the commissions did not only tackle the issue of the rights of children but also other age groups (e.g. the elderly). This strengthened the response to the obstacles and addressed them in a more systematic manner. The overall project framework provided UNICEF and the local commissions with the more comprehensive task of exploring and providing an evidence base and then, based on the data, to develop Action Plans to address the gaps in the social protection and social inclusion systems for vulnerable groups. While UNICEF’s support in most municipalities addressed the needs of children in particular, interviews with the commissions and a desk review of the adopted action plans revealed that these policy documents also addressed issues of other vulnerable groups. Further analysis showed that the action plans also included allocated budgets. This is an extremely important achievement as it shows that the UNICEF assisted municipalities now understand the complexity of the needs in their communities and have taken budgeted measures to address them. A review of the adopted measures shows that the action plans include dimensions of work of the municipal administrations with and through multiple stakeholders as well as different angles for addressing social protection and inclusion. The action plans include innovative ways to ensure that services are provided in the local communities by adopting the models and approaches for tackling the issues of early childhood development and social inclusion shared by UNICEF.
27
Evaluation question 3. Extent to which the project design and its objectives are relevant in regard to national policies and strategies?
As mentioned in this analysis, as per EQ 1, the overall project addresses the remaining issues of the IDP and returnee populations as defined in the Revised Strategy. The Strategy identifies obstacles to solutions in the following main groupings: livelihood, health, education, electrification, water supply, roads and other infrastructure, social protection, housing and property, and compensation for property that cannot be recovered.
The UNICEF Project Component addresses the social protection and inclusion issues by engaging authorities at the local level as coordinators of the project structures (working groups, etc.) to lead targeted actions to resolve problems that affect whole communities and/or groups. The UNICEF Project Component addressed these issues through establishment/support for the functioning of permanent municipal SPI commissions and the identification of vulnerable groups, using the human rights based approach to programming (HRBA). It developed action plans and secured protocols of cooperation between social sector service providers in the target communities. It supported programmes to promote the rights and address the needs of vulnerable groups, training curriculums on early childhood intervention (ECI) and the scaling-up of social services, such as ECD, education and the prevention of juvenile delinquency, for children and families, especially for children coming from the IDP and returnee populations.
Evaluation question 4. The extent to which the project is designed according to international norms and agreements on Human Rights (HR) and Gender Equality (GE) and in line with national strategies to advance HR & GE?
The project design corresponds to international norms and agreements on human rights. Although the principle of gender equality was not at the forefront of the project strategy, there was a good gender balance among the beneficiaries and professionals.
At the project’s core was the application of the human rights based approach (HRBA) to programming with emphasis on the principles of non-discrimination and participation. Through the application of HRBA, the project helped the selected municipalities to address the actual needs of the local population and to develop models and best practice that can be expanded in the future. The project applied child rights indicators as tools to measure the obligations of the country. More specifically, the progressive obligation to protect, promote and fulfil child rights. All of which relates to the international treaty body obligations of BiH and the development agenda as specified by EU inclusion policies. This was done through active support provided to municipal administrations to organise policies, mechanisms and services that would enable access to child rights in line with international obligations and EU inclusion policies.
When it came to gender equality norms and principles the assessment of the project documentation showed that gender was not at the forefront of the strategy, even though the actual implementation of activities reached out to both genders. Disaggregated gender data collection as per the project indicators was inconsistent and represents the most important weakness of the project component.
28
4.2 Effectiveness: To what extent does the project meet the outcomes as defined by the DoA?
This section is organised in such a way as to provide responses to the evaluation questions relating to the OECD-DAC effectiveness criteria.
EQ 5. Extent to which the planned results have been achieved to date (quantitatively and qualitatively)?
The UNICEF Project Component focused specifically on overall project framework Objective 3: Enable vulnerable returnees and IDPs to access their rights and inclusive social services. This objective focused on strengthening social protection and inclusion systems in the targeted localities, with a focus on vulnerable returnees and IDPs, and enhancing the capacities to deliver inclusive services to those identified as the most vulnerable returnees and IDPs in the targeted localities. The evaluation process provided evidence that the planned results were achieved, with a positive perspective on effectiveness (see Table 4 below). However, the analysis was hindered somewhat by the fact that baselines were not established at the onset of implementation of the UNICEF Project Component.
Table 4 Achievement of Results per Indicator
Objectives (results within the overall Action framework)
Indicators Achievements as per UNICEF monitoring data
R5: Social protection and inclusion systems strengthened in the targeted localities, with particular focus on vulnerable returnees and IDPs.
I5.1: Social protection and inclusion commissions establishment in at least 10 municipalities.
I5.2: Evidence-based Action Plans on social protection and inclusion developed in at least 10 municipalities.
Achieved: SPI commissions established in 10 municipalities.
Achieved: Action Plans on SPI developed in 10 municipalities.
R6: Enhanced capacity to deliver inclusive services to those identified as the most vulnerable returnees and IDPs in the targeted localities.
I6.1: At least 300 vulnerable children and caregivers provided with access to inclusive social services (education, health, social protection and child protection).
Achieved: 873 children and their caregivers provided with access to inclusive social services (education, health, social protection and child protection).
R8: Awareness raised and positive attitudes and behavioural change initiated in relation to returnees and IDPs in the targeted localities.
8.1: A 20% increase in positive attitudes toward returnees and IDPs achieved in the targeted localities.
Achieved: KAP survey showed an increase in positive attitudes toward returnees and IDPs in the targeted localities.
Key achievements in the establishment of an evidence base on the vulnerabilities of local communities were recorded as support for the complex process of data collection aimed at elaborating a municipal Situational Analysis. These documents set the foundation for the development of Action Plans whose budgets are allocated by the municipal governments. This represents a major step forward, particularly when you consider the fact that the
29
targeted communities are (with the exception of Brčko, Prijedor and Gradiska) mostly poor and underdeveloped. An electronic database has been established and is available in all of the municipalities. The database will administer and unify data collected through the circulation of sectorial questionnaires through the centres for social welfare, public health and education institutions, the ministries of internal affairs, the judiciary and civil society organisations in the target municipalities. The database contains all relevant sectorial data collected by the SPI commissions and has the capability to generate sectorial and cross-sectorial reports. It serves as a municipal database available to all relevant institutions and sectors and therefore ensures the sustainability of the SPI model. This is an important contribution as municipalities now have baseline data on their communities that can be updated and serve to monitor trends and new needs in communities. Organised municipal services, in many cases funded by municipalities (e.g. Foča and Maglaj), have been instrumental in advancing the inclusion and development of children. Interviews conducted with different stakeholders showed that investment in building the capacities of service providers from the education, health and social welfare sectors to provide inclusive services has proven an additional motivational factor for these professionals and brought new knowledge and to some extent skills. Lastly, direct work with children materialised into strong messages sent by children (through films and other communication tools) about exclusion, poverty and access to rights. Specific contributions and results are elaborated further in the analysis as per EQ6 to EQ10 below.
Evaluation question 6. Extent and ways in which the project has contributed to the enhancement of systems and capacities to deliver inclusive services at the local level, with a focus on IDP and returnee populations?
The project interventions have contributed to the enhancement of systems and capacities to deliver inclusive services at the local level to socially excluded groups, with contributions to the IDP and returnee populations. The UNICEF Project Component was organised within Specific Objective 3 of the overall Action (Specific Objective 3: To enable vulnerable returnees and IDPs to access their rights and inclusive social services), which was taken as the overall objective of the UNICEF Project Component. The three related results outlined in the overall Action were taken as specific objectives of the UNICEF Project Component (See Table 1 of this report).
The first UNICEF objective (Result 5 of the overall project) focused on strengthening social protection and inclusion systems in the targeted localities, with a focus on vulnerable returnees and IDPs. The second UNICEF objective (Result 6 of the overall project) focused on enhancing capacities to deliver inclusive services to persons identified as the most vulnerable returnees and IDPs in the targeted localities. The third UNICEF objective (Result 8 of the overall project) focused on raising awareness and initiating positive attitudes and behavioural change in relation to IDPs and returnees in the targeted localities (see Table 1 of this report and the overall log frame in Annex 3).
In relation to the first objective of the UNICEF Project Component, assistance to the municipal administrations focused in particular on the respective departments for social affairs. The assistance was directed in such a way that it supported these departments in establishing and/or enhancing cooperation, coordination and networking between different sectors and local institutions (health, education, social welfare, justice, police, civil society, etc.) in order to develop action plans based on evidence collected by the commissions for SPI. The evaluation process collected evidence that showed that the commissions have been functional: all of the commissions successfully completed Situational Analyses that formed the basis for the development of Action Plans for SPI.
30
An analysis of the available budget information from the target municipalities showed that each municipality planned and utilised funds for SPI activities envisaged in the Action Plans. This was done either through budget lines for implementation of the Action Plan (Zivinice, Maglaj, Gradiska*, Derventa and Bosanski Petrovac) or through contributions approved by
the mayor for Action Plan activities (Prijedor, Mostar, Foča, Bijeljina and Brčko District).
The intensive support provided also resulted in the signing and/or revitalising of protocols on the referral of services in each municipality. Referral Protocols enable official lines of cooperation and referral by endorsing official authorisations for the referral of cases between institutions. The value of the Referral Protocol lies in the fact that it lays out a common understanding of the lines for referral and the ways in which relevant actors should communicate in cases when referral is necessary. Signing/revitalising protocols represents another successful achievement of the project, especially when you consider that the protocols were put into use by the commissions. According to the interviews conducted for the purposes of this evaluation, the commissions now actively work on cases in a more coordinated manner. Concrete examples of such cases were not recorded by the evaluation for reasons of confidentiality of data in such operations. The visited municipalities confirmed that they have mechanisms in place and that they are using them. Working within the context of the larger programme, focused on solving housing, legal aid, social and other issues of IDPs and returnees, also helped strengthen the commissions and gave them more authority and power as the main point of contact (and decision maker) for many project related issues and the wider sphere.
Interviews and field observations show that while at the onset the commissions were faced with significant challenges in terms of the collection of data on IDPs and returnees they have seized the opportunity to become directly engaged and taken the ‘driver’s seat’ in interventions of local importance. As discussed above in EQ 5, the data collection process was conducted through the distribution of sectorial questionnaires to the centres for social welfare, public health institutions, institutions of education, ministries of internal affairs, the judiciary and civil society organisations in the target municipalities; the questionnaires were completed and returned to the commissions. Interviews conducted with members of the commissions (representatives of different sectors) confirmed that they experienced a major challenge in collecting data on indicators with disaggregation per IDP and the returnee population in particular.
This was especially true when it came to children, because none of the institutions placed emphasis on recording this kind of data on children. As per their definition, returnees were in a community prior to the war yet their status was not followed-up in accordance with these parameters. The IDP population has been resident in host communities for more than twenty years and therefore, in some cases, even the parents of the children in question were born in the host community and therefore children constitute the second generation. This situation also applies to the returnee population to large extent. Yet despite these problems, the interviewed commissions confirmed that once the data had been collected it provided a better and more comprehensive overview and context of their communities and the related vulnerabilities of different groups. Unfortunately, because of the limited timeframe for this evaluation, it was not possible to obtain more and further detailed evidence on the functionality of the commissions via field observations.
Simultaneous to its support for the establishment of the commissions and their work UNICEF also worked on capacity and skills development for professionals dealing with different dimensions of SPI in the local communities, including but not limited to commission members. Support was provided to professionals in education, the centres for social work and health institutions. UNICEF worked closely with the Pedagogical Institute of Republika Srpska when conducting training on social inclusion within the education system in the entity. This cooperation resulted in three sets of training for teachers from the targeted communities within the entity of Republika Srpska. Group discussions showed that this approach was appropriate
31
as it provided an institutional framework and continuity for such training; the UNICEF supported trainings were part of a larger framework of training provided by this institution. Teachers participating in the group discussions confirmed that the training helped them gain further theoretical knowledge but only limited practical skills. The latter is an area for further investment aimed at ensuring enhanced skills for inclusive education.
Discussions with social service providers, teams from the centres for social work and health institutions confirmed that the UNICEF supported trainings and skills building in the area of early childhood development and early detection were of critical importance for improving their understanding of how early detection methods can be applied to assist children with developmental delays. The imparted knowledge will help these professionals to assist children and their families to cope and speed up these children’s development during the critical age prior to starting school. Within the scope of this intervention, the Early Childhood Intervention (ECI) Working Group in collaboration with the civil society organisation EDUS developed two guides on child assessment and creating individual development plans for children with developmental delays and disabilities aged 0-3 years and 3-6 years. These guides will assist ECI educators. In addition, a Curriculum on Parent Education for ECI educators and a Brochure for Parents, on work with children with developmental delays and disabilities, were developed.
A review of the project information showed that 99 SPI commission and working group
members were trained in the SPI model, while 88 SPI commission and working group members were trained in PCM, results-based budgeting and effective communication. The project reached out to 317 service providers within child protection, 43 within early childhood development and 11 within social protection. They received training that focused on specific areas of their expertise with the aim to provide inclusive services to vulnerable children and their families.
UNICEF supported capacity building for a total of 317 professionals aimed at the advancement of practices for working with children that come into contact with the law, i.e. either as victims, perpetrators and/or witnesses of criminal offences. The particularities of working with child victims of different forms of violence were also included in the UNICEF child protection component. Workshops that engaged lecturers from academia and the independent human rights institution ‘Ombudsperson for Children in Republika Srpska’ focused on sharing input on current national and international standards on child protection. These workshops engaged professionals in active discussions on recorded their views on the major bottlenecks that could potentially hinder putting these standards into practice. The workshops also presented an opportunity to further elaborate and map the requirements for tackling the challenges related to working with children at risk and enhancing the capacities to respond to the needs of these children. Throughout the discussions it became evident that more work needs to be done in schools on how to recognise and respond to cases of violence, abuse and neglect of children as well as cases involving referral to other relevant institutions.
Feedback from the interviews and discussion groups with the SPI commissions, teachers and service providers together with the desk review and interviews with UNICEF confirmed that systems and capacities have been strengthened through the support provided through the project. All of the interviewed stakeholders agreed that their municipalities had a more institutionalised approach to SPI that is more resilient to the political and socioeconomic challenges within these municipalities. This was confirmed by increases in budgets and sustained support for the Action Plans for SPI. The findings of the survey among clients of these services also showed a high rating for the services supported by UNICEF. However, this evaluation was not able to establish any evidence on the level of satisfaction among pupils/students, the police and the clients of the health centres and the centres for social work.
32
Evaluation question 7. Extent and way in which the project has contributed to raising awareness on the social inclusion of the IDP and returnee populations?
Advocacy for the rights and social inclusion of vulnerable groups with special focus on IDPs and returnees was organised through a range of activities with and for commission and social service providers, as discussed above. These activities included focused activities such as but not limited to One Minute Junior films with a mixed group of IDPs/returnees and other pupils in selected primary schools and roundtable discussions and social media campaigns aimed at raising awareness on the social inclusion of IDPs and returnees. A total of 10 workshops were held involving 102 children resulting in the production of 102 films. In addition, 10 TV shows intended to raise awareness on the social inclusion of IDPs and returnees were broadcast by 5 local TV stations and via social media.
Interviews and site observations showed that activities with the commissions and service providers were very important for raising awareness on the needs of the IDP and returnee populations, especially when you consider the fact that prior to this the service providers never made any formal distinction between the domicile, IDP or returnee populations. This means that previously their particularities and needs were not at the forefront of strategic planning in these institutions. The commissions and service providers reflected that the project requirement to explore and learn about these populations (as well as the domicile population) was difficult at the onset but important in terms of raising awareness on the multiple deprivations and social needs of these groups, particularly those communities that are outside of urban areas (villages - some of which are far from the municipal centres). The commissions confirmed that the collected data is stored in databases that are now more comprehensive and easier to update and that these databases provide a more accurate picture of the different contexts within the municipalities. However, this evaluation could not find evidence that these populations will continue to be monitored in the period after the project finishes.
The awareness raising activities with children mentioned above were highly interactive and a valuable experience for the children as they had the opportunity to express their creativity and at the same time learn about human rights and the vulnerabilities of their peers coming from the IDP and returnee populations. A review of the one-minute junior films underlined some important, moving and creative stories captured by the children that represent a valuable investment in building confidence and trust among children coming from different groups in the target communities. Lastly, the project supported the production of a TV series featuring the major results of the project and its benefits for children in coordination with RTV USK. The series was broadcast on RTV TK, HTV Oscar C, HIT TV Brčko, TV Zenica and RTV BPK Goražde. While direct evidence on the outcome of awareness raising and enhanced positive attitudes could not be established by this evaluation, the project data shows that there was a 20 per cent increase in positive attitudes toward returnees and IDPs within the targeted localities. The latter was confirmed by the baseline and final Knowledge, Attitudes and Practice surveys.
Evaluation question 8. Extent to which the project has contributed to strengthening the capacities of service providers to provide quality assistance to the target groups?
Most direct benefits to children in need, including those coming from IDP and returnee groups, materialized through the project contribution to strengthening the capacities of service providers to provide quality assistance. The project monitoring data shows that 873 children benefited from access to inclusive social services (education, health, social protection and child protection), although disaggregated data on IDP and returnee children was not available. The project worked closely with service providers both in public institutions and on the CSO side. Best practice models were shared and lessons learned discussed throughout the implementation of the project activities. The number of service providers trained in different areas of education, early detection of developmental delays and disabilities, and interventions
33
are listed below.
Year I: 99 SPI commission and working group members trained in the SPI model and 88 SPI commission and working group members trained in PCM, results-based budgeting and effective communication.
Year II: 244 service providers from the education, health and social welfare sectors.
Year III: 371 service providers (317 under child protection, 43 early childhood development and 11 social protection).
Interviews and focus groups with professionals coming from different areas of SPI showed that the capacity building interventions were both timely and appropriate, particularly in terms of linking newly arising needs to a better evidence base established in the municipalities within the situation analyses. Capacity building interventions were seen as an integral part of the holistic approach. Building professional skills aligns and responds to new needs and the initiatives of local communities envisaged in the Local Action Plans and stirred by the commissions for SPI at the local level. The interviews showed that the training brought new values and methodologies to work with children, particularly those with special needs.
The desk review and interviews showed that investment in training went hand in hand with UNICEF support for the Special Focus Projects (SFPs), which provided the target municipalities with opportunities to invest in infrastructure and new services (e.g. speech therapy, special education teacher services, resource rooms and day care centres) that were identified as needed under the situational analyses. These projects were an important part of the overall approach: The commissions had a policy and stirring function, while the possibility to organise needs-based services gave them the opportunity to apply Action Plan measures and test new services with mentoring support provided by UNICEF for the first phase of implementation. A review of the project data showed that 873 children along with 285 parents were reached and included in direct service provision.
Children and their families, respondents to the survey conducted within the scope of this evaluation, used a range of services supported by the project; almost one-third of respondents used the speech therapy service (see Table 5 below).
Table 5 What services did you use?
Early assessment and intervention 11.5%
Counselling work with parents 1.0%
Educational workshops in the field of sport 4.3%
Educational workshops on the development of volunteerism among young people
1.9%
Educational workshops on the literature 2.9%
Educational workshops in the field of fine arts 3.3%
Educational workshops in the field of psychosocial support 1.0%
Educational workshops in the field of first aid 1.0%
Educational workshops in the field of journalism 1.0%
Early assessment and intervention 0.5%
Speech and language therapy 24.9%
Somatopedic treatment 5.7%
Advisory therapeutic treatment with a neuropsychiatrist and a psychologist
1.9%
Observation and preparation of individual plans for a child 0.5%
Educational workshops for parents 0.5%
Educational workshops for children 2.9%
Individual and advisory work with children 2.4%
34
Group work with children with disabilities 2.9%
Group work with parents of children with disabilities 1.0%
All of the above 29.2%
Children became clients of services through different channels, most often through various means of information, a referral or invitation from service providers (see Table 6 below).
Table 6 How did your child and you become clients of the services?
A referral from a health centre / hospital / clinic 7.2%
An invitation from a social worker 1.4%
A referral from the centre for social work 12.4%
A referral from a kindergarten / school 14.4%
Through a NGO 0.5%
An invitation from a teacher 7.7%
A call from the centre for social work 7.7%
Through the Red Cross 2.4%
Volunteer 1.4%
Other source of information 44.0%
Other (invitation from a speech therapist or association) 8.1%
Respondents to the survey found that many features of the services supported by UNICEF (listed in Table 5 above) a motivating factor to start using them (as presented in Table 7 below). Almost all respondents stated that the level of information on voice and speech impediments was the feature that motivated them to use the services.
Table 7 What motivated you to become a client of the service for your child?
Answer Options Response Percent
Response Count
All of the below mentioned features of the service 68.6% 118
Quality of the service 12.2% 21
Availability of the service in the local community 9.9% 17
Innovativeness 0.0% 0
Level of emotional, spiritual and social support 10.5% 18
Education on how to understand children’s needs 4.7% 8
Education on and an understanding of what constitutes adequate development in children
6.4% 11
Other 12.2% 21
answered question 172
skipped question 38
The survey respondents confirmed that the services (as listed above in Table 5) supported by UNICEF were not previously available in their communities (as presented in Graph 4).
35
Graph 3 Did these or similar services exist in your community prior to the establishment of the service you are currently using?
The respondents presented various features of the services (see the list of services in Table 5 above) that were beneficial to them. These included but were not limited to the benefits of children being taught to pronounce letters, syllables and sentences correctly; to quote correctly and being assisted in losing a stutter. Other benefits were reading and writing, improved communication, obtaining information on how to treat a child who has trouble speaking and ways to help children overcome these difficulties. Exercises aimed at increasing a child’s interest in socialising with his or her peers, professional guidance on how to work with a child, the possibility of individual work with a child and programmes adapted to all aspects of child development were cited by respondents. Other benefits included how a child qualifies for inclusion in their school, the possibility of individual work and early intervention, socialisation of the child and improvements in relation to the child's development through ‘individual treatments’. Parents also benefitted directly through training for parents, workshops for children and parents, children becoming independent and traveling without escort that enables parents to have some relief. Additional benefits included the daily work of special education teachers, sensory rooms, the work of sociologists, etc.
As shown in Graph 4 above, almost half of the respondents (43%) confirmed that such services (as listed above in Table 5) did not exist in their communities before the project, while 36 per cent of them stated that they used to use the service in another community of relative proximity to theirs (e.g. speech therapy and day care centre services). The respondents confirmed the vast majority of service benefits for children, including improved speech abilities, socialising with peers, children qualifying for inclusion in their school and children becoming independent as well as the overall benefits in terms of the wellbeing and sense of empowerment of children and parents.
Generally, the respondents were satisfied with the services provided and saw them as available, accessible and affordable (as presented above in Graph 3). All but five of the respondents thought that the teams providing the services were open and accessible. The five respondents (out of a total of 160 respondents for the related Question - 17) who had a different opinion stated that the teams working on group work with parents of children with special needs were not interested, while seven stated that the teams were not interested in group work activities with children and parents.
5.9%
43%
11%
36%
Yes
No
I don't know
Before, we used the serviceoutside our community
36
Graph 4 How would you rate the services you have been using?
The respondents thought that many of the new services would be desired, including sessions with psychologists and psychiatrists, physiotherapy and more frequent workshops for children, etc.
The respondents saw the main limitations of the services offered by the health centres, centres for mental health and centres for social work as the low number of employees and working hours (some of these institutions, such as the Resource Room in Prijedor do not work every day). Other limitations mentioned included too large a number of children in need, the inadequate number of therapy sessions, the financial limitations, the lack of professionals (e.g. speech therapists and special education teachers in communities), the lack of adequate resources and the unclear future status of such services once the project ends. In one community (Zivinice), respondents mentioned the lack of accessibility and the inhospitable attitude of staff at the centre for social work.
Another important intervention was entrepreneurship learning for primary school children. UNICEF, in cooperation with the CSO ‘Nesto Vise’, implemented the ‘Life skills for the future – Let’s learn, let’s do it!’ Project, which contributes to the development of the entrepreneurial skills of primary school children in the municipalities included in the Annex VII Action. The initiative reached 200 primary school pupils and 20 school teachers from five of the target
municipalities (Bosanski Petrovac, Derventa, Foča, Mostar and Zivinice), who attended entrepreneurship trainings and gained knowledge and skills on how to develop business plans. The result of this process was the entrepreneurship clubs established in each of the participating schools. Additionally, five representatives of local NGOs were trained to provide
0102030405060708090
100
I am very satisfiedwith the service I
used
I am satisfied withthe service I used
I am not satisfiedwith the service I
used
Services areavailable,
affordable andaccessible
Services are notavailable,
affordable andaccessible
I don't know
Early assessment and intervention
Individual work with parents
Counselling work with parents
Advisory - therapeutic work with parents
Educational workshops for parents
Speech and language therapy
Somatopedic treatment
Advisory - therapeutic treatment with a neuropsychiatrist and a psychologist
physiotherapy
Observation and preparation of individual plans for a child
Educational workshops for parents
Educational workshops for children
37
support for school teams. Twenty entrepreneurship projects were implemented and pupils participated in regional school fairs. While this evaluation could not establish evidence on the impact that these initiatives had on school pupils and teachers, interviews conducted with the CSO team and UNICEF brought forth positive examples of the children’s ability to explore new ideas and business potential. Some schools even managed to acquire funds through small business activities such as the production and sale of chalks to the school. Money earned through these entrepreneurial activities was used for school activities such as excursions.
Evaluation question 9. Major factors influencing the achievement or non-achievement of the project objectives to date?
The major driver of the UNICEF Project Component of the project was a well-established model and holistic approach to SPI at the local level. UNICEF has been developing this model for years through implementation of various phases of the SPIS Project, which is also supported by the EU. The model that UNICEF furnished for implementation of this project component brought the best and most operational approach. UNICEF utilised its team members with clear ToR to support the commissions (one team member with direct responsibility for this) and support the protocols and working groups (one team member), while the ECD sector of the UNICEF country office worked on the ECD components and the educational sector worked on the education component, child protection and C4D. This presented a lean approach and clear responsibilities resulting in better integration of the interventions, which was not the case in the previous phases. Analysis of historical data from the SPIS Project and the UNICEF Project Component that is the focus of this evaluation showed that UNICEF capitalised on previous lessons learned and achievements leading toward the final shaping of the model for the specific circumstances in Bosnian and Herzegovinian. Another driver of success was the fact that the commissions were in the ‘driver’s seat’ throughout implementation of this project, which gave them both the authority and the motivation to invest their efforts in the quality implementation of the project but also the measures contained in their action plans.
On the other side, a major hindering factor for the planned achievements was the ever-present political situation and the resulting challenges that have a strong influence at the local level in particular. UNICEF mitigated these challenges by ensuring that the commissions were institutionalised into the municipal structures through the official approval of the mayors and the municipal assemblies/councils; however, there is the ongoing threat that the successes of the project will be diminished through political changes in the municipalities. Other hindering factors include the challenging overall socioeconomic situation, the persistent beliefs and attitudes expressed toward IDP and returnee families and children, and the slow pace of reform of the social protection and inclusion system in general and for children and families in particular.
Evaluation question 10. Extent of the additional (not directly planned by the project) significant contributions/outcomes toward the development of services and social inclusion for the most vulnerable and excluded children, with a focus on children from the IDP/returnee populations?
The overall project (including the UNICEF Project Component) has had a multiplier effect at the local level through investments made by other project partners. These included tackling the most important needs of the IDP and returnee populations through an approach that simultaneously addressed their most important needs in terms of housing, access to rights
and socioeconomic empowerment. This involved the reconstruction of housing units and connecting these units to public utilities (Result 1 of the overall project), strengthening value chains to support the economic integration of returnees and IDPs (Result 2 of the overall project) and livelihood support secured for vulnerable
38
returnees and IDPs (Result 3 of the overall project). Free legal aid for returnees and IDPs was also provided in the target localities (Result 7 of the overall project). This evaluation focused on the UNICEF Project Component and it was not possible to establish the level of achievement within other components of the overall project.
4.3 Efficiency: To what extent did the management of the project ensure timelines and the efficient utilisation of resources?
The analysis of efficiency is structured as per the evaluation questions relevant to this section.
EQ 11. How well has the implementation of activities been managed, to what extent have the activities been implemented as scheduled and what management and monitoring tools were used?
Evidence collected through the desk review and interviews with UNICEF and UNHCR showed that the implementation of activities was managed smoothly, after initial issues deriving from setting up the larger project in coordination with UNHCR and other partners. A desk review of the available UNICEF Project Component reports showed that there was a five and a half month delay in signing the agreement between UNHCR and UNICEF, which caused delays in the project activities. In agreement with UNHCR, a request for amendment of the Agreement was submitted to UNHCR immediately after signing the Agreement. This was done to reflect the changes in the Action Plan caused by the fact that the 12-month activities plan had to fit under the five and a half month implementation period. EUD approved Amendment 001 in November 2014. In addition, in December 2014, UNHCR and UNICEF agreed on a three-month no-cost implementation extension for the planned 2014 activities.
Besides the five and a half month delay in the finalisation of the Agreement between UNICEF and UNHCR and the devastating floods, an additional two-month delay was caused by the confusion over contradicting instructions by implementing partners, without proper coordination and consultation with UNICEF, on the composition and proper structure of the SPI Commissions, Municipal Operations Teams (MoTs) and the Working Groups. The desk review revealed a dispersed approach among project partners (overall project framework), which caused confusion and uncertainty among municipal interlocutors. This is why, upon request of UNICEF, UNHCR, as the lead coordinator of the Annex VII Action, prepared a clarification of duties and responsibilities for both the municipalities and partners. As per this clarification, UNICEF ensured that the SPI commission structures and other components to be implemented by UNICEF on the basis of its prior experience and expertise in BiH were formed and guided in a manner that would ensure their full potential and contribution to the implementation of the Annex VII Revised Strategy. UNHCR coordinated the activities of other project partners in order to ensure that the different components of the Action formed a coherent strategy and worked effectively together toward the same objective.
The situation concerning overall coordination and the delays incurred put a strain on UNICEF, which had to ensure that all mechanisms (SPI commissions and the working groups) were in place for the other components to run smoothly. The UNICEF team mitigated these challenges through prompt action. Interviews with the commissions revealed that strong coordination and backstopping by the UNICEF team helped them to work more efficiently toward achieving better coordination of activities within the different areas of the overall project intervention.
UNICEF applied its internal monitoring and evaluation tools to the project in order to ensure that all data was recorded accurately and filed. A key tool for monitoring process was tracking progress against indicators, as set out in the project log frame. Written progress reports were shared with UNHCR on a regular basis, while monitoring was conducted through regular field visits and backstopping by the project team. UNICEF engaged a Project Assistant located in Banja Luka and a SPI consultant to provide guidance to the SPI working groups. Project
39
oversight was further maintained through regular programme coordination meetings involving all UNICEF sections: education, early childhood development, communication, and social and child protection. These meetings were used to discuss and agree on the overall progress of the project implementation from a cross-sectorial perspective that included the budget utilisation plan and expenditure status.
Importantly, through desk research and the field phase, the evaluation found evidence that UNICEF had reviewed and incorporated lessons learned through past experience not only in the overall project design (focused only on the local level) but also in view of how to focus project interventions on what could realistically be achieved within the given limitations (time, resources, political and socioeconomic factors).
Consulted project partners recognised the positive role played by the UNICEF team and its expertise in the subject matter, knowledge of the complexity of the country context and choice of approaches to tackle the gaps and needs as well as their personal dedication. The team was praised for its flexibility, which allowed project partners to adapt plans according to the changing context and evolving insights.
Evaluation question 12. Were the financial resources used in a quality manner (were funds managed in a cost-effective manner), what was the correlation between the funds utilised and the outputs, were results achieved and could the same results have been achieved with less resources?
The desk review and interviews conducted with the UNICEF team revealed that UNICEF used the available project funds efficiently. The budget for the UNICEF Project Component was EUR 887,900 and the component lasted for 30 months. Despite the initial delays, UNICEF succeeded in completing the originally planned activities and achieved the envisaged results as measured against the set indicators (see Table 4 above). The decision to engage a project assistant and a SPI consultant to complete the UNICEF in-house team was a good measure that ensured both the cost-effectiveness of human resources and continuous backstopping and technical assistance for the municipal authorities and partners.
No significant deviations from the planned budget were recorded, while the ratio of programme versus operational costs remained balanced. A review of the budget expenditures showed that operational costs amounted to 16.3 per cent of the total budget, including salaries and local office costs (this includes the salaries of staff working on the implementation of project activities). The remaining balance after project closure was USD 2,194.52 per all three contracts or 0.17 per cent. This can be attributed to various reasons, including a fluctuating exchange rate and shorter implementation period caused by the delays in signing the contracts. UNICEF and UNHCR agreed on budget revisions for each year based on actual costs accrued per each contract.
Additional efficiency features of the project consisted of the use of available premises in hospitals (Dom Zdravlja) and kindergartens for the delivery of IECD services. This was in order to minimise costs and to ensure as far as possible coordinated implementation schedules at the local level. The procurement process was transparent and guided by the value-for-money principle. Viewed from this perspective, the efficiency of the project can be perceived as high.
Evaluation question 13. To what extent did the project ensure co-ordination with other similar interventions in order to encourage synergy and avoid overlap?
The overall project framework included a number of project partners with UNHCR having a coordination role. The role of UNICEF was to ensure that the SPI commissions and working groups were established, functional and able to coordinate the overall project effort. The
40
evaluation process showed that UNICEF managed to support these bodies in such a way that ensured coordination with other project partners, encouraged synergies and avoided overlaps. Project level coordination with all implementing partners was organised through periodic project implementation team (PIT) meetings, stirred by UNHCR. This evaluation was unable to find evidence of instances of cooperation with other UN and/or international actors external to the overall project.
Evaluation question 14. Extent of the flexibility of the project design in terms of adapting to the changing environment (impact of the recent flooding, etc.)?
The project remained flexible in order to adapt to the changing environment, particularly the challenges that arose after the flooding that occurred in 2014 and the political turmoil both during and after the local elections in 2016. For example, the UNICEF team continuously monitored the work of the SPI commissions and the working groups providing assistance when needed and adapting its approach to the changing circumstances in the local communities. This was most visible in the communities of Bijeljina, Zivinice and Gradiska, where the situation became complex after political changes to the local administration following local elections held in 2016. In Bijeljina, for example, political changes resulting from the elections meant that most of the SPI commission members were dismissed from their positions. Because of this, the commission stopped its work and this caused all other project activities to cease. UNICEF continuously sought ways/systems through which the ECD centres could be sustained without UNICEF support but within the institutional set-up.
Evaluation question 15. Extent to which human rights and gender equality were a priority in the overall project budget and implementation?
The project’s underlying principle and approach in all interventions was the human rights based approach (HRBA) and therefore the budget, while not explicitly stating this, covered human rights. Gender equality was not at the forefront of the project and there was a lack of gender-disaggregated data within project interventions.
41
4.4 Impact: To what extent has the project improved the social protection and inclusion systems in the target municipalities?
EQ 16. What is extent of the project’s contribution toward the promotion of a multi-sectorial approach to SPI through the establishment of protocols on cooperation and referral mechanisms in the targeted municipalities?
The evaluation process shows that UNICEF succeeded in achieving all of its envisaged results (as presented in Table 4 above), including but not limited to establishment of the SPI commissions and the working groups, and the signing of protocols on cooperation and referral mechanisms in all of the target municipalities. Yet the existence of formal mechanisms within the municipal administrations do not by themselves mean anything unless there is commitment and measures to utilise these mechanisms to improve approaches to social protection and inclusion. To ensure this, UNICEF applied a coherent approach that provided intensive and continuous support to local stakeholders through backstopping and advice by the UNICEF team to the SPI commissions and service providers supported under the UNICEF Project Component. This approach ensured that local actors had all of the support needed to implement the various steps in the development of the local systems and mechanisms as per the SPI model. In such a way, UNICEF successfully supported structural reform of the social protection and inclusion systems at the local level and the institutionalisation of systematic sets of measures to tackle the most pressing issues of the target groups proactively.
With all of the model’s features in place in the municipalities, the local governments now have established evidence based Action Plans and most (with the exception of Brčko, Prijedor and Gradiska) of them are budgeted. The municipalities have improved knowledge, human resources and material capacities (resulting from the SPFs and investment in capacity building of SPI professionals as well as experience in the provision of social services). This means that the municipalities now have a strong foundation from which to ensure a more adequate and systematic response to the needs of vulnerable groups in their communities, including IDPs and returnees. However, the real impact of this foundation depends on a number of factors, most of which are beyond the control of the project. This relates in particular to political and economic factors, especially the commitment of local governments to ensure funds to accompany the action plans once the project ends and for the foreseeable future.
Evaluation question 17. What intended and unintended and positive and negative results have the interventions had on the lives of children?
Institutionalised evidence based policies and mechanisms to tackle social protection and inclusion have multidimensional benefits for communities. This can be in the form of better policies and coordination and improved access to services for groups in need, including IDPs and returnees. The UNICEF Project Component assisted local governments and their partners from the SPI service providers to establish and institutionalise such mechanisms (SPI commissions, protocols on cooperation and referral mechanisms, Action Plans). Through direct work on capacity building of SPI professionals improvements are already visible, as per the desk review, in the work with and inclusion of children in education, health and social services. Social services provided in the target communities reached out to 873 children and 285 parents through direct service provision as well as 390 children through child participation events. Attention in these communities was drawn to the IDP and returnee communities, which represents a significant contribution to these groups. It was not possible to ascertain the exact number of IDP and returnee children benefiting from these services because of a lack of monitoring data. However, feedback from awareness raising campaign participants showed positive changes in the way children think about and relate to each other.
42
While this evaluation could not establish conclusive evidence as to the extent to which the investment in policies will have a positive impact on all of the children, particularly those coming from the IDP and returnee populations and suffering from a lack of access, theoretically it can be assumed that if the mechanisms are applied consistently and based on updated evidence they will have a positive impact.
Question 18. What were the unintended results of the intervention on human rights and gender equality and were they positive or negative and in which ways did they affect the different stakeholders?
UNICEF ensured the rights of vulnerable groups through the application of HRBA, engraining this into the approaches, mechanisms and services supported. An improved evidence base on the social context of communities resulting from the support provided for a situational analysis at the municipal level strengthened capacities and institutional mechanisms (SPI commissions, budgeted action plans and improved services) and contributed to improved respect for and the protection of human rights, particularly those of children and especially children from the IDP and returnee populations. This evaluation could not establish any specific results on gender equality, because of the lack of gender-disaggregated data collected within the project.
4.5 Sustainability: To what extent are the achieved project outcomes sustainable?
The prospect of sustainability was assessed against the evaluation questions as per the Terms of Reference for this evaluation and therefore the analysis was guided by the set criterion.
EQ 19. Did the project design include an appropriate sustainability strategy (including promoting local ownership, use of local capacity, etc.) to support positive change for the most vulnerable groups after the end of the intervention?
Neither the project DoA or the UNICEF Project Component included a comprehensive (standalone) sustainability strategy. Yet the project envisaged continued efforts toward the institutionalisation of the mechanisms and services established and/or supported as critical drivers for the sustainability of the achieved changes. By its nature, the UNICEF approach included important steps to ensure the sustainability of the changes achieved in relation to the most vulnerable groups. These steps included but were not limited to capacity development and the building of know-how, synergies between the partners, institutionalisation of the established mechanisms (based on evidence collected as their foundation) and justification of the selected direction for SPI. These steps contributed to building a solid basis for sustainable SPI.
In essence, the UNICEF sustainability strategy is reflected in the fact that both the commissions and the working groups (known as Operational Teams) are permanent municipal bodies with ToR that are adaptable to changing circumstances in their communities. The desk review showed that all of the commissions and working groups have already adopted Work Plans for 2017. These documents envisage the development of follow-up Action Plans for SPI for the period 2018 -2019 as an integral activity to take place in all of the communities in June next year. In addition, the established electronic database is a sustainable measure and a tool for data collection and monitoring trends.
Evaluation question 20. Extent to which the new knowledge and skills have been integrated into regular activities of professionals working with children?
All of the interviewed representatives of service providers confirmed that the acquired knowledge and skills have been important drivers of their work, helping them to understand the complexities of social inclusion and the ways and approaches to act inclusively in their workplace. The first outcomes of newly acquired approaches to diagnostics and care have
43
already benefitted the lives of children included in the improved services. Local governments have recognised these investments and now contribute funds for new services in local
communities (e.g. Foča and Maglaj fund speech therapist services both in the town and in
returnee communities) or budget Action Plans for SPI, which is a strong measure of sustainability. However, the level to which the government counterparts will succeed after the closure of this project in retaining and expanding this knowledge remains in question. The survey showed ambivalent feedback by parents: while 60 per cent of parents ‘believed’ or ‘strongly believed’ that the services would continue to be available for as long as children need them, 28.1% ‘believed’ that due to a lack of donor funds or government budget to support these services they would not be available after the project closed. 11.7 per cent of respondents were ‘not sure’ or ‘did not know’, because of budget constraints (see Graph 5 below).
Graph 5 Do you believe that services will continue to be available for as long as you and your child need them?
Areas for improvement include the recruitment of more professionals, the inclusion of more children, longer duration and increased frequency of services, to secure more tools and equipment for services and to continue the work already started. Respondents also emphasised the need to change the legislative solutions for children and young people with special needs and to work on the reform of the centres for social work.
UNICEF plans to follow-up on the work of the commissions in 2017-2018 as part of its Country Programme commitments to support local level policymaking in the field of social protection and inclusion.
Evaluation question 21. Extent to which the project promoted the strengthening of existing partnerships and the establishment of new ones as well as strengthening inter-sectorial and cross-sectorial cooperation?
Evidence collected throughout the evaluation process shows that investment in the SPI commissions and working groups along with investment in protocols and referral systems promoted and strengthened partnerships among actors in the SPI sectors within the communities. Interviews conducted with members of the SPI commissions across the board in the sampled municipalities confirmed unanimously that the project was an important driver for
20%
40%
09%
02%
17%
01% 07%
04%
I firmly believe
I believe
I do not believe, due to thedepletion of donor funds
I do not believe, because of lack offunds to continue the work
I do not believe, because of budgetconstraints
I do not believe, because of lack ofspace
I am not sure
I do not know
44
building on and deepening the links, lines of cooperation and referral mechanisms. Success in facilitating protocols on cooperation between the relevant government institutions helped forge new partnerships between institutions that previously did not work so closely together. Interviews with relevant stakeholders from all levels of governance confirmed that partnerships and cooperation, particularly in the systems for referral, have increased. However, all interviewed stakeholders agreed that continued partnership, cooperation and exchange would largely depend on personal contact and personal motivation to move things forward.
45
5. Lessons Learned
The evaluation findings and feedback from the interviewed stakeholders within the evaluation process as well as the analysis of the survey results point to a number of emerging good practices and lessons learned.
Investment in strengthening social protection and inclusion at the local governance level remains relevant in Bosnia and Herzegovina. Implementation of the UNICEF Project Component confirmed the relevance of this type of intervention selection at the local level. Interventions to support local commissions for SPI to develop and implement relevant SPI action plans, sign and apply referral protocols and provide social services for the most vulnerable groups of children fit well within the strategic direction of Bosnia and Herzegovina, especially in view of the EU and BiH’s international commitments in relation to the fulfilment of child rights. The project responded to the developmental context and challenges in the target communities where it was active.
Returnee and IDP populations in Bosnia and Herzegovina still need support to ensure their access to rights, particularly the right to basic services such as health, social protection and education. Difficult living conditions caused by poverty and social exclusion create multidimensional challenges especially for children of families already deprived and particularly those affected by developmental challenges or living in remote and excluded rural areas. Multiple studies conducted by UNICEF show that children with special needs face multiple discrimination and that IDP or returnee status creates additional challenges for accessing rights and solving basic social protection needs. The project supported systematic solutions for social protection and inclusion at the local level by which new services and mechanisms have been developed and/or reinforced.
Investment in the empowerment and protection of rights holders remains relevant. The project has enabled the empowerment of children and their families as rights holders by opening access to services and strengthening policies at the local level. Work in this field remains important for ensuring systematic and equitable access to services for children and their families.
An understanding of and approach to the gender dimension of social exclusion is important in projects dealing with social protection and inclusion. While this project has contributed to the empowerment of women its intervention logic and related reports do not follow a gender mainstreaming approach.
46
6. Conclusions
In the focus of this evaluation, the UNICEF intervention is part of a larger project led by UNHCR, funded by the European Union and implemented in cooperation with government counterparts. The aim of the intervention was to enable national actors to address the remaining obstacles to finding solutions for displaced persons and returnees in line with the goals of the Revised Annex VII Implementation Strategy. The UNICEF Project Component falls under Specific Objective 3: To enable vulnerable returnees and IDPs to access their rights and inclusive social services.
The project is relevant to the national priorities for IDPs and returnees and in line with the Revised Strategy as it tackles the remaining obstacles to social protection and inclusion of returnees and IDPs through the institutionalisation of local mechanisms for social protection and inclusion. The established commissions and the working groups within the auspices of the commissions did not only tackle the rights of children but also other populations, which strengthened the response to the obstacles in a more systematic manner. Direct service provision and investment in strengthening the knowledge of professionals addressed the need for improved access and inclusion of marginalised groups, particularly children in need.
The contribution to the envisaged results was positive: contributing to the envisaged outcomes of the UNICEF Project Component as well as the overall project objective. Evidence collected through this evaluation confirmed that the project achieved all of its planned objectives and there is evidence of contributions progress toward the envisaged outcomes. Particularly strong contributions were made in relation to strengthening the systems and capacities to deliver quality inclusive services in local communities. This was especially the case in returnee and IDP areas, most of which are outside the urban centres of the targeted municipalities. Another strong achievement is the fact that most Action Plans for SPI developed by the SPI commissions were evidence based and budgeted either through special budget lines for the Action Plan or mayors’ decisions on funding activities and measures stemming from these documents.
The evaluation found that the utilisation of resources was efficient and contributed to stronger effectiveness of the interventions. The UNICEF team engaged with local actors, particularly members of the SPI commissions and the working groups, on a continuous basis, which was a functional approach that assisted both the efficiency of the project and benefitted local stakeholders. Interviews and focus groups showed that this was a positive driver of the project and the achievements of the commissions. The project incurred significant delays at the onset of implementation yet UNICEF succeeded in overcoming them through intensive work with partners.
The evaluation found evidence of a contribution to the improvement of the social protection and inclusion systems, in the establishment of foundations that will form the basis for systemic change. The establishment of an evidence base on exclusion and the social context in communities as a foundation for action plans was a strong investment in selecting the right priorities and measures for the target groups. The established systems, mechanisms and services already show positive effects on children directly included and benefiting from the innovative approaches and collaborative efforts of partners, while within the scope of the larger project the activities of the working groups have already had a positive impact on the well-selected target groups of final beneficiaries. However, this evaluation was unable to find any evidence of a particular impact of UNICEF’s work on IDP and returnee children.
Overall, the sustainability prospects of the achievements are relatively positive. Policy solutions are in place and represent a positive foundation for sustainable and coordinated
47
service provision for the most vulnerable children. Action plans in most communities are accompanied by budget allocations and this is another important factor of sustainability; however, financial sustainability in the longer term remains an open question. Until such time that solid rights-based foundations for practices and procedures are built and capacities are in place to ensure that these mechanisms run effectively external support will remain crucial for increasing the access and coverage of vulnerable children.
48
7. Recommendations
The findings and conclusions of this evaluation point to the recommendations presented below. The recommendations are based on the findings and conclusions of the evaluation as well as consultations with key stakeholders interviewed during the field phase. Each interview, focus group and discussion group checked the perceptions of various stakeholders concerning the top priorities for SPI reform that need to be addressed in the coming years and consequently the role that UNICEF should play. The recommendation section presents strategic and operational recommendations for UNICEF and the donor. Each recommendation has an addressee and a proposed timeframe. For ease of reference, recommendations are divided into two categories: 1) operational and 2) strategic and programming recommendations.
No Recommendations Addressee Timing
Operational Recommendations (O)
O1 Organise trainings based on the findings of a Training Needs Assessments in order to ensure that the trainings serve their purpose wherein education (and other) professionals acquire practical skills.
The evaluation found that a Training Needs Assessment did not precede the training for education professionals, which points to the concrete need for training to be organised. The training allowed the education professionals (and others) to acquire the theoretical knowledge they required but less practical skills.
This is why UNICEF should conduct a Training Needs Assessments for its training components of future projects, particularly those based on the SPIS model.
UNICEF and partners
2017-2018
O2 Gender should be mainstreamed into programming for UNICEF support for SPI.
Gender equality, the human rights approach and gender mainstreaming are important crosscutting pillars for UNICEF. UNICEF manages to apply the human rights based approach in its programmes yet gender mainstreaming is an area that requires improvement, as evidenced by this evaluation. Effort should be invested to devise a set of measures to mainstream and integrate gender into interventions that focus on SPI created and implemented in cooperation with governments, such as ensuring that the principles of gender equality are adhered to and monitored. All monitoring data should be gender disaggregated in order to provide a clear record of the gender perspective of interventions.
UNICEF 2017-2018
49
No Recommendations Addressee Timing
O3 Strengthen further the monitoring and reporting practices at the programme level and the means to evaluate the impact.
The Theory of Change for this project was developed retrospectively for the purposes of this evaluation. It is much more useful to construct the theory of change at the beginning of a project, because the outcomes and processes are viewed differently with hindsight. A Theory of Change approach can sharpen the planning and implementation of a programme or project. In the design phase, it increases the likelihood that stakeholders will clearly specify the initiative's intended outcomes and outputs, the activities to be implemented to achieve these outcomes and the contextual factors that are likely to influence them. UNICEF should design its Country Programme and associated projects in such a way that allows the impact to be evaluated, with clear baseline indicators and targets disaggregated by gender and with established ‘control’ and ‘treatment’ groups.
UNICEF Up until the end of 2018
Strategic and Programming Recommendations (SP)
SP1 Scale-up the SPI model countrywide to ensure that at least 50% of municipalities apply the model in the improvement of SPI, particularly for children.
UNICEF has been developing and perfecting the SPI model for years in a number of municipalities. Lessons learned from implementation of the UNICEF Project Component confirm that the model brings multiple positive benefits for communities and vulnerable groups. Opportunities should now be explored with donors and government partners to scale-up the model throughout the country, as a systematic investment in SPI at the local level. Taking into account the fact that in the (near) future the EU will not fund these types of activities, ways should be explored to ensure that funding is available to support the target and other municipal administrations to establish and implement the model in at least 50% of BiH municipalities.
UNICEF, government partners and donors (particularly EUD)
2017-2020
SP2 Ensure the continuation of support for IECD and other services established/supported through the project in order to strengthen their mechanisms for sustainability.
IECD and other services established through the project are highly vulnerable in terms of sustainability due to a mix of factors. Further funding opportunities should be explored in order to ensure continuous support for the established services and to advocate for their
Government, with the support of UNICEF and donors
2017 – onwards
50
No Recommendations Addressee Timing
institutionalisation. This is vital as these services provide an innovative caring and valuable space for children, particularly those with disabilities, where they can receive prompt and adequate care and support.
SP3 Advocate for the development of a framework for securing financial sustainability and long-term support and empowerment for the services developed through the project, particularly those targeting children from IDP/returnee families.
It is recommended that UNICEF continue to advocate for more efficient financing of social protection and inclusion services at the local level.
Government in partnership with municipalities, UNICEF and international donors
2017-2020
Annex 1. TERMS OF REFERENCE
FINAL EVALUATION
"Support for durable solutions of Revised Strategy for Implementation of Annex VII of the Dayton Peace Agreement"
-UNICEF supported component-
1. Rationale
The EUR 8.1 million project „Support to durable solutions of the Revised strategy for implementation of Annex VII of the Dayton Peace Agreement “ is funded by the European Union (in the amount of EUR 7 million through IPA 2012), co-funded (with EUR 1.1 million) and implemented by UNHCR in cooperation with seven project partners, including UNICEF, in ten priority municipalities. The purpose of the project is to assist at least 2.400 vulnerable displaced persons and returnees in Bosnia and Herzegovina, including women victims of war, in finding solutions to social challenges that they are facing every day, through close cooperation between local authorities, local community, civil society and project partners, together with the Ministry for Human Rights and Refugees of Bosnia and Herzegovina.
There are around 85,000 internally displaced persons (IDPs) and more than 470,000 minority returnees, currently living in BiH. In mid-2010, the Parliamentary Assembly of Bosnia and Herzegovina adopted the Revised Strategy for the Implementation of Annex VII of the Dayton Peace Agreement, in order to speed up the process of identifying durable solutions for refugees, internally displaced persons and returnees. However, according to the survey conducted by the United Nations High Commissioner for Refugees (UNHCR), in cooperation with the Union for Sustainable Return and Integration (UZOPI), during 2013 and 2014, a significant number of IDPs and returnees still live in unacceptable conditions.
In 2013, the authorities in Bosnia and Herzegovina and international partners recognized the importance of this issue and signed a joint statement, announcing further operationalization of the Annex VII Revised Strategy. In 2014, UNHCR has accordingly launched the project called ‘Support to durable solutions under the Revised Strategy for Implementation of Annex VII of the Dayton Peace Agreement’.
The main project activities are carried out in 10 BiH municipalities: Bijeljina, Bosanski Petrovac, Derventa, Foča, Gradiška, Maglaj, Mostar, Prijedor, Živinice and Brčko District. The main goal of the project is to ensure durable solutions for the most vulnerable families of IDPs, minority returnees and women victims of war, through a teamwork and cooperation among local authorities, local communities, civil society sector and project partners.
UNHCR entrusted the implementation of the project to partners who have years of experience in implementing similar activities. These are: the United Nations Development Fund (UNDP), United Nations Children's Fund (UNICEF), International Organization for Migration (IOM), Hilfswerk Austria International (HWA), Bosnian Humanitarian Logistics Service (BHLS), Foundation of Local Democracy (FLD) and Vaša Prava BiH (VP).
52
UNICEF is focusing on supporting the Municipalities in the development of Action Plans for social protection and inclusion (SPI), based on detailed vulnerability assessments. The vulnerability assessments represent the situation of vulnerable groups in each Municipality, with a focus on returnees and IDPs, in the areas of education, health, social welfare, protection and justice. The project established Commissions for social protection and inclusion in each Municipality, with an aim to promote and strengthen SPI governance and inter-sectorial cooperation. In addition, UNICEF supported local Governments to ensure the provision of SPI services for more than 300 vulnerable children and families and strengthened capacities of more than 200 professionals from social, education and health sectors.
Within this Project, UNICEF does not support the activities only for children from IDP and returnee families, but it contributes to creating the favourable environment required to enable all vulnerable children and their families to enjoy the social inclusion. This is the only way to establish a sustainable, long-term support to all vulnerable children and ensure better social inclusion and cohesion.
2. Objective
The main objective of the assignment is to conduct a final evaluation of the project „Support to durable solutions of the Revised strategy for implementation of Annex VII of the Dayton Peace Agreement” (the Project), in order to evaluate the final (end) results and achievements of the UNICEF supported component of the project.
More specifically the evaluation objectives are to:
1. Provide feedback to UNICEF BiH office, UNHCR and national counterparts on the soundness (defined as relevance, effectiveness, efficiency, sustainability) and impact of the Project approach in strengthening the social protection and inclusion for vulnerable families and children, in particular for IDPs and returnees, with the aim to:
a. Reveal good practices and its implementation modalities in with particular focus on local levels of governments
b. Identify systemic gaps in application of SPI approaches at local levels
c. Evaluate Project Impact following Description of the Action and Logical Framework
2. Extract lessons learned and recommendations aimed at further enhancement of the Social Protection and Inclusion system in BiH at local level based on the experiences from the Project implementation.
3. Document (in a form of a separate brief) how the Social Protection and Inclusion methodology serves the purpose of targeting specific needs of IDP and returnee population group? What are the benefits of the SPI methodology for this specific target group and how it can be further improved to better serve its purpose? The brief should include specific examples from selected target Municipalities.
53
4. Document (in a form of a separate situation brief) the remaining gaps and bottlenecks for full integration of returnee and IDPs children into society as well as the recommended programmatic approach to address those issues.
3. Scope
The Project evaluation should cover the entire project implementation period (15 June 2014 – 31 December 2016).
Description of the Action (DoA) and Logical Framework (LF) should be used as the main reference point the assess the project, because they capture relevant information. Since the Project had no Theory of Change (ToC), it is expected that a retrospective ToC be created by the evaluator.
4. FINAL EVALUATION QUESTIONS
The evaluation will look at the relevance, effectiveness, efficiency, impact and sustainability of the project (OECD-DAC evaluation criteria as the humanitarian ones are not relevant for this task). The evaluation will specifically address the following categories of questions with respect to UNICEF’s contribution to system level changes.
1) Relevance: To what extent is the Project responding to the needs of beneficiaries?
To what extent does the Project respond to the needs of IDP and returnee population?
To what extent is the Project design relevant in regard to the overall Project goal and the achievement of its objectives in the given period of time?
To what extent are the Project design and its objectives relevant in regard to national policies and strategies?
Was the Project designed according to international norms and agreements on Human Rights (HR) and Gender Equality (GE) and in line with national strategies to advance HR & GE?
Optional - What does the literature and current experience suggest about the appropriateness of the current or proposed strategy? If successfully implemented would this strategy be likely to address the key issues affecting most marginalized groups?
2) Effectiveness: To what extent does the Project meet the outcomes as defined by the DoA and LF?
To what extent have the planned results been achieved to date (quantitative and qualitative)?
To what extent and how has the project contributed to the enhancement of systems and capacities to deliver inclusive services at the local level, with a focus on IDP and returnee population?
To what extent has the Project contributed to raising awareness on social inclusion of IDP and returnee population?
To what extent has the Project contributed to strengthening capacities of service providers to provide quality assistance to target groups?
What were the major factors influencing the achievement or non-achievement of the Project objectives to date?
Has the project provided any additional (not directly planned by the Project) significant contribution/outcomes towards development of services and social
54
inclusion of most vulnerable and excluded children, with a focus on children from IDP/returnee population?
3) Efficiency: To what extent did the management of the Project ensure timelines and efficient utilization of resources?
How well have the implementation of activities been managed? To what extent were activities implemented as scheduled? What management and monitoring tools have been used?
How well have the financial resources been used / were funds managed in a cost-effective manner / what is the correlation between funds utilized and outputs / results achieved / could the same results be achieved with less resources?
Did the project ensure co-ordination with other implementing partners to encourage synergy and avoid overlaps?
How flexible was the Project design in adapting to the changing environment (impact of the recent flood, etc.)?
To what extent are HR & GE a priority in the overall Project budget and implementation?
4) Impact: To what extent has the Project improved the social protection and inclusion systems in target Municipalities?
To what extent has the project contributed to the promotion of multi sectorial approach to SPI through establishment of protocols of cooperation and referral mechanisms in targeted municipalities?
What were the results in children’s lives of the interventions - intended and unintended, positive and negative - including the social, economic and environmental effects on targeted groups?
Were there any unintended results on human rights & gender equality in the intervention? Were they positive or negative and in which ways did they affect the different stakeholders?
5) Sustainability: To what extent are the Project outcomes achieved sustainable? Sustainability looks to the probability of continued long-term benefits to most marginalized groups.
Did the Project design include an appropriate sustainability strategy (including promoting local ownership, use of local capacity, etc.) to support positive changes for the most vulnerable groups after the end of the intervention?
To what extent are new knowledge and skills integrated into regular activities of professionals working with children?
To what extent has the Project promoted strengthening of already existing partnerships and establishment of new ones and to strengthening of inter-sectoral and cross-sectoral cooperation?
5. EXISTING INFORMATION SOURCES
Initial documents and analysis to be taken in consideration:
Project DoA and SOPs
ROM report 2016
SPI Manual
Municipal SPI Action Plans 2016 – 2017 and monitoring matrices
SPI Commissions and SPI working groups ToRs; relevant MoVs
55
SPIS evaluation reports (optional)
Other relevant project documents
6. METHODOLOGY
The evaluation methodology will be guided by the Norms and Standards of the United Nation
Evaluation Group (UNEG).
(http://www.uneval.org/normsandstandards/indexs.isp?doc_cat_source_id=4)10.
The contractor is expected to observe the UNEG ethical guidance to evaluation as guiding principle
to ensure quality of evaluation process.
(http://www.uneval.org/search/index.jsp?q=ETHICAL+GUIDELINES)
Based on the OECD/DAC questions above listed and on the UNDG Evaluation Guidelines, the Consultant is expected to develop a detailed research design and methodology prior to the evaluation (Inception Report), which should be approved by the UNICEF team.
The Final evaluation will be based on primary and secondary data and information collected at national/entity/district and local level. Information collected will be explicitly triangulated.
The evaluation process will be based on:
Desk reviews and analyses of existing project related documents
Analysis of existing national/entity/district/local policy and planning documents
Analysis of primary and secondary data
Data collection instruments, such as, questionnaires
Field visits to the target municipalities (structured observation and/or focus groups discussions with service providers)
Interview of key stakeholders (structured interviews with government and non-governmental sector)
7. DELIVERABLES, WORK PLAN and EVALUATION MANAGEMENT
The final evaluation report should follow UNEG Norms and Standards, UNICEF Evaluation Report Standards and should follow the GEROS Quality Assessment System.
10 UNEG Norms: http://www.uneval.org/indexAction.cfm?module=Library&action=GetFile&DocumentAttachmentID=1491 UNEG Standards: http://www.uneval.org/indexAction.cfm?module=Library&action=GetFile&DocumentAttachmentID=1496
56
The consultant is responsible for:
Inception report (5 working days);
Field work (10 working days);
The Draft Evaluation Report (10 working days);
The Final Evaluation Report, which should include executive summary, description of sampling and evaluation methodology, data collection instruments, types of data analysis, assessment of methodology (including limitations), findings, conclusions, recommendations, lessons learned, attachments with developed list of indicators and questionnaires (6 working days);
A brief on how the Social Protection and Inclusion methodology serves the purpose of targeting specific needs of IDP and returnee population group (5 working days);
A brief on remaining gaps and bottlenecks for the integration of children from returnee/IDP population group (5 working days);
The Final evaluation report will be assessed and rated against the assessment tool.
The reports should be provided in electronic form in English in the required UNICEF format. Completed data sets are to be submitted to UNICEF (filled out questionnaires, records of individual interviews and focus group discussion, etc.). Contractor should be sensitive to beliefs and act with integrity and respect to all stakeholders. In the report evaluators may not refer to individual children. Contractor may not share findings with media in BiH or abroad concerning individual children or individual institutions.
UNICEF premises will be available during the time spend in BiH if needed. Printers, photocopying services, and other similar services will be provided by UNICEF. It is expected that contractor will bring his/her own laptop.
Management: The evaluation will be managed by the UNICEF Social Protection Specialist, in close cooperation with Child Rights Monitoring and Evaluation Specialist as well as Education Officer, C4D Officer, ECD& Health Officer, Child Protection Officer and Communication Specialist. The contractor will be supervised by Social Protection Specialist. The management of the evaluation includes the development of the terms of reference, assignment of the evaluation consultant, liaison between the consultant and partners/stakeholders involved (supporting organization of meetings/ interviews and field visits), as well as quality assurance of the report.
Organization: Individual consultancy is required for this evaluation.
Annex 2. Evaluation Matrix
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
RELEVANCE - to what extent is the Project responding to the needs of stakeholders and beneficiaries?
EQ 1 To what extent does the Project respond to the needs of IDP and returnee population?
Project design founded on the needs assessments, social exclusion and poverty data relating to IDP and returnee population.
Alignment of the project intervention with needs and priorities identified in national and local strategies aimed to guide and advance social services and other social protection and inclusion measures for IDPs and returnee population
Evidence of consistency between needs and priorities for social protection and inclusion of IDP and returnee population and the strategy/approach developed by the project
National and local reports, research studies
Interviews with key stakeholders
Project reports
EQ 2 To what extent is the Project design relevant in regard to the overall Project goal and the achievement of its objectives in the given period of time?
The Project activities and outputs respond to the objectives of the project as assessed by:
linkages between
The project has a well-defined intervention logic, demonstrating how the outputs will produce the intended results
project activities in combination will deliver the projected outputs / results
Project document
Project reports
58
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
outputs/results
linkages between outputs and outcomes
influence of external factors
assumptions and risks
project outputs in combination are sufficient to produce outcomes, given the identified assumptions
EQ 3 To what extent are the Project design and its objectives relevant in regard to national policies and strategies?
Extent to which the project activities are in line with overarching objectives and priorities for improvement of social protection and inclusion of children in Bosnia and Herzegovina
Project activities are in line with international and European commitments and needs of Bosnia and Herzegovina
Evidence of consistency between needs and priorities for creating the favourable environment required to enable all vulnerable children and their families to enjoy the social inclusion and the content of deliverables of the project
National and local reports, research studies
Interviews with key stakeholders
Project reports
EQ 4 Was the Project designed according to international norms and agreements on Human Rights (HR) and Gender Equality (GE) and in line with
Extent to which the project interventions are based on and integrating international agreements on
The project design reflecting the standards and norms of HR and GE
Evidence of consistency between strategies to advance
National and local reports, research studies
Interviews with key stakeholders
59
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
national strategies to advance HR & GE?
human rights and gender equality
Extent to which the project interventions are based on and integrating national strategies to advance human rights and gender equality
HR and GE and the strategy/approach developed by the project
Project reports
EQ 5 Optional - What does the literature and current experience suggest about the appropriateness of the current or proposed strategy? If successfully implemented would this strategy be likely to address the key issues affecting most marginalized groups?
EFFECTIVENESS: To what extent does the Project meet the outcomes as defined by the DoA and LF?
EQ 6 To what extent have the planned results been achieved to date (quantitative and qualitative)?
The project produced the planned outputs
The outputs produced the intended results (quantitative and qualitative)
Intended results (i) have been achieved, (ii) have been partially achieved (in which areas) or (iii) have not been achieved
% outputs and results achieved (indicators)
Quality of outputs and results
Evidence and examples of high/poor effectiveness
Examples of factors which contributed or hampered the effective achievement of outputs and results
Examples of where final beneficiaries have taken up/used the outputs made available
Project reports (annual and monitoring)
Interviews with stakeholders, discussion groups, mini survey
Site visits to a selected number of municipalities, including interviews/ group discussions with end beneficiaries to the extent possible
EQ 7 To what extent and how has the project contributed to the enhancement of systems and capacities to deliver inclusive services at the
The extent to which the project contributed to strengthening the Social Protection and Inclusion systems at
Evidence and examples of new/updated Social Protection and Inclusion systems in targeted communities
No. and quality of modalities of inclusive services at the local
Strategies and legislation in the Social protection and inclusion thematic area in Bosnia and Herzegovina
60
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
local level, with a focus on IDP and returnee population?
local level
Quality of models of inclusive services at the local level developed with the project support
The extent to which the project contributed to the mobilisation and capacity building of selected municipalities for delivery of inclusive services at the local level, with a focus on on IDP and returnee population
level, with a focus on IDP and returnee population
Evidence and examples of municipal initiatives for delivery of inclusive services at the local level
Evidence of new decisions put to practice in order to ensure delivery of inclusive services at the local level
Quality of models observed; no major shortcomings identified
Examples of successful/ unsuccessful results of interventions at local level
National and local reports, research studies
Interviews with key stakeholders
Project reports
Site observations
Mini survey
EQ 8 To what extent has the Project contributed to raising awareness on social inclusion of IDP and returnee population?
The extent to which the project contributed to raising awareness on on social inclusion of IDP and returnee population
Evidence of raised awareness on social inclusion of IDP and returnee population
Project reports
Site visits to a selected communities
Interviews with key stakeholders
Group discussions
Mini survey
61
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
EQ 9 To what extent has the Project contributed to strengthening capacities of service providers to provide quality assistance to target groups?
The extent to which the project contributed to the mobilisation and capacity building of service providers to provide quality assistance to the target groups
Evidence and examples of municipal initiatives for improvement of provision of quality assistance to the target groups
Evidence and examples of quality services to target groups
Project reports (yearly, monitoring)
Municipal decisions/regulations
Site visits
Interviews with key stakeholders and discussion groups
Mini survey
Site observations
EQ 10 What were the major factors influencing the achievement or non-achievement of the Project objectives to date?
Extent to which external factors affect the operations of the Project
Extent to which the risk mitigation strategy was effective in ensuring project results are achieved
Evidence of external factors and their effects on the Project operations
Evidence of successful mitigation strategies for risks and assumptions
Project reports (yearly, monitoring)
Site visits
Interviews with key stakeholders
EQ 11 Has the project provided any additional (not directly planned by the Project) significant contribution/outcomes towards development of services and social inclusion of most vulnerable and excluded children, with a focus on
Identification and assessment of additional (planned and unplanned) outcomes
The identified additional outcomes are (not) classified into positive or negative
Evidence through examples of additional outcomes and their appraisal
Effects (positive or negative) of identified outcomes
Government policy reviews and reports on social protection and inclusion of children
Research studies and reports developed within the project
Site visits, site observations
62
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
children from IDP/returnee population?
Interviews with key stakeholders and discussion groups
Mini survey
EFFICIENCY: To what extent did the management of the Project ensure timelines and efficient utilization of resources?
EQ 12 How well have the implementation of activities been managed? To what extent were activities implemented as scheduled? What management and monitoring tools have been used?
Management of the project ensured timeliness and efficient use of resources
Chosen management and implementation modalities are in line with best practices of other UNICEF or donors’ interventions
Evidence that chosen management modalities provided for needed efficiency, timely delivery and adaptation/flexibility in project implementation
Examples of management intervention for overcoming barriers and constraints in project implementation
Project reports (annual, monitoring)
Interviews with stakeholders and beneficiaries
Site visits to selected projects
EQ 13 How well have the financial resources been used / were funds managed in a cost-effective manner / what is the correlation between funds utilized and outputs / results achieved / could the same results be achieved with less resources?
Financial and human resources spent for the achievement of outputs and results
Results could have been achieved at a lower cost (or not)
Same / better results could have been achieved (or not) at same cost using other means
Examples of project activities with a good/poor cost-effectiveness level
Examples of alternative ways of minimising costs of achieving the same or better outcomes
Project reports (annual, monitoring)
Interviews with stakeholders
EQ 14 Did the project ensure co-ordination with other similar interventions to encourage synergy and avoid overlaps?
Judgement will be based on the examination of: o Complementarity
with other projects
Coherence between the project and similar interventions’ objectives; co-ordinated implementation schedules
Demonstrable effects of
Projects documentation
Interviews with key informants
63
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
o Functioning donor coordination and consultation processes with stakeholders and beneficiaries
complementarity or/and overlaps, both upstream on the level of donor coordination and downstream on project implementation level
Minutes of coordination meetings (if available)
EQ 15 How flexible was the Project design in adapting to the changing environment (impact of the recent flood, etc.)?
The extent to which the Project design was flexible in adapting to the changing environment (impact of the recent flood, etc.)?
Evidence of mitigation measures to adapt to the changing environment (impact of the recent flood, etc.)
Projects documentation
Interviews with key informants
EQ 16 To what extent are HR & GE a priority in the overall Project budget and implementation?
The extent to which the HR&GE is been priorities in the overall budget and implementation.
Evidence and examples of prioritisation of HR&GE in the overall budget and implementation
Projects documentation
Interviews with key informants
IMPACT: To what extent has the Project improved the social protection and inclusion systems in target Municipalities?
EQ 17 To what extent has the project contributed to the promotion of multi sectorial approach to SPI through establishment of protocols of cooperation and referral mechanisms in targeted municipalities?
Contribution of the project to the increase of children from target groups benefiting from quality services, cooperation and referral mechanisms in targeted municipalities
Services are used by families from vulnerable groups
Quantitative evidence that the project made a visible contribution to referral mechanisms in targeted municipalities
Evidence of beneficiaries’ increased use of services
Project documentation
National statistics and reports
Reports of international organisations (EC, CRC, WB, etc.)
Research studies and assessments
Interviews with key stakeholders
64
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
Site visits and focus groups, Mini survey
EQ 18 What were the results in children’s lives of the interventions - intended and unintended, positive and negative - including the social, economic and environmental effects on targeted groups?
New measures and services developed with support of the project have positive (or negative) - - including the social, economic and environmental - effects in terms of benefits for target groups
Identified systemic barriers (administrative, institutional, financial, human resources, etc.) which reduce the identified impact of the project
Evidence and examples of positive effects and benefits of measures and services developed with support of the project for children from target groups (skills, abilities and knowledge)
Factors reducing the impact of projects (external and internal to the management of the project)
Project documentation
Research studies and assessments
Interviews with key stakeholders
Feedback from discussion groups
Mini survey
EQ 19 Were there any unintended results on human rights & gender equality in the intervention? Were they positive or negative and in which ways did they affect the different stakeholders?
Identification and assessment of unintended (positive/negative) results on human rights & gender equality in the intervention
Identification and assessment of
Evidence through examples of intended or unintended human rights & gender equality in the intervention
Project documentation
Research studies and assessments
Interviews with key stakeholders
Feedback from discussion groups
65
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
positive or negative results and they effect on the different stakeholders
Mini survey
SUSTAINABILITY: To what extent are the Project outcomes achieved sustainable? Sustainability looks to the probability of continued long-term benefits to most marginalized groups
EQ 20 Did the Project design include an appropriate sustainability strategy (including promoting national/local ownership, use of local capacity, etc.) to support positive changes for the most vulnerable groups after the end of the intervention?
Project design includes an appropriate sustainability strategy (including promoting national/local ownership, use of local capacity, etc.) to support positive changes for the most vulnerable groups after the end of the intervention
Evidence of existence of an appropriate sustainability strategy (including promoting national/local ownership, use of local capacity, etc.) to support positive changes for the most vulnerable groups after the end of the intervention
Project documentation
EQ 21 To what extent are new knowledge and skills integrated into regular activities of professionals working with children?
The extent to which professionals integrate their newly acquired knowledge into regular activities to be judged by: o Extent to which
new approaches are integrated in the regular activities of professionals
o Further staff development planning, based on
No. of professionals applying new knowledge and skills in their regular activities, measured through extent of integration of new methodologies in their work practice
Evidence of staff development plans, based on capacity building packages developed by the project, for keeping abreast with professional challenges
Project documentation
Analytical reports by government and independent experts
Interviews with key stakeholders
Site visits and feedback from professionals
66
Evaluation Questions (EQ) Judgement Criteria Indicators Sources of information
capacity building packages developed by the project, for keeping abreast with professional challenges
EQ 22 To what extent has the Project promoted strengthening of already existing partnerships and establishment of new ones and to strengthening of inter-sectoral and cross-sectoral cooperation?
Extent to which the Project promoted strengthening of already existing partnerships and establishment of new ones
Extent to which the Project promoted strengthening of inter-sectoral and cross-sectoral cooperation
Evidence through examples of already existing partnerships strengthened and new ones established
Evidence through examples of strengthened inter-sectoral and cross-sectoral cooperation
Project documentation
Analytical reports by government and independent experts
Interviews with key stakeholders
Site visits
Group discussions with professionals
Annex 3. Logframe of the Overall Action
LOGICAL FRAMEWORK FOR THE PROJECT
Intervention logic Objectively verifiable
indicators of achievement
Sources and means of verification
Assumptions
Overall objectives
O1: To enable national actors to address the remaining obstacles to solutions for displaced persons and returnees in accordance with the Revised Annex VII DPA Implementation Strategy goals.
Progress Report on implementation of the Revised Annex VII DPA Strategy goals
Specific objective
SO1: To provide durable housing solutions to vulnerable returnee and IDP families
SO2: To provide sustainable economic opportunities to returnee and IDP families
SO3: To enable vulnerable returnees and IDPs to access their rights and inclusive social
SOI1: 125 families moved into their homes
SOI2: 480 families generating additional or new income
SOI3: At least 500 persons provided with access to rights and/or access to inclusive social services
- Project implementation reports, field assessments
- Competent authorities’ reports - Other relevant independent
agencies’ reports
- Project implementation reports, field assessments
- Competent authorities’ reports
- Project implementation reports, field assessments
- Competent authorities’ reports
Stable political and security environment in BiH
Continuous support of major stakeholders for the implementation of the Revised Strategy for the implementation of Annex VII
68
services - CSOs reports on human rights - Other relevant independent
agencies’ reports
Expected results
R1: Prioritised housing units constructed / reconstructed
R2: Constructed / reconstructed housing units connected to public utilities
R3: Strengthened value chains to support economic integration of returnees and IDPs
R4: Livelihood support secured for vulnerable returnees and IDPs
R5: Strengthened social
I1.1: At least 60 housing units constructed and at least 65 housing units reconstructed or rehabilitated, with provision of full ownership rights or security of tenure in case of social housing
I2.1: 125 housing units connected to public utilities
I3.1: Increased productivity in selected value chains by 5% (baseline: initial value chains analysis)
I3.2: Average gross revenue per unit of production increased by 5% in selected value chains (baseline: initial value chains analysis)
I4.1: 480 families received basic economic inputs to aid income-generation
I1.1.a: Monitoring and progress reports of implementing partners
I1.1.b: Municipal Commissions’ handover reports
I2.1.a: Monitoring and progress reports of competent authorities, implementing partners
I2.1.b: Signed contracts with utility companies
I3.1: Assessment of a representative sample of farmers
I3.2: Assessment of a representative sample of farmers
I4.1.a: Monitoring and progress reports of implementing partners
I4.1.b: Signed assistance receipts and contracts between municipal authorities and beneficiaries
Stable political and security environment in targeted localities
Coordination and cooperation among all relevant stakeholders at state, entity/cantonal and municipal levels, including civil society participation
Conducive legal and policy environment at higher government levels, in particular in relation to agriculture/rural development and social inclusion
69
protection and inclusion systems in targeted localities, with a particular focus on vulnerable returnees and IDPs
R6: Enhanced capacities to deliver inclusive services to identified most vulnerable returnees and IDPs in targeted localities
R7: Free legal aid provided to returnees and IDPs in targeted localities, and in other locations selected by UNHCR
R8: Awareness raised and positive attitude and behaviour change initiated regarding
I5.1: Social Protection and Inclusion Commissions established in at least 10 municipalities
I5.2: Evidence-based Action Plans on Social Protection and Inclusion developed in at least 10 municipalities
I6.1: At least 300 vulnerable children and caregivers provided with access to inclusive social services (education, health, social protection, child protection)
I6.2: At least 200 vulnerable elderly returnees and IDPs provided with access to inclusive social services (psychosocial support, health care to elderly)
I7.1: At least 500 individuals in targeted municipalities receive legal aid to facilitate access to rights, and at least 300 more in other locations selected by UNHCR.
I8.1: 20% increase in positive attitudes towards
I5.1: Municipal Councils/Assemblies decisions on the establishment of Commissions
I5.2: Action Plan documents
I6.1: Monitoring reports
I6.2: Monitoring reports
I7.1.a: Monitoring and progress reports of implementing partners
I7.1.b: Partner’s data base
Readiness to support change towards inclusion of returnees and IDPs in targeted localities
70
returnees and IDPs in targeted localities
returnees and IDPs in targeted localities
I8.1: Knowledge, attitude and practice baseline and evaluation survey
Activities
A1.1. Construction, reconstruction and / or rehabilitate homes for vulnerable IDP and returnee families in prioritised municipalities
A1.1.1 Preparatory activities
A1.1.1.1 Establish Project Governance Structure
A1.1.1.2 Development / updating of Methodology / Standard Operating Procedures documents and information session s to municipal staff; dissemination of the documents among other stakeholders
A1.1.1.3 Public information and media promotion
A1.1.2. Selection of project beneficiaries
A1.1.2.1 Selection of
Means:
A1 and A2
UNHCR/Implementing partners
Human Resources: Project Manager, 3 Technical Coordinators/Engineer Supervisors (70%), Desk Officer (17%), Coordinator of Media Campaign (part time 20%), FLD Coordinator for Expert Working Group (EWG) (33%)
Other: travel, vehicle rental, equipment and supplies, fees of architect, architect’s assistant and electrical engineer , construction costs, infrastructure connection costs, cost of press conferences and visibility materials, cost of producing publications,
Costs:
Total project value:
EUR 8,107,500
UNHCR contribution:
EUR 1,107,500
Requested EU contribution:
EUR 7,000,000
Total cost breakdown:
Human resources:
EUR 1,549383.13
Travel:
EUR 8,050
- Partnership arrangements function accordingly
- Readiness of local authorities to cooperate
- Participation of IDPs and returnees in community-based programme interventions
71
beneficiaries using evidence-based data
A1.1.2.2 Selection of beneficiaries through municipal calls
A1.1.2.3 Assistance to beneficiaries in the application process
A1.1.2.4 Evaluation of applications and final selection of housing beneficiaries
A1.1.2.5 Signing of Tripartite Agreements (TPAs)
A1.1.3 Construction, reconstruction and rehabilitation of dwellings and connection of dwellings to public utilities
A1.1.3.1 Final definition and provision of building plots, preliminary and final (re)construction designs and building permits
A1.1.3.2 Preparation of tender documents and call
costs of meetings/training sessions and fees for trainers and consultants
Equipment and supplies:
EUR 7,491.98
Local office:
EUR 127,555.77
Other costs and services:
EUR 344,434.45
Other:
EUR 5,540,187.46
Indirect costs (7%):
EUR 530,397.20
72
for bids from design companies
A1.1.3.3 Evaluation of bids, selection of sub-contractors and contracting (design companies)
A1.1.3.4 Preparation of project documents (housing)
A1.1.3.5 Preparation of tender documents
A1.1.3.6 Launch tender calls for (re)construction and official technical inspection
A1.1.3.7 Evaluation of bids and selection of sub-contractors for (re)construction and official technical inspection;
A1.1.3.8 Contracting, start of works and supervision (technical acceptance optional)
A1.1.3.9 Hand over of housing units to beneficiaries
Cost breakdown by Agency:
UNHCR
Human resources: EUR 1,294,43.13
Travel: EUR 600.00
Equipment and supplies: EUR 7,491.98
Local office: EUR 72,085.77
Other costs and services:
EUR 177,853.60
Other: EUR 4,199,148.32
Indirect costs (7%): EUR 402,641.60
UNDP
Human resources: EUR 130,240.00
Travel: EUR 3,850.00 EUR
Equipment and supplies: 0
73
A2.1 Connection of constructed, reconstructed or rehabilitated homes to public utilities
A2.1.1 Preparation of tender documents and call for bids for the selection of design companies
A2.1.2. Evaluation of bids, selection of sub-contractors and contracting (design companies)
A2.1.3. Preparation of technical specifications
A2.1.4 Preparation of tender documents
A2.1.5 Launch tender calls for construction/connection and official technical inspection
A2.1.6 Evaluation of bids and selection of sub-contractors for works and official technical inspection
A2.1.7 Contracting, start of works and supervision
A2.1.8 Technical acceptance and handing over of infrastructure to beneficiaries
Local office: EUR 33,990.00
Other costs and services:
EUR 44,980.86
Other: EUR 704,379.14
Indirect costs (7%): EUR 62,839.00
UNICEF
Human resources: EUR 130,240.00
Travel: 0
Equipment and supplies: 0
Local office: EUR 18,000.00
Other costs and services:
EUR 121,600.00
Other: EUR 618,060.00
Indirect costs (7%): EUR 62,153.00
IOM:
74
A3.1. Value chain analysis in 5 targeted areas
A3.1.1. Map high value SME prospects in selected areas
A3.1.2. Select high priority value chains
A3.1.3. Define individual projects and targeted beneficiaries based on value chains methodology
A3.2. Provide assistance to farmers for income generation and self-employment
A3.2.1. Create new production units for income generation
A3.2.2. Support farm production through technical assistance and training
A3.2.3. Improve food safety and product standardisation through certification of SMEs and farmers for GlobalGAP, HACCP and other standards
A3.3. Horizontal strengthening of supply chains
A3.3.1. Local sourcing analysis
A3
UNDP
Human resources: Value Chains Expert
Other: transportation, office, extension/advisory services, studies, agricultural infrastructure, agricultural inputs
Human resources: EUR 13,800.00
Travel: EUR 3,600.00
Equipment and supplies: 0
Local office: EUR 3,480
Other costs and services: 0
Other: EUR 18,600.00
Indirect costs (7%): EUR 2,763.60
- Significant number of SMEs operating in the selected localities
- Local governments’ support to the private sector development and economic recovery
75
and link of the primary producers into new or existing value chains
A3.3.2. Strengthen business infrastructure and logistics to enhance SME competitiveness
A3.4. Support access to market
A3.4.1. Improve foreign trade skills of selected SMEs in targeted areas
A3.4.2. Develop promotional material for selected export-oriented SMEs in targeted areas
A3.4.3 Organize promotional trips (trade shows and fairs)
A4.1. Provide income-generation grants to vulnerable IDP and returnee families
A4.1.1 Preparatory activities
A4.1.1.1 Establish Projcect Governance Structure
A4.1.1.2 Developing / updating of Methodology/ Standard Operating Procedures documents
A4
UNHCR/Implementing partners
Human Resources: Engineer of Technology/Agronomist, Sustainability Coordinator IDP, Sustainability Coordinator RET, Administration Officer,
76
and information sessions to municipal ; dissemination of the documents among other stakeholders
A4.1.1.3 Public information and media promotion
A4.1.2 Select economic-sustainability (grant) recipients
A4.1.2.1 Economic analyses of housing (and RHP) beneficiaries
A4.1.2.2 Pre-selection and final selection of economic beneficiaries
A4.1.3. Provision of economic grants
A4.1.3.1 Creation of final grants list
A4.1.3.2 Provision of grants
A4.1.3.3 Delivery of grants
A4.1.4. Business advisory and consulting
A4.1.4.1 Pre-granting “case-by-case” advisory
Assistant Project Manager, Office Clerk/Cleaner, Project Manager, Finance Manager Mission BiH average 30% (40% Y1, 25% Y2, 25% Y3), Project/Admin Assistant 50%, Finance Assistant 50%, Country Director 30%, Coordinator of Media Campaign (part time 60%), FLD Coordinator for Expert Working Group (EWG) (33%),
Other: travel, vehicle costs, office rental and supplies, grants for income generation and training, cost of press conferences and visibility materials, postal fees for RHP beneficiaries, cost of producing publications, costs of meetings/training sessions and fees for trainers and consultants
- Partnership arrangements function accordingly
- Readiness of local authorities to cooperate
- Active participation of IDPs and returnees in community-based programme interventions
77
A4.1.4.2 Post-granting business advisory and consulting
A4.1.4.3 Advisory-based monitoring of use of grants
A5.1. Establish and develop the capacity of Social Protection and Inclusion Commissions in at least 10 municipalities
A5.1.1. Establish Social Protection and Inclusion Commissions in at least 10 municipalities
A5.1.2. Develop capacity of members of SPI Commissions
A5.2. Conduct an assessment to identify vulnerable returnees and IDPs and their needs in in line with the obstacles to solutions as defined in the Revised Strategy for
A5, A6.1, 6.2 and A.8
UNICEF
Human resources: C4D Officer (38.7%), Project Assistant (45.3%), Driver
78
the Implementation of Annex VII in at least 10 municipalities
A5.2.1. Develop capacity to conduct situation analysis
A5.2.2. Situation analysis in the areas of education, health, social welfare, protection and justice at the municipal level, with a focus on the most vulnerable returnees and IDPs
A5.3. Develop evidence-based Action Plans on Social Protection and Inclusion in at least 10 municipalities
A5.3.1. Development of municipal action plans to respond to the needs identified through the assessment, with a focus on most vulnerable groups
A5.3.2. Adoption of Social Protection and Inclusion Municipal Action Plans by the Municipal Councils, in at least 10 municipalities
A6.1: Develop the capacity of service providers from the education, health and social welfare sectors to provide
(45.3%), M&E Specialist (33.33%)
Other: publications, surveys and assessments, training workshops, round tables, Consultants (SPI, C4D), grants, translation, travel, utilities, office supplies, visibility
A6.3 and 6.4
79
inclusive services to vulnerable children and their families
A6.1.1. Capacity development in social protection and inclusion systems
A6.1.2. Capacity development in child protection and case management
A6.1.3 Capacity development in inclusive education
A6.1.4 Capacity development in health, Early Childhood Development and Early detection of developmental delays
A6.2: Support service providers in at least 10 municipalities to ensure access to inclusive social services to vulnerable children and families (R6)
A.6.2.1. Strengthen networks and referral mechanisms
A.6.2.2. Support the provision of services to the vulnerable returnees and IDPs, with a focus on children and families
A6.3. Undertake needs
IOM
Human resources: 1 Project Coordinator Assistant (66.67%); 1 Administrative and Finance Assistant (13.33%);
Other: transportation, office, Group Psychosocial Workshops Consultant for Needs Assessment of healthcare in rural areas; Consultant for psychosocial needs assessment; Psychologist
80
assessment in ten pilot municipalities to identify home health support needs of elderly and disabled IDPs and returnees and required capacity building of municipal health services
A6.3.1. Undertake assessments to ensure full awareness and familiarity with the most pertinent health issues within ten targeted municipalities;
A6.3.2. Conduct assessments of municipal/cantonal service providers in order to identify gaps and define needs for a wider intervention;
A6.3.3. Based on assessments, design methodology and strategy to build the capacities of municipal/cantonal service providers to deliver home health support.
A6.4: Provide psychosocial support in targeted localities to especially vulnerable returnees and IDPs and implement capacity building of Centres for Social Work in two pilot municipalities
A6.4.1. Undertake psychosocial assessments of the target
- Partnership arrangements function accordingly
- Readiness of local authorities to cooperate
- Active participation of IDPs and returnees in community-based programme interventions
81
group to design methodology for psychosocial support
A6.4.2. Provide support and referral assistance as deemed appropriate;
A6.4.3. Organise educational experiential workshops for target group;
A6.4.4. Workshops and on-the-job training to transfer methodology to Centres for Social Welfare through targeted capacity building.
A7.1. Support to individuals in accessing various individual rights
A7.1.1 Legal representation of beneficiaries in cases related to accessing rights
A8.1. Conduct baseline and evaluation survey on Knowledge Attitude and Practices towards IDPs and returnees, among representative sample of general population
A8.1.1. Assess knowledge, attitudes and practices towards IDPs and returnees
A7: UNHCR/Implementing partners
Human Resources: Executive Director (50%), Assistant Director for Finance, Database / Info Officer, Lawyer (9 persons), Admin / Messenger / Driver, Finance Assistant, Interpreter, Information officer, Paralegal, Programme Officer (73.82%)
Other: fuel, office supplies, communication costs, utilities
82
A8.1.2. Develop a Behaviour Change Communication Strategy
A8.2. Change perceptions and promote child participation through school-based activities
A8.2.1 Organize One Minute Junior Workshops with a mixed group of IDPs/returnees and other students, in selected primary schools
A8.2.2 School based workshops to change perceptions towards IDPs and returnees
A8.3. Raise awareness on social inclusion of IDPs and returnees through round table discussions and social media
A8.3.1. Round tables for inter-generational dialogue with students, service providers, municipal authorities and local media
A8.3.2. Raise awareness on social inclusion of IDPs and returnees through social media.
Annex 4. Interview guides
General methodological notes:
Each interview and discussion group will start with the presentation of the Evaluator and of the evaluation objectives, followed by the presentation of the interlocutors. Whenever necessary, a brief presentation of the Project will be also done. The questions will be sent in advance to the people who are going to be interviewed.
The participants in discussion groups will be briefed in advance about the major topics to be discussed during the meeting. The groups will be composed of 6-8 people. The discussion groups will last 1.5-2 hours each and will take place in the municipalities sampled for site visits and in-depth review.
In line with best evaluation practices, the interviews and focus groups and discussion groups will be attended only by the Evaluator and the interviewees.
Interview Guide for UNICEF management and project staff
1. To what extent has the project achieved its intended overall objectives and outputs? 2. Please provide examples of project contribution to the attainment of its specific objective
of UNICEF Component: SO3: To enable vulnerable returnees and IDPs to access their rights and inclusive social services
3. What types of mechanisms/ services were most improved by the project activities? 4. Which capacity building activities and mechanisms were the most / least successful in
achieving the planned results and outcomes and why? 5. Did some municipalities perform better than others and why? 6. Which, if any, contextual factors (e.g. political, economic, social) have affected the work
of the project and your own efforts in this area? 7. To what extent were relevant actors and stakeholders included in UNICEF’s programming
and implementation? 8. What efforts did UNICEF undertake, and what challenges did it face in view of managing
available funds efficiently? 9. Looking ahead, which of the achievements made to date are likely to be sustained or
expanded without further external support? Which of them will require further support? 10. The project was aiming to contribute to the longer-term goal “To enable national actors
to address the remaining obstacles to solutions for displaced persons and returnees in accordance with the Revised Annex VII DPA Implementation Strategy goals.” Looking beyond the work of individual actors, how far or close do you feel Bosnia and Herzegovina currently is from that goal? What are the main bottlenecks/obstacles to achieving this goal?
Interview Guide for Institutional Partners
1. To what extent was the project aligned with explicit priorities and needs of the municipality government and/or your institution?
2. What positive changes has the project contributed to?
84
As regards strengthening relevant policy frameworks and budgets related to social protection in your municipality?
As regards enhancing the capacity of your institution? (Please specify what capacities and how) Which capacity building activities and mechanisms were the most / least successful in enabling your institution to improve access for vulnerable children?
As regards improving access to social services for children from vulnerable groups (special focus on IDPs and returnees)?
Other changes?
3. What, if any, contextual influences (e.g. political, economic, social) have influenced the work of the project as well as your own efforts in this area?
4. To what extent were relevant actors and stakeholders included in UNICEF programming and implementation?
5. Looking ahead, which of the achievements made to date are likely to be sustained or expanded without further external support? Which of them would require further support?
6. What do you consider the key factors likely to support or hinder the sustainability of results?
7. The project was aiming to contribute to the longer-term goal “To enable national actors to address the remaining obstacles to solutions for displaced persons and returnees in accordance with the Revised Annex VII DPA Implementation Strategy goals.” Looking beyond the work of individual actors, how far or close do you feel Bosnia and Herzegovina currently is from that goal?
8. Do you have any other observations or comments that you would like to share with us?
85
Guide for Discussion Groups with local actors in target communities
Introduction
Introduction of the consultant to the group
Provision of information on background to the discussion group: - The purpose of the discussion - The intended recipients of findings and how they will be used - How feedback will be handled (issues of anonymity, confidentiality, data protection) - Rules of the discussion group: who speaks when and agreement on how to indicate
when one wants to speak - The time allocated for discussion and explanation of the discussion group approach - Answering any questions participants may have.
Discussion
1) How do you see your role in the process of improving social protection and inclusion, particularly with regards to children, particularly from IDPs and returnee population? What are the measures your institution took to support the process of improving access to social services?
2) How did the project support your efforts? What do you think have been the biggest achievements of the project (e.g. capacity building, introduction of new approaches to working with children, assisting the reform processes initiated in the community, inclusion of children from vulnerable groups, particularly from IDPs and returnee population, in social services)?
3) Would it have been possible to achieve these changes (if any) without the project? 4) Has your municipality ensured funds for continuation of models and practices initiated by
the project? If yes, in what way? 5) What are the challenges ahead and ways to overcome them?
End of Discussion
Thanking participants for attending and giving feedback
86
Annex 5. BENEFICIARY SATISFACTION SURVEY
Interviewee:
Parent……..1
Child……….2
Jointly……...3
Sex of the respondent:
Male………..1
Female…….2
Sex of the child:
Male………..1
Female…….2
1. Which services did your child use?
…………
…………
…………
…………
…………
2. How did you join (…………………….name of the service)?
Invitation………………………….……………………………..1
Referred by hospital/clinic ……………………………….……2
Referred by Center for Social Work…………………………..3
Volunteered…………….……………………………………….4
Informants/word-of-mouth……………………………………..5
Other (specify) 6
3. What was the motivation for joining (……………………name of the service)?
Quality of services………………………..1
Innovative services…………………….…2
87
Emotional, spiritual & social support……3
Education, awareness & prevention…….4
All of the above…………………………….5
Other (specify) 6
4. How long have you been associated with (…………………name of the service)?
Less than a month…………………………1
Between 1 & 2 months…………………….2
Between 2 & 4 months…………………….3
More than 4 months……………………….4
5. Did this or similar service exist prior to the project in your municipality?
Yes……………………1 If yes, what type of service?...........................................................
No……………………..2
Don’t know.………..…3
Not applicable….…….4
6. Please, rate services you use?
Service My child and I enjoy it totally
My child and I enjoy it a lot
My child and I enjoy it somewhat
My child and I don't enjoy it mostly
My child and I don’t enjoy it
7. Are there other services that you would have liked provided?
88
Yes….………………1
No………..…………2
Other (specify) 3
8. If yes, please list them.
9. Are you satisfied with social services you receive from your social service provider(s)?
Yes……………………………1
No……………………………..2
Indifferent……………………..3
10. How would you rate your level of satisfaction with the services that are provided to you and your family? (NOTE: rate only those services mentioned in Section A question 6 above).
Service provided Level of satisfaction
Good Fair Poor
11. What is your opinion on the provided services in terms of availability, affordability and accessibility?
89
Services provided i. ii. iii.
Easily available?
What is the staff attitude like?
Comment
Yes..………..1
No.………….2 Don’t know...3
N/A.…………8
Warm, open..………..1
Indifferent.…..2
Negative….....9
N/A.……….…8
12. What are the main constraints of the Project/social service you use? Please, write down all main constraints.
13. How confident are you that the services you currently receive will be sustained or continued as long as you need them?
Very confident………………………………………..1
Moderately confident…………………………………2
Least confident due to donor fatigue……………….3
Least confident due to budgetary constraints…..…4
Not sure…………………………………………….….5
Don’t know………………………………………….…6
Other (specify) 7
14. What would be your suggestions for improvement of the services offered by (………………name of the service)?
THANK YOU FOR YOUR TIME AND PARTICIPATION!
90
Annex 6. List of interviewed persons
Name Position
Selma Kazic UNICEF
Anna Riatti UNICEF
Danijela Alijagic-Dolovac UNICEF
Vera Bartel UNICEF
Nela Kacmarcik-Meduna UNICEF
Dzenana Sabic UNICEF consultant
Sanja Tica EUD In BiH
Vivien Savoye UNHCR
Adem Lisičić Citizens' Association ''Nešto Više''
Prijedor
Silvana Jojic Grad Prijedor
Zorica Bubija Gradska uprava
Rade Surlan Gradska uprava
Zoran Indjic CJB Prijedor inspector policije
Branka Jandric- Vukovic CSR Prijedor
Snezana Javoric D3 Prijedor
Sanja Munjuza JU OS “B. Bolic”
Prim. Dr. Azra Pasalic Dom zdravlja – radna grupa
Rajka Zdjelar Grad Prijedor
Foca
Bojana Bajic- Prodanavic Clan radne grupe, dipl. soc. radnik
Sanja Kulic Clan radne grupe, dip. pedagog
Zorana Djokic Dipl. psiholog
Ivana Buzovik Clan komisije, dipl. pegagog
Radenka Sruolevi Opstina Foca
Kulic Dragica SZI Komisija
Bora Kovac JUSSC Hoca prof. clan komisije
Sladjana Vladicic Direktor
Zoran Cosovic Opstina Foca
Borka Kovac Clan komisije
Sukalo Lutvija Opstina Foca
Dusanka Stankovic
Zivinice
Said Cerkezovic Predsjednik komisije
Azra Hodzic Clanica kom- zamjenica predsjednika, radna grupa za soc.inkluziju coordinator PUP
Zufija Hodzic Clanica komisije
Gradiska
Sasa Ratkovic Pomocnik koordinatora Gradiska
Zora Malesevic Obrazovanje
91
Suzana Sorla CSW, clan Komisije
Dragana Bijelic Odjeljenje za drzavnu djelatnost
Zorica Blagic Predstavnica IOPS
Sakiba Sahinovic NUO Povratak
Suzana Cekic Pedagogo OS
Dijana Lucar Soc. radnik u OS
Derventa
Slavica Taskovic
Maja Celar VSI
Sladjana Cicavac CSW
Branimir Palagetic Policija
Vladen Popovic JU Buducnost
Elvira Mujic Skola
Dragana Pejovic Pd stranice
Dijana Tubak Pedijatar, Dom zdravlja
Snjezana Kovacevic Opstina
Ljubinka Lazic CSW
Biljana Radiskovic Dom zdravlja
Gordana Asentic Pedagog
Ankica Stjepanovic Uciteljica
Slaven Simulja Nastavnik
Maglaj
Midhat Karovic PU Maglaj, clan komisije
Azra Cakrana Opstina
Jasmina Hasanic CSW, Soc. zastita
Marko Kelavic NVO i direktor zavoda za zaposljavanje ZENICA
Haris Bradaric Dom zdravlja Maglaj
Samra Jusufbasic Logoped, radi na PUP-U, radi I ankete
Arnela Tumbic Novinar- PR Opstine
Bisermisa Gracic Zamjenica predsjenice komije, Referent u IDP
Brcko
Nataša Stevanović Predsjednica SZI Komisije Brčko distrikta
92
Annex 7. Survey data
Enclosed as a separate excel file.