UNICEF Lebanon Terms of reference: Accelerated Immunization Activities evaluation

PROJECT/ASSIGNMENT TITLE: Evaluation of the Accelerated Immunization Activities within the Health and

Nutrition programme at UNICEF country programme in Lebanon (2017-2019)

BRIEF SUMMARY

The evaluation of the Accelerated Immunization Activities (AIA) covering the period from November 2017 till June 2019 is aimed to

understand primarily the impact of such intervention and secondarily how the approach and strategy shifted from EPI System

Strengthening (in phase I) to mitigating the spread of measles cases (in Phases II and Phase III).

The AIA started in November 2017, aiming to strengthen the existing Expanded Programme of Immunization (EPI) of the Ministry

of Public Health (MoPH), and to prevent the transmission of Polio from Syria in 2017 and other vaccine preventable outbreaks in

Lebanon (such as measles) through working on both the supply and quality of service at the health center level and to increase

demand through finding children defaulting from immunization and enhancing the immunization seeking behavior at community

level. AIA was first piloted in Bekaa and Mount Lebanon in September 2017 for almost one-month period before expanding to more

areas. However, with scarcity, unpredictability and the increase in earmarked funding, UNICEF now would like to evaluate the

Accelerated Immunizations Activities to measure their effectiveness, efficiency, relevance, sustainability and impact of the approach

both on the measles outbreak response, as well as their contribution to Immunization System Strengthening and to provide specific

recommendations on the way forward to strengthen the MoPH immunization programme.

The evaluation is planned to start in December 2019 for 3 months period. The evaluation team will be supported by and reporting to

the M&E Specialist in coordination with the Health and Nutrition section.

BACKGROUND

During the development of the UNICEF Lebanon Country Programme Document (2017-2020), all stakeholders (MoPH, UN

agencies and implementing partners) agreed that immunization should be a main pillar under the Health and Nutrition programme for

the coming 4 years. As such, UNICEF in its CPD, added Outcome (RAM) indicators to measure the progress towards children’s

immunization, against Penta 1 and Penta 3 and measles.

Throughout various country programmes, UNICEF has been supporting the MoPH for both Routine Immunizations and Campaigns

through procurement of routine vaccines and vaccination commodities, in the quality cold chain at the EPI points for quality storage of

vaccines, and campaigns implementation. In addition, UNICEF in past years, supported MoPH and UN agencies (namely WHO) in

conducting several immunization campaigns. Lebanon constantly reported high national immunization coverage (JRF reports).

However, the EPI cluster survey (WHO, 2016), identified low immunization cadasters in Lebanon. In addition, a circulating vaccine

derived Polio outbreak in Syria was declared in 2017, with high risk of contamination to Lebanon. All of this indicated that Lebanon’s

existing EPI system needs further strengthening.

The Accelerated Immunization Activities project was initiated to originally target the low immunization coverage cadasters (identified

in the EPI cluster survey and in consultation with central and regional MoPH focal points). The AIA was modified a bit later in 2018

to respond to the measles outbreak and mitigate the spread of measles cases and try to control the outbreak. In 2019 (Phase III) and due

to continued increase in measles cases, especially in Great Bekaa and Great North, AIA was prioritized in these governorates to control

the spread of cases.

Overall, the AIA implementation was prioritized in cadasters, with the following consideration:

- Equity: based on areas with low immunization coverage cadasters, to reach vulnerable children defaulting from immunization

- Quality: under the leadership of the MoPH, to provide quality vaccines, with priority to Measles and Polio containing

vaccines, but with a referral to the EPI system, to ensure access to quality vaccines and quality vaccine service provision and

full immunization of the child.

- Vulnerability: most of the areas identified as low immunization coverage, were also part of the vulnerable localities identified

in the UN vulnerability mapping

- Engagement with different stakeholders: to implement AIA, the Ministry of Public Health at central and regional levels, with

UNICEF support, involved implementing partners (national and international NGOs) and local authority (such as

municipalities, gatekeepers) and with the personnel at the identified primary health care facilities (PHCC or a dispensary).

The AIA intervention, is aligned with meeting the indicators set in the CPD and are aligned to the SDG 3.

AIA Conceptual Framework – Theory of Change:

The AIA is divided into 3 phases:

-Phase I: November 2017 till December 20171

-Phase II: June 2018 till December 2018

-Phase III: January 2019 till current

When AIA Phase I started, in 2017, its main aim was to: strengthen the existing MoPH EPI system and prevent Polio transmission

from Syria and vaccine preventable outbreak in Lebanon. As such, the AIA Phase I approach focused on:

- Targeting low immunization cadasters identified in the EPI cluster survey conducted in 2016 (WHO) and cadasters there were

identified as vulnerable based on joint discussions between UNICEF and MoPH central and regional level.

- Trained community outreach workers, data entry clerks, vaccinators from identified PHC facilities, qadaa level MoPH staff,

on Interpersonal communication skills, AIA approach, data organization and flow of work

- Implementing partner, jointly with MoPH PHC coordinators and public health officers, scheduled meetings with local

authorities (mainly municipalities) and briefed them about the AIA project and asked them to support in accessing households

in their localities.

- Implementing partner mapped each cadaster

- Outreach teams visited all houses in the cadasters and screened the vaccination cards of all children under age of 15 years old;

identify defaulting children and provide them with a referral voucher with unique barcode.

- Referred children go to the identified PHCs near them and they are provided with needed vaccine and their barcode and their

information is registered

- Children who were identified as defaulters but did not go to the identified PHCs for their vaccination as advised had their data

shared with the implementing partner to follow-up on their immunization

- Implementing partner followed-up on children through phone calls and visits

1 Some implementing partners continued till February 2018 (based on extensions of their PDs)

Also, during Phase I, WHO with MoPH mapped all Informal Tented Settlements (ITSs) into 2 categories A&B, where by all ITSs

under category A (essential): children under 15 years of age, were provided on-spot immunization for defaulting children (measles,

MMR, PCV 13, OPV and IPV) and referred defaulting children to identified PHCs for follow-up on remaining vaccines. And children

in category B were screened and referred to nearest PHC (treated as a Household visit).

AIA phase II – specifically after measles outbreak; MoPH with support of UNICEF adapted the original AIA approach, to include

‘campaign mode’ option in areas with high measles attack rate and that do not have a nearby PHC facility. As such the AIA phase II

approach focused on:

- First, cadasters with measles cases; second, cadasters with low immunization coverage that were targeted in AIA phase I, but

were not completed; finally, cadasters with low immunization coverage that were not approached in Phase I.

- During planning phase jointly with MoPH and implementing partner(s), each cadaster was assigned as potentially needing a

Mobile Vaccination Unit or only outreach workers screening and referring children to nearest PHC and then following up on

them.

- Data is collected electronically using KOBO with direct link to MOPH at central level.

AIA Phase III: UNICEF continued to focus on areas with measles cases; as such the intervention in first 6 months of 2019 was like

that adopted in phase II. In addition:

- UNICEF with MoPH mobilized the health sector, especially in Great North and Great Bekaa, where some sector partners

adopted AIA approach and supported MoPH in targeting children in areas with measles cases.

- UNICEF started discussions with UNICEF partners from different sections, who already conduct outreach, to also support

without additional cost in reaching out to children under 15 years of age and referring defaulters to nearest identified PHC.

- All data is collected through MERA at outreach and center level and data is triangulated at MOPH central level to provide

feedback to health centers and partners on children defaulting from immunization who did not attend the service following

the referral.

Objectives, Purpose & Expected results:

Purpose:

UNICEF is commissioning a summative evaluation on the accelerated immunization activity program to generate substantial

knowledge and learning on the results of AIA project in order to understand how the project (i) reached its objective of strengthening

MoPH national EPI system; (ii) increased immunization coverage in Lebanon, and (iii) had an effect on the measles outbreak

(controlling the spread of cases, or decreasing the number of measles cases, or had no effect on the outbreak). The evaluation is also

intended to propose recommendations for such a project, to make it sustainable, cost effective and included within MoPH EPI

system.

UNICEF will be the primary user of the evaluation report. Ministries, donors and implementing partners will be the secondary users.

Objectives:

The evaluation will explore how the AIA project has contributed towards improving access and coverage of routine immunization and

its effect on the measles outbreak.

More specifically, the objectives of the evaluation are to:

1. Assess the effectiveness of different AIA approaches (door to door mobilization, temporary vaccination sites, mobile

vaccination units…) on enhancing routine immunization, including uptake by most vulnerable girls and boys and attitudes,

knowledge and behaviors.

2. Assess how AIA II and III had any effect on controlling the spread of measles cases in Lebanon; and if AIA I had any

‘protective factor’ on some cadasters and prevented or at least delayed the onset of measles outbreak

3. Assess the effect of AIA on strengthening the MoPH expanded programme of immunization (EPI) system; including the

effect of AIA on the enhancement of EPI data/registry system through KOBO/MERA application.

4. Provide recommendations on how to move forward with an immunization strengthening approach, which is relevant, efficient,

cost-effective and sustainable, especially with less and less funding available in the health sector. These recommendations

will be utilized bythe MoPH, UNICEF and the main health stake-holders to feed into future strategies for the Health and

Nutrition programme and upcoming UNICEF Lebanon Country Programme Document and could influence the health chapter

of the LCRP.

a. Scope of work:

The scope of the evaluation will focus on the UNICEF Lebanon AIA project, from 2017 till June 2019 in all governorates of Lebanon

including Akkar, North, Bekaa, Baalbeck-Hermel, Beirut, Mount Lebanon, South and Nabatieh.

The evaluation will specifically look at the following pillars:

1. Sustainable strengthening MoPH national EPI system;

2. Increasing immunization coverage in Lebanon;

3. Controlling the measles outbreak

The evaluation will take into consideration, the CPD as the highest level of outcome, whereby the AIA project falls as one of the tools

to reach the CPD Child Survival outcome. In addition, the rolling work plans of Health & Nutrition section of 2017, 2018 and 2019

will be considered when measuring the outcome of AIA project.

b. Time covered by the evaluation

The period 2017-2019 will be considered as the time frame for the evaluation which includes the 3 phases (AIA I+II+III)

c. Timing of the evaluation

The evaluation will be used to inform next years rolling work plan and shape the needed Immunization strategy. It will also guide in

the planning of the next country programme document (2022-2026). The start of the evaluation will be December 2019 – for a period

of 3 months.

Evaluation questions:

The following questions for this evaluation are initial suggestions formulated based on the OECD DAC criteria, and will be further

developed by the research institute/consultancy firm, that will contribute in developing the evaluation methodology and instruments

(including questionnaires). Overall, the evaluation aims to answer the following questions focusing to understand the relevance,

effectiveness, efficiency, coverage, and sustainability of the project:

1. How relevant was the AIA project to MoPH EPI national priorities and the needs of the most vulnerable boys and girls in Lebanon?

a. To what extent did the AIA project suit the priorities of the national MoPH strategy, MoPH EPI programme, UN LCRP,

Health sector plan and UNICEF CPD, and annual and rolling workplans?

b. To what extent are the objectives of the AIA programme still valid and will be valid - in regards to epidemiology (outbreaks)

and immunization coverage of children in Lebanon - for the coming country programme document?

c. To what extent are the activities and outputs of the programme consistent with the overall goal and achievement of its initial

objectives considering the shift from phase I to II and III? (Was the shift in the approach successful and useful or UNICEF

just had to remain in the initial AIA approach?)

2. To what extent was the project effective in meeting the intended goal?

a. To what extent was the project effective in strengthening the functionality of the MOPH EPI system?

b. How did provision of immunization data to health centers enable them to know the immunization coverage within their

catchment area and informed them on the children defaulting from immunization within their area?

c. What real difference has the project made to targeted PHC staff and centers in terms of immunization knowledge, motivation

and ownership?

d. To what extent did UNICEF collaborate with the right (key) partners to ensure achievement of the results? Were there any

challenges in achieving the intended goal?

e. To what extent did the AIA project result in changes in the knowledge and behavior of caregivers of children (boys and girls)

under 15 years of age towards immunizing their children and following up on their routine vaccination? And increasing their

trust in PHCs?

f. To what extent did the AIA project protect from vaccine preventable outbreaks? And to what extent did it control the spread

of measles outbreak?

3. To what extent did the AIA project increase routine immunization coverage at national level, or at least at targeted cadasters?

a. To what extent was the project effective in reaching most vulnerable children (boys and girls) in Lebanon including children

defaulting from routine vaccines? To what extent was the project generating evidence to inform equity EPI priorities? To what

extent was it gender equitable?

b. What reasons accounted for not reaching an identified defaulter child (boys and girls) with measles and polio vaccination?

What lessons learnt can be drawn and applied for strategy revision/strengthening in the near future?

c. To what extent was the project tailored to meet the needs of Lebanese and non-Lebanese children(boys and girls) in rural and

urban setting?

d. To what extent was the project tailored to target children (boys and girls) with disabilities?

4. To what extent was the project cost efficient (reflecting any differences between urban and rural areas; between one implementing

partner and another)

5. To what extent is the AIA project sustainable?

a. To what extent did the benefits of AIA project continue when interventions (due to limiting funding) stopped? Or intervention

was moved from one area to another? (what does sustainability for a vaccination programme look like? Will MoH continue

the programme? Are vaccines available without UNICEF project? Will the staff continue to use the knowledge?)

-If yes: What were the major factors that influenced the achievement of sustainability of the project?

-If no: what were the major factors that influenced the non-achievement of the sustainability of the project?

b. To what extent have the AIA project contributed to or embed sustained local ownership and involvement of local

communities in increasing demand and utilization of routine immunization services considering gender disaggregation of

caregivers?

c. Did the AIA project contributed to sustainable strengthening of the MoPH EPI system (in capacity building, in community

outreach, in EPI registry system)?

d. To what extent did MoPH, PHCs and local authority have sufficient capacity to take on tasks without UNICEF support

(especially that with decrease in funds, UNICEF will not be able to provide HR support to MoPH and PHCs)?

Evaluation stakeholders:

A preliminary mapping of relevant stakeholders of the AIA programme identified the below list of stakeholders. In addition, these

stakeholders are divided into four sub-categories depending on their relative level of “Interest” and relative “Influence”. These

stakeholders are at the core of the design of the programme with whom the team will engage.

Lo

w

I

NF

LU

EN

CE

H

igh

Municipalities Ministry of Public Health;

World Health Organization

Municipalities

Primary Health Care Center Directors

Primary Health Care Center Directors

Caregivers of children under 15

Primary Health Care Center Directors

Implementing partners

Caregivers of children under 15

Low INTEREST High

Evaluation Methods:

The methodology for this evaluation method will be a mix of quantitative (analysis of existing secondary data) and qualitative

tools (KII, FGDs). . The overall methodology should be participatory and should ensure that the various stakeholder including

the most vulnerable communities, municipalities, mayors, partners and others will be reached and their voices elicited.

It is expected that the evaluation will use the following methods:

- Desk Review of key documents: analysis of data from partners and MoPH reports, national surveys and internal compilation

of data and documentation of AIA.

- Key informant interviews with:

• Staff from MoPH PHC department (head of PHC department, senior PHC coordinator, EPI focal point), MoPH qada

offices (qada physicians, PHC coordinators, supporting staff), MoPH IT department (IT project manager and IT AIA

focal points), PHCs (nurses, data entry clerks, directors).

• Project manager from main UNICEF implementing partners and non-implementing partners: WHO, IOCC,

Makhzoumi, Amel, Islamic health association, Al Resaleh Scouts, Al Midan, LRC, Medecin sans frontiers, Nudge. Staff

from UNICEF Health & Nutrition, UNICEF Planning, Monitoring and Evaluation and Communication for

Development and others if needed.

• Municipalities (Member, Mayor, staff involved in AIA, Internal Security Forces)

• Geographical area: Akkar, North, Bekaa, Baalbek Hermel, Beirut, Mount Lebanon, South and Nabatieh considering

partners that targeted Palestinian camps (Burj El Barajneh, Chatila)

- Focus group discussions with:

• Parents/caregivers (men and women)

• Children defaulters and vaccinated (10 to 15 years old)

• Geographical area: Akkar, North, Bekaa, Baalbek Hermel, Beirut, Mount Lebanon, South and Nabatieh considering

targeted Palestinian camps (Burj El Barajneh, Chatila)

The team can further elaborate these requirements in the inception report as appropriate and needed.

List of documents to inform evaluation planning and desk review (This list is non-exhaustive; the evaluation team will be expected to

expand upon this list during their planning phase and desk review.)

The following are some of the key information sources for the evaluation:

- Programme document from different implementing partners and budgets

- LCRP strategy and progress reports (2017 – 2020)

- Presentations on AIA and other internal AIA documents.

- Monthly reports received by MoPH on AIA data throughout the 3 phases

- Partner reports

- Nudge final report

- Lists of municipalities involved in AIA and list of PHCCs and dispensaries that participated in AIA

- Etc…

Limitations:

Some of the foreseeable limitations to this evaluation include in some cases the absence of good quality data (different formats of

same data, incomplete data).

Gender considerations:

- The research team, involved in data collection should be gender balanced, with the division of responsibilities as equally divided

as possible.

- The research team will ensure that locations for FGDs are easily accessible for female participants and individuals with disabilities

and that the date and time for FGDs will be set so as not to interfere with women/girls, men/boys’ routines. Same-sex focus group

discussions are generally more valuable as women/girls may feel more comfortable speaking about certain topics without the

presence of men. Female moderators will be assigned to female participants in the FGDs.

Ethical considerations:

In line with the Standards for UN Evaluation in the UN System (developed by the UN Evaluation Group), all those engaged in

designing, conducting and managing evaluation activities will aspire to conduct high quality and ethical work guided by professional

standards and ethical and moral principles. The proposal must identify actual or potential ethical issues, as well as measures and

methods adopted to mitigate against these issues. All interviewees will be informed with the purpose of the study and their role and

what information is required specifically from them. No deceptive practices are adopted in the research methodology. Confidentiality

of participants and right of withdrawal are ensured. If interviewees will include minors, a written consent should be taken from the

persons in charge of their care. Most importantly, the research methodology and every personnel involved in the study must ensure and

abide by the “Do No Harm” principle. This means that during all the stages of the research, the team needs to avoid putting the

participants at any risk, providing false information, or giving false promises. The harm can be physical, psychological, social, or

financial. The three pillars of the Do No Harm are Respect, Beneficence and Non-maleficence, and Justice. The least to ensure Do No

Harm, the research team must abide by the aforementioned considerations (see UNICEF Procedure for Ethical Standards in Research,

Evaluation, Data Collection and Analysis for details). All the documents, including data and fieldwork instruments, developed in the

course of this consultancy are the intellectual property of UNICEF.

All tools developed must be in line with the Institutional Review Board (IRB) or the Ethical Review Board (ERB). The IRB is a

constituted review body established or designated by an institution to protect the rights and welfare of human subjects recruited to

participate in biomedical or behavioral or social science research. IRBs attempt to ensure, both in advance and by periodic review,

protection of subjects by reviewing research proposals and related materials. IRB protocols assess the ethics of research, evaluations

or data collection and analysis and their methods, promote fully informed and voluntary participation by prospective subjects capable

of making such choices (or, if that is not possible, informed permission given by a suitable proxy), and seeks to maximize the safety of

subjects. The below provides more details on the main ethical aspects, that are further explored in details in the UNICEF Procedure for

Ethical Standards in Research, Evaluation, Data Collection and Analysis.

A. Informed Consent

The purpose of the informed consent is to assure that the fundamental rights and welfare of subjects are protected. Signing the informed

consent ensures that the document has been provided to a prospective subject, risks and benefits have been explained, participants agree

to participate, and a contact has been given incase of any complaints or queries. As per the IRB guidance, the informed consent should

include the following:

An age-appropriate statement that the study involves research, an explanation of the purposes of the research and the expected

duration of the subject's participation, a description of the procedures to be followed, and identification of any procedures which

are experimental;

- A description of any reasonably foreseeable risks or discomforts to the subject;

- A description of any benefits to the subject or to others which may reasonably be expected from the research;

- A disclosure of appropriate alternative procedures or courses of assistance, if needed, that might be advantageous to

the subject;

- A statement describing the extent, if any, to which privacy and confidentiality of records identifying the subject will

be maintained, and any limitations to confidentiality (e.g., mandatory reporting of abuse, etc.);

- For research involving more than minimal risk, an explanation of the risk involved and any support services that will

be made available;

- An explanation of whom to contact for answers to pertinent questions about the research and research subjects' rights, and

whom to contact in the event of a research-related injury to the subject; and

- A statement that participation is voluntary and negotiable, refusal to participate or choose not to respond to a particular

question will involve no penalty or loss of benefits to which the subject is otherwise entitled, and the subject may discontinue

participation at any time without penalty or loss of benefits to which the subject is otherwise entitled.

B. Protection of Data

To maintain the integrity of stored data, project data should be protected from physical damage as well as from tampering, loss, or theft

by limiting access to data. Principal Investigators should decide which project members are authorized to access and manage stored

data. Data stored on paper, such as surveys or notebooks should be kept together in a safe, secure location away from public access,

e.g., a locked file cabinet. Confidentiality and anonymity can be assured by replacing names and other information with encoded

identifiers, with the encoding key kept in a different secure location. Ultimately, the best way to protect data may be to fully educate

all members of the research team about data protection procedures. Data protection should be a part of every project's plan for data

storage. The best way to protect data, whether in written or electronic form, is by limiting access to the data. Electronic data storage

offers many benefits but requires additional consideration and safeguards. Theft and hacking are concerns with electronic data. Many

research projects involve electronic collection and maintenance of human subjects’ data and other confidential records that could

become the target of hackers.

C. Protection of Human Subjects’ Identities

If it is essential to collect and link identifying information (e.g., subjects' names) to subjects’ responses (e.g., questionnaire answers),

researchers must do their best and may need to be creative to provide the utmost confidentiality of subject data. Providing subject

anonymity of information collected means that either the project does not collect identifying information of individual subjects (e.g.,

name, address, email address, etc.), or the project cannot link individual responses with participants’ identities Maintaining

confidentiality of information collected from research participants means that only the investigator(s) or individuals

collecting/analyzing data can identify the responses of individual subjects. However, the researchers must make every effort to prevent

anyone outside of the project from connecting individual subjects with their responses. Examples of increases the level of

confidentiality is use a unique subject code instead of recording identifying information and/or encrypt identifiable data.

D. Protection of Human Subjects’ Safety

The primary concern of researchers should be subject safety. Protecting subject safety requires researchers to use all available

information to identify potential risks to subjects, to establish means of minimizing those risks, and to continually monitor the ongoing

research for adverse events experienced by subjects. Researchers must be prepared to stop the study if risks arise. To assess risks and

benefits, researchers need to explain and potential risks, as well as benefits, that are encountered by the research project. The term risk

refers to the possibility that harm might occur. There are many kinds of risks, such as psychological, physical, legal, social and

economic hardship. The term benefit in research refers to something positive as related to health or welfare. Risks and benefits affect

not only subjects, but also their families and society at large. The IRB will decide whether the risks to the subjects are justified.

Three Basic Ethical Principles:

- Respect for subjects: protecting the autonomy of all people and treating them with courtesy and respect and allowing for

informed consent. Researchers must be truthful and conduct no deception.

- Beneficence: The philosophy of "Do no harm" while maximizing benefits for the research project and minimizing risks to the

research subjects. Subjects are treated in an ethical manner not only by respecting their decisions and protecting them from

harm, but also by making efforts to secure their well-being.

- Justice: ensuring reasonable, non-exploitative, and well-considered procedures are administered fairly — the fair distribution

of costs and benefits to potential research participants — and equally to each person an equal share.

- The basics of all human research subject protections is the fulfillment of nine functions:

- The risks of the research are minimized;

- The risks to subjects are reasonable in relation to anticipated benefits;

- The selection of subjects is fair;

- Each participant gives a voluntary and informed consent;

- When appropriate, the research plan makes adequate provisions for monitoring the data collected to ensure the safety of

subjects;

- There are adequate provisions to protect the privacy of subjects and to maintain the confidentiality of data; Conflicts of interest

are transparent and appropriately managed;

- Consideration is given to what additional protections, if any, are needed for vulnerable populations; and

- Proper training in human subjects’ protections is provided for research personnel.

Management of the evaluation:

The planning, monitoring and evaluation specialist will directly manage the evaluation team in coordination with the Health and

Nutrition Officer to ensure neutrality of the evaluation. MoPH and the Communication for Development Section as well as the Health

& Nutrition Section will also be involved in the evaluation and the H&N section.

The Planning, Monitoring and Evaluation specialist with the Health and Nutrition Officer will give final approval for all the

deliverables, including inception report and the final evaluation report (considering MoPH and communication for development focal

points inputs) prior to final payment.

In addition, UNICEF team (Planning, monitoring and evaluation specialist and health and nutrition officer) will support the

coordination of the evaluation, by facilitating the evaluation team and providing necessary assistance and information to effectively

support the AIA programme evaluation.

UNICEF team will be responsible in the evaluation of bids and act as the selection panel following the rules and regulations of UNICEF,

which will be the contacting party.

Deliverables:

The tasks to be completed by the contractor (or team of consultants) includes, but are not necessarily limited to the following:

Phases Tasks Deliverables

Timeline

(Working

Days)

Phase 1 Review background documentation, including

analysis of data from partners and MoPH

reports, national surveys and internal

compilation of data and documentation of AIA.

Bibliography 5 days

Phase 2 Develop an inception report including

evaluation design and detailed

methodology/tools, work plan for data

collection, and an evaluation matrix.

Inception Report with

annexes submitted to UNCEF

Lebanon country office

10 days

Presentation of Inception report to Health &

Nutrition and Planning, Monitoring and

Evaluation sections.

Feedback on Inception report

provided by UNICEF, MoPH

7 days

Research team to address inputs provided by

UNICEF

Feedback on Inception report

addressed

5 days

Phase 3 Desk Review;

Data collection: Meet/interview/group

discussion with relevant key stakeholders

and beneficiaries

Desk review, Interviews and group

discussion with key stakeholders

conducted in due time.

20 days

Phase 4 Perform analysis and produce draft

preliminary findings and Produce draft

evaluation report

Preliminary findings and draft

report available and shared with

UNICEF Lebanon country office

and MoPH.

20 days

Presentation of draft report to UNICEF

Leabanon Country Office and MoPH, as

well stakeholders including WHO and

Implementing Partner Organizations.

Feedback on final report provided

by UNICEF, MoPH

7 days

Research team to addresse inputs provided by

UNICEF LCO comments

Feedback on final report addressed 5 days

Evaluation team presents the findings of the AIA

program evaluation final report.

Final report is shared with UNICEF

Lebanon Country Office and MoPH

(The final report should be 40 to 60

pages.). A workshop will be

organized to present key findings of

the evaluation.

10 days

Responsibilities:

The research institute/consultancy firm is responsible to produce the following by the stated deadline:

1. Desk review and preparation of the inception report: 27 days

2. Data collection: 20 days

3. Final report draft submission: 32 days

4. Final report: 10 days

Payment schedule:

Payment proportion Phases

30% of total cost Upon delivery of deliverables in phase 1 & 2

30% of total cost Upon delivery of deliverables in phase 3

40% of total cost Upon delivery of deliverable in phase 4

Reporting requirements:

- The research institute/company needs to meet UNICEF’s Quality Assurance requirements to an evaluation. Guidance and

tools relevant to these requirements at any stage of work and respective deliverable: Inception report and full evaluation

report, as well as ethical review requirements, will be shared with the selected service provider (Evaluation quality

assurance/ Ethical review (UNICEF), integrating human rights and gender equality in evaluations (UNEG).

- The report structure will be as per UNEG required standard. UNICEF-Adapted UNEG Evaluation Reports Standards will

be shared with the research company.

- The Consultant will report to the Planning, Monitoring and Evaluation Specialist.

- The reports will be electronically submitted to the Planning, Monitoring and Evaluation Specialist.

- The final report will be submitted in the evaluation database Evidence Information System Integration (EISI). The Global

Evaluation Reports Oversight System (GEROS) will report on the quality of the evaluations, by reviewing and assessing

the quality of final evaluation reports commissioned by UNICEF Offices. The quality of the evaluation report is then

reported to senior management mainly through three channels: a) annual report of the Director to the Executive Board; b)

the Global Evaluation Dashboard, and c) inclusion of this information in the Global Evaluation database;

Qualification Requirements:

This evaluation is looking for research institutes/companies with proven expertise in programme evaluation in an emergency and

development context, and especially within the Health and Nutrition field. UNICEF will ensure neutrality and no conflict of interest

in the recruitment process.

The staff from the research institute/company should have the following qualifications:

Team of

consultants

Work Experience and academic qualification

Team Leader

• Advanced University degree, ideally MPH/PhD, in a relevant Health discipline (for instance,

Public Health, Epidemiology, Bio-Statistics, Health Management) with at least 10 years of

relevant work experience (both international and national) in health, specifically in

immunization programming;

• A minimum of 8 years of practical experience in implementing evaluations (having conducted

evaluation on Immunization programme (routine and campaigns) and Immunizations System

Strengthening is an asset);

• Qualitative and quantitative methods expertise;

• Excellent report writing and analytical skills in English;

• Knowledge of Arabic language is mandatory;

• Familiarity with UNICEF's programming strategies and organizational culture;

• Previous experience in undertaking evaluations of health programmes or similar work

experience in the region specifically in Lebanon will be an asset;

• Understanding of human rights-based approaches to programming, gender considerations and

participatory approaches considered in programme evaluations;

• Strong inter-personal, teamwork and organizational skills;

• Familiarity with information technology, including proficiency in word processing,

spreadsheets, and presentation software.

Overall team

requirements

• Master’s degree in one of the disciplines relevant to public health (for instance epidemiology,

bio-statistics, Health Management, health promotion).

• Good understanding on the use of evaluation methodologies including both quantitative and

qualitative methods for data collection and analysis;

• Previous experience in undertaking evaluations especially for health programming including

Immunizations is considered an advantage;

• Understanding of human rights-based approaches to programming’, gender considerations and

participatory approaches considered in programme evaluations;

• Familiarity with UNICEF's programming strategies and organizational culture;

• Strong inter-personal, teamwork and organizational skills;

• Familiarity with information technology, including proficiency in word processing, spreadsheets,

and presentation software;

• Fluent in both English and Arabic languages.

Evaluation process of the bids:

1-Technical Evaluation Criteria:

Technical Evaluation Criteria Max. Points

Obtainable

1 Overall Response

1.1 Completeness of response and Understanding of UNICEF requirements. 5

5

2 Overall Experience of Research Institute/Company and Key Personnel Proposed for the assignment

2.1

Proven experience with UN agencies (The team leader having conducted at least 2 projects) (2.5 points

per previous contracts).

Familiarity with UNICEF’s health programming and UNICEF’s organizational culture.

5

2.2

Evidence of experience in conducting evaluation of humanitarian and/ or development projects in

Lebanon and internationally (including MENA).

(The team leader having conducted at least 4 projects) (2.5 points per project)

10

2.3 Evidence of previous experience in undertaking evaluations of health programmes specifically

Immunizations. 10

2.4 Proven experience in similar methods (qualitative and quantitative) as proposed with analytical skills 10

35

3 Proposed Approach

3.1 Logical proposal of deliverables and timelines for the consultancy respecting the proposed milestones. 10

3.2 Adequacy of the evaluation proposal and work plan, as per requested methodology including

evaluation questions. (Focus on language and local expertise) 20

30

TOTAL TECHNICAL SCORE 70

Technical evaluation is composed of 70 points

Minimum successful score for the technical evaluation is 49 points

2- Financial Evaluation Criteria:

- Only bidders obtaining the minimum pass mark in the technical evaluation (49 points) will be considered for the financial

evaluation.

- Financial evaluation is composed of 30 points. The lowest financial offer will obtain 30 points.

- Service Providers should fill the attached table for calculation of the financial evaluation.

Timing/Duration of Contract:

UNICEF will receive the final evaluation report 3 months from the date the contract is signed.

Duty Station: Beirut

Administrative issues:

It is not mandatory for the entire team (team leader) to be based in Lebanon on the condition that regular visits to Lebanon are

conducted.

Project management:

The project will be managed by PRIME and Health and Nutrition units of UNICEF.

ANNEX: Financial Proposal