evaluation seminar1
TRANSCRIPT
EVALUATION OF A NATIONAL
HEALTH PROGRAMME
PRESENTERS: Dr. Shanthosh Priyan
S
Dr. Joymati Oinam
MODERATOR: Dr. Shantibala K
OUTLINE OF PRESENTATION
1. Introduction
2. Monitoring & Evaluation definitions
3. Evaluation Vs Monitoring
4. History of evaluation
5. Why, What & When to evaluate
6. Types of evaluation
7. Principles of evaluation
8. The evaluation process
9. Conclusion
INTRODUCTION
Health services have become more complex
There has been a growing concern about their
functioning both in the developed and developing
nations
Questions are raised about the quality of medical care,
utilization and coverage of health services, benefits to
community health and improvement in the health status
of the recipients of care
An evaluation study addresses itself to these issues
So, program evaluation is an essential organizational
practice in public health
However, it is not practiced consistently across
program areas, nor is it sufficiently well-integrated into
the day-to-day management of most programs
Thus, a framework for understanding program
evaluation and facilitating integration of evaluation
throughout the public health system is needed
The purposes of this seminar is to
• summarize the essential elements of program evaluation;
• provide a framework for conducting effective program
evaluations;
• clarify the steps in program evaluation;
• review standards for effective program evaluation; and
• address misconceptions regarding the purposes and
methods of program evaluation
WHAT IS MONITORING?
Monitoring is the periodic oversight of the
implementation of an activity which seeks to establish the
extent to which input deliveries, work schedules, other
required actions and targeted outputs are proceeding
according to plan, so that timely action can be taken to
correct deficiencies detected
"Monitoring" is also useful for the systematic checking
on a condition or set of conditions, such as following the
situation of women and children
MEANING OF EVALUATION
Evaluation has its origin from the Latin word
“Valupure” which means the value of a particular
thing, idea or action
Evaluation, thus, helps us to understand the worth,
quality, significance amount, degree or condition of any
intervention desired to tackle a social problem
Evaluation means finding out the value of something
Evaluation simply refers to the procedures of fact
finding
WHAT IS EVALUATION?
“The systematic investigation of the worth, merit, or
significance of an ‘object’ ”
Michael Scriven
“The systematic collection of information about the
activities, characteristics and outcomes of programs to
make judgments about the program, improve program
effectiveness, and/or inform decisions about future
program development”
CDC
Evaluation has been defined as
“ The systematic and scientific process determining
the extent to which an action or set of actions were
successful in the achievement of pre-determined objectives.
It involves measurement of adequacy, effectiveness and
efficiency of health services”
-WHO
Monitoring
Recording
Analysis
Reporting
Corrective action at the operational
level
Information
Recording Recommendations
Analysis
Information
from other
sources
Information from
Monitoring
Affirmation or modification in
objectives, resources, and process
Relationship of Monitoring and
EvaluationEvaluation
Objective
Monitoring Evaluation
To track changes from
baseline conditions to
desired outcomes
To validate what results were
achieved, and how and why
they were or were not
achieved
Methodology Tracks and assesses
performance through
analysis and
comparison of
indicators over time
• Evaluates achievement or
outcomes by comparing
indicators before and after
the intervention
• Involves Value Judgment
• Relies on monitoring data
and information from
external sources
Characteristics
Continuous and
systematic by
Programme/ Project
managers and key
partners
Time-bound, periodic, in-
depth by Internal or
External evaluators and
partners
Uses Alerts managers about
problems in
performance, provides
options for corrective
actions and helps
demonstrate
accountability
Provides managers /
donors / stakeholders with
strategy and policy
options, provides basis for
learning and demonstrates
accountability
PROGRAM
“An organized response to eradicate or eliminate or
reduce 1/more problems where the response includes 1/
more objectives & the expenditure of resources”
“Any set of organized activities supported by a set of
resources to achieve a specific and intended result.”
Research vs. Evaluation
Knowledge intended for use
Program- or funder-derivedquestions
Judgmental quality
Action setting
Role conflicts
Often not published
Multiple allegiances
Production of generalizableknowledge
Researcher-derived questions
Paradigm stance
More controlled setting
Clearer role
Published
Clearer allegiance
Research EvaluationSystematic
Methods
HISTORY OF EVALUATION
Began in the field of education
Strengthened during the 1960’s emphasis on social
programs and determining their effect on society
Further strengthened during the 1990’s emphasis on
outcomes measurement and quality improvement
WHY EVALUATE PROGRAMS?
To gain insight about a program and its operations – to see
where we are going and where we are coming from, and to
find out what works and what doesn’t
To improve practice – to modify or adapt practice to enhance
the success of activities
To assess effects – to see how well we are meeting objectives
and goals, how the program benefits the community, and to
provide evidence of effectiveness
To build capacity - increase funding, enhance skills,
strengthen accountability
WHAT CAN BE EVALUATED?
Direct service interventions
Community mobilization efforts
Research initiatives
Surveillance systems
Policy development activities
Outbreak investigations
Laboratory diagnostics
Communication campaigns
Infrastructure-building
projects
Training and educational
services
Advocacy works
– MMWR, 1999
Framework for Program Evaluation in Public
Health
WHEN TO CONDUCT EVALUATION?
Conception Completion
Planning a
NEW
program
Assessing a
DEVELOPING
program
Assessing a
STABLE,MATURE
program
Assessing a
program after
it has ENDED
• Formative evaluation
Evaluation intended to improve performance, most
often conducted during the design and/or
implementation phases of projects or programs
Its intent is to assess ongoing project activities
Formative evaluation has two components:
implementation evaluation and progress evaluation
Implementation(Process) evaluation:
The purpose of implementation evaluation is to assess
whether the project is being conducted as planned
Progress evaluation:
To assess progress in meeting the goals of the
program and the project. It involves collecting
information to learn whether or not the benchmarks of
participant progress were met and to point out
unexpected developments
Evaluation Area
(Formative assessment )
Evaluation Question Examples of Specific
Measurable Indicators
Staff Supply Is staff supply sufficient? Staff-to-client ratios
Service Utilization What are the program’s usage
levels?
Percentage of utilization
Accessibility of Services How do members of the target
population perceive service
availability?
• Percentage of target
population who are
aware of the program in their
area
• Percentage of the “aware”
target population who know
how to access the service
Client Satisfaction How satisfied are clients? Percentage of clients who
report being
satisfied with the service
received
• Summative(Outcome/Impact) evaluation
An evaluation conducted at the end of an intervention
to determine the extent to which anticipated outcomes were
produced such as improved survival or reduced disability
The traditional outcome components are the 5 D’s of ill
health, viz. disease, discomfort, dissatisfaction,
disability and death
Evaluation Area
(Summative Assessment)
Evaluation question Examples of specific
measurable indicators
Changes in Behaviour Have risk factors for
cardiac disease have
changed?
Compare proportion of
respondents who reported
increased physical activity
Morbidity/Mortality • Has lung cancer
mortality decreased by
10%?
• Has there been a
reduction in the rate of
low birth weight babies?
• Age-standardized lung
cancer mortality rates for
males and females
•Compare annual rates of
low-birth weight babies over
five years period
• Participatory evaluation
Evaluation which provides the active involvement of
those with a stake in the program, (providers, partners,
beneficiaries, and any other interested parties) in designing,
carrying out and interpreting an evaluation
• 360 degree Evaluation (evaluation by planner &
implementers)
An evaluation by those who are entrusted with the
design and delivery of a project
TYPES OF EVALUATION Contd..
PURPOSE OF EVALUATION
To improve health programs and the services for
delivering them
To guide the allocation of resources in current and
future programs
Should be used constructively & not for the
justification of past actions/merely to identify their
inadequacies
Decision-oriented tool and to link closely with
decision making, whether at the operational/the
policy level
The very process of carrying out an evaluation often
induces a better understanding of the activities being
evaluated, and a more constructive approach to their
implementation and to any further action required
CONSTRAINTS OF EVALUATION
Evaluation, difficult in any field, presents particular
problems in health work owing to the very nature of the
activities, which often do not lend themselves easily to
the measurement of what has been attained against
predetermined, quantified objectives
It is therefore often unavoidable to apply qualitative
judgment, supported by reliable, quantified information
Changes in a health situation are often brought about by
elements outside the health sector, making evaluation,
particularly of effectiveness & impact, even more
difficult
Development and selection of sensitive and specific
indicators a challenging and difficult task
There is a certain in-built resistance in principle to
accepting evaluation & its results as a valid management
tool
PLACE OF EVALUATION IN THE
HEALTH DEVELOPMENT PROCESS
The purpose of the managerial process for national
health development is to build up the health system in a
rational & systematic way
Health program evaluation is part of the managerial
process for national health development
RESPONSIBILITY OF EVALUATION
Central level: Director-general of health services in the
ministry of health, the minister of finance, Directors, Deans
District level: Directors of district hospitals, public health
laboratories, environmental health services, training schools
& finally, the district health officer
Local level: Communities themselves
COMPONENTS OF EVALUATION
PROCESS
Relevance:
Relates to the appropriateness of the services, whether it
is needed at all
For example, vaccination against small pox is now
irrelevant because the disease no longer exists
Adequacy:
Implies that sufficient attention has been paid to certain
previously determined courses of action, such as the
various issues to be considered during broad
programming
For example, the staff allocated to a certain program
may be described as inadequate if sufficient attention
was not paid to the quantum of workload and targets to
be achieved
Accessibility:
Proportion of the given population that can be expected
to use a specified facility, service, etc.
The barriers of accessibility may be physical(distance,
travel, time); economic(travel cost, fee charged); or
social and cultural(caste or language)
Acceptability:
The services provided may be accessible but not
acceptable(male sterilisation, rectal cancer screening)
Efficiency:
Expression of the relationships between the results
obtained from a health program/activity and the efforts
expended in terms of human, financial and other
resources, health processes and technology, and time
(number of immunizations provided in an year as
compared with an accepted norm)
Cost-benefit analysis will be useful for this purpose
To find out if optimal utilization of available resources is
being made
Effectiveness:
Expression of the desired effect of a program,
service, institution or support activity in reducing a
health problem or improving an unsatisfactory health
situation
Measures the degree of attainment of the
predetermined objectives & targets of the program,
service/institution
Impact:
Expression of the overall effect of a program,
service/institution on health and related socio-
economic development
Aimed at identifying any necessary change in the
direction of health programs so as to increase their
contribution to overall health and socioeconomic
development
Sustainability:
Meeting needs without compromising the ability of
future generations to meet their needs (project will
continue after donors intervention)
FREQUENCY OF EVALUATION
While evaluation is a continuing process, its results have to
be summarized & reported on at given times/specified
intervals
The frequency will vary ranging from relatively short
intervals for the assessment of progress and efficiency to
much longer intervals for the assessment of effectiveness
and impact
INDICATORS AND CRITERIA FOR
EVALUATION
Indicators are variables that help to measure changes
They are evaluation tools which can measure changes
directly/indirectly
Indicators should be:
valid
reliable
sensitive &
specific
Health policy indicators:
Allocation of adequate resources to health for all
Level of community involvement in attaining health for
all
Social and economic indicators:
Rate of population increase
Adult literacy rate
Adequacy of housing as expressed as number of persons
per room
Indicators of provision of health care
Availability
Physical accessibility
Utilization of services
Health status indicators
Infant mortality rate
Child mortality rate
Under 5 mortality rate
Life expectancy
Criteria are standards by which actions are measured
2 types- Technical and Social
Technical-normally highly specific to program
For example, technical criteria for the guarantee of
safety of drinking water would be certain technical
standard for purity of drinking water
A social criteria for the guarantee of the continuation of
the water supply would be the existence of community
organizations for the maintenance of the supply
Main purpose of criteria is to provoke thought leading
to judgment
Indicators and criteria should be included in the
program at the planning stage, so that the information
requirements can be determined early on
INFORMATION SUPPORT
Evaluation has to be based on valid, relevant and
sensitive information
The types of information required may include
political, social, cultural, economic, environmental
and administrative factors influencing the health
situation as well as mortality and morbidity statistics
STANDARDS FOR EFFECTIVE
EVALUATION
Utility: Who needs the information and what information
do they need?
Feasibility: How much money, time, and effort can we
put into this?
Propriety: What steps need to be taken for the
evaluation to be ethical?
Accuracy: What design will lead to accurate
information?
Utility:
Ensures that the information needs of intended users are
met.
Who needs the evaluation findings?
What do the users of the evaluation need?
Will the evaluation provide relevant (useful) information
in a timely manner?
Feasibility
Ensures that evaluation is realistic, prudent,
diplomatic, and frugal.
Are the planned evaluation activities realistic given
the time, resources, and expertise at hand?
Propriety
Ensures the evaluation is conducted legally, ethically,
and with due regard for the welfare of those involved
and those affected.
Does the evaluation protect the rights of individuals
and protect the welfare of those involved?
Does it engage those most directly affected by the
program and by changes in the program, such as
participants or the surrounding community?
Accuracy:
• Ensures that the evaluation reveals and conveys
technically accurate information.
• Will the evaluation produce findings that are valid and
reliable, given the needs of those who will use the
results?
The evaluation process includes a number of
distinct stages-
A. planning evaluations
B. gathering or recording baseline data
C. managing evaluations
D. implementing evaluations
E. using evaluations
Why do the evaluation?
Identify number of evaluation purposes
Identify who needs what information and for what
purpose
Four common reasons for evaluations are:
Formative Evaluation
Summative Evaluation
Learning Lessons For Future Application
Accountability
When to do an evaluation?
The timing of a major evaluation is affected by:
its relation to the sector/ministry plan, within the
country development plan
a significant problem identified in the course of
monitoring
a donor request
What should be evaluated?
The scope of work of those conducting
evaluations-
Description: What happened and how does this
compare with what was expected?
Analysis: Why and how did it happen or not
happen?
Prescription: What should be done about it?
An evaluation may focus on different levels of
results of a service/programme or project-
Inputs-outputs
Processes
Outcomes Or Impacts
Development of a frame logical model
It is a flow chart that shows the program’s components, the
relationships between components and the sequencing of
events.
The following key issues provide another focus in formulating the main questions an evaluation should address:
Effectiveness - Is the project or programme achieving satisfactory progress toward its stated objectives?
Efficiency - Are the effects being achieved at an acceptable cost, compared with alternative approaches to accomplishing the same objectives?
Relevance - Are the project objectives still relevant?
Impact - What are the results of the project?
Sustainability - Is the activity likely to continue
after
donor funding, or after a special effort, such as a
campaign, ends?
Sample evaluation questions: What
might stakeholders want to know?
Program clients:
• Does this program provide us with high quality service?
• Are some clients provided with better services than other clients? If so, why?
Program Staff:
• Does this program provide our clients with high quality service?
• Should staff make any changes in how they perform their work, as individuals and as a team, to improve program processes and outcomes?
Program managers:
• Does this program provide our clients with high quality service?
• Are there ways managers can improve or change their activities, to improve program processes and outcomes?
Funding bodies:
• Does this program provide its clients with high quality service?
• Is the program cost-effective?
• Should we make changes in how we fund this program or in the level of funding to the program?
Who will do the evaluation?
The evaluation plan should:
identify which entity (entities) will
manage/supervise the evaluation
indicate what type of person (s) should conduct it,
i.e., internal or external evaluators or a combination
How to answer the evaluation
questions?
Determining information needs is an initial
step-
Existing data should be identified -
Monitoring documents, previous evaluations, and
other documentation (audits, mid-term/annual
reviews) of the project
Government routine reporting systems records or
evaluations of similar programmes in agency or
donor offices (WHO, other ministries, NGOs, etc.)
Identify the minimum amount of new information
needed to answer the evaluation questions
Effective evaluations concentrate on collecting
timely, relevant and useable data
Decide what data should be used for the
evaluation-
While planning an evaluation, the sponsors should
work to specify evaluable questions and design a
feasible data collection plan
Involves deciding which indicators to use to
measure
progress
Decide on criteria to judge progress-
Every evaluation should measure progress and compare it to some standard-
programme/project objectives
past performance
national targets
baseline data
similar services or project areas
Match data-collection methods with evaluation
purposes
There are four fundamental ways of obtaining
information:
1. collecting, tabulating, reviewing already available data
2. questioning people through interviews, focus groups
3. conducting surveys
4. observing people and things though field visits
Types of Data, Uses, and Collection Methods
for different Evaluation Focuses
Focus Type of Data Use/
purpose
Data Collection
Methods
Input Financial, material,
personnel
Delivered to
project?
Administrative
records
Output Services provided
and used
Reach target
group?
Administrative
records, RAP,
surveys
Outcome
/impact
Change in
beneficiary status
Effects
attributable
to
programme?
Routine
reports,RAP,
survey, views of
informed people
Efficienc
y
Costs of input,
outputs, impacts
Most effect
for cost?
Cost-
effectiveness
comparisions
What resources are needed and
available for evaluation?
Early in planning an evaluation make an estimate
of its costs
Costs depend on the nature and size of the service
/programme and design of the evaluation
Evaluation planners should consider the non-
financial or
indirect costs of the evaluation
The amount of resources available, determined
during programme planning, should be built into the
budgets of all the entities supporting the
programme financially
B.Data sources for
Evaluation Existing data includes-
Primary and secondary data
Data from the service or project -
monitoring documents [progress reports and field trip
notes] & evaluations [midterm reviews]
Relevant outside information include
evaluations of similar programmes,
special studies and researches of the problem
data from government census, surveys, sentinel
sites, or
C.Managing Evaluations
Principally consists of -
negotiating the evaluation plan
preparing the Terms of Reference
selecting and working with the evaluation
team
Negotiating the evaluation plan with
others
Negotiating skills and understanding how to
design evaluation
The various roles must be taken into account in
deciding who should be consulted in determining
evaluation objectives and methods
Drawing Up the Terms of Reference
The Terms of Reference, sometimes called a
Scope of Work (SOW), set out the formal
agreements about the evaluation
It is like a contract with the evaluators spelling out
what they are to accomplish
Prepared four to six months before the evaluation
to allow adequate time for planning
A good TOR paves the way for a good evaluation
Suggested Contents of the Terms of Reference
1. Background and Purpose
2. Evaluation Questions or Objectives
3. Evaluation Methods
4. Composition of the Evaluation Team
5. Schedule of Major Tasks
6. Deliverables
7. Financial Requirements and Logistical Support
Recruiting and selecting the
evaluation team
Include people from the country on the team
If women are the subject of the study, recruiting women
team members
Size of the team depends also on the duration of the
evaluation
Steering committee members can help recruit
Finally, discuss TOR with those selected and make anymutually agreed upon modifications before initiatingcontractual arrangements
Working with the evaluation team
The person(s) designated to manage the evaluation and supervise the evaluation team should meet with the team before it begins-
to give instructions
review the work plan and answer questions
to send background materials to team members before the orientation meeting
Supervising the evaluation team
Regular contact with the team leader
Maintaining frequent communication and a
cooperative relationship enables problems to be
identified early, tackled together, and the workplan
modified accordingly
Finally, supervisors should evaluate the work of
the evaluation team and assess the quality of the
evaluation and report by:
a. providing feedback to team members, individually
or in a group.
b. asking team members to offer their suggestions
for
improving the process next time
c. assessing the evaluation report and discussing its
strengths and limitations with the team
D.Conducting
EvaluationsTasks of Evaluators
(1) familiarize themselves and refine the evaluation plan
(2) gather data
(3) analyse existing data and those collected to formulate findings and recommendations
(4) write the evaluation report
(5) debrief the interested parties on the findings and recommendations
(6) Disseminate evaluation report
Refining the evaluation plan
Evaluators must choose the type of sample, the
sites and sample size
Site selection is required for most evaluations, and
should be done carefully so that sites are
representative
Formulating Findings
Findings may be called conclusions or lessons learned-
a. describe project/programme/service results
b. compare them to what was planned and/or some other
standard
c. judge whether "enough progress" was made
d. identify major reasons for successes, failures and
constraints
Developing recommendations
Start with the findings
Directed to different kinds of decision makers
Avoid vague, general and impractical
Always list in priority order and include costs of
implementing them
A proposed timetable
Preparing the Evaluation Report
Suggested Contents of Evaluation Report -
1. Title page
2. Table of Contents
3. Acknowledgments (optional)
4. Executive Summary
5. Introduction
6. Evaluation Objectives and Methodology
E.Using Evaluation
Results
Develop a new/revised implementation plan in
partnership with stakeholders
Monitor the implementation of evaluation
recommendations and report regularly on the
implementation progress.
Plan the next evaluation
Conclusions
Monitoring and evaluating are key functions to
improve the performance of those responsible for
implementing health services
During programme preparation and start up it is
necessary to specify general arrangements and
make budgetary provisions for future evaluations
The evaluation plans should be outlined in the
country
programme plan of operations, annual plans of
action and
annual reports for each programme
Interventions that are effective in developed
countries may
not be effective in developing countries
Rigorous program evaluation of interventions in
various
resource-limited settings is needed to determine
which