Role of board in Monitoring and Evaluation
Professor Benon C Basheka, PhD, FCIPS
Uganda Technology and Management University
LEARNING OUTCOMES
By the end of the training, board members should be able to:-
1. Describe the importance of monitoring and evaluation in organizational effectiveness
2. Apply M and E knowledge in the day today activities of their work in ABiTRust
3. Relate M and E requirements to the DEC standards
4. Oversees establishment of an effective M and E system for the organization
TRAINING METHODOLOGY
1. Presentations
2. Individual experiences
3. Discussions
Session coverage
The training will generally cover three units:-
1. General Introduction and Role of Boards of Trustees (with relevance to M and E)
2. Requirements for establishing an M and E system (and role of the board)
3. The Evaluation Process and Roles the board
UNIT 1: GENERAL INTRODUCTION AND ROLE OF THE BOARD IN M & E
INTRODUCTION
• Monitoring and evaluation are increasingly being emphasized in all endeavors
• The need to determine what interventions are working or not and why now necessitate organizations to invest in building sound monitoring and evaluation systems
• In almost all organizations where boards exist, boards generally have oversight functions-overseeing all activities of the organization (M and E Inclusive)
• The range of things where evaluation applies is limited but can include projects, programmes, processes, systems etc
Monitoring/Evaluation is to be seen as:-
• As a profession with a set of standards
• As a field of practice (Monitoring and evaluation units and staff therein)
• As a discipline of study which is an academic field now declared to be in
adolescent stages (universities and other higher educational institutions
Organizations have advisory or policy organs
1. Board of governors
2. Board of managers
3. Board of regents
4. Board of trustees, and
5. Board of visitors
Different Modes of Board Governance
1. Advisory Board Model
2. Patron Model
3. Co-operative Model
4. Management Team Model
5. Policy Board Model [AbiTrust?]
Abi-TRUST Governance
Founders Committee
Management Staff M and E unit
Board of Trustees
Typical duties of boards of directors include:
1. Governing the organization by establishing broad policies and objectives
2. Selecting, appointing, supporting and reviewing the performance of the chief executive
3. Ensuring the availability of adequate financial resources
4. Approving annual budgets
5. Approving annual plans
6. Approving and reviewing Reports for management activities
Working Methodology of Boards
• In larger organizations, the structure of the board and its committees usually mirrors the structure of the organization's administration.
• Just as there are staff responsible for human resources, fund-raising, finance, planning, and programs, the board creates committees with responsibility for these areas.
• One of the areas of board committee will be monitoring and evaluation
• Even the performance of the board or its committees needs to be measured
Strategically
1. Mobilize resources
2. Look for networks
3. Design and approve policy
4. Provide oversight
5. Participate in evaluation
Boards Strength lies on :-
1. Composition
2. Expertise
3. Experience
4. Qualifications
5. Networks
CONTEXTUALLY
• The Board of Trustees of aBi Trust has a four-fold mandate:
1. To protect the Trust’s assets over time and ensure survival and the prosperity of the Trust in a transparent, accountable and responsible manner
2. To guide the Trust in fulfilling its Vision, mission and objectives
3. To give strategic direction to aBi Trust Management
4. To protect the Trust’s interests.
What is the implication?
1. The mandates and functions of the board cannot be efficiently and effectively achieved without sound knowledge of M and E
2. M and E supplies knowledge for oversight and decision making
3. M and E supplies the board with necessary tools and methodologies
4. M and E provides the right attitude and mindset for involvement
5. M and E shapes the decision making to support building required systems
Is Evaluation new?
Evaluation is as old as the world itself and has moved side by side the journey of human civilization
The scriptures tell us under genesis 1:31 that when God created the earth, the light in the darkness, the firmament in the midst in the midst of the waters, the plants, the animals, and finally man, at the end of the fifth day’…God saw everything that he had made, and behold, it was very good
He used some criteria unknown to us and this enabled him to make an assessment on whose findings he was able to make a fundamental decision of not scrapping what he had done
— Michael Q. Patton
God’s archangel came then, asking, “God, how do you know that what you have created is ‘very good’? What are your criteria? On what data do you base your judgment? Just what results were you expecting to attain? And aren’t you a little close to the situation to make a fair and unbiased evaluation?”
God thought about these questions and that day and God’s rest was greatly disturbed. On the eighth day God said, “Lucifer go to hell.”
The second example
• From the philosophical works of Socrates, Plato and Aristotle to the mathematical methodologies of Pythagoras and Euclid, the ideas of the ancient Greeks shaped many institutions and contributions to many fields including evaluation.
• Existing scholarly accounts inform us that the Delphic oracle of the ninth to the third centuries BC was the first central intelligence database of the ancient world, an interdisciplinary think tank of approximately 90 priests, deemed the best educated experts of antiquity. • They collected and evaluated information and advised ordinary people and leaders, among them
Alexander the Great. Major project management in the fourth century BC existed.
In ancient Greece,
• the practices of their 12 gods, called Olympians because they were stationed on Olympus, the highest mountain in Greece shed some light on how evaluation was done.
• The council of the Olympian gods and goddesses made collective decisions with input from an expert panel, which consisted of Zeus (the president of the gods), Athena (the goddess of wisdom), Hermes (the god of information and commerce), and any other god whose area of expertise would be pertinent to the subject in question.
In their Working methodology
• Meetings were problem-oriented participatory sessions, characterized by intense discussions and searches for best solution.
• The gods' decisions were persuasively communicated to mortals and powerfully implemented with follow-up reports (Theofanides 1999).
• The Olympian style of management and decision making is illustrated in the steps below:-
1. Identify the problem or theme for action. Collect all relevant information and data, through the intelligence work by Hermes, the god of informatics.
2. Search for solutions via dialogue with all participants. Discuss step 1 with all concerned parties and propose alternative solutions.
3. Select the best problem solution or action theme, mainly by conferring with the concerned party or parties
4. Announce the decision of the gods to all mortals concerned through the god of informatics, Hermes. Send Peitho, the goddess of persuasion, to illuminate the best solution in step 3 as the decision of the gods of Olympus
5. Use lightning and thunderbolts to implement the Olympian decisions in step 4 to achieve the desired goals identified in steps 1 and 3.
6. Implement all decisions, supervised by Hermes, the god of informatics, who announces to the Olympian gods the results of their action taken in step
In contemporary times….
• Monitoring and evaluation (M&E) is an essential part of any program, project, policy and is often used in all kinds of contexts.
• Development partners increasingly expect their partners to have sound M and E systems
• Monitoring and evaluation can tell us :-
• Whether a program, policy or project is making a difference and for whom;
• It can identify program areas that are on target or aspects of a program that need to be adjusted or replaced.
• Information gained from M&E can lead to better decisions about program investments.
• It can demonstrate to program implementers and funders that their investments are paying off
• It is a tool for ensuring accountability to other stakeholders
Monitoring and evaluation can:
• Help identify problems and their causes;
• Suggest possible solutions to problems;
• Raise questions about assumptions and strategy;
• Push you to reflect on where you are going and how you are getting there;
• Provide you with information and insight;
• Encourage you to act on the information and insight;
• Increase the likelihood that you will make a positive development difference.
We conduct evaluations….
To help people make better decisions and achieve better outcomes
To provide better services (public and private)
• By: Comparing policy options objectively and rigorously
Calculating empirically the size of likely impacts
Calculating empirically the diversity/variance of impacts
Getting more precise estimates of risk and bias
Establishing a cumulative evidence base for decision making
1. Does it work?
2. How well does it work?
3. Does it do what we want it to?
4. Does it work for the reasons we think it does?
5. Is it cost effective?
6. Are the benefits worth it?
7. What are the unintended consequences?
An evaluation answers questions such as….
Challenges facing developing countries in M and E
Masuku, Ngengeezi and Ijeoma EOC (2015: 5) report the following:-
1. Designing M and E
2. Context Challenges
3. Cooperation and coordination
4. Institutional challenges
….
5. Lack of stakeholder involvement
6. Compliance
7. Linking planning, budget, priorities and M and E
8. Lack of integration with other strategic approaches
Other Challenges include:-
1. Capacity challenges
2. Poor coordination
3. Lack of legislative structures
4. Locus and focus problems
5. Elite capture vs stakeholder involvement
6. Absence of theory of change
7. Lack of evidence and truth
Types of Evaluation
• Evaluation Type by Level
1) Project-level evaluation
2) Program-level evaluation
3) Sector program evaluation
4) Thematic evaluation
5) Policy-level evaluation
• Evaluation Types by Stages of Project Cycle1) Ex-ante Evaluation 2) Mid-term evaluation3) Terminal evaluation4) Impact evaluation5) Ex-post evaluation
Types of evaluation based on the results chain
1. Context evaluation
2. Input evaluation
3. Process evaluation
4. Output evaluation
5. Outcome evaluation
6. Impact evaluation
Professionalization of evaluation• By 2010, there were more than 65 national
and regional evaluation organizations throughout the world, most in developing countries
• Although specialized training programs have existed for several decades, graduate degree programs in evaluation have emerged only recently
– Australasia– Africa– Canada– Central America– Europe (not every country)– Japan– Malaysia– United Kingdom
Professional Standards
• Utility
• Feasibility
• Propriety
• Accuracy
• Evaluation Accountability
DISCIPLINES FROM WHICH EVALUATION BORROWS?
•‘Social Research Methods’•Sociology•Economics•Statistics•Development studies•Public Administration•Social Anthropology
•Education
•Project Management
•Management
•Engineering
•Policy Analysis
•History
Structure of an evaluation-Commissioner’s perspective
1. Concept Paper
2. RFQ/Proposal
3. Evaluation of EOI/Proposal
4. Contact negotiation
5. Providing contacts and support
6. Quality control
7. Providing information
8. Approval of the report
9. Discussion of results
10. Discussion of consequences
11. Managing implementation of recommendations
Structure of an evaluation-Evaluator’s Perspective
1. EOI/Proposal
2. Contract Negotiation
3. Planning workshop meeting
4. Clarifying organizational questions
5. Inception report
6. Data collection, analysis and interpretation
7. Continuous coordination and exchange of information among parties
8. Draft evaluation report
9. Final evaluation report
10. Closing workshop
11. Follow-up
Policy Maker’s Perspective???
1. Emerging questions
2. Directives on how to address the questions
3. Participation as respondents
4. Participation in workshops discussing results
5. Utilization of evaluation results
6. Change in policy as a result of the evaluation
Tasks
1. As board members, what comes in your mind when you hear or read
about Monitoring and evaluation?
2. What came into your mind when you were told you were going to do a
training in M and E?
Principles of Evaluation
In every evaluation, it is clear certain questions have to be addressed:-
1. What is to be evaluated (evaluands)
2. Why should the evaluation be conducted (purpose)
3. What criteria should be applied
4. Who should conduct the evaluation
5. When should the evaluation be conducted
6. How should the evaluation be conducted
1. What do we evaluate?
•The things to be evaluated (evaluands) now days range from :-• Laws•Products •Services •Organizations •People •Processes •Social state of affairs of any kind (Stockman & Meyer 2013:67)
• On the political Global level:
1. Global Goals (the Millennium Development Goals)
2. International Conventions (the Geneva Conventions)
3. DECD Standards?
Millennium Development Goals
1) Eradicate extreme poverty and hunger
2) Achieve universal primary education
3) Promote gender equality and empower women
4) Reduce child mortality
5) Improve maternal health
6) Combat HIV/AIDS, malaria, and other diseases
7) Ensure environmental sustainability
8) Develop a global partnership for development
• On the political Africa level
1. African political federation
2. African Union peace and military initiatives
3. African participation in ICC
4. African Governance Mechanisms (APRM)
5. New Partnership for African Development (NEPAD)
• On political regional level (East African Community level)
1. East African political federation
2. East African Customs union
3. Northern Corridor interventions
4. Oil pipeline interventions
5. Standard railway-guage interventions
On the political country level
1. National Development Plans
2. Vision 2040
3. The NRM Manifesto (2011-2016)
4. Strategies ( NAADS interventions, Basket funding strategies)
• The Programme Level
• Things become a bit more concrete when we move on to the programme level
• Interventions take place and are followed up by Monitoring & Evaluation.
• The idea here is to assess how programmes work.
• The following are the most common examples of evaluation objects:
• Programmes
• Projects
• Single Measures/Activities/Outputs
• Competences/Resources/Inputs
• The System Level (Board plays a leading role)
• At the system level, things again become more abstract and less easy to handle.
• Typical objects are:
• Structures/Systems
• Networks
• Organisations
• Institutions
• Rules/Norms/Curricula/Agreements
• The Performance Level
The question shifts to the way a policy/ Programme/system/intervention/etc. evolves.
Nowadays there is a much greater focus on the performance aspect of programmes (and also on the systemic view) than in former times.
The main objects of performance evaluations are:
1. Results/Impacts
2. Performances/Processes
3. Management/Governance
4. Operations/Actions
• The Individual Level
• The assessment focuses on either group processes or individual behaviour and the attitudes behind them.
• So the main objects are:
1) Interaction/Group Behaviour
2) Communication/Exchange –
3) Individual Behaviour
4) Cognitive Processes/Attitudes
2. Why evaluation?
1. Providing gainful insights
2. Exercising control
3. Initiation of development and learning processes
4. Legitimization of measures, projects or prgrammes implemented
5. Accountability roles
6. Observe implementation processes
7. Assessing the feasibility of a programme (programme development phase-formative)
8. Supporting Managers in management (during implemenation phases)
3. What assessment criteria?
• Development Assistance Committee (DAC) of the Organisation for Economic Cooperation and Development(OECD) has developed criteria towards which many national organizations orient themselves
A. Relevance
B. Effectiveness
C. Efficiency
D. Impact
E. Sustaianability
Note
• If there are standards like the ones of DAC, they are directly stated by the client
• Sometimes, it is left to the evaluator to determine the criteria as he or she is considered to be an expert who ought to know what criteria best on what is to be evaluated-this is knowledge-or experience-based
• It is rare for the criteria to be set by the target group
4. Who should conduct the evaluation?
• Evaluation is best thought of as a team effort.
• Although one person heads an evaluation team and has primary responsibility for the project, this individual will need assistance from others on your staff.
• An evaluation team will work together on the following tasks:
1. Determining the focus and design of the evaluation.
2. Developing the evaluation plan, performance indicators, and data collection instruments.
3. Collecting, analyzing, and interpreting data.
4. Preparing the report on evaluation findings.
Options on who to conduct evaluation
1) Hiring an outside evaluator (option 1).
2) Using an in-house evaluation team supported by an outside consultant and program staff (option 2).
3) Using an in-house evaluation team supported by program staff (option 3).
Note:-
•Evaluators are diverse:•They might be economists concerned with efficiency and costs; •Management consultants interested in the smooth running of the organization; •Policy analysts with a commitment to public sector reforms and transparency; •Scientists concerned to establish truth, generate new knowledge and confirm or
disconfirm hypothesis.
5. When should the evaluation be conducted?
1. Before the intervention
2. During the implementation
3. Mid-way the implementation process
4. After the implementation
6. How will the evaluation be conducted?
1. Scientific-oriented approaches
2. Management-oriented approaches
3. Participant-oriented approaches
4. Qualitative-oriented approaches
Session 2: Building Monitoring and Evaluation systems
Board establishes M and E System
• One may define an M&E system as a collection of people, procedures, data and technology that interact to provide timely information for authorized decision-makers
• M and E systems are systems used to monitor and evaluate a project, program or organization to see if it is on track to achieve its overall outcomes
If you have a project for example….
61
A good M/E system will
1.Monitor the use of project inputs
2.Monitor the effectiveness of the project implementation process
3. Monitor the production of project outputs
4.Assess project impacts on the target communities
5.Assess the effectiveness of project outputs in producing the intended short-term and long-term impacts.
6.Assess the extent to which these impacts can be attributed to the effects of the project.
Why build an M and E system?
1) Supports planning activities at the sectoral and program level
2) Provides information for a more efficient allocation of public funds
3) Facilitates program management`
4) Helps re-designing and improving programs
5) Promotes transparency and accountability
6) Enriches policy discussion by incorporating rigorous evidence
M and E Systems has 12 key features
1. Organizational Structures with M&E Functions
The M&E unit whose main purpose is to coordinate all the M&E functions.
Some organizations prefer to outsource such services.
M&E unit should have its roles defined, its roles should be supported by the
organizations hierarchy and other units within the organization should be
aligned to support the M&E functions.
…
2. Human Capacity for M&E
An effective M&E implementation requires not only adequate staff but they must have the
necessary M&E technical know-how and experience.
Necessary to have the human resource that can run the M&E function by hiring employees
who have adequate knowledge and experience in M&E implementation
Ensure that the M&E capacity of these employees are continuously developed through
training and other capacity building initiatives to ensure that they keep up with current and
emerging trends in the field
….
3. Partnerships for Planning, Coordinating and Managing the M&E System
A prerequisite for successful M&E systems whether at organizational or national levels is the existence of M&E partnerships.
Partnerships for M&E systems complement the organization’s M&E efforts in the M&E process and they act as a source of verification for whether M&E functions align to intended objectives.
Partnerships also serve auditing purposes where line ministries, technical working groups, communities and other stakeholders are able to compare M&E outputs with reported outputs.
….
4. M&E frameworks/Logical Framework
The M&E framework outlines the objectives, inputs, outputs and outcomes of the intended project and the indicators that will be used to measure all these.
It also outlines the assumptions that the M&E system will adopt.
The M&E framework is essential as it links the objectives with the process and enables the M&E expert know what to measure and how to measure it.
…
5. M&E Work Plan and costs
Closely related to the M&E frameworks is the M&E Work plan and costs.
While the framework outlines objectives, inputs, outputs and outcomes of the intended project, the work plan outlines how the resources that have been allocated for the M&E functions will be used to achieve the goals of M&E.
The work plan shows how personnel, time, materials and money will be used to achieve the set M&E functions
….
6. Communication, Advocacy and Culture for M&E
This refers to the presence of policies and strategies within the organization to promote M&E functions.
Without continuous communication and advocacy initiatives within the organization to promote M&E, it is difficult to entrench the M&E culture within the organization.
Such communication and strategies need to be supported by the organizations hierarchy.
The existence of an organizational M&E policy, together with the continuous use of the M&E system outputs on communication channels are some of the ways of improving communication, advocacy and culture for M&E
….
7. Routine Programme Monitoring
M&E consists of two major aspects: monitoring and evaluation.
This component emphasizes the importance of monitoring.
Monitoring refers to the continuous and routine data collection that takes place during project implementation.
Data needs to be collected and reported on a continuous basis to show whether the project activities are driving towards meeting the set objectives.
They also need to be integrated into the program activities for routine gathering and analysis.
….
8. Surveys and Surveillance
This involves majorly the national level M&E plans and entails how frequently relevant national surveys are conducted in the country.
National surveys and surveillance needs to be conducted frequently and used to evaluate progress of related projects.
For example, for HIV and AIDS national M&E plans, there needs to be HIV related surveys carried at last bi-annually and used to measure HIV indicators at the national level.
…
9. National and Sub-national databases
The data world is gradually becoming open source.
More and more entities are seeking data that are relevant for their purposes.
The need for M&E systems to make data available can therefore not be over-emphasized.
This implies that M&E systems need to develop strategies of submitting relevant, reliable and valid data to national and sub-national databases.
…
10. Supportive Supervision and Data Auditing
Every M&E system needs a plan for supervision and data auditing.
Supportive supervision implies that an individual or organization is able to supervise regularly the M&E processes in such a way that the supervisor offers suggestions on ways of improvement.
Data auditing implies that the data is subjected to verification to ensure its reliability and validity.
Supportive supervision is important since it ensures the M&E process is run efficiently, while data auditing is crucial since all project decisions are based on the data collected.
…
11. Evaluation and Research
One aspect of M&E is research and the other is evaluation.
Evaluation of projects is done at specific times most often mid- term and at the end of the project.
Evaluation is an important component of M&E as it establishes whether the project has met he desired objectives.
It usually provides for organizational learning and sharing of successes with other stakeholders.
….
12. Data Dissemination and use
The information that is gathered during the project implementation phase needs to be used to inform future activities, either to reinforce the implemented strategy or to change it.
Additionally, results of both monitoring and evaluation outputs need to be shared out to relevant stakeholders for accountability purposes.
Organizations must therefore ensure that there is an information dissemination plan either in the M&E plan, Work plan or both.
Session three: The Evaluation Process and Role of the Board
The evaluation manager’s role, in consultation
with the steering
committee
The evaluator’s role
Clarify policy/ programme
objectives and intended outcomes
Clarify intended evaluation
purpose, users and uses
Develop relevant evaluation questions
Select the evaluation
approach and methods
Identify data sources and
collection and analysis procedures
Identify the necessary
resources and governance
arrangements
Prepare the TOR; commission (and possibly tender) the
evaluation
Conduct and Manage the evaluation
Our main focus in this session
The evaluation process – an overview
Terms of Reference
START
Evaluation Assessment
(2-3 months)
Contracting(3-4 months)
Field Work/ Analysis
(6-8 months)Report & Recommendations
(2-4 months)
Management Responses(1 month)
Executive Approval;
Internet Posting(2-3 months)
Implementing Change/Follow Up
Large evaluations typically take 12-18 months to
complete. Some phases may overlap.
Cycle of conducting and managing evaluations
Standards
Utility
Feasibility
Propriety
Accuracy
Engage Stakeholders
Focus the
Evaluation
Design
Describe the
program
Gather credible
evidence
Justify
conclusions
Use and share
lessons learned
Steps
Steps of the Evaluation process
Step 1- Engage Stakeholders
Who are the stakeholders?
Those involved in program operations, those affected by the program operations, and
users of evaluation results
Step 2 - Describe the Program
What are the goals and specific aims of the program?
What problem or need is it designed to address?
What are the measurable objectives? What are the strategies to achieve the objectives?
What are the expected effects?
What are the resources and activities?
How is the program supposed to work?
What do you want to know?
Consider the purpose, uses,
questions, methods, roles,
budgets, deliverables etc.
An evaluation cannot
answer all questions
For all stakeholders.
Step 3 - Focus the evaluation design
Data collected must address the evaluation questions Evidence must be believable, trustworthy and relevant
Information scope, sources, quality, logistics, methodology & data collection.
Who is studied and when?
Step 4 - Gather credible evidence
Consider the data you have:
• Analysis and synthesis
- determine findings.
• Interpretation
- what do findings mean?
• Judgments
- what is the value of findings based on accepted standards?
• Recommendations –
- what claims can be made?
- what are the limitations of your design?
Step 5 - “Justify” Conclusions
Share lessons learned with stakeholders!
Provide feedback, offer briefings. disseminate findings
Implement evaluation recommendations
Develop a new/revised implementation plan in partnership with stakeholders
Step 6 - Use and share results
Several options (not mutually exclusive)
Reconstructing baseline data ex post: recall method (more later)
Use key informants and triangulate (mostly qualitative)
Reconstruct a baseline “scenario” with secondary data (not always practical given absence and quality of baseline studies)
Single difference with econometric techniques: some practical obstacles (workload, time constraints, availability of trained specialists)
Dealing with lack of baseline data
Identify the implementation logic and theory of change
Allow for the inception report phase
Deal with missing baseline and other gaps
Gather data
Examine the effort using various criteria
Draw conclusions and recommendations
Conduct reporting
Ensure quality
Feedback on the evaluation
Management response
Dissemination findings
Feedback and lessons learnt
Best practices in managing evaluations
If I had know too much
information would make it complicated, I wouldn’t have asked for it!!!
• Value for money a key concern
• Underfunding as wasteful as over-funding
• Balance between cost and quality
• Quality ultimately more important
• But also relevance for purpose
• Make sure all aspects adequately funded including consultation with stakeholders, reporting and dissemination
• Ensure evaluation design appropriate to budget as well as aims of programme
Evaluation budget
Conclusions
1) Monitoring and evaluation is now a condition for development partners and government collaboration
2) Board’s oversight role is well done if there are functioning M and E systems
3) International standards need to be integrated into the local M and E systems
4) M and E units need to be adequately staffed and their capacity enhanced
5) M and E budget needs to be supported by the board