final outline plan for webinar evaluation and impact assessment mof 2004
TRANSCRIPT
www.wateraid.org
Webinar on final evaluation and impact assessment of
Governance and Transparency Fund ProgrammeTuesday 23rd April 2013
www.wateraid.org
Costa RicaHondurasNicaraguaGuatemala
Burkina Faso Ghana Mali
Nigeria
Ethiopia Kenya
MadagascarMalawi
Uganda Zambia
India Bangladesh
WA/FAN GTF programme
www.wateraid.org
Background and purpose of the two exercises
• The evaluation is primarily for Accountability and the impact assessment is primarily for Learning
• Complementary exercises
• CPs and partners are primary users of results
• Results for communication, fundraising etc
• Progress against the baseline data is critical
www.wateraid.org
Overview of all evaluation and learning processes and how they link together
Mid Term Review – WHAT so far? Evaluation – WHAT? Impact assessment – So WHAT? Learning review – HOW? Most significant change
analysis – the WHAT about the SO WHAT?
www.wateraid.org
Key Stakeholders in the process
DFID/KPMG
www.wateraid.org
Which country programme is doing what
www.wateraid.org
Different levels and how to deal with this
• 7 countries are doing a full scale evaluation • 9 countries are doing small scale evaluation• All countries are doing an impact
assessment except from Kenya
• Small scale = updating Mid Term Review• Full Scale = in depth assessment based on
key areas
www.wateraid.org
Length of the consultancy and how to use your time
• Total number of days = 25 to be shared between two exercises
• Rough Guideline: Step 1: Understanding the context - understanding of the
problem in country that GTF is addressing• Background reading - 1 day• Working with country prog staff and key informants - reinforcing
understanding of programme, stakeholders and intervention design, findings of MTR (if there was one) and conclusions - up to 3 days
Step 2: Enquiry – conducting self-assessment, semi-structured interviews, FGD…..8 days
Step 3: Analysis - Self-assessment collation of results, coding of qualitative data….2.5 accountability and 2.5 days for learning analysis;
Step 4: Write the first draft of the report - 4 days Step 5: Revisions and redraft of the report - 4 days
www.wateraid.org
Timeline Dates Actions 4th April ToR for Evaluation and Impact Assessment sent out to all countries
19th April Each country to sign contracts with local consultants 23th April Webinar with all consultants
May 17th Local consultants to submit draft Impact assessment section of report
May 31st Local consultants to submit draft Evaluation reports June 24th Local consultants to submit final Evaluation report including Impact
assessment July 11th MoF to submit global consolidated impact assessment
Week of July 22nd CM to submit draft Evaluation report and share report to KPMGWeek of July 29th Last Annual learning meeting
Mid Sept CM to submit Final Global Evaluation report
End of Sept /mid October
Papa to share report with KPMG
End of October Submission of WaterAid PCR to KPMG
www.wateraid.org
The difference between an evaluation and impact
assessment
www.wateraid.org
In summary....Questions Monitoring Evaluation Impact Assessment
Why do we do it?
Measures on-going activities
Measures performance against objectives
Assesses change in peoples lives
What is the main focus?
Focus on programme interventions
Focus on programme interventions Focus on stakeholders
At what level? Outputs Outcomes/impact Impact and change
What are the key questions to ask?
•What is being done?•Is our programme progressing as planned?
•What happened? Did we achieve what we set out to achieve in terms of:
•Effectiveness•Efficiency •Relevance•Sustainability •Impact
•So what actually changed? •For whom? •How significant is it for them?•Will it last?•What, if anything, did our programme contribute?
www.wateraid.org
The Evaluation Component
Catherine Currie
www.wateraid.org
GTF Summative Evaluation• We are conducting a critical analysis of
the GTF Programme in order to assess whether or not it achieved its goals
Whether the planned activities occurred Whether the activities led to achievement of goals; How effective the project was; How costly the project was; etc.
• This is a summative or end of programme evaluation
www.wateraid.org
Purpose of evaluation
• For accountability- to enable beneficiaries, board members, etc to know how funds have been used;
• Our country evaluations will assess: – objectives against logframe targets and
milestones;– programme performance by OECD DAC
criteria of effectiveness, efficiency, relevance, sustainability, replicability and impact
www.wateraid.org
A useful Global Evaluation Report
• Top tips include:– i) the process of collation, analysis and write up– ii) enhanced rigour and comparability of results and
reports…………...so
• A consistent stance• Support and advice through online forum• Verification of evidence• Step-by-step validation of evaluation results• Quality assurance processes• Prioritise the use of country systems• Use a set of agreed working definitions for key terms • Use the WaterAid report template
www.wateraid.org
Evaluation QuestionsRelevance: • What we expect: Details of the programme’s significance
with respect to increasing voice, accountability and responsiveness within the local context.
Evaluation Questions:1. How well did the programme relate to governance priorities at
local, national or internal levels? Please demonstrate with examples in relation to: i)increasing voice; ii) accountability; and, iii) responsiveness within the local context.
2. How well did the programme relate to the Country Strategy Paper aims and objectives? Of WaterAid and where applicable of the FAN network – ie regional secretariats and of DFID.
3. How logical is the current theory of change?
www.wateraid.org
Effectiveness• What we expect: An assessment of how far the
intended outcomes were achieved in relation to targets set in the original logical framework.
Evaluation Questions:1. Have interventions achieved the objectives? At
country regional and global level.2. How effective and appropriate was the programme
approach? How effective was the MEL system and framework?
3. With hindsight, how could it have been improved?
www.wateraid.org
Partnership• What we expect: How well did the
partnership and management arrangements work and how did they develop over time? Please consider areas such as monitoring, evaluation and learning arrangements. If possible, consider from a regional perspective.
www.wateraid.org
Advocacy• What we expect: To what extent has GTF
contributed to WaterAid influencing targets? Evaluation Questions:1.How has the programme helped implement
successful advocacy strategies? Are there any lessons learned about measuring influencing.
2.How has the programme contributed to the overall in country advocacy strategy?
www.wateraid.org
Equity• What we expect: Discussion of social differentiation (e.g. by
gender, ethnicity, socio economic group, disability, etc) and the extent to which the programme had a positive impact (from an accountability perspective) on the more disadvantaged groups.
Evaluation Questions:1. How did the programme actively promote gender equality? 2. What was the impact of the programme on children, youth and
the elderly? 3. What was the impact of the programme on ethnic minorities? 4. If the programme involved work with children, how were child
protection issues addressed? 5. How were the needs of excluded groups, including people with
disabilities and people living with HIV/AIDS addressed within the programme?
www.wateraid.org
Value for Money• What we expect: Good value for money is the optimal use of
resources to achieve the intended outcome. Evaluation Questions:1. Has economy been achieved in the implementation of
programme activities?2. Could the same inputs have been purchased for less money?3. Were salaries and other expenditures appropriate to the
context?4. What are the costs and benefits of this programme? 5. Is there an optimum balance between Economy, Efficiency
and Effectiveness? Overall, did the programme represent good value for money?
www.wateraid.org
Efficiency• What we expect: How far funding, personnel, regulatory,
administrative, time, other resources and procedures contributed to or hindered the achievement of outputs.
Evaluation Questions:1. Are there obvious links between significant expenditures and
key programme outputs? How well did the partnership and management arrangements work and how did they develop over time?
2. How well did the financial systems work? 3. Were the risks properly identified and well managed? 4. For advice on measuring value for money in governance
programmes see DFID’s Briefing Note (July 2011) Indicators and VFM in Governance Programming, available at: www.dfid.gov.uk
www.wateraid.org
Sustainability• What we expect: Potential for the
continuation of the impact achieved and of the delivery mechanisms following the withdrawal of existing funding.
Evaluation Questions:1.What are the prospects for the benefits of the
programme being sustained after the funding stops? Did this match the intentions?
2.How have collaboration, networking and influencing of opinion support sustainability?
www.wateraid.org
Innovation & Replicability• What we expect: How replicable is the
process that introduced the changes/impact? Refer especially to innovative aspects which are replicable.
Evaluation Questions:1.What aspects of the programme are
replicable elsewhere? 2.Under what circumstances and/or in what
contexts would the programme be replicable?
www.wateraid.org
Expected impact and change• What we expect: Details of the broader economic, social, and political
consequences of the programme and how it contributed to the overall objectives of the Governance and Transparency Fund (increased capability, accountability and responsiveness) and to poverty reduction.
Evaluation Questions:1. It is critical to demonstrate the progress in relation to the indicators included in the
GTF programme logframe. The focus is on accountability for the impact. 2. What was the programme’s overall impact and how does this compare with what was
expected? Please demonstrate from an accountability perspective if the perceived impact was achieved and if not, why not.
3. Did the programme address the intended target group and what was the actual coverage? Again from an accountability perspective, was the coverage reached? If not, why not, if yes, how?
4. Who were the direct and indirect/wider beneficiaries of the programme? Again, the importance here is to set out who these were for accountability purposes.
5. What difference has been made to the lives of those involved in the programme? Describe the impact.
6. As you are aware, the Consultant is also conducting more detailed critical analysis on Impact for learning purposes.
www.wateraid.org
Vertical Logic of ProgrammeImpact is the higher level situation that the project
contributes towards achieving
Outcome identifies what will change and who benefits during the lifetime of the
project
Outputs are specific deliverables
Human Resource and financial inputs
LEARNING:For the GTF Global
Consultants this requires evidence of :
‘so what’?ACCOUNTABILITYFor the GTF Global
Consultants this requires evidence against
programme specific objectives
www.wateraid.org
The Impact Assessment Component
www.wateraid.org
Why conduct the impact assessment component?
• To learn and improve: • To enable Country programme staff, stakeholders in
country, Wateraid staff and others to really understand what changed as a result of the programme and to apply this to future plans
• To test and refine our understanding of how change happens and how successful we have been in supporting positive changes for our stakeholders:• To what extent did we work with the right people? In the right way? How did
this all link up ?• To what extent did the changes we expected to see along the way support
the long term changes we were aiming to influence?• What does this tell us about the way we think we can influence change? • What should we do differently next time?
www.wateraid.org
The Impact Assessment – Focus on the “so what question”
• what’s changed? • For whom? • How significant/lasting are these changes for
different stakeholder groups? • In what ways did the programme contribute
– Expect the unexpected - we are looking for evidence of positive/negative/ intended and unintended changes,
– Prioritise analysis over gathering information – need for open and probing questions
www.wateraid.org
Key questions for the Impact Assessment
www.wateraid.org
Background and context – what we need to know (Country Programme Theory of
Change):• The local and national context, including key social,
political and environmental conditions and how they have changed over the life time of the programme
• Key issues that the programme planned to address• The target groups who would ultimately benefit from
the programme and how each would benefit?• The process or sequence of changes that would lead
to the desired long-term goal • The assumptions that the programme made about the
anticipated process of change• The other actors/factors who had the potential to
influence the changes sought, both positively or negatively.
www.wateraid.org
Four Domains of Change
1. Changes in the ways in which CSOs function and network, and their capacity to influence the design, implementation and evaluation of effective WASH policies at all levels
2. Changes in the ways that CSOs, including those representing marginalised groups, are able to engage in decision-making processes affecting the WASH sector.
3. Changes in the ways in which members of local communities demand accountability and responsiveness from governments and service providers in the WASH sector
4. Changes in the ways that Governments and service providers are accountable to citizens and end users in the WASH sector
www.wateraid.org
Each Domain is broken down further into “areas of enquiry”
These are the key questions you need to explore across all Domains:
1. What has actually changed for each of the different stakeholder groups, especially the poorest and most marginalized communities in relation to WASH (positive, negative, intended and/or unintended changes)
2. How significant and/or sustainable are these changes for the different target groups?
3. To what extent do these changes compare with baselines and changes that were planned and expected?
4. How do they link together and/or influence each other? 5. To what extent did the GTF programme contribute to these changes?
How? 6. Who or what else might have contributed to these changes? How?7. How confident are you in these findings (levels of evidence)?
www.wateraid.org
Areas of Enquiry Domain 1
Domain 1 Key Areas of Enquiry
Changes in the ways in which CSOs function and network, and their capacity to influence the design, implementation and evaluation of effective WASH policies at all levels
Ways in which networks have developed and function over time
Shifts in CSO capacity How this capacity change has influenced
policy and practice at o local levelso National level
Note: we will provide further guidelines on how to assess these areas of enquiry in the next week
www.wateraid.org
Areas of Enquiry Domain 2
• PDomain 2 Key Areas of Enquiry
Changes in the ways that CSOs, including those representing marginalized groups, are able to engage in decision-making processes affecting the WASH sector.
Shift in awareness, knowledge and confidence of marginalized groups
Shifts in the ways that people have been able to demand their rights
The extent to which the voices of marginalized people are making a difference to policy and practice
Ways in which different CSO strategies have influenced change (e.g. budget tracking, participation in stakeholder reviews, etc…)
Note: we will provide further guidelines on how to assess these areas of enquiry in the next week
www.wateraid.org
Areas of Enquiry Domain 3
Domain 3 Key Areas of Enquiry
Changes in the ways in which members of local communities demand accountability and responsiveness from governments and service providers in the WASH sector
• Levels of awareness of rights in local communities
• Ways in which media coverage supports understanding of rights
• Ways in which citizens are influencing policy and practice over time
• Changes in community access to WASH • Changes in community influence over
natural resources
Note: we will provide further guidelines on how to assess these areas of enquiry in the next week
www.wateraid.org
Areas of Enquiry Domain 4
Domain 4 Key Areas of Enquiry
Changes in the ways that Governments and service providers are accountable to citizens and end users in the WASH sector
Changing levels of governance, transparency and compliance
Changes in policy and regulation (e.g. new policies, laws, standards, political and institutional framework) – and the consequences of these
Changes in practice relating to WASH (e.g. delivery of new services and systems) and the consequences
Note: we will provide further guidelines on how to assess these areas of enquiry in the next week
www.wateraid.org
Methodology for both components Restrict yourselves to using a few tried and tested
tools. We suggest:– Facilitated self assessment: building on MTR which will
support the evaluation component• How to do this and who should be involved • Note: we will be adding some more change questions this time
– Follow up workshop to validate findings and focus on the impact assessment element
• How to do this and who should be involved– Other in depths interviews /FGD with key informants as
required • This might enable a deeper understanding of e.g. how changes
affected particular target groups
www.wateraid.org
Guiding Principles for Methodology
• Create an atmosphere where informants feel able to be honest and provide critical feedback. Use an appreciative enquiry approach
• Ensure a mix of both qualitative and quantitative data is gathered
• For the impact assessment – ask open and probing questions for a deeper understanding of change
• Findings must be backed up with evidence and be set against the original baseline
www.wateraid.org
Consideration of sample sizeQuestions: • How many people to interview?• What is a “good enough sample?”Answers: • Be pragmatic (you have limited time but need to be
representative). • Plan with in country focal point:
– Include people/groups/interventions which represent • good/strong• Medium• Poor/weak
• Explain your sampling decisions in the methodology section of the report ( with an indication of the level of rigour you believe this provides)
www.wateraid.org
The Report We will provide more detailed guidance over the next week but this is
the guide35 -40 pages to include:• Executive Summaries x 2 - (4 pages in total) • Contents + Abbreviations (1 page)• Methodology + challenges and limitations (2 pages )• Country Context and introduction to the programme (3 pages )• Evaluation Report (10 pages) – findings and conclusions under the
following headings:– Relevance, Effectiveness, Partnership, Advocacy Equity, Value for
Money, Efficiency, Sustainability, Innovation and Replicability, Expected Impact and change.
• Impact Assessment Report (10 pages) – findings and conclusions under the following headings: – Changes under each Domain– Overall analysis of impact for different target groups– What difference the programme has made overall
• Overall conclusions and learning for Country Programmes, for the sector and globally (4 pages)
• Annexes
www.wateraid.org
Other ways to present findings
• Opportunity for Wateraid to share findings and learning with a wide group of stakeholders -
• Target groups:– Country programme staff– Networks in country– Partners– Wateraid donors– Sector specialists
• Supplementary ways of presenting findings (optional)– Case studies– Video footage– Photos
www.wateraid.org
Next steps• Maureen and Catherine to send more
detailed guidelines by Friday April 26th • In country evaluation team to take stock of
the outcomes of the webinar and• Prepare and send proposal to Catherine and
Maureen copied to Marta and Papa by Friday April 26th with a brief overview of your plan including:– Time line– Your methodology– Key informants – Sample size and rationale for this
www.wateraid.org
Support and assistance• Guidance notes to follow:
– Some thematic guidance– Self-assessment format to use– Learning questions– Report format
• Online forum: you will be able to post questions, debate issues and findings with other consultants involved in the GTF final evaluation and impact assessment.– Please expect an email in your inbox with a password and user details from
either Catherine or her colleague Erica Packington Erica will manage access to the private forum.
– Please action the email immediately to guarantee access.• E mail contacts:
– Technical advice• Catherine Currie [email protected]• Maureen O’Flynn [email protected]
– Logistics:• Papa Diouf [email protected]• Marta Barcelo [email protected]
Please copy everything to Marta
www.wateraid.org
Questions?