changing the way the new zealand aid programme monitors and evaluates its aid ingrid van aalst...
TRANSCRIPT
Changing the way the New Zealand Aid Programme monitors and evaluates its Aid
Ingrid van Aalst
Principal Evaluation Manager
Development Strategy & Effectiveness Division
Key national environmental drivers
The New Zealand Government’s priorities
enhancing the operational effectiveness of ODA focus on effective aid that supports sustainable economic development priority to the Pacific Region and targeted approach to Asia, very small
programmes in Africa and LA focus on NZ’s comparative advantage and to address our partner’s needs requirements for greater accountability for the use of public money and
demonstrate value for money
Key international environmental drivers
the effectiveness of aid expenditure is now at the forefront of development thinking
transparency, accountability, and confidence that significant aid investments are both effective and efficient in fulfilling their objectives
commitments to Paris Declaration on Aid Effectiveness, the Accra Agenda for Action and the Millennium Development Goals, with better monitoring and evaluation for both donors and development partners
An effective and focussed monitoring and evaluation function
contribute to our understanding of whether our aid interventions are effective, efficient and provide value for money
help us improve activities, programmes, and how we deliver aid what is working effectively (or not) & why what progress are we making towards outcomes
be aligned to the New Zealand Aid Programme’s core strategic and operational goals, and priorities - be at the forefront and integrated into our work
support accountable requirements for spending by providing credible, independent and objective assessments of New Zealand Aid Programme’s key strategies, policies, sectors, programmes and activities - we need to better demonstrate results or “tell our story”
be integrated with the New Zealand Aid Programme’s internal and external planning and reporting
New Zealand Aid Programme’s Performance Framework
Planning · Statement of Intent · Information Supporting
Estimates · MFAT Strategic Directions
Planning · Programme Plans · Sector Plans(??)· Joint Commitment for
Development / Country Strategic Frameworks (incld Results Frameworks)
Planning · Activity Designs (incld
Results Frameworks)
Reporting NZ Accountability · MFAT Annual Report · Section 32 A Report International Reporting · Aid Effectiveness Reporting
(e.g. Cairns Compact)Internal Management · Quality Management
Review Report· Dashboard · Aid Programme Focus and
Concentration Report
Reporting· MFAT Annual Report · Programme Reports· Sector Reports(??)· JCfD / CSF Annual Reviews· Aid Programme Focus and
Concentration Report
Reporting · MFAT Annual Report · AMAs and ACAs· Implementing Partner
Reporting (based on Results Framework)
Level 1- New Zealand Aid Programme
Level 2 - Programme (including Sector Strategies)
Level 3 - Activities
Qu
alit
y A
ssu
ran
ce
Qu
ality
Man
agem
ent R
evi
ew
In
tern
al A
udi
t
New Zealand Aid Programme Performance Framework
International Development
PolicyStatement
NZ Aid Programme
Strategy
Individual Performance Plans Divisional Operational Plans
Str
ateg
ic E
valu
ati
on
Wo
rk P
rog
ram
me
Pro
gra
mm
e, P
olic
y, S
ect
or
and
Re
spo
nsiv
e (
Ma
nage
d b
y D
SE
)
Monitoring &
Evaluation
Ac
tiv
ity
Ev
alu
atio
ns
Pri
ma
rily
man
aged
by
Pro
gram
mes
Monitoring &
Evaluation
Monitoring &
Evaluation
New Zealand Aid Programme’s monitoring and evaluation
Development, Strategy and Effectiveness Division
New Zealand Aid Programme’s strategy and policy, work programme, planning, performance and reporting
monitoring, evaluation and research
cross cutting co-ordination
procurement
process improvement, policy coherence, development outreach
best practice programme and activity management
The Evaluation team
developing and managing a strategic evaluation work programme
building evaluation capacity in and across the New Zealand Aid Programme, including at activity level
reviewing and developing evaluation policies, strategy, plans, processes and systems
The New Zealand Aid Programme’s strategic evaluation work programme
Four strands in the work programme
Sector Programme – The New Zealand Aid Programme has 24 Programmes Policy Responsive
a rolling programme of 5-7 evaluations each year across a number of years report to the Evaluation and Research Governance Group anticipate participating in a number of joint evaluations focus first is on developing the strategic evaluation work programme
Activity level monitoring and evaluation
simple and manageable monitoring and evaluation approach, which builds on previous monitoring and evaluation approaches
supports design of the activity at start and during implementation
greater focus on results (outputs, outcomes and goal) - away from inputs and activities
imbedded into the Activity lifecycle management processes and systems - ‘part of business as usual’
activity monitoring and evaluation is the responsibility of the Programme Teams
Activity Results Framework
Results Diagram identifies logic supporting the Activity and intended results
Results Measurement Table how results will be monitored and measured, including indicators and
targets
Monitoring and Evaluation Work Plan tasks and resources required to support monitoring and evaluating
results
Increased use of clean water
Improved health of community
Key Outputs
Long-term Outcomes
Activity’s Goal: For a healthier population contributing to a more prosperous country
Medium-term
Outcomes
Short-term Outcomes
Water management
training provided Wells Built
Community engagement /
training provided
Improved water quality
Improved water practices by community
Improved water management
processes
Water management
operations plan developed
Improved guidance for operations
Results Measurement Table
Results Indicators Baseline and Targets Methodology & Data Sources
Long-term Outcomes: Improved health of the Community
1. Cause specific morbidity rates (per 100,000)*2. Selected infectious diseases rates (per 100,000)*
Baseline: 450 (2005) and 470 (2010)Target: 460 (2012) and 400 (2015)Baseline: 460 (2005) and 480 (2010)Target: 480 (2012) and 420 (2015)
Information is available each October from the MoH. There is a one year lag in available information.
Medium-term Outcome: Increased use of clean water
1. % of households making use of clean water
Baseline: no information currently available Target: all households using clean water (2013)
Information will be collected through a specifically designed survey and captured by the MoH. Available each November.
Short-term Outcome: Improved water quality
1. % of water testing samples that comply with agreed quality standards
2. Incidence water born diseases in the community*
Baseline: 20% of samples comply with quality standards Target: 90% of samples comply (2012)
Baseline: 60 incidence per year Target: 30 incidence (2012) and less than 10 incidence (2015)
Information will be collected through a specifically designed audit process and captured by the MoH.
Output: Wells Built
1. Wells built to specification to time and to budget^
Baseline: no information available Target: Compliance achieved (2012)
Information will be collected through a specifically designed audit process and captured by the MoH. Information will be available in November.
M&E Work Plan
Tasks Approach (methods, processes)
Timeline Roles and Responsibilities
Deliverables and Reporting
Indicative Costs
Evaluation (Note: must be based on soon to be released Evaluation Policy and Guidelines)
Plan and undertake year three evaluation
4.1 Develop terms of reference
Identify in consultation with key stakeholders the purpose, scope, objectives / Document and confirm ToR with the Steering Committee
June-July 2012 Lead: MFAT Documented and agreed terms of reference
TA funding: 2 days ($4k)
4.2 Consult with stakeholders and develop eval. Plan
Identify and contract evaluator / Develop and agree eval. Plan
June –July 2012 Lead: MFAT Agreed and high quality eval. Plan
Funding for contract evaluator: $16k
4.3 Undertake year three evaluation
Complete the evaluation June- Aug 2013 Lead: Lead Evaluator Eval report signed-off by Eval. Steering Group
Funding for contract evaluator (s): $60k
4.4 Year three evaluation: Report results and develop action plan to strengthen on-going delivery
a) Present results and b) Workshop to determine response to results and how services will be strengthened
Sep 13 Lead: Evaluator Presentation on the results / Workshop: response to the evaluation, work plan to strengthen services
Grant funding: $20k
Overall Monitoring and Evaluation Budget Grant Funding $106k
TA Funding $14k
Full Funding $120k
New Zealand Aid Programme’s Performance Framework
Planning · Statement of Intent · Information Supporting
Estimates · MFAT Strategic Directions
Planning · Programme Plans · Sector Plans(??)· Joint Commitment for
Development / Country Strategic Frameworks (incld Results Frameworks)
Planning · Activity Designs (incld
Results Frameworks)
Reporting NZ Accountability · MFAT Annual Report · Section 32 A Report International Reporting · Aid Effectiveness Reporting
(e.g. Cairns Compact)Internal Management · Quality Management
Review Report· Dashboard · Aid Programme Focus and
Concentration Report
Reporting· MFAT Annual Report · Programme Reports· Sector Reports(??)· JCfD / CSF Annual Reviews· Aid Programme Focus and
Concentration Report
Reporting · MFAT Annual Report · AMAs and ACAs· Implementing Partner
Reporting (based on Results Framework)
Level 1- New Zealand Aid Programme
Level 2 - Programme (including Sector Strategies)
Level 3 - Activities
Qu
alit
y A
ssu
ran
ce
Qu
ality
Man
agem
ent R
evi
ew
In
tern
al A
udi
t
New Zealand Aid Programme Performance Framework
International Development
PolicyStatement
NZ Aid Programme
Strategy
Individual Performance Plans Divisional Operational Plans
Str
ateg
ic E
valu
ati
on
Wo
rk P
rog
ram
me
Pro
gra
mm
e, P
olic
y, S
ect
or
and
Re
spo
nsiv
e (
Ma
nage
d b
y D
SE
)
Monitoring &
Evaluation
Ac
tiv
ity
Ev
alu
atio
ns
Pri
ma
rily
man
aged
by
Pro
gram
mes
Monitoring &
Evaluation
Monitoring &
Evaluation
Building the monitoring and evaluation capacity and capability across the New Zealand Aid
Programme developing simple user-friendly, practice-oriented guidelines and templates covering all aspects of monitoring, and the process of commissioning and managing Activity evaluations
developing and delivering result monitoring and evaluation training
providing coaching and support for the more complex Activity evaluation challenges
re-establishing a Community of Practice for results monitoring and evaluation
supporting results monitoring and evaluation champions in each Division
Monitoring and evaluation policies, processes and systems
In the next 12 months
developing activity and programme results monitoring and evaluation policy, processes, systems and guidelines, and evaluation policy and strategy
establishing the Evaluation and Research Governance Group, and a panel/pool of results monitoring and evaluation providers
developing processes for quality review of evaluation deliverables; protecting the independence of evaluators and their reports; disseminating learnings; management response and action from evaluations; link to planning, reporting and management decision-making
Any questions?