effectiveness: improve it framework - ben jackson, bond
DESCRIPTION
TRANSCRIPT
bond.org.uk
Introduction to the Improve It Framework -
Effectiveness:
Improving and Proving It Conference
Edinburgh November 2012
Ben Jackson [email protected]
Chief Executive, Bond
Context
bond.org.uk 3
Consortium Effectiveness Programme
1. Developing agreement and supporting implementation of a sector wide
framework of common outcome indicators and data collection
tools (Improve it Framework); and online organisational self-
assessment and resource portal (Organisational Health Check;
NIDOS Effectiveness Tool); and agreed quality standards regarding
NGO evidence.
2. Building knowledge and skills to support members to measure and
manage effectiveness and value for money
3. Creating an enabling environment that encourages and supports
organisations to deliver improvements in their effectiveness
…supports UK NGOs in improving how they measure, manage and demonstrate their contribution to social change
bond.org.uk 4
Consortium Effectiveness Programme:
Key elements
• NIDOS Effectiveness Tool / Bond Organisational Health Check
• Value for Money
• NGO Evidence Principles
• Improve it Framework
bond.org.uk 5
Contributors to the programme
Consortium partners:
bond.org.uk 6
How the strands of programme fit together
Strong systems, culture, leadership etc
Bond Organisational Health Check/
NIDOS Effectiveness Tool
Robust, practical
approaches to measuring
effectiveness
Appropriate approaches to managing cost efficiencies & effectiveness
Development effectiveness
Improve It Framework
bond.org.uk
The Improve it Framework
What is it?
• A tool for improving how UK NGOs measure,
learn from and demonstrate outcomes and
impact
• A platform for sharing information on
outcomes, indicators and data collection tools
Why are we developing it?
• Developing robust methods for measuring
effectiveness is a collective challenge for UK
NGOs
8
• UK NGOs need to start telling a more robust story at sector level as well as
at organisational level about their added value
bond.org.uk 9
Improve it Framework: what it is and is not
• …about creating a single way of
assessing effectiveness –it’s about
encouraging greater harmonisation
and consistency where appropriate
• …going to offer an off the shelf
answer to measuring effectiveness,
but it will provide a common starting
point
• …going to produce an encyclopaedia
of indicators and tools. There will be
an element of prioritisation.
• …an organisation specific tool;
seeing 60% of your organisation in it
is good enough
• …about providing a collective resource
that UK NGOs can draw on when
developing their own context specific
M&E frameworks
• …about developing shared
approaches where we can
• ….about providing organisations with
the tools to be able to tell a more
robust story of how they are
contributing to change
• …going to be practical & simple
• …going to be dynamic & will continue
to evolve
It is not… It is…
bond.org.uk
1 2 3 4 5 6
Where the Improve it Framework can help in
your work
Focus of Improve it Framework
Identify the problem & underlying
causes
Articulate your vision of change
Identify who you will work
with & influence &
how
Identify what you want to
change
Identify the evidence
that you will collect and
how
Reflect and learn from the data
Communicate results
7 8
Collect, analyze and
draw conclusions for the data
Outcomes Indicators &
tools Principles
In practice
Capacity development
Improve it
Framework
1) All children access quality
education where they learn useful knowledge and
skills
2
Governments and other mandated bodies
ensure all children access quality
education
3
Schools are well managed, safe and well
resourced
4
All children receive their full allocation of high quality teaching
6
Communities actively support the education
of all children
Outcomes
a) Mothers, fathers and other caregivers support all their children to access education
b) Communities actively support and participate in children’s education
c) CSOs support and advocate for improved education for all
Outcomes
a) School management committees exist and manage schools effectively
b) School managers access, manage, and deploy resources in an effective and transparent way
c) Teachers, resources and
infrastructure are in place to
support education
d) Schools are safe and
healthy environments for girls
and boys
Outcomes
a) Teaching is high quality,
child-centred and child-
friendly
b) Teachers deliver a full
allocation of lessons
c) Teachers are trained and
supported to deliver high
quality teaching
d) The curriculum is relevant
and appropriate
e) Teaching supports equality
and empowerment
Domains of Change Framework for Education
Outcomes
a) Children participate in a meaningful way in decisions that affect their education
b) Children participate in their own and others’ learning
c) Children have a positive attitude towards education
Outcomes
a) Laws, policies and practices are in place that support quality education that is accessible for all children
b) Adequate resources are allocated and adequate teachers recruited and trained for the provision of quality education for all
c) Performance in education is monitored at a national and local level
5
All children participate in their education and
in school life
1) All children access quality
education where they learn useful knowledge and
skills
2
Governments and other mandated bodies
ensure all children access quality
education
3
Schools are well managed, safe and well
resourced
4
All children receive their full allocation of high quality teaching
6
Communities actively support the education
of all children
Outcomes
a) Mothers, fathers and other caregivers support all their children to access education
b) Communities actively support and participate in children’s education
c) CSOs support and advocate for improved education for all
Outcomes
a) School management committees exist and manage schools effectively
b) School managers access, manage, and deploy resources in an effective and transparent way
c) Teachers, resources and
infrastructure are in place to
support education
d) Schools are safe and
healthy environments for girls
and boys
Outcomes
a) Teaching is high quality,
child-centred and child-
friendly
b) Teachers deliver a full
allocation of lessons
c) Teachers are trained and
supported to deliver high
quality teaching
d) The curriculum is relevant
and appropriate
e) Teaching supports equality
and empowerment
Domains of Change Framework for Education
Outcomes
a) Children participate in a meaningful way in decisions that affect their education
b) Children participate in their own and others’ learning
c) Children have a positive attitude towards education
Outcomes
a) Laws, policies and practices are in place that support quality education that is accessible for all children
b) Adequate resources are allocated and adequate teachers recruited and trained for the provision of quality education for all
c) Performance in education is monitored at a national and local level
5
All children participate in their education and
in school life
1) All children access quality
education where they learn useful knowledge and
skills
2
Governments and other mandated bodies
ensure all children access quality
education
3
Schools are well managed, safe and well
resourced
4
All children receive their full allocation of high quality teaching
6
Communities actively support the education
of all children
Outcomes
a) Mothers, fathers and other caregivers support all their children to access education
b) Communities actively support and participate in children’s education
c) CSOs support and advocate for improved education for all
Outcomes
a) School management committees exist and manage schools effectively
b) School managers access, manage, and deploy resources in an effective and transparent way
c) Teachers, resources and
infrastructure are in place to
support education
d) Schools are safe and
healthy environments for girls
and boys
Outcomes
a) Teaching is high quality,
child-centred and child-
friendly
b) Teachers deliver a full
allocation of lessons
c) Teachers are trained and
supported to deliver high
quality teaching
d) The curriculum is relevant
and appropriate
e) Teaching supports equality
and empowerment
Domains of Change Framework for Education
Outcomes
a) Children participate in a meaningful way in decisions that affect their education
b) Children participate in their own and others’ learning
c) Children have a positive attitude towards education
Outcomes
a) Laws, policies and practices are in place that support quality education that is accessible for all children
b) Adequate resources are allocated and adequate teachers recruited and trained for the provision of quality education for all
c) Performance in education is monitored at a national and local level
5
All children participate in their education and
in school life
bond.org.uk 16
School management committees exist and manage schools effectively
Indicators Tools
• # and % of schools with a Parent Teacher
Association (PTA)
• Level of activity of the Parent Teacher
Association (PTA)
• # and % of schools with a School
management committee (SMC)
• Level of activity of the School
management committee (SMC)
The Right to Education indicators and form for section 9: right to transparent and accountable schools.
The Plan UK child friendly tool Section 1: School management
The VSO quality scale for education services quality areas on education management and community engagement
Timelines
bond.org.uk 22
What to look forward to
• Organisational Health Check pilot starts (Summer 2012)
• Launch of pilot Evidence Principles (October 2012)
• Beta version of the Improve it Framework (December 2012)
• Comic Relief, Big Lottery Fund, DFID, Bond research into the cost
of measuring effectiveness (Spring 2013)
• Guide to demystifying theories of change (Spring 2013)
bond.org.uk 23
Thank you
Contact:
Ben Jackson
Chief Executive
Regent’s Wharf • 8 All Saints Street • London • N1 9RL • United Kingdom
+44 (0)20 7837 8344 • [email protected] • bond.org.uk
bond.org.uk
What is it?
• An online self-assessment that:
• Identifies strengths and weaknesses in
organisational capacity / functionality
• Facilitates benchmarking with peers
• Sign posts to other initiatives, tools, training and
support
Why are we developing it?
• Strong systems, leadership etc are key building
blocks for delivering good development outcomes
24
Organisational Health Check
• An online self-assessment enables bond to provide more tailored
capacity support
• A common framework can help facilitate structured peer learning and the
sharing of best practice
bond.org.uk
Why are we doing it?
• At the heart of the VfM agenda are important questions about how UK NGOs
use their resources
• VfM is a part of the wider results agenda
• Want to ensure VfM remains appropriate and proportionate to NGO
operations and programmes
Value for Money (VfM)
What are we doing?
• Providing common tools and frameworks
for NGOs to navigate the VfM agenda
• Bringing NGOs together to share emerging
experience
• Ensuring constructive dialogue with DfID
25
bond.org.uk
NGO Evidence Principles (pilot to May 2013)
1. Voice and Inclusion: We listen to beneficiaries’ perspectives on the
effects of the intervention, and can identify who has been affected and
how.
2. Appropriateness: We use methods for data collection and analysis
that are justifiable given the nature of the intervention and purpose of the
assessment
3. Triangulation: We make conclusions about the intervention’s effects by
using a mix of methods, data sources, and perspectives
4. Contribution: We show how change happened, explore a range of
explanations for why it happened, test our theory of how we thought it
would happen, and explain how we contributed
5. Transparency: We are open about the data sources and methods
used, the results achieved, and the strengths and limitations of the
evidence
26
bond.org.uk
NGO Evidence Principles – the checklist
A NGO checklist for assessing the quality of evidence (DRAFT)
Piece of evidence being assessed: Name of assessor:
Principle Criteria 1 2 3 4 NA Comments/evidence
1)VoiceandInclusion
Welistento
beneficiaries’
perspectivesonthe
effectsofthe
intervention,andcan
identifywhohasbeen
affectedandhow
1a.Aretheperspectivesofbeneficiariesincludedinthe
evidence?
1b.Aretheperspectivesofthemostexcludedand
marginalisedgroupsincludedintheevidence?
1c.Arethefindingsdisaggregatedaccordingtosex,
disabilityandotherrelevantsocialdifferences?
1d.Dobeneficiariesplayanactiveroleindesigningthe
assessmentprocessand/oranalysingdata?
Scoreforvoiceandinclusion(averageofcriteriascores):
2)Appropriateness
Weusemethodsthat
arejustifiablegiven
thenatureofthe
interventionand
purposeofthe
assessment
2a.Arethemethodsusedtocollectdataappropriategiven
thenatureoftheinterventionandthepurposeofthe
assessment?
2b.Isthesizeandcompositionofthegroupfromwhich
dataiscollectedappropriategiventhenatureofthe
interventionandpurposeoftheassessment?
2c.Isthecompositionoftheteamcollectingandanalysing
thedataappropriategiventhenatureoftheintervention
andpurposeoftheassessment?
2d.Arethemethodsusedtoanalysethedataappropriate
giventhenatureoftheinterventionandpurposeofthe
assessment?
Scoreforvoiceandinclusion(averageofcriteriascores):
bond.org.uk
NGO Evidence Principles – the results
• A quality assurance
framework
• A kite mark / seal of
approval
• A tool for identifying skill
gaps and building capacity
28
Voice&Inclusion
13/16Appropriateness
7/16
Triangula on
12/16
Contribu on
4/16
Transparency
9/16