how the british council evaluates impact
DESCRIPTION
How the British Council evaluates impactCorporate Planning and Performance shannon.west@britishcouncil.orgwww.britishcouncil.org1HighlightsWhole organisation performance measurement system developed over the past decade adapting commercial and non-commercial models. Designed to facilitate a value for money and return on investment approach. Integrated into organisational systems and processes. Quantitative and qualitative methods of monitoring and evaluating. Monitors and evaluates: acrTRANSCRIPT
www.britishcouncil.org 1
How the British Council evaluates impactCorporate Planning and Performance
www.britishcouncil.org 2
HighlightsWhole organisation performance measurement system developed over the past decade adapting commercial and non-commercial models.
Designed to facilitate a value for money and return on investment approach.
Integrated into organisational systems and processes.
Quantitative and qualitative methods of monitoring and evaluating.
Monitors and evaluates: across the whole logic model, at all organisational levels and, allowing matrix cuts of data.
Dashboarding and detailed reports.
A global support team and formal and informal staff capacity building.
Continuous adaptation and innovation = evolution.
www.britishcouncil.org 3
Corporate level indicators
www.britishcouncil.org 4
Project logic model
www.britishcouncil.org 5
Values driven approachProfessionalism
• externally-referenced and benchmarked
Creativity• continual improvement and gradual change, experimental collection and presentation data
Mutuality• designed and results shared with staff and stakeholders
Integrity• evidence-based
Valuing people• developmental staff capacity building, inclusive of variety of stakeholder views and opinions, staff and stakeholder involvement in all development and change
www.britishcouncil.org 6
Data collectionStandardised corporate tools covering the whole logic model:
• Cultural relations mapping – establishes country context and local stakeholder needs and interests• Financial information (including donated services and off-books income) • Taxonomy of audience and output/activity types • Immediate response and learning questionnaires that allow for localisation• Annual long term impact measurement linked to standard outcomes: Personal and learning development, Institutional benefits, Closer ties to UK
Supplemented by milestone and end of programme evaluations (often 3rd party) that validate results and add specific detail relevant to the context of the programme or location of delivery.Reported at different frequencies depending on data source.
www.britishcouncil.org 7
Data collection example - participationStandardised audience taxonomy used to monitor global participation in programmes and services.
www.britishcouncil.org 8
Data collection example - outcomesEvaluation of long-term outcomes (ELTO) – standardised annual outcome measurement tool used for 8 years.
Collects quantitative and qualitative outcome data from participants 12-24 months after initial participation.
Measures extent of achievement against three performance indicators:
• Personal benefits• Institutional benefits• Closer ties/links to the UK.
Local and corporate staff socialisation around results to reflect on performance.
www.britishcouncil.org 9
Data collection example – people and partnersIncludes measurement of internal and external climate for the people and partners involved in our work.
Annual Staff Survey indicators of staff engagement
Annual Diversity Assessment Framework (DAF) – process measuring the extent to which our equal opportunity and diversity strategy and policy is integrated into our internal and external work.
Suite of partnership measurement approaches (in development 2011-12).
www.britishcouncil.org 10
Operationalising£700k per annum (0.1% of turnover) for dedicated support staff and headquarters team.
Organisation-wide online system (SAP) for collating and reporting all quantitative data:
• customized input and reporting screens• input at local level by project managers• reported at all levels from local project level to corporate• management dashboarding through add-ons.
Standardised data tagging allowing for multiple cuts and combinations of data (e.g. financial and activity data to support efficiency measurement). Tags include: country, region, program, business unit, funding type, product types and accreditation bodies, plan, forecast and year to date results.
www.britishcouncil.org 11
IntegrationCorporate – reported annually as part of annual report and integrated into annual corporate plan.
Business unit – standardised activity and outcomes aligned to the logic model.
Programme – activity and outcome indicators part of design and approval process and linked to annual and quarterly planning and reporting.
Individual – forms part of individual operational staff duties for data collection, collation and reporting.
www.britishcouncil.org 12
Staff capacity buildingGuidance notes and documentation –documents, diagrams, video demonstrations.
Dedicated local support staff who are members of global evaluation team.
Online induction to monitoring processes and rules.
Refresher webinars and online demonstrations.
Professionalisation through UK-accredited evaluation qualification framework (participants become members of community of practice).
www.britishcouncil.org 13
ReportingStandardised monthly and quarterly reporting to Executive Board and Board of Trustees.
Dashboarding through add-ons for standardised management reports (in development).
Wide system access for flexible localised reporting.
Integration of data for efficiency reporting – e.g. financial and activity data, unit cost data.
www.britishcouncil.org 14
Reports integrating qualitative and quantitative
Business unit quarterly report
Programme milestone report