Meta-Evaluation of the Impacts and Legacy of the London 2012 Olympic Games and Paralympic Games

Jonathan France and Anna Woodham

June 2nd 2011

0

50

100

150

200

250

300

350

400

450

19

95

-01

19

96

-01

19

97

-01

19

98

-01

19

99

-01

20

00

-01

20

01

-01

20

02

-01

20

03

-01

20

04

-01

20

05

-01

20

06

-01

20

07

-01

20

08

-01

20

09

-01

20

10

-01

Av

era

ge

ho

us

e p

ric

es

(£

00

0's

)

England

Five Host Boroughs

Greenwich

Hackney

London

Newham

Tower Hamlets

Waltham Forest

2

Contents

• Overview of the meta-evaluation– Objectives– Phases– Methods

• Planning and prioritisation

• The evaluation approach

• Key areas for further research

3

Study objectives

Comprehensive and systematic meta-evaluation of the Impacts and Legacy

of the London 2012 Olympic Games and Paralympic Games

• Considering Sport, Economic, Social and East London, plus Disability and Sustainability

• Outputs, intermediate results, and outcomes, and the additional impact and VFM of legacy investments

• Tangible and intangible/intended and unintended effects

• Lessons learnt around planning and delivery of a lasting legacy

• Influencing role (interim evaluation)

• Inform and advance the evidence base on hosting mega-events

• To develop and enhance methods of meta-evaluation (ESRC supported)

4

Overview of the 3 phases

Phase 1: May 2010 – April 2011

Project Initiation

• Project Initiation Document

• Project Plan

• Project set-up tasks

Report 1 (Scope)

• Identification of project scope

• Research questions

• Logic chains

• Data strategy

Report 2 (Methodology)

• Meta-evaluation methodology

• Lessons learnt

Phase 2 : Feb 2011 – March 2012

Report 3 (Baseline)

• Baseline

• Counterfactual

Report 4 (Interim Evaluation)

• Evaluation synthesis

• Primary research

• Initial modelling

• Impacts and lessons learnt to date

Phase 3: April 2012 – March 2013

Report 6 (Post Games Evaluation)

• Baseline & counterfactual update

• Evaluation synthesis

• Primary research

• Economic modelling

• VFM

• Lessons learnt

• Longer-term impacts

• Meta-evaluation methods

Phase 4 planned for (March 2013-2020)

5

Contents

• Overview of meta-evaluation

• Setting up and planning the meta-evaluation– Project scope– Logic chains– Research questions

• The evaluation approach

• Key areas for further research

6

Defining scope

• Legacy initiatives:– New public programmes and investments developed as a result of

Games– Existing programmes and investments refocused, expanded, or

accelerated– Private and third sector investments inspired by the Games

• Distributional impacts:– Geography (East London, Nations and Regions)– Groups (disabled, BME, young people, gender)

• Temporal impacts:– Pre, during and post Games

Thoughts refined through desk research, DCMS workshop, stakeholder consultation and logic chains.

7

Logic chains

Rationale and Objectives

London 2012 involves a major regeneration programme providing jobs, homes, infrastructure, training and opportunities for UK businesses. This helps overcome a coordination failure whereby firms and individuals are unwilling to invest in the area even though if they all did together they would benefit.

The Games will also:- act as a showcase for British expertise and capabilities, overcoming similar

problems with coordinated promotion to encourage inward investment, tourism and improve export potential;

- provide a focal point for national identity, representing Britain and British culture to the world and creating civic and national pride that inspires new attitudes and behaviours;

- inspire greater participation in sport, providing positive externalities through improved health.

Outcomes/impacts

- Economic benefits –increased GVA, job creation and jobs safeguarded

- Better health outcomes

- Better quality of life/ improved wellbeing

- More cohesive and inclusive communities

- More sustainable communities.

Outputs- Engagement with individuals (sports,

culture and employment support)- Improvements to sports

infrastructure - Engagement with businesses- Creation of opportunities

(volunteering, disabled people etc)- Marketing and awareness campaigns - Building of venues according to new

sustainable construction standards.

Results - New participants in sport and cultural activity - People into jobs and gaining qualifications - More new businesses and more contracts traded with

existing businesses - New inward investment to East London and UK- Increased visitor numbers to London and UK- Greater sense of community cohesion and inclusion - More sustainable behaviour of individuals and

businesses - Improved accessibility for disabled people in accessing

sport, culture employment and transport.

Gross to Net Conversion

- Need to take account of additionality - extent to which opportunities would not have been provided or not be available from other sources in the counterfactual scenario.

- Need some consideration of extent to which programme of intervention is responsible for observed outcomes.

- Adjustments for displacement, leakage, substitution, crowding out and multiplier effects, as relevant.

Activities- Sport - Harnessing the UK's passion for

sport- Economy - Exploiting opportunities for

economic growth- Social - Promoting social engagement

and participation- East London - Driving the regeneration

of East London.

Research questions Designed to tease out what the impacts have been in legacy areas

• participation in sport and physical activity the

development of competitive and elite sport

• economic impacts, particularly in terms of employment and

GVA

• social impacts, particularly in terms of

volunteering and development of the 'big

society'

• contribution to the regeneration of

East London

• CROSS CUTTING• Changes in attitudes to disability; increased participation of

disabled people in sport, economy, volunteering and culture• Contribution to sustainable development

• Effects on well-being• Effects on the international profile and reputation of the

UK, London and East London• Benefits to target groups/communities

• value for money• Whether impacts have been sustained

• Lessons learned about how to maximise the benefits to the host country and city from the staging of mega-

events, particularly in terms of organisational lessons and change

9

This is a meta-evaluation

• Requires robust programme evaluations, that meet needs of Meta-Evaluation, so…

1. Identify gaps or lack of granularity

2. Influence evaluations

• Evaluation Steering Group

• Work with Legacy Boards

• Internal and external seminars

3. Ongoing assessment of relevance and robustness of project-level evaluations

4. Last resort develop own plans for primary research

10

Contents

• Overview of meta-evaluation

• Setting up and planning the meta-evaluation

• The evaluation approach– Framework– Value for money– Challenges

• Key areas for further research

11

Framework for assessing impact

• Builds upon existing evaluation frameworks:– Treasury Green Book input – output – outcome model– Regional Development Agency Impact Assessment Framework– Magenta Guidance– PWC 2012 Games Evaluation Framework

• Each logic model grounded in theory of change and common outcome measures, which can then be tested out through evaluation evidence

• Issues of deadweight must be tested and will apply to all themes; displacement and substitution more to economic themes

• Relevant impact assessment frameworks then devised for each theme, but top down and bottom up analysis required in all cases

12

Sport: Logic chain

Rationale and Objectives

The 2012 Games provides an unparalleled opportunity to catalyse investment in new and improved sports infrastructure, as well as inspiring people to do more and achieve more in sport. Social and economic benefits can be derived from health benefits. Elite achievement in sport can deliver important benefits in terms of boosting national pride, providing sport role models, encouraging community and elite participation in sport, and improving the UK's reputation and influence abroad.

The GOE’s Sports Legacy Strategy Paper, set out five promises:

- Places: Transforming the places where people play sport, making the benefits of London 2012 visible in cities, towns and villages across the country

- People: Inspiring people to make sport happen at the local level- Play: create the sporting opportunities and challenges that give everyone the

chance to become part of the mass participation legacy.

Outcomes/impacts

- Increased participation in in active sport

- Increased life expectancy and better health

- Increased excellence and elite sport achievements

- Increased sporting influence abroad

Outputs

- Youths involved in Olympic and Paralympic style school sport competitions

- Adults and young people involved in community sport and physical activity programmes

- Coaches trained to support sports- Hours of volunteering for sports- Investment in elite sport and NGBs- Young people engaged in international sport- Bids for other sporting events

Results - Increased access to sport opportunities, sport

satisfaction, club membership- New participants in sport and less drop-off- Increased confidence of pupils, higher school

attendance, and academic motivation- Better pathways and support to elite sport- Higher level of sport participation for disabled- Increased access to sport volunteering and

higher interest and commitment in volunteering- Increased influence abroad- Mega sporting event bids won

Gross to Net Conversion

Need to take account of additionality - extent to which opportunities would not have been provided or not be available from other sources in the counterfactual scenario.

Need some consideration of extent to which programme of intervention is responsible for observed outcomes.

ActivitiesLegacy activity is to take place in three areas:

-Places: Upgrading up to a thousand local sports clubs and facilities;· investing in iconic multi-sport facilities; protecting and improving hundreds of playing fields.- People: Recruiting, training and deploying 40,000

sports leaders- Play: Motivate over 100,000 adults to test

themselves in multiple Olympic and Paralympic sports;; investment in sport for disabled people.

- Support for Elite and Internationalsport

13

NATIONAL EVALUATIONS National evaluations with a regional dimension

eg. CompeteFor

PRIMARY AND SECONDARY DATA Secure additional questions in surveys

Commission studies if needed

PROJECT MONITORING Project/programme monitoring information

Input data per project

MACRO ECONOMIC MODELLING Impacts of construction expenditure

Defining counterfactualEstimating secondary (indirect and induced) impacts

QUALITATIVE DATA Case studies

Data sources: collation; QA; synthesis; and

aggregation

14

Examples

• Sport and social themes:

Amendments to national culture and sport surveys in England, facilitating statistical tests of association between sports participation and variables measuring engagement in the Games; medals performance; time-series data/scenarios for volunteering

Bottom-up evidence from programme evaluations; post Games venue planning; selective primary research for Games Makers (survey), disability (media analysis) and sustainability (case studies)

• Economy and East London

• Mix of statistical techniques (differences in differences/control areas; economic modelling); amendments to national tourism surveys; UK trade and investment data

• Bottom-up evidence from programme evaluations; stakeholder interviews (SAV); academic studies; resident and business surveys

15

Valuing benefits

Techniques include:

• Benefits transfer (other studies)

• Econometric modelling

• Revealed preference techniques

• Stated preference techniques

On benefits transfer, some benchmarks available through the DCMS CASE programme:

• Doing sport once a week generated Subjective Well Being equivalent to £11,000 increase in household income;

• Health cost savings of doing sport are £5,000 for a 30-39 year old.

Benefits/Outputs

Baseline

No 2012 Games

2012 Games (Without legacy)

2012 Games (with legacy)

16

Thematic challenges

• Sport– Measuring the sustainability of participation (or use of proxies)– Methodologies to analyse elite sport networks are underdeveloped– Top-down sport for development highly criticized and under-evaluated

• Social– Lack of participant research – access to LOCOG Games Maker database– Wider influences, e.g. Big Society

• Economy– Aggregating the short-term and ongoing impact of the Games on economy– Assessing the longer-term influence of the Games on image/competitiveness;

– need to influence tourism agencies to capture impacts over time– wider impact of the Games on inward investment and export trends

(longitudinal approach preferred)

17

Broader challenges

• Questions have been prioritised, but scope still vast and complex

• Internal organisation and systems

• What is additional?

• Influencing, building support (e.g. LOCOG, academic community), access

• Synthesis or aggregation of outcome data?

• Assessing additionality

• Developing counterfactuals without evaluations

• Developing counterfactuals with evaluations

• Geographical and E&D sensitivities

• Mega-events have not previously been measured in such a comprehensive way: no real precedent

18

Contents

• Overview of meta-evaluation

• Planning the meta-evaluation

• The evaluation approach

• Key areas for further research

19

Areas for further research:

Theme Primary Data Requirements

East London • Household survey and focus groups with residents• Private sector leverage• Stakeholder lessons from park conversion and convergence

Economic • Visitor experience survey• FDI attracted• Sustainability of employment outcomes

Sport • Impact of infrastructure investments• Elite Sport contribution

Community Engagement • Inspire Programme outcomes• Games Maker survey

Disability and Sustainability

• Perceptions of disability• Additional small scale primary research

Nations and Regions • Pre-Games Training Camps impact• Local Authority 2012 activities and outcomes

20

How you can get involved

• Share knowledge on existing and planned 2012 evaluations and research

• Share findings of 2012 project level evaluations and research

• Influence surveys and research work planned

• Views and input on methodology and alternative approaches

• Contact: anna.woodham@culture.gsi.gov.uk

22

Questions…