pro bono economics
Post on 07-May-2015
777 Views
Preview:
DESCRIPTION
TRANSCRIPT
2© Pro Bono Economics
PBE aims to match professional economists with charities;
providing pro bono help to measure performance and understand charity impact, and
fostering a culture of volunteering within the economics profession
3© Pro Bono Economics
History2009 - incorporated and registered with Charity Commission
11 projects – no management – none completed
2010 – 2 members of staff from SeptemberManaging 24 projects – 1 completed
2012 – + part-time economistManaging ± 40 projects – 41 completed to date
4© Pro Bono Economics
PeopleSupported by over 260 volunteers from the public sector (Government Economic Service, Bank of England and others), private consultancy firms (Charles Rivers Associates, Europe Economics, Frontier Economics, FTI Consulting, and Oxera), academics and individual economistsTrustees Martin Brookes (Chair), Andy Haldane (Treasurer), Kate Barker,Lynne Berry, Lucy Heady,Nicola Pollock, Dave RamsdenPatrons include Gus O’Donnell and Adair Turner
5© Pro Bono Economics
Projects160 charities have registered 168 projects
Where: 127 London & South East, 25 North & East, 5 in Scotland,1 Northern Ireland and 2 WalesWho for: 13 with homeless people,13 with ex-offenders, 9 with disable people, 4 with the elderly, 8 with other charitiesWhat: 12 employment support, 15 using arts or sport, 20 looking for analysis to support campaigning88 proposals have been withdrawn or rejected 79 projects have had 162 economists allocated to them41 reports completed to date16 projects in pipeline, 14 being scoped, 9 being worked on
© Pro Bono Economics
Vfm of interventions UK analysis
Data collection
advice
Other economic
advice
Some charities PBE has helped
The counterfactual
8© Pro Bono Economics
Types of evaluation
Process: how and how well is the project working?
Impact: what change occurred as a result of the project?
Economic: were the benefits worth the costs? requires an impact evaluation
9© Pro Bono Economics
Measuring outcomesMeasure a characteristic before the interventione.g. number of people with a coldDeliver the interventione.g. cold and flu medicineMeasure the same characteristic after the interventionDifference = impact?
Most people get better from a cold with or without medicine
10
© Pro Bono Economics
Measuring outcomesMeasure a characteristic before the intervention
e.g. school attendanceDeliver the interventione.g. mentoring scheme to improve attendanceMeasure the same characteristic after the interventionDifference = impact?What other things might be going on at the same time? New breakfast club New attendance officer
11
© Pro Bono Economics
What would have happened anyway?i.e. without the interventionN.B. the intervention did take place, so we can never know for certainCan we measure changes in outcomes for a credible comparison group, who didn’t get the intervention?
credible = no reason to think that there may be differences (bias) which affect the outcomese.g. employment programme for women cf. national employment outcomes – unbiased?
12
© Pro Bono Economics
Control GroupsNational average
PBE: Foundation Training CompanyBeforePBE: MEAM pilots – adults with multiple needsMatched records in admin datasetPeterborough Social Impact Bond/MoJ datalabNatural experiment in charity dataPBE: BarnardosRCTEducation Endowment Foundation/Social Research Unit
13
© Pro Bono Economics
Estimating impact
Measure outcomesBest estimate of what would have happened anyway (the counterfactual)Outcomes minus counterfactual = impactA measurement minus an estimate = an estimate
14
© Pro Bono Economics
Discuss
What are you using/might you use as a control group?
Data: methods & issues
16
© Pro Bono Economics
Data 1: project admin
Can provide data on activities and outputs
(monitoring)
Rarely useful for outcomes (evaluation)
Tends to be less reliable over time at an individual
level
Often essential for identifiers to link to large data sets need to ensure this is gathered from the start
17
© Pro Bono Economics
Data 2: surveyProduce your own survey Pros:
can tailor to your outcomesCons: time consuming (design, trialing, collection)usually need specialist inputbiases data collection from control groups
Sometimes possible to link to other large surveys
18
© Pro Bono Economics
Data 3: large data setsEducational attainment
National Pupil Database (NPD) Re-offending PNC, prison, probation, MoJ Data labPros: good measures of specific outcomes can ‘match’ / control – ‘synthetic’ counterfactualCons: access (security)use….needs v high level of expertise bias (unrecorded variable bias)
19
© Pro Bono Economics
Data 4: RCTs
Randomised control trials
Randomly allocating participants between ‘treatment / project’ and ‘control’ groups & compare outcomes.
Produces the best counterfactual with least bias → best estimation of impact
20
© Pro Bono Economics
Issues 1: biasResponse bias
especially to surveysUnrecorded variable biasself selection attitude personality
RCTs are the best way to overcome bias
21
© Pro Bono Economics
Issues 2: sample size
Only relevant once bias is eliminated or known
Needs to be large enough to be representative, but too large is wasteful
What is large enough?Ask a statistician
22
© Pro Bono Economics
Issues 3: timing
Outcomes happen as a result of activity Therefore can only be measured after the activity not during it!
Can be difficult for funders who want reports at a project’s close
23
© Pro Bono Economics
DiscussSample evaluations:
What level of evidence?Strengths and weaknesses?
24
© Pro Bono Economics
Levels of Evidence
Valuation
26
© Pro Bono Economics
DefinitionsSocial value
monetary worth of something to society orSocial value act“how what is proposed to be procured might improve the economic, social and environmental well-being of the relevant area”Social cost benefit analysisCBA including wider benefitsSocial return on investmentparticular methodology for social CBA emphasising role and input of stakeholders
27
© Pro Bono Economics
Whose perspective?Value to society
won’t include transfer payments e.g. tax and benefitsValue to individualschange in income including benefits (+)Value to ExchequerIncrease in tax receipts (+) or increase in benefit payments (-)Potential costs avoidedCashable savings
28
© Pro Bono Economics
Which benefits?Hard outcomes
Qualifications → employment → lifetime earningsCosts avoided by public servicesSoft outcomesEarly stages of work in this areaRelated to income, so only for those who are earningBut still important so
No numbers without stories, no stories without numbers
29
© Pro Bono Economics
Which costs and values?
Input costsProgramme costsVolunteer inputOther activities
Benefit valuesAverage costsMarginal cost
30
© Pro Bono Economics
DiscussWhat type of valuation might be most relevant to you?
top related