collecting and analysing data
DESCRIPTION
Collecting and Analysing Data. Chris Dayson Research Fellow Presentation to: Involve/CRESR Social Impact Masterclass 26th September 2013. 1. Collecting data. What types of data are there?. Primary data. This is the data that you have collected or could collect Examples include: - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/1.jpg)
Collecting and Analysing Data
Chris DaysonResearch Fellow
Presentation to:Involve/CRESR Social Impact Masterclass26th September 2013
![Page 2: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/2.jpg)
1. Collecting data
![Page 3: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/3.jpg)
What types of data are there?
• This is the data that you have collected or could collect• Examples include:
• Monitoring information• User surveys• Outcome stars• Case management information• Project outputs• Qualitative
Primary data
![Page 4: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/4.jpg)
What types of data are there?
• This is the data that others have collected that might help you
• Examples include:• Area level data:
• Census• Benefits claimants• Deprivation• Crime
• User level:• Health• Crime
Secondary data
![Page 5: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/5.jpg)
What types of data should you collect?
• Think about the types of data you already collect• Could you use it more effectively?• Does this tell you what you need to know about your
outcomes and impact?• What gaps are there: does it cover all key stakeholders?
• Think about the data that others might collect about your stakeholders
• Could you use it to demonstrate your impact better?• Could it help explain context• Can you access it?
• Consent and data protection• Negotiation and mutual benefits
Some tips
![Page 6: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/6.jpg)
2. Collecting primary data
![Page 7: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/7.jpg)
Starting out
• Map out what it is you need to know• What outcomes are you trying to measure• What would outcome change look like in practice
• What level/scale will you start with?• The whole organisation• A project• A particular stakeholder group• Think about your priorities
• Decide on the tools and methods you will use• Who will be responsible for data collection?• Surveys: face-to-face; postal; web/email• Outcome Stars: practitioner/user led
Some considerations
![Page 8: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/8.jpg)
Starting out
• How will collate you the data?• Data entry onto a spreadsheet/database• Bespoke platforms• Who will be responsible for regular data entry
• How will you analyse and use the data?• Frequency of reporting• Who will be responsible for analysis and reporting?
• Think about the skills, capacity and resources• Are there any skills gaps?• Need to create time and space to do it well• Can you build it into funding bids etc
Some considerations
![Page 9: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/9.jpg)
Developing outcome indicators
• Don't reinvent the wheel:• use existing measures where possible• sources include: ONS, WikiVois, UK Data archive, similar
organisations• Measure distance travelled:
• baseline; during; post-intervention; to provide evidence of change and how long it lasts
• retrospective measurement possible but less effective• capture evidence re additionality and impact
• Sampling:• decide how many beneficiaries you need to measure/track• need to expect attrition in the sample over time
Some key principles
![Page 10: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/10.jpg)
3. Analysing data
![Page 11: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/11.jpg)
Analysing quantitative outcome data
• Key it simple and relevant• What will you organisation find useful?• What will funders etc expect to see?
• Focus on change• How much change have you observed?
• how many/what proportion have improved?• how much improvement has there been?• have there been different levels of improvement within
different groups?
Key principles
![Page 12: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/12.jpg)
4. A worked example
![Page 13: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/13.jpg)
Background info
• Aim: to improve the employability of people 'disadvantaged in the labour market' through volunteering based interventions
• Stakeholder: individual beneficiaries of the programme• 'Hard' outcomes:
• individuals undertake volunteering• individuals move into employment• employment is sustained
• 'Soft' outcomes:• individuals move closer to the labour market• individuals have improved health and well-being• individuals are more involved in their communities
A volunteering employability programme
![Page 14: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/14.jpg)
Mapping the change
• Overarching outcome: individuals move closer to the labour market
• Specific outcomes:• Greater motivation to find work• More confidence in holding down a job• More skills and experience• Better able to complete job applications• More confident in attending interviews
Soft outcomes
![Page 15: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/15.jpg)
Analysing the data
• Distance travelled data:• 200 participants - all completed a baseline• 100 participants completed a distance travelled
questionnaire after 4 months
Distance travelled findings
![Page 16: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/16.jpg)
25 24 24 26 26
1513 15 12
17
0
5
10
15
20
25
30
35
40
45
Motivation to findwork
Confidence inholding down a
regular job
Skills andexperience to find a
job
Completing jobapplications to agood standard
Confidence inattending job
interviews
Perc
enta
ge o
f res
pond
ents
Increased by one or two points Increased by three points or more
![Page 17: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/17.jpg)
40 38 3731
36
32 33 36
32
33
0
10
20
30
40
50
60
70
Motivation to findwork
Confidence inholding down a
regular job
Skills andexperience to find
a job
Completing jobapplications to agood standard
Confidence inattending job
interviews
Perc
enta
ge o
f res
pond
ents
A lot A little
![Page 18: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/18.jpg)
Analysing the data
• We can now extrapolate for all service users• Of 200 beneficiaries:
• 80 were more motivated to find work• 74 were more confident in holding down a job• 78 had more skills and experience• 76 were better able to complete job applications• 86 were more confident in attending interviews
• High levels of additionality for each outcome• Qualitative interviews corroborate quantitative findings• Next steps:
• Tracking beneficiaries for a longer period• Valuing outcomes?
Interpreting the data to identify change
![Page 19: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/19.jpg)
5. Taking it further: valuing outcomes
![Page 20: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/20.jpg)
Putting a value on the outcome
• Towards social return on investment (SROI)• Aim: assigning a value to something without a market
price• A range of options:
• Cost savings - to the public sector (and others)• Real money - net gains in income• Willingness to pay - how much would they pay for the
outcome• Revealed preference - build up the value from other
market values• Other proxies: Travel cost and household spending
• Don't forget the stakeholder's perspective
Approaches to valuation/monetisation
![Page 21: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/21.jpg)
Valuation in practice
• How would we value?• Greater motivation to find work• More confidence in holding down a job• More skills and experience• Better able to complete job applications• More confident in attending interviews
Returning to the worked example
![Page 22: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/22.jpg)
Valuation in practice
• Some considerations:• Are we looking at more than one outcome...• ...or a subset of outcomes linked to 'being better equipped
to find work'?• Is being better equipped to find work an interim outcome
on the way to actually getting a job?• From a valuation perspective, does getting a job usurp
any outcome on the way to finding work• Would we be 'double counting' if we valued each outcome
for each participant?
The proxification conundrum
![Page 23: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/23.jpg)
Valuation in practice
• One solution:• For each participant that finds work, identify a proxy value
for that work• For participants that do not find work, but are better able
to find work, identify a proxy value for that outcome• This approach ensures outcomes are not double-counted
and that outcomes are not over-valued
Disentangling the proxification conundrum
![Page 24: Collecting and Analysing Data](https://reader035.vdocuments.net/reader035/viewer/2022062305/56816269550346895dd2d716/html5/thumbnails/24.jpg)
Collecting and Analysing Data
Chris DaysonResearch Fellow
Contact details:email: [email protected]: 0114 225 3539web: www.shu.ac.uk/cresr
Any questions?