measuring outcomes - sopact documents... · tips for collecting beneficiary-level data 14 •select...
TRANSCRIPT
MEASURING OUTCOMES
1
JANUARY 2020
HOW DO YOU THINK ABOUT IMPACT?
LOOK BEYOND THE OUTREACH
The “What” … So What?
• Did lives improve?• Do the youth get jobs?• Do these new jobs pay more?• Has health improved?• How are lives transformed?
Our mission is to create dignified jobs for the poorest of the
poor by building scalable businesses with investment and consulting support. 4
17 BUSINESSES | 14,500+ JOBS
Inputs
What do you exist to do?
What are the inputs you bring to bear?
BE SYSTEMATIC: USE A LOGIC MODEL
6
Activities
What do you doto carry out your mission?
if / then
Outputs
What happens as a directresult of those activities?
if / then
Outcomes
What happens indirectly, as a longer term consequence of those activities?
What is outside your direct sphere of influence?
if / then
Impact
What outcomes are proven to be attributed to your actions?
What is the causal connection?
if / then
Mission
Our mission is to create dignified jobs for the poorest of the poor by building scalable businesses with investment and consulting support
Activities
Screen and select
Patient equity
Accelerator and deeper support
Outputs
Profitable, self-sustaining and scalable enterprises
New job creation
Outcomes
Stabilized and increased income for jobholders
Jobholder progress out of poverty including:• Improved housing
quality• Improved food
security• Increase in the
number of household assets
UPAYA’S LOGIC MODEL
7
Mission
Our mission is to create dignified jobs for the poorest of the poor by building scalable businesses with investment and consulting support
Activities
Screen and select
Patient equity
Accelerator and deeper support
Outputs
Profitable, self-sustaining and scalable enterprises
New job creation
Outcomes
Stabilized and increased income for jobholders
Jobholder progress out of poverty
UPAYA’S LOGIC MODEL
8
• # of channels• # of applicants• # of businesses accelerated• Amount of capital committed for
investment• # of investments
• Annual revenue growth• % that are profitable• Amount of follow-on capital attracted• # of jobs created• Average % income increase• Change in poverty scores (PPI™)
• % moving to more secure homes• Average % increase in food
expenditure• # and type of new assets procured
after the job
Resulting Metrics
MEASURING THE QUALITY OF A JOB
• Track at household level• Gather all previous income sources • Gather all current income sources• Standardize for same time period (daily)• Do not overlook discontinued income sourcesà Calculate % change in incomeà Calculate what % is from partner business
INCREASE IN INCOME
Rs 200 / week Rs 10,000 / season Rs 1,600 / month
Previous
Rs 1,000 / week
Current
IS INCOME MORE STABLE?
OVERALL JOB SATISFACTION?
ASSETS PURCHASED AFTER HIRE
LONG TERM ASPIRATIONS
10
MEASURING THE QUALITY OF A JOB
11
80% Ave Income Increase($4.15 to $7.50)
MEASURING THE QUALITY OF A JOB
12
This Photo by Unknown Author is licensed under CC BY-NC-SA
MEASURING THE QUALITY OF A JOB
Challenges in Impact Management
13
• Selecting the most appropriate metricsConsider industry standards (IRIS+, Poverty Probability Index, etc.)
• Lack of resources to support IM activitiesFunders must make IM a priorityOrganizations should include costs in their program budgets
• Systems designed for mission-driven organizationsData storageAlignment with industry standardsAnalysis and visualizationReport generation
Tips for Collecting Beneficiary-level Data
14
• Select most critical metricsA number between 15-20 is ideal What indicators can your organization directly / indirectly effect? Can you act on the feedback?
• Design surveyWord questions to prevent biased answer(Ex: Are you satisfied? VS Would you recommend this to your friends?)(Ex: Has this improved your life? VS What are your goals with this new income?)Consider multiple choice (vs free form) à Ex: Asset buckets (comfort, household, productive)Administer within 10 minutes
• Work with enumeratorsSelect third-party, objective groupTraining on questions, flow, and testing very importantProvide interpretation guide to accompany survey à what is the goal for every question?
Designing Metrics: Considerations
15
• What is objective / observable?Ex: Assets / housing condition as indicators of poverty
• Do not discount respondent’s own perceptionsEx: Is your income more stable than before?
• Methods to assess “impact”Track same respondents over timeCompare cohorts of similar respondents (ex: compare by 1-year versus 3-year tenure)Focus group discussions before survey design à create metrics around themes that emerge
16
POVERTY PROBABILITY INDEX (“PPI™”)
0-432% < $1.90/day96% < $3.80/day
45-492% < $1.90/day38% < $3.80/day
80-840% < $1.90/day3% < $3.80/day
95-1000% < $1.90/day0% < $3.80/day
Please visit povertyindex.org
for more information