evaluation of sme and entreprenuership programme - jonathan potter & stuart thompson
Upload: oecd-local-economic-and-employment-leed-programme-and-its-trento-centre
Post on 08-Feb-2017
647 views
TRANSCRIPT
![Page 1: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/1.jpg)
EVALUATION OF SME AND ENTREPRENEURSHIP PROGRAMMESJonathan Potter and Stuart Thompson
Trento13th November 2015
![Page 2: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/2.jpg)
Structure of the presentation
1. Role and importance of evaluation2. Approaches to evaluation – measures and
techniques3. Evaluation examples4. Planning and organising evaluation5. Some conclusions
![Page 3: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/3.jpg)
Part 1
Role and importance of evaluation
![Page 4: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/4.jpg)
OECD Istanbul Ministerial Declaration
Recognised“The need to develop a strong evaluation culture in ministries and agencies responsible for SME policies and programmes” “Evaluation provides a means of ensuring that SME programmes remain cost-effective and adapt to changing conditions in a dynamic world”
![Page 5: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/5.jpg)
What do we mean by evaluation?
Papaconstantinou and Polt (1997): “Evaluation refers to a process that seeks to determine as systematically and objectively as possible the relevance, efficiency and effectiveness of an activity in terms of its objectives” Purpose is to provide decision makers with the best information to answer crucial questions, e.g.• Does my programme address the key
issues? • What is working and not working? • What can I do better next time?
![Page 6: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/6.jpg)
Why evaluate?
• Learning from experience• To establish the impact of policies and
programmes against their objectives• To make informed decisions about the
allocation of funds across a portfolio• To show the taxpayer whether the
programme is a cost-effective use of public funds
• To stimulate informed debate• To achieve continued improvements in the
design and administration of programmes
![Page 7: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/7.jpg)
Some evaluation questions
• Is the activity still relevant? • Is it achieving its objectives? • Is it cost effective and proportional to
what it seeks to achieve? • Should the activity be continued or
terminated? • If continued, how can it be improved?• Can it be used in different contexts?• What issues will occur when scaling up a
pilot project?
![Page 8: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/8.jpg)
Part 2
Approaches to evaluation – measures and techniques
![Page 9: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/9.jpg)
Inputs
Resources required for the programme (finance, people etc)
Often defined in advance, needs to be sufficient
↓
Activities
Actual actions taken (training courses, advice services etc)
Determined from technical considerations and experience
↓
Outputs
Immediate measure of activities (number of trainees, people advised)
Determined from previous experience and evaluations
↓
Outcomes
Change in behaviour of target (number of start-ups etc)
Requires detailed analysis and research
↓
Impacts
Overall effect (noting what would have happened anyway etc)
Often only calculable well after the programme is complete
Measures at different levels
![Page 10: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/10.jpg)
Measure Definition Typical QuestionsRelevance The extent to which the activity is
suited to the priorities and policies of the target group, recipient and government.
To what extent are the objectives of the programme still valid?Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives?
Effectiveness The extent to which the intervention's objectives were achieved, or are expected to be achieved, taking into account their relative importance.
To what extent were the objectives achieved / are likely to be achieved?What were the major factors influencing the achievement or non-achievement of the objectives?
Efficiency The outputs in relation to the inputs. It signifies that the intervention uses the least costly resources possible in order to achieve the desired results.
Were activities cost-efficient?Were objectives achieved on time?Was the programme or project implemented in the most efficient way compared to alternatives?
Impact
The positive and negative changes produced by an intervention, directly or indirectly, intended or unintended on the local social, economic, environmental and other development indicators.
What has happened as a result of the programme or project?What real difference has the activity made to the beneficiaries?How many people have been affected?
Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.
To what extent did the benefits of a programme or project continue after funding ceased?What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?
Typical evaluation measures
![Page 11: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/11.jpg)
Monitoring Step I Take up of program Step II Recipients’ opinions Step III Recipients’ views of the differences
made by the program Evaluation Step IV Comparison of the performance of
‘assisted’ with ‘typical’ ventures Step V Comparison with ‘matched’ ventures Step VI Taking account of selection bias
Approach - six steps
![Page 12: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/12.jpg)
The ideal approach – Randomised control group
![Page 13: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/13.jpg)
Example - GATE
• Open to all, but targeted at the unemployed
• Applicants randomly given assistance or not
• No barriers to using other sources of advice
• Result showed robust evidence of an impact
Impaq International (2009), Growing America Through Entrepreneurship: Final Evaluation of Project GATE
![Page 14: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/14.jpg)
Regression model
N
NN N
N
N N
N
NN
N
N
NN
N N
N
NN
N
B
B
BB B
BBB
B
BB
B
BB
B
B
B
B
Out
put v
aria
ble
(res
ult o
f a v
arie
ty o
f fa
ctor
s af
fect
ing
firm
cha
ract
eris
tics)
Firm characteristics (control variables and value of policy support)
Beneficiaries
Non-beneficiaries
![Page 15: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/15.jpg)
Matching (propensity scoring)
![Page 16: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/16.jpg)
Differences in differences
![Page 17: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/17.jpg)
Regression discontinuity
![Page 18: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/18.jpg)
Part 3
Evaluation examples
![Page 19: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/19.jpg)
Example – Prince’s Trust Youth Business Initiative
Greene, Francis J (2005), Evaluating Youth Entrepreneurship: the Case of the Prince’s Trust
• The Prince’s Trust is one of the longest running programmes to support youth entrepreneurship in the United Kingdom. It uses a mix of soft (mentoring) and hard support (loans and grants) and has maintained the same approach over a long period.
• The programme’s specific aims are:• Supporting individuals who would be unlikely
to start up an enterprise• Deepening the human capital of participants• Improving survival rates
![Page 20: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/20.jpg)
Prince’s Trust - methodology
• Survey of recipients• Survey of non-recipients• Survey of non-survivors• Matched survey with sample
selection• 3600 respondents• Survival rates assessed over 3 years
![Page 21: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/21.jpg)
Prince’s Trust - results
• Only mentoring appears to have a positive effect
• This is at variance with simpler evaluations which only looked at the views of participants, who were very positive
• No evidence that entrepreneurship support assisted employability
![Page 22: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/22.jpg)
Example – Start-up grants to unemployed immigrants in Germany
• Germany has had long term policies of providing support and “welfare bridges” to the unemployed i.e. continued subsidies even when they are self-employed. Exact levels of benefits have varied over time.
• Immigrants have different characteristics from the general population and may respond to policies differently.
• There have been a number of positive evaluations of the scheme as a whole, but these do not mean that it works in the same way for all groups of the unemployed, for example immigrants.Robinson, Eric (2011), Start-Up Grants to Unemployed Immigrants in
Germany: A Means to Reach Employment Parity?
![Page 23: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/23.jpg)
Methodology
• Use of long term socio-economic database, the German Socio Economic Panel – including more than 140,000 individuals tracked between 1983 and 1993.
• Regression techniques, controlling for sex, age, marital status, education, work experience, sector, type of self-employment
![Page 24: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/24.jpg)
Results
• Unlike natives, foreigners are more likely to lapse back into unemployment when start-up benefits are high. This difference is statistically significant.
• Consideration of skill levels suggests that this is because high benefits give an incentive for ill-prepared individuals to start up business.
![Page 25: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/25.jpg)
Part 4
Planning and managing evaluations
![Page 26: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/26.jpg)
Evaluation and the policy cycle
ProgrammingWhat is the objective?
IdentificationWhat are the barriers?
FormulationHow can we address it?
ResourcingCan we mobilise resources?
ImplementationIs it on track?
Ex Post EvaluationDid it produce the results?
Ex Ante EvaluationOngoing/Interim
Evaluation
Meta evaluationsBenchmarking
Terminal Evaluation
![Page 27: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/27.jpg)
• A timetable for evaluations• Responsibility for commissioning or
performing evaluations• Quality assurance• Mobilisation of resources
Evaluation Framework - management
![Page 28: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/28.jpg)
• Identifying stakeholder needs• Deciding what to evaluate• Key Performance Indicators• Timely and accurate information• Qualitative information• Information regarding assumptions• Baselines• Counterfactual
Evaluation framework - information
![Page 29: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/29.jpg)
Getting the information?Cost
Complexity
Administrativereports
Managementreports
Availablestatistics
Adapted availablestatistics
Interviews withbeneficiaries
Specialisedsurveys
![Page 30: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/30.jpg)
Dealing with the results
• Is there logic in the report?• Is it peer-reviewed? Quality assured?• Have things changed?• What needs to be public?• What is politically necessary?
![Page 31: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/31.jpg)
Part 5
Conclusions
![Page 32: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/32.jpg)
Conclusions
• It is important to evaluate consistently and robustly across programmes
• Effective and efficient evaluation requires planning from the beginning of implementation
• Evaluation often involves deciding on proportionality – the amount of effort involved in gathering data against the usefulness of the result
• Managing authorities need to develop capacity in commissioning evaluations rather than in actually doing them
• Full impact evaluations can be difficult
![Page 33: Evaluation of SME and entreprenuership programme - Jonathan Potter & Stuart Thompson](https://reader035.vdocuments.net/reader035/viewer/2022070509/589aceb81a28abc93a8b4b43/html5/thumbnails/33.jpg)
Questions