evaluation of business start-up policies by jonathan potter, senior economist, oecd leed programme
Upload: oecd-local-economic-and-employment-leed-programme-and-its-trento-centre
Post on 25-Dec-2014
1.083 views
DESCRIPTION
Presentation from the capacity building seminar “Financing business start-up by under-represented groups”, 27-29 June 2012, Trento – Italy; organised by the Local Economic and Employment Development (LEED) Programme and its Trento Centre at the OECD in collaboration with the Directorate-General Employment, Social Affairs and Inclusion of the European Commission. See www.trento.oecd.orgTRANSCRIPT
Evaluation of Business Start-up Policies
Jonathan Potter
Senior Economist
OECD Centre for SMEs, Entrepreneurship & Local Development (CFE)
www.oecd.org/cfe
OECD-European Commission
Capacity-building seminar
‘Financing Business Start-up by Disadvantaged and
Under-Represented Groups’, Trento, 27-29 June 2012
Presentation structure
1. Principles of evaluation
2. Evaluation approaches
– logical framework
– indicators
– qualitative and quantitative evaluation
3. Conclusions
1.1 What is evaluation?
Papaconstantinou and Polt (1997):
“Evaluation refers to a process that seeks to determine as systematically and objectively as possible the relevance, efficiency and effectiveness of an activity in terms of its objectives”
Purpose is to provide decision makers with the best information to answer crucial questions . . .
1.2 Evaluation questions
• Is the activity relevant?
• Is it achieving its objectives?
• Is it cost effective and proportional to what it seeks to achieve?
• Should the activity be continued or terminated?
• If continued, how can it be improved?
1.3 Assessment criteria
Criterion Explanation
Appropriateness (relevance) Extent to which the programme objectives are relevant to the needs of the economy, and address a market failure or social inequity
Effectiveness Extent to which the activity is achieving the programme objectives, net of what would have happened in the absence of the initiative
Efficiency Extent to which costs are minimised in achieving the programme’s objectives
Sustainability To what extent can benefits be expected to last after the intervention is completed
Process management Extent to which the programme is well managed
1.4 The counterfactual
Outputs, e.g.
number of
businesses
started by
the target
group
Time
Baseline Project
Do nothing
option
Assistance
option
Additionality
What would happen without the programme
Our interest is in the difference between the assisted output and the non-assisted output
Non-
additionality
1.5 Evaluation in the policy cycle
Definition of rationale for
policy
Appraisal of policy
options
Implementation of programmes
and projects
Monitoring of progress
Evaluation of results
(benefits – costs)
Feedback
2.1 The logical framework – concept
• Analytical planning tool to clarify causal relations in a policy intervention
• Operational chain of activities, immediate results and subsequent impacts
• Can be used at programme level and at project level
• Can be used for strategy design and for monitoring and evaluation of projects
2.2 Logic framework – microfinance and business support
project
Resources /inputs
E.g. public investment in guarantees of
non-performing loans; fund
administration cost; public
cost of business support
E.g. numbers of loans
provided; hours of
mentoring provided; new contacts with
banks and business support agencies
Activities Outputs
E.g. number of new firms created by
unemployed people;
increased skills;
improved business
proposals; new relationships with banks
Outcomes
E.g. reduction in number of participants
who are unemployed;
increase in number of
starts; increase in survival of
starts; increased lending to
target group
Impacts
E.g. reduction in
unemployment in the target
group; increased
entrepreneur-ship rate
2.3 EQUAL Business Creation Theme Logic Model
2.4 Indicators
Type Examples
Inputs Funds allocated Total budget Funds allocated as % of budget
Activities Number of people contacted Number of people having received services Number of proposals for the creation of new businesses Number of advisors % of beneficiaries belonging to target groups
Outputs/outcomes Attitudes to entrepreneurship Business start rate Survival rate of enterprises Reduction in unemployment rate
Other Satisfaction of beneficiaries with the programme Satisfaction of managers with the programme
2.5 Methods for evaluation
Qualitative approach
• Based on stakeholder and beneficiary interviews
• Deep questioning on processes and changes in decisions and behaviour
• Strength is that it explores processes and engages participants in policy learning
Quantitative approach
• Compares performance of assisted and matched firms and individuals
• Accounts for selection bias
• Strength is that is gives robust impact estimates
12
3. Conclusions
• Evaluation is critical to policy success but is not yet properly integrated in policy process
• At this stage of the EU policy process the emphasis is on ex ante evaluation, implying:
– development of logical frameworks
– Identification of indicators to measure, targets to reach and creation of a monitoring programme
• Once activities have been implemented, will need to consider the nature of evaluations to commission; may need some data collection now