applying performance based funding to literacy and essential skills
DESCRIPTION
Applying performance based funding to literacy and essential skills Boris Palameta, Karen Myers, Natalie Conte. January 16, 2013. The context. Transition to a new economy, in which skills are the new currency - PowerPoint PPT PresentationTRANSCRIPT
APPLYING PERFORMANCE BASED FUNDING TO LITERACY AND
ESSENTIAL SKILLS
BORIS PALAMETA, KAREN MYERS, NATALIE CONTE
January 16, 2013
Transition to a new economy, in which skills are the new currency
Many jurisdictions moving towards integrated service delivery models along an employment continuum
Interest in re-aligning incentives to improve outcomes for job-seekers, employers, and tax payers
Can performance-based funding (PBF) drive system-wide change?
What can we learn from the experiences of other jurisdictions?Today’s presentation draws a State of Knowledge Review that was conducted in partnership with Workplace Education Manitoba and funded by HRSDC’s Office of Literacy and Essential Skills
The context
2
Literature review - Review evidence on various PBF models (Canada, US, UK, Australia)
Expert review – Interviews with PBF experts in other jurisdictions
Consultations – Consultations with practitioners and government officials in Manitoba and Nova Scotia
Expert panel – Canadian Economics Association conference, June 2012
Throughout the process our approach was guided by input from the project reference group which was comprised of officials from MB & NS.
Our approach
3
A tool for allocating resources to service providers based on measurable performance targets
Shifts the focus from inputs to outcomes
Assumption is this shift will drive innovation in service delivery and achieve desired long-term outcomes
4
Introduction
What is performance based funding?
1. Design matters - PBF systems are complex and vary widely in design and effectiveness
2. Better design can mitigate risk – PBF risks generating unintended consequences, but ‘second-generation’ designs are more successful in mitigating these risks
3. Promising approaches – Establish meaningful links between practice and performance by paying for client progress along employment and learning pathways
Intermediate outcome milestones (“tipping points”) as performance indicators
State of knowledge
Key findings
5
Examples: “Work first” – Job placement; human capital development; poverty reduction; productivity
SYSTEM GOALSPolicy objectives
Target population
Examples: Employment status; income status; work readiness; human capital, demographic
Outcomes of interest
Immediate Short-term Longer-term
Client outcomes Process
Performance indicators
PROCUREMENT MODEL
Less competitive-non marketLess competitive-non market Open competition-quasi-marketOpen competition-quasi-market
DESIGN OF INCENTIVE SYSTEM
% service-based payments% service-based payments
By outcome By outcome By client characteristics
By client characteristics
Financial Financial Non-Financial (e.g. star ratings)Non-Financial (e.g. star ratings)
% outcome-based payments% outcome-based payments
Benchmark attainment (x$ if x% of clients achieve
outcome A)
Benchmark attainment (x$ if x% of clients achieve
outcome A)
Payment per outcome (y$ per each client achieving
outcome A)
Payment per outcome (y$ per each client achieving
outcome A)
By local economic conditionsBy local economic conditions By client characteristicsBy client characteristics
By speed of placement
By speed of placement
Payment based on absolute performance
Payment based on absolute performance
Payment based on relative performance
Payment based on relative performance
Type of incentive
Scale of risk
Performance targets
Payment weighting
Adjustment for factors outside provider control
Competition for incentive
State of knowledge
1. PBF systems are complex and vary widely
U.S. Job Training Partnership Act (1982-1998)
Australia Job Network / Job Services (1998-current)
Washington St. Student Achievement Initiative (2007 – current)
Context Employment services Employment services Adult education (college system)
Policy objectives ROI in human capital development
Work first; job placement, cost cutting
Educational attainment leading to labour market attachment
Performance indicators
Post-program (employment and earnings at 13 weeks)
Post-program (13 & 26 week job placements)
In-program continuum of learning outcomes, from basic skills to credential attainment
Funding model & scale of risk
6% of state allocations for incentive awards
Competitive bidding – up to 50% for outcome payments
2% funding cut reallocation
Performance payment For meeting federal or state-set targets (% attaining outcome)
Per outcome achieved Per point gained within learning continuum (momentum points model)
Leveling playing field Targets adjusted to local conditions and client characteristics
Client tiering; payment weighted by type of client
Momentum points determined by principle of equivalent effort
Competition between providers
In some states Yes; star ratings Each college assessed against its own historical performance
Three contrasting models
7
Even small amounts of PBF may change behaviour, but not all changes are in the desired direction
Early models particularly fraught with unintended consequences (cream-skimming, parking, gaming)
Second generation models are more promising with built-in features that aim to avoid these pitfalls
Key to mitigating risk is not only careful system design, but also commitment to continuous improvement
State of knowledge
2. Better design and monitoring can mitigate risk
8
Choice of measures crucial in determining incentive architecture ◦ Poorly chosen performance measures may create conflicting
incentives – obtaining performance payments vs. serving clients
Performance measures have often been:◦ 1) Outside provider control; i.e. based entirely on outcomes that
happen after clients leave the program No clear connection between services providers offer and outcomes
they are paid for
◦ 2) Based on attainment of levels rather than gains from a starting point Incentives to pick “winners”
Performance measures
9
Performance measures have often been:◦ 3) Poor proxies for quality
Program outcomes of interest are often long-delayed Performance measures typically use short-term proxies (e.g.
employment at 13 weeks) for outcomes of interest (e.g. longer-term employment)
But chain of evidence is often lacking No clear connection between the short-term outcomes providers are
paid for and longer-term program impacts “Hitting the target, missing the point”
Performance measures (cont’d)
10
1) Use in-program performance measures > In-program measures establish a more immediate and meaningful connection between day-to-day practice and performance
Allow providers to track progress in a timely fashion, understand where and why learners succeed and where they falter, and design interventions to accelerate progress
2) Measure gains not levels > Most measures have focused on levels attained by clients at the time performance is assessed.
Need measures that include starting points and magnitudes of improvement to convey information about provider’s impact on learner achievement.
Guiding principles for designing PBF systems that work
11
3) Measure what counts > Avoid mission narrowing by ensuring that performance measures recognize the full range of program objectives
PBF changes cost/benefit calculus, may encourage development of costly but innovative services; on the other hand, what you do not pay for may be left undone
4) Identify key milestones > Identify intermediate milestones that can be used to track the progress of clients who may enter at different points (e.g. with different levels of skill, employment readiness, etc.)
Select milestones based on points along the pathway where learners stall or struggle – meaningful transitions
Guiding principles for designing PBF systems that work
12
5) Monitor system performance > Build a continuous learning process to respond to unplanned behaviour
E.g. ‘teaching to the test’
6) ‘Right-size’ incentives > Ensure performance incentives are neither too big nor too small
Too big risk management rather than innovation
Too small if costs of meeting performance targets > performance bonuses, incentives will be ignored
Guiding principles for designing PBF systems that work
13
7) Flexible approach to performance targets > Pre-set performance targets are often either too ambitious or not ambitious enough; both can lead to strategic behaviour
A more open-ended approach encourages continuous improvement
E.g. awarding performance dollars according to ‘momentum points’ (i.e. total number of milestones achieved along a learning pathway)
8) Ensure all targeted clients are served > ‘Level the playing field’
Design incentives using the principle of equivalent effort whereby each momentum point should require roughly the same intensity of effort to attain. Recognizes that clients with more barriers may require greater effort to transition between milestones.
Guiding principles for designing PBF systems that work
14
9) Build provider capacity > Providers may lack knowledge or resources to respond effectively to incentives
Limit competition for performance dollars; encourage collaboration to build tools and practices
10) Link in-program measures to post-program impacts > Use longitudinal research to establish a chain of evidence between intermediate milestones (potential ‘tipping points’) and longer-term, post-program impacts (e.g. employment, earnings, etc.)
Follow-up with learners to establish the connection between measured performance and client success in the long-term
Use results to refine and improve performance measurement framework
Guiding principles for designing PBF systems that work
15
Basic skills gains
Pre-college math/english gains
Transition to college level
15 College level credits (0.5yr)
30 College level credits (1 yr)
Washington State Student Achievement Initiative
16
Key transitions milestones within a student’s pathway (identified by research as ‘tipping points’). Provides incentives to focus on full range of skill levels
Rewards achievement of key milestones – Encourages client progress by rewarding achievement of key milestones that, if reached, are associated with further progress and ultimately long-term labour market success
Focuses on balanced set of ‘in-program’ measures – Which helps providers understand where clients succeed and where they falter, and thus provide the data to drive innovation
Driven by a balance of competition and collaboration - Allocates performance dollars according to total number of milestones achieved. Thus while providers have strong incentives to innovate, they are not in competition with each other. Indeed they may be motivated to collaborate to improve outcomes
State of knowledge
3. Key features of promising approaches
17