emerging practices in vr: moving from pilot project to system integration 6 th annual summit on vr...
TRANSCRIPT
Emerging Practices in VR: Moving from Pilot Project
to System Integration
6th Annual Summit on VR Program Evaluation & Quality Assurance
September 16, 2013
Chip Kenney, Deborah Greene
Session Objectives• Share methods used by Southeast TACE
to evaluate integration of emerging practices in VR systems
• Review/Update findings from 2012 evaluation of Southeast TACE pilot projects designed to introduce practices.
• Present protocols and tools used to evaluate impact and measure/track systems integration.
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
2
Overview
• Since late 2009, the Southeast TACE has been partnering with a number of state vocational rehabilitation (VR) agencies to integrate innovative practices within the VR service delivery mainstream.
• Various states are piloting or integrating approaches such as Customized Employment and Asset Development.
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
3
Overview, cont.
• Five of these “Innovations Projects” have progressed to a point where their impact can be assessed to inform future action at local, state and regional levels.
• In 2012 TACE worked with project teams to evaluate individual project experiences as well as the Innovations initiative as a whole, in terms of their impact on the adoption of best practices within VR systems in the Southeast.
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
4
Overview, cont.
• All 5 projects are now taking steps to integrate piloted practices into VR delivery systems
• Three additional Innovations Projects are under way
• Evaluation of project activities continues through networking, tracking tools and Integration Scale measures
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
5
Overview, cont.Current status
•5 Southeast states, 5 state agencies
•8 projects in various stages of design or implementation, focused on…
Aspects of Customized Employment (7) Asset Development (1)
•5 have completed a pilot & are implementing further integration strategies
•2 projects from original count on hold due to agency internal changes
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
6
Overview, cont.Summary of evaluation findings
•VRCs in 4 states have viable option to serve jobseekers with most significant disabilities, where previously they had no effective option
•Results directly attributable to pilot projects
•Value of investment evidenced in… States’ further strategies to fully adopt the practice Refinements/Evolution of the practice itself Continuous improvement of the Innovations initiative
•New design/evaluation tools (Roadmap, Integration Scale, Community of Practice)
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
7
Innovations strategy: Goal
Increase the capacity of state agencies and their community partners to improve employment and self-sufficiency outcomes for individuals with significant and most disabilities
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
8
Innovations strategy: Premise Adoption of innovative practices requires more than training, hence a phased process from pilot to sustainability to integration•Incubator to understand/address feasibility and systems implications before taking it to scale
•Partners can try the practice out, get their feet wet, make it their own, get buy-in
•Admin. and service delivery partners experience and influence the process – credibility, support for the practice and its feasibility
•Phases designed according to local needs, resources, readiness, logistics
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
9
Approach to Innovation Projects
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012
10
Innovations strategy: Process • Meet with agency leadership/staff re. goals, success
measures, strategy
• Agree on multi-phase design
• Assign project manager/team
• Develop project roadmap (workplan, resources)
• Identify pilot sites, partners, resources
• Conduct performance-based training
• Facilitate partnerships and infrastructure supports
• Evaluate pilot phase
• Refine plans for expansion/sustainability/integration
• Launch subsequent phases
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
11
Description of Pilots
Original 5 projects•Florida (2)
Customized Employment - Transition Discovery Certification - Transition and Adult
•Mississippi Customized Employment - Adult
•Georgia Customized Employment - Transition and Adult
•Kentucky Customized Employment - Adult
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
12
Description of Pilots, cont.
3 new projects since 2012
•Kentucky Asset Development
•Georgia Discovery in Residential Setting - Adult
•North Carolina Customized Employment - Transition
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
13
Evaluation, cont.
Design principles•Demonstrate value of Innovations approach to support adoption of emerging practices
•Assess impact to date of individual projects and initiative as a whole
•Directly connect results with the intervention
•Establish process/protocols for ongoing evaluation and continuous improvement
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
14
Evaluation, cont.
Targets•Outputs
Trainings conducted, numbers trained Meetings facilitated Products generated (reports, tools, samples)
•Impact Change in attitude/behavior, i.e., awareness/
understanding/utilization of the practice Systems changes, e.g., partnerships and
infrastructure measures to operationalize the practice
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
15
Evaluation, cont.
Targets, cont.•Return on investment: Start-up and operational costs, value-added
•Customer satisfaction: Agency/Partner assessment of project experience and quality of TACE services
•Jobseeker outcomes Completed profiles Job placements Financial security measures
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
16
Evaluation, cont.• Subjects: Projects that…
Completed the pilot or training phase (at least), and Took or are considering measures to ramp up
• Types of data Quantitative (trainings conducted, # participants,
active practitioners, etc.) Qualitative (partnerships & infrastructure, attitudes &
behavior)
• Overall Innovations initiative as a TACE technical assistance activity
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
17
Evaluation, cont.Data sources•Tracking tools, reports and other documentation (communications, products, etc.)
•Southeast TACE team (staff & consultants involved in development of initiative, design and execution of projects, etc.)
•Agency leadership
•Project leads & participants (incl. jobseekers and their families)
•Communities of practice
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
18
Evaluation, cont.Process•Agree on measures, design evaluation strategy with input from TACE team, agencies, projects
•Develop/Update evaluation Work Plan
•Design data collection and interview protocol (source documents, key informant interviews, focus group)
•Collect/Analyze prelim. Data
•Draft individual project reports and aggregate discussion paper
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
19
Evaluation, cont.Process, cont.•Conduct focus group among project leads
•Synthesize findings – focus group report
•Disseminate white paper summarizing findings and recommendations (TACE website)
•Publish article in Journal of Rehabilitation Administration
•Construct new evaluation tools Project Roadmap Integration Scale
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
20
Evaluation, cont.Process, cont.•Convene communities of practice
Customized Employment Asset Development
•Update evaluation measures and strategy Outputs (projects, trainings, TA activities, certified or
authorized practitioners) Systems integration/impact indicators (policies,
MOUs, curricula, expansion activities) Operations indicators – changes in attitudes, practices Client impact indicators – quantitative, qualitative
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
21
Evaluation, cont.Findings: Infrastructure•CE/Discovery defined/endorsed as discrete services in policy/regulation (MS, GA, FL, KY)
•CE/Discovery incl. in VR strategic plans (FL, GA)
•Fee code changes (MS, GA, FL, KY)
•Long term supports FL: DD agencies GA: MOUs between GVRA and DD/MH KY: HCBS waiver – require referral to VR MS: HCBS waiver – in discussion
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
22
Evaluation, cont.Findings: Capacity-building•Internal Capacity-building
Creation/Modification of positions to support CE Creating mentor/support teams
•Provider/Partner Capacity-building Outreach/Funding strategies to train/develop a pool of
VR-approved CRPs (GA) Communities of practice (FL, MS, GA) Mentoring strategies (MS, FL, KY) University-based training or credentialing program in
progress (GA, FL, KY)TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
23
Evaluation, cont.Findings: Partnerships•School Districts (early collaboration for transition)
•Employment First
•DD and MH agencies
•Universities: capacity-building, evaluation
•Joint outreach & tracking strategies
•Shared recognition of practices as cross-system continuum of services requiring collaboration
•Reduced ‘finger pointing’
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
24
Evaluation, cont.
Findings: Systems Change•Awareness of CE as an option and increasingly requesting CE or Discovery for a given jobseeker
•Understanding of the innovation as a change in VR practice
•Use of CE by VR systems practitioners
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
25
Roadmap• Step-by-step guidance in developing Innovation
projects
• Ensures successful implementation, fidelity Objectives, success measures, evaluation Phases Stakeholders, partners Expectations Advance work Workplan, resources Deliverables
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
26
Roadmap, cont.• Tools to plan project, conduct meetings,
negotiate agreements, e.g. SWOT analysis Resource assessment Readiness/Feasibility assessment Risk assessment Meeting agenda template
• Use for design, development, tracking and evaluation
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
27
Integration Scale
Key indicators
•Agency Buy-in
•Policy and Procedures
•Capacity/Resources
•Data Collection
•Accountability/ Results Reporting
•Evaluation/Impact/
Continuous Improvement TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012
28
Integration Scale: Agency Buy-In
• Agency Leadership
• Agency Champion
• Administrative Level
• Field Service Level
• Partner Level
• Success Measures incorporated in strategic plan, agency identity, accountability
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012
29
Integration Scale: Policy/Procedures
• Policy documents, communications
• Fee structure that adequately compensates providers
• Field Service Manual
• Built into staff job descriptions, performance expectations (management, supervision, field services and support)
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012
30
Integration Scale: Capacity/Resources • Competencies Identified
• Qualified Providers
• Skilled Staff (management, supervision, field services and support)
• Adequate financial resources
• Ongoing training
• Certification Process for Providers
• Mentoring
• Tools
• Technical Assistance
• Partners/ Formal Agreements
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012
31
Integration Scale: Data Collection
• Tracking Employment outcomes Client satisfaction Longitudinal
• Planned Service
• Purchased Service
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012
32
Integration Scale:Accountability/ Results Reporting
• Performance expectations
• Reporting requirements
• Periodic reviews
• Community of Practice with designated agency liaison
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012
33
Integration Scale:Eval./ Continuous Improvement
• Evaluation strategy, workplan
• Periodic review of results
• Results factored into strategic plan, milestones, infrastructure, capacity-building
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012
34
Community of Practice
• Representatives of Innovation Projects meet at least quarterly Customized Employment Asset Development
• Ongoing technical assistance
• Peer networking, exchange
• Ongoing needs assessment
• Ongoing accountability and evaluation
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2012
35
Impact MeasuresExtent to which advances in integration can be attributed to TACE strategy
•Outputs (projects, trainings, TA activities, certified or authorized practitioners)
•Systems integration indicators (policies, MOUs, curricula, expansion activities)
•Operations indicators – changes in attitudes, knowledge; use of practices; referral rates
•Client impact indicators – quantitative, qualitative
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
36
Observations: Highlights• Leadership willingness to take risks, devote
resources to a practice untested in their system
• Systems changes that substantially increased capacity to serve jobseekers with the most significant disabilities
• Outcomes clearly attributable to the intervention
• Costs deemed worth the return on investment
• Pilot results are basis for decisions to expand, replicate, sustain the practice
• New products, new directions (e.g., residential)
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
37
Observations: Challenges
• Buy-in: Reinforcing that this is about serving a population for whom VR traditionally has not had a successful strategy; that building capacity will take time/effort and will be a departure from business as usual
• Consistently engaging leadership & management commitment sufficient to support and sustain a project (even w/ personnel changes)
• Project planning to create environment for successful pilot
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
38
Observations: Challenges
• Articulating success measures directly connected to project activities
• Identifying terms/triggers for moving from one phase of implementation to another
• Prioritizing resources for project management and face-to-face training and technical assistance.
• Communicating project parameters, expectations
• Logistics: Scheduling training, assigning cases
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
39
Recommendations
• Phased approach for practices that are a significant departure from business as usual
• Success measures clearly connected to project activities and decisions to expand/ sustain the practice
• Project workplan covering all phases (pilot to evaluation/replication to full implementation)
• Project management and facilitation resources
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
40
Recommendations• Clear communication of expectations/
implications in advance of committing to pilot
• Supplemental resources to finance project activities and ongoing operations
• Distance learning platforms rather than conference calls
• Leadership program for administrators providing info. on promising practices
• Supports to sustain/integrate practices (Roadmap, Integration Scale, Community of Practice)
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
41
Questions
&
Discussion
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
42
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
43
Southeast TACE Region IVToll-free: (866) 518-7750 [voice/tty]
Fax: (404) 541-9002
Web: TACEsoutheast.org
MyTACE Portal: TACEsoutheast.org/myportal
Email: [email protected]
TACE Center: Region IV, a project of the Burton Blatt Institute.Funded by RSA Grant # H264A080021. © 2013
44
Disclaimer
This presentation was developed by the TACE Center: Region IV ©2013
with funds from the U.S. Department of Education, Rehabilitation Services
Administration (RSA) under the priority of Technical Assistance and Continuing Education
Projects (TACE) – Grant #H264A080021. However, the contents of this presentation do
not necessarily represent the policy of the RSA and you should not assume endorsement by the
Federal Government [34 CFR 75.620 (b)].