1 ichh learning and evaluation framework interagency council on housing and homelessness framework...
TRANSCRIPT
1 ICHH Learning and Evaluation Framework
Interagency Council on Housing and Homelessness
Framework for Learning & Evaluation
May 20, 2009
2 ICHH Learning and Evaluation Framework
Meeting Agenda
Purpose:
Agree on an evaluation &
learning framework for
ICHH Networks.
Agenda
1. Welcome and Introduction (One Family)
2. Overview
3. Outcome Evaluation
4. Innovation Evaluation
5. Network Evaluation
6. Roles, Resources and Timelines
7. Next Steps
3 ICHH Learning and Evaluation Framework
Steering Committee
• Bob Pulster, ICHH• Sue Beaton, Fireman Foundation• Deborah Fung, Fireman Foundation• Dennis Culhane, University of Pennsylvania• Emily Cohen, One Family• Joe Finn, MHSA• Madeleine Taylor, Innovation Network for
Communities• John Cleveland, Innovation Network for
Communities
4 ICHH Learning and Evaluation Framework
Context Overview
5 ICHH Learning and Evaluation Framework
The Stakes Are High
• The ICHH funding is one-time funding designed to push a paradigm shift in the state approach to homelessness.
• Innovation in the field will drive future state policy around know best practice.
• We need a rigorous evaluation and learning processes to make a convincing case for systems reform.
6 ICHH Learning and Evaluation Framework
The Paradigm Shift
Shelter
Day CareEmployment Assistance
Family Supt Services
MH/SA Services
Prevailing Model Emerging Model
Housing Stabilization
Prevention, diversion & rapid re-housing
Day Care
Employment Assistance
Shelter
Family Supt Services
MH/SA Services
Housing Placement
7 ICHH Learning and Evaluation Framework
Systems Transformation
• From “the Continuum” to “the Network”• Creating a New Community of Practice:
Housing Stabilization• New Service Priorities:
– Diversion– Tenancy Sustainment– Service Coordinators– Benefits Counselors– Housing Relocation Specialists– Family Mediation– Home Visitors
8 ICHH Learning and Evaluation Framework
ICHH Evaluation Mandates in 5 Yr Plan
• Improve statewide data gathering systems.
• Create an information management system to monitor performance and outcomes.
• Assess progress on key benchmarks and use information to make system adjustments.
9 ICHH Learning and Evaluation Framework
Evaluation Overview
10 ICHH Learning and Evaluation Framework
Three Areas of Evaluation
Purpose: To understand what works and doesn’t work to reduce individual and family homelessness, and align state
policy with this knowledge.
Areas of Learning
Outcomes
What were the targeted outcomes and to what extent
were they achieved?
Innovations
What innovations were used, and how did they affect the
outcomes?
Networks
What is the structure of the networks, and
what effect did it have on program
outcomes?
11 ICHH Learning and Evaluation Framework
Three Areas of LearningArea What We Want to Learn
Outcomes • What were the targeted outcomes (impact on homelessness)
• To what degree were these outcomes achieved?
Innovations • What practice innovations did the networks implement?
• How were these practices different from traditional practice?
• What impact did the practice innovations have on outcomes?
Networks • What is the structure of the network?• How is the network governed and managed?• How well is the network developing trust and
creating value for members?• What is the relationship between the network
structure and innovations?
12 ICHH Learning and Evaluation Framework
Three Processes
Process What it Involves
Regional Network Processes
• Regions develop systems for collecting common data elements
• Regions collect outcomes data• Regions assess their innovations and
networks
Technical Assistance Support
• MHSA, One Family and Innovation Network for Communities provide TA to regional networks
• Site visits; interviews; phone consultations
Peer Learning • Network representatives meet quarterly to share best practice learning in specific areas
13 ICHH Learning and Evaluation Framework
Measuring Outcome Data
14 ICHH Learning and Evaluation Framework
Collecting Outcome Data
• ICHH’s goal is to implement an improved state-wide HMIS system.
• In implementing HMIS, regions can utilize existing platforms if they meet appropriate data exchange standards.
• However, outcome data planning and collection cannot wait for the completion of the HMIS system.
• All regions will be asked to collect “common data elements” to evaluate outcomes.
15 ICHH Learning and Evaluation Framework
Common Data Elements• HUD has issued new HMIS standards for the
Homelessness Prevention and Rapid Rehousing Program (HPRP)
• DHCD will adopt these new data elements for HPRP providers and regional networks
• New elements include 1) tracking HPRP eligibility, 2) services provided, and 3) outcomes
• Client tracking and recertification for assistance until exit
• Quarterly reports due to HUD and DHCD starting January 2010
16 ICHH Learning and Evaluation Framework
Outcome Evaluation Resources
• MHSA will work with each network on implementation of the “Home and Healthy for Good” evaluation methodology
• Dennis Culhane will serve as an overall consulting resource to ICHH on outcomes data collection and analysis
• One Family will provide TA to networks on evaluation of family outcomes
17 ICHH Learning and Evaluation Framework
Home and Healthy for Good Methodology
• Secure informed consent• Data collection through one-on-one interviews:
– Entry Interview, Follow-Up Interviews, and Exit Interview
• Entry of data by MHSA into secure databank• Analysis of data:
– Costs
– Medicaid
– Quality of life
– Sub-analysis
• Comparison of data across networks
18 ICHH Learning and Evaluation Framework
Assessing Innovations
19 ICHH Learning and Evaluation Framework
Common Types of Innovations
Early Warning Systems
Prevention and Diversion
Assessment and Intake
Case Planning & Management
Access to Housing
Supportive Services
20 ICHH Learning and Evaluation Framework
Common InnovationsType of Innovation Elements
Early Warning Systems
• Training for risk factors across the network• Development of alert systems
Prevention & Diversion
• Tenancy preservation & eviction prevention services• Stabilization teams; Legal services and mediation• Direct diversion to housing, bypassing shelter system
Assessment & Intake • Mobile assessment• Pre-assessment tools; uniform assessment tool• Case triage and targeting tools• Common databases
Case Planning • Comprehensive case planning (Individual Service Plans)• Case management and case coordination
Access to Housing • Rapid re-housing; shallow subsidies• Low threshold housing• Housing location systems; housing searches; housing database
management• Landlord network and relationship management• Housing development
Supportive Services • Flexible funding for short-term and shallow subsidies• Employment and training; Work First; employment counseling• Financial management training
21 ICHH Learning and Evaluation Framework
Things To Learn About Innovations• What specific practices did the innovation consist of?
• How did the practice vary across regions? Why?
• How did the practice vary across populations? Why?
• How did the innovation affect outcomes?
• Were networks used in the innovation? If so, what kind?
• How critical was the network design to the innovation success?
• What are the “best practices” in each category?
• What are the economics of each innovation? How much do they cost and what do they save?
• What are the policy implications?
22 ICHH Learning and Evaluation Framework
Innovation Evaluation Process
• One Family and MHSA site visits to document actual innovation practices
• Debrief with network staff on details of implementation: What practices were actually implemented? What made a difference in outcomes?
• Development of best practice examples for peer learning sessions
23 ICHH Learning and Evaluation Framework
Network Evaluation
24 ICHH Learning and Evaluation Framework
Network Evaluation
Area of Focus Points of Inquiry
Network Structure and Composition
How is the network structured and how are members connected to each other?
Network Governance and Management
How does the network make decisions and manage its work?
Network Production
What does the network actually do? What services does it deliver?
Network Health What is the level of trust between network members? How highly do members value the network?
25 ICHH Learning and Evaluation Framework
Innovations and Networks
Type How A Network Model Might Change Practice
Early Warning Systems
• Common training across a broad range of network members
• Common risk factors and alert systems
Prevention • Collaboration on prevention across the network
• Joint interventions
Assessment & Intake
• Common intake information and triage processes
• Common data systems
Case Planning
• Network-wide case managers• Common ISP formats
Diversion • Sharing of diversion resources• Collaborative case management on diversion process
Access to Housing
• Network collaboration on housing searches and location systems
• Common landlord relationship management and standards
Supportive Services
• Involvement of service providers in the network• Preferred access for network clients
26 ICHH Learning and Evaluation Framework
Different Types of Networks
CONNECTIVITYInformation
ALIGNMENTIdentity
PRODUCTIONInitiative
Connects people to allow easy flow of and access to information and relationships
Aligns people to develop and spread an identity and collective value proposition
Individuals come to share a set of ideas, language, or standards
Fosters joint action for specialized outcomes by aligned people
•Mobilization•Advocacy•Learning and dissemination•Delivering products and services
Increased levels of complexity
27 ICHH Learning and Evaluation Framework
Different Networks for Different Purposes
Example: Early Warning System
•Connecting – Shared alert system
•Aligning – Common standards on risk factors
•Producing – Joint interventions
28 ICHH Learning and Evaluation Framework
Network Evaluation Process• Develop network baseline map
– Material reviews
– On-line survey
– On-site interviews or focus groups
– Develop electronic map
• Develop baseline assessment– Develop network scorecard
• Collect follow up data in 9 months and at completion
29 ICHH Learning and Evaluation Framework
Network Evaluation Products
• Graphic network maps that show baseline and evolving structure
• Network scorecards
• Case studies
• Network evaluation for each region
• Summary network evaluation
30 ICHH Learning and Evaluation Framework
Peer Learning Sessions
31 ICHH Learning and Evaluation Framework
Peer Learning Sessions
• Quarterly meetings (Up to 6 over the 18 month time frame)
• 1-4 people per region (up to 50 people total)• Half day; facilitated by the Innovation Network for
Communities• Agenda will include:
– Specific case studies on the focus topic
– Presentation of outside points of view
– Round-robin discussion on best practices
– Identification of challenges and barriers
– Policy options
32 ICHH Learning and Evaluation Framework
Roles and Resources
33 ICHH Learning and Evaluation Framework
RolesWho What
Regions • Develop evaluation plans• Collect and analyze outcome data• Participate in site visits; share information• Participate in peer learning
ICHH • Establish common data elements• Aggregated data and prepare ICHH reports
Dennis Culhane • Assist in design and analysis of common data
MHSA • Support regions in HHFG evaluation• Provide TA on individual innovations
One Family • Support regions in family outcome evaluation• Provide TA on family innovations• Convene peer learning sessions
Innovation Network for Communities
• Collect data for network evaluations• Provide TA to regions on network design• Facilitate peer learning sessions
34 ICHH Learning and Evaluation Framework
Evaluation Plan Elements
1. Who is responsible for overall evaluation strategy.
2. Strategy for collecting and analyzing outcomes data and connecting with HMIS system.
3. Strategy for documenting and assessing specific innovation practices.
4. Strategy for network evaluation.
5. Evaluation timeline and budget.
35 ICHH Learning and Evaluation Framework
Resources
• Evaluation and learning resources being provided by the Fireman Foundation
• $25,000 grants to each network to support data collection; participation in site visits; and participation in peer learning
• One Family support provided free of charge
• Separate TA and evaluation contracts
• Some ICHH resources to support peer learning process
36 ICHH Learning and Evaluation Framework
Timeline Elements
What When
Regions Submit Evaluation Plans July 1, 2009
Baseline Site Visits July, 2009
Follow Up Site Visits Q4, 2009
Peer Learning Events Quarterly, beginning in July, 2009
Outcome Data Collection Monthly, beginning
Interim Report to ICHH ?
Final Report to ICHH ?
37 ICHH Learning and Evaluation Framework
Summary of Key Points• Every region will need an evaluation plan. The plan will have three
components:– Outcome evaluation and common data elements
– Evaluation of innovations
– Network evaluation
• In addition to the development of the plan, participating in the evaluation process will involve:– Participating in interviews with evaluators
– Responding to information requests (such as on-line surveys)
– Analyzing root causes; participating in peer learning processes
• ICHH has organized resources to support this process.– $25,000 grant to each network
– Technical assistance resources
38 ICHH Learning and Evaluation Framework
Additional Questions and Discussion