aea presentation: using data to influence programs and policy
TRANSCRIPT
Solutions for Health, Housing and Land ● www.cloudburstgroup.com
Using Data to Influence Programs and PolicyLindsey Barranco, Ph.D. Jamie Taylor, Ph.D.
American Evaluation AssociationChicago, ILNovember 14, 2015
2
Session Objectives1. To identify innovative ways existing data is used to
influence program and policy directions2. To examine multiple ways data results can be used for
mid-course corrections in programs and long-term policy impacts
3. To understand the use of data visualization tools to promote data utilization, stakeholder discussion and new policy directions
3
Using Administrative Data to Drive Systems Change and Improve Programs
4
Background• Homeless Management Information System (HMIS) is a
locally administered, electronic data collection system that stores longitudinal person-level information about persons who access the homeless service system.
• HMIS is HUD’s response to a Congressional Directive to capture better data on homelessness
• Can also provide communities with a comprehensive view of the nature and response to homelessness AND foster collaboration
• HUD is now requiring communities use HMIS data to report on system performance toward ending homelessness
55
HUD System Performance Measures
6
Benefits of Analyzing HMIS Data• Provides HUD way to benchmark measure
progress around ending homelessness • Provides communities with an indicator of their
success and challenges• Provides agencies with information about how
their program is contributing to overall system performance
• Allows communities to look at the needs of the homeless population and what is and is not working
• Allows community to monitor performance of agencies
7
2014 Annual Homeless Assessment Report – Published 11/13/15
HMIS data entered at client level, aggregated quarterly for AHAR.
8
2014 Annual Homeless Assessment Report – Published 11/13/15
9
Using the Data to Understand the Local Homeless Service System
10
• National trends in homelessness can be tracked to monitor investments made in targeted efforts, i.e. Ending Veterans homelessness and Ending Chronic Homelessness by 2016
• Great example of efforts to really USE the data collected as part of Federal reporting requirements
• Complicated process at the federal level determining how to define each measure accurately enough to ensure all communities are measuring performance in the same manner
• Communities need support in understanding how to USE their data
• Communities need support in understanding what programmatic or policy changes will “move the needle”
Lessons Learned
11
RRH Analytics ProjectGOAL OF PROJECT: Capacity building to empower community leaders to use data to meet local and federal policy goals to end homelessness. Policy focus on Rapid Re-Housing (RRH) – a housing assistance approach that provides time-limited rent payments to quickly move households out of shelter and back into housing. RRH DATA ANALTICS PROJECT: Homeless Management Information System (HMIS) data analyzed in six sites across the country. Community leaders engaged in learning community: weekly meetings to review data quality, share system learning around program/provider variation & promote inquiry for RRH system level improvement plans.
RRH Data Analytics Project: 2 states, 3 cities, 1 county
12
RRH Data Analytics ProjectFour-month Learning Community Process
HMIS Data Pull
Data Preparatio
n Data
AnalysisIterative
Data Reviews
Data Dashboard
s
12
• Weekly Meetings with Project Leadership– HMIS Data Pull, Data Analysis Design– Data Preparation, Data Reviews, Data Visualization
• Collaborative assessment with community partners ensured analysis congruent with expected program data
• Theory of Change based on Data Literacy Intervention
13
Theory of Change to End Homelessness has Changed
Person falls into homelessness
Person sheltered
Person enters Transitional Housing
Person “ready” to re-enter community
End of Person Homelessness
14
Need for Rapid Re-Housing Evidence Base
15
RAPID CYCLE EVALUATION on RRH IMPACTSPropensity Score Matching (PSM) employed to assess the effect of RRH on reducing the risk of return to homelessness. With Phoenix/Maricopa County HMIS data, comparison groups that statistically looked the same were created to assess the true effects of RRH assistance.
Households were matched on: age, type of household, single parent, education, income, previous shelter stay, race, ethnicity, mental health disability, physical disability, substance use disability explanatory variables.
1616
Evidence supports the claim that RRH reduces the risk of return to homelessness
17
Additional PSM Analysis: Families vs. Singles
Single RRH Households Family RRH Households
*Returns to homelessness were significantly lower for households receiving RRH than for similar households that received usual care. Significance at 5% level of significance
18
Lessons LearnedSystem transformation impacts of RRH data analytics project:• Accelerated state and local shift towards collective
understanding of local RRH impact on ending homelessness• Creative data visualization of analysis motivated broad
stakeholder engagement, system-wide program improvements
• Data analytic results were used by leaders to move support from temporary/transitional housing to RRH approach
• System-Rapid cycle evaluation technique (PSM) engaged social investment, motivated policy review based on evidence of RRH effects on ending homelessness
19
Displaying Surveillance Data to Influence Resource Allocation
20
BackgroundSuicide Deaths in Alaska – One of Highest in the Country
21
Data and MethodsQuantify the Problem, Quantify the Solution
22
Data and MethodsQuantify the Problem, Quantify the Solution
23
Lessons LearnedData reports developed using effective visualization tools promoted positive results of training initiative in decreasing suicide risk
Communities, agencies, organizations and school systems now expanding their use of suicide prevention trainings for all staff
New suicide prevention policies and procedures are being developed locally to increase community awareness and actions on suicide risk, and prevent youth suicide death.
24
Other experiences using existing data to influence policy?
25
Recommendations• To empower communities to use data for program and policy
improvements, rapid evaluation methods and data literacy development is critical
• Knowledge sharing across multi-sector stakeholders requires inquiry, and the capacity to review what is and is not working with data as neutral evidence
• To motivate policy change around complex public health issues, both rigorous research evidence and the effective use of data visualization tools are critical to promote cross-sector understanding and political will
• Data-driven decision making is necessary for continuous, system-wide improvement planning that is geared towards continual mid-course corrections
26
Recommendations• Evaluation does not have to cost a fortune!
• Often there is a wealth of data available to communities for planning at little to no cost
• Look for opportunities – up front and throughout – to apply analysis results in a way that creates meaningful change
• When using administrative data – ensure you have a full understanding of the pros and cons – and what data cleaning may be required
• Include community stakeholders or practitioners in the analysis process
