system level performance measurement: using hmis to drive change darlene mathews, the community...

System Level Performance Measurement: Using HMIS to Drive Change

Darlene Mathews, The Community PartnershipLianna Barbu, Community Shelter Board

September 23, 2008

2008 HMIS Training: Setting the Standard - U.S. Department of Housing and Urban Development2

Learning Objectives

1. Learn the fundamentals of performance measurement.

2. Learn how to develop and use HMIS-based performance measures to support local system planning and policymaking on homelessness.

3. Identify strategies to use HMIS-based performance measures to improve program and system performance.


Why HUD is Interested in Performance Measurement

Accountable to Congress, OMB, the White House, and taxpayers

HUD’s overall objective: to “end chronic homelessness and to move homeless families and individuals to permanent housing”

At least 71.5% of formerly homeless individuals will remain housed in HUD permanent housing projects for at least 6 months

At least 63.5% of homeless persons will move from HUD transitional housing into permanent housing

At least 19% of persons exiting HUD homeless assistance projects will be employed

CoC’s accountable for achieving & documenting results

Performance Measurement


Performance Measurement Fundamentals


What is Performance Measurement?

Performance measurement is a process that systematically evaluates whether your Continuum of Care’s efforts are making an impact on the

clients you are serving.


Multiple Levels of Performance Measurement

Local Service Provider

1. Program Level APR or Other Performance Measurement Tool

CoC

2. CoC/System LevelCoC Application or Other Performance Measurement

Tool

3. State Level State-Specific Performance Measurement Tool

4. National Level GRPA and PART Reviews


Purpose of System Measurement

• Are actions achieving intended goals?– What is the purpose of the system? Goals?– Does the system work?– If yes, what makes it work?– If no, what part doesn’t work, and how do you fix it

to make it work?

• Note that you may have system goals that only relate to certain types of clients or parts of the system (e.g., different goals for severely disabled persons)


Performance Measurement Building Blocks

• Inputs include resources dedicated to, or consumed by, the system—e.g., money, programs, facilities.

• Activities are what the system does with the inputs to fulfill its mission, such as providing shelter, feeding the homeless, or providing job training.

• Outputs are the direct products of system activities. They usually are presented in terms of the volume of work accomplished—e.g., number of participants served, number of service engagements, number of shelter nights.

• Outcomes are benefits or changes among clients during or after participating in system activities. Outcomes may relate to change in client knowledge, attitudes, values, skills, behaviors, conditions, or other attributes.


Example: Single Adult Shelter System’s Impact Over Time

75% of clients will have a plan to obtain and maintain stable housing.

50% of clients who exit will obtain stable housing.

85% of clients who obtain stable housing will not return to shelter within 12 months (43% of all exits).

400 single adults will receive shelter and access to services.

350 clients will receive individualized housing information and referral assistance.

180 clients will receive weekly check-up calls and problem solving assistance, as needed, after exiting to stable housing.

Short-term

Long-term

Outputs Outcomes


Framework for Converting System Goals into Outcomes

Who is the base for measuring

results?

Step 1

What do you hope to achieve

with this population?

Step 2

Within the base, how many persons achieved it?

Step 3

Who is the base population for measuring results?

Within the base, how many persons achieved it?

Step 4

Outcome (%)


Converting System Goals into Outcomes: Example

Program Goal: Exiting single adult shelter clients into “stable” housing

Persons who exited during the past year

(N= 400)

Step 1

Obtain stable housing

Step 2

200 persons obtained stable

housing

Step 3

400

200

Step 4

50% obtainedstable housing


Look Out for Ambiguous Concepts

For example, what do we mean by…

Developing and measuring performance outcomes often invites ambiguous concepts into the process.

• …obtaining stable housing?

• …obtaining employment?• …increasing income?• …accessing services?• …becoming more self-sufficient?

Which data elements and responses will count?


Key Underlying Assumptions

• Staff are collecting the required information at entry and exit for each client.

• Staff are recording this information into the HMIS in a timely fashion (e.g., 4th business day of every month for prior month).

• The target population is clearly defined.

• The program goals are clearly defined for the target population.

Outcomes can be calculated consistently for all clients if, and only if:


Developing Performance Measures: Key Elements

• For each measure (output or outcome):

a) Purpose: identifies the reason for the measure

b) Application: identifies the programs and/or system for which a measure applies

c) Definition: definition for the measure

d) Goal-Setting Methodology: used by program or system to set goals around the measure

e) Reporting Methodology: detailed description of how measure is calculated

• Use HMIS data elements

• Develop and test before implementation


Example: Successful Housing Outcome

a) Purpose: Indicates program’s success in ending homelessness by linking households to appropriate next step housing which includes transitional and permanent housing. A higher number and rate are considered positive.

b) Programs: All

c) Definition: the number of distinct households that exit (i.e., latest exit for households with multiple stays during report period) to successful housing as defined in Table X and the percentage this represents of total distinct households exited during the report period.

d) Goal-setting methodology: Meet or exceed CoC goal or prior performance. Multiply the percentage goal by the projected number of exited households (i.e., latest exit for households with multiple stays during the report period).

e) Reporting methodology: Calculate the total number of distinct household exits and the total number of destinations that are considered successful housing outcomes. Divide the number of successful housing outcomes by the number of total exits during the report period.


Considerations When Developing Measurement Methodology

Specify all parts of query/calculation:• Programs

– Across a program type (e.g., all outreach, all PSH programs)– All programs– Programs that serve a specific subpopulation

• Clients (de-duplicated across programs)– All Exiters– All participants– All currently enrolled– Those who stayed > 6 months

• Data Elements– Same HMIS elements as program measures– Need rules to reconcile data across programs if client was enrolled in more

than one program

• Timeframe– Define date range– Specify date field (entry date, exit date, placement date, assessment date,

etc.)


Wherever Possible Use HMIS to Define Your Concepts

• Emergency shelter • Apartment or house that you own

Based on the Destination HMIS data element, we can define stable housing (narrowly) using the following response categories:

• Transitional housing

• Permanent housing

• Substance abuse facility or detox center

• Hospital (non-psychiatric)

• Jail, prison, or juvenile detention center

• Room, apartment, or house that you rent

• Don’t Know

• Staying/living with family

• Staying/living with friends

• Hotel or motel voucher paid for without ES voucher

• Place not meant for human habitation

• Other

• Foster care home or group home

• Refused


Performance Measurement Methods


Performance Measurement Methods

1. Measure Interim Outcomes by Applying Program Measures System-wide

2. Use Program Performance Results to Change or Manage the System

3. Measure System Results with HMIS-Based Impact Outcomes


1st System Performance Measurement Method

Measure Interim Outcomes byApplying Program Measures System-wide

• What: Examine how the system (across all programs or across part of a system) performs on a specific measure– Example: What percentage of all homeless persons have greater

incomes when they leave the homeless system?– Example: What percentage of all homeless single men leave the

shelter system for transitional or permanent housing?

• Use: Mid-term (e.g., annually)– To assess and communicate the overall system’s success in

helping clients achieve intended outcomes– To indicate interim success towards overall impact goal– To reinforce or adjust current expenditures and support new

funding requests


Establishing System Performance Measures

• Adopt system goals

– Past performance data

– Account for state and federal goals

– Account for local goals related to achievement of long-term impact

• Establish program performance expectations based on system goals

– Negotiate with agencies

– Incorporate into CoC/funding agreements

– Include quarterly, semi-annual, annual goals

– Response to low performance

• Performance reports

– Set frequency

– Determine audience/distribution

– Use data that meets quality standards

– System and program level performance


Case Study: Columbus, Ohio Performance Measures

Prevention Outreach Emergency Shelter

Resource Specialists

Rapid

Re-Housing

Permanent Supportive

Housing

Number Served Successful Housing Outcomes Average Length of Stay Average Length of Participation

Recidivism Successful Income Outcomes

(Tier II shelter)

Direct Client Assistance Utilization

Occupancy (Tier II

shelter)

Housing Stability Housing Retention


Case Study: Columbus, OHPerformance Reporting

• Annual Evaluation– System & program measures compared to period goal – Used to help determination annual funding– Data mostly derived from HMIS – Programs scored as:

• High: no less than one not achieved • Medium: half or more achieved • Low: less than half achieved

• Quarterly System & Program Indicator Reports– System & program performance measures compared to period goal– Used to identify system(s) and program(s) “of concern”, need for

intervention

• Annual Community Report on Homelessness – Annual and trend data – Point-in-time count data

• Reports issued to CSB Board of Trustees, CoC Steering Committee, funders, www.csb.org


Low Barrier and Severe Weather Shelters are not required to set measures for the baseline Performance Indicators listed above because they are not indications on the quality of services offered. However, like all other programs, they are responsible for ensuring that their data is accurate so that we may calculate Occupancy, Clients Served and Chronically Homeless numbers per site.

Self Sufficiency Temporary, Transitional and Permanent Supportive Housing Programs required to submit a Self Sufficiency Indicator will have to choose from Substance Abuse, Education, Mental Illness Assistance or Employment.

Case Study: Washington, DC Performance Measures

Clients Served (all)

Clients Served (chronic)

Facility OccupancyRate

Clients Engaged in Case Mgmt

Exits(Positive & Negative)

Clients w/ Income Upon Exit

Length of Stay

Self Sufficiency

Low Barrier & Severe Weather

Temporary Shelters Transitional Housing Permanent Supportive Housing


• Outputs/Efficiency Measures– Clients Served– Chronically Homeless Served– Occupancy: the rate at which program was used

• Interim Outcome Measures– Permanent Housing

• Positive client destinations at exit (TH programs)• Retain clients for 6+ months (PSH programs)

– Income: the amount of income or sources obtained– Self-Sufficiency: change in substance use, education,

mental illness or employment

• Measures apply differently to each program type and are supplemented with qualitative data for ranking purposes

Case Study: Washington, DC Performance Measures (con’t)


2nd System Performance Measurement Method

Use Program Performance Results to Change or Manage the System

• What: Compare performance results on selected program measures from one program to another. (Risk adjust for client differences.)– Our system is not as effective as we want in increasing client

incomes… which programs are working best or least well?

• Use: Mid-term (e.g., annual)– To help understand whether individual programs are working– To help decide how to allocate dollars or rank programs for

funding– To identify best practices or agencies that need TA – To determine how performance expectations should differ for

different populations, and how to refer clients most appropriately


Caveats of System Performance Measurement & Analysis

• May require a lag in time even beyond the client’s completion of program involvement– Measuring recidivism requires a certain period of

time (12 months?) after program exit in which return to the system might occur

• If the HMIS only includes data from a subset of programs, the system analysis will provide an incomplete picture– Perceived decline in recidivism may only reflect

absence from participating providers as opposed to not being homeless anymore


Apples to Apples: Risk Adjustment

• Problem: Comparing program results can encourage programs to “cream” to ensure strong results

• Solution: Risk adjustment allows differences to be accounted for in client populations when comparing results across programs

Results can be adjusted on the basis of…• Client characteristics, such as demographics, family size, disability• Client history, such as past eviction, criminal background • Client functionality or level of engagement/commitment to change

• To adjust for client differences, programs need to collect consistent data on clients to use during analysis of program results

– Agree on these standards beforehand

• Develop an analysis plan for how to adjust for client differences– Engage a researcher to help develop the plan

• Even if results not formally adjusted, acknowledge that different programs may have different outcome expectations based on differences in clients targeted and/or served


3rd System Performance Measurement Method

Measure System Results with HMIS-based Impact Outcomes

• What: Define a process to track the impact of the system’s efforts on the need/problem.

– Is the system reducing the time that people stay homeless, and therefore reducing the point-in-time prevalence?

• Use: Long-term (e.g., three years)– To determine if you need to continue actions, expand current efforts (add

resources to the current system), tweak current efforts, or fundamentally change the structure and approach of your homeless system

– To evaluate whether past actions (expanding the system or adding money) helped address the issue

– To communicate the overall system’s success (such as, when justifying funding requests)

• Theory: In the long-term, important to demonstrate that the collective actions of the system solve the problem.


Sample Impact Measures

• Incidence of homelessness - Is homelessness declining?

• Incidence of street or CH - Is street or chronic homelessness declining?

• Length of stay in system, across all homeless programs - Do people stay homeless for shorter periods of time?

• Rates of Recidivism – Are repeat occurrences of homelessness avoided or declining?

Cross-tabulate results by core characteristics to understand if/how results vary for different

subpopulations


Some notes of caution…

• There is more to performance measurement than conducting the analysis

– Educate, train, obtain buy-in

• Be careful about how data is interpreted and used

1. Jump in, but don’t be careless in how results are used

2. Look at the results within the context of all the outputs, interim measures and impact measures to validate the interpretation that’s being made

3. Vet the results before publicly releasing anything4. Appropriately caveat the limitations of the data and

analysis


Comparative Trends in Bed Availability and Families Served in the Temporary Shelter System

0

200

400

600

800

1000

1200

2002 2003 2004 2005 2006

Year Round TemporaryShelter Slots forFamilies

Families Served inTemporary Shelter

Look before you leap….

It appears that the rate of family homelessness is going down.


Comparative Trends in Bed Availability & Families Served in the Emergency Shelter System

1124

806

651597

442

184 171 171 153 171

6 6

8

12

18

0

200

400

600

800

1000

1200

2002 2003 2004 2005 2006

0

2

4

6

8

10

12

14

16

18

20

Families Served in Emergency Shelter

Year-Round Emergency Shelter Slots for Families

Length of Stay (months)

…Lesson Learned

Reality: Programs couldn’t accept new families as beds were still occupied by existing clients who had dramatically longer LOS.


Value of Performance Measurement

• Communicates more accurate description of population (vs. individual, anecdotal needs)

• Creates higher level of understanding of community problem (builds community support)

• Informs resource allocation (annual funding) • Informs program development/improvement

(CQI) • Informs policy options (more shelter vs.

housing)


Additional Resources

• What Gets Measured Gets Done: A Toolkit on Performance Measurement for Ending Homelessness, National Alliance to End Homelessness, www.endhomelessness.org

• The Community Partnership for the Prevention of Homelessness (Washington, DC) www.community-partnership.org

• Community Shelter Board (Columbus, OH) www.csb.org

http://www.endhomelessness.org/

http://www.community-partnership.org/

http://www.csb.org/

Questions & Discussion

system level performance measurement: using hmis to drive change darlene mathews, the community...

Documents

system performance

department of housing

system work

system goals

hud transitional housing

hud permanent housing

job training

urban developmentexample