2.12 creating a yardstick: developing a performance measurement system
DESCRIPTION
TRANSCRIPT
Performance Performance MeasurementMeasurement
Some Handy ThoughtsSome Handy ThoughtsIain De JongIain De Jong
OrgCode Consulting, Inc.OrgCode Consulting, Inc.
AgendaAgenda1.1. Intro/Definition of Performance Measurement Intro/Definition of Performance Measurement
2.2. Ingredients for Successful Performance Ingredients for Successful Performance Measurement Measurement
3.3. When we know we need to focus on performance When we know we need to focus on performance measurementmeasurement
4.4. Approaches to performance measurement; moving Approaches to performance measurement; moving from program to system level from program to system level
5.5. Priority setting, coordinating performance Priority setting, coordinating performance measurement across the system; metrics to look at measurement across the system; metrics to look at
6.6. Performance measurement and HEARTH, PM Tools Performance measurement and HEARTH, PM Tools
7.7. Q & A/Session Close Q & A/Session Close
To Get StartedTo Get StartedIf you want copies of the presentations, If you want copies of the presentations, send an email (contact details for each send an email (contact details for each presenter are at the end of this presenter are at the end of this presentation). presentation).
At some point presentations will be on the At some point presentations will be on the NAEH website – NAEH website – www.naeh.org - as well as - as well as oodles of other cool resources.oodles of other cool resources.
The OrgCode website - The OrgCode website - www.orgcode.com - - will also contain some of the presentations will also contain some of the presentations and other materials you may find useful. and other materials you may find useful.
Intro/Definitions of Intro/Definitions of Performance Performance MeasurementMeasurement
Definitions of Definitions of Performance Performance MeasurementMeasurement
Performance measurement is the Performance measurement is the process by which criteria is process by which criteria is established to determine the established to determine the quality of activities based on quality of activities based on organizational goals using organizational goals using quantitative evidence.quantitative evidence.
A simple, effective system for A simple, effective system for determining whether an determining whether an organization is meeting objectives.organization is meeting objectives.
What What Performance Performance
Measurement is Measurement is NotNotIt is NOT a waste of time.It is NOT a waste of time.
It is NOT your HMIS. That is a data retention, It is NOT your HMIS. That is a data retention, management and report generating system – management and report generating system – not a performance measurement system.not a performance measurement system.
It is NOT something you consider once per It is NOT something you consider once per year when doing a funding application.year when doing a funding application.
It is NOT a collection of anecdotes.It is NOT a collection of anecdotes.
It is NOT something you look at with passing It is NOT something you look at with passing interest and do nothing about.interest and do nothing about.
6 Things That 6 Things That Performance Performance
Measurement Does Measurement Does (When Done Right)(When Done Right)1.1. Ensures that service and system requirements are Ensures that service and system requirements are metmet
2.2. Sets transparent sensible objectivesSets transparent sensible objectives
3.3. Provides internal and external standards for Provides internal and external standards for comparisoncomparison
4.4. Provides a “scoreboard” for people to monitor Provides a “scoreboard” for people to monitor their performance leveltheir performance level
5.5. Highlights problems that deserve priority Highlights problems that deserve priority attentionattention
6.6. Provide feedback to drive the improvement effortsProvide feedback to drive the improvement efforts
4 Purposes to 4 Purposes to Performance Performance
Measurement at the Measurement at the System LevelSystem Level1.1. Identify chronic underperformersIdentify chronic underperformers
2.2. Identify those organizations where, Identify those organizations where, with coaching and extra attention, with coaching and extra attention, performance standards can be metperformance standards can be met
3.3. Ensure prudent financial stewardshipEnsure prudent financial stewardship
4.4. Acknowledge (reward?) strong Acknowledge (reward?) strong performersperformers
From the From the TrenchesTrenches
Between 1 in 5 and 1 in 3 organizations Between 1 in 5 and 1 in 3 organizations report that staff have felt pressure to lie report that staff have felt pressure to lie about their data at least once in the past about their data at least once in the past yearyear
3 out of 5 organizations that use an HMIS 3 out of 5 organizations that use an HMIS also use one or more other approaches to also use one or more other approaches to collecting information on servicecollecting information on service
Almost half of all staff report they need Almost half of all staff report they need help analyzing datahelp analyzing data
From the From the TrenchesTrenches
Between 40-50% of frontline staff are Between 40-50% of frontline staff are uncomfortable with data and uncomfortable with data and performance measurementperformance measurement
3 out of the top 5 ways in which 3 out of the top 5 ways in which organizations use data pertains to organizations use data pertains to funding (applications, soliciting funding (applications, soliciting donations/fundraising, and meetings donations/fundraising, and meetings with funders) and performance with funders) and performance measurement tends to track less than measurement tends to track less than 50%50%
From the From the TrenchesTrenches
4 out of 5 organizations report that data 4 out of 5 organizations report that data is consistent with their organization’s is consistent with their organization’s Mission, but less than 2 out of 5 report Mission, but less than 2 out of 5 report that the data adequately demonstrates that the data adequately demonstrates the efforts that go into the workthe efforts that go into the work
7 out of 10 service providers do not have 7 out of 10 service providers do not have a process for assessing acuity of clients a process for assessing acuity of clients to point them in the direction of the to point them in the direction of the resources best able to end their resources best able to end their homelessnesshomelessness
From the From the TrenchesTrenches
Less than half of service providers have Less than half of service providers have logic models for each homeless or logic models for each homeless or housing programhousing program
Of those that have a logic model, only Of those that have a logic model, only two-thirds regularly update and two-thirds regularly update and measure against the logic modelmeasure against the logic model
Ingredients for Ingredients for Successful Successful
Performance Performance MeasurementMeasurement
Intensive Supports
Mid-level
support Little If
Any Support
Homeless Population – Not Homogeneous
“Funnel” Of Homeless Services
Intake & Assessment – Acuity Determined
Driving Change Through Driving Change Through Performance Performance MeasurementMeasurement
2009 2019
For Successful For Successful Operational Operational PerformancePerformanceThe intake into the system should create service The intake into the system should create service
pathways for clients based upon pathways for clients based upon theirtheir needs. needs.
Prioritize those most in need of service first.Prioritize those most in need of service first.
Support plans should be informed by information Support plans should be informed by information gleaned at intake – seamlessly.gleaned at intake – seamlessly.
Track how the acuity of clients change over time.Track how the acuity of clients change over time.
Look beyond Look beyond outputs outputs to to outcomes.outcomes.
Have a logic model, follow it and update against Have a logic model, follow it and update against it.it.
Getting StartedGetting StartedDefine your strategic objectives.Define your strategic objectives.
Define your strategic metrics/key Define your strategic metrics/key performance indicators.performance indicators.
Set performance targets.Set performance targets.
Establish a baseline.Establish a baseline.
Determine when you need to look at Determine when you need to look at which metrics to determine which metrics to determine performance against targets.performance against targets.
To Be Successful To Be Successful In Getting StartedIn Getting Started
See if people share the same perceptions See if people share the same perceptions and values across services.and values across services.
Look at the system as a whole and the place Look at the system as a whole and the place of every organization within it.of every organization within it.
Review the quality of your existing HMIS Review the quality of your existing HMIS data and other data that may sit with other data and other data that may sit with other sources.sources.
If you need to refresh/update your plan to If you need to refresh/update your plan to end homelessness, this is a good time to do end homelessness, this is a good time to do so.so.
Want the Want the Simplest Simplest
Approach?Approach?Make sure your data and your approach is meaningful, Make sure your data and your approach is meaningful, unambiguous and widely understoodunambiguous and widely understood
Spend time on working with all data collecting and Spend time on working with all data collecting and measuring organizations so that they “own” and manage measuring organizations so that they “own” and manage the processthe process
Up front work on ensuring there is a high level of data Up front work on ensuring there is a high level of data integrityintegrity
Embed data collection as part of “normal” duties instead Embed data collection as part of “normal” duties instead of something people do in addition to “real” workof something people do in addition to “real” work
Take action based upon that which is learned in Take action based upon that which is learned in measurementmeasurement
Link it all to your strategic objectivesLink it all to your strategic objectives
A Good A Good Performance Performance Measurement Measurement
ApproachApproachLooks at the system as a whole as well as individual Looks at the system as a whole as well as individual agents/performersagents/performers
Focuses the attention on what matters most, not everything Focuses the attention on what matters most, not everything that is possible to measurethat is possible to measure
Moves from data collection to the creation of a narrative Moves from data collection to the creation of a narrative based upon that which is collected, when, by whom, for based upon that which is collected, when, by whom, for what purpose and with which degrees of accuracywhat purpose and with which degrees of accuracy
Ensures measurement of the right thingsEnsures measurement of the right things
Embraces positive deviantsEmbraces positive deviants
Creates a common language for an organizationCreates a common language for an organization
Is independently verifiableIs independently verifiable
Performance Performance Measurement Is Only Measurement Is Only As Good As the DataAs Good As the DataConsistent definitionsConsistent definitions
Timely entry by staffTimely entry by staff
Non-duplication of entries (including Non-duplication of entries (including aliases)aliases)
Agreement on data updates by programAgreement on data updates by program
Exits are duly notedExits are duly noted
Information is completeInformation is complete
""You get what you You get what you measure. Measure measure. Measure the wrong thing the wrong thing and you get the and you get the
wrong behaviors.”wrong behaviors.”- John H. Lingle- John H. Lingle
High Performing High Performing OrganizationsOrganizations
Know exactly which individuals/families they have Know exactly which individuals/families they have the expertise to serve and which programs to the expertise to serve and which programs to provide. They are NOT all things to all people.provide. They are NOT all things to all people.
Prioritize which individuals they serve. They are Prioritize which individuals they serve. They are NOT first come, first serve.NOT first come, first serve.
Internally monitor their performance and are Internally monitor their performance and are dedicated to being reflective practitioners.dedicated to being reflective practitioners.
Possess a simple, transparent and operational Possess a simple, transparent and operational logic model for each program that they run.logic model for each program that they run.
Have internal performance standards and Have internal performance standards and operational standards that they measure against.operational standards that they measure against.
Signs of System Signs of System Level Performance Level Performance
Measurement FailureMeasurement FailureFunders receive information and do not Funders receive information and do not de-fund chronic underperformersde-fund chronic underperformers
Failure to learn and improve (excuse Failure to learn and improve (excuse culture instead of a solution focused culture instead of a solution focused culture)culture)
Delay in reporting out resultsDelay in reporting out results
Continuing to function as a collection of Continuing to function as a collection of projects rather than the system as a wholeprojects rather than the system as a whole
Anecdotal tampering of findingsAnecdotal tampering of findings
ProcessProcess
DefineDefine
MeasureMeasure
LearnLearn
ImproveImprove
When We Know We When We Know We Need to Focus on Need to Focus on
Performance Performance MeasurementMeasurement
Approaches to Approaches to Performance Performance MeasurementMeasurement
Moving from the Program to System LevelMoving from the Program to System Level
““A zebra does A zebra does not change not change its spots.”its spots.”
- Al Gore- Al Gore
Advice for Advice for SuccessSuccess
1.1. Begin the design of your performance Begin the design of your performance measurement system with the end in mind. measurement system with the end in mind.
2.2. Have solid strategic objectives and test Have solid strategic objectives and test against the change that you think needs to against the change that you think needs to happen is happening. happen is happening.
3.3. Create a data loving culture. Create a data loving culture.
4.4. Ensure the data collection and performance Ensure the data collection and performance measurement has valuemeasurement has value
5.5. Focus on getting better and better and better. Focus on getting better and better and better.
Advice for Good Advice for Good Strategic Strategic
ObjectivesObjectives1. Is it measurable or verifiable? 1. Is it measurable or verifiable?
2. Is it achievable or feasible? 2. Is it achievable or feasible?
3. Is it flexible or adaptable? 3. Is it flexible or adaptable?
4. Is it consistent with the rest of your strategic 4. Is it consistent with the rest of your strategic plan?plan?
5. Does it stretch your people without breaking 5. Does it stretch your people without breaking them? them?
6. Is it clear, easy to understand, and inviting to 6. Is it clear, easy to understand, and inviting to achieve? achieve?
Performance Performance Measurement & Your Measurement & Your
Funding ProgramsFunding ProgramsWhile meeting reporting expectations While meeting reporting expectations for funding, the key to success in for funding, the key to success in thinking like a system is to find thinking like a system is to find commonality across requirements and commonality across requirements and structuring community activities into structuring community activities into sectors of service rather than funding sectors of service rather than funding programs. programs.
Sectors of ServiceSectors of ServiceApproach 1Approach 1
PreventionPrevention
OutreachOutreach
Emergency ShelterEmergency Shelter
Transitional HousingTransitional Housing
Rapid Re-housingRapid Re-housing
Permanent Permanent Supportive HousingSupportive Housing
Approach 2Approach 2
PreventionPrevention
OutreachOutreach
ShelterShelter
Drop-insDrop-ins
Housing SupportsHousing Supports
Example of Performance Indicators by Example of Performance Indicators by Program TypeProgram Type
Prevention Outreach Emergency Shelter
Transitional Housing
Rapid
Re-Housing
Permanent Supportive
Housing
Number of Unique Individuals Served
Successful Housing Outcomes
Average Length of Stay Recidivism Successful Income Outcomes
Direct Client Assistance Utilization
Occupancy
Example of Example of Sector of ServiceSector of Service
Outreach
Strategic Objective Decrease Street Homelessness by 50% or more by the next Point in Time Count by housing people, helping them access shelter and/or reuniting with family/friends, focusing on those with highest acuity first.
Performance Indicators • # of unique individuals served• successful housing outputs• successful shelter outputs• successful family/friend reunification• recidivism
Performance Targets • each outreach team to work with no more than 25 unique individuals per month• 4 successful housing outputs per outreach team per month (at least 2 higher acuity per month)• 8 successful shelter outputs per outreach team per month (at least 4 higher acuity per month)• 1 successful family/friend reunification per outreach team per month
To Achieve To Achieve Excellence in Excellence in
Sectors of ServiceSectors of ServiceCreate and publish standards or core service Create and publish standards or core service expectations for each sector of service.expectations for each sector of service.
Ensure that all Strategic Objectives are Ensure that all Strategic Objectives are SMART!SMART!
Set targets that are achievable, not just an Set targets that are achievable, not just an aspiration.aspiration.
Define the service pathways and linkages Define the service pathways and linkages between the sectors of service to allow for between the sectors of service to allow for unambiguous service navigation.unambiguous service navigation.
The Data Analysis The Data Analysis PlanPlan
1.1. ProgramsPrograms
2.2. PopulationsPopulations
3.3. Data ElementsData Elements
4.4. Data CalculationsData Calculations
5.5. FrequencyFrequency
6.6. Reporting FormatReporting Format
7.7. Report AudienceReport Audience
Data Analysis Data Analysis Plan: ProgramsPlan: Programs
By individual programBy individual program
Across all programs (duplicate users Across all programs (duplicate users across programs; individual program across programs; individual program users; average number of users per users; average number of users per program)program)
Data Analysis Data Analysis Plan:Plan:
PopulationsPopulationsSpecific groups to potentially investigate Specific groups to potentially investigate (examples):(examples):
By genderBy genderBy age group(s)By age group(s)By military serviceBy military serviceBy presence of a specific diagnosis or life By presence of a specific diagnosis or life issueissueBy level of acuityBy level of acuityBy length of homelessnessBy length of homelessnessBy location within cityBy location within cityBy indoor or outdoorBy indoor or outdoorEtc.Etc.
Data Analysis Data Analysis Plan:Plan:
Data ElementsData ElementsYou can use HMIS measuresYou can use HMIS measures
Data Analysis Data Analysis Plan:Plan:
Data CalculationsData CalculationsNeed to write out how exactly the Need to write out how exactly the conclusions were reached, especially conclusions were reached, especially when moving beyond descriptive when moving beyond descriptive statisticsstatistics
Data Analysis Data Analysis Plan:Plan:
FrequencyFrequencyNeed to pre-determine which data will Need to pre-determine which data will be analyzed to measure performancebe analyzed to measure performance
Need to determine what the data Need to determine what the data ranges are for the analysis:ranges are for the analysis:
Intake dateIntake date
Program enrollment dateProgram enrollment date– Assessment datesAssessment dates– Exit dateExit date
Data Analysis Data Analysis Plan: Reporting Plan: Reporting
FormatFormatProduct ConsiderationsProduct Considerations
• Web-updateWeb-update
• Paper reportPaper report
• Annual ReportAnnual Report
• Report to the BoardReport to the Board
• Report to FunderReport to Funder
Style ConsiderationsStyle Considerations
• TableTable
• GraphsGraphs
• ContextContext
• AnalysisAnalysis
• Outcome ImpactsOutcome Impacts
Data Analysis Data Analysis Plan:Plan:
Report AudienceReport Audience• BoardBoard
• FunderFunder
• General publicGeneral public
• StaffStaff
• Supervisory TeamSupervisory Team
• Senior Management Senior Management TeamTeam
• Executive DirectorExecutive Director
• Partner agenciesPartner agencies
• 10 Year Plan 10 Year Plan CommitteeCommittee
• Elected OfficialsElected Officials
Quality AssuranceQuality Assurance• ""When dealing with numerical data, When dealing with numerical data,
approximately right is better than approximately right is better than precisely wrong."- Carl G. Thorprecisely wrong."- Carl G. Thor
Quarterly Report Quarterly Report ConsiderationsConsiderations
• Target(s) for the quarterTarget(s) for the quarter
• Outputs for the quarterOutputs for the quarter
• Target(s) year to dateTarget(s) year to date
• Outputs year to dateOutputs year to date
• Quarter over quarterQuarter over quarter
• Year over yearYear over year
• Outcomes to dateOutcomes to date
Can
arie
s in
C
an
arie
s in
C
oal M
ines
Coal M
ines
•h
ttp://p
op
watc
h.e
w.c
om
/20
08
/12
/22
/site
-of-th
e-
http
://pop
watc
h.e
w.c
om
/20
08
/12
/22
/site
-of-th
e-
14
//1
4//
Considerations for Considerations for Process Review at the Process Review at the
System LevelSystem Level• Evidence of client prioritizationEvidence of client prioritization
• Evidence of seamless linking of clients Evidence of seamless linking of clients to programs that meet their needsto programs that meet their needs
• Funding expensed by quarterFunding expensed by quarter
• Percentage of sector target achieved by Percentage of sector target achieved by organizationorganization
• Site visitsSite visits
Priority setting, Priority setting, coordinating coordinating performance performance
measurement across measurement across the system; metrics to the system; metrics to
look at look at
Performance Performance Measurement & Measurement &
HEARTH, PM TOOLSHEARTH, PM TOOLS
Q & AQ & A
Iain De JongOrgCode Consulting [email protected]@orgcodewww.orgcode.com
Susan McGeeHomeward [email protected] @susanmcgeewww.homewardtrust.ca
Kim WalkerCenter for Capacity BuildingNational Alliance to End [email protected]
Amanda SternbergHomeless Action Network Detroit313-964-3666 [email protected]