Mark PlanigaleResearch & Consultancy

results by design

Homelessness assistance sector, Australia, November 2010

2

Performance measurement

Compliance reporting

Program logic

3

Performance measurement Bigger picture of organisational performance

measurement

Compliance reporting currently focuses on effort

Many organisations monitor inputs and client satisfaction

But are we making a difference?

National frameworks: National Affordable Housing Agreement, National Partnership Agreement on Homelessness

‘Manufacturing-based’ performance models need to be enhanced

4

Program logic

InputsPeople and resources

• Clients

• Human resources

• Other resources

ActivitiesWhat gets done

• Functions

• Processes

• Actions

OutputsWhat clients receive

• Client participation benefits

• Client direct benefits

OutcomesChanges in clients’ lives

• Short term

• Long term

• Interim

ImpactsChanges in society

• Social outcomes

Environment • Problem or need• Barriers• Enablers

Objectives Aims

Mission Vision

5

Useful information

is produced through

well-planned systems.

6

Purpose and scope

Defined outcomes

Defined measures and tools

Data collection and storage processes

Analysis and reporting processes

Use of outcomes information

Strategies for sustaining the system

7

Clarify purpose and scopeWhy measure outcomes? Questions to consider

Compliance

Individual assessment and planning

Oversight

Quality improvement

Advocacy

Which programs?

Which populations?

What do we mean by outcomes?

8

Define key outcome areasOutcome: Questions to consider

“... a change or absence of change that results (at least in part) from actions of staff of the organisation”

Desired vs. undesired

Short term, long term, interim

Domains vs. locus of change

If the program works really well, how are clients’ lives better?

If things go wrong for our clients, where do we see this in their lives?

What is our core business?

9

Example outcomes1. Person obtains appropriate, affordable long-term

housing

2. Person takes medication more regularly

3. Person has a better understanding of tenancy rights and responsibilities

4. Person is placed on the OOH Early Housing waiting list

5. Person gains part-time employment

6. Person becomes homeless

10

Select measures and toolsMeasure: Questions to consider

“...an observable, measurable characteristic of a person or their situation, which is linked with a state or condition of interest to us”

Base measures, derived measures and KPIs

Varied vantage points

Use modules for flexibility and consistency

If our key outcomes occurred, how would we know?

Which measures have the greatest communicative power?

What data do we already have?

11

Example measuresStatus change/maintenance scale:

1. % of clients who were homeless at the end of the period

Level of functioning scale:

2. % of clients who rate their parenting skills better at exit than they did at entry

Goal attainment scaling:

3. % of clients who achieved better than expected outcomes in majority of goals reviewed this period

12

Example tool: Outcomes Star

Triangle Consulting / London Housing Foundation. http://www.homelessoutcomes.org.uk/The_Outcomes_Star.aspx

13

Collect dataData collection and entry: Questions to consider:

Has a huge impact on data quality

Should be integrated as far as possible with service delivery

Requires client consent

May need to respond flexibly to circumstances of client

Who will collect the data? What training will they require?

All clients or a sub-sample?

How and when will data be collected?

How will data be stored?

14

Analyse and reportOutcomes reporting enablers: Questions to consider:

Who will use the reports?

What levels of aggregation are useful?

How can complexity be acknowledged?

A knowledge of stakeholder requirements

Data analysis skills

Standardised:

calculation procedures

report templates

database queries

Access to data!

15

Example: control graphProportion of service episodes with housing situation improved at completion

38.4%

34.5%

31.8%

35.2%34.3%

33.3%34.4%

37.4%

28.5%

28.6%

34.9%

41.3%

20%

30%

40%

Jun

2009

Sep

2009

Dec

2009

Mar

2010

Jun

2010

Sep

2010

Dec

2010

Mar

2011

Jun

2011

Percentage of

completed service

episodes

(current quarter

n = 165,

cumulative

n = 1097)

Quarterly proportion Cumulative proportion

16

Use outcomes informationContexts for use: Questions to consider:

Individual casework: assessment, planning and review

Service review and reflection

Strategic planning: “turning the curve”

Advocacy

How can outcomes information be shared with clients?

How can staff play a part in “giving meaning” to outcomes information?

17

Caution! Outcomes monitoring data by itself cannot “prove”

service effectiveness

To build your case, triangulate with other evidence

18

Sustain the systemWhat helps?

Using the information – complete the loop

Strong commitment to and endorsement of outcomes measurement by senior management

Performance expectations – staff to collect outcomes data

Ongoing resourcing

Regular review of framework

19

Risks

Implementation process

The bottom line

20

Risks Poor value for money

Outcome measurement can be resource-intensive and time-consuming

Information produced may not be high quality

Distortion of service delivery

Staff opposition

Data collection burden

Feeling scrutinised

Negative results

21

Process tips Involve stakeholders early and ongoingly

Board, Executive, management, service delivery staff, clients

Combine top-down and bottom-up elements

Pilot locally, implement sequentially

Typically 6 – 24 months to end of pilots

Varies with size of organisation, complexity of services, and level of resourcing

22

The bottom line Client wellbeing comes first

Client rights are respected

Information produced is useful

Relevant

Reliable

Valid

System is affordable

Data collection burden on staff is minimised

23

Useful resourcesResults-Based Accountability

www.raguide.org

Friedman (2009) Trying Hard Isn’t Good Enough

UK Approaches (incl. Outcomes Star) www.homelessoutcomes.org.uk

Burns & Cupitt (2003) Managing Outcomes: A Guide for Homelessness Organisations

US: National Alliance to End Homelessness Spellman & Abbenante (2008) What Gets Measured, Gets Done: A Toolkit on

Performance Measurement for Ending Homelessness

Reference – technical and implementation issues Planigale (2010) Literature Review: Measurement of Client Outcomes in Homelessness

Services. http://www.homeground.org.au/assets/literature-review-measurement-of-client-outcomes-in-homelessness-services.pdf

Mark PlanigaleResearch & Consultancyresults by design

Phone: 0429 136 596

Mail: PO Box 754, Macleod VIC 3085

Email: results@planigale.com

Web: www.planigale.com