Are we making a difference?

Measuring the outcomes of disability related services and supports

Presenter: Dr Erin Wilson

School of Health and Social Development, Deakin University

Acknowledgements: Dr Nick Hagiliassis and

Dr Robert Campain, Scope

Background- what do I bring?

• Worked with Scope; Perth Home Care Services; WA Mental Health

Commission and others

• Worked with team of researchers to develop a range of data

collection and outcome measurement tools in a range of contexts:

– Individualised funding and planning

– Early Childhood Intervention Services

– Flexible funding in early childhood

– Therapy and psychology services

– Community based mental health services

– Assistive technology

– Range of disability service models

Selection of data collection and analysis tools developed and trialed

Scope Outcomes Framework (used for analysis) • A whole of life outcomes framework across seven life domains

Scope Outcomes and Impacts tool • A one page survey rating scale to measure impact (positive and negative) of service on life

domains of individual or family

Individualised Planning/Review Tool • A tool based on the categories of a person centred plan to rate achievement against goals

Measurement of Services and Supports (MOSS) tool • A pre and post measure of change resulting from individual goal setting and services based on

this goal

Family Stress and Coping tool AND Family Capacity tool • Two survey scales rating a family’s self perceived level of coping and capacity

Early Childhood Individualised Support Plan tool • Attachment to Individualised Support Plan to rate outcomes at end of service period, and

barriers and enablers to these

1 in 4 poll Social Inclusion Survey • A population level survey rating levels of social inclusion related to service exclusion, economic

exclusion and social exclusion

Overview

• Key terms

• WHAT outcomes might we use – selecting and defining outcomes

• HOW to measure outcomes – guiding principles

How do we know if we’re making a difference?

3 key ideas/terms

•Process

•Output

•Outcome

Terminology – key elements What are ‘outcomes’?

In broadest terms, ‘outcome’ means: what happened

BUT different things are focused on and there is confusion over

terms and their relationship:

1. PROCESS: did we receive or deliver a quality service? The ways in

which program services and goods are provided. Practices used.

2. OUTPUT: how much service did we receive or deliver, how many

people received it etc? The goods and services produced (quantity,

efficiency: e.g., number of people served, speed of response to

complaints).

3. OUTCOME: what happened (for the person)? The broader results

achieved through the provision of goods and services (Horsch, 2005)

• Under the proposed NDIS, it is possible that measurement will occur across

outcomes, but also outputs and processes.

– e.g., outputs - number of clients serviced, respite hours, equipment payments,

home and vehicle modification data;

– e.g., process – degree of service coordination, how disputes are managed.

• Measuring process or output is frequently used as a ‘proxy’ measure of

outcome

• This is based on the thinking (and sometimes research) that where a

given process is used (or level of output occurs) this leads to the

desired outcome (eg. Support to exercise leads to better health –

measure whether support to exercise provided)

• Department of Human Services Standards – largely oriented to

measure ‘good practice’ i.e process

• Output or Process measures ≠ Outcomes.

– i.e., They do not reveal what happened as a result of the service or

intervention or activity.

Steps in developing an outcomes measurement approach for human services

1. Decide the desired areas of change – i.e desired outcomes and for whom

2. Define these areas & unpack complex terms – eg. what do we mean by ‘communication competence’, ‘awareness’,

‘community inclusion’, ‘support’, ‘partnership’?

3. Identify the indicators of these changes – how will we know, what will we see if change occurs?

4. Decide how these indicators can be measured

– How to collect data. How to analyse data. Do measures exist? Can existing measures be modified? Do measures need to be developed?

5. Combine and present the resulting information in a clear and informative way. (Adapted from ABS, 2001, ch 1).

1. Decide the desired

areas

2. Define these areas & unpack

complex terms

3. Identify the

indicators

4. Design how to

measure

5. Present information

Steps of outcomes measurement Task

1. Decide the desired areas of change (what are the outcomes?)

Check if specific outcome areas are identified by funder, government policy, or by others such as individuals who set specific goals for services. Consult stakeholders (practitioners and service recipients)

2. Define these areas & unpack complex terms

Check published literature for concept definition and sub domains of concepts. Consult stakeholders

3. Identify the indicators of these changes Check published literature to identify indicators; consult with stakeholders

4. Decide how these indicators can be measured

Check published literature for existing data collection instruments and methods; consult stakeholders; design methods

5. Combine and present the resulting information

Report and publication writing

Step 1: decide the desired areas – YY

Which processes should we measure? Processes / practice approaches we might measure:

– Family centred practice • (Process measurement tool is Measure of Processes of Care (MPOC), King,

Rosenbaum & King, 1995)

– Person centred practice

– Individualised service delivery

– Strengths based practice

– Recovery based practice

– Etc, etc

Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.

An example: Family Centred Practice

1. Decide the desired

areas

2. Define these areas & unpack complex

terms

3. Identify the

indicators

Family Centred Practice

5 Domains: 1. Enabling and

Partnership 2. Providing general

information 3. Providing specific

information 4. Co-ordinated and

comprehensive care 5. Respectful and

supportive care

(King, Rosenbaum and King, 1995)

Enabling and Partnership e.g., Parents felt like partners in their child’s care

Providing general information e.g., Parents received adequate information about their child’s disability

Co-ordinated and comprehensive care e.g., Looking at the needs of the ‘whole’ child (e.g. Mental, emotional and social needs) instead of just physical needs

Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.

An example: Department of Human Services Standards

1. Decide the

desired areas

2. Define these areas & unpack complex

terms

3. Identify the

indicators

Department of Human Services

Standards (VIC)

4 Areas: 1. Empowerment 2. Access and

Engagement 3. Wellbeing 4. Participation

(DHS, 2012)

Empowerment e.g.,

Rights and responsibilities are developed and provided in an accessible format that facilitates understanding by all people

Access and Engagement e.g.,

Service-delivery hours are responsive to the needs of people accessing the service

Wellbeing e.g., People actively participate in an assessment of their strengths, risks, wants and needs

Participation e.g., The service provider supports people to develop and maintain their personal, gender, sexual, cultural, religious and spiritual identity

Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.

An example: Person Centred Practice

1. Decide the desired

areas

2. Define these areas & unpack complex

terms

3. Identify the

indicators

Person Centred Practice

Domains / major areas:

??

??

Step 1: decide the desired areas of change – yy

Which outcomes should we measure?

Outcomes for whom?

• Person with a disability

• Family /carer of person

• Service provider

• Service sector

• Community members/organisations

• Wider society and social structures

Frequently capacity building and social change outcomes

Step 1: decide the desired areas of change – yy

Which outcomes should we measure?

• What outcomes were seen as

important by panel?

• What others would you add?

• What are the most important?

Step 1: decide the desired areas of change -Which outcomes should we measure?

• Outcomes for persons with disability might include:

– Quality of life

– Social inclusion

– Participation

• (community participation, cultural participation, social

participation, political participation, economic participation….)

– Satisfaction (with services, with life…)

– Loneliness

– Autonomy, independence, choice….

– Etc, etc

What are the outcomes sought under the NDIS?

“An important component of the evidence base under the

NDIS will be the systematic collection of data on outcomes

of particular services or interventions for people with

disabilities. Data on outcomes in employment, education,

social participation, and capacity for self-care, and on the

measures that contributed to those outcomes, would help

to build an evidence base for analysing which interventions

or forms of assistance are more effective, and why”. p 576, Productivity Commission Report, Disability Care and Support, 2011

What are the outcomes sought in other important policy documents and Conventions?

Convention on the Rights of Persons with Disabilities (UN, 2006): • Full and effective participation of all people with disabilities in all areas of community

and economic life

• equality, autonomy, independence, opportunity, accessibility, identity, participation

and inclusion.

National Disability Strategy: • Inclusive and accessible communities

– Proportion of people using public transport; proportion participating in common cultural and

recreational activities

• Rights, protection, justice and legislation – Feelings of safety; participation in civic life

• Economic security – Average income; housing stress; participation in labour market

• Personal and community support – Access to and unmet demand for disability services, etc

• Learning and Skills

• Health and Wellbeing

Step 2: Define these areas & unpack complex terms – eg. what do we mean by ‘social inclusion’

Step 3. Identify the indicators of these changes – how will we know, what will we see if change

occurs?

2. Define these areas & unpack complex

terms

3. Identify the

indicators

Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.

An example: Social Inclusion (as used in 1 in 4 poll)

1. Decide the desired

areas of change

2. Define these areas & unpack complex

terms

3. Identify the

indicators

Social inclusion

Social exclusion Service exclusion

Economic exclusion

(Saunders et al 2007a, 2007b & 2008)

Social exclusion e.g., Having social contact with other people

Service exclusion e.g., Accessing medical services

Economic exclusion e.g., Having enough money to get by on

The 1 in 4 Poll - Social Inclusion Survey (example items from on line and paper based survey in Easy English)

4. Design how to

measure

Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.

An example: Quality of life

1. Decide the desired

areas of change

2. Define these areas & unpack complex

terms

3. Identify the

indicators

Quality of Life

1. Emotional wellbeing 2. Interpersonal

relations 3. Material wellbeing 4. Personal

development 5. Physical wellbeing 6. Self determination 7. Social inclusion 8. Rights

(Schalock, 2004)

Emotional wellbeing e.g., Contentment (moods, enjoyment); Self concept (eg self esteem)

Material wellbeing e.g., Financial status (income, benefits); Employment; Housing Physical wellbeing e.g., Health; Activities of daily living; leisure

Steps 1,2, 3: Decide desired areas; Define these areas; Identify indicators.

An example: Outcomes Star (Services Connect)

1. Decide the desired

areas of change

2. Define these areas & unpack complex

terms

3. Identify the

indicators

Outcomes Star

Services Connect

1. Motivation and taking responsibility 2. Self care and living skills 3. Managing money and personal

administration 4. Social networks and relationships 5. Drug and alcohol misuse 6. Physical health 7. Emotional and mental health 8. Meaningful use of time 9. Managing tenancy and

accommodation 10. Offending (Triangle, n.d)

Reflections on WHAT outcomes to measure: Guiding principles

1. Outcomes selected must be meaningful

• Must capture what is important to people most

affected.

– Consider whole of life rather than focus on single aspects

– Consider broad outcomes list to include ALL that is

important to people

– Consider allowing individuals to name own outcomes (eg

goal focused) – able to be configured to the individualised

service and support context.

Example: A Whole Of Life, Whole Of Person Approach

Scope Outcomes Framework (Wilson, 2006)

• Encapsulates nine life areas:

Personal, Social, Economic, Political, Educational, Recreation and Leisure, Cultural, Spiritual, Environment

• Reflects

– multidimensional aspects of a person’s life

– move away from outcomes measurement approaches that have a narrower focus, e.g., those that focus on physical functioning exclusively.

– whole of life/person approach (not just the piece of it the service addresses).

• Drawn from a wide range of literature (and concepts) such as:

– Personal Outcomes Measures (Accreditation Ontario, 2000),

– National Core Indicators (Human Service Institute, 2003),

– Quality of life and subjective well being concepts (e.g., Cummins, 2007; Shalock,

2004; International Wellbeing Group,2005),

– wide range of other literature (happiness, friendship, social capital, etc),

– lived experiences of clients, families and communities involved in Scope services.

Scope Outcomes Framework / Life domains model (Wilson, 2006)

Personal and family wellbeing

That is, in the area of health and function, happiness, wellbeing, safety, sense of independence and choice.

Social life That is, in the area of friendship and relationship, community involvement and sense of belonging

Political life That is, in the area of having a say about things that affect you (eg in a local service or community group, about your area, school, funding etc

Cultural life That is, being involved in cultural activities (eg. arts, music, theatre, dance at any level). This might be through attending activities or playing an active part. Or being part of your own cultural group

Recreational and leisure life

That is, being involved in recreational or leisure activities at any level. This might be through attending activities, playing an active part or doing what you enjoy

Economic life That is in terms of your finances, employment or business.

Educational life That is, any aspect of your family’s education, training, personal or professional development.

Spiritual life That is, any aspect of your family’s religious or spiritual activities

Your environment

That is, your family’s access to and enjoyment of public spaces (eg parks, pools, theatres, shopping centres, public transport etc) or your own private space (eg. your home). Making environments you use more accessible and appropriate (eg playgroups, kindergartens, library, etc).

Outcomes across whole of life focus: Outcomes and Impacts tool based on the life domains model (Wilson, 2006)

Thinking about whole of life or breadth of outcome coverage Scope/Wilson life domains Personal Outcome Measures (Council on Quality Leadership, 2012)

Personal and family wellbeing

People are safe People have the best possible health People exercise rights People are treated fairly People are free from abuse and neglect People experience continuity and security People decide when to share personal information People choose where and with whom they live People choose services People choose personal goals People realize personal goals

Social life People are connected to natural support networks People have intimate relationships People interact with other members of the community People perform different social roles People participate in the life of the community People have friends People are respected

Political life

Cultural life

Recreational and leisure life

Economic life People choose where they work

Educational life

Spiritual life

Your environment People use their environments People live in integrated environments

Reflections on WHAT outcomes to measure: Guiding principles continued

2. Information collected should be able to guide

improvement

– “The point is not to collect information - it is to learn enough

about how things are going so we can keep going or change

direction with confidence. We want monitoring which leads to

action to improved… [practice], and to have a positive impact on

people’s lives” (Ritchie et al, 2001: 177)

– Identification of factors affecting outcomes: enablers, barriers.

2. What worked for you? What helped you achieve your goal? • People, activities, equipment/aids/resources, finances, environment, policies, time. What else? • Which of these was the biggest help? Comments from person with a disability Comments from others (please specify role, eg. paid support worker, family): Practitioner comments:

3. What stopped you from achieving your goal? What made achieving your goal difficult? • People, activities, equipment/aids/resources, finances, environment, policies, time. What else? • Which of these was the biggest barrier? Comments from person with a disability: Comments from others (please specify role, eg. paid support worker, family): Practitioner comments:

Example from MOSS: Enablers and Barriers focus

Step 4: Design HOW these indicators can be measured

Guiding principles for HOW to measure:

1. Self report

2. Achievable

3. Accessible

4. Effective

See also Andresen, 2000

4. Design how to measure

HOW to measure: Guiding principles

1. Self report of person with disability (or those most

affected, e.g. carer)

– Capture the view point of a person with a disability

– Create alternative ways a person can give information directly

about themselves

• E.g. Use of multi media

• Provide a scribe to assist write, type or enter data

– If must use proxy to report, give explicit instruction to report

from the viewpoint of person they represent

– Be clear who is providing data

• Identify different respondent views – distinguish between person

with disability (service recipient), staff, and others

– Have a mechanism for informed consent of the person

Examples of self report

• Ensuring the perspective of the person with a disability is captured – explicit instruction to proxies;

– 1 in 4 poll:

67.4% of respondents completed the survey independently;

15% had carers complete the survey on their behalf.

\

The survey is being completed by:

A person with disability without assistance

A person with a disability with assistance

A carer of person with disability (someone speaking

on behalf of a person with a disability who cannot

express their views themselves)

Where support is provided, the survey should be completed from the person with a disability’s view point

Examples of self report

• Ensuring the perspective

of the person with a

disability is captured;

– MOSS:

offers three rating scales in

different formats to enable

individuals to select which

one is meaningful to them.

Accessible for some people

with intellectual disability

\

2. What worked for you? What helped you achieve your goal? • People, activities, equipment/aids/resources, finances, environment, policies, time. What else? • Which of these was the biggest help? Comments from person with a disability Comments from others (please specify role, eg. paid support worker, family): Practitioner comments:

3. What stopped you from achieving your goal? What made achieving your goal difficult? • People, activities, equipment/aids/resources, finances, environment, policies, time. What else? • Which of these was the biggest barrier? Comments from person with a disability: Comments from others (please specify role, eg. paid support worker, family): Practitioner comments:

Example from MOSS: Different viewpoints

HOW to measure: Guiding principles

2. Achievable

– Brief and easy to use

– Avoids administrative burden

– Considers time, workload, training, data analysis allocations

– Considers BOTH how data will be collected AND how data will analysed

HOW to measure: Guiding principles

3. Accessible

– Instrument can be used (with and without support by others) by

people with a wide range of disabilities including people with

intellectual disabilities;

– Has alternative formats:

• Easy English format and/or visual supports;

• simple visual scaling;

• physical administration demands low;

• allows for simple verbal responses (e.g., ‘yes’ or ‘no’);

– Considerations to item/instrument bias; reliability, validity vs

internal inconsistency; acquiescence;

– Culture/ language adaptations.

Examples of accessibility

• Value in engaging Easy English experts to

translate content

• Checked by people with intellectual disability

E.g. 1 in 4 poll:

High ‘user friendliness’ ratings (mean rating 8.5)

Examples of accessibility MOSS tool

• offer simplified choices and rating scales

HOW to measure: Guiding principles

4. Effective

– Items well match the outcome concepts identified (concept capture

and validity)

– Indices of change/measurement are responsive – sensitive to and

able to capture small changes

– Considers timing of data collection – when will outcomes be

achieved and apparent? (at moment of intervention, long term)

– Has credibility: indicators provide sufficient information about a

condition or result to convince both ‘supporters’ and ‘sceptics’ (Andresen, 2000)

– Allows for aggregating up for generating comparative data, e.g.,

comparison to general population, benchmarking data

• Data can be analysed at the individual, service provider, disability system and

societal level.

Deciding the ‘when’ of outcomes: outcomes at what moment?

– e.g., outcomes for the person with a disability

Example: Deciding the ‘when’

Outcomes for the person with complex communication

needs

– at the moment of intervention (e.g. use of communication

board in a shop)

• can communicate effectively

• able to make an effective transaction

• increased confidence

– beyond the moment of intervention - ie on person’s life as a

whole

• increased ability to shop independently (decrease support costs)

• increased choice and range of places can participate in

Issues we’ve confronted

• Selecting the ‘right’ indices of measurement (e.g.

performance, achievement, satisfaction, impact, difficulty,

etc)

• Aggregating data – moving from a focus on an individual’s

outcomes (highly individualised) to whole of service /

population outcomes

• Data analysis – difficulties in quantifiers, thematising etc –

time consuming and often highly subjective

• Administration compliance – getting staff to administer

data collection tools (correctly)

• And more…..

Data collection and analysis tools developed and trialed

Scope Outcomes Framework (used for analysis) • A whole of life outcomes framework across seven life domains

Scope Outcomes and Impacts tool • A one page survey rating scale to measure impact (positive and negative) of service on life

domains of individual or family

Individualised Planning/Review Tool • A tool based on the categories of a person centred plan to rate achievement against goals

Measurement of Services and Supports (MOSS) tool • A pre and post measure of change resulting from individual goal setting and services based on

this goal

Family Stress and Coping tool AND Family Capacity tool • Two survey scales rating a family’s self perceived level of coping and capacity

Early Childhood Individualised Support Plan tool • Attachment to Individualised Support Plan to rate outcomes at end of service period, and

barriers and enablers to these

1 in 4 poll Social Inclusion Survey • A population level survey rating levels of social inclusion related to service exclusion, economic

exclusion and social exclusion

Caution

Outcome measurement is a not an exact science – it tells us some

things but not everything, only part of the story

Not everything that can be counted counts,

and not everything that counts can be counted. (Albert Einstein)

It is very difficult to establish what CAUSED outcomes – that is, to attribute

the change to your service/support. Be cautious in your conclusions.

Be careful not to spend more time in outcome measurement and its

complexities than in taking action to improve the lives of people

Relevant publications by research team Campain, R.& Wilson, E. (under review) Life is what happens while you’re busy making plans: issues in

the development of outcome measurement for individualised plans for people with disability. Quilliam, C. & Wilson, E. (2011). Literature Review - Outcomes measurement in disability services: a

review of policy contexts, measurement approaches and selected measurement tools. Melbourne: Deakin University.

Hagiliassis, N., Nicola-Richmond, K., Mackay, A., & Wilson, E. (2011). Administration Manual - Measuring Outcomes in Services and Supports (MOSS) tool: a means to assess the outcomes of person-directed goals. Melbourne: Scope.

Nicola-Richmond, K., Wilson, E., Hagiliassis, N., & Mackay, A. (2011). Summary Sheet - Measuring Outcomes in Services and Supports (MOSS) tool. Melbourne: Scope.

Quilliam, C., Wilson, E., Hagiliassis, N. & Nicola-Richmond, K. (2010). Measurement of Services and Supports (MOSS): Project Overview and Findings. Melbourne: Deakin University.

Wilson, E. & Campain, R. (2011), Above and Beyond: Exploring outcomes and practices of Scope Southern Region Early Childhood Intervention Service for children with disability, Melbourne: Scope (Vic).

Wilson, E. & Campain, R. (2008). Problem solving, people skills, and personalised arrangements. Early results of research into individualised approaches for young adults receiving ‘Futures’ funding at Scope. C.Bigby & C. Fyffe (Eds.). Achieving their own lives. Proceeding from the Third Intellectual Disability Policy Roundtable, La Trobe University, October

Wilson, E. (2006). Defining and measuring the outcomes of inclusive community for people with disability, their families and the communities with whom they engage. C. Bigby; C. Fyffe & J. Mansell (Eds). From Ideology to Reality: Current issues in implementation of intellectual disability policy. Proceedings of the Roundtable on Intellectual Disability Policy. Bundoora: La Trobe University

Wilson, E.; Jenkin, E. & Campain, R. (2011). Outcome Measurement of Community Based Mental Health Services in Western Australia: Literature and Concept Summary. Melbourne: Inclusion Matters

PLUS many conferences and workshops and seminars

References ABS (2001) 4160.0 Measuring Wellbeing: Frameworks for Australian Social Statistics. Viewed at:

http://www.abs.gov.au/Ausstats/abs@.nsf/0/0120F9081822AAEBCA256B5F00804216?Open, on 31 October 2005.

Andreson E (2000), Tools of disability outcomes research: criteria for assessing the tools of disability outcomes research, Archives of physical medicine and rehabilitation, Vol. 81, No. 12, pp. S1-S4.

Commonwealth of Australia (2011) . 2010-2020 National Disability Strategy. An initiative of the Council of Australian Governments. Canberra. http://www.fahcsia.gov.au/sites/default/files/documents/05_2012/national_disability_strategy_2010_2020.pdf

Department of Human Services (2012) Department of Human Services Standards self-assessment report and quality improvement plan. http://www.dhs.vic.gov.au/about-the-department/documents-and-resources/reports-publications/department-of-human-services-standards-self-assessment-report-and-quality-improvement-plan

Horsch, K. (2005). Indicators: Definition and use in a Results-Based Accountability System. Harvard Family Research Project viewed at http://www.gse.harvard.edu/hfrp/pubs/onlinepubs/rrb/indicators.html on 31 Oct 2005.

King, S.; Rosenbaum, P. & King, G (1995). The Measure of Processes of Care: A means to assess family-centred behaviours of health care providers, McMaster University, Neurodevelopmental Clinical Research, Hamilton, Canada.

Schalock, R.L. (2004). The concept of quality of life: what we know and do not know, Journal of intellectual disability research, 48, (3): 203-216.

Saunders, P., Naidoo, Y. & Griffiths, M. (2008). Left out and missing out: Disability and disadvantage. Social Policy Research Centre, Sydney, Australia.

Saunders, P., Naidoo, Y. & Griffiths, M. (2007a). Deprivation & social exclusion in Australia. Social Policy Research Centre, Sydney, Australia.

Saunders, P., Naidoo, Y. & Griffiths, M. (2007b). Towards new indicators of disadvantage: Deprivation and social exclusion in Australia. Social Policy Research Centre, Sydney, Australia.

Triangle (n.d.) Outcomes Star. http://www.outcomesstar.org.uk/

Contact

Dr Erin Wilson

Senior Lecturer, School of Health & Social Development

Deakin University

erin.wilson@deakin.edu.au

Dr Nick Hagiliassis

Research Coordinator

Scope

nhagiliassis@scopevic.org.au