quality of life measures: mental health services and consumer satisfaction

Quality of Life Measures:

Mental Health Services and Consumer Satisfaction

THE WASHINGTON INSTITUTE FOR MENTAL HEALTH RESEARCH & TRAINING

For further information, please contact: Dennis McBride: (253) 756-2335 | [email protected]

Catherine M. Wilson: (253) 761-7573 | [email protected]

Part 1

Consumer Satisfaction Surveys

The Mental Health Statistical Improvement Program(MHSIP)


For further information, please contact: Dennis McBride: (253) 756-2335 | [email protected]

Catherine M. Wilson: (253) 761-7573 | [email protected]

MHSIP Emerges 1976 – MHSIP Advisory Group (developed data standards for public

mental health systems) 1987 – National Research Institute (NRI) to Support MHSIP 1989 – MHSIP Policy Group (Refined Data Standards for Mental Health

Systems) 1996 – MHSIP Consumer-Oriented Report Card (Developed base

survey we now use). National Association of State Mental Health Program Directors (NASMHPD) was created to further develop the Application of MHSIP.

1997 – Five State Feasibility Project (refined survey) 1998-2001 – 16 State Study on Mental Health Performance Measures

(refined survey) 2001- 2014 – Current Survey

Goal & Audiences

The consumer surveys are designed to examine service quality issues related to Washington State’s delivery of state-funded mental health services.

Using Survey Data for:Federal ReportingPerformance Indicator Work GroupsProvider Agencies and RSNs (e.g., EQROs)Consumer Groups


Consumer Satisfaction Surveys 2013

Federal Uses• Data Infrastructure Grants (DIGS)• National Outcome Measures (NOMS)• Government Performance and Results Act (GPRA)• Uniform Reporting System (URS).

The MHSIP Consumer Survey is now being implemented in 55 states/territories for the adult survey and 54 states/territories for youth surveys.

Surveys to Meet CMS Requirements

Survey Year

Statewide Sample Based Outcomes Survey

1998

Children with Special Needs 1999

MHSIP Adult Consumer Survey (10 surveys)

2002-2013

MHSIP Child and Family Consumer Survey (9 surveys)

2002-2013



2013 MHSIP Survey Samples

• Persons receiving at least one hour of service within a 6 month time frame: May-Oct 2012

Adult Survey: 18+ years oldYouth Survey: 13-20 years oldFamily Survey: <13 years old

• Stratified Random Sample (Stratified by RSN, Age, Minority Status

• Probability Proportionate to Size (Size adjusted for small RSNs)



WIMHRT’s CATI Lab System



Sources of Data



Contact Informati

on

ProviderOne Data System

Research and Data Analysis

(DSHS)

ACES Barcode data – Economic

Services Administration

(DSHS)

Selecting Participants



49,746

5,646

1,334

30,311

3,924

924

Adult Consumer Survey (ACS)

Sample Frame

Drawn

Sample

Respondent Sample

Child & Family Consumer Survey (CFCS)

Sample Frame

Drawn Sample

Respondent Sample

F=604Y=320

ACS & CFCS Disposition


Consumer Satisfaction Surveys 2013 Adult

Disposition N (%)

Family/YouthDisposition

N (%)

Completions 1344 (23.6) 924 (23.5)

Refusals 647 (11.5) 260 (6.6)

Incorrect Numbers 2073 (36.7) 1015 (25.9)

No MH Services 46 (0.8) 54 (1.4)

Language Barrier 114 (2.0) 9 (0.2)

Unavailable 310 (5.5) 86 (2.2)

Deceased 33 (0.6) 1 (0.03)

No Answer 932 (16.5) 1534 (39.1)

Other 157 (2.8) 41 (1.04)

TOTAL 5646 3924

Response & Cooperation Rates Over Time



0102030405060708090

100ACS % Coopera-tion Rate

ACS % Response Rate

CFCS % Coopera-tion Rate

CFCS % Response Rate

2004 Data CFCS % Cooperation Rate N/ACFCS % Response Rate N/A

States’ MHSIP Completion Rates 2008

State Sample Method

Survey Method

Adult N Adult % Y/F N Y/F %

Texas Stratified Random

Mail Out 334 19% 389 18%

Wisconsin* Random Mail Out 803 36% 524 39%

Vermont Random Mail Out 737 36% 251 20%

Washington State

Stratified Random

Telephone/Mail Out

1500 24% 906 33%

Illinois Random Mail Out 561 19% 586 17%

Oregon Total Population

Mail Out 3240 24% Combined



Source: Sampling and the MHSIP Consumer Surveys: Techniques, Models, Issues. Mary E. Smith & Vijay Ganju (June, 2008).

*Removes non-contacts from the denominator.

Total number of attempts/calls: 32,628

ACS: 22,023CFCS: 10,605

Average number of attempts/calls per completion: 14.45



2013 Calls & Completions



Total number of attempts/calls: 341,078

ACS: 231,995CFCS: 109.083

Total number of completions: 24,005

ACS: 15,592CFCS: 8,413

Total Calls & Completions

Representativeness



Age Hours Age Hours0

10

20

30

40

50

Respondent Sample Drawn Sample Sample Frame

Female % Minority % Female % Minority %0

10203040506070

ACS CFCS

Adult/Youth Perception of Outcomes

As the result of services I received:

1. I am better at handling daily life.2. I get along better with family members.3. I get along better with friends and other

people.4. I am doing better in school and/or work.5. I am better able to cope when things go wrong.6. I am satisfied with my family life right now.

Scoring: 1-Strongly Disagree; 2-Disagree; 3-Undecided; 4-Agree; 5-Strongly Agree THE WASHINGTON INSTITUTE FOR MENTAL HEALTH RESEARCH & TRAINING

Consumer Satisfaction Surveys 2013 Alpha = 0.90

Scale

Scale Construction



Scale ACS CFCSGeneral Satisfaction .87 .92

Appropriateness/Quality of Services .87 .91

Participation in Treatment Goals .56 .70

Perceived Outcomes .91 .90

Perception of Access .82 .48

Satisfaction with Staff NA .84

Cultural Sensitivity of Staff NA .84

NOMS Functioning* .87 NA

NOMS Social Connectedness* .79 .81

Stigma* .86 .85(Youth only)

*Added in 2007

ACS Scales

General Sat-isfaction

Quality of Service

Participation in Treatment

Perception of Access

Perceived Outcomes

0%

25%

50%

75%

100%

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Strongly Agree

Agree

Undecided

Disagree

Strongly Disagree

Average ScorePerc

ent S

atisfi

ed



CFCS Scales



0%

25%

50%

75%

100%

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Strongly Agree

Agree

Undecided

Disagree

Strongly Disagree

Average Score

Perc

ent S

atisfi

ed

Other Relationships

• Demographics and Total Service Hours• Living Situation by Age, Gender, & Ethnicity• Employment Status by Age, Gender, & Ethnicity• Employment Status by Age, Gender, & Ethnicity• Medical/Insurance Information by Age, Gender, &

Ethnicity• Satisfaction Scales by Age, Gender, & Ethnicity• Satisfaction Scales by RSN



What two things do you like the most about the services you received?

Services14%

Support21%

Therapy/Counseling11%

Medications1%

Access6%

Staff1%

Professional Staff22%

Environment1%

No Response21%

General Positive 1% General Negative

1%

ACS First Response

Services

Support

Therapy/Counseling

Medications

Access

Staff

Professional Staff

Environment

No Response

General Positive

General Negative



Open-Ended Responses: Adult – Like Most



Support:

”The support that they give me. If I need someone to talk with, they are

there. If not, they call right back. They are good listeners and very

understanding .”

”You are not rejected on account of anything you might be

experiencing/feeling. If there’s one counselor that you’re just not clicking

with, they will rotate you to another. They’ll bear with you and just keep

going until you can identify your feelings and handle them.”

What two things do you like the least about the services you received?



Services16%

Support21%

Therapy/Counseling8%

Medications1%

Access7%

Staff2%

Professional Staff17%

No Response25%

General Positive 2% General Negative

1%

CFCS First Response

Services

Support

Therapy/Counseling

Medications

Access

Staff

Professional Staff

No Response

General Positive

General Negative

Open-Ended Responses: Family – Like Least



Access:

”The location and difficulty in getting after school appointments .”

”Long waiting list and it takes months to see a psychiatrist.”

”I liked least that they are short of time. They had so few choices

of time. My child would miss school for appointments.’

STIGMA





ACS Perceived Stigma by Gender/Age

Female Male 18 to <21 21 to <41 41 to <61 61 to <760%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0

0.5

1

1.5

2

2.5

3

3.5

4

Most Stigmatized (3.5-4)

More Stigmatized (2.5-3.49)

Less Stigmatized (1.5-2.49)

Least Stigmatized (1-1.49)

Average ScorePerc

ent S

tigm

atize

d

Consumer Satisfaction Surveys 2013 ACS Perceived Stigma by Gender/Ethnicity



ACS Perceived Service Outcomes by Stigma


ACS Consumer Satisfaction Surveys 2012 -13

Service Perception Scores Stigma Scale

Satisfaction with Services -.124

Appropriateness and quality of Services

-.175

Participation in Treatment Goals

-.166

Perceived Outcome of Service

-.241

Perception of Access to Service

-.189

NOMS Functioning Scale -.232

NOMS Social Connectedness Scale

-.312

N = 2578 p < .001

Correlations: Service Perception by Stigma

Part 2

Trends


ACS Perceived Stigma by Year



2007 2008 2009 2010 2011 2012 20130%

25%

50%

75%

100%

0

0.5

1

1.5

2

2.5

3

3.5

4

Most Stigmatized (3.5-4)More Stigmatized (2.5-3.49)Less Stigmatized (1.5-2.49)Least Stigmatized (1-1.49)Average Score

Perc

ent S

tigm

atize

d

Trends in Mean ACS Satisfaction Scores Over Time



2002 2004 2006 2007 2008 2009 2010 2011 2012 20133.2

3.3

3.4

3.5

3.6

3.7

3.8

3.9

4

4.1

4.2

General Satisfation Quality of ServiceParticipation in Treatment Perception of AccessPerceived Outcome



Trends in Mean CFCS Satisfaction Scores Over Time

2002 2005 2007 2008 2009 2010 2011 2012 20133.4

3.6

3.8

4

4.2

4.4

General Satisfation Satisfation with StaffPerception of AccessParticipation in Treatment Cultural Sensitivity of StaffAppropriateness of Services Perceived Outcome



Functioning/Connectedness Scores Over Time

2007 2008 2009 2010 2011 2012 20133.5

3.6

3.7

3.8

3.9

4

4.1

4.2

4.3

ACS Social ConnectednessACS Functioning CFCS Social ConnectednessCFCS Functioning

States’ Adult MHSIP Scores 2010

ACS Measures Texas Wisconsin Vermont Washington State

Illinois Oregon

Access to Services

78.40% 74.40% 82.20% 68.20% 84.60% 74.00%

Quality/Appropriateness of Services

83.70% 79.60% 84.60% 80.70% 83.20% 82.90%

Outcome of Services

57.20% 57.20% 69.30% 61.50% 67.30% 57.20%

Participation in Treatment

71.00% 67.30% 76.90% 71.90% 85.90% 66.80%

Overall Satisfaction

89.00% 76.70% 84.40% 77.10% 85.70% 80.40%



Source: 2010 CMHS Uniform Reporting System Output Tables

Note: Scores represent the average of those positively agreeing or strongly agreeing with each scale.

Summary of Findings: ACS

• Satisfaction scores on most scales remain high, with at least 70% of consumers on most scales saying they are satisfied or very satisfied with services.

• Since 2002, satisfaction score trends have shown slight but steady improvement over time.

• Across years, Perceived Outcomes has remained the lowest, with more than one third of consumers being undecided or dissatisfied in this area.

• Consumers continue to be most satisfied with Appropriateness and Quality of Services.

• Perceived Stigma affects nearly half of consumers and has remained relatively constant since 2007.



• Satisfaction scores on most scales remain high, with at least three quarters of consumers on most scales saying they are satisfied or very satisfied with services.

• Since 2002, all areas of youth and family service satisfaction have continued to improve.

• Cultural Sensitivity of Staff and Satisfaction with Staff continue to be the highest rated areas.

• Perceived Outcomes is consistently the lowest rated area.



Summary of Findings: CFCS

HTTPS://DEPTS.WASHINGTON.EDU/PBHJP/DEV/PROJECTS-PROGRAMS/PAGE/MENTAL-HEALTH-STATISTICS-IMPROVEMENT-PROGRAM-CONSUMER-SATISFACTION-SURVEYS



quality of life measures: mental health services and consumer satisfaction

Documents

mhsip consumer survey

mental health services

family consumer survey

survey data

adult survey

mhsip survey samplespersons

base survey

washington institute