presented by peg capo, executive director, ddrb kathy williams, project manager, ddrb lisa richter,...

Presented byPeg Capo, Executive Director, DDRBKathy Williams, Project Manager, DDRBLisa Richter, richterIntelCynthia Cluff, richterIntel

5 Habits of Highly Effective Consumer Satisfaction Research

A case study of capturing meaningful consumer feedback for the DDRB of St. Charles county

MACDDS Annual ConferenceOctober 2009

Introduction

DDRB is a public taxing entity that enters into funding contracts with agencies that serve individuals with developmental disabilities in St. Charles County. This public tax was established in 1977 when voters of St. Charles County approved to tax themselves up to sixteen cents per $100.00 of assessed property value.

DDRB funds:

29 agencies plus case management for DMH and DDRB

11 service areas or groupings of programs

53 programs, plus 2 case management programs

Serving approximately 1,500 consumers in St. Charles County

A little less than half of the consumers receive 1 service, a little more than one-third of the consumers receive 2-3 services, and a little less than 20% receive 3 or more services

We needed feedback directly from the families and individuals that we serve in a way that is consistent across agencies and meaningful to us.

Two key uses of Consumer Satisfaction:

Planning tool for continuous improvement

Accountability that we are good stewards of the money we are handling for our constituents

It is a systematic and scientific approach to collecting feedback from the consumer’s viewpoint.

Shifts planning and decision making from intuitive information gathering to orderly and objective investigating

Emphasizes understanding consumer’s needs and measuring success as decided by the consumer

Characteristics of valuable customer satisfaction research are:

Timely – research is conducted on a schedule where it can be used to influence planning and decision making, the data is current

Complete – right amount of information

Relevant – collects measures that are meaningful to customer satisfaction, anticipates the kinds of information that will be useful

Quality – Data represents the true situation

What is Consumer Satisfaction Market Research?

What Does Consumer Satisfaction Research Allow You to Do?

Describe:

Paint a picture of the situation from the customer’s viewpoint – who, what, when …

Diagnose:

Explain the data – reasons behind something from the customer’s view point, dig deeper

Predict:

Using statistics predict the results of something

Habit #1: First Things First – Develop a Blueprint

First step in consumer satisfaction research is a well laid out plan.

A well laid out plan includes determining:a. Your OBJECTIVE for conducting the research

b. Who are you wanting to talk to – SAMPLE PLAN

c. How are you going to collect the data – METHODOLOGY

A. Clear and precise OBJECTIVE of what is to be achieved, what information is being sought.

DDRB county-wide Consumer and Family Satisfaction Survey objective:By listening directly to the voice of the consumer and their families or guardians measure the quality and effectiveness of DD services in St. Charles County

Identify areas for quality improvement

Measure satisfaction in a consistent format across agencies

Look at survey results in the following ways:

• System-wide

• By Consumers and Parents/Guardians

• By Service Area (11)

• Taking each of 29 agencies plus 2 case management agencies and look at results for each their funded programs


B. A well defined SAMPLE PLAN - how many and who

DDRB was going to sample the entire population.

DDRB definition of population included:

All consumers 16 years of age or older

The parent/guardian of every consumer

A survey for each service consumer received – one to consumer if 16+ and one to the parent, could be multiple surveys to one consumer and one parent

Served in the last fiscal year – a definite time frame, go back no more than 12 months


A database was developed, capturing all consumers, every service they received, their parent/guardian.

Sent Excel spreadsheet to each agency requesting a list of their population served in fiscal year, with defined columns of data needed, exact format specified

Assumed no email addresses available.

One row of data per consumer with 1 service listed, many consumers were listed multiple times for each service received.

This step of database development took a lot of time!

End result was 5,000+ records merged together from all 31 agencies.

Agency

Program

Last Name-Consumer

First Name-Consumer

Month Born

Year Born

Gender of Consumer

Consumer-Street Address

Consumer- City

Consumer-State

Consumer Zip

Consumer Phone #

Last Name-Parent

First Name-Parent

Parent-Street Address

Parent- City

Parent-State

Parent-Zip

Parent Phone #


C. Identify most appropriate METHOD for collecting data considering the situation:

Characteristics of population

Database information you have on population

Budget

DDRB study done by mail. Targeted postcard reminders. Targeted telephone follow-up.

Things to keep in mind when doing a mail survey.

Lower cost compared to other methods (i.e., telephone)

Takes longer time to collect the data

Have to be very clear – wording and instructions, no interviewer to clarify

Have to be aware of the length of questionnaire – can’t be too long

Hard copy only – risk of order bias

Lose control of sample – it’s self selection

• Many won’t respond

• Can’t control who actually answers the survey

• Those not responding may impact the representativeness of the sample obtained

Habit #2: Begin With the End in Mind – Develop a Questionnaire

Purpose of the questionnaire: takes the objectives and translates them into specific questions that are meaningful to the client and understandable to the respondent.

Develop a valid and reliable QUESTIONNAIRE.Valid - Measuring what it is supposed to measure – do the scope of questions cover the topic area?Reliable - Provide consistent results

Validity – What is important to consumers and families to be satisfied?Use expertise, knowledge

Stakeholder panel – agencies, board members …

Past surveys

Other organization’s surveys

Reliability – Will it provide consistent results over time and under different circumstances?

Clear wording, simple, understandable, not ambiguousOne question addresses one thingCan person answer, is it at their level, can they recallReliable measurement scale

• At least 5 point scale, to get intensity of feelings, label scale either every pt or end pts

• Balanced – equal satisfied and not satisfied points on scale, some exceptions

• Non-forced choice – give them a don’t know or doesn’t apply option, some exceptions

Habit #2: Begin With the End in Mind – Develop a Questionnaire

Give clear directions. Break down barriers to participation

Give them comfort in the privacy of their feedback – Tell them it is anonymous and their name won’t be tied to them or identified to the sponsor of the study in any way

Give them a contact to reach if they have any questions, issues, or need clarifications

• DDRB cover letter with survey did this

Identify the consumer, the agency, and the service – they are to think of when they answer this questionnaire

Pre-test!!!

Confirms if questionnaire is valid and reliable.

Have a small number of consumers and families complete the survey. Be there in person. See how they answer things. Ask them if anything is unclear, or left out.

Habit #2: Begin With the End in Mind

Benchmark2005 DDRB Consumer Satisfaction

Q2. eliminate

Q6. eliminate

Habit #2: Begin With the End in MindGuardian Satisfaction Survey

Thinking about the specific AGENCY and the specific PROGRAM listed in the box below, tell us what you think by checking a response for each item. If you do not know about an item or it does not apply to you, please check ‘don’t know / doesn’t apply’ for that item.Your opinions about this AGENCY: «Agency1» «Agency1_ID»

Your opinions about this PROGRAM: «SERVICE1» «SERVICE1_CODE»Consumer for whom you are Guardian: merge in consumer first and last name

1. Staff is respectful and helpful reworded2. Staff follows through in a timely manner

(with appointments, meetings, requests, care plans, etc.) NEW3. Staff returns phone calls promptly NEW4. Staff meets your needs NEW5. Staff keeps you informed NEW6. Staff is well trained and knowledgeable NEW7. Program you participate in helps you 8. Program you participate in is well organized NEW9. Program service location is safe and healthy 10. Program service location is convenient NEW11. Overall rating for quality of staff and program reworded

Poor Not Very Average Very Excellent Don’t Know orThinking of this service the consumer Good Good Doesn’t Applyreceives from this agency, give us youropinion on… 1 2 3 4 5 0

12. What do you like best about the staff and this program? (Please be as specific as possible) NEW – directed open ended questions

13. If you rated any of the above areas a ‘Poor’ or ‘Not Very Good,’ how can we improve this area? (Please be as specific as possible) NEWNot At Not Very Neutral Somewhat Very Don’t Know orAll Likely Likely Likely Likely Doesn’t Apply

1 2 3 4 5 014. Likely to recommend this program to a friend

15. Is there anything else that can be done to better meet your needs? (Please be as specific as possible) NEW«G_ID»

NEW- ID consumer to think about

NEWScale and DK

NEWScale and DK

NEW-directions NEW pre-coding

Habit #2: Begin With the End in Mind

Benchmark2005 DDRB Case Management Satisfaction Survey

Eliminated most of the questions:

Q1, Q2, Q3, Q4, Q4a, Q5a

Habit #2: Begin With the End in MindGuardian Satisfaction Survey

Thinking about the specific AGENCY and the specific PROGRAM listed in the box below, tell us what you think by checking a response for each item. If you do not know about an item or it does not apply to you, please check ‘don’t know / doesn’t apply’ for that item.Your opinions about this AGENCY: «Agency1» «Agency1_ID»

Your opinions about this PROGRAM: «SERVICE1» «SERVICE1_CODE»Consumer for whom you are Guardian: merge in consumer first and last name

1. Staff is respectful and helpful NEW

(with appointments, meetings, requests, care plans, etc.) NEW3. Staff returns phone calls promptly NEW4. Staff meets your needs NEW5. Staff keeps you informed NEW6. Staff is well trained and knowledgeable NEW7. Staff makes you aware of services NEW8. Staff is an advocate for your needs NEW9. Staff’s ability to solve problems NEW10. Have consistent case management staff NEW11. Overall rating for quality of case mgmt staff reworded

Poor Not Very Average Very Excellent Don’t Know orThinking only of case management you Good Good Doesn’t Applyreceive from this agency, give us youropinion on… 1 2 3 4 5 0

12. What do you like best about the case management staff and service you receive? NEW – directed open ended questions

13. If you rated any of the above areas a ‘Poor’ or ‘Not Very Good,’ how can we improve this area? (Please be as specific as possible) NEW

14. Do you have an Annual Written Plan? Yes No Don’t Know definition of plan provided15. Have you seen or do you have a copy of your Annual Written Plan? Yes No Don’t Know NEW

16. Is there anything else that can be done to better meet your needs? (Please be as specific as possible) NEW«G_ID»

NEW- ID consumer

NEWConsistent scaleConsistent q’s

A definition of Case Management was stated

NEW pre-coding

Q7-Q10 unique to CM

Habit #3: Two Ears One Mouth – Listen to the Consumer

COLLECT THE DATA.

This takes time! DDRB took about 7 weeks to collect the data.

TOTAL: 31 AGENCIES Consumers(16 years or older) Guardians Total

Questionnaires Mailed Out 3,322 3,908 7,230

Can’t do, Returned past cut-off (210) (186) (396)

Undeliverable/Unreachable (151) (113) (264)Completed Questionnaires 1,081 1,347 2,428

TOTAL RETURN RATE (%) 1081/2961= 37%1347/3609= 37%

2428/6570= 37%

29 Funded Agencies:Questionnaires Mailed Out

5,200

29 Funded Agencies: Completed Questionnaires

785 9831,768

(38% return rate)

DHM & DDRB Case Mgmt:Questionnaires Mailed Out

2,030

DMH & DDRB Case Mgmt: Completed Questionnaires 296 364

660

(35% return rate)

Habit #3: Two Ears One Mouth – Listen

Follow up to achieve return rate.

Post card reminders

DDRB email reminders to agencies and several public administrators

Telephone follow up to targeted smaller populations, in certain agencies, service areas, or programs.

Things to keep in mind with telephone surveying:

Can obtain people that may not complete mail self administer – more representative sample (all public opinion polls are done by telephone because it is currently accepted as the method that gives the most accurate representation of the population)

Capturing open ended responses, maybe more likely to answer, interviewer can probe

Telephone is more costly compared to mail or web-based

Habit #4: Synergize – Tell a Story

Weave the data and findings into a story that paints a picture of what consumers and parents/guardians say.

Reveal strengths and opportunities.

Include not only key findings but drill down to tie in underlying reasons (i.e., using open-ended responses).

Provide insightful and actionable recommendations that go beyond identifying a broad service area for improvement.

Tools to help weave a story and organize information. Organize performance attributes – For DDRB listed attributes most important to least

important.

We conducted some statistics and determined the strongest correlating/highest impact attributes on Overall Performance.

Decide on a reporting system – For DDRB we reported top two box score (rating Excellent/Very Good)

Handle open-ended questions – Group open ended responses and quantify


2005 Story

Questions in order of questionnaire

6 numbers reported for each question – 42 %’s in total on table for Guardians –

Guardians on one slide, consumers on another, no total combining

Habit #4: Synergize – Tell a Story DDRB funded agencies receive a high score of 90% for Overall quality of staff

and program. Excellent work by agencies!

Key strength is Staff is respectful and helpful (92%).

Several attributes receive moderate scores of between 80% - 87%. These are areas for improvement.

29 DDRB Funded Agencies Total Consumers Guardians

OVERALL QUALITY OF STAFF AND PROGRAM 90% 90% 90%

Staff is well trained and knowledgeable 89% 90% 88%

Staff follows through in a timely manner (with appointments, meetings, requests, care plans, etc.)

88% 88% 88%

Staff meets your needs 88% 88% 88%

Staff is respectful and helpful 92% 92% 93%

Program service location is convenient 87% 89% 86%

Program you participate in is well organized 87% 88% 86%

Program service location is safe and healthy 93% 92% 93%

Program you participate in helps you 91% 90% 91%

Staff keeps you informed 84% 85% 83%

Staff returns phone calls promptly 87% 86% 88%

High Score 88%+Moderate Score 80%-87%Low Score < 80%

HIG

HER

IM

PA

CT

LO

WER

IMP

AC

TM

OD

ER

ATE

IMP

AC

T

• Attributes ordered by impact on Overall

• High/low score targets set

• Top 2 box score for each attribute-11 %’s for each grouping of data

• Show total, consumers, & guardians on one slide


2005 Story – open ended “Comments” listed in total and were reported by agency in the format as seen here.

• The lady I got is super nice, he/she helps me 100 percent. He/She is very responsive and he/she was there for me even right after his/her surgery. I cannot read or write and the burden was lifted of my chest after I got this service. Thank you very much!!

• The staff/ teachers are absolutely awesome! My (child) has made huge changes in his/her speech and behavior in the last year. These guys are wonderful we would have never been ready for kindergarten next fall if it wasn’t for them!

• He/She is happy and well cared for. The staff are all caring and provide a loving family atmosphere for him/her. I can not say enough good things about (Agency). I feel confident that all his/her needs are being met.

• The staff are Saints! The staff are all angels! Our association with this (Service) gives (Name) an indescribable excitement and anticipation for the activities he/she has done. The work, sweat, planning, staffing of this activity is worth all their hard work for the participants! They talk to each other about the classes & can’t wait to begin each year! Thanks to you for the facilities and the volunteers. The kids LOVE them.

• The agency has been great for our child. It has allowed him/her to participate in group sporting activities & to succeed with the help of the staff. The peer & staff support has allowed our (child) to feel confident in his/her "abilities."

• It took them 9 months before they would return my call. Once they did come out, they said they were going to send me information and never did; there was no follow up.

• Should respond quicker and let me know why you can’t fill requests.• They promise things but do not follow through• Need more employees• Inconsistent discipline• My child complains that the staff people there always yell and he/she thinks that is not necessary• Staff not always as responsive as I think they should be• I am not satisfied with the cleanliness of the carpet in the group home. Dirty and unsafe condition.• When there was trouble or a problem on the job, always sided with employer• Need more staff


Organize open ended questions by creating mutually exclusive and exhaustive answer categories. Assign the participant’s answers to the categories.

Tell a story of the open ended comments by frequency of response.

29 DDRB Funded Agencies Total Consumers Guardians

Staff is concerned, caring, nice, helpful, respectful, excellent care

30% 32% 29%

Program helps consumer, consumer enjoys, fun, provides learning and socialization

15% 18% 12%

Staff well trained, experienced, knowledgeable 7% 4% 9%

Good activities, variety, benefit consumer 5% 7% 4%

Keep us informed, good communication 4% 4% 5%

Good structure, organized, good hours, location, 4% 3% 5%

Safe for consumer 1% 1% 1%

Provides relief for guardian 1% 1% 2%

What do you like best about the staff and this program?

• With coding of open ends can diagnose systematically

• Can sort by agency, program, service area

• No new areas of importance are mentioned that were not included on the survey

Base for total is all 1768 respondents, 41% of the total gave no answer (not shown in table above).Multiple responses accepted.

29 DDRB Funded Agencies Total Consumers GuardiansImprove responsiveness, better communication, return phone calls/e-mails, stay in touch

30% 22% 34%

Better structure, more organized, need timely notice of activities/events, reduce wait time for assistance, provide transportation, inconvenient location, facility improvements

22% 16% 26%

Better trained staff, i.e., on disabilities, patient interaction, more helpful to consumer 16% 12% 19%

Less staff turnover, more staff 6% 5% 6%

Additional/better activities, other times 2% 2% 2%

Need cleanliness of home, personal hygiene 1% - 1%


• Can obtain insight in consumer’s or guardian’s own words as to areas to improve – in a systematic way

If you rated any of the above areas a ‘Poor’ or ‘Not Very Good,’ how can we improve this area?

Base for total is 146 respondents (146 out of 1768) who gave at least one ratings question a ‘Poor’ or ‘Not Very Good.’ Of this 146, 29% of the total gave no answer (not shown in above table).Multiple responses accepted.


Told the story of Consumer and Family Satisfaction by organizing and telling the findings:

With a System Wide Report & Report for the Board With an Agency Report Card


Conclusion from 2005

It would appear that there are at least four areas, where quality improvement initiatives can be focused. These areas would be person centered planning, case management, Individualized Support Services, and Employment Services. An examination of the qualitative comments provided by respondents can also be helpful in combination with survey data to generate areas of focus for quality improvement initiatives.


Employment

OVERALL QUALITY OF STAFF AND PROGRAM 85%

Staff is well trained and knowledgeable 85%Staff follows through in a timely manner 84%Staff meets your needs 81%Staff is respectful and helpful 88%Program service location is convenient 87%

Program you participate in is well organized 83%Program service location is safe and healthy 91%Program you participate in helps you 85%Staff keeps you informed 78%Staff returns phone calls promptly 82%

Employment


HIG

HER

IM

PA

CT

LO

WER

IMP

AC

TM

OD

ER

ATE

IMP

AC

T

2008 Employment Service Area – Staff keeps you informed looks like an opportunity area.



HIG

HER

IM

PA

CT

LO

WER

IMP

AC

TM

OD

ER

ATE

IMP

AC

T

Community LivingPEP

(n=197)

Day Habilitate

(n=53)

Resident (n=50)

Respite(n=26)*

ILA(n=13)*

Employ.(n=45)

OVERALL QUALITY OF STAFF AND PROGRAM

94% 93% 92% 84% 85% 81%

Staff is well trained and knowledgeable 93% 92% 92% 71% 85% 86%Staff follows through in a timely manner 92% 94% 94% 83% 75% 77%

Staff meets your needs 88% 94% 94% 86% 85% 79%Staff is respectful and helpful 95% 91% 98% 88% 85% 87%Program service location is convenient 90% 90% 93% 72% 78% 90%Program you participate in is well organized 91% 92% 94% 73% 82% 87%

Program service location is safe and healthy 94% 94% 96% 84% 70% 92%

Program you participate in helps you 94% 96% 98% 92% 82% 82%

Staff keeps you informed 85% 90% 90% 78% 75% 75%Staff returns phone calls promptly 87% 96% 95% 75% 75% 77%Likely to recommend to a friend 96% 96% 98% 88% 90% 89%

Employment Follow Along

Improve responsiveness

“Numerous times late or no show, didn't listen to specific information given.” “Make sure phone calls are returned next day.” “Consistency between job coaches; letting parents know more information.”

Habit #5: One Mouth Two Ears - Talk, Plan, Action, Reward

Take the data and recommendations and connect with stakeholders, develop action plans for improvement areas, reinforce positive areas.

System-wide recommendations to the DDRB of St. Charles.

1. Communicate to agency’s staff appreciation for care being given

2. Communicate that programs are making a difference

3. Strengthen communications and responsiveness given to families

4. Strengthen organizational or structural aspects

5. Provide additional training on disabilities, job knowledge, and appropriate conduct

6. Consider demand or need for enhancements to programs, additional programs or activities

7. Put forth effort to have less staff turnover

DDRB examples of using the data for quality improvement.

It is a ___________ and ____________ approach to collecting feedback from the ____________ viewpoint

Characteristics of valuable customer satisfaction research are:

________ – research is conducted on a schedule where it can be used to influence planning and decision making, the data is current

________ – right amount of information

________ – collects measures that are meaningful to customer satisfaction, anticipates the kinds of information that will be useful

________ – Data represents the true situation

What is Consumer Satisfaction Market Research

The questionnaire take the objectives and translates them into specific questions that are meaningful and understandable.

A questionnaire should be:

__________________ Scope of questions cover the topic area

__________________ Provide consistent results

What is something you can do with the questionnaire before the data is actually collected to test if the questionnaire is of good quality.

___________________

What is Consumer Satisfaction Market Research

Conclusion

We hope these 5 Habits to Highly Effective Consumer Satisfaction Research will help bring even more success to the children and families and the adults that you serve.

Content for these slides prepared by Cynthia Cluff

presented by peg capo, executive director, ddrb kathy williams, project manager, ddrb lisa richter,...

Documents