sara galantowicz july 11, 2006 using consumer surveys for quality management: a modified pes as one...

Sara GalantowiczJuly 11, 2006

Using Consumer Surveys for Quality Management: A Modified PES as One

Option

Presentation to Cash & Counseling States

Consumer Surveys for Self-Direction2

Agenda

• Consumer Surveys and Quality Management

• Modified Participant Experience Survey as one Survey Option– PES Overview

– ME CD-PAS version

– WV hybrid

• Additional Issues for Consideration


Consumer Surveys

• Consumer surveys are one of many sources of quality data– Consumer experience

– Knowledge, attitudes, and practices

– Satisfaction

• Where you want to end up is where you want to start– Research questions

– Data gaps

– Suspected problem areas

– Mandates

– Changes/policies requiring evidentiary support

– CMS and other reporting

– Who needs to know what


Consumer Surveys Options

• Designing your own instrument– Flexible

– Specific

– Reflects stakeholder input

• Using/modifying existing instruments– Stealing can be good!

• Efficient

• Testing

• Comparability

• Either approach requires involving the right players and technical expertise


Looking at Existing Tools

• Multiple tools available

• Vary on several dimensions– Target population– Domains or areas of emphasis– Method of administration– Role of proxy respondents– Length of survey– Resulting measures

• Evaluation Criteria– Scope– Testing

• Reliability and validity– Cost– Availability of benchmark data and support


Participant Experience Survey

PES is a technical assistance tool for States– Solicits participant feedback on HCBS waiver services/supports

– Generates indicators for QA/QI

– Developed by The Medstat Group, Inc. for CMS/DHHS

• Experience survey for participants only– Not a satisfaction survey

– No proxy version

– Designed to be administered in-person

– Optional

• Extensively tested before release– Cognitive testing of individual items

– Field testing to evaluate administration and tool properties


PES Development

• Combination of quantitative and qualitative methods

– Work Group provided technical assistance and oversight, and specified priority areas

– Review of related instruments

– Draft items to operationalize domains

– Cognitive testing (Phase I)

– Field testing (Phase II)


PES: Current Versions

PES E/D– Frail elderly and adults with physical disabilities– 33 indicators– Free automated survey software available

• Responses entered directly into laptops• Pull-down menus for data analysis

• PES MR/DD– Adults with mental retardation or developmental disabilities– 51 indicators– Includes 8 “core” questions for participants with severe cognitive

impairments– Automated version

• PES BI– Adults with acquired brain injuries– Up to 58 indicators (two optional modules)


PES: Domains

• Access to Care– Are participants’ needs for personal assistance, adaptive

equipment, and case manager access being met?

• Choice and Control– Do program participants have input into the types of services they

receive and who provides them?

• Respect and Dignity– Are program participants treated with respect by providers?

• Community Integration– Do program participants engage in activities and events of their

choice outside their homes, when they want to?


Sample Items

• Access to Care– Do you ever go without eating when you need to?

– Is this because there is no one there to help you?

– Have you ever talked with your case manager or support coordinator about any special equipment, or changes to your home, that might make your life easier?

• Choice and Control– Do you help pick the people who are paid to help you?

– Would you like to help pick the people who are paid to help you?

– When you are at home, can you eat when you want to? (MR/DD)

– Can you talk to your case manager or support coordinator when you need to?


Sample Items

• Respect and Dignity– Do the people paid to help you treat you respectfully in your home? (E/D)– Do support staff in other places, such as at work, or at a day program, say

“please” and “thank you” when they ask for something? (MR/DD)– Does anyone ever do mean things to you, such as yell at you? (MR/DD)

• Community Integration– Do you like to go out to eat? (MR/DD)– Do you go out to eat? (MR/DD)– Do you get to help pick where you eat out? (MR/DD)– Is there anything you want to do outside your home that you don’t do now?


Maine’s CD-PAS Survey

• State seeking feedback from CD-PAS participants– Non-elderly adults with physical disabilities

• Reviewed PES E/D and identified gaps

• Crafted new survey items

• Stakeholder input– Operating agency– Medicaid agency– Provider (Alpha One)– Consumers

• Testing– Cognitive– Field


Maine’s CD-PAS Survey

• Domains– Program Referral and Support

– Access to Services

– Respect and Dignity

– Technology and Community Integration

– Interviewer comments and observations

• Second year of use– Interviews conducted by staff from the Muskie School’s survey unit

– Early results reflected political uncertainty

– Modifications made based on findings


Modifying the PES in Maine

• Added items specific to self-direction– How PAs located and hired

– PAs’ relationship to waiver participant

– PA retention and back-up

– Support from Alpha One Independent Living Specialists

– Experience with self-direction

• Retained relevant items from PES– Respect and dignity

– Unmet need for personal assistance

– Adaptive equipment

– Physical and verbal abuse, theft


Crafting the new items in Maine

• General design principles and traps to avoid– Consistent response categories– Appropriate response categories– Adequate response categories– “Problem” words– Sound-alike words– Compound or double-barreled questions– Jargon and unfamiliar words– Complicated concepts– Relevance to consumers (screening questions)– Sensitive, appropriate language– Hypothetical questions– Scales– Specificity of times, dates, frequency, and amounts


Testing the new items in Maine

• How people answer a survey question– Comprehension

– Retrieval

– Estimation

– Response

• Each area is a potential source of bias

• Cognitive testing: determining if the question you think you are asking is the one waiver participants are hearing and answering

• Iterative process of asking draft questions with probes– Small n, longer interviews

– Discuss and act on results

– Resulted in many changes


Field Testing in Maine

• Larger n, shorter interviews

• Evaluate survey logistics– Reaching consumers

– Scheduling interviews

– Interview length

– Interview acceptability

• Obtain baseline data

• Evaluate appropriateness of tool

• Examine utility of findings


West Virginia’s Combined PES Tool

• State seeking feedback from both traditional E/D waiver participants and those enrolling in new Personal Options waiver

• Developed hybrid tool with separate skip patterns for traditional and self-directing waiver participants

• Multiple players– Medicaid agency

– Operating agency

– UCED staff

– Stakeholder advisory group


WV’s Combined PES tool (con.)

• Many common items– Access to care/unmet need– Respect and Dignity– Abuse/theft

• Choice and control items for traditional waiver– Primarily same questions from PES E/D– Some new items

• New Program Referral and Support items for Personal Options participants

– Modified the Maine questions– Additional items on resource consultants and financial

management services


Testing the WV Survey

• Iterative development process– Reviewed PES E/D and Maine tool

– Developed new questions

– Lots of review and feedback

• Consumer review

• Expert review

– Limited cognitive testing

• Field testing to begin this summer


Issues for Consideration

• What are your goals for the survey effort?– Whom do you want to survey and why?– Where will these data fit into your QM strategy?– What additional information would you like to capture?– With whom do you plan to share the data?– One-time vs. longitudinal use

• What is your target population?– Sampling frame

• Criteria for inclusion or exclusion– Sampling plan

• Sample size• Stratification• Proxies?


Additional Issues for Consideration

• Interviewer Selection and Training– Who will be conducting interviews?

– Who will attend training? Who will conduct it?

– Level of effort

• Ongoing data collection vs. limited period

• Contacting participants– How will sample respondents be contacted?

• State staff, case managers, other

– Scheduling and briefing respondents

– Data collection on non-respondents


Additional Issues for Consideration

• Pre-survey information– What kind of information can you access about participants before

the interview?

– How can you use this information to tailor the interview and provide appropriate supports

– Assess ability to complete survey

• Special issue for consideration!

• Timeframes– When will interviews start

– How long will the process last

– When would you like to be able to have and share results


Other Issues for Consideration

• Electronic data entry

• Psychometrics– Inter-rater reliability studies

– Validity concerns/cognitive screening

• Data Analysis– Who will develop the analysis plan?

– Who will conduct the analysis

• Acting on and sharing your results– Who should see the findings and how best to share them?

– Determining appropriate QI projects

• Involving stakeholders

• Need to consider this up front

sara galantowicz july 11, 2006 using consumer surveys for quality management: a modified pes as one...

Documents

selfdirection8 pes

selfdirection9 pes

modified pes

support slide

consideration slide

survey option pes overview

technical expertise

cmsdhhs experience survey