Conducting Statewide Conducting Statewide Customer Satisfaction SurveysCustomer Satisfaction Surveys

Brad Sickles

USDOL / ETA

ETA Performance Accountability Team

Measuring Customer Measuring Customer SatisfactionSatisfaction

The The MeasuresMeasures

Participant Customer SatisfactionParticipant Customer Satisfaction

Employer Customer SatisfactionEmployer Customer SatisfactionCustomer ratings on each of the three core Customer ratings on each of the three core questions regarding satisfaction are questions regarding satisfaction are weighted to account for regional weighted to account for regional differences and reported on a 1 to 100 differences and reported on a 1 to 100 scale. Computation of the ACSI is scale. Computation of the ACSI is discussed in TEGL No. 6-00.discussed in TEGL No. 6-00.

The American Customer Satisfaction The American Customer Satisfaction Index (ACSI) is used to report statewide Index (ACSI) is used to report statewide results to the Department.results to the Department.

The Core ACSI QuestionsThe Core ACSI Questions1. Utilizing a scale of 1 to 10 where “1” means “Very 1. Utilizing a scale of 1 to 10 where “1” means “Very Dissatisfied” and “10” means “Very Satisfied” what is your Dissatisfied” and “10” means “Very Satisfied” what is your overall satisfaction with the services provided from overall satisfaction with the services provided from ________?________?

2. Considering all of the expectations you may have had 2. Considering all of the expectations you may have had about the services, to what extent have the services met about the services, to what extent have the services met your expectations? “1” now means “Falls Short of Your your expectations? “1” now means “Falls Short of Your Expectations” and “10” means “Exceeds Your Expectations” and “10” means “Exceeds Your Expectations.”Expectations.”

3. Now think of the ideal program for people in your 3. Now think of the ideal program for people in your circumstances. How well do you think the services you circumstances. How well do you think the services you received compare with the ideal set of services? “1” now received compare with the ideal set of services? “1” now means “Not very close to the Ideal” and “10" means “Very means “Not very close to the Ideal” and “10" means “Very Close to the Ideal.”Close to the Ideal.”

Participant Customer Participant Customer SatisfactionSatisfaction

The The MeasuresMeasures

The weighted average of participant ratings on The weighted average of participant ratings on each of the three questions regarding overall each of the three questions regarding overall satisfaction reported on a 1 to 100 scale. satisfaction reported on a 1 to 100 scale.

Who:Who: Exiters Exiters

When:When: 1to 60 days after date of 1to 60 days after date of hard exit hard exit or or date a date a soft exitsoft exit was determined was determined

How:How: Telephone survey Telephone survey

Required completions:Required completions: 500 statewide 500 statewide

Required response rate:Required response rate: 50% first year; 70% 50% first year; 70% following yearsfollowing years

Employer Customer SatisfactionEmployer Customer Satisfaction

The The MeasuresMeasures

The weighted average of employer ratings on The weighted average of employer ratings on each of the three questions regarding overall each of the three questions regarding overall satisfaction reported on a 1 to 100 scale. satisfaction reported on a 1 to 100 scale.

Who:Who: Employers who receive a substantial Employers who receive a substantial service. service.

When:When: 1to 60 days after date of the 1to 60 days after date of the completion of a service. For employers who completion of a service. For employers who listed a job order where no referrals were made, listed a job order where no referrals were made, contact should occur 30 to 60 days after a job contact should occur 30 to 60 days after a job order was listed.order was listed.How:How: Telephone survey Telephone survey

Required completions:Required completions: 500 statewide 500 statewide

Required response rate:Required response rate: 50% first year; 70% 50% first year; 70% following yearsfollowing years

Capturing Customer Satisfaction Capturing Customer Satisfaction Results to Assess Local WIBsResults to Assess Local WIBs

States may use the ACSI questions and approach to States may use the ACSI questions and approach to measure satisfaction at the local levelmeasure satisfaction at the local level

States may also develop their own customer satisfaction States may also develop their own customer satisfaction instruments and approach for assessing local WIBsinstruments and approach for assessing local WIBs

Approaches must be applied Approaches must be applied uniformly and consistently to all uniformly and consistently to all WIBsWIBs

TEGL 6-00 outlines parameters TEGL 6-00 outlines parameters for collecting customer satisfaction results for collecting customer satisfaction results

Revisions to TEGL No. 6-00Revisions to TEGL No. 6-00

Response rates: 50% for the first year of data collection; Response rates: 50% for the first year of data collection; 70% for following years.70% for following years.

Response levels below the specified minimums Response levels below the specified minimums willwill invalidate the results.invalidate the results.

States are required to determine the appropriate sample States are required to determine the appropriate sample sizes. States are no longer required to draw minimum sizes. States are no longer required to draw minimum sample sizes of 1,000.sample sizes of 1,000.

Respondents Respondents mustmust be told responding to the survey is be told responding to the survey is voluntary and the information will be kept confidential. voluntary and the information will be kept confidential.

States States must must create a process for creating an up-to-date create a process for creating an up-to-date customer list, capturing information on each customer’s customer list, capturing information on each customer’s address and telephone number.address and telephone number.

Surveys to Obtain State Level Results…Surveys to Obtain State Level Results…

Disregard earlier guidance that stated the sampling frame only included participants and employers eligible for the surveys who had valid contact information.

A survey is complete when valid answers are provided by the respondents for each of the core ACSI questions. Valid answers are responses 1 through 10.

The calculation of the ACSI score includes only the results from complete surveys.

The introductory statements for both the participant survey and the employer survey were revised to better set the context for the interviews.

Revisions to TEGL No. 6-00Revisions to TEGL No. 6-00Surveys to Obtain State Level Results…Surveys to Obtain State Level Results…

Local program staff must inform the customer during the Local program staff must inform the customer during the registration process about the importance of satisfying registration process about the importance of satisfying customers and the possibility of being contacted for customers and the possibility of being contacted for information on his or her experience with the services. information on his or her experience with the services.

The interview should be limited to 15 minutes or less. The interview should be limited to 15 minutes or less. A minimum of five follow-up attempts is required, A minimum of five follow-up attempts is required,

involving various times of the day before closing the involving various times of the day before closing the record.record.

Revisions to TEGL No. 6-00Revisions to TEGL No. 6-00Surveys to Obtain State Level Results: Surveys to Obtain State Level Results: Additions to Ensure Adequate Response RatesAdditions to Ensure Adequate Response Rates

Local programs should collect alternate contact information.

A letter in advance of the survey should be sent out informing the customer that he or she can expect to be contacted about his or her satisfaction with the services.

Revisions to TEGL No. 6-00Revisions to TEGL No. 6-00Surveys to Obtain State Level Results: Surveys to Obtain State Level Results: Additions to Ensure Adequate Response RatesAdditions to Ensure Adequate Response Rates

Creating Questions to Creating Questions to Supplement the ACSISupplement the ACSI

Customer serviceCustomer service is measured by standards set is measured by standards set by you or your agency.by you or your agency.

Customer satisfactionCustomer satisfaction is measured is measured by the customer’s standards for by the customer’s standards for the services, regardless of whether the services, regardless of whether they make sense. they make sense.

Customer Satisfaction Not the Same as Customer Satisfaction Not the Same as Customer ServiceCustomer Service

Drivers of SatisfactionDrivers of Satisfaction

• ConvenienceConvenience• AccessibilityAccessibility• Ease of useEase of use• TimelinessTimeliness• SafetySafety• ReliabilityReliability• AccuracyAccuracy• ThoroughnessThoroughness• FairnessFairness• AppropriatenessAppropriateness• Attractiveness and cleanlinessAttractiveness and cleanliness

• CourteousCourteous• ProfessionalProfessional• AttentiveAttentive• FriendlyFriendly• HelpfulHelpful• KnowledgeableKnowledgeable• PromptPrompt• InformativeInformative• HonestHonest• CandidCandid

Service Quality Service Quality MeasuresMeasures

Customer Service Customer Service CharacteristicCharacteristic

Characteristics of Firms With Characteristics of Firms With High Customer SatisfactionHigh Customer Satisfaction

Customers define qualityCustomers define quality Variety of servicesVariety of services Customization of servicesCustomization of services Convenience of servicesConvenience of services Timeliness of servicesTimeliness of services Continually identify factors that influence satisfactionContinually identify factors that influence satisfaction Continually identify expectations and set customer Continually identify expectations and set customer

service standards well above these expectations (to service standards well above these expectations (to “delight” customers)“delight” customers)

Writing Effective QuestionsWriting Effective Questions Remember your survey's purpose: to provide Remember your survey's purpose: to provide

you feedback on the level of satisfaction.you feedback on the level of satisfaction. If you have doubts about a question, discard it.If you have doubts about a question, discard it. State your questions simply.State your questions simply. Stay focused when writing questions; avoid vague areas.Stay focused when writing questions; avoid vague areas. If a question can be misinterpreted by the respondent, chances are If a question can be misinterpreted by the respondent, chances are

that it will be.that it will be. Include only one subject per question.Include only one subject per question. Avoid questions that lead respondents. Avoid questions that lead respondents. Consider optional ways to ask questions that deal with sensitive Consider optional ways to ask questions that deal with sensitive

areas.areas.

Writing Effective QuestionsWriting Effective Questions Create questionnaire items to determine the key drivers Create questionnaire items to determine the key drivers

of satisfaction…of satisfaction…

Constructing Effective Constructing Effective Response ItemsResponse Items

Answer options need to be mutually Answer options need to be mutually exclusive and exhaustive.exclusive and exhaustive.

Keep open‑ended questions to a Keep open‑ended questions to a minimum.minimum.

Respondents interpret time-oriented Respondents interpret time-oriented responses differently, such as “never,” responses differently, such as “never,” “sometimes,” and “always.”“sometimes,” and “always.”

Consider a "don't know" responses when you create answer Consider a "don't know" responses when you create answer sets.sets.

Create meaningful scales for rating services.Create meaningful scales for rating services.

The first series of questions should be easy for the The first series of questions should be easy for the respondent to understand and should capture his or her respondent to understand and should capture his or her attention and interest. attention and interest.

Start with general questions and gradually shift to more Start with general questions and gradually shift to more specific questions. specific questions.

Group questions in logical sections and Group questions in logical sections and sequence sections or questions in a well sequence sections or questions in a well thought out order.thought out order.

Introduce each new section with a Introduce each new section with a summary statement so that participants summary statement so that participants have an opportunity to adjust to the have an opportunity to adjust to the new set of questionsnew set of questions..

Ordering QuestionsOrdering Questions

Ordering QuestionsOrdering Questions Position questions about personal or sensitive issues toward the Position questions about personal or sensitive issues toward the

end of the questionnaire.end of the questionnaire. The order of similar items on a list can bias results. Randomly The order of similar items on a list can bias results. Randomly

or alphabetically order items and indicate in the instructions how or alphabetically order items and indicate in the instructions how they are ordered, reducing the likelihood that respondents will they are ordered, reducing the likelihood that respondents will see the first items as most important of the group. see the first items as most important of the group.

Put demographic questions at the end of the survey form. Put demographic questions at the end of the survey form. Seek to minimize the number of times the a “skip rule” is Seek to minimize the number of times the a “skip rule” is

followed as confusion can result with too many skips or when followed as confusion can result with too many skips or when skips become overly complicated.skips become overly complicated.

ResourcesResources Fowler, F. J., Jr. Fowler, F. J., Jr. Survey Research Methods (2nd ed.)Survey Research Methods (2nd ed.). Thousand Oaks, CA: Sage . Thousand Oaks, CA: Sage

Publications, Inc., 1993. Publications, Inc., 1993. Hayes, Bob E. Hayes, Bob E. Measuring Customer Satisfaction: Development and Use of Measuring Customer Satisfaction: Development and Use of

QuestionnairesQuestionnaires. Milwaukee: ASQC Quality Press,1993. . Milwaukee: ASQC Quality Press,1993. Payne, S. L. Payne, S. L. The Art of Asking QuestionsThe Art of Asking Questions. Princeton, NJ: Princeton University . Princeton, NJ: Princeton University

Press, 1951.Press, 1951. Schuman, H and Presser, S. Schuman, H and Presser, S. Questions and Answers in Attitude Surveys: Questions and Answers in Attitude Surveys:

Experiments in Question Form, Wording, and ContextExperiments in Question Form, Wording, and Context. New York: Academic . New York: Academic Press,1981. Press,1981.

Simply Better! Simply Better! The Voice of the Customer, Second EditionThe Voice of the Customer, Second Edition. Washington: U.S. . Washington: U.S. Department of Labor, Employment and Training Administration, 1998.Department of Labor, Employment and Training Administration, 1998.

Sudman, S. and Bradburn, N. Sudman, S. and Bradburn, N. Asking Questions: A Practical Guide to Asking Questions: A Practical Guide to Questionnaire DesignQuestionnaire Design. San Francisco, CA: Jossey‑Bass Inc., 1982 . San Francisco, CA: Jossey‑Bass Inc., 1982

Vavra, T. G. Vavra, T. G. Improving Your Measurement of Customer SatisfactionImproving Your Measurement of Customer Satisfaction. Milwaukee: . Milwaukee: ASQC Quality Press, 1997.ASQC Quality Press, 1997.