improvement coach professional development...

Improvement Coach Professional Development ProgramVirtual Workshop 2

February 22, 201712-3 PM Eastern

Agenda2

Time Item Who

12:00 PM Welcome and warm up Linson

12:10 PM Tools spotlight: Tree diagrams Phyllis

12:35 PM Transition to WebEx breakouts All

12:40 PM Breakouts: WIP presentations Bill, Phyllis,

Christina, Karen

facilitate

1:40 PM Break and transition back to main room All

1:55 PM Tools spotlight: Scatter plots and surveys Phyllis, Karen

2:50 PM Next steps and evaluation Linson

3:00 PM Adjourn All

WebEx quick reference

• Please use chat to “All Participants” for questions

• For technology issues only, please chat to “Host”

• WebEx Technical Support: 866-569-3239

• Dial-in Info: Communicate, Join Teleconference (in menu)

3

Raise your hand

Select Chat recipient

Enter Text

You should see a small phone icon or a

headset icon next to your name.

WebEx connection

When chatting…

Please send your message to

All Participants

5

Please mute your line

• Please keep your line

on mute when you are

speaking to minimize

background noise

• You can mute or

unmute using *6 on

your phone

6

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=Ng_N8VBr3t90SM&tbnid=dqXADtowQ5MuCM:&ved=0CAUQjRw&url=http://theexpiredmeter.com/2009/11/another-dude-thinks-chicagos-parking-meter-lease-was-a-good-deal/shhhh/&ei=2dzfUq69DdHKsQSNuYDQCw&bvm=bv.59568121,d.eW0&psig=AFQjCNGTKkn4GvZGDGLG9J5FkTo26I4FNA&ust=1390489143664422

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=Ng_N8VBr3t90SM&tbnid=dqXADtowQ5MuCM:&ved=0CAUQjRw&url=http://theexpiredmeter.com/2009/11/another-dude-thinks-chicagos-parking-meter-lease-was-a-good-deal/shhhh/&ei=2dzfUq69DdHKsQSNuYDQCw&bvm=bv.59568121,d.eW0&psig=AFQjCNGTKkn4GvZGDGLG9J5FkTo26I4FNA&ust=1390489143664422

Who’s on the line with us today…

Phyllis Virgil Bill Peters Karen Baldoza

Christina

Gunther-Murphy

Linson Naval

Warm-up chat

• In the chat box, please chat to All

Participants...

Your favorite Spring activity!

8

Tools Spotlight: Tree DiagramsPhyllis

What is a tree diagram?

Tool can be used to organize and understand the structure and content of any problem, issue or objective

Diagram resembles a tree when completed

Helps teams think systematically, logically and comprehensively

Each branch is exploded and explained

Allows an easy to see graphic view of increasing detail

Moves thinking step by step from the general to the specific

Can be developed horizontally or vertically

Also called: systematic diagram, tree analysis, analytical tree, or hierarchy diagram

10

Adapted from: The Improvement Handbook: Model, Methods, and Tools for Improvement, API by Phyllis M. Virgil

1. Problem Solving Tree (Why)

2. Planning Tree (How)

3. Analysis Tree (What)

4. Mixed

Types of Tree Diagrams11

Phyllis M. Virgil as adapted From: http://asq.org/learn-about-quality/new-management-planning-tools/overview/tree-diagram.html

12Readmissions Problem Solving (Why)

Source: Phyllis M. Virgil

Strategic Planning for Healthy Weight (How)


Work Breakdown Structure Tree (How)

Source: http://www.criticaltools.com/wbschartprosoftware.htm

Used Extensively by Project Management Professionals

What does that mean?




Customer Requirement (VOC)

Good Cup of Coffee

Hot

Temp between 91° and 96°

Tasteful

Flavorful

Perfect Brew

Strength

Golden Ratio 17.42 units of

water to 1 unit of coffee

Not Bitter

PH of 5

Quality Characteristic Analysis Tree (What)

Source: Phyllis M. Virgil, Strategic Quality Planning Workshop

Six Sigma’s DMAIC

Critical To Quality Tree (What)

Source: http://isoconsultantpune.com/voc-data-collecting-tools/

http://i0.wp.com/isoconsultantpune.com/wp-content/uploads/2015/05/CTQ-tree.png

http://i0.wp.com/isoconsultantpune.com/wp-content/uploads/2015/05/CTQ-tree.png

Driver Diagram is a Tree that Maps Out Your Theory (Mixed)

Source http://www.ihi.org/resources/Pages/Publications/WhatsYourTheoryDriverDiagrams.aspx

http://www.ihi.org/resources/Pages/Publications/WhatsYourTheoryDriverDiagrams.aspx

Whiteboard Video: Intro to Driver Diagrams18

http://www.ihi.org/education/IHIOpenSchool/resources/Pages/AudioandVideo/Whiteboard9.aspx

http://www.ihi.org/education/IHIOpenSchool/resources/Pages/AudioandVideo/Whiteboard9.aspx

New Driver Diagram Paper

http://www.ihi.org/resources/Pages/Publications/WhatsYourT

heoryDriverDiagrams.aspx

http://www.ihi.org/resources/Pages/Publications/WhatsYourTheoryDriverDiagrams.aspx

Benefits of Tree Diagrams

• Makes it less likely essential items will be left out

• Encourages discussion and dialogue

• Improves communication and presentations

• Versatile and easy to use

20

Adapted from: The Improvement Handbook: Model, Methods, and Tools for Improvement

Use of Tree Diagrams

A tree diagram can be used in all phases of

improvement work:

Break project down into sub-objectives

Uncover the causes of problems

Understand customers needs

Develop our measure of change

Identify change ideas

Detail out action plans

Adapted by Phyllis M. Virgil from: The Improvement Handbook: Model, Methods, and Tools for Improvement

What are we trying toaccomplish?

How will we know that achange is an improvement?

What change can we make thatwill result in improvement?

Model for Improvement

PlanAct

DoStudy

API

How to Develop a Tree Diagram

1. Clearly state the problem, opportunity, objective or issue of interest. Place

this as the first box at the far right/left or top of your work space (page, flip

chart, white board)

2. Ask questions of Why, What or How to develop the next level of detail.

Place each answer in a box (or on a line) at next level and link them by

drawing a line.

Problem Solving -- Why

Planning -- How

Analysis -- What

3. Repeat step 2 process until a useful level of detail is achieved.

Adapted by Phyllis M. Virgil from Minnesota Department of Health QI Tool Box http://www.health.state.mn.us/divs/opi/qi/toolbox/print/tree.pdf

Questions to Ask

– Planning Tree: “How can this be accomplished?” or

“What Tasks must be done to accomplish this?”

– Problem Solving Tree: “Why does this happen?” or

“What causes this?”

– Analysis Tree: “What are the components?” “What

goes into it?” “What drives this?” “What comes next?”

– For unclassified tree diagrams ask whatever

question is appropriate, What, Why, How...

Adapted by Phyllis M. Virgil from: http://asq.org/learn-about-quality/new-management-planning-tools/overview/tree-diagram.html

23

Tips for Constructing Tree Diagrams

Sticky notes are often useful to use when developing tree diagrams.

Best when done by a small team who has detailed knowledge

Develop by brainstorming, clumping, labeling and sequencing ideas

Another way is to logically and sequentially move from level to level

Measures, dates and responsibility can also be added diagrams

Typically taken to third level of detail

Use nested diagrams for more detail

Do a “necessary, valid and sufficient” check

Adapted from ASQ and Minnesota Department of Health

Use SmartArt to Create in MS Word or PowerPoint

• Go to Insert Tab > SmartArt > Hierarchy

• Use Design and Format tabs to create and format.

Content Design: Phyllis M. Virgil

25

Summary

Tree Diagrams are versatile and easy to use.

They help you to think systematically, logically and

comprehensively about any problem, issue or objective

They can be used in any phase of improvement work.


26

Appendix: Six More Examples

1. CRX Problem Solving Tree (Problem Solving)

2. Healthy Weight Action Plan Tree (Planning)

3. Customer Service CTQ Tree (Analysis)

4. Decreasing Fuel Costs (Driver Diagram)

5. Behavior Health Primary Care Integration (Driver Diagram)

6. Grocery Store Customer Satisfaction (Driver Diagram)

28

1991 CRX will not start

Dead Battery

Fluid Levels Low

did not fill Did not check

Would not charge

Generator not working

Solenoid broken

Left Lights onUsed lights during day

Did not drive in 3 months

Too busy to drive it

No Gas

Fuel Line broken

Rubber hose worn

Rubber hose old

Gas filter clogged

Not changed in 3 years

Not on maintenance

schedule

Gas tank empty

Fuel gage broken

Forgot to reset

odometer

Starter malfunction

Wires not connected

Vibration caused

disconnect

Parts WornFriction and

Age

Engine Malfunction

CarburetorChoke

Malfunction

Distributor Timing Off

Problem

Solving

Tree (Why)

Adapted from: The Improvement Handbook: Model, Methods, and Tools for Improvement, API

HW Action Plan Tree (How) nested diagramSee HW Strategy Tree


DMAIC Critical to Quality Tree (What)

Source: http://isoconsultantpune.com/voc-data-collecting-tools/

Basic

Driver

Diagram

Source: Quality Institute, NHS, UK

Complex

Collaborative

Driver Diagram

Source: CCRMC BH/PC Collaborative, 2015

Change Idea Tree (aka Driver Diagram)

WIP PresentationsKaren, Linson

WIP presentations: Purpose

The purpose of the Works in Progress (WIP) sharing is to provide an environment that enables Improvement Coaches to:

• Lead effective two-way communication about coaching and improvement efforts

• Elicit useful feedback to accelerate personal skill building and project gains, and in turn, provide coaching to fellow Improvement Coaches

• Stimulate peer-to-peer learning and creative ideas about coaching and improvement among Improvement Coaches and faculty.

35

WIP presentations: Format

• For each presentation, the presenter-coach will have

12 minutes as follows:

– 7 minute uninterrupted presentation

– 5 minute discussion that the presenter-coach facilitates

• All should speak up or use chat to provide feedback,

ask questions, etc.

• We will capture the chat so presenter-coaches have

the notes and can concentrate on the discussion

• Faculty will keep time and send time checks via the

chat

36

WIP presentations: Timing

Time (Eastern) Presentation

12:35 PM-12:40 PM Transition and get started

12:40 PM-12:55 PM Presentation 1




1:40 PM-1:55 PM Break and transition back to the main room

37

WebEx breakouts

• It is imperative that everyone be connected correctly to

fully participate in WebEx Breakouts for the Project

Presentations

• When you join WebEx, wait until the Pop Up window

comes up and gives you the number to dial, session

number, and attendee ID#.

• You need to dial all of the numbers to connect

completely.

You should see a small phone icon or a

headset icon next to your name.

Today we will use WebEx Breakouts for the

Project Presentations

• When we are ready to start the

Breakouts, you will see a pop-up

window that invites you to join.

Please click YES when it pops up.

• You will then see a pop-up

window that tells you that your

audio line will automatically be

transferred to the Breakout. Click

YES again.

Breakouts

• Once you are in your virtual breakout room, Karen, Bill, Christina, or Phyllis will help facilitate the project presentations. Linson will load the slides for each breakout room.

• You will be able to speak, raise your virtual WebEx hand, and use the chat box while you are in the breakout room.

• Linson will send message to provide a 5-minute warning when we are nearing the end of our breakout sessions.

• Five minutes later, Linson will send a message that requests that you return to the main session.

• Do not click refresh or back- sometimes it takes a few seconds for the breakouts to begin and end- please be patient!

Transition to Project Presentation Breakouts

15 minute break

Tools Spotlight: Scatter Plots

A tool to understand relationshipsPhyllis Virgil

47

What is a Scatter Plot?

A scatter plot is a graphic representation of the association between

two variables (X and Y)

It is a dot plot of paired data

Which helps to test the direction and strength of possible

relationships

Also known as Scatter Diagram, or X-Y Graph

Content Design: Phy llis M. Virgil

48

When do you use a Scatter Plot?

Use to test hunches and theories about relationships

In improvement work they can help us

1. Select and focus improvement projects

2. Verify relationship of process and

outcome measures

3. Target areas for change

Content Design Phy llis M. Virgil

49

What are we trying toaccomplish?

How will we know that achange is an improvement?

What change can we make thatwill result in improvement?

Model for Improvement

PlanAct

DoStudy

API

Independent Variable X Y Dependent Variable

Positive relationship > upward slope > as X increases Y increase

Negative relationship > downward slope > as X increases Y decreases

No relationship > no slope up or down

X = Ave. Temp.

Y=

Ele

ctr

ic B

ill

Positive Relationship

X = Weight of Car

Y=

Mile

s p

er

Gallo

n

NegativeRelationship

X = Shoe SizeY

= IQ

Te

st S

co

re

No Relationship

Direction of the Relationship

Positive or Negative

upward slope > as


Images courtesy of http://sixsigmatraining.com

50

X Y

Strong relationship > tight clustering of dots

Mild or Weak relationship > loose clustering of dots

X = Miles to Destination

Y=

Air

Fare

Mild Relationship

X = Waiting Time

Y=

Cu

sto

me

r

Sati

sfa

cti

on

Strong Relationship

Strength of the Relationship

Strong or Weak

51


Images courtesy of http://sixsigmatraining.com

http://sixsigmatraining.com/

http://sixsigmatraining.com/

More Complex Non Linear Scatter Plots

Stopping Distance by Speed

Fuel Used by SpeedReading Score by

Hours of Sleep

Examples of Curvilinear & Quadratic (Parabola) Scatter Plots

Stopping Distance by Speed : http://athometuition.com/modelling-bivariate.aspxReading Scores and Sleep: https://www.aea267.k12.ia.us/assessment/statistics/descriptive-statistics/correlation/curvilinear-relationship/ Anxiety and Performance : https://webcourses.ucf.edu/courses/950845/pages/the-correlational-research-strategyFuel Used by Speed: http://bolt.mph.ufl.edu/6050-6052/unit-1/case-q-q/linear-relationships/

How to Construct a Scatter Plot

Gather paired data from the same observation unit (time, person, place, thing)

By hand = scale and label x and y axis, plot dot of intersection for each paired valued, circle duplicates

By Excel = Highlight data > Insert Tab > Select Scatter Plot

X Y

Date# Patients

Late

Ave. Waiting

Time

4-Apr 3 15

5-Apr 4 17

6-Apr 5 20

7-Apr 6 29

8-Apr 6 32

11-Apr 6 36

12-Apr 4 25

13-Apr 3 10

14-Apr 2 7

15-Apr 5 30

18-Apr 5 30

0

5

10

15

20

25

30

35

40

0 1 2 3 4 5 6 7

Ave. Waiting Time

# Patients Late

Ave. Waiting Time by # Patients Late

53


Trend lines can be added...

Eyeball Method = draw line in middle of dots (half above, half below)

Excel = Grab Graph > Layout Tab > Select Trendline.

0

5

10

15

20

25

30

35

40

2 3 4 5 6 7

Ave Waiting Time

# Patients Late

Ave Waiting Time by # Patients Late

54


Select and Focus Improvement Projects

1.0

2.0

3.0

4.0

5.0

6.0

7.0

0 20 40 60

Ave. Pat Satisfaction

Ave. Weekly Wait Time

Patient Satisfaction by Wait Time

Finding: Fairly Strong negative relationship between wait time and patient satisfaction.

Conclusion: Working on decreasing wait time makes sense.

55


Use after C/E Brainstorming

To Confirm Change Ideas Before Testing

Finding: Fairly Strong positive relationship between wait time and # of patient late.

Conclusion: Testing change ideas aimed reducing # late patients makes sense.

0

5

10

15

20

25

30

35

40

2 3 4 5 6 7

Ave Waiting

Time

# Patients Late

Ave Waiting Time by # Patients Late

C/E Image Credit: https://www.nursingtimes.net/Journals/2013/04/12/k/x/z/Using-fishbone-analysis--to-investigate-problems-160413.pdf

Content Source: Phyllis M. Virgil

https://www.nursingtimes.net/Journals/2013/04/12/k/x/z/Using-fishbone-analysis--to-investigate-problems-160413.pdf

Beware of Spurious (false) Scatter Plots57

Spurious scatter plot is often a result of a third factor that is not apparent at the

time of examination. Spurious comes from the Latin word spurious, which

means illegitimate or false.

Killer Ice Cream?


Scatter Plot Source: http://tellingthestory.typepad.com/telling_the_story/2010/06/

Study shows killers love ice cream!

Conclusion: Think before you believe.

http://tellingthestory.typepad.com/.a/6a010536ce7016970b0133f1910952970b-pi

http://tellingthestory.typepad.com/.a/6a010536ce7016970b0133f1910952970b-pi

http://tellingthestory.typepad.com/telling_the_story/2010/06/

http://tellingthestory.typepad.com/telling_the_story/2010/06/

Scatter Plots are about relationships!

Scatter plots do not prove anything!

Rather they provide “evidence”

They help you:

Understand relationships

Understand the direction and strength of the relationships

Scatter Plots in Action

59

Physical Function by Steps per Day

Content Design and Conclusion: Phyllis M. Virgil

Content and Image Source http://www.rehab.research.va.gov/jour/04/41/4/smith.html

Shows No Relationship

Conclusion: Think twice about focusing on increasing steps per day

as a means to improve physical function.

60

http://www.rehab.research.va.gov/jour/04/41/4/smith.html

Service Quality Rating and Patient Safety

Content Design and Conclusion by Phyllis M. Virgil

Content and Image Source by http://www.psqh.com/enews/1107feature.html

Shows a Very Moderate Positive Relationship

Conclusion: Patient Safety is not a driver of satisfaction,

but likely a Prerequisite. More study is required.

61

Patient Satisfaction by Nurse Contact Time

62


Content and Image Source http://www.qihub.scot.nhs.uk/knowledge-centre/quality-improvement-tools/scatter-plot.aspx

Shows a Moderate Positive Relationship

Conclusion: Looks like 10-20 minutes is a sweet spot.

Do more investigation.

Sick Days to Case Load

Scatter sick days to case load

Scattergram

30 35 40 45 50 55 60 65

Case Load (X)

0

2

4

6

8

10

12

14

16

Days S

ick (

Y)

Scatter sick days to case load

Scattergram

30 35 40 45 50 55 60 65

Case Load (X)

0

2

4

6

8

10

12

14

16

Days S

ick (

Y)


Shows Moderate, Curvilinear

Positive Relationship

63

Source: The Health Care Data Guide: Learning from Data for Improv ement By Lloyd P. Provost, Sandra Murray


by Department

64

Case Load Vs Sick Days Total

30 35 40 45 50 55 60 65Case Load (X)

0

2

4

6

8

10

12

14

16

Days S

ick (

Y)

Case Load Vs Sick Days Dept A

40 45 50 55 60 65Case Load (X)

0

2

4

6

8

10

12

14

16

Days S

ick (

Y)

Case Load Vs Sick Days Dept B

30 35 40 45 50 55 60 65Case Load (X)

0

2

4

6

8

10

12

14

16

Days S

ick (

Y)

Case Load Vs Sick Days Dept C

35 40 45 50 55 60Case Load (X)

0

2

4

6

8

10

12

14

16

Days S

ick (

Y)

Dept. A

Shows a Positive, Moderate

Curvilinear Relationship

Dept. C


Dept. B


Conclusion, focus improvement effort only on Department A

and do more investigation

Total

Shows a Positive, Moderate

Curvilinear Relationship

Source: The Health Care Data Guide: Learning from Data for Improv ement By Lloyd P. Provost, Sandra Murray

Nonsense (or spurious) Scatter Plots65

450

500

550

600

650

700

750

9 9.5 10 10.5 11 11.5

Civil Engineering

Degrees Awarded

Mozzarella Cheeze Consumption Per Capata in Pounds

9.2

9.4

9.6

9.8

10

10.2

10.4

10.6

10.8

11

11.2

450 550 650 750

US Per Capita

Mozzarella Cheeze

Consumption in Pounds

Civil Engineering Doctorate Degrees US


Content and Image Source by http://tylervigen.com/old-version.html

Conclusion: just because any two variables can be

put on a scatter plot, does not mean it makes sense.

Use rational judgment when discerning associations.

Mozzarella Cheese Consumption and Civil Engineering Degrees

Show a Strong Positive Relationship.

Correlation Coefficient and Regression Lines

More advanced applications of the Scatter Plot include the use of correlation coefficients (R) and regression lines

R Indicates how closely observed values fall around regression line – both the clustering about line & direction of association.

R standardizes the degree of association, regardless of units of measurement

Ranges between -1 and 1

Direction of the relationship indicated by a + or -

positive r = positive relationship

negative r = negative relationship

0 = no relationship

General guide for interpreting strength of the relationship R (absolute value)

0 - .2 = weak, slight

.2 - .4 = mild

.4 - .6 = moderate

.6 - .8 = strong

.8 - 1.0= very strong

In Excel select graph > Layout Tab > Trendline > More Options > Display R squared value on chart


66

Review of Scatter Plot Patterns67

No Relationship

Scattered

No Relationship

Horizontal Cluster

Quadratic

Relationship

Curvilinear

Relationship

Strong Positive

Relationship

Strong Negative

Relationship

Weak Positive

relationship

Weak Negative

relationship

Images from: http://slideplayer.com/slide/4682680/, as published by Justin Jacobs

http://slideplayer.com/slide/4682680/

Summary

Scatter plots test association between two variables

They tell us if there is “evidence” of a relationship

Direction: is it positive, negative, curvilinear, or quadratic?

Strength: is it strong or weak (tight or loose clustering of dots)?

Can use them in first three questions MFI

Focus aim

Select measures

Target areas for change


68

Tools spotlight: SurveysTools and methods to gather informationKaren


Reference: The Improvement Guide, page 417

Session objective

• Summarize how surveys can be used to gather

information and for data collection to enhance

your improvement effort

Session agenda

Topic Time

Overview 5 minutes

Written surveys 5 minutes

Developing and using surveys, analyzing the results 5 minutes

Kano method 5 minutes

Chat in

• Think about the last survey you participated in.

• Chat in your answers to the following:

– What was it?

– What did you like about the survey?

– What didn’t you like about it?

72

Surveys (as we’ll define it here)

• A method of collecting information directly from people about their feelings, motivations, plans, beliefs, experiences, and backgrounds

• In relation to the Model for Improvement and PDSA cycles, a survey is a type of data collection process to answer a question(s) posed in Question 2 and the planning phase of the cycle


73

Reasons to survey

• To build the knowledge of those obtaining the

information (duh!)

• To obtain a desired effect on the relationship

between those obtaining the information and

those providing the information


74

Methods for obtaining information

• Informal conversation

• Written surveys

• Personal interviews

• Group interviews

• Observations

• Trading places


75

Considerations:

• Purpose

• Reliability and validity

• Usefulness and credibility of results

• Cost

• Anonymity

• Convenience

• Complexity of information

• Time – to administer and respond

• What you will do with the information

Written survey example76

Thank you for participating in our workshop today. Please help us improve our service to you by completing this survey.

Please list three ways we can improve our service to you.

1. _____________________________________

2. _____________________________________

3. _____________________________________

Please rate your experience with our services today (circle one)

Poor Fair Good Very Good Excellent

Other suggestions:

_______________________________________________________________

_______________________________________________________________

Written surveys

• Delivery: intentional or opportunistic?

• Design considerations:

– Order of the questions

– Selection of response options

– Wording


77

Written surveys

Advantages Disadvantages

Beneficial for those who are better at

written than verbal communication

Respondent has no say about what is

covered

Can be administered to preserve

confidentiality

Usually a large percent do not

complete

Less expensive than interviews Questioner cannot build on answers

Useful in reaching a geographically

diverse group of peopleRespondents have to interpret the

questions

Respondent assumes most of the

work


78

Steps for conducting a survey

1. Clarify the purpose of the

survey

2. Consider why a survey is the

best method for obtaining

the desired information

3. What questions are to be

answered by the survey?

4. Decide on the type of survey

(e.g., written, phone,

interview)

5. Select the survey design,

content, questions, and test

the form


79

6. Decide who should

participate and when (will

sampling be used?)

7. Will there be follow-up

surveys?

8. Attend to the survey

administer (e.g., collecting

forms, conducting

interviews) (Do!)

9. Analyze, interpret, and

report results (Study!)

10. Act (if appropriate) on the

results

Survey questions

• Consider the respondents’ point of view – how complicated should the questions be and at what level should they be written?

• Open-ended– Offers insight as to why people believe what they do

– Interpretation can be difficult

• Closed-ended– Easy to use, score, and code

– Reliable to the extent they provide uniform data

– Validity usually requires extensive testing


80

Closed-ended questions

• Each question should be meaningful to the respondents

• Use standard, simple language

• Make questions concrete

• Avoid biased words and phrases

• Check your own biases

• Don’t get too personal

• Each question should have just one thought (no double-barreled questions)

• Make sure the response options match the question

• Tip: State questions/statements in the extreme


81

82Written survey examples

Learning in real time

Hayes, H., Scott, V., Scaccia, J.P., Abraszinskas, M., Stout, S., & Wandersman, A. (under review). A Multi-Method Approach to Evaluating Training

83

Methods for analyzing survey data

• Depends on the type of questions asked

• Closed-ended questions:– Frequency plots (possibly

with stratification)

– If collected over time, run and control charts

• Open-ended questions:– Clustering techniques with

written summaries of the clusters/themes – e.g., affinity diagram or Pareto chart


84

0

1

2

3

4

5

6

7

8

9

Poor Fair Good Very good Excellent

Nu

mb

er o

f re

spo

nse

s

Response

Customer Satisfaction

Kano survey method

• Useful for testing new ideas and innovations with

customers to determine if the ideas might

change their expectations (future needs and

wants)

• Incorporates the concept of “attractive” and

“must-be” quality elements


85

Kano two-dimensional recognition

mode

86

Customer

Satisfied

Attractive One-dimensional

Not Fulfilled Fulfilled

Must-be

Customer

Dissatisfied


Three classifications of quality-characteristics of a produce or service:• One-dimensional Element – gives satisfaction when fulfilled and results in dissatisfaction when

not fulfilled• Attractive Quality Element – give satisfaction when fulfilled but is acceptable when not fulfilled• Must-be Quality Element – take for granted when fulfilled but results in dissatisfaction when

not fulfilled

Method

• Ask questions in pairs

– How would you feel if you had coffee at the

workshop?

– Delighted, I expect it and like it, No feeling, Live with it, Do

not like, Other

– How do you feel when you do not have coffee at the

workshop?

– Delighted, I expect it and like it, No feeling, Live with it, Do

not like, Other


87

Two-dimensional chart of survey results for each

quality element

88

Response to “Negative Wording” of Question

Re

spo

nse

to “

Po

sitiv

e

Wo

rdin

g” o

f Q

ue

stio

n

Delighted I expect and like it

No feeling

Live with it

Do not like

Other

Delighted Skeptical Attractive Attractive Attractive One-

dimensional

Other

I expect and like it

Reverse Indifferent Indifferent Indifferent Must-be Other

No feeling Reverse Indifferent Indifferent Indifferent Must-be Other

Live with it Reverse Indifferent Indifferent Indifferent Must-be Other

Do not like Reverse Reverse Reverse Reverse Skeptical Other

Other Other Other Other Other Other Other

• One-dimensional: Satisfied when fulfilled, dissatisfied when not fulfilled

• Attractive: Satisfied when fulfilled, no feeling when not fulfilled

• Must-be: No feeling when fulfilled, dissatisfied when not fulfilled

• Indifferent: No feeling of satisfaction or dissatisfaction regardless of fulfillment

• Reverse: Not expected, dissatisfied when fulfilled or satisfied when not fulfilled

• Skeptical: Doubtful if question is understood; difficult question, poor expression

• Other: None of the responses apply; other consideration

89

Summary Survey Results Example: Trends in Quality Elements

Survey Form Example: Kano

Questionnaire at a Hotel

Next steps and evaluation

• Continue working with your team

• Our next Virtual Workshop is on:

March 2, 2017

12:00 PM – 3:00 PM Eastern

• Please give us your feedback! Chat in:– What you think “went well”

– Improvement suggestions

– Questions you still have that you would like us to address

We really appreciate your feedback and questions!

Thank you!

SurveysAppendix

92

Personal interviews

• Discussion between an interviewer and a respondent to obtain information on a pre-arranged set of topics

• Can be specific questions or set of topics

• Emphasis on a particular topic depends on value of the information being obtained and effect we hope to get


93

Personal interviews


Respondent can provide input into what will

be discussed

Difficult to administer to people in varied

locations

Interviewer can:

• Check for understanding and clarify

• Change which topics are emphasized

• Pursue fruitful unforeseen areas

• Observe body language

Variation is introduced by multiple

interviewers

Presents an opportunity to build or enhance

relationships

Lacks confidentiality

Less work for interview than a written

survey

Time-consuming for interviewers

Expensive compared to written surveys

Format is not standard


94

Group interviews or focus groups

• Like personal interviews

• Discussion between an interviewer and a group

of respondents to obtain information on a pre-

arranged set of topics


95

Group interviews


Mutual stimulation of the group can

result in more information than

individual interviews

Respondents may be reluctant to

discuss sensitive issues in a group

Fruitful unforeseen areas can be

pursued by the entire group

Interviewer must be more skilled than

for a personal interview

Can help to establish or enhance

relationships between members of the

group

Interactions between members of the

group may suppress or bias

information from one or more

members

Less time consuming than personal

interviews

Expensive if the group is

geographically diverse

May create other agendas for some

members of the group


96

Observation

• Observe users of your

product, process,

equipment, performing a

function

• Can be used as the basis for

innovation


97

Observation


Information obtained under realistic

conditions

The presence of an observer may

create artificial conditions

Observer needs less skill to obtain the

information than an interviewer

Limited to the conditions under which

the observations were made

Useful for generating ideas for

innovation

The use of many products and

services is difficult to observe

Simple May miss important but infrequently

occurring events

Useful for validation of personal or

group interviews

If not careful to be objective, observers

can sometimes see what they want to

see


98

Trading places

• Experience the product

or service for yourself


99

Trading places


Allows the person trading places to

experience feelings that could not be

expressed by respondents in an

interview

Perceptions are obtained from only

one or a few people who assume the

role

Very useful for building or enhancing a

relationship

Presence of the person assuming the

role may create artificial conditions

Important, but minute, level of detail

can be experienced

May need certain skills to assume

someone’s role

May be too short to form a useful

perception

Familiarity with the product or service

may make it impossible to assume the

position of a first time user


100

improvement coach professional development...

Documents