survey methods & applications in healthcare

Survey Methods & Applications in Healthcare

Nawanan Theera-Ampornpunt, M.D., Ph.D.Faculty of Medicine Ramathibodi Hospital

Mahidol Universitywww.SlideShare.net/Nawanan

2

2003 M.D. (First-Class Honors) (Ramathibodi)2009 M.S. in Health Informatics (U of MN)2011 Ph.D. in Health Informatics (U of MN)2012 Certified HL7 CDA Specialist

• Deputy Executive Director for Informatics (CIO/CMIO)Chakri Naruebodindra Medical Institute

• Lecturer, Department of Community MedicineFaculty of Medicine Ramathibodi HospitalMahidol University

[email protected]://groups.google.com/group/ThaiHealthITwww.SlideShare.net/Nawanan

Introduction

3

• Overview of Surveys• Survey Methodology• A Sample Survey

Outline

4

OVERVIEW OF SURVEYS

5

• An activity in which many people are asked a question or a series of questions in order to gather information about what most people do or think about something

Survey

Merriam Webster Dictionary

6

• A written set of questions that are given to people in order to collect facts or opinions about something

Questionnaire

Merriam Webster Dictionary

7

• To know something– Personal information– Knowledge, Opinions & Attitudes– Behaviors & Practice– etc.

• About someone– Individuals– Organizations

• In order to understand, create knowledge, or make decisions

Why Do a Survey?

8

Examples of Surveys

Say2KFC.com

Service Satisfaction Survey (Online)

9

Examples of Surveys

http://www.kmutt.ac.th/building/pdf/application_form_2552_1.pdf

Service Satisfaction Survey (Paper-based)

10

Examples of Surveys

Image Source: http://pixabay.com/en/agent-business-call-center-18741/

Service Satisfaction Survey (Telephone)

11

SURVEY METHODOLOGY

12

• Survey Design– Study Design– Modes of Data Collection

• Instrument Design• Sampling• Survey Conduct• Data Analysis• Reports

Survey Methodology

13

WHAT WOULD YOU DO TO INCREASE SURVEY

RESPONSES?

14

The Tailored Design Method

1978 1999 2007 2008

Dillman et al.

15

The Tailored Design Method

2014

Dillman et al.

16

• “The development of survey procedures that create respondent trust and perceptions of increased rewards and reduced costs for being a respondent, which take into account features of the survey situation and have as their goal the overall reduction of survey error”

The Tailored Design

Dillman et al. (2007)

17

• Rewards• Costs• Trust

Social Exchange Theory


Image Source: http://horsebusinessschool.com/using-stick-and-carrot-to-motivate-employees/

18

• “The likelihood of responding to the request to complete a self-administered questionnaire, and doing so accurately, is greater when the respondent trusts that the expected fix rewards of responding will outweigh the anticipated costs”

Social Exchange


19

• Providing Rewards– Show positive regard– Say thank you – Ask for advice– Support group values– Give tangible rewards– Make the questionnaire interesting– Give social validation– Inform respondents that opportunities to

respond are scarce

Survey & Social Exchange Theory


20

• Reducing Social Costs– Avoid subordinating language– Avoid embarrassment– Avoid inconvenience– Make questionnaires appear short & easy– Minimize requests to obtain personal

information– Keep requests similar to other requests to

which a person has already responded



21

• Establishing Trust– Provide a token of appreciation in advance– Sponsorship by legitimate authority– Make the task appear important– Invoke other exchange relationships



22

• Study Design– Cross-sectional– Longitudinal

Survey Design

23

• Mode of Data Collection– Self-administered survey

• Paper-based• Online• Telephone (Automated/IVR)

– Interviewer-administered survey (structured interview)

• In-person• Telephone

– Mixed-mode survey

Survey Design

24

• Choose simple over specialized words• Choose as few words as possible to pose the

question• Use complete sentences to ask questions• Avoid vague quantifiers when more precise

estimates can be obtained• Avoid specificity that exceeds respondent’s

potential for having an accurate, ready-made answer

• Use equal numbers of positive & negative categories

Instrument Design


25

Problem• Number of years lived in Idaho

Years• Your city or town

City or Town• Your county

County

Use complete sentences


26

Revision• How many years have you lived in Idaho?

Years• In what city or town do you live?

City or Town• In what Idaho county do you live?

Idaho County

Use complete sentences


27

Problem• How often did you attend religious services

during the past year? Never Rarely Occasionally Regularly

Avoid vague quantifiers


28

Revision• How often did you attend religious services

during the past year? Not at all A few times About once a month Two to three times a month About once a week More than once a week

Avoid vague quantifiers


29

Problem• About how many books have you read for

leisure during the past year?Number of books

Avoid too much specificity


30

Problem• About how many books have you read for

leisure during the past year? less that 10 11-25 26-50 51-75 76 or more

Avoid too much specificity


31

• Distinguish undecided from neutral by placement at the end of the scale

• Avoid bias from unequal comparisons• State both sides of attitude scales in the

question stems• Eliminate check-all-that-apply question formats

to reduce primacy effects• Develop response categories that are mutually

exclusive

Instrument Design


32

Problem• Which one of the following do you feel is most

responsible for recent outbreaks of violence in American’s schools? Irresponsible parents School policies Television programs

Avoid bias from unequal comparisons


33

Revision• Which one of the following do you feel is most

responsible for recent outbreaks of violence in American’s schools? The way children are raised by parents School policies Television programs

Avoid bias from unequal comparisons


34

Problem• From which one of these sources did you first

learn about the tornado in Derby? Radio Television Someone at work While at home While traveling to work

Mutually Exclusive


35

• Use cognitive design techniques to improve recall

• Provide appropriate time referents• Be sure each question is technically accurate• Choose question wordings that allow essential

comparisons to be made with previously collected data

• Avoid asking respondents to say yes in order to mean no

• Avoid double-barreled questions

Instrument Design


36

Problem• Should the city build a new swimming pool that

includes lanes for swimming laps that is not enclosed for winter use? Yes No

Double Barreled


37

• Soften the impact of potentially objectionable questions

• Avoid asking respondents to make unnecessary calculations

Instrument Design


38

• Choose simple over specialized words• Choose as few words as possible to pose the

question• Use complete sentences to ask questions• Avoid vague quantifiers when more precise

estimates can be obtained

Instrument Design


39



Survey Methodology

40

• Target Population• Sample• Inclusion & Exclusion Criteria• Sampling Frame• Sampling Method/Technique

Sampling Design

41

• Census• Random (probability) sampling

– Simple random sampling– Systematic random sampling– Stratified random sampling– Cluster random sampling– Multi-stage random sampling

• Non-probability sampling– Purposive sampling– Convenience sampling– Snowball sampling– etc.

Sampling Design

42

Example sequence• Pre-notice Letter• 1st Questionnaire Mailout• Reminder + 2nd Questionnaire Mailout

Survey Conduct

Simplified from Dillman et al. (2007)

43

• Sampling error• Coverage error• Measurement error• Nonresponse error

– Questionnaire effects– Data collection mode effects– Interviewer effects– Respondent effects

• Processing error

Errors in Surveys

Dillman et al. (2007) & Office of Management and Budget (2001)

44

• Sampling error• Coverage error• Measurement error• Nonresponse error• Processing error

– Data entry error– Pre-edit coding errors– Editing errors– Imputation errors

Errors in Surveys

Dillman et al. (2007) & Office of Management and Budget (2001)

45

A SAMPLE SURVEY

46

NAWANAN’S DOCTORAL STUDY

47

Unknown State of IT Adoption in Thai Hospitals

Need to improve theoretical knowledge

This Study

Dual Opportunities

48

• To describe current state of IT adoption in Thai hospitals nationwide

• To test proposed conceptual framework & explore relationships between organizational characteristics, IT management, and IT adoption

Study Objectives

49

Hypothesized Model

50

• Study Design: Nationwide cross-sectional mail survey• Sample: All hospitals in Thailand except pilot (N =

1,302)• Pilot: 5 hospitals (10 respondents each)• Sampling Frame: List of hospitals from Ministry of

Public Health’s Web site• Subjects: Hospital’s staff responsible for managing

information systems (CIO/IT manager or equivalent; hospital director if N/A)

• Data Collection Period: 16 weeks

Design & Population

51

• Modified from original instrument• Face & content validity established

(Theera-Ampornpunt, 2009)

• Further modified based on pilot findings

• Translated to Thai

Section 1 Hospital ProfileSection 2 IT Adoption & Use ProfileSection 3 Respondent’s Information

English version

Survey Instrument

52

• Managerial: To what extent do you agree or disagree with each statement? e.g.,

• Those who will use the information systems are fully involved in hospital IT development

• Functional: How much is each activity supported by computerized information systems in your hospital?

• Technological: To what extent is each technology made available in your hospital?

• Integration: To what extent is information shared or transmitted among information systems within/outsideyour hospital?

Sample Questions

53

• 150-baht (~US$5) incentive if completed• Endorsed by President of the Thai Medical Informatics Association• Funded by a leading medical school with known informatics focus• Anonymous unless contact information provided for incentive & results

mailing

Survey Methodology (Nationwide)

54

• 64% response rate• Some items problematic

– Differing within-hospital responses on total & IT budgets, No. of IT staff, quality accreditation status

– Poor interrater reliability for some dimensions• Quality accreditation status dropped• Item wording revised & survey shortened• Integration sophistication items restructured

Pilot Study Findings

55

Hospital Characteristic Site 1 Site 2 Site 3 Site 4 Site 5Response rate 40% 50% 70% 70% 90%Hospital beds

Authoritative source

30 ± 0

30

120.2 ± 0.4 (120-121)

120

360 ± 0

335

303.1 ± 9.4 (282-307)

305

1,058.1 ± 187.1(863-1,500)

938PublicPrivate

100%0%

0%100%

100%0%

100%0%

100%0%

Accreditation statusNot accredited &

without planNot accredited,

with plan but no significant progress

Not accredited, with plan and significant progress

Accredited

25%

75%

0%

0%

0%

40%

40%

20%

0%

0%

0%

100%

0%

14%

86%

0%

0%

0%

0%

100%Number of IT staff

None1-56-2021-5051 or more

0%75%25%0%0%

0%80%20%0%0%

0%100%

0%0%0%

0%43%57%0%0%

0%0%

22%11%67%


56

Hospital Characteristic Site 1 Site 2 Site 3 Site 4 Site 52009 total budget (million baht)

22.0[n=1]

300.0[n=1]

578.0[n=1]

368.4 ± 93.7[n=3]

7,000.0 ± 1,414.2 [n=2]

2009 IT budget (million baht)

0.4[n=1]

10.0[n=1]

2.1 ± 1.6 [n=3]

5.5± 0.7[n=2]

93.0± 40.0[n=3]

Number of computers in hospital

23.8 ± 4.8 (20-30)

106.7 ± 90.2(20-200)

170.0 ± 108.9(10-300)

207.1 ±82.2

(100-290)

2,350.0 ± 1,332.3

(100-4,000)Calculated percentage of 2009 IT budget according to provided amount

1.8%[n=1]

3.3%[n=1]

0.5%[n=1]

1.4%[n=2]

1.3%[n=1]

Subjective estimated percentage of 2009 IT budget (if amount not provided above)

Below 1%1-4%5-8%Above 8%

0%75%25%0%

0%40%40%20%

20%60%20%0%

0%86%14%0%

0%17%50%33%


57

Construct Overall Site 1 Site 2 Site 3 Site 4 Site 5Managerial Sophistication

3.6 ± 0.4 3.2 ± 0.2 4.2 ± 0.4 3.9 ± 0.5 3.8 ± 0.4 3.2 ± 0.9

Technological Sophistication

3.5 ± 0.3 3.1 ± 0.2 3.4 ± 0.6 3.7 ± 1.0 3.4 ± 0.5 3.8 ± 0.5

Functional Sophistication

4.0 ± 0.3 3.5 ± 0.5 4.4 ± 0.4 4.1 ± 0.6 4.2 ± 0.5 4.0 ± 0.3

Integration Sophistication(Within Hospital)

3.8 ± 0.3 3.8 ± 0.3 4.2 ± 1.2 3.8 ± 0.6 3.7 ± 0.5 3.4 ± 0.7

Integration Sophistication (Outside Hospital)

2.3 ± 0.9 1.1 ± 0.04 2.5 ± 1.4 3.6 ± 0.8 2.0 ± 0.9 2.1 ± 0.5

Overall IT Sophistication

3.4 ± 0.4 2.9 ± 0.2 3.7 ± 0.7 3.9 ± 0.6 3.4 ± 0.3 3.3 ± 0.4


58

• IT Sophistication ItemsConstruct Intraclass

CorrelationCronbach’s

AlphaManagerial Sophistication 0.26* 0.91Technological Sophistication 0.04 0.81Functional Sophistication 0.20 0.93Integration Sophistication(Within Hospital)

0.00 0.89

Integration Sophistication(Outside Hospital)

0.50* 0.97

Overall IT Sophistication 0.30* 0.96*p < 0.05 on F-test.


59

• 4 of 1,302 hospitals ineligible• Response rate 69.9%

Characteristic Overall Responding Hospitals

Non-Responding

HospitalsN of eligible hospitals 1,298 908 390Bed size** 106.9 117.5 82.9Public status**

PrivatePublic

24.0%76.0%

17.4%82.6%

39.2%60.8%

Geography*CentralEastNorthNortheastSouthWest

33.4%7.5%11.1%27.1%15.3%5.6%

31.1%7.8%13.5%26.9%14.9%5.8%

39.0%6.7%5.4%27.7%16.2%5.1%

*p < 0.01, **p < 0.001.

Nationwide Findings

60

Characteristic Number of Responses Statistic†Public status

PrivatePublic

908158750

17.4%82.6%

Teaching statusNon-teachingTeaching

901716185

79.5%20.5%

Total employees 890 368.2 ± 573.5 (10-5269)IT employees 901 4.3 ± 5.3 (0-60)Total budget (million baht) 443 146.67 ± 313.60 (0.25-3,067)IT budget (million baht) 598 2.77 ± 8.79 (0-100)Ratio of IT budget to total budget‡

< 1%1-4%5-8%> 8%

4161352184023

2.7% ± 4.6% (0-43.3%)32.5%52.4%9.6%5.5%

Extent of overall IT utilizationVery lowLowModerateHighVery high

9055

35169454242

0.6%3.9%

18.7%50.2%26.7%

Total PCs in use 883 126.1 ± 218.6 (0-3,000)

Nationwide Findings

61

Estimate (Partial or Complete Adoption) NationwideBasic EHR, outpatient 86.6%Basic EHR, inpatient 50.4%Basic EHR, both settings 49.8%Comprehensive EHR, outpatient 10.6%Comprehensive EHR, inpatient 5.7%Comprehensive EHR, both settings 5.3%order entry of medications, outpatient 96.5%order entry of medications, inpatient 91.4%order entry of medications, both settings 90.2%order entry of all orders, outpatient 88.6%order entry of all orders, inpatient 81.7%order entry of all orders, both settings 79.4%

Adoption Estimates

62

Final Model

63

• High IT adoption rates• Drastic changes in adoption landscape• Local context might play a role

– Supply Side– Demand Side

• International Comparison– Relatively higher adoption

Discussion

64

• Overview of Surveys• Survey Methodology• A Sample Survey

Recap

65



Recap

survey methods & applications in healthcare

Education

sample survey

survey situation

survey responses

survey methods applications

tailored design dillman

rewards costs

social costs

examples of surveys