1 class 8 choosing measures to pretest, creating a questionnaire and questionnaire guide, methods...
TRANSCRIPT
![Page 1: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/1.jpg)
1
Class 8
Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for
Pretesting Measures
November 9, 2006
Anita L. Stewart Institute for Health & Aging
University of California, San Francisco
![Page 2: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/2.jpg)
2
Basic Steps in Selecting Appropriate Measures
1. Specify context 2. Define concept for your study3. Identify and review potential measures for
a) conceptual and psychometric adequacyb) practicality and acceptabilityc) translation available if needed
4. Select best candidates5. Pretest selected measures in your target population6. Choose best ones based on pretest results OR7. Adapt if necessary to address problems
![Page 3: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/3.jpg)
3
Overview of Class 8
Choosing measures for pretesting Creating a questionnaire and a guide to
measures in your questionnaire Methods for pretesting measures
![Page 4: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/4.jpg)
4
4. Select Best Candidate Measures of Your Concept to Pretest
Select best measures for all concepts in your conceptual framework– Preferably at least 2 measures per concept
Pay special attention to priority concepts– Outcome measures or main independent
variables
![Page 5: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/5.jpg)
5
Strategies for Selecting Best Measures
Select existing instrument in its entirety Select subscales of relevant domains from
existing instruments– only those that meet your needs
Supplement existing instrument with additional measures– core plus modules approach
![Page 6: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/6.jpg)
6
What if no “Best” Measure?
For priority concepts with inadequate measure, could select optimal measure and adapt or modify
For priority concepts with no measure, may require development of new measure
![Page 7: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/7.jpg)
7
Overview of Class 8
Choosing measures for pretesting Creating a questionnaire and a guide to
measures in your questionnaire Methods for pretesting measures
![Page 8: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/8.jpg)
8
Process of Creating Questionnaire
A. Introduction
B. Statement of confidentiality
C. Length
D. Sections, section headings to break it up
E. Order of questions, measures
F. Formatting
G. Conclusion
![Page 9: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/9.jpg)
9
Process A. Introduction
Inform respondent of purpose of overall study and of this questionnaire
Who is conducting the study Topics included Expected time to complete Assurance that participation is voluntary,
can skip any questions Specific instructions for completing
![Page 10: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/10.jpg)
10
Process B. Statement of Confidentiality
Purpose: inform respondents of extent to which their answers are protected if there are risks to confidentiality, state them
Sample Statement: “All information that would permit identification of
individuals will be regarded as strictly confidential, will be used for purposes of evaluating the study, and will not be disclosed or released for any other purposes without prior consent, except as required by law.”
![Page 11: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/11.jpg)
11
Statement of Confidentiality (cont.)
Note that this is a very high reading level Would need to simplify for lower SES
group
![Page 12: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/12.jpg)
12
Process C. Length of Questionnaire
Acceptable length depends on: Commitment of respondents to topic area Health, free time of respondents
Short forms of most measures can help For those too ill or busy to complete long
survey Minimizes bias due to systematic nonresponse
of those more ill
![Page 13: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/13.jpg)
13
Process D. Sections of Questionnaire
Break up with sections, grouped by topic (minimizes psychological burden)
Introduce new topics with– Phrases - “the next questions are about how
you’ve been feeling”
– Simple section headings, e.g.,
How You’ve Been Feeling
![Page 14: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/14.jpg)
14
Process E. Question Order
Order can affect willingness to complete survey
Begin with general questions– Easy, non-threatening, interesting, related to
purpose of study Proceed with more specific questions,
more personal and sensitive questions Conclude with demographics (least
interesting, sensitive)
![Page 15: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/15.jpg)
15
Process F. Formatting Self-Administered Questionnaires
Goal - make the tasks of reading questions, following instructions, and recording answers as easy as possible for respondents – Clear instructions for indicating their answer
– Easy to track separate questions
![Page 16: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/16.jpg)
16
Principles of Formatting
Create a lot of space on page Use very light background for best contrast Number all questions Allow sufficient space for open-ended
questions– Lines far enough apart for large script
Special issues for older adults:– Larger font size (14)– Higher contrast (black on white)
![Page 17: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/17.jpg)
17
Indicate How to Respond
Give specific instructions for answering questions, e.g.,– Circle all that apply
– Circle one number
– Put a check in the box that best represents howyou feel
If no instructions, subjects will have to figure out what to do and may not do it correctly
![Page 18: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/18.jpg)
18
Types of Formats: Horizontal
How would you rate your health in general?
(circle one number)
1 Excellent 2 Good 3 Fair 4 Poor
(easy to get confused)
![Page 19: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/19.jpg)
19
Types of Formats: Horizontal
How would you rate your health in general?
(check one box)
Excellent Good Fair Poor
(Still easy to get confused)
![Page 20: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/20.jpg)
20
Types of Formats: Vertical
(circle one number)
No, not limited at all ........... 1
Yes, limited a little .......….. 2
Yes, limited a lot ...........…. 3
During the past 4 weeks, did your healthlimit you in walking one block..
![Page 21: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/21.jpg)
21
Types of Formats: Matrix (circle one number)
How often during thepast 4 weeks did No, not Yes, Yes,your health limit limited limited limitedyou in ... at all a little a lot
Walking one block 1 2 3
Walking several blocks 1 2 3
Climbing 1 flight of stairs 1 2 3
![Page 22: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/22.jpg)
22
Types of Formats: Matrix with Boxes
(Check one box on each row)How often during thepast 4 weeks did No, not Yes, Yes,your health limit limited limited limitedyou in ... at all a little a lot
Walking one block � � �Walking several blocks � � �Climbing 1 flight of stairs � � �
Easier to administer, but harder for data entry
![Page 23: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/23.jpg)
23
Process G. Conclude Questionnaire
Add a brief thank you for completing the questionnaire (for their time and effort)
If questionnaire is to be mailed back, include instructions for mailing– Always include a pre-stamped, preaddressed
return envelope
![Page 24: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/24.jpg)
24
Sample Questionnaires
See sample questionnaires– (sample formats)
![Page 25: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/25.jpg)
25
Questionnaire Guide: Organization of Survey Content Identifies purpose of each measure in the
survey (questionnaire)– Describes source of data, time points– Describes the number of scales for each
measure Documents sources of measures and items
before you forget For modified measures, helps you know
exactly how you modified it
![Page 26: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/26.jpg)
26
See Organization of Survey Content Handout Type of variable Concept Measure Data source Data points Number of items/survey question numbers Number of scores or scales for each measure References for measure sources
![Page 27: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/27.jpg)
27
Overview of Class 8
Choosing measures for pretesting Creating a questionnaire and a guide to
measures in your questionnaire Methods for pretesting measures
![Page 28: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/28.jpg)
28
Step 5: Pretest in Target Population Pretesting essential for measures being applied
to any new population group– Especially priority measures (e.g., outcomes)
Pretest is to identify:– problems with method of administration,
respondent burden, procedures– problems with questions or response choices
![Page 29: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/29.jpg)
29
Types of Pretests
General debriefing pretest (N=10) In-depth cognitive interviewing
(N=5-10 each group) Large pretest (N=100)
– test measurement properties prior to major study
![Page 30: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/30.jpg)
30
Types of Pretests
General debriefing pretest (N=10) In-depth cognitive interviewing
(N=5-10 each group) Large pretest (N=100)
– test measurement properties prior to major study
![Page 31: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/31.jpg)
31
General Debriefing Pretest
Goal
– Identify problems with the procedures
– Estimate time needed to complete instrument
– Identify serious problems with items Procedures
– Subjects answer entire questionnaire
– At end, debrief
– Close to true task
![Page 32: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/32.jpg)
32
Debriefing Questions After Administration of Survey..
Ask respondents: Were any questions confusing? Which words were hard to understand? Which questions were difficult to answer?
caused distress? Was questionnaire too long? Confusing instructions?
![Page 33: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/33.jpg)
33
Patient Acceptance Scale
6-item scale about burden of a survey Negative affect burden
– Feeling embarrassed, upset, annoyed, uncomfortable
Questionnaire length Ease of answering questionnaire
Zimmerman M et al., Med Care, 1994;32:603-608
![Page 34: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/34.jpg)
34
Example of General Pretest to Select Best Measure of Patient Satisfaction
Compared 4 questionnaires on– Response rates, missing data, completion time,
patient ratings of the questionnaire 10 evaluation items at end of questionnaire
– Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions
Perneger TV et al., A randomized trial of four patient satisfactionquestionnaires Med Care, 2003;41(12):1343-1352
![Page 35: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/35.jpg)
35
Example of General Pretest to Select Best Measure of Patient Satisfaction
Compared 4 questionnaires on– Response rates, missing data, completion time,
patient ratings of the questionnaire 10 evaluation items at end of questionnaire
– Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions
Perneger TV et al., A randomized trial of four patient satisfactionquestionnaires Med Care, 2003;41(12):1343-1352
![Page 36: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/36.jpg)
36
Problems with General Pretests
Respondents… often don’t understand the task. don’t want to appear as if they didn’t
understand have a hard time telling you anything was
wrong easier to say everything was fine
![Page 37: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/37.jpg)
37
Types of Pretests
General debriefing pretest (N=10) In-depth cognitive interviewing
(N=5-10 each group) Large pretest (N=100)
– test measurement properties prior to major study
![Page 38: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/38.jpg)
38
In-Depth Cognitive Interviews
Derived from social and cognitive psychology to explore processes respondents use to answer survey questions
![Page 39: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/39.jpg)
39
Cognitive Interviews Examine 4 Steps in Answering Questions
Comprehend the question (as intended) Retrieve the information
– various strategies used to access memory Judgment formation - formulate an answer
– calculate or judge the correct information Edit response - decide what to report
– is answer embarrassing, socially undesirable?
Sudman S et al., Thinking About Answers, Jossey-Bass, San Francisco, 1996
![Page 40: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/40.jpg)
40
Purpose of Cognitive Interviews
To learn .. .. if respondents understand words and phrases
as intended (meaning) .. about the process of answering the questions .. whether items are unacceptable .. about the usefulness of response choices
– whether response choices are adequate
– how they use the response choices
![Page 41: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/41.jpg)
41
Two Types of Cognitive Interviews
Think aloud interviews– Respondent asked to think aloud as they
answer question Probe interviews
– Interviewer asks specific questions to elicit how respondent answered question
– Scripted and spontaneous probing
![Page 42: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/42.jpg)
42
Think Aloud Interviews
Thorough examination of the entire thought process of creating answers
Require that respondents verbalize their thought processes as they are answering items– Hard task for most people
Mostly done at “cognitive interviewing” labs, e.g., at National Institutes of Health
![Page 43: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/43.jpg)
43
Steps in Cognitive Interview Pretesting
Identify most problematic items and develop scripted “probes”
Decide on order of administration– Probes after each question or at the end
Recruit sample for cognitive interviews Administer entire questionnaire
– Administer probe questions Summarize results
![Page 44: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/44.jpg)
44
Probe Questions to Explore Meaning
What does the word _______ mean to you?
What does the phrase ________ mean to you?
![Page 45: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/45.jpg)
45
Example:Probing the Meaning of a Phrase
I asked you about how the office staff treated you personally ….
What does the phrase “office staff” mean to you?
![Page 46: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/46.jpg)
46
Example:Probing the Meaning of a Phrase
What does the phrase “office staff” mean to you?
“the receptionist and the nurses”
“nurses and appointment people”
“the person who takes your blood pressure and the clerk in the front office”
![Page 47: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/47.jpg)
47
Example:Probing the Meaning of a Phrase
What does the phrase “office staff” mean to you?
“the receptionist and the nurses”
“nurses and appointment people”
“the person who takes your blood pressure and the clerk in the front office”
We changed the question to receptionist and appointment staff
![Page 48: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/48.jpg)
48
Types of Probes (cont.)
Retrieval– How did you remember that?
Judgment– Why did you pick that number for your answer?
Response– Do you think that most people answer this
question honestly?Collins D. Quality of Life Research 2003. 12:229-38.
![Page 49: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/49.jpg)
49
Types of Probes (cont.) Redundancy
– How is the phrase “give you advice about your diet and exercise” different from the phrase “talk to you about your diet and exercise”?
![Page 50: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/50.jpg)
50
Types of Cognitive Probes (cont.)
Cultural appropriateness and meaning:– I asked you how often doctors asked you about
your health beliefs? What does the term ‘health beliefs’ mean to you?
“.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in
going to doctors?”
![Page 51: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/51.jpg)
51
Types of Cognitive Probes (cont.)
Cultural appropriateness and meaning:– I asked you how often doctors asked you about
your health beliefs? What does the term ‘health beliefs’ mean to you?
“.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in
going to doctors?” We changed the question to “personal beliefs
about your health”
![Page 52: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/52.jpg)
52
Overcoming “Social Desirability” in Cognitive Interviews
Ask respondents whether they think others…– would have difficulty answering a question
– would answer the question honestly
![Page 53: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/53.jpg)
53
Example of Probe on Difficulty: CES-D Item
“During the past week, how often have you felt that you could not shake off the blues, even with help from family and friends”
Probe: Do you feel this is a question that people would or would not have difficulty understanding?– Latinos more likely than other groups to report
people would have difficulty
TP Johnson, Health Survey Research Methods, 1996
![Page 54: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/54.jpg)
54
Example:Probing the Process of Answering
I asked you _____ and you answered____. – Why did you pick this answer?– What were you thinking of when you picked this
answer? – Can you tell me what you were thinking when
you answered this way?»Can you give me some examples?
What came to mind when I asked you _____?
![Page 55: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/55.jpg)
55
Example: Use of Response Scale
Do diverse groups use the response scale in similar ways?
Re questions about cultural competence of providers– Interviewers reported that Asian respondents
who were completely satisfied did not like to use the highest score on the rating scale
CPEHN Report, 2001
![Page 56: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/56.jpg)
56
Sample Result: Use of Response Scale
In an exercise class of Samoans, instructor asked them to rate the difficulty of the exercise he just did on a 1-10 scale
They did not understand what he meant by a 1-10 scale – “Western” metric?
![Page 57: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/57.jpg)
57
Cognitive Interviewing:Example of Probing Questions
Acceptability of Questions I asked you _____.
– Did you find this question offensive?
– Was it distressing to answer this question?
![Page 58: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/58.jpg)
58
Example of Probe for Acceptability
When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given).– Were you offended by this question?
![Page 59: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/59.jpg)
59
Steps in Cognitive Interview Pretesting
Identify most problematic items and develop scripted “probes”
Decide on order of administration– Probes after each question or at the end
Recruit sample for cognitive interviews Administer entire questionnaire
– Administer probe questions Summarize results
![Page 60: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/60.jpg)
60
Handouts: Examples of Probe Questions
Sample Cognitive Interview Protocol– Ron Hays and Leo Morales
Selected probes from Interpersonal Processes of Care study
Summary of NCHS working paper series on cognitive methods (website)
![Page 61: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/61.jpg)
61
Homework for Week 9
Create a short “questionnaire” of the measure, including instructions
Identify 4-5 questions in the measure you selected that might be a problem for your target population
– Phrases, length, response choices Write a “probe” question for each problem
– Include some on the meaning of a word or phrase and some on the adequacy of the response choices.
![Page 62: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/62.jpg)
62
Vogt et al.
Role of focus groups in adapting measures for diverse populations
How focus groups led to expansion and addition of constructs, confirmation of existing items, and identification of relevant quotes that could be used for new items
Appendices provide examples of construct definitions and a focus group guide
![Page 63: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/63.jpg)
63
Harris-Kojetin Reading
Example of Measure Undergoing Cognitive Testing – Consumer Assessment of Health Plans Study
(CAHPSTM) Describes extensive pretesting in
developing and refining the CAHPSTM
Provides example of good pretesting
![Page 64: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/64.jpg)
64
Pasick et al. reading
Provides examples of what can be learned through various types of pretests
Methods included– interviewing investigators and interviewers– focus groups– cognitive interview testing
Suggests that for important measures, multiple approaches to pretesting are useful
![Page 65: 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart](https://reader035.vdocuments.net/reader035/viewer/2022062221/56649ef35503460f94c05daa/html5/thumbnails/65.jpg)
65
Switzer et al. reading
From class 3 section of class binder p 405-406 – modifying measures