1 class 8 pretesting measures and creating a questionnaire november 12, 2009 anita l. stewart...
TRANSCRIPT
![Page 1: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/1.jpg)
1
Class 8
Pretesting Measures and Creating a Questionnaire
November 12, 2009
Anita L. Stewart Institute for Health & Aging
University of California, San Francisco
![Page 2: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/2.jpg)
2
Overview of Class 8
Choosing best measure(s) for pretesting Creating a questionnaire Pretesting measures - types of pretests Cognitive interviewing Designing your own pretest
![Page 3: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/3.jpg)
3
Process of Selecting Good Measures for Your Studies
Define concept (variable)
Identify potential measures
Review measures’ properties--conceptual adequacy
--psychometric adequacy
Pretest best 1-2 measures
Select final measure
![Page 4: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/4.jpg)
4
Select Best Candidate Measures to Pretest
Select best measures for all concepts in your conceptual framework
Pretest measures you plan to use – Pretest 2 measures of key concepts if
possible so you can choose the best one
![Page 5: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/5.jpg)
5
Strategies for Selecting Best Measure of “Unidimensional” Concept
Select existing measure in its entirety Select two existing measures
– Have a choice later based on psychometric results
![Page 6: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/6.jpg)
6
Strategies for Selecting Best Measure of “Multidimensional” Concept
Select existing instrument in its entirety Select subscales of only relevant domains
from existing instruments – those that meet your needs
Supplement either of the above with subscales from other measures
![Page 7: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/7.jpg)
7
What if There is No “Best” Measure?
For priority concepts with inadequate measure, select the one that is the closest in your review– Use pretest or other qualitative studies as basis to
adapt or modify For priority concepts with no measure that
meets your needs, may require development of new measure
![Page 8: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/8.jpg)
8
Documenting Measures in Your Study
Handout – sample guide to measures used in questionnaires
Before you forget – – Summarizes source of all measures to be
administered in your study
![Page 9: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/9.jpg)
9
Overview of Class 8
Choosing best measure(s) for pretesting Creating a questionnaire Pretesting measures - types of pretests Cognitive interviewing Designing your own pretest
![Page 10: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/10.jpg)
10
Considerations in Creating a Questionnaire
Introduction Statement of confidentiality Length Sections, section headings to break it up Order of questions, measures Formatting Conclusion
![Page 11: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/11.jpg)
11
Introduction: Inform respondent …
Purpose of overall study and of this questionnaire
Who is conducting the study Topics included in questionnaire Expected time to complete Assurance that participation is voluntary,
can skip any questions Specific instructions for completing
![Page 12: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/12.jpg)
12
Statement of Confidentiality
Inform respondents of extent to which their answers are protected if there are risks to confidentiality, state
them Can use section from consent form
pertaining to survey data
![Page 13: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/13.jpg)
13
Sample Statement of Confidentiality
“All information that would permit identification of individuals will be regarded as strictly confidential, will be used for purposes of evaluating the study, and will not be disclosed or released for any other purposes without prior consent, except as required by law.”
![Page 14: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/14.jpg)
14
Statement of Confidentiality (cont.)
Note that this is a very high reading level– Would need to simplify for lower SES group
“All information about you will be kept as confidential as possible. Your name will not be used in any published reports about this study.”
![Page 15: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/15.jpg)
15
Length of Questionnaire
Acceptable length depends on: Interest of respondents in topic area Health Free time of respondents
![Page 16: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/16.jpg)
16
Create Sections of Questionnaire
Break up with sections, grouped by topic (minimizes psychological burden)
Introduce new topics with– Phrases - “the next questions are about how
you’ve been feeling”
– Simple section headings, e.g.,
How You’ve Been Feeling
![Page 17: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/17.jpg)
17
Question Order
Order can affect willingness to complete survey
Begin with general questions– Easy, non-threatening, interesting, related to
purpose of study Proceed with more specific questions,
personal and sensitive questions Conclude with demographics (least
interesting, sensitive)
![Page 18: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/18.jpg)
18
Indicate How to Respond
Give specific instructions for answering questions, e.g.,– Circle all that apply
– Circle one number
– Check the box that best represents howyou feel
If no instructions, subjects will have to figure out what to do and may not do it correctly
![Page 19: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/19.jpg)
19
Formatting Self-Administered Questionnaires
Make the tasks of reading questions, following instructions, and recording answers as easy as possible for respondents – Clear instructions for indicating their answer
– Easy to track separate questions
![Page 20: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/20.jpg)
20
Principles of Formatting
Create a lot of space on page Use very light background for best contrast
– Minimize use of color as background Number all questions (unique number for each
one helps data entry) Allow sufficient space to write answers to
open-ended questions Special issues for older adults:
– Larger font size (14)– Higher contrast (black on white)
![Page 21: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/21.jpg)
21
Types of Formats: Horizontal
How would you rate your health in general?
(check one box)
Excellent Good Fair Poor
(Easy to get confused)
![Page 22: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/22.jpg)
22
Types of Formats: Respondent Writes Response Choice
For each question, write in one of the following numbers: 1 = poor
2 = fair
3 = good
4 = excellent
___ 1. How would you rate the availability of medical information by phone?
___ 2. How would you rate the length of time spent waiting at the office to see a provider?
![Page 23: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/23.jpg)
23
Types of Formats: Vertical
(circle one number)
No, not limited at all ........... 1
Yes, limited a little .......….. 2
Yes, limited a lot ...........…. 3
During the past 4 weeks, did your healthlimit you in walking one block..
![Page 24: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/24.jpg)
24
Types of Formats: Matrix
(Circle one number on each line)
How often during the past 4 weeks did your health limit you ….
No, not limitedat all
Yes,limiteda little
Yes,limited
a lot
1. Walking one block 1 2 3
2. Walking several blocks 1 2 3
3. Climbing one flight of stairs 1 2 3
![Page 25: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/25.jpg)
25
Types of Formats: Matrix with Boxes
(Check one box on each line)
How often during the past 4 weeks did your health limit you ….
No, not limitedat all
Yes,limiteda little
Yes,limited
a lot
1. Walking one block � � �2. Walking several blocks � � �3. Climbing one flight of stairs � � �
![Page 26: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/26.jpg)
26
Conclude Questionnaire
Add a brief thank you for completing the questionnaire (for their time and effort)
If questionnaire is to be mailed back, include instructions for mailing– Always include a pre-stamped, preaddressed
return envelope
![Page 27: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/27.jpg)
27
Resources for Creating Questionnaires
Mullin et al article on formatting principles
Handout: formatting tips for using WORD See sample questionnaires in class
– (sample formats)
![Page 28: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/28.jpg)
28
Overview of Class 8
Choosing best measure(s) for pretesting Creating a questionnaire Pretesting measures - types of pretests Cognitive interviewing Designing your own pretest
![Page 29: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/29.jpg)
29
Pretest in Target Population Pretesting essential for measures being
applied to any new population group– Especially priority measures (e.g., outcomes)
Pretest is to identify:– problems with procedures
» method of administration, respondent burden
– problems with questions» Item stems, response choices, and instructions
![Page 30: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/30.jpg)
30
Types of Pretests
General debriefing pretest (N=10) Large pretest (N=100)
– test measurement properties prior to major study
In-depth cognitive interviewing pretest
![Page 31: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/31.jpg)
31
Types of Pretests
General debriefing pretest (N=10) Large pretest (N=100)
– test measurement properties prior to major study
In-depth cognitive interviewing
![Page 32: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/32.jpg)
32
General Debriefing Pretest
Goal
– Identify problems with the procedures
– Estimate time needed to complete instrument
– Identify serious problems with items Procedures
– Subjects answer entire questionnaire
– At end, debrief
– Close to true task
![Page 33: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/33.jpg)
33
Sample Debriefing Questions After Administration of Survey..
Ask respondents: Were any questions confusing? Which words were hard to understand? Which questions were difficult to answer?
caused distress? Was questionnaire too long? Were instructions confusing?
![Page 34: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/34.jpg)
34
“Questionnaire” Asking Opinions About another Questionnaire/Survey
Six questions about burden of a survey– Questionnaire length
– Ease of answering questionnaire
– Negative affect burden (4 items)» Feeling embarrassed, upset, annoyed,
uncomfortable
M Zimmerman et al., Med Care, 1994;32:603-608
![Page 35: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/35.jpg)
35
4-item Negative Affect Burden Scale
How much were you annoyed by being asked these questions?
How much were you embarrassed by being asked these questions?
How much did it upset you to be asked these questions?
How uncomfortable did it make you feel to be asked these questions?
Responses: very much, somewhat, a little, not at allZimmerman M et al.1994
![Page 36: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/36.jpg)
36
Problems with General Pretests
Respondents… – often don’t understand the task.
– don’t want to appear as if they didn’t understand
– have a hard time telling you anything was wrong
– easier to say everything was fine
![Page 37: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/37.jpg)
37
Types of Pretests
General debriefing pretest (N=10) Large pretest (N=100)
– test measurement properties prior to major study
In-depth cognitive interviewing
![Page 38: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/38.jpg)
38
Large Pretest – N=100
Test psychometric properties and procedures for large-scale administration– Administer surveys– Examine item distributions, missing data– Calculate internal-consistency reliability– Conduct some preliminary validity studies
Only large-scale studies have this luxury
![Page 39: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/39.jpg)
39
Large Pretest of 4 Patient Satisfaction with Hospital Care Measures (N=2,850)
Compared 4 questionnaires on– Response rates, missing data, completion time
10 evaluation items at end of questionnaire (1-5 agree-disagree scale)– Clarity of questions, ease of finding an answer,
questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions
TV Perneger et al., Med Care 2003;41:1343-1352
![Page 40: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/40.jpg)
40
Medical Outcomes Study Measures
On long-form measures, conducted 9 full-scale pilot studies over a 1-year period– Physical functioning, health perceptions,
energy/fatigue, sleep, role functioning, pain, symptoms, family functioning, sexual functioning
Samples ranged from 50-100 each
Stewart AL and Ware JE, Measuring Functioning and Well-Being, 1992, p. 72..
![Page 41: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/41.jpg)
41
Types of Pretests
General debriefing pretest (N=10) Large pretest (N=100)
– test measurement properties prior to major study
In-depth cognitive interviewing pretest (N=5-10 each group)
![Page 42: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/42.jpg)
42
Overview of Class 8
Choosing best measure(s) for pretesting Creating a questionnaire Pretesting measures - types of pretests Cognitive interviewing Designing your own pretest
![Page 43: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/43.jpg)
43
In-Depth Cognitive Interviews
Derived from social and cognitive psychology to explore processes respondents use to answer survey questions
Goal: understand thought processes used to answer questions
Can help write/adapt questions
![Page 44: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/44.jpg)
44
Types of Problems with Questions or Response Choices
Are all words/phrases understood as intended? Are questions interpreted similarly by all
respondents? Are some questions not answered? Are any questions offensive or irrelevant? Does each closed-ended question have an
answer that applies to each respondent?– Are the response choices adequate?
![Page 45: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/45.jpg)
45
Cognitive Interviews Examine 4 Steps in Answering Questions
Comprehend question (as intended) Retrieve information - various strategies
used to recall information Judgment formation - calculate the
correct response Response - decide what to report
Sudman S et al., Thinking About Answers, Jossey-Bass, San Francisco, 1996
![Page 46: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/46.jpg)
46
Comprehend Questions as Intended
Bulk of cognitive interview pretests assess whether your items are interpreted by respondents as you intended
![Page 47: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/47.jpg)
47
Retrieve Information
Information needed to answer the question must be retrieved from memory– Episodic data – stored in detail
– Schemas – generalizations about events Frequent, routine events more likely to be
stored as “schema” with no detail
![Page 48: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/48.jpg)
48
Example of Retrieval Schema
People with chronic conditions and a lot of doctor visits tend to have “generic” memories of these visits– Multiple visits blurred together
– Could describe a typical visit but not any visit in particular
– Hard to recall how many visits
![Page 49: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/49.jpg)
49
Judgment Formation
Involves using the recalled information to estimate frequency of an event, or to decide whether one agrees or disagrees with a statement
Cues taken from the item response choices
![Page 50: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/50.jpg)
50
Response
After respondent has judged what the best answer is, decides how to respond– Accurately?
– To improve social desirability?
– To minimize embarrassment?
![Page 51: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/51.jpg)
51
Probe Interviews: Types of Probes
Interpret meaning as you intended Ease of answering – appropriate response
choices Acceptability of questions
– Culturally appropriate Clarity, difficulty Elicit socially desirable responding
![Page 52: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/52.jpg)
52
Probes to Determine if Respondents Understand Meaning as Intended
Wording of pretest item should be non-threatening:
Example: What do you think we meant by the phrase ________? – Threatening – implies there is a right answer
![Page 53: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/53.jpg)
53
More Appropriate Probe Questions to Explore Meaning
What does the word _______ mean to you?
What do you understand by _______ ? What came to mind when I asked you
_____? Ask person to paraphrase a question
– Repeat question in your own words
![Page 54: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/54.jpg)
54
Probing Meaning of a Phrase
I asked you about how the office staff treated you personally ….– What does the phrase “office staff” mean to you?
I asked you how often doctors asked you about your health beliefs? – What does the term ‘health beliefs’ mean to you?
![Page 55: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/55.jpg)
55
Probing Meaning of Phrase
After a series of questions about the person’s main job– You mentioned earlier that you had more
than one job. How did you decide which job was your main job?
DeMaio TJ et al., Answering Questions, Jossey-Bass, 1996, p. 177..
![Page 56: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/56.jpg)
56
Probing Meaning of Entire Question
Do you go to the doctor for checkups even when you’re well?– What does the word “checkup” mean to you?
– Can you rephrase this question in your own words?
Pasick RJ et al., Pub Health Rep, 2001;116:223.
![Page 57: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/57.jpg)
57
Probing Meaning of Entire Question
In the last 6 months, have you had problems getting to see a specialist? (yes/no)– Tell me more about that.– What kind of doctor do you think of as a
specialist?– If yes - What kinds of problems have you
had?
![Page 58: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/58.jpg)
58
Probing Confidence in Answer
After respondent answers a question– How confident are you in the answer you
gave? » Very confident, somewhat confident, not
confident at all
Lack of confidence is a flag for a problem
![Page 59: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/59.jpg)
59
Probing Use of Response Scale
I asked you _____ and you answered____.
What were you thinking of when you picked this answer?
Can you tell me why you picked that number for your answer?
How did you decide to choose that answer? Were you able to find the right answer to the
question from the response options?
![Page 60: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/60.jpg)
60
Probing Retrieval
How did you remember that? Did you have a particular time period in
mind? How did you calculate your answer? Can you tell me what you were thinking
when you answered this way?»Can you give me some examples?
![Page 61: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/61.jpg)
61
Probe for Acceptability
I asked you _____.– Did you find this question offensive?
– Was it distressing to answer this question?
![Page 62: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/62.jpg)
62
Probe for Acceptability (cont)
When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given).– Were you offended by this question?– Do you think people would be offended by this
question?
![Page 63: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/63.jpg)
63
Probe on Difficulty: CES-D Item
“During the past week, how often have you felt that you could not shake off the blues, even with help from family and friends”– Do you feel this is a question that people would
or would not have difficulty understanding?
TP Johnson, Health Survey Research Methods, 1996
![Page 64: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/64.jpg)
64
Overcoming “Social Desirability” in Cognitive Interviews
Ask respondents whether they think others– would have difficulty answering a question
– would answer honestly
Example: – Do you think that most people would
answer this question honestly?
![Page 65: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/65.jpg)
65
Overview of Class 8
Choosing best measure(s) for pretesting Creating a questionnaire Pretesting measures - types of pretests Cognitive interviewing Designing your own pretest
![Page 66: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/66.jpg)
66
Steps in Cognitive Interview Pretesting
Identify problematic items– Develop scripted “probes” for each one
Recruit sample for cognitive interviews Administer questionnaire and probe
questions Tape record and transcribe Summarize results
![Page 67: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/67.jpg)
67
Identify Potentially Problematic Items
Optimal: a small debriefing pretest to find out potential problems
Investigator judgment– You will be good at spotting these
Identify problems with item stems and response choices
![Page 68: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/68.jpg)
68
Write Scripted Probes for Each Problem
Create a cognitive interview version of the questionnaire– Not the real questionnaire
Underneath each problem item, write a “probe” question you would ask each pretest respondent
Examples in slides and handouts
![Page 69: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/69.jpg)
69
Handouts: Sample Cognitive Interviews
Sample Cognitive Interview Protocol– Ron Hays and Leo Morales (UCLA)
» Experts at cognitive interviewing pretests
Cognitive interview protocols for Interpersonal Processes of Care study– Divided into two parts due to length – Each part: a questionnaire and the cognitive
interview questions accompanying it
![Page 70: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/70.jpg)
70
Recruiting for Cognitive Interviewing
The task of your pretest respondent is:– Answering questions (normal interview)
– Being debriefed or answering additional questions about their answers (cognitive interviewing)
Difficult to explain this during recruitment
![Page 71: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/71.jpg)
71
Recruiting for Cognitive Interviewing (cont.)
Pay subjects (if you can)– Special demand on them
If survey is long, may have to pretest different sections on different subjects– Requires more total subjects
![Page 72: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/72.jpg)
72
Recruiting Scripts for Cognitive Interviewing
Explain how their help fits into the process of creating questions
Ways to explain
– “help us learn how to ask better questions”
– “help us make questions clearer for others”
– “help us learn how we can improve this questionnaire”
– “help us find problems with the questions”
![Page 73: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/73.jpg)
73
Administer Questionnaire and Probes
Administer entire questionnaire– Or break into sections
Self-administration or interview?– Interview identifies more problems
Administer probes Audiotape entire interview including probes
– Transcribe audiotapes
RESULT: information on the entire process of administering the questionnaire
![Page 74: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/74.jpg)
74
Two Sequences to Administering Probes
Concurrent probing– Ask each probe immediately after respondent has
given answer to survey item– Advantage-information is fresh in respondent’s
mind Retrospective probing (preferred)
– Ask probes after entire interview– Advantage-able to assess standard administration of
items– Doesn’t break up the respondent’s task
![Page 75: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/75.jpg)
75
Interviewer’s Role: Spontaneous Probing
Probe on items that appear to be problematic
If a long pause in answering..– “I noticed you pausing - what came to mind
when you I asked you that question?”
![Page 76: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/76.jpg)
76
Other Cues to Problems in Face-to- Face Pretests
When administering the survey… Be aware of behavioral cues related to
specific items or to questions in general– Discomfort
– Yawning
– Looking at their watch
![Page 77: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/77.jpg)
77
Other Cognitive Interview Methods Resources
Using cognitive interviews to develop structured surveys (May 2007)– Annotated bibliography of articles describing
methods of cognitive interviews
![Page 78: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/78.jpg)
78
Homework for Class 9/10
Select best measure Create “mini questionnaire”
– Formatted for self-report, including instructions Write 4 “probe” questions of selected items
– At least one on the meaning of a word/phrase and one on adequacy of response choices
Identify source of 2 pretest “subjects” similar to those you would include in your research
– Write brief 1-paragraph script for recruiting them
![Page 79: 1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San](https://reader030.vdocuments.net/reader030/viewer/2022032707/56649e1a5503460f94b0872c/html5/thumbnails/79.jpg)
79
Next and Last Class
December 3 Blend of class 9 and 10