1 class 7 pretesting measures and considerations in modifying or adapting measures november 10, 2005...
TRANSCRIPT
![Page 1: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/1.jpg)
1
Class 7
Pretesting measures and considerations in modifying or adapting measures
November 10, 2005
Anita L. Stewart Institute for Health & Aging
University of California, San Francisco
![Page 2: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/2.jpg)
2
Steps 5-7: Pretesting and Adapting
1. Specify context 2. Define concept for your study3. Identify and review potential measures for
a) conceptual and psychometric adequacyb) practicality and acceptabilityc) translation available if needed
4. Select best candidates5. Pretest selected measures in your groups6. Choose best ones based on pretest results, OR7. Adapt if necessary to address problems this week
![Page 3: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/3.jpg)
3
Purposes of Pretest by Context
In your search for measures, can look for evidence of pretesting– Evaluate whether measure was pretested and
adequacy of methods When creating a battery of measures
– You can conduct pretest to help you decide between alternative measures
When using specific measures– You can conduct pretest to identify problems
![Page 4: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/4.jpg)
4
Example of Measure Undergoing Cognitive Testing
Consumer Assessment of Health Plans Study (CAHPSTM)– Harris-Kojetin et al., 1999 (in readings)
Describes extensive pretesting in developing and refining the CAHPSTM
Provides example of good pretesting
![Page 5: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/5.jpg)
5
Step 5: Pretest in Target Population Pretesting essential for measures being applied
to any new population group– Especially priority measures (e.g., outcomes)
Pretest is to identify:– problems with method of administration,
respondent burden, procedures– problems with questions or response choices
![Page 6: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/6.jpg)
6
Types of Pretests
General debriefing pretest (N=10) In-depth cognitive interviewing
(N=5-10 each group) Large pretest (N=100)
– test measurement properties prior to major study
![Page 7: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/7.jpg)
7
Types of Pretests
General debriefing pretest (N=10) In-depth cognitive interviewing
(N=5-10 each group) Large pretest (N=100)
– test measurement properties prior to major study
![Page 8: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/8.jpg)
8
General Debriefing Pretest
Goal
– Identify problems with the procedures
– Estimate time needed to complete instrument
– Identify serious problems with items Procedures
– Subjects answer entire questionnaire
– At end, debrief
– Close to true task
![Page 9: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/9.jpg)
9
Debriefing Questions After Administration of Survey..
Ask respondents: Were any questions confusing? Which words were hard to understand? Which questions were difficult to answer?
caused distress? Was questionnaire too long? Confusing instructions?
![Page 10: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/10.jpg)
10
Patient Acceptance Scale
6-item scale about burden of a survey Negative affect burden
– Feeling embarrassed, upset, annoyed, uncomfortable
Questionnaire length Ease of answering questionnaire
Zimmerman M et al., Med Care, 1994;32:603-608
![Page 11: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/11.jpg)
11
Example of General Pretest to Select Best Measure of Patient Satisfaction
Compared 4 questionnaires on– Response rates, missing data, completion time,
patient ratings of the questionnaire 10 evaluation items at end of questionnaire
– Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions
Perneger TV et al., A randomized trial of four patient satisfactionquestionnaires Med Care, 2003;41(12):1343-1352
![Page 12: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/12.jpg)
12
Example of General Pretest to Select Best Measure of Patient Satisfaction
Compared 4 questionnaires on– Response rates, missing data, completion time,
patient ratings of the questionnaire 10 evaluation items at end of questionnaire
– Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions
Perneger TV et al., A randomized trial of four patient satisfactionquestionnaires Med Care, 2003;41(12):1343-1352
![Page 13: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/13.jpg)
13
Problems with General Pretests
Respondents… often don’t understand the task. don’t want to appear as if they didn’t
understand have a hard time telling you anything was
wrong easier to say everything was fine
![Page 14: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/14.jpg)
14
In-Depth Cognitive Interviews
Derived from social and cognitive psychology to explore processes respondents use to answer survey questions
![Page 15: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/15.jpg)
15
Cognitive Interviews Examine 4 Steps in Answering Questions
Comprehend the question (as intended) Retrieve the information
– various strategies used to access memory Judgment formation - formulate an answer
– calculate or judge the correct information Edit response - decide what to report
– is answer embarrassing, socially undesirable?
Sudman S et al., Thinking About Answers, Jossey-Bass, San Francisco, 1996
![Page 16: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/16.jpg)
16
Purpose of Cognitive Interviews
To learn .. .. if respondents understand words and phrases
as intended (meaning) .. about the process of answering the questions .. whether items are unacceptable .. about the usefulness of response choices
– whether response choices are adequate
– how they use the response choices
![Page 17: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/17.jpg)
17
Two Types of Cognitive Interviews
Think aloud interviews– Respondent asked to think aloud as they
answer question Probe interviews
– Interviewer asks specific questions to elicit how respondent answered question
– Scripted and spontaneous probing
![Page 18: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/18.jpg)
18
Think Aloud Interviews
Thorough examination of the entire thought process of creating answers
Require that respondents verbalize their thought processes as they are answering items– Hard task for most people
Mostly done at “cognitive interviewing” labs, e.g., at National Institutes of Health
![Page 19: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/19.jpg)
19
Steps in Cognitive Interview Pretesting
Identify most problematic items and develop scripted “probes”
Decide on order of administration– Probes after each question or at the end
Recruit sample for cognitive interviews Administer entire questionnaire
– Administer probe questions Summarize results
![Page 20: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/20.jpg)
20
Probe Questions to Explore Meaning
What does the word _______ mean to you?
What does the phrase ________ mean to you?
![Page 21: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/21.jpg)
21
Example:Probing the Meaning of a Phrase
I asked you about how the office staff treated you personally ….
What does the phrase “office staff” mean to you?
![Page 22: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/22.jpg)
22
Example:Probing the Meaning of a Phrase
What does the phrase “office staff” mean to you?
“the receptionist and the nurses”
“nurses and appointment people”
“the person who takes your blood pressure and the clerk in the front office”
![Page 23: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/23.jpg)
23
Example:Probing the Meaning of a Phrase
What does the phrase “office staff” mean to you?
“the receptionist and the nurses”
“nurses and appointment people”
“the person who takes your blood pressure and the clerk in the front office”
We changed the question to receptionist and appointment staff
![Page 24: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/24.jpg)
24
Types of Probes (cont.)
Retrieval– How did you remember that?
Judgment– Why did you pick that number for your answer?
Response– Do you think that most people answer this
question honestly?Collins D. Quality of Life Research 2003. 12:229-38.
![Page 25: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/25.jpg)
25
Types of Probes (cont.) Redundancy
– How is the phrase “give you advice about your diet and exercise” different from the phrase “talk to you about your diet and exercise”?
![Page 26: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/26.jpg)
26
Types of Cognitive Probes (cont.)
Cultural appropriateness and meaning:– I asked you how often doctors asked you about
your health beliefs? What does the term ‘health beliefs’ mean to you?
“.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in
going to doctors?”
![Page 27: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/27.jpg)
27
Types of Cognitive Probes (cont.)
Cultural appropriateness and meaning:– I asked you how often doctors asked you about
your health beliefs? What does the term ‘health beliefs’ mean to you?
“.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in
going to doctors?” We changed the question to “personal beliefs
about your health”
![Page 28: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/28.jpg)
28
Overcoming “Social Desirability” in Cognitive Interviews
Ask respondents whether they think others…– would have difficulty answering a question
– would answer the question honestly
![Page 29: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/29.jpg)
29
Example of Probe on Difficulty: CES-D Item
“During the past week, how often have you felt that you could not shake off the blues, even with help from family and friends”
Probe: Do you feel this is a question that people would or would not have difficulty understanding?– Latinos more likely than other groups to report
people would have difficulty
TP Johnson, Health Survey Research Methods, 1996
![Page 30: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/30.jpg)
30
Example:Probing the Process of Answering
I asked you _____ and you answered____. – Why did you pick this answer?– What were you thinking of when you picked this
answer? – Can you tell me what you were thinking when
you answered this way?»Can you give me some examples?
What came to mind when I asked you _____?
![Page 31: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/31.jpg)
31
Example: Use of Response Scale
Do diverse groups use the response scale in similar ways?
Re questions about cultural competence of providers– Interviewers reported that Asian respondents
who were completely satisfied did not like to use the highest score on the rating scale
CPEHN Report, 2001
![Page 32: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/32.jpg)
32
Sample Result: Use of Response Scale
In an exercise class of Samoans, instructor asked them to rate the difficulty of the exercise he just did on a 1-10 scale
They did not understand what he meant by a 1-10 scale – “Western” metric?
![Page 33: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/33.jpg)
33
Cognitive Interviewing:Example of Probing Questions
Acceptability of Questions I asked you _____.
– Did you find this question offensive?
– Was it distressing to answer this question?
![Page 34: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/34.jpg)
34
Example of Probe for Acceptability
When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given).– Were you offended by this question?
![Page 35: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/35.jpg)
35
Interviewer’s Role: Spontaneous Probing
Probe on items that appear to be problematic
If a long pause in answering..– “I noticed you pausing - what came to mind
when you I asked you that question?”
![Page 36: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/36.jpg)
36
Other Cues to Problems in Face-to- Face Pretests
When administering the survey… Be aware of behavioral cues related to
specific items or to questions in general– Discomfort
– Yawning
– Looking at their watch
![Page 37: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/37.jpg)
37
Steps in Cognitive Interview Pretesting
Identify most problematic items and develop scripted “probes”
Decide on order of administration– Probes after each question or at the end
Recruit sample for cognitive interviews Administer entire questionnaire
– Administer probe questions Summarize results
![Page 38: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/38.jpg)
38
Two Approaches to Probing
Concurrent probing– Ask probes immediately after respondent has given
answer to survey item– Advantage-information is fresh in respondent’s
mind Retrospective probing
– Ask probes after entire interview– Advantage-able to assess standard administration of
items
![Page 39: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/39.jpg)
39
Steps in Cognitive Interview Pretesting
Identify most problematic items and develop scripted “probes”
Decide on order of administration– Probes after each question or at the end
Recruit sample for cognitive interviews Administer entire questionnaire
– Administer probe questions Summarize results
![Page 40: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/40.jpg)
40
Special Script for Recruiting People for Cognitive Interviewing
Respondents often do not understand their role Explain to them how their help fits into the
larger study, goal of main study, process of creating questions
Explain their role clearly:
– “help us learn how to ask better questions”
– “help us make questions clearer for others”
![Page 41: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/41.jpg)
41
Recruiting Subjects for Cognitive Interviewing
Explain who you are, purpose of pretest, their role, expected duration (1-1.5 hours)– Subjects told it is a pretest and that their task is to
help by identifying problems Pay subjects since it is a special demand on
them (if you can) If survey is long, may have to pretest different
sections on different subjects Requires more total subjects
![Page 42: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/42.jpg)
42
Steps in Cognitive Interview Pretesting
Identify most problematic items and develop scripted “probes”
Decide on order of administration– Probes after each question or at the end
Recruit sample for cognitive interviews Administer entire questionnaire
– Administer probe questions Summarize results
![Page 43: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/43.jpg)
43
Administer Questionnaire and Probes
Administer entire questionnaire– Or break into sections
Administer probes Audiotape entire interview including probes
– Transcribe audiotapes
RESULT: information on the entire process of administering the questionnaire
![Page 44: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/44.jpg)
44
Steps in Cognitive Interview Pretesting
Identify most problematic items and develop scripted “probes”
Decide on order of administration– Probes after each question or at the end
Recruit sample for cognitive interviews Administer entire questionnaire
– Administer probe questions Summarize results
![Page 45: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/45.jpg)
45
Summarize Data on Pretest Interviews
Summarize problems and nature of problems for each item
Determine how important problems are Data become basis for possible
revisions/adaptations
![Page 46: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/46.jpg)
46
Behavioral Coding
Review transcripts to identify problems Two types of problems:
– Problems with standard administration of items» respondent and interviewer “problem” behaviors
– Problems identified with specific probe questions
![Page 47: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/47.jpg)
47
Examples of Behavioral Codes for Standard Administration
Interviewer behavior Hard to read - interviewer experiences
difficulty reading question
Respondent behavior Repeat question - respondent asks to have
the question repeated Asks for clarification
![Page 48: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/48.jpg)
48
Summarize Behavioral Coding For Each Item
Proportion of interviews with each problematic behavior
For standard administration– # of occurrences of each problem divided by N
» e.g., 7/48 respondents requested clarification For probes
– # of occurrences of each problem divided by N for probe
» e.g., 2/15 respondents did not understand meaning based on response to probe
![Page 49: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/49.jpg)
49
Behavioral Coding Summary Sheet: Standard Administration (N=20)
Item #
Interviewer: difficulty reading
Subject: asks to repeat Q
Subject: asks for
clarification
1 2/20 0 1/20
2 0 0 0
3 1/20 3/20 2/20
4 0 1/20 0
![Page 50: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/50.jpg)
50
Examples of Behavioral Codes Based on Probes
Probe on meaning – Open-ended responses indicate lack of
understanding Probe on use of response options
– Open-ended responses indicate response options are problematic
![Page 51: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/51.jpg)
51
Example:Probing the Meaning of a Phrase
I asked you how often doctors asked you about your health beliefs. What does the term “health beliefs” mean to you? “.. I don’t want medicine”
“.. How I feel, if I was exercising…” “.. Like religion? --not believing in
going to doctors?” Behavioral coding: Count number of times a subject
did NOT understand meaning as intended
![Page 52: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/52.jpg)
52
Behavioral Coding Summary Sheet: Standard Administration (N=20) + Probes
Item # ProbesMeaning unclear
Interviewer -difficulty
reading
Subject: asks to
repeat Q
Subject: asks for
clarification
1 10 2/10 2/20 0 1/20
2 0 0 0 0 0
3 15 4/15 1/20 3/20 2/20
4 15 0 0 1/20 0
![Page 53: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/53.jpg)
53
Summarize Behavioral Coding
Determine if problem is common– Items with only a few problems may be fine– Items are questionable when
» several types of problems were found» several subjects experienced the same problem
![Page 54: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/54.jpg)
54
Behavioral Coding Summary Sheet: Standard Administration (N=20) + Probes
Item # ProbesMeaning unclear
Interviewer -difficulty
reading
Subject: asks to
repeat Q
Subject: asks for
clarification
1 10 2/10 2/20 0 1/20
2 0 0 0 0 0
3 15 4/15 1/20 3/20 2/20
4 15 0 0 1/20 0
![Page 55: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/55.jpg)
55
Summarize Behavioral Coding (cont.)
Determine if common problems with an item are serious– Gross misunderstanding of the question– Yields completely erroneous answer– Couldn’t answer the question at all
Some less serious “common” problems can be addressed by improved instructions or a slight modification
![Page 56: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/56.jpg)
56
Now What!
Issues in adapting measures based on pretest results
![Page 57: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/57.jpg)
57
Options for Adapting or Revising
Depend on whether you are using a standard measure or a new measure
![Page 58: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/58.jpg)
58
Writing New Items
One approach to finding problems with a standard measure in diverse groups– Write some new items that you think will be better for
this group– Write them in the same format as the standard measure
Can test the adequacy of the original measure and a “modified” measure that includes your new items– A way to begin to develop new items
![Page 59: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/59.jpg)
59
Adapting Measures
If a measure needs modification in your opinion– Can contact author(s) to see if they would like to
work with you– Be sure your opinion is based on extensive pretests
in several groups by your group and/or other researchers
– Work with a measurement specialist to assure that your proposed modifications are likely to solve the problem
![Page 60: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/60.jpg)
60
Internal versus External Validity
Internal validity - your study is able to answer your research questions with a measure
External validity - your results on a measure can be compared to other studies using the same measure
![Page 61: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/61.jpg)
61
Tradeoffs of Using Adapted Measures
Modifying measures– Can improve internal validity - able to answer your
question with this measure – You lose external validity - can’t compare your scores
to other studies Can test measure in original and adapted
form– If “adapted” measure works better, use it
If “adapted” measure does not work better– Can still use original measure (it may not work either)
![Page 62: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/62.jpg)
62
Handouts: Examples of Probe Questions
Sample Cognitive Interview Protocol– Ron Hays and Leo Morales
Selected probes from Interpersonal Processes of Care study
Summary of NCHS working paper series on cognitive methods (website)
![Page 63: 1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University](https://reader034.vdocuments.net/reader034/viewer/2022051316/56649e9f5503460f94ba129e/html5/thumbnails/63.jpg)
63
Homework for Next Class (Dec 2)
Identify 4-5 questions in measure you have selected that might be a problem for your target population– Phrases, length, response choices
Write one “probe” question for each problem Conduct 2 pretests on persons similar to those
you are interested in Summarize briefly your results