building face and content validity into health outcome measures in the context of rasch analysis
DESCRIPTION
The quality of patient reported outcome scales depends on factors such as a coherent and valid measurement model, quality item generation, face validity (for patients), content validity, acceptability to patients and fit to the Rasch model.TRANSCRIPT
Stephen McKenna and James Twiss
Galen Research Ltd, Manchester, UK
Email: [email protected]
Building face and content validity into health outcome measures in the context of Rasch analysis
Creating a scale
Let’s create a new scale ……..
We need to write a lot of items and then apply Rasch
analysis
Introduction
• But what will it really measure?
• How relevant will the items be to respondents?
The value of Rasch analysis is in perfecting the scale-
not in generating its content
Random item generation
Impact of MS Impact of MS
QoL
QoL
Func
Satis
Symp
Targeted item generation based on model of
construct
Impact of MS
QoL
What is quality of life?
• Not symptoms, activity limitations, utility or clinical
outcome
• Uniquely patient-based
• Holistic – overall impact of disease and its treatment
on the patient
• Unidimensional
“Human life quality is dependent upon the satisfaction of certain basic needs - lack of disease, mobility, adequate nutrition and shelter.”
Sir Thomas More (1478-1535)
Needs-based model
Patient interviews found:
• Impact of disease related to inability to meet needs
rather than functional limitations
• Life gains its quality from the ability and capacity of
individuals to satisfy their needs
• QoL is an unidimensional construct - providing an
index rather than a profile
Needs-based QoL measures:
• Based on coherent model of construct
• Content derived from patients
• Unidimensional – provide index of QoL
• Highly sensitive to changes in QoL
• Major advance in our ability to measure impact of
disease from patient’s perspective
Face and content validity
• Face validity appears to respondent to be asking the
right questions
• Content validity the extent to which the scale covers
important aspects of the construct being measured
Need to cover all (or most) relevant issues
Problem for computer adaptive testing?
PRIMUS
• Patient Reported Indices of Multiple Sclerosis (PRIMUS)
• consists of 3 scales:
– Symptoms (impairment)
– Activity limitations
– Needs-based quality of life
• Identifies and measures patient-perceived impact of MS
in each of these areas
Item generation: Patient interviews
• 35 in depth, unstructured qualitative interviews
• Interviews:
– Conducted in interviewee’s home
– Lasted between 1 and 3 hours
– Audio recorded
– Transcribed
Interview Sample (n=35) n (%)
Gender Male
Female
15 (43)
20 (57)
Employment Full-time
Part-time
Not working dues to MS
2 (6)
3 (9)
26 (74)
Perceived
current severity
of MS
Mild
Moderate
Severe
10 (29)
18 (51)
5 (14)
Perceived
general health
Very good
Good
Fair
Poor
2 (6)
17 (49)
10 (29)
3 (9)
Item generation: Interview content analyses
• Thematic and interpretive phenomenological (IPA) analyses
conducted on transcripts to:
– Identify key impact areas
– Interpret interviewees’ perception of that impact
• Item selection guided by measurement models / definitions:
– The WHO for symptoms and activity limitations
– Needs-based model for QoL
• Created a pool of several hundred potential items for each scale
Key areas of impact on quality of life
• Impact of diagnosis on self-esteem and identity
• Physical incapacity – dependency, freedom
• Ability to fulfil household and family roles
• Ability to care for self (personal care, appearance)
• Identity, status, time structure etc (work life)
• Intimate relationships
• Social needs with friends and family
• Impact on freedom of choice
Item reduction: QoL scale
• Remove potential items that:
– Clearly measure a different construct
– Duplicate other items
– Poorly worded
– Express more than one idea
– Difficult to translate
– Appear personal to one interviewee
– Specific to UK culture
• Resulted in 47-item scale
Item reduction: Cognitive debriefing interviews
• 15 face to face interviews with MS patients to assess:
– Ability to understand and answer
– Judgment of relevance
– Judgment of comprehensiveness
– Time to complete
• Potential to:
– Remove items that do not work well
– Change wording to improve understanding
• 2 items removed – leaving 45-item scale
Sample items from the PRIMUS QoL scale
• I feel I am a burden to others
• I have little freedom to do what I want to do
• I have to push myself to do things
• I avoid physical intimacy
• I feel as if I have nothing to offer anyone
Conclusions
Quality of patient reported outcome scale depends on:
• Coherent and valid measurement model
• Quality item generation
• Face validity (for patients)
• Content validity
• Acceptability to patients
• Fit to Rasch model