Stephen McKenna and James Twiss

Galen Research Ltd, Manchester, UK

Email: smckenna@galen-research.com

Building face and content validity into health outcome measures in the context of Rasch analysis

Creating a scale

Let’s create a new scale ……..

We need to write a lot of items and then apply Rasch

analysis

Introduction

• But what will it really measure?

• How relevant will the items be to respondents?

The value of Rasch analysis is in perfecting the scale-

not in generating its content

Random item generation

Impact of MS Impact of MS

QoL

QoL

Func

Satis

Symp

Targeted item generation based on model of

construct

Impact of MS

QoL

What is quality of life?

• Not symptoms, activity limitations, utility or clinical

outcome

• Uniquely patient-based

• Holistic – overall impact of disease and its treatment

on the patient

• Unidimensional

“Human life quality is dependent upon the satisfaction of certain basic needs - lack of disease, mobility, adequate nutrition and shelter.”

Sir Thomas More (1478-1535)

Needs-based model

Patient interviews found:

• Impact of disease related to inability to meet needs

rather than functional limitations

• Life gains its quality from the ability and capacity of

individuals to satisfy their needs

• QoL is an unidimensional construct - providing an

index rather than a profile

Needs-based QoL measures:

• Based on coherent model of construct

• Content derived from patients

• Unidimensional – provide index of QoL

• Highly sensitive to changes in QoL

• Major advance in our ability to measure impact of

disease from patient’s perspective

Face and content validity

• Face validity appears to respondent to be asking the

right questions

• Content validity the extent to which the scale covers

important aspects of the construct being measured

Need to cover all (or most) relevant issues

Problem for computer adaptive testing?

PRIMUS

• Patient Reported Indices of Multiple Sclerosis (PRIMUS)

• consists of 3 scales:

– Symptoms (impairment)

– Activity limitations

– Needs-based quality of life

• Identifies and measures patient-perceived impact of MS

in each of these areas

Item generation: Patient interviews

• 35 in depth, unstructured qualitative interviews

• Interviews:

– Conducted in interviewee’s home

– Lasted between 1 and 3 hours

– Audio recorded

– Transcribed

Interview Sample (n=35) n (%)

Gender Male

Female

15 (43)

20 (57)

Employment Full-time

Part-time

Not working dues to MS

2 (6)

3 (9)

26 (74)

Perceived

current severity

of MS

Mild

Moderate

Severe

10 (29)

18 (51)

5 (14)

Perceived

general health

Very good

Good

Fair

Poor

2 (6)

17 (49)

10 (29)

3 (9)

Item generation: Interview content analyses

• Thematic and interpretive phenomenological (IPA) analyses

conducted on transcripts to:

– Identify key impact areas

– Interpret interviewees’ perception of that impact

• Item selection guided by measurement models / definitions:

– The WHO for symptoms and activity limitations

– Needs-based model for QoL

• Created a pool of several hundred potential items for each scale

Key areas of impact on quality of life

• Impact of diagnosis on self-esteem and identity

• Physical incapacity – dependency, freedom

• Ability to fulfil household and family roles

• Ability to care for self (personal care, appearance)

• Identity, status, time structure etc (work life)

• Intimate relationships

• Social needs with friends and family

• Impact on freedom of choice

Item reduction: QoL scale

• Remove potential items that:

– Clearly measure a different construct

– Duplicate other items

– Poorly worded

– Express more than one idea

– Difficult to translate

– Appear personal to one interviewee

– Specific to UK culture

• Resulted in 47-item scale

Item reduction: Cognitive debriefing interviews

• 15 face to face interviews with MS patients to assess:

– Ability to understand and answer

– Judgment of relevance

– Judgment of comprehensiveness

– Time to complete

• Potential to:

– Remove items that do not work well

– Change wording to improve understanding

• 2 items removed – leaving 45-item scale

Sample items from the PRIMUS QoL scale

• I feel I am a burden to others

• I have little freedom to do what I want to do

• I have to push myself to do things

• I avoid physical intimacy

• I feel as if I have nothing to offer anyone

Conclusions

Quality of patient reported outcome scale depends on:

• Coherent and valid measurement model

• Quality item generation

• Face validity (for patients)

• Content validity

• Acceptability to patients

• Fit to Rasch model