the application of anchoring vignettes to the eq-5d-5l:a possible solution to reporting...

Paula LorgellyDeputy DirectorVisiting Professor, Division of Cancer Studies, Kings College London

9th June 2016

The application of anchoring vignettes to the EQ-5D-5L:a possible solution to reporting heterogeneity in PROMs

Application of anchoring vignettes to the EQ-5D-5L:a possible solution to reporting heterogeneity in PROMs

Acknowledgments• Australian Research Council Discovery Project Grant

(DP110101426)• Investigators: Paula Lorgelly, Bruce Hollingsworth, Mark Harris,

Nigel Rice, John Wildman, William Greene• Researchers: Rachel Knott, Nicole Black (Au)

• BankWest Curtin Economics Centre Research Grants Program • Mark Harris, Nigel Rice, Paula Lorgelly, Rachel Knott

• Faculty of Business and Economics Research Grant Scheme • Paula Lorgelly, Rachel Knott, Mark Harris


Background•Individual and household surveys often rely on self-reported measures of health

• In general, would you say your health is: excellent, very good, good, fair or poor?

•Analyses using measures of self-reported health (SRH) rely on the measure being an accurate reflection of the true health of the groups or individuals concerned•But responses to questions on subjective scales will be inaccurate if groups of individuals systematically differ in their use and/or interpretation of the response categories•Systematic variation in the use of response categories is known as reporting heterogeneity or response scale heterogeneity or differential item functioning (DIF)

3


EQ-5D• The EQ-5D is the most commonly used instrument for measuring

preference-based health-related quality of life (HRQoL) • The responses to the five health domains can be used

descriptively as health profiles (12112) or converted to a preference-weighted summary index which reflects health-related utility (where 0 is dead and 1 is full health), thus can be used to estimate QALYs

• Most commonly used in economic evaluations, but the EQ-5D is increasingly being used as a measure of population health status and is included in a number of population health surveys • When used to measure and compare health profiles or utilities

across sub-groups of the population, the results will be misleading if groups systematically differ in use of response categories

• Could the EQ-5D suffer from DIF like other SRH measures?

4

Application of anchoring vignettes to the EQ-5D-5L:a possible solution to reporting heterogeneity in PROMs5


Differential Item Functioning

τ4

τ3

τ2

τ1

No problems

Slight problems

Moderate problems

Severe problems

Unable to walk

Group 2

Und

erly

ing

late

nt h

ealth

sca

le fo

r mob

ility τ4

τ3

τ2

τ1

No problems

Slight problems

Moderate problems

Severe problems

Unable to walk

Group 1High mobility

Low mobility

Group 2’s mean health

Group 1’s mean health


Programme of research• ARC funded project (starting 2011) assessing reporting

behaviour/heterogeneity and it’s consequences• Focus on SRH (generic likert scale) in panel surveys• Addition of primary research looking at relatively new

phenomenon of anchoring vignettes• Limited research considering DIF in the EQ-5D, none using

anchoring vignette technique• Programme of research

• Necessary to design vignettes, given identifying assumptions• Explore feasibility of eliciting responses• Robustly test if can be used to adjust for DIF• Consider broader applications


Anchoring vignettes• In order to obtain any meaningful comparison between the health

of groups 1 and 2 it is essential to adjust for DIF • Anchoring vignettes (King et al. 2004) can be used to adjust for

DIF • Previously been used to address DIF in political efficacy,

job/income/life satisfaction, general/specific health measures• Vignette - a brief health description of a hypothetical individual• Respondents are asked to rate the health state described by the

vignette using the same ordered categories they use to rate their own health

• Since the actual level of health of the people in the vignettes is the same for all respondents, the variation in ratings can be used to identify and correct for DIF


Anchoring vignettes• Example of a vignette for the mobility domain:

Belinda walks for one or two kilometres and climbs three flights of stairs every day without tiring.

Select the one option that best describes Belinda’s mobility:

She has no problems with walking around She has slight problems with walking around She has moderate problems with walking around She has severe problems with walking around She is unable to walk around


Anchoring vignettes• Typically, a series of vignettes are presented for each health

construct of interest, at varying levels of severity• Suppose we give groups 1 and 2 two vignettes to rate, of differing

severity:• Vignette 1 – limited problems in walking around• Vignette 2 – more problems in walking around

10


Anchoring vignettes

τ4

τ3

τ2

τ1

No problems

Slight problems

Moderate problems

Severe problems

Unable to walk

Group 2

Und

erly

ing

late

nt h

ealth

sca

le fo

r mob

ility

Vignette 2

Vignette 1

High mobility

Low mobility

τ4

τ3

τ2

τ1

No problems

Slight problems

Moderate problems

Severe problems

Unable to walk

Group 1


Necessary assumptions• Vignette equivalence (VE) holds if all respondents interpret the

health states described by the vignettes in the same way and on the same uni-dimensional scale, aside from random error.• VE is demonstrated in the example above by the horizontal

dotted lines • Response consistency (RC) is where respondents rate the health

of the hypothetical people described in the vignettes in the same way or using the same underlying scale that they would rate their own health. • RC would be violated if, for example, respondents rated the

health described by the vignettes either more or less harshly than they did their own health


Stage 1: Qualitative assessment of RC• Initial ARC study question• Research questions:

• Is the rating of vignettes for the EQ5D-5L feasible?• How do the vignette ratings compare by version?• Informal test for VE – is the ordering of vignettes consistent

with global ordering?• Understand thought process when rating vignettes – do

respondents rate hypothetical individuals in the same way as themselves? Does RC hold? (qualitative perspective).


Methods – vignette development• Gary King has a library of vignettes

http://gking.harvard.edu/vign/eg/ • Used these where possible and amended according to EQ-5D

attributes mobility, self care, usual activities, pain, anxiety/depression

• Version A: 15 vignettes - single health dimensions. Asks EQ-5D-5L by health dimension

• Version B: 3 vignettes - combined health state. Asks EQ-5D-5L as a whole including the VAS

• Respondents asked to rate the health of people in the vignettes before rating their own health to help with priming

• Vignette names were gender specific• Respondents were instructed to assume the hypothetical people

were of the same age and background as themselves

http://gking.harvard.edu/vign/eg/

http://gking.harvard.edu/vign/eg/


Pluralistic research design• Online survey: socio-demographic questions, long term illnesses,

health seeking behaviour, objective health measures, vignettes and EQ-5D-5L (+ SAH).

• Randomisation of survey version (A or B) • Data collection: April to May 2012• Phase 1: Online survey + face-to-face interview

• Interview to assess survey (clarity of instructions, wording & formatting) and feasibility of vignette task (clarity of the vignettes, level of concentration required, and thought processes).

• Phase 2: Online survey only• Additional questions on thoughts during vignette rating task

• Subjects: staff, students and people from a database of past research participants recruited through Monash University online newsletter and emails.


Results – feasibility• Interview feedback:

• Survey was straight forward and easy• Vignettes were easy to understand and the descriptions

seemed real and imaginable.• 3 younger respondents (aged 18-24) found some scenarios

difficult to imagine for someone their age.• Version A: one respondent noted the difficulty in rating a

person’s health “…without any other background or other knowledge of other aspects of their health” (Male, 30-34). – Highlights trade-off between simplicity of vignettes and lack

of context in a single health dimension description.• Version B was equally easy to understand as A• But, version B required more concentration than A


Response consistency • Did you assume the people in the vignettes were of the same age

and background as yourself? • Total sample, yes = 69%

• Interview: If no, why? • “When I read someone more disabled than myself I thought

they were possibly older and if less disabled, possibly younger”. (Male, 55-59, VA).

• “Most of them seemed older than me. I probably don’t see many people with those symptoms my age”. (Male, 18-24, VA)


Response consistency• Did you imagine yourself in the health state of the people in the

vignettes (at least for some of them)? • Online only: yes = 77%• Many in interview also demonstrated this. For example:

• “I pictured myself in that position. It’s easier to judge whether something is bad or not if it happens to you.” (Male, 18-24, VA).

• Others in interview took an external view. For example:• “I didn’t think of myself as them – I thought they were another

person. I rated myself quite separately from the vignettes” (Female, 25-29, VB)

• “I was trying to think of a view that a medical or paramedic person would put on it.” (Male, 70+, VA)


Response consistency • Did you rate your own health on the same scale as the

hypothetical individuals?• Online only: 39% strongly agree; 39% somewhat agree; 15%

disagree; 6% unsure• More people in version B strongly agree (50%) than in version A

(29%). • Suggests describing vignette as a whole health state rather than

as independent health dimensions does a better job at encouraging response consistency.

• Combined responses (interview and online only) suggest 37% demonstrated response consistency (28% for version A, 46% for version B).


Stage 1: Summary• Evidence that vignettes for the EQ-5D-5L are feasible• Suggested improvements required in the wording in order to

improve response consistency• Health states age neutral• … imagine yourself …

• Several avenues to explore in future work

• Au and Lorgelly (2014) Anchoring vignettes for health comparisons: an analysis of response consistency. QoLR.

• Knott et al (2016) Response scale heterogeneity in the EQ-5D. Health Economics


Stage 2: Quantitative exploration• Second ARC study question plus BankWest study• Research questions:

• Can the anchoring vignette approach be used to identify DIF in the EQ-5D-5L?

• Does it pass ‘strong’ tests for RC and VE?• What is the impact of DIF on inter-group comparisons?


Data• Two online surveys of a sample of representative Australian

residents, recruited via a survey panel company (April 2014 and Aug/Sept 2015)

• First survey compared versions A and B (and priming effect), second only used version B, analysis focuses on version B vignettes, of which their were two

• Total n=4,095

22


Vignette 1• REBECCA/ROB is able to walk distances of up to 500 metres

without any problems but feels puffed and tired after walking one kilometre or walking up more than one flight of stairs. She/he is able to wash, dress and groom her/himself, but it requires some effort due to an injury from an accident one year ago. Her/his injury causes her/him to stay home from work or social activities about once a month. Rebecca/Rob feels some stiffness and pain in her/his right shoulder most days however her/his symptoms are usually relieved with low doses of medication, stretching and massage. She/he feels happy and enjoys things like hobbies or social activities around half of the time. The rest of the time she/he worries about the future and feels depressed a couple of days a month.


Vignette 2• CHRISTINE/CHRIS is suffering from an injury which causes her/him

a considerable amount of pain. She/he can walk up to a distance of 50 metres without any assistance, but struggles to walk up and down stairs. She/he can wash her/his face and comb her/his hair, but has difficulty washing her/his whole body without help. She/he needs assistance with putting clothes on the lower half of her/his body. Since having the injury Christine/Chris can no longer cook or clean the house her/himself, and needs someone to do the grocery shopping for her/him. The injury has caused her/him to experience back pain every day and she/he is unable to stand or sit for more than half an hour at a time. She/he is depressed nearly every day and feels hopeless. She/he also has a low self-esteem and feels that she/he has become a burden.


Data• Standard socio-demographic questions, self reports of own health

(EQ-5D-5L and SRH), vignettes, additional health questions• First survey included ‘objective’ health measures to test RC• Considered heterogeneity in following groups

• Age, gender, education and country of birth


Econometric Analysis• Hierarchical ordered probit (HOPIT) model

• Extension of OP but allows for variation in the inter-category thresholds by modelling them as a function of covariates

• We estimated five separate HOPITs for each domain of the EQ-5D-5L

• DIF is tested for using LR that restrict the threshold covariates to be zero

• Impact of DIF on EQ-5D-5L indices assessed by simulating data given distribution of latent health using the estimated parameters of the mean function of the HOPIT and the characteristics of each individual, apply the predicted thresholds at sample means of the covariates

• EQ-5D-5L values from Australian DCE (Norman et al, 2013)


Testing the assumptions• Bago d’Uva et al (2011) developed tests for VE and RC• RC test based on objective measures

• Inter-category thresholds should be the same for across the health and the vignette equations

• VE tests that no systematic difference in perceptions ofhte health states of the vignette persons• Interactions between individual characteristics and vignette

severity (for all but one vignette)


Results regarding assumptions

Degrees of freedom χ2 test statistic p-value

Response consistency Mobility 13 15.12 0.300 Self-care 13 18.31 0.146 Usual activities 13 8.14 0.835 Pain/discomfort 13 18.86 0.127 Anxiety/depression 13 19.44 0.110

Vignette equivalence Mobility 13 100.06 <0.001 Self-care 13 178.69 <0.001 Usual activities 13 170.03 <0.001 Pain/discomfort 13 241.63 <0.001 Anxiety/depression 13 172.44 <0.001


Vignette equivalence – age groups Degrees of freedom χ2 test statistic p-value

Age 20-34 Mobility 8 21.785 0.005 Self-care 8 65.791 <0.001 Usual activities 8 54.208 <0.001 Pain/discomfort 8 68.995 <0.001 Anxiety/depression 8 38.895 <0.001Age 35-44 Mobility 8 28.017 <0.001 Self-care 8 75.826 <0.001 Usual activities 8 56.664 <0.001 Pain/discomfort 8 79.472 <0.001 Anxiety/depression 8 45.601 <0.001Age 45-54 Mobility 8 67.563 <0.001 Self-care 8 110.842 <0.001 Usual activities 8 93.543 <0.001 Pain/discomfort 8 129.923 <0.001 Anxiety/depression 8 82.278 <0.001Age 55-65 Mobility 8 8.296 0.600 Self-care 8 9.427 0.492 Usual activities 8 11.675 0.307 Pain/discomfort 8 15.076 0.129 Anxiety/depression 8 24.061 0.007


Evidence of DIF for 55-65 (N=914)

Mobility Self-care Usual

activitiesPain/

discomfortAnxiety/

depression

LR test statistic 94.82 57.71 64.73 74.89 74.57

p-value 0.000 0.043 0.008 0.001 0.001

Degrees of freedom 40 40 40 40 40


Mobility Self care Usual activities

Pain/ Discomfort

Anxiety/ Depression

Female -0.165* -0.005 0.059 0.131*** 0.035(0.087) (0.052) (0.046) (0.050) (0.047)

Education (base category low)Medium -0.128 -0.088 0.014 -0.109* 0.047

(0.095) (0.061) (0.054) (0.057) (0.055)High -0.251** -0.168** -0.073 -0.142** -0.03

(0.107) (0.067) (0.057) (0.061) (0.058)Country of Birth (ref. Australia)Oth English speaking 0.099 0.125 -0.097 0.188** 0.119

(0.160) (0.095) (0.094) (0.089) (0.088)Asia 0.168 0.037 0.025 0.055 0.02

(0.105) (0.073) (0.065) (0.070) (0.066)Other 0.399** 0.159 0.142 0.201 0.118

(0.179) (0.133) (0.121) (0.126) (0.123)Marital status (ref. never married)Married/de facto -0.335*** -0.165** -0.005 -0.063 0.008

(0.103) (0.074) (0.070) (0.074) (0.073)Divorced/widowed -0.259** -0.123 0.066 -0.034 0.092

(0.123) (0.084) (0.079) (0.084) (0.081)Employment status (ref. NILF)Employed -0.009 -0.032 -0.074 -0.044 -0.087*

(0.084) (0.053) (0.048) (0.051) (0.048)Unemployed -0.333 -0.127 0.018 -0.023 -0.269**

(0.265) (0.128) (0.102) (0.113) (0.120)

Res

ults

for t

he fi

rst t

hres

hold

–

betw

een

extre

me

and

seve

re


DIF adjusted indices

Female Male

Low ed

ucati

on

Med ed

ucatio

n

High ed

ucatio

n

Born A

ustra

lia

Other E

ngl. S

p.Asia

Other

Married

/de fa

cto

Divorce

d/wido

wed

Never

married

Employe

d

Unemplo

yed

NILF/re

tired

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Index based on self-reports DIF-adjusted index

EQ-5

D In

dex

Female Male

Low ed

ucati

on

Med ed

ucatio

n

High ed

ucatio

n

Born A

ustra

lia

Other E

ngl. S

p.Asia

Other

Married

/de fa

cto

Divorce

d/wido

wed

Never

married

Employe

d

Unemplo

yed

NILF/re

tired

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Index based on self-reports DIF-adjusted index

EQ-5

D In

dex

Diffe

rence = 0.04

9

Diffe

rence = 0.09

5


Stage 2: Summary• Vignettes can be used identify DIF in the EQ-5D-5L (at least in

certain age groups)• Failure to adjust for DIF can lead to conclusions that are

misleading• Further work is needed to achieve vignette equivalence

• Earlier work increased RC (rate vignettes as if it were themselves, imagine person of similar age, avoided age specific diseases) but did this come at the expense of VE?

• Further work required to understand what this means for economic evaluations

• Knott et al (2016) Differential item functioning in the EQ-5D: an exploratory analysis using anchoring vignettes. Working paper


Stage 3: external application• Funded by Monash Faculty of Business grant• Often voiced concern is that the inclusion of vignettes in studies,

particularly clinical trials is not costless• Application of vignettes has typically been limited to datasets

where they are collected• Recent work (Harris et al, 2015) showed that it is possible to

correct for DIF using vignette responses collected externally to the main dataset, using SRH and HILDA

• Research question• Is it possible to adjust for DIF in the EQ-5D within a dataset that

did not include vignettes?• If it’s possible, what effect does it have?


Data sources• Vignette data as before• Multi Instrument Comparison (MIC) study recruited 8,000+

respondents in 6 countries to complete 6 of the most common MAUIs, including the EQ-5D-5L (Richardson et al, 2012)

• Targeting of morbidity groups and the healthy public• Wave 1 Australian sample N=1,341• Given RC and VE only exist in 55+ age group, MIC external

sample N=656 and vignette sample N=914• Key issue: how similar are the two groups, how applicable will the

vignette responses in the external data be to the MIC respondents? Is the DIF problem in this sample the same as in the other?


Descriptive statisticsMIC sample (N = 656) Vignettes sample (N = 914)Mean St. Dev. Mean St. Dev.

Female 0.447 0.498 0.497 0.500Male 0.553 0.498 0.503 0.500Aged 55-64 0.566 0.496 1 -Aged 65+ 0.435 0.496 0 -University degree (high) 0.349 0.477 0.309 0.462Certificate/diploma (medium) 0.245 0.431 0.330 0.471High school or less (low) 0.405 0.491 0.361 0.481 Born in Australia 0.686 0.465 0.756 0.430Employed 0.244 0.430 0.528 0.500Married 0.654 0.476 0.650 0.477Asthma 0.061 0.240 0.166 0.373Cancer 0.200 0.400 0.101 0.301Respiratory 0.093 0.291 0.067 0.250Depression 0.067 0.250 0.318 0.466Diabetes 0.180 0.384 0.149 0.356


Gender Education group Migrant statusRaw DIF-adjusted Raw DIF-adjusted Raw DIF-adjusted

Male Female Male Female High Med Low High Med Low Born Aus Migrant

Born Aus Migrant

Mobility

None 0.474 0.468 0.755 0.754 0.550 0.447 0.417 0.873 0.776 0.639 0.456 0.505 0.740 0.786Slight 0.275 0.266 0.240 0.246 0.231 0.248 0.320 0.127 0.224 0.353 0.271 0.272 0.258 0.209Mod 0.174 0.191 0.006 0 0.131 0.224 0.199 0 0 0.008 0.191 0.160 0.002 0.005Severe 0.069 0.072 0 0 0.074 0.075 0.064 0 0 0 0.073 0.063 0 0Unable 0.008 0.003 0 0 0.013 0.006 0 0 0 0 0.009 0 0 0

Self-care

None 0.774 0.860 0.810 0.771 0.878 0.745 0.797 0.904 0.683 0.763 0.793 0.854 0.789 0.801Slight 0.135 0.089 0.185 0.218 0.079 0.174 0.109 0.096 0.317 0.218 0.127 0.087 0.202 0.194Mod 0.074 0.041 0.006 0.010 0.035 0.068 0.075 0 0 0.019 0.064 0.049 0.009 0.005Severe 0.014 0.010 0 0 0.009 0.006 0.019 0 0 0 0.016 0.005 0 0Unable 0.003 0 0 0 0 0.006 0 0 0 0 0.000 0.005 0 0

Usual activities

None 0.499 0.468 0.501 0.406 0.537 0.466 0.451 0.624 0.342 0.387 0.500 0.451 0.436 0.510Slight 0.295 0.331 0.468 0.560 0.301 0.298 0.327 0.367 0.621 0.564 0.284 0.369 0.527 0.471Mod 0.140 0.157 0.030 0.034 0.122 0.161 0.162 0.009 0.037 0.049 0.151 0.141 0.038 0.019Severe 0.058 0.031 0 0 0.035 0.050 0.053 0 0 0 0.053 0.029 0 0Unable 0.008 0.014 0 0 0.004 0.025 0.008 0 0 0 0.011 0.010 0 0

Pain/ discomfort

None 0.201 0.188 0.088 0.003 0.227 0.217 0.154 0.135 0.012 0 0.182 0.223 0.036 0.083Slight 0.408 0.372 0.826 0.689 0.450 0.317 0.387 0.769 0.820 0.729 0.387 0.403 0.787 0.718Mod 0.267 0.294 0.085 0.307 0.227 0.304 0.308 0.096 0.168 0.271 0.289 0.257 0.178 0.199Severe 0.113 0.116 0 0 0.070 0.149 0.132 0 0 0 0.122 0.097 0 0Extreme 0.011 0.031 0 0 0.026 0.012 0.019 0 0 0 0.020 0.019 0 0

Anxiety/ depression

None 0.526 0.546 0.625 0.642 0.550 0.565 0.504 0.668 0.615 0.613 0.529 0.549 0.602 0.699Slight 0.273 0.222 0.295 0.311 0.262 0.230 0.252 0.271 0.323 0.316 0.231 0.291 0.320 0.262Mod 0.124 0.184 0.061 0.038 0.122 0.149 0.177 0.061 0.031 0.053 0.164 0.121 0.058 0.034Severe 0.058 0.038 0.019 0.010 0.048 0.037 0.056 0 0.031 0.019 0.056 0.034 0.020 0.005Extreme 0.019 0.010 0 0 0.017 0.019 0.011 0 0 0 0.020 0.005 0 0


DIF adjustment – group differences

Male - Female High educ - Low educ Migrant - Born Aus Employed - Not employed

Married - Alone Aged 65 plus - Under 65

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

-0.004

0.0539999999999999

0.0379999999999999

0.093

0.0650000000000001

0.08

Unadjusted scores DIF-adjusted scores

Diff

eren

ce in

EQ

-5D

-5L

indi

ces

Male - Female High educ - Low educ Migrant - Born Aus Employed - Not employed

Married - Alone Aged 65 plus - Under 65

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.016

0.079

0.037

0.141

0.0960000000000001 0.097

Unadjusted scores DIF-adjusted scores

Diff

eren

ce in

EQ

-5D

-5L

indi

ces

MID=0.074


Stage 3: summary• It is possible to correct for DIF using responses to anchoring

vignettes that are collected externally to the main dataset of interest

• Resulting QALY measures can be considered comparable across different population groups• Assuming reporting behaviour in each sample is the same

• Knott & Lorgelly (2016) Adjusting for differential item functioning in the EQ-5D using externally-collected vignettes. HESG Paper (Gran Canaria)


Where to next?• Better understanding of the vignette equivalence failure issue

• Will there always be a trade-off with response consistency?• Is there value in exploring DIF cross-culturally?

• Multi-national clinical trials, often apply one country’s tariff as if all respondents are within that country

• Is the external adjustment as good as (or a close substitute for) collecting them within a study?

• What does this mean for economic evaluations and the decisions they inform? • Could response behaviour change over time?

the application of anchoring vignettes to the eq-5d-5l:a possible solution to reporting...

Presentations & Public Speaking