Measuring worker productivity:

outcome or cost?

Dorcas Beaton, BScOT, PhDHealth Policy Management & Evaluation, University of Toronto,Mobility Program Clinical Research Unit, St Michael’s Hospital,

Institute for Work & Health.

CAN Workshop, November 2009

Perspective: OMERACT Outcome Measurement in Rheumatological Clinical

Trials (est. 1992)

Goal: to improve outcomes used in clinical trials Principle of consensus What to measure, and how Allow better communication across trials, across

stakeholders. Clinicians, consumers, researchers, regulators, industry (pharma)

Meets bi-annually informal networking meeting with work in between Work began being addressed at OMERACT 8 (2006)

Decision making – OMERACT Style Data driven decisions:

OMERACT Filter (Boers, J Rheum1998)

Truth – validity (content, validity) Feasibility – costs, ease Discrimination – between treatment and control arms

in a trial (reliability, responsiveness)

Evidence gathered, recommendation made Consensus vote for endorsement

Challenges and advances- so far! Challenges

Defining our target Contextual nature of work

Advances Available measures Methods to blend absenteeism and presenteeism

Challenge #1: Defining our targetPerspective: Workplace productivity = productivity at level of workplace

Number of cars off line Worker productivity = experience/contribution of the

individual worker Our target for arthritis research

Defining worker productivity Absenteeism = time not at work (but employed) Presenteeism = at-work productivity loss = loss efficiency or

effectiveness while on the job. Both important to arthritis research

Interest in Work in arthritis research As an outcome state

Off work / RTW (absenteeism) Amount of difficulty at work (at-work productivity

loss or presenteeism)

As an indicator of cost Cost of days off work +/- replacement workers

(absenteeism costs) Cost of at-work productivity loss (presenteeism

costs)

Target: the impact of arthritis on work at the level of the worker

Absenteeism Presenteeism/ At-work productivity loss

Outcome State YES YES

Cost estimate YES YES

Goal: to have solid indicators of each of these cells. Solid in terms of breadth & depth, measurement propertiesPossibly same measure for Cost and Outcome goals!

Challenge #2 Contextual nature of work Difficulty experienced with work is, in part, defined

by the job. Physical, mental, time demands, where work is done

Case scenario XX was having a lot of difficulty at work, working what he

felt was ~ 75% productivity, so had to leave ideal job and get a less demanding role where he could function without difficulty

Numbers say 75% 100% productivity. Is he “better”?

Context of work At an individual level we can sort out context

At a group level, numbers from our scales are on their own. Can we validly add together numbers for a group

mean with meaning? Can we interpret change?

Contextual factors to be considered Environment

Nature of job - Physical, control over tasks and pace Flexibility of workplace to accommodate Workplaces organizational practices around work

Individual factors adaptations and coping behaviours used to stay at work

(Gignac, 2005) Adjust time (inside and outside of work), use vacation, Get help from others, Modify activities to conserve energy, make things easier Anticipate and avoid problems (organize, pace, take breaks).

Solutions? I suggest we agree on four cells, whether we

wish to move towards a goal of same indicators across outcome & economic studies

Contextual factors need to be sorted out Funded for qualitative study – decision making

around absenteeism/presenteeism

Challenges Advances

Advance #1 Available Measures OMERACT 9

Absenteeism > days absent from work What about half days? Vacation days?

Break out groups, consensus votes What should be included in “absenteeism”

Absenteeism 94% OMERACTer’s agreed:

Days off sick due to arthritis Vacation days taken because of arthritis Part days/hours missed because of arthritis Change in number of hours worked per week Temporary work cessation (work disability/sick leave) Permanent work cessation due to arthritis

Consider further: All cause work absence Cessation of work due to choice or retirement

OMERACT 9, 2008

At-work productivity loss 21 different scales (Escorpizo, 2007; Beaton, 2009)

5 direct comparisons Different levels of work limitations (Lavigne,

2003) Low correlations between measures Little information in arthritis

HLQ, Osterhaus(Lavigne, 2003)

WLQ25, SPS13(Turpin, 2004)

WLQ25, WPSI(Ozminkowski, 2004)

QQ, HLQ(Meerding, 2005)

Osterhaus, QQ(Brouwer, 1999)

Head to head comparison Five measures in 250 persons with arthritis followed

over one year period (OA and IA) Validity and responsiveness of individual instruments

demonstrated (Beaton, in press) Predictive validity – predictive of work transitions

(change job, hours, duties because of arthritis) presented yesterday (Tang, ACR Meeting 2009)

Correlations between scales were low to moderate

Cost estimates – varied up to 7-fold depending on indicator of presenteeism (Zhang, under review)

clear differences between scales in quantifying work

Direct comparison: conceptual differencesDifferent concepts, different strengths

WALS – Work Activity Limitations Questionnaire (Gignac, 2004)

Difficulties in tasks of work. Based on HAQ type questions aimed at work

WIS – Work Instability Scale (Gilworth, 2003) Designed for RA, seems okay in OA, workers. Mismatch between person and job, tipping point to going off work.

WLQ – Work Limitations Questionnaire (Lerner, 2002) Proportion of time experiencing any level of work limitation.

Convertible to dollar cost estimate

Patient/consumer viewPatient votes on preferred scale:

0 5 10 15 20 25 30 35 40 45 50

Percent of sample

√

√

EWPS

SPS6

WLQ

WIS

WALS

Measurement properties

0 2 4 6 8 10 12 14

WALS

WIS

WLQ-sum

SPS6-R

EWPS

Discrimination between those hindered (46%) or not hindered (54%) at their work

0.5 0.7 0.9

WALS

WIS

WLQ-sum

SPS6-R

EWPS

Reliability

Validity

√

√

Picking up change over 1 year

0 0.2 0.4 0.6 0.8 1

WALS

WIS

WLQ-sum

SPS6-R

EWPS

Mean change/SD change Change in productivity

0 0.2 0.4 0.6 0.8 1

WALS

WIS

WLQ-sum

SPS6-R

EWPS

Mean change/SD change

Change in difficulty doing job

Instruments are sensitive to change in their intended target

√

The winner is….???

Preference Reliability Validity Responsiveness – difficulty

Responsiveness - productivity

EWPS √

SPS ~ ~

WLQ √ √

WIS √ √ √

WALS √ ~ √ √

Potential candidates for At work productivity – OMERACT Vote – not quite there

1) WALS2) RA-WIS

3) WLQ4) WPAI

5) WPS-RA6) HPQ

7) Other 8) No opinion 6.8

16

16

48

45

39

54

47

0 20 40 60 80 100

OMERACTEndorsement to continue

Percent endorsement to continue to work with this measure based on filter evidence

Advance #2: blending absenteeism and presenteeism Currently: measured separately.

At-work productivity = missing data for those absent from work in study

Losing important information – they are definitely at one end of at-work spectrum

Statistical solutions Hierarchical modelling

Working yes / no, if yes then level of at-work productivity

Structural equation models / path analyses

Issues People vary in course

Absent-present-absent Modelling longitudinal course

ML will impute data if we use presenteeism scores BUT this missing is MNAR

Clearly absenteeism has meaning Methods to blend dichotomous variable with

continuous Depeng Jiang’s idea…

Distribution of Work Status and At-Work Disability

Sample Descriptive Statistics and Individual Trajectories

Approach 1: 2 part Mixed Model

Advantages of two-part mixed model Able to jointly model the two outcomes (absenteeism and at-work

disability) simultaneously. Able to find the interesting associations between the trajectory of

two outcomes.

Problems of two-part mixed model Unable to model the missing mechanism. Those estimates from the two-part mixed model might be biased

if the dropout is not MAR. Unable to model the diversity of trajectories of absenteeism and at-

work disability.

Approach 2: 2 part mixture models Part 1 manages work status and missing due to

drop out

Part 2 allows cluster analysis Subgroups defined by intraindividual differences

in course!! More meaningful results

Two-part Mixture Model: Result

Two-part Mixture Model: Result

Note: The top blue line represents those sustained off-work having higher disability score (censored above the maximum possible score)

Two-part Mixture Model: Result Summary

Class Dropout rate

at 3-, 6-, 12M

At-work disability P-value for change in at-work disability

1 Sustained at work

(24.9%)

<10%, <10%, <15%

Decreased 20.7% over 12 months

0.006

2 High presenteeism

(9.6%)

54%, 75%,86% No significant change 0.92

3 Sustained off work

(25.9%)

<10%, <10%, <15%

Can not evaluate (no WLQ measures)

NA

4 Working at most time points (8.7%)

<10%, <10%, <15%

Decreased 44.3% over 12 months

<.001

5 Initially off work with RTW (11.8%)

46%, 60%, 76% Decreased 80.5% over 12 months

<.001

6 Initially at work, later off (19.2%)

<10%, <10%, <15%

Increased over 60% in 3 months

<.001

Dropout is non-ignorable missing. The likelihood of dropout, the level of presenteeism and level of at-work disability were significantly associated.

Structural equation modeling

At work difficulty

At work productivity loss

Absent from work

(WALS) (WLQ)

Person-job instability

(WIS)

Potential to build more complex models to explain the relationship between different measures

final thoughts… Which way to go with instrumentations

Measures are not equivalent

Still need to conquer contextual factors. Which are priority

New modelling methods can help us to blend absenteeism and presenteeism into one “ruler” for studies What do we do for smaller studies, clinical practice

Loss in productivity

Full time

status

Part time

Modified

Absenteeism-Sick-Vacation-Hours off …

Presenteeism-Ability to work-Effective hours lost

Job contex

t

Unemployed

Retired, age