How effectively do accident compensation regulators use research

evidence in policy and practice decision making?

Pauline Zardo PhD Candidate, Department of Epidemiology and Preventive Medicine, Monash University &

Institute of Safety, Compensation and Recovery Research. &

Dr Alex Collie Chief Research Officer, Institute of Safety, Compensation and Recovery Research & Associate

Department of Epidemiology and Preventive Medicine, Monash University

Overview • Background • Research Questions • Methodology • Results • Findings • Implications • Applications

Background • Growing interest in use of academic

research evidence to inform injury compensation and rehabilitation policy and practice decision making.

• Limited research on use of evidence in

Australian policy settings.

Definitions

• Evidence = Academic and scientific research evidence.

• Policy = Operational, organisational level policy.

The problem – by way of example (i)

1601 – Lancaster shows that lemon juice can eliminate scurvy amongst sailors. 1747 – Lind demonstrates the same for citrus juice. 1795 – British Navy first uses citrus juice for sailors (194 years after discovery). 1854 – British Board of Trade begins using citrus juice for sailors (253 years after discovery).

Research & Compensation Policy

• Great deal of research that could potentially be used to inform compensation policy

The problem – by way of example (ii) • 2003 – Roberts-Yates reports that aspects of the compensation

process in South Australian workers comp system lead to poor client satisfaction. (Roberts-Yates C. Disab & Rehab, 2003)

• 2004 – Strunin & Boden report similar findings in USA workers compensation. (Strunin L & Boden L. Am J Ind Med, 2004)

• 2005 – Sager & James report similar findings in NSW workers compensation. (Sager L & James C. Aust Occ Therapy Journal, 2005)

• 2006 – Lippel replicates and extends findings in Quebec workers compensation system. (Lippel K, Int J Psychiatry & Law, 2006)

• 2011 – Murgatroyd et al report similar findings in NSW motor accident compensation system. (Murgatroyd D et al, Injury Prevention 2011)

The problem – by way of example (iii)

•‘Compensation’ a significant predictor of health care utilisation (e.g. Harris et al. 2009)

•Long term outcomes (functional outcome, return to work) up to 2 times worse for people using compensation systems (Gabbe et al. 2007)

•Legislative change can have significant impact on health outcomes (Cameron et al 2008, Cassidy et al. 2000)

Source: Arno Akkermans 2011

Use of Research • Only 8 - 15% of research evidence contributes to

a change in policy or practice

• For evidence-informed practice, we need more practice informed evidence.

• Lack of evidence of intervention effectiveness.

Source: Best & Holmes 2010; Perrier (2011); Quimet (2011)

Barriers to use of evidence

Source: Lavis 2009; Mitton 2007: systematic review evidence

Facilitators to use of evidence

What research is likely to be used?

Source: WHO, 2004

Actionable messages

Synthesis of research evidence

Individual studies, articles and reports

Basic science, theoretical and methodological innovations

Mor

e lik

ely

to b

e us

ed

Context Specific Research • To increase or improve use of evidence

in a particular context, an in-depth understanding of that context is required.

• There is no published research on how evidence is used in Australian compensation policy.

• TAC provides compensation for the treatment and rehabilitation of thousands of Victorians injured in a transport accident.

- ~16,000 new injury claims per annum. - $900 million spent on health and rehabilitation services in 09/10.

Transport Accident Commission

Research Questions

1. What types of evidence are referred to in the TACs injury and rehabilitation compensation policies?

2. What is the purpose of reference to evidence in the TACs injury and rehabilitation compensation policies?

Hypotheses

• Evidence Based Medicine • Clinical Justification Framework in the TAC • Clinical Panel in the TAC

In Treatment, compared to Non-Treatment policies, it was expected that there would be: Hypothesis 1 Greater reference to academic research evidence Hypothesis 2 Greater reference to clinical judgement Hypothesis 3 Greater reference to external legislation

Methodology

• Quantitative content analysis of all (N=128) TAC injury and rehabilitation compensation policies as at 30 December 2010

Policy Type Purpose of Reference to

Evidence

Evidence Type

Treatment Support Policy Position Academic/Scientific Evidence

Non-Treatment Support Compensation Decision Making

TAC Legislation

External Legislation

TAC Policy

External Policy

Clinical/Medical Judgement

Costs Evidence

Other Evidence.

• 45 of the 128 policies were inter-rated to test the content analysis method.

• There was 90% agreement between Rater 1 and

Rater 2 for all categories and frequencies, except for the Costs Evidence category.

• The Costs Evidence category was subsequently

removed from further analyses.

Methodology

Results Categories (Overall) Frequency of

Reference to Evidence Median & Range Number of Policies

References to Evidence 2383

15.5 (0-67) 128

Treatment vs. Non-Treatment

References to Evidence in Treatment Policies

1169

16 (3-67) 65

References to Evidence in Non-Treatment policies

1214

13 (0-58) 63

Evidence to Support Policy vs. Evidence to Support Compensation References to Evidence to Support Policy position

600

3 (0-52) 125

References to Evidence to Support Compensation Decision Making

1783 11 (0-65) 120

Type of Evidence Median & Range

Ref’s per policy overall

Ref’s to Support Policy Position per policy

Ref’s to Support Compensation per policy

TAC Policy 6 (0-36)

1 (0-27) 4.5 (0-35)

Clinical/Medical Judgement

2.5 (0-28)

0 (0-3) 2 (0-28)

TAC Legislation 1 (0-11)

1 (0-11) 0 (0-4)

Other Evidence 1 (0-13)

0 (0-5) 1 (0-13)

External Policy 0 (0-24)

0 (0-15 0 (0-18)

External Legislation 0 (0-16)

0 (0-16) 0 (0-5)

Academic/ Scientific Research

0 (0-7) 0 (0-3) 0 (0-5)

Results

Type of Evidence Median & Range Significance Level

Ref’s per Non Treatment Policy

Ref’s per Treatment Policy

Difference between Treatment and Non-Treatment policy: Mann-Whitney U Test; p= 0.05

TAC Policy 6 (0-35) 6 (1-36) p = 0.785

Clinical/Medical Judgement

2 (0-25) 3 (0-28) p = 0.029*

TAC Legislation 1 (0-11) 1 (1-7) p = 0.017*

Other Evidence 1 (0-13) 1 (0-3) p = 0.279

External Policy 0 (0-24) 1 (0-17) p = 0.415

External Legislation 0 (0-16) 0 (0-2) p = 0.001*

Academic/ Scientific Research

0 (0-5) 0 (0-7) p = 0.000*

Results

Findings • TAC policy was the most common type of

evidence referenced. – Internal sources of evidence are heavily relied on to inform policy

decision making.

• Clinical judgement was the next most common type of evidence referenced.

– Raises the issue that not all clinical judgement is always evidence based hence the need for EBM and EIP.

Findings

• The purpose of most references to evidence was to support compensation decision making.

– TAC Policy and Clinical Judgement were the main types of

evidence used to support compensation decision making. – There were close to 3 times more references to support

compensation than references to support the policy position.

Findings

• Significantly greater reference to internal and external legislation in treatment policies.

- Development and implementation of TAC treatment policy may be more complex than non-treatment policy.

Findings • Academic research evidence was the least

common type of evidence referenced. – Total references= 50, 2.1% of total references to evidence. – Total policies that refer to scientific/academic evidence = 30,

23.43% of total TAC policies.

• There were significantly more references to academic evidence in treatment policy. – There were 40 references to evidence in 26 treatment policies;

compared to 10 references in 4 non-treatment policies.

Findings

• The most common reference to academic research evidence was ‘recent peer reviewed journal article’.

–6 policies referred to a specific piece of published research evidence. –1 policy referred to ‘clinical evidence’ and 1 policy referred to ‘clinical trials’ and ‘high quality evidence’. –No specific requirement for evidence of treatment effectiveness from high quality research designs such as: systematic reviews, RCT or cohort studies.

Implications • At present, there is an opportunity to

improve and increase use of research evidence to inform injury and rehabilitation compensation policy and practice. – Government decision making about what injury and rehabilitation

treatments should be compensated for could be better informed by high quality research evidence.

Implications • Where relevant, high quality, research

evidence exists this should be used to inform the development of compensation policy and practice.

– Whilst much relevant research exists, there remains a need for more compensation policy relevant research.

Implications

• Document content analysis is an effective way of identifying the types of evidence used in specific compensation policy and practice environments. – Research identifying how evidence is used in a particular policy

and practice context can inform design and implementation of interventions aimed at increasing use of evidence in that context.

21st November 2011 References to evidence in compensation policy: a TAC policy content analysis.

Applications

• This research has been used by the TAC to inform the design of a Policy Development Guideline document – Policies will now be required to consider scientific/academic

research evidence where relevant – Policies will now all be required to be updated on 3 yearly

basis – TAC had started this process and used the findings to

support and inform detail and direction

Questions???

Results Type of Evidence

Range Treatment Policy

Range Non-Treatment Policy

TAC Policy (1-36) (0-35)

Clinical/Medical Judgement

(0-28) (0-25)

TAC Legislation

(1-7) (0-11)

Other Evidence

(0-3) (0-13)

External Policy

(0-17) (0-24)

External Legislation

(0-2) (0-16)

Academic/ Scientific Research

(0-7) (0-5)

Type of Evidence

Median & Range Significance Level

Ref’s per policy overall

Ref’s to Support Policy Position per policy

Ref’s to Support Compensation per policy

Ref’s per Non Treatment Policy

Ref’s per Treatment Policy

Difference between Treatment and Non-Treatment policy: Mann-Whitney U Test; p= 0.05

TAC Policy 6 (0-36)

1 (0-27) 4.5 (0-35)

6 (0-35) 6 (1-36) p = 0.785

Clinical/Medical Judgement

2.5 (0-28)

0 (0-3) 2 (0-28) 2 (0-25) 3 (0-28) p = 0.029*

TAC Legislation 1 (0-11)

1 (0-11) 0 (0-4) 1 (0-11) 1 (1-7) p = 0.017*

Other Evidence 1 (0-13)

0 (0-5) 1 (0-13) 1 (0-13) 1 (0-3) p = 0.279

External Policy 0 (0-24)

0 (0-15 0 (0-18) 0 (0-24) 1 (0-17) p = 0.415

External Legislation

0 (0-16)

0 (0-16) 0 (0-5) 0 (0-16) 0 (0-2) p = 0.001*

Academic/ Scientific Research

0 (0-7)

0 (0-3) 0 (0-5) 0 (0-5) 0 (0-7) p = 0.000*

21st November 2011 References to evidence in compensation policy: a TAC policy content analysis.

Results

Barriers Facilitators

Individual: • Decreased researcher and research

user interaction • Low relevance: topic & timing • Accordance in values between

researchers & research users • Capacity to assess research • Mistrust • Negative attitudes to change

Individual: • Increased interaction • Ongoing collaboration • Research on values • Use of networks • Building trust between researchers and

users • Clearly defined roles

Organisational: • Practice culture unsupportive of research • Competing interests • Incentives for researchers • High staff turnover

Organisational: • Collaborative research partnerships • Sufficient resources • Authority to implement change • Capacity building initiatives • Readiness for change