searching for gold standards

Why David Healy is WrongWhy David Healy is Wrong

WHY RANDOMISE?WHY RANDOMISE?

Why RCTs are the best…I really mean it… cross my heart

and hope to die….

Why RCTs are the best…I really mean it… cross my heart

and hope to die….

Insert Picture of Sliced BreadInsert Picture of Sliced Bread

Why do we bother with RCTs?Why do we bother with RCTs?

Why bother to randomise?Why bother to randomise?

Slater and Sargeant, 1963Slater and Sargeant, 1963

• If they fail to demonstrate any differences between a placebo and a drug which everybody knows to be effective, this means only that the work has not been done well enough

• If they fail to demonstrate any differences between a placebo and a drug which everybody knows to be effective, this means only that the work has not been done well enough

We are never going to learn how to treat depressions properly from double blind sampling in an MRC statistician’s office

We are never going to learn how to treat depressions properly from double blind sampling in an MRC statistician’s office

Sargeant, BMJ, 1963

If you can believe fervently in your treatment, eve n though controlled tests show it is quite useless, then you r results are much better, your patients are much better and your income is much better too. I believe this accounts for the remarkable success of some of the less gifted, but more credulous members of our profession, and also for t he violent dislike of statistics and controlled test w hich fashionable and successful doctors are prone to dis play

If you can believe fervently in your treatment, eve n though controlled tests show it is quite useless, then you r results are much better, your patients are much better and your income is much better too. I believe this accounts for the remarkable success of some of the less gifted, but more credulous members of our profession, and also for t he violent dislike of statistics and controlled test w hich fashionable and successful doctors are prone to dis play

Richard Asher, 1972

So what is unique about the randomised controlled trial?So what is unique about the randomised controlled trial?

So what is unique about the RCT?So what is unique about the RCT?

• Is it having a control group?• Is it having a control group?

So what is unique about the RCT?So what is unique about the RCT?

• Is it having a control group?• Is it the blindness?

• Is it having a control group?• Is it the blindness?

So what is unique about RCT?So what is unique about RCT?

• Is it having a control group?• Is it the blindness?• Is it controlling for the effects of chance?

• Is it having a control group?• Is it the blindness?• Is it controlling for the effects of chance?

Have the Samaritans Reduced the Suicide Rate?

RANDOMISATION IS THE ONLY WAY TO CONTROL

CONFOUNDERS

RANDOMISATION IS THE ONLY WAY TO CONTROL

CONFOUNDERS

And what happens if you don’t randomise?

And what happens if you don’t randomise?

YOU GET THE WRONG ANSWER

YOU GET THE WRONG ANSWER

Randomization: the Gold StandardRandomization: the Gold Standard

Randomization Non randomassignment

Cheatingdifficult (%)

Cheating easy(%)

Proportion ofstudies withp < 0.05

8.8 24.4 58.1

Randomization Non randomassignment

Cheatingdifficult (%)

Cheating easy(%)

Proportion ofstudies withp < 0.05

8.8 24.4 58.1

Chalmers et al, NEJM, 1983

Boys behaving badlyBoys behaving badly

Cambridge Somerville Youth Study

Cambridge Somerville Youth Study

• 750 delinquent boys• randomised• “sustained friendly counselling”• Intervention lasted five years

• 750 delinquent boys• randomised• “sustained friendly counselling”• Intervention lasted five years

5, 10, 20 and 30 years later, one group had more

5, 10, 20 and 30 years later, one group had more

• criminal convictions• unemployment• premature death• alcoholism• Serious mental illness

• criminal convictions• unemployment• premature death• alcoholism• Serious mental illness

McCord, 1978

Men behaving badlyMen behaving badly

“FORWARD PSYCHIATRY” – The PIE Principles

“FORWARD PSYCHIATRY” – The PIE Principles

• Proximity

• Immediacy

• Expectancy

• Proximity

• Immediacy

• Expectancy

RFA Canberra

What else can we do about psychological trauma?

What else can we do about psychological trauma?

What is debriefing?What is debriefing?

• Introduction• Describing traumatic incident• Psychological reactions to incident• Emotional processing/catharsis• Identifying symptoms• Teaching - symptoms are normal• Re entry/closure

• Introduction• Describing traumatic incident• Psychological reactions to incident• Emotional processing/catharsis• Identifying symptoms• Teaching - symptoms are normal• Re entry/closure

Debriefing has been used for….Debriefing has been used for….

• Police after shooting incidents• Sailors after maritime collisions• Red Cross personnel• Medical students whose patients have

died• Families whose children are

undergoing transplants• Soldiers on grave registration duties• Train drivers who have witnessed

suicides• Jurors in murder trials• Air Force personnel on whose Base

there has been a fatal accident• Casualty staff after trauma incidents

• Police after shooting incidents• Sailors after maritime collisions• Red Cross personnel• Medical students whose patients have

died• Families whose children are

undergoing transplants• Soldiers on grave registration duties• Train drivers who have witnessed

suicides• Jurors in murder trials• Air Force personnel on whose Base

there has been a fatal accident• Casualty staff after trauma incidents

• Burns victims• accident victims• Hospital staff after failed

resuscitations• Nurses in cancer care• patients who have recovered from

cancer• Rape victims• Rescue workers after any natural

disaster• The entire New York Fire Department• The entire New York Police

Department• Children attending a school where a

pupil has died• Ward staff where a suicide has

occurred

• Burns victims• accident victims• Hospital staff after failed

resuscitations• Nurses in cancer care• patients who have recovered from

cancer• Rape victims• Rescue workers after any natural

disaster• The entire New York Fire Department• The entire New York Police

Department• Children attending a school where a

pupil has died• Ward staff where a suicide has

occurred

“the experiences of 700 CISM teams in more than 40,000 debriefings cannot be ignored,

especially so when the overwhelming majority of reports are extremely positive”

“the experiences of 700 CISM teams in more than 40,000 debriefings cannot be ignored,

especially so when the overwhelming majority of reports are extremely positive”

Mitchell & Everly, 2003

Does it work? How dare you even ask?

“numerous studies have already shown positive results….proves the clinical

effectiveness beyond reasonable doubt”

“numerous studies have already shown positive results….proves the clinical

effectiveness beyond reasonable doubt”

Single session psychological debriefing definitely does not

work…

Single session psychological debriefing definitely does not

work……..and increases the risk of PTSD…..and increases the risk of PTSD

Rose, Wessely, Bisson – Cochrane Review 2003: Emmerinket al, Lancet, 2002

So why doesn’t debriefing work?

So why doesn’t debriefing work?

• Not everyone wants or needs to talk• “retraumatisation”• Interferes with natural recovery processes• Suggestion• Professionalisation of distress• Impedes people talking to who they want, when

they want

• Not everyone wants or needs to talk• “retraumatisation”• Interferes with natural recovery processes• Suggestion• Professionalisation of distress• Impedes people talking to who they want, when

they want

Everyone behaving badlyEveryone behaving badly

This can’t be rightThis can’t be right

Graded Exercise Therapy (GET) for CFS: (Fulcher & White, BMJ 1997)

Graded Exercise Therapy (GET) for CFS: (Fulcher & White, BMJ 1997)

• Setting: National Sports Centre

• Design: RCT

• Treatment: 12 weeks Graded Exercise Therapy

• Control Flexibility exercises

• Patients: 66 CFS patients

• Results: GET superior

• Conclusions: GET safe and effective

• Setting: National Sports Centre

• Design: RCT

• Treatment: 12 weeks Graded Exercise Therapy

• Control Flexibility exercises

• Patients: 66 CFS patients

• Results: GET superior

• Conclusions: GET safe and effective

Percentage improved with GETPercentage improved with GET

0

10

20

30

40

50

60

70

Fulcher Wearden Powell

GETControl

Percentage improved with GETPercentage improved with GET

0

10

20

30

40

50

60

Moss-Morris

Wallman

GETControl

What is CBT?What is CBT?

• Collaborative• Identifies cognitive barriers• Sets behavioural targets, not exercise• Emphasises predictability/consistency• Does relapse prevention

• Collaborative• Identifies cognitive barriers• Sets behavioural targets, not exercise• Emphasises predictability/consistency• Does relapse prevention

CBT for CFS: the Oxford RCT (Sharpe et al, BMJ 1997)

CBT for CFS: the Oxford RCT (Sharpe et al, BMJ 1997)

• Setting: Medical out patient clinic

• Treatment: 16 sessions CBT given by skilled therapists

• Patients: 60 CFS patients

• Results: CBT superior at 12 months on symptoms and function

• Setting: Medical out patient clinic

• Treatment: 16 sessions CBT given by skilled therapists

• Patients: 60 CFS patients

• Results: CBT superior at 12 months on symptoms and function

1996

CBT for CFS: The King’s RCT (Deale et al, 1997)

CBT for CFS: The King’s RCT (Deale et al, 1997)

• Setting: CFS Clinic

• Treatment: 12 sessions CBT

• Control 12 sessions relaxation

• Patients: 72 patients with CFS

• Results: CBT better on symptoms and disability

• Five year follow up Differences less, but still pr esent

• Setting: CFS Clinic

• Treatment: 12 sessions CBT

• Control 12 sessions relaxation

• Patients: 72 patients with CFS

• Results: CBT better on symptoms and disability

• Five year follow up Differences less, but still pr esent

Outcome at 6 months follow upOutcome at 6 months follow up CBT Relaxation

Much improved (Increase of 50 or more on MOS score)

16 (53%) 3(10%)

Improved 8 (27%) 11 (37%)

Unimproved (incl drop outs)

6 (20%) 16 (53%)

CBT Relaxation

Much improved (Increase of 50 or more on MOS score)

16 (53%) 3(10%)

Improved 8 (27%) 11 (37%)

Unimproved (incl drop outs)

6 (20%) 16 (53%)

CBT for CFS: a multi-centre RCT (Prins et al Lancet 2001)CBT for CFS: a multi-centre RCT (Prins et al Lancet 2001)

• Setting: Medical out patient clinic

• Treatment: CBT given by newly trained therapists; Support group or Natural course.

• Patients: 278 patients with CFS

• Results: CBT group: 35% improved on fatigue; 49% physical functioning; 50% self rated improvement.

• Conclusions: CBT was more effective than guided support and natural course with non specialist therapists.

• Setting: Medical out patient clinic

• Treatment: CBT given by newly trained therapists; Support group or Natural course.

• Patients: 278 patients with CFS

• Results: CBT group: 35% improved on fatigue; 49% physical functioning; 50% self rated improvement.

• Conclusions: CBT was more effective than guided support and natural course with non specialist therapists.

RCT of patient education to encourage graded exerci se in CFS

(Powell et al BMJ 2001)

RCT of patient education to encourage graded exerci se in CFS

(Powell et al BMJ 2001)

• Setting: Medical out patient clinic

• Patients: 148 patients with CFS

• Treatment: standard medical care; GET; telephone advice + GET: face to face advice + GET

• Results: 69% improved in the intervention groups 6% of controls improved

• 4 years later Improvements maintained

• Setting: Medical out patient clinic

• Patients: 148 patients with CFS

• Treatment: standard medical care; GET; telephone advice + GET: face to face advice + GET

• Results: 69% improved in the intervention groups 6% of controls improved

• 4 years later Improvements maintained

Routine Clinical Practice (Chalder et al)

• Setting: King’s

• Design: Prospective uncontrolled study

• Treatment: Routine CBT

• Patients: 293 patients with CFS

• Results: 58% rated themselves as very/much or much better; 26% were a little better; 16% were the same or worse on global outcome

• Conclusions: Outcomes only slightly less than in the RCTS

Game, set and match ……… .Game, set and match ……… .

Action for ME Website(www.afme.org.uk)

Action for ME Website(www.afme.org.uk)

Q. What treatments are available?

A. Research has proven two therapies bring relief for many people with ME…………...

Q. What treatments are available?

A. Research has proven two therapies bring relief for many people with ME…………...

Action for ME Website(www.afme.org.uk)

Action for ME Website(www.afme.org.uk)

…….magnesium injections and the dietary supplement Efamol (a

combination of evening primrose oil and marine oil)

…….magnesium injections and the dietary supplement Efamol (a

combination of evening primrose oil and marine oil)

COMP ARISO N OF T R EATM ENT APP RO A CH ES

GRO UPS RESEAR C H sample he lped m ade no m adeSEVERELY AFFEC TE D SURVEY diffe rence w orse

PACI NG GRO UPS 257 88% 9% 3%

SEVERELY AFFEC TE D 2,180 89% 9% 1%

GR AD ED EX ERCIS E TH ER APY GRO UPS 209 39% 22% 39%

SEVERELY AFFEC TE D 1,214 34% 15% 50%

COG NITIVE B EH AVIOU R AL TH ER APY GRO UPS 113 55% 32% 13%

SEVERELY AFFEC TE D 285 7% 67% 26%

Action for ME Self help Group Survey

Psychiatry is the dustbin of the medical profession

(Clare Francis, 1988)

Psychiatry is the dustbin of the medical profession

(Clare Francis, 1988)

Psychiatry is opinion dressed up as science, stupid and hypocritical

(Clare Francis, 1996)

Psychiatry is opinion dressed up as science, stupid and hypocritical

(Clare Francis, 1996)

“People with postviral fatigue syndrome often have to put with a lot of disbelief-there were many doctors who diagnosed this as a psychiatric disorder although on the whole

it is taken much more seriously now”

“People with postviral fatigue syndrome often have to put with a lot of disbelief-there were many doctors who diagnosed this as a psychiatric disorder although on the whole

it is taken much more seriously now”

“Watchdog to look into ME resources”: Dundee Courier and Advertiser, 11.11.1994

“ME is an imaginary illness, for which the best treatment is

psychiatric”

“ME is an imaginary illness, for which the best treatment is

psychiatric”

Unnamed physician, cited in Steincamp, 1989

Without the RCT we would never have been able to

improve the care of patients

Without the RCT we would never have been able to

improve the care of patients

People don’t do clinical trials in mental health because........

People don’t do clinical trials in mental health because........

• I know my treatment works• My treatment is too individual• The results are not generalisable

• I know my treatment works• My treatment is too individual• The results are not generalisable

What’s the real reason?What’s the real reason?

They are bloody difficultThey are bloody difficult

Problems with Psychiatry Trials: Our Fault

Problems with Psychiatry Trials: Our Fault

• Not big enough• Not good enough• Too many rating scales• Testing wrong kind of things

• Not big enough• Not good enough• Too many rating scales• Testing wrong kind of things

Cochrane Depression/Anxiety Group- Interventions

Cochrane Depression/Anxiety Group- Interventions

10%

23%

67%

Other

Psychotherapy

Pharmacotherapy

Problems with Psychiatry Trials: Not Our Fault

Problems with Psychiatry Trials: Not Our Fault

• Problems with follow up/drop outs• Fuzzy outcome measures• Large non specific effects• Complex interventions• Ethical dilemmas (capacity, coercion)• Bureaucratic nightmare which is getting worse

• Problems with follow up/drop outs• Fuzzy outcome measures• Large non specific effects• Complex interventions• Ethical dilemmas (capacity, coercion)• Bureaucratic nightmare which is getting worse

So what is the moral of the story?So what is the moral of the story?

When in doubt….When in doubt….

Randomise!!Randomise!!