how can we evaluate...

UNIVERSITY FRIBOURG, CH| Department of Psychology | Chair of Clinical Psychology and Psychotherapy| PD Dr. Peter Wilhelm, Spring 2018 1

How Can We Evaluate Interventions?

Peter Wilhelm

28.2.2018


Overview of Today’s Lecture

How can we show that treatments work?

Case studies

First outcome studies of psychotherapy

A critical review of the efficacy of psychotherapy

When we do not need a control group?

The first clinical trial: A historical example

Major threats to internal validity

What kind of control groups can we us to evaluate our treatment?


Case studies

„Little Albert“ (Watson & Rayner, 1920). Demonstrates how anxiety can be acquired via classical conditioning

„The wolfs man“ „Anna O.“ (Freud), central for the development of Psychoanalysis

Vivid description of the psychotherapeutic process and its principals

Example is usually highly convincing

Necessary for documentation of very rare cases.

Useful for the generation of hypotheses.


Problems with case studies

Based on clinical observations of therapist

Selective and biased perception (confirmation)

No proof that therapeutic intervention produce the observed effect.

Problem of generalizability


First outcome study of Psychoanalysis (Fenichel, 1930, cited from Knight, 1941, p. 440)

"apparently cured": definite and complete recovery, which could be attributed only to PA.

"much improved“: improvement was considerable and was attributable to the PA, but in which the analyst felt that a complete cure was lacking

"improved“: improvement was of lesser degree and might be attributable to other factors.


Eysenck’s (1952) review of early outcome studies

Eysenck (1952) reviewed available studies that were conducted to show that psychotherapy (PT) improves patients’ condition.

He included 19 studies (7000 neurotic patients) in his report

• 5 studies evaluating psychoanalytic treatments

• 14 studies evaluating eclectic treatments

Assessment: Therapists reported how much “neurotic” patients improved after treatment

Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16, 319–327.


Results of Eysenck’s review of early outcome studies

Results of 5 psychoanalytic treatment studies (Eysenck, 1952)


Results of Eysenck’s review of early outcome studies

Results across 14 eclectic treatment studies 64% improved

Results across 5 psychoanalytic treatment studies: 44% improved

• 1/3 broke off before 6 months. Eysenck considered them as failures


Eysenck’s base line estimates to contrast results of PT

Eysenck contrasted results with those of a long term study based on insurance data of 500 “neurotic” patients (Denker, 1946): Patients were unable to work for at least 3 months “treated by general practitioners …, and not by accredited specialists or

sanatoria” Treatment: “sedatives, tonics, suggestion, and reassurance” Criteria of "recovery“: a) return to work; b) no further or very slight

difficulties; (c) successful social adjustments.

after 1 year: 45% improved after 2 year: 72% improved after 5 year: 90% improved


Eysenck’s conclusion

Conclusion: Results of treatment studies don’t show efficacy After 2 years patients without PT are better of

than patients with PT

Eysenck’s alternative explanation: Observed improvement was not necessarily due

to treatment but could be due to spontaneous recovery


Eysenck’s (1952) summary of the evaluation of the state of research

“The figures fail to support the hypothesis that psychotherapy facilitates recovery from neurotic disorder.

In view of the many difficulties attending such actuarial comparisons, no further conclusions could be derived from the data

whose shortcomings highlight the necessity of properly planned and executed experimental studies into this important field.”

Eysenck’s review stimulated randomized controlled experiments to evaluate efficacy of psychotherapeutic treatments


When we do not need a control group?

• When the treatment effect operates against our common knowledge and is dramatic • E.g. effect of jumping out of a plane without parachute (based

on common knowledge) vs. with a parachute

The experiment of José Delgado “Matador’ With a Radio Stops Wired Bull” (NYT, 1965)


When we do not need a control group? The experiment of José Delgado

Delgado implanted a stimoceiver in the caudate nucleus of a bull. The next day Delgado challenged the bull like a matador and the bull attacked him. Delgado stopped the animal mid-way by using a remote control to send an electric impulse into the bulls brain. “The experiment […], was probably the most spectacular demonstration ever performed of the deliberate modification of animal behavior through external control of the brain.” New York Times, May 17, 1965, p. 1, 20


The world’s first controlled clinical trial The problem: Scurvy

Scurvy was the number one cause of mortality for sailors until the 19th century. More sailors died from scurvy, than from battles, or storms. Between 1500 and 1800 scurvy killed at least two million sailors.

Symptoms of Scurvy • “Their gums were rotten even to the very roots of their very teeth,

and their cheeks hard and swollen, • the teeth were loose neere ready to fall out ... • their breath a filthy savour. • The legs were feeble and so weak, that they were full of aches and

paines, with many blewish and reddish staines or spots, some broad and some small like flea-biting.” (William Clowes, who had served as a surgeon in Queen Elizabeth's fleet; Edzard & Singh, 2009, p. 14).


The world’s first controlled clinical trial Treatments for scurvy until the end of 18th century

Physicians proposed the following remedies: bloodletting mercury paste salt water vinegar sulphuric acid, hydrochloric acid, Moselle wine

burying the patient up to his neck in sand

hard labour

• doctors observed that scurvy was generally associated with lazyness. However, it was scurvy that caused sailors to be lazy, rather than laziness that made sailors vulnerable to scurvy.


The world’s first controlled clinical trial The desperate situation and the turning point

“Learned men around the world would fabricate arcane theories about the causes of scurvy and debate the merits of various cures, but nobody seemed capable of stopping the rot that was killing hundreds of thousands of sailors.” (Edzard & Singh, 2009, p. 16) 1747, came a major breakthrough. Naval surgeon James Lind (1716-1794) began his service on board of the HMS Salisbury. After several months, several sailors had scurvy. Lind did something new and unusual: He systematically observed what would happen if he treated the ill sailors in different ways? “His sharp brain and meticulous mind allowed him to discard fashion, prejudice, anecdote and hearsay, and instead he tackled the curse of scurvy with extreme logic and rationality.” (Edzard & Singh, 2009, p. 16)


The world’s first controlled clinical trial: Patients

Lind identified 12 sailors

with similarly serious symptoms of scurvy.

their hammocks were placed in the same portion of the ship

all received the same breakfast, lunch and dinner • biscuits, salted meat, dried fish

Lind was helping to guarantee a fair test, because all patients were

similarly sick,

similarly housed

similarly fed

Thus, Lind held potential confounding variables constant.


The world’s first controlled clinical trial: Treatments

He divided the sailors into six pairs and gave each pair a different treatment:

a quart of cider,

twenty-five drops of vitriol (sulphuric acid) three times a day,

two spoonfuls of vinegar three times a day,

half a pint of sea water a day,

medicinal paste consisting of garlic, mustard, radish root and gum myrrh,

two oranges and a lemon each day.

In addition, sailors who were ill and got the normal ship diet were observed (control group)).


The world’s first controlled clinical trial: Results

After 6 days:

sailors who were consuming lemons and oranges were almost completely recovered

all other patients were still suffering from scurvy

except for the cider drinkers who slightly improved


The world’s first controlled clinical trial: Conclusion

Although numbers of patients were small, results were striking: oranges and lemons were the key to curing scurvy (effects of treatment were huge). Lind had no idea, why oranges and lemons were effective. However this was not important. Demonstrating that a treatment is effective

is the priority. Understanding the exact details of the

underlying mechanism can be left as a problem for subsequent research

Lind published his results 6 years later in a book which did not have an immediate impact.

1780 the physician Gilbert Blane read Lind’s book and replicated the trial, with many sailors.

Later he became responsible for determining naval medical procedures. On 5 March 1795 the Board and the Admiralty agreed that sailors' were issued a daily ration of lemon juice.


Why oranges and limes cure scurvy? The discovery of the causal mechanism

1927 Hungarian biochemist Albert Szent-Györgyi (1893-1986) isolated a compound he called hexuronic acid,

1932, The American biochemist Charles Glen King (1896-1988) isolated the same compound from lemon juice and demonstrated the connection between hexuronic acid and scurvy in guinea pigs. The acid was renamed: ascorbic acid; Vitamin C

Vitamins are organic nutrients that cannot be produced by the body and have to be supplied through food.

Vitamin C is used to produce collagen, which glues together the body's muscles, blood vessels and other structures, and so helps to repair cuts and bruises.

A lack of vitamin C results in bleeding and the decay of cartilage, ligaments, tendons, bone, skin, gums and teeth. A scurvy patient disintegrates gradually and dies painfully.

Studies on humans were conducted during WW II with conscientious objectors in the U.K. and in the 1960s with prisoners in the U.S • Symptoms of scurvy could be induced by an experimental scorbutic diet with extremely low

vitamin C content. • Symptoms could be completely reversed by additional vitamin C supplementation of only 10 mg

per day.

100 g lemons or oranges contain about 53 mg Vitamin C.


What can we learn from this example?

If you want to know whether a treatment is efficacious :

Compare groups of persons who get different treatments and no treatment to figure out which treatment works better than no treatment

Make sure that the groups are equal (comparable) before the treatment starts

Hold conditions in the groups constant except for the treatment condition you want to investigate

-> Realize a randomized controlled trial (RCT)


What can we learn from this example?

Effects have to be replicated before further conclusions can be drawn It takes time to implement new discoveries into clinical practice

Efficacy research aims providing evidence that a treatment really works

• It aims to provide technological knowledge

Efficacy research does (usually) not tell you why a treatment works If a treatment is efficacious, it can be beneficially used without having any idea

or even a wrong idea why it works

Discovering the causal pathway is usually the goal of fundamental research Knowing the causal pathway will help to optimize treatment and develop an

efficacious prevention • E.g. dropping unnecessary components • Adding components that facilitate or enhance the effects


Why do we need an appropriate control group?

to rule out alternative explanations (threats to internal validity) (Cook & Campbell, 1979; Trochim, 2001):

1. Maturation • Spontaneous remission

• Cyclic course of disease (e.g. Major Depression)

2. History and external factors: • Patients get additional treatment (e.g. medication) • external factors (e.g. economic crisis, terrorism etc.)

3. Mortality: • Patients, who do not profit, break up earlier

4. Regression to the mean .


Regression to the mean (Trochim, 2001)

Statistical phenomenon

between two variables that are not perfectly related

when an extreme group is selected based on the values of one variable

• E.g. Galton investigated heritability of body size. He found counterintuitive results: Big fathers had smaller sons

Small fathers had bigger sons

Explanation: Size of fathers is the variable for the selection of extreme groups. There is no perfect correlation between size of fathers and size of sons, because size of sons depends 50 % on genes of fathers and 50 % on genes of mothers. Therefore, the size of sons will be closer to the population mean, than the size of fathers

• E.g. Pre-Post Design A group of patients with low psychosocial functioning is selected at beginning

of treatment. After the treatment social functioning is measured again (retest reliability = .60). Even if there was no treatment effect the value of the post measure will be closer to the population mean.


Regression to the mean


Regression to the mean (Trochim, 2001)

Regression to the mean is stronger:

the more the selected group deviates from the population mean

the less the two variables are related

Regression to the man is a function of the correlation r between the variables : Percent Regression = 100 (1 - r)

r = 1.00 Regression to the mean = 0%

r = .50 Regression to the mean = 50%




Regression to the mean (Trochim, 2001) Example 1: Pre-post design; no treatment effect

Correlation: r Pre Post = .50

Selected group: pretest < 40 (Mpre = 30)





No treatment effect in the posttest for the population (Mpre = 50; Mpost = 50).





No treatment effect in the posttest for the population (Mpre = 50; Mpost = 50).

However, in the selected extreme-group Mpost = 40.

Thus, a pseudo effect (gain of 10 points) occures due to regression to the mean


Regression to the mean (Trochim, 2001) Example 2: Pre-post design, with treatment effect



Treatment effect (gain of 15 points) in the posttest for the population (Mpre = 50; Mpost = 65)


Regression to the mean (Trochim, 2001) Example 2: Pre-post design, with treatment effect



Treatment effect (gain of 15 points) in the posttest for the population (Mpre = 50; Mpost = 65)

However, in the selected extreme group Mpost = 55, it appears

that the improvement was larger in the selected group (25 points). A pseudo effect, due to regression to the mean.


Why do we need an appropriate control group?

to rule out alternative explanations (threats to internal validity) (Cook & Campbell, 1979):

1. Maturation • Spontaneous remission

• Cyclic course of disease (e.g. Major Depression)

2. History and external factors: • Patients get additional treatment (e.g. medication) • external factors (e.g. economic crisis, terrorism etc.)

3. Mortality: • Patients, who do not profit, break up earlier

4. Regression to the mean 5. Testing (reapeated measures design)

• E.g. Reactivity of pre measurement

6. Instrumentation (reapeated measures design) • E.g. Bias of diagnostician


Possible control groups (Comer & Kendall, 2013)

• Control group without any treatment • Participants only participate in the assessment, but they do not expect treatment

• Wait-list control group • Participants participate in the assessment • Participants expect to receive treatment after a certain period of time

• Pill-placebo • Participants get medication, without knowing whether it contains an active substance or not

• „Psychotherapy placebo“

• credible treatment for the participant • but no specific effect (e. g. discussion group, relaxation, reading disorder related books, self

help manuals etc.)

• Adding or Subtracting components of a treatment (dismantled comparison)

• Treatment with or without specific elements: e.g. CBT with relaxation, vs. CBT without relaxation, vs relaxation.

• Treatment as usual (TAU)

• Well established, evidence based treatments PT or medication


Problems with untreated control groups

• Demoralization

• Patients drop out

• Patients seek other treatments

• Problem, if patients run into crisis

• Ethical problems: How can we justify to withholding a patient being efficaciously treated?

• Depends on condition (e.g. depression vs. social phobia)


The Placebo Concept does not Work for

Psychotherapy

Pharmacologic Placebo (sham medication)

• identical appearance of pharmacol. substance and placebo

• only difference = pharmacol. effect

• separation of substance and procedure of giving the substance

• expectancy effect controlled (double blind)

„Psychotherapy Placebo“ (minimal treatment)

• Substantial difference between intervention and PT-placebo -> credibility?

• Intervention and PT-placebo are different in several aspects that have an effect on experience and behavior

• Technique and procedure can‘t be separated

• Therapist knows that PT-placebo is supposed to be less efficacious -> Expectancy effects


Problems with

Psychotherapy Placebos

• PT-Placebo without effect is not possible

• Treatment needs to be plausible for patients and therapists -> Assessment of treatment credibility

• Contains unspecific components:

• contact with therapist, appreciation, expectation of improvement, optimism

• Ethical problem:

• Patients do not get more effective treatment

• Patients need to be informed and willing to obtain a psychotherapy placebo

• PT-Placebo = Control condition with low treatment -> The more similar placebo and treatment the smaller the difference


Problems with treatment as usual

(TAU) (Westen et al., 2004)

Quality of TAU often low:

• Low budget

• Low frequency of sessions

• Often badly trained therapists

• Therapists often have high work load


What does a particular control group control?

Untreated

Wait-list Pill-

Placebo

« Therapy

Placebo»

Minimal

Treatment

Treatment

as usual

(TAU)

Evidence

Based

Treatment

Passage of time

X X X X X X

Expectation that there will be a treatment and improvement soon

X

Having contact with doctor / therapist

X X X X

Receiving treatment

X X X X

Expectation that treatment will help

(X) (X) X X

Common factors: attention, warmth, appreciation, empathy, information …

(X) (X) X X

Specific effect of treatment

(X) X

(X) Depends on the realization of the condition and / or participants’ expectations and believes


Requirements for drawing valid conclusions from

comparisons with control groups

• Participants in the treatment and control group need to be similar before treatment starts

• Methods to ensure equality: • Participants will be randomly assigned to treatment conditions

• requires large samples

• Randomized blocks assignment • Matching participants in subgroups that are comparable on key dimensions

(baseline severity of disorder)

• Members of group will then be randomly assigned to treatment condition


Problems despite randomized control

groups (Westen et al., 2004)

• Randomization does not work well with small samples (n > 40 per group)

• Participants drop out (attrition, mortality) • Differences between participants who drop out and participants who

complete the study

• Different drop out rates in different groups • E.g. drop out rate higher in wait-list control group than in treatment group


Additional requirements for drawing valid

conclusions from comparisons with control

groups

• Procedures across treatments must be equal for key variables: • Duration of treatment

• Length, intensity and frequency of contact with clients

• Credibility of the treatment rationale

• Setting (individual vs. group)

• Degree of involvement of persons close to client (partner, friends)


Take home message

Artifacts (threats to internal validity, like regression to the mean) may let us believe that treatments are efficacious when in fact they are not

To demonstrate that a treatment is efficacious we need to apply an experimental control group design that can effectively rule out alternative explanations

Randomized Controlled Trials (RCT) are the gold standard to determine efficacy

The size of treatment effect depends on the control (comparison) group

Despite randomization of groups inequalities may occur which weaken the strength of conclusions that a treatment is efficacious


References

Comer, J. S. & Kendall, P. C. 2013). Methodology, design, and evaluation in psychotherapy research. In J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (6th ed., pp. 21-48). Hoboken, NJ: Wiley.

Cook, T. D. & Campbell, D. T. (1979). Quasi -Experimentation. Design and analysis issues for field settings. Chicago: Rand McNally College Publishing Company.

Edzard, E. & Shing, S. (2009). Trick or treatment? Alternative medicine on trial. London: Bantam Press.

Gratis available: http://cpp.in.ua/wp-content/uploads/2013/06/10_Simon_Singh_and_Edzard_Ernst_Trick_or_TreatmentBookos.org_.pdf

Knight, R. O. (1941). Evaluation of the results of psychoanalytic therapy. American Journal of Psychiatry, 98, 434-446.

Trochim, W. M. K. (2001). The research methods knowledge base (2nd ed.). Cincinnati, OH (USA): Atomic Dog Publishing. https://www.socialresearchmethods.net/kb/regrmean.php

Westen, D., Novotny, C. M. & Thompson-Brenner, H. (2004). The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin, 130, 631-663.

https://www.socialresearchmethods.net/kb/regrmean.php

how can we evaluate...

Documents