Public health systems: holding governments accountable. Establishing standards,

measuring implementation

Paul BoltonApplied Mental Health Research Group

Johns Hopkins Bloomberg School of Public HealthBaltimore, USA.

Site population problems interventionSouthwest Uganda adults depression IPT-G

Northern Uganda Displaced adolescents

Maladaptive behavior, depression

IPT-GCreative play

Indonesia (Aceh) Adults affected by conflict

Depression, distress Non-specific counseling

Kurdistan (Iraq) Genocide, repression

Depression, anxiety, PTSD, Traumatic Grief

CPTBAnon-specific

Southern Iraq Conflict, repression.

As above CPT, CETA

Thai Burma border Displacement, repression,

Depression, anxiety, PTSD

CETA

Colombia Repression Depression, anxiety, trauma

CETA

Democratic Republic of Congo

Female sexual violence

Depression, anxiety, trauma, stigma

CPT VSLA

Zambia Child sexual abuse Depression, withdrawal trauma, stigma

TF-CBT

Outcome Measures

• Survivors participate in the rehabilitation process (GC3)

• Programmes…take into account a victim’s culture, personality, history and background (GC3)

• So how can we have standard instruments/measures when client needs and situations vary?

Some mental health problems are both predictable and similar:

– Depression (Hopkins Symptom Checklist or HSCL)– Anxiety (HSCL)– Trauma (Harvard Trauma Questionnaire or HTQ)

But others vary

A Qualitative Approach to outcome measures

1. Start with free listing asking:‘what are the problems of people who have been tortured?’

2. List all commonly mentioned problems, then choose priority problems based on frequency and severity.

3. Key informant interviews to explore those problems.

• Feeling handicapped • Social injustice. We are not treated equally• Divorce• Poverty• Drinking alcohol • We are not respected as we should be. We feel inferior • We are insulted; especially women and girls are called

names, that you are raped. • We regret helping this government. • Social relationships have become weak.• Disappointment. Nothing has been done for us. • Rage • No-one is honest. So we are obliged to do bad things

Qualitative Problem descriptions used to:

• Other problems are included as separate questions

• Add to depression/anxiety/trauma instruments to increase local validity– Thinking too much– Cannot accept person is gone– hating the world

Qualitative Problem descriptions used to:

• Used for translation

• Decide on appropriate interventions

Program Monitoring

• Monitoring – constantly tracking indicator.• Purpose of monitoring is to identify and address

problems as they occur (iterative)• Evaluation – determining if there is a change in

the indicator between program beginning and end.

• However: Everything that is evaluated is also monitored.

program monitoring and evaluation for “effective Implementation”*†

FidelityAvailability/Access*UptakeSurvivor Compliance/CooperationAppropriateness/Acceptability*FeasibilityCostEffectiveness*

†Dissemination and Implementation Research

Fidelity Monitoring

Gender M Main Syxs Nightmares, anxious, tense, sleeping problems

Behavior Act Need Score (/9)

3

Trauma Type

Prison, Torture, Military attack, taken from family

Total Score 59 Functioning Items (general/out of 84)

55

Relax Need Score (/12)

12 Live Exposure Score (/3) 0

Age 40 Clinical Observations: rubbing hands and looking around room Flow ___ Standard _X__ + Relaxation

___ + Behavioral Activation ___ + Live Exposure

Accessibility/Reach

Can be defined in various ways:distancecosttimeopportunity cost.

Uptake

• How many of those who have access to services and know about them, try them?

• Can be defined as:

Uptake

Uptake

Compliance/cooperation

• How many survivors who begin treatment complete it?

Compliance

Appropriateness/Acceptability

• Combination of uptake and compliance:

• If high uptake and compliance, program is considered acceptable.

• If either is low, acceptability is considered low.

Feasibility

• Requires a vision of who is going to pay for the services for as long as they are needed.

• Can this payer(s) afford to pay for the duration?• Is this payer(s) willing to pay for the duration?

Effectiveness

• Does NOT refer to whether survivors improve.

Refers to what would happen to the survivor in the absence of services:

survivor would be worse off: effectivesurvivor would be the same: not effectivesurvivor would be better off: not effective

and harmful.

Reduction of depression symptoms by group

25

30

35

40

45

50

Baseline Follow-Up

Dep

ress

ion

Sym

ptom

Sc

ores

Reduction of depression symptoms by group

25

30

35

40

45

50

Baseline Follow-Up

ControlIPT-GCP

Dep

ress

ion

Sym

ptom

Sc

ores

Effectiveness: what outcomes?

• “Rehabilitation…refers to the restoration of function or the acquisition of new skills required as a result of the changed circumstances of a victim in the aftermath of torture or ill-treatment.”

• Ultimate goal is restoration of dignity.

• Interpretation: Main aim is the restoration of survivor’s roles in terms of self, family and society.

How to operationalize this?

Main impact outcomes are the functions that make up locally defined roles.

These vary, so there is no single instrument.

Replace single instrument with process that has survivors define them locally.

Function Assessment

• Qualitative methods:– Free listing

• What are the activities that men/women normally do (to take care of themselves/family/community?

• What do children normally do?• How do you know when a man/woman/child is doing

well.• What are the activities that survivors cannot do that

they need to do?

Iraq (women)• Housework• Cooking• Other manual labor• Caring for family

members• Giving advice• Exchanging ideas• Having harmonious

relationship with family• Raising children

correctly• Contributing to the

community• Sympathize with others

• Visiting and socializing• Asking for help • Getting help• Making decisions• Taking part in family

activities • Taking part in community

activities• Learning something new• Concentrating• Dealing with strangers • Attending mosque or

religious gathering• Assisting others

Holding states accountable

• Treat the state as a partner whose priorities must also be met.

• Try to help the state meet them.• Typical state priorities:

– Increase access to effective health services– Reduce mortality for the whole population– Reduce morbidity for the whole population

Meeting State and Survivor Needs

•Address priority problems OF torture survivors, not just focus on problems DUE TO torture.•Most problems of torture survivors are shared by many others.•Where possible, support services that deal with these problems for everyone. Survivors should access these same services as others.

New Structure in Iraq• Most survivors receive mental health and counseling services integrated

into physical health system which addresses these problems for everyone (ie, also non-torture survivors)

• Providers are primary clinic staff with little mental health background who are trained to provide effective psychotherapy

• Supervisors are mental health professionals based in psychiatric centers and in torture treatment centers.

• Referrals (non-torture) to psychiatric centers• Referrals (torture) to TTC.• Advantages:

– Government more supportive (supports wider need)– Clients like it better (more anonymity, less singling them out)– Reach and access enhanced +++++ through integration– Priorities better match client priorities and more accessible– Torture survivors who need it still get specialist care– TTC can focus on those who really need them.

Apprenticeship Model of Training and Supervision

• Key to expanding quality treatment access in low resource countries

• Based on research on training:– One-off trainings are ineffective for behavior change.

– “Train and hope” approach to implementation does not work (e.g., Kelly et al., 2000)

– Ongoing supervision with on-the-job training is critical

SUPERVISION MODEL Monitoring quality Continuous Training

Continuous intervention development

Trainers

Supervisors

Counselors

Clients

Purpose of apprenticeship training

• Provides real skills development of provider and supervisor by learning while doing

• Allows non professionals to really learn to provide treatment while assuring survivor gets quality treatment.

• Iterative correction/improvement of survivor and provider and supervisor problems

• Cares for providers – monitors and prevents/treats burnout.

Impact Assessment

Standard evaluation vs Experimental evaluation

• Standard evaluation– Post intervention measure only, OR– Pre and post intervention measure– Assesses whether problem improved– Does not assess why.

• Research– Pre and post evaluation(s)– Compare with controls– Assesses whether problem improved and whether

this was because of the intervention.

Uganda

• Intervention study:

• Pre-intervention - 110/117 (94%) depressed• Post-intervention - 64/117 (54.7%)

depressed

(p<0.001)

Uganda

• Control arm:• Pre-intervention - 110/117 (94%) depressed• Post-intervention - 64/117 (54.7%) depressed

• Intervention arm:• Pre-intervention - 92/107 (86%) depressed• Post-intervention - 7/107 (6.5%) depressed

(p<0.001)

Pre Post0

1

2

3

4

5

6

7

controlintervention

Pre Post0

1

2

3

4

5

6

7

controlintervention

Pre Post0

1

2

3

4

5

6

7

controlintervention

Pre Post0

1

2

3

4

5

6

7

controlintervention

Reduction of depression symptoms by group

25

30

35

40

45

50

Baseline Follow-Up

Dep

ress

ion

Sym

ptom

Sc

ores

Reduction of depression symptoms by group

25

30

35

40

45

50

Baseline Follow-Up

ControlIPT-GCP

Dep

ress

ion

Sym

ptom

Sc

ores

Which one?

• Use Standard Evaluation when other factors stable.

• Use experimental evaluation when likely effect of intervention is not known and other factors not stable.

– Control group is essential to answering the question: ‘was change due to the intervention?’

RCT as program evaluation

RCTs perceived as:• Unnecessary• Unethical – making people wait.• Too complex/difficult• Too expensive - $,time,effort.• Waste/diversion of resources from ‘real’ aid• Not appropriate for low resource/difficult

environments

500

250

250

250

250

Assess

Randomize

Reassess and compare

Service capacity=500

1,000

500

500

500

500

Assess

Randomize

Reassess and compare

Service capacity=500

What makes RCTs costly/complex?

• Design/training/implementation of intervention• Monitoring quality of intervention• Supervision of workers• Monitoring acceptability/feasibility/compliance• Assessing who has priority for services• Reassessing persons after treatment.

True costs beyond normal M&E

• Randomization (cheap)• Analysis (cheap)• Assessing additional cases (relatively cheap –

economies of scale).

• Small compared with savings from stopping ineffective programs– Saves money/time/effort of funder and population– More ethical

When is an RCT Appropriate?

• Existing evidence base poor

• Programs vulnerable to cross-cultural variation

• Situation unstable

• Therefore, uncertainty about local impact of interventions.

• EFFECT SIZE is a measure of the difference between intervention and control in a way that is comparable across studies

50

Effect Size Percent (%) of Controls with Scores below the Average of Intervention Clients

0.2 58%0.5 69%0.8 79%1.4 92%2.0 98%2.5 99%

Interpretation of Effect Size

• Interpretation:0.0 = no effect0.2 = small effect0.5 = moderate effect0.8 = large effect

51

Comparison of Effect Sizes (Iraq)

52

Barriers to Treatment Access

• Most trauma affected persons in low resource countries have multiple problems.

• Many treatments but each focused on 1-2 problems

• Therefore, ?need to provide multiple treatments.