using behavioural insights and randomised controlled ... symp... · using behavioural insights and...

26
Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher CabinetOffice Behavioural Insights Team

Upload: nguyenkien

Post on 29-Jun-2018

229 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

Using Behavioural Insights and Randomised

Controlled Trials in public policy

Dr Rory Gallagher

CabinetOfficeBehavioural Insights Team

Page 2: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Behavioural insights (BI) offers new evidence

and frameworks for influencing behaviour

1. Regulation

2. Incentives

3. Information

Behavioural Insights

Page 3: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

BI can help change the way we think about and

approach policy problems

Page 4: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

“Our government will find intelligent ways to encourage, support and enable people to make better choices for themselves.”

Page 5: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

• ‘Gold Standard’ in evaluation

• Widely used in medicine, international development

and business, but less so in public policy

• Integral part of BIT’s methodology

RCTs are a good way of evaluating whether our

(BI) interventions are working

.

Page 6: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Lots of myths about using RCTs in publioc

1. Pointless

2. Expensive2. Expensive

3. Unethical

4. Difficult to run

Page 7: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

• Understand what works

– Evidence-based policy; VfM; prove efficacy

• Context is everything

Why we use RCTs

• Context is everything

– Lab vs field; different policy areas/locations/people

• Details matter

– Small details = big effects; co-design

Page 8: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Multi-arm trials can highlight important nuances

35.1

35.9

37.2

39.0% paying tax debts after 23 days

33.6

35.1

Control (8,558) UK Norm (8,300) Local Norm (8,403) Debt Norm (8,779) Local + Debt Norm

(8,643)

Page 9: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Different measures also reveal nuances

5.00%

10.40%

15.40%

Proportion Leaving a Legacy Gift

£3,300 £3,100

£6,610 Size of Legacy Gifts

5.00%

Control Just Ask Passion Ask Control Just Ask Passion Ask

Page 10: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

21

35 35

25

30

35

40

Response rate of Doctors to HMRC letters

Don’t always produce the results you expect

4

21

0

5

10

15

20

Generic HMRC

letter

Tax Health Plan

letter

BI letter BI letter + social

norms

Page 11: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Don’t always produce the results you want!

22

28

25

20

25

30

% Discontinuing council tax discount

0

5

10

15

Original letter (control) New BI letter New BI letter + signature up

front

Page 12: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

• Operational constraints– Policy interventions?

– Buy-in

– Sample size

– Data sharing

• Political constraints

Constraints and trade-offs

• Political constraints– Established programmes

– Transparency - ‘nil’ results/ ‘failures’

– Time frames

• Trade-offs– Who/how randomise

– No. of arms

– Impact (‘bundle’) vs specific causal effect

Page 13: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

• Randomisation

• Design

Things to watch out for

• Implementation

• Measurement

Page 14: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

BIT paper: Test, Learn, Adapt

Page 15: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

BI and RCTs in action:

Two case studiesTwo case studies

Page 16: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

What is this letter about?

Page 17: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Actions/consequences unclear

Is the

heading

clear? What is a

distress

Contact info

on the back

page

distress

warrant?

Lots of possible

consequences –

diluting effect

Key action –

how contact?

Primes

appeals

Is this info

necessary?

Page 18: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Court fines

Page 19: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Court fines

£9

£13

£11£12

£8

£11

£9£10

10

12

14

16

Av

era

ge

am

ou

nt

pa

id (

£) Trial 1

Trial 2

£4

£9 £8 £9

0

2

4

6

8

10

No text Standard Personal Amount Personal +

Amount

Av

era

ge

am

ou

nt

pa

id (

£)

Text condition

Page 20: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Test, Learn, Adapt – 9 steps

1. Intervention - test effect of text messages on payment of court fines

5 arms = control, standard, personal, amount, personal+ amount

2.Outcome - % individuals paying court fines by the deadline date

3. Unit - randomise at individual level

(rather than different courts or local authority areas)

4. Numbers - to demonstrate a 10% increase in payment rates, we needed a

sample size of 2,000 people.

5. Randomisation - individuals randomly allocated to 5 conditions

6. Administration - Courts Service sent text messages to the individuals

randomly assigned to receive them

7-9. Test, Learn, Adapt - 3 trials, impressive results.

HMCTS exploring national roll-out.

Page 21: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Getting people into employment

Page 22: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Getting people into employment

1. Cut down process 2. Commitments 3. Strengths identification

Page 23: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Getting people into employment

69%

66%65%

70%

80%% off-flow from benefits after 13 weeks

55% 55%57%

56%58%

61%

53%

40%

50%

60%

Feb Mar April May June July Aug

Page 24: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

Pragmatic stepped wedge randomised trial:

job centres across the county

Job

CentreReadiness

Implementation

Order

(TBC)

On-flow

per

month

Off-flow at

13 weeks

Months of trial

0 1 2 3 4 5 6 7 8 9 10 11 12 etc....

1

Ready

Harlow 608 51.70% *M1 *M2 *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....

2 Chelmsford 727 54.00% *M2 *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....

3 Colchester 734 52.80% *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....

4 & 5Canvey & 235 & 55.5% &

*M4 *M5 *M6 *M7 *M8 *M9 etc....

UNCLASSIFIED

4 & 5

Almost

Ready

Canvey &

Rayleigh

235 &

308

55.5% &

57.3%*M4 *M5 *M6 *M7 *M8 *M9 etc....

6 & 7Clacton &

Harwich

441 &

129

48.8% &

47.0%*M5 *M6 *M7 *M8 *M9 etc....

8 Grays 683 50.50% *M6 *M7 *M8 *M9 etc....

9

Not ready

Basildon 901 53.50% *M7 *M8 *M9 etc....

10 & 11Braintree &

Witham

444 &

177

55.2% &

52.2%*M8 *M9 etc....

12 Southend 930 50.20% *M9 etc....

Dark grey = current jobseeker process (control)

Blue = new jobseeker process (intervention)

*MX = measurement of 13 week off flow for cohort which began jobseeker claim in month X of trial (in bold = first month of outcome measure for that cluster)

Page 25: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

CabinetOfficeBehavioural Insights Team

• BI is having major impacts

• Provides new frameworks and approach - humility

• RCTs increasing - driving ‘what works’

• Test, Learn, Adapt: agile implementation

Conclusions

• Test, Learn, Adapt: agile implementation

• Context, Details, People … matter

• Early days for BI & RCTS- long way to go, just scratching surface

• Next phase = sustainability and segmentation

• Future enablers = ‘big’ data, digital, transparency, VfM

Page 26: Using Behavioural Insights and Randomised Controlled ... Symp... · Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher Cabinet Office Behavioural

http://blogs.cabinetoffice.gov.uk/behavioural-insights-team