using behavioural insights and randomised controlled ... symp... · using behavioural insights and...

Using Behavioural Insights and Randomised

Controlled Trials in public policy

Dr Rory Gallagher

CabinetOfficeBehavioural Insights Team


Behavioural insights (BI) offers new evidence

and frameworks for influencing behaviour

1. Regulation

2. Incentives

3. Information

Behavioural Insights


BI can help change the way we think about and

approach policy problems

“Our government will find intelligent ways to encourage, support and enable people to make better choices for themselves.”


• ‘Gold Standard’ in evaluation

• Widely used in medicine, international development

and business, but less so in public policy

• Integral part of BIT’s methodology

RCTs are a good way of evaluating whether our

(BI) interventions are working

.


Lots of myths about using RCTs in publioc

1. Pointless

2. Expensive2. Expensive

3. Unethical

4. Difficult to run


• Understand what works

– Evidence-based policy; VfM; prove efficacy

• Context is everything

Why we use RCTs

• Context is everything

– Lab vs field; different policy areas/locations/people

• Details matter

– Small details = big effects; co-design


Multi-arm trials can highlight important nuances

35.1

35.9

37.2

39.0% paying tax debts after 23 days

33.6

35.1

Control (8,558) UK Norm (8,300) Local Norm (8,403) Debt Norm (8,779) Local + Debt Norm

(8,643)


Different measures also reveal nuances

5.00%

10.40%

15.40%

Proportion Leaving a Legacy Gift

£3,300 £3,100

£6,610 Size of Legacy Gifts

5.00%

Control Just Ask Passion Ask Control Just Ask Passion Ask


21

35 35

25

30

35

40

Response rate of Doctors to HMRC letters

Don’t always produce the results you expect

4

21

0

5

10

15

20

Generic HMRC

letter

Tax Health Plan

letter

BI letter BI letter + social

norms


Don’t always produce the results you want!

22

28

25

20

25

30

% Discontinuing council tax discount

0

5

10

15

Original letter (control) New BI letter New BI letter + signature up

front


• Operational constraints– Policy interventions?

– Buy-in

– Sample size

– Data sharing

• Political constraints

Constraints and trade-offs

• Political constraints– Established programmes

– Transparency - ‘nil’ results/ ‘failures’

– Time frames

• Trade-offs– Who/how randomise

– No. of arms

– Impact (‘bundle’) vs specific causal effect


• Randomisation

• Design

Things to watch out for

• Implementation

• Measurement


BIT paper: Test, Learn, Adapt

BI and RCTs in action:

Two case studiesTwo case studies


What is this letter about?


Actions/consequences unclear

Is the

heading

clear? What is a

distress

Contact info

on the back

page

distress

warrant?

Lots of possible

consequences –

diluting effect

Key action –

how contact?

Primes

appeals

Is this info

necessary?


Court fines


Court fines

£9

£13

£11£12

£8

£11

£9£10

10

12

14

16

Av

era

ge

am

ou

nt

pa

id (

£) Trial 1

Trial 2

£4

£9 £8 £9

0

2

4

6

8

10

No text Standard Personal Amount Personal +

Amount

Av

era

ge

am

ou

nt

pa

id (

£)

Text condition


Test, Learn, Adapt – 9 steps

1. Intervention - test effect of text messages on payment of court fines

5 arms = control, standard, personal, amount, personal+ amount

2.Outcome - % individuals paying court fines by the deadline date

3. Unit - randomise at individual level

(rather than different courts or local authority areas)

4. Numbers - to demonstrate a 10% increase in payment rates, we needed a

sample size of 2,000 people.

5. Randomisation - individuals randomly allocated to 5 conditions

6. Administration - Courts Service sent text messages to the individuals

randomly assigned to receive them

7-9. Test, Learn, Adapt - 3 trials, impressive results.

HMCTS exploring national roll-out.


Getting people into employment



1. Cut down process 2. Commitments 3. Strengths identification



69%

66%65%

70%

80%% off-flow from benefits after 13 weeks

55% 55%57%

56%58%

61%

53%

40%

50%

60%

Feb Mar April May June July Aug


Pragmatic stepped wedge randomised trial:

job centres across the county

Job

CentreReadiness

Implementation

Order

(TBC)

On-flow

per

month

Off-flow at

13 weeks

Months of trial

0 1 2 3 4 5 6 7 8 9 10 11 12 etc....

1

Ready

Harlow 608 51.70% *M1 *M2 *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....

2 Chelmsford 727 54.00% *M2 *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....

3 Colchester 734 52.80% *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....

4 & 5Canvey & 235 & 55.5% &

*M4 *M5 *M6 *M7 *M8 *M9 etc....

UNCLASSIFIED

4 & 5

Almost

Ready

Canvey &

Rayleigh

235 &

308

55.5% &

57.3%*M4 *M5 *M6 *M7 *M8 *M9 etc....

6 & 7Clacton &

Harwich

441 &

129

48.8% &

47.0%*M5 *M6 *M7 *M8 *M9 etc....

8 Grays 683 50.50% *M6 *M7 *M8 *M9 etc....

9

Not ready

Basildon 901 53.50% *M7 *M8 *M9 etc....

10 & 11Braintree &

Witham

444 &

177

55.2% &

52.2%*M8 *M9 etc....

12 Southend 930 50.20% *M9 etc....

Dark grey = current jobseeker process (control)

Blue = new jobseeker process (intervention)

*MX = measurement of 13 week off flow for cohort which began jobseeker claim in month X of trial (in bold = first month of outcome measure for that cluster)


• BI is having major impacts

• Provides new frameworks and approach - humility

• RCTs increasing - driving ‘what works’

• Test, Learn, Adapt: agile implementation

Conclusions

• Test, Learn, Adapt: agile implementation

• Context, Details, People … matter

• Early days for BI & RCTS- long way to go, just scratching surface

• Next phase = sustainability and segmentation

• Future enablers = ‘big’ data, digital, transparency, VfM

http://blogs.cabinetoffice.gov.uk/behavioural-insights-team

using behavioural insights and randomised controlled ... symp... · using behavioural insights and...

Documents