![Page 1: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/1.jpg)
Kevin Chadwick
ANU Centre for Social Research and Methods
Applying EAST to
email reminders
![Page 2: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/2.jpg)
Acknowledgements Department of the Environment and Energy
The International Operations Section
Permission to publish
PhD support
Lani Perlesz, BETA
ATO research staff
Liz Hobman, CSIRO
Behavioural Insights Team
![Page 3: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/3.jpg)
Overview
Policy context
Proposed / imposed solution
Methodology
Results – test, learn, adapt
Lessons
![Page 4: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/4.jpg)
Policy context - APS
Sweeping budget cuts
Commission of review
Staff cut by up to 25%
Deregulation agenda
Red tape reduction targets
![Page 5: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/5.jpg)
Policy context –
International operations section The Ozone Act
Companies must report
SGG imports each quarter
through OLaRS
Email reminders at the end
of each reporting period
Regulatory burden
Unsatisfactory compliance
with reporting obligations
![Page 6: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/6.jpg)
Policy context –
behavioural economics
First departmental BE work program
Very small team
No specialists, everything we did was new
Equity concerns = risk aversion?
“You only get one shot, do not miss your
chance to blow, this opportunity comes
once in a lifetime, yo” – Eminem, Lose Yourself
![Page 7: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/7.jpg)
Proposed / imposed solution
Redesign email reminders sent to companies
No scope for pre-trial
research into companies’
compliance behaviour
Hypothesis/assumption:
Better reminder design
increases compliance
![Page 8: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/8.jpg)
Trial methodology RCT - stepped wedge variation
Period 1: control plus treatment 1
Period 2: control plus treatments 1 and 2
Period 3: control plus treatments 1, 2, 3 and 4
Period 4: abandoned due to operational requirements
![Page 9: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/9.jpg)
Balance check on past
reporting behaviour
Difference to control groupGroups 1 2 3 4 Control
N = 132 128 138 132 138
1st time report +1% -2% +1% -1% -
Late last period +3% -1% +4% +3% -
Repeatedly late +6% +2% +6% +8% -
Later analysis found no statistical difference in past reporting compliance between the groups
![Page 10: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/10.jpg)
First reporting period
Treatment 1 (n = 132)
Control
(n = 138)
Trial sample
[n = 668]
![Page 11: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/11.jpg)
Reminders: Control vs treatment
![Page 12: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/12.jpg)
Applying EAST EASY
Simplify the message
Reporting information upfront
‘Button’ link to web page
ATTRACTIVE
Appealing photo to attract
attention
Required actions in coloured text
boxes
SOCIAL
Most people report on time
TIMELY
Not yet, but watch this space…
![Page 13: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/13.jpg)
Period 1 results
5.02%
33.51%
4.85%-1.40%
Treatment 1 Late last Nil report 1st report
Conditional probit model
***
* p < 0.05, ** p < 0.01, *** p < 0.001
![Page 14: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/14.jpg)
Second reporting period
Treatment 1 (n = 132)
Treatment 2 (n = 128)
Control
(n = 138)
Trial sample
[n = 668]
![Page 15: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/15.jpg)
Early reminder notification
Christmas and summer break
End of reporting period 1 Jan
Reporting deadline 14 January
Sent to all companies
Standard procedure
Report before end of period
![Page 16: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/16.jpg)
Treatment 2: 2nd redesigned reminder
![Page 17: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/17.jpg)
Applying EAST again EASY
Removed photo – distraction?
Report deadline clearer
What & why information upfront
Three simple steps to report
ATTRACTIVE
Australian Government crest
SOCIAL
Most licence holders report on
time
TIMELY
Not yet, but watch this space…
![Page 18: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/18.jpg)
Period 2 results
-6.36%
4.76%
25.12%
-2.28%
Treatment 1 Treatment 2 Late last Nil report
Conditional probit model
***
Important observation:
A greater proportion
of companies in all
groups reported on
time in this period.
Historical reporting
data indicated that
reporting rates at this
time of year were
higher in previous
years, too.
Was this due to the
early reminder?
* p < 0.05, ** p < 0.01, *** p < 0.001
p=0.071
![Page 19: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/19.jpg)
New insights revealed
Reminder strategy varied over time
Number of reminders
Timing of reminders
Company characteristics varied
Previous reporting behaviour
Experience with reporting
Panel analysis – ten reporting periods
![Page 20: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/20.jpg)
Insights from previous periods
10.35%
-2.50% -3.87%
8.20%
-0.39%
0.12%
Late last Nil report First report No.
reminders
1st
reminder
Last
reminder
Factors influencing compliance –
panel data, ten reporting periods
***
*
* p < 0.05, ** p < 0.01, *** p < 0.001
Average decrease in
likelihood to report on
time for each day the first
reminder was sent before
the reporting deadline
***
****
Companies that
reported late in one
period were more
likely to report on
time the next period
Average increase in likelihood
to report on time for each
additional reminder sent
compared with a single
reminder
![Page 21: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/21.jpg)
Third reporting period
T 1
(n = 132)
T 2
(n = 128)
T 3
(n = 138)
T 4
(n = 132)Control
(n = 138)
Trial sample
[n = 668]
![Page 22: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/22.jpg)
Third period
ControlStandard reminder
Treatment 1 1st redesign
Treatment 2 2nd redesign
Treatment 3Early
reminder2nd redesign
Treatment 4Early
reminder2nd redesign
Last minute reminder
Mid-March 1 April 13 April 14 April
due date
![Page 23: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/23.jpg)
Treatment 4: Last-minute reminder notification
Based on Redesign 2
Sent 10am day before deadline
‘Due tomorrow’ in subject line
Increase salience/present bias
Most of your peers have reported
![Page 24: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/24.jpg)
Period 3 results
2.88%
0.94%
8.08%
18.93%
Treatment 1 Treatment 2 Treatment 3 Treatment 4
Unconditional probit model
***
p=0.096
* p < 0.05, ** p < 0.01, *** p < 0.001
![Page 25: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/25.jpg)
Period 3 results
3.57%
-1.78%
7.15%
21.82% 22.29%
-0.20%
Treatment 1 Treatment 2 Treatment 3 Treatment 4 Late last Nil report
Conditional probit model
* p < 0.05, ** p < 0.01, *** p < 0.001
******
p=0.125
![Page 26: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/26.jpg)
Period 3 results
Cumulative per cent of licence holders reported
control
T1
T2
T3
T4
Early reminder
End of period
reminder
Easter
Last-minute reminder
Companies in all groups
increased rates of
reporting just before the
due date, but those that
received the last-minute
reminder were even
more likely to report
As observed in historical
data, each reminder
prompted a short spike
in reporting from the
companies that
received them
![Page 27: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/27.jpg)
Third reporting period results
Before early Early Standard Last-minute
Proportion of outstanding reports submitted
Control
T1
T2
T3
T4
The last minute
reminder led to
a clear increase
in reporting just
before the due
date.
The early
reminder
prompted ten
times as many
companies to
report before
the due date.
![Page 28: Applying EAST to email reminders · ANU Centre for Social Research and Methods kevin.chadwick@anu.edu.au Applying EAST to email reminders. Acknowledgements Department of the Environment](https://reader031.vdocuments.net/reader031/viewer/2022041017/5eca5981c38f4e40c93ea2ef/html5/thumbnails/28.jpg)
Lessons learned
Don’t decide on a solution before
you’ve explored the context
Design trials to test, learn, and adapt
Importance of historical data analysis
Make sure you have all the information
Allow plenty of time for data cleaning
Test interventions before implementation