the online laboratory: perfecting your outreach through email testing
DESCRIPTION
Presentation delivered at Netroots Nation 2013 by CCAH's Damien Shirley and PFAW's Ben Betz.TRANSCRIPT
![Page 1: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/1.jpg)
The Online Laboratory:
Perfecting Your Outreach
Through Email Testing
#NNOnlineLab
![Page 2: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/2.jpg)
WHO ARE YOU?
• Ben Betz
– Online Strategy Manager, People For the
American Way
• Damien Shirley
– Account Executive, Chapman Cubine
Adams + Hussey
![Page 3: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/3.jpg)
WHAT IS AN EMAIL TEST?
• Send multiple versions of an email
• Identical except for a single test variable
• To a random, representative sample of your
email list
• Measuring which version performs better
• Goal: Try a new approach and see if it works
VS.
![Page 4: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/4.jpg)
WHY SHOULD I TEST?
• Every email list is different
• There are best practices, but they don’t always apply to every email list.
• You may know how your supporters respond to your emails, but not why
• If you’re not testing, you’re guessing
• You might be leaving something on the table
• Bottom line: Test to optimize your email engagement.
![Page 5: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/5.jpg)
WHAT DO I NEED TO GET
STARTED?
• Email list
• Email platform (CRM)– Many have built-in testing modules
• Email messaging program
• Confidence calculator
![Page 6: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/6.jpg)
WHAT SHOULD I TEST?
• Anything
• Test approaches that could improve
your email performance
– Let your data guide you
• Make your tests scalable
• Think outside the inbox
![Page 7: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/7.jpg)
WHAT SHOULD I TEST: TESTING
IDEAS
• Sender
• Send time
• Send date
• Subject line
• Personalization
• Callout box position
• Callout box image
• Salutation
• Signer
• Length
• Tone
• Landing page
• Landing page
layout
![Page 8: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/8.jpg)
HOW DO I TEST?
• Components:
– Control and test email versions
• Only difference: your test variable
• Only one test variable at a time
– Control and test audiences
• Random, representative samples of your list
– Bottom line: the only difference between
email versions should be the variable
you’re testing
![Page 9: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/9.jpg)
HOW DO I TEST: TEST TYPES• A/B split test
– One control email variant & one test email
variant
– Most common
– Easiest to set up
Control Test
![Page 10: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/10.jpg)
• Multivariate Test
– One control email variant & multiple test
email variants
– Still only one test variable
– Better for larger lists
Control Test 1 Test 2
HOW DO I TEST: TEST TYPES
![Page 11: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/11.jpg)
• Pre-Launch Test
– Send your control and test variant(s) to
small segments of your email list
– Launch the winning version to the rest of
your list
– Takes more time to set up
Control TestTo Full List
HOW DO I TEST: TEST TYPES
![Page 12: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/12.jpg)
HOW OFTEN SHOULD I TEST?
• Always
• Or, as often as you can
• Continue until you get significant
results
![Page 13: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/13.jpg)
HOW DO I KNOW WHAT WON?
• Check your data
• Use a confidence calculator
–Determine statistical significance
–Calculate your confidence interval
![Page 14: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/14.jpg)
HOW DO I KNOW WHAT WON?
• What if neither variant wins?
–No significant difference: go with
your preference
–Mixed results: prioritize your metrics
![Page 15: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/15.jpg)
WHAT NOW?
• Apply your results
• Plan your next test
• Build a testing library
• Embrace failure
• You’re not done yet: retesting
– No set schedule
– Let your data guide you
![Page 16: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/16.jpg)
WHAT TO TEST: MEMBER DRIVE
• Test idea: Test including a member
drive versus standard renewal
messaging.
• Test type: A/B split test
• Control: Standard renewal messages.
• Test: Messages included the member
drive goal for new or renewed
members.
![Page 17: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/17.jpg)
Standard Renewal Member Drive
WHAT TO TEST: MEMBER DRIVE
![Page 18: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/18.jpg)
• Results: – The member drive test had a higher
clickthrough rate.
– The member drive also test generated more revenue per donor.
– Using a confidence calculator, we determined that both of these results were statistically significant.
• Recommendation: – Include member drives in future renewal
campaigns.
WHAT TO TEST: MEMBER DRIVE
![Page 19: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/19.jpg)
WHAT TO TEST: EMAIL LAYOUT
Personalized Member Invoice
vs. Standard Email Format
Invoice was the winner
![Page 20: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/20.jpg)
WHAT TO TEST: CALLOUT BOX
IMAGE
Member Card had a higher
clickthrough rate
Thermometerhad a higher response
rate (winner)
![Page 21: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/21.jpg)
WHAT TO TEST: SENDER
• Testing campaign-y names as sender
versus names of staff
– Stop GOP Witch Hunts vs. NAME, PFAW
– Overturning Citizens United vs. NAME,
PFAW
– PFAW Membership Director vs. NAME,
PFAW
![Page 22: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/22.jpg)
WHAT TO TEST: MESSAGE
• Message A vs. Message B
• Roll out a message to smaller segment
of the email list to test its strength prior
to full-list launch
![Page 23: The Online Laboratory: Perfecting Your Outreach Through Email Testing](https://reader036.vdocuments.net/reader036/viewer/2022081401/559caae81a28abe75b8b4791/html5/thumbnails/23.jpg)
QUESTIONS?