2013 dma post conference certification
DESCRIPTION
I presented this presentation at the close of the 2013 DMA show.TRANSCRIPT
The Answer is Always, “It Depends” – So Test It!
Ryan PhelanVice President, Strategy
Acxiom
• Prior Industry Experience– Vice President, Strategy at BlueHornet
– Director, Email Marketing & Acquisition at Sears Holdings
– Responsible for East Coast Operations at Responsys
• Thought Leadership– DM News : Email Gets Personal (Cover Story)
– Keynote address – March 2012, EEC12
– Ranked as one of the top 40 Digital Marketing Strategists in the country by OMI
– Co-Chair of the EEC
– Member of:Ryan PhelanVice President, Global Strategic Services
Join the Email Evolution Council for education, events, networking and
resources!
“OMG Honey, look at that compelling subject line ….totally made me open it”
This is more like it…
Tier 3:Advanced
Tier 2:Medium Complex
Tier 1: Foundation programs
Existingprograms
Persona development, cluster analysis, next logical product, behavioral \ attitudinal segmentation, dynamic messaging, shopping cart abandonment,
preference center phase 2, social messaging
Video in email, creative testing, promotional optimization, triggers, win-back, preference center, social media
Testing & Reporting
Welcome, transactional messaging, opt-down, acquisition, promotional, attrition
Optimize programs based on easy changes (Low Hanging Fruit)
Testing & Reporting
Testing & reporting
Email Program Development
1,000 Consumers from across the United States
Nearly 49% of respondents have an email account for emails they rarely intend to open
Lesson: Ensure that when you ask for an email address, you make the reason very compelling
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
9
When an email is saved to be read
later, 60% never read it
Lesson: Make your CTA immediate, urgent, laced with benefit and time sensitive in email
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
Lesson: 21% to get updates and 13% because they love the brand10
40% of consumers sign up for email to receive discounts
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
11
36% of respondents check email, social media and texts before doing anything else after they wake up
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
Lesson: Think about your message and test the optimal time of the day to send your message.
Lesson: Test when you send and be wary of complex CTA in the morning.12
21% check their email before breakfast
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
Consumers have shifted their consumption and are active at the very start of the day
Lesson: Try testing in the evening or even Friday evening for retail based business.2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
14
Things are tiny in the morning…
Lesson: If you have to pinch it, you’re doing it wrong. Track mobile opens and design a template.
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
15
Phone calls, texts, browsing the internet and email are the top uses for
smart phones
Lesson: Lesson: Track statistics for your consumers that are consuming email on a mobile device.
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
Lesson: Track statistics for your consumers that are consuming email on a mobile device. 16
The shift in consumption is increasing based on consumer need for and ease of access
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
17
Consumers don’t see silos, they see devices
Lesson: Consumers have adopted more devices and marketers have to be truly Omni-Channel
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
Lesson: Capture and track the mobile opens of your subscribers and implement a template18
91% of consumers check email on their mobile phones
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
19
…25% of those that use iPhone Passbook used it to access coupons
Of those that use iPhone Passbook (33%), 22% use it for Movie Tickets…
Lesson: Try linking a coupon for a sale or event to the Passbook functionality and then track use
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
20
72% of consumers read email when they are bored…
…29% read email while in the bathroom
Lesson: Don’t make your email creative boring – inspire, delight and amaze
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
Nothing anyone says in marketing is right(except me of course)
It depends…(the strategists mantra)
QUIZ: WHICH TEST WON?DMA2013 | Email Testing in the Digital Age
Which Test Won?
Up or Down Arrows
Which Test Won?
Up or Down Arrows
Which Test Won
Which Test Won
Which series won?Images Courtesy of:
Which series won?Images Courtesy of:
CHALLENGES OF THE EMAIL MARKETER
DMA2013 | Email Testing in the Digital Age
The email marketer’s challenge• More subscribers• More data• More email
• More targeting• More expectations• Poor attribution
What do we struggle with
You’re not alone…everyone struggles with email
Chart 4.21 Email campaign element testing and optimizationWhich of the following email campaign elements do you routinely test to optimize performance? Please select all that apply.
Subject line
Call-to-action
Message (eg greeting, body, closing)
Days of the week sent
Layout and images
Time of day sent
Landing page
Target audience
Personalization
From line
Layout and images specifically for mobile viewing
Other
None of the above
86%
62%
58%
48%
47%
46%
44%
44%
42%
32%
26%
Source: ©2013 MarketingSherpa Email Marketing Benchmark Survey Methodology: Fielded December 2012, N=264
2%
2%
Message (e.g. greeting, body, closing)
Why do we care?
• Campaign on 10/17– 166
individual email segments
– 84 were tests
Images Courtesy of:
Does this look bad???
The highlighting worked…for a while
Images Courtesy of:
Testing Challenges
We don’t (as an industry) know how to test
Types of TestingA|B Testing• Testing one element against a control
Multi-Variant• Testing multiple elements against a control
Local Control Groups• Isolation of a population on a campaign level to see % of lift
Universal Control Groups• Isolation of a permanent population to see percentage of lift over time
A|B Testing
Cons• Populations must be equal• Time periods must be
significant to judge results• Limited to one variable so
extended testing can be long
Definition• 2 email creative that are
identical but with one element changed in one version
Pros• Simple testing that is built into
most email platforms• Systems usually handle
division of population• Results are easy to
understand and act upon
Multivariate Testing
Cons• Populations in each group
must be equal• Most email populations do
not have enough equal parts to be statistically relevant
DefinitionUsing one region in multiple email creative with changes in each sent to equal populations to determine a better performing email
Pros• Multiple elements can be
tested at the same time
Local Control Groups
Cons• Sometimes hard to manage
from a population• Some people that are
active/buyers will not get a message
• Must involve pre-planning
DefinitionIsolation of a small but significant population from an email campaign to see what happens with their behavior against those that received an email
Pros• Can show the influence that
and individual email has on a campaign level
• Population must be reflective of the entire list
Universal Control Group
Cons• Isolation of a population
means that some customers don’t get an email
• Hard sell internally• Must educate various groups
internally
DefinitionIsolating a population over a longer period of time to see what their behavior is against those that receive email
Pros• Gives email a true ROI
number
CONFIDENCE INTERVALSDMA2013 | Email Testing in the Digital Age
Sample Size DeterminationThere is no perfect answer in determining sample size. It is a trade-off between sample size and the difference between A and B it is important for us to detect.
Acxiom presents a chart (as seen below) to help clients find their optimal balance.
43
The smaller the difference we want to be able to detect, the greater the required
sample size
Example Confidence Interval
• We have some results from an A/B Test:
oHow much confidence do we have in these estimates?
oDo we feel comfortable enough in the observed gain of .3% to switch to
B?
–It’s all about Sample Size when considering confidence in Response Rates.
44
Example Confidence Interval
• The 95% Confidence Interval indicates we are 95% certain the RANGE of the interval captures the True response rate
• We observed B as 0.3% greater than A.
• We can now use our confidence interval for the difference (B – A) to establish how tight that 0.3% difference is based on our sample sizes
• The range includes A being .8% greater than B all the way to B being 1.4% greater than A. With this wide range of possibilities, taking action based on our estimates becomes very dangerous
45
Example Confidence Interval
• Let’s say we have the same results, but instead of results based on samples of 1,000 each, they are now based samples of 50,000 each.
• We now have more evidence, due to our greater sample size. This results in greater belief in our results (estimates), and hence tighter intervals
• We still observe B as 0.3% greater than A, however now we can now conclude B is greater than A by between 0.1% and 0.5%.
46
THE RULES OF TESTINGDMA2013 | Email Testing in the Digital Age
buck·shot mar·ket·ing n. Marketing without a plan, clue, intelligent design or path toward success
Rules for Testing
1. What needs to be tested2. Get a plan3. Execute the test4. Report on the results
What needs to be tested
• To develop a plan, take time to define what’s broken– Mobile creative– Subjects– CTA– Lifestyle images vs action images– personalization– Audience/Segment– Landing page– Discount type
Get a Plan
1. Develop a plan that lasts from 1-3 months1. Set rules for populations
1. What determines a valid population
2. What is success in each test1. What is the KPI that will be judged2. Define from prior test/campaigns what the range of KPI success
exists
3. Determine the right amount of time to build a significance to achieve the desired KPI
4. What are your exclusions5. Get extra pairs of eyes – make it a team effort
One point to remember about the plan
• Does the result really prove the point?– Always work to validate
your testing– Sometimes thing “win”
because they’re different
• Recognize the “shiny objects”
Images Courtesy of:
Execute the Test
• Define equal populations• Define the optimal timeline for response and
adhere to it• Monitor results• Verify that the test has been carried out
Report on the Results
• Each test should have a post-mortem report– Reason– Goal– Creative– Population– Results
• Should be one page per report and stored• How does the result inform the next test
1. Track results1. You must be able to track testing results2. Determine who’s on the strike team to examine results
EXERCISESDMA2013 | Email Testing in the Digital Age
YOU NOW ALL WORK FOR
…AND GET PAID $1,000,000 A YEAR(SO DON’T SUCK)
• Come up with a testing plan for this creative
Images Courtesy of:
Small Groups• Define what you could test
(10 things over 3 months)– Cannot be subject line– Why are you testing it
• Define how you would test it• Define the audience• Define the exclusions
Assumptions• Customer is a male• 42 years old• Lives in Half Moon
Bay, CA• 20 miles south
of SF• Single
Teams will present their testing plan to the group. Best one, wins something gooooooood.
Go…now…be smart
Conclusion
• Think about your testing plan• Find equal populations• Test things that move the needle• Stop testing only subject lines
Questions?
© 2013 Acxiom Corporation. All Rights Reserved.
Ryan PhelanVice President, Strategic [email protected]
Thank you!
Keep the learning going. Follow me on twitter @ryanpphelan @acxiom
APPENDIX
Example Confidence Interval
64
How did we get the confidence intervals?
Confidence Interval for one proportion:
Test A = 0.015 +/- 1.96*SqRt ( (0.015 x 0.985) / 1,000 ) = 0.015 +/- 0.008 = (0.7%, 2.3%) Test B = 0.018 +/- 1.96*SqRt ( (0.018 x 0.982) / 1,000 ) = 0.018 +/- 0.008 = (1.0%, 2.6%)
Confidence Interval for difference between two proportions:
Test (B – A) = 0.003 +/- 1.96*SqRt ( ((0.015 x 0.985) / 1,000 ) + ((0.018 x 0.982) / 1,000) )Test (B – A) = 0.003 +/- 0.011 = Test (B – A) = (-0.8%, 1.4%)