testing & optimization - a deeper look
DESCRIPTION
TRANSCRIPT
Testing & Optimization:A Deeper Look
January 16, 2013Brendan Regan
Marketing Optimization SpecialistAnalytics Pros
BestPractices – Seattle, WA – 12.6.12
Late last year, I presented “Building a Successful Optimization Program.”
TO GET THE SLIDES:http://www.slideshare.net/analyticspros/best-practices-optimizationpresentationbrendanregan
TODAY, we’re focusing in for a deeper look…
People | Process | ToolsPeople | Process | Tools
Why is Optimization “process” worth a
webinar?I’m glad you asked.
“Process is boring.”
Process is the stuff of superheroes!
Process lets you…
1. Achieve valid, effective tests2. Take full advantage of business
opportunities
Without Process With Process
Unclear, misleading results Statistically valid results
Results that don’t add $ Results that add to the bottom line
High effort, low return Reasonable effort, huge return
Opinion-based testing Data-driven testing
“Inconclusive” results (a costly tie) Clear winners and losers
Take advantage of opportunities
1. Time to market– How fast can you get
improvements in place?2. Test the right stuff, in the right
order– Low Investment, high Return
Maximize Return Over Time
Today, You Will…
1. Learn the Scientific Method…for Marketers
2. Learn a proven Optimization process
3. Discover 4 hidden BestPractices nuggets
The Scientific Method…for Marketers
1. Define the problem2. Research/observe the problem3. Form a hypothesis4. Conduct experiment5. Analyze experiment results6. Form a conclusion7. Socialize results
Look familiar?
Optimization Process
Optimization Process
Science Marketing
1) Business Goals
Image credit: Square2Marketing.com
1) Business Goals• In science-speak, this is “define
the problem”• Resist temptation to:– skip it– assume
• Multiple goals OK– prioritize
1) Business Goals• Example:–Conversion Rate = primary business goal– Forward to a friend engagement =
secondary– Test for Conversion Rate–Analyze/monitor F2F engagement
Hidden BestPractice Nugget #1:
Test as “close to” revenue as you can. Testing further upstream can be misleading.
2) Understand Target Audience
Image credit: Closed Loop Marketing
• In science-speak, this is “research the problem”
• Generally qualitative data (“market research”)– personas– feedback forms– focus groups– social media listening– chat transcripts– call center monitoring– demographics/psychographics
2) Understand Target Audience
3) Web Data Analysis
• In science-speak, this is “research the problem”•Web Analytics– High traffic points– Decision/conversion points– Persuasion points– High abandonment points
• Survey Data
3) Data Analysis (Quantitative)
4) Heuristic Analysis
Image credit: Printed Circuit University
• In science-speak, this is “research the problem”• Are “best practices” (rules of thumb)
applied?• Are you at parity with competitors
and industry leaders?
4) Heuristic Analysis
5) Capture Challenges (roadblocks)
Image credit: YoJoe.com
No, not this kind of Roadblock.
5) Capture Challenges (roadblocks)
Image credit: ArtsJournal.com
• Challenges presented by your site/marketing to your prospects• Ex: “Landing page lacks ‘social
proof’ content.”
5) Capture Challenges (roadblocks)
6) Challenges » Hypotheses
• Every challenge/roadblock has potential solution(s)• Best guess, best solution =
hypothesis• Ex: “Adding ‘social proof’
content will increase conversion rate.”
6) Challenges » Hypotheses
Hidden BestPractice Nugget #2:• Hypothesis: “Adding ‘social proof’
content will increase conversion rate.”• Research Question: “Which version of
social proof content will perform best?”
7) Design Experiment
• Formal documentation• Capture:– Goal(s)– Problem/challenge– Quantitative/Qualitative proof– Hypothesis– Specifications
7) Design Experiment
Hidden BestPractice Nugget #3:“Confidence level” - the statistical probability that your results are not due to chance.
Own your confidence level. Most tools are 95%. Get buy-in on tool’s default or other confidence level.
8) Design Variations
• Sketch/wireframe• Copy deck• Designs/mockups• Approval process–Per experiment design–Marketing/Legal blessing
8) Design Variations
9) Build Variations
• You may have to write code • You may “throw away” code• Page load speed needs to be
consistent!
9) Build Variations
10) Implement Experiment
• Integrate w/ testing tool• Test before you test– X-browser– Acceptance testing?
• Launch & Monitor– 7 days minimum to mitigate “day-of-week”– Patience, please
10) Implement Experiment
11) Analyze Results
No, not this kind of analysis.
11) Analyze Results
• Google Analytics integrations– Optimizely, Visual Website Optimizer, etc.
11) Analyze Results
•What does the result mean about the target audience?• Are the learnings applicable
elsewhere?– Other sites? – Other campaigns? – Other channels?
11) Analyze Results
12) Report Findings
• If +, tie it to revenue– “estimated”
12) Report Findings
• If +, tie it to revenue– “estimated”
• If -, what did you learn?
12) Report Findings
• Install winning version– You don’t make $ until you install
the winner!– Production-ready code, SEO, etc.
• Follow-up test ideas• Add to Marketing knowledge
base
13) Next Steps
Hidden BestPractice Nugget #4:
Optimize your Optimization. Stay lean, learn as you go, smooth out the process using Six Sigma or other methodologies.
Image credit: htt
p://rahuleink.wordpress.com
/2010/04/23/what-is-six-sigm
a/
Key Takeaways
1. Optimization Process is your friend2. Optimization Process maps to science…– …but bows to Marketing
3. Optimize your own Optimization efforts– “…the best is the enemy of the good.”
Voltaire (1694–1778)
Questions?