opticon 2015-hypothesize, analyze, optimize: using hidden insights to perfect your experiment...
TRANSCRIPT
#opticon2015
Hypothesize, Analyze, Optimize:
Using Hidden Insights to Perfect your Experiment Process
Abby Lokshin Mobile Product Manager, Rue La La
Abel Ramos Product Manager, Ecommerce Princess Cruises
@abelramos58 [email protected]
#opticon2015
Iteration – The Key to your Testing Success Abel Ramos Product Manager, Ecommerce Princess Cruises
@abelramos58 [email protected]
“Every strike brings me closer to the next homerun.” – Babe Ruth
“We Failed”
Our Hypothesis
The next greatest search UI ……or so we thought.
Oops, What Happened?
Let the fun begin. Where do we start with these types of results? Time to regroup.
Introducing the Next Greatest Search UI…… no, really
OK, Someone Want to Interpret These Results?
Could we be getting closer?
“Hey, what about our mobile traffic?”
– Joseph Espana, Web Designer Princess Cruises
Understanding your Business Segments
How is your Mobile Traffic affecting your results ?
“Life is a series of experiences, each one of which makes us
bigger, even though sometimes it is hard to realize this.”
– Henry Ford
Know your Audience
Segment your website traffic for your tests.
What are the Obstacles Preventing Testing Success? How are your promotional efforts affecting your testing?
Review all of your Data Points Constantly
Use all available data resources to help with your testing.
What Does the Future Hold for Testing at Princess Cruises?
Are your testing goals inline with larger company initiatives?
“We Won” Testing is an iterative process and it takes time.
#opticon2015
Thank you!
Abel Ramos Product Manager, Ecommerce Princess Cruises
@abelramos58 [email protected]
#opticon2015
What should we test (again)?
The testing lifecycle
Abby Lokshin Mobile Product Manager, Rue La La
#opticon2015
RUE LA LA
LEADING E-COMMERCE LIFESTYLE DESTINATION
15M+ MEMBERS
NEW BOUTIQUES DAILY
EXPIRE IN 2-3 DAYS
AN OVERVIEW
RUE LA LA USER FLOW
Kicking off the testing process
TEST IDEATION WHAT HAPPENS
MULTIPLE STAKEHOLDERS
UNIQUE KPIS
DISORGANIZED BACKLOG
Determine common themes to define primary goals
TESTING BUCKETS GROUPED BY PRIMARY GOALS
Prioritization
TEST PRIORITIZATION 60 SECOND BUSINESS CASE
Coordination Effort Creative Effort
Initial Development
EffortRollout Effort
Strategic Alignment
Revenue Impact
Customer Experience
Value
Lowers Cost PM Priority
RANK: HIGH, MEDIUM, LOW
SCORE: 100 PTS. TOTAL
+
EFFORT
BUSINESS VALUE
TEST PRIORITIZATION 60 SECOND BUSINESS CASE
An organized backlog leads to a stronger sprint
EXPIRING FREE SHIPPING TEST
TESTING GOALS:• USER SEES QUICK TIP
• USER ADDS TO CART
• USER CHECKS OUT
• PRIMARY GOAL: REVENUE
RESULTS: 4.2%+ CONVERSION
NEXT STEPS: ROLL OUT!
QUICK TIP ON PRODUCT DETAIL
LEVEL OF EFFORT: LOW
BUSINESS VALUE: HIGH
What if the test is inconclusive?
MEN’S SUIT SHOP MIX & MATCH BOUTIQUE
TESTING GOALS:• USER BUILDS A SUIT
• USER ADDS TO CART
• USER CHECKS OUT
• PRIMARY GOAL: REVENUE
RESULTS: NO CLEAR WINNER
What impacted the primary goal?
MEN’S SUIT SHOP SEVERAL ISSUES
TOO MANY STEPS
CONFUSING UI
AUDIENCE TOO SMALL
STATISTICALLY INSIGNIFICANT
MEN’S SUIT SHOP NEXT STEPS
RE-RUN, WITH MODIFICATIONS:• FEMALE AUDIENCE
• FEWER STEPS
• LONGER TIME FRAME
• MORE BOUTIQUES
What if converting isn’t the problem?
CATEGORY SORT TEST SHOPPING PREFERENCES
TESTING GOALS:• USER INTERACTS WITH
SORTING
• USER CHECKS OUT
• PRIMARY GOAL: REVENUE
RESULTS: NO CLEAR WINNER
VAR. A VAR. B
CATEGORY SORT TEST WHAT HAPPENED
NO IMPACT ON THE WHOLE
MANY CATEGORY COMBINATIONS
CAN WE FIND ANY TRENDS?
VAR. A VAR. B
CATEGORY SORT TEST INTERESTING TRENDS
TREND IN VARIATION A: FEMALES RANK WOMEN’S FIRST
• 4%+ IN $/VISIT
TREND IN VARIATION B: MALES RANK MEN’S FIRST
• 31%+ IN $/VISIT
• 32%+ IN PURCHASES
VAR. A VAR. B
CATEGORY SORT TEST INTERESTING TRENDS
NEW TESTS:• DEFAULT MEN TO MEN’S
SECTION
• RE-OPEN TO LAST VIEWED CATEGORY
VAR. A VAR. B
REMEMBER… TESTING IS NEVER ONE AND DONE
CONSIDER THE DATA
COLLECT FEEDBACK
REMEMBER THE PRIMARY GOAL
KEEP ITERATING
#opticon2015
Questions