expt panel hive_data_rp_20130320_final-1
Post on 16-Apr-2017
129 Views
Preview:
TRANSCRIPT
Controlled Experimentation (A/B Testing)!• Method to study effects of a treatment#
• Concept:!- Randomly split users into two groups#
➥ A : Control#
➥ B: Treatment#- A and B are identical to each other except
for the treatment being evaluated#- Collect performance metrics from the
experiment#- Run statistical tests to determine if
differences between A and B are purely by chance#
2
Randomly Divide
A (Control) B (Treatment)
Measure & Evaluate
Controlled Experimenta=on Panel
Why Run Controlled Experiments?!• Commonly used approach in clinical trials!- What is the effect of a particular drug / treatment?#
• Systematically validate hypotheses with data!!• Concurrently run the treatment and control!- The difference (if any) is#➥ Because of the treatment OR#➥ Due to random chance#
• Determine if a treatment is causal in nature!- E.g., Making the search box bigger causes increase in queries / user#
3 Controlled Experimenta=on Panel
Controlled Experimentation: Use Cases! #
4 Controlled Experimenta=on Panel
A B Stract Widget Company
_________________ _________________ _________________ _________________ BUY NOW
A B Stract Widget Company
_________________ _________________ _________________ _________________ BUY NOW
Website Variants
Controlled Experimentation: Use Cases! #
5 Controlled Experimenta=on Panel
Free Trial Play Now
Mobile Call to Ac=on
Controlled Experimentation: Use Cases! #
6 Controlled Experimenta=on Panel
Top deal highlighted
Email Template Design
Controlled Experimentation: Use Cases! #
7 Controlled Experimenta=on Panel
Backend changes (e.g., Personaliza=on Algorithm)
8
Controlled Experimentation: Use Cases!
# • Follow-up message for users who previously clicked on an ad#
• Incentive campaign to re-engage lapsed users#
• Think of this as placing filters / guards on a randomly chosen user population#
Controlled Experimenta=on Panel
Custom Defined User Segments
9
Key Components of an Experimentation Platform!Hashing function!
!
!
!!
!
Logging!
!
!
!
!
Metrics – suite of KPI!
!
!
!!
!
Dashboard!
F( ) Group 0
Group 1
Group N-‐1
Time Spent
Revenue
Click-‐Through Rate
Session Length
Abandonment
Purchase Rate
• Metric improvements and Sta=s=cal Significance in a central place
• Detailed logging of all user interac=ons
Controlled Experimenta=on Panel
Ensure Identical Control and Treatment!
• Custom Segments#
10
CONTROL TREAMENT
Gender
Male
Female
CONROL TREATMENT
Region Size
Small
Medium
Large
CONROL TREATMENT
Prior Exposure
No
Yes
δ%
Controlled Experimenta=on Panel
• Frequency Distribution#
• Large Difference in Prior Exposure Rate violates assumptions#
A/A Tests!• Run an experiment with two identical variants#
• Helps to determine if:#- Users are being split uniformly at random#- Correct data is being logged#- Variance between identical populations of users is acceptable#
• Challenge:!- Few purchases of high value deals render statistically significant
difference between treatment and control#
11
SPAIN TRIP $1,999
Controlled Experimenta=on Panel
Monitor Each Variant!• Place yourself in each variant
to validate the experience#!
!
12
Carefully inspect each variant
Controlled Experimenta=on Panel
• Wrong sort order!!
!
Objective Function!#
#
Conversion Revenue
P(conversion)
• Favors lower price deals
E(rev) = P(conversion) * price
• More expensive deals can dominate
Need to balance mul=ple, oZen conflic=ng objec=ves
13 Controlled Experimenta=on Panel
Measure Overall Impact!• Test focuses on#- A particular area of
the website#- A sub-population of
users#
• Measure!- Improvement on the
sub-segment AND#- Entire population!#
14
Measure overall impact to guard against cannibaliza=on
Controlled Experimenta=on Panel
Thanks to many talented individuals at Groupon I am privileged to work with!#• Data Science#• Engineering#• Marketing / Market Research#
Acknowledgements#
16 Controlled Experimenta=on Panel
top related