groupon_controlled experimentation_panel_the hive
TRANSCRIPT
![Page 1: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/1.jpg)
!Rajesh Parekh!
Controlled Experimentation to Guide Product Innovation!
![Page 2: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/2.jpg)
Controlled Experimentation (A/B Testing)!• Method to study effects of a treatment#
• Concept:!- Randomly split users into two groups#
� A : Control#
� B: Treatment#- A and B are identical to each other except
for the treatment being evaluated#- Collect performance metrics from the
experiment#- Run statistical tests to determine if
differences between A and B are purely by chance#
2
Randomly)Divide)
A)(Control)) B)(Treatment))
Measure)&)Evaluate)
Controlled)Experimenta=on)Panel)
![Page 3: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/3.jpg)
Why Run Controlled Experiments?!• Commonly used approach in clinical trials!- What is the effect of a particular drug / treatment?#
• Systematically validate hypotheses with data!!• Concurrently run the treatment and control!- The difference (if any) is#� Because of the treatment OR#� Due to random chance#
• Determine if a treatment is causal in nature!- E.g., Making the search box bigger causes increase in queries / user#
3 Controlled)Experimenta=on)Panel)
![Page 4: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/4.jpg)
Controlled Experimentation: Use Cases! #
4 Controlled)Experimenta=on)Panel)
A"B"Stract"Widget"Company"
_________________)_________________)
_________________)_________________) BUY)NOW)
A"B"Stract"Widget"Company"
_________________)_________________)
_________________)_________________) BUY"NOW"
Website)Variants)
![Page 5: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/5.jpg)
Controlled Experimentation: Use Cases! #
5 Controlled)Experimenta=on)Panel)
Free)Trial) Play)Now)
Mobile)Call)to)Ac=on)
![Page 6: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/6.jpg)
Controlled Experimentation: Use Cases! #
6 Controlled)Experimenta=on)Panel)
Top)deal)highlighted)
Email)Template)Design)
![Page 7: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/7.jpg)
Controlled Experimentation: Use Cases! #
7 Controlled)Experimenta=on)Panel)
Backend)changes)(e.g.,)Personaliza=on)Algorithm))
![Page 8: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/8.jpg)
8
Controlled Experimentation: Use Cases!
# • Follow-up message for users who previously clicked on an ad#
• Incentive campaign to re-engage lapsed users#
• Think of this as placing filters / guards on a randomly chosen user population#
Controlled)Experimenta=on)Panel)
Custom)Defined)User)Segments)
![Page 9: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/9.jpg)
9
Key Components of an Experimentation Platform!Hashing function!!
!
!
!
!
Logging!!
!
!
!
Metrics – suite of KPI!!
!
!
!
!
Dashboard!
F())))))))))))))
Group)0)
Group)1)
Group)NU1)
Time"Spent"
Revenue"
Click>Through"Rate"
Session"Length"
Abandonment"
Purchase"Rate"
• Metric)improvements)and)Sta=s=cal)Significance)in)a)central)place)
• Detailed)logging)of)all)user)interac=ons)
Controlled)Experimenta=on)Panel)
![Page 10: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/10.jpg)
Ensure Identical Control and Treatment!
• Custom Segments#
10
CONTROL" TREAMENT"
Gender""
Male)
Female)
CONROL" TREATMENT"
Region"Size"
Small)
Medium)
Large)
CONROL" TREATMENT"
Prior"Exposure"
No)
Yes)
δ%"
Controlled)Experimenta=on)Panel)
• Frequency Distribution#
• Large Difference in Prior Exposure Rate violates assumptions#
![Page 11: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/11.jpg)
A/A Tests!• Run an experiment with two identical variants#
• Helps to determine if:#- Users are being split uniformly at random#- Correct data is being logged#- Variance between identical populations of users is acceptable#
• Challenge:!- Few purchases of high value deals render statistically significant
difference between treatment and control#
11
SPAIN"TRIP"$1,999"
Controlled)Experimenta=on)Panel)
![Page 12: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/12.jpg)
Monitor Each Variant!• Place yourself in each variant
to validate the experience#!
!
12
Carefully)inspect)each)variant)
Controlled)Experimenta=on)Panel)
• Wrong sort order!!
!
![Page 13: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/13.jpg)
Objective Function!#
#
Conversion" Revenue"
P(conversion)))
• Favors)lower)price)deals)
E(rev))=)P(conversion))*)price))
• More)expensive)deals)can)dominate)
Need)to)balance)mul=ple,)oZen)conflic=ng)objec=ves))
13 Controlled)Experimenta=on)Panel)
![Page 14: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/14.jpg)
Measure Overall Impact!• Test focuses on#- A particular area of
the website#- A sub-population of
users#
• Measure!- Improvement on the
sub-segment AND#- Entire population!#
14
Measure)overall)impact)to)guard)against)cannibaliza=on)
Controlled)Experimenta=on)Panel)
![Page 15: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/15.jpg)
Panel Discussion: Questions!
15 Controlled)Experimenta=on)Panel)
![Page 16: Groupon_Controlled Experimentation_Panel_The Hive](https://reader035.vdocuments.net/reader035/viewer/2022062419/55a22dc61a28abf3348b4605/html5/thumbnails/16.jpg)
Thanks to many talented individuals at Groupon I am privileged to work with!#• Data Science#• Engineering#• Marketing / Market Research#
Acknowledgements#
16 Controlled)Experimenta=on)Panel)