airbnb offline experiments
DESCRIPTION
Elena Grewal presented these slides on a/b testing in the real world (offline experiments not online) at the Big Data Innovation Summit on April 9, 2014.TRANSCRIPT
![Page 1: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/1.jpg)
A/B Testing In The Real WorldHow to run experiments in an “offline” setting
Elena Grewal 2014-04-09
Big Data Innovation Summit
![Page 2: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/2.jpg)
The Plan
2
1
2
“Offline” experiments: what and why?
Some experiment pitfalls and advice
3 Conclusions
![Page 3: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/3.jpg)
But first, you might ask: What is Airbnb?
![Page 4: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/4.jpg)
Airbnb is an online marketplace for accommodations
4
Part of the “sharing economy”
![Page 5: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/5.jpg)
Search in San Francisco
5
![Page 6: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/6.jpg)
Come Stay In My Home!
6
![Page 7: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/7.jpg)
That looks like a website. What do we mean by “offline”?
![Page 8: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/8.jpg)
Guest Journey
![Page 9: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/9.jpg)
Host Journey
![Page 10: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/10.jpg)
Offline Operations Departments
+ Customer Support
+ Local Operations
+ Professional Photography
+ Many others…
10
![Page 11: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/11.jpg)
Customer Support
![Page 12: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/12.jpg)
Local Ops Teams
![Page 13: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/13.jpg)
Photography
!
+ 3,000 Photographers worldwide
+ Over 100k listings photographed
+ Almost 2 million professional photos13
![Page 14: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/14.jpg)
Stepping back
14
+ Many companies have offline operations + Can optimize these using experiments
!
!
!
Online Experiments:
We run these all the time too.
If you are curious about on our online experimentation see Jan Overgoor’s tech talk
http://nerds.airbnb.com/tech-talks/
![Page 15: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/15.jpg)
Why Do We Need Experiments?
![Page 16: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/16.jpg)
Before and after won’t work
16
• Often very little data before professional photos are added • Seasonality and other confounding factors bias results
![Page 17: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/17.jpg)
Selection bias often impacts analysis
17
• Listings that opt to get professional photography are not the same as listings that do not get photography
![Page 18: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/18.jpg)
Without an experiment, we don’t know the causal effect
18
This is the same reason we need online experiments
Date01−01 01−15 02−01 02−15 03−01 03−15
Product Launch
Product Rollback
Launch initiative: e.g. Offered Free Professional Photography
![Page 19: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/19.jpg)
Traditional A/B Testing Online
Great sources: http://mcfunley.com/design-for-continuous-experimentation http://www.evanmiller.org/how-not-to-run-an-ab-test.html
Control Treatment
19
![Page 20: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/20.jpg)
-5%
-4%
-3%
-2%
-1%
0%
1%
2%
3%
4%
5%
0 4 8 12 16 20 24 28 32 36
Del
ta
Treatment Effect for Price Filter Experiment
Initial Results Look Good
20
Δ > 0 : “positive”
0.00
0.10
0.20
0.30
0.40
0 4 8 12 16 20 24 28 32 36
p-va
lue
Days since start of experiment
P-Value
p < 0.05 : “significant”
![Page 21: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/21.jpg)
-5%
-4%
-3%
-2%
-1%
0%
1%
2%
3%
4%
5%
0 4 8 12 16 20 24 28 32 36
Del
ta
Treatment Effect for Price Filter Experiment
Actually, NeutralStatistical significance by itself does not tell the whole story
p = 0.4 : “noise”
Δ = 0 : “neutral”
21
0.00
0.10
0.20
0.30
0.40
0 4 8 12 16 20 24 28 32 36
p-va
lue
Days since start of experiment
P-Value
p < 0.05 : “significant”
![Page 22: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/22.jpg)
Offline Experiment Examples
![Page 23: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/23.jpg)
Professional PhotographyLet’s run an experiment!
23
More bookings?
![Page 24: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/24.jpg)
Beware of CannibalizationThe unit of randomization depends on the effect we want to estimate
24
!
!
![Page 25: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/25.jpg)
Local Operations: Market Level Experiment
25
!
+ Smaller “long tail” markets < 100 reviewed listings
Randomize Markets 93 Treatment / 92 Control
Assess impact of operational strategy on market growth
+ Statistically measure the lift due to local ops teams + Measuring active listings, hosts, reviewed listings, and
bookings
![Page 26: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/26.jpg)
Market Distribution U.S. & Europe
26
![Page 27: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/27.jpg)
Finding: Local Ops Efforts Have Positive Impact on Growth
27
Active Listings
Control17% Growth
Local Ops Kickoff
Treatment 31% Growth
![Page 28: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/28.jpg)
Case Study: Campos do Jordão, BR
28
+ Market grew 9x + Over 90% of the new listings are from new users + Low CPA + Primary approach is phone sales + Other approaches were less successful
+ 862%
+ 7%
Use qualitative research to understand what happened
Active Listing Growth
Treatment Control
![Page 29: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/29.jpg)
Host EducationImproving listings through outreach
29
+ Initially not launched as an experiment and found positive impact + Launched as an experiment and found neutral impact + Don’t need market level approach here! !
![Page 30: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/30.jpg)
Some takeawaysUse context to improve operations
30
+ Can investigate heterogeneity in treatment effects with higher N + Word of caution: can’t just compare those who were reached
by a call or email to the control (selection bias strikes again)
![Page 31: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/31.jpg)
Compare entire treatment to entire control
31
!Treatment
!!
!Control
!!
Called
vs.
![Page 32: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/32.jpg)
Additional Offline vs. Online Considerations
32
+ Opt-in biases + You know you are in an experiment (Hawthorne/John Henry effects) + Monetary incentives impact external validity, trade-off take-up rate
+ Takes time to adjust to a change + Sample size may be limited by ops capacity + Stakeholders may be less data-savvy + Real people delivering the experiment! + Ethical considerations !
Always partner with customer support.
!
![Page 33: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/33.jpg)
Takeaways
+ Controlled experiments are the way to go if you want to make causal inference + Use them to optimize operations! !
but: + Level of randomization - what impact do you want to measure? + Cannibalization + Compare the right groups - no selection bias + Break down results to get the most from the analysis + Be practical/ethical - you are dealing with real people here
33
![Page 35: Airbnb offline experiments](https://reader035.vdocuments.net/reader035/viewer/2022081413/548141f6b4795937578b4884/html5/thumbnails/35.jpg)