discovering context effects from raw choice data13-11-00)-13... · 2019-06-10 · discovering...
TRANSCRIPT
![Page 1: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/1.jpg)
Discovering Context Effects
from Raw Choice Data
ARJUN SESHADRI, STANFORD UNIVERSITY
ALEX PEYSAKHOVICH, FACEBOOK ARTIFICIAL INTELLIGENCE RESEARCH
JOHAN UGANDER, STANFORD UNIVERSITY
ICML 2019
![Page 2: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/2.jpg)
Modelling in Discrete Choice
Data of the form where “alternative is chosen from the set ” and is a subset of , the universe of alternatives
Discrete choice settings are ubiquitous
![Page 3: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/3.jpg)
Modelling in Discrete Choice
Data of the form where “alternative is chosen from the set ” and is a subset of , the universe of alternatives
Discrete choice settings are ubiquitous
![Page 4: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/4.jpg)
Encompasses Many Fields
Recommender Systems
Inverse reinforcement learning
Virtual Assistants
Structural Modeling
![Page 5: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/5.jpg)
Independence of Irrelevant
Alternatives (IIA) Fully determines the workhorse Multinomial Logit (MNL) Model
Main (strong) assumption:
The Good:
inferentially tractable, powerful, and interpretable
The Bad:
When IIA does not hold, out of sample predictions are wildly miscalibrated
Cannot account for the wide literature on context effects (e.g. Compromise Effect)
Size
Sa
vin
gs
Compromise Effect
![Page 6: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/6.jpg)
Problems we address
Modelling individual choice behavior
Behavioral economics “anomalies” are all over the place
Search Engine Ads (Ieong-Mishra-Sheffet ’12, Yin et al. ’14)
Google Web Browsing Choices (Benson-Kumar-Tomkins ’16)
Need to model while retaining parametric and inferential efficiency
Statistical tests for violations of IIA
General, global tests are intractable (Seshadri & Ugander ‘19, Long & Freese ‘05)
Model based approaches challenging due to identifiability issues (Cheng & Long, ‘07)
“ad group quality”
![Page 7: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/7.jpg)
Context Dependent Utility Model
(CDM)
Developing the CDM
Universal logit model (McFadden et al., ’77)
![Page 8: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/8.jpg)
Context Dependent Utility Model
(CDM)
Developing the CDM
Universal logit model (McFadden et al., ’77)
Decompose the model (Batsell & Polking, ’85)
![Page 9: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/9.jpg)
Context Dependent Utility Model
(CDM)
Developing the CDM
Universal logit model (McFadden et al., ’77)
Decompose the model (Batsell & Polking, ’85)
Truncate to 2nd
order (effects are pairwise)
Full Rank CDM
![Page 10: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/10.jpg)
Context Dependent Utility Model
(CDM)
Developing the CDM
Universal logit model (McFadden et al., ’77)
Decompose the model (Batsell & Polking, ’85)
Truncate to 2nd
order (effects are pairwise)
Full Rank CDM
Make a low rank approximation (parameters linear in items)
Low Rank CDM
![Page 11: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/11.jpg)
Context Dependent Utility Model
(CDM)
Developing the CDM
Universal logit model (McFadden et al., ’77)
Decompose the model (Batsell & Polking, ’85)
Truncate to 2nd
order (effects are pairwise)
Full Rank CDM
Make a low rank approximation (parameters linear in items)
Low Rank CDM r-dimensional
latent feature
vector
r << n items
Other items
change how
features are
traded off
![Page 12: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/12.jpg)
A Theoretical Preview
![Page 13: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/13.jpg)
A Theoretical Preview
Identifiability
Sufficient:
Necessary:
More generally:
![Page 14: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/14.jpg)
A Theoretical Preview
Identifiability
Sufficient:
Necessary:
More generally:
Convergence Guarantees
![Page 15: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/15.jpg)
A Theoretical Preview
Identifiability
Sufficient:
Necessary:
More generally:
Convergence GuaranteesHypothesis Testing
![Page 16: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/16.jpg)
Unifying Existing Choice Models
Low Rank CDM
![Page 17: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/17.jpg)
Unifying Existing Choice Models
Tversky-Simonson Model
Low Rank CDM
(Tversky & Simonson, 1993)
![Page 18: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/18.jpg)
Unifying Existing Choice Models
Tversky-Simonson Model
Low Rank CDM
Batsell-Polking Model
(Tversky & Simonson, 1993)
(Batsell & Polking, 1985)
![Page 19: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/19.jpg)
Unifying Existing Choice Models
Tversky-Simonson Model
Low Rank CDM
Blade-Chest Model
Batsell-Polking Model
(Tversky & Simonson, 1993)
(Batsell & Polking, 1985)
(Chen & Joachims, 2016)
![Page 20: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/20.jpg)
An Empirical Preview: Performance
and Interpretability
![Page 21: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/21.jpg)
An Empirical Preview: Performance
and Interpretability
Transportation Preferences (Koppelman & Bhat, ‘06)
Survey of transportation choices for residents in various San Francisco neighborhoods
Low Rank CDMs significantly outperform MNL and MMNL
![Page 22: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/22.jpg)
An Empirical Preview: Performance
and Interpretability Not Like the Other (Heikinheimo & Ukkonen, ‘13)
Individuals are shown triplets of nature photographs
asked to choose photo most unlike the other two
CDM illustrates intuitive property of dataset: similar items have negative target-context inner product
Induces grouping by similarity in both target and context vectors
Transportation Preferences (Koppelman & Bhat, ‘06)
Survey of transportation choices for residents in various San Francisco neighborhoods
Low Rank CDMs significantly outperform MNL and MMNL
![Page 23: Discovering Context Effects from Raw Choice Data13-11-00)-13... · 2019-06-10 · Discovering Context Effects from Raw Choice Data ARJUN SESHADRI, STANFORD UNIVERSITY ALEX PEYSAKHOVICH,](https://reader034.vdocuments.net/reader034/viewer/2022042909/5f3a9d5d3963b178e055f1f9/html5/thumbnails/23.jpg)
Conclusions
CDM models context effects with efficiency guarantees and enables
practical tests of IIA
Can be easily applied to many pipelines by modifying “the final layer”
Simultaneously brings both:
Machine Learning rigor to Econometrics models (identifiability, convergence)
Econometrics modeling (choice set effects) into Machine Learning research
Thanks!!
Discovering Context Effects from Raw Choice DataArjun Seshadri, Alex Peysakhovich, and Johan Ugander
Poster: Pacific Ballroom #234