making fashion recommendations with human-in-the-loop machine learning
TRANSCRIPT
![Page 1: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/1.jpg)
Making fashion recommendations with human-in-the-loop machine learning
Brad Klingenberg, Stitch [email protected]
Machine learning meets fashionKDD2016 | San Francisco | August 2016
Three lessons
![Page 2: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/2.jpg)
Personal styling recommendations with humans in the loop
Fashion with humans in the loop:
It works really well, but it’s complicated
![Page 3: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/3.jpg)
Personal styling recommendations with humans in the loop
Fashion with humans in the loop:
It works really well, but it’s complicated
Lesson 1: You have more than one feedback loop
Lesson 2: Human selection changes your objective function
Lesson 3: Even humans need feature selection
![Page 4: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/4.jpg)
Humans in the loop for fashionat Stitch Fix
![Page 5: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/5.jpg)
Stitch Fix
![Page 6: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/6.jpg)
Stitch Fix
![Page 7: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/7.jpg)
Stitch Fix
![Page 8: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/8.jpg)
Stitch Fix
![Page 9: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/9.jpg)
Styling at Stitch Fix
Personal styling
Inventory
![Page 10: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/10.jpg)
Styling at Stitch Fix: personalized recommendations
Inventory Algorithmic recommendations
Machine learning
![Page 11: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/11.jpg)
Styling at Stitch Fix: expert human curation
Human curation
Algorithmic recommendations
![Page 12: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/12.jpg)
Lesson 0: Humans and machines are a winning combination
![Page 13: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/13.jpg)
Combining art & science
Humans and machines are a winning combination.
![Page 14: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/14.jpg)
Combining art & science
Humans and machines are a winning combination.
Human judgement
● helps leverage unstructured data
![Page 15: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/15.jpg)
Combining art & science
Humans and machines are a winning combination.
Human judgement
● helps leverage unstructured data● provides a human element to the
process (empathy, creativity)
![Page 16: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/16.jpg)
Combining art & science
Humans and machines are a winning combination.
Human judgement
● helps leverage unstructured data● provides a human element to the
process (empathy, creativity)● frees the algorithm developer from
edge cases
![Page 17: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/17.jpg)
Lesson 1: You have more than one feedback loop
![Page 18: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/18.jpg)
Traditional recommenders
Learning through feedback
![Page 19: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/19.jpg)
Humans in the loop
Learning through feedback
![Page 20: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/20.jpg)
Humans in the loop
Learning through feedback
Human interaction
![Page 21: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/21.jpg)
Lesson 2: Human selection changes your objective function
![Page 22: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/22.jpg)
Training a model
What should you predict?
![Page 23: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/23.jpg)
Training a model
What should you predict?
Historical shipment data is plentiful
![Page 24: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/24.jpg)
Training a model
What should you predict?
Naive approach: ignore selection and train on historical shipment data
Advantages
● “traditional” supervised problem● simple historical data
![Page 25: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/25.jpg)
Training a model
Problem 1: selection can censor your data
![Page 26: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/26.jpg)
Censoring through selection
Problem: selection can censor your data
![Page 27: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/27.jpg)
Censoring through selection
Problem: selection can censor your data
![Page 28: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/28.jpg)
Censoring through selection
Problem: selection can censor your data
Arms flaunted
SuccessYes
No
Yes No
?
?
p
1-p
![Page 29: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/29.jpg)
Success given selection
Problem 2: success probabilities can make for terrible recommendations
![Page 30: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/30.jpg)
Success given selection
Problem: success probabilities can make for terrible recommendations
![Page 31: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/31.jpg)
Success given selection
Problem: success probabilities can make for terrible recommendations
“Probability the stylist should
choose this item”
![Page 32: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/32.jpg)
Success given selection
Problem: success probabilities can make for terrible recommendations
“Probability the stylist should
choose this item”
Probability the stylist will be able to send this to the right client
![Page 33: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/33.jpg)
Success given selection
clients
Y_i = 1
Y_i = 0
Example: A low-coverage item
![Page 34: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/34.jpg)
Success given selection
clients
Y_i = 1
Y_i = 0
An edgy dress. Not many people will like it, but easy for a stylist to identify these clients
Example: A low-coverage item
![Page 35: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/35.jpg)
Success given selection
clients
Y_i = 1
Y_i = 0
High score!
Example: A low-coverage item
![Page 36: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/36.jpg)
Success given selection
Example: A high-coverage item
clients
Y_i = 1
Y_i = 0
![Page 37: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/37.jpg)
Success given selection
clients
Y_i = 1
Y_i = 0
More neutral item. Many clients will like it, but it is hard for stylists to identify these clients
Example: A high-coverage item
![Page 38: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/38.jpg)
Success given selection
clients
Y_i = 1
Y_i = 0Average score!
Example: A high-coverage item
![Page 39: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/39.jpg)
A useful recommendation?
High score
Low score
![Page 40: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/40.jpg)
The lesson
Effective recommendations require understanding human selection
In both of these cases,
![Page 41: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/41.jpg)
Effective recommendations require understanding human selection
Generally, selection data will be much larger and more complicated to collect and work with
● Negative cases: logging the set of things that was available to be selected but was not selected
● Presentation effects○ similar to the way search engine results are studied
![Page 42: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/42.jpg)
Lesson 3: Even humans need feature selection
![Page 43: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/43.jpg)
Recall the styling process
Human curation
Algorithmic recommendations
![Page 44: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/44.jpg)
Creating features for the human classifier
Unstructured data can be overwhelming, even for humans
![Page 45: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/45.jpg)
Creating features for the human classifier
Derived feature 1
Derived feature 2
Feature engineering: creating useful summaries for human consumption
![Page 46: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/46.jpg)
Creating features for the human classifier
Derived feature 1
Derived feature 2
![Page 47: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/47.jpg)
Creating features for the human classifier
It is important to focus on
● Interpretability● Evidence● Orthogonality
For fashion data especially, this is hard!
![Page 48: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/48.jpg)
Creating features for the human classifier
A / B
Ultimately, this is an empirical question
Run experiments with
● Production systems● Simulations for human classifiers
![Page 49: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/49.jpg)
Personal styling recommendations with humans in the loop
Fashion with humans in the loop:
It works really well, but it’s complicated
Lesson 1: You have more than one feedback loop
Lesson 2: Human selection changes your objective function
Lesson 3: Even humans need feature selection
![Page 50: Making fashion recommendations with human-in-the-loop machine learning](https://reader034.vdocuments.net/reader034/viewer/2022052117/58ad3d0a1a28ab8b598b4e47/html5/thumbnails/50.jpg)
Thanks!
Questions?