product management for ai/ml
TRANSCRIPT
![Page 1: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/1.jpg)
Product Management for AI/ML
The Product Mentor, Season 7
Resources available:https://goo.gl/TfUxac
![Page 2: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/2.jpg)
Chris ButlerDirector of Prod Strat @ Philosophie NYC
The Best Product Person 2016
17 years of product and BD
Microsoft, Waze, Horizon Ventures, KAYAK, and started my own company (failed)
@chrizbot
![Page 3: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/3.jpg)
![Page 4: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/4.jpg)
Product management for AI/ML
● What do I need to know about these things?
● How do they impact product’s role○ Purpose and strategy
○ Learning
○ Building
○ Prioritizing
○ Measuring
○ Technical
![Page 5: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/5.jpg)
What is...artificial intelligence?...artificial general intelligence?...narrow artificial intelligence?
...a neural network?...machine learning?
...deep learning?…?
![Page 6: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/6.jpg)
![Page 7: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/7.jpg)
Where to start?
![Page 8: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/8.jpg)
Learns from from (good) data
![Page 9: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/9.jpg)
Attempts to reduce an error against desired outcomes
![Page 10: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/10.jpg)
![Page 11: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/11.jpg)
Why are AI programs different?
● Content: models, not programs
● Process: training, not debugging
● Release: retraining, not patching
● Uncertainty: of objective
● Uncertainty: of action and recommendation
● Uncertainty: propagates through model
![Page 12: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/12.jpg)
Training and inference
![Page 13: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/13.jpg)
Types of problems it can solve (possibly)
● Ranking - Google search results
● Recommendation - Netflix movie recommendations
● Regression (or prediction) - Zillow predicting house prices
● Classification - Image is a cat or dog
● Clustering - Tumblr social network analysis to find groups
of topics
![Page 14: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/14.jpg)
● Supervised
● Unsupervised
● Supervised
● Unsupervised
● Reinforcement
● Semi-supervised
● One shot
● Few shot
Types of learning
● Transfer
● Active
● Imitation
● Q
● Transduction
● ...
![Page 15: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/15.jpg)
Resources to start learningBooks
● Programming Collective Intelligence by Toby Segaran● The Master Algorithm by Pedro Domingos● Introduction to Machine Learning by Nils Nilsson● Data Mining by Ian Whitten● Data Science for Business by Foster Provost● Neural Networks and Deep Learning by Michael
Nielsen● Make Your Own Neural Network by Tariq Rashid
Courses
● Introduction to Machine Learning by Andrew Ng (highly recommended)
● Machine Learning Engineer by Udacity● Machine learning is Fun! by Adam Geitgey● How to use Tensorflow for Classification by Siraj
Raval● Learning AI if you suck at Maths by Daniel Jeffries● Machine Learning Mastery by Jason Brownlee● Machine Learning by Georgia Tech (Udacity)
Must-Reads:
● WTF is Artificial Intelligence by Sam DeBrule● Machine learning for Product Managers by Ken
Norton● AI, Deep Learning, and Machine Learning: A Primer
by Frank Chen● Artificial Intelligence is the new electricity by Andrew
Ng● The current state of Machine Learning by Shivon Zilis● How Google is remaking itself a ‘Machine Learning
First’ company by Steven Levy● An executives guide to machine learning by Dorian
Pyle (Mckinsey)● Experience Design in the Machine Learning Era by
Fabien Girardin● A human’s guide to Machine learning by Sam DeBrule
(subscribe to his newsletter)● What every manager should know about Machine
Learning by Mike Yeomans● An introduction to Machine Learning theory and its
application by Nick Mccrea● Machine Learning pitfalls by Ben Hamner
https://hackernoon.com/machine-learning-and-product-managers-930b691b1b37
![Page 16: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/16.jpg)
Fast.ai MOOC is very hands on
![Page 17: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/17.jpg)
Unless you are in research, the real focus should be on what differentiates your product and gives it meaning. Not finding a better way to detect the difference between cats and dogs in ImageNet images.https://uxdesign.cc/robots-need-love-too-empathy-mapping-for-ai-59585ad3548d
![Page 18: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/18.jpg)
Purpose and strategy
![Page 19: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/19.jpg)
![Page 20: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/20.jpg)
![Page 21: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/21.jpg)
![Page 22: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/22.jpg)
![Page 23: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/23.jpg)
![Page 24: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/24.jpg)
Without human purpose, a computer is just a rock that we tricked into thinking.
https://uxdesign.cc/robots-need-love-too-empathy-mapping-for-ai-59585ad3548d
![Page 25: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/25.jpg)
![Page 26: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/26.jpg)
![Page 27: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/27.jpg)
Design Thinking, Lean, and Agile?
![Page 28: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/28.jpg)
Design Thinking(AKA Human-Centered Design)
![Page 29: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/29.jpg)
![Page 30: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/30.jpg)
Human at the center being assisted and augmented by AI/ML
![Page 31: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/31.jpg)
Lean
![Page 32: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/32.jpg)
![Page 33: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/33.jpg)
Start small with simple models that build confidence in what you are doing
![Page 34: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/34.jpg)
Agile
![Page 35: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/35.jpg)
![Page 36: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/36.jpg)
Continuously improve the data, the model, training, etc.
![Page 37: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/37.jpg)
Design Thinking, Lean, Agile, and AI are about emergent practice
![Page 38: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/38.jpg)
Design Thinking, Lean, Agile, and AI are about learning and adapting
![Page 39: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/39.jpg)
How to learn about purpose
![Page 40: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/40.jpg)
When using AI, you want to know:
● Are we helping solve a problem?
● Do they trust the information?
● Do they feel comfortable giving feedback to the system?
![Page 41: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/41.jpg)
Rich Picture
![Page 42: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/42.jpg)
Empathy Mapping (for the machine)
![Page 43: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/43.jpg)
Confusion matrix
![Page 44: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/44.jpg)
Decision boundaries
![Page 45: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/45.jpg)
Patterns for building with purpose
![Page 46: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/46.jpg)
Self driving cars classifications
![Page 47: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/47.jpg)
Does everything Learning Watching
Approving Confident Recommending
Veto’ing Proven Taking action
Human System State Machine Action
![Page 48: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/48.jpg)
Intelligent CTA
![Page 49: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/49.jpg)
“Calculating” and explainability
![Page 50: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/50.jpg)
Be up front about possible errors
and sometimes wrong
![Page 51: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/51.jpg)
Feedback
![Page 52: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/52.jpg)
Wizard of Oz to learn
![Page 53: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/53.jpg)
Bootstrapping
![Page 54: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/54.jpg)
How to prioritize with purpose
![Page 55: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/55.jpg)
Prioritization
![Page 56: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/56.jpg)
Outcome Mapping
![Page 57: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/57.jpg)
How to measure with purpose
![Page 58: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/58.jpg)
Research questions
● Think back to the last time you did this, how did you come to
that decision?
● Do you trust these suggestions for what to do next?
● How do you think the system decided [action]?
● Was there enough information for you to [take action]?
● How much do you trust the system to make the right decision in
the future? It is more or less than before?
![Page 59: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/59.jpg)
Gathering feedback from people
![Page 60: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/60.jpg)
Gathering feedback from people
![Page 61: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/61.jpg)
Back to the confusion matrix!
![Page 62: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/62.jpg)
Technical concerns
![Page 63: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/63.jpg)
Simulation and QA
![Page 64: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/64.jpg)
Scaling
![Page 65: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/65.jpg)
Technical debt
***
![Page 66: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/66.jpg)
In closing
![Page 67: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/67.jpg)
In closing
● We give machines their purpose - focus on problems you
are solving, not new toys
● Building these systems are a journey - iterate and learn
● We deal with nondeterministic systems all day in our
teams, industries, and markets - AI is no different
![Page 68: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/68.jpg)
Don’t get stuck with a rock that doesn’t help you meet your purpose
![Page 70: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/70.jpg)
![Page 71: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/71.jpg)
![Page 72: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/72.jpg)
Appendix: removed slides
![Page 73: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/73.jpg)
Background for internal review● Audience: product people (new and experienced)
● When: 9/17/17
● Alternate use: planning on using parts for Design Thinking for AI workshops
● Feedback needed:○ Good enough overview of AI? Design Thinking/Lean?
○ Does it feel like a good journey/order?
○ Anything unnecessary? Missing?
○ Did you learn something?
![Page 74: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/74.jpg)
Definition
“A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.”
-Tom Mitchell, 1997
![Page 75: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/75.jpg)
![Page 76: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/76.jpg)
![Page 77: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/77.jpg)
![Page 78: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/78.jpg)
Design Thinking and Lean, not either/or
![Page 79: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/79.jpg)
Diverge and converge
![Page 80: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/80.jpg)
The “spiral”
![Page 81: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/81.jpg)
Stanford d.school Design Thinking process
Learn
Build
Measure
Iterate
Lean
![Page 82: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/82.jpg)
![Page 83: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/83.jpg)
![Page 84: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/84.jpg)
Empathy Mapping (for the machine)
![Page 85: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/85.jpg)
Iterate
![Page 86: Product Management for AI/ML](https://reader031.vdocuments.net/reader031/viewer/2022021815/5a6503497f8b9abd5b8b488d/html5/thumbnails/86.jpg)
Perfect is the enemy of good… ...and not possible with AI