big data & artificial intelligence
TRANSCRIPT
Big Data & Artificial Intelligence2014 Technology Review and Primer
Zavain Dar
High Level
Data —> Infrastructure —> Enables more Data —> Analytics, Applications, & Artificial Intelligence
If we buy the above, we see ‘AI’, ‘Big Data’, ‘Deep Learning’, etc… not as buzz words, but as a logical next step of technological
progress from the past 20 years
2
Outline
• Historical Context: The Web, Big Data, & Distributed Computing
• Modern Infrastructure
• Artificial Intelligence
• Learnings & Thesis Directions
3
Computing Infrastructure pre Web
• Storage Paradigm: Relational Databases (Oracle, MySQL, etc…)
• Access Paradigm: Relation Algebra (SQL)
• Each computer owned its data, computation was generally done on a single computer
CC
C
DD
D
1984: 100 Nodes convert to TCP/IP
• Until 1984, there was no unified ‘internet’, rather a collection of fragmented networks using one-off protocols
• In 1984, the most connected 100 nodes switched to TCP/IP. Modern Internet was born
The Web as a ‘Big Data’-base
• We can view the Web itself as the first big database
• Storage Paradigm: HTML, DOM, Relational Databases (Oracle, MySQL)
• Access Paradigm: HTTP
C
C
C
C
D
D
D
D
The Web emerged as the first ‘Big Data’-set
• Other than HTTP requests, which were slow and clunky - we had no way to index, and parse web content
• A handful of search engines came and went, but all struggled to effectively deploy algorithms atop this massive distributed data set
7
Google in 1998• Data uniformly distributed across
computers
• Storage Paradigm: GFS (Google Filing System)
• Access Paradigm: ???
• Google kept Access Paradigm proprietary for years
CCC C
D
2004: Big Data leaves Google’s confines
• Jeff Dean and Sanjay Ghemawat publish seminal paper outlining MapReduce, a distributed data access paradigm
• Storage Paradigm: GFS
• Access Paradigm: MapReduce
Modern Big Data• Apache Hadoop was born as an open source project form Yahoo in 2005.
Followed Google’s GFS and Google MapReduce implementations
• Hadoop consisted of HFS (Hadoop Filing System) and Hadoop Map Reduce
• It took years for the open source framework to become enterprise ready. In the interim, Cloudera and HortonWorks began offering enterprise solutions based around Hadoop
• Others wrote completely black box, proprietary versions based on GFS and Map Reduce. Examples: Palantir and Discovery Engine
• Palantir only recently switching over to Hadoop based code. 10
Emergent Themes• Commoditization of Infrastructure
• Early infrastructure providers have plateaued in value; Hortonworks a recent example with a down round IPO
• DevOps
• As computing models changed from local and heterogeneous-hardware based, new solutions emerge to help pace innovation
• ‘Appification' and Analytics atop Hadoop
11
DevOps: Docker
• Programming on and testing on a laptop different than running on Dell x86 clusters or mobile+HP server.
• Docker creates a portable container (eg docker) around an application, making it easy to port to heterogenous environments
laptop x86 x86
x86 x86
HP iOS
Application
Application
Application
DevOps: Mesosphere
• The old world had Virtual Machines which sliced single computers into numerous ‘virtual instances’ for security, debugging, etc…
• Now we need the opposite, to view entire clusters as a singe computer with shared and (hence) optimized storage, network, and compute C
CC
C’
Artificial Intelligence
Traditional AI broken into 2 categories
1. Computational Logic (this guy!) & Search+Planning
2. Machine Learning
14
Computational Logic + Planning• Based on implementing static rules for a computer to follow. The
end algorithm and rules are independent of the data
• Old school (Chomskyan) NLP and chess playing followed this approach
• Planning based on route optimization and ‘graph search’
• Eg how do you efficiently plan a UPS route, or guide a robotic arm around obstacles of a pre known course
15
Computational Logic + Planning• From 1940s through the early 1990s this was the preferred methodology for AI
• Key assumption: The world is guided by rules, and it’s just going to be a while before we can encode the minimal viable set before computers can deduce future outcomes and propositions
• AI slowed in results, and hence funding from the 70s through the 80s.This was known as the AI Winter. Largely due to heavy academic emphasis on these methods
• The early 90s showed focus on statistical methods - commonly dubbed the Bayesian Revolution
• This lead to the proliferation and growth of machine learning
16
Machine Learning• Premise for machine learning:
• Have a dataset
• Have an algorithm f(D)
• f(D) applied to a dataset gives a new function (model) m(i)
• m(i) applied to any input i predicts an output o
17
D
f
Machine Learning (Pictorially)
18
D f m(i) o
1. The machine learning algorithm f is applied to the dataset D, giving the model m
2. For any input i, the model m predicts an output o
1) Supervised Learning
• D consists of pairs of input, output types: <i, o>
• The larger D the more generalized and accurate the end model m is
• Learn by example
3 Types of Machine Learning
19
D f m(i) o
3 Types of Machine Learning2) Unsupervised (Topological) Learning
• D consists of just inputs: <i>
• Generally end up with a partitioning of D
• Good at finding patterns20
D f m(i) o
3 Types of Machine Learning3) Reinforcement Learning
• You add some derivative of the output back to the initial dataset, and reoptimize your model
• Eg Learning to play chess by playing over and over again. Ideally the more you play the less you lose
21
D f m(i) o
?
Deep Learning• Deep Learning and Neural Nets are synonymous
• Deep Learning is a subset of machine learning, it is a class of functions f from the previous slides
• Deep learning algorithms take in a data set and spit out another function, or model, m
• Can be deployed in structured, unstructured, and reinforced contexts
22
Deep Learning
• First theorized and worked on in the 80s
• However, lacked the infrastructure and data to meaningfully deploy
• Has seen a massive resurgence 2009 onwards
• Loosely inspired by (vague) knowledge of brain - layers of abstraction23
Deep Learning• Useful for noisy, large, human generated data
• That is data for which, even the correct form of model input i can be tricky to characterize
• When I see a picture of a human face, I immediately recognize eyes, a nose and ears … hence a face
• When a computer receives the same image, it’s a rectangular grid of RGB values. How do we map the computer’s input space to our semantic space?
• Types of data that this makes sense for: Text, Visual (images & video), Audio, User behavior (my patterns on Twitter or Facebook), Basketball (player millisecond movement), etc…
24
Deep Learning
25
Image Models
Functions Artificial Neural Nets Can Learn
Audio: “sh ang hai res taur aun ts”
Good Fine-grained Classification
“hibiscus” “dahlia”
Sensible Errors
“snake” “dog”
Embeddings are Powerful
fallen
draw
fell
drawn
drew taketaken
took
givegiven
gave
fall
sentence rep
PCA
LSTM for End to End Translation
linearly separable!wrt subject vs object
Generating Image Captions from Pixels
Human: A young girl asleep on the sofa cuddling a stuffed bear.!
Model sample 1: A close up of a child holding a stuffed animal.!
Model sample 2: A baby is asleep next to a teddy bear.
Work in progress by Oriol Vinyals et al.
Generating Image Captions from Pixels
Human: A young girl asleep on the sofa cuddling a stuffed bear.!
Model sample 1: A close up of a child holding a stuffed animal.!
Model sample 2: A baby is asleep next to a teddy bear.
Work in progress by Oriol Vinyals et al.
Generating Image Captions from Pixels
Human: A young girl asleep on the sofa cuddling a stuffed bear.!
Model sample 1: A close up of a child holding a stuffed animal.!
Model sample 2: A baby is asleep next to a teddy bear.
Work in progress by Oriol Vinyals et al.
Current LandscapeGPUs, FPGAs, ASICs (User wants specialized deployments either for the learning function f or the end model m):
Select examples: Nervana Systems, TerraDeep, Qualcomm Neuromorphic Group
APIs, SDKs (USer wants to use prewritten algos on their datasets):
Select examples: Metamind, Skymind.io, Vicarious, Deep Mind
Vertical (Technology is black-boxed from user):
Select examples: Clarifai, Butterfly Networks, Binatix, etc…
26
Artificial Intelligence
27
Computational Logic & Planning
Machine Learning
Statistical Regressions
Deep Learning
etc…
Applications
• NLP • Computer Vision • Robotics • Audio • Sports • Genetics • Finance • Anomaly Detection
LearningsStatic software commoditizes
• Early big data infrastructure providers stagnating
• Google’s algorithms are essentially public (PageRank etc..)
• Deep Learning algos are an arms race & race to bottom
Defensibility and ability to grow into large 100M+ company is in owning proprietary data from which you can train better models and/or have network or scale effect
Why is now special? We’re sitting at the intersection of:
1. a matured big data infrastructure driven by well understood distributed storage and data access paradigms
2. data continues to explode. Not only though web, but also via noisy sensor and human generated data
3. have AI tools necessary to make sense of unstructured and noisy datasets whose features don’t map well to our a priori intuition
28
‘Virtuous’ Feedback LoopsGoing back to Google:
29
CCC C
D
f m(i) o
D’
‘Virtuous’ Feedback LoopsGoing back to Google:
30
CCC C
D
f m(i) o
D’
CommoditizedCommoditized
Feedback Loops• Google collects click-data with each user - this enables better search
for next user: n+1th user has a better experience than nth user • Google increases margin from competition the more we use it • Leads to a run-away effect • Can explain Google’s monopoly in search • Same analogy with Facebook/Twitter-adds and other large tech co’s • Prediction: Early movers who can bootstrap initial feedback loop will
be big, potentially winner-take-all, winners
31
Data —> Infrastructure —> Enables more Data —> Analytics, Applications, & Artificial Intelligence
32
Empirical Timeline
MapReduce
Empirical Timeline
Hadoop
Empirical Timeline
Big Data
Empirical Timeline
Deep Learning
