building serverless machine learning models in the cloud [pydata dc]

clda.co/pydata-‐dc10/08/2016

Building Serverless Machine Learning models

in the Cloud

http://clda.co/pydata-dc

http://pydata.org/dc2016/
































































































































About Me

@alex_casalboni

clda.co/pydata-‐dc

Bachelor in Computer Science

Master in Sound & Music Engineering

6+ years of Web Development

3+ years of Python


https://cloudacademy.com





























































Have you ever met a Data Scien?st?

Bob

Bob is very smart and curious

Bob loves Python & (Big)Data

Bob builds great ML models

Bob likes user-‐centered ML































































Bob can build a model in ~30 LOC!

clda.co/ML-codeBob






























































http://clda.co/ML-code


But…































































Phase 1: Analysis & Modeling

+

+

Bob

Data

Time































































Phase 2: Visualiza?on & Prototyping

+

+

Bob

Data

Time

ML Model

Data VisualisaKon

Prototype

+

+































































Phase 3: Disillusion

+

+

Bob

Data

Time

ML Model

Data VisualisaKon

Prototype

+

+

ProducKon Code































































Phase 4: Prod-‐ready Integra?on

+

+

Bob

Data

Time

ML Model

Data VisualisaKon

Prototype

+

+

WebDev

DevOps

A lot of Time

+

+































































The lack of ownership

!=

Data ScienKst DevOps

Mathema?cal modeling Sta?s?cal analysis

Data mining

(Cloud) Opera?ons System administra?on So\ware best prac?ces































































Why is deploying ML models a challenge?

Bob’s prototypes are’t producKon-‐ready

Bob needs ElasKcity & High Availability

Bob will deploy hundreds of models

Bobs wants plenty of features

Bob































































Machine Learning as a Service (MLaaS)

AmazonMachine Learning

AzureMachine Learning

GooglePredic?on API

IMBWatson Analy?cs

BigML






























































http://cloudacademy.com/blog/aws-machine-learning/









































http://cloudacademy.com/blog/azure-machine-learning/






































http://cloudacademy.com/blog/google-prediction-api/

http://cloudacademy.com/blog/bigml-machine-learning/
























































































































































https://www.ibm.com/analytics/watson-analytics/

















































































































































































































































Deployment Strategies

1. Web-‐app controller

how will Bob update his model(s)?

same website auth layer?

shared uWSGI processes?

It’s disarmingly simple, but…

Bob
































































2. Fleet of servers

same problems as before

Bob has many more machines to maintain

sKll no elasKcity (over-‐provisioning)

Bigger capacity and no code changes, but…

Load Balancing

Bob Bob2 Bob3
































































<Bob>

3. Auto Scaling

sKll shared resources? containers?

even bigger lack of ownership

what about caching, versioning and auth?

Bob achieved elasKcity and availability , but…

Load Balancing + Auto Scaling
































































4. Serverless Compu?ng

+Amazon API Gateway

AWS Lambda

Happy Bob

Managed infrastructure + BYOC

Transparent elasKcity and PAYG

Many useful features for free

Bob doesn’t need (to be) a DevOps anymore!































































There is always a server somewhere, you just don't have to worry about it :)































































Benefits of Serverless Machine Learning

Versioning, staging & caching

1 model = 1 microservice

IntuiKve RESTful interface

High Availability (no downKme)

Very li_le operaKonal effort

Transparent elasKcity (PAYG)

Failure isolaKon Offline training phase

ProducKon-‐ready prototypes A/B tesKng through composiKon































































A simple example: Serverless Sen?ment Analysis

clda.co/ML-‐Lambda






























































https://d2ya98r07n6vbt.cloudfront.net/sentiment.html

http://clda.co/ML-Lambda


A real-‐world example

Serverless ML @ Cloud Academy

MulK-‐model architecture

RESTful interface for each ML model

S3 + RDS for storage

Periodic training (offline)































































Limita?on of Serverless ML

AWS Lambda

No real-‐Kme models (only pseudo real-‐Kme)

Deployment package management: size limit and OS libraries

Not suitable for model training yet (5 min max execuKon Kme)

Cold start Kme is long and hard to avoid

Unit/integraKon tests help, but not enough































































The Serverless Ecosystem

AWS Lambda

Microsoh Azure FuncKons

Google Cloud FuncKons

IBM Bluemix OpenWhisk

hook.io

Cloud Placorms Func?on as a Service

iron.io

Auth0 (SSO)

Misc

Firebase (DB)






























































https://aws.amazon.com/lambda/

https://azure.microsoft.com/en-us/services/functions/

https://cloud.google.com/functions/

http://www.ibm.com/cloud-computing/bluemix/openwhisk/

http://hook.io/

https://www.iron.io/

https://auth0.com/

https://firebase.google.com/

Thank you!

jobs.cloudacademy.comP.S. we are hiring!
































































































































