Introducing KryloveBay AI Platform - Machine Learning Made Easy

GPU Technology Conference, 2018

Henry SaputraTechnical Lead for Krylov - eBay Unified AI Platform

1. Data Science and Machine Learning at eBay2. Introducing Krylov3. Compute Cluster and Accelerator Support with Nvidia GPU4. Quickstart Example5. Future Roadmap6. Q & A

Agenda

Data Science and Machine Learning at eBay

eBay Patterns - Tools and FrameworksTools

• Languages: R, Python, Scala, C++

• IDE-like: RStudio, Notebooks (Juptyer), Python IDE

• Frameworks: NumPy, SciPy, matplotlib, Scikit-learn, Spark MLLib, H2O

Weka, XGBoost, Moses

• Pipelines: Cron, Luigi, Apache Airflow, Apache Oozie

Patterns for ML Training

• Single node

• Distributed training

• Deep learning (GPUs)

Deep LearningDistributed Training Key takeaway = CHOICE

1. Flexibility of software 2. Flexibility of hardware

configuration

1. 50%-70% is plumbing worka. Accessing and moving secured datab. Environment and tools setupc. Sub-optimal compute instances - NVIDIA GPUs and High memory/ CPUs instancesd. Long wait time from platform and infrastructure

2. Lost of productivity and opportunitiesa. ML lifecycle management of models and featuresb. Building robust training model pipelines: prepare data, algorithm, hyperparameters tuning, cross

validation

3. Collaborations almost impossible4. Research vs Applied ML

Problems and Challenges

Introducing Krylov: Unified eBay AI Platform

● Krylov is the core project of the eBay unified AI Platform initiative to enable easy to use and powerful cloud-based data science and machine learning platform.

● The objective of the project is to enable machine learning jobs with easy access to secured-data and eBay cloud computing resources.

● The main goals for the Krylov initiative are:○ Easy and secure access to training datasets○ Access to compute in high performance machines, such as GPUs, or cluster of

machines.○ Familiar tools and flexible software to run machine learning model training jobs○ Interactive data analysis and visualization, with multi-tenancy support to allow quick

prototyping of algorithms and data access○ Sharing and collaboration of ML work between teams in eBay

Overview

ML Lifecycle Management

Lifecycle

MODEL INFERENCINGDeployable, Scalable

MODEL BUILDING

Interactive, iterative

MODEL RE-FITTINGInteractive, iterative

MODEL RE-TRAININGInteractive, iterative

Data + Lifecycle Management

MODEL TRAININGAutomatable, repeatable, scalable

Krylov Staircase Design for AI Platform

eBay AI Platform Components

Infrastructure - Krylov

AI Engine - Krylov

LearningPipelines

Model Experimentation

Data ScientistWorkspaces

Model LifecycleManagement

GPU Tall instances

Fast Storage

Data

Preparation

Movement

Discovery

AccessAI Hub(Shared

Repository)

AI Modules

Speech Recognition Machine TranslationComputer Vision Information RetrievalNatural Language Understanding …

Inferencing

Krylov High Level Architecture

1. Client Command Line Interface (CLI) via krylovctl program

2. ML Application and Run Specification

3. ML Pipelines: Workflow and Workspace

4. Namespaces - For quota and data isolation

5. Jobs and Runs - Managed by Krylov Tools and Minions

6. Secure Data Access - HDFS, NFS, OpenStack Swift, Custom

Krylov Main Features and Concepts

Krylov CLI - krylovctl

● Krylov ML Application is a versioned unit of deployment that contains declaration of the

developers’ programs

● Implemented as client project used as source to build deployment artifact

● Three main parts:

○ mlapplication.json and artifact.sjon configuration files

○ Source code of the programs

○ Dependencies management via Dockerfile

● Supported types of programs: JVM languages (Java, Scala), Python, Shell script

● Using the ML Application as source, developers can build deployment artifact that can be

used by the Run Specification file to deploy it into one of the nodes in the cluster

Krylov ML Application

{

"tasks": {

"prepare_data": {

"program": "com.ebay.oss.krylov.workflow.JvmMainProgram",

"parameters": {

"className": "com.ebay.krylov.helloai.HelloWorld"

}

},

"train_model": {

"program": "com.ebay.oss.krylov.workflow.PythonProgram",

"parameters": {

"file": "helloai-python/helloai/helloworld.py",

"args": []

}

},

...

Krylov ML Application Example

● The Krylov Run Specification is a runtime configuration to add override configuration and

parameter passing for each Task in the ML Application job submissions

● It tells Krylov master API server of which the artifact created by ML Application will be used in

the compute cluster

● Defined as runspec.json file or can be passed as argument to krylovctl client program.

● The runspec.json file also has definition for the compute resources, such as which NVIDIA

GPUs to use, CPU, memory, and which Docker image for dependencies used in ML

Application programs

Krylov Run Specification

{

"jobName": "job-sample",

"artifact": "myartifact",

"artifactTag": "latest",

"mlApplication": "com.ebay.oss.krylov.workflow.app.GenericMLApplication",

"applicationParameters": {

},

"tasks": {

"prepare_data": {

"taskParameters": {

"prepare_data_parameter_key": "prepare_data_parameter_value"

}

}

}

Krylov Run Specification Example

● Krylov ML batch lifecycle pipeline is defined as Krylov Workflow definition

○ Declarative

○ Default Generic Workflow

● Important concepts for Krylov Workflow:

○ Workflow - A single pipeline defined within Krylov and the unit of deployment for an ML Application

■ Each Workflow contains one or more Tasks

■ The Tasks are connected to each other as Directed Acyclic Graph (DAG) structure

○ Task - smallest unit of execution that run developers’ Program and executed in a single machine

○ Flows - Contains one or more key-value pairs of name and declaration of Tasks DAGs

○ Flow - The chosen key that will be run from possible selection in the Flows definition

Krylov ML Pipelines: Workflow

{

"tasks": {

...

},

"flows": {

"sample_flow": {

"prepare_data":

["train_model"],

"train_model":

["output"]

}

},

"flow": "sample_flow"

}

Workflow Example in mlapplication.json

Workflow Runs Flow

● A Workspace is an interactive web application to allow developers to use web

browser to do ML model prototyping, data preparation and exploration

● The Workspace is run as Jupyter Notebook servers and launched on high CPU/

memory or NVIDIA GPU instances

● Enhance the JupyterHub project to allow distributed launching of multi-tenants

Jupyter Notebook servers in Krylov compute cluster using Kubernetes

● Krylov Workspace uses configuration file on creation time to override and

customize default parameters

Krylov ML Pipelines: Workspace

Workspace Deployment Flow

Krylov Compute Cluster

Krylov Cluster Infrastructure

Krylov Compute Cluster Deployment

● Metrics - Grafana, InfluxDb, and Telegraf for GPU monitoring

Krylov Cluster Monitoring

Krylov Metrics Management Flow

Krylov Compute Resources Management

Quickstart Example

1. Download krylovctl program from Krylov release repository

2. Run `krylovctl project create` to create new project in the local machine

3. Update or add code to the Krylov project for the machine learning programs

4. Register them as Program within a Task in the mlapplication.json

5. Add new Flow for the defined Tasks to construct the Workflow as a Directed Acyclic Graph (DAG)

6. Run `krylovctl project build` to build the project.

7. Run `krylovctl artifact create` to copy the runnables of the program into an artifact file

8. Run `krylovctl artifact upload` to upload the artifact file for remote execution

9. Run `krylovctl job run` for local execution, or `krylovctl job submit` for running it in the computing

cluster

Steps to Submit Krylov Workflow Job with CLI

● Here we go ...

Demo Time

Future Roadmap

1. Inferencing Platform

2. Exploration and documentation of RESTful APIs for job management

3. Data Source and Dataset abstraction via Krylov SDKs

4. Managed ML Pipelines - Computer Vision, NLP, Machine Translation

5. Distributed Deep Learning

6. AutoML - Hyper Parameters Tuning

7. AI Hub to share ML Applications and Datasets

Future Roadmap

Question?