Upload File

deep learning frameworks - hpc.pnl.gov

11
Where we are and where we should be going DEEP LEARNING FRAMEWORKS JACK LEE | University of Toronto AMY WANG | Huawei Canada

Upload: others

Post on 30-Nov-2021

7 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

Where we are and where we should be going

DEEP LEARNING FRAMEWORKS

JACK LEE | University of TorontoAMY WANG | Huawei Canada

Page 2: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

BACKGROUNDArchitecture

Frontend API

Graph IR

Graph Executor

Kernel Library

INPUTS OUTPUTS

TensorFlow Frontend

Graph IR

Kernel Implementation

Page 3: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

MOTIVATIONSFrontend Interface

TensorFlow Frontend

Autograph

PyTorch Frontend

Page 4: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

MOTIVATIONSGraph Optimizations

PyTorch Frontend

Trace-based JIT

AST-based JITTensorFlow Graph IR

XLA Lower Level opsAutomatic differentiation every iteration.

Page 5: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

MOTIVATIONSKernel Specialization

XLA Lower Level ops

Benchmarks

Deep Fusion Tiling

Graph Lowering

Frontend API

INPUTS OUTPUTS

Graph IR

Compiler IR

COMPILED NETWORK

Page 6: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

MOTIVATIONSKernel Specialization

NNVM API

TVM API

OUTPUTS

Compiler IR

Generated GPU Code

INPUTS OUTPUTS

NNVM Graph IR

TVM Halide IR

COMPILED KERNELS

CUSTOM RUNTIME

Page 7: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

STATE OF THE ART SUMMARY

TENSORFLOW TENSORFLOW XLA PYTORCH PYTORCH - GLOW NNVM + TVM

Staged Frontend

✘ ✘

Native Frontend

✘ ✘ ✘

GraphOptimization

✘

Kernel Specialization

✘ ✘

Runtime Specialization

✘ ✘ ✘ ✘ ✘

ExecutionLevel

C++ C++ Python Machine Code C++

Page 8: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

THE DVM FRONTENDDeep Learning Compilation Framework

TENSORFLOW PYTORCH NATIVE SYNTAX

IR Transformation IR Transformation Parser (Clang/Python AST)

IR Builder

Page 9: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

THE DVM MIDENDDeep Learning Compilation Framework

SSA-based IRLow level opsControl Flow

Data Flow

Automatic Differentiation

Graph Optimizations

Profile Guided Optimizations

Compiler Optimizations

Page 10: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

THE DVM BACKENDSDeep Learning Compilation Framework

Default Runtime Codegen

Compatible Compiler

Specialized Runtime Source

Code

Handwritten Kernel Source

Code

Compiled Network

Runtime + Kernel Codegen

ClusteredSpecialized

Runtime Source Code

FusedSpecialized

Kernel Source Code

Compatible Compiler

Compiled Network

Page 11: DEEP LEARNING FRAMEWORKS - hpc.pnl.gov

Q&A

Default Runtime Codegen

Compatible Compiler

Specialized Runtime Source Code

Handwritten Kernel Source Code

Compiled Network

Runtime + Kernel Codegen

ClusteredSpecialized Runtime

Source Code

FusedSpecialized Kernel

Source Code

Compatible Compiler

Compiled Network

TENSORFLOW PYTORCH NATIVE SYNTAX

IR Transformation IR Transformation Parser (Clang/Python AST)

IR Builder

SSA-based IRLow level opsControl Flow

Data Flow

Automatic Differentiation

Graph Optimizations

Profile Guided Optimizations

Compiler Optimizations


DeepLM: Large-Scale Nonlinear Least Squares on Deep ......DeepLM: Large-scaleNonlinear Least Squares on Deep Learning Frameworks using Stochastic Domain Decomposition Jingwei Huang

Effectively Scaling Deep Learning Frameworks · Effectively Scaling Deep Learning Frameworks (To 40 GPUs and Beyond) ... you can often start with fairly small datasets that can easily

Designing Deep Learning · 2017-01-13 · Frameworks for Deepening Understanding and Building Capacity. WOW! Webinar resources ... Deep Learning. Task or Product “Your learning

Building Effective Network Security Frameworks using Deep

Usability Study of Distributed Deep Learning Frameworks ... · With the popularity of deep learning, the increasing complexity of deep learning models, and the availability of very

An Empirical Study towards Characterizing Deep Learning … · 2019-09-17 · An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks

Intel Powered Deep Learning Frameworks

Deep Learning for Computer Vision: Software Frameworks (UPC 2016)

Regional and site geological frameworks – proposed Deep

Deep Learning at Scale on NVIDIA V100 Accelerators · Deep Learning at Scale on NVIDIA V100 Accelerators HPC and AI Innovation Lab Rengan Xu, Frank Han, ... •Deep Learning Frameworks

Deep Learning Frameworks Deep Learning Framewor… · Deep Learning Frameworks Fabio A. González Machine Learning 2019-1. ... Optimized libraries ★OpenCL: Similar to CUDA but multiplatform

Comparison of deep learning frameworks from a viewpoint of double backpropagation

Deep Learning frameworks for wave propagation-based damage ... · Deep Learning frameworks for wave propagation-based damage detection in 1D-waveguides Mahindra Rautela1*, S ... which

Deep Support for SOA in EA Frameworks & Meta Models

NVIDIA Containers For Deep Learning Frameworks · 2020-01-27 · Docker Containers NVIDIA Containers For Deep Learning Frameworks DU-08518-001_v001 | 2 that are not affected by the

Deep learning frameworks v0.40

An Empirical Study towards Characterizing Deep Learning ... · An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms

The Frontier of Define-by-Run Deep Learning Frameworks · The Frontier of Define-by-Run Deep Learning Frameworks GTC 2019 @ San Jose. Mar. 20, 2019 Seiya Tokui, Preferred Networks,

Which Is Deeper - Comparison Of Deep Learning Frameworks On Spark

NVIDIA Containers and Deep Learning Frameworks · NVIDIA Containers and Deep Learning Frameworks DU-08518-001_v001 | ii TABLE OF CONTENTS Chapter 1. Docker Containers

Legislative and Administrative Frameworks for Deep Sea Minerals

BENCHMARKING DEEP LEARNING FRAMEWORKS FOR THE … · 2016. 6. 7. · BENCHMARKING DEEP LEARNING FRAMEWORKS FOR THE CLASSIFICATION OF VERY HIGH RESOLUTION SATELLITE MULTISPECTRAL DATA

QualityDeepSense: Quality-Aware Deep Learning Framework ... · The existing mobile deep learning frameworks usually treat every sensor input equally over time, lacking ... proposed

Differences of Deep Learning Frameworks

Deep Learning Frameworks - GitHub Pages...Deep Learning Frameworks COSC 7336: Advanced Natural Language Processing Fall 2017 Today’s lecture Deep learning software overview TensorFlow

Deep Learning Tools & Frameworks - cpsschool.eu · • Pros: general-purpose deep learning framework, flexible interface, good-looking computational graph visualizations, and Google’ssignificant

TASO: Optimizing Deep Learning Computation with Automatic ...odedp/taso-sosp19.pdf · Existing deep neural network (DNN) frameworks optimize ... Optimizing Deep Learning Computation

A Deep Dive Into Testing Vapor - Broken Hands · 2017-09-02 · DEEP DIVE INTO TESTING VAPOR THE PROBLEM WITH MOCKING FRAMEWORKS Reliance on frameworks May not be testing what you

Artificial Intelligence (A.I.), Machine and Deep Learning ...biconsulting.hu/letoltes/...balint_deep_learning_past_present_future.pdf · (Near) future… High level frameworks. Standardized

Deep Learning Frameworks slides

Effectively Scaling Deep Learning Frameworkson-demand.gputechconf.com/gtc/2017/presentation/s7543... · 2017-05-16 · Effectively Scaling Deep Learning Frameworks (To 40 GPUs and

Leveraging deep learning hyperparameter tuning frameworks · Leveraging deep learning hyperparameter tuning frameworks Derek Jensen, Donald Lucas, Cli˙ord Anderson-Bergman, Sonia

Deep Learning Frameworks with Spark and GPUson-demand.gputechconf.com/gtc/2017/presentation/s7737-subbu-rama... · DL4J can import neural net models from most major frameworks via

Wafer-Scale Deep Learning · Deep Learning: The Most Important Computational ... • Neural network models expressed in common ML frameworks • Cerebras interface to framework extracts

Deep500 – A Deep Learning Meta-Framework and HPC Benchmarksimipro.github.io/papers/Deep500 Simon Huber Bachelor.pdf · There are currently many Deep Learning frameworks. Most of