Upload File

deluceva: delta-based neural network inference …...deluceva: delta-based neural network inference...

  • Home
  • Documents
  • Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information
22
Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics Jingjing Wang and Magdalena Balazinska University of Washington 1

Upload: others

Post on 22-May-2020

5 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Deluceva:Delta-BasedNeuralNetworkInferenceforFastVideoAnalytics

Jingjing WangandMagdalenaBalazinskaUniversity of Washington

1

Page 2: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Deluceva:Delta-BasedNeuralNetworkInferenceforFastVideoAnalytics

• Largevolumeofimages/videoswithvaluableinformation• Alargesetofneuralnetworkmodelsforimages• Objectclassification,detection,…

• Nextstep:videoanalytics• Largervolume• Efficiencycritical,liveoutput

2

Page 3: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

VideoAnalyticsUsingNeuralNetworks

3

… …

DeepNeuralNetworkRichsetofestablishedimagemodels

Page 4: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

KeyObservation:TemporalRedundancy

4

- =

Page 5: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

ProcessDeltasInsteadofFullFrames

5

… …

State

Query

“IncrementalQueryEvaluation”

Page 6: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

ProcessDeltasInsteadofFullFrames

6

DataStream

State

Query

“IncrementalQueryEvaluation”

…newboatdiscoveredwithboundingboxB

…

…pixel[x,y,z]has

changedby(0,0,1)…

Page 7: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Delta-BasedInferenceforVideos

• Problem:• Input: a videostream,areferencemodel• Output: similartothereferencemodel’sresult

• Approach:• Acceleratemodelinferencebyperforminglesscomputation

7

Page 8: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Delta-BasedInferenceforVideos:Overview

• Modifyneuralnetworktotakedeltasasinputs• Decidewhichdeltasaresignificantenoughtoprocess• Generate a network of mixed-type (dense or delta-based) operators

8

Page 9: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Delta-BasedInferenceforVideos:Overview

• Modifyneuralnetworktotakedeltasasinputs• Decidewhichdeltasaresignificantenoughtoprocess• Generate a network of mixed-type (dense or delta-based) operators

9

Page 10: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Frame2– Frame1

Frame1

Neural Network with SparseDeltas

10

=

+

Model(original)

Frame2

Model(delta)

Model(delta)

Page 11: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

ExampleNeuralNetwork Model

11

Model(original)

OpOp ……

• Exampleoperators:convolution,maxpooling,ReLU,…

Page 12: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

ExampleNeuralNetwork Model

12

Model(delta)

OpOp ……

• Modifyoperatorstooperateondeltas

DeltaOperatorUnit

Page 13: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

FilterOperator

DeltaOperatorUnit

• Sparseoperator:takessparsedeltas&outputsdelta• Saves#ofFLOPsbyprocessingdeltascalarsonly

• Filter:sendonlysignificantdeltastotheoperator• Buildshistogram,keepssmalldeltas&outputslargedeltas

13

Indices,Values

abs

Accum.deltas

Kernel

・ +

SparseHow much to filterforthecurrentframe?

Page 14: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Delta-BasedInferenceforVideos

• Modifyneuralnetworktotakedeltasasinputs• Decidewhichdeltasaresignificantenoughtoprocess• Generate a network of mixed-type (dense or delta-based) operators

14

Page 15: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

DeltaImpactsOutputQuality

15

Conv …

Conv …

Frame1 Frame2

Frame2– Frame1

Groundtruths:

1s

5s

90%

20%

Page 16: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

DynamicFilteringPercentage

16

• Filteringpercentage• Higherisfasterbutrisky• Lowerissafer butslower

• Targetfilteringpercentage:largestpercentagethatgeneratesgoodresult• Appliestoalloperators• Two approaches: PI controller / Machine learning

Safer,slower Risky,faster

abs(delta)

90%

Page 17: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Delta-BasedInferenceforVideos

• Modifyneuralnetworktotakedeltasasinputs• Decidewhichdeltasaresignificantenoughtoprocess• Generate a network of mixed-type (dense or delta-based) operators

17

Page 18: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

MixedNetwork

18

OpOp ……

OpOp ……

Filteringpercentageis:low(e.g.0)high(e.g.99)

medium(e.g.50)

OpOp ……

Page 19: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

DeltaOpUnit

MixedNetwork• Logical plan: a DAG of operators• Physical plan: choose between delta opunit/ originaldenseimplementation• Profileeachoperatorwithdifferentfilteringpercentages• Pickthefastervariant

19

Op2(sparse)Filter

Op2(dense)Filteringpercentage=99

Op1… …Op3

Page 20: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Evaluation:Setup

20

• Threeobjectdetectionmodels

• Six10-minutevideosfromthreeYouTubelivestreams– Takenatdifferenttimes(e.g.day/night)for eachstream– Typicalobjects:people,cars,buses,boats,…– One frame per second

• TensorFlow,oneCPUthread, AmazonEC2r3.2xlarge

Model Abbrv. #ofFLOPs

SSD-VGG16 ssd-vgg 123B

FRCNN-RESNET101 frcnn-res 550B

FRCNN-INCEPTION-RESNET-V2 frcnn-incep 1395B

Time

3s

16s

41s

Page 21: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Evaluation:End-to-EndComparison

21

• HighestruntimesavingsbyPIcontroller– Whenerrorlessthanathreshold

Model / Datasetssd-vgg

je jn kd kn vd vnfrcnn-res

je jn kd kn vd vnfrcnn-incep

je jn kd kn vd vn

0

10

20

30

40

50

60

Avg.

Pct

. of S

avin

g

videos

Page 22: Deluceva: Delta-Based Neural Network Inference …...Deluceva: Delta-Based Neural Network Inference for Fast Video Analytics •Large volume of images/videos with valuable information

Deluceva:Conclusion

• Observerichtemporalredundancyinvideos• Acceleratemodelinferencebyprocessingsignificantdeltasonly

• ModifyNNmodelstoconsistofsparse&denseops• Adjustthefilteringgranularityadaptively• Generateanetwork of mixed-type operatorsbasedoncostmodels

• Improveruntimeupto67%with lowerror• Appliestoconvolutionalneuralnetworkmodels

• Ongoing:GPUimplementation,comparetootherwork

22


Complex Inference in Neural Circuits with Probabilistic ...papers.nips.cc/paper/4555-complex-inference-in... · Complex Inference in Neural Circuits with Probabilistic Population

Dynamic Neural Relational Inference...Dynamic Neural Relational Inference Colin Graber Alexander Schwing University of Illinois at Urbana-Champaign {cgraber2, aschwing}@illinois.edu

Pruning convolutional neural networks for resource efficient inference

Property Inference Attacks on Fully Connected Neural ...seclab.illinois.edu/wp-content/uploads/2018/08/ganju2018property.pdf · neural networks, property inference, permutation equivalence

DELPHI: A Cryptographic Inference Service for Neural Networks · 2019-10-15 · DELPHI: A Cryptographic Inference Service for Neural Networks Pratyush Mishra Ryan Lehmkuhl Akshayaram

VINE: A VARIATIONAL INFERENCE-BASED BAYESIAN NEURAL ... · ComputingBased Realization of Spiking Neural Networks” also known as “VINE: A Variational Inference- Based-Bayesian

RESEARCH ARTICLE The Causal Inference of Cortical Neural ...hjjens/PLOS_ONE_Music.pdf · RESEARCH ARTICLE The Causal Inference of Cortical Neural Networks during Music Improvisations

1 Characterizing the Deep Neural Networks Inference ... · 1 Characterizing the Deep Neural Networks Inference Performance of Mobile Applications Samuel S. Ogden, Tian Guo Abstract—Today’s

Factorised Neural Relational Inference for Multi

Learning Structured Inference Neural Networks with Label ...mori/research/papers/hu-cvpr16.pdf · Learning Structured Inference Neural Networks with Label Relations Hexiang Hu1,2,

Improving Inference for Neural Image Compression · 2020. 9. 26. · Improving Inference for Neural Image Compression Yibo Yang, Robert Bamler, Stephan Mandt Department of Computer

Inference Attacks Against Graph Neural Networks

Inference in Probabilistic Graphical Models by …Inference in Probabilistic Graphical Models by Graph Neural Networks 2. Related Work Several researchers have used neural networks

Scalable inference of overlapping communities. Neural Information Processing Systems, 2012

Neural Network and Adaptive Neuro- Fuzzy Inference System ...cdn.intechopen.com/pdfs/36647.pdf · Neural Network and Adaptive Neuro- Fuzzy Inference System Applied ... and adaptive

Accelerated Physical Emulation of Bayesian Inference in ... · neural networks, neural sampling, probabilistic inference 1. INTRODUCTION The aggressive pursuit of Moore’s law in

Adaptive Neural Fuzzy Inference Systems (ANFIS): Analysis and Applications

Band-limited Training and Inference for Convolutional Neural …proceedings.mlr.press/v97/dziedzic19a/dziedzic19a.pdf · Band-limited Training and Inference for Convolutional Neural

3 Adaptation of Fuzzy Inference System Using Neural … · several works related to the integration of neural networks and fuzzy inference systems [1, 2, 3, 15, 17, 23 ... Neural

Graph Neural Network Inference on FPGA SEPTEMBER 2019

Deep Convolutional Neural Network Inference with Floating

Fast Inference for Neural Vocoders - GPU Technology …on-demand.gputechconf.com/gtc/2017/presentation/s7544...Fast Inference for Neural Vocoders Welcome, everyone! I’m excited to

EIE: Efficient Inference Engine on Compressed Deep Neural ...smk9u/CS6501F16/p243-han.pdf · EIE: Efficient Inference Engine on Compressed Deep Neural Network Song Han∗ Xingyu

Chlorophyll a spatial inference using artificial neural - SciELO

Deep Convolutional Neural Network Inference with Floating ...Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activations Figure 1. Examples

Variational Inference for Bayesian Neural NetworksVariational Inference for Bayesian Neural Networks Jesse Bettencourt, Harris Chan, Ricky Chen, Elliot Creager, Wei Cui, Mo-hammad

GINA: Neural Relational Inference From Independent Snapshots

Inference of Gas-liquid Flowrate using Neural Networks

ncnn -universal neural network inference with vulkan

Artificial Neural Network Fuzzy Inference System(1)

Natural language grammatical inference with recurrent neural … · 2007-03-04 · Natural Language Grammatical Inference with Recurrent Neural Networks Steve Lawrence, Member, IEEE,C.LeeGiles,Fellow,

Neural Relational Inference with Efficient Message Passing

Complex Inference in Neural Circuits with Probabilistic Population

Neural Multisensory Scene Inference - arXiv · 2019. 11. 11. · Neural Multisensory Scene Inference Jae Hyun Limyx123, Pedro O. Pinheiro 1, Negar Rostamzadeh , Christopher Pal1234,

Neural Networks Cortical circuits for perceptual inference