NeuralNetworks TheNewMooresLaw

ChrisRowen,PhD,FIEEECEOCognite Ventures

December2016

Outline MooresLawRevisited:EfficiencyDrivesProductivity EmbeddedNeuralNetworkProductSegments EfficiencyandProductivityinEmbeddedNeuralNetworks BreadthofNeuralNetworks:

VisionandthePixelExplosion Speech NaturalLanguage

Efficiency,ProductivityandNeuralNetworks TheEmbeddedSystemsInnovationSpace TheEvolutionofElectronicDesignandCognitiveComputing

MooresLawRevisited:EfficiencyDrivesProductivityMooresLaw:numberoftransistorsina[economicallyviable]denseintegratedcircuitdoublesapproximatelyeverytwoyears

DennardScalingImpact ofscaling Lby (

EmbeddedNeuralNetworkProductSegmentsAutonomousVehiclesandRobotics

MonitoringandSurveillance

Human-MachineInterface

PersonalDeviceEnhancement

Vision Multi-sensor:image,depth, speedEnvironmentalassessmentFull surroundviews

AttentionmonitoringCommand interfaceMulti-modeASR

SocialphotographyAugmentedReality

Audio Ultrasonic sensingAcoustic surveillanceHealthandperformancemonitoring

MoodanalysisCommand interface

ASRsocialmediaHands-freeUIAudiogeolocation

NaturalLanguage

AccesscontrolSentiment analysis

MoodanalysisCommand interface

Real-timetranslationLocalbotsEnhancedsearch

ThePixelExplosion Computingandcommunicationdrivenbynewdata

in/out CMOSsensorstriggerimagingexplosion 99%ofofcapturedrawdataispixels(dwarfssounds

andmotion)

1010 sensorsx108 pixels/sec=1018 rawpixels/sec

Rapidgrowthofvision-basedproductsandservices Starting2015:moreimagesensorsthanpeople NewAge:Makingsenseofpixelsrequires computer

cognition 0

2E+09

4E+09

6E+09

8E+09

1E+10

1.2E+10

1.4E+10

1.6E+10

1.8E+10

2E+10

1990 1995 2000 2005 2010 2015 2020

WorldPopulation

Three-yearsensorpopulation

Vision

0

10

20

30

40

50

2011 2012 2013 2014 2015 2016ImageN

etTop

-1Error

%

Year

RapidProgressonAccuracy

01020304050

0 5 10 15 20 25

ImageN

etTop

-1

Error%

GMACsperimage

BoundedComputeLoad

01020304050

0 50,000,000 100,000,000 150,000,000

ImageN

etTop

-1Error

%

ModelCoefficients

ModelsGettingMoreManageable

Computervisionisbig,obviousNNdomain Manyrelatedtasks:classification,localization,segmentation,objectrecognition,captioning

Hugecomputationinembeddedinference Visionisfundamentallyhard!! Example:ImageNetClassification:

1000categories 120speciesofdogs

23.5

24

24.5

25

25.5

0 2 4 6 8ImageN

etTop

-1Error

%

GMACs

OptimizationDoublesEfficiency

ResNet50,101Cadence

Tibetanmastiff Shih-Tzu Norwegianelkhound

Speech:Fromsoundstowords Automatedspeechrecognition(ASR)pipeline:

Waveformtospectralsamples Spectralsamplestophonemes

Shese

llsse

ashellsbytheseasho

re

Tuske etal,AcousticModelingwithDeepNeuralNetworksUsingRawTimeSignalforLVCSR

Phonemestowords

RNNorHiddenMarkovModel RNNorHiddenMarkovModel

Movingtounifiedend-to-endtrainedneuralnetwork

Year WordErrorRateonSwitchboard ASR1995 40%2001 20%2015 6.3%

NaturalLanguage Realtimenaturallanguageinterpretationcrucialforrich

human-machineinterfaces Buthowdoweautomaticallyfindwordmeanings?? Powerfulapproach: Automaticallylearnhigh-dimensionvectorembedding(N=300-600)fromthewordusageinlargedata-sets Wordswhosevectorsareclosehavestrongrelationship Differentdimensionsreflectdifferentkindsofwordrelationships:

good:bettergood:finegood:badgood:product

Coolapplication:vectorarithmeticrevealscomplexrelationships:

V(king) V(man)+V(woman) V(queen)

TranslationgoesNN:

Fullsentence-at-a-timetranslation

With[GoogleNeuralMachineTranslation],GoogleTranslateisimprovingmoreinasingleleapthanweveseeninthelasttenyearscombined- BarakTurovsky

Source:https://research.googleblog.com/2016/09/a-neural-network-for-machine.htmlSee:Mikolov etal,EfficientEstimationofWordRepresentationsinVectorSpace

EfficiencyandNeuralNetworks

Efficiency: Conventionalwisdom- deepneuralnetworksmuchlessefficientthanhand-tunedfeaturerecognitionmethods(butmoreeffective)

Convolutionalneuralnetworksallow Highparallelism Lowbitresolution Structured,specializedarchitectures Manageablememorybandwidth

~1000xenergyimprovementoverGPCPUmaycompensateforefficiencygap

10

100

1,000

10,000

10 100 1,000 10,000 100,000

GMAC

SPe

rWatt

GMACs

NeuralNetworkPlatforms

VisionDSPcore1 VisionDSPcore1cluster VisionDSPcore2VisionDSPcore2cluster EmbeddedGPUcorecluster DataCenterGPU1clusterEmbeddedGPUcluster FPGA1 FPGA2Convolutionalneuralnetwork(CNN)engine CNNenginecluster DataCenterGPU2cluster

DataCenterGPUs

FPGAs

EmbeddedGPUs

VisionDSPs

Vision+NNDSPs

CNNengines

GPCPU

ProductivityandNeuralNetworksmachinelearningtechnologyisonthecuspofeatingsoftware-AlexWoodie

Productivity: Trainingneuralnetworksoftenmucheasierthancodingofapplication-specificfeatures

Neuralnetworkmethodsroutinelyperformbetterthanbestmanualmethods

MassiveeducationalshiftunderwaytomakemachinelearningabasicCSskill

MOOCs:AndrewNgsStanfordMachineLearningcourse:100,000studentsregistered. Hypestillexceedsreality

ProductivityImpactatThreeLevels:1.Programming:TrainedNNreplacehand-designoffeaturedetectors

2.Applications:RichmachinelearningframeworksmakeopenssophisticatedNNtonon-experts

3.Business:DeepLearningmethodsareenoughbetterandfastertoinfluencethewholetechsector.

Weareinthefirstphase,whenveryrapidproductivitygainsbecomepossibleinpartsoftheeconomythataresimplytoosmalltoaffecttheoverallnumbers [T]hetechsectorshouldexperiencegreaterproductivitygainsoveragreaterrangeofbusinesses,potentiallynudgingmeasuredproductivityupward.-- TheEconomist,July23,2014

TheEmbeddedSystemsInnovationSpace

InnovationProductivity

Implem

entatio

nEfficiency

Moo

resLaw

DeepLearningIm

proved

algorith

ms

TheEvolutionofElectronicDesignCognitiveComputingwillbealong-termdriverforelectronics

1930 1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 2040

AnalogCircuits:tubediscreteICIP Reuse

DigitalCircuits:discreteTTLRTLIP Reuse

ProcessorBased:AssemblyHLLEcoSystemOpen Source

CognitiveComputing

Circuitsim Customlayout Extraction/DRC

3-statesim Logicsynthesis Std CellP&R Statictiming Formalverification

Optimizingcompiler RTOS Debugger/profilerMPprogramming Appdev framework

Parallelstochasticnetworktraining Networkstructuresynthesis Autodatalabeling&augmentation

neuralnetworktechnologyandapplications