recent developments in embedded vision: algorithms ......© 2017 embedded vision alliance 3 vision...
TRANSCRIPT
![Page 1: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/1.jpg)
© 2017 Embedded Vision Alliance 1
Jeff Bier
Founder, Embedded Vision Alliance | President, BDTI
Vision Systems Design Webcast — January 25, 2017
Recent Developments in Embedded Vision:
Algorithms, Processors, Tools and
Applications
![Page 2: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/2.jpg)
© 2017 Embedded Vision Alliance 2
Computer vision: research and fundamental
technology for extracting meaning from images
Machine vision: factory applications
Embedded vision: thousands of applications
• Consumer, automotive, medical, defense, retail,
gaming, security, education, transportation, …
• Embedded systems, mobile devices, PCs and the
cloud
The Evolution of Vision Technology
![Page 3: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/3.jpg)
© 2017 Embedded Vision Alliance 3
Vision is the Sensing Opportunity
-
10,000
20,000
30,000
40,000
2010 2011 2012 2013 2014 2015 2016 2017 2018
Un
it p
er
Ye
ar
(M)
Sensor Unit Volume: Source: Semico Research, 2014
Microphone
Image
Gyroscope
Accelerometer
Ambient light
Proximity
Magnetometer
Pressure
Touch
Fingerprint
Chemical/gas
Temperature
Ultrasonic
IR
Biological
Humidity
Hall effect
UV
ECG
EMG
Ultrasonic
EEG
Source: Chris Rowen, Cadence
![Page 4: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/4.jpg)
© 2017 Embedded Vision Alliance 4
Vision is the Sensing Opportunity
-
10,000
20,000
30,000
40,000
2010 2011 2012 2013 2014 2015 2016 2017 2018
Un
it p
er
Ye
ar
(M)
Sensor Unit Volume: Source: Semico Research, 2014
Microphone
Image
Gyroscope
Accelerometer
Ambient light
Proximity
Magnetometer
Pressure
Touch
Fingerprint
Chemical/gas
Temperature
Ultrasonic
IR
Biological
Humidity
Hall effect
UV
ECG
EMG
Ultrasonic
EEG
0.00E+00
2.00E+12
4.00E+12
6.00E+12
8.00E+12
1.00E+13
1.20E+13
1.40E+13
2010 2011 2012 2013 2014 2015 2016 2017 2018D
ata
-Rate
We
igh
ted
Vo
lum
e p
er
Ye
ar
(M
un
its *
bit
s p
er
se
co
nd
)
Sensor Volume Adjusted for Data Rate
Source: Chris Rowen, Cadence
![Page 5: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/5.jpg)
© 2017 Embedded Vision Alliance 5
The Internet of Things That See: Compology
Video: bit.ly/2jnl8Po
![Page 6: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/6.jpg)
© 2017 Embedded Vision Alliance 6
• For embedded vision to achieve its potential, we need:
• High-performance, energy-efficient, inexpensive processors
• More-capable, more integrated and less expensive image sensors
• More capable and reliable algorithms
• Improved software development productivity
• More skilled engineers
Overcoming Critical Challenges
![Page 7: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/7.jpg)
© 2017 Embedded Vision Alliance 7
Processors
![Page 8: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/8.jpg)
© 2017 Embedded Vision Alliance 8
• Ideally, heterogeneous
processors combine:
• Most of the performance and
efficiency of specialize
processors
• Most of the ease of
development of CPUs
• But it comes at a cost:
• complexity
Trend: Heterogeneous Processors for
Performance and Efficiency
Performance/$
Performance/W
Development
Effort
![Page 9: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/9.jpg)
© 2017 Embedded Vision Alliance 9
Processor Chips:
• Analog Devices BF609
• Inuitive NU3000
• MobileEye EyeQ5
• Movidius (now Intel) Myriad 2
• NXP S32V
• Texas Instruments TDA3x, TDA2Ec
Trend: Vision-specific Processors
Processor Cores:
• Apical (now ARM) Spirit
• Cadence Vision P5, Vision P6
• CEVA MM-3101, XM-4, XM-6
• Synopsys EV5x, EV6x
• Vivante VIP7000,
GC7000-XS VX
Inuitive M3 Reference Design Movidius Myraid 2
![Page 10: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/10.jpg)
© 2017 Embedded Vision Alliance 10
Type of Processor Used for Vision Tasks—
Ranked as One of Top 3 (July 2016)
75%
64%
44%
31% 31% 26%
4%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
CPU GPU Mobile Vision-specific FPGA DSP Other
![Page 11: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/11.jpg)
© 2017 Embedded Vision Alliance 11
Maximizing Processing per Watt: ODG
![Page 12: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/12.jpg)
© 2017 Embedded Vision Alliance 12
Algorithms
![Page 13: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/13.jpg)
© 2017 Embedded Vision Alliance 13
• Infinitely varying inputs in many
applications
• Uncontrolled conditions: lighting,
orientation, motion, occlusion
• Leads to ambiguity…
• Leads to the need for complex,
multi-layered algorithms to extract
meaning from pixels
• Plus:
• Lack of analytical models means
exhaustive experimentation is
required
• Numerous algorithms and algorithm
parameters to choose from
Vision Algorithms Are Challenging
www.selectspecs.com
![Page 14: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/14.jpg)
© 2017 Embedded Vision Alliance 14
Embedded Vision Challenge: Algorithms
Source: xkcd.com
![Page 15: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/15.jpg)
© 2017 Embedded Vision Alliance 15
Source: hitl.washington.edu/artoolkit Source: xkcd.com
Embedded Vision Challenge: Algorithms
![Page 16: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/16.jpg)
© 2017 Embedded Vision Alliance 16
Deep Neural Networks: Learning Machines
Source: NVIDIA
![Page 17: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/17.jpg)
© 2017 Embedded Vision Alliance 17
Use of Neural Networks to Perform
Computer Vision Functions (July 2016)
Yes, extensively
19%
Yes, in a minor role 19%
Not yet, but planning to
37%
No 22%
Don't know 3%
![Page 18: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/18.jpg)
© 2017 Embedded Vision Alliance 18
Originally used solely for classification, deep
neural networks are now also being used for:
• Detection
• Segmentation
• Sequences (e.g., video captioning)
• Optical flow
• Visual motor control
Expanding Applicability of Deep Learning
Source: Long, Shelhamer, Darrell. CVPR’15
Source: Dosovitskiy et al., ICCV ‘15
![Page 19: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/19.jpg)
© 2017 Embedded Vision Alliance 19
Deploying Deep Learning: Nauto
![Page 20: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/20.jpg)
© 2017 Embedded Vision Alliance 20
Trend: Processors for Deep Learning
Degree of Specialization
Software Packages/
Libraries, Frameworks
Architecture Enhancements
Dedicated
Co-processor
None
Architecture Focus
Tensor
Processing
Unit
Movidius
(now Intel)
Myriad 2
Nervana
(now Intel)
NVIDIA
Tegra X1
![Page 21: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/21.jpg)
© 2017 Embedded Vision Alliance 21
Sensors
![Page 22: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/22.jpg)
© 2017 Embedded Vision Alliance 22
Sensor Innovation Accelerates
Image Sensors World
ims—chips.de
Chronocam
Edn-europe.com
![Page 23: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/23.jpg)
© 2017 Embedded Vision Alliance 23
Proliferation of 3D Sensors
Infineon.com
Inuitive-tech.com
Scandy.co
Photographylife.com
![Page 24: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/24.jpg)
© 2017 Embedded Vision Alliance 24
Understanding the World in 3D: RetailNext
Retainlext.com
TechCrunch.com
![Page 25: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/25.jpg)
© 2017 Embedded Vision Alliance 25
Software Development
![Page 26: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/26.jpg)
© 2017 Embedded Vision Alliance 26
Developing efficient vision software is challenging, due to:
• Demanding, complex, ever-changing algorithms
• Complex, heterogeneous, cutting-edge processors
• Limited infrastructure (tools, libraries, frameworks, etc.)
Challenge: Vision Software Development
Qualcomm Conceptdraw hitl.washington.edu/artoolkit
![Page 27: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/27.jpg)
© 2017 Embedded Vision Alliance 27
• New Khronos standard API for acceleration of
vision applications
• Addresses heterogeneous processors (CPU,
vision co-processor, GPU, DSP, etc.)
• Promises portability across chips
• Developer specifies data flow graph of vision
algorithm kernels
• Kernels may be implemented in any language
• OpenVX framework partitions the graph
among processing engines, handles memory
transfers, etc.
• Designed for mobile, useful in other domains
Solution: Standards-based Abstractions
Vision
Accelerator
Application
Application
Application
Application
Vision
Accelerator Vision
Accelerator Vision
Accelerator
Khronos Group
![Page 28: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/28.jpg)
© 2017 Embedded Vision Alliance 28
Libraries and APIs Used for Vision Tasks—
Ranked as One of Top 3 (July 2016)
82%
47%
26% 26%
15%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
OpenCV OpenGL FastCV OpenVX Other
![Page 29: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/29.jpg)
© 2017 Embedded Vision Alliance 29
• Caffe
• Vuforia
Solution: Domain-specific Frameworks
Source: CEVA
Source: Gravity Jack
![Page 30: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/30.jpg)
© 2017 Embedded Vision Alliance 30
Vision-enabled Autonomy: DJI Phantom 4
dcrainmaker.com
![Page 31: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/31.jpg)
© 2017 Embedded Vision Alliance 31
Vision has been out of reach for most potential applications,
so few engineers in industry have vision expertise
The Embedded Vision Alliance (www.Embedded-Vision.com)
is a partnership of 60+ vision technology suppliers
Mission: Inspire and empower product creators to incorporate
visual intelligence into their products
The Alliance provides high-quality, practical technical
educational resources for engineers
• Alliance website offers tutorial articles, video “chalk
talks,” forums
• Embedded Vision Insights newsletter—news and updates
Register for updates at www.Embedded-Vision.com
Challenge: Skilled Engineers
![Page 32: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/32.jpg)
© 2017 Embedded Vision Alliance 32
• Embedded vision enables software-defined sensors, uniquely capable of
delivering insights about the real world
• Accelerating advances in processors, sensors, algorithms and
development tools make it increasingly practical to deploy vision
• Embedded vision is becoming ubiquitous
• Vision is becoming a huge creator of value—for technology suppliers,
end-product developers, and users
Take-aways
![Page 33: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/33.jpg)
© 2017 Embedded Vision Alliance 33
The only industry event focused on enabling
developers to create “machines that see”
• “Awesome! I was very inspired!”
• “Fantastic. Learned a lot and met great people.”
• “Wonderful speakers and informative exhibits!”
Embedded Vision Summit 2017 highlights:
• Inspiring keynotes by leading innovators
• Practical technical, business and product talks
• Exciting demos of the latest technologies
• Visit www.EmbeddedVisionSummit.com for details
• Register by February 1 using discount code
vsdwebcast0125
Join us at the Embedded Vision Summit –
May 1-3, 2017—Santa Clara, California
![Page 34: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/34.jpg)
© 2017 Embedded Vision Alliance 34
Embedded Vision Alliance Member Companies
![Page 35: Recent Developments in Embedded Vision: Algorithms ......© 2017 Embedded Vision Alliance 3 Vision is the Sensing Opportunity -10,000 20,000 30,000 40,000 2010 2011 2012 2013 2014](https://reader033.vdocuments.net/reader033/viewer/2022043009/5f9ce29a4280e04260330603/html5/thumbnails/35.jpg)
© 2017 Embedded Vision Alliance 35
Email me ([email protected]) for:
• A copy of these slides
• Links to videos of cool embedded-vision-based products
• Questions about the Embedded Vision Summit, May 1-3, 2017,
Santa Clara, California, USA
Questions?
Jeff Bier
President, BDTI
Founder, Embedded Vision Alliance
Chairman, Embedded Vision Summit
www.BDTI.com
www.embedded-vision.com
+1 925-954-1411
Walnut Creek, CA 94596 U.S.A.