beyond real-time video surveillance analytics with gpus
TRANSCRIPT
[email protected] – www.intuvisiontech.com
Beyond real-time video surveillance
with GPU accelerated Panoptes
Dr. Sadiye Guler intuVision, Inc. August 14, 2013
GeoInt Accelerator • Platform to start developing with GPUs
• GPU access, GPU-accelerated apps and
libraries
• Register to learn more:
http://www.nvidia.com/geoint
Webinar Feedback Submit your feedback for a chance to win Tesla K20 GPU* https://www.surveymonkey.com/s/intuvision
* Offer is valid until September 1st 2013
intuVision, Inc. Proprietary
Outline
• Introduction
• Intelligent Video Surveillance Background
• intuVision Approach
• CPU vs CPU+GPU
– Utilizing GPU’s in Video Analysis
– Benefits in Performance and Accuracy
• OpenCV CUDA Advantage
• intuVision Panoptes System
• Concluding Remarks
intuVision, Inc. Proprietary
Outline
• Introduction
• Intelligent Video Surveillance Background • intuVision Approach
• CPU vs CPU+GPU
– Utilizing GPU’s in Video Analysis
– Benefits in Performance and Accuracy
• OpenCV CUDA Advantage
• intuVision Panoptes System
• Concluding Remarks
intuVision, Inc. Proprietary
Intelligent Video Surveillance: Promise
Intelligent analysis turns data into knowledge
Data collections
knowledge/ intelligence
information
Intelligent Video Surveillance … turns video cameras into
smart sensors…
Stacks of HD’s, reams of tapes
Walls of TV monitors
Decisions
Actions
intuVision, Inc. Proprietary
If an image is worth a thousand words, a second of video may have 15-60 frames…1000 x framerate… worth a lot more
Once the frame contents are extracted into metadata it is more efficient to store/index/search
Intelligent Video Surveillance: Premise
intuVision, Inc. Proprietary
Object motion and appearance features
Intelligent Video Surveillance
Object trajectories, Object classes
Higher level analyses
Activities, events, anomalies
Real time Alerts, Warnings / Post Analysis Investigation
intuVision, Inc. Proprietary
Video Object Detection & Tracking Basics
• Background Model and Background Subtraction – Thresholded difference from a background model
– Different background models, Gray Scale, RGB color, Advanced color, Mean/Median, Mixture of Gaussians
– Provides approximate object silhouette
• Object detection and Tracking – Identifying foreground object pixels in each frame
– Correlating objects from frame to frame
– Classification of foreground objects (e.g. person, vehicle, etc.)
• Frame differencing – Thresholded difference of consecutive frames
– Provides a rough object region for moving objects
– Computationally inexpensive
– Stopped objects are no longer detected
intuVision, Inc. Proprietary
Outline
• Introduction
• Intelligent Video Surveillance Background
• intuVision Approach • CPU vs CPU+GPU
– Utilizing GPU’s in Video Analysis
– Benefits in Performance and Accuracy
• OpenCV CUDA Advantage
• intuVision Panoptes System
• Concluding Remarks
intuVision, Inc. Proprietary
intuVision Video Tracking Framework
Communications
Peripheral Tracker
Vision Tunnels
Stationary Object Layer
Objects
Tracks
Objects
BG Model
SO Model
Video
Input
Communications
Scene Description Layer
Peripheral Tracker
Object Layer
Stationary Object Layer
Video
Input
• Layered tracking approach inspired by human Multiple Object Tracking process
• Fast Peripheral Vision- Spatially based quick glance of the scene
– Fast but coarse detection and tracking of objects
• Object Tunnels- Object based focus for the tracked targets
– Detailed analysis and object classification
• Scene Description Layer
• Pixel, Edge and Noise Background Models
• Stationary Object Layer
• Detection of stopped or intermittent objects
• Task dedicated “as needed” processing
• Computational efficiency
intuVision, Inc. Proprietary
Peripheral Tracking Layer
Stationary Object layer
Scene Description Layer
Scene Description Layer
Peripheral Tracking Layer
Video Tracking Layers
intuVision, Inc. Proprietary
intuVision, Inc. Proprietary
Peripheral Tracking Layer
Stationary Object LLayer
Scene Description Layer
Scene Description Layer
Peripheral Tracking Layer
Object Layer
Object Object Object Object
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0 5 10 15 20 25 30 35
intuVision, Inc. Proprietary
Video Tracking Layers
intuVision, Inc. Proprietary
Input Frame
Background Model
Peripheral motion
Noise
Shadow pixels
Foreground object pixels
Video Tracking Layers
intuVision, Inc. Proprietary
Video Event Detection
Input Frames
Intermediate Representation
Event Detection Relative motion and spatial states of detected objects
Classification of detected objects
intuVision, Inc. Proprietary
Parallelizable tasks in intuVision framework
• Detecting short term Peripheral object motion
• Maintaining a long term background model for every pixel
– Update background mean and standard deviation
– Support for GrayScale, RGB and La*b* color models for background
• Determining foreground pixel probability validated by Peripheral motion compare Background with each new frame
• Detecting stationary and intermittent objects
• Maintaining False Motion Models to account for noise due to dynamic scene elements
• Applying morphological filters to foreground pixels
intuVision, Inc. Proprietary
Outline
• Introduction
• Intelligent Video Surveillance Background
• intuVision Approach
• CPU vs CPU+GPU – Utilizing GPU’s in Video Analysis
– Benefits in Performance and Accuracy
• OpenCV CUDA Advantage
• intuVision Panoptes System
• Concluding Remarks
intuVision, Inc. Proprietary
CPU vs GPU
• CPU architecture – few general purpose processors capable of multithreading
– optimized for instruction level parallelism
• GPU architecture – many small specialized processors executing Single Instruction on
Multiple Data (SIMD)
– optimized for data
level parallelism
core
[0,1]
core
[0,2]
core
[0,6]
core
[1,1]
core
[1,2]
core
[1,6]
core[3,
1]
core
[3,2]
core
[3,6]
core
[0,0]
core
[1,0]
core
[3,0]
core
[0,3]
core
[1,3]
core
[3,3]
core[0,
4]
core
[1,4]
core
[3,4]
core
[0,5]
core
[1,5]
core
[3,5]
core
[2,1]
core
[2,2]
core
[2,6]
core
[2,0]
core
[2,3]
core
[2,4]
core
[2,5]
core
[0,7]
core
[1,7]
core
[3,7]
core
[2,7]
G
P - 0
U
G
P - 1
U
G
P - 2
U
G
P - 3
U
Reduced computing time
Reduced hardware costs
Reduced power consumption
intuVision, Inc. Proprietary
All pixel based operations are independent of their neighbors and
can be performed in parallel on the GPU
– Reduced computing time - More frames processed per second
– Reduced hardware costs - Low-end computers can be boosted with GPU’s
Video Analysis with GPU Acceleration
GPU
[0,1]
GPU
[0,2]
GPU
[0,6]
GPU
[1,1]
GPU
[1,2]
GPU
[1,6]
GPU
[3,1]
GPU
[3,2]
GPU
[3,6]
GPU
[0,0]
GPU
[1,0]
GPU
[3,0]
GPU
[0,3]
GPU
[1,3]
GPU
[0,4]
GPU
[1,4]
GPU
[3,4]
GPU
[0,5]
GPU
[1,5]
GPU
[3,5]
GPU
[2,1]
GPU
[2,2]
GPU
[2,6]
GPU
[2,0]
GPU
[2,3]
GPU
[2,4]
GPU
[2,5]
GPU
[0,7]
GPU
[1,7]
GPU
[3,7]
GPU
[2,7]
intuVision, Inc. Proprietary
CPU only CPU + GPU
CPU
GPU
[0,1]
GPU
[0,2]
GPU
[0,7]
GPU
[1,1]
GPU
[1,2]
GPU
[1,7]
GPU
[3,1]
GPU
[3,2]
GPU
[3,7]
GPU
[0,0]
GPU
[1,0]
GPU
[3,0]
CPU
Time
High processing fps, left object detected in time
intuVision, Inc. Proprietary
• In the GPU architecture the data parallel portions of an application are executed as kernels in parallel
• The CUDA processing model uses a grid of 1D, 2D or 3D blocks with each block running multiple threads
• In this model it is important to minimize the control flow divergence as each thread executes the same kernel and any divergence results in the slowing down of all the threads
• A group of 32 threads is defined as warp in the CUDA model and it corresponds to the smallest executable unit of parallelism on the GPU device
GPU Instruction Execution
intuVision, Inc. Proprietary
GPU Memory to Remember
• The CUDA memory model contains multiple memory spaces: – Thread, local, block, shared, global, constant and texture memory spaces
– Global, constant and texture memory spaces are persistent across kernels
– Global, local and texture memory have the greatest access latency
intuVision CUDA implementation employs:
• Reduced memory access and improved memory access patterns
• Memory coalescing – Enforces consecutive threads of a warp to concurrently request consecutive
logical addresses from global memory to minimize the global memory access
• Use of CUDA textures – cached unlike the global memory
Side by Side
Vehicle Counting
but far apart… intuVision, Inc. Proprietary
GPU Performance Benefit Example
Video Resolution CPU (msec) GPU (msec)
GrayScale RGB La*b* GrayScale RGB La*b*
640 x 480 16.50 21.05 58.05 10.66 10.21 12.82
1280 x 720 39.06 56.80 184.23 16.10 16.07 23.63
Use of the GPU enables faster than real time processing with the standard RGB model and the more computationally intensive advanced color La*b* model
Off-the –shelf hw platform: CPU – Intel Core i7-3770 (4 Cores)
GPU – NVIDIA GTX690 (3092 Cores)
Better than real-time
intuVision, Inc. Proprietary
GPU Performance Benefits in Numbers
CPU+ GPU processing speed up factor over CPU
0
1
2
3
4
5
6
7
8
9
640 x 480 1280 x 720
Grayscale
RGB
La*b*
Video Resolution
GP
U-C
PU
sp
ee
du
p fa
cto
r
Intel Core (4) i7-3770 NVIDIA GTX690
intuVision, Inc. Proprietary
Full scale Video Analysis Video Object Detection
L*a*b* Color Background Model RGB Background Model
Grayscale Background Model
Missed pixels
Shadows
GPU Accuracy Benefits
intuVision, Inc. Proprietary
GPU Accuracy Benefits (before)
Large area monitoring with wide angle view HD camera
Traditional CPU based processing cannot maintain a real time frame rate
– Results in background model deterioration and missed detections
intuVision, Inc. Proprietary
GPU Accuracy Benefits (after)
Large area monitoring with wide angle view HD camera
GPU based processing of the video correctly learns the background
– Results in reduced missed detections, better motion history improved tracking
intuVision, Inc. Proprietary
intuVision, Inc. Proprietary
Outline
• Introduction
• Intelligent Video Surveillance Background
• intuVision Approach
• GPU vs CPU
– Utilizing GPU’s in Video Analysis
– GPU Benefits in Performance
• OpenCV CUDA Advantage • intuVision Panoptes System
• Concluding Remarks
intuVision, Inc. Proprietary
intuVision use of OpenCV CUDA
• More than 250 optimized functions
• Supports filtering, feature extraction, object detection, optical flow, etc.
– Computation intensive tasks like object detection can be performed in real time
• Haar Cascade Classifier – 6x speed up
• Allows for quick prototyping of algorithms and determining the GPU speedup
intuVision, Inc. Proprietary
Clutter and dynamic texture detection
Without clutter and dynamic texture removal
With clutter and dynamic texture removal
Input Image with boat&wake detection
Foreground Image
intuVision, Inc. Proprietary
Example use of OpenCV: Clutter and dynamic texture removal filter
• Identify techniques for clutter and dynamic texture detection
• Validate a prototype using OpenCV CUDA implementations find out GPU speedup potential
• Prototype optimization
– Develop optimized version of algorithm components
– Replace off the shelf algorithms with custom implementation
• GPU implementation of clutter and dynamic texture detection results in more than 2.5x speedup
intuVision, Inc. Proprietary
Clutter and dynamic texture removal
Without clutter and dynamic texture removal
With clutter and dynamic texture removal
intuVision, Inc. Proprietary
Outline
• Introduction
• Intelligent Video Surveillance Background
• intuVision Approach
• GPU vs CPU
– Utilizing GPU’s in Video Analysis
– GPU Benefits in Performance
• OpenCV CUDA Advantage
• intuVision Panoptes System • Concluding Remarks
intuVision, Inc. Proprietary
Panoptes Real-Time and Beyond Video Event Monitoring
Panoptes “the all seeing”
Argus Panoptes –the God with 1000 eyes
Event detection and logging , alarm schedules, alarm metadata and triggers
Faster than real-time processing of archived video (up to 120fps)
Video from live or archive streams
BW, Color and IR
CIF to HD resolutions
Panoptes Event:
Wrong Direction
9:38 am Jan 15, 2010
intuVision, Inc. Proprietary
Panoptes Intuitive User Interface
Color coded List of Detected Alarms
Current Alarm Detection marked with a flashing red frame
IR Camera Monitoring
Selected Alarm Event Details
intuVision, Inc. Proprietary
Panoptes
Object Classification User trainable object
classification
Automatically collected
training samples
Robust models for
people and vehicles
Learned models apply to
similar views Training Samples
Person Vehicle Group
intuVision, Inc. Proprietary
Panoptes Events
Abandoned Object Objects left unattended for a period of time
Activity Objects detected in a Region of Interest
Crowd Density People gathering for a set period of time
Enter/Exit An object enters/exits the view from a specified area
Idle Object A moving object coming to stop for a duration
Line Crossing An object moving passed over a specified line
intuVision, Inc. Proprietary
Panoptes Events
Object Taken Objects moved from marked regions
Object Counting Counting objects in a Region of Interest
Perimeter Intrusion A moving object entering into a marked zone
Speeding Object An object moving too quickly between set lines
Wrong Way Movement in the specified wrong direction
Smoke & Fire Detection Detecting sections of a scene with smoke or fire
intuVision, Inc. Proprietary
Panoptes Counting Events
Count any Panoptes Event
Use with any object type
Person
Vehicle
Animal
Boat
Etc.
Easy-to-use count options
Overlay on camera view
List in the object counting
window
intuVision, Inc. Proprietary
Panoptes Compound Events
Compound Events:
Specified events happening in combination over camera(s)
Wrong Direction in Camera-1 Stopped vehicle in Camera-2
AND
intuVision, Inc. Proprietary
Camera-2
Idle Vehicle
Intrusion
Compound Event
Panoptes
Floor Plan Camera Monitoring
Monitor cameras
from an intuitive
floor-plan layout
Observe camera
interactions at a
glance
Receive notifications
and respond quickly
as events unfold
intuVision, Inc. Proprietary
Panoptes Compound Event
Get-away car & indoor office activity after hours.
intuVision, Inc. Proprietary
Outline
• Introduction
• Intelligent Video Surveillance Background
• intuVision Approach
• CPU vs CPU+GPU
– Utilizing GPU’s in Video Analysis
– Benefits in Performance and Accuracy
• OpenCV CUDA Advantage
• intuVision Panoptes System
• Concluding Remarks
intuVision, Inc. Proprietary
Summary
• Video Content Analysis continues to be a fast growing technology area
with lots of potential and applications
• Accuracy reliability, scalability, performance and robustness matter!
• GPU’s offer low cost options for increased performance and scalability
– With increased performance more complex algorithms can be run in real-
time (and beyond) resulting in increased accuracy
• intuVision Video Analytics are GPU accelerated
– Performance gains up to 12x are obtained with addition of Nvidia GPU
cards to off-the-shelf low-end hardware platforms
– Optimization to newer GPU platforms is ongoing
intuVision, Inc. Proprietary
Video Content Analysis: Practice
data knowledge/
intelligence information incomplete
uncertainty
noise
Decisions
Actions
• Several interesting problems remain to be solved in extracting video content:
• Multi camera environments, understanding events at different granularity…
• Also several not-so-interesting problems remain to be solved to have robust
systems:
• Illumination variations, shadows, occlusions, camera jitter, varying frame rates…
Human
in the
loop
intuVision, Inc. Proprietary
Thank you!
Dr. Sadiye Guler intuVision, Inc.
www.intuvisiontech.com
Surveillance bee
Upcoming GTC Express Webinars
Register at www.gputechconf.com/gtcexpress
August 15 - CUDA 5.5 Production Release: Features Overview
September 5 - Data Discovery through High-Data-Density Visual
Analysis using NVIDIA GRID GPUs
September 10 - Virtualizing Tough 3D Workloads with VMware
Horizon View and NVIDIA Technologies
September 12 - Guided Performance Analysis with NVIDIA
Visual Profiler
September 17 - ArrayFire: A Productive GPU Software Library
for Defense and Intelligence Applications
GTC 2014 Call for Submissions
Looking for submissions in the fields of
Science and research
Professional graphics
Mobile computing
Automotive applications
Game development
Cloud computing
Submit by September 27 at www.gputechconf.com
GeoInt Accelerator • Platform to start developing with GPUs
• GPU access, GPU-accelerated apps and
libraries
• Register to learn more:
http://www.nvidia.com/geoint
Webinar Feedback Submit your feedback for a chance to win Tesla K20 GPU* https://www.surveymonkey.com/s/intuvision
* Offer is valid until September 1st 2013