optimizing pedestrian detection for real-time …...scope of the presentation ! what you will see in...
TRANSCRIPT
![Page 1: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/1.jpg)
Optimizing Pedestrian Detection For Real-time Automotive Applications Vladimir Glavtchev Automotive Computer Vision Engineer NVIDIA
![Page 2: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/2.jpg)
Agenda
§ Scope of this presentation
§ Introduction to Pedestrian Detection
§ Survey of Pedestrian Detection Techniques
§ Academic Focus
§ Production ADAS Focus
§ Optimizations
§ A Path to Production
§ Demo
![Page 3: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/3.jpg)
Scope of the Presentation
§ What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection
— High-level concepts
— The gap between academia and industry in ADAS
— Optimizations needed to bridge this gap
— Hardware recommendations for various algorithms
§ What you will not see in this talk: — Low-level CUDA-specific optimizations
— Marketing pitch / jargon
— Code!
![Page 4: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/4.jpg)
Introduction to Pedestrian Detection
§ Examples
Image courtesy: Dollar et al– “Pedestrian Detection: An Evaluation of the State of the Art”
![Page 5: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/5.jpg)
Research Examples
§ Example vehicles — VisLab
— Lexus
Image courtesy VisLab Image Courtesy Lexus
![Page 6: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/6.jpg)
Our goal: Integrated system
§ Windshield mounted camera(s)
Image courtesy Valeo
![Page 7: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/7.jpg)
Production Examples
§ Current night vision pedestrian detection systems — German automakers
— Volvo, Lexus, etc
Image courtesy BMW AG
![Page 8: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/8.jpg)
Terminology
§ Introduction to terminology: — True positive (correct alarm)
§ Detecting an existing pedestrian.
— False positive (false alarm) § System alarms of pedestrian when none is present.
— True negative (correct non-alarm) § System correctly does not alarm because no pedestrian is present.
— False negative (incorrect non-alarm) § System failing to alarm when a pedestrian is actually present.
![Page 9: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/9.jpg)
Terminology
§ Introduction to terminology: — Preprocessing
§ Steps taken to rectify, undistort, enhance, and otherwise enable easier classification and tracking.
— Classification § Detecting a pedestrian in an image.
— Tracking § “Following” a pedestrian in a sequence of images after a classification.
![Page 10: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/10.jpg)
Classification State-of-the-Art Techniques
• Haar wavelets
• Histograms of Oriented Gradients (HOG)
• Local Binary Patterns (LBP)
![Page 11: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/11.jpg)
Classification State-of-the-Art Techniques
§ Current state-of-the-art is: — Sped up version of ChnFtrs1 classifier (Integral Channel Filters)
— HOG-like features + Cascade structure + SVM2
1 Dollar et al – “Fastest pedestrian detector in the west”
2 Benenson et al – “Pedestrian detection at 100 frames per second”
![Page 12: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/12.jpg)
Academic Focus
§ Accuracy
§ Novelty
§ Trade-offs
§ Grayscale / color
§ Depth / stereo
§ Public datasets — Not necessarily representative of in-vehicle camera footage (INRIA,
TUD Brussels)
![Page 13: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/13.jpg)
Academic Detection Pipeline
Input Frame
Preprocessing
Classification
False Positive Reduction
Tracking
Naive N models, 1 image scale
Traditional Approach 1 model, N image scales
Image courtesy: Benenson et al – “Pedestrian Detection at 100 frames per second”
![Page 14: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/14.jpg)
Naïve Implementation
![Page 15: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/15.jpg)
Traditional Academic Implementation
![Page 16: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/16.jpg)
Traditional Academic Implementation
![Page 17: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/17.jpg)
Traditional Academic Implementation
![Page 18: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/18.jpg)
§ Runtime performance — ChnFtrs runs at ~1 fps on modern desktop HW
P. Dollár, C. Wojek, B. Schiele and P. Perona “Pedestrian Detection: An Evaluation of the State of the Art”
Academic Pipeline Performance
![Page 19: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/19.jpg)
§ Scaling is expensive
§ Reduce scaling by a factor of K
Classification Optimization – Approach I
Dollar “FPDW” Approach 1 model, N / K image scales
![Page 20: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/20.jpg)
§ Runtime performance — Optimization achieves 3–7 fps on average
— But that’s still not fast enough!
Image courtesy: Dollár et al - “Pedestrian Detection: An Evaluation of the State of the Art”
Classification Optimization – Approach I
![Page 21: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/21.jpg)
§ Don’t scale image
§ Scale model by a factor of K
Classification Optimization – Approach II
Benenson “100 fps” approach N / K models, 1 image scale
![Page 22: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/22.jpg)
Benenson et al – “Pedestrian detection at 100 frames per second”
Performance: Approach II
Core i7 870 NVIDIA GeForce GTX 470 (14 SMs)
NVIDIA GeForce GT 640 (2 SMs)
Target Automotive GPU (1 SM)
0.08 fps 1.38 fps 0.20 fps 0.10 fps
§ Runtime performance — Performance for VGA (640x480) images
— GPU offers 15x performance boost over CPU
— Simple scaling of performance to number of SMs
![Page 23: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/23.jpg)
Search Space Reduction: Approach II
§ Reduce search space using stixels
Benenson et al – “Fast stixel computation for fast pedestrian detection”
![Page 24: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/24.jpg)
Production ADAS Focus
§ In-vehicle integration
§ Real-time operation
§ Accuracy
§ No false positives — Can put driver in dangerous situations
— Reduces driver confidence in the system
§ Cost — ASICs, FPGAs, General-purpose processors
§ Power
![Page 25: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/25.jpg)
From Academia to Production
§ Cameras — Color vs. Grayscale
§ Implications of grayscale: Retraining classifiers, reduced detection rate
— Monocular vs. Stereo § Implications of mono: No depth information, larger search space
— Infrared § Can significantly simplify night vision detection
§ Sensor fusion — Availability of lidar / radar
§ Can provide depth information for monocular cameras
![Page 26: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/26.jpg)
Proposed Solution: Motion Estimation
§ Observation: To an observer on a moving vehicle, closer objects move faster than objects farther away
![Page 27: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/27.jpg)
Towards ADAS: Motion Estimation
§ Calculate motion vector for each N x N pixel block — Compute motion from previous frame to current frame
— Possible methods: § Optical flow algorithms
§ Lucas-Kanade, Block-matching, Horn and Schunck
§ Block-based Iterative Motion Estimation (used for video encoding)
— Great fit for GPU because these algorithms are very parallel § Most operate on blocks of pixels
- Function available in OpenCV library
![Page 28: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/28.jpg)
Towards ADAS: Motion Example
![Page 29: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/29.jpg)
![Page 30: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/30.jpg)
![Page 31: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/31.jpg)
Towards ADAS: Motion Segmentation
§ Segment blocks of pixels which have a motion vector with: — High confidence
— High enough magnitude
— Similar direction
§ These segments represent the “foreground” — Objects which are moving faster than those around them
![Page 32: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/32.jpg)
![Page 33: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/33.jpg)
Towards ADAS: Geometry Reduction
§ Reducing the classification search space using geometric constraints
— Pedestrians cannot be taller than a certain height
— Pedestrians cannot be shorter than a certain height
— Pedestrians cannot be detached from the ground
§ What is needed? — Estimate of the ground plane
— Vehicle / camera pitch information
![Page 34: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/34.jpg)
![Page 35: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/35.jpg)
Tracking Bounding Box
Classification Bounding Box
Max. allowed height
Estimated Horizon
Min. allowed height
Estimated pedestrian base
Towards ADAS: Geometry Reduction
![Page 36: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/36.jpg)
![Page 37: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/37.jpg)
Pedestrian tracking
§ Why do we need to track? — Classification does not give us successful results at each frame
— Gives us a better approximation of a pedestrian’s trajectory
§ Tracking using motion information — Median Flow
§ Closed-loop tracking — MeanShift, CamShift, and TemplateMatching
- Function available in OpenCV library
![Page 38: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/38.jpg)
Tracking: MeanShift
Histogram of tracking box in Hue space
![Page 39: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/39.jpg)
Input Frame
Preprocessing
Classification
False Positive Reduction
Tracking
Input Frame
Preprocessing
Geometry Reduction
Depth/Motion Segmentation
Classification
Tracking
Traditional Academic Pipeline
Proposed Optimized Pipeline
15x
15x
Motion Estimation Free
8x
44x
Gain HW
Gain HW
![Page 40: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/40.jpg)
Demo
§ Video demo
![Page 41: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/41.jpg)
Benenson et al – “Pedestrian detection at 100 frames per second”
Integrated ADAS Performance
NVIDIA GeForce GTX 470 (14 SMs)
NVIDIA GeForce GT 640 (2 SMs)
Target Automotive GPU (1 SM)
50 fps 7.14 fps 3.57 fps
§ Monocular results
NVIDIA GeForce GTX 470 (14 SMs)
NVIDIA GeForce GT 640 (2 SMs)
Target Automotive GPU (1 SM)
135 fps 19.29 fps 9.64 fps
§ Stereo stixels + ground plane results — Motion Estimation & Geometric Reduction
![Page 42: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/42.jpg)
Conclusions
§ Full-frame classification is not fast enough yet
§ GPU acceleration can lead to a large speed-up in classification
§ Classification search space must be significantly reduced to achieve real-time results
§ Recent progress in academic research is employing practical system deployment concepts
§ Advances in sensors and processors will enable very high frame rates which will free up resources for other tasks
![Page 43: Optimizing Pedestrian Detection For Real-time …...Scope of the Presentation ! What you will see in this talk: — Introduction to state-of-the-art techniques in Pedestrian Detection](https://reader036.vdocuments.net/reader036/viewer/2022063006/5fb5fcaaf8ac8529f92eee48/html5/thumbnails/43.jpg)
Thank You!
§ Special thanks to:
— Elif Albuz
— Phillip Smith
— Shalini Gupta
— Khanh Duc