s7348: deep learning in ford's autonomous vehicles · s7348: deep learning in ford's...
TRANSCRIPT
![Page 1: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/1.jpg)
S7348: Deep Learning in Ford's Autonomous Vehicles
Bryan Goodman
Argo AI
9 May 2017
1
![Page 2: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/2.jpg)
Today: examples from• Stereo image processing
• Object detection• Using RNN’s
• Motorsports
2
Ford’s 12 Year History in Autonomous Driving
![Page 3: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/3.jpg)
Stereo Matching Problem
• Determining the correspondences in stereo images
• Calculating the disparities
• But what is the correct correspondence?
• Basic stereo matching algorithm− Compare pixels on the same
epipolar line in two images
− Choose the best match
3
![Page 4: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/4.jpg)
Deep neural networks for stereo matching
• The brain can estimate the distance of an object using the visual information from two eyes.
• We can use deep neural networks
4
Right Stereo Camera
Deep Convolutional Neural Networks
Post-Processing
Left Stereo Camera
Distance Map Estimation
![Page 5: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/5.jpg)
Proposed deep convolutional neural network• AV driving requires an intelligent distance map estimation, which filters out the
objects not of interest.• Network I
− General network
− Encoding and decoding layers
− Retain objects of interest in the training data sets
5
Co
nv1 Conv5
Co
nv2
Co
nv3
Co
nv6
Deco
nv6
Co
nv7
De
con
v7
Deco
nv8
Co
nv8
Deco
nv9
Encoder Decoder
Co
nv9
Loss Function
Deco
nv1
0
Co
nv1
0
Co
nv4
![Page 6: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/6.jpg)
Proposed deep convolutional neural network II
− Specialized network
− Encoding and decoding layers
− The cross correlation layers force the network to look for correspondence on the epipolar line
− The weights in the encoding layers are shared
6
Co
nv1
L
Conv4L
Loss
Fun
ction
Encoder Decoder
Co
nv2
L
Co
nv3
L
Co
nv1
R
Conv4R
Co
nv2
R
Co
nv3
R
CC
5
Co
nv5
Co
nv6
CC
6D
econ
v6
Co
nv7
CC
7
Deco
nv7
Deco
nv8
Co
nv8
Deco
nv9
Co
nv9
![Page 7: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/7.jpg)
Proposed deep convolutional neural network
• Cross correlation (CC) layer− Computes CC values between each pairs of
patches
− Outputs the CC values for each pair of patches
− Does not lose any information
• Loss function− In AV driving, closer objects are more important
than distant ones
− Assigns more weight to the closer objects
− The closer object distance is estimated more accurately
7
0.2 0.4 0.6 0.8 1
1
0.4
0.2
α
d
0.6
0.8
![Page 8: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/8.jpg)
Performance on synthetic and real stereo data
• Synthetic data generation− Generate 14,000 pairs of RGB stereo images
− Synthetic distance maps are only generated for the objects of interest, e.g. cars or pedestrians
− Gaussian noise added to the stereo images
8
![Page 9: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/9.jpg)
Performance on synthetic and real stereo data• Fine tuning with LIDAR data sets
− Project LIDAR point clouds onto the camera images
− The baseline and optic axes are not the same as the synthetic data
9
Left camera Right camera Network I Network II
![Page 10: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/10.jpg)
1/2x
10
![Page 11: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/11.jpg)
Comparing Manual Annotation to DNN Model
11
![Page 12: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/12.jpg)
12
Detection Result Original Image Enhanced Contrast
Network’s detection outperforms human labelerin low-contrast areas
Pedestrian detection Pedestrian misdetection Detected, but not labeled
![Page 13: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/13.jpg)
Introducing Recurrence in Detection and Tracking
• Use RNN’s to detect occluded objects• Remember location of static objects
• Predict location of non-static objects
13
Image 0
FeatureMap
RNN Conv
Image 1
FeatureMap
Image 2
FeatureMap
RNN Conv RNN Conv
Detector Detector Detector
![Page 14: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/14.jpg)
Orange = ground truth; Green = model prediction
14
![Page 15: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/15.jpg)
15
Classifying NASCAR images
The Ford team reviews pictures during the race
![Page 16: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/16.jpg)
16
Classifying NASCAR images
Looking for damage and other performance indicators
Gap
![Page 17: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/17.jpg)
17
Results –Boxing the Cars
Using ~2k images labeled
with boxes around the
vehicles, the model does
well detecting cars
![Page 18: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/18.jpg)
18
Results –Boxing the Cars
![Page 19: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/19.jpg)
Classifying NASCAR images
Next –determine car
number:labeled ~30k
images
![Page 20: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/20.jpg)
Classifying NASCAR images
Outliers easy to find in review
![Page 21: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/21.jpg)
Classifying NASCAR images
Human: ???Model: 78
Confidence: 0.999
![Page 22: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/22.jpg)
Classifying NASCAR images
Human: ???Model: 42
Confidence: 0.985
![Page 23: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/23.jpg)
Inspecting the Neural Network
23
Activated Filter Input Image
The Model is not a black box. We can see that it is detecting the numbers – important for robustness when the paint changes
![Page 24: S7348: Deep Learning in Ford's Autonomous Vehicles · S7348: Deep Learning in Ford's Autonomous Vehicles Bryan Goodman Argo AI 9 May 2017 1](https://reader035.vdocuments.net/reader035/viewer/2022070915/5fb5cd0553c13c0bb33ae7f1/html5/thumbnails/24.jpg)
Argo AI
• Argo AI is an artificial intelligence company, established to tackle one of the most challenging applications in computer science, robotics and artificial intelligence: self-driving vehicles
• Engineering hubs in Pittsburgh, Southeastern Michigan and the Bay Area of California
• For more information regarding Argo AI and its work, please talk to me at GTC or visit: www.argo.ai
24