rpug2019 - pavement distress detection using advanced ... › pdfs › pe-2019 ›...
TRANSCRIPT
![Page 1: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/1.jpg)
Pavement Distress Detection Using Advanced Machine Learning Methods
with Intensity and Depth Data By
Matthew Connelly-Taylor
Andrea Annovi
Fugro Roadware
![Page 2: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/2.jpg)
2
Slide about who Fugro Roadware is - Coming from Allan
Fugro Roadware
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data 2
Founded in 1969
1st fully integrated road data collection vehicle (ARAN) in 1980
2019
56 ARANs operating in 18 countries
Over 10 Million miles of ARAN roadway data to date
Over 500 Thousand miles of ARAN roadway data each year
![Page 3: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/3.jpg)
3
Overview
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
• Why develop crack detection algorithms?
• Automation increases value of pavement data:
• Less human intervention = less subjectivity = more dependable results
• Faster results = more time to use the data = better decisions
• Current automated algorithms aren’t good enough
• Why Machine Learning?
• Rapidly improving field
• Excellent at solving complex problems with unstructured data
• Why us?
• We have 50 years of experience in pavement condition analysis
• We have a lot of accessible pavement data = 3 PetaBytes = 2 Million Miles
• …and it is already annotated
![Page 4: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/4.jpg)
4
Pave3DX Stereoscopic Imaging and Measurement
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
4.3m width 7296 pixels 0.6mm pixel
Image 2Image 1
![Page 5: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/5.jpg)
5
1mm Lines Combined into Image Frames
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
100 km/h
![Page 6: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/6.jpg)
6
2 Images from different angles
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
Image 2 (Right)Image 1 (Left)
![Page 7: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/7.jpg)
7
Combine Stereo Images
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
3D Model
Depth Image
3D Point Cloud
![Page 8: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/8.jpg)
8
Pave3DX + WiseCrax Processing Pipeline
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
Right Stereo
Image
Left Stereo
Image
3D
Reconstruction
Depth
Image
Colour
Image
Image
Processing
Computer
Vision
Machine
Learning
Crack Detection
Longitudinal Profile
Surface Texture
Transverse Profile
Rut Depth
Pothole Detection
…
Learning Feed
Forward
Learning Feed Back
3D
Model
3D
Point
Cloud
![Page 9: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/9.jpg)
9
WiseCrax Detection Pipeline
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
Noise removal (autoencoder) Image tiles
Supervised
Unsupervised
ML
model
Roadware Vision
DB
![Page 10: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/10.jpg)
10
LeNet-5 – A Classic CNN Architecture
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf
![Page 11: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/11.jpg)
11
Convolution Operation
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
Source: http://cs231n.github.io/convolutional-networks/
![Page 12: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/12.jpg)
12
Instance Segmentation
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 13: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/13.jpg)
13
U-Net: Convolutional Networks
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
https://arxiv.org/pdf/1505.04597.pdf
![Page 14: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/14.jpg)
14
Data Augmentation
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
• Generate batches of image data with real-time data augmentation
• The data will be looped over (in batches)
• 250,000 images as Data Augmentation
• Transformations applied:
o Rotation
o Flip
o Translation
o Gaussian Noise
o Scale
o Mirroring
![Page 15: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/15.jpg)
15
Measure similarity between two images:
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
Modified Pixel-wise-based Method
1. Introduce buffer regions by applying erosion on the original crack map
2. Convert the 'thick crack line' to 1-pixel-wide crack line using Skeletonization
Accuracy = Skeleton of TP / Skeleton of Union
![Page 16: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/16.jpg)
16
Detection Performance Definitions
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 17: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/17.jpg)
17
Crack Detection Metrics - Precision
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
Average : 0.909
![Page 18: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/18.jpg)
18
Crack Detection Metrics - Recall
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
Average : 0.999
![Page 19: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/19.jpg)
19
Crack Detection Metrics – F1 Score
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
Average: 0.948
![Page 20: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/20.jpg)
20
Example 1 – Real Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 21: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/21.jpg)
21
Example 1 – Depth Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 22: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/22.jpg)
22
Example 1 – Crack Detection
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 23: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/23.jpg)
23
Example 2 – Real Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 24: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/24.jpg)
24
Example 2 – Depth Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 25: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/25.jpg)
25
Example 2 – Crack Detection
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 26: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/26.jpg)
26
Example 3 – Real Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 27: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/27.jpg)
27
Example 3 – Depth Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 28: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/28.jpg)
28
Example 3 – Crack Detection
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 29: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/29.jpg)
29
Example 4 – Real Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 30: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/30.jpg)
30
Example 4 – Depth Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 31: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/31.jpg)
31
Example 4 – Crack Detection
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 32: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/32.jpg)
32
Example 5 – Real Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 33: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/33.jpg)
33
Example 5 – Depth Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 34: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/34.jpg)
34
Example 5 – Crack Detection
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 35: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/35.jpg)
35
Example 6 – Real Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 36: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/36.jpg)
36
Example 6 – Depth Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 37: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/37.jpg)
37
Example 6 – Crack Detection
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 38: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/38.jpg)
38
Example 7 – Real Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 39: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/39.jpg)
39
Example 7 – Depth Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 40: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/40.jpg)
40
Example 7 – Crack Detection
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 41: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/41.jpg)
41
Example 8 – Real Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 42: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/42.jpg)
42
Example 8 – Depth Image
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 43: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/43.jpg)
43
Example 8 – Crack Detection
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
![Page 44: RPUG2019 - Pavement Distress Detection Using Advanced ... › PDFs › PE-2019 › Connelly-Taylor.pdfPavement Distress Detection Using Advanced Machine Learning Methods with Intensity](https://reader033.vdocuments.net/reader033/viewer/2022060409/5f102f797e708231d447ddee/html5/thumbnails/44.jpg)
44
Conclusion
Pavement Distress Detection Using Advanced Machine Learning Methods with Intensity and Depth Data
Deep Learning - Well suited for pavement distress detection
WiseCrax with UNet – Very promising results
Precision = 90.9%
Recall = 99.9%
F1 Score = 94.8%
Continuing to learn rapidly
Large data sets improve results
Data Augmentation
Reduces burden of annotation
Increases speed of improvement