deep learning quality assurance - autonomous driving · ai failing marius reuther, dlqa @...

DEEP LEARNING QUALITY ASSURANCE

Marius Reuther, Founder DLQA

SOCIAL MEDIA AND AI

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 2

https://www.chefblogger.me/2016/02/18/sind-frauen-

wirklich-so-auf-social-media/

https://techcrunch.com/wp-

content/uploads/2016/03/tay.jpg?w=1390&crop=1

AI FAILING

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 3

https://techcrunch.com/wp-content/uploads/2016/03/tay.jpg?w=1390&crop=1

Tay.ai

• Microsoft‘s AI Chatbot

• Published in March 2016 on

Twitter

• Emits salacious and abusive

tweets

• Shutdown only 16 hours after the

releasehttps://techcrunch.com/wp-

content/uploads/2016/03/tay.jpg?w=1390&crop=1

Semantic Segmentation Road ModelObject Detection

CAMERA AI APPLICATIONS

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 4

https://www.youtube.com/watch?v=_kxX09i4fds https://www.youtube.com/watch?v=PNzQ4PNZSzc https://www.youtube.com/watch?v=ETT34fn7OfE

Collision PreventionObject Detection Semantic Segmentation

LIDAR AI APPLICATIONS

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 5

https://www.youtube.com/watch?v=KdrHLXpYYlg https://www.youtube.com/watch?v=7NNpvtdrHkU https://www.youtube.com/watch?v=nXlqv_k4P8Q

AI TrainingDataset

DATASET CREATION PROCESS

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 6

Oil Rig Refinery

Collect DataAI Developer

Data

Labeling

Datasets for Machine Learning in AD typically consist of 10k – 10 mio images.

Annotation may take 1h or more.

LABELING LANDSCAPE

Labeling Service Labeling ToolingTooling + Service

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 7

Computer Vision Annotation Tool

(CVAT)

LabelMe

LABELME – OPEN SOURCE LABELING TOOL

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 8

LabelMe

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 9

Car

car,

CAR,

Automobile,

Object,

Auto,

Vehicle,

Vehcl,

MotorizedVehicle,

motorized vehicle,

Motorized Vehicle,

…

LABELME – OPEN SOURCE LABELING TOOL

LabelMe

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 10

Car

car,

CAR,

Automobile,

Object,

Auto,

Vehicle,

Vehcl,

MotorizedVehicle,

motorized vehicle,

Motorized Vehicle,

…

https://www.pinterest.it/pin/423479171186745584/

LABELME – OPEN SOURCE LABELING TOOL

CVAT – OPEN SOURCE LABELING TOOLS

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 11

CVAT

Matlab Ground Truth Labeler Alp‘s Labeling Tool (ALT)

MORE OPEN SOURCE LABELING TOOLS

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12

Overview about labeling tools:

https://de.slideshare.net/yuhuang/annotation-102542284

OPEN SOURCE DATASETS

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 13

Overview about open-source datasets:

https://de.slideshare.net/yuhuang/public-dataset-for-autonomous-driving

TIME SPEND ON CREATING AI PROJECTS

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 14

https://www.scribd.com/document/381514649/TRAIN-AI-2018-Andrej-Karpathy-Tesla(

datasets

models and algorithms

GOOGLE PAPER ON MACHINE LEARNING & DATA

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 15

If data replaces code in ML

systems, […] then […] testing

of input data is critical to a

well-functioning system.

https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

FREQUENT QUALITY ISSUES

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 16

http://blogs.yis.ac.jp/20hepburng/2016/05/24/english-unit-3-reflection-perspective-and-prejudice-english/

FREQUENT QUALITY ISSUES

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 17

Note:

An unbiased and labeling-

independent QA workflow

helps to improve the overall

dataset quality while

keeping track on the

requirements, latest updates

and doubtful situations as

well as insufficiencies!

AI TrainingDataset

DATASET CREATION PROCESS (WITH QA)

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 18

Oil Rig Refinery

Collect Data

Data

Labeling

Quality

Assessment

Inspection

|✓

AI Developer

AI TrainingDataset

DATASET CREATION PROCESS (WITH QA)

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 19

Oil Rig Refinery

Collect Data

Data

Labeling

Quality

Assessment

Inspection

|✓

AI Developer

THANKS

12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 20

Marius Reuther, Founder DLQA

DEEP LEARNING QUALITY ASSURANCE

www.dlqa.de