combining ros and ai for fail-operational automated driving...© virtual vehicle comet k2-center...

© VIRTUAL VEHICLE 1

Combining ROS and AI for

fail-operational automated driving

Prof. Dr. Daniel WatzenigVirtual Vehicle Research Center, Graz, AustriaandInstitute of Automation and Control at Graz University of Technology

© VIRTUAL VEHICLE

COMET K2-Center

December 2017 Software Defined Vehicles 2

AUTOMOTIVE RAIL AEROSPACE

Shareholder:

� Gegründet: 2002

� Mitarbeiter: 204

� Umsatz: 20,3 Mio. EUR

� Standort: Graz

� Website: www.v2c2.at

Dr. Jost BernaschGeschäftsführer

Prof. Hermann SteffanWissenschaftlicher Leiter

© VIRTUAL VEHICLE

Our automated driving research activities

Embedded control and software functions

Real-time sensor fusion

Sensor self-diagnostics and fail-operational architectures

Dependable computing and reliable in vehicle control (strong multi-core expertise in both SW and HW)

Functional safety analyses (ISO PAS 21448)

Virtual prototyping of automated driving functions

Validation of real driving scenarios (real-time co-simulation)

Traffic simulation (micro, macro) / infrastructure integration

AI in both function development and vehicle operation


© VIRTUAL VEHICLE

Outline

� Motivating example

� The challenge “fail-operational”

� ROS and AI integration• Simulation, visualization, and communication

� AD demo car

� AI example for active safety design

� Summary


© VIRTUAL VEHICLE

Motivating example

Virtual Vehicle

� Research Center in Graz (about 200 employees)

� Expertise in simulation and experimental verification• Focus on Automated Driving

Dependable Systems Group

� Functional safety

� Verification and validation


© VIRTUAL VEHICLE

Motivating example

Motivating example – Uber car

� Safety is top priority for user acceptance of automated driving

http://orf.at/stories/2384924/2384925/


© VIRTUAL VEHICLE

Motivating example

Motivating example – consequences

� Achieve user acceptance• Avoid accidents and hazards (see example)

� Keep development and verification efforts reasonable• Easy to use simulation environment during development

• Verify software on different test platforms

− MiL, SIL, HiL, and vehicle

� Simple goals, hard to achieve…


© VIRTUAL VEHICLE

Technical challenges

Reference architecture

Sensoryprocessing

Environmentmodel

Behaviourgeneration

Valuejudgement

Sensors Actuators

Perceived objects and events

Situationevaluation

Predicted Input

Plan

State

Plan evaluation

Sensorinputs

Planresults

Update

Actions

ANSI Reference architecture of intelligent systems


Information and data uncertainties!

© VIRTUAL VEHICLE

Technical challenges

Vehicle technologies

� Sensory processing• Camera, RADAR, LiDAR, …

� Value judgement and environment model• High performance computing (HPC)

• Segmentation (using AI)

• Situation identification

• Behaviour estimation of other traffic participants

� Behaviour generation• Electronic Control Units (ECU)

• Vehicle control algorithms

Camera

LiDAR

RADAR

ECUHPC

Steering

Accelaration/Brake


© VIRTUAL VEHICLE

Limits of sensors


As effective sensors are, they have some drawbacks• Limited range• Performance is susceptible to common environmental conditions (rain, fog,

varying lighting conditions) • “False positives”• Range determination not as accurate as required• The use of several sensor types can ensure a higher level of confidence in

target detection and characterization

� Robust sensors and sensor self-diagnosis� Redundancy in HW and SW (“fail-operational”)� Sensor fusion (raw data? objects? hybrid?)

© VIRTUAL VEHICLE

• Use of “heat maps” (buffer where each pixel of successfully detected window adds “1”)

• Average the buffer from the last 10-20 frames

• Use only pixels that survived a certain amount of frames

• Extract component bounding blocks � vehicle(s)

Vehicle detection: false positives


© VIRTUAL VEHICLE

Enhancement of view


Austrian test region ALP.Lab

© VIRTUAL VEHICLE

Enhancement of view


© VIRTUAL VEHICLE

System architecture – automated driving

December 2017 14Software Defined Vehicles

plan safeactions under

currentconditions

steer functionsand vehicle

unpredictableconditions

fuse all input to

one model

Reliable sensor data processing and fusion• Raw data analysis of all sensors• Deliver consistent environmental model

Scene understanding, driver monitoring, decision making, and planning• Situation and behaviour prediction• Planning of provably safe trajectories• Handover/takeover planning

Fail-operational X-by-wire actuation (low level control)

System Performance and Driver Monitoring

• Out-of-position• Warning• Intervention• Measure reliability

and uncertainty• Detection and

decision on functionavailability

• System degradation• Sensor self-diagnosis• Ensuring fail-

operational behaviour• …

Radar LIDAR Cameras Infrared Ultrasonic V2XOther

Sensors

[Source: based on ECSEL Project RobustSense, 2015]

© VIRTUAL VEHICLE

Towards fail-operational: e.g. power steering


• Fail-safe (what we have now)• No emergency operation necessary

• Safe state: system off, driver immediately in control loop

• High-availability

• Safe state: system off, driver immediately in control loop

• Emergency operation is desirable but not required

• Minimize hazardous situation in case of potential misuse

• Fail-operational

• Emergency operation is required (10-15s)

• Eyes-off, brain-off

• Achieved by adding measures to all vital parts

[Source: Steindl, Miedl, Safetronic, 2015]

© VIRTUAL VEHICLE

Homogeneous vs. diversity concept


• Homogeneous redundancy

• uses a minimum of two equal instances in parallel

• the effort in development can be reduced due to the identical components

• because of the equality, this approach only protects against random faults caused by aging, deterioration or bit flips

• probability for a complete system crash is higher than in approaches with diverse components

• Redundancy by diversity (avionics)

• The calculating components are heterogeneous, e.g. from different manufacturers

• System SW of each unit is different or uses at least different HW components

• this system SW diversity complements functional diversity of the application SW

• Different implementations result in a lower probability of failure for the system due to the lower probability that the diverse components show the same misbehavior at the same time

© VIRTUAL VEHICLE

Fail-operational architecture

• EC Project “PRYSTINE” (2018 to 2021, Programmable systems for intelligence in automobiles)

• Infineon, Scania, Ford, BMW, Virtual Vehicle, TU Graz…


© VIRTUAL VEHICLE

Fail-operational architecture

• Massive redundancy will not be the solution (2x and 3x, STANAG 4626)

• We need “smart” solutions!

• Mechanism for HW fault detection (e.g. BIST, built-in self tests)

• HW extensions for predictive diagnosis

• Reconfiguration strategies (isolation of faulty function and “shift of functions”)

• Degradation strategies (critical vs. non-critical functions)

• …


© VIRTUAL VEHICLE

PRYSTINE


© VIRTUAL VEHICLE

ROS Introduction


© VIRTUAL VEHICLE

Robot Operating System (ROS)

� Open-source, meta-operating system for robots• Originally developed by Stanford AI Lab and Willow Garage in 2007

• Maintained by the Open Source Robotics Foundation (OSRF)

• Runs on top of e.g. Ubuntu/Linux

� Designed for many kinds of robots• Provides tools for building/running code

− Including software libraries

• Hardware abstraction

• Allows for low-level device control

• Provides communication system

ROS Introduction


© VIRTUAL VEHICLE

ROS Introduction

ROS Communication

� Peer-to-peer communication

� Central “service” registration

� Supports UDP and TCP

Advantages

� Flexible configuration

� Fast communication

� Simple distribution of functions on computationplatforms


© VIRTUAL VEHICLE

ROS Introduction

Integration of common solutions

� Sensor data processing

− Library for 2D/3D image and point cloud processing

− Filtering, feature detection, …

− Tracking on camera data

� Motion planning• Support of multiple planning algorithms

� 3D simulation and visualization• Dynamics, kinematics, sensors, …

� Data transformation• Coordination system transformations for sensor data


© VIRTUAL VEHICLE

ROS Introduction

ROS Simulation

� 3D environment

� Sensor simulation• Laser scanner built in

• Camera available

• Extension possible

� Vehicle simulation• Position

• Orientation

• Speed

• Steering

• …


Sensorsimulation

© VIRTUAL VEHICLE

ROS Introduction

Driving Simulation

� Detailed street layouts• Number of lanes

• Street markings

� Traffic management• Traffic signs and signals

� Different weather conditions• Sunshine, snow, rain, …

� Sensor simulation• Multiple sensor types available

• Extension possible


https://vires.com/vtd-vires-virtual-test-drive/

Kom

ple

xe S

traß

enve

rläufe

Änderu

ngen im

Str

aß

enve

rlauf

Ändern

de U

mw

eltb

edin

gungen

Unte

rsch

iedlic

he

Str

aß

enbesc

haff

enheit

© VIRTUAL VEHICLE

ROS – Function Implementation

AI Integration

Simulation

� Exchangeability of simulation and real environment• Replacement of sensor data format

� Software deployment can be freely chosen


DrivingSimulation

Real worlddata

Sensor dataprocessing

Behaviourgeneration

Segmentation(AI function)

Decisionmaking

ROS –Simulation

ROS –Converter

ROSmessages

HPC #1 HPC #2 ECU

Sensordata

available available

© VIRTUAL VEHICLE

AI inference

Standard Software

AI Integration


AI function control flow

- Read image

- Copy to GPU memory

- Matrix operations(multiply and add)

- Mass non-linear function(ReLU, sigmoid, …)

call

return result

CPU GPU

- Read result

- Send message

ROS specific part can be implemented “as usual”

© VIRTUAL VEHICLE

Our automated drive test vehicle

ROS Deployment

� Use ROS function implementation in vehicle• No change of SW needed

� Integrated sensors• Six radar sensors

− Four long range, two short range

• 6 Cameras

− Front, two side, back

• One Mobile Eye

� Full vehicle control• Steering, acceleration, brake, …


Virtual Vehicle – Automated Drive test vehicle

Virtual Vehicle – AD test vehicle trunk

© VIRTUAL VEHICLE

Motivation: Development process active safety systems

December 2017 30

Problem: Complexity and high variation of accidents

� Safety functions for every combination of accident type and cause

� Function design based an quantitative (e.g. DESTATIS) and qualitative accident databases (e.g. GIDAS Pre-Crash-Matrix)

Software Defined Vehicles

First 50%: 26 types and causes of accidents

Last 50%: 5287 types and causes of accidents

An exponential effort in the development of individual active safety systems

is required to cover only significant increase in the addressing of accidents!

Type:

Cause: speeding, slippery road, etc

© VIRTUAL VEHICLE

Function development based on real world data

December 2017 31

Method analysis: From the accident to the active safety system

� Accident recording by highly automated vehicles (SAE Level 3)

Data recording: Course of the ego vehicle and other traffic participantsroad geometry, driver behavior

1) Evaluation of active safety systems with recorded traffic data

2) Learning the function behavior of active safety systems directly from recorded data

Software Defined Vehicles

[Here] [Kostal][BMW]

Backend

� 360° environment recognition� High-precision digital maps incl.

localization� Driver monitoring� Backend communication

[Source: BMW and Virtual Vehicle, cooperative R&D project, 2017]

© VIRTUAL VEHICLE

Vision: Optimization based on real world traffic data

December 2017 32Software Defined Vehicles

� 360°sensors� High-precision maps � Driver monitoring� Backend

Crossing scenarios Total scenarios Accidents

Training data 1840 242

Test data 1829 243

Project use case: Crossing pedestrian (75% of all pedestrian accidents)

� Generated pedestrian scenarios from the effectiveness analysis

© VIRTUAL VEHICLE

Simulation results: False positives vs. speed reduction


Neural Network

0 5 10 15 20 25 30 35 400

10

20

30

40

50

60

70

80

90

100

False positives

Sp

ee

d r

ed

uct

ion

[%]

Reference algorithm

NN

NN (Extended Features)

NN (Reduced Training Data)

NN (Extended Features / Reduced Training Data)

0 5 10 15 20 25 30 35 400

10

20

30

40

50

60

70

80

90

100

False positives

Sp

ee

d r

ed

uct

ion

[%]

Reference algorithm

RF

RF (Extended Features)

RF (Reduced Training Data)

RF (Extended Features / Reduced Training Data)

Random Forest

Variation by the algorithm evaluation:1) Feature set (feature variation)

2) Training data (reduced speed range pedestrian)

Speed reduction

Reference Implementation 73.7%

Random Forest 83.4%

Neural Network 92.0%

© VIRTUAL VEHICLE

AI-based systems – challenges to be faced


• Requirements engineering (Machine Learning algorithms)

• Identification of Key Performance Indicators for ML algorithms• e.g. accuracy (fault tolerances but also mis-detection), speed, etc.

• HW requirements for different DNN structures

• Identification of relevant safety analyses according to ISO 26262 and ISO PAS 21448• Determination of safety measures, time tolerances for detection etc.

• How to measure code and structure coverage of DNNs?

• Selection of training and test data

• Work out criteria based on statistical methods, quality, and versatility

• SW unit tests and integration test

• Verification and validation of SW safety requirements

© VIRTUAL VEHICLE

Summary

� Automated driving requires redundancy (SW and HW)

• Fail-operational architectures � ISO PAS 21448

• Minimum redundancy but maximum reliability

• Homogeneous redundancy vs. redundancy by diversity � trade-off

� AI functions have already proven their usefulness• Object detection and localization (segmentation)

• End-to-End driving vs. modular approach

• AI in function development

� ROS is already widely used

• Sound basis for rapid prototyping

• Seamless transition from simulation to real hardware

• ROS is a suitable development and test environment for AI functions


© VIRTUAL VEHICLEDecember 2017 Software Defined Vehicles 37

Prof. Dr. Daniel WatzenigEmail: [email protected]

Thank you for your attention.

combining ros and ai for fail-operational automated driving...© virtual vehicle comet k2-center...

Documents