Philipp Slusallek

German Research Center for Artificial Intelligence (DFKI)

Research Area: Agents and Simulated Reality

Saarland University

Excellence Cluster Multimodal Computing and Interaction (MMCI)

Understanding the World with AI:Training & Validating Autonomous Vehicles with Synthetic Data

Today’s Talk

• Overview of DFKI

• What is AI?

• Learning through Synthetic Data from Simulations

• Genesis: Open Platform for Collaboration on AI

• “CERN for AI”

• Conclusions

Saarland Informatics Campus

German Research Center for

Artificial Intelligence (DFKI)

• Motto

„Computer with Eyes, Ears, and Common Sense“

• Overview

Largest AI research center worldwide (founded in 1988)

Germany’s leading research center for innovative SW technologies

5 sites in Germany

• Saarbrücken, Bremen, Kaiserslautern; Berlin, Osnabrück

18 research areas, 10 competence centers, 7 living labs

More than 510 employees (>900 with research assistants)

Budget of more than 44 M€ (2017)

More than 80 spin-offs

DFKI is a Joint Venture of:

Rhineland-Palatinate

SaarlandSaarland

Bremen

Deutschland GmbH

‘Blue Sky‘

Basic Research

Commercialization/

Exploitation

DFKI Covers the Complete Innovation Cycle

Labs at the

University

Application-

oriented

Basic

Research

Applied

Research

and

Development

Transfer

Projects

DFKI projects

for

external

clients and

shareholders

Spin-off

Companies

with DFKI

equity

DFKI

projects

for

federal

government,

EU

DFKI

projects

for

state

governments,

clients and

shareholders

External

Clients

Shareholders

Agents & Simulated Reality:AI & Graphics & HPC & Security

Research:

Topics &

Teams

Philipp Slusallek

Linked Data

Representations

René Schubotz

Large-Scale Virtual

Environments

Philipp Slusallek

Multimodal

Computing

and

Interaction

High-Performance

Graphics & Computing

Richard Membarth

Distributed Realistic

Graphics

Philipp Slusallek

Computational

3D Imaging

Tim Dahmen

Knowledge- and Technology Transfer

VisCenter

Georg Demme

Strategic

Relations

Hilko Hoffmann

SW-Engineering &

Organization

Georg Demme

Scientific Director

Survivable Systems

and Services

Philipp Slusallek

Intelligent

Information Systems

Matthias Klusch

Multi-agent

Systems

Klaus Fischer

Behavior, Interaction

& Visualization

Georg Demme

Visual

Computing

Philipp Slusallek

Application

DomainsAutonomous

Driving

Christian Müller

Industrie 4.0

Ingo Zinnikus

Smart

Environments

Hilko Hoffmann

Autonomous

Driving

Christian Müller

Smart System

Security

Andreas Nonnengart

High-Performance

Computing

Richard Membarth

Computational

Sciences

Tim Dahmen

DFKI Agenten und Simulierte Realität 8

Intelligent Human Simulation,

e.g. in Production Environments

Collaborative Robotics and

Simulated Reality

Autonomous Driving: Training

using Synthetic Sensor Data

What is AI?

What is AI?

What is AI?

• Unique set of capabilities that facilitate: Perception

• … to find out what the world around “looks” like

Machine Learning (Deep Learning, etc.)• … to automatically build/adapt models of reality

Reasoning• ... to infer and generalize new models based on existing knowledge

Knowledge Representation, Communication, and Interaction• … to represent, process, and communicate models of the environment

Planning• … to plan and simulate in the virtual world, exploring possible options, evaluating

their performance, and making decisions

Acting• … to manipulate and move around in the real/virtual world

Supporting Technology• Validation/Verification, Security, Ethics/Social Aspects, SW/HW-Platform

What is AI?

• AI Research: Three distinct application areas

AI as a Tool• Enabling supportive and adaptive technology

• Systems aware of their environment and (re-)acting accordingly

AI for Cognitive Sciences• Understanding the human brain and cognition

• Helps building tools that understand humans

AI for Artificial Humans• A non-goal for our research

• Cause of fear in the general public (sci-fi movies, etc.)

What is AI?

• AI Technologies (roughly) Logical Reasoning (rule-based systems)

• Various forms of logic/algorithms (proof theory)

• First-Order, Higher-Order, Fuzzy, Modal, …

Statistical Reasoning• Deriving properties via probability densities from data

• Mainly Bayesian Reasoning

Machine Learning (data-driven approach)• Learning and predictions based on data

• Decision trees, Clustering, SVMs, Reinforcement-Learning, …

Deep-Learning (recursive pattern recognition)• Layers of weighted, non-linear “neurons”

• “Programming” via network structure and providing examples

• Learning via “Back-Propagation” of errors through the network

What is Deep Learning?

• From Neural Nets to Deep Learning

Neurons• Brain is a vast network of connected

neurons each with simple characteristics– Human: ~86 GNeurons, <10k synapses each

Early (Artificial) Neural Networks (NN)• A few layers of neurons (e.g. perceptron)

• Back-propagation for learning weights (𝑤𝑖)

Deep Neural Networks (DNN)• Often hundred layers and more

• Different structures (CNN, RNN, LSTM, …)

What is Deep Learning?

• What DNN Actually Learn

© Honglak Lee at al.

Lowest Level

Medium Level

High Level

State of AI

• Amazing progress in recent years

Most visible due to Deep Neural Networks (DNNs)

Supported by • More data

• Better HW

• Improved algorithms

• Success stories

Perception: vision, speech, …

Game playing: Chess, Go, simple video games, …

Some complex tasks: translation, autonomous driving, …

• Still many scientific and practical challenges

Key AI Challenges

• Validation and verification Lack of formal methods for DNN only leaves

statistical validation via systematic testing (e.g. this talk)

• Transparency and Explanation Ability to explain behavior of AI systems to humans

• Modularity From end-to-end learning to composable building blocks

• Integration with classical AI techniques Rich ecosystem of existing techniques, often with formal methods

• Limits of the technology Understand the landscape of limits from physics, computing,

economics, and theory

Using Synthetic Data to Train and

Validate Autonomous Systems

Why Do We Need Validation

via Synthetic Data?

Autonomous Driving:

The Problem

• Our World can be extremely complex Geometry, Appearance, Motion, Weather, Environment, …

• Systems must make accurate and reliable decisions Especially in Critical Situations

Increasingly making use of (deep) machine learning

• Learning of critical situations is essentially impossible Too little data even for “normal” situations

Critical situations rarely happen in reality – per definition!

Extremely high-dimensional models

Goal: Scalable Learning from synthetic input data Training, benchmarking & validation (“Virtual Crash-Test“)

Autonomous Driving:

The Problem

Learning from

real data

Typical situations

with high probability

Critical situations

with low probability

Learning from

synthetic data

Learning forLong-Tail Distributions

Goal: Validation of correct behavior by

covering high variability of input data

Example: Learning to Play Go

via Simulations

• Paper from Google DeepMind [Nature, Oct´17]

Given: Rules + Deep-Learning + Reinforcement-Learning

Learning by simulation (here: playing against itself)

• But: In reality „rules“ are complex AND unknown!

Simultaneously learn rules, simulate, and train AI system

Quality of Gameplay Prediction of „human moves“

Reality

Car

Reality

• Training and Validation in Reality (e.g. Driving)

Difficult, costly, and non-scalable

Reality

Digital

Reality

CarCar

Digital Reality

Validation

• Training and Validation in the Digital Reality

Arbitrarily scalable (given the right platform)

But: Where to get the models and the training data from?

SzenarienSzenarien

Teil-ModelleTeil-ModelleTeil-Modelle

Reality

Digital

Reality

Partial

Models

(Rules)

Simulation/

Rendering

Scenarios

Concrete

Instances of

Scenarios

Configuration &

Learning

Modeling &

Learning

Geometry

Material

Behavior

Motion

Environment

…

e.g. kid in front of the car e.g. kid in front of the car – in this way

Synthetic Sensor Data,

Labels, …

Traffic

Situation

Adaptation to the

Simulated Environment

(e.g. used sensors)

Coverage of Variability via

Parameterized Instantiation &

Genetic Programming

CarCar

Digital Reality: Training

Validation

SzenarienSzenarien

Teil-ModelleTeil-ModelleTeil-Modelle

Reality

Digital

Reality

Partial

Models

(Rules)

Simulation/

Rendering

Scenarios

Concrete

Instances of

Scenarios

Configuration &

Learning

Modeling &

Learning

Geometry

Material

Behavior

Motion

Environment

…

e.g. kid in front of the car e.g. kid in front of the car – in this way

Synthetic Sensor Data,

Labels, …

Traffic

Situation

Adaptation to the

Simulated Environment

(e.g. used sensors)

Coverage of Variability via

Parameterized Instantiation &

Genetic Programming

Car

Continuous

Validation &

Adaptation

Car

Digital Reality: Training Loop

AI für Autonomous Driving

• Requirements and Challenges Learning of critical situations

• Creating proper models of the real world (via learning)

• Generating the required input data (via models)

• Covering the required variability of the input data

• Requirements on the size, accuracy and robustness of the data?

Benchmarking the development processes • Reproducible and standardized test scenarios

• Scalable and fast simulation, rendering, and learning

• Open architecture for integrating different models & simulations

Validating the learned behavior for autonomous driving• Calibrating synthetic data against real data

• Identifying and adapting insufficient and missing models

• Setting up a ”Virtual TÜV” for autonomous vehicles (systems?)

Initial Scenario: Autonomous

Breaking System (Euro NCAP)

• Autonomous breaking test conducted by ADAC

Significant efforts invested in tests in 2016

Images from: https://www.adac.de/infotestrat/tests/crash-test/notbremsassistent_2016/default.aspx?ComponentId=250194&SourcePageId=31956

Initial Scenario: Autonomous

Breaking System (Euro NCAP)

• Insufficient Results in Recent Test by ADAC (2016)

Problems: Night, speed, moving legs, only forward looking, …

But even difficult cases were handled by at least some cars

• Important for Pedestrian Protection (30% of deaths)

Potentially strong impact with relatively simple setup

Easier tests, wider coverage, less false positives, …

[Nils Tiemann, Ein Beitrag zur Situationsanalyse im vorausschauenden Fußgängerschutz, Diss, Uni Duisburg-Essen, 2012]

Challenges: e.g. Recognition

of Subtle Motions

• Long history in motion research (>40 years)

E.g. Gunnar Johansson's Point Light Walkers (1974, below)

Significant interdisciplinary research (e.g. psychology)

• Humans can easily discriminate different styles

E.g. gender, age, weight, mood, ...

Based on minimal information

• Can we teach machines the same?

Detect if pedestrian will cross the street

Parameterized motion model & style transfer

Predictive models & physical limits

Genesis: Open Platform for Learning,

Simulation, Training, and Validation of

Autonomous Systems

Digital Reality for

Autonomous Driving (DFKI)

Intelligent Simulated Reality for

Autonomous Systems (BMW)

Using PreScan Data (TASS)

for Semantic Segmentation

OpenDS

• Highly visible automotive DFKI platform

Open Source driving simulation platform

Validated tasks for psychological driver distraction research

Internationally widely used in research and industry• More than 10k registered users

• Including: Google, Bosch, Continental, Honda, TomTom, Nuance, …

• Including: CMU, Stanford, Berkeley, MIT, TUM, TU-Berlin, …

Also used for driver education (e.g. developing countries)

USP: Highly flexible infrastructure for research

• Key ingredient of Genesis platform

REACT Project

• BMBF-funded Project at DFKI

Autonomous Driving: Modeling, Learning & Simulation Environment for Pedestrian Behavior in Critical Situations

3 year project with 6 researchers

• Exploring Key Challenges

Motion Models and Motion Synthesis for Pedestrians

Modeling and Simulating High-Level Behavior

Hybrid Deep Learning for Agent-Based Simulations

Automated Creation & Evaluation of Critical Situations

High-Performance Deep Learning (w/ AnyDSL)

DFKI Competence Center

Autonomous Driving (CCAD)

• Thematic Research Center at DFKI

Cutting across research of several research areas at DFKI

Exploiting DFKI-internal synergies

Increased visibility in research and industry

One-Stop-Shop for external contacts & collaborations

Common platform and infrastructure projects

Lead by Christian Müller (DFKI Fellow)

• Contributing various DFKI technologies for Genesis

Projects in DL and Different

Application Areas (only ASR)

• Deep Learning High-Performance Deep-Learning on heterogeneous HW, new ExCluster project Large-Scale Deep-Learning Framework (HP-DLF), with FhG-ITWM, TU Dresden Deep-Learning Training Center, at UdS and DFKI Deep-Learning Competence Center, at UdS and DFKI (submitted)

• Computational Sciences TransRegio SFB, TR244 (excellent review), jointly with UdS, TU Ilmenau, TU Karlsruhe (KIT) SFB „3D Microstructure Evolution“ (first review), jointly with MS, CS, and Math at UdS Learning Sensor Systems (BMBF), jointly with FhG, U. Würzburg, U. Bamberg Understanding SKA data based on models (BMBF proposal, Max-Planck for Radio-Astronomy) Strong focus on AI now at sciences, also at associations like DPG and DGM

• Autonomous Driving Motion Synthesis for Learning Pedestrian Behavior (BMW/VW, Intel) Dreams4Cars, EU project, together with DFKI-IUI and -RIC VDA Leitinitiative „Autonomous Driving“ (in all 3 planned projects: AI, V+V, SetLevel4to5)

• Human Understanding Excellence Cluster proposal „Digital Reality“, SL Informatics Campus (final proposal) EU Flagship proposal “Humane AI” (10 year, 100 M€/year, submitted 1st phase) EU “AI on Demand Platform” (20 M€ for AI community, due mid of April)

Future of AI

• AI as a fundamental game changer in many areas

Cars, robots, virt. assistants, urban planning, energy mgmt., …

Nat. sciences, data analytics, social, finance, …

Needs scientists and experts from many different fields

• Common approach and challenges

Simultaneously learn models & good actions (given models)

Requires large-scale learning, simulation, collaboration

• Need for a joint research platform & community

We have done this before: Human Genome project & CERN

Can we learn from these approaches?

EU Flagship Proposal:

Humane AI

„CERN for AI“

• Mission:

Bring together the fragmented AI landscape in Europe

Collaborative, scientific effort to accelerate and consolidate the development and uptake of AI for the benefit all humans and our environment

Continuously improve our understanding of the world around us and use this information to explore and evaluate better ways to act and interact in this world

Provide a transparent, open, and flexible platform supporting a wide range of research capabilities while facilitating transfer and exchange with industry

Broadly discuss and implement values and ethics for using AI

Humane AI Flagship

• Very large EU project

Graphene, Human Brain, Quantum Technologies (starting)

10 years, 100 M€/a (50% EU funding)

• New Flagship call in 2018

1. Phase (DL: Mid February)• „Humane AI“ proposal submitted by small AI consortium, sent copy

2. Phase (DL: Mid September)• Planning already started

3. Preparatory phase (1 year, essentially during 2019)

Final proposal is result of Preparatory Phase

Humane AI Flagship

Toward AI Systems That Augment and Empower Humans by Understanding Us, our Society and the World Around Us

• Develop world-leading insights and AI technologies From fundamental algorithms to disruptive AI applications

and platforms

Making technology easily and widely available

• Technologies and tools for enhancing human cognitive capabilities Channeling human creativity, inventiveness and intuition

Empower humans to make important decisions in a more informed way

Helping us understand and foresee long-term implications

Humane AI Flagship

• Technologies that allow seamless, productive interaction among humans, intelligent machines and complex, dynamic, real-world environments Develop AI systems that can intelligently interact within complex

settings Adapt to changing, open-world environments

• Explainable, transparent, validated and thus trustworthy AI systems Dealing with the increasing complexity of a networked globalized

world Overcome unpredictable “black box” systems

• Adding values, ethics, privacy, and security as core design considerations AI systems that make their values accessible and understandable

to humans

Conclusions

• AI will be a Central Component of Future Systems

• Validation of AI Systems is Essential to Keep Us Safe

Only scalable method: Systematic testing with synthetic data

• Digital Reality: Deriving provably adequate behavior

Need to learn good models and simulation of the real world

Loop of model learning, simulation, training, and validation

• Big Challenges Ahead

Many promising partial results – but largely islands so far

Requires open collaboration of industry & academia

Needs large-scale European and national initiatives