humandrive: the most complex autonomously controlled
TRANSCRIPT
#CAM2019 @LCV_Event
HumanDrive: The Most Complex
Autonomously Controlled Journey in the UKNick Blake
Chief Innovation Strategist and Head of Big Data Labs – Hitachi
Research & Development Europe
CAM Seminar Hall Sponsor
© Hitachi Europe, Ltd. 2019. All rights reserved.
Data-Driven Perception and Planning
Methodologies for Autonomous Vehicles
04/09/2019
Nick Blake, Ph.D
Head of European Big Data Lab
European Big Data Lab
London, United Kingdom
Syed Adnan Yusuf, Ph.D
Senior Research Scientist
European Big Data Lab
London, United Kingdom
© Hitachi Europe, Ltd. 2019. All rights reserved.
Presentation Contents
3
1. Hitachi Autonomous Vehicle European R&D Activities
2. HumanDrive Project
3. Autonomous Driving Software Paradigms
4. HumanDrive Software Paradigm – Hitachi Approach
5. Key Lessons
© Hitachi Europe, Ltd. 2019. All rights reserved. 4
1. Hitachi Autonomous Vehicle European R&D Activities
© Hitachi Europe, Ltd. 2019. All rights reserved.
Hitachi Autonomous Vehicle European R&D Activities C
olla
bo
rativ
e re
se
arc
hK
ey techno
logie
s
Connected Autonomous Mobility & Advanced
Telematics Solutions (ATS)
Build
ing b
locks
Vehicle Dynamics
Cognition and Judgement
Communication & Cooperativeness
Adv. Telematics
Autonomous Vehicle Control & Monitoring
Sensor Fusion, Deep Learning, AI
Connected Car,Cooperative AD Data Analysis, Machine
Learning, AI, Pred. Maintenance, Fleet Opt.
Experimental
Analysis
Mathematics and
Data Analysis
Simulation
Machine Learning
and AI
European Project
activitiesDriver Characterisation
TCU
Standardization
TCU: Telematics Control Unit
2 test/demo vehicles
Sensor Fusion
© Hitachi Europe, Ltd. 2019. All rights reserved. 6
2. HumanDrive Project
▪ Scope
▪ Consortium
▪ Video
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Project – Scope
7
2017 2018 2019
Project start
Dynamic Trials
Static Env Trials
Grand Drive
Projectcomplete
OVERTAKING OF CYCLISTS
ROUNDABOUT
NATURAL ROAD POSITION
Technical challenges for human like driving using machine learning
The
BIG
Ambition
200
Miles
100%
AD
Collaboration between global and UK CAV experts in industry, academia and Government agencies.
Establish autonomous vehicle R&D team in Nissan UK.
Hitachi create CAV team in UK, with unique AI capability.
World Class cyber security capability developed between Atkins and SBD.
Using test UK test facilities at Horiba MIRA and Cranfield Univ
Benefit and Impact
SMOOTH AUTONOMOUS VEHICLE CONTROL
https://humandrive.co.uk/
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Project – Consortium
8
Our consortium is made up of 10 members, all of whom have specific responsibilities
and areas of expertise:
Sponsored by the Innovate UK (the UK’s innovation agency) and CCAV office:
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Project – Video
9
© Hitachi Europe, Ltd. 2019. All rights reserved. 10
3. Autonomous Driving Software Paradigms
© Hitachi Europe, Ltd. 2019. All rights reserved.
Autonomous Driving Software Paradigms
11
Mediated Perception Behaviour Reflex
• Software Reconfigurability and Debuggability
• Component Level Safety Assurance
• High Software Complexity
• Fixed Driving Behaviour
• Low Software Complexity
• Customisable Driving Behaviour
• Safety Assurance
• Debuggability
Perception Control
PlanningLocalization AI Agent
• Construct a direct mapping from sensory inputs
e.g. cameras, lidars etc. to a driving action e.g.
accel/brake or steer.
• Problem is decomposed into separate sub-
modules to solve the perception, localisation,
planning and control tasks.
Source: http://deepdriving.cs.princeton.edu/paper.pdf
© Hitachi Europe, Ltd. 2019. All rights reserved. 12
4. HumanDrive Software Paradigm – Hitachi Approach
▪ Perception System
▪ Planning System
▪ Integration & Testing
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Software Paradigm – Hitachi Approach (1)
13
Machine Learning to Develop
Natural Human-Like Control
Software Safety Assurance
Software Reconfigurability,
Debuggability, Maintainability
Cross-Platform Software
Integration
Challenges to Solve
1
2
3
4
Proposed Solution
• Combine the benefits of Mediated and
Reflex paradigms to enable natural
human-like control whilst providing the
required safety assurance.
• Perception and Planning are the two
key sub-components responsible for
“Cognition” and “Decision-Making”.
AI Perception Control
AI PlanningLocalization
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Software Paradigm – Hitachi Approach (2)
14
Safety
Control
Software Diagnostics
➢ Numerical/Computational Issues ➢ Process Performance ➢ HW Health Status ➢ Log Info
Planning
Perception
Camera Based
Lidar Based
Camera + Lidar Based
➢ Occupied Area
➢ Detect/Track/Predict
➢ Detect/Track/Predict
➢ Drivable RoadLid
ars
+ C
am
era
sG
PS
+ O
do
m
Localization
➢ Manipulate actuators
to follow trajectory
➢ Localise on the map
➢ Determine desired
(coarse) route
Ego-carDesired route
Human-Like path
➢ Check if AI-based
path is within a set
of predefined
boundaries. If not,
then raise a flag to
indicate unsafe
operation.
Ego-car
Drivable-road
Desired route
Human-Like paths
Min/Max Limiter Rate Limiter
➢ Data-driven imitation
of human driving
behaviour
➢ Recurrent
Convolutional Neural
Network(s)
➢ Training using on pre-
recorder/ selected
human driving data
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Software Paradigm – Perception System (1)
15
System Requirements
Occupancy Grid Drivable AreaDetection, Tracking,
Prediction
Real-Time
Execution
Top-Level Perception System Architecture
Sensing Area
Camera
LiDAR
Machine/Deep Learning Layer Engineering Layer Output Layer
Camera-Based
• CNN for
object
detection and
classification
Lidar-Based
• CNN for
object
detection and
classification
Lidar/Camera-
Based
• FCN for
semantic
segmentation
Fusion
Filtering/Tracking
PredictionPast states
Predicted states
PredictedMeasured
Optimal
• Reduce false positives
• Increase robustness
• Future predictions
• Data visualisation
Planning…
GridVector
• Data representation
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Software Paradigm – Perception System (2)
16
Open source Hitachi labelling tool available on GitHub:
https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor
Iterative Performance Improvement
Machine Learning Factory Off-Line & On-Line System Performance Evaluation
Data Engineering*
Training & Validation Pipeline
Occupancy Grid
Lidar Based Detection
Camera Based Detection
Camera View
BEV View
Truck 2
Car 2
Car 1Truck 1
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Software Paradigm – Planning System (1)
17
System Requirements
Human-Like Data-Driven Safety AssuranceReal-Time
Execution
Top-Level Planning System Architecture*
*Patent Pending
Planning Network (PlanNet)
Trajectory Generator
Occupancy Grid
Desired Route
Input Layer
Perception Sequence
t
t-1
t-n⋱
𝑥𝑦
𝜃
ሶ𝑥
ሶ𝜃
𝑡 + Δ𝑡 =
𝑥𝑦𝜃ሶ𝑥ሶ𝜃
(𝑡) +
ሶ𝑥
ሶ𝜃sin ሶ𝜃Δ𝑡 + 𝜃 −
ሶ𝑥
ሶ𝜃sin 𝜃
−ሶ𝑥
ሶ𝜃cos ሶ𝜃Δ𝑡 + 𝜃 +
ሶ𝑥
ሶ𝜃cos 𝜃
ሶ𝜃Δ𝑡00
Trajectory Generator
Planning Network (PlanNet)
RCNN LSTM
CO
NC
AT
INA
TE
FC
LSTM
FC
Vehicle Centric
Yaw Rate
Sequence
Speed
Sequence
Example
Network
Architecture
• Generate trajectory based on future desired
yaw rates and speeds
• Constant Turn Rate Vehicle (CTRV) model
can be used
Output Trajectory
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Software Paradigm – Planning System (2)
18
Planning Network
Machine Learning Factory
Key Enabler: Driving Behaviour Analysis Toolchain
Imitation Learning: Important Consideration
Bias Variance Non-Smooth Driving
© Hitachi Europe, Ltd. 2019. All rights reserved.
HumanDrive Software Paradigm – Integration and Testing
19
• Containerised (Docker) development
environment allows seamless integration
on multiple vehicles
• Robot Operating System (ROS) is used
as the main meta-operating system
System Integration System Testing
© Hitachi Europe, Ltd. 2019. All rights reserved. 20
5. Key Lessons
© Hitachi Europe, Ltd. 2019. All rights reserved.
Key Lessons
21
Data quality and quantity is key enabler for creating natural human-like vehicle control
Data science and engineering is a necessary step prior to any machine learning activity
Machine learning do not replace rigorous engineering – it is an enabler rather than a disruptor
Natural human-like vehicle control is not the same for every passenger
Data-driven engineering is essential to unlock personalised autonomous vehicles
© Hitachi Europe, Ltd. 2019. All rights reserved.
Data-Driven Perception and Planning
Methodologies for Autonomous Vehicles
04/09/2019
Nick Blake, Ph.D
Head of European Big Data Lab
European Big Data Lab
London, United Kingdom
Syed Adnan Yusuf, Ph.D
Senior Research Scientist
European Big Data Lab
London, United Kingdom