summercourse-france-2018-presentation-v0.3 - modo de...

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018

Intelligent Interaction between Vehicles and VRUs

Miguel Ángel Sotelo

SPAIN

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 2

Motivation

Proposed approach

Pedestrian pose measurement

Dimensionality reduction and Prediction

Experimental results

Conclusions and Future Work

Content

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 3

Motivation

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 4

Motivation

• Pedestrian Path Prediction in the Automotive:

– Further improvement in state-of-the-art ADAS by meansof action classification based on body language reading

– Walking, Stopping, Starting, Bending-in

– Improvement of accuracy in 30-50 cm:

– Difference between effective and non-effectiveintervention in emergency braking systems

– Initiation of emergency braking 0.16 s in advance canpotentially reduce severity of accidents injuries by 50%

– Early recognition of pedestrians stopping/walking actionscan provide more accurate last-second active interventions

Strong gains are expected in the performance and reliability of active

pedestrian protection systems

Strong gains are expected in the performance and reliability of active

pedestrian protection systems

Strong gains are expected in the performance and reliability of active

pedestrian protection systems

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 5

Motivation

Proposed approach

Pedestrian pose measurement

Dimensionality reduction and Prediction

Experimental results

Conclusions and Future Work

Content

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 6

Proposed Approach

Global Scheme

Off-line Motion Capture System

Recovery of 3D pose and position

Lateral predicted position

On-line

Probabilistic Training

Knowledge of

Pedestrians

dynamics

Stereo Cameras

Transformation to latent space +

prediction

Geometric processing

3D Pedestrian Pose

Estimation

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 7

Motivation

Proposed approach

Pedestrian pose measurement

Dimensionality reduction and Prediction

Experimental results

Conclusions and Future Work

Content

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 8

Pedestrian Pose Measurement

Pedestrian Skeleton considered in this research

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 9

Pedestrian Pose Measurement

Body parts detection using Deep Learning

St: Confidence MapsLt: Parts Affinity Maps (PAF) (Z. Cao. CVPR 2017)

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 10

Pedestrian Pose Measurement

Body parts detection using Deep Learning

Confidence maps of the right wrist (first row) and PAFs(second row) of right forearm across stages

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 11

Pedestrian Pose Measurement

Body parts detection using Deep Learning

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 12

Pedestrian Pose Measurement

Body parts detection using Deep Learning

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 13

Motivation

Proposed approach

Pedestrian pose measurement

Dimensionality reduction and Prediction

Experimental results

Conclusions and Future Work

Content

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 14

Dimensionality Reduction

Major Goal

Learning the pedestrians motion dynamics by reducingthe dimensionality of input data and to make it moremanageable and interpretable in a low dimensional latentembedding

Method considered

GPDM (Gaussian Process with Dynamical Model)

Input data-set

CMU (41 body joints obtained with motion capture syst.)

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 15

Dimensionality Reduction

CMU Sequence Example

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 16

Dimensionality Reduction

GPDM (1/3)

1. Nonlinear probabilistic generalization of PCA: marginalization of the mapping (W) and optimization of the latent variables (X)

2. Conditional density in GPDM for a centered observed data Y, latent variable X, and kernel hyperparameters θ

where W is a scaling matrix and K is a kernel matrix with:

where δi,j is the Kronecker delta function

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 17

Dimensionality Reduction

GPDM (2/3)

3. Dynamic mapping from latent coordinates:

where Xout=[X2, … , XN], d is the model dimension and KX is the kernel built from {X1, … , XN-1} with:

where βi are hyperparameters

4. Minimization of –ln p(X,θ,β,W|Y) using SCG

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 18

Dimensionality Reduction

GPDM (3/3)

Prediction

1. Predicted latent position in the next frame (kX is a column vector)

2. Reconstructed 3D pose (kY is a column vector)

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 19

Learning Pedestrian Motion

Pedestrian walking 6 steps (using B-GPDM)

Green: projection of pedestrian motion sequence onto the subspace.

Variance in color code: cold (small) to warm (big)

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 20

Experimental Results

Video sequence showing prediction results

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 21

Motivation

Proposed approach

Pedestrian pose measurement

Dimensionality reduction and Prediction

Experimental results

Conclusions and Future Work

Content

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 22

General Method - Overview

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 23

Datasets

Frame Rate of training data

CMU data-set: 120 fps (frames per second)

Feature Vector

UAH: 11 body joints (3D pose + 3D motion: 11x6)

CMU: 41 (we consider only the 11 joints in common with UAH)

Pedestrian Behaviors

Walking (toward the curb to enter the road lane)

Stopping (at the curb)

Starting (from the curb)

Standing

Prediction Horizon

Up to 1.0 s

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 24

Experimental Results

UAH Dataset - Breakdown

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 25

Manual Labelling of Pedestrian Poses

Standing-Starting Transition

Criteria: movement of leg forward

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 26

Manual Labelling of Pedestrian Poses

Starting-Walking Transition

Criteria: pedestrian places foot on the ground after one stride

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 27

Manual Labelling of Pedestrian Poses

Walking-Stopping Transition

Criteria: pedestrian places foot on the ground in the final stride before full stop

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 28

Manual Labelling of Pedestrian Poses

Stopping-Standing Transition

Criteria: pedestrian comes to a full stop

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 29

Activity Recognition

Clues for prediction of “Starting” behaviors

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 30

Activity Recognition

Clues for prediction of “Stopping” behaviors

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 31

Activity Recognition

Pedestrian activity modelled as HMM

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 32

Activity Recognition

Pedestrian activity modelled as HMM

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 33

Activity Recognition

Pedestrian activity modelled as HMM

Prior

Emission (Likelihood)

α: SSE for pedestrian poseβ: SSE for pedestrian displacement

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 34

Activity Recognition

Recognition results for different features and #joints

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 35

Experimental Results

Probabilistic Action Classification

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 36

Experimental Results

Detection Delay - Summary

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 37

Experimental Results

UAH Prototype vehicle - DRIVERTIVE

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 38

Experimental Results

Experiments in UTAC proving ground (France) – BRAVE H2020 Project

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 39

Experimental Results

Experiments in UTAC proving ground (France) – BRAVE H2020 Project

Expected Contributions to Euro NCAP

- Recommendations to Euro NCAP WG on AD on how to measure performance of VRU prediction systems

- Enhanced performance in Euro NCAP tests by including predictions

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 40

Experimental Results

Experiments in UTAC proving ground (France) – BRAVE H2020 Project

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 41

Experimental Results

Experiments in UTAC proving ground (France) – BRAVE H2020 Project

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 42

Intelligent Interface with VRUs

GRAIL – GReen Assistant Interface Light

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018

Use Case 1 – Pedestrians (GRAIL-1)

The pedestrian stops and looks at the ego-vehicle

Ego-vehicle decelerates until full stop and indicates to thepedestrian that s/he can cross.

The pedestrian continues and completely crosses.

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018

Use Case 1 – Pedestrians (GRAIL-2)

The ego-vehicle decelerates, the pedestrian turns his/her head to the left indicating a possible intention to cross

The ego-vehicle comes to a full stop while indicating the pedestrian that s/he can cross.

The pedestrian crosses.

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018

Use Case 2 – Pedestrians (passing-by)

Pedestrian walking along the road

The ego-vehicle decelerates

Passes the pedestrian slowly.

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018

Use Case 3 – Pedestrians (avoidance)

25%

Crossing scenario with impact point at 25%.

AES activation to avoid the impact.

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018

Use Case 4 – Cyclists (passing-by)

It’s possible to overtake so deceleration takes place

and overtaking with the correct distance with the bicycle.

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018

Use Case 5 – Cyclists (traffic context)

The ego-vehicle decelerates and waits to the oncoming traffic to pass

After that it overtakes.

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 49

Intelligent Interface with VRUs

GRAIL – GReen Assistant Interface Light

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 50

Intelligent Interface with VRUs

GRAIL will be demonstrated at IEEE IROS 2018

First week of October, Madrid

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018 51

Motivation

Proposed approach

Pedestrian pose measurement

Dimensionality reduction and Prediction

Experimental results

Conclusions and Future Work

Content

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018

Conclusions and Future Work

Conclusions

- Accurate path pedestrian prediction is possible up to 1.0 s using body language traits and probabilistic dimensionality reduction techniques.

- Predictions are carried out in the latent space, yielding results with a potential accuracy of 15-20 cm at a time horizon of 1.0 s

- The generative solution is not scalable to thousands of examples. A new approach is needed for massive learning.

Future Research Challenges

- Taking a discriminative approach to the prediction problem: Deep Learning? Recurrent Neural Networks?

- VRU recognition (standing pedestrian, seating pedestrian, cyclist, motorcyclist).

- Context-based action prediction:

- Gaze direction, group behavior, type of road.

- Use of Probabilistic Graphical Models (Bayesian Networks).

Summer School on Cooperative Interacting Automobiles, Loire Valley, France. September 2018

Merci beaucoup!

Miguel Ángel Sotelo

miguel.sotelo@uah.es University of Alcalá, SPAIN

Intelligent Interaction between Vehicles and VRUs