safety criteria study for new assessment/test methods of

Safety Criteria study forNew Assessment/Test Methodsof Automated Driving System

(3rd Report)

page1

Transmitted by the expert of Japan

Informal Document VMAD-03-053rd VMAD IWG, July 1-2,2019Agenda item X.

July 1-2, 2019

Overall approach

page2

2nd VMAD IWG

Reprint

3rd VMAD IWG

Traffic Scenario Structure Reprint

Road geometryEgo-vehicle

behaviorTraffic participants’

positionsurrounding vehicle

behavior

Define 8+1 position around ego-vehicle and movement which can invade the ego trajectory.Basically, traffic disturbance scenario can be described with two vehicles

Extract the most demanding parameter from real environmentdata for each road geometry classification based existing laws and regulations.

1

2

1 2

Ego-vehicle behavior

Lane keep Lane change

Main roadway

Free DrivingFollowing

Lane changeOvertaking

Merge Merging in front Merge

Branch --- Branch

Ramp Free DrivingFollowing

Lane changeOvertaking

Ro

ad g

eom

etry

Road Structure Ordinance

page3

Applicable scenarios to Lane keeping system

Applicable traffic scenarios for Lane Keeping system

Other Vehicle's

maneuver

Traffic ScenarioImage

ParametersInitial condition Vehicle motion

InitialVelocity

【Vo0】

Initial distance Lateral motion Deceleration Acceleration

Longitudinal distance

【dx0】

Lateral distance

【dy0】

Max LateralVelocity

【Vy_max】

motiontiming

Max decel G

【Gx_max】

Decel rate

【dG/dt】

Decel timing

Max accel G

【Gx_max】

Accelrate

【dG/dt】

Accel timing

Cut in ----- ----- -----

Cut out ----- ----- -----

Acceleration ----- ----- ----- ----- ----- -----

Deceleration ----- ----- ----- ----- ----- -----

Sync. ----- ----- ----- ----- ----- ----- ----- ----- -----

page4

Today‘s scenario

Within ODD

Rationally Foreseeable

Preventable Unpreventable

Unforeseeable

Within ODD, AD shall not cause rationally foreseeable and preventableaccident resulting injury or death

[WP29 Framework Document Draft]

Out of ODD

Best effort

page5

A

Schematic structure of the safety requirement

Foreseeable: It is important to cover the events occurring in the actual traffic society.

=>Specify the foreseeable range based on the actual traffic data in line with the scenario structure.

Preventable: Socially acceptable criteria for AD needs to be defined through further discussion

ADs are expected to drive safelywithin this area

How to define boundaries?How to define boundaries?

B

A

B

A

B

Foreseeable

Rationally Foreseeable Unforeseeable

page6

A

A

■Classify and accumulate the trajectory data for each scenario

■Define the critical parameter and model the scenario

Derivation Process of Foreseeable Scenario

Extract Statistical distribution of trajectories' parameters

Analysis

Real world data

ParameterSelection

ScenarioStructure

Scenario DB

Trajectory dataw/Road structure

Parametric dataDriving Database

Collect and analyze the traffic flow data including near miss which represents the actual traffic environment.

Based on the statistical analysis, the parameter range which covers not only a)the measured data but

also b) the reasonably foreseeable future events is defined.

0

10

20

30

40

50

60

70

80

0 20 40 60 80 100 120 140 160③

dx0

(In

itia

l dis

tan

ce) [

m]

① Ve0 (Ego vehicle velocity) [km/h]-15

-10

-5

0

5

10

15

20

0 50 100 150 200

②V

e0

-Vo

(R

ela

tive

ve

loci

ty)

[m/s

]

① Ve0 (Ego vehicle velocity) [km/h]-15

-10

-5

0

5

10

15

0 50 100 150

②V

e0

-Vo

(R

ela

tive

ve

loci

ty)

[m/s

]

③ dx0 (Initial distance) [m]

■Extract the applicable data from the trajectory data

■Analyze the data and correlation, and define the parameter range

Parameter Selection

page7

b)Reasonably foreseeable

a)Real data distribution

fitting

Using the statistical analysis,

Define the range including the foreseeable events

Define the possible range of parameters' combination by analyzing the correlation

(e.g.: ego-vehicle velocity x intravehicular distance x relative velocity)

Foreseeable scenario

Traffic data collection in Japan

page8

Measurement vehicles Fixed camera

Data SourceTUAT* Driving Recorder Data

(~2018~)

JAMA Driving Recorder Data

(2008)

Driving Database (2017)

On road Recognition

Database (2017)

Measurement Vehicle Data

Correction (2018~)

Fixed LocationCamera(2018~)

LoggingVehicleｓ

- 60 vehicles 30 vehicles 6 vehicles 3 vehicles3 cameras / each

location

Driver Taxi driver Ordinal driver Ordinal driver Staff Staff -

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

Ego

EgoEgo

Ego

Ego

Ego

Ego

Ego

Ego

Parameter available (Camera/Lidar)

Video only

visible

Not recorded

/

Ongoing data collection ※TUAT：Tokyo University of Agriculture and Technology

actual traffic data collection by third party.

In order to effectively establish the holistic coverage of traffic monitoring data, the fixed

camera is utilized in specific area like ramp, marge, and exit.

計測車両だけでなく定点カメラでランプや合流など特定地点をカバーし、効率的にシナリオ全体をカバー

Traffic data accuracy evaluation

Fixed camera and reference data（X-Y）

定点撮影

撮影画角映像出力

（4K/FHD）

車両検出・追尾処理

追尾軌跡オルソ変換

高精度軌跡（MMS搭載車）（経度・緯度）

比較による軌跡精度確認

車両検出ライブラリ

軌跡リファレンス値

軌跡計測値

経度・緯度平面直角座標変換

リファレンス車

Object detection libraryFixed camera

Reference

Video data output

High precision GPS Car

Detection ＆Tracking

Trajectory generation

Fixed camera value (x, y)

Reference value (x, y)

Data processing

page9

Precision comparison

Coordination convert

Coordinate X [m]Co

ord

inat

e Y

[m]

Blue : ReferenceRed : Measurement

Generate the trajectory data from the traffic flow monitoring data and classify it

according to the scenario structure.

[Accuracy verification]Sufficient accuracy of the data has been ensured by using the reference vehicle.

VelocityLong.

Distance

Lat.

DistanceDece. G

Dece.

Rate

Lat.

velocity

Scenario

Initial velocity(Ve)

VelocityLong.

Distance

Lat.

DistanceDece. G

Dece.

Rate

Lat.

velocity

Scenario


VelocityLong.

Distance

Lat.

DistanceDece. G

Dece.

Rate

Lat.

velocity

Scenario


VelocityLong.

Distance

Lat.

DistanceDece. G

Dece.

Rate

Lat.

velocity

Scenario


VelocityLong.

Distance

Lat.

DistanceDece. G

Dece.

Rate

Lat.

velocity

Scenario


Trajectory dataw/Road structure

0

0.05

0.1

0.15

0.2

0.25

0 10 20 30 40 50 60

Freq

uen

cy

①Ve0(Ego velocity) [m/s]

Measured cases Gauss distribution

Parameter Range Derivation

Statistically analyze the actual data to derive the parameter range.

While ensuring the a) sufficiency of data amount, derive the appropriate parameter range

by b) extrapolating the non-measured events and c) eliminating the inconceivable

events.

a) Sufficiency of data amount

b) Extrapolation of non-measured events c) Elimination of inconceivable events

Using the statistical analysis, define the reasonable parameter

range

=>Define the range including the foreseeable future events

b) Theoretical

foreseeable

future events

a) Measured

data 0

10

20

30

40

50

60

70

80

0 20 40 60 80 100 120 140 160

③d

x0 (I

nit

ial d

ista

nce

) [m

]

① Ve0 (Ego vehicle velocity) [km/h]

-15

-10

-5

0

5

10

15

20

0 50 100 150 200

②V

e0

-Vo

(R

ela

tive

ve

loci

ty)

[m/s

]

① Ve0 (Ego vehicle velocity) [km/h]

-15

-10

-5

0

5

10

15

0 50 100 150

②V

e0-V

o (

Rel

ativ

e ve

loci

ty)

[m/s

]

③ dx0 (Initial distance) [m]

11

11.5

12

12.5

13

13.5

0 100 200 300 400 500 600 700 800

Va

lue

of

3σ

[m

/s]

Number of data [cases]

The value of 3σ became stable as the data amount increased.

=> Sufficiency of data amount has been ensured by analyzing

the convergence of distribution change.

*Sampled 5 times from parent population

page10

N=911

Analyze the correlation between parameters and define the possible

parameter range

(e.g.: ego-vehicle velocity x intravehicular distance x relative velocity)

Correlation of Field data and Foreseeable data

page11

Precondition data :Ve0=80kph, Vy=1.72m/s

Measured data

2.2220

-2-4-6

-8

-10-10.32

420

-2-4-6

-8

-10-10.09

4.12 6420

-2-4

-6

-8-9.84

6.066420

-2-4

-6

-8-9.64

8.018

6420

-2-4

-6

-8-9.36

9.958

6420

-2-4

-6

-8-9.12

11.89

810

6420

-2-4

-6

-8-8.95

12

810

13.1910.2 20 30 40 50 60 66.7

dx0 (Initial distance) [m]V

e0-V

0 (

Rel

ativ

e ve

loci

ty)

[m/s

]

⇒ Not only the observed data but also the data that may theoretically occurin the future is covered.

Foreseeable data

Preventable

Preventable Unpreventable

page12

B

B

Precondition of Case study

In order to verify crush avoidance performance, Ego vehicle‘s driving model is necessary.

Parameters of expert driver model and AD vehicle model are set as examples.

Perception

Action (time to reach target control

amount)

Avoidance

performance

Judgment [time from perception to

action (start braking)]

❶ Perception timing of accident/near miss

❷ Accelerator release time

❸ Brake onset time

❹ Time to reach max. deceleration G

❺ Max. deceleration G

Ego Lead

Brake！！Risk

Perceive

Risk Perceive

timing

❷

Start braking point

Accelerator release time

Brake onset time❸

❹

Max. deceleration G

❶Arrival time

❺

Deceleration[G]

Time(sec)

page13

1.9

0

1

2

3

4

5

6

Tim

e t

o C

olli

sion[s

]

0.3

4.9

Parameters of expert driver model

Reference: Makisita et al.(2001)

0.95

0.3

0.774 ・General driver ：0.689G

・Driving trainees* ：0.774G

1.25

*Driving trainees: Trainees of Japan Safe Driving Center (JSDC)

Central Training Academy for Safe Driving

0

5

10

15

20

25

30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.85G

0.675G

0.5G

Fre

quency c

ount

Tim

e t

o a

ccele

rato

r re

lease

[s]

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

(1) (2) (3) (4)

1.2

0.7

Reference No.

Tim

e t

o b

rake a

pplication[s

]

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

(2)(1)(3)(5)(6)(4)(7)

1.0

1.5

Reference No. time to reach stated deceleration[s]

Min.

Max..

Ave..

0.95

1.25

0.2 0.4

0.3

1.9

Risk Perceive

timing

❷

Start braking point

Accelerator release time

Brake onset time❸

❹

Max deceleration G

❶Arrival time

❺

Deceleration[G]

Time(sec)

(TTC=1.9)

❷ ❸ ❹❶

❺

Trainees

Regular drivers

The difference in distribution of the trainees and

the regular drivers average deceleration values

0.774

A study example of emergency braking

characteristics of driving trainees in Japan

Emergency avoidance performance model

by braking

Reference: Development of an FCW

Algorithm Evaluation Methodology With

Evaluation of Three Alert Algorithms

(NHTSA June2009)

Evaluation subjects:

Average age (35yrs, 18-68 yrs)

66 Males and 43 Females

Evaluation data of attentive

drivers

page14

Virtual test result

Through the virtual test by using the performance of the expert driver, the preventable level could be determined. Then, using this, the test scenario catalog database will be created.

Ve0-V

0(R

ela

tive V

elo

city)[

m/s

]

dx0(Initial Distance) [m]

Result (Ve0=80kph, Vy=1.72m/s)

✔：Success（non-crash), ✘：Fail（Crash） page15

Expert Driver Model

AD Vehicle Model

Access the required information through the Test Scenario Catalog according to the test

environment

<?xml version="1.0" encoding="UTF-8"?>-<SimScenario Name="LS-G1-2-E1-10-00B-3030"><Description>Lane keeping(Steady running) Ego vehicle constant speed、Side vehicle(Pr-s) changes the lane(Cut in: (right -> left)</Description><Road ID="2"/>-<Object Name="Ego vehicle" Category="EgoVehicle"><VehicleInit Speed="40" Abscissa="0" Ordinate="0" Lane="1stDrivingLane"/></Object>-<Object Name="Side vehicle(Pr-s)" Category="OtherVehicle"><VehicleInit Speed="40" Abscissa="0" Ordinate="100" Lane="2ndDrivingLane"/><VehicleScene Speed="40" Abscissa="0" Lane="2ndDrivingLane" Time="3"/><VehicleScene Speed="20" Abscissa="0" Lane="1stDrivingLane" Time="1.5"/><VehicleScene Speed="20" Abscissa="0" Lane="1stDrivingLane" Time="200"/></Object></SimScenario>

Way to apply Test Scenario Catalog Database

<?xml version="1.0" encoding="utf-8"?><OpenDRIVE><header revMajor="1" revMinor="1" name="" version="1.00"

date="2018/10/18 0:00:00" north="0.0000000000000000e+00" south="0.0000000000000000e+00" east="0.0000000000000000e+00" west="0.0000000000000000e+00" vendor="TOYOTA"></header><road name="Mainroad_1Line" length="3000" id="1" junction="-1"><link>

</lane><lane id="2" type="border" level="0"><width sOffset="0.0000000000000000e+00"

a="2.5000000000000000e+00" b="0.0000000000000000e+00" c="0.0000000000000000e+00" d="0.0000000000000000e+00" /></road></OpenDRIVE>

















Scenario XML

Road geometry XMLAudit (Virtual tests)

Real-traffic tests, Proving ground tests

Test Scenario Catalog Database

VelocityLong.

Distance

Lat.

DistanceDece. G

Dece.

Rate

Lat.

velocity

Scenario


Test procedure

Test datapage16

The process to define criteria which is possibly applicable for certification by using a concrete case was shown

Need to discuss1) Definition of “rationally Foreseeable and Preventable”

criteria2) Qualified real world driving data

To Consider common understanding of the socially acceptable criteria issues if each country uses this process to derivate the

criteria

Summary

page17

Next step

safety criteria study for new assessment/test methods of

Documents