autonomous driving validation using game engine …...autonomous vehicle validation [ õ] and hence...
TRANSCRIPT
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 1
Co-Authors:
Jithin Jose, Tata Elxsi
Jiljith John, Tata Elxsi
Dashlin Jeno, Tata Elxsi
Arun R, Tata Elxsi
Muhammad Asif V, Tata Elxsi
Autonomous Driving Validation using Game Engine
Technology
Jihas Khan, Tata Elxsi
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 2
TABLE OF CONTENTS
Abstract ......................................................................................................................................................... 3
Introduction .................................................................................................................................................. 4
Challenges in Real-World Validation and Testing of Autonomous Driving ................................................... 5
What Can be Done about it? ......................................................................................................................... 7
Leveraging Game Engine Technology For Autonomous Driving validation .................................................. 8
Key highlights .............................................................................................................................................. 10
Conclusion ................................................................................................................................................... 14
About Tata Elxsi ........................................................................................................................................... 15
References .................................................................................................................................................. 16
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 3
ABSTRACT
In this paper, we present the various challenges and risks associated with real-world validation of AI-based
Autonomous Vehicles (AVs) and how the prototyping and production of these vehicles would benefit from
virtual validation. This paper provides statistical and graphical analysis of the potential risks and
challenges involved in AV validation and how these risks and challenges exponentially increase when
dealing with a fleet of such vehicles. In order to seek out an effective solution, we benchmarked various
Autonomous Driving (AD) virtual validation platforms based on several criteria essential for AD validation
and realized their limitations. This paper gives you a peek into the advantages of using game engine
technology to re-create a real-world test scenario fully in a virtual manner. A virtual validation tool
enabling automatic generation of millions of scenarios and even allowing manual creation of user-specific
scenarios will provide much-needed validation of AD systems before deployment.
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 4
INTRODUCTION
With the current advancements in technologies such as Artificial Intelligence (A.I.), it is clear that a future with fully autonomous vehicles is just around the corner. A fully autonomous vehicle has many advantages over its human-driven counterpart. While reduced accidents would be the primary advantage of autonomous driving, these vehicles will also have better fuel efficiency and reduced emissions. Hence, it is of no surprise that most automobile manufacturers are leaning towards this proposed future.
Due to the relative infeasibility of real-world validation and testing of autonomous driving solutions, the automotive sector leans towards the more practical solution of virtual simulation. Currently, many platforms enable virtual validation and testing of autonomous driving. CARLA is one such platform, which uses game engine technology. CARLA [1] possesses numerous realistic vehicles, pedestrians, infrastructures and sensors models required for creating a virtual environment, with road network creation possible using tools such as RoadRunner. IPG CarMaker [2] and Mechanical Simulation CarSim [3] are two other platforms that provide similar functionality with Sil, Mil, Hil, and Dil support and MatLab/Simulink interface. dSPACE ASM [4] supports a multitude of sensor models with support to only dSPACE platforms, while TESIS veDYNA [5] can be used in any hardware but requires modification of the plan model for functionality. TASS International PreScan [6] is a simulation platform that provides real-time physics and collisions but lacks the ability to use external models. VIRES VTD [7] is another platform that provides most of these features along with realistic visuals. With support to automotive standards such as OpenDRIVE and OpenCRG, Vires VTD provides robust features. Figure 1 shows the comparison between these platforms with respect to their vehicle dynamics models, sensor support, 3D rendering, real-time physics and collisions, actor maneuvers and behaviors, model libraries and hardware dependencies.
This paper discusses the shortcomings of real-world validation of testing of AD solutions, compares various
virtual validation platforms that currently exists in the market and discusses how a game engine can be
used to provide end-to-end solution for validation of Autonomous Driving algorithm with its powerful,
adaptable and fully customizable sensor models, environment simulation, vehicle models, vehicle
Figure 1: High Level Comparison between existing virtual autonomous driving validation platforms
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 5
dynamics, static and dynamic actors, map generation, physics simulation, and photorealistic rendering.
UE4 (Unreal Engine 4) is the gaming engine used to achieve the 3D photorealistic real-world rendering,
where sensors can perceive and provide precise ground truth data for every object/property in an
environment. The user has full control over customization and creation of edge/complex scenarios instead
of testing in a real-world environment. This, in turn, helps to accelerate the development, testing, and
deployment of Autonomous Vehicles.
CHALLENGES IN REAL-WORLD VALIDATION AND
TESTING OF AUTONOMOUS DRIVING
In real-world validation and testing of autonomous driving solutions, the test vehicle with the AD
algorithm is driven through various scenarios and conditions in the real world, for long distances and
durations, in order to collect data for training and testing the AD algorithm as well as to analyze its
performance. This approach has drawbacks to it, which are described below.
In order to train and validate the A.I. based AD algorithm in these vehicles, it needs to be exposed
to millions of scenarios. The sheer volume of scenarios required makes it impractical for real-world
validation and testing
These scenarios must include traffic conditions of various intensity, different road structure and
surface types, dynamic weather phenomenon, variations in temperature and lighting and the day-
night cycle. The creation of such scenarios in a real vehicle in the real world would be expensive
and time-consuming.
Further, these scenarios must also include dynamic interaction of the test vehicle with other
dynamic actors and a multitude of rare events. Construction works, road hazards, accidents,
pedestrians that suddenly cross the road and rogue vehicles that don’t follow the traffic rules are
a few examples of the rare events. These rare events raise additional challenges, as manufacturing
these scenarios in the real world is not a viable solution.
Another cause for concern regarding real-world testing is the multitude of health and safety risks
that need to be considered. The major issue here is the one thing that makes AD vehicles what
they are – the absence of the human driver. In a conventional vehicle or even one with Advanced
Driver Assistance System (ADAS) features, the driver can intervene at times of emergency. Further,
while testing of ADAS features if the said system does fail, the driver can assume control of the
vehicle. This is not the case when the driver is the software being tested. Scenarios that push
algorithm to its limits are fundamentals in any testing procedure. Figure 2 shows the number of
disengagements in AD test vehicles in California during the period from December 2017 to
November 2018 [8]. Miles per disengagement is a measure of distance traveled by the AD vehicle
(in miles) before the AD technology disengages due to the failure of the AD technology. This shows
the frequency of disengagements in AD test vehicles and hence the risk they pose during real-
world testing.
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 6
Figure 2: Number of AD disengagements in vehicles tested in California from Dec 2017 to Nov 2018
The creation of edge conditions where the AD vehicle is about to collide with a pedestrian or with
another vehicle would have obvious risks. If Ad technology were to fail or disengage, it could lead
to catastrophic outcomes
To collect sufficient data, these vehicles must be driven for long durations and long distances while
being controlled by their AD algorithm
The challenges presented above were merely those present for the validation and testing of a
single AD vehicle. Training and validating a fleet of AD vehicles would significantly multiply these
challenges. Further, these test scenarios must be repeated until the results obtained are of
statistical significance. This further shows the impracticality of validating and testing AD
algorithms in the real world. Figure 3 compares the miles per disengagement for AD test vehicles
in California during the period from December 2017 to November 2018. This comparison along
with the number of disengagements from Figure 2 provides us with an estimate of the amount of
distance that was covered for the testing of these vehicles. This vast distance traveled and the
amount of time taken to do so acts as limitations for real-world testing.
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 7
Figure 3: Miles per disengagement for AD vehicles tested in California from Dec 2017 to Nov 2018 (Source:
thelastdriverlicenseholder.com)
WHAT CAN BE DONE ABOUT IT?
A more practical solution to Autonomous Driving validation and testing would be to train, test and validate
these algorithms in a virtual vehicle in a simulated environment. This will enable the creation of test
scenarios to be created in a controlled virtual world, which would otherwise be impossible to recreate in
the real world. Further being a virtual simulation, test automation and batch execution of test scenarios
enable accelerated validation and testing.
A game engine, that blurs the line between reality and simulations, facilitates the creation of the simulated
environment. We can use game engine technology for providing reliable ground truth data for
autonomous vehicle validation [9] and hence use it in HiL, SiL, ViL, PiL, MiL and DiL applications. Game
engines enable usage of complex collisions and realistic physics in order to simulate realistic interactions
between the ego vehicle (vehicle under test) and other vehicles, pedestrians and animals. Further, this
technology enables the creation of a vast variety of environments such as cities and rural areas, with
editable terrain. Environment simulation can also be extended to dynamic weather and lighting which
enables a vast range of weather and time-of-day simulations that can influence test parameters and ego
vehicle movement. This is done using realistic lighting and particle effects. Lastly, the high-end game
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 8
engine technology enables faithful visual recreation of vehicles, pedestrians and static objects. This
enables added visual realism to be present in a virtual environment. The end result is that the AD algorithm
would be tricked into believing that it truly is in the real world.
Figure 4: A virtual scenario visualized using Unreal Engine 4
LEVERAGING GAME ENGINE TECHNOLOGY FOR
AUTONOMOUS DRIVING VALIDATION
A test scenario would be a combination of values consisting of road details, environmental conditions, and
traffic details. According to these values, the proposed tool simulates a virtual environment. The tool
facilitates the creation of scenarios in two ways:
1) Manual - Creating the scenario using a graphical user interface, where the user can interact to create
the full scenario as per his/her requirements.
2) Automatic - An intelligent scenario generation algorithm generates unique and realistic scenarios based
on some rules and preconditions set by the user.
This scenario will have a road network in a defined environment with both static and dynamic actors. The
road network can either be created by the user or loaded from a standard format such as OpenDRIVE. It
would consist of various road geometries, tunnels, bridges, lane and road markings, road and roadside
objects, and traffic signs and signals. The static actors would consist of various static infrastructures like
buildings and vegetation, while the dynamic actors consist of the Ego and other POV vehicles, pedestrians
and animals. Finally the environment would consist of the terrain information, weather, and lighting.
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 9
The trajectory of the Ego vehicle could be pre-defined by the user or could be controlled during run-time
by an external algorithm using CAN, ROS, TCP, UDP and FMI modes of data transfer. The trajectories of the
other POV vehicles, pedestrians and animals are either pre-defined by the user in manual mode or defined
by the tool in automatic mode.
Figure 5: Rendered 2D Scenario
Figure 6: Rendered 3D Scenario
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 10
The created scenario is visualized using the Unreal Engine 4 (UE4) game engine. The use of a game engine
like UE4 facilitates faithful recreation of the created test scenario. Realism is achieved by the usage of
vehicle, pedestrian and animal models that have realistic aesthetics, physics, and behavior. This enables
faithful recreation of real-world interactions between these dynamic actors as in the case of accident
scenarios or vehicle and pedestrian detections. The tool will have an inbuilt vehicle dynamics model. The
user can also interface vehicle dynamics models of the user’s choice, e.g. MatLab models via a MatLab
interface, further enabling user customizability.
UE4 also facilitates modeling of various sensors: Ultrasonic, RADAR, LIDAR, Camera, GPS, IMU, and Wheel-
tick sensors. These sensors can be used as ideal models for benchmarking real sensors or can be used as
realistic models by the introduction of errors. These errors are generated by mathematical distributions
or using UE4’s dynamic simulation environment. Semantic segmentation and data annotation for each
frame of the rendered scenario is performed. The sensor outputs can be transferred through CAN, TCP,
UDP and ROS channels. The sensor models would also be FMI compliant, enhancing modularity.
Figure 7: LIDAR point cloud data visualization
KEY HIGHLIGHTS
Our analysis of the currently existing AD virtual validation platforms has shown that there is a need for a
solution that encompasses all the merits of each of these platforms and overcome their flaws. The
proposed Tata Elxsi solution does so while also having a set of aspects unique to it. These are:
Realistic 3D modeling and simulation –
The use of Unreal Engine 4 enables faithful 3D recreation of dynamic models and the
environment. These dynamic models include vehicle models that accurately represent their real-
world counterparts.
User-friendly GUI –
The solution facilitates quick and easy creation of test scenarios. This, combined with the user-
friendly nature of the GUI, which enables any user to create test scenarios without much training,
enables accelerated test scenario generation.
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 11
Automatic scenario generation –
The solution has the ability to automatically create scenarios based on a set of rules or parameters
defined by the user. The tool does so using a generation algorithm, which is fine-tuned so that
each generated scenario is logical, realistic and unique. The parameters defined by the user
include the type, count and behavior of both static and dynamic actors, using which the
generation algorithm generates a random but logical scenario. Figure 7 describes the approach
for automatic scenario generation.
Figure 8: Approach for automatic generation of scenarios based on user-defined parameters
Real-time physics and collisions –
Unreal Engine 4 provides the ability to simulate fast and accurate physics interactions between
actors and realistic collision detections. This enables the simulated scenario to have simulated
realism without the need for external models or plug-ins.
A vast and editable library –
Users will have access to a vast library of static and dynamic actors. Further, the user has the
ability to customize these actors or import a model of their choice into the library. This added
layer of customizability promotes the flexibility of the tool for each user.
Sensor simulation –
The solution facilities realistic and ideal sensor modeling of n-number of sensors of each type –
Ultrasonic, RADAR, LIDAR, Camera, GPS, IMU, and Wheel-tick.
Integration with external vehicle dynamics models –
Users can integrate external vehicle dynamics models, such as MatLab models through a MatLab
interface. This promotes further customizability of the solution to fit the user’s requirements.
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 12
Data annotation –
The solution provides data annotation output for each frame of the rendered scene. This output
contains information regarding the bounding box, location, and orientation of each actor in the
frame. This can be used as ground truth data required for validation and testing. Figure 6 show an
example of such an output. Figure 9 shows the annotated data output from the solution.
Figure 9: Annotated data output
Test automation –
Batch execution of test scenarios coupled with automatic scenario generation leads to the
solution accelerating the validation and testing process.
Modularity –
Each fundamental component of the solution is capable of being scaled, modified and interfaced
with as per the requirements. These fundamental parts include- scenario creation, road network
creation, sensor configuration, and scenario visualization.
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 13
SAMPLE USE CASE
This sample use case shows how a version of the proposed solution has been customized as per the
requirements of a leading Tier 1. This version has been customized to automatically generate a vast
number of scenarios that would then be used to train the AI algorithm present in an AD camera ECU. The
tool generates each frame of each rendered scenario along with a semantic segmented image and
annotated data for that frame. These would then be used as ground truth data for training the AI algorithm
implemented in the AD camera ECU. During the testing phase, the algorithm output for each scenario is
compared with the ground truth by the test automation framework. The test result is generated as a
report of the detected and expected data. The architecture of the proposed version is shown in Figure 10.
Figure 10: Test system architecture for AD camera ECU training
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 14
CONCLUSION
How many miles of road testing would an Autonomous Vehicle would require? How much time would
that take? Without simulation it is difficult to test all the testing scenarios an Autonomous Driving vehicle
would find in a real dynamic environment. This paper identifies existing challenges in the validation of an
AI-based Autonomous Driving (AD) algorithm. These challenges include the sheer volume of data to be
processed, the time and monetary constraints required to train and test these algorithms in real-world
environments. Further simulating corner and rare cases in the real-world would add further difficulty in
testing and validation. Users should have the ease of option to generate millions of AD scenarios
automatically facilitating faster scenario generation, which in turn would promote workload reduction.
The use of game engine technology enables rendering realistic 3D simulation, which helps attain higher
accuracy and efficiency. Further, due to the entire simulation process being in a virtual scenario, real-
world risks would be non-existent. Further to the game engine for validating AD technology, the option of
simulating sensor models and vehicle dynamics would be required to fully validate the AD controller
algorithm or the ECU. This simulated game engine based solution would result in an accelerated,
economical, efficient, and risk-free solution to AD testing and validation. The proposed method of using
game engine technology for validation of autonomous driving technology is well accepted by OEMs and
Tier 1s, for activities that were merely achievable using existing tools in the market.
A 3D simulation solution “VDRIVE” from Tata Elxsi which uses game engine technology overcomes the
challenges discussed in this paper and is an effective tool to test the complexity of AI-based Autonomous
Driving algorithms.
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 15
ABOUT TATA ELXSI Tata Elxsi is amongst the world’s leading providers of design and technology services for product design,
engineering, solutions, electronics and software for the Automotive industry. Accredited with Automotive
SPICE Level 5 certification, premium member of the AUTOSAR consortium and other industry partnerships
help us gain a competitive advantage in the global market place.
Tata Elxsi works with leading OEMs and suppliers in the automotive and transportation industries for R&D,
design and product engineering services from architecture to launch and beyond. Tata Elxsi stays invested
in developing solutions that would help clients achieve a faster time to market. Supported global brands
in the automotive space (OEMs and Tier-1 suppliers) for testing and validation of their programs. Tata
Elxsi has designed and developed 40+ HILS for testing single and multiple ECU’s (Infotainment, BCM,
Chassis, and Powertrain ECUs).
Autonomous Driving Validation Using Game Engine Technology
[email protected] © Tata Elxsi 2019 16
REFERENCES
1. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio López, and Vladlen Koltun. CARLA: An
Open Urban Driving Simulator
2. CarMaker: Virtual testing of automobiles and light-duty vehicles. Accessed on: Nov. 20, 2019.
[Online]. Available: https://ipg-automotive.com/products-services/simulation-
software/carmaker/
3. CarSim Mechanical Simulation. Accessed on: Nov. 20, 2019. [Online]. Available:
https://www.carsim.com/products/carsim/index.php
4. Automotive Simulation Models (ASM): Tool suite for simulating the engine, vehicle dynamics,
electrical system, and traffic environment, 2019. [Online]. Available:
https://www.dspace.com/shared/data/pdf/2019/dSPACE_ASM_Catalog2019.pdf
5. veDYNA: More efficiency in component and controller development with comprehensive vehicle
dynamics simulation, 2018. [Online]. Available:
https://www.tesis.de/fileadmin/Downloads/Flyer_Broschueren/veDYNA/Product_overview_ved
yna.pdf
6. preScan | TASS International. Accessed on: Nov 20, 2019. [Online]. Available:
https://tass.plm.automation.siemens.com/prescan
7. VIRES VTD Details 201403, March 2014. [Online]. Available:
https://www.vires.com/docs/VIRES_VTD_Details_201403.pdf
8. Mario Herger, UPDATE: Disengagement Reports 2018 – Final Results, Feb. 13, 2019. Accessed on:
Nov 20, 2019. [Online]. Available: https://thelastdriverlicenseholder.com/2019/02/13/update-
disengagement-reports-2018-final-results/
9. S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games.
In European Conference on Computer Vision (ECCV), 2016.