f

Networked Embedded Applications and Technologies Lab

Department of Computer Science and Information Engineering National Cheng Kung University, TAIWAN

Implementation of In-Vehicle Multi-sensor Information Fusion Gateway for Cooperative Driving

Authors: Ting-Ying Wei, Zhi-Liang Qiu, Chung-Ping YoungPresenter: Zhi-Liang Qiu

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 2NEAT

Outline

Introduction Multi-Sensor Information Fusion Hardware architecture

Vehicle Gateway Electronic Control Units (ECUs) Communication Sensor Software

Conclusion

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 3NEAT

Introduction

The traffic crashes caused by the human factor in US are about 80%-90%

The safety concern is the most important issue for vehicle operation Cooperative driving

Enhance the driving information Increase the response time

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 4NEAT

Introduction

Precaution warning or compensatory control before the pre-crash stage of vehicle safely

Heterogeneous multi-core processor: One ARM core for I/O control and system management Two DSP cores for intensive computation of information

fusion Signal processing Location mapping Trajectory prediction Risk assessment

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 5NEAT

Introduction

Two types of tasks: Real-time Non-real-time

The ARM processor: General Purpose Processor (GPP) for the non-real-time tasks

PAC DSP: Special Purpose Processor(SPP) for the real-time tasks

Electronic Control Unit(ECU): Gathering the surrounding information

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 6NEAT

Multi-Sensor Information Fusion

Vehicle gateway implements the information fusion In-vehicle sensor data Wireless communication packets

GPP will inform the data to Information fusion engine in SPP whenever the data is received

SPP generates several safety indexes and produce warning messages

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 7NEAT

Multi-Sensor Information Fusion

Procedure: Retrieving information Integrating the information in a time sequential linked list Cross-evaluating the information Mapping each car in the tree structure onto a highway map Predicting the upcoming path Generating warning messages Broadcasting to surrounding vehicles

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 8NEAT

Multi-Sensor Information Fusion

Procedure:

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 9NEAT

Multi-Sensor Information Fusion

Time synchronization base on the GPS signal from each car

Image processing DSPs detect the lane departure and blind spot Camera frame rate: 30fps Sensor sampling rate: 200Hz

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 10NEAT

Multi-Sensor Information Fusion

Location mapping: GPS data is only be used for global positioning route

planning or track recording Merging the signals from radar, accelerometer, wheel speed

sensor, and yaw sensor generates the relative distance, position and velocity

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 11NEAT

Multi-Sensor Information Fusion

Trajectory prediction: Using the current and past vehicle information as well as the

circumstance event to estimate the next few seconds path If there are any specific events, like emergency brake, we will

receive the warning from the event car

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 12NEAT

Multi-Sensor Information Fusion

Risk assessment:

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 13NEAT

Vehicle Gateway

The vehicle gateway is a core of this system, it includes many different interfaces of communication as shown below

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 14NEAT

Vehicle Gateway

The realization of vehicle gateway

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 15NEAT

Vehicle Gateway

PAC Duo DSP development platform

•The PAC Duo platform is developed by Industry Technology Research Institute (ITRI) in Taiwan•Each DSP core communicates with ARM core through AXI-AHB bus

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 16NEAT

Electronic Control Units

Microchip PIC18Fxx8 microcontroller Renesas R32C/118

The microcontrollers are the middle layers between PAC Duo and the surrounding sensors

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 17NEAT

Communication

There are several wired or wireless of communication interfaces are required CAN serial bus protocol OBDII IEEE 802.15.4 DSRC 3G

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 18NEAT

Sensors

Different types of sensors are used for acquiring vehicle diagnosis or driving status Infrared night vision camera The GPS receiver

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 19NEAT

Software

The software is separated into several parts Non-real-time OS, Meego

Open source Linux-based mobile OS Supported by Intel and Nokia Provide the GUI

Real-time on PAC DSP Non-OS on each microcontroller board

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 20NEAT

Conclusion

A heterogeneous multi-core platform is establishing. The multi-sensor information fusion is implemented for

computing the relative position, velocity and acceleration to the host vehicle.

Assess the potential of collision risk and provide the cooperative driving feature to enhance the active safety

Department of Computer Science and Information EngineeringNational Cheng Kung University, TAIWAN 21NEAT

Thank you for attention!