Behavior Planning For Connected Autonomous Vehicles UsingFeedback Deep Reinforcement Learning

Songyang Han, Fei Miao1*

1University of Connecticut371 Fairfield Way, Department of Computer Science and Engineering

Storrs, Connecticut 06269songyang.han@uconn.edu, fei.miao@uconn.edu

Abstract

With the development of communication technologies, con-nected autonomous vehicles (CAVs) can share informationwith each other. We propose a novel behavior planningmethod for CAVs to decide actions such as whether to changelane or keep lane based on the observation and shared in-formation from neighbors, and to make sure that there ex-ist corresponding control maneuvers such as acceleration andsteering angle to guarantee the safety of each individual au-tonomous vehicle. We formulate this problem as a hybridpartially observable Markov decision process (HPOMDP) toconsider objectives such as improving traffic flow efficiencyand driving comfort and safety requirements. The discretestate transition is determined by the proposed feedback deepQ-learning algorithm using the feedback action from an un-derlying controller based on control barrier functions. Thefeedback deep Q-learning algorithm we design aims to solvethe critical challenge of reinforcement learning (RL) in aphysical system: guaranteeing the safety of the system whilethe RL is exploring the action space to increase the reward.We prove that our method renders a forward invariant safe setfor the continuous state physical dynamic model of the sys-tem while the RL agent is learning. In experiments, our be-havior planning method can increase traffic flow and drivingcomfort compared with the intelligent driving model (IDM).We also validate that our method maintains safety during thelearning process.

IntroductionThe development of the Dedicated Short-Range Commu-nication (DSRC) technology (Tangade, Manvi, and Lorenz2018) and 5G technology enables Vehicle-to-Vehicle (V2V)and Vehicle-to-Infrastructure (V2I) communication. TheU.S. Department of Transportation (DOT) estimated thatDSRC based V2V communication could potentially addressup to 82% of crashes in the United States and preventthousands of automobile crashes every year (Kenney 2011).When basic safety message (BSM), including current ve-locities and positions, is communicated for connected au-tonomous vehicles (CAVs), control approaches for scenarios

*This work was supported by NSF S&AS-1849246, NSF CPS-1932250 grants.Copyright © 2021, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

such as cross intersections or lane-merging (Rios-Torres andMalikopoulos 2017; Lee and Park 2012; Ort, Paull, and Rus2018) have shown the advantage of information sharing.

When a lane-changing decision is already made, continu-ous state space controllers are designed for vehicles to mergeinto or leave the platoon (Li et al. 2017). However, existingcontrol frameworks for CAVs, such as platooning (Liang,Martensson, and Johansson 2016), adaptive cruise control(ACC) (Nilsson et al. 2016), and cooperative adaptive cruisecontrol (CACC) (Gritschneder, Graichen, and Dietmayer2018) mainly focus on the controller design when a decisionabout lane changing or keeping has been provided (Guanetti,Kim, and Borrelli 2018). End-to-end neural network-basedapproaches to learn control signals for steering angle andacceleration directly, based on camera images has been de-signed (Codevilla et al. 2018; Bansal, Krizhevsky, and Ogale2018; Amini et al. 2020). RL is also used based on the pre-dicted trajectory (Henaff, LeCun, and Canziani 2019), tolearn traffic control policies of a group of autonomous vehi-cles (Jang et al. 2019). However, these deep learning-basedapproaches only consider lane-keeping scenarios withoutlane-changing cases or safety guarantees.

To solve autonomous vehicle’s challenges, one paramountconcern raised by RL is how to ensure the safety of thelearned driving policy. It has been proved that merely as-signing a negative reward for accident trajectories cannot en-sure driving safety–it enlarges the variance of the expectedreward, which in turn prohibits the convergence of stochas-tic gradient descent algorithm for policy learning (Shalev-Shwartz, Shammah, and Shashua 2016). Instead, researcheshave used deep learning to learn a policy to mimic the con-trol law of model predictive control (MPC) (Chen et al.2018) or used MPC to generate samples for training in theguided policy search (Zhang et al. 2016). Paper (Cheng et al.2019) adds control barrier functions (CBF) to the model-freeRL to ensure end-to-end safe training. The main differencebetween this work and ours is: (1). We consider a hybrid sys-tem where the CBF and RL have different action and statespace, while in previous work they share the same actionand state space; (2). We have stronger motivation to applymodel-free RL algorithms since the state transition is com-pletely unknown, while in their work they have a rough sys-tem dynamic model at the beginning and the model is get-

arX

iv:2

003.

0437

1v2

[cs

.AI]

1 O

ct 2

020

ting more accurate along with training. The other popularapproach is to decompose the learning and control phases.Learning-based methods can make a high-level decision,including lane-changing or lane-keeping (Pan et al. 2017).However, splitting learning and control steps may cause in-consistency in control-based and learning-based decisions.Is there a better way to integrate the strength of both? More-over, when a CAV gets extra knowledge about the environ-ment via V2X communication, how to make tactical deci-sions such as whether to change lane or keep lane to improvetraffic efficiency, to best utilize the shared information is stillunsolved challenges for hybrid systems.

The existing multi-agent reinforcement learning litera-ture (MARL) (Zhang, Yang, and Basar 2019) has not fullysolved the CAV challenges yet–either how communicationamong agents will improve systems’ performance or safepolicy learning is not analyzed clearly. The recent advancesin MARL include Multi-Agent Deep Deterministic PolicyGradient (MADDPG) and League training. MADDPG usesa centralized Q-function to alleviate the problem caused bythe non-stationary environment (Lowe et al. 2017). Leaguetraining is used in AlphaStar to avoid the cycles that ap-pear in self-play and increase the diversity of the strategies(Vinyals et al. 2019). In this work, we focus on safe policylearning of each individual autonomous vehicle in the CAVsystem based on shared information, while the other vehi-cles (CAVs and non-CAVs) are considered as the environ-ment that the ego vehicle can observe. MARL is out of thescope of this paper and will be considered as future work.

Hence, we design a learning algorithm that uses shared in-formation to enhance behavior planing performance for fu-ture CAVs with feedback from controllers and safety guar-antees. We show the benefit of V2V communication in im-proving both system efficiency and safety. We focus on thebehavior planning challenge for CAVs, where we want toensure the safety for both discrete and continuous dynamicswhen learning a good policy for a hybrid system. We modelthe problem as a hybrid partially observable Markov deci-sion process (HPOMDP) with information sharing. The dis-crete state transition policy is explored by a RL algorithmand the continuous state is controlled under a CBF-basedquadratic programming (QP). We add a feedback processto guarantee the action to be explored is safe and feasible.We also prove the CBF-QP can be used to render a for-ward invariant safe set even when there is a bounded noiseof the continuous dynamics. From experiment results, thefeedback DRL based behavior planning improves the sys-tem efficiency in terms of the traffic flow and driving com-fort while ensuring safety during the entire learning process.

The main contributions of this work are:

• We propose a feedback deep Q-learning algorithm for ahybrid system to explore a policy for discrete state transi-tions. This algorithm has a feedback process to ensure thesafety of both the discrete and continuous dynamics of theproposed HPOMDP.

• We prove that the quadratic programming with controlbarrier functions renders a forward invariant safe set forthe continuous dynamics of the physical system. We also

prove that the forward invariant set still exists when thecontinuous dynamics has a bounded noise.

• We utilize the proposed feedback deep Q-learning methodto solve the behavior planning problem for CAVs givenshared information. We show in experiments that thesafety is guaranteed and the system efficiency is improvedin terms of traffic flow and driving comfort.

Problem FormulationThe V2V and V2I communications extend the informationgathered by a single-vehicle further beyond its sensing sys-tem. This work addresses how to utilize shared informa-tion to make better behavior decisions such as when tochange/keep lane for CAVs, considering safety guaranteesof CAVs and the transportation system efficiency. The ve-hicles can only observe the state of the world by its ownsensors (observation) or use the information shared by oth-ers (shared information). As shown in Fig. 1, we formulateour problem as a HPOMDP defined as follows.

Definition 1 (Hybrid Partially Observable Markov De-cision Process (HPOMDP)). A HPOMDP is a collectionH = (S,A,O,M,R, πθ, T ,o, r,X ,U , Init,F , C) where

• S is a set of states describing the world.• A is a discrete set of actions including all behaviors. The

action determines the discrete state of this hybrid system.• O is a set of observations.• M is a set of shared information.• R is a set of reward.• πθ : O×M×A → [0, 1] is a policy parameterized by θ.• T : S ×A → S is a state transition function.• o : S → O is a private observation that the agent receives

correlated with the state.• r : S × A → R is reward assigned for each state-action

pair.• X ⊆ Rn is a compact set of continuous states of the un-

derlying controller.• U ⊆ Rm is a compact set of the control inputs of the

underlying controller.• Init ⊆ A×X is the set of initial behavior and underlying

states.• F : A×X ×U → X assigns to each a ∈ A a continuous

vector field F(·, a), a function from X × U to X .• C is a safe set that defines safe state x ∈ X .

We consider the vector fieldF(·, a) in control affine form:

xt+1 = f(xt) + g(xt)ut, (1)

with f and g locally Lipschitz, x ∈ X ⊂ Rn is the state andu ∈ U ⊂ Rm is the control input. Here we use a discretiza-tion of the continuous time state space system, because weconsider a controller with 100 Hz frequency in our problem.We define forward invariant within a safe set and the con-trol barrier function for guaranteeing safety of this systemas follows.

a1!"

s1 s2 s3

o1a2

o2

m2

!o o

"(sk+1 !sk, ak) "(sk+1 !sk, ak)

xt+1 = f(xt) + g(xt)ut

!"

m1xt+1 = f(xt) + g(xt)ut

Figure 1: The hybrid partially observable Markov decisionprocess.

Definition 2 (Forward invariant (Ames et al. 2019)). De-note the safe set C by the super-level set of a continuouslydifferential function h : Rn → R,

C : {x ∈ X ∈ Rn : h(x) ≥ 0}. (2)

The set C is forward invariant if ∀x0 ∈ C, xt ∈ C ∀t ≥0. The system is safe with respect to the set C if the set Cis forward invariant, i.e., the system state x always remainswithin the safe set C.

Definition 3 (Control Barrier Function (CBF) (Cheng et al.2019)). Given a set C ⊂ X ⊂ Rn, the continuously differen-tiable function h : X → R is a discrete-time control barrierfunction for dynamic system (1) if there exists η ∈ [0, 1]such that for all xt ∈ X ,

suput∈U

{h(xt+1) + (η − 1)h(xt)

}≥ 0, (3)

where η represents how strongly the barrier function”pushes” the state inwards within the safe set C. Whenη = 0, this condition is the Lyapunov condition.

For example, if the state x of a vehicle is always within thesafe set C, it means this vehicle always stay in some bound-ing box/range and will not collide with other obstacles. TheCBF is a condition to add the forward invariant property ofthe safe set C. Once x0 ∈ C, then xt ∈ C ∀t ≥ 0.

We want to find a policy in the HPOMDP to maximize itstotal expected return Gk =

∑Ki=k+1 γ

i−k−1ri where γ is adiscount factor and K is the time horizon. At the same time,there exists a controller that generates control inputs u ∈ Uto guarantee the underlying continuous state x ∈ C and thesafe set C is forward invariant. To achieve these goals, weuse a feedback deep Q-learning algorithm to learn the actionvalue function Q(o,m, a) and a feedback process to gener-ate feedback action a′ for the underlying controller to imple-ment. The policy to generate the feedback action π′(a′|o,m)can maximize the total expected return while guaranteeinga′ is safe, i.e., the underlying state x ∈ C during the learn-ing process. We use t and k to distinguish the underlyingcontroller time step and reinforcement learning time step.

Feedback Deep Q-LearningWe consider the challenge that the state transition functionT is not available. Traditional sample-based RL is good at

exploiting historical data to find a good policy. However, itdoes not guarantee either the optimality or the safety prop-erty of the converged policy. Different from the traditionalDRL, we design a feedback controller to the continuous vec-tor fieldF(·, a) to introduce a safe set for the continuous satex. In this section, we first introduce the feedback deep Q-learning algorithm and explain why the feedback does notlower the data efficiency. Then we introduce the quadraticprogramming (QP) formulated based on the CBF and provehow it induces a forward invariant set. We call the final pol-icy a feedback policy that generates actions implemented tothe environment, and the feedback policy π′(a′|o,m) has theability to utilize the learned action-value function and ensurethe safety during the learning process.

Algorithm

Environment

Agent

𝑅!"#

Controller

𝑂 !

𝑅!𝑟𝑒𝑤𝑎𝑟𝑑

Ob𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛

𝑎𝑐𝑡𝑖𝑜𝑛𝐴!

𝑂!"#

𝐴!$𝑓𝑒𝑒𝑑𝑏𝑎𝑐𝑘𝑎𝑐𝑡𝑖𝑜𝑛

Figure 2: The agent-environment interaction in the feedbackdeep reinforcement learning.

The agent-environment interaction is shown in Fig. 2 forthe feedback DRL. The DRL agent is like a smart toddlerwho is good at learning new knowledge but may do some-thing dangerous, such as eating inedible things and playingsharp items. In this case, an adult (the feedback controller)needs to look after her and guide her as a guardian. For thehybrid system, when the RL algorithm is exploring differentactions a ∈ A, the underlying state may have x ∈ X∧x /∈ C.For example, lane-changing may lead to collisions undersome traffic scenarios. The underlying controller will mon-itor action a from the DRL agent and determine feedbackaction a′ ∈ A based on the current traffic condition.

The process to get the feedback action is shown in Fig. 3.Once the behavior policy (e.g., the ε-greedy) selects anaction, the underlying CBF-QP-based controller evaluateswhether it is feasible to implement this action. This QP willbe defined in detail in the following subsection. If the CBF-QP has a feasible solution, it means a is safe, then a′ = a; ifthere is no solution, the controller will search other actionsin A in descending order according to their action-value Qand find a safe one. If all the actions in A are not safe in theworst case, then the underlying controller will implementthe Emergency Stop (ES) process.

Meanwhile, the transition experience is represented by(o,m, a′, r, o′,m′) in our HPOMDP, where o and m is thecurrent observation and shared information, a′ is the feed-back action, r is the reward, and o′ and m′ is the next ob-servation and shared information. This algorithm is shownin Alg. 1. This algorithm learns an action-value function Q∗by minimizing the Bellman loss:

L(θ) = Eo,m,a′,r,o′,m′ [(y −Q∗(o,m, a′; θ))2], (4)

Figure 3: Get the feedback action.where y = r + γ · maxaQ(o,m, a; θ). The correspondingpolicy can be derived as

πθ(a|o,m) = argmaxa

Q∗(o,m, a). (5)

Algorithm 1: Feedback deep Q-learning1 Initialize replay memory D with a capacity N ;2 Initialize action-value function Q with random

weights θ;3 Initialize target action-value function Q with θ = θ;4 for episode = 1, E do5 Initialize o ∈ O ∧m ∈M;6 for k = 1,K do7 Choose a ∈ A based on o,m and policy

derived from Q-function (e.g., ε-greedy);8 Give a to the feedback controller and get

feedback action a′;9 Take action a′, get r, o′,m′;

10 Store transition (o,m, a′, r, o′,m′) in D;11 Sample random a minibatch from D;12 Set target y = r + γ ·maxaQ(o,m, a; θ);13 Do gradient descent on (y −Q(o,m, a′; θ))2

with respect to θ;14 Assign Q = Q for some time steps;15 end16 end

Notably, the feedback does not lower the data efficiencyfor the RL. In traditional RL, the transition experience isrepresented by (o, a, r, o′). In feedback RL, the correspond-ing experience is represented by (o, a′, r, o′), where a′ is thefeedback action executed by the feedback controller. There-fore, the feedback does not slow down the growth of theexperience replay buffer.

Controller And Safety

We use standard basis vectors in Rn to denote each action,where n is the number of total actions in the action spaceA.The action an = (0, . . . , 0, 1, 0, . . . , 0)T is a vector equal tozero in every entry except for the nth entry, where it equalsto one. In other words, (an)i = 1 when i = n and (an)i = 0

when i 6= n. The action a = {an|an ∈ A} is selected basedon the policy πθ(a|o,m) learned by the Alg. 1.

Assemble the underlying controller’s references for eachaction as a matrix as follows.

U =

u11 u21 · · · un1u12 u22 · · · un2

......

. . ....

u1m u2m · · · unm

, (6)

where un1, un2, · · · , unm is the reference for action an. Thereference of a controller is also called the setpoint. It is atarget for the controller to track. The reference to track atrajectory for CAVs will be introduced in the next section.

In experiment, we consider an affine barrier function ofform h = pTx + q, (p ∈ Rn, q ∈ R). This restrictionmeans the set C is a polytope constructed by the intersect-ing of half spaces, since the safe area for a vehicle is usuallyrepresented by a bounding box (Cesari et al. 2017). By ap-plying a Frenet frame, the curve road can also be regardedas a straight road (Werling et al. 2010).

We formulate the following Quadratic Programming (QP)based on the CBF condition (3) that can be solved efficientlyat each time step:

argminut

‖ut − U · a‖2

s.t. pTf(xt) + pTg(xt)ut + q ≥ (1− η)h(xt).(7)

Lemma 1. If there exists η ∈ [0, 1] such that there exists asolution to the QP (7) for the dynamical system (1) for allxt ∈ C (defined in (2)), then the controller derived from (7)renders set C forward invariant.

Proof. For any xt ∈ C, we have h(xt) ≥ 0 according to thedefinition (2). Therefore,

h(xt+1) = pTf(xt) + pTg(xt)ut + q

≥ (1− η)h(xt) ≥ 0.(8)

Thus, xt+1 ∈ C and C is forward invariant.

In fact, we can relax the constraint in (7), and the forwardinvariant property holds for a larger set. Consider this QP:

argminut,ζ

‖ut − U · a‖2 +Mζ

s.t. pTf(xt) + pTg(xt)ut + q ≥ (1− η)h(xt)− ζζ ≥ 0,

(9)where M is a large constant.Theorem 1. If there exists η ∈ [0, 1] and Z ≥ 0 such thatthere exists a solution to the QP (9) for the dynamical system(1) for all xt ∈ C with ζ ≤ Z, then the controller derivedfrom (9) renders set C′ : {x ∈ Rn : h′(x) = h(x) + Z

η ≥ 0}forward invariant.

Proof. From Z ≥ ζ ≥ 0 ≥ −ηh(xt), we have h(xt) + Zη ≥

0. Thus, xt ∈ C′ with h′(xt) ≥ 0. Also, we have

h(xt+1) = pTf(xt) + pTg(xt)ut + q

≥ (1− η)h(xt)− ζ ≥ (1− η)h(xt)− Z,(10)

h(xt+1) +Z

η≥ (1− η)

[h(xt) +

Z

η

],

h′(xt+1) ≥ (1− η)h′(xt) ≥ 0.

(11)

Thus, xt+1 ∈ C′ and C′ is forward invariant. Note that itsimplifies to Lemma 1 with set C = C′ when Z = 0.

The value of Z denotes how large the CBF condition (3)is violated for the original h(xt). In this case, the safety con-dition should be formulated according to the set C′, which isforward invariant.

When dynamic system has noise in the model in reality:

xt+1 = f(xt) + g(xt)ut + wt. (12)

We can also find a forward invariant set as long as the noiseis bounded, i.e., ‖wt‖ ≤W . Consider the following QP:

argminut,ζ

‖ut − U · a‖2 +Mζ

s.t. pTf(xt) + pTg(xt)ut + q − pTW · 1≥ (1− η)h(xt)− ζζ ≥ 0.

(13)

Theorem 2. If there exists η ∈ [0, 1] and Z ≥ 0 such thatthere exists a solution to the QP (13) for the dynamical sys-tem (12) for all xt ∈ C with ζ ≤ Z and bounded noise‖wt‖ ≤ W , then the controller derived from (13) rendersset C′ : {x ∈ Rn : h′(x) = h(x) + Z

η ≥ 0} forward invari-ant.

Proof. Similar with the Theorem 2, xt ∈ C′ with h′(xt) ≥0. Also, we have

h(xt+1) = pTf(xt) + pTg(xt)ut + pTwt + q

≥ pTf(xt) + pTg(xt)ut + q − pTW · 1≥ (1− η)h(xt)− ζ ≥ (1− η)h(xt)− Z,

(14)

h(xt+1) +Z

η≥ (1− η)

[h(xt) +

Z

η

]. (15)

Thus, xt+1 ∈ C′ and C′ is forward invariant.

Behavior Planning of CAVsIn this section, we apply the feedback RL to the behaviorplanning problem of CAVs. We consider a scenario withautonomous vehicles running on a multi-lane freeway, asshown in Fig. 4.

Lane#1

Lane#2

Lane#3

Egovehicle

KeepLane

ChangeLeft

ChangeRight

EmergencyStop

Figure 4: The scenario of a 3-lane freeway. The ego vehicle’sbehavior is determined based on observation and shared in-formation.

The behavior planning process based on Alg.1 is shownin Fig. 5 for the CAV hybrid system. The discrete state tran-sition policy πθ(a|o,m) is learned by a RL agent with 2 Hz;

Figure 5: The concept diagram of behavior planning.the underlying continuous state is controlled by a CBF-QP(13) with 100 Hz. After the RL selects an action a, the con-trol reference will be calculated by trajectory tracking intro-duced in the following subsection. The CBF-QP will findpossible feasible control inputs to track the trajectory. If ais safe, then the feedback action a′ = a; if not, the feed-back action will be searched in descending order accordingto their action-value. If ∀a ∈ A, there is no feasible solutionof the CBF-QP, then the vehicle will implement the Emer-gency Stop (ES) process.

Physical Dynamic Model of an Ego VehicleThe physical dynamics of an ego vehicle is described by akinematic bicycle model that achieves a balance between ac-curacy and complexity (Brito et al. 2019). The discrete-timeequations of this model can be obtained by applying an ex-plicit Euler method with a sampling time Ts as follows:

xt+1 =

x1(t) + x4(t)cos(x3(t))Tsx2(t) + x4(t)sin(x3(t))Ts

x3(t)x4(t)

+

0 00 0

x4(t)Ts

L 00 Ts

ut,(16)

where x = [px, py, ψ, v]T ∈ X includes the two coordi-nates of the Center of Gravity (CoG), its orientation and itsvelocity at the CoG. The input u = [tanδ, v] ∈ U are thetangent of the steering angle and its acceleration. The pa-rameter L is the distance between the front and rear axle.Equation (16) is the vector field F(·, a).

Trajectory Tracking. For each action, we can use state-of-the-art trajectory planning methods to generate trajecto-ries σ = [σ1, σ2]T = [px, py]T, such as potential fields, celldecomposition, MPC (Dixit et al. 2018; Cesari et al. 2017).One can use endogenous transformation to compute the con-troller’s references to track the trajectories for each action:

u1 =σ1σ1 + σ2σ2√

σ21 + σ2

2

, u2 =σ1σ2 − σ2σ1(σ2

1 + σ22)

32

. (17)

Plugging them in equation (6), the references are gener-ated for the CBF-QP (13).

RL ModelAction. The set of actions A includes:• Keep Lane (KL): The ego vehicle stays in the current lane

with this action. It may either accelerate or decelerate.

• Change Left (CL): The ego vehicle changes to a neighborlane on the left. The incentive to change lane is to achievehigher speed or to avoid obstacles/traffic jams.

• Change Right (CR): The ego vehicle changes to a neigh-bor lane on the right.This action set is defined for the freeway scenario in

Fig. 4. Our method can also apply for other scenarios byextending the action set, for example, adding turn-left andturn-right at intersections.

Observation. The ego vehicle observes the environmentby its own sensors. The observation set O includes obser-vations from its own sensors, like radar, camera, and liar.Besides its sensors, we also add an observation for lane-changing frequency to count the total lane-changing timeswithin a fixed time window. The motivation is passengersmay feel uncomfortable if a vehicle changes lane frequently.Definition 4 (Observation for lane-changing frequency).The observation for lane-changing frequency is

O(k) = −Kf∑i=1

Ichange(k − i), (18)

where k is the current time instant, Kf is a constant deter-mining the window size of [k−Kf , k− 1] and the indicator

Ichange(i) =

{1, if lane-changing from time i;0, otherwise.

(19)

Shared Information. For CAVs, the ego vehicle can alsogather information from its peers by communication. Ac-cording to the current development of BSM and DSRCsecurity, the message can be both authenticated and en-crypted (Tangade, Manvi, and Lorenz 2018). Hence, inthis work, we assume that vehicular communication is trueinformation that is not manipulated by attackers. We as-sume autonomous vehicles to share information with theirε-neighbors defined as follows.Definition 5 (ε-neighbors). One vehicle j is one ε-neighborof a vehicle i if ‖pit − p

jt‖ ≤ ε, where i and j are vehicles’

indexes, pit and pjt represent the positions of vehicle i andj at time t respectively, and ε is a constant. The set that in-cludes all the ε-neighbors of the vehicle i except for i itself isdenoted as Ni(ε). This set is called vehicle i’s ε-neighbors.

In this application, we consider autonomous vehicles toshare their current velocity vt and lane number lt (labeledas 1, 2, 3) with Ni(ε). When the reinforcement learning ismaking decision at step k, the ego vehicle uses the most re-cently information received from other neighbors. The near-est information determine vk and lk. One problem is the to-tal number of shared information increases proportionally to| Ni(ε) |. To reduce the curse of dimensionality, we definethe shared information for the average velocity of lane #l as:Definition 6 (Shared information for the average velocity).The shared information for the average velocity of lane #l is

M(k, l) =1

N

∑i:i∈Nl

vik, (20)

where Nl = {i|i ∈ Ni(ε), lik = l}, N =| Nl |, lik ∈{1, 2, 3}, vik is the velocity of the vehicle i at time k.

This function is the average velocity of CAVs on lane #l. It helps to avoid unnecessary lane-changing. For example,even though the close vehicles on the neighbor lane havea higher speed than the ego vehicle, there may be a trafficjam in front of them. The ego vehicle could not observe thistraffic jam because it cannot see through its neighbors. Inthis case, there is no need to change lane.

Based on this definition, the shared information setM ={M(k, l−1),M(k, l),M(k, l+1)}, where the lane #(l−1)and #(l + 1) is the left and right neighbor lane respectively.

Reward. After taking an action, the ego vehicle gets a re-ward from the environment. We consider a behavior plan-ning policy to improve two system-level evaluation criteria:traffic flow F and driving comfort C. When there is a cen-tral cloud center to collect information from all the agents,a global reward function can be used for the MARL (Waiet al. 2018). Therefore, the reward function is defined as aweighted sum of the above two criteria:Definition 7 (Reward function). The reward function is

r(s, a) = w · F + C, (21)

where s ∈ S, a ∈ A, w is a trade-off weight.

• Traffic Flow: The traffic flow reflects the quality of theroad throughout with respect to the traffic density. Thetraffic density ρ is the ratio between the total number ofvehicles and the road length. It is calculated as

F = ρ× v, (22)

where v is the average velocity of all the vehicles (Rios-Torres and Malikopoulos 2018).

• Driving Comfort: The driving comfort of a road segmentis defined to be the average driving comfort of all the ve-hicles on this segment. The driving comfort of a single-vehicle is related to its acceleration and driving behavior.Define the driving comfort for the vehicle i’s accelerationvit at time t and its action a ∈ A as follows:

Csingle(vit, a) =

3, if vit < Θ and a = KL;

2, if vit ≥ Θ and a = KL;

1, if a = CL/CR;

0, if in ES.

(23)

where Θ is a predefined acceleration threshold. Therefore,driving comfort (for the freeway) is calculated as

C =1

N · T∑i

∑t

Csingle(vit, a), (24)

where N is the total number of all the vehicles.

ExperimentIn experiments, vehicles are randomly scattered on differ-ent lanes of a 1000-long freeway as their initial positions.The total number of vehicles ranges from 100 to 900. Allvehicles loop on this freeway. For different traffic densities,

experiments run for 4000 time steps. Traffic flow and driv-ing comfort are evaluated based on the statistics in the last1000 time steps. Also, the experiment runs 30 times underdifferent initialization for each traffic density.

Freeway ScenarioWe compare the traffic flow and driving comfort under dif-ferent traffic densities between the feedback RL agent usingAlg. 1 and an intelligent driving model (IDM). In this IDM,the vehicle’s acceleration is a function of its current speed,current and desired spacing, and the leading and followingvehicles’ speed (Talebpour and Mahmassani 2016). Build-ing on top of these IDM agents, we add lane-changing func-tionality using the gap acceptance method by consideringthe positions of the leading vehicles in the current and targetlane and the following vehicle on the target lane (Butakovand Ioannou 2014).

0.2 0.4 0.6 0.8

Traffic density

0.5

0.6

0.7

0.8

0.9

1

Tra

ffic

flo

w

IDM

RL

0.2 0.4 0.6 0.8

Traffic density

2.8

2.85

2.9

2.95

3

Drivin

g c

om

fort

IDM

RL

Figure 6: The comparison between feedback RL and IDM.As shown in Fig. 6, the feedback RL agent gets both larger

traffic flow and better driving comfort when traffic density ρis low. When ρ grows, the result of the RL agent gets worse,but it is still comparable with the IDM. When the road is sat-urated, lane-changing tends to downgrade passengers’ com-fort but cannot bring higher speed. Consequently, a betterchoice is to keep a lane when ρ is high, and there is no sig-nificant difference between the RL and the IDM.

Table 1 shows the number of collisions avoided by thefeedback process. It is also the number of actions altered bythe feedback where a′! = a. This number is averaged overthe last 10 episodes, which has a maximum epoch of 2000.Without the feedback process, each episode terminates veryearly in the first few epochs because of a collision. For ex-ample, when ρ = 0.1, one episode can be lucky to run for18 epochs after 2000 episodes. With the feedback process,each episode can run safely for the maximum epoch.Table 1: Total number of collisions avoided by the feedbackprocess.

ρ 0.1 0.2 0.3 0.4 0.5# 4416 9510 21897 43491 61689

ρ 0.6 0.7 0.8 0.9 -# 91154 133135 191404 242976 -

The average headway of all the vehicles is shown in thefirst 500 epochs of one episode when ρ = 0.6 in Fig. 7. Wesee that the headway of vehicles is always greater than 0 inthe feedback RL with the minimum value of 3.33. Neverthe-less, the vanilla RL without the feedback process is likelyto have a negative headway, which will cause collisions in

reality.

0 50 100 150 200 250 300 350 400 450 500

Time step

-20

-15

-10

-5

0

5

10

He

ad

wa

y

vanilla RL

feedback RL

Figure 7: The comparison between feedback RL and vanillaRL without feedback.

Road Closure Scenario

Lane#1

Lane#2

Figure 8: The road closure scenario with two closures onlane #1.

This experiment shows a case study where there are tworoad closures on one lane. As shown in Fig. 8, the road clo-sures may be caused by accident, or they are under construc-tion. These road closures both have a length of 200. We wantto test whether the learning algorithm can work in a morecomplicated scenario. CAVs are assumed to know the po-sition of the road closures when it reaches the ε-neighborrange of these segments. They can share this informationwith its ε-neighbors (add as additional information to beshared in this case) or gather this information through V2Icommunication from infrastructure.

0.2 0.4 0.6 0.8

Traffic density

0.4

0.42

0.44

0.46

0.48

0.5

Tra

ffic

flo

w

IDM

RL

0.2 0.4 0.6 0.8

Traffic density

2.94

2.95

2.96

2.97

2.98

2.99

3

Drivin

g c

om

fort

IDM

RL

Figure 9: Feedback RL in the road closure scenario.

The result is shown in Fig. 9. The feedback RL agent getsa comparable traffic flow while improving driving comfort alot. Though the feedback RL agent does not use the lane withclosures to overtake and get higher speed, it avoids massiveacceleration to provide better driving comfort.

ConclusionThis paper focuses on the behavior planning challenge forCAVs, and designs a novel policy learning algorithm thatconsiders feedback from controllers and safety require-ments with shared information to enhance operational per-formance. We model a hybrid partially observable Markovdecision process (HPOMDP) for this hybrid system and de-sign a feedback DRL framework to explore a safe behaviorplanning policy for both discrete and continuous dynamics.Our method combines the advantages of both DRL and CBF

and guarantees a safe learning process for this physical sys-tem. Though our method for the HPOMDP is proposed forthe CAVs, it is applicable to other hybrid systems where thediscrete transition policy is explored by the RL and the un-derlying controller renders a forward invariant safe set. Moreapplications using this method can be investigated in the fu-ture. The recent advances in multi-agent RL can be used tofurther improve the result, like MADDPG and league train-ing.

ReferencesAmes, A. D.; Coogan, S.; Egerstedt, M.; Notomista, G.;Sreenath, K.; and Tabuada, P. 2019. Control barrier func-tions: Theory and applications. In 2019 18th European Con-trol Conference (ECC), 3420–3431. IEEE.

Amini, A.; Gilitschenski, I.; Phillips, J.; Moseyko, J.; Baner-jee, R.; Karaman, S.; and Rus, D. 2020. Learning RobustControl Policies for End-to-End Autonomous Driving FromData-Driven Simulation. IEEE Robotics and AutomationLetters 5: 1–1. doi:10.1109/LRA.2020.2966414.

Bansal, M.; Krizhevsky, A.; and Ogale, A. S. 2018. Chauf-feurNet: Learning to Drive by Imitating the Best and Syn-thesizing the Worst. CoRR abs/1812.03079. URL http://arxiv.org/abs/1812.03079.

Brito, B.; Floor, B.; Ferranti, L.; and Alonso-Mora, J.2019. Model Predictive Contouring Control for CollisionAvoidance in Unstructured Dynamic Environments. IEEERobotics and Automation Letters 4(4): 4459–4466.

Butakov, V. A.; and Ioannou, P. 2014. Personalizeddriver/vehicle lane change models for ADAS. IEEE Trans-actions on Vehicular Technology 64(10): 4422–4431.

Cesari, G.; Schildbach, G.; Carvalho, A.; and Borrelli, F.2017. Scenario model predictive control for lane changeassistance and autonomous driving on highways. IEEE In-telligent Transportation Systems Magazine 9(3): 23–35.

Chen, S.; Saulnier, K.; Atanasov, N.; Lee, D. D.; Kumar, V.;Pappas, G. J.; and Morari, M. 2018. Approximating explicitmodel predictive control using constrained neural networks.In 2018 Annual American control conference (ACC), 1520–1527. IEEE.

Cheng, R.; Orosz, G.; Murray, R. M.; and Burdick, J. W.2019. End-to-end safe reinforcement learning through bar-rier functions for safety-critical continuous control tasks. InProceedings of the AAAI Conference on Artificial Intelli-gence, volume 33, 3387–3395.

Codevilla, F.; Miiller, M.; Lopez, A.; Koltun, V.; and Doso-vitskiy, A. 2018. End-to-end driving via conditional imita-tion learning. In 2018 IEEE International Conference onRobotics and Automation (ICRA), 1–9. IEEE.

Dixit, S.; Fallah, S.; Montanaro, U.; Dianati, M.; Stevens,A.; Mccullough, F.; and Mouzakitis, A. 2018. Trajectoryplanning and tracking for autonomous overtaking: State-of-the-art and future prospects. Annual Reviews in Control 45:76–86.

Gritschneder, F.; Graichen, K.; and Dietmayer, K. 2018.Fast Trajectory Planning for Automated Vehicles usingGradient-based Nonlinear Model Predictive Control. In2018 IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS), 7369–7374. IEEE.

Guanetti, J.; Kim, Y.; and Borrelli, F. 2018. Control of con-nected and automated vehicles: State of the art and futurechallenges. Annual Reviews in Control 45: 18 – 40. ISSN1367-5788. doi:https://doi.org/10.1016/j.arcontrol.2018.04.011. URL http://www.sciencedirect.com/science/article/pii/S1367578818300336.

Henaff, M.; LeCun, Y.; and Canziani, A. 2019. Model-predictive policy learning with uncertainty regularization fordriving in dense traffic. In 7th International Conference onLearning Representations (ICLR).

Jang, K.; Vinitsky, E.; Chalaki, B.; Remer, B.; Beaver, L.;Malikopoulos, A. A.; and Bayen, A. 2019. Simulationto Scaled City: Zero-Shot Policy Transfer for Traffic Con-trol via Autonomous Vehicles. In Proceedings of the 10thACM/IEEE International Conference on Cyber-PhysicalSystems, ICCPS ’19, 291–300. New York, NY, USA: Asso-ciation for Computing Machinery. ISBN 9781450362856.doi:10.1145/3302509.3313784. URL https://doi.org/10.1145/3302509.3313784.

Kenney, J. B. 2011. Dedicated Short-Range Communica-tions (DSRC) Standards in the United States. Proceedingsof the IEEE 99(7): 1162–1182. ISSN 0018-9219. doi:10.1109/JPROC.2011.2132790.

Lee, J.; and Park, B. 2012. Development and Evaluation of aCooperative Vehicle Intersection Control Algorithm Underthe Connected Vehicles Environment. IEEE Transactionson Intelligent Transportation Systems 13(1): 81–90. ISSN1524-9050. doi:10.1109/TITS.2011.2178836.

Li, B.; Zhang, Y.; Ge, Y.; Shao, Z.; and Li, P. 2017. Op-timal control-based online motion planning for cooperativelane changes of connected and automated vehicles. In 2017IEEE/RSJ International Conference on Intelligent Robotsand Systems (IROS), 3689–3694. IEEE.

Liang, K. Y.; Martensson, J.; and Johansson, K. H. 2016.Heavy-Duty Vehicle Platoon Formation for Fuel Efficiency.IEEE Transactions on Intelligent Transportation Systems17(4): 1051–1061. ISSN 1524-9050. doi:10.1109/TITS.2015.2492243.

Lowe, R.; Wu, Y. I.; Tamar, A.; Harb, J.; Abbeel, O. P.;and Mordatch, I. 2017. Multi-agent actor-critic for mixedcooperative-competitive environments. In Advances in neu-ral information processing systems, 6379–6390.

Nilsson, P.; Hussien, O.; Balkan, A.; Chen, Y.; Ames, A. D.;Grizzle, J. W.; Ozay, N.; Peng, H.; and Tabuada, P. 2016.Correct-by-Construction Adaptive Cruise Control: Two Ap-proaches. IEEE Transactions on Control Systems Technol-ogy 24(4): 1294–1307. ISSN 1063-6536. doi:10.1109/TCST.2015.2501351.

Ort, T.; Paull, L.; and Rus, D. 2018. Autonomous vehi-cle navigation in rural environments without detailed prior

maps. In 2018 IEEE International Conference on Roboticsand Automation (ICRA), 2040–2047. IEEE.Pan, X.; You, Y.; Wang, Z.; and Lu, C. 2017. Virtual toreal reinforcement learning for autonomous driving. arXivpreprint arXiv:1704.03952 .Rios-Torres, J.; and Malikopoulos, A. A. 2017. A Surveyon the Coordination of Connected and Automated Vehiclesat Intersections and Merging at Highway On-Ramps. IEEETransactions on Intelligent Transportation Systems 18(5):1066–1077. ISSN 1524-9050. doi:10.1109/TITS.2016.2600504.Rios-Torres, J.; and Malikopoulos, A. A. 2018. Impact ofpartial penetrations of connected and automated vehicles onfuel consumption and traffic flow. IEEE Transactions onIntelligent Vehicles 3(4): 453–462.Shalev-Shwartz, S.; Shammah, S.; and Shashua, A. 2016.Safe, multi-agent, reinforcement learning for autonomousdriving. arXiv preprint arXiv:1610.03295 .Talebpour, A.; and Mahmassani, H. S. 2016. Influence ofconnected and autonomous vehicles on traffic flow stabilityand throughput. Transportation Research Part C: EmergingTechnologies 71: 143–163.Tangade, S.; Manvi, S. S.; and Lorenz, P. 2018. Decentral-ized and scalable privacy-preserving authentication schemein VANETs. IEEE Transactions on Vehicular Technology67(9): 8647–8655.Vinyals, O.; Babuschkin, I.; Czarnecki, W. M.; Mathieu, M.;Dudzik, A.; Chung, J.; Choi, D. H.; Powell, R.; Ewalds,T.; Georgiev, P.; et al. 2019. Grandmaster level in Star-Craft II using multi-agent reinforcement learning. Nature575(7782): 350–354.Wai, H.-T.; Yang, Z.; Wang, Z.; and Hong, M. 2018. Multi-agent reinforcement learning via double averaging primal-dual optimization. In Advances in Neural Information Pro-cessing Systems, 9649–9660.Werling, M.; Ziegler, J.; Kammel, S.; and Thrun, S. 2010.Optimal trajectory generation for dynamic street scenariosin a frenet frame. In 2010 IEEE International Conferenceon Robotics and Automation, 987–993. IEEE.Zhang, K.; Yang, Z.; and Basar, T. 2019. Multi-Agent Rein-forcement Learning: A Selective Overview of Theories andAlgorithms. arXiv preprint arXiv:1911.10635 .Zhang, T.; Kahn, G.; Levine, S.; and Abbeel, P. 2016. Learn-ing deep control policies for autonomous aerial vehicleswith MPC-guided policy search. In 2016 IEEE InternationalConference on Robotics and Automation (ICRA), 528–535.ISSN null. doi:10.1109/ICRA.2016.7487175.