inverse optimal heuristic control for imitation...

424

Inverse Optimal Heuristic Control for Imitation Learning

Nathan Ratliff, Brian Ziebart, Kevin Peterson,J. Andrew Bagnell, Martial Hebert, Anind K. Dey

Robotics Institute, MLD, CSDCarnegie Mellon University

Pittsburgh, PA 15213

Siddhartha SrinivasaIntel Research

Pittsburgh, PA 15213

Abstract

One common approach to imitation learning isbehavioral cloning (BC), which employs straight-forward supervised learning (i.e., classification)to directly map observations to controls. A sec-ond approach is inverse optimal control (IOC),which formalizes the problem of learning se-quential decision-making behavior over long hori-zons as a problem of recovering a utility func-tion that explains observed behavior. This pa-per presents inverse optimal heuristic control(IOHC), a novel approach to imitation learn-ing that capitalizes on the strengths of bothparadigms. It employs long-horizon IOC-stylemodeling in a low-dimensional space where in-ference remains tractable, while incorporating anadditional descriptive set of BC-style features toguide a higher-dimensional overall action selec-tion. We provide experimental results demon-strating the capabilities of our model on a sim-ple illustrative problem as well as on two realworld problems: turn-prediction for taxi drivers,and pedestrian prediction within an office envi-ronment.

1 Introduction

Imitation learning is an important tool for modelingdecision-making agents (Ziebart et al., 2008b) as wellas for crafting automated decision making algorithms,particularly in robotics (Pomerleau, 1988; LeCun etal., 2006; Ratliff et al., 2006; Silver, Bagnell, & Stentz,2008; Kolter, Abbeel, & Ng, 2007). The majority ofimitation learning techniques rely on an approach wewill term behavioral cloning (BC) (Bain & Sammut,1995), in which one attempts to predict actions di-rectly from an observed feature vector that describesthe environment. This approach has been applied suc-cessfully in a number of applications. Perhaps themost well-known of these applications is the problem oflearning to steer a vehicle within a road lane or around

Appearing in Proceedings of the 12th International Confe-rence on Artificial Intelligence and Statistics (AISTATS)2009, Clearwater Beach, Florida, USA. Volume 5 of JMLR:W&CP 5. Copyright 2009 by the authors.

Figure 1: Left: Obstacles in the roadway that could in-fluence a driver’s actions in a behavioral cloning approach.Right: The inverse optimal control setting for navigating(cyan line) to a destination (red X). Driver behavior is bestexplained as a combination of these two approaches.

obstacles (Figure 1) using camera images (Pomerleau,1988; LeCun et al., 2006).

An alternate approach that has recently been suc-cessful for achieving coherent decision making– es-pecially sequential decisions over multiple time-stepsabout distant future goals– has been to treat the imita-tion learning problem as one of inverse optimal control(IOC). Here the goal is to take training examples in theform of trajectories through the state space (Figure 1)and train a planning algorithm to predict the same setof decisions. Moreover, these algorithms train plan-ning behavior that generalizes to new domains unseenduring training. They demonstrate strong empiricaland theoretical success on a wide variety of problemsranging from taxi route preference modeling and driverbehavior modeling (Ziebart et al., 2008a; Abbeel & Ng,2004) to heuristic learning for legged locomotion andmobile robot navigation through rough terrain (Ratliffet al., 2006; Ratliff, Bagnell, & Zinkevich, 2006; Silver,Bagnell, & Stentz, 2008; Kolter, Abbeel, & Ng, 2007).

In this work, we integrate these two previously dis-parate forms of imitation learning to engage the ca-pabilities of both paradigms. The success of inverseoptimal control centers around its ability to move be-yond a one-step look-ahead paradigm and to learn apolicy that correctly predicts entire sequences of ac-tions. When applicable, these algorithms substantiallyoutperform alternative techniques such as behavioral

425


cloning. Unfortunately, as the dimensionality of thestate space increases, the corresponding Markov Deci-sion Process (MDP) becomes increasingly difficult tosolve, severely limiting the application of these tech-niques. By contrast, behavioral cloning methods ben-efit from the scalability of existing supervised learningalgorithms to handle high-dimensional feature spaces,but they ignore modern planning technologies andmake decisions without explicitly considering their fu-ture consequences.

We frame the training of our combined model as anoptimization problem. Although the resulting objec-tive function is non-convex, we present a collectionof convex approximations that may be optimized assurrogates. Further, we demonstrate both empiricallyand theoretically that the objective function is nearlyconvex. Optimizing it directly leads to improved per-formance across a range of imitation learning prob-lems. Section 5 begins by illustrating the theoreti-cal properties of our algorithm on a simple problem.We then demonstrate the algorithm and compare itsperformance to previous approaches on a taxi routeprediction problem (Section 5.2) and on a pedestrianprediction problem (Section 5.3) using real-world datasets.

Prior work has previously examined combining aspectsof behavioral cloning and inverse optimal control (un-der the names direct and indirect approaches) (Neu &Szepesvari, 2007), however, the authors focus on onlythe relationships between the loss-functions typicallyconsidered under the two approaches. The techniquesdescribed in that paper remain limited by the low-dimensionality restrictions of inverse optimal control.Formally, similar to previous work (Neu & Szepesvari,2007; Ramachandran & Amir, 2007; Ziebart et al.,2008a), our technique fits a Gibbs/Maximum Entropymodel over actions based on features in the environ-ment. In this paper, however, we take a direct ap-proach to training and propose to use our Gibbs-basedmodel to learn a stochastic policy that predicts theprobability that the expert takes each action given anobservation.

2 Inverse optimal heuristic control

In this section, we examine a relationship between be-havioral cloning and inverse optimal control that be-comes clear through the use of Gibbs distributions.After discussing this relationship in Section 2.1, wepropose a novel Gibbs model in Section 2.2 that com-bines the strengths of these individual models. Section2.3 presents an efficient gradient-based learning algo-rithm for fitting this model to training data.

2.1 Gibbs models for imitation learning

Recent research in inverse optimal control has intro-duced a Gibbs model of action selection in whichthe probability of taking an action is inversely pro-portional to that action’s exponentiated Q-value in

the MDP (Neu & Szepesvari, 2007; Ramachandran &Amir, 2007). Denoting the immediate cost of takingan action a from state s as c(s, a), and the cost-to-go from a state s′ as J(s′), we can write Q∗(s, a) =c(s, a) + J(T a

s ), where T as denotes the deterministic

transition function.1 The Gibbs model is therefore

p(a|s) =e−c(s,a)−J(T a

s )∑a′∈As

e−c(s,a′)−J(T a′s )

, (1)

where the function Q∗(s, a) = c(s, a)+J(T as ) is known

as the energy function of the Gibbs model.

The form of this inverse optimal control model is strik-ingly similar to a multi-logistic regression classifica-tion model (Nigam, Lafferty, & McCallum, 1999). Wearrive at a straight-forward behavioral cloning modelbased on multi-logistic regression by simply replacingthe energy function with a linear combination of fea-tures: E(s, a) = wT f(s, a), where f(s, a) denotes afunction mapping each state-action pair to a represen-tative vector of features.

2.2 Combining inverse optimal control andbehavioral cloning

The observations presented in Section 2.1 suggest thata natural way to combine these two models is to designan energy function that utilizes the strengths of bothparadigms. While inverse optimal control has demon-strated better generalization than behavioral cloningin real-world settings (Ratliff, Bagnell, & Zinkevich,2006), the technique’s applicability is limited by itsreliance on an MDP solver. On the other hand, therange of application for behavioral cloning has histor-ically surpassed the modeling capacity of MDPs.

To prevent needlessly restricting our model, we con-sider a general class of policies in which the agent sim-ply maps observations to actions p(a|o) ∝ e−

eE(o,a),where a ∈ A is an arbitrary action and o ∈ O is agiven observation.

In many cases, there exists a problem specific MDPM that can model a subproblem of the decision pro-cess. Let SM and AM be the state and actionspaces of our lower-dimensional MDP M. We re-quire a mapping φ(o, a) from an observation-actionpair to a sequence of state-action pairs that repre-sents the behavior exhibited by the action throughthe lower-dimensional MDP. Specifically, we denoteφ(o, a) = {(s1, a1), (s2, a2), . . .} and use T a

o to indicatethe state resulting from following the action-trajectoryφ(o, a). Throughout the paper, we denote the cost ofa trajectory ξ through this MDP as C(ξ), although forconvenience we often abbreviate C(φ(o, a)) as C(o, a).

Our combined Gibbs model, in this setting, uses thefollowing energy function

E(o, a) = E(o, a) + Q∗M(o, a),

1In this paper, we restrict ourselves to deterministicMDPs for modeling the lower-dimensional problem.

426

Ratliff, Ziebart, Peterson, Bagnell, Hebert, Dey, Srinivasa

where Q∗M(o, a) = C(o, a) + JM(T a

o ) denotes thecumulative cost-to-go of taking the short action-trajectory φ(o, a) and then following the minimum costpath from the resulting state to the goal. Intuitively,the learning procedure chooses between BC and IOCparadigms under this model, or finds a combination ofthe two that better represents the data.

In what follows, we denote a trajectory as a sequenceof observation-action pairs ξ = {(ot, at)}

Tξ

t=1, the setof all such trajectories starting from a given observa-tion o as Ξo, and the set of actions that can be takengiven a particular observation o as Ao. Sections 3 and4 present some theoretical results for the Gibbs IOCmodel. In those sections, we refer to only a singleMDP, and we can talk about states in place of obser-vations. When applicable, we replace the observationo with a state s in this notation.

Choosing linear parameterizations of both terms of theenergy function we can write our model as

p(a|s) =e−wT

v fv(o,a)−wTc F∗M(o,a)∑

a′∈A e−wTv fv(o,a)−wT

c F∗M(o,a), (2)

where we define FM(ξ) =∑

t fc(st, at) as the sum ofthe feature vectors encountered along trajectory ξ, anddenote

F ∗M(o, a) = FM(φ(o, a)) +

∑(st,at)∈ξ∗

fc(st, at),

where ξ∗ is the optimal trajectory starting from T ao

and fc(s, a) denotes a feature vector associated withstate-action pair (s, a). Below we use w to denote thecombined set of parameters (wv, wc). This notationutilizes the common observation that for linear costs,the cumulative cost-to-go of a trajectory through anMDP is a linear combination of the cumulative featurevector (Ng & Russell, 2000). Choosing wv = 0 resultsin a generalized Gibbs model for IOC in which eachaction may be a small trajectory through the MDP.

We often call fv(s, a) and fc(s, a) value features andcost features respectively because of their traditionaluses for value function approximation and cost param-eterization in the separate behavioral cloning and in-verse optimal control models.

2.3 Gradient-based optimization

Following the tradition of multi-logistic regressionfor behavioral cloning, given a trajectory ξ ={(ot, at)}Tt=1, we treat each observation-action pair asan independent training example and optimize thenegative log-likelihood of the data. Given a set oftrajectories D = {ξi}Ni=1, the exponential form of ourGibbs distribution allows us to write our objective

l(D;wv, wc) = − log∏N

i=1 p(ξi) as

l(D;wv, wc) =N∑

i=1

Ti−1∑t=1

wTv fv(ot, at) + wT

c F ∗M(ot, at)

+ log∑a∈A

e−wTv fv(ot,a)−wT

c F∗M(ot,a) +λ

2‖w‖2, (3)

where we denote Ti = Tξifor convenience; λ ≥ 0 is

a regularization constant. In our gradient expressionsand discussion below, we suppress the regularizationterm for notational convenience.

This objective function is piecewise-differentiable; atpoints of differentiability the following formula givesits gradient in terms of both wv and wc:

∇wvl(D) =

N∑i=1

Ti−1∑t=1

fv(ot, at)− Epw(a|ot)[fv(ot, a)]

∇wcl(D) =

N∑i=1

Ti−1∑t=1

F ∗M(ot, at)− Epw(a|ot)[F

∗M(ot, a)],

where we use pw(a|o) ∝ exp{−wTv fv(o, a) −

wTc F ∗

M(s, a)} to denote policy under our Gibbs modelparameterized by the combined vector of parametersw = (wv;wc). At points of nondifferentiability, thereare multiple optimal paths through the MDP; choos-ing any one of them in the above formula results ina valid subgradient. Algorithm 1 presents a simpleoptimization routine based on exponentiated gradientdescent for optimizing this objective.

3 On the efficient optimization ofinverse optimal heuristic control

Our experiments indicate that the simple gradient-based procedure of Section 2.3 is robust to variationsin starting point, a property often reserved for convexfunctions. This section demonstrates that, in manyways, our objective closely resembles a convex func-tion. We first note that for any fixed wc the objectiveas a function of wv is convex. Moreover, we demon-strate below that for wv = 0, the resulting (general-ized) IOC Gibbs model is almost-convex in a rigoroussense. In combination, these results suggest that aneffective strategy for optimizing our model is to:

1. Set wv = 0 and optimize the almost-convex IOCGibbs model.

2. Fix wc and optimize the resulting convex problemin wv.

3. Further optimize wc and wv jointly.

Optionally, in (2), one may use the fixed Q-values asfeatures thereby guaranteeing that the resulting modelcan only improve over the Gibbs model found in step

427


Algorithm 1 Optimization of the negative log-likelihood via exponentiated gradient descent

1: procedure Optimize( D = {(ξi,Mi)}Ni=1 )2: Initialize log-parameters wl

v ← 0 and wlc ← 0

3: for k = 0, . . . ,K do4: Initialize gv = 0 and gc = 05: for i = 1, . . . , N do6: Set wv = ewl

v and wc = ewlc

7: Construct cost map csai = wT fc(s, a) for

our low-dimensional MDPMi

8: Compute cumulative feature vectorsF ∗M(o, a) through MDPMi for each

potential action from each observa-tion found along the trajectory

9: for t = 1, . . . , Ti do10: gt

v ← fv(ot, at)− Epw(a|ot)[fv(ot, a)]11: gt

c ← FMi(ot, at)−Epw(a|ot)[F

∗Mi

(ot, a)]12: end for13: end for14: Update wl

v ← wlv − αk

∑Ti

t=1 gtv and

wlc ← wl

v − αk

∑Ti

t=1 gtc

15: end for16: return Final values wv = ewl

v and wc = ewlc

17: end procedure

(1). The final joint optimization phase can then onlyadditionally improve over the model found in (2).

In what follows, we derive our almost-convexity resultsstrictly in terms of the Gibbs model for IOC. Similararguments can be used to prove analogous results forour generalized IOC Gibbs model as well.

Definition 3.1: Almost-convexity. A functionf(x) is almost-convex if there exists a constant c ∈ Rand a convex function h(x) such that h(x) ≤ f(x) ≤h(x) + c for all x in the domain.

The notion of almost-convexity formalizes the intu-ition that the objective function may exhibit the gen-eral shape of a convex function, while not necessarilybeing precisely convex. In what we show below, thenegative log-likelihood of the Gibbs model for IOC islargely dominated by a commonly seen function inthe machine learning literature known as the percep-tron objective.2 The nonconvexities of the negativelog-likelihood arise from a collection of bounded dis-crepancy terms that measure the difference betweenthe hard-min and the soft-min functions.

2We call this function the perceptron objective becausethe perceptron algorithm that has been used in the past forvarious structured prediction problems (Collins & Roark,2004) is a particular subgradient algorithm for optimizingthe function. We note, however, that technically, as anobjective, this function is degenerate in the sense that itis successfully optimized by the zero function. Many ofthe properties of the perceptron algorithm cited in the lit-erature are specific to that particular algorithm, and notgeneral properties of this objective.

The negative log-likelihood of an example trajectoryξ, under this model is

− log p(ξ) = −T−1∑t=1

loge−Q∗(st,at)∑

a∈Aste−Q∗(st,a)

(4)

=T−1∑t=1

Q∗(st, at) + log∑

a∈Ast

e−Q∗(st,a)

.

By expanding Q∗(s, a) = c(s, a) + J(T as ), and then

pushing the sum through, we can write this as

− log p(ξ) = C(ξ) +T−1∑t=1

J(T atst

) + log∑

a∈Ast

e−J(st,a)

,

where we denote the cumulative cost of a path asC(ξ) =

∑T−1t=1 c(st, at).

By noting that T atst

= st+1 (taking action at from statest gets us to the next state along the trajectory in adeterministic MDP), we rewrite the second term as

T−1∑t=1

J(T atst

) =T∑

t=2

J(st) = −J(s1) +T−1∑t=1

J(st).

For the final simplification, we added and subtractedJ(s1) = minξ∈Ξs1

C(ξ) and use the fact that the cu-mulative cost-to-go of the goal state sT is zero (i.e.,J(sT ) = 0). We additionally note that J(s) =mina∈As

c(s, a) + J(T as ). Our negative log-likelihood

expression therefore simplifies to

− log p(ξ) = C(ξ)− minξ∈Ξs1

C(ξ)

+T−1∑t=1

mina∈Ast

Q∗(st, a) + log∑

a∈Ast

e−Q∗(st,a)

.

Finally, if we denote the soft-min function bymins

a∈AsQ∗(s, a) = − log

∑a∈As

e−Q∗(s,a), we canwrite the negative log-likelihood of a trajectory as

− log p(ξ) =C(ξ)− minξ∈Ξs1

C(ξ) (5)

+T−1∑t=1

min∆a∈Ast

Q∗(st, a).

where we use the notation min∆i ci = mini ci −mins

i ci

to denote the discrepancy between the hard- and soft-min operators over a set of values {ci}.When the cost function is defined as a linear com-bination of features, the cost function itself is linear,and the Q∗ function, as a min over linear functions,is therefore concave (making −Q∗ convex). Thus, thefirst two terms are both convex. In particular, theyform the convex structured perceptron objective. In-tuitively, these terms contrast the cumulative cost-to-go of the given trajectory with the minimum hypoth-esized cost-to-go over all trajectories.

428


The non-convexity of the negative log-likelihood ob-jective arises from the final set of terms in Equation5. Each of these terms simply denotes the differencebetween the soft-min function and the hard-min func-tion. We can bound the absolute difference betweenthe hard- and soft-min by log n, where n is the num-ber of elements over which the min operates. In ourcase, this means that each hard/soft-min discrepancyterm can contribute no more to the objective than theconstant value log |A|. In particular, we can state thefollowing

Theorem 3.2: Gibbs IOC is almost-convex. De-note the negative log-likelihood of ξ by f(w) and leth(w) = C(ξ) − minξ∈Ξs1

C(ξ). If n ≥ |As| for alls and T = |ξ| is the length of the trajectory, thenh(w) ≤ f(w) ≤ h(w) + |ξ| log n.

Proof. The soft-min is everywhere strictly less thanthe hard-min. Each discrepancy term min∆

a∈AstQ∗(st, a)

is therefore positive. Thus, h(w) ≤ f(w). Additionally, weknow that mini ci−log n ≤ − log

Pi e−ci for any collection

{ci}ni=1. For a discrepancy term, this bound gives

min∆i ci = min

ici + log

Xi

e−ci

≤ mini

ci − (mini

ci − log n) = log n.

Applying this upper bound to our objective gives the de-sired result. 2

Section 5.1 presents some simple experiments illustrat-ing these results.

4 Convex approximations

The previous section demonstrates that the general-ized Gibbs model for inverse optimal control is almost-convex and suggests that directly optimizing the ob-jective following algorithm 1 will work well on a widerange of problems. While our experimental resultssupport this analysis, without a proof of (approxi-mate) global convergence, we cannot guarantee theseobservations to hold across all problems. This sec-tion, therefore, presents three convex approximationsto the negative log-likelihood that can be efficiently op-timized to attain a good starting point for algorithm2.3 should arbitrary initialization fail.

We present the first two results for the traditional IOCGibbs model, although analogous results hold for ourgeneralized Gibbs model as well. However, to empha-size its application to our full combined model, the dis-cussion in Section 4.3 of what we call the soft-backupapproximation is presented in full generality. In par-ticular, steps (1) and (2) of the optimization strategyproposed in Section 3 may be replaced by optimizingthis convex approximation.

4.1 The perceptron algorithm

Given the discussion of almost-convexity in Section 3,the simplest convex approximation is the perceptron

objective

h(w) =N∑

i=1

C(ξi)− minξ∈Ξi

C(ξ). (6)

See (Collins & Roark, 2004) for details regarding theperceptron algorithm. Since each discrepancy term ispositive, this approximation is a convex lower bound.

4.2 Expert augmentation

Our second convex approximation can be derived byaugmenting how we compute the action probabilitiesof the Gibbs model. Equation 1 dictates that theprobability of taking action a from state st is in-versely proportional to E(st, a) = Q∗(st, a) throughour MDP. We will modify this energy function only forthe value at the action at chosen by the expert fromthat state. Specifically, we will prescribe E(st, at) =C(ξt)−

∑Tτ=t c(st, at); the energy of the expert’s action

now becomes the cumulative cost-to-go of the trajec-tory taken by the expert.

The negative log-likelihood of the resulting policy isconvex. Moreover, for any parameter setting w, sinceQ∗(st, at) ≤ Q(ξt) and the energies of each alternativeaction remain unchanged, the probability of taking theexpert’s action cannot be smaller under the actual pol-icy than it was under this modified policy. This obser-vation shows that our modified negative log-likelihoodforms a convex upper bound to the desired negativelog-likelihood objective.

4.3 Soft-backup modification

Arguably, the most accurate convex approximation wehave developed comes from deriving a simple soft-backup dynamic programming algorithm to modifythe energy values for the expert’s action used by theGibbs policy. Applying this procedure backward alongthe example trajectory ξ = {(ot, at)}Tt=1 starting fromthe oT and proceeding toward o1 modifies the policymodel in such a way that the contrasting hard/soft-min terms of Equation 5 cancel.

Specifically, as detailed in Algorithm 2 the soft-backup algorithm proceeds recursively from observa-tion oT replacing each hard-min J-value J∗M(ot) =mina∈A C(ot, a) + JM(T a

ot) with the associated soft-

min. We define

JM(ot) = − log∑a∈A

e−C(ot,a)− eJM(T aot

), (7)

with J(T aot

) = J(T aot

) when a 6= at. Our new policyalong the example trajectory, therefore, becomes

p(a|ot) =e−E(ot,a)−C(ot,a)− eJM(T a

ot)∑

a′∈Ase−E(ot,a′)−C(ot,a′)− eJM(T a′

ot)

(8)

429


Algorithm 2 Soft-backup procedure for convex ap-proximation

1: procedure SoftBackup( ξ, c : S → A )2: Compute cost-to-go values JM(s) for s in M

that can be reached by at most one actionfrom any ot ∈ ξ.

3: Initialize JM(sT ) = 04: for t = T − 1, . . . , 1 do5: Set αt =

∑a∈A\at

e−E(ot,a)−C(ot,a)−JM(T ast

)

6: Set αt = e−E(ot,at)−C(ot,at)− eJM(ot+1)

7: Update JM(ot) = − log (αt + αt)8: end for9: return Updated J-values.

10: end procedure

Theorem 4.1 : The negative log-likelihood of themodified policy of Equation 8 is convex and takes theform

l(w; ξ) =∑

(ot,at)∈ξ

(E(ot, at) + C(ot, at))− JM(o1).

Proof (sketch). The true objective function is given inEquation 3. Under the modified policy model, each en-ergy value of the Gibbs model becomes E(o, a)+ C(o, a)+eJM(T a

o ). In particular, for each t along the trajec-tory, the modified J-value in that expression cancelswith the log-partition term corresponding to observation-action pair (ot+1, at+1). Therefore, summing across alltimesteps leaves only the energy and cost segment termsP

t E(ot, at)+C(ot, at) and the first observation’s modified

J-value eJ(o1). This argument demonstrates the form of theapproximation. Additionally, since each backup operationis a soft-min performed over concave functions, each termeJ(ot) is also concave. The term − eJ(o1) is therefore convex,which proves the convexity of our approximation. 2

Setting the value features to zero wv = 0, we can addi-tionally show that this final approximation l(w) is alsobounded by the perceptron h(w) in the same was asour negative log-likelihood objective l(w). Moreover,l(w) ≤ l(w) everywhere. Combining these bounds, wecan state that for a given example trajectory ξ and forall w,

h(w) ≤ l(w) ≤ l(w) ≤ h(w) + |ξ| log |A|. (9)

This observation suggests that the soft-backup con-vex approximation is potentially the tightest of theapproximations presented here. Our experiments inSection 5.1 support this claim.

5 Experimental results

In this section, we first demonstrate our algorithm ona simple two-dimensional navigational problem to il-lustrate the overall convex behavior of optimizing the

negative log-likelihood, and compare that to the per-formance of the convex approximations discussed insection 4. We then present two real-world experiments,and compare the performance of the combined model(with value features) to that of the Gibbs model alone(without value features).

5.1 An illustrative example

In this experiment, we implemented Algorithm 1 onthe simple navigational problem depicted in the left-most panel of Figure 2 and compared its performanceto that of each convex approximation presented in Sec-tion 4 using only the traditional IOC Gibbs model.We manually generated 10 training examples chosenspecifically to demonstrate stochasticity in the behav-ior. Our feature set consisted of 14 randomly posi-tioned two-dimensional radial basis features along witha constant feature. We set our regularization parame-ter to zero for this problem.

As we predicted in Section 4, the backup approxima-tion performs the best on this problem. Although itconverges to a suboptimal solution, the negative log-likelihood levels off and does not significantly increasefrom the minimum attained value in contrast to thebehavior seen in the perceptron and replacement ap-proximations. The center two panels show the costfunctions learned for this problem using Algorithm1 (center-left) and by optimizing the soft-backup ap-proximation (center-right).

5.2 Turn Prediction for Taxi Drivers

We now apply our approach to modeling the routeplanning decisions of drivers so that we can predict,for instance, whether a driver will make a turn atthe next stoplight. The imitation learning approachto this problem (Ziebart et al., 2008a), which learnsa cost function based on road network features, hasbeen shown to outperform direct action modeling ap-proaches (Ziebart et al., 2008b), which estimate theaction probabilities according to previous observationproportions (Simmons et al., 2006; Krumm, 2008).However, computational efficiency in the imitationlearning approach comes at a cost: the state space ofthe corresponding Markov Decision Process must bekept small. Table 1 shows how the number of statesin the Markov decision process grows when paths oflength K decisions are represented by the state. Pre-viously, the model was restricted to using only thedriver’s last decision as the state of the Markov De-cision Process to provide efficient inference.

As a consequence, given the driver’s intended destina-tion, decisions within the imitation learning model areassumed to be independent of the previous driving de-cisions that led the driver to his current intersection.We incorporate the following action value features thatare functions of the driver’s previous decisions to relaxthis independence assumption:

430


Figure 2: This figure shows the examples (left), the cost map learned by directly optimizing the negative log-likelihood(middle-left), the cost map learned by optimizing the soft-backup approximation (middle-right), and a plot comparingthe performance of each convex approximation in terms of its ability to optimize the negative log-likelihood (right). Thetraining examples were chosen specifically to exhibit stochasticity. See Section 5.1 for details.

Table 1: State space size for larger previous segmenthistory.

History (K) States1 3157042 9010463 24961224 77336205 23281701...

...k ≈ 3k ∗ 105

• Has the driver previously driven on this road seg-ment on this trip?

• Does this road segment lead to an intersectionpreviously encountered on this trip?

We compare our approach with a combination of theseaction value features and additional cost-to-go featuresthat are road network characteristics (e.g., length,speed limit, road category, number of lanes) with amodel based on only cost-to-go features. Additionally,the Full model contains additional unique costs for ev-ery road segment in the network. We use the problemof predicting a taxi driver’s next turn given final desti-nation on a withheld dataset (over 55,000 turns, 20%of the complete dataset) to evaluate the benefits of ourapproach. We include baselines from previously ap-plied approaches to this problem (Ziebart et al., 2008b)and compare against the Gibbs model without valuefeatures, and our new inverse optimal heuristic control(IOHC) approach, which includes value features.

Table 2 shows the accuracy of each model’s mostlikely prediction and the average log-probability of thedriver’s decisions within each model. We note for bothsets of cost-to-go features a roughly 18% reduction inthe turn prediction error over the Gibbs models whenincorporating action value features (IOHC), which isstatistically significant (p < 0.01). We also find cor-responding improvements in our log-probability met-ric. We additionally note better improvement in bothmetrics over the best previously applied approaches for

Table 2: Turn prediction evaluation for various mod-els.

Model Accuracy Log ProbabilityRandom Guess 46.4% -0.781Markov Model 86.2% -0.319

MaxEnt IOC (Full) 91.0% -0.240Gibbs (Basic) 88.8% -0.319IOHC (Basic) 90.8% -0.246Gibbs (Full) 89.9% -0.294IOHC (Full) 91.9% -0.226

Figure 3: The three images shown here depict the officesetting in which the pedestrian tracking data was collected.

this task.

5.3 Pedestrian prediction

Predicting pedestrian motion is important for manyapplications, including robotics, home automation,and driver warning systems, in order to safely inter-act in potentially crowded real-world environments.Under the assumption that people move purposefully,attempting to achieve some goal, we can model a per-son’s movements using an MDP and train a motionmodel using imitation learning.

In this experiment, we demonstrate that trajectoriessampled from a distribution trained using momentum-based value features better match human trajecto-ries than trajectories sampled from a model withoutvalue features. The resulting distribution over statesis therefore a superior estimate of future behavior ofthe person being tracked.

Tracks of pedestrians were collected in an office en-vironment using a laser-based tracker (see Figure 3).The outline of the room and the objects in the room

431


Figure 4: This figure compares the negative log-likelihoodprogressions between a traditional Gibbs model (withoutvalue features) and an IOHC model (with value features)on a validation set. Access to features encoding dynamiccharacteristics of the actions substantially improves mod-eling performance. The full collection of pedestrian trajec-tories is shown on the right from an overhead perspective.

were also recorded. The laser map was discretized into15cm by 15cm cells and convolved with a collection ofsimple Gaussian smoothing filters. These filtered val-ues and one feature representing the presence of anobject make up the state-based feature set. Addition-ally, we include a set of action value features consistingof a history of angles between the current and previ-ous displacements. These action features incorporatea smoothness objective that would otherwise require ahigher-dimensional state space to incorporate.

We constructed a Gibbs model (without value fea-tures) and an IOHC model (with value features) us-ing a set of 20 trajectories and we tested the modelon a set of 20 distinct validation trajectories. Figure4 compares the negative log-likelihood progression ofboth models on the test set during learning. The algo-rithm is able to exploit features that encode dynami-cal aspects of each action to find a superior model ofpedestrian motion.

6 Conclusions

We have presented an imitation learning model thatcombines the efficiency and generality of behavioralcloning strategies with the long-horizon prediction per-formance of inverse optimal control. Our experimentshave demonstrated empirically the benefits of this ap-proach on real-world problems. In future work, weplan to explore applications of the pedestrian predic-tion model to developing effective robot-pedestrian in-teraction behaviors. In particular, since stochasticsampling may be implemented using efficient MonteCarlo techniques, the computational complexity ofpredictions can be controlled to satisfy real-time con-straints.

Acknowledgments

The authors gratefully acknowledge the support of theDARPA Learning for Locomotion contract, the Quality ofLife Technology Center/National Science Foundation con-tract under Grant No. EEEC-0540865, and Intel Research,Pittsburgh.

References

Abbeel, P., and Ng, A. Y. 2004. Apprenticeshiplearning via inverse reinforcement learning. In Proc.ICML, 1–8.

Bain, M., and Sammut, C. 1995. A framework forbehavioral cloning. In Machine Intelligence Agents,103–129. Oxford University Press.

Collins, M., and Roark, B. 2004. Incremental parsingwith the perceptron algorithm. In Proc. ACL, 111–118.

Kolter, J. Z.; Abbeel, P.; and Ng, A. 2007. Hier-archical apprenticeship learning with application toquadruped locomotion. In Proc. NIPS. 769–776.

Krumm, J. 2008. A markov model for driver routeprediction. Society of Automative Engineers (SAE)World Congress.

LeCun, Y.; Muller, U.; Ben, J.; Cosatto, E.; andFlepp, B. 2006. Off-road obstacle avoidance throughend-to-end learning. In Proc. NIPS. 739–746.

Neu, G., and Szepesvari, C. 2007. Apprenticeshiplearning using inverse reinforcement learning andgradient methods. In Proc. UAI, 295–302.

Ng, A. Y., and Russell, S. 2000. Algorithms for inversereinforcement learning. In Proc. ICML, 663–670.

Nigam, K.; Lafferty, J.; and McCallum, A. 1999. Usingmaximum entropy for text classification. In IJCAI-99 Workshop on Machine Learning for InformationFiltering.

Pomerleau, D. A. 1988. Alvinn: an autonomous landvehicle in a neural network. In Proc. NIPS, 305–313.

Ramachandran, D., and Amir, E. 2007. Bayesian in-verse reinforcement learning. In Proc. IJCAI, 2586–2591.

Ratliff, N.; Bagnell, J. A.; and Zinkevich, M. 2006.Maximum margin planning. In Proc. ICML, 729–736.

Ratliff, N.; Bradley, D.; Bagnell, J. A.; and Chestnutt,J. 2006. Boosting structured prediction for imitationlearning. In Proc. NIPS, 1153–1160.

Silver, D.; Bagnell, J. A.; and Stentz, A. 2008. Highperformance outdoor navigation from overhead datausing imitation learning. In Proceedings of RoboticsScience and Systems.

Simmons, R.; Browning, B.; Zhang, Y.; and Sadekar,V. 2006. Learning to predict driver route and desti-nation intent. Proc. Intelligent Transportation Sys-tems Conference 127–132.

Ziebart, B.; Maas, A.; Bagnell, J. A.; and Dey, A.2008a. Maximum entropy inverse reinforcementlearning. In Proc. AAAI, 1433–1438.

Ziebart, B.; Maas, A.; Dey, A.; and Bagnell, J. A.2008b. Navigate like a cabbie: Probabilistic reason-ing from observed context-aware behavior. In Proc.UbiComp, 322–331.

inverse optimal heuristic control for imitation...

Documents