Importance Sampling Actor-Critic Algorithms

Jason L. Williams John W. Fisher III Alan S. WillskyLaboratory for Information and Decision Systems andComputer Science and Artificial Intelligence Laboratory

Massachusetts Institute of TechnologyCambridge, MA 02139

jlwil@mit.edu fisher@csail.mit.edu willsky@mit.edu

Abstract— Importance Sampling (IS) and actor-critic are twomethods which have been used to reduce the variance ofgradient estimates in policy gradient optimization methods. Weshow how IS can be used with Temporal Difference methodsto estimate a cost function parameter for one policy usingthe entire history of system interactions incorporating manydifferent policies. The resulting algorithm is then applied toimproving gradient estimates in a policy gradient optimization.The empirical results demonstrate a 20-40× reduction in vari-ance over the IS estimator for an example queueing problem,resulting in a similar factor of improvement in convergence fora gradient search.

I. INTRODUCTION

Many problems of practical interest can be formulated and,conceptually, solved optimally using dynamic programming(cf [1]). However, the practical applicability of the methodto problems with large state spaces is limited due to theso-called curse of dimensionality. The absence of exactmodels for the systems of interest further limits applicability,through the so-called curse of modelling. Approximationswhich address both of these difficulties have been studiedextensively over the past decade, and may be divided intotwo broad categories: cost function approximation methodsand policy approximation methods.

Cost function approximation methods seek to approximatethe optimal cost-to-go function with a particular parametricform, such as a linear combination of basis functions. Theparameter vector can be learned from simulation using themethod of Temporal Differences [2]. In policy approxima-tion, one chooses a particular parameterized policy family,and seeks to find the parameter value which minimizes thecost of employing the policy. The minimization is commonlyperformed using stochastic gradient methods. The policygradient method [3] obtains a noisy estimate of the gradientof the objective with respect to the policy parameter froma single simulation trajectory. The actor-critic method [4]improves this estimate by incorporating cost function approx-imation: by retaining information from previous simulations,and constraining the estimates of the cost-to-go functionto a low-dimensional subspace, the variance is reducedsubstantially.

Frequently, interaction with the system (or simulationof the system) is expensive, computationally or otherwise,hence it is desirable to exploit the limited simulation datawhich is available as much as possible. The Importance

Sampling (IS) method of [5], discussed in detail in Section II-B, provides a means of utilizing information from the entirehistory of interactions with the system (using many differentpolicies) to compute a reduced variance estimate of theobjective gradient. We present an algorithm in Section IIIthat combines the IS algorithm with a cost function approx-imation method. By restricting the cost estimates to lie ina low-dimensional subspace, the variance of the gradientestimate is reduced substantially. The simulation results inSection IV demonstrate a 20-40× reduction in variance overthe IS estimator for an example queueing problem.

Whereas Konda and Tsitsiklis’ actor-critic method relieson two time constants, one of which controls the faster adap-tion rate of the approximation parameter, and the other whichcontrols the slower adaption rate of the policy parameter,our method adaptively weights the entire system interactionhistory to calculate approximation parameters. This allowsthe system designer to trade off the number of interactionsrequired with the system against computational cost.

II. BACKGROUND

Consider a Markov Decision Process (MDP) with statess ∈ S and actions a ∈ A. We assume a randomized policyparameterized by θ, such that action a is selected in state swith probability uθ(s, a). If action a is selected in state s,the immediate cost incurred is ga(s), and the next state isdrawn from the transition distribution on S, p(·|s, a).

For our purposes, we consider stochastic shortest pathformulations1 in which, under all policies, there is a singlerecurrent class of states T ⊂ S and that these states are cost-free (ga(s) = 0 ∀ s ∈ T ). Thus the goal of the policy is tominimize the cost incurred before reaching the terminal setT . As in [3], we assume that the starting state is drawn fromthe probability distribution on S, π(·). We denote by Jθ(s)the expected cost-to-go from state s using the policy definedby the parameter vector θ. Consequently, the objective wewish to minimize is:

χ(θ) =∫S

π(s)Jθ(s)ds (1)

We assume that the state is known at each decision step.We make the common assumptions that ∃n < ∞ such that

1Analogous statements can be made for average cost per stage problemsby considering renewal intervals, as per the development in [3].

Proceedings of the 2006 American Control ConferenceMinneapolis, Minnesota, USA, June 14-16, 2006

WeC07.1

1-4244-0210-7/06/$20.00 ©2006 IEEE 1625

the probability of terminating within n steps from any stateand under any policy is uniformly bounded below by ε >0, and that the policy distribution uθ(s, a) is differentiablew.r.t. θ and ∇θuθ(s,a)

uθ(s,a) is bounded.

A. λ-Least Squares Temporal Difference

λ-Least Squares Temporal Difference (λ-LSTD) providesa method of approximate policy evaluation with convergenceat a much faster rate than traditional temporal differencemethods [6], [7]. Given a set of basis functions, the cost-to-go is approximated by:

J(s, r) = φ(s)T r (2)

To apply it to a stochastic shortest path problem, we take aseries of independent simulation trajectories and accrue forthe i-th simulation, xi = (si

0, ai0, . . . , s

ini−1, a

ini−1, s

ini), the

following quantities:

A(xi) =ni−1∑m=0

zim[φ(si

m+1) − φ(sim)]T

b(xi) =ni−1∑m=0

zimg(si

m) (3)

where zim =

∑mk=0 λm−kφ(si

k). We can then combineN simulation trajectories (x1, . . . , xN ) to give an overallestimate:

AN =1N

N∑m=1

A(xi)

bN =1N

N∑m=1

b(xi)

rN = −A−1N bN (4)

This is equivalent to the method described in [6], except thatthe eligibility trace zi is reset after each simulation (whichis natural for a stochastic shortest path problem). Under mildconditions,2 AN and bN will converge to their expectedvalues as N → ∞ w.p. 1, hence the parameter vector rN

also converges. The choice of basis functions φ(s) is problemdependent; Section IV provides an example for a queueingproblem.

B. Importance Sampling

Policy gradient methods estimate ∇θχ(θ) via a smallnumber of Monte Carlo simulations under an essentiallyfixed policy. The resulting estimate often exhibits highvariance, increasing convergence times while decreasingthe utility of the resulting policy. Peshkin and Shelton [5]suggest an Importance Sampling (IS) approach, using sampletrajectories under varied policies, as a means of incorporatingprevious simulations and, consequently, reducing the vari-ance. The importance sampling estimate of E{f(x)}, using

2The trajectories {xi} are i.i.d., the per-stage costs are bounded and theusual termination assumption of stochastic shortest path problems are met(i.e. A(xi) and b(xi) have finite variance).

samples {xi}Ni=1 drawn from distribution q(·), is expressed

by:

f =1N

N∑i=1

pθ(xi)q(xi)

f(xi)

f =1N

∑Ni=1

pθ(xi)q(xi) f(xi)

1N

∑N

i=1pθ(xi)q(xi)

(5)

The estimate f is unbiased, provided that q(x) > 0 ∀ x :pθ(x) > 0, and converges to the true value as the numberof samples increases. The normalized estimate f reducesthe variance in f at the expense of inducing a bias (whichvanishes asymptotically). Samples corresponding to manydifferent parameter values {θi, i ∈ {1, . . . , N}} may beadmitted by treating them as collectively belonging to themixture density q(x) = 1

N

∑N

i=1 pθi(x), where pθi

(x) is thelikelihood of trajectory x under the policy defined by the i-th parameter θi. In a stochastic shortest path problem, eachsimulation trajectory forms an independent sample, startingat an independently drawn initial state s0 ∼ π(·), and endingwith sn ∈ T . The likelihood of obtaining a given trajectoryx = (s0, a0, . . . , sn−1, an−1, sn) may be calculated as:

pθ(x) =

[p(s0)

n∏i=1

p(si|si−1, ai−1)

]·

[n−1∏i=0

uθ(si, ai)

](6)

where only the second term in brackets has dependence onthe policy parameter vector θ. Consequently, in calculatingthe ratios p(·)

q(·) to estimate the cost at parameter value θ, thetransition probabilities cancel leaving:

pθ(x)q(x)

=∏n−1

i=0 uθ(si, ai)1N

∑N

j=0

∏n−1i=0 uθj

(si, ai)(7)

In order to estimate the gradient of the objective, one canuse a simple estimate of the cost along a trajectory, J(x) =∑n

k=0 gak(sk). Noting that E{J(x)|s0 ∼ π(·)} = χ(θ), we

then have: 3

∇θχ(θ) =∫

∇θpθ(x)J(x)dx

=∫

pθ(x)∇θpθ(x)

pθ(x)J(x)dx (8)

Assuming that ∇θpθ(x)pθ(x) is bounded, we apply Eq. (5) to

obtain

∇θχ(θ, {xi}Ni=1) =

1N

N∑i=1

∇θpθ(xi)q(xi)

J(xi) (9)

3Note that the random variable to which the variable of integration inEq. (8) corresponds may be discrete or mixed, depending on the stateand action spaces. Furthermore, it is of variable dimension, dependingon the length of the trajectory. For the sake of clarity we use normalRiemann integration notation; the ideas may be made precise using measuretheoretic notation, replacing the importance sampling weights with theRadon-Nikodym derivatives.

1626

We estimate the corresponding cost similarly:

χ(θ, {xi}Ni=1) =

1N

N∑i=1

pθ(xi)q(xi)

J(xi) (10)

III. IMPORTANCE SAMPLING ACTOR-CRITIC

The method described in the previous section applies ISto a simple trajectory-based cost estimator J(x) to estimatethe objective for different parameter values. The algorithmdeveloped below, referred to as Importance Sampling Actor-Critic (ISAC), uses a cost function approximation methodto constrain these estimates to a low-dimensional subspace,reducing the variance of the resulting estimates.

While the recursive forms of the TD(λ) method [2] andλ-LSPE [6] algorithms are convenient for online processing,they do not allow IS to be applied to evaluate the cost of onepolicy using simulations from other policies. However, thestructure of the λ-LSTD estimator does provide a form whichallows the quantities being accrued to be broken up into amean of independent samples, so that IS can be applied. Weoutline how this can be done in Section III-A, and apply theresult to policy gradient estimation in Section III-B.

A. Importance Sampling Least Squares Temporal Difference

Suppose we want to use λ-LSTD to estimate the cost of thepolicy resulting from parameter θ. Eq. (4) provides a meansof taking a sequence of N i.i.d. trajectories and estimating thequantities Aθ = E{A(x)} and bθ = E{b(x)}, and hence thecost function approximation parameter rθ = A−1

θbθ. Using

Eq. (5), we may also obtain unbiased estimates of thesequantities, using simulations from different policies throughimportance sampling:

Aθ({xi}Ni=1) =

1N

N∑i=1

pθ(xi)q(xi)

A(xi)

bθ({xi}Ni=1) =

1N

N∑i=1

pθ(xi)q(xi)

b(xi) (11)

Since the probability weights are the only variables whichdepend on the parameter vector, we can estimate the gradi-ents of these quantities as:4

∇θAθ({xi}Ni=1) =

1N

N∑i=1

∇θpθ(xi)q(xi)

A(xi)

∇θbθ({xi}Ni=1) =

1N

N∑i=1

∇θpθ(xi)q(xi)

b(xi) (12)

B. Application to Actor-Critic

If we use an estimate of the expected cost of a simulationJ(θ, x), which depends on the value of the policy parameter,the gradient of the expected cost in Eq. (8) becomes

∇θχ(θ) =∫

∇θpθ(x)J(θ, x)dx +∫

pθ(x)∇θJ(θ, x)dx

(13)

4In general, the gradient of the matrix Aθ w.r.t. the vector θ is a tensor;we will treat it as a collection of matrices, each member of which is thederivative of Aθ with respect to a different component of θ.

Noting that if J is not a function θ, then the second term inEq. (13) is zero, we see that Eq. (13) is a generalization ofEq. (8). Now consider an alternative form of cost estimator,based on the λ-LSTD algorithm. The cost of a simulationtrajectory x = (s0, a0, . . . , sn−1, an−1, sn) is estimated as:

J(θ, x) = φ(s0)T rθ (14)

where from Eq. (11) rθ = −A−1θ

bθ. Using this estimator,only the starting state of the trajectory is used for the costestimate: the impact of the policy parameter on the cost istaken into account through the second term in Eq. (13). Sincethe cost estimate depends only on the starting state, Eq. (13)may be rewritten as:

∇θχ(θ) = ∇θ

∫π(s0)J(θ, s0)ds0

=∫

π(s0)∇θJ(θ, s0)ds0 (15)

Because the distribution of starting state is not a functionof the policy parameter, the first term in Eq. (13) is zero.Evaluation of Eq. (15) requires the gradient of J(θ, s0):

∇θJ(θ, s0) = φ(s0)T∇θrθ

= −φ(s0)T∇θ[A−1θ

bθ] (16)

= φ(s0)T {A−1θ

[∇θAθ]A−1θ

bθ − A−1θ

∇θbθ}(17)

Eq. (17) should be read as meaning that the i-th elementof ∇θJ(θ, s0) is the RHS evaluated with gradients takenw.r.t. the i-th term of θ. The gradient estimate can then beevaluated as: (where all estimates are functions of the sample{xi}N

i=1)

∇θχ(θ, {xi}Ni=1) =

[1N

N∑i=1

φ(si0)

T

]·{

A−1θ

∇θAθA−1θ

bθ − A−1θ

∇θbθ

}(18)

While the individual estimates for Aθ , ∇θAθ, bθ and ∇θbθ

are unbiased, the nonlinear composition in Eq. (18) will bebiased. However, as the number of samples N grows, theindividual estimates converge to the respective true parametervalues, hence the gradient estimate will be asymptoticallyunbiased. The cost itself can be estimated similarly:

χ(θ, {xi}Ni=1) =

[1N

N∑i=1

φ(si0)

T

]A−1

θbθ (19)

C. Remarks

The form of the actor-critic algorithm of [4] providesinsight into a subspace which the basis functions mustspan for the purpose of obtaining the gradient estimate. Ananalogous insight for the ISAC estimator is not obvious, andremains an open question. However, the empirical results inthe following section demonstrate the dramatic improvementin convergence which can be achieved using a well-chosenlow-dimensional cost estimate, as well as the bias which canresult from a poor choice of basis functions.

1627

The approximation architecture is primarily employed inISAC to estimate the cost from the starting state to the end ofthe simulation. Therefore, it is intuitively desirable to selectthe approximation parameter which minimizes the differencebetween the true cost and the approximate cost accordingto a norm weighted by the distribution of starting states.Convergence bounds for temporal difference algorithms [2](including λ-LSTD) are in terms of the L2 norm weighted bythe steady state distribution. Accordingly, if the distributionof starting state is vastly different from the steady statedistribution, then the error in the resulting cost estimate maybe large.

A practical issue which affects the IS and ISAC algorithmsalike is computational complexity, due to the growing historyof system interactions over which the estimates are calcu-lated. In practice, one would commonly retain a subset ofpast interactions; in the experiments to follow, we retain themost recent 10,000 simulations. Selection of this memorylength effectively allows the system designer to trade offthe number of interactions required with the system againstcomputational cost.

IV. EXPERIMENTAL RESULTS

We compare the performance of the algorithm presentedin Section III was compared to the IS estimator discussedin Section II-B for a queueing problem. So that we mightcompare performance to the optimal solution, the problemwas chosen to have a small enough size such that theactual cost-to-go function can be calculated. The state inthe problem corresponds to the length of a queue, s ∈{0, . . . , B}, where B = 100. The control a ∈ {0, 1} affectsthe probability that the queue length is reduced:

P (sk = y|sk−1 = x, ak−1 = a) =⎧⎪⎪⎪⎨⎪⎪⎪⎩1, y = x = 01 − βa, y = x > 0βa, y = x − 1 ≥ 00, otherwise

(20)

where β0 = 0.2, and β1 = 0.5. The initial state is drawnfrom a modified geometric distribution:

P (s0 = x) =

⎧⎪⎨⎪⎩0, x = 0(1 − ω)x−1ω, 1 ≤ x < B∑∞

y=B (1 − ω)y−1ω, x = B

(21)

where ω = 0.03. The cost per stage is given by ga(s) =s + aη, where η = 35. The stochastic shortest path problemterminates when the queue is emptied (s = 0). A reason-able parameterized policy family for this problem is a softthreshold function:

uθ(s, a = 1) =exp[0.1(s − θ)]

1 + exp[0.1(s − θ)](22)

0 20 40 60 80 1003200

3400

3600

3800

4000

4200

4400

4600

4800

θ

Exp

ecte

d co

st

Cost approximation using 10,000 simulations

0 20 40 60 80 1003200

3400

3600

3800

4000

4200

4400

4600

4800

θ

Exp

ecte

d co

st

Cost approximation confidence regions

χ(θ)ISACIS normalized

χ(θ)ISACIS normalizedIS unnormalized

Fig. 1. Expected cost from starting state to termination as a function of θ.Upper figure shows the true cost, and costs estimated from a single set of10,000 simulations using the ISAC and normalized IS estimators. Lower plotshows the 1-σ confidence bounds for estimates using 10,000 simulations,estimated using 400 sets of 10,000 simulations.

For any given θ, the true cost of the policy can be found bysolving Bellman’s equation: [1]

Jθ(s) =∑

a

uθ(s, a)ga(s) +∑a,y

uθ(s, a)P (y|s, a)Jθ(y),

s ∈ {1, . . . , B} (23)

where Jθ(0) = 0. The value Jθ(s) is the expected cost toreach termination from state s using the policy defined bythe parameter θ. From Eq. (1), the quantity which we seek tominimize is the expected cost to reach termination when thestarting state is drawn from the initial distribution in Eq. (21),χ(θ) =

∑s P (s0 = s)Jθ(s). We use the value λ = 1 in the

LSTD algorithm, and feature vectors φ(s) = [s s2 s3]T .

A. Cost function estimates

In order to give a qualitative comparison of the rela-tive merit of the two approximations, the cost estimatesof Eq. (10) and Eq. (19) were evaluated for a range ofvalues of θ with simulations computed using random policyparameter values, θ ∼ U(0, 100). The upper plot in Fig. 1compares the true cost to the cost estimates obtained usingthe two methods with a single set of 10,000 simulations.The gradient estimators of Eq. (9) and Eq. (18) correspondto calculating the derivative of the respective curves in Fig. 1,

1628

0 50 100 150 200 250 300 350 4003400

3600

3800

4000

4200

4400

4600

Step number

Tru

e co

stConvergence of gradient search

ISACISOptimal cost

0 50 100 150 200 250 300 350 4003400

3500

3600

3700

3800

3900

4000

4100

Step number

Tru

e co

st

Convergence of gradient search

ISACISOptimal cost

Fig. 2. Convergence of gradient search algorithms. Upper plot shows meanand standard deviation of the true cost achieved after different numbersof gradient steps using the ISAC and IS gradient estimates with step sizeτ = 0.05. Lower plot shows the same data for ISAC alongside results usingIS gradient estimate with step size τ = 0.0025. Some lower error boundsfall below the optimal cost due to skewness in the distributions.

hence the diagram allows one to anticipate the behaviorof a gradient optimization procedure to some degree. Thelarge variability of the IS estimate demonstrates that gradientestimates combining this number of simulations will stillhave a large variance. While there are still fluctuations inthe ISAC estimate (and some local minima), their varianceis reduced substantially. The lower plot in Fig. 1 showsthe 1-σ confidence bounds for the estimates obtained from10,000 simulations (the confidence bounds were estimatedusing 400 different sets of 10,000 simulations). The reductionin the standard deviation of the ISAC estimator over thenormalized IS estimator is a factor of between four andsix, corresponding to a reduction in variance by a factor ofbetween 20 and 40.

B. Gradient search

To compare the performance of the gradient search proce-dures, we tested the two algorithms from the same startingpoint for 150 random values of θ0 ∼ U(0, 100). In each gra-dient iteration, we performed 100 Monte Carlo simulationsusing the current policy parameter θk, calculated the gradientusing the respective estimate (Eq. (9) and Eq. (18)), andimplemented the gradient step θk+1 = θk − τ∇χ. Gradientestimates were calculated using a sliding window of the

0 20 40 60 80 1003400

3600

3800

4000

4200

4400

4600

4800

θ

Approximations of χ(θ) induced by basis function selections

χ(θ)

φ(s) = [s s2 s3]T

φ(s) = s2

φ(s) = s

Fig. 3. Approximation of objective function resulting from different choicesof basis functions.

last 10,000 simulations. The behavior of the algorithms isillustrated in Fig. 2. The upper plot shows the mean and stan-dard deviation of the true costs of the policies after differentnumbers of gradient steps, with step size τ = 0.05. The plotillustrates that the ISAC method consistently converges tothe optimal solution, while the IS method oscillates betweenapparent local minima. The lower plot shows the same datafor ISAC alongside results using IS gradient estimate withstep size τ = 0.0025. With the smaller step size, the ISestimator exhibits slow convergence towards the minimum,but a far greater number of steps is required to achieve thesame degree of convergence as ISAC.

C. Impact of approximation architecture

The ISAC method discussed in Section III-B effectivelyoptimizes the function χ(θ) = E{φ(s0)T }rθ, where rθ isthe approximation architecture parameter vector supplied bythe λ-LSTD algorithm. If λ = 1, the LSTD algorithm willconverge to the weighted projection of the true cost functiononto the approximation architecture subspace. The approxi-mations χ(θ) resulting from these projections are shown fordifferent selections of approximation architecture in Fig. 3,demonstrating the importance of choosing an approximationarchitecture which provides sensitivity to variations in thecost due to the changing parameter vector. In this problem,a quadratic cost approximation visually appears to provide agood fit to the true cost (i.e., Jθ(s) is well-approximated byr · s2 for most policies), yet the resulting approximation ofχ(θ) loses much of the true structure. Comparatively, a linearcost approximation visually seems poorer, but the resultingapproximation is reasonable.

V. CONCLUSIONS

This paper has shown how importance sampling can beapplied to estimate the parameter of a cost function approx-imation architecture using λ-LSTD, and how the resultingalgorithm can be applied to improve gradient estimates in apolicy gradient optimization by restricting cost estimates to a

1629

low-dimensional subspace. Our empirical results demonstratethe utility of the proposed method in several ways: analysisof the cost function showed that the proposed method resultsin a significantly better approximation, while gradient imple-mentations were able to use much larger step sizes, resultingin significantly faster convergence behavior.

REFERENCES

[1] D. P. Bertsekas, Dynamic Programming and Optimal Control, 2nd ed.Belmont, MA: Athena Scientific, 2000.

[2] J. N. Tsitsiklis and B. Van Roy, “An analysis of temporal-differencelearning with function approximation,” IEEE Transactions on AutomaticControl, vol. 42, no. 5, pp. 674–690, May 1997.

[3] P. Marbach, “Simulation-based methods for markov decision pro-cesses,” PhD Thesis, Massachusetts Institute of Technology, Cambridge,MA, 1998.

[4] V. R. Konda and J. N. Tsitsiklis, “On actor-critic algorithms.” SIAMJournal on Control and Optimization, vol. 42, no. 4, pp. 1143–1166,2003.

[5] L. Peshkin and C. R. Shelton, “Learning from scarce experience,” inProceedings of the Nineteenth International Conference on MachineLearning, 2002, pp. 498–505.

[6] A. Nedic and D. P. Bertsekas, “Least squares policy evaluation algo-rithms with linear function approximation,” Discrete Event DynamicSystems: Theory and Applications, vol. 13, pp. 79–110, 2003.

[7] D. Bertsekas, V. Borkar, and A. Nedic, “Improved temporal differencemethods with linear function approximation,” Massachusetts Instituteof Technology, Tech. Rep. LIDS-TR-2573, 2003.

1630