employing the algebraic riccati equation for a parametrization of the solutions of the...

Systems & Control Letters 54 (2005) 693–703

www.elsevier.com/locate/sysconle

Employing the algebraic Riccati equation for a parametrizationof the solutions of the finite-horizon LQproblem:

the discrete-time case�

Augusto Ferrantea,∗, Lorenzo NtogramatzidisbaDipartimento di Elettronica e Informatica, Università di Padova, via Gradenigo 6/B, 35131 Padova, Italy

bDipartimento di Elettronica, Informatica e Sistemistica, Università di Bologna, viale Risorgimento 2, 40136 Bologna, Italy

Received 24 October 2003; received in revised form 22 October 2004; accepted 14 November 2004Available online 4 January 2005

Abstract

In this paper, a new methodology is developed for theclosed-formsolution of a generalized version of the finite-horizonlinear-quadratic regulator problem for LTI discrete-time systems. The problem considered herein encompasses the classicalversion of the LQ problem with assigned initial state and weighted terminal state, as well as the so-called fixed-end pointversion, in which both the initial and the terminal states are sharply assigned. The present approach is based on a parametrizationof all the solutions of the extended symplectic system. In this way, closed-form expressions for the optimal state trajectory andcontrol law may be determined in terms of the boundary conditions. By taking advantage of standard software routines forthe solution of the algebraic Riccati and Stein equations, our results lead to a simple and computationally attractive approachfor the solution of the considered optimal control problem without the need of iterating the Riccati difference equation.© 2005 Elsevier B.V. All rights reserved.

Keywords:Discrete-time systems; Quadratic cost function; Finite-horizon LQ problem; Extended symplectic system

1. Introduction

The finite-horizon linear-quadratic regulator (LQR)is a well-known and deeply investigated problemin control theory. The solution of this optimization

� Partially supported by the ministry of higher education ofItaly (MIUR), under project Identification and Control of IndustrialSystems.

∗ Corresponding author. Tel.: +390498277681.E-mail addresses:[email protected](A. Ferrante),

[email protected](L. Ntogramatzidis).

0167-6911/$ - see front matter © 2005 Elsevier B.V. All rights reserved.doi:10.1016/j.sysconle.2004.11.008

problem is obtained in the literature by resorting to thesolution of a suitable Riccati differential/differenceequation (see, for example,[1,8,11]), which is verydemanding from a computational point of view. Tolighten this computational burden, in[5] a new ap-proach has been presented for the solution, in thecontinuous-time case, of the finite-horizon LQ prob-lem, when both the initial and terminal states areassigned as well as in the case when either is assignedand the other is weighted in the performance index.This method is based on a parametrization of all the

http://www.elsevier.com/locate/sysconle

mailto:[email protected]

mailto:[email protected]

694 A. Ferrante, L. Ntogramatzidis / Systems & Control Letters 54 (2005) 693–703

state-costate functions satisfying the Hamiltonian dif-ferential equation.The present paper is intended to be the counterpart

of [5] for discrete-time systems. However, the exten-sion is not straightforward as we consider the generalcase when the matrix weighting the control input ispossibly singular and without any assumptions on theinvertibility of the state dynamics of the system. Werefer to [3,12], for continuous and discrete-time sin-gular LQ problems, respectively.We remark that the problem considered in this pa-

per is pretty general, since we allow each one of theinitial and terminal states to be either fixed or quadrat-ically weighted in the performance index, thus encom-passing the standard LQ problem, in which the initialstate is assigned and the terminal state is quadraticallyweighted, and the so-called fixed end-point LQ prob-lem, in which the extreme states are both sharply as-signed. We refer to[7,14] and references therein for adiscussion on the presence of constraints in the initialand terminal states. See also[2] for a feedback com-pensation scheme that guarantees optimality only fora given set of initial states. The contribution of this pa-per is that of providingclosed-formexpressions for theinput and state evolutions, which depend only on thestabilizing solution of a discrete-time algebraic Ric-cati equation and on the solution of a symmetric Steinequation (also known as discrete-time Lyapunov equa-tion) and donot require the iteration of a differenceRiccati equation. This result hinges on a parametriza-tion of the solutions of the extended symplectic systemassociated with the optimal control problem. To obtainthis parametrization, several ancillary results of inde-pendent interest on the discrete-time Riccati equationare derived.The approach presented is particularly convenient

for manifold reasons:

• Several versions of the LQ problem (encompass-ing the classical and the fixed end-point ones) aretreated in a unified framework.

• The closed-form expression of the optimal trajec-tory and of the optimal cost can be exploited forthe analysis of the structural properties of the op-timal solutions and for solving more general (pos-sibly parametric) optimization problems, e.g. prob-lems having the LQ as a subproblem. For example,consider the case of two (or more) optimal control

problems that are coupled by constraints or by aweight on the respective boundary conditions (seeSection 5.1).

• The optimal state-costate trajectories and controllaw are expressed as functions of two stable ma-trices, thus ensuring the robustness of the obtainedsolution even for very large time-horizons.

• Standard software routines can be exploited for thesolution of the algebraic Riccati and Stein equa-tions.

Throughout this paper, the symbolRn×m denotesthe space ofn × m real matrices,I denotes the iden-tity matrix andA+ denotes the Moore–Penrose pseu-doinverse of matrixA. The symbol�(A) denotes thespectrum ofA.

2. Description of the problem

Consider the linear time-invariant discrete-timestate equation

x(k + 1)= Ax(k)+ Bu(k), (1)

where, for allk�0, x(k) ∈ Rn is the state,u(k) ∈Rm is the control input,A ∈ Rn×n andB ∈ Rn×m.Consider ann× n matrixQ, anm×m matrixR andann×m matrixSsuch that

� :=[Q S

ST R

]= �T�0.

The set of matrices(A,B,�) is often referred toas a Popov triplet, and will be denoted by�. Letkf >0, andP0, Pf ∈ Rn×n be symmetric and posi-tive semidefinite. Moreover, define the quadratic costfunctional

J (x, u) := xT(0)P0x(0)

+kf−1∑k=0

[xT(k) uT(k)][Q S

ST R

] [x(k)

u(k)

]

+ xT(kf )Pf x(kf ). (2)

In this paper, we will consider the following optimalcontrol problems.

Problem 1 (Assigned initial and terminal states).Given x0, xf ∈ Rn, find a control law u(k),

A. Ferrante, L. Ntogramatzidis / Systems & Control Letters 54 (2005) 693–703 695

k ∈ {0, . . . , kf − 1},minimizing(2)with P0=Pf =0under the constraintsx(0)= x0, x(kf )= xf and (1).

Note that in this casekf must be greater than thereachability index of the pair(A,B) to guarantee theexistence of the solution for anyx0, xf ∈ Rn.

Problem 2 (Assigned initial state and weighted ter-minal state). Givenx0 ∈ Rn, find u(k), k ∈ {0, . . . ,kf − 1}, minimizing(2) with P0 = 0 under the con-straintsx(0)= x0 and (1).

Problem 3 (Assigned terminal state and weighted ini-tial state). Given xf ∈ Rn, find u(k), k ∈ {0, . . . ,kf − 1}, minimizing(2) with Pf = 0 under the con-straintsx(kf )= xf and (1).

Problem 4 (Weighted initial and terminal states). Findu(k), k ∈ {0, . . . , kf − 1}, minimizing(2) under theconstraint(1).

In the case of Problem 4, a trivial optimal solutionis the identically zero state trajectory and control law.

3. Mathematical background and preliminaryresults

It is well-known (see e.g.[6,9]) that the optimalstate trajectory and control law satisfy the extendedsymplectic system ESS(�):

x(k + 1)= Ax(k)+ Bu(k), (3)

�(k)=Qx(k)+ AT�(k + 1)+ Su(k), (4)

0= STx(k)+ BT�(k + 1)+ Ru(k), (5)

with k ∈ {0, . . . , kf − 1}, which is obtained by ex-tending the statex(k) of system (1) with the costate�(k) ∈ Rn. More precisely, ifx(k) is the optimal statetrajectory of any of the Problems 1–4 (whatever theassigned initial or terminal conditions), then a costatetrajectory�(k) and an input functionu(k) exist suchthat the vector[xT(k) �T(k) uT(k)]T satisfies (3)–(5).Note that (3)–(5) can be written in the descriptor form

Fp(k + 1)=Gp(k), k ∈ {0, . . . , kf − 1}, (6)

where

F :=[I 0 00 −AT 00 −BT 0

], G :=

[A 0 B

Q −I S

ST 0 R

]

and p(k) :=[x(k)

�(k)u(k)

].

The matrix pencilzF − G is known as the extendedsymplectic pencil associated with�.

Remark 1. If the extended symplectic pencilzF −Gis regular, the system of equations (6) has 2n + m

linearly independent solutions, spanning a(2n+m)-dimensional space (see e.g.[4,10]). Note however that,sinceu(kf ) is irrelevant both to the value of the costfunction and to the satisfaction of the constraints (foreach one of the four optimal control problems con-sidered), we may regard as equivalent two solutionsp1(k) andp2(k) of (6) if they only differ for the valueof u(kf ). On the other hand, a solutionp(k) of (6)remains such when we exchangeu(kf ) with an arbi-trary u ∈ Rm. Therefore, the dimension of the spaceof non-equivalent solutions of (6) equals 2n. We setas a representative element of each equivalence classof the solutions of (6) the vectorp(k) such that thecorrespondingu(kf ) is zero.

We make the following standing assumptions:

(A1) the pair(A,B) is reachable;(A2) the extended symplectic pencilzF−G associated

to� is regular and has no generalized eigenvalueson the unit circle.

Moreover, without loss of generality we may assumethat� := [BT R]T is full-column rank. Indeed, if�has a non-trivial kernel, there exists a subspaceU0of the input space that does not influence the statedynamics and the cost function. Then, by performinga suitable (orthogonal) change of basis in the inputspace, we may eliminateU0 and obtain an equivalentproblem for which this condition is satisfied.Recall that the discrete-time algebraic Riccati equa-

tion DARE(�)

P = ATPA− (ATPB + S)(R + BTPB)−1× (BTPA+ ST)+Q (7)


has a (unique) symmetric positive semidefinite solu-tionP+ such that all the eigenvalues of the closed-loopmatrix

A+ = A− B(R + BTP+B)−1(BTP+A+ ST) (8)

lie inside the unit disk if and only if the pair(A,B) isstabilizable and assumption (A2) holds, and a (unique)symmetric negative semidefinite solutionP− such thatall the eigenvalues of

A− = A− B(R + BTP−B)−1(BTP−A+ ST) (9)

are outside the unit disk if assumptions (A1)–(A2)hold and in addition the extended symplectic pencilzF −G has no generalized eigenvalues at zero.The matricesP+ andP−, when they exist, are, re-

spectively, called the stabilizing and anti-stabilizingsolutions of DARE(�).If A is invertible, we define as in[6, p. 105] the

time-reversed system associated with (1) as

x(k)= Ax(k + 1)+ Bu(k), (10)

whereA=A−1 andB=−A−1B. The weightingmatrixassociated with (10) is

� :=[Q S

ST R

],

where Q := A−TQA−1, R := R − STA−1B −BTA−TS + BTA−TQA−1B and S := A−TS −A−TQA−1B. The set of matrices� := (A, B, �) isthe time-reversed Popov triplet. The link between theextreme solutions of the DARE(�) and DARE(�)is described by the following theorem (see e.g.[6,p. 106]).

Theorem 1. Let A be invertible. LetP be a symmet-ric solution ofDARE (�). Then−P is a solution ofDARE (�) if and only if the extended symplectic pencilzF −G is regular and has no generalized eigenvaluesat zero or, equivalently, if and only ifR− STA−1B isnon-singular.

Recall the following theorem about the Stein equa-tion.

Theorem 2. LetA ∈ Rn×n andB ∈ Rn×m. The sym-metric Stein equation

AXAT −X + BBT = 0 (11)

has a unique symmetric solutionX ∈ Rn×n if and onlyif the spectrum of A does not contain reciprocal pairs,i.e., if and only if � ∈ �(A) implies that�−1 /∈�(A).Moreover, if the pair (A,B) is reachable, then A isa stability matrix if and only if(11) admits a uniquepositive definite solution.

In the following theorem, we present an efficientmethod for the computation of the maximal solutionP+ of the time-reversed DARE and of the correspond-ing closed-loop system matrixA+, which is basedon the solution of a symmetric Stein equation. Thisresult is an extension of a well-known property ofcontinuous-time systems (see[13, p. 354]). Then, inSection 4, it will be shown that, even when the sys-tem matrixA is singular and the time-reversed system� is not defined, these formulas still provide two ma-trices, sayP∗ andA∗, that can be exploited to derivea parametrization of all the solutions of the extendedsymplectic system.

Theorem 3. Let A be invertible. LetP+ and P+ bethe positive semidefinite solutions ofDARE(�) andDARE(�), respectively. Then, the matrix� := P+ +P+ is positive definite and it is the inverse of the uniquesolution W of the Stein equation

A+WAT+ −W + B(R + BTP+B)−1BT = 0. (12)

Proof. First, suppose thatL := R − STA−1B is in-vertible. In this case, DARE(�) has a maximal posi-tive semidefinite solutionP+. By virtue of Theorem 1,the matrix−P+ is a solution of DARE(�), and beingnegative semidefinite it is indeedP−. Hence,� equalsthe differenceP+−P−, and satisfies the homogeneousalgebraic Riccati equation

� = AT+�A+ + AT+�BR−11 B

T�A+, (13)

whereR1 := R + BTP+B − BT�B; moreover,�0 isa solution of (13) if and only if�0=P+ −P0, whereP0 is a solution of DARE(�) (see[16, Lemma 3.2]).Now, consider the Stein equation (12). Since(A,B)is reachable,(A+, B) is such, and hence(A+, B(R+BTP+B)−1/2) remains reachable in view of the non-singularity of (R + BTP+B)−1/2. By virtue of The-orem 2, it follows that Eq. (12) does admit a uniquesolutionW, and this solution is positive definite. More-over, since the extended symplectic pencil is regular


and has no generalized eigenvalues at zero, we mayconclude thatA+ is non-singular. It follows that (12)can be written as

W = A−1+ WA−T+ − A−1+ B(R + BTP+B− BTW−1B + BTW−1B)−1

× BTA−T+=A−1+ WA−T+ − A−1+ �B(R + BTP+B

− BTW−1B + BTW−1A+A−1+ �B)−1

× BT�TA−T+ ,

where� := WA−T+ AT+W−1=I . Both sides of the for-mer can be inverted, and by applying the well-knownmatrix inversion lemma, we get

W−1 = AT+W−1A+ + AT+W−1BR−11 B

TW−1A+.

Conversely, the same calculation in the reverse ordershows that if� is a non-singular solution of (13), then

�−1

is a solution of (12). As a result,W−1 is theunique positive definite solution of (13). Nowwe showthatW−1 = �. As already observed, sinceW−1 is asolution of (13), we haveW−1 = P+ − P0, whereP0is a solution of DARE(�). On the other hand,P− isthe minimal solution of DARE(�), henceW−1 = P+−P0�P+ −P− =�. It follows that� is positive defi-nite as well. SinceW−1 is the unique positive definitesolution of (13), it follows that� =W−1.Now, let L be singular. A real�>0 can be found

such that, for any ∈ (0, �), by definingR := R +Im, the matrix

L := R − STA−1B (14)

isnon-singular.Chooseforexample� := min{|Re{�}| :� ∈ �(L)\{0}}. Note that for all ∈ [0, �) the matrix

� :=[Q S

ST R

]

is symmetric positive semidefinite. In fact, more istrue: for all 0�1�2< �we have 0��1��2.Let � := (A,B,�) and � be the correspondingtime-reversed Popov triplet. Since� = �T

�0, twomatricesC andD exist such that� can be par-titioned as� = F T F, with F := [C D], whereD is full-column rank sinceR = DT

D>0 for all in (0, �). The injectivity ofD for all ∈ (0, �),assumptions (A1)–(A2) and the non-singularity of

L ensure the existence and uniqueness of the pos-itive semidefinite solution of DARE(�) and ofDARE(�), here denoted byP+, and P+,, respec-tively. Moreover,P+, and P+, are continuous in[0, �) since the mapping −→ � is continuous in[0, �) and monotonically non-decreasing on[0, �),i.e. P+, −→ P+,0 = P+ andP+, −→ P+,0 = P+ as −→ 0+ (see1 [17, p. 107]). Let A+, be the sta-ble closed-loop matrix corresponding toP+, and letV := B(R + BTP+B)−1BT. Note thatA+, andVare continuous in[0, �) asR andP+, are continuousin [0, �). It follows that the solutionW of the Steinequation

A+,WAT+, −W + V = 0 (15)

is continuous in[0, �). Moreover,W is invertible byvirtue of Theorem 2 since the pair(A+,, V) is reach-able, as can be proved by using the same argumentspresented in the case ofL invertible. It follows thatW is invertible by virtue of Theorem 2, and its in-verse� := W−1

is continuous in[0, �). Now, sincefor all ∈ (0, �) the matrixL is non-singular, it fol-lows thatP+, + P+, = � for all ∈ (0, �). There-fore, since the two functionsP+, + P+, and� arecontinuous in[0, �) and they are equal in(0, �), thenthey are equal at = 0 as well. As a consequence,�0=P+,0+ P+,0=P+ + P+ =�. By the continuity of� in [0, �) it follows that�−1 is the unique solutionof (15) at = 0, which is exactly (12). �

Theorem 4. Let A be invertible. LetP+ and P+ bethe positive semidefinite solutions ofDARE (�) andDARE(�), respectively. LetA+ be the stable closed-loop matrix of the time-reversed system, defined by

A+ := A− B(R + BTP+B)−1(BTP+A+ ST). (16)The following relation holds:

A+ = �−1AT+�. (17)

Proof. First suppose thatL := R − STA−1B is in-vertible. As already seen in the proof of Theorem 3,

1 In [17, p.107], the further assumption of full-column ranknessof matrix D0 is not necessary here: in fact in that context, theinjectivity of matrixD for all ∈ [0, �) guarantees the existenceand uniqueness of the positive semidefinite solution of DARE(�),which is here ensured by assumptions (A1)–(A2).


in this case we haveP− = −P+. Define (as it is donein [6, p. 107])

G(P−) := R + BTP−B,H(P−) := ATP−B + S,K(P−) := ATP−A− P− +Q.By exploiting DARE(�), which can be written asK(P ) = H(P )G−1(P )HT(P ), and the identityL =G(P−)−HT(P−)A−1B (see[6, p. 107]), we get

LTG−1(P−)HT(P−)= (G(P−)−HT(P−)A−1B)TG−1(P−)HT(P−)=HT(P−)− BTA−TK(P−)= ST + BTA−TP− − BTA−TQ,

yielding the following identity:

(R − BTA−TS)−1(ST + BTA−TP− − BTA−TQ)= (R + BTP−B)−1(BTP−A+ ST). (18)

On the other hand, note thatA−1+ , in view of the well-known matrix inversion lemma, can be expressed as

A−1+ = A+ AB(R − BTP−B− (ST − BTP−A)AB)−1(ST − BTP−A)A

=A− B(R + STB)−1(STA− BTP−)=A− B(R − BTA−TS)−1

× (ST + BTA−TP− − BTA−TQ).

Hence, by taking into account the latter, (9) and (18),it follows thatA− = A−1+ . On the other hand, by using(8) and (9) and by exploiting the fact that bothP+ andP− are solutions of DARE(�), it can be checked thatA− = �−1A−T+ �. Hence (17) holds.Now let L be singular. Consider a real�>0 such

that for any ∈ (0, �) the matrixL defined in (14)is non-singular. Let� be defined as in the proof ofTheorem 3 and let� be its time-reversed represen-tation. The existence and uniqueness of the positivesemidefinite solutions of DARE(�) and DARE(�),sayP+, and P+,, respectively, and their continuityin [0, �) was proved in the proof of Theorem 3. It wasalso shown that their difference� is continuous andpositive definite in[0, �), whereasA+, is continuousin [0, �). As a consequence, the product�−1

AT+,� is

continuous in[0, �). Moreover, sinceA+, is definedby applying (16) to the time-reversed representation of

�, A+,is obviously continuous in[0, �). The func-tions A+, and�−1

AT+,� are continuous in[0, �)

and equal in(0, �). It follows that they are equal at = 0. Hence (17) follows. �

4. Optimal state-costate trajectories and controllaw

In this section, it is first shown how the solutions ofthe extended symplectic system can be parametrizedin closed form. Then, as a second step, we show how toexploit such parametrization for the solution of Prob-lems 1–4.

4.1. Solutions of the extended symplectic system

The following theorem parametrizes the set of rep-resentatives (in the sense of Remark 1) of the solutionsof the extended symplectic system (6) in terms of twon-dimensional vectorsp andq.

Theorem 5. Let assumptions(A1)–(A2)hold. Denoteby P+ the stabilizing solution ofDARE(�) and by�the inverse of the(unique) positive definite solution ofthe Stein equation(12).LetP∗ := � − P+,

A∗ := �−1AT+�, (19)

and letK+ andK∗ be the stabilizing gains defined by

K+ := (R + BTP+B)−1(BTP+A+ ST), (20)

K∗ := (R + BTB)−1(BT(AA∗ − I )+ STA∗ − BTP∗). (21)

The set of representatives of the solutions of the ex-tended symplectic system(6) is parametrized by thefollowing expression in terms ofp, q ∈ Rn:

[x(k)

�(k)u(k)

]=

[I

P+−K+

]Ak+p+

[A∗

−P∗A∗−K∗

]

×Akf−k−1∗ q 0�k�kf − 1,[

I

P+0

]Akf+ p +

[I

−P∗0

]k = kf .

(22)

Proof. First, supposeA to be invertible. By virtue ofTheorems 3 and 4, it is found thatP∗ = P+ andA∗ =


A+. Now we prove thatK∗ = K+, whereK+ is thestabilizing gain matrix for�, defined as

K+ := (R + BTP+B)−1(BTP+A+ ST). (23)

As a consequence of (23) andA∗ = A+ we have

AA∗ − BK+ = I, (24)

which can be proved by direct substitution. DARE(�)can be written as

A−T(P∗ +Q)A−1 − P∗ − (A−TS − A−T

(P∗ +Q)A−1B)K+ = 0. (25)

By premultiplying (25) byBT and by adding the iden-tity thus obtained with (23), one gets

BTP∗ = STA∗ − RK+. (26)

From the injectivity of[BT R]T, it follows thatBTB+R is invertible. Hence, by premultiplying (24) byBT

and taking into account (26), we getK+ =K∗.We are now ready to show that (22) are indeed so-

lutions of the ESS(�). First, letk < kf −1. Substitute(22) in (6), and note that the terms inp are equal since[

I

−ATP+−BTP+

]A+ =

[A− BK+

Q− P+ − SK+ST − RK+

]. (27)

Indeed, the first block-row equation holds by definitionof A+ and the remaining are equivalent to (7) with(20). Likewise, the terms inq are equal since[

I

ATP∗BTP∗

]A∗ =

[AA∗ − BK∗

QA∗ + P∗A∗ − SK∗STA∗ − RK∗

]A∗ (28)

holds owing to (24), (26) and by noting that the secondblock-row equality follows by premultiplying (25) by−AT.Now, let k = kf − 1. Since (27) and (28) still hold,

relation (22) satisfies (6) fork = kf − 1 as well.Now, letAbe singular. A preliminary state feedback

u(k)=Fx(k) can be performed so as to obtain a non-singular systemmatrixAF := A+BF (note that this isalways possible since(A,B) is reachable). As shownin [6, p. 96], to this feedback transformation therecorresponds a newPopov triplet�F := (AF , BF ,�F )

with BF = B and

�F :=[QF SFSTF RF

],

whereQF := Q + SF + F TST + F TRF , RF := R

and SF := S + F TR. Note thatP+ is the maximalsolution of DARE(�) if and only if it is the maxi-mal solution of DARE(�F ) for all F (see[6, p. 100]).Moreover,A+, � andA∗ are independent ofF. As aconsequence, Eq. (19) provides the stable closed-loopsystem matrix of the time-reversed representation of�F , say�F , for anyF such thatAF is non-singular.Hence, Eq. (22), written with respect to�F and to�F , furnishes for allp, q ∈ Rn solutions of ESS(�F ).However, by taking into account (3)–(5), it can beeasily checked thatpF (k) := [xTF (k) �TF (k) u

TF (k)]T

is a solution of ESS(�F ), if and only if p(k) :=[xT(k) �T(k) uT(k)]T, wherex(k) := xF (k), �(k) :=�F (k) andu(k) := uF (k) + FxF (k), is a solution ofthe ESS(�). Note that[

I

−ATFP+−BTFP+

]A+ =

[AF − BF (K+ + F)

QF − P+ − SF (K+ + F)STF − R(K+ + F)

],

(29)

which follows after simple algebraic manipulations:the equality of the first block-row derives from thedefinition ofA+; the second is implied by the relationobtained by (27) written with respect to�F and by thedefinition ofK+; the third follows from (27) as well.Moreover, there holds[

I

ATFP∗BTFP∗

]A∗

=[

AFA∗ − BF (K∗ + FA∗)QFA∗ + P∗A∗ − SF (K∗ + FA∗)

STFA∗ − RF (K∗ + F)

]A∗ (30)

which follows from (28) written with respect to�F .As a result of (29) and (30), it follows that for allk ∈{0, . . . , kf − 1}[xF (k)

�F (k)uF (k)

]=

[I

P+−K+ − F

]Ak+p

+[

A∗−P∗A∗

−K∗ − FA∗

]Akf−k−1∗ q (31)

for anyp andq in Rn. Eq. (31) shows that the vectorp(k) defined above is a trajectory of (22).Conversely, let us show that all functions satisfying

ESS(�) can be expressed by means of (22). To this


aim, we show that (22) has 2n linearly independenttrajectories. Let in fact

V1 :=[

I

P+−K+

], V2 :=

[A∗

−P∗A∗−K∗

].

It is easy to verify thatFV 1A+ = GV 1 andFV 2 =GV 2A∗, whereF andGare defined in Section 3. It fol-lows thatV1 andV2 are basis matrices for two deflat-ing subspaces of the symplectic pencilzF −G, sinceA+ −�I andI−�A∗ are both regular pencils (see e.g.[15]). From[9, Theorem 1.6.5], it follows thatV :=im V1∩ im V2 is still a deflating subspace ofzF −G.Moreover, since the generalized eigenvalues ofzF−Gcorresponding to the deflating subspaces imV1 andim V2 are the eigenvalues ofA+ (hence stable) and thereciprocal of the eigenvalues ofA∗ (hence with mod-ulus greater than 1 and possibly infinity), respectively,the set of generalized eigenvalues ofzF − G corre-sponding toV is empty. Hence,V= {0}. Hence, forany given pairp, q ∈ Rn, the two trajectoriesV1Ak+pandV2A

kf−k−1∗ q are linearly independent. Therefore,

the dimension of the linear space of trajectories in (22)is given by the sum of the dimensionsn1 andn2 ofthe subspaces of trajectories of (22) corresponding top = 0 and toq = 0, respectively. Settingk = 0 andk = kf , we easily see thatn1 = n andn2 = n, respec-tively. In conclusion, (22) has 2n linearly independentsolutions. Since the dimension of the linear space ofnon-equivalent solutions of ESS(�) is exactly 2n (seeRemark 1), it follows that (22) represents the completeset of solutions of ESS(�) with u(kf )= 0. �

4.2. Optimal trajectories

In Theorem 5 it has been shown that under assump-tions (A1)–(A2) the matricesP+ andP∗ yield an ex-plicit formula parametrizing in terms ofp and q allthe solutions of the extended symplectic system (6),which is a set of necessary conditions for an optimum.Now, from this set of trajectories we select those thatsatisfy the boundary conditions, so that the trajectoriesthus obtained are solutions of the complete set of Pon-tryagin equations written for the optimal control prob-lem considered. Note that, in general, the Pontryaginequations are only necessary for an optimum. How-ever, in the present case, they are also sufficient, bythe convexity of Problems 1–4 and by the reachabil-

ity of (A,B) (the latter ensuring that a state and inputfunctions satisfying the constraints of any of Problems1–4 exist).Now we show how to compute the values of the pa-

rametersp andq in terms of the boundary conditions.Let := [pT qT]T.

Problem 1 (Assigned initial and final states). In thiscase the boundary equations arex(0)=x0 andx(kf )=xf . By evaluating(22) at k = 0 andk = kf we get

�1 =N1 where �1 :=[x0xf

]and

N1 :=[I A

kf∗Akf+ I

]. (32)

The corresponding values of p and q can be computedby matrix inversion, sinceN1 is invertible. In fact, forany x0, xf ∈ Rn, a state trajectory such thatx(0) =x0 and x(kf ) = xf exists sincekf is supposed to begreater than the reachability index. Hence, an optimaltrajectory for Problem1 exists by the convexity ofProblem1 and, as a result, there exist p and q suchthat (32)holds. The arbitrariness ofx0 andxf impliesthatN1 is invertible.

Problem 2 (Assigned initial state and weighted ter-minal state). Since the terminal state is quadraticallyweighted in the performance index, the Pontryagin setof equations includes a boundary condition on�(kf ),expressed by�(kf ) = Pf x(kf ). From this equationand from(22)we obtain

(Pf − P+)Akf+ p + (Pf + P∗)q = 0. (33)

Equationx(0)=x0 and(33), in view of(22)evaluatedat k = 0, leads to the compact expression

�2 =N2 where �2 :=[x00

]and

N2 :=[

I Akf∗

(Pf − P+)Akf+ Pf + P∗

]. (34)

If N2 is invertible, p and q are obtained from the for-mer by inversion. IfN2 is singular, the convexity ofProblem2 ensures the existence of an optimal statetrajectory. Hence�2 ∈ imN2. As a consequence, theset of all parameters p and q providing optimal solu-tions is given by=N+

2 �2+Kv, where K be a basis


matrix for ker N2. Then, the set of all optimal statetrajectories are given by(22), where the values of pand q are parametrized by the latter.

Problem 3 (Assigned terminal state and weightedinitial state). Since the initial state is quadraticallyweighted in the performance index, the Pontryagin setof equations includes a boundary condition on�(0),expressed by�(0)= −P0x(0).We obtain(P0 + P+)p + (P0 − P∗)A

kf∗ q = 0. (35)

Equationsx(kf )= xf and (35), in view of(22) eval-uated atk = kf , lead to the compact expression

�3 =N3 where �3 :=[0xf

]and

N3 :=[P0 + P+ (P0 − P∗)A

kf∗Akf+ I

]. (36)

Let K be a basis matrix for kerN3. Again,p andqcan be obtained by=N+

3 �3+Kv, since�3 ∈ imN3.

Problem 4 (Weighted initial and terminal states).Relation(22) evaluated atk = 0 and k = kf can beexpressed in compact form as[00

]=N4

[p

q

]where

N4 :=[

P0 + P+ (P0 − P∗)Akf∗

(Pf − P+)Akf+ Pf + P∗

]. (37)

Let K be a basis matrix forker N4. Clearly, p and qcan be obtained by =Kv.

4.3. Optimal cost

In this section, we present general expressions forthe optimal value of the cost function corresponding tothe four optimal control problems described in Section2. First, we express such value as a quadratic form in := [pT qT]T.

Lemma 1. The following identities hold.

P+ − AT+P+A+ −Q+ SK+ +KT+ST

−KT+RK+ = 0, (38)

P∗ − AT∗P∗A∗ − AT∗QA∗ + AT∗SK∗+KT∗ STA∗ −KT∗RK∗ = 0, (39)

AT+P∗ − P∗ −QA∗ + SK∗ +KT+STA∗−KT+RK∗ = 0. (40)

Proof. The first equality reduces to DARE(�) writtenwith respect toP+ by virtue of (8). The second andthird identities follow from (24), (26) and (28). �

Theorem 6. Consider Problem i, i ∈ {1,2,3,4}. Theoptimal cost isJ o = TVi, where

V1

:=[P+ − (AT+)kf P+A

kf+ (AT+)kf P∗ − P∗A

kf∗(AT∗ )kf P+ − P+A

kf+ P∗ − (AT∗ )kf P∗A

kf∗

],

(41)

V2

:=

P+ + (AT+)kf (AT+)kf (Pf + P∗)

×(Pf − P+)Akf+ −P∗A

kf∗(Pf − P+)A

kf+ (Pf + P∗)− (AT∗ )kf

+(AT∗ )kf P+ ×P∗Akf∗

,(42)

V3

:=

(P+ + P0)− (AT+)kf (AT+)kf P∗

×P+Akf+ +(P0 − P∗)A

kf∗(AT∗ )kf (P+ + P0) ×P∗ + (AT∗ )kf

−P+Akf+ (P0 − P∗)A

kf∗

,(43)

V4 := 0. (44)

Proof. Consider Problem 1. From (2) and (22), theoptimal cost can be written as

J o =kf−1∑k=0

[pT(AT+)k qT(AT∗ )kf−k−1]

×[M11 M12MT12 M22

] [Ak+p

Akf−k−1∗ q

],

with

M11 := Q− SK+ −KT+ST +KT+RK+= P+ − AT+P+A+,

M12 := QA∗ − SK∗ −KT+STA∗ +KT+RK∗=AT+P∗ − P∗A∗,

M22 := AT∗QA∗ − AT∗SK∗ −KT∗ STA∗ +KT∗RK∗= P∗ − AT∗P∗A∗,


where the equalities follow from Lemma 1. Note that

kf−1∑k=0

(AT+)k(P+ − AT+P+A+)Ak+

= P+ − (AT+)kf P+Akf+ ,

kf−1∑k=0

(AT+)k(AT+P∗ − P∗A∗)Akf−k−1∗

= (AT+)kf P∗ − P∗Akf∗ ,

kf−1∑k=0

(AT∗ )kf−k−1(P∗ − AT∗P∗A∗)Akf−k−1∗

= P∗ − (AT∗ )kf P∗Akf∗ ,

which, by substitution, yield (41). Consider Problem2. By virtue of (22) evaluated atk= kf , the quadraticform xT(kf )Pf x(kf ) is summed by (41) to obtain(42). Consider Problem 3. By virtue of (22) evaluatedat k = 0, the quadratic formxT(0)P0x(0) is summedby (41) to obtain (43). �

Alternative expressions of the optimal cost are pro-vided as a quadratic form in the assigned states.

Corollary 1. The following results hold.

• Consider Problem1. LetN1 be defined in(32) andV1 be defined in(41). Let x := [xT0 xTf ]T. The op-timal cost isJ o = xTN−T

1 V1N−11 x.

• Consider Problem2. LetM ∈ R2n×n be such that = Mx0. Let V2 be defined in(42). The optimalcost isJ o = xT0MTV2Mx0.

• Consider Problem3. Let N ∈ R2n×n be such that = Nxf . Let V3 be defined in(43). The optimalcost isJ o = xTf NTV3Nxf .

5. An example and concluding remarks

5.1. An example

The formulas that provide the optimal value of thecost can be exploited to solve non-standard optimalcontrol problems. For example, consider the casewhen

x(0)= x0 and the optimal cost to be minimized isJ (x, u)= (x(kf )− �f )

TPf (x(kf )− �f )

+kf−1∑k=0

[xT(k) uT(k)]

×[Q S

ST R

] [x(k)

u(k)

], (45)

where�f ∈ Rn is given. This non-standard LQ prob-lem can be tackled as follows. First, determine themin-imum of the sum in (45) withx(0)=x0 andx(kf )=xf .Such a minimum is a quadratic form inx0 andxf (seeTheorem 6 and Corollary 1):

J ′(x0, xf )= minx(kf )=xf

kf−1∑k=0

[xT(k) uT(k)]

×[Q S

ST R

] [x(k)

u(k)

]

= [xT0 xTf ]N−T1 V1N

−11

[x0xf

],

whereN1 is defined in (32) andV1 is defined in (41).Now, the minimum ofJ (x, u) can be attained by per-forming a minimization overxf :

minJ (x, u)= minxf

[(xf − �f )TPf (xf − �f )

+ J ′(x0, xf )].Therefore, the problem reduces to the minimizationof a static quadratic function inxf , which is standardand easy to solve. Oncexf is determined,p and qare computed by inversion of Eq. (32), and (22) pro-vides the optimal control. To the best of the authors’knowledge, this problem has not been solved (and isnot easy to solve) with the classical approach to theLQ problem.

5.2. Concluding remarks

A novel approach has been presented for the solu-tion of a class of finite-horizon linear quadratic prob-lems in the discrete-time case. It has been shown thatthe set of solutions of the extended symplectic system,which represents necessary conditions for optimality,can be parametrized in closed-form in terms of twovectorsp and q. Their values can be determined by


imposing the boundary conditions corresponding tothe particular problem considered.This procedure is particularly convenient since it

yields reliable software algorithms for the solution ofall the optimal control problems considered herein: infact, the result presented in Theorem 5 involves thestabilizing solution of a discrete-time algebraic Riccatiequation and the solution of a Stein equation, whichmay be computed by standard and robust algorithmsavailable in any control package (see the MATLAB�

routinesdare.m and dlyap.m ). Furthermore, theexpression of the optimal control is given in terms ofpowers of strictly stable matrices in the overall timeinterval, thus ensuring that the solution proposed isrobust even for large time-horizons.

References

[1] B.D.O. Anderson, J.B. Moore, Optimal Control: LinearQuadratic Methods, Prentice-Hall International, London,1989.

[2] F. Blanchini, F.A. Pellegrino, Relatively optimal control andits linear implementation, IEEE Trans. Automat. Control AC-48 (2003) 2151–2162.

[3] D.J. Clements, B.D.O. Anderson, Singular optimal control:the linear quadratic problem, in: A.V. Balakrishnan, M. Thoma(Eds.), Lecture Notes in Control and Information Sciences,vol. 5, Springer, New York, 1978, p. 93.

[4] A. Ferrante, B.C. Levy, Canonical form for symplectic matrixpencils, Linear Algebra Appl. 274 (1998) 259–300.

[5] A. Ferrante, G. Marro, L. Ntogramatzidis, Employing thealgebraic Riccati equation for the solution of the finite-horizon

LQ problem, in: Proceedings of the 42nd IEEE Conferenceon Decision and Control (CDC 2003), Hyatt Regency Maui,Hawaii, USA, December 2003.

[6] V. Ionescu, C. Oarˇa, M. Weiss, Generalized Riccati theoryand robust control, a Popov function approach, Wiley, 1999.

[7] E. Kondo, T. Kiyota, S. Sunaga, Closed-loop expressions offixed-end-point LQ optimal control for digital systems, in:Proceedings of the 12th IFAC Congress, vol. 1, July 1993,pp. 173–178.

[8] H. Kwakernaak, R. Sivan, Linear Optimal Control Systems,Wiley, New York, 1972.

[9] P. Lancaster, L. Rodman, Algebraic Riccati Equations,Clarendon Press, Oxford, 1995.

[10] B.C. Levy, A. Ferrante, Characterization of stationary discrete-time Gaussian reciprocal processes over a finite interval,SIAM J. Matrix Anal. Appl. 24 (2) (2002) 334–355.

[11] F.L. Lewis, V. Syrmos, Optimal Control, Wiley, New York,1995.

[12] G. Marro, D. Prattichizzo, E. Zattoni, A geometric insightinto the discrete time cheap and singular LQR problems,IEEE Trans. Automat. Control 47 (1) (2002) 102–107.

[13] B.P. Molinari, The time-invariant linear-quadratic optimalcontrol problem, Automatica 13 (1977) 347–357.

[14] L. Ntogramatzidis, A simple solution to the finite-horizon LQproblem with zero terminal state, Kybernetika 39 (4) (2003)483–492.

[15] C. Oara, P. Van Dooren, An improved algorithm for thecomputation of structural invariants of a system pencil andrelated geometric aspects, Systems Control Lett. 30 (1997)39–48.

[16] A.C.M. Ran, H.L. Trentelman, Linear quadratic problemswith indefinite cost for discrete time systems, SIAM J. MatrixAnal. Appl. 14 (3) (1993) 776–797.

[17] A. Saberi, P. Sannuti, B.M. Chen, H2 Optimal Control. Systemand Control Engineering, Prentice-Hall International, London,1995.

employing the algebraic riccati equation for a parametrization of the solutions of the...

Documents