model reduction notes- siep weiland tueindhoven_part 1

Model Reduction – 2013

Class 8

Department of Electrical EngineeringEindhoven University of Technology

Siep Weiland

Class 8 (TUE) Model Reduction – 2013 Siep Weiland 1 / 39

Outline

1 Function approximationspectral decompositionsproper orthogonal decompositions

2 Projection frameworklinear systemsnon-linear systemsinfinite dimensional systems

3 Reduction of spatial-temporal systemsa wave propagation examplehow to choose basis functions?how to derive coefficients?results on wave propagation (Fourier)results on wave propagation (POD)

4 Summary


Function approximation



Function approximation spectral decompositions

spectral decompositions

Image RGB decomposition



Decompositions of light and sound




Given function w ∈ W, find function wr ∈ Wr so as to minimize‖w − wr‖.

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5





0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5originalpolynomial degree 5

• Wr polynomials of degree ≤ r

• wr (x) = a0 + a1x + . . .+ arxr , r = 5





0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4



• wr (x) = a0 + a1x + . . .+ arxr , r = 10





0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4



• wr (x) = a0 + a1x + . . .+ arxr , r = 15





0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5originalFourier degree 5

• Wr trigonometric functions of frequency ≤ r2π/|X|• wr (x) = a0 + a1 cos(πx) + . . .+ ar cos(rπx), r = 5





0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4





Solution by projection

Projection framework

Equip function space W with inner product 〈·, ·〉

with

• Norm ‖w‖2 = 〈w ,w〉 induced by inner product

• Orthogonal projection Π :W →Wr onto Wr

Projection theorem

w∗r in Wr minimizes ‖w − wr‖ if and only if






with



Projection theorem


w∗r = Πw






with



Projection theorem


w − w∗r ⊥ Wr






with



Projection theorem


〈w − w∗r , v〉 = 0 for all v ∈ Wr






with



Projection theorem


w∗r = U(U>U)−1U>w with im(U) =Wr



Solution by spectral decompositions

Write w ∈ W as

w(x) =∑

k akϕk(x)

• ϕk basis functions in W that are orthonormal in the sense that

〈ϕk , ϕ`〉 =

{1 if k = `

0 otherwise

• ak = 〈w , ϕk〉 are (Fourier) coefficients

If Wr = span(ϕ1, . . . , ϕr ) then

w∗r (x) =∑r

k=1 akϕk(x)

minimizes ‖w − wr‖ over all wr ∈ Wr .



Spectral decompositions - example

Joseph Fourier (1768-1830)

• W = L2([0, 1]) with innerproduct

〈f , g〉 =

∫ 1

0f (x)gH(x) dx

• Fourier basis

ϕk(x) = e ikπx

• Optimal approximation of w

w∗r (x) =r∑

k=0

〈w , ϕk〉ϕk(x)




0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−0.2

0

0.2

0.4

0.6

0.8

1


r = 4


〈f , g〉 =

∫ 1

0f (x)gH(x) dx

• Fourier basis

ϕk(x) = e ikπx


w∗r (x) =r∑

k=0

〈w , ϕk〉ϕk(x)




0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−0.2

0

0.2

0.4

0.6

0.8

1


r = 8


〈f , g〉 =

∫ 1

0f (x)gH(x) dx

• Fourier basis

ϕk(x) = e ikπx


w∗r (x) =r∑

k=0

〈w , ϕk〉ϕk(x)




0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−0.2

0

0.2

0.4

0.6

0.8

1


r = 20


〈f , g〉 =

∫ 1

0f (x)gH(x) dx

• Fourier basis

ϕk(x) = e ikπx


w∗r (x) =r∑

k=0

〈w , ϕk〉ϕk(x)




0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−0.2

0

0.2

0.4

0.6

0.8

1


r = 100


〈f , g〉 =

∫ 1

0f (x)gH(x) dx

• Fourier basis

ϕk(x) = e ikπx


w∗r (x) =r∑

k=0

〈w , ϕk〉ϕk(x)




Josiah Wilard Gibbs (1839-1903)


〈f , g〉 =

∫ 1

0f (x)gH(x) dx

• Fourier basis

ϕk(x) = e ikπx


w∗r (x) =r∑

k=0

〈w , ϕk〉ϕk(x)




Gives strong convergence

limr→∞

∫ 1

0|w(x)− wr (x)|2 dx = 0

but possibly

limr→∞

|w(x)− wr (x)| 6= 0

for some x .(no pointwise convergence).


〈f , g〉 =

∫ 1

0f (x)gH(x) dx

• Fourier basis

ϕk(x) = e ikπx


w∗r (x) =r∑

k=0

〈w , ϕk〉ϕk(x)



The POD basis problem

Given a collection of M functions w1, w2, . . . , wM in W

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−4

−3

−2

−1

0

1

2

3

4

5

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

Find r dimensional subspace Wr = span(ϕ1, . . . , ϕr ) such that the averageerror

1

M

M∑j=1

‖wj − Πwj‖2

is minimal.


Function approximation proper orthogonal decompositions



Given data w1, . . . ,wM ∈ W, find orthonormal basis {ϕk , k = 1, 2, . . .} ofW such that the error

J(ϕ1, . . . , ϕr ) =M∑j=1

‖wj −r∑

k=1

〈wj , ϕk〉ϕk‖2

is minimal for all truncation levels r .

• Basis will be data dependent

• Needs inner product 〈f , g〉 on W.

• Orthonormal means

〈ϕk , ϕ`〉 =

{1 if k = `

0 otherwise


Student

Line


Solution of POD basis problem

Theorem

Suppose that W = RN . An orthonormal basis {ϕk , k = 1, . . . ,N} of W isa POD basis if and only if

WW>ϕk = λkϕk

where λ1 ≥ · · · ≥ λn, n = rank(W ) and

W =(w1 · · · wM

)∈ RN×M .

Moreover, in that case

J(ϕ1, . . . , ϕr ) =∑k>r

λk

POD basis can be obtained from eigenvalue decomposition!


Student

Line

Student

Line


Proof

For any orthonormal basis we have

J(ϕ1, . . . , ϕr ) =M∑j=1

‖∑k>r

〈wj , ϕk〉ϕk‖2 =

=∑k>r

M∑j=1

〈wj , ϕk〉2 =∑k>r

ϕ>k WW>ϕk

(if): If WW>ϕk = λkϕk then J =∑

k>r λk is minimal for all r sinceλ1 ≥ · · · ≥ λn.

(only if): If {ϕk}Nk=1 is a POD basis and {ψ`}n`=1 eigenvectors of WW>, then

J =∑k>r

ϕ>k WW>ϕk =∑k>r

n∑`=1

λ` 〈ϕk , ψ`〉

is minimal for all r only if 〈ϕk , ψ`〉 = δk,`. But then ϕk is ev of WW>.



Computation of POD basis by SVD

Need to solve WW>ϕk = λkϕk

• Suppose n = rank(W ). Compute singular value decomposition

W = UΣV>

where• U =

(u1 · · · uN

)∈ RN×N , unitary

• V =(v1 · · · vM

)∈ RM×M unitary

•

Σ =

(Σ 00 0

); Σ = diag(σ1, . . . , σn)

• Then WW>uk = σ2kuk and λk = σ2

k .

ϕk = uk is POD basis


Student

Line

Student

Line


Computation of POD basis by EVD

• data matrix W wide (N ≤ M)Compute eigenvalue decomposition

WW>uk = λkuk , ‖uk‖ = 1, k = 1 . . . , n

• data matrix W tall (N ≥ M)Compute eigenvalue decomposition

W>Wvk = λkvk , ‖vk‖ = 1, k = 1 . . . , n

Set uk = 1λ2kWvk

ϕk = uk is POD basis


Projection framework linear systems


Complex model

• Model {w = Aw + Bu

y = Cw + Du

• Variable projection

w ≈ Uwr

• Vector field projection

imV ⊥ [w − Aw − Bu]

Reduced order model

• Combined:

w ≈ Uwr

V>w = V>Aw + V>Bu

• Reduced order model{V>Uwr = Arwr + Bru

y = Crwr + Dru

whereAr = V>AU, Br = V>B,Cr = CU.



Projection matrices U and V

Ar= V>

A U

U and V projection matrices that project on imU and imV

U>U = Ir , V>V = Ir

Reduction methods differ in selection of U and V .



How to select U and V ?

• Use gramiansbalancing type of algorithms

• Use eigenvectors of Amodal truncations/time scale separations

• Use U = VGalerkin projections

• Use time series data• measurement w(t), t = 1, . . . ,M• use SVD to pick relevant directions

• Use frequency domain data• measurement W (ω), ω1, . . . ωM

• use SVD to pick relevant directions

• Use Krylov spacesnot discussed in this course. . .


Student

Line

Projection framework non-linear systems

Projection framework for non-linear systems

Complex nonlinear model

• Model{w(t) = f (w(t), u(t))

y(t) = g(w(t), u(t))


w ≈ Uwr


imV ⊥ [w − f (w , u)]

Reduced order model

• Combined:

w ≈ Uwr

V>w = V>f (w , u)

• Reduced order model{V>Uwr = fr (wr , u)

y = gr (wr , u)

wherefr (wr , u) = V>f (Uwr , u),gr (wr , u) = g(Uwr , u).


Projection framework infinite dimensional systems

Projection framework for distributed systems

PDE or distributed system

• Model{∂w∂t = F (∂w∂x , . . . , )

y(x , t) = g(w(x , t), u(x , t))


w(x , t) ≈ Uwr (x , t)

more difficult to define !!


imV ⊥[∂w

∂t− F (

∂w

∂x, . . . , )

]

Reduced order model

• Combined:

w(x , t) ≈ Uwr (x , t)

V>∂w

∂t= V>F (

∂w

∂x, . . . , )

• Reduced order model⟨v , ∂wr

∂t

⟩=⟨v ,F (∂wr

∂x , . . . , )⟩

wr = 〈u,w〉y(x , t) = g(wr , u)

where v ∈ imV , u ∈ imU and〈·, ·〉 is some inner product (seebelow)


Student

Line

Projection framework infinite dimensional systems

Projection framework for distributed systems

PDE or distributed system

• Model{∂w∂t = F (∂w∂x , . . . , )

y(x , t) = g(w(x , t), u(x , t))


w(x , t) ≈ Uwr (x , t)

more difficult to define !!


imV ⊥[∂w

∂t− F (

∂w

∂x, . . . , )

]

Reduced order model

• Combined:

w(x , t) ≈ Uwr (x , t)

V>∂w

∂t= V>F (

∂w

∂x, . . . , )

• Reduced order model⟨v , ∂wr

∂t

⟩=⟨v ,F (∂wr

∂x , . . . , )⟩

wr = 〈u,w〉y(x , t) = g(wr , u)

where v ∈ imV , u ∈ imU and〈·, ·〉 is some inner product (seebelow)


Reduction of spatial-temporal systems a wave propagation example

Reduction of spatial-temporal systems

Example: wave propagation equationConsider solutions w(x , t) of partial differential equation

∂2w

∂t2− κ2∂

2w

∂x2= 0

where• space: x ∈ X = [0, 1];• time: t ∈ T = [0, 1]• initial and boundary conditions

w(x , 0) = w0(x), w(0, t) = 0, w(1, t) = 0,∂w

∂t(x , 0) = w1(x)

We wish to make a spectral decomposition of solution:

w(x , t) =∑∞

k=1 ak(t)φk(x)



Wave propagation in a string

(click to animate)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−0.25

−0.2

−0.15

−0.1

−0.05

0

0.05

0.1

0.15

0.2

0.25

Solution wave equation for κ = 3.



Three questions

What basis functions ϕk(x) should we take ??

How do we obtain the coefficients ak(t) ??

How do we project to get a simple reduced order model ??


Reduction of spatial-temporal systems how to choose basis functions?

What basis functions ϕk should we take ??

To talk about

• normalized basis functions: we need structure of a normed vectorspace

• orthogonal basis functions: we need structure of an inner product.

For every t, the solution w(·, t) therefore needs to belong to an innerproduct space1: (

W, 〈·, ·〉)

that describes the relevant waves.

First need a relevant inner product – but what’s relevant here?

1Better even: a separable Hilbert space.Class 8 (TUE) Model Reduction – 2013 Siep Weiland 24 / 39


Choice of inner product

Possible inner product choices (1)

• Square integrable functions

W = L2([0, 1],R) = {f : [0, 1]→ R | ‖f ‖22 :=

∫ 1

0f 2(x)dx <∞}

with inner product

〈f , g〉 :=

∫ 1

0f (x)g(x)dx

• Basis This set has an orthonormal basis of Fourier modes:

1,√

2 cos(kπx),√

2 sin(kπx)

which implies for every f ∈ W the classical Fourier decomposition:

f (x) =∞∑k=1

ak√

2 cos(kπx) +∞∑`=1

b`√

2 sin(`πx) + cm




Possible inner product choices (2)• Square integrable functions

W = {f ∈ L2([0, 1],R) | f (0) = 0 = f (1)}

with same inner product.(This takes the initial conditions of the wave equation into account!)





W = {f ∈ L2([0, 1],R) | f (0) = 0 = f (1)}


• Basis: Has orthonormal basis of Fourier modes:

ϕk(x) =√

2 sin(kπx); x ∈ [0, 1]; k = 1, 2, · · ·

and implies the classical Fourier decomposition:

f (x) =∞∑k=1

ak√

2 sin(kπx)

with ak = 〈f , ϕk〉 =∫ 1

0

√2 sin(kπx)f (x)dx .





W = {f ∈ L2([0, 1],R) | f (0) = 0 = f (1)}


• Basis: Has orthonormal basis of Legendre polynomials:

ϕk(x) =8k

2 · k!

dk

dxk(x2 − x)k , k = 1, 2, · · ·

and implies a Legendre decomposition (not used in this course):

f (x) =∞∑k=1

akϕk(x)

with ak = 〈f , ϕk〉 =∫ 1

0 f (x)ϕk(x)dx .Class 8 (TUE) Model Reduction – 2013 Siep Weiland 26 / 39



Possible inner product choices (3)

• Discretization firstDiscretize X = [0, 1] in N disjoint intervals and assume that w(x , t) isonly interesting at samples xk = k∆x , ∆x = 1/N. Then

W = RN

with its usual (standard) inner product

〈f , g〉 = f >g .

• POD basisConsider M time samples w(x , t1), . . . , w(x , tM) of measured dataand store them in an N ×M snapshot matrix

Wsnap =

w(x1, t1) · · · w(x1, tM)...

...w(xN , t1) · · · w(xN , tM)



A data dependent basis

Now let Wsnap = UΣV> be an SVD of Wsnap with

U =(ϕ1 · · · ϕN

); ϕk ∈ RN .

Then vectors {ϕk , k = 1, . . . ,N} define an orthonormal basis of RN .

This is a POD basis!

Thus the time-averaged error

M∑`=1

‖w(x , t`)− wr (x , t`)‖2 =M∑`=1

N∑k=r+1

a2k(t`)

is minimal for all truncation levels r


Reduction of spatial-temporal systems how to derive coefficients?


Next question: Get coefficients ak(t)

Once an orthonormal basis {ϕk(x), k = 1, 2, . . .} has been decided upon,the coefficients ak(t) in

w(x , t) =∞∑k=1

ak(t)ϕk(x)

satisfy

ak(t) = 〈w(x , t), ϕk(x)〉 =

∫ 1

0w(x , t)ϕk(x)dx

Nice, but useless as we do not know w(x , t). . .




Alternative: Substitute expansion in PDE ∂2w∂t2 = κ2 ∂2w

∂x2 . This yields:

∞∑k=1

ak(t)ϕk(x) = κ2∞∑k=1

ak(t)ϕk(x)

Then project both sides on ϕn, n = 1, 2, . . ., to infer that⟨ ∞∑k=1

ak(t)ϕk(x), ϕn(x)

⟩=

⟨κ2∞∑k=1

ak(t)ϕk(x), ϕn(x)

⟩

Using the orthonormality of the inner product, this yields:

an(t) = κ2∑∞

k=1 ak(t) 〈ϕk(x), ϕn(x)〉, n = 1, 2, . . .




Now have a close look at:

an(t) = κ2∑∞

k=1 ak(t) 〈ϕk(x), ϕn(x)〉, n = 1, 2, . . .

• This is an ordinary differential equation in the time-varyingcoefficients an

• So we went from a PDE to an ODE!

• To solve it, we need to compute ϕk and 〈ϕk(x), ϕn(x)〉 for all k andall n.

not something to look forward to. . .

• We will be interested in a finite number of an’s only.

• To compute these, the infinite sum is replaced by a finite one too.


Reduction of spatial-temporal systems results on wave propagation (Fourier)

Results on wave propagation (Fourier)

Let’s try the Fourier basis!

• Fourier basis from “Possible choice 2”: ϕk(x) =√

2 sin(kπx).

• Then, ϕk(x) = −(k2π2)ϕk(x).

• Then, using orthonormality, the inner products simplify to

〈ϕk(x), ϕn(x)〉 =

{0 if k 6= `

−n2π2 if k = `

• Then, the coefficients satisfy a set of decoupled ODE’s:

an(t) = −κ2n2π2an(t); n = 1, 2, 3, · · ·

And we can explicitly solve these!




Indeed:an(t) = −κ2n2π2an(t); n = 1, 2, 3, · · ·

is a second order ordinary diff. eqn. which has as its solution

an(t) = An cos(ωnt) + Bn sin(ωnt); ωn = κnπ

where constants An, Bn follow from initial conditions:

An = an(0) =

∫ 1

0w0(x)ϕn(x) dx =

√2

∫ 1

0w0(x) sin(nπx) dx

Bn =1

ωnan(0) =

√2

ωn

∫ 1

0w1(x) sin(nπx) dx

This gives the complete solution




For equation lovers, the explicit expression for the solution is:

w(x , t) =∞∑k=1

ak(t)ϕk(x) =

∞∑k=1

[√2

∫ 1

0w0(x) sin(kπx) dx cos(ωkt)+

+

√2

ωk

∫ 1

0w1(x) sin(kπx) dx

]√

2 sin(kπx)

which satisfies the boundary conditions

w(0, t) = w(1, t) = 0; t ≥ 0

(by construction of the basis) and

w(x , 0) = w0(x);∂w

∂t(x , 0) = w1(x).



How do we get a simple reduced order model ??

The truncated expansion

wr (x , t) =∑r

k=1 ak(t)φk(x)

is the approximate solution of order r , and requires construction of ak(t)for 1 ≤ k ≤ r only.




(click to animate) Simulation with ϕk(x) =√

2 sin(kπx):

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−0.25

−0.2

−0.15

−0.1

−0.05

0

0.05

0.1

0.15

0.2

0.25 original degree=100

approximation degree=5

Solutions wave equation

position

Solution wave propagation with κ = 3, blue: exact solution, red: r = 5thorder Fourier approximation


Reduction of spatial-temporal systems results on wave propagation (POD)

Results on wave propagation (POD)

Let’s try the POD basis!

• POD basis from “possible choice 3”: ϕk(x) on discretized grid.

• Hence, ϕk(x) needs to be approximated numerically on the grid by

ϕk,xx(xj) :=ϕk(xj−1)− 2ϕk(xj) + ϕk(xj+1)

(δx)2δx grid size

• Compute inner products 〈ϕk,xx , ϕn〉, 1 ≤ k ≤ r

• Solve coupled ODE’s:

an(t) = κ2r∑

k=1

ak(t) 〈ϕk,xx , ϕn〉 , n = 1, 2, . . . , r .

• This is of the form a = Aa with a = col(a1, . . . , an).

Again, we can explicitly solve this (in Matlab)!


Reduction of spatial-temporal systems results on wave propagation (POD)

Results on wave propagation (POD)

Simulations: (click to animate)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−0.25

−0.2

−0.15

−0.1

−0.05

0

0.05

0.1

0.15

0.2

0.25original degree=100

Fourier approx. degree=5

POD approx. degree=5

Solutions wave equation

position

Solution wave equation for κ = 3, red: 5th order Fourier approximation,green: 5th order POD approximation.


Summary

Summary

• Most model reduction techniques require projection of the statevariable and state equations to define low order models

• This idea generalizes to nonlinear systems and systems described byPDE’s

• For systems described by PDE’s we consider spectral expansions

w(x , t) =∞∑k=1

ak(t)ϕk(x)

and their truncations as (approximate) solutions.• Functions ϕk are selected to form an orthonormal basis of some

Hilbert space/inner product space.• Many choices are possible, including data-based functions

(POD-method)• Once basis functions are fixed, coefficients ak satisfy ODE’s that we

can solve.• Low order POD approximations work well on wave propagation

example.

Previous class Next class


model reduction notes- siep weiland tueindhoven_part 1

Documents