L25 Numerical Methods part 5• Project Questions• Homework• Review• Tips and Tricks• Summary

1

H24

2

( 1) ( )

( 1) ( 1) ( 1)

( 1) ( ) ( )

( 1)( )

( 1) ( )

( 1) ( )

( ) ( + ) ( )'( )=0( ) ( ) ( ) 0

since( )

( ) 00

k k

k T k k

k k k

kk

k k

k k

f f ffdf f dd d

ddf

x x d

x x xx

x x dx d

x dc d

F

F

10.57 revisited

3

c1 4 0.29787234 0.057143 -4.8E-10c2 8 -0.021276596 -0.11429 1.58E-11c3 6 -0.170212766 0.114286 8.84E-10

||c|| 10.7703 0.3437 0.1714 0.0000

β 0.0010 0.2487 0.0000d1 -4 -0.301946582 -0.13224 4.77E-10d2 -8 0.013128112 0.117551 -1.6E-11d3 -6 0.164101403 -0.07347 -8.8E-10

-1.24771E-10 -1.2E-10 1.85E-18{=MMULT(TRANSPOSE(C13:C15),B19:B21)}=C13*B19+C14*B20+C15*B21

( 1) ( ) 0k k c d

( 1) ( ) 0k k c d

10.67

4

Prob10.67

Iteration 1 2 3 4 GRG

x1 1 2 4 4x2 1 0.5 2 2

c1 -4 -1 3.51E-08 -8.3E-07c2 2 -2 -5.2E-08 1.34E-06||c|| 4.5 2.2 0.0 0.0

β 0.25 7.85E-16 632.9881d1 4 2 -3.5E-08 -2.1E-05d2 -2 1.5 5.19E-08 3.15E-05

-1.22E-07 -7.6E-09 9.86E-14

α 0.25 1 5 5

xnew1 2 4 4 3.25 4xnew2 0.5 2 2 1.75 2

f (x) -5.5 -8 -8 -7.6875 -8

2 21 2 1 1 2

1 2

2 1 *

( ) 2 4 22 4 2( *) 4 2

x

f x x x x xx xf x x

x

c x

( 1) ( ) 0k k c d

( ) ( ) ( 1)

( ) ( ) ( 1) 2( )

( 1) ( )

Conjugate Gradient

( / )

k k k

k k k k

k k

search direction

step size

d c dc c

x x d

10.72

5

Prob 10.72

Iteration 1 2 3 4 GRGx1 1 0.382978723 0.257143 -1.6E-10x2 1 -0.234042553 -0.22857 3.85E-11x3 1 0.074468085 0.142857 1.19E-10

c1 4 0.29787234 0.057143 -2.4E-10c2 8 -0.021276596 -0.11429 7.03E-11c3 6 -0.170212766 0.114286 5.52E-10||c|| 10.8 0.3 0.2 0.0

β 0.00101856 0.248726 1.26E-17d1 -4 -0.301946582 -0.13224 2.44E-10d2 -8 0.013128112 0.117551 -7E-11d3 -6 0.164101403 -0.07347 -5.5E-10

-1.24771E-10 -6.6E-11 -8.1E-18

α 0.154255 0.416748768 1.944444 5

xnew1 0.382979 0.257142857 -1.6E-10 1.06E-09 0xnew2 -0.23404 -0.228571429 3.85E-11 -3.1E-10 0xnew3 0.074468 0.142857143 1.19E-10 -2.6E-09 0

f (x) 0.053191 0.028571429 5.38E-20 1.63E-17 0

2 2 21 2 3 1 2 2 3

1 2

2 1 3

3 2 *

( ) 2 2 2 2(1,1,1)

2 2( *) 4 2 2

4 2x

f x x x x x x x

x xf x x x

x x

xx

c x

( 1) ( ) 0k k c d

Prob 10.76

6

Prob 10.76

Iteration 1 2 3 4 GRGx1 1 3.072336 3.113395 2.937977x2 2 1.770152 0.322241 0.32624x3 3 2.069488 1.349908 0.760734x4 4 1.979566 2.12286 2.29332

c1 -1118 22.93872 38.65691 10.06746c2 124 239.4148 -51.58023 -0.341454c3 502 107.2371 99.79143 -1.66613c4 1090 -53.0963 -31.14542 4.609562||c|| 1644.8 268.6 122.8 11.2

β 0.026674 0.209012 0.00832d1 1118 6.883109 -37.21827 -10.37712d2 -124 -242.722 0.848449 0.348514d3 -502 -120.628 -125.004 0.626082d4 -1090 24.02137 36.16616 -4.308656

-0.00256 3.01E-06 -4.43E-05

α 0.00185 0.00597 0.00471 0.09420

xnew1 3.0723 3.1134 2.9380 1.9605 2.51E-10xnew2 1.7702 0.3222 0.3262 0.3591 0xnew3 2.0695 1.3499 0.7607 0.8197 0.001241xnew4 1.9796 2.1229 2.2933 1.8874 0.001241

f (x) 259.8004 44.5808 15.6172 11.0456 0.0000

2 2 4 41 2 3 4 2 3 1 4

31 2 1 4

31 2 2 3

33 4 2 3

33 4 1 4 *

( ) ( 10 ) 5( ) ( 2 ) 10( )(1,2,3,4)

2( 10 ) 40( )20( 10 ) 4( 2 )

( *) 10( ) 8( 2 )

10( ) 40( )x

f x x x x x x x x

x x x xx x x x

fx x x xx x x x

xx

c x

( 1) ( ) 0k k c d

Conjugate Gradient

7

( ) ( )

( 1) ( ) ( )

( ) ( ) ( 1)

( ) ( ) ( 1) 2( )

( 1) ( )

Steepest Descent

Conjugate Gradient

( / )

k k

k k k

k k k

k k k k

k k

search directionstep size

search direction

step size

d cx x d

d c dc c

x x d

Proof: http://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf

“Deflected” Steepest Descent

8

A comparison of the convergence of gradient descent with optimal step size (in green) and conjugate vector (in red) for minimizing a quadratic function associated with a given linear system. Conjugate gradient, assuming exact arithmetic, converges in at most n steps where n is the size of the matrix of the system (here n=2). Wik

http://en.wikipedia.org/wiki/Conjugate_gradient_method

Higher Order Methods

9

1( ) ( )

( 1) ( ) ( )

1( ) ( ) ( )

( 1) ( )

( 1) ( ) ( )

( 1)

Modified Newton

Marquardt's Compromise

( ). . 0.5 2

k k

k k k

k k kk

k k

k k k

k

search directionstep size

search directionfraction or multiple

e g orstep size

d H cx x d

d H I c

x ( )k note no alphax d

Optimization Project

• Formulating• Computer Modeling• Solving/executing• Evaluating (your “solution”)• Analyzing the sensitivity of your solution

10

1. Tips: Formulating

11

• Functional requirements (HoQ) • Eng. Characteristics (i.e. quantifiable measures)• Identify design variables, names, symbols, units, limits• Develop Objective function• Retrieve or develop analytical formulas/models • Develop constraints (laws of nature, man & economics)• “Principal of Optimum Sloppiness”, significant figures?

2. Tips: Computer Modeling• Pre-test custom-written code• Hand-check (w/calculator): f(x), g(x), h(x) at some x(1) • Eliminate ratios, if possible (to avoid divide by zero)• Eliminate non-differentiable functions (such as abs(), max())• Check analytical derivatives w/FD derivatives• Exploit available library routines • Scale variables and or constraints if difficulties arise

12

3. Tips: Solving/Executing

13

• Test “optimizer” w/ known problems/solutions• Solve from multiple starting points• If algorithm fails, monitor each iteration• Record statistics: constraint values, solutions

4. Tips: Evaluating the Solution

14

• Hand-check (calculator): f(x*), g(x*), h(x*)• Evaluate constraint activity

– Violated– Non-binding/inactive– Binding/active

• Do results make physical sense?

5. Tips: Analyzing the sensitivity

15

• Relax R.H.S. • Record Δf(x) for Δx• Change cost coefficients in f(x)• Vary parameters in g(x), h(x)

• Remember: A, b and c’s• Look for opportunity!

Test 5 on Wed

• T/F• Region elimination methods• Steepest descent algorithim • Conjugate Gradient algorithm• Be prepared to do hand calculations.

16

Alternate Equal Interval

Golden Section

Equal Interval aka “Exhaustive”

Fractional Reduction

17

ln( )1ln(0.618)

FRN

Add these formulas to your notes for next test!

1/ (0.618)nnew oldFR I I

2ln( )1ln(2 / 3)

FRN

2/1new oldFR I I

n

2 1NFR

/2/ (2 / 3)nnew oldFR I I

Summary• Steepest descent algorithm may stall• Conjugate Gradient

Convergence in n iterations (n=# of design var’s)Still has lots of Fcn evals (in line search)May need to restart after n+1 iterations

• Use “TIPS” to facilitate your project

18