Solutions for Nonlinear Equations
Lecture 8
Alessandra Nardi
Thanks to Prof. Newton, Prof. Sangiovanni, Prof. White, Jaime Peraire, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy
Last Lecture Review
• How to represent circuits– MNA– Voltage sources– How to assemble, X-stamp
• How to solve linear systems– Gaussian elimination– LU decomposition
Outline
• Nonlinear problems• Iterative Methods• Newton’s Method
– Derivation of Newton– Quadratic Convergence– Examples– Convergence Testing
• Multidimensonal Newton Method– Basic Algorithm – Quadratic convergence– Application to circuits
Need to Solve
( 1) 0d
t
VV
d sI I e
Nonlinear Problems - Example
0
1
IrI1 Id
0)1(1
0
11
1
1
IeIeR
III
tV
e
s
dr
11)( Ieg
Nonlinear Equations
• Given g(V)=I
• It can be expressed as: f(V)=g(V)-I
Solve g(V)=I equivalent to solve f(V)=0
Hard to find analytical solution for f(x)=0
Solve iteratively
Nonlinear Equations – Iterative Methods
• Start from an initial value x0
• Generate a sequence of iterate xn-1, xn, xn+1 which hopefully converges to the solution x*
• Iterates are generated according to an iteration function F: xn+1=F(xn)
Ask• When does it converge to correct solution ?• What is the convergence rate ?
Newton-Raphson (NR) MethodConsists of linearizing the system.
Want to solve f(x)=0 Replace f(x) with its linearized version and solve.
Note: at each step need to evaluate f and f’
functionIterationxfxdx
dfxx
xxxdx
dfxfxf
iesTaylor Serxxxdx
dfxfxf
kkkk
kkkkk
)()(
))(()()(
))(()()(
11
11
***
Newton-Raphson Method – Algorithm
Define iteration
Do k = 0 to ….
until convergence
• How about convergence?
• An iteration {x(k)} is said to converge with order q if there exists a vector norm such that for each k N:
qkk xxxx ˆˆ1
)()(1
1 kkkk xfxdx
dfxx
2* * * 2
2
*
0 ( ) ( ) ( )( ) ( )( )
some [ , ]
k k k k
k
df d ff x f x x x x x x x
dx dx
x x x
Mean Value theoremtruncates Taylor series
But10 ( ) ( )( )k k k kdf
f x x x xdx
by Newtondefinition
Newton-Raphson Method – Convergence
Subtracting
21 * 1 * 2
2 ( ) [ ( )] ( )( )k k kdf d f
x x x x x xdx d x
Convergence is quadratic
21 * * 2
2( )( ) ( )( )k k kdf d fx x x x x x
dx d x
Dividing through
21
2
21 * *
Let [ ( )] ( )
then
k k
k k k
df d fx x K
dx d x
x x K x x
Newton-Raphson Method – Convergence
Newton-Raphson Method – Convergence
Local Convergence Theorem
If
Then Newton’s method converges given a sufficiently close initial guess (and
convergence is quadratic)
boundedisK
boundeddx
fdb
roay from zebounded awdx
dfa
)
)
2
2
21
2 21 * * *
* 2
2
1 *
*
( ) 2
( ) 1 0,
2 ( ) 1
2 ( ) 2 ( )
1( ) (
( 1
)2
)
k k
k k k k
k k k k k
k kk
dfx x
dx
x x x x
x x x x x x x
f x x fin
x
d x x
or x x x xx
Convergence is quadratic
Newton-Raphson Method – ConvergenceExample 1
1 2
1 *
* *1
2 *
( ) 2
2 ( 0) ( 0)
10 0
( ) 0,
for 02
1( ) ( )
2
0
k k
k k k
k k k
k k
dfx x
dx
x x x
x x x x
or x x x
f x
x
x x
Convergence is linear
Note : not bounded
away from zero
1df
dx
Newton-Raphson Method – ConvergenceExample 2
0 Initial Guess,= 0x k
Repeat { 1
k
k k kf x
x x f xx
} Until ?
1 1? ?k k kf x thresholdx x threshold
1k k
Newton-Raphson Method – Convergence
Need a "delta-x" check to avoid false convergence
X
f(x)
1kx kx *x
1 a
kff x
1 1 a r
k k kx xx x x
Newton-Raphson Method – ConvergenceConvergence Checks
Also need an " " check to avoid false convergencef x
X
f(x)
1kx kx
*x
1 a
kff x
1 1 a r
k k kx xx x x
Newton-Raphson Method – ConvergenceConvergence Checks
X
f(x)
Convergence Depends on a Good Initial Guess
0x
1x2x 0x
1x
Newton-Raphson Method – ConvergenceLocal Convergence
Nodal Analysis
1 2At Node 1: 0i i
1bv
+
-
1i2i 2
bv
3i+ -
3bv
+
-
NonlinearResistors
i g v
1v2v
1 1 2 0g v g v v
3 2At Node 2: 0i i
3 1 2 0g v g v v
Two coupled nonlinear equations in two unknowns
Nonlinear Problems – Multidimensional Example
* *Problem: Find such tha 0t x F x * N N N and : x F
Multidimensional Newton Method
functionIterationxFxJxx
atrixJacobian M
x
xF
x
xF
x
xF
x
xF
xJ
iesTaylor SerxxxJxFxF
kkkk
N
NN
N
)()(
)()(
)()(
)(
))(()()(
11
1
1
1
1
***
Multidimensional Newton MethodComputational Aspects
)()()( :solve Instead
sparse).not is(it )( computenot Do
)()(:
1
1
11
kkkk
k
kkkk
xFxxxJ
xJ
xFxJxxIteration
Each iteration requires:
1. Evaluation of F(xk)
2. Computation of J(xk)
3. Solution of a linear system of algebraic equations whose coefficient matrix is J(xk) and whose RHS is -F(xk)
0 Initial Guess,= 0x k
Repeat {
1 1Solve for k k k k kFJ x x x F x x
} Until 11 , small en ug o hkk kx x f x
1k k
Compute ,k kFF x J x
Multidimensional Newton MethodAlgorithm
If
1) Inverse is boundedkFa J x
) Derivative is Lipschitz ContF Fb J x J y x y
Then Newton’s method converges given a sufficiently close initial guess (and
convergence is quadratic)
Multidimensional Newton Method Convergence
Local Convergence Theorem
Application of NR to Circuit Equations
0
1
IrI1 Id
01
,
0
1
11
1
11
11
1
1
11
1
1
1
11
k
dk
k
k
k
v
kk
r
VV
S
I
VV
Sk
k
VV
S
kkkkk
VVeV
I
RIeI
R
VVf
eIVgIVgR
VVf
VVVfVfVf
Note: G0 and Id depend on the iteration count k G0=G0(k) and Id=Id(k)
Application of NR to Circuit EquationsCompanion Network – MNA templates
Modeling a MOSFET
• Need continuous models with continuous derivatives
• Example: – simple MOS model valid from weak inversion to
saturation
VVV
DS
TGS
eVX
X
XKI
2
2
1ln2
1
Implications• Device model equations must be continuous with continuous
derivatives and derivative calculation must be accurate derivative of function (not all models do this - Poor diode models and breakdown models don’t - be sure models are decent - beware of user-supplied models)
• Watch out for floating nodes (If a node becomes disconnected, then J(x) is singular)
• Give good initial guess for x(0)
• Most model computations produce errors in function values and derivatives. Want to have convergence criteria || x(k+1) - x(k) || < such that > than model errors.
Summary
• Nonlinear problems
• Iterative Methods
• Newton’s Method– Derivation of Newton
– Quadratic Convergence
– Examples
– Convergence Testing
• Multidimensional Newton Method– Basic Algorithm
– Quadratic convergence
– Application to circuits
Methods for Ordinary Differential Equations
Lecture 10
Alessandra Nardi
Thanks to Prof. Jacob White, Deepak Ramaswamy Jaime Peraire, Michal Rewienski, and Karen Veroy
Outline
• Transient Analysis of dynamical circuits– i.e., circuits containing C and/or L
• Examples
• Solution of Ordinary Differential Equations (Initial Value Problems – IVP)– Forward Euler (FE), Backward Euler (BE) and
Trapezoidal Rule (TR)– Multistep methods– Convergence
Ground Plane
Signal Wire
LogicGate
LogicGate
• Metal Wires carry signals from gate to gate.• How long is the signal delayed?
Wire and ground plane form a capacitor
Wire has resistance
Application ProblemsSignal Transmission in an Integrated Circuit
capacitor
resistor
• Model wire resistance with resistors.• Model wire-plane capacitance with capacitors.
Constructing the Model• Cut the wire into sections.
Application ProblemsSignal Transmission in an IC – Circuit Model
Nodal Equations Yields 2x2 System
C1
R2
R1 R3 C2
Constitutive Equations
cc
dvi C
dt
1R Ri v
R
Conservation Laws
1 1 20C R Ri i i
2 3 20C R Ri i i
1
1 2 21 1
2 22
2 3 2
1 1 1
0
0 1 1 1
dvR R RC vdt
C vdv
R R Rdt
1Ri1Ci
2Ri
2Ci3Ri
1v 2v
Application ProblemsSignal Transmission in an IC – 2x2 example
1(0) 1v
2 (0) 0v
Notice two time scale behavior• v1 and v2 come together quickly (fast eigenmode).• v1 and v2 decay to zero slowly (slow eigenmode).
Application ProblemsSignal Transmission in an IC – 2x2 example
Circuit Equation Formulation
• For dynamical circuits equations can be written compactly:
• For sake of simplicity, we shall discuss first order ODEs in the form:
riablescircuit va of vector theis where
)0(
0),,)(
(
0
x
xx
txdt
tdxF
),()(
txfdt
tdx
Ordinary Differential EquationsInitial Value Problems (IVP)
Typically analytic solutions are not available
solve it numerically
.condition initial given the intervalan in
)(
),()(
:(IVP) Problem Value Initial Solve
00
00
x,T][t
xtx
txfdt
tdx
• Not necessarily a solution exists and is unique for:
• It turns out that, under rather mild conditions on the continuity and differentiability of F, it can be proven that there exists a unique solution.
• Also, for sake of simplicity only consider
linear case:
0),,( tydt
dyF
We shall assume that has a unique solution 0),,( tydt
dyF
00 )(
)()(
xtx
tAxdt
tdx
Ordinary Differential Equations Assumptions and Simplifications
First - Discretize Time
Second - Represent x(t) using values at ti
ˆ ( )llx x t
Approx. sol’n
Exact sol’n
Third - Approximate using the discrete ( )l
dx t
dtˆ 'slx
1 1
1
ˆ ˆ ˆ ˆExample: ( )
l l l l
ll l
d x x x xx t or
dt t t
Lt T1t 2t 1Lt 0
t t t
1t 2t 3t Lt0
3x̂ 4x̂1x̂
2x̂
Finite Difference MethodsBasic Concepts
lt 1lt t
x
1( ) ( )slope l lx t x t
t
slope ( )l
dx t
dt
1( ) ( ) ( )l l lx t x t t A x t
1
1
( ) ( )( ) ( )
or
( ) ( ) ( )
l ll l
l l l
x t x tdx t A x t
dt t
x t x t t A x t
Finite Difference Methods Forward Euler Approximation
1t 2t t
x
(0)tAx
11 ˆ( ) (0) 0x t x x tAx
3t
2 1 12 ˆ ˆ ˆ( )x t x x tAx
1ˆtAx
1 1ˆ ˆ ˆ( ) L L LLx t x x tAx
Finite Difference Methods Forward Euler Algorithm
lt 1lt t
x 1slope ( )l
dx t
dt
1( ) ( )slope l lx t x t
t
1 1( ) ( ) ( )l l lx t x t t A x t
11 1
1 1
( ) ( )( ) ( )
or
( ) ( ) ( )
l ll l
l l l
x t x tdx t A x t
dt t
x t x t t A x t
Finite Difference Methods Backward Euler Approximation
1t 2t t
x
1ˆtAx
2ˆtAx
1 11 ˆ ˆ( ) (0)x t x x tAx
Solve with Gaussian Elimination
1ˆ[ ] (0)I tA x x
1 1ˆ ˆ( ) [ ]L LLx t x I tA x
2 1 12 ˆ ˆ( ) [ ]x t x I tA x
Finite Difference Methods Backward Euler Algorithm
1
1
1
1 1
1( ( ) ( ))
21
( ( ) ( ))2
( ) ( )
1( ) ( ) ( ( ) ( ))
2
l l
l l
l l
l l l l
d dx t x t
dt dt
Ax t Ax t
x t x t
t
x t x t tA x t x t
t
x
1( ) ( )slope l lx t x t
t
slope ( )l
dx t
dt
1slope ( )l
dx t
dt
1 1
1 1( ( ) ( )) ( ( ) ( ))
2 2l l l lx t tAx t x t tAx t
Finite Difference Methods Trapezoidal Rule Approximation
1t 2t t
x
1ˆ (0)2 2
t tI A x I A x
Solve with Gaussian Elimination
1 11 ˆ ˆ( ) (0) (0)
2
tx t x x Ax Ax
12 1
2
11
ˆ ˆ( )2 2
ˆ ˆ( )2 2
L LL
t tx t x I A I A x
t tx t x I A I A x
Finite Difference Methods Trapezoidal Rule Algorithm
1
1( ) (( ) )) )( (l
l
t
l l t
dx t x t x t A dx t xA
dt
lt 1lt
1
( )l
l
t
tAx d
1( )ltAx t BE
( )ltAx t FE
( ) ( )2 l l
tAx t Ax t
Trap
Finite Difference Methods Numerical Integration View
Trap Rule, Forward-Euler, Backward-Euler Are all one-step methods
Forward-Euler is simplest No equation solution explicit method. Boxcar approximation to integralBackward-Euler is more expensive Equation solution each step implicit method Trapezoidal Rule might be more accurate
Equation solution each step implicit method
Trapezoidal approximation to integral
1 2 3ˆ ˆ ˆ ˆ is computed using only , not , , etc.l l l lx x x x
Finite Difference Methods Summary of Basic Concepts
Definition: A finite-difference method for solving initial value problems on [0,T] is said to be
convergent if given any A and any initial condition
0,
ˆmax 0 as t 0lT
lt
x x l t
exactx
tˆ computed with
2lx
ˆ computed with tlx
Convergence Analysis Convergence Definition
Definition: A multi-step method for solving initial value problems on [0,T] is said to be order p
convergent if given any A and any initial condition
0,
ˆmaxpl
Tl
t
x x l t C t
0for all less than a given t t
Forward- and Backward-Euler are order 1 convergentTrapezoidal Rule is order 2 convergent
Convergence Analysis Order-p Convergence
Multistep Methods – Convergence Analysis
Two types of error
made.been haserror previous no assuming ,solution theof value
exact theand ˆ valuecomputed ebetween th difference theis
at methodn integratioan of (LTE)Error Truncation Local The
1
1
1
)x(t
x
t
l
l
l
exactly.known iscondition
initial only the that assuming ,solution theof value
exact theand ˆ valuecomputed ebetween th difference theis
at methodn integratioan of (GTE)Error Truncation Global The
1
1
1
)x(t
x
t
l
l
l
• For convergence we need to look at max error over the whole time interval [0,T]– We look at GTE
• Not enough to look at LTE, in fact:– As I take smaller and smaller timesteps t, I would
like my solution to approach exact solution better and better over the whole time interval, even though I have to add up LTE from more timesteps.
Multistep Methods – Convergence Analysis
Two conditions for Convergence
1) Local Condition: One step errors are small (consistency)
2) Global Condition: The single step errors do not grow too quickly (stability)
Typically verified using Taylor Series
All one-step methods are stable in this sense.
Multistep Methods – Convergence Analysis
Two conditions for Convergence
Definition: A one-step method for solving initial value problems on an interval [0,T] is said to be consistent if for any A and any initial condition
1ˆ0 as t 0
x x t
t
One-step Methods – Convergence Analysis
Consistency definition
Forward-Euler definition
22
2
0()
2 0
dxtdxxtxt
dtdt
Expanding in about zero yields t
1ˆ00 xxtAx
Noting that (0)(0) and subtractingd
xAxdt
22
12 ˆ
2
tdxxxt
dt
Proves the theorem if
derivatives of x are bounded
0,t
One-step Methods – Convergence Analysis
Consistency for Forward Euler
Forward-Euler definition
1l
xltxlttAxlte
Expanding in about yields tlt 1
ˆˆ ˆlll
xxtAx
where is the "one-step" error bounded byl
e
2
2
[0,]2 , where 0.5maxl
T
dxeCtC
dt
One-step Methods – Convergence Analysis
Convergence Analysis for Forward Euler
ˆ Define the "Global" errorll
Exxlt
1ˆˆ 1
lllxxltItAxxlte
Subtracting the previous slide equations
Taking norms and using the bound on l
e
1 lllEItAEe
2 1
21
ll
l
EItAECt
tAECt
One-step Methods – Convergence Analysis
Convergence Analysis for Forward Euler
Example
+-
I1
R
C
V2
VS
V1
SVV
IVR
VR
VRdt
dVCV
R
2
121
21
1
011
011
Sk
kkk
kk
kk
Sk
kkk
kkk
k
VV
IVR
VR
t
VCV
Rt
VCV
R
VV
IVR
VR
VRt
VVCV
R
21
11
21
11
12
1
111
1
21
11
21
11
21
1111
1
011
11
011
011
BE
Sk
kkk
kk
kk
Sk
kkk
kkk
k
VV
IVR
VR
VRt
VCV
Rt
VC
VV
IVR
VR
VRt
VVCV
R
2
121
21
11
1
2
121
211
11
011
11
011
011
FE