stochastic hybrid systems: modeling, analysis, and applications to networks and biology
DESCRIPTION
research supported by NSF. Stochastic Hybrid Systems: Modeling, analysis, and applications to networks and biology. João P. Hespanha Center for Control Engineering and Computation University of California at Santa Barbara. Talk outline. A model for stochastic hybrid systems (SHSs) - PowerPoint PPT PresentationTRANSCRIPT
Stochastic Hybrid Systems:Modeling, analysis, and applications to
networks and biology
João P. Hespanha
Center for Control Engineeringand Computation
University of Californiaat Santa Barbara
research supported by NSF
Talk outline
1. A model for stochastic hybrid systems (SHSs)2. Examples:
• network traffic under TCP• networked control systems
3. Analysis tools for SHSs• Lyapunov• moment dynamics
4. More examples …
Collaborators:Stephan Bohacek (U Del), Katia Obraczka (UCSC), Junsoo Lee (Sookmyung Univ.), Mustafa Khammash (UCSB)
Students:Abhyudai Singh (UCSB), Yonggang Xu (UCSB)
Disclaimer: This is an overview, details in papers referenced…
Deterministic Hybrid Systems
guardconditions
reset-maps
continuousdynamics
q(t) 2 Q={1,2,…} ´ discrete state x(t) 2 Rn ´ continuous state
right-continuousby convention
we assume here a deterministic system so the invariant sets would be the exact complements of the guards
Stochastic Hybrid Systems
reset-maps
continuousdynamics
transition intensities(probability of transition in interval (t, t+dt])
Continuous dynamics:
Reset-maps (one per transition intensity):
Transition intensities:
q(t) 2 Q={1,2,…} ´ discrete state x(t) 2 Rn ´ continuous state
Stochastic Hybrid Systems
reset-maps
continuousdynamics
Special case: When all are constant, transitions are controlled by a continuous-time Markov process
q = 1 q = 2
q = 3
specifies q(independently of x)
transition intensities(probability of transition in interval (t, t+dt])
Formal model—Summary
State space: q(t) 2 Q={1,2,…} ´ discrete statex(t) 2 Rn ´ continuous state
Continuous dynamics:
Reset-maps (one per transition intensity):
Transition intensities:
# of transitions
Results:1. [existence] Under appropriate regularity (Lipschitz) assumptions, there exists
a measure “consistent” with the desired SHS behavior2. [simulation] The procedure used to construct the measure is constructive and
allows for efficient generation of Monte Carlo sample paths3. [Markov] The pair ( q(t), x(t) ) 2 Q£ Rn is a (Piecewise-deterministic)
Markov Process (in the sense of M. Davis, 1993)[HSCC’04]
Stochastic Hybrid Systems with diffusion
reset-maps
stochasticdiff. equation
transition intensities
Continuous dynamics:
Reset-maps (one per transition intensity):
Transition intensities:
w ´ Brownian motion process
Example I: Transmission Control Protocol
server clientnetwork
packets droppedwith probability pdrop
transmitsdata packets
receivesdata packets
congestion control ´ selection of the rate r at which the server transmits packetsfeedback mechanism ´ packets are dropped by the network to indicate congestion
r
Example I: TCP congestion control
server clientnetwork
transmitsdata packets
receivesdata packets
TCP (Reno) congestion control: packet sending rate given bycongestion window (internal state of controller)
round-trip-time (from server to client and back)• initially w is set to 1• until first packet is dropped, w increases exponentially fast (slow-start)• after first packet is dropped, w increases linearly (congestion-avoidance)• each time a drop occurs, w is divided by 2 (multiplicative decrease)
packets droppedwith probability pdrop
congestion control ´ selection of the rate r at which the server transmits packetsfeedback mechanism ´ packets are dropped by the network to indicate congestion
r
Example I: TCP congestion control
per-packetdrop prob.
pckts sentper sec
£ pckts droppedper sec=
# of packetsalready sent
TCP (Reno) congestion control: packet sending rate given bycongestion window (internal state of controller)
round-trip-time (from server to client and back)• initially w is set to 1• until first packet is dropped, w increases exponentially fast (slow-start)• after first packet is dropped, w increases linearly (congestion-avoidance)• each time a drop occurs, w is divided by 2 (multiplicative decrease)
many SHS models for TCP…long-lived TCP flows
(no delays)
on-off TCPflows with delay
long-lived TCP flows with delay
on-off TCP flows(no delays)
[SIGMETRICS’03]
network
Example II: Estimation through network
encoder decoder
white noisedisturbancex
x(t1) x(t2)
process
encoder logic ´ determines when to send measurements to the networkdecoder logic ´ determines how to incorporate received measurements
state-estimator
for simplicity:• full-state available• no measurement noise• no quantization• no transmission delays
network
Stochastic communication logic
encoder decoder
white noisedisturbancex
x(t1) x(t2)
process
encoder logic ´ determines when to send measurements to the network
state-estimator
decoder logic ´ determines how to incorporate received measurements
for simplicity:• full-state available• no measurement noise• no quantization• no transmission delays
network
Stochastic communication logic
encoder decoder
white noisedisturbancex
x(t1) x(t2)
process state-estimator
Error dynamics:
reset error to zero
prob. of sending data in [t,t+dt) depends on current error e
for simplicity:• full-state available• no measurement noise• no quantization• no transmission delays
network
Stochastic communication logic
encoder decoder
white noisedisturbancex
x(t1) x(t2)
with:• measurement noise• quantization• transmission delay
process state-estimator
reset error to nonzero random variables
prob. of sending data in [t,t+dt) depends on current encoder error enec
[ACTRA’04]
error at encoder sidebased on data sent
error at decoder sidebased on data received
Example III: And now for something completely different…
Decaying-dimerizing chemical reactions (DDR):
SHS model population of species S1
population of species S2
reaction rates
S2 0S1 0 2 S1 S2
c1 c2c3
c4
Inspired by Gillespie’s Stochastic Simulation Algorithm for molecular reactions [Gillespie, 76]
Example III: And now for something completely different…
Decaying-dimerizing chemical reactions (DDR):
SHS model population of species S1
population of species S2
reaction rates
S2 0S1 0 2 S1 S2
c1 c2c3
c4
Inspired by Gillespie’s Stochastic Simulation Algorithm for molecular reactions [Gillespie, 76]
Disclaimer: Several other important applications missing. E.g., air traffic control [Lygeros,Prandini,Tomlin] queuing systems [Cassandras] economics [Davis]
Generalizations of the SHS model
1. Stochastic resets can be obtained by considering multiple intensities/reset-maps
One can further generalize this to resets governed by to a continuous distribution [see paper]
Generalizations of the SHS model
2. Deterministic guards can also be emulated by taking limits of SHSs
The solution to the hybrid system with a deterministic guard is obtained as # 0+
# 0+
-1 1
This provides a mechanism to regularize systems with chattering and/or Zeno phenomena…
barrierfunction
g(x)
Example: Bouncing-ball
t
yc 2 (0,1) ´ energy absorbed at impact
Zeno-time
The solution of this deterministic hybrid system is only defined up to the Zeno-time
g
y
g
0 5 10 15-0.2
0
0.2
0.4
0.6
0.8
1
1.2
0 5 10 15-0.2
0
0.2
0.4
0.6
0.8
1
1.2
0 5 10 15-0.2
0
0.2
0.4
0.6
0.8
1
1.2
= 10–2 = 10–3 = 10–4
mean (blue) 95% confidence intervals (red and green)
Example: Bouncing-ball
g
y
g
c 2 (0,1) ´ energy absorbed at impact
Analysis—Lie Derivative
Given scalar-valued function Ã: Rn £ [0,1) ! R
derivativealong solution
to ODE Lf ÃLie derivative of Ã
One can view Lf as an operator
space of scalar functions onRn £ [0,1)
space of scalar functions onRn £ [0,1)
à (x, t) Lf Ã(x, t)
Lf completely defines the system dynamics
Generator of a SHS
continuous dynamicsreset-mapstransition intensities
Given scalar-valued function Ã: Q £ Rn £ [0,1) ! Rgenerator for the SHS
where
Lie derivative
reset term
L completely defines the SHS dynamics
Dynkin’s formula(in differential form)
diffusion term
Disclaimer: see following paper for technical assumptions [HSCC’04]
instantaneous variation
intensity
Given scalar-valued function Ã: Q £ Rn £ [0,1) ! R
where
Generator of the SHS with diffusion
Dynkin’s formula(in differential form)
Attention: These systems may have problems of existence of solution due to jumps!
(stochastic Zeno)
E.g.
no global solution no local solutionfor x(0) > 1
L completely defines the SHS dynamics
“jumping makes jumping more likely”
“probability of multiple jumps in short interval not sufficiently small”
Disclaimer: see following paper for technical assumptions [HSCC’04]
Stochastic communication logicserror dynamics
in remote estimation
Long-lived TCP flows
slow-start congestionavoidance
long-lived TCP flows(with slow start)
Lyapunov-based stability analysis
Expected value of error:
Dynkin’s formula
)2nd moment of the error:
)
error dynamicsin NCS
Lyapunov-based stability analysis
Expected value of error:
Dynkin’s formula
For constant rate: (e) =
)2nd moment of the error:
)
assuming (A – /2 I ) Hurwitz
error dynamicsin NCS
Lyapunov-based stability analysis
Dynkin’s formula
One can show…
For constant rate: (e) =
1. E[ e ] ! 0 as long as > <[(A)]2. E[ || e ||m ] bounded as long as > 2 m <[(A)]
For polynomial rates: (e) = (e0 P e)k P > 0, k ¸ 0
1. E[ e ] ! 0 (always)2. E[ || e ||m ] bounded 8 m
both always true 8 ¸ 0 if A Hurwitz(no jumps needed for boundedness)
getting more moments bounded requires
higher jump intensities
Moreover, one can achieve the same E[ || e ||2 ] with a smaller
number of transmissions…
[ACTRA’04, CDC’04]
error dynamicsin NCS
Analysis—Moments for SHS state
z (scalar) random variable with mean and variance 2
Tchebychev inequalityMarkov inequality(8 > 0) Bienaymé inequality
(8 > 0, a2R , n2N)
continuous dynamicsreset-mapstransition intensities
often a few low-order moments suffice to study a SHS…
Polynomial SHSs
A SHS is called a polynomial SHS (pSHS) if its generator maps finite-order polynomial on x into finite-order polynomials on xTypically, when
are all polynomials 8 q, t
continuous dynamicsreset-mapstransition intensities
Given scalar-valued function Ã: Q £ Rn £ [0,1) ! Rgenerator for the SHS
where
Dynkin’s formula(in differential form)
Moment dynamics for pSHS
E.g,
continuous state discrete statex(t) 2 Rn q(t) 2 Q={1,2,…}
Monomial test function: Given
Uncentered moment:
for short x(m)
Moment dynamics for pSHS
continuous state discrete statex(t) 2 Rn q(t) 2 Q={1,2,…}
for short x(m)
Uncentered moment:
)
For polynomial SHS…
) monomial on x polynomial on x linear comb. ofmonomial test functions
linear moment dynamics
Monomial test function: Given
Moment dynamics for TCP
long-livedTCP flows
Moment dynamics for TCP
long-livedTCP flows
on-off TCP flows(exp. distr. files)
Truncated moment dynamicsExperimental evidence indicates that the (steady-state)
sending rate is well approximated by a Log-Normal distribution
w0 10 20 30 40 50
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Data from: Bohacek, A stochastic model for TCP and fair video transmission, INFOCOM’03
PDF of the congestion window size w
z Log-Normal
r Log-Normal (on each mode) )
)
Moment dynamics for TCP
finite-dimensionalnonlinear ODEs
long-livedTCP flows
on-off TCP flows(exp. distr. files)
Long-lived TCP flows
0 1 2 3 4 5 6 7 8 9 10
0.02
0.1
0 1 2 3 4 5 6 7 8 9 100
100
200
sending rate mean
0 1 2 3 4 5 6 7 8 9 100
50
100
sending rate std. deviation
Monte Carlo (with 99% conf. int.) reduced model
moment dynamics in response to abrupt change in drop probability
long-lived TCP flows with delay
(one RTT average)
Truncated moment dynamics (revisited)
For polynomial SHS…
linear moment dynamics
Stacking all moments into an (infinite) vector 1
infinite-dimensional linear ODEIn TCP analysis…
lower ordermoments of interest
moments of interestthat affect dynamics
approximated bynonlinear function of
Truncation by derivative matchinginfinite-dimensional linear ODE
truncated linear ODE(nonautonomous, not nec. stable)
nonlinear approximatemoment dynamics
Assumption: 1) and remain bounded along solutions to
and
2) is (incrementally) asymptotically stable
class KL function
Theorem: 8 > 0 9 N s.t. if
then
Disclaimer: Just a lose statement. The “real” theorem is stated in [HSCC'05]
Truncation by derivative matchinginfinite-dimensional linear ODE
truncated linear ODE(nonautonomous, not nec. stable)
nonlinear approximatemoment dynamics
Proof idea:1) N derivative matches ) & match on compact interval of length T2) stability of A1 ) matching can be extended to [0,1)
Assumption: 1) and remain bounded along solutions to
and
2) is (incrementally) asymptotically stable
class KL function
Theorem: 8 > 0 9 N s.t. if
then
Assumption: 1) and remain bounded along solutions to
and
2) is (incrementally) asymptotically stable
class KL function
Theorem: 8 > 0 9 N s.t. if
then
Truncation by derivative matchinginfinite-dimensional linear ODE
truncated linear ODE(nonautonomous, not nec. stable)
nonlinear approximatemoment dynamics
Proof idea:1) N derivative matches ) & match on compact interval of length T2) stability of A1 ) matching can be extended to [0,1)
Given , finding N is very difficult
In practice, small values of N (e.g., N = 2) already yield good results
Can use
to determine (¢): k = 1 ! boundary condition on
k = 2 ! linear PDE on
Moment dynamics for DDR
Decaying-dimerizing molecular reactions (DDR):
S2 0S1 0 2 S1 S2
c1 c2c3
c4
Truncated DDR model
by matching
for deterministic distributions
[IJRC, 2005]
Decaying-dimerizing molecular reactions (DDR):
S2 0S1 0 2 S1 S2
c1 c2c3
c4
Monte Carlo vs. truncated model
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
x 10-3
0
10
20
30
40populations means
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
x 10-3
0
1
2
3
4populations standard deviations
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
x 10-3
-1.1
-1.05
-1
-0.95
-0.9
populations correlation coefficient
E[x1]
E[x2]
Std[x1]
Std[x2]
x2,x2](lines essentially
undistinguishable at this scale)
Fast time-scale(transient)
Parameters from: Rathinam, Petzold, Cao, Gillespie, Stiffness in stochastic chemically reacting systems: The implicit tau-leaping method. J. of Chemical Physics, 2003
Monte Carlo vs. truncated model
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
1
2
3
4
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-1
-0.5
0
0.5
populations means
populations standard deviations
populations correlation coefficient
E[x1]
E[x2] Slow time-scaleevolution
Std[x1]
Std[x2]
x2,x2]
error only noticeable when populations become very small
(a couple of molecules,still adequate to study cellular
reactions)
Parameters from: Rathinam, Petzold, Cao, Gillespie, Stiffness in stochastic chemically reacting systems: The implicit tau-leaping method. J. of Chemical Physics, 2003
Conclusions
1. A simple SHS model (inspired by piecewise deterministic Markov Processes) can go a long way in modeling network traffic
2. The analysis of SHSs is generally difficult but there are tools available(generator, Lyapunov methods, moment dynamics, truncations)
3. This type of SHSs (and tools) finds use in several areas(traffic modeling, networked control systems, molecular biology, population dynamics in ecosystems)
Bibliography
• M. Davis. Markov Models & Optimization. Chapman & Hall/CRC, 1993• S. Bohacek, J. Hespanha, J. Lee, K. Obraczka. Analysis of a TCP hybrid model. In Proc. of the
39th Annual Allerton Conf. on Comm., Contr., and Computing, Oct. 2001.• S. Bohacek, J. Hespanha, J. Lee, K. Obraczka. A Hybrid Systems Modeling Framework for Fast
and Accurate Simulation of Data Communication Networks. In Proc. of the ACM Int. Conf. on Measurements and Modeling of Computer Systems (SIGMETRICS), June 2003.
• J. Hespanha. Stochastic Hybrid Systems: Applications to Communication Networks. In R. Alur, G. Pappas, Hybrid Systems: Computation and Control, LNCS 1993, pp. 387-401, 2004.
• João Hespanha. Polynomial Stochastic Hybrid Systems. In Manfred Morari, Lothar Thiele, Hybrid Systems: Computation and Control, LNCS 3414, pp. 322-338, Mar. 2005.
• J. Hespanha, A. Singh. Stochastic Models for Chemically Reacting Systems Using Polynomial Stochastic Hybrid Systems. To appear in the Int. J. on Robust Control Special Issue on Control at Small Scales, 2005.
• A. Singh, J. Hespanha. Moment Closure Techniques for Stochastic Models in Population Biology. Oct. 2005. Submitted to publication.
• Y. Xu, J. Hespanha. Communication Logic Design and Analysis for NCSs. ACTRA’04.• Y. Xu, J. Hespanha. Optimal Communication Logics for NCSs. In Proc. CDC, 2004.
All papers (and some ppt presentations) available athttp://www.ece.ucsb.edu/~hespanha
END
TCP window-based control
1st packet sent
e.g., w = 3
t
2nd packet sent3rd packet sent 1st packet received & ack. sent
2nd packet received & ack. sent3rd packet received & ack. sent1st ack. received )
4th packet can be sent
t
source f destination f
w effectively determines the sending rate r:
round-trip time
t0
t1
t2
t3
0
1
2
w (window size) ´ number of packets that can remain unacknowledged for by the destination
Built-in negative feedback:
r increases congestion
RTT increases (queuing delays)
r decreases
TCP Reno congestion avoidance1. While there are no drops, increase w by 1 on each RTT (additive increase)2. When a drop occurs, divide w by 2 (multiplicative decrease)
(congestion controller constantly probes the network for more bandwidth)
additiveincrease
multiplicative decrease
per-packetdrop prob.
pckts sentper sec
£ pckts droppedper sec
=
total # of packetsalready sent
Three feedback mechanisms: As rate r increases ) …a) congestion ) RTT increases (queuing delays) ) r decreasesb) congestion ) pdrop increases (active queuing) ) r decreasesc) multiplicative decrease more likely ) r decreases
TCP Reno slow start3. Until a drop occurs double w on each RTT4. When a drop occurs divide w by 2 and transition to congestion avoidance
(get to full bandwidth utilization fast)
TCP has several other modes that can be modeled by hybrid systems[Bohacek, Hespanha, Lee, Obraczka. A Hybrid Systems Modeling Framework for Fast and Accurate Simulation of Data Communication Networks. In SIGMETRICS’03]
# of packetsalready sent
Long-lived TCP flows
long-livedTCP flows
On-off TCP model
• Between transfers, server remains off for an exponentially distributed time with mean off
• Transfer-size is a random variable with cumulative distribution F(s)determines duration of on-periods
On-off TCP model
• Between transfers, server remains off for an exponentially distributed time with mean off
• Transfer-size is exponentially distributed with average k (packets)determines duration of on-periods
simple but not very realistic…
Analysis—Generator of the SHS
continuous dynamicsreset-mapstransition intensities
Given scalar-valued function Ã: Q £ Rn £ [0,1) ! Rgenerator for the SHS
where
Lf ÃLie derivative of Ã
reset instantaneousvariation
intensity
L completely defines the SHS dynamics
Disclaimer: see following paper for technical assumptionsHespanha. Stochastic Hybrid Systems: Applications to Communication Networks. HSCC’04
Dynkin’s formula(in differential form)
Analysis—PDF for SHS state
functional PDEvery difficult to solve
When ( ¢ ; t) is smooth, one can deduce from the generator that
continuous dynamicsreset-mapstransition intensities
Let ( ¢ ; t) be the probability density function (PDF) for the state (x,q) at time t :
inverse of
Long-lived TCP flows
Monte-Carlo & theoreticalsteady-state distribution
(vs. drop probability)
long-lived TCP flows with delay
(one RTT average)
On-off TCP flows
Transfer-sizes exponentially distributed with mean k = 30.6KB (Unix’93 files)Off-time exponentially distributed with mean off = 1 secRound-trip-time RTT = 50msec
SHS for on-off TCP flows
On-off TCP flows
Monte Carlo (solid) reduced model (dashed)
0 0.5 1 1.5 2 2.5 3
0.02
0.1
0 0.5 1 1.5 2 2.5 30
0.05
0.1
0.15
0 0.5 1 1.5 2 2.5 30
10
20
0 0.5 1 1.5 2 2.5 30
20
40
60
sending rate mean
sending rate std. dev.
probabilities
ss
ca
ss
ca
total
ssca
total