williams et.al. – approximate dynamic programming for ...mitchell/class/cs532m.2007w... · p)...
TRANSCRIPT
![Page 1: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/1.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Williams et.al. – Approximate DynamicProgramming for Communication-Constrained
Sensor Network ManagementAn Experimental Review and Visual Tool
Jonatan Schroeder
Project Outline – CPSC532MUniversity of British Columbia
Mar 17, 2008
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 2: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/2.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Outline
1 Problem Introduction
2 Dynamic Programming Formulation
3 Project
Based on: J. L. Williams, J. W. Fisher III, and A. S. Willsky. Approximatedynamic programming for communication-constrained sensor networkmanagement. IEEE Transactions on Signal Processing, 55(8):4300–4311,August 2007.
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 3: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/3.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Sensor Networks
Spatially distributed autonomous devicesSensor to monitor:
temperature, sound, pressurepollutantsexternal agentsbattlefield surveillance
Wireless communicationLimited energy
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 4: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/4.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Example
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 5: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/5.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Example
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 6: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/6.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Example
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 7: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/7.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Example
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 8: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/8.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Example
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 9: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/9.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Example
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 10: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/10.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Example
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 11: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/11.jpg)
Problem IntroductionDynamic Programming Formulation
Project
The Problem
Identify the state (position, velocity) of the objectProbability Distribution Function (pdf)Estimate the object’s next state
Subset of sensors and a leader sensor
Objectives:Maximize the information estimation performanceMinimize the communication cost
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 12: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/12.jpg)
Problem IntroductionDynamic Programming Formulation
Project
The Problem
Identify the state (position, velocity) of the objectProbability Distribution Function (pdf)Estimate the object’s next state
Subset of sensors and a leader sensorObjectives:
Maximize the information estimation performanceMinimize the communication cost
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 13: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/13.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Dynamic Programming Approach
Objective function: estimation performance orcommunication cost
Lagrangian relaxation
State: object actual positionPolicy: leader and sensor subset selectionRolling discrete horizon
In each step, N steps are calculated, and one is taken
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 14: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/14.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Object Dynamics
Object state: xk =
pxvxpyvy
Object dynamics: xk+1 = Fxk + wk
F =
1 T 0 00 1 0 00 0 1 T0 0 0 1
wk is a white Gaussian noise process
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 15: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/15.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Measurement Model
Measurement model: zsk = h(xk , s) + vs
k
h(xk , s) =a
‖Lxk − ys‖22 + b
vsk is a white Gaussian noise process
z0:k is the history of all measurements {z0, z1, . . . , zk}Estimation: in progress
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 16: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/16.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Communication
Cost (per bit) of direct communication: C̃ij ∝∥∥y i − y j
∥∥22
Cost considering path {i0, i1, . . . , inij}:
Cij =
nij∑k=1
C̃ik−1ik
Transmission of probabilistic model: Bp bitsTransmission of measurements: Bm bits
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 17: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/17.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Constrained Dynamic Programming
Conditional belief state: Xk , p (xk |z0:k−1) ∈ P(X )
Control: uk = (lk ,Sk ), lk ∈ S and Sk ⊂ SControl policy – time k : µk (Xk , lk−1)
Control policy set: πk = {µk , . . . , µk+N−1}
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 18: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/18.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Optimization Problem
minπ
E
[k+N−1∑
i=k
g (Xi , li−1, µi (Xi , li−1))
]
s.t.E
[k+N−1∑
i=k
G (Xi , li−1, µi (Xi , li−1))
]≤ M
Constraint addressed through Lagrangian relaxation:
Lk (Xk , lk−1, πk , λ) = E
[k+N−1∑
i=k
g (Xi , li−1, µi (Xi , li−1))
+ λ
(k+N−1∑
i=k
G (Xi , li−1, µi (Xi , li−1))−M
)]
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 19: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/19.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Optimization Problem
minπ
E
[k+N−1∑
i=k
g (Xi , li−1, µi (Xi , li−1))
]
s.t.E
[k+N−1∑
i=k
G (Xi , li−1, µi (Xi , li−1))
]≤ M
Constraint addressed through Lagrangian relaxation:
Lk (Xk , lk−1, πk , λ) = E
[k+N−1∑
i=k
g (Xi , li−1, µi (Xi , li−1))
+ λ
(k+N−1∑
i=k
G (Xi , li−1, µi (Xi , li−1))−M
)]Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 20: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/20.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Dynamic Programming Function
Lk (Xk , lk−1, πk , λ) = E
[k+N−1∑
i=k
g (Xi , li−1, µi (Xi , li−1))
+ λ
(k+N−1∑
i=k
G (Xi , li−1, µi (Xi , li−1))−M
)]
JDk (Xk , lk−1, λ) = min
πkLk (Xk , lk−1, πk , λ)
JLk (Xk , lk−1) = max
λ≥0JD
k (Xk , lk−1, λ)
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 21: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/21.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Dynamic Programming Function
Lk (Xk , lk−1, πk , λ) = E
[k+N−1∑
i=k
g (Xi , li−1, µi (Xi , li−1))
+ λ
(k+N−1∑
i=k
G (Xi , li−1, µi (Xi , li−1))−M
)]
JDk (Xk , lk−1, λ) = min
πkLk (Xk , lk−1, πk , λ)
JLk (Xk , lk−1) = max
λ≥0JD
k (Xk , lk−1, λ)
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 22: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/22.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Complexity Evaluation
Ns sensorsNp samples for each policyN steps (horizon)Ns2Ns possible controlsNs2NsNp new total samplesO(NN
s 2NsNNNp )
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 23: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/23.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Linearized Gaussian Approximation
Considering the smoothness of zSkk
Function for entropy evaluation is simplifiedO(NN
s 2NsN)
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 24: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/24.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Greedy Sensor Subnet Selection
Each stage is divided in substagesA new problem is defined, algebraically equivalent to theoriginal problemSensors that increase the cost in a substage are discardedSensors are limited to a small neighbourhoodWorst-case complexity: O(NN3
s )
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 25: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/25.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Dynamic Programming Problem Implementation
Implementation of the problem described in this paperBoth communication contraint (CC) and informationconstraint (IC) approachesRepeat simulation resultsObtain extended resulsTool for visualizing:
Agents positionObject stateLeader and subset selection
Jonatan Schroeder Approximate DP for Sensor Network Management
![Page 26: Williams et.al. – Approximate Dynamic Programming for ...mitchell/Class/CS532M.2007W... · p) Jonatan Schroeder Approximate DP for Sensor Network Management. Problem Introduction](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6c8393e9435218f26ff57e/html5/thumbnails/26.jpg)
Problem IntroductionDynamic Programming Formulation
Project
Williams et.al. – Approximate DynamicProgramming for Communication-Constrained
Sensor Network ManagementAn Experimental Review and Visual Tool
Jonatan Schroeder
Project Outline – CPSC532MUniversity of British Columbia
Mar 17, 2008
Jonatan Schroeder Approximate DP for Sensor Network Management