Distributed Optimization

Yen-Ling KuoDer-Yeuan Yu

May 27, 2010

Outline [Yu]

• Optimized Sensing: From Water to the Web• Distributed Dynamic Programming• Distributed Solutions to Markov Decision

Problems

Optimized Sensing

• Problem Statement• Greedy Algorithms and Submodularity• Robust Sensing Optimization with Saturate

Algorithm• Application in Blogs

Problem Statement

• How do we detect contamination in drinking water distribution networks?

• Which blogs should we read to learn about the biggest, newest stories on the Web?

• Fundamental Question: How can we get the most useful information at minimum cost (limited resources)?

Solutions to Optimized Sensing

• Covers fields of statistics, machine learning, sensor networks, and robotics

• With partially observable Marko decision processes, we can get optimal solutions

• But it is difficult to scale POMDP to large problems

• Introducing a new algorithm based on submodularity

Formulation

• Sensing quality function F(A)– A: the set of sensor locations Si (i=1~k)– V: the set of all locations

• We can also have cost constraints– Total cost of sensor deployment no greater than

the budget• Goal: Find A*

– This is NP-hard already

Greedy Algorithm

• Iteratively find Si

• This naïve algorithm actually performs pretty well– Why? Submodularity– We get near-optimal solutions

• Submodularity: diminishing returns

Diminishing Returns

Cost-Effective Lazy Forward-Selection (CELP)

• Greedy algorithm• Lazy evaluations

– Delaying computation until the result is required– A computational technique

Robust Sensing Optimization

• Idea: Protect system against adversaries that know of our deployment of sensors

• Goal: Maximize the worst-case detection performance

• Approach

• Unfortunately, this naïve extension can fail

Failure of Greedy Algorithm on Worst-Case Scenarios

• I1, I2: two contamination events• S1, S2, S3: three possible sensor locations

– S1: detect I1 immediately, but never I2– S2: detect I2 immediately, but never I1– S3: detect both I1 and I2, but only after a long time

• We can only place two sensors• Greedy would pick S3 first and then either S1 or S2• But we know the optimal solution should be S1 and

S2• Solution? Saturate algorithm

Saturate Algorithm

• Idea: reduce the non-submodular worst-case objective to a submodular optimization problem– Transform non-submodular to submodular

• Transformation– Guess optimal solution value C using binary search– Try to find A such that F(A) is no less than C

Performance of Saturate

From Water to the WebBlog Reading

• Problem: Information cascading

Improvements

• Number-of-posts (NP) model– Reading a big blog can be time-consuming, so they define

the cost to be the number of posts

• CELP tends to choose blogs with many posts• NP model tends to choose summarizer blogs

– But stories appear in summarizer blogs a little late

Other Thoughts

• What if we are looking for stories to read instead of blogs to read?– We can reverse our information management goal– Find posts instead of blogs– Ref. 10

• End of Paper

Distributed Dynamic Programmingfor Path Planning

• Asynchronous Dynamic Programming• Learning Real-Time A*

Asynchronous Dynamic Programming

• Propagate costs from target to start locations

Learning Real-Time A* (LRTA*)

LRTA*(n)

• LRTA with n agents• Faster

– Agents break ties differently– They can share the same h-value table

LRTA*(2)

Distributed Solutions to Markov Decision Problems

• As previously mentioned in the Water to Web paper, MDPs can be difficult to scale to big problems

• Solution: Exploit independence properties• We address the modularity of actions

Action Selection in multiagent MDPs

Implementation

Subtask Distribution

• A global problem is broken down into subtasks

• Subtasks are distributed among agents

• Each agent has different capabilities

Problem

25

Contract Net

• Stages– Recognition– Announce– Bidding– Awarding & Expediting

• Initial assignment: Not optimal• Anytime property

– Improve assignment in negotiation process

26

Assignment problem

• Problem definition– A set N of n agents– A set X of n objects– A set M N × X of possible assignment pairs, and⊆– A function v : M → R

• Find optimal assignment

X NM

27

Corresponding Linear Program

• Linear program (LP) formulation

Profit maximization

Resource constraint

Optimal solution

• Any LP can be solved in polynomial time O(n3)

28

Competitive Equilibrium

• Consider a price vector p = (p1, …, pn)– The utility from an assignment j to agent i is

u(i, j) = v(I, j) - pj

• A feasible assignment S and a price vector p are in competitive equilibrium when for every pairing (i, j)

S it is the ∈ case that ∀ k, u(i, j) ≥ u(i, k)

29

Every agent will not change its selection

S is a optimal solution

Naïve Auction Algorithm

• Round-robin style• Bid increment is the difference between the

utility to i of the best and second-best object

30

The agent will not overbid

Problem in Naïve Auction

• When more than one object offers maximal utility for an agent– Bid increment is zero

31

Terminating Auction Algorithm

• Modify the bid increment–

32

ε-competitive equilibrium: u(i, j) + ε ≥ u(i, k)Agents may overbid some objects

Scheduling Problem

• Problem definition– N is a set of n agents– X is a set of m discrete and consecutive time slots– q = (q1, . . . , qm) is a reserve price vector

– v = (v1, . . . , vn), where vi is the valuation function of agent I

• Find optimal allocation

33

F

Corresponding Integer Program

• Integer program (IP) formulation

• IPs are not solvable polynomial time

34

Competitive Equilibrium – General Form

• Definition– For all i N it is the case that ∈

Fi = argmaxT X ⊆ (vi(T) − ∑j|xj T ∈ pj)

– For all j such that xj F∈ ∅ it is the case that pj = qj

– For all j such that xj F∈ ∅ it is the case that pj ≥ qj

• May not exist competitive equilibrium

35

Has a competitive equilibrium solution ↕

The LP relaxation of the associated integer program has a integer solution.

Ascending Auction Algorithm

• Center advertise an ask price• Bid increment is constant

36

Problem in Ascending Auction

• If the increment is too large

• May not converge to optimal solution37

Social Laws and Conventions

• Social law– A restriction on the given strategies of the agents– Induce a sub-game

• Social convention– The sub-game consists of a single strategy for all agent

• Other topics– Social goal negotiation– Social norm negotiation– ….

38