distributed control in multi-agent systems: design and analysis

Distributed Control in Multi-agent Distributed Control in Multi-agent Systems: Design and AnalysisSystems: Design and Analysis

Kristina LermanAram Galstyan

Information Sciences InstituteUniversity of Southern California

01/23/2002

ISI

USC Information Sciences Institute K. LermanDistributed Control in MAS

Design of Multi-Agent SystemsDesign of Multi-Agent Systems

Multi-agent systems must function inDynamic environmentsUnreliable communication channelsLarge systems

SolutionSimple agents

No reasoning, planning, negotiationDistributed control

No central authority

01/23/2002

ISI


Advantages of Distributed Advantages of Distributed ControlControl

• Robust• tolerant of agent error and failure

• Reliable• good performance in dynamic environments

with unreliable communication channels• Scalable

• performance does not depend on the number of agents or task size

• Analyzable• amenable to quantitative analysis

01/23/2002

ISI


Analysis of Multi-Agent SystemsAnalysis of Multi-Agent SystemsTools to study behavior of multi-agent systems• Experiments

• Costly, time consuming to set up and run• Grounded simulations: e.g., sensor-based simulations of

robots• Time consuming for large systems

• Numerical approaches• Microscopic models, numeric simulations

• Analytical approaches • Macroscopic mathematical models• Predict dynamics and long term behavior • Get insight into system design

• Parameters to optimize system performance• Prevent instability, etc.

01/23/2002

ISI


DC: Two Approaches and DC: Two Approaches and AnalysesAnalyses

• Biologically-inspired approach• Local interactions among many simple agents

leads to desirable collective behavior• Mathematical models describe collective

dynamics of the system• Markov-based systems

• Application: collaboration, foraging in robots• Market-based approach

• Adaptation via iterative games• Numeric simulations • Application: dynamic resource allocation

Biologically-Inspired ControlBiologically-Inspired Control

01/23/2002

ISI


Analysis of Collective BehaviorAnalysis of Collective Behavior

Bio control modeled on social insects• complex collective behavior arises in simple,

locally interacting agents

Individual agent behavior is unpredictable• external forces – may not be anticipated• noise – fluctuations and random events • other agents – with complex trajectories• probabilistic controllers – e.g. avoidance

Collective behavior described probabilistically

01/23/2002

ISI


Some Terms DefinedSome Terms Defined

• State - labels a set of agent behaviors• e.g., for robots Search State = {Wander,

Detect Objects, Avoid Obstacles}• finite number of states• each agent is in exactly one of the states

• Probability distribution• = probability system is in

configuration n at time t • where Ni is number of agents in

the i’ th of L states

),( tnP

),,( 1 LNNn

01/23/2002

ISI


Markov SystemsMarkov Systems

• Markov property: configuration at time t+t depends only on configuration at time t

• also, • change in probability density:

.),(),|,(),(

n

tnPtnttnPttnP

n

n

tnPtnttnP

tnPtnttnPtnPttnP

),(),|,(

),(),|,(),(),(

n

tnttnP 1),|,(

01/23/2002

ISI


Stochastic Master EquationStochastic Master Equation

In the continuum limit,

with transition rates

nn

tnPtnnWtnPtnnWdt

tndP ),();|(),();|(),(

ttnttnPtnnW

t

),|,(lim);|(0

0t

01/23/2002

ISI


Rate EquationRate Equation

Derive the Rate Equation from the Master Eqn

• describes how the average number of agents in state k changes in time

• Macroscopic dynamical model

k

kk

kk tNtkkWtNtkkW

dttNd

)();|()();|()(

Collaboration in RobotsCollaboration in Robots

01/23/2002

ISI


Stick-Pulling Experiments Stick-Pulling Experiments (Ijspeert, Martinoli & Billard, 2001)(Ijspeert, Martinoli & Billard, 2001)

• Collaboration in a group of reactive robots• Task completed only through collaboration• Experiments with 2 – 6 Khepera robots• Minimalist robot controller

A. Ijspeert et al.

01/23/2002

ISI


Experimental ResultsExperimental Results

Key observations• Different

dynamics for different ratio of robots to sticks

• Optimal gripping time parameter

Flowchart of robot’s controllerstart look for sticks

object detected?

obstacle?

gripped?

grip & wait

time out?

teammatehelp?

release

obstacleavoidance

success

Ijspeert et al.

Y

N

Y

N

N

NN

Y

Y

Y

State diagram for amulti-robot system

search

grip

s u

01/23/2002

ISI


Model VariablesModel Variables• Macroscopic dynamic variables

Ns(t) = number of robots in search state at time tNg(t) = number of robots gripping state at time tM(t) = number of uncollected sticks at time t

• Parameters• connect the model to the real system = rate of encountering a stickRG = rate of encountering a gripping robot = gripping time

01/23/2002

ISI


Mathematical Model of Mathematical Model of CollaborationCollaboration

);()()()(

)()()()()()(

ttNtMtN

tNtNRtNtMtNdt

tdN

gs

gsGgss

Initial conditions: 00 )0(,0)0(,)0( MMNNN gs

find & grip stickssuccessful collaboration

unsuccessful collaboration0NNN gs

consttM )( for static environment

01/23/2002

ISI


Dimensional AnalysisDimensional Analysis

• Rewrite equations in dimensionless form by making the following transformations:

• only the parameters and appear in the eqns and determine the behavior of solutions

• Collaboration rate• rate at which robots pull sticks out

)()(1)(~),,( 00 NfNtntntR

G

s

RMN

MtMtNtNtn

~,/

,,/)()(

00

000

01/23/2002

ISI


Searching Robots vs TimeSearching Robots vs Time

=5=0.5

01/23/2002

ISI


Collaboration Rate vs Collaboration Rate vs

Key observations• critical • optimal gripping

time parameter=1.0

=1.5

=0.5

01/23/2002

ISI


Comparison to Experimental Comparison to Experimental ResultsResults

=1.0

=1.5

=0.5

Ijspeert et al.

01/23/2002

ISI


Summary of ResultsSummary of Results

• Analyzed the system mathematically• importance of • analytic expression for c and opt• superlinear performance

• Agreement with experimental data and simulations

Foraging in RobotsForaging in Robots

01/23/2002

ISI


Robot ForagingRobot Foraging• Collect objects scattered in

the arena and assemble them at a “home” location

• Single vs group of robots• no collaboration• benefits of a group

• robust to individual failure• group can speed up collection

• But, increased interference

Goldberg & Matarić

01/23/2002

ISI


Interference & Collision Interference & Collision AvoidanceAvoidance

• Collision avoidance

• Interference effects• robot working alone is more efficient• larger groups experience more interference• optimal group size: beyond some group size,

interference outweighs the benefits of the group’s increased robustness and parallelism

01/23/2002

ISI


State DiagramState Diagram

start look for pucks

object detected?

obstacle? avoid obstacle

grab puck

go home

searching homing

avoidingavoiding

01/23/2002

ISI


Model VariablesModel Variables• Macroscopic dynamic variables

Ns(t) = number of robots in search state at time tNh(t) = number of robots in homing state at time tNs

av(t), Nhav(t) = number of avoiding robots at time t

M(t) = number of undelivered pucks at time t• Parameters

r = rate of encountering a robotp = rate of encountering a puck = avoiding timeh

0 = homing time in the absence of interference

01/23/2002

ISI


Mathematical Model of ForagingMathematical Model of Foraging

savh

hsSr

havhsp

s NNNNNNNMNdt

dN

11][ 0

hh

Ndt

dM1

Initial conditions: 00 )0(,)0( MMNNs

Average homing time: 0

0 1 Nrhh

hh

havhhr

havhsp

h NNNNNNNMNdt

dN

11][][ 0

01/23/2002

ISI


Searching Robots and Pucks vs Searching Robots and Pucks vs TimeTime

robots

pucks

01/23/2002

ISI


Group Efficiency vs Group SizeGroup Efficiency vs Group Size

=1

=5

01/23/2002

ISI


Sensor-Based SimulationsSensor-Based Simulations

Player/Stage simulatornumber of robots = 1 - 10number of pucks = 20arena radius = 3 mhome radius = 0.75 mrobot radius = 0.2 m robot speed = 30 cm/s puck radius = 0.05 m rev. hom. time = 10 s

01/23/2002

ISI


Simulations ResultsSimulations Results

0

200

400

600

800

1000

1200

1400

1600

0 2 4 6 8 10

number of robots

time

(s)

avoid time = 3savoid time = 1smodel (3 s)model (1 s)

01/23/2002

ISI


Simulations ResultsSimulations Results

0

0.002

0.004

0.006

0 2 4 6 8 10

number of robots

effic

ienc

yt=3 smodel 3 st=1 smodel 1 s

01/23/2002

ISI


SummarySummary• Biologically inspired mechanisms are

feasible for distributed control in multi-agent systems

• Methodology for creating mathematical models of collective behavior of MAS

• Rate equations • Model and analysis of robotic systems

• Collaboration, foraging• Future directions

• Generalized Markov systems – integrating learning, memory, decision making

Market-Based ControlMarket-Based Control

01/23/2002

ISI


Distributed Resource AllocationDistributed Resource Allocation

• N agents use a set of M common resources with limited, time dependent capacity LM(t)

• At each time step the agents decide whether to use the resource m or not

• Objective is to minimize the waste

where Am(t) is the number of agents utilizing resource m

t m

mm tLtAw 2))()((

01/23/2002

ISI


Minority GamesMinority Games

• N agents repeatedly choose between two alternatives (labeled 0 and 1), and those in the minority group are rewarded

• Each agent has a set of S strategies that prescribe a certain action given the last m outcomes of the game (memory) 00

0001

010

011

100

101

110

111

0 1 1 0 0 1 0 1

strategy with m=3

• Reinforce strategies that predicted the winning group

• Play the strategy that has predicted the winning side most often

inputaction

01/23/2002

ISI


MG as a Complex SystemMG as a Complex System

• Let be the size of the group that chooses ”1” at time t• The “waste” of the resource is measured by the standard

deviation - average over

time• In the default Random Choice Game (agents take either action

with probability ½) , the standard deviation is

2/N

)(tA

,)()( 22 tAtA

0

0.25

0.5

0.75

1

1.25

1.5

2 4 6 8 10 12 14

memory length

stan

dard

dev

iatio

nFor some memory size the waste is smaller than in the random choice game

Coordinated phase

01/23/2002

ISI


Variations of MGVariations of MG• MG with local information

Instead of global history agents may use local interactions (e.g., cellular automata)

• MG with arbitrary capacitiesMG with arbitrary capacities The winning choice is “1” if where is the capacity, is the number of agents that chose “1”

LtA )( )(tAL

To what degree agents (and the system as a whole) can coordinate in externally changing environment?

01/23/2002

ISI


MG on Kauffman NetworksMG on Kauffman NetworksSet of N Boolean agents: Nisi ..1},1,0{

Kjk j ..1},{

)()(),()( 01

tLLtLtstAN

ii

Each agent hasA set of K neighbors A set of S randomly chosen Boolean functions of K variables

))(),...(()1(1

tstsFtsK

MAX

kkj

ii

SjF ji ..1,

Dynamics is given by

The winning choice is “1” if where)()( tLtA

Global measure for optimality:

T

ttLtA

T 1

22 )]()([1

For the RChG (each agent chooses “1” with probability )

NtLt /)(

T

ttdtT

N0

2 ]1[1

01/23/2002

ISI


Simulation ResultsSimulation ResultsK=2 networks show a tendency towards self-organization into a coordinated phase characterized by small fluctuations and effective resource utilization

K=2 Traditional MG m=6

01/23/2002

ISI


Results (continued)Results (continued)

Coordination occurs even in the presence of vastly different time scales in the environmental dynamics

01/23/2002

ISI


ScalabilityScalability

For K=2 the “variance” per agent is almost independent on the group size,

constN

In the absence of coordination N

01/23/2002

ISI


Phase Transitions in Kauffman Phase Transitions in Kauffman NetsNets

Kauffman Nets: phase transition at K=2 separating ordered (K<2) and chaotic (K>2) phases

For K>2 one can arrive at the phase transition by tuning the homogeneity parameter P (the fraction of 0’s or 1’s in the output of the Boolean functions)

K=378.0cP

The coordinated phase might be related to the phase transition in Kauffman Nets.

01/23/2002

ISI


Summary of ResultsSummary of Results

• Generalized Minority Games on K=2 Kauffman Nets are highly adaptive and can serve as a mechanism for distributed resource allocation

• In the coordinated phase the system is highly scalable

• The adaptation occurs even in the presence of different time scales, and without the agents explicitly coordinating or knowing the resource capacity

• For K>2 similar coordination emerges in the vicinity of the ordered/chaotic phase transitions in the corresponding Kauffman Nets

01/23/2002

ISI


ConclusionConclusion

• Biologically-inspired and market-based mechanisms are feasible models for distributed control in multi-agent systems

• Collaboration and foraging in robots• Resource allocation in a dynamic environment

• Studied both mechanisms quantitatively• Analytical model of collective dynamics• Numeric simulations of adaptive behavior

distributed control in multi-agent systems: design and analysis

Documents

lermandistributed control

set of agent behaviorse

average number of agents

collective dynamics

avoidancecollective

time t dt

probability system

long term behavior