Agent-Organized Networks for Dynamic Team Formation

Multi-Agent Planning and Learning Laboratory (MAPLE)

Department of Computer Science and EE

University of Maryland Baltimore County

Matt Gaston

mgasto1@cs.umbc.edu

Marie desJardins

mariedj@cs.umbc.edu

Agent-Organized Networks 2

Overview

Introduction and Motivation

Team Formation

Agent-Organized Networks

Experimental Results

Related Projects: Connections Model AONs AONs for Production and Exchange Stable Team Formation

Future Work and Conclusions

Agent-Organized Networks 3

Introduction and Motivation

Agent-Organized Networks 4

Introduction: Multi-Agent Systems

Agent: Autonomous, intelligent software system. Physical (robot, autonomous vehicle, mobile sensor) or virtual

(search, travel planning, trading / e-commerce, information retrieval)

MAS: “Community” of agents – competitive or cooperative Connections form a “social network” of agents

Agent-Organized Networks 5

Why Adapt?

Multi-agent systems are growing in popularity and size Technologies like the Semantic Web support the

deployment and evolution of large-scale, dynamic multi-agent systems

Agent cognitive capacities are limited, preventing all agents from knowing/interacting with all other agents

Previous findings suggest that network structure plays an essential role in understanding team formation dynamics in multi-agent systems

Identifying the “best” network structure is difficult or impossible to do a priori

Solution: Agent-Organized Networks

Agent-Organized Networks 6

Team Formation[AAMAS 2005]

Agent-Organized Networks 7

Multi-Agent Team Formation Model

Agents must form teams to complete tasks

Agent states: Uncommitted Committed Active

Tasks are advertised to the network of agents

A valid team: Connected path in network Task skill requirements met Formed within time

constraints

1 1

22

2

22

3

1, 1, 2, 3

2, 2

1, 2, 2, 2, 4

4

Agent-Organized Networks 8

Multi-Agent Team Formation Model

Some model details Parameters

Number of agents: N Skill diversity: Task introduction interval: Team/task size: T Advertisement duration: Task duration: Network structure

Organizational Efficiency# of tasks successfully completed

total # of tasks advertisedefficiency =

1 1

22

2

22

3

4

Agent-Organized Networks 9

Team Joining Strategy

With some initiation probability,start a new team if needed:

Always join a team if it’s already beenstarted, and it needs your skill.

Considering each task in random order...

Agent-Organized Networks 10

Agent-Organized Networks

Agent-Organized Networks 11

Agent-Organized Networks

Definition: An agent-organized network (AON) is an organizational network structure, or agent-to-agent interaction topology, that is the result of local rewiring decisions made by the individual agents in a networked multi-agent system.

Design considerations: Local perception of global performance Adaptation triggers Rewiring strategies

Evaluation metrics: Learning rate Stability Structural properties of resulting networks

Agent-Organized Networks 12

Structure-Based Adaptation

Adapt based on preferential attachment Natural network formation process that leads to scale-free networks

Adaptation trigger (random): Probability of adaptation for each uncommitted agent: 1/N

Rewiring strategy: Disconnect from a random neighbor Connect to some neighbor’s neighbor with probability

Agent-Organized Networks 13

Performance-Based Adaptation

Adaptation trigger: Adapt if performance drops below neighbors’ average performance:

Rewiring strategy: Drop the lowest-performing neighbor:

Add a connection to the highest-performing neighbor ak of the highest-performing neighbor al:

Agent-Organized Networks 14

Results

Agent-Organized Networks 15

Experimental Setup

Initial network structure: Random geometric graph Randomly place agents in a unit square Connect agents that are closer than d units apart Use the minimal d that guarantees all neighbors have at least one

edge

Run team formation with no adaptation to establish baseline Run with each adaptation strategy separately Results are an average of 50 runs

Agent-Organized Networks 16

Results: Summary

Significant performance improvement (over baseline) for both AON methods

Agent-Organized Networks 17

Stability of Networks

Structure-based AONs outperform performance-based AONs, but result in substantially more rewirings

Performance-based AONs are more efficient (“better value” if adaptation cost is in similar units to performance measure)

Agent-Organized Networks 18

Evolution of the Network: Structure-Based

Converges to a network with hub structure and short average path length

Agent-Organized Networks 19

Evolution of the Network: Performance-Based

Convergence to short-average-path-length structure happens more slowly

Qualitatively similar structure to strategy-based (but in this case not by design!)

Agent-Organized Networks 20

Connections Model AONs[AAAI 2005 Workshop on Multi-Agent Learning]

Agent-Organized Networks 21

The (Symmetric) Connections Model

Symmetric when ij = and cij = c for all i and j

0 < < 1 is the value of a relationship, discounted by distance

c is the cost of a direct connection

(Jackson & Wolinsky 1996; Jackson 2002)

Agent-Organized Networks 22

Dynamic Network Formation in SCM

Based on pairwise stability (Watts 2001): At each iteration:

Two agents meet (are selected) at random (synchronous) If they have a connection, they remove the connection if at least one

of them benefits -- unilateral deletion If the do not have a connection, they add a connection if it is

mutually beneficial -- bilateral creation

But . . .

Agent-Organized Networks 23

Experiment: Watts Dynamic Network Formation

= 0.9, c = 0.8, optimal = 7878.42

Agent-Organized Networks 24

A (Simple) Multi-Agent Learning Approach

Goals: Eliminate need for “global” knowledge Eliminate need for “global” computation Maintain bilateral network formation (agents agree to create link) Follow dynamic network formation process of Watts On-line learning

Approach Stateless Q-Learning (Claus & Boutilier 1998) A = { add, delete, nothing }

Agents add connection if both have largest Q value for add (bilateral) Agents remove connection if one has largest Q value for delete (unilateral) Reinforcement signal comes from omniscient oracle (!)

Agent-Organized Networks (AONs) “Distributed Annealing”

Agent-Organized Networks 25

Experiment: Learning to Form Networks

Adaptive Learning Rate: Win or Lose Fast (WoLF) (Bowling & Veloso 2002)

= 0.9, c = 0.8, optimal = 7878.42

Agent-Organized Networks 26

Experiment: Adding an Unselfish Agent

= 0.9, c = 0.8, optimal = 7878.42

Agent-Organized Networks 27

AONs for Production and Exchange[AAAI 2005]

Agent-Organized Networks 28

A Model of Production and Exchange

n agents in an artificial economy with two goods

Each agent i possesses g1i units of good 1 and g2

i units of good 2

Each agent is a producer of either good 1 or good 2

At each iteration of the model, the agents are selected in random order and choose between initiating trade with another agent or producing their respective good in order to maximize utility

Agent utility:

(Wilhite 2001: 2003)

fully rational behavior

Agent-Organized Networks 29

Push Referral AON Strategies

Random referral: agent selected randomly from Nj(i)

Degree referral:

Production referral:

Definition: Assuming that agent i is adapting its connection to agent j, a push referral is a local rewiring by i from j to an agent in Nj(i)

Agent-Organized Networks 30

Results

production referral

degree referral

random selection

n = 400, q = 30, = 0.05= = 0.1= = 0.1initialized values to 1

Agent-Organized Networks 31

Stable Team Formation[AAAI 2004 Workshop on Team/Coalition Formation]

Agent-Organized Networks 32

Economic Model of Team Formation

Share-based scheme for pay-off distribution Team’s revenue is stored in “team account” Team members get shares for joining and working Share value = team account / # outstanding shares

Agents bound to the team by a contract Joining Shares, Sjoin : sign-on bonus

Commission, Scomm : shares given to the agent for every task completed by the team in which the agent actively participates

Dividend, Sdiv : shares given to the agent for every task completed by the team in which the agent does not participate

(Dividend < Commission) Penalty, p : the amount to be paid to the team when leaving the team

Agent-Organized Networks 33

Results: Effect of Deadlines

Agent-Organized Networks 34

Results: Stable vs. Dynamic Agents

Agent-Organized Networks 35

Conclusions and Future Work

Summary: AONs based only on local knowledge can improve team

formation in networked MAS AON ideas can also be applied to other MAS domains and

models Stability can be achieved through a contractual model of team

formation

Future Work: Quantitative analysis of post-adaptation network structures Learning individual agent team selection strategies

[JAAMAS 2006] Skill placement and replacement for dynamic team formation

Agent-Organized Networks 36