Evolution as a Tool for Understanding and Designing Collaborative SystemsWilfried ElmenreichMobile Systems Group/Lakeside LabsInstitute for Networked and Embedded SystemsAlpen-Adria Universität Klagenfurt

2Wilfried Elmenreich

Overview

• Cooperation theory

• The evolutionary approach

• Selection models

• Agent behavior rules

• A tool for evolving SOS

• Application examples

• Challenges

3Wilfried Elmenreich

Cooperation Theory

• Cooperation theory addresses the common tension between – what is good for the individual actor in the short run,

and – what is good for the group in the long run

• Typically employs game theory as the basis for analysis

• Robert Axelrod published Evolution of Cooperation in 1984– Still we are not done with answering this question

today...

4Wilfried Elmenreich

Application Examples

• A soccer team– Discipline, selfishness,

assistance/cooperation…

• Employees/company– „Teamwork“ issue

• Energy consumers and producers in the „Smart Grid“

Source: Wikimedia

5Wilfried Elmenreich

Prisoner‘s Dilemma

• Characterization of 1-move 2-person Prisoner’s Dilemma:

R: the reward for mutual cooperation

P: the punishment for mutual defection T: payoff for unilateral defectionS: the sucker’s payoff

• The Prisoner’s Dilemma is defined by T > R > P > S

• Mutual cooperation should be in favor over coordinated alternation of unilateral cooperation:R > (T+S) / 2

R,R

T,S

S,T

P,P

6Wilfried Elmenreich

Prisoner‘s Dilemma Strategies

• Always defect:– Best for individual actor in the short run– Bad for overall group

• Always cooperate:– Good for group– Rewards antisocial behavior

• A sophisticated strategy– Tit-for-Tat– Forgiveness concept– Retaliation concept

7Wilfried Elmenreich

Public Goods Game

• A social dilemma• Different story, same properties as

Prisoneer‘s Dilemma• Typical much more than 2 players• Altruistic behavior best for group• Best to defect for an individual (Freerider)

– Tragedy of the commons

8Wilfried Elmenreich

Some questions

• What is the most rational strategy?• What are the boundary conditions for a

strategy to be optimal?• Why do real systems converge to equilibria

with non-rational behavior?

• Put your research topic here!

9Wilfried Elmenreich

The Evolutionary Approach

• Research the evolution of rules via Darwinian process

• Model the social game (e.g. Public Goods Game) as the problem

• Evolve the behavioral rules of the agents

10Wilfried Elmenreich

Individual selection based onindividual fitness

• Select the soccer players that score the most goals

• A: altruistic agent• S: selfish agent

A A

AA

A S

S S

SS

S

S S

11Wilfried Elmenreich

Individual selection based ongroup fitness

• Select the best performing agents from the best performing groups

A

S

AS

S

S

A

S

AS

S

S

S

S

S

12Wilfried Elmenreich

Group selection based ongroup fitness

• Select the best performing agents from the best performing groups

A

S

A

S

S

S

A

S

A

A

S

A

A

S

A

13Wilfried Elmenreich

Kin selection

• Hamilton’s rule of inclusive fitness:if r b > c then cooperater relatednessb benefitc cost

• r = probability that two individuals have homologous alleles identical by descent

• J.B.S. Haldane: “I would lay down my life for two brothers or eight cousins.”

14Wilfried Elmenreich

The ongoing discussion: Group vs. Kin selection

• Social insects make a strong point for kin selection– Cooperation without being able to reproduce– But does not cover all evolved cooperations

• Spatial models make a strong point for group selection – Group selection is especially in favor for areas

with natural barriers– Analysis requires now a connection (network)

model

15Wilfried Elmenreich

Searching for the Rules

• Simulation of target system as playground

• Evolvable model of local behavior (e.g., fuzzy rules, ANN)

• Define goal via fitness function (e.g., maximize throughput in a network)

• Run evolutionary algorithm to derive local rules that fulfill the given goal

System modelGoals (fitness function)

Simulation of game

System modelGoals (fitness function)

Simulation of game

Explore solutions

Explore solutions

Evaluate & Iterate

Evaluate & Iterate

Analyze results

Analyze results

16Wilfried Elmenreich

Evolutionary Algorithm

17Wilfried Elmenreich

Agent behavior to be evolved

• Controls the agents of the SOS• Processes inputs (from sensors) and produces

output (to actuators)• Must be evolvable

– Mutation– Recombination

• We cannot easily do this with an algorithm represented in C code…

Agent

Control System„Agent‘s Brain“

18Wilfried Elmenreich

Artificial Neural Networks

• Each neuron sums up theweighted outputs of the other connected neurons

• The output of the neuronis the result of an activation function (e.g. step, sigmoid function) applied to this sum

• Neural networks are distinguished by their connection structure– Feed forward connections (layered)

– Recursive (Ouput neurons feed back to input layer)

– Fully meshed

19Wilfried Elmenreich

Evolving Neural Networks

3.2 -1.2

3.2

3.2

-0.1

-4.2

0.20.0

3.5 -1.2

3.2

3.2

-0.1

-4.2

0.20.0

Mutation

0.0 -1.2

1.2

3.2

-0.1

1.2

1.20.0

3.5 2.2

3.2

3.2

-0.1

-4.2

0.20.5

Recombination

3.2 -1.2

3.2

3.2

-0.1

-4.2

0.20.0

20Wilfried Elmenreich

A Framework for Evolutionary Design

• FREVO (Framework for Evolutionary Design)

• Modular Java tool allowing fast simulation and evolution

• „Frevo“ means also a hot, „boiling“ dance around here

21Wilfried Elmenreich

Framework for Evolutionary Design

• FREVO defines flexible components for – Controller representation

– Problem specification

– Optimizer

22Wilfried Elmenreich

Giving FREVO a Problem

• Basically, we need a simulation of the problem• Interface for input/output connections to the

agents– E.g. for the public goods game:– Your input last round– Your revenue

• Feedback from a simulation run -> fitness value

23Wilfried Elmenreich

Example: Evolving Team Play in Robot Soccer

• Simulated robot soccer teams• Goal evolve behavior for individual players• Team selection strategy

– Not the top scorer but thewinning teams are selected

– This selection forcescooperation within team

24Wilfried Elmenreich

Modeling of individual soccer players

25Wilfried Elmenreich

Agent I/O by example of an ANN

26Wilfried Elmenreich

Composite fitness function

27Wilfried Elmenreich

Results (movie)

http://www.youtube.com/watch?v=cP035M_w82shttp://www.youtube.com/watch?v=cP035M_w82s

28Wilfried Elmenreich

Evolution of cooperative behavior

29Wilfried Elmenreich

Evolution of cooperative behavior

• Public goods game model• 6 players randomly selected from the pool• Neural network model for each player• Player decides how much money to contribute to

a pot• Pot function: money is tripled and evenly

distributed among all• Players have knowledge about their own and

other‘s decisions from previous round

30Wilfried Elmenreich

Freerider behavior evolves

31Wilfried Elmenreich

A synergy effect

• Make the pot function overproportional to the submitted money

• E.g. by a square function: pot = c x2

• This significantly increases the motivation for cooperation

• Does this immediately lead to cooperation?

32Wilfried Elmenreich

Public Goods Game with Synergy Model

• Still nocooperation?

33Wilfried Elmenreich

Cooperation evolves in the long term

• It needs some random fluctuations to set the coopertive strategy loose

• Quickly convergesto (partial) cooperative behavior

34Wilfried Elmenreich

Not the end of the story…

• Spatial model could enable group-selection-like evolution of altruism– But what spatial model?

• 2D von Neumann neighborhood• 2D Moore neighborhood• Hexagon neighborhood• Hypercubes• Network models

• Extending the model with team selection for agents could enable kin selection feature– At least as many questions

35Wilfried Elmenreich

Challenges

• Evolutionary modeling gives new insight to the problem– Answers the question: Why is a particular non-optimal

behavior so prevalent in a population?

• Evolutionary modeling also raises new questions– How should the agent‘s brain be structured?– How should the selection function be modeled?– What is an appropriate spatial model?

36Wilfried Elmenreich

Visit us

• (open source):

www.frevotool.tk• Project MESON (Design Methods for Self-

Organizing Systems):meson.lakeside-labs.com

• Lakeside Labs Clusterwww.lakeside-labs.com