1

How Scenario Analysis Can Contribute toABMS Validation

Yusuke Goto (Iwate Pref. Univ.)Shingo Takahashi (Waseda Univ.)

The 7th International Workshop on Agent-based Approachesin Economic and Social Complex SystemsJanuary 17th, 2012Kansai University, Osaka, Japan

Background of Our Study

2

Current issues of organization management:• Increased uncertainty and complexity• Limited knowledge of the organization•Organizational behavior is unpredictable

Decision support for managers:1) Optimization approach

2) Better-informed decision making (North & Macal, 2007)

•Predict the organizational behavior•Identify an optimal solution

•Deepen the understanding of a solution’spossible effects on the organizational behavior•Collaboratively choose and invent a solution

North and Macal: Managing Business Complexity, Oxford Univ. Press (2007)

Better-informed Decision Making

3

Tool for supporting better informed decision making•Agent-based modeling and simulation (ABMS)•Scenario analysis:

-possible outcomes-a mechanism that results in a specific outcome

in a given situation from the simulation of a solution

to gain the information about

Hurdle for ABMS practitioners•Clients do not appreciate the ABMS analysis

•Persuading their clients of ABMS validity

Clients perceive ABMS analysis as an opaque,arbitrary process.

If clients do not trust the ABMS analysis, they will reject their proposals or be less committedto implement them.

Aims and outline

4

Aims of our study

•Originally intend to help ABMS researchersanalyze the simulation results in a valid manner•Persuade stakeholders of the validity ofABMS-based proposals

How can scenario analysis contribute to ABMS validation?

Outline• Introduction of ABMS• Scenario analysis:landscape analysis & micro-dynamics analysis

• Validating ABMS:Arbitrariness of presenting simulation results

• Discussion:ABMS validation by stakeholders

Agent-based Modeling

5

•Consider which MCS can achieve a business goal•Indirectly control the organizational behavior•Lack sufficient information to model the organization

Target situation

Architecture of Agent-based Models (ABMs)

Characteristics of ABMs

6

•Modelers lack sufficient information about elements and interactions in the system.•They cannot determine the valid elements and interactions in advance.•They must model the elements and interactions on the basis of their current understanding of the system.

Business complexity:uncertainties and complex interactions within and outside of the organizations

Autonomous agents:•Agent’s internal decision-making model consists of many parameters representing his/her values, attributes, perception of the situation...•This model may have a probabilistic deterministic process to present fluctuations in the behavior.

Agent-based Simulation

7

1)uncertain factors of the ABM:

To run an agent-based simulation (ABS)ABMS researchers must determine...

2)a random seed for fluctuating factors

•They must make assumptions and create specifications of uncertain factors•If their original assumptions are inadequate, they revise their assumptions and re-specify the uncertain factors

ABMS is an iterative process (≠ waterfall process)

•To generate pseudo-random number sequences- For representing fluctuations of behaviors- For determining specific values of uncertain parameters

•To assure ABMS researchers of the reproducibility of ABS results

Characteristics of ABS

8

Running an ABSThe behavior observed after every run of the ABS can vary considerably because of

•Complex interactions between agents•Uncertainty of agents’ decision-making model•Fluctuations of agents’ behavior•Micro-macro links

Evaluating ABS results•Evaluation based on a single run is inadequate•Multiple runs are required:

- Effect of uncertainties: which values of uncertain parameters are realized-Behavioral fluctuation: how probabilistic behaviors are realized

Scenario Analysis

9

TerminologyScenario:

•ABM generates various paths in a given scenario•Each path represents a possible outcome of the scenario

Set of critical experimental parameters that may have an effect on future system behaviors

Path:Time-series system behavior generated by an ABS

Aims and methodsTo deepen the understanding of the target system:

•Possible outcomes in given scenarios•Mechanism by which an outcome achieved

1)Landscape analysis2)Micro-dynamics analysis

Scenario analysis consists of two sub-analysis:

Landscape Analysis

10

Landscape of possible outcomes that result from the ABS in scenarios of concern

Methods of Landscape Analysis

11

Drawing a landscape

Scenario orderingQualitative and independent scenarios:

•Possible maximum performance value•Possible average performance value•Range of possible outcomes

Sort the scenarios from the perspective of similarity of each scenario’s characteristics to those of other scenario characteristics

1)Scenarios defined by the managerial intention2)Performance index that reflects the system’s behavior

3)A point in time4)Scenario ordering

Sort the scenarios from a specific analytic perspective

Micro-dynamics Analysis (1/2)

12

To explain why a particular outcome is generated from the viewpoint of the dynamics of micro-level parameters.

Aims

Method (typical approach)

•Every macro-level behavior is formed by micro-level agents’ behaviors•Explanation is logical, consistent with the theories and assumptions of the ABM

1)Choose two types of outcomes in the target scenario•a divergent outcome•an ordinary outcome

2)Comparative analysis of the two selected outcomes•Identify a cause of the divergence•Heuristic method that relies on analyst’s sense•Validity or robustness is uncertain

Micro-dynamics Analysis (2/2)

13

Quantitative information for judging validity•Calculating the degree of approximation between the two outcomes’ micro-dynamics•Clustering the possible outcomes

Validating ABMS: Brief Review

14

Verification and validation (V&V)Play an essential role in persuading decision-makers to ABMS and ABMS researchers’ proposals

•Conceptual validity•Operational validity•Data validity

Degree of homomorphism between one system and a second system that purportedly represents

ValidityFormal definition:

Practical definition:

Validating ABMS: Validation Strategy

15

•ABM is a common language or text•ABM reflects stakeholders’ perception and understanding •ABM should continually be revised

Model resolution & validation strategy

Participative approach

Abstract model Middle range model Facsimile model

Purpose

Strategy

Scope

Understanding of a complex target

systemCommon theory of a

targeted problem class Scenario analysis

•Docking•M2M analysis

•Stylized Fact Analysis•History Friendly Approach

?ABMS

researchers ABMS researchers ABMS researchers,Stakeholders

To obtain stakeholder acceptance through participative model building

Participative approach

16

•There is no perfect means of validating ABMSABMS validation is a type of a social process•North stresses the importance of ...

ABMS validation for business organization

-Users’ belief in the ABMS results- Fulfilling user’s for business applications

•Time constraints to participate in the complete modeling processDifficulty of achieving sufficient stakeholder validation through only the participative modeling•Share the ABMS results by ABMS researchers after stakeholders’ partial participation in the modeling process

Participative approach in practice

Sharing ABMS results

17

Traditional presentation of ABMS result:1.Outcome of a typical run 2.statistical result

Why is this outcome selected? How is the result generated?

•Arbitrariness of choosing and presenting results of ABMS•Traditional presentation has limited effectiveness in persuading stakeholders

Discussion (1/2)

18

Our idea for validationLandscape analysis:Yields a landscape of possible outcomes in the scenarios

Micro-dynamics analysis:

•Identify possible outcomes in the landscape•Confirm their positions in the possible outcomes when ABMS researchers choose outcomes for further micro-dynamics analysis

Links micro-level parameter dynamics to the macro-level behavior of the system

•Find a logical explanation of why the chosen outcome is appeared

Scenario analysis have no arbitrariness of choosing and presenting results of ABMS

Discussion (2/2)

19

Framework of ABMS validation by stakeholders

Characteristics of the framework•Previous participatory approaches emphasize the stakeholder participation in the modeling phase•Organizational learning: re-modeling and re-running

Summary

20

• Landscape analysis:Stakeholders can evaluate the validity of choosing the outcomes

• Micro-dynamics analysis:Stakeholders can evaluate the validity of the explanation

How can scenario analysis contribute to validation by stakeholders?

Scenario analysis solves the problem of arbitrariness of presenting ABMS results

21

Discussion (3/3)

22

System needs for effective persuasion:

•To perform scenario analysis in real time & interactively•Rich graphical visualization•To draw a landscape of possible outcomes•To assist ABMS researchers in performing micro-level log traces