1

SIGMA WorkshopPart 1: Pattern-oriented Testing and AnalysisGönenç Yücel, Can SücüllüSESDYN Research GroupBoğaziçi University, Istanbul

Similarity of dynamic behaviors• Comparing/evaluating model outcomes

• Calibration• Validation• Behavior analysis• Scenario and policy analysis

• Similarity in terms of what?• Pattern characteristics

Problem identification/description◦ Oscillating inventory◦ Exponentially growing resource consumption◦ Declining sales

Model calibration and behavior validation

Sensitivity analysis◦ Numeric sensitivity vs. behavior sensitivity

Policy analysis

Thinking in patterns

Page 10.00 25.00 50.00 75.00 100.00

Time

1: Noname 4

Page 10.00 25.00 50.00 75.00 100.00

Time

1: Noname 4

Page 10.00 25.00 50.00 75.00 100.00

Time

1: Noname 4

4

Patternwise Comparison

23.07.2014 BATS - Can Sücüllü & Gönenç Yücel - ISDC 2014 Delft

5

Behavior Analysis and Testing Software (BATS)

Overview of BATS with External Parties

7

BATS

USER

Excel

View theResults

Control theSoftware

Parse Commands

ReceiveSimulation results

ImportData-series

ExportData-series

8

• Excel• txt, csv, tab• Drawn by the user• Vensim results files (.vdf)• Vensim models through Model Docking Window

Data Importing Methods

Set parameter values

Model Docking Window of BATS

Send simulation results to Workspace in order to further analyze the model output using data analysis features

Send commands; Run

Exporting of simulation results from Vensim models

Overview of Features of BATS

Data ImportingLoad From FileModel Docking WindowDraw

Data PreparationSplitSelectExponential SmoothingMoving Average< Trend >

Data AnalysisClassifyTrendAutocorrelationAutocorrelation TestSpectral DensityAmplitude EstimationCrosscorrelationSummary StatsGraphical Comparison

Model AnalysisHypothesis TesterBehavior Space ClassifierBehavior Class Mapper

Data VisualizationPlot

*

**

**

Graphical Comparison

11

BATS Hypothesis Tester

Extreme conditions are imposed

parameter values

Model is run and output is obtained

ISTS Algorithm is called

IF hypothesized behavior <-3 FAILED

IF hypothesized behavior >-3 PASSED

IF multiple >-3 INCONCLUSIVE

Behavior Space Classifier

13

Settings

BATS Behavior Space Classifier

Parameter 1 Range 1

Parameter 2 Range 2

Outcome Variable

For each cell, a

simulation run is made

Model output is

taken from Vensim ISTS

Algorithm is called

25 likelihoods

are obtained

The highest value is

entered in the

Classifier Grid cell

Param1=1Param2=9

oscct

Behavior Class Mapper

15

Settings

BATS Behavior Class Mapper

For each point, a

simulation run is made

Model output is

taken from Vensim ISTS

Algorithm is called

25 likelihoods

are obtained

The likelihood value for pre-specified class is plotted on the

graph

Pre-specified class is sshgr

(s-shaped-growth)

-3.99Param1=8Param2=16

Identified Usage Modes of BATS• Structure-oriented behavior validity testing

• Extreme condition tests

• Behavior pattern validity testing• Barlas’ Multi-Step procedure and Graphical Inspection

• Sensitivity analysis• Behavior pattern sensitivity with respect to parameter changes

• Model calibration• Parameter calibration based on behavior patterns

• Policy design• Policy parameter specification based on behavior patterns

• Policy analysis• Policy structure specification based on pattern characteristics of model behaviors

18

• Current version of BATS can be downloaded from the following link (Only for Windows for now. Mac version will be coming soon)• Website:

http://www.ie.boun.edu.tr/labs/sesdyn/projects/bats/index.html• Download links:

http://www.ie.boun.edu.tr/labs/sesdyn/projects/bats/bats_v1_11_win.ziphttp://www.ie.boun.edu.tr/labs/sesdyn/projects/bats/bats_v1_11_win.rar

• Please, keep in mind that this is a beta version.

• And if you give it a try, do not forget to send us some feedback that can help us to improve it.

Gönenç Yücel, Assist. Prof. Boğaziçi UniversityIndustrial Engineering Dept.gonenc.yucel@boun.edu.tr

For further questions

Thank you…