interactive parallel & distributed simulation

24.11.200324.11.2003 Interactive Parallel and Distributed SimulatInteractive Parallel and Distributed Simulation - DSS - Jan Klírion - DSS - Jan Klír 11

Interactive Parallel & Interactive Parallel & Distributed SimulationDistributed Simulation

24.11.200324.11.2003 Interactive Parallel and Distributed SimulatInteractive Parallel and Distributed Simulation - DSS - Jan Klírion - DSS - Jan Klír 22/39/39

OutlineOutline Introduction to Parallel and Introduction to Parallel and

distributed simulation (basic distributed simulation (basic concepts)concepts)

Methods in Interactive PADSMethods in Interactive PADS• OverviewOverview• ApproachesApproaches

Conclusion, current state, future Conclusion, current state, future workwork


Area DefinitionArea Definition

Time-steppedTime-stepped: : time advances by time advances by fixed time fixed time incrementsincrements

Event drivenEvent driven: : time advances time advances occur with irregular occur with irregular incrementsincrements

computersimulation

discretemodels

continuousmodels

eventdriven

time-stepped


Distributed SimulationDistributed Simulation

As fast as possibleAs fast as possible Analytical Analytical

simulationssimulations High Performance High Performance

Computing Computing CommunityCommunity

Conservative and Conservative and Optimistic (Time Optimistic (Time Warp) Warp) synchronizationsynchronization

RealtimeRealtime Virtual Virtual

environments, environments, hardware-in-the-hardware-in-the-looploop

Defense M&S Defense M&S communitycommunity

SIMNET -> DIS -> SIMNET -> DIS -> HLAHLA


Conservative SynchronizationConservative Synchronization

LP 4

LP 3

LP 2

LP 1

Simulation TimeT

events ready to be processed

T+L•L – L – LookaheadLookahead – Any message sent by a LP must have – Any message sent by a LP must have timestamp at least T + Ltimestamp at least T + L•Necessary to allow concurrent processing of events with different time stamps•Relies on model props, complicates the model developmentRelies on model props, complicates the model development

events not ready to be processed


Optimistic SynchronizationOptimistic Synchronization

LP 4

LP 3

LP 2

LP 1

Simulation TimeT

processed events

unprocessed events

rollback

•LP processes events without checking if an event in its past LP processes events without checking if an event in its past could later arrivecould later arrive•If an event arrives in the LPs past (If an event arrives in the LPs past (straggler messagestraggler message), the ), the computation is rolled backcomputation is rolled back


Optimistic SynchronizationOptimistic Synchronizationlocal control mechanismlocal control mechanism

Input Queue(event list)

straggler message arrives in the past, causing rollback

12 21 35

processed eventunprocessed eventsaved stateanti-message

State Queue

1212Output Queue(anti-messages)

•LP processes events in time order like sequential simulatorLP processes events in time order like sequential simulator•The state before processed events must be saved, the The state before processed events must be saved, the messages sent to other LPs must be also savedmessages sent to other LPs must be also saved•When a message arrives in the past (straggler message), the When a message arrives in the past (straggler message), the rollback is initiatedrollback is initiated


Optimistic SynchronizationOptimistic Synchronizationconcepts and notionsconcepts and notions

State savingState saving – various algorithms – various algorithms (Copy SS, Periodic SS, Incremental (Copy SS, Periodic SS, Incremental SS, …)SS, …)

Anti messagesAnti messages• In case of rollback are sent to In case of rollback are sent to

destination LPsdestination LPs• If LP gets anti message, the If LP gets anti message, the

corresponding waiting event is removed corresponding waiting event is removed from the queue (from the queue (annihilationannihilation))


Optimistic SynchronizationOptimistic Synchronizationconcepts and notionsconcepts and notions

Global virtual timeGlobal virtual time• Minimum timestamp of any unprocessed Minimum timestamp of any unprocessed

message or anti-message in the whole systemmessage or anti-message in the whole system• Used to reclaim memory used by SS and Used to reclaim memory used by SS and

output queues (output queues (fossil collectionfossil collection))• Also serves to commit blocking I/O operationsAlso serves to commit blocking I/O operations• Synchronous X Asynchronous GVT computationSynchronous X Asynchronous GVT computation• Wide spectrum of GVT algorithms exist today, Wide spectrum of GVT algorithms exist today,

the selection depends on application. The the selection depends on application. The algorithms are non-trivial in distributed algorithms are non-trivial in distributed environments.environments.


PADS ConclusionPADS Conclusion Conservative or Optimistic approachConservative or Optimistic approach Optimistic much more used than Optimistic much more used than

conservativeconservative

Next part: Interactive methods based Next part: Interactive methods based on PADSon PADS


Interactive simulationInteractive simulation

Methods in Interactive PADSMethods in Interactive PADS


Interactive simulation - outlineInteractive simulation - outline Introduction – applications & Introduction – applications &

requirementsrequirements Description of Interactive PADS Description of Interactive PADS

approachesapproaches• Multiple path explorationMultiple path exploration• Input / outputInput / output• Variable resolution modelingVariable resolution modeling

Summary, open issues, current stateSummary, open issues, current state


Applications of Interactive Applications of Interactive SimulationSimulation

Decision support - (Air) traffic control, Decision support - (Air) traffic control, planning, …planning, …

Interactive system analysis – Interactive system analysis – Networks, transportation systems, Networks, transportation systems, ecological systems, logistics, ecological systems, logistics, battlefield, …battlefield, …

Virtual environments – Training Virtual environments – Training simulators, medicine (virtual simulators, medicine (virtual surgery), … surgery), …


RequirementsRequirementswhat should an interactive simulator supportwhat should an interactive simulator support Exploration of different simulation Exploration of different simulation

futuresfutures• Rewind to some point and continue with Rewind to some point and continue with

different inputdifferent input• Execute multiple simulations Execute multiple simulations

concurrentlyconcurrently Fast input and outputFast input and output Variable resolution models Variable resolution models

• Interactive detail level changingInteractive detail level changing


Multiple Path ExplorationMultiple Path Exploration Uses – explore different simulation Uses – explore different simulation

paths based on different input paths based on different input parametersparameters

ApproachesApproaches• Multiple sequential simulations with Multiple sequential simulations with

different parametersdifferent parameters• Interactive simulation steering with Interactive simulation steering with

rollback & reexecutionrollback & reexecution• Cloning (Virtual processes)Cloning (Virtual processes)


Rollback & ReexecutionRollback & Reexecution Input events called ‘steering’ events – Input events called ‘steering’ events –

modify the simulation path according the modify the simulation path according the inputinput

Special ‘reexecution’ events allow to Special ‘reexecution’ events allow to rewind to the pastrewind to the past• Scheduled to the pastScheduled to the past• The simulation can take a new path by The simulation can take a new path by

inputting new data via steering eventsinputting new data via steering events• Suitable for Time Warp simulators that utilize Suitable for Time Warp simulators that utilize

the rollback as a part of the synchronizationthe rollback as a part of the synchronization


Rollback & ReexecutionRollback & Reexecution ProblemsProblems

• There must be a saved state prior the There must be a saved state prior the time of the reexecution eventtime of the reexecution event

• The available rollback length is limited - The available rollback length is limited - due to fossil collection in Time Warp due to fossil collection in Time Warp simulators the old saved states are simulators the old saved states are reclaimed as the GVT passesreclaimed as the GVT passes

• Some state saving schemes were Some state saving schemes were developed but the rollback is still limiteddeveloped but the rollback is still limited


Cloning (Virtual Processes)Cloning (Virtual Processes) Multiple simulation futures are computed Multiple simulation futures are computed

in parallelin parallel The futures share the computation until The futures share the computation until

they branch with different parametersthey branch with different parameters Uses the Virtual processes paradigmUses the Virtual processes paradigm

• All the computed simulations contain virtual All the computed simulations contain virtual processesprocesses

• All corresponding virtual processes are mapped All corresponding virtual processes are mapped to a single physical process, until the branchto a single physical process, until the branch


Virtual ProcessesVirtual Processes

2 virtual simulations, cloned on process A2 virtual simulations, cloned on process A

A2 B2C22

VA1 VB1 VC1

VA2 VB2 VC2

PA2 PB1 PC1PA1

A1 B1C11

Virtual simulation planesVirtual simulation planes Mapping to phys. processes


Virtual ProcessesVirtual Processes CommunicationCommunication

• B -> C: message to PB -> C: message to PCC and thus to V and thus to VC1C1 and Vand VC2C2

• B -> A: message to PB -> A: message to PA1A1 (V (VA1A1) and copied ) and copied to Pto PA2A2 (V (VA2A2))

• A -> B (A1 to B1): PA -> B (A1 to B1): PB2B2 created and created and message sent to Pmessage sent to PB1B1 (V (VB1B1))


CloningCloning 3 phases: Both simulations identical; 3 phases: Both simulations identical;

Incremental cloning; Both Incremental cloning; Both simulations fully clonedsimulations fully cloned

Cloning rate depends on the amount Cloning rate depends on the amount of communication of communication


Multiple Path ExplorationMultiple Path Explorationsummarysummary

Reexecution more interactive, cloning Reexecution more interactive, cloning faster (offers more parallelism)faster (offers more parallelism)

Reexecution - limited rollback length; Reexecution - limited rollback length; cloning – all paths must be known before cloning – all paths must be known before the branchthe branch

Open issuesOpen issues• Greater rollback length, merging convergent Greater rollback length, merging convergent

sim. paths, create new branch point in the past sim. paths, create new branch point in the past and keep the current path, …and keep the current path, …


Input / OutputInput / Output Purpose: To provide accurate and Purpose: To provide accurate and

fast I/O to distributed simulatorsfast I/O to distributed simulators Several classes of I/O events:Several classes of I/O events:

• Blocking X non-blockingBlocking X non-blocking• Various types of access to the Various types of access to the

simulation statesimulation state Speed-up: optimistic outputSpeed-up: optimistic output


I/O EventsI/O Events 3 classes of output events:3 classes of output events:

• Sampling: provide a subset of state information Sampling: provide a subset of state information in regular intervalsin regular intervals

• Notification: occur in predefined transient Notification: occur in predefined transient conditioncondition

• Queries: require access to arbitrary state in Queries: require access to arbitrary state in previously unknown timepreviously unknown time

Input:Input:• Events injected by external entitiesEvents injected by external entities• Blocking requests for input scheduled by the Blocking requests for input scheduled by the

simulatorsimulator


Optimistic OutputOptimistic Output Output is often visually rich and Output is often visually rich and

computation intensivecomputation intensive Speedup:Speedup:

• Output is non-blocking eventOutput is non-blocking event• It’s optimistically precomputed and It’s optimistically precomputed and

thrown away in case of rollbackthrown away in case of rollback• Good for faster-than-realtime simulators Good for faster-than-realtime simulators

and computationally intensive outputand computationally intensive output


Input / OutputInput / Outputconclusionconclusion

Open issuesOpen issues• Queries in distributed simulation, over a Queries in distributed simulation, over a

wide time rangewide time range• Optimistic output:Optimistic output:

Limiting the optimism to save resourcesLimiting the optimism to save resources Relationship between optimistic output and Relationship between optimistic output and

the probability of inputthe probability of input


Variable Resolution ModelingVariable Resolution Modeling Why use VRM?Why use VRM?

• Speed-up – execute the focused portion Speed-up – execute the focused portion at full detail and the other at a lower d.at full detail and the other at a lower d.

• Zooming – overview the whole and focus Zooming – overview the whole and focus on details to better comprehend the on details to better comprehend the simulated phenomenonsimulated phenomenon

• Whole-greater-than-sum-of-parts Whole-greater-than-sum-of-parts relationshiprelationship

• Combine different legacy models with Combine different legacy models with diff. detail -> heterogeneous sim. diff. detail -> heterogeneous sim.


Variable Resolution ModelingVariable Resolution Modelingapproachesapproaches

Selective viewingSelective viewing• Simulate only the most detailed model Simulate only the most detailed model • Calculate the other views from the most Calculate the other views from the most

detailed modeldetailed model Aggregation/DisaggregationAggregation/Disaggregation

• One level is executed at one timeOne level is executed at one time• When needed, low resolution entity is When needed, low resolution entity is

replaced with a set of high res. entities replaced with a set of high res. entities (disaggregation) and vice versa(disaggregation) and vice versa


Variable Resolution ModelingVariable Resolution Modelingapproachesapproaches

Concurrent representationConcurrent representation• All the entities are in the model All the entities are in the model

concurrentlyconcurrently• Only the selected detail level is Only the selected detail level is

executedexecuted• Other detail levels maintain consistency Other detail levels maintain consistency

by transforming the results from the by transforming the results from the running entitiesrunning entities

Existing methods use mix of these Existing methods use mix of these approachesapproaches


Multi Representation EntitiesMulti Representation Entities Single MRE has Single MRE has

multiple detail levelsmultiple detail levels Communication with Communication with

other entities at other entities at appropriate levelappropriate level

Consistency Consistency maintained by C.E. maintained by C.E. which has a set of which has a set of mapping functions to mapping functions to convert the state convert the state valuesvalues

MRE E1

A

B1 B1 B1

Consistencyenforcer

E2

E3


Hooking TransactionsHooking Transactions Model designed at multiple detail Model designed at multiple detail

levelslevels At the run time, the channel leading At the run time, the channel leading

to the LRE can be redirected to a to the LRE can be redirected to a HRE to increase the detail (zoom in)HRE to increase the detail (zoom in)

The disaggregated LRE can continue The disaggregated LRE can continue or is stripped completelyor is stripped completely

Simulation language primitives exist Simulation language primitives exist to be used by modelersto be used by modelers


Hooking TransactionsHooking Transactions

LRE 1 LRE 2 LRE 3

HRE 1


Hooking TransactionsHooking Transactions Consistency: Attribute ghostingConsistency: Attribute ghosting

• Attributes of the striped entity are Attributes of the striped entity are calculated from the newly hooked entitycalculated from the newly hooked entity

Mapping consistencyMapping consistency• The state of the HRE on disaggregation The state of the HRE on disaggregation

must be properly initialized from the must be properly initialized from the LRE’s state (and vice versa) LRE’s state (and vice versa)


Aggregation / DisaggregationAggregation / Disaggregation Used often in the HLA areaUsed often in the HLA area Low resolution entities (aircraft patrols) Low resolution entities (aircraft patrols)

automatically disaggregate to a set of high automatically disaggregate to a set of high resolution entities (single aircrafts) when resolution entities (single aircrafts) when approaching other HREs (ground radars)approaching other HREs (ground radars)

HLA is enriched with a set of functions that HLA is enriched with a set of functions that enable the registration of HREs and LREs, enable the registration of HREs and LREs, together with transformation functions and together with transformation functions and disaggregation conditionsdisaggregation conditions


Aggregation / DisaggregationAggregation / Disaggregation

LRE 1 LRE 3LRE 2HRE2

HRE1

HRE3


Other ApproachesOther Approaches Optimization based MRMOptimization based MRM

• Models at different resolution level Models at different resolution level provide a set of inputsprovide a set of inputs

• Mathematical function describing Mathematical function describing differences between the outputs is differences between the outputs is defineddefined

• Using mathematical analysis the inputs Using mathematical analysis the inputs are calculated so the differences are calculated so the differences between the outputs are minimizedbetween the outputs are minimized


Variable Resolution ModelingVariable Resolution Modelingsummarysummary

VRM – rich area with variety of possible VRM – rich area with variety of possible applicationsapplications

Hooking Hooking • Highly interactiveHighly interactive• Currently only for sequential simulationCurrently only for sequential simulation

MREMRE• Solid designSolid design• Not widely used (high complexity and demands on Not widely used (high complexity and demands on

modelers)modelers) Aggregation/DisaggregationAggregation/Disaggregation

• Oldest approach, simpleOldest approach, simple• Good for real-time simulation area (HLA) Good for real-time simulation area (HLA)


Current StateCurrent Stateand future workand future work

Prof. Prof. ŠafaříkŠafařík - 2 diploma theses to - 2 diploma theses to incorporate optimistic and incorporate optimistic and conservative synchronization to J-Sim conservative synchronization to J-Sim ((C-Sim) C-Sim)

Base on this and incorporate the Base on this and incorporate the interactivity as shown in this interactivity as shown in this presentation, trying to overcome the presentation, trying to overcome the shortcomings of the presented shortcomings of the presented approachesapproaches


The EndThe EndQuestions ???Questions ???

Some pictures were reproduced from the Parallel and Distributed Simulation Some pictures were reproduced from the Parallel and Distributed Simulation course by M. Hybinette, PADS presentations by R. Fujimoto, and from various course by M. Hybinette, PADS presentations by R. Fujimoto, and from various papers cited in the papers cited in the Methods in Interactive Parallel and Distributed Simulation Methods in Interactive Parallel and Distributed Simulation

paper paper

interactive parallel & distributed simulation

Documents