Building Simulation Model

• In this lecture, we are interested in whether a simulation model is accurate representation of the real system.

• We are interested in building a valid, credible, and verified simulation model.

• Verification: is to determine the conceptual simulation model has been correctly translated into a computer program.

• Validation: is the process of determining whether a simulation model is an accurate representation of the system.

Validation Versus Verification

Real world(News)

English Report

Arabic translator

Arabic News

Verification is to check whether the translation is correct

Validation to check whether the news is correct or not

Credibility

• A simulation model has a credibility if the manager and other key project personnel accept them as correct.

Guidelines for determining the level of model detail

• Define the specific issues to be investigated and the performance measures that will be used for evaluation.

• The entity moving through the simulation model need not be the entity moving in the actual system. – Example: it might be better to model a box of

parts in a manufacturing model as an entity rather than modeling each part as an entity.

• Use Subject Mater Experts (SMEs) and sensitivity analysis to help determine the level of model detail. – SMEs are People who are familiar with the

system are asked what components are likely to be added to the model

– Sensitivity analysis is used to determine what factors that have a great impact on the performance measure.

• Do not include excessive amount of model detail. Start by a moderate detail and add more details as needed.

Guidelines cont.

• Do not have more detail in the model more than is necessary to address the issues of interest.

• The model detail should be consistent with the type of data available.

• Time and Money constraints are a major factor.

• If the number of factors is large. Perform analytical analysis to determine the important factors. – There are some commercial software packages

available for this purpose.

Verification of Simulation Computer Program

Some of these techniques are valid for any computing program.

Technique1: Divide the program into subprograms and then debug these subprograms individually.– Sometimes computer simulation programs may

go to 10,000 statements– Start by a moderate simulation model then

gradually increase complexity as needed.

Technique 2: Make a structured walk-through of the program by other persons as one may not able to criticize himself.

Verification cont.• Technique 3: Run the simulation model under

several settings of the input parameters. Try it first for known system performance.

• Technique 4: One of the powerful techniques is to trace the state of the simulated system and compare with a hand simulation.

• Technique5: Run the model under simplified assumptions for which its true characteristics are known.

– Example: replace the M/E2/1 queuing model by M/M/1 … etc.

Verification cont

• Technique 6: If it is possible, use animation to trace entities in the model.

• Technique 7: Compute a sample mean and sample variance for each simulation input probability and compare with the desired known mean and variance.

• Technique 8: Use commercial simulation packages carefully to reduce the amount of programming time.

Techniques to increase model validity and credibility

Technique 1: Collect high quality information and data on the system. Make use of the existing information including:

A. Conversation with Subject Mater Experts (SME’s)

B. Observations of the system: Existing Theory Use previous data that have been introduced from

similar simulation studies. Use the experience and intuition of the modelers to

hypothesize how certain components of a complex system operation.

A. Conversation with Subject Mater Experts (SME’s)

Example: for a communication network, relevant people may include: • End users

• Network designers

• Technology experts

• System administrators

• Application architects

• Maintenance personnel

• Managers

B. Observations of the system:

If the system exists or a similar system exists then collect data from this system. It is important to follow these two principles.

• The modelers need to make sure that the data requirements are specified precisely to the people who provide the data.

• The modeler should understand the process of produced data.

The following errors may occur during the collection of data:

1. Data are not representatives of what we need Example: the data collected in a military

field test may not represent the actual combat field.

2. Data are not of the appropriate type or format Example: the machine downtime in a

manufacture should read only the time when the work is on not the wall clock.

Data may contain measurements, recording or rounding errors.

Data may be biased because of self interest

Data may have inconsistent units. Km, miles, …etc.

• Technique 2: Interact with the manager on a regular basis.

• Technique 3: Maintain an assumptions of document and perform a structural walk-through:– Record all assumptions that have been taken

in the simulation model including• Project goals• Detailed description of each subsystem• What simplifying assumptions were made and why• Summary of data such as mean, histogram, … etc.• Sources of important and controversial information.

– Use sensitivity analysis to determine model factors that have a significant impact on the desired measures of performance.

– Examples include• The value of a parameter• The choice of a distribution that the entity

moving in the system• The level of detail of subsystems • What data are the most crucial to collect

Technique 4: Validate components of the

model by using quantitative techniques.

– Compare the output with a similar existing system to check whether the output of the simulation model reflects the real life performance or not.

Technique 5: Validate output from the overall simulation model.