challenges in modeling
DESCRIPTION
Challenges in Modeling. COMPLEXITIES OF MODELS. Large State Space (e.g. Bedrock, Wireless handoff) Model construction problem Model solution problem Model Stiffness. Fast and slow rates acting together Failure And Recovery/Repair (HSP Markov model in Bedrock) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/1.jpg)
Challenges in Modeling
![Page 2: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/2.jpg)
COMPLEXITIES OF MODELS
• Large State Space (e.g. Bedrock, Wireless handoff)– Model construction problem
– Model solution problem
• Model Stiffness.
Fast and slow rates acting together– Failure And Recovery/Repair (HSP Markov model in Bedrock)
– Performance and failure (Wireless handoff)
![Page 3: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/3.jpg)
COMPLEXITIES OF MODELS
(Continued)• Modeling Non-Exponential Distributions
(e.g. N+1 problem)
• Believability/Understandability/Usability
• What about software?
![Page 4: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/4.jpg)
Potential Solutions
• Largeness
– Largeness Tolerance
– Largeness Avoidance
![Page 5: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/5.jpg)
LARGENESS TOLERANCE
• Automated Model Construction
– Loops in the specification of CTMC (SHARPE)
– Stochastic Petri nets (SPNP, SHARPE)
– High level languages (SAVE, QNAP, ASSIST, SDM)
– Fault-Tree + Recovery Info (HARP)
– Object-Oriented Approaches (TANGRAM)
![Page 6: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/6.jpg)
LARGENESS TOLERANCE (Continued)
• Efficient numerical solution techniques
– Sparse Storage
– Accurate and Efficient Solution Methods
We have Generated and Solved Models
with 1,000,000 states (has gone up
considerably recently)
Steady-State : NEAR-Optimal SOR
Transient: Modified Jensen's method
![Page 7: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/7.jpg)
MODEL SPECIFICATION LANGUAGES
• Different languages can be used to specify a
single model type:
SAVE, QNAP, SPNP all appear very different;
underlying model type is Markov
• Same language can be used to specify different
model types:SPNP input language used for
Markovian SPN analytic numeric solution or
non-Markovian SPN simulation solution
![Page 8: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/8.jpg)
MODEL SPECIFICATION LANGUAGES (Continued)
• Languages can be domain specific:
– Reliability: HARP, SDM
– Availability: SAVE
– Performance: RESQ, QNAP
• Language can be domain independent:
– SHARPE, SPNP
![Page 9: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/9.jpg)
LARGENESS AVOIDANCE
• Non-State-Space methods
– Reliability block diagrams
– Fault-trees
– Product-Form Queuing Networks
• Approximate solutions
– State Truncation
SAVE, SPNP (Kantz and Trivedi: PNPM91)
![Page 10: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/10.jpg)
Case Study: JPL REE System Availability Modeling in Spacecraft Architecture
![Page 11: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/11.jpg)
LARGENESS AVOIDANCE (Cont.)
• Stochastic Petri Nets (State-space-based modeling)
• State truncation by introducing guard function
Guard g is defined as
If (mark(“…_dn”) >= K)
return (0);
else
return (1);
![Page 12: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/12.jpg)
SPN MODELING
![Page 13: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/13.jpg)
AVAILABILITY MEASURES
![Page 14: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/14.jpg)
LARGENESS AVOIDANCE (Continued)
• Approximate solutions
– Hierarchical Decomposition
and Fixed-Point Iteration among submodels:
• Heidelberger and Trivedi; IEEE-TC,1983
(Queueing Models)
• Ciardo and Trivedi; PNPM91 (SPN Models)
• Tomek and Trivedi (Availability Models)
• Lanus, Liang & Trivedi: (Bedrock)
• Wireless handoff work: Ma, Han & Trivedi
![Page 15: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/15.jpg)
LARGENESS AVOIDANCE (Continued)
• Approximate solutions
– Performability:
Multiprocessor example
– Fluid Approximation:
Mitra; Kulkarni; Ciardo; Nicol, and Trivedi;
FSPN
![Page 16: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/16.jpg)
Difficulties in Modeling Using MRMs
• Stiffness
Causes numerical difficulties in solution– Stiffness Tolerance
Develop stiffness tolerant numerical
solution methods – Stiffness Avoidance
Avoid generating stiff models through
decomposition
![Page 17: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/17.jpg)
Potential Solutions (Continued)
• Stiffness
– Stiffness Tolerance
– Stiffness Avoidance
• Modeling Non-Exponential Distributions
– Stage-type expansion, MRGP, NHCTMC, DES
![Page 18: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/18.jpg)
STIFFNESS TOLERANCE
• Automatic Detection of Stiffness (HARP)
• Special Stable ODE Solver
Reibman and Trivedi (TR-BDF2)
Computers and Operations Research, 1988.
Malhotra and Trivedi (Pade, Implicit RK)
![Page 19: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/19.jpg)
STIFFNESS TOLERANCE (Continued)
• Uniformization for Stiff Markov Chains
Muppala and Trivedi
We can solve models with rate ratios of 108 or
higher
Implemented in SHARPE & SPNP
![Page 20: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/20.jpg)
STIFFNESS AVOIDANCE
• Model-level decomposition
– Hierarchical Composition (SHARPE)
Composition of Submodel solutions without
generating a single one-level overall model
(Bedrock example)
– Fixed-Point Iteration (Wireless handoff example)
![Page 21: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/21.jpg)
STIFFNESS AVOIDANCE (Continued)
• Importance Sampling (simulation)
– Lewis, Goyal, Heidelberger, Shahbuddin, Geist, Nicola
– Can also apply to analytic-numeric methods
(Heidelberger, Muppala, and Trivedi; Performance 93)
• Importance splitting (Simulation)
– Tuffin and Trivedi; Tools’ 00
![Page 22: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/22.jpg)
Non-Exponential Behavior
• Non state space models: Fault Trees, Reliability
Graphs, RBDs; no problem
![Page 23: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/23.jpg)
Non-Exponential Behaviorin State Space Models
![Page 24: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/24.jpg)
![Page 25: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/25.jpg)
NON-EXPONENTIAL DISTRIBUTIONS
• Phase-Type Expansions
– N+1 example
• Non-Homogeneous Markov Chains
CARE III, HARP
Soft Rel model with imperfect repairs solved
using SHARPE
![Page 26: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/26.jpg)
NON-EXPONENTIAL DISTRIBUTIONS (Continued)
• Semi-Markov Chains N+1 example• Markov Regenerative Processes: Choi, Logothetis, Kulkarni, Trivedi• DSPN and MRSPN:
Choi, Kulkarni, Trivedi• Discrete-Event Simulation Now in SPNP (FSPN and Non-Markovian SPN
Simulation), RESQ, QNAP, Bones, SES workbench
![Page 27: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/27.jpg)
CASE STUDY: AT & T
• GSHARPE:– A Preprocessor to SHARPE developed at Bell Labs by
a Duke Student.– User can specify Weibull Failure times and lognormal
and other repair time distributions.– GSHARPE fits these to phase type distributions and
produces a Markov model that is generated for processing by SHARPE
![Page 28: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/28.jpg)
Potential Solutions (Continued)
• Believability/Understandability/Usability
– GUI, many practical examples, short-courses, tools, Boeing SDM project
• Incorporation in the design process
– VHDL Availability Model,
– C Program Perf. Model
– Ada Program SPN Perf. Model (SPC)
• Connection between measurements & models
![Page 29: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/29.jpg)
BELIEVABILITYUNDERSTANDABILITY
• Integration of Measurements and Models
– Measurements Provide Parameters to Models
– Models Provide Guidelines For Measurements
– Models Validated Against Measurements
• Integration of Different Modeling Tools
– Boeing SDM project
![Page 30: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/30.jpg)
BELIEVABILITY/UNDERSTANDABILITY
(Continued)
• Many Case-Studies of Validations Needed
– Vaxcluster Availability Model: Wein & Sathaye
– Hsueh, Iyer and Trivedi; IEEE-TC, Apr. 1988
– Lucent Validation of ESS; Veena Mendiratta
• Technology Transfer
– Short courses
– Development and Dissemination of Tools
(SHARPE, SPNP)
![Page 31: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/31.jpg)
BELIEVABILITY/UNDERSTANDABILITY
(Continued)
• Application of the Techniques and Tools
– Motorola
– Cisco
– 3Com
– HP
– Sun
![Page 32: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/32.jpg)
CASE STUDY: BOEING
• An Integrated Reliability Environment
• A working prototype
• Developed a high-level modeling language (SDM)
• Designed and implemented an intelligent interpreter
![Page 33: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/33.jpg)
CASE STUDY: BOEING (Continued)
• Interpreter determines which solution method is applicable
• Translator translates the SDM input file into an input file of any of the engines down below
• Five different modeling engines are integrated:
– CAFTA, SETS, EHARP, SHARPE and SPNP.
![Page 34: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/34.jpg)
MODELING AND MEASUREMENTS: INTERFACES
• Measurements supply Input Parameters to Models
(Model Calibration or Parameterization)
Confidence Intervals should be obtained
Boeing, Draper, Union Switch projects
• Model Sensitivity Analysis can suggest which Parameters to Measure More Accurately: Blake, Reibman and Trivedi: SIGMETRICS 1988; Fricks and Trivedi: 1997
![Page 35: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/35.jpg)
MODEL CALIBRATION
What is ?
• Fault Model for Each Component– Design,Manufacturing: Heisenbugs, Bohrbugs
– Operational: Permanent, Intermittent,Transient
– Human
• Fault Arrival Processes (PP,Weibull,NHPP)
• Failure Rates (Sources:MIL-STD)
![Page 36: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/36.jpg)
MODEL CALIBRATION (Continued)
What is c ?
• Field Data
• Fault/Error Injection (FIAT,MESSALINE)
• Analytic Coverage Model
What is ?
• Maintenance Model Corrective; dispatch , travel, repair time, dead on arrival, imperfect repair
Preventive
![Page 37: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/37.jpg)
MODEL CALIBRATION (Continued)
What is r ?• Binary: Up & Down
• Capacity-Oriented:
Number of Operational Resources in Each State• Performance-Oriented:
Evaluate Perf. in Each Degraded Level of Syst. Config.
1. Measurements
2. Simulation Model
3. Analytic Model -- SHARPE, SPNP
![Page 38: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/38.jpg)
– Validation: Does the conceptual model faithfully
reflect the behavior of the system?
– Verification: Has the conceptual model been
correctly implemented?
VALIDATION&VERIFICATION
![Page 39: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/39.jpg)
MODEL VALIDATION (Continued)
• Three step process outlined by Naylor and Finger– Face validation: Discussion with the experts
– Input-Output validation: Compare results obtained from model with those from measurements
– Validation of model assumptions: Either prove that the assumptions are correct or do statistical testing
![Page 40: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/40.jpg)
MODEL ASSUMPTIONS/ERRORS
• Errors in Model Structure
– Missing or Extra Arcs
– Missing or Extra States
– Use Face Validation to avoid these errors.
• Errors Due to Non-Independence
• Distributional Errors
• Parametric Errors
![Page 41: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/41.jpg)
MODEL ASSUMPTIONS/ ERRORS(Continued)
• Errors Due Approximations
– Decomposition/Aggregation/Iteration
– State Truncation
• Numerical Solution Errors
– Discretization Errors
– Round-Off Errors
![Page 42: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/42.jpg)
Model Verification
• Programming Errors
• Approximation errors: Tight bounds due to
approximations are desirable
• Numerical: Errors in numerical algorithms
should be bounded
![Page 43: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/43.jpg)
What about software?
• Testing phase– Software reliability estimation
• Black-box based approach
• Architecture-based approach
• Operational phase– Fault tolerance coverage (c in Markov model)
– Countering software aging
• Symptom-based fault management
![Page 44: Challenges in Modeling](https://reader036.vdocuments.net/reader036/viewer/2022062500/5681587e550346895dc5e02d/html5/thumbnails/44.jpg)
Conclusions:• Availability evaluation is very important in
characterizing systems
• Evaluation can be performed either through measurements, simulation or analytical modeling
• Model verification and validation should form an integral part of the modeling process