june 2, 2005slide 1 analyzing the risks of information security investments with monte- carlo...
Post on 30-Dec-2015
219 Views
Preview:
TRANSCRIPT
June 2, 2005 Slide 1
Analyzing the Risks of Information Security Investments with Monte-Carlo Simulations
WEIS05Workshop on the Economics of Information Security
James R. Conrad, University of Idaho Department of Computer Science conr2286@uidaho.edu
June 2, 2005 Slide 2
Contents
Introduction to the Problem
The Monte-Carlo Solution
Overview of Monte-Carlo Simulations
Example
Analysis and Critique
Conclusions
June 2, 2005 Slide 3
Introduction to the Problem
• An information security investment may need to compete for resources with other business opportunities
• But many information security models rely upon experts’ estimates
• And the experts’ estimates may include significant uncertainty
• How can the analyst communicate an opportunity when so much is uncertain?
June 2, 2005 Slide 4
Monte-Carlo Solution
• Apply the Monte-Carlo technique to simulate and express uncertainty in information security models
• This is not a new model --- this is an enhancement of existing models
• While less common in the Computer Science discipline, many financial decision makers are already familiar with the Monte-Carlo approach
June 2, 2005 Slide 5
Monte-Carlo Simulations
• Specify uncertainty in probability distributions
• Monte-Carlo engine samples distributions
• Engine executes the security model once for each of several thousand iterations
• Monte-Carlo engine captures and collects the result of each iteration
distributions
engine model
results
June 2, 2005 Slide 6
Monte-Carlo Simulations
• Engine simulates uncertainty in the model parameters
• Model continues to operate with discrete values
• Extra complexity largely confined to the Monte-Carlo engine
• Results can be charted as probability distributions
distributions
engine model
results
June 2, 2005 Slide 7
Monte-Carlo Example
• Based upon Longstaff et al’s example appearing in “Are we Forgetting the Risks of Information Technology?” of IEEE Computer, December 2000
• Simulates the benefit/cost ratio of a proposed infosec investment for a financial enterprise
• Modeling parameters are similar to Longstaff’s example with an added complication…
• …The experts don’t agree!
June 2, 2005 Slide 8
Original (pre-Monte-Carlo) Parameters & Model
Intrusion Rate Parametersr1 2 Simulated annual intrusion count w/o investmente 5.00E-01 Effectiveness of investmentr2 =r1*e Annual intrusion count with investment
Other Parametersp1 =r1/365 Daily probability of intrusion w/o investmentp2 =r2/365 Daily probability of intrusion with investmentX $20,000,000,000,000 Asset valuey1 $100,000,000 Cost of software assurance w/o investmenty2 $200,000,000 Cost of software assurance with investmentz1 1.00% Losses w/o investmentz2 0.50% Losses with investment
Model Calculationsd1 =p1*z1 Calc damage w/o investmentd2 =p2*z2 Calc damage with investmentD =y2-y1 Calc cost to provide software assurance with investmentd =d1-d2 Calc percentage of losses prevented by investmentb =d*X-D Calc net benefit of investmentbcr =b/D Calc benefit/cost ratio for investment (bcr=7.22)
intrusionrates
otherparameters
model
benefit/costratio, bcr
June 2, 2005 Slide 9
Uncertainty in the Revised Example
• Consider a case in which the experts don’t agree upon an single value estimate for the annual intrusion rate (fixed at r1=2 events/year in the original problem)
• The hypothetical disagreement stems from uncertainty in anticipated business practices
• Experts do agree there exists a 20% chance that business practices will change in a way that will raise the intrusion rate to 20 events/year and an 80% chance that those practices will remain unchanged
June 2, 2005 Slide 10
Uncertainty in the Revised Parameters• Model variability of optimistic intrusion rate as a Poisson process (for purposes of this example), ro=randpoisson(2)
• Model variability of pessimistic intrusion rate as a Poisson process, rp=randpoisson(20)
• Model uncertainty of anticipated business conditions by choosing the optimistic rate 80% of the time and the pessimistic rate 20% of the time using randdiscrete(0.80,0.20,ro,rp)
• Variability refers to a truly random process
• Uncertainty refers to the experts’ inability to anticipate future business conditions
June 2, 2005 Slide 11
Revised Params & Model
Intrusion Rate Parametersro =randpoisson(2) Optimistic annual intrusion count w/o investmentrp =randpoisson(20) Pessimistic annual intrusion count w/o investmentr1 =randdiscrete(0.8,0.2,ro,rp) 80% Chance of ro. 20% Chance of rp.e 5.00E-01 Effectiveness of investmentr2 =r1*e Annual intrusion count with investment
Other Parametersp1 =r1/365 Daily probability of intrusion w/o investmentp2 =r2/365 Daily probability of intrusion with investmentX $20,000,000,000,000 Asset valuey1 $100,000,000 Cost of software assurance w/o investmenty2 $200,000,000 Cost of software assurance with investmentz1 1.00% Losses w/o investmentz2 0.50% Losses with investment
Model Calculationsd1 =p1*z1 Calc damage w/o investmentd2 =p2*z2 Calc damage with investmentD =y2-y1 Calc cost to provide software assurance with investmentd =d1-d2 Calc percentage of losses prevented by investmentb =d*X-D Calc net benefit of investmentbcr =b/D Calc benefit/cost ratio for investment
intrusionrates
otherparameters
model
benefit/costratio, bcr
June 2, 2005 Slide 12
Simulation of Revised Example
• randpoisson() and randdiscrete() sample the probability distributions in each iteration of the simulation
• The Monte-Carlo engine recalculates the model for each iteration and captures the results (bcr)
• The Monte-Carlo engine charts the captured simulation results (next slide)
June 2, 2005 Slide 13
Simulation Results
June 2, 2005 Slide 14
Why not use a weighted average of r1 and r2?• Why doesn’t the revised model simply compute a weighted average of the two possible intrusion rates?
r1 = randpoisson(2)*0.8+randpoisson(20)*0.2
• The randdiscrete() simulation preserves the bimodal nature of the experts’ disagreement.
• Any attempt to “average away” that uncertainty conceals the truth: The experts don’t agree.
June 2, 2005 Slide 15
Analysis
• The results reflect the experts’ strong preference for the optimistic intrusion rate in which the benefit/cost ratio remains unchanged at 7.22. Risk-tolerant decision makers might manage to this value.
• The mean value lies at 22 between the two modes.
• The results also reflect a second mode at about 81 along with a 10% chance of the benefit/cost ratio exceeding 81. Risk-adverse decision makers might manage to this value to avoid a catastrophe “on their watch.”
June 2, 2005 Slide 16
Critique
• But are real experts willing to provide even more estimates?
• The author’s industry experience with Monte-Carlo models is that many experts are relieved to disclose the uncertainty they know to be in their estimates
• What real experts truly dislike is being held accountable to an expected value they know is merely representative of the possibilities
June 2, 2005 Slide 17
Additional Critique
• Given a tool to express uncertainty as probability distributions, which distributions closely model the empirical evidence?
• How to extend the Monte-Carlo approach to graphical models?
June 2, 2005 Slide 18
Conclusions
• Monte-Carlo techniques offer an approach to simulate uncertainty in expert estimates
• Enables the use of probability distributions for model parameters and forecast results
• The Monte-Carlo engine simulates random variables, allowing a security model to continue to manipulate discrete values with only minimal changes
• May be particularly useful for visualizing the potential of an extreme event, the unlikely possibility of a catastrophic outcome
June 2, 2005 Slide 19
Questions and Optional Slides
June 2, 2005 Slide 20
Why Poisson Distribution?
• The example problem uses a Poisson process to approximate intrusion attempts
• If and/or when the Poisson process usefully reflects empirical intrusion attempts is an open question
• Review: Models the number of events occurring during a specified time interval for a Poisson process
• Review: Continuous opportunity for independent events to occur
• Review: Long-term rate is constant
• Review: Used to model lightening strikes in a storm
June 2, 2005 Slide 21
Correlated Parameters
• “Every iteration of a… model must be a scenario that could physically occur.” -- Vose.
• The parameters must “make sense” to the security model!
• One correlated parameter can usually be expressed as a function (relation) of another.
• Consider r1 and r2 in the example. These are likely related which is why r2 is calculated as a function of r1.
• If the relationship (e) between r1 and r2 is also uncertain, this too can be simulated.
June 2, 2005 Slide 22
Variability and Uncertainty
• Yes, this example lumped (simulated) variability and uncertainty together for simplicity
• Vose (Risk Analysis, 2000) offers an excellent treatment of this subject for those who need to keep them separated
June 2, 2005 Slide 23
Partitioning
• Yes, partitioning is an alternative technique
• The Monte-Carlo technique might be viewed as an automated approach to partitioning
• …and the Monte-Carlo technique avoids the subjective choice of partition boundaries
• …and the Monte-Carlo technique has commercial tool support for systems-level models.
June 2, 2005 Slide 24
Commercial Tools
• Yes, commercial off-the-shelf tools are available
• They are most useful for systems-level security models.
• They are less useful for low-level combinatorics security models
• Search for “monte carlo simulation” and pay particular attention to the “Sponsored Links”
June 2, 2005 Slide 25
Performance
• The author’s industry experience includes Monte-Carlo simulations using “hundreds” of random distribution parameters
• Yes, they required several hours to run…
• In 1997!
• My computer is more than 10X faster today.
• Simulation multiplies model complexity by n, the number of iterations. A simulation of an O(m2) model becomes n*O(m2).
• Opportunities for parallel approaches when n cannot be ignored.
top related