![Page 1: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/1.jpg)
Security MetricsWhat Can We Measure?
Zed AbbadiThe Public Company Accounting Oversight Board
![Page 2: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/2.jpg)
What is a “Metric”A metric is a system of related measures enabling quantification of some characteristic. A measure is a dimension compared against a standard.*Security metric is a system of related dimensions (compared against a standard) enabling quantification of the degree of freedom from possibility of suffering damage or loss from malicious attack.*
*QoP’ 06, Oct. 2006
![Page 3: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/3.jpg)
Do We Really Need Metrics?"If you cannot measure it, you cannot
improve it."
Lord Kelvin
"In physical science the first essential step in the direction of learning any subject is to find principles of numerical reckoning and practicable methods for measuring some quality connected with it. I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science, whatever the matter may be." [PLA, vol. 1, "Electrical Units of Measurement", 1883-05-03]
“You cannot manage what you cannot measure"
![Page 4: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/4.jpg)
Drivers For Metrics
Money mattersAsset vs. liability
Governance You claim it is a science?
Do as good as math, physics and astrology
Decision aidHow are we doing with security
![Page 5: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/5.jpg)
Good Metrics. vs. Metrics
QuantitativeObjectiveBased on a formal modelHas a time dimensionUniversally acceptableHas ground truthInexpensiveObtainableRepeatable
![Page 6: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/6.jpg)
Data Collection
Vulnerabilities, exploits and attacksOrganization vs. industry vs. everyone elseDisclosure PoliciesAccuracyStatistical Significance
![Page 7: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/7.jpg)
Attempts at Measuring SecurityTCSEC (Orange book)ITSEC (Europe’s Orange book)CTCPEC (Canada’s Orange book)Common Criteria (everyone’s Orange book)
Framework rather than a list of requirements
SSE-CMMNIST FIPS-140 seriesNIST SP 800-55
![Page 8: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/8.jpg)
Security Metrics Types
Process Security MetricsNetwork Security MetricsSoftware Security MetricsPeople Security MetricsOther
![Page 9: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/9.jpg)
Process Security Metrics
Measure processes and procedures Imply high utility of security policies and processesRelationship between metrics and level of security not clearly definedCompliance/Governance drivenGenerally support better securityActual impact hard to define
![Page 10: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/10.jpg)
Examples
No. of Policy Violations% of systems with formal risk assessments% of system with tested security controls% of weak passwords (non-compliant)No. of identified risks and their severity% of systems with contingency plans
![Page 11: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/11.jpg)
Network Security Metrics
Driven by products (firewalls, IDS etc)Readily availableWidely usedGives sense of controlNice charts and interfacesCan be misleading
![Page 12: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/12.jpg)
Examples
Successful/unsuccessful logonsNo. of incidentsNo. of viruses blockedNo. of patches appliedNo. of spam blockedNo. of virus infectionsNo. of port probesTraffic analysis
![Page 13: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/13.jpg)
Software Security MetricsSoftware measures are
troublesome (LOC, FPs, Complexity etc) “Laws of Physics” are missingMetrics are context sensitive and environment-dependentArchitecture dependentAggregation may not
lead to strength
![Page 14: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/14.jpg)
Examples
Size and complexityDefects/LOCDefects (severity, type) over timeCost per defectAttack surface (# of interfaces)Layers of securityDesign Flaws
![Page 15: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/15.jpg)
People Security Metrics
RelevanceUnique characteristics
Risk perception skewed “optimism Bias”Limited memory and attention spanBehavior modeling is difficult
Awareness training?
![Page 16: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/16.jpg)
Reliability vs. Security
Similar but differentWe care more about reliabilityDifferent adversary model
Reliability models exist, but…Security is a moving target
![Page 17: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/17.jpg)
Most Common Security Metric
Risk- We love this thing!Source for profit
Where is the data? Non monetary consequencesAdversary behavior modelsAccuracy against ground truthMission/system/support models
Dynamic in nature
![Page 18: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/18.jpg)
Future Of Security MetricsConsumers demand better security metricsGovernment involvement is increasedScience evolves to provide better measuresVendors volunteer (forced to) develop universal accurate metricsSome vendors cheat, a watchdog is createdSecurity problems continue, no change in level of risk
![Page 19: Security Metrics What Can We Measure?€¦ · exploits and attacks Organization vs. industry vs. everyone else Disclosure Policies Accuracy Statistical Significance. Attempts at Measuring](https://reader030.vdocuments.net/reader030/viewer/2022040606/5ead465c4f556477e90d9f6b/html5/thumbnails/19.jpg)