bsidessf 2014 fix what matters:why cvss sucks

Fix What Matters: !

Why CVSS Sucks And How To

Do Better

Once Jailbroke an Iphone 3G

Michael Roytman

Proud Owner of Remote Controlled AirplaneRecently a Naive Grad Student

Data Scientist, Risk I/ODoes Not Wake Up Before 11 CST

qualifications:

PART 1: !

YOU SUCK AT YOUR JOB

!

(and don’t even know it yet)

Why Are We Here?

Empirical Failures of CVSSProper Remediation Frameworks (Yeah, they exist)

CVSS SUCKS

Analytical Failures of CVSS

(+Data Driven Alternatives)

Remove the Threat

RemediationAccept the Risk

Repair the Vulnerability

C(ommon) V(ulnerability) S(coring) S(ystem)

“CVSS is designed to rank information system vulnerabilities”

Exploitability/Temporal (Likelihood)

Impact/Environmental (Severity)

The Good: Open, Standardized Scores

“It is a capital mistake to theorize before one has data.

!

!

!

Insensibly, one begins to twist facts to suit theories, instead of

theories to suit facts.”

FAIL: Data FundamentalismDon’t Ignore What a Vulnerability Is: Creation Bias http://blog.risk.io/2013/04/data-fundamentalism/ !

Jerico/Sushidude @ BlackHat https://www.blackhat.com/us-13/briefings.html#Martin

!

Luca Allodi - CVSS DDOS http://disi.unitn.it/~allodi/allodi-12-badgers.pdf

F1: Data FundamentalismSince 2006 Vulnerabilities have declined by 26 percent.” http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf !

!

The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ” http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf

FAIL 2: A Priori Modeling“Following up my previous email, I have tweaked my equation to try to achieve better separation between adjacent scores and to have CCC have a perfect (storm) 10 score...There is probably a way to optimize the problem numerically, but doing trial and error gives one plausible set of parameters...except that the scores of 9.21 and 9.54 are still too close together. I can adjust x.3 and x.7 to get a better separation . . .”

F3: Logical InconsistencyTemporal Scores Hurt Decision Making

Report Confidence is Useless

Base Rate Fallacy

F4: Stochastic Ignorance

Attackers Change Tactics Daily

F4: Stochastic Ignorance

Empirical Failures of CVSSObjective: Remediate the riskiest vulnerabilities

Constraint: Can’t measure impact/priority

Need:

MOAR DATA!!!

Repair the Vulnerability

I Love It When You Call Me Big Data50,000,000 Live Vulnerabilities

1,500,000 Assets

2,000 Organizations

I Love It When You Call Me Big Data

3,000,000 Breaches

Baseline AllthethingsProbability (You Will Be Breached On A Particular Open Vulnerability)?

=(Open Vulnerabilities | Breaches Occurred On Their CVE) /(Total Open Vulnerabilities)

2%

Probability A Vuln Having Property X Has Observed Breaches

RANDOM VULN

CVSS 10

CVSS 9

CVSS 8

CVSS 6

CVSS 7

CVSS 5

CVSS 4

Has Patch

0.000 0.010 0.020 0.030 0.040

PART 2: !

FIX WHAT MATTERS

Proper Framework

Know which vulnerabilities put you most at risk.

Counterterrorism

Known Groups

Surveillance

Threat Intel, Analysts

Targets, Layouts

Past Incidents, Close Calls

Uh, Sports?

Opposing Teams, Specific Players

Gameplay

Scouting Reports, Gametape

Roster, Player Skills

Learning from Losing

InfoSec?

Defend Like You’ve Done It Before

Groups, Motivations

Exploits

Vulnerability Definitions

Asset Topology, Actual Vulns on System

Learning from Breaches

Work With What You’ve Got:

Akamai, Safenet

ExploitDB, Metasploit

NVD, MITRE

Bad Alternatives

Why Don’t I Just Patch The Important Assets?

Good Alternatives

Probability A Vuln Having Property X Has Observed Breaches

Random Vuln

CVSS 10

Exploit DB

Metasploit

MSP+EDB

0.0 0.1 0.2 0.2 0.3

Data Is Everything And Everything Is Data

Data Is Everything And Everything Is Data

Be Better Than The Gap

Data is Everything and Everything is DataSpray and Pray = 2%

CVSS 10 = 4%

Metasploit and Exploit DB = 30%

Holler!www.risk.io@mroytman