meaningful security metrics
DESCRIPTION
Dr. John D. Johnson delivers a presentation on security metrics to the Cyber Security Strategies Summit, Washington, DC, May 2011.TRANSCRIPT
John Johnson, PhD, CISSP Sr. Security Program Manager
} Setting the Stage ◦ Defining Terms and Setting Expectations
} Exploring the Problem ◦ A historical perspective ◦ Reaching out to other disciplines
} Gathering Data ◦ Both qualitative and quantitative data sources
} Models for Making Sense of the Data ◦ Making sense of large data sets ◦ Asking the right question, getting the right answer
} Deriving Value and Driving Improvements ◦ Examples of Operational, Strategic & Business Metrics ◦ Building Your Security Metrics Program
} Audience Discussion
Setting the Stage
} Performance metrics: measure how well an organization performs and drive process improvements and demonstrate value-add
} What are we actually measuring? ◦ Are we more secure today than yesterday? ◦ How do we compare to our peers?
} We are often stuck with what our tools provide ◦ The Cycle: Detect Report Prioritize Remediate
} Security metrics be made meaningful; this means they
should provide value to stakeholders ◦ We need to learn to ask the right questions, if our results are
going to be meaningful ◦ The best metrics are SMART: Specific, Measurable, Attainable,
Repeatable & Time-Dependent
} Coming up with meaningful security metrics is an inherently difficult problem, but we can: ◦ Draw upon examples from other disciplines
◦ Realize that there are many ways to tell a story so that it is
meaningful to stakeholders, focus on impact and outcomes ◦ Recognize what we do not know and cannot measure, and still do
our best to account for these threats ◦ Find ways to quantify security activities
◦ Gather more data from different sources
◦ Work together to define standard frameworks for analyzing data
and drawing conclusions
From: Stars in the Water by Lesley DuTemple, Illustration by Jack Oyler
} born 1546 / died 1601
} A Danish nobleman, known for hisprecise and comprehensive planetary and astronomical observations.
} “I've studied all available charts of the planets and stars and none of them match the others. There are just as many measurements and methods as there are astronomers and all of them disagree. What's needed is a long term project with the aim of mapping the heavens conducted from a single location over a period of several years.” – does this sound familiar?
} Also known for wearing a golden nose, after losing his real nose in a duel, and for dying due to bladder complications after throwing a really good party, but refusing to leave to use the bathroom.
“I've studied all available charts of the planets and stars and none of them match the others. There are just as many measurements “I've studied all available charts of the planets and stars and none of them match the others. There are just as many measurements “I've studied all available charts of the planets and stars and none
} born 1571/ died 1630
} Worked with Tycho Brahe, until Brahe’s untimely death.
} Kepler had access to volumes of quantitative data on the planets and he developed a scientific theory (a model of planetary motion).
} Kepler’s Laws provided a foundation for Newton’s Law of Gravity and transformed forever the way we see the night sky.
} Born 1564 / died 1642
} He built the first telescopefor observing the heavens.(early adopter)
} He drew what he observed, but had no model, so his results turned out to be nonsense (later corrected by Huygens).
} Nevertheless, he did not give up and his many observations (phases of Venus, Jupiter’s moons) vindicated the heliocentric model and led to the birth of modern science.
Grainy Data, No Model Good Data, Good Model
Galileo did the best he could with what he had…
As the tools and models improve, knowledge improves
Coarse Data, Confusing
Graphic showing oil prices (© Pedro Monteiro of the What Type blog):
Jon Peltier added better and finer-grained data, and asked more meaningful questions to come up with graphics that were easier to interpret. Good data visualization should be easy to read. Which of these would you take to your management?
http://peltiertech.com/WordPress/replacement-for-oil-price-radial-chart/
This is a very simplistic graph, showing the kind of data you could get from AV tools in 2006. Reporting was limited and data was coarse, but it was combined with pertinent facts to explain fluctuations.
} Ordinal numbers are used to rank, or create stop light graphics ◦ Red, Yellow, Green ◦ High, Medium, Low ◦ No units or scales
} Cardinal numbers have units and can better
be interpreted to add value ◦ More precise ◦ Can compare across business units, companies… ◦ Can be used to establish a baseline
} Motivators ◦ Regulations - Compliance ◦ Audits (both internal and external) ◦ Money (security is rarely a profit center) ◦ Responding to new threats ◦ Enabling new technology and business processes
} Qualitative and Quantitative Data ◦ Traditionally, ordinal data and storytelling has been
“good enough” ◦ Quantitative data can be automated & more consistent ◦ New threats and shorter exploit times means detection
and response needs to be quicker ◦ A mix of both types of data and more standard models
are needed to respond
} Good standard risk assessment frameworks exist to address some of our concerns ◦ Examples: FAIR, VERIS… ◦ Provide standard taxonomy for describing risk ◦ Standard for gathering and expressing data in consistent manner ◦ Allows for analyzing complex risk scenarios
} SEIM projects can add value, if you are ready to do something
with all the data you collect, and if someone is going to look at it ◦ Large data sets can be difficult to filter and reduce, while maintaining
integrity
} Not all industries have the same risks and priorities
} There can be legal issues, when dealing with various data types, that vary by country
} Data visualization and mining tools can help discover issues by looking at data from different vantage points, and prompting drill-down and “asking better questions”
Good standard risk assessment frameworks exist to address
https://verisframework.wiki.zoho.com/
© 2011 Verizon. All Rights Reserved. MC14949 04/11. The Verizon and Verizon Business names and logos and all other names, logos, and slogans identifying Verizon’s products and services are trademarks and service marks or registered trademarks and service marks of Verizon Trademark Services LLC or its affiliates in the United States and/or other countries. All other trademarks and service marks are the property of their respective owners.
“VERIS is a set of metrics designed to provide a common language for describing security incidents in a structured and repeatable manner. The overall goal is to lay a foundation on which we can constructively and cooperatively learn from our experiences to better manage risk.” Each incident is a chain of events, composed of: Agent, Action, Asset, Attribute.
} Most companies have various regulatory reporting requirements (i.e. SOX, PCI…). Suggested metrics: ◦ Manager sign-off on access controls ◦ A&A control artifacts ◦ Audit reports/findings (number, severity, BU) ◦ Exception reporting/tracking ◦ PCI Compliance status, dates
Do you feel your compliance metrics are just CYA measures, and not driving real security
improvements or risk management? What are you going to do about it?
} Tactical/Operational: ◦ Incident metrics
� IDS � Forensics cases � Help desk cases: i.e. infected desktops � DLP: IP “leakage” ◦ Risk Metrics
� Time to patch � Vulnerabilities: severity, time to remediate � Daily AV status: # blocked, definition updates ◦ Maintenance
� Project management: milestones, deadlines � Security application/infrastructure support � Change management
http://www.mcafee.com/us/products/epolicy-orchestrator.aspx
http://www.arcsight.com/products/
} Strategic Planning ◦ Budgets ◦ Something to show policies are being followed ◦ Something to show controls/processes effective ◦ Something to show improvement over time ◦ Something to compare to industry best practice,
baseline ◦ Something to help prioritize response and resource
allocation � This can involve both compliance and operation
outputs
Seems like something is missing!
} A good example of how quantitative data can help security practitioners set priorities. ◦ What are the results for your industry vertical? ◦ Internal vs. External ◦ Where does data leakage occur?
} Apply 80/20 rule to focus on the 20% that leads
to 80% of your data loss.
http://www.verizonbusiness.com/go/2011dbir © 2011 Verizon. All Rights Reserved. MC14949 04/11. The Verizon and Verizon Business names and logos and all other names, logos, and slogans identifying Verizon’s products and services are trademarks and service marks or registered
trademarks and service marks of Verizon Trademark Services LLC or its affiliates in the United States and/or other countries. All other trademarks and service marks are the property of their respective owners.
© 2011 Verizon. All Rights Reserved. MC14949 04/11. The Verizon and Verizon Business names and logos and all other names, logos, and slogans identifying Verizon’s products and services are trademarks and service marks or registered trademarks and service marks of Verizon Trademark Services LLC or its affiliates in the United States and/or other countries. All other trademarks and service marks are the property of their respective owners.
© 2011 Verizon. All Rights Reserved. MC14949 04/11. The Verizon and Verizon Business names and logos and all other names, logos, and slogans identifying Verizon’s products and services are trademarks and service marks or registered trademarks and service marks of Verizon Trademark Services LLC or its affiliates in the United States and/or other countries. All other trademarks and service marks are the property of their respective owners.
© 2011 Verizon. All Rights Reserved. MC14949 04/11. The Verizon and Verizon Business names and logos and all other names, logos, and slogans identifying Verizon’s products and services are trademarks and service marks or registered trademarks and service marks of Verizon Trademark Services LLC or its affiliates in the United States and/or other countries. All other trademarks and service marks are the property of their respective owners.
} We are familiar with the equation: Risk = Threat x Vulnerabilities x Value } We must leverage historical data when we have it, or turn to industry trends
as expressed in DBIR. Lacking data it is reasonable to make best estimates.
} Communicate with stakeholders, and determine what they think is important, then measure and develop balanced scorecard they will find meaningful
} Find a way to add business value: ◦ Meeting regulatory requirements ◦ Consolidation of tools, reduction of resources ◦ Demonstrate reduced costs by reduction in help desk cases ◦ Business leaders take the loss of IP seriously ◦ Have security seen as a business enabler. New technologies come
with risks, but they may also lead to new innovations and competitive advantage.
} “Align the capabilities of the organization, and the exposure of the organization, with the tolerance of the data owners.” – Jack Jones, Huntington Bank(author of FAIR; a quantitative risk analysis framework)
} What are security professionals good at? ◦ Responding to a crisis ◦ Qualitative risk assessments (ordinal numbers) ◦ Justifying security spend in terms of risk management ◦ Making graphs that show how clever we are ◦ Developing operational metrics ◦ Talking to our peers, and reaching a consensus on how
to respond to new threats or enable new technologies ◦ Asking Gartner what we should think and do! ◦ Thinking holistically about security
} Where can security professionals improve? ◦ Preventing a crisis in the first place (always tough to
show you prevented what didn’t happen!) ◦ Gathering (better & fine-grained) quantitative data for
assessing risk (cardinal numbers) ◦ Justifying security spend in terms of business value ◦ Presenting results in easy to understand format ◦ Prioritizing based on predictive modeling ◦ Developing more strategic metrics that show security
efficacy and improvement ◦ Talking to business leaders in terms they understand ◦ Thinking holistically about the business
} Decide on your goals and objectives at the onset ◦ Long-term and short-term goals
} Identify key metrics (SMART) to generate ◦ Will these be qualitative or quantitative? ◦ Will these be manual or automated? ◦ Will these be based on a standard framework, or vetted against peers, or
using some other model? ◦ Will these be operational, strategic or business metrics?
} Establish a baseline and targets
} Determine how best to present metrics in a consistent way, for
audience and frequency
} Get stakeholder buy-in and feedback
} Develop a process for continuous improvement
} Dr. Deming is often incorrectly quoted as saying, "You can't manage what you can't measure." In fact, he stated that one of the seven deadly diseases of management is running a company on visible figures alone.
} "The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.”
} “In God we trust; all others must bring data.”
} Verizon DBIR ◦ http://www.verizonbusiness.com/go/2011dbir
} Verizon VERIS Framework ◦ https://www2.icsalabs.com/veris/
} FAIR Framework ◦ http://fairwiki.riskmanagementinsight.com/
} Open Source Security Testing Methodology, http://is.gd/Jyg757 (review)
http://www.osstmm.org/ (home)
} Trustwave SpiderLabs Global Security Report ◦ https://www.trustwave.com/GSR
} Ponemon Institute ◦ http://www.ponemon.org
} Security Metrics: Replacing Fear, Uncertainty, and Doubt, Andrew Jaquith (2007)
} Metrics and Methods for Security Risk Management, Carl Young (2010)
} John D. Johnson, Ph.D., CISSPSr. Security Program Manager, John [email protected] http://twitter.com/johndjohnson