Security Metrics Rehab Breaking Free from Top ‘X’ Lists, Cultivating Organic Metrics,

& Realizing Operational Risk Management

April 11, 2014

About Me

2

!   Author of P.A.S.T.A threat modeling methodology (risk centric) (Wiley Life Sciences)

!   Nearly 20 years of IT/Security experience (Developer, Security Arch, Dir of RM, CISO roles)

!   Founder of VerSprite, global consulting security firm based in Atlanta

!   OWASP ATL Chapter/ Project Leader for past 6 years

!   VerSprite works across Retail, Hospitality, Financial, Gov’t, Higher Ed, Healthcare, Info Svc

!   Co-organizer to BSides Atlanta grassroots security conference

!   B.S from Cornell in International Finance

!   Contact Info: !   tonyuv@versprite.com !   @t0nyuv / @VerSprite

About This Talk

3

!  Counterculture presentation on metrics, governance, and risk

!  Depict pros/ cons around existing metrics/ frameworks in public domain

!   Introduce seed of thought around building organic security metrics

Consider the Following

4

If you had to only pick one… !   Option A: Fully developed security controls

framework w/ supporting metrics based upon leading industry lists and control frameworks

!   Option B: Fully developed risk framework where

inherent and residual values are quantifiable, supported, and tied to business impact scenarios

METRICS VS “BSETRICS”

Separating Fact from Fiction

Metrics !   Objective focused !   Built from ‘What Do I Need’ (e.g. –

goal of providing evidence to effective technology/process management)

!   Data source is dependable & vast !   Metrics should have a reliable data

source that augments over time !   Outliers are factored out !   Support clearly defined IT/ Biz goals

“BSetrics” !   Metrics that ‘feel/look

good’ (e.g. – closed risk issues)

!   Built from ‘What Do I have’ (e.g. – tool begins to shape metrics discussion

!   Based upon “industry standard” !   Keeping Up w/ the Jones’

Metrics !   Building metrics to manage

perception !   Data set is limited (e.g. – time,

breadth, pre-fixed) !   Outliers are not factored out

6

Bad Metrics

7

!   Issues remediated !   Unanswered: Tested/ not tested !   Unanswered: Was issue resolved, closed, transferred !   Unanswered: Is this ‘issue’ important

!   # High Vulnerabilities Closed !   Deemed ‘High’ by whom? !   No context (High risk to a Low value asset)

!   # of Code Imperfections to a Top X List (SAST Scan) !   Top X List begins to drive your risk perception devoid of anything

else !   Cultivating responses, remediation, and reports solely on top X items

!   # of Pen Tests | # WebAppScans Conducted !   Doesn’t factor in automated or poorly conducted testing

STATUS QUO SECURITY METRICS

Process Metrics !   Is a SDL Process used? Are security

gates enforced? !   Secure application development

standards and testing criteria? !   Security status of a new application

at delivery (e.g., % compliance with organizational security standards and application system requirements).

!   Existence of developer support website (FAQ's, Code Fixes, lessons learned, etc.)?

!   % of developers trained, using organizational security best practice technology, architecture and processes

Management Metrics !   Management Metrics !   % of applications rated

“business-critical” that have been tested.

!   % of applications which business partners, clients, regulators require be “certified”.

!   Average time to correct vulnerabilities (trending).

!   % of flaws by lifecycle phase. !   % of applications using

centralized security services. !   Business impact of critical

security incidents.

9

Examples of AppSec Metrics Today

AppSec Metrics in Vuln Management

!   Number and criticality of vulnerabilities found. !   Most commonly found vulnerabilities. !   Reported defect rates based on security testing (per

developer/team, per application) !   Root cause of “Vulnerability Recidivism”. !   % of code that is re-used from other products/projects*

!   % of code that is third party (e.g., libraries)* !   Results of source code analysis**:

!   Vulnerability severity by project, by organization !   Vulnerabilities by category by project, by organization !   Vulnerability +/- over time by project !   % of flaws by lifecycle phase (based on when testing occurs)

Source: * WebMethods, ** Fortify Software

ROOM FOR IMPROVEMENT

Forrester Survey: “What are your top three drivers for measuring information security?”

Source: “Measuring Information Security Through Metrics And Reporting”, Forrester Research, Inc., May 2006”

63%

11%

23%

26%

37%

51%

Manage risk

Report progress to business

Better stewardship

Loss of reputation

Regulations

Justification for security spending

Report progress to business

Better stewardship Base: 40 CISOs and senior security managers

Good Metrics – Align w/ Maturity Model

Metrics  ma+er  most  when  they  have  direct  or  indirect  relevance  to  opera5onal/  strategic  goals  

Align  to  Biz/  IT  Goals  

Directly  or  indirectly,  categories  to  be  measured  need  to  map  to  key  indicators  that  ma+er  in  IT  Ops,  Sales,  Finance  

Relate  to  Business  Processes  

Good  start  is  to  map  metric  areas  to  key  processes  sustained  by  a  BIA  

Map  to  a  Business  Impact  

!   Start simple !   Forget what everyone

else is doing – for now !   Perform an internal

PoC with LOBs/ BUs !   Grow base of coverage

over time !   Mature metrics by

benchmarking against industry reports/ analysis

13

Opportunities for Metrics - Secure Development Life Cycle (SDL)

14

Secure questions during interviews

Concept Designs Complete

Test plans Complete

Code Complete

Deploy Post Deployment

Threat analysis

Security Review

Team member training

Data mutation & Least Priv Tests

Review old defects Check-ins checked Secure coding guidelines Use tools

Security push/audit

= on-going

Learn & Refine

External review

Source: Microsoft

Software assurance activities conducted at each lifecycle phase

Organizing Metric Types Process Metrics

Information about the processes themselves. Evidence of maturity.

Vulnerability Metrics Metrics about application vulnerabilities themselves

Management Metrics specifically designed for senior management

Examples !  Secure coding standards in use !  Avg. time to correct critical vulnerabilities

Examples !  By vulnerability type !  By occurrence within a software development

life cycle phase

Examples !  % of applications that are currently security

“certified” and accepted by business partners

!  Trending: critical unresolved, accepted risks

Our Security Metric Challenge

“A major difference between a "well developed" science such as physics and some of the less "well-developed" sciences such as psychology or sociology is the degree to which things are measured.” Source: Fred S. Roberts, ROBE79

“Give information risk management the quantitative rigor of financial information management.” Source: CRA/NSF, 10 Year Agenda for Information Security Research, cited by Dr. Dan Geer

BREAKING FREE FROM TOP ‘X’ LISTS

Let’s Rethink Security Lists

Pros !   Great content from various

sources: OWASP Top Ten, SANS 20 Critical Security Controls, MITRE CWE Top 25, WASC TC v2, OWASP Top 10 - Mobile

!   Provide a benchmark for testing | measurement

!   Brings broader industry perspective

!   Better suited for more mature programs where benchmarking is timely

Cons !   This defines an AppSec’s

program baseline !   Used as ground floor level

of metrics !   Tempts programs to look

outwardly vs. inwardly !   Doesn’t foster for Good

Metrics to take root !   Tools don’t make quitting

this trend easy (pre-defined profiles)

!   Not a real basis for threat or risk analysis

How Do Lists Break Us Free from This Cycle?

METRICS & LISTS – TIMING IS EVERYTHING

OWASP OpenSAMM Project !   Evaluate an organization’s

existing software security practices

!   Build a balanced software security assurance program in well-defined iterations

!   Demonstrate concrete improvements to a security assurance program

!   Define and measure security-related activities throughout an organization

! http://www.opensamm.org !   Dedicated to defining,

improving, and testing the SAMM framework

!   Always vendor-neutral, but lots of industry participation

!   Open and community driven !   Targeting new releases every

6-12 months !   Change management process

SAMM in a nutshell

!   Evaluate an organization’s existing software security practices !   Build a balanced software security assurance program in well-

defined iterations !   Demonstrate concrete improvements to a security assurance

program !   Define and measure security-related activities throughout an

organization

OWASP OpenSAMM (Software Assurance Maturity Model)

!  Look inward !  Start with the core

activities tied to SDLC practices

!  Named generically, but should resonate with any developer or manager

Leveraging Lists at the Right Maturity Level !  Measure what you need across a framework’s

(OpenSAMM) area !   Identify ‘indicators’ that support business/ product

goals & objectives !  Apply use of lists for benchmarking as maturity

level rise

Develop ‘Organic’ Security Metrics

Reasons !   Supports contextual

analysis based upon internal operations

!   Top down approach to regressing to security metrics that matter

!   Will substantiate security initiatives across non-InfoSec areas

Baking Organic Metrics Organiza5onal  Objec5ves  

Opera5onal  Processes    

Suppor5ng  Technology  &  Infrastructure  

BU/  LoB  Objec5ves  

Revenue  Growth  • Reputa5onal  Loss  • Non-­‐Compliance  

Cost  Reduc5on  • Fines  &  Penal5es  

Product/  Service  Objec5ves  Product  Innova5on  •  IP  Security  •  Insider  Threats  •  Incident  Handling/  Response  

Efficient  Service  Delivery  • Con5nuity  • Data  Integrity  

Revisiting Lists

!   Build your processes first

!   Design metrics mapped to activities for those processes

!   Develop scorecards that report on organic security metrics that relate to operational, financial areas

!   Bake-in industry ‘lists’ in order to reflect more advanced quantitative analysis (Level 4)

Creating Scorecards

!   Gap analysis !   Capturing scores from detailed

assessments versus expected performance levels

!   Demonstrating improvement !   Capturing scores from before

and after an iteration of assurance program build-out

!   Ongoing measurement !   Capturing scores over

consistent time frames for an assurance program that is already in place

THANK YOU!