cybersecurity test and evaluation - home - itea symposium...cybersecurity test and evaluation alex...

Office of Test & Evaluation

Science and Technology Directorate

Cybersecurity Test and Evaluation

Alex HooverTest Area Manager

Cyberspace & Homeland Security Enterprise Programs

202-254-5615

[email protected]

Policy

Practice

Threat Assessment

COI/MOE/MOP

Agenda

2

Purpose Improve operational resilience of network-enabled capabilities

and inform major acquisition decisions.

Applicability Acquisition programs subject to DOT&E oversight will

incorporate these procedures into all future TEMPs and OT&E Plans.

Programs will include cybersecurity in TEMPs

Mission context, threat description, stakeholders, evaluation framework, integrated

T&E, and resources

OTAs will include cybersecurity in OT&E concepts, plans, & reports

Realistic threat portrayal to determine mission effects

DOT&E will include cybersecurity in LOAs

Effectiveness, Suitability, and Cybersecurity

Procedures for Cybersecurity OT&E

3

Cybersecurity-Informed Acquisition

4

Is the capability sufficiently cyber secure

to enter initial production/deployment?

Refine T&E

Strategy

Conduct

Develop-

mental T&E

NEEDANALYZE/

SELECTOBTAIN

PRODUCE/

DEPLOY/

SUPPORT1 2 2B 2C 3

Test &

Evaluation

TEMP

OTEP

LOALOA

OTER OTERTEMP

Input to

Operational

Requirements

Develop T&E

StrategyConduct

Operational

T&E

LOA

TEMP

5

Lifecycle Cybersecurity T&E Activities

5

Cybersecurity Requirements

Attack SurfaceIntent

T&E

• MNS

• ORD

• CONOPS

• Threat

Assessment

• Local

• Adjacent

• Network

• Denial

• Disruption

• Modification

• Exfiltration

• Pivot

• Security T&E

• Blue & Red Team Assessments

• Realistic threats in cyber domain

Rigorous T&E is essential to close

the gap between authorities to operate and

operating securely

IOT&E FOT&E

NEEDANALYZE/

SELECTOBTAIN

PRODUCE/

DEPLOY/

SUPPORT1 32C2 LRIP

JRC 2BDT&E

ST&EST&E

Attack Surface

• Local

• Adjacent

• Network

Kill Chain

• Tactics

• Exploits

Kill Chain

• Tactics

• Exploits

Clearly define the threat(s) to the system and corresponding missions

Threat assessment should answer the following general questions:

Which threat actors may target the missions that the system supports?

What is their intent?

What do they view as the critical terrain to accomplish their intent?

What are their capabilities in terms of knowledge, tools, and operations?

What are their most likely and most dangerous attack vectors based upon their

intent and capabilities?

DHS does not have an existing process or office of primary

responsibility for program-specific threat assessments

Use Requests for Information thru Component and Department intelligence offices

DOT&E working long-term solution with DUSM, DHS I&A, and JRC

Program-Specific Threat Assessment

6

Denial – Blocking completion of mission tasks.

Degradation – Decreasing the speed, quality, or other performance

characteristics for mission tasks.

Manipulation – Altering the information available to decision makers.

Exfiltration – Gaining information about mission details to be

exploited against other assets.

Pivot – Using access to one system/network to gain access to a

partner system/network.

Intent

7

Capabilities

8

Minimal Limited Moderate Advanced

Kn

ow

led

ge

General Systems Home market hardware,

networks and, general-

purpose languages. Basic

user OS and applications.

Public cryptography/

authentication. Public exploits

of known vulnerabilities.

Common hardware, firmware, and

defensive devices. Enterprise

network and OS. Industry data

protocols. 0-day exploits of less

common/more vulnerable software,

custom software.

Custom hardware, embedded

systems, and less common

network/protocols, specialized

firmware. Biometric-based

authentication. 0-day exploits of

more common/less vulnerable

software.

Classified systems, platforms, and

software. Cross-domain devices,

cryptography and associated hardware.

0-day exploits of restricted government

systems and industrial control systems.

Target Network and

Systems

Information found from

commonly available open

sources or from external

reconnaissance of target

organization.

Knowledge of network and system

specifications and type/configuration

of host-based defenses equivalent

to an authorized user in the target

environment.

Knowledge of network and

system specifications and

type/configuration of networked

defenses equivalent to an

authorized administrator in the

target environment.

Knowledge of network and system

specifications and defenses equivalent

to an authorized domain administrator in

the target environment.

Target Operations Information found from

commonly available open

sources or from external

reconnaissance of target

organization.

Knowledge from more specialized

literature or equivalent to prior

experience with target operations,

including key information or

supporting systems.

Knowledge equivalent to

substantial prior experience with

target operations, including work

flow and sub-task objectives.

Knowledge of current target operations

equivalent to an experienced authorized

operator.

To

ols

Hardware Inexpensive home market

hardware.

Hard-ware, clusters, costing

$10,000s or dozens of man hours.

Hardware costing $100,000s or

hundreds of man hours.

Custom hardware costing $1,000,000s

or thousands of man hours.

Software Freeware and inexpensive

commercial tools.

Commercial software. Custom software, polymorphic

malware, rootkits.

Custom software, firmware-resident

malware.

Infrastructure Access through publically

available infrastructure.

Direct control of leveraged public

infrastructure.

Covert remote access tools and

loggers.

Covert close access.

Op

era

tio

ns

Planning Opportunistic actions, no

planning.

Intent and short-range plans formed

on-the-fly as needed.

Organizes one or more

operations with specific target

systems and associated effects

on target organization

Organizes multiple operations against

separate targets, synchronizing timing,

accesses, and planned second-order

effects

Procedures No demonstrated stealth, non-

attribution or efficient use of

resources

Countermeasures for common

defensive systems. Non-attribution.

Efficiency in use of resources

consistent with intent.

Advanced and custom non-

attribution tools. Efficiency in use

of resources consistent with

intent

High degree of control of defensive

infrastructure. Non-attribution, false flag

operations. Efficiency in use of resources

consistent with intent

Persistence Intermittent, directed activity. Gradual, low level passive

operations.

Repeated active operations. 24/7 monitoring and control of offensive

capabilities.

8

Critical Operational Issue

Is this capability resilient to cyber attack?

Measures of Cybersecurity

How resilient is this mission to DOS attack of this capability?

How resilient are the tasks to cyber degradation?

How resilient are the procedures to data manipulation?

How resilient is the mission to data exfiltration of the key cyber terrain?

How well does this system protect against attack from/to interfaced capabilities?

Possible Evaluation Questions

9

10

Cybersecurity

Is this capability resilient to cyber attack?

Denial of Service (Mission Impact)

- Probability of Occurrence - Repeatability

- Duration - Attack Resources

Degradation of Service (Task Impact)

- Probability of Occurrence - Degree of Degradation


- Repeatability - Defend Resources

- Probability of Detection

Data Manipulation (Task Impact)

- Probability of Occurrence - Degree of Manipulation




Data Exfiltration (Enterprise Impact)

- Probability of Occurrence - Significance of Exfiltration




External Pivoting (Enterprise Impact)

- Probability of Occurrence - Probability of Detection



Effectiveness

Measure 1

Measure 2

Measure 3

Suitability

Measure 1

Measure 2

Measure 3

Cybersecurity

Measure 1

Measure 2

Measure 3

Understand Collective

Impact on Mission/Task

Accomplishment

Sample

Cybersecurity

Evaluation Structure

Rules of Engagement

11

Purpose – Support evaluation of ...

Threat Assessment

Actors (FIS, Terrorist, Criminal, Activist, Mercenary, Hackers)

Intent

Capabilities (Historical, Projected, Surrogates)

Strategic Goals

Definition of Capability Under Test

System Boundary

Included Systems

Excluded Systems

Mission Impacts (DOS, DEG, DMAN, EXFIL, PIVOT)

Operational Objectives

Targeted Data (leads to ...) Deliberate

Targeted Systems (leads to ...) Deliberate

Targeted Networks (leads to ...) Deliberate 80 / Exploratory 20

Targeted Interfaces (leads to ...) Deliberate 50 / Exploratory 50

Relevant Vulnerabilities Deliberate 20 / Exploratory 80

Rules of Engagement (cont’d)

12

Tactical Plan

Schedule

Operational Objective, Capability (Surrogate) Start, End

Initial Access

TTP by Scheduled Event (planned and contingency)

OCO Actions

Limits of Action

Prohibited Actions

Termination Conditions/Notification

DCO Posture

Active Events – Events the DCO will carry out

Stop Events – Events where the DCO will report detection to the red team and the event will stop

Passthrough Actions – Events where the DCO will report detection and the event will proceed

Data Handling

cybersecurity test and evaluation - home - itea symposium...cybersecurity test and evaluation alex...

Documents