U.S. ARMY TEST AND EVALUATION COMMAND

Cybersecurity Test and Evaluation: The ATEC PerspectiveMike Zwiebel, Director of Test Management

27 March 2019

Agenda

1. What does ATEC Need to Know about Cybersecurity2. Cybersecurity Evaluation Data Sources3. Cybersecurity T&E Phases4. Cybersecurity Testing Opportunities5. DEVOPS / Rapid Acquisition: Notional T&E Strategy6. Challenges7. Back-Up Charts

2

Cybersecurity: What Does ATEC Need To Know• U.S. Army Evaluation Center (AEC) must evaluate effectiveness, suitability,

survivability.• Operational survivability concern: Is the system robust and resilient against hostile

cyber activity?o Does the system meet all Federal and DoD cybersecurity regulations,

guidelines, and best practices? o Does the system introduce exploitable cyber vulnerabilities to the systems and

networks with which it interoperates?o Are vulnerabilities introduced to system survivability when integrated into its

end-state employment? o Does the system provide the capability to detect the loss of system or data

integrity, and to restore the system and data to a known good (trusted) state?• Operational system/network cybersecurity capabilities:

o Prevent compromise by threato Mitigate effects when compromisedo Recover system to pre-compromised state Compliance with IA controls

are necessary but not sufficient

3

Cybersecurity: Evaluation Data Sources• Documentation

o DOD Architecture Framework (Operational/System/Technical Views)o Program Protection Plano Contract Language (Request For Proposal)o Technical Reviewso System Engineering Plans

• Risk Management Framework (RMF) Artifacts• Contractor • System Integration Labs (SILs) - PM or AFC Combat Capabilities Development

Command (CCDC)• Test Teams

o CCDC/ Data and Analysis Center (D&AC)/Lethality, Survivability & Human Systems Integration Division (LSH)

o PM ITTS, Threat Systems Management Office (TSMO)• ATEC Test Centers?

4

Cybersecurity: T&E Phases

• Develop Cybersecurity T&E Strategy

• Develop Evaluation Methodology inclusive of Cybersecurity

Outcome: MS A TEMP

• Develop DT&E Framework• Update Cybersecurity T&E

Strategy• Incorporate Cyber Attack

Surface elements into test plans

• Define resources for cybersecurity DT&E

Outcomes: MS B TEMP and input to RFP, PDR,

CDR

• In cooperation with SE develop understanding of system vulnerabilities

• Assess system for vulnerabilities

• Provide feedback to SE

Outcome: Test plans; initial vulnerability

assessments; input to CDR

• Execute adversarial cybersecurity DT&E event within realistic mission environment.

• Use of Cyber ranges

Outcome: Input to DT&E Assessment,

MS C TEMP

• Overt and cooperative review of the system to characterize operational cybersecurity status

• Determine residual risk as well as readiness for the Adversarial Assessment.

Outcome: POA&M for documented

vulnerabilities

Understand Cybersecurity Requirements

Characterize Cyber Attack

Surface

Cooperative Vulnerability Identification

Adversarial Cybersecurity

DT&E

Full RateProduction

Decision Review

Technology Maturation &

Risk Reduction

Engineering & Manufacturing Development

Production and Deployment O&S

SRR SFR CDR TRR SVRASR

MaterielSolutionAnalysis

MDD

DRAFTCDDAOA CDD CPD

IATT

Cooperative Vulnerability and

Penetration Assessment

IOT&E

T&E Phases

OTRRDT&EEvent

Adversarial Assessment

CDD Validation

Dev RFP Release Decision

DT&E Assess-

ment

DT&E Assess-

ment

PDR

ATOA B C

• Full operational test and evaluation of the system’s defensive cyberspace performance in the operational environment.

Outcome: OTA and DOT&E Reports

Phases are iterative and executed as part of the Acquisition continuum.

Developmental Test Integrated DT/OT

Operational Test

This is Cybersecurity “Shift Left”5

6

Network

Environment (Live, Virtual, Collective)

Systems Under Test (SUTs)

Electronic Attack

Cyber Eletromagnetic Activity

ATEC• Test Methodology/Environment• Instrumentation• System Access/Operation• System Knowledge• Evaluation (ESS/Safety)

AFC Lethality Survivability and HSI

Division• Army CVPA lead• Augments TSMO during

Test• Conducts assessment

during Operation Test:o Prevento Mitigateo Recover

PM ITTS Threat Systems Management Office

• NSA certified Red Team• Cyber Aggressors• Validated Cyber Threat

Pilot Studies• ATC• EPG• RTC

National Cyber Range• Operated by TRMC• Provides realistic

Cybersecurity T&E

System Access• Network• Physical• Electronic• Other

Cybersecurity: Testing Opportunities

Test Center Involvement7

Other capabilities

Early Soldier

Involvement

8

Deployment (Refinement by CPTs at Ft. Gordon)

Test and Evaluation(1 to 6 months of testing at the forge facilities with dedicated team)

Capability Development (TBD)

Demonstrations, software and code analysis

Fielding Decision

Dedicated and Independent Test Team & Contractors located at the forge facilities –Enables Continuous, Real Time Data Analysis and Report Writing

Report

DT/ Lab testing

Robust testing in an operational realistic

environment

Limited User Event

TestingFix

Find

Test

DT/ Lab testing

FixFind

Test

Limited User Event

testingFixFind

TestAdditional T&E as required

Ktr/ MIL operators testing (CPT)

In stride assessments & iterative testingKTR MILKTR / MIL

Agile Test Team Focus: Learn, Assess, Find, Identify, Recommend, Fix, and Verify

Entry Point for prototype

DT/OT OT

Month 1

1 2 3 4

Month 2

1 2 3 4

Month 3

1 2 3 4

Month 4

1 2 3 4

Month 5

1 2 3 4

Month 6

1 2 3 4

Month 7

1 2 3 4

Month 8

1 2 3 4

Month 9

1 2 3 4

Month 10

1 2 3 4

Month 11

1 2 3 4

Month 12

1 2 3 4

DEVOPs / Rapid AcquisitionNotional T&E Strategy

• T&E strategy ideally integrated into a dedicated test and training site • Must represent OT-like environment

• Notional schedule aligns with proposed development/fielding needs of end user (e.g., 1 to 6 months T&E window)

• Small continuous events to provide feedback and a tailored evaluation product • Requires a dedicated and independent Test Team; in-stride T&E

• Cybersecurity is embedded throughout the process• Ideally still requires exercises at least annually for OT-like assessment(s)

Capability Drop

Cyber Tabletop Example of a Capability Drop

Learn

Tech Evolution Intel Driven

Recursive process to support simultaneous

capability drops of multiple RDPs

Capability Development, Test and Evaluation, Deployment

Test Concept Brief Emerging Results Brief

DT: Developmental Test OT: Operational Test

Challenges• Cybersecurity as Systems Engineering Discipline • Contractual Language for systems with IT

• Build Cybersecurity into design• Accountability for Cybersecurity findings discovered in testing• Data accessibility

• Reducing introduction of vulnerabilities of integrated systems• Understanding Operational Requirements and Impacts

• “If there is a computer in something, it can be cyber-attacked, and we need to be able to harden it and defend it.” the Pentagon’s Deputy Chief Information Officer for Cybersecurity Mr. Richard Hale

• “The Joint Staff has recently put out a formal requirement document that includes cybersecurity as a key part of the survivability key performance parameter (KPP) for every new system”

• Metrics Defensible Systems

9

Back-Up Charts

10

A B CIATT OTRR

OT Cooperative Vulnerability

and Penetration Assessment

OT Adversarial Assessment

FRP

Phase 5CVPA

Phase 6AA

Proposed Cybersecurity T&E Events

DT Adversarial Assessment

Phase 4Phase 3

DT Cooperative Vulnerability

& Penetration Assessment (CVPA)

Events derived from draft DASD(DT&E) DoD Cybersecurity Test and Evaluation Guidebook, and DOT&E Cybersecurity Operational Test and Evaluation Guidance Memo (01 August 2014)

Phase 2Phase 1Understand

Cybersecurity Requirements

Characterize Cyber Attack

Surface

Developmental Test Integrated DT/OT Operational Test

Analysis phase

Test phase

RMF

Cooperative Vulnerability Identification

Adversarial Cybersecurity

DT&E

11

Cybersecurity T&E Process (4)“Shift Left”

Yesterday’s Intermediate Threat is Today’s Novice Threat

Compliance with IA controls / standards and profiles are

necessary but not sufficient

Fielded systems found to have novice vulnerabilities during OT, which is problematic and costly.

Threat

Program Start

Program OT

12