UNCLASSIFIED

Threat Model Test Capability Enhancements

Supporting Acquisition Program Testing

Ms. Melody JohnsonAir Force Range Oversight

Test Resource Management CenterDecember 12, 2018

UNCLASSIFIED

Purpose

Provide an overview of ongoingThreat Modeling and Simulation (M&S) Working Group

threat model roadmap development efforts

2

UNCLASSIFIED

Threat Models Supportthe LVC Test Environment

T&E Threat Models

VISION: Agile Infrastructure That Supports Agile Acquisition

3

UNCLASSIFIED

All T&E Users BenefitFrom Threat Model Development

Digital System Models (DSMs)• AFMC (SIMAF)• F-35 JSE/F-22 ACS• ITASE• DIADS

Threat H/W in the Loop (HITL)• Army RF Sim System at Redstone Arsenal• Navy Missile-on-the-Mountain at China

Lake South Range • ECSEL at Pt Mugu

Installed System Test Facilities (ISTFs)• Navy ACETEF at Patuxent River• Air Force BAF at Edwards AFB• JPRIMES at Eglin AFB• Army AvSTIL at Redstone Arsenal

Open-Air Ranges (OAR)• China Lake South Range RSDEs• Nevada Test & Training Range RF Stimulators

Infrared (IR) Scene Prediction Models

4

UNCLASSIFIED

The Threat Model Need

• Several threat model shortfalls exist, to include:– Lack of common threat models in use across the DoD– Models are not necessarily endorsed by the Intelligence Community (not

authoritative)– No broadly implemented enterprise management process– Need for improved organizational, functional, and technical linkages between T&E

and intelligence communities (to improve needs definition and model management)– Key Performance Parameter/Key System Attribute (KPP/KSA) threat models are not

sufficient to meet current and emerging acquisition program test requirements– Current threat model development efforts are stove-piped, frequently proprietary,

and serve a single, specific use– Test and Evaluation (T&E) investments not informed, nor aligned, across the

enterprise– Acquisition programs typically bear the burden of funding T&E threat M&S– Inability to acquire emerging advanced threat systems for T&E use– Lack of T&E/acquisition community knowledge of T&E threat M&S capabilities– No systematic process to capture test data to improve threat modelsThe Threat M&S Working Group has begun to addressthese shortfalls for infrared (IR) and radio frequency (RF) threat M&S for T&E.

5

UNCLASSIFIED

Importance of Common, Authoritative Threat Models

• Ensure Threat M&S capability for DoD weapon systems are prepared to counter adversary IR and RF threat systems

– DoD test facilities can only test 21% of high-priority RF threats and 22% of medium- and low-priority RF threats in a specific theater of operations

– For worldwide IR, tests can be conducted against only 28% of high-priority threats and 26% of medium- and low-priority threats

• Ensure DoD can sufficiently test and train against existing and emerging Western Pacific and other regional IR and RF threats

• Ensure air combat weapon systems will be fielded with adequate assessment of system performance for effectiveness, suitability, and survivability against critical IR and RF threats

– Without this assessment, the right information will not be provided to decision-makers

• Ensure model development based on an enterprise management processproviding development and interoperability standards to ensure data correlation with threat models across a variety of DT and OT environments

– Across the T&E spectrum, from integration labs to open-air ranges

6

UNCLASSIFIED

Importance of Common, Authoritative Threat Models (cont.)

• Common, authoritative threat model development best addresses the need for high-density threat environments and known shortfalls with test capabilities that will:

– Reduce proprietary and/or single-use threat models– Augment costly open-air range testing and enable testing where threat assets cannot be replicated– Provide opportunities for test where none currently exist– Account for changes to threat system software as needed– Reduce threat model management and sustainment cost– Provide more complete weapon systems performance information to decision-makers

• An enterprise approach will improve organizational, functional, and technical linkages between the T&E and Intelligence Communities

– Better awareness of threat M&S shortfalls and ability to lead-turn future test threat M&S needs– Clearly defined enterprise requirements and priorities– Means to communicate the urgent need for threat M&S development for T&E– Ability to rapidly account for changes and present more realism into test

7

UNCLASSIFIED

T&E Threat Model Development(CCB Process)

IPC Model Developers

TETRA T&EModel

ManagerT&E Users

SYSEA/EPResponse

RMC EMSIG

IC BaselineThreat Models

& Analysis

T&E ThreatModels

Wrapped

T&E Threat ModelRepository

Version Control

T&E DistributionControl

VV&A andReference Reports

Certification Data

12

3

3 Distinct Models Require Life-Cycle Management

Modify ifNeeded

Modify For Install

AnomalyReporting

AnomalyReporting

Hardware-in-the-Loop(HITLs)

Installed System TestFacilities (ISTFs)

Open-Air RangeSimulators

T&E Digital Simulators

8

STIMS 3.0AMBERCHIMERA

LIVEWEEW

UNCLASSIFIED

Threat M&S Working Group Background

• Despite past studies and investment efforts, the need remains to address IR and RF T&E threat M&S shortfalls

– Based on current and emerging IR and RF threats

• TRMC is partnering with:– Director, Operational Test and Evaluation– Office of the Assistant Secretary of Defense, Developmental Test and Evaluation– Services and Agencies– Intelligence Production Centers (IPCs)

o Missile and Space Intelligence Center (MSIC), National Air and Space Intelligence Center (NASIC), Office of Naval Intelligence (ONI), and National Ground Intelligence Center (NGIC)

• In order to:– Document threat M&S test capability gaps, vetted by Service T&E representatives– Identify an Intelligence and T&E Community validated prioritized threat list– Define gap filling projects to address threat M&S needs for T&E– Deliver a T&E Threat M&S Capability Investment Roadmap (IR/RF)– Establish an enterprise process for threat model life cycle management– Fund identified needs – Continue the effort beyond FY 2018

9

UNCLASSIFIED

Threat M&S Working Group Key Stakeholders and Partners

DOT&EJHU/APL

TRMCTETRA DT&E IDA

Service T&E RepsIPCs

T&E Ranges

T&E ISTFs/HITLs T&E Labs

Service T&E Reps

CCB

Working Group

10

OUSD(R&E) EW EXCOM DOT&E DT&E TrainingOUSD(I) T&E Executives TRMC Acquisition

Champions

TETRA

OTAsDT&E

UNCLASSIFIED

Threat M&S Working Group Approach

Drafts developed; Most Solutions Priced

Costs across FYDP determined

CAPE approved RF Pacific Rim use case for RF/worldwide IR use cases developed, briefed, and assessed for need

Service prioritized needs and gaps received; gap analysis conducted and gaps drafted

Solutions (projects) drafted and ranked by working group members

IR ASE and 4 RF Programs of Record (PoR) Drivers DefinedDrivers

Use Cases

Performance Attributes

Threat Model Capability Gaps

Potential Gap Filling Projects

Business Case Analysis

Investment Roadmap

Community Coordination

Program Risk vs Infrastructure Cost

Trac

eabi

lity

PoR test requirements determined via use cases

Collaboration and endorsement

A collaborative process leading to high fidelity investment trade-space for leadership decision.

11

UNCLASSIFIED

Threat M&S Working Group Scope

M&S

SPACE

LAND

SEACYBER

AIR

• RF Drivers: Deliberate decision to start with Air Combat Systems (ACS)– Limited to the APR-39D Radar Warning Receiver, F-15 Eagle Passive Active Warning Survivability

System (EPAWSS), Next Generation Jammer (NGJ), and B-2 Defensive Management System Modernization (DMS-M)

– Used CAPE approved Pacific Rim use case

• IR Drivers: Aircraft Survivability Equipment (ASE) platforms and advanced development systems‒ Such as the AAR-47 Missile Warning Systems (MWS), AAR-57 Common MWS, Common Infrared

Countermeasures (CIRCM) program, Advanced Threat Warning System, and Advanced Threat Detection System (ATDS)

‒ Worldwide use cases used

Threat RF M&S needs exist in the other warfare areas (Land, Sea, Space, Cyber)

12

UNCLASSIFIED

Threat M&S Working Group Use Cases

(Derived from Authoritative Sources)• RF use cases:

– B2 DMS-M day 1 or early phase mission− Day 1 attack bombing mission; live auto-route modifications due to changing threat indications and layout.

– Electronic support, opposed CSAR with land-based fighter support − Rescue Combat Air Patrol mission; air-to-air and air-to-ground components in a threat environment (F-15

EPAWSS, F-22 with Small Diameter Bomb, F-15E).

– Electronic attack, rescue/escort CSAR− EA-18G with NGJ; target jamming/strike mission to protect aircraft during campaign day 3-4.

– Electronic support, MV-22/APR-39 combat search and rescue (CSAR) − Aircraft lost, downed crewmember; MV-22 in threat ring.

• IR use cases:– 1v1 IR guided missile engagement

− Missile warning system (MWS) detect/locate; laser jammer and/or flare countermeasures to defeat; any missile type, any host aircraft, any weather condition and background

– 1v1 hostile fire engagement− MWS to detect and locate HF threats; any HF threat type, any host aircraft, any atmospheric condition

and background

– Multiple concurrent missile and/or hostile fire engagement− Same situation and conditions as 1v1, but multiple concurrent engagements

– Missile and hostile fire warning systems in alternate spectral bands− Multiple spectral bands employed; same conditions as previous use cases

13

UNCLASSIFIED

Threat M&S Working Group Threats Identified

Specific threats were identified through acquisition program use cases and vetted through working group members• 81 RF threats (representative listing)

– Long-range Surface-to-Air-Missiles (SAM) and associated radars– Long-range Air-to-Air Missiles (AAM)– Airborne Radars/Airborne Intercept Radars– Target Acquisition Radars– Target Engagement Radars– Naval SAMs– Naval radars– Coastal Surveillance Radars– Passive Surveillance Radars– Ground-based Radars– Airborne Early Warning and Control Systems

• 73 IR threats – 38 Surface-to-Air Missiles– 10 Air-to-Air Missiles– 25 Hostile Fire threats

RF threats are applicable to other air combat systems beyond the selected drivers.

14

UNCLASSIFIED

Threat M&S Working Group Roadmap Development

• Specific threats were identified through program of record use cases and vetted through working group members

– 81 RF and 73 IR threats (prioritized with Air Force and Navy Service T&E representatives; T&E Executives reviewed)

• Gap analysis conducted based on drivers, use cases, threats identified, and current T&E threat model capability

• Working Group developed RF and IR roadmaps – 38 RF projects developed to provide:

o New models, updates to existing models, improved multi-spectral signatures and RF data Electronic Warfare Integrated Reprogramming (EWIR) database enhancements and Next Generation Electronic

Warfare Environment Generator (NEWEG) support as exampleso Improve threat M&S characterization for T&E

Recurring Intelligence Production Center (IPC) all-source analysis; Specific threat analysis for 10 identified threatso Improved threat M&S management and infrastructure

– 34 IR projects developed to provide:o IR missile performance, Hostile Fire performance, missile source signature, hostile fire source signature,

background, target, and on-aperture signature modelso Joint Standard Instrumentation Suite (JSIS) full operational capabilityo Improved threat M&S for T&E management

• Results vetted through working group members and Service T&E and Intelligence communities stakeholders

15

UNCLASSIFIED

Threat M&S Issue Funding Breakout

• $113M added to the FY 2019 President’s Budget– Model development (10 RF and 10 IR) $65.8M (managed via the CTEIP PE)– JSIC FOC $47.2M (managed by DOT&E)

• 10 high-priority, high-fidelity RF threat models (8 LIVE, 2 CHIMERA)– 8 STIMS 3.0 architecture Threat Modeling and Analysis Program (TMAP) models (MSIC developed)

o Provides scripted, effects-based evaluation without an operator-in-the-loopo Emulates RF missile threats

– 8 Laboratory Intelligence Validated Emulator (LIVE) – RF missile threats (MSIC developed)o Real-time model provides level of fidelity previously unavailableo A state-of-the-art closed-loop, low-power, RF direct-inject radar emulatoro Provides emulative fidelity closed-loop capability to capture electronic attack (EA) responsivenesso Enables electronic warfare (EW) technique evaluation, at RFo Allows for operator-in-the-loop capabilityo (Field Programmable Gate Array (FPGA) rack, STIMS model, radar main computer, test kits, tools, data

analysis, displays, validation)

– 2 AMBER architecture TMAP-based models for RF threat radars (NASIC developed)o Simulink-based analysis tool for ground-based radars (supports ground-based read analysis)o Capable of being used in real-time scenarios with enhanced EA/electronic protection (EP) functionality

16

UNCLASSIFIED

– 2 CHIMERAo Real-time, real-signal, high-fidelity model for use in all stages of testing and trainingo Consumes, processes, and transmits real signalso Provide a surrogate or faithful representation of a specific threat system or class of threat systemso For use in laboratories, HITLs, and ranges o (PC in a rack with FPGA, Amber model, radar main computer, low-fidelity test kits, tools and data

analysis, validation)

• 10 high-priority, high-fidelity IR missile fly-out models• Joint Standard Instrumentation Suite (JSIS)

– Fully integrated capability to collect ground truth and plume signature data during IR missile and Hostile Fire (HF) munitions live fire events

– Data collected will be used to update/improve IR and HF threat models for T&E use

Threat M&S Issue Funding Breakout (cont.)

17

UNCLASSIFIED

Threat Models for T&EEnterprise Management Approach

• Develop a plan to identify, develop, upgrade, manage, distribute and sustain threat models across the threat model life cycle

• Review current threat M&S policies and make recommendations for improvements or new policies

18

UNCLASSIFIED

ICBandwidth

Threat M&S Enterprise-Wide Efficiencies

• Senior leadership direction & limited funding demand that we improve the process

• Enterprise-wide approach needed to help synch the various efforts of the Intelligence community and T&E threat M&S community

19

Enterprise Approach

Service Priorities Closed

Loop & Open Loop

Parallel EW Efforts

MTEP M&S

Standard Wrappers

Configuration Management

EW EXCOM

M&S Efforts

IC Bandwidth & Throughput

T&E Schedules

Authoritative Models

Training Needs

FME Efforts

Various Funding Efforts

Vertical Testability

Prog

ram

Tes

t Sc

hedu

les

Competing visions for

M&S

DisparateEW efforts

Stovepipe M&S

$

OutdatedModels

UNCLASSIFIED

T&E Facility Site Visit Findings

• The Working Group visited the following T&E facilities:– Electronic Warfare Integration Labs (EWIL), Pt. Mugu, CA– Electronic Combat Range (ECR), China Lake, CA (China Lake South Range)– Benefield Anechoic Facility (BAF)/ Digital Integrated Air Defense System (DIADS), 412th Test

Wing, Edwards AFB, CA– Air Combat Environment Test and Evaluation Facility (ACETEF), NAS Patuxent River, MD– Electronic Warfare Avionics Integrated Support Facility (EWAISF), Warner-Robins AFB, GA– Guided Weapons Evaluation Facility (GWEF), 96th Test Wing, Eglin AFB, FL– Joint Preflight Integration of Munitions and Electronic Systems (JPRIMES), 96th Test Wing– Integrated Demonstrations and Applications Lab (IDAL), Wright-Patterson AFB, OH– Naval Surface Warfare Center, Crane Division, IN

• Key take-aways include the need for:– Both high-fidelity and lower fidelity IR and RF models– Open-loop Electronic Warfare Integrated Reprogramming (EWIR) data– Standardized wrappers for facility model integration– Common background environment/clutter models (IR/RF)– Personnel to integrate/update threat models

20

UNCLASSIFIED

Threat M&S Working GroupSo What?

• Identified threat M&S needs (RF and IR) for T&E valued at ~$372 million (FY19-23)

– Developed an Intelligence Community validated and prioritized threat list– Documented threat M&S test capability gaps, vetted by Service T&E representatives– Developed solutions to fill the gaps; cross-walked threats to gaps to solutions– Drafted IR and RF time-phased investment plans (roadmaps)– Funding provided in the FY 2019 President’s Budget for model development

• Began improvement of organizational, functional, and technical linkages between the T&E and the Intelligence communities

• Validated the need for effective threat M&S life cycle management– Established an RF Configuration Control Board (CCB)– Initiated a threat models for T&E enterprise management approach

• Successfully executed a LIVE demonstration at the Benefield Anechoic Facility (BAF)

21

UNCLASSIFIED

Way Forward

• Near Term– Begin execution of the IR and RF threat model production plans beginning in FY 2019 – Continue collaboration with CTEIP staff for project execution/management requirements – Establish the “enterprise” process as a priority– Continue engagement with Industry, programs, Service training organizations, and other T&E

facilities

• Beyond in 2018– Continue to identify additional model development requirements– Map out additional projects required and update roadmaps as needed– Continue T&E facility engagement to identify integration needs (wrapper/common interface)– Ensure a CCB-like enterprise management process for RF is formalized– Document policy changes needed to establish an enterprise threat model management process– Continue to champion T&E and Intelligence Community cross-talk– Develop “Model 101” terms definitions (fidelity, resolution, standards, etc.)

22

UNCLASSIFIED

Questions?

23