test and evaluation matters for warfighters and we need ... · been discovered without realistic...

Dr. Catherine Warner

Science Advisor

April 2017

Test and Evaluation matters for warfighters—and we need your help to make it better

Operational testing exposed a critical failure in the F-22’s communication with its missiles

These communication issues would not have been discovered without realistic live shots

We have to think carefully about the range of realistic combat scenarios a system may face

Requirements documents are often missingimportant mission considerations

Sometimes mission outcome is subjective

Sometimes mission outcome is subjective

Survey regarding

improved situational awareness

Strongly agree

Agree

Slightly agree

Slightly disagree

Disagree

Strongly disagree

Scope of missions depend on system under test and what it brings to the fight

Scope of missions depend on system under test and what it brings to the fight

Link 16

Training events can be helpful,but the scenarios are limited

Live-Virtual Constructive seems the obvious choice… but it needs better validation

Operational Space

Range of

threat types

Threat lethality


Operational Space

Range of

threat types

Threat lethality

Modeling and

simulation


Operational Space

Range of

threat types

Threat lethality

Modeling and

simulation

live

testing

Cybersecurity’s effects on a missionare unclear

Ideally, operational testing will demonstratea new system is better than its predecessor

Ideally, operational testing will demonstratea new system is better than its predecessor

Target SAM

Operational testing provides critical information to warfighters about new systems…

Before warfighters’ lives and missions depend on them

Operational testing provide critical information to warfighters about new systems…

Before warfighters’ lives and missions depend on them

Time to correct problems

Time to restrict missions

Operationally Realistic Testing

By the early 1980s,Congress’ concerns were growing

d4c-35816.jpg (JPEG Image, 600 × 400 pixels) http://www.456fis.org/MISC/d4c-35816.jpg

1 of 1 1/20/17, 4:49 PM

Congress established DOT&E separate from the Services’ operational testing agencies

Department of Defense

Office of the

Secretary of Defense

Director,

Operational Test

and Evaluation

(DOT&E)



Office of the


Director,

Operational Test

and Evaluation

(DOT&E)

Service Operational Testing Agencies

Army Navy & Marines Air Force



Office of the


Director,

Operational Test

and Evaluation

(DOT&E)

Service Operational Testing Agencies

Army Navy & Marines Air Force

Congress

We take a scientific approach tooperational testing

Carrier variantConventional Short takeoff/vertical landing


Mission Areas Air Threat Ground Threat

Air-Surface

Strike

Destruction/Suppression of Enemy

Air Defenses

Defensive counter air

Offensive counter air

Close air support

Search and rescue



Close air support

power estimates Large detectable difference

Medium detectable difference

Small detectable difference

Number of tests0 30

1


Characterization across operational envelope –Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense

Weapons

Production

Facility


Weapons

Production

Facility

Radar

Surface to

Air Missile

Radar

Radar

Radar

Surface to

Air Missile


Weapons

Production

Facility

Radar

Surface to

Air Missile

Radar

Radar

Characterization across operational envelope –Response Variables

Lots of measures to capture:

Mission outcomes

Air to Air Performance

Air to Surface Performance

System sensor capabilities

Targeting Accuracy Striker

Striker First Track Range

Striker First Hostile Declaration Range

Striker First Shot Range

Red Air First Detection Range

Red Air First Shot Range

Striker SAM Track Time

Proportion of Valid Weapon Releases to Number

of Valid Weapon Releases Required to Meet

Mission Tasking

Proportion of Assigned Air to Surface Targets

Removed

Proportion of Striker Kill Removed

Striker to Red Air Exchange Ratio

Geolocation Find Time

Fix Time

DEAD Time

Targeting Accuracy Escort

Escort SAM Track Time

Proportion of Assigned SAM Elements Removed

Proportion of Assigned SAM Elements Engaged

Exchange Ratio

Closest Red Air Range to Strike Package

Blue Striker Encroachment Range

Escort First Track Range

Escort First Hostile Declaration Range

Escort First Shot Range

Red Air First Detection Range

Red Air First Shot Range

Proportion of Escort Blue Strikers that reach

their Weapons Release Point

Proportion of Protected Aircraft (Strikers) Not

Kill Removed

Proportion of Escort F-35 Kill Removed

Escort to Red Fighter Exchange Ratio

Two mission designs, executed in a 5th

generation scenario

Power calculations provided justification for number of trials

Power

Target location power

Variant power

Environment power (in/out of band)

Signal-to-Noise Ratio

0 2

1

.8

1

Design of Experiments supports characterization of DOD information systems*

Variability in outcomes could exist due to manyoperational factors:

Human users!

Network load

Database errors

Dynamic Environment

Memory Leaks

…

*such as financial, command and control, medical records, and logistics management

Combinatorial designs assume system is fully deterministic

Useful for finding bugs in software



Testing combination of 10

settings requires 210 = 1,024

tests!



Testing combination of 10

settings requires 210 = 1,024

tests!

Only 12 tests are needed to

test all 120 3-way interactions!

But…the outcomes are binary and completely deterministic!

Creating a combinatorial design and then augmenting up to a classical design* allows you to find problems AND fit a statistical model to explain variability

* D-optimal, fractional factorial, full factorial, etc…

Combination of widgets Y/N

Timeliness (location, network,

time of day)

DOE supports timeliness analysis across operational space

Elmendorf SAMMC

72 61 68 410

60

120

180

240

300

360

420

Morning Afternoon Morning Afternoon

Do

wn

loa

d T

ime

(s)

Operator location affected timeliness

Future Test Challenges

Modern software-defined threats pose major challenges to current and future test and evaluation

Threat capability may

change significantly with

a new SW load

Diminishes usefulness of

mission data libraries

Machine cognition is being pursued to counter advanced adaptive threats

Adapt Jamming

We need to think carefully aboutthe challenges we face in the future

Big data Workforce

Cyber AutonomySpace

test and evaluation matters for warfighters and we need ... · been discovered without realistic...

Documents