1

Planning and Executing an Integration Test Strategy for a Complex Aerospace System

Mr. Derrick Hinton

Principal Deputy Director, Test Resource Management Center

Complex Aerospace Systems Exchange

September 11, 2012

Pasadena, CA

2

Secretary of DefenseHonorable Leon

Panetta

Deputy Secretary of Defense

Honorable Ashton Carter

Director, OperationalTest & Evaluation

Honorable J. Michael Gilmore

Under Secretary of Defense for Acquisition, Technology & Logistics

Honorable Frank KendallPrincipal Deputy, vacant

Assistant Secretary of Defense for Acquisition

(ASD(A)) Honorable Mrs. Katrina

McFarland

Assistant Secretary of Defense for Research

& Engineering

Honorable Zachary Lemnios

Deputy Assistant Secretary of Defense

for Systems Engineering

Mr Stephen Welby

Deputy Assistant Secretary of Defense for Developmental Test &

Evaluation

Mr Ed Greer

Serves concurrently asDirector, TRMC

Director, Test Resource

Management Center

Mr Ed Greer

Section 231 of the 2003 NDAA established TRMC as a DoD Field Activity

TRMC OrganizationOffice of the Secretary of Defense

3

Deputy Director,

Corporate

Operations

DASD(DT&E)

Dir, TRMC

PD, TRMC

Deputy Director,

T&E Range Oversight

Air Force

Range

Oversight

Navy

Range

Oversight

Army

Range

Oversight

Agency

Range

Oversight

Strategic

Planning

Coordinator

International

Facilities

Coordinator

Deputy Director,

Test Capabilities

Development

CTEIP

Deputy Director,

Technology

Development

T&E/S&T Program

Deputy Director,

Interoperability

JMETC Program

Cyberspace,

Interoperability &

Distributed Testing

T&E Budget

Cert

Coordinator

TRMC OrganizationTRMC Organizational Structure

4

“Bob Stump” NDAA for FY03 (PL107-314)

Sec 231: Establish a DoD Test Resource Management Center (TRMC) as

a Field Activity under USD(AT&L) that will:

Complete and maintain a Strategic Plan for T&E Resources

Review and provide oversight of proposed DoD T&E budgets

Certify Service and Defense Agency T&E proposed budgets

Administer:

• Central Test and Evaluation Investment Program (CTEIP)

• Test and Evaluation/Science and Technology (T&E/S&T)

Concern about managing the test infrastructure – 1999 and 2000 studies.

Recognition that the right range infrastructure resources enable and support

adequate T&E of weapon systems.

Need to assess duplicative test capabilities as essential or unnecessary costs.

These identified concerns and needs resulted in the 2002 legislation:

TRMC BackgroundNeed for Test Range Oversight

Codified in

Title 10

U.S. Code

§ 196

5

JMETC

Cyber

TRMC BackgroundAdditional Responsibilities

“Effective October 1, 2012, I designate the Test Resource Management Center (TRMC) as

the responsible organization for the National Cyber Range (NCR), as part of the TRMC mission to

ensure adequate and available range capabilities for the Department.”

-- Mr. Frank Kendall USD(AT&L)

Memorandum dated August 28, 2012

Subject: National Cyber Range Management Transition

• The 2005 Program Decision Memorandum II directed the stand-up of the Joint Mission

Environment Test Capability (JMETC) Program under the Office of the Under Secretary of

Defense for Acquisition, Technology, and Logistics (OUSD [AT&L]) to provide the test

infrastructure necessary for conducting Joint distributed test events

• USD(AT&L) assigned responsibility for execution to the Director, TRMC

• The JMETC Program Office stood up in October 2006

PDM II

Nov 30, 2005

USD(AT&L)

Memo

Aug 28, 2012

6

Our Cyberspace T&E Problem

Cyberspace vulnerabilities exist in DoD weapons systems

Inadequate environments for conducting

cyberspace T&E and experimentation

Participant “stove pipes” delay

integration of cyberspace capabilities

A persistent, operationally realistic cyberspace environment with:

Interoperability Standards that break down “stove pipes”

Rapidly integrated Live, Virtual, & Constructive capabilities

Common test & experimentation tools and applications

DoD

Needs

Building a federated cyberspace environment

is costly and time consuming

7

Cyberspace T&E within the Acquisition Process

HardwareFabrication

Component/InterfaceDefinition

Allocated Functions &Performance Requirements

Requirements Definitions (JCIDS) Transition

RequirementsAnalysis

LogicalAnalysis

DesignSolution

Integration

Verification

Validation

Implementation

Tech Data &Training Pubs

SoftwareCreation/Coding

DetailedDesign

ConfirmationAudits

Realization &Assessment

Decomposition& Definition

InputOutput to Next Phase of

Development or Lifecycle SupportOperational Need

Measures of Effectiveness/Suitability

SystemMeasures of Performance

Element DesignCriteria

Spec

ialt

y En

gin

eeri

ng

Dev

elo

pm

ent/

De

sign

En

gin

eeri

ng

Dec

isio

n

Au

tho

rity

Identify Cyberspace

Measures

Cyberspace

Vulnerability

Assessments

Component Level Testing

to the “chipset” level

Component

Characterization

Integrated

Component Testing

DT&E Verification

in Realistic

Environment

OT&E Validation

against actual

Cyber Threats

“V” Chart Source: DAU Acquisition Community Connection

Identify

cyberspace

System and

sub-system

Dependencies

Test System of

Systems

Mitigation of

Cyberspace

Threats

8

Tenets of “Good” Government and Industry Systems Integration Testing

• Derives from Mission-Focused

Requirements & Test Plans

• Combines a Tiered Management

Structure & Accountability

• Utilizes Agile Development Concepts

Realistic MissionTest Environment

• Leverages Existing Government and Industry Test Capabilities

• Employs Interoperability Standards that Enable Rapidly

integrated Live, Virtual, & Constructive Capabilities

• Shares common Test & Experimentation Tools and Applications

• Unifies approach in conducting both Cyberspace and

“Traditional” Testing

• Applies Effective Knowledge Management Techniques &

Structures

DoD Acquisition Infrastructure Enterprise

9

Army HWILs Navy

HWILs

AF HWILsDISA

HWILs

DISA SILs

Army SILs

NavySILs

AFSILs

AFRanges

ArmyRanges

NavyRanges

AFISTFs

NavyISTFs

Hardware-in-the-LoopTest Laboratories

SystemIntegration

Laboratories

Installed SystemTest Facilities

Open-Air Test Ranges

Realistic MissionTest Environment

Enterprise Approach mitigates duplication, improves efficiency and reusability, and optimizes long-range improvements and modernizations across the Department

Enterprise Approach Benefits:

Mitigates duplicative, inefficient, and “stovepipe” technical solutions

Fosters standards & common test procedures improving data quality

Provides confidence in out-year availability to acquisition programs

Defines cost model for usage Optimizes

modernization/investment plans to corporate priorities

Enhances reusability across DoD

TRMC Mission is to ensure allDoD Test Infrastructure is operated,

improved, and sustained to meet current & future DoD requirements

Persistently connected via JMETC

and Joint IO Range

TENA-enabled Interoperability

TENA: Test and Training Enabling Architecture

Industry Test Facilities

IndustryHWILs

Industry SILs

IndustryISTFs

IndustryLabs

10

Keys to an Effective Knowledge Management Approach

• Trusted processes across government and industry that identify

acquisition problems sooner rather than later

– Result: Develops mutual trust in testing products

• Accessibility of knowledge & data to legitimate users

– Result: Efficiencies gained through reducing the amount of testing

required to obtain knowledge

• Discoverability of knowledge & data obtained over time

– Result: Enables data scientists to leverage data mining and analytical

tools that rapidly answer questions and discover “unknown unknowns”

• Availability of knowledge through common tools &

technologies – Example: Cloud storage and services

– Result: Efficiencies gained from readily available & shared technology,

hardware, software, and processes

Primary Goal of T&E: Discern Knowledge from Basic Measurement Data

11

BACKUP

12

TRMC ResponsibilitiesInvestment Programs: T&E/S&T

Program Goal: To exploit new technologies and expedite

their transition from the laboratory to the T&E community

Program Philosophy

• Work closely with test capability developers (e.g., CTEIP projects, Service

Range I&M projects) to mature and reduce risk in new test technologies

• Leverage heavily from ongoing efforts in industry, academia, and the

DoD S&T community (DARPA, ONR, ARL, AFRL, etc.)

Current T&E/S&T Test Technology Areas

• Cyberspace Test

• Directed Energy Test

• Electronic Warfare Test

• High Speed Systems Test

Test & Evaluation Science & Technology Program

Technology Development

• Net-Centric Systems Test

• Unmanned & Autonomous Systems Test

• Advanced Instrumentation Systems Technology

• Spectrum Efficient Technology

13

Established in FY02

Joint DDR&E / DOT&E Initiative

Transitioned to TRMC in FY05

RDT&E Budget Activity 3 funds

Purpose

High Risk / High Payoff R&D for Testing

Foster technology transition to major DoD test ranges

Risk reduction for test capabilities developments

FY 2012

80 ActiveProjects

Annual Broad Agency Announcements (BAAs) Academia

Industry

Government Laboratories

Tri-Service working groups Validate requirements

Evaluate proposals

Facilitate technology transition

Central Oversight – Distributed Execution

High Speed

Systems

13 Active Projects

Current Test Technology Areas

Directed Energy

17 Active Projects

Cyberspace

2 Active Projects

Net-Centric Systems

7 Active Projects

Electronic Warfare

16 Active Projects

Unmanned &

Autonomous Systems

3 Active Projects

Advanced

Instrumentation

8 Active Projects

Shaping Technology into Tomorrow’s T&E Capabilities

Spectrum

Efficiency

14 Active Projects

Mission: Develop Technologies Required to Test Future Warfighting Capabilities

TRMC ResponsibilitiesInvestment Programs: T&E/S&T

14

• Develops high priority test capabilities that resolve

Joint / multi-Service requirements shortfalls

• Promotes common test infrastructure and standard test

methodologies

• Minimizes duplicative engineering development and life

cycle support costs

• Partnership with Services and Agencies

– Service T&E Executives prioritize needs

– OSD centrally manages funding

– Service field activities lead development

TRMC ResponsibilitiesInvestment Programs: CTEIP

Central Test and Evaluation Investment Program

Test Capability Development

15

Resource Enhancement

Project

(REP)

Joint Improvement

& Modernization

(JIM)

Threat Systems

Project

(TSP)

• 3-5 year requirement horizon

• EMD of Major Test Capabilities

• Must address joint requirements

• Services & Agencies budget for O&M over Life-Cycle of delivered

capabilities

• $110-120M/year

• 1-2 year requirement horizon

• EMD of target capabilities

• Address shortfalls in threatsystems representation

• Coordinated with DOT&E

• $3-5M/year

• 1-2 year requirement horizon

• EMD of instrumentation needed to address an emergent requirement

• Must address OT shortfalls

• Coordinated with DOT&E

• $18 -20M/year

Initiated DEPSECDEF – 9 November 1988

Established in FY91 by Congress

RDT&E Budget Activity 4 funds

Purpose

Have multi-Service utility

Be developmental

Be non-procurement

Mission: Develop or Improve Major Test Capabilities that have Multi-Service Utility

11 TSP, 19 JIM, 13 REP = 43 Projects

T&E Master Plan (TEMP) References

MILSATCOM Atmospheric Scintillation Simulator

AEHF survivability and effectiveness testing in realistic

atmospheric environments

Ground Mounted Seeker Simulation

Advanced open-air SAM simulator to support IDECM Block

IV effectiveness testing

Prec Tgt Sig - Performance Mover (PTS-RPM)

Radar reflective/mobile targets for Gray Eagle UAS testing

FY2012

43 Active

Projects

TRMC ResponsibilitiesInvestment Programs: CTEIP

16

• Provide a persistent and robust infrastructure to integrate Live, Virtual, and

Constructive systems for test and evaluation in a Joint Systems-of-Systems

environment

– Persistent connectivity, integration software, tools, reuse repository, technical expertise

• Supports distributed testing across the acquisition life-cycle

– Address SoS interoperability issues early

– Generate adequate numbers of systems under test

– Create adequate “supporting cast” of systems

– Link the right threat types, fidelity, and density for live testing

JMETC Users Group

• Focus on tech requirements & solutions relevant to current & future distributed testing needs:

• Identify infrastructure requirements, use cases, and opportunities to collaborate

• Identify, investigate, and resolve issues

• Discuss available solutions, tools, and techniques; share lessons learned

• For the Schedule and Location of next quarterly meeting: WWW.JMETC.ORG

TRMC ResponsibilitiesInvestment Programs: JMETC

FY2012

70+ Distributed

Event Activities

Joint Mission Environment Test Capability

Interoperability

17

TRMC ResponsibilitiesInvestment Programs: JMETC