NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
1
Developmental Test and EvaluationCommittee
Status and Plans December 2013
Beth Wilson, RaytheonSteve Scukanec, Northrop Grumman
Industry Co-Chairs
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
2
DT&E Committee2013 Status
Activity Plans for 2013 Status/PlansScientific Test and Analysis TechniquesSTAT for T&E
Effective use of statistical approaches for test optimization (including DOE) with examples
White paper completePreparing journal publicationsPresented results at SE conference
Modeling & Simulation
Evaluate test as a stakeholder Joint meetings with SED meetingsPropose for 2014
ArchitectureNDIA T&E Conf DT&E panel, STAT report Conference cancelled
NDIA SE Conf DT&E tracks Joint tracks, metrics/STAT reports
Added Task Dec 2012: Metrics
Provide more information on leading indicators for T&E
Status: draft reportPlans: results at SE conference
Added Task Dec 2012: Reliability
Develop a better test strategy for reliability (collaborate with IEEE Reliability Society)
Discussed at June 2013 meetingLeads identified Nov 2013
Added Task Jun 2013:Cyber Testing
Develop recommendations for Cyber Guidelines 2.0 in 2014
Lead and team identifiedKicked off Nov 2013 (joint with ITEA)
Added Task Jun 2013:Chief Developmental Tester
Develop recommendations for industry interaction with Chief Developmental Tester role
Lead and team identified
Complete In Process Cancelled
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
3
DT&E Committee - 2014 Task PlanProjects Working Group
Proposed 2014 Tasks:• Cyber Testing Project• Chief Developmental Tester Project• Collaboration with IEEE Reliability• Joint meetings with SED Committees
• Modeling and Simulation• Architecture• Systems Security Engineering• Education and Training
Deliverables/Products• Cyber Testing Recommendations white paper• Chief Developmental Tester white paper
• Candidate KSABs for role (E&T)• Industry interaction recommendation• Tutorial on role for SE conference
• Inputs to reliability standards effort• Joint meeting notes
Schedule / Resources• Cyber Testing Draft Recommendations Feb 2014
• Weekly teleconferences started 12/3/2013• Full white paper later in 2014
• Chief Developmental Tester• Joint meeting with E&T with SED meeting• Tutorial at SE conference
• Reliability standards input for Jan/Aug meetings• Test as a Stakeholder joint meetings with SED
• Modeling and Simulation• Architecture• Systems Security Engineering
Issues / Concerns:• Industry engagement• NDIA T&E conference may be cancelled (Seattle
in March)• ITEA conference (Denver in September) may be
postponed and moved
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
4
Summary of DT&E Committee2013 Results/2014 Plans
Topic Activity
T&E
Test Optimization 2013: Scientific Test and Analysis Techniques report/articles
Cyber Testing 2013-2014: Industry recommendations for cyber testing
Chief Developmental Tester 2013-2014: Industry interaction with new role
SE
Performance Measurement 2013: Requirements Verification leading indicators report
Reliability 2013-2014: Recommendations for better reliability testing
Modeling and Simulation 2014: T&E Perspective for Modeling/Simulation products
Architecture 2014: T&E Perspective for Architecture views
Education and Training 2014: Chief Developmental Tester role
Systems Security Engineering
2014: Cyber testing guidelines connections to Program Protection Planning
Completed Current Proposed
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
5
DT&E Committee 2012-2013Statistical Test Optimization
2012 SE Conference summit/workshop thread- Tutorials on Monday 10/22- Presentations on Wednesday 10/24- Synthesis Panel on Wednesday 10/24
Follow-on Results: - DT&E Committee white paper - ITEA Journal article- SE Conference track Oct 2013
Activity Plans for 2013 Status/PlansScientific Test and Analysis TechniquesSTAT for T&E
Examples of effective use of statistical approaches for test optimization (including DOE) for an implementation framework
White paper completeSE Conference track: 2012 and 2013ITEA journal publication MORS conference Nov 2013
2012 - 2013
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
6
DT&E Committee 2012-2013Leading Indicator Metrics
2012 SE Conference workshop- Follow-on workshop with System Performance Measurement WG - Focused on Requirements Verification not addressed in first report
Potential Measures Identified: - System Maturity Level- Verification Requirement Maturity- Technical Measures and Stakeholder Need
Activity Plans for 2013 Status/PlansMetrics(with Performance Measurement WG)
Provide more information on leading indicators for T&E
Draft reportPresented results at SE conference
2012 - 2013
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
7
DT&E Committee 2012-2013Reliability Testing
Reliability Testing Focus- Software reliability standard- Cyber testing initiatives- Data for reliability growth measures
Input/Feedback Opportunities: - IEEE meetings January and August- Periodic revision periods
Activity Plans for 2013 Status/PlansReliability Testing(with IEEE)
Develop a better test strategy for reliability (collaborate with IEEE Reliability Society)
Solicit feedback on existing standardsIdentify recommendations to improve reliability testing
Project Leads: Lou Gullo, Taz Daughtrey
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
8
DT&E Committee 2013-2014Cyber Testing
Activity Plans for 2013 - 2014 Status/PlansCyber Testing Review what industry does for
cyber testing, expand focus to include interoperability
Recommendations for Cyber Testing guidelines
Project Lead: Dave DesjardinsJoint Project with ITEA
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
9
DT&E Committee 2013-2014Chief Developmental Tester
Activity Plans for 2013-2014 Status/PlansChief Developmental Tester
Review service policies for new role and industry implementations
Propose model for industry interactionWhite paperT&E/SE conference presentations
The Secretary of Defense shall require that each major defense acquisition program be supported by—‘‘(A) a chief developmental tester; and‘‘(B) a governmental test agency, serving as lead developmental test and evaluation organization for the program.
Coordinating DT&E activitiesInsight into Contractor activitiesOversee T&E activitiesInform government PM about contractor DT&E results
Project Lead: Joe Manas
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
10
Backup
DT&E Committee Results 2008-2013
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
11
Summary of DT&E Committee Test and Evaluation Efforts
Topic Activity
DT&E Committee
DoD T&E Policy Study 2006 – 2008: Workshops and Study ReportImproving T&E in the DoD Acquisition Process
Integrated Testing 2008 – 2010: Integrated Test Study NDIA presentations and tutorialITEA journal article
RFP Language for T&E 2010 – 2011: Comments for Update to OSD GuideIncorporating T&E into DoD Acquisition Contracts
Test Optimization 2012: Statistical Optimization Conference Thread2013: Scientific Test and Analysis Techniques report/articles
Cyber Testing Started 2013: Industry recommendations for cyber testing
Chief Developmental Tester Started 2013: Industry interaction with new role
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
12
Summary of DT&E Committee Systems Engineering Efforts
SED Committee Activity
SE Collaboration
Software 2009: SW T&E Summit Recommendations
System of Systems 2010 – 2011: T&E for SoS Workshop and Initiatives2012: Final Report on Best Practices Model
Performance Measurement
2012: Leading Indicators for T&E workshop
2013: Requirements Verification leading indicators report
Modeling and Simulation
2011: Effective Use of Modeling and Simulation for T&E 2012: Distributed Model-Based Testing
Reliability Started 2013: Recommendations for better reliability testing
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
13
T&E Themes
ForConferences
SEDCommittee
CollaborativeProjects
DT&E Committee Current Structure Since 2010
NDIA Systems Engineering Conference (Q4) Collaborative Projects
NDIA Test and Evaluation Annual Conference (Q1)Periodic T&E Committee-Focused Conferences
Cooperation with other NDIA Divisions
NDIA Industrial Committee on Test and Evaluation
(ICOTE)
DT&E Committee Focus: T&E initiatives aligned with SE, DT&E
T&E Themes & Issues
Project ResultsDT&E Themes
NDIATest and Evaluation
Division
NDIASystems Engineering
Division
NDIADT&E Committee
SE Themes & Issues
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
14
DoD T&E Policy Study August 2006: DT&E Committee Kickoff
Policy Study:“Improving T&E in the DoD Acquisition Process”Industry T&E policy recommendations
Workshops:August 2007January 2008
Focus Areas:1. Earlier contractor and tester involvement2. Integrated DT/OT and DT operational relevance3. Suitability
April 2008: Report Summarized Results:10 Findings15 Recommendations
2006-2008
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
15
Integrated Testing (CT/DT/OT)Implementation Framework
2008-2010
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
16
Software Test and EvaluationSoftware Summit
RFP Language
How Much Testing
is Enough?
Lifecycle and
End-to-End Software T&E
Changing Paradigms
Training & Competency ModelPolicy, Guidance, and Standards
Tools, Automation, Methodologies, Process2009
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
17
RFP LanguageIndustry
#
Master #
Reviewer Line# Comment and Rationale Recommended Input A/R/P
Industry Comments for Update:Incorporating Test and Evaluation
Into Department of DefenseAcquisition Contracts
Recommendations from SW Summit
2010-2011
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
18
Test and Evaluation forSystems of Systems
2010: Workshop2009: “Sleepless Nights” List of Issues
2010: “Sominex” Resulting Initiatives
InitiateSoS
PlanSoS
Update
EvolveSoSArch
EvolveSoSArch
ImplementSoS
Update
PlanSoS
Update
ContinueSoS Analysis
ImplementSoS
Update
PlanSoS
Update
ContinueSoS Analysis
ConductSoS Analysis
ContinueSoS Analysis
ImplementSoS
Update
DevelopSoSArch
External Environment
2011: Best PracticesWave Model
2009-2012 2012: Final Report
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
19
Effective Use of Modeling and Simulation for Test and Evaluation
Joint Meeting in August 2011• Distributed Testing, the Joint Mission
Environment Test Capability (JMETC) and the Test and Training Enabling Architecture (TENA)
• DoD M&S Community of Interest Data Management Working Group
• LVC Architecture Roadmap Implementation (LVCAR‐I) Gateways Effort Applicability to T&E
• OSD T&E Working Group • Raytheon Presentation on M&S for T&E • Potential Topics for November AMSWG
Meeting
2011
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
20
Modeling & Simulation forDistributed Testing
Benefits:• Find integration issues earlier• Test to learn in ‘safe’ environment• Protect proprietary information• Facilitate DT to OT transition• Increase performance testing range in operating environments • Support end to end studies throughout the programBarriers:• Security• Lack of persistent network• Early consideration of technical issues• Perceived value• Disconnect between the communities (M&S and T&E)Recommendations:• Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata)• Create a framework for reusing and repurposing M&S through the product model• Establish M&S as part of statistical test design
• Determine what tests are conducted to acquire data for model validation.• Fewer test events with better models.
• Recommend the use of M&S to do I&T • Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T
Joint TrackSE Conference October 2012
Joint MeetingAugust 2012
2012