lessons learned from army interoperability certification ... · lessons learned from army...

12
Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF (254) 554-1888 (254) 532-8321 x2759 RDCS Technology, INC [email protected]

Upload: others

Post on 15-Jun-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

Lessons Learned from Army Interoperability Certification Testing

Robert BoerjanCTSF Capability Set Coordinator, CTSF(254) 554-1888(254) 532-8321 x2759RDCS Technology, [email protected]

Page 2: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

Analysis of Software Block 2 (SWB 2) Test Incident Reports (TIRs)

Background: • The CTSF is the Army Interoperability Certification (AIC) agent for

LandWarNet/Battle Command systems• The Army incorporates a blocking process of multiple systems to

introduce new capability set into the Army• Software Block 2 had 50+ systems • SWB 2 was initially scheduled for 9 months of Test-Fix-Tes, AIC and

Backward Compatibility (BWC) testing – however it took over 2 years before the CIO G6 certified the block and fielding began.

Oct 06 Jan07 Apr07 Jul07 Oct 07 Jan08 Apr08 Jul08 Oct 08 Jan09 Apr09

Original Plan

TFT AIC 1 TIR Val AIC 2 RegressionBWC

First Unit Equipped

Page 3: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

Methodology

Results

BCID PCR Master Tracking

Sheet

Configuration Management

Database

TIR Database

Focused on Level 1-3 TIRs Date range is 10/06 thru 12/15/08 Did not evaluate BWC issues Segregated the TIRs into 5 Test Windows: Test-Fix-Test; AIC 1; TIR Evaluation; AIC 2; Regression 1 & 2

Evaluated the following type of data: Number of TIRs over time for entire 2-year period Number of TIRs over time for each Test Window Individual system TIRs and sub-category by severity level TIRs by Issue Category System of System (SoS) vs. System TIRs Graphic TIRs CTSF Configuration Management database of software deliveries: date and purpose of deliveries Battle Command Integration Directorate (BCID) SWB 2 Product Change Request (PCR) Master Log:

Page 4: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

Requirements Changed

Suspense date for TCM & PM certification of threads was 18 Sept 06.

Start of TFT to Dec 08, 216 approved PCR changes (count after removing PCR duplicates, archived Threads, and denied PCRs)

38% of PCRs occurred during AIC 1

Maneuver and Aviation PM/TCMs requested the majority of PCRs during AIC 1

Majority of PCRs during Validation and AIC 2 were in response to Thread and SW TIRs

High number of INTEL PCR changes during Regression due to DCGS-A replacing ASAS

6

2512

5 439 4

70

143 1

36

5315

10

1020304050607080

TFT AIC 1 TIR Val AIC 2 REG

Approved Thread PCR Requests by BOS

SANMMVRINTELFSCSSAVNADA

0%20%40%60%80%

100%

2020 9 3 5

1081 41 10 74

PCRs vs. TIRs for SWB 2

PCRs

Admin Thread TIRs + Thread TIRs

Changing Requirements Consume Resources

216 PCRs

Page 5: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

71

10

17

10

51 1 2

67

15 14

5 52

4

97

1 2 15

1

20

2 1 13

1

26

51

31

0

10

20

30

40

50

60

70

80

SW Error Thread TIR Class Error Admin Thread TIR

Requirements Standards Architecture Hardware

TFT AIC 1 Validation AIC 2 Regression

SWB 2 TIRS by CategoriesSW Error: Software failure that was fixed with a SW patchThread: System’s requirement not accurately depicted by the threadTIR Class Error: A TIR that either should not have made it out of the DAG; post- score discovery issue was operator induced; or post-score discovery of an inaccurate assessment. Admin Thread TIR: TIRs levied against the thread proponent to ensure thread issue is corrected. Requirements: Change in requirements as directed from TCMs; for example System A received TIR for inability to display graphic from System B, and TCM determined System A did not have a requirement to display graphic from System B.Standards: Conflict between two standards – for example USMTF & VMFArchitecture: Data product issues or missing systems from a particular echelonHardware: Hardware failures

193

39 34

18 11 11 10 3

319 Total TIRs

Page 6: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

Total Level 1-3 TIRs for SWB 2

14

31

2420 18

10

39

56

17

10 12

4 610

7 5

13

2 2

116

20

10

20

30

40

50

60

10/1/

2006

11/1/

2006

12/1/

2006

1/1/20

072/1

/2007

3/1/20

074/1

/2007

5/1/20

076/1

/2007

7/1/20

078/1

/2007

9/1/20

0710

/1/20

0711

/1/20

0712

/1/20

071/1

/2008

2/1/20

083/1

/2008

4/1/20

085/1

/2008

6/1/20

087/1

/2008

8/1/20

089/1

/2008

10/1/

2008

11/1/

2008

SWB 2 TIRs by Test WindowsTFT

AIC 1TIR Validation

AIC 2Regression 1 …

Test Windows

Total of 319 LvL 1-3 TIRs 6.9% of TIRs were LvL 1 81.5% of TIRs were LvL 2 11.6% of TIRs were LvL 3 5 systems out 31 Systems with TIRs accounted for 52% of TIRs

SWB 2 Remove

d off Test Floor

Page 7: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

14

31

24

2018

10

39

56

17

1012

46

107

5

13

2 2

11

6

2

0

10

20

30

40

50

60

Unexpected Results of AIC & Regression Testing

Expected Outcome

Actual Outcome

There was a only a 1.6% decrease in LvL 1-3 TIRs going from the TFT (117 TIRs) to the AIC 1 (112 TIRs)

The TIR Validation Event was to validate PM SW patches corrected AIC 1 TIRs. In addition to closing TIRs, there was an additional 26 TIRs scored

There was a .6% increase in TIRs from TIR Validation to AIC 2 (28 TIRs)

There was a 2.5% increase in TIRs from AIC2 through Regression Windows (36 TIRs)

Last 3 Windows averaged 30 TIRs per Test Window

SWB 2 Remove

d off Test Floor

TFTAIC 1

TIR ValidationAIC 2

Regression 1 …

Test Windows

Page 8: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

SWB 2 TIRs by Severity Level

10 72 3 0

05

1015

LvL 1 TIRs

LvL 1 TIRs

99 101

20 17 23

050

100

TFT AIC 1 TIR Validation

AIC 2 Regression

LvL 2 TIRs

LvL 2 TIRs

84 4

813

05

1015

TFT AIC 1 TIR Validation

AIC 2 Regression

LvL 3 TIRs

LvL 3 TIRs

10 LvL 1 TIRs during TFT were within expectations because it was first time systems evaluated within a robust integrated architecture

7 LvL 1 TIRs for AIC 1 not expected and resulted in inability to fully test systems’ capability

AIC 1 accounted for 39% of all LvL 2 TIRs

The TIR Validation window had 20 new LvL 2 TIRs

The 17 LvL 2 TIRs from AIC 2 were corrected during Regression, but an additional 16 TIRs were scored ( NOTE: 4 OOC systems first test against SWB 2 accounted for an additional 7 LvL 2 TIRs)

LvL 3 TIRs actually increased during AIC 2 and Regression (NOTE: A LvL 3 TIR is a LvL 1 or 2 TIR that has been reduced to a LvL 3 with a valid Technical Bulletin.)

Page 9: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

Graphics

Requirements9%

Standards1%

SW Error74%

TIR Class Error16%

SWB 2 Graphics TIRs

1

18

118 8

3

1922

14

1 1 1 105

10152025

SWB 2 Graphic TIRs over Time SWB2 had 108 Graphic related TIRs out of 319 TIRs

Graphic Software Error TIRs accounted for 41% (80 out of 193) of all Software Error TIRs for SWB 2

74% of Graphic TIRs were software errors (80 out of 108) – Systems delivered software that fixed issue

TIR Class Errors result of operators building incorrectly and systems configured incorrectly

Graphic TIRs became a non-factor after AIC 1 window

Not a new issue – SWB 1 had two Graphic Summits to work graphics

TFTAIC 1

TIR ValidationAIC 2

Regression 1 & 2

Test Windows108

Page 10: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

Software Deliveries 43 Systems turned in software or software

patches after the TFT Window through the first week of AIC 1 Window.

16 of those systems contributed 88 LvL 1-3 TIRs scored during AIC 1 Window

Fire Support systems and Aviation systems did not participate in the TFT event. 10 LvL 1-2 TIRs are attributed to Aviation assets and issues with FBCB2 Operation Center (OPS CNTR) and Fire support.

45 software deliveries to CTSF CM for inclusion in TIR Validation from Nov 07 thru 01 Feb 08. 12 of the SW deliveries were multiple drops from 6 systems. 7 of the systems accounted for 23 out of the 26 TIRs

16 systems delivered software during AIC 2. 6 of the systems accounted for 17 LvL 1-3 TIRs

85

11

4 3 47

10

5

10

15

4-Mar-07 14-Mar-07 24-Mar-07 3-Apr-07 13-Apr-07 23-Apr-07

PM SW Deliveries pre-AIC 1

# SW Deliveries per Date

10

39

3 26 5 7

0

20

5-Oct-07 25-Oct-07 14-Nov-07 4-Dec-07 24-Dec-07 13-Jan-08 2-Feb-08 22-Feb-08

# SW Deliveries During TIR Validation

# SW Deliveries per Date

1 14 6

3 4

05

10

2-Feb-08 17-Feb-08 3-Mar-08 18-Mar-08 2-Apr-08 17-Apr-08 2-May-08 17-May-08

# of SW Deliveries During AIC 2

# of SW Deliveries per Date

SoS Maturation is required after systems deliver software

Page 11: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

SoS vs. System TIRs 74% of LvL 1-3 Software TIRs were SoS TIRs

37% of LvL 1-3 Software TIRs found in AIC 1 were System TIRs.

9 of the systems, with AIC 1 TIRs delivered SW for the TIR Validation Test and received additional TIRs. 5 out of the 9 systems had TIRs that were System TIRs

0 10 20 30 40 50 60 70

Regression

AIC 2

TIR Validation

AIC 1

TFT

8

6

5

25

10

28

22

4

42

61

SoS vs. System TIRs

SoSSystem

1

2

1

2

SoS Events find both SoS and system errors

Page 12: Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army Interoperability Certification Testing Robert Boerjan CTSF Capability Set Coordinator, CTSF

Summary

SoS Interoperability development is a process that is dependent upon:– Stable Requirements– Software Maturity– SoS Integration Capability

Goal is a disciplined and repeatable process