1 iv&v facility nasa software iv&v facility metrics data program june 5, 2003 mike chapman...
TRANSCRIPT
![Page 1: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/1.jpg)
1
IV&V Facility
NASA Software IV&V Facility
Metrics Data Program
June 5, 2003
Mike Chapman [email protected] 304-367-8341Pat Callis [email protected] 304-367-8309Ken McGill [email protected] 304-367-8300
![Page 2: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/2.jpg)
2
IV&V FacilityThe Metrics Data Program Goal
Establish a centralized repository that provides consistent, Establish a centralized repository that provides consistent, fully-involved fully-involved software product datasoftware product data across multiple domains: across multiple domains:
•To improve the effectiveness of software assuranceTo improve the effectiveness of software assurance•To improve the effectiveness of software researchTo improve the effectiveness of software research•To improve the ability of projects to predict software errors To improve the ability of projects to predict software errors
early in the lifecycleearly in the lifecycle
![Page 3: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/3.jpg)
3
IV&V Facility
Data, Data Everywhere: The Data Drought
There is very little fully-involved software product data…There is very little fully-involved software product data…
Error data associated to smallest functional unit Error data associated to smallest functional unit
Requirements traced through the design to the smallest Requirements traced through the design to the smallest functional unitfunctional unit
……available to those who need it.available to those who need it.
![Page 4: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/4.jpg)
4
IV&V FacilitySARP PI Survey
To what degree has a lack of software defect data from actual NASA projects impacted your SARP funded research?
- Greatly (9). Lack of software defect data from actual NASA project has seriously hampered my research
- Moderately (8). It would be nice to have more (or some) real project data, but I have found non-NASA data sources or other workarounds.
- Not significantly (5). My project has not been impacted because I either have all the data I need or I don't need software defect data
Note: The totals are in parentheses.
![Page 5: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/5.jpg)
5
IV&V Facility
The Lack of Data from a Project Perspective
Error data associated to smallest functional unitError data associated to smallest functional unit•There is little value in this activity for the projectThere is little value in this activity for the project
Requirements traced through the design to the smallest Requirements traced through the design to the smallest functional unitfunctional unit
•Projects only need to trace requirements to the executable Projects only need to trace requirements to the executable (CSC level or above)(CSC level or above)
Fully-involved data made available to the research communityFully-involved data made available to the research community•Vulnerability of the programVulnerability of the program•Proprietary issues Proprietary issues
![Page 6: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/6.jpg)
6
IV&V Facility
The Quest for the Holy Grail
Requirements
Design
Code(smallest functional unit)
Traceability
ProblemReports
dev, test, user, maintenance
Regardless of the development model the following is needed:
Association to the smallest functional unit.
(Many to one issue.)
![Page 7: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/7.jpg)
7
IV&V Facility
Recruitment of Project Data
Existing repository data Existing repository data
Error Data Enhancement ProgramError Data Enhancement Program
Identifier Language LOC Domain Error Data
JM-1 C 315K Real-Time 8 years
KC C++ ~750K Data System 5 years
CM-1 C 20K Instrument 2 years
CM-4 C ~40K Flight N/A
![Page 8: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/8.jpg)
8
IV&V Facility
Error Data Enhancement Program
The goal of the enhancement effort is to successfully recruit projects to work with the MDP to provide fully-involved software product data – the Holy Grail.
The MDP team will provide:Requirements analysis and traceability supportConfiguration Management Support (Error Tracking)Metrics generation and analysisDatabase and web supportMachine learning analysis
![Page 9: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/9.jpg)
9
IV&V Facility
Project Error Data
MDP Web Site
ClearQuestInterface
Firewall
Stored Metrics and Error Data
ClearQuest
Project SecurityProject1………………….Project n
Project Metrics and Error Data
MDP Repository
Queries
Sanitized Data
Sanitized Data
Other Project Data
![Page 10: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/10.jpg)
10
IV&V Facility
Agency benefits:•The improved ability to predict error early in the lifecycle•The improved ability to assess the quality of the software
The research community benefits:•Availability of quality error and metric data•Availability of a support team for data needs
Participating projects benefit:•Additional metrics analysis•Additional error analysis•Problem tracking tool•Other support such as requirements traceability
Benefits
![Page 11: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/11.jpg)
11
IV&V Facility
Where to Find More Information
mdp.ivv.nasa.gov
![Page 12: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/12.jpg)
12
IV&V Facility
Site Metrics
Web Site activity for 3 months:Web Site activity for 3 months:
596 hits596 hits46 accounts46 accounts146 logins146 logins85 downloads of data85 downloads of data
![Page 13: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/13.jpg)
13
IV&V Facility
Special Data Requests
Five time stamps of KC-2 dataFive time stamps of KC-2 dataSanitized activity fields of JM-1 dataSanitized activity fields of JM-1 dataError Reports from CSCI of KCError Reports from CSCI of KC
Note: Five papers have been written from the repository data so far.
![Page 14: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/14.jpg)
14
IV&V Facility
Current Data Request
CM-1 requirements data and associated errors - JPLCM-1 requirements data and associated errors - JPLCM-1 data including metrics – Dolores Wallace SATCCM-1 data including metrics – Dolores Wallace SATCKC semantic metrics generation – Letha Etzkorn UA – HuntsvilleKC semantic metrics generation – Letha Etzkorn UA – HuntsvilleJM-1 time stamp data (five sets) – Tim Menzies WVUJM-1 time stamp data (five sets) – Tim Menzies WVU
![Page 15: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/15.jpg)
15
IV&V Facility
A Study
Metric Combination % of Errors % of Code Probability of Failure (%)
l >= 0.047015525 50% 34% 71%
loc >= 9.197AND l >= 0.066 30% 20% 37%
•v(G) – cyclomatic complexity – independent linear paths•ev(G) – essential complexity – unstructured constructs•e – programming effort – mental effort•l – (2/u1)*(u2/N2)-program level – the level at which a program can be understood
Metric Combination Probability of Module Error Probability of Failure (%)
% of Severity One Errors
v(g)>45.9 & e>=1340053 80% 0% 50% (8/16)
v(g)>45.9 & loc>= 150 73% 0% 33% (23/69)
![Page 16: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/16.jpg)
16
IV&V Facility
Background Slides on Metrics
![Page 17: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/17.jpg)
17
IV&V Facility
v(G): cyclomatic complexity = # path(ish)s = edges-nodes+1
m = # one entry/one exit sub-graphs
ev(G): essential complexity = v(G) – m
iv(G): design complexity (reflects complexity
of calls to other modules)
Halstead: programmers read code.
Too many “words” error
Mccabe: paths between “words”Twisted paths error
McCabe Metrics
![Page 18: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/18.jpg)
18
IV&V Facility
e.g. 2+2+3N1 = 3N2 = 2µ1 = 2µ2 = 2µ1’ = 2(ish)µ2’ = #input parameters
µ1µ2
N1N2
µ = µ1+ µ2N = length = N1+N2V = volume = N*log2(µ)V’ = (2+ µ2’)*log2(2+ µ2’)
Could be found via simple tokenizers
Halstead Metrics
L = level = V’/VD = difficulty = 1/LL’ = 1/DE = effort = V/L’T = time = E/18
![Page 19: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/19.jpg)
19
IV&V Facility
Operators
! % & * + , -
. / ; < > ? ^ |
~ = >= <= == != >>
<< += -= *= /= %= &=
^= |= >>= <<= && || ++
-- -> return size of enum struct delete
continue new break union
if (…) … if (…) … else
switch (……) default:
case <label>: goto <label>
do … while (…) while (…) …do
for (… ; …; …) this->
[ ] { }
<function name>( ) ( ) in any other cases not covered
… ‘?’ … ‘ ‘: …
![Page 20: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/20.jpg)
20
IV&V Facility
Operands
Variables and identifiersConstants (numeric literal/string)Function names when used during calls
![Page 21: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/21.jpg)
21
IV&V Facility
Module LOC ER ED* 6 mth 1 yr 2 yrCSC1/mod1 59 6 101.7 3 4 5
CSC2/mod1 190 6 31.58 2 4 5
CSC1/mod9 40 5 125 1 2 3
CSC7/mod27 47 3 63.83
CSC4/mod10 287 2 6.9 1 2 2
CSC6/mod51 482 10 20.8 1 4
*ED = ER/KLOC
Error Metrics
![Page 22: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/22.jpg)
22
IV&V Facility
OO Metrics
Number of Children (NOC) – number of sub-classesNumber of Children (NOC) – number of sub-classesDepth – level of class in the class hierarchyDepth – level of class in the class hierarchyResponse for Class (RFC) – number of local methods plusResponse for Class (RFC) – number of local methods plusthe number of methods called by local methods (>100)the number of methods called by local methods (>100)Weighted Methods per Class (WMC) – sum of the complexitiesWeighted Methods per Class (WMC) – sum of the complexitiesof the methods (>100)of the methods (>100)Coupling Between Object Classes (CBO) – dependency on classesCoupling Between Object Classes (CBO) – dependency on classesoutside the class hierarchy (>5)outside the class hierarchy (>5)Lack of Cohesion of Methods (LOCM) – the use of local Lack of Cohesion of Methods (LOCM) – the use of local instance variable by local methods instance variable by local methods
![Page 23: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/23.jpg)
23
IV&V Facility
ARM Metrics
Weak Phrases (adequate, be able to) – clauses that cause uncertaintyWeak Phrases (adequate, be able to) – clauses that cause uncertaintyIncomplete (TBD, TBR) – Words and phrases that indicate the Incomplete (TBD, TBR) – Words and phrases that indicate the spec may not be fully developedspec may not be fully developedOptions (can, optionally) – Words that give the developer latitudeOptions (can, optionally) – Words that give the developer latitudeImperatives (shall, may, will, should) – Words that are explicitImperatives (shall, may, will, should) – Words that are explicitContinuances (below, as follows, and) Extensive use of continuances Continuances (below, as follows, and) Extensive use of continuances can indicate complex requirements.can indicate complex requirements.Directives (for example, figure, table) – examples or illustrationsDirectives (for example, figure, table) – examples or illustrationsLines of text (size)Lines of text (size)Document structure (levels) Document structure (levels)
![Page 24: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/24.jpg)
24
IV&V Facility
Program Background Slides
![Page 25: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/25.jpg)
25
IV&V Facility
Problem Report Fields
Error Identifier: (Alpha-numeric) Headline: (text- short description)
Submitted-on: (Date yymmdd) Severity: (1 thru 5)
Status: (NVTCDARMB) System Mode: (operations versus test)
Request type: (Problem or enhancement) Problem-type: (requirements, design, source code, COTS, documentation, hardware,etc)
Problem Mode: (PR or Action Item)
Assigned-to: CCB Approval: (Date)
Impacts-csci: (high level design element) Impacts-csc: CSC
Impacts-class/file: class/file Impacts-method/function/module: Method/function/module
Impacts – Requirement Impacts – design element
Resolution: source code, COTS, documentation, not a bug, unreproducible
Problem: (text) Analysis: (text)
Resolution: (text) Closed-on: (Date)
![Page 26: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/26.jpg)
26
IV&V Facility
Project Non-Specific - Universal
Error Identifier: (Alpha-numeric)*Submitted-on: (yymmdd)Severity: (1 thru 5)Status: (NVTCDARMB)Mode: (operations versus test)Request type: (Problem or enhancement)Problem-type: (requirements, design, source code, COTS, documentation,etc)Impacts-csci: (high level software design element)*Documents: (What documents are affected?)CCB Approval: (Date)Resolution: source code, COTS, documentation, not a bug, unreproducibleVerified-on: (Date)Closed-on: (Date)
* May need sanitized.
![Page 27: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/27.jpg)
27
IV&V Facility
Project Non-specific - Expanded
Impact data: Costing Data:Impacts-csc: CSC*Impacts-class/file: class/file*Impacts-method/function/module: Method/function/module*Recommend-change: source code Process data:How-found: (e.g., Acceptance Test)When-found: (e.g., Acceptance Testing)Analysis-due: 020322Assigned-Eval-on: 020322Assigned-Implement-on: 020323Implement-due: 020325Fix-date: 020325Fixed-on: (date)In-Test-on: 020325Test-name: (Numeric id)*Test-system: (hardware)Verify-date:Merge-build-id:Deferred-on:Build-name: (Alpha-numeric identifier)Patch-rel-name: Patch-rel-date: (date)Automated history entries: (Alpha-numeric) * May need sanitized
Cost: (high, medium, low)Est-fix-hours: Est-fix-date: (date)Est-Num-SLOC: Rev-fix-time: Rev-fix-date: (date)SLOC-Type:SLOC-count: Fix-hours:
Miscellaneous Data:Operating-system:Priority: (High, Medium, Low)Enhancement: (Y or N)Workaround: YIteration: (Version bug identified in)
![Page 28: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/28.jpg)
28
IV&V Facility
Project Specific - Universal
Headline: (text – short problem description)Problem: (text – expanded problem description)Analysis: (text)Resolution: (text)Closure: (text)Submitter-id: Assigned-to: Closer-id:
Question: Can Project Specific data be sanitized?
![Page 29: 1 IV&V Facility NASA Software IV&V Facility Metrics Data Program June 5, 2003 Mike Chapman chapman@ivv.nasa.gov 304-367-8341chapman@ivv.nasa.gov Pat Callis](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649e115503460f94afcf7a/html5/thumbnails/29.jpg)
29
IV&V Facility
Project Specific - Universal
Headline: (text – short problem description)Problem: (text – expanded problem description)Analysis: (text)Resolution: (text)Closure: (text)Submitter-id: Assigned-to: Closer-id:
Question: Can Project Specific data be sanitized?