1.basic metrics and measurement

8/6/2019 1.Basic Metrics and Measurement

1/19

SQE Vol. 1 R.L. Probert, 2001

MODULE 1: Quality,

Process and Management

SOFTWARE QUALITY

MANAGEMENTManagement Objectives

Measurements & Metrics

(Ch. 3, 4. Of Kan)

Process Metrics

Product Metrics

Cost of Quality

INDUSTRIAL SCALE TESTING

Best Principles

Best PracticesBest Automation Strategies


2/19


Management Objectives

Optimize product quality Development cost

Time to market

Personnel development


3/19


An Example of the Waterfall Process Model

Requirements

Gathering

and Analysis

Architectural

Design

HLD/IO LLD/I1 CODE/12 UT

Integration

Subsystem Test

RAISE System Test

Early Customer

Feedback and Beta

Test Programs

Release

HLD: High-level design

IO: HLD Inspection

LLD: Low-level design

I1: LLD InspectionI2: Code Inspection

UT: Unit Test

Raise: Reliability, Availability, Install,

Serviceability, and Ease of use


4/19


Development Process

Requirements Gathering &

Validation Design

Design reviews and inspections

Code

Code inspection

Debug and development test

Integration (of components and

modules to form the product) Formal machine testing

Early customer programs


5/19


Possible Testable

Hypotheses For software projects, the higher

the percentage of the designs andcode that are inspected, the lower

the defect rate that will beencountered at the later phase offormal machine testing.

The more effective the designreviews and the code inspections as

scored by the inspection team, thelower the defect rate that will beencountered at the later phase offormal machine testing.

The more thorough the developmenttest (in terms of test coverage)done before integration, the lowerthe defect rate that will beencountered at the formal machinetesting phase.


6/19


Four Levels of Measurement

Nominal scale Ordinal scale

Interval scale

Ratio scale

Note that the measurementscales are hierarchical


7/19


Basic Measures

Ratio Proportion

PercentageRequirements bugs were 15% of the total,

design bugs were 25% of the total, codingbugs were 50% of the total, and other bugs

made up 10% of the total.

Vs.

The project consists of 8 thousand lines ofcode (KLOC). During its development a total

of 200 defects were detected and removed,giving a defect removal rate of 25 defectsper KLOC. Of the 200 defects,

requirements bugs constituted 15%, designbugs 25%, coding bugs 50%, and other bugs

made up 10%.


8/19


Percentage Distributions ofDefect Type by Project

Type of Defect Project A Project B Project C(%) (%) (%)

Requirements 15.0 41.0 20.3

Design 25.0 21.8 22.7

Code 50.0 28.6 36.7

Others 10.0 8.6 20.3

Total 100.0 100.0 100.0(N) (200) (105) (128)

Of the total 20 defects for the entire project of 2 KLOC,

there were 3 requirements bugs, 5 design bugs, 10coding bugs, and 2 others.


9/19


Percentage Distributions ofdefects Across project by Defect

Type

Project

Type of Defect A B C Total (N)

Requirements (%) 30.3 43.4 26.3 100.0 (99)

Design (%) 49.0 22.5 28.5 100.0 (102)

Code (%) 56.5 16.9 26.6 100.0 (177)

Others (%) 36.4 16.4 47.2 100.0 (55)


10/19



11/19


Reliability and Validity

Reliability refers to the consistency of anumber of measurements taken using thesame measurement method on the samesubject Reliability can be expressed in terms of the

size of the standard deviations of the

repeated measurements. Index of variation (IV) is used

IV = Standard deviationMean

Validity refers to whether themeasurement or metric really measureswhat we intend it to measure.

Construct validity:

Validity of the operational measurement ormetric

Criterion-related validity: Is predictive validity Content validity refers to the degree to which

a measure covers the range of meanings


12/19



13/19


Measurement Errors

Systematic Random

M = T + s + eM is the observed/measured score, T is the true

score, s is systematic error, e is random error

The correlation between the true score and theerror term is zero.

Three is no serial correlation between the truescore and the error term.

The correlation between errors on distinctmeasurements is zero.

Expected value of observed scores is equal to thetrue score:

E(M) = E(T) + E(e)= E(T) + 0= E(T)= T


14/19


Assess the Impact of e onthe Reliability of the

Measurements

M = T + e

Var(M) = var(T) + var(e) (var represents variance. Thisrelationship is due to the

assumption on error terms)

Reliability = Vm = var(T)/var(M)

= [var(M)-var(e)]/var(M)= 1[var(e)/var(M)].

Reliability of a metric varies between 0 and 1.

If all variance of the observed scores is due torandomerrors, then the reliability is zero


15/19


Test/Retest Method for

Estimating Reliability

M1M2

e1 e2Test

Retest

T


16/19



17/19


y = mx + b (straight line)


18/19


Criteria for Causality

1. The first requirement in a causalrelationship between two variablesis that the cause precedes theeffect in time or as shown clearlyin logic.

2. The second requirement in a causal

relationship is that the twovariables be empirically correlatedwith one another.

3. The third requirement for a causal

relationship is that the observedempirical correlation between twovariables is not because of aspurious relationship.


19/19


Spurious Relationships

Z

X

Y

X Z Y

C

X Y

ex ey

1.basic metrics and measurement

Documents