combinatory metrics for software development

Combinatory Metrics for Software Development

Aligning Software Development with Customer’s Needs

Dr. Thomas M. Fehlmann

Euro Project Office AG

Zeltweg 50

CH-8032 Zurich, Switzerland

Phone: +41 1 253 1306 Fax: +41 1 253 1364

E-mail: [email protected]

V1.0-6, 22 May, 2002 © 2002 by QFD Institut Deutschland e.V.

and the author

22 May, 2002 V1.0-6

Combinatory Metrics for Software of 16 Dr. Thomas Fehlmann

Combinatory Metrics for Software Development

Dr. Thomas M. Fehlmann Euro Project Office AG, Zurich, Switzerland

Summary Combinatory Metrics is the science of defining metrics for topics that are not directly measurable. Such topics exist in abundance in the area of software development.

This paper explains how to establish a network of combinators to cover the topics for which we want to get metrics. The basic techniques are the cause – effect combinators known from QFD (“Quality Function Deployment”). The combination technique is taken from Combinatory Algebra, a theory used in computer science to understand the nature of computing with knowledge about topics.

We also present the experiences we made when applying these techniques to a medium-sized software development organization that targets a niche market. People do not necessarily like metrics, but combinatory metrics are very helpful when developing products that satisfy customer’s needs.

Thomas M. Fehlmann, Dr. sci. Math. ETH, Euro Project Office AG, Zeltweg 50, CH-8032 Zurich, phone +41 1 253 1306, fax +41 1 253 1364, E-mail: [email protected]

22 May, 2002 V1.0-6


Table Of Contents

1. Combinatory Metrics........................................................................... 4 1.1 Why metrics? ..........................................................................................................4 1.2 The problem with metrics........................................................................................4 1.3 Software ..................................................................................................................4 1.4 Topics......................................................................................................................5 1.5 Combinatory Metrics ...............................................................................................6 1.6 Quality Function Deployment..................................................................................7 1.7 Combinatory metrics and QFD ...............................................................................9

2. A Practical Case Study ..................................................................... 10 2.1 Metrics for the Software Development Model.......................................................10 2.2 Listening to the Customer’s Voice ........................................................................11 2.3 How does it translate into software development? ...............................................11 2.4 Aligning to the market using combinators.............................................................12 2.5 Testing software with combinators .......................................................................13 2.6 Metrics Precision...................................................................................................13 2.7 Filling the Gaps .....................................................................................................14 2.8 Case Study Metrics Overview...............................................................................14

3. Conclusions....................................................................................... 15 Table of Figures Figure 1: Sample Software Development Model......................................................................5 Figure 2: How much does this feature contribute?...................................................................6 Figure 3: A Cause–Effect Combinator with derived metric for solution characteristics ...........7 Figure 4: Comprehensive QFD – Sample use of combinators ................................................8 Figure 5: Sample Lanchester combinator linking competition to business processes ............9 Figure 6: Metrics the for Software Development Model .........................................................10 Figure 7: Customer’s Needs for a Hotel Reservation System ...............................................11 Figure 8: CMM Assessment Result........................................................................................12 Figure 9: Sample testing combinator......................................................................................13 Figure 10: Overview of Combinatory Metrics used for the Case Study .................................14

22 May, 2002 V1.0-6


1. Combinatory Metrics

1.1 Why metrics? When working on complicated matters that are hard to understand, men always tried to use measurements to help assessing where he stands. Measurements were at the very beginning of seafaring, and already the old Greeks in the 2nd century B.C. constructed analog computers to navigate in stormy seas [20].

However, when talking about metrics in software development, enthusiasm is limited and the audience skeptical. So many well-meant approaches failed to yield the benefits people had hoped for. Technology in software development changes so rapidly that before a metric could establish itself, it was already outdated.

1.2 The problem with metrics The main problem with metrics is that you need something to measure. When talking software, it is hard to find anything tangible. One of the best candidates is the Function Points method (IFPUG 4.1). Function Points are independent from the software techniques used [14].

Function Points is a good example that measurements and metrics are not the same. Function Point is an excellent way of measuring the size of software. They are therefore very suitable as metrics for software size and — more general — for the size of ICT (Information & Communication Technology) applications, and can therefore be used easily and successfully for derived metrics such as defect density and system availability and reliability, and thus whatever relates to the size of an application.

However, when we try to use Function Points to estimate the cost of developing the software, we run into a number of problems. Software development cost depend on a variety of things, such as scope definition, availability of requirements and of customers, knowledge of the application domain and — at the same time — of the logic behind software development, and other skills, development tools used, business processes in effect, and much more. At the end, the size of the generated software is then only one cause among others that contribute to development cost1.

If we want to use Function Points to predict time and effort needed to develop the software, then we must first study the cause – effect relationships between size, time and effort in software development. To do this, we use combinatory metrics.

1.3 Software For practical purpose, we use a slightly different definition for “software”. We define software as an expert service that is delivered at a remote location, at

1 In a recent Web – enabled software product development project, the cost of development split as

follows: 85% getting the requirements right; 15% design, coding, testing and packaging.

22 May, 2002 V1.0-6


a time chosen by the user of the service with the help of ICT, without having the experts at hand that once prepared this service (see [6]).

This definition both captures the technical aspect of software, namely that software is running a program (sequence of executables) which manipulates data, as well as what users expect from the software part of a web service: Providing valuable services fast and with almost no dependency from the personal availability of experts, sometimes with the possibility to enter a consequential contractual business agreement2. People resources that might be needed to complement such service (e.g. a help desk) are not part of software; neither are hardware and other ICT equipment that is necessary to provide such service. Consequently, software engineering describes the techniques needed to define such service, i.e. the various tools for defining, creating and documenting what the service provides such as design, programming, and documentation tools.

1.4 Topics When creating software, we need to consider various aspects. We call such aspects topics. Topics3 are identified by the software development methodology.

Component Design & Programming

System Architecture

Business Process Specification & Implementation

Business Strategy & Realization

Customer’s Needs

Use Cases

BusinessProcesses

SoftwareFeatureTests

User-Centered Tests

Architectural Tests

SoftwareFeatures

Programming

Business Process Tests

Specification

Verification

Specification

Verification

Specification

Verification & Validation

Figure 1: Sample Software Development Model

2 Imagine how e-Commerce and other Web-based services could flourish if the providers would start to

understand that they must excel in service quality. 3 Sometimes people use the term “views”, such as engineering view or customer’s view, instead of

“topic”. However, “view” assumes that the essence of the matter in question is the same, but it is not. For instance, time-to-market and maintainability are two different topics when developing software, not different views.

22 May, 2002 V1.0-6


As an example, we show nine topics that pertain to web software testing in the Figure 1 above.

1.5 Combinatory Metrics Combinatory Metrics is the science of how to derive metrics for topics that cannot be measured directly, but have an impact on things we would like to know. For instance, we would like to know in advance whether adding a feature to our software would increase competitiveness. Therefore we need a metric for the software feature. Competitiveness is easily measurable: We need to look at the market share, and its variations over time. However, finding a metric that tells us the impact of a given feature on overall success in the market is hard. We have two options: Asking a representative couple of people, or do cause – effect analysis.

StartStart

GoalGoal

Time

Val

ue

Figure 2: How much does this feature contribute?

Asking people has several inherent problems. First, we may not have a customer readily available, who is responsible and can give us his require-ments and answer questions. Second, people often lack the imagination to understand the impact of a proposed feature. This is the reason, why Requirements Engineering is so hard.

On the other hand, we know quite well how to conduct a customer’s needs analysis. Cause – effect analysis does transform such insight into software requirements and design choices. Moreover, the QFD Methods yields a metric for these topics, by assigning a value for its relative contribution to satisfying customer’s needs.

Cause – effect analysis is not easy either, but it has the big advantage, that it is not much dependent on time. Neither technology advances, nor people’s perception have much impact. Once cause – effect relations are understood, much of the hard work is done for good. Cause – effect analysis can be reused as long as the topics remain the same. What does change indeed, is the relative value of the topics

Thus the first step in cause – effect analysis may be hard, but combining the results is easy.

22 May, 2002 V1.0-6


1.6 Quality Function Deployment The best-known application of combinatory metrics is QFD.

QFD allows for three basic operations:

1. It describes how a solution approach contributes to solve a problem. Traditionally, the problem has been stated as how to satisfy customer's needs; the solution to it being the optimum choice of solution components that constitute the solution approach. The tools used are correlation matrices such as the famous “House of Qualities”4.

We call the items being correlated to each other, such as solution approaches and the problem under question, Deployment Topics. The target Deployment Topics we call Goal Topics (the left–hand side of the correlation matrix), the means describing the solution approach we call Solution Topics (bottom of the matrix).

We refer to these links between the Deployment Topics described by such correlation matrices as Cause–Effect Combinators5.

Solution Quality Requirements Solution Quality Requirements

Deployment

Focu

s on f

ew

rele

vant

thin

gs

Pro

vid

e u

ndo a

nd j

um

p b

ack

Dynam

ic e

lem

ents

Erg

onom

ic layout

Vis

ual fe

edback

Real-

tim

e D

B u

pdate

Acc

ess

to d

ata

base

s

Pri

vate

are

as

Hig

h a

vaila

bili

ty

Modula

r desi

gn

Obje

ct m

odel

Fail-

Save b

ehavio

ur

Defined t

est

sta

tus

Inte

gra

ted s

upport

Conta

ct a

vaila

bili

ty

Sale

s Poin

ts

Ready t

o P

ay

Com

petition

Cust

om

er

Segm

ents

Sele

ctio

n C

rite

ria

Customer's Needs SC

-1.1

SC

-1.2

SC

-1.3

SC

-1.4

SC

-1.5

SC

-2.1

SC

-2.2

SC

-2.3

SC

-2.4

SC

-3.1

SC

-3.2

SC

-3.3

SC

-3.4

SC

-4.1

SC

-4.2

5 4 5 3

CN-1.1 Know who is operating the site 3 3 2 4 5CN-1.2 Unambiguous liability 3 2 2 5 4CN-2.1 Know where to stay 5 5 5 4 3CN-2.2 Accommodate party size 3 4 3 3 4CN-2.3 Be sure it has been understood 5 5 5 4 4CN-3.1 Trust to get value for money 4 4 3 3 4CN-3.2 Payment reaches destination 2 1 1 4 5CN-4.1 Be sure booking remains valid 3 2 2 4 4CN-4.2 Be able to adapt booking 4 4 3 5 4CN-5.1 Find the place 4 5 4 4 3CN-5.2 Track effective cost 3 2 2 4 3CN-5.3 Being remembered 5 5 5 5 4

Raw Weight 183 162 167 151 178 178 208 186 132 127 152 150 168 180 179 2502 208

Weight % 7 6 7 6 7 7 8 7 5 5 6 6 7 7 7

Weight 1-5 Scale 4 4 4 4 4 4 5 4 3 3 4 4 4 4 4

Weig

ht

Tota

l

Max.

0

1

2

3

4

5

0 1 2 3 4 5

Cause - Effect RelationshipStrong 9Medium 3Weak 1

Solution Topics

Goal Topic

s

Problem

Weight

What

do we m

easure?

Howdo we contribute?

Figure 3: A Cause–Effect Combinator with derived metric for solution characteristics

Figure 3 shows a sample Cause–Effect Combinator used in the ICT project explained below, see section 2 of this article. We derive a metric for the solution topics — in this case the solution quality characteristics — from the goal topic, which consists of the customer’s needs.

2. Furthermore, QFD describes how Cause–Effect Combinators F * G may be applied to each other. The Solution Topics that support the Goal Topics of the Cause–Effect Combinator F, become in turn the Goal Topics in Cause–

4 The “House of Qualities” is usually the first of a sequence of correlation matrices that describe the

quality deployment for a given project, product, or service. 5 Note that there exist other approaches to cause–effect analysis that do not provide the capability of

being combinators or provide metrics.

22 May, 2002 V1.0-6


Effect Combinator G. Such combinations are referred as Comprehensive QFD, or QFD in the Broad Sense [2].

Linking topics by combinators allows deriving metrics for each of the topics that contribute to the goal topic.

StartStart

GoalGoal

Time

Val

ue

Cause - EffectMeasurementDependency

Cust

omer

’s N

eeds

Solution Characteristics

Solution Design

Sol

ution C

har

acte

rist

ics

SW Features

Sol

ution D

esig

n

Feature Tests

SW

Fea

ture

s

Figure 4: Comprehensive QFD – Sample use of combinators

3. Third, QFD integrates measurement methods for measurable topics. The best-known example is the Voice Of The Customer method, which normally stands at the beginning of every deployment (for example, see [10], [21], [22]).

There exist a variety of methods to measure customer’s needs. This topic constitutes in most deployments the starting point for the series of combi-nators that finally lead to the optimization of the solution approach. Traditional QFD is based on this one measurement.

However, there are many more applicable measurement methods. Topics such as cost, market share can be measured with traditional marketing and controlling tools; scope, defect density, and other metrics related to the size of the generated software use the Function Points method. Such measure-ments are the crosschecks in the measurement networks.

For instance, the 2nd law of Lanchester [17] allows to link various product characteristics to market share. Using this law, one can predict the outcome of clients’ product evaluations and therefore predict the attainable market share. The Lanchester combinator is similar to the cause – effect combinator introduced above, however, it calculates two directions: One is the cause-effect correlation, how well the competition does with satisfying customer’s needs, or delivering the wanted solution characteristics. The others are the actual market share measurements.

22 May, 2002 V1.0-6


Competitive Advantage Business Process

Att

ract

cust

om

er

Info

rm c

ust

om

er

about

off

ers

Cust

om

er'

s se

lect

ion

Contr

act

ing

Dow

npaym

ent

Retr

ievin

g t

he r

ese

rved h

ote

l

Check

in into

the h

ote

l

Explo

re t

he h

ote

l

Check

-out

and p

ay t

he f

inal

bill

Retr

ieve t

he W

eb s

ite

Conta

ctin

g e

x-c

ust

om

ers

Managin

g m

em

ori

es

of

cust

om

ers

Competition SC

-1.1

SC

-1.2

SC

-1.3

SC

-1.4

SC

-1.5

SC

-2.1

SC

-2.2

SC

-2.3

SC

-2.4

SC

-3.1

SC

-3.2

SC

-3.3

Rating Re la tive

Our System 4 9 8 6 5 6 6 8 8 6 8 9 9 352 4

"Gorgeous Views" - Reservations 5 8 6 9 6 8 7 9 6 7 3 6 5 400 5"Best Price" - Reservations 4 7 6 7 6 9 8 7 7 5 5 7 6 336 4"High Cuisine" - Reservations 3 6 8 8 5 5 7 8 5 4 4 4 2 185 2"Pittoresque" - Reservations 2 8 6 5 7 4 5 5 6 8 5 2 2 139 2"Business Travel" - Reservations 2 7 8 8 6 7 8 8 6 7 2 5 4 137 2

Solution Characteristics Weight Factors 4 5 5 2 2 3 3 2 3 3 2 5Our Weight 38 39 30 12 14 16 23 14 21 20 22 45 4 45

Competition Weight 31 32 38 15 17 19 22 11 21 10 13 21 16 38 45Our Achievement 1-5 Scale 4 4 3 1 2 2 3 2 2 2 2 5 33 1 1072

Competition's Achievement 1-5 Scale 3 4 4 2 2 2 2 1 2 1 1 2 28 0 765(relative to Weapon's Strength) Better than the Competition 18% E2= 1.40

Competition: yellowOur Solution.: green

Mark

et

Sh

are

Weig

ht

0 1 2 3 4 5

0

1

2

3

4

5

Attract

custo

mer

Inform

custo

mer ab

out o

ffers

Custom

er's s

electi

on

Contra

cting

Downp

aymen

t

Retriev

ing th

e res

erved

hotel

Check

in in

to the

hotel

Explore

the hote

l

Check

-out a

nd pay

the f

inal bil

l

Retriev

e the

Web

site

Contac

ting e

x-cus

tomers

Managin

g mem

ories

of cu

s...

N0 = 3.72

Figure 5: Sample Lanchester combinator linking competition to business processes

Using such measurements, one can establish a metrication scheme for the whole software development project. In the example in section two of this paper, we used four measurement methods to calibrate the combinatory metrics: An assessment of the software development processes according the Capability Maturity Model (CMM) criteria [11], a Voice of the Customer approach at the beginning of product development, a software size measure-ment using Function Points to define defect density, and two Lanchester combinators for the market share (see Figure 10: Overview of Combinatory Metrics used for the Case Study).

1.7 Combinatory metrics and QFD Combinatory metric is an extension of QFD in two directions. First, we observe that the combinators do not only work in the direction of the deployment. A deployment answers the question, which solution mix causes the wanted effect. There is the other way round as well: If we have a solution, then the same combinator answers the question, how well this solution supports the goal topic.

This two-way calculation has been repeatedly used for combinators that link solution topics like testing and risk exposure to goal topics such as customer’s needs and solution approach. The mappings are not opposite to each other. In general they have a different meaning, as shown in [7].

Another difference is that combinatory metrics are not limited to measuring topics. You can lift the combinators to a higher level and define metrics for how well an organization is capable to apply new topics to new areas [8].

22 May, 2002 V1.0-6


2. A Practical Case Study The following case study has been altered in business domain, in order to protect sensitive data of the customer.

A medium – sized software house builds software products for a niche market, namely Web – based hotel reservation systems for travel agencies and city tourist offices. It develops a software product for a worldwide market. It has a small but very active and highly skilled sales force.

2.1 Metrics for the Software Development Model The metrics for the software development model as presented in Figure 1 are defined by the following structure.

First we have the classical QFD Deployment

( Business Process j Customer’s Need) i ( Use Case k Business Process) j

( Software–Feature m Use Case ) k

Component Design & Programming

System Architecture

Business Process Specification & Implementation

Business Strategy & Realization

Cust

om

er’s

Nee

ds

Business Processes

Use Cases

Busi

nes

s Pro

cess

es

Software Feature

Use

Case

s

Cust

om

er’s

Nee

ds

User Centered Test Methods

Business Process Test Methods

Busi

nes

s Pro

cess

es

Architectural Test Methods

Use

Case

s

Software Feature Test Methods

Soft

ware

Fea

ture

Figure 6: Metrics the for Software Development Model

On the right side, we have the following four test deployments.

( User–centered tests i’ Customer’s Needs ) i ( Application Test j’ Business Process) j

( Integration Test k’ Use Case ) k ( Feature Test m’ Software Feature) m

22 May, 2002 V1.0-6


The model starts with “Customer's Needs” as the Goal Topics of the first Cause–Effect Combinator. Such deployments have been discussed in [10], [12], [16] et.al. Thus we get combinatory metrics for the selection of the Business Processes that we want to support, of the Use Cases that are necessary for the Business Processes, and for the selection of software features. As we have seen, we need to establish calibration measurements to make the metrics useful.

2.2 Listening to the Customer’s Voice The most useful measurement is the customer’s needs. In this case, we used a questionnaire, developed according the method of Kano[12], to establish the base for the combinatory metrics network. The questionnaire focused on the topics of needs and wants, but did also ask for their preferred solution characteristic. It was not meant to get actual solution ideas from customer, but rather to find out what their unexpressed needs are. The software developers have much better solutions in mind than the customer possibly could imagine. But they did need the voice of the customer to decide what to realize first.

The outcome6 of the assessment is shown in the following list: Customer's Needs Explanations

CN-1 Trust CN-1.1 Know who is operating the site The need to know who is operating the Web business.

CN-1.2 Unambiguous liability Know how to act in case of incidents or fraud

CN-2 Booking CN-2.1 Know where to stay The need to identify the place where the hotel is.

CN-2.2 Accommodate party size Availability of suitable rooms for all the party.

CN-2.3 Be sure it has been understood Unambiguousity across language and cultural borders.

CN-3 Pricing CN-3.1 Trust to get value for money Have the opportunity to compare price levels.

CN-3.2 Payment reaches destination Know where to pay and where the payment goes.

CN-4 Traveling CN-4.1 Be sure booking remains valid … and you don't arrive at midnight and won't find a bed.

CN-4.2 Be able to adapt booking Flexibility of coping with change of travel plans.

CN-5 Stay CN-5.1 Find the place Actually retrieve the hotel upon arrival.

CN-5.2 Track effective cost Get a detailed bill that identifyes cost items.

CN-5.3 Being remembered Have the feeling to be highly estimated as a guest.

0 1 2 3 4 5

Figure 7: Customer’s Needs for a Hotel Reservation System

2.3 How does it translate into software development? To understand the customer is one condition for success. To be able to use this understanding effectively is the next one. It was therefore felt that there is a need to assess the management processes of the company.

In order to have a second calibration measurement for the combinatory metrics network, we conducted a CMM assessment of the company. To do this, we established additional combinators

( Solution Characteristics j Customer’s Needs ) i ( Management Processes k Solution Characteristics ) j

( CMM Criteria m Management Processes ) k

6 The assessment shown here is for demonstration purposes only. These are not the actual results.

22 May, 2002 V1.0-6


This yields a metric for the importance of the CMM criteria for this particular company.

The result of the assessment was as shown in Figure 8. Note that the combinator ( CMM Criteria k Management Processes ) m is used both ways. One way is to derive the needed CMM – level for the management processes such that the market expectations can be met.

Top Processes Assessment

Importance Results

The Four Tops

PL-1.1 Voice of the Customer 5.00 1.20 PL-5.2 Risk Management 4.18 2.18 PL-6.8 Acceptance Testing 4.00 1.59 PL-1.3 Requirements Engineering 3.96 1.95

The Ten Valuable

PL-7.4 Knowledge Management 3.95 1.17 PL-1.2 Project Proposal 3.74 1.18 PL-2.3 Realization 3.70 1.36 PL-4.1 Hotline 3.61 1.14 PL-6.7 Application Testing 3.56 2.36 PL-2.2 Object Design 3.55 1.33 PL-6.4 Code Review 3.35 2.12 PL-8.2 Metrics 3.30 1.50 PL-7.5 Repository 3.12 1.81 PL-2.4 Release Planning 2.88 1.68

The Disregarded

PL-6.3 Formal Review 2.83 PL-8.6 Problem Reporting 2.73 PL-2.1 Business Analysis 2.70 PL-3.3 Going Life 2.70 PL-5.7 Supplier Management 2.49

3.71 1.61 Goal Actual

Voic

e of

the

Cust

om

er

Ris

k M

anagem

ent

Acc

epta

nce

Tes

ting

Req

uirem

ents

Engin

eering

Know

ledge

Managem

ent

Proje

ct P

roposa

l

Rea

lization

Hotlin

e

Applic

ation T

esting

Obje

ct D

esig

n

Code

Rev

iew

Met

rics

Rep

osi

tory

Rel

ease

Pla

nnin

g

0

1

2

3

4

5SW Development Excellence

Figure 8: CMM Assessment Result

On the other hand, the same combinator is used to assess the actual performance. The standard Software CMM assessment questions had been used[17]; however the assessment was limited to the most important key process areas, due to cost constraints.

2.4 Aligning to the market using combinators To understand the impact on the market, we used New Lanchester theory to construct the combinator that links the Business Processes to the actual and the predicted market share of competition. It came out as shown in Figure 5.

Note that the New Lanchester combinator

( Business Processes k Competition ) m

is used again in both directions, comparing the Web software with existing market share. According this comparison, a market share gain of 18% can be expected, provided all planned features were implemented and the competition remained on the actual state.

At the time of this writing, it is not yet known if the New Lanchester predictions will become true. There exist other case studies showing that such predictions are amazingly correct [5]. However, we cannot yet speak about statistically sound experience involving New Lanchester and combinatory metrics. It would pay off to learn more about this technique.

22 May, 2002 V1.0-6


2.5 Testing software with combinators Finally, the testing combinators, discussed in detail in [9], provide a means to optimize the testing effort such that customer’s needs, and the features needed for the planned market share gains, are the focus of the testing effort. A sample testing combinator is shown in Figure 9.

Architectural Tests Architectural Tests

Coverage

Nav

igat

ional

com

ple

tenes

s

Auto

mat

ic U

RL

chec

ks

Con

text

dep

enden

cy

Info

rmat

ion d

ispla

y co

mple

tenes

s

Conte

xt s

ensi

tivi

ty t

est

Info

rmat

ion r

etri

eval

tes

t

Por

tal fu

nct

ional

ity

test

Sea

rch e

ngin

es t

ests

Links

outb

ound

Vis

ibili

ty t

est

Rea

dab

ility

tes

t

Bro

wse

r te

sts

Links

inbou

nd

Use Case WT-B

.2

WT-E

.2

WT-B

.1

WT-C

.1

WT-C

.2

WT-A

.2

WT-A

.1

WT-E

.4

WT-E

.1

WT-D

.1

WT-D

.2

WT-D

.3

WT-E

.3

3090 442

UC-2.10 Customer Photo Gallery 4.9 5.0 435 14% 4.9

UC-1.1 Welcome Page 4.3 4.3 442 14% 5.0

UC-2.6 Reservation Notification 3.1 4.4 315 10% 3.6

UC-2.9 Customer Profile 3.1 4.4 305 10% 3.5

UC-2.4 Preference Selection 2.2 4.3 222 7% 2.5

UC-1.2 Link Page 1.6 2.9 237 8% 2.7

UC-2.3 Reservation Status 1.1 2.5 198 6% 2.2

UC-2.2 Room Information Page 1.0 2.3 198 6% 2.2

UC-2.1 Room Images 0.9 2.7 152 5% 1.7

UC-1.3 Navigation Bar 0.9 2.7 145 5% 1.6

UC-1.4 Quicktime Panorama 0.5 2.1 109 4% 1.2

UC-2.8 Downpayment Confirmation 0.5 2.0 110 4% 1.2

UC-2.5 Room Reservation Page 0.5 1.9 118 4% 1.3

UC-2.7 Downpayment Method Selection 0.4 1.8 102 3% 1.2

Raw Weight 176 175 172 166 159 155 144 142 141 132 132 124 111 1931 176

Weight % 9 9 9 9 8 8 7 7 7 7 7 6 6

Weight 1-5 Scale 5 5 5 5 5 4 4 4 4 4 4 4 3

Wei

ghte

d C

ove

rage

Wei

ght

of U

se C

ase

0

1

23

4

5

0 1 2 3 4 5

Cause-Effect Relationship

Strong 9

Medium 3

Weak 1

Figure 9: Sample testing combinator

The testing combinator compares actual test coverage (dark brown profile, on the right side of the matrix) with the importance regarding customer’s needs (light green profile). The test case metrics shows their respective importance regarding customer’s needs (which includes the market ‘s expectations according the New Lanchester Strategy).

2.6 Metrics Precision It is an often-raised question how reliable such a combinatory metric network actually is. At first, it depends from the accuracy of the cause – effect matrices in use. Thus they depend from the domain know-how of the people who did the analysis. However, there is also a statistical method of assessing its dependability. This is in studying the effects induced by variations of the cell values coupling causes to effects. The usual method applied by the author is to change each value by one in each direction, and check if the order of importance changes. It is known that using the mathematical properties of this linear mapping between topics could yield more reliable results, however for all practical purposes so far it was sufficient to do such statistical sensitivity analysis.

22 May, 2002 V1.0-6


However, it is much more important to identify the topics correctly, and keep them separate. It requires common understanding of the domain scope between the people in charge. It is therefore suggested to call such common understanding “Organizational Knowledge”. Using combinatory metrics, we may measure organizational knowledge by its effect on the company’s financial results.

2.7 Filling the Gaps The main advantage of Combinatory Metrics is that the combinators are usable in both directions. One direction is the cause – effect analysis used in QFD to deploy goal topics like the customer’s needs in all branches of the delivery process. The other direction allows assessing the actual performance. In general, such gaps exist and they are very important to know for successful operations.

2.8 Case Study Metrics Overview Figure 10 shows a complete overview over the combinatory metrics used for the “Hotel Reservation” system software delivery organization, the topics assessed and measured, and all topics for which metrics have been defined. The gaps in the development processes became immediately apparent in the early stage and we could start software development process improvement where there was still time to fix those shortcomings that has most impact on the planned software development.

Goal:Goal:becomebecomeNo. OneNo. One

Time

Val

ue

Customer’sNeeds

Competition

CustomerSegments

MarketSegments

CustomerQuestionnaire

KanoAnalysis

CustomerBudget

Readyto pay?

Start:Start:No. 2 AsiaNo. 2 AsiaNo. 3 USNo. 3 US

BusinessProcesses

Use CasesSoftwareFeatures

UserCentered

Tests

Business Process Test

ArchitecturalTests

Competitor’sFeatures

SoftwareFeatureTests

SoftwareDevelopment

Competitor’sProcesses

SolutionCharacteristics

CMM Criteria

Cause - EffectMeasurementDependency

New Lanchester

New Lanchester

Application Test Coverage

Acceptance Test Coverage

Architectural Test Coverage

Feature Test Coverage

Feature Deployment

Use Case Deployment

Process Deployment

ManagementProcesses

CompetitiveAnalysis

CompetitiveAnalysis

“House of Qualities”

SW Development Processes

CMM Assessment Combinator

Figure 10: Overview of Combinatory Metrics used for the Case Study

Setting up the combinators was not a major challenge; there was enough domain knowledge and software expertise at hand to do it efficiently. We

22 May, 2002 V1.0-6


used an Excel tool to record the combinators and to calculate Combinatory Metrics.

Combinatory Metrics is especially useful for requirements engineering and testing. On every level combinatory metrics relate requirements and test cases to customer ‘s needs and market expectations. We used this metrics to keep the development plans focused on the market needs. Every engineering task has a priority index attached to it, derived from customer’s needs.

During development, the market and the economic environment did evolve. Using the cause – effect combinators, we keep the product features in line with the evolving needs of the market.

The pilot application is now on the market in spring 2002 and the initial reaction was as expected. We expect to have numbers by end of the year.

The effort to set up this metric network was little compared with the standard quality assurance activities; it was well below 1% of the total development effort. Quality assurance (reviews and tests) counted for 20 – 25%.

3. Conclusions The main achievement of applying combinatory metrics in the case study was, that after years of constantly missed schedules, the new product is now on the market just in time and with the quality required (status May 2002).

Combinatory Metrics is a very promising approach to manage complex systems using locally applicable metrics. In the experience gained so far, there was not a single failure. Combinatory metrics deserve an in-depth exploration of its capabilities and characteristics. Its computer science background metrics is very compelling too [3], and more research on this techniques and the enabling theory should be performed. The payoff for complex systems will be tremendous.

22 May, 2002 V1.0-6


References

[1] Yoji Akao et.al.: Quality Function Deployment (QFD); Productivity Press 1990 [2] Yoji Akao, QFD and the international stamdard ISO 9001, 7th International QFD

Conference, October 2001, Tokyo, Japan [3] Erwin Engeler, The Combinatory Programme, Birkhäuser 1995 [4] Thomas Fehlmann, Quality Function Deployment for the Full Business Life Cycle, EOQ

Proceedings Vienna, April 1999. [5] Thomas Fehlmann, Measuring Competitiveness in Service Design, in: QFD Institute

(Ed.): 12th Symposium On Quality Function Deployment; June 2000 Novi, MI [6] Thomas Fehlmann, Christian Hauri, Measuring Project Management Excellence, in: 3rd

European Conference on Software Measurement and ICT Control, FESMA – AEMES, Oct. 2000, Madrid, Spain

[7] Thomas Fehlmann, Risk Exposure Measurements on Web Sites, in: 4th European Conference on Software Measurement and ICT Control, FESMA – DASMA, May 2001, Heidelberg, Germany

[8] Thomas Fehlmann, QFD as Algebra of Combinators, 7th International QFD Conference, October 2001, Tokyo, Japan

[9] Thomas Fehlmann, Business–oriented testing in E-Commerce, in: Software Quality and Software Testing in Internet Times, ed. Dirk Meyerhoff, SQS AG, Köln 2001

[10] Georg Herzwurm, Sixten Schockert, Werner Mellis: Qualitätssoftware durch Kunden-orientierung. Die Methode Quality Function Deplyoment (QFD). Grundlagen, Praxis-leitfaden, SAP R/3 Fallbeispiel. Vieweg-Verlag Braunschweig – Wiesbaden 1997

[11] Managing the Software Process, Watt S. Humphrey, Addison-Wesley, 1989 [12] Norikai Kano et.al., Attractive Quality and Must-be Quality, 12th Annual Meeting of the

Japanese Society of Quality Control, 1982 [13] Glenn Mazur, QFD for Service Organizations, by Japan Business Consultants, Ltd.,

1993 [14] IFPUG 4.1.1, International Funtion Point User Group, Function Point Counting

Practices Manual, Release 4.1.1, Princeton Junction, NJ, April 2000 [15] Shigeru Mizuno, ed. 1988; Management for Quality Improvement: The 7 New QC

Tools, Productivity Press, 1988 [16] Shigeru Mizuno and Yoji Akao, ed. 1994; QFD: The Customer-Driven Approach to

Quality Planning and Deployment, translated by Glenn Mazur, Tokyo: Asian Productivity Organization, 1994

[17] Mark C. Paulk, Bill Curtis, Mary Beth Chrissis, and Charles V. Weber, "Capability Maturity Model for Software, Version 1.1", Software Engineering Institute, CMU/SEI-93-TR-24, DTIC Number ADA263403, February 1993

[18] Nobuo Taoka: Lanchester Strategy – An Introduction, Lanchester Press Inc, 1997 [19] Ernest Wallmüller: Software – Qualitätsmanagement in der Praxis, Carl Hanser Verlag,

May 2001 [20] The apparatus of Antikyra. Technical Musemum at Thessaloniki, exposed in: World

Exhibition Hannover 2000 [21] Richard E. Zultner: Quality Function Deployment (QFD) for Software: Structured

Requirements Exploration. In:. Schulmeyer/McManus (ed.): Handbook of Software Quality Assurance.2nd Ed., Zurich 1992, S. 297-319

[22] Richard E. Zultner: Blitz QFD: Better, Faster, and Cheaper Forms of QFD. In: American Programmer. October 1995, S. 25-36

combinatory metrics for software development

Documents