integrating management and engineering processes in...

Integrating Management and Engineering Processes in Software

Product Development

Daniel Karlström

Department of Communication Systems

Lund Institute of Technology

Integrating Management and Engineering Processes in Software Product Development

2

ISSN 1101-3931 ISRN LUTEDX/TETS- -1069- -SE+230P Printed in Sweden E-husets tryckeri Lund 2004


4

This thesis is submitted to Research Board FIME - Physics, Informatics, Mathematics and Electrical Engineering - at Lund Institute of Technology (LTH), Lund University, in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Engineering.

Contact Information:

Daniel Karlström Department of Communication Systems Lund University Box 118 SE-221 00 LUND Sweden Tel: +46 46 222 08 84 Fax: +46 46 14 58 23 email: [email protected] http://www.telecom.lth.se/Personal/danielk/

5

Abstract

The intangible nature of software means that traditional processes for managing product development are not sufficiently effective. This, in combination with the increasing share of software in technical products, has had the effect of making it difficult to identify improvement proposals for engineering processes in many cases. This thesis uses both qualitative and quantitative research methods to investigate methods that involve the entire development organisation in software process improvement.

The research is performed in cooperation with several companies developing software products in different countries. The studied organisations range from small development companies, using light, informal methodologies, to large corporations, with thousands of developers, using large frameworks of formal process models. The need for organisations to use both local experience as well as generally accepted best practice in process improvement is addressed.

A method that uses input from the whole organisation for strategic software process improvement is created and evaluated. The introduction of one agile methodology, Extreme Programming, is studied in a small team and a decision support method is evaluated for introducing the methodology. Also, a framework based approach to process improvement is applied to testing practices. The approach allows a small, rapidly evolving, company to introduce suitable practices that solve problems apparent in the current state of the company. This allows the processes to evolve in small steps without introducing formal practices too early in the company’s evolution.

An integration of agile methods and high-level product management processes is proposed and evaluated on both the engineering level and the product management level of organisations. The integration allows contrasting elements of the processes to remain effective within each respective domain and in some instances even perform synergetically.


6

7

Acknowledgements

This work was partly funded by The Swedish Agency for Innovation Systems (VINNOVA), under a grant for the Centre for Applied Software Research at Lund University (LUCAS). The research performed in Japan was funded using contributions from the Sweden-Japan Foundation, Per Westling’s memorial foundation, Hosei University Tokyo, the Ernhold Lundström scholarship, and the Saskawa foundation. Travel support was also received from the Royal Physiographic Society in Lund.

First and foremost I would like to thank my supervisor, Dr. Per Runeson, for supporting, guiding, inspiring and restraining me during the work presented in this thesis. I would also like to thank my second supervisor, Dr. Martin Höst for always being available to answer my questions.

I would like to take this opportunity to thank Prof. Claes Wohlin for convincing me to start as a PhD student in the group.

I would also like to thank my co-authors and all other people who have contributed to the research performed in the thesis.

A special thanks to all my colleagues at the department of Communication systems for creating a professional working environment. Especially I would like to thank the members of the Software Engineering Research Group, Carina Andersson, Enrico Johansson, Lena Karlsson, Johan Natt och Dag, Josef Nedstam, Bengt Ljungquist, Dr. Björn Regnell, Dr. Thomas Thelin, and former group members Dr. Magnus C. Ohlsson, Dr. Tomas Berling Dr. Håkan Petersson and Thomas Olsson for providing the opportunity for many rewarding discussions both on and off the topic of software engineering. Thanks also to the Telecom floor ball players for allowing me to clobber them every Monday evening.

Finally, I am grateful to Kerstin, my friends outside the department and my family. Thank you for putting up with me, especially during the last year while performing my studies away from home and, of course, during the last few months of finishing the thesis.


8

9

Contents

Introduction ......................................................................................13 1 Background.......................................................................................16 2 Research Methodology in Software Engineering................................42 3 Research Presentation........................................................................49 4 References .........................................................................................56

Paper 1: Aggregating Viewpoints for Strategic Software Process Improvement – A Method and a Case Study ........................................63 1 Introduction .....................................................................................64 2 Processes Distortion and Viewpoints.................................................66 3 Method.............................................................................................68 4 Case Study at Fuji Xerox...................................................................72 5 Summary and Conclusions................................................................87 6 Acknowledgements ...........................................................................88 7 References .........................................................................................88

Paper 2: Introducing Extreme Programming – A Case Study.................91 1 Introduction .....................................................................................92 2 Methodology ....................................................................................92 3 Online Telemarketing.......................................................................93 4 The Development Project .................................................................94 5 Experiences of the XP Practices .........................................................96 6 Quality of Conclusions ...................................................................101 7 Summary and Conclusions..............................................................102 8 Acknowledgements .........................................................................103 9 References .......................................................................................104


10

Paper 3: Decision Support for Extreme Programming Introduction and Practice Selection ............................................................................. 105 1 Introduction ...................................................................................106 2 XP Introduction Strategies ..............................................................106 3 Methodology ..................................................................................109 4 Evaluation Results...........................................................................114 5 Summary ........................................................................................119 6 Acknowledgements .........................................................................119 7 References .......................................................................................120

Paper 4: Minimal Test Practice Framework for Emerging Software Organisations ................................................................................... 121 1 Introduction ...................................................................................122 2 Related Work..................................................................................123 3 The Minimal Test Practice Framework...........................................126 4 Derivation of the Framework ..........................................................131 5 Application of the Framework.........................................................137 6 Survey Validation of the Framework ...............................................142 7 Validity of the Studies.....................................................................146 8 Summary and Future Work ............................................................147 9 Acknowledgements .........................................................................147 10 References .......................................................................................148

Paper 5: Agile Methods and Software Product Management – An Integrative and Hybrid Approach ...................................................... 151 1 Introduction ...................................................................................152 2 Background.....................................................................................154 3 Integrating Project Management Models and Agile Methods ..........162 4 Example ..........................................................................................171 5 Challenges for the Future................................................................177 6 Acknowledgement...........................................................................178 7 References .......................................................................................178

11

Paper 6: Integrating Agile Software Development into Stage-Gate Managed Product Development ........................................................ 181 1 Introduction ...................................................................................182 2 Study Scope ....................................................................................184 3 Case Contexts .................................................................................184 4 Study Design ..................................................................................186 5 Study Validity .................................................................................191 6 Results ............................................................................................193 7 Summary and Conclusions..............................................................203 8 Acknowledgements .........................................................................204 9 References .......................................................................................204

Paper 7: Exploring Agile Influences in Stage-Gate Management of Software Product Development ......................................................... 207 1 Introduction ...................................................................................208 2 Vodafone Group Global Product Development ..............................210 3 Study Design ..................................................................................212 4 Study Validity .................................................................................216 5 Results ............................................................................................218 6 Summary and Conclusions..............................................................228 7 Acknowledgements .........................................................................228 8 References .......................................................................................229

Reports on Communication Systems ................................................. 232


12

13

Introduction

Software has become a part of almost every product being developed today. The intangible nature of software means that traditional processes for managing industrial projects are not effective. This, in combination with the increasing size and complexity of software products, has had the effect that improvement strategies for development processes are in most cases hard to identify.

The field of software processes and process improvement has had to evolve constantly in order to keep up with changes in the type, size and complexity of the software being developed. In addition to this, the number of people performing the development has increased greatly as well as the variety of different types of software. An excellent example is the extent of Internet software developed in the early 1990s compared to early 2000. Currently there are two quite different approaches to process improvement being used in the industry, (1) methods based on maturity models, and (2) agile methodologies. The maturity models approach is regarded a more formal and ‘heavy’ approach and has evolved from the Capability Maturity Model (CMM) from the beginning of the 90’s. The agile approaches have evolved partly as a reaction to the first and are considered more lightweight. Currently the agile approach is thought less compatible with large-scale software product development, often performed in strict, staged, management models. The research performed in this thesis indicates that there may indeed be several advantages both on the engineering and management levels of the organisation by using agile concepts instead of traditional heavyweight concepts.

The research presented in this thesis is performed in both small development organisations, using light, less formal methodologies, known as agile methodologies, and large organisations, with thousands of developers, using large, more formal development process frameworks.


14

A method that uses input from the whole organisation for strategic software process improvement is created and evaluated. This method allows the process owners to base software process improvement on input from the entire development organisation. This is thought to anchor the decision more firmly in the organisation.

The introduction of one of the main agile approaches, Extreme Programming, is studied in a case study of a small development project in a real world context as well as in a large stage gate based product development context in two other cases.

A decision support method is evaluated for identifying practices within Extreme Programming that might be perceived to be more or less effective in the developer group before the introduction. Hence the introduction of the practices can be tailored based on the results of the analysis.

A framework approach to process improvement for testing is investigated that allows the company to introduce suitable practices that solve problems apparent in the current state of the company. This allows the test processes to evolve in small steps without introducing too many formal practices too early in the company’s evolution.

The research presented in this thesis attempts to reduce inherent resistance to process changes in organisations by increasing the involvement of the whole organisation in the improvement work and thereby increasing acceptance for process improvement.

The seven research papers in the thesis can be classified into research performed in a context of a small or large company and into research in the use of traditional or agile methods. This classification is shown in Table 1.1. The papers could be read in any order, but have been numbered chronologically as the studies have been performed. The papers are provided exactly as they have been published, only the layout has been altered to suit the format of this thesis. This has the advantage of making each paper more understandable as a unit but has the disadvantage of some repetition of material in the introduction and background .parts of the papers.

Large context Small context Traditional Paper 1

Paper 4

Agile Paper 5 Paper 6 Paper 7

Paper 2 Paper 3

Table 1.1 Paper Classification

15

The first part of this thesis is an introductory part that summarises and packages the work performed. It is divided into the following sections:

1. Background – This section briefly explains the background of software system engineering and software engineering processes, the structure of the discipline, the terminology involved and the challenges present today. 2. Research Methodology – This section discusses research methodology used in software engineering research. Parts of this section are based on the following publication. Karlström, D., “Effects of Introducing Agile Methodologies in Software System Engineering”, 3rd National Conference on Software Engineering Research and Practice in Sweden, SERPS'03, Lund, Sweden, October 2003. 3. Research Presentation – This section presents the research performed in this thesis, the strategies used, the main contributions made and an outline for further research in the field. Parts of this section are based on the following publication. Karlström, D. Runeson, P., “Combining Stage-Gate Product Development with Agile Software Development: Observations from three industrial cases”, Submitted the special issue of IEEE Software on software engineering project management, May/June, 2005.

The second part of this thesis contains the seven research papers that constitute the main contribution of the thesis:

Paper 1 - Aggregating Viewpoints for Strategic Software Process Improvement – A Method and a Case Study Karlström, D, Runeson, P. , Wohlin, C., Published in IEE Proceedings Software, October 2002. Paper 2 - Introducing Extreme Programming - An Experience Report Karlström, D., Published in proceedings of the third International Conference on Extreme Programming and Agile Processes in Software Engineering, 2002. Paper 3 - Decision Support for Extreme Programming Introduction and Practice Selection Karlström, D., Runeson, P., Published in proceedings of the fourteenth International Conference on Software Engineering and Knowledge Engineering, 2002.


16

Paper 4 - Incremental Introduction of a Structured Test Practice Framework in Emerging Software Development Organisations Karlström, D., Runeson, P., Nordén, S., Accepted for publication in the Journal of Software Testing Verification and Reliability, preliminary publication issue: June 2005. Paper 5 - Agile Methods and Software Product Management – An Integrative and Hybrid Approach Runeson, P., Karlström, D., Technical report, Lund Institute of Technology, CODEN : LUTEDX (TETS-7203) / 1-34 / (2004) & local 16, 2004. Paper 6 - Integrating Agile Software Development into Stage-Gate Managed Product Development Karlström, D., Runeson, P., Submitted to the Journal of Empirical Software Engineering, 2004. Paper 7 - Exploring Agile Influences in Stage-Gate Management of Software Product Development Karlström, D., Submitted to IEEE Transactions on Engineering Management, 2004.

1 Background The background to this thesis starts with an introduction to software engineering (SwE), system engineering (SE) and software system engineering (SwSe). The area of the thesis is mapped into the software engineering body of knowledge (SWEBOK) [1] followed by an introduction to SwE processes, software process improvement (SPI). Process maturity models are described in a separate section as are Agile Methodologies and project management models as these areas are central to the thesis topic.

1.1 Software Engineering and Software System Engineering

Software system engineering (SwSE) is considered the application of system engineering (SE) on software development [2]. SE is the practical application of scientific, engineering and management skills required to move from a need to a technical realisation of that need. At the SE level the product is generic, i.e. the product does not necessarily have to include software at all, can be part software or a complete software

17

product. A standard for SE is provided in IEEE Std. 1220-1998 [3]. Here, five aspects of SE are identified: Problem definition, Solution analysis, Process planning, Process control, and Product evaluation.

Relationships between SE, SwSE and software engineering are well described by Thayer [2] and are illustrated in Figure 1.1. Here, a sequential V-type model [4] relationship is implied for the activities on all levels. However, there are activities on all levels throughout the course of the product development. System safety is often a very important factor in large systems. The software part of the system is often the hardest part in which to ensure safety. Other aspects of system engineering such as logistic engineering, integrated logistics support (ILS) and logistics support analysis (LSA), although they may be remotely affected by a software development methodology change, fall outside the scope of this thesis. The Software Engineering Body of Knowledge (SWEBOK) [1] lists systems engineering as a related discipline to software engineering.

Figure 1.1 Engineering relationships between system engineering, software system engineering and software engineering [2]

1.1.1 The Structure of the Software Engineering Discipline

Software Engineering is a relatively new discipline and it is only recently that structure and terminology has become well established. The collaborative effort creating the Software Engineering Body of Knowledge (SWEBOK) [1] has begun the important task of creating an introduction

System analysis

System design

Software requirements

analysis

Architectural software design

Detailed software design

Code and unit test

System testing

System integration

testing

Software system testing

Software integration

testing

Software subsystem

testing

System engineering

Software system engineering

Software engineering


18

to the body of knowledge. This is, however, a difficult task as the discipline is constantly changing. The overview provided by this initiative is provided in Figure 1.6.

SWEBOK divides the discipline into ten knowledge areas, requirements, design, construction, testing, maintenance, configuration management, engineering management, process, tools and methods, and quality. The knowledge areas represent areas where research is being performed today and have been established at different times during the evolution of software engineering processes up to the present. The first five knowledge areas are ordered according to the steps in the traditional waterfall development process [6]. The knowledge areas themselves, however, are not on the same level of abstraction as each other and many are very closely related. Some sub-areas are even part of several other areas. For instance, the software process knowledge area is a meta-area compared to the first five knowledge areas. All the areas however play a significant role in the development of software today.

The research presented in this thesis can be categorised into the software engineering process knowledge area of the SWEBOK [1], but parts of the software engineering management area are also closely related to the work. An overview of the guide to the SWEBOK is shown in Figure 1.2 and relevant areas are highlighted. The process knowledge area is subdivided into the following sub-areas:

• Process implementation and change This sub-area focuses on organisational change. It describes the infrastructure, activities, models and practical considerations for process implementation and change.

• Process definition This sub-area focuses on how processes are defined. This can be as procedures, a policy or a standard.

• Process assessment This sub-area focuses on how process assessment or appraisal is carried out.

• Process and product measurement This sub-area focuses on how measurement is performed both to evaluate the need for process change and the effects of process changes made.

Figu

re 1

.2 T

he S

oftw

are

Eng

inee

ring

Bod

y of

Kno

wle

dge

Ove

rvie

w [

1]

(Hig

hlig

hts i

ndic

ate

the

area

s mos

t rel

evan

t for

this

thes

is)

Soft

war

e R

equi

rem

ents

So

ftw

are

Des

ign

Soft

war

e C

onst

ruct

ion

Soft

war

e T

esti

ng

Soft

war

e M

aint

enan

ceSo

ftw

are

Con

figur

atio

n M

anag

emen

t

Soft

war

e E

ngin

eeri

ng

Man

agem

ent

Soft

war

e E

ngin

eeri

ng

Proc

ess

Soft

war

e E

ngin

eeri

ng T

ools

an

d M

etho

ds

Rel

ated

D

isci

plin

es

Soft

war

e Q

ualit

y

Soft

war

e R

equi

rem

ents

fu

ndam

enta

ls

Req

uire

men

ts

Proc

ess

Req

uire

men

ts

Elic

itatio

n

Req

uire

men

ts

Ana

lysi

s

Req

uire

men

ts

Spec

ifica

tion

Req

uire

men

ts

Val

idat

ion

Prac

tica

l C

onsi

dera

tion

s

Soft

war

e D

esig

n Fu

ndam

enta

ls

Key

Iss

ues i

n So

ftw

are

Des

ign

Soft

war

e St

ruct

ure

and

Arc

hite

ctur

e

Soft

war

e D

esi g

n Q

ualit

y an

alys

is

and

Eva

luat

ion

Soft

war

e D

esig

n N

otat

ions

Soft

war

e D

esig

n St

rate

gies

and

M

etho

ds

Soft

war

e C

onst

ruct

ion

Fund

amen

tals

Man

agin

g C

onst

ruct

ion

Prac

tica

l C

onsi

dera

tion

s

Soft

war

e T

estin

g Fu

ndam

enta

ls

Tes

t Lev

els

Tes

t T

echn

ique

s

Tes

t Rel

ated

M

easu

res

Tes

t Pr

oces

s

Soft

war

e M

aint

enan

ce

Fund

amen

tals

Key

Iss

ues

in

Soft

war

e M

aint

enan

ce

Mai

nten

ance

Pr

oces

s

Tec

hniq

ues

for

Mai

nten

ance

Man

agem

ent o

f th

e SC

M P

roce

ss

Soft

war

e C

onfi

gura

tion

Iden

tific

atio

n

Soft

war

e C

onfi

gura

tion

Con

trol

Soft

war

e C

onfi

gura

tion

Stat

us A

ccou

ntin

g

Soft

war

e C

onfi

gura

tion

Aud

itin

g

Soft

war

e R

elea

se

Man

agem

ent

and

Del

iver

y

Initi

atio

n an

d Sc

ope

Def

init

ion

Soft

war

e Pr

ojec

t Pl

anni

ng

Soft

war

e Pr

o jec

t E

nact

men

t

Rev

iew

and

E

valu

atio

n

Clo

sure

Sof t

war

e E

ngin

eeri

ng

Mea

sure

men

t

Proc

ess

Impl

emen

tati

on

and

Cha

nge

Proc

ess

Def

init

ion

Proc

ess

Ass

essm

ent

Proc

ess

and

Prod

uct

Man

agem

ent

Soft

war

e T

ools

Soft

war

e E

ngin

eeri

ng

Met

hods

Soft

war

e Q

ualit

y Fu

ndam

enta

ls

Soft

war

e Q

ualit

y M

anag

emen

t an

d Pr

oces

ses

Prac

tica

l C

onsi

dera

tion

s

Com

pute

r E

ngin

eeri

ng

Com

pute

r Sc

ienc

e

Man

agem

ent

Mat

hem

atic

s

Pro j

ect

Man

agem

ent

Qua

lity

Man

agem

ent

Soft

war

e E

rgon

omic

s

Syst

ems

Eng

inee

ring

Gui

de to

the

Soft

war

e E

ngin

eeri

ng B

ody

of K

now

ledg

e (2

004

Ver

sion

)


20

1.2 Software Engineering Processes As software engineering processes have developed, the terminology has evolved successively. Sommerville defines the software process as: “The software process is the set of activities and associated results which produce a software product” [5]. We can model this basic concept by using an elementary process model adapted to software development which gives us a basic software process. This is illustrated in Figure 1.3.

Figure 1.3 An illustration of the software process [5]

Any software process needs to at least address the following activities in some form in order to develop software [5]:

• Specification. Decide what is to be developed. • Development. Develop the product. • Validation. Ensure the product meets the specification. • Evolution. Handle changes in the product.

In today’s methods, the execution of the software process is done in

many different ways. An example of software development that does not address these activities is the methods that sparked the software crisis, also known as ‘code and fix’ methods. The lack of planning and requirements in these methods, made the following problems common:

• Poor structure. After each fix the structure of the code is destroyed making subsequent fixes and additions more and more difficult.

• Inaccurate results. The resulting program is hardly ever what the customer desired in the first place due to the lack of requirements.

Software process

Product idea Software Product

Resources

21

• Expensive. Because of poor structure and lack of planning all modifications and fixes become very expensive.

When these problems were observed, new models that addressed

planning, requirements, test and modification were developed. Sommerville identifies four different types of such software

development models currently in professional use [5]:

• Waterfall type [6]. Each activity - specification, development, validation and evolution - is executed and signed off sequentially, one by one. • Evolutionary type [5]. The activities are interleaved to rapidly produce prototypes of increasing complexity and correctness with regards to customer requirements. • Formal transformation [7]. The system is specified as a mathematical system and is then transformed into the finished product using formal mathematical transformations. • Component based [8]. The software system is assembled from pre-developed parts.

Of these, the waterfall and evolutionary types are most widely used in

industry today, but the potential effectiveness of software reuse has caused much interest in component based software engineering. In the classification above, agile methodologies [9] are classified as evolutionary by Sommerville. These are, however, commonly regarded as a separate type. These methodologies are very new and there is a need for more research into the area. The agile methodologies are discussed in section 1.5.

One of the current problems within software engineering is the use of many different terms for similar items within different areas of the discipline. This has arisen over time as certain terms have been associated with certain methodologies or have become negatively charged due to their usage in different circumstances. New software processes have changed the terminology in order not to be associated with old values or just in plain ignorance of existing terminology. The terms that are used within this thesis are shown in Table 1.2


22

1.2.1 Software Engineering Process Evolution

Software engineering processes can be divided into three distinct periods chronologically and we are currently moving into a fourth period [10]. The periods are illustrated in Figure 1.4 and a brief summary is provided below

The first period can be referred to as ‘code and fix’ [11]. Much programming was performed using low-level programming languages, program size was relatively small and the software environment was less complicated with fewer external interactions [12]. This implied that many of today’s problems were not apparent during this period, although surprisingly many current problems were mentioned by Brooks in his classic book [13] in 1975. Performing software development was relatively simple compared to today’s elaborate systems as it meant that bugs could be fixed. Systems could actually be developed that were completely fault free. Term Definition Process A sequence of steps performed for a given purpose [14].

Model A model is a simplified representation of a system or phenomenon with

any hypotheses required to describe the system or explain the phenomenon, often mathematically. It is an abstraction of reality emphasizing those aspects that are of interest to someone [15].

Software process The set of activities and associated results which produce a software product [5].

Practice A specific activity within a process or method.

Practice framework A collection of practices creating a form of a software process.

Method A reasonably complete set of rules and criteria that establish a precise and repeatable way of performing a task and arriving at a desired result [16].

Methodology A collection of methods, procedures and standards that defines an integrated synthesis of engineering approaches to the development of a product [16].

Improvement method Defines the strategy and the process by which improvement is pursued. It specifies how to assess a process, to initiate corrective actions, to evaluate them, and to further promote new improvement initiatives.

Maturity model Maturity model defines the ideal profile of a process, i.e., the (minimum) set of requirements that a process should satisfy to qualify for a specific status.

Table 1.2 Terminology within software engineering processes

23

The second period can be referred to as the structured methods period [11]. This period started after the software crisis in the late 1960’s when new hardware advances enabled software systems to be developed that were too large for the traditional development methods used at the time [5]. Elementary software development models were developed and used in industry. A major characteristic of this period is that the software engineering process was being described graphically. Examples thereof are waterfall and spiral processes [6, 17]. A large portion of practicing software development companies today are still using gate processes similar to these early processes although they are tackling functionality several orders of magnitude larger than in the end of the 70s. This evolution of the waterfall method is commonly referred to as gate or milestone models [18, 19, 5, 20].

Figure 1.4 Periods in software engineering process evolution

The third period, process maturity, introduced a meta-level to the

development processes. During this period, software process improvement (SPI) processes and models were developed and implemented [10]. The essence of SPI is to use an improvement process, functioning on a meta-level with respect to the traditional software development process. The current development process is assessed compared to a reference process model or framework and then adjusted accordingly. The most well known example of work originating from this period is the Capability Maturity Model (CMM) from the Software Engineering Institute (SEI), Carnegie Mellon University [21]. Another example is ISO 9000 and 9001 [22, 23, 24, 25]. Most of the third period models are appropriate for larger companies though the research community has made several attempts at adapting them to smaller organisation types [26, 27, 28].

The fourth period was predicted to be the industrialised software engineering phase during the third period [29]. The engineering methods used would resemble the production methods present in other more

~1970 ~1990 ~2000

Period 2: Structured methods

Period 3: Process Maturity

High Maturity

Agile Method-ologies

?Period 1: Code & Fix


24

established engineering disciplines such as civil engineering or manufacturing. This fourth period, however, is currently only beginning to take form. There has been a distinct change in the software process community with the introduction of the processes known as the agile methodologies that appear to be in contrast to the continued evolution of the maturity models towards working with higher maturity levels. Both of these have evolved from software engineering using third period methodologies but with different approaches to the problems discovered.

Table 1.3 compares an example of inherent properties of software compared to traditionally industrially manufactured items. The properties indicate some of the problems that were thought to be solvable with fourth period methods. Due to the different nature of software, however, the industrialisation may never be possible.

Software Traditional Industry 1 Build once Build all the time 2 Reproduction low cost Reproduction high cost 3 Very flexible Inflexible 4 Intangible Physically present 5 Customer often undecided Customer determined 6 High maintenance cost Rel. low maintenance cost

Table 1.3 Comparisons between inherent properties of software and traditional industrial artefacts [30]

The creation of software can be compared to preparing for industrial production of a product. Here the community has identified some of the most useful items to apply in the field of SE, such as Total Quality Management (TQM) and Plan-Do-Check-Act (PDCA) [31, 32].

The fourth period has started with a division of the SE community. The successful practitioners of the third period models have continued their improvement into high maturity practices. The companies that failed in their intent to introduce third period processes, or never even attempted this, have taken to agile methods as the natural next step. In this second group, there is a strong reaction towards unnecessary formalism and overhead in the third period processes.

The development over the next few years will show the approaches converging and being adapted to different circumstances in different environment and it will become more apparent that both groups are striving towards a common goal of producing quality software as efficiently as possible, with as much control as possible, and with content people within the organisations. To do this, influences will be needed from within the entire scope of the practice, and theory, of SE today and

25

that each organisation producing a different type of software will require a highly adapted methodology for their software type, organisation type and individual people types involved in development. This can be identified in Brooks’ early paper [33] from which we can quote the classic “There is no silver bullet for software development”.

1.3 Software Process Improvement Software process improvement (SPI) involves working with the existing software process in a company and improving it. The area can be divided into the process for performing the improvement work and the reference needed for comparison [34].

1.3.1 Software Process Improvement Methods

A generic SPI method can be simplified to three steps: start, assess and change. The SPI initiative starts by the insight that the current development process can and needs to be improved. In this first step, start, sponsorship from senior management is ensured. This is in order to make sure that support and resources are available and to spread knowledge and acceptance about the effort. During this initial step a general plan is created and the goals for the initiative are also drawn up.

Assessment of the software process in the next step, assess, concerns making an evaluation of the current process and comparing it to a reference of some kind. This can be performed both internally by the company, externally by assessors or a combination of both.

The actual change is performed in the final step, change. Typical methods for introducing change include training and introduction of new process documentation.

Typical examples of process improvement methods are SEI’s IDEAL method [35], specialized for self-assessment and capability evaluation, CMM based appraisal for internal process improvement (CBA-IPI) [36], Software Capability Evaluation (SCE) [37], and the forthcoming SPICE/ISO15504 standard [38].

The IDEAL method was developed by SEI as a way of using the CMM based appraisal method for process improvement. As the method does not refer to a specific model for comparison, this model could be used in combination with other models than the CMM. The IDEAL method is illustrated in Figure 1.5. Note the similar structure to the basic phases described earlier in this section. Here the initialising phase of the


26

IDEAL model corresponds to the start phase, diagnosing corresponds to the assess phase and establishing, acting and leveraging correspond to the change phase.

Figure 1.5 SEI IDEAL process improvement model [35]

A process improvement should be relevant, i.e. actually improve the process, as well as applicable, i.e. possible to introduce both technically and socially in the target context.

In order to be effective a process improvement should therefore contain both improvements based on the local situation as well as general improvements. This dual nature of process improvement is not reflected clearly in the SPI methods referenced. It is referred to as tailoring by some but no real guidelines are provided.

Improvements based on local situation have the positive characteristics that they are easily accepted in the organization and are also effective in that organization. They have the negative characteristics that they are most probable not found in general SPI frameworks and so can be difficult to identify. Secondly they have a low degree of transferability. A generic improvement process based on the local situation in a company is shown in Figure 1.6.

sponsorship

improvement

Stimulus for and establish

Set context

infrastructure

improvement

Establish

strategy

Set

phase results

and document

recommendations Develop

characterize

Appraise and

lessons

Plan actions

action teams

practice

current

priorities

and

Plan and

and measures

processes

Define

process Establish

and analyse

Document

installation

and track

execute

Plan,

pilots

execute Acting

Leveraging

Initialising

approach

Establishing

Diagnosing

organisational

Revise

27

Improvements based on general frameworks have positive characteristics that their need may not be apparent in the organisation despite their relevance. Secondly they have a high degree of transferability. They have the negative characteristics that they can be perceived to be forced onto the practitioners in the organisation and may not be as effective. A generic improvement process based using a general framework is shown in Figure 1.7.

Figure 1.6 Improvement based on local situation

Figure 1.7 Improvement based on general framework

The most effective SPI process therefore would attempt to use

improvements based on both the local situation and general frameworks. This hybrid process is illustrated in Figure 1.8.

Actual behaviour

Observe Improve DecideWhat & How

Actual behaviour

Compare Improve DecideHow

Reference


28

Figure 1.8 Improvement based on both local situation and general framework

A further issue here is identifying general frameworks that apply to

different context within SE. For example frameworks for small vs. large companies, product development vs. contract based development and so on. This creates an in-between to completely general frameworks such as the CMM [16] and completely local based improvement methods such as TQM [31, 39, 40].

Looking at SPI and SPI models on another level we can examine the origin of the reference framework and improvement method and how these were established. Most frameworks are established as collections of best practices based on the aggregated experience of a varying number of individuals or developing organisations. This ranges from frameworks without any source traceability what so ever to the CMM [16], which can be classified as one of the largest collections of software practices performed.

The generic SPI processes are composed of three types of steps: The first is either observe or measure, the second decide and the third do. Each step can potentially be improved using a different strategy from SwE research methodology, presented in Section 2, due to that the goal of the work involved are similar to that of the research performed:

• Observe – Qualitative research validity theory (see Section 2.2). • Measure – Quantitative research validity theory.(see Section 2.1) • Decide – using Internal & external experts or analysis, different

sources of experience and decision support methods. • Do – improve anchorage, motivation and support for process

changes.

Actual behaviour

Observe

Improve

DecideWhat & How

Compare DecideHow

Reference

29

In addition, the analogy can be extended to flexible research being

similar to non-framework based SPI and fixed research being similar to framework based SPI. The synergy effects of integrating fixed and flexible research approaches can therefore be analogous to integrating the different types of SPI methodologies. We can use this analogy to improve both the application of software process improvement in the target context as well as to improve creation of software process frameworks.

One of the most difficult steps is the decision step. Based on observations made, and potentially comparisons to a framework, a decision is taken based on the decision takers’ knowledge, experience and beliefs. The person or group making the decision as well as the apparent basis he takes the decision on affects this decision significantly. This aspects is often not included in SPI methodologies

A general process framework is created through a large number of observations and/or experiences generalised to a framework. The generalisation step involves deciding how to incorporate differences between cases, which can be difficult. The process, which is parallel in nature, is illustrated in Figure 1.9

Once created, a general process framework needs to evolve to keep up with industry practice. This process is more serial in nature and is illustrated in Figure 1.10.

Figure 1.9 Genesis of a general framework

Actual behaviour

Observe Generalise

Actual behaviour

Actual behaviour

General Framework

Observe Generalise

Observe Generalise

Different C

ontexts (similar and different in nature)


30

Figure 1.10 Evolution of a general framework

Two cases, CMM [16] and agile methodologies [9], are discussed here to demonstrate the evolution of two different software process models that are of significance in the third and fourth periods respectively. It is interesting to note the opposite direction of introduction in the two cases. In the case of agile methodologies the source of the methodology is industry and the introduction is bottom up from developer to manager whereas in the CMM case the source of the method is academia in cooperation with the US Department Of Defence (DoD).

The cases illustrate different roles that academia has taken during the evolution of two different software processes. The current trends in software processes must affect not only what the academia is researching, but also how the research is performed. For instance, in the CMM case the intent of each part of the process is clear, but the actual industrial effect is unclear and needs to be investigated. This was done for the lower maturity levels in the CMM as many companies reached the lower levels relatively quickly. The effects of the higher levels have not really been seen until the latter part of the 90’s and the content thereof can be seen as largely speculative based on what was though to be needed once the initial practices at the lower levels were in place. This has resulted in updates due to experience gathering at for example high maturity workshops [41].

In the second case however the intent of the practices are not as clear to the academic community as the formulation of the practices have not involved the academic community. As in the CMM case the actual effects of the practices need to be studied, but the fact that the models from an academic point of view seems to contain several ‘dangerous’ activities and excludes several activities proven to be effective and are still used needs to be investigated.

Framework


Actual behaviour

Framework


Actual behaviour

Framework


Actual behaviour

31

1.3.2 Challenges in SPI

Humphrey describes six basic principles that need to be followed during the course of process improvement. These provide an overview of some basic issues involved in software process change [42]: 1. Major changes to the software process must start at the top.

Senior management leadership is required to launch the change effort and to provide continuing resources and priority.

2. Ultimately everyone must be involved. Software engineering is a team effort and everyone who does not participate in the improvement will miss the benefits and may even inhibit progress.

3. Effective change requires a goal and knowledge of the current process. To use a map you have to know where you are.

4. Change is continuous. SPI is not a one shot effort; it involves continuous learning and growth.

5. Software process changes will not be retained without conscious effort and periodic reinforcement. Effort must be taken not only to introduce changes, but also to retain them.

6. SPI requires investment. SPI takes planning, dedicated people, management time and capital investment.

One of the goals of the improvement effort is to improve the

relationship between actual required resources compared to the estimation of resources. The process can be affected in two ways:

• Predictability Moving the estimation of the required resources towards the average of the actual required resources.

• Control Decreasing the variance of the actual required resources.

The official process described in a company exists in different forms depending on which phase of the process improvement we investigate it. A presentation of the different kinds of processes that can be present in an organisation is given in Table 1.4 [43].


32

Process Name Description Reference The sources from which the process

ideas originate (Theoretical papers, books, etc).

Desired The process that the person interpreting the references wants to introduce.

Official The process described on paper by the organisation.

Perceived The process that the performing subjects believe they should perform

Actual What is actually performed. Observed The observed process.

Table 1.4 Different types of the same process [43]

Each process type is related to the next by a mechanism of transfer. For

example the perceived process is obtained from the official process through teaching, peer information and self-study. The mechanisms are summarised in Table 15.

It can be seen that three of the processes shown in the table are transferred by interpretation. Interpretation is highly subjective, i.e. highly dependant on the person involved. A person’s subjective interpretation of the process is known as that person’s viewpoint [44]. From To Mechanism Description Reference Desired Interpretation Reference processes are interpreted by the

person(s) creating the official process. Desired Official Formulating The official process is formulated from the

desired process and written down as the official process.

Official Perceived Interpretation The user interprets the official process based on what he is taught, what he reads and how his co-workers influence him.

Perceived Actual Performing What is actually performed is a result of the perceived new process and the old process.

Actual Observed Interpretation The observed process is a result of an interpretation of the actual process.

Table 1.5 Mechanisms of transfer between types of processes

33

The actual process is unique in the sense that this is the process that actually creates the product. In many ways this makes it the most important process, as this is the process that we must ultimately improve. It is observed through the interpretations of observers but cannot be objectively evaluated by observation alone. The use of some form of metrics, however, offers an objective unbiased picture of the results of the actual process [45]. This indicates the importance of metrics in SPI work.

SPI involves changing existing practices, processes and structures. The study of the phenomena observed when introducing changes to a group of people belongs more to the social sciences than engineering science. It is however very important to remember that there are humans involved in the development process and any action made needs to take this fact into account. All changes must therefore be practised, improved and will then eventually become a natural part of the company’s practices after the introduction. A typical response to change and the effort exerted by the subject is illustrated in Figure 1.11.

It must also be remembered that the current process being used at a company is the result of many years of experience, and most of it is usually good practice.

Figure 1.11 Typical human response to change [10]

1.4 Process Maturity Models There are several process maturity models used in practice, both commercial and non-commercial. Without any doubt the most used is the CMM model [16]. The framework quagmire [46] shows an overview of many of the alternatives that exist. How these relate to each other also gives an idea of how they have evolved. Here the term framework is used as a collective terms for process standards, quality standards, maturity or capability models, appraisal methods and guidelines so as to compare them directly. The choice of framework is, however, often much more limited due to pre-set requirements from, for example, contractors, management, the current process and the improvement goal itself. The

Time

Stunned paralysis

Anger, Rage

Denial

Bargaining

Testing

Depression

Effort Acceptance

Status Quo


34

most influential frameworks in the quagmire are the CMM-based models [16], ISO 9000 based standards [22], and various military standards. These frameworks are also the most widely used today and many companies are assessed in several standards.

Process improvement models are not only different in their composition and content. They also have different purposes and uses. Some models are intended for use in one special project because the customer demands it, while others are intended to be used consistently across a whole organisation, such as the CMM.

The Software-CMM (SW-CMM) [16] contains a framework of practises that define different levels of maturity in the software development process. The model is composed of five levels of increasing maturity. Each level contains practises that would normally be expected in a software process at the corresponding maturity level. Organisations that do not implement CMM are said to be at level 1, the initial stage.

Each level, except for level 1, is divided into several key process areas (KPA), which describe the areas that the level addresses. For an organisation to be performing at a certain CMM level, it must successfully implement the practices described at that level and at all lower levels. The CMM can be used in five different ways [16]:

1. CMM based assessment to identify process strengths and weaknesses.

2. Evaluating subcontractors for selection. 3. Developing new appraisal methods based on the CMM. 4. Teaching upper management about SPI introduction. 5. As a guide for technical staff and process improvement groups in

their work to define and improve their processes.

Since its introduction the SW-CMM [16] has evolved in to the Capability Maturity Model Integrated (CMMI) [47], by being updated and integrated with the system engineering CMM (SE-CMM) [48]. This model is available both in a staged version such as in the original SW-CMM, or a continuous version from the SE-CMM.

When using the CMM models it should be remembered that they were created at the initiative of the Department of Defence and that their way of acquiring software is usually to select one of several contractors to complete the project at a fixed price. The projects involved are usually quite long, not market driven or of a product type and have relatively stable requirements. The software industry has changed radically during the 90s with the popularisation of the Internet and exponential growth of

35

amount of software being developed.

1.5 Agile Methodologies Agile methodologies have arisen as a reaction to the more strict processes employed during the third period of software engineering processes The development of these occurred in parallel at the end of the 90’s. The most widespread of these methodologies are:

• Extreme Programming (XP) [49, 50] XP was created by Ward Cunningham and Kent Beck and involved 12 practices that individually are not new to the SE community. The methodology became widespread around 2000 and a new updated version is currently being released.

• Cockburn's Crystal Family [51] Crystal was developed by Alistair Cockburn in the early 1990s with the primary focus of improving communication in software projects. It was intended to include three versions for different sizes of projects, but only the smallest version has been completed so far.

• Adaptive Software Development (ASD) [52] ASD was created by Jim Highsmith. The method focuses more on the philosophy behind the development approach instead of individual practices. Currently considered complementary to Crystal, where the details in Crystal complement the philosophy in ASD.

• Scrum [53] Scrum was created by Ken Schwaber. The method is considered a meta-process or ‘wrapper’, independent of the actual development methodology used. It is, for instance, quite common to use Scrum and XP in combination.

• Feature Driven Development (FDD) [54] FDD was developed by Jeff De Luca and Peter Coad. FDD development is a process designed to produce frequent, tangible, results and is a collection of best practices.

• Dynamic System Development Method (DSDM) [55] DSDM is a framework focused on delivering a quality solution quickly. The method focuses specifically on


36

incremental development, user involvement and most important first.

1.5.1 The Agile Manifesto

The Agile methodologies have many common aspects and in 2001 a core group of people from the agile community formulated the agile manifesto [9] describing the most fundamental aspects of agile development. The principles of the agile manifesto are shown in Table 1.4. These principles indicate the difference in focus of agile methodologies compared to the maturity models. Agile methodologies focus on [9]:

• Individuals and interactions over processes and tools. • Working software over comprehensive documentation. • Customer collaboration over contract negotiation. • Responding to change over following a plan.

This means that there is still value placed in the practices on the right,

but those on the left are valued more. A common misconception of the agile methodologies is that the practices to the right of the list above are strictly forbidden. This is not the case. All of the methodologies described in the previous subsection share these values and the principles behind them. Indeed, it is possible to use only the principles to influence the current practice in an organisation without actually using any of the methodologies mentioned.

The agile methodologies recognise the importance of making the path from the person(s) that are familiar with what functionality is desired (i.e. the customer) to the developers that are going to produce the product as short as possible in order to make communication as quick and effective as possible. As the agile methodologies propose a different perspective on scheduling development, based on fixing time and resources and varying functionality, changes to requirements late in the development cycle has no effect on the schedule. This different focus of scheduling is illustrated in Figure 1.12. By maximising work not to be done the team can focus on the most important functionality without being distracted by less important features.

37

Principles of the Agile Manifesto:

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

7. Working software is the primary measure of progress. 8. Agile processes promote sustainable development. The sponsors, developers,

and users should be able to maintain a constant pace indefinitely. 9. Continuous attention to technical excellence and good design enhances agility. 10. Simplicity, the art of maximizing the amount of work not done, is essential. 11. The best architectures, requirements, and designs emerge from self-organizing

teams. 12. At regular intervals, the team reflects on how to become more effective, then

tunes and adjusts its behaviour accordingly.

Table 1.4 Principles of the Agile Manifesto [9]

By focusing on delivering an actual product in working software as quickly as possible and consequently delivering increments often, a concrete measure of progress is achieved compared to traditional methodologies. Agile methodologies also recognise the need for good communication both within the development team and with management. This is best achieved through frequent face-to-face communication, a much richer form of communication compared to documents. Through communication trust is established between team members as well as between the team and management. Agile methodologies promote self-organisation and self improvement in the development teams. This makes sure that improvement is performed at the level where the product is created and not imposed from above.

The development team is expected to work at a pace which they can sustain indefinitely. It is considered that overtime or death march projects only lead to lower quality products, increasing the lead time and cost in the long run. As the software product is a technical product it is essential to promote technical excellence and good design.


38

Figure 1.12 Requirement flex in Agile Development [55]

These agile approaches have only been in widespread use since around 2000 as a result of the amount of media attention received by the extreme programming methodology and we have only recently begun seeing experience reports on the usage of such methods [56, 57]. A brief introduction to Extreme Programming is provided in the introduction to Paper 3 in this thesis and a complete case study on introducing Extreme Programming is provided in Paper 2.

1.5.2 Effects on the Development of Software Systems

The effects on SwSE and SwE of introducing an agile approach in the software development will, of course depend on many factors. For instance whether the change is introduced in a small group within the software developers’ organisation or if all the software developers in a large project are to use a more agile approach. Another factor will be the extent of software in the product being developed. If the product is largely hardware with an embedded software part, the software development must be strongly coordinated with the hardware development. The fixed nature of hardware makes hardware development a less volatile process.

If we examine the four aspects of the agile manifesto we can discuss how each of these in turn is thought to affect the development of the software system. The issues discussed will affect the software system development in different ways in specific instances, but the most general principles will most probably be similar.

Functionality

Time Functionality

Resources Time

Resources

Fixed

VariableTraditional

Agile

39

Individuals and interactions over processes and tools The shift of focus towards interactions between humans must ensure that communication previously handled by the process and its artefacts must still be in place in order to successfully develop the product. Instead of putting that responsibility on the process, the responsibility is given to the people working in the project. Empowering them to optimise the way they work together. Tools will be necessary to complete certain tasks and interactions although it is the completion of the tasks and results of interactions per se that are truly important. An example of easing interactions between people can be to make sure that a working version of the product is available to discuss over. Working software over comprehensive documentation The shift from a document oriented development strategy to a software-oriented strategy may sound dangerous to many system development organisations. Documentation is required to synchronise development between development units, both software and non-software, for future maintenance of the system. Often documentation can be for the sake of complying to quality standards themselves. The shift implies reducing unnecessary documentation. Documents that are not updated as the product evolves or document the code line by line are examples of candidates for unnecessary documentation, but documents can still be essential when fit for their purpose Customer collaboration over contract negotiation The shift towards communicating with a customer stakeholder is part of increasing the feedback rate within the project. Instead of the project being one big feedback loop, with the goal set at the beginning and feedback provided by the stakeholder at delivery, the project becomes a series of rapid feedback steps with feedback provided as often as is effective. The SwSE environment implies many complex customer relationships, e.g. software groups may be customers to hardware groups and visa versa, and it may be important to have customer groups and customer group coordination activities as well as an extended project community. Responding to change over following a plan The shift towards responding to change over following a plan seems contradictory to the ordered flow shown in Figure 1.1 and it is a large challenge to introduce this focus shift in such a context. However, if all


40

units producing the product are tightly coordinated with short feedback loops and proceeding at optimal pace, then the plan becomes less important than ensuring that the project continues to proceed optimally. This is not saying that a software system should be developed without a plan, plans are important to further coordinate different units within the product.

1.6 Project Management Models A project management model (PM) defines a framework for setting up and managing a project [18]. The framework as such is general, applicable to any kind of project. The PM focuses on monitoring, management and decision issues on a business level, rather than the specific technical work, and typically defines:

• A set of phases. • Criteria for decisions related to the phases. • Stakeholder roles. • Project management artefacts.

In Figure 1.13, the principal constituents of a generic PM are outlined.

In this model, there are four phases; prestudy, feasibility study, execution and completion phases. This is the terminology used in the PROPS model by Ericsson [19] but most PM models used have phases with similar intent. Project progress is measured through the passing of predefined milestones (MS) and at the end of each phase a tollgate (TG) must be passed, which involves a decision whether the project is to continue and be assigned further funding, or if it is to be terminated.

Stakeholder roles are defined within the PM, such as project manager, project sponsor, quality assurance manager, and for the internal project organization, subproject managers. The customer stakeholder is typically only involved at the start of the project and after delivery. Project documentation standards are defined in the PM, such as the content and format of project specifications, project plans, quality plans, progress reports, etc.

41

Figure 1.13 Stage-Gate™ model [18]

In order to control the actual software development in the project, the

organization integrates the management model with a software development process. This process is much more technically oriented and fits into the specific application domain for the project. There may be several sub-processes for different logical parts of the software and even sub-processes for hardware development that must be coordinated. The general phases and decision criteria are defined in terms of a technical content and technical documentation is added and linked to the different phases.

The project management model can thus be seen as a layered model. The technical development process is still on a rather high level of abstraction even though it describes work of technical nature.

An incremental approach can be defined, where development products are split into increments, delivered to other stakeholders according to a defined increment plan [58]. Milestones are defined for each of these incremental deliveries similar to those shown in Figure 1.13, and these are reported and monitored on the management level using the same reporting mechanism.

Launch

Discovery

Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

Post launch review

G2 G1

G3 G4 G5

Scoping Build business case

Development Testing & validation


42

2 Research Methodology in Software Engineering

Research is the expansion of human knowledge, and research methodology deals with methods for doing this. Traditionally, empirical research methodology has been divided into two main paradigms, the qualitative and the quantitative methodologies. Each has been used with success in different disciplines and applications [59]. These paradigms can also be referred to as fixed and flexible designs respectively [59, 60]. These terms regard the amount of preparation performed by the researcher before the study takes place.

Research can be performed from various philosophical perspectives. These can be said to be positivist, interpretive and critical. Positivist research assumes that the researcher can perform the research without affecting the studied subject or environment. It also assumes that the subject can actually be measured upon and is accessible for direct measurement. Interpretative research assumes that the subject can only be accessed indirectly through tools such as language and nomenclature. Critical research assumes that social reality is historical and that it is produced and reproduced by people. This research focuses on the oppositions, conflicts and contradictions in society and strives to solve these.

Examples of other distinctions made in research are: objective versus subjective, being concerned with general laws (nomothetic) versus being concerned with uniqueness of each particular situation (idiographic), as aimed at prediction and control versus being aimed at explanation and understanding, as taking an outsider (etic) versus taking an insider (emic) perspective [61].

When preparing a research project the researcher must choose a research methodology. This can be done by discussing the purpose of the research and the epistemological status of the area. If there is a low level of previous knowledge, an exploratory study is probably the best selection of methodology. If a little more is known and a more detailed result is required, a descriptive methodology might be the better selection. If a lot is known and relationships are to be confirmed an explanatory methodology is probably the best selection. This can be illustrated as a research method staircase, as shown in Figure 2.1. In the figure, the type of knowledge in the research area changes as you go up the staircase.

43

Qualitative methods are generally lower in the staircase and quantitative methods are generally towards the upper part of the staircase [62]. Table 2.1 shows appropriate strategies for research depending on the aim of the research.

Figure 2.1 Research method staircase [62]

Type Research aim Requires control

over behavioural events?

Experiment How, why

Yes

Survey Who, what, where, how many, how much

No

Case study How, why

No

Table 2.1 Relevant situations for different research strategies, adapted from Yin [63]

2.1 Quantitative Research Quantitative research has evolved through the natural sciences in order to explore the world. This scientific approach has the common feature of starting with a theory, which is then proven or disproved by an experiment. The scientific researcher is concerned with a certain theory and only this theory during the course of testing it. This implies that there is already much knowledge about the researched area and explicit relationships are to be proven as opposed to a more general exploration of the area. Figure 2.2 shows an overview of quantitative research. The researcher starts by creating a theoretical hypothesis that is then proved or disproved by performing an experiment. If the theory does not hold, the researcher can revise the theory and perform a new experiment [64].

Form

Scope, Frequency Explore

Describe

Explain

Relationship X->Y


44

Figure 2.2 Quantitative research overview

The quantitative approach can be said to involve 5 steps [59]:

1. Deducing a hypothesis. 2. Expressing the hypothesis. 3. Testing the hypothesis. 4. Examining the outcome. 5. Eventual modification of the theory and return to the first step.

As can be seen these steps create a very orderly and rigid research

procedure with clearly defined steps that are easily followed. The first two steps are performed in the theory part of Figure 2.2, and the third is performed in the practice part. The fourth step is comprised of comparing the results from the practical experiment to the theory. Typical examples of quantitative research methods are experiments and surveys.

Experiments are a form of research performed in a very controlled environment, for instance a classroom environment, where complete control of all variables is possible. The method is intended to make the practical observation environment as similar as possible to the constructed theory. The experimentation process is illustrated in Figure 2.3. The theoretical construction created by the researcher is divided into the cause construct, the cause-effect construct and the effect construct. This theory is then tested in the controlled environment. The controlled independent variables are subjected to the treatment and the results measured. The actual outcome is compared to the effect construct and based on this the hypothesis is rejected or accepted. In cases where true experimentation is not possible due to lack of control of one or more variables, quasi-experiments can be performed instead [59, 64]. A comprehensive treatment of experimentation within the field of software engineering is provided by Wohlin et al. [64], Juristo and Moreno [65] and Kitchenhham et al. [66].

Theory Practice

2 1

45

Figure 2.3 Quantitative Experiment Research Principles [64]

Surveys are usually more exploratory or descriptive in nature. This implies that survey methods belong closer to the second, describe, step in Figure 2.1 than to the first, explore, step. Methods of data collection in surveys are mostly questionnaires or interviews. Surveys however focus on the aggregation and analysis of the questionnaires and/or interviews and are in this way more quantitative in nature. Surveys can be used to provide a snapshot of a given situation. If the focus of the study is on examining different groups in the sample, the study is a cross-sectional study. If the same sample is surveyed at different points in time the study is known as a longitudal study [59].

In quantitative research, validity is traditionally divided into four different areas: Conclusion validity, internal validity, construct validity and external validity. Each area contains a series of threats that need to be addressed. Not all threats apply to all quantitative methods and some threats even contradict each other. In this case the researcher must decide which threats are vital to eliminate for the specific results that he needs. These threats have been mapped into the experimentation process shown in Figure 2.3 by Wohlin et al. [64] as follows:

• Conclusion validity is relevant between the treatment and

outcome, ensuring that the relationship between these is significant.

Cause construct

Treat-ment

Outcome

Effect construct

Independent variable

Dependent variable Experiment

operation

Experiment objective

Cause-effect construct

Treatment-outcome

Theory

Observation


46

• Internal validity ensures that the relationship between the treatment is not due to a variable we do not have control over.

• Construct validity ensures that the relationship between the theoretical constructs and the actual treatment and outcome is correct.

• External validity ensures that the results from the study are generalisable to a greater context than that in which the experiment was performed.

These threats provide a complete framework for evaluating the results from a performed experiment as well as guidelines for improving the experiment itself.

2.2 Qualitative Research Qualitative research has evolved from the social sciences to explore the social and cultural world. The qualitative approach, in contrast to the quantitative research methods usually start with a very vague idea of the theory and examines the practical world to understand and further develop the theory. During the research the researcher is interested in actively finding facts that will change the theory dynamically, as this will improve the theory, in contrast to the quantitative researcher who would have to discard the theory and start over. The qualitative research methods claim to better deal with ‘messy’ real world research involving complicated scenarios as a lower degree of control over variables is required. The researcher, having only a vague idea of the theory, looks for a contribution to his theory in the practical part of the diagram first. Each time a contradiction occurs the theory is adjusted and the work carries on. If enough subjects have been examined without the theory changing significantly, the researcher can start to consider whether the research is approaching a concluding state or not [61].

Typical examples of methods that are used for qualitative research are case studies, evaluations, observations, interviews and questionnaires. It should be mentioned for the sake of clarity that some of these techniques can also be used for quantitative research.

In qualitative research a different terminology is used for validity compared to in quantitative research. Although some of the same basic problems as in quantitative research need to be addressed, the prerequisites are different depending on the fundamental differences between the two research types. In qualitative research the following four

47

categories are used to address validity: Credibility, transferability, reliability and confirmability [59].

Credibility can be compared to internal validity in the quantitative methodology. The main question to be answered here can be formulated as: Are the results correct given the observed situation?

Transferability can be compared to external validity in quantitative research. It determines whether the conclusions of the study have applications in a wider context than the one observed in the study.

Reliability concerns the actual process of study. A reliable study method is consistent, independent of time, researchers and methods.

Confirmability confirms that the results depend on the subjects and conditions of the study rather than being dependent on the enquirer. The issue here is if the study could be re-performed using a different researcher.

Figure 2.4 Qualitative Research overview

2.3 Research on SE Processes The research performed in this thesis is performed in close cooperation with industrial software development organisations. There are aspects of software engineering that cannot be researched in a laboratory environment as a controlled experiment. When performing research on software engineering processes we must separate the concepts of context for, and control of, the research. The context for the research can be either in the field, i.e. in a real development organisation or in a lab environment recreated purely for research purposes. The control of the experiment denotes the power the researcher wields over the subject of study.

Software engineering research is an applied research discipline, it would not exist without the industry of software engineering. Of course, there are areas that benefit from being isolated from the industrial setting and investigated by research in a controlled environment. A combination of experiments performed in the field and controlled experiments

Theory Practice

2 1


48

performed in a lab setting and using the most appropriate approach or combination of approaches depending on the nature of the research would be the optimal strategy.

An experiment or survey in the field is harder to create valid results from but can provide entirely different kinds of results compared to a laboratory experiment. It is equally important to sternly address validity when performing experiments or surveys in the field whether they are qualitative or quantitative. In software process research, where the practice is so tightly connected to organisational and human factors, it is difficult to recreate environments and situations in the laboratory that are realistic. Therefore fieldwork is very important in this area.

Other areas within SE are easier to perform controlled experiments on, for example certain verification and validation situations can be recreated in a constructed situation and are therefore suitable for controlled experimental methodologies. This has implied that this area and others similar to it have been more extensively explored than areas that are by their nature more difficult. However, the need for research is equally large in all areas so it is imperative that the community does not only pick easy targets for research and makes an effort to perform research as needed and not as possible in the areas [66].

2.4 Effect of Agile Methods on Research Practice Within the agile community, few experimental studies and limited case studies have been performed so far. This leaves the explore and describe steps discussed previously in this Section largely unutilised. Thus those few experiments that are actually performed, are performed without a preceding thorough understanding of the general concepts within the domain.

Problems apparent when performing research on software development in the field compared to an isolated experimental environment include lack of control of significant variables. Organisations are under pressure to complete products and are unwilling to try new, untried methods. Politics also affect method decisions in companies, for instance a new manager can introduce a new method in order to make his mark. Different motivators for method effectiveness have been identified between managers and developers. Developers are more convinced by the local practical effectiveness of the method whereas management are more convinced by general measured effectiveness [67]. Overall, as research is performed close to the practice, human factors such as viewpoints [Paper

49

1, 44] become significant threats to validity if not addressed in the research methodology.

On the exploratory step of the research staircase presented in Figure 2.1, we are studying more general concepts, whereas on the third step we are extracting the relationships to be studied from the context, excluding the effects of many of the uncontrollable variables. Research in between these extremes, be it quasi-experimental or quasi-exploratory, demands contemplation on how to address validity threats. Validity in these quasi-methodologies must be addressed from both qualitative and quantitative angles in order to ensure completeness and a thorough understanding of the concepts involved is vital.

A case study of observational nature can suffice to both increase the academic community’s understanding of important aspects of an industrial setting in the way described by Potts [68] and also return improvement suggestions based on both new research and suggestions of established results [Paper 4]. Observational case studies can of course only serve as a tool to increase knowledge and understanding when exploring new phenomenon [Paper 2].

The low number of published studies on agile methodologies means that a general lack of general knowledge compared to the knowledge base of more traditional methods. The generalisability of for example the Capability Maturity Model [16], evolved through a huge qualitative study involving feedback from hundreds of industrial software organisations is of an entirely different nature to that of the entire collected research performed on agile methodologies thus far. Research in the field of system engineering and agile methods is in comparison nonexistent.

One solution that could achieve larger scale studies and thereby increased generalisability would be coordinated exploratory research, but this puts demands on the research community to communicate through research networks such as the European network of excellence Network for Agile Methodologies Experience (NAME) [69], or the International Software Engineering Research Network (ISERN) [15].

3 Research Presentation This section presents the research contained in the papers that constitute the main contribution of this thesis. The strategies that have been used to perform the work are presented as well as the contributions of each paper. The research is summarised in Table 3.1.


50

3.1 Paper Contributions and Strategies Paper 1 - Aggregating Viewpoints for Strategic Software Process Improvement – A Method and a Case Study Karlström, D, Runeson, P. , Wohlin, C., Published in IEE Proceedings Software, October 2002. This study investigates SPI in a very large software developing company using CMM framework based process improvement [36]. The study attempts to identify elements external to the framework that might not otherwise have been thought of by performing a case study in order to identify candidate improvement suggestions in a qualitative manner. An analytical hierarchy (AHP) [70] survey is conducted among the staff in order to rank the suggestions. The study contributes a method of involving the entire organisation directly in a process improvement initiative as well as investigating existence of possible local improvement factors as discussed in section 1.3.1. Paper 2 - Introducing Extreme Programming - An Experience Report Karlström, D., Published in proceedings of the third International Conference on Extreme Programming and Agile Processes in Software Engineering, 2002. In this study we have investigated XP [49, 50] in an industrial application at a small software developing company. The study was performed shortly after the widespread publication of XP and was therefore one of the first structured case studies conducted of XP application experiences. The results of this study have been related to directly by Tessem [71] in a three week qualitative study in a more controlled environment. Our study was performed as a case study using interviews and observations to collect data. Paper 3 - Decision Support for Extreme Programming Introduction and Practice Selection Karlström, D., Runeson, P., Published in proceedings of the fourteenth International Conference on Software Engineering and Knowledge Engineering, 2002. This study applies the survey methodology used in paper 1, AHP [70], to the problem of introducing XP [49, 50] practices in a company. By involving all practitioners in grading the practices, a suitable subset can be selected if partial introduction is desired. If complete introduction is

51

preferred, which is currently the accepted best practice in the XP community, it is easy to identify which practices are thought to require more introduction support. The method is employed at the same small company as in paper 2 as well as on a student control group. Paper 4 - Incremental Introduction of a Structured Test Practice Framework in Emerging Software Development Organisations Karlström, D., Runeson, P., Nordén, S., Accepted for publication in the Journal of Software Testing Verification and Reliability, preliminary publication issue: June 2005. This study investigates improving the sub set of practices regarding testing in a small Internet software development company. The company was expected to grow rapidly at the time and was thus concerned that all practices introduced would be scalable to suit their needs. A framework of test practices and an introduction method are identified using a case study. These are applied at the company and the experiences of the first stage are gathered. A survey of 12 other software-producing companies is performed to investigate the transferability of the framework. The study combines qualitative research of observatory nature with interviews and a survey as well as looking at quantitative indicators of the effects of the framework. Paper 5 - Agile Methods and Software Product Management – An Integrative and Hybrid Approach Runeson, P., Karlström, D. Technical Report, Lund Institute of Technology, CODEN : LUTEDX (TETS-7203) / 1-34 / (2004) & local 16, 2004. This paper contains a theoretical integration of stage-gate software product management [18] and agile methodologies [9]. A general background is provided as well as the actual hybrid model proposed through analytic reasoning. Finally an example of how the model would function at one tollgate in the staged management approach, tollgate 2, is drawn up as a concrete example. The paper is to be considered as an introduction to the work in papers 6 and 7. Paper 6 - Integrating Agile Software Development into Stage-Gate Managed Product Development Karlström, D., Runeson, P., Submitted to the Journal of Empirical Software Engineering, 2004.


52

This study is an inductive qualitative study investigating the experiences of integrating pilot XP teams [49, 50] into native stage gate management models [18] at two large software development companies building on the theories discussed in Paper 5. Success factors and key experiences are identified and presented. Two different teams at two different companies are studied using interviews and archival analysis of internal documents. Paper 7 - Exploring Agile Influences in Stage-Gate Management of Software Product Development Karlström, D. Submitted to IEEE Transactions on Engineering Management, 2004. This study is an inductive qualitative study investigating the possibilities of integrating agile concepts [9] into the staged product management process [18] at a large software producing company. The perspective in this study has been from the programme management perspective rather than developer and project management perspective in paper 6. This new perspective was identified as relevant during the study presented in paper 6. Possible guidelines improving the way of working in the product management process are identified and suggested together with improvement strategies.

3.2 Technology Transfer The most obvious effects of the research involved is at the companies involved in the studies. The results at each company are accounted for in the research papers. The fact that the companies actually use the research results is considered a validity measure for the research. Also, mutual gain for the company and the researcher implies a better research connection between the parties and therefore more accurate results. Secondly, technology transfer is achieved through the publication of the papers in journals, at conferences and in this thesis. During the final stages of the research for this thesis a course was given on agile software development with correspondents from both industry as well as the university graduate student. The research group also holds seminars for the local software industry, thereby disseminating research results into the community. Networking is important to this strategy and the research group is active in the local software process improvement network (SPIN).

53

Paper Company

involved Size Process

type Strategy Research

type Methods used

1 Fuji Xerox Large Trad. Qual. & Quant.

Case study & Survey

Observations, Archival analysis, AHP

2 Online Telemarketing

Small Agile Qual. Case study Observations, Interviews

3 Online Telemarketing

Small Agile Quant. Survey AHP

4 Scalado, SPIN-Syd1

Small Trad.. Qual. & Quant.

Case study Observations, Participation, Interviews

5 Ericsson Large Agile Qual. Theoretical Analytic reasoning

6 ABB Automation Products, Ericsson Microwave

Large Agile Qual. Case study Interviews, Archival analysis

7 Vodafone Large Agile Qual Case study Interviews, Archival analysis

Table 3.1 Research strategies, types and specific methods used in the papers presented in the thesis

3.3 Further Research This section describes further research that can be performed within the area for this thesis. Some of the research is closely related to the contents of one or more papers as they are a continuation or refinement of the research performed and may therefore even be mentioned in the actual paper. Other further research suggestions are in related areas that continue the research in a different context.

3.3.1 Overall Research Direction

The current trend in software engineering processes with agile methods contrasting to higher maturity levels in maturity frameworks will evolve and change the way that software is produced. It is natural to continue to

1 For the survey part of this study, 12 companies from the local SPIN group were involved.


54

study this evolution using various methods. It is necessary to perform more controlled experiments on specific areas of agile methods, such as those performed on pair programming [72, 73, 74]. In order to study a complete software process in its context however, it is also necessary to continue to use case-study methodology to follow the evolution in industry. Qualitative research in agile software processes and their improvement has so far been performed in relatively limited scopes compared to sociology studies [59]. A desirable goal for these studies would be to coordinate them further in order to be able to produce a much larger picture of current industrial practice. As these methods have evolved from industry instead of from the academics it is even more important to use case-study research to keep the academic community up to date. This could be coordinated through an Agile Alliance programme [75] using a series of workshops at the major agile conferences currently organised several times a year.

3.3.2 Research Refinement and Continuation

These further research areas describe research that involves a direct continuation or refinement of the research performed in the papers presented in the thesis.

Paper 1: Investigate how an increase in maturity spreads through a large organisation during a prolonged SPI initiative. When introducing a new process or a process change in a large organisation this is impossible to do throughout the entire organisation at once. A natural continuation of the research performed in the study is to attempt to map the spreading of a process initiative and process attitudes throughout a large organisation and compare this to the intended spread of the process. Paper 2: Identify and study further XP teams, gathering experiences, structuring and coordinating these with other case studies and experiences. This is especially interesting now that Beck is publishing a revised version of the original XP book with the lessons learned during the first four years of widespread use in the community [76]. A natural question is whether XP as a practice has evolved in the way that is described in the revised book as well as how the revised methodology can be adopted by non-agile teams

55

Paper 3: By performing the survey at a large number of companies, both currently performing XP and not performing XP, it would be possible to attempt to identify patterns in the perceptions of the practices depending on the individual context of the companies. Factors that might affect practice profiles might be product type, number of employees, market situation, etc.

Paper4: (a) - By continuing to study the application of the framework at Scalado we can investigate the effects of the higher levels of the framework in the same context as in the original study.

(b) - By assessing the actual introduction of the practice framework at additional companies we can validate the gereralisability of the framework further than was possible using the survey method used in the paper.

(c) – Further evaluate the SPI strategy of starting entirely from scratch within different process areas than testing. Examples of relevant areas would be requirements, development and external software acquisition. Papers 5, 6 & 7: (a) - By continuing to study the companies involved in our case studies we can assess the long-term effects of using the methodologies within these companies. Although most of the groups studied have been restructured, the companies are planning further teams using agile methodologies in the future.

(b) – By studying further companies using a similar research strategy we can further assess how agile teams work in combination with stage gate models and how the agile principles work at a management level. This should be done in cooperation with other research groups in order to increase the validity of the studies. This research path is considered the most important further research point as much software development is done in stage gate managed models and there is currently not much research on combining these models with agile methodologies.


56

4 References [1] Abran, A., Bourque, P., Tripp, L., (Editors) “Guide to the Software Engineering

Body of Knowledge”, SWEBOK IEEE – Ironman Version, 2004. [2] Thayer, H., “Software System Engineering: a Tutorial”, IEEE Computer April

2002, pp. 68-73. [3] IEEE Std. 1220-1998, Standard for Application and Management of the

System Engineering Process, IEEE Press, 1998. [4] IABG (1992). The V-model – General Directive 250, Software Development

Standard for the German Federal Armed Forces, August, IndustrieanlagenBetriebsgesellschaft mbH-IABG; Ottobrum, Germany.

[5] Sommerville, I., “Software Engineering”, Addison-Wesley, 2001. [6] Royce,W., W., “Managing the development of large software systems: concepts and

techniques”, Proceedings IEEE WESTCON, August 1970. [7] Liu, S., “SOFL: A Formal Engineering Methodology for Industrial Applications”,

IEEE Transactions on Software Engineering, Vol. 24, No. 1, January, 1998, pp. 24-45.

[8] Crnkovic, I., Larsson, M., Luders., F., “State of the practice: Component-based

software engineering course.” In Proceedings of 3rd International Workshop of Component-Based Software Engineering. IEEE Computer Society, January 2000.

[9] http://agilemanifesto.org last accessed 041103. [10] Zahran, S., “Software Process Improvement - Practical Guidelines for Business

Success”, Harlow, England: Addison-Wesley,1997. [11] Boehm, B., W., “A spiral model of software development and enhancement”,

IEEE Computer, Vol.21, No 5, May 1988, pp. 61-72. [12] Whittaker, J. A., Voas, J., M., “50 Years of Software: Key Principles for

Quality”, IT Professional, November/December 2002, pp.28-35. [13] Brooks, F., P., Jr., “The Mythical Man-Month”, Addison Wesley, 1995. [14] IEEE Standard Glossary of Software Engineering Terminology 610.12-1990.

In IEEE Standards Software Engineering, 1999 Edition, Volume One: Customer and Terminology Standards. IEEE Press, 1999.

57

[15] International Software Engineering Research Network, http://www.iese.fhg.de/ISERN/, last accessed 041103.

[16] Paulk, M.,C., Curtis, B., Chrissis, M., B., and Weber, C., V., “Capability

Maturity Model for Software,Version1.1”, Software Engineering Institute, CMU/SEI-93-TR-24, February 1993.

[17] Boehm, B., W., ”Software Engineering Economics”, Prentice Hall, 1981. [18] Cooper, R. G., Winning at New Products, 3rd edition, Perseus Publishing,

Cambridge, MA, 2001. [19] Mulder, L., “The importance of a common project management method in the

corporate environment”, Blackwell publishing: R&D Management, 27(3):189-196, 1997.

[20] Wallin, C., Ekdahl, F., Larsson, S., “Integrating Business and Software

Development Models”, IEEE Software, November/December 2002, pp. 28-33.

[21] Paulk, M. C., Weber, C. V., Curtis, B., “The Capability Maturity Model:

Guidelines for improving the Software Process”, Addison Wesley, 1995. [22] ISO9001:2000 International Organisation for Standardization, 2000. [23] Hass, A. M. J., Johansen, J., Pries-Heje, J., “Does ISO 9001 increase software

development maturity?”, Euromicro Conference, 1998. Proceedings 24th , Volume: 2 , 25-27 Aug. 1998 pp. 860 – 866.

[24] Rozman, I., Vajde Horvat, R., Gyorkos, J., “United view on ISO 9001 model

and SEI CMM”, of the 1994 IEEE International Engineering Management Conference, 1994. 'Management in Transition: Engineering a Changing World', Proceedings, 17-19 Oct. 1994 pp. 56 – 63.

[25] Stelzer, D., Mellis, W., Herzwurm, G.,, ”Software process improvement via

ISO 9000? Results of two surveys among European software houses” System Sciences, 1996., Proceedings of the Twenty-Ninth Hawaii International Conference on , , Volume: 1 , 3-6 Jan. 1996, pp. 703 – 712.

[26] Laryd, A., Orci, T., “CMM for small companies – Level 2”, Umeå University,

Department of Computing Science, UMINF 00.10, ISSN-0348-0542. [27] Grunbacher, P., “A software assessment process for small software enterprises”,

EUROMICRO 97. 'New Frontiers of Information Technology'., Proceedings of the 23rd EUROMICRO Conference , 1-4 Sept. 1997, pp. 123 – 128.

[28] Straub, P., Guzman, D., “Incremental, collaborative software process

improvement in a tiny software group”, Computer Science Society, 2002.


58

SCCC 2002. Proceedings. 22nd International Conference of the Chilean , 6-8 Nov. 2002, pp. 187 – 194.

[29] Lai, R., “The move to mature process”, IEEE Software July, 1993, pp. 14-17. [30] Matsumoto, Y. , “A Software Factory: An Overall Approach to Software

Production,”, Tutorial: Software Reusability, P. Freeman, ed., IEEE CS Press, 1987.

[31] Deming, W., E., “Out of the Crisis”, Cambridge University Press, 1986. [32] Humphrey, W.S., “Software and the factory paradigm”, Software Engineering

Journal , Volume: 6 , Issue: 5 , Sept. 1991, pp. 370 – 376. [33] Brooks, F., P., Jr., “No Silver Bullet - Essence and Accidents of Software

Engineering” IEEE Computer, April 1987, pp. 10-19. [34] Dowson, M., “Software Process Themes and Issues”, Proceedings 2nd

International Conference on the Software Process, pp. 54-62, 1993. [35] McFeely, R., “IDEAL: A User’s Guide for Software Process Improvement”,

SEI Handbook CMU/SEI-96-HB-001, February,1996. [36] Dunaway, D. K. , Masters, S., “CMM - based appraisal for internal process

improvement (CBA IPI): Method description”, SEI Technical Report CMU/SEI-96-TR-007, 1996.

[37] Byrnes, P., Philips, M., “Software Capability Evaluation Version 3: Method

Description”, SEI Technical Report CMU/SEI-96-TR-002 1996. [38] ISO/IEC, “Information Technology – Software Process Assessment – Concepts

and Introductory Guide”, DTR 15504-1, May 1998. [39] Juran, J., M., Juran on Leadership for Quality, New York, Free Press, 1989. [40] Ishikawa, K., What is Total Quality Control? The Japanese Way, Englewood

Cliffs, NJ, Prentice-Hall, 1985. [41] Paulk, M. C. , “Practices of High Maturity Organizations,” The 11th Software

Engineering Process Group (SEPG) Conference, Atlanta, Georgia, 8-11 March 1999.

[42] Humphrey, W. S., “Managing the Software Process”, Addison-Wesley, 1989. [43] Bandinelli, S., Fuggetta, A., Lavazza, L., Loi, M., &Picco, G. P. “Modelling and

improving an Industrial Software Process”, IEEE Transactions on software Engineering, 1995 Vol. 21, No. 5, pp. 440-454.

59

[44] Sommerville, I., ”Managing Process Inconsistencies Using Viewpoints”, IEEE Transactions on Software Engineering vol. 25, No. 6, 1999, pp. 784-799.

[45] van Solingen, R., Berghout, E., ‘”The Goal/Question/Metric Method”

McGraw-Hill International (UK) Limited, 1999. [46] Software Product Consortium http://www.software.org/quagmire/, homepage

visited 041103. [47] CMMI Product Team, CMMISM for Software Engineering, Version 1.1,

Staged Representation, Technical Report CMU/SEI-2002-TR-029. [48] Bate, R., Kuhn, D., Wells, C., Armitage, J., Clark, G., Cusick, K., Garcia, S.,

Hanna, M., Jones, R., Malpass, P., Minnich, I., Pierson, H., Powell, T., Reichner, A., “A Systems Engineering Capability Maturity Model, Version 1.1” SECMM-95-01, CMU/SEI-95-MM-003, Software Engineering Institute, Carniege Mellon University, 1995.

[49] Beck, K., “Extreme Programming Explained: Embrace Change”, Addison

Wesley, 1999. [50] Beck, K., “Embracing Change with Extreme Programming”, IEEE Computer,

October 1999, pp. 70-77. [51] Crystal Methodologies, http://www.crystalmethodologies.org, homepage

visited 2004-10-16. [52] Highsmith, J., A., “Adaptive Software Development: A Collaborative

Approach to Managing Complex Systems”, Dorset House, 2000. [53] Schwaber, K., Beedle, M., “Agile Software Development with SCRUM”,

Prentice Hall, 2001. [54] Palmer, S., R., Felsing, J., M., “A Practical Guide to Feature-Driven

Development”, Prentice Hall PTR, 2002. [55] Stapleton, J., “DSDM Dynamic Systems Development Method: The Method

in Practice”, Addison-Wesley, 1997. [56] IEEE Software, “Reports from the Field”, Special Issue: Extreme Programming

Experiences, Volume: 18 Issue: 6 , Nov.-Dec. 2001. [57] IEEE Software, “Extreme Programming”, Special Issue: Extreme

Programming, Volume: 20 Issue: 3, May-Jun. 2003. [58] Gilb, T., Principles of Software Engineering Management, Addison-Wesley,

1998. [59] Robson,C., “Real World Research”, Blackwell Publishers, Oxford, 2003.


60

[60] MacDonald, G., Evaluating the effectiveness of social interventions. In A. Oakley

and H. Roberts, eds., Evaluating social interventions: a report of two workshops funded by the economic and social research council. Ilford: Barnado’s.

[61] Myers, M. D. "Qualitative Research in Information Systems," MIS Quarterly

(21:2), June 1997, pp. 241-242. MISQ Discovery, archival version, June 1997, http://www.misq.org/misqd961/isworld/. MISQ Discovery, updated version, last modified: http://www.auckland.ac.nz/msis/isworld/.

[62] Rosengren, K.E., Arvidsson, P., “Sociologisk metodik”, Almqvist & Wiksell,

2001. [63] Yin, R., K., “Case Study Research Design and Methods”, Sage Publications,

Beverly Hills, California, 1994. [64] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., Wesslén, A.,

“Introduction to Experimentation in Software Engineering”, Kluwer Academic Publishers, 2000.

[65] Juristo, N., Moreno, A. M., Basics of Software Engineering Experimentation,

Kluwer Academic Publishers, 2001. [66] Kitchenham, B., Pfleeger, S., L., Hoaglin, D., C., El Emam, K., Rosenberg, J.,

”Preminary Guidelines for Empirical Research in Software Engineering”, IEEE Transactions on Software Engineering, Vol 28, No. 8, August 2002, pp. 721-734.

[67] Rainer, A., Hall, T., Nathan, B., “Persuading developers to ‘buy into’ software

process improvement: local opinion and empirical evidence”, the second symposium on empirical software engineering. ISESE 2003, Rome, September 30 - October 1 2003.

[68] Potts, C., “Software Engineering Research Revisited”, IEEE Software,

September 1993, pp. 19-28. [69] http://name.case.unibz.it/ last accessed 041103. [70] Saaty, T. L., The Analytic Hierarchy Process, (McGraw-Hill, New York, 1980). [71] Tessem, B., “Experiences in Learning XP Practices: A Qualitative Study”,

proceedings of the 4th international conference on extreme programming and agile processes, Genova, Italy, May 2003, pp. 131-137.

[72] Padberg, F., Muller, M.M., “Analyzing the cost and benefit of pair

programming”, proceedings of the Ninth International Software Metrics Symposium, 3-5 Sept. 2003, pp. 166 – 177.

61

[73] Williams, L., McDowell, C., Nagappan, N., Fernald, J., Werner, L., “Building pair programming knowledge through a family of experiments”, Proceedings of the 2003 International Symposium on Empirical Software Engineering, 30 Sept.-1 Oct. 2003, pp. 143 – 152.

[74] Williams, L., Kessler, R.R., Cunningham, W., Jeffries, R., “Strengthening the

case for pair programming”, Software, IEEE , Volume: 17 , Issue: 4 , July-Aug. 2000, pp. 19 – 25.

[75] http://www.agilealliance.com/home last accessed 041103. [76] Beck, K., Extreme Programming Explained: Embrace Change, 2nd edition,

Addison Wesley Professional, 2005.


62

63

Aggregating Viewpoints for Strategic Software Process Improvement – A Method and a Case Study

Karlström, D., Runeson, P., Wohlin, C., “Aggregating Viewpoints for Strategic Software Process Improvement – A Method and a Case Study”, IEE Proceedings Software, October 2002.

Abstract Decisions regarding strategic software process improvement (SPI) are generally based on the management’s viewpoint of the situation, and in some cases also the viewpoints of some kind of an SPI group. This may result in strategies, which are not accepted throughout the organisation, as the views of how the process is functioning are different throughout the company. This paper describes a method for identifying the major factors affecting a process improvement goal and how the perception of the importance of the factors varies throughout the organisation. The method lets individuals from the whole development organisation rate the expected effect of these factors from their own viewpoint. In this way the strategic SPI decision can be taken using input from the entire organisation, and any discrepancies in the ratings can also give important SPI decision information.

The method is applied in a case study performed at Fuji Xerox, Tokyo, which is reported in this paper. In the case study, significantly different profiles of the factor ratings came from management compared to the ones from the engineering staff. This result can be used to support the strategy decision as such, but also to anchor the decision in the organisation.

1


64

1 Introduction The need for software process improvement (SPI) is well known and also increasingly accepted as a means for a software company to stay competitive in a rapidly changing environment. However, when it comes to which strategy to follow in an SPI program, there are many different opinions in a software organisation. For example, quality assurance staff may tend to stress the importance of the documented processes, while engineers may tend to rely on better tools. One of the contributing factors to such differences is that the documented process is not the same as the actual work performed. Roles and workflows are documented in official descriptions, but the actual roles and work performed are a combination of past practices, each person’s interpretation of the official documents and the effects of training programmes.

Decisions made to improve the development processes as a part of a software process improvement initiative, are in most cases taken by a select group of people with their specific views of the entire process. If the different viewpoints in the organisation are not taken into account, there is a risk that the SPI initiative is not sufficiently accepted, and hence the goals will not be effectively implemented.

After the need for an improvement effort is realised and accepted by the company, an initial phase begins when sponsorship, goals and strategies are confirmed for the duration of the improvement effort. The current status of the company’s development must then be measured or assessed before any improvement attempts can be made. The steps to be taken for SPI as described by Humphrey [1] can be generalised into the steps shown in Figure 1. These are also the main steps of the more advanced SPI method called the IDEAL process [2] introduced by SEI.

Figure 1 General SPI model

Improve Analyse Verify Goal

65

This paper focuses on the goal and analysis part of the SPI model shown in Figure 1. It presents a method that supports SPI decisions to be taken with knowledge about the different interpretations that each person has made of the current status of the development process and their own role in the process. The method is divided into five main steps, for which a flow diagram is provided in Figure 2.

A: Determine goal. B: Identify major factors believed to be affecting the goal. C: Allow the organisation to rate the factors anonymously

based on their personal viewpoints. D: Analysis of data. Aggregate and analyse the rating results

and identify discrepancies. E: SPI strategy decision.

Figure 2 Overview of the method

The method provides decision support information to SPI responsible individuals by identifying the perceived most effective improvement factors and by analysing discrepancies between different viewpoints throughout the organisation. The advantage is that it provides systematically derived information, which supports decision makers in the decisions on what to improve and how to improve it. Metric analysis can be one method of identifying improvement factors. A general process framework/appraisal method such as the CMM [2] is another. However, factors outside these frameworks may be just as important since they are derived from the individual company. Hence, they should most definitely not be excluded. An example of such factors is provided by Cattaneo et al. [3] by demonstrating the need for organisational restructuring within development companies prior to, or in combination with, CMM based SPI work.

Herbsleb et al. show the importance of management involvement in successful CMM based SPI in their survey of companies using the

Goal Rate

Factors

SPI Strategy

Decision

Identify Major

Factors

Analyse

Rating data

A B DC E


66

framework [4]. By making information about the views on process improvement alternatives, the method presented in this paper is thought to be able to improve management involvement in organisations.

The paper is structured as follows: Section 2 presents the general problem of how processes are interpreted from different viewpoints of different persons in the organisation. In Section 3, the method and its constituents are presented and in Section 4, a case study of the application of the method is presented. The case study was performed at Fuji Xerox in Tokyo, Japan.

2 Processes Distortion and Viewpoints In a process improvement initiative, there may be different opinions on what should be improved in order to reach the improvement goal. To some extent, this depends on the different viewpoints on what the process actually is.

Development processes are documented in official documents, but as they are interpreted, different versions appear depending on the perception of the process. Bandinelli et al. describe these different perceptions of the process and names these as the official, perceived and desired process [5]. To this list we can add the process that is actually performed, the actual process and the observations made by SPI responsible people, the observed process. The relations between the processes are illustrated in Figure 3. Note that of all the processes shown in the figure, only the actual process produces the product and the only absolutely accessible process is the official process.

From To Mechanism Description Reference Desired Comprehension The reference process is interpreted by the person(s) creating

the official process. Desired Official Formulating The official process is formulated from the desired process

and written down as the official process. Official Perceived Comprehension The user based on what he is taught, what he reads and how

his co-workers influence him interprets the official process. Perceived Actual Performing What is actually performed is a result of the perceived new

process and the old process and is also dependant on individual opinions, mistakes and external stimuli.

Actual Observed Comprehension The observed process is a result of an interpretation of the actual process.

Table 1 Mechanisms of transfer between types of processes

67

Figure 3 The Process feedback loop

The mechanisms between the different processes are summarised in

table 1. The most complicated process transition is between the official and the perceived process. The individual person’s perceived process is affected by an individual interpretation of the official process, both as a result of reading it directly and as a result of some form of training. The perceived process is also affected by how Co-workers communicate their perceptions of the process and by observation of the co-workers’ actual execution of the process. Finally the processes that have been in effect in the past also affect the perceived process.

Of the mentioned processes, the observed, desired and perceived processes are subject to viewpoints [6] by each person that is involved with this process. This means that each individual person’s perception of this aspect of the process is different. The only externally observable processes are the official process, documented by official process documents, and the actual process, which consists of the actual actions that agents perform to produce the product. As the product is the result of the actual process, the SPI strategy must be to improve this process and

Observed Process

Desired Process

Perceived Process

Official Process

Actual Process

Reference Processes

Tradition, co-workers and personal opinions

Product


68

make inconsistencies between the actual and the official processes as few and small as possible.

Sommerville et al. introduces the concept of viewpoints to software processes [6]. Several of the processes described in this paper thus far are subject to different persons in the company having different viewpoints. The observed process is subject to each SPI responsible persons interpretation, or viewpoint. The desired process is subject to the interpretation and application of reference processes and personal opinions of the SPI people. The perceived process is, as previously described, affected by many factors and each person has their own perceived process.

Herbsleb et al. [4] analyse responses from respondents with differing roles within the companies using CMM based SPI in their survey. No statistical differences are shown between different roles, but only one person in each role is used from each company.

This paper focuses on using people’s perception, or viewpoint, of the official and actual processes to help decision-making in process improvements.

3 Method In order to support the strategy choice for an SPI initiative, a method is developed which involves the viewpoints of different roles in the software development organisation. In the description of the method, the general model for SPI presented in Figure 1 has been followed, with the additional steps shown in Figure 2, regarding the selection of which factors to implement.

3.1 Determine Goal The first stage of an SPI initiative is to determine the goal for the process improvement. The goal has to be set in line with the business goals of the organisation. Any method that works in the organisation and produces a list of goals can be used. For instance, the goals of the CMM [2] can be used according to the current maturity level of the company.

69

3.2 Identify Major Factors Once a goal has been determined, factors thought to be affecting this goal need to be identified, see Figure 4. The main focus of this paper is not, however, this identification of factors, but the following rating and use of the results of these ratings.

Each goal must be possible to realize by factors affecting the development process. If it is not, it is an unrealisable goal, and therefore uninteresting to the SPI work. We define a factor as a possible change in the process, organisation, or environment that is believed to cause a change in the development process towards the goals associated with the factor.

Figure 4 Relationship between goals and factors

Each factor has an actual effect and an actual cost that cannot be estimated accurately beforehand, if at all. However, the perception of the effect and cost of the introduction of the factors may provide enough information to identify the best factors for implementation. As these factors are to be rated for one goal at a time, the exact correlation between the factors and the goal is not as important as covering all factors that are possible to implement for the goal. This is because factors that are thought not to affect the goal during the rating should be rated low.

The number of factors used in the formal evaluation exercise is constrained by the time available for each person to complete the rating procedure. This must be decided taking into account the expected benefit of the whole procedure. In the next method step, pair-wise comparisons between the factors are performed, and an estimation of the time taken for the persons to complete the rating procedure can be performed by estimating a maximum and minimum time for reading an introductory text for the rating procedure and a maximum and minimum time for

FactorGoal

Goal

Goal

Factor

Factor

Factor

Factor

Factor

Affected by


70

each comparison. From this a low and high estimation can be calculated for the total completion time and a cost/benefit decision can be taken. If the method has proven successful previously, a larger number of factors may be used the next time. With the rating scheme used in this paper, n*(n-1)/2 comparisons are performed to compare n factors. If the number of people in the organisation is large, some form of sampling should be undertaken. The subjects were randomly assigned one of four randomly ordered rating pages to ensure that the order of the comparisons did not affect the results.

The goal/factor structure presented here is similar to the structure of goals, questions and metrics in the GQM method [7]. The GQM method, however, uses direct measures available in the actual process to measure aspects thereof in order to achieve the set goal by answering the questions associated with the goal. The method presented in this paper uses the perceptions of the individual persons to indicate the major factors contributing to the improvement goal. The method is, of course, affected by the factors selected for the rating. If a major factor is not identified, or omitted, this will mean that the finally selected factor might not be the ultimate one. The method of selecting factors should therefore be carefully selected and tailored to the situation.

Other methods for identifying factors can be studying past SPI strategies and their effect. Studying the effect of SPI in other companies is a further possibility.

3.3 Rate the Factors The rating of the factors is to be performed by a suitable sample of all people involved in the development organisation that are active in the process. In a small organisation, a complete sample can be used, but in a large organisation this is not possible. The sample should be designed to ensure that all vertical levels of the organisation (i.e. management levels) and all horizontal groups (i.e. development teams, departments) are represented. The purpose of this is to ensure that all sources of different viewpoints should be represented in the sample.

The method used for rating the factors is called the Analytic Hierarchy Process (AHP). The AHP is a method for evaluating alternatives in a choice situation [8, 9]. It uses pair-wise comparisons and measures each alternative’s relative contribution to the rating. Each subjects’ individual set of comparisons is put into a judgement matrix. From this matrix the relative weights for the compared items can be calculated for each

71

individual. These relative weights are known as priorities. The process also gives the possibility of calculating how consistent the performed ratings are for each individual as the pair-wise comparisons imply that the alternatives are indirectly compared several times.

The AHP process has been implemented for applications outside the process improvement domain. Karlsson et al. [8], for example, uses the process to select between customer requirements. This method of comparing requirements has been further developed into a commercial tool developed by Focalpoint AB [10]. This tool was not used for the experiment as it does not give full control over the algorithms used for calculating the results.

3.4 Analysis of Data This step consists of two parts. First the whole data set is analysed and then comparisons are made between groups in the data in order to identify discrepancies within the organisation. To compare the relative importance of each factor, the rating results are aggregated first overall and then in each group. When aggregating results from an AHP comparison process we must decide whether we wish to aggregate the judgements of each individual or the priorities calculated for each individual. This is referred to as aggregation of individual judgements (AIJ) and aggregation of individual priorities (AIP) respectively. The decision is determined by whether the group is assumed to act as a unit or as separate individuals [11]. Secondly it must be decided whether to use the arithmetic or geometric mean for the aggregation. For AIJ the geometric mean must be used. For AIP any of the two may be used, but the arithmetic mean has the advantage of being comparable to the original values. We have chosen to use AIP aggregation with arithmetical means.

The statistical methods described in this section are examples of appropriate methods for analysis of the data. There are several other methods that are appropriate and could be used.

3.4.1 Overall Analysis

An overall view of the data can be given by using box plots and descriptive statistics. Further analysis of the ratings can be performed using an ANOVA test [12], if the data is normally distributed and the variances are equal. Significant rating differences can then be established within each group using Fisher’s PLSD test [12]. The normal properties


72

of the data is checked using a Kolmogorov-Smirnov test for normal distribution. If the assumptions for ANOVA are not met by the data, a Kruskal-Wallis non-parametric test can be performed instead [13].

3.4.2 Identify Discrepancies

Discrepancies between groups of persons can give vital SPI strategy information. The ratings of one factor by two groups are compared using an unpaired t-test if the ratings are normally distributed. If the ratings are not normally distributed a Mann-Whitney test is performed [13]. A comparison of ratings in each group can be performed using the methods described for the overall results in section 3.4.

3.5 SPI Strategy Decision The final outcome of the method is an overview of the expected impact of factors to improve in a development organisation and an idea of the difference between the groups in the organisation. The systematic nature of the method implies that we can be fairly sure that the results are valid as decision support material for the SPI strategy. The final decision will of course be taken taking other information into account as well. This is not an automatic decision taking method.

4 Case Study at Fuji Xerox

4.1 The Company The method presented in Section 3 is applied in a case study at Fuji Xerox in Tokyo, Japan. Fuji Xerox Group is a joint venture owned by Fuji Photo Film Co., Ltd. and Xerox Ltd. of the United Kingdom. Fuji Xerox operates in the Asia-Pacific and Oceania regions as a member of the world wide Xerox Group. The company has approximately 15000 employees in its operating area.

The company’s principal business is the manufacturing and selling of office automation equipment such as copiers and low-end laser printers, collectively referred to as the document services business. Other businesses include logistics and educational services. Fuji Xerox also performs research and development, marketing and service activities on behalf of the global Xerox Group.

73

4.2 Fuji Xerox SPI Background The SPI process at Fuji Xerox (FX) began in 1995 when a proposition to introduce the SW-CMM won management approval. The company had an initial CMM assessment in April 96 and was found to be operating at level 1.

Improvement Action Teams (IAT) were started in July 96 to prepare for pilot projects operating at level 2. The pilot projects were started in April 97 and were reviewed in June of the same year. FX was assessed to be operating at level 2 in April 98. The SPI work has continued and the organisation was due to start pilot projects for level 3 operations in April 2000.

Occurrences of major events during the process initiative are illustrated in Figure 5. The goal of the program was to increase productivity and quality in the development process by working in the following three areas:

• Increase management visibility into ongoing projects. • Activate more capability from the engineers using CMM. • Achieve dynamic resource rotation.

Figure 5 Major events in the Fuji Xerox SPI initiative

Fuji Xerox was at the time of starting the program experiencing problems in the development of software. Two major factors were identified as causes:

• The size of the code in many projects has increased a factor 10 from 1990 to 1995 (approx. 100 KLOC to approx. 1 MLOC).

• The invisible nature of software.

The size of the code meant that a team consisting of one team leader and a group of developers was not sufficient to develop the software product in reasonable time. Instead the projects needed to be split up into several groups of developers and group leaders. This introduced new

95 97 98 99 00 0196

IAT created

L2 Pilot project

Management approval

2nd assessment L3 Pilot project1 st assessment SEPG

Created


74

management and communication aspects that were hard to solve. During the spring of 2001 Fuji Xerox has been successfully assessed at

CMM level 3 and is operating several projects at level 3. The degree of deployment of level 2 was unknown in the spring of 2000 but it was estimated by members of the process improvement group that among the projects that have started the initiative approximately 80% of the key process areas for level 2 are satisfied.

The Copier Platform and Systems (CP&S) unit, where the case study was performed, has approximately 800 engineers working with software development for the Fuji Xerox document systems. The software is both embedded software for integration in copiers and printers and also independent software products for systems concerned with document handling.

Fuji Xerox employs a matrix organisational structure [14] for development projects. Project managers are responsible for coordinating and completing projects while the group leaders are responsible for providing the functional resources to the projects.

4.3 Determine the Goal The goal determined at Fuji Xerox was identified by looking at the original intent of the SPI initiative and discussing this with concerned management. One of the important goals of the SPI initiative was to increase management visibility into ongoing projects. It was hence decided that this was one of the main goals to analyse.

Visibility concerns the information flow between the ongoing development projects and the responsible management and can be defined as: “The ease and accuracy with which it is possible to assess the status of a project's cost, schedule, functionality, or other characteristics.” [15]

Information reaches management in three ways:

1. Periodical reporting Routine procedures that require reports or other forms of information to be produced and sent to management. 2. On demand information

Management asks for specific information from the project. 3. General impressions

Impressions obtained during day-to-day activities.

75

In an organisation with low visibility, information is only available and

accurate in a close vicinity of the source. Hence if someone outside this area requires information the information has to be collected. This is usually performed in a non-standard fashion in a low visibility organisation. The information is therefore not comparable to information from other parts of the organisation and when it has been collected it may well already be out of date. In an organisation with high visibility, relevant information is collected in a standardised fashion and made available to authorised people, as they need it.

4.4 Identify Major Factors The next step is to identify the factors, which affect the goal. The factors affecting managements’ visibility into development projects were identified by a qualitative study, observing the SPI work performed at Fuji Xerox. The study was limited to SPI work within the Controller Platform and Service development (CP&S) unit as this unit has been making a software process improvement effort since 1995. Many of the observations were made in conjunction with the improvement action team (IAT) meetings held under two weeklong periods during the spring of 2000.

The study performed is of a single execution nature and it is of the holistic type as the study concerns the SPI work within one division of Fuji Xerox during one single period of time [16]. During the case study no consideration was given to the situation within specific departments.

The observations made can be divided into 3 different categories depending on the source of the information: 1. General information provided as a background to Fuji Xerox

development. 2. Impressions obtained during regular meetings with representatives

from Fuji Xerox process improvement group. 3. Impressions obtained during IAT workshop weeks.

General information was collected through impressions during the stay at Fuji Xerox, brochures and internal documents provided by Fuji Xerox. The meetings held every two weeks with Fuji Xerox process experts were intended to keep track of the project, but also provided information about the actual development performed at Fuji Xerox and an opportunity for questions to be answered.


76

The IAT workshops were the most productive sources of information for process improvement information. This was because of the following reasons:

1. Process improvement is the main focus of these meetings. 2. A CMM expert consultant was present and willing to discuss the

Fuji Xerox situation. 3. An interpreter was present and thereby solved any direct

communication difficulties.

The IAT workshops were held on two occasions during the course of the case study. Each workshop was a weeklong session dealing with current issues of the SPI work. The main goal of the SPI work was intended to progress the development process on the CMM scale. The following people were present at the workshops:

• IAT team members. • The CMM consultant and certified lead assessor. • A Japanese-English interpreter. • People relevant to the each current topic.

During each IAT workshop a special session was allocated to

discussions regarding the SPI work described in this paper. A summary of the seven factors finally used in the AHP rating is presented in table 2. These factors affect the software process throughout the whole company.

4.5 Rate the Factors Senior Fuji Xerox management authorised a survey scope of approximately 160 subjects based on the estimated completion range of 24 to 52 minutes per subject. The subjects were to be chosen from within the CP&S unit consisting of 800 people in total. After consulting with Fuji Xerox representatives, the strategy for choosing subjects was determined as follows: 1. Subjects should be chosen from projects that are a part of the

process improvement initiative. 2. Project traits should be comparable using the survey (i.e. a

sufficient number of people from each project should be sampled to establish an impression of differences between projects).

77

3. Organisation level traits should be comparable (i.e. differences between management and engineers should be apparent if present).

The AHP rating was performed using a web based form. The web pages were created in English and then translated to Japanese by a translator. The Japanese content of the web pages was reviewed by Fuji Xerox process experts together with the author of the form so that the content was as correct as possible in the following aspects:

• Content is in correct Japanese. • Content is correct in Software Engineering terminology. • Content is presented in a Japanese style.

This translation strategy implies that the content of the Japanese version is not a word for word exact translation of the English original. This was deliberately avoided in order to make the survey as effective as possible as the terminology involved is very different in the two languages.

The 7 major factors affecting the SPI goal identified at Fuji Xerox: 1. Deployment: Increase the number of projects using new

processes in the organisation and optimise for those already using the standard process.

2. Maturity: Increase process maturity level as rapidly as possible in projects that are already in the improvement program.

3. Tools: Introduce new more powerful tools to aid the process, for instance automatic reporting tools and development tools.

4. Training: Ensure quality and acceptance of CMM training and integrate into all levels of the FX training program.

5. Support-group process introduction: Introduce process oriented thinking to operations outside the engineering function such as management, marketing, planning and production.

6. Culture change: Adapting the current CMM level 2 process to be more company specific.

7. Standardising data: Introducing standards to increase homogeneity of information and numerical data used throughout the organisation.

Table 2 The SPI Factors Identified in the Case Study


78

The web pages first present the purpose of the rating and instructions for the rating system. Then the concept of visibility used in this study is explained. This is followed by a description of all the factors to be compared. Finally the actual rating form is presented. The rating form itself is divided into two parts. The first part is intended to gather information from the subject in order to classify the subject. Classification is performed using the subject’s role in the functional organisation, either management or engineer. Management is defined as the three levels called division manager, group leader and function manager in the Fuji Xerox functional organisation. Engineer is defined as the two levels called sub leader and engineer.

The second part of the survey is the rating part. Each rating is presented with the factors to be rated on each side of a series of buttons that each represents a degree of choice in the AHP scale. All keywords are hyper-linked to short descriptions. The rating sheet is randomly chosen as one of four different randomly ordered rating sheets to address validity issues. An example of a rating is shown in Figure 6. When all the ratings have been completed the form is submitted and a thank you page is shown. The results are written to a database on the web server.

Test runs were performed on two occasions with two subjects on each occasion. The subjects’ impressions from these test runs were used to improve the contents of the survey. The test runs also verified that the server was reachable from all relevant development facilities around Tokyo, many at significant distances from the physical location of the server.

The actual rating started with an email sent to all 148 selected subjects explaining the purpose and reason for the rating and how to access the server. The server was available for a period of 11 days. During this time 75 subjects completed the rating procedure. Of these, 8 subjects were later removed due to lack of consistency in their ratings. The consistency of each individual’s ratings was appraised using the consistency ratio calculated according to the AHP method. The consistency ratio limit was set to 0.15, well below the limit of 0.2 that has been deemed acceptable in practice [8], which is good.

79

Figure 6 A rating from the web rating system

4.6 Analysis The analysis of the rating data is divided into two steps as described in the method section. First we analyse all the data to get an overall view of the trends in the data. Then we check for significant differences between the groups.

4.6.1 Overall Analysis

All the ratings were plotted in a box plot in order to get an overview of the whole data set. The box plot is shown in Figure 7.

The results were first aggregated using the arithmetic mean of the ratings for each factor, after outliers had been removed based on their consistency ratio, as described in section 3.4. The means are presented in the ‘mean’ column in Table 3. A Kolmogorov-Smirnov test was then performed on the data to check if it was normally distributed. It was found that the data for four out of the seven factors was normally distributed. In the case of the other three, the test could not reject that the data was normally distributed at a 0.05 significance level.


80

Figure 7 Box plot of all ratings

The next stage of analysis was to perform an ANOVA test. This test is normally used on a ratio scale and as the results from the AHP are of a ratio nature compared to the ordinal nature of the ratings we can apply the test. In order to investigate the relationships between the rated factors, we formulate our null-hypothesis such that there are no significant differences between the 7 factors:

H0: R1=R2=...=R7

We formulate our alternative hypothesis such that there is a significant difference between at least one pair of the 7 factors:

H1: Rm<>Rn, for at least one pair of factors m,n

The test showed that there is a significant difference in the data set at the 0.05 significance level. As we could not prove the normal distribution of all the factors we also performed a Kruskal-Wallis test. This test gave the same result as the ANOVA test. The final test that we would like to perform is the Fisher PLSD test to investigate the individual relationships in the data, but this requires that the preconditions for the ANOVA are satisfied. As the ANOVA provided the same result at 0.05 significance level as the Kruskal-Wallis test and taking into account the robustness of the ANOVA test towards fit of normality, the Fisher PLSD test was used nonetheless [17]. The whole analysis procedure is illustrated in Figure 8.

81

Overall Rating Mean Std Dev Std Err 1 Standardisation 0.1733 0.0668 0.007 2 Tools 0.1614 0.0801 0.009 3 Deployment 0.1429 0.0556 0.010 4 Maturity 0.1422 0.0700 0.006 5 Training 0.1408 0.0513 0.008 6 Culture change 0.1238 0.0776 0.010 7 Support group 0.1152 0.0612 0.008

Table 3 Aggregated Results from the AHP calculations

Figure 8 Data analysis procedure for all subjects

The results of the PLSD show that the Standardisation factor is rated significantly higher than Training, Deployment, Maturity, Support group and Culture change. Tools is rated significantly higher than Support group and Culture change. Finally Training, Deployment and Maturity are rated significantly higher than Support group. These relationships are illustrated in Figure 9. In the figure, if hypothesis H0 is rejected for a pair [m,n] at the 0.05 level, this is denoted by a ‘S’ at the point of intersection between two factors. The factors are ordered according to their arithmetic mean rating.

The results indicate that the factors Standardisation and Tools seem to be the most firmly anchored in the organisation. To investigate if this finding is tied to a specific group, the analysis is continued by dividing the data into the two different groups.

AHP Data

Kolmogorov-Smirnov

Anova

Kruskal-Wallis

Fisher PLSD


82

Figure 9 Significant relationships between factors according to Fisher’s PLSD

4.6.2 Identify Discrepancies

For the purpose of identifying discrepancies, the data was divided into two groups according to the role of the subjects, the management group and the engineer group. The hierarchical nature of the organisation implies that there are more engineers available to perform the comparisons. In total 10 managers performed the comparisons, versus 56 engineers. A Kolmogorov-Smirnov test for normal distribution was performed on the data in each group as in the case with the whole data set.

Next we formulate the hypotheses as follows:

H0: R(manager)n=R(engineer)n, n=1..7 H1: R(manager)n<>R(engineer)n, n=1..7

Our null hypothesis states that there is no statistical difference between

each pair of factors from respective group and our alternative hypothesis states that there is a statistical difference.

Each factor was then compared between the groups using a Mann-Whitney test. If no significant relationship (0.05 level) was discovered by this and both groups were normally distributed for the factor a t-test was performed instead. This analysis procedure is illustrated in Figure 10.

Standardisation

Training

Tools

Culture change

Deployment

Support group

Maturity

S

S

S

S

S S

S

S S S

83

Figure 10 Data analysis procedure for comparison between groups

Significant differences were found between the two groups for factors Maturity (using the Mann-Whitney test) and Standardisation (using the t-test). Next, the analysis procedure performed for the entire data set was repeated to investigate the rated relationships between the factors within each group. The results for this analysis are summarised in Figures 11 and 12 for engineers and management respectively. It can be seen that the order of factors is different between the two groups.

Figure 11 Significant relationships between factors in the ‘Engineering’ group according to Fisher’s PLSD

AHP Data

Kolmogorov-Smirnov

t-test

Mann-Whitney

Standardisation

Maturity

Tools

Culture change

Deployment

Support group

Training

S

S

S

S

S S

S

S S

S

S S


84

Figure 12 Significant relationships between factors in the ‘Management’ group according to Fisher’s PLSD

The results indicate that there is a difference between the groups regarding Maturity and Standardisation. It is interesting to note though that the groups agree on the importance of Tools.

4.7 Validity The threats to the validity of the survey have been evaluated according to the lists presented by Wohlin et al. [18]. Only significant threats to validity are discussed.

4.7.1 Conclusion Validity

Reliability of measures is a validity concern during the translation of the survey form from English to Japanese. It is not certain that the translator understands the context of the English version and therefore may provide a translation that misleads the subjects. This validity concern was reduced by arranging review sessions with the author of the English text and English-speaking process experts from Fuji Xerox.

Cultural differences were addressed by performing the case study over the relatively long period of six months and by reviewing observations with English-speaking members of Fuji Xerox staff.

The formulation of the factor descriptions and the formulation of the instructions will affect the ways in which the subjects complete the survey.

Maturity

Deployment

Tools

Training

Culture change

Support group

Standardisation S

S

S

S S

85

The effects of these validity concerns where reduced by putting a lot of time into the construction of the survey and by test running the form twice. Good consistency values during the test runs and during the actual runs suggest that the subjects have understood both the instructions for performing the survey and the meaning of each factor.

The order of the ratings may affect the results from the rating procedure. The randomisation of the rating order is an attempt to reduce this factor.

Reliability of treatment implementation is a validity concern for the survey as the form was applied over a web interface there was no control of the environment during the duration of the study. The subjects were free to fill out the form in any situation they chose and it is assumed that most of the subjects filled out the form in his/her normal working place.

Random irrelevancies in experimental setting: The effects of the subjects’ normal working environment cannot be eliminated as a validity threat as there was no control over this.

Threats to the conclusion validity concerning statistical tests are under control. The data is tested for normal properties, and the tests applied (ANOVA and t-test) are standard test for the analysis. In the cases where normality cannot be established, non-parametric tests are used instead (Kruskal-Wallis and Mann-Whitney tests).

4.7.2 Internal Validity

Maturation: The subjects could become more acquainted with the different rating factors as the survey progresses. The randomisation of the order of the ratings should reduce this threat.

Instrumentation: Wordings in the web form affect the grounds for the rating choices made and will therefore be a validity threat. Wordings in the instructions to the rating will affect the way the subjects perform the study and may therefore threaten the validity. The relatively high level of consistency and the general homogeneity of the results suggests that the effects of these two threats are low, unless the descriptions explicitly make a certain rating order favourable.

Selection: The basis on which the subjects were selected does not provide a correct sample from the group as Fuji Xerox representatives chose projects that they were interested in. The low response rate caused by the fact that there was no method of forcing form completion means that only subjects willing to fill out the form did so.


86

4.7.3 Construct Validity

Inadequate pre-operational explication of constructs: Although great care was taken in defining the constructs the limited total time of the study implied that there came a time when there simply was no more time to work on these and the study had to be performed.

Evaluation apprehension: Subjects can be purposely not answering the survey in order to demonstrate how busy they are and how devoted they are to working with their primary tasks.

4.7.4 External Validity

The nature of the case study method usually implies low external reliability [16]. As the factors were identified using observations made at Fuji Xerox the study has external limitations. Fuji Xerox however faces the same challenges as most other large corporation developing software. Making changes in the way the company functions takes an extremely long time and standardising is difficult due to the large number of people and opinions involved. The case study does, however, show that the method presented in the paper is useful in one software engineering organisation and it provides information useful to other companies interested in trying the method.

4.7.5 Validity Summary

The lack of control of the environment where the subjects performed the tests because of the web interface is a major validity threat together with translation and formulation effects in the form itself. The main threat to internal validity in this study is the selection of the subjects. Even though the results are specific to Fuji Xerox they can provide useful information to other organisations in similar situations.

4.8 Case Study Summary The case study shows that the factors can be prioritised and that differences exist between different groups in the company. Especially the study shows that there are several differences between subjects at the management level and at the engineering level in the organisation.

The methods used in the case study prove adequate for identifying the factors and performing a prioritisation between the factors. The analytical hierarchy process worked extremely well for rating the factors provided from the case study. Especially, the ability to calculate the consistency of

87

the subjects gave a good idea of the quality of the results obtained and also provided a means to remove inconsistent subjects.

It is interesting to note that the comparison between the groups showed a significant difference in one of the most highly rated factors overall, Standardisation. This should have a major impact on future SPI work. The Fuji Xerox management group decided to investigate introducing standardisation of low-level work processes through the use of some kind of reference model such as the Personal Software Process [19] after the results of the investigation were presented.

5 Summary and Conclusions In a software improvement program, there is a risk that the improvement strategy is not well anchored in the organisation. Different persons in the organisation have different viewpoints on what the process is, and how it should be improved.

To support the establishment of an improvement program strategy, which takes these different viewpoints into account, a method is developed and evaluated in a case study at Fuji Xerox, in Tokyo, Japan. The method supports the improvement program by providing decision support information for SPI work.

Based on a determined goal and a set of factors that affect the goal, people from the organisation rate the factors, using the analytic hierarchy process (AHP), which basically means pair-wise comparisons of the factors. The AHP provides ranking of the factors as well as a measure of how consistent the rankings are. There is also an opportunity to compare the ranking throughout the organisation and thereby identify any discrepancies. With the information collected in this way, management is expected to be able to make better decisions concerning SPI strategy.

With the information at hand, which was derived by using the method, it was shown in the case study that management was able to make better decisions on which strategy to choose, and how to create a more homogenous process perception throughout the organisation. The method provided information on the viewpoints of different stakeholders in the organisation, which was a support in the selection of improvement factors as well as in the identification of change support needed. Further, the involvement of people at different levels in the organisation provides in itself a more firmly anchored improvement program.


88

6 Acknowledgements The authors would like to thank Fuji Xerox for their extensive cooperation and support for the project. Especially Dr. Jun Miyazaki and Mr. Atsushi Nakamura. Mr. Mark Amaya of Software Process People and Technology provided many illuminating insights into the process improvement work at Fuji Xerox. Finally we would like to thank the Sweden-Japan Foundation for the funding of the research performed in Japan associated with the project.

7 References [1] Humphrey, W. S., Managing the Software Process, Addison-Wesley, 1989.

[2] Paulk, M.,C., Curtis, B., Chrissis, M., B., and Weber, C., V., “Capability Maturity Model for Software, Version 1.1”, Software Engineering Institute, CMU/SEI-93-TR-24, February 1993.

[3] Cattaneo, F.,Fuggetta, A., Sciuto, D., ”Pursuing Coherence in Software Process Assessment and Improvement”, Software Process Improvement and Practice 2001, pp. 3-22.

[4] Herbsleb, J., D., Goldeson, D., R., “A Systematic Survey of CMM Experience and Results”, 18th International Conference on Software Engineering (ICSE’96). IEEE Computer Society Press.

[5] Bandinelli, S., Fuggetta, A., Lavazza, L., Loi, M., Picco, G. P. ‘Modelling and Improving an Industrial Software Process’, IEEE Transactions on software Engineering, Vol. 21, No. 5, 1995, pp. 440-454.

[6] Sommerville, I., Sawyer, P., Viller, S., “Managing Process Inconsistencies Using Viewpoints”, IEEE Transactions on Software Engineering vol. 25, No. 6, 1999, pp. 784-799.

[7] van Solingen, R., Berghout, E., The Goal/Question/Metric Method, McGraw-Hill International (UK) Limited, 1999.

[8] Karlsson, J., Ryan, K., “A Cost-Value Approach for Prioritising Requirements”, IEEE Software September/October, 1997, pp. 67-74.

[9] Saaty, T. L., The Analytic Hierarchy Process, McGraw-Hill, New York, 1980.

89

[10] Home page of Focal Point AB, accessed 041103, http://www.focalpoint.se/index_eng.html.

[11] Forman, E., Peniwati, K., “Aggregating Individual Judgements and Priorities with the Analytic Hierarchy Process”, European Journal of Operational Research, 1998, pp. 165-169.

[12] Montgomery, D. C., Design and Analysis of Experiments, John Wiley and sons, 1997.

[13] Siegel, S., Castellan Jr., N. J., Nonparametric Statistics, McGraw-Hill International, 1998.

[14] Thayer, R., H., “Software Engineering Project Management”, from Software Engineering Project Management, edited by Thayer, R., H., pp. 72-104, IEEE Computer Society, 2000.

[15] McConnell, S., Software Project Survival Guide, Microsoft Press, 1998.

[16] Yin, R. K., Case Study Research Design and Methods, Sage Publications, Beverly Hills, California, 1994.

[17] Bratthall, L., Wohlin, C., “Understanding Some Software Quality Aspects from Architecture and Design Descriptions”, International Workshop on Program Comprehension, 2000, pp.27-36.

[18] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., Wesslén, A., Introduction to Experimentation in Software Engineering, Kluwer Academic Publishers, 2000.

[19] Humphrey, W.S., A Discipline for Software Engineering, Addison-Wesley, 1995.


90

91

Introducing Extreme Programming – A Case Study

Karlström, D., “Introducing Extreme Programming - An Experience Report”, Proceedings Third International Conference on Extreme Programming and Agile Processes in Software Engineering (XP'2002), Sardinia, Italy, 2002.

Abstract This paper presents a single case study reporting the experiences of introducing extreme programming (XP) in a small development project at Online Telemarketing in Lund, Sweden. The project was a success despite the fact that the customer had a poor idea of the system required at the start of the development. This success is partly due to the introduction of practically all of the XP practices. The practices that worked best were the planning game, collective ownership and customer on site. The practices that were found hard to introduce, and not so successful, were small releases and testing.

2


92

1 Introduction Extreme programming (XP) [1] is a methodology that has received much attention during 2000 and 2001. XP is a package of several practices and ideas, most of which are not new. The combination and packaging of all of these is, however new. One of the features that makes XP different to most other methodologies is that it is centred on the developer and gives him or her more responsibility in the creation of the product. This paper provides an experience report from the introduction of XP at Online Telemarketing in Lund, Sweden. The company decided to use XP to develop a sales support system for use in their principal line of business, telemarketing. The paper presents a brief introduction to qualitative research methodology, which can be said to be the research methodology used for this experience report in Section 2. Section 3 contains a brief introduction to the company, followed by an introduction to the development project in section 4. The experiences of the XP practices are accounted for in section 5 and a discussion of the quality of the conclusions is accounted for in section 6. Finally the conclusions and a summary are presented in section 7.

So far, relatively few experience reports have been made available with regards to XP. Especially, well-structured reports of attempts to fully introduce XP are rare. Experience reports not only provide insight into specific situations in which the method may work and not work, but also provide practical examples to illustrate the method. Organisations considering XP can gain much needed prior experience of what to expect when introducing practices, irrespective if they are implementing one practice or implementing XP fully.

2 Methodology The majority of the information presented in this experience report was gathered in two different ways. The first way was by direct observation of the developers during the course of the project, and the second was by interviews with both the developers and the development management. The interviews with the developers gave a lot of information about attitudes towards different XP practices, while the interviews with the management gave information mostly about how the practices were being followed.

As the information presented is of a qualitative nature, a brief

93

discussion of qualitative methodology and threats is in order. It should be mentioned that the study performed is not intended to be a complete formal qualitative investigation and that auditing is not used to validate the results [7]. This kind of validation is only applicable and practical in much larger studies. By addressing the methodology behind the research techniques we can at least make an informed attempt at improving the quality of the information obtained.

In qualitative research the trustworthiness of the investigation, which is usually called validity in quantitative research, can be addressed using four criteria: credibility, transferability, dependability and confirmability [6].

These criteria are briefly summarised below. Further information can be found in the works of Lincoln [6], Robson [7], and Miles and Huberman [8]. References are made in the summaries below to corresponding criteria in quantitative validity theory. Wohlin et al. [9] contains a comprehensive quantitative validity section.

• Credibility corresponds to internal validity in quantitative

research. The aim of this criterion is to ensure that the subject of the enquiry has been accurately identified and described. This can be achieved by, for example, triangulation of sources or methods.

• Transferability corresponds to external validity in quantitative research. This criterion addresses how far outside the observed domain the results are applicable.

• Dependability addresses whether the process of the study produces the same results, independent of time, researcher and method.

• Confirmability addresses the issue of researcher biases and ensures that the researcher affects the results as little as possible.

An attempt to evaluate the study according to these four criteria is

performed in the quality of conclusions section, section 6.

3 Online Telemarketing Online Telemarketing is a small company specialising in telephone-based sales of third party goods. The company has its head office in Lund, Sweden, and regional branches in Uppsala, Visby and Umeå. Recently the company has expanded internationally with operations in Denmark, Norway and Finland. The company consists of a small core of fulltime staff that manages and supports a large number of temporarily employed


94

staff. This implies that the company has a very flat organisation. The primary task of the temporary staff is performing the actual sales calls.

Management realised in the autumn of 2000 that a new sales support system would be required and started planning for a system for use within the company. ‘Commercial off the shelf ’ (COTS) alternatives were evaluated but discarded due to being too expensive and due to the fact that it would be both difficult and expensive to incorporate specialised functionality. The management at Online Telemarketing had several novel ideas for features not present in the systems available on the market that they considered crucial for the future expansion and business success of the company.

The person responsible for systems development at Online Telemarketing realised that the lack of detailed requirements from management and the fact that no similar systems had been created before meant that traditional development with a big up-front design and detailed requirements documents would prove expensive and not very efficient. An alternative was found in XP [1].

4 The development project

4.1 Project Overview Online Telemarketing decided on a strategy for developing the product that involved using their own system responsible person and employing part time developers to perform the coding work. To start with, four systems-engineering students were employed part time in parallel with their coursework at the university. After three months a further four people were employed and integrated into the development team in order to increase the absolute velocity of the project. The developers were employed as regular employees and there was no connection whatsoever between their position at Online and their university course-work. The employees were selected by interviewing applicants answering adverts placed throughout the student community both virtual and real.

The product was coded using Microsoft Visual Basic and SQL in a Microsoft development environment. The customer for the project was internal at Online Telemarketing and no considerations were made for eventually selling the product outside the company.

The size of the product is estimated to approximately 10 000 lines of code after all the initial functionality has been developed. The development was started in December 2000 and the first functional

95

system was launched in mid April 2001. The system has been in full commercial operation since the end of August 2001.

4.2 Roles The traditional XP roles described by Beck in [1] were assigned to the various members of the team at the start of the project. The employed developers assumed the roles of programmer. They also assumed the roles of testers, working together with the customers to create and run functional tests. The senior management at Online assumed the role of customer, as they were the people who had the original idea of the system. The tracker’s responsibilities were assumed by the IT executive at Online as he had a good overview of the work performed by the group and was in direct contact with the developers daily. The coach role was assumed mainly by the IT executive, but at the beginning of the project, when XP was new to the team, the author shared some of the coach’s responsibilities. Finally the Online senior management also assumed the roles of bosses for the project as they were providing all means for the development, such as computers, location and funding.

4.3 Configuration Management The configuration management was solved by a simple solution. As there were no branches in the configuration management and the system was relatively small, the team used a checkout directory to copy source code manually instead of using a tool for this purpose. This solution proved effective during the first part of the project when only two pairs of programmers were working. Common sense, combined with the fact that all the developers were in the same room, made sure that the configuration management worked well. As the product grew, and the number of developers doubled, problems did arise on occasion. One of the effects of the problems was work being deleted on a few occasions due to versions overwriting each other because of misunderstandings. When this showed to be causing problems for the developers, a quick and dirty solution was introduced. Using simple text files to administrate copies to checkout directories, the problem was solved.

Awareness of what was happening in the product was intended to be handled by the developers sitting in the same room and communicating all the time. The problem that became apparent with this strategy was when people were absent or working different schedules.


96

Code is integrated continuously several times a day and several times for each task. The alternative of employing a tool for the configuration management might, in retrospect, have been a more effective solution. The basic system of copying files to checkout folders solves the basic issues addressed by these tools and, as no configuration branches were to be used at all, the simple solution worked once the communication problems were fixed.

5 Experiences of the XP Practices This section discusses the experiences gathered through the observations and interviews made at Online Telemarketing practice by practice. This strategy of structuring the experiences seemed at the time to provide the most complete account for describing the experiences gained. If the experiences were not structured by practice it was thought that experiences not thought vital for this group might not be included and thereby not available to other groups.

The developers were introduced to XP by a half-day seminar with an introduction and an extreme hour exercise [10]. The developers had guidance from the coach regarding how they should implement XP at all times. XP books [1, 11, 12] were also made available to them. The developers were also instructed to look at XP websites to keep up to date on recent developments in the XP community [10, 13, 14, 15].

5.1 The Planning Game Using story cards proved to be one of the greatest successes of all the XP practices. The story cards provided all parties involved with a picture of the status of the work and an overview of the product as a whole. Approximately 150 stories have been implemented in total. The stories were written by the customer and then prioritised together with the development manager as he had the best overview of the technical status of the product. The estimation worked well once the management understood the three levels of prioritisation [1, 2, 3].

New stories were added continuously during the whole project. This was due to the fact that the management did not have a clear picture of the product at the start of the project. This meant that functionality was continuously added during the entire project. The time estimation of the stories was difficult at first due to the lack of practical experience of

97

estimating, but after a few weeks the estimating worked very well according to the group members. The estimation quality was not confirmed using quantitative methods. The whole group performed estimations together during planning meetings.

Breaking the stories into tasks was difficult for the developers to grasp. The developers ended up drawing flow charts for the work, which was not the idea. Some of the story cards were very similar to tasks, i.e. at a too detailed a level for story cards. The problem was thought to be due to the difficulty of setting some kind of a common detail level for the stories.

The developers selected the stories to develop in conjunction with the development manager. This way of working with stories and tasks is an area that was continuously looked at and improved during the course of the project.

5.2 Small Releases Creating a minimal framework for each part of the system proved to take longer than the following smaller releases. The very first iteration took much more time than intended due to lack of experience in using the XP methods and traditional development thinking dominating. Once a complete bare working system was implemented, however, small releases were easier to implement.

During the long initial releases it was important to keep good communication between the customers and developers so that the project did not proceed in the wrong direction. As an afterthought, this practice seems fundamental to the success of XP. Maybe more effort should have been exerted to keep the initial release time shorter.

5.3 Metaphor The system metaphor created before the actual start of the development was a little too detailed. It was almost an attempt at a complete requirements document. This was partly due to the fact that this document was written before the XP methodology was first thought of for the project. The document was not altered after XP was selected as the preferred development method. The metaphor document was also not properly updated as the system evolved during the course of the project. This is most probably also due to the too detailed level of the system metaphor. A common picture of the system was gained throughout the project by looking at the system directly and discussing individual cards.


98

This common picture could have been improved by creating an accurate system metaphor.

5.4 Simple Design The development team has strived to implement the simplest possible solution at all times in accordance with this XP practice. A further evaluation of this practice was deemed to be difficult to perform in a reasonable amount of time.

The philosophy of always assuming simplicity was thought to have saved time in the cases where a much larger solution would otherwise have been implemented. Time was also believed to have been saved due to the fact that developers did not have to cope with a lot of unnecessarily complicated code.

5.5 Testing Test-first programming was difficult to implement at first. Determining how to write tests for code proved difficult to master. The developers thought that the tests were hard to write and they were not used to thinking the test-first way. It was found difficult to see how many tests were enough to satisfy that the desired functionality would be implemented correctly.

The VB Unit test structure [12, 13] was used to create the automatic unit tests. VB Unit takes quite a long time to get used to and set up according to the developers. The unit tests that were written take less than one minute to run in total. The whole set of tests were run each time new code was integrated. During the course of the project the developers started to ignore writing tests first, especially when the project came under time pressure a few months in. The developers understood why tests are important but thought it involved too much work and did not see the short term benefits. It is believed that this was due to the inexperience of the developers. A more rigorous approach to the testing practices would most probably have been preferable.

The developers found programming by intention difficult. Programming by intention involves deciding the functionality and structure of the code in advance so that the test cases can be created beforehand. The development manager, who is experienced in coding these kinds of systems, found this way of working natural. He actually found that the way he usually worked was very close to the way described

99

by XP. It was found that database code was much easier to write test code for than business rule code. The graphical user interface (GUI) code was also, as expected beforehand, hard to write automated tests for. Because of the limited nature of the GUI it was decided that an automated test tool for GUI testing would probably take longer to take into practice than manual user testing.

As the project came under pressure to release the fully operational version, the test first method of working ceased completely. The time pressure was due to the expansion of the company into a new region earlier than first expected. This meant that a portion of the new system was desired to go into operation earlier than the initial planning.

The functionality of the system was tested by the customer before each release as well as spontaneously during the development. When the functionality was not as the customer had intended it, a correction card was written. At first the customer just interrupted the developers when they found the functionality inconsistent with the desired functionality, but this was found to be too disruptive so a correction card strategy was adopted. The functional testing provided a good view of how the product is progressing.

5.6 Refactoring No tools for refactoring were used in the project. All project members performed minor refactoring continuously. No major refactoring of the code was performed, but assessment of the code was performed continuously regarding the benefits of a major refactoring in case it was necessary. No education or training was given either beforehand or during the course of the project in refactoring methods or theory such as those presented by Fowler [16].

5.7 Pair Programming The developers used pair programming at all times. The only exceptions were when illness intervened or the developers had demanding schedules at the university. The developers adopted pair programming cautiously at first, but then gradually started to work naturally and effectively in the pairs.

The fact that the developers had no prior professional experience probably made the introduction of pair programming much easier than if they had been used to working in a traditional single-programmer manner.


100

When alone, the programmers often seemed to seize and get stuck when solving a problem. Also the tendency to carry on with a nonworking solution seemed more frequent. The developers found it easier to keep their concentration on the task at hand when working in pairs.

The development leader estimates that the pair programming produced the code faster than if the same programmers would be working separately. However the inexperience of the developers made them much slower than experienced professionals.

The pair programming worked excellently when introducing new people into the project. For the first part of the project the pairs were been fixed so that the developers could synchronise their schedules easily, but during the second phase of the project when 8 people were working full time on the project, the pairs were changed continuously. The original 4 developers also chose their own pair-programming buddy, but the second group were assigned into pairs by management.

5.8 Collective Ownership Collective ownership worked well in the project. This contributed to solving some minor irritation among the developers due to defects found in the code. When the programmers thought of defects as a group issue, rather than someone else’s ‘private’ defect the irritation disappeared and a constructive atmosphere was created. The only problem observed in this practice was due to the configuration management or rather lack of effectiveness in the communication in the handling of the configuration management. The developers were on occasion afraid to change parts of the code due to the risk of loosing work if not in direct contact with the other pairs.

5.9 Continuous Integration Continuous integration proved to be natural in the development environment created for the project. As soon as code was finished it was integrated into the product. The ease with which this practice was implemented is notable in itself.

101

5.10 40-Hour Week As the developers all worked part time, 20-hours per week, this practice was adjusted to accommodate this. Only the development manager and senior management worked full time.

5.11 On Site Customer The customer was available throughout the course of the project. This worked very well. The only problems were the flexible work hours of both developers and management and everyone’s busy schedules. While the senior management of the company had the role of customer, they were not been able to devote all of their available time to this project, because of other meetings and responsibilities in running the company. At the start of the project the customer had many opinions on the functionality in the product. As soon as a release was made the customer wanted to modify or add to it. This decreased during the course of the project, partly due to the system evolving into what the customer wanted and partly due to that the customer became better at writing story cards describing the desired functionality to the developers more efficiently.

5.12 Coding Standards A coding standard document was created at the start of the project. This was used extensively at first and added to when needed. After a while the developers became more relaxed and used the coding standard less. This was at the time identified as an issue and was re-enforced with success. The outcome of this practice has however not been evaluated by comparing sections of the actual code with the coding standard.

6 Quality of Conclusions In this section the criteria discussed in section 3, methodology, are discussed with regards to the research methodologies employed in this paper.

• Credibility The fact that both interviews and observations were used in the study increases credibility. The resulting observations do not seem to be incredible. The resulting observations seem to be correct when reviewed by the development manager at Online Telemarketing.


102

• Transferability The experiences from introducing XP in this project should be of considerable help to other projects introducing XP, either in part or fully. Consideration should be taken to the facts that the developers were working part time and were otherwise university students, not full time, experienced professionals. • Dependability Due to the limited nature of the study in this experience report it is difficult to assess the dependability of the study. • Confirmability The confirmability is increased by the review by the development manager at Online Telemarketing.

The quality of the conclusions is increased by the triangulation of

qualitative research methods. Both interviews and direct observations were used and the results were reviewed by a representative of the participating subjects.

7 Summary and Conclusions In conclusion, the project at Online Telemarketing was a success. The product was created and is now functioning live. The experiences of the actual XP practices are a mostly successes, but also a few failures. All of these experiences are relevant to projects considering introducing XP.

The planning game was easy to introduce and effective. This can be partly due to the fact that the extreme hour, used to initially introduce XP, focuses on the planning game, as does extensive parts of the XP literature [e.g. 1, 11, 12].

Small releases proved difficult for the first releases for each part of the system. Even though they were expected to take a little longer than the other releases, they took longer than planned. An increased focus on only creating an absolute minimal framework system might help this.

The system metaphor was too complicated to start with. This resulted in a metaphor document that did not evolve with the system. It should not be difficult to keep the metaphor up to date if it is simple from the start.

Simple design was thought to work well, but was not verified by code inspections. The developers believed that by thinking in terms of simple solutions as much as possible, they saved a lot of time by not having to try

103

to understand unnecessarily complicated code. Testing was found to be one of the hardest practices to implement. It

requires careful preparation of the testing unit and also a strict discipline among the testers to always write the tests first. The testing practice was the first practice to cease when the project came under pressure.

Refactoring was performed on a small scale all the time. This is, however, natural in normal programming. Larger scale refactoring was not performed, although the possibility of large scale refactoring was continuously evaluated.

Pair programming worked excellently for the developers in the project. It seemed to help them solve difficult problems faster and identify potential dead-end solutions earlier. The pair programming also worked very well when introducing new people into the project

Although it was different from what the developers were used to from the start, collective ownership proved to be effective for the team spirit.

Continuous integration was not hard to implement and was found a natural way to work in the development environment created in the project.

The on-site customer practice worked well. The customer solved many misunderstandings of functionality early and was available to complete or clarify any poorly written story cards. As the customer did not really know the full extent of the product at the start of the project, this practice appears to be one of the major reasons for the success of the project.

The coding standard practice worked well. When the developers started to get sloppy in the middle of the project, the development manager enforced the coding standard again.

Keeping in mind the issues raised in the quality of conclusions section, section 7, these experiences should be of interest to any development team considering introducing XP.

8 Acknowledgements This work was partly funded by The Swedish Agency for Innovation Systems (VINNOVA), under a grant for the Center for Applied Software Research at Lund University (LUCAS). The author would also like to thank Johan Norrman (Online Telemarketing) and Per Runeson (LTH) for their contributions to the paper.


104

9 References [1] Beck, K., Extreme Programming Explained: Embrace Change, Addison Wesley,

1999. [2] Beck, K., “Embracing Change with Extreme Programming”, IEEE Computer,

October 1999, pp. 70-77. [3] Martin, R., C., “Extreme Programming Development through Dialog”, IEEE

Software, July/August 2000, pp.12-13. [4] Haungs, J., “Pair Programming on the C3 Project”, IEEE Computer, February

2001, pp. 118-119. [5] Hicks, M., XP Pros Take it to the Extreme, ZDNet eWeek News, confirmed

010903, http://www.zdnet.com/eweek/stories/general/0,11011,2714342,00.html .

[6] Lincoln, Y., S., Guba, E., G., Naturalistic Inquiry, Sage Publications, 1985. [7] Robson,C., Real World Research, Blackwell Publishers, Oxford, 1993. [8] Miles, M.B., Huberman, A.M., Qualitative Data Analysis, Sage Publications,

1994. [9] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., Wesslén, A.,

Introduction to Experimentation in Software Engineering, Kluwer Academic Publishers, 2000.

[10] Multiple Authors, The Extreme Programming Roadmap, last confirmed

010903, http://www.c2.com/cgi/wiki?ExtremeProgrammingRoadmap . [11] Beck, K., Fowler, M., Planning Extreme programming, Addison Wesley, 2000. [12] Jeffries, R., Anderson, A., Hendrickson, C., Extreme Programming Installed,

Addison Wesley, 2000. [13] Jeffries, R., (Ed.), XProgramming.com, last confirmed 010903,

http://www.xprogramming.com . [14] Wells, D., Extreme Programming: A Gentle Introduction, last confirmed

010903, http://www.extremeprogramming.org/ . [15] Multiple authors, Extrem Programmering, (in Swedish), last confirmed

010903, http://oops.se/cgi-bin/wiki?ExtremProgrammering . [16] Fowler, M., Refactoring: Improving the Design of Existing Code, Addison Wesley,

2000.

105

Decision Support for Extreme Programming Introduction and Practice Selection

Karlström, D., Runeson, P., “Decision Support for Extreme Programming Introduction and Practice Selection”, Proceedings The Fourteenth International Conference on Software Engineering and Knowledge Engineering (SEKE'02), Ischia, Italy, 2002.

Abstract This paper presents an investigation concerning the introduction of Extreme Programming (XP) in software development organisations. More specifically the concept of using a decision support method known as the Analytical Hierarchy Process (AHP) is evaluated by a group of students and a group of developers and the outcome is compared to experiences from an XP case study. The results provide an indication that different practices are thought to be easier and more effective to implement in the two groups. A company considering implementing only a few practices can use this as help for deciding which practices to implement. Companies introducing all practices can use the results of this kind of method to see where more attention might be needed after or during the introduction of XP.

3


106

1 Introduction Extreme programming (XP) [1,2] has come into focus recently as an agile approach to software development. Extreme programming consists of a set of practices, which can be implemented separately or in combination. When introducing XP, it is in many circumstances impossible to start with all the practices involved. This paper discusses different strategies for introducing XP in various organisations and how to support the strategy decision.

Software process improvement in general suffer from that there are different viewpoints of which process is used, and which process should be used. Development processes are documented in official documents, but as they are interpreted differently by different individuals, different versions appear depending on the perception of the process. Bandinelli et al. describe these different perceptions of the process and name these as the official, perceived and desired process [3]. The viewpoints are typically different depending upon the individual’s role in the organisation [4]. The same situation is expected with respect to opinions on the different practices in XP.

In order to support the strategy decision regarding XP introduction, a decision support methodology called the Analytical Hierarchy Process (AHP) [5] is used in order to investigate the issues involved in selecting XP practices. A case study is performed with two different groups prioritising XP practices. The outcome is validated by comparing it to experiences gained from a case study of introducing XP [6].

The paper is structured as follows. In Section 2, the practices of XP and the possible strategies for introducing these are presented. Section 3 presents the methodology for the empirical study, while Section 4 presents the outcome of the investigation. Section 5 interprets the data and finally Section 6 presents a summary.

2 XP Introduction Strategies Two distinct strategies are possible when introducing XP into an organisation [1]. Either (1) all practices are introduced at once or (2) a fewer number practices are selected for introduction.

In the first case, the methods described in this paper can be used to investigate where more introduction support might be needed to avoid difficulties, or where the XP framework might need tailoring to suit the particular organisation in the future. In the second case, the method

107

might be used to select the practice that is thought to be the most effective.

An introduction of the complete XP framework in a large organisation provides greater challenges than in a small one. Consideration needs to be taken to differences throughout the organisation, communication between different groups in the organisation and also differences in attitudes towards XP or a change in methodology at all. It is common to limit initial introduction to a pilot group to study the effects before a total introduction is performed in a large organisation. Before a change in methodology, the use of decision support tools can give help in making crucial decisions such as full or partial introduction, or which practices to implement.

2.1 XP Practices XP is composed of a few fundamental values, principles and activities, which are implemented by 12 practices. The fundamental values are communication, simplicity, feedback and courage. The fundamental principles are: rapid feedback, assume simplicity, incremental change, embracing change and quality work. The basic activities are coding, testing, listening and designing.

The 12 practices that are intended to realise all this are described in the following list [1].

• Planning Game Quickly determine the scope of the next release by combining business priorities and technical estimates. As reality overtakes the plan, update the plan. • Small Releases Put a simple system into production quickly, and then release new versions on a very short cycle. • Metaphor Guide all development with a simple shared story of how the whole system works. • Simple Design The system should be designed as simply as possible at any given moment. Extra complexity is removed as soon as it is discovered. • Testing Programmers continually write unit tests, which must run flawlessly for development to continue. Customers write tests demonstrating that features are finished.


108

• Refactoring Programmers restructure the system without changing its behaviour to remove duplication, improve communication, simplify, or add flexibility. • Pair Programming All production code, i.e. code that is actually used in the final product, is written with two programmers at one machine. • Collective Ownership Anyone can change any code anywhere at any time. • Continuous Integration Integrate and build the system many times a day, every time a task is implemented. • 40-hour Week Work no more than 40 hours a week as a rule. Never work overtime a second week in a row. • On-site Customer Include a real, live user on the team, available full-time to answer questions. • Coding Standard

Programmers write all code in accordance with a set of rules emphasizing communication through code.

The practices can be introduced independently or as a whole

depending on the situation in the development organisation. The practices are intended as a starting point for a development team. The team should start using the practices as they are described in XP and then through continuously adapting and evolving the methodology used, optimise them towards the team’s own preferred method of working under the current circumstances.

2.2 XP Roles An XP project works best if certain roles are assigned to the team members so that they each have different responsibilities regarding the previously described practices. The roles do not necessarily need to represent individual persons. One role can be assumed by several people and conversely one person can assume several roles if need be. The roles in XP are briefly described in the following list [1].

109

• Programmer The programmer is at the heart of XP. The programmer creates the product based on the story cards written by the customer. • Customer The customer provides the functionality to be implemented in the form of story cards. The customer also creates, sometimes with the help of the tester, functional tests to ensure that the product does what it is supposed to do. • Tester The tester role in XP is focused towards the customer to ensure that the desired functionality is implemented. The tester runs the tests regularly and broadcasts the results suitably. • Tracker The tracker role is to keep an eye on the estimates and compare these to the performance of the team. The tracker should keep an eye out for the big picture and remind the team, without disrupting it, when they are off track. • Coach The coach’s role in XP is to be responsible for the process as a whole, bringing deviations to the team’s notice and providing process solutions. • Boss The boss’s role in XP is basically to provide for the needs of the team to ensure it get the job done.

3 Methodology

3.1 Decision Support Method The decision support method is basically composed of four steps:

A: Present the XP practices to the subjects. B: Allow the subjects to rate the factors anonymously by

pair-wise comparisons, based on their personal viewpoints.

C: Analyse the data using AHP. Aggregate and analyse the rating results and identify discrepancies.

D: Strategy decision. Decide strategy based on the outcome of the analysis


110

The rating of the practices should be performed by a suitable sample of all people involved in the development organisation that are active in the development. In a small group, a complete sample can be used, but in a large organisation this would not be practically possible. In such a case, the sample should be designed to ensure that the results are affected as little as possible by the sampling.

The method used for rating the factors is the Analytic Hierarchy Process (AHP). The AHP is a method for evaluating alternatives in a choice situation [5]. It uses pair-wise comparisons and measures each alternative’s relative contribution to the rating. Each subjects’ individual set of comparisons is put into a judgement matrix. From this matrix the relative weights for the compared items can be calculated for each individual. These relative weights are known as priorities. The process also gives the possibility of calculating how consistent the performed ratings are for each individual as the pair-wise comparisons imply that the alternatives are indirectly compared several times. This was one of the main reasons for selecting the method.

The AHP process has been implemented for applications outside the process improvement domain. Karlsson et al. [7], for example, use the process to select between customer requirements. This method of comparing requirements has been further developed into a commercial tool developed by FocalPoint AB [8].

The XP practices are rated with respect to their expected effect in the organisation and with respect to their expected ease of introduction. The effect is of primary interest when selecting a subset of practices while the ease is of primary interest when preparing for an introduction of all the practices. However, the selection of a subset will always be a trade-off between ease and effect. These factors are used in a general sense, providing a simple concept for the rating of the practices rather than being based on exact definitions. It is thought that the members of the project would capture the group’s characteristics and needs and rate the factors according to this. If certain other more specific factors have been identified as important, these factors could be added adjacent to ease and effect in order to further investigate the perceptions of the group and provide further information on which to base decisions.

The outcome of the rating is a vector of relative priorities for each rated item, which in turn is the input to the analysis step. When using the AHP method with several subjects, there are some options to be chosen among for the aggregation of the rating results.

When aggregating results from an AHP comparison process we must

111

decide whether we wish to aggregate the judgements of each individual or the priorities calculated for each individual. This is referred to as aggregation of individual judgements (AIJ) and aggregation of individual priorities (AIP) respectively. The decision is determined by whether the group is assumed to act as a unit or as separate individuals [9].

Secondly it must be decided whether to use the arithmetic or geometric mean for the aggregation. For AIJ the geometric mean must be used. For AIP any of the two may be used, but the arithmetic mean has the advantage of being comparable to the original values. We have hence chosen to use AIP aggregation with arithmetical means.

The outcome from the AHP analysis is then analysed using descriptive statistics and statistical analyses. The statistical methods described in this section are examples of appropriate methods for analysis of the data. There are several other methods that are appropriate and could be used.

An overall view of the data can be given by using bar plots of the means for each group. Further analysis of the ratings can be performed using an ANOVA test [10], if the data is normally distributed and the variances are equal. Significant rating differences can then be established within each group using Fisher’s PLSD test [10]. The normal properties of the data are checked using a Kolmogorov-Smirnov test for normal distribution. If the assumptions for ANOVA are not met by the data, a Kruskal-Wallis non-parametric test can be performed instead [11].

Figure 1 Decision support using the method

The results from the analysis are intended to support the decisions taken prior to XP practice introduction. The choices performed prior to introduction are illustrated in Figure 1. The first choice, partial or

AHP

Com-plete

Partial

Low support

High support

Removed practices

Adapted practices

Even profile

Un-even profile

High pos. effect

Low pos. effect

High pos. effect

Low pos. effect

Discrepancy analysis and cost evaluation


112

complete introduction is often decided beforehand, but results from the method may change this. For example, if a complete introduction has been decided on beforehand and certain practices appear difficult to implement with low positive effect, a partial introduction may be considered instead. If a complete introduction is selected, the outcome of the method is used to identify practices that need more support, e.g. training courses, to ensure that they are introduced properly. Similarly, practices may be identified that need less support. If a partial introduction has been chosen, the outcome provides information on which practices to implement and which to reject. Discrepancies between people and groups provide additional information for the decision makers as well as the ease of introduction.

Discrepancies between groups of persons can give vital SPI strategy information. The ratings of one factor by two groups are compared using an unpaired t-test if the ratings are normally distributed. If the ratings are not normally distributed a Mann-Whitney test is performed [11].

The final outcome is a list of which practices are given priority by each group, and a list of significant differences between the groups, if any. This information can be used in the decision process. If all XP practices shall be used, the results about ease of introduction is a support when planning for training and coaching efforts. If a subset of practices shall be used, the expected effect may guide the selection. Significant differences between groups provide certain information on that there are different expectations on the XP practices.

3.2 Evaluation Method In order to evaluate the method, it was applied in a case study. The outcome of the case study, in turn, was validated against data from a case study based on the actual introduction of the XP practices in a development project [6]. The purpose of this first study was to investigate the introduction and use of XP in a real-world environment.

The case study application of the method presented in this paper was conducted during spring 2001 as a part of a graduate course in software engineering at Lund University. The students taking this course are in fact the subjects, referred to as “students” in the study. In addition, a group of industry people acted as subjects, referred to as “industry”. The case study applied one treatment, i.e. the decision support method, to two types of subjects, i.e. one group of industry people and one group of students. The industry people represent one viewpoint and the student group represent

113

another viewpoint. The industry group was a team of 5 developers about to commence development using XP and the students were a group of 26 graduate students at the computer science and engineering graduate programme at Lund University. The students were working in pairs.

The subjects were given an introduction to XP. The students were asked to read an introduction to extreme programming [2] and were presented with the practices in a class-room environment. The developers were presented to extreme programming through an introduction day with an introductory presentation, discussions and an extreme hour [6], a group exercise performed during one hour illustrating a simplified version of XP.

After the XP introduction, the subjects were asked to rate the factors first according to their expected positive effect on development and then according to their expected ease of introduction. Both groups performed the rating of the practices via a web-based interface, and the AHP calculations were performed using the FocalPoint tool [8]. The students performed the rating in a classroom environment in pairs, while the developers performed the rating in the office environment one-by-one. The FocalPoint tool allows incomplete ratings, i.e. that all pairs are not compared, but all the subjects in the study performed ratings of all pairs of XP practices.

The outcome from the ratings is a vector of ratings for each of the five industry subjects and one for each of the 13 pairs of student subjects. Furthermore, a consistency index is calculated, which indicates how consistent the subjects are in their ratings. The rating vectors are analysed according to the method and the outcome is a list of which of the practices where considered significantly more easy and more effective than others. In addition, significant differences between the subject groups are identified.

Finally, the outcome of this case study is compared to the qualitative judgement of the XP practices based on the use of XP in another case study. Section 4 presents the result of this evaluation.

3.3 Validity The purpose of the empirical study is not to come to some certain conclusion on XP practices, but to illustrate and make credible that the decision support method is useful and provides relevant information to decision makers. Below, the issues of threats to internal, conclusion, construct and external validity are briefly discussed [12].


114

The mostly discussed internal validity threat is related to selection of subjects. Using students as subjects tend to be criticized, but in this case study, the most important characteristics of the subject groups is that they are somewhat different with respect to the context in which they perform the method. This is, however, used to illustrate the ability of the method to identify such differences.

The subjects used in the study did not have very much experience in XP other than the brief training provided. This was intended to produce a similar situation to a company introducing XP. Before the introduction of XP the people involved are not likely to have much experience of XP in a real world situation.

Threats to conclusion validity concern statistical tests. Standardized and robust tests are used. In cases of non-normality, these are reported openly and the analyses are mainly based on non-parametric tests.

Threats to construct validity concern the definition of the study itself. This may be a threat as the method is not compared to other methods. However, the validation against an observational study of the industry subjects as they complete a project using XP [6] is intended to reduce this threat.

Finally, threats to external validity concerns generalization. As the study is a case study, there is not intention of finding a general result, but to illustrate the outcome in a certain context and show that the method can provide a useful result.

4 Evaluation Results

4.1 Method Results The case study was performed successfully, and all the 31 subjects completed their ratings of the XP practices. The students performed the rating in pairs due to practical reasons, making the number of student ratings 13. The consistency ratio was calculated and found to be sufficient for all except one subject in each group. The subjects having a consistency ratio of 0.2 or more were removed as outliers [7]. The limit of 0.2 is fully satisfactory in a practical application of AHP. In total there were 4 industry subjects and 12 pairs of student subjects remaining in the study after this step.

The individual ratings were first aggregated using the arithmetic mean of the ratings for each factor. This data were plotted in bar plots, see Figure 2. A Kolmogorov-Smirnov test was then performed on the data to

115

check if it was normally distributed. The next stage of analysis was to perform an ANOVA test. As we could not formally prove the normal distribution of all the factors, we also performed the non-parametric alternative, the Kruskal-Wallis test. This test gave the same result as the ANOVA test for the expected effect, but differed in one case for the expected ease.

The final test that we would like to perform is the Fisher PLSD test to investigate the individual relationships in the data, but this requires that the preconditions for the ANOVA are satisfied. In the cases where the ANOVA provided the same result as the Kruskal-Wallis test, at a significance level of 0.05, the Fisher PLSD test was used nonetheless [10,13].

The bar plots (Figure 2) indicate that the industry subjects rank short releases as the most effective practice and on-site customer as the second most effective. The students rank the testing approach having the largest effect, while the collective ownership is ranked lowest. Industry subjects rank pair programming as the by far easiest practice to introduce, while the students expect 40-hour week to be the easiest. The Kolmogorov–Smirnov test was performed and the data was found to be normally distributed on a significance level >0.9 as reported in Table 1.

Ease Effect Industry 5 5 Student 8 10

Table 1 Number of normally distributed cases out of the 12 practices

Both ANOVA and Kruskal-Wallis tests were run to identify whether there are any significant differences in the rankings. The p-values are reported in Table 2. It can be noted that the ANOVA test and the Kruskal-Wallis test give significant results on the same cases for effect. Hence, we can identify significant differences concerning the ranking of effects, while the differences concerning ranking of ease of introduction are not statistically significant.

Ease Effect ANOVA K-W ANOVA K-W Industry 0.014* 0.15 0.065* 0.011* Student 0.19 0.51 0.007* 0.010*

Table 2 P-value of ANOVA and Kruskal-Wallis tests (* indicates significant results)


116

Figure 2 Bar graphs showing mean ratings for: (a) developers rating effect of practice, (b) developers rating ease of introduction of practice, (c) students rating effect of practice, (d)

students rating ease of introduction of practice

In the cases where there are significant differences, we use the Fisher

PLSD test to identify which differences are significant. As seen in Table 1, all data is not normally distributed, but as the test are robust and ANOVA and Kruskal-Wallis provide the same result, we continue with the Fisher PLSD test. The PLSD test provides comparisons between each pair of practices. The full analysis is too extensive to report here, but we highlight a few interesting results.

For the effect rankings by the industry subjects, the highest ranked practice, short releases, is significantly different from five out the remaining eleven practices. The second ranked practice, on-site customer,

0

,03

,05

,08

,1

,13

,15

,17

,2

Cel

l Mea

n

Pla

nnin

g G

ame.

2

Sho

rt R

elea

ses.

2

Met

apho

r.2

Sim

ple

Des

ign.

2

Tes

ting.

2

Ref

acto

ring.

2

Pai

rpro

gram

min

g.2

Col

lect

ive

Ow

ners

hip.

2

Con

tinuo

us In

tegr

atio

n.2

For

ty-h

wee

k.2

On-

site

Cus

tom

er.2

Cod

ing

Sta

ndar

d.2

Cell

Interaction Bar Plot for PracticesEffect: Category for PracticesInclusion criteria: cost ovo from all.svd

0

,02

,04

,06

,08

,1

,12

,14

Cel

l Mea

n

Pla

nnin

g G

ame.

2

Sho

rt R

elea

ses.

2

Met

apho

r.2

Sim

ple

Des

ign.

2

Tes

ting.

2

Ref

acto

ring.

2

Pai

rpro

gram

min

g.2

Col

lect

ive

Ow

ners

hip.

2

Con

tinuo

us In

tegr

atio

n.2

For

ty-h

wee

k.2

On-

site

Cus

tom

er.2

Cod

ing

Sta

ndar

d.2

Cell

Interaction Bar Plot for PracticesEffect: Category for PracticesInclusion criteria: effect ovo from all.svd

0

,02

,04

,06

,08

,1

,12

,14

Cel

l Mea

n

Pla

nnin

g G

ame.

2

Sho

rt R

elea

ses.

2

Met

apho

r.2

Sim

ple

Des

ign.

2

Tes

ting.

2

Ref

acto

ring.

2

Pai

rpro

gram

min

g.2

Col

lect

ive

Ow

ners

hip.

2

Con

tinuo

us In

tegr

atio

n.2

For

ty-h

wee

k.2

On-

site

Cus

tom

er.2

Cod

ing

Sta

ndar

d.2

Cell

Interaction Bar Plot for PracticesEffect: Category for PracticesInclusion criteria: effect students from all.svd

0

,02

,04

,06

,08

,1

,12

,14

Cel

l Mea

n

Pla

nnin

g G

ame.

2

Sho

rt R

elea

ses.

2

Met

apho

r.2

Sim

ple

Des

ign.

2

Tes

ting.

2

Ref

acto

ring.

2

Pai

rpro

gram

min

g.2

Col

lect

ive

Ow

ners

hip.

2

Con

tinuo

us In

tegr

atio

n.2

For

ty-h

wee

k.2

On-

site

Cus

tom

er.2

Cod

ing

Sta

ndar

d.2

Cell

Interaction Bar Plot for PracticesEffect: Category for PracticesInclusion criteria: cost students from all.svd

(a)

(d)(b)

(c)Industry, Effect Student, Effect

Industry, Ease Student, Ease

117

is significantly different from only two of the remaining eleven practices. Except for these, there is only one significant difference, i.e. between the planning game and the continuous integration practices.

The case is similar for the student ratings. The highest ranked testing approach is significantly different from seven of the other practices and the lowest ranked collective ownership is significantly lower than 4 other practices. In addition, simple design is found to significantly higher ranked than two other practices.

A further analysis was performed by a pair-wise comparison of the means for each practice between the student and industry group. Normality was checked by using the Kolmogorov–Smirnov test for normality on each set of means. The results rejected normality in one of the four sets, the factor ease in the group industry. The results are summarised in Table 3.

Ease Effect Industry Rejected >0.9999 Student >0.9999 >0.9999

Table 3 Normality check for the means of practices within each group

In the case of comparing the two groups for effect, this can be performed using the pair wise t-test [10] as both groups are normally distributed. The test shows a significant difference between the groups. As the groups are not normally distributed for the factor ease, the preconditions for the t-test are not fulfilled. A non-parametric test was attempted instead, the Wilcoxon Signed rank test. This test could not, however, show a significant difference between the groups. With this information at hand, the decision on improvement strategy can take place, see Figure 1. In the case of selecting a subset of XP practices, short releases, on-site customer and testing are ranked the highest. The two groups have different top-ranks, and hence the rationale behind their ranking should be investigated.

In the case of introduction of all XP practices, it can be concluded that the industry group assumes that pair programming and collective ownership require little support for the introduction, while the other practices are evenly ranked. The students assume 40 hour week and on-site customer will be the easiest to introduce.

The fact that a significant difference could be shown using the t-test is interesting. If the groups were within a large development organisation, a different XP implementation strategy might be considered where groups are significantly different.


118

Note that the results from the application of the method are specific to the environment within which the experiment was performed. The intent of this paper is not to investigate a general ease and effect profile for XP but rather to provide and test a tool that can show this relationship in specific cases. The actual ease/effect results therefore are therefore irrelevant to the reader. The factors ease and effect could in other companies be replaces by other factors considered more important to the XP introduction, for instance lead-time reduction or quality improvement

4.2 XP Application Case Study A qualitative XP case study is conducted at the company where the industrial subjects performed the development project using XP [6]. Here the XP practices were used by 10 people during the project. The experiences on the ease of introduction and effect of applying the practice have been summarised and are presented in Table 5. The practices are judged qualitatively with respect to ease and effect using the scheme in Table 4.

-- - 0 + ++ Ease Very hard Hard Neutral Easy Very easy Effect Very neg. Neg. Neutral Pos. Very pos.

Table 4 Coding Scheme

XP Practice Ease Effect Planning Game - ++ Small Releases + ++ Metaphor - - Simple Design + + Testing -- ++ Refactoring + + Pair Programming ++ ++ Collective Ownership - + Continuous Integration ++ + 40-hour week + 0 On-site customer + ++ Coding standard - +

Table 5 Case Study Summary

It can be seen in the table that some of the practices identified using the method presented in this paper before the start of the project as effective, were also actually found to be effective during the course of the project., for instance short releases and on-site customer. The students

119

ranked testing very high, which in the case study was considered having very large effect.

However, a major difference between the results of this qualitative evaluation and the systematic decision support method, is that the qualitative evaluation does not involve any systematic ranking and comparison. Hence, two practices judged on the same ease or effect level, cannot be compared. The decision support method provides complete comparisons between all pairs of practices.

5 Summary Extreme programming (XP) has been set in focus recently as the most widely spread example of agile methodologies for software development. As XP consists of a set of practices, the introduction of XP in an organisation involves strategic decisions whether to introduce all practices, or a subset. In the latter case, decision support is needed to decide which subset to select. In the first case, information about expected problems in the introduction is valuable in the practical planning of for example training and coaching efforts.

In this paper, a decision support method, based on the Analytic Hierarchy Process (AHP) is presented. The method is applied to a case study with industry and student subjects, which purpose was to rate the XP practices with respect to ease of introduction and expected effect of the practice. Furthermore, the outcome is validated by comparing the results with the qualitative evaluation of the application of XP in a project with 10 participants.

The outcome of the application is that the industry group and the student group have different views of both ease and effect of the practices. This is an interesting result as such, since this indicates that measures have to be taken to motivate or train people in the process improvement process.

The proposed method hence gives support for the decision process by visualising opinions in the organisation, and thereby improving the chances for good decisions, that are firmly anchored in the organisation.

6 Acknowledgements This work was partly funded by The Swedish Agency for Innovation Systems (VINNOVA), under a grant for the Center for Applied Software Research at Lund University (LUCAS). The authors would also like to thank Martin Höst (LTH) for his contributions to the paper.


120

7 References [1] Beck, K., Extreme Programming Explained: Embrace Change, Addison

Wesley, 1999. [2] Beck, K., “Embracing Change with Extreme Programming”, IEEE Computer,

October 1999, pp. 70-77. [3] Bandinelli, S., Fuggetta, A., Lavazza, L., Loi, M., Picco, G. P. “Modelling and

Improving an Industrial Software Process”, IEEE Transactions on software Engineering, Vol. 21, No. 5, 1995, pp. 440-454.

[4] Karlström, D. , Runeson, P., Wohlin, C., ”Aggregating Viewpoints for

Strategic Software Process Improvement – A Method and a Case Study” proceedings 6th International Conference on Empirical Assessment & Evaluation in Software Engineering, EASE 2002.

[5] Saaty, T. L., “The Analytic Hierarchy Process” McGraw-Hill, New York, 1980. [6] Karlström, D., “Introducing Extreme Programming – An Experience

Report”, proceedings 3rd International Conference on eXtreme Programming and Agile Processes in Software Engineering, XP2002.

[7] Karlsson, J., Ryan, K., ”A Cost-Value Approach for Prioritising Requirements”, IEEE Software September/October, 1997, pp. 67-74.

[8] Home page of Focal Point AB, accessed 02/03/05,

http://www.focalpoint.se/index_eng.html.

[9] Forman, E., Peniwati, K., “Aggregating Individual Judgements and Priorities with the Analytic Hierarchy Process”, European Journal of Operational Research, 1998, pp. 165-169.

[10] Montgomery, D. C., “Design and Analysis of Experiments”, John Wiley and

sons, 1997. [11] Siegel, S., Castellan R., N. ., “Nonparametric Statistics”, McGraw-Hill

International, 1998. [12] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., Wesslén, A.,


[13] Bratthall, L., Wohlin, C., “Understanding Some Software Quality Aspects from

Architecture and Design Descriptions”, International Workshop on Program Comprehension, 2000, pp.27-36.

121

A Minimal Test Practice Framework for Emerging Software Organisations

Karlström, D., Runeson, P., Nordén, S., "A Minimal Test Practice Framework for Emerging Software Organisations", accepted for publication in the journal of Software Testing Verification and Reliability, preliminary publication issue: June 2005.

Abstract Testing takes a large share of software development efforts, and is hence of interest when seeking improvements. Several test process improvement frameworks exist, but they are extensive and much too large for smaller organisations to be effective. This paper presents a minimal test practice framework (MTPF) that allows the incremental introduction of appropriate practices at the appropriate time in rapidly expanding organisations. The process for introducing the practice framework tries to minimise resistance to change by maximising the involvement of the entire organisation in the improvement effort and ensuring that changes are made in small steps with a low threshold for each step. The created practice framework and introduction method are evaluated at one company by applying the framework for a one-year period. 12 local software development companies also evaluate the produced framework in a survey.

4


122

1 Introduction Testing is an activity in software development known to consume a lot of effort. Hence it is also a prime candidate for change in order to improve the effectiveness and efficiency of the software development. There are many improvement frameworks, specifically focused on testing, for example the Testability Maturity Model (TMM) [1], the Test Improvement Model (TIM) [2] and the Test Process Improvement model (TPI) [3]. All of these suffer from being too extensive for small and medium-sized enterprises (SME) where the resources needed are simply not available. In particular taking the first improvement step is a huge effort for an SME. This is especially important in the light of the fact that SMEs tend not to consider documented processes as a key asset, but rather rely on the knowledge and skills of their staff [4].

In order to meet the needs of such companies, this paper proposes a structured framework of test practices and an introduction method, taking a minimalist approach. The context is an emerging software development organisation, which is expected to grow significantly in the near future and the current practice is of an ad-hoc nature [4]. The approach focuses on following the growth of the company while minimising the resources spent introducing the practices. Another concern that is treated in the approach is the need for continuous adaptation of the practices in order for them to stay current, effective and useful. The approach also attempts to reduce the resistance to change and increase involvement of the entire organisation by frequent feedback and adaptation opportunities.

The method used to create this framework is based on observational qualitative research performed at a small and emerging software company, Scalado in Lund, Sweden. Empirical research is accepted and promoted within software engineering. It is considered that a combination of qualitative and quantitative approaches is the most beneficial as it is very difficult to isolate the observed phenomena from their contexts to achieve control over variables. For this kind of research, aiming at observing a change at a small company, flexible, deductive design studies are most suitable [5]. By starting with a blank sheet of paper, the test process needs in a small and growing software company are investigated. Required practices are documented in a minimal test practice framework (MTPF). Other considerations taken in the creation of the practice framework are avoiding negative effects of viewpoints and process distortion [6,7,8] by

123

involving the whole organisation in the test practices and their improvement.

The framework is evaluated by actual use within the same company and observations noted during the first year of use. Furthermore, the transferability of the framework is evaluated in a survey among twelve software companies based in the region. It is concluded that the framework is useful to small and emerging companies as a first step in their test process improvement.

The paper is structured as follows. In Section 2, other work on test improvement models and frameworks is presented and related to the minimal test practice framework. Section 3 presents the framework, along with the introduction method. Section 4 presents the procedures applied to derive the framework. In Section 5, the validation through use of the framework is presented, and in Section 6, the survey validation is presented. An overall validity analysis is provided in Section 7 and finally, Section 8 summarises the work and outlines further work.

2 Related Work Testing in software companies is by no means an unexplored area in software engineering research. This section presents some relevant related work to the work presented in this paper.

2.1 Test Practice Frameworks The Capability Maturity Model (CMM) [9] has been one of the most successful and influential software process improvement models to date. The areas of CMM mostly related to testing are found in the Software Product Engineering Key Process Area (KPA). Further testing practices are found in the Training Program, Technology Change Management and Process Change Management KPA:s. Common for the testing practices in the CMM is a high level of formalism, making them difficult to introduce in a small organisation. The CMM states that testing should be performed on four levels, unit, integration, system, and acceptance tests. Regression tests are required to verify changes. The test plan should be reviewed before completion and test process standards should be organised together with the software process standards. Testers require proper training and a separate system test group is responsible for performing independent testing. Test processes need to be continuously improved. Hence the CMM is on a more strategic level, and there is a


124

need for more practice-oriented support. To facilitate the transition from ad-hoc to a more mature approach, the MTPF is designed to comply with the CMM KPA:s, if the organisation decides to take a CMM improvement approach later on when this is appropriate [10].

The Testability Maturity Model (TMM) [1] is modelled after the CMM and has six Key Support Areas (KSA). These are: test friendly infrastructure, test-aware project planning, test-friendly product information, test-aware software design, testware and test environment design. The TMM however has only two improvement steps from the initial level in contrast to the CMM’s four. This makes the levels difficult to reach. The small steps of the MTPF proposed in this paper are intended to provide an easier path to follow. The Test Improvement Model (TIM) [2] incorporates the notion of levels from the CMM and TMM and it is composed of five key areas: Organisation, Planning and Tracking, Test Cases, Testware, and Reviews. These areas are very similar to the areas in the framework proposed in this paper. A key difference, however, is that the effort needed for the initial introduction of the first phase is much higher in the TIM case, compared to the MTPF low-effort approach. The Test Process Improvement model (TPI) [3] is composed of 20 Key Areas classified into levels of maturity in a test maturity matrix. To ensure correct classification of the maturity level of a Key Area a number of checkpoints are assigned to each level. In addition a number of improvement suggestions are provided for each level. The model has evolved largely from knowledge and experience gained in the field of administrative automation. The sheer size and complexity of the TPI model makes introduction into a small organisation very difficult. Just identifying appropriate parts of the model and extracting and adapting these to a small organisation would take more resources than a small organisation is willing to spend on test process improvement. The completeness of the model however implies that most of the practices identified as important in the MTPF presented in this paper are included in the TPI model.

2.2 Introduction Methods In order to effectively introduce a test practice framework there is a need for a suitable introduction method. This strategy of separating practices and introduction method can be found in CMM [9], where the IDEAL improvement process defines the introduction steps [11]. Plan Do Check

125

Act (PDCA) or Deming circles [12] is another well-known approach which aims at the same goal. TPI uses a similar approach.

An introduction method has been developed, supporting the MTPF. To ensure that the practices are suitable for use within the targeted context, the introduction method includes a pre-analysis step aiming to adapt the general knowledge from the framework to the specialised situation at the company.

2.3 Research Methodology It is well accepted in software engineering research to use empirical methods, although the practical use of empirical methods still is limited [13]. Empirical research methods comprise experiments, case studies and surveys. The former are quite well described and supported by guidelines [14, 15, 16], while the latter still rely on guidelines from social science research [5, 17], except for quantitative approaches [16, 18].

Quantitative studies or fixed design studies, e.g. experiments, are suited for use in assessing technical issues, like comparison of two object model representations or two inspection methods easily extracted from their context in the software organisation and controlled in an experimental environment. When extending the study scope to comprise a complete project or an organisation, qualitative studies or flexible design studies are better suited to capture the wide variety of aspects involved in such a situation. By their critics, qualitative studies are considered less scientific than quantitative ones [5]. The design and analysis methodology is less precise in nature, but applying a well structured and rigorously reported qualitative design methodology, the study may capture aspects of the studied situation that cannot be captured in figures.

In this study, an approach to participant observation in identifying improvement needs in a small development organisation is applied initially. Qualitative analysis is used to prioritise the needs. The outcome is documented in terms of a draft framework. Secondly, the draft framework is validated in two steps, using semi-structured interviews within the studied company. Thirdly, the framework and its transferability to other contexts is validated in a survey comprising a set of other companies.


126

3 The Minimal Test Practice Framework This section presents the Minimal Test Practice Framework in its final version, validated both by introduction of phase 1at Scalado and by the survey of local software companies.

3.1 Overview The MTPF defines the kind of practices that are needed in small and emerging software companies. It is structured in five categories, and is levelled in three phases.

The five categories correspond to areas in testing and test organisation: • Problem and experience reporting. The systematic reporting and

tracking of defects found in the product and experiences gained in the projects.

• Roles and organisation issues. The organisational structure of test functions and the roles determined within the organisation for test purposes.

• Verification and validation. Activities conducted to verify and validate the product.

• Test Planning. Planning of the tests. • Test Administration. Administration of the testing environment.

The categories are derived from the observed needs of the small

organisation studied and are validated by actual application within the organisation as well as in a survey in several different companies. The details of how the categories are derived are presented in Section 4, and their validation is presented in Sections 6 and 7.

The phases correspond to the growth of the company. The first phase includes practices that are suitable for a company with approximately 10 employees in development. The second phase includes practices that are suitable for a company with approximately 20 employees in development. The third phase includes practices that are suitable for a company with 30+ employees in development. The idea is that as the organisation grows the new practices solve the new issues created in the new, larger organisation. It is possible that a fourth phase might be appropriate for extremely large development teams but it is believed that such a step preferably involves a more rigid framework such as TPI [3]. The number of people involved in development described in the framework is intended as a guide to using the framework, not to dictate the exact point

127

of transition from one phase to another. This decision must be left to the company itself as it is based on too many variables to be defined in a framework of this kind. The framework is summarized in Table 1. The table shows the categories from left to right and the phases from bottom to top. In this section an extensive description of each area is provided.

Along with the practice framework, an introduction method has been derived, which is inspired by the IDEAL process. A significant difference is that the method proposed here enables a feedback and revision step before actual introduction of practices in order to increase the involvement of the practitioners.

The introduction method consists of five steps, prepare phase, introduce phase, revise phase, perform, and evaluate. These steps are illustrated in the flowchart shown in Figure 1. In the figure the dashed rectangle indicates the normal operating mode of the organisation. Here the practices are continuously evaluated (1) and revised if needed (2). As soon as the need for a new phase is identified (3) a phase preparation step is initiated. In this phase everything required for the introduction of the coming phase is prepared. This includes training material, database systems and other preparations for introducing the phase practices.

Pha

se 3

(

30+

)

Maintain & Evolve System Define Teams Perform

Inspections

Risk Manageme

nt

Pha

se 2

(~

20)

Create System Define Roles Perform Walkthroughs Test Cases

Coordinate Software Quality

Assurance

Pha

se 1

(

~10

)

Define Syntax Define Responsibility

Use Checklists

Basic Admin. of

Test Environ-

ment

Test Plan

Category Problem & Experience Reporting

Roles & Organisation

Verification & Validation

Test Administr

ation

Test Planning

Table 1 Overview of MTPF


128

The prepared practices are presented to the employees in the second step. This is done so that feedback is easily given from all parts of the organisation, for instance in a seminar form. Before the actual performing of the new practice employee feedback is used to revise the practices, ensuring that the practices are really tailored to the context in which they will be used.

Figure 1 Overview of Introduction Method

3.2 Test Practices In this section each test category is presented phase-by-phase and described in detail.

3.2.1 Problem and Experience Reporting

Phase 1 - Define Syntax. A common problem recording syntax is introduced in the company. This will provide developers and testers with a common language to describe and record problems in the product.

Phase 2 - Create System. A problem reporting and storage system is introduced. This system should use the common syntax created during phase 1. The syntax should have stabilised by now and be accepted throughout the company. Routines are established for gathering, documenting, storing and using experiences gathered during the course of each project. This could be done as an extension of the problem reporting system created in this phase.

Phase 3 – Maintain and Evolve System. It is ensured that the system from phase 2 can handle recording defects in the new state of the company.

3.2.2 Roles and Organisation

Phase 1 – Define Responsibility. Testing responsibility is made clear within the company. The responsibilities include: developing a test plan for each new project, administrate the test environment, administrate the problem reporting, update checklists, continuously assess the testing

n=1..3

Prepare Phase n Practices

Introduce Phase n Practices

Perform Phase 1..n Practices

Continuous Evaluation of

Practices Phase 1..n

n++

Revise Practices

Phase 1..n

(1) (2)

(3)

Normal operation

Start n=1

129

practices and monitor the need for the next phase. Phase 2 – Define Roles. A test manager is appointed and the developer

roles are assigned. Several people at the company now perform testing. Responsibilities for testers include: administrate walkthroughs, administrate test case development, support the problem reporting system and administrate experience reporting.

Phase 3 - Define Teams. An independent test team is created and separated from development. This requires that the practices from the previous phases are well introduced and accepted by the personnel. Without, for example, a well-functioning problem reporting system that is used by the personnel, a division of the test team and the development team will be inefficient. The roles within the test team are clearly defined. Some testers will, for example, concentrate on system testing whereas others will concentrate on security and performance testing. This enables the test team to perform a more thorough testing of the software. It also gives the chance for the testers to become experts in a specific type of software testing.

3.2.3 Verification and Validation

Phase 1- Use Checklists. Checklists are created for the most important tasks such as GUI testing and platform testing. Initial checklists should be written prior to a new project. If checklists already exist, they should be reviewed and updated to fit the needs of the new project.

Phase 2 – Perform Walkthroughs. Walkthroughs should be performed before the software is ready for execution, ideally in the design phase. The walkthrough team mainly consist of developers and designers. One should assign one tester to the walkthrough team, so that comments can be made from a testing viewpoint.

Phase 3 – Perform Inspections. Inspections will more often replace walkthroughs. As the organisation becomes larger, a more formal way of performing reviews is required. Inspections are more formal than walkthroughs as they require the participants to prepare in advance. A tester should be assigned to the inspection team as to the walkthrough teams. Routines and responsibilities for inspections are established.

3.2.4 Test Administration

Phase 1 – Basic Administration of Test Environment. The test environment must always be available for testing when necessary. Basic administrative activities are: Organise a testing environment for each project and make this available, update the testing environment, and


130

document the test environment and how it is used. Phase 2 – Test Cases. Test cases are derived to assure that the most

common situations and actions are tested. The development of these cases takes a lot of time and several testers will need to work with this. Test scenarios are developed that include several test cases. The test cases will be used when testing a system and will especially be helpful when performing regression testing, as the actions are determined beforehand. The advantage of using test cases is that the test team can work in parallel with the development. Instead of describing how to reproduce the failure and how the failure is made manifest, the tester only needs to describe the effect of the fault and the test case that produces the failure. Tests defined by test cases cannot, however replace all testing performed on the product. Some testing must be performed in a more ad-hoc nature, but it must still be documented.

Phase 3 – Risk Management. Risk management aims to identify problem areas at the beginning of the project and avoid or reduce the impact of issues within these areas. To be able to adapt risk management, a well-organized problem database must be in place as well as experience reporting.

3.2.5 Test Planning

Phase 1 – Test Plan. A test plan is created in order to collect all test related planning in one document. The IEEE Standard for Software Test Documentation [19] is an example of a standard that can be used as a template for this purpose. An important part of the test plan is the creation of milestones for the project.

Phase 2 and 3 – Coordinate SQA. Quality assurance routines should be established. This will ensure that software will not be released before it reaches a certain level of quality and will enable the sales department to give a measurement of the quality of the software to their customers. This will increase the trustworthiness of the products and the whole organisation.

3.3 Summary This minimal test practice framework is a lightweight approach to meet the needs of a small but emerging software development organisation. Focusing on these categories and introducing the relevant practices implies a very cost effective improvement approach utilising well proven general practices as well as the possibility to adapt these to the special local situation at the target company.

131

4 Derivation of the Framework The MTPF was originally developed in a case study in a small and emerging company in Sweden, Scalado. The test practices to be introduced at Scalado were identified using an observational empirical study [5, 17, 20]. This section contains a summary of the procedures for deriving the test practice framework, which is intended to strengthen its validity. An overview of the work performed in order to identify the test practices is provided in Figure 2.

The first step was to observe the current situation and provide background information. The result of this step was a draft set of practices (version 0), identified through observations and interviews. The second step was to introduce the draft to the developers and other employees that were involved in the future use of the practices, providing feedback on the framework. The test practices were then revised according to the feedback produced and were further developed and prepared for actual introduction in the company (version 1). Experiences from three months of application were fed back and integrated in the framework (version 2). Then a survey of 12 software companies validated the framework, which again was adjusted based on the validation feedback (version 3), which is the version presented in Section 3.

Figure 2 Overview of Framework Construction

Observations

Application

Feedback

MTPF v.1

MTPF v.0

Survey validation

MTPF v.2

MTPF v.3


132

The Initial observations and Feedback steps are presented below, while the Application step is presented in Section 5 and the Survey validation in Section 6.

4.1 Company Background Scalado was founded in the year 2000 and has around 25 employees today. Scalado develops tools for creation, presentation and maintenance of image-based digital information on the Internet, mobile Internet and hand-held devices. The company has grown rapidly and is expected to continue to expand in the future, although due to the economic recession during the last couple of years, Scalado has been forced to cut down some personnel.

Testing has previously been performed largely ad hoc but as the company is in an expanding phase, the need for a more thorough test method has increased lately. Scalado has not yet come to the point where projects are running over budget by far more than expected, but management and all project members are certain that this situation will arise if testing practices are not structured.

Scalado has on occasion attempted partial adaptation of fragments of the Extreme Programming (XP) [20, 21] development process such as pair programming. The development can despite this be most appropriately described as completely ad hoc. Testing is not adopted according to XP at all. Instead a separate test person is performing system test, as the customer is not available at site.

Integration testing is performed by using CPP Unit [22], a test-framework that integrates units to an existing system. The tester performs system testing by using the application and reports problems found. A test environment is built with virtual machines for easier administration of test. Acceptance testing is performed shortly before release.

4.2 Research Aims In order to more systematically record the current situation, an observational study was launched. The research aims of this study are:

1. Identify a set of testing practices appropriate for introduction in an emerging, rapidly expanding, development organisation.

133

2. To determine how these practices be organised into a series of phases where each phase follows naturally from the last and is appropriate for a larger number of developers.

The aims are investigated using qualitative, observational research techniques, both participatory and non-participatory, and interviews [5, 14, 17, ] at Scalado.

4.3 Data Collection The data collection in this study is performed using different means:

1. Observation of testing 2. Open-ended interviews 3. Observation at design meetings 4. Participant observation 5. Semi-structured feedback interviews

The units of analysis in the study are defined as the following three

units: The current development process, the current method of testing and improvement suggestions to the existing test method. These units of analysis are summarized in Table 2.

The goal of the first data collection occasion was to observe how testing was conducted before any change was proposed. A tester performed testing on a new patch during the observations. After the observation, the tester was asked questions about certain actions during the testing. Based in this initial observation a set of questions were prepared for the second data collection occasion, open-ended interviews.

The open-ended interviews consisted of questions to all six members of one project in the same area: how new systems are designed initially and how testing is performed today. The questions were slightly differently formulated depending on the role of the interviewee; developers, testers and a project manager. The open-ended interviews generated a clearer picture of the software development process; especially how development interfaced with testing.

The third data collection occasion consisted of attending design meetings and mostly verified and clarified the results of the open-ended interviews.


134

Unit of analysis

Subjects Timing Scope

Development process

Developers, project manager

Beginning of the data collection

Collect data to define an overview of the development process. Define the interfaces between development and testing. Seek parameters in the development that can affect software testing negatively or positively. Understand the type of product being developed.

The existing test method

Testers, project manager, developers

Beginning, parallel to defining the overview of the development process

Collect data to define the overview of the existing test method. Scale-up the overview and collect data to define practices for framework.

Suggestions about change to the existing test method

Testers, project manager and developers

Entering criteria: Development process is defined, existing test method is defined and enough theoretical knowledge to suggest recognized solutions to common testing problems.

Collect data about positive and negative areas in the existing test method. Present the found bottlenecks and collect comments. Present solutions to the bottlenecks and collect comments. Space for the staffs’ own solutions to negative areas in the existing test method. Collect attitudes about process change. Collect data about how critical testing is for the product quality.

Table 2 Units of Analysis

The fourth data collection occasion consisted of participatory observation while testing a new patch. Observations concerning practical difficulty in software testing were collected. Also, ideas from developers concerning improvements of the existing test method were gathered.

The fifth data collection occasion was intended to provide feedback on the results from the analysis of the data collected on the previous occasions. The collection and its results are therefore described in Section 4.6.

4.4 Analysis of Observations and Interviews The analysis discovered some bottlenecks in the existing test approach. The open-ended interviews and the attendance at the design meetings revealed that testing was not planned for in advance. This resulted in deadlines overrunning and project cost increases. Also, testing was assigned very little time and resources and was performed too late in the project.

135

The participant observation, while testing a patch, revealed that the test environment was badly organized which lead to inefficiency as the test environment had to be reorganized before the testing could start. The participant observation generated ideas about attributes needed for the problem reporting syntax such as including the machine message as an item.

Most of the problems are of a managerial nature rather than technical. The improvement proposals from the interviewees on important areas for successful testing are:

• Using checklists • Developing a test plan • Using a problem reporting system or problem reporting syntax

Furthermore, the interviewees thought that a time plan with milestones

would be the most important part of a test plan for effective testing. When it comes to checklists, they thought that it is most important to develop a checklist prior to walkthroughs. However, when they saw examples of checklists, they thought that the GUI checklist would be most useful. The questions about the test environment identified the need for someone to take the responsibility for the test environment and for the test environment to be realistic. However, the interviewees also pointed out that it is important that the test environment is available. Other issues that distinguish a good test environment are that:

• It is specified how the test environment is built • The test environment is fast • Enough time is assigned to test if the test environment is slow

The existing problem reporting syntax was inefficient and

unstructured according to the interviewees. They agreed that an interactive problem reporting system is the best way of administrating the problem reporting. Some form of improved common syntax provides the second best alternative and the worst alternative is the existing problem reporting syntax. Important items of a problem reporting syntax are according to the interviewees:

• Instructions on how to reproduce the problem • Priority to problems


136

• Search function among the problem reports • A minimum of fields to fill in – simplicity • A field to assign responsibility for each problem

All interviewees agreed that it is beneficial to have a common coding

standard. An informal coding standard did exist but everyone did not use it. There are different opinions about the importance of walkthroughs and their efficiency. However, most interviewees thought walkthroughs are more efficient for the purpose of learning from others experience than for finding faults in the product.

4.5 Test Practices Structure Based upon the analysis of the observations the most important areas identified to introduce as early as possible are:

• A problem reporting system • Checklists • A test plan • Defined responsibility for test

It is also concluded that walkthroughs should not be introduced until

the organization becomes larger and that a coding standard should be introduced but is not top priority for effective testing.

This ended up in a draft of the MTPF composed of five areas, as defined in Section 3:

• Problem and Experience Reporting • Roles and Organisation • Verification and Validation • Test Administration • Test Planning

4.6 Feedback The proposed practice framework was shown to the employees at Scalado during an open interview and their comments were recorded.

The interviewees generally thought that the draft of the test method would meet the needs at Scalado. The problem reporting system, checklists, a test plan and responsibility for test are considered to be

137

important parameters for effective testing. Walkthroughs, the test environment and a code standard are less important parameters.

The MTPF was reworked regarding wordings and other minor issues based on the feedback.

5 Application of the Framework The next step in the development of the MTPF was applying it at Scalado. Observations were mainly done in the first three-month period and after one year of operation. As the company comprised about 10 developers, the first phase of the framework was mostly in focus.

The first month of application was dedicated solely to the introduction of each practice of the test method. The next two months, the introduced practices were used and evaluated by performing software testing in three different projects. Each project is of a different type as described below:

• The first project involved testing a patch of an existing product. • The second project involved the reengineering of a sub-system of

an existing product. The project started and ended during a period of three months.

• The third project involved testing a totally new product

In this section the lessons learned from the application of phase 1within each test practice category are summarised. The general structure of the framework is considered sufficient, while the application experience adds to the detailed level of the framework. It is important to remember that the framework would be applied differently in a different case. This level is to be tailored for each organisation, but the examples are reported for illustration purposes.

5.1 Problem Reporting The application identifies Define Syntax as an important feature of the problem reporting. The problem reporting syntax had to fulfil the following requirements:

• Include a set of fields that together describe a problem in any project

• It must follow the workflow of the tester • It must be easy to understand for the developer and give

motivation to correct the problem


138

• A problem report must have a unique number • The syntax must allow grouping of problems

A set of fields was developed that fulfilled the above requirements and suited the development at Scalado.

It is highly important that the syntax is defined to allow flexibility. If the syntax is defined using an advanced database system (i.e. entering phase 2 directly) it will probably be hard to change the syntax even though it has shown to be ineffective. To avoid this scenario, the strategy at Scalado was to define a syntax using Microsoft Excel as reporting tool and then evaluate the result after using it for a while. The syntax probably needs to go through another (and a larger) project before it can be thoroughly evaluated. It is, however, already observed that some adjustments must be made before creating a problem reporting system.

Another level of defining the problem reporting syntax is of a linguistic character. Experiences at Scalado announce that it is important to discuss some basic linguistic definitions prior to testing a project. The long-term solution on the linguistic difficulty is probably to let the syntax develop as a consequence of striving towards constancy when writing problem reports.

5.2 Roles and Organisation The responsibility for software testing was assigned to one person to avoid the problems experienced earlier, when testing was performed by the support manager, and no test team existed.

During the period of introduction, the test manager did the software test planning and performed software testing in the three projects mentioned above. The introduction became flexible since the person who experienced the practical difficulties with software testing was also the one who introduced the practices.

The result of defining the responsibility of software testing has affected the structure and organisation of testing at Scalado in a positive way. Not only is testing more structured but the marketing department can also more accurately plan the release date for each product.

The development department has also gained from the defined responsibilities as they are working with a person who is specialised in software testing. The developer does not need to think as much about the overall quality of the developed software at system level and can instead concentrate on the details when correcting faults at unit level. The testers’ role is to find faults indicated by product failures, distribute problem

139

reports of these to the developers, perform regression testing and communicate the status of the project to the project manager. Currently hired consultants perform the tests. It is the aim of the company to have a permanent test responsible team that can be complemented by consultants when needed.

5.3 Verification and Validation The MTPF introduces checklists as a verification and validation practice in phase 1.

Prior to the platform testing, checklists describing the different configurations to be tested upon, were developed. This type of checklist was very useful as it can easily be transformed into a status report telling which platforms that are supported and which are not.

In the second and third project the sections Features to be tested and Pass/Fail criteria of the test plan came to be the basis of a functionality checklist, see Section 5.5. This type of checklist enables the tester to get an overall picture of the functions tested. In the second project this concept was further developed. By developing a test case description, which included the features to be tested and the pass criteria for each feature, regression testing could be performed faster.

A GUI checklist was developed but there has not been any need for this sort of checklist in the three projects.

5.4 Test Administration The existing test environment at Scalado was very fragile. In some situations it was hard to decide if the test environment or the software under test caused a failure. The test manager and the system administrator started to reorganize the test environment by reinstalling its components and naming well defined configurations. After installing a configuration it was tested by using it for a while to confirm that it represented a stable test environment. This generated a more reliable test environment.

The following lessons were learnt:

• The test environment becomes a bottleneck since parallel testing cannot be performed.

• After setting up a test environment, the test machine itself must be tested.


140

This practice was, surprisingly, one of the hardest to introduce. Test

environments appear to be taken for granted by developers and scarce testing hardware resources are often taken over by development, considered more critical. Alternative test environments are currently being investigated at Scalado where the test environment consists of three computers, two test computers and one administrator computer.

5.5 Test Planning The Test Plan is the first practice of test planning. A test plan has been written prior to the testing in each project using parts of the IEEE Software Test Plan [19] as a template. The sections included in the test plan are somewhat different between the three projects but the main sections are:

1. Features to be tested 2. Pass/Fail Criteria 3. Schedule 4. Responsibilities

An important experience from planning the three projects is that the

sequence of how testing is performed may determine the final software quality. By performing systematic functionality testing on a stable system before starting the platform testing, the time spent in relation to the given software quality will increase. As the most troublesome functional faults are corrected before platform testing is performed, the number of failures that may occur in the platform testing decreases and therefore the fault causing the failure on a specific platform can be more thoroughly investigated.

The second project differed from the others, as it was the only project that started and ended during the three months period. This enabled the test manager to start the planning of the project well before the testing phase had begun. The flexibility of the small organization made it possible to change the original project plan and start the test phase approximately two weeks before it was originally planned to begin. By starting the testing while the last functionality was being implemented, higher software quality was gained than in other projects. The conclusion of this experience is that to be able to plan successful testing, the test manager must dare to make use of the flexibility that exists to a higher

141

degree in small organisations. The definition of the responsibility also resulted in testing not being

overlooked when calculating the total time of the project. Testing was, however, still dependent on the development being completed in time. If development overruns its deadline, the time available for testing will decrease, as small organisations are often completely dependent on one product with short time to market.

5.6 Phase 2 Currently only a web-based defect reporting system has been introduced from phase two. Other practices from phase two are intended to be introduced once the organisation starts to grow again.

5.7 Quantitative Indicators of Application Quantitative measures were taken from two similar projects at Scalado, one before the application of the framework (Project A) and the other after the application of phase 1 of the framework (Project B). It is of course impossible to find identical projects as these are always of differing size and nature, but the two that have been used are as similar as is possible. The projects are releases of image zooming software for use in web applications, both developed with around five developers. The data has been extracted from fault reports and time reporting from the projects and is presented in Table 3.

Project A Project B % of time spent testing 5% 20% Number of documented faults 20 400 Number of documented corrected faults 0 300

Table 3 Quantitative indicators of the application of the framework

Before the introduction of the framework none of the projects were writing or using test cases.

After the introduction projects with more than 6 weeks duration wrote and used test cases. Before the introduction of the framework no projects used test planning. After the introduction projects with more than 6 weeks duration always use test planning. Projects under 6 weeks duration still use ad-hoc testing.

Project A’s indicators are the result of the code and fix style of ad-hoc


142

working prior to the framework. The product is in development almost until it is released, there is a low number of documented faults in total, most faults are not documented at all. The indicators for Project B demonstrate a clear increase in the time spent testing the product. The increase in the number of documented faults is due to the fact that faults are now actually being documented in a structured fashion according to the framework as well the larger portion of time is spent in testing. The product quality is perceived higher in Project B due to the more effective fault removal of the framework, i.e. the larger number of faults does not indicate a lower quality product in this case.

5.8 Summary of Application Based on the application, it can be concluded that the first phase of the MTPF is feasible and meets acceptance among practitioners. The lightweight approach is appealing to them and the focus on involvement in the introduction method creates acceptance as it begins with the needs of the organization. The quantitative measures used indicate that the framework has significantly changed the way of working at the company.

6 Survey Validation of the Framework To investigate the transferability of the MTPF, it was validated using a survey with representatives from twelve regional software engineering companies. Size wise, these span from one-man consulting firms to 100+ developer development units. These obviously have different expectations of a testing framework.

The economic situation at the time of the survey, summer 2003, deserves to be mentioned. The recession has affected all of the companies and, as was seen in the survey, none are growing rapidly. It would have been very interesting to perform the validation in an economic climate similar to that of the late nineties, with many companies growing very rapidly.

6.1 Method The survey methodology [5] was chosen in order to be able to quickly reach many subject companies, as the transferability of the MTPF was considered the most important issue. Representatives of companies

143

involved in a regional software process improvement network, SPIN-Syd2, were used as subjects. The framework was presented to the representatives first in an oral presentation and then in the form of a written summary. After the initial presentation the survey forms were handed out and the subjects were given as much time as they needed to complete it. During the survey the subjects had access to the written summary and were also free to ask questions.

The first section of the survey gathered general information about the company. The information requested was the number of developers in the company, the age of the company, the current rate of change in the company size and the relationship the company had to testing methodologies. The relationship was determined by classifying the companies into three groups: (i) Companies that have gotten further than the concepts involved in

the framework. Here the primary interest is whether the framework would have been useful in attaining their current level.

(ii) Companies that are currently using a similar method to the current framework. Here the primary interest is how well the methodologies match.

(iii) Companies that have not focused very much on testing. Here the primary interest is if the framework would be a suitable tool to begin with.

All of the companies involved in the study can therefore contribute

with useful information regarding the MTPF from their unique perspective.

In the second part of the survey the subjects were asked to look at the framework stage by stage and comment on each area with respect to the situation at their company. Thirdly the subjects were asked explicitly if they thought anything useful to their company was missing in the framework. The final survey sections regarded the subjects’ perception of how the framework introduction method would perform in their respective company. An overview of the survey is shown in Table 4.

2 www.spin-syd.org


144

Survey section Content 1 Subject classification, Size, age, growth rate and relation to test

frameworks 2 Framework evaluation stage by stage 3 Completion of the framework 4 Perceived introduction method performance in the subject’s own

organization

Table 4 Overview of Survey Instrument

6.2 Survey Results The subjects largely filled out the survey form satisfactorily, although three did not complete section 2. All of these were however positive concerning the appropriateness of the survey to their specific company.

The results of the first section are summarised in Table 5. Size denotes the number of software developers in the company. Age denotes the number of years since the company’s foundation. Growth denotes the rate of change of the company’s size, on a scale of - - strong reduction, - reduction, sq status quo, + growing, + + strongly growing. Relation denotes the company’s relationship to working with testing, 0 not worked much with this, = are currently using a similar approach, + are more advanced than the issues discussed here.

Here it can be seen that the range of companies spans from 1 to 100+ in number of developers and from start-ups to relatively mature companies, for software development that is. Most of the companies are currently not growing. The four that are growing seem evenly distributed across both size and age. A trend can be seen in the relation to working with testing. Larger companies appear to have come further than medium and smaller companies.

ID# 1 2 6 7 12 11 3 8 5 4 10 9

Size 1 1-2 3 1 2 7 10 15 20 70 95 100+

Age 0.9 1 2 1 0 2+ 15 >10 15 old 40 ~20

Growth sq sq + sq sq + sq sq + sq + sq

Relation 0 = 0 + = 0 0 0 + + = +

Table 5 Classification Results

Smaller companies explicitly verified that simple defect tracking in a file is adequate at this stage. Also clear from most of the subjects is that responsibility benefits from being defined clearly as early as possible.

Generally it can be seen that the larger the company the more of the

145

parts of the framework are in place, in the staged order prescribed. Many companies mentioned automatic testing as a completion of the

framework. This is probably due to the large amount of attention given lately to eXtreme Programming (XP) [21] including its view on testing. This could, if found useful by the organisation, easily be integrated into the first stage in the MTPF under test administration. Several companies also mentioned requirements engineering and the connection to testing and that this should be integrated in some way into the framework. The point is very relevant and should not be disregarded by any company doing software development. Including requirements focus however, would make the MTPF more a complete software process description, which it is not intended to be.

All subjects except one responded positively to whether the introduction method might perform well in their company. This exception expressed a need to vary the ambition level of verification and validation from project to project depending on the customer and product being developed.

One subject mentioned patterns as a good method for experience gathering and setting syntax early. Another subject suggested emptying the framework prior to the introduction of each phase in order to focus on that which is most relevant to the current situation. A few subjects responded in a discussion after the survey that the logic of what testing activity was placed in each area was a little different to the thinking at their company. As a result of this, minor revisions were made in the positioning of activities in the MTPF.

One of the largest companies reported being stuck with an outdated problem-reporting syntax in a reporting system. This demonstrates the importance of working out a good syntax prior to locking it in a system, but it is of course impossible to be completely insured against future needs for syntax changes.

6.3 Survey Conclusions In general the MTPF appears suitable, or would have been suitable earlier, to most of the represented companies. As a result only minor structural revisions were performed after the validation. Suggestions for improvements were noted but are apparently not relevant to more than a single company in the group. The flexibility of the introduction method would allow companies with special needs to accommodate these.

A general observation noted during the survey is that one of the most


146

important needs for the companies seems to be a need for structure in the improvement work rather than extremely detailed specific practices. This supports the flexible framework approach used.

7 Validity of the Studies This paper presents a minimal test practice framework intended for small and emerging software development organisations. An important and relevant question to raise is how valid the MTPF is, firstly, regarding the validity of the results for Scalado, and secondly for transferring the results to other companies in similar situations. The first issue is addressed by reporting measures taken in the MTPF derivation, and the second issue is addressed by a survey among potential users of the framework. The validity issues regarding the derivation as such are referred to as credibility, confirmability and reliability, while the issues on transferring the results are referred to as transferability [5].

The MTPF was derived during a period of three months, and then internally validated during another three months. This persistent observation reduces the threat that the characteristics of the organisation are not known to the researchers. The many sources of information in this study allow a high degree of triangulation, i.e. having multiple sources for the information, not least the peer debriefing which provides feedback opportunity increases both the confirmability and credibility in the study. Reliability was addressed by rigorously researching the theory behind observational research and interviews [5, 14, 17] and careful preparation before each data collection round.

Transferability is a central point in the creation of the framework. If there is no transferability the created framework is only applicable for the presented situation at Scalado. In order to investigate the transferability, a survey was launched with twelve local software development companies. This survey is presented in section 6, while the validity of the survey as such is analysed below, and reported in Table 6. Given these threats and the measures taken to avoid them it can be concluded that validity of the survey is adequate. Further, the results of the survey indicate that the transferability of the MTPF is good.

147

Threat Countermeasures Subjects misunderstanding the framework and introduction method.

Publication of an early version of the framework to increase clarity, presentation of framework and introduction method before survey in both oral and written forms, expert support during the survey completion.

Subjects misunderstanding the survey questions and their intent.

Review of survey instrument prior to use, expert support during the survey completion.

Subjects do not care enough to give the right answers.

Promise feedback of results from survey to aid the company’s process improvement work.

The subjects represent the most relevant people to answer the survey.

The subjects were asked to recommend more suitable people to pass on the survey to if they were not the most suitable. None did.

The subjects are unwilling to give information that makes their organisation or themselves look bad, or not advanced.

None, but the SPIN-Syd forum for companies is very open and sincere so this should not affect significantly.

The researchers misinterpret the intent of the subjects’ survey answers.

Feedback both of the results to the subjects and of the subjects’ comments on the results to the researchers.

Table 6 Survey validity measures

8 Summary and Future Work In this paper a minimal test practice framework and a corresponding introduction method suitable for introduction in a small, rapidly expanding development company have been created. The first phase of the MTPF and the introduction method have been evaluated by introducing them in the subject company. A significant contribution is the low threshold for introducing each phase compared to other test processes and practice frameworks. The framework is derived and validated using rigorous and well-established research methodologies, which secures the validity of the framework.

The natural continuation of the work presented in this paper is to continue introducing the MTPF at Scalado and to continue to observe the results. A second natural continuation of the work is to try to introduce the framework in other different organisations in a similar situation. It will be interesting to see how the framework is adapted in order to suit new circumstances.

9 Acknowledgements This work was partly funded by The Swedish Agency for Innovation Systems (VINNOVA), under a grant for the Center for Applied Software Research at Lund University (LUCAS). The authors would like to thank Scalado for their extensive cooperation and support for the project and SPIN-Syd for providing a stimulating forum for software engineering


148

practice and research in the south of Sweden and cooperation with the survey.

10 References [1] Gelperin, D., “A Testability Maturity Model”, Fifth International Conference

on Software Testing, Analysis and Review, Orlando, Florida, 1996. [2] Ericson, T., Subotic, A., Ursing, S., “TIM- A Test Improvement Model”,

Software Testing, Verification and Reliability, 7: 229-246, 1997. [3] Koomen, T., Pol, M., Test Process Improvement – A practical step-by-step guide

to structured testing, Addison-Wesley, 1999. [4] Runeson, P., Andersson, C., Höst, M., “Test Processes in Software Product

Evolution – A Qualitative Survey on the State of Practice”, Journal of Software Maintenance and Evolution, 15(1):41-59, 2003.

[5] Robson,C., Real World Research, Blackwell Publishers, Oxford, 1993. [6] Sommerville, I., ”Managing Process Inconsistencies Using Viewpoints”, IEEE

Transactions on Software Engineering, 25( 6):784-799, 1999. [7] Karlström, D., Runeson, P., Wohlin, C., “Aggregating Viewpoints for Strategic

Software Process Improvement”, IEE Proceedings - Software Engineering, 149(5):143-152, 2002.

[8] Bandinelli, S., Fuggetta, A., Lavazza, L., Loi, M., Picco, G. P., “Modelling

and Improving an Industrial Software Process”, IEEE Transactions on software Engineering, 21(5): 440-454, 1995.

[9] Paulk, M., C., Curtis, B., Chrissis, M., B., and Weber, C., V., Capability

Maturity Model for Software, Version 1.1, Software Engineering Institute, CMU/SEI-93-TR-24, February 1993.

[10] Paulk, M., C., Weber, C., V., Curtis, B., The Capability Maturity Model –

Guidelines for Improving the Software Process, Addison Wesley, 1995. [11] Deming, W., E., Out of the Crisis, Cambridge University Press, 1986. [12] Ramesh, V, Glass, R. L., and Vessey, I. , “Research in computer science: an

empirical study”, Journal of Systems and Software, 70(1-2): 165-176, 2004. [13] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., Wesslén, A.,


149

[14] Juristo, N, and Moreno, A., Basics of Software Engineering Experimentation,

Kluwer Academic Publishers, 2001. [15] Kitchenham, B.A.; Pfleeger, S.L.; Pickard, L.M.; Jones, P.W.; Hoaglin, D.C.;

El Emam, K.; Rosenberg, J., “Preliminary guidelines for empirical research in software engineering”, IEEE Transactions on Software Engineering, 28(8)721-734, 2002.

[16] Yin, R. K., Case Study Research Design and Methods, Sage Publications, Beverly

Hills, California, 1994. [17] Kitchenham, B.; Pickard, L.; Pfleeger, S.L., “Case studies for method and tool

evaluation”, IEEE Software, 12(4):52-62, 1995. [18] IEEE Std 829-1998 (1998). IEEE Standard for Software Test Documentation.

Software Engineering Technical Committee of the IEEE Computer Society. [19] Karlström, D., “Introducing Extreme Programming - An Experience Report”,

Proceedings Third International Conference on Extreme Programming and Agile Processes in Software Engineering, Sardinia, Italy, 2002.

[20] Beck, K., Extreme Programming Explained: Embrace Change, Addison Wesley,

1999. [21] Jeffries, R., Anderson, A., Hendrickson, C., Extreme Programming Installed,

Addison Wesley, 2001.


150

151

Agile Methods and Software Product Management – An Integrative and Hybrid Approach

Runeson, P., Karlström, D., “Agile Methods and Software Product Management – An Integrative and Hybrid Approach”, Technical Report, Lund Institute of Technology, CODEN : LUTEDX (TETS-7203) / 1-34 / (2004) & local 16, 2004.

Abstract Agile methods have evolved as a down-to-earth approach to software development, which use has gained positive attention. However, in many cases software is only one part of the development project, such as for example embedded products, agile methods must therefore coexist with project management models, typically of the stage-gate type. Further, agile methods in their original form are intended only for small-scale development teams, and there is a need for scaling this up. We have observed projects using an agile approach for development that is hidden from management, but we propose to turn this into an approach that utilizes the strengths of the agile methods to improve the manageability of the whole project. This paper presents a hybrid model, integrating the advantages of agile methods into the context of the stage-gate project management models currently used by these companies. We define a process model and how it interfaces towards agile methods. In an example of an important management decision point we illustrate the information available using the agile approach, and how this can be utilized to improve the decision made at this point.

5


152

1 Introduction Agile methods have evolved as a new approach to developing software products [1]. The methods have been successfully implemented in small to medium-sized projects, and the principles of agile methods have inspired software developers in wide-ranging application domains. The agile methodologies offer down-to-earth approaches to software development that focus on simplifying the software development process, focusing on the developers, close contact with the customers, and the final product. Agile methods are primarily used for small-scale development.

In large-scale industrial product development, however, software development is not an isolated activity. It usually exists as sub-projects in an environment composed of hardware development, marketing, production planning, product rollout, etc. which all must be managed and coordinated concurrently. Furthermore, software development seldom starts from scratch; legacy software constitutes the core of new products to which new functionality and other types of customer values are added. Another common situation is migrating legacy software to new platforms.

This situation is particularly true for software products of the embedded type, i.e. where the software is an integrated part of a product e.g. a cell phone. The scale-up issue is relevant for packaged software as well, i.e. software acquired by the customer/user and installed on a general PC or PDA platform. In the case of embedded software, the software development sub-project has to be coordinated and integrated with other kinds of development sub-projects, e.g. electronics and mechanics. In the case of market-driven development, both kinds of development have to be integrated with marketing activities, such as advertising campaigns. Independently of product type, the sheer size of the software requires a division of the software development into several sub-projects, putting more emphasis on sub-project co-ordination. A key issue to manage in software development is the balance between 1) requirements on functionality, 2) the quality of the product, and 3) the related development cost and schedule. The three dimensions are often contradictory and management involves a trade-off between the three.

The traditional approach when managing a complex development project, is to use a project management model (PM) such as Cooper’s Stage-gate model [2]. The PM gives support not only for the

153

communication within the project, but also for decision makers sponsoring or acquiring the project. The concepts of a PM are not connected to any specific domain; these models are sufficiently generic to be applied to any kind of project. However, PM:s tend to have a basic sequential structure, i.e. distinct phases of prestudy, feasibility study, execution and conclusion. This is similar to software development methods of the waterfall type [4] that are currently considered a thing of the past in the software engineering community.

Traditional PM models have already been integrated with more iterative software development models in general [3] and the Rational Unified Process (RUP) in particular [4], and also integration with agile methods has been presented [5], although only on an overview level so far. We have observed software projects using agile methods executed in traditional PM environments, but the main characteristics of the agile methods are not designed with high level product management or management of integrated projects, and their advantages are therefore not utilized in the management of the project. The agile development has in these cases had no effect on the project management and overall management of the project despite demonstrating success on the development level.

Traditional PM are compatible with maturity-oriented software process improvements, using for instance the CMM [6] or CMMI frameworks [7]. Both PM and the maturity models aim at defining structures, roles and responsibilities within the organization, but do not intend to actually dictate how the actual technical project work is to be performed. The agile methodologies can be considered a reaction to these maturity-oriented approaches within the software development process field, but we do not believe that the two schools are contradictory [8]. Maturity-oriented approaches seem to be preferred by quality departments and management, while technical staff tend to be in favour of agile methods. To fit in a mature organizational environment, software projects using agile methods must be able to coexist with traditional PM:s. Mature organizations have their role in adding not only rigor and order to a chaotic environment, but also demonstrate certain characteristics similar to a successful agile development project [9, 10] [11]. There is currently room for research on effective integration between PM and agile methods with the aim of not only integrating the models, but optimizing them to interact optimally and identifying irreconcilable differences.


154

In this paper we outline a set of issues for the integration of agile methods into a PM, based on a generic PM model and pinpoint challenges for these different cultures to meet and cooperate in an integrated or hybrid model. The approach is similar that of Wallin et al. [5], although this paper is specific to agile methods and presents more detailed description of interfaces and communication means used to integrate PM and agile methods. Further, a specific example is provided using one of the most widespread agile methods, Extreme Programming (XP) [12].

The rest of the paper is structured as follows. In Section 2 background information is provided for the areas of software development processes, project management models and agile methods are presented. Section 3 presents the proposed hybrid approach, and Section 4 illustrates the approach, using the go-ahead management decision point of a project based on a context from a real industrial management model, as an example. In Section 5 we elaborate the challenges for further work and research on this hybrid approach.

2 Background In order to present an integration of the PM and a software development process based on agile methods, we elaborate on the constituents of and factors affecting the model in the following subsections. We determine that the PM is oriented towards general project management and there must exist a more software specific technical development process according to which the actual software development is performed (Section 2.1). We elaborate on the concept of PM (Section 2.2) and software development processes, both traditional (Section 2.3) and agile ( Section 2.4). We also elaborate on factors affecting the PM and software development such as the product stakeholder situation, product characteristics and organizational factors such as structure, competence and experience (Section 2.5).

2.1 Software Development Process As software development processes have developed, the terminology has been defined successively. Sommerville defines the software process as: “The software process is the set of activities and associated results which produce a software product.” [13] We can model this basic concept by

155

using an elementary process model adapted to software development giving us a basic software process.

The evolution of software development processes can be divided into three distinct periods chronologically and we are currently moving into a fourth period [14, 15].

The first period can be referred to as ‘code and fix’ [16]. Much programming was performed using low-level programming languages and program size was small. Nevertheless, some of the problems observed today were already apparent [17].

The second period can be referred to as the structured methods period [14]. This period started after the software crisis in the late 1960's when new hardware advances enabled software systems to be developed that were too large for the traditional development methods used at the time [18]. It was at the beginning of this phase that the term Software Engineering was coined, specifically at the 1968 Nato conference in Garmisch Partenkirchen devoted to the software crisis. A major part of this period was devoted to graphically describe the development process. Examples thereof are waterfall and spiral processes [19, 20].

The third period, process maturity, introduced a meta level to the development processes. During this period, software process improvement (SPI) processes and models were developed and implemented [14]. The most well-known example of work from this period is the Capability Maturity Model (CMM) [7, 9]. Most of the third period models are more appropriate for larger companies though the research community has made several attempts at adapting them to smaller organization types [21].

The fourth period was predicted to be the industrialized software engineering phase during the third period [22]. The engineering methods used would resemble the engineering methods present in other more established engineering disciplines such as civil engineering or manufacturing. This fourth period, however, is currently only beginning to take form. There has been a distinct change in the software process community with the introduction of “agile methodologies” [1] that are in contrast to the continued evolution of the maturity models.

The fourth period has started with a division of the software engineering community. The successful practitioners of the third period models have continued their improvement into high maturity practices and the frameworks have evolved based on experiences gained. The companies that have failed in their intent to introduce third period processes or never even attempted this, have taken agile methods as the


156

next step. In this latter group, there is a strong reaction against the formalism and size of the third period processes.

We believe that there is a potential in combining the best of the third and fourth periods. By combining the engineering empowerment in the agile movement and the business perspective in the PM, we foresee a strong combination for the future.

2.2 Project Management Models A project management model (PM) defines a framework for setting up and managing a project [2]. The framework as such is general, applicable to any kind of project. The PM focuses on monitoring, management and decision issues on a business level, rather than the specific technical work, and typically defines:

• a set of phases, • criteria for decisions related to the phases, • stakeholder roles and • project management artefacts.

In Figure 1a, the principal constituents of a generic PM are outlined.

In this model, there are four phases; prestudy, feasibility study, execution and completion phases. This is the terminology used in the PROPS model by Ericsson [23] but most PM models used have phases with similar intent. Project progress is measured through the passing of predefined milestones (MS) and at the end of each phase a tollgate (TG) must be passed, which involves a decision whether the project is to continue and be assigned further funding, or if it is to be terminated.

Stakeholder roles are defined within the PM, such as project manager, project sponsor, quality assurance manager, and for the internal project organization, subproject managers. The customer stakeholder is typically only involved at the start of the project and after delivery. Project documentation standards are defined in the PM, such as the content and format of project specifications, project plans, quality plans, progress reports, etc.

157

Figure 1. a) Project Management Model b) Technical Software Development Process.

In order to control the actual software development in the project, the organization integrates the management model with a software development process, see Figure 1b. This process is much more technically oriented and fits into the specific application domain for the project. There may be several sub-processes for different logical parts of the software and even sub-processes for hardware development that must be coordinated. The general phases and decision criteria are defined in terms of a technical content and technical documentation is added and linked to the different phases.

The project management model can thus be seen as a layered model, as depicted in Figure 1. The technical development process is still on a rather high level of abstraction even though it describes work of technical nature.

An incremental approach can be defined, where development products are split into increments, delivered to other stakeholders according to a defined increment plan [3]. Milestones are defined for each of these incremental deliveries similar to those shown in Figure 1b, and these are reported and monitored on the management level using the same reporting mechanism.

TG1 TG2 TG3 TG4

Prestudy Feas-ability

Execution Comple-tion

R1 R2

TG1 TG2 TG3 TG4

Proto-type

Req LL Design Integration

MS1 MS2 MS3 MS4 MS6 MS5

HLD

Testing

P2 R3 FR P1 R4 R5 R6 R7 R8b)

a)


158

2.3 Management of the Technical Software Development Process

The project management model and the development process model, including their documentation, are intended to support the decisions taken regarding the execution of the project as well as technical decisions regarding the product under development. This enables upper management to achieve an overall picture of the running projects and their status.

Managing the work involves synchronization with other activities, in particular for projects that involve more than software development. An example is software embedded in a cell phone product that also involves a large portion of hardware development. On the project level, synchronization regards resources within the organization, as well as potential target market activities and trends. On the technical level, the issues for synchronization regard technical issues, synchronizing dependencies with other technical sub-projects, for example hardware development or production planning.

Management decisions within a project management model are of two types: 1) discrete, planned decisions; and 2) continuous day-by-day decisions. The former type of decisions are officially defined, well prepared, documented and communicated, represented by tollgates in Figure 1. The continuous kind of decision is taken more spontaneously, based on the formal or informal roles of the people involved in the project. In practice, the informal decisions are often the basis leading up to the formal ones, hence these constitute a very important part of the actual formal decision.

2.4 Agile Methods Agile methods have arisen as a reaction to the use of strict software engineering processes. The development of these occurred in parallel towards the end of the 90’s. The most widespread of these methodologies are: Extreme Programming (XP) [12, 11], Cockburn's Crystal Family [1], Highsmith's Adaptive Software Development [24], Scrum [25], Coad's Feature Driven Development [26], and Dynamic System Development Method (DSDM) [27].

These methodologies have many common aspects and in 2001 a core group of people from the agile community met and formulated the agile manifesto [28], which defines that agile methods shift the focus of software development in four general aspects:

159

• Individuals and interactions over processes and tools. • Working software over comprehensive documentation. • Customer collaboration over contract negotiation. • Responding to change over following a plan.

Each of the agile methodologies implements this shift in different ways

but the fundamental principles are similar. For the sake of this paper we focus on XP in order to provide a more detailed analysis of the integration discussed than is possible just studying these fundamental principles.

In the XP case, the focus shift is implemented through the introduction of a set of 12 development practices and by defining a few simple, clearly defined roles within the development team. The focus of XP is, however, on the technical development of the product, dictating on a rather specific level exactly how the work is to be performed, at least at the initial stage of introduction. Large-scale project management is outside the scope of the method, as is the issue of how the development team communicates with project management and how coordination is performed with teams outside the XP group, both software, using XP or an other methodology, as well as non-software.

The fundamental values of XP are communication, simplicity, feedback and courage. The fundamental principles are: rapid feedback, assume simplicity, incremental change, embracing change and quality work. The basic activities are coding, testing, listening and designing. The practices that are intended to realize these values and principles are summarized in Table 1 [12].

XP is an extremely iterative development method where the product is integrated and built several times every day and released to stakeholders every few weeks. The overall strategy lies in developing a minimal running framework as early as possible and then continuously integrate functionality into this. In this way a working version of the product is always available and progress can be measured by the functionality in the product, not in time or documentation as in traditional development. The current product is therefore always available to be demonstrated to stakeholders and can function as an explicit communication media. Quality and technical co-ordination is ensured through automatic test cases that are created for all functionality and executed at each integration. The customer stakeholder’s influence is increased by requiring the customer to be on site all the time to answer questions about the desired functionality.


160

Practice Description Planning Game Quickly determine the scope of the next release by combining

business priorities and technical estimates. As reality overtakesthe plan, update the plan.

Small Releases Put a simple system into production quickly, and then releasenew versions on a very short cycle.

Metaphor Guide all development with a simple shared story of how thewhole system works.

Simple Design The system should be designed as simply as possible at any givenmoment. Extra complexity is removed as soon as it is discovered.

Testing Programmers continually write unit tests, which must runflawlessly for development to continue. Customers write testsdemonstrating that features are finished.

Refactoring Programmers restructure the system without changing itsbehaviour to remove duplication, simplify, or add flexibility.

Pair Programming All production code, i.e. code that is actually used in the finalproduct, is written with two programmers at one machine.

Collective Ownership Anyone can change any code anywhere at any time. Continuous Integration Integrate and build the system many times a day, every time a

task is implemented. 40-hour Week Work no more than 40 hours a week as a rule. Never work

overtime a second week in a row. On-site Customer Include a real, live user on the team, available fulltime to answer

questions. Coding Standard Programmers write all code in accordance with a set of rules

emphasizing communication through code.

Table 1 Extreme programming practices

An XP project works by assigning roles to the team members so that they each have different responsibilities regarding the previously described practices. One role can be assumed by several people and vice versa. The roles are briefly described summarised from the original XP method description in Table 2 [12]. Note that these roles are described only from a perspective of making the development team as effective as possible.

Depending on, among other factors, the size of an organization, the organizational structure is differs. In the small-size environment of 3-15 programmers, in which most of the agile projects have been run [12], the communication channels are short and informal, while in larger organizations, there is a need for more formal structures and separated roles.

In a development project and its environment there are many different stakeholders. Many of these have conflicting interests regarding the decisions to be taken, e.g. priorities of requirements [29]. Further, they have access to different kinds of information and they may speak different “languages”, depending on whether their background is technical,

161

financial, managerial, or if they are representing the users or the application domain [30].

Role Description Programmer The programmer is at the heart of XP. The programmer

creates the product based on the story cards written bythe customer.

Customer The customer provides the functionality to beimplemented in the form of story cards. The customeralso creates functional tests to ensure that the productdoes what it is supposed to do.

Tester The tester supports the customer to ensure that thedesired functionality is implemented. The tester runs thetests regularly and broadcasts the results suitably.

Tracker The tracker’s role is to keep an eye on the estimates andcompare these to the performance of the team. Thetracker should monitor the overall performance andremind the team when they are off track.

Coach The coach’s role is to be responsible for the process as awhole, bringing deviations to the team's notice andproviding process solutions.

Boss The boss’s role is basically to provide for the needs of theteam to ensure it get the job done.

Table 2 Roles in extreme programming

programmercustomeruser

projectacquiring org. producing org.

Figure 2. a) Stakeholders in a small customer-oriented project using agile methods.b) Stakeholders in a large market-oriented project using traditional PM.

projectmanager

programmersub-projectmanager

projectmanager

projectsponsor

customeruser productmanager

projectacquiring org.

producing org.

b)

a)

marketmanager


162

To illustrate the extremes of these combinations, we define two

examples of stakeholder sets, business models and size of organizations. The first, presented in Figure 2a, is a small project in a contract situation, using agile methods. The programmers in the team may communicate directly with the customer, or even with the end user. As the project is contracted at actual cost, no project sponsor has to be involved in the decisions regarding how much can be spent on the project. The project manager involvement is to ensure that the overall goal of the project is reached.

The other example, shown in Figure 2b, contains a situation with a large number of stakeholders. This case performs market-driven development. As there is no contract between the producing and the acquiring organizations, there is a market manager or department within the producing organization, whose task it is to represent the intended market regarding requirements and priorities for the product. Further, as the project is an investment for the company, a project sponsor “owns” the funds and decides how to invest them. In the PM context, there is a hierarchy of sub-projects for different aspects or parts of the product, and finally, there are programmers who are going to develop the product.

Another aspect that may be present in the larger case is product management. The company may have a product line strategy for their software products [31]. In that case, there are also product management stakeholders involved, to make sure that e.g. long-term architectural requirements are taken into account, and balanced towards the short-term project requirements. These product stakeholders are usually organized independently of development and marketing departments and answer directly to management.

The former situation is very much what agile methods are developed for, and it is clear from these examples that the latter situation is much more complex. Still, there is very much to gain from being inspired by the agile methods, and we therefore propose an integrated approach.

3 Integrating Project Management Models and Agile Methods

In order to support the use of agile methods in an environment primarily governed by traditional project management processes and principles, and also to scale up the use of agile methods, we define a hybrid project management model. This model aims at bridging the gaps between the

163

traditional project management models and the agile methods. It is intended to support software development subprojects that use agile methods for a part or the whole of the software development as well as integration towards non-software parts of the project. Section 3.1 elaborated the gaps between PM and agile methods. Section 3.2 presents the basic issues to be handled in the integration. In Section 3.3 we present and overview of the proposed process model and in Section 3.4 we elaborate on the process content.

3.1 Integration Needs The four general aspects of agile methods, defined by the agile manifesto [28] and presented in Section 2.4 summarize the cultural gap between agile methods and traditional project management (PM):

• Individuals and interactions vs. processes and tools. • Working software vs. comprehensive documentation. • Customer collaboration vs. contract negotiation. • Responding to change vs. following a plan.

Agile methods focus on more informal person to person

communication, while PM focuses on processes, tools and documentation. Agile methods focus on the product, which also is in focus of PM, although often in the form of representations of the product, e.g. specifications. Agile methods involve the customer closely in the project in order to eliminate mistakes as quickly as possible, while PM tries to create a contract-like situation in order to control requirements changes, even in case the business model is market-driven. Agile methods focus on rapidly adapting to changes in the environment, while PM is much more plan-driven and inflexible.

At a first glance, the gap seems impossible to bridge. However, the success experienced when applying agile methods motivates an attempt to integration between the two areas. The successful aspects are the empowerment of engineers, the product-focus, the customer-driven development and the iterative approach [12]. We therefore use these as a starting point for the integrated approach.


164

3.2 Issues for Integration The integrated approach must follow some form of PM model in order to enable communication with the world outside the software sub-project, i.e. other sub-projects and stakeholders. Every stakeholder of a large project cannot interact with the sub-project members in person. Hence, there is a need for a process acting as a roadmap and a means for communication. However, the process must enable agile development work at the technical level, without adding more bureaucracy than absolutely necessary.

The technical documentation defined in the integrated process should be governed by the principles of agile methods. This means that the documentation should as far as possible be constituted of the running software, its test programs, and documentation that can be automatically generated from those artefacts. Such artefacts are design documents and test specifications. Any further documentation should only be used if motivated by sincere importance for a stakeholder.

The collaboration between the development sub-project and its customer and other stakeholders, should take the form of a stakeholder representative, as the number of stakeholders in these projects is generally large. Since the stakeholder representative must communicate with many different stakeholders, some form of contracts or specifications are unavoidable. There is also a need for interpretations and priorities among stakeholders, which must be taken care of in the project, although these should be kept to a minimum.

Coordinating software development activities with other activities, for example, hardware development or market introduction, requires planning. However, this should not be a traditional task oriented plan, with time estimates, and where the time in the end tends to be the degree of freedom. Since the development now follows an iterative, release driven plan as a result of the agile approach, the plan is of a constant time boxed nature of the agile release planning with the list of delivered features as the degree of freedom in the plan. This plan is to be synchronized with occasions of milestones and tollgate decisions.

In addition to these four areas it can be concluded based on the principles presented in Section 2, that there are some principles that are contradictory between traditional PM and agile methods. These contradicting principles are presented in Table 3, together with approaches to solutions.

165

Issue PM Agile methods Integrated

approach Priority: Cost/ Time/Functionality/Quality

Time or cost is primary focus, Functionality second and Quality follows

Quality is fixed as high as possible, Functionality is primary focus and Time or Cost is second.

Use the agile methods approach, which reduces risks for delays

Customer/Market communication

Many layers of communication

Direct communication with customer

Stakeholder representative as interface between agile methods and PM

Progress Specification finished Core and X% of the functionality running

Use the agile methods approach, making progress very visible and certain.

Integration of embedded product

Waterfall concept, leading to big bang integration

Requires simulated environment if hardware is developed in parallel

Integration phase acts as an interface

Communication media Document Personal and product Personal and product within agile methods part, documents outside but try to use the product as much as possible.

Table 3 Conflicting principles in traditional PM vs. agile methods

In all project management, there is a conflict between goals of which functions shall be implemented, the quality of the functions, and the cost and schedule. Functions can be developed with high quality, but this takes time and costs resources. Functions can be delivered on time, at budgeted cost, but the quality may not be sufficient. PM is focused on the time or cost. Deliveries should be on time, and quality is thereby given least priority. Agile methods has functionality in focus, ensures fixed quality by continuous tests. Agile methods can deliver on time, but may not deliver all the functions. We propose using the latter approach since it is shown that a small share of software’s functions is really used. It is therefore more important that the delivered functions really are of high quality and are implemented most important first.

In large organizations, there are many layers of stakeholders between the customer and the programmer, while agile methods proposes a direct contact. We don’t think that direct continuous contact between developers and the customer is possible in large projects, but the focus on personal communication as interfaces between agile subprojects and the PM environment should be encouraged as far as possible. One observed


166

company has started introducing customer groups to early product prototypes at a very early stage of development with beneficial results. We have seen a technical product manager taken on the customer role in a very effective way. Layers of stakeholders should be short-circuited as far as possible.

The progress measure on the technical level is much more trustworthy using the agile methods approach, with real actual executing functionality. Hence this is part of the integrated approach. This is also related to the integration of embedded products. By having running products all the time, the risks connected to big-bang integrations are reduced. Nevertheless there is a need for a final integration phase, actually running the software on the target hardware.

Finally, the communication media from agile methods is preferred as far as possible. The document producer must understand why the document is needed, otherwise this person will not produce a good quality document. Further, documents can be produced as ordinary agile tasks, making it clear that there is a cost related to producing the document and communicating this effectively to the person requiring the document making cost/benefit analysis possible.

3.3 Process Model We propose an integrated process model, constituted of two layers, the project management model and a technical development process. The model can be used either when the software constitutes the whole development project, or when the software is one sub-project. Below, we refer to both cases as “project”. The main stakeholder of the process model is project management, top management, and quality management. Engineers working in the project should focus on the product itself and the short-term plans, agile style.

The shape of the PM part remains very much the same as in the general case (see Figure 1a) although the content of the management decisions and the artefacts are changed. On the technical level, we propose a development process, that reflects the iterative nature of agile methods (see Figure 3), although there are some distinct phases defined for coordination purposes. Furthermore, the character of the technical decisions, and the management decisions change compared to the traditional approach. As continuous evaluation of the product is possible, we may utilize this to avoid a large step in the wrong direction by taking several small steps and continuously evaluate the direction, agile style.

167

Figure 3. Technical Development Process using agile methods.

The engineers work within the development process and deliver

product parts and documents on defined occasion. However, they do not relate very much to the process model, but plan their tasks in short iterations and deliver accordingly. The integration towards the overall project is conducted by project management.

3.4 Process Contents In the technical process, we define five types of phases, see Figure 3:

• Prototype, the developers are focused on developing prototypes in order to explore both possible technical solutions, understand these better and as well as investigating the underlying needs that are to be solved.

• Core, the developers are focused on developing a minimalist core product framework [12] with only the absolutely most fundamental functionality included. The end product of this phase is a part of the real product on to which the remaining functionality is added and the final system can evolve from.

• Iteration (Rn), the developers continuously add functionality to the product, starting with the most important according to the customer and delivers to the stakeholders continuously. The end product of the phase is a complete unit and function tested product, ready for integration with other parts of the product and testing in its target environment. In addition to the product growth, further prototyping or other kinds of investigations may

Proto-type

Co- re

H.O.

R1

R2

R3

R4

R5

R6

R7

R8

R ...

R ...

R ...

R ..

R ...

R ...

R ...

Rn

R ...

R ...

R ...

Int.

TG1 TG2 TG3 TG4

MS1 MS4 MS2

P2 FR P1 R

MS3

R


168

be needed during the iterations. This is referred to as spikes in the XP context.

• Integration (Int), the developers focus on the final integration aspects of the software into the complete product and issues that arise during final verification and validation of the end product. The deliverable of the phase is the final product, integrated in its target hardware environment, accepted by the stakeholders and coordinated with other related activities, such as marketing and production, i.e. completely productified.

• Hand-over (HO), focused on handing over the software and the experience gained during the project, to the product owner organization so that the product can be supported for the remainder of its life cycle.

The project management process in our model contains the same four

phases as shown in Figure 1a. • The prestudy phase is mapped to the prototype phase of the

technical process. • The feasibility study phase covers the core phase and the first few

release iterations of the iteration phase. • The execution phase is mapped to the remaining iterations of the

iteration phase. • The completion phase covers the integration and hand-over phases.

The purpose of the prestudy phase is to evaluate whether the project is

technically and commercially feasible. To judge feasibility, the project must gain as much knowledge as possible regarding the product space and possible implementation techniques through prototype development work. Here the project manager and possibly end users will act as customer for the developed prototypes. A metaphor for the product should be derived at this stage as well in order to enhance communication in the project. Parallel sub-projects or activities investigate market, hardware, production and similar issues, which all have an impact on the decision to move on into a feasibility study. The material used as a basis for the TG1 decision is therefore the prototypes developed, and the outcome of the experiences gained through the prototypes and the other concurrent activities mentioned.

The purpose of the feasibility study in traditional software projects, as

169

described in Section 2, is to establish a strategy and plans, in terms of requirements specifications, high-level design specifications and project plans, from which the development may begin. Using agile methods, a firmer technical basis can be established instead, in terms of the core product framework, and the most fundamental, i.e. the most important, functionality that demonstrates the feasibility of the chosen solution. In addition, estimates on required effort for feature development can be based on the first cycles of the iteration phase, and hence be more accurate than estimates in traditional development projects based solely on experience from previous projects. The material for the TG2 decision is therefore the core product, and actual performance data collected during the first cycles. This material is further elaborated in the example provided in Section 4.

The execution phase contains the main part of the project development work. The development work is performed in iterations, using the principles of agile methodology. This implies, in contrast to traditional PM:

• Deliveries are primarily planned by date, and secondarily by the

functional content (time boxing). The quality is controlled by continuous testing, setting a clear, constantly high, quality threshold.

• Priorities are set by the stakeholder representative, acting as a liaison between the project, the sponsor, the product owner, the customer, the users etc.

• Status reporting is performed in terms of a list of implemented features, possible to demonstrate using the product itself, thus increasing the impact of the results of the development. This implies that status reports can be made available continuously, not only at discrete reporting points in time. Still, to interface towards the PM, status is reported on specific points in time, as defined by the process model.

• The developers are more involved in the project management, as the developers are an important part of the day to day micro-planning. If micro planning is accurate and effective, macro-plans are better adhered to.

• The developers gain direct feedback from stakeholders continuously rather than at the end of the development. Fixing deviations from stakeholder wishes as quickly as possible, reducing the cost of change.


170

In addition to the iterative development work, we define an integration

phase of the technical process. This phase would, however, be shorter than in traditional development as the product has been more thoroughly tested during the iterative development by automated testing both at unit and system level. Using agile methods, the work product is growing continuously towards its final shape, and the product may be released in its current form to the stakeholders, including the final customers and users, at any time. Nevertheless there is a need for synchronizing the product with other sub-projects, to allow for needs within a complicated development project environments such as:

• Hardware development • Subcontracted software development • System integration and testing of software and hardware • Production organization and procedures • Support organization • Customer acceptance testing (contract business case) • Marketing (market business case)

From a development point of view, this is considered a delivery to the

customer, whether the customer is an internal stakeholder within the organization or an external one.

3.5 Roles The roles of an XP team, as defined in Table 2, interface towards the PM environment naturally using the functionality in the implemented product as the main communication media. The boss and the customer roles are the most important interfaces, as they act in both the agile methods and the PM environment but the shift of methodology puts increased focus on the developers as they are now working with the most significant artefact in the project. The boss role corresponds to the sub-project or project manager and is responsible for providing the same resources as in the original XP methodology with the addition of interfacing with more senior management and parallel management. The day-to-day customer role should preferably be chosen among technical product management. It must, however, be realized that the customer situation is very complex in large systems. Customers can include end users, top management, marketing for example and these should be

171

allowed to influence the project at different stages and suitable intervals. The tracker is a role completely within the XP team, as are the programmer and tester roles and these are therefore relatively unchanged. The coach is an important supporting role that can be internal or external to the company. It is important that the coach is well acquainted with the concepts of the PM model as well as the technical process.

3.6 Summary of Proposal The proposed model is developed with the aim that agile methods can coexist beneficially with more traditional process-oriented project management approaches. Our approach adapts the interfaces to the software developing part of the project that are using agile methods and, by keeping the changes at the project level small, the result is in effect an encapsulation of the agile method in the development part of the organization, but still utilizing the increased control capabilities available by using agile methods for the development. The communication purpose is kept from the PM, but the communication is filled with information that is unique to agile methods.

As a result, the software development can utilize the benefits of agile methods, and still it can coexist in a proven traditional project management environment. In addition, the increased control available to management constitutes a major motivational factor when introducing the method.

4 Example To illuminate the concept of our proposal we present an example in which we investigate the material prepared for the decision to enter the main project execution phase in more detail. The decision is one of the most critical in the whole project, as it is at this point that the project ramps up to full scale with respect to committing resources and priorities. The outcome of the decision may be pass, terminate project, or prepare a better decision basis.

The example is based on a project using the PROPS model [23]; a company proprietary project management model at Ericsson, see Figure 4. The model has been used in a large number of industrial projects both in pure software projects and in concurrent hardware-software development. The outer “U” is the PM and the inner “U is the technical development model.


172

Pre-study

Feasibility

Execution

Conclusion

Figure 4. Overview of PROPS project management model

TG2

TG1

TG3

TG4

TG5

We study the information available at the start of execution tollgate,

denoted TG2, in the current model and compare this to the information that would be available had an agile approach been used at the development level instead. In the PROPS project management model, each tollgate assessment is based on three perspectives:

• Project status • Business situation • Use of resources

Project status involves reporting the technical progress of the

development, the requirements set on the project, how the project

TG1

TG2

TG3

TG4

TG5

173

performs, and any risks or uncertainties observed. The business situation comprises judgement regarding the customers, the direction of the business, observations regarding market and other trends and calculations of profitability of the product. Finally, the resources issues concern the project’s use of resources and to total project portfolio. Issue PM Agile methods Project status Technical progress Documents:

-Requirements specification -Design specification

Code: -Functioning, minimal core -X% of feature list

Requirements High-level: Specified in Requirements specification Medium-level: Specified in Design specification

High-level: Specified in story cards Low-level: Specified in test programs as we go

Performance Monitored via documents

Monitored via executable code

Risks Project scoping risks moderate for all features Implementation and integration risks unknown

Project scoping risks high for features with low importance and low for features with high importance Implementation and integration risks low

Business situation

Customers Analysed Involved

Market and trends Analysed Analysed or partly involved

Profitability analysis Based on estimates of project costs, which may be unsure

Based on different scenarios regarding how much functionality is implemented within budget

Resources Resources Relation between resource usage and final outcome hard to know

Prediction of resource use can be made based on actual development

Project portfolio Product line management

No functionality planned for the future

Table 4 Comparison of decision basis for traditional projects and agile methods projects


174

Applied to a software development project using agile methods, these three perspectives can be given new interpretations related to the difference towards traditional software development projects. Table 4 shows the information available regarding these issues in a traditional software project, compared to a project using agile methods.

To assess project status at TG2 in traditional projects, documents are used as indicators of technical progress, e.g. requirements specifications and design specifications. Specifications have a defined status in terms of the quality assurance reviews they undergo. However, there is always a large amount of interpretation and judgement regarding the quality of a document, which leads to risks in using them as progress indicators. Projects using agile methods can demonstrate the project status at TG2 in terms of a running core of the product, validating the basic realization concepts, as well as one or a few of the most important features implemented and running. These progress indicators involve much less interpretation and judgement and hence a reduced risk.

The requirements in a traditional project are specified on a high level in requirements specifications and on lower, more design-oriented levels, as design specifications. Mostly, plain text is used with a defined structure in a template format. More strict notations, like use cases [32] also exist, although text is still predominating. Using agile methods, high-level requirements are expressed using story cards, which tell, from a user perspective, the aim and user context for each feature. The low-level requirements are defined in terms of engineering tasks and test code, written before the actual program code is written. The requirements are in the agile case more precisely defined for parts already worked on, while the long-term requirements are less elaborated. This means that resources are spent on parts that are actually developed instead of parts that are later discarded.

The project performance is monitored via progress reports and other kinds of documents in the traditional case, while agile methods focus on monitoring progress through demonstrating implemented functionality in the executable code. This forces the development environment to support continuous integration and shipping, but reduces the risk for false judgements, such as “40% of the code is written, but nothing is running yet”, which gives no real indication of progress and creates a feeling of uncertainty in the project, in turn affecting progress.

Project scoping risks are moderate in the traditional case since the intended product is fully specified, or at least is supposed to be fully specified. Implementation and integration risks however are unknown,

175

since it is postponed until later stages of the project to verify the chosen concepts and the implemented code actually works as intended. In the agile case, on the contrary, the scoping risks are higher since the intention is not to specify all details from the beginning but to define the scope as time passes. The most important feature first principle, however, works to reduce the scoping risk for the most important features and move this to the less important features where the effect is less important to the stakeholders. The customer/user representative is the guarantee for the long-term project scoping. This approach leads to reduced implementation and integration risks in total, since this is performed continuously and is therefore verified continuously.

Regarding the business situation, the differences are smaller between traditional methods and agile methods. Customers have to work closer to an agile project, since the project scoping is defined by the internal or external customers continuously, depending on the business situation. Hence the customer situation is analysed in the traditional case, while the customer is involved in the agile case. Using agile methods, a customer may also be internal in the organization, e.g. a technical product manager.

The analysis of the market and ongoing trends is the same independently of approach, although the agile project may adjust more easily to unexpectedly changing markets and trends.

The profitability analysis is the same on the income side, while the agile project has the advantage of having better means for estimating the investment cost in the project. In a traditional project, the costs are based on estimates, which are very unsure in the software business, in many cases not more substantial than experienced guesswork. The agile methods offer the opportunity to launch a product into the market at different stages, and the profitability analysis can the be performed as a trade-off between investment costs on a feature level and the expected market value of different variants of the product with different feature contents.

The resource analysis in traditional development suffers from the uncertain estimates regarding time and effort for development tasks. In the agile case, better estimates are made possible by collecting data from the implementation of the early iterations, and using this information to predict the resources use for later iterations. Further, a team using agile methods have very good control over the day to day micro-planning.

Regarding project portfolio management, product lines is an approach used in traditional projects. Software components can be developed to be used in multiple products within a product family. Agile methods focus in


176

principle on one delivery of one product, not spending effort on designing general solutions for possible future needs. Hence, this is an issue where agile methods have to be adjusted to fit into a situation where a product line approach is beneficial.

In summary, the agile approach of measuring progress in terms of running software reduces risks, both regarding the end of project (the product can be delivered at any time, with known quality) and the estimates (we know exactly how much time the currently running features took to develop). Hence, by focusing on running software, instead of time plans, delivery precision may improve. On the other hand, the lack of a traditional requirements specification can be interpreted by people used to traditional big-up-front methodologies as involving a risk regarding the scope of the project. In many cases the sheer size of the project scope impedes the development from proceeding at an optimal rate. The stakeholder representative is a very important liaison between many involved parties: project staff, project manager, project sponsor, market/ customer etc.

The tollgate decision as such, i.e. pass or fail, has to be based on the presented material. If the technical material available to base the decision on is insufficient, one or two additional iterations may provide a sufficient basis.

This example focusing on the decision to enter the execution phase of a project, has illustrated some of the conflicting principles between traditional project management and agile methods in Table 4. In Table 5 we cross reference the impact of the integrated project management approach and issues in the TG2 decisions.

Issue Integrated approach TG2 issue Priority: Cost/Time/ Functionality/Quality

Use the agile methods approach, which reduces risks for delays

Risks (project scoping) Profitability analysis Resources

Customer/Market communication

Stakeholder representative Customers Markets and trends

Progress Use the agile methods approach, which is possible to verify

Technical progress Performance

Integration of embedded product

Integration phase acts as an interface

Risks

Communication media Personal and product within the agile methods part, documents outside

Technical progress Requirements

Table 5 Principles in integrated approach at TG2

177

5 Challenges for the Future We have outlined a hybrid project management model to support the use of agile methods in an environment governed by traditional project management models, either as software-only projects, or more interestingly, as software sub-projects in development of complex systems such as embedded systems. The model consists of a process and defined interfaces and communication media, where the process part is very much in line with the proposal by Wallin et al. [5]. For a true integration of agile methods and project management models, the hybrid approach must have constituents from both origins. We have defined a few process steps to support a joint roadmap with other sub-projects. Further, we stress a minimalistic approach regarding documentation, and define a role of stakeholder representative to act as a liaison between the stakeholders of a project, such as customers, users, project sponsor, product owner etc. providing rapid feedback to cut development costs.

Further work regarding integration of project management models and agile methods is to evaluate the model in pilot projects in real environments. Specific issues to bridge are those of cultural differences between management and technical staff [30]. The technical staff appreciates the technically oriented approach in agile methods, while this is a hinder for management who are not (and should not be) so technically involved. Further, the focus on performance regarding project progress, rather than time, is also a cultural shift. Our hypothesis is that by focusing on the running software, risks are reduced and as management have a firmer basis for decision making. Finally, the focus on oral communication over documents, tends to conflict with, especially, quality management. However, many of the quality models have evolved from traditional development models and may need revision if agile approaches are proven more effective for high quality software development. The stakeholder liaison is in this respect a critical resource, and the pilot project has also to evaluate whether this role is possible to take or not.

Another issue for further work is investigating how process improvement can be integrated into these models. Effective process improvement ensures that the process is as effective as possible in the given context at the company at the time of the work. Challenges here are the different languages used at different levels of the company and the different views on what is to be improved [33].


178

6 Acknowledgement Thanks to Bertil I. Nilsson, dept. of Production Management, Lund University for valuable comments on an earlier version of this paper. The work is financially supported by the Swedish Agency for Innovation Systems, VINNOVA, under grant for LUCAS, Center for Applied Software Research. The work was also in part supported by The Fulbright Commission, the Royal Swedish Academy of Sciences and The Royal Physiographic Society in Lund.

7 References [1] Cockburn, A., Agile Software Development, Addison-Wesley, Boston, 2002. [2] Cooper, R. G., Winning at New Products, 3rd edition, Perseus Publishing,

Cambridge, MA, 2001. [3] Gilb, T., Principles of Software Engineering Management, Addison-Wesley,

1998. [4] Royce, W., Software Project Management—A Unified Approach, Addison-

Wesley Longman, 1998. [5] Wallin, C., Ekdahl, F., Larsson, S., “Integrating Business and Software


[6] Paulk, M. C., Curtis, B., Chrissis, M. B., Weber, C. V., Capability Maturity

Model for Software, Version 1.1, Software Engineering Institute, CMU/SEI-93-TR-24, February 1993.

[7] CMMI Product Team, CMMISM for Software Engineering, Version 1.1,

Staged Representation, Technical Report CMU/SEI-2002-TR-029 [8] Karlström, D., Runeson, P., “Decision Support for Extreme Programming

Introduction and Practice Selection”, Proceedings The Fourteenth International Conference on Software Engineering and Knowledge Engineering (SEKE'02), Ischia, Italy, 2002.

[9] Paulk, M. C., Weber, C. V., Curtis, B., The Capability Maturity Model:

Guidelines for improving the Software Process, Addison Wesley, 1995. [10] Paulk, M. C., “Practices of High Maturity Organizations,” The 11th Software


179

[11] The IEEE dynabook on Extreme Programming http://computer.org/seweb/Dynabook/Index.htm , homepage last visited 2003-06-03.

[12] Beck, K., “Embracing Change with Extreme Programming”, IEEE Computer,

October 1999, pp. 70-77. [13] Sommerville, I., Software Engineering, Addison-Wesley, 2001. [14] Zahran, S., Software Process Improvement - Practical Guidelines for Business

Success, Harlow, England: Addison-Wesley,1997. [15] Whittaker, J. A., Voas, J. M., “50 Years of Software: Key Principles for

Quality”, IT Professional, November/December 2002, pp.28-35. [16] Boehm, B. W., “A spiral model of software development and enhancement”,

IEEE Computer, Vol.21, No 5, May 1988, pp. 61-72. [17] Brooks Jr., F. P., The Mythical Man-Month, 20th anniversary edition, Addison

Wesley, 1995. [18] Naur, P., Randell, B., editors. Software Engineering. Report on a conference,

NATO Scientific Affairs Division, Garmisch, 1968. [19] Royce, W. , “Managing the development of large software systems: concepts

and techniques”, Proceedings IEEE WESTCON, August 1970. [20] Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981 [21] Paulk, M. C., “Using the Software CMM in Small Organizations,” The Joint

1998 Proceedings of the Pacific Northwest Software Quality Conference and the Eighth International Conference on Software Quality, Portland, Oregon, 13-14 October 1998, pp. 350-361

[22] Lai, R. ,“The move to mature process”, IEEE Software July, 1993, pp. 14-17. [23] Mulder, L., “The importance of a common project management method in the


[24] Highsmith, J., A., Adaptive Software Development: A Collaborative Approach to

Managing Complex Systems, Dorset House, 2000. [25] Schwaber, K., and Beedle, M., Agile Software Development with SCRUM,

Prentice Hall, 2001. [26] Palmer, S., R. and J. M. Felsing, A Practical Guide to Feature-Driven

Development, Prentice Hall PTR, 2002.


180

[27] Stapleton, J., Dsdm Dynamic Systems Development Method: The Method in

Practice, Addison-Wesley, 1997. [28] http://agilemanifesto.org, last visited 2004-06-30 [29] Regnell, B., Höst, M., Natt och Dag, J., Beremark, P., Hjelm, T., “An

Industrial Case Study on Distributed Prioritization in Market-Driven Requirements Engineering for Packaged Software,” Requirements Engineering Journal, 6:51-62, 2001.

[30] McKinney, D., “Six Translations between Software-Speak and Management-

Speak”, IEEE Software, November/December 2002, pp. 50-52. [31] Bosch, J., Design and Use of Software Architectures: Adopting and Evolving a

Product-Line Approach, Addison-Wesley 2000. [32] Weidenhaupt, K., Pohl, K., Jarke, M., Haumer, P., “Scenarios in System

Development: Current Practice”, IEEE Software, pp 34-45, March/April 1998. [33] Karlström, D., Runeson, P., Wohlin, C., “Aggregating Viewpoints for Strategic

Software Process Improvement”, IEE Proceedings Software, 149(5): 143-152, 2002.

181

Integrating Agile Software Development into Stage-Gate Managed Product Development

Karlström, D., Runeson, P., “Integrating Agile Software Development into Stage-Gate Managed Product Development”, submitted to the Journal of Empirical Software Engineering, 2004.

Abstract Agile methods have evolved as a bottom-up approach to software development. However, as the software in embedded products is only one part of development projects, agile methods must coexist with project management models typically of the stage-gate type. This paper presents a qualitative case study of two large independent software system projects that have used Extreme Programming (XP) for software development within contexts of stage-gate project management models. The study comprises interviews with managers as well as practitioners. We conclude that it is possible to integrate XP in a gate model context. Key issues for success are the interfaces towards the agile subproject and management attitudes towards the agile approach.

6


182

1 Introduction Agile methods have evolved as a new approach to developing software products [1]. The methods have been successfully implemented in small to medium sized projects, and the principles of the agile methods have inspired software developers in wide-ranging application domains [e.g. . 2, 3, 4, 5, 6, 7, 8, 9]. The agile methodologies offer down-to-earth approaches to software development that focus on simplifying and improving the software development process, making the customers, the developers and the final product the most important [1]. A summary of the main principles involved in the agile methods as formulated by the agile manifesto is given in Table 1.

In real-life, however, software development projects are not isolated activities. They usually exist as sub-projects in an environment composed of hardware development, marketing, production planning etc. which all must be managed and coordinated concurrently [10]. Furthermore, software development seldom starts from scratch; legacy software constitutes the core of new products to which new functionality and other types of customer value are added, or legacy software is moved to new platforms. The traditional approach when managing a complex development project situation is to use a project management model (PM) such as Cooper’s Stage-gate model [10], see Figure 1. The PM gives support not only for the communication within the project, but also for decision makers sponsoring the project or acquiring the outcome of the project. The concepts of a PM are not connected to any specific domain; these models are sufficiently generic to be applied to any kind of project. However, PM:s currently tend to have a basic sequential structure, i.e. distinct phases of pre-study, feasibility study, execution and conclusion. From a management point of view this is quite natural.

Figure 1 Stage-Gate™ model [10]

Launch

Discovery


Post launch review

G2 G1

G3 G4 G5



183

1. Individuals and interactions over processes and tools. 2. Working software over comprehensive documentation. 3. Customer collaboration over contract negotiation. 4. Responding to change over following a plan.

Table 1 The agile manifesto [1]

Traditional PM:s are compatible with maturity-oriented software process improvement work, using for instance the CMM framework [11]. Both PM:s and the maturity models aim at defining structures, roles and responsibilities within the organisation, but do not intend to actually dictate how the actual technical project work is to be performed. The agile methodologies can be considered a reaction to these maturity-oriented approaches within the software development process field, but we do not believe that the two schools are contradictive. Software projects using agile methods must be able to exist within environments structured using traditional PM. Mature organizations add not only rigour and order to a chaotic environment, but also certain characteristics similar to a successful agile development project [12, 13, 14].

Traditional PM models have already been integrated with more iterative development models in general [15] and the Rational Unified Process (RUP) in particular [16] Recently an example of the possibility of integration with agile methods has been presented [17], although only at an overview level so far. Software projects using agile methods have indeed been executed in traditional PM environments, but the main characteristics of the PM are not originally designed for the management of software projects using agile methods, and their advantages are therefore not utilised in the management of the project. The agile development has in these cases had no effect on the project management and overall management of the project despite demonstrating success on the development level.

The research presented by Wallin et al. [17] presents the overall concepts and problems involved in combining a project management model with several different software development processes, including RUP and XP. The study however, is lacking in detail of how the interfaces are defined and how the different processes need to be modified.

Rainer et al. present interesting work on buy in for software development process improvement [18]. This is an especially relevant issue for agile methods as these are often introduced bottom up in contrast to top down for most other software development processes. They present the importance of showing local effectiveness of the process to developers in contrast to metrics from other locations.


184

In this paper we investigate the experiences from integrating agile teams in traditional project management models. Two cases are studied, both within large system development companies, comprising both software and hardware development. One company is in the domain of airborne defence radar systems and the other in industrial control systems. In both cases the agile method used was XP, but each company used their own variant of the gate model.

This paper is structured as follows. In Section 2, the study scope is presented and related work is discussed in Section 3. In Section 4, background information about the companies and development teams involved is presented. The study design is presented in Section 5 and study validity in Section 6. Finally, the results are presented in Section 7 and a summary is provided together with the conclusions in Section 8.

2 Study Scope The study aims to investigate the effects of introducing an agile software development process within a traditional stage gate model style project management system. The study specifically aims to identify the main problems with this approach as well as key success factors. The specific projects studied were using extreme programming within their own variants of the gate model. To narrow the scope of the investigation it was decided to focus on the decision point gate 3 (or G3 as shown in Figure 1), i.e. deciding on start of full-scale development, as this is considered a crucial point in projects. Here it is decided whether the project should proceed into full-scale development and a large increase in resources is to be allocated. The teams that were investigated however were far beyond G3 and as such could only describe this as they remembered it. Furthermore, specific focus was set on the interfacing between the XP team and the surrounding environment.

3 Case Contexts The context of the case study is presented according to the recommendations by Kitchenham et al. [19] stating that the experimental context needs the three elements background information, discussion of research hypotheses, and information about related research. The two former are presented below, while the latter is presented in Section 3.

185

3.1 Background Ericsson Microwave Systems Case Ericsson Microwave Systems (EMW) is a part of Ericsson global communications company and is developing systems for radar and microwave communication. The particular section that is studied in this research constructs airborne military radar systems. The company culture is influenced by the military application domain, as they are used to requirements on document-driven development processes. EMW has about 2000 employees and Ericsson as a whole has around 50 000 employees.

The organisation of EMW is of a matrix type with a line management and a functional management. The functional management is still structured according to the physical parts of the radar system when it was constructed completely in hardware. The system is moving more and more towards software solutions and as a part of this the optimal functional organisational structure is becoming unclear.

The team that was studied completed two projects during using agile methods. The first was a target tracking function within the radar system. This function is relatively isolated related to the overall system and communicates with relatively few other parts of the system. The second function was a data integration function, communicating with virtually all parts of the system. The team size varied between 6 and 10 people during the course of the projects and during a part of the project the team was split into two.

When the team was formed in order to commence the first project they were eager to try a new methodology. At the time XP was frequently mentioned in software engineering media and the team thought it would suit them well. Management were indifferent to the suggestion and the team decided to go ahead. The team used no external resources to introduce the method. Instead they studied the literature available in a workshop format and applied the methods to their specific situation.

The project management model used at Ericsson is known as PROPS [20]. This model is defined on the corporate level and is very similar to Coopers Stage-Gate model [10] and allows for a large degree of flexibility in the choice of development method. There is a development process on a more technical level which is integrated in the gate model, which is company-tailored. It was, however, unclear and unplanned how it would work together with XP.

The EMW context can in summary be characterized by its military background and the corporate level gate model. The introduction of XP was bottom-up, without explicit management support.


186

3.2 Background ABB SCADA Case ABB SCADA is a part of Asea Brown Boveri (ABB) global company with more than 100 000 employees in total. The development facility studied consists of more than 100 developers in total and produces hardware and software for automated industrial control systems. The organization studied was integrated in ABB through acquisition some 10 years ago. The company culture has its roots in a small, but steadily growing software development company.

The project studied is a part of the matrix organisational structure with both functional and line management. ABB also has a system of technical and market product managers for each product that work in cooperation with the developers. The studied development team consisted of eight people, but only six of these were part of the XP group. The other two were located off-site and used a more traditional development approach.

The project members and the project leader back in 2002 were inspired by the local SPIN (Software Process Improvement Network) [21] on XP and decided to try the method. An external consultant was hired for the initial introduction of the method. The compliance with the ABB gate model, was not accounted for at the start, but a study by ABB corporate research was conducted and resulted in a part of the results presented by Wallin et al.[17]. The ABB stage gate model is also similar to Cooper’s Stage Gate model [10], although more integrated with the technical processes.

The ABB context can in summary be characterized by its small and growing company background, the corporate gate model with a technical focus. The introduction of XP was accepted by management and supported by external consultants.

4 Study Design The case study performed is of the applied research type, implying that the purpose is to understand the nature of and the problems present within a certain context in society [22]. The study takes a broad view within the area of integrating agile methods into gate models and some scooping is described in section 3. The units of analysis are individual members at different positions in the organisation, the teams that these people are organised in and the organisation as a whole, including its process descriptions. The study is focused on the experiences gained during the time that the teams were using agile methods in the gate

187

model context compared to previous experiences within the same organisation without agile methods.

4.1 Subjects The subject sampling strategy was to interview a complete sample of the teams involved and their management in the immediate surrounding. However, a few people were unavailable at the time of the study. All participants were interviewed voluntarily and all data were treated strictly confidentially after the interview occasions. In total, 9 people were interviewed as presented in Table 2.

ABB EMW Department manager Subproject manager Developer Developer Developer

Department manager Developer Developer Developer

Table 2 Interview subjects in the study

4.2 Research Strategy Based on the low epistemological level within software engineering in general, and specifically in the area of integrating agile methods in software engineering, we approach the research using a qualitative research strategy [22, 23, 24, 25]. Inspired by Patton’s classification of Farming Systems Research and Development (FSRD) [22], we elect to study the software engineering environment as a complete dynamic system based on the following observations. Software engineering research and practice…

… is a team effort. … is interdisciplinary. … takes place in the field. … is collaborative. … is comprehensive. … is inductive and exploratory. … begins with qualitative description. … is sensitive to context. … is interactive, dynamic and process oriented. … is situationally responsive and adaptive.


188

These are the characteristics defined up by Patton for another area of research, from this we apply the same observations on software engineering and thereby conclude a similar research strategy.

4.3 Research Methods The main source of information in this investigation is the interviews performed with the development team members and the management in proximity to the teams in the studied organisations.

The interviews were performed as open-ended interviews [26], more in the form of a discussion using the interview instrument as a guide of subjects available to discuss. Interesting facts mentioned were followed up immediately with each subject instead of strictly following the instrument. The interview instrument was constructed by two researchers and was adapted as the interviews progressed. The interview instrument consisted of the following main sections:

• Introduction • Personal, group and company background • General experiences of introducing XP • Experiences of combining XP and gate models • Ending

The interviews were approximately one hour in length with two

researchers interviewing one subject. Some notes were taken during the interview but the main form of documentation was the sound recording intended for transcription as a part of the analysis.

4.4 Analysis Following is a complete description and discussion of the analysis steps taken in the study, an overview of which is given in Figure 2.

During the course of the analysis information takes on several forms at different levels of abstraction. The subjects act in and observe the reality within their organisation. Their reflections and observations on the area of research are discussed in the open-ended interviews and recorded as an audio file. These recordings are then fully transcribed and interesting quotes are identified. Next, quotes are coded and grouped to support each other and finally individual results are identified from these groups. The grouping was conducted using plain tables in a word processing product.

189

The different forms that the information takes in this investigation are numbered 1 to 7 in Figure 2. The first form is the actual event in the study context. The second form is the subject’s perception of the event, and so on as shown in Table 3. Each change in form is due to some kind of treatment and we refer to this change in form as a transition. The transitions are denoted through in Figure 2 and the subsequent tables. Each transition introduces additional validity threats to the results depending on the treatment performed in the transition. Treatments and the validity threats introduced are summarised in Table 4. Countermeasures to these validity threats are addressed in Section 5. These countermeasures can also be seen in part throughout the study design, as the one of the goals of a good study design is to minimise the effects of threats. Some of the transitions in the analysis were performed redundantly by two researchers independently of each other. The results of the individual analyses in these transitions were compared and aggregated into a common analysis result. This is indicated by a star next to transitions involving redundancy in Tables 4 and 5.

The analysis performed in the study is of the inductive type, implying that patterns, themes and categories of analysis come from the data itself in contrast to being imposed prior to the data collection, as is the case in deductive analysis. All approaches used in inductive analysis according to Patton have been considered in the analysis, except for the synthesising of studies [22]. This is however currently under preparation as further work is being planned.

Figure 2 Forms and transitions in the research analysis process. The forms in each step are presented in Table 3

EMW ABB 1

2

α

β 3γ 4δ 5ε 6ζ 7


190

Form# Description 1 Actual events within case context 2 Subjects’ perceptions based on their

observations and experiences 3 Sound recording of interview 4 Transcription of recording 5 Coded quotes from transcribed material 6 Grouped quotes 7 Independent results

Table 3 Information forms in the research process

Transition From

form To form

Treatment description

Threats

α 1 2 Native observations Misconceptions, misunderstanding, lack of objectivity

β* 2 3 Interview Misunderstandings, omissions,

γ 3 4 Transcription Audibility of source

δ* 4 5 Identifying quotes and coding

Misunderstanding, misinterpretation of intent

ε* 5 6 Grouping Incorrect grouping due to effects of previous threats

ζ* 6 7 Identifying specific results Incorrect results due to effects of previous threats

Table 4 Transitions between information forms in the research process (* indicate redundancy in transition)

In addition to the interview data, other information sources in the study include archival analysis of internal documents, including process description documents as well as experience documents and informal information gained through conversations outside the interviews.

4.5 Validation The preliminary results from the study were presented back to the teams in a workshop and their opinions were collected and any misinterpretations were resolved. Final conclusions were based on all the gathered information.

The interview study was performed in a short time space, three days in total, but the companies in the study are involved in a long-term cooperation with the research group and there is therefore much tacit knowledge already present, both generally, concerning the overall company organisation, products and working climate, as well as specifically the technical development details within the teams concerned.

191

5 Study Validity The qualitative strategy selected involves using the researcher himself as the instrument for the research. A short discussion of the effects of the particular researchers involved is therefore appropriate [23].

The main author has been involved in industrial product development since 1993 and has been performing research within the area for over four years. The main focus of research has been software process and software process improvement through involving developers more in how they work and improve their work. He has followed the agile movement as it has developed from a small community to a global phenomenon. Methodologically, he has used both empirical experiments and qualitative case studies extensively in his research.

The second author has worked with a traditional process modelling and quality assurance perspective, both in industry and research. The main focus is processes for verification and validation, although the whole process is within the interest scope. He has gradually adopted ideas from agile methods, through critical analysis of its effects. He uses a wide range of research methodologies, both qualitative and quantitative.

To reduce the threats to validity of the study, countermeasures are taken in the study design and during the whole study. Further, an analysis of threats to the validity and corresponding strategies are presented below, using the Lincoln and Guba model [23], see Table 6. Specific countermeasures to the validity threats brought up in Section 5 are summarised in Table 5.

Transition Threats Countermeasures α - Misconceptions

- Misunderstanding - Lack of objectivity

- Use several sources with similar perspectives. - Identify inconsistencies between interviews.

β* - Misunderstandings - Omissions

- Feedback to subjects before final results - Multiple researchers during interview.

γ - Audibility of source - Use high quality microphone and recording equipment.

δ* - Misunderstanding - Misinterpretation of intent

- Feedback to subjects before final results. - Multiple researchers in quoting and coding.

ε* - Incorrect grouping due to effects of previous threats

- Previous countermeasures - Multiple researchers in grouping.

ζ* - Incorrect results due to effects of previous threats

- Previous countermeasures - Multiple researchers in conclusions.

Table 5 Countermeasures to validity threats in the study


192

Threat to validity Strategy Reactivity Researcher

bias Respondent bias

Prolonged involvement

Reduces threat Increases threat Reduces threat

Triangulation Reduces threat Reduces threat Reduces threat Peer debriefing No effect Reduces threat No effect Member checking Reduces threat Reduces threat Reduces threat Negative case analysis

No effect Reduces threat No effect

Audit trail No effect Reduces threat No effect

Table 6 Strategies for dealing with threats to validity [23]

The researchers have a long-term research involvement with the case

study companies, while the specific case study observations are conducted during a short period of time. This is assumed to provide a balance between the researcher bias threats and the respondent and reactivity threats.

Triangulation is attained in four different ways, which further increases the validity of the study. A summary is provided below: • Data triangulation

Multiple data sources were used in the study, interviews, archival and informal.

• Investigator triangulation Interviews were performed by two researchers together. Important analysis steps were performed by two researchers independently.

• Theory triangulation Multiple perspectives are represented in the subject set as well as in the feedback phase

• Methodological triangulation Multiple methods were used, both qualitative interviews and qualitative archival analysis and a few quantitative measures to investigate relevant metrics.

Peer debriefing is not used to any larger extent in the study, except in the very late stage of the analysis. In order to be effective, a peer from a different research angle should be used, and this was not available. Member checking on the other hand is used continuously, by feeding back both transcripts and analyses to the respondents.

The two researchers have conducted several steps independently from each other, as mentioned previously, as investigator triangulation. This approach also provides negative case analysis.

193

An extensive approach to audit trail is used in the study. The analysis is conducted with rigour. Each statement of interest in the transcribed material is encoded and summarized in tables. The results are the then derived from the tables, still keeping the full traceability back to each statement in the interviews, which the results are based on.

The large number of projects currently being coordinated using stage-gate models [10] implies that, at least from the perspective of using such a model, transferability should be high, although a case study always is coloured by its specific context. Using two different cases with two different sets of application domain, company cultures etc. also increases the transferability of the study.

In summary, all reasonable countermeasures to validity threats have been implemented in the study. It is therefore contended by the authors that the validity of the study is sound.

6 Results This section presents the results identified in the case studies through the analysis procedure described in the paper. The results are presented individually and aggregated conclusions are presented in Section 7. We present the findings structured into five major groups in Section 6.1-6.5:

• General • Process • Communication • Timing • Values

Further, among the observations, there are three contrasting perspectives on the observations; agile vs. non-agile teams, management vs. engineers, and EMW vs. ABB. The observations are discussed with respect to these perspectives in Section 6.6.

6.1 General

6.1.1 EMW

During the project management were negative to the agile method used at the engineering level due to the fact that they felt uneasy with a different method. They felt less in control of the project and were unable to squeeze extra functionality into the project without something else


194

being removed. The developers however felt a strong sense of control in the project and were very pleased. The opinion was that the information that the management required existed, they just did not look in the right places.

In retrospect of the project the results are causing positive reactions within management at the company. The product quality achieved in the project is significantly higher than other parts of the organisation and there is now an interest to continue investigating the potential of agile approaches.

6.1.2 ABB

The project was started during quarter 3 in the year 2000 using the standard development process at ABB. XP was introduced in quarter 1 of 2001 and product releases were made at the end of 2001 and 2002, as well as spring 2004. These events are shown on a timeline in Figure 3.

The team has observed a measured increase in quality with significantly less defects in similar parts of the product as well as much fewer problems in integration. The defect data recorded in the defect management system at ABB has been extracted and is shown in the graph presented in Figure 4. Note that the data for 2000 and 2004 is only for the part of the year during which the project was active.

Currently, the team has been dispersed as it has completed the project ahead of schedule and members have been relocated to help other projects that have not able to comply with their schedule.

Figure 3 Significant events in the ABB development team

2000 2001 2002 2003 2004

Release P1 Release P3Release P2

XPIntro

Project start

195

Figure 4 Defect report statistics from the team at ABB

6.2 Process

6.2.1 Gate Model Structure

A general observation made is that the ABB stage gate model (Wallin, 2002) defines more low-level technical development details than the Ericsson PROPS model (Mulder, 1997). In the ABB model, the technical process model is more tightly integrated into the management process model, while in the PROPS case, the connections can be more easily adapted. Still there is a resistance towards making the necessary adaptations. Nevertheless, the PROPS model therefore seems to be more flexible and insensitive to the underlying development process.

6.2.2 Gate Model Adherence

The gate models in the companies are not strictly adhered to in any of the companies, with or without XP. This is especially the case when a project comes under time pressure. The advantage of XP in this situation is that it emphasises this known problem, which is often ignored. The gate models are too inflexible to accommodate software development in any form. The solution must be to make the models more tailorable. This is particularly valid in the ABB case where the project management model is highly connected to the technical process model.

For instance at the time a complete requirement specification is

0

100

200

300

400

500

600

700

800

2000 2001 2002 2003 2004Year

#Def

ect R

epo

rts

Total submitted

Closed

Remaining Open


196

required by the gate model, the XP team will be sitting on a functional basic version of the system. How can this best be accommodated in a project management model?

6.2.3 Documents

The traditional document structures used within the gate based project management models need revision to fit an XP project. In the studied cases, informal documents were created as the projects progressed. These contained much of the information management needed, but this was not realized by management. Formal documents required for gates were fed into the XP planning game as tasks and treated like any other work in the project. Here again, XP emphasizes a known problem in the development process. Documents are inflexible and difficult to keep updated. Suggestions for improving this are using more flexible document formats and automatically generated documents. In one of the cases, design specifications were successfully generated automatically at any time that management asked for it.

6.3 Communication

6.3.1 Level of Abstraction

In the agile teams, communication takes place at a lower level of abstraction than in the non-agile teams, especially between the customer (project/product manager) and developers. This results in that detailed functional issues are identified earlier than in non-agile teams and can be resolved earlier. Also technical problems are raised earlier leading to a risk that these hinder progress. On the other hand we observed frustration from higher-level management about having to bother about so technical issues so early.

This problem is solved in one of the study objects through the direct involvement of the technical product management in the planning game of the team to resolve these issues as quickly as possible. Automatic and manual tests were continuously used to promote technical communication and coordination with good results.

A significant advantage of the early communication is that the customer needs are made very clear early in the project.

197

6.3.2 Team Isolation

The XP team is considered more isolated than teams were previously. This results in a separation of concerns between developers and management and breathing space to ‘get on with the tasks at hand’. An enabler for this is an increased reliance on communication through tests.

The increased isolation, however, did on some occasions increase the amount of suspicion and speculation about what was really going on in development. The important customer communication must not be affected by this isolation, which was the case in one of the companies. Ensuring that the customer role is clearly defined, even if several people assume the role, can help this. Also, an increased emphasis on this relationship may help. Especially management seem to be in need of an attitude change here.

6.3.3 Customer

A clear customer role is essential to keep the XP team running smoothly. This must therefore be provided from the project management model. In both our cases the customer interfaced towards the gate model through some form of a requirements document. In the EMW case, the deliverables were sometimes “put on the shelf” which had a negative impact on the progress. In the ABB case, the customer role player - a technical product manager (TPM) - was continuously testing the deliverables, giving feedback, and thus providing a driving force in the development.

The customer role ensures fast, continuous and flexible feedback. The main problem seems to be identifying a suitable person or persons to assume the role. The most important factors appear to be that the customer is active and present within the team planning game and acceptance of continuous releases. Project managers, technical product managers and marketing product managers are examples of suitable customer candidates.

6.3.4 Internal Communication

Communication within the agile development team is perceived to be much better than in traditional teams. This is believed to be due to continuous pair programming, daily stand-up meetings and bi-weekly planning games and deliveries. One important factor affecting communication is the physical office environment. When one of the


198

teams was spread over two floors, communication was severely impeded. Positive factors here include increased competency spreading, and increased project status awareness. The only negative factor reported was that of possible increased team isolation due to this, the results of which are discussed in Section 6.3.2.

6.4 Timing

6.4.1 Iteration Frequency

The XP teams use a much higher iteration frequency compared to the gate model, even where an incremental development process is used. XP iterations are in the magnitude of weeks while the increments in the main projects are in the magnitude of many months. The higher frequency provides much faster feedback both for developers and management. In the studied cases the developers adjusted to this easier than management. The problem with this is shows itself in the form of a mismatch between the information required by the gate model and the activities dictated by the model as well as other teams. Other teams that are dependent on the work performed by the XP team operate at a different iteration frequency and, for instance, will ask for complete requirements document at a specific stage. This must be solved through clearly defined interfaces with the rest of the organisation and gate model.

6.4.2 Good Micro Planning

The XP teams perceive that they have much better plans for the coming weeks compared to working in common teams. The XP teams are totally focused on the work at hand and everybody is certain about what they are responsible for. The problem that became apparent was that management was more concerned with macro planning and status reporting. The solution must be to combine the successful XP micro planning with more traditional macro planning. Another plausible improvement is to try to change management attitudes to the importance of micro planning.

6.5 Values

6.5.1 Flexibility

The XP teams see no problems with continuously changing requirements in contrast to how they worked previously. This flexibility works well

199

both when the requirements are clear and when they are more diffuse. The flexibility supports the extreme work breakdown required for the micro planning and increased incremental delivery.

The only problem associated with this is the adherence to current quality standards that the companies are certified to. Neither of the cases had done any deeper investigation about quality standard compliance, but they delivered the required documentation, which was defined as any development task in the planning game.

6.5.2 “Under Control”

All people involved in the projects have a strong feeling of having control, with the exception of managers. The teams have increased confidence in the code due to the automatic test suites, the incremental delivery in small steps with continuous feedback give more confidence in the functions that are developed and the micro planning gives control of time allocation.

Management perceived that they lost some control, as they did not recognise their usual planning models. Neither did the process model satisfactorily describe the ongoing work. Management also complained that when they asked development to squeeze in new functionality; they were in return asked what they wanted to be replaced, since the developers were so good at micro planning.

6.5.3 Quality

Both engineers and managers consider product quality being higher than when using traditional development methods. The advantage of increased quality showed itself in the fact that there was no big-bang integration phase that previously took several months. This in turn meant that the projects adhered better to the macro plan, even finishing ahead of schedule. Part of this was perceived to be caused by engineering resources being allocated more efficiently to tasks.

6.5.4 Attitude

XP is often introduced bottom up by motivated engineers. This appears to scare management, although several managers express a need for a change in the way they work with software products. The advantages of a bottom up approach include that the engineers feel empowered in their


200

environment and are extremely motivated. The management attitudes must change in order for bottom up process change and responsibility to work. One way of creating such an attitude change is management training. Further, successful pilot projects, like the ones studied, may provide evidence for the gains of the approach.

6.6 Contrasting Comparison In this section we summarize the observations with respect to three contrasting perspectives on the observations in the study; agile vs. non-agile teams, management vs. engineers, and EMW vs. ABB.

Comparing the first contrasting perspectives of agile versus non-agile teams, summarized in Table 7, we can identify that the gate model is in practice not adhered to but this is accepted and worked with in the agile situation and ignored in the non-agile situation. Similarly, the need for documents and information are continuously worked with in the agile case where the cost of creating documents becomes apparent as these are planned and scheduled as regular tasks, making cost/benefit analysis and direct prioritization between different types of activities possible. The focus on the product and technical issues in agile teams means that communication is performed on a lower level of detail than in regular teams. Communication with the world outside the agile team is more limited than the non-agile team. This is positive in that the team can get on with the work at hand undisturbed, but negative in that the team appears isolated from the rest of the organization. Communication issues regarding dependencies arise and must be solved by other means than in the non-agile case. The agile teams, however, perceive the communication within the group as much better than previously in their non-agile teams, partially due to pair-programming and team isolation. The iteration frequency is much higher in the agile teams, even compared to other incremental approaches. This is perceived as effective as it increases feedback and keeps the team on track with both the schedule and the stakeholders’ requirements on the product. The agile teams have no problems handling changes in requirements whereas non-agile teams have to go back to the early stages of the development process and redo a lot of work. In retrospect, the products created by agile teams have been considered to be of significantly better quality than that of non-agile teams.

201

Finding Agile vs. non-agile Gate model adherence Model not adhered to in any case. Agile team emphasises need for

more flexibility.

Documents More informal documents used in agile teams. Document writing planned as normal agile tasks.

Level of abstraction Agile teams communicate on a lower level of abstraction.

Team isolation Agile teams work, for good and for bad, more isolated.

Internal communication The agile teams perceive the internal communication better than in non-agile teams.

Iteration frequency Agile teams have higher iteration frequency, also compared to other incremental approaches.

Flexibility Being flexible to e.g. changing requirements is an integrated part of the agile teams.

Quality The resulting product quality is considered better for agile teams.

Table 7 Summary of observations on the contrasts between agile and non-agile teams

Looking at the second contrasting perspective, management versus engineers, (Table 8) we can identify the following interesting observations. Management dislikes handling detailed level issues. It is believed that the engineers should resolve these, whereas engineers need support to resolve detailed issues early before they turn into bigger issues. Concerning the iteration frequency, it was observed that the engineers adjusted to the higher frequency easier than management. This could be due to that the way that management plans at a high level today is not optimised for a highly iterative way of working making the integration difficult for management. Management expressed concern with the good micro planning at the engineering level, as the engineers were unable to slip things into the schedule without taking something else out. This schedule control is perceived as one of the reasons of the success of the agile teams and, in contrast to the perceptions of management in these cases, should increase control of management as well, if properly understood and managed. Well functioning micro-planning seem to lead to better adherence to the macro-plans management are responsible for. The engineers experience this as an increased control of their situation while management feel that they are loosing control, as they are not adapting to the new situation. This leads to positive attitudes from engineering compared to negative attitudes from management, at least during the course of the projects. As accounted for previously, management is in retrospect interested in the methods due to quality increases and adherence to plans and schedules.


202

Finding Management vs. engineers Level of abstraction Management dislike handling the detailed level issues.

Iteration frequency Engineers adjusted better to the higher frequency than management.

Good micro planning Management finds sometimes, paradoxically, good micro planning as a

problem, since they are concerned with macro planning.

“Under control” Engineers find agile project work “under control”, while managers feel they are loosing control.

Attitude Engineers are positive towards agile methods while managers are sceptical.

Table 8 Summary of observations on the contrasts between management and engineers

The third contrasting perspective, EMW versus ABB, is summarised in Table 9. In general ABB had a more positive approach to the introduction of the methodologies. Support was provided from management in the form of resources for external consulting and time for education. This meant that the management was more involved in the whole initiative. At ABB, however, the gate model is more fixed and closely related to the actual technical work making it harder to accommodate a different methodology of this kind. Neither company managed to accommodate the agile team with complete success; indeed the successes described were achieved without changing the gate models at all from the management’s perspective. For the customer role, ABB appointed this to the TPM, making this role very clear and available to the engineers, while this role was very unclear at EMW. Finding EMW vs. ABB General ABB was more positive towards the agile team than EMW. ABB has a

more flexible company culture than EMW has.

Gate model structure The ABB model structure is more fixed than the EMW structure, but neither company managed to adapt the model to accommodate the agile teams.

Customer ABB appointed a customer role to the TPM, while the customer was less clear in the EMW case.

Table 9 Summary of observations on the contrasts between EMW and ABB

203

7 Summary and Conclusions The results of the study show that it is indeed possible to successfully integrate project management models the stage-gate type with XP projects. This has been achieved through complete isolation of the engineering teams and attempting to make these look as similar to a regular team as possible. Based on the results we identify certain key issues that are believed would make a more effective project management involving agile methods. These include:

• Involving developers early in the product development to quickly identify and eliminate technical issues and clearly outline plausible solutions;

• Adapting the project planning to accommodate for agile micro planning in combination with macro project planning;

• Identifying critical feedback loops and make these as short and fast as possible;

• Striving to make an early version of the actual product as quickly as possible to exemplify solutions and use for communication;

• Using technical tools for technical coordination, such as for example automated testing of technical dependencies;

• Making the customer-developer roles and interactions as clear end effective as possible;

• Working chiefly with management attitudes to accommodate uncertainties due to them not feeling at ease with the new processes.

An observed benefit due to the increased quality of the developed code

was the lack of “big-bang” integration problems making the overall project much more predictable.

The increased control felt both by the customers and developers is also considered to be an advantage in the cases studied. This is in contrast to the contradictive lack of control felt by the management.

In summary there are several interesting benefits of integrating agile teams into traditional stage-gate project management models and we have identified some key issues regarding these. There are also some key issues that must be addressed in order to be successful, some of which have been identified in this paper.


204

8 Acknowledgements The authors would like to thank all the people that were available for interviews at ABB and EMW.

9 References [1] http://agilemanifesto.org last accessed 040629. [2] Vanhanen, J., Kähkönen, T., "Practical Experiences of Agility in the Telecom

Industry", Extreme Programming and Agile Processes in Software Engineering, 4th International Conference, XP 2003, Genova, Italy, May 2003.

[3] Fuqua, A. M., Hammer, J. M., "Embracing Change: An XP Experience

Report", Extreme Programming and Agile Processes in Software Engineering, 4th International Conference, XP 2003, Genova, Italy, May 2003.

[4] Rassmusson, J., "Introducing XP into Greenfield Projects: Lessons Learned",

IEEE Software, May/June 2003, pp. 21-28. [5] Schuh, P.,"Recovery, Redemption, and Extreme Programming", IEEE

Software, November/December 2001, pp. 34-40. [6] Grenning, J., "Launching Extreme Programming at a Process Intensive

Company", IEEE Software, November/December 2001, pp. 27-33. [7] Poole, C., Huisman, J. W., "Using Extreme Programming in a Maintenance

Environment", IEEE Software, November/December 2001, pp. 42-50. [8] Murru, O., Deias, R., Mugheddu, G., "Assessing XP at a European Internet

Company", IEEE Software, May/June 2003, pp. 37-42. [9] Karlström, D., “Introducing Extreme Programming - An Experience Report”,

Proceedings of the third International Conference on eXtreme Programming and Agile Processes in Software Engineering (XP’02), Sardinia, Italy, 2002.

[10] Cooper, R. G., Winning at New Products, 3rd edition, Perseus Publishing,

Cambridge, MA, 2001. [11] Paulk, M. C., Curtis, B., Chrissis, M. B.,Weber, C. V., Capability Maturity


[12] Paulk, M. C., Weber, C. V., Curtis, V., The Capability Maturity

Model:Guidelines for improving the Software Process, Addison Wesley, 1995.

205

[13] Paulk99 M. C. Paulk, “Practices of High Maturity Organizations,” The 11th Software Engineering Process Group (SEPG) Conference, Atlanta, Georgia, 8-11 March 1999.

[14] XPdynabook The IEEE dynabook on Extreme Programming,

http://computer.org/seweb/ Dynabook/Index.htm, homepage last visited 2004-06-15.

[15] Gilb, T., Principles of Software Engineering Management, Addison-Wesley,

1998. [16] Royce, W. Software Project Management—A Unified Approach, Addison-Wesley

Longman, 1998. [17] Wallin, C., Ekdahl. F., Larsson, S., “Integrating Business and Software



process improvement: local opinion and empirical evidence”, Proceedings of the second symposium on empirical software engineering. ISESE 2003.

[19] Kitchenham, B., Pfleeger, S., L., Hoaglin, D., C., El Emam, K., Rosenberg, J.,

”Preliminary Guidelines for Empirical Research in Software Engineering”, IEEE Transactions on Software Engineering, 28(8):721-734, 2002.

[20] Mulder, L., “The importance of a common project management method in the


[21] South Sweden Software Process Improvement Network, www.spin-syd.org, last

accessed 040615 [22] Patton, M., Q., Qualitative Research & Evaluation Methods, Sage Publications;

3rd edition October 2001. [23] Robson, C., Real World Research, Blackwell Publishers, Oxford, 2002. [24] Yin, R., K., “Case Study Research Design and Methods”, Sage Publications,

Beverly Hills, California, 1994. [25] Rosengran, K.E., Arvidsson, P., “Sociologisk metodik ” (in Swedish), Almqvist

& Wiksell, 1992. [26] Lantz, A., Intervjumetodik, (in Swedish) Studentlitteratur, 1993.


206

207

Exploring Agile Influences in Stage-Gate Management of Software Product Development

Karlström, D., “Exploring Agile Influences in Stage-Gate Management of Software Product Development”, submitted to IEEE Transactions on Engineering Management, 2004.

Abstract Management of the development of products involving software is almost always done using a stage gate type of management model. Process improvement at this level in a company using can be performed from many perspectives, such as marketing, product management and engineering, causing confusion in the organisation. This paper presents a qualitative case study performed at Vodafone Group global product development investigating the possibilities of integrating agile concepts in the stage gate product management environment. The agile approach has been shown to be possible to introduce successfully at the engineering level of two different organisations with a similar context in a previous study and the research is performed as a continuation. The study comprises interviews with product mangers, programme managers, marketing managers and engineering managers. We conclude that there are several current issues within Vodafone product development where an agile approach is perceived to be beneficial in the management process by focusing on increased person to person communication, faster feedback, quicker decisions and spending less time on functionality that is not to be developed.

7


208

1 Introduction Software development projects are seldom isolated activities. They often exist as sub-projects in an environment composed of hardware development, marketing, and production planning etc. which all must be managed and coordinated concurrently [1]. Furthermore, software development seldom starts from scratch; legacy software constitutes the core of new products to which new functionality and other types of customer value are added, or legacy software is moved to new platforms. The traditional approach when managing a complex development project situation is to use a project management model (PM) such as Cooper’s Stage-gate model [1], see Figure 1. The PM gives support not only for the communication within the project, but also for decision makers sponsoring the project or acquiring the outcome of the project. The concepts of a PM are not connected to any specific domain; these models are sufficiently generic to be applied to any kind of project. However, PM:s currently tend to have a basic sequential structure, i.e. distinct phases of pre-study, feasibility study, execution and conclusion.

Figure 1 Stage-Gate™ model [1]

Traditional PM:s are compatible with maturity-oriented software process improvement work, using for instance the CMM framework [2]. Both PM:s and the maturity models aim at defining structures, roles and responsibilities within the organisation, but do not intend to actually dictate how the actual technical project work is to be performed. The agile methodologies can be considered a reaction to these maturity-oriented approaches within the software development process field, but we do not believe that the two schools are contradictive. Software projects using agile methods must be able to exist within environments structured

Launch

Discovery


Post launch review

G2 G1

G3 G4 G5



209

using traditional PM. Mature organizations add not only rigour and order to a chaotic environment, but also certain characteristics similar to a successful agile development project [3, 4, 5].

Agile methods have evolved as a new approach to developing software products [6]. The methods have been successfully implemented in small to medium sized projects, and the principles of the agile methods have inspired software developers in wide-ranging application domains [e.g. 7, 8, 9, 10, 11, 12, 13, 14]. The agile methodologies offer down-to-earth approaches to software development that focus on simplifying and improving the software development process, making the customers, the developers and the final product the most important [6]. A summary of the main principles involved in the agile methods as formulated by the agile manifesto is given in Table 1.

1. Individuals and interactions over processes and tools.

2. Working software over comprehensive documentation. 3. Customer collaboration over contract negotiation. 4. Responding to change over following a plan.

Table 1 The agile manifesto [6]

Traditional PM models have already been integrated with more iterative development models in general [15] and the Rational Unified Process (RUP) in particular [16] Recently an example of the possibility of integration with agile methods has been presented [17], although only at an overview level so far. Software projects using agile methods have indeed been executed in traditional PM environments, but the main characteristics of the PM are not originally designed for the management of software projects using agile methods, and their advantages are therefore not utilised in the management of the project. The agile development has in these cases had no effect on the project management and overall management of the project despite demonstrating success on the development level.

In our previous study we have investigated the incorporation of agile (specifically XP) teams into a stage-gate product management models and shown examples of successful projects. Some key issues were identified and these are related to in the discussion section of this paper [Paper 6].

The research presented by Wallin et al. [17] presents the overall concepts and problems involved in combining a project management model with several different software development processes, including extreme programming and the Rational Unified Process. The study however, is lacking in detail of how the interfaces are defined and how the


210

different processes need to be modified. The study does not look at how the stage gate model itself can benefit from agile concepts.

In this paper we investigate the possibilities for introducing agile concepts into the product development process from a programme and product management perspective. An interview study is performed at Vodafone Global product development.

This paper is structured as follows. In Section 2 the context at Vodafone is presented. The study design is presented in Section 3 and study validity in Section 4. The observations made in the study are presented and analysed in Section 5. Finally, a summary is provided together with the conclusions in Section 6.

2 Vodafone Group Global Product Development

The context of the case study is presented according to the recommendations by Kitchenham et al. [19] stating that the experimental context needs the three elements background information, discussion of research hypotheses, and information about related research. The two former are presented below, while the latter is presented in Section 1.

Vodafone Global Group is responsible for developing and providing common products and services in the Vodafone network worldwide. At this time Vodafone has approximately 140 million subscribers in 26 countries. Specifically the global division of the company delivers standard platforms and services that are released to the operating companies (opco) and localised to reflect each respective opco’s individual technology, price plans and so on. As the Vodafone opco network has grown through acquisition of operators, the current generation of technology used in the opcos is not standardised, creating many difficulties in the global development. Future globally coordinated generations of technology, once in place, will reduce this difficulty. The opcos vary considerably in size, both in terms of the number of subscribers and the number of local product development staff.

The study was started due to interest shown for agile methods at several levels of the organisation at around the same time. The company is continuously adapting its processes to optimise their work and there is a strong spirit of continuous improvement present. The process improvement work has been affected much by the merger of several different development companies and centres. The main product development for Vodafone Global is handled largely by what used to be a

211

portal development company, based in London with agencies throughout Europe. Vodafone acquired this company in 2000. This has lead to a huge change for the people involved with product management and development in that the company that they are working in now is suddenly very much larger. These changes have occurred under a relatively short period of time and it seems that the company is still getting used to the situation. Some parts of Vodafone Group have previously defined a stage-gate process known as their global product delivery process (referred to as GPDP in this paper). This process is outlined in figure 2.

Programme management has recently defined a product proposition delivery process describing how the company intends to go from consumer product concepts to release, launch and localisation in the various Vodafone operator countries. The process is intended to coordinate all product work into the release schedule. The proposition process is in practice slightly less formal than the GPDP process although the underlying concept of a staged product management process is similar as are the general purposes of stages involved. The GPDP is a much more complete process model as it is intended to manage the development of a single product, while the proposition process manages an aggregation of products. These processes are being coordinated in the current process improvement work at Vodafone. The proposition delivery process overview is illustrated in Figure 3. The process starts 16 months before a release, every six months. The process is also terminated every six months. This means that there are always three releases active at one time. The gates are approved at various management levels in the company in both processes; the highest approving authority is the Consumer Marketing Council, which approves the proposition, the revised proposition and is involved in the final review of the release. These approval points are shown as rhombi in Figures 2 and 3. Note that these approvals can be located within one of the defined stages as in the Deploy stage of the proposition delivery process.

Figure 2 Vodafone proposition delivery process overview

Proposition Concept Development

Evaluation Deploy

Review Launch

1 1a 2 2a 3


212

Figure 3 Vodafone global product development process (GPDP) overview

Development of the global products and platforms is currently distributed over three locations, London, Düsseldorf and California. Each centre is responsible for a certain product or group of products.

The overall product architecture divides the system into platform, enablers and customer facing features. Another large area concerned with in the global division is the acquisition of global content applicable for all opcos. The product is extremely market driven and the competition for subscribers is fierce. The pace of development in the company is therefore high.

3 Study Design The case study performed is of the applied research type, implying that the purpose is to understand the nature of and the problems present within a certain context in society [20]. The study takes a broad view within the area of exploring agile influences in gate models. The units of analysis are individual members at different positions in the organisation, the teams that these people are organised in and the organisation as a whole, including its process descriptions. The study is focused on the experiences of the subjects in their day-to-day activities and how they perceive that specific agile concepts may affect their work. With agile concepts we mean the application of the agile manifesto [6] shown in Figure 1, the principles behind the Agile Manifesto as well as how the key issues identified in our previous study translate to the higher level studied at Vodafone [Paper 6].

Proposition

Ideas Scope Proof of Concept Evaluate

…

Launch

Develop Retire Review …

a b c d

e f

213

3.1 Scope The study aims to follow up on the results of a previous study investigating the effects of introducing an agile software development process within a traditional stage gate model style project management process [Paper 6]. The study aims to investigate the effects of introducing agile concepts at the product management level. The study specifically aims at investigating the perceived effects of introducing these concepts and does not study actual introduction of the concepts.

3.2 Subjects The subject sampling strategy was to interview relevant people that would have key insight into the product development at Vodafone. This involved, programme management, product managers, technical leaders and process owners. A few of the selected subjects, 3 out of 14, were unfortunately unavailable at the time of the study. All participants took part in the interviews voluntarily and all data were treated strictly confidentially after the interview occasions. In total, 11 interviews were performed.

3.3 Research Strategy We approach the research using a qualitative research strategy [20, 21, 22 23]. By the same reasoning as in our previous study [Paper 6] using Patton’s classification of a similar research area [20], we elect to study the software engineering environment as a complete dynamic system.

3.4 Research Methods The main source of information in this investigation is the interviews performed with the subjects at Vodafone in London and Düsseldorf. The interviews were performed as open-ended interviews [24], more in the form of a discussion using the interview instrument as a guide of subjects available to discuss. Interesting facts mentioned were followed up immediately with each subject instead of strictly following the instrument. The interview instrument was constructed by one researcher and was adapted as the interviews progressed. The interview instrument consisted of the following main sections:


214

• Introduction. • Personal, group and company background. • Product process experiences. • Specific thoughts on integrating agile concepts into the product

process. • General informing about the agile concepts and spontaneous

reactions. • Ending.

The interviews were approximately one hour in length with one

researcher interviewing one subject. The same researcher performed all the interviews with all the subjects. Some notes were taken during the interview but the main form of documentation was the sound recording intended for complete transcription as a part of the analysis.

3.5 Analysis This section contains a complete description and discussion of the analysis steps taken in the study. An overview of these steps can be found in Figure 4.

During the course of the analysis information takes on several forms at different levels of abstraction. The subjects act in and observe the reality within their organisation. Their reflections and observations on the area of research are discussed in the open-ended interviews and recorded as an audio file. These recordings are then fully transcribed and interesting quotes are identified. Next, quotes are coded and grouped to support each other and finally individual results are identified from these groups. The grouping was conducted using plain tables in a word processor.

The different forms that the information takes in this investigation are numbered 1 to 7 in Figure 4. The first form is the actual event in the study context. The second form is the subject’s perception of the event, and so on as shown in Table 3. Each change in form is due to some kind of treatment and we refer to this change in form as a transition. The transitions are denoted α through ζ in Figure 4 and the subsequent tables. Each transition introduces additional validity threats to the results depending on the treatment performed in the transition. Treatments and the validity threats introduced are summarised in Table 4. Countermeasures to these validity threats are addressed in Section 5. These countermeasures can also be seen in part throughout the study design, as the one of the goals of a good study design is to minimise the

215

effects of threats. The analysis performed in the study is of the inductive type, implying

that patterns, themes and categories of analysis come from the data itself in contrast to being imposed prior to the data collection, as is the case in deductive analysis. All approaches used in inductive analysis according to Patton have been considered in the analysis.

Figure 4 Forms and transitions in the research analysis process

Form# Description 1 Actual events within case context 2 Subjects’ perceptions based on their observations

and experiences 3 Sound recording of interview 4 Transcription of recording 5 Coded quotes from transcribed material 6 Grouped quotes 7 Independent results

Table 3 Information forms in the research process

1

2

α

β 3γ 4δ 5ε 6ζ 7


216

Trans-ition

From form

To form Treatment description

Threats

α 1 2 Native observations Misconceptions, misunderstanding, lack of objectivity

β 2 3 Interview Misunderstandings, omissions,

γ 3 4 Transcription Audibility of source

δ 4 5 Identifying quotes and coding

Misunderstanding, misinterpretation of intent

ε 5 6 Grouping Incorrect grouping due to effects of previous threats

ζ 6 7 Identifying specific results

Incorrect results due to effects of previous threats

Table 4 Transitions between information forms in the research process

In addition to the interview data, other information sources in the study include archival analysis of internal documents, including process description documents as well as working documents and informal information gained through conversations outside the interviews.

3.6 Validation The preliminary results from each individual subject were presented back to that subject via email. The subjects were also asked to comment on the accuracy of the results and any further changes in that area since the interview. This was done via email as meetings were very hard to schedule. Finally, the results were presented in a workshop format and the participants’ opinions were collected and any misinterpretations were resolved. Final conclusions were based on all the gathered information.

The interview study was performed over a period of several months and several days were spent observing the general working environment at Vodafone, attending meetings with different purposes etc.

4 Study Validity The qualitative strategy selected involves using the researcher himself as the instrument for the research. A short discussion of the effects of the particular researchers involved is therefore appropriate [21].

The author has been involved in industrial product development since 1993 and has been performing research within the area for over four years. The main focus of research has been software process and software process improvement through involving developers more in how they work and improve their work. He has followed the agile movement as it has developed from a small community to a global phenomenon.

217

Methodologically, he has used both empirical experiments and qualitative case studies extensively in his research.

To reduce the threats to validity of the study, countermeasures are taken in the study design and during the whole study. Further, an analysis of threats to the validity and corresponding strategies are presented below, using the Lincoln and Guba model [21], see Table 6. Specific countermeasures to the validity threats brought up in Section 5 are summarised in Table 5.

Trans-ition

Threats Countermeasures

α - Misconceptions - Misunderstanding - Lack of objectivity

- Use several sources with similar perspectives. - Identify inconsistencies between interviews.

β - Misunderstandings - Omissions

- Feedback to subjects before final results

γ - Audibility of source - Use high quality microphone and recording equipment.

δ - Misunderstanding - Misinterpretation of intent

- Feedback to subjects before final results.

ε - Incorrect grouping due to effects of previous threats

- Previous countermeasures - Researcher has experience of previous studies with similar methodology

ζ - Incorrect results due to effects of previous threats

- Previous countermeasures

Table 5 Countermeasures to validity threats in the study

Threat to validity Strategy Reactivity Researcher

bias Respondent bias

Prolonged involvement

Reduces threat

Increases threat

Reduces threat

Triangulation Reduces threat

Reduces threat

Reduces threat

Peer debriefing No effect Reduces threat

No effect

Member checking

Reduces threat

Reduces threat

Reduces threat

Negative case analysis

No effect Reduces threat

No effect

Audit trail No effect Reduces threat

No effect

Table 6 Strategies for dealing with threats to validity [21]


218

The researcher has been involved long-term with the case study companies for over six months. This is assumed to provide a balance between the researcher bias threats and the respondent and reactivity threats.

Triangulation is attained through data triangulation by using multiple data sources: interviews, observations, archival and informal. Theory triangulation is achieved by using subjects with multiple perspectives.

Peer debriefing is not used to any larger extent in the study, except in the very late stage of the analysis. In order to be effective, a peer from a different research angle should be used, and this was not available. Member checking on the other hand is used, by feeding back analysis results to the respondents.

An extensive approach to audit trail is used in the study. The analysis is conducted with rigour. Each statement of interest in the transcribed material is encoded and summarized in tables. The results are the then derived from the tables, still keeping the full traceability back to each statement in the interviews, which the results are based on.

The large number of projects currently being coordinated using stage-gate models [1] implies that, at least from the perspective of using such a model, transferability should be high, although a case study always is coloured by its specific context.

In summary, all reasonable countermeasures to validity threats have been implemented in the study. It is therefore contended by the authors that the validity of the study is sound.

5 Results This section presents the results identified through the case study using the analysis procedure described in the previous section. We present the observation summary structured into five major groups in the subsections 5.1.1-5.1.5 with the following categorisation:

• Process • Product • Planning and Scoping • Communication • Engineering

It can be difficult to absolutely categorise the observations as many observations are related to several of the areas. These are mentioned in the most significant area, but analysed as belonging to several.

219

5.1 Observation Summary

5.1.1 Process The current process improvement work is performed on a high level within the company and on a high abstraction level. The next step is said to be to specify the models on a lower level, making them even more detailed. It is unclear to the subjects involved whether the processes denote what is a future goal for the organisation, what is actually being done or what is mandatory to do. This appears to be causing some confusion in different areas of the company. Similarly, the purpose of the process work per se is unclear in many cases. A common attitude is that there is no point in having a process if it is not used or enforced.

There are currently a number of different processes, the two main being the GPDP and the product proposition process and it is unclear what actually is the official process. In some cases GPDP is mentioned, but it is often referred to as too elaborate. When discussing different process models using official process documents there was quite a low degree of recognition of the different process models, but the general concepts were very well known. The only parts of the company that knew one process model well were those that have been using the GPDP for a long time. Other parts of the company have inherited GPDP as they have been integrated into the Vodafone Group. The GPDP is very different from the style of working that derives from the culture before the acquisition referred to in Section 2. Local teams in London prefer the flexibility of the old style but can also see the need for some more discipline in the new larger organisation. GPDP however is considered too formal. One suggestion made is to allow the type of deliverable to influence the process used. For a new platform a more formal process might be used compared to when rapidly developing customer features.

Top down process improvement work is being performed from both programme management and marketing management perspectives. These initiatives are being coordinated to a certain extent, but it is difficult in such a large organization. There is, for instance, product process improvement work being performed from a marketing perspective that does not take engineering practice into account. Input for the initiatives is being obtained through workshops with key individuals from the organisation. The process improvement work is chiefly being performed from the perspective of each process owner’s perspective. Currently, bottom up process improvement initiatives are not facilitated from higher levels in the company.


220

There is a general desire to move towards iterative development in the local development teams and some parts of the product management. In contrast to this, there appears to be significant resistance in remote development teams. The pressure of the day-to-day schedule to meet deadlines also hampers moving towards an iterative process. The change is considered to be large and quite difficult and no significant resources are available to make the change. Many of the subjects speculated on the technical difficulties of building the products iteratively, whereas engineering subjects thinks this is possible but will need some time to sort out. It is perceived difficult to identify smaller steps to make the process change easier to introduce gradually.

Other hindrances to process change mentioned by the subjects include: The physical distance between development teams in remote locations, different processes in different parts of the company, different languages, terminologies, cultures and roles in different countries. Especially the role terminologies were mentioned as hindering the process work as, for example, a product manager role can have very different responsibilities in different parts of the organization. In one part he would have a technical responsibility that in another would be handled by the technical product owner.

The global organization is going through very rapid change. Different parts of the organizations are from different companies with different styles of working. Parts that are used to a less formal way of working do not see the need for the formal process models and, visa versa, parts using the more formal processes are unable to work less formally. Subjects used to the way of working before the acquisition experienced that the product management was easier. The distance between product managers and engineering was less, there were more personal relationships. There is also a realisation that the new, larger, organisation requires more disciplined processes.

One strategy for incremental process improvement suggested in the interviews was using project retrospectives at the end of each release in order to evaluate the product and processes in use.

5.1.2 Product The product architecture reflects the nature of the developed functionality. It is thought that that the processes used to develop the product do not reflect this. The platform or enabling parts of the system require longer horizons and stability whereas customer services and features need to be implemented quickly. Different parts of the product

221

also require different means of communication. Customer features require fast feedback from customer representatives, whereas platform parts of the product require more internal coordination and input.

5.1.3 Planning and Scoping The scoping of features to be developed has been a huge problem in the past releases. Everyone mentions this as a problem, but also that they are working to improve this in future releases. It is very difficult to motivate removal of features early on in the process and hence many features are therefore specified in too much detail despite that they will not be developed. Engineering mentioned that by putting in early engineering resources, development resources may be reduced without affecting projected revenue by defining the feature so as to make it easier to engineer. This does take time away from engineering features currently in the development phase of another release though.

There is always too much to be developed compared to the time available. The planning appears to be optimised to provide a long-term stable perspective on the product features. This is especially important for the parts of the product that involve the phone terminals as these have a much longer product cycle than the software services. The long-term perspective in the planning leads to a lot of re-planning work as soon as a change is introduced.

Development teams currently prioritise between working with maintenance and new feature development by themselves. Variations in how this is prioritised affects the amount of development capacity for new features and this is not apparent to programme management. System maintenance is often of lower priority than new features, degrading non-functional properties of the system until the situation becomes critical and a whole release needs to be devoted to improving non-functional properties.

It is always the last phases that are compromised in the process, even due to slippage in earlier phases. This means that testing and product localisation is often performed in less time than planned for and under unnecessary time pressure. This can even be due to slippage occurring very early in the process, involving features that might not even make it into the end product. An example situation was described in one interview where an early, low-level, detailed change request took several months to be approved by the change control board and therefore affected the whole development schedule.


222

5.1.4 Communication There is an awareness of the importance of communication throughout the organization. Between the technical product owners and the product managers there are examples of good interpersonal communication. Currently mock-ups and demos are used early on in the process as communication tools, but it is thought that this could be done even more. There is no focus on using iterative development to produce a rough working actual product early, although this is thought to be a good idea and possible to integrate with the future iterative process model. Product managers try to stay away from the technical details, except when these are central to decision making and planning.

Due to the different perspectives of the marketing, engineering and product management teams there is a conflict of interests between future strategy (marketing) versus current and past development (engineering) with the product managers in-between. It is for instance difficult to make the marketing group interested in evaluating the past release as they have already moved their interest two releases further on.

Communication between the global product managers and the representatives from the operating countries has shown to be more effective when a personal trust has been established on location. Much of the communication is performed via tele- and video-conferencing, which is difficult if you have never met the person on the other end. The need for early feedback from the opcos has been identified by almost everyone and is currently being realized through a lead opco program. This implies that one or two opcos will start using the release earlier in order to give earlier feedback. Requirements are collected from the opcos, as are the best solutions developed on a local level. Workshops are held by the global product managers where best practices in the OpCos are identified and spread. This can also be done through the file sharing team rooms used at Vodafone. Despite these different efforts, it is not absolutely clear in all cases what the role of the global group is compared to the OpCos.

Communication with the end customer is mostly done by marketing and third party marketing companies. There is no direct contact between engineering and the end customer.

Traceability is being improved in the product process. In earlier versions of the product, there was no way of tracing the proposed concepts through the development process. It would be especially useful to be able to compare the proposed concepts to what is actually delivered.

The global development is currently spread over three locations in United Kingdom, USA and Germany. It will soon be extended to

223

additional sites. This physical distance between the groups has severely impacted development in the past. The USA team is almost considered as a third party software vendor by some.

There is a recognised need to improve the process for the parts of the product that are being developed by third party software vendors, especially the communication between Vodafone and the software vendor during the development. These parts of the product are specified, contracted and the contractor goes on his way to complete the product. There is hardly any communication between the start of the development and the delivery point, neither regarding features nor progress.

5.1.5 Engineering Parts of the engineering group have identified some best practices that they think should be spread to other parts of the product. These include using sandbox environments to be able to test run interfacing products early as well as product building environments. Sandboxing is thought to enable an improvement in the technical coordination between different product parts early in the development.

5.2 Analysis The observations made in the case study give a picture of a very volatile software product environment with many distributed teams and extreme time to market pressure. The organisation delivers products on time, but can be severely pushed to make deadlines. The possibility and effect of introducing some agile concepts on the product management level was discussed in the interview series. For some of the major issues identified in Vodafone product development we present a corresponding application of the agile concepts. The issues and corresponding agile strategies identified are summarised in Table 7.

5.2.1 Agile Improvement Concepts

The first issue identified is that the processes and process improvement work is difficult to access for the organisation. The more detailed the processes are specified, the more complex and inaccessible they become. They also become more difficult to change. An agile approach to working with processes at this, high, level is to work with values and simple practices more in order to make the process work simpler and easier to spread.


224

Vodafone Issue Agile Concept application 1 Process Improvement

Inaccessible Values & practices versus process detailed process definitions

2 Communication terminology discrepancies

Increase personal contacts in order to identify and resolve these.

3 Changes in planned functionality

Iterative short term planning versus long term Gantt chart. (In the parts of the product where this is suitable, not. for the terminals)

4 Scoping into development window is accurate

Continuous development versus development window. Create a rough initial working framework and iterate from there. Use time boxing instead of feature boxing.

5 Overload of development Make sure that only requirements that can be developed are committed to in the process. Make sure development is proceeding as fast as possible without being disturbed by undelivered features.

6 Feedback/Control Reduce time spent waiting for decisions and feedback as much as possible.

7 Communication mediums Functioning product versus demos and mock-ups over documentation

8 Communication with different customers

Involve at least one opco throughout the entire development of the product as a customer. Involve end user groups to evaluate early versions of customer features. Make sure development meets end user in person.

9 Physical distance Build interpersonal relations and trust though frequent meetings in person. Ensure that key people spend time on important locations.

10 Engineering best practice Make sure that engineering practices support the communication between different parts of the product. E.g. sandboxing provides quick feedback, architectures affect iteration rates and so on. Ensure spread of best practices.

11 Different perspectives and interests in product community

Increase time spent face to face versus communicating via documents. Generate continuous interest by defining customer-deliverer roles clearly.

12 Organisational Change Organisational change involves many people making large changes, let it take time and make the changes in small steps.

Table 7 Issues identified at Vodafone and the corresponding agile solution

225

Communication terminology discrepancies and misunderstandings might be reduced by committing resources to identify and resolve these through person-to-person contacts throughout the organisation. This type of terminology misunderstanding is easily overlooked when working with documents.

Changes in the functionality in the planned releases of the product could be more easily accommodated using an agile approach to the planning. Instead of working with large, predetermined work packages a more continuously prioritised list can be used. This in combination with an iterative approach would enable greater flexibility in parts of the product where this is possible, such as customer related features.

As the development phase is consistently overloaded it is necessary to reduce the number of concepts that are committed to development. It is also important to spend as little resources as possible on those concepts that are not developed in the end. The prioritising focus must be on maximising the business objectives in the long run.

The rate of feedback on completed or proposed work appears central to making the process more efficient. This can be stretched to include controlling feedback decisions on what is to be developed. All time spent waiting on feedback implies a waste of resources. A lot of the communication in Vodafone is through workshops and face-to-face meetings, only important decision material is put into documents, such as the requirements specification. Demos and mock-ups are used as early communication mediums and proof of concept. A further evolvement of this that works well with the iterative approach to development is to use an early working version of the product as a communication media on the product management level.

Communication with the customer could benefit from further improvement through involving one or two lead opcos throughout the entire product management process. Contact with end users can also serve to provide faster feedback on end-user functionality. A direct meeting between end-users and development at some point would ensure direct feedback. Much of the communication problems at Vodafone are due to the physical distance between the distributed teams. This needs to be addressed through the building of firm person-to-person relations by frequent meetings in person. It seems important for key individuals to spend time at the remote locations in order to understand the local issues. The communication issue can also be addressed using technological means. The Vodafone team room file sharing system does this and could be extended to further forms of communication such as wiki webs,


226

discussion forums and chat applications. Engineering communication will also affect product management. If good engineering coordination is achieved technically there will be less issues for product management to coordinate. Making sandbox environments available for other products to run tests against and automatically testing dependent interfaces are examples of technical coordination methods.

The process work at Vodafone is performed from many different perspectives and interests, taking into account different parts of the operation. The most important issue to resolve is roles and responsibilities within the organisation and ensure that the processes support this. By defining multiple customer-deliverer relationships it is thought that clarity might be increased. The changes currently going on and proposed in the future for the Vodafone organisation are very large. Changing the working practice of this many people will take time and will need to be done in several smaller steps so as not to conflict with day to day operations, unless significant resources in both time and money are provided to make a larger change.

One example of a smaller step for process change in the organisation is making the initial process change continuous instead of pulsed and coordinated with a specific release. This could be a first step to making the development phase continuous instead of a discrete development window, which is a much larger change. A continuous concept elicitation phase reflects reality, i.e. new concepts can be thought of at any time, however it is considered an advantage to be able to use the deadline to motivate completion of concept descriptions.

5.2.2 Comparison to Earlier Study

The observations made in our previous study on integrating XP teams into a stage gate model [Paper 6] are compared to the observations made in this study. How the key issues, identified from the engineering perspective, translate to the product management at Vodafone is presented in Table 8. As can be seen in the table many of the key issues either translate to the higher level or support the product management level from the engineering level. The bottom-up process improvement at the engineering level in the previous study appears in contrast to the more top-down approach used at Vodafone. It seems, however, that the approaches are not contradictory, rather different areas are addressed by each approach and it seems that both approaches might be needed in order to cover both low-

227

level engineering process improvement as well as organisational level process improvement. Key Issues on Engineering

Level Translate to Product Management Context

1 Involving developers early in the product development to quickly identify and eliminate technical issues and clearly outline plausible solutions.

Ensure good quality communication between stakeholders early in the product development process. Engineering, marketing, product mangers, opcos and end customers all need to be involved at an early stage in order to quickly identify the most important features and ensure technical feasibility.

2 Adapting the project planning to accommodate for agile micro planning in combination with macro project planning.

Use agile micro planning in combination with macro project planning. Ensure that different planning horizons are used for different parts of the product requiring different planning horizon, such as terminal hardware.

3 Identifying critical feedback loops and making these as short and fast as possible.

Same. The larger context at the product management level shows several areas where feedback rate could be improved such as control decisions and product feedback.

4 Striving to make an early version of the actual product as quickly as possible to exemplify solutions and use for communication.

This is perceived to be beneficial on PM level. Mock-ups, demos or technical spikes can in some cases replace an actual product at an early stage.

5 Using technical tools for technical coordination, such as for example automated testing of technical dependencies.

Not applicable. Coordination must be handled at the engineering level if it is to be handled automatically. On the product management level coordination must be done through discussions and documents.

6 Making the customer-developer roles and interactions as clear and effective as possible;

Define this type of relationship in several steps from engineering to project management to marketing and direct customer. Involve all parties concurrently in order to give feedback on the decisions made.

7 Working chiefly with management attitudes to accommodate uncertainties due to them not feeling at ease with the new processes.

Working with attitudes throughout the organisation where uncertainties are apparent. This involves creating positive process improvement work on all levels of the company.

Table 8 Translation of the key issues on the engineering level to the product management context


228

6 Summary and Conclusions The goal of the study has been to explore the possibilities of utilising agile influences in product development at Vodafone, not to create a complete agile product management process model. A stage-gate process seems unavoidable at the product management level of the company and will most probably consist of the same general stages. It seems, however, that agile concepts can provide important general guidelines for working on this level of the company, both in day-to-day work and in the process improvement work.

A key question in parts of Vodafone Group product development today appears to be how to keep working in the spirit of the previous small company but plan and manage it with the discipline needed in a larger company. The company needs to allow and support the different attributes of product parts as varied as terminals, software platforms and software customer features and coordinate this into a complete functioning release. The application of the agile concepts presented in the paper might act as a guide for doing this.

7 Acknowledgements The author would like to thank the people that were available for interviews at Vodafone and Prof. Per Runeson for his comments on the paper.

229

8 References [1] Cooper, R. G., Winning at New Products, 3rd edition, Perseus Publishing,

Cambridge, MA, 2001. [2] Paulk, M. C., Curtis, B., Chrissis, M. B.,Weber, C. V., Capability Maturity


[3] Paulk, M. C., Weber, C. V., Curtis, V., The Capability Maturity

Model:Guidelines for improving the Software Process, Addison Wesley, 1995. [4] Paulk, M. C., “Practices of High Maturity Organizations,” The 11th Software


[5] XPdynabook The IEEE dynabook on Extreme Programming,

http://computer.org/seweb/Dynabook/Index.htm, homepage last visited 2004-06-15.

[6] http://agilemanifesto.org last accessed 040629. [7] Vanhanen, J., Kähkönen, T., "Practical Experiences of Agility in the Telecom

Industry", Extreme Programming and Agile Processes in Software Engineering, 4th International Conference, XP 2003, Genova, Italy, May 2003.

[8] Fuqua, A. M., Hammer, J. M., "Embracing Change: An XP Experience

Report", Extreme Programming and Agile Processes in Software Engineering, 4th International Conference, XP 2003, Genova, Italy, May 2003.

[9] Rassmusson, J., "Introducing XP into Greenfield Projects: Lessons Learned",

IEEE Software, May/June 2003, pp. 21-28. [10] Schuh, P., "Recovery, Redemption, and Extreme Programming", IEEE

Software, November/December 2001, pp. 34-40. [11] Grenning, J., "Launching Extreme Programming at a Process Intensive

Company", IEEE Software, November/December 2001, pp. 27-33. [12] Poole, C., Huisman, J. W., "Using Extreme Programming in a Maintenance

Environment", IEEE Software, November/December 2001, pp. 42-50. [13] Murru, O., Deias, R., Mugheddu, G., "Assessing XP at a European Internet

Company", IEEE Software, May/June 2003, pp. 37-42.


230

[14] Karlström, D., “Introducing Extreme Programming - An Experience Report”, Proceedings of the third International Conference on eXtreme Programming and Agile Processes in Software Engineering (XP’02), Sardinia, Italy, 2002.

[15] Gilb, T., Principles of Software Engineering Management, Addison-Wesley,

1998. [16] Royce, W., Software Project Management—A Unified Approach, Addison-

Wesley Longman, 1998. [17] Wallin, C., Ekdahl F., Larsson, S., “Integrating Business and Software



process improvement: local opinion and empirical evidence”, Proceedings of the second symposium on empirical software engineering. ISESE 2003.

[19] Kitchenham, B., Pfleeger, S., L., Hoaglin, D., C., El Emam, K., Rosenberg, J.,

”Preliminary Guidelines for Empirical Research in Software Engineering”, IEEE Transactions on Software Engineering, 28(8):721-734, 2002.

[20] Patton, M., Q., Qualitative Research & Evaluation Methods, Sage Publications;

3rd edition October 2001. [21] Robson, C., Real World Research, Blackwell Publishers, Oxford, 2002. [22] Yin, R., K., “Case Study Research Design and Methods”, Sage Publications,

Beverly Hills, California, 1994. [23] Rosengran, K. E., Arvidsson, P., “Sociologisk metodik ” (in Swedish), Almqvist

& Wiksell, 1992. [24] Lantz, A., Intervjumetodik, (in Swedish) Studentlitteratur, 1993.


232

Reports on Communication Systems

101. On Overload Control of SPC-systems Ulf Körner, Bengt Wallström, and Christian Nyberg, 1989. CODEN: LUTEDX/TETS- -7133- -SE+80P

102. Two Short Papers on Overload Control of Switching Nodes Christian Nyberg, Ulf Körner, and Bengt Wallström, 1990. ISRN LUTEDX/TETS- -1010- -SE+32P

103. Priorities in Circuit Switched Networks Åke Arvidsson, Ph.D. thesis, 1990. ISRN LUTEDX/TETS- -1011- -SE+282P

104. Estimations of Software Fault Content for Telecommunication Systems Bo Lennselius, Lic. thesis, 1990. ISRN LUTEDX/TETS- -1012- -SE+76P

105. Reusability of Software in Telecommunication Systems Anders Sixtensson, Lic. thesis, 1990. ISRN LUTEDX/TETS- -1013- -SE+90P

106. Software Reliability and Performance Modelling for Telecommunication Systems Claes Wohlin, Ph.D. thesis, 1991. ISRN LUTEDX/TETS- -1014- -SE+288P

107. Service Protection and Overflow in Circuit Switched Networks Lars Reneby, Ph.D. thesis, 1991. ISRN LUTEDX/TETS- -1015- -SE+200P

108. Queueing Models of the Window Flow Control Mechanism Lars Falk, Lic. thesis, 1991. ISRN LUTEDX/TETS- -1016- -SE+78P

109. On Efficiency and Optimality in Overload Control of SPC Systems Tobias Rydén, Lic. thesis, 1991. ISRN LUTEDX/TETS- -1017- -SE+48P

110. Enhancements of Communication Resources Johan M. Karlsson, Ph.D. thesis, 1992. ISRN LUTEDX/TETS- -1018- -SE+132P

233

111. On Overload Control in Telecommunication Systems Christian Nyberg, Ph.D. thesis, 1992. ISRN LUTEDX/TETS- -1019- -SE+140P

112. Black Box Specification Language for Software Systems Henrik Cosmo, Lic. thesis, 1994. ISRN LUTEDX/TETS- -1020- -SE+104P

113. Queueing Models of Window Flow Control and DQDB Analysis Lars Falk, Ph.D. thesis, 1995. ISRN LUTEDX/TETS- -1021- -SE+145P

114. End to End Transport Protocols over ATM Thomas Holmström, Lic. thesis, 1995. ISRN LUTEDX/TETS- -1022- -SE+76P

115. An Efficient Analysis of Service Interactions in Telecommunications Kristoffer Kimbler, Lic. thesis, 1995. ISRN LUTEDX/TETS- -1023- -SE+90P

116. Usage Specifications for Certification of Software Reliability Per Runeson, Lic. thesis, May 1996. ISRN LUTEDX/TETS- -1024- -SE+136P

117. Achieving an Early Software Reliability Estimate Anders Wesslén, Lic. thesis, May 1996. ISRN LUTEDX/TETS- -1025- -SE+142P

118. On Overload Control in Intelligent Networks Maria Kihl, Lic. thesis, June 1996. ISRN LUTEDX/TETS- -1026- -SE+80P

119. Overload Control in Distributed-Memory Systems Ulf Ahlfors, Lic. thesis, June 1996. ISRN LUTEDX/TETS- -1027- -SE+120P

120. Hierarchical Use Case Modelling for Requirements Engineering Björn Regnell, Lic. thesis, September 1996. ISRN LUTEDX/TETS- -1028- -SE+178P

121. Performance Analysis and Optimization via Simulation Anders Svensson, Ph.D. thesis, September 1996. ISRN LUTEDX/TETS- -1029- -SE+96P

122. On Network Oriented Overload Control in Intelligent Networks Lars Angelin, Lic. thesis, October 1996. ISRN LUTEDX/TETS- -1030- -SE+130P

123. Network Oriented Load Control in Intelligent Networks Based on Optimal Decisions Stefan Pettersson, Lic. thesis, October 1996. ISRN LUTEDX/TETS- -1031- -SE+128P

124. Impact Analysis in Software Process Improvement Martin Höst, Lic. thesis, December 1996. ISRN LUTEDX/TETS- -1032- -SE+140P


234

125. Towards Local Certifiability in Software Design Peter Molin, Lic. thesis, February 1997. ISRN LUTEDX/TETS- -1033- -SE+132P

126. Models for Estimation of Software Faults and Failures in Inspection and Test Per Runeson, Ph.D. thesis, January 1998. ISRN LUTEDX/TETS- -1034- -SE+222P

127. Reactive Congestion Control in ATM Networks Per Johansson, Lic. thesis, January 1998. ISRN LUTEDX/TETS- -1035- -SE+138P

128. Switch Performance and Mobility Aspects in ATM Networks Daniel Søbirk, Lic. thesis, June 1998. ISRN LUTEDX/TETS- -1036- -SE+91P

129. VPC Management in ATM Networks Sven-Olof Larsson, Lic. thesis, June 1998. ISRN LUTEDX/TETS- -1037- -SE+65P

130. On TCP/IP Traffic Modeling Pär Karlsson, Lic. thesis, February 1999. ISRN LUTEDX/TETS- -1038- -SE+94P

131. Overload Control Strategies for Distributed Communication Networks Maria Kihl, Ph.D. thesis, March 1999. ISRN LUTEDX/TETS- -1039- -SE+158P

132. Requirements Engineering with Use Cases – a Basis for Software Development Björn Regnell, Ph.D. thesis, April 1999. ISRN LUTEDX/TETS- -1040- -SE+225P

133. Utilisation of Historical Data for Controlling and Improving Software Development Magnus C. Ohlsson, Lic. thesis, May 1999. ISRN LUTEDX/TETS- -1041- -SE+146P

134. Early Evaluation of Software Process Change Proposals Martin Höst, Ph.D. thesis, June 1999. ISRN LUTEDX/TETS- -1042- -SE+193P

135. Improving Software Quality through Understanding and Early Estimations Anders Wesslén, Ph.D. thesis, June 1999. ISRN LUTEDX/TETS- -1043- -SE+242P

136. Performance Analysis of Bluetooth Niklas Johansson, Lic. thesis, March 2000. ISRN LUTEDX/TETS- -1044- -SE+76P

137. Controlling Software Quality through Inspections and Fault Content Estimations Thomas Thelin, Lic. thesis, May 2000 ISRN LUTEDX/TETS- -1045- -SE+146P

138. On Fault Content Estimations Applied to Software Inspections and Testing Håkan Petersson, Lic. thesis, May 2000. ISRN LUTEDX/TETS- -1046- -SE+144P

235

139. Modeling and Evaluation of Internet Applications Ajit K. Jena, Lic. thesis, June 2000. ISRN LUTEDX/TETS- -1047- -SE+121P

140. Dynamic traffic Control in Multiservice Networks – Applications of Decision Models Ulf Ahlfors, Ph.D. thesis, October 2000. ISRN LUTEDX/TETS- -1048- -SE+183P

141. ATM Networks Performance – Charging and Wireless Protocols Torgny Holmberg, Lic. thesis, October 2000. ISRN LUTEDX/TETS- -1049- -SE+104P

142. Improving Product Quality through Effective Validation Methods Tomas Berling, Lic. thesis, December 2000. ISRN LUTEDX/TETS- -1050- -SE+136P

143. Controlling Fault-Prone Components for Software Evolution Magnus C. Ohlsson, Ph.D. thesis, June 2001. ISRN LUTEDX/TETS- -1051- -SE+218P

144. Performance of Distributed Information Systems Niklas Widell, Lic. thesis, February 2002. ISRN LUTEDX/TETS- -1052- -SE+78P

145. Quality Improvement in Software Platform Development Enrico Johansson, Lic. thesis, April 2002. ISRN LUTEDX/TETS- -1053- -SE+112P

146. Elicitation and Management of User Requirements in Market-Driven Software Development Johan Natt och Dag, Lic. thesis, June 2002. ISRN LUTEDX/TETS- -1054- -SE+158P

147. Supporting Software Inspections through Fault Content Estimation and Effectiveness Analysis Håkan Petersson, Ph.D. thesis, September 2002. ISRN LUTEDX/TETS- -1055- -SE+237P

148. Empirical Evaluations of Usage-Based Reading and Fault Content Estimation for Software Inspections Thomas Thelin, Ph.D. thesis, September 2002. ISRN LUTEDX/TETS- -1056- -SE+210P

149. Software Information Management in Requirements and Test Documentation Thomas Olsson, Lic. thesis, October 2002. ISRN LUTEDX/TETS-1057- -SE+122P

150. Increasing Involvement in Software Process Improvement Daniel Karlström, Lic. thesis, November 2002. ISRN LUTEDX/TETS- -1058- -SE+125P

151. Changes to Processes and Architectures; Suggested, Implemented and Analyzed from a Project viewpoint Josef Nedstam, Lic. thesis, November 2002. ISRN LUTEDX/TETS- -1059- -SE+124P


236

152. Resource Management in Cellular Networks -Handover Prioritization and Load Balancing Procedures RolandZan der, Lic. thesis, March 2003. ISRN LUTEDX/TETS- -1060- -SE+120P

153. On Optimisation of Fair and Robust Backbone Networks Pål Nilsson, Lic. thesis, October 2003. ISRN LUTEDX/TETS- -1061- -SE+116P

154. Exploring the Software Verification and Validation Process with Focus on Efficient Fault Detection Carina Andersson, Lic. thesis, November 2003. ISRN LUTEDX/TETS- -1062- -SE+134P

155. Improving Requirements Selection Quality in Market-Driven Software Development Lena Karlsson, Lic. thesis, November 2003. ISRN LUTEDX/TETS- -1063- -SE+132P

156. Fair Scheduling and Resource Allocation in Packet Based Radio Access Networks Torgny Holmberg, Ph.D. thesis, November 2003. ISRN LUTEDX/TETS- -1064- -SE+187P

157. Increasing Product Quality by Verification and Validation Improvements in an Industrial Setting Tomas Berling, Ph.D. thesis, December 2003. ISRN LUTEDX/TETS- -1065- -SE+208P

158. Some Topics in Web Performance Analysis Jianhua Cao, Lic. thesis, June 2004. ISRN LUTEDX/TETS- -1066- -SE+99P

159. Overload Control and Performance Evaluation in a Parlay/OSA Environment Jens K. Andersson, Lic. thesis, August 2004. ISRN LUTEDX/TETS- -1067- -SE+100P

160. Performance Modeling and Control of Web Servers Mikael Andersson, Lic. thesis, September 2004. ISRN LUTEDX/TETS- -1068- -SE+105P

161. Integrating Management and Engineering Processes in Software Product Development Daniel Karlström, Ph.D. thesis, December 2004. ISRN LUTEDX/TETS- -1069- -SE+230P