IEEE TRANSACTIONS ON SOFTWARE ENGINEERING 1

Variability in Software Systems – A Systematic Literature Review

Matthias Galster, Danny Weyns, Dan Tofan, Bartosz Michalik, and Paris Avgeriou

Abstract—CONTEXT: Variability (i.e., the ability of software systems or artifacts to be adjusted for different contexts) became a key property of many systems. OBJECTIVE: We analyze existing research on variability in software systems. We investigate variability handling in major software engineering phases (e.g., requirements engineering, architecting). METHOD: We performed a systematic literature review. A manual search covered 13 premium software engineering journals and 18 premium conferences, resulting in 15430 papers searched and 196 papers considered for data collection. To improve reliability and to increase reproducibility, we complemented the manual search with a targeted automated search. RESULTS: Software quality attributes have not received much attention in the context of variability. Variability is studied in all software engineering phases, but testing is underrepresented. Data to motivate the applicability of current approaches are often insufficient; research designs are vaguely described. CONCLUSIONS: Based on our findings we propose dimensions of variability in software engineering. This empirically grounded classification provides a step towards a unifying, integrated perspective of variability in software systems, spanning across disparate or loosely coupled research themes in the software engineering community. Finally, we provide recommendations to bridge the gap between research and practice and point to opportunities for future research.

Index Terms—Variability, systematic review, software engineering.

1 INTRODUCTION

xxxx-xxxx/0x/$xx.00 © 200x IEEE

————————————————•

•

•

•

•

2 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

1.1 Variability – An Introduction

1.2 Lack of Existing Reviews

1.3 Goals and Contributions

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 3

1.4 Paper Structure

2 RESEARCH METHOD

2.1 Research Questions

Fig. 1. Systematic literature review process (adapted from [10]).

Fig. 2. Process to develop review protocol (adapted from [29]).

4 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

2.2 Search Strategy

2.3 Scope of Search and Sources Searched

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 5

TABLE 2 Searched Conferences

TABLE 1 Searched Journals

6 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

2.4 Inclusion and Exclusion Criteria

•

•

•

•

•

•

•

•

2.5 Search Process

2.6 Quality Criteria

Fig. 3. Search process.

TABLE 3 Questions to Assess Study Quality

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 7

2.7 Data Collection

2.8 Data Analysis

2.9 Protocol Review

TABLE 4 Data Collection Form

8 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

3 RESULTS 3.1 Demographic Data

Fig. 4. Number of papers per journal.

Fig. 6. Studies per software engineering area of publication venue.

Fig. 5. Number of papers per conference.

Fig. 7. Studies per year.

5

0

5

11

25

2

10

42

7

0

36

0

5

10

15

20

25

30

ASEJ

ESE

IEEE

SW IST

JSS

REJ

SCP

SMR

P

SoSy

M

SPE

STVR TS

E

TOSE

M

Num

ber o

f pap

ers

Journal

General SE85%

Requirements engineering

3%

Architecture / design

8%

Implementation / integration

0%

Testing / verification

0%

Maintenance4%

9

13

20

67

58

40 0

8

14 3 4

36

6

0

5

10

15

20

25

30

35

40

ASE

CAi

SE

EASE

ECO

OP

ECSA FS

E

GPC

E

ICSE

ICSM

ICST

ISST

A

MO

DEL

S

OO

PSLA

QoS

A

RE

SLE

SPLC

WIC

SA

Num

ber o

f pap

ers

Conference

0

5

10

15

20

25

30

35

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

Num

ber o

f pap

ers

Year

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 9

3.2 RQ1: What Methods to Handle Variability in Software Systems Exist?

3.2.1 RQ1.1: What Types of Variability do these Methods Handle?

3.2.2 RQ1.2: What Activities in the Software Development Process are Affected by these Methods?

3.2.3 RQ1.3: What Runtime and Design Time Quality Attributes are Addressed by these Methods?

•

•

•

•

•

•

•

•

• • •

• •

TABLE 5 Software Engineering Activities Addressed by Studies

10 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

3.3 RQ2: What is the Evidence that Motivates the Adoption of Existing Methods?

Fig. 8. Frequency analysis of quality scores for each question.

Fig. 9. Distribution of total quality scores.

TABLE 6 Studies with a total Quality Score of 6 and 5.5

0

20

40

60

80

100

120

140

160

0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1

Q1 Q2 Q3 Q4 Q5 Q6

Freq

uenc

y of

sco

res

Non-empirical Empirical

0

5

10

15

20

25

30

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

Freq

uenc

y of

tota

l sco

res

Total score

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 11

3.4 RQ3: What are the Limitations of Existing Methods?

•

•

•

•

Fig. 10. Distribution of evidence levels.

TABLE 8 Studies with Highest Evidence Level

TABLE 7 Studies with Citation Count > 100

Level 0, 19, 10%

Level 1, 106, 54%

Level 2, 20, 10%

Level 3, 26, 13%

Level 4, 25, 13%

Level 5, 0, 0%

12 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

4 DISCUSSION OF RESULTS 4.1 Application of Methods to Handle Variability in

Software Engineering

4.2 Applicability of Results in Practice

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 13

4.3 Dimensions of Variability

4.3.1 Definition of Dimensions

4.3.2 Overview of the Dimensions

Fig. 11. Relevance versus rigor.

11 34 35

7

40

2

19

7

21

4

11

5

-1

0

1

2

-1 0 1 2 3

Rele

vanc

e

Rigor

14 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

TABLE 9 Dimensions of Variability

4.3.3 Evaluation of Dimensions

4.3.4 Usefulness of the Classification

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 15

TABLE 10 Dimensions of Variability – Type Cluster

16 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

TABLE 11 Dimensions of Variability – Mechanism Cluster

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 17

4.4 Variability Handling in Software Engineering versus Variability Management in SPL

5 DEVIATIONS FROM REVIEW GUIDELINES, THREATS TO VALIDITY AND ASSESSMENT OF REVIEW

5.1 Deviations from Systematic Literature Review Guidelines

18 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

5.2 Threats to Validity

5.3 Assessment of Review

6 CONCLUSIONS

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 19

20 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

APPENDIX

TABLE 12 Sources of Searched Venues

TABLE 14 Number of Conference Papers Searched, and Papers left

after each Search Stage

TABLE 13 Number of Journal Papers Searched, and Papers Left

after each Search Stage

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 21

TABLE 15 Studies Included in the Final Review

22 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 23

24 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

ACKNOWLEDGMENT

REFERENCES

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 25

26 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

Matthias Galster is a Lecturer at the University of Canterbury, Christchurch, New Zealand. Danny Weyns is a professor at Linnaeus University, Sweden.

Dan Tofan is a doctoral student at the University of Groningen, Netherlands. Previously, he worked six years in the software industry.

Bartosz Michalik completed his PhD at the Katholieke Universiteit Leuven, Belgium. He currently works in the software industry. Paris Avgeriou is professor and head of the Software Engineering and Architecture group at the University of Groningen, Netherlands.