variability: from software product lines to self-adaptive...

Variability: From Software Product Lines to Self-Adaptive Systems Danny Weyns www.lnu.se/adaptwise

Dynamic Software Product Lines, DSPL Florence, September 15, 2014

Observation

•  Variability •  “The ability of software systems or artifacts to be adjusted for

different contexts”* •  Mainly been studied in the context of software product lines

•  Variability beyond traditional software product lines •  Runtime service composition •  Self-adaptation •  Dynamic software product lines…

* van Gurp, Bosch, Svahnberg, On the Notion of variability in software product lines, WICSA 2011

Research Objective

•  Addressing variability in a more holistic manner

•  Analyze and characterize handling of variability in software systems from researchers viewpoint

- Identify trends in research on variability handling in software systems in general - Assess evidence of variability research, identify open problems and new opportunities - Obtain a more rigorous understanding of variability in software engineering

Research Method

•  Systematic Literature Review*

GALSTER ET AL.: VARIABILITY IN SOFTWARE SYSTEMS – A SYSTEMATIC LITERATURE REVIEW 3

!""#$%&'$(&'"$)*&'&+,*$-)+#."-$)*/)$/0012$31"/'$'"-"/'3*$(")*&#-4$0&&'$"5.#"!3"$3&+1#$("/!$)*/)$(&'"$#/)/$/6&+)$"7.-).!,$(")*&#-$ %&'$ */!#1.!,$ 5/'./6.1.)2$ -*&+1#$ 6"$ 3&181"3)"#4$'"-"/'3*$)'"!#-$/!#$&0"!$0'&61"(-$3&+1#$0&.!)$)&$"73.).!,$ '"-"/'3*$ &00&')+!.)."-9$ :+')*"'(&'";$ )*"$ '"5."<$<.11$ *"10$ 0&-.).&!$ !"<$ '"-"/'3*$ /3).5.)."-=$ :&'$ !"<$ '"8-"/'3*;$<"$!""#$)&$ .#"!).%2$ )*"$-)/)"8&%8)*"8/')$/!#$(/>"$31"/'$ <*"'"$ !"<$ '"-"/'3*$ %.)-$ .!)&$ )*"$ 3+''"!)$ 6&#2$ &%$>!&<1"#,"9$ :.!/112;$ 6/-"#$ &!$ &+'$ %.!#.!,-$ <"$ 0'&0&-"$#.("!-.&!-$ %&'$ 5/'./6.1.)2$ )&$ 0'&5.#"$ /$ (&'"$ '.,&'&+-$+!#"'-)/!#.!,$&%$5/'./6.1.)2$.!$-&%)</'"$"!,.!""'.!,9$?*.-$31/--.%.3/).&!$ %&'$ 5/'./6.1.)2$ .!$ #.%%"'"!)$ #.("!-.&!-$ 3/08)+'"-$>"2$%/3")-$&%$5/'./6.1.)29!

1.4 Paper Structure @!$ A"3).&!$ B$ <"$ #.-3+--$ )*"$ '"-"/'3*$ (")*&#9$ A"3).&!$ C$0'"-"!)-$ #"(&,'/0*.3$ .!%&'(/).&!$ /!#$ )*"$ '"-+1)-$ <"$&6)/.!"#$ %'&($ /!/12D.!,$ )*"$ #/)/$ "7)'/3)"#$ %'&($ '"85."<"#$-)+#."-9$E$#.-3+--.&!$/!#$ .!)"'0'")/).&!$&%$ )*&-"$'"-+1)-$ )/>"-$ 01/3"$ .!$ A"3).&!$ F9$ @!$ A"3).&!$ G;$<"$ #.-3+--$1.(.)/).&!-$&%$&+'$'"5."<9$A"3).&!$H$3&!31+#"-$)*.-$0/0"'9$

2 RESEARCH METHOD ?*"$ -2-)"(/).3$ 1.)"'/)+'"$ '"5."<$ (")*&#$ IGJ$ .-$ /$ <"118#"%.!"#$ /00'&/3*$ )&$ .#"!).%2;$ "5/1+/)"$ /!#$ .!)"'0'")$ /11$'"1"5/!)$-)+#."-$'",/'#.!,$/$0/').3+1/'$'"-"/'3*$K+"-).&!;$)&0.3$ /'"/$ &'$ 0*"!&("!&!$ &%$ .!)"'"-)$ ICLJ9$ ?*.-$(")*&#$</-$3*&-"!$6"3/+-"$<"$</!)"#$)&$,")$/$%/.';$3'"#.61"$/!#$+!6./-"#$"5/1+/).&!$&%$/00'&/3*"-$%&'$*/!#1.!,$5/'./6.18.)2$ .!$ -&%)</'"$ -2-)"(-9$ ?*"$ -2-)"(/).3$ '"5."<$</-$0"'8%&'("#$/-$-*&<!$.!$:.,9$M$N/#/0)"#$%'&($IMLJO9$

E!$.(0&')/!)$-)"0$.-$)*"$#"5"1&0("!)$&%$/$'"5."<$0'&8)&3&1$)&$"!-+'"$'.,&'$/!#$'"0"/)/6.1.)29$@!$&'#"'$)&$'"#+3"$'"-"/'3*"'-P$ 6./-$ <*"!$ 0"'%&'(.!,$ )*.-$ '"5."<;$ <"$/#/0)"#$ )*"$ 0'&3"--$ %&'$ #"5"1&0.!,$ /$ '"5."<$ 0'&)&3&1$0'&0&-"#$ 62$E1.$ ")$ /19$ N:.,9$ BO$ IBQJ9$E%)"'$ .#"!).%2.!,$ )*"$'"-"/'3*$K+"-).&!-;$<"$#"3.#"#$)&$+-"$/$(/!+/1$-"/'3*$/-$&+'$-"/'3*$-)'/)",2$/!#$#"%.!"#$/$-"/'3*$-3&0"9$E-$0/')$&%$)*.-$ -)"0;$ <"$ 0"'%&'("#$ 0.1&)$ -"/'3*"-9$ ?*"!;$ <"$ #"5"18&0"#$ -)+#2$ .!31+-.&!$/!#$"731+-.&!$ 3'.)"'./;$ /!#$#"%.!"#$

)*"$ -"/'3*$ 0'&3"--9$ E1-&;$ <"$ 0'&0&-"#$ &+'$ -)'/)",2$ %&'$/--"--.!,$)*"$K+/1.)2$&%$-)+#."-$)*/)$<"$3&!-.#"'"#$.!$)*"$'"5."<9$R"7);$6/-"#$&!$&+'$'"-"/'3*$K+"-).&!-;$<"$.#"!).8%."#$<*/)$#/)/$"1"("!)-$)&$"7)'/3)$%'&($"/3*$-)+#2$%&+!#$.!$ )*"$ -"/'3*9$ E$ #/)/$ 3&11"3).&!$ %&'($ </-$ 0.1&)"#$ <.)*$-"5"'/1$0/0"'-$%&+!#$.!$)*"$0.1&)$-"/'3*9$:.!/112;$)&$3'&--83*"3>$)*"$-"1"3).&!$&%$)*"$0'.(/'2$-)+#."-$.#"!).%."#$.!$)*"$(/!+/1$ -"/'3*$ <"$ #"%.!"#$ -"/'3*$ )"'(-$ %&'$ /$ )/',")"#$1.(.)"#$/+)&(/)"#$ -"/'3*9$E1-&;$<"$#"%.!"#$&+'$ -)'/)",2$)&$ /!/12D"$ "7)'/3)"#$#/)/$ /!#$ )&$0'"-"!)$ )*"$ '"-+1)-9$S"8)/.1-$/'"$0'"-"!)"#$.!$)*"$'"(/.!#"'$&%$)*.-$-"3).&!9$

:.5"$'"-"/'3*"'-$#.-)'.6+)"#$/3'&--$%&+'$1&3/).&!-$<"'"$.!5&15"#$.!$)*"$'"5."<9$T&((+!.3/).&!$)&&>$01/3"$&!1.!"$/!#$.!$-"5"'/1$&!8-.)"$("").!,-9$?&&1-$+-"#$%&'$)*"$'"5."<$.!31+#"#$U!#R&)"$VF;$S'&06&7$/!#$W.3'&-&%)$U73"19$

2.1 Research Questions X"$ %.'-)$ %&'(+1/)"#$ )*"$ '"5."<$ ,&/1$ )*'&+,*$ Y&/18Z+"-).&!8W")'.3$ 0"'-0"3).5"-$ N0+'0&-";$ .--+";$ &6["3);$5."<0&.!)O$ICMJ=$$"#$%&'()! !"!#$%&'!"(')*!+!),&+-%&4$*''#()$ *!"(#-".'/0'1!+-!2-#-,$4$+,-(./)$ -"'3/0,4!+&'3$3,&534$01(2%&13/)! 0+/5'!'+&3&!+)*&+63'7/-",'/0'1-&49$$\/-"#$&!$)*"$,&/1;$<"$#"'.5"#$'"-"/'3*$K+"-).&!-=$

456)$ X*/)$(")*&#-$ )&$ */!#1"$ 5/'./6.1.)2$ .!$ -&%)</'"$-2-)"(-$"7.-)]$

45676)$ X*/)$)20"-$&%$5/'./6.1.)2$#&$)*"-"$(")*&#-$*/!8#1"]$

45678)' X*/)$ /3).5.)."-$ .!$ )*"$ -&%)</'"$ #"5"1&0("!)$0'&3"--$/'"$/%%"3)"#$62$)*"-"$(")*&#-]$

45679)$ X*/)$'+!).("$/!#$#"-.,!$).("$K+/1.)2$/))'.6+)"-$/'"$/##'"--"#$62$)*"-"$(")*&#-]$

458)$ X*/)$.-$)*"$"5.#"!3"$)*/)$(&).5/)"-$)*"$/#&0).&!$&%$"7.-).!,$(")*&#-]$

459)$ X*/)$/'"$)*"$1.(.)/).&!-$&%$)*"$"7.-).!,$(")*&#-]$^ZM$.-$(&).5/)"#$62$)*"$!""#$)&$#"-3'.6"$)*"$-)/)"8&%8

)*"8/')$&%$*&<$"7.-).!,$(")*&#-$*/!#1"$5/'./6.1.)29$^ZM$.-$'"%.!"#$ .!)&$ )*'""$ -+68K+"-).&!-$ )*/)$ /.($ )&$ /!-<"'$<*")*"'$5/'./6.1.)2$&33+'-$/!#$.-$)'"/)"#$/)$#"-.,!$).("$&'$'+!).("$ N.9"9;$ )*"$ )20"-$ &%$ 5/'./6.1.)2O;$ <*.3*$ -&%)</'"$

$Fig. 1. Systematic literature review process (adapted from [10]).

$Fig. 2. Process to develop review protocol (adapted from [29]).

* Galster, Weyns, Tofan, Michalik, Avgeriou, Variability in software systems, a systematic literature review, IEEE Transactions on Software Engineering, 40(3):282-306, 2014

Research Questions

•  1. What methods to handle variability in software systems exist? •  - Design time vs runtime variability? •  - Quality attributes addressed? •  - Activities in software development affected?

•  2. What is the evidence that motivates the adoption of existing methods?

•  3. What are the limitations of existing methods?

Search Strategy

•  Manual search of articles published at quality venues

since 2000 (18 conferences, 13 journals)

•  Complemented with targeted automatic search •  Pair-wise review of batches of papers

Data Collection


!"#$$%&'()!*+,-.$*!'*-)+/$#*$-,"*01$+!(')2*$(!"$#*-+*34$+52*3!'*+'6$*$7!$)!5*'#*3)'58*94* (),.1:();*3!'*+'6$*$7!$)!5*/$*-<'(:$:*)$;.$,!();*+!-!$6$)!+*/"$#$*-1!"'#+*&#'<(:%$:* ').4* .(6(!$:* ()='#6-!(')* !'* -)+/$#* !"$* -++$++6$)!*01$+!(')+* >?@A8* B-,"* 01-.(!4* -++$++6$)!* 01$+!(')* /-+*-)+/$#$:*C4*-++(;)();*-*)16$#(,-.*<-.1$*DE*F*34$+52*G*F*3)'52* -):*G8H* F* 3!'* +'6$* $7!$)!5I8* J"$)2* -* 01-.(!4* +,'#$*/-+* ;(<$)* !'* -* +!1:4* C4* +166();* 1&* !"$* +,'#$+* ='#* -..*01$+!(')+*='#*-*+!1:4*>?@A8*K+*,-)*C$*+$$)*()*J-C.$*L2*!"$+$*+,'#$+* #$=$#* !'* !"$* 01-.(!4* '=* !"$* #$&'#!();* '=* -* +!1:42*#-!"$#* !"-)* (!+* -,!1-.* 01-.(!48* M'!$* !"-!* 01-.(!4* ,#(!$#(-*/$#$*)'!*1+$:*='#*=(.!$#();*&1#&'+$+*!'*(),.1:$*'#*$7,.1:$*&#(6-#4*+!1:($+*()*!"$*#$<($/*C1!*!'*-++$++*!"$*01-.(!4*'=*-..* (),.1:$:* +!1:($+8* N1#!"$#6'#$2* !"$* 01-.(!4* ,#(!$#(-*/$#$*-&&.($:*!'*-..*+!1:($+*(),.1:$:*()*!"$*#$<($/2*#-!"$#*!"-)*').4*!'*&-&$#+*!"-!*#$&'#!*$6&(#(,-.*+!1:($+8**

2.7 Data Collection N'#* $-,"* +!1:42* !"$* ()='#6-!(')* +"'/)* ()* J-C.$* O* /-+*,'..$,!$:8*P!1:($+*/$#$* #$-:* ()*:$!-(.* !'*$7!#-,!* !"$*:-!-8*J"#$$*#$+$-#,"$#+*#$-:*!"$*+$.$,!$:*+!1:($+*()*&-#-..$.8*N'#*$-,"* &-&$#2* :-!-* /-+* $7!#-,!$:* C4* ')$* #$+$-#,"$#* -):*,"$,Q$:* -;-()+!* !"$* &-&$#* C4* -)'!"$#* #$+$-#,"$#8* R(+-%;#$$6$)!+* /$#$* #$+'.<$:* C4* :(+,1++(')+* C$!/$$)* #$%+$-#,"$#+*'#*C4*,')+1.!();*-)*-::(!(')-.*#$+$-#,"$#8*

R-!-* ='#* NE* !'* NS* /$#$* ,'..$,!$:* ='#* :',16$)!-!(')*&1#&'+$+*!'*Q$$&*6$!-%()='#6-!(')*'=*&-&$#+8*NT*#$,'#:+*!"$*,(!-!(')*,'1)!*='#*$-,"*&-&$#2*C-+$:*')*U'';.$*P,"'.-#*D-+* '=* V-4* ?GELI8* NW* #$,'#:+* !"$* 01-.(!4* +,'#$* ='#* $-,"*&-&$#2*C-+$:*')*,#(!$#(-*()!#':1,$:*()*P$,!(')*?8S8*N@*-):*NEG*#$,'#:*01-.(!4*-!!#(C1!$+8*X$*#$,'#:$:*01-.(!4*-!!#(C%1!$+* !"-!* -#$* -::#$++$:* ()* +!1:($+2* '#* C4* &#'&'+$:* !$,"%)(01$+8*NEE*#$,'#:+*(=*!"$#$*(+*-)4*!''.*+1&&'#!8*J''.*+1&%&'#!* ,'1.:*C$* -1!'6-!(,2* +$6(%-1!'6-!(,*'#*6-)1-.8*J"$*$<(:$),$*.$<$.*DNE?I*'=*-*+!1:4*(+*,#(!(,-.*='#*#$+$-#,"$#+*!'*(:$)!(=4*)$/* !'&(,+* ='#* $6&(#(,-.* +!1:($+2* -):* ='#* &#-,!(%!(')$#+* !'* -++$++* !"$*6-!1#(!4* '=* -* &-#!(,1.-#*6$!"':* '#*!''.8* Y(!,"$)"-6* &#'&'+$:* =(<$* .$<$.+* '=* +!1:4* :$+(;)*>LGA8*X$*1+$:*-*#$<(+$:*,.-++(=(,-!(')*!'*6-Q$*!"$*-++$++%6$)!*6'#$*&#-,!(,-.* >?HA8*N#'6*/$-Q$+!* !'* +!#');$+!2* !"$*

,.-++(=(,-!(')*(+*-+* ='..'/+Z*G*F*)'*$<(:$),$[*E*F*$<(:$),$*=#'6*:$6')+!#-!(')* '#* !'4* $7-6&.$+[* ?* F* $<(:$),$* =#'6*$7&$#!* '&()(')+* '#* 'C+$#<-!(')+[* L* F* $<(:$),$* =#'6* -,-%:$6(,* +!1:($+* D$8;82* ,')!#'..$:* .-C* $7&$#(6$)!+I[* O* F* $<(%:$),$*=#'6*():1+!#(-.*+!1:($+*D$8;82*,-1+-.*,-+$*+!1:($+I[*H*F*$<(:$),$*=#'6*():1+!#(-.*&#-,!(,$8*K,,'#:();*!'*K.<$+*$!*-.8* >?HA2* ():1+!#(-.* &#-,!(,$* ():(,-!$+* !"-!* -* 6$!"':* "-+*-.#$-:4*C$$)*-&&#'<$:*-):*-:'&!$:*C4*():1+!#(-.*'#;-)(%\-!(')+8* J"1+2* &#-,!(,$* +"'/+* ,')<(),();* $<(:$),$* !"-!*+'6$!"();*/'#Q+*-):*(+*!"1+*#-)Q$:*+!#');$+!8*K,!(<(!($+*-::#$++$:* DNELI* -#$* #$01(#$6$)!+* $);()$$#();2* -#,"(!$,%!1#$* ]* :$+(;)2* (6&.$6$)!-!(')2* !$+!();* -):* <$#(=(,-!(')2*-):*6-()!$)-),$8 ^(6(!-!(')+*DNEOI*-):*!4&$+*'=*<-#(-C(.%(!4* DNEHI* /$#$* #$,'#:$:* -+* !$7!8* J4&$+* '=* <-#(-C(.(!4* -#$*:4)-6(,* <-#(-C(.(!4* D(8$82* <-#(-C(.(!4* #$+'.<$:* -!* #1)!(6$I*-):* +!-!(,* <-#(-C(.(!4* D(8$82* <-#(-C(.(!4* #$+'.<$:* :1#();*:$+(;)*]*:$<$.'&6$)!* !(6$I8*NES*#$,'#:+* (=*-*+!1:4* (+*-)*$6&(#(,-.*&-&$#*'#*)'!8*

2.8 Data Analysis R-!-* =#'6* +!1:($+* /$#$* ,'..-!$:* !'* -)+/$#* !"$* #$+$-#,"*01$+!(')+8* V'+!* &-&$#+* ()* '1#* #$<($/* -#$* ;#'1):$:* ()*01-.(!-!(<$* #$+$-#,"8* K+* -#;1$:* C4* R4C-* -):* R();+'4#*>O?A2*6$!-%-)-.4+(+*6(;"!*)'!*C$*+1(!-C.$*='#*+4)!"$+(\();*01-.(!-!(<$*:-!-8*J"1+2*!"$*:-!-*/-+*6-)1-..4*#$<($/$:*()*!"$*+&#$-:+"$$!+*/"$#$* (!*/-+*+!'#$:8*K+*='1):*C4*'!"$#*#$+$-#,"$#+2*!-C1.-!();*!"$*:-!-*/-+*1+$=1.*:1#();*-;;#$%;-!(')*>EGA8*_)*&-#!+*'=*!"$*:-!-*/$*1+$:*:$+,#(&!(<$*+!-%!(+!(,+* -):* =#$01$),4* -)-.4+(+8* N#$01$),4* -)-.4+(+* "-+*C$$)*1+$:*C4*+4+!$6-!(,*#$<($/+*/"(,"*:$-.*/(!"*01-.(!-%!(<$*:-!-*>?GA8*

2.9 Protocol Review J"$*&#'!','.*/-+*#$<($/$:*C4*-..*#$+$-#,"$#+*()<'.<$:*()*!"$*#$<($/*-+*/$..*-+*C4*$7!$#)-.*#$+$-#,"$#+*)'!*()<'.<$:*()* !"$* #$<($/8* `$<(+(')+* !'* !"$* &#'!','.* /$#$* 6-:$* -,%,'#:();.4*D$8;82*,.-#(=(,-!(')*'=*#$+$-#,"*01$+!(')+*-):*!"$*:-!-*,'..$,!(')*='#62*#$=()$6$)!*'=*+$-#,"*<$)1$+I8*V'#$%'<$#2*!"$*+4+!$6-!(,*#$<($/*/-+*<-.(:-!$:*-+*='..'/+Z*E8 `$+$-#,"$#+*1+$:*-* +1C+$!*'=* #$+'1#,$+* !'* !$+!* !"$*

+$-#,"*&#',$++8*a#'C.$6+*()*#$&.(,-!();*!"$*&#',$++*/$#$*(:$)!(=($:*-):*!"$*&#',$++*#$<(+$:8*

?8 U-&+*()*+$-#,"*<$)1$+*/$#$*(:$)!(=($:*-):*<$)1$+*,"-);$:*!'*(),.1:$*#$.$<-)!*<$)1$+8*

L8 J"$*#$.(-C(.(!4*'=*:-!-*$7!#-,!(')*/-+* !$+!$:8*K*#$%+$-#,"$#*/-+*;(<$)*-*+$!*'=*&-&$#+*-):*-+Q$:*!'*=(..*()*!"$*:-!-*$7!#-,!(')*='#68*9-+$:*')*!"$*=$$:C-,Q*#$<(+(')+*/$#$*6-:$*!'*!"$*&#'!','.8*

R1#();*!"$*#$<($/*/$*:$<(-!$:*=#'6*!"$*'#(;()-.*&#'%!','.* -!* +'6$* &'()!+8* b)(!(-..4* /$* &.-))$:* !'* #$,'#:* !"$*!4&$*'=*+!1:4* ='#*$-,"*&-&$#* D$8;82*$6&(#(,-.*+!1:42* !"$'%#$!(,-.* +!1:4I2* (=* &-&$#+* (),.1:$* -* :(+,1++(')* ')* (6&-,!*-++$++6$)!*'=*<-#(-C(.(!42*'#*(=*!"$#$*(+*-)4*+'#!*'=*=$-!1#$*"-):.();* 1+$:8* c'/$<$#2* (!* !1#)$:* '1!* !"-!* !"(+* ()='#%6-!(')*/-+*:(==(,1.!* !'*'C!-()[* ()*&-#!(,1.-#2*:(==$#$)!(-!%();* !4&$+*'=* +!1:($+* !1#)$:*'1!* !'*C$*)'!*&'++(C.$8*J"1+2*!"$* ,'##$+&'):();* :-!-* =($.:+* /$#$* #$6'<$:* =#'6* !"$*:-!-*='#68*

TABLE 4 Data Collection Form

!" #$%&'" (%)%*+,-"./%)0$12"NE* K1!"'#D+I* )]-*N?* d$-#* )]-*NL* J(!.$* )]-*NO* e$)1$*Df'1#)-.*'#*,')=$#$),$I* )]-*NH* Y$4/'#:+* )]-*NS* KC+!#-,!* )]-*NT* g(!-!(')*,'1)!* `h?*NW* h1-.(!4*+,'#$* `h?*N@* `1)!(6$*01-.(!4*-!!#(C1!$+* `hE8L*NEG* R$+(;)*!(6$*01-.(!4*-!!#(C1!$+* `hE8L*NEE* ^$<$.*'=*!''.*+1&&'#!* `hL*NE?* B<(:$),$*.$<$.* `h?*NEL* K,!(<(!($+*-::#$++$:*C4*-&&#'-,"* `hE8?*NEO* ^(6(!-!(')+* `hL*NEH* J4&$+*'=*<-#(-C(.(!4* `hE8E*NES* J4&$*'=*+!1:4* `h?**

Main Results

Demographic data Design time vs runtime variability Engineering activities addressed Level of evidence Reported limitations

Demographic Data

Manually searched 15430 papers Resulted in 196 papers for data collection

8 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

!

3 RESULTS 3.1 Demographic Data "#!$%&'%(()! *#%+,-#.!/0123!4%4#+*5!"#! (667#.!%8! 9::/!;6'+&%(! 4%4#+*! <=%>(#! /2! ?&! %44#&.?@A! %&.! 912:! ,6&B#+C#&,#!4%4#+*!<=%>(#!/1!?&!%44#&.?@A5!=-#!&'$>#+!6B!4%4#+*!%B8#+!D8%E#!F!G%*!2:3!</HI!;6'+&%(!4%4#+*!J!FFF!,6&B#+#&,#!4%4#+*A5! K?L#! 4%4#+*! %(*6! +#*'(8#.! B+6$! 8-#! 8%+E#8#.! %'C86$%8#.!*#%+,-5!M'+?&E!.%8%!,6((#,8?6&N!$6+#!4%4#+*!G#+#!+#$6L#.!.'#!86!8-#?+! (%,7!6B!+#(#L%&,#!B6+!8-#!*8'.)!<*##!%(*6!D#,8?6&!F50A5!=-?*!+#*'(8#.!?&!/:H!4%4#+*!B6+!8-#!B?&%(!+#L?#G! <*##! =%>(#! /0! ?&! 8-#! %44#&.?@A5! K?E5! 1! *-6G*! 8-#!&'$>#+!6B!4%4#+*!?&,('.#.!?&!8-#!+#L?#GN!4#+!;6'+&%(5!K?E5!0!*-6G*!8-#!&'$>#+!6B!4%4#+*!4#+!,6&B#+#&,#5!!

"#! ,6$4%+#.! 6'+! 4+?$%+)! *8'.?#*! G?8-! +#(%8#.! +#CL?#G*!(?*8#.!?&!D#,8?6&!/5F5!"#!&68?,#.!6&()!%!*$%((!6L#+C(%4! >#8G##&! 6'+! 4+?$%+)! *8'.?#*! %&.! 4+?$%+)! *8'.?#*!?&,('.#.! ?&!68-#+!+#L?#G*5!K6+!#@%$4(#N!O-#&!%&.!P%>%+!<QR! *)*8#$%8?,! +#L?#G! 6B! #L%('%8?6&! 6B! L%+?%>?(?8)! $%&C%E#$#&8!%44+6%,-#*!?&!*6B8G%+#!4+6.',8!(?&#*SA!+#L?#G#.!:9! 4%4#+*5! T&()! /1! 4%4#+*! %+#! %(*6! ?&,('.#.! ?&! 6'+! +#CL?#G5! =-#! ,6$4(#8#! ,6$4%+?*6&! ?*! %L%?(%>(#! %*! *'44(#C$#&8%+)! $%8#+?%(! ?&! %! 8#,-&?,%(! +#46+8! U2:V5! =-#! (?$?8#.!6L#+(%4!?*!.'#!86!8-#!.?BB#+#&8!E6%(*!%&.!B6,?!6B!8-#!*8'.C?#*5! T&! 8-#! 68-#+! -%&.N! G#! ,6&,('.#! 8-%8! 6'+! #@8#&.#.!*,64#! <>#)6&.! 4+6.',8! (?&#*A! ?*! ?&.##.! +#B(#,8#.! ?&! 8-#!*8'.?#*!?&,('.#.!?&!6'+!B?&%(!+#L?#G5!!WDD! -%*! >##&! 8-#! $6*8! 4+6$?&#&8! ;6'+&%(! 86! 4'>(?*-!

L%+?%>?(?8)C+#(%8#.! +#*#%+,-N! G-#+#%*! DXYO! ?*! 8-#! $6*8!4+6$?&#&8! ,6&B#+#&,#! B6+! L%+?%>?(?8)! +#*#%+,-5! D#%+,-?&E!

8-#! 8G6! L#&'#*! 8-%8! B6,'*! 6&! #$4?+?,%(! +#*#%+,-! <ZRDZ!%&.!ZDZA!.?.!&68!+#*'(8!?&!%!*?E&?B?,%&8!&'$>#+!6B!*8'.?#*5!=-?*!,6&B?+$*!68-#+!+#L?#G*!8-%8!%(*6!,6'(.!&68!?.#&8?B)!%!*8'.)!6&!L%+?%>?(?8)!$6.#(?&E!4'>(?*-#.!?&!ZDZ!UF3V5!K'+C8-#+$6+#N! &6&#! 6B! 8-#! 8#*8?&E! %&.! L#+?B?,%8?6&! L#&'#*!<[OD=N![DD=RN!D=\]A!6+!.#L#(64$#&8C+#(%8#.!,6&B#+#&,#*!<ZOTTXN! TTXDYRA! *##$*! 86! >#! %88+%,8?L#! L#&'#*! B6+!L%+?%>?(?8)!+#*#%+,-5!!K?E5! H! *-6G*! 8-#! .?*8+?>'8?6&! 6B! *8'.?#*! >%*#.! 6&! 8-#!

L#&'#*! +#(%8#.! 86! 8-#! *6B8G%+#! #&E?&##+?&E! %+#%*! ?&8+6C.',#.!?&!D#,8?6&!F5F5!K?E5!H!*-6G*!8-%8!$6*8!*8'.?#*!-%L#!>##&!4'>(?*-#.!?&!E#&#+%(!*6B8G%+#!#&E?&##+?&E!L#&'#*5!!!

K?&%(()N! K?E5! 9! *-6G*! 8-#! .?*8+?>'8?6&! 6B! *8'.?#*! 6L#+!8-#!)#%+*!B+6$!F333!86!8-#!#&.!6B!F3//5!=-?*!B?E'+#!*-6G*!%&! ?&,+#%*?&E! 8+#&.! B6+! 4'>(?*-?&E! L%+?%>?(?8)! +#*#%+,-!'&8?(!F33I5!RB8#+!F33IN! 8-#! ?&,+#%*#! ?&!4'>(?,%8?6&*! *##$*!86!*(6G!.6G&5!T&#!B%,86+!8-%8!#@4(%?&*!8-?*!8+#&.!,6'(.!>#!8-#!#$#+E?&E!?&8#+#*8!?&!8-#!'*#!6B!L%+?%>?(?8)!?&!.)&%$?,!*#+L?,#!,6$46*?8?6&5!

!

Fig. 4. Number of papers per journal.

Fig. 6. Studies per software engineering area of publication venue.

Fig. 5. Number of papers per conference.

Fig. 7. Studies per year.

5

0

5

11

25

2

10

42

7

0

3 6

0

5

10

15

20

25

30

ASEJ

ESE

IEEE

SW IST

JSS

REJ

SCP

SMRP

SoSy

M

SPE

STVR TS

E

TOSE

M

Num

ber o

f pap

ers

Journal

General SE85%

Requirements engineering

3%

Architecture / design

8%

Implementation / integration

0%

Testing / verification

0%

Maintenance4%

9

13

20

67

58

40 0

8

14 3 4

36

6

0

5

10

15

20

25

30

35

40

ASE

CAiS

E

EASE

ECO

OP

ECSA FS

E

GPC

E

ICSE

ICSM

ICST

ISST

A

MO

DELS

OO

PSLA

QoS

A RE SLE

SPLC

WIC

SA

Num

ber o

f pap

ers

Conference

0

5

10

15

20

25

30

35

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

Num

ber o

f pap

ers

Year


!

3 RESULTS 3.1 Demographic Data "#!$%&'%(()! *#%+,-#.!/0123!4%4#+*5!"#! (667#.!%8! 9::/!;6'+&%(! 4%4#+*! <=%>(#! /2! ?&! %44#&.?@A! %&.! 912:! ,6&B#+C#&,#!4%4#+*!<=%>(#!/1!?&!%44#&.?@A5!=-#!&'$>#+!6B!4%4#+*!%B8#+!D8%E#!F!G%*!2:3!</HI!;6'+&%(!4%4#+*!J!FFF!,6&B#+#&,#!4%4#+*A5! K?L#! 4%4#+*! %(*6! +#*'(8#.! B+6$! 8-#! 8%+E#8#.! %'C86$%8#.!*#%+,-5!M'+?&E!.%8%!,6((#,8?6&N!$6+#!4%4#+*!G#+#!+#$6L#.!.'#!86!8-#?+! (%,7!6B!+#(#L%&,#!B6+!8-#!*8'.)!<*##!%(*6!D#,8?6&!F50A5!=-?*!+#*'(8#.!?&!/:H!4%4#+*!B6+!8-#!B?&%(!+#L?#G! <*##! =%>(#! /0! ?&! 8-#! %44#&.?@A5! K?E5! 1! *-6G*! 8-#!&'$>#+!6B!4%4#+*!?&,('.#.!?&!8-#!+#L?#GN!4#+!;6'+&%(5!K?E5!0!*-6G*!8-#!&'$>#+!6B!4%4#+*!4#+!,6&B#+#&,#5!!

"#! ,6$4%+#.! 6'+! 4+?$%+)! *8'.?#*! G?8-! +#(%8#.! +#CL?#G*!(?*8#.!?&!D#,8?6&!/5F5!"#!&68?,#.!6&()!%!*$%((!6L#+C(%4! >#8G##&! 6'+! 4+?$%+)! *8'.?#*! %&.! 4+?$%+)! *8'.?#*!?&,('.#.! ?&!68-#+!+#L?#G*5!K6+!#@%$4(#N!O-#&!%&.!P%>%+!<QR! *)*8#$%8?,! +#L?#G! 6B! #L%('%8?6&! 6B! L%+?%>?(?8)! $%&C%E#$#&8!%44+6%,-#*!?&!*6B8G%+#!4+6.',8!(?&#*SA!+#L?#G#.!:9! 4%4#+*5! T&()! /1! 4%4#+*! %+#! %(*6! ?&,('.#.! ?&! 6'+! +#CL?#G5! =-#! ,6$4(#8#! ,6$4%+?*6&! ?*! %L%?(%>(#! %*! *'44(#C$#&8%+)! $%8#+?%(! ?&! %! 8#,-&?,%(! +#46+8! U2:V5! =-#! (?$?8#.!6L#+(%4!?*!.'#!86!8-#!.?BB#+#&8!E6%(*!%&.!B6,?!6B!8-#!*8'.C?#*5! T&! 8-#! 68-#+! -%&.N! G#! ,6&,('.#! 8-%8! 6'+! #@8#&.#.!*,64#! <>#)6&.! 4+6.',8! (?&#*A! ?*! ?&.##.! +#B(#,8#.! ?&! 8-#!*8'.?#*!?&,('.#.!?&!6'+!B?&%(!+#L?#G5!!WDD! -%*! >##&! 8-#! $6*8! 4+6$?&#&8! ;6'+&%(! 86! 4'>(?*-!

L%+?%>?(?8)C+#(%8#.! +#*#%+,-N! G-#+#%*! DXYO! ?*! 8-#! $6*8!4+6$?&#&8! ,6&B#+#&,#! B6+! L%+?%>?(?8)! +#*#%+,-5! D#%+,-?&E!

8-#! 8G6! L#&'#*! 8-%8! B6,'*! 6&! #$4?+?,%(! +#*#%+,-! <ZRDZ!%&.!ZDZA!.?.!&68!+#*'(8!?&!%!*?E&?B?,%&8!&'$>#+!6B!*8'.?#*5!=-?*!,6&B?+$*!68-#+!+#L?#G*!8-%8!%(*6!,6'(.!&68!?.#&8?B)!%!*8'.)!6&!L%+?%>?(?8)!$6.#(?&E!4'>(?*-#.!?&!ZDZ!UF3V5!K'+C8-#+$6+#N! &6&#! 6B! 8-#! 8#*8?&E! %&.! L#+?B?,%8?6&! L#&'#*!<[OD=N![DD=RN!D=\]A!6+!.#L#(64$#&8C+#(%8#.!,6&B#+#&,#*!<ZOTTXN! TTXDYRA! *##$*! 86! >#! %88+%,8?L#! L#&'#*! B6+!L%+?%>?(?8)!+#*#%+,-5!!K?E5! H! *-6G*! 8-#! .?*8+?>'8?6&! 6B! *8'.?#*! >%*#.! 6&! 8-#!

L#&'#*! +#(%8#.! 86! 8-#! *6B8G%+#! #&E?&##+?&E! %+#%*! ?&8+6C.',#.!?&!D#,8?6&!F5F5!K?E5!H!*-6G*!8-%8!$6*8!*8'.?#*!-%L#!>##&!4'>(?*-#.!?&!E#&#+%(!*6B8G%+#!#&E?&##+?&E!L#&'#*5!!!

K?&%(()N! K?E5! 9! *-6G*! 8-#! .?*8+?>'8?6&! 6B! *8'.?#*! 6L#+!8-#!)#%+*!B+6$!F333!86!8-#!#&.!6B!F3//5!=-?*!B?E'+#!*-6G*!%&! ?&,+#%*?&E! 8+#&.! B6+! 4'>(?*-?&E! L%+?%>?(?8)! +#*#%+,-!'&8?(!F33I5!RB8#+!F33IN! 8-#! ?&,+#%*#! ?&!4'>(?,%8?6&*! *##$*!86!*(6G!.6G&5!T&#!B%,86+!8-%8!#@4(%?&*!8-?*!8+#&.!,6'(.!>#!8-#!#$#+E?&E!?&8#+#*8!?&!8-#!'*#!6B!L%+?%>?(?8)!?&!.)&%$?,!*#+L?,#!,6$46*?8?6&5!

!

Fig. 4. Number of papers per journal.

Fig. 6. Studies per software engineering area of publication venue.

Fig. 5. Number of papers per conference.

Fig. 7. Studies per year.

5

0

5

11

25

2

10

42

7

0

3 6

0

5

10

15

20

25

30

ASEJ

ESE

IEEE

SW IST

JSS

REJ

SCP

SMRP

SoSy

M

SPE

STVR TS

E

TOSE

M

Num

ber o

f pap

ers

Journal

General SE85%

Requirements engineering

3%

Architecture / design

8%

Implementation / integration

0%

Testing / verification

0%

Maintenance4%

9

13

20

67

58

40 0

8

14 3 4

36

6

0

5

10

15

20

25

30

35

40

ASE

CAiS

E

EASE

ECO

OP

ECSA FS

E

GPC

E

ICSE

ICSM

ICST

ISST

A

MO

DELS

OO

PSLA

QoS

A RE SLE

SPLC

WIC

SA

Num

ber o

f pap

ers

Conference

0

5

10

15

20

25

30

35

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

Num

ber o

f pap

ers

Year

Design vs Runtime Variability

128 studies focus on design time variability

Top: flexibility, reusability, modifiability 50 studies focus on runtime variability

Top: performance, flexibility, availability 18 remaining studies no clear differentiation

Engineering Activities Addressed

130 studies focus on one activity 22 studies focus on two activities One study focuses on 6 activities


3.2 RQ1: What Methods to Handle Variability in Software Systems Exist?

!"# $%&# '()*+))# ),+'(")# %$# -./(.0(1(,2# ($# )%3,&./"# )2)4,"5)#6),+'(")#./"#1(),"'#($#,7"#.88"$'(9:#;.01"#<=>?#

3.2.1 RQ1.1: What Types of Variability do these Methods Handle?

;%#.$)&"/#,7()#@+"),(%$:#&"#'/"&#%$#'.,.#"9,/.*,"'#0.)"'#%$#A<=#6,28")#%3#-./(.0(1(,2>#3/%5#,7"#'.,.#"9,/.*,(%$#3%/5?#B-"/.11:#<CD#),+'(")# 3%*+)#%$#'")(E$#,(5"#-./(.0(1(,2#.$'#=F#),+'(")#%$#/+$,(5"#-./(.0(1(,2?#;7"#/"5.($($E#<D#),+'4(")# '%#$%,# *1"./12#'(33"/"$,(.,"# 0",&""$# /+$,(5"# .$'#'"4)(E$#,(5"#-./(.0(1(,2?##!"#$%&'()*+#,-G")(E$# ,(5"# -./(.0(1(,2# *.$# 0"# *%$)('4

"/"'# ,7"# *+//"$,# ,/"$'# ($# -./(.0(1(,2# /")"./*7?# H%&"-"/:#/"*"$,# ,/"$'):#)+*7#.)#)"134.'.8,(-"#)2),"5)#IJKL#/+$,(5"#%/*7"),/.,(%$# %3# )"/-(*"40.)"'# )2),"5)# IJJL# .$'# ,7"#"5"/E($E#'%5.($#%3#'2$.5(*#8/%'+*,#1($")#IJ=L#7.)#1"'#,%#($*/".)($E#.,,"$,(%$#3%/#/+$,(5"#-./(.0(1(,2?#

3.2.2 RQ1.2: What Activities in the Software Development Process are Affected by these Methods?

;%#.$)&"/#,7()#@+"),(%$:#&"#+)"'#'.,.#"9,/.*,"'#0.)"'#%$#A<K# 6.*,(-(,(")#.''/"))"'#02#.88/%.*7># 3/%5#,7"#'.,.#"94,/.*,(%$#3%/5?#!"#&"/"#($,"/"),"'#($#-./(.0(1(,2#,7.,#*%+1'#%**+/# .$2&7"/"# ($# ,7"# )%3,&./"# '"-"1%85"$,# *2*1"?# G"4-"1%85"$,#.*,(-(,(")#.''/"))"'#./"#)(5(1./#,%#,7"#)%3,&./"#"$E($""/($E#./".)#+)"'#3%/#)"1"*,($E#)"./*7#-"$+")?#;7"/"43%/":# ,7"# .*,(-(,(")# +)"'# '+/($E# '.,.# .$.12)()# &"/"# /"4@+(/"5"$,)# "$E($""/($E:# ./*7(,"*,+/":# '")(E$:# (581"5"$4,.,(%$:# ,"),($E# .$'# -"/(3(*.,(%$:# .$'# 5.($,"$.$*"?# !"#3%+$'#,7.,#)%5"#),+'(")#'('#$%,#"981(*(,12#.''/"))#.$2#%3#,7")"# .*,(-(,("):# &7"/".)# %,7"/# ),+'(")# .''/"))"'# 5%/"#,7.$# %$"# .*,(-(,2?# M$# *.)")# &7"/"# 5%/"# .*,(-(,(")# &"/"#.''/"))"':#&"# .))(E$"'# ),+'(")# ,%# .11# .*,(-(,(")# ,7"2# .'4'/"))?# ;7+):# ,7"# )+5# %3# ,7"# )"*%$'# *%1+5$# ($# ;.01"# =#"9*""')#,7"#$+50"/#%3#),+'(")?##!"#$%&'()*+#,# B-"/.11:# <KF# ),+'(")# %$12# .''/"))# %$"#

)($E1"#.*,(-(,2?#;.01"#=#)7%&)#,7.,#5%),#),+'(")#'".1#&(,7#-./(.0(1(,2# ($# ,7"# ./*7(,"*,+/"# .$'# '")(E$# 87.)"?# N''(4,(%$.112:#&"#3%+$'#,7.,#CC#),+'(")#.''/"))#0%,7#.*,(-(,("):#./*7(,"*,+/"#.$'#'")(E$# 6),+'(")#KO:#=O:#P<:#PC:#P=:#QP:#DF:#<KP:# <KD:# <JK:# <JP:# <PQ:# <PO:# <QF:# <Q<:# <QC:# <QK:# <QJ:# <DQ:#<DO:#<O<:#<OK>?#B$12#%$"#),+'2#6MG#=<>#.''/"))")#)(9#.*,(-(4,(")#6.11#"9*"8,#'")(E$>?#

3.2.3 RQ1.3: What Runtime and Design Time Quality Attributes are Addressed by these Methods?

;%# .$)&"/# RS<?K:# &"# +)"'# '.,.# "9,/.*,"'# 0.)"'# %$# AO#6/+$,(5"#@+.1(,2#.,,/(0+,")#),+'("'>#.$'#A<F#6'")(E$#,(5"#@+.1(,2# .,,/(0+,")# ),+'("'>?# ;%# *%11"*,# .11#8%))(01"#@+.1(,2#.,,/(0+,")#.$'#,%#0"#$%,#1(5(,"'#,%#.#3(9"'#1(),:#&"#'('#$%,#.8812# .# 8/"'"3($"'# 1(),# %3# @+.1(,2# .,,/(0+,")?# M$),".':# ,7"#1(),#%3#@+.1(,2#.,,/(0+,")#"5"/E"'# 3/%5#,7"#@+.1(,2#.,,/(04+,")# "9,/.*,"'# 3/%5# ,7"# ),+'(")?# R+$,(5"# @+.1(,2# .,,/(04+,")#,7.,#7.-"#0""$#5"$,(%$"'#($#),+'(")#./"T#

• N-.(1.0(1(,2# 6CF# ),+'(")>T# N0(1(,2# ,%# 0"# %8"/.,(%$.1#.$'#.**"))(01"#&7"$#/"@+(/"'#3%/#+)"#IJPL?#

• U-%1-.0(1(,2# 6Q# ),+'(")>#.$'# 31"9(0(1(,2# 6C<# ),+'(")>T#N0(1(,2#%3#.#)2),"5#,%#.**%55%'.,"#*7.$E")#($#/"4@+(/"5"$,)# ,7/%+E7# 1(3")8.$# &(,7# 1%&# *%),# &7(1"#5.($,.($($E#./*7(,"*,+/.1#($,"E/(,2?#

• M$,"/%8"/.0(1(,2#6J#),+'(")>T#N0(1(,2#,%#($,"/.*,#&(,7#%$"#%/#5%/"#)8"*(3("'#)2),"5#IJQL?#

• V"/3%/5.$*"# 6CJ# ),+'(")>T# N0(1(,2# ,%# .**%581()7#'")(E$.,"'# 3+$*,(%$)# &(,7($# E(-"$# *%$),/.($,)# ($#,"/5)#%3#)8""':#.**+/.*2:#%/#5"5%/2#+).E"#IJPL?#

• R"1(.0(1(,2# 6K# ),+'(")>T#N0(1(,2# ,%# 8"/3%/5# /"@+(/"'#3+$*,(%$)# +$'"/# ),.,"'# *%$'(,(%$)# 3%/# .# )8"*(3("'#8"/(%'#%3#,(5"#IJPL?#

• W*.1.0(1(,2# 6K# ),+'(")>T#N0(1(,2#%3#7.$'1($E# ($*/".)"#($#*%58+,($E#*.8.*(,2#IJDL?#

• W"*+/(,2# 6C# ),+'(")>T#N0(1(,2#%3# )2),"5# ,%# /")(),#+$4.+,7%/(X"'#+).E"#&7(1"#),(11#8/%-('($E#(,)#)"/-(*"#,%#1"E(,(5.,"#+)"/)#IKKL?#

• W.3",2# 6<#),+'2>T#N0(1(,2# ,%#.-%('#*.,.),/%87(*#*%$4)"@+"$*")#%$#,7"#+)"/)#.$'#,7"#"$-(/%$5"$,#IJDL?#

W"-"$#),+'(")#5"$,(%$#S+.1(,2#%3#W"/-(*"# 6S%W># ($# ,7"#*%$,"9,# %3# -./(.0(1(,2:# &(,7%+,# 3+/,7"/# )8"*(32($E# &7.,#@+.1(,2#.,,/(0+,"#S%W#"9.*,12#/"3"/)#,%?#A+/,7"/5%/":#)%5"#.88/%.*7")#'%#$%,#5"$,(%$#.$2#/+$,(5"#@+.1(,2#.,,/(0+,"#.,# .11?# W(5(1./12:# 5%),# ),+'(")# '%# $%,# "981(*(,12# '()*+))#'")(E$#,(5"#@+.1(,2#.,,/(0+,")?#G")(E$#,(5"#@+.1(,2#.,,/(04+,")#.''/"))"'#02#),+'(")#($*1+'"#,7"#3%11%&($ET#

• U-%1-.0(1(,2#6J#),+'(")>#• A1"9(0(1(,2#6<K#),+'(")>#• Y%'(3(.0(1(,2# 6($*1+'($E# /"*%$3(E+/.0(1(,2# .$'#

5.($,.($.0(1(,2Z#J#),+'(")>#• V%/,.0(1(,2#6C#),+'(")>#• R"+).0(1(,2#6P#),+'(")>#!"#$%&'()*+#,#!"#*%+1'#$%,#('"$,(32#)",)#%3#*%55%$12#

.''/"))"'# /+$,(5"# %/# '")(E$# ,(5"# @+.1(,2# .,,/(0+,")?#R+$,(5"#@+.1(,2#.,,/(0+,")#./"#.''/"))"'#5%/"#"9,"$)(-"412# ,7.$# '")(E$# ,(5"# @+.1(,2# .,,/(0+,")?#H%&"-"/:# %+/# /"4)+1,)# ($'(*.,"# ,7.,# 7.$'1($E# -./(.0(1(,2# ($# @+.1(,2# .,,/(04+,")# ()# +)+.112# $%,# ,7"# 3%*+)# %3# *+//"$,# .88/%.*7"):# (?"?:#-./(.0(1(,2# ($# @+.1(,2# .,,/(0+,")# ()# +)+.112# $%,# "98/"))"'#"981(*(,12?#N#/".)%$#3%/#,7()#()#,7.,#5.$2#8.8"/)#),"5#3/%5#,7"# 8/%'+*,# 1($"# '%5.($# &7(*7# ,/.'(,(%$.112# 3%*+)")# %$#-./(.0(1(,2#($#3".,+/")#6&(,7#3".,+/")#+$'"/),%%'#.)#)",)#%3#3+$*,(%$.1(,2>?###

TABLE 5 Software Engineering Activities Addressed by Studies

#N0)%1+,"# [#

R"@+(/"5"$,#"$E($""/($E# KC# <K?F#N/*7(,"*,+/"# QC# CO?C#G")(E$# DP# KJ?D#M581"5"$,.,(%$# C=# <F?<#;"),($E#.$'#-"/(3(*.,(%$# =# C?F#Y.($,"$.$*"# CQ# <F?O#

#

Level of Evidence

Level 0: no evidence Level 1: demonstration or toy example Level 2: expert opinions


!"#$%&'()&*+(&,-$,.$#/&%#-0#$1#/#1'$&'$'",2-$&-$3&45$67$8900,)%&-4$ (,$ #/&%#-0#$ 1#/#1'$ &-$ :#0(&,-$ ;5<=5$ !"&'$ .&4+)#$'",2'$ ("9($>,'($ '(+%&#'$ "9/#$ 9$ 1,2$ #/&%#-0#$ 1#/#1$ 9-%$#/&%#-0#$"9'$,-1?$*##-$,*(9&-#%$.),>$%#>,-'()9(&,-'$,)$(,?$ #@9>A1#'5$B-$ ("#$ ,("#)$ "9-%C$ ("#)#$ &'$ -,$ '(+%?$ ("9($'",2'$ &-%+'()&91$ #/&%#-0#5$ 3+)("#)>,)#C$ 3&45$ 67$ '",2'$("9($>9-?$'(+%&#'$"9/#$("#$'9>#$#/&%#-0#$1#/#1$8#/&%#-0#$1#/#1$ 6=$ 9-%$ ("+'$ 09--,($ *#$ %&..#)#-(&9(#%$ *9'#%$ ,-$ ("#$#/&%#-0#$ 1#/#15$ 3&-911?C$ 2#$ 1&'($ '(+%&#'$ 2&("$ ("#$ "&4"#'($#/&%#-0#$1#/#1$&-$!9*1#$D5$E49&-C$("#'#$'(+%&#'$9)#$A,(#-F(&911?$ >,'($ )#1#/9-($ .,)$ )#'#9)0"#)'$ 9-%$ A)90(&(&,-#)'5$G#9)',-H'$0,))#19(&,-$0,#..&0&#-('$.,)$#/&%#-0#$1#/#1$/#)'+'$*,("$ ("#$ 9/#)94#$ 0&(9(&,-$ 0,+-($ 9-%$ ("#$ (,(91$ 0&(9(&,-$0,+-($ 9)#$ 01,'#$ (,$ I#),5$ 3&.(?F.&/#$ A9A#)'$ 9)#$ #>A&)&091$A9A#)'$ 8&5#5C$ &-01+%#$ 9-$ #>A&)&091$ '(+%?C$ #545C$ 9'$ A9)($ ,.$#/91+9(&-4$9$>#(",%=5$!"#$9/#)94#$J+91&(?$'0,)#$,.$#>A&)F&091$A9A#)'$ &'$ K5D5$!"#$9/#)94#$J+91&(?$ '0,)#$,.$ ,("#)$ A9FA#)'$ &'$ ;5D5$ :&>&19)$ 9'$ 2&("$ J+91&(?$ '0,)#'C$ 2#$ .,+-%$ -,$0,))#19(&,-$ *#(2##-$ ("#$ 9/#)94#$ 0&(9(&,-$ 0,+-($ 9-%$ ("#$#/&%#-0#$1#/#1'$,.$911$A9A#)'$8G#9)',-H'$0,))#19(&,-$0,#..&F0&#-($ !$ L$757KMMN$ !$ L$ F7577DM$ .,)$ 0,))#19(&,-$*#(2##-$ (,(91$0&(9(&,-$0,+-($9-%$#/&%#-0#$1#/#1'=5$!"#$%&'()*+#,$ E+(",)'$ A),/&%#$ 9$ %#'0)&A(&,-$ ,.$ ("#$

>,(&/9(&,-$ 9-%$ 0,-(#@($ .,)$ ("#$ )#'#9)0"C$ *+($ )#'#9)0"$%#'&4-C$%9(9$(,$'+AA,)($.&-%&-4'C$9-%$0)&(&091$)#.1#0(&,-$9)#$A,,)1?$%#'0)&*#%5$!"#$ (,(91$ J+91&(?$ '0,)#'$ .,)$ ("#$ '(+%&#'$9)#$ >,%#)9(#C$ 9-%$ 91(",+4"$ ("#'#$ -+>*#)'$ )#.#)$ (,$ ("#$29?$'(+%&#'$9)#$)#A,)(#%C$&($&'$%&..&0+1($(,$O+%4#$("#$J+91&F(?$,.$9$'(+%?$&.$&($&'$A,,)1?$)#A,)(#%5$3&-911?C$>,'($'(+%&#'$

"9/#$9$1,2$#/&%#-0#$1#/#15$:#/#)91$)#'#9)0"#)'$&-$("#$2&%F#)$ ',.(29)#$ #-4&-##)&-4$ 0,>>+-&(?$ "9/#$ )#A#9(#%$ ("#$-##%$ .,)$ '?'(#>9(&0$ #/91+9(&,-C$ #545C$ PMQC$ R7S5$ B+)$ '(+%?$0,-.&)>'$("9($("&'$,*'#)/9(&,-$91',$9AA1&#'$(,$("#$/9)&9*&1F&(?$"9-%1&-4$&-$',.(29)#$#-4&-##)&-45$

3.4 RQ3: What are the Limitations of Existing Methods?

!,$ 9-'2#)$ ("&'$ J+#'(&,-C$ 2#$ +'#%$ .&#1%$ 36M$ ,.$ ("#$ %9(9$0,11#0(&,-$ .,)>5$E1',C$2#$ 9-91?I#%$ %9(9$ 9*,+($ (,,1$ '+AFA,)($A),/&%#%$&-$'(+%&#'$8366=N$190T$,.$(,,1$'+AA,)($2,+1%$*#$ 9$ 1&>&(9(&,-$ .,)$ ("#$ A)90(&091$ 9AA1&09*&1&(?$ ,.$ 9-$ 9AFA),90"5$ U9'#%$ ,-$ ("#$ %9(9$ .),>$ 36MC$ 2#$ &%#-(&.&#%$ ("#$.,11,2&-4$)#0+))&-4$("#>#'$&-$)#A,)(#%$1&>&(9(&,-'V$

• W,))#0(-#''$9-%$0,-'&'(#-0?$9)#$%&..&0+1($(,$4+9)9-F(##N$ 9+(",)'$ )#A,)($ ("#$ -##%$ .,)$ /#)&.&09(&,-C$ 0,-F'&'(#-0?$0"#0T&-4C$#(05$

• X&>&(9(&,-'$)#49)%&-4$J+91&(?$9(()&*+(#'C$&-$A9)(&0+F19)$A#).,)>9-0#$,/#)"#9%C$190T$,.$'+AA,)($.,)$'#0+F)&(?$ -##%'$ 9-%$ %?-9>&0$ .#9(+)#'C$ '+0"$ 9'$ )+-(&>#$+A4)9%9*&1&(?5$$

• G,,)$ +'#)F.)&#-%1&-#''C$ #545$ &-$ (#)>'$ ,.$ .1#@&*&1&(?$9-%$+'#)$'+AA,)($.,)$/9)&9*&1&(?$"9-%1&-45$

• X&>&(9(&,-'$)#49)%&-4$("#$&%#-(&.&09(&,-$,.$/9)&9*&1F&(?$ 9-%$ /9)&9-('C$ #545C$ ',>#$ 9+(",)'$ )#A,)($ ("9($>,)#$ 9%/9-0#%$ (#0"-&J+#'$ '+0"$ 9'$ %9(9$ >&-&-4$0,+1%$*#$9AA1&#%5$$

B("#)$1&>&(9(&,-'$9)#$)#19(#%$(,$'A#0&.&0$&''+#'C$'+0"$9'$-,(9(&,-'$,)$>,%#1'$ (,$*#$+'#%5$Y&("$)#49)%$(,$ (,,1$'+AFA,)($8366=$2#$.,+-%$("9($67M$'(+%&#'$&-01+%#$',>#$',)($,.$(,,1$ '+AA,)(5$ :&@(?F'#/#-$ '(+%&#'$ %,$ -,($ A),/&%#$ (,,1$'+AA,)($ 9-%$ ("#$ )#>9&-&-4$ '(+%&#'$ %,$ -,($ %&'0+''$ 9-?$(,,1$'+AA,)(5$B("#)$1&>&(9(&,-'$)#19(#%$(,$A,,)$/91&%9(&,-$,)$'(+%?$J+91&(?$"9/#$*##-$%&'0+''#%$&-$:#0(&,-$K5K5$!"#$%&'()*+#,$!"#$)#A,)(#%$%9(9$0,-.&)>$'#/#)91$.&-%F

&-4'$2#$91)#9%?$%&'0+''#%C$&-$A9)(&0+19)C$1&>&(#%$0,/#)94#$,.$ J+91&(?$ 9(()&*+(#'C$ A,,)$ /91&%9(&,-C$ 9-%$ ("#$ -##%$ .,)$*#((#)$ (,,1$ '+AA,)(5$ Z-$ 9%%&(&,-C$ (2,$ &%#-(&.&#%$ ()90T'$ ,.$

Fig. 10. Distribution of evidence levels.

TABLE 8 Studies with Highest Evidence Level

-(./01234#1 5&)/$+6$17$&$71KC$[C$6<C$6QC$M[C$R6C$R<C$<MC$D<C$Q;C$Q<C$66KC$66[C$66<C$66DC$6;;C$6;[C$6;DC$6;QC$6K;C$6M[C$6R6C$6[RC$6[[C$6QR$

M$

$

TABLE 7 Studies with Citation Count > 100

-(./01231 8)('()*+16*.+(1 9&$%':$1 6)('()*+16*.+(1

;*('71 <.'7)(01#6*%$1

R$ M;D$ <6$ [57$<$ 6QD$ 6R$ 657$66$ 6K7$ 6;$ M5R$6M$ 6[D$ 6<$ K5R$6<$ ;;Q$ ;R$ R5R$;7$ 6;[$ 6M$ ;5R$;Q$ 6<K$ KR$ ;5R$KK$ MQ6$ MQ$ K5R$KR$ 6Q;$ ;6$ ;57$KD$ 6Q<$ ;7$ R57$MR$ 67<$ 6K$ K5R$R7$ 6R7$ K7$ ;5R$RM$ ;7;$ ;Q$ M5R$<R$ 66M$ 6Q$ M57$<[$ 6QK$ ;6$ K57$D6$ 6;D$ 66$ M5R$DR$ 67D$ 6;$ M57$Q[$ 66[$ 66$ K57$Q<$ 6;M$ 66$ K57$66K$ 6<6$ ;6$ M57$6;M$ ;6D$ MM$ K5R$6KM$ 676$ KM$ ;5R$6KR$ 6Q<$ ;7$ 65R$6MR$ KD7$ MD$ 65R$6<7$ KMR$ MK$ ;57$6<K$ [D6$ R;$ ;5R$6DR$ 6[<$ ;D$ ;57$6Q;$ ;;R$ ;R$ 65R$6QM$ 67[$ KR$ M5R$$

Level 0, 19, 10%

Level 1, 106, 54%

Level 2, 20, 10%

Level 3, 26, 13%

Level 4, 25, 13%

Level 5, 0, 0%

Level 3: academic studies (experiments, etc.) Level 4: industrial studies (e.g., case studies) Level 5: evidence from industrial practice

Observations Related to Evidence (1/2) Quality score of publications Q1: rationale Q2: context of the research Q3: research design Q4: data available Q5: role of researchers Q6: discussion credibility results


!

3.3 RQ2: What is the Evidence that Motivates the Adoption of Existing Methods?

"#$%&'(&! )*! +%*,)! ,-*,*.&%! /&)0*%.! $.! %&-$#&%! 1-*/!)0-&&!)2,&.!*1!$'1*-/+)$*'3!40&!56+7$)2!.(*-&.!89:;!+'%!)0&!&#$%&'(&! 7&#&7! 89<=;>! 96-)0&-/*-&?! @&! 6.&%! )0&! )2,&! *1!.)6%2!89<A;!)*!$'#&.)$B+)&!$1!)0&-&!$.!+!%$11&-&'(&!C&)@&&'!&/,$-$(+7!+'%!'*'D&/,$-$(+7!,+,&-.>!9$'+772?!@&!6.&%!)0&!($)+)$*'! (*6')! )*! $%&')$12! $1! )0&-&! $.! +'2! (*--&7+)$*'! C&D)@&&'!&#$%&'(&!+'%!($)+)$*'.!*1!+!,+,&-!89E;>!!9$B>!:!.0*@.!+!1-&56&'(2!+'+72.$.!*1!)0&!.(*-&.!1*-!&+(0!

56+7$)2!56&.)$*'!$'!4+C7&!F>!96-)0&-/*-&?!9$B>!:!%$11&-&')$D+)&.!.(*-&.!1*-!&/,$-$(+7!+'%!'*'D&/,$-$(+7!.)6%$&.>!G*.)!.)6%$&.! ,-*#$%&! +! -+)$*'+7&! 1*-! @02! )0&! ,-*,*.&%! @*-H!@*67%! C&! '&&%&%! 8I<;>! J7.*?! /*.)! .)6%$&.! %&.(-$C&! 8+)!7&+.)!,+-)$+772;! )0&! (*')&K)! $'!@0$(0! )0&2!0+#&!C&&'! (*'D%6()&%!8I=;>!L*@&#&-?!/*.)!.)6%$&.!%*!'*)!,-*,&-72!%&D.(-$C&!+!-&.&+-(0!%&.$B'!8IF;!+'%!1+$7!)*!(-$)$(+772!&K+/$'&!)0&! -*7&! *1! -&.&+-(0&-.! $'#*7#&%! $'! )0&! .)6%2! 8IM;>!G*.)!.)6%$&.!,-&.&')!*'72!,+-)$+7!%+)+!)0+)!@*67%!.6,,*-)!1$'%D$'B.! 1-*/! +! .)6%2! 8IN;>! O$/$7+-72?! /*.)! .)6%$&.! ,-&.&')!*'72!+!7$/$)&%!%$.(6..$*'!*1!)0&$-!(-&%$C$7$)2!8IA;>!4*!$%&'D)$12!@0&)0&-! )0&-&! $.! +! %$11&-&'(&! C&)@&&'! &/,$-$(+7! ,+D,&-.! +'%! '*'D&/,$-$(+7! ,+,&-.! @$)0! -&B+-%! )*! 56+7$)2!.(*-&.?! @&! ,&-1*-/&%! +! G+''DP0$)'&2! Q! )&.)! *'! )@*!$'%&,&'%&')! B-*6,.! 8&/,$-$(+7! ,+,&-.?! '*'D&/,$-$(+7!,+,&-.;>!40&! )&.)! .0*@&%!+! .)+)$.)$(+772! .$B'$1$(+')!%$11&-D&'(&!@$)0! -&B+-%! )*! +77! .$K! 56+7$)2! 56&.)$*'.>! R'! B&'&-+7?!&/,$-$(+7! .)6%$&.! +(0$&#&%!0$B0&-! .(*-&.! +(-*..! +77! 56&.D)$*'.>!40$.!*C.&-#+)$*'!(*'1$-/.!)0+)!&/,$-$(+7!.)6%$&.!+-&!(*'%6()&%! +'%! -&,*-)&%! $'! +!/*-&! .2.)&/+)$(! +'%! .)-$()!/+''&->!L*@&#&-?!$)!$.!$')&-&.)$'B!)0+)!&#&'!)0&!/$'*-$)2!*1! &/,$-$(+7! ,+,&-.! (-$)$(+772! &K+/$'&.! )0&! -*7&! *1! -&D.&+-(0&-.!8IM;>!

9$B>! S! .0*@.! )0&! %$.)-$C6)$*'! *1! )*)+7! 56+7$)2! .(*-&.>!40&!/+K$/6/!)*)+7!.(*-&!$.!A!8$1!+!.)6%2!-&(&$#&%!+!.(*-&!*1! <! 1*-! &#&-2!56+7$)2!56&.)$*';>!G*.)! .)6%$&.! -&(&$#&%!+!.(*-&! C&)@&&'! <>M! +'%! F>M>! 40$.! /&+'.?! *'! +#&-+B&! )0&!56+7$)2! *1! )0&! .)6%$&.! $.! '&$)0&-! ,&-1&()?! '*-! %*&.! $)! .6BDB&.)!)0+)!.)6%$&.!+-&!(*/,7&)&72!17+@&%>!!!!!!

R'!4+C7&!A!@&! 7$.)! )0&! .)6%$&.!@$)0! )0&!0$B0&.)!56+7$)2!.(*-&.!8$>&>?!+!56+7$)2!.(*-&!*1!A!+'%!M>M;>!O)6%2!$%&')$1$&-.!8O)6%2!RT;!6.&%!)0-*6B0*6)!)0&!,+,&-!-&1&-!)*!)0&!$%&')$D1$&-.!*1!.)6%$&.!+.!6.&%!$'!4+C7&!<M!$'!)0&!+,,&'%$K>!!

R'!4+C7&!E!@&!7$.)!)0&!.)6%$&.!@$)0!+)!7&+.)!<UU!($)+)$*'.>!P&!7$.)!.)6%$&.!@$)0!)0&!0$B0&.)!56+7$)2!.(*-&.!+'%!+!($)+D)$*'! (*6').! C&(+6.&! )0&.&! .)6%$&.! +-&! /*-&! 7$H&72! )*! C&!6.&%!C2!-&.&+-(0&-.!+'%!,-+()$)$*'&-.>!J!($)+)$*'!(*6')!*1!<UU!$.!(0*.&'!C&(+6.&!)0$.!$.!)0&!($)+)$*'!(*6')!*1!)0&!)*,!<MV!,+,&-.>!L*@&#&-?!)0&.&!'6/C&-.!+-&!*'72!6.&%!+.!+'!+-C$)-+-2! )0-&.0*7%! )*!,*$')!*6)!0$B0D56+7$)2! .)6%$&.>!4+DC7&! E! +7.*! $'(76%&.! )0&!'6/C&-!*1! +#&-+B&! ($)+)$*'.!6')$7!=U<F?! C+.&%!*'! )0&!,6C7$(+)$*'!2&+->!40&-&! $.!'*! (*--&7+D)$*'! C&)@&&'! )0&! +#&-+B&! ($)+)$*'! (*6')! +'%! )0&! )*)+7!56+7$)2! .(*-&! 1*-! +77! <SA! ,+,&-.! 8W&+-.*'X.! (*--&7+)$*'!(*&11$($&')! !! Y! U><<AEZ! !! Y! U>UFSA! $1! (*--&7+)$*'! C&)@&&'!)*)+7!($)+)$*'!(*6')!+'%!)*)+7!56+7$)2!.(*-&!$.!(*/,6)&%;>!!!!!!!!!

Fig. 8. Frequency analysis of quality scores for each question.

Fig. 9. Distribution of total quality scores.

TABLE 6 Studies with a total Quality Score of 6 and 5.5

!"#$%&'(& )#*+,"%&-./01&M! A!<A! A!N=! A!NA! A!ME! A!<E! M>M!M=! M>M!MS! M>M!AA! M>M!<NN! M>M!<NA! M>M!<:E! M>M!

!

0

20

40

60

80

100

120

140

160

0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1Q1 Q2 Q3 Q4 Q5 Q6

Freq

uenc

y of

sco

res

Non-empirical Empirical

0

5

10

15

20

25

30

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

Freq

uenc

y of

tota

l sco

res

Total score

no to some extent

yes

Observations Related to Evidence (1/2) Quality score of publications Q1: rationale Q2: context of the research Q3: research design Q4: data available Q5: role of researchers Q6: discussion credibility results


!


"#$%&'(&! )*! +%*,)! ,-*,*.&%! /&)0*%.! $.! %&-$#&%! 1-*/!)0-&&!)2,&.!*1!$'1*-/+)$*'3!40&!56+7$)2!.(*-&.!89:;!+'%!)0&!&#$%&'(&! 7&#&7! 89<=;>! 96-)0&-/*-&?! @&! 6.&%! )0&! )2,&! *1!.)6%2!89<A;!)*!$'#&.)$B+)&!$1!)0&-&!$.!+!%$11&-&'(&!C&)@&&'!&/,$-$(+7!+'%!'*'D&/,$-$(+7!,+,&-.>!9$'+772?!@&!6.&%!)0&!($)+)$*'! (*6')! )*! $%&')$12! $1! )0&-&! $.! +'2! (*--&7+)$*'! C&D)@&&'!&#$%&'(&!+'%!($)+)$*'.!*1!+!,+,&-!89E;>!!9$B>!:!.0*@.!+!1-&56&'(2!+'+72.$.!*1!)0&!.(*-&.!1*-!&+(0!


9$B>! S! .0*@.! )0&! %$.)-$C6)$*'! *1! )*)+7! 56+7$)2! .(*-&.>!40&!/+K$/6/!)*)+7!.(*-&!$.!A!8$1!+!.)6%2!-&(&$#&%!+!.(*-&!*1! <! 1*-! &#&-2!56+7$)2!56&.)$*';>!G*.)! .)6%$&.! -&(&$#&%!+!.(*-&! C&)@&&'! <>M! +'%! F>M>! 40$.! /&+'.?! *'! +#&-+B&! )0&!56+7$)2! *1! )0&! .)6%$&.! $.! '&$)0&-! ,&-1&()?! '*-! %*&.! $)! .6BDB&.)!)0+)!.)6%$&.!+-&!(*/,7&)&72!17+@&%>!!!!!!







!

0

20

40

60

80

100

120

140

160

0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1Q1 Q2 Q3 Q4 Q5 Q6

Freq

uenc

y of

sco

res


0

5

10

15

20

25

30

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

Freq

uenc

y of

tota

l sco

res

Total score

Average of empirical papers = 3.8 None empirical papers = 2.8


!


"#$%&'(&! )*! +%*,)! ,-*,*.&%! /&)0*%.! $.! %&-$#&%! 1-*/!)0-&&!)2,&.!*1!$'1*-/+)$*'3!40&!56+7$)2!.(*-&.!89:;!+'%!)0&!&#$%&'(&! 7&#&7! 89<=;>! 96-)0&-/*-&?! @&! 6.&%! )0&! )2,&! *1!.)6%2!89<A;!)*!$'#&.)$B+)&!$1!)0&-&!$.!+!%$11&-&'(&!C&)@&&'!&/,$-$(+7!+'%!'*'D&/,$-$(+7!,+,&-.>!9$'+772?!@&!6.&%!)0&!($)+)$*'! (*6')! )*! $%&')$12! $1! )0&-&! $.! +'2! (*--&7+)$*'! C&D)@&&'!&#$%&'(&!+'%!($)+)$*'.!*1!+!,+,&-!89E;>!!9$B>!:!.0*@.!+!1-&56&'(2!+'+72.$.!*1!)0&!.(*-&.!1*-!&+(0!


9$B>! S! .0*@.! )0&! %$.)-$C6)$*'! *1! )*)+7! 56+7$)2! .(*-&.>!40&!/+K$/6/!)*)+7!.(*-&!$.!A!8$1!+!.)6%2!-&(&$#&%!+!.(*-&!*1! <! 1*-! &#&-2!56+7$)2!56&.)$*';>!G*.)! .)6%$&.! -&(&$#&%!+!.(*-&! C&)@&&'! <>M! +'%! F>M>! 40$.! /&+'.?! *'! +#&-+B&! )0&!56+7$)2! *1! )0&! .)6%$&.! $.! '&$)0&-! ,&-1&()?! '*-! %*&.! $)! .6BDB&.)!)0+)!.)6%$&.!+-&!(*/,7&)&72!17+@&%>!!!!!!







!

0

20

40

60

80

100

120

140

160

0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1Q1 Q2 Q3 Q4 Q5 Q6

Freq

uenc

y of

sco

res


0

5

10

15

20

25

30

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6

Freq

uenc

y of

tota

l sco

res

Total score

no to some extent

yes

Observations Related to Evidence (2/2)

No correlation between number of citations and total quality score of the papers


!"#$%&'()&*+(&,-$,.$#/&%#-0#$1#/#1'$&'$'",2-$&-$3&45$67$8900,)%&-4$ (,$ #/&%#-0#$ 1#/#1'$ &-$ :#0(&,-$ ;5<=5$ !"&'$ .&4+)#$'",2'$ ("9($>,'($ '(+%&#'$ "9/#$ 9$ 1,2$ #/&%#-0#$ 1#/#1$ 9-%$#/&%#-0#$"9'$,-1?$*##-$,*(9&-#%$.),>$%#>,-'()9(&,-'$,)$(,?$ #@9>A1#'5$B-$ ("#$ ,("#)$ "9-%C$ ("#)#$ &'$ -,$ '(+%?$ ("9($'",2'$ &-%+'()&91$ #/&%#-0#5$ 3+)("#)>,)#C$ 3&45$ 67$ '",2'$("9($>9-?$'(+%&#'$"9/#$("#$'9>#$#/&%#-0#$1#/#1$8#/&%#-0#$1#/#1$ 6=$ 9-%$ ("+'$ 09--,($ *#$ %&..#)#-(&9(#%$ *9'#%$ ,-$ ("#$#/&%#-0#$ 1#/#15$ 3&-911?C$ 2#$ 1&'($ '(+%&#'$ 2&("$ ("#$ "&4"#'($#/&%#-0#$1#/#1$&-$!9*1#$D5$E49&-C$("#'#$'(+%&#'$9)#$A,(#-F(&911?$ >,'($ )#1#/9-($ .,)$ )#'#9)0"#)'$ 9-%$ A)90(&(&,-#)'5$G#9)',-H'$0,))#19(&,-$0,#..&0&#-('$.,)$#/&%#-0#$1#/#1$/#)'+'$*,("$ ("#$ 9/#)94#$ 0&(9(&,-$ 0,+-($ 9-%$ ("#$ (,(91$ 0&(9(&,-$0,+-($ 9)#$ 01,'#$ (,$ I#),5$ 3&.(?F.&/#$ A9A#)'$ 9)#$ #>A&)&091$A9A#)'$ 8&5#5C$ &-01+%#$ 9-$ #>A&)&091$ '(+%?C$ #545C$ 9'$ A9)($ ,.$#/91+9(&-4$9$>#(",%=5$!"#$9/#)94#$J+91&(?$'0,)#$,.$#>A&)F&091$A9A#)'$ &'$ K5D5$!"#$9/#)94#$J+91&(?$ '0,)#$,.$ ,("#)$ A9FA#)'$ &'$ ;5D5$ :&>&19)$ 9'$ 2&("$ J+91&(?$ '0,)#'C$ 2#$ .,+-%$ -,$0,))#19(&,-$ *#(2##-$ ("#$ 9/#)94#$ 0&(9(&,-$ 0,+-($ 9-%$ ("#$#/&%#-0#$1#/#1'$,.$911$A9A#)'$8G#9)',-H'$0,))#19(&,-$0,#..&F0&#-($ !$ L$757KMMN$ !$ L$ F7577DM$ .,)$ 0,))#19(&,-$*#(2##-$ (,(91$0&(9(&,-$0,+-($9-%$#/&%#-0#$1#/#1'=5$!"#$%&'()*+#,$ E+(",)'$ A),/&%#$ 9$ %#'0)&A(&,-$ ,.$ ("#$

>,(&/9(&,-$ 9-%$ 0,-(#@($ .,)$ ("#$ )#'#9)0"C$ *+($ )#'#9)0"$%#'&4-C$%9(9$(,$'+AA,)($.&-%&-4'C$9-%$0)&(&091$)#.1#0(&,-$9)#$A,,)1?$%#'0)&*#%5$!"#$ (,(91$ J+91&(?$ '0,)#'$ .,)$ ("#$ '(+%&#'$9)#$ >,%#)9(#C$ 9-%$ 91(",+4"$ ("#'#$ -+>*#)'$ )#.#)$ (,$ ("#$29?$'(+%&#'$9)#$)#A,)(#%C$&($&'$%&..&0+1($(,$O+%4#$("#$J+91&F(?$,.$9$'(+%?$&.$&($&'$A,,)1?$)#A,)(#%5$3&-911?C$>,'($'(+%&#'$

"9/#$9$1,2$#/&%#-0#$1#/#15$:#/#)91$)#'#9)0"#)'$&-$("#$2&%F#)$ ',.(29)#$ #-4&-##)&-4$ 0,>>+-&(?$ "9/#$ )#A#9(#%$ ("#$-##%$ .,)$ '?'(#>9(&0$ #/91+9(&,-C$ #545C$ PMQC$ R7S5$ B+)$ '(+%?$0,-.&)>'$("9($("&'$,*'#)/9(&,-$91',$9AA1&#'$(,$("#$/9)&9*&1F&(?$"9-%1&-4$&-$',.(29)#$#-4&-##)&-45$

3.4 RQ3: What are the Limitations of Existing Methods?

!,$ 9-'2#)$ ("&'$ J+#'(&,-C$ 2#$ +'#%$ .&#1%$ 36M$ ,.$ ("#$ %9(9$0,11#0(&,-$ .,)>5$E1',C$2#$ 9-91?I#%$ %9(9$ 9*,+($ (,,1$ '+AFA,)($A),/&%#%$&-$'(+%&#'$8366=N$190T$,.$(,,1$'+AA,)($2,+1%$*#$ 9$ 1&>&(9(&,-$ .,)$ ("#$ A)90(&091$ 9AA1&09*&1&(?$ ,.$ 9-$ 9AFA),90"5$ U9'#%$ ,-$ ("#$ %9(9$ .),>$ 36MC$ 2#$ &%#-(&.&#%$ ("#$.,11,2&-4$)#0+))&-4$("#>#'$&-$)#A,)(#%$1&>&(9(&,-'V$

• W,))#0(-#''$9-%$0,-'&'(#-0?$9)#$%&..&0+1($(,$4+9)9-F(##N$ 9+(",)'$ )#A,)($ ("#$ -##%$ .,)$ /#)&.&09(&,-C$ 0,-F'&'(#-0?$0"#0T&-4C$#(05$

• X&>&(9(&,-'$)#49)%&-4$J+91&(?$9(()&*+(#'C$&-$A9)(&0+F19)$A#).,)>9-0#$,/#)"#9%C$190T$,.$'+AA,)($.,)$'#0+F)&(?$ -##%'$ 9-%$ %?-9>&0$ .#9(+)#'C$ '+0"$ 9'$ )+-(&>#$+A4)9%9*&1&(?5$$

• G,,)$ +'#)F.)&#-%1&-#''C$ #545$ &-$ (#)>'$ ,.$ .1#@&*&1&(?$9-%$+'#)$'+AA,)($.,)$/9)&9*&1&(?$"9-%1&-45$

• X&>&(9(&,-'$)#49)%&-4$("#$&%#-(&.&09(&,-$,.$/9)&9*&1F&(?$ 9-%$ /9)&9-('C$ #545C$ ',>#$ 9+(",)'$ )#A,)($ ("9($>,)#$ 9%/9-0#%$ (#0"-&J+#'$ '+0"$ 9'$ %9(9$ >&-&-4$0,+1%$*#$9AA1&#%5$$

B("#)$1&>&(9(&,-'$9)#$)#19(#%$(,$'A#0&.&0$&''+#'C$'+0"$9'$-,(9(&,-'$,)$>,%#1'$ (,$*#$+'#%5$Y&("$)#49)%$(,$ (,,1$'+AFA,)($8366=$2#$.,+-%$("9($67M$'(+%&#'$&-01+%#$',>#$',)($,.$(,,1$ '+AA,)(5$ :&@(?F'#/#-$ '(+%&#'$ %,$ -,($ A),/&%#$ (,,1$'+AA,)($ 9-%$ ("#$ )#>9&-&-4$ '(+%&#'$ %,$ -,($ %&'0+''$ 9-?$(,,1$'+AA,)(5$B("#)$1&>&(9(&,-'$)#19(#%$(,$A,,)$/91&%9(&,-$,)$'(+%?$J+91&(?$"9/#$*##-$%&'0+''#%$&-$:#0(&,-$K5K5$!"#$%&'()*+#,$!"#$)#A,)(#%$%9(9$0,-.&)>$'#/#)91$.&-%F

&-4'$2#$91)#9%?$%&'0+''#%C$&-$A9)(&0+19)C$1&>&(#%$0,/#)94#$,.$ J+91&(?$ 9(()&*+(#'C$ A,,)$ /91&%9(&,-C$ 9-%$ ("#$ -##%$ .,)$*#((#)$ (,,1$ '+AA,)(5$ Z-$ 9%%&(&,-C$ (2,$ &%#-(&.&#%$ ()90T'$ ,.$

Fig. 10. Distribution of evidence levels.

TABLE 8 Studies with Highest Evidence Level

-(./01234#1 5&)/$+6$17$&$71KC$[C$6<C$6QC$M[C$R6C$R<C$<MC$D<C$Q;C$Q<C$66KC$66[C$66<C$66DC$6;;C$6;[C$6;DC$6;QC$6K;C$6M[C$6R6C$6[RC$6[[C$6QR$

M$

$

TABLE 7 Studies with Citation Count > 100

-(./01231 8)('()*+16*.+(1 9&$%':$1 6)('()*+16*.+(1

;*('71 <.'7)(01#6*%$1

R$ M;D$ <6$ [57$<$ 6QD$ 6R$ 657$66$ 6K7$ 6;$ M5R$6M$ 6[D$ 6<$ K5R$6<$ ;;Q$ ;R$ R5R$;7$ 6;[$ 6M$ ;5R$;Q$ 6<K$ KR$ ;5R$KK$ MQ6$ MQ$ K5R$KR$ 6Q;$ ;6$ ;57$KD$ 6Q<$ ;7$ R57$MR$ 67<$ 6K$ K5R$R7$ 6R7$ K7$ ;5R$RM$ ;7;$ ;Q$ M5R$<R$ 66M$ 6Q$ M57$<[$ 6QK$ ;6$ K57$D6$ 6;D$ 66$ M5R$DR$ 67D$ 6;$ M57$Q[$ 66[$ 66$ K57$Q<$ 6;M$ 66$ K57$66K$ 6<6$ ;6$ M57$6;M$ ;6D$ MM$ K5R$6KM$ 676$ KM$ ;5R$6KR$ 6Q<$ ;7$ 65R$6MR$ KD7$ MD$ 65R$6<7$ KMR$ MK$ ;57$6<K$ [D6$ R;$ ;5R$6DR$ 6[<$ ;D$ ;57$6Q;$ ;;R$ ;R$ 65R$6QM$ 67[$ KR$ M5R$$

Level 0, 19, 10%

Level 1, 106, 54%

Level 2, 20, 10%

Level 3, 26, 13%

Level 4, 25, 13%

Level 5, 0, 0%

Reported Limitations

Correctness and consistency are difficult to guarantee; need for verification techniques Impact on quality attributes, in particular performance overhead, support for security needs, dynamic features Poor user-friendliness, lack of flexibility, support for users (2 on 3 studies provide some kind of tool support) Identification of variability and variants, need for advanced techniques such as data mining

Dimensions of Variability

Aim to make a step forward in establishing a more holistic characterization of variability in software systems

Capture key facets of variability (“dimensions”) Baseline to identify variability needs and compare approaches

We identified initial set of dimensions from a number of existing approaches documented in the literature Pilot (20% papers with best quality score) to test dimensions and identify different options (“domain”) for each dimension

Dimensions of Variability: Two Clusters


!

"#$!%&'()*)("+),#!,*!-".)"/)0)+12!3"%'$!,#!+4'!$"+"!(,00'(+5'$! )#! +4'! &)0,+6! 7'! ,/%'.-'! +4"+! +4'! .'89).':'#+%! +4"+!-".)"/)0)+1! "$$.'%%'%! ".'! "&&.,;):"+'01! '89"001! $)-)$'$!/'+7''#! *9#(+),#"0! "#$! 89"0)+1! .'89).':'#+%! <.'89).'5:'#+! +1&'=2! >4'! $,:)#"+)#?! "&&.,"(4'%! +,! .'&.'%'#+!-".)"/)0)+1!".'! .90'%!@!(,#$)+),#%!"#$!-".)"#+! 0"/'0%@!"#5#,+"+),#%! )#! %,*+7".'! ".+)*"(+%! <.'&.'%'#+"+),#=2! >4'! &.)5:".1!'0':'#+%!+4"+!".'!%9/A'(+!,*!-".)"/)0)+1!".'!".(4)+'(5+9.'! "#$! (,:&,#'#+! <".+)*"(+=6! "#$! +4'! :"A,.)+1! ,*! +4'!%+9$)'%! %&'()*1! -".)"/)0)+1! )#! %'&"."+'$! $)%+)#(+! ".+)*"(+%!<,.+4,?,#"0)+1=2!!>4'! %'(,#$! (09%+'.6!!"#$%&'(!() *+) ,%-'%.'/'016! .'*'.%! +,!

+4'!7"1!-".)"/)0)+1!)%!"(+9"001!.'"0)B'$2!>4'!+.)??'.!+,!/)#$!-".)"#+%!:,%+01!(,:'%!*.,:!%+"C'4,0$'.%6!*,00,7'$!/1!+4'!'#-).,#:'#+!,*! +4'! %1%+':!<+.)??'.=2!>4'!$,:)#"#+! +'(45#)("0! "&&.,"(4'%! *,.! +4'! .'"0)B"+),#! ,*! -".)"/)0)+1! ".'! %'50'(+),#! ,*! -".)"#+%! *,00,7'$! /1! .',.?"#)B"+),#! ,*! %1%+':!".+)*"(+%! <.'"0)B"+),#! +'(4#)89'=2! >4'! +):'!74'#! -".)"#+%!".'! /,9#$! )%! "&&.,;):"+'01! '89"001! $)%+.)/9+'$! ,-'.!%,*+7".'! (,#%+.9(+),#@'-,09+),#! "#$! .9#+):'! <+):'! ,*!/)#$)#?=6!"#$!"0:,%+!"00!%+9$)'%!&.,-)$'!%,:'!C)#$!,*!+,,0!%9&&,.+!+,!/)#$!-".)"#+%!+,!-".)"+),#!&,)#+%!<"9+,:"+),#=2!

TABLE 9 Dimensions of Variability

!"#$%%&'()*+,-(.*'%/'+%0#$-.1.-/(.*'%*1%2/)./3.4.("%

5.6$'0.*'% 5*6/.'%7',63$)%*1%0(,+.$0%.'%#.4*(8%

D'89).':'#+!+1&'! E9#(+),#"0!<FG=!!!H9"0)+1!<FI=!

D'&.'%'#+"+),#!! E'"+9.'!:,$'0!<J=!!D90'%@(,#$)+),#%!<FG=!K".)"#+!0"/'0%@"##,+"+),#%!<I=!L.,*)0'%!<J=!M4"#?'!%('#".),%!<N=!

O.+)*"(+! P('#".),!<G=!39%)#'%%!&.,('%%!<F=!O.(4)+'(+9.'!<FJ=!M,:&,#'#+!<FJ=!M,$'!*."?:'#+!<G=!K".)"/0'!<N=!

Q.+4,?,#"0)+1! P'&"."+'$!<FG=!!R#+'?."+'$!<I=!9$-:/'.060%%

;*<%+*$0%2/)./3.4.("%(/=$%#4/-$%*)%.0%3)*,>:(%/3*,(%>.)??'.! P+"C'4,0$'.!<FJ=!

39%)#'%%!&.,('%%!<F=!!P1%+':!<G=!!S#-).,#:'#+!<T=!

D'"0)B"+),#!+'(4#)89'! D',.?"#)B"+),#!<I=!!P'0'(+),#!<FT=!K"09'!"%%)?#:'#+!<N=!!M,$'!?'#'."+),#!<N=!

>):'!,*!/)#$)#?! P,*+7".'!(,#%+.9(+),#@'-,09+),#!<FU=!D9#+):'!<FV=!

O9+,:"+),#! W"#9"0!<V=!!P':)5"9+,:"+)(!<FU=!!O9+,:"+)(!<X=!

!

4.3.3 Evaluation of Dimensions S-"09"+)#?!"!(0"%%)*)("+),#!*.":'7,.C!)%!)#!?'#'."0!"!$)**)5(90+! +"%C2! O! %1%+':"+)(! '-"09"+),#6! %9(4! "%! +4'! ,#'! &.,5&,%'$!/1!Y,!:'B5L'!.'B!ZGI[6!7,90$!.'89).'!"!*,.:"0!%&'(5)*)("+),#!,*!,9.!(0"%%)*)("+),#!*.":'7,.C6!74)(4!)%!/'1,#$!+4'! %(,&'! ,*! +4)%! %+9$12! 3"%'$! ,#! ZGI[6! 7'! $)%(9%%! +4.''!):&,.+"#+! '-"09"+),#! (.)+'.)"! ,*! +4'! &.,&,%'$! (0"%%)*)("5+),#\! (,#%)%+'#(16! (,:&0'+'#'%%6! "#$! ';&"#$"/)0)+12!M,#5%)%+'#(1! .'*'.%! +,!74'+4'.! )+! )%!&,%%)/0'! +,!,/+")#! (,#+."5$)(+,.1!(,#(09%),#%! *.,:!"&&01)#?! +4'!$):'#%),#%2!O*+'.!"#!)#)+)"0!&4"%'!,*!+4'!&)0,+6!7'!$'*)#'$!"((9."+'!$'%(.)&5+),#%! ,*! +4'! $):'#%),#%! "#$! +4').! ,&+),#%!7'.'! %+.")?4+5*,.7".$! +,! "&&012!]4)0'! +4'.'! ".'! ,/-),9%! .'0"+),#%4)&%!/'+7''#! +4'! $):'#%),#%6! /"%'$! ,#! ,9.! ';&'.)'#('%!7)+4!(,00'(+)#?!+4'!$"+"!$9.)#?!+4'!&)0,+6!7'!("#!(,#(09$'!+4"+!+4'! $):'#%),#%! (,-'.! $)%A,)#+! /9+! (,:&0':'#+".1! *"('+%!,*!-".)"/)0)+12!M,:&0'+'#'%%!.'*'.%! +,! +4'!(,-'."?'!,*! +4'!$):'#%),#%6! "#$! $):'#%),#%! +4"+! :"1! 4"-'! /''#! ,-'.50,,C'$2!O%!';&0")#'$6!+4'!)#+'#$'$!%(,&'!,*!+4'!(0"%%)*)("5+),#! )%!#,+! +,!/'!(,:&0'+'6!/9+! +,!$'*)#'!"! %'+!,*! (,.'!$)5:'#%),#%!+,!$'%(.)/'!"#$!(,:&".'!C'1!*"('+%!,*!-".)"/)05)+12!E.,:!,9.!';&'.)'#('%!7)+4!+4'!.'-)'7!)#!?'#'."0!"#$!+4'!&)0,+! )#!&".+)(90".6!7'!(,#(09$'!+4"+!+4'!(0"%%)*)("+),#!(,-'.%!"!.'&.'%'#+"+)-'!%'+!,*!(,.'!$):'#%),#%!*,.!$'%(.)/5)#?!-".)"/)0)+12!O%!7'!9%'$!,#01!"!%9/%'+!,*!&.):".1!%+9$5)'%6! +4'! ."#?'%! ,*! %,:'! ,*! +4'! $):'#%),#%! 7)00! #,+! /'!(,:&0'+'2!E,.!';":&0'6!:,.'!,&+),#%!*,.!.'"0)B"+),#!+'(45#)89'!:"1! "&&'".!74'#! +4'!$):'#%),#%!7)00! /'! "&&0)'$!+,! ,+4'.! %+9$)'%2! >4)%! /.)#?%! 9%! +,! ';&"#$"/)0)+16! 74)(4!.'*'.%! +,! +4'! "/)0)+1! *,.! ';+'#$)#?! +4'! (0"%%)*)("+),#! <)2'26!"$$)#?! #'7! $):'#%),#%! +,! +4'! (0"%%)*)("+),#! "#$! #'7!,&+),#%! +,! ';)%+)#?! $):'#%),#%=6!7)+4,9+! "0+'.)#?! +4'! %'+!,*!7'005$'*)#'$!&.,&'.+)'%2!>4'!^,&'#_!#"+9.'!,*!+4'!&.,5&,%'$!(0"%%)*)("+),#!"00,7%!.'%'".(4'.%! +,!'"%)01!"$$!#'7!$):'#%),#%!,.!,&+),#%!*,.!$):'#%),#%2!

4.3.4 Usefulness of the Classification >4'!(0"%%)*)("+),#!,**'.%!"!-,("/90".1!*,.!.'%'".(4'.%!"#$!'#?)#''.%!+4"+!"00,7%!%&'()*1)#?!"#$!(,:&".)#?!+4'!-".)5"/)0)+1!&.,&'.+)'%!9#$'.!(,#%)$'."+),#!)#!+4').!%+9$)'%!"#$!&.,A'(+%2!>4'!$):'#%),#%!("#!/'!9%'$!/,+4!$9.)#?!%1%+':!.'"0)B"+),#!"#$!.'-'.%'!'#?)#''.)#?!"(+)-)+)'%2!>4'!$):'#5%),#%! %9&&,.+! %,*+7".'! '#?)#''.%!7)+4! 4"#$0)#?! "#$!$'5%(.)/)#?! +4'! C'1! *"('+%! ,*! -".)"/)0)+1! )#! +4').! %1%+':%! ';5&0)()+016!74)(4!)%!(.9()"0!+,!:"C'!)#*,.:'$!$'()%),#%2!>4'!(0"%%)*)("+),#!"0%,!'#"/0'%!(,:&".)%,#!,*!-".)"/)05

)+1! %,09+),#%! "(.,%%! $)**'.'#+! %1%+':%! "#$! $)**'.'#+! $,5:")#%2! R*! ';+'#$'$! +,! "! *900! (0"%%)*)("+),#! +4"+! (,-'.%! "00!&.):".1!%+9$)'%!)#!+4)%!0)+'."+9.'!.'-)'76!+4'!(0"%%)*)("+),#!7,90$! /'! "! ?.'"+! "%%'+! *,.! )$'#+)*1)#?! .'0"+'$!7,.C! )#! "!%&'()*)(! ".'"!,*!-".)"/)0)+16! "#$! )$'#+)*1)#?! %&'()*)(! ".'"%!)#!-".)"/)0)+1! .'%'".(4! +4"+!$'%'.-'! *9.+4'.! "++'#+),#2!]'!0'"-'!+4)%!'**,.+!*,.!*9+9.'!7,.C2!R#!(,#(09%),#6!7'!/'0)'-'!+4"+!+4)%!':&).)("001!?.,9#$5

'$!(0"%%)*)("+),#!&.,-)$'%!"!%,0)$!%+'&!+,7".$%!"!9#)*1)#?6!)#+'?."+'$!&'.%&'(+)-'!,*!-".)"/)0)+1! )#! %,*+7".'!%1%+':%6!%&"##)#?! "(.,%%! (9..'#+01! $)%&"."+'! ,.! 0,,%'01! (,9&0'$!.'%'".(4!+4':'%! )#! +4'!%,*+7".'!'#?)#''.)#?!(,::9#)+12!P9(4!9#)*1)#?!&'.%&'(+)-'!4"%! +4'!&,+'#+)"0! +,! "(('0'."+'!+4'! ';(4"#?'%! ,*! )$'"%! "(.,%%! .'%'".(4! +4':'%6! .'$9()#?!


!

"#$!%&'()*)("+),#!,*!-".)"/)0)+12!3"%'$!,#!+4'!$"+"!(,00'(+5'$! )#! +4'! &)0,+6! 7'! ,/%'.-'! +4"+! +4'! .'89).':'#+%! +4"+!-".)"/)0)+1! "$$.'%%'%! ".'! "&&.,;):"+'01! '89"001! $)-)$'$!/'+7''#! *9#(+),#"0! "#$! 89"0)+1! .'89).':'#+%! <.'89).'5:'#+! +1&'=2! >4'! $,:)#"+)#?! "&&.,"(4'%! +,! .'&.'%'#+!-".)"/)0)+1!".'! .90'%!@!(,#$)+),#%!"#$!-".)"#+! 0"/'0%@!"#5#,+"+),#%! )#! %,*+7".'! ".+)*"(+%! <.'&.'%'#+"+),#=2! >4'! &.)5:".1!'0':'#+%!+4"+!".'!%9/A'(+!,*!-".)"/)0)+1!".'!".(4)+'(5+9.'! "#$! (,:&,#'#+! <".+)*"(+=6! "#$! +4'! :"A,.)+1! ,*! +4'!%+9$)'%! %&'()*1! -".)"/)0)+1! )#! %'&"."+'$! $)%+)#(+! ".+)*"(+%!<,.+4,?,#"0)+1=2!!>4'! %'(,#$! (09%+'.6!!"#$%&'(!() *+) ,%-'%.'/'016! .'*'.%! +,!

+4'!7"1!-".)"/)0)+1!)%!"(+9"001!.'"0)B'$2!>4'!+.)??'.!+,!/)#$!-".)"#+%!:,%+01!(,:'%!*.,:!%+"C'4,0$'.%6!*,00,7'$!/1!+4'!'#-).,#:'#+!,*! +4'! %1%+':!<+.)??'.=2!>4'!$,:)#"#+! +'(45#)("0! "&&.,"(4'%! *,.! +4'! .'"0)B"+),#! ,*! -".)"/)0)+1! ".'! %'50'(+),#! ,*! -".)"#+%! *,00,7'$! /1! .',.?"#)B"+),#! ,*! %1%+':!".+)*"(+%! <.'"0)B"+),#! +'(4#)89'=2! >4'! +):'!74'#! -".)"#+%!".'! /,9#$! )%! "&&.,;):"+'01! '89"001! $)%+.)/9+'$! ,-'.!%,*+7".'! (,#%+.9(+),#@'-,09+),#! "#$! .9#+):'! <+):'! ,*!/)#$)#?=6!"#$!"0:,%+!"00!%+9$)'%!&.,-)$'!%,:'!C)#$!,*!+,,0!%9&&,.+!+,!/)#$!-".)"#+%!+,!-".)"+),#!&,)#+%!<"9+,:"+),#=2!

TABLE 9 Dimensions of Variability

!"#$%%&'()*+,-(.*'%/'+%0#$-.1.-/(.*'%*1%2/)./3.4.("%

5.6$'0.*'% 5*6/.'%7',63$)%*1%0(,+.$0%.'%#.4*(8%

D'89).':'#+!+1&'! E9#(+),#"0!<FG=!!!H9"0)+1!<FI=!

D'&.'%'#+"+),#!! E'"+9.'!:,$'0!<J=!!D90'%@(,#$)+),#%!<FG=!K".)"#+!0"/'0%@"##,+"+),#%!<I=!L.,*)0'%!<J=!M4"#?'!%('#".),%!<N=!

O.+)*"(+! P('#".),!<G=!39%)#'%%!&.,('%%!<F=!O.(4)+'(+9.'!<FJ=!M,:&,#'#+!<FJ=!M,$'!*."?:'#+!<G=!K".)"/0'!<N=!

Q.+4,?,#"0)+1! P'&"."+'$!<FG=!!R#+'?."+'$!<I=!9$-:/'.060%%

;*<%+*$0%2/)./3.4.("%(/=$%#4/-$%*)%.0%3)*,>:(%/3*,(%>.)??'.! P+"C'4,0$'.!<FJ=!

39%)#'%%!&.,('%%!<F=!!P1%+':!<G=!!S#-).,#:'#+!<T=!

D'"0)B"+),#!+'(4#)89'! D',.?"#)B"+),#!<I=!!P'0'(+),#!<FT=!K"09'!"%%)?#:'#+!<N=!!M,$'!?'#'."+),#!<N=!

>):'!,*!/)#$)#?! P,*+7".'!(,#%+.9(+),#@'-,09+),#!<FU=!D9#+):'!<FV=!

O9+,:"+),#! W"#9"0!<V=!!P':)5"9+,:"+)(!<FU=!!O9+,:"+)(!<X=!

!

4.3.3 Evaluation of Dimensions S-"09"+)#?!"!(0"%%)*)("+),#!*.":'7,.C!)%!)#!?'#'."0!"!$)**)5(90+! +"%C2! O! %1%+':"+)(! '-"09"+),#6! %9(4! "%! +4'! ,#'! &.,5&,%'$!/1!Y,!:'B5L'!.'B!ZGI[6!7,90$!.'89).'!"!*,.:"0!%&'(5)*)("+),#!,*!,9.!(0"%%)*)("+),#!*.":'7,.C6!74)(4!)%!/'1,#$!+4'! %(,&'! ,*! +4)%! %+9$12! 3"%'$! ,#! ZGI[6! 7'! $)%(9%%! +4.''!):&,.+"#+! '-"09"+),#! (.)+'.)"! ,*! +4'! &.,&,%'$! (0"%%)*)("5+),#\! (,#%)%+'#(16! (,:&0'+'#'%%6! "#$! ';&"#$"/)0)+12!M,#5%)%+'#(1! .'*'.%! +,!74'+4'.! )+! )%!&,%%)/0'! +,!,/+")#! (,#+."5$)(+,.1!(,#(09%),#%! *.,:!"&&01)#?! +4'!$):'#%),#%2!O*+'.!"#!)#)+)"0!&4"%'!,*!+4'!&)0,+6!7'!$'*)#'$!"((9."+'!$'%(.)&5+),#%! ,*! +4'! $):'#%),#%! "#$! +4').! ,&+),#%!7'.'! %+.")?4+5*,.7".$! +,! "&&012!]4)0'! +4'.'! ".'! ,/-),9%! .'0"+),#%4)&%!/'+7''#! +4'! $):'#%),#%6! /"%'$! ,#! ,9.! ';&'.)'#('%!7)+4!(,00'(+)#?!+4'!$"+"!$9.)#?!+4'!&)0,+6!7'!("#!(,#(09$'!+4"+!+4'! $):'#%),#%! (,-'.! $)%A,)#+! /9+! (,:&0':'#+".1! *"('+%!,*!-".)"/)0)+12!M,:&0'+'#'%%!.'*'.%! +,! +4'!(,-'."?'!,*! +4'!$):'#%),#%6! "#$! $):'#%),#%! +4"+! :"1! 4"-'! /''#! ,-'.50,,C'$2!O%!';&0")#'$6!+4'!)#+'#$'$!%(,&'!,*!+4'!(0"%%)*)("5+),#! )%!#,+! +,!/'!(,:&0'+'6!/9+! +,!$'*)#'!"! %'+!,*! (,.'!$)5:'#%),#%!+,!$'%(.)/'!"#$!(,:&".'!C'1!*"('+%!,*!-".)"/)05)+12!E.,:!,9.!';&'.)'#('%!7)+4!+4'!.'-)'7!)#!?'#'."0!"#$!+4'!&)0,+! )#!&".+)(90".6!7'!(,#(09$'!+4"+!+4'!(0"%%)*)("+),#!(,-'.%!"!.'&.'%'#+"+)-'!%'+!,*!(,.'!$):'#%),#%!*,.!$'%(.)/5)#?!-".)"/)0)+12!O%!7'!9%'$!,#01!"!%9/%'+!,*!&.):".1!%+9$5)'%6! +4'! ."#?'%! ,*! %,:'! ,*! +4'! $):'#%),#%! 7)00! #,+! /'!(,:&0'+'2!E,.!';":&0'6!:,.'!,&+),#%!*,.!.'"0)B"+),#!+'(45#)89'!:"1! "&&'".!74'#! +4'!$):'#%),#%!7)00! /'! "&&0)'$!+,! ,+4'.! %+9$)'%2! >4)%! /.)#?%! 9%! +,! ';&"#$"/)0)+16! 74)(4!.'*'.%! +,! +4'! "/)0)+1! *,.! ';+'#$)#?! +4'! (0"%%)*)("+),#! <)2'26!"$$)#?! #'7! $):'#%),#%! +,! +4'! (0"%%)*)("+),#! "#$! #'7!,&+),#%! +,! ';)%+)#?! $):'#%),#%=6!7)+4,9+! "0+'.)#?! +4'! %'+!,*!7'005$'*)#'$!&.,&'.+)'%2!>4'!^,&'#_!#"+9.'!,*!+4'!&.,5&,%'$!(0"%%)*)("+),#!"00,7%!.'%'".(4'.%! +,!'"%)01!"$$!#'7!$):'#%),#%!,.!,&+),#%!*,.!$):'#%),#%2!

4.3.4 Usefulness of the Classification >4'!(0"%%)*)("+),#!,**'.%!"!-,("/90".1!*,.!.'%'".(4'.%!"#$!'#?)#''.%!+4"+!"00,7%!%&'()*1)#?!"#$!(,:&".)#?!+4'!-".)5"/)0)+1!&.,&'.+)'%!9#$'.!(,#%)$'."+),#!)#!+4').!%+9$)'%!"#$!&.,A'(+%2!>4'!$):'#%),#%!("#!/'!9%'$!/,+4!$9.)#?!%1%+':!.'"0)B"+),#!"#$!.'-'.%'!'#?)#''.)#?!"(+)-)+)'%2!>4'!$):'#5%),#%! %9&&,.+! %,*+7".'! '#?)#''.%!7)+4! 4"#$0)#?! "#$!$'5%(.)/)#?! +4'! C'1! *"('+%! ,*! -".)"/)0)+1! )#! +4').! %1%+':%! ';5&0)()+016!74)(4!)%!(.9()"0!+,!:"C'!)#*,.:'$!$'()%),#%2!>4'!(0"%%)*)("+),#!"0%,!'#"/0'%!(,:&".)%,#!,*!-".)"/)05

)+1! %,09+),#%! "(.,%%! $)**'.'#+! %1%+':%! "#$! $)**'.'#+! $,5:")#%2! R*! ';+'#$'$! +,! "! *900! (0"%%)*)("+),#! +4"+! (,-'.%! "00!&.):".1!%+9$)'%!)#!+4)%!0)+'."+9.'!.'-)'76!+4'!(0"%%)*)("+),#!7,90$! /'! "! ?.'"+! "%%'+! *,.! )$'#+)*1)#?! .'0"+'$!7,.C! )#! "!%&'()*)(! ".'"!,*!-".)"/)0)+16! "#$! )$'#+)*1)#?! %&'()*)(! ".'"%!)#!-".)"/)0)+1! .'%'".(4! +4"+!$'%'.-'! *9.+4'.! "++'#+),#2!]'!0'"-'!+4)%!'**,.+!*,.!*9+9.'!7,.C2!R#!(,#(09%),#6!7'!/'0)'-'!+4"+!+4)%!':&).)("001!?.,9#$5

'$!(0"%%)*)("+),#!&.,-)$'%!"!%,0)$!%+'&!+,7".$%!"!9#)*1)#?6!)#+'?."+'$!&'.%&'(+)-'!,*!-".)"/)0)+1! )#! %,*+7".'!%1%+':%6!%&"##)#?! "(.,%%! (9..'#+01! $)%&"."+'! ,.! 0,,%'01! (,9&0'$!.'%'".(4!+4':'%! )#! +4'!%,*+7".'!'#?)#''.)#?!(,::9#)+12!P9(4!9#)*1)#?!&'.%&'(+)-'!4"%! +4'!&,+'#+)"0! +,! "(('0'."+'!+4'! ';(4"#?'%! ,*! )$'"%! "(.,%%! .'%'".(4! +4':'%6! .'$9()#?!

Requirement Type •  Functional •  E.g. support for different types of missions in a command and

controller simulator provided by optional components

Quality E.g. an alternative service is selected and invoked when the original service is not available

Representation •  Feature model •  E.g., feature graph with links between features and requirement

specifications to represent variability •  Rules/conditions •  E.g., expressions that refer to elements of design patterns to express

variability of patterns •  Variant labels/annotations •  E.g., component annotated with properties (e.g., required resources) to

discriminate between alternatives enabling dynamic selection of variants •  Profiles •  E.g., use of meta-data (composition policy) together with context data to

dynamically select and add service (e.g., authentication) •  Change scenarios

E.g., scenarios that specify a specific set of choices supporting decision making for variability of a product line

Trigger •  Stakeholder •  E.g., user commands trigger the selection of variants of GUI forms that

guide the user through the process of generating legal documents •  Business process •  E.g., firing conditions of a transition in a business process trigger the

execution of a variant of a region of the process •  System •  E.g., a service system optimizes its workflow by replacing services

with services of better quality when the current quality differs from the prediction by more than a give threshold

•  Environment •  E.g., changes in the context, such as quality of the network or physical

location trigger the system to replace components by variants that guarantee better utility

Time of Binding •  Software construction/evolution •  E.g., an analyst selects the best architecture among a set of

variants during system development based on the impact analysis for a set of modifiability scenarios

•  Runtime •  E.g., dynamic content adaptations of mobile devices based on

device profiles and users’ service level agreements by selecting matching services from a repository of variants

Conclusions •  Variability is a key issue of many software systems today, if not all

•  SLR aimed at studying variability handing in software systems beyond SPL

•  Resulting classification offers vocabulary to support specification of variability properties under consideration •  - Enables comparison of variability solutions across different systems/domains

- Potential asset for identifying related work in specific areas of variability

•  Step towards a unifying and integrated perspective on variability in software systems •  - Spanning currently disparate or loosely connected fields/communities •  - Potential to accelerate exchange of ideas and solutions between fields

Examples in Our Planned & Ongoing Research

•  Variability as a mechanism to support self-adaptation

•  Focus on multiple concerns •  MAPE-K feedback loop(s): •  - Maintain(s) models of different quality properties •  - Reason(s) about tradeoffs of architectural configurations •  - Select(s) most suitable variant to adapt system to the given context

Examples in Our Planned & Ongoing Research •  Self-adaptation as a variability mechanism

•  Autonomic software product lines* •  - Separate domain logic from adaptation logic in SPL

- Adaptation logic = reusable building blocks of feedback loops - Complemented with supporting artifacts

Domain quality attribute scenarios Responsibility structures Reasoning framework - Supporting systematic reuse for quality properties of interest

* Abbas, Andersson, Löwe. Autonomic Software Product Lines (ASPL), ECSA 2010 Abbas, Andersson, Weyns, Modeling variability in product lines using domain quality attribute scenarios,, WICS/ECSA Companion Volume, 2011

Thank you! Questions, comments?

Galster, Weyns, Tofan, Michalik, Avgeriou, Variability in software systems, a systematic literature review, IEEE Transactions on Software Engineering, 40(3):282-306, 2014

http://homepage.lnu.se/staff/daweaa/papers/2013TSE.pdf Technical report with all SLR material:

http://iwi.eldoc.ub.rug.nl/FILES/root/2013/VariaSoftSystGalster/variability_in_software_systems_data.pdf

Bibliography •  van Gurp, Bosch, Svahnberg, On the Notion of variability in software

product lines, WICSA 2011

•  Galster, Weyns, Tofan, Michalik, Avgeriou, Variability in software systems, a systematic literature review, IEEE Transactions on Software Engineering, 40(3):282-306, 2014

•  Abbas, Andersson, Löwe, Autonomic Software Product Lines (ASPL), ECSA 2010

•  Abbas, Andersson, Weyns, Modeling variability in product lines using domain quality attribute scenarios, WICS/ECSA Companion Volume, 2011

•  Nadeem Abbas, Andersson, Weyns Knowledge evolution in autonomic software product lines, DSPL, 2011

variability: from software product lines to self-adaptive...

Documents