sip: optimal product selection from feature models using ... · • ibea performed best. abdel...
TRANSCRIPT
![Page 1: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/1.jpg)
SIP: Optimal Product Selection from Feature Models using Many-Objective Evolutionary Optimisation
(1) BrunelUniversityLondon,UK(2) UniversityofSeville,Spain
(3) NorthwesternPolytechnicalUniversity,China
RobertM.Hierons(1),MiqingLi(1),XiaoHuiLiu(1),SergioSegura(2),WeiZheng(3)
![Page 2: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/2.jpg)
Introduction
Problem
Approach
Results
Conclusions
![Page 3: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/3.jpg)
IntroductionProduct lines
![Page 4: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/4.jpg)
IntroductionSoftware product lines
• Family of Products
• Built from a set of reusable assets
• There can be constraints
• Evidence of significant use in industry
![Page 5: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/5.jpg)
Catalogue
Shopping cart
Payment
Search
Login
Banners
IntroductionSoftware product lines
![Page 6: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/6.jpg)
IntroductionFeature Models
• Set of features
• There can be constraints such as:• Or/Alternative• Requires/Excludes• Mandatory
• A valid product is:• A set of features that satisfies the constraints
• An FM is typically represented as a tree
![Page 7: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/7.jpg)
Mandatory Optional
Or
Requires
Feature
= {E-shop + Catalogue + Payment + Credit Card + Security + High + Search}
Excludes
Alternative
IntroductionFeature Models
![Page 8: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/8.jpg)
14How many products?
Automated analysis of feature models:Computer-aided extraction of information from FMs
Introduction
![Page 9: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/9.jpg)
No, two constraints are violated
Is P a valid product?
Automated analysis of feature models:Computer-aided extraction of information from FMs
P = {Mobile Phone + Screen + Colour + Media + Camera}
Introduction
![Page 10: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/10.jpg)
Introduction
• Much interest in the automated analysis of feature models.
• Catalogues with up to 30 analysis operations
• Tools that implement these operations• e.g. FaMa Framework, SPLOT, …
• Automation is via e.g. SAT solvers.
Feature models
![Page 11: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/11.jpg)
Connectivity
WifiBluetooth
USB
Name: CostDomain: RealValue: 85.5
Name: MaxSpeedDomain: RealValue: 3.6
Name: MemoryDomain: IntegerValue: 725
Name: MemoryDomain: IntegerValue: 425
Name: CostDomain: RealValue: 50
Name: MaxSpeedDomain: RealValue: 2.1
Name: CostDomain: RealValue: 35.5
Name: MaxSpeedDomain: RealValue: 12
Name: MemoryDomain: IntegerValue: 179
IntroductionAttributed feature models
![Page 12: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/12.jpg)
Introduction
Problem
Approach
Results
Conclusions
![Page 13: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/13.jpg)
Problem
• Usually, it is not feasible to produce all valid products.
• Reasons for choosing a particular product:
• What product to release first• many features, low cost, …
• What product to test• different combinations of features, low cost …
Motivation
![Page 14: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/14.jpg)
Problem
• We require a valid product:• We want to minimise the number of constraints failed.
• We want to optimise certain properties eg:• Cost (of building product)• Number of features• Historical faults
Motivation
![Page 15: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/15.jpg)
Ex: “Find a product that minimises the number ofconstraints violated, maximises the number of features,minimises cost, minimises the number of known defects,and minimises the number of changes since the lastrelease.”
Optimal product selection ProductAttributed
feature model
Objective functions
Problem
![Page 16: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/16.jpg)
Problem
• Several aspects (objective functions) to optimise.
• So: a multi-objective optimisation problem.
• We could weight the objectives and optimise but ...• The choice of values for weights would be important.• How would we choose the weights?
![Page 17: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/17.jpg)
Pareto Dominance
• Candidate (product) C1 dominates candidate C2 if:• C1 is at least as good as C2 on all criteria• C1 is better than C2 on at least one criterion.
• Used by some evolutionary optimisation algorithms.• Aim is:
• get as close as possible to the Pareto front.
• Returns a set of solutions:• These are incomparable under Pareto dominance• Provide alternative compromises.
![Page 18: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/18.jpg)
• Maximising – which dominate others
Pareto Dominance
![Page 19: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/19.jpg)
Many-objective problems
• Many-objective problems• Those with four or more objectives• Typically, much more difficult to solve.
• Normally, Pareto dominance becomes ineffective
• There are algorithms that do not use Pareto dominance• e.g. IBEA, SPEA2+SDE, MOEA/D
![Page 20: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/20.jpg)
Previous workInitial work
• Recognised that it is a many-objective problem• Applied several evolutionary multi/many-objective
optimisation algorithms (EOMs) including:• NSGA-II, SPEA2, IBEA
• Used two feature models, up to 290 features.
• Difficult to find valid products.• IBEA performed best.
Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based Software Engineering: A Case Study in Software Product Lines. 2013 International Conference on Software Engineering (ICSE ’13)
![Page 21: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/21.jpg)
Previous workEnhancements
• Improvements by Sayad et al. included:• Removing core features (those that are in all valid
products)• Removing dead features (those that are in no valid
products)• Planting an initial seed (a valid product).
Abdel Salam Sayyad, Joseph Ingram, Tim Menzies, and Hany Ammar. Scalable product line configuration: A straw to break the camel’s back. 2013 IEEE/ACM International Conference on Automated Software Engineering. 465–474.
Abdel Salam Sayyad. 2014. Evolutionary Search Techniques with Strong Heuristics for Multi-Objective Feature Selection in Software Product Lines. Ph.D. Dissertation. West Virginia University.
![Page 22: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/22.jpg)
• Evaluated on more models:• Seven models.• Included one large model (6,888 features) – the Linux
kernel feature model.
• Required an initial phase to find a seed• took approximately three hours for the large model.
![Page 23: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/23.jpg)
Previous workEnhancements
• Additional enhancements involved:• Adapting mutation (cannot be applied in certain
circumstances, avoiding some invalid products).• Placing greater emphasis on the number of constraints
that fail (a weight)
• New mutation and replacement operators that used a SAT solver
Abdel Salam Sayyad, Joseph Ingram, Tim Menzies, and Hany Ammar. Scalable product line configuration: A straw to break the camel’s back. 2013 IEEE/ACM International Conference on Automated Software Engineering. 465–474.
Christopher Henard, Mike Papdakis, Mark Harman, and Yves Le Traon. Combining Multi-Objective Search and Constraint Solving for Configuring Large Software product Lines. 2015 International Conference on Software Engineering (ICSE ’15).
![Page 24: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/24.jpg)
Previous workSummary
• Search as a many-optimisation problem.• Found to be difficult.• A number of enhancements help.
• However:• Enhancements complicates the approach• New operators are more computationally expensive• Can be specific to certain optimisation algorithms
• Can we produce a simpler search-based approach?
R. M. Hierons, M. Li, X. Liu, S. Segura, and W. Zheng: SIP: Optimal Product Selection from Feature Models using Many-Objective Evolutionary Optimisation, ACM Transactions on Software Engineering and Methodology, 25 2, 2016.
![Page 25: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/25.jpg)
Our initial work
• Initial motivation:• Try different evolutionary algorithms – those that have
been found to be good at many-objective optimisation.
• We wanted approaches that return a range of solutions• Provide alternative products – user can choose
between these.
• Initial experiments on two case studies:• WebPortal and E-Shop
• It was difficult to find valid products (similar to earlier work).
![Page 26: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/26.jpg)
Finding products
• We conjectured that two aspects made this difficult:• The presence of many constraints.• As the number of objectives increases, there is less
evolutionary pressure towards valid products.
• We aimed to address these two points.
• We also aimed to use case studies with realistic attribute values.
![Page 27: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/27.jpg)
Novelencoding(ShrInk)
Prioritizedobjectives(Prioritise)
Realisticcasestudies
1+nEMOalgorithmscomparison
ApproachSIP: Shrink and Prioritise
![Page 28: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/28.jpg)
Prioritizedobjectives(Prioritise)
Realisticcasestudies
1+nEMOalgorithmscomparison
Novelencoding(ShrInk)
ApproachSIP: Shrink and Prioritise
![Page 29: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/29.jpg)
GPS
Routing Interface
Screen
LCDTouch
Traffic avoiding
Auto-rerouting Keyboard3D map
Radio
AM FM Digital
1 1 0 1 1 0 0 0 0 1 1 1 0 1Direct
ApproachStandard encoding
![Page 30: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/30.jpg)
ApproachNovel encoding
• Basic idea:
• No need to include Core features (Sayad at al. and Henard et al. also observed this).
• Did not remove dead features• Feature models should not have these.• They should be found and removed in advance.
• Sometimes we don’t need to include some parents (e.g. if children are in a OR relationship).
![Page 31: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/31.jpg)
GPS
Routing Interface
Screen
LCDTouch
Traffic avoiding
Auto-rerouting Keyboard3D map
Radio
AM FM Digital
1 1 0 1 1 0 0 0 0 1 1 1 0 1Direct
0 1 1 0 0 0 1 0 1SIP
ApproachNovel encoding
![Page 32: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/32.jpg)
GPS
Routing Interface
Screen
LCDTouch
Traffic avoiding
Auto-rerouting Keyboard3D map
Radio
AM FM Digital
Encoding
GPS
Rou
ting
3D m
ap
Aut
o-re
rout
ing
Traf
fic a
void
ing
Rad
io
FM AM
Dig
ital
Inte
rfac
e
Key
boar
d
Scre
en
Touc
h
LCD
Direct ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
SIP ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
ApproachNovel encoding
![Page 33: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/33.jpg)
Approach
• This helps since:
• We have a smaller search space.
• Some constraints can be enforced through the encoding.• Should make it easier to generate valid products.
Novel encoding
![Page 34: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/34.jpg)
Realisticcasestudies
EMOalgorithmscomparison
Novelencoding(ShrInk)
Prioritizedobjectives(Prioritise)
1+n
ApproachSIP: Shrink and Prioritise
![Page 35: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/35.jpg)
ApproachPrioritised objectives: 1+n
• Traditional, (n+1)-approach:• All of the objectives are considered together.
• The 1+n-approach:• We first consider the number of constraints failed, only
then the remaining (n) objectives.• Motivation: invalid products have no value.
• Implemented as:• First compare individuals using number of constraints
failed.• If equal on this, then compare as normal (for the given
algorithm).
![Page 36: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/36.jpg)
Approach
• Mating selection:• Compare individuals on number of constraints failed.• If equal on this, then compare as normal (for the given
algorithm).• Environmental selection (determining which survive):
• Group individuals on number of constraints failed.• Keep the best groups.• Use normal approach (for EMO) if we need to select a
subset from a group.
Implementing 1+n
![Page 37: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/37.jpg)
Minimise size
Maximise features
Minimise complexity
Minimise changes
Minimise violations
…
n+1n+1 1+n
ApproachPrioritised objectives
![Page 38: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/38.jpg)
EMOalgorithmscomparison
Novelencoding(ShrInk)
Prioritizedobjectives(Prioritise)
1+nRealisticcase
studies
ApproachSPI: Shrink and Prioritise
![Page 39: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/39.jpg)
Features 48 79
Attributes 22 17
Products 2.09 • 109 66,528
Published FMs with random attributes.Randomly generated FM with 10,000 features.
ApproachRealistic case studies
![Page 40: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/40.jpg)
Novelencoding(ShrInk)
Prioritizedobjectives(Prioritise)
1+nRealisticcase
studiesEMOalgorithmscomparison
ApproachSIP: Shrink and Prioritise
![Page 41: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/41.jpg)
Evaluation
• Four used:• Direct encoding (one bit for each feature). Used in
initial work by Sayyard et al.• Core encoding – do not include core features. Used by
Sayyad et al. and Henard et al.• Hierarchical encoding – do not include parents unless
necessary• The (proposed) novel encoding
• Combines the above enhancements• (core + hierarchical)
Encodings
![Page 42: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/42.jpg)
NSGA-II[Deb et al. 2002]
IBEA [Zitzler et al. 2004]
SPEA2+SDE [Li et al. 2014]
MOEA/D-WS[Zhang and Li 2007]
MOEA/D-TCH[Zhang and Li 2007]
MOEA/D-PBI[Zhang and Li 2007]
EvaluationEMO algorithms
![Page 43: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/43.jpg)
Evaluation
• We initially used:• (n+1)-approach• Direct encoding
• Two feature models (E-shop and WebPortal).
• Found that:• All EMOs returned relatively few valid products.
• Motivated the enhancements.
Initial experiments
![Page 44: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/44.jpg)
Research Questions
• RQ1: How does the encoding affect performance?• RQ2: Does the 1+n approach make it easier to find
valid products?• RQ3: Are some EMO algorithms more effective than
others?• RQ4: Is the relative performance similar with realistic
attribute values?• RQ5: Is the relative performance similar for larger
models?• RQ6: How does the number of valid products returned
vary?• RQ7: How does execution time differ?
![Page 45: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/45.jpg)
Evaluation
SPL # Features # CTC # AttributesPer feature
BerkleyDB 13 0 4ERS 36 0 7WebPortal 43 6 4E-shop 290 21 4Drupal 48 21 22AmazonEC2 79 0 17Random(generated by SPLAR)
10,000 0 4
• All feature models were written in the SXFM format.
First four previously used
Set of Subjects
![Page 46: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/46.jpg)
Evaluation
• For the four previously used models:• We used the same five objectives as previous
experiments• For Amazon and Drupal, two experiments for each:
• We used the same five objectives (synthetic values).• We had eight objectives for each (realistic values).
• For the large randomly generated model:• The same five objectives as previous work (for
comparison).
• All sets included correctness (number of constraints not satisfied).
Objectives
![Page 47: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/47.jpg)
Subjects with realistic attributes
• Two case studies.
• Drupal:• Previously, 22 non-functional attributes extracted from
GIT repository.
• Amazon EC2:• Configuration space of Amazon Elastic Computing
Service.• Attributes were not fixed – there are constraints.• We randomly assigned values that satisfied the
constraints.
![Page 48: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/48.jpg)
Evaluation
• Each experiment repeated 30 times.• Termination criterion: 50,000 fitness evaluations• Population size:
• 100 for most• For MOEA/D – closest possible integer to 100
(depends on number of objectives).
• Parameter values used:• Those recommended in the literature.
Experimental setup
![Page 49: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/49.jpg)
Evaluation
• Three metrics:• Hypervolume (HV): the volume of the objective
space between the solutions and a reference point.• Only used the valid solutions returned.
• The number of executions that returned at least one valid product (VN).
• Mean rate of valid individuals in the final population (VR).
• Note that HV and VR were averaged over populations that had at least one valid solution.
Performance metrics
![Page 50: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/50.jpg)
Hypervolume
• Influenced by:• Choice of reference point.• Scaling of space (dimensions).
• Reference point used:• Nadir point of the problem’s range (the point
constructed with the worst value on each objective).
• Scaling:• we normalised the objective values according to
the range of values in the objective space.
![Page 51: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/51.jpg)
Hypervolume
• Computed exact values when fewer than eight objectives.
• Literature suggests this is infeasible with seven or more objectives
• For the large example (eight objectives) • We estimated the HV result.• We used Monte Carlo sampling with
10,000,000 sampling points.
![Page 52: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/52.jpg)
Results
![Page 53: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/53.jpg)
Results
• An example, direct encoding and (n+1)
Results with E-shop
Algorithm HV VN (/30) VRNSGA-II 0.0000 0 0%IBEA 0.0000 0 0%MOEA/D-WS 0.016184 26 21.42%MOEA/D-TCH 0.0000 0 0%MOEA/D-PBI 0.18815 4 10.50%SPEA2+SDE 0.0000 0 0%
![Page 54: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/54.jpg)
Results
• An example, novel encoding and (n+1)
Results with E-shop
Algorithm HV VN (/30) VRNSGA-II 0.136545 13 100%IBEA 0.169191 16 100%MOEA/D-WS 0.184810 5 100%MOEA/D-TCH 0.199697 1 100%MOEA/D-PBI 0.166157 5 100%SPEA2+SDE 0.144341 15 100%
![Page 55: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/55.jpg)
Results
• An example, core encoding and (n+1)
Results with E-shop
Algorithm HV VN (/30) VRNSGA-II 0.003343 28 2.07%IBEA 0.26741 30 33.91%MOEA/D-WS 0.074223 30 26.74%MOEA/D-TCH 0.0000 0 0%MOEA/D-PBI 0.070765 30 30.62%SPEA2+SDE 0.0000 0 0%
![Page 56: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/56.jpg)
Results
• An example, novel encoding and 1+n
Results with E-shop
Algorithm HV VN (/30) VRNSGA-II 0.162943 30 100%IBEA 0.190496 30 100%MOEA/D-WS 0.222875 30 100%MOEA/D-TCH 0.226257 30 100%MOEA/D-PBI 0.192485 30 100%SPEA2+SDE 0.159930 30 100%
![Page 57: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/57.jpg)
Results
• Direct encoding:• With, (n+1), only two algorithms returned valid
solutions (VR of 4/30 and 26/30).• With 1+n, all (sometimes) produced valid solutions.
• Hierarchical and core encodings:• More effective, core better than hierarchical.
• Novel encoding:• Almost all EMOs had 100% VR and VN.
Results with E-shop
![Page 58: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/58.jpg)
Results
• HV for novel:• With 1+n, the best results were with some variants
of MOEA/D.• With (n+1), IBEA best (similar to previous work).
• We tried E-shop with 500,000 evaluations:• Better results.• However, only 1+n (with core/novel) gave 100%
VN for all algorithms.
Results with E-shop
![Page 59: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/59.jpg)
Results
• Results similar to E-shop but better.• Seems to be an easier problem.
• 1+n approach always gave 100% VN and VR.
• (n+1) approach gave 100% VN, VR for core and novel encodings.
• IBEA tended to give the best HV.
Results with Webportal
![Page 60: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/60.jpg)
Results
• Similar results.
• 1+n approach and novel always gave 100% VN and VR.
• The only combination that did this.
• HV comparisons were quite variable. Highest values for:• SPEA2+SDE (Amazon, Berkley)• IBEA (Drupal)• NSGA-II (ERS).
Other results (five objectives)
![Page 61: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/61.jpg)
Results
• Drupal:• Relatively easy to find valid products.• Only (n+1)/direct failed to have 100% VR,VN for all
EMOs.
• Amazon:• Results were much poorer.• Almost no experiments produced valid products
with direct/(n+1).• Most were effective with novel/1+n.
Other results (five objectives)
![Page 62: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/62.jpg)
• Results for 1+n with the novel encoding.
Results
Algorithm HV VN (/30) VRNSGA-II 0.001844 30 100%IBEA 0.001897 17 100%MOEA/D-WS 0.001877 30 100%MOEA/D-TCH 0.001688 30 100%MOEA/D-PBI 0 0 0%SPEA2+SDE 0.002001 30 100%
Amazon (real attributes)
![Page 63: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/63.jpg)
Results
• Search found this difficult.• Only novel/1+n returned valid solutions
• All others had 0% for VN, VR.
Algorithm HV VN (/30) VRNSGA-II 0.014588 24 100%IBEA 0.020762 25 100%MOEA/D-WS 0.042142 15 100%MOEA/D-TCH 0.025037 19 100%MOEA/D-PBI 0.042513 18 0%SPEA2+SDE 0.018173 28 100%
Large, randomly generated model
![Page 64: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/64.jpg)
Results
Model Best performingMOEA
E-shop. 50k MOEA/D-TCHE-shop, 500k MOEA/D-WSWebPortal IBEAAmazon SPEA2+SDEBerkley SPEA2+SDEDrupal IBEAERS NSGA-IIDrupal, real SPEA2+SDEAmazon, real SPEA2+SDELarge, random SPEA2+SDE
• Comparing HV values.
• Not all differences are statistically significant (many were).
Best performing (novel, 1+n)
![Page 65: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/65.jpg)
Statistical tests used
• First tested the hypothesis that all EMO algorithms perform equally (different techniques, same approach):• Kruskal-Wallis test. Always rejected.
• Then used pairwise comparisons:• post-hoc Kruskal-Wallis test• with Bonferroni adjustment
• Effect size using:• Mann-Whitney U test• many of the differences had large effect sizes
(between 0.8 and 0.9).
![Page 66: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/66.jpg)
• Time in seconds.• For novel encoding; results similar (but usually slower)
for other encodings.
Results
EMO (n+1) 1+nNSGA-II 101.76 98.758IBEA 154.399 131.465MOEA/D-WS 135.98 104.44MOEA/D-TCH 132.9 99.7MOEA/D-PBI 131.9 103SPEA2+SDE 247.786 223.777
Time (large model)
![Page 67: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/67.jpg)
Results
The encoding does affect performance, with the proposed encoding proving to be most effective.
The 1+n approach, where one objective is given priority over the rest, is more effective that treating all the objectives equally.
There is not clear “best” EMO algorithm.
The choice of the encoding and approach have more impact on performance than the choice of EMO algorithm
![Page 68: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/68.jpg)
Future Work• Incorporate novel operators from e.g. Henard et al.
• Further enhance the representation:• Aim to encode more constraints.
• Consider approaches that use repair.
• Additional studies:• Others with realistic attribute values – do the approaches
perform less well on these?• Varying the number of objectives.
![Page 69: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/69.jpg)
§ The encoding affects performance, with the proposed encoding providing to be most effective.
§ The 1+n approach is more effective than the (n+1) approach.
§ Both enhancements help.
§ There is no clear “best” EMO algorithm.
§ SPEA2+SDE performs best most often (and for the most difficult) but takes more time.
Summary
![Page 70: SIP: Optimal Product Selection from Feature Models using ... · • IBEA performed best. Abdel Salam Sayyad, Tim Menzies, and Hany Ammar. On the Value of User Preferences in Search-based](https://reader036.vdocuments.net/reader036/viewer/2022070921/5fba06e4c89b7b013f4d51ef/html5/thumbnails/70.jpg)
SIP: Optimal Product Selection from Feature Models using Many-Objective Evolutionary Optimisation
(1) BrunelUniversityLondon,UK(2) UniversityofSeville,Spain
(3) NorthwesternPolytechnicalUniversity,China
RobertM.Hierons(1),MiqingLi(1),XiaoHuiLiu(1),SergioSegura(2),WeiZheng(3)
Thanks!