benchmarking for graph transformation
DESCRIPTION
Benchmarking for Graph Transformation. Gergely Varró gervarro @cs.bme.hu Andy Schürr [email protected] Dániel Varró [email protected]. Talk o verview. Paradigm & tool features. GT overview. Introduction. Benchmarking terminology. Measurements. Model transformation in MDD. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/1.jpg)
Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Benchmarking for Graph Transformation
Gergely Varró [email protected]
Andy Schü[email protected]ániel Varró [email protected]
![Page 2: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/2.jpg)
2Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
![Page 3: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/3.jpg)
3Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Model transformation in MDD• Model Driven Development (MDD)
– thorough system modeling before implementation– precise and visual SW models
• on a high abstraction level• from different viewpoints
– automatic code generation• Model transformation
– automated translations within and between visual modeling languages
• MDD requirements on model transformation:– Specification: high-level specification language– Execution: efficient algorithms and techniques– Industrialization: extensive tool support
• Upcoming standard: QVT
![Page 4: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/4.jpg)
4Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Specification techniques for MT
• QVT• Main features:
– object based patterns– precondition– unidirectional– bidirectional
• Problems:– immature tool support– lack of experience
• GT• Main features:
– similar to QVT– wide tool support
• Problems:– non-bidirectional
(except for TGGs)
• Pattern matching phase:– most critical issue
from performance viewpoint
– same as in QVT
![Page 5: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/5.jpg)
5Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Benchmarking
• Aim:– systematic and reproducable measurements– on performance– under different and precisely defined
circumstances
• Overall goal: help system engineers in selecting tools
• Popular approach in different fields– AI– relational databases– rule-based expert systems
![Page 6: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/6.jpg)
6Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Benchmarking in model transformation
• Specification examples for MT– Goal: assessing expressiveness
– UML-to-XMI, object-relational mapping, UML-to-EJB, etc.
• No performance benchmarks for MT
• In the paper:Benchmarks for graph transformation tools– terminology for benchmarking
– quantitative comparison of tools
• Ongoing research:– Benchmarks for general model transformation tools
![Page 7: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/7.jpg)
7Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
![Page 8: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/8.jpg)
8Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Metamodel
Metamodeling
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
next
blocked
held_by releasetoken
request*
1
*
1*
111
***
*
Class
Association
Multiplicity constraint
At most one
Arbitrary
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
Resource
Process
Object
Link
Slot
Inheritance
Attribute
Instance model
![Page 9: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/9.jpg)
9Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Graph transformation rule
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
Negative applicationc
ondition
p1:Process
r:Resource
GiveRule
n:nextp2:Process
rel:release
p1:Process
r:Resource
n:nextp2:Process
t:token
LHS RHS
Left-HandSide
Right-HandSide
![Page 10: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/10.jpg)
10Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Pattern matching phase
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
![Page 11: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/11.jpg)
11Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Updating phase
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
rl:release
![Page 12: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/12.jpg)
12Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
![Page 13: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/13.jpg)
13Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Paradigm features
• Definition: problem characteristics
• Model size– number of nodes and edges in the model graph
• Pattern size– number of nodes and edges in the LHS graph
• Fan-out (maximum degree of model nodes)– number of edges adjacent to a certain node
• Number of matchings
• Transformation sequence length
![Page 14: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/14.jpg)
14Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Tool features
• Definition: typical optimization strategies supported by different tools
• Multiplicity based optimization– More powerful strategy when traversing edges with ‘at
most one’ (0..1) multiplicity
• Parameter passing
• Parallel rule execution
• ‘As long as possible’ (ALAP) rule application
• Indexes (for key/index attributes)
• Incremental pattern matching
• ...
![Page 15: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/15.jpg)
15Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Parameter passing I.
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
rl1:release
•Between consecutive rule applications•No recalculation in later steps
![Page 16: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/16.jpg)
16Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Parameter passing II.
rl1:release
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
p1:Process
r:Resource
GiveRule
n:nextp2:Process
rel:release
p1:Process
r:Resource
n:nextp2:Process
t:token
LHS RHS
t1:token
![Page 17: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/17.jpg)
17Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Parallel rule execution (PM)
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
4
1 2
3
1
43
2
4
3
2
1
•Calculation of all matchings•Independent matchings
![Page 18: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/18.jpg)
18Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Parallel rule execution (Update)
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
4
1 2
3
1
43
2
4
3
2
1
rl1:release
rl3:release
rl2:release
rl4:release
•No re-evaluation of valid matchings•Transaction block
![Page 19: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/19.jpg)
19Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
![Page 20: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/20.jpg)
20Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology I.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –
– Broad application field of graph transformation• model analysis• model transformation• visual languages with dynamic operational
semantics
– Informal characteristics
![Page 21: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/21.jpg)
21Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology II.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –Benchmark example + + – – – –
next
blocked
held_by releasetoken
request*
1
*
1*
111
**
*
*
Resource
Process
TakeRule
RequestRule
GiveRule
ReleaseRule
LHS RHS• Incarnation of a scenario– fulfills the informal characteristics
– e.g. Mutex for the simulation of visual languages
![Page 22: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/22.jpg)
22Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology III.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –Benchmark example + + – – – –Test set + + PD PD PD –
h1:held_by
nN:next
n1:next
hN:held_by
p1:Process
p2:Process
pN:ProcessrN:Resource
r1:Resource
Instance model
1. N ReleaseRule
2. N GiveRule
3. N RequestRule
4. N TakeRule
• Parametric and deterministicspecification
![Page 23: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/23.jpg)
23Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology IV.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –Benchmark example + + – – – –Test set + + PD PD PD –Test case + + PD PD PD +
• Multiplicity based optimization: OFF
• Parameter passing: OFF
• Parallel rule execution: ON
![Page 24: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/24.jpg)
24Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology V.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –Benchmark example + + – – – –Test set + + PD PD PD –Test case + + PD PD PD +Run + + + + + +
• Complete characterization
• Reproducable measurements
N = 10
![Page 25: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/25.jpg)
25Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
![Page 26: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/26.jpg)
26Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Categorization
• Style– Interpreted: AGG, GTDB
– Compiled: PROGRES, Fujaba
• Base algorithm– Constraint satisfaction: AGG, GTDB
• variables + constraints
– Local searches: Fujaba, PROGRES • step-by-step extension of the matching
![Page 27: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/27.jpg)
27Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Environment
• Hardware and OS– 1500 MHz Pentium
– 768 MB RAM
– Linux kernel of version 2.6.7
• Tool related– Runs without GUI
– Rule applications guided by Java programs• PROGRES: guided by C program
– Standard services of the default distribution• fine-tuning based on developers’ suggestions
![Page 28: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/28.jpg)
28Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Tool specific issues
• AGG– Manual implementation of parameter passing
• Progres– GRAS (default underlying graph-oriented database)– Prolog-style cuts (determinism)– User undo/redo mechanism being switched off
• Fujaba– Additional node for storing processes and resources– Slightly altered models to provide better performance
• GTDB– MySQL (version 4.1.7)– No additional optimization
![Page 29: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/29.jpg)
29Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Tool feature support
Multiplicity based
optimization
Parameter passing
Parallel rule application
AGG – + –
PROGRES + + +
Fujaba + + +
GTDB – + +
![Page 30: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/30.jpg)
30Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Quantitative comparison
Updating phase
0.01
0.1
1
10
100
10 100 1000
N
av
g. t
ime
(m
s)
AGG
PROGRES
FUJABA
DB
• Result set clean-up procedure• Constant overhead distributed over more rule applications
• Compilation step from graphsto categories in each GT step
![Page 31: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/31.jpg)
31Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Effect of multiplicity based optimization
10 100 1000 10 100 1000 10 100 1000
0.1
1
10
100
1000
10 100 1000
GiveRule
10 100 1000 10 100 1000 10 100 1000
0.1
1
10
100
1000
10 100 1000
ReleaseRule
AGG PROGRES FUJABA GTDB
![Page 32: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/32.jpg)
32Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Effect of parameter passing
10 100 1000 10 100 1000 10 100 1000
0.1
1
10
100
1000
10 100 1000
AGG PROGRES FUJABA GTDB
STS test set, Pattern matching phase, ReleaseRule
![Page 33: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/33.jpg)
33Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Effect of parallel rule application
0.1
1
10
100
1000
10 100 1000 10 100 1000 10 100 1000 10 100 1000
AGG PROGRES FUJABA GTDB
ALAP test set, Pattern matching phase, ReleaseRule
![Page 34: Benchmarking for Graph Transformation](https://reader036.vdocuments.net/reader036/viewer/2022062323/5681584e550346895dc5ab92/html5/thumbnails/34.jpg)
34Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Thanks for your kind attention
• Contributions to model transformation benchmarking are welcome– additional benchmark examples, test sets, test
cases
– participation of further tools
– new optimization strategies
• Benchmark example descriptions, specifications, and measurement results– http://www.cs.bme.hu/~gervarro/benchmark/