under the supervision of christian pérez insa de rennes
Post on 02-Jan-2016
17 Views
Preview:
DESCRIPTION
TRANSCRIPT
Ph.D. Defense by Julien Bigot, December 6th 2010
Generic Support of Composition Operators in
Software Component Models,Application to Scientific Computing
Under the supervision of Christian Pérez
INSA de Rennes
INRIA Team-Projects PARIS(IRISA) / GRAAL(LIP)
2
Context: Scientific Applications
• Complex applications
Coupling of independent codes
→ Developed by distinct teams
Long life-cycle→ longer than hardware
• Computational intensive (HPC),
=> requires complex hardware
Supercomputers
Computing clusters
Computing grids
Clouds
Thermal
Optics
Dynamics
Structural Mechanics
SAN SAN
Cluster
LAN
WAN
Supercomputer
Grid
Supercomputer
Cluster
Satellite modeling ? programmingmodels
3
Context:Parallel/Distributed Computing Paradigms
• Shared memory
Implementations→ OS (node level)→ kDDM (cluster level)→ Juxmem (grid level)→ …
DSM
4
Context:Parallel/Distributed Computing Paradigms
• Shared memory
Implementations→ OS (node level)→ kDDM (cluster level)→ Juxmem (grid level)
• Message passing
Implementations→ MPI→ PVM→ Charm++→ …
Variations→ Collective operations
• Barrier• Broadcast• Scatter• …
5
Context:Parallel/Distributed Computing Paradigms
• Shared memory
Implementations→ OS (node level)→ kDDM (cluster level)→ Juxmem (grid level)
• Message passing
Implementations→ MPI→ PVM→ Charm++
Variations→ Collective operations
• Remote procedure/methods calls
Implementations→ ONC RPC→ Corba→ Java RMI→ …
Variations→ Parallel procedure call (PaCO++)→ Master / worker (GridRPC)
6
Context:Parallel/Distributed Computing Paradigms
• Shared memory
Implementations→ OS (node level)→ kDDM (cluster level)→ Juxmem (grid level)
• Message passing
Implementations→ MPI→ PVM→ Charm++
Variations→ Collective operations
• Increased abstraction level
Easier programming
Various implementations for various hardware
• Centered on communication medium
Application architecture not dealt with
Makes reuse difficult
• Remote procedure/methods calls
Implementations→ ONC RPC→ Corba→ Java RMI
Variations→ Parallel procedure call (Paco++)→ Master / worker (GridRPC)
7
Context:Composition-Based Programming
• Components
Black boxes
Interactions through well defined interaction points
• Application
Component instance assembly
High level view of architecture
• Implementation
Primitive: C++, Fortran, Java
Composite (Assembly)
• Dimension
Spatial (classical models)→ CCA, CCM, Fractal, SCA
Temporal (workflow/dataflow)→ Kepler, Triana
C3
C2 C1
T1 T2
e
c
d
a
b
f
8
Context:Composition-Based Programming
• Algorithmic skeletons
Predefined set of composition patterns→ Farm (master/worker)→ Pipeline→ Map→ Reduce→ …
Focus: decoupling→ User:
application related concerns (what)→ Skeleton developer:
hardware related concerns (how)
Implementations→ Multiple libraries→ P3L→ Assist→ …
Farm
9
Context:Components for Parallel Computing
• Parallel/distributed programming paradigms
Hidden architecture
Reuse complex
Hardware resource abstraction
Efficiency on various hardware
• Composition-based models
High-level view of architecture
Eased reuse
Hardware specific assemblies
Parallelism-oriented composition-based model
10
Context:Components for Parallel Computing
• Memory sharing between components
CCA & CCM Extensions
• Parallel components
CCA, SCIRun2, GridCCM
• Collective communications
CCM Extension
• Parallel method calls
SCIRun2, GridCCM
• Master / worker support
CCA & CCM Extensions
• Some algorithmic skeletons in assemblies
STKM
• Two type of features
Component implementations→ ≈ skeletons
Component interactions
11
Context:Existing Models Limitations
• Limited set of features in each model
=> Combination in a single model
• Unlimited set of features (including application specific)
=> Extensibility of composition patterns
Thermal
Structural Mechanics
Dynamics
Optics
CollComm
CollComm
12
Context:Problematic
• Goals for an extensible component model
User code reuse
Reusable composition patterns
Reusable inter-component interactions
Efficiency on various hardware
• Required concepts
Components
User-definable skeletons
User-definable connectors
Multiple implementations with hardware-dependant choice
• Problems
How to support user-defined skeleton implementations ?
How will all the concepts behave when combined ?
13
Outline
• Context of the Study
• Software Skeletons as Generic Composites
• HLCM: a Model Extensible with Composition Patterns
• HLCM in Practice
• Conclusion
14
Software Skeletons with Components:Motivating Example
coord pixel
• Goal
Generating pictures of the Mandelbrot set
• Embarrassingly parallel problem
C = (x,y)
Z0 = 0, Zn+1 = Zn² + C→ Bounded → black→ Unbounded → blue
• Hardware
Quad-core processor
• Task farm
4 workers
Input: coordinate
Output: pixel color
15
Software Skeletons with Components:A Component Based Farm
Coord Pixel
Coord
Disp
Pix
Coll
Mandel
Worker
Mandel
Worker
Mandel
Worker
Mandel
Worker
MandelbrotFarm
16
Software Skeletons with Components:Limitations of Component Based Farm
• Hard Coded in the composite
Type of the worker
Type of the interfaces
Number of workers
• Limited reuse
For a distinct calculation
For distinct hardware
Coord Pixel
Coord
Disp
Pix
Coll
Mandel
Worker
Mandel
Worker
Mandel
Worker
Mandel
Worker
MandelbrotFarm
17
Software Skeletons with Components:A Component Based Farm
Coord Pixel
Coord
Disp
Pix
Coll
Mandel
Worker
Mandel
Worker
Mandel
Worker
Mandel
Worker
MandelbrotFarm
18
Software Skeletons with Components:Introducing Parameters
Coord Pixel
Coord
Disp
Pix
Coll
W
W
W
W
Farm<W>
19
Software Skeletons with Components:Introducing Parameters
I O
Disp<I> Coll<O>
W
W
W
W
Farm<W, I, O>
20
Software Skeletons with Components:Introducing Parameters
Disp<I> Coll<O>
W
W
W
Farm<W, I, O, N>
.
.
.N times
GenericityI O
21
Software Skeletons with Components:Concepts for Genericity
• Generic artifacts
Accept 2nd order parameters
Use the values of parameters in their implementation
public class GenClass<T> {
private T _val;
public T getVal() {
return _v;
}
…
}
template<typename T>
T increment (T val) {
T result = val;
result += 1;
return result;
}
Java
C++
GenCmp<C>
C CmpA
22
MyComposite
Software Skeletons with Components:Concepts for Genericity
• Specializations
Use of generic artifacts
Arguments bind parameters to actual values
GenClass<String> gs = new GenClass<String>();
String s = gs.getVal();
GenClass<Integer> gi = new GenClass<Integer>();
…
int i = 42;
i = increment<int>(i);
float f = 35.69;
f = increment<double>(f);
…
Java
C++
GenCmp<MyCmp>
MyCmp
AnotherMyCmp CmpA
23
Software Skeletons with Components:Metamodel-based Transformation
GenericCMMeta-Model
S:GenericCMsource files
D:CM
source filesS Model D Model
CMMeta-Model CM:
Non-generic Supported
GenericCM: Generic Unsupported
component App { content { composite { decl { Decrement_Composite D; Viewer v; Decrement d1; d1.h_log_port -- v.v_log_port; D.h_log_port -- v.v_log_port; d1.dp_dec_port -- D.du_dec_port; d1.du_dec_port -- D.dp_dec_port; set d1.Number 3; } service run { seq { exec d1.export; exec D; } } } }}
component App { content { composite { decl { Decrement_Composite D; Viewer v; Decrement d1; d1.h_log_port -- v.v_log_port; D.h_log_port -- v.v_log_port; d1.dp_dec_port -- D.du_dec_port; d1.du_dec_port -- D.dp_dec_port; set d1.Number 3; } service run { seq { exec d1.export; exec D; } } } }}
Semantically equivalent
GenericCMMeta-ModelParsing Transformation Dump
A
TB<T>
B<A>C
A
AB_0
D
B<D>
B_0C
B_1
DB_1
D
24
Software Skeletons with Components:Deriving a Metamodel with Genericity
• Example
Generic artifact→ ComponentType
Artifact usable as parameter→ PortType
• Additional modifications
Default values for parameters
Constraints on parameter values
Explicit specializations
Meta-programming
Port
PortTypeParam
PortTypeArgComponentType
PortType
ComponentInstance
25
Software Skeletons with Components:Transformation
• Specialized compilation
C++ approach
Makes template meta-programming possible
• Algorithm
Copy non generic elements
For each generic reference→ Create a context
(parameter value)→ Copy the generic element in that
context
A
TB<T>B<A>
C
A AB_0
D B<D>
B_0C
B_1
DB_1
D
26
Software Skeletons with Components:Summary
• Defined genericity for component models
Straight-forward meta-model extension
Independent of the component model→ SCA (100 class) GenericSCA (+20 class)→ ULCM (71 class) GULCM (+24 class)
• Proposed an approach for genericity execution support
Transformation→ generic model non-generic, executable equivalent
Relies on model-driven engineering approach→ Implemented using Eclipse Modeling Framework (EMF)
• Make skeleton implementation possible
Implemented skeletons: task farm (pixel, tile), …→ Computed pictures of the Mandelbrot set
27
Outline
• Context of the Study
• Software Skeletons as Generic Composites
• HLCM: a Model Extensible with Composition Patterns
• HLCM in Practice
• Conclusion
28
Combining All Concepts in HLCM:Aims & Approach
• Aims
Support user-defined extensions Hierarchy
Genericity→ Interactions: connectors→ Adaptation to hardware: implementation choice
Maximize reuse→ Existing user code => component models
• Approach
Abstract model: primitive concepts taken from backend model→ HLCM Specializations (HLCM + CCM => HLCM/CCM)
Deployment time transformation→ Hardware resources taken into account→ Fall back to backend
→ Skeletons:
29
Combining All Concepts in HLCM:Component Implementation Choice
• Distinguish component type & component implementation
Component type→ Generic→ List of ports
Component implementation→ Generic→ Implemented component type
specialization→ Content description
• Primitive• Composite
• Choice at deployment time
Match used specialization & implemented specialization
Choose amongst possible
CmpA<T>
A2
A1 A3<X>
T=XT=B
T=C
CmpA<B>
A3<B>
T=B
component CmpA<component T> exposes {provide<Itf> p;
}
primitive A1implements CmpA<B>{ …}
composite A3<component X>implements CmpA<X>{ …}
30
UP
Combining All Concepts in HLCM:Introducing Connectors from ADLs
• Without connectors
Direct connection between ports
Predefined set of supported interactions
• Connectors
A concept from ADLs
Connectors reify interaction types→ A Name→ Set of named roles
Instance are called connections→ Each role fulfilled by a set of ports
Fisrt class entity
CmpA CmpB
CmpC CmpD
CmpA CmpB
CmpC CmpDEvent
ports
roles
user provider
31
Combining All Concepts in HLCM:Generators
• Generator = connector implementation
• 1 connector multiple generators
Distinct constraints→ Port types→ Component placement
• Two kinds of generators
Primitive
User-defined→ Composite
• Intrinsically generic
Type of ports fulfilling roles generic parameters
UP
interface = UT interface = PT
When PT subtype of UT &&user.host == provider.host
UP
connector UP < role user, role provider >;
generator IPC < interface UT, interface PT >implements UP<user={UT}, provider={PT}>when UT super PT && user.host == provider.host{ Stub<UT> stub; Skel<PT> skel; …}
Skel<PT>
UPStub<UT>
Unixsocket
user provider
IPC
33
SharedData
Combining All Concepts in HLCM: Concepts Interactions
C0
ParallelA ParallelB
.
.
.
M times
MPI
MPI
.
.
.
N times
C1
CM
D0
D1
DN
Adapt Adapt
0
1
M
0
1
N
C
C
C
D
D
D
34
Shared
Data
Combining All Concepts in HLCM: Concepts Interactions
ParallelA ParallelB
.
.
.
MPI
MPI
.
.
.M times N times
C0
C1
CM
D0
D1
DN
0
1
M
0
1
N
C
C
C
D
D
D
35
Combining All Concepts in HLCM:Introducing Open Connections
ParallelA ParallelB
.
.
.M times
.
.
.N times
SharedData
SharedData
SharedData
SharedData
SharedData
Merge
SharedData
SharedData
SharedData
C
C
C
Merge
Merge
Merge
Merge
Merge
Merge
MPI
MPI
MPI
MPI
MPI
MPI
D
D
D
SharedData
Shared
Data
36
Combining All Concepts in HLCM:Introducing Open Connections
• Connection:
Connector
Mapping: role -> set(ports)
• Merge({ ConnA, ConnB })
Pre: ConnectorA == ConnectorB
Each ( role r )→ Mapping: r -> set(Port)A union set(Port)B
• Component
Expose named connection→ Open connection
Composite→ Expose internal connection
Primitive→ Fill exposed connections roles
UP
UPUP
merge
UPUP
merge
UP
37
Combining All Concepts in HLCM:Introducing Open Connections
ParallelA ParallelB
.
.
.M times
.
.
.N times
UP
UP
UP
MPI
MPI
MPI
MPI
MPI
MPI
UP
UP
UP
UP UP
User:multiply<MatrixBlock>
User:multiply<Matrix>
Provider:multiply<Matrix>
Provider:multiply<MatrixLine>
38
UP
• Bundle Regroup multiple open connections to fulfill a
role
• Connection adaptor Supports open connection polymorphism
→ Supported connection→ Behavior definition
Two views→ A connection implementation→ An open-connection exposer
Implemented by an assembly
Used only if necessary
Combining All Concepts in HLCM:Connection Adaptors & Bundles
adaptor PushPull supports UseProvide < user={Receptacle<Push>}, provider={} > //< supported as UseProvide < user={}, provider={Facet<Pull>} > //< this{ BufferComponent buffer; merge({ buffer.pushSide, supported }); merge({ this, buffer.pullSide });}
Receptable<Push> Facet<Pull >
UPUP UPQ
UP UPQ
Receptable<Push> Facet<Pull >
39
Combining All Concepts in HLCM:Summary
• Approach
Abstract model: primitive elements from backend→ E.g. HLCM + CCM HLCM/CCM
Transformation: HLCM specialization backend→ E.g. HLCM/CCM pure CCM
• Source model (PIM)
Four main concepts→ Hierarchy, genericity, implementation choice, connectors
Additional concepts→ Open connections, merge, connection adaptors, bundles
• Transformation
Merge connections (use adaptors if required)
Choose implementation
Expose composite content
• Destination model (PSM)
Direct map to backend
Core: 127 Ecore classes
CCM specialization: 3 Ecore classes
41 Ecore classes
40
Outline
• Context of the Study
• Software Skeletons as Generic Composites
• HLCM: a Model Extensible with Composition Patterns
• HLCM in Practice
• Conclusion
41
Experimental Validation:A Framework for HLCM Implementation
• Model-transformation based
Eclipse Modeling Tools
• HLCM/CCM source model (PIM)
490 Emfatic lines (130 Ecore classes) 25 000 generated Java lines
2000 utility Java lines
• HLCM destination model (PSM)
160 Emfatic lines (41 Ecore classes) 1500 generated Java lines
800 utility Java lines
• Transformation engine
4000 Java lines
• Already implemented connectors (HLCM/CCM)
Shared Data
Collective Communications
Parallel method calls
42
Experimental Validation:Implementing shared memory
C3
C1
C2SharedMem
C3C1
C2
LocalMemoryStore
LocalUseProvide
provider
role user
generator LocalSharedMem<Integer N> implementsSharedMem< access = each(i:[1..N]) { LocalReceptacle<DataAccess> } >{ LocalMemoryStore<N> store; each(i:[1..N]) { store.access[i].user += this.access[i]; }}
43
Experimental Validation:Implementing shared memory
C3
C1
C2
PosixSharer
PosixSharer
PosixSharer
UseProvide
role user role provider
Locality Constraint«same process»
generator PosixSharedMem<Integer N> implementsSharedMem< access = each(i:[1..N]) { LocalReceptacle<DataAccess> } >when samesystem ( each(i:[1..N]) { this.access[i] } ){ each(i:[1..N]) { PosixSharer node[i]; node[i].access.user += this.access[i]; }}
C3
C1
C2
SharedMem
44
C1
C3
Experimental Validation:Implementing shared memory
C2
JuxMemManager
JuxMemPeer
JuxMemPeer
JuxMemPeer
UseProvide
generator JuxMem<Integer N> implementsSharedMem < access = each(i:[1..N]) { LocalReceptacle<DataAccess> } >{ JuxMemManager<N> manager; each(i:[1..N]) { JuxMemPeer peer[i]; peer[i].access.user += access[i]; merge ({ peer[i].internal, manager.internal[i] }); }}
C3
C1
C2
SharedMem
45
Experimental Validation:Parallel Method Calls Server
component ServiceProvider exposes{ UseProvide<provider={Facet<Service>}> s;}
bundletype ParallelFacet<Integer N, interface I>{ each(i:[1..N]) { UseProvide<provider={Facet<ServicePart>}> part[i]; }}
composite ParallelServiceProvider<Integer N> implements ServiceProvider { each(i:[1..N]) { PartialServiceProvider p[i]; } this.s.provider += ParallelServiceFacet<N> { each(i:[1..N]) { part[i] = providerPart[i].s; } }}
S2
S1
S0
ParallelServiceProvider<3>
PartialServiceProvider
ProviderPartialServiceFacet
ParallelServiceFacet<N>
provider
PartialServiceUser
C1
ParallelServiceUser<2>
PartialServiceUser
C0Merge
46
Experimental Validation: Parallel Method Calls Result
UsersideRedistributor<2,3>
UsersideRedistributor<2,3>
ServersideRedistributor<2,3>
ServersideRedistributor<2,3>
ServersideRedistributor<2,3>
C0
C1
S0
S1
S2
PartialServiceProvider
PartialServiceProvider
PartialServiceProvider
47
Experimental Validation:Parallel Method Calls Performance
Ban
dwith
par
cal
ler
(Mb/
s)
Message size (Byte)
• Comparison
HLCM/CCM
Paco++
• Single cluster
1Gb Ethernet
• Parallelism
3 clients
4 servers
48
Experimental Validation:Conclusion
• HLCM Implementation Developed
HLCMi: a framework for HLCM specializations implementations
HLCM/CCM: a CCM based specialization
Other specializations→ Plain Java→ CCM + plain C++→ Charm++
• Validated
Good expressiveness
Good performance
No real automated choice yet
49
Outline
• Context of the Study
• Software Skeletons as Generic Composites
• HLCM: a Model Extensible with Composition Patterns
• HLCM in Practice
• Conclusion
50
Conclusion
• Identified need for an extensible component model User-defined inter-component interactions (composite generators)
User-defined parameterized component implementations (skeletons)
• Introduced genericity in component models Supports user-defined skeleton implementation
Applicable to existing component models
Model transformation approach
• Described an extensible component model: HLCM Abstract model that relies on a backend for primitive concepts
Concepts combined in HLCM→ Hierarchy→ Genericity→ Implementation choice→ Connectors with open connections & connection adaptors
• Implemented & used
A model-based implementation framework using EMF: HLCMi
Used for synthetic example implementations→ memory sharing→ parallel method calls
51
Perspectives
• More validation
MapReduce ANR project (2010 - 2013)
Domain decomposition, Adaptative Mesh Refinement→ Salome (EDF, CEA) (w. Vincent Pichon)
OpenAtom (Molecular Dynamic)→ Pr Kale @ UIUC (Charm++) → Complex application, 4D/2D interaction
• Introduce the time dimension
Usage→ Change of implementation choices during execution (QoS, Data, …)→ Time related applications description (dataflow, workflow)
Problem→ Keep logical & primitive model in sync
• Study choice algorithms
Choice criteria attached to model elements (w. scheduling community)
Interactions with resource management systems (w. Cristian Klein)
top related