1 ludovic henrio paris, 2006-06-08 an open source middleware for the grid programming wrapping...
Post on 15-Jan-2016
221 views
TRANSCRIPT
1
Ludovic Henrio
Paris, 2006-06-08
An Open Source Middleware for the GridProgramming Wrapping Composing Deploying
2
Table of Contents
1. Distributed Objects
2. Deployment
3. Monitoring
4. Composing
5. Conclusion
3
A uniform framework: An Active Object pattern A formal model behind: Determinism (POPL’04)
Programming Model (Active Objects):• Asynchronous Remote Invocations, Wait-By-Necessity
• Groups, Mobility, Components, Security, Fault-tolerance, Load balancing
Environment:
• XML Deployment Descriptors, File Transfers • Interfaced with: rsh, ssh, LSF, PBS, Globus, Jini, SUN Grid Engine
• Graphical Visualization and monitoring: IC2D
In the www. ObjectWeb .org Consortium
(Open Source LGPL)
ProActive: A Java API + Tools for Parallel, Distributed Computing
4
Model• Activity : coarse-grained structuring entities (subsystems)
• possibly owns many passive objects
• has exactly one thread, and one active object.
• No shared passive object -- Parameters are passed by deep-copy
• Asynchronous Communication between active objects
• Futures (promised replies) and wait-by-necessity.
JVM
copy!distant.foo(object)
5
An object created with A a = new A (obj, 7);
can be turned into an active and remote object:
• Object-based:a = (A)ProActive.turnActive (a, node);
• Instantiation-based:A a = (A)ProActive.newActive(«A», params, node);
The “node” is the AO container.
Remaining of the code unchanged “Transparency”
Creating active objects
6
Future: result of a method call
• A call on an active object consists in 2 steps• A request : name of the method, parameters…
• A Reply : the result of the method call
• A request returns a Future object which is a placeholder for the result
• The callee will update the Future when the result is available
• The caller can continue its execution event if the Future has not been updated
Reply future = distant.query (parameters);
7
A
Wait-by-necessity
Proxy
Java Object
A ag = newActive (“A”, […], VirtualNode)V v1 = ag.foo (param);V v2 = ag.bar (param);...v1.bar(); //Wait-By-Necessity
V
Wait-By-Necessity
is a
Dataflow
Synchronization
JVM
A
JVM
Active Object
Future Object Request
Req. Queue
Thread
v1v2 ag
WBN!
8
Collective Communications: Groups
Typed and polymorphic Groups of active and remote objectsDynamic generation of group of resultsLanguage centric, Dot notation
• Manipulate groups of Active Objects, in a simple and typed manner:
• Be able to express high-level collective communications (like in MPI):• broadcast, • scatter, gather, • all to all
9
A
Creating AO and Groups
Typed Group Java or Active Object
A ag = newActiveGroup (“A”, […], VirtualNode)V v = ag.foo(param);...v.bar(); //Wait-by-necessity
V
Typed groups, and Asynchrony
are crucial for Components and GRID
JVM
10
Weak Migration of active objects
Migration is initiated by the active object itself through a primitive: migrateTo
Can be initiated from outside through any public method
The active object migrates with:• all pending requests• all its passive objects • all its future objects
Automatic and transparent forwarding of:• requests (remote references remain valid)• replies (its previous queries will be fullfilled)
2 Techniques : Forwarders or Centralized server
11
Table of Contents
1. Distributed Objects
2. Deployment
3. Monitoring
4. Composing
5. Conclusion
12
How to deploy on the multiple Grids
Internet
Clusters
ParallelMachine
LargeEquipment
Internet
Job management forembarrassingly parallel application (e.g. SETI)
Internet
Servlets EJBs Databasesenterprise
scientific edge computing
intranet
13
Abstract Deployment Model> Problems:
• Difficulties and lack of flexibility in deployment
• Avoid scripting for: configuration, getting nodes, connecting, etc.
A key principle: Virtual Node (VN) in XML deployment file
• Abstract Away from source code:– Machines names– Creation/Connection Protocols– Lookup and Registry Protocols
• Interface with various protocols and infrastructures:– Cluster: LSF, PBS, SGE , OAR and PRUN(custom protocols)– Intranet P2P, LAN: intranet protocols: rsh, rlogin, ssh– Grid: Globus, Web services, ssh, gsissh
14
> Virtual Node (VN):• Identified as a string name• Used in program source• Configured (mapped) in the XML descriptor file --> Nodes
XML Deployment files
Program Source Descriptor (RunTime)|------------------------------------------- ---------------------------------------------| Activities (AO)--> VN VN --> JVMs --> Hosts Runtime structured entities: 1 VN --> n Nodes in m JVMs on k Hosts
> Operations specified in descriptors:• Mapping of VN to JVMs (leads to Node in a JVM on Host)
• Register or Lookup VNs
• Create or Acquire JVMs
• Security, Fault-tolerance
15
Same application, many deployments
One Host Local Grid Distributed Grids
Internet
16
Security: Key Features
ProActive Security Features
• Authentication of users and applications (PKI X 509 certificates)
• Authentication, Integrity and Confidentiality of communication
• In XML deployment files, Not In Source
• Mobility Aware
• Dynamically negotiated policies
• Easily adaptable to the deployment
17
Pure P2P: Definition
Only PEERs, no above everything, top level, server(s)
Every peer is, somehow, also a server
No master … No slave !
System gets organized dynamically, without static configuration
Coherent, desired behavior, dynamically emerges
18
Peer-to-Peer ComputingModel & Infrastructure
•Dynamic Computational Peer-to-Peer (P2P):•Intranet & Internet
•Propose a High Level Model•Dynamic JVMs Network (computation nodes) [infrastructure]
•P2P Programming Model for Branch and Bound (B&B) problems
•ProActive Context: no modification of Java language, of JVMs, …
Inspired from Gnutella
Use Sparse CPU Cycles from Desktop Workstations/clusters
19
Fault-tolerance in ProActiveRollback-Recovery fault-tolerance
• After a failure, revert the system state back to some earlier and correct version
• Based on periodical checkpoints of the active objects
• Stored on a stable server
Two protocols are implemented
• Communication Induced Checkpointing (CIC)
• Low failure free overhead
• Slow recovery
• Pessimistic Message Logging (PML)
• Higher failure free overhead
• Fast recovery
Fault-tolerance is set in deployment descriptors
• Fault-tolerance service attached to virtual nodes
• No source code alteration
Transparent and non intrusive
20
Table of Contents
1. Distributed Objects
2. Deployment
3. Monitoring
4. Composing
5. Conclusion
21
IC2D:Interactive Control & Debug for Distribution
• Graphical Visualisation
• Textual Visualisation
• Control and Monitoring
• Job Management
22
IC2D: Application Monitoring
Main Features:
- Hosts, JVM,
- Nodes
- Active Objects
- Topology
- Migration
23
Table of Contents
1. Distributed Objects
2. Deployment
3. Monitoring
4. Composing
5. Conclusion
24
Content
Controller
ProActive Components
25
Table of Contents
1. Distributed Objects
2. Deployment
3. Monitoring
4. Composing
5. Conclusion
26
Appli: JEM 3D : Java 3D Electromagnetism together with Said El Kasmi, Stéphane Lanteri (caiman)
Maxwell 3D equation solver, Finite Volume Method (FVM)Pre-existing Fortran MPI version: EM3D (CAIMAN team @ INRIA)
Execution Time on a cluster
0
100
200
300
400
500
600
700
800
900
0 10 20 30 40 50 60 70nombre de processeurs
tem
ps
(sec
on
des
)
21*21*21
31*31*31
43*43*43
55*55*55
81*81*81
97*97*97
113*113*113
121*121*121
Mesh Size
27
ProActive and (De Facto) Standards
• RMI, RMI-Ibis, Jini, HTTP
• rsh, ssh, scp
• Globus GTx, sshGSI, Unicore, EGEE gLite
• LSF, PBS, OAR, Sun Grid Engine
• Fractal Components
• Web Services
• OSGi
28
That’s it!
A Strong Programming Model + A Strong Deployment
Presented features:
Active Objects, XML descriptors, Security, P2P, Fault-tolerance
Features not detailed:
OO SPMD, Web Services, load balancing, legacy wrapping, and more…
Perspectives:• Better Error Management (Exceptions, …)• Behavior Specification (Model Checking) • Tools for Distributed Programming: Code Analysis, Transformation, Integrated IDE
Available in Open Source http://ProActive.ObjectWeb.org
29
30
31
f3
f1
Structure
ASP
foo
foo
f2
Active(a)
32
Openness
• Open Source
• Easy to extend• Add new Deployment Protocols
• Add new File Transfer Protocols
• Not so hard to add new Transport (one could imagine hardware specific implementations)
• Add Checkpointing Protocols
• Add localization strategies for mobility
33
ASP: Summary and Results
An Asynchronous Object Calculus:
• Structured asynchronous activities
• Structured communications with futures
• Data-driven synchronization
ASP Confluence and Determinacy
Future updates can occur at any time
Execution characterized by the order of request senders
Determinacy of programs communicating over trees, …
34
Interactive Composition in IC2D
Content
Composition ViewInstead of
XML ADL
<processDefinition id="linuxJVM">
<jvmProcess class="org.objectweb.proactive.core.process.JVMNodeProcess"/>
</processDefinition>
<processDefinition id=”sshProcess">
<sshProcess class="org.objectweb.proactive.core.process.ssh.SSHJVMProcess"
hostname="sea.inria.fr">
<processReference refid="linuxJVM"/>
</sshProcess>
</processDefinition>
35
Perspective for Components - PSE Graphical Composition, Monitoring, Migration
36
Perspective for Components - PSE Graphical Composition, Monitoring, Migration