three types of remote process invocation single (jobtype=single) grpc_function_handle_init( … );...
TRANSCRIPT
Three types of remote process invocation
Single (jobtype=single)Single (jobtype=single)grpc_function_handle_init(…);
Multiple (jobtype=multiple)Multiple (jobtype=multiple)grpc_function_handle_array_init_np(…);
ClientServer
gatekeeper
ClientServer
gatekeeper
MPI (jobtype=MPI)MPI (jobtype=MPI)grpc_function_handle_init(…);“Backend=MPI” is specified in the IDL
Three types of remote process invocation (cont’d)
ClientServer
gatekeeper
mpirunprun
Ninf-G fnctions for fault detection
Detects errors appropriately and an error code is reDetects errors appropriately and an error code is returnedturned
server processes diednetwork disconnection
Other functions for fault detectionOther functions for fault detectiontimeout
for initialization (e.g. stacked in the queue, etc.)for rpc (something unexpected would happen)for heatbeat (heatbeat messages have been continuously missed)
If these faults are detected, API returns an error and the error code indicates the timeout error.
Remarkable Features of Ninf-G4 (summary)
Enabling RPC via any middlewareEnabling RPC via any middlewarePre-WS GRAM, WS GRAM, UNICORECondor, SSH, NAREGI SSAny others
Client configuration file that allows detailed descripClient configuration file that allows detailed description of execution environmentstion of execution environments
Server, Client, Function attributes, etc.Capability for fault detectionCapability for fault detection
Explicit faultsServer process died, network disconnection, etc.
Implicit faultsTimeout of Invocation, Execution, Heatbeat
High quality as the softwareHigh quality as the software
National Institute of Advanced Industrial Science and Technology
Ninf-G
How to use Ninf-GHow to use Ninf-G
How to use Ninf-G
Build remote libraries on server machinesBuild remote libraries on server machinesWrite IDL filesCompile the IDL filesBuild and install remote executables
Develop a client programDevelop a client programProgramming using GridRPC APICompile
RunRunCreate a client configuration fileGenerate a proxy certificateRun
Rough steps for RPC (client-side programming)
InitializationInitialization
Create a function handleCreate a function handleabstraction of a connection to a remote executable
Call a remote libraryCall a remote library
grpc_function_handle_t handle;
grpc_function_handle_init( &handle, host, port, “lib_name”);
grpc_call(&handle, args…); orgrpc_call_async(&handle, args…);grpc_wait( );
grpc_initialize(config_file);
Terminloogy (cont’d)
Ninf-G ExecutableNinf-G ExecutableNinf-G executable is an executable file that will be invoked by Ninf-G systems. It is obtained by linking a user-written function with the stub code, Ninf-G and the Globus Toolkit libraries.
SessionSessionA session corresponds to an individual RPC and it is identified by a non-negative integer called Session ID.
GridRPC APIGridRPC APIApplication Programming Interface for GridRPC. The GridRPC API is going to be standardized at the GGF GridRPC WG.
Terminology
Ninf-G Client Ninf-G Client This is a program written by a user for the purpose of controlling the execution of computation.
Ninf-G IDLNinf-G IDLNinf-G IDL (Interface Description Language) is a language for describing interfaces for functions and objects those are expected to be called by Ninf-G client.
Ninf-G StubNinf-G StubNinf-G stub is a wrapper function of a remote function/object. It is generated by the stub generator according to the interface description for user-defined functions and methods.
National Institute of Advanced Industrial Science and Technology
Ninf-G
How to build Remote LibrariesHow to build Remote Libraries
- server side operations -- server side operations -
Ninf-G remote libraries
Ninf-G remote libraries are implemented as executNinf-G remote libraries are implemented as executable programs (able programs (Ninf-G executablesNinf-G executables) which) which
contain stub routine and the main routinewill be spawned off by GRAM or other middleware
The stub routine handlesThe stub routine handlescommunication with clients and Ninf-G system itselfargument marshalling
Underlying executable (main routine) can be written Underlying executable (main routine) can be written in C, C++, Fortran, etc.in C, C++, Fortran, etc.
Ninf-G remote libraries (cont’d)
Ninf-G provides two kinds of Ninf-G remote executaNinf-G provides two kinds of Ninf-G remote executables:bles:
FunctionStatelessDefined in standard GridRPC API
Ninf-G objectstatefulenables to avoid redundant data transfersmultiple methods can be defined
initializationcomputation
How to build Ninf-G remote libraries (1/3)
Write an interface information using Ninf-G Interface Description LanguWrite an interface information using Ninf-G Interface Description Language (Ninf-G IDL).age (Ninf-G IDL).Example:Example:
Module mmul;Module mmul;Define dmmul (IN int n,Define dmmul (IN int n, IN double A[n][n], IN double A[n][n], IN double B[n][n], IN double B[n][n], OUT double C[n][n]) OUT double C[n][n])Require “libmmul.o”Require “libmmul.o”Calls “C” dmmul(n, A, B, C);Calls “C” dmmul(n, A, B, C);
Compile the Ninf-G IDL with Ninf-G IDL compilerCompile the Ninf-G IDL with Ninf-G IDL compiler
% ng_gen <IDL_FILE>% ng_gen <IDL_FILE>
ns_gen generates stub source files and a makefile (<module_name>.mns_gen generates stub source files and a makefile (<module_name>.mak)ak)
How to build Ninf-G remote libraries (2/3)
Compile stub source files and generate Ninf-G exeCompile stub source files and generate Ninf-G executables and LDIF files (used to register Ninf-G recutables and LDIF files (used to register Ninf-G remote libs information to GRIS).mote libs information to GRIS).
% make –f <module_name>.mak% make –f <module_name>.mak
Publish the Ninf-G remote libraries (optional)Publish the Ninf-G remote libraries (optional)
% make –f <module_name>.mak install% make –f <module_name>.mak install
How to build Ninf-G remote libraries (3/3)
MDS<module>.mak
Ninf-G IDL file<module>.idl
Ninf-G IDL file<module>.idl
ng_gen
_stub_goo.c
_stub_goo
_stub_bar.c
_stub_bar
_stub_foo.c
_stub_foo
Library programlibfoo.a
Library programlibfoo.a
<module>::goo.ldif<module>::bar.ldif
<module>::foo.ldif
Client
make –f <module>.mak
National Institute of Advanced Industrial Science and Technology
Ninf-G
How to call Remote LibrariesHow to call Remote Libraries
- client side APIs and operations -- client side APIs and operations -
(Client) User’s Scenario
Write client programs in C/C++/Java using APIs provideWrite client programs in C/C++/Java using APIs provided by Ninf-Gd by Ninf-G
Compile and link with the supplied Ninf-G client compile Compile and link with the supplied Ninf-G client compile driver (driver (ng_ccng_cc))
Write a Write a client configuration fileclient configuration file in which runtime environ in which runtime environments can be describedments can be described
Run Run grid-proxy-initgrid-proxy-init command (if you use GRAM) command (if you use GRAM)
Run the programRun the program
GridRPC API / Ninf-G APIAPIs for programming client applications
The GridRPC API and Ninf-G APIGridRPC APIGridRPC API
Standard C API defined by the GGF GridRPC WG.Provides portable and simple programming interface.Enable interoperability between implementations suchas Ninf-G and NetSolve.
Ninf-G APINinf-G APINon-standard API (Ninf-G specific)complement to the GridRPC APIprovided for high performance, usability, etc.ended by _np
eg: grpc_function_handle_array_init_np(…)
Rough steps for RPC
InitializationInitialization
Create a function handleCreate a function handleabstraction of a connection to a remote executable
Call a remote libraryCall a remote librarysynchronous or asynchronous call
grpc_function_handle_t handle;
grpc_function_handle_init( &handle, host, port, “func_name”);
grpc_call(&handle, args…); orgrpc_call_async(&handle, args…);grpc_wait( );
grpc_initialize(config_file);