june 1999 chaims1 neal sample stanford university objective: investigate new approaches to...

June 1999 CHAIMS 1

Neal Sample

Stanford University

Objective:Investigate new approaches tolarge-scale software composition.

Approach:Develop and validate a composition-only language (CLAM), a protocol for large, distributed, heterogeneous and autonomous megamodules (CPAM), and a supporting system.

CHAIMS: A Reuse and Composition System for Services

CHAIMS

June 1999 CHAIMS 2

Shift in Programming Tasks

Coding

Integration, Composition, Reuse

1970 1990 2010

June 1999 CHAIMS 3

Composition of Processes...

• versus composition and integration of Components

– reusing small components (copy/paste, shared library, local install, etc.)

– large distributed components within same “domain” (e.g. within one bank or airline)

June 1999 CHAIMS 4

Domain expert, Programmers

local computer orlocal distributed system

I/O I/O

Purchasing Local Components

a bc

de

e

and copy

Distributor

purchase

June 1999 CHAIMS 5





• versus composition and integration of Data–data-warehouses–wrapping data available on the web

June 1999 CHAIMS 6

Accessing Remote Data

Control &Computation

Services

Wrappers

s e

r v

e r

sc

l i e

n t

I/O I/O

a bc

de

Domain expert, Programmers

local computer orlocal distributed system

DataResources

June 1999 CHAIMS 7





• versus composition and integration of Data– data-warehouses– wrapping data available on the web

CHAIMS/CLAM:

– compose remote autonomous services

June 1999 CHAIMS 8

Accessing Remote Services

a

b

cd

e

s e

r v

e r

s a

tpr

ovid

er's

sit

esc

l i e

n t

s

cont

rol

data

IO module IO module

Domain expert

Client workstation

June 1999 CHAIMS 9

Accessing Remote Servicess

e r

v e

r s

at

prov

ider

's s

ites

c l i

e n

t s

cont

rol

data

IO module IO module

Domain expert

Client workstation

IO module IO module

Domain expert

Client workstation

a

b

cd

e

June 1999 CHAIMS 10

IO module IO module

Domain expert

Client workstation

Accessing Remote Servicess

e r

v e

r s

at

prov

ider

's s

ites

c l i

e n

t s

cont

rol

data

IO module IO module

Domain expert

Client workstation

a

b

cd

e

IO module IO module

Domain expert

Client workstation

June 1999 CHAIMS 11

Composing Computational Services

Megamodules

IO module

a

b

cd

e

s e

r v

e r

s a

tpr

ovid

er's

sit

esc

l i e

n t

CPAM and distribution systems

cont

rol

data

IO module

Domain expert

Client workstation

RemoteComputation

Services

DataResources

June 1999 CHAIMS 12

Example of Remote ServicesBio-informatics:

» several research institutes develop algorithms for modeling and matching molecule structures

» several knowledge bases available, many of them with additional computational services

» a scientist has to combine several of these processing units to produce desired result

June 1999 CHAIMS 13

Typical Scenario: LogisticsA general has to ship troops and/or various material from Los Angeles to Washington DC:

– different kind of material: criteria for preferred transport differ

– not every airport equally suited

– congestion, prices

– actual weather

– certain due or ready dates

Today: calling different companies, looking up information on the web, reservations by hand

» hand-coded systems

Tomorrow: system proposes possibilities that take into account various conditions

» composition of processes

June 1999 CHAIMS 14

Advantages

Knowledge» separation of concerns (provider - composer)» technical expertise (databases …)» maintenance, newest algorithms

Availability» pay per usage instead of pay per instance» lower barrier for clients to try out new modules» choice of several providers even at run-time

Resources» access to computation intensive modules for everybody» minimal local resources possible

Yet there are also challenges...

Advantages of reuse by accessing remote services:

June 1999 CHAIMS 15

Challenge: Computation Intensive Services

What we would like:==> exploiting parallelism among services without

complex coding on client side

==> partial extraction of results when they are ready

==> progressive extraction of results

==> monitoring progress of service invocation and results

==> cost estimation prior to invocation of a service for possibility to choose cheapest or fastest service

Services are not free for a client:• execution time of a service

• transfer time and fee for data

• fees for services

June 1999 CHAIMS 16

Challenge: Autonomy

Separation of control ==> client has no direct control over:

• availability of services • resources provided for services

Heterogeneity:• heterogeneity concerning languages,

platforms as well as distribution protocols

• legacy code, applications and libraries

June 1999 CHAIMS 17

Challenge: Composition Task

Services must somehow be composed on the client-side

• automated: generation of effective access code

• composition-oriented: composition only language or interface

• easy: simple for domain experts to use

June 1999 CHAIMS 18

other language

or

interpreter

or

incremental programming

server side client side

CHAIMS Architecture

CPAM protocol

Distribution Systems (CORBA, RMI…)

Composer

interacts with

ed

a

b

cMEGA Modules

CHAIMS Repository

adds information to

MegamoduleProvider

providesmegamodules

Wrapper Templates

June 1999 CHAIMS 19

Implementation of CPAMOn top of various protocols:

» RMI, CORBA, DCE, plain TCP/IP, http» each primitive is a procedure call, or a pair of XML messages» or mimicked by other interfaces: SWAP, several methods

Method parameters:» simple data element, or recursive structure of data elements» each data element has also type and descriptive name

information» encoding: now ASN.1, future XML» either megamodule conformity or ontology mediator

megamodules

Wrappers for CPAM compliance:» asynchrony and state management (client-specific

parameters, results)» progress information and cost estimation» data encoding

June 1999 CHAIMS 20

server side client side

CHAIMS Architecture

CPAM protocol

Distribution Systems (CORBA, RMI…)

Composer

writes

Client Side Run Time

CLAM program

CHAIMS Compiler

generates ed

a

b

cMEGA Modules

CHAIMS Repository

adds information to

MegamoduleProvider

providesmegamodules

Wrapper Templates

June 1999 CHAIMS 21

Architecture: CLAM andCHAIMS-Protocols

Megaprogram

Megaprogrammer

M e g a m o d u l e s

CLAM

CHAIMS-protocols

CORBA-idl DCE-idl Java-class

Compiler/Interpreter

June 1999 CHAIMS 22

CLAM Primitives

Pre-invocation:SETUP: set up the connection to a megamodule

SET-, GETPARAM: in a megamodule

ESTIMATE: for optimization

Invocation and result gathering:INVOKE

EXAMINE: test progress of an invoked method

EXTRACT: extract results from an invoked method

Termination:TERMINATE: terminate a method invocation/connection to

a megamodule

Control:WHILE, IF

June 1999 CHAIMS 23

Pre-invocation: SETUP• Establish communication between

client CLAM program and a megamodule

• Returns a megamodule handle

• Maps logical name of service to provider

• ex:mm_handle = SETUP (“TransportationModule”)

• CLAM programmers avoid module location and transportation protocol issues

• Authentication occurs here (open question).

June 1999 CHAIMS 24

Pre-invocation: ESTIMATE• Returns cost of a method invocation in terms of

fee, time, and data volume

• Considers values of method parameters

• ESTIMATE for runtime module selection

• Useful for scheduling and cost minimization.

June 1999 CHAIMS 25

Cost Estimation for SelectionChoosing megamodules:

RouteChoose - Optimum - …

BestPick- BestRoute- ...

CHAI

MS client

tim

e

SETUP () SETPARAM (attributes essential for cost estimation)(f1=fee, t1=time) = ESTIMATE (“Optimum”)

SETUP () SETPARAM (attributes essential for cost estimation)(f2=fee, t2=time) = ESTIMATE (“BestRoute”)

calculate cost function==> today BestPick is better

BestPick.INVOKE (“BestRoute”, …)

June 1999 CHAIMS 26

Pre-invocation: SET-, GETPARAM• GET-: returns current parameter values

• SET -: overrides current parameter values

• Operates on a megamodule handle

• ex:mm_handle.SETPARAM (city = “Amsterdam”)

• Utility

–presetting for ESTIMATE

–avoid resetting with repeated invocations

–simple input perturbation.

June 1999 CHAIMS 27

Example: parameters of method TripPlan

general default valuesclient-specific settings for client A

method TripPlan, client A:invocation-specific settings

client-specific settings for client B

from deststart-date

end-date number

1JAN1998 1JAN1998

6OCT19984OCT1998

7OCT19984OCT1998

LAS BWI 1

2

actual values used in invocation TripPlan1

SJO ZRH

SFO

1

1

SFO

SJO6OCT19984OCT1998 ZRH

ZRHactual values used in invocation TripPlan2

FRA

7OCT1998

9DEC19981DEC1998

9DEC19981DEC1998 2LAS FRAactual values used in invocation TripPlan1

method TripPlan, client A:invocation-specific settings

method TripPlan, client B:invocation-specific settings

Hierarchical Setting of Parameters

June 1999 CHAIMS 28

CLAM Primitives








a megamodule

Control:WHILE, IF

June 1999 CHAIMS 29

Invocation: INVOKE• Starts execution of a method

• Operates on a megamodule handle

• Returns an invocation handle

• ex:in_handle = mm_handle.INVOKE (“GenericMethod”)

in_handle = mm_handle.INVOKE (“RoutePlanMethod”, Pair_of_Cities = cities)

• Achieves asynchrony in sequential clients.

June 1999 CHAIMS 30

Invocation: EXAMINE• Determines the state of a invocation

• Operates on an invocation handle

• Returns:– (status, “progress information”)

– status = {DONE, NOT_DONE, PARTIAL, ERROR}

• Utility

–Progressive extractions

–Partial extractions

–Optimized module execution scheduling

–Steering.

June 1999 CHAIMS 31

Invocation: EXAMINEquick_route: found at time 30

optimal_route: found at time 100

• Method Status:WHILE (in_handle.EXAMINE() != DONE) {}

(routeVar = quick_route) = in_handle.EXTRACT()

• Attribute Status:WHILE (in_handle.EXAMINE(quick_route) != DONE) {}

(routeVar = quick_route) = in_handle.EXTRACT()

• Steering:IF (in_handle.EXAMINE(quick_route) != PARTIAL)

… ask user (or scheduler) to continue or halt…

June 1999 CHAIMS 32

Invocation: EXTRACT• Collects data from a megamodule method

• Operates on an invocation handle

• Returns: assignment of values to parameters

• ex:(routeVar = quick_route) = in_handle.EXTRACT()

• Allows partial data retrieval

–saves volume: abandon uninteresting elements

–saves time: termination of useless invocation

• Allows progressive data retrieval.

June 1999 CHAIMS 33

CLAM Data Types• Simple data types still have a role

– opaque //exclusively used for user data

– integer

– string

– Boolean

– real

• No classes or amalgamated types

• Only be assigned and compared

• Used for control (not opaque)

June 1999 CHAIMS 34

CLAM Primitives








a megamodule

Control:WHILE, IF

June 1999 CHAIMS 35

TERMINATE• Ends method invocation or closes a

connection to a megamodule

• Why necessary?– delayed extractions

– repeated/progressive extractions

– partial extractions

– early termination

• ex:in_handle.TERMINATE ()

mm_handle.TERMINATE ()

June 1999 CHAIMS 36

CLAM Composition Language

Purely compositional:– no primitives for arithmetic – no primitives for input/output

Splitting up CALL-statement:– parallelism by asynchrony in sequential program

– novel possibilities for optimizations– reduction of complexity of invoke statements

Higher-level language: – assembler => HLLs,

– HLLs => composition/megamodule paradigm

June 1999 CHAIMS 37

Ongoing & Upcoming Research Issues

CPAM

automatedcomposition

CLAM

graphicalcomposition

incrementalcomposition

automatedscheduling ofinvocations

electronicmarketplaceof services exploiting

XML web-services

direct dataflowsbetween megamodules

optimization ofcompositions

repositoryin XML, ...

directoryservices

CHAIMSbroker

repositorybrowser

compositionwizard

sign-up andauthentication

cost estimation

incrementalresult extraction

c o m p o s i t i o n

e - c o m m e r c e p r o t o c o l

Repository

June 1999 CHAIMS 38

Conclusion

• New reuse model» computational modules stay at provider’s site

» reuse of computational modules by accessing computational services over the internet

• New issues» cost-estimation, progress monitoring

» easy and efficient composition of remote, computation-intensive services

June 1999 CHAIMS 39

Questions?http://www-db.stanford.edu/CHAIMS/

June 1999 CHAIMS 40

Other Approaches

Traditional Languages/Extensions– FX: Self-contained language for module building

composition. Much finer grain than service composition.

– Orca, Opus: Extensions to HPF. Programmers require intimate knowledge of all process data.

Composition Languages– MANIFOLD: Similar IWIM model, with black box

process

– Myriad scripting languages, generally assume complete ontological control of services

June 1999 CHAIMS 41

Megaprogram Example: Codeio_mmh = SETUP ("InputOutput")route_mmh = SETUP ("RouteInfo")...best2_mmh.SETATTRIBUTES (criterion = "cost")

cities_default = route_mmh.GETPARAM(Pair_of_Cities)input_cities_ih = io_mmh.INVOKE ("input”, cities_default)WHILE (input_cities_ih.EXAMINE() != DONE) {}cities = input_cities_ih.EXTRACT()...route_ih = route_mmh.INVOKE ("AllRoutes", Pair_of_Cities = cities)WHILE (route_ih.EXAMINE() != DONE) {}routes = route_ih.EXTRACT() …

IF (best1_mmh.ESTIMATE("Best_Route") < best2_mmh.ESTIMATE("Optimum") ) THEN {best_ih = best1_mmh.INVOKE ("Best_Route", Goods = info_goods, Pair_of_Cities = cities, List_of_Routes = routes, Cost_Ground = cost_list_ground, Cost_Air = cost_list_air)}ELSE {best_ih = best2_mmh.INVOKE ("Optimum", Goods = info_goods, …...best2_mmh.TERMINATE()

// Setup connections to megamodules.

// Set global variables valid for all invocations // of this client.

// Get information from the megaprogram user // about the goods to be transported and about// the two desired cities.

// Get all routes between the two cities.

//Get all city pairs in these routes.//Calculate the costs of all the routes.

// Figure out the optimal megamodule for// picking the best route.

//Pick the best route and display the result.

// Terminate all invocations

June 1999 CHAIMS 42

Name of Person

Data objects: Gentype bagsMinimal Typing within CHAIMS:

Integer, Boolean only for controlAll else is placed into an ASN.1 bag, transparent to compiler :

A Gentype is a triple of name, type and value, where value is either a simple type or a list of other gentypes (i.e. a complex type).

Simple types: given by ASN.1, the ASN.1-conversion library for C++, our own conversion routines.

Example: Person_Information complex

First Name string Joe Last Name string Smith

Personal Data complex Address

Date of Birth date 6/21/54 Soc.Sec.No string 345-34-3456

June 1999 CHAIMS 43

Scheduling: Simple Example

1 cost_ground_ih = cost_mmh.INVOKE ("Cost_for_Ground", 1 List_of_City_Pairs = city_pairs,Goods = info_goods)

2 WHILE (cost_ground_ih.EXAMINE() != DONE) {} 3cost_list_ground = cost_ground_ih.EXTRACT()

3 cost_air_ih = cost_mmh.INVOKE ("Cost_for_Air", 2 List_of_City_Pairs = city_pairs,Goods = info_good)

4 WHILE (cost_air_ih.EXAMINE() != DONE) {} 4cost_list_air = cost_air_ih.EXTRACT()

order inunscheduledmegaprogram

order in automaticallyprescheduled megaprogram

june 1999 chaims1 neal sample stanford university objective: investigate new approaches to...

Documents

b c d e e

b c d e s e r v e r

remote services s e

b c d e slide

services slide

b c d e domain expert

i e n t io

local components