Economics Paradigm for “Resource Management and

Scheduling” for Service Oriented P2P/Grid

Computing

Rajkumar Buyya

Melbourne, Australiawww.buyya.com/ecogrid

WW Grid

2

3

Need Honest Answers!

I want to have access to your Grid resources & want to knowhow many of you are willing to give me access ? (following cases)

I am unable to give you access our Australian machines, but I want to have access to yours! [social]

Want to solve academic problems Want to solve business problems

I am willing to gift you Kangaroos! [bartering] I am willing to give you access to my machines, if you

want. (sharing, but no measure & no QoS) [bartering] I am willing to pay you dollars on usage basis.

[economic incentive, market-based, and QoS]

WW Grid

4

Overview

A quick glance at today’s Grid computing Resource Management challenges for next

generation Grid computing A Glance at Approaches to Grid computing. Grid Architecture for Computational Economy Economy Grid = Globus + GRACE Nimrod-G -- Grid Resource Broker Scheduling Experiments Case Study: Drug Design

Application on Grid Conclusions

Scheduling Economics

Grid

EconomyGrid

5

2100

2100 2100 2100 2100

2100 2100 2100 2100

Desktop SMPs or SuperComputers

LocalCluster

GlobalCluster/Grid

PERFORMANCE

Inter PlanetaryGrid!

•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Universe

Administrative Barriers

EnterpriseCluster/Grid

?

Scalable HPC: Breaking Administrative Barriers & new challenges

6

Why Grids? Large Scale Explorations need them—Killer

Applications. Solving grand challenge applications using

modeling, simulation and analysis

Life Sciences

CAD/CAM

Aerospace

Military ApplicationsDigital Biology Military ApplicationsMilitary Applications

Internet & Ecommerce

7

8

What is Grid ?

An infrastructure that logically couples distributed resources:

Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc;

Software – e.g., ASPs renting expensive special purpose applications on demand;

Catalogued data and databases – e.g. transparent access to human genome database;

Special devices – e.g., radio telescope – SETI@Home searching for life in galaxy.

People/collaborators. and presents them as an integrated global resource. It enables the creation of virtual enterprises (VEs)

for resource sharing.

Widearea

data archives

9

P2P/Grid Applications-Drivers

Distributed HPC (Supercomputing): Computational science.

High-Capacity/Throughput Computing: Large scale simulation/chip design & parameter studies.

Content Sharing (free or paid) Sharing digital contents among peers (e.g., Napster)

Remote software access/renting services: Application service provides (ASPs) & Web services.

Data-intensive computing: Virtual Drug Design, Particle Physics, Stock Prediction...

On-demand, realtime computing: Medical instrumentation & Mission Critical.

Collaborative Computing: Collaborative design, Data exploration, education.

Service Oriented Computing (SOC): Computing as Utility: New paradigm and new industries.

10

Building and Using Grids require

Services that make our systems Grid Ready! Security mechanisms that permit resources

to be accessed only by authorized users. (New) programming tools that make our

applications Grid Ready!. Tools that can translate the requirements of

an application/user into the requirements of computers, networks, and storage.

Tools that perform resource discovery, trading, selection/allocation, scheduling and distribution of jobs and collects results.

Globus

?

11

Players in Grid Computing

12

What users want ?Users in Grid Economy &

Strategy Grid Consumers

Execute jobs for solving varying problem size and complexity

Benefit by selecting and aggregating resources wisely Tradeoff timeframe and cost

Strategy: minimise expenses Grid Providers

Contribute “idle” resource for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity

Strategy: maximise return on investment

Challenges for Next Generation Grid

Technology Development

14

Challenges for Grid Computing

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Development Tools

15

Sources of Complexity in Resource Management for World Wide Grid

Computing Size (large number of nodes, providers, consumers) Heterogeneity of resources (PCs, Workstations, clusters, and

supercomputers, instruments, databases, software) Heterogeneity of fabric management systems (single system image OS,

queuing systems, etc.) Heterogeneity of fabric management polices Heterogeneity of application requirements (CPU, I/O, memory, and/or

network intensive) Heterogeneity in resource demand patterns (peak, off-peak, ...) Applications need different QoS at different times (time critical results). The

utility of experimental results varies from time to time. Geographical distribution of users & located different time zones Differing goals (producers and consumers have different objectives and

strategies) Unsecure and Unreliable environment

16

Traditional approaches to resource management & scheduling are NOT useful

for Grid ? They use centralised policy that need

complete state-information and common fabric management policy or decentralised consensus-based

policy. Due to too many heterogenous parameters in the Grid it is impossible to

define/get: system-wide performance matrix and common fabric management policy that is acceptable to all.

“Economics” paradigm proved to effective institution in managing decentralization and heterogeneity that is present in human economies!

Fall of USSR & Emergence of US as world superpower! (monopoly?) So, we propose/advocate the use of computational economics principles

in management of resources and scheduling computations on world wide Grid.

Think locally and act globally approach to grid computing!

17

Benefits of Computational Economies

It provides a nice paradigm for managing self interested and self-regulating entities (resource owners and consumers)

Helps in regulating supply-and-demand for resources. Services can be priced in such a way that equilibrium is maintained.

User-centric / Utility driven: Value for money! Scalable:

No need of central coordinator (during negotiation) Resources(sellers) and also Users(buyers) can make their own decisions and try to

maximize utility and profit. Adaptable It helps in offering different QoS (quality of services) to different applications

depending the value users place on them. It improves the utilisation of resources It offers incentive for resource owners for being part of the grid! It offers incentive for resource consumers for being good citizens There is large body of proven Economic principles and techniques available, we can

easily leverage it.

18

New challenges of Computational Economy

Resource Owners How do I decide prices ? (economic models?) How do I specify them ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …..

Resource Consumers How do I decide expenses ? How do I express QoS requirements ? How I trade between timeframe & cost ? ….

Any tools, traders & brokers available to automate the process ?

19

mix-and-match

Object-oriented

Internet/partial-P2P

Network enabled Solvers

Market/Computational Economy

20

Many Grid Projects & Initiatives

Australia Economy Grid Nimrod-G Virtual Lab Active Sheets DISCWorld ..new coming up

Europe UNICORE MOL Lecce GRB Poland MC Broker EU Data Grid EuroGrid MetaMPI Dutch DAS XW, JaWS and many more...

Japan Ninf DataFarm and many more...

USA Globus Legion Javelin AppLeS NASA IPG Condor Harness NetSolve AccessGrid GrADS and many more...

Cycle Stealing & .com Initiatives Distributed.net SETI@Home, …. Entropia, UD, Parabon,….

Public Forums Global Grid Forum P2P Working Group IEEE TFCC Grid & CCGrid conferences

http://www.gridcomputing.com

21

Many Testbeds ? & who pays ?,

who regulates supply and demand ?

GUSTO (decommissioned)

Legion Testbed

NASA IPG

World Wide Grid

WW Grid

22

Testbeds so far -- observations

Who contributed resources & why ? Volunteers: for fun, challenge, fame, charismatic apps, public

good like distributed.net & SETI@Home projects. Collaborators: sharing resources while developing new

technologies of common interest – Globus, Legion, Ninf, Ninf, MC Broker, Lecce GRB,... Unless you know lab. leaders, it is impossible to get access!

How long ? Short term: excitement is lost, too much of admin. Overhead

(Globus inst+), no incentive, policy change,… What we need ? Grid Marketplace!

Regulates supply-and-demand, offers incentive for being players, simple, scalable solution, quasi-deterministic – proven model in real-world.

23

Building an Economy Grid(Next Generation Grid

Computing!)

To enable the creation and promotion of:Grid Marketplace (competitive)

ASPService Oriented Computing

. . .And let users focus on their own work (science, engineering, or commerce)!

24

Grid Node N

GRACE: A ReferenceGrid Architecture for Computational Economy

Grid User

Application

Grid Resource Broker

Grid Service Providers

Grid Explorer

Schedule Advisor

Trade Manager

Job ControlAgent

Deployment Agent

Trade Server

Resource Allocation

ResourceReservation

R1

Misc. services

Information Server(s)

R2 Rm…

Pricing Algorithms

Accounting

Grid Node1

…

Grid Middleware Services

…

…

HealthMonitor

Grid Market Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign-on

Grid Bank

See PDPTA 2000 paper!

25

Economic Models

Price-based: Supply,demand,value, wealth of economic system

Commodity Market Model Posted Price Model Bargaining Model Tendering (Contract Net) Model Auction Model

English, first-price sealed-bid, second-price sealed-bid (Vickrey), and Dutch (consumer:low,high,rate; producer:high, low, rate)

Proportional Resource Sharing Model Monopoly (one provider) and Oligopoly (few players)

consumers may not have any influence on prices. Bartering

Shareholder Model Partnership Model

See SPIE ITCom 2001 paper!: with Heinz Stockinger, CERN!

26

Grid Open Trading Protocols

Get Connected

Call for Bid(DT)

Reply to Bid (DT)

Negotiate Deal(DT)

Confirm Deal(DT, Y/N)

….

Cancel Deal(DT)

Change Deal(DT)

Get Disconnected

Trade Manager Trade Server

Pricing Rules

DT - Deal Template - resource requirements (BM) - resource profile (BS) - price (any one can set) - status - change the above values - negotiation can continue - accept/decline - validity period

API

27

GridFabric

GridApps.

GridMiddleware

GridTools

Networked Resources across Organisations

Computers Clusters Data Sources Scientific InstrumentsStorage Systems

Local Resource Managers

Operating Systems Queuing Systems TCP/IP & UDP

…

Libraries & App Kernels …

Distributed Resources Coupling Services

Security Information … QoSProcess

Development Environments and Tools

Languages Libraries Debuggers … Web toolsResource BrokersMonitoring

Applications and Portals

Prob. Solving Env.Scientific …CollaborationEngineering Web enabled Apps

Resource Trading

Grid Components

Market Info

28

Economy Grid = Globus + GRACE

Applications

MDS

GRAMGlobus Security Interface

Heartbeat MonitorNexus

Local Services

LSF

Condor GRD QBank

PBS

TCP

SolarisIrixLinux

UDP

High-level Services and Tools

DUROC globusrunMPI-G Nimrod/GCC++

Grid Status

GASS

GRACE-TS

GARA

GridFabric

GridApps.

GridMiddleware

GridTools

GBankGMD

eCash

JVM

DUROC

Core Services

Science

Engineering Commerce Portals ActiveSheet……

See IPDPS HWC 2001 paper!

……

29

GRACE components

A resource broker (e.g., Nimrod/G) Various resource trading protocols for different

economic models A mediator for negotiating between users and

grid service providers (Grid Market Directory) A deal template for specifying resource

requirements and services offers Grid Trading Server Pricing policy specification Accounting (e.g., QBank) and payment

management (GridBank, not yet implemented)

30

Pricing, Accounting, Allocations and Job Scheduling Flow @ each site/Grid Level

QBankQBank

Resource Manager44

IBM-LL/PBS/….

00

55 88

66 77

Compute Resourcesclusters/SGI/SP/...

0. Make Deposits, Transfers, Refunds, Queries/Reports1. Clients negotiates for access cost.2. Negotiation is performed per owner defined policies. 3. If client is happy, TS informs QB about access deal.4. Job is Submitted5. Check with QB for “go ahead”6. Job Starts7. Job Completes8. Inform QB about resource resource utilization.

Trade Server 3311

Pricing PolicyPricing Policy22

DB@Each SiteDB@Each Site

GRID BankGRID Bank(digital transactions)(digital transactions)00

31

Service Items to be Charged

CPU - User and System time Memory:

maximum resident set size - page size amount of memory used page faults: with/without physical I/O

Storage: size, r/w/block IO operations Network: msgs sent/received Signals received, context switches Software and Libraries accessed Data Sources (e.g. Protein Data Bank)

32

How to decide Price ?

Fixed price model (like today’s Internet) Dynamic/Demand and Supply (like tomorrow’s Internet) Usage Period Loyalty of Customers (like Airlines favoring frequent flyers!) Historical data Advance Agreement (high discount for corporations) Usage Timing (peak, off-peak, lunch time) Calendar based (holiday/vacation period) Bulk Purchase (register 100 .com domains at once!) Voting -- trade unions decide pricing structure Resource capability as benchmarked in the market! Academic R&D/public-good application users can be offered at

cheaper rate compared to commercial use. Customer Type – Quality or price sensitive buyers. Can be Prescribed by Regulating (Govt.) authorities

33

Payments- Options & Automation

Buy credits in advance / GSPs bill the user later--”pay as you go”

Pay by Electronic Currency via Grid Bank NetCash (anonymity), NetCheque, and Paypal NetCheque: - http://www.isi.edu/gost/info/netcash/

Users register with NC accounting servers, can write electronic cheques and send (e.g email). When deposited, balance is transferred from sender to receiver account.

NetCash - http://www.isi.edu/gost/info/netcheque/ It supports anonymity and it uses the NetCheque system to

clear payments between currency servers. Paypal.com– account+email is linked to credit card.

Enter the recipient’s email address and the amount you wish to request.

The recipient gets an email notification and pays you at www.PayPal.com

Nimrod-G:The Grid Resource Broker

Soft Deadline and Budget-based Economy Grid Resource Broker

for Parameter Processing on P2P Grids

35

Parametric Computing(What Users think of Nimrod

Power)

Multiple RunsSame ProgramMultiple Data Killer Application for the Grid!

ParametersAge Hair

23 CleanAge Hair

23 Clean23 Beard28 Goatee

Age Hair23 Clean23 Beard

Age Hair23 Clean23 Beard28 Goatee28 Clean

Age Hair23 Clean23 Beard28 Goatee28 Clean19 Moustache

Age Hair23 Clean23 Beard28 Goatee28 Clean19 Moustache10 Clean

Age Hair23 Clean23 Beard28 Goatee28 Clean19 Moustache10 Clean

-4000000 Too much

Courtesy: Anand Natrajan, University of Virginia

Magic Engine

See IPDPS 2000 paper!

36

P-study Applications -- Characteristics

Code (Single Program: sequential or threaded)

High Resource Requirements Long-running Instances Numerous Instances (Multiple Data) High Computation-to-Communication

Ratio Embarrassingly/Pleasantly Parallel

37

Sample P-Sweep ApplicationsSample P-Sweep Applications

Bioinformatics: Bioinformatics: Drug Design / Protein Drug Design / Protein

ModellingModelling

SensitivitySensitivityexperiments experiments

on smog formationon smog formation

Combinatorial Combinatorial Optimization:Optimization:

Meta-heuristic Meta-heuristic parameter estimationparameter estimation

Ecological Modelling: Ecological Modelling: Control Strategies Control Strategies

for Cattle Tickfor Cattle Tick

Electronic CAD: Electronic CAD: Field Programmable Field Programmable

Gate ArraysGate ArraysComputer Graphics: Computer Graphics: Ray TracingRay Tracing

High Energy High Energy Physics: Physics:

Searching for Searching for Rare EventsRare Events

Finance: Finance: Investment Risk AnalysisInvestment Risk Analysis

VLSI Design: VLSI Design: SPICE SimulationsSPICE Simulations

Aerospace: Aerospace: Wing DesignWing Design

Network SimulationNetwork SimulationAutomobile:Automobile:

Crash Simulation Crash Simulation

Data MiningData Mining

Civil Engineering:Civil Engineering:Building Design Building Design

astrophysics astrophysics

38

Thesis

Perform parameter sweep (bag of tasks) (utilising distributed resources) within “T” hours or early and cost not exceeding $M.

Three Options/Solutions: Using pure Globus commands Build your own Distributed App & Scheduler Use Nimrod-G (Resource Broker)

39

Remote Execution Steps

Choose Resource

Transfer Input Files

Set Environment

Start Process

Pass Arguments

Monitor Progress

Read/Write Intermediate Files

Transfer Output Files

Summary ViewJob ViewEvent View

+Resource Discovery, Trading, Scheduling, Predictions, Rescheduling, ...

40

Using Pure Globus commands

Do all yourself! (manually)

Total Cost:$???

41

Build Distributed Application & Scheduler

Build App case by case basisComplicated Construction

E.g., AppLeS/MPI based Total Cost:$???

42

Use Nimrod-G

Aggregate Job SubmissionAggregate View

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East

West

North

South

Submit & Play!

43

Nimrod & Associated Family of Tools

P-sweep App. Composition: Nimrod/

EnfusionResource Management and Scheduling:

Nimrod-G BrokerDesign Optimisations:

Nimrod-OApp. Composition and Online Visualization:

Active SheetsGrid Simulation in Java:

GridSimDrug Design on Grid:

Virtual Lab

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East

West

North

South

Remote Execution Server(on demand Nimrod Agent)

File Transfer Server

Upcoming?: HEPGrid (+U. Melbourne), GAVE(+Rutherford Appleton Lab)Grid (Un)Aware Virtual Engineering

44

A resource broker for managing, steering, and executing task farming (parametric sweep/SPMD model) applications on Grid based on deadline and computational economy.

Based on users’ QoS requirements, our Broker dynamically leases services at runtime depending on their quality, cost, and availability.

Key Features A single window to manage & control experiment Persistent and Programmable Task Farming Engine Resource Discovery Resource Trading Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & results Steering & data management Accounting

Nimrod/G : A Grid Resource Broker

45

A Glance at Nimrod-G Broker

Grid Middleware

Nimrod/G Client Nimrod/G ClientNimrod/G Client

Grid Information Server(s)

Schedule Advisor

Trading Manager

Nimrod/G Engine

GridStore

Grid Explorer

GE GISTM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

Globus, Legion, Condor, etc.

G

G

CL

Globus enabled node.Legion enabled node.

GL

Condor enabled node.

RM & TSRM & TS

C LSee HPCAsia 2000 paper!

46

Globus Legion

Fabric

Nimrod Broker

Nimrod ClientsP-Tools (GUI/Scripting)(parameter_modeling)

Legacy Applications

P2P GTS

Farming Engine

Dispatcher & Actuators

Schedule Advisor

Trading Manager

Grid Explorer

Customised Apps(Active Sheet)

Monitoring and Steering Portals

Algorithm1

AlgorithmN

Middleware

. . .

Computers Storage Networks InstrumentsLocal Schedulers

G-Bank. . .

Agents

Resources

Programmable Entities Management

Jobs Tasks

. . .

AgentScheduler JobServer

PC/WS/Clusters Radio TelescopeCondor/LL/Mosix/ . . .Database

Meta-Scheduler

Nimrod/G Grid Broker Architecture

Globus-A

Channels

Legion-A P2P-A. . .

Database(Postgres)

XML

Condor GMD

XML?

IP hourglass ?

47

A Nimrod/G Monitor

A Nimrod/G Monitor

CostCostDeadlineDeadline

Legion hosts

Globus Hosts

Bezek is in both Globus and Legion Domains

Arlington

Alexandria

Richmond

HamptonNorfolk

Virginia BeachChesapeakePortsmouth

Newport News

Roanoke

Ap p om a toxRive r

Ja m esRive r

Shena nd oa hRive r

Ra p p a ha nnoc kRive r

Potom a cRive r

VIRGINIA77

81

64

64

66

85

48

User Requirements: Deadline/Budget User Requirements: Deadline/Budget

49

Active Sheet: Spreadsheet Processing on Grid

NimrodNimrodProxyProxy

Nimrod/GNimrod/G

See HPC 2001 paper!

50

51

Nimrod/G Interactions

Grid Infoservers

Resource Discovery

QueuingSystem

Processserver

Resource allocation

(local)

Userprocess

File accessI/Oserver

Gatekeeper node

NimrodAgent

Computational node

Dispatcher

Root node

Scheduler

FarmingEngine

Grid Trade Server

“Do this in 30min. for $10?”

52

Discover Discover ResourcesResources

Distribute JobsDistribute Jobs

Establish Establish RatesRates

Meet requirements ? Remaining Meet requirements ? Remaining Jobs, Deadline, & Budget ?Jobs, Deadline, & Budget ?

Evaluate & Evaluate & RescheduleReschedule

Discover Discover More More

ResourcesResources

Adaptive SchedulingAlgorithms

Execution Time (not beyond deadline)

Execution Cost (not beyond budget)

Time Minimisation Minimise Limited by budgetCost Minimisation Limited by deadline MinimiseNone Minimisation Limited by deadline Limited by budget

Adaptive Scheduling Algorithms

Compose & Compose & ScheduleSchedule

See HPDC AMS 2001 paper!

53

Cost Model

Without cost ANY shared system becomes un-managable

Charge users more for remote facilities than their own

Choose cheaper resources before more expensive ones

Cost units (G$) may be Dollars Shares in global facility Stored in bank

54

Cost Matrix @ Grid site X

Non-uniform costing Encourages use of

local resources first Real accounting

system can control machine usage

11 33

22 11User 5User 5

Mach

ine 1

Mach

ine 1

User 1User 1

Mach

ine 5

Mach

ine 5

Resource Cost = Function (cpu, memory, disk, network, software, QoS, current demand, etc.)

Simple: price based on peaktime, offpeak, discount when less demand, ..

55

Deadline and Budget-based Cost Minimization Scheduling

1. Sort resources by increasing cost.2. For each resource in order, assign as

many jobs as possible to the resource, without exceeding the deadline.

3. Repeat all steps until all jobs are processed.

56

M - Resources, N - Jobs, D - deadline Note: Cost of any Ri is less than any of Ri+1 …. Or Rm

RL: Resource List need to be maintained in increasing order of cost Ct - Time when accessed (Time now) Ti - Job runtime (average) on Resource i (Ri) [updated periodically]

Ti is acts as a load profiling parameter. Ai - number of jobs assigned to Ri , where:

Ai = Min (No.Unassigned Jobs, No. Jobs Ri can complete by remaining deadline) No.UnAssignedJobsi = Diff( N, (A1+…+Ai-1)) JobsRi consume = RemainingTime (D- Ct) DIV Ti

ALG: Invoke Job Assignment() periodically until all jobs done. Job Assignment()/Reassignment():

Establish ( RL, Ct , Ti , Ai ) dynamically – Resource Discovery. For all resources (I = 1 to M) { Assign Ai Jobs to Ri , if required}

Deadline-based Cost-minimization Scheduling

57

Deadline and Budget Constraint (DBC) Time Minimization

Scheduling

1. For each resource, calculate the next completion time for an assigned job, taking into account previously assigned jobs.

2. Sort resources by next completion time.3. Assign one job to the first resource for

which the cost per job is less than the remaining budget per job.

4. Repeat all steps until all jobs are processed. (This is performed periodically or at each scheduling-event.)

58

Deadline and Budget Constraint (DBC) Time+Cost Min. Scheduling

1. Split resources by whether cost per job is less than budget per job.

2. For the cheaper resources, assign jobs in inverse proportion to the job completion time (e.g. a resource with completion time = 5 gets twice as many jobs as a resource with completion time = 10).

3. For the dearer resources, repeat all steps (with a recalculated budget per job) until all jobs are assigned.

4. [Schedule/Reschedule] Repeat all steps until all jobs are processed.

Evaluation of Scheduling Heuristics

A Hypothetical Application on

World Wide Grid

WW Grid

60

Globus+LegionGRACE_TS

Australia

Monash Uni.:

Linux cluster

Solaris WS

Nimrod/G

Globus +GRACE_TS

Europe

ZIB/FUB: T3E/Mosix Cardiff: Sun E6500Paderborn: HPCLineLecce: Compaq SCCNR: ClusterCalabria: Cluster CERN: ClusterPozman: SGI/SP2

Globus +GRACE_TS

Asia/Japan

Tokyo I-Tech.:ETL, Tuskuba

Linux cluster

Globus/LegionGRACE_TS

North America

ANL: SGI/Sun/SP2USC-ISI: SGIUVa: Linux ClusterUD: Linux clusterUTK: Linux cluster

Internet

World Wide Grid (WWG)

Globus +GRACE_TS South America

Chile: Cluster

WW Grid

WW Grid

61

Experiment-1 Setup

Workload: 165 jobs, each need 5 minute of cpu time

Deadline: 1 hrs. and budget: 800,000 units

Strategy: minimise cost and meet deadline

Execution Cost with cost optimisation AU Peaktime:471205 (G$) AU Offpeak time: 427155 (G$)

62

Resources Selected & Price/CPU-sec.

Resource Type & Size

Owner and Location

Grid services

Peaktime Cost (G$)

Offpeak cost

Linux cluster (60 nodes)

Monash, Australia

Globus/Condor 20 5

IBM SP2 (80 nodes)

ANL, Chicago, US

Globus/LL 5 10

Sun (8 nodes) ANL, Chicago, US

Globus/Fork 5 10

SGI (96 nodes) ANL, Chicago, US

Globus/Condor-G

15 15

SGI (10 nodes) ISI, LA, US Globus/Fork 10 20

63

Execution @ AU Peak Time

0

2

4

6

8

10

12

Time (minutes)

Jo

bs

Linux clus ter - Monash (20) Sun - ANL (5) SP2 - ANL (5) SGI - ANL (15) SGI - ISI (10)

64

Execution @ AU Offpeak Time

0

2

4

6

8

10

12

Time (minutes)

Jo

bs

Linux clus ter - Monash (5) Sun - ANL (10) SP2 - ANL (10) SGI - ANL (15) SGI - ISI (20)

65

AU peak: Resources/Cost in Use

0

50

100

150

200

250

300

350

400

450

500

Tim e (in m in.)

Co

st o

f R

eso

urc

es in

Use

0

5

10

15

20

25

30

35

40

Tim e (in m in.)

Res

ou

rces

(N

o. o

f C

PU

s) in

Use

After the calibration phase, note the difference in pattern of two graphs. This is when scheduler stopped using

expensive resources.

66

AU offpeak: Resources/Cost in Use

0

50

100

150

200

250

300

350

Time (in min.)

Co

st o

f R

eso

urc

es i

n U

se

0

5

10

15

20

25

30

Time (in min.)

Res

ou

rces

(N

o.

of

CP

Us)

in

Use

67

Experiment-2 Setup

Workload: 165 jobs, each need 5 minute of CPU time

Deadline: 2 hrs. and budget: 396000 units Strategy: minimise time / cost Execution Cost with cost optimisation

Optimise Cost: 115200 (G$) (finished in 2hrs.) Optimise Time: 237000 (G$) (finished in 1.25 hr.) In this experiment: Time-optimised scheduling run

costs double that of Cost-optimised. Users can now trade-off between Time Vs. Cost.

68

Resources Selected & Price/CPU-sec.

Resource & Location

Grid services & Fabric

Cost/CPU sec.or unit

No. of Jobs Executed

Time_Opt Cost_Opt.

Linux Cluster-Monash, Melbourne, Australia

Globus, GTS, Condor

2 64 153

Linux-Prosecco-CNR, Pisa, Italy

Globus, GTS, Fork 3 7 1

Linux-Barbera-CNR, Pisa, Italy

Globus, GTS, Fork 4 6 1

Solaris/Ultas2

TITech, Tokyo, Japan

Globus, GTS, Fork 3 9 1

SGI-ISI, LA, US Globus, GTS, Fork 8 37 5

Sun-ANL, Chicago,US Globus, GTS, Fork 7 42 4Total Experiment Cost (G$) 237000 115200

Time to Complete Exp. (Min.) 70 119

69

DBC Scheduling for Time Optimization

0

2

4

6

8

10

12

Time (in Minute)

No.

of

Tas

ks i

n E

xecu

tion

Condor-Monash Linux-Prosecco-CNR Linux-Barbera-CNR

Solaris /Ultas2-TITech SGI-ISI Sun-ANL

70

DBC Scheduling for Cost Optimization

0

2

4

6

8

10

12

14

Time (in Minute)

No.

of

Tas

ks i

n E

xecu

tion

Condor-Monash Linux-Prosecco-CNR Linux-Barbera-CNR

Solaris /Ultas2-TITech SGI-ISI Sun-ANL

Application Case Study

The Virtual Laboratory Project: "Molecular Modelling for Drug Design" on Peer-to-Peer Grid

72

Virtual Drug Design: Data Intensive Computing on Grid

A Virtual Laboratory for “Molecular Modelling for Drug Design” on Peer-to-Peer Grid.

It provides tools for examining millions of chemical compounds (molecules) in the Protein Data Bank (PDB) to identify those having potential use in drug design.

In collaboration with: Kim Branson, Structural

Biology, Walter and Eliza Hall Institute (WEHI)

http://www.csse.monash.edu.au/~rajkumar/vlab

73

Virtual Drug DesignA Virtual Lab for “Molecular Modeling for Drug Design” on P2P Grid

“Screen 2K molecules in 30min. for $10”

Grid Market Directory

ResourceBroker

Grid Info. Service

GTS

GTS

GTS

GTS

“Give me list PDBs sourcesOf type aldrich_300?”

“serv

ice co

st?”

(GTS - Grid Trade Server)

PDB2

“get mol.10 from pdb1 & screen it.”

Data Replica Catalogue

“service providers?”

GTS

PDB1

“mol.10 please?”

“mol.5 please?”

(RB maps suitable Grid nodes and Protein DataBank)

74

DataGrid Brokering

Nimrod/GComputational

Grid Broker

Data Replica CataloguePDB Broker

Algorithm1

AlgorithmN

. . .

PDB Service

PDB2

“Screen mol.5 please?”

GSP1 GSP2 GSP4GSP3(Grid Service Provider)

GSPm

PDB Service

GSPn

1

“advise PDB source?

2“selection & advise: use GSP4!”

5Grid Info. Service

3

“Is GSP4 healthy?”

4

“mol.5 please?”6

“PDB replicas please?”

“Screen 2K molecules in 30min. for $10”

7

“process & send results”

75

Software Tools

Molecular Modelling Tools (DOCK) Parameter Modelling Tools (Nimrod/enFusion) Grid Resource Broker (Nimrod-G) Data Grid Broker Protein Data Bank (PDB) Management and Intelligent Access

Tools PDB databse Lookup/Index Table Generation. PDB and associated index-table Replication. PDB Replica Catalogue (that helps in Resource Discovery). PDB Servers (that serve PDB clients requests). PDB Brokering (Replica Selection). PDB Clients for fetching Molecule Record (Data Movement).

Grid Middleware (Globus and GrACE) Grid Fabric Management (Fork/LSF/Condor/Codine/…)

76

DOCK code*(Enhanced by WEHI, U of

Melbourne)

A program to evaluate the chemical and geometric complementarities between a small molecule and a macromolecular binding site.

It explores ways in which two molecules, such as a drug and an enzyme or protein receptor, might fit together.

Compounds which dock to each other well, like pieces of a three-dimensional jigsaw puzzle, have the potential to bind.

So, why is it important to able to identify small molecules which may bind to a target macromolecule?

A compound which binds to a biological macromolecule may inhibit its function, and thus act as a drug.

Thus disabling the ability of (HIV) virus attaching itself to molecule/protein!

With system specific code changed, we have been able to compile it for Sun-Solaris, PC Linux, SGI IRIX, Compaq Alpha/OSF1

* Original Code: University of California, San Francisco: http://www.cmpharm.ucsf.edu/kuntz/

77

Dock input filescore_ligand yesminimize_ligand yesmultiple_ligands norandom_seed 7anchor_search notorsion_drive yesclash_overlap 0.5conformation_cutoff_factor 3torsion_minimize yesmatch_receptor_sites norandom_search yes . . . . . . . . . . . .maximum_cycles 1ligand_atom_file S_1.mol2receptor_site_file ece.sphscore_grid_prefix ecevdw_definition_file parameter/vdw.defnchemical_definition_file parameter/chem.defnchemical_score_file parameter/chem_score.tblflex_definition_file parameter/flex.defnflex_drive_file parameter/flex_drive.tblligand_contact_file dock_cnt.mol2ligand_chemical_file dock_chm.mol2ligand_energy_file dock_nrg.mol2

Molecule to Molecule to be screenedbe screened

78

score_ligand $score_ligandminimize_ligand $minimize_ligandmultiple_ligands $multiple_ligandsrandom_seed $random_seedanchor_search $anchor_searchtorsion_drive $torsion_driveclash_overlap $clash_overlapconformation_cutoff_factor $conformation_cutoff_factortorsion_minimize $torsion_minimizematch_receptor_sites $match_receptor_sitesrandom_search $random_search . . . . . . . . . . . .maximum_cycles $maximum_cyclesligand_atom_file ${ligand_number}.mol2receptor_site_file $HOME/dock_inputs/${receptor_site_file}score_grid_prefix $HOME/dock_inputs/${score_grid_prefix}vdw_definition_file vdw.defnchemical_definition_file chem.defnchemical_score_file chem_score.tblflex_definition_file flex.defnflex_drive_file flex_drive.tblligand_contact_file dock_cnt.mol2ligand_chemical_file dock_chm.mol2ligand_energy_file dock_nrg.mol2

Parameterized Dock input file

Molecule to be Molecule to be screenedscreened

79

parameter database_name label "database_name" text select oneof "aldrich" "maybridge" "maybridge_300" "asinex_egc" "asinex_epc" "asinex_pre" "available_chemicals_directory" "inter_bioscreen_s" "inter_bioscreen_n" "inter_bioscreen_n_300" "inter_bioscreen_n_500" "biomolecular_research_institute" "molecular_science" "molecular_diversity_preservation" "national_cancer_institute" "IGF_HITS" "aldrich_300" "molecular_science_500" "APP" "ECE" default "aldrich_300";

parameter score_ligand text default "yes";parameter minimize_ligand text default "yes";parameter multiple_ligands text default "no";parameter random_seed integer default 7;parameter anchor_search text default "no";parameter torsion_drive text default "yes";parameter clash_overlap float default 0.5;parameter conformation_cutoff_factor integer default 5;parameter torsion_minimize text default "yes";parameter match_receptor_sites text default "no";parameter random_search text default "yes"; . . . . . . . . . . . .parameter maximum_cycles integer default 1;parameter receptor_site_file text default "ece.sph";parameter score_grid_prefix text default "ece";parameter ligand_number integer range from 1 to 200 step 1;

Dock PlanFile (contd.)

Molecules to be Molecules to be screenedscreened

80

task nodestart copy ./parameter/vdw.defn node:. copy ./parameter/chem.defn node:. copy ./parameter/chem_score.tbl node:. copy ./parameter/flex.defn node:. copy ./parameter/flex_drive.tbl node:. copy ./dock_inputs/get_molecule node:. copy ./dock_inputs/dock_base node:.endtasktask main node:substitute dock_base dock_run node:substitute get_molecule get_molecule_fetch node:execute sh ./get_molecule_fetch node:execute $HOME/bin/dock.$OS -i dock_run -o dock_out copy node:dock_out ./results/dock_out.$jobname copy node:dock_cnt.mol2 ./results/dock_cnt.mol2.$jobname copy node:dock_chm.mol2 ./results/dock_chm.mol2.$jobname copy node:dock_nrg.mol2 ./results/dock_nrg.mol2.$jobnameendtask

Dock PlanFile

81

Nimrod/TurboLinux enFuzion GUI tools for Parameter Modeling

82

Docking Experiment Preparation

Setup PDB DataGrid Index PDB databases Pre-stage (all) Protein Data Bank (PDB) on replica sites Start PDB Server

Create Docking GridScore (receptor surface details) for a given receptor on home node.

Pre-Staging Large Files required for Docking: Pre-stage Dock executables and PDB access client on Grid nodes, if

required (e.g., dock.Linux, dock.SunOS, dock.IRIX64, and dock.OSF1 on Linux, Sun, SGI, and Compaq machines respectively). Use globus-rcp.

Pre-stage/Cache all data files (~3-13MB each) representing receptor details on Grid nodes.

This can can be done demand by Nimrod/G for each job, but few input files are too large and they are required for all jobs). So, pre-staging/caching at http-cache or broker level is necessary to avoid the overhead of copying the same input files again and again!

83

Protein Data Bank

Databases consist of small molecules from commercially available organic synthesis libraries, and natural product databases.

There is also the ability to screen virtual combinatorial databases, in their entirety.

This methodology allows only the required compounds to be subjected to physical screening and/or synthesis reducing both time and expense.

84

Target Testcase

The target for the test case: electrocardiogram (ECE) endothelin converting enzyme. This is involved in “heart stroke” and other transient ischemia.

Is·che·mi·a : A decrease in the blood supply to a bodily organ, tissue, or part caused by constriction or obstruction of the blood vessels.

85

Nimrod/G in Action:Screening on World-Wide

Grid

86

Any Scientific Discovery ? Did your collaborator invent new drug for

xxxx?

Not Yet

Anyway, checkout the announcement of Nobel-

prize winners for next year

?

Conclude with a comparison with the Electrical

Grid………..

Where we are ????

Courtesy: Domenico Laforenza

Alessandro Volta in Paris in 1801 inside French National Institute shows the battery

while in the presence of Napoleon I

Fresco by N. Cianfanelli (1841) (Zoological Section "La Specula" of National History Museum of Florence

University)

89

….and in the future, I imagine a worldwidePower (Electrical) Grid …...

What ?!?!This is a mad man…

Oh, monDieu !

90

2001 - 1801 = 200 Years

91

” I think there is a world market for about five computers.”Thomas J. Watson Sr., IBM Founder, 1943

Can we Predict its Future ?

92

Summary and Conclusions P2P and Grid Computing is emerging as a next generation

computing platform for solving large scale problems through sharing of geographically distributed resources.

Resource management is a complex undertaking as systems need to be adaptive, scalable, competitive,…, and driven by QoS.

We proposed a framework based on “computational economies” and discussed several economic models for resource allocation and for regulating supply-and-demand for resources.

Scheduling experiments on World Wide Grid demonstrate our Nimrod-G broker ability to dynamically lease or rent services at runtime based on their quality, cost, and availability depending on consumers QoS requirements.

Economics paradigm for QoS driven resource management is essential to push P2P/Grids into mainstream computing!

93

Download Software & Information

Nimrod & Parameteric Computing: http://www.csse.monash.edu.au/~davida/nimrod/

Economy Grid & Nimrod/G: http://www.buyya.com/ecogrid/

Virtual Laboratory/Virtual Drug Design: http://www.buyya.com/vlab/

Grid Simulation (GridSim) Toolkit (Java based): http://www.buyya.com/gridsim/

World Wide Grid (WWG) testbed: http://www.buyya.com/ecogrid/wwg/ Looking for new volunteers to grow

Please contact me to barter your & our machines!

Want to build on our work/collaborate: Talk to me now or email: rajkumar@csse.monash.edu.au

94

Thank You… Any ??

Thank You… Any ??

95

Further Information

Books: High Performance Cluster Computing, V1,

V2, R.Buyya (Ed), Prentice Hall, 1999. The GRID, I. Foster and C. Kesselman (Eds),

Morgan-Kaufmann, 1999. IEEE Task Force on Cluster Computing

http://www.ieeetfcc.org Global Grid Forum

www.gridforum.org

IEEE/ACM CCGrid’xy: www.ccgrid.org CCGrid 2002, Berlin: ccgrid2002.zib.de

Grid workshop - www.gridcomputing.org

96

Further Information

Cluster Computing Info Centre: http://www.buyya.com/cluster/

Grid Computing Info Centre: http://www.gridcomputing.com

IEEE DS Online - Grid Computing area:

http://computer.org/dsonline/gc

Compute Power Market Project http://www.ComputePower.com