rapid application development & grid computing …...nlp - nonlinear programming cns –...

1

Rapid Application Development & Grid Computing Using GAMS

Software DemonstrationINFORMS San Francisco 2005

2

Demo Objectives

Basics / Algebraic ModelingData Exchange / ChartingTying things together (procedural elements)Grid ComputingModel Deployment

3

Company Profile

General Algebraic Modeling System Started as a research projectat the World Bank 1976Went commercial in 1987Offices in Washington, D.C and CologneProfessional software tool providerOperating in a segmented niche marketBroad academic and commercial user base

4

Modeling Systems

Describe problems to a computer system in the same way that people describe problems to each other.Simplify the model building and solution processCreate maintainable modelsAdapt models quickly to new situations

5

Basic Technical Principles

Separation of model and dataSeparation of model and solution methodsComputing platform independenceMultiple model types, solvers, platformsBalanced mix of declarative and procedural approachesModel is a data base operator and/or object

6

Basic Technical Principles II

Open architecture and interfaces to other systems:

GUIExcel, DatabasesProgramming Languages etc.

Maintainable models and protection of investments

7

Multiple Model Types

LP - Linear ProgrammingMIP - Mixed Integer ProgrammingQCP - Quadratically Constrained ProgrammingNLP - Nonlinear ProgrammingCNS – Constrained Nonlinear SystemsMINLP - Mixed Integer Nonlinear ProgrammingMPEC - NLP with Complementarity ConstraintsMPSGE - General Equilibrium ModelsStochastic Optimization

8

Multiple Solvers & Platforms

9

Basic Portfolio Selection

MarkowitzClosing Values of Stocks over TimeCalculate Return, Deviation, Co-variance using Closing ValuesSelect Portfolio to

Minimize Variances.t. Some Return GoalMinimum Portfolio Size per Stock

10

Basic Portfolio Selection Model QP1

11

Data from Excel QP2

12

Gams Data eXchange

Separation of Responsibility for Data and Model (Data contract)Gams Data eXchange (GDX):

A priori validation of data contractComplements the ASCII text data input

Application GDX GAMS

13

GDX Tools

GDX

gdxxrw(MS Office)

gams

IDE gdxsql

gdxdiff

gdxdump

gdxmerge

gdxsplitGDXAPI

14

Reporting

Integrated Standard Reporting Facilitiestext filesGDX: MS Office, DB, Charting, …

Specialized ReportingScenario Management/Data Cube: VEDAGeographical Information System: MapInfoVisualization with MATLAB

15

16

GAMS/MapInfo

Less Than 0

0 - 199

200-1000

1000-3000

Increase in Ktons Per Year

17

GAMS/MATLAB

18

Procedural Elements

Contrast pure declarative approach:e.g. AMPL, MPL, OPL

Scripting added outside languageAMPL Script, OPL scriptProgramming Languages API (OptiMax)

Experience shows importance of procedural paradigmPart of the GAMS Language

19

Procedural Elements QP3

20

What is Grid Computing?

A pool of connected computers managed and available as a common computing resource

Allows parallel task executionAllows effective sharing of CPU powerFor us, not necessarily distributedScheduler handles management tasksCan be rented or owned in commonE.g. Condor, Sun Grid Engine, Globus

21

Economics of Grid Computing

Yearly cost, 2-CPU workstation: $5200Hardware - $1200Software - $4000

Hourly cost on the grid: $2$1/hour for CPU time (to grid operator)$1/hour for software (GAMS, model owner)

1 workstation == 50 hrs/week grid timeUp-front vs. deferred, as-needed costs

22

Energy planning problems

Given expected energy demands, costs, and supplies, one can quickly form an optimal plan.Uncertain data many possible scenariosPlanning time increases dramatically

30 sec/scen * 4K scen = 33 hoursLimits the number of scenarios allowed

23

Solution A: Optimize for Speed

Modify model to batch scenariosEliminate generation time

GAMS generates each model from scratchFraction of total time used is non-trivialRequires significant programmingIncreased chance of errorNot maintainable/flexible/layered

Speedup: 1.5 (at most!), or 22 hours

24

Solution B: Use a Grid

Solve the scenarios in parallel, e.g.100 CPUs: 20 minutes

Marginal cost is $66No programming requiredModel stays maintainableSeparation of model and solution maintained

25

Grid Specifics

echo "#!/bin/bash" > ${3}runit.shecho $1 $2 >> ${3}runit.shecho gmscr_ux.out $2 >> ${3}runit.shecho mkdir ${3}finished >> ${3}runit.shchmod 750 ${3}runit.sh${3}runit.sh > /dev/null &

26

Parallel Submission QP4

27

Massively Parallel MIP

MIP/B&C Algorithm ideal to parallelizeSeymour problem solved that waySoftware: FATCOP/Condor, BCP/PVMA priori subdivision in n independent problems

Open Pit Mining Problem (openpit in GAMS Model library)

Used integer variables to subdivide Model into into 4096 sub-problemsExperiments (Ferris) at UW using Condor Pool

28

Results

Submission start Nov. 9 at 6 pmAll job submitted by Nov. 9 at 7:30 pmAll jobs returned by Nov 10, 10:15 am

16 hours wall time, 1812 CPU hoursPeak number of CPUs: 270

Different Instance:24 hours wall time, 3000 CPU hours

29

Condor Statistics

30

Other Examples

Student-centric scheduling at USMA (West Point)

Models used intensively 4 times/year4000 independent MIP models

Hill & Associates (energy/coal experts)National Power Model (coal & electricity)Multi-period (20 yr) model: runs 10 hours Tens of runs & quick turnaround required

31

Model Deployment

All options are availableSpreadsheets, databasesWeb servicesGUI (IDE, custom built)On the Grid

Independent of model or data sourcesBasic philosophy: layers of separation

32

Fully Embedded Application

Server

GAMS

WEB CLIENT

JAVA ServletSpawns GAMS

Communication via HTML/JAVA

Servlets

Web Front End

SERVER

Access / Excel Database

Return Solution

Graph Solution as HTML (JAVA

Servlets)

BACK END APPLICATION

33

Web Front End

User may specify:

• Expected Return

• Investment Period (Days)

• Number of Stocks

• Solver

• Database (Excel or Access)

34

Output

Output:

• User input parameters

• Real-time GAMS output of model solve

35

Output (continued)

Output:

• GAMS Return Codes

• Investment distribution

•Graphical output