1
Rapid Application Development & Grid Computing Using GAMS
Software DemonstrationINFORMS San Francisco 2005
2
Demo Objectives
Basics / Algebraic ModelingData Exchange / ChartingTying things together (procedural elements)Grid ComputingModel Deployment
3
Company Profile
General Algebraic Modeling System Started as a research projectat the World Bank 1976Went commercial in 1987Offices in Washington, D.C and CologneProfessional software tool providerOperating in a segmented niche marketBroad academic and commercial user base
4
Modeling Systems
Describe problems to a computer system in the same way that people describe problems to each other.Simplify the model building and solution processCreate maintainable modelsAdapt models quickly to new situations
5
Basic Technical Principles
Separation of model and dataSeparation of model and solution methodsComputing platform independenceMultiple model types, solvers, platformsBalanced mix of declarative and procedural approachesModel is a data base operator and/or object
6
Basic Technical Principles II
Open architecture and interfaces to other systems:
GUIExcel, DatabasesProgramming Languages etc.
Maintainable models and protection of investments
7
Multiple Model Types
LP - Linear ProgrammingMIP - Mixed Integer ProgrammingQCP - Quadratically Constrained ProgrammingNLP - Nonlinear ProgrammingCNS – Constrained Nonlinear SystemsMINLP - Mixed Integer Nonlinear ProgrammingMPEC - NLP with Complementarity ConstraintsMPSGE - General Equilibrium ModelsStochastic Optimization
8
Multiple Solvers & Platforms
9
Basic Portfolio Selection
MarkowitzClosing Values of Stocks over TimeCalculate Return, Deviation, Co-variance using Closing ValuesSelect Portfolio to
Minimize Variances.t. Some Return GoalMinimum Portfolio Size per Stock
10
Basic Portfolio Selection Model QP1
11
Data from Excel QP2
12
Gams Data eXchange
Separation of Responsibility for Data and Model (Data contract)Gams Data eXchange (GDX):
A priori validation of data contractComplements the ASCII text data input
Application GDX GAMS
13
GDX Tools
GDX
gdxxrw(MS Office)
gams
IDE gdxsql
gdxdiff
gdxdump
gdxmerge
gdxsplitGDXAPI
14
Reporting
Integrated Standard Reporting Facilitiestext filesGDX: MS Office, DB, Charting, …
Specialized ReportingScenario Management/Data Cube: VEDAGeographical Information System: MapInfoVisualization with MATLAB
15
16
GAMS/MapInfo
Less Than 0
0 - 199
200-1000
1000-3000
Increase in Ktons Per Year
17
GAMS/MATLAB
18
Procedural Elements
Contrast pure declarative approach:e.g. AMPL, MPL, OPL
Scripting added outside languageAMPL Script, OPL scriptProgramming Languages API (OptiMax)
Experience shows importance of procedural paradigmPart of the GAMS Language
19
Procedural Elements QP3
20
What is Grid Computing?
A pool of connected computers managed and available as a common computing resource
Allows parallel task executionAllows effective sharing of CPU powerFor us, not necessarily distributedScheduler handles management tasksCan be rented or owned in commonE.g. Condor, Sun Grid Engine, Globus
21
Economics of Grid Computing
Yearly cost, 2-CPU workstation: $5200Hardware - $1200Software - $4000
Hourly cost on the grid: $2$1/hour for CPU time (to grid operator)$1/hour for software (GAMS, model owner)
1 workstation == 50 hrs/week grid timeUp-front vs. deferred, as-needed costs
22
Energy planning problems
Given expected energy demands, costs, and supplies, one can quickly form an optimal plan.Uncertain data many possible scenariosPlanning time increases dramatically
30 sec/scen * 4K scen = 33 hoursLimits the number of scenarios allowed
23
Solution A: Optimize for Speed
Modify model to batch scenariosEliminate generation time
GAMS generates each model from scratchFraction of total time used is non-trivialRequires significant programmingIncreased chance of errorNot maintainable/flexible/layered
Speedup: 1.5 (at most!), or 22 hours
24
Solution B: Use a Grid
Solve the scenarios in parallel, e.g.100 CPUs: 20 minutes
Marginal cost is $66No programming requiredModel stays maintainableSeparation of model and solution maintained
25
Grid Specifics
echo "#!/bin/bash" > ${3}runit.shecho $1 $2 >> ${3}runit.shecho gmscr_ux.out $2 >> ${3}runit.shecho mkdir ${3}finished >> ${3}runit.shchmod 750 ${3}runit.sh${3}runit.sh > /dev/null &
26
Parallel Submission QP4
27
Massively Parallel MIP
MIP/B&C Algorithm ideal to parallelizeSeymour problem solved that waySoftware: FATCOP/Condor, BCP/PVMA priori subdivision in n independent problems
Open Pit Mining Problem (openpit in GAMS Model library)
Used integer variables to subdivide Model into into 4096 sub-problemsExperiments (Ferris) at UW using Condor Pool
28
Results
Submission start Nov. 9 at 6 pmAll job submitted by Nov. 9 at 7:30 pmAll jobs returned by Nov 10, 10:15 am
16 hours wall time, 1812 CPU hoursPeak number of CPUs: 270
Different Instance:24 hours wall time, 3000 CPU hours
29
Condor Statistics
30
Other Examples
Student-centric scheduling at USMA (West Point)
Models used intensively 4 times/year4000 independent MIP models
Hill & Associates (energy/coal experts)National Power Model (coal & electricity)Multi-period (20 yr) model: runs 10 hours Tens of runs & quick turnaround required
31
Model Deployment
All options are availableSpreadsheets, databasesWeb servicesGUI (IDE, custom built)On the Grid
Independent of model or data sourcesBasic philosophy: layers of separation
32
Fully Embedded Application
Server
GAMS
WEB CLIENT
JAVA ServletSpawns GAMS
Communication via HTML/JAVA
Servlets
Web Front End
SERVER
Access / Excel Database
Return Solution
Graph Solution as HTML (JAVA
Servlets)
BACK END APPLICATION
33
Web Front End
User may specify:
• Expected Return
• Investment Period (Days)
• Number of Stocks
• Solver
• Database (Excel or Access)
34
Output
Output:
• User input parameters
• Real-time GAMS output of model solve
35
Output (continued)
Output:
• GAMS Return Codes
• Investment distribution
•Graphical output