solving business problems with sas analytics and optmodel · gartner: magic quadrant for advanced...

Copyr i g ht © 2016, SAS Ins t i tu t e Inc . A l l r ights reser ve d .

SOLVING BUSINESS PROBLEMS

WITH SAS ANALYTICS AND OPTMODEL


SAS ANALYTICS

AND OPTMODELWHAT WE’LL COVER

• Background: SAS and SAS Analytics

• Introductory OPTMODEL Example: Political Redistricting

• Pharmaceutical Launch Revenue Optimization

• Product Formulation Optimization

• Pricing Optimization

• Technical Overview and Future Features


BACKGROUND: SAS AND SAS ANALYTICS

ED HUGHES


SAS ANALYTICS

AND OPTMODELSAS BACKGROUND

• SAS Institute Inc. founded in 1976

• SAS users, partners, and offices around the world

• Broad range of integrated analytic products

• Data access and integration

• Data mining

• Statistical analysis

• Forecasting and econometrics

• Text analytics

• Optimization, simulation, and scheduling


SAS ANALYTICS

AND OPTMODELOPTIMIZATION WITH SAS/OR

• OPTMODEL: algebraic optimization modeling language

• Linear Programming:

• Primal, dual, network simplex; interior point; concurrent

• Mixed Integer Linear Programming:

• Branch and cut: heuristics, conflict search, option tuning

• Decomposition algorithm (MILP and LP)

• Quadratic Programming:

• Interior point

• Nonlinear Programming:

• Active set; interior point; concurrent; multistart algorithm


SAS ANALYTICS

AND OPTMODELOPTIMIZATION WITH SAS/OR

• Network Algorithms

• 11 diagnostic and optimization algorithms

• Constraint Logic Programming

• Solve CSPs (general and scheduling)

• Local Search

• Hybrid parallel approach; multiple algorithms


SAS ANALYTICS

AND OPTMODELSAMPLE CUSTOMER PROJECTS: OPTIMIZATION

• Television advertising optimization (network)

• Chemical mixture optimization (Procter & Gamble)

• Optimal ATM replenishment (DBS Singapore)

• Public water management (national utilities agency)

• Optimal loan assignment (bank)

• Pharmaceutical launch revenue optimization


SAS ANALYTICS

AND OPTMODELRECENT CUSTOMER PROJECTS: SIMULATION

• Hospital room/bed assignment

• Neonatal intensive care unit (Duke Children’s Hospital)

• Prison population planning (NC Sentencing Commission)

• Digital parts distribution (parts distributor/retailer)


SAS ANALYTICS

AND OPTMODELOPTIMIZATION-DRIVEN SOLUTIONS

• SAS Marketing Optimization

• SAS Size Optimization

• SAS Service Parts

Optimization

• SAS for Service Operations

Optimization

• SAS Risk Management for

Banking, Insurance

• SAS Markdown Optimization

• SAS Credit Scoring for Banking

• SAS Customer Link Analytics

• SAS Fraud Framework

• SAS Inventory Optimization

Workbench

• SAS Launch Revenue

Optimization


IDCWORLDWIDE ADVANCED AND PREDICTIVE ANALYTICS

SOFTWARE 2014 VENDOR SHARES

Source: IDC, Worldwide Advanced and Predictive Analytics Software Market Shares, 2014: The Rise of the Long Tail (Doc # 257344). , July 2015.


GARTNER: MAGIC QUADRANT FOR ADVANCED ANALYTIC PLATFORMS

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from SAS. Gartner does not endorse any vendor, product or service depicted in its research publications, and

does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with

respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Gartner Magic Quadrant for Advanced Analytics Platforms by Lisa Kart, Gareth Herschel, Alexander Linden, Jim Hare 9 February 2016.


INTRODUCTORY OPTMODEL EXAMPLE:

POLITICAL REDISTRICTING

ROB PRATT


GRAPH

PARTITIONING PROBLEM DESCRIPTION

• Given undirected graph, node weights, positive integer 𝑘, target 𝑡 ∈ [ℓ, 𝑢]

• Partition node set into 𝑘 subsets, each with total weight within [ℓ, 𝑢]

• Usual objective: minimize sum of edge weights between parts of partition

• Our objective: find most “balanced” partition

• Minimize total absolute deviation of subset weight from target (𝐿1 norm)

• Minimize maximum absolute deviation of subset weight from target (𝐿∞ norm)

• Connectivity constraints: each of the 𝑘 resulting induced subgraphs must be

connected


GRAPH

PARTITIONINGMOTIVATION

• Dennis E. Shasha, Doctor Ecco’s Cyberpuzzles (2002)

• Political districting

• Advanced Analytics and Optimization Services consulting engagement

• Assigning service workers to customer regions

• INFORMS OR Exchange posting (Jan. 2013)

www.or-exchange.com/questions/7248/graph-partitioning-with-connectivity-constraints

• Forestry

http://www.or-exchange.com/questions/7248/graph-partitioning-with-connectivity-constraints


GRAPH

PARTITIONING MAP FROM SHASHA (2002)


GRAPH

PARTITIONING MAP REPRESENTED AS GRAPH


GRAPH

PARTITIONING NODE WEIGHTS (POPULATIONS)


GRAPH

PARTITIONING

FEASIBLE SOLUTION FOR 14 CLUSTERS,

BOUNDS 12000 ± 100


GRAPH

PARTITIONING SOLUTION APPROACHES

• Compact MILP formulation with binary variables, nonnegative flow variables,

flow balance constraints

• Row generation, dynamically excluding disconnected subsets

• Column generation, dynamically generating connected subsets

• Column generation, statically generating all possible connected subsets


GRAPH

PARTITIONING PROC OPTMODEL CODE: MILP SOLVER

Master MILP problem: Given all feasible subsets, solve cardinality constrained set partitioning problem

1. /* master problem: partition nodes into connected subsets, minimizing deviation from target */

2. /* UseSubset[s] = 1 if subset s is used, 0 otherwise */

3. var UseSubset {SUBSETS} binary;

4.

5. /* L_1 norm */

6. num subset_population {s in SUBSETS} = sum {i in NODES_s[s]} p[i];

7. num subset_deviation {s in SUBSETS} = abs(subset_population[s] - target);

8. min Objective1 = sum {s in SUBSETS} subset_deviation[s] * UseSubset[s];

9.

10. /* each node belongs to exactly one subset */

11. con Partition {i in NODES}:

12. sum {s in SUBSETS: i in NODES_s[s]} UseSubset[s] = 1;

13.

14. /* use exactly the allowed number of subsets */

15. con Cardinality:

16. sum {s in SUBSETS} UseSubset[s] = num_clusters;

17.

18. problem Master include

19. UseSubset Objective1 Partition Cardinality;


GRAPH

PARTITIONING STATIC COLUMN GENERATION

1. Generate all possible connected subsets with weights close enough to

target

• Relax connectivity and find all possible subsets

• Postprocess to filter out disconnected subsets

2. Solve cardinality constrained set partitioning master problem


GRAPH

PARTITIONING FEATURES USED IN PROC OPTMODEL

• MILP solver to compute maximum subset size for each node (COFOR loop)

• Network solver (all-pairs shortest path) to generate valid cuts that encourage

connectivity

• Constraint programming solver to generate subsets (multiple solutions)

• Network solver (connected components) to check connectivity (COFOR loop)

• MILP solver to solve set partitioning problem


GRAPH

PARTITIONING COLUMN GENERATION SUBPROBLEM

• Given node populations 𝑝𝑖 and target population range [ℓ, 𝑢]

• Find all connected node subsets close to target population

• Binary variable 𝑥𝑖 indicates whether node 𝑖 appears in subset

• Range constraint ℓ ≤ 𝑖∈𝑁 𝑝𝑖𝑥𝑖 ≤ 𝑢

• Connectivity hard to impose directly, so relax (but do not ignore)


GRAPH

PARTITIONING

NODE SUBSET 𝟏, 𝟐, 𝟑, 𝟒, 𝟓 SATISFIES POPULATION

BOUNDS, BUT IS DISCONNECTED


GRAPH

PARTITIONING VALID CUTS FOR NODES TOO FAR APART

• 𝑚𝑖: max # nodes among

feasible subsets that

contain node 𝑖

• If shortest path distance

𝑑𝑖𝑗 ≥ min 𝑚𝑖 , 𝑚𝑗 then

𝑥𝑖 + 𝑥𝑗 ≤ 1


GRAPH


1. /* maxnodes[i] = max number of nodes among feasible subsets that contain node i */

2. num maxnodes {NODES};

3.

4. /* UseNode[i] = 1 if node i is used, 0 otherwise */

5. var UseNode {NODES} binary;

6.

7. max NumUsed = sum {i in NODES} UseNode[i];

8.

9. con PopulationBounds:

10. target - deviation_ub <= sum {i in NODES} p[i] * UseNode[i] <= target + deviation_ub;

11.

12. problem MaxNumUsed include

13. UseNode NumUsed PopulationBounds;

14. use problem MaxNumUsed;

15.

16. put ’Finding max number of nodes among feasible subsets that contain node i...’;

17. cofor {i in NODES} do;

18. put i=;

19. fix UseNode[i] = 1;

20. solve with MILP / loglevel=0 logfreq=0;

21. maxnodes[i] = round(NumUsed.sol);

22. unfix UseNode;

23. end;


GRAPH

PARTITIONING PROC OPTMODEL CODE: NETWORK SOLVER

1. /* distance[i,j] = shortest path distance (number of edges) from i to j */

2. num distance {NODES, NODES};

3. /* num one {EDGES} = 1; */

4. put ’Finding all-pairs shortest path distances...’;

5. solve with network /

6. links = (include=EDGES /*weight=one*/)

7. shortpath

8. out = (spweights=distance);


GRAPH

PARTITIONING VALID CUTS FOR SMALL-POPULATION NODES

• If 𝑥𝑖 = 1 and 𝑝𝑖 < ℓ

then 𝑥𝑗 = 1 for some

neighbor 𝑗 of 𝑖

• 𝑥𝑖 ≤ 𝑗𝜖𝑁𝑖 𝑥𝑗


GRAPH

PARTITIONING VALID CUTS FOR SMALL-POPULATION SETS

• More generally, if 𝑥𝑖 = 1 for some 𝑖 ∈ 𝑆 ⊂ 𝑁 such that 𝑗∈𝑆 𝑝𝑗 < ℓ then 𝑥𝑘 = 1

for some neighbor 𝑘 in moat around 𝑆

• 𝑥𝑖 ≤ 𝑘𝜖( 𝑗𝑁𝑗)∖𝑆 𝑥𝑘

• Known as “ring inequalities” in the literature


GRAPH

PARTITIONING

PROC OPTMODEL CODE: CONSTRAINT PROGRAMMING

SOLVER1. set NODE_PAIRS = {i in NODES, j in NODES: i < j};

2.

3. /* cannot have two nodes i and j that are too far apart */

4. con Conflict {<i,j> in NODE_PAIRS: distance[i,j] >= min(maxnodes[i],maxnodes[j])}:

5. UseNode[i] + UseNode[j] <= 1;

6.

7. /* if UseNode[i] = 1 and p[i] < target - deviation_ub

8. then UseNode[j] = 1 for some neighbor j of i */

9. con Moat1 {i in NODES: p[i] < target - deviation_ub}:

10. UseNode[i] <= sum {j in NEIGHBORS[i]} UseNode[j];

11.

12. /* if (UseNode[i] = 1 or UseNode[j] = 1) and p[i] + p[j] < target - deviation_ub

13. then UseNode[k] = 1 for some neighbor k in moat around {i,j} */

14. con Moat2_i {<i,j> in NODE_PAIRS: p[i] + p[j] < target - deviation_ub}:

15. UseNode[i] <= sum {k in (NEIGHBORS[i] union NEIGHBORS[j]) diff {i,j}} UseNode[k];

16. con Moat2_j {<i,j> in NODE_PAIRS: p[i] + p[j] < target - deviation_ub}:

17. UseNode[j] <= sum {k in (NEIGHBORS[i] union NEIGHBORS[j]) diff {i,j}} UseNode[k];

18.

19. problem Subproblem include

20. UseNode PopulationBounds Conflict Moat1 Moat2_i Moat2_j;

21. use problem Subproblem;

22.

23. put ’Solving CLP subproblem...’;

24. solve with CLP / findallsolns;

25. /* use _NSOL_ and .sol[s] to access multiple solutions */

26. SUBSETS = 1.._NSOL_;

27. for {s in SUBSETS} NODES_s[s] = {i in NODES: UseNode[i].sol[s] > 0.5};


GRAPH

PARTITIONING CONSTRAINT PROGRAMMING SOLVER LOG

NOTE: Problem generation will use 4 threads.

NOTE: The problem has 66 variables (0 free, 0 fixed).

NOTE: The problem has 66 binary and 0 integer variables.

NOTE: The problem has 5123 linear constraints (5122 LE, 0 EQ, 0 GE, 1 range).

NOTE: The problem has 42926 linear constraint coefficients.

NOTE: The problem has 0 nonlinear constraints (0 LE, 0 EQ, 0 GE, 0 range).

NOTE: All possible solutions have been found.

NOTE: Number of solutions found = 4472.

CARD(SUBSETS)=4472


GRAPH

PARTITIONING PROC OPTMODEL CODE: NETWORK SOLVER

1. /* check connectivity */

2. set NODES_THIS;

3. num component {NODES_THIS};

4. set DISCONNECTED init {};

5. cofor {s in SUBSETS} do;

6. put s=;

7. NODES_THIS = NODES_s[s];

8. solve with network /

9. links = (include=EDGES)

10. subgraph = (nodes=NODES_THIS)

11. concomp

12. out = (concomp=component);

13. if or {i in NODES_THIS} (component[i] > 1)

14. then DISCONNECTED = DISCONNECTED union {s};

15. end;

16. SUBSETS = SUBSETS diff DISCONNECTED;


GRAPH

PARTITIONING NETWORK SOLVER LOG

s=1

NOTE: The SUBGRAPH= option filtered 145 elements from ’EDGES.’

NOTE: The number of nodes in the input graph is 4.

NOTE: The number of links in the input graph is 3.

NOTE: Processing connected components.

NOTE: The graph has 1 connected component.

NOTE: Processing connected components used 0.00 (cpu: 0.00) seconds.

...

s=28





NOTE: The graph has 2 connected components.


...

s=4472





NOTE: The graph has 1 connected component.


CARD(SUBSETS)=2771


GRAPH


1. /* master problem: partition nodes into connected subsets, minimizing deviation from target */

2. /* UseSubset[s] = 1 if subset s is used, 0 otherwise */

3. var UseSubset {SUBSETS} binary;

4.

5. /* L_1 norm */

6. num subset_population {s in SUBSETS} = sum {i in NODES_s[s]} p[i];

7. num subset_deviation {s in SUBSETS} = abs(subset_population[s] - target);

8. min Objective1 = sum {s in SUBSETS} subset_deviation[s] * UseSubset[s];

9.

10. /* each node belongs to exactly one subset */

11. con Partition {i in NODES}:

12. sum {s in SUBSETS: i in NODES_s[s]} UseSubset[s] = 1;

13.

14. /* use exactly the allowed number of subsets */

15. con Cardinality:

16. sum {s in SUBSETS} UseSubset[s] = num_clusters;

17.

18. problem Master include

19. UseSubset Objective1 Partition Cardinality;

20.

21. ...

22.

23. use problem Master;

24. put ’Solving master problem, minimizing Objective1...’;

25. solve;


GRAPH

PARTITIONING MILP SOLVER LOG







NOTE: The OPTMODEL presolver is disabled for linear problems.

NOTE: The MILP presolver value AUTOMATIC is applied.

NOTE: The MILP presolver removed 1092 variables and 0 constraints.

NOTE: The MILP presolver removed 7089 constraint coefficients.

NOTE: The MILP presolver modified 0 constraint coefficients.

NOTE: The presolved problem has 1679 variables, 67 constraints, and 10245 constraint coefficients.

NOTE: The MILP solver is called.

NOTE: The parallel Branch and Cut algorithm is used.

NOTE: The Branch and Cut algorithm is using up to 4 threads.

Node Active Sols BestInteger BestBound Gap Time

0 1 0 . 96.3214286 . 1

...

0 1 0 . 113.1320755 . 1

NOTE: The MILP solver added 40 cuts with 2752 cut coefficients at the root.

3 0 2 124.0000000 124.0000000 0.00% 3

NOTE: Optimal.

NOTE: Objective = 124.


GRAPH

PARTITIONING

SOLUTION FOR 14 CLUSTERS, BOUNDS 12000 ± 100,

OBJECTIVE1


GRAPH


1. /* L_infinity norm */

2. var MaxDeviation >= 0;

3. min Objective2 = MaxDeviation;

4. con MaxDeviationDef {s in SUBSETS}:

5. MaxDeviation >= subset_deviation[s] * UseSubset[s];

6.

7. put ’Solving master problem, minimizing Objective2...’;

8. MaxDeviation = max {s in SUBSETS} (subset_deviation[s] * UseSubset[s].sol);

9. solve with MILP / primalin;


GRAPH

PARTITIONING MILP SOLVER LOG







NOTE: The MILP presolver value AUTOMATIC is applied.

NOTE: The MILP presolver removed 1092 variables and 2733 constraints.

NOTE: The MILP presolver removed 10901 constraint coefficients.

NOTE: The MILP presolver added 1798 constraint coefficients.

NOTE: The MILP presolver modified 329 constraint coefficients.

NOTE: The presolved problem has 1680 variables, 105 constraints, and 11944 constraint coefficients.

NOTE: The MILP solver is called.

NOTE: The parallel Branch and Cut algorithm is used.

NOTE: The Branch and Cut algorithm is using up to 4 threads.

Node Active Sols BestInteger BestBound Gap Time

0 1 1 37.0000000 17.1147541 116.19% 2

NOTE: The MILP presolver is applied again.

0 1 1 37.0000000 29.0000000 27.59% 3

0 0 2 29.0000000 29.0000000 0.00% 3

NOTE: Optimal.

NOTE: Objective = 29.


GRAPH

PARTITIONING

SOLUTION FOR 14 CLUSTERS, BOUNDS 12000 ± 100,

OBJECTIVE2


GRAPH

PARTITIONINGPAPER

SAS Global Forum 2015 paper:

Using the OPTMODEL Procedure in SAS/OR to Solve Complex Problems

http://support.sas.com/resources/papers/proceedings15/SAS1502-2015.pdf


PHARMACEUTICAL LAUNCH REVENUE

OPTIMIZATION

SCOTT SHULER


PRODUCT FORMULATION OPTIMIZATION

IVAN OLIVEIRA


PRICING OPTIMIZATION

SCOTT SHULER


TECHNICAL OVERVIEW AND FUTURE FEATURES

PATRICK KOCH


OPTIMIZATION IN

MACHINE LEARNINGOVERVIEW

1. Training Predictive Models

• Optimization problem

• Parameters – opt and train

2. Tuning Predictive Models

• Manual, grid search, optimization

• Optimization challenges

3. Distributed / parallel processing

4. Sample results


OPTIMIZATION IN

MACHINE LEARNINGPREDICTIVE MODELS

Transformation of relevant data to high quality descriptive and predictive models

Ex.: Neural Network

• How to determine weights, activation

functions?

• How to determine model

configuration?…

.

….

….

Input layer Hidden layer Output layer

x1

x2

x3

x4

xn

wij

wjk

f1(x)

f2(x)

fm(x)


OPTIMIZATION IN

MACHINE LEARNINGMODEL TRAINING – OPTIMIZATION PROBLEM

• Objective Function

• 𝑓 𝑤 =1

𝑛 𝑖=1𝑛 𝐿 𝑤; 𝑥𝑖 , 𝑦𝑖 + 𝜆1 𝑤 1 +

𝜆2

2𝑤 2

2

• Loss, 𝐿(𝑤; 𝑥𝑖 , 𝑦𝑖), for observation (𝑥𝑖 , 𝑦𝑖) and weights, 𝑤

• Stochastic Gradient Descent (SGD)

• Variation of Gradient Descent 𝑤𝑘+1 = 𝑤𝑘 − 𝜂𝛻𝑓(𝑤𝑘)

• Replaces 𝛻𝑓 𝑤𝑘 with mini-batch gradient

• 𝛻𝑓𝑡(𝑤) =1

𝑚 𝑖=1𝑚 𝐿(𝑤; 𝑥𝑖 , 𝑦𝑖)

• Several tricks to improve performance…


OPTIMIZATION IN

MACHINE LEARNINGMODEL TRAINING – OPTIMIZATION PARAMETERS (SGD)

• Learning rate, η

• Too high, diverges

• Too low, slow performance

• Momentum, μ

• Too high, could “pass” solution

• Too low, no performance improvement

• Regularization parameters, λ1 and λ2

• Too low, has little effect

• Too high, drives iterates to 0

• Other parameters

• Adaptive decay rate, β

• Mini-batch size, m

• Communication Frequency, c

• Annealing Rate, r


OPTIMIZATION IN

MACHINE LEARNINGHYPERPARAMETERS

• Quality of trained model governed by so-called ‘hyperparameters’

No clear defaults agreeable to a wide range of applications

• Optimization options (SGD)

• Neural Net training options

• Number of hidden layers

• Number of neurons in each hidden layer

• Random distribution for initial

connection weights (normal, uniform, Cauchy)

• Error function (gamma, normal, Poisson, entropy)

….

….

….

Input layer Hidden layer Output layer

x1

x2

x3

x4

xn

wij

wjk

f1(x)

f2(x)

fm(x)


OPTIMIZATION IN

MACHINE LEARNINGMODEL TUNING

• Traditional Approach: manual tuning

Even with expertise in machine learning algorithms and their parameters, best settings

are directly dependent on the data used in training and scoring

• Hyperparameter Optimization: grid vs. random vs. optimization

x1

x2

Standard Grid Search

x1

x2

Random Latin Hypercube

x1

x2

Random Search


OPTIMIZATION IN

MACHINE LEARNINGHYPERPARAMETER OPTIMIZATION CHALLENGES

x

Objective

blows up

Categorical / Integer

Variables

T(x)

Noisy/Nondeterministic

Flat-regions.

Node failure

Tuning objective, T(x), is validation error score (to avoid increased overfitting effect)


OPTIMIZATION IN

MACHINE LEARNINGPARALLEL HYBRID DERIVATIVE-FREE OPTIMIZATION STRATEGY

Default hybrid search strategy:

1. Initial Search: Latin Hypercube Sampling (LHS)

2. Global search: Genetic Algorithm (GA)

• Supports integer, categorical variables

• Handles nonsmooth, discontinuous space

3. Local search: Generating Set Search (GSS)

• Similar to Pattern Search

• First-order convergence properties

• Developed for continuous variables

All easy to parallelize


OPTIMIZATION IN

MACHINE LEARNINGDATA PARALLEL & MODEL PARALLEL

0

2

4

6

8

10

12

2 4 8 16 32

Tim

e (

hours

)

Maximum Models Trained in Parallel

Handwritten Data Train 60k/Validate 10k

0

1

2

3

4

5

6

7

8

9

10

1 2 4 8 16 32 64

TIm

e (

min

ute

s)

Number of Nodes for Training

Credit Data49k Train / 21k Validate

“Data Parallel”

Distributed Data,

Sequential Tuning

“Data Parallel & Model Parallel”

Distributed Data,

Parallel Tuning


OPTIMIZATION IN

MACHINE LEARNINGSAMPLE HYPERPARAMETER OPTIMIZATION RESULTS

• Default train, 9.6 % error

• Iteration 1 – Latin Hypercube best,

6.22% error

• Very similar configuration can have

very different error…

• Best after tuning: 2.61% error

0

20

40

60

80

100

0 50 100 150

Err

or

%

Evaluation

Iteration 1: Latin Hypercube Sample Error

0

2

4

6

8

10

12

0 5 10 15 20

Obje

ctive (

Mis

cla

ssific

ation E

rror

%)

Iteration

Handwritten Data Train 60k/Validate 10k


OPTIMIZATION IN

MACHINE LEARNINGMANY TUNING OPPORTUNITIES

• Other Machine Learning algorithms

• SVM - Support Vector Machines

• Decision Trees

• Random Forest

• Gradient Boosting Trees…

• Other search methods, extending hybrid solver framework

• Bayesian Optimization

• Surrogate-based Optimization

Better (generalized) Models = Better Predictions = Better Decisions


THANK YOU!

solving business problems with sas analytics and optmodel · gartner: magic quadrant for advanced...

Documents