debasis samanta - iitkgp

Solving Optimization Problems

Debasis Samanta

IIT Kharagpur

[email protected]

06.03.2018

Debasis Samanta (IIT Kharagpur) Soft Computing Applications (IT60108) 06.03.2018 1 / 22

Introduction to Solving Optimization Problems

Today’s Topics

Concept of optimization problem

Defining an optimization problem

Various types of optimization problems

Traditional approaches to solve optimization problems

Limitations of the traditional approaches


Concept of optimization problem

Optimization : Optimum value that is either minimum or maximumvalue.

y = F (x)

Example:

2x − 6y = 11

ory = (2x − 11)÷ 6

Can we determine an optimum value for y?Similarly, in the following case

3x + 4y ≥ 56.

These are really not related to optimization problem!



Suppose, we are to design an optimal pointer made of some material withdensity ρ. The pointer should be as large as possible, no mechanicalbreakage and deflection of pointing at end should be negligible.

The task is to select the best pointer out of many all possible pointers.

Diameter d

Length l

Suppose, s is the strength of the pointer.

Mass of the stick is denoted by

M = 13Π

(d2

)2 ∗ l ∗ ρ = 112Π ∗ d2 ∗ l ∗ ρ

Deflection : δ = f1 (d , l , ρ)

Strength : s = f2(d , l , ρ)Debasis Samanta (IIT Kharagpur) Soft Computing Applications (IT60108) 06.03.2018 4 / 22


The problem can be stated as

Objective function

Minimize M = 112Π ∗ d2 ∗ l ∗ ρ

Subject to

δ ≤ δth, where δth = allowable deflections ≥ sth, where sth = required strength

and

dmin ≤ d ≤ dmax

lmin ≤ l ≤ lmax


Defining Optimization Problem

An optimization problem can be formally defined as follows:

Maximize (or Minimize)

yi = fi (x1, x2, · · · , xn)where i = 1, 2 · · · k, k ≥ 1

Subject to

gi (x1, x2, · · · xn) ROPi ci

where i = 1, 2, ..., j , j ≥ 0. ROPi denotes some relational operatorand ci = 1, 2, · · · j are some constants.andxi ROP di , for all i=1,2...n (n ≥ 1)Here, xi denotes a design parameter and di is some constant.


Some Benchmark Optimization Problems

Exercises: Mathematically define the following optimization problems.

Traveling Salesman Problem

Knapsack Problem

Graph Coloring Problem

Job Machine Assignment Problem

Coin Change Problem

Binary search tree construction problem


Types of Optimization Problem

Unconstrained optimization problemProblem is without any functional constraint.

Example:

Minimize y = f (x1, x2) = (x1 − 5)2 + (x2 − 3)3

where x1, x2 ≥ 0

Note: Here, gj = NULL



Constrained optimization problemOptimization problem with at one or more functional constraint(s).

Example:

Maximize y = f (x1, x2, · · · , xn)

Subject togi (x1, x2, · · · , xni ) ≥ ciwhere i = 1, 2, · · · , k and k > 0andx1, x2, · · · , xn are design parameters.



Integer Programming problemIf all the design variables take some integer values.

Example:

Minimize y = f (x1, x2) = 2x1 + x2

Subject tox1 + x2 ≤ 35x1 + 2x2 ≤ 9andx1, x2 are integer variables.



Real-valued problemIf all the design variables are bound to take real values.

Mixed-integer programming problemSome of the design variables are integers and the rest of the variables takereal values.



Linear optimization problem

Both objective functions as well as all constraints are found to be somelinear functions of design variables.

Example:

Maximize y = f (x1, x2) = 2x1 + x2

Subject tox1 + x2 ≤ 35x1 + 2x2 ≤ 10andx1, x2 ≥ 0



Non-linear optimization problem

If either the objective function or any one of the functional constraints arenon-linear function of design variables.

Example:

Maximize y = f (x1, x2) = x21 + 5x32

Subject tox41 + 3x22 ≤ 6292x31 + 4x32 ≤ 133andx1, x2 ≥ 0


Traditional approaches to solve optimizationproblems

Optimization Methods

Linear Programming

Method

Non linear Programming

MethodSpecialized Algorithm

Graphical Method

Simplex Method

Single Variable Multi Variable

Numerical

Method

Analytical

Method

Elimination Method Interpolation Method

Dynamic Programing

Branch & Bound

Greedy Method

Unrestricted method

Exhaustive method

Fibonacci method

Dichotomous Search

Golden Section method

Quadratic

Cubic

Direct root

Constrained

OptimizationUnconstrained

Optimization

Unrestricted method

Exhaustive method

Fibonacci method

Random Walk

Univeriate Method

Pattern Search

Steepest Descent

Conjugate Gradient

Quasi Newton

Variable Match

Divide & Conquer

Lagrangian method


Example : Analytical Method

Suppose, the objective function: y = f (x). Let f (x) be a polynomial ofdegree m and (m > 0)

If y ′ = f ′(x) = 0 for some x = x∗, then we say thaty is optimum (i.e. either minimum or maximum point exist) at the pointx = x∗.

If y ′ = f ′(x) 6= 0 for some x = x∗, then we say thatthere is no optimum value at x = x∗ (i.e. x = x∗ is an inflection point)

An inflection point is also called a saddle point.



Note:An inflection point is a point, that is, neither a maximum nor a minimumat that point.

Following figure explains the concepts of minimum, maximum and saddlepoint.

x1*

Maximum

Minimum

Saddle Points

y

x

x2*



Let us generalize the concept of ”Analytical method”.

If y = f (x) is a polynomial of degree m, then there are m number ofcandidate points to be checked for optimum or saddle points.

Suppose, yn is the nth derivative of y.To further investigate the nature of the point, we determine (firstnon-zero) (n − th) higher order derivativeyn = f n(x = x∗)

There are two cases.Case 1:If yn 6= 0 for n=odd number, then x∗ is an inflection point.

Case 2:If yn = 0 for n = odd number, then there exist an optimum point at x∗.



In order to decide the point x∗ as minimum or maximum, we have to findthe next higher order derivative, that is yn+1 = f n+1(x = x∗).There are two sub cases may be:

Case 2.1:If yn = f n(x = x∗) is positive then x is a local minimum point.

Case 2.2:If yn = f n(x = x∗) is negative then x is a local maximum point.

x1* x4*x3*x2*z1* z4*z3*z2*

y

x

If yn+1 = f n+1(x = x∗) = 0, then we are to repeat the next higher orderderivative.Debasis Samanta (IIT Kharagpur) Soft Computing Applications (IT60108) 06.03.2018 18 / 22

Question

y = f (x)d2ydx = +ve ⇒ x = x∗1d4ydx = −ve ⇒ x = x∗2d6ydx = ±ve ⇒ x = x∗3

y

x

x=x2*

x=x1* x=x3*

Optimal Solution

Is the analytical method solves optimization problem with multiple inputvariables?If ”Yes”, than how?If ”No”, than why not?Debasis Samanta (IIT Kharagpur) Soft Computing Applications (IT60108) 06.03.2018 19 / 22

Exercise

Determine the minimum or maximum or saddle points, if any for thefollowing single variable function f (x)

f (x) = x2

2 + 125x

for some real values of x.


Duality Principle

Principle

A Minimization (Maximization) problem is said to have dual problem if itis converted to the maximization (Minimization) problem.The usual conversion from maximization ⇔ minimizationy = f (x)⇔ y∗ = −f (x)y = f (x)⇔ y∗ = 1

f (x)

y x

y = f(x)

y* = f(x)

Maximization Problem

Minimization Problem


Limitations of the traditional optimization approach

Computationally expensive.

For a discontinuous objective function, methods may fail.

Method may not be suitable for parallel computing.

Discrete (integer) variables are difficult to handle.

Methods may not necessarily adaptive.

Soft Computing techniques have been evolved to address the abovementioned limitations of solving optimization problem with traditionalapproaches.


Evolutionary Algorithms

The algorithms, which follow some biological and physical behaviors:

Biologic behaviors:

Genetics and Evolution –> Genetic Algorithms (GA)Behavior of ant colony –> Ant Colony Optimization (ACO)Human nervous system –> Artificial Neural Network (ANN)

In addition to that there are some algorithms inspired by some physicalbehaviors:

Physical behaviors:

Annealing process –> Simulated Annealing (SA)Swarming of particle –> Particle Swarming Optimization (PSO)Learning –> Fuzzy Logic (FL)

Debasis Samanta (IIT Kharagpur) Soft Computing Applications 06.03.2018 3 / 26

Genetic Algorithm

It is a subset of evolutionary algorithm:

Ant Colony optimizationSwarm Particle Optimization

Models biological processes:GeneticsEvolution

To optimize highly complex objective functions:

Very difficult to model mathematicallyNP-Hard (also called combinatorial optimization) problems (whichare computationally very expensive)Involves large number of parameters (discrete and/or continuous)


Background of Genetic Algorithm

Firs time itriduced by Ptrof. John Holland (of Michigan University, USA,1965).But, the first article on GA was published in 1975.

Principles of GA based on two fundamental biological processes:Genetics: Gregor Johan Mendel (1865)Evolution: Charles Darwin (1875)


A brief account on genetics

The basic building blocks in living bodies are cells. Each cell carriesthe basic unit of heredity, called gene

Chromosome

Nucleus

Other cell bodies

For a particular specie, number of chromosomes is fixed.

ExamplesMosquito: 6Frogs: 26Human: 46Goldfish: 94etc.



Genetic code

Spiral helix of protein substance is called DNA.For a specie, DNA code is unique, that is, vary uniquely from oneto other.DNA code (inherits some characteristics from one generation tonext generation) is used as biometric trait.



Reproduction

+ =

x y

gamete

haploid(Reproductive cell has

half the number of

chromosomes)

Organism’s cell :

Cell division

Each chromosome from

both haploids are combined

to have full numbers

diploid

diploid


A brief account on geneticsCrossing over

Information from two different

organism’s body cells

Combined into so that diversity in information is possible

Random crossover points makes infinite diversities

Kinetochore


A brief account on evolutionEvolution : Natural Selection

Four primary premises:

1 Information propagation: An offspring has many of itscharacteristics of its parents (i.e. information passes from parentto its offspring). [Heredity]

2 Population diversity: Variation in characteristics in the nextgeneration. [Diversity]

3 Survival for exitence: Only a small percentage of the offspringproduced survive to adulthood. [Selection]

4 Survival of the best: Offspring survived depends on theirinherited characteristics. [Ranking]


A brief account on evolution

Mutation:

To make the process forcefully dynamic when variations in populationgoing to stable.


Biological process : A quick overview

Genetics


Working of Genetic Algorithm

Definition of GA:

Genetic algorithm is a population-based probabilistic search andoptimization techniques, which works based on the mechanisms ofnatural genetics and natural evaluation.


Framework of GA

Start

Initial Population

Converge ?

Stop

Selection

Yes

No

Reproduction

Note:An individual in the

population is

corresponding to a

possible solution


Working of Genetic Algorithm

Note:

1 GA is an iterative process.2 It is a searching technique.3 Working cycle with / without convergence.4 Solution is not necessarily guranteed. Usually, terminated with a

local optima.


Framework of GA: A detail view

Start

Initialize population

Converge ?

Stop

Evaluate the fitness

Select Mate

Crossover

Mutation

Inversion

Yes

No

Re

pro

du

cti

on

Define parameters

Parameter representation

Create population

Apply cost

function to each of

the population

Se

lec

tio

n


Optimization problem solving with GA

For the optimization problem, identify the following:

Objective function(s)

Constraint(s)

Input parameters

Fitness evaluation (it may be algorithm or mathematical formula)

Encoding

Decoding


GA Operators

In fact, a GA implementation involved with the realization of thefollowing operations.

1 Encoding: How to represent a solution to fit with GA framework.

2 Convergence: How to decide the termination criterion.

3 Mating pool: How to generate next solutions.

4 Fitness Evaluation: How to evaluate a solution.

5 Crossover: How to make the diverse set of next solutions.

6 Mutation: To explore other solution(s).

7 Inversion: To move from one optima to other.


Different GA Strategies

Simple Genetic Algorithm (SGA)

Steady State Genetic Algorithm (SSGA)

Messy Genetic Algorithm (MGA)


Simple GA

Start

Create Initial population

of size N

Convergence

Criteria meet ?

Stop

Select Np individuals

(with repetition)

Create mating pool (randomly) (Pair of

parent for generating new offspring)

Perform crossover and

create new offsprings

Mutate the offspring

Perform inversion on

the offspring

Yes

No

Evaluate each individuals

Replace all individuals in the last generation

with new offsprings created

Return the individual(s) with

best fitness value

Re

pro

du

ctio

n


Important parameters involved in Simple GA

SGA Parameters

Initial population size : N

Size of mating pool, Np : Np = p%ofN

Convergence threshold δ

Mutation µ

Inversion η

Crossover ρ


Salient features in SGA

Simple GA features:

Have overlapping generation (Only fraction of individuals arereplaced).

Computationally expensive.

Good when initial population size is large.

In general, gives better results.

Selection is biased toward more highly fit individuals; Hence, theaverage fitness (of overall population) is expected to increase insuccession.The best individual may appear in any iteration.


Steady State Genetic Algorithm (SSGA)

Start

Generate Initial population of size N

Reject the

offspring if

duplicated

Stop

Evaluate the offspring

If the offspring are better than the

worst individuals then replace the

worst individuals with the offspring

Yes

No

Evaluate each individuals

Return the solutions

Select two individual without

repetition

Crossover

Mutation

Inversion

Convergence

meet ?


Salient features in Steady-state GA

SGA Features:

Generation gap is small.Only two offspring are produced in one generation.

It is applicable when

Population size is small

Chromosomes are of longer length

Evaluation operation is less computationally expensive (compare toduplicate checking)


Salient features in Steady-state GA

Limitations in SSGA:

There is a chance of stuck at local optima, ifcrossover/mutation/inversion is not strong enough to diversify thepopulation).

Premature convergence may result.

It is susceptible to stagnation. Inferiors are neglected or removedand keeps making more trials for very long period of time withoutany gain (i.e. long period of localized search).


***

Any Questions??


debasis samanta - iitkgp

Documents