lastlecture% heuris,c%op,mizaon% · 2013-07-03 · lastlecture% heuris,c%op,mizaon%...
TRANSCRIPT
Last Lecture Heuris,c Op,miza,on (using some slides from earlier lectures)
Handout 12-‐2-‐11
2
So Why would you want to take this course?
• For many problems there is no feasible way to find good solu,ons except by using heuris,cs methods.
• For other problems, heuris,cs are easier to use and possibly as good as non heuris,c methods
• You will learn about methods that are easy to program and to understand that can be applied to almost all op,miza,on problems and related problems like parameter es,ma,on.
3
Your use of these methods
• There is a very high probability that you will make use of these algorithms either alone or in combina,on with other op,miza,on methods.
• It will be fun to be able to crea,vely generate your own algorithm modifica,ons and to try them out on complex project case studies.
• If you do use these methods later in your job or degree program, drop me an email. It helps me know how to improve the course.
4
Advantages and Disadvantages of Different Op,miza,on Methods
• No one op,miza,on method is best for all problems. (No Free Lunch Theorem)
• Problems that are linear or convex and differen,able have special methods that are quite efficient computa,onally (e.g. linear programming or deriva,ve-‐based nonlinear programming) .
• Combinatorial (discrete variable) and global op,miza,on (con,nuous variable) are very difficult and might be solved best by a heuris,c.
5
Linear Programming (LP) Advantages
• Very efficient for many variables
• User Friendly soZware
• Users don’t need to understand algorithm
• If you have a problem to solve, consider first if it can be solved by linear programming.
6
Linear Programming Disadvantages
• Restricted to linear mathema,cal form with con,nuous variables
• (Some nonlinear problems can be approximated by linear mathema,cal form.)
7
Integer Programming Advantages
• User Friendly soZware
• For problems with a modest number of integer variables, and linear func,ons, MIP is oZen the most efficient technique
8
Integer Programming (IP)
• Restricted to linear mathema,cal form
• Computa,onal efficiency is poor for large number of integer variables so IP is not necessarily faster than a heuris,c. (Heuris,cs have been shown to get beaer solu,ons in some cases than IP given a fixed number of itera,ons.)
9
Nonlinear Programming Advantage
Fastest for differen,able problems with con,nuous variables and a nonlinear, but convex response surface
Only one local minimum
10
Global versus Local Minima Mul$-‐Modal Problems have Mul$ple local minima
These are the problems for which heuris$cs are o:en the best choice
F(x)
X (parameter value)
Local minimum
Global minimum
11
Disadvantages of Global Op,miza,on Problems
• There is no way to tell how close to op,mal a solu,on is, • (By contrast for NLP, if the deriva,ve of Cost(s)
evaluated at the current solu,on is close to zero, then the current solu,on is probably close to an op,ma (maybe a local op,mum)).
• It usually takes a lot more func,on evalua,ons to find an op,mial or near op,mal solu,on to a global problem than a local op,miza,on problem.
• The difficulty is with the problem (e.g it is mul,modal) not because of the solu,on method
12
Why do We Care About Mul,-‐Modal Func,ons?
• With Complex nonlinear models, we typically do not know if the model is mul,-‐modal (i.e. has many local minima) or unimodal.
• Hence to analyze a complex model we need to search for a global minimum even if we do not know if it is mul, modal.
• Hence global op,miza,on capability is important when dealing with complex nonlinear models.
13
Nonlinear Programming Disadvantages
Cannot be used for discrete variables or most nonconvex problems
Calcula,on of deriva,ves can require excessive amounts of CPU and results may not be accurate
High exper,se required May find only local op,mum
14
Heuris,c Methods Advantages (SA, GA, TS, DDS, RW, etc.)
Easy to learn Easy to Implement (in serial or parallel) Doesn’t stop at local op,mum and might find global op,mum
Heuris,cs tend to parallelize rela,vely efficiently (which will be increasingly important in the future).
15
Heuris,c Methods Advantages con,nued
Can find good answers for a very wide class of problems (integer, binary, con,nuous variables; nonlinear or even discon,nuous func,ons; and even where Cost(S) is a code, not a single equa,on)
Human effort to code and run is typically much less for heuris,cs than for nonlinear methods. (CPU ,me is rela,vely cheap.)
It’s fun !
16
Heuris,c Methods: Disadvantages
For problems that sa,sfy the assump,ons of other algorithms, the heuris,c method will probably take more (serial) CPU ,me and give a less accurate answer.
• Perhaps no powerful way to incorporate knowledge of mathema,cal structure into algorithm
17
Heuris,c Methods: Difficul,es
There is no strong theore,cal background for describing speed of convergence for most problems or even for the best selec,on of algorithm parameters values.
Hence comparison between algorithms needs to be largely empirical and sta,s,cal.
However for global op,miza,on problems there is oZen no alterna,ve to using heuris,c methods.
18
CPU ,me, Human Effort and Size of Func,on Evalua,on
For problems for which there exists non heuris,c algorithms, you need to weigh the
human ;me saved by using a simple general method (e.g. a heuris,c) versus
The CPU ;me taken to solve the problem.
19
Computa,onal Efficiency • In order to save on expensive human ,me, it is oZen worthwhile to use a simple robust method that uses more CPU ,me and less human ,me
• However, there is a limit to how long a business can wait to get an answer.
• Hence for computa,onally expensive Cost(S) func,ons, having a faster algorithm is worthwhile
• The poten,al disadvantage of heuris,c methods is lessened with parallel compu,ng since many heuris,cs are rela,vely more efficient (in comparison to more mathema,cal methods) when calcula,ons are done on a large number of parallel or distributed processors
Punchline • The best choice of algorithm will depend on the type of problem.
• For many problems, heuris,cs are probathe only op,on for solu,on (e.g. mul,modal mixed integer)
• For some problems, there are more mathema,cally based methods that might be faster in terms of CPU but take more human ,me because they are more specialized so you need to weigh whether CPU or human ,me is more important.
What are the Big Ideas in Each of the Algorithms We Studied?
• Simulated Annealing
• Gene,c Algorithm
• Tabu Search
What are the Big Ideas in Each of the Algorithms We Studied?
• DES (Derandomized Evolu,onary Strategy)
• DDS (Dynamically Dimensioned Search)
• DE Differen,al Evolu,on (Theory Sec,on)
What are the Big Ideas in Each of the Algorithms We Studied?
• Mul, Objec,ve
• Gene,c Programming
• Response Surfaces with Evolu,onary Algorithms (Theory Sec,on)
What is the Big Idea In
• Sta,s,cal Comparisons
• No Free Lunch