Global Optimization Algorithms Theory and Application 3rd

EdAuthor: Thomas Weisetweise@gmx.deEdition: thirdVersion: 2011-12-07-15:31Newest Version: http://www.it-weise.de/projects/book.pdfSources/CVS-Repository: goa-taa.cvs.sourceforge.net2Don,tprintme!Also think aboutthe trees thatwould have todieforthepaper!This book is much moreuseful as electronic resource:youcansearchtermsandclicklinks. You can t do that in aprintedbook.,Thebookisfrequentlyupdatedandimprovedaswell.Printedversionswilljustoutdate.Don,tprintme!PrefaceThis e-book is devoted to Global Optimization algorithms, which are methods for ndingsolutions of high quality for an incredible wide range of problems. We introduce the basicconcepts of optimization and discuss features which make optimization problems dicultand thus, should be considered when trying to solve them. In this book, we focus onmetaheuristic approaches like Evolutionary Computation, Simulated Annealing, ExtremalOptimization, Tabu Search, and Random Optimization. Especially the Evolutionary Com-putation methods, which subsume Evolutionary Algorithms, Genetic Algorithms, GeneticProgramming, Learning Classier Systems, Evolution Strategy, Dierential Evolution, Par-ticle Swarm Optimization, and Ant Colony Optimization, are discussed in detail.In this third edition, we try to make a transition from a pure material collection andcompendium to a more structured book. We try to address two major audience groups:1. Our book may help students, since we try to describe the algorithms in an under-standable, consistent way. Therefore, we also provide the fundamentals and muchbackground knowledge. You can nd (short and simplied) summaries on stochastictheory and theoretical computer science in Part VI on page 638. Additionally, appli-cation examples are provided which give an idea how problems can be tackled withthe dierent techniques and what results can be expected.2. Fellow researchers and PhD students may nd the application examples helpful too.For them, in-depth discussions on the single approaches are included that are supportedwith a large set of useful literature references.The contents of this book are divided into three parts. In the rst part, dierent op-timization technologies will be introduced and their features are described. Often, smallexamples will be given in order to ease understanding. In the second part starting at page530, we elaborate on dierent application examples in detail. Finally, in the last part fol-lowing at page 638, the aforementioned background knowledge is provided.In order to maximize the utility of this electronic book, it contains automatic, clickablelinks. They are shaded with dark gray so the book is still b/w printable. You can click on1. entries in the table of contents,2. citation references like Heitkotter and Beasley [1220],3. page references like 253,4. references such as see Figure 28.1 on page 254 to sections, gures, tables, and listings,and5. URLs and links like http://www.lania.mx/ccoello/EMOO/ [accessed 2007-10-25].11URLs are usually annotated with the date we have accessed them, like http://www.lania.mx/ccoello/EMOO/ [accessed 2007-10-25]. We can neither guarantee that their content remains unchanged, northat these sites stay available. We also assume no responsibility for anything we linked to.4 PREFACEThe following scenario is an example for using the book: A student reads the text andnds a passage that she wants to investigate in-depth. She clicks on a citation which seemsinteresting and the corresponding reference is shown. To some of the references which areonline available, links are provided in the reference text. By clicking on such a link, theAdobe ReaderR 2will open another window and load the regarding document (or a browserwindow of a site that links to the document). After reading it, the student may use thebackwards button in the Acrobat Readers navigation utility to go back to the text initiallyread in the e-book.If this book contains something you want to cite or reference in your work, please usethe citation suggestion provided in Chapter A on page 945. Also, I would be very happyif you provide feedback, report errors or missing things that you have (or have not) found,criticize something, or have any additional ideas or suggestions. Do not hesitate to contactme via my email address tweise@gmx.de. Matter of fact, a large number of people helpedme to improve this book over time. I have enumerated the most important contributors inChapter D Thank you guys, I really appreciate your help! At many places in this bookwe refer to Wikipedia The Free Encyclopedia [2938] which is a great source of knowledge.Wikipedia The Free Encyclopedia contains articles and denitions for many of the aspectsdiscussed in this book. Like this book, it is updated and improved frequently. Therefore,including the links adds greatly to the books utility, in my opinion.The updates and improvements will result in new versions of the book, which willregularly appear at http://www.it-weise.de/projects/book.pdf. The completeLATEX source code of this book, including all graphics and the bibliography, is hostedat SourceForge under http://sourceforge.net/projects/goa-taa/ in the CVSrepository goa-taa.cvs.sourceforge.net. You can browse the repository underhttp://goa-taa.cvs.sourceforge.net/goa-taa/ and anonymously download thecomplete sources of it by using the CVS command given in Listing 1.3cvs -z3 -d:pserver:anonymous@goa-taa.cvs.sourceforge.net:/cvsroot/goa-taacheckout -P BookListing 1: The CVS command for anonymously checking out the complete book sources.Compiling the sources requires multiple runs of LATEX, BibTEX, and makeindex because ofthe nifty way the references are incorporated. In the repository, an Ant-Script is provided fordoing this. Also, the complete bibliography of this book is stored in a MySQL4database andthe scripts for creating this database as well as tools for querying it (Java) and a MicrosoftAccess5frontend are part of the sources you can download. These resources as well as allcontents of this book (unless explicitly stated otherwise) are licensed under the GNU FreeDocumentation License (FDL, see Chapter B on page 947). Some of the algorithms providedare made available under the LGPL license (see Chapter C on page 955)Copyright c _ 2011 Thomas Weise.Permission is granted to copy, distribute and/or modify this document under the terms ofthe GNU Free Documentation License, Version 1.3 or any later version published by the FreeSoftware Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-CoverTexts. A copy of the license is included in the Chapter B on page 947 entitled GNU FreeDocumentation License (FDL).2The Adobe ReaderR is available for download at http://www.adobe.com/products/reader/ [ac-cessed 2007-08-13].3More information can be found at http://sourceforge.net/scm/?type=cvs&group_id=2643524http://en.wikipedia.org/wiki/MySQL [accessed 2010-07-29]5http://en.wikipedia.org/wiki/Microsoft_Access [accessed 2010-07-29]ContentsTitle Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Part I Foundations1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.1 What is Global Optimization? . . . . . . . . . . . . . . . . . . . . . . . . . . 211.1.1 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221.1.2 Algorithms and Programs . . . . . . . . . . . . . . . . . . . . . . . . . 231.1.3 Optimization versus Dedicated Algorithms . . . . . . . . . . . . . . . 241.1.4 Structure of this Book . . . . . . . . . . . . . . . . . . . . . . . . . . . 241.2 Types of Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . 241.2.1 Combinatorial Optimization Problems . . . . . . . . . . . . . . . . . . 241.2.2 Numerical Optimization Problems . . . . . . . . . . . . . . . . . . . . 271.2.3 Discrete and Mixed Problems . . . . . . . . . . . . . . . . . . . . . . 291.2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301.3 Classes of Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . 311.3.1 Algorithm Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311.3.2 Monte Carlo Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 341.3.3 Heuristics and Metaheuristics . . . . . . . . . . . . . . . . . . . . . . 341.3.4 Evolutionary Computation . . . . . . . . . . . . . . . . . . . . . . . . 361.3.5 Evolutionary Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 371.4 Classication According to Optimization Time . . . . . . . . . . . . . . . . . 371.5 Number of Optimization Criteria . . . . . . . . . . . . . . . . . . . . . . . . 391.6 Introductory Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 Problem Space and Objective Functions . . . . . . . . . . . . . . . . . . . . 432.1 Problem Space: How does it look like? . . . . . . . . . . . . . . . . . . . . . 432.2 Objective Functions: Is it good? . . . . . . . . . . . . . . . . . . . . . . . . . 443 Optima: What does good mean? . . . . . . . . . . . . . . . . . . . . . . . . . 513.1 Single Objective Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513.1.1 Extrema: Minima and Maxima of Dierentiable Functions . . . . . . 523.1.2 Global Extrema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533.2 Multiple Optima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543.3 Multiple Objective Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 556 CONTENTS3.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553.3.2 Lexicographic Optimization . . . . . . . . . . . . . . . . . . . . . . . 593.3.3 Weighted Sums (Linear Aggregation) . . . . . . . . . . . . . . . . . . 623.3.4 Weighted Min-Max . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643.3.5 Pareto Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653.4 Constraint Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703.4.1 Genome and Phenome-based Approaches . . . . . . . . . . . . . . . . 713.4.2 Death Penalty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713.4.3 Penalty Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713.4.4 Constraints as Additional Objectives . . . . . . . . . . . . . . . . . . 723.4.5 The Method Of Inequalities . . . . . . . . . . . . . . . . . . . . . . . 723.4.6 Constrained-Domination . . . . . . . . . . . . . . . . . . . . . . . . . 753.4.7 Limitations and Other Methods . . . . . . . . . . . . . . . . . . . . . 753.5 Unifying Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 753.5.1 External Decision Maker . . . . . . . . . . . . . . . . . . . . . . . . . 753.5.2 Prevalence Optimization . . . . . . . . . . . . . . . . . . . . . . . . . 764 Search Space and Operators: How can we nd it? . . . . . . . . . . . . . . 814.1 The Search Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 814.2 The Search Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834.3 The Connection between Search and Problem Space . . . . . . . . . . . . . . 854.4 Local Optima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 884.4.1 Local Optima of single-objective Problems . . . . . . . . . . . . . . . 894.4.2 Local Optima of Multi-Objective Problems . . . . . . . . . . . . . . . 894.5 Further Denitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 905 Fitness and Problem Landscape: How does the Optimizer see it? . . . . 935.1 Fitness Landscapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 935.2 Fitness as a Relative Measure of Utility . . . . . . . . . . . . . . . . . . . . . 945.3 Problem Landscapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 956 The Structure of Optimization: Putting it together. . . . . . . . . . . . . 1016.1 Involved Spaces and Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1016.2 Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1036.3 Other General Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046.3.1 Gradient Descend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046.3.2 Iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1056.3.3 Termination Criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . 1056.3.4 Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066.3.5 Modeling and Simulating . . . . . . . . . . . . . . . . . . . . . . . . . 1067 Solving an Optimization Problem . . . . . . . . . . . . . . . . . . . . . . . . 1098 Baseline Search Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1138.1 Random Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1138.2 Random Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1148.2.1 Adaptive Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1158.3 Exhaustive Enumeration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1169 Forma Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119CONTENTS 710 General Information on Optimization . . . . . . . . . . . . . . . . . . . . . . 12310.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12310.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12510.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . . . . . 12610.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135Part II Diculties in Optimization11 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13912 Problem Hardness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14312.1 Algorithmic Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14312.2 Complexity Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14512.2.1 Turing Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14512.2.2 T, AT, Hardness, and Completeness . . . . . . . . . . . . . . . . . . 14612.3 The Problem: Many Real-World Tasks are AT-hard . . . . . . . . . . . . . 14712.4 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14713 Unsatisfying Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15113.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15113.2 The Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15213.2.1 Premature Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . 15213.2.2 Non-Uniform Convergence . . . . . . . . . . . . . . . . . . . . . . . . 15213.2.3 Domino Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . 15313.3 One Cause: Loss of Diversity . . . . . . . . . . . . . . . . . . . . . . . . . . . 15413.3.1 Exploration vs. Exploitation . . . . . . . . . . . . . . . . . . . . . . . 15513.4 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15713.4.1 Search Operator Design . . . . . . . . . . . . . . . . . . . . . . . . . . 15713.4.2 Restarting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15713.4.3 Low Selection Pressure and Population Size . . . . . . . . . . . . . . 15713.4.4 Sharing, Niching, and Clearing . . . . . . . . . . . . . . . . . . . . . . 15713.4.5 Self-Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15813.4.6 Multi-Objectivization . . . . . . . . . . . . . . . . . . . . . . . . . . . 15814 Ruggedness and Weak Causality . . . . . . . . . . . . . . . . . . . . . . . . . 16114.1 The Problem: Ruggedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16114.2 One Cause: Weak Causality . . . . . . . . . . . . . . . . . . . . . . . . . . . 16114.3 Fitness Landscape Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . 16314.3.1 Autocorrelation and Correlation Length . . . . . . . . . . . . . . . . . 16314.4 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16415 Deceptiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16715.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16715.2 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16715.3 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16816 Neutrality and Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17116.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17116.2 Evolvability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17216.3 Neutrality: Problematic and Benecial . . . . . . . . . . . . . . . . . . . . . 17216.4 Neutral Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17316.5 Redundancy: Problematic and Benecial . . . . . . . . . . . . . . . . . . . . 17416.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17516.7 Needle-In-A-Haystack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1758 CONTENTS17 Epistasis, Pleiotropy, and Separability . . . . . . . . . . . . . . . . . . . . . 17717.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17717.1.1 Epistasis and Pleiotropy . . . . . . . . . . . . . . . . . . . . . . . . . 17717.1.2 Separability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17817.1.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17917.2 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17917.3 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18217.3.1 Choice of the Representation . . . . . . . . . . . . . . . . . . . . . . . 18217.3.2 Parameter Tweaking . . . . . . . . . . . . . . . . . . . . . . . . . . . 18317.3.3 Linkage Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18317.3.4 Cooperative Coevolution . . . . . . . . . . . . . . . . . . . . . . . . . 18418 Noise and Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18718.1 Introduction Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18718.2 The Problem: Need for Robustness . . . . . . . . . . . . . . . . . . . . . . . 18818.3 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19019 Overtting and Oversimplication . . . . . . . . . . . . . . . . . . . . . . . . 19119.1 Overtting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19119.1.1 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19119.1.2 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19319.2 Oversimplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19419.2.1 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19419.2.2 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19520 Dimensionality (Objective Functions) . . . . . . . . . . . . . . . . . . . . . . 19720.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19720.2 The Problem: Many-Objective Optimization . . . . . . . . . . . . . . . . . . 19820.3 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20120.3.1 Increasing Population Size . . . . . . . . . . . . . . . . . . . . . . . . 20120.3.2 Increasing Selection Pressure . . . . . . . . . . . . . . . . . . . . . . . 20120.3.3 Indicator Function-based Approaches . . . . . . . . . . . . . . . . . . 20120.3.4 Scalerizing Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . 20220.3.5 Limiting the Search Area in the Objective Space . . . . . . . . . . . . 20220.3.6 Visualization Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 20221 Scale (Decision Variables) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20321.1 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20321.2 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20421.2.1 Parallelization and Distribution . . . . . . . . . . . . . . . . . . . . . 20421.2.2 Genetic Representation and Development . . . . . . . . . . . . . . . . 20421.2.3 Exploiting Separability . . . . . . . . . . . . . . . . . . . . . . . . . . 20421.2.4 Combination of Techniques . . . . . . . . . . . . . . . . . . . . . . . . 20522 Dynamically Changing Fitness Landscape . . . . . . . . . . . . . . . . . . . 20723 The No Free Lunch Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 20923.1 Initial Denitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20923.2 The Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21023.3 Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21023.4 Innite and Continuous Domains . . . . . . . . . . . . . . . . . . . . . . . . 21323.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213CONTENTS 924 Lessons Learned: Designing Good Encodings . . . . . . . . . . . . . . . . . 21524.1 Compact Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21524.2 Unbiased Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21524.3 Surjective GPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21624.4 Injective GPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21624.5 Consistent GPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21624.6 Formae Inheritance and Preservation . . . . . . . . . . . . . . . . . . . . . . 21724.7 Formae in Genotypic Space Aligned with Phenotypic Formae . . . . . . . . . 21724.8 Compatibility of Formae . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21724.9 Representation with Causality . . . . . . . . . . . . . . . . . . . . . . . . . . 21824.10Combinations of Formae . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21824.11Reachability of Formae . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21824.12Inuence of Formae . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21824.13Scalability of Search Operations . . . . . . . . . . . . . . . . . . . . . . . . . 21924.14Appropriate Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21924.15Indirect Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21924.15.1Extradimensional Bypass: Example for Good Complexity . . . . . . . 219Part III Metaheuristic Optimization Algorithms25 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22525.1 General Information on Metaheuristics . . . . . . . . . . . . . . . . . . . . . 22625.1.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 22625.1.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22725.1.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 22725.1.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22826 Hill Climbing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22926.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22926.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22926.3 Multi-Objective Hill Climbing . . . . . . . . . . . . . . . . . . . . . . . . . . 23026.4 Problems in Hill Climbing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23126.5 Hill Climbing With Random Restarts . . . . . . . . . . . . . . . . . . . . . . 23226.6 General Information on Hill Climbing . . . . . . . . . . . . . . . . . . . . . . 23426.6.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 23426.6.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23426.6.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 23426.6.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23526.7 Raindrop Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23526.7.1 General Information on Raindrop Method . . . . . . . . . . . . . . . 23627 Simulated Annealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24327.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24327.1.1 Metropolis Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 24327.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24527.2.1 No Boltzmanns Constant . . . . . . . . . . . . . . . . . . . . . . . . . 24627.2.2 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24727.3 Temperature Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24727.3.1 Logarithmic Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . 24827.3.2 Exponential Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . 24927.3.3 Polynomial and Linear Scheduling . . . . . . . . . . . . . . . . . . . . 24927.3.4 Adaptive Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24927.3.5 Larger Step Widths . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24910 CONTENTS27.4 General Information on Simulated Annealing . . . . . . . . . . . . . . . . . . 25027.4.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 25027.4.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25027.4.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 25027.4.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25128 Evolutionary Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25328.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25328.1.1 The Basic Cycle of EAs . . . . . . . . . . . . . . . . . . . . . . . . . . 25428.1.2 Biological and Articial Evolution . . . . . . . . . . . . . . . . . . . . 25528.1.3 Historical Classication . . . . . . . . . . . . . . . . . . . . . . . . . . 26328.1.4 Populations in Evolutionary Algorithms . . . . . . . . . . . . . . . . . 26528.1.5 Conguration Parameters of Evolutionary Algorithms . . . . . . . . . 26928.2 Genotype-Phenotype Mappings . . . . . . . . . . . . . . . . . . . . . . . . . 27028.2.1 Direct Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27028.2.2 Indirect Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27228.3 Fitness Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27428.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27428.3.2 Weighted Sum Fitness Assignment . . . . . . . . . . . . . . . . . . . . 27528.3.3 Pareto Ranking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27528.3.4 Sharing Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27728.3.5 Variety Preserving Fitness Assignment . . . . . . . . . . . . . . . . . 27928.3.6 Tournament Fitness Assignment . . . . . . . . . . . . . . . . . . . . . 28428.4 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28528.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28528.4.2 Truncation Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28928.4.3 Fitness Proportionate Selection . . . . . . . . . . . . . . . . . . . . . 29028.4.4 Tournament Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . 29628.4.5 Linear Ranking Selection . . . . . . . . . . . . . . . . . . . . . . . . . 30028.4.6 Random Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30228.4.7 Clearing and Simple Convergence Prevention (SCP) . . . . . . . . . . 30228.5 Reproduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30628.6 Maintaining a Set of Non-Dominated/Non-Prevailed Individuals . . . . . . . 30828.6.1 Updating the Optimal Set . . . . . . . . . . . . . . . . . . . . . . . . 30828.6.2 Obtaining Non-Prevailed Elements . . . . . . . . . . . . . . . . . . . . 30928.6.3 Pruning the Optimal Set . . . . . . . . . . . . . . . . . . . . . . . . . 31128.7 General Information on Evolutionary Algorithms . . . . . . . . . . . . . . . 31428.7.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 31428.7.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31628.7.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 31728.7.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32229 Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32529.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32529.1.1 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32529.2 Genomes in Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 32629.2.1 Classication According to Length . . . . . . . . . . . . . . . . . . . . 32629.2.2 Classication According to Element Type . . . . . . . . . . . . . . . . 32729.2.3 Classication According to Meaning of Loci . . . . . . . . . . . . . . 32729.2.4 Introns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32729.3 Fixed-Length String Chromosomes . . . . . . . . . . . . . . . . . . . . . . . 32829.3.1 Creation: Nullary Reproduction . . . . . . . . . . . . . . . . . . . . . 32829.3.2 Mutation: Unary Reproduction . . . . . . . . . . . . . . . . . . . . . 33029.3.3 Permutation: Unary Reproduction . . . . . . . . . . . . . . . . . . . . 333CONTENTS 1129.3.4 Crossover: Binary Reproduction . . . . . . . . . . . . . . . . . . . . . 33529.4 Variable-Length String Chromosomes . . . . . . . . . . . . . . . . . . . . . . 33929.4.1 Creation: Nullary Reproduction . . . . . . . . . . . . . . . . . . . . . 34029.4.2 Insertion and Deletion: Unary Reproduction . . . . . . . . . . . . . . 34029.4.3 Crossover: Binary Reproduction . . . . . . . . . . . . . . . . . . . . . 34029.5 Schema Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34129.5.1 Schemata and Masks . . . . . . . . . . . . . . . . . . . . . . . . . . . 34129.5.2 Wildcards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34229.5.3 Hollands Schema Theorem . . . . . . . . . . . . . . . . . . . . . . . . 34229.5.4 Criticism of the Schema Theorem . . . . . . . . . . . . . . . . . . . . 34529.5.5 The Building Block Hypothesis . . . . . . . . . . . . . . . . . . . . . 34629.5.6 Genetic Repair and Similarity Extraction . . . . . . . . . . . . . . . . 34629.6 The Messy Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 34729.6.1 Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34729.6.2 Reproduction Operations . . . . . . . . . . . . . . . . . . . . . . . . . 34729.6.3 Overspecication and Underspecication . . . . . . . . . . . . . . . . 34829.6.4 The Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34829.7 Random Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34929.8 General Information on Genetic Algorithms . . . . . . . . . . . . . . . . . . 35129.8.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 35129.8.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35229.8.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 35329.8.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35630 Evolution Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35930.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35930.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36030.3 Recombination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36230.3.1 Dominant Recombination . . . . . . . . . . . . . . . . . . . . . . . . . 36230.3.2 Intermediate Recombination . . . . . . . . . . . . . . . . . . . . . . . 36230.4 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36430.4.1 Using Normal Distributions . . . . . . . . . . . . . . . . . . . . . . . . 36430.5 Parameter Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36730.5.1 The 1/5th Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36830.5.2 Endogenous Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 37030.6 General Information on Evolution Strategies . . . . . . . . . . . . . . . . . . 37330.6.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 37330.6.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37330.6.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 37330.6.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37631 Genetic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37931.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37931.1.1 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37931.2 General Information on Genetic Programming . . . . . . . . . . . . . . . . . 38231.2.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 38231.2.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38331.2.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 38331.2.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38631.3 (Standard) Tree Genomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38631.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38631.3.2 Creation: Nullary Reproduction . . . . . . . . . . . . . . . . . . . . . 38931.3.3 Node Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39231.3.4 Unary Reproduction Operations . . . . . . . . . . . . . . . . . . . . . 39312 CONTENTS31.3.5 Recombination: Binary Reproduction . . . . . . . . . . . . . . . . . . 39831.3.6 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40031.4 Linear Genetic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . 40331.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40331.4.2 Advantages and Disadvantages . . . . . . . . . . . . . . . . . . . . . . 40431.4.3 Realizations and Implementations . . . . . . . . . . . . . . . . . . . . 40531.4.4 Recombination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40731.4.5 General Information on Linear Genetic Programming . . . . . . . . . 40831.5 Grammars in Genetic Programming . . . . . . . . . . . . . . . . . . . . . . . 40931.5.1 General Information on Grammar-Guided Genetic Programming . . . 40931.6 Graph-based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40931.6.1 General Information on Grahp-based Genetic Programming . . . . . 40932 Evolutionary Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41332.1 General Information on Evolutionary Programming . . . . . . . . . . . . . . 41432.1.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 41432.1.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41432.1.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 41432.1.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41733 Dierential Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41933.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41933.2 Ternary Recombination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41933.2.1 Advanced Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42033.3 General Information on Dierential Evolution . . . . . . . . . . . . . . . . . 42133.3.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 42133.3.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42133.3.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 42133.3.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42434 Estimation Of Distribution Algorithms . . . . . . . . . . . . . . . . . . . . . 42734.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42734.2 General Information on Estimation Of Distribution Algorithms . . . . . . . 42834.2.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 42834.2.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42834.2.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 42934.2.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43134.3 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43134.4 EDAs Searching Bit Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . 43434.4.1 UMDA: Univariate Marginal Distribution Algorithm . . . . . . . . . 43434.4.2 PBIL: Population-Based Incremental Learning . . . . . . . . . . . . . 43834.4.3 cGA: Compact Genetic Algorithm . . . . . . . . . . . . . . . . . . . . 43934.5 EDAs Searching Real Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . 44234.5.1 SHCLVND: Stochastic Hill Climbing with Learning by Vectors ofNormal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44234.5.2 PBILC: Continuous PBIL . . . . . . . . . . . . . . . . . . . . . . . . 44434.5.3 Real-Coded PBIL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44434.6 EDAs Searching Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44734.6.1 PIPE: Probabilistic Incremental Program Evolution . . . . . . . . . . 44734.7 Diculties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45234.7.1 Diversity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45234.7.2 Epistasis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454CONTENTS 1335 Learning Classier Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45735.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45735.2 Families of Learning Classier Systems . . . . . . . . . . . . . . . . . . . . . 45735.3 General Information on Learning Classier Systems . . . . . . . . . . . . . . 45935.3.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 45935.3.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45935.3.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 45935.3.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46236 Memetic and Hybrid Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 46336.1 Memetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46336.2 Lamarckian Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46436.3 Baldwin Eect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46436.4 General Information on Memetic Algorithms . . . . . . . . . . . . . . . . . . 46736.4.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 46736.4.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46736.4.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 46736.4.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47037 Ant Colony Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47137.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47137.2 General Information on Ant Colony Optimization . . . . . . . . . . . . . . . 47337.2.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 47337.2.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47337.2.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 47337.2.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47638 River Formation Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47738.1 General Information on River Formation Dynamics . . . . . . . . . . . . . . 47838.1.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 47838.1.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47838.1.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 47838.1.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47939 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 48139.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48139.2 The Particle Swarm Optimization Algorithm . . . . . . . . . . . . . . . . . . 48139.2.1 Communication with Neighbors Social Interaction . . . . . . . . . . 48239.2.2 Particle Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48239.2.3 Basic Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48239.3 General Information on Particle Swarm Optimization . . . . . . . . . . . . . 48339.3.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 48339.3.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48339.3.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 48339.3.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48640 Tabu Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48940.1 General Information on Tabu Search . . . . . . . . . . . . . . . . . . . . . . 49040.1.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 49040.1.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49040.1.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 49040.1.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49114 CONTENTS41 Extremal Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49341.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49341.1.1 Self-Organized Criticality . . . . . . . . . . . . . . . . . . . . . . . . . 49341.1.2 The Bak-Sneppen model of Evolution . . . . . . . . . . . . . . . . . . 49341.2 Extremal Optimization and Generalized Extremal Optimization . . . . . . . 49441.3 General Information on Extremal Optimization . . . . . . . . . . . . . . . . 49641.3.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 49641.3.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49641.3.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 49641.3.4 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49742 GRASPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49942.1 General Information on GRAPSs . . . . . . . . . . . . . . . . . . . . . . . . 50042.1.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 50042.1.2 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 50042.1.3 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50043 Downhill Simplex (Nelder and Mead) . . . . . . . . . . . . . . . . . . . . . . 50143.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50143.2 General Information on Downhill Simplex . . . . . . . . . . . . . . . . . . . 50243.2.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 50243.2.2 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 50243.2.3 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50244 Random Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50544.1 General Information on Random Optimization . . . . . . . . . . . . . . . . . 50644.1.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 50644.1.2 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 50644.1.3 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506Part IV Non-Metaheuristic Optimization Algorithms45 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50946 State Space Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51146.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51146.1.1 The Baseline: A Binary Goal Criterion . . . . . . . . . . . . . . . . . 51146.1.2 State Space and Neighborhood Search . . . . . . . . . . . . . . . . . . 51146.1.3 The Search Space as Graph . . . . . . . . . . . . . . . . . . . . . . . . 51246.1.4 Key Eciency Features . . . . . . . . . . . . . . . . . . . . . . . . . . 51446.2 Uninformed Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51446.2.1 Breadth-First Search . . . . . . . . . . . . . . . . . . . . . . . . . . . 51546.2.2 Depth-First Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51646.2.3 Depth-Limited Search . . . . . . . . . . . . . . . . . . . . . . . . . . . 51646.2.4 Iteratively Deepening Depth-First Search . . . . . . . . . . . . . . . . 51646.2.5 General Information on Uninformed Search . . . . . . . . . . . . . . . 51746.3 Informed Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51846.3.1 Greedy Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51946.3.2 ASearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51946.3.3 General Information on Informed Search . . . . . . . . . . . . . . . . 520CONTENTS 1547 Branch And Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52347.1 General Information on Branch And Bound . . . . . . . . . . . . . . . . . . 52447.1.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 52447.1.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52447.1.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 52448 Cutting-Plane Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52748.1 General Information on Cutting-Plane Method . . . . . . . . . . . . . . . . . 52848.1.1 Applications and Examples . . . . . . . . . . . . . . . . . . . . . . . . 52848.1.2 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52848.1.3 Conferences and Workshops . . . . . . . . . . . . . . . . . . . . . . . 528Part V Applications49 Real-World Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53149.1 Symbolic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53149.1.1 Genetic Programming: Genome for Symbolic Regression . . . . . . . 53149.1.2 Sample Data, Quality, and Estimation Theory . . . . . . . . . . . . . 53249.1.3 Limits of Symbolic Regression . . . . . . . . . . . . . . . . . . . . . . 53549.2 Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53649.2.1 Classication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53649.3 Freight Transportation Planning . . . . . . . . . . . . . . . . . . . . . . . . . 53649.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53749.3.2 Vehicle Routing in Theory and Practice . . . . . . . . . . . . . . . . . 53849.3.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54049.3.4 Evolutionary Approach . . . . . . . . . . . . . . . . . . . . . . . . . . 54249.3.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54549.3.6 Holistic Approach to Logistics . . . . . . . . . . . . . . . . . . . . . . 54949.3.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55150 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55350.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55350.2 Bit-String based Problem Spaces . . . . . . . . . . . . . . . . . . . . . . . . . 55350.2.1 Kaumans NK Fitness Landscapes . . . . . . . . . . . . . . . . . . . 55350.2.2 The p-Spin Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55650.2.3 The ND Family of Fitness Landscapes . . . . . . . . . . . . . . . . . . 55750.2.4 The Royal Road . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55850.2.5 OneMax and BinInt . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56250.2.6 Long Path Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56350.2.7 Tunable Model for Problematic Phenomena . . . . . . . . . . . . . . 56650.3 Real Problem Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58050.3.1 Single-Objective Optimization . . . . . . . . . . . . . . . . . . . . . . 58050.4 Close-to-Real Vehicle Routing Problem Benchmark . . . . . . . . . . . . . . 62150.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62150.4.2 Involved Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62550.4.3 Problem Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63250.4.4 Objectives and Constraints . . . . . . . . . . . . . . . . . . . . . . . . 63450.4.5 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635Part VI Background16 CONTENTS51 Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63951.1 Set Membership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63951.2 Special Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64051.3 Relations between Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64051.4 Operations on Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64151.5 Tuples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64351.6 Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64351.7 Binary Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64451.8 Order Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64551.9 Equivalence Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64651.10Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64651.10.1Monotonicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64751.11Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64751.11.1Sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64951.11.2Searching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65051.11.3Transformations between Sets and Lists . . . . . . . . . . . . . . . . . 65052 Graph Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65153 Stochastic Theory and Statistics . . . . . . . . . . . . . . . . . . . . . . . . . 65353.1 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65353.1.1 Probabily as dened by Bernoulli (1713) . . . . . . . . . . . . . . . . 65453.1.2 Combinatorics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65453.1.3 The Limiting Frequency Theory of von Mises . . . . . . . . . . . . . . 65553.1.4 The Axioms of Kolmogorov . . . . . . . . . . . . . . . . . . . . . . . . 65653.1.5 Conditional Probability . . . . . . . . . . . . . . . . . . . . . . . . . . 65753.1.6 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65753.1.7 Cumulative Distribution Function . . . . . . . . . . . . . . . . . . . . 65853.1.8 Probability Mass Function . . . . . . . . . . . . . . . . . . . . . . . . 65953.1.9 Probability Density Function . . . . . . . . . . . . . . . . . . . . . . . 65953.2 Parameters of Distributions and their Estimates . . . . . . . . . . . . . . . . 65953.2.1 Count, Min, Max and Range . . . . . . . . . . . . . . . . . . . . . . . 66053.2.2 Expected Value and Arithmetic Mean . . . . . . . . . . . . . . . . . . 66153.2.3 Variance and Standard Deviation . . . . . . . . . . . . . . . . . . . . 66253.2.4 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66453.2.5 Skewness and Kurtosis . . . . . . . . . . . . . . . . . . . . . . . . . . 66553.2.6 Median, Quantiles, and Mode . . . . . . . . . . . . . . . . . . . . . . 66553.2.7 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66753.2.8 The Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . . . . 66853.3 Some Discrete Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 66853.3.1 Discrete Uniform Distribution . . . . . . . . . . . . . . . . . . . . . . 66853.3.2 Poisson Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 67153.3.3 Binomial Distribution B(n, p) . . . . . . . . . . . . . . . . . . . . . . 67453.4 Some Continuous Distributions . . . . . . . . . . . . . . . . . . . . . . . . . 67653.4.1 Continuous Uniform Distribution . . . . . . . . . . . . . . . . . . . . 67653.4.2 Normal Distribution N_, 2_ . . . . . . . . . . . . . . . . . . . . . . 67853.4.3 Exponential Distribution exp() . . . . . . . . . . . . . . . . . . . . . 68253.4.4 Chi-square Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 68453.4.5 Students t-Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 68753.5 Example Throwing a Dice . . . . . . . . . . . . . . . . . . . . . . . . . . . 68953.6 Estimation Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69153.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69153.6.2 Likelihood and Maximum Likelihood Estimators . . . . . . . . . . . . 69353.6.3 Condence Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696CONTENTS 1753.7 Statistical Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69953.7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 699Part VII Implementation54 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70555 The Specication Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70755.1 General Denitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70755.1.1 Search Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70855.1.2 Genotype-Phenotype Mapping . . . . . . . . . . . . . . . . . . . . . . 71155.1.3 Objective Function . . . . . . . . . . . . . . . . . . . . .