Slide 1

Two objectives of surrogate fittingTo fit a surrogate we minimize an error measure, called also loss function.We also like the surrogate to be simple:Fewest basis functionsSimplest basis functionsFlatness is desirable (given y=1 for x=i, i=1,10 we dont fit a sine to the data. Why?)

The process of fitting a surrogate involves minimizing an error measure, such as the mean square error, sometimes called the loss function. It is also desirable to fit the data with a small number of simple base functions. For example, if we can fit the data with a linear polynomial with the same accuracy as with a quadratic polynomial we would prefer the former.

Another desirable property is flatness. For example, given that y=1 for xi=i, we would rather fit a straight line rather than a sine.1Support Vector RegressionCombines loss function and flatness as a single objective.Support vector machines developed by Vapnik and coworkers for optical character recognitions in Russia in the 1960s.First use for regression in 1997.Besides regression has become popular as classifier to divide design space into feasible domain (where constraints are satisfied) and infeasible domain (where they are not).

Support vector regression is a surrogate that explicitly minimizes the weighted sum of the loss function and flatness.

Its origins is in Russia where it was developed by Vapnik and coworkers as Support Vector Machines for optical character recognition and object recognition (Vapnik and Lerner, 1963). Continued by Vapnik at Bell Labs. Application to regression since 1997. First part of lecture based on Smola and Scholkopf (2004)

In surrogate based optimization research, support vector machines is often use as a classifier rather than as surrogate to define the feasible domain where all the constraints are satisfied (e.g., Basoudhar and Missoum)

Basudhar and Missoum, An improved adaptive sampling scheme for the construction of explicit boundaries Structural and Multidisciplinary Optimization, 42 (4) (2010), 517-529SMOLA and SCHOLKOPF A tutorial on support vector regression, Statistics and Computing 14: 199222, 2004Vapnik V. and Lerner A. 1963. Pattern recognition using generalized portrait method. Automation and Remote Control, 24: 774780.Vapnik V., Golowich S., and Smola A. 1997. Support vector method for function approximation, regression estimation, and signal processing. In: Mozer M.C., Jordan M.I., and Petsche T. (Eds.) Advances in Neural Information Processing Systems 9, MA, MIT Press, Cambridge. pp. 281287.

2Epsilon-insensitive loss functionSupport vector regression can use any loss function, but the one most often associated with it is epsilon-insensitive.It is less sensitive to one bad data point.

Figures from Gunns Support Vector Machines for Classification and Regression

3Flatness measure

4SVR optimization problem

5Example

6SolutionFigure compares the two fits

7Problems