managing approximation models in collaborative optimization

DOI 10.1007/s00158-004-0492-y

RESEARCH PAPER

Struct Multidisc Optim (2005) 30: 11–26

B.-S. Jang · Y.-S. Yang · H.-S. Jung · Y.-S. Yeun

Managing approximation models in collaborative optimization

Received: 12 April 2004 / Revised manuscript received: 12 August 2004 / Published online: 7 January 2005 Springer-Verlag 2005

Abstract Collaborative optimization (CO), one of the mul-tidisciplinary design optimization techniques, has been cred-ited with guaranteeing disciplinary autonomy while main-taining interdisciplinary compatibility due to its bi-level op-timization structure. However, a few difficulties caused bycertain features of its architecture have been also reported.The architecture, with discipline-level optimizations nestedin a system-level optimization, leads to considerably in-creased computational time. In addition, numerical difficul-ties such as the problem of slow convergence or unexpectednonlinearity of the compatibility constraint in the system-level optimization are known weaknesses of CO.

This paper proposes the use of an approximation modelin place of the disciplinary optimization in the system-leveloptimization in order to relieve the aforementioned difficul-ties. The disciplinary optimization result, the optimal dis-crepancy function value, is modeled as a function of theinterdisciplinary target variables, and design variables ofthe system level. However, since this approach is hinderedby the peculiar form of the compatibility constraint, it ishard to exploit well-developed conventional approximationmethods. In this paper, neural network classification is em-ployed as a classifier to determine whether a design point

B.-S. Jang (B)Structure/Shipbuilding & Plant R&D Institute, Samsung Heavy In-dustries, Koje-City, Kyungnam, 656-710, KoreaE-mail: [email protected]

Y.-S. YangDepartment of Naval Architecture and Ocean Engineering, SeoulNational University, San 56-1, Shillim-dong, Kwanak-gu, Seoul, 151-742, KoreaE-mail: [email protected]

H.-S. JungRolling Stock Research Department, Structural Mechanics ResearchGroup, Korea Railroad Research Institute, 60-1, Woulam-Dong,Uiwang-City, Kyonggi-Do, 437-050, KoreaE-mail: [email protected]

Y.-S. YeunDepartment of Mechanical Design Engineering, Daejin University,San 11-1, Sundanri, Pochen, Kyonggi-do, 467-711, KoreaE-mail: [email protected]

is feasible or not. Kriging is also combined with the classi-fication to make up for the weakness that the classificationcannot estimate the degree of infeasibility.

In addition, for the purpose of enhancing the accuracy ofthe predicted optimum, this paper also employs two approx-imation management frameworks for single-objective andmulti-objective optimization problem in the system-leveloptimization. The approximation is continuously updatedusing the information obtained from the optimization pro-cess. This can cut down the required number of disciplinaryoptimizations considerably and lead to a design (or Paretoset) near to the true optimum (or true Pareto set) of thesystem-level optimization.

Keywords Approximation management framework · Col-laborative optimization · Neural network classification

1 Introduction

Most engineering design is carried out by multiple teamsthrough the process of decomposition into a set of tractabledesign problems for which a decision is made through an-alysis. These subproblems are hard to solve independentlysince they have interdependent (or coupled) relationships.The field of multidisciplinary design optimization (MDO)has emerged to develop approaches for efficiently optimiz-ing the design of such large coupled systems (Balling andSobieszcznski–Sobieski 1996).

Collaborative optimization (CO) was thereafter de-veloped to follow the multidisciplinary characteristics ofengineering design. It basically consists of a bi-level opti-mization architecture. It is the job of the discipline teamsto satisfy constraints while working to define a design onwhich all the teams involved can agree. The system teamis in charge of adjusting the target values so that suchagreement is possible while minimizing or maximizingthe system-level objective. This architecture is designed topromote disciplinary autonomy while maintaining interdis-ciplinary compatibility (Braun 1996a; Kroo and Manning2000). Due to this structure, CO is judged highly advan-

12 B.-S. Jang et al.

tageous in its applications to practical engineering designproblems.

However, there is insufficient quantitative informationavailable to demonstrate its merit. Although its variousadvantageous features have been demonstrated in Braunet al. (1996b), Kroo et al. (1994), and Kroo (1997), CO isstill relatively immature and little experience in actual indus-trial environments is available yet.

Some difficulties associated with the inherent features ofthe architecture have been reported. Numerical difficultiescaused by certain mathematical manipulations have beencited in Kroo and Manning (2000), Alexandrov and Lewis(1999, 2000). The use of quadratic forms for the system-level compatibility constraints means that changes in thesystem target variables near the solution have little effect onthe constraint values. Specifically, the gradient approacheszero, leading to difficulties for many optimizers, especiallythose that rely on linear approximations to the constraints.This leads to a slow rate of convergence of the system nearthe presumed solution. Several choices are possible for theforms of the disciplinary objective functions and the system-level compatibility constraints. These have been investigatedin a number of previous studies (DeMiguel and Murray2000; McAllister et al. 2000). In spite of such endeavors,in common with most multi-level schemes, the system-leveloptimization of CO may be sensitive to the selection ofdiscipline-level optimization parameters such as feasibilityor optimality tolerances.

In addition, the price that must be paid for the advan-tages of decomposition is an increased computational time— some studies have cited extremely large computationaltime. This unexpected cost is mostly caused by the fact thatthe architecture nests discipline-level optimization in thesystem-level optimization, that is, every disciplinary designshould be performed once in order to evaluate the compati-bility constraint of the system-level optimization. This prob-lem is one of the reasons that hinders the application of CO

Fig. 1 Use of approximation in CO

to engineering design, especially when a disciplinary designcannot be automated or when it requires time-consuming an-alysis.

As an alternative to relieve the aforementioned prob-lems, the use of an approximation model has been proposedin place of the disciplinary design in CO (Sobieski et al.1998a). This models the result of the disciplinary design asa function of system-level design variables (i.e. d∗

i = f(z))as depicted in Fig. 1. The system-level optimizer uses thisapproximation instead of the disciplinary optimizations forestimating the interdisciplinary compatibility constraint.

This concept was addressed initially in Sobiebski et al.(1998a). The paper uses the response surface method andsuggests two approaches to determine a quadratic fit tothe disciplinary optimal results: modeling directly the dis-crepancy function and the optimal interdisciplinary designvariables as a function of the target variables. This paper(Sobiebski et al. 1998a, and Sobiebski 1998b) uses a trustregion approach (Alexandrov et al. 1998) to refine the re-sponse surface sequentially.

This concept is particularly appealing in CO for severalreasons. When used to model the results of disciplinary op-timization, the dimensionality of the approximation can bemuch smaller than would be required for fitting an inte-grated analysis system. Along with the usual approximationfeatures that aid in parallel execution and load balancing,this approach renders this very robust, but inefficient, opti-mizer acceptable. That is, the direct search method such asHooke and Jeeves method or the probabilistic search methodsuch as genetic algorithm or simulated annealing methodcan be used instead of the gradient-based method due to thenearly free computational cost of approximations. This canrelieve the convergence problem of CO (Kroo and Manning2000).

What distinguishes the method of this paper from themethod of Sobiebski et al. (1998a) is that this paper adaptsglobal approximations instead of local approximations. The

Managing approximation models in collaborative optimization 13

sequential response approaches, such as the trust regionmethod, sequentially isolate a small region of good designthat can be accurately represented by a low-order poly-nomial response surface model. However, this is not appro-priate for a multi-objective optimization problem becausethe response region of interest will never be reduced toa small neighborhood that is good for all conflicting ob-jectives. Finally the converged design of the trust regionmethod tends to depend considerably on its starting pointand move limit.

However, there is a difficulty in the use of global ap-proximation. Because of the peculiar form of the compati-bility constraint, it is hard to use conventional approxima-tion methods, such as the response surface method, kriging,neural network, and so on, directly. In this paper, neural net-work classification is employed as a classifier to determinewhether a design point is feasible or not. Also, kriging iscombined with the classifier in order to estimate the degreeof infeasibility. Kriging, spatial correlation modeling, hasbeen asserted to be an approximation technique that showsgood promise for building accurate global approximations ofa design space (Cressie 1993; Simpson 1998b; Trosset andTorczon 1997).

As an effort to reduce the inaccuracy due to the use ofapproximations, this paper also adopts two approximationmanagement frameworks for single-objective and multi-objective optimization problems. Their purpose is not onlyto reduce the computational cost but also to obtain a nearlytrue optimum or Pareto set by sequentially updating the ap-proximations. This framework was proven to reduce the re-quired function calls and to provide a more accurate optimalresult than a method just using approximation without suchan updating process (Yang et al. 2002).

The paper is organized as follows: in Sect. 2, the com-bination of neural network classification and kriging is ex-plained. In Sect. 3, a brief description of the approximationmanagement strategy is provided. Its availability is demon-strated through illustrative examples in Sect. 4. Finally, con-clusions are laid out in Sect. 5.

2 Approximation in system-level optimization in CO

In this section, the characteristic of the discrepancy functionis described in comparison with a conventional constraintand the resultant difficulty in the modeling using conven-tional approximation techniques is cited in Sect. 2.1. Sec-tion 2.2 explains what extra-point information in CO is andhow to utilize it in the approximation. The employment ofneural network classification and the combination of it withkriging is discussed in Sect. 2.3. To begin with, two mathe-matical examples to assist the discussion of this section areintroduced.

Mathematical example 1

This problem was initially introduced in Simpson (1998a) asan illustrative example to demonstrate the usefulness of krig-ing. However, it is modified here into a constrained problemin order to reformulate it as a collaborative optimization

problem. The definition as a standard optimization problemis described in Fig. 2.

The problem can be formulated as a collaborative opti-mization problem as Fig. 3. The system-level optimizer hasone interdisciplinary target variable (y∗) and one interdisci-

Fig. 2 The standard optimization problem definition for example 1

Fig. 3 The collaborative optimization problem definition for ex-ample 1

Fig. 4 Optimal solution of the system-level optimization problem ofexample 1


plinary compatibility constraint. A discipline only evaluatesthe objective function value ( f ), while, the other disciplineexecutes an optimization and passes the optimal discrepancyfunction value (d ) to the system level. The profiles of f andd depicted in Fig. 4 can be obtained by repeatedly perform-ing the discipline-level evaluation and the optimization forvarious values of y∗.

Mathematical example 2

This problem requires the solution of a coupled analysis toevaluate constraints and an objective function. As shown inFig. 5, there exists a link between the evaluation of y1 and

Fig. 5 The standard optimization problem definition for example 2

Fig. 6 The collaborative optimization problem definition for ex-ample 2

Fig. 7 Discrepancy function profiles with varying system-level targetvariables

y2, that is, one of them requires the other as input to itscomputation. In the collaborative optimization problem de-picted in Fig. 6, the system level coordinates the couplingbetween the coupling target variables, y∗

12 and y∗21, which en-

able two disciplines to compute their actual function valuesindependently and to determine the design variables, x1 andx2, separately . Figure 7 depicts the shape of the discrepancyfunction obtained from a series of discipline-level optimiza-tions varying the target variables.

2.1 Difficulty in modelling the discrepancy function usingconventional approximation techniques

In this subsection, the character of the discrepancy functionis explained and the difficulty in its approximation is demon-strated through the aforementioned examples.


The compatibility constraint for system-level optimiza-tion (discrepancy function, di = 0) has a peculiar form com-pared with a conventional inequality constraint (gi ≤ 0). Itcan take on a non-negative value — positive in the infeasibleregion and zero in the feasible region, while the conven-tional constraint may take a negative value in the feasibleregion. Therefore, there is a remarkable change in the trendof its response surface near the boundary between the feas-ible region and the infeasible region. This feature causes theslow convergence problem for CO and makes it difficult toemploy conventional approximation methods directly. Evena small fitting error of the approximation in the feasibleregion may lead to a misjudgment of a feasible design asinfeasible. This may eventually restrict the feasible region.This problem can be observed in the aforementioned mathe-matical example 1 and example 2, as shown in Figs. 4 and 7.

Figure 8a and b depict the profiles of the discrepancyfunction approximated by kriging and neural network, re-spectively in comparison with the exact profile for ex-ample 1. The approximation models are fitted to three sam-ple points. Figure 9a, b, and c show three profiles of theexact shape, kriging and neural network for the discrepancyfunction 1 of example 2, respectively. In this problem, nine

Fig. 8 Profiles of a kriging and exact function, and b neural networkand exact function for example 1

sample points and four extra points are used for the model-ing — extra points will be discussed in Sect. 2.2.

The kriging model produces “over-fitting” in the feasibleregion, as shown in Figs. 8a and 9b. This may lead to seri-ously wrong approximation of the discrepancy function inthe feasible region. The neural network seems to model theresponse in the feasible region correctly for the Example 1.However, there exists a considerable error in the feasible re-gion, which will have a critical influence on the feasibilitydecision. The value of d1 of the exact models is of the orderof 10−20∼−30; however, the value approximated by the neu-ral network is of the order 10−2∼−4, which is comparable tothe allowable tolerance of the compatibility constraint givenby the system-level optimizer. Figure 9c shows a consider-ably reduced feasible region when a reasonable tolerance isgiven. The feasible region is very sensitive to the size of thetolerance.

As observed in the examples, the feasible region esti-mated by the approximation is considerably affected by thecharacter of the approximation. It is necessary to employ anapproximation method that is suited to the character of thediscrepancy function for more accurate modeling.

2.2 Using implicit extra-point information

It was asserted that the incorporation of approximationmodels into CO is very effective in relieving the prob-lems concerning the practical application of CO describedin Sect. 1. Meanwhile, special information can be utilizedfor improving the efficiency of the approximate model gen-eration. This information can reduce the number of disci-plinary optimizations required for the approximation andalso improve the quality of approximation due to the spe-cial properties of the disciplinary optimization (Sobiebski1998b). This subsection introduces this information.

A discipline of CO performs its disciplinary optimizationone time for given target values of the interdisciplinary vari-ables {z∗} and yields the achieved interdisciplinary values{z}. The key observation is that, if a second disciplinaryoptimization were performed where the target values {z∗}were set equal to the optimal solution from the first dis-ciplinary optimization (i.e.{z∗} = {z}), the discipline wouldachieve the target values exactly and the resulting discrep-ancy function value would be equal to zero. Since the resultis known a priori, there is no need to actually perform an-other disciplinary optimization; rather an extra disciplinaryoptimization solution is obtained implicitly, with no addi-tional analysis.

The extra-point information is explained through themathematical example 2. The shaded region of Fig. 10 de-picts the range of actual interdisciplinary variable values thatcan be provided by discipline 1 without violating its con-straints. If target values of the interdisciplinary variables lo-cated outside the range are given by the system level, disci-pline 1 cannot agree with the target values without violatingits constraint, thus, the discrepancy function has a nonzerovalue. Figure 10b shows nine sample points, (y∗

12, y∗21), of

on a 3×3 grid and corresponding extra points, (y12, y21),along with the contour line of exact d1. Here, a sample


Fig. 9 Profiles of a the exact discrepancy function 1, and b kriging model c neural network model for example 2

Fig. 10 Extra-point information for example 2: a the range of interdisciplinary variables to be provided by discipline 1, and b sample pointsand corresponding extra points

point means a set of interdisciplinary target variable valuespassed from the system-level optimizer, and an extra pointmeans a set of variable values actually accomplished by dis-

cipline 1. Among the nine sample points, five sample pointshave their corresponding extra points — four extra pointsamong them are shown in Fig. 10b while the other extra


point lies outside the square region. This means that if oneof the five sets of target values is given to discipline 1 by thesystem-level optimizer, the discipline would fail to coincidewith the target values and produce different interdisciplinaryvariable values. As observed in Fig. 10b, extra points are al-ways located at the boundary between the region of d1 = 0and the region of d1 > 0 — the boundary divides the feas-ible region from the infeasible region for the compatibilityconstraint. An optimal design point of the system-level op-timization of most of the CO problems is located at theboundary. Therefore, the extra points are highly critical tothe accurate modeling of the boundary and the exact searchfor the optimal point.

This special property is created by the decompositionstrategy of CO: that the disciplinary design problems areposed to guarantee a domain-specific feasibility. In order toassure that each discipline can always achieve a design thatis feasible with respect to its local constraints, the systemlevel controls all interdisciplinary variables and each disci-pline is allowed to vary their copies. However, this may be atthe cost of a higher degree of freedom for both the system-level optimization and the disciplinary optimization. Thus,it burdens the optimizers with additional computational ex-pense.

2.3 Introduction of classification neural network

In this subsection, as an alternative to overcome the diffi-culty discussed at Sect. 2.1, neural network classification issuggested.

A neural network, multilayer perceptron (Haykin 1994),is employed as a classifier to decide whether a design pointlies in the feasible region or not. The neural network classi-fication approximates the “region” itself while conventionalapproximation models approximate the “response value”.Because of the different object of the modeling, the inac-curacy of the approximation can be reduced considerably.The misjudgment of the feasibility may be limited to theneighborhood of the boundary between the feasible and in-feasible regions. However, due to extra points located along

Fig. 12 Profiles of a exact discrepancy function 1, and b the neural network classification model for example 2

the boundary, the error of the judgment can be further re-duced.

Learning and mapping procedures are as follows. First,if a sample point (i.e. a set of interdisciplinary target vari-ables values) is given to a discipline, the discipline performsdisciplinary optimization and yields an optimal discrepancyfunction value. According to the value, the sample point isclassified as “feasible” or “infeasible” and assigned {1, 0}or {0, 1} as its output pattern, {O1, O2}, respectively. Thetolerance used in this paper to qualify feasibility is about0.5% of the average value of the interdisciplinary variable.That is, if the difference between the optimized interdisci-plinary value and its target value is smaller than 0.5% ofits average value, it can be judged feasible. The neural net-work to be trained has the same number of input nodes asthe target variables and two output nodes. After learning, theneural network makes a decision by comparing the resultsobtained at the output nodes, {O1, O2}. It judges a certaindesign point feasible if its output results in O1 > O2, other-wise infeasible, as shown in Fig. 11.

The neural network classification is applied to the mathe-matical example 2. Nine sample points and four extra pointsare used to construct the neural network. The feasible region

Fig. 11 Multilayer perceptron as a classifier


Fig. 13 Learning and mapping process of the combination of the neural network classification and kriging model

Fig. 14 Combined neural network classification — kriging model applied to example 1: a respective profiles, and b combined profile

approximated by the neural network is very similar to theexact region, as depicted in Fig. 12.

However, in spite of the aforementioned advantage ofthe classifier, it cannot evaluate the degree of infeasibilityfor an infeasible design point. It just provides a judgmentof whether a design point is feasible or not. The degree ofinfeasibility is important for any kind of optimization al-gorithm, even for a direct or a global search method likethe simulated annealing or the genetic algorithm (Goldberg1989). In this paper, kriging is combined with the classifica-tion in order to estimate the infeasibility for a query point,once the point is classified into an infeasible class by theclassifier. This process is explained in Fig. 13. Sample dataitself is used for building kriging and classified data is usedfor training the classification. When predicting the responseof a query point, it is first filtered by the classification. If itis judged feasible, its output is assigned a value of zero, oth-erwise its infeasibility level is approximated by kriging orthe neural network. Since the response obtained in this waymay have a discontinuity at the boundary of the feasible andinfeasible regions, the genetic algorithm is thus used in thispaper instead of the gradient-based search methods.

This combined approximation method is applied to ex-ample 1. Figure 14a shows both profiles of kriging and neu-ral network classification fitted to three sample points and

Fig. 14b shows the combined profile of the two. Althoughthe combined response has a discontinuity between the re-gions judged feasible and infeasible by the classification, itfollows an overall trend of the exact discrepancy functionbetter than the two.

In the case of utilizing extra-point information, the learn-ing process can be modified slightly, as explained in Fig. 15.Conventional approximation is fitted to sample points lo-

Fig. 15 Learning strategy when extra-point information is available


cated in the infeasible region and corresponding extra points.Sample points in the feasible region are excluded. Since theconventional approximation plays the role of estimating theinfeasibility level, it is not necessary to use all the samplepoints. The sample points within the feasible region maycause over-fitting, thus it may deteriorate the accuracy of theestimation in the infeasible region.

This section discussed the appropriate selection of ap-proximation methods and their combination. In the follow-ing section, the approximation management strategy, ratherthan the approximation method itself, is introduced in brief.

3 Managing approximation models in optimization

In engineering problems, computationally intensive high-fidelity models or expensive computer simulations hinderthe use of standard optimization techniques because theyhave to be invoked repeatedly during optimization, despitethe tremendous growth of computer capability. Therefore,these expensive analyses are often replaced with approxi-mation models that can be evaluated with considerably lesseffort. However, due to their limited accuracy, it is prac-tically impossible to find an actual optimum exactly (ora set of actual non-inferior solutions) of the original single-or multi-objective optimization problem. Significant effortshave been made to overcome this limitation. The modelmanagement framework is one of such endeavours (Bookeret al. 1999; Dennis and Torczon 1997). The approximation

Fig. 16 Adaptive approximation in single-objective optimization

models are sequentially updated during the iterative opti-mization process in such a way that their capability to modelthe original functions accurately, especially in our region ofinterest, can be improved. The models are modified and im-proved by using one or several sample points generated bymaking good use of the predictive ability of the approxima-tion models. In the management strategy proposed in thispaper, which will be called adaptive approximation in sin-gle optimization (AASO) henceforth, ‘a predicted optimum’to be obtained by solving an optimization problem definedby surrogates in place of high-fidelity models is used forthe modification of the surrogates (Toropov 1989; Toropovet al. 1996; Yang et al. 2000). This strategy is applied tosystem-level optimization of CO, and disciplinary optimiza-tion is replaced by the approximation. Figure 16 shows a de-tailed algorithm of AASO.

In multi-objective optimization, a genetic algorithm thatcan treat multiple objectives (MOGA, Kim 1994) is used forgenerating the Pareto set. MOGA predicts a Pareto set usingapproximation models fitted to sample points that have beenpreviously observed. In the Pareto set, a few points will beselected for reanalysis and further adapting the approxima-tion models of the true analysis. The idea of the maximindistance design criteria (Johnson et al. 1990), one of thespace-filling design criteria, is adopted to select a few pointsfrom the Pareto set. As the name implies, it is used to maxi-mize the minimum distance (or Euclidean distance) betweenany two sample points. It is intended to prevent the sam-ple points from clustering in one portion of the design space


Fig. 17 Adaptive approximation in multi-objective optimization

and thereby to ensure that they are spread out. That is, themost isolated points in the design variable space are selectedfor the reanalysis. This algorithm, named AAMOGA, wasdeveloped in (Yang et al. 2002) and is explained in Fig. 17.

In the following section, the AASO and AAMOGAstrategies using the combination of classification neural net-work and kriging are demonstrated through various CO ex-amples.

4 Illustrative examples

In the following examples, the approximation is used inplace of the compatibility constraint of the system-level op-timization. In addition, the approximation is sequentiallyupdated according to the AASO or AAMOGA strategy.

4.1 Mathematical example 1 — AASO in CO

In this subsection, the AASO strategy is applied to the math-ematical problem 1 already introduced in Sect. 2. AASOwith the combination of classification neural network andkriging is verified through this example. In this problem,the use of extra-point information is intentionally avoidedbecause an extra point obtained from a disciplinary opti-mization is the very optimal point. At first, three samplepoints are used to construct the initial approximation forthe compatibility constraint (discrepancy function = 0) atthe system-level optimization. As shown in Fig. 18, five ad-ditional sample points (making a total of eight points) arerequired for the convergence of the AASO strategy.

The initial approximation model fitted to three samplepoints has a remarkable discontinuity at the boundary be-tween the feasible and infeasible regions, which is drawn bythe trained classification neural network. However, as sev-eral approximate optimal points are to be found near theboundary and these are used as new sample points during theAASO strategy, the discontinuity fades out and the profileof the approximation becomes more similar to the true one,especially near the boundary.

4.2 Mathematical example 2 — AASO in CO

In example 2 already introduced in Sect. 2, five kinds ofapproximation methods in place of two compatibility con-straint (i.e. discrepancy functions) will be tried for AASOin CO: kriging, neural network, classification only, the com-bination of classification neural network and kriging (clas-sification + kriging), and the combination of classificationneural network and neural network (classification + neuralnetwork). Here, the neural network is used just for the ap-proximation of the response value itself, separately from theneural network classification.

Two cases are considered: one case utilizing extra-pointinformation and the other case without. Table 1 summarizesthe results of the first case. Figure 19 depicts five approxi-mate 3-D shapes of the discrepancy function 1 as well as theactual model when AASO ends up converging to a solution.

The 3-D shape of kriging looks considerably differentfrom the exact model, that is, over-fitting arises in the cen-tral part of the feasible region where d1 = 0. However, thecase using kriging succeeded in finding exact solution be-cause extra points near the exact solution have a strong effecton exactly modelling that region and eventually lead to theoptimum.

On the other hand, although the shape of the neural net-work seems to be very similar to the exact model in thefeasible region, there is a remarkable difference between theexact optimum and the result of the neural network. Thisis because there exists a decisive error in the feasible re-gion that has a critical influence on feasibility. The value ofd1 of the exact model is of the order of 10−20∼30, howeverthe neural network result is of the order 10−2∼−4, whichis roughly the allowable tolerance of the compatibility con-straint given by the system-level optimization. The case ofclassification, the case of classification + kriging, and the


Fig. 18 Profiles of the approximation model, the combination of clas-sification and kriging, for the discrepancy function when the numberof sample points is a 3, b 4, c 7, and d 8

Table 1 Results of mathematical example 2 when utilizing extra-pointinformation

Approximation No. of Optimal design point Objectivesample pts of system-level problem

Exact optimum 51 (17.732, 25.296) 37.677(SQP) (fcn call)Kriging 13 (17.843, 25.451) 37.756Neural network 14 (19.706, 25.451) 41.481Classification 16 (18.333, 25.529) 38.665Classif. + kriging 15 (17.745, 25.451) 37.559Classif. + NN 15 (17.647, 25.294) 37.509

case of classification + neural network provide not only al-most the same shape but also a value exactly equal to zero inthe feasible region. This fact demonstrates the usefulness ofclassification, especially for the exact approximation of thefeasible region. Among the three, the case of classification +kriging gives the most accurate optimal point.

Next, for the purpose of investigating the effect of theextra-point information, an approach not utilizing extra-point information is also taken. Table 2 summarizes the re-sults. The case of kriging fails in achieving the true opti-mum, while the case of classification + kriging succeedsand gives the best result among the five attempts, which isthe same result as the previous approach utilizing extra-pointinformation. This proves that the use of the extra-point infor-mation has a strong influence on the success of the case ofkriging but little influence on that of the case of classification+ kriging. This fact will be identified again in the followingtwo-member hub frame example.

4.3 Two-member hub frame example — AASO in CO

This problem is to find the optimal cross-sectional dimen-sions of a two-member hub frame, as shown in Fig. 20. Itsdetail description is laid out in Balling and Sobieszczanski-Sobieski (1995).

The CO formulation of the problem and the use of ap-proximations are shown in Fig. 21. The coupling variables,the member forces of PQ and PR, are removed by perform-ing structural analysis of discipline 0 prior to the optimiza-tions of discipline 1 and 2. This leads to the reduction ofnot only the size of the system-level optimization problemby eliminating six system-level design variables, but also thecomplexity of the discrepancy function of each disciplineby cutting down the dimension of the function. In spite ofthis advantage, it leads to the problem that the extra-pointinformation cannot be utilized for the approximation of thediscrepancy function because the information is availableonly when copies of all the interdisciplinary variables re-lated to a discipline can be controlled by that discipline.However, discipline 1 and 2 do not control the copies of themember forces of PQ or PR, which influence the result oftheir optimization.

The volume is an objective function to be minimized andit is computed directly without approximation. The allow-able displacement constraint at P is treated at the systemlevel. Without the removal of coupling variables, the mem-


Fig. 19 3-D shapes of discrepancy function 1 in a the exact model, b kriging, c neural network, d classification + kriging, e classification +kriging, and f classification + neural network

ber forces of PQ and PR, the displacement constraint shouldbe treated at discipline 0. Translational and rotational dis-placement constrains at P are accumulated into a new con-straint, d0, which is approximated by kriging.

Allowable stress and buckling stress constraints for eachmember are to be satisfied in the corresponding discipline.Each of two discrepancy functions, d1 and d2, is modeledas a function of system-level target variables: the area and


Table 2 Results of mathematical example 2 not utilizing extra-pointinformation

Approximation No. of Optimal design point Objectivesample pts of system-level problem

Exact optimum 51 (17.732, 25.296) 37.677(SQP) (fcn call)Kriging 11 (19.020, 19.804) 48.435Neural network 18 (19.902, 22.314) 45.712Classification 15 (18.039, 25.588) 39.007Classif. + kriging 14 (17.745, 25.294) 37.704Classif. + NN 16 (18.137, 25.451) 38.344

Fig. 20 Two-member hub frame problem

Fig. 21 CO formulation for the two-member hub frame

moment of inertia of the two members, A∗1, I∗

1 , A∗2, and,

I∗2 .

In the same manner as example 2, five approximationmethods are applied for comparative purposes and five tri-als for each case are made. Table 3 summarizes the results.Solving CO problem by sequential quadratic programming(SQP) without approximations requires 148 function calls(i.e. discipline-level optimizations).

In this problem, the case of kriging fails to convergeto a true solution. As mentioned at Sect. 4.2, this can beexplained by the fact that this problem cannot utilize theextra-point information. The points lie along the boundary

Table 3 The results of the two-member hub frame problem

Approximation Average no. Average No. of convergenceof sample pts objective / total no. of trials

Kriging 25.4 2036.2 5/5Neural Network 92.0 1253.0 1/5Classification 27.0 1245.0 5/5Classification 35.8 1240.4 5/5+ KrigingClassification + NN 34.6 1265.4 5/5

Exact Solution (without approximation)No. of discipline level optimization = 148

{A1, I1, A2, I2} = {3.345, 42.567, 3.672, 35.750}Volume = 1236.158


of the feasible and infeasible regions and the optimal pointof this problem is located somewhere on that boundary. Thecase of the neural network offers solutions close to the exactsolution, but it mostly fails to achieve convergence withinthe limit of the number of sample points, 100. The casesof classification, classification + kriging and classification+ neural network succeed in converging to points near theoptimal point. Among them, the case of classification +kriging gives the best result. The number of function calls(i.e. discipline-level optimizations) is considerably reducedcompared to the case of not using approximations.

4.4 Two-member hub frame example — AAMOGA in CO

In this subsection, adaptive approximation in multi-objectivegenetic algorithm (AAMOGA) is applied to a multi-objec-tive collaborative optimization problem. The two-memberhub frame example introduced in Sect. 4.3 is extended toa multi-objective optimization problem by adding anotherobjective function, translational displacement at P. The ef-fect of AAMOGA is illustrated through comparison withfive different cases. Their differences are summarized inTable 4. Figure 22 shows each resultant Pareto set obtainedfrom the cases. Each result is compared with the result ofthe weighting method in standard optimization, which canbe regarded as an exact Pareto set.

Case I is the strategy newly proposed in this paper. Thediscrepancy functions of disciplines 1 and 2 are modelledby the combination of classification and kriging. The ac-cumulated constraint of discipline 0 and the translationaldisplacement at P are modelled by kriging. The strategyof AAMOGA is applied to the system-level optimization.This case yields a Pareto set that has a good correspon-dence with the exact Pareto set with 66 function calls, i.e.discipline-level optimizations, as shown in Fig. 22a. Thisnumber can be regarded as quite small compared with thecases not using approximation models such as Case V ofFig. 22e or Case VI of Fig. 22f. Case V and VI requireabout 30–40 times as many as Case I because they carry

Table 4 The six approaches to solving the two-member hub frame example

Approximation type and updating

Case title multi-objective d0 D1 d2 translational updatingoptimization displacement or not?

I AAMOGA with MOGA Kriging classification classification kriging Yesclassification + kriging + kriging + kriging

II AAMOGA with kriging MOGA Kriging kriging kriging kriging Yes

III One approximation MOGA Kriging classification classification kriging No+ kriging + kriging

IV AASO with weighting method Kriging classification classification kriging Yesclassification + kriging + GA + kriging + kriging

V MOGA without approximation MOGA not using approximations

VI Weighting method without weighting method not using approximationsapproximation + SQP

out discipline-level optimization instead of using approxi-mations.

Case II employs only kriging for all of the approxima-tions. Its results converge to a quite different result from theexact Pareto set, spending just 30 function calls as depictedin Fig. 22b. This may be seen as premature convergence andshows that kriging has a limitation in correctly approximat-ing the discrepancy function when not using classification.

Case III is the same as Case I except that there is noupdating process. Though the failure of Case III shown inFig. 22c, it can be judged that the updating process, the coreof AAMOGA, is a decisive factor for convergence to theexact Pareto set.

Case IV utilizes the same approximation models asCase I, the combination of classification and kriging. How-ever, this case employs the weighting method to treat mul-tiple objectives instead of MOGA. Since the weightingmethod changes a multi-objective optimization problem intoa single-objective optimization problem, the AASO strat-egy replaces AAMOGA as an approximation managementframework. The Case IV generates Pareto solutions one byone as varying weighting factors of design objectives. InAASO, sample points cluster on the local region near eachPareto solution, thus, the resultant Pareto solution may bemore exact than one of Pareto set obtained from AAMOGA.However, in order to generate the same number of Pareto so-lutions as AAMOGA, it would require many more functioncalls than AAMOGA. On the other hand, AAMOGA selectssample points in such a way that all sample points are spreadacross a comparatively large region in which Pareto solu-tions are predicted to be located by approximation models.This strategy focuses on improving the accuracy of the ap-proximations all over the region of interest and utilizingsample points as efficient as possible. As shown in Fig. 22d,Case IV consumes as many as 150 function calls in orderto generate only 11 Pareto solutions, while Case I uses 66function calls for 99 Pareto solutions.

This example illustrates the usefulness of the applicationof AAMOGA in multi-objective CO problem.


Fig. 22 Pareto set of the two-member hub frame example: a AAMOGA with classification and kriging, b AAMOGA with kriging, c oneapproximation, d AASO with classification and kriging, e MOGA without approximation, and f weighting method without approximation

5 Conclusion

This paper focuses on the use of approximation modelsfor modeling the disciplinary optimal result in collabora-tive optimization. The target of the approximation is nota time-consuming analysis but an optimization. The result,

the optimal discrepancy function value, is approximated asa function of the system-level design variables. The approx-imation is utilized in the system-level optimization insteadof executing the discipline-level optimizations directly. Inthis paper, neural network classification is newly introducedfor the approximation. Since it approximates the decision ofwhether a design is feasible or not instead of the response of


a function, it can avoid the difficulty in modeling caused bythe particular trend of the profile of the discrepancy function.In addition, in order to overcome the weakness of classifi-cation that cannot estimate the degree of infeasibility, this iscombined with kriging. The classification plays the role ofdiscriminating whether a design point satisfies the compat-ibility constraint or not and kriging plays the role of evaluat-ing the infeasibility level of a design point if it is estimatedinfeasible by the classification.

Two strategies (AASO and AAMOGA) to update theapproximation models using the information obtained fromthe optimization are used for the system-level optimiza-tion using approximations. The following advantages areexpected. First, the number of required disciplinary opti-mizations can be reduced remarkably. Since the disciplinarydesign is nested in the system-level optimization, this canmake it easy to apply collaborative optimization to engin-eering design, especially when a disciplinary design cannotbe automated or when it requires time-consuming analysis.Second, the use of approximation allows the system-leveloptimization to use a robust search method like a director a global search method, thus it can overcome the con-vergence problem of collaborative optimization. These ad-vantages are demonstrated through numerical examples, andstructural optimization problems.

Acknowledgement This work was supported by the Advanced ShipEngineering Research Center of the Korea Science & EngineeringFoundation.

References

Alexandrov NM, Dennis Jr JE, Lewis RM, Torczon V (1998) A trustregion framework for managing the use of approximation modelsin optimization. Struct Optim 15:16–23

Alexandrov NM, Lewis RM (1999) Comparative Properties of Collab-orative Optimization and Other Approaches to MDO. TechnicalReport No. 99-24, ICASE, Institute for Computer Applicationsin Science and Engineering, NASA Langley Research Center,Hampton, VA

Alexandrov NM, Lewis RM (2000) Analytical and computa-tional properties of distributed approaches to MDO. Proc. 8thAIAA/USAF/NASA/ISSMO Symp. on Multidisciplinary An-alysis and Optimization (held in Long Beach, CA), pp 13–42,AIAA-2000-4718

Balling RJ, Sobieszczanski-Sobieski J (1995) An algorithm for solv-ing the system-level problem in multilevel optimization. StructOptim 9:168–177

Balling RJ, Sobieszczanski-Sobieski J (1996) Optimization of coupledsystems: a critical overview of approaches. AIAA J 34(1):6–17

Booker AJ, Dennis Jr JE, Frank PD, Serafini DB, Torczon V, TrossetMW (1999) A rigorous framework for optimization of expensivefunctions by surrogates. Struct Optim 17:1–13

Braun RD (1996a) Collaborative optimization: an architecture forlarge-scale distributed design. Ph.D. Thesis, Stanford University

Braun RD, Moore AA, Kroo IM (1996b) Use of the collaborativeoptimization architecture for launch vehicle design. Proc. 6thAIAA/USAF/NASA/ISSMO Symp. on Multidisciplinary Analy-sis and Optimization, (held in Bellevue, Washington), pp 306–318, AIAA Paper No. 96-4018

Cressie NAC (1993) Statistics for spatial data. Revised Edition. Wiley,New York

DeMiguel A, Murray W (2000) An analysis of collaborative optimiza-tion methods. Proc. 6th AIAA/USAF/NASA/ISSMO Symp. onMultidisciplinary Analysis and Optimization, (held in Bellevue,Washington), AIAA-2000-4720

Dennis JE, Torczon Jr V (1997) Managing approximation models inoptimization. In: Alexandrov N, Hussaini MY (eds) Multidisci-plinary design optimization: state-of-the-art, pp 330–347, SIAM,Philadelphia, Also available as Technical Report 95-19, Depart-ment of Computational and Applied Mathematics, Rice Univer-sity, Houston, Texas 77005-1892

Goldberg DE (1989) Genetic algorithms on search, optimization, andmachine learning, Addison-Wesley, Reading, MA

Haykin S (1994) Neural networks, a comprehensive foundation.Macmillian College, New York

Johnson ME, Moore LM, Ylvisaker D (1990) Minmax and maximindistance design. J Stat Plann Inference 26:131–148

Kim GH (1994) Multicriteria structural optimization by genetic algo-rithm (in korean). Ph.D. Thesis, Department of Naval Architecture& Ocean Engineering, Seoul National University, Seoul, Korea

Kroo I, Altus S, Braun R, Gage P, Sobieski I (1994) Multidisciplinaryoptimization methods for aircraft preliminary design. Proc. 5thAIAA/USAF/NASA/ISSMO Symposium on MultidisciplinaryAnalysis and Optimization, (held in Panama City, FL), Vol. 1,pp 697–707, AIAA 94-4325

Kroo IM (1997) MDO for large-scale design. In: Alexandrov NM,Hussaini MY (eds) Multidisciplinary design optimization: state ofthe art, pp 22–44, SIAM, PA

Kroo IM, Manning VM (2000) Collaborative optimization: statusand directions. Proc. 8th AIAA/USAF/NASA/ISSMO Symp.on Multidisciplinary Analysis and Optimization, (held in LongBeach, CA), AIAA-2000-4721

McAllister CD, Simpson TW, Yukish M (2000) Goal programmingapplications in multidisciplinary design optimization. Proc. 8thAIAA/USAF/NASA/ISSMO Symp. on Multidisciplinary Analy-sis and Optimization, held in Long Beach, CA), AIAA-2000-4717

Sobieski IP, Manning VM, Kroo IM (1998a) Response surface es-timation and refinement in collaborative optimization. Proc. 7thAIAA/USAF/NASA/ISSMO Symp. on Multidisciplinary Analy-sis and Optimization, (held in St. Louis, Missouri), pp 359–370,AIAA 98-4753

Sobieski IP (1998b) Multidisciplinary design using collaborative opti-mization. Ph.D. Thesis, Department of Aeronautics and Astronau-tics, Stanford University

Sobieski IP, Kroo IM (2000) Collaborative optimization using re-sponse surface estimation. AIAA J 38(10):1931–1938

Simpson TW (1998a) A concept exploration method for product fam-ily design. Ph.D. Thesis, Department of Mechanical Engineering,Georgia Institute of Technology

Simpson TW (1998b) Spatial correlation metamodels for global ap-proximation in structural design optimization. Proc. DETC98(ASME Design Engineering Technical Conf. held Atlanta, GA)

Toropov VV (1989) Simulation approach to structural optimization.Struct Optim 1:37–46

Toropov V, van Keulen F, Markine V, de Boer H (1996) Refine-ments in the multi-point approximation method to reduce theeffects of noisy structural responses. Proc. 6th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis andOptimization (held in Bellevue, WA), Vol. 2, pp 941–951AIAA-96-4087-CP

Trosset MW, Torczon V (1997) Numerical optimization using com-puter experiments. Technical Report 97-38, ICASE, NASA Lan-gley Research Center, Hampton, VA

Yang YS, Jang BS, Yeun YS, Jung HS (2000) A framework for man-aging approximation models in place of expensive simulations inoptimization. Proc. 2nd ASMO UK/ISSMO Conference on En-gineering Design Optimization (held in Swansea, Wales, UK),pp 249–256

Yang YS, Jang BS, Yeun YS, Ruy WS (2002) Managing approxima-tion models in multi-objective optimization. Struct Optim 24:141–156

managing approximation models in collaborative optimization

Documents