Complex & Intelligent Systems (2021) 7:1185–1194https://doi.org/10.1007/s40747-020-00240-6

ORIG INAL ART ICLE

Genetic programming with separability detection for symbolicregression

Wei-Li Liu1 · Jiaquan Yang1 · Jinghui Zhong1 · Shibin Wang2

Received: 30 June 2020 / Accepted: 23 November 2020 / Published online: 4 January 2021© The Author(s) 2021

AbstractGenetic Programming (GP) is a popular and powerful evolutionary optimization algorithm that has awide range of applicationssuch as symbolic regression, classification and program synthesis. However, existing GPs often ignore the intrinsic structureof the ground truth equation of the symbolic regression problem. To improve the search efficacy of GP on symbolic regressionproblems by fully exploiting the intrinsic structure information, this paper proposes a genetic programming with separabilitydetection technique (SD-GP). In the proposed SD-GP, a separability detection method is proposed to detect additive separablecharacteristics of input features from the observed data. Then based on the separability detection results, a chromosomerepresentation is proposed, which utilizes multiple sub chromosomes to represent the final solution. Some sub chromosomesare used to construct separable sub functions by using separate input features, while the other sub chromosomes are usedto construct sub functions by using all input features. The final solution is the weighted sum of all sub functions, and theoptimal weights of sub functions are obtained by using the least squares method. In this way, the structure information canbe learnt and the global search ability of GP can be maintained. Experimental results on synthetic problems with differingcharacteristics have demonstrated that the proposed SD-GP can perform better than several state-of-the-art GPs in terms ofthe success rate of finding the optimal solution and the convergence speed.

Keywords Genetic programming · Least squares method · Multi-chromosome · Symbolic regression · Separability detection

Introduction

Genetic programming (GP) is a popular and powerful evolu-tionary optimization algorithm that solves user-defined tasksby the evolution of computer programs [7,10,23]. It hasattracted increasing attention from researchers in variousresearch fields and a number of enhanced GP variants havebeen proposed recently, such as Cartesian genetic program-ming [14], Semantic Genetic Programming (SGP) [4,9,15],Grammatical evolution (GE) [16], Gene Expression Pro-gramming (GEP) [8,22,23], Linear Genetic Programming(LGP) [3] and Multiple Regression Genetic Programming(MRGP) [1]. So far, GP and its variants have been applied to anumber of practical applications, including symbolic regres-sion [24], classification [7], time series prediction problem[23] and program synthesis [11,20].

B Jinghui Zhongjinghuizhong@scut.edu.cn

1 South China University of Technology, Guangzhou, China

2 Henan Normal University, Xinxiang, China

In GPs, a solution is constructed by combining a set ofprimitives consisting of functions and input features. Thus,without exploiting the structure information in advance, tosome extent, the searching process is merely recombinationwith the given building blocks.Most existingGPs [4,8,14,16]search solutions in the entire search space directly withoututilizing the relationships among input features to acceleratethe search. As a result, they often suffer from low searchefficacy on complicated symbolic regression problems.

In practical applications, however, the physical system thatgenerates the observed data usually can be decomposed intoa number of separable sub systems. Accordingly, the analyt-ical function to model the system can be decomposed intoa number of separable sub functions. By searching the subfunctions in smaller sub spaces first and then building thefinal analytical function in a bottom-up manner, the solutionstructure can be fully exploited and the search efficacy couldbe improved significantly. Inspired by this, Luo et al. [12]recently proposed a block building programming (BBP) forsymbolic regression. The BBP adopted a separability detec-tionmethod to judge whether the target function is separable.

123

1186 Complex & Intelligent Systems (2021) 7:1185–1194

Then, based on the detection results, the original target func-tion is divided into several blocks and further into factors. Thefactors are then modeled by an optimization engine such asGP. However, to accomplish the separability detection task,the BBP requires the system being able to generate desiredtraining data that satisfy certain constrains, which limitsits application scopes. Udrescu and Tegmark [19] recentlyalso developed a symbolic regression system (named AIFeynman) with a separability detection method. However,the separability detection method in AI Feynman requires ahighly accurate neural network to predict the output of giveninput features, which is hard to construct. Besides, the sep-arability detection method in AI Feynman has not yet beenintegrated with GP to improve the performance of GP.

To address the aforementioned issues, this paper pro-poses an efficient Genetic Programming with SeparabilityDetection (SD-GP) for symbolic regression. In the proposedSD-GP, the separability detectionmethod in BBP is extendedto detect additive separable characteristic of the target model.Besides, by using a gaussian process surrogate model, theproposed method does not require the system be able togenerate new data for separability detection, which is moreflexible for practical use. Then, based on the separabilitydetection results, a chromosome representation is proposed toencode solutions. The proposed chromosome representationconsists of multiple sub chromosomes. These sub chromo-somes generally consist of two kinds: the first kind is simplyencoded by all the input features, while the second kind isthe separated chromosomes that are encoded by the separablefeatures. Notably, the chromosome encoded by all the inputfeatures is still maintained so as to enhance the stability ofour method, considering that there can be some ground truthequation having no separable features. The final solution isa weighted sum of all sub functions, and the weights of subfunctions are optimized using the least squares method. Inthis way, the proposed SD-GP can hopefully attain optimalsolution efficiently and can ensure the global search abilityeven when the separability detection is not accurate enough.

The rest of the paper is organized as follows. “Preliminar-ies” presents related background techniques. “The proposedalgorithm” describes the proposed SD-GP algorithm, fol-lowed by the experiment study in “Experiments and results”.Finally, “Conclusion”draws the conclusions and futurework.

Preliminaries

In [12], Luo et al. proposed a separability detection techniquenamed as the Bi-Correlation Test (BiCT). In this paper, wefurther extended the BiCT to detect the additive separablecharacteristics of input features in a more flexible manner.To facilitate reader better comprehend ourmethod,we brieflydescribe the BiCT technique in this section. Generally, the

BiCT aims to detect whether the given system is partiallyseparable by calculating the correlation coefficient of certainsample data generated by the system.

Definition 1 A scalar function f (x) with n continuous vari-ablesx = [x1, x2, . . . , xn] ( f : Rn �→ R, x ∈ R

n) is partiallyseparable if and only if it can be rewritten as

f (x) = c0 ⊗ ϕ1(x1) ⊗ ϕ2(x2) ⊗ · · · ⊗ ϕc(xc) (1)

where c0 is a constant, xi represents a partition of the vectorx that contains ni variables, ϕi is a scalar sub function (ϕi :Rni �→ R), and the binary operator⊗i can be plus (+), minus

(−) and times (∗), c is the number of separable partitions.

In order to test whether xtest containing a certain part ofthe input variables can be separated from the input vector x toform xi in Eq. (1), BiCT first constructs four input matricesXA,XB,XC andXD for sampling. The structures of matricesXA and XB are described in (2).

XA = (Xfixed,Xrandom_A)

=

⎛⎜⎜⎜⎜⎝

x (fixed)1,1 · · · x (fixed)

1,m · · · x (random_A)1,n

x (fixed)2,1 · · · x (fixed)

2,m · · · x (random_A)2,n

.... . .

.... . .

...

x (fixed)N ,1 · · · x (fixed)

N ,m · · · x (random_A)N ,n

⎞⎟⎟⎟⎟⎠

XB = (Xfixed,Xrandom_B)

=

⎛⎜⎜⎜⎜⎝

x (fixed)1,1 · · · x (fixed)

1,m · · · x (random_B)1,n

x (fixed)2,1 · · · x (fixed)

2,m · · · x (random_B)2,n

.... . .

.... . .

...

x (fixed)N ,1 · · · x (fixed)

N ,m · · · x (random_B)N ,n

⎞⎟⎟⎟⎟⎠

(2)

where m and n are the variable numbers of vector xtest =(x1, x2, . . . , xm) and x = (x1, x2, . . . , xn), and N is thenumber of samples. The values of the first m columnswhich correspond to the sampling values of the variablesin xtest, are the same in both XA and XB and make upthe fixed matrix Xfixed. The rest n − m columns whichrepresent the samples of remaining variables in x, arerandomly sampled and form random matrices Xrandom_A

and Xrandom_B, respectively. The structures of matricesXC and XD are similar to XA and XB, but the ran-dom matrices in XC and XD correspond to the variablesin xtest, while the fixed matrix represents the samples of theremaining variables in the vector x.

The corresponding sampled outputs of XA, XB, XC andXD are represented as YA, YB, YC and YD, respectively.Then, the correlation coefficient rAB between YA and YB isestimated as follows:

rAB = 1

N − 1

N∑i=1

ai − a

σa· bi − b

σb(3)

123

Complex & Intelligent Systems (2021) 7:1185–1194 1187

where N is the number of samples in YA and Yb, ai and biare the i-th values inYA andYb, a and b are the samplemeanvalues of YA and Yb, and σa and σb are the sample standarddeviations of YA and Yb. Similarly, rCD can be obtainedbased on YC and YD. If both rAB and rCD are both equalto 1, then it can be judged that the variables in xtest can beseparated from the input variables as a whole and the originaldata are partially separable.

Based on the above descriptions, we can find that theBiCTtechnique could not distinguish whether the system is sepa-rable in the form of addition or multiplication. Besides, theBiCT requires that the system can provide the outputs of ran-domly generated input features (i.e., XA, XB, XC and XD).This requirement is usually not easy to satisfywhen the givenproblem only provides a fixed set of data for training.

The proposed algorithm

In this section, we first describes the proposed surrogate-assisted additive separability detection method named asSurrogate-assisted Additive Bi-Correlation Test (SAA-BiCT). Then, the proposed genetic programming with SAA-BiCT is presented.

The surrogate-assisted additive bi-correlation testmethod

Although BiCT can effectively detect the separability fromthe observed data automatically, there are two shortcom-ings that limit its application. First, BiCT can only detectthe features with separability, but is disable to distinguishwhether the separable features are additive or multiplicative.For example, variables in f1(x1, x2, x3) = x1 + x2 + x3 andf2(x1, x2, x3) = x1 ∗ x2 ∗ x3 are considered to be partiallyseparable when tested by BiCT, but the former function isadditive combination and the latter is multiplicative combi-nation. Second, if the problemonlyprovides afixed set of datafor training, the special sampling input and output matricesfor detection may not be available, which makes it inappli-cable. To overcome the aforementioned issues, we proposedan enhanced additive separability detection method namedSAA-BiCT.

In the SAA-BiCT, the same method as mentioned inSect. 2 is utilized to construct input matrices XA, XB, XC

andXD. To avoid the limitation caused by insufficient data inthe original data set, we adopt a Gaussian process regression(GPR) as the surrogate model to predict the output matri-ces of the four input matrices. Specially, in this paper, theSquared Exponential with Periodic Element is adopted as thekernel function of the GPR model, which can be expressedas follows:

k(x, x′) = σ 2

f exp

[−(x − x

′)2

2l2

]+ σ 2

n δ(x, x ′), (4)

where σ f , l and σn are hyper-parameters of the kernel func-tion. The JADE [21] is adopted as the solver to optimize thehyper parameters of the GPR model.

After generating the input and output matrices, the pro-posed SAA-BiCT does not directly use their correspondingoutput matrices to calculate the linear correlation coeffi-cients. Assuming the data dimension is n, and x1, x2, . . . , xmare tested variables and xtest = (x1, x2, . . . , xm), we definethe matrices �test and �rest as follows:

�test =

⎛⎜⎜⎜⎝

ζ(xtest(1))ζ(xtest(2))

...

ζ(xtest(N ))

⎞⎟⎟⎟⎠ , �rest =

⎛⎜⎜⎜⎝

ζ(xrest(1))ζ(xrest(2))

...

ζ(xrest(N ))

⎞⎟⎟⎟⎠ , (5)

where N is the number of samples,xrest =(x1, x2, . . . , xn−m)

is a vector made up of the remaining input variables, and ζ

is a nonlinear function. Then matrices Y′A, Y

′B, Y

′C and Y

′D

can be obtained by

Y′A = YA + �test, Y

′B = YB + �test,

Y′C = YC + �rest, Y

′D = YD + �rest.

(6)

We use the newly constructed Y′A and Y

′B to calculate the

correlation coefficient rAB and Y′C, and Y

′D to calculate rCD

by Eq. (3). If the results of rAB and rCD are 1, the variablesin xtest can be considered to be additively separable.

Based on the above mechanism, the proposed SAA-BiCTcan distinguish the additive separable structures from theoriginal data model. SAA-BiCT applies an additive sepa-rability factor to the process of testing, so that it can only testthe model from the perspective of additive separability. Forexample, function f (x1, x2, x3, x4) = x1 ∗ x2 ∗ x3 ∗ x4 is amultiplicative model. When SAA-BiCT tests whether x1 canbe additively separable, a factor ζ(x1) is added to the origi-nal model and extends the function to be f (x1, x2, x3, x4) =x1 ∗ x2 ∗ x3 ∗ x4 + ζ(x1). Then by calculating the correlationcoefficients, SAA-BiCT can judge that x1 is not separable.It is worth pointing out that different ζ forms can lead todifferent detection effects. In this paper, we adopt a simpleζ function as shown in (7) which works well based on ourempirical test.

ζ(x) =∑xi∈x

x2i (7)

123

1188 Complex & Intelligent Systems (2021) 7:1185–1194

Fig. 1 General framework of SD-GP

Genetic programming with separability detection

To harness the separability characteristic of the given prob-lem to improve the search performance, the proposed SD-GPadopts a multi-chromosome representation method. In SD-GP, each sub chromosome represents a sub function of themodel to be solved, and the final model is the weighted sumof all sub functions. Figure 1 illustrates the general frame-work of the proposed SD-GP. It first uses the SAA-BiCT todetect the additive separability of the observed model. Then,the original input variables are disassembled for each subchromosome according to the additive separation features.The least squares estimator method is utilized to optimizethe weight of each sub function. The population of chromo-somes is repeatedly evolved using genetic operators such ascrossover, mutation and selection, until the termination con-dition is met.

When testing separability on the observed data, travers-ing all combinations takes a lot of time. However, it is notnecessary to perform the SAA-BiCT on all input variablecombinations one by one under some circumstances. Forinstance, if the input variables are x1, x2, x3, there are 7combinations, which are (x1), (x2), (x3), (x1, x2), (x1, x3),(x2, x3) and (x1, x2, x3). A data set that contains n input vari-ables containsC1

n +C2n +· · ·+Cn

n combinations of variables.As the number of variables increases, the computationaltime will increase dramatically. Whereas, in SAA-BiCT,

Fig. 2 An example of decoding a chromosome

we only need to test (x1), (x2) and (x3) and totally testC1n + C2

n + · · · + C�n/2n times to achieve the same detec-

tion effect. In this paper, we aim to reduce the number ofvariables in each separated part so as to reduce the difficultyof searching the sub functions. Therefore, the proposed SAA-BiCT starts with combinations with fewer variables. That is,each single variable is tested at the beginning, and then thecombinations of them. In addition, all variables to be testedare put into a vector before detection. Once variables or theircombinations is separable, these variables are removed fromthe vector. If only combinations formedby the variables in thevector are detected, unnecessary detections can be avoidedand the separable feature set can be obtained.

The encoded solution I of a chromosome in SD-GP isobtained by

I = β0 + β1C1 + β2C2 + · · · + βkCk, (8)

where Ci is one of the sub chromosomes, βi is the weightcalculated by the least squares estimator method and k isthe number of sub chromosomes. In this paper, we adoptsthe fixed-length gene expression encoding method to encodeeach sub chromosome Ci .

Basic structure and implementation details ofgenetic programming

The basic structure of the gene expression chromosome con-tains a Head section and a Tail section. The Head section cancontain both function symbols (e.g., +, ∗, sin) and terminalsymbols (e.g., x1, x2), while the Tail section can only containterminal symbols. To ensure the whole chromosome can bedecoded correctly, the length of Head (h) and Tail (t) mustsatisfy t = h ·(u−1)+1, where u is the number ofmaximumoperands of function symbols.

123

Complex & Intelligent Systems (2021) 7:1185–1194 1189

Table 1 Synthetic problems Category No. Function Data set

I 1 y = x1x22 + x3 + x4x5 U [−2, 2, 100]2 y = 0.8x1x2 + 0.95x3 + 0.4x24 + 0.1x5 U [0, 4, 100]3 y = x1x2x3 + x4x5 + x6x7 U [−2, 2, 100]4 y = 1.78x1x2 + 2.45x3x4 + 6.2x5x6 + 4.98x7x8 U [−2, 2, 100]

II 5 y = 2.5x21 + 3.1x22 + 4.5x23 + 1.7x24 + 2.8x25 + 5 U [−2, 2, 100]6 y = (x1 − x2)2 + (x3 − x4)2 + · · · + (x7 − x8)2 U [−2, 2, 100]7 y = ∑8

i=1 x2i U [−2, 2, 100]

8 y = ∑5i=1 x

3i U [−2, 2, 100]

9 y = (x1 + x2)2 + x23 x4 + x5x26 U [−2, 2, 100]III 10 y = sin(x1) + sin(x2 + x3) + x4 U [0, 4, 100]

11 y = sin(x1) + cos(x2) + sin(x3) + cos(x4) + sin(x5) U [−2, 2, 100]12 y = ex1 + ex2 + ex3 + ex4 + ex5 U [−2, 2, 100]13 y = ex1 + x2x23 + x24 U [−2, 2, 100]14 y = x21 + ex2 + x3x4 + x25 U [−2, 2, 100]15 y = ln(x1) + x2x3 + ln(x4 ∗ x5) U [0, 4, 100]

The process of using a chromosome to represent a math-ematical function is as shown in Fig. 2. Given a string-basedrepresentation in Fig. 2, one should treat the first operatoras the root node in encoding tree. Then, the rest symbolsin the string are traversed in a breath-first-search fashion.That is, + and x1 should be the children of root node −, and∗ and sin should be the children of the second symbol +,and so on. With the constructed encoding tree, the equationx1 ∗ x1 + sin(x2) − x1 can be obtained naturally.

The generation of sub chromosomes generally dependson the obtained additive separation features. Each separablegroup of variables is assigned to a sub chromosome, and thesub chromosome can only utilize the separable group of vari-ables to construct sub function. To ensure the global searchability of the algorithm, an additional sub chromosome isadded to each chromosome, which can utilize all variablesinvolved to construct sub function. In SD-GP, the minimumnumber of sub chromosomes Kmin (e.g., Kmin = 5) is set inadvance. When the number of separable variable sets foundby SAA-BiCT is smaller than Kmin , the remaining sub chro-mosomes can utilize all variables to construct sub functions.

TheSD-GPadopts SL-GEP [22] as theGP solver to evolvethe above multi-chromosomes. The SL-GEP is a recentlypublished GP variant which can efficiently search solutionby using extended genetic operators borrowed from differ-ential evolution (DE) [17,21]. It should be noted that theSL-GEP adopts a gene expression chromosome representa-tion with automatically defined functions (C-ADFs). In thispaper, we only borrow the genetic operators of SL-GEP toevolve the proposed chromosomes. Generally, the evolution-ary process in SL-GEP is composed of fourmain steps,whichare initialization, mutation, crossover and selection. In the

initialization, according to the chromosome structure, cor-responding symbols are randomly assigned to the Head andTail sections to form the initial chromosomes. Then the chro-mosomes are iteratively evolved throughmutation, crossoverand selection, until the termination condition is satisfied. Inthis paper, the algorithm will terminate when a pre-definedmaximum number of generations is reached or a pre-definedsatisfying solution is found.

Experiments and results

This section investigates the performance of the proposedSD-GP algorithm. First, the experimental settings, includingthe synthetic test problems, the compared algorithms and theparameter settings of all algorithms are described. Then, thecomparison results are presented. Finally, the effectivenessof the separability detection of SAA-BiCT is discussed.

Experiment settings

To test the effectiveness of the proposed SD-GP for solvingproblems with separable additive structure, a synthetic prob-lem set containing 15 symbolic regression problems withdifferent features is designed for testing, as listed in Table 1.The problems can be divided into three categories: CategoryI is made up of basic multivariate problems. The problemsin Category II have more complex forms with more squareor even cubic structures, which are harder to be solved. Cat-egory III contains more function primitives, including sin,cos, exp and ln. The format U [a, b, N ] in column Data Set

123

1190 Complex & Intelligent Systems (2021) 7:1185–1194

Table 2 Important parametersetting of SD-GP

Parameter description Value

The population size (P) 50

The maximum number of generations (G) 20,000

The set of function symbols (Ψ ) {+,−, ∗, /, sin, cos, exp, ln}The mutation factor of DE in SL-GEP (F) rand(0,1)

The crossover factor of DE in SL-GEP (CR) rand(0,1)

The sample size of separability detection (S) 20

Table 3 Comparison results Category No. S-SLGEP SMGGP GPTIPS-2 SD-GPSuccess Success Success Success SAA-BiCT

I 1 5 30 21 30 tc

2 0 30 20 30 tc

3 1 30 27 30 tw

4 0 30 30 30 tc

II 5 0 9 14 30 tc

6 0 0 0 7 tc

7 0 0 0 30 tc

8 0 0 0 6 tc

9 0 0 7 24 tc

III 10 10 30 29 30 pc = (x1, x4)

11 1 30 28 30 pc = (x1, x3, x4, x5)

12 29 30 25 30 tc

13 1 30 20 30 tc

14 0 30 27 30 tc

15 30 30 29 30 pc = (x1, x2, x3)

a All algorithms are run 30 times for each synthetic problem and column Success records the total successfultimes.b Column SAA-BiCT records the additive separability detection results of SD-GP. The used abbreviations: “tc”means totally correct on all variables, “pc=(xi , . . . )” means partly correct on variables xi , . . . , “tw” meanstotally wrong on all variables

means that the number of samples is N and each input vari-able is randomly sampled in the interval (a, b).

Tobetter evaluate the performanceof the proposedSD-GP,we choose three other GP variants for comparison. The firstalgorithm is amodified SL-GEP [22] (labelled as S-SLGEP),which removes the ADFs in chromosome representation anduses least squares estimatormethod to create constants. Sincethe S-SLGEP is adopted as the fundamental component inthe proposed method, comparison with S-SLGEP indicatesthe effectiveness of the proposed preprocessing component,SAA-BiCT.

The second algorithm is a multi-gene GP (SMGGP),which adopts the chromosome representation and geneticoperators in SD-GP but abandons the separability detectiontechnique. Comparison with SMGGP serves as componentanalysis to indicate the efficacy of the separability compo-nent. The last algorithm is theGenetic ProgrammingToolboxfor the Identification of Physical Systems version 2 (GPTIPS-2) [18], a widely used open source genetic programming

toolbox for multi-gene symbolic regression on MATLABplatform [2,5,6,13]. SMGGP and GPTIPS-2 are multi-genealgorithms, but unlike SD-GP, they do not use the knowl-edge obtained from the observed data to assist the searchingprocess, so the total number of their sub chromosomes (K )is fixed in advance. In our experiments, the parameter Kof SMGGP and GPTIPS-2 is set to 5. For SD-GP, we setthe minimum number of sub chromosomes Kmin = 5. Thelength of the string representing the sub chromosome (L) inSD-GP and SMGGP is 21. Because S-SLGEP is a single-chromosome GP algorithm, so to enhance its expressionability, its chromosome can contain more symbols than otherthree algorithms and its L is set to 81. For GPTIPS-2, themaximum depth (Dmax) of the GP tree is set to 4. In addi-tion, the fitness value V of all algorithms is calculated by

V = 1

N

N∑i=1

(yi − oi )2, (9)

123

Complex & Intelligent Systems (2021) 7:1185–1194 1191

Fig. 3 Comparison onconvergence speeds of problem1–9

123

1192 Complex & Intelligent Systems (2021) 7:1185–1194

Fig. 4 Comparison on convergence speeds of problem 10–15

where yi is the output value calculated by the indicate chro-mosome and oi is the real output value in the observed dataset. The common parameters of SD-GP and S-SLGEP areset the same as suggested in SL-GEP [22]. Table 2 lists theimportant parameter settings of the proposed SD-GP.

Results and discussion

For each test problem, we run all algorithms for 30 indepen-dent times and count the total successful times for solvingproblems correctly. The experimental results are recorded inTable 3. Column SAA-BiCT in SD-GP shows whether SAA-BiCT is able to detect all additive separable features correctlyfrom the input variables.

123

Complex & Intelligent Systems (2021) 7:1185–1194 1193

Table 4 Additive separability test for misjudgment

Function BiCT SAA-BiCT

f = x1x2(x3 + x4) x2 –

f = x1x2x3x4 x3, x4 –

f = sin(x1 + 0.5)cos(x2 + x3) + 1 x2, x3 x2, x3

f = 0.6x1x2x3x4 + 1.2 x3, x4 –

The experimental results show that the proposed SD-GPperforms much better than other compared algorithms. Ingeneral, SMGGP, GPTIPS-2 and SD-GP, which adopt multi-gene encoding technique, perform significantly better thanS-SLGEP which uses only a single chromosome. Amongthe three multi-gene algorithms, SD-GP performs the best,especially when SAA-BiCT can correctly detect all the addi-tive separable features of the observed data. It is difficult forSMGGPandGPTIPS-2 to solvemultivariate problemswhichcontain many additive high-power structures, such as prob-lems in Category II. However, with the detected knowledge,SD-GP can search efficiently on these problems.

In SD-GP, the number of sub chromosomes is deter-mined by the number of separable features detected. In thisway, SD-GP can adaptively adjust the chromosome size soas to fit the complexity of the problem. Meanwhile, bothSMGGP and GPTIPS-2 use a fixed size of sub chromosomesto solve all problems, which is less flexible than the pro-posed SD-GP. For example, when they solve Problem 11that requires 8 sub chromosomes, they cannot automaticallyincrease the number of sub chromosomes, thereby weaken-ing their problem-solving ability. Owing to SAA-BiCT, theSD-GP can overcome the above weakness. It should be notedthat even when SAA-BiCT fails to detect correct separablefeatures, SD-GP still has promising anti-interference abilityto correct errors, just as what happened during the solutionof Problem 3, Problem 10, Problem 11 and Problem 15. OnProblem 10, SAA-BiCT wrongly considers that both x2 andx3 can be separated independently, but in reality they are onlyseparable as a whole. However, since SD-GP uses extra subchromosomes to contain all variables and limits the mini-mum number of sub chromosomes to 5, the separable partsin(x2 + x3) can be found by additional sub chromosomessuccessfully.

In order to further test the efficiency of SD-GP in findingcorrect solutions, we plot the convergence graphs of fitnessvalues on nine canonical problems in Fig. 3, and six prob-lems with relatively complex primitives in Fig. 4. The fitnessvalues are obtained by Eq. (9). Thus the closer the fitness isto zero, the better the solution is. The results in Fig. 3 andFig. 4 show that SD-GP always converges the fastest amongthe compared algorithms, which indicates that the proposedmethod is effective to improve the search efficiency of GP.

Finally, we test the additive separability detection capabil-ity of SAA-BiCTandBiCT.The variables that aremistakenlyjudged to be additive separable are listed in Table 4. It can beobserved that BiCT has misjudgments on all multiplicativefunctions, which donates that it has no discerning abil-ity to the multiplicative separation and additive separation.Meanwhile, SAA-BiCT can correctly identify the additiveseparability features on most cases. It is worth pointing outthat the detection accuracy of SAA-BiCT is limited by theprediction accuracy and the calculation error, so misjudg-ments may occur sometimes.

Conclusion

In this paper, we proposed a genetic programming with sep-arability detection technique (SD-GP) to improve the searchefficacy of GP on symbolic regression problems with sepa-rable structures. SD-GP uses a separability detection methodcalled SAA-BiCT to discover the additive separable featuresin the observed model, and then it generates and acceleratesthe evolution of sub chromosomes with the obtained knowl-edge. Experiments have indicated that SD-GPhas advantagesin problemswith seperable structures compared to S-SLGEP,SMGGP and the widely used MATLAB toolbox GPTIPS-2.There are several future research directions. First, the pro-posed method can be further improved by designing a moreaccurate prediction method to assist the separability detec-tion. Second, different forms of ζ in Eq. (7) can be tried toachieve better separability detection results. Hereafter, withthe separability detection component, the proposed algorithmcan be further extended to high-dimensional scenario byreducing the computation overhead. In addition, the proposedalgorithm framework can be further improved by incorporat-ing other useful knowledge learned from the observed data,such as symmetry characteristic of input features.

Acknowledgements This work is partially supported under the KeyProject of Science and Technology Innovation 2030 supported by theMinistry of Science andTechnologyofChina (Grant no. 2018AAA0101303), the National Natural Science Foundation of China (Grant no.62076098), and the Fundamental Research Funds for the CentralUniversities (Grant no. D2191200).

Funding This work was supported in part by the Key Project of Scienceand Technology Innovation 2030 supported by the Ministry of Scienceand Technology of China (Grant no. 2018AAA0101303) in part by theNational Natural Science Foundation of China (Grant no. 62076098),and the Fundamental Research Funds for the Central Universities(Grant no. D2191200).

Data availibility statement The data sets used and /or analyzed dur-ing the present study are available from the corresponding author onreasonable request.

123

1194 Complex & Intelligent Systems (2021) 7:1185–1194

Compliance with ethical standards

Conflict of interest The authors declare that they have no competinginterests.

Code availability If published, the custom code described in thepaper can be downloaded fromhttps://www.jianguoyun.com/p/DQ5TLQUQrdfiCBj869ED.

Open Access This article is licensed under a Creative CommonsAttribution 4.0 International License, which permits use, sharing, adap-tation, distribution and reproduction in any medium or format, aslong as you give appropriate credit to the original author(s) and thesource, provide a link to the Creative Commons licence, and indi-cate if changes were made. The images or other third party materialin this article are included in the article’s Creative Commons licence,unless indicated otherwise in a credit line to the material. If materialis not included in the article’s Creative Commons licence and yourintended use is not permitted by statutory regulation or exceeds thepermitted use, youwill need to obtain permission directly from the copy-right holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

References

1. Arnaldo I, Krawiec K, O’Reilly UM (2014) Multiple regressiongenetic programming. In: Proceedings of the 2014 annual confer-ence on genetic and evolutionary computation, GECCO ’14, pp.879–886. Association for Computing Machinery, New York, NY,USA (2014). https://doi.org/10.1145/2576768.2598291

2. Astarabadi SSM, Ebadzadeh MM (2019) Genetic programmingperformance prediction and its application for symbolic regressionproblems. Inf Sci 502:418–433

3. Brameier MF, Banzhaf W (2007) Linear genetic programming.Springer, Berlin

4. Castelli M, Vanneschi L, Silva S (2014) Semantic search-basedgenetic programming and the effect of intron deletion. IEEE TransCybern 44(1):103–113

5. D’Angelo G, Palmieri F (2020) Knowledge elicitation basedon genetic programming for non destructive testing of criticalaerospace systems. Future Gen Comput Syst 102:633–642

6. D’Angelo G, Pilla R, Tascini C, Rampone S (2019) A proposal fordistinguishing between bacterial and viral meningitis using geneticprogramming and decision trees. Soft Comput 23(22):11775–11791

7. Espejo PG,Ventura S, Herrera F (2010)A survey on the applicationof genetic programming to classification. IEEE Trans Syst ManCybern Part C (Applications and Reviews) 40(2):121–144

8. Ferreira C (2001) Gene expression programming: a new adaptivealgorithm for solving problems. Complex Syst 13(2):p8–129

9. Ffrancon R, Schoenauer M (2015) Memetic semantic genetic pro-gramming. In: Proceedings of the 2015 annual conference ongenetic and evolutionary computation, pp 1023–1030. ACM

10. Koza JR (1992)Genetic Programming: vol. 1, On the programmingof computers by means of natural selection, vol. 1. MIT press

11. LangdonWB,HarmanM(2015)Optimizing existing softwarewithgenetic programming. IEEE Trans Evol Comput 19(1):118–135

12. Luo C, Chen C, Jiang Z (2017) A divide and conquer method forsymbolic regression. arXiv e-prints arXiv:1705.08061

13. Mehr AD, Jabarnejad M, Nourani V (2019) Pareto-optimal mpsa-mggp: a new gene-annealing model for monthly rainfall forecast-ing. J Hydrol 571:406–415

14. Miller JF, Thomson P (2000) Cartesian genetic programming.Genetic programming. Springer, Berlin, pp 121–132

15. Moraglio A, Krawiec K, Johnson CG (2012) Geometric semanticgenetic programming. International conference onparallel problemsolving from nature. Springer, Berlin, pp 21–31

16. O’Neill M, Ryan C (2001) Grammatical evolution. IEEE TransEvol Comput 5(4):349–358

17. Price K, Storn R, Lampinen J (2005) Differential evolution: a prac-tical approach to global optimization, natural computing series.Springer, Berlin

18. Searson DP (2015) GPTIPS 2: an open-source software platformfor symbolic data mining. Springer, Cham, pp 551–573. https://doi.org/10.1007/978-3-319-20883-1_22

19. Udrescu SM, Tegmark M (2020) Ai feynman: a physics-inspiredmethod for symbolic regression. Sci Adv 6:16. https://advances.sciencemag.org/content/6/16/eaay2631

20. Weise T, Wan M, Tang K, Yao X (2014) Evolving exact integeralgorithms with genetic programming. In: 2014 IEEE congress onevolutionary computation (CEC), pp 1816–1823

21. Zhang J, Sanderson AC (2009) Jade: adaptive differential evo-lution with optional external archive. IEEE Trans Evol Comput13(5):945–958

22. Zhong J, Ong YS, Cai W (2016) Self-learning gene expressionprogramming. IEEE Trans Evol Comput 20(1):65–80

23. Zhong J, Feng L, Ong Y (2017) Gene expression programming: asurvey [review article]. IEEE Comput Intell Mag 12:54–72

24. Zhong J, Feng L, Cai W, Ong Y (2018) Multifactorial genetic pro-gramming for symbolic regression problems. In: IEEE transactionson systems, man, and cybernetics: systems, vol. 50, no. 11, pp.4492–4505, Nov. 2020

Publisher’s Note Springer Nature remains neutral with regard to juris-dictional claims in published maps and institutional affiliations.

123