ELSEVIER Fuzzy Sets and Systems 93 (1998) 223 230

FUZ2¥ sets and systems

Universal approximation by hierarchical fuzzy systems Li-Xin Wang

Department of Electrical and Electronic Enyineerin 9, The Hong Kon.q University of Science and Technology, Clear Water Bay, Kowloon, Hon 9 Kong

Received January 1996; revised July 1996

Abstract

A serious problem limiting the applicability of standard fuzzy controllers is the rule-explosion problem; that is, the number of rules increases exponentially with the number of input variables to the fuzzy controller. A way to deal with this "curse of dimensionality" is to use the hierarchical fuzzy systems. A hierarchical fuzzy system consists of a number of hierarchically connected low-dimensional fuzzy systems. It can be shown that the number of rules in the hierarchical fuzzy system increases linearly with the number of input variables. In this paper, we prove that the hierarchical fuzzy systems are universal approximators; that is, they can approximate any nonlinear function on a compact set to arbitrary accuracy. Our proof is constructive, that is, we first construct a hierarchical fuzzy system in a step-by-step manner, then prove that the constructed fuzzy system satisfies an error bound, and finally show that the error bound can be made arbitrarily small. (~) 1998 Elsevier Science B.V.

Keywords: Control theory; Linguistic modelling; Approximation theory

1. Introduction

Although fuzzy control techniques have been successfully applied to a variety of problems, the applications are usually limited to systems with very few variables, say two to four. A fundamental limitation of standard fuzzy systems is that as the number of variables increases, the number o f rules increases exponentially.

Specifically, suppose there are n input variables and m fuzzy sets are defined for each variable, then the number of rules in the standard fuzzy system is m n, which is a huge number for large n. For example, with n - m = 5, which is quite common in practice, we have ran= 3125 rules. As the number of variables increases, the rule base will quickly overload the memory and make the fuzzy controller unimplementable. We must find some ways to deal with this rule-explosion problem if we want to apply fuzzy control techniques to more complex systems.

In fact, it is a common phenomenon that the complexity o f a problem increases exponentially with the number o f variables involved; this is not unique to fuzzy systems. This phenomenon was called by Bellman as the "curse of dimensionality [2]". Researchers in different fields have been studying this problem using different approaches. In statistics, the projection pursuit algorithm [8,9] was proposed which tries to find the most important variables one by one; other examples are recursive partitioning and related methods [4]. In mathematics, the famous Kolmogorov theorem [10,12] shows that every continuous function on [0, 1] n can

0165-0114/98/$19.00 (~) 1998 Elsevier Science B.V. All rights reserved PH S01 65-01 14(96)00197-2

224 L..X. Wang~Fuzzy Sets and Systems 93 (1998) 223-230

Y~ [fuzzysyste~

[fuzzysyste~

T,, Xl x2 x 3 x4

Yn-I=Y

[fuzzy systen]

Yn-2

X n

Fig. 1. An example of n-input hierarchical fuzzy system which comprises n - 1 two-input fuzzy systems.

be represented as the additive superposition of continuous one-dimensional functions. In neural networks and wavelets, this topic has been extensively addressed [1,3,7]. In signal processing and control, the classical principal component method is an example. In fuzzy systems, we have the hierarchical fuzzy systems [15,16].

The hierarchical fuzzy systems were proposed by Raju et al. in [15] as a way to deal with the "curse of dimensionality" for fuzzy control applications. Their hierarchical fuzzy system consists of a number of low- dimensional standard fuzzy systems connected in a hierarchical fashion. Fig. 1 shows a typical example of hierarchical fuzzy systems. We see that this n-input hierarchical fuzzy system comprises n - 1 low-dimensional fuzzy systems, with each low-dimensional fuzzy system having two inputs. If we define m fuzzy sets for each variable, including the internal variables y j . . . . , Y n - 2 , the total number of rules is (n - 1)m 2 which is a linear function of the number of input variables n. In general, it can be shown that the number of rules in the hierarchical fuzzy system increases linearly with the number of input variables (see Section 2). Hence, the hierarchical fuzzy systems provide a good candidate for solving high-dimensional problems.

It is well known that the standard fuzzy systems are universal approximators [5,20,22,23], thus they are suitable for a wide variety of applications. What is the approximation capability of hierarchical fuzzy systems? If they can only represent a restricted class of nonlinear functions, their applicability should be limited. Surprisingly, we will show in this paper that the hierarchical fuzzy systems are also universal approximators, despite their small rule bases and structure constraints.

In Section 2, we construct the hierarchical fuzzy systems and show some basic properties. In Section 3, we prove, in a constructive manner, that the hierarchical fuzzy systems are universal approximators. Section 4 concludes the paper.

2. Construction and basic properties of hierarchical fuzzy systems

The idea of a hierarchical fuzzy system is to put the input variables into a collection of low-dimensional fuzzy systems, instead of a single high-dimensional fuzzy system as in the usual case. The low-dimensional fitzzy systems are Takagi-Sugeno-Kang (TSK) fuzzy systems which were proposed in [17,18] as an alternative

L.-X. WanoI Fuzzy Sets and Systems 93 (1998) 223-230 225

to the standard fuzzy systems. The TSK fuzzy system is constructed from the following rules:

1 I F x l isA{ and - . . and Xn, is A;n~, THEN y i=h i (x l . . . . ,Xn,), (1)

where AJ are fuzzy sets, h~ are linear or nonlinear functions (e.g., polynormals), and l = 1,2 . . . . ,Mi. That is, the IF parts o f the rules are the same as in the ordinary fuzzy IF-THEN rules, but the THEN parts are linear or nonlinear functions of the input variables. Given an input xi = (x l , . . . ,x , , )T E U i C R ";, the output f i (x i ) E R of the TSK fuzzy system is computed as the weighted average of the h~'s in (1), i.e.

: , ( x ; ) - - E ,h:(xl . . . . . x°,)w: E~_-'l w: ' (2)

where the weights w[ are computed as

t / i

w: = l - ["A' (xJ) (3) j=l

Each low-dimensional TSK fuzzy system constitutes a level in the hierarchical fuzzy system. Suppose that there are n input variables xl . . . . . x~, then the hierarchical fuzzy system is constructed as follows.

Construction of general hierarchical fuzzy systems: • The first level is a TSK fuzzy system with nl input variables xl . . . . . xn,, that is, it is the f i (xi) of (2) with

i = 1 . • The ith level (i > 1) is a TSK fuzzy system with ni -1- 1 (ni >~ 1) input variables which are constructed

from the following rules:

IF XN,+l is A t a n d - . , and XN;+,; is A l and Yi-1 i s C[_I, Ni+ l Ni+ni l THEN Yi = hi(xu,+l,... ,XN,+n;, Yi-1 ), (4)

where Ni = ~ - I nj, and l = 1 ,2 , . . . ,Mi . According to (2) and (3), this TSK fuzzy system is

' .--N,+n, hi(XN;+l," " ,XNi+ni, Yi--1) 1 Ij=N,+I #AS(XJ )~Clt_ , (Yi--I )

f i (XNi+l, ' ' . ,XNi+ni, Yi-l ) = ~ 1 F[Ni+n" (5) l lj=Ni+l I~A;(Xj )ktC/_, (Y i - I )

• The construction continues until i = L such that ~ )=1 nj = n, that is, until all the input variables are used in one of the levels. We see that the first level converts nl variables xl . . . . . x,, into one variable Yl which is then sent to the

second level. In the second level, some other n2 variables xn,+l . . . . ,x,~+,~ and the variable Yl are combined into another variable Y2 which is sent to the third level. This process continues until all the variables x~,. . . ,x, are used.

A special case of the hierarchical fuzzy system is to choose n l = 2 and ni ---- 1 for i = 2, 3 . . . . . L. In this case, all the TSK fuzzy systems in the hierarchy have two inputs, and there are L = n - 1 levels. This special hierarchical fuzzy system is illustrated in Fig. 1.

We now show that the number of rules in the hierarchical fuzzy system is a linear function of the number of input variables (Theorem 1) and that the rule number reaches its minimum in the special case of Fig. 1 (Theorem 2).

Theorem 1. Let n, ni and L be the same as in the construction of general hierarchicalJuzzy systems. Suppose that m fuzzy sets are defined for each variable, includin9 the input variables xt . . . . . x, and the intermediate

226 L.-~ Wang~Fuzzy Sets and Systems 93 (1998) 223-230

variables yl . . . . . YL, and that the f u z z y system in the ith level ( i - 2 . . . . . L) is constructed f rom m n~+l rules (a complete rule set) and for the first level the rule number is m ~' . I f n l = ni + 1 --- c (constant) fo r i = 2, 3 . . . . , L, then the total number o f rules in the whole hierarchical f u z z y system is

m c M = (n - 1). (6)

e - 1

Proof. Obviously, we have

L

M = m n~ + Z mn'+l = LmC" i=2

L Since n = 2~=1 ni = c + Ei=2(c - 1) = Lc - L + 1, we have

L - - - ( n - 1 ) / c - 1

Substituting (8) into (7), we obtain (6). []

(7)

(8)

Since mC/(c - 1 ) is a constant, we see from (6) that the number of rules M in the whole hierarchical fuzzy system increases linearly with the number of input variables n. For the case o f m = 3, c -- 2 and n = 5, we have M = 324 = 36; if we use the standard fuzzy system, the number of rules is m n = 3 s = 243. The reduction of rules is even greater for larger m and n.

Theorem 2. Let the assumptions in Theorem 1 be true. I f m >~ 2, then the total number o f rules M of(6) is minimized when c=2, that is, when the T S K f u z z y systems in all the levels have two inputs as shown in Fig. 1.

The proof of this theorem can be found in [15].

3. Universal approximation by the hierarchical fuzzy systems

In this section, we prove that the hierarchical fuzzy systems are universal approximators. Our proof is constructive, that is, we first construct a hierarchical fuzzy system and then show that this hierarchical fuzzy system has the universal approximation property. For notational simplicity, we consider the three-input case; however, the approach can be generalized to the n-input case. We first specify the problem.

The Problem: Let g(x ) be a continuous function on the compact set U = [~t,fll] >( [~2, fl2] )< [CZ3,fl3] c R 3. Suppose that the analytic formula of g(x) is unknown, but for any x c U, we can obtain the value g(x) (that is, g(x) is a black-box). Our task is to design a hierarchical fuzzy system which approximates g(x).

We now design such a hierarchical fuzzy system in a step-by-step manner. Design o f a hierarchical f u z z y system:

• Step 1: Let the domains o f x i be [~i, fli] (i = 1,2,3) and the domain of yl be [0, 1] (we will design the first-level fuzzy system such that its output Yl E [0, 1]). Define m fuzzy sets A] . . . . ,A m in [~i, fli] with the following equal-spaced triangular membership functions:

~I (xi) -- ~ I (xi; e), e~, e~),

j--1 j j + l #A;(Xi)=PA~(xi;e i , e i , e i ), f o r j = 2 , 3 , . . . , m - - 1 , and

#A:~'(X/) = I~A~'(Xi; m--1 m e i , e i , e~ ),

(9)

(10)

(11)

L.-X. WangIFuzzy Sets and Systems 93 (1998) 223-230

C 4 ~ Y2

C 2

x 2 3 A 4 g ~ x2 / - " ~

2 " ~ ~ ~ A31 A3 2 A3 3 A3 4

A 2 1 ~ l~x I

F V V N Al I Al2 A[3 Ai4

Fig. 2. An example of fuzzy sets defined for each variable (the m = 4 case).

227

where i = 1,2,3, e { = ei + ( j - 1)bi with b i = (fli - o~i)/( m - 1), and the triangular membership functions are defined according to

(x - a ) / ( b - a) ,

g,~(x; a , b , c ) : (c - x ) / ( c b),

0,

a ~ x 4 b ,

b ~ x ~ c ,

x < a o r x > c .

(12)

Similarly, define m fuzzy sets C l . . . . . C m in [0, 1] with the equal-spaced triangular membership functions as in (9)-(11). Fig. 2 shows the case of m = 4. S t e p 2: Define the constants ~Pq = ( p + m ( q - 1))/m 2, where p , q = 1,2,. . . ,m. The TSK fuzzy system in the first level is designed as

m m Yl = fffxl,x2) = ~-~p=l ~q=l YPq[I2A~( xl )]2dq(X2)]

~-~ p=l m m ~ q = I [ ~ A ~ ' ( X l ) ~ A g ( X 2 ) ] ' (13)

which is a standard fuzzy system - a special case of the TSK fuzzy system (2). Since 0 < )Pq ~< 1 and f l ( X l , X 2 ) is a weighted average of the )sPq's, we have yl E [0, 1]. S t e p 3: Let the TSK fuzzy system in the second level be

m ~im--1 ~ j = l hiJ(Y 1 )[/2c' (Yl)#A~ (X3)]

Y2 = fz(yi,x3) = ~i"~-1 ~j=lm [l~c,(y I )#A~(X3) ] ,

where the functions hi j (y l ) are chosen as the following m - l 's order polynomials of YI:

(14)

hij(Yl ) = doij + d l i jY l + " " + d(m-l) i jY'~ - l (15)

228 L.-X. W a n o / F u z z y Sets and Sys tems 93 (1998) 223-230

and the. m 3 parameters doij . . . . . d(m_l)i j ( i , j = 1,2 . . . . . m) are determined as follows. Since ktA/(eJ ) = 1,

/L4~(eg) = 0 for k ~ j and at most two /~c,(Yl)'S will be nonzero for any yl E [0, 1], we have from (14)

that

- pq j -pq -pq -pq -pq f2(Yl , e 3 ) = h i j ( Y l )ktc'(Yl )+h( i+l) j (Yl )t2c,+~(Y~ ), (16)

for some i E { 1 , 2 . . . . . m 1}. Let f 2 ( f2 fq , e j ) P q j -- = 9 ( e I ,e2,e3) and substituting (15) into (16), we have

gteP e q eJ~ [,2Ci(f2fq)[doij "q- d l i j f f ; f q - ~ - . . . -~- d ( m _ l ) i j ( y f q ) m - l ] k 1~ 2' 3 : ~

-~-]2Ci+l ( y f q ) [ d o ( i + l ) j -~- d l ( i + l ) j Y f q - 4 - . . . q - d ( m _ l ) ( i + l ) j ( f ? f q ) m - l ] .

For j fixed and p ,q = 1,2 . . . . ,m, collect the m 2 equations of (17) into the matrix form

(17)

gj=~dj, (18)

1 1 j l J 1 m J m m J ~]T where the m 2 × 1 vectors gj [9(el,e2,e 3), 9(e~, • dj = . . . , e 2, e3 ) , . . , g ( e I , e 2 , e 3 ) , . . . , g ( e 1 , e 2 , e 3 JJ , = [d01j . . . . . d(m-l)lj . . . . ,domj . . . . ,d(m_l)mj] T, and the m 2 x m 2 matrix Wj is obtained from (17) accordingly. Compute the m 3 parameters doi/,. . . ,d(m-1)ij ( i , j = 1 ,2 , . . . ,m) from

dj = wj- lgj . (19)

Substituting the doij, . . . , d(,n-t)ij into (15) and the resulting hij(yl) into (14), we obtain the second-level TSK fuzzy system.

• Step 4: The overall hierarchical fuzzy system is obtained as

f ( X l , X2, X3 ) = f2 ( f l (Xl, X2 ), X3 ), (20)

where the f l and f2 are given by (13) and (14), respectively. We now show that the maximum difference between the unknown function 9 and the hierarchical fuzzy

system f designed above over U is bounded by a linear function of the lengths of the sampling intervals bi = ( f l i - ~ i ) / ( m - 1), and from this result we conclude that the hierarchical fuzzy systems are universal approximators.

Theorem 3. Let f ( x l , x2 ,x3 ) be the hierarchical f u z z y system & (20) and g(xl,x2,x3) be the unknown func- tion to be approximated. I f 9(xl ,x2,x3) is continuously differentiable on U = [~h/~l] × [c%f12] x [(x3, f13], then

[[Y-f[[oo <, ~ 09 + ~ oo bi, i=1

where the infinite norm [l*]loo is defined as Ila(x)lloo = SUpx~v la(x)l, and bi = ( f l i - o ~ i ) / ( m - 1).

(21)

Proof . First, we show that f(ePl, q J P q J e2, e 3 ) = g(e l , e2, e 3 ) for p, q, j ---- 1,2 . . . . , m. According to the member- ship functions defined in Step 1, we have from (13) that f~(eP, e q) = yfq for p ,q = 1 ,2 , . . . ,m. Since the

doij . . . . • d (m- l ) i j determined from (19) guarantee (17) which in turn implies f2(PPlq, e j ) = ff(e p ,e2, q e 3 j ) , we have P q J q J -Pq J P q J f ( e l ,e2,e3 ) = f2[f l (e~,e2 ),e3] =fe(Y l ,e3) = 9(el ,e2,e3).

L.-X. Wan O I Fuzzy Sets and Systems 93 (1998) 223~30 229

Let uPqJ=[eP, e p+l ] × t~2, ~2roq ,,q+l~j × [e~, eJ+l], where p, q , j = 1,2, . . . , m - 1 . Since [~i, fli] = [e~, e2i] U [e2i, e 3 ] U m - - I in . . . U [ e i ,e i ], i = l , 2 , 3 , w e h a v e

m--I m--1 m--I

U U U p = l q = l j = l

(22)

which implies that for any x E U, there exists U pqj such that x E U pqj. Now let x be an arbitrary point in U, so there exist fixed numbers Px, qx,jx E {1,2 . . . . . r n - 1} (corresponding to the x) such that x E U p'q'j ' . Let e x = toP,- oq, ,S, )T. Using the mean-value theorem and the fact that f ( e X ) = 9(e x), we have ~ 1 ' ~2 ' ~3

3 ( ~ Ix~ 0f ojx, ) [g(x) - f ( x ) [ ~< Ig(x) - g(eX)[ + I f (x ) - f(eX)[ <~ ~ - e;l + - e~] , i=1

(23)

where ei ~ is the ith element of e x. Since x E U p'q'j', we have I x i - eX[ <<. bi, and hence

Ig(x) - f ( x ) [ ~< ~ ~ + bg. i=1 oo

(24)

Since the x in (24) is an arbitrary point in U, we obtain (21) from (24). []

Using the error bound (21), we obtain the following universal approximation theorem.

Universal approximation theorem. For any continuous function y (x) on the compact set U C R 3 and arbi- trary e, > O, there exists a hierarchical fuzzy system in the form o f (20) such that

sup I9(x) - f(x)l < e. (25) xEU

Proof. Since y(x) and f ( x ) are continuous functions and U is compact, ]lS9/Oxil]~ and I]Of/Oxil[~ are finite

numbers. Hence, by choosing bi sufficiently small we can make ~ = l ( l l a g / a x i t l ~ + II~f/Ox~ll~)bi < ~. Since bi = ( f l i - ~ i ) / ( m - 1) and fli and ~i are finite, we can make bi arbitrarily small by choosing large m, that is, by defining sufficiently large number of fuzzy sets for each variable. []

Remark. Although the total number of rules in the hierarchical fuzzy system is much smaller than that in the corresponding standard fuzzy system, each rule is more complicated because the THEN part is a ruth order polynomial of the input variables. This is the price we have to pay in order to achieve the nice universal approximation capability. In fact, we must have sufficient number of free parameters in order for the hierarchical fuzzy systems to become universal approximators. Essentially, our strategy is to move the complexity from the IF parts to the THEN parts. More specifically, in the standard fuzzy systems the IF parts are required to cover the whole domain, thus the task of the THEN parts is simple and constants can do the job; in the hierarchical fuzzy system, on the other hand, the number of rules is reduced so that the IF parts cannot exhaustively cover the whole domain, but the THEN parts are made sophisticated enough to take care of the uncovered regions. We can say that in the hierarchical fuzzy systems the burden is somewhat "uniformly" distributed over the IF and THEN parts, whereas in the standard fuzzy systems the burden is put on the IF parts only.

230 L.-X. Wang~Fuzzy Sets and Systems 93 (1998) 223-230

4. Conclusions

In this paper, we proved that the hierarchical fuzzy systems can approximate any continuous function over a compact set to arbitrary accuracy. Our proof is constructive, that is, we first designed a hierarchical fuzzy system and then showed that this hierarchical fuzzy system has the universal approximation property. This result provides a theoretical justification for using the hierarchical fuzzy systems in a wide variety of applications.

References

[1] A. Barron, Universal approximation bounds for superpositions of a sigmoidal function, IEEE Trans. Inform. Theory 39 (1993). [2] R. Bellman, Adaptive Control Processes (Princeton University Press, Princeton, 1966). [3] L. Breiman, Hinging hyperplanes for regression, classification and function approximation, IEEE Trans. InJorm. Theory 39 (1993)

999-1013. [4] L. Breiman, J. Friedman, J. Olshen and C. Stone, Classification and Regression Trees (Wadsworth, Belmont, CA, 1984). [5] J.J. Buckley, Universal fuzzy controllers, Automatica 28 (1992) 1245-1248. [6] C. Chui, An Introduction to Wavelets (Academic Press, San Diego, 1992). [7] D. Donoho and I. Johnstone, Projection-based approximation and a duality with kernel methods, Ann. Statist. 17 (1989) 58-106. [8] J. Friedman and W. Stuetzle, Projection pursuit regression, J. Amer. Statist. Assoc. 76 (1981) 817-823. [9] P. Huber, Projection pursuit (with discussion), Ann. Statist. 13 (1985) 435-475.

[10] A. Kolmogorov, On the representation of continuous functions of several variables by superposition of continuous functions of one variable and addition, Dokl. Akad. Nauk. BSSR 114 (1957) 679-681.

[11] C.C. Lee, Fuzzy logic in control systems: fuzzy logic control- Parts I and I1, IEEE Trans. Systems Man Cybernet. 20(2) (1990) 404-435.

[12] G. Lorentz, The 13th problem of Hilbert, in: F. Browder Ed., Mathematical Developments Arising from Hilbert Problems (AMS, Providence, R1, 1976).

[13] W. Pedrycz, Fuzzy Control and Fuzzy Systems (2nd edn., Research Studies Press, 1993). [14] M.J.D. Powell, Approximation Theory and Methods (Cambridge University Press, Cambridge, 1981). [15] G.V.S. Raju, J. Zhou and R.A. Kisner, Hierarchical fuzzy control, Internat. J. Control 54 (1991) 1201-1216. [16] G.V.S. Raju and J. Zbou, Adaptive hierarchical fuzzy controller, IEEE Trans. Systems Man Cybernet. 23 (1993) 973-980. [17] M. Sugeno and G.T. Kang, Fuzzy modeling and control of multilayer incinerator, Fuzzy Sets and Systems 18 (1986) 329-346. [18] T. Takagi and M. Sugeno, Fuzzy identification of systems and its application to modeling and control, IEEE Trans. Systems Man

Cybernet. 15 (1985) 116-132. [19] L.X. Wang, Adaptive Fuzzy Systems and Control: Design and Stability Analysis (Prentice-Hall: Englewood Cliffs, NJ, 1994). [20] L.X. Wang, Fuzzy systems are universal approximators, Proc. 1EEE Internat. Conf. on Fuzzy Systems (San Diego, 1992)

1163-1170. [21] R.R. Yager and D.P. Filev, Essentials of Fuzzy Modeling and Control (Wiley, New York, 1994). [22] H. Ying, Sufficient conditions on general fuzzy systems as function approximators, Automatica 30 (1994) 521-525. [23] X.J. Zeng and M.G. Singh, Approximation theory of fuzzy systems - MIMO case, IEEE Trans. Fuzzy Systems 3(4) (1995).