ANFIS: Adaptive Neuro-Fuzzy Inference SystemANFIS: Adaptive Neuro-Fuzzy Inference System

Chapter 12: ANFIS

Chapter 12: ANFIS

2

OutlineOutlineSoft computingFuzzy logic and fuzzy inference systemsNeural networksNeuro-fuzzy integration: ANFIS

• ANFIS: Adaptive Neuro-Fuzzy Inference Systems• Learning methods for parameter ID

Input selection for ANFIS modeling• Heuristic and exhaustive searches• Performance index

Application examples• Hair dryer modeling• Box-Jenkins furnace data

Chapter 12: ANFIS

3

Neuro-Fuzzy and Soft ComputingNeuro-Fuzzy and Soft Computing

Neural networks

Fuzzy inf. systems

Model space

Adaptive networks

Derivative-free optim.

Derivative-based optim.

Approach space

SoftSoftComputingComputing

Chapter 12: ANFIS

4

Fuzzy SetsFuzzy Sets

Sets with fuzzy boundaries

A = Set of tall people

Heights(cm)

170

1.0

Crisp set A

Membershipfunction

Heights(cm)

170 180

.5

.9

Fuzzy set A1.0

Chapter 12: ANFIS

5

Membership Functions (MFs)Membership Functions (MFs)

About MFs• Subjective measures• Not probability functions

MFs

.5

.8

.1

“tall” in Taiwan

“tall” in the US

“tall” in NBA

180 Heights(cm)

Chapter 12: ANFIS

6

Fuzzy If-Then RulesFuzzy If-Then Rules

• Mamdani styleIf pressure is high then volume is small

high small

• Sugeno styleIf speed is medium then resistance = 5*speed

mediumresistance = 5*speed

Chapter 12: ANFIS

7

Fuzzy Inference System (FIS)Fuzzy Inference System (FIS)

If speed is low then resistance = 2If speed is medium then resistance = 4*speedIf speed is high then resistance = 8*speed

Rule 1: w1 = .3; r1 = 2Rule 2: w2 = .8; r2 = 4*2Rule 3: w3 = .1; r3 = 8*2

Speed2

.3

.8

.1

low medium high

Resistance = Σ(wi*ri) / Σwi= 7.12

MFs

Chapter 12: ANFIS

8

First-Order Sugeno FISFirst-Order Sugeno FIS

• Rule baseIf X is A1 and Y is B1 then Z = p1*x + q1*y + r1

If X is A2 and Y is B2 then Z = p2*x + q2*y + r2

• Fuzzy reasoning

A1 B1

A2 B2

x=3

X

X

Y

Yy=2

w1

w2

z1 =p1*x+q1*y+r1

z =

z2 =p2*x+q2*y+r2

w1+w2

w1*z1+w2*z2

Π

Chapter 12: ANFIS

9

Fuzzy Inference Systems (FIS)Fuzzy Inference Systems (FIS)

Also known as• Fuzzy models• Fuzzy associate memories (FAM)• Fuzzy controllers

Rule base(Fuzzy rules)

Data base(MFs)

Fuzzy reasoning

input output

Chapter 12: ANFIS

10

Neural NetworksNeural Networks

Supervised Learning• Multilayer perceptrons• Radial basis function networks• Modular neural networks• LVQ (learning vector quantization)

Unsupervised Learning• Competitive learning networks• Kohonen self-organizing networks• ART (adaptive resonant theory)

Others• Hopfield networks

Chapter 12: ANFIS

11

Single-Layer PerceptronsSingle-Layer Perceptrons

Network architecture

x1

x2

x3

w1

w2

w3

w0

y = signum(Σwi xi + w0)

∆wi = κ t xi

Learning rule

Chapter 12: ANFIS

12

Single-Layer PerceptronsSingle-Layer Perceptrons

Example: Gender classification

h

v

w1

w2

w0

Network Arch.

y = signum(hw1+vw2+w0)-1 if female1 if male=

y

Training data

h (hair length)

v (v

oice

freq

.)

Chapter 12: ANFIS

13

Multilayer Perceptrons (MLPs)Multilayer Perceptrons (MLPs)

Learning rule:• Steepest descent (Backprop)• Conjugate gradient method• All optim. methods using first derivative• Derivative-free optim.

Network architecture

x1

x2

y1

y2

hyperbolic tangentor logistic function

Chapter 12: ANFIS

14

Multilayer Perceptrons (MLPs)Multilayer Perceptrons (MLPs)

Example: XOR problem

Training data

x1

x2

y

Network Arch.

x1 x2 y0 0 00 1 11 0 11 1 0

x1

x2

x1

x2

y

Chapter 12: ANFIS

15

MLP Decision BoundariesMLP Decision Boundaries

A B

B A

AB

XOR Interwined General

1-layer: Half planes

A B

B A

AB

2-layer: Convex

A B

B A

AB

3-layer: Arbitrary

Chapter 12: ANFIS

16

Adaptive NetworksAdaptive Networks

Architecture:• Feedforward networks with diff. node functions• Squares: nodes with parameters• Circles: nodes without parameters

Goal:• To achieve an I/O mapping specified by training data

Basic training method:• Backpropagation or steepest descent

x

yz

Chapter 12: ANFIS

17

Derivative-Based OptimizationDerivative-Based Optimization

Based on first derivatives:• Steepest descent• Conjugate gradient method• Gauss-Newton method• Levenberg-Marquardt method• And many others

Based on second derivatives:• Newton method• And many others

Chapter 12: ANFIS

18

Fuzzy ModelingFuzzy Modeling

Unknown target systemUnknown target system

Fuzzy Inference systemFuzzy Inference system

y

y*

x1

xn

. . .

• Given desired i/o pairs (training data set) of the form(x1, ..., xn; y), construct a FIS to match the i/o pairs

• Two steps in fuzzy modelingstructure identification --- input selection, MF numbersparameter identification --- optimal parameters

Chapter 12: ANFIS

19

Neuro-Fuzzy ModelingNeuro-Fuzzy Modeling

Basic approach of ANFIS

Adaptive networks

Neural networks Fuzzy inferencesystems

Generalization Specialization

ANFIS

Chapter 12: ANFIS

20

ANFISANFIS

• Fuzzy reasoning

A1 B1

A2 B2

w1

w2

z1 =p1*x+q1*y+r1

z2 =p2*x+q2*y+r2

z = w1+w2

w1*z1+w2*z2

x y

• ANFIS (Adaptive Neuro-Fuzzy Inference System)A1

A2

B1

B2

Π

ΠΣ

Σ/

x

y

w1

w2

w1*z1

w2*z2

Σwi*zi

Σwi

z

Chapter 12: ANFIS

21

Four-Rule ANFISFour-Rule ANFIS

• ANFIS (Adaptive Neuro-Fuzzy Inference System)A1

A2

B1

B2

Σ

Σ

/

x

y

w1

w4

w1*z1

w4*z4

Σwi*zi

Σwi

z

Π

Π

Π

Π

• Input space partitioning

A1

B1

A2

B2

x

y x

y

A1 A2

B1

B2

Chapter 12: ANFIS

22

ANFIS: Parameter IDANFIS: Parameter ID

Hybrid training method

A1

A2

B1

B2

Σ

Σ

/

x

y

w1

w4

w1*z1

w4*z4

Σwi*zi

Σwi

z

Π

Π

Π

Π

nonlinearparameters

linearparameters

fixed

least-squares

steepest descent

fixed

forward pass backward passMF param.(nonlinear)

Coef. param.(linear)

Chapter 12: ANFIS

23

Parameter ID: Gauss-Newton MethodParameter ID: Gauss-Newton Method

Synonyms:• linearization method• extended Kalman filter method

Concept:general nonlinear model: y = f(x, θ)linearization at θ = θnow:

y = f(x, θnow)+a1(θ1 - θ1,now)+a2(θ2 - θ2,now) + ...LSE solution:

θnext = θnow + η(A A) A BT T-1

Chapter 12: ANFIS

24

Param. ID: Levenberg-MarquardtParam. ID: Levenberg-Marquardt

Formula:θnext = θnow + η(A A + λI) A B

Effects of λ:λ small Gauss-Newton methodλ big steepest descent

How to update λ:greedy policy make λ smallcautious policy make λ large

T T-1

Chapter 12: ANFIS

25

Param. ID: ComparisonsParam. ID: Comparisons

Steepest descent (SD)• treats all parameters as nonlinear

Hybrid learning (SD+LSE)• distinguishes between linear and nonlinear

Gauss-Newton (GN)• linearizes and treats all parameters as linear

Levenberg-Marquardt (LM)• switches smoothly between SD and GN

Chapter 12: ANFIS

26

FIS data structure in FLTFIS data structure in FLT

FIS file and FIS matrix

FIS file(on disk)

FIS Data Structure

MF 1Label: Small

Type: GaussianParams: [5 -10]

MF 2Label: Large

Type: TriangleParams: [-5 10 20]

[Input 1]Name: PositionRange: [-10 10]

MF #: 2

MF 1Label: SmallType: Gbell

Params: [5 2 -40]

MF 2Label: Large

Type: SParams: [-50 0 50]

[Input 2]Name: VelocityRange: [-50 50]

MF #: 2

MF 1Label: Neg. Big

Type: ZParams: [-10 -5 0]

MF 2Label: Neg. Small

Type: GbellParams: [5 2 -3]

MF 3Label: Pos, Small

Type: GbellParams: [5 2 3]

MF 4Label: Pos. Big

Type: SParams: [0 5 10]

[Output 1]Name: Force

Range: [-10 10]MF #: 4

[Rules]Rule list

Rule weightsRule types

[System]Name: mam21Type: mamdani

. . .

FIS matrix(in workspace)

readfiswritefis

Chapter 12: ANFIS

27

From Data Sets to FISFrom Data Sets to FIS

Flow chart: From data sets to FIS

FLTGUI tools

genfis1.m

genfis2.m

anfis.mTrainingdata

Initial FIS

Trainingdata

Checkingdata

Final FIS

Chapter 12: ANFIS

28

ANFIS: Structure IDANFIS: Structure ID

Input selection

Input space partitioning

Grid partitioning Tree partitioning Scatter partitioning

• CART method •C-means clustering•mountain method

To select relevant input for efficient modeling

Chapter 12: ANFIS

29

Input SelectionInput Selection

Optimal search: direct exhaustive search

x2x2x1x1 x3x3 x4x4 x5x51 input

x1, x2x1, x2 x1, x3x1, x3 x1, x4x1, x4 x1, x5x1, x5 x2, x3x2, x3 . . .2 inputs

x1, x2, x3x1, x2, x3 x1, x2, x4x1, x2, x4 x1, x2, x5x1, x2, x5

x1, x2, x3, x4x1, x2, x3, x4 x1, x2, x3, x5x1, x2, x3, x5

3 inputs

4 inputs

x1, x3, x4x1, x3, x4 . . .

x1, x2, x4, x5x1, x2, x4, x5 . . .

Chapter 12: ANFIS

30

Input SelectionInput Selection

Suboptimal search• One-pass ranking• Sequential forward selection• Generalized sequential forward selection• Sequential backward selection• Generalized sequential backward selection• ‘Add m, remove n’ selection• Generalized ‘add m, remove n’ selection

Chapter 12: ANFIS

31

Input SelectionInput Selection

Sequential forward selection

x2x2 x3x3 x4x4x1x1 x5x5

x2, x1x2, x1 x2, x3x2, x3 x2, x4x2, x4 x2, x5x2, x5

x2, x4, x1x2, x4, x1 x2, x4, x3x2, x4, x3 x2, x4, x5x2, x4, x5

x2, x4, x3, x1x2, x4, x3, x1 x2, x4, x3, x5x2, x4, x3, x5

1 input

2 inputs

3 inputs

4 inputs

Chapter 12: ANFIS

32

Performance IndexPerformance Index

How to effectively contruct a model and evaluate it properly?

• Model construction:ANFIS with one-epoch training(Use least-squares method only once)

• Model evaluation:- Training RMSE (root-mean-squared error)- Bipartite regularity criterion- Leave-one-out regularity criterion

Chapter 12: ANFIS

33

Performance IndexPerformance Index

Bipartite regularity criterion:

[ ] [ ]( ){ }( ){ }

RCA

t F xB

t F x

x t

x t

F

F

iA

B iA

iB

A iB

ii

iA

iA

iB

iB

A

B

= − + −⎧⎨⎩

⎫⎬⎭

•

•

∑∑1 12

2 2

| |( )

| |( )

;

;

( ):

( ):

r r

r

r

Data Set A =

Data Set B =

Model identified using data set A

Model identified using data set B

Chapter 12: ANFIS

34

Regularity CriterionRegularity Criterion

Bipartite RC (used in GMDH)

Data set A

Data set B

Model AModel A

constructionconstruction

MSEAevaluationevaluation

Model BModel B MSEB

constructionconstruction

evaluationevaluation

RC = (MSEA + MSEB)/2

Chapter 12: ANFIS

35

Regularity CriterionRegularity Criterion

Leave-one-out RC

( )rx ti i; Model kModel k

( )rx t1 1;

( )rx t2 2;

( )rx tn n;

...

...

...ni/o

pairs

constructionconstruction

MSEk( )rx ti i; evaluationevaluation

RC = (Σk MSEk)/n

Chapter 12: ANFIS

36

Regularity CriterionRegularity Criterion

Computational complexity of ANFIS:• Bipartite RC: TBRC = n*(tL+tE)• Leave-one-out RC: TLRC = 2n*tL + n*tE

(tL: time for one-entry sequential LSE update)(tE: time for one-entry model computation)

• If tL = 4tE, then TLRC/TBRC = 9/5 ~= 2.

Chapter 12: ANFIS

37

Input Selection Based on RCInput Selection Based on RC

Test example: y = (1 + x + x )Data source:

• Sugeno and Yasukawa, Tran. on Fuzzy SystemsData size:

• Total: 50, data set A: 25, data set B: 25Two dummy input variables: x3 and x4

-1.52

-21

2

Chapter 12: ANFIS

38

Input Selection Based on RCInput Selection Based on RC

x1x1(.57)(.57)

x2x2(.74)(.74)

x3x3(1.20)(1.20)

x4x4(1.47)(1.47)

x1, x2x1, x2(.33)(.33)

x1, x3x1, x3(1.18)(1.18)

x1, x4x1, x4(1.10)(1.10)

x1, x2, x3x1, x2, x3(21.49)(21.49)

x1, x2, x4x1, x2, x4(18.94)(18.94)

Level 1

Level 2

Level 3

Chapter 12: ANFIS

39

Gas Furnace ModelingGas Furnace Modeling

Data source: Box-Jenkins gas furnace data

Sampling time: 9 seconds

Gas furnaceGas furnace y(k)y(k)(CO(CO22 concentration)concentration)

u(k)u(k)(gas flow rate)(gas flow rate)

u(k)u(k)

y(k)y(k)

Chapter 12: ANFIS

40

Gas Furnace ModelingGas Furnace Modeling

10 potential inputs for ANFIS:• Group 1: y(k), y(k-1), y(k-2), y(k-3)• Group 2: u(k), u(k-1), u(k-2), u(k-3), u(k-4), u(k-5)

1 output: y(k+1)System model:

y(k+1) = F(y(k), ... y(k-3), u(k), ... u(k-5))Data size: 296

Chapter 12: ANFIS

41

Gas Furnace ModelingGas Furnace Modeling

Sequential forward selection with bipartite RC:• data set A: the first 148 entries• data set B: the second 148 entries

Level 1Level 1Level 2Level 2

Level 3Level 3

Selected inputs:Selected inputs:y(k) and u(ky(k) and u(k--2)2)

(RC = .186)(RC = .186)

(27.5 s)(27.5 s)

Chapter 12: ANFIS

42

Gas Furnace ModelingGas Furnace Modeling

Sequential forward selection with bipartite RC:• data set A: odd-indexed 148 entries• data set B: even-indexed 148 entries

Level 1Level 1Level 2Level 2

Level 3Level 3Level 4Level 4

Selected inputs:Selected inputs:y(k), y(ky(k), y(k--1), and u(k1), and u(k--3)3)

(RC = .107)(RC = .107)

(70.9 s)(70.9 s)

Chapter 12: ANFIS

43

Gas Furnace ModelingGas Furnace Modeling

Sequential forward selection with leave-one-out RC:

Selected inputs:Selected inputs:y(k), y(ky(k), y(k--1), and u(k1), and u(k--3)3)

(RC = .095)(RC = .095)

Level 1Level 1Level 2Level 2

Level 3Level 3Level 4Level 4

Chapter 12: ANFIS

44

Hair Dryer ModelingHair Dryer Modeling

Data source: Chap. 17 of System Identification., Ljung, 1984

u(k): binary random signal shifting between 3.41 and 6.41 Vsampling time: 0.08s

Hair dryerHair dryer y(k)y(k)(temperature)(temperature)

u(k)u(k)(voltage)(voltage)

y(k)

u(k)

Chapter 12: ANFIS

45

Hair Dryer ModelingHair Dryer Modeling

10 potential inputs for ANFIS:• Group 1: y(k), y(k-1), y(k-2), y(k-3)• Group 2: u(k), u(k-1), u(k-2), u(k-3), u(k-4), u(k-5)

1 output: y(k+1)System model:

y(k+1) = F(y(k), ... y(k-3), u(k), ... u(k-5))Data size: 600

• data set A: 300• data set B: 300

Chapter 12: ANFIS

46

Hair Dryer ModelingHair Dryer Modeling

Level 1Level 1Level 2Level 2

Level 3Level 3

Level 4Level 4

Sequential forward selection with bipartite RC:

Level 5Level 5

Selected inputs:Selected inputs:y(k), y(ky(k), y(k--1), u(k1), u(k--2) and u(k2) and u(k--3)3)

(RC = .002)(RC = .002)

(829 s)(829 s)

Chapter 12: ANFIS

47

Hair Dryer ModelingHair Dryer ModelingTo select 3 inputs out of ten:

Sequential forward selectionSequential forward selection(22.2 s)

Direct exhaustive searchDirect exhaustive search(54.7 s)

Chapter 12: ANFIS

48

Hair Dryer ModelingHair Dryer Modeling

ARX modeltraining RMSE = .114test RMSE = .072

ANFIS modeltraining RMSE = .038test RMSE = .044