fuzzy signature based radial basis neural...

Fuzzy Signature Based Radial Basis Neural Network

Wei Fan

4 November 2011

A report submitted for the degree of Master of Computing of

the Australian National University

Under the supervision of Prof. Tom Gedeon

Research School of Computer Science

College of Engineering and Computer Science

Australian National University

Fuzzy Signature Based Radial Basis Neural Network Wei Fan (u4924027)

2

Acknowledgements

The work in this report would not have been a success without my nice supervisor Tom Gedeon and his PhD student Dingyun Zhu. I also thank my friends Huajie Wu and Tengfei Sheng.


3

Abstract

This report first takes a broad look at neural network and fuzzy signatures in the literature, and then focuses on fuzzy signature based Radial Basis Function (RBF) neural networks. The term fuzzy signature in this context indicates a solution for fuzzy rule based systems, used to reduce the rule explosion common in such fuzzy rule based systems. The other term Radial Basis Function in this report is a real value function where the value depends only on the distance from the centroid of that function. This report presents two implementations. The first implementation is a basic neural network, and is a Radial Basis Function neural network using fuzzy signatures as neurons. Fuzzy signatures for the implementation are constructed by either a data clustering method or manually specified. The second implementation is a testing suite which is based on the fuzzy signatures RBF neural network implementation. It is able to process a number of benchmarks. Finally, this report represents experimental evaluation for the fuzzy signatures based RBF neural network by two experiments. The result shows this approach is viable and worth further investigation.


4

List of Abbreviations

NN Neural Networks (synonym for ANN) ANN Artificial Neural Networks RBF Radial Basis Function VVFS Vector Valued Fuzzy Sets Cascor Cascade Correlation Neural Network sNN Symmetric Nearest Neighbor


5

Table of Contents

Acknowledgements ...................................................................................................................................................... 2 Abstract ............................................................................................................................................................................. 3 List of Abbreviations .................................................................................................................................................... 4 Table of Contents ........................................................................................................................................................... 5 List of Figures ................................................................................................................................................................. 7 List of Tables ................................................................................................................................................................... 8 1. Introduction ............................................................................................................................................................ 9

1.1 Motivation ................................................................................................................................................... 9 1.2 Purpose ...................................................................................................................................................... 10 1.3 Contribution ............................................................................................................................................. 10 1.4 Relevant techniques .............................................................................................................................. 10 1.5 Organization of the report .................................................................................................................. 11

2. Background and relevant techniques .......................................................................................................... 11 2.1 Neural network ....................................................................................................................................... 11 2.2 Radial Basis Function ............................................................................................................................ 13 2.3 Fuzzy signatures ..................................................................................................................................... 14 2.4 Fuzzy signature based Radial Basis Function neural network ............................................. 16

3. Design and Implementation ........................................................................................................................... 17 3.1 Programing language choice .............................................................................................................. 18 3.2 Construction of RBF neural network .............................................................................................. 19

3.2.1 Clustering ...................................................................................................................................... 19 3.2.2 Obtain fuzzy signature information .................................................................................... 21 3.2.3 Create neural network ............................................................................................................. 22 3.2.4 Training the neural network.................................................................................................. 23

3.3 Testing suite.............................................................................................................................................. 25 3.3.1 Formatting & re-organization of raw data ....................................................................... 26 3.3.2 Extracting network information .......................................................................................... 26 3.3.3 Testing network & gathering benchmarks ....................................................................... 27

4. Evaluation of Fuzzy signature based RBF neural network ................................................................. 28 4.1 Dataset description ................................................................................................................................ 28

4.1.1 Data sets from UCI ..................................................................................................................... 28 4.1.2 Data set from another source ................................................................................................ 31

4.2 Experiment1: Benchmarks for Heart data set ............................................................................. 31 4.2.1 Experiment description ........................................................................................................... 31 4.2.2 Experiment setup ....................................................................................................................... 31 4.2.3 Experimental results ................................................................................................................ 32

4.3 Experiment2: Benchmarks comparison between Fuzzy signature based RBF neural network and other approaches .................................................................................................................... 38

4.3.1 Experiment description ........................................................................................................... 38


6

4.3.2 Experiment setup ....................................................................................................................... 38 4.3.3 Experimental results ................................................................................................................ 38

5. Conclusion and future work ........................................................................................................................... 41 5.1 Conclusion ................................................................................................................................................. 41 5.2 Future work .............................................................................................................................................. 42

References ...................................................................................................................................................................... 43 Appendix A ..................................................................................................................................................................... 45


7

List of Figures

FIGURE 1: EXAMPLE OF A BASIC NEURON ................................................................................................................... 11 FIGURE 2: EXAMPLE OF A NEURAL NETWORK ............................................................................................................ 13 FIGURE 3: EXAMPLE FUZZY SIGNATURE ..................................................................................................................... 15 FIGURE 4: AGGREGATION OF FUZZY SIGNATURES ....................................................................................................... 15 FIGURE 5: EXAMPLE OF FUZZY SIGNATURE BASED RADIAL BASIS FUNCTION NEURAL NETWORK ......................... 17 FIGURE 6: GENERAL ARCHITECTURE DESIGN FOR THE FUZZY SIGNATURE BASED RBF NEURAL NETWORK PROJECT

................................................................................................................................................................................. 18 FIGURE 7: ALGORITHM OF AGGLOMERATIVE HIERARCHICAL CLUSTERING .............................................................. 20 FIGURE 8: MANHATTAN DISTANCE AND GAUSSIAN FUNCTION ................................................................................. 24 FIGURE 9: ARCHITECTURE OF TRAINING MODULE ..................................................................................................... 25 FIGURE 10: ACTUAL OUTPUT AND DESIRED OUTPUT................................................................................................. 28 FIGURE 11: CLASS DISTRIBUTION WITH NUMBER OF CLUSTERS FROM 2 TO 5 ........................................................ 33 FIGURE 12: HINTON DIAGRAMS WITH NUMBER OF FUZZY NEURONS FROM 4 TO 7 ................................................ 34 FIGURE 13: ACCURACY RATE FOR TRAINING AND TESTING DATA SET WITH NUMBER OF NEURONS FROM 2 TO 15.

................................................................................................................................................................................. 35 FIGURE 14: MEAN SQUARE ERROR FOR “HEART” DATA SET WITH FUZZY NEURONS FROM 2 TO 15..................... 36 FIGURE 15: ACCURACY RATE FOR TRAINING AND TESTING DATA SET WITH EPOCHS FROM 1 TO 2000. .............. 36 FIGURE 16: MEAN SQUARED ERROR FOR “HEART” DATA SET WITH EPOCHS FROM 1 TO 2000 ........................... 37 FIGURE 17: TEST ACCURACY PERCENTAGE RESULTS WITH DIFFERENT CLASSIFICATION APPROACHES ................ 41 FIGURE 18: CLASS DISTRIBUTION WITH NUMBER OF CLUSTERS FROM 6 TO 9 ........................................................ 54 FIGURE 19: HINTON DIAGRAMS WITH NUMBER OF FUZZY NEURONS FROM 2 TO 15 ............................................. 55


8

List of Tables

TABLE 1: RELATED TECHNIQUES ..................................................................................................................................... 10 TABLE 2: ADVANTAGES AND DISADVANTAGES BETWEEN DIFFERENT CLUSTER METHODS ........................................ 19 TABLE 3: SIGNAL AND ASSOCIATED AGGREGATION FUNCTION ..................................................................................... 22 TABLE 4: BENCHMARKS FILE SPECIFICATION ................................................................................................................. 27 TABLE 5: DATA SETS INFORMATION ................................................................................................................................ 29 TABLE 6: HARDWARE ENVIRONMENT INFORMATION ................................................................................................... 32 TABLE 7: LOCAL OPTIMIZED NETWORK INFORMATION ................................................................................................ 39 TABLE 8: TEST ACCURACY PERCENTAGE RESULTS WITH DIFFERENT CLASSIFICATION APPROACHES ....................... 40


9

1. Introduction

1.1 Motivation

Nowadays, classification is applied in a variety of fields, such as math, business, industry, artificial intelligence and medical science. It refers to a mathematic procedure for assigning a known set of input data into a given number of categories. Specific examples include medical diagnosis, handwritten digit recognition, voice recognition, and video tracking and internet search engines. However, input data in reality may contain millions of records and also contain noise (outliers), inconsistent data and missing values. Therefore how to classify datasets correctly and efficiently with understandable outcomes is becoming a key issue in these fields. The traditional technique to solve this kind of issue is statistical classification. The typical example is the Bayesian procedure [14]. It has been shown by Swain et al [1] that this method can work very well if the distribution functions of the information classes are known. Unfortunately, there are two major limitations for statistical classification. The first drawback is that users do not always know the distribution functions, especially for data sets that are from multiple sources. The second limitation is that performance of the statistical classification is sensitive to the model of the data. Therefore, users must have a good understanding about the data before this technique can be applied with good effect. The neural networks approach is a technique based on bio-inspired models to solve problems such as pattern recognition. It has an advantage that it is distribution-free, which means the user does not need to know the statistical distribution of the data. However, the neural network approach can be significantly complex computationally, especially for networks that contain hundreds of input neurons. Therefore, the efficiency of the neural network approach is more dependent on the complexity of the data. A similar efficiency issue called rule explosion affects fuzzy rule based system. The complexity of fuzzy rule based system grows exponentially with the number of input variables. In this case, it has been reported by Gedeon et al [2][4][5] that fuzzy signatures can be used in the fuzzy rule based system for solving the rule explosion problem. Based on this knowledge, the fuzzy signatures based Radial Basis Function neural network approach has been introduced. It is an artificial neural network that uses radial basis functions as the activation function. More deeply, each hidden neuron in the Radial Basis Function network has a fuzzy signature implementing it. Theoretically, this novel approach should take advantage from the benefits of neural networks and avoid the efficiency issues of fuzzy systems.


10

1.2 Purpose

The aim of this report is to provide a detailed investigation for Radial Basis Function neural networks using fuzzy signatures as neurons, and evaluate this approach by using real world data.

1.3 Contribution

The contribution of this report includes three main parts. It presents techniques and methodologies to implement such a basic fuzzy signature neural network. A suitable test suite for the implementation is also demonstrated in this report. The evaluation experiment for the fuzzy signature neural network is described. The feasibility and future works of such a neural network based on the results of the experiments is discussed as well.

1.4 Relevant techniques

In this report, several relevant techniques or methodologies will be introduced in the later chapters. The table below shows the technique names and the chapter in which they are discussed.

Techniques Involved section Neural network Section 2.1 Radial Basis Function Section 2.2 Euclidean distance Section 2.2 Taxicab distance Section 2.2 Fuzzy signature Section 2.3 Hierarchical clustering Section 3.2.1 K-means clustering Section 3.2.1 Ward linkage Section 3.2.1 Gaussian function Section 3.2.4 Cascor Section 4.3 sNN Section 4.3 Non-linear Bayes Classifier Section 4.3

Table 1: related techniques

All of these techniques are background to this work, and are introduced as required for the implementation and evaluation of fuzzy signature based Radial Basis Function neural networks.


11

1.5 Organization of the report

Chapter 2 gives a background/overview of the relevant techniques and concepts, including neural networks, radial basis function and fuzzy signatures, which are essential in understanding fuzzy signature based Radial Basis Function neural networks. Furthermore, it looks at relevant literature and especially focuses on relevance to the fuzzy signature based Radial Basis Function neural network. Chapter 3 demonstrates the techniques and methodologies of design and implementation of basic fuzzy signature based Radial Basis Function neural networks and the test suite for the implementation. Chapter 4 is about evaluating this approach based on the implementation in Chapter 3. In the final chapter, conclusions and future work are presented.

2. Background and relevant techniques

2.1 Neural network

A neural network or artificial neural network is a programing construction that consists of a number of interconnecting artificial neurons to solve specific artificial intelligence problems such as classification problems. It has an ability to solve some problems that do not have a known statistical model. Additionally, they are good at solving problems that human beings are good at solving. Before I introduce neural networks, the basic neuron shall be demonstrated first.

X1

X2

⋮ Output

Activation function

Xn

Input data Weights

Figure 1: Example of a basic neuron

𝑓(∑) �

Wn

W2

W1


12

Figure 1 displays a basic neuron. According to the figure, it connects a number of input data either from original data or the output of other neurons. The strength of the connection between input data and neurons is called a weight. The weighted sum of the input has been collected by the neuron and then the neuron processes this kind of data by a function called activation function. The output that is generated by activation function will either be sent to other neurons or be treated as the final output. The activation function in this case is the key point of the basic neuron. It is shown by the formula:

𝑦 = 𝑓 ��𝜔𝑖𝑥𝑖

𝑛

𝑖=0

� (2.1.1)

Where f is the specific function. For example sigmoid function

y is the output of the neuron n is the number of inputs of the neuron 𝑥𝑖 is the value of input i of the neuron 𝜔𝑖 is the weight value of input i Compared with a single neuron, neural networks consist of numbers of neurons so that they are able to handle more complex problems. There are three different types of neurons involved in neural networks. They are input neurons, hidden neurons and output neurons. Each neuron is located at different layers which are called input layer, hidden layer and output layer respectively. Figure 2 shows an example neuron network. Before using neural network to solve classification problems, there is a compulsory procedure called training the neural network. The goal of training neural networks is to find suitable values of the weight matrix that will lead to the difference between desired output from data set and actual output from network to be as small as possible. During this process, neurons learn the weights iteratively by being given a number of training data. This process is finished when the network has stabilized. For example, the training process may complete when the weights of the neural network do not change between two consecutive iterations. However, the efficiency of the neural network depends significantly on the training samples. This is because finding such a suitable value of the weight matrix takes time when the data set size is large. This issue would be enlarged for the case that the training sample contains large numbers of data. The size of the weight matrix really depends on the complexity of data. The more dimensions the data contains, the larger the size of the weight matrix. Therefore, it is time consuming when the training samples contain large numbers of data with complex data structure.


13

Figure 2: Example of a neural network

2.2 Radial Basis Function

A radial basis function (RBF) is a real-valued function whose value is determined by the distance between the origin point 𝑥 and some other point 𝑐 [15]. The general radial basis function is:

∅(𝑥, 𝑐) = ∅‖𝑥 − 𝑐‖ (2.2.1)

Normally, the distance measure used in the radial basis function is the Euclidean distance [16]. For one dimension, the distance between two points is the absolute value of the numerical difference. For n dimensional point p and q, the distance is:

𝑑(𝑝, 𝑞) = �(𝑝1 − 𝑞1)2 + (𝑝2 − 𝑞2)2 + ⋯+ (𝑝𝑛 − 𝑞𝑛)2 (2.2.2)

Where 𝑝 = (𝑝1,𝑝2, … 𝑝𝑛) and 𝑞 = (𝑞1, 𝑞2, … 𝑞𝑛)

There are other distance measures which can be used in the radial basis function. The taxicab

Output layer

Output neuron

Output neuron

Weight matrix

Hidden layer

Hidden neuron

Input layer

Input neuron

Input neuron

Input neuron

Weight matrix

Hidden neuron


14

distance (Manhattan distance) is another common measure [17]. The taxicab distance between two points p and q in an n dimensional space is:

𝑑(𝑝, 𝑞) = �|𝑝𝑖 − 𝑞𝑖| (2.2.3)𝑛

𝑖=1

Where 𝑝 = (𝑝1,𝑝2, … 𝑝𝑛) and 𝑞 = (𝑞1, 𝑞2, … 𝑞𝑛)

One advantage of the taxicab distance measure is that the computation cost is less than the Euclidean distance since the Euclidean distance costs more time in calculating the square root and the squares.

2.3 Fuzzy signatures

Fuzzy rule based systems are a popular approach in the artificial intelligence research area since the ability to assign linguistic labels [6], and to manage uncertainty is useful in many decision making and classification settings. However, traditional fuzzy rule based systems suffer from high computational complexity, especially for large size of data with many dimensions. The formula below determines the number of rules that are required for traditional fuzzy rule based systems.

|R| = 𝑂(𝑇𝐾) (2.3.1)

T determines the number of terms for each input dimension and K is the number of input dimensions. The number of rules is able to grow significantly with either or both variable T and K as they increase. Therefore, the traditional fuzzy rule based applications are mainly limited to a few dimensions of input and have few terms per dimension. Based on the formula 2.3.1, there are two basic way to solve the rule explosion problem in fuzzy rule based system. First principle is to decrease the value of variable T which is the number of terms per input and second principle is to reduce the value of variable K which is the number of input dimension. Based on these ideas, there are 4 possible solutions to handle fuzzy rule based systems that have high number of input dimensions and/or terms within those input dimensions. They are sparse fuzzy rule based systems, hierarchical fuzzy rule based systems, sparse hierarchical fuzzy based systems and fuzzy signatures. The fuzzy signatures method is the focus of this section. It is reported by Mendis (2008, p18) that a “Fuzzy Signature is a Vector Valued Fuzzy Set (VVFS), where each vector component is another VVFS (branch) or an atomic value (leaf). It can


15

also be denoted as below” [6]:

A : X → [𝑎𝑖]𝑖=1𝑘 (2.3.2)

Where 𝑎𝑖 = � [𝑎𝑖𝑖]𝑖=1𝑘𝑖 ; 𝑖𝑓 𝑏𝑏𝑎𝑏𝑐ℎ

[0,1] ; 𝑖𝑓 𝑙𝑙𝑎𝑓

Additionally, Figure 3 below demonstrates a fuzzy signatures structure for SARS patients [6]. Each fuzzy signature in this fuzzy signature structure is a vector that describes certain symptoms for a patient. The relationship between different fuzzy signature levels is determined by a set of fuzzy aggregation functions. Figure 4 shows the aggregation of SARS patient fuzzy signatures using max, min and average as aggregation functions. According to the figure, these aggregation functions transfer these fuzzy signatures into single fuzzy signatures and eventually single values.

⎣⎢⎢⎢⎢⎢⎢⎢⎡

�

𝑓𝑙𝑓𝑙𝑏 𝑎𝑎 8 𝐴𝐴𝑓𝑙𝑓𝑙𝑏 𝑎𝑎 12 𝑃𝐴𝑓𝑙𝑓𝑙𝑏 𝑎𝑎 4 𝑃𝐴𝑓𝑙𝑓𝑙𝑏 𝑎𝑎 8 𝑃𝐴

�

� 𝑆𝑦𝑆𝑎𝑆𝑙𝑖𝑐 𝑏𝑙𝑆𝑆𝑑 𝑝𝑏𝑙𝑆𝑆𝑝𝑏𝑙𝐷𝑖𝑎𝑆𝑎𝑆𝑙𝑖𝑐 𝑏𝑙𝑆𝑆𝑑 𝑝𝑏𝑙𝑆𝑆𝑝𝑏𝑙�

𝑏𝑎𝑝𝑆𝑙𝑎𝑎𝑏𝑑𝑆𝑎𝑖𝑏𝑎𝑙 𝑝𝑎𝑖𝑏

⎦⎥⎥⎥⎥⎥⎥⎥⎤

Figure 3: Example Fuzzy Signature

⎣⎢⎢⎢⎢⎢⎢⎡

�

0.60.70.50.4

�

�0.20.5�0.40.8

⎦⎥⎥⎥⎥⎥⎥⎤

Max

⎣⎢⎢⎢⎡

0.7�0.20.5�0.40.8 ⎦

⎥⎥⎥⎤ Min �

0.70.20.40.8

� AVG

[0.525]

Figure 4: Aggregation of fuzzy signatures There are some advantages of using fuzzy signature in fuzzy rule based systems, which have been reported by Wong et al [2]. First and most importantly, it is able to reduce the high computation cost via its sparse and hierarchical data representation. Second, fuzzy signatures have the ability to handle noisy and missing value by using specific aggregation functions. Last


16

but not least, new information can be easily added in without redesigning the structure of the data representation.

2.4 Fuzzy signature based Radial Basis Function neural network

In regards to Chapter 2.1, traditional neural networks cost much computation time to train the neurons when the training samples contain large numbers of data with a complex data structure. Therefore, using neural networks on such data sets is limited in reality. A similar limitation also applies to fuzzy rule based systems which is called rule explosion. One solution is using fuzzy signatures in the fuzzy rule based system (chapter 2.3). Based on this technique, using fuzzy signatures in the radial basis function neural network should be able to overcome both limitations. A fuzzy signature based radial basis function neural network is a type of neural networks where the radial basis function is treated as the activation function in the neuron [13]. For example, the Euclidean distance is calculated from the evaluation point to the sample point in each neuron. Additionally, each hidden neuron (fuzzy signature neuron) in the neural network has a specific fuzzy signature associated with it and the output of the hidden neuron is the similarity between input vector and the fuzzy signature based on the specific aggregation function. Figure 5 represents the general architecture of the fuzzy signature based radial basis function neural network. Note that the strengths between the input neurons and hidden neurons (fuzzy signature neurons) are constants. They do not change during the training time. The whole training task is only performed by the weight matrix between hidden neurons (fuzzy signature neuron) and the output neurons. Therefore, the time to train these neural networks should reduce since the smaller size of weight matrix, and reduced number of layers. Previous work on a similar network has been shown by Chandra [11]. It is an application of fuzzy signature neural network, written in the C language. However, this application has several limitations. Firstly, it was only used for a specific data set, so the application was not able to handle data sets from other sources. Secondly, the aggregation function in the network is pre-defined and only limited to the average method, so users were not able to specify the aggregation function by themselves. Thirdly, only crisp membership values were used. Finally, the result of the application is text only. It may be difficult for user to analyzing the result without any graphic interface, especial when the result contains large amounts of text. Therefore, this implementation significantly extends the previous work, re-designs and constructs the fuzzy signature based RBF network in a different and more general way.


17

Figure 5: Example of Fuzzy signature based radial basis function neural network

3. Design and Implementation

This chapter describes techniques and methodology of how to implement a fuzzy signature based RBF neural network in detail. In general, there are two main tasks for the whole project implementation. They are constructing the RBF neural network and implementing the test suite. The programing language used to implement the project is MATLAB. The implementation can be divided into seven modules. The fundamental architecture of the implementation is displayed in Figure 6. According to this figure, the RBF network construction consists of four modules, they are clustering, obtaining fuzzy signature information, create and train the network. The testing suite contains three modules; they are formatting & re-organize raw data, extracting network information, and testing network and gather results. Furthermore, the construction of the network should be embedded into the test suite, which means it should be part of the test suite.

Weight matrix

Output layer

Output neuron

Output neuron

Hidden layer

Input layer

Input neuron

Input neuron

Input neuron

Fuzzy Signature Neuron

Fuzzy Signature Neuron


18

Construct RBF neural network

Test suite

Figure 6: General architecture design for the fuzzy signature based RBF neural network project

3.1 Programing language choice

The programing language that is used in this project is MATLAB. It is used for implementing both the networks and the test suites. There are several reasons why MATLAB is used for this project. Firstly, it is good at handling complex data structures such as vectors or matrices, and the implementation of the network contains a number of different vectors and matrices associated with it. Secondly, MATLAB has many built-in functions and tools to support the implementation of this neural network and its test suite. For example, the network construction starts with clustering, and MATLAB has various types of built-in clustering methods. This reduces the cost of the project since the result of the clustering can safely be assumed to be correct and there is no need for the coding to be verified. Finally, MATLAB allows us to make sure of the appropriate numerical precision in the final result. In this case, the benchmarks from test suites can be assumed to be numerically accurate by using MATLAB. Therefore, MATLAB is appropriate to be the programing language of this project.

Formatting & re-organization

of raw data

Clustering input data

Obtain Fuzzy

signature

Create neural

network

Train neural

network

Testing network &

gather results

Extracting network

information


19

3.2 Construction of RBF neural network

This section describes the procedure to construct fuzzy signature based radial basis function neural networks in detail. As referred to figure 6 on the general architecture of the network, the whole network procedure is divided into four different modules and will be introduced as follows:

3.2.1 Clustering

Clustering is the pre-processing method to obtain the fuzzy signature neurons in this implementation. However, this is not always a compulsory stage in this implementation since the fuzzy signature neurons can be determined manually. The advantage of using a clustering technique is that the users do not need to consider how to construct the fuzzy signature themselves. Usually, extracting fuzzy signature manually is time consuming and can be difficult, especially when the input data set contains large numbers of records. There are various types of clustering methods. Each clustering method has advantages and disadvantages. Table 2 demonstrates two different clustering methods’ advantages and disadvantages. This implementation uses the agglomerative hierarchical clustering method. This is partly for its hierarchical nature and because it is a deterministic clustering method. That means the same results will be generated with the same dataset at different run times, which is useful during program development and debugging.

Clustering method

k-means clustering Agglomerative hierarchical clustering

Type Flat clustering Hierarchical clustering Advantage Speed may be faster than

hierarchical clustering May generate tight clusters

Do not need user to specify the number of clusters

Algorithm is deterministic Outputs more informative

than flat clustering Disadvantage Difficulty in comparing quality

of the clusters produced since different initial values affects the result.

Sensitive to outliers Need to specify the number of

clusters

Lower efficiency compared with flat clustering

Table 2: Advantages and disadvantages between different cluster methods


20

Agglomerative Hierarchical Clustering

Agglomerative hierarchical clustering is a bottom up hierarchical clustering method where each cluster can be another cluster’s sub-cluster. It begins with each single object in a separate cluster. Then it agglomerates similar clusters based on similarity criteria, until all data merge into one cluster. The detailed algorithm is demonstrated in the figure below (figure 7): Yes

No

Figure 7: Algorithm of agglomerative hierarchical clustering The similarity criterion for this implementation is based on the Ward linkage method since it is efficient. It uses the incremental sum of squares to calculate the distance. The sum of squares measure is defined as in the formula below:

𝑑(𝑏, 𝑆) = �2𝑏𝑟𝑏𝑠

(𝑏𝑟 + 𝑏𝑠)‖𝑥𝑟 − 𝑥𝑠‖ (3.2.1.1)

where 𝑥𝑟 and 𝑥𝑠 are the centroids of clusters r and s

‖𝑥𝑟 − 𝑥𝑠‖ is Euclidean distance between 𝑥𝑟 and 𝑥𝑠 𝑏𝑟 and 𝑏𝑠 are the number of elements in clusters r and s MATLAB has a built in statistical toolbox for Agglomerative Hierarchical Clustering with the Ward linkage method. Function 𝑙𝑖𝑏𝑙𝑎𝑙𝑙(𝑦,𝑎𝑙𝑎ℎ𝑆𝑑) takes input data set 𝑦 and the specific method name as arguments then creates a hierarchical cluster tree. In this case, the formatted data combined with a string as specific parameters pass though this function. This string indicates Ward linkage method. Furthermore, the function 𝑐𝑙𝑝𝑆𝑎𝑙𝑏(𝑍,′ 𝑎𝑎𝑥𝑐𝑙𝑝𝑆𝑎′,𝑏)

Assign each object into separate cluster

Construct distance matrix

based on the similarity criteria

Does the distance matrix

contain only one element?

Clustering complete

Find the pair of clusters

with closest distance

Remove the pair from distance

matrix and update the value of matrix


21

constructs clusters from a hierarchical cluster tree 𝑍 and number of clusters 𝑏 as arguments. The clustering module in this implementation uses both functions to cluster the formatted data set.

3.2.2 Obtain fuzzy signature information

The fuzzy signature neurons are based on either selected clusters (which has been mentioned in Section 3.2.1) or manually input fuzzy signatures. In general, each fuzzy signature represents the similarity between the given input and the specific cluster’s centroid in a data set. Therefore, a matrix would be extracted after the hierarchical clustering method is applied on the formatted data set. It is essential to create a neural network that contains the cluster’s information such as the centroid, and each dimension’s minimum and maximum values for further usage. The detailed structure information of this matrix is shown below:

�

𝐶𝑙𝑏𝑎11 𝐶𝑙𝑏𝑎12 … 𝐶𝑙𝑏𝑎1𝑛 𝐴𝑎𝑥11 … 𝐴𝑎𝑥1𝑛 𝐴𝑖𝑏11 … 𝐴𝑖𝑏1𝑛𝐶𝑙𝑏𝑎21 𝐶𝑙𝑏𝑎22 … 𝐶𝑙𝑏𝑎2𝑛 𝐴𝑎𝑥21 … 𝐴𝑎𝑥2𝑛 𝐴𝑖𝑏21 … 𝐴𝑖𝑏2𝑛

⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮𝐶𝑙𝑏𝑎𝑖1 𝐶𝑙𝑏𝑎𝑖2 … 𝐶𝑙𝑏𝑎𝑖𝑛 𝐴𝑎𝑥𝑖1 … 𝐴𝑎𝑥𝑖𝑛 𝐴𝑖𝑏𝑖1 … 𝐴𝑖𝑏𝑖𝑛

�

Where j is the number of clusters (3.2.2.1) n is the number of dimensions of input data

Centjn is the coordinate of centroid point at dimension n in cluster j Maxjn is the coordinate of maximum value at dimension n in cluster j Minjn is the coordinate of minimum value at dimension n in cluster j

Moreover, various aggregation functions can be applied on partial or whole fuzzy signatures. For example, assume a fuzzy signature instance based on Figure 4 is [[0.7, 0.8, 0.2, 0.1] [0.2, 0.5] 0.1, 0.3], the maximum aggregation function is applied on the sub-vector [0.7, 0.8, 0.2, 0.1] and minimum aggregation function applied on the sub-vector [0.2, 0.5], then the average aggregation is applied on the whole fuzzy signature. In order to get the final membership value, it is important to obtain the fuzzy signature structure information as well. In this case, the module reads structure information from a file and stores it into another matrix. The structure information about the matrix is as follows:

�

𝑆𝑎𝑎𝑏𝑎1 𝐸𝑏𝑑1 𝑆𝑖𝑙𝑏𝑎𝑙1𝑆𝑎𝑎𝑏𝑎2 𝐸𝑏𝑑2 𝑆𝑖𝑙𝑏𝑎𝑙2⋮ ⋮ ⋮

𝑆𝑎𝑎𝑏𝑎𝑖 𝐸𝑏𝑑𝑖 𝑆𝑖𝑙𝑏𝑎𝑙𝑖

� (3.2.2.2)

Where i indicates a specific sub-vector in the fuzzy signature

Starti is the start position value of sub-vector i in the fuzzy signature


22

Endi is the end position value of sub-vector i in the fuzzy signature 𝑆𝑖𝑙𝑏𝑎𝑙𝑖is the signal value of sub-vector i in the fuzzy signature

The table 3 shows the relationship between signal value and type of aggregation.

Signal value Aggregation function description 0 There is no need to do any process with the specific

vector 1 Compute the mean and standard deviation for specific

vector and replace all values of the vector with [mean – standard deviation, mean + standard deviation], then return new vector

2 Return mean from the specific vector 3 Return maximum value from the specific vector 4 Return minimum value from the specific vector

Table 3: Signal and associated aggregation function

According to the matrix 3.2.2.2, the first row in the matrix indicates the lowest level sub-vector in the fuzzy signature and the last row indicates the highest level in the fuzzy signature (the whole fuzzy signature). Therefore, the order of rows in the matrix indicates the structure of the data structure from bottom to up. In this case, the structure information for Figure 4 should be like:

�1 4 35 6 41 8 2

� (3.2.2.3)

3.2.3 Create neural network

After fuzzy signature information has been obtained, the neural network is created. The following parameters need to be specified by the creation process: 1. Number of neurons in the hidden layer (Fuzzy signature neurons) 2. Number of training epochs 3. Centroid point of each cluster 4. Radius of each cluster in each dimension 5. Weights matrix applied to the neural network The number of fuzzy signature neurons and the number of training epochs are determined by the user at the beginning of the program. The centroid point of each cluster is extracted by the


23

module that is described in section 3.2.2. The radius of each cluster in each dimension depends on the cluster coordinates of centroid value, and the maximum and minimum values. The formula below shows extracting the radius based on these three variables:

𝑏𝑎𝑑𝑖𝑝𝑆𝑖 = 𝑎𝑎𝑥𝑖𝑎𝑝𝑎�(𝐴𝑎𝑥𝑖 − 𝐶𝑙𝑏𝑎𝑖), (𝐶𝑙𝑏𝑎𝑖 − 𝐴𝑖𝑏𝑖)� (3.2.3.1) Where 𝑎𝑎𝑥𝑖𝑎𝑝𝑎 is the function that returns the greater of two arguments

𝐶𝑙𝑏𝑎𝑖 is the coordinate of centroid point at dimension i 𝐴𝑎𝑥Ri is the coordinate of maximum value at dimension i 𝐴𝑖𝑏𝑖 is the coordinate of minimum value at dimension i 𝑏𝑎𝑑𝑖𝑝𝑆𝑖 is the value of radius at dimension i

The size of the weight matrix is based on the number of hidden neurons and the number of output dimensions. Furthermore, the neural network weight matrix values are initially set to small random values. This provides the training procedure with a good starting stage to work towards a solution. In this case, one function in the MATLAB statistics toolbox called 𝑏𝑎𝑏𝑑𝑆𝑎 is embedded into the implementation. It takes the number of columns and rows combined with a mean and standard deviation as arguments then generates a matrix with random values. After all possible parameters have been determined; one data container called ℎ𝑎𝑏𝑑𝑙𝑙𝑆 packs all of them and sends the package to the training module.

3.2.4 Training the neural network

The training module is the core module for constructing the network. It finds suitable values for the weight matrix that will lead the error between the desired output from data set and the actual output from the network to be as small as possible. Therefore, providing a suitable training method is an essential pre-condition for solving a specific task such as a classification problem. This module starts to train the neural network when a number of training data are given. After the input neurons receive input data, hidden neurons compute the Manhattan distance between the cluster’s centroid and the given input at each dimension as in Figure 8(a). The Manhattan distance measure is used as it is computational faster than the Euclidean distance measure. A Gaussian function is then applied on the distance. In regard to Figure 8(b), the result ranges between 0 and 1, and shows the similarity between the given data and the cluster’s centroid. The value 1 means the observation data and centroid have the highest similarity (the value of input data is the same as cluster centroid), on the other hand, the value 0 means the observation data and centroid has no similarity (the observed data is far from the centroid). Therefore, the further a neuron is from the evaluation point, the lower value the result is.


24

Activation

Vertical distance

Horizontal distance

Distance

Distance = Vertical distance + Horizontal distance

(a) Manhattan distance (b) Gaussian function

Figure 8: Manhattan distance and Gaussian function

Once the hidden neurons finalize the similarity computation in all input dimensions, the result is encapsulated by a structured data representation which is called a fuzzy signature. In order to send a single value to the output neurons, a specific function called 𝑎𝑙𝑙𝑏𝑙𝑙𝑎𝑎𝑙𝑑𝑎𝑎𝑎(𝑓, 𝑆) then takes the fuzzy signature 𝑓 and fuzzy signature structure matrix 𝑆 as arguments and produces the membership value. The matrix 𝑆 refers to the formula 3.2.2.3 in section 3.2.2. The algorithm for the function 𝑎𝑙𝑙𝑏𝑙𝑙𝑎𝑎𝑙𝑑𝑎𝑎𝑎(𝑓, 𝑆) follows: 1. Get first row from 𝑆 and store it into a vector called r. 2. Apply aggregation function based on the start position, end position and signal value from r. 3. Replace the value of sub-vector with the result of step 2 from start position to end position

in r. 4. Check whether r is the end row of data 𝑆 or not. 5. If not, update the start position and end position from next row to the end row, and fetch

next row into r, then go to step 2. 6. If the result of step 4 is yes, the function is finish. In order to get the actual output of the neuron network, the sum of the weighted membership values is collected by output neurons based on the formula 2.1.1 with weight matrix and clusters’ membership value. The neural network then compares the difference between the actual output and the desired output value, and updates the weight matrix values based on the delta rule. The figure 9 shows the process of the neural network training module. Once all training data has been used in the training module, one training cycle (Epoch) is complete. However, this does not mean the end of the training phase since many training cycles are generally needed to train the neural network. This training module takes the number of


25

training cycles from the creation module mentioned in section 3.2.3 and performs the training procedure iteratively.

Figure 9: Architecture of training module

Delta Rule

The delta rule is a gradient descent update rule for tuning the weights matrix of the neurons. In this implementation, it is demonstrated in simplified form with the update function as follows:

∆𝜔𝑖𝑖 = 𝛼 (𝑎𝑖 − 𝑦𝑖)𝑥𝑖 (3.2.4.1) where 𝛼 is a constant with small value called learning rate 𝑎𝑖 is the value of desired output at dimension i 𝑦𝑖 is the value of actual output at dimension i 𝑥𝑖 is the value of input at dimension i The training module in this implementation uses this delta rule to minimize the error between actual output and desired output. The learning rate can be determined by the users, with a default value of 0.01.

3.3 Testing suite

The testing suite implementation generates a number of benchmarks for the neural network. Those benchmarks include mean squared error and accuracy rates for both training and testing data sets. The testing suite also produces neural network information such as the centroid

Receive data from input layer

Apply radial basis function based on

fuzzy signature

Get membership value based on

aggregate function

Generate output value with weights

Compare difference between actual and

desired output

Update weights value


26

point, minimum value and maximum value of each cluster, and the weight matrix. From Figure 6, there are three modules in the testing suite, called “formatting and re-organization of raw data”, “extracting network information”, and “testing network and gather benchmarks”. Once the testing suite has been implemented, the constructed network is embedded into the testing suite to generate the benchmarks as shown in the general architecture of the network diagram in the beginning of chapter 3 (Figure 6). This section concentrates on demonstrating these three modules.

3.3.1 Formatting & re-organization of raw data

To start with this part, it is essential to format and re-organize the raw data. There are two main advantages to have such a module. First, formatting allows the neural network to handle different type of data sets without modifying the core code. Second, to re-organize the raw data leads to more accurate benchmarks since the training set and testing set are isolated. Usually, raw data sets include two parts. They are the input data set and the corresponding desired output data set. Each column in the input data set indicates an attribute of the observation. Each row in the input dataset demonstrates one observation. The desired output data sets represent the class that each specific observation belongs to. Both input and output data sets may contain missing values. According to the UCI machine learning repository [7], missing values are represented as question marks or null normally. Therefore, it is essential to replace the missing value by using some specific method, or use a method that is not affected by missing values. This module scans all raw data sets, replaces missing values with the specific attribute mean then splits the whole data set into the input data set and the desired output data set. All data in the input data will be converted into decimal. Moreover, the desired output will be converted into a vector of binary values. For example, if the data set consists of 5 different classes and observation x belongs to class 3. The desired output for the observation is [0, 0, 1, 0, 0]. The organizing method is performed in this test suite for the k-fold cross validation scheme. For example, if the variable k is determined as 4 by the user, both input data set and desired data set are divided into 4 sets, where three of them are treated as training set randomly and the remaining one is treated as test set, the total number of iterations is 4. The advantage of achieving this is that the user is able to choose how large each test set is and how many iterations are averaged over independently.

3.3.2 Extracting network information

Once the neural network has been trained, all information that directly relates to the network has been determined. This module collects network information as a kind of benchmark and stores it into a file. The table below describes the files and the benchmarks that are associated


27

with them. The Appendix A at the end of this report also shows an example of network information based on the “Heart” data set that is associated with the experiment in section 4.2.

File Name Benchmarks Description Cluster_details.txt This file describes cluster information in

detail. It includes the class distribution, centroid value, minimum value and max value for each cluster.

Training_details.txt This file contains training information i.e. the weight matrix for each iteration.

Testing_details.txt This file shows testing information based on the trained network. It includes the desired output, optimized output and actual output information for unmatched results.

Final_result.txt This file shows the general result benchmarks. It includes accuracy rate and mean square error by iterations. Moreover, the average accuracy rate and mean squared error is included as well.

Table 4: benchmarks file specification

Further, Hinton diagrams are also generated by this module. They provide qualitative display of the weight matrix values in trained networks. In the Hinton diagram, each square’s color demonstrates a specific sign of weight. In this implementation, the red color (dark if printed out without color) means negative weights and the green color (light if printed out without color) displays positive weights. The absolute value of a weight is shown by the size of the square.

3.3.3 Testing network & gathering benchmarks

After all network information has been extracted, the network will be tested using a specific data set. In this case, both training data set and testing data set will be used during this testing procedure with k-fold cross validation scheme, and benchmarks include accuracy rate and mean squared error are produced by iteration. The accuracy rate indicates the percentage of the desired output which matches the output that is obtained by the network. As mentioned in section 3.3.1, each desired output is a vector of binary values that represents the class to which this observation belongs. On the other hand, the actual result from the neural network is usually a vector consisting of decimal numbers. Therefore, a specific mapping function is performed to produce optimized output. In this module, the mapping function is able to convert the result from Figure 10(a) to the result in Figure 10(b). The detailed mechanism for


28

the mapping function is as follows: 1. Get the vector of actual results from the network. 2. Find the position j that indicates maximum value in the vector. 3. Create a new vector and initialize all elements to 0. The number of elements shall be the

same as the actual result vector. 4. Assign the value of element at position j to 1. 5. Return this new vector as optimized output. Once the optimized output has been generated, the module then compares it with the desired output and produces the value of accuracy and mean squared error one by one. Furthermore, the final benchmarks including average accuracy rate and average mean squared error are generated at the end of the procedure. This module also shows the trend of benchmarks by diagrams, so users can easily understand the quality of the network with those benchmarks.

(a) Actual output (b) Desired output

Figure 10: Actual output and desired output

4. Evaluation of Fuzzy signature based RBF neural network

4.1 Dataset description

4.1.1 Data sets from UCI

There are 12 data sets described in section 4.1.1 from the University of California Irvine (UCI) machine learning data set repository. Table 5 shows the general information for these data sets.

0

0.5

1

1 2 3 4 5

Result

membershipvalue

Class 0

0.5

1

1 2 3 4 5

Result

membershipvalue

Class


29

Data set Number of input

dimensions

Number of output

dimensions

Number of observation

Contain missing value

Cancer 9 2 699 Yes Card 51 2 690 Yes Diabetes 8 2 768 Yes Gene 120 3 3175 No Glass 9 6 214 No Heart 35 2 920 Yes Heartc 35 2 303 Yes Horse 58 3 364 Yes SARS 8 4 4000 No Soybean 35 19 683 Yes Thyroid 21 3 7200 No Ionosphere 34 2 351 No Wine 13 2 178 No

Table 5: data sets information

Moreover, detailed descriptions for all data sets are listed at the UCI website [7]. A brief synopsis of each data set is as follows:

Cancer

This data set refers to a classification task that diagnoses patients’ breast cancer. The task is try to indicate a tumor as either benign or malignant. There are 9 input dimensions and 2 desired output dimensions. The total number of records is 699.

Card

This data set refers to a binary classification task that predicts whether a customer’s credit card should be approved or not. There are 51 inputs that represent a real credit card application. The 2 outputs show the decision on the credit card. The total number of records is 690.

Diabetes

This data set refers to a classification task that diagnoses Pima indians’ diabetes. The aim of the data set is try to determine whether a Pima indian individual is diabetes positive or not. There are 8 input dimensions and 2 desired outputs dimensions. The total number of instances is 768.

Gene

This data set demonstrates the detection of intron/exon boundaries in nucleotide sequences. There are 120 inputs which encode 60 DNA sequence elements. Each DNA element is a


30

four-valued nominal attribute. The output consists of 3 dimensions. The total number of observations is 3,175.

Glass

This data set indicates the classification task that classifies glass types. 9 inputs consist of content percentage of 8 different elements plus the refractive index. The output shows the 6 different types of glass. The number of observations is 214.

Heart

This data set describes the prediction of heart disease. There are 35 input dimensions associated with personal data such as age, sex, smoking habits, subjective patient pain descriptions and results of various medical examinations such as blood pressure and electro cardiogram result. The 2 output dimensions determine whether at least one of four vessels is reduced in diameter by more than 50%. Total number of observations is 920.

Heartc

This data set is an alternate version of the “Heart” data set. The structure of input and output is the same as the heart data set. The difference between “Heart” and “Heartc” is the “Heartc” dataset comes from a different source, the Cleveland Clinic Foundation. The total number of observations is 303.

Horse

This data set refers to a classification task that predicts the fate of a colic horse. It indicates whether the horse will survive, will die or will be euthanized based on the result of veterinary examination. The structure of the data set consists of 58 input dimensions and 3 desired output dimensions. The total number of records is 364.

Soybean

This data set indicates the classification task that recognizes different diseases of soybeans. The 35 inputs demonstrate the features of the bean and plant leaves plus information about each plant’s life history. There are 19 different diseases of soybeans, and there are 19 desired output dimensions. The total number of observations is 683.

Thyroid

This data set refers to a classification task that diagnoses patients’ thyroid condition. There are 21 inputs, 7200 examples. The desired output has 3 types which indicate whether the patient’s thyroid has overfunction, normal function, or underfunction.

Ionosphere

This data set refers to radar information that was obtained by a system in Goose Bay, Labrador. It is used to determine whether some type of structure exists in the ionosphere. There are 34 input dimensions that indicate values of the electromagnetic signal, and 2 desired output


31

dimensions show whether the structure exists or not. Therefore, this is a binary classification task. Total number of observations is 351.

Wine

This data set indicates a chemical analysis of three different type of wine that grown in the same region in Italy. It consists of 13 input dimensions which describe the information of the wine and three output dimensions which indicate three types of wines. Total number of observations is 178.

4.1.2 Data set from another source

SARS

This data set describes the severe acute respiratory syndrome (SARS) patients’ information such as fever temperature at different time, blood pressure, nausea and abdominal pain by 8 input dimensions. The 4 outputs indicate 4 types of patients. They are SARS, normal, pneumonia and hypertension. The total number of observation is 4,000 (Wong et al [2]).

4.2 Experiment1: Benchmarks for Heart data set

4.2.1 Experiment description

This experiment takes the “Heart” data set which is a classification problem and generates text and graphic benchmarks based on the fuzzy signatures based RBF network. The goal of this experiment is to discover basic rules for optimizing the network with a specific data set. Experiments were conducted with all twelve data sets shown in table 5 and results for the “Heart” data set are reported in detail below.

4.2.2 Experiment setup

This experiment, based on fuzzy signature based on the RBF neural network model, requires analyzing the results of the “Heart” data set. It is essential to specify the experimental environment. As described in chapter 3, the network and test suite was implemented in MATLAB, More specifically; in MATLAB 7.8.0 (R2009a) Windows version and was run on a laptop. The detailed information about the hardware environment is shown in table 6. The experiment requires analyzing a number of benchmarks from networks with different structures. Therefore several parameters have been modified during the experiment. The fuzzy neuron number for this experiment was set between 2 and 15, and the epochs were set


32

between 1 and 2000. The learning rate was set to 0.01. The validation method used in this experiment is k-fold cross and k was set to 5. The mean function was used as the aggregation function for generating the membership values. Accuracy rate as part of benchmarks is determined by the average. These values were chosen from experience in practice with this model during design and construction (for the range of fuzzy neurons), and from the course COMP8420 Bio-inspired computing and its materials for the other values.

CPU Intel(R) Core(TM) i7 CPU (1.2GHz) Num of Cores 2 Memory 4GB Hard Disk 500GB System Windows 7-64 bit Video Card Nvidia Geforce GT335m Network Single Port Gigabit Ethernet

Table 6: Hardware environment information

4.2.3 Experimental results

Clustering result

Figure 11 presents the distribution of each cluster based on the hierarchical clustering method with specific numbers of clusters. The leftmost bar in each cluster represents the number of patients with “vessel is reduced” and the rightmost bar in each cluster displays the number of patients with “no vessel is reduced”. The contents of the figure depend on the number of clusters. For example, figure 11(a) shows the class distribution for two clusters and figure 11(d) shows the class distribution with five clusters. According to those figures, each cluster contains both patients with “vessel is reduced” and “no vessel is reduced”, so there is no simple relation that can be extracted so far. This section only shows the distributions with cluster numbers from 2 to 5. Figure 18 displays the distribution from 6 to 10 clusters, and can be found in appendix A. For all detailed cluster information, please refer to the results folder for the test suite application since the information is too long to be shown in the report.


33

(a) Class distribution for two clusters (b) Class distribution for three clusters

(c) Class distribution for four clusters (d) Class distribution for five clusters

Figure 11: Class distribution with number of clusters from 2 to 5

Network result

In Figure 12, four Hinton diagrams are presented. They show the strength of neural connections in the network from 4 fuzzy neurons to 7 fuzzy neurons. According to the data, the weakest connection is displayed in Figure 12(c) between the 5th neuron and the 1st output. Also, the Hinton diagrams for 6 and 7 fuzzy neurons are similar, which may mean that a good or acceptable number of neurons were recorded at 6 neurons. Please also see Appendix A, as these similarities are less clear for more fuzzy neurons.

151

260 162

347

0

200

400

Cluster 1 Cluster 2

Class 1

Class 2

69

191 151 241

106 162

0

200

400

Cluster 1 Cluster 2 Cluster 3

Class 1 Class 2

20

171

69

151

35 71

241

162

0

50

100

150

200

250

300

Cluster 1 Cluster 2 Cluster 3 Cluster 4

Class 1 Class 2

106

45 20

171

69

18

144

35 71

241

0

50

100

150

200

250

300

Cluster1

Cluster2

Cluster3

Cluster4

Cluster5

Class 1 Class 2


34

(a) Hinton diagram with 4 fuzzy neurons (b) With 5 fuzzy neurons

(c) With 6 fuzzy neurons (d) With 7 fuzzy neurons

Figure 12: Hinton diagrams with number of fuzzy neurons from 4 to 7

Benchmarks with different number of fuzzy neurons

Figure 13 shows the line graph of accuracy values for both training and testing data sets. The vertical axis demonstrates the accuracy rate for the result and the horizontal axis indicates the numbers of fuzzy neurons included in the network. The line with diamonds shows the accuracy rate trend for the training data set and the line with squares displays the accuracy rate trend for the testing data set. The number of epochs is set to be 50 in this part of the experiment. According to the results, the fuzzy neurons from 2 to 5 shows a rapid climb in the rate of accuracy from 55% to 80%, which is followed by a leveling off at 80% for neurons more than 5. Also it can be noticed that the accuracy rate was the lowest when the number of fuzzy neurons was determined to be 2, and there is not much change while increasing the number of fuzzy neurons from 5 to 15. Therefore, the number of fuzzy neurons is suggested to be 5. Figure 14 below shows the same situation in a different way. Figure 14(a) displays the trend of mean squared error for the training data set with fuzzy neurons increasing from 2 to 15. The figure 14(b) indicates the trend of mean squared error for the testing data set. The vertical axis for both Figure 14(a) and (b) shows the value of the mean squared error and the horizontal axis indicates the numbers of fuzzy neurons involved in the network. According to the results, the mean squared error of network in Figure 14(a) and Figure 14(b) are similar and follow the same trend, the fuzzy neurons from 2 to 5 saw a significant drop in the mean squared error from 0.25 to 0.15, which was followed by a slow reduction while the


35

number of neurons increased to more than 5. The minimal mean squared error for the testing data set is 0.13 while the number of fuzzy neurons is 10. Overall, the local optimized result by various analyses (number of neurons between 2 and 15) is suggested to be between 5 and 10 neurons.

Figure 13: Accuracy rate for training and testing data set with number of neurons from 2 to 15.

(a) Mean squared error for training data set with fuzzy neurons from 2 to 15

50556065707580859095

100

2 3 4 5 6 7 8 9 10 11 12 13 14 15

Training accuracyTesting accuracy

Accuracy

Number of fuzzy neurons

0.1

0.15

0.2

0.25

0.3

2 3 4 5 6 7 8 9 10 11 12 13 14 15

MSE for training

Value



36

(b) Mean squared error for testing data set with fuzzy neurons from 2 to 15

Figure 14: Mean square error for “Heart” data set with fuzzy neurons from 2 to 15

Benchmarks with different epochs

Figure 15 below demonstrates the trend of network accuracy rate while the epochs of training increase from 1 to 2000. In this part of the experiment, the number of fuzzy neurons is set to be 5, based on the suggestion from the previous part.

Figure 15: Accuracy rate for training and testing data set with epochs from 1 to 2000. According to the data, the accuracy rate for both training and testing rate has a significant rise while the epochs increase from 1 to 20, then the training set accuracy rate remains stable as the

0.1

0.15

0.2

0.25

0.3

2 3 4 5 6 7 8 9 10 11 12 13 14 15

MSE for testing

Value


50

55

60

65

70

75

80

85

1 10 20 50 100 200 500 1000 2000

Traing accuracyTesting accuracy

Epochs

Accuracy


37

epochs increase from 20 to 2000. On the other hand, the accuracy rate for the testing data set has a slight increase when the epochs increase from 20 to 100, and a peak has been reached when the epochs are set to 100, then the trend of accuracy rate has a slight decrease with the epochs are over 100. One possible reason for this decrease is that the network is over-fitting when the epochs are over 100. Therefore, the value of epochs for the heart data set is suggested to the range between 10 and 100. Further, the mean squared errors for the training and testing data are displayed in Figure 16. The mean squared error for both training and testing data sets reduces rapidly while the epochs increase from 1 to 50 and there is not much change for the epochs over 50.

(a) Mean squared error for training data set with epochs from 1 to 2000

(b) Mean squared error for testing data set with epochs from 1 to 2000

Figure 16: Mean squared error for “Heart” data set with epochs from 1 to 2000

0.1

0.15

0.2

0.25

0.3

1 10 50 100 200 500 1000 2000

MSE for training

Value

Epochs

0.1

0.15

0.2

0.25

0.3

1 10 50 100 200 500 1000 2000

MSE for testing

Epochs

Value


38

4.3 Experiment2: Benchmarks comparison between Fuzzy

signature based RBF neural network and other approaches

4.3.1 Experiment description

This experiment produces related benchmarks with the data sets that are described in section 4.1 and compares them with other classification approaches. The classification approaches include other neural network methods and statistical methods. The aim of this experiment is to discover the feasibility and performance of fuzzy signature based RBF networks for a range of data sets.

4.3.2 Experiment setup

In order to allow for comparison between fuzzy signature based RBF neural network and other classification method, a series of benchmarks were performed on the data sets that were described in section 4.1. Moreover, the fuzzy signature RBF network benchmarks for each data set have been tuned as in section 4.2. The experiment setup was the same as used previously except that the range of fuzzy neurons was between 2 and 20. There are four approaches used for the benchmark comparisons. The neural network method used for comparison is Cascade Correlation Neural Network (Cascor) [8]. It is a neural network model which begins with a minimal network and adds hidden neurons one by one during training [12]. The other methods used for comparisons in this experiment are Symmetric Nearest Neighbor (k-sNN) [9], Nonlinear Bayes classifier [10] and Polymorphic Fuzzy Signatures [5][6].

4.3.3 Experimental results

For each data set, various numbers of fuzzy neurons and epochs have been used to find the maximum accuracy rate and minimum mean squared error. The procedure is the same as described in the previous experiment in section 4.2. The local optimized results for all data sets are listed in table 7 below. More specifically, each row in table 7 shows one specific data set benchmarks, and includes the local optimized fuzzy neurons (number between 2 and 20), local optimized epochs (epochs between 1 and 2000), mean and standard deviation of accuracy rate (results of 5 fold cross validation) for both training and testing data set, and mean squared error for both the training and testing data sets.


39

Data set Fuzzy Neurons

Epoch Training data set Mean StDv MSE

Testing data set Mean StDv MSE

Cancer 2 500 96.1 0.44 0.03 96.4 1.48 0.04 Card 20 200 81.4 1.53 0.13 82.2 2.88 0.14 Diabetes 9 400 75.1 1.59 0.17 77.6 1.59 0.17 Gene 18 600 79.7 0.74 0.11 79.4 2.61 0.11 Glass 12 200 65.0 2.33 0.09 63.7 9.94 0.09 Heart 10 50 81.1 0.51 0.14 80.3 3.06 0.14 Heartc 3 200 82.2 1.96 0.13 83.1 9.17 0.12 Horse 3 200 63.9 1.90 0.16 63.2 4.58 0.16 Ionosphere 8 200 91.2 0.92 0.11 90.9 3.12 0.09 SARS 17 5 97.9 0.28 0.08 97.7 1.06 0.08 Soybean 7 2000 55.8 1.34 0.04 53.3 6.29 0.04 Thyroid 3 50 93.1 0.09 0.05 92.1 0.38 0.05 Wine 6 50 95.1 0.92 0.08 96.2 1.52 0.08

Table 7: Local optimized network information

According to the results, there is no common optimized number of fuzzy neurons or epochs in this experiment, and each data set’s accuracy rate is different. For example, the “Cancer” dataset required only 2 fuzzy neurons and achieved 96% accuracy while the “Card” data set required 20 neurons to get 82% accuracy. Once the benchmarks for all data sets have been obtained, the benchmarks from other classification methods are obtained from relevant publication [5][6][8][9][10]. All information is collected and listed in Table 8. In table 8, the fuzzy signature RBF network includes benchmarks for all 13 data sets, the Cascor network results consist of 10 data sets and has no benchmark for data sets “ionosphere”, “SARS” & “Wine”. Symmetric Nearest Neighbor has accuracy rates for data sets “Cancer”, “Diabetes”, “Horse” and “Wine”. The accuracy rates for the Non-linear Bayes method are available for data sets “Diabetes”, “Ionosphere” and “Wine”, and Polymorphic fuzzy signatures method has result for the data set “SARS”. An empty field in Table 8 means the benchmarks for that specific data set cannot be found. The final result shows the average accuracy rate based on the data sets. In general, Polymorphic fuzzy signatures method with only one result got over 95% accuracy rate, and other three methods got similar results (range between 80% and 84%). Cascor method (83.4% on 9 datasets) and non-linear Bayes method (83.1% on 3 datasets) both got more than 83% accuracy on average, fuzzy signature RBF network got more than 81% accuracy rate on 13 datasets and the k-sNN got the lowest result (80.7% on 5 datasets). The fuzzy signature RBF network was best overall on the “Diabetes”, “Heart”, “Heartc”, “Ionosphere”, “Wine” and “SARS” datasets. Please note, the final results in table 8 do not measure which method is better than


40

others overall, since the benchmarks for some data sets are missing. For example, Polymorphic Fuzzy Signatures method has only one benchmark (96.7%) on data set “SARS”, and it is lower than the benchmark (97.9%) from Fuzzy signature RBF network, so it can only be concluded that fuzzy signatures method is better on this dataset. The benchmarks are also compared with related previous work [11]. That produced similar results to the current work for the SARS dataset only.

Data set Fuzzy signature

RBF network Mean StDv

Cascor

Mean StDv

k-sNN

Mean

Nonlinear Bayes Mean

Polymorphic Fuzzy

Signatures

Cancer 96.4 1.48 98.1 0.38 62.5 Card 82.2 2.88 86.4 0.43 Diabetes 77.6 1.59 76.5 1.44 69.8 70.4 Gene 79.4 2.61 86.6 0.47 Glass1 63.7 9.94 65.2 5.88 Heart 80.3 3.06 80.1 1.58 75.1 Heartc 83.1 9.17 80.5 1.28 Horse 63.2 4.58 73.6 2.58 70.9 ionosphere 90.9 3.12 87.5 SARS 97.7 1.06 96.7 Soybean 53.3 6.29 90.5 0.86 Thyroid 92.1 0.38 97.0 1.15 Wine 96.2 1.52 87.64 91.3 Average 81.2 83.4 80.7 83.1 96.7

Table 8: Test accuracy percentage results with different classification approaches Figure 17 represents the benchmarks comparison by using bar diagrams. The leftmost bar for each specific data set shows the testing accuracy rate for the fuzzy signature RBF network, the second bar indicates the testing accuracy rate for the Cascor approach. The third one demonstrates the Symmetric Nearest Neighbor (k-sNN)’s testing accuracy rate. The fourth bar displays the Non-linear Bayes method’s testing accuracy. The rightmost bar for each data set indicates the Polymorphic fuzzy signature method’s testing accuracy. Please note some bars for specific data set are missing since the benchmarks for them cannot be found. According to figure 17, the fuzzy signature RBF network got the best result with data sets “Diabetes”, “Heart”, “Heartc”, “Ionosphere”, “Wine” and “SARS”. Furthermore, it has a reasonably good result on data sets “Cancer”, “Card”, “Heart”, “Thyroid”. However, the worst result for fuzzy signature RBF network has occurred with the data set “Soybean”, where the difference of accuracy between the Cascor and Fuzzy signature network is more than 30%.


41

Overall though, the fuzzy signature RBF network got good, competitive results compared with other approaches in this experiment.

Figure 17: Test accuracy percentage results with different classification approaches

5. Conclusion and future work

5.1 Conclusion

I have implemented the fuzzy signature based RBF neuron network for classification. The techniques and methodologies to implement such a network and suitable test cases have been demonstrated. Using real world data sets from UCI, the benchmarks have been compared with different parameters in the network and different classification methods. The fuzzy signature based RBF neural network achieved good, competitive results in the experiments, demonstrating that this approach is viable and worth further investigation.

50556065707580859095

100

Fuzzy signature RBF network cascork-sNN Non-linear BayesPolymorphic fuzzy signature

Accuracy


42

5.2 Future work

In this report, I have shown the feasibility of fuzzy signature based RBF neuron network with real world data sets. The network got good results in the experiments. Therefore, the most important future work will be to implement and test more aggregation functions for the network, and to automate their selection for optimizing the results. During the experiment, few data sets got extremely low benchmarks. This may be because the experiment focused on getting benchmarks with optimized fuzzy neuron numbers and epochs. Different aggregation functions may affect the network benchmarks significantly. Therefore, it should be considered to implement more complex aggregation methods such as WRAO [6], and compare the benchmarks to see the results. The other possible reason for the observation is that the mapping function that is described in section 3.3.3 is not suitable. In the current implementation, the highest value in the result indicates the category of the observation. In some cases of the experiment, the result shows the observation has high similarity for more than one category and the expected result is the category that has a high similarity value but is not the highest one. Therefore, implementing another mapping function should be considered as well. Another useful direction would be visualizing the implementation. Currently, the implementation is semi-visualized since the program starts with typing commands at a MATLAB terminal window and generating the visualized benchmarks. This could be difficult for users who are MATLAB novices. Therefore, it would be useful to implement a GUI application which starts the interface for novice users and allows them to input data such as network parameters and the benchmarks that need to be compared from other classification methods, and to browse details of the benchmarks files generated.


43

References

[1] Swain, H. Philip and Jon A. Benediktsson, 1990, 'Neural Network Approaches Versus Statistical Methods in Classfication of Multisource Remote Sensing Data', IEEE Transactions on geoscience and remote sensing, pp. 540-551. [2] K.W. Wong, T.D. Gedeon, L.T. Kóczy, 2004, ‘Construction of Fuzzy Signature from Data: An Example of SARs Pre-clinical Diagnosis System’, Proceedings of IEEE International Conference on Fuzzy Systems – FUZZ-IEEE 2004, Budapest, Hungary, p. 1353. [3] T. Vamos, L.T. Koczy, and G. Biro, 2001, ‘Fuzzy Signatures in Data Mining’, Proceedings of the joint 9 IFSA World Congress, 2001, pp. 2842-2846. [4] B.S.U Mendis, T.D. Gedeon, L.T. Kóczy, 2007, ‘On the issue of learning weights from observations for fuzzy signatures’, Proceedings WAC06, Budapest, pp. 1-6. [5] B.S.U. Mendis, T.D. Gedeon, J. Botzheim, L.T. Kóczy, 2007, ‘Generalised weighted relevance aggregation operators for hierarchical fuzzy signatures’, Proceedings International Conference on Intelligent Agents, Web Technologies and Internet Commerce, DOI: 10.1109/CIMCA.2006.110. [6] B.S.U. Mendis, 2008, Fuzzy Signatures: Hierarchical Fuzzy Systems and Applications, PhD thesis, Department of Computer Science, The Australian National University, Australia, March 2008. [7] The UCI Machine Learning Repository, 1987, Center for Machine Learning and Intelligent Systems, viewed 5 September 2011, <http://archive.ics.uci.edu/ml/datasets.html>. [8] N.K. Treadgold, T.D. Gedeon, 2001, ‘Exploring constructive cascade networks’, Industrial Electronics Society, 2001. IECON '01. The 27th Annual Conference of the IEEE, vol. 1, pp. 25-30. [9] Nock, R., Sebban, M., and Bernard, D. 2003, ‘A Simple locally adaptive nearest neighbor rule with application to pollution forecasting’, Internal Journal of Pattern Recognition and Artificial Intelligence, pp.1369-1382. [10] M.L. Raymer, T.E. Doom, L.A. Kuhn, W.F. Punch, 2003 ‘Knowledge Discovery in Medical and Biological Datasets Using a Hybrid Bayes Classifier/Evolutionary Algorithm’, Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, vol. 33, pp. 802-813. [11] P. Chandra, 2010, Fuzzy Signature Neural Network for Rule Discovery, COMP3006 Project Report, Research School of Computer Science, The Australian National University, Australia,

http://archive.ics.uci.edu/ml/datasets.html


44

October 2010. [12] S. Fahlman, and C. Lebiere, 1990. ‘The Cascade-Correlation Learning Architecture,’ Advances in Neural Information Processing Systems 2, D.S. Touretsky, ed., pp. 524-532. [13] Pawel Strumillo, Wladyslaw Kaminski, 2002, ‘Radial Basis Function Neural Networks: Theory and Applications’, proceedings of the Sixth International Conference on Neural Network and Soft Computing, Zakopane, Poland, pp.107-119. [14] Bernardo, José M., and Smith, Adrian F. M., 1994, Bayesian Theory, 1st edn, Wiley, UK. [15] M. D. Buhmann, 2003, Radial Basis Functions: Theory and Implementations, 1st edn, Canmbridge University Press 2003, UK. [16] Deza, M.M., and Deza, E., 2009, Encyclopedia of Distance, Springer 2009, US [17] Eugene F. Krause, 1987, Taxicab Geometry: An Adventure in Non-Euclidean Geometry, Dover Publications 1987, US


45

Appendix A

The files below show an example of the text benchmarks with 9 fuzzy neurons and 100 epochs, the experiment environment is recorded in chapter 4.2.2 File: training_details.txt Weight matrix learnt after run number 1

�−0.644 −0.849 −0.464 0.655 −0.572 −1.964 2.990 −0.607 2.3150.792 0.773 0.689 −0.687 0.654 2.182 −2.613 1.053 −2.344�

Weight matrix learnt after run number 2

�−0.525 −1.090 −0.253 0.852 −0.570 −2.116 3.034 −0.509 2.0440.692 1.015 0.479 −0.889 0.626 2.336 −2.646 0.956 −2.078�


�−0.585 −0.926 −0.271 0.790 −0.643 −1.921 2.854 −0.560 2.1470.736 0.865 0.493 −0.808 0.748 2.096 −2.481 1.017 −2.190�


�−0.447 −1.098 −0.466 0.922 −0.619 −1.954 2.922 −0.509 2.0950.576 1.057 0.695 −0.968 0.678 2.166 −2.543 0.959 −2.109�


�−0.474 −1.024 −0.333 0.894 −0.563 −2.194 3.004 −0.589 2.0840.580 0.977 0.590 −0.937 0.638 2.391 −2.598 1.049 −2.142�

First 100 lines of file: testing_details.txt Failed results in run number 1 Observation 4 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.73 0.27 Observation 17 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.67 0.33


46

Observation 19 does not match the desired result Desired classification - 1 0 Neural net classification (rounded) - 0 1 Neural net classification (actual) - 0.48 0.50 Observation 31 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.63 0.41 Observation 63 does not match the desired result Desired classification - 1 0 Neural net classification (rounded) - 0 1 Neural net classification (actual) - 0.47 0.51 Observation 112 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.60 0.42 Observation 132 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.85 0.17 Observation 133 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.63 0.43 Observation 140 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.66 0.36 Observation 146 does not match the desired result Desired classification - 1 0 Neural net classification (rounded) - 0 1 Neural net classification (actual) - 0.32 0.67 Observation 182 does not match the desired result


47

Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.80 0.20 Observation 188 does not match the desired result Desired classification - 1 0 Neural net classification (rounded) - 0 1 Neural net classification (actual) - 0.30 0.69 Observation 198 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.64 0.42 Observation 207 does not match the desired result Desired classification - 1 0 Neural net classification (rounded) - 0 1 Neural net classification (actual) - 0.44 0.56 Observation 211 does not match the desired result Desired classification - 1 0 Neural net classification (rounded) - 0 1 Neural net classification (actual) - 0.49 0.54 Observation 212 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.89 0.12 Observation 230 does not match the desired result Desired classification - 1 0 Neural net classification (rounded) - 0 1 Neural net classification (actual) - 0.22 0.79 Observation 291 does not match the desired result Desired classification - 1 0 Neural net classification (rounded) - 0 1 Neural net classification (actual) - 0.44 0.53 Observation 293 does not match the desired result Desired classification - 1 0 Neural net classification (rounded) - 0 1


48

Neural net classification (actual) - 0.45 0.51 Observation 318 does not match the desired result Desired classification - 0 1 Neural net classification (rounded) - 1 0 Neural net classification (actual) - 0.69 0.36

File: cluster_details.txt

Details for 9 clusters

Details for cluster number 1 with number of elements = 73 are: Percentage of members in class 1 is 22 30.2 % Percentage of members in class 2 is 51 69.8 %

Cluster centroid - [0.605, 0.808, 0.151, 0.041, 0.151, 0.658, 0.000, 0.463, 0.000, 0.319, 0.041, 0.000, 0.123, 0.027, 0.000, 0.000, 1.000, 0.000, 0.533, 0.000, 0.000, 0.438, 0.000, 0.466, 0.000, 0.082, 0.890, 0.027, 0.000, 0.338, 0.014, 0.274, 0.123, 0.575, 0.027] Minimum values of each argument [0.240, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.167, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 1.000, 0.000, 0.200, 0.000, 0.000, 0.000, 0.000, 0.205, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000] Maximum values of each argument [0.880, 1.000, 1.000, 1.000, 1.000, 1.000 , 0.000, 0.933, 0.000, 0.921,1.000, 0.000, 1.000, 1.000, 0.000, 0.000, 1.000, 0.000, 0.900, 0.000, 0.000, 1.000, 0.000, 0.773, 0.000, 1.000, 1.000, 1.000, 0.000, 1.000, 1.000, 1.000, 1.000, 1.000,1.000] Details for cluster number 2 with number of elements = 116 are: Percentage of members in class 1 is 23 19.8% Percentage of members in class 2 is 93 80.2% Cluster centroid - [0.561, 0.905, 0.034, 0.052, 0.060, 0.853, 0.000, 0.452, 0.009, 0.239, 0.207, 0.000, 0.259, 0.026, 0.534, 0.207, 0.259, 0.000, 0.490, 0.000, 0.000, 0.629, 0.000, 0.436, 0.009, 0.388, 0.388, 0.172, 0.052, 0.172, 0.371, 0.034, 0.052,0.862, 0.052] Minimum values of each argument [0.140, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000,0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000] Maximum values of each argument [0.940, 1.000, 1.000, 1.000, 1.000, 1.000, 0.000, 1.000, 1.000, 0.515, 1.000, 0.000, 1.000, 1.000,


49

1.000, 1.000, 1.000, 0.000, 0.893, 0.000, 0.000, 1.000, 0.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000] Details for cluster number 3 with number of elements = 106 are: Percentage of members in class 1 is 9 8.5% Percentage of members in class 2 is 97 91.5% Cluster centroid - [0.492, 0.915, 0.000, 0.000, 0.000, 1.000, 0.000, 0.457, 0.000, 0.216, 0.358, 0.000, 0.094, 0.179, 1.000, 0.000, 0.000, 0.000, 0.412, 0.000, 0.000, 0.792, 0.000, 0.462, 0.000, 0.000, 0.991, 0.009, 0.000, 0.000, 0.953, 0.000, 0.113,0.198, 0.689] Minimum values of each argument [0.060, 0.000, 0.000, 0.000, 0.000, 1.000, 0.000, 0.100, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 1.000, 0.000, 0.000, 0.000, 0.113, 0.000, 0.000, 0.000, 0.000, 0.193, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000] Maximum values of each argument [0.940, 1.000, 0.000, 0.000, 0.000, 1.000, 0.000, 1.000, 0.000, 0.996, 1.000, 0.000, 1.000, 1.000, 1.000, 0.000, 0.000, 0.000, 0.647, 0.000, 0.000, 1.000, 0.000, 0.636, 0.000, 0.000, 1.000, 1.000, 0.000, 0.000, 1.000, 0.000, 1.000, 1.000, 1.000] Details for cluster number 4 with number of elements = 85 are: Percentage of members in class 1 is 32 37.6% Percentage of members in class 2 is 53 62.4% Cluster centroid - [0.539, 0.694, 0.024, 0.224, 0.553, 0.200, 0.000, 0.414, 0.024, 0.183, 0.376, 0.000, 0.106, 0.071, 0.835, 0.141, 0.024, 0.000, 0.491, 0.000, 0.000, 0.388, 0.000, 0.429, 0.000, 0.035, 0.894, 0.071, 0.000, 0.086, 0.647, 0.424, 0.024, 0.141, 0.412] Minimum values of each argument [0.160, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.068, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000] Maximum values of each argument [0.960, 1.000, 1.000, 1.000, 1.000, 1.000, 0.000, 1.000, 1.000, 0.592, 1.000, 0.000, 1.000, 1.000, 1.000, 1.000, 1.000, 0.000, 0.847, 0.000, 0.000, 1.000, 0.000, 0.705, 0.000, 1.000, 1.000, 1.000, 0.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000] Details for cluster number 5 with number of elements = 62 are: Percentage of members in class 1 is 16 25.8% Percentage of members in class 2 is 46 74.2%


50

Cluster centroid - [0.579, 0.871, 0.016, 0.000, 0.016, 0.968, 0.000, 0.466, 0.016, 0.281, 0.129, 0.000, 0.242, 0.048, 0.000, 1.000, 0.000, 0.000, 0.432, 0.000, 0.000, 0.710, 0.000, 0.439, 0.000, 0.065, 0.548, 0.097, 0.290, 0.000, 1.000, 0.000, 0.000, 0.000, 1.000] Minimum values of each argument [0.180, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 1.000, 0.000, 0.000, 0.147, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 1.000, 0.000, 0.000, 0.000, 1.000] Maximum values of each argument [0.980, 1.000, 1.000, 0.000, 1.000, 1.000, 0.000, 0.833, 1.000, 0.583, 1.000, 0.000, 1.000, 1.000, 0.000, 1.000, 0.000, 0.000, 0.827, 0.000, 0.000, 1.000, 0.000, 0.864, 0.000, 1.000, 1.000, 1.000, 1.000, 0.000, 1.000, 0.000, 0.000, 0.000, 1.000] Details for cluster number 6 with number of elements = 57 are: Percentage of members in class 1 is 12 21.1% Percentage of members in class 2 is 45 78.9% Cluster centroid - [0.547, 0.912, 0.018 , 0.088, 0.070, 0.825, 0.000, 0.361, 0.000, 0.073, 0.772, 0.000, 0.053, 0.649, 0.719, 0.246, 0.018, 0.018, 0.485, 0.000, 0.000, 0.421, 0.000, 0.454, 0.000, 0.579, 0.000, 0.404,0.018, 0.023, 0.930, 0.000, 0.000, 0.000, 1.000] Minimum values of each argument [0.080, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.060, 0.000, 0.000, 0.000, 0.000, 0.125, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000,1.000] Maximum values of each argument [0.920, 1.000, 1.000, 1.000, 1.000, 1.000, 0.000, 0.750, 0.000, 0.717, 1.000, 0.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 0.813, 0.000, 0.000, 1.000, 0.000, 0.750, 0.000, 1.000, 0.000, 1.000, 1.000, 0.667, 1.000, 0.000, 0.000, 0.000, 1.000] Details for cluster number 7 with number of elements = 124 are: Percentage of members in class 1 is 106 85.5% Percentage of members in class 2 is 18 14.5% Cluster centroid - [0.489, 0.532, 0.073, 0.306, 0.419, 0.202, 0.000, 0.417, 0.000, 0.310, 0.000, 0.000, 0.121, 0.000, 0.500, 0.008, 0.492, 0.000, 0.670, 0.000, 0.000, 0.105, 0.000, 0.346, 0.000, 0.831, 0.129, 0.040, 0.000, 0.126, 0.024, 0.992, 0.000, 0.000, 0.008] Minimum values of each argument [0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.117, 0.000, 0.123, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.240, 0.000, 0.000, 0.000, 0.000, 0.295, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000]


51

Maximum values of each argument [0.980, 1.000, 1.000, 1.000, 1.000, 1.000, 0.000, 0.833, 0.000, 0.638, 0.000, 0.000, 1.000, 0.000, 1.000, 1.000, 1.000, 0.000, 0.947, 0.000, 0.000, 1.000, 0.000, 0.636, 0.000, 1.000, 1.000, 1.000, 0.000, 1.000, 1.000, 1.000, 0.000, 0.000, 1.000] Details for cluster number 8 with number of elements = 55 are: Percentage of members in class 1 is 20 36.4% Percentage of members in class 2 is 35 63.6% Cluster centroid - [0.628, 0.964, 0.073, 0.127, 0.327, 0.473, 0.000, 0.000, 1.000, 0.233, 0.236, 0.000, 0.382, 0.000, 0.345, 0.527, 0.127, 0.000, 0.000, 1.000, 0.000, 0.000, 1.000, 0.010, 0.982, 0.018, 0.000, 0.000, 0.982, 0.000, 1.000, 0.000, 0.000, 0.036, 0.964] Minimum values of each argument [0.140, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 1.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 1.000, 0.000, 0.000, 1.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 1.000, 0.000, 0.000, 0.000, 0.000] Maximum values of each argument [0.920, 1.000, 1.000, 1.000, 1.000, 1.000, 0.000, 0.000, 1.000, 0.577, 1.000, 0.000, 1.000, 0.000, 1.000, 1.000, 1.000, 0.000, 0.000, 1.000, 0.000, 0.000, 1.000, 0.523, 1.000, 1.000, 0.000, 0.000, 1.000, 0.000, 1.000, 0.000, 0.000, 1.000,1.000] Details for cluster number 9 with number of elements 242 are: Percentage of members in class 1 is 171 70.7% Percentage of members in class 2 is 71 29.3% Cluster centroid - [0.413, 0.748, 0.058, 0.397, 0.264, 0.281, 0.000, 0.428, 0.004, 0.253, 0.165, 0.000, 0.107, 0.083, 0.785, 0.153, 0.058, 0.004, 0.531, 0.000, 0.000, 0.140, 0.000, 0.300, 0.029, 0.033, 0.017, 0.000, 0.950, 0.003, 0.983, 0.054, 0.070, 0.062, 0.814] Minimum values of each argument [0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.047, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000] Maximum values of each argument [0.920, 1.000, 1.000 , 1.000, 1.000, 1.000, 0.000, 1.000, 1.000, 0.854, 1.000, 0.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 0.867, 0.000, 0.000, 1.000, 0.000, 0.580, 1.000 , 1.000, 1.000, 0.000 , 1.000, 0.333, 1.000, 1.000, 1.000, 1.000, 1.000] File: final_result.txt


52

Final result for iteration 1 Training data set success rate = 0.800 Testing data set success rate = 0.837 Training data set mean square error = 0.135 Testing data set mean square error = 0.133 Final result for iteration 2 Training data set success rate = 0.808 Testing data set success rate = 0.772 Training data set mean square error = 0.141 Testing data set mean square error = 0.159 Final result for iteration 3 Training data set success rate = 0.796 Testing data set success rate = 0.853 Training data set mean square error = 0.140 Testing data set mean square error = 0.115 Final result for iteration 4 Training data set success rate = 0.800 Testing data set success rate = 0.821 Training data set mean square error = 0.138 Testing data set mean square error = 0.130 Final result for iteration 5 Training data set success rate = 0.815 Testing data set success rate = 0.761 Training data set mean square error = 0.134 Testing data set mean square error = 0.147 Final benchmarks Average training success rate = 0.804 Average testing success rate = 0.809 Average training mean square error = 0.138 Average testing mean square error = 0.137


53

(a) Class distribution for six clusters

(b) Class distribution for seven clusters

(c) Class distribution for eight clusters

41 28

106

45 20

171 150

91

18

144

35

71

0

50

100

150

200

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6

Class 1 Class 2

16 12

41

106

45 20

171

46 45

150

18

144

35

71

0

50

100

150

200

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7

Class 1 Class 2

9 32

16 12

106

45 20

171

97

53 46 45 18

144

35

71

0

50

100

150

200

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7 Cluster 8

Class 1 Class 2


54

(d) Class distribution for nine clusters

Figure 18: Class distribution with number of clusters from 6 to 9

(a) Hinton diagram with 2 fuzzy neurons (b) with 3 fuzzy neurons

(c) with 4 fuzzy neurons (d) with 5 fuzzy neurons

(e) with 6 fuzzy neurons (f) with 7 fuzzy neurons

22 23 9

32 16 12

106

20

171

51

93 97

53 46 45 18

35

71

0

50

100

150

200

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7 Cluster 8 Cluster 9

Class 1 Class 2


55

(g) Hinton diagram with 8 fuzzy neurons (h) with 9 fuzzy neurons

(i) with 10 fuzzy neurons (j) with 11 fuzzy neurons

(k) with 12 fuzzy neurons (l) with 13 fuzzy neuron

(m) with 14 fuzzy neurons (n) with 15 fuzzy neurons

Figure 19: Hinton diagrams with number of fuzzy neurons from 2 to 15

fuzzy signature based radial basis neural...

Documents