Neural Networks

Chapter 8

8.1 Feed-Forward Neural Networks

Figure 8.1 A fully connected feed-forward neural network

Node 1

Node 2

Node i

Node j

Node k

Node 3

Input Layer Output LayerHidden Layer

1.0

0.7

0.4

Wjk

Wik

W3i

W3j

W2i

W2j

W1i

W1j

Table 8.1 • Initial Weight Values for the Neural Network Shown in Figure 8.1

Wlj

Wli

W2j

W2i

W3j

W3i

Wjk

Wik

0.20 0.10 0.30 –0.10 –0.10 0.20 0.10 0.50

Equation 8.1

Neural Network Input Format

valueattribute possiblelargest theis uemaximumVal

attribute for the valuepossiblesmallest theis ueminimumVal

converted be to value theis lueoriginalVa

range interval [0,1] thein falling valuecomputed theis newValue

where

ueminimumValuemaximumVal

ueminimumVallueoriginalVanewValue

Neural Network Output Format

Equation 8.2

The Sigmoid Function

2.718282.by edapproximat logarithms natural of base theis

where

1

1)(

e

xexf

Figure 8.2 The sigmoid function

0.000

0.200

0.400

0.600

0.800

1.000

1.200

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6

f(x)

x

8.2 Neural Network Training: A Conceptual View

Supervised Learning with Feed-Forward Networks

• Backpropagation Learning

• Genetic Learning

Table 8.2 • A Population of Weight Elements for the Network in Figure 8.1

PopulationElement

Wlj

Wli

W2j

W2i

W3j

W3i

Wjk

Wik

1 0.20 0.10 0.30 –0.10 –0.10 0.20 0.10 0.502 0.14 0.38 0.19 0.25 –0.17 0.27 0.11 0.543 0.20 0.10 0.38 –0.16 –0.16 0.24 0.12 0.534 0.23 0.10 0.39 –0.18 –0.17 0.26 0.15 0.54

Unsupervised Clustering with Self-Organizing Maps

Figure 8.3 A 3x3 Kohonen network with two input layer nodes

Output Layer

Input Layer

Node 2Node 1

8.3 Neural Network Explanation

• Sensitivity Analysis

• Average Member Technique

8.4 General Considerations

• What input attributes will be used to build the network? • How will the network output be represented?• How many hidden layers should the network contain?• How many nodes should there be in each hidden layer?• What condition will terminate network training?

Neural Network Strengths

• Work well with noisy data.• Can process numeric and categorical data.• Appropriate for applications requiring a time element.• Have performed well in several domains.• Appropriate for supervised learning and unsupervised clustering.

Weaknesses

• Lack explanation capabilities.• May not provide optimal solutions to problems.• Overtraining can be a problem.

8.5 Neural Network Training: A Detailed View

The Backpropagation Algorithm: An Example

Equation 8.3

Backpropagation Error Output Layer

k nodeat function sigmoid theinput to the

function sigmoid theof derivativeorder -first The)('

erroroutput actual The)(

k nodeat output computed The

output target The

where

)]('[)()(

k

k

k

k

kk

x

xf

OT

O

T

xfOTkError

Equation 8.4

Backpropagation Error Output Layer

)1()()( kkk OOOTkError

Equation 8.5

Backpropagation Error Hidden Layer

)1(

toevaluates )(' 8.3, Eq.in As j. nodeat function sigmoid theinput to The

function sigmoid theof derivativeorder -first The)('

k nodeoutput and j node betweenlink the withassociated weight The

k nodeat error output computed The)(

where

)(')()(

jj

jj

j

jk

jk

jk

OO

xfx

xf

W

kError

xfWkErrorjError

Equations 8.6 and 8.7

The Delta Rule

jkjkjk wcurrentwneww )()(

j node ofoutput The

k nodeat error computed The)(

01 withparameter rate learning The

where

))](()[(

j

jjk

O

kError

rr

OkErrorrw

Equation 8.8

Root Mean Squared Error

nodeoutput th and instance th for theoutput computed the

nodeoutput th theand instance nth for theoutput target the

nodesoutput ofnumber total the

instancesset trainingofnumber totalthe

where

)(

inO

iT

i

n

ni

n iOT

in

in

inin

Kohonen Self-Organizing Maps: An Example

Figure 8.4 Connections for two output layer nodes

Node 1

Node 2 Node j

Input Layer Output Layer

0.4

0.7

W1i = .2Node i

W2j = .6

W1j = .3

W2i = .1

Equation 8.9

Classifying a New Instance Output Node = j

jnodeoutputatnodeinputthithewithassociatedweighttheisijw

inodeinputatvalueattributetheisin

ijwi in2)(

Equation 8.10

Adjusting the Weight Vectors Output Node = j

10

)(

where

)()(

r

wnrw

wcurrentwneww

ijiij

ijijij

Building Neural Networks with iDA

Chapter 9

9.1 A Four-Step Approach for Backpropagation Learning

1. Prepare the data to be mined.

2. Define the network architecture.

3. Watch the network train.

4. Read and interpret summary results.

Example 1: Modeling the Exclusive-OR Function

Table 9.1 • The Exclusive-OR Function

Input 1 Input 2 XOR

1 1 00 1 11 0 10 0 0

Figure 9.1A graph of the XOR function

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Input 2

Input 1

A B

AB

Step 1: Prepare The Data To Be Mined

Figure 9.2 XOR training data

Step 2: Define The Network Architecture

Figure 9.3 Dialog box for supervised learning

Figure 9.4 Training options for backpropagation learning

Step 3: Watch The Network Train

Figure 9.5 Neural network execution window

Step 4: Read and Interpret Summary Results

Figure 9.6 XOR output file for Experiment 1

Figure 9.7 XOR output file for Experiment 2

Example 2: The Satellite Image Dataset

Step 1: Prepare The Data To Be Mined

Figure 9.8 Satellite image data

Step 2: Define The Network Architecture

Figure 9.9 Backpropagation learning parameters for the satellite image data

Step 3: Watch The Network Train

Step 4: Read And Interpret Summary Results

Figure 9.10 Statistics for the satellite image data

Figure 9.11 Satellite image data: Actual and computed output

9.2 A Four-Step Approach for Neural Network Clustering

Step 1: Prepare The Data To Be Mined

The Deer Hunter Dataset

Step 2: Define The Network Architecture

Figure 9.12 Learning parameters for unsupervised clustering

Step 3: Watch The Network Train

Figure 9.13 Network execution window

Step 4: Read And Interpret Summary Results

Figure 9.14 Deer hunter data: Unsupervised summary statistics

Figure 9.15 Output clusters for the deer hunter dataset

9.3 ESX for Neural Network Cluster Analysis