back propagation network by bhabanath sahu

7/30/2019 Back Propagation Network by bhabanath sahu

1/21

Back Propagation Network


2/21


What is BPN?

A single-layer NN has many restriction.

limited classes of task.

Minsky ans Papert (1969) showed that a two

layer FF Network can over come many

restriction, but they did not present the

solution to the problem as how to adjust the

weight from input to hidden layer?


3/21

Back Propagation Network (Continue)

In 1986 an answer was presented by

Rumelhart, Hinton and williams.

The central idea behind this solution is that,

the error for the hidden layer units are

determined by back-propagating the errors of

the output layer units.

This method is often called the Back-

propagation learning rule.


4/21


Back-propagation is a systematic method of

training artificial neural networks.

A BackProp network consist of at least three

layers of units:

An input layer,

At least one intermediate hidden layer, and

An output layer.


5/21


Units are connected in a feed-forward fashion

(i.e. i/p layer neuron is fully connected to

hidden layer neuron and hidden layer neuron

is fully connected to output layer neuron.)

When a BackProp n/w is cycles, an input

pattern is propagated forward to the o/p units

through the i/p to hidden and hidden to o/pweights.

The o/p of BackProp n/w is interpreted as a

classification decision or result of application.


6/21


With BPN, learning occurs during trainingphase. The steps are:

1. Each i/p pattern in a training set is applied to

the i/p units and then propagated forward.2. At the o/p layer error is calculated. (i.e.

compression between n/w o/p and target).

3. The error signal for each such target o/ppattern is then back-propagated from the o/pto the i/p in order to adjust the weights ineach layer of the n/w


7/21


4. After BackProp n/w has learned the correct

classification for a set of i/ps, it can be tested

for untrained pattern.

Learning: implement AND function in NN

AND function

AND

X1 X2 Y

0 0 0

0 1 0

1 0 0

1 1 1

X1

X2

Y

I1

I2

Output OW1

W2


8/21


There are 4 inequalities in the AND function

and they must be satisfied.

W1

0 + W2

0 < , W1

0 + W2

1 < ,

W11 + W20 < , W11 + W21 >

One possible solution: if both weight are set

to 1 and the threshold is set to 1.5, then

(1)(0) + (1)(0) < 1.5 assign 0, (1)(0) + (1)(1) < 1.5 assign 0,

(1)(1) + (1)(0) < 1.5 assign 0, (1)(1) + (1)(1) > 1.5 assign 1,


9/21


Example 1

AND Problem

AND

X1 X2 Y

0 0 0

0 1 0

1 0 0

1 1 1

X1

X2

Y

I1

I2

Output O

W1

W2

The o/p of the n/w is determined by

calculating a weighted sum of its two i/p and

comparing this value with a threshold .

if the net input (net) is greater than , than

the o/p is 1, else it is 0.

the computation is performed by the o/p

unit is.

net= W1 I1 + W2 I2

if net > then O = 1, otherwise O = 0.


10/21

Simple Learning Machines

Rosenblatt proposed learning n/w called

Perceptron.

A perceptron consist of a set of input units

and a single output unit.

As in the AND N/w, the output of the

perceptron is calculated by comparing the net

i/p net = (Wi Ii) where i = 1 to n, and a

threshold .

If net > the O/p = 1, else O/p = 0


11/21

Simple Learning Machines (Continue)

The learning rule that Rosenblatt developed, is

based on determining the difference between

the actual o/p of the n/w with the target o/p

(0 or 1), called ErrorMeasure.

Error Measure (Learning rule): Difference

between actual o/p of the n/w with target o/p

(0 or 1).


12/21


Error Measure (Learning rule) : Difference

between actual o/p of the n/w with target o/p

(0 or 1).

If the i/p vector is correctly classified (i.e., zero

error), then the weight are left unchanged, and

the next i/p vector is presented.

Else there are two cases to consider


13/21


Case 1: if the o/p unit is 1 and need to be 0 then.

the threshold is incremented by 1.

if the i/p Ii is 0, then the corresponding weight Wi is

left unchanged.

If the i/p Ii is 1, then the corresponding weight Wi is

decreased by 1.

Case 2: If O/p unit is 0 but need to be 1 then theopposite changes are made.


14/21


Perceptron Learning Rule : Equations

The perceptron learning rules are govern by two

equations,

1. Change in threshold

2. Change in weights.

The change in threshold is given by = -(tp op) = -dp

The change in the weight are given by

Wi = (tp op) Ipi = dp Ipi


15/21


Learning by Example: Consider the Multilayer

feed-forward back-propagation network

1 1 1

2 2 2

l m n

II1

II2

IIl

OI1

OI2

OIl

V11

V21

Vij

IH1

IH2

IHm

OH1

OH2

OHm

W11W21

Wij

IO1

IO2

IOn

OO1

OO2

OOn

Input Layer i nodes Hidden Layer m nodes O/p Layer n nodes


16/21


Consider a problem in which an nset of l

inputs and the corresponding nset of n

output data is shown in table

No Input Output

l1

L2..

ll

O1

O2

O

l

1 0.3 0.4 0.8 0.1 0.56 0.82

2

.

.

Nset


17/21

Input Layer Computation

Consider linear activation function.

if the o/p of input layer is the input of the i/p

layer and the transfer function is 1, then

{ O }I = { I }I

lx 1 lx 1(denotes matrix row, column size)

The i/p to the hidden neuron is the weightedsum of the o/ps of the i/p neurons. Thus

IHP = V1pOI1 + V2p OI2+ .+ V1p OIl so that

{ I }H = [V]T

{ O }I


18/21

Hidden Layer Computation

Consider sigmoid function. The output of the pth

hidden neuron is given by

OHP

= 1/(1 + e-(IHP-HP))

The input to the o/p neuron is the weighted sum

of the o/p of the hidden neuron. Ioq the input to

the qth output neuron is given by

Ioq = W1q Ohp + W2q OH2+.+ Wmq Ohm

So that,

{ I }o = [W]T

{ O }H


19/21

Output Layer Computation

Consider sigmoid function. The output of the qth

output neuron is given by

Ooq

= 1/(1 + e-(Ioq-oq)

Where : Ooq is the o/p of the qth o/p neuron,

Ioq is the i/p of the qth o/p neuron,

oq is the threshold of the qth

o/pneuron,


20/21

Calculation of Error

Consider any rth output neuron,

The error norm in the output for the rth output

neuron is calculated by:

E1r = (1/2) e2

r = (1/2) (T-O)2


21/21

Basic algorithm loop structure

Initialize the weight

Repeat

foreach training pattern

Train on that pattern

EndUntil the error is acceptably low.

back propagation network by bhabanath sahu

Documents