neural network 1

8/8/2019 Neural Network 1

1/34

Intro to Neural NetworksIntro to Neural NetworksNiranjan PandaNiranjan Panda

29.10.09


2/34

Before we startBefore we start

Information processing technologyInformation processing technologyinspired by studies of brain and theinspired by studies of brain and the

nervous system.nervous system.


3/34

What is a Neural Network?What is a Neural Network?

"...a computing system made up of a number of"...a computing system made up of a number of

simple, highly interconnected processingsimple, highly interconnected processing

elements, which process information by theirelements, which process information by their

dynamic state response to external inputs.dynamic state response to external inputs.


4/34

Biological NeuronsBiological Neurons

Inputs Outputs

Connection between cells


5/34

ContdContd

The brain is made up of neurons which have

A cell body (soma)

Dendrites (inputs)

An axon (outputs)

Synapses (connection between cells)

There are around 1011 neurons, 1014 synapses

are present in human body and they are connected

in massively parallel.


6/34

What is a Artificial Neural Network?What is a Artificial Neural Network?

An ANN is a network of many very simpleAn ANN is a network of many very simple

processors ( units ), each possibly having a ( smallprocessors ( units ), each possibly having a ( small

amount of) local memory. The units are connected byamount of) local memory. The units are connected byunidirectional communication channels (connections"),unidirectional communication channels (connections"),

which carry numeric ( as opposed to symbolic ) data.which carry numeric ( as opposed to symbolic ) data.

The units operate only on their local data and on theThe units operate only on their local data and on the

inputs they receive via the connections.inputs they receive via the connections.


7/34

HISTORICALHISTORICAL

19431943 ------ McCulloch and Pitts (start of the modern eraMcCulloch and Pitts (start of the modern eraof neural networks).of neural networks). Logical calculus of neuralLogical calculus of neuralnetworks. A network consists of sufficient number ofnetworks. A network consists of sufficient number ofneurons (using a simple model) and properly setneurons (using a simple model) and properly setsynaptic connections can compute any computablesynaptic connections can compute any computable

function.function.

19491949 ------ Hebb's book "The organization ofHebb's book "The organization ofbehavior".behavior". An explicit statement of a physiologicalAn explicit statement of a physiologicallearning rule for synaptic modification was presentedlearning rule for synaptic modification was presented

for the first time.for the first time.

Hebb proposes that the connectivity of the brain isHebb proposes that the connectivity of the brain iscontinually changing as an organism learns differingcontinually changing as an organism learns differingfunctional tasks, and that neural assemblies arefunctional tasks, and that neural assemblies are

created by such changes.created by such changes.


8/34


19581958 ------ Rosenblatt introduced Perceptron ARosenblatt introduced Perceptron Anovel method of supervised learning. Perceptronnovel method of supervised learning. Perceptronconvergence theorem.Least meanconvergence theorem.Least mean--square (LMS)square (LMS)algorithmalgorithm

19691969 ------ Minsky and Papert showed limits onMinsky and Papert showed limits onperceptron computation. Minsky and Papertperceptron computation. Minsky and Papertshowed that there are fundamental limits onshowed that there are fundamental limits on

what singlewhat single--layer perceptrons can compute.layer perceptrons can compute.

19821982 ------ Hopfield's networks Hopfield showedHopfield's networks Hopfield showedhow to use "Ising spin glass" type of model tohow to use "Ising spin glass" type of model tostore information in dynamically stable networks.store information in dynamically stable networks.


9/34


19821982 ------ Kohonen's selfKohonen's self--organizing maps (SOM)organizing maps (SOM)Kohonen's selfKohonen's self--organizing maps is capable oforganizing maps is capable ofreproducing important aspects of the structure ofreproducing important aspects of the structure ofbiological neural nets: Data representation usingbiological neural nets: Data representation usingtopographic maps (which are common in the nervoustopographic maps (which are common in the nervous

systems). SOM also has a wide range of applications.systems). SOM also has a wide range of applications.

19851985 ------ Ackley, Hinton, and Sejnowski, developedAckley, Hinton, and Sejnowski, developedBoltzmann machine, which was the first successfulBoltzmann machine, which was the first successful

realization of a multilayer neural network.realization of a multilayer neural network.

19861986 ------ Rumelhart, Hinton, and Williams developedRumelhart, Hinton, and Williams developedthe backthe back--propagation algorithmpropagation algorithm ------ the most popularthe most popularlearning algorithm for the training of multilayerlearning algorithm for the training of multilayer

perceptrons.perceptrons.


10/34

Why Neural Networks?Why Neural Networks?

Adaptive learning:Adaptive learning:An ability to learn how to doAn ability to learn how to dotasks based on the data given for training ortasks based on the data given for training orinitial experience.initial experience.

SelfSelf--OrganizationOrganization: An ANN can create its own: An ANN can create its own

organization or representation of the informationorganization or representation of the informationit receives during learning time.it receives during learning time.

Real Time OperationReal Time Operation: ANN computations may: ANN computations maybe carried out in parallel, and special hardwarebe carried out in parallel, and special hardwaredevices are being designed and manufactureddevices are being designed and manufactured

which take advantage of this capability.which take advantage of this capability.Fault Tolerance via Redundant InformationFault Tolerance via Redundant InformationCodingCoding: Partial destruction of a network leads to: Partial destruction of a network leads tothe corresponding degradation of performance.the corresponding degradation of performance.However, some network capabilities may beHowever, some network capabilities may be

retained even with major network damage.retained even with major network damage.


11/34

Artificial Neural NetworkArtificial Neural Network

ANNs incorporate the two fundamental components

of biological neural networks:

1. Neurons (nodes)

The basic computational unit.

2. Synapses (weights)

Connection links characterizedby certain weight known as synaptic weight.


12/34

Synapse

vs.

weight

Neuron

vs.

Node


13/34

Application areaApplication area

AerospaceAerospace

AutomotiveAutomotiveBankingBanking

DefenseDefense

ElectronicsElectronicsEntertainmentEntertainment

Finance &Finance &

InsuranceInsurance

Several areas are there where Neural Network works:Several areas are there where Neural Network works:

ManufacturingManufacturing

MedicalMedical

ApplicationsApplications

Oil and GasOil and Gas

RoboticsRobotics

Speech ProcessingSpeech Processing

TelecommunicationTelecommunication

TransportationTransportation


14/34

Example ApplicationsExample Applications

NETtalk (Sejnowski and Rosenberg, 1987)NETtalk (Sejnowski and Rosenberg, 1987)

Maps character strings into phonemes forMaps character strings into phonemes forlearning speech from text.learning speech from text.

Neurogammon (Tesauro and Sejnowski, 1989)Neurogammon (Tesauro and Sejnowski, 1989) Backgammon learning programBackgammon learning program

Speech recognition (Waibel, 1989)Speech recognition (Waibel, 1989)

Converts sound to textConverts sound to text

Character recognition (Le Cun et al., 1989)Character recognition (Le Cun et al., 1989)

Face Recognition (Mitchell)Face Recognition (Mitchell)

ALVINN (Pomerleau, 1988)ALVINN (Pomerleau, 1988)


15/34

A Simple Artificial Neuron

Synapticweights

wij

Summing

function

Input

signal

Bias

b

Activation

functionLocal

Field

vOutput

y

x1

xj

xm wim

wi1

/ /

)(N/


16/34

Contd..

The neuron is the basic information processing unit of aThe neuron is the basic information processing unit of a

NN. It consists of:NN. It consists of:

11 A set ofA set ofsynapsessynapses ororconnecting linksconnecting links, each link, each link

characterized by acharacterized by a weightweight known asknown as synaptic weightsynaptic weight::WWi1i1 , W, Wi2i2, ,W, ,Wiijj ,, W,, Wimim

22 AnAn adderadderfunction (linear combiner)function (linear combiner)

which computes the weighted sumwhich computes the weighted sum

of the inputs:of the inputs:

33 Activation functionActivation function (squashing function)(squashing function) for limitingfor limiting

the amplitude of the output of the neuron.the amplitude of the output of the neuron.

! !

m

1 jijxwu j

)(uy b!N


17/34

Contd..Contd..

BiasBias b (b (weight is w0 but with a fixed input of x0 =1weight is w0 but with a fixed input of x0 =1))has the effect of applying anhas the effect of applying an affine transformationaffine transformation toto uu

i.e.i.e. it serves the purpose of increasing or decreasingit serves the purpose of increasing or decreasingthe net input of activation function depending onthe net input of activation function depending on

whether it is +ve orwhether it is +ve or ve.ve.

v = u + bv = u + b

vvis theis the induced fieldinduced field of the neuronof the neuron

!!

m

1jijxwu

j

v

u


18/34

Contd..Contd..

Input

signal

Synaptic

weights

Summing

function

Activation

functionLocal

Field

vOutput

y

x1

x2

xm

w2

wm

w1

/ / )(N

w0x0= +1

bw

xwvm

i

!

! !

0

0

Bias is an external parameter of the neuron.Can be modeled by adding an extra input.


19/34

Activation FunctionsActivation Functions

Threshold Function:Threshold Function:

(v) = 1 , v(v) = 1 , v 0 0

= 0 , v < 0= 0 , v < 0

(v) output

1

0 v input


20/34

Contd..Contd..

Signum Function:Signum Function:

(v) = +1 , v >(v) = +1 , v >

== --1 , v 1 , v

(v) output

1

0 v input

-1


21/34


22/34

Contd..Contd..

Sigmoid Function:Sigmoid Function:

(v) = where(v) = where is the slope parameter

= 0.5 , v = 0= 0.5 , v = 0

= 1 , v = = 1 , v =

= 0 , v == 0 , v = --

1

1 + exp ( -v )


23/34

Contd..Contd..

Hyperbolic Tangent Function:Hyperbolic Tangent Function:

(v) = tan hv(v) = tan hv

(v) 1 , v infinite(v) 1 , v infinite

--1 , v1 , v --infiniteinfinite


24/34


25/34

Single Layer Feedforward NetworkSingle Layer Feedforward Network

The nodes on the left are in the soThe nodes on the left are in the so--calledcalled input layer.input layer.

The input layer neurons are to only pass and distribute theThe input layer neurons are to only pass and distribute the

inputs and perform no computation. Thus, the only true layer ofinputs and perform no computation. Thus, the only true layer of

neurons is the one on the right.neurons is the one on the right.

Each of the inputsEach of the inputsx1,x2,xNx1,x2,xN is connected to every artificialis connected to every artificial

neuron in the output layer through the connection weight.neuron in the output layer through the connection weight.Since every value of outputsSince every value of outputs y1,y2,yNy1,y2,yNisis

calculated from the same set ofcalculated from the same set of

input values, eachinput values, each

output is aried basedoutput is aried based

on the connection weights.on the connection weights.

Although the presented network isAlthough the presented network isfully connectedfully connected, the true biological neural, the true biological neural

network may not have all possible connectionsnetwork may not have all possible connections -- the weight value of zero can bethe weight value of zero can be`` "`` "

I

nputlayer Outputlayer


26/34

Multilayer Feedforward NetworkMultilayer Feedforward Network

Input

layer

Hidden Layer

Output

layer

To achieve higher level of computational capabilities, a moreTo achieve higher level of computational capabilities, a more

complex structure of neural network is required. Figure showscomplex structure of neural network is required. Figure showsthethe multilayer neural networkmultilayer neural networkwhich distinguishes itself from thewhich distinguishes itself from thesinglesingle--layer network by having one or morelayer network by having one or more hidden layershidden layers. In. Inthis multilayer structure, the input nodes pass the information tothis multilayer structure, the input nodes pass the information tothe units in the first hidden layer, then the outputs from the firstthe units in the first hidden layer, then the outputs from the first

hidden layer are passed to the next layer, and so on.hidden layer are passed to the next layer, and so on.

Multilayer network can be also viewed as cascading of groupsMultilayer network can be also viewed as cascading of groupsof singleof single--layer networks.layer networks.

The level of complexityThe level of complexity

in computing can bein computing can be

Seen by the fact thatSeen by the fact thatmany singlemany single--layerlayer

networks are combinednetworks are combined

into this multilayer network.into this multilayer network.

The designer of an artificialThe designer of an artificial

neural network should consider how many hidden layers areneural network should consider how many hidden layers are


27/34

Recurrent NetworksRecurrent Networks

In this type of networks there exists at least 1 feedback loop.In this type of networks there exists at least 1 feedback loop.Output layer is again feed the outputs to the input layer(Output layer is again feed the outputs to the input layer(arbitrary neurons ) as inputs of the same network.arbitrary neurons ) as inputs of the same network.

There could be neurons with self feedback links.There could be neurons with self feedback links.


28/34

Learning Algorithm

There are a lot of learning algorithm classified as

supervised learning and unsupervised Learning.

Supervised Learning uses a set of inputs for which

the appropriate (desired) output areknown.Computed output and correct output are

compared to determine error.

Unsupervised Learning only input stimuli are shown

to the network. The network is self-Organizing. Thesystem learns of its own by discovering and adapting

to structural features in input pattern.


29/34

2 Main Types of ANN

UnsupervisedSupervised

e.g:

Adaline

Perceptron

MLP

RBF

Fuzzy ARTMAP

etc.

e.g:

Competitive learning networks

- SOM- ART families

- neocognition

- etc.


30/34

Supervised Network

Teacher

ANN

+

-

error


31/34

Unsupervised ANN

Teacher

ANN


32/34

How does an ANN learn

I

N

P

U

T

S

I

G

N

A

L

S

O

U

T

P

U

T

S

I

G

N

A

L

SInput

layer

Middle

layer

Output

Layer

weights

neurons

Connected by links-each link

has a numerical weight

Weight

basic means of long-term

memory in ANNs

Express the strength

Learns through repeated

adjustments of theseweights


33/34

Learning Process of ANN

Compute

output

Is

DesiredOutput

achieved

Stop

Adjust

Weight

yes

No

Learn from experience

Learning algorithms

Recognize pattern of

activities

Involves 3 tasks

Compute outputs

Compare outputs with

desired targets

Adjust the weights and

repeat the process


34/34

NN Learning Algorithm

Unsupervised Learning Reinforced Learning( Outpt Based )

Supervised Learning( ErrorBased )

ErrorCorrectionGradient descent

HebbianStochastic

Competitive

LeastMean Square

Backpropagation

Learning Algorithm

Classification

neural network 1

Documents