lect 1(intro)

8/7/2019 Lect 1(Intro)

1/19

Artificial Neural Networks

- Introduction -


2/19

Reference Books and Journals

Neural Networks: A Comprehensive

Foundation by Simon Haykin

Neural Networks for Pattern Recognition byChristopher M. Bishop

Some papers


3/19

Overview

Neural Network (NN) or Artificial Neural Networks

(ANN) is a computing paradigm

The key element of this paradigm is

the novel structure of the information processing system

consisting of a large number of highly interconnected

processing elements (neurons) working in unison to solve

specific problems

Development of NNs date back to the early 1940sMinsky and Papert, published a book (in 1969)

summed up a general feeling of frustration (against neural

networks) among researchers


4/19

Overview (Contd.)

Experienced an upsurge in popularity in the

late 1980s

Result of the discovery of new techniques anddevelopments and general advances in computer

hardware technology

Some NNs are models of biological neural

networks and some are not


5/19

Overview (Contd.)

Historically, much of the inspiration for the

field of NNs came from the desire to produce

artificial systems capable ofsophisticated, perhaps intelligent, computations

similar to those that the human brain routinely

performs, and thereby possibly to enhance our

understanding of the human brain.


6/19

Overview (Contd.)

Most NNs have some sort oftraining rule. In other words,

NNs learn from examples

as children learn to recognize dogs from examples of dogs) and

exhibit some capability forgeneralization beyond the training data

Neural computing must not be considered as a competitor to

conventional computing.

Rather should be seen as complementary

Most successful neural solutions have been those which operate in

conjunction with existing, traditional techniques.


7/19

Overview (Contd.)

Digital Computers Deductive Reasoning. Weapply known rules to input data

to produce output

Computation is centralized,

synchronous, and serial. Memory is packetted, literally

stored, location addressable

Not fault tolerant. One transis-

tor goes and it no longer works.

Exact. Static connectivity.

Applicable if well defined rules

with precise input data.

Neural Networks Inductive Reasoning. Given input

and output data (training

examples), we construct rules

Computation is collective,

asynchronous, and parallel.

Memory is distributed,

internalized, short term and

content addressable.

Fault tolerant, redundancy, and

sharing of responsibilities.

Inexact.

Dynamic connectivity.

Applicable if rules are unknown

or complicated, or if data are

noisy or partial.


8/19

Why Neural Networks

Adaptive learning

An ability to learn how to do tasks based on the data given for training or initial

experience.

Self-Organization

An ANN can create its own organization or representation of the information itreceives during learning time.

Real Time Operation

An ANN computations may be carried out in parallel, and special hardware

devices are being designed and manufactured which take advantage of this

capability.Fault Tolerance via Redundant Information Coding:

Partial destruction of a network leads to the corresponding degradation of

performance. However, some network capabilities may be retained even with

major network damage.


9/19

What can you do with an NN

and what not?

In principle, NNs can compute any computable function,

i.e., they can do everything a normal digital computer can

do.

In practice, NNs are especially useful forclassification andfunction approximationproblems.

NNs are, at least today, difficult to apply successfully to

problems that concern manipulation of symbols and

memory.There are no methods for training NNs that can magically

create information that is not contained in the training

data.


10/19

Who is concerned with NNs?

Computer scientists want to find out about the propertiesof non-symbolic information processing with neural netsand about learning systems in general.

Statisticians use neural nets as flexible, nonlinearregression and classification models.

Engineers of many kinds exploit the capabilities of neuralnetworks in many areas, such as signal processing andautomatic control.

Cognitive scientists view neural networks as a possibleapparatus to describe models of thinking andconsciousness (High-level brain function).

Neuro-physiologists use neural networks to describe andexplore medium-level brain function (e.g. memory,sensory system, motorics).


11/19

Who is concerned with NNs?

Physicists use neural networks to model phenomena instatistical mechanics and for a lot of other tasks.

Biologists use Neural Networks to interpret nucleotidesequences.

Philosophers and some other people may also be interestedin Neural Networks for various reasons


12/19

Biological inspiration

Animals are able to react adaptively to changes in their

external and internal environment, and they use their

nervous system to perform these behaviours.

An appropriate model/simulation of the nervous systemshould be able to produce similar responses and

behaviours in artificial systems.

The nervous system is build by relatively simple units, the

neurons, so copying their behavior and functionality

should be the solution.


13/19

Biological inspiration (Contd.)


14/19

Biological inspiration (Contd.)

The brain is a collection of about 10 billion interconnected

neurons

Each neuron is a cell that usesbiochemical reactions to receive,

process and transmit information.

Each terminal button is connected to other neurons across

a small gap called a synapse

A neuron's dendritic tree is connected to a thousand

neighbouring neurons

When one of those neurons fire

a positive or negative charge is received by one of the dendrites.

The strengths of all the received charges are added together

through the processes of spatial and temporal summation.


15/19

Artificial neurons

Neurons work by processing information. They receive and

provide information in form of spikes.

The McCullogh-Pitts model

Inputs

Outputw2

w1

w3

wn

wn-1

.. .

x1

x2

x3

xn-1

xn

y)(;

1

zHyxwzn

i

ii ===


16/19

Artificial neurons

Nonlinear generalization of the McCullogh-Pitts

neuron:

),( wxfy=

y is the neurons output, x is the vector of inputs, and w

is the vector of synaptic weights.

Examples:

2

2

2

||||

11

a

wx

axw

ey

ey

T

=

+

= sigmoidal neuron

Gaussian neuron


17/19

Activation Functions

The activation function is generally non-linear

Linear functions are limited because the output is simply

proportional to the input


18/19

Activation Functions (Contd.)


19/19

Artificial neurons

Nonlinear generalization of the McCullogh-Pitts

neuron:

),( wxfy=

y is the neurons output, x is the vector of inputs, and w

is the vector of synaptic weights.

Examples:

2

2

2

||||

11

a

wx

axw

ey

ey T

=

+

= sigmoidal neuron

Gaussian neuron

lect 1(intro)

Documents