1 neural networks - basics artificial neural networks - basics uwe lämmel business school institute...

1 Neural Networks - Basics

ArtificialNeural Networks - Basics

Uwe Lämmel

Business SchoolInstitute of

Business Informatics

www.wi.hs-wismar.de/~laemmel

[email protected]


Neural Networks

• Idea

• Artificial Neuron & Network

• Supervised Learning

• Unsupervised Learning

• Data Mining – other Techniques


Artificial Neuron & Network

• Idea• An artificial Neuron• Neural Network• Example• Learning• Application


Idea

• A human being learns by example “learning by doing”

– seeing(Perception), Walking, Speaking,…

• Can a machine do the same?

• A human being uses his brain. A brain consists of millions of single cells.A cell is connected with ten thousands of other cell.

• Is it possible to simulate a similar structure on a computer?


Idea

Artificial Neural Network– Information processing similar to

processes in a mammal brain– heavy parallel systems, – able to learn– great number of simple cells

? Is it useful to copy nature ? – wheel, aeroplane, ...


Idea

• we need:– software neurons– software connections between

neurons– software learning algorithms

An artificial neural network functions in a similar way a natural neural network does.


A biological Neuron

• Dendrits: (Input) Getting other

activations• Axon: (Output ) forward the

activation (from 1mm up to 1m long)

• Synapse: transfer of activation:– to other cells, e.g. Dendrits of

other neurons– a cell has about 1.000 to 10.000

connections to other cells• Cell Nucleus: (processing)

evaluation of activation

cell and nucleus

Axon(Neurit)

Dendrits

Synapsis


Natural vs. Artificial Neuron

cell and nucleus

Axon(Neurit)

Dendrits

Synapsis

j

jjii ownet

),( iiacti netfact

)( iouti actfo

w1i w2i wji...

oi


Abstraction

• Dendrits: weighted (real number) connections• Axon: output: real number• Synapse: ---

(identity: output is directly forwarded)• Cell nucleus:

unit contains simple functionsinput = (many) real numbersprocessing = activation functionoutput = real number (~activation)


An artificial Neuron

net : input from the networkw : weight of a connectionact : activationfact : activation function

: bias/thresholdfout : output function (mostly

ID)o : output

j

jjii ownet

),( iiacti netfact

)( iouti actfo

w1i w2i wji...

oi


Exercise: AND/OR -LTU

• Built a „network“ that works like an AND-function

• Built an OR-network• Try to built an XOR-network

LTU – Linear Threshold Unit


A simple switch

• Neuron = AND-function• Find parameters:

– Input neurons 1,2 : a1,a2 input pattern,

– weights of edges: w1, w2

– bias • Now evaluate output o !

a1=__ a2=__

o=__

net = o1w1+o2

w2a = 1, if net >= 0, otherwise

o = a

w1=__ w2=__


Mathematics in a Cell

• Propagation function (neuron input) neti(t) = ojwj = w1i o1 + w2i o2 + ...

• Activationai(t) – Activation at time t

• Activation function fact :ai(t+1) = fact(ai(t), neti(t), i)

i – bias

• Output function fout :

oi = fout(ai)


Activation

rea l

u n re str icted

[0 ,1 ][-1 ,+ 1 ]

In te rva l

co n tin uo us

0 ,1-1 ,+ 1

- ,+

b in ary

{ -1 ,0 ,+ 1}{ -10 0 ,... ,+ 10 0}

m u lti v a lue

d iscre te

A c tiv a tion


Bias function

-1,0

-0,5

0,0

0,5

1,0

-4,0 -2,0 0,0 2,0 4,0

Identity

-4,0

-2,0

0,0

2,0

4,0

-4,0 -2,5 -1,0 0,5 2,0 3,5

Activation Functions activation functions

aresigmoid functions


y = tanh(c·x)

-1,0

-0,5

0,5

1,0

-0,6 0,6 1,0

c=1c=2

c=3

-1,0

Activation Functions

y = 1/(1+exp(-c·x))

0,5

1,0

-1,0 0,0 1,0

c=1c=3

c=10

Logistic function:

activation functions

aresigmoid functions


Structure of a network

• layers– input layer – input neurons– output layer – output neurons– hidden layer – hidden neurons

• An n-layer network has:– n layer of connections which can be

trained


Definition: A Neural Network …

• … is characterized by – connections of many (a lot of) simple units

(neurons) and– units exchanging signals via these

connections

• … is a – coherent, directed graph which has– weighted edges and– each node (neuron, unit )

contains a value (activation).


XOR-network

• weights are set by hand

• fact = 1, net > = 0, otherwise

= 1.0

input

output

weight matrix wij: i\j 1 2 3 4 1 0 0 1 1.2 2 0 0 1 1.2 3 0 0 0 -2 4 0 0 0 0


XOR-Example

• standard propagation function: neti(t) = oj(t)wji

• activation function = bias function– ai = 1, if neti(t)> i e.g. =0.5

0, otherwise– output function = Identity: oj = aj

• use EXCEL and built the XOR-example network!


Learning

Network

changing network parameters

evaluation network errorlearning

examples


Learning - can be done by:

1. Modification of the weight of a connection – most frequently used

2. Deleting connections– can be done by (1): w=0 w0

3. Modification of the bias of a neuron– Can be done by (1) using an extra neuron

4. changing functions (activation, propagation, output function)

5. Building new cells (GNG)6. Building new connections7. Deleting cells


Learning

• supervised learning– We know the results for certain input pattern:

teaching input teaching output

– Network error is used to adapt weights– Fast, but not natural

• reinforced learning– We know whether output is right or wrong– Information is used to adapt weights – Slower than supervised; natural

• unsupervised learning– Network has to learn by itself;– Slow , natural


Applications

Pattern recognition (text, numbers, faces)

Checking the quality of a surface Control of autonomous vehicles Monitoring of credit card accounts Data Mining


Applications

Speech recognition Control of artificial limbs classification of galaxies Product orders (Supermarket) Forecast of energy

consumption Stock value forecast


Application - Summary

Classification Clustering Forecast Pattern recognition

Learning by examples, generalization Recognition of not known structures

1 neural networks - basics artificial neural networks - basics uwe lämmel business school institute...

Documents