www.spatialanalysisonline.com
Chapter 8
Geocomputation Part B:
Artificial Neural Networks (ANNs) & Genetic Algorithms (GAs)
3rd edition www.spatialanalysisonline.com 2
Geocomputation: ANNs
In this presentation on geocomputation:ANNs discussed include
Multi-level perceptrons (MLPs) Radial basis function neural networks (RBFNNs) Self organising feature maps (SOFMs)
ANNs are particularly concerned with Function approximation and interpolation Image analysis and classification Spatial interaction modelling
3rd edition www.spatialanalysisonline.com 3
Geocomputation: Evolutionary computing
In this presentation on geocomputation:
EC elements discussed include Genetic algorithms (GAs) Genetic programming (GP)
EC is particularly concerned with Complex problem solving using GAs Model design using GP methods
3rd edition www.spatialanalysisonline.com 4
Geocomputation
Artificial Neural Networks (ANNs) A computational model based on emulating
biological neural networks A form of non-linear modelling tool Often a 3-layer network structure is used:
input, hidden, output The output layer of such structures are typically
modified weighted sums of intermediate layers, which are modified weighted sums of the input layer
3rd edition www.spatialanalysisonline.com 5
Artificial Neural Networks
Hence at each output node (hidden or final) a two-step process takes place:
g
3rd edition www.spatialanalysisonline.com 6
Artificial Neural Networks Simple 3-layer feedforward ANN Fully inter-connected; each connection
is given a weight, w Known as a Multi-level perceptron
(MLP) In this case: 3 input nodes, 5 hidden
nodes, 2 output nodes and 2 bias nodes (bias, B, is similar to the constant term in regression models)
At hidden node 1 we have:
where the wij are weights to be determined, b1=1, and the xi are the observed input values
2
1
3
1
2
3
4
5
1
2
INPUT HIDDEN OUT PUT
B1 B2
n=3 m=5 p=2
1
*01 1 11 1 21 2 31 3h w b w x w x w x
3rd edition www.spatialanalysisonline.com 7
Artificial Neural Networks
is simply a linear weighted sum of the inputs. To generate a non-linear output it must be modified by some (well behaved) non-linear function, g(), e.g. the logistic function:
i.e.
1 01 1 11 1 21 2 31 3h g w b w x w x w x
1
*01 1 11 1 21 2 31 3h w b w x w x w x
1( )
1 hg h
e
0
0.5
1
-5 -4 -3 -2 -1 0 1 2 3 4 5
Logistic (1+Tanh)/2 Step Linear
Sample activation functions
3rd edition www.spatialanalysisonline.com 8
Artificial Neural Networks
We can now compute the output layer values as the weighted sum
Suppose we have known input values x1=1, x2=-3, x3=5, and known outputs of 0 and 1. Can we select the weights to ensure the inputs generate the known outputs?
Suggestion: <build your own worked example & program here>
*1 01 2 11 1 21 2 31 3 41 4 51 5
*1 1
, and thusy z b z h z h z h z h z h
y g y
3rd edition www.spatialanalysisonline.com 9
Artificial Neural Networks
Learning Supervised learning
Split training/test data sets (control data) Known inputs and output (target) values for training data (Network output-Target output) = Error signal, e Systematically adjust weights to minimise sum of e2
Adjustment typically based on backpropagation and gradient descent
Used in many classification/pattern recognition applications and in function approximation
Unsupervised learning No training data Must create clusters by analysing dataset for patterns/clusters
3rd edition www.spatialanalysisonline.com 10
Artificial Neural Networks
Some basic issues: local vs global minimisation Initialisation and selection Data normalisation and coding Momentum Model design and over-fitting Overtraining Interpolation vs Extrapolation/Forecasting
3rd edition www.spatialanalysisonline.com 11
Artificial Neural Networks
MLP: Example 1 function approximation
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0 100 200 300 400 500 600 700 800 900 10000
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Epochs
RM
SE
(R
oot
mea
n sq
uare
d er
ror)
source data fitted solution curve RMSE vs epochs
3rd edition www.spatialanalysisonline.com 12
Artificial Neural Networks
MLP Example 2: LCM
3rd edition www.spatialanalysisonline.com 13
Artificial Neural Networks
MLP Example 2: LCM
3rd edition www.spatialanalysisonline.com 14
Artificial Neural Networks
MLP Example 2: LCM
3rd edition www.spatialanalysisonline.com 15
Artificial Neural Networks
MLP Example 2: LCM
weights matrix
3rd edition www.spatialanalysisonline.com 16
Artificial Neural Networks
MLP Example 3: Spatial interaction model Generalised model: Tij=f(Oi,Dj,dij) Sample data format
(log transformed):
3rd edition www.spatialanalysisonline.com 17
Artificial Neural Networks
MLP Example 3: Spatial interaction model
3rd edition www.spatialanalysisonline.com 18
Artificial Neural Networks
Radial Basis Function NetworksBasic functional form:
Gaussian RBF:
1
( ),
where denotes the Euclidean norm.
n
k ij i ji
w
y x c
2 2/ 2
2
1( )
2
xx e
3rd edition www.spatialanalysisonline.com 19
Artificial Neural Networks
Self organising function maps SOM as an output space Neighbourhood relations Grid size, form and topology
3rd edition www.spatialanalysisonline.com 20
Artificial Neural Networks
Self organising function maps Dimensional reductions Mapped output – similar vectors (units) are
close to each other Typically an unsupervised procedure Spatial mapping of SOM can follow using
simple assignment to best matching unit (BMU)
3rd edition www.spatialanalysisonline.com 21
Artificial Neural Networks
Self organising function maps Choose a grid size, form and topology Train the network
Identify the best matching units Modify the BMU and its neighbours (spatially biased
learning rule) Map the trained network
3rd edition www.spatialanalysisonline.com 22
Artificial Neural Networks
Self organising function maps – some issues Initialisation Pre-processing Normalisation Missing data Masking and weighting Learning and tuning Distance metrics Neighbourhood functions (kernels) Learning rate functions
3rd edition www.spatialanalysisonline.com 23
Artificial Neural Networks
Self organising function maps – Idrisi
3rd edition www.spatialanalysisonline.com 24
Artificial Neural Networks
Self organising function maps – Idrisi
3rd edition www.spatialanalysisonline.com 25
Genetic Algorithms
Solutions are represented as individuals Individuals are modelled as chromosomes Chromosomes are comprised of genes Genes have values known as alleles Chromosomes have a measurable fitness New chromosomes (children) are created by reproduction and mutation processes
The fittest individuals survive The creation process is then iterated
3rd edition www.spatialanalysisonline.com 26
Genetic Algorithms
GAs: Example 1 - TSP
chromosome
genes
allele=12 (ID of town in TSP problem set)
Each chromosome contains complete list of towns• create a set of m randomly permuted strings and compute lengths, d• evaluate the fitness of each string (e.g. 1/d)• select random pairs of tours (biased by fitness)• combine pairs by crossover operation• evaluate fitness of offspring• apply replacement rule (fittest retained) and iterate till stable
3rd edition www.spatialanalysisonline.com 27
Genetic Algorithms
GA components Encoding or representation – binary, list, tree etc Fitness function selection – use of rank transforms Population initialisation Selection: roulette, tournament, uniform random Reproduction Crossover e.g. A = [a b c d e f g h] B = [1 2 3 4 5 6 7 8]
and the crossover point is 3, the following children are generated:
child 1 = [a b c 4 5 6 7 8] child 2= [1 2 3 d e f g h] Mutation Local search Termination
3rd edition www.spatialanalysisonline.com 28
Genetic Algorithms
GAs: application areas TSP (as above) Clustering Map labelling Optimum location with capacity constraints Concept can be extended to alleles that are
expressions or program elements rather than numerical values Genetic programming