Data Mining and Data Mining and Neural NetworksNeural Networks

Danny LeungDanny LeungCS157B, Spring 2006CS157B, Spring 2006Professor Sin-Min LeeProfessor Sin-Min Lee

22

Artificial Intelligence Artificial Intelligence for Data Miningfor Data Mining Neural networks are useful for data mining and Neural networks are useful for data mining and

decision-support applications.decision-support applications.

People are good at generalizing from People are good at generalizing from experience.experience.

Computers excel at following explicit instructions Computers excel at following explicit instructions over and over. over and over.

Neural networks bridge this gap by modeling, on Neural networks bridge this gap by modeling, on a computer, the neural behavior of human a computer, the neural behavior of human brains.brains.

33

Neural Network Neural Network CharacteristicsCharacteristics

Neural networks are useful for pattern Neural networks are useful for pattern recognition or data classification, recognition or data classification, through a learning process.through a learning process.

Neural networks simulate biological Neural networks simulate biological systems, where learning involves systems, where learning involves adjustments to the synaptic adjustments to the synaptic connections between neuronsconnections between neurons

44

Anatomy of a Neural Anatomy of a Neural NetworkNetwork

Neural Networks map a Neural Networks map a set of input-nodes to a set of input-nodes to a set of output-nodesset of output-nodes

Number of inputs/outputs Number of inputs/outputs is variableis variable

The Network itself is The Network itself is composed of an arbitrary composed of an arbitrary number of nodes with an number of nodes with an arbitrary topologyarbitrary topology

Neural Network

Input 0 Input 1 Input n...

Output 0 Output 1 Output m...

55

Biological BackgroundBiological Background

A neuron: many-inputs / one-output unitA neuron: many-inputs / one-output unit

Output can be Output can be excitedexcited or or not not excitedexcited

Incoming signals from other neurons Incoming signals from other neurons determine if the neuron shall determine if the neuron shall exciteexcite ("fire")("fire")

Output subject to attenuation in the Output subject to attenuation in the synapses, synapses, which are junction parts of the which are junction parts of the neuronneuron

66

Basics of a NodeBasics of a Node

A node is an A node is an element which element which performs a performs a functionfunction

y = fy = fHH(∑(w(∑(wiixxii) + ) + WWbb))

fH(x)

Input 0 Input 1 Input n...

W0 W1 Wn

+

Output

+

...

Wb

ConnectionConnection

NodeNode

77

A Simple PreceptronA Simple Preceptron

Binary logic Binary logic applicationapplication

ffHH(x) [linear (x) [linear threshold]threshold]

Wi = random(-1,1)Wi = random(-1,1)

Y = u(W0X0 + Y = u(W0X0 + W1X1 + Wb)W1X1 + Wb)

fH(x)

Input 0 Input 1

W0 W1

+

Output

Wb

88

Preceptron TrainingPreceptron Training

It’s a single-unit networkIt’s a single-unit network

Adjust weights based on a how well the current Adjust weights based on a how well the current weights match an objectiveweights match an objective

Perceptron Learning RulePerceptron Learning Rule

Δ WΔ Wii = η * (D-Y).I = η * (D-Y).Iii

– η = Learning Rateη = Learning Rate– D = Desired OutputD = Desired Output

99

Neural Network Neural Network LearningLearning From experience: examples / training From experience: examples / training

datadata

Strength of connection between the Strength of connection between the neurons is stored as a weight-value for neurons is stored as a weight-value for the specific connectionthe specific connection

Learning the solution to a problem = Learning the solution to a problem = changing the connection weightschanging the connection weights

1010

Neural Network Neural Network LearningLearning Continuous Learning ProcessContinuous Learning Process

Evaluate output Evaluate output

Adapt weightsAdapt weights

Take new inputsTake new inputs

Learning causes stable state of the Learning causes stable state of the weightsweights

1111

Learning PerformanceLearning Performance

SupervisedSupervised– Need to be trained ahead of time with lots of dataNeed to be trained ahead of time with lots of data

Unsupervised networks adapt to the inputUnsupervised networks adapt to the input– Applications in Clustering and reducing Applications in Clustering and reducing

dimensionalitydimensionality– Learning may be very slowLearning may be very slow– No help from the outsideNo help from the outside– No training data, no information available on the No training data, no information available on the

desired outputdesired output– Learning by doingLearning by doing– Used to pick out structure in the input:Used to pick out structure in the input:

– ClusteringClustering– CompressionCompression

1212

Topologies – Back-Topologies – Back-Propogated NetworksPropogated Networks Inputs are put Inputs are put

through a through a ‘Hidden Layer’ ‘Hidden Layer’ before the output before the output layerlayer

All nodes All nodes connected connected between layersbetween layers

Input 0 Input 1 Input n...

Output 0 Output 1 Output o...

O0 O1 Oo

H0 H1 Hm...

...

Hidden Layer

1313

BP Network – BP Network – Supervised TrainingSupervised Training

Desired output of the training examplesDesired output of the training examples

Error = difference between actual & desired outputError = difference between actual & desired output

Change weight relative to error sizeChange weight relative to error size

Calculate output layer error , then propagate back to Calculate output layer error , then propagate back to previous layerprevious layer

Hidden weights updatedHidden weights updated

Improved performanceImproved performance

1414

Neural Network Neural Network Topology Topology CharacteristicsCharacteristics

Set of inputsSet of inputs

Set of hidden nodesSet of hidden nodes

Set of outputsSet of outputs

Increasing nodes makes network more Increasing nodes makes network more difficult to traindifficult to train

1515

Applications of Neural Applications of Neural NetworksNetworks Prediction – weather, stocks, diseasePrediction – weather, stocks, disease

Classification – financial risk assessment, Classification – financial risk assessment, image processingimage processing

Data association – Text Recognition (OCR)Data association – Text Recognition (OCR)

Data conceptualization – Customer Data conceptualization – Customer purchasing habitspurchasing habits

Filtering – Normalizing telephone signals Filtering – Normalizing telephone signals (static)(static)

1616

OverviewOverview

AdvantagesAdvantages– Adapt to unknown situationsAdapt to unknown situations– Robustness: fault tolerance due to network Robustness: fault tolerance due to network

redundancyredundancy– Autonomous learning and generalizationAutonomous learning and generalization

DisadvantagesDisadvantages– Not exactNot exact– Large complexity of the network structureLarge complexity of the network structure

1717

Referenced WorkReferenced Work

Intro to Neural Networks - Computer Vision Applications and Training Intro to Neural Networks - Computer Vision Applications and Training Techniques. Doug Gray. www.soe.ucsc.edu/~taoswap/ Techniques. Doug Gray. www.soe.ucsc.edu/~taoswap/ GroupMeeting/NN_Doug_2004_12_1.pptGroupMeeting/NN_Doug_2004_12_1.ppt

Introduction to Artificial Neural Networks. Nicolas Galoppo von Introduction to Artificial Neural Networks. Nicolas Galoppo von Borries. www.cs.unc.edu/~nico/courses/ comp290-58/nn-Borries. www.cs.unc.edu/~nico/courses/ comp290-58/nn-presentation/ann-intro.ppt presentation/ann-intro.ppt