neural networ;

8/3/2019 Neural Networ;

1/38

For other uses, seeNeural network (disambiguation).

Simplified view of a feedforward artificial neural network

The term neural network was traditionally used to refer to a network or circuit ofbiological

neurons.[1]

The modern usage of the term often refers toartificial neural networks, which arecomposed ofartificial neuronsor nodes. Thus the term has two distinct usages:

1. Biological neural networksare made up of real biological neurons that are connected orfunctionally related in theperipheral nervous systemor thecentral nervous system. In thefield ofneuroscience, they are often identified as groups of neurons that perform a

specific physiological function in laboratory analysis.2. Artificial neural networksare composed of interconnecting artificial neurons

(programming constructs that mimic the properties of biological neurons). Artificial

neural networks may either be used to gain an understanding of biological neural

networks, or for solving artificial intelligence problems without necessarily creating amodel of a real biological system. The real, biological nervous system is highly complex:

artificial neural network algorithms attempt to abstract this complexity and focus on what

may hypothetically matter most from an information processing point of view. Goodperformance (e.g. as measured by good predictive ability, low generalization error), or

performance mimicking animal or human error patterns, can then be used as one source

of evidence towards supporting the hypothesis that the abstraction really captured

something important from the point of view of information processing in the brain.Another incentive for these abstractions is to reduce the amount of computation required

to simulate artificial neural networks, so as to allow one to experiment with larger

networks and train them on larger data sets.

This article focuses on the relationship between the two concepts; for detailed coverage of the

two different concepts refer to the separate articles:biological neural networkandartificialneural
http://en.wikipedia.org/wiki/Neural_network_(disambiguation)http://en.wikipedia.org/wiki/Neural_network_(disambiguation)http://en.wikipedia.org/wiki/Neural_network_(disambiguation)http://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neural_network#cite_note-0http://en.wikipedia.org/wiki/Neural_network#cite_note-0http://en.wikipedia.org/wiki/Neural_network#cite_note-0http://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Peripheral_nervous_systemhttp://en.wikipedia.org/wiki/Peripheral_nervous_systemhttp://en.wikipedia.org/wiki/Peripheral_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/File:Neural_network_example.svghttp://en.wikipedia.org/wiki/File:Neural_network_example.svghttp://en.wikipedia.org/wiki/File:Neural_network_example.svghttp://en.wikipedia.org/wiki/File:Neural_network_example.svghttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Peripheral_nervous_systemhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Neural_network#cite_note-0http://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neural_network_(disambiguation)


2/38

This article is about cells in the nervous system. For other uses, seeNeuron (disambiguation).

"Brain cell" redirects here. For other uses, seeGlial cell.

Neuron: Nerve Cell

Drawing bySantiago Ramn y Cajalof neurons in the

pigeon cerebellum. (A) DenotesPurkinje cells, an example

of a multipolar neuron. (B) Denotesgranule cells, which

are also multipolar.

NeuroLexID sao1417703748

vd e

A neuron ( /njrn/NEWR-on; also known as a neurone or nerve cell) is anelectricallyexcitablecellthat processes and transmits information by electrical and chemicalsignaling.

Chemical signaling occurs viasynapses, specialized connections with other cells. Neurons

connect to each other to formnetworks. Neurons are the core components of thenervous system,which includes thebrain,spinal cord, and peripheralganglia. A number of specialized types ofneurons exist:sensory neuronsrespond to touch, sound, light and numerous other stimuli

affecting cells of thesensory organsthat then send signals to the spinal cord and brain.Motor

neuronsreceive signals from the brain and spinal cord, causemuscle contractions, and affect

glands. Interneurons connect neurons to other neurons within the same region of the brain orspinal cord.
http://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Neuron_(disambiguation)http://en.wikipedia.org/wiki/Neuron_(disambiguation)http://en.wikipedia.org/wiki/Neuron_(disambiguation)http://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Santiago_Ram%C3%B3n_y_Cajalhttp://en.wikipedia.org/wiki/Santiago_Ram%C3%B3n_y_Cajalhttp://en.wikipedia.org/wiki/Santiago_Ram%C3%B3n_y_Cajalhttp://en.wikipedia.org/wiki/Purkinje_cellhttp://en.wikipedia.org/wiki/Purkinje_cellhttp://en.wikipedia.org/wiki/Purkinje_cellhttp://en.wikipedia.org/wiki/Granule_cellshttp://en.wikipedia.org/wiki/Granule_cellshttp://en.wikipedia.org/wiki/Granule_cellshttp://en.wikipedia.org/wiki/NeuroLexhttp://en.wikipedia.org/wiki/NeuroLexhttp://www.neurolex.org/wiki/sao1417703748http://en.wikipedia.org/wiki/Template:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template_talk:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template_talk:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template_talk:Infobox_neuron/sandboxhttp://en.wikipedia.org/w/index.php?title=Template:Infobox_neuron/sandbox&action=edithttp://en.wikipedia.org/w/index.php?title=Template:Infobox_neuron/sandbox&action=edithttp://en.wikipedia.org/w/index.php?title=Template:Infobox_neuron/sandbox&action=edithttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Electricityhttp://en.wikipedia.org/wiki/Electricityhttp://en.wikipedia.org/wiki/Electricityhttp://en.wikipedia.org/wiki/Cell_(biology)http://en.wikipedia.org/wiki/Cell_(biology)http://en.wikipedia.org/wiki/Cell_(biology)http://en.wikipedia.org/wiki/Signal_(biology)http://en.wikipedia.org/wiki/Signal_(biology)http://en.wikipedia.org/wiki/Signal_(biology)http://en.wikipedia.org/wiki/Synapsehttp://en.wikipedia.org/wiki/Synapsehttp://en.wikipedia.org/wiki/Synapsehttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Nervous_systemhttp://en.wikipedia.org/wiki/Nervous_systemhttp://en.wikipedia.org/wiki/Nervous_systemhttp://en.wikipedia.org/wiki/Brainhttp://en.wikipedia.org/wiki/Brainhttp://en.wikipedia.org/wiki/Brainhttp://en.wikipedia.org/wiki/Spinal_cordhttp://en.wikipedia.org/wiki/Spinal_cordhttp://en.wikipedia.org/wiki/Spinal_cordhttp://en.wikipedia.org/wiki/Gangliahttp://en.wikipedia.org/wiki/Gangliahttp://en.wikipedia.org/wiki/Gangliahttp://en.wikipedia.org/wiki/Sensory_neuronhttp://en.wikipedia.org/wiki/Sensory_neuronhttp://en.wikipedia.org/wiki/Sensory_neuronhttp://en.wikipedia.org/wiki/Sensehttp://en.wikipedia.org/wiki/Sensehttp://en.wikipedia.org/wiki/Sensehttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Muscle_contractionhttp://en.wikipedia.org/wiki/Muscle_contractionhttp://en.wikipedia.org/wiki/Muscle_contractionhttp://en.wikipedia.org/wiki/Glandhttp://en.wikipedia.org/wiki/Glandhttp://en.wikipedia.org/wiki/File:PurkinjeCell.jpghttp://en.wikipedia.org/wiki/File:PurkinjeCell.jpghttp://en.wikipedia.org/wiki/Glandhttp://en.wikipedia.org/wiki/Muscle_contractionhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Sensehttp://en.wikipedia.org/wiki/Sensory_neuronhttp://en.wikipedia.org/wiki/Gangliahttp://en.wikipedia.org/wiki/Spinal_cordhttp://en.wikipedia.org/wiki/Brainhttp://en.wikipedia.org/wiki/Nervous_systemhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Synapsehttp://en.wikipedia.org/wiki/Signal_(biology)http://en.wikipedia.org/wiki/Cell_(biology)http://en.wikipedia.org/wiki/Electricityhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/w/index.php?title=Template:Infobox_neuron/sandbox&action=edithttp://en.wikipedia.org/wiki/Template_talk:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template:Infobox_neuron/sandboxhttp://www.neurolex.org/wiki/sao1417703748http://en.wikipedia.org/wiki/NeuroLexhttp://en.wikipedia.org/wiki/Granule_cellshttp://en.wikipedia.org/wiki/Purkinje_cellhttp://en.wikipedia.org/wiki/Santiago_Ram%C3%B3n_y_Cajalhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Neuron_(disambiguation)


3/38

A typical neuron possesses a cell body (often called thesoma),dendrites, and anaxon. Dendrites

are filaments that arise from the cell body, often extending for hundreds of micrometres andbranching multiple times, giving rise to a complex "dendritic tree". An axon is a special cellular

filament that arises from the cell body at a site called theaxon hillockand travels for a distance,

as far as 1 m in humans or even more in other species. The cell body of a neuron frequently gives

rise to multiple dendrites, but never to more than one axon, although the axon may branchhundreds of times before it terminates. At the majority of synapses, signals are sent from the

axon of one neuron to a dendrite of another. There are, however, many exceptions to these rules:

neurons that lack dendrites, neurons that have no axon, synapses that connect an axon to anotheraxon or a dendrite to another dendrite, etc.

All neurons are electrically excitable, maintainingvoltagegradients across theirmembranesby

means of metabolically drivenion pumps, which combine withion channelsembedded in the

membrane to generate intracellular-versus-extracellular concentration differences ofionssuch as

sodium,potassium,chloride, andcalcium. Changes in the cross-membrane voltage can alter thefunction ofvoltage-dependent ion channels. If the voltage changes by a large enough amount, an

all-or-none electrochemical pulse called anaction potentialis generated, which travels rapidlyalong the cell's axon, and activates synaptic connections with other cells when it arrives.

Neurons of the adult brain do not generally undergocell division, and usually cannot be replaced

after being lost, although there are a fewknown exceptions. In most cases they are generated byspecial types ofstem cells, althoughastrocytes(a type ofglial cell) have been observed to turn

into neurons as they are sometimespluripotent

Artificial neural network

From Wikipedia, the free encyclopedia

Jump to:navigation,search

This article needs additionalcitationsforverification.Please helpimprove this articleby addingreliable references. Unsourced material may bechallenged

andremoved.(March 2009)

An artificial neural network (ANN), usually calledneural network(NN), is amathematical

modelorcomputational modelthat is inspired by the structure and/or functional aspects of

biological neural networks. A neural network consists of an interconnected group ofartificialneurons, and it processes information using aconnectionistapproach tocomputation. In most

cases an ANN is anadaptive systemthat changes its structure based on external or internal

information that flows through the network during the learning phase. Modern neural networksarenon-linearstatisticaldata modelingtools. They are usually used to model complexrelationships between inputs and outputs or tofind patternsin data.
http://en.wikipedia.org/wiki/Soma_(biology)http://en.wikipedia.org/wiki/Soma_(biology)http://en.wikipedia.org/wiki/Soma_(biology)http://en.wikipedia.org/wiki/Dendritehttp://en.wikipedia.org/wiki/Dendritehttp://en.wikipedia.org/wiki/Dendritehttp://en.wikipedia.org/wiki/Axonhttp://en.wikipedia.org/wiki/Axonhttp://en.wikipedia.org/wiki/Axonhttp://en.wikipedia.org/wiki/Axon_hillockhttp://en.wikipedia.org/wiki/Axon_hillockhttp://en.wikipedia.org/wiki/Axon_hillockhttp://en.wikipedia.org/wiki/Voltagehttp://en.wikipedia.org/wiki/Voltagehttp://en.wikipedia.org/wiki/Voltagehttp://en.wikipedia.org/wiki/Cell_membranehttp://en.wikipedia.org/wiki/Cell_membranehttp://en.wikipedia.org/wiki/Cell_membranehttp://en.wikipedia.org/wiki/Ion_transporterhttp://en.wikipedia.org/wiki/Ion_transporterhttp://en.wikipedia.org/wiki/Ion_transporterhttp://en.wikipedia.org/wiki/Ion_channelhttp://en.wikipedia.org/wiki/Ion_channelhttp://en.wikipedia.org/wiki/Ion_channelhttp://en.wikipedia.org/wiki/Ionhttp://en.wikipedia.org/wiki/Ionhttp://en.wikipedia.org/wiki/Ionhttp://en.wikipedia.org/wiki/Sodiumhttp://en.wikipedia.org/wiki/Sodiumhttp://en.wikipedia.org/wiki/Potassiumhttp://en.wikipedia.org/wiki/Potassiumhttp://en.wikipedia.org/wiki/Potassiumhttp://en.wikipedia.org/wiki/Chloridehttp://en.wikipedia.org/wiki/Chloridehttp://en.wikipedia.org/wiki/Chloridehttp://en.wikipedia.org/wiki/Calciumhttp://en.wikipedia.org/wiki/Calciumhttp://en.wikipedia.org/wiki/Calciumhttp://en.wikipedia.org/wiki/Voltage-dependent_ion_channelhttp://en.wikipedia.org/wiki/Voltage-dependent_ion_channelhttp://en.wikipedia.org/wiki/Voltage-dependent_ion_channelhttp://en.wikipedia.org/wiki/Action_potentialhttp://en.wikipedia.org/wiki/Action_potentialhttp://en.wikipedia.org/wiki/Action_potentialhttp://en.wikipedia.org/wiki/Mitosishttp://en.wikipedia.org/wiki/Mitosishttp://en.wikipedia.org/wiki/Mitosishttp://en.wikipedia.org/wiki/Neurogenesis#Adult_neural_stem_cellshttp://en.wikipedia.org/wiki/Neurogenesis#Adult_neural_stem_cellshttp://en.wikipedia.org/wiki/Neurogenesis#Adult_neural_stem_cellshttp://en.wikipedia.org/wiki/Stem_cellhttp://en.wikipedia.org/wiki/Stem_cellhttp://en.wikipedia.org/wiki/Stem_cellhttp://en.wikipedia.org/wiki/Astrocyteshttp://en.wikipedia.org/wiki/Astrocyteshttp://en.wikipedia.org/wiki/Astrocyteshttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Pluripotenthttp://en.wikipedia.org/wiki/Pluripotenthttp://en.wikipedia.org/wiki/Pluripotenthttp://en.wikipedia.org/wiki/Artificial_neural_network#mw-headhttp://en.wikipedia.org/wiki/Artificial_neural_network#mw-headhttp://en.wikipedia.org/wiki/Artificial_neural_network#mw-headhttp://en.wikipedia.org/wiki/Artificial_neural_network#p-searchhttp://en.wikipedia.org/wiki/Artificial_neural_network#p-searchhttp://en.wikipedia.org/wiki/Artificial_neural_network#p-searchhttp://en.wikipedia.org/wiki/Wikipedia:Citing_sources#Inline_citationshttp://en.wikipedia.org/wiki/Wikipedia:Citing_sources#Inline_citationshttp://en.wikipedia.org/wiki/Wikipedia:Citing_sources#Inline_citationshttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edithttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edithttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edithttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Computational_modelhttp://en.wikipedia.org/wiki/Computational_modelhttp://en.wikipedia.org/wiki/Computational_modelhttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Non-linearhttp://en.wikipedia.org/wiki/Non-linearhttp://en.wikipedia.org/wiki/Statisticalhttp://en.wikipedia.org/wiki/Statisticalhttp://en.wikipedia.org/wiki/Data_modelinghttp://en.wikipedia.org/wiki/Data_modelinghttp://en.wikipedia.org/wiki/Data_modelinghttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/File:Question_book-new.svghttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Data_modelinghttp://en.wikipedia.org/wiki/Statisticalhttp://en.wikipedia.org/wiki/Non-linearhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Computational_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edithttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Citing_sources#Inline_citationshttp://en.wikipedia.org/wiki/Artificial_neural_network#p-searchhttp://en.wikipedia.org/wiki/Artificial_neural_network#mw-headhttp://en.wikipedia.org/wiki/Pluripotenthttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Astrocyteshttp://en.wikipedia.org/wiki/Stem_cellhttp://en.wikipedia.org/wiki/Neurogenesis#Adult_neural_stem_cellshttp://en.wikipedia.org/wiki/Mitosishttp://en.wikipedia.org/wiki/Action_potentialhttp://en.wikipedia.org/wiki/Voltage-dependent_ion_channelhttp://en.wikipedia.org/wiki/Calciumhttp://en.wikipedia.org/wiki/Chloridehttp://en.wikipedia.org/wiki/Potassiumhttp://en.wikipedia.org/wiki/Sodiumhttp://en.wikipedia.org/wiki/Ionhttp://en.wikipedia.org/wiki/Ion_channelhttp://en.wikipedia.org/wiki/Ion_transporterhttp://en.wikipedia.org/wiki/Cell_membranehttp://en.wikipedia.org/wiki/Voltagehttp://en.wikipedia.org/wiki/Axon_hillockhttp://en.wikipedia.org/wiki/Axonhttp://en.wikipedia.org/wiki/Dendritehttp://en.wikipedia.org/wiki/Soma_(biology)


4/38

An artificial neural network is an interconnected group of nodes, akin to the vast network of

neuronsin thehuman brain.

Contents

[hide]

1 Background 2 Models

o 2.1 Network functiono 2.2 Learning

2.2.1 Choosing a cost functiono 2.3 Learning paradigms

2.3.1 Supervised learning 2.3.2 Unsupervised learning 2.3.3 Reinforcement learning

o 2.4 Learning algorithms 3 Employing artificial neural networks 4 Applications

o 4.1 Real-life applicationso 4.2 Neural networks and neuroscience

4.2.1 Types of models 4.2.2 Current research

5 Neural network software 6 Types of artificial neural networks 7 Theoretical properties

o 7.1 Computational powero 7.2 Capacityo 7.3 Convergence
http://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Human_brainhttp://en.wikipedia.org/wiki/Human_brainhttp://en.wikipedia.org/wiki/Human_brainhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_network#Backgroundhttp://en.wikipedia.org/wiki/Artificial_neural_network#Backgroundhttp://en.wikipedia.org/wiki/Artificial_neural_network#Modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Network_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Network_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Choosing_a_cost_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Choosing_a_cost_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_paradigmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_paradigmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Supervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Supervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Unsupervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Unsupervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Reinforcement_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Reinforcement_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_algorithmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_algorithmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Employing_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Employing_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Real-life_applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Real-life_applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_networks_and_neurosciencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_networks_and_neurosciencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Current_researchhttp://en.wikipedia.org/wiki/Artificial_neural_network#Current_researchhttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_network_softwarehttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_network_softwarehttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Theoretical_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Theoretical_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Computational_powerhttp://en.wikipedia.org/wiki/Artificial_neural_network#Computational_powerhttp://en.wikipedia.org/wiki/Artificial_neural_network#Capacityhttp://en.wikipedia.org/wiki/Artificial_neural_network#Capacityhttp://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/File:Artificial_neural_network.svghttp://en.wikipedia.org/wiki/File:Artificial_neural_network.svghttp://en.wikipedia.org/wiki/File:Artificial_neural_network.svghttp://en.wikipedia.org/wiki/File:Artificial_neural_network.svghttp://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Capacityhttp://en.wikipedia.org/wiki/Artificial_neural_network#Computational_powerhttp://en.wikipedia.org/wiki/Artificial_neural_network#Theoretical_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_network_softwarehttp://en.wikipedia.org/wiki/Artificial_neural_network#Current_researchhttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_networks_and_neurosciencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Real-life_applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Employing_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_algorithmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Reinforcement_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Unsupervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Supervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_paradigmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Choosing_a_cost_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Network_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Backgroundhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Human_brainhttp://en.wikipedia.org/wiki/Neuron


5/38

o 7.4 Generalization and statisticso 7.5 Dynamic properties

8 Criticism 9 Gallery 10 See also 11 References 12 Bibliography 13 Further reading 14 External links

[edit] Background

The original inspiration for the termArtificial Neural Networkcame from examination ofcentralnervous systemsand theirneurons,axons,dendrites, andsynapses, which constitute the

processing elements ofbiological neural networksinvestigated byneuroscience. In an artificial

neural network, simple artificialnodes, variously called "neurons", "neurodes", "processingelements" (PEs) or "units", are connected together to form a network of nodes mimicking the

biological neural networkshence the term "artificial neural network".

Because neuroscience is still full of unanswered questions, and since there are many levels ofabstraction and therefore many ways to take inspiration from the brain, there is no single formal

definition of what an artificial neural network is. Generally, it involves a network of simple

processing elements that exhibit complex global behavior determined by connections betweenprocessing elements and element parameters. While an artificial neural network does not have to

be adaptive per se, its practical use comes with algorithms designed to alter the strength

(weights) of the connections in the network to produce a desired signal flow.

These networks are also similar to thebiological neural networksin the sense that functions are

performed collectively and in parallel by the units, rather than there being a clear delineation ofsubtasks to which various units are assigned (see alsoconnectionism). Currently, the term

Artificial Neural Network (ANN) tends to refer mostly to neural network models employed in

statistics,cognitive psychologyandartificial intelligence.Neural networkmodels designed withemulation of thecentral nervous system(CNS) in mind are a subject oftheoretical neuroscience

andcomputational neuroscience.

In modernsoftware implementationsof artificial neural networks, the approach inspired by

biology has been largely abandoned for a more practical approach based on statistics and signal

processing. In some of these systems, neural networks or parts of neural networks (such as

artificial neurons) are used as components in larger systems that combine both adaptive and non-adaptive elements. While the more general approach of suchadaptive systemsis more suitable

for real-world problem solving, it has far less to do with the traditional artificial intelligence

connectionist models. What they do have in common, however, is the principle of non-linear,distributed, parallel and local processing and adaptation.
http://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Generalization_and_statisticshttp://en.wikipedia.org/wiki/Artificial_neural_network#Generalization_and_statisticshttp://en.wikipedia.org/wiki/Artificial_neural_network#Dynamic_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Dynamic_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Criticismhttp://en.wikipedia.org/wiki/Artificial_neural_network#Criticismhttp://en.wikipedia.org/wiki/Artificial_neural_network#Galleryhttp://en.wikipedia.org/wiki/Artificial_neural_network#Galleryhttp://en.wikipedia.org/wiki/Artificial_neural_network#See_alsohttp://en.wikipedia.org/wiki/Artificial_neural_network#See_alsohttp://en.wikipedia.org/wiki/Artificial_neural_network#Referenceshttp://en.wikipedia.org/wiki/Artificial_neural_network#Referenceshttp://en.wikipedia.org/wiki/Artificial_neural_network#Bibliographyhttp://en.wikipedia.org/wiki/Artificial_neural_network#Bibliographyhttp://en.wikipedia.org/wiki/Artificial_neural_network#Further_readinghttp://en.wikipedia.org/wiki/Artificial_neural_network#Further_readinghttp://en.wikipedia.org/wiki/Artificial_neural_network#External_linkshttp://en.wikipedia.org/wiki/Artificial_neural_network#External_linkshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=1http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=1http://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Axonshttp://en.wikipedia.org/wiki/Axonshttp://en.wikipedia.org/wiki/Axonshttp://en.wikipedia.org/wiki/Dendriteshttp://en.wikipedia.org/wiki/Dendriteshttp://en.wikipedia.org/wiki/Dendriteshttp://en.wikipedia.org/wiki/Synapseshttp://en.wikipedia.org/wiki/Synapseshttp://en.wikipedia.org/wiki/Synapseshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Node_(neural_networks)http://en.wikipedia.org/wiki/Node_(neural_networks)http://en.wikipedia.org/wiki/Node_(neural_networks)http://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Statisticshttp://en.wikipedia.org/wiki/Statisticshttp://en.wikipedia.org/wiki/Cognitive_psychologyhttp://en.wikipedia.org/wiki/Cognitive_psychologyhttp://en.wikipedia.org/wiki/Cognitive_psychologyhttp://en.wikipedia.org/wiki/Artificial_intelligencehttp://en.wikipedia.org/wiki/Artificial_intelligencehttp://en.wikipedia.org/wiki/Artificial_intelligencehttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Theoretical_neurosciencehttp://en.wikipedia.org/wiki/Theoretical_neurosciencehttp://en.wikipedia.org/wiki/Theoretical_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Adaptive_systemshttp://en.wikipedia.org/wiki/Adaptive_systemshttp://en.wikipedia.org/wiki/Adaptive_systemshttp://en.wikipedia.org/wiki/Adaptive_systemshttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Theoretical_neurosciencehttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Artificial_intelligencehttp://en.wikipedia.org/wiki/Cognitive_psychologyhttp://en.wikipedia.org/wiki/Statisticshttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Node_(neural_networks)http://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Synapseshttp://en.wikipedia.org/wiki/Dendriteshttp://en.wikipedia.org/wiki/Axonshttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=1http://en.wikipedia.org/wiki/Artificial_neural_network#External_linkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Further_readinghttp://en.wikipedia.org/wiki/Artificial_neural_network#Bibliographyhttp://en.wikipedia.org/wiki/Artificial_neural_network#Referenceshttp://en.wikipedia.org/wiki/Artificial_neural_network#See_alsohttp://en.wikipedia.org/wiki/Artificial_neural_network#Galleryhttp://en.wikipedia.org/wiki/Artificial_neural_network#Criticismhttp://en.wikipedia.org/wiki/Artificial_neural_network#Dynamic_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Generalization_and_statistics


6/38

[edit] Models

Neural network models in artificial intelligence are usually referred to as artificial neuralnetworks (ANNs); these are essentially simple mathematical models defining a function

or a distribution over or both and , but sometimes models are also intimately associated

with a particular learning algorithm or learning rule. A common use of the phrase ANN modelreally means the definition of a class of such functions (where members of the class are obtained

by varying parameters, connection weights, or specifics of the architecture such as the number of

neurons or their connectivity).

[edit] Network function

See also:Graphical models

The word networkin the term 'artificial neural network' refers to the interconnections betweenthe neurons in the different layers of each system. An example system has three layers. The first

layer has input neurons, which send data via synapses to the second layer of neurons, and thenvia more synapses to the third layer of output neurons. More complex systems will have more

layers of neurons with some having increased layers of input neurons and output neurons. Thesynapses store parameters called "weights" that manipulate the data in the calculations.

An ANN is typically defined by three types of parameters:

1. The interconnection pattern between different layers of neurons2. The learning process for updating the weights of the interconnections3. The activation function that converts a neuron's weighted input to its output activation.

Mathematically, a neuron's network function is defined as a composition of other functions, which can further be defined as a composition of other functions. This can be conveniently

represented as a network structure, with arrows depicting the dependencies between variables. A

widely used type of composition is the nonlinear weighted sum, where ,

where (commonly referred to as theactivation function[1]

) is some predefined function, such asthehyperbolic tangent. It will be convenient for the following to refer to a collection of functions

as simply a vector .

ANN dependency graph
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=2http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=2http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=3http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=3http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=3http://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Hyperbolic_tangenthttp://en.wikipedia.org/wiki/Hyperbolic_tangenthttp://en.wikipedia.org/wiki/Hyperbolic_tangenthttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/Hyperbolic_tangenthttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=3http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=2


7/38

This figure depicts such a decomposition of , with dependencies between variables indicated by

arrows. These can be interpreted in two ways.

The first view is the functional view: the input is transformed into a 3-dimensional vector ,

which is then transformed into a 2-dimensional vector , which is finally transformed into .

This view is most commonly encountered in the context ofoptimization.

The second view is the probabilistic view: therandom variable depends upon the

random variable , which depends upon , which depends upon the random

variable . This view is most commonly encountered in the context ofgraphical models.

The two views are largely equivalent. In either case, for this particular network architecture, the

components of individual layers are independent of each other (e.g., the components of are

independent of each other given their input ). This naturally enables a degree of parallelism inthe implementation.

Recurrent ANN dependency graph

Networks such as the previous one are commonly calledfeedforward, because their graph is a

directed acyclic graph. Networks withcyclesare commonly calledrecurrent. Such networks arecommonly depicted in the manner shown at the top of the figure, where is shown as being

dependent upon itself. However, an implied temporal dependence is not shown.

[edit] Learning

What has attracted the most interest in neural networks is the possibility oflearning. Given aspecific taskto solve, and a class of functions, , learning means using a set ofobservations to

find which solves the task in some optimal sense.

This entails defining acost function such that, for the optimal solution ,

(i.e., no solution has a cost less than the cost of the optimal solution).

Thecost function is an important concept in learning, as it is a measure of how far away a

particular solution is from an optimal solution to the problem to be solved. Learning algorithms

search through the solution space to find a function that has the smallest possible cost.

For applications where the solution is dependent on some data, the cost must necessarily be a

function of the observations, otherwise we would not be modelling anything related to the data. It
http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Random_variablehttp://en.wikipedia.org/wiki/Random_variablehttp://en.wikipedia.org/wiki/Random_variablehttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Feedforward_neural_networkhttp://en.wikipedia.org/wiki/Feedforward_neural_networkhttp://en.wikipedia.org/wiki/Feedforward_neural_networkhttp://en.wikipedia.org/wiki/Directed_acyclic_graphhttp://en.wikipedia.org/wiki/Directed_acyclic_graphhttp://en.wikipedia.org/wiki/Path_(graph_theory)http://en.wikipedia.org/wiki/Path_(graph_theory)http://en.wikipedia.org/wiki/Path_(graph_theory)http://en.wikipedia.org/wiki/Recurrent_neural_networkhttp://en.wikipedia.org/wiki/Recurrent_neural_networkhttp://en.wikipedia.org/wiki/Recurrent_neural_networkhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=4http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=4http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=4http://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=4http://en.wikipedia.org/wiki/Recurrent_neural_networkhttp://en.wikipedia.org/wiki/Path_(graph_theory)http://en.wikipedia.org/wiki/Directed_acyclic_graphhttp://en.wikipedia.org/wiki/Feedforward_neural_networkhttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Random_variablehttp://en.wikipedia.org/wiki/Optimization_(mathematics)


8/38

is frequently defined as astatisticto which only approximations can be made. As a simple

example, consider the problem of finding the model , which minimizes , for

data pairs drawn from some distribution . In practical situations we would only have

samples from and thus, for the above example, we would only minimize

. Thus, the cost is minimized over a sample of the data rather than the

entire data set.

When some form ofonline machine learningmust be used, where the cost is partiallyminimized as each new example is seen. While online machine learning is often used when is

fixed, it is most useful in the case where the distribution changes slowly over time. In neural

network methods, some form of online machine learning is frequently used for finite datasets.

See also:Optimization (mathematics),Estimation theory, andMachine learning

[edit] Choosing a cost function

While it is possible to define some arbitrary,ad hoccost function, frequently a particular costwill be used, either because it has desirable properties (such asconvexity) or because it arises

naturally from a particular formulation of the problem (e.g., in a probabilistic formulation the

posterior probability of the model can be used as an inverse cost). Ultimately, the cost functionwill depend on the desired task. An overview of the three main categories of learning tasks is

provided below.

[edit] Learning paradigms

There are three major learning paradigms, each corresponding to a particular abstract learning

task. These aresupervised learning,unsupervised learningandreinforcement learning.

[edit] Supervised learning

Insupervised learning, we are given a set of example pairs and the aim is to find a

function in the allowed class of functions that matches the examples. In other words, wewish to inferthe mapping implied by the data; the cost function is related to the mismatch

between our mapping and the data and it implicitly contains prior knowledge about the problem

domain.

A commonly used cost is themean-squared error, which tries to minimize the average squared

error between the network's output, f(x), and the target value y over all the example pairs. When

one tries to minimize this cost usinggradient descentfor the class of neural networks calledmultilayer perceptrons, one obtains the common and well-knownbackpropagation algorithmfor

training neural networks.

Tasks that fall within the paradigm of supervised learning arepattern recognition(also known as

classification) andregression(also known as function approximation). The supervised learning

paradigm is also applicable to sequential data (e.g., for speech and gesture recognition). This can
http://en.wikipedia.org/wiki/Statistichttp://en.wikipedia.org/wiki/Statistichttp://en.wikipedia.org/wiki/Statistichttp://en.wikipedia.org/wiki/Online_machine_learninghttp://en.wikipedia.org/wiki/Online_machine_learninghttp://en.wikipedia.org/wiki/Online_machine_learninghttp://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Estimation_theoryhttp://en.wikipedia.org/wiki/Estimation_theoryhttp://en.wikipedia.org/wiki/Estimation_theoryhttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=5http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=5http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=5http://en.wikipedia.org/wiki/Ad_hochttp://en.wikipedia.org/wiki/Ad_hochttp://en.wikipedia.org/wiki/Ad_hochttp://en.wikipedia.org/wiki/Convex_functionhttp://en.wikipedia.org/wiki/Convex_functionhttp://en.wikipedia.org/wiki/Convex_functionhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=6http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=6http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=6http://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=7http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=7http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=7http://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Mean-squared_errorhttp://en.wikipedia.org/wiki/Mean-squared_errorhttp://en.wikipedia.org/wiki/Mean-squared_errorhttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Multilayer_perceptronhttp://en.wikipedia.org/wiki/Multilayer_perceptronhttp://en.wikipedia.org/wiki/Backpropagationhttp://en.wikipedia.org/wiki/Backpropagationhttp://en.wikipedia.org/wiki/Backpropagationhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Backpropagationhttp://en.wikipedia.org/wiki/Multilayer_perceptronhttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Mean-squared_errorhttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=7http://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=6http://en.wikipedia.org/wiki/Convex_functionhttp://en.wikipedia.org/wiki/Ad_hochttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=5http://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Estimation_theoryhttp://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Online_machine_learninghttp://en.wikipedia.org/wiki/Statistic


9/38

be thought of as learning with a "teacher," in the form of a function that provides continuous

feedback on the quality of solutions obtained thus far.

[edit] Unsupervised learning

Inunsupervised learning, some data is given and the cost function to be minimized, that can beany function of the data and the network's output, .

The cost function is dependent on the task (what we are trying to model) and our a priori

assumptions (the implicit properties of our model, its parameters and the observed variables).

As a trivial example, consider the model , where is a constant and the cost

. Minimizing this cost will give us a value of that is equal to the mean of the

data. The cost function can be much more complicated. Its form depends on the application: forexample, in compression it could be related to themutual informationbetween x and y, whereas

in statistical modeling, it could be related to theposterior probabilityof the model given the data.

(Note that in both of those examples those quantities would be maximized rather thanminimized).

Tasks that fall within the paradigm of unsupervised learning are in general estimationproblems;the applications includeclustering, the estimation ofstatistical distributions,compressionand

filtering.

[edit] Reinforcement learning

Inreinforcement learning, data are usually not given, but generated by an agent's interactionswith the environment. At each point in time , the agent performs an action and the

environment generates an observation and an instantaneous cost , according to some(usually unknown) dynamics. The aim is to discover apolicy for selecting actions that minimizes

some measure of a long-term cost; i.e., the expected cumulative cost. The environment'sdynamics and the long-term cost for each policy are usually unknown, but can be estimated.

More formally, the environment is modeled as aMarkov decision process(MDP) with states

and actions with the following probability distributions: the instantaneous

cost distribution , the observation distribution and the transition ,

while a policy is defined as conditional distribution over actions given the observations. Taken

together, the two define aMarkov chain(MC). The aim is to discover the policy that minimizesthe cost; i.e., the MC for which the cost is minimal.

ANNs are frequently used in reinforcement learning as part of the overall algorithm.

Tasks that fall within the paradigm of reinforcement learning are control problems,gamesand

othersequential decision makingtasks.

See also:dynamic programmingandstochastic control
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=8http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=8http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=8http://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Mutual_informationhttp://en.wikipedia.org/wiki/Mutual_informationhttp://en.wikipedia.org/wiki/Mutual_informationhttp://en.wikipedia.org/wiki/Posterior_probabilityhttp://en.wikipedia.org/wiki/Posterior_probabilityhttp://en.wikipedia.org/wiki/Posterior_probabilityhttp://en.wikipedia.org/wiki/Estimationhttp://en.wikipedia.org/wiki/Estimationhttp://en.wikipedia.org/wiki/Estimationhttp://en.wikipedia.org/wiki/Data_clusteringhttp://en.wikipedia.org/wiki/Data_clusteringhttp://en.wikipedia.org/wiki/Data_clusteringhttp://en.wikipedia.org/wiki/Statistical_distributionshttp://en.wikipedia.org/wiki/Statistical_distributionshttp://en.wikipedia.org/wiki/Statistical_distributionshttp://en.wikipedia.org/wiki/Data_compressionhttp://en.wikipedia.org/wiki/Data_compressionhttp://en.wikipedia.org/wiki/Data_compressionhttp://en.wikipedia.org/wiki/Bayesian_spam_filteringhttp://en.wikipedia.org/wiki/Bayesian_spam_filteringhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=9http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=9http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=9http://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Markov_decision_processhttp://en.wikipedia.org/wiki/Markov_decision_processhttp://en.wikipedia.org/wiki/Markov_decision_processhttp://en.wikipedia.org/wiki/Markov_chainhttp://en.wikipedia.org/wiki/Markov_chainhttp://en.wikipedia.org/wiki/Markov_chainhttp://en.wikipedia.org/wiki/Gamehttp://en.wikipedia.org/wiki/Gamehttp://en.wikipedia.org/wiki/Gamehttp://en.wikipedia.org/w/index.php?title=Sequential_decision_making&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Sequential_decision_making&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Sequential_decision_making&action=edit&redlink=1http://en.wikipedia.org/wiki/Dynamic_programminghttp://en.wikipedia.org/wiki/Dynamic_programminghttp://en.wikipedia.org/wiki/Dynamic_programminghttp://en.wikipedia.org/wiki/Stochastic_controlhttp://en.wikipedia.org/wiki/Stochastic_controlhttp://en.wikipedia.org/wiki/Stochastic_controlhttp://en.wikipedia.org/wiki/Stochastic_controlhttp://en.wikipedia.org/wiki/Dynamic_programminghttp://en.wikipedia.org/w/index.php?title=Sequential_decision_making&action=edit&redlink=1http://en.wikipedia.org/wiki/Gamehttp://en.wikipedia.org/wiki/Markov_chainhttp://en.wikipedia.org/wiki/Markov_decision_processhttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=9http://en.wikipedia.org/wiki/Bayesian_spam_filteringhttp://en.wikipedia.org/wiki/Data_compressionhttp://en.wikipedia.org/wiki/Statistical_distributionshttp://en.wikipedia.org/wiki/Data_clusteringhttp://en.wikipedia.org/wiki/Estimationhttp://en.wikipedia.org/wiki/Posterior_probabilityhttp://en.wikipedia.org/wiki/Mutual_informationhttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=8


10/38

[edit] Learning algorithms

Training a neural network model essentially means selecting one model from the set of allowed

models (or, in aBayesianframework, determining a distribution over the set of allowed models)

that minimizes the cost criterion. There are numerous algorithms available for training neural

network models; most of them can be viewed as a straightforward application ofoptimizationtheory andstatistical estimation. Recent developments in this field useparticle swarm

optimizationand otherswarm intelligencetechniques.

Most of the algorithms used in training artificial neural networks employ some form ofgradient

descent. This is done by simply taking the derivative of the cost function with respect to the

network parameters and then changing those parameters in agradient-relateddirection.

Evolutionary methods,simulated annealing,expectation-maximizationandnon-parametric

methodsare some commonly used methods for training neural networks.

See also:machine learning

Temporalperceptual learningrelies on finding temporal relationships in sensory signal streams.

In an environment, statistically salient temporal correlations can be found by monitoring thearrival times of sensory signals. This is done by theperceptual network.

[edit] Employing artificial neural networks

Perhaps the greatest advantage of ANNs is their ability to be used as an arbitrary function

approximation mechanism that 'learns' from observed data. However, using them is not so

straightforward and a relatively good understanding of the underlying theory is essential.

Choice of model: This will depend on the data representation and the application. Overlycomplex models tend to lead to problems with learning.

Learning algorithm: There are numerous trade-offs between learning algorithms. Almostany algorithm will work well with the correcthyperparametersfor training on a

particular fixed data set. However selecting and tuning an algorithm for training onunseen data requires a significant amount of experimentation.

Robustness: If the model, cost function and learning algorithm are selected appropriatelythe resulting ANN can be extremely robust.

With the correct implementation, ANNs can be used naturally inonline learningand large data

set applications. Their simple implementation and the existence of mostly local dependenciesexhibited in the structure allows for fast, parallel implementations in hardware.

[edit] Applications
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=10http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=10http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=10http://en.wikipedia.org/wiki/Bayesian_probabilityhttp://en.wikipedia.org/wiki/Bayesian_probabilityhttp://en.wikipedia.org/wiki/Bayesian_probabilityhttp://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Statistical_estimationhttp://en.wikipedia.org/wiki/Statistical_estimationhttp://en.wikipedia.org/wiki/Statistical_estimationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Swarm_intelligencehttp://en.wikipedia.org/wiki/Swarm_intelligencehttp://en.wikipedia.org/wiki/Swarm_intelligencehttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient-relatedhttp://en.wikipedia.org/wiki/Gradient-relatedhttp://en.wikipedia.org/wiki/Gradient-relatedhttp://en.wikipedia.org/wiki/Evolutionary_methodshttp://en.wikipedia.org/wiki/Evolutionary_methodshttp://en.wikipedia.org/wiki/Simulated_annealinghttp://en.wikipedia.org/wiki/Simulated_annealinghttp://en.wikipedia.org/wiki/Simulated_annealinghttp://en.wikipedia.org/wiki/Expectation-maximizationhttp://en.wikipedia.org/wiki/Expectation-maximizationhttp://en.wikipedia.org/wiki/Expectation-maximizationhttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Perceptual_learninghttp://en.wikipedia.org/wiki/Perceptual_learninghttp://en.wikipedia.org/wiki/Perceptual_learninghttp://en.wikipedia.org/w/index.php?title=Perceptual_network&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Perceptual_network&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Perceptual_network&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=11http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=11http://en.wikipedia.org/wiki/Hyperparameterhttp://en.wikipedia.org/wiki/Hyperparameterhttp://en.wikipedia.org/wiki/Hyperparameterhttp://en.wikipedia.org/wiki/Online_algorithmhttp://en.wikipedia.org/wiki/Online_algorithmhttp://en.wikipedia.org/wiki/Online_algorithmhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=12http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=12http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=12http://en.wikipedia.org/wiki/Online_algorithmhttp://en.wikipedia.org/wiki/Hyperparameterhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=11http://en.wikipedia.org/w/index.php?title=Perceptual_network&action=edit&redlink=1http://en.wikipedia.org/wiki/Perceptual_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Expectation-maximizationhttp://en.wikipedia.org/wiki/Simulated_annealinghttp://en.wikipedia.org/wiki/Evolutionary_methodshttp://en.wikipedia.org/wiki/Gradient-relatedhttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Swarm_intelligencehttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Statistical_estimationhttp://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Bayesian_probabilityhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=10


11/38

The utility of artificial neural network models lies in the fact that they can be used to infer a

function from observations. This is particularly useful in applications where the complexity ofthe data or task makes the design of such a function by hand impractical.

[edit] Real-life applications

The tasks artificial neural networks are applied to tend to fall within the following broad

categories:

Function approximation, orregression analysis, includingtime series prediction,fitnessapproximationand modeling.

Classification, includingpatternand sequence recognition,novelty detectionandsequential decision making.

Data processing, including filtering, clustering, blind source separation and compression. Robotics, including directing manipulators,Computer numerical control.

Application areas include system identification and control (vehicle control, process control),quantum chemistry,

[2]game-playing and decision making (backgammon, chess, racing), pattern

recognition (radar systems, face identification, object recognition and more), sequence

recognition (gesture, speech, handwritten text recognition), medical diagnosis, financial

applications (automated trading systems),data mining(or knowledge discovery in databases,"KDD"), visualization ande-mail spamfiltering.

[edit] Neural networks and neuroscience

Theoretical andcomputational neuroscienceis the field concerned with the theoretical analysis

and computational modeling of biological neural systems. Since neural systems are intimately

related to cognitive processes and behavior, the field is closely related to cognitive andbehavioral modeling.

The aim of the field is to create models of biological neural systems in order to understand howbiological systems work. To gain this understanding, neuroscientists strive to make a link

between observed biological processes (data), biologically plausible mechanisms for neural

processing and learning (biological neural networkmodels) and theory (statistical learningtheory andinformation theory).

[edit] Types of models

Many models are used in the field defined at different levels of abstraction and modelingdifferent aspects of neural systems. They range from models of the short-term behavior ofindividual neurons, models of how the dynamics of neural circuitry arise from interactions

between individual neurons and finally to models of how behavior can arise from abstract neural

modules that represent complete subsystems. These include models of the long-term, and short-

term plasticity, of neural systems and their relations to learning and memory from the individualneuron to the system level.
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=13http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=13http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=13http://en.wikipedia.org/wiki/Function_approximationhttp://en.wikipedia.org/wiki/Function_approximationhttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Time_series_predictionhttp://en.wikipedia.org/wiki/Time_series_predictionhttp://en.wikipedia.org/wiki/Time_series_predictionhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Statistical_classificationhttp://en.wikipedia.org/wiki/Statistical_classificationhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Novelty_detectionhttp://en.wikipedia.org/wiki/Novelty_detectionhttp://en.wikipedia.org/wiki/Novelty_detectionhttp://en.wikipedia.org/wiki/Data_processinghttp://en.wikipedia.org/wiki/Data_processinghttp://en.wikipedia.org/wiki/Roboticshttp://en.wikipedia.org/wiki/Roboticshttp://en.wikipedia.org/wiki/Computer_numerical_controlhttp://en.wikipedia.org/wiki/Computer_numerical_controlhttp://en.wikipedia.org/wiki/Computer_numerical_controlhttp://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-Balabin_2009-1http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-Balabin_2009-1http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-Balabin_2009-1http://en.wikipedia.org/wiki/Data_mininghttp://en.wikipedia.org/wiki/Data_mininghttp://en.wikipedia.org/wiki/Data_mininghttp://en.wikipedia.org/wiki/E-mail_spamhttp://en.wikipedia.org/wiki/E-mail_spamhttp://en.wikipedia.org/wiki/E-mail_spamhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=14http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=14http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=14http://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Information_theoryhttp://en.wikipedia.org/wiki/Information_theoryhttp://en.wikipedia.org/wiki/Information_theoryhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=15http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=15http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=15http://en.wikipedia.org/wiki/Biological_neuron_modelshttp://en.wikipedia.org/wiki/Biological_neuron_modelshttp://en.wikipedia.org/wiki/Biological_neuron_modelshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=15http://en.wikipedia.org/wiki/Information_theoryhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=14http://en.wikipedia.org/wiki/E-mail_spamhttp://en.wikipedia.org/wiki/Data_mininghttp://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-Balabin_2009-1http://en.wikipedia.org/wiki/Computer_numerical_controlhttp://en.wikipedia.org/wiki/Roboticshttp://en.wikipedia.org/wiki/Data_processinghttp://en.wikipedia.org/wiki/Novelty_detectionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Statistical_classificationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Time_series_predictionhttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Function_approximationhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=13


12/38

[edit] Current research

This section does notciteanyreferences or sources.Please help improve this section by adding citations toreliable sources. Unsourced material may be

challengedandremoved.(June 2010)

While initial research had been concerned mostly with the electrical characteristics of neurons, aparticularly important part of the investigation in recent years has been the exploration of the role

ofneuromodulatorssuch asdopamine,acetylcholine, andserotoninon behavior and learning.

Biophysicalmodels, such asBCM theory, have been important in understanding mechanisms for

synaptic plasticity, and have had applications in both computer science and neuroscience.

Research is ongoing in understanding the computational algorithms used in the brain, with somerecent biological evidence forradial basis networksandneural backpropagationas mechanisms

for processing data.

Computational devices have been created in CMOS for both biophysical simulation andneuromorphic computing. More recent efforts show promise for creatingnanodevicesfor very

large scaleprincipal componentsanalyses andconvolution. If successful, these effort could usherin a new era ofneural computingthat is a step beyond digital computing, because it depends on

learningrather thanprogrammingand because it is fundamentallyanalograther thandigitaleven

though the first instantiations may in fact be with CMOS digital devices.

[edit] Neural network software

Main article:Neural network software

Neural network software is used tosimulate,research,developand apply artificial neural

networks,biological neural networksand in some cases a wider array ofadaptive systems.

[edit] Types of artificial neural networks

Main article:Types of artificial neural networks

Artificial neural network types vary from those with only one or two layers of single directionlogic, to complicated multiinput many directional feedback loop and layers. On the whole, these

systems use algorithms in their programming to determine control and organization of their

functions. Some may be as simple, one neuron layer with an input and an output, and others canmimic complex systems such asdANN, which can mimic chromosomal DNA through sizes atcellular level, into artificial organisms and simulate reproduction, mutation and population

sizes.[3]

Most systems use "weights" to change the parameters of the throughput and the varying

connections to the neurons. Artificial neural networks can be autonomous and learn by inputfrom outside "teachers" or even self-teaching from written in rules.
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=16http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=16http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=16http://en.wikipedia.org/wiki/Wikipedia:Citing_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Citing_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Citing_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Neuromodulatorshttp://en.wikipedia.org/wiki/Neuromodulatorshttp://en.wikipedia.org/wiki/Neuromodulatorshttp://en.wikipedia.org/wiki/Dopaminehttp://en.wikipedia.org/wiki/Dopaminehttp://en.wikipedia.org/wiki/Dopaminehttp://en.wikipedia.org/wiki/Acetylcholinehttp://en.wikipedia.org/wiki/Acetylcholinehttp://en.wikipedia.org/wiki/Acetylcholinehttp://en.wikipedia.org/wiki/Serotoninhttp://en.wikipedia.org/wiki/Serotoninhttp://en.wikipedia.org/wiki/Serotoninhttp://en.wikipedia.org/wiki/Biophysicshttp://en.wikipedia.org/wiki/Biophysicshttp://en.wikipedia.org/wiki/BCM_theoryhttp://en.wikipedia.org/wiki/BCM_theoryhttp://en.wikipedia.org/wiki/BCM_theoryhttp://en.wikipedia.org/wiki/Synaptic_plasticityhttp://en.wikipedia.org/wiki/Synaptic_plasticityhttp://en.wikipedia.org/wiki/Radial_basis_networkshttp://en.wikipedia.org/wiki/Radial_basis_networkshttp://en.wikipedia.org/wiki/Radial_basis_networkshttp://en.wikipedia.org/wiki/Neural_backpropagationhttp://en.wikipedia.org/wiki/Neural_backpropagationhttp://en.wikipedia.org/wiki/Neural_backpropagationhttp://en.wikipedia.org/w/index.php?title=Nanodevice&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Nanodevice&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Nanodevice&action=edit&redlink=1http://en.wikipedia.org/wiki/Principal_componenthttp://en.wikipedia.org/wiki/Principal_componenthttp://en.wikipedia.org/wiki/Principal_componenthttp://en.wikipedia.org/wiki/Convolutionhttp://en.wikipedia.org/wiki/Convolutionhttp://en.wikipedia.org/wiki/Convolutionhttp://en.wikipedia.org/wiki/Neural_computinghttp://en.wikipedia.org/wiki/Neural_computinghttp://en.wikipedia.org/wiki/Neural_computinghttp://en.wikipedia.org/wiki/Learninghttp://en.wikipedia.org/wiki/Learninghttp://en.wikipedia.org/wiki/Programminghttp://en.wikipedia.org/wiki/Programminghttp://en.wikipedia.org/wiki/Programminghttp://en.wikipedia.org/wiki/Analog_signalhttp://en.wikipedia.org/wiki/Analog_signalhttp://en.wikipedia.org/wiki/Analog_signalhttp://en.wikipedia.org/wiki/Digitalhttp://en.wikipedia.org/wiki/Digitalhttp://en.wikipedia.org/wiki/Digitalhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=17http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=17http://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Simulationhttp://en.wikipedia.org/wiki/Simulationhttp://en.wikipedia.org/wiki/Simulationhttp://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Technology_developmenthttp://en.wikipedia.org/wiki/Technology_developmenthttp://en.wikipedia.org/wiki/Technology_developmenthttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=18http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=18http://en.wikipedia.org/wiki/Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects#Software_librarieshttp://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects#Software_librarieshttp://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects#Software_librarieshttp://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-2http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-2http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-2http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-2http://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects#Software_librarieshttp://en.wikipedia.org/wiki/Types_of_artificial_neural_networkshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=18http://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Technology_developmenthttp://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Simulationhttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=17http://en.wikipedia.org/wiki/Digitalhttp://en.wikipedia.org/wiki/Analog_signalhttp://en.wikipedia.org/wiki/Programminghttp://en.wikipedia.org/wiki/Learninghttp://en.wikipedia.org/wiki/Neural_computinghttp://en.wikipedia.org/wiki/Convolutionhttp://en.wikipedia.org/wiki/Principal_componenthttp://en.wikipedia.org/w/index.php?title=Nanodevice&action=edit&redlink=1http://en.wikipedia.org/wiki/Neural_backpropagationhttp://en.wikipedia.org/wiki/Radial_basis_networkshttp://en.wikipedia.org/wiki/Synaptic_plasticityhttp://en.wikipedia.org/wiki/BCM_theoryhttp://en.wikipedia.org/wiki/Biophysicshttp://en.wikipedia.org/wiki/Serotoninhttp://en.wikipedia.org/wiki/Acetylcholinehttp://en.wikipedia.org/wiki/Dopaminehttp://en.wikipedia.org/wiki/Neuromodulatorshttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Citing_sourceshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit&section=16


13/38

[edit] Theoretical properties

[edit] Computational power

The multi-layerperceptron(MLP) is a universal function approximator, as proven by the

Cybenko theorem. However, the proof is not constructive regarding the number of neuronsrequired or the settings of the weights.

Work byHava SiegelmannandEduardo D. Sontaghas provided a proof that a specific recurrentarchitecture with rational valued weights (as opposed to full precisionreal number-valued

weights) has the full power of aUniversal Turing Machine[4]

using a finite number of neurons

and standard linear connections. They have further shown that the use of irrational values for

weights results in a machine withsuper-Turingpower.

[edit] Capacity

Artificial neural network models have a property called 'capacity', which roughly corresponds to

their ability to model any given function. It is related to the amount of information that can be

stored in the network and to the notion of complexity.

[edit] Convergence

Nothing can be said in general about convergence since it depends on a number of factors.Firstly, there may exist many local minima. This depends on the cost function and the model.

Secondly, the optimization method used might not be guaranteed to converge when far away

from a local minimum. Thirdly, for a very large amount of data or parameters, some methodsbecome impractical. In general, it has been found that theoretical guarantees regarding

convergence are an unreliable guide to practical application.

[edit] Generalization and statistics

In applications where the goal is to create a system that generalizes well in unseen examples, the

problem of over-training has emerged. This arises in convoluted

neural networ;

Documents