neural networks and deep learning (part 1 of 2): an introduction - valentino zocca, real data...
TRANSCRIPT
An introduction to neural nets
by Valentino [email protected]
[email protected]+39 333 789 2692+1 202 640 1381
Deep Learning in AIDigitalisation
Features
Classification/Prediction
AnalysisInteraction
The World
Sensor
Machine Learning
1642• Pascaline: first mechanical adder, invented by
French mathematician Blaise Pascal, using a system of gears and wheels could add and subtract numbers.
1694• Machine by Gottfried Wilhelm Von Leibniz
who also developed calculus and invented the binary system. His machine could also multiplicate and divide.
1801• Joseph Marie Charles invents the Jacquard
loom to weave different patterns using cards punched with holes. Precursor for modern computers and data storage.
1890• Herman Hollerith, founder of the Tabulating
Machine Company (later merged into IBM), creates a mechanical tabulator using punched cards to store data to calculate statistics. He is regarded as the father of modern machine data processing.
1957• Frank Rosenblatt invents the perceptron
algorithm, the first neural networks implementation. It was later proved by Marvin Minsky and Seymour Papert in 1969 that it could not learn the XOR function.
1974• Paul Werbos’s Ph.D. thesis describes the
process of training neural nets through back-propagation.
Supervised Learning
1. Input Data2. Process the information3. Check the output4. Learn new rule5. New rule is applied to better performance
Neural Networks
The theory of neural networks arises from the attempt to mimic our biological brain in order to create machines that can learn or perform pattern recognition in order to make predictions.
Neural Networks
Neural networks are systems comprised of many ”neurons”, which are the units of the neural net.
Neural Networks
In neural networks, a space of ”weights” is defined alongside the input space. The weights and the input together define the activity rules that in turn will define the output according to specified activation functions.
Neural Networks
The weights can change with time as the neural network learns and their change may be specified by some learning rule which will generally be depending on the activities of the neurons.
Perspective
• Biggest artificial neural network to-date: Over 11 billion parameters. (1.1 * 10ˆ10).
• Number of neurons in brain 10ˆ11, each with about 10ˆ4 connections for a total of 10ˆ15 parameters.
A model for a neuron
Images from upcoming book "Python Deep Learning"
Neural Networks
x w wx
Neural Networks
x w1
w1x+w2y
y w2
Neural Networks
x w1
w1x+w2y+b
y w2
1
b
w1x+w2y+b > 0
w1x+w2y > -b
Neural Networks
x w1 σ(w1x+w2y+b)
y w2
1
b
Neural Networks
The Universal Approximation Theorem
Neural networks with a single hidden layer can be used to approximate any continuous function to any desired precision.
A logistic sigmoid: 1/(1+exp(-x))
In general: 1/(1+exp(-wx-b))
Why the Universal Approximation theorem is true
A step function
Picture by John Kaufhold
Classic Neural Networks
Deep neural networks in the 80’s
Recent Developments in Deep Learning by Geoff Hintonhttps://www.youtube.com/watch?v=vShMxxqtDDs
Learning Representations
Neural Networks for Machine Learning by Geoff Hintonhttps://class.coursera.org/neuralnets-2012-001/lecture
Learning Representations
Neural Networks for Machine Learning by Geoff Hintonhttps://class.coursera.org/neuralnets-2012-001/lecture
Building High-level Features Using Large ScaleUnsupervised Learning
http://research.google.com/archive/unsupervised_icml2012.html
http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf
http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf
http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf
http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf
http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf
http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf
http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf
Picture by John Kaufholdhttp://www.cs.toronto.edu/~rsalakhu/ISBI1_pdf_version.pdf
Next Talk (Part II)
• Ising Models• Restricted Boltzmann
Machines• Convolutional Networks
Various Resources
Courses • Geoff Hinton Coursera course-http://www.coursera.org/course/neuralnets)• Andrew Ng Coursera course-http://www.coursera.org/course/ml
An introduction to neural nets
An Introduction by Valentino [email protected]
[email protected]+39 333 789 2692+1 202 640 1381