neural networks and deep learning (part 1 of 2): an introduction - valentino zocca, real data...

An introduction to neural nets

by Valentino Zoccavzocca@realdatamachines.com

vzocca@gmail.com+39 333 789 2692+1 202 640 1381

Deep Learning in AIDigitalisation

Features

Classification/Prediction

AnalysisInteraction

The World

Sensor

Machine Learning

1642• Pascaline: first mechanical adder, invented by

French mathematician Blaise Pascal, using a system of gears and wheels could add and subtract numbers.

1694• Machine by Gottfried Wilhelm Von Leibniz

who also developed calculus and invented the binary system. His machine could also multiplicate and divide.

1801• Joseph Marie Charles invents the Jacquard

loom to weave different patterns using cards punched with holes. Precursor for modern computers and data storage.

1890• Herman Hollerith, founder of the Tabulating

Machine Company (later merged into IBM), creates a mechanical tabulator using punched cards to store data to calculate statistics. He is regarded as the father of modern machine data processing.

1957• Frank Rosenblatt invents the perceptron

algorithm, the first neural networks implementation. It was later proved by Marvin Minsky and Seymour Papert in 1969 that it could not learn the XOR function.

1974• Paul Werbos’s Ph.D. thesis describes the

process of training neural nets through back-propagation.

Supervised Learning

1. Input Data2. Process the information3. Check the output4. Learn new rule5. New rule is applied to better performance

Neural Networks

The theory of neural networks arises from the attempt to mimic our biological brain in order to create machines that can learn or perform pattern recognition in order to make predictions.

Neural Networks

Neural networks are systems comprised of many ”neurons”, which are the units of the neural net.

Neural Networks

In neural networks, a space of ”weights” is defined alongside the input space. The weights and the input together define the activity rules that in turn will define the output according to specified activation functions.

Neural Networks

The weights can change with time as the neural network learns and their change may be specified by some learning rule which will generally be depending on the activities of the neurons.

Perspective

• Biggest artificial neural network to-date: Over 11 billion parameters. (1.1 * 10ˆ10).

• Number of neurons in brain 10ˆ11, each with about 10ˆ4 connections for a total of 10ˆ15 parameters.

A model for a neuron

Images from upcoming book "Python Deep Learning"

Neural Networks

x w wx

Neural Networks

x w1

w1x+w2y

y w2

Neural Networks

x w1

w1x+w2y+b

y w2

1

b

w1x+w2y+b > 0

w1x+w2y > -b

Neural Networks

x w1 σ(w1x+w2y+b)

y w2

1

b

Neural Networks

The Universal Approximation Theorem

Neural networks with a single hidden layer can be used to approximate any continuous function to any desired precision.

A logistic sigmoid: 1/(1+exp(-x))

In general: 1/(1+exp(-wx-b))

Why the Universal Approximation theorem is true

A step function

Picture by John Kaufhold

Classic Neural Networks

Deep neural networks in the 80’s

Recent Developments in Deep Learning by Geoff Hintonhttps://www.youtube.com/watch?v=vShMxxqtDDs

Learning Representations

Neural Networks for Machine Learning by Geoff Hintonhttps://class.coursera.org/neuralnets-2012-001/lecture

Learning Representations

Neural Networks for Machine Learning by Geoff Hintonhttps://class.coursera.org/neuralnets-2012-001/lecture

Building High-level Features Using Large ScaleUnsupervised Learning

http://research.google.com/archive/unsupervised_icml2012.html

http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf

http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf

http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf

http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf

http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf

http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf

http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf

Picture by John Kaufholdhttp://www.cs.toronto.edu/~rsalakhu/ISBI1_pdf_version.pdf

Next Talk (Part II)

• Ising Models• Restricted Boltzmann

Machines• Convolutional Networks

Various Resources

Courses • Geoff Hinton Coursera course-http://www.coursera.org/course/neuralnets)• Andrew Ng Coursera course-http://www.coursera.org/course/ml

An introduction to neural nets

An Introduction by Valentino Zoccavzocca@gmail.com

vzocca@realdatamachines.com+39 333 789 2692+1 202 640 1381