neural networks and handwriting recognitiondyong/math164/2007/sloss/presentation.pdf · neural...

NeuralNetworks andHandwritingRecognition

Steven SlossMath 164

Background

NeuralNetworks

Neural NetworkStructure

Training NeuralNetworks

Motivation

Problem

Solution

Single-CharacterRecognition

MultipleCharacterRecognition

RecognizingMath

Outlook

Neural Networks and Handwriting Recognition


26 April 2007



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Presentation Outline

1 Background

2 Neural NetworksNeural Network StructureTraining Neural Networks

3 Motivation

4 Problem

5 SolutionSingle-Character RecognitionMultiple Character RecognitionRecognizing Math

6 Outlook



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Artificial Intelligence

Todays computers can perform many computations much,much faster than a human being can.

Example

Integrate 10

1 x2dx

My laptop: 0.3867... seconds.

Me: 1.3 minutes



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Artificial Intelligence, Contd.

There are many areas where computers fall short, however.

Example

Find the swingset:

My 2 year old neighbor: 1.2 seconds

A computer: ???



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Artificial Intelligence, Contd.

Artificial Intelligence is the field of mathematics andcomputer science that tries to give computers human-likecognitive abilities.

Neural Networks are an important way to do this.



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook


1 Background


3 Motivation

4 Problem


6 Outlook



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Neural Networks

Neural networks - teaching a computer to do patternrecognition like a human brain!



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Biological Neural Networks versus Artificial NeuralNetworks

Lots of parallels between artificial and biological neuralnetworks.

Both biological and artificial neural networks use neurons.



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook


Biological neurons:Accepts signal from Dentries.Upon accepting a signal, that neuron may fireIf it fires, a signal is transmitted over the neurons axon,leaving the neuron over the axon terminalsThis signal is then transmitted to other neurons or nerves



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook


Artificial neurons: Artificial neurons are based on digitalsystems (computers) rather than analogue systems(dentries),

Receives a number of inputs (from other neurons or theprogram itself)

Each input has a weight

Each neuron has a activation threshold



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Solving Problems with Neural Networks

Problems not suited to neural networks

Deterministic problems

Programs that can be written with a flowchart

Where the logic of the program is likely to change

Where you must know how the solution was derived



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Solving Problems with Neural Networks

Problems suited to neural networks

Problems that cant be solved as a series of steps

Pattern recognition

Classification



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

The Neuron

The basic building block of the neural network.

Individual neurons are connected to one another

Each connection is assigned a weight.

These connection weights determine the output of theneural network



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

The Neuron

Input/Output

Receives input from other neurons or the users programSends output to other neurons or the users program

A neuron fires or actives when the sum of its inputs

f (x) = K

(i

wigi (x)

)is high enough. We may use one of many activation functions,like

y = 1

Tanh:

tanh(u) =eu eu

eu + eu

Sigmoid:

y =1

1 + ex



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Neural Network Structure

Split into two parts, neurons and layers

Neurons - basic element. Interconnected, with eachconnection having a weight.

Layers - groups of neurons.



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Neuron Connection Weights

Neurons are connected together by weighted connections

These weights allow the neural network to recognizepatterns

If you adjust the weights, the neural network will recognizea different pattern.

Training a neural network is merely adjusting the weightsbetween the neurons until we get the desired output



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Layers of Neurons

Neurons are commonly grouped in layers

Layers - groups of neurons that perform similar functions

Three types1 Input layer - receives input from the user2 Output layer - sends data to user3 Hidden layer - neurons connected only to other neurons



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Training

Remember: neurons are connected via weightedconnections, these weights determine the output of thenetwork

Training methodology:1 Assign random numbers to weights2 Determine validity of neural network (see next slides)3 Adjust weights according to validation results



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Training Methods

Supervised Training

Most common form of neural network trainingGive the neural net a set of sample data with anticipatedoutputs for each sample.Progresses through several iterations (epochs) until theactual neural network matches the anticipated outputwithin error.

Unsupervised Training

Give the neural network a set of sample data withoutanticipated outputsUsed when the neural network needs to classify the inputsinto several groups

Hybrid Approaches



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Validation

We must check that training has gone correctly!

We determine whether we need further training

Validation data is separate from training data1 Use half the data to train the network2 Use other half to make sure neural networks weights

correspond to correct solutions.



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Feed-Forward Networks

A feed-forward neural network.

A feed-forward neuralnetwork is one whereconnections betweenneurons do not form adirected cycle.



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook


1 Background


3 Motivation

4 Problem


6 Outlook



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Problem Motivation

Computers are now very goodat recognizing printed text likethis: But very bad at recognizing

text like this:



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Problem Motivation

The former is called Optical Character Recognition and thelatter is called Intelligent Character Recognition.

Optical Character Recognition is a very well-studiedproblem (see: Google Library Project), and error rates areapproaching 1 in several hundred.

Intelligent character recognition is not very well studied

Almost no one has done, properly, handwriting to LATEX.No one has done it with a Neural Network yet.



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook


1 Background


3 Motivation

4 Problem


6 Outlook



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Problem Definition

Transform handwritten equations into LATEX form using neuralnetworks.



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook


1 Background


3 Motivation

4 Problem


6 Outlook



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Three Steps

There are three major subproblems inherent in transforminghandwritten equations into LATEX.

1 Single-character recognition

2 Multiple character recognition

3 Positional recognition (i.e. fractions, exponents)



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Single-Character Recognition

This can be broken down into several steps:

1 Find bounds of user-drawn letter

2 Downsample letter

3 Feed into Kohonen Neural Network



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Finding Bounds

A very easy task!

Iterate through each direction to see if there is a pixel onthat line, if not, keep going

The final result looks like (graphically)



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Downsampling

We downsample the image to a 5 7 gridGraphically looks like:



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Kohonen Neural Network

Single-layer feed-forward network where output neuronsare arranged in 2D grid.

Each input is connected to all output neurons.

With every neuron there is a weight vector with the samedimensionality as input vectors.



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

The NeuralOCR Application

Screenshot:



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Recognizing Multiple Characters

We can do better -multiple characterrecognition!

Multiple bounding boxes -one for each letter

Each character getsrecognized separatelyby the neural networkThis should be a validassumption for math -people dont writemath in cursive.



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Recognizing Math!!!

We can do better still! - multiple character

recognition!

Multiple bounding boxes - one for each

character

Store position data for each

character after you recognize it

Use a second neural network to

recognize position to determine the

difference between:

Normal math

Subscript

Superscript

Fractions (more on this

later)



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Fractions are Hard

The hangup is recognizingfractions with this method



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook


1 Background


3 Motivation

4 Problem


6 Outlook



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Outlook

Future work:

Multiple-character recognition (not that hard!)

Positional recognition



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

References

All images from Wikipedia (thanks guys!)



Background

NeuralNetworks



Motivation

Problem

Solution



RecognizingMath

Outlook

Questions?

Quotes (upon hearing research topic)

I will personally pay you lots and lots of money if you getthis working. - Prof. Yong

Dude ... why are you setting yourself up to fail likethat? - Anonymous Former Neural Nets Student

. . . hahahahahahaha - Anonymous Neural NetsResearcher

Questions?

BackgroundNeural NetworksNeural Network StructureTraining Neural Networks

MotivationProblemSolutionSingle-Character RecognitionMultiple Character RecognitionRecognizing Math

Outlook

neural networks and handwriting recognitiondyong/math164/2007/sloss/presentation.pdf · neural...

Documents

J Handwriting

Impact of Handwriting Training on Handwriting Fluency ... · Impact of Handwriting Training on Handwriting Fluency, Spelling and Text Quality in Primary School Children ... Visual-motorIntegration

순환신경망을 이용한 한글 필기체 인식 · 2017. 5. 29. · 순환신경망을 이용한 한글 필기체 인식 Hangul Handwriting Recognition using Recurrent Neural

Caption Contest Sponsored by Dr. Lesley Sloss Award Bottle of Whiskey

Handwriting Policy 2013 · Heavers’Farm’Primary’School’Handwriting’Policy’JAN’2013’ 1 ... Handwriting ’should’be’ ... The’adultshould’model’clearly’using’the

Reserve design and the SLOSS debate

Handwriting Development - Warwick Hospital€¦ · HANDWRITING DEVELOPMENT . 2 South Warwickshire Foundation Trust ... Handwriting grip & Arm position 25 -26 Handwriting Tips for

Dr L L Sloss Principal Environmental Consultant

Jimbo Project-Handwriting Recognition Using an Arificial Neural Network

Handwriting Skill

Improving handwriting without teaching handwriting: The

A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting

Professional Handwriting

Year 1 Autumn 2 Week 1 W.B. 2.11 · 2020. 11. 4. · Week 1 W.B. 2.11.20. Handwriting . Handwriting. Handwriting. Handwriting. Handwriting. Handwriting. Handwriting. English . Listen

Handwriting Analysis

Graham Handwriting

1992-8645 NEURAL NETWORK MODEL OF ARTIFICIAL … · Keywords: Neural Network Model, Handwriting Recognition, Feature Extraction, Classification, Artificial Intelligence, Writer Verification

Gender and Handedness Prediction from Offline Handwriting ...downloads.hindawi.com/journals/complexity/aip/3891624.pdf · Handwriting using Convolutional Neural Networks Ángel Morera1,

1922 - Handwriting

Lecture 23: Artificial neural networks - pha.jhu.eduneufeld/numerical/lecturenotes23.pdf• Confluence of statistical mechanics, applied math, ... handwriting, underwriting) ... From

Handwriting Recognition Using Neural Networks

Artificial Neural Networks: Self Organizing Networks€¦ · Artificial neural networks which are currently used in tasks such as speech and handwriting recognition are based on learning

Handwriting VB

On the Benefits of Convolutional Neural Network …...On the Benefits of Convolutional Neural Network Combinations in Offline Handwriting Recognition Dewi Suryaniy, Patrick Doetsch

Angel Sloss Makerspaces

Sloss Fall 2009, Criminal Law Outline

Ofﬂine Arabic Handwriting Recognition with ...graves/arabic_ocr_chapter.pdf · Ofﬂine Arabic Handwriting Recognition with Multidimensional Recurrent Neural Networks Alex Graves

Margaret Sloss Women’s Center Newsletter · Feminist Food For Thought: Halloween Costumes The Margaret Sloss Women's Center promotes equity and social change on the Iowa State University

Ofﬂine Arabic Handwriting Recognition with ...graves/arabic_ocr_chapter.pdf · Ofﬂine Arabic Handwriting Recognition with Multidimensional Recurrent Neural ... ease to, for example,

A Comparison of Sequence-Trained Deep Neural Networks … A Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling for Handwriting Recognition

Neural networks and Chinese character recognition...Deep learning examples Neural networks Handwriting recognition Signatures Signatures The mth level of the signature of a path in

A Neural Network Approach to Handwriting Recognition in ...thesis.honors.olemiss.edu/692/1/ECA_thesis.pdf · Neural networks are biologically-inspired con gurations of "neurons" which

Offline Handwritten Character Recognition using Neural Networkshabdbooks.com/gallery/395-april2020as.pdf · Neural Networks, Geometric Feature Extraction, Segmentation, Handwriting

Neural Networks in the Wild: Handwriting Recognition

Sloss Furnaces Alabama