lip reading computer

LIP READING COMPUTER

COMPILED BY

RUTUJA SHAH

TE IT

AGENDA

Introduction of the System. System Construction. Working of the System. Areas of Application. List of software distributors.

INTRODUCTION

What ability do humans already have? What is exactly meant by “LIP

READING”? Computer vision is been already used for

lip reading or “feature extraction”. Sequences of lip shapes and lip motions

vary strongly with the language we speak.

INTRODUCTION A computer developed at the

University of East Anglia in Norwich in UK.

Identifies the language spoken by an individual

Has very high accuracy. Very first time a computer has been

actually “taught” to detect different languages.

SYSTEM CONSTRUCTION The proposed system is constructed

on a commercial base personal computer (PC).

Video capture card (Video Camera) is equipped on the PC to get lip movement in real time.

SYSTEM CONSTRUCTION

Size of each image is 640 * 480 pixels.

To achieve fast capturing operation, the captured images are directly stored in the memory by using DMA (Direct Memory Access) technique.

WORKING OF THE SYSTEM

SEQUENCE :

IMAGE FROM VCC

SACM

HMM


System

Flowchart is:


Fast and exact lip shape extraction is indispensable for developing the real time lip reading system.

The modified SACM (Sampled Active Contour Model) is a kind of the active contour models proposed by M. KASS.

WORKING OF SACM…contd.

JOB OF SACM :

Input from VCC. Extract lip shapes from series of face

images. Facilitate fast and exact extraction.

WORKING OF SACM…..contd.

LIP SHAPE WHILE UTTERING : “a”

OUTER LIP SHAPE INNER LIP SHAPE


LIP SHAPE WHILE UTTERING “i”



LIP SHAPE WHILE UTTERING “O”


WORKING OF SACM…contd. As shown in this

figure, the modified SACM is a closed curve and some contour points are located on it.

On each contour point, 4 forces operate.

WORKING OF SACM….contd.

Location of these forces


EXTRACTION OF LIP SHAPE USING SACM


Recognition parameters : From coordinates of the contour points on

the converged modified SACM, parameters for recognition are obtained in the following manners.

Obtain the center of the converged modified SACM. Draw lines from the center every 30 degrees.

Calculate intersection points of these lines and the converged modified SACM.

Obtain distances from center to the intersection points.

Calculate deviations of these distances from previous image.

WORKING OF SACM….contd.

The 4 forces act on every contour point are essential for image edge extraction.

Thus, these image edge extraction are used as parameters for different lip shapes which is stored in database of lip reading computer.

HMM FOR RECOGNITION

JOB OF HMM : The new lip reading parameters are

investigated and are recognized by HMM (Hidden Markov Model).

Our preference here is the one adopted by Ferguson and Rabiner et al.

OVERVIEW OF HMM….contd.

HMM is finite state machine with some transition rules and states.

A hidden Markov model is defined as a pair of stochastic processes (X,Y).

The X process is a first order Markov chain, and is not directly observable.

While the Y process is a sequence of random variables taking values of observations.

OVERVIEW OF HMM…contd.Notation : aij represents the probability of state transition (probability of being

in state Sj given state Si ) aij = P(qt+1=Sj / qt=Si) ..................(1) bj(wk) is the wk symbol probability distribution in a state Sj w is the alphabet and k is the number of symbols in this

alphabet. pie= {1 0 0 0 0 } is the initial state probability distribution.

The model is completely defined by these three sets of parameters a, b, and p and the model of N states and M observations can be referred to by :

l = (A , B , pie ) ..................(2) where A = {aij}, B = {bj(wk)} 1 < i , j <=N and 1< k < M. It is called Left-Right HMM as derived from its way of behaviour and its topology (moving from left to right during state transition). The reason for using the L-R topology of HMM is due to its inherent

structure which can model the temporal flow of speech signal over time.

OVERVIEW OF HMM …contd.

ASSOCIATING HMM WITH SPEECH : Understanding speech formation.

different positions of lips consequently producing the stream of

sounds that form the speech signal. each articulatory position could be

represented by a state of different and varying duration.

OVERVIEW OF HMM…contd.

Accordingly, the transition between different articulatory positions (states) can be represented by A = {aij}.

The observations in this case are the sounds produced in each position and due to the variations in the evolution of each sound

This can be also represented by a probabilistic function B = {bj(wk)}.

OVERVIEW OF HMM…contd.

These HMMs are trained by using the Baum-Welch algorithm.

Baum-welch algorithm uses forward-backward algorithm.

The algorithm first computes a set of forward probabilities .

The algorithm also computes a set of backward probabilities.

The forward-backward algorithm can thus be used to find the most likely state for a hidden Markov model at any time.

EXPERIMENT ANALYSIS Much of the error can be

attributed to the presence of plosives in the beginning and end of some of the words .

E.g.India and Zambia are similar sounding ,have same vowel part but differs only in their unvoiced beginning..hence it recorded a low 65% accuracy.

Words like Spain and Mexico being different from others recorded very high percentage

WORD SPEAKER1 SPEAKER2 SPEAKER3India 60% 64% 64.50%Spain 97% 99.20% 98.30%Germany 95.50% 91% 92%Zambia 65% 70% 61.40%Mexico 98% 99% 97.23%

AREAS OF APPLICATION

For deaf people, in Military units serving in overseas.

Police wanting to know what people in CCTV footage are saying.

Video surveillance systems. Recognize which language your

teenager speaks.

LIST OF DISTRIBUTORS

1) Learn to Lipread: an Introductory course (2000) (Dr Mary Allen) Australia

Web: www.lipread.com.au2) Conversation Made Easy (2002) (Dr Nancy Tye-

Murray)USA Web:

www.asha.org/about/publications/leader-online/reviews/conversation.htm

3) Lipreader (2004) (David Smith Software) UK Web: www.lipreader.co.uk

THANK YOU !

END

lip reading computer

Documents