Lip Feature Extraction Using Red Exclusion

Trent W. Lewis and David M.W. Powers

Flinders University of SA

VIP2000

01/12/2000 Lip Feature Extraction Using Red Exlucsion

2

Overview

• Context

• Lip Feature Extraction– Related Work (greyscale, horizontal edges, red and hue colour spaces)

– Red Exclusion

• AVSR: Results and Issues

• Summary

01/12/2000 Lip Feature Extraction Using Red Exlucsion

3

Context

• Audio Speech Recognition (ASR)

• Psycholinguistic Research

• Audio Visual Speech Recognition (AVSR)

01/12/2000 Lip Feature Extraction Using Red Exlucsion

4

Context - ASR

• Up to 99% word accuracy

• However,– limited context– limited vocabulary– trained on individual– close microphone, cannot handle noise

01/12/2000 Lip Feature Extraction Using Red Exlucsion

5

Context - Psycholinguistic

• McGurk Effect– A[ba] + V[ga] [da]

• Viseme– visual phonemes– form complementary sets

• Demonstrates vision can assist the perception of speech AVSR

01/12/2000 Lip Feature Extraction Using Red Exlucsion

6

Context - AVSR

• Acoustic Features• Visual Features

– width

– height

– oral cavity

• Integration– Early

– Late

01/12/2000 Lip Feature Extraction Using Red Exlucsion

7

Lip Feature Extraction

• Pixel-Based Model

01/12/2000 Lip Feature Extraction Using Red Exlucsion

8

Lip Feature Extraction

• Pixel-Based Model– raw pixels or minimal processing– retain linguistically relevant data– large amounts of data, time– shift and lighting variant– normalisation and PCA

01/12/2000 Lip Feature Extraction Using Red Exlucsion

9

Lip Feature Extraction

• Pixel-Based Model– reduced input to set of hand-crafted features– width, height, average intensity, etc.– less features, time– model fitting, time– lose linguistically relevant features

01/12/2000 Lip Feature Extraction Using Red Exlucsion

10

Lip Feature Extraction

• Pixel-Based Model– feature extraction– Steps

• preprocess to enhance contrast

• locate mouth edges

• identify corners, height, and other key features

• train recogntion engine

Our Approach

01/12/2000 Lip Feature Extraction Using Red Exlucsion

11

Lip Feature Extraction

• Database

1 2 3

01/12/2000 Lip Feature Extraction Using Red Exlucsion

12

Lip Feature Extraction

• Preprocessing Techniques– Grey-scale– Horizontal Edges– Red Analysis– Hue, Saturation, and Value (HSV)– Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

13

Lip Feature Extraction

• Grey-scale– vertical position of mouth

• minimum row sum

– threshold minimum row• average of min and max of row

– search for above threshold pixels

01/12/2000 Lip Feature Extraction Using Red Exlucsion

14

Lip Feature Extraction

• Grey-scale

01/12/2000 Lip Feature Extraction Using Red Exlucsion

15

Lip Feature Extraction

• Grey-scale

01/12/2000 Lip Feature Extraction Using Red Exlucsion

16

Lip Feature Extraction

• Horizontal Edges– high horizontal edge content– 3x3, DY Prewitt operator

111

000

111

DY

01/12/2000 Lip Feature Extraction Using Red Exlucsion

17

Lip Feature Extraction

• Horizontal Edges

“Found” Corners Binary Image

01/12/2000 Lip Feature Extraction Using Red Exlucsion

18

Lip Feature Extraction

• Red Analysis– overcome bearded subjects– used for face location

limlim UG

RL

01/12/2000 Lip Feature Extraction Using Red Exlucsion

19

Lip Feature Extraction

• Red Analysis

01/12/2000 Lip Feature Extraction Using Red Exlucsion

20

Lip Feature Extraction

• HSV– disentangles illumination from colour Illumination > Hue

otherwise

whhw

hhhf o

o

,0

,)(

1)( 2

2

01/12/2000 Lip Feature Extraction Using Red Exlucsion

21

Lip Feature Extraction

• HSV

01/12/2000 Lip Feature Extraction Using Red Exlucsion

22

Lip Feature Extraction

• Red Exclusion– needed extraction method for AVSR– similar to Red Analysis

• face predominantly red

• variations occur in the blue and green colours

B

Glog

01/12/2000 Lip Feature Extraction Using Red Exlucsion

23

Lip Feature Extraction

• Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

24

Lip Feature Extraction

• Corners found using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

25

Lip Feature Extraction

• ComparisonAlgorithm

Subject 1 female

Subject 2 male bearded

Subject 3 male thin lips

Reliable Corners

Other Features

Grey-scale

Edge

Red Analysis

HSV Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

26

AVSR: Results and Issues

• Application for red exclusion

• Used in finding lip features– Width– Height– Key pixels

01/12/2000 Lip Feature Extraction Using Red Exlucsion

27

AVSR: Results and Issues

• Visual Speech RecognitionStatic (%) Dynamic (%)

Voicing 32.2 30.8

Viseme 54.7 51.3

Phoneme 14.7 13.6

01/12/2000 Lip Feature Extraction Using Red Exlucsion

28

AVSR: Results and Issues

• AVSR - IntegrationEarly

Static

Early

Dynamic

Late

Voice/Vis

Late Error

Voice/Vis

Phoneme20.1 18.0 29.0 19.5

01/12/2000 Lip Feature Extraction Using Red Exlucsion

29

Summary

• Vision can help ASR– AVSR

• Needed good extraction technique– Red Exclusion

• AVSR is difficult when both signals degraded

01/12/2000 Lip Feature Extraction Using Red Exlucsion

30

Questions?

Trent W. Lewis

BSc (Cognitive Science)

Flinders University

lewi0146@ist.flinders.edu.au