sign language translator ieee power point

Dept of Biomedical Engineering

AUTOMATIC LANGUAGE TRANSLATION SOFTWARE FOR AIDING COMMUNICATION BETWEEN INDIAN

SIGN LANGUAGE AND SPOKEN ENGLISH USING LABVIEW

By

YELLAPU MADHURI,

Reg.No.1651110002,

MTECH II YEAR,

SRM University.

Guided by Ms.G.ANITHA

Assistant professor (O.G) /BME

Dept of Biomedical Engineering

INTRODUCTION

SIGN LANGUAGE (SL)

Natural way of communication of speech and/or hearing-impaired people.

SIGN

Movement of one or both hands, accompanied with facial expression, which

corresponds to a specific meaning.

TRANSLATOR

Communication between speech and/or sound impaired person and person that do

not understand sign language, avoiding by this way the intervention of an

intermediate person. And allow communication using their natural way of speaking.

Dept of Biomedical Engineering

ANATOMY OF HUMAN EAR

Dept of Biomedical Engineering

EVENTS INVOLVED IN HEARING

Dept of Biomedical Engineering

SPEECH CHAIN

Dept of Biomedical Engineering

AIM

To develop a mobile interactive application software for automatic translation of

Indian sign language into speech in English and vice-versa to assist the

communication between speech and/or hearing impaired people with normal

people. This language translator should be able to translate one handed finger

spelling input of Indian Sign language alphabets A-Z and numbers 1-9 into spoken

English audio output and 165 spoken English words input to Indian Sign language

picture display output.

Dept of Biomedical Engineering

GRAPHICAL ABSTRACT

Dept of Biomedical Engineering

OBJECTIVE

For Sign to Speech conversion

1. Acquire images using the inbuilt camera of the device.

2. Perform vision analysis functions in the operating system and provide speech output

through the inbuilt audio device.

For Speech to Sign conversion

1. Acquire speech input using the inbuilt microphone of the device.

2. Perform speech analysis functions in the operating system and provide visual sign

output through the inbuilt display device.

Minimize hardware requirements and expense.

Dept of Biomedical Engineering

LITERATURE REVIEW

1. Jose l. Hernandez-rebollar et al [2004]

Discusses a novel approach for capturing and translating isolated gestures of ASL into spoken and written words using combined acceleglove and a two-link arm skeleton.

2. Paschaloudi N. Vassilia et al [may 2006]

Extensible system to recognize GSL modules for signs or finger-spelled words, using isolation or combined neural networks

3. Beifang yi [ may 2006] Explorations in the areas of computer graphics, interface design, and human-computer interactions with emphasis on software development and implementation in ASLT

4. Andreas domingo et al [2007]

ASLT using pattern-matching algorithm.

5. Rini akmeliawatil et al [may 2007]

ASLT for real-time english translation of the malaysia SL using neural networks.

6. Abang irfan halil et al [ 2007]

Extent of development details on recognition system by using state-of-the-art graphical programming software

Dept of Biomedical Engineering

ALGORITHM CRITERION

1. REAL-TIME

2. VISION-BASED

3. AUTOMATIC AND CONTINUOUS OPERATION

4. EFFICIENT TRANSLATION

Dept of Biomedical Engineering

MATERIALS

Software Tools used: National Instruments LabVIEW and toolkits

LABVIEW 2012 version Vision Development Module Vision acquisition Module

Hardware tools used:

Laptop inbuilt webcamera- Acer Crystal Eye Laptop inbuilt speaker-Acer eAudio

Dept of Biomedical Engineering

GUI OF SOFTWARE

Dept of Biomedical Engineering

PAGE 2- SPEECH TO SIGN LANGUAGE TRANSLATOR

Dept of Biomedical Engineering

BLOCK DIAGRAM OF SPEECH TO SIGN LANGUAGE TRANSLATOR

Dept of Biomedical Engineering

FLOW CHART OF SPEECH TO SIGN LANGUAGE TRANSLATION

Dept of Biomedical Engineering

WINDOWS SPEECH RECOGNITION TUTORIAL

Dept of Biomedical Engineering

WINDOWS SPEECH RECOGNITION SOFTWARE GUI

Dept of Biomedical Engineering

USER INTRFACE OF SPEECH TO SIGN LANGUAGE TRANSLATOR

Dept of Biomedical Engineering

PAGE 3- TEMPLATE PREPARATION

Dept of Biomedical Engineering

IMAGE ACQUISITION SEQUENCE OF FRAMES

Dept of Biomedical Engineering

USER INTERFACE OF TEMPLATE PREPARATION FOR SIGN LANGUAGE TO ENGLISH TRANSLATION

Dept of Biomedical Engineering

FLOW CHART OF TEMPLATE PREPARATION FOR SIGN LANGUAGE TO ENGLISH TRANSLATION

Dept of Biomedical Engineering

PAGE 4- PATTERN MATCHING

Dept of Biomedical Engineering

USER INTERFACE OF PATTERN MATCHING FOR SIGN LANGUAGE TO ENGLISH TRANSLATION

Dept of Biomedical Engineering

BLOCK DIAGRAM OF SIGN LANGUAGE TO SPEECH TRANSLATOR

Dept of Biomedical Engineering

DATABASE OF ONE HANDED ALPHABETS AND NUMBERS OF SIGN LANGUAGE

Dept of Biomedical Engineering

ADVANTAGES

Eliminates the need for an interpreter for communication between sign language and speech language.

Easy to incorporate and execute in any supporting operating system. Real time translation. Does not require any additional hardware.

Dept of Biomedical Engineering

FUTURE APPLICATIONS

Web conferenceCOMPUTER AND VIDEO GAMESPRECISION SURGERYDOMESTIC APPLICATIONSWEARABLE COMPUTERS

Dept of Biomedical Engineering

CHALLENGES

Background subtraction for robust usage.Making the system user independent.Pattern matching training.

Dept of Biomedical Engineering

LIMITATIONS

System is trained on a limited database..

Possibility of misinterpretation for closely related gestures.

Translates only static signs.

Not trained to translate dynamic signs.

Facial expressions are not considered.

Possibility of misinterpretation for words of similar pronunciation.

Dept of Biomedical Engineering

CONCLUSION

The feature vectors which include whole image frames containing all the aspects of

the sign are considered.

The geometric features which are extracted from the signers’ dominant hand, improve

the accuracy of the system to a great degree.

Training the speech recognition for shorter phrases is difficult than longer phrases.

Dept of Biomedical Engineering

FUTURE WORK

To increase the performance and accuracy of the ASLT, the quality of the training database used should be enhanced to ensure that the ASLT picks up correct and significant characteristics in each individual sign and further improve the performance more efficiently.

Current collaboration with Assistive Technology researchers and members of the Deaf community for continued design work should be considered for continued progress.

This project did not focus on facial expressions although it is well known that facial expressions convey important part of sign-languages.

This system can be implemented in many application areas examples include accessing government websites whereby no video clip for deaf and mute is available or filling out forms online whereby no interpreter may be present to help.

Dept of Biomedical Engineering

REFERENCES Andreas Domingo, Rini Akmeliawati, Kuang Ye Chow ‘Pattern Matching for Automatic

Sign Language Translation System using LabVIEW’, International Conference on

Intelligent and Advanced Systems 2007.

Beifang Yi Dr. Frederick C. Harris ‘A Framework for a Sign Language Interfacing

System’, A dissertation submitted in partial fulllment of the requirements for the degree

of Doctor of Philosophy in Computer Science and Engineering May 2006 University of

Nevada, Reno.

Helene Brashear & Thad Starner ‘Using Multiple Sensors for Mobile Sign Language

Recognition’, ETH - Swiss Federal Institute of Technology Wearable Computing

Laboratory 8092 Zurich, Switzerland flukowicz, junker g@ife.ee.ethz.ch

Dept of Biomedical Engineering

Jose L. Hernandez-Rebollar1, Nicholas Kyriakopoulos1, Robert W. Lindeman2 ‘A

New Instrumented Approach For Translating American Sign Language Into Sound

And Text’, Proceedings of the Sixth IEEE International Conference on Automatic

Face and Gesture Recognition (FGR’04) 0-7695-2122-3/04 $ 20.00 © 2004 IEEE.

K. Abe, H. Saito, S. Ozawa: Virtual 3D Interface System via Hand Motion

Recognition From Two Cameras. IEEE Trans. Systems, Man, and Cybernetics, Vol.

32, No. 4, pp. 536–540, July 2002.

Dept of Biomedical Engineering

THANK YOU