hear me sign: sign language translator - development and test...title [specialty track (specialty...

Copyright 2019 Accenture. All rights reserved.

Hear me sign:Sign language translator -development and testDaria LashkevichAccenture. Riga, Latvia

TAPOST 2019


How you can understand that I need

your urgent help if you don’t know

British sign language?


BusinessHear me sign

BY 2050

900M HEARING LOSS

AND 70M MUTE

PEOPLE

130 DIFFERENT

SIGN LANGUAGES

A LOT OF TIME

LOST


Sign language Basic sentence structure

Source https://www.youtube.com/watch?v=glJvYftJZM4

https://www.youtube.com/watch?v=glJvYftJZM4


TaskScope

Create AI application

which can recognize

British Sign Language

alphabet

3 AI Engineers 4 weeks


SOLUTION ArchitectureProcess flow

frontend

video

stream

each frame is

analysed by

skeleton extractor

JSON pose

coordinates

LSTM

module 1

• Signs recognition

• Translation

textJSON

text

text to

speechvoice

backend

AI Pose

recognition


Landscape reviewPossible solutions

IOT GLOVES FOR HAND

TRACKING

CAMERA ONLYIMAGE + DEPTH (E.G.

LEAP MOTION)

https://www.youtube.com/watch?v=oZ_53T2jBGg https://github.com/CMU-Perceptual-Computing-Lab/openposehttps://cdn-blog.adafruit.com/uploads/2016/05/3059616-poster-p-2-these-

students-built-a-glove-that-translates-sign-language-into-english.jpg


Person recognitionBody, face and face key points


Desktop prototype Hear me sign


SolutionHear me signHow to address the problem?

A mobile application translating between sign

language / audio:


SolutionHear me sign


Creating dataset: first problems

• Learning BSL

• Translations problems

• Uncountable quantity

of cases to show sign



• Left and right handed signs



• Gender differences for coordinate defining

• Not all needed coordinates can be defined


Creating dataset: solutions

• Letters separately (Alphabet first)

• Based on pictures first, after on videos

• Equal quantity of men and women for dataset

• The highest camera quality

• Dataset filtration



• All different cases also should be shown in data set



• All negative cases also should be shown in data set


Model training – how to test

• Different backgrounds

• Different cameras (desktop and mobile)

• Different people

• (doesn’t matter men or women)

• Testing with British sign language natives


Software/Hardware requirements

Not so easy to test such heavy components:

• GPU NVIDIA driver

• Caffe framework

• Big capacity for video implementation

• Client – server Application


Solution integration: testing

• Testing integration: client – server

• More then one person in the camera –

• who should be recognized

• Unknown signs

• Connection delays

• Fast signs show

• Hard outdoor conditions: train, rain, fog


Recognition

• The second place winner in Global Technology Innovation

Contest 2018 Europe ATC

• The conference SQA-Days EU-1 2019 invited participant


Demo indoor


Demo outdoor


What next?

mobile only wearables + mobileSign language avatar

(inversed translator)


Sign language avatar(inversed translator) - ISL


CONTACTS

Accenture RigaLiquid Studio

DARIA LASHKEVICH

Riga Liquid Studio

Innovation Test Team Lead

[email protected]

+371 23202147

Visit us @ Latvia.Accenture.lv

mailto:[email protected]

hear me sign: sign language translator - development and test...title [specialty track (specialty...

Documents