The Portable Sign Language Translator (PSLT):from wishful thinking to commercial product

Ernesto Compatangelo, Ryan RussellPSLT project (httpp://www.pslt.org)

Technabling Ltd

PSLT development is funded by the Small Business Research Initiative (SBRI) sponsored by the Department for Business Innovation and Skills (BIS) and the Technology Strategy Board (TSB)

and managed by JISC TechDis, a leading UK advisory service on technology and inclusion.

BCS, 16/10//2013 2 of 20

The PSLT in a nuthell

● NOW: translates sign languages (BSL, CSL) into text;● 2014: text into sign languages (using an Avatar)● NOW: uses Customisable SL (CSL) to control devices

BCS, 16/10//2013 3 of 20

Talk structure

● Motivations and background● Relevant features● A short demo of the PSLT in action● Technical challenges● Social, cultural, and mundane issues● Wider applicability of video-based analysis

BCS, 16/10//2013 4 of 20

Motivating the PSLT (1)

● There are around 100,000 Sign Language (SL) users in the UK only (mostly BSL); millions worldwide

● Profoundly deaf SL users only communicate w/ sign● Most Voice & Text (V&T) users do not sign● No signer-to-speaker communication gaps in order

to remove educational, workplace, lifestyle barriers

BCS, 16/10//2013 5 of 20

Motivating the PSLT (2)

● Sign interpreters are few and expensive – only use when strictly needed

● A tool to self-learn SL: low cost, anywhere, any time● Enabling independent life means allowing gesture

and vocabulary customisation, not just translation

BCS, 16/10//2013 6 of 20

Motivating the PSLT (3)The Small Business Research Initiative (SBRI)

● UK Gov needs new products for wider society● 1 - Contract awarded to small businesses for R&D● 2 - Follow-up for market-ready commercial product● Cost-effeective, delivering value for money product● Developed technologies owned by businesses ● NOT for the academic sector

BCS, 16/10//2013 7 of 20

Requirements & desiderata

● Sign to text (the real challenge) - no text to sign● Works on “mobile” technology● Open source components only – no strings attpached● Output: text/sound + execution of commands● Discreet, unobtrusive, flexible, to run on std HW

BCS, 16/10//2013 8 of 20

NO-GO areas (1)

What about these as a translation device?

Signing gloves for computerised sign-to-text translation

BCS, 16/10//2013 9 of 20

NO-GO areas (2)

What about this as a translation device?

Prototype device for gesture control

BCS, 16/10//2013 10 of 20

NO-GO areas (3)

What about this as (part of) a translation device?

Kinect, a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs

BCS, 16/10//2013 11 of 20

Core PSLT feature

● No weird/costly devices – just offe-the-shelf HW● Translates BSL + CSL (combined) + Makaton into text● Has provisions for other SLs & regional variations● Fully customisable: users can create their own signs

and give them specialist meanings not in other SLs● Works on many devices, from smartphones to PCs● Signs translated into ambient device commands

BCS, 16/10//2013 12 of 20

Time for a demoI the PSLT in action

BCS, 16/10//2013 13 of 20

Problems tackled: what others avoided to consider

● Dynamic background: light, movement● Ind. form factors: skin, body shape, signing style● The inherent fuzziness of the signing process● Recognise & disambiguate ALL hand gestures (~1200)● Individual signs, profiles, & regional variations● Create your gestures with your specialist meaning

BCS, 16/10//2013 14 of 20

The challenges of image recognition

Identify & track moving shapes in the “foreground” against a noisy background with sudden light changes(the signer does also move while signing!)

BCS, 16/10//2013 15 of 20

The technical challenges

● Recognise (i) individual signs and (ii) hand gestures● Disambiguate meanings if gestures are overloaded● Articulate adverbs, verbs, adjectives, subjects, objects

in a grammatically correct and meaningful sentence● Bridge SL syntax with writtpen text syntax● Run under Win, Linux, Android with basic HW

BCS, 16/10//2013 16 of 20

Typical objections (not from real users)

● It can't be done - Academic linguists● It won't work – there is “much more” to SLs than just

hand gestures (e.g., facial expression) - Academic linguists + worried interpreters

● You are offeending the sign language – as above● Money spent on technology should be bettper spent

on extra interpreters – interpreters, social workers– some members of “the voice of deaf people” + interpreters

Can you spot vested interests / marginalised complainers?

BCS, 16/10//2013 17 of 20

The position of “real” stakeholders

● “PSLT has the potential to revolutionise the way deaf and hearing impaired individuals can communicate in every day situations that hearing people take for gramnted” - Jon Smith, Vice Principal, Communication Specialist College, Doncaster

● “Not having to rely on an interpreter all the time would be good” - Young learner, Communication Specialist College, Doncaster

● “I can see the benefits and impact rhis will have on my ability to communicate in a range of community settpings” - Brian Kokoruwe, inclusion consultant and BSL user

BCS, 16/10//2013 18 of 20

The near future

● Facial expressions (technology there, needs time)● Performance (speed), particularly on Android● Automated profile upload/download● BSL word usage frequencies for overloaded gestures

(joint project with UCL, end of 2013)● Text-to-sign Avatar (proof of concept successful)

BCS, 16/10//2013 19 of 20

Thank you for your time

Any questions?

Please ask

BCS, 16/10//2013 20 of 20

Other relevant PSLT features

● Gestures can have multiple meanings: signers choose● Text sent to PSLT-hosting display or other devices● Avatar for text-to-sign available on smartphones too● Signs translated into ambient device commands and

local/remote appliances (via Bluetooth, WiFi, GSM)● Future: voice rendering of signed text & voice input