bridging the language divide

67
Waibel, A. - Bridging the Language Divide Bridging the Language Divide Alex Waibel and the InterACT Team Carnegie Mellon University Karlsruhe Institute of Technology [email protected] [email protected] [email protected]

Upload: vukien

Post on 22-Dec-2016

235 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Bridging the Language Divide

Waibel, A. - Bridging the Language Divide

Bridging the Language Divide

Alex Waibel and the InterACT Team

Carnegie Mellon University

Karlsruhe Institute of Technology

[email protected]

[email protected]

[email protected]

Page 2: Bridging the Language Divide
Page 3: Bridging the Language Divide

Waibel, A. - Bridging the Language Divide

“Everyone Speaks English”…

???

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

English language knowledge (not mother tongue)

In Europe:

Page 4: Bridging the Language Divide

Human Effort

Page 5: Bridging the Language Divide

Waibel, A. - Bridging the Language Divide

• German is the most widely-spoken first language in the

EU (~100 million speakers)

• Most Germans speak at least two languages (English,

French, and Russian are most common)

• Recognized minority languages:

– Danish

– Plattdeutsch

– Sorbian

– Romany

– Frisian

Languages in Germany

5

Page 6: Bridging the Language Divide

Waibel, A. - Bridging the Language Divide

• Germany has one official language (German)

• Real life is something else:

– Immigration

– Tourism

– Trade and commerce

– Regional development, governance, and cooperation

• Mobility and traffic

• Energy and climate change

• Environment and natural resources

– Cross-border legal issues (e.g. marriage, birth, contracts)

Isn’t Germany monolingual?

6

Page 7: Bridging the Language Divide

Waibel, A. - Bridging the Language Divide

Neighboring languages

7

Polish

CzechFrench

Dutch

Danish

Page 8: Bridging the Language Divide

Refugee Crisis 2015

Page 9: Bridging the Language Divide

Refugee Crisis 2015

Germany is the second-most popular immigration

destination (after the US)20% of residents in Germany have some roots outside Germany

6.4 million come from outside the EU

Page 10: Bridging the Language Divide

New Challenges

Page 11: Bridging the Language Divide

New ChallengesMajor immigrant languages

have included:Turkish (>2 million speakers)

Kurdish

Polish

Balkan languages

Russian

….

Page 12: Bridging the Language Divide

New ChallengesMajor immigrant languages

have included:Turkish (>2 million speakers)

Kurdish

Polish

Balkan languages

Russian

….

Page 13: Bridging the Language Divide

Refugee Registration

Page 14: Bridging the Language Divide

Communication

Effective Communication is not only Text,

But:

– Speech

– Images

– Ill-formed Text

“lol-jah I want hr to be like dat…”, Hppyyyy BD, CU, LMK

….what is

he saying?

你们的评估准则是什么

Page 15: Bridging the Language Divide

The daunting challenge requires

innovative solutions

Page 16: Bridging the Language Divide

An Interpreting Machine

To Build a Language Communicator

– 6 Component-Engines: Automatic Speech Recognition, Machine

Translation, and Text-to-Speech Synthesis

– Each is in Principle Language Independent,

but Requires Language Dependent Models

– Models are Automatically Trained but Require Large Corpora

– Certain Language Dependent Challenges still Persist

Page 17: Bridging the Language Divide

First Speech Translation VideoCall ‘91-92

• 1992 – C-STAR Consortium for Speech Translation Advanced Research

• 1993 – Public C-STAR Demo, ATR-CMU-UKA-Siemens

Page 18: Bridging the Language Divide

First Feasibility Demo

• 1991 – First Public Demonstration of Speech

July 27, 1991 – UKA, CMU, ATR

Page 19: Bridging the Language Divide

Mobile Consecutive Interpretation

Technologies for Cross-Lingual Dialog

Page 20: Bridging the Language Divide

2009

Page 21: Bridging the Language Divide
Page 22: Bridging the Language Divide
Page 23: Bridging the Language Divide

Jibbigo on Apple Commercials

Page 24: Bridging the Language Divide

Humanitarian Deployment

Page 25: Bridging the Language Divide
Page 26: Bridging the Language Divide

Cobra Gold’11

Thailand

Page 27: Bridging the Language Divide
Page 28: Bridging the Language Divide
Page 29: Bridging the Language Divide

Cambodia

Page 30: Bridging the Language Divide
Page 31: Bridging the Language Divide

San Jose , Honduras

Page 32: Bridging the Language Divide
Page 33: Bridging the Language Divide

Simultaneous Interpretation

Domain Unlimited Translation

of Monolingual Monologues

Page 34: Bridging the Language Divide

Domain Unlimited

Domain Unlimited Translators for:

– TV/Radio Broadcast Translation

– Translation of Lectures and Speeches

– Parliamentary Speeches (UN, EU,..)

– Telephone Conversations

– Meeting Translation

你们的评估准则是什么

Page 35: Bridging the Language Divide

End-to-End Speech Translation

Page 36: Bridging the Language Divide

www.eu-bridge.eu27.10.2015

Text für Fußzeile

Alex Waibel / EU-BRIDGE Overview

The work leading to these results has received funding from the European Union under grant agreement n° 287658

EU-BRIDGE –

Bridges across the Language Divide

Page 37: Bridging the Language Divide

www.eu-bridge.eu27.10.2015

Text für Fußzeile

Alex Waibel / EU-BRIDGE Overview

EU-BRIDGE Partners

Page 38: Bridging the Language Divide

www.eu-bridge.eu27.10.2015

Text für Fußzeile

Alex Waibel / EU-BRIDGE Overview

ASR

MT

Use Case 2

Engines Services Use Cases

Language Service

Customization,

Adaptation

Develop and InsertImproved Technology

Language Services

for User and Developer

Communities

Page 39: Bridging the Language Divide

Subtitling: BBC Weatherview

Page 40: Bridging the Language Divide

Subtitling & Translation: Euro-News

Euronews

Language ID + multilingual ASR + MT

8 Euronews languages

Page 41: Bridging the Language Divide

University Lectures

êß*0vúbØi∫BA¬pysUêÍ}hÿ5

≈ƒÄ<„y‡ëŒkû¢OFˇØ∏kô#å

¯«Zeû

Page 42: Bridging the Language Divide
Page 43: Bridging the Language Divide

Lecture Translation

Page 44: Bridging the Language Divide
Page 45: Bridging the Language Divide

Lecture Transcription/Translation at KIT

• Speech more Spontaneous than TED

• Real-Time Requirement

• Specialist Vocabularies

Page 46: Bridging the Language Divide

Lecture Translator in Karlsruhe

Page 47: Bridging the Language Divide

Lecture Translation E->F

Page 48: Bridging the Language Divide

Lecture Translation G->E

Page 49: Bridging the Language Divide

• Translation of Power Point Slides

• Presentation by Sub-Titles

Tools for Students

Page 50: Bridging the Language Divide

Can Tech Support Human Interpretation?

Page 51: Bridging the Language Divide

EP Rectors’ Conferences Nov.’12-’14

Page 52: Bridging the Language Divide

EP Rectors’ Conferences

Nov.’12-’14

• Demonstrating automatic real-time lecture interpretation

• University Presidents; Interpretation Training & Services

• Promising but Controversial

Page 53: Bridging the Language Divide

Three Use Cases:

– Terminology Support

– Named Entity Support

– Interpreter’s ‘Cruise Control’

Human-Machine Symbiosis

Page 54: Bridging the Language Divide

Voting Sessions

Observations:

Interpreting Voting Sessions is…

– Boring and Repetitive

– Still Stressful, and Demanding

– Many Numbers

and Named Entities

Page 55: Bridging the Language Divide

Field Test at the EP (Dec.14)

Page 56: Bridging the Language Divide

Interactive Systems Labs

Why is this so Hard ?

Language is Ambiguous at All Levels:

– Semantics:

• The Spirit is Willing but the Flesh is Weak

• The Vodka is Good but the Meat is Rotten

– Syntax:

• Time Flies Like an Arrow 6 Different Parses

– Phonetics:

• This Machine Can Recognize Speech

This Machine Can Wrack a Nice Beach

• Give me a New Display Give me a Nudist Play

Page 57: Bridging the Language Divide

Why is German so Hard?

• German has some particularly difficult peculiarities:

– Wordorder:

Ich schlage Ihnen einen Termin für nächste Woche in meinem

Büro am Adenauerring in Karlsruhe, in dem ….. vor.

I propose [hit?] a meeting for next week at my office in

Karlsruhe on the Adenauerring…

– Inflections and Agreement:

Zu der nächsten wichtigen interessanten Vorlesung

– Compounds:

Worterkennungsfehlerrate

Word Recognition Error Rate

Page 58: Bridging the Language Divide

Compounding

Die Fehlerstromschutzschalterprüfung

Die Wirtschaftsdelegationsmitglieder

Die Bankwirtschaftsfreigabeerklärung

Die Lehrverpflichtungserklärungen

Die Schiffskommunalschuldverschreibungen

Die Vorkaufsrechtverzichtserklärung

Das Mehrzweckkirschentkerngerät

Die Gemeindegrundsteuerveranlagung

Die Nummernschildbedruckungsmaschine

Der Mehrkornroggenvollkornbrotmehlzulieferer

Die Verkehrsinfrastrukturfinanzierungsgesellschaft

Die Feuerwehrrettungshubschraubernotlandeplatzaufseherin

Das Rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz

Page 59: Bridging the Language Divide

Compounding

Zentraleuropa:

Zentral-Europa Central Europe

Zentrale-Ur-Opa Headquarter-Great-Grandpa

Dramatisch:

drama-t-isch dramatic

drama-tisch drama table

Asiatisch:

asia-t-isch asian

asia-tich asia table

Page 60: Bridging the Language Divide

Interpreting Language

„Ich freue mich, dass Sie heute so zahlreich....“

you, she, they ?

„If the baby does not like the milk, boil it“

es, sie ?

Page 61: Bridging the Language Divide

Words, Words, Words….

• Technical Terms & Special Usage

– epstral-Koeffizienten, Wälzlagerungen Roller Bearings

– Klausur Final Exam (not Retreat), Vorzeichen Sign (not Omen)

• Formulas:

– Eff von Ix f(x)

• Foreign Words in a German Lecture

– Computer Science- English Expressions

– “Cloud”, “iPhone”, “iPad”, “Laser”

• Declinations and Compounding incl. foreign Words

– Web-ge-casted, down-ge-loaded

– Cloudbasierter Webcastzugriff

Page 62: Bridging the Language Divide

Scientific Challenge

Language Problems can only be Conquered,

if Machines Embrace, Represent, Process:

– Ambiguity:

Scores, Statistics, Neural Activations, ..

– Learning:

Build Models, Extract Knowledge from

Human Data & Interaction, Automatically

Performance Depends on Data & Computing

Page 63: Bridging the Language Divide

Neural Nets: Bigger, Deeper, Faster

(1987) (1989) (2013)

TDNN: Shift-Invariance, Waibel ‘87 Modular (deep) TDNN: Waibel ’87 Waibel et al. Babel, 2013

Weights: ~6,000 ~40,0000 ~33,000,000

TrnData[hrs]: ~0.1 ~1 ~1,000

Time[weeks] ~1 ~1 ~1

Page 64: Bridging the Language Divide

English Text Copora

0

200

400

600

800

1000

1200

1400

1600

2007 2008 2009 2010 2011 2012

News Shuffle SizeM

illi

on

Wo

rds

• Computer MT or ASR systems train on >> 1GWords

– News Shuffle, GigaWord, Europarl, VideoLectures, …

• Human speaks 0.5 GigaWords in a Lifetime!!

Page 65: Bridging the Language Divide

The Data Challenge

• Machine Learning + Massive Data

Lead to Better Performance

• Is the problem too hard? Is it too easy?

Already done? Google Translate?

• Effective Language Solutions

– Not only from/to English, but from/to German, …

– Minority Languages and Regional Dialects

– Need targeted solutions in domain/application

– Privacy and Security

– Dissemination, not only Assimilation

• European Language Solutions

– Language (and technology) must be cultivated and treasured

– Data Volume and Access Key Challenge

Page 66: Bridging the Language Divide

Conclusion

Communication between the people of the world

– Bridging the Linguistic Divide

– Technology can already make helpful contributions

– Methods: Machine Learning from Data

Adaptation, Error Recovery, Learning, Forgetting

– User Interaction, Appropriate Interfaces

– More Data, more Robust Performance

– Better Language Portability

– Integration into Services

Page 67: Bridging the Language Divide