language technology for multilingual europe: current state and future plans

Co-funded by the 7th Framework Programme and the ICT Policy Support Programme of the European Commission through the contracts T4ME, CESAR, METANET4U, META-NORD (grant agreements no. 249119, 271022, 270893, 270899).

Language Technology for Multilingual Europe: Current State and Future Plans

Georg Rehm

Network Manager META-NET German Research Center for Artificial Intelligence (DFKI), Berlin, Germany

European Languages in the Age of Technology: Quo vadis? – Vilnius, Lithuania

November 14, 2012

Outline

q  Introduction

q  Language White Paper Series

q  Strategic Research Agenda

q  Conclusions

http://www.meta-net.eu 2

Multilingual Europe

3 http://www.meta-net.eu

q  Challenge: Providing each language community with the most advanced technologies for communication and information so that maintaining their mother tongue does not turn into a disadvantage.

q  While research has made considerable progress in recent years, the pace of progress is not fast enough to meet the challenge within the next 10-20 years.

q  All stakeholders – researchers, LT user and provider industries, language communities, funding programmes, policy makers – should team up for a major dedicated push.

Objectives

META-NET is a network of excellence dedicated to fostering the tech-nological foundations of the European multilingual information society.


Four EU-Funded Projects

q  Initial project: T4ME (FP7; 13 partners, 10 countries)

q  Three ICT-PSP consortia since Feb. 2011: CESAR, METANET4U, META-NORD

q  All EU member states and several non-member states covered.

q  META-NET in Nov. 2012: 60 members in 34 countries.


http://www.meta-net.eu/members

Language White Paper Series META-VISION


Language White Paper Series


q  Reports on the state of our languages in the digital age and the level of support through language technology.

q  Series covers 30 languages.

q  Key communication instruments to address decision makers and journalists.

q  Inform about societal and technological problems and challenges as well as economic opportunities.

q  >2 years in the making.

q  >200 national experts as contributors. q  >8.000 copies printed and distributed to

politicians and journalists.

30 Languages Covered

q  Basque q  Bulgarian* q  Catalan q  Czech* q  Danish* q  Dutch* q  English* q  Estonian* q  Finnish* q  French*

q  Galician q  German* q  Greek* q  Hungarian* q  Icelandic q  Irish* q  Italian* q  Latvian* q  Lithuanian* q  Maltese*

q  Norwegian q  Polish* q  Portuguese* q  Romanian* q  Serbian q  Slovak* q  Slovene* q  Spanish* q  Swedish* q  Croatian


* = Official EU language

Cross-Lingual Ranking

q  In four application areas, each language is assigned to one of five clusters, ranging from excellent LT support to weak/no support:

1.  Machine Translation

2.  Speech Processing

3.  Text Analysis

4.  Resources

q  Results finalised at a meeting in Berlin with representatives of all 30 languages (October 21/22, 2011).


MT


English

good

French, Spanish

moderate fragmentary

Catalan, Dutch, German, Hungarian, Italian, Polish, Romanian

weak or no support

Basque, Bulgarian, Croatian, Czech, Da-nish, Estonian, Finnish, Galician, Greek, Icelandic, Irish, Latvian, Lithuanian,

Maltese, Norwegian, Portuguese, Serbian, Slovak, Slovene, Swedish

excellent

Czech, Dutch, Finnish, French, German,

Italian, Portuguese, Spanish


Basque, Bulgarian, Catalan, Danish, Estonian, Galician, Greek,

Hungarian, Irish, Norwegian, Polish, Serbian, Slovak, Slovene, Swedish

weak or no support

Croatian, Icelandic, Latvian, Lithuanian, Maltese, Romanian

excellent

English

good

Spee

ch

English

good

Dutch, French, German, Italian,

Spanish


Basque, Bulgarian, Catalan, Czech, Danish, Finnish, Galician, Greek, Hungarian, Norwegian, Polish, Portuguese, Romanian, Slovak,

Slovene, Swedish

weak or no support

Croatian, Estonian, Icelandic, Irish, Latvian, Lithuanian, Maltese, Serbian

excellent

English

good

Czech, Dutch, French, German, Hungarian,

Italian, Polish, Spanish, Swedish


Basque, Bulgarian, Catalan, Croatian, Danish, Estonian, Finnish, Galician,

Greek, Norwegian, Portuguese, Romanian, Serbian, Slovak, Slovene

Icelandic, Irish, Latvian, Lithuanian, Maltese

weak/no support excellent

Res

ourc

es

Text

Ana

lysi

s

Europe’s Languages and LT


Dutch French German Italian

Spanish

Catalan Czech

Finnish Hungarian

Polish Portuguese

Swedish

Basque Bulgarian

Danish Galician

Greek Norwegian Romanian

Slovak Slovene

Croatian Estonian Icelandic

Irish Latvian

Lithuanian Maltese Serbian

English

good support through Language Technology

weak or no support

Not enough R&I on European languages

➔  LT research on European languages, except for English, is too weak and too slow

➔  Many languages are badly covered

0

50

100

150

200

250

300

350

400

450

English

Ch

inese

Germ

an, Stand

ard

Fren

ch

Spanish

Japane

se

Arabic

Dutch

Portugue

se

Czech

Danish

Swed

ish

Hind

i Ko

rean

Turkish

Ita

lian

Russian

Finn

ish

Hebrew

Hu

ngarian

Sloven

e Urdu

Romanian

Zulu

Bulgarian

Catalan-‐Va

lencian-‐Ba

lear

Greek

Thai

Welsh

Estonian

Basque

Ge

rman, Swiss

InukStut

Indo

nesia

n Ineseñ

o LaSn

Marathi

Malay

Pushto

Serbian

Syria

c Tamil

UgariS

c Ukrainian

Uspanteko

Vietnamese

Languages treated in the 2010 editions of Journal of Computational Linguistics and Conferences of ACL, EMNLP and COLING. Many European languages without any reference: Slovak, Maltese, Lithuanian, Irish, Albanian, Croatian, Galician etc.

Key Observations


q  When it comes to Language Technology support, there are massive differences between Europe’s languages and technology areas.

q  LT support for English is ahead of any other language.

q  Even support for English is far from being perfect.

q  The gap between English and the other languages keeps widening!

q  Several languages – Icelandic, Latvian, Lithuanian, Maltese – receive this weakest score in all four areas!

q  At least 21 European languages in danger of digital extinction!(Languages put into the “weak or no support” category at least once.)

White Paper Press Campaign

q  Headline of press release:

At Least 21 European Languages in Danger of Digital Extinction. Good News and Bad News on the European Day of Languages.

q  Sent out to journalists, politicians and other stakeholder groups before the European Day of Languages (September 26).

q  Overwhelmed by the huge interest in the topic and our key findings!

q  520+ mentions in the online and traditional press.

q  40+ interviews with META-NET representatives (television, radio).

q  News came in from 41 countries in 35 different languages.


Coverage by Country


Spain, 15.90%

Bulgaria, 10.80%

International, 7.90%

Latvia, 5.30%

Netherlands, 4.80%

Greece, 4.60% Romania, 4.40%

Serbia, 4.40%

Italy, 4.20%

Germany, 3.50%

Russia, 3.50%

Estonia, 2.90%

France, 2.60%

Slovenia, 2.40%

Iceland, 2.20% Malta, 2% USA, 1.50%

Denmark, 1.30%

Latin America, 1.30%

Lithuania, 1.30%

Ireland, 1.30% UK, 1.10%

Belgium, 0.90%

Finland, 0.70% Sweden, 0.70%

Poland, 0.70%

Norway, 0.40%

Mexico, 0.40%

Brazil, 0.40%

Slovakia, 0.40%

Basque Country, 0.40% Portugal, 0.40% Austria, 0.20%

New Zealand, 0.20%

Hungary, 0.20%

Bosnia and Herzegovina, 0.20%

Costa Rica, 0.20%

Cyprus, 0.20%

Canada, 0.20%

Australia, 0.20%

Spain

Bulgaria

International

Latvia

Netherlands

Greece

Romania

Serbia

Italy

Germany

Russia

Estonia

France

Slovenia

Iceland

Malta

USA

Denmark

Latin America

Lithuania

Response: Examples

q  Austria: Der Standard. q  Denmark: Politiken, Berlingske Tidende. q  Finland: Tiede. q  Germany: Heise Newsticker, Süddeutsche Zeitung. q  Greece: in.gr, Πρώτο Θέµα, Prosilipsis. q  Iceland: Fréttablaðið, Morgunblaðið. q  Italy: Wired. q  Lithuania: Delfi, Elektronika, KaunoDiena. q  Norway: Computerworld. q  Slovenia: Delo, Dnevnik, Demokracija. q  Serbia: Politika. q  Spain: El Mundo. q  UK: Huffington Post. q  USA: Mashable, NBC News, Reddit.


Date 30 September 2012 Page 16

Copyright material. This may only be copied under the terms of a Newspaper Licensing Agency agreement (www.nla.co.uk) or with written publisher permission. For external republishing rights see www.nla-republishing.com

Press Campaign: Examples


38

!!"# $"%&'() *+,-) .*#… µ&/,0# *//"#&(1, 2+34 ('& '5(*!64 1//*4 *753+'8(64

9/:;;*4, ;0µ%3#' µ* +'#*753+'8-() 6(<*;" µ* !"# 7+,95'%) 200 ('& +/6,# *&.&(:#. = ;79(*(5&µ6#" µ*/6-!" ."µ,;&*0!"(* '+2 !, *+&;!"µ,#&(2 .>(!7, ?@AB-C@A µ* '%,5µ) !" -!*-;&#) @753+'8() =µ65' D/3;;:#.

D&' !&4 '#19(*4 !"4 65*7#14 !,74, 9/3;;,/29,& '+2 34 -:5*4 !"4 D"-5'&14 =+*>5,7 E'<µ,/29";'# !&4 .&'<6;&µ*4 9/3;;&(64 7+"5*;>*4 ('& ."µ&,059";'# 6#' «F*7(2 G&-E/>,» 9&' (1<* *753+'8() 9/:;;'. H!" µ*/6!" !,74, ,& *&.&(,> '#'I)!"-;'# µ*!'J0 1//3# !6;;*5' E';&(1 "/*(!5,#&(1 *59'/*>', ."/'.) !"# 0+'5J" '7!2µ'!"4 µ*!1%5';"4, !" .7#'!2!"!' %3#"!&()4 '//"/*-+>.5';"4 ('& $"%&'()4 '#1/7;"4 (*&µ6#,7, *#: !'7!2-5,#' .&*5*7#)-<"(* ('& " .&'<*;&µ2!"!' 9/3;;&(:# +253# ) +"9:#.

H* +5:!" %1;" *J6!';'# !&4 &;!,-;*/>.*4 +,7 *+&!56+,7# ;!,74 -5)-;!*4 #' (1#,7# µ*!'%51;*&4 online, 2+34, 9&' +'51.*&9µ', " 7+"5*;>' !,7 (,/,;;,0 +/"5,%,5&()4 Google Translate. A"# >.&' :5', *J*!1;!"(* ('& " «*+&(,&#3#>'» !3# *//"#2%3-#3# -5";!:# µ* !&4…;7;(*764 !,74, 2+34 9&' +'51.*&9µ' " .7#'!2!"!'

#' «µ&/);*&» (1+,&,4 ;!, GPS ;!" µ"!5&() !,7 9/:;;'. K& *5*7#"!64 ('!6/"J'# ;!, ;7µ+65';µ' 2!& 7+15-,7# !6!,&*4 ;7;(*764, '//1 .*# *>#'& !2;, .&'.*.,µ6#*4 2;, ,& '99/2%3#*4. A, «-57;2» µ*!1//&, ('!'(!1,

2+34 *>#'& 1//3;!* ('& /,9&(2, " '99/&() 9/:;;'. K& '99/2%3#,& -5)-;!*4 6-,7# !"# ('/0!*5" .7#'!) !*--#,/,9&() 7+,;!)5&J", (1!& !, ,+,>, *7#,*> !"# +*5'&!653 *J1+/3;" !"4 9/:;;'4. B+2 «!*-#,/,9&(2 '+,-(/*&;µ2» (&#.7#*0,7# +*5&;;2!*5, " &;/'#.&(), " /*!,#&(), " /&<,7'#&() ('& " µ'/!6I&(" 9/:;;', *#: ;* />9, ('/0!*5" µ,>5' E5>;(,#!'& " *//"-#&(), " E,7/9'5&(), " ,7995&() ('& " +,/3#&(), +,7 2+34 '#'%65*& " 65*7#' 6-,7# «'+,;+';µ'!&()» !*--#,/,9&() 7+,;!)5&J".

«?6!5&'» -'5'(!"5>I*!'& " 7+,-;!)5&J" -5";!:# ;* ,//'#.&(), 9'/-/&(), 9*5µ'#&(), &!'/&() ('& &;+'#&() 9/:;;'. K& *+&(*%'/)4 !"4 *+&;!"-µ,#&()4 ,µ1.'4, !"#$ %&'()*+,- ('& .(/0*1( 2/µ, '#'%65,7# -'5'(!"5&-;!&(1: «L+15-,7# .5'µ'!&(64 .&'%,-564 ;!"# 7+,;!)5&J" !"4 9/3;;&()4

!*-#,/,9>'4 '#1µ*;' ;!&4 .&1%,5*4 *753+'8(64 9/:;;*4. A, -1;µ' µ*!'-J0 “µ&(5:#” ('& “µ*91/3#” 9/3;;:# ,/,6#' ('& .&*750#*!'&. M56+*& #' *J';%'/>;,7µ* !,# *%,.&';µ2 !3# µ&(52!*53# ('& /&92!*5, +/,0;&3# ;* $"%&'(,04 +25,74 9/3;;:# µ* !&4 '+'5'>!"!*4 E';&(64 !*-#,/,9>-*4. N&'%,5*!&(1, ,& 9/:;;*4 '7!64 *>#'& ('!'.&(';µ6#*4 ;* $"%&'() *J'%1#&;"».

?1/&;!', ,& *&.&(,> !,#>I,7# 2!& -3-5>4 '+,%';&;!&() .51;" ,& 9/:;;*4 '7!64 .0;(,/' <'… *+&E&:;,7# ;!,# $"%&'(2 (2;µ,7 !,7 21,7 '&:#'. = (. 3"*4" ."5*,6748)&, µ6/,4 !"4 *+&-;!"µ,#&()4 ,µ1.'4 '+2 !, O#;!&!,0!,

@+*J*59';>'4 !,7 F29,7 @5*7#"!&(2 P6#!5, B<"#1, /6*& ;!,# «@.A.»: «= 65*7#' '7!) .*# /6*& 2!& .*# <' I);*& " *//"#&() 9/:;;' ) 2!& (&#.7#*0*& µ* *J'%1#&;"». = *&.&(24 *J"9*> 2!& 2;, 7+15-,7# 1#<53+,& +,7 µ&/1-#*, 951%,7# ('& *+&(,&#3#,0# µ* µ&' 9/:;;', !2!* '7!) <' ;7#*->;*& #' 7+15-*&. @>#'& ;"µ'#!&(2, 2µ34, #' 6-,7# 2/,& ,& -5);!*4 !" .7#'!2!"!' #' «µ&/);,7#» ;!&4 µ"-'#64, 2+34 !' GPS !,74, ;!' *//"#&(1 ('& #' 6-,7# ;!" .&1<*;) !,74 9/3;;&(1 *59'/*>' "/*(!5,#&(:# 7+,/,9&;!:#.

?*!'J0 '7!:# !3# «*59'/*>3#» *>#'& ,& .&,5<3!64 ,5<,95'%&(:# ('& ;7#!'(!&(:# /'<:#, +,7 -5";&µ,+,&-,0#!'& ('<"µ*5&#1 '+2 *('!,#!1.*4 @//"#*4 -5);!*4 ('& E';>I,#!'& ;!" 9/3;;&() !*-#,/,9>'. M'5’ 2/' '7!1, !,#>I*& 2!& " $"-

%&'() *J1+/3;" µ&'4 9/:;;'4 *>#'& ;"µ'#!&() «N*# *>#'& ;!' -65&' !,7 µ6;,7 -5);!". K& *(1;!,!* (7E*5-#);*&4, " @753+'8() @#3;" ('& , &.&3!&(24 !,µ6'4 +56+*& #' -5"µ'-!,.,!);,7# !"# '#1+!7J" '7!)4 !"4 !*-#,/,9>'4 9&' 2/*4 !&4 9/:;;*4», '#'%65*& ('& ;7#*->I*&: «K& -5);!*4, 2µ34, +56+*& #' '+'&!,0# #' 7+15--,7# ('& ;!" 9/:;;' !,74 !' µ6;' '7!1 ('& #' µ"# &('#,+,&,0#!'& µ* !' '99/&(1». Q

!"µ#$% 27 &'#$'µ()*+, 2012 !"!#$!%&' (#)&'

LifeMKFF@H @LRSMBTP@H DFSHH@H U@SRKLCABO A@VCKFKDOPB… W@M@RBH?@C@H

!" #$%&'() "*'%+,&-$ (&,./,"01/, 2' "33$,&(+

!"!*+ ,!%-&#[email protected]

= 9/:;;' !"4 '+,J6#3;"4…

XX GREEKLISH

-. /0012345+. 6)78$'9 "6+,5 $%5 :/1;$')% <,5/$7 $'65+1+0.:7 ,#+8$7).=%, 0'0+529 #+, ',5+'* $%5 #')/.$")4 '=>#148% $%9 01?88/9

.! GREEKLISH /9,(),#:-#);# 97<)# µ<=: µ6#&µ+--:# > email ), 9/*,==0-/*), #<), -6$ ?@*"$ µ"$. A"*+ -) 1/1)#0$ 0-, -" -/7/&--"4" ?*0#," &9+*?)&# -" 17:==,(+ /*1"7/4", -" )9)4" /9,-*<9)&# -6 ?*>=6 -6$ /776#,(>$ 1*"µµ"-)-=/,*+$, <B65), (", #<), /#>7,(/$ B"4#/-", 0-, 8/# <?)&# «"1("7,+=/,» "&-<$ -,$ -/?#)7)14/$. % ("C6-16->$ .7:==)7)14"$, (. .,@*1)$ 39"µ9,#,@-6$, 7</, =-)# «D.E.»: «E" greeklish /4#", 9*0576µ" 1," -6# /776#,(> 17@==", ,84:$ 1," "#C*@9)&$ #<"$ 67,(4"$ 1," <#"# ("C"*+ 17:==,(0 701). 3/ -6 ?*>=6 -:# greeklish "9)F/#@#)#-", "90 -6 µ)*B> -6$ 7<F6$ > 09:$ 7<µ/ -) /-&µ)7)1,(0 4#8"7µ" 9)& 867@#/-", µ/ -6# )*C)1*"B4" -6$ 7<F6$ (", =&#8</-", (", µ/ -6 =6-µ"=4" -6$ 7<F6$ (", µ/ -6# 9*)<7/&=> -6$». % (4#8&#)$, µ/ -)# )9)4) <*?)#-", "#-,-µ<-:9), ), #<), +#C*:9),, /4#", 6 "9)F<#:=6 "90 -6 1*"9-> µ)*B> -6$ 17@=-="$. G&-> 6 «),(/,0-6-"», 0µ:$, 5)6C+/, (", =-6# ("-"#06=6 -6$ =6µ"=4"$ "77+ (", -6# 9*)<7/&=6 -6$ 7<F6$. «G&-> 6 "9)F<#:=6 8/# /4#", +#/& =6µ"=4"$», "#"B<*/, ) /,8,(0$, ) )9)4)$ /F61/4 0-, 6 8,"8,("=4" -6$ 1*"B>$ 5)6C+/, #" /#-&9:-C/4 6 7<F6 (", #" =&#8/C/4 µ/ +77/$ )µ0**,'/$ 7<F/,$. «%-"# ?*6=,µ)9),/4-", "&-> 6 µ)*B> /9,(),#:#4"$, ("--"=-*<B)#-",, "-)#);#. H/# /4#", 9*)$ C+#"-), "77+ C" (+#/, '6µ,+», "#"B<*/, ) (. 39"µ9,#,@-6$, ) )9)4)$ =&µ5)&7/;/, -)&$ ?*>=-/$ #" /9,7<1)&# -6# /776#,(> 1*"µµ"-)=/,*+.

-/01234 .56µ5/7/0894.

Website: Visitors Overview


began sending out press release

European Day of Languages

unusually high traffic

Website: Visitors’ Cities


Strategic Research Agenda META-VISION


Three Ingredients

21

Appropriate Programme

Vision & Agenda

Appropriate Actors

Research & Commercialisation

Appropriate Support

Funding

http://www.meta-net.eu

Strategic Research Agenda


q  META-NET Strategic Research Agenda for Multilingual Europe 2020.

q  Addresses the problems we found when preparing the white papers.

q  Three priority research themes and application/innovation scenarios.

q  Can put Europe ahead of its competitors in this technology area.

q  190+ contributors.

q  Final version to be ready in Nov. 2012.

q  SRA will be presented to the EC and national bodies.

Strategic Research Agenda


Priority Themes: 3 + 2

q  Three Priority Research Themes: §  Translation Cloud

§  Social Intelligence and e-Participation

§  Socially-Aware Interactive Assistant

q  Two additional themes: §  European Language Technology

Platform

§  Core Technologies for Language Analysis and Production


Gartner Hype Cycle 2012

13 of the 48 emerging technologies are

language technologies!

Conclusions META-NET


Conclusions

q  Our white paper press campaign shows that Europe is extremely interested in and passionate about its languages.

q  Two Parliamentary Questions in the European Parliament on the “digital extinction of languages” topic.

q  Now is the time to move forward with a continent-wide, systematic push and to invest in strategic research.

q  A modest investment is required.

q  This push will generate a countless number of opportunities.

q  Horizon 2020 and Connecting Europe Facility can provide sufficient resources to make our visions for Europe’s citizens and economy a reality.


Labai ačiū!

Thank you very much! [email protected] http://www.meta-net.eu http://www.facebook.com/META.Alliance

28

Q/A

2013 2020

Maturity

Database Technology Language TechnologyNetwork Technology

Language Technology

Web Technology