how to evaluate mt quality based on effectiveness (adam lamontagne, language technology dev. &...
TRANSCRIPT
![Page 1: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/1.jpg)
Evaluating (MT) quality based on effectiveness
Adam LaMontagne (Language Technology Dev. & Deployment Manager, Moravia)
TAUS Roundtable, 15 March 2016, Vienna
![Page 2: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/2.jpg)
Agenda
Quality Usability Effectiveness An Interconnected System Beyond Individual Quality Measures Discussion
TAUS Roundtable, 15 March 2016, Vienna
![Page 3: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/3.jpg)
TAUS Roundtable, 15 March 2016, Vienna
What is Quality?
![Page 4: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/4.jpg)
TAUS Roundtable, 15 March 2016, Vienna
Measuring QualityAutomated
BLEU F-Measure METEOR Levenshtein NIST ROUGE TER(p) WER
Human
![Page 5: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/5.jpg)
TAUS Roundtable, 15 March 2016, Vienna
What is Usability?
• Effectiveness - can users complete tasks, achieve goals with the product, i.e. do what they want to do?
• Efficiency - how much effort do users require to do this? (Often measured in time)
• Satisfaction – what do users think about the products ease of use?
Did the user achieve their goal?
Credit: http://www.usabilitynet.org/
![Page 6: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/6.jpg)
Measuring Usability
TAUS Usability Comprehension tests Questionnaires Participant observation Screen recording Think-Aloud Protocols Eye Tracking
TAUS Roundtable, 15 March 2016, Vienna
![Page 7: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/7.jpg)
What is Effectiveness?
Help the user to take action:• User documentation• Help content• Training material• User Interface
Help the user to make a decision:• Product/Service
descriptions
• Marketing content Help the user to
communicate:• Chat• Email• Social media
Help to protect the user:• Signage• Legal
TAUS Roundtable, 15 March 2016, Vienna
What does the content really want?Different content types have different purposes:
![Page 8: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/8.jpg)
Measuring Effectiveness
Active User Feedback Passive User Feedback Metadata (Indirect/abstracted measures of
effectiveness)
TAUS Roundtable, 15 March 2016, Vienna
![Page 9: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/9.jpg)
Measuring Effectiveness
TAUS Roundtable, 15 March 2016, Vienna
Active User Feedback User ratings User feedback & continuous
improvement
![Page 10: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/10.jpg)
Measuring Effectiveness
TAUS Roundtable, 15 March 2016, Vienna
Example: Microsoft Collaborative Translation Framework (CTF)
Live examples from: https://support.microsoft.com/fr-fr/kb/274703CTF Overview: https://www.microsoft.com/en-us/translator/ctf.aspx
![Page 11: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/11.jpg)
Measuring Effectiveness
TAUS Roundtable, 15 March 2016, Vienna
Passive User Feedback Web analytics : “User Engagement”
Clicks Click depth Duration Conversion Bounce rate Drop-off rate
Screenshot from Google Analytics
![Page 12: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/12.jpg)
Measuring Effectiveness
TAUS Roundtable, 15 March 2016, Vienna
Metadata (Indirect/Abstracted effectiveness Metrics) SEO results Native-language support communication Call/help center costs Sales/revenue data by market
![Page 13: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/13.jpg)
An Interconnected System
EffectivenessUsabilityQuality
TAUS Roundtable, 15 March 2016, Vienna
![Page 14: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/14.jpg)
Beyond Individual Quality Measures
TAUS Roundtable, 15 March 2016, Vienna
Combining & correlating evaluation metrics
Quality Metrics Automated QA & scoring LQA DQF metrics Errors/error typology Automated MT metrics
Effectiveness Metrics User feedback Web analytics Metadata
Usability Metrics Comprehension tests Questionnaires Participant observation…
![Page 15: How to evaluate MT quality based on effectiveness (Adam LaMontagne, Language Technology Dev. & Deployment Manager, Moravia)](https://reader035.vdocuments.net/reader035/viewer/2022070600/589a0d821a28ab7a318b5f33/html5/thumbnails/15.jpg)
Discussion
TAUS Roundtable, 15 March 2016, Vienna