how to keep post-editors engaged and prevent attrition. (jose sanchez, ebay)

ENGAGING POST-EDITORS

JOSÉ LUIS BONILLA SÁNCHEZOctober 14, 2015

ENGAGING POST-EDITORS 2

TOPICS

• BACKGROUND: MT AT EBAY

• THE CHALLENGES OF POST-EDITING (INDUSTRY AND EBAY)

• THE EBAY EXPERIENCE: PROBLEMS AND SOLUTIONS

• A LOOK AT THE FUTURE

• DISCUSSION


MT AT EBAY

•Since 2013

•Home-grown, statistical (Moses) engines

•Covering 10 language pairs (FR IT ESES ESLA DE BRPT •RU ZH USEN UKEN)

•Content translated: listings (item titles and descriptions), keywords


CHALLENGES OF POST-EDITING


INDUSTRY CHALLLENGES

Post-Editing projects add extra complications to the regular L10n flow. For instance:

- Quality expectations can be unclear (definitions of light and full post-editing vary).

- No universal agreement on rates (Per hour? Percentage discount? Edit distance?).

- Every project (and engine) comes with its own difficulties: statistical vs rule-based engines, technical vs social content…

Sanchez, Jose

not sure what you mean by stress test here - please rephrase and make your intention clear.Often "unclear" - would rephrase by "can be unclear" - we are very clearEllaborate on engine (hybrid-rulesbased- statisticalNot sure about the picture at the bottom - what is this


EBAY CHALLENGES

At eBay, we deal with especially complex projects:

- 12k+ eBay categories, many with their own terminology

- User-generated content: Unpredictable quality, slang, non-standard acronyms and abbreviations…

- Very specific requirements: Our goal is not polished content, but content which can be understood and useful to train the engine

Sanchez, Jose

especially challenging projects12K domains? Please leave the number out unless you can explain what domains are - domains for me are categories and itemsUser generated content - leave out not professionally written - everybody knows thatI don't get the last point - just say "our goal is not polished content, but content which can be understood and useful to train the engine)

Our Solutions


Modular Guidelines…

Language-

specific

Content-specific

General


…Structured to Facilitate Learning

General introduction

PE-Specific Instructions

Item Titles Languages

RU

BRPT

ESLA

FR

IT

ESES

DE

ZH

Item Descriptions

QueriesGeneral

Translation Instructions

Rioufol, Sabine

Hard to read - there is a lot to digest on this slideCan you elaborate with bullet poimts instead of a mind chart?


Recorded Trainings

The more specialized the training……the more important to preserve the information in recorded format so future post-editors can refer to it.

Rioufol, Sabine

what do you mean by "specialized training" - please give an example - what do you mean by "preserve"? Record to enable the end-user to go back to itthird fail - this is the first time you are mentiong Fails. This should be in a different slide


Escalation as Needed

3rd fail triggers a call with the vendor. • Participants: Linguists, PjM, Quality Manager• Agenda:- Diagnosis- Vendor Action Plan- Feedback for Client

Rioufol, Sabine

what do you mean by "specialized training" - please give an example - what do you mean by "preserve"? Record to enable the end-user to go back to itthird fail - this is the first time you are mentiong Fails. This should be in a different slide


2014 RESULTS (FAILS VS PASSES)

1 2 3 4 5 6 7 8 9 10 11 120

5

10

15

20

25

30

35

40

45

Reviews Pass Fail

Rioufol, Sabine

these are stats - not "how we do it"Please revise and explain.


2015


EARLY SAMPLE REVIEWS

In-progress review for most problematic language combinations:

First 2k words of the project, in the first project week

Rioufol, Sabine

first 2K of what?first project week - unclear


ADDITIONAL REFERENCE MATERIAL

Providing vendors with MT translations from 2 engines (generic and customized) so they pick the best

Out of Vocabulary words are unknown to the system, and left in English, which often makes them easy to mistake for brand names. By automatically tagging OOV, we allow the vendor to focus on them.

eBay listing titles can be difficult to understand without context (images, descriptions).

Alternative MT translation

Tagging “suspect” terms (OOV)

Providing full context (HTML files)

Rioufol, Sabine

remove the bottomOther actions - seems vague to meMultiple MT options - what do you mean? ElaborateTagging (some people will not kb

Rioufol, Sabine

elaborate on multiple MT optionsTagging (explain) - spell out the acronymElaborate on HTML file contextremove the bottom part which belongs to the context


RESULTS - PRODUCTIVITY

Series10%5%

10%15%20%25%30%35%40%45%

Productivity Increase

Alternative MT Translation

Tagging "suspect" terms (OOV)

Providing full context


RESULTS - QUALITY

1 2 3 4 5 6 7 8 9 10 11 120

5

10

15

20

25

30

35

40

45

Pass/Fail over Time - 2014

Reviews Pass Fail

1 2 3 4 5 6 7 8 9 10 11 120

10

20

30

40

50

60

70

Pass/Fail over Time- 2015

Reviews Pass Fail

THE FUTURE

Divider sub-headline goes here


THE NEXT BREAKTHROUGHS

By predicting the quality of the MT output, we could:- Filter and send to our vendors the best (or worst) translations.- Map QE score to time spent, and use it to calculate more accurate initial rates.

Tools like iOmegaT or Matecat offer time tracking, edit distance analysis and even action recording – usable to:- Analyze post-editor behavior and identify

areas to help them improve.- Calculate accurate rates.

The future is for online tools – they will allow for more direct interaction (early sampling, continuous communication, centralized information repository), further integrating the post-editors with the team.

Quality Estimation

Behavior Tracking

Online Collaboration


DISCUSSION TIME

how to keep post-editors engaged and prevent attrition. (jose sanchez, ebay)

Presentations & Public Speaking