Timed Text at Netflix

Rohit Puri (rpuri@netflix.com)

Engineering Manager, Video Systems

Digital Supply Chain, NetflixFebruary 19, 2016 0

The Netflix Content Processing System

• Video systems team develops cloud-scalable systems and tools– Audio/Video: ingestion/inspections and packaging/DRM

• e.g., IMF, QuickTime, and MP4-DASH– Timed Text: entire processing pipeline

• e.g., W3C TTML, W3C WebVTT

1

Ingestionand

Inspections

Packagingand

DRMTrans-coding

sourcefromcontentpartner

downloadableto CDN

February 19, 2016

Acknowledgements

• Dae Kim (dakim@netflix.com)

• Shinjan Tiwary (stiwary@netflix.com)

• Harold Sutherland (hsutherland@netflix.com)

• David Ronca (dronca@netflix.com)

• Glenn Adams (Skynav Inc.) - member W3C Timed Text Working Group (TTWG)

February 19, 2016 2

Netflix after January 6, 2016

• > 75 million subscribers• ~ 190 countries• > 12,000,000,000 hours streamed in Q4 2015• 20+ languages

3February 19, 2016

Talk Outline

• History and Legacy Workflow

• New Workflow

• Standards Activity and Roadmap

4February 19, 2016

Talk Outline

• History and Legacy Workflow

• New Workflow

• Standards Activity and Roadmap

5February 19, 2016

A Brief History of Timed Text

• Latin Alphabet (2014 and before)– First subtitles delivered in 2009. Bottom-centered, yellow,

italics and underline options (dfxp-simplesdh)

– Follow-up 2012 offering. multiple colors, background, outline, generic font family, font size, position information (dfxp-ls-sdh) (ls => ‘less simple’)

– First WebVTT output profile was created in 2013

• Global Subtitles (2015+)– TTML2 (Timed Text Markup Language) based Japanese

subtitles in 2015

– Bidirectional text with worldwide launch in January 2016February 19, 2016 6

Legacy Workflow (2014 and before)

• Two step-procedure– source inspection– source conversion to output

• Sources– CEA-608 based Scenarist Closed Captions (.scc)– EBU Subtitling data exchange format (.stl)– SubRip (.srt)– Timed Text Markup Language 1 (.ttml, .dfxp)

• Outputs– feature restricted TTML1 outputs– feature restricted WebVTT outputs

February 19, 2016 7

Legacy Workflow: Inspections

8

ProprietaryCanonical

model

Semantic inspections

STL parser and syntax inspections

SCC parser and syntax inspections

TTML1 parser and syntax inspections

.scc

.stl

.ttml

SRT parser and syntax inspections

.srt

Legacy Workflow: Conversions

9

ProprietaryCanonical

model

STL parser

SCC parser

TTML1 parser

.stl

.ttml

SRT parser.srt

TTMLwriter

WebVTTwriter

.scc

Talk Outline

• History and Legacy Workflow

• New Workflow

• Standards Activity and Roadmap

10February 19, 2016

Japanese Subtitles

• Essential Features– vertical text– ruby annotations– horizontal-in-vertical– Unicode

11February 19, 2016

Japanese Subtitles Challenges

• New source format - Videotron Lambda (.cap)– inaccessible specification– ambiguity in format– non-interoperable vendor implementations

• TTML1 does not support essential Japanese features, TTML2 needed

• Netflix device SDK did not support essential rendering features– delivery of image-based subtitles

– significant investment in open source software (OSS) tools 12February 19, 2016

Japanese Subtitles Conversion Workflow (1-off)

• .cap sources converted to TTML2, and then to image subtitles or webVTT (webVTT specification appears incomplete w.r.t. Japanese)

• Image subtitles archive contains .png images + manifest with timing and positioning information

• Green modules available as OSS (https://github.com/skynav/ttt)

13

cap2tt

TTPETTX

WebVTT writer

Archiver

ttml2

ttml2 ISD .png images

February 19, 2016

New Workflow: Inspections

14

TTML2based

canonical model

Semantic inspections

STL parser and syntax inspections

SCC parser and syntax inspections

CAP parserand syntax inspections

TTML1,TTML2,IMSC1

SRT parser and syntax inspections

TTML familyparser and syntax inspections

model serialized to disk

Sourceformat and

charsetdetection

inputfile

February 19, 2016

New Workflow: Conversions

• Configurable filter-based architecture• Bank of model-domain filters• Output writer generates text or images

15

model-domain filter chain

deserializedmodel FilterN

Output generator

Filter1output

specificfilters

outputwriter

outputfile

February 19, 2016

Global Subtitles

• i18n-grade timed text processing workflow

• All languages of this world (+Klingon)

• New TTML2-based output profile (“nflx-ttml-gsdh”)– vertical text– horizontal-in-vertical– ruby annotations– bidirectional text

• Both text and image delivery options16February 19, 2016

Talk Outline

• History and Legacy Workflow

• New Workflow

• Standards Activity and Roadmap

17February 19, 2016

Our Experience with Source Formats

• SCC– limited to Latin alphabet– non-standard use of SMPTE timecode

• EBU-STL– dated1, ambiguous, non-interoperable industry practices– no support for Asian character set

• SRT– Too simple - positioning information not used in practice

• LambdaCAP– ambiguous format - hard to find official specification

• TTML1– not self-contained - needs “Document Processing Context”– no support for Japanese rendering features

181“The medium for exchange is a 3.5-inch high-density portable magnetic disk (microfloppy). The disk is formatted for 1.44 Mbytes (2 sides, 80 tracks, 18 sectors/track).”

TTWG Standards

• IMSC1 (Internet Media Subtitles and Captions) is TTML1-based W3C candidate recommendation; mandatory for IMF– multiple Netflix sponsored implementations were announced to

TTWG on February 1, 2016– Netflix plans 100% support for IMF– Netflix ingest implementation will support IMSC-T (text profile)

• We have multiple TTML2 implementations in development - will support TTML2-based IMSC2

• Netflix is enthusiastic about HTMLCue

19February 19, 2016

Open Source Activity

• regxmllib (https://github.com/sandflow/regxmllib)– sponsored by Netflix– tools that provide essential building blocks for authoring of

IMF CPL

• ttt (https://github.com/skynav/ttt)– sponsored by Netflix– tools for validation and rendering of TTML1/2

• photon (https://github.com/Netflix/photon) (December 2015)– developed at Netflix– complete set of tools for validation of IMF packages

February 19, 2016 20

Talk Summary

• Subtitles experience core to Netflix business

• Netflix committed to TTWG standards– multiple IMSC1 and TTML2 implementations declared or

in flight

• Netflix plans to be 100% IMF

• Netflix is actively involved in OSS efforts around timed text as well as IMF

21February 19, 2016

2015 Tech Emmy for Netflix

• Netflix was a co-recipient of 2015 Technology and Engineering Emmy Award “Standardization and Pioneering Development of Non-Live Broadcast Captioning”

22

Questions?

• Rohit Puri (rpuri@netflix.com)• https://www.linkedin.com/in/rohit-puri-0a13b02

February 19, 2016 23