yapi.js introduction (mopcon 2016 version)
TRANSCRIPT
![Page 1: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/1.jpg)
ADAPTIVE MEDIA PLAYER BASED ON HTML5
陳建⾠ jessechen
yapi.js
![Page 3: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/3.jpg)
OTT VOD SERVICE台⽇韓中超⼤⽚庫
⼤數據精準推薦
離線播放
投放⾄電視
![Page 4: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/4.jpg)
![Page 5: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/5.jpg)
PLAYBACK EXPERIENCEON WEB
WITH HTML5
![Page 6: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/6.jpg)
影⽚yapi .js
(台)
![Page 7: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/7.jpg)
WHAT IS BEING PLAYEDmedia
![Page 8: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/8.jpg)
analog-to-digital adapter
digital raw data
PREPARE MEDIA1. Make it digital
analog signal
![Page 9: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/9.jpg)
H.264 / H.265Encode
PREPARE MEDIA2. Encode video file
Raw data .mp4 / .ts file
Easier for storage and transference
![Page 10: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/10.jpg)
server holds media
VINTAGE WAY
Player get it and play
single file
super slow
![Page 11: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/11.jpg)
server holds media
EFFICIENT WAY
Player get part of video and
start play
streaming(progressive download)
Keep requesting rest part of video during
playback
inflexible
![Page 12: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/12.jpg)
PREPARE MEDIA3. Fit adaptive need
.mp4 / .ts file
more storage needed but much flexible
different bitrate
+
+
fragments
manifestfile
![Page 13: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/13.jpg)
MANIFEST TELLS- adaptions: video, audio or subtitle
- available bitrates
- how file is fragmented
- encryption
![Page 14: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/14.jpg)
ADAPTIVE STREAMING TYPES
SS Smooth Streaming
HLS HTTP Live Streaming
DASH Dynamic Adaptive
Streaming through HTTP
![Page 15: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/15.jpg)
.mp4 / .ts file different bitrate
+
+
fragments
manifestfileRaw data
imaging there are thousands of media needs to handle
![Page 16: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/16.jpg)
CONTENT OF KKTV IS PREPARED WITH
MASShttps://github.com/KKBOX/mass
https://goo.gl/6TCmWqintroduction by Jinkuen:
![Page 17: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/17.jpg)
download
adaptive streamingclient
yapi.js
media
manifestfile
adaptivealgorithm
fragments
![Page 18: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/18.jpg)
adaptive algorithm decides which quality to download
![Page 19: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/19.jpg)
• network status
• how long does buffered duration left
algorithm considering…
![Page 20: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/20.jpg)
adaptive
starting lag
buffering times / duration
overall bitrate
![Page 21: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/21.jpg)
a/v sync
decode
decrypt
‘playback’client
yapi.js
media
manifestfile
fragments
![Page 22: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/22.jpg)
decrypt
encryption + decryption
![Page 23: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/23.jpg)
H.264 / H.265Encode
ENCRYPTION
Raw data .mp4 / .ts file
encryption implemented
here
![Page 24: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/24.jpg)
CLIENT SIDE DECRYPTION
• Client get a license instead of a key
• A blackbox/sandbox get the key by processing license
• That blackbox/sandbox provide key for decryption and directly output to display
• This solution called DRM(Digital Right Management)
![Page 25: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/25.jpg)
DRM ON BROWSER
• For browser, that blackbox called CDM (Content Decrypt Module)
• Each browser support different DRM
context - “a blackbox or sandbox get the key by processing license”
![Page 26: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/26.jpg)
DRM ON BROWSER
Widevine FairplayPlayready PrimetimeWidevine
![Page 27: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/27.jpg)
INTRODUCE EME
• Even though browser support its own DRM, W3C defines a EME spec, in order to expose same api to use
• prefixed api was implemented on earlier version of chrome(smart tv)
Encrypted Media Extension
![Page 28: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/28.jpg)
YAPI’S PROTECTION LOGIC
• implement protection on client side through EME
• give what DRM server needs to retrieve license
• deal with different browsers (versions)
![Page 29: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/29.jpg)
EMECDM
attempting to play encrypted media
get ‘challenge’
DRM license server
request with challenge
get license
yapi
provide license for CDM to decrypt content
EMECDM
PROTECTION LOGIC FLOW
![Page 30: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/30.jpg)
a/v sync
decode
html5 media element + MSE
![Page 31: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/31.jpg)
MEDIA ELEMENT
var videoNode = document.createElement(‘video’);
videoNode.src = VIDEO_URL;
With html5 media element, you can play single video source easily
![Page 32: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/32.jpg)
single source multiple source
+
+
![Page 33: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/33.jpg)
MSE
“ MSE (Media Source Extension) extends HTMLMediaElement to allow JavaScript to generate media streams for playback.
Allowing JavaScript to generate streams facilitates a variety of use cases like adaptive streaming and time shifting live streams. “
media source extension
![Page 34: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/34.jpg)
var video = document.createElement(‘video’);
video.src = VIDEO_URL
MEDIASOURCE IS A ‘SOURCE’
set ‘src’ attribute of video element to an url pointed to media source
new window.MediaSource();window.URL.createObjectURL(ms);
![Page 35: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/35.jpg)
SOURCE BUFFER
sourceBufferVideo = ms.addSourceBuffer(VIDEO_CODEC); sourceBufferAudio = ms.addSourceBuffer(AUDIO_CODEC);
// get stream buffer via network
sourceBufferVideo.appendBuffer(buffer);
// sourcebuffer provides buffer info after append complete
![Page 36: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/36.jpg)
BUFFER INFO
var buffered = sourceBuffer.buffered; buffered.length; // how many discontinuous buffered time range
buffered.start(0); // first buffer start time
buffered.end(0); // first buffer end time
get buffer information from buffered attribute
![Page 37: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/37.jpg)
MSE EXTENDS MEDIA ELEMENT
• MSE focus on providing stream buffer to media element
• playback behavior still hold by media elemente.g play, pause, seek
![Page 38: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/38.jpg)
a/v sync
decode
‘playback’client
yapi.js
media
manifestfile
fragments
decrypt
![Page 39: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/39.jpg)
streaming
adaptive
protection / EME
mediaElement / MSE
client
yapi.js
![Page 40: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/40.jpg)
BETWEEN MODULES1. hold reference
problems
• dependency
streaming adaptive
![Page 41: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/41.jpg)
BETWEEN MODULES2. a parent module
problems
• defining ‘parent’
• multiple parent modules?
streaming
adaptive
parent
parent?
![Page 42: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/42.jpg)
BETWEEN MODULES3. event system
• modules are parallel
• semantic event name indicates current ‘state’ of player
streaming
adaptive
protection / EME
mediaElement / MSE
eventBus
![Page 43: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/43.jpg)
streaming
adaptive
MSECtrl
eventBus
eventBus
eventBus
eventBus
![Page 44: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/44.jpg)
function downloadFragment() {}streaming
adaptive
eventBus
eventBus
MSECtrl eventBus
function calculateQuality() {}
function appendToSourceBuffer() {}
function onDownloaded() { // notify FRAGMENT_DOWNLOADED }
function onAppended() { // notify FRAGMENT_APPENDED }
![Page 45: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/45.jpg)
register
eventBus.mapHandler(‘FRAGMENT_DOWNLOADED’, )adaptive. calculateQuality
eventBus.mapHandler(‘FRAGMENT_DOWNLOADED’, )MSECtrl. appendToSourceBuffer
eventBus.mapHandler(‘FRAGMENT_APPENDED’, )streaming. downloadFragment
![Page 46: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/46.jpg)
{ }
FRAGMENT_DOWNLOADED: [ , ],
adaptive. calculateQuality
MSECtrl. appendToSourceBuffer
FRAGMENT_APPENDED: [ ]streaming. downloadFragment
eventBus
![Page 47: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/47.jpg)
function downloadNextFragment() {}streaming
adaptive
eventBus
eventBus
MSECtrl eventBus
function calculateQuality() {}
function appendToSourceBuffer() {}
function onDownloaded() { eventBus.dispatch(‘FRAGMENT_DOWNLOADED’) }
function onAppended() { // notify FRAGMENT_APPENDED }
dispatch event
![Page 48: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/48.jpg)
eventBus
{ }
FRAGMENT_DOWNLOADED: [ , ],
adaptive. calculateQuality
MSECtrl. appendToSourceBuffer
FRAGMENT_APPENDED: [ ]streaming. downloadFragment
invoke callbacks
![Page 49: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/49.jpg)
function downloadNextFragment() {}streaming
adaptive
eventBus
eventBus
MSECtrl eventBus
function calculateQuality() {}
function appendToSourceBuffer() {}
function onDownloaded() { eventBus.dispatch(‘FRAGMENT_DOWNLOADED’) }
function onAppended() { eventBus.dispatch(‘FRAGMENT_APPENDED’) }
dispatch event
![Page 50: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/50.jpg)
eventBus
{ }
FRAGMENT_DOWNLOADED: [ , ],
adaptive. calculateQuality
MSECtrl. appendToSourceBuffer
FRAGMENT_APPENDED: [ ]streaming. downloadFragment
invoke callbacks
![Page 51: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/51.jpg)
works with app level
yapi.js
app
ui
client
api
![Page 52: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/52.jpg)
![Page 53: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/53.jpg)
works with app level
yapi.js
app
ui
client
events
api
![Page 54: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/54.jpg)
MEDIA EVENTS
http://www.w3.org/2010/05/video/mediaevents.html
Dispatched by HTML5 media element, notifying playback process
![Page 55: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/55.jpg)
YAPI PROXY/PATCH EVENTS
Media Element event event with same name
yapi.js
![Page 56: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/56.jpg)
USEFUL MEDIA EVENTS
loadstart: Indicate loading media begins, this fires when setting src attribute
loadedmetadata: while element has basic info of playback, e.g duration
timeupdate: while current time of playback is ticking
seeking/seeked: while conducting seek
ended: playback ends
play/playing: playback resume from other status to playing
![Page 57: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/57.jpg)
ADDITIONAL EVENTSloadedmanifest: after manifest is loaded/parsed
bitratechanged: when bitrate is changed
enableabr: when adaptive activation changed
buffering: when playback pending
cuechanged: webvtt subtitle cue changed (in and out)
![Page 58: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/58.jpg)
yapi.load(MANIFEST_URL); // exposed api
yapi.addEventListener(‘playing’, onPlaying);
function onPlaying() { // app logic // or ui reaction, e.g showing playing status ui }
// onPlaying invoked when playing
app yapi ui
![Page 59: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/59.jpg)
LESSONS
![Page 60: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/60.jpg)
1. test case helps
![Page 61: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/61.jpg)
2. handle errors
![Page 62: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/62.jpg)
3. boss wants ‘numbers’
![Page 63: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/63.jpg)
streaming
adaptive
protection / EME
mediaElement / MSE
client
yapi.js
statisticplaying smoothly is not
enough
you need to tell how good it is
![Page 64: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/64.jpg)
![Page 66: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/66.jpg)
mse spec: https://goo.gl/L3HPKP media event sample: https://goo.gl/BsrH18MDN media element: https://goo.gl/JsojcN HTML MediaElement doc: https://goo.gl/Y3lbO0dash.js: https://goo.gl/xAm2Ve
![Page 67: yapi.js introduction (mopcon 2016 version)](https://reader034.vdocuments.net/reader034/viewer/2022042611/588232001a28ab52368b7115/html5/thumbnails/67.jpg)
THANK YOUKKTV is hiring
web / android / iOS / backend developers