a 7 architecture sustaining line live

What is LINE LIVE?

Video streaming and hosting service

Watch streams from celebrities and companies

Host or watch streams from other users

Chat communication

Showing appreciation with Loves

History of LINE LIVE Feb 2014 LINE LIVE CAST was released


Sep 2015 Kick off meeting for LINE LIVE



Dec 2015 LINE LIVE was released



Dec 2015 LINE LIVE was released

Aug 2016 LINE LIVE Version 2 is out

Tools and Staff

Java, Nginx, MySQL, Redis, Elasticsearch, Fluentd, CDN, etc.

Server (media 260+, chat 110+, API and other 100+)

Over 20 core engineers

Over 30 other related engineers (LINE app, storage, network, etc.)

LINE LIVE Architecture

OBS I nhouse S t o r age

Sys t em

Hosting

Viewing

RTMP Media Servers

API Servers

LINE

Billing

CMS

Chat

WebSocke t

JSON API

CDN Akama i

LINE Servers Upload HLS F i les

Cache

S e n d O p e r a t i o n O n J S O N A P I

S e n d E v e n t s O n J S O N A P I

Relationship with LINE

Data Analysis

For deta i ls , see the presenta t ion,

New Stream Processing Pla t form with Apache Fl ink"

Data Labs

Real-time viewer aggregation

Streaming program KPI measurement, etc.

Video Stream Delivery

HLS (HTTP Live Streaming)

HLS

l ive-1_001.ts

l ive-1_002.ts

l ive-1_003.ts

l ive-1_004.ts

l ive-1_005.ts

(HTTP Live Streaming)

HLS

All compiled into one video

l ive-1_001.ts

l ive-1_002.ts

l ive-1_003.ts

l ive-1_004.ts

l ive-1_005.ts


HLS

l ive-1.m3u8

l ive-1_001.ts l ive-1_002.ts l ive-1_003.ts l ive-1_004.ts l ive-1_005.ts . . .

l ive-1_001.ts

l ive-1_002.ts

l ive-1_003.ts

l ive-1_004.ts

l ive-1_005.ts


HLS

l ive-1.m3u8

l ive-1_001.ts l ive-1_002.ts l ive-1_003.ts l ive-1_004.ts l ive-1_005.ts . . .

l ive-1_001.ts

l ive-1_002.ts

l ive-1_003.ts

l ive-1_004.ts

l ive-1_005.ts


CDN

HLS l ive-1_720.m3u8

l i v e -1_720_ 00 1 . t s l i v e -1_720_ 00 2 . t s l i v e -1_720_ 00 3 . t s . . .

l ive-1.m3u8

l i v e -1_720 . m3 u8 l i v e -1_480 . m3 u8 l i v e -1_360 . m3 u8 . . .

l ive-1_480.m3u8


l ive-1_360.m3u8


l i v e -1_720 _0 0 1 . t s

l i ve -1_720 _0 0 2 . t s

l i ve -1_720 _0 0 3 . t s

l i ve -1_480 _0 0 1 . t s

l i ve -1_480 _0 0 2 . t s

l i ve -1_480 _0 0 3 . t s

l i ve -1_360 _0 0 1 . t s

l i ve -1_360 _0 0 2 . t s

l i ve -1_360 _0 0 3 . t s

Architecture

Media Servers

API Servers

OBS Inhouse Storage System

CDN Akamai

Hosting Viewing

RTMP

Architecture

Media Servers

API Servers


CDN Akamai

Hosting Viewing

RTMP

Upload HLS Fi les

Architecture

Media Servers

API Servers


CDN Akamai

Hosting Viewing

Send Events On JSON API

RTMP

Upload HLS Fi les

Architecture

Media Servers

API Servers


CDN Akamai

Hosting Viewing


RTMP

Send a

Push

Upload HLS Fi les

Architecture

Media Servers

API Servers


CDN Akamai

Hosting Viewing


Cache Download HLS Fi les

RTMP JSON API

Upload HLS Fi les

HLS Good Hack on LINE LIVE

HLS Good Hack for Data LIVE

http:///{encrypt(objectId, startAt, durationSec)}_720_xxx.ts

http:///{encrypt(objectId, startAt, durationSec)}.m3u8

http:///FooBar_720_xxx.ts

http:///FooBar.m3u8

Example : s tar tAt=19:00 dura t ionSec=3600

Prepare encrypted URL on server in advance

HLS Good Hack for Data LIVE

GET 200 GET 200

18:00 21:00 19:00 20:00

http:///FooBar.m3u8


HLS Good Hack for Data LIVE Example : s tar tAt=19:00 dura t ionSec=3600

GET 404 GET 404 GET 404 GET 404

18:00 21:00 19:00 20:00

http:///FooBar.m3u8


HLS Good Hack for Auto play

http:///{encrypt(objectId, startAt, durationSec)}.m3u8


Prepare encrypted URL on server in advance

Example : s tar tAt=2sec dura t ionSec=6

001. ts 002. ts 003. ts 004. ts 005. ts xxx. ts

HLS Good Hack for Auto play Example : s tar tAt=2sec dura t ionSec=6

001. ts 002. ts 003. ts 004. ts 005. ts xxx. ts

GET 200 GET GET GET

http:///AcmeAcme.m3u8

foo-bar.m3u8 002 . t s 003 . t s 004 . t s

Flow When Stream Is Terminated

Media Servers

API Servers


CDN Akamai

Hosting Viewing

RTMP

Upload HLS Fi les

Stream Terminated

Media Servers

API Servers


CDN Akamai

Hosting Viewing

RTMP

Upload HLS Fi les

Stream Terminated

Media Servers

API Servers


CDN Akamai

Hosting Viewing

2. Cache

RTMP

Upload HLS Fi les

1 . GET HLS 2. Fetch Or ig in

Stream Terminated

Media Servers

API Servers


CDN Akamai

Hosting Viewing

3. 404 4. 404

RTMP

Upload HLS Fi les


Standby Until Recovery

Media Servers

API Servers


CDN Akamai

Hosting Viewing

3. 404 4. 404

RTMP

Upload HLS Fi les

Burst API Server

6. SUSPEND

5. Pol l ing

SUSPEND Event


Recovery Flow

Media Servers

API Servers


CDN Akamai

Hosting Viewing

1. RTMP restar t

Upload HLS Fi les

Burst API Server

2. RESTART event

Recovery Flow

Media Servers

API Servers


CDN Akamai

Hosting Viewing

1. RTMP restar t

Upload HLS Fi les

Burst API Server

4. RESTART

3. Pol l ing

2. RESTART event

Recovery Flow

Media Servers

API Servers


CDN Akamai

Hosting Viewing

Cache 5. Download HLS Fi les

1 . RTMP restar t

Upload HLS Fi les

Burst API Server

4. RESTART

3. Pol l ing

2. RESTART event

Improved Hosting Performance GPU CPU

720p 1080p

220%

267%

Chat Overview CMS

Hosting Viewing

WebSocket

CHAT

API Servers

JSON API

JSON API

JSON API WebSocket

Chat Overview Server 1 Server 2 Server 3

Room 1 Room 1

Redis Cluster

Publ ish/Subscr ibe

Chat Overview Server 1 Server 2 Server 3

Room 1 Room 1

Room 2 Room 2

Redis Cluster

Room 2

Log Archive

l ive-1.chatlog

l ive-1_001.json l ive-1_002.json l ive-1_003.json l ive-1_004.json l ive-1_005.json . . .

l ive-1_001.json

l ive-1_002.json

l ive-1_003.json

l ive-1_004.json

l ive-1_005.json

Upload to the OBS

Issues

Issues

Over 10,000 messages per minute

LOVE

Love-mashing is fun "Love-mashing is communication

Love

Cl ients Store ( Increment ) F lush

Burst API

Redis Cluster

MySQL

Clients

- ranking (ZSET) - Tota l count (NUM)

Data ( log)

Using Lua Scr ip t

Love Total Ranking

Get

API Redis Cluster

Clients

Using ZREVRANGE

Love Total Ranking

Get

(Arch ive)

API Redis Cluster

Clients

MySQL

Using ZREVRANGE

Example 1

11 million Loves in 120 minutes

Example 2

2.3 million Loves in 30 minutes

Chat and Love Issues Are Also Explained on our Blog

http://developers.linecorp.com/blog/ja/?p=3653

http://developers.linecorp.com/blog/ja/?p=3771

Summary

Provided overview of LINE LIVE

Talked about video streaming methods

Talked about implementation of traffic-inviting chat and Love features

Any Questions?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

a 7 architecture sustaining line live

Technology