GamingAnywhere: An Open-Source Cloud Gaming Testbed Chun-Ying Huang, De-Yu Chen, Cheng-Hsin Hsu, and Kuan-Ta Chen

ACM Multimedia 2013 OSS Competition, Barcelona, Spain

1

Cloud Gaming is Hot • Cloud gaming is expected to lead the future growth of

computer games: 9 times in 6 years [CGR]

[CGR] http://www.cgconfusa.com/report/documents/Content-5minCloudGamingReportHighlights.pdf

T5-Labs

2

What is Cloud Gaming

3

Real-time game playing using light-weight clients

Selling Points of Cloud Gaming • Gamers’ perspectives: • Frees gamers from indefinitely upgrading their computers

• Eliminates setup overhead and compatibility issues for trying out a game, as everything needed will be settled in data centers by game operators

• Enables gamers to play games anywhere, anytime

• Game manufacturers’ perspectives: • Allows developers to support more platforms

• Reduces the production cost

• Prevents pirating

4

Limitations of Existing Services • OnLive demands for 5 Mbps for reasonable quality • OnLive dictates a backbone latency of 22 ms, and partially

copes with it by setting up 3 data centers (CA, VA, TX) • Only people who live in 1000 mile radius from a data center can play

the game • and more…

• We, researchers, have tons of ideas to improve cloud gaming services, but all cloud gaming systems are proprietary and closed

5

Solution

• GamingAnywhere is the first cloud gaming platform for researchers, service providers, and users

6

Design Objectives • Extensibility

• Components can be implemented as modules • Features can be easily expanded

• Portability

• Cross-platform ready

• Configurability • Exposes as many configurations as possible

• Openness • It has been released to the public since April 2013

7

Cloud Gaming Scenario

8

System Architecture • The client and the server, with many components • Implemented by leveraging open-source packages

Game console

Data Flow

Control Flow

Running the selected game

Age

nt

Proc

ess/

Thre

ad

Game Server Game Client

Internet

Audio / VideoEncoder

RTSP / RTP / RTCP

Audio / VideoCapturerReplay User Inputs

(Keyboard, Mouse, ...)

Decode Input Events(Customized Protocol)

RTSP / RTP / RTCP

Audio / VideoDecoder

Encode Input Events(Customized Protocol)

Audio / VideoPlayer User Inputs

(Keyboard, Mouse, ...)

9

Implementation Overview • GamingAnaywhere core is implemented as a library

• With several modules and executable • Server components

• Video source (modularized, platform dependent) • Audio source (modularized, platform dependent) • Encoders (modularized, currently w/ ffmpeg) • Replayer (modularized, platform dependent) • RTSP/RTP server (in library, w/ ffmpeg)

• Client components • RTSP/RTP client (using live555) • Decoders (in library, w/ ffmpeg) • Controller (using SDL) • Renderer (using SDL)

10

License • GamingAnywhere adopts 3-clause BSD license

• Less restrictive than GPL

• Libraries used by GamingAnywhere

11

Library License Library License ffmpeg LGPL zlib zlib glibc LGPL lame LGPL libasound2 LGPL libvpx 3-clause BSD libSDL zlib opus 3-clause BSD libX11 LGPL x264 GPLv2 live555 LGPL libva MIT

GamingAnywhere Screenshots

12

Server

Client #1

Client #2

Client #3

Performance Evaluation • Compare against OnLive and StreamMyGame • Environment setup • Sample results are presented

14

Response Delay Measurement – For Closed Systems (OnLive and SMG)

0.1 sec 0.2 sec 0.3 sec

Screen Response delay

Response delay (RD) = Network delay (ND) + Processing delay (PD) + Playout delay (OD)

• Kuan-Ta Chen, Yu-Chun Chang, Po-Han Tseng, Chun-Ying Huang, and Chin-Laung Lei, “Measuring the Latency of Cloud Gaming Systems,” ACM Multimedia 2011

15

t0 (Key event sent) t1 (Key event received)

t2 (Frame sent) t3 (Frame received)

Server Client

Menu frame Menu screen shown

t4 (Frame displayed)

• Network delay (ND) : network RTT • Processing delay (PD) : t2 – t1 t3 – t0 – ND

• Playout delay (OD) : t4 – t3

16

GA Has The Lowest Response Delay

17

With GA: For a strict 100 ms RD requirement, ND can be as long as 52 ms

From Barcelona to Greenland (~4000 km)

18

http://gaminganywhere.org/

• In 6 months • Web: 16,103 visits, 9,689 unique visitors

• Forum: 53 topics, 178 posts

Visitor Distribution

19

Conclusion • GamingAnywhere is the first open cloud gaming platform

• May be used by researchers, engineers, and gamers

• We hope it will stimulate more studies on cloud gaming,

• Codecs for cloud gaming • Usability study for cloud gaming interfaces • Cloud gaming in heterogeneous networks • 3D/Stereoscopic cloud gaming • … and many more!

• We need your participation!

20

QUESTIONS? Join us at http://gaminganwhere.org

21

BACKUP

22

Server Overview

23

Audiosource

Desktop/Game

Videosource

Videoencoder

Audioencoder

Audiobuffer

Videobuffer

(1a)audio capture

(1v)video

capture

Threads

Shared buffers

(2a)write audio

frames

(2v)write a video frame

(3a)wake upencoder

(3v)wake upencoder

(4a)read audio

frames

(4v)read a video frame

(5a)encode andsend

(5v)encode and

send

Object owner

RTSP server thread

Data Flow Connections (RTSP/RTP/RTCP)

(1n)handleclients

Inputreplayer

Control Flow Connections

(2i)replayinputevents

(1i)receiveinputevents

Process Video Frames w/ a Shared Buffer • A shared buffer for a single screenshot

• The writer could be blocked by readers

24

Process Video Frames • Process a video frame

1. Video source: Capture a frame 2. Converter: Convert from RGB format to YUV format 3. Encoder: encoding and packetization

25

Process Video Frames in Parallel • Suppose the targeted inter-frame delay is ∆t • The response delay may greater than ∆t

• frame capture + color space conversion + encoding

• It could degrade encoding bitrate • Process in parallel

26

F1 F2

F1

F1

F2

F2

F3

F3

F3

Video source(frame capture)

t

Video encoder

F4

Color space converter(RGB to YUV)

t+∆ t t+2∆ t t+3∆ t

Process Audio Frames • Each audio encoder has an audio source buffer

• A queue for captured audio frames

27

Client Overview

28

Game Interaction

Mainthread

Control Flow Connections

(1i)receiveinputevents

(2i)sendinputevents

SDL Rendering Input Events

Data Flow Connections

RTSP

clie

nt th

read

Videobuffer

Audiobuffer

Threads

Buffers

Object owner

(1r)receiveencodedA/Vframes

(2rv)buffer

an encoded

audio frames

(2ra)bufferencodedaudio frames

(3rv)decode and render

video frames

(3ra)decode and renderaudio frames(callback)

Video Playout Buffering • The 1-frame buffering strategy

• Based on the RTP marker bit • An H.264 frame can be split into different numbers of packets • The marker bit (with a value of 1) indicates the last packet of a

frame • An example: 30KB frame transmitted in a 50Mbps network would have

a buffering time of 30KB*8-bit/50Mbps =~ 5ms

29

M M M M M

RTP Packets

Frame #1 Frame #2 Frame #3 Frame #4 Frame #5

time

Performance Evaluation (Cont’d) • Audio: The LAME encoder • Video: The x264 encoder

• One of the VideoLan projects

• With the below default parameters • $r is the bitrate variable, default 3Mbps

--profile main --preset faster --tune zerolatency --bitrate $r --ref 1 --me dia --merange 16 --intra-refresh --keyint 48 --sliced-threads --slices 4 --threads 4 --input-res 1280x720

30

Delay Decomposition of GA • Processing delay

• Memory lock + copy • RGB to YUV • x264 encoder • Packetization (RTP packets)

• Playout delay • 1-frame buffering • S/W decoder • SDL rendering

31

GA Incurs Low to Moderate Network Loads

• Configured to capture at 50 fps with a bitrate of 3 Mbps

32

GA Provides Superior Video Quality

33