Opinionated containers and the

future of game servers

1

2

• Brendan Fosberry

• Software Engineer

• Shopkeep POS

• Core Services Team

3

• Background

• Containerizing Applications

• Containerizing a Simple Application

• Containerizing a Complex Application

• The Future

Overview

Opinionated Applications

4

• IPad POS Software and Hardware

• Web based reporting and customization

• 24/7 Support

Background

5

Background

• Founded in 2008

• 180 employees

• 4 Development teams + Devops

• Rails, Objective C, Golang

• 11k active registers

• 100k transaction per hour at peak

• 700k transactions per day

6

Background

Redesign

Microservices

• Transaction Service (Riak)

• Reporting Service

• Authentication Service

• API Service

• Etc..

Docker Stack

• Apps

• Docker CI Pipeline

• Routing

• Monitoring

PB over RPC

7

Background

Dockerfile

FROM debian:jessie

RUN mkdir -p /opt/go

ADD ./core-api /opt/go/core-api

VOLUME /var/log/ /opt/go/logs/

EXPOSE 8080

ENTRYPOINT ["/opt/go/core-api"]

Procfile

web: core-api

Containers

8

Containerizing Applications

Best Practices

Based on Dockerfile best practices

Applicable for in-house/black box applications

Assumes working Dockerfile

Mostly common sense

9

Containerizing Applications

Best Practices

1. File-less configuration• Load configuration from environment

• Use Fracker to write environment file

• Pass environment file to container

• Otherwise use confd

• Avoid chef

• Disconnects container and application configuration

10

Containerizing Applications

Best Practices

1. File-less configuration

2. Separate tasks• Container = 1 instance of 1 application task

• Separate web heads, crons, job runners

• Makes resource allocation simpler

11

Containerizing Applications

Best Practices

1. File-less configuration

2. Separate tasks

• Assume container is ephemeral

• Log to stdout

• Log to mounted volume

3. Log to sink

12

Containerizing Applications

Best Practices

1. File-less configuration

2. Separate tasks

• Assume container is disposable

• Externalize persisted state

• External db or mounted volume

3. Log to sink

4. Separate datastores

13

Containerizing Applications

Best Practices

• Use application methods if possible

• Dump to simple sink

• Don’t juggle multiple tasks

4. Separate datastores

5. Externalize data extraction

1. File-less configuration

2. Separate tasks

3. Log to sink

14

Containerizing Applications

Best Practices

• Control process via container

• Send signals with docker

• Control through custom ports

4. Separate datastores

5. Externalize data extraction

6. Don’t daemonize

1. File-less configuration

2. Separate tasks

3. Log to sink

15

Containerizing Applications

Why Game Servers?

• Each server is unique

• Poor existing automation

• Poor open source tooling

• Daniel Gibbs

• Poor isolation

• Why not?

16

Containerizing Applications

Game Servers

Containerization Levels Goals

1. Globally Deployable

2. Uniform Configuration

3. Disposable

1. Container Image

2. Docker

3. Clustered with Fleet

17

Simple Application

Simple Application

Teamspeak

Gaming Infrastructure

Black box, client-server

Clients negotiate state changes

Server dispatches voice streams

Simple setup - download and run

18

Best Practices

1. File-less configuration

• Requires configuration files

• Use confd

Simple Application

19

Best Practices

1. File-less configuration

2. Separate tasks

• Single Task

• No background Jobs

Simple Application

20

Best Practices

1. File-less configuration

2. Separate tasks

• Configure log sink

• Mount volume to host

3. Log to sink

Simple Application

21

Best Practices

• Mount volume for sqlite DB

• Use external DB

• Write db config file

4. Separate datastores

Simple Application

1. File-less configuration

2. Separate tasks

3. Log to sink

22

Best Practices

• Can parse docker logs

• Or mount logging volume

• Use secondary container to parse token

• Use MachineOf

4. Separate datastores

5. Externalize data extraction

Simple Application

1. File-less configuration

2. Separate tasks

3. Log to sink

23

Best Practices

• Run in foreground

4. Separate datastores

5. Externalize data extraction

6. Don’t daemonize

Simple Application

1. File-less configuration

2. Separate tasks

3. Log to sink

24

Results

Simple Application

• No state within container

• If host dies logs are lost

• Single DB per container

• Connect using mapped port

• Runs on fleet

25

Simple Application

Teamspeak

Containerization Levels Goals

1. Globally Deployable

2. Uniform Configuration

3. Disposable

1. Container Image

2. Docker

3. Clustered with Fleet

26

Complex Application

Complex Application

Team Fortress 2

Game Server

Black box, client-server

Clients negotiate state changes

Server dispatches voice streams

Server dispatches state changes

27

Installing Tf2

#update_script

@NoPromptForPassword 1

@ShutdownOnFailedCommand 1

login anonymous

force_install_dir /opt/server/

app_update 4020 validate

quit

Complex Application

#!/bin/bash

install_steamcmd.sh

steamcmd.sh +runscript ./update_script

28

Running Tf2

./srcds_run -game tf2 +map “ctf_2fort”

+maxplayers 24 -ip 0.0.0.0 -port 27015

Complex Application

Great opportunity for extraction

• Write simple server.cfg

• Run srcds_run

• Open/forward ports

29

Best Practices

1. File-less configuration• Requires configuration files

• Use confd

Complex Application

30

Best Practices

1. File-less configuration

2. Separate tasks

• Single Task

• No background Jobs

Complex Application

31

Best Practices

1. File-less configuration

2. Separate tasks

• Configure log sink

• Mount volume to host

3. Log to sink

Complex Application

32

Best Practices

• No persisted state

• External DB where mods need it

4. Separate datastores

Complex Application

1. File-less configuration

2. Separate tasks

3. Log to sink

33

Best Practices

• Don’t need any data

• Monitoring

4. Separate datastores

5. Externalize data extraction

Complex Application

1. File-less configuration

2. Separate tasks

3. Log to sink

34

Best Practices

• Run in foreground

4. Separate datastores

5. Externalize data extraction

6. Don’t daemonize

Complex Application

1. File-less configuration

2. Separate tasks

3. Log to sink

35

Results

Complex Application

No state within container.

If host dies logs are lost.

Connect using mapped port.

8Gb is large for an image.

36

Port Mapping

Complex Application

Source servers report ip and port.

Steam master list references static ports.

Favouriting references static port.

Direct connection works.

37

Complex Application

Port RemappingRemap the traffic to the correct port

• External remapper (inbound proxy)

• Cluster/Host-level dynamic remapper

• Container-level dynamic remapper

• Static port bindings - requires port allocator

We can’t control the return journey

38

Host

49168 49170Container

49168

Complex Application

Tf2

49168

Client/Master

Server

HostIP:49168

Port Remapping

Container

27015

Proxy

27015 -> HostIP:49170

HostIP:49168

Tf2 becomes dependent upon Proxy

39

Host

49168Container

27015

Complex Application

Tf2

49168

Client/Master

Server

HostIP:49168

Port Mapping

HostIP:49168 Remapper

27015 -> 49168

Fewer containers, but multipurpose

40

Complex Application

Tcp/Udp Mapping

Source Engine Server binds to both tcp and udp.

Can’t separate port bindings.

Docker may map to different port numbers.

Open PR to allow EXPOSE 12345/tcpudp

41

Complex Application

Favouriting

A favourite is an IP + Port.

Invalidated by container changing Host/Port.

Remapping containers allow port changes.

No simple solution for host changes.

1. Container Image

2. Docker

3. Clustered with Fleet

42

Complex Application

Tf2

Containerization Levels Goals

1. Globally Deployable

2. Uniform Configuration

3. Disposable

43

Complex Application

Game Server Ecosystem

Persisted state is critical.

Many servers are windows only.

Some assets/code are proprietary.

Limited configuration options.

Hackers promote constrictive environment.

Obscured docs, varied user base.

44

The Future

The future of game servers

Apply lessons from services to increase compatibility

• Game Server Containers are becoming

commonplace

• Standardize configuration methods

• Allow persisted state to be externalized

• Dedicated linux

• Support clustering

45

Summary

Summary

• Containerizing opinionated apps is easy

• Deploying an opinionated container may not be

• Container deployment will become increasingly relevant

• Container compatibility will become increasingly defining

• Game servers have a lot of work to do

46

Summary

Questions?

brendan@shopkeep.com

@brendanfosberry

bfosberry

Fozz

Thank You.

47