Stackless Python in EVE

Kristján Valur Jónsson

kristjan@ccpgames.com

CCP Games inc.

EVE

• MMORPG Space game

• Client / server

• Single shard massive server

• 120.000 active players, >24.000 concurrent users

• World concurrency record on a shard

• Relies on Stackless Python

The Tranquility cluster• 400 GHz CPU / 200 Gb

RAM• 2 Routers (CISCO Alteon)• 14 Proxy servers (IBM

Blade) • 55 Sol servers (IBM x335)• 2 DB servers (clustered,

IBM Brick x445)• FastT600 Fiber, 56 x FC

15k disks, DS4300 + 3*EXP700

• Windows 2000, MS SQL Server

• Currently being upgraded• AMD x64

EVE Architecture

• COM-like basic architecture

• Python tighly integrated at an early stage

• Home-grown wrapping of BLUE objects

python modulesblue modules

ExeFile.exe

Netclient.dll

Trinity.dll

...

blue.dll

Pythonlib.libautoexec.py

...

bar.py

foo.py

Stackless Python

• Tasklets

• Threads of execution. Not OS threads

• Lightweight

• No pre-emption

• Channels

• Tasklet rendezvous point

• Data passing

• Scheduling

• Synchronization

Stackless?

• No C stack

• Python stack in linked

frame objects

• Tasklet switching by

swapping frame chain

• Compromise

• stackless where

possible.

• C stack whisked away

if necessary

main()

PyStackless_CallMethod_Main(“MyMain”)

def MyMain(self): cmodule.Frobnicate(self.Callback)

PyFrobnicate() { PyObject_CallObject(cb, …); }

def Callback(self): self.channel.send(42)

Stack switch

Channels

Channel

Readers Writers

Tasklet 1

Tasklet 2

Channel semantics

• Send on a channel with no receiver blocks tasklet.

• Send on a channel with a (blocked) receiver,

suspends tasklet and runs receiver immediately.

Sender runs again in due course.

• Symmetric wrt. Send and Receive.

• “balance”, can have a queue of readers or writers.

• Conceptually similar to Unix pipes

Channel semantics, cont.

• Scheduling semantics are precise:

• A blocked tasklet is run immediately

• Usable as a building block:

• semaphores

• mutex

• critical section

• condition variables

Stackless in EVE

• BLUE foundation: robust, but cumbersome

• RAD

• Stackless Python: Python and so much more

• EVE is inconceivable without Stackless

• Everyone is a programmer

The main loop

• Establish stackless context

int WinMain(...) {

PyObject *myApp = new EveApp();

PyObject *r = PyStackless_CallMethod_Main(MyApp,

“WinMain”, 0);

return PyInt_AsLong( r );

PyObject* EveApp::WinMain(PyObject *self, PyObject *args) { PyOS->ExecFile("script:/sys/autoexec.py"); MSG msg; while(PeekMessage(&msg, 0, 0, 0, PM_REMOVE)){

TranslateMessage(&msg);DispatchMessage(&msg);

} for (TickIt i = mTickers.begin(; i !=

mTickers.end(); i++)i->mCb->OnTick(mTime, (void*)taskname);

}

• Regular Windows message loop• Runs in Stackless context• The “Main Tasklet”

The main loop cont.

Autoexec.py

import blue

def Startup(): import service srvMng = service.ServiceManager() run = ["dataconfig", "godma", “ui", …] srvMng.Run(run)

#Start up the client in a tasklet!if CheckDXVersion(): import blue blue.pyos.CreateTasklet(Startup, (), {})

Tickers

• Tickers are BLUE modules:

• Trinity (the renderer)

• Netclient

• DB (on the server)

• Audio

• PyOS (special python services)

• …

The PyOS tick:

• Runs fresh tasklets

• (sleepers awoken elsewhere)

Tick() {

…

mSynchro->Tick() PyObject *watchdogResult; do {

watchdogResult = PyStackless_RunWatchdog(20000000);

if (!watchdogResult)PyFlushError("PumpPython::Watchdog");

Py_XDECREF(watchdogresult); } while (!watchdogResult);

blue.pyos.synchro

• Synchro:

• Provides Thread-like tasklet utilities:

• Sleep(ms)

• Yield()

• BeNice()

1. Sleep: A python script makes the call

blue.pyos.Sleep(200)

2. C++ code runs:

1. Main tasklet check

2. sleeper = New Sleeper();

mSleepers.insert(sleeper);

PyObject *r =

PyChannel_Receive(sleeper-

>mChannel);

3. Another tasklet runs

blue.pyos.synchro cont.

blue.pyos.synchro, ctd.4. Main tasklet in windows loop enters PyOS::Tick()5. mSleepers are examined for all that are due we

do:mSleepers.remove(sleeper);PyChannel_Send(sleepers.mChannel,

Py_NONE);6. Main tasklet is suspended (but runnable), sleeper

runs.

Points to note:

• A tasklet goes to sleep by calling

PyChannel_Receive() on a channel which has

no pending sender.

• It will sleep there (block) until someone sends

• Typically the main tasklet does this, doing

PyChannel_Send() on a channel with a reader

• Ergo: The main tasklet may not block

Socket Receive

• Use Windows asynchronous file API

• Provide a synchronous python API. A python script

calls Read().

• Tasklet may be blocked for a long time, (many

frames) other tasklets continue running.

• Do this using channels.

Receive, cont.

1. Python script runs:

foo, bar = socket.Read()

2. C code executes the request:

Request *r = new Request(this);WSAReceive(mSocket, …);mServe->insert( r );PyChannel_Receive(r->mChannel);

3. Tasklet is suspended

Receive, cont.

4. Socket server is ticked from main loop

5. For all requests that are marked completed, it

transfers the data to the sleeping tasklets:

PyObject *r =

PyString_FromStringAndSize(req->mData,

req->mDataLen);

PyChannel_Send(req->mChannel, r);

Py_DECREF(data);

delete req;

4. The sleeping tasklet wakes up, main tasklet is

suspended (but runnable)

Receive completed

EveApp::WinMain()

Netclient::OnTick()

PyChannel_Send(mReceiver, data);

Stack switch

Cmodule::Work(PyObject *cb)

def TaskletMain():

Stack switch

def ProcessStuff:

def Callback():

Socket::Receive(PyObject *cb)

d = PyChannel_Receive(mReceiver);

Main Tasklet• The one running the windows loop• Can be suspended, allowing other tasklets to run• Can be blocked, as long as there is another tasklet to

unblock it (dangerous)• Is responsible for waking up Sleepers, Yielders, IO

tasklets, etc. therefore cannot be one of them• Is flagged as non-blockable

(stackless.get_current().block_trap = True)

Channel magic• Channels perform the stackless context switch.• If there is a C stack in the call chain, it will magically

swap the stacks.• Your entire C stack (with C and python invocations)

is whisked away and stored, to be replaced with a new one.

• This allows stackless to simulate cooperative multi-threading

Co-operative multitasking

• Context is switched only at known points.

• In Stakcless, this is channel.send() and

channel.receive()

• Also synchro.Yield(), synchro.Sleep(), BeNice(),

socket and DB ops, etc.

• No unexpected context switches

• Almost no race conditions

• Program like you are single-threaded

• Very few exceptions.

• This extends to C state too!

Tasklets• Tasklets are cheap• Used liberally to reduce perceived lag

• UI events forked out to tasklets

• A click can have heavy consequences.• Heavy logic• DB Access• Networks access

• special rendering tasks forked out to tasklets.• controlling an audio track

• “tasklet it out”• Use blue.pyos.synchro.BeNice() in large loops

Example: UI Event:

• Main tasklet receives window messages such as

WM_CLICK

• Trinity invokes handler on UI elements or global

handler

• Handler “tasklets out” any action to allow main

thread to continue immediately.

def OnGlobalUp(self, *args): if not self or self.destroyed: return mo = eve.triapp.uilib.mouseOver if mo in self.children: uthread.new(mo._OnClick) class Action(xtriui.QuickDeco): def _OnClick(self, *args): pass

That’s all

• For more info:

• http://www.ccpgames.com

• http://www.eve-online.com

• http://www.stackless.com

• kristjan@ccpgames.com