DevelopingThe Bachelor

What is The Bachelor?

1. The showReality dating game show

2. CompetitionContestants compete to be selected through elimination rounds.

3. Limited ParticipantsOnly 25 participants in the TV show; a Facebook game lets theaudience play too!

The Development Team

Nicholas Asch

Alice Bevan-McGregor

Nicolas Cadou

Blaise Laflamme

Egor Miadzvedeu

Zachary Allatt

Infrastructure

Libraries

Main libsPyramid

MongoEngine

Mako

zc.buildout

nose and lettuce

AlsoSCSS

futures

apscheduler

marrow.mailer

Architecture

Run of the mill MVC-ishControllerPyramid Routes + Views

ModelCore API + Data Access Layer

ViewViews + Templates

Architecture

Thin views

@view_config(route_name='audition_enter', http_cache=0, renderer='bachelor:templates/show/audition/enter.mako')def audition_enter(self): return dict(show=self._show)

@view_config(route_name='audition_enter_confirm', request_method='POST', xhr=True, renderer='json', http_cache=0)def audition_ajax(self): if self.request.POST.get('photo'): photo = json.loads(self.request.POST.get('photo'))['url'] else: photo = None

self.show_api.audition(self._show, picture=photo)

return dict(next=self.request.route_path('audition_list'))

Architecture

Fat views (yuk!)

@view_config(route_name='event_list', http_cache=0, renderer='bachelor:templates/event/list.mako')def list_(self): """ Return a list of events, customized for the user. """ page = int(self.request.params.get('page', 1)) reward = Tuneable.get_event_gain(self.user_api.level(self.user)) cost = Tuneable.get_topic_cost(self.user_api.level(self.user)) score = self.user_api.get_score() can_participate = cost.validate(score) details = cost.validate(score, True) notifications = self.user_api.notifications(self.user)

allowed_cities = Tuneable.get_allowed_cities(self.user_api.level()) current_city = self.request.session.get('current_city', None)

if self.request.method == 'POST' and 'city' in self.request.params: current_city = self.request.POST.get('city', None)

if not current_city: current_city = allowed_cities[0] elif current_city not in allowed_cities: current_city = 'hell' # easter egg for those who like to play with # data

self.request.session['current_city'] = current_city self.request.session['allowed_cities'] = allowed_cities

paginated_events = Pagination(self.event_api.get_user_events( with_events=True, city=current_city), Tuneable.get_per_page('events'), Tuneable.get_pages_per_chunk())

return dict( user_events=paginated_events.get_page_items(page), pagination=paginated_events, current_page=page, reward=reward, requirement=cost, can_participate=can_participate, req_details=details, notifications=notifications, cities=Tuneable.get_cities(),

current_city=current_city, allowed_cities=allowed_cities, )

Architecture

Core API and Data AccessWe keep them separate

Instead of littering the API with stuff like this

u = UserIO.one(user_id)

if not u: topic = []else: topic = u['answers'].get(event_id, {}).get(str(topic_id), [])

We use object model methods

topic = UserEvent.get_topic_answers(user_id, event_id, topic_id)

The code then become easier to understand

def matches(self, user_id, event_id, topic_id): """ Return a list of users whose answer is similar. """ topic = UserEvent.get_topic_answers(user_id, event_id, topic_id)

if topic is None: return None

return UserEvent.matches(user_id, event_id, topic_id)

MongoDB

schema-less

non-relational

full of awesomeness!

Well-suited to building clusters.

Harder to break.

The State of MongoDB in PythonChoices for MongoDB in Python are limited to MongoEngine and MongoKit.

So we rolled our own on top of raw pymongo.

But then we were even more limited.

Switching to MongoEngine.

Really nice!

Migrations

NoSQL == no migrations, right?

WRONG!!

DB is schema-less

But for the sake of sanity, app shouldn't be

Data and content can change

Migration modes

In Python, on-the-fly

Migration scripts

Auto-migrate on deployment

Lettuce

Is lettuce a vegetable or a testing platform?!

Feature: Main show In order ward off boredom and hopefully get laid As a facebook user and bachelor wannabe I want to take part in the best of the best And that means playing the Bachelor game

# Actors: # - B: Bachelor # - C: Contestant # - S: System # - F: Friend

Scenario: B starts a show Given B has enough points to start a show When B initiates a new show And B selects four of his pictures And B starts the audition Then B should loose the expected points to start a show And B should not yet have access to the audition results And B should not be able to start another show

Scenario: S selects the audition winners Given the allotted audition time has elapsed And 24 C or more have entered the audition When S selects the 24 best matches Then there should be 24 C in the show And the lives of countless others shall be ruined

Lettuce

Lettuce steps map to Python code:

Scenario: C looses episode 1 Given C has lost episode 1 When C asks for episode 1 results Then C should be informed that he failed the profound cuteness test And C should gain the points earned by episode 1 losers And C should should not have access to episode 2

@step(u'Then C should be informed that he failed the profound cuteness test')def then_c_should_be_informed_that_he_failed_the_profound_cuteness_test(step): w.golem.as_bachelor() losers = w.golem.get('12_roses_losers', True)

for l in losers: user = w.golem.user_api.get_user(l) w.golem.as_contestant(user) show = w.golem.show_api.get_by_id(w.golem.show_id) is_winner = w.golem.show_api.collect_12roses(show=show) assert not is_winner

Facebook IntegrationFacebook lets you take advantage of an existing account, their friends, andsocial sharing. Great for marketting, no fun for developers.

# What's the purpose of this... there's no id when you ask to add the# app so it raises the exception before a user can register# ncadou: the purpose of this check is to prevent the creation of blank# users when the facebook API actually fails. See commit 7bfef9df70f7.#if not 'id' in user:# # TODO log something# raise Exception('Facebook API error')if user.get('error', None): return dict()

Facebook API IssuesOnly the most critical ones...

Downtime

Speed (Latency in every request)

Sudden Changes

Documentation - Comparable to Early-Stage FOSS

How Your Public API Can BeBetterK.I.S.S.

Namespaces are one honking great idea -- let's domore of those!Seriously, hierarchical organization is easy to understand and use.

Document every API callExplain what it does and any nuances.

Document every variableType, possible options (and what each option means), and if itsrequired.

Mapping Game Logic to ObjectsMultiplayer social games are step and role based.

System Tasks and TimersCertain tasks have time limits or system requirements

def make_steps(): g = GameSteps()

# Episode 0 - Audition g.add(bachelor=g.step('show_create', episode=0)) g.add(bachelor=g.step('audition_wait', has_url=False, episode=0, can_skip=True), contestant=g.step('audition_enter', episode=0)) g.add(contestant=g.step('audition_wait', can_skip=True, has_url=False, episode=0)) g.add(system=g.step('select_contestants', has_url=False, is_stop=True, timer=Tuneable().get_durations(0), callback=callback.select_contestants)) g.add(bachelor=g.step('audition_result', label='Reveal Audition Results', episode=0), contestant=g.step('audition_result', label='Reveal Audition Results', episode=0))

Worker ProcessesMongoDB as RPC.

Needed for timed game events.

Needed because Facebook's API is slow.

Background task execution.

Immediate or scheduled.

Captures responses and exceptions.

Acts as a state machine with locking.

Worker ProcessesMongoDB as RPC.

Stores job data in a collection.

Notifications via capped collection.

Uses Futures for parallel execution of tasks.

Uses APScheduler for timed execution of tasks.

Atomic locking using "update if current" mechanic.

Capped CollectionsMongoDB as a low-latency queue.

Limited size, optionally limited document count.

Ring buffer design.Insert order.Updates allowed… mostly.

Used by MongoDB for replication.

Tailable cursors.Long-poll push, like IMAP IDLE.

Live demonstration time!

Example Job RecordStored in a permanant collection.

{ "_id" : ObjectId("4ea3717f9bfbb601d2000002"), "state" : "new", // pending, dead, cancelled, running, finished "callable" : "c__builtin__\nprint\np1\n.", "args" : [ "Task", 0 ], "kwargs" : { }, "created" : ISODate("2011-10-23T01:44:31.446Z"), "creator" : [ "Lucifer", 298, 466 ], "owner" : null, // [ "Lucifer", 324, 456 ] // If scheduled, not immediate: "when": ISODate("...") // If in progress or completed... "acquired" : ISODate("..."), // If completed... "result" : null, "exception" : null, "completed" : ISODate("..."),}

Example Notification RecordStored in the capped collection.

// Workaround for MongoDB quirk.{ "_id" : ObjectId("4ea371629bfbb601c8000000"), "nop" : true }

{ // New job. "_id" : ObjectId("4ea371769bfbb601d2000001"), "job_id" : ObjectId("4ea371769bfbb601d2000000"), "creator" : [ "Lucifer", 298, 466 ]}

{ // Finished job. "_id" : ObjectId("4ea371769bfbb601c8000001"), "job_id" : ObjectId("4ea371769bfbb601d2000000"), "creator" : [ "Lucifer", 324, 456 ], "result" : true}

Example Queue RunnerPython generators are teh win.

def queue(collection, query=None): if not collection.find(): # This is to prevent a terrible infinite busy loop while empty. collection.insert(dict(nop=True)) last = None query = query or {} cursor = collection.find(query, slave_ok=True, tailable=True, await_data=True) while True: # Primary retry loop. try: while cursor.alive: # Inner record loop; may time out. for record in cursor: last = record['_id'] yield record except OperationFailure: pass retry_query = {"_id": {"$gte": last}} retry_query.update(query) cursor = collection.find(retry_query, slave_ok=True, tailable=True, await_data=True)

Example Job HandlerJob locking to prevent accidental execution.

def handler(self, job_id): # Build the dictionary update. update = dict(acquired=datetime.utcnow(), state="running", owner=self.identity) try: result = self.jobs.update(dict(_id=job_id, state="pending", owner=None), {"$set": update}, safe=True) except: raise AcquireFailed() if not result['updatedExisting']: raise AcquireFailed() try: job = self.jobs.find(dict(_id=job_id), limit=1, fields=['callable', 'args', 'kwargs'])[0] except: # This should, in theory, never happen unless MongoDB goes away. raise AcquireFailed() obj = pickle.loads(job['callable'].encode('ascii')) args = job.get('args', []) kwargs = job.get('kwargs', {}) return obj(*args, **kwargs)

Questions?Comments?