First Indico Workshop

HTTP APIAdrian Mönnich

29-27 May 2013 CERN

APIUhh… what’s an API and why do we need one?

An application programming interface (API) is a protocol intended to be used as an interface by software components to communicate with each other.

Wikipedia

Before the apiexport.py

Not powerful at all:Only events inside categoriesXML, RSS, iCal or HTML Accessing it requires an XML parserProtection? Not really Must be restricted to trusted machines by IPReusable code? None

/ export.py?fid=2l12&date=today&days=1000&of=xml

And the developers said…

…let there be a proper API…

…and they created one

Design ideasHow to make the API awesome?

Handle authentication in a clean and secure way (API Keys, OAuth2)Various security levels available (API keys, HMAC signatures, …)Let developers write code for the actual taskSerialize Python data structures as JSON, XML, …Provide utilities for common things (relative dates, pagination)

Using the APIBefore we go into details, look at how it worksIndico Administrators: API Modes:

Control how strict the API key requirements areAt CERN: We always require an API key, but a signature is only

needed to access non-public information

Persistency:Lowers security a little bit (one leaked URL is valid forever)Allows people to use e.g. a RSS reader (cannot sign requests)

Using the APIBefore we go into details, look at how it worksIndico Users: Creating an API key is easy – just one click!Keys can be reset at any time, e.g. in case of a (suspected) leakLast request including IP address and full path is displayedPersistence can be toggled (if enabled globally)

Using the APIBefore we go into details, look at how it worksThe same data – available as XML, JSON and sometimes additional more formats (iCal, RSS, HTML) / export/categ/4751.[json|xml|

html]

Using the APIBefore we go into details, look at how it worksWhat it we need a signed request? /export/categ/123.json?limit=10

1. Add the API key to the params …&ak=api-key

2. Add the current timestamp to the params …&timestamp=1234567890

3. Sort the query string params alphabetically…?ak=api-key&limit=10&timestamp=123…

4. Merge path and the sorted query string/export/categ/123.json?ak=api-key&limit=10&timestamp=1234567890

5. Create a HMAC-SHA1 signature of this string using the secret key as the key.6. Append the hex-encoded signature to the query string:

…&signature=xxx

EXTENDING THE APICreating a custom API endpoint

Let’s keep it simple: We want an API that returns a range of numbers or characters.This does not access any indico-specific data but demonstrates some of the utilities available when using the API as a developer./ export/num/1-

10.json/ export/char/a-z.json

EXTENDING THE API

EXTENDING THE APIRelevant classes

HTTPAPIHook: Hooks into the /export/ URL subtree, handles argumentsIteratedDataFetcher: Provides pagination, sorting, relative dates, etc.IFossil: Describes how an object is converted to primitives (dict, list, int, str, bool) that can be serialized (JSON, XML, …) later

EXTENDING THE API

EXTENDING THE APIThe hook, Pt. 1: Metadata

@HTTPAPIHook.registerclass RangeHook(HTTPAPIHook): TYPES = ('num', 'char') RE = r'(?P<start>[0-9]+|[a-z])-(?P<end>[0-9]+|[a-z])' DEFAULT_DETAIL = 'simple' MAX_RECORDS = { 'simple': 10, 'palindrome': 5 }

EXTENDING THE APIThe hook, Pt. 2: Arguments

Perform additional validation if necessaryUse self._pathParams for arguments from the path regexUse self._queryParams to access arguments in the query string

def _getParams(self): super(RangeHook, self)._getParams() self._start = self._pathParams['start'] self._end = self._pathParams['end']

EXTENDING THE APIThe hook, Pt. 3: The export methods

Perform actions/validations specific to the export typeMust return an iterator, usually provided by an IteratedDataFetcher

def export_num(self, aw): try: start = int(self._start) end = int(self._end) except ValueError: raise HTTPAPIError('Invalid value', 400) return RangeFetcher(aw, self).numbers(start, end)

def export_char(self, aw): if len(self._start) != 1 or len(self._end) != 1: raise HTTPAPIError('Invalid character', 400) return RangeFetcher(aw, self).chars(self._start, self._end)

EXTENDING THE API

EXTENDING THE APIThe fetcher, Pt. 1: Metadata

Remember DEFAULT_DETAIL and MAX_RECORDS? Here we specify how objects are fossilized in those detail levels.More about fossilization later, let’s get some data first!

class RangeFetcher(IteratedDataFetcher): DETAIL_INTERFACES = { 'simple': IDummyFossil, 'palindrome': IDummyPalindromeFossil }

EXTENDING THE APIThe fetcher, Pt. 2: Python iterator magic

Remember: We call these methods from our export methods.In this case we just iterate over numbers/characters in the given range.DummyObject? You will see soon…

def numbers(self, start, end): iterable = xrange(int(start), int(end) + 1) iterable = itertools.imap(str, iterable) iterable = itertools.imap(DummyObject, iterable) return self._process(iterable)

def chars(self, start, end): iterable = itertools.imap(chr, xrange(ord(start), ord(end) + 1)) # Besides that first line everything is the same like in numbers()

EXTENDING THE APIThe Dummy Object

Usually we deal with custom objects in Indico. Because of historical reasons it is common to have getter methods (yes, we know it is ugly and unpythonic).The fossilization system is built around this thin wrapper providing a getter for our primitive string values.

class DummyObject(object): def __init__(self, value): self.value = value def getValue(self): return self.value

EXTENDING THE API

EXTENDING THE APIFossils

IDummyFossil is really simple: It fossilizes the getValue() return value of the actual object to a field named value.For the palindrome detail level we specify a custom callable to generate the value on the fly. The field name is determined automatically again.

class IDummyFossil(IFossil): def getValue(self): pass

class IDummyPalindromeFossil(IDummyFossil): def getPalindrome(self): pass getPalindrome.produce = lambda x: x.value + x.value[::-1]

EXTENDING THE APIDoes it work?

/ export/num/128-256.json?detail=palindrome/ export/num/128-256.json/ export/num/1-1.xml

EXTENDING THE APIYes! Did you notice the pagination?

/ export/num/128-256.json?detail=palindrome/ export/num/128-256.json/ export/num/1-1.xml

What elseThe API is even more powerful!

Caching (unless disabled by the user per-request)POST (to modify things, e.g. booking a room)Additional URL prefixes besides /export/ (e.g. /api/)Many things (like the export_* method name) can be changed easily when subclassing HTTPAPIHookReading the existing code is the best way to get used to it!

EXTENDING THE API

EXTENDING THE APIAnother example, using POST

@HTTPAPIHook.registerclass AnnouncementHook(HTTPAPIHook): PREFIX = 'api' TYPES = ('announcement',) RE = r'set' GUEST_ALLOWED = False VALID_FORMATS = ('json', 'xml') COMMIT = True HTTP_POST = True

def _getParams(self): super(AnnouncementHook, self)._getParams() self._message = get_query_parameter(self._queryParams, ['message'], '')

def api_announcement(self, aw): am = getAnnoucementMgrInstance() am.setText(self._message) return {'message': self._message}

Adrian Mönnich

Questions?

https://github.com/ThiefMaster

View the example code on GitHub: https://goo.gl/btdVc [Diff]https://goo.gl/wGJXG [Gist]