Crafting [Better] API Clients

PyTennesee 2015- Ben Lopatin

Partner and developer @ Wellfire Interactive

@bennylope

Clients, not Services

Why?

A great API is necessary but insufficient to be

useful.

This is the right way.

This is the right way. This is some good ideas.

Acknowledge that you’re making design decisions.

Some assumptions

Data Model

Why?

Database API

Real world

Layers of abstraction!

Connections & data

Designing the Python data interface

Explicit data attributes or implicit collection?

class DataResource: def __init__(self, id, first_name, last_name): self.id = id self.first_name = first_name self.last_name = last_name resource.first_name

resource = { 'id': 1, 'first_name': 'Ben', 'last_name': 'Lopatin', } resource['first_name']

class DataResource(dict): def update(self, client): # Do something cool here client.update(self.id, self.first_name, self.last_name)

Data or Resources?

• If you think about everything as a resource, then shouldn’t it have the same methods?

• Or is it data and the API is just providing some

• You’re not obliged to represent 1:1

Be mindful of what you discard

Second, we wanted to give you a heads up that we're announcing a new feature soon -- additional fields. This will allow people to get congressional districts, state legislative districts, timezones, and school districts with a forward or reverse lookup. We are looking to add more additional fields in the future (namely Census data).

Errors

Why?

What?

API application errors Authentication errors

Account errors …

If only there were a concept to encapsulate

errors in Python…

…that would be exceptional!

Exceptions for obvious flow control

try: response = client.request()except APIPaymentError: # Email corporate accounts payable raiseexcept APIAuthError: # Redirect to account settings raise

Map exceptions to API errors

class SmartyStreetsError(Exception): """Unknown SmartyStreets error""" def __str__(self): return self.__doc__class SmartyStreetsInputError(SmartyStreetsError): """HTTP 400 Bad input. Required fields missing from input or are malformed."""class SmartyStreetsAuthError(SmartyStreetsError): """HTTP 401 Unauthorized. Authentication failure; invalid credentials"""class SmartyStreetsPaymentError(SmartyStreetsError): """HTTP 402 Payment required. No active subscription found."""class SmartyStreetsServerError(SmartyStreetsError): """HTTP 500 Internal server error. General service failure; retry request."""

ERROR_CODES = { 400: SmartyStreetsInputError, 401: SmartyStreetsAuthError, 402: SmartyStreetsPaymentError, 500: SmartyStreetsServerError,}

…maybe not all errors

Errors aren’t necessarily limited to HTTP status

codes

Exceptions +

Error codes

Where?

Logging

Why?

What?

Verify requests issued

Request/response count

API performance

API errors

How?

@decorators?

@log_requestsdef _req(self, method='get', verb=None, headers={}, params={}, data={}): url = self.BASE_URL.format(verb=verb) request_headers = {'content-type': 'application/json'} request_params = {'api_key': self.API_KEY} request_headers.update(headers) request_params.update(params) return getattr(requests, method)(url, params=request_params, headers=request_headers, data=data)

def log_requests(func): def decorator(*args, **kwargs): request_id = str(uuid.uuid4()) logger.info("Requesting %s" % request_id) try: resp = func(*args, **kwargs) except: logger.exception("Request error %s" % request_id) raise logger.info("Response %s" % request_id) return resp return decorator

Testing

Why?

How?

Live API Mock servers

Mocked responses

That’s not nice (or fun)

Mock server?

Live testing’s hard, let’s go mocking!

Mocking requests

responses HTTPretty

responses HTTPretty betamax

@responses.activate def test_auth_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=401, content_type='application/json') self.assertRaises(SmartyStreetsAuthError, self.client.street_addresses, [{}, {}])

@responses.activate def test_payment_error(self): responses.add(responses.POST, 'https://api.smartystreets.com/street-address', body='', status=402, content_type='application/json') self.assertRaises(SmartyStreetsPaymentError, self.client.street_addresses, [{}, {}])

@httpretty.activate def test_auth_error(self): """Ensure an HTTP 403 code raises GeocodioAuthError""" httpretty.register_uri(httpretty.GET, “http://api.geocod.io/v1/parse", body="This does not matter", status=403) self.assertRaises(GeocodioAuthError, self.client.parse, "")

There can be benefits to some live testing

Performance

Connections

Batching

The right API methods

Data structures

class APIData: def init(self, id, name, image): self.id = id self.name = name self.image = image

class APIData: __slots__ = ['id', 'name', 'image'] def init(self, id, name, image): self.id = id self.name = name self.image = image

Concurrency

response_iter = ( grequests.post( url=url, data=json.dumps(data_chunk), params=params, headeresponse_iter=headeresponse_iter, ) for data_chunk in chunker(data, 100) ) responses = grequests.imap(response_iter, size=parallelism)status_codes = {}addresses = AddressCollection([])for response in responses: if response.status_code not in status_codes.keys(): status_codes[response.status_code] = 1 else: status_codes[response.status_code] += 1 if response.status_code == 200: addresses[0:0] = AddressCollection(response.json())

session = FuturesSession(max_workers=parallelism)session.headers = headerssession.params = paramsfutures = [ session.post(url, data=json.dumps(data_chunk)) for data_chunk in chunker(data, 100) ] while not all([f.done() for f in futures]): continuestatus_codes = {}responses = [f.result() for f in futures]addresses = AddressCollection([])for response in responses: if response.status_code not in status_codes.keys(): status_codes[response.status_code] = 1 else: status_codes[response.status_code] += 1 if response.status_code == 200: addresses[0:0] = AddressCollection(response.json())

You know Twisted can do that, right?

Security

HTTPS

Python 2.7

• pyOpenSSL

• ndg-httpsclient

• pyasn1

Authenticating

Docs

Django

No?

Python library

Anything Django related

• Create the client *first* then Django integration is a bonus.

• Mapping models to a distant API.

• It’ll be easier for you to maintain and test

• Easier for other people to use for other things

class SomeModel(models.Model): my_field = models.CharField(max_length=100) def sync(self, client=None): if client is None: client = APIClient() client.update(id=self.id, name=self.my_field)

Code Gen

Starting from (DSL) API documentation, automatically

generate matching client code

Crank ‘em out

In [hypothetical] practice?

Suitability?

The End