Building Microservices with API Gateway and Lambda

@mattjbarlowgithub.com/mattjbarlow

DevOps Days Austin 2016Revenge of the Devs

What we will talk aboutSwagger

API Gateway

Lambda

DynamoDB

What we will talk aboutSwagger

API Gateway

Lambda

DynamoDB

API Spec File

Proxy / Router / URL

Backend Code

Data

With Swagger you can auto-generate...Documentation Monitoring Tests

Integration Tests

Client Libraries

We Never Edit API Gateway Directly We Edit Swagger

which is imported into...

API Gateway Monitoring Docs Tests

Example Microservice

Teammates want the ability to run a Lambda Function exactly one time in the future.

Solution

At Job Microservice

A microservice that mimics the Unix at command for Lambda.

The At Command

First appears in 1979 as part of Unix Version 7.

Design

API MethodsPath Operation OperationID Description

/atq GET list_jobs List all jobs

/atq POST create_job Create an at job

/atq/{id} GET describe_job Describe an at job

/atq/{id} DELETE delete_job Delete an at job

A method is a combination of a resource path and an operation.

Objects

Name Type Format Required?

jobid string uuid yes

lambdaArn string arn no

time string dateTime no

atJob

The object will define the response that our API returns.

CodeOperationID Pseudocode

list_jobs DynamoDB BatchGetItem

describe_job DynamoDB Query on jobid

create_job Create CloudWatch Event and PutItem into DynamoDB.

delete_job Delete CloudWatch Event and DeleteItem out of DynamoDB.

These Operation IDs are defined in Swagger and passed through to Lambda code as part of the event object.

Initialize Directory

git clone git@github.com:mattjbarlow/microservice-template.git

File Description

service/service.py * Python module that will run in Lambda.

circle.yml Circle CI instructions.

deploy.yml Ansible playbook for provisioning microservice.

destroy.yml Ansible playbook for deleting microservice.

swagger.yml * Spec file that describes your API.

template.json AWS resources required by the microservice.

version.yml Ansible playbook that versions your Lambda code.

* The bulk of your edits will be in these two files.

Update Swagger Paths

Remember these?Path Operation OperationID Description

/atq GET list_jobs List all jobs

/atq POST create_job Create an at job

/atq/{id} GET describe_job Describe an at job

/atq/{id} DELETE delete_job Delete an at job

We Insert Them Directly Into SwaggerRESOURCE PATH

HTTP OPERATION

Object Definition

Then we define our objects

Deploy

ansible-playbook -e “prefix=devopsdays” deploy.yml

Ansible Deployment Playbook

1. Zips up Python module and uploads it to S3

2. Creates the API Gateway

3. Provisions AWS resources

a. Lambda

b. DynamoDB

c. IAM Roles and Policies

d. Lambda Permission

4. Adds mapping templates to API Gateway

5. Sets stage variables

6. Exports Swagger file from API Gateway

Import Swagger Into Postman

Open Postman

Click Import

Select Swagger File

Start Sending Requests

Tail Lambda Logs

Logging in Lambda

API Gateway turned our HTTP request data into an Event Object. To start with, we log the entire Event Object.

pip install awslogs

awslogs allows us to mimic tail -f behavior on our Lambda function’s Log Group.

Event Object Close-Up

Post Body

Headers

Path ParamsQuery Params

API Gateway Variables

Helpful for filtering logs.

Tracing Our Request

GET Request Lifecycle

Method Response

Integration Response

Lambda

Method Request

Integration Request

Client

Cloudwatch Logs

HTTP GET Request Is Sent

GET /dev/v1/atq HTTP/1.1Host: cg4e6xg82i.execute-api.us-east-1.amazonaws.comConnection: keep-aliveCache-Control: no-cacheUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36Postman-Token: 265fcfe1-e196-0114-89bf-442a34c0180dAccept: */*Accept-Encoding: gzip, deflate, sdchAccept-Language: en-US,en;q=0.8

Mapped To Event Object

Event Object Received by Lambda

Logged to CloudWatch

Editing Code

Dev WorkflowEdit Eval

Push

Request

Parse

create_job function

Push Code To Lambda

ansible-playbook -e “prefix=dod” version.yml

Test create_jobImportant!

create_job response

Persisted in DynamoDB

POST Request Lifecycle

Method Response

Integration Response

Lambda

Method Request

Integration Request

Client

Cloudwatch Logs

EventsDynamo

Removing Lambda

GET Request With LambdaStep What Happens

Send GET request API Gateway Receives HTTP Request Data from client.

Transform API Gateway Transforms Request Data into Event Object

Proxy API Gateway POSTs Event Object to Lambda

Read Lambda Reads data from DynamoDB

Return API Gateway Receives return values from Lambda

Transform API Gateway Transforms backend data into HTTP Response

Respond API Gateway Responds back to the client.

GET Request without LambdaStep What Happens

Send GET request API Gateway Receives HTTP Request Data from client.

Transform API Gateway Transforms Request Data into Event Object

Proxy API Gateway POSTs DynamoDB Query to Dynamo

Read Lambda Reads data from DynamoDB

Return API Gateway Receives return values from Dynamo

Transform API Gateway Transforms backend data into HTTP Response

Respond API Gateway Responds back to the client.

Mapping GET Request To DynamoDB

Mapping DynamoDB Response to Client Response

Automating Tests

CI WorkflowCI

Step 1 Deploy Microservice

Step 2 Validate API Calls

Step 3 Destroy Microservice

git push POST: /project/:tree/:branch

Validating API Calls With Flexschema = load('swagger.awsexport.json')validate_api_call(schema, raw_request=r.request, raw_response=r)

1. Receives Swagger spec file which is our source of truth.2. Makes an HTTP Request to the API Gateway URL.3. Ensures the response matches what we said it would in Swagger.

Loads Swagger spec into memory

AuthIt has been 1 days since we talked about auth.

Built In Auth OptionsGET /dev/v1/atq HTTP/1.1Host: cg4e6xg82i.execute-api.us-east-1.amazonaws.comConnection: keep-alivex-api-key: bkayZOMvuy8aZOhIgxq94K9Oe7Y70Hw55

Option 2: Signature Version 4 signing with IAM(Powerful, but requires client having AWS API Key)

Option 1: API Keys managed by API Gateway API(Not really useful for user auth in public APIs)

Custom Authorizers

Receive Token from IdentitySend Token to /auth endpoint Pass Through

Responds With JWT

Make request with JWT Custom Authorizer Intercept

Caches temporary policyAllows Request

Generates JWT

Validates JWT

Returns temporary IAM policy

Client Library API Gateway Lambda$:

Validates Identity Token

Links

https://github.com/mattjbarlow/microservice-template

https://github.com/mattjbarlow/at

http://editor.swagger.io/#/

http://flex-swagger.readthedocs.io/en/latest/