open stack ocata summit enabling aws lambda-like functionality with openstack swift and openwhisk
TRANSCRIPT
Enabling AWS Lambda-like functionality with OpenStack Swift and OpenWhiskShaun Murakami, Senior Cloud Architect, @stmurakaAndrew Bodine, Software Engineer, @bodine_andrew
Development in the Cloud
Bare metal
Virtual machines
ContainersFunctions
Decreasing concern (and control) over stack implementation
Incr
easi
ng fo
cus
on b
usin
ess
logi
c
Serverless can handle many cloud native 12 Factors appI Codebase Handled by developer (Manage versioning of functions on their own)
II Dependencies Handled by developer, facilitated by serverless platform (Runtimes and packages)
III Config Handled by platform (Environment variables or injected event parameters)
IV Backing services Handled by platform (Connection information injected as event parameters)
V Build, release, run Handled by platform (Deployed resources are immutable and internally versioned)
VI Processes Handled by platform (Single stateless containers often used)
VII Port binding Handled by platform (Actions or functions are automatically discovered)
VIII Concurrency Handled by platform (Process model is hidden and scales in response to demand)
IX Disposability Handled by platform (Lifecycle is hidden from the user, fast startup and elastic scale is prioritized)
X Dev/prod parity Handled by developer (The developer is the deployer. Scope of what differs is narrower)
XI Logs Handled by platform (Developer writes to console.log, platform handles log streaming)
XII Admin processes Handled by developer (No distinction between one off processes and long running)
Event driven programming
Execute app logic in response to database triggers
Execute app logic in response to sensor data
Execute app logic in response to cognitive trends
Execute app logic in response to scheduled tasks
Provide easy server-side backend for mobile app
Amazon Lambda• An event-driven compute service where AWS Lambda runs
your code in response to events, such as changes to data or in response to HTTP requests.
http://docs.aws.amazon.com/lambda/latest/dg/welcome.htmlhttp://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html
OpenWhisk
An open source cloud platformthat executes code
in response to events
https://github.com/openwhisk/openwhisk
OpenWhisk supports many growing workloads
OpenWhisk can help power various mobile, web and IoT app use cases by simplifying the programming model of orchestrating various services using events without a dedicated backend.
Digital app workloads Big Data/Analytics pipelineComplex data pipelines for Big Data/Analytics tasks can be scripted using changes in data services or streams for near real-time analytics and feedback.
DevOps and infrastructure as codeOpenWhisk can be used to automate DevOps pipelines based on events triggered from successful builds, or completed staging, or a go-live event.
Microservices builderOpenWhisk can be used to easily build microservices given the footprint and programming model desired by microservices
OpenWhisk Programming model
Developers use Rules to…associate how Actions handle events emitted by Triggers generated by services
R T A:=
riggersA class of events that can occurT
Social events
Data changes
Device readings Location updates
User input
ctionsCode that runs in response to an event (i.e., an event handler)
A
Supports a variety of languages such asJavaScript, Python, Java, and Swift ...
function main(params) {return { message: 'Hello, ' +
params.name + ' from ' + params.place };};
... even as a Docker container
Actions can be chained together
AA := A1 + A2 + A3
AB := A2 + A1 + A3
AC := A3 + A1 + A2
OpenWhisk can implement RESTful microservices
Send HTTP requestHTTP GET
app.com/customers
1
Invoke OpenWhiskaction get-customers
Browser
Mobile App
Web App
2
JS Swift Docker …
OpenWhisk
API Proxy
OpenStack Swift Middleware• Swift uses middleware to add custom behaviors.• The majority of Swift middleware is applied to the
Proxy Server.• Python WSGI Middleware (or just “middleware”)
can be used to “wrap” the request and response of a Python WSGI application (i.e. a webapp, or REST/HTTP API), like Swift’s WSGI servers (proxy-server, account-server, container-server, object-server)
http://docs.openstack.org/developer/swift/development_middleware.html
http://docs.openstack.org/admin-guide/objectstorage-arch.html
OpenStack Swift System Metadata• Provides a means to store potentially private custom metadata
with associated Swift resources in a safe and secure fashion.• May be set on accounts and containers by including headers
with a PUT or POST request. • Maybe set on objects using only PUT requests.• Takes the form of X-<type>-Sysmeta-<key>: <value>
• <type> depends on the resources type (i.e. Account, Container, Object)
• <key> and <value> are set by trusted code running in a Swift WSGI Server
http://docs.openstack.org/developer/swift/development_middleware.html#sysmeta
AWS S3 + Lambda
1. A user uploads an object to the source bucket in Amazon S3 (object-created event).
2. Amazon S3 detects the object-created event.3. Amazon S3 publishes the s3:ObjectCreated:* event
to AWS Lambda by invoking the Lambda function and passing event data as a function parameter.
4. AWS Lambda executes the Lambda function by assuming the execution role that you specified at the time you created the Lambda function.
5. From the event data it receives, the Lambda function knows the source bucket name and object key name. The Lambda function reads the object and creates a thumbnail using graphics libraries, and saves it to the target bucket
http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html
OpenStack Swift + OpenWhisk
JS Swift Docker …
OpenWhisk
API P
roxy 1. A user uploads an object to a OpenStack Swift
Container.2. OpenStack Swift detects an object has been
modified in the Container.3. The Swift Webhook middleware triggers an
OpenWhisk Action by invoking the OpenWhisk API specified as the container’s webhook URL.
4. The OpenWhisk credentials are validated against the OpenWhisk authorization service.
5. From the request payload, the OpenWhisk action knows the source container and object name. The OpenWhisk Action creates a thumbnail using graphics libraries and saves it to a target container
Swift
Mid
dlew
are
SourceContainer
TargetContainer
2
3
4
WhiskAction
1
5
OpenStack Swift Middleware (webhook.py)class WebhookMiddleware(object):
@wsgifydef __call__(self, req):
req.headers[SYSMETA_WEBHOOK] = req.headers['x-webhook']req.headers[SYSMETA_WEBHOOK_AUTH] = req.headers['x-webhook-auth']
resp = req.get_response(self.app)
if obj and is_success(resp.status_int) and req.method in [’PUT’,’DELETE’]:webhook_req = urllib2.Request(container_info['sysmeta'].get('webhook'))
webhook_req.add_header('Authorization', base64.b64encode(container_info['sysmeta'].get('webhook-auth')))webhook_req.add_data(json.dumps({
’url’: swiftUrl, ‘token’: req.headers[‘x-auth-token’], ‘container’: container, ‘object’: obj, ‘method’: req.method}))
...try:
webhook_resp = urllib2.urlopen(webhook_req).read()….
return resp
Install Webhook Middleware1. Place webhook.py in middleware of swift-proxy…/swift/common/middleware/webhook.py
2. Update proxy-server.conf…[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck ... webhook proxy-server
[filter:webhook]
paste.filter_factory = swift.common.middleware.webhook:webhook_factory
...
OpenWhisk Action DockerfileFROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update \&& apt-get install -y curl
# Node.jsRUN curl -sL https://deb.nodesource.com/setup_4.x | bash - \
&& apt-get install -y nodejs
WORKDIR /root/server
COPY server .RUN npm install
EXPOSE 8080
CMD node ./app.js
Future workSwift Asynchronous Event Notification Framework• Support Multiple Notification Use Cases
• OpenWhisk / Serverless computing• Cloud inter-service notifications• Metadata search indexing• Resource / traffic monitoring
• Common Notification APIs• Enable / disable notifications on an account or container• Allow for user / system level policies• Desire to become a de-facto standard in Swift for notification management
• Deployment Flexibility• Can be adapted to multiple message queue technologies• Initial submission will include Kafka and Zaqar
• Patch WIP• https://review.openstack.org/388393
Resources• AWS Lambda
• Example: http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html
• OpenWhisk• Code: https://github.com/openwhisk/openwhisk• On IBM BlueMix: http://www.ibm.com/cloud-computing/bluemix/openwhisk/
• OpenStack Swift Middleware• http://docs.openstack.org/developer/swift/development_middleware.htm
• Demo Code• https://github.ibm.com/stmuraka/OpenStackSwift-OpenWhisk