aws re:invent 2016: using aws lambda to build control systems for your aws infrastructure (svr401)
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Randall Hunt, Developer Evangelist, AWS
Using AWS Lambda to Build Automated
Control Systems for Your AWS
Infrastructure
December 1, 2016
SVR401
Welcome to re:Invent!
Randall Hunt - @jrhunt
Developer Evangelist at AWS
Formerly of SpaceX, NASA
Agenda
• Brief overview of AWS Lambda
• Why automate?
• Why Lambda for automation and control systems?
• Event-driven policy enforcement
• Lambda as an infrastructure control plane
• Best practices
Operations and management Scaling
Provisioning and utilization Responsible for availability
and fault tolerance
Owning servers means dealing with ...
Serverless compute: AWS Lambda
COMPUTE SERVICE
EVENT-DRIVEN
Run code
without
managing
servers
Code runs only
when it needs to
run
Code is all you need(Native: Java/Python/NodeJS)
Event-driven scaling
Never pay for idle servers Availability and fault tolerance built in
Going serverless with AWS Lambda
Things to remember: Lambda function
Memory = “Power level”
• Higher levels offer more memory and more CPU power
Functions don’t have a notion of state
• Use Amazon DynamoDB, Amazon S3, or Amazon ElastiCache
• Wrap your config in a function and call it from your published code
Use the right access control for downstream services
• IAM roles and permissions for AWS services
• VPC for private endpoints
Benefits of automation done correctly
Increased Agility Improved Quality
Improved Innovation
Increased Consistency
Reduced Expenses and
Labor Costs
Reduced Human Errors
Dangers of incorrect automation
Code Maintenance Issues Bad Permissions Control Auditing Nightmares
Everyone Blames You!Accountability
Amazon
API GatewayAWS
IoT
Amazon
Alexa
Automatic Infrastructure Cleanup Allow other endpoints and AWS services to
trigger infrastructure change
Create copies/snapshots of data in response
to events
Integrate / think outside the box
Amazon EC2 Amazon ECS Amazon ECR
Amazon RDS Amazon
DynamoDB
Instant Core Infrastructure Changes
Amazon
VPC
Amazon
Route 53AWS
CloudFormation
Lambda as a building block for automaton
Because Lambda is event-driven, it offers a very
powerful framework for automated infrastructure
control planes and policy engines.
AWS Lambda
Near real-time reactionEvent
Benefits for automation with AWS Lambda
Single knob configuration Easy to integrate Logging and auditability
Rapid delivery and versioning Get started fast
Event-driven automation as a building block
Being proactive rather than reactive to changes in your
infrastructure is key. However, event-driven automation can
be part of the solution. We can remove human errors when
we need to be reactive through use of automation.
Amazon
S3
Amazon
DynamoDB
Amazon
Kinesis
AWS
CloudFormation
AWS
CloudTrail
Amazon
CloudWatch
Logs
Amazon
SNSAmazon
SES
Amazon
API Gateway
Amazon
Cognito
AWS
IoT
Cron events
Trigger an event based on Data Store changes Interact with other endpoints and AWS services
to trigger infrastructure change
Respond to custom events as they happen
near real-time
Respond to messaging or scheduled
events in your infrastructure
Event sources
Policy enforcement
What do you mean by policy enforcement?
You have the freedom to define what
policy enforcement means to you!
VPC policy enforcement
Lambda Policy Engine
• Allocate Address
• Associate Address
instance instance instance
instance instance instance
instance instance instance
Validate Security Group
instance instance instance
Identity and access management enforcement
Lambda Policy Engine
Validate Role Path
AWS IAM
CreateRole
Verify No Role Elevation
Verify No Resource Level Elevation
Allow Role
Delete Roles
Tagging enforcement
Lambda Policy Engine
instance instance instance
instance instance instance
RunInstancesValidate Team Tag
Validate Billing Group Tag
Validate Environment Stage Taginstance instance instance
StopInstances
SNS Notification
instance instance instance
Infrastructure control plane
Lambda gives you the freedom to
design the automation you need for
your infrastructure.
Infrastructure automation
Amazon CloudWatch Events
Auto Scaling Event AWS API Call EC2 Instance State Change
Automatic security group placement
Lambda Policy Engine
instance instance instance
instance instance instance
RunInstancesCheck VPC
Validate Security Group
Remove SG
Add Correct SG
instance instance instance
Do Nothing
Create/Update Amazon Route 53 records from tag
Lambda Policy Engine
instance instance instance
instance instance instance
RunInstances
Do Nothing
instance instance instance
Check VPC
Update DNS instance
Get CNAME Tag
Add DNS Record
instance instance
Bootstrapping With Lambda and Await/Notify
• Need to load a schema into RDS? Lambda function.
• Need to load data into RDS? Lambda function.
• Need to load data into DynamoDB? Lambda function.
• Need to perform a complex bootstrapping action?
Lambda function.
• Need to depend on an external action? Lambda function
+ API Gateway.
• For everything else there’s EC2 dependson/await/notify.
Best practices
Log Everything
Bite-Sized Code
Report Failures
Version Lambda Functions
Use Source Control
Iterate
Recap
• Event-Driven Response to Policy Management and Infrastructure Events.
• AWS Config Rules backed by Lambda to visualize compliant versus non-compliant
infrastructure.
• Think outside the box. Number of available CloudWatch Events API Call triggers is
large.
• Don’t forget the ability to schedule AWS Config rule validation, as well as Scheduled
CloudWatch Event Triggers to Lambda.
• CloudWatch Events can point to multiple Targets (Not just Lambda functions).