improving infrastructure governance on aws - aws june 2016 webinar series
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chris Munns, Business Development Manager - DevOpsJune 2016
DevOps on AWS:Improving Infrastructure Governance on AWS
https://secure.flickr.com/photos/mgifford/4525333972
Why are we here today?
Why are we here today?
Using cloud based infrastructure changes how to think about governing our infrastructure: Infrastructure can be provisioned in seconds.. and go
away just as quickly! Development teams expect a higher level of flexibility and
self control in interfacing with their infrastructure needs Being API driven means that the way people provision and
manage infrastructure in the cloud has changed compared to on-premises
Why are we here today?
That doesn’t mean that our basic governance needs change: We still need to have some ability to drive best
practices/patterns in our organizations We need to make sure that we’re able to audit
and track changes to our infrastructure for both regulation and security purposes
We need to make sure that we understand how resources are related and integrated
What can we do?
There are a few areas to focus on that can help us accomplish both the freedom to rapidly provision, manage, and update our infrastructure while meeting our governance needs: Policy as Code Infrastructure standardization (via code!) Self service environments Logging/Auditing/Reacting to infrastructure change
Policy as Code builds off of infrastructure as code practices by
allowing organizations to codify infrastructure and system configurations
allowing them to monitor and enforce compliance dynamically and at scale.
Infrastructure as Code is a practice in which infrastructure is provisioned
and managed using code and software development techniques,
such as version control and continuous integration.
Infrastructure as Code “levels”
AWS Resources
Operating System and Host Configuration
Application Configuration
Infrastructure as Code “levels”
AWS Resources
Operating System and Host Configuration
Application Configuration
allOfThis == $Code
Browse and launch
AWS ConfigAWS CloudTrail
Use and modify
Users Admin
Putting the AWS Management services together
AWS Service Catalog
Provision with Tags
API calls Configuration checks and reactions to change
Troubleshoot and Audit
Create templates of your infrastructure
CloudFormation provisions AWS resources based on dependency needs
Version control/replicate/update templates like code
Integrates with development, CI/CD, management tools
AWS CloudFormatio
n
Template CloudFormation Stack
JSON formatted fileParameter definitionResource creation
Configuration actions
Configured AWS resourcesComprehensive service support
Service event awareCustomizable
FrameworkStack creationStack updates
Error detection and rollback
CloudFormation – Components & Technology
Template File Defining Stack
The entire infrastructure can be represented in an AWS
CloudFormation template.
Many Stacks & Environments from One Template
Template File Defining Stack
The entire infrastructure can be represented in an AWS
CloudFormation template.
Use the version control system of your choice to store and track changes to this template
Many Stacks & Environments from One Template
GitPerforce
SVN…
Template File Defining Stack
GitPerforce
SVN…
Dev
Test
Prod
The entire infrastructure can be represented in an AWS
CloudFormation template.
Use the version control system of your choice to store and track changes to this template
Build out multiple environments, such as for Development, Test, Production and even DR using the same template
Many Stacks & Environments from One Template
CloudFormation example use cases:
Have “full stack” templates that can be used to stand up common application patterns inside your organization such as a 3-tier application template that: uses Lambda custom resources to look up appropriate VPC
information (VPC ID, Subnets, etc) based on tags creates an Elastic Beanstalk environment that supports Multi-AZ,
AutoScaling, CloudWatch Metrics, and Elastic Load Balancing contains security controls such as AWS Identity and Access
Management (IAM) roles, profiles, and policies, and Security Groups allows the user to specify the language of their application allows a user to specify which database they want (SQL or NoSQL)
and then creates the appropriate resource
Using Parameters and Conditionals are two key ways to make a single template much more dynamic:"Parameters" : { "Database": { "Type" : "String", "Default" : "RDS", "AllowedValues" : ["RDS", "DynamoDB", "None"], "Description" : "Database to create. Select None if using an existing database.” } },
"Conditions" : { "CreateRDS" : {"Fn::Equals" : [{"Ref" : "Database"}, "RDS"]}, "CreateDynamoDB" : {"Fn::Equals" : [{"Ref" : "Database"}, "DynamoDB"]}, "CreateNone" : {"Fn::Equals" : [{"Ref" : "Database"}, "None"]} },
“Resources” : { ”RDSdb01" : { "Condition" : " CreateRDS ", "Type" : "AWS::RDS::Instance",
https://secure.flickr.com/photos/wscullin/3770015991
Now that we have these templates, what can we do to help simplify our developers’ lives further and increase our infrastructure standardization?
Customized catalogs of products
Manage products centrally
Personalized, self-service portal
Integrate with existing systemsAWS Service Catalog
What is AWS Service Catalog?
AWS Service Catalog allows organizations to create and manage catalogs of IT services. It enables users to quickly deploy the approved IT services they need in a self-service manner.
Organizations Developers
ControlStandardization
Governance
AgilitySelf-service
Time to market
Creates portfolio
Adds constraints and grant access
1
4
5
AdministratorPortfolio
Users
Browse Products
6Launch ProductsAWS CloudFormation template
Creates product3Authors template2
ProductX
ProductY
ProductZ
7Deploys stacks
EventsEvents
88
Service Catalog
Create custom services
and grant access
Use a personalized
portal to find and launch services
Service Catalog use cases:
You can remove the need for developers to understand how all AWS services work. Treat infrastructure provisioning like buying components from a retail site: provide standardized Service Catalog products around
common internal application frameworks/architectural patterns provide common application component products such as
databases, queues, caches, worker tiers, etc build logging, monitoring, metrics into these stacks leverage service discovery tools when possible build in the same best practices across development, staging,
production environments with these provided products
We’ve helped solve some of our developer’s access and standardization issues, but how can we now go about auditing changes to our infrastructure?
https://www.flickr.com/photos/atoach/7623237104
AWS CloudTrail
Records AWS API calls for your account
Delivers log files of API calls to S3
Delivery typically within 15 minutes of API call
Logs contain detailed information
Log files can be encrypted and have their integrity verified by you
AWS CloudTrail
CloudTrail can help you achieve many tasks Security analysis Track changes to AWS resources, for
example VPC security groups and NACLs Compliance – log and understand AWS API
call history Prove that you did not:
Use the wrong region Use services you don’t want
Troubleshoot operational issues – quickly identify the most recent changes to your environment
AWS CloudTrail logs can be delivered cross-account
CloudTrail can help you achieve many tasks Accounts can send their trails to a central
account Central account can then do analytics Central account can:
Redistribute the trails Grant access to the trails Filter and reformat Trails (to meet
privacy requirements)
CloudTrail
CloudTrail – Amazon CloudWatch Logs Integration
Continuous recording
Inventory of AWS resources
New & deleted resources
Configuration change & compliance notifications
AWS Config
Relationships
Bi-directional map of dependencies automatically assigned
Change to a resource propagates to create Configuration Items for related resources
Example: Security Group sg-10dk8ej and EC2 instance i-123a3d9 are “associated with” each other
Configuration Item
All configuration attributes
Normalized
Point in time
Captured on configuration change
Component Description Contains
Metadata Information about this configuration item
Version ID, Configuration item ID, Time when the configuration item was captured, State ID indicating the ordering of the configuration items of a resource, MD5Hash, etc.
Common Attributes Resource attributes Resource ID, tags, Resource type. Amazon Resource Name (ARN)Availability Zone, etc.
Relationships How the resource is related to other resources associated with the account
EBS volume vol-1234567 is attached to an EC2 instance i-a1b2c3d4
Current Configuration Information returned through a call to the Describe or List API of the resource
e.g. for EBS VolumeState of DeleteOnTermination flagType of volume. For example, gp2, io1, or standard
Related Events The AWS CloudTrail events that are related to the current configuration of the resource
AWS CloudTrail event ID
Configuration Item
Sample Configuration Item "configurationItemVersion": "1.0", "configurationItemCaptureTime": "2014…", "configurationStateID": “….", "configurationItemStatus": "OK", "resourceId": "vol-ce676ccc", "arn": "arn:aws:us-west-………", "accountId": "12345678910", "availibilityZone": "us-west-2b", "resourceType": "AWS::EC2::Volume", "resourceCreationTime": "2014-02..", "tags": {}, "relationships": [ { "resourceId": "i-344c463d", "resourceType": "AWS::EC2::Instance", "name": "Attached to Instance" } ],
"relatedEvents": [ "06c12a39-eb35-11de-ae07-db69edbb1e4", ],
Metadata
Common Attributes
Relationships
Related Events
Sample Configuration Item"configuration": { "volumeId": "vol-ce676ccc", "size": 1, "snapshotId": "", "availabilityZone": "us-west-2b", "state": "in-use", "createTime": "2014-02-……", "attachments": [ { "volumeId": "vol-ce676ccc", "instanceId": "i-344c463d", "device": "/dev/sdf", "state": "attached", "attachTime": "2014-03-", "deleteOnTermination": false } ], "tags": [ { "tagName": "environment", "tagValue": "PROD"
Configuration
Having this data is one thing, but how can we react to change?
https://www.flickr.com/photos/livenature/204420128/
CloudTrail – Amazon CloudWatch Logs Integration
CloudTrail – Amazon CloudWatch Logs Integration
Trigger a CloudWatch Alarm based on API call!
Check configuration changes
Pre-built rules provided by AWS
Custom rules using AWS Lambda
Continuous assessment
Dashboard Compliance visualization Identify offending changes
GitHub repo: Community sourced rules
AWS Config Rules
Normalize
RecordChanging Resources
AWS Config & Config RulesDeliver
Stream
Snapshot (ex. 2014-11-05)AWS Config
APIs
Store
History
Rules
Config Rules
Config Rules
Config Rules GitHub repository
AWS Management Tools Partners
FIN, ACKWe’ve seen a quick run through today of the ways you can improve your governance on AWS: Treat your infrastructure and host configuration as code! This lends itself to being able to use services like Service
Catalog to enable self-service in your organization Track, trend, and alert on CloudTrail API logs to keep on top
of access to your AWS resources Use Config and Config Rules to understand the relationship
between resources and react to policy violations Putting all this together is what gives you Policy as Code!
Browse and launch
AWS ConfigAWS CloudTrail
Use and modify
Users Admin
Putting the AWS Management services together
AWS Service Catalog
Provision with Tags
API calls Configuration checks and reactions to change
Troubleshoot and Audit
But wait, there’s more!
Resources to learn more: More on DevOps: https://aws.amazon.com/devops/ AWS Management Services: https://aws.amazon.com/products/management/ AWS CloudFormation
https://aws.amazon.com/cloudformation/ AWS Service Catalog
https://aws.amazon.com/servicecatalog/ AWS CloudTrail
https://aws.amazon.com/cloudtrail/ AWS Config / Config Rules
https://aws.amazon.com/config/ GitHub repo: https://github.com/awslabs/aws-config-rules
Thank you!
Happy Deploying!