testing on aws - aws il meetup
TRANSCRIPT
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Boaz Ziniman, Technical Evangelist, AWS
@ziniman
Testing on AWSIsrael AWS User Group – January 2018
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Make your pipeline safer
1. Automating tests to catch regressions early
2. Identifying production issues quickly
3. Deploying changes safely
4. Automatically deciding when to release changes
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous production testing
3. Manage deployment health
4. Segment production
5. Halt promotions
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Tools used for Deployment and Testing
MonitoringAmazon CloudWatch
Software DevelopmentAmazon SNS
AWS Lambda
Build & TestAWS CodeBuild
DeploymentAWS CodeDeploy
AWS CodePipeline
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MyAppCodeCommit
Source
BuildCodeBuild
Build
DeployToIntegCodeDeploy
Integration
IntegTestEnd2EndTester
DeployToProdCodeDeploy
Production
Source
Build
Deploy to Integration Stack
Integration Tests
Deploy to Production
Model the release process in CodePipeline
Pipeline RunAction
Stage
Pipeline
Source change • starts a run and• creates an artifact to be used by
other actions.
Change 1
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Release and deploy process: Starting point
MyAppCodeCommit
Source
BuildCodeBuild
Build
DeployToIntegCodeDeploy
Integration
IntegTestEnd2EndTester
DeployToProdCodeDeploy
Production
CodeDeploy
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. Automated testing2. Continuous production testing3. Manage deployment health 4. Segment production5. Halt promotions
Techniques
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Be aware when a code change fails tests
Problem: A code change can fail automated tests without developer knowledge.
Consequence:Your pipeline becomes block and changes stop flowing to production.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Integrate testing into your pipeline
1. Run unit tests during build process
2. Deploy to integration environment
3. Run integration and UI tests
4. Notify developer chat room if the build or tests fail
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Configure CodeBuild
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. Trigger pipeline on source change2. Build and unit tests3. Deploy to integration environment4. Execute UI tests5. Execute integration tests
Tests
Step 1: Build and unit testsSource
MyAppSourceCodeCommit
Build
BuildAndUnitTestsCodeBuild
IntegrationDeployCodeDeploy
TestOnChromeCodeBuild
TestOnChromeCodeBuild
IntegTestEnd2EndTester
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Configure CloudWatch Event notifications
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 2: Notify on failed build and test
Lambda FunctionNotifySlackOnPipelineActionFailure()
Change 1
CloudWatchEvents
(Failed Action)
Tests
Source
MyAppSourceCodeCommit
Build
BuildAndUnitTestsCodeBuild
IntegrationDeployCodeDeploy
TestOnChromeCodeBuild
TestOnChromeCodeBuild
IntegTestEnd2EndTester
Change 2
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous production testing
3. Manage deployment health
4. Segment production
5. Halt promotions
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use synthetic traffic to simulate real users
• Test all business critical functionality (UI and APIs)
• Tests must run quickly
• Measure client latencies
• Check for reachability
• Alerts and Notifications
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous production testing
3. Manage deployment health
4. Segment production
5. Halt promotions
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Add safety to rolling deployments
1. Validate each host’s health
2. Ensure a minimum percentage of the fleet is healthy
3. Rollback if the deployment failed
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous production testing
3. Manage deployment health
4. Segment production
5. Halt promotions
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lower deployment risk by segmenting
1. Break production into multiple segments• Single Host• AZ• Regions
2. Deploy to a segment
3. Test a segment after a deployment
4. Repeat 2 & 3 until done
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous production testing
3. Manage deployment health
4. Segment production
5. Halt promotions
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Don’t compound problems during an outage
Pipeline Problem: The pipeline is unaware of the health of the infrastructure it is deploying to
Consequence: Production changes, usually deployments, can make it difficult for an operator to resolve a production event
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source
MyAppSourceCodeCommit
Build
MyAppBuildCodeBuild
DeployToProd
MyAppCodeDeploy
Change 1Change 2
Automatically stop deploying to production during an event
CloudWatch SyntheticTraffic
deployschecks
CloudWatchEvents (1m)
triggers
emitsdisables
disableTransition() CW alarm
EC2 instance
SNS
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Code is available online
• github.com/awslabs/aws-codepipeline-synthetic-tests
• github.com/awslabs/aws-codepipeline-block-production
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Related re:Invent Sessions
• DEV322-R - Continuous Integration Best Practices for
Software Development Teams
• DEV343 - How to Reduce Deployment Failures through
Code Reviews and Continuous Integration
• DEV325-R - Application Deployment Techniques for
Amazon EC2 Workloads with AWS CodeDeploy