architecting for the cloud storage build test

Download Architecting for the cloud storage build test

Post on 27-Aug-2014




1 download

Embed Size (px)


This is more of Day 5 covering buld and test


  • Architecting for the Cloud Len and Matt Bass Build/Test
  • Overview What is the problem? Deployment pipeline Integration testing Staging
  • Goal of Developers Get code into production with minimum errors and maximum speed In this lecture we are concerned with Errors caused by working in a team Errors caused by having multiple versions of a system simultaneously Automating the testing process Errors equate to delays since it takes time to fix them. Between team coordination is another source of delays. We have already seen a coordination-free architectural style.
  • Errors caused by working in a team Consider the following scenarios A team member suddenly leaves. One team member overwrites the code of another team member Code written by one team member must be integrated with code written by another Preventing these errors is the purpose of a configuration management tool.
  • Errors caused by having multiple versions of a system simultaneously Consider the following scenarios A version of the system is in production and a new version is being tested A problem develops in the production version and The problem can be fixed without affecting either the production version or any development versions Two different members of the team are working on different versions of the system Preventing these errors is the purpose of a version control system
  • Branches Version control systems Contain a main trunk Support the creation of branches for particular sets of modifications. Branches allow for independent evolution of different streams of work. Might have production version, version being UAT tested, version with new features, version with bug fix being tested Once a branch has been released into production, it should be merged back into the trunk May be time consuming depending on how trunk has evolved since the creation of the branch (delay). Best practice is to restrict the number of branches.
  • Overview What is the problem? Deployment pipeline Integration testing Staging
  • The Deployment Pipeline Version control and configuration management are combined into a continuous integration server. Build and integration tests are performed as a portion of continuous integration User Acceptance Testing and performance testing are performed as the last stage before deploying to production. We will call these steps staging. A developer commits to a branch and the remainder of the deployment pipeline is for that branch.
  • Environments Before we worry about delivery, we need to worry about the steps preceding it. An environment is a set of software running on a collection of virtual machines with defined external connections. Entry is always through a load balancer External references are through URLs An environment
  • Changing environments Changing the external connections changes the behavior of the system. The difference between using a production database and a test database is the URL of the database. Production Database Test Database
  • Developer tests locally and commits latest version of their module to version control system Build is triggered that draws latest version of each module used from version control system If build is successful a machine image is stored in the build environment. If build unsuccessful, a message is sent to the development team. If developers module does not compile then not only that build will break but builds by any team member whose module depends on the newly committed module will also break. (delay) What happens when a developer commits a change to a branch? Build environment Success Failure Pre-committests(local) VersionControlSystemPre-committests(local) ... CompileandBuild Commit
  • Overview What is the problem? Deployment pipeline Integration testing Staging
  • Integration Tests If the newly committed module does not pass testing then testing any commit by other team members will also fail. Continuous integration servers can be used to test a build prior to committing. Fixing a broken build or broken integration test is high priority. Tests can use copy of production environment for services not a portion of newly built process. URLs used to specify location of services build environment copy of production environment Build Environment Test Harness Test Input Test output Copy of Production Environment
  • Managing the database for integration testing Integration testing will read and write a database. Use a separate test database to prevent impacting the production database The test database is preloaded with test data. Executing the tests will produce writes to the database. Check the state of the database as a portion of verifying the tests.
  • Complications with test database The initial state of the database may not be the same from one test sequence to another because tests write to the database Reinitialize every time a test sequence is run The sequence of tests is important. Some tests may be dependent on the results of other tests Create account must be done before Add money to account The correctness of the database after tests are run must be tested by a special utility designed for that purpose.
  • Feature Toggles (switches/flags) A feature toggle is an alternative to creating a new branch If toggle is on then execute new code else execute old code Widely used in internet companies such as Facebook, Google, Netflix. Requires feature toggle manager to coordinate the setting of global values for feature toggles. Archaius is open source system released by Netflix that, among other things, acts as a feature toggle manager.
  • Pluses and Minuses of Feature Toggles Pluses Allows uncompleted code to be checked in. Toggle feature off and it will not be executed during test and, hence, will not break the build Allows for flexible deployment process. We will see this later. Minuses Clutters code. Misuse of feature toggle can have serious consequences. Knight Capital lost (US) $440 Million in 45 minutes partially as a result of a repurposed feature toggle. Best practice: Remove feature toggles once the code has been committed to production.
  • Overview What is the problem? Deployment pipeline Integration testing Staging
  • Staging Testing After successful integration testing Machine image does not change from successful integration testing Mirror input from production system to create an input stream for performance testing Production system input stream will not test new features. Additional tests are required for new features. Staging Environment Production Environment
  • Staging Testing Database Make copy of production database. Assumes reasonable size production database Some cloud vendors make hourly back ups of production database, can use one of these as test. Ensure writes to database do not affect production database After test completes you will need to check test database for correctness.
  • Where do tests come from? Test Driven Development advocates writing tests before developing a feature. Results in a collection of tests Regression tests past problems with the system result in a collection of test cases Edge/boundary cases how well are edge cases handled? Error cases what happens when erroneous input occurs?
  • Additional tests Static analyzers Analyze code for known problems. E.g. SQL injection, buffer overflow Detect some problems at code level with modularity Dynamic analyzers Mainly for security Penetration testing. Looks for back door ways into a system.
  • Continuous Deployment/Delivery Continuous deployment and continuous delivery are two terms meaning almost the same thing. Both terms imply automated movement between stages of the deployment pipeline. Continuous delivery: a developers commit triggers a sequence that automatically results either in failure to pass tests or in a machine image prepared for production, i.e. in UAT environment. Continuous deployment: a developers commit triggers a sequence that automatically results either in failure to pass tests or in a machine image being placed into production, i.e. moved from UAT into production.
  • Continuous Delivery Not all organizations can adopt continuous delivery. Banks, for example, require that a human certify any release A human examines the output of the AUT/Performance testing to ensure compliance with various financial regulations Continuous delivery assumes a go


View more >