architecting for the cloud storage build test

30
Architecting for the Cloud Len and Matt Bass Build/Test

Upload: len-bass

Post on 27-Aug-2014

362 views

Category:

Software


2 download

DESCRIPTION

This is more of Day 5 covering buld and test

TRANSCRIPT

Page 1: Architecting for the cloud storage build test

Architecting for the Cloud

Len and Matt Bass

Build/Test

Page 2: Architecting for the cloud storage build test

Overview

• What is the problem?

• Deployment pipeline

• Integration testing

• Staging

Page 3: Architecting for the cloud storage build test

Goal of Developers

• Get code into production with minimum errors and maximum speed

• In this lecture we are concerned with – Errors caused by working in a team

– Errors caused by having multiple versions of a system simultaneously

– Automating the testing process

• Errors equate to delays since it takes time to fix them.

• Between team coordination is another source of delays. We have already seen a coordination-free architectural style.

Page 4: Architecting for the cloud storage build test

Errors caused by working in a team

• Consider the following scenarios

– A team member suddenly leaves.

– One team member overwrites the code of another team member

– Code written by one team member must be integrated with code written by another

• Preventing these errors is the purpose of a configuration management tool.

Page 5: Architecting for the cloud storage build test

Errors caused by having multiple versions of a system simultaneously

• Consider the following scenarios – A version of the system is in production and a new

version is being tested

– A problem develops in the production version and • The problem can be fixed without affecting either the

production version or any development versions

– Two different members of the team are working on different versions of the system

• Preventing these errors is the purpose of a version control system

Page 6: Architecting for the cloud storage build test

Branches

• Version control systems – Contain a main “trunk” – Support the creation of branches for particular sets of

modifications.

• Branches allow for independent evolution of different streams of work. – Might have production version, version being UAT tested,

version with new features, version with bug fix being tested

• Once a branch has been released into production, it should be merged back into the trunk – May be time consuming depending on how trunk has evolved

since the creation of the branch (delay).

• Best practice is to restrict the number of branches.

Page 7: Architecting for the cloud storage build test

Overview

• What is the problem?

• Deployment pipeline

• Integration testing

• Staging

Page 8: Architecting for the cloud storage build test

The Deployment Pipeline

Version control and configuration management are combined into a continuous integration server. • Build and integration tests are performed as a portion of

continuous integration • User Acceptance Testing and performance testing are performed as

the last stage before deploying to production. We will call these steps staging.

• A developer commits to a branch and the remainder of the deployment pipeline is for that branch.

Page 9: Architecting for the cloud storage build test

Environments

• Before we worry about delivery, we need to worry about the steps preceding it.

• An environment is a set of software running on a collection of virtual machines with defined external connections. – Entry is always through a load

balancer – External references are through

URLs An environment

Page 10: Architecting for the cloud storage build test

Changing environments

• Changing the external connections changes the behavior of the system.

• The difference between using a production database and a test database is the URL of the database.

Production Database Test Database

Page 11: Architecting for the cloud storage build test

• Developer tests locally and commits latest version of their module to version control system

• Build is triggered that draws latest version of each module used from version control system

• If build is successful a machine image is stored in the build environment. • If build unsuccessful, a message is sent to the development team. • If developer’s module does not compile then not only that build will break but

builds by any team member whose module depends on the newly committed module will also break. (delay)

What happens when a developer commits a change to a branch?

Build environment

Success

Failure

Pre-commit tests (local)Version Control SystemPre-commit tests (local)

...

Compile and Build

Commit

Page 12: Architecting for the cloud storage build test

Overview

• What is the problem?

• Deployment pipeline

• Integration testing

• Staging

Page 13: Architecting for the cloud storage build test

Integration Tests

• If the newly committed module does not pass testing then testing any commit by other team members will also fail.

• Continuous integration servers can be used to test a build prior to committing.

• Fixing a broken build or broken integration test is high priority.

• Tests can use copy of production environment for services not a portion of newly built process.

• URLs used to specify location of services – build environment – copy of production environment Build

Environment

Test Harness

Test Input

Test output

Copy of Production Environment

Page 14: Architecting for the cloud storage build test

Managing the database for integration testing

• Integration testing will read and write a database.

• Use a separate test database to prevent impacting the production database

• The test database is preloaded with test data.

• Executing the tests will produce writes to the database.

• Check the state of the database as a portion of verifying the tests.

Page 15: Architecting for the cloud storage build test

Complications with test database

• The initial state of the database may not be the same from one test sequence to another because tests write to the database • Reinitialize every time a test sequence is run

• The sequence of tests is important. Some tests may be dependent on the results of other tests • Create account must be done before • Add money to account

• The correctness of the database after tests are run must be tested by a special utility designed for that purpose.

Page 16: Architecting for the cloud storage build test

Feature Toggles (switches/flags)

• A feature toggle is an alternative to creating a new branch

• If “toggle is on” then execute new code else execute old code

• Widely used in internet companies such as Facebook, Google, Netflix.

• Requires feature toggle manager to coordinate the setting of global values for feature toggles. – Archaius is open source system released by Netflix that,

among other things, acts as a feature toggle manager.

Page 17: Architecting for the cloud storage build test

Pluses and Minuses of Feature Toggles

• Pluses – Allows uncompleted code to be checked in. Toggle feature

off and it will not be executed during test and, hence, will not break the build

– Allows for flexible deployment process. We will see this later.

• Minuses – Clutters code. – Misuse of feature toggle can have serious consequences.

Knight Capital lost (US) $440 Million in 45 minutes partially as a result of a repurposed feature toggle.

• Best practice: Remove feature toggles once the code has been committed to production.

Page 18: Architecting for the cloud storage build test

Overview

• What is the problem?

• Deployment pipeline

• Integration testing

• Staging

Page 19: Architecting for the cloud storage build test

Staging Testing

• After successful integration testing

• Machine image does not change from successful integration testing

• Mirror input from production system to create an input stream for performance testing

• Production system input stream will not test new features. Additional tests are required for new features. Staging

Environment Production Environment

Page 20: Architecting for the cloud storage build test

Staging Testing Database

• Make copy of production database.

– Assumes reasonable size production database

– Some cloud vendors make hourly back ups of production database, can use one of these as test.

• Ensure writes to database do not affect production database

• After test completes you will need to check test database for correctness.

Page 21: Architecting for the cloud storage build test

Where do tests come from?

• Test Driven Development advocates writing tests before developing a feature. – Results in a collection of tests

• Regression tests – past problems with the system result in a collection of test cases

• Edge/boundary cases – how well are edge cases handled?

• Error cases – what happens when erroneous input occurs?

Page 22: Architecting for the cloud storage build test

Additional tests

• Static analyzers

– Analyze code for known problems. E.g. SQL injection, buffer overflow

– Detect some problems at code level with modularity

• Dynamic analyzers

– Mainly for security

– Penetration testing. Looks for back door ways into a system.

Page 23: Architecting for the cloud storage build test

Continuous Deployment/Delivery

• Continuous deployment and continuous delivery are two terms meaning almost the same thing.

• Both terms imply automated movement between stages of the deployment pipeline.

• Continuous delivery: a developer’s commit triggers a sequence that automatically results either in failure to pass tests or in a machine image prepared for production, i.e. in UAT environment.

• Continuous deployment: a developer’s commit triggers a sequence that automatically results either in failure to pass tests or in a machine image being placed into production, i.e. moved from UAT into production.

Page 24: Architecting for the cloud storage build test

Continuous Delivery

• Not all organizations can adopt continuous delivery. – Banks, for example, require that a human certify any release – A human examines the output of the AUT/Performance testing

to ensure compliance with various financial regulations

• Continuous delivery assumes a good test suite. • Some organizations will use continuous delivery with quick

roll backs in case of errors (live testing). • Continuous delivery

– Virtues – reduces manual intervention and speeds up the delivery process.

– Drawbacks - implies adherence to a number of constraints that we will discuss.

Page 25: Architecting for the cloud storage build test

Configuration Parameters

• All external references from the environment are specified by URLs – These constitute the difference between operating in a build

environment, the UAT environment, or the production environment

• URLs for the different environments can be specified in a database. • Other configuration parameters can also be specified in the database.

These may vary based on environment or circumstances. E.g. – Feature toggle values – Maximum number of virtual machines in autoscaling group.

• At start up, the system is informed of the location of its configuration parameters, it reads them in, and uses them to perform its duties.

Page 26: Architecting for the cloud storage build test

Tools and Scripts

• Specialized tools exist for the different stages of the deployment pipeline

– Continuous integration tools

– Environment specification and migration tools

• Programs for these tools are called “scripts”

• One (and only one) script should exist that invokes a system and tells it where to find its environmental parameters.

Page 27: Architecting for the cloud storage build test

Traceability

• Everything to goes into a component in production should be traceable – What version of which modules are included in the

component – What version of the database is being accessed – What version of dependent systems are being used – What configuration parameters were set – What script was used to build the environment and

place the system into production. – What values of the features toggles were used. – …

Page 28: Architecting for the cloud storage build test

Achieving traceability

• All of the information could be available from a version control system

• This requires that everything be version controlled – code, scripts, and dependent systems.

• During operational problems, there is a tendency to modify the production system via the console. This is not traceable.

• Best practice is to lock down the environment creation and modification process to authorized users and require everyone to go through the process of modifying a script and checking it in before using it.

Page 29: Architecting for the cloud storage build test

Architectural implications of build and test

• Use feature toggles

• Interact with external world only through URLs

• Small modules are easier to test.

Page 30: Architecting for the cloud storage build test

Summary

• The deployment pipeline describes the steps and the processes to move code from development to production – Build – Staging – Production

• Associating an environment with each step and restricting interaction from within the environment to outside the environment to URLs allows simpler movement through the steps

• Continuous deployment has movement through all of the steps automated

• Everything should be configuration controlled and access to the production system should be restricted to configuration controlled items.