issues with the use of canaries in upgrade
DESCRIPTION
TRANSCRIPT
NICTA Copyright 2012 From imagination to impact
Issues with the use of Canaries
Len Bass
NICTA Copyright 2012 From imagination to impact
Goal of Presentation
• For you to understand there are interesting problems associated with canaries.
• For me to be sure that I understand how canaries are used in practice.
• For me to get feedback about appropriate testing results.
2
NICTA Copyright 2012 From imagination to impact
Various Upgrade Strategies
• How many at once?– One at a time (rolling upgrade)– Groups at a time (staged upgrade, e.g. canaries)– All at once (big flip)
• What happens to old versions?– Replaced en masse– Maintained for some period for compatibility purposes
• This talk will focus on the canary strategy and examine some issues associated with canaries
3
NICTA Copyright 2012 From imagination to impact
Context
• Deep service dependency hierarchy – may be 70 deep
• Upgrading one service in this hierarchy
• Need to consider both service and its clients
4
Figure from Netflix Tech Blog
NICTA Copyright 2012 From imagination to impact
Current state of major internet provider
• Each service has an owner• Every service instance is instrumented• When a canary is deployed, service owner
examines monitoring data (next slide) and uses judgment to decide when to move to production.
• Canary testing is currently based on functionality. No stress testing of canaries.
• Research question – what scientific criteria can be used to make judgment of when to go into production?
5
NICTA Copyright 2012 From imagination to impact
Netflix Monitoring Sequence
6
• Client outbound (start/end)• Network (start/end)• Service network (inbound start/end)• Service processing (start/end)• Service outbound (start/end)• Network (start/end)• Client inbound (start/end)
NICTA Copyright 2012 From imagination to impact
Common upgrade strategy
• Require all versions to be backward compatible with previous versions
• Require changes associated with new version to be software switchable.
• Clients of a service must be version aware in order to know whether to utilize new functionality.
• Once all instances have been upgraded to new versions, send signal to turn on changes both in the new version and their clients.
• When using canaries only turn on changes for a subset of services and their clients.
7
NICTA Copyright 2012 From imagination to impact
Canary Issues
• Canaries are a form of live testing. Put a new version into limited production to test its correctness.
• Issues– How long are new versions tested to determine
correctness?• Period based – for some period of time • Load based – under some utilization assumptions • Result based – until some criteria is met
– How are clients of new version chosen and how is this choice enforced?
– How are the canaries deployed?
8
NICTA Copyright 2012 From imagination to impact
General Picture
Client
Top Level load
balancer
Second level load balancer
Server for Version A
Server for Version A
Server for Version B
Second level load balancer
Server for Version A
Server for Version B
9
Client• Version aware
• Must know about new versions In order to take advantage of new functionality
• May be implicitly version aware based on, e.g. cluster
• Version unaware clients will only use old functionality and these can be served by any server since services are backward compatible.
In addition:• Load variation may
trigger elasticity rules. • Deciding whether to
load new version or old version raises other issues.
NICTA Copyright 2012 From imagination to impact
More Detail on Upgrade Process
• Canaries are deployed and allowed to run for a period without turning on new features.
• This is to test backward compatibility.• Once canaries pass this test, then the new
features are turned on.
10
NICTA Copyright 2012 From imagination to impact
Question 1 – how are clients messages routed?
• Three cases:1. Clients are separated, a priori, into those utilizing
new version and those not.
2. Messages are routed arbitrarily by load balancer and those that are received by new version of service cause client to be designated as utilizing new version.
3. All services are capable of being old version or new version and choose based on version of message they receive. (seems contrary to canary strategy)
11
NICTA Copyright 2012 From imagination to impact
More Questions
2. After turning on new functionality, how does one decide that the canaries have been sufficiently functionally tested for the fixed set of clients
– Are there results from the testing community that pertain here? I don’t know.
– After answering this question, one can add additional clients to those being routed to Version B until the metric available(?) from the testing community passes some threshold.
3. How can one perform stress testing in a live environment?
– We are examining a metric called “Performance Nonscability Likelihood” for its applicability
12
NICTA Copyright 2012 From imagination to impact
Summary• We have identified the problem of determining when canary testing
is adequate as one that could use more rigor.• Multiple different strategies for connecting new version clients to
new version services • Outstanding questions are
– How long before all of the benefit of using canaries has been realized and the new functionality can be turned on?
– How is stress testing performed?
13