how we tripled our load in production for testing...

How we tripled our load in productionHow we tripled our load in productionHow we tripled our load in productionHow we tripled our load in productionfor testing purposesfor testing purposesfor testing purposesfor testing purposesand survived to tell the story!and survived to tell the story!and survived to tell the story!and survived to tell the story!

(and how we did it with AWS)(and how we did it with AWS)(and how we did it with AWS)(and how we did it with AWS)

Carlos Arguelles, Senior SDETDan-Constantin Florescu, SDETWebsite Applications Platform

AgendaAgendaAgendaAgenda

• Why we did this

• What choices we considered

• Why we picked ours

• How we implemented it

• Q&A

What I’d like you to get out of thisWhat I’d like you to get out of thisWhat I’d like you to get out of thisWhat I’d like you to get out of this

• Start thinking about Load testing (nownownownow)

• Large-scale load testing in Production is an option

• Key questions you can use to guide your testing

• What you can do as a service owner to design more testable services TIP

Why we did thisWhy we did thisWhy we did thisWhy we did this

• Load Testing…

– Every team says “It’d be nice to do…”

– But often it doesn’t get enough priority

• Large scale load testing

– Is expensive (hardware, SDE-time)

• Large scale load testing in production

– Is risky (downtime, customer data loss)

Why we did thisWhy we did thisWhy we did thisWhy we did this

• During Black Friday 2010, one of our services had a number of issues due to scaling problems

–Customer impact: Many services across Amazon unable to make real-time data-driven decisions during one of the most important shopping day of the year

Why we did Why we did Why we did Why we did thisthisthisthis

Black FridayBlack FridayBlack FridayBlack FridayCyber MondayCyber MondayCyber MondayCyber Monday

How we designed the How we designed the How we designed the How we designed the Load TestLoad TestLoad TestLoad Test

• We made a list of questions list of questions list of questions list of questions to guide us

– No right/wrong answer, no “Golden Formula”

– Cost/risk/benefitCost/risk/benefitCost/risk/benefitCost/risk/benefit for each decision

– Product specific

Open QuestionsOpen QuestionsOpen QuestionsOpen Questions

• What do you really want to do?What do you really want to do?What do you really want to do?What do you really want to do?

• What is the architecture and data flow?

• What are the dependencies?

• Where should we test?

• How long?

• What data? How much?

• How do I assess the impact of the load?

• How to scale the load gen?

• How do I ensure the right pace?

• How do I perform an emergency-stop?

What do you really want to do?What do you really want to do?What do you really want to do?What do you really want to do?

Performance Testing

Performance Testing

ResilienceTesting

ResilienceTesting

StressTestingStressTesting

LoadTestingLoadTesting

Figure out what you *really* want to do before you start testing!


Performance Testing

LoadLoadLoadLoadTestingTestingTestingTesting

StressTesting

ResilienceTesting

Can it handle an expected realistic load?

How much hardware do I need for my peak?

Miss SLAs? (i.e. Latency)System down?Data lossErrors…

Emphasis on “prod“prod“prod“prod----like”like”like”like”behavior


Performance Testing

LoadTesting

StressStressStressStressTestingTestingTestingTesting

ResilienceTesting

When & Where will it break?

Bottlenecks?Contention? Locks?

Not on “prod“prod“prod“prod----like”like”like”like”behavior

Emphasis on breaking the breaking the breaking the breaking the systemsystemsystemsystem


Performance Testing

LoadTesting

StressTesting

ResilienceResilienceResilienceResilienceTestingTestingTestingTesting

• How will it handle underlying failures?

• Data center down• Slow network• Dependencies failing• CPU/Memory hogs• …


Performance Performance Performance Performance TestingTestingTestingTesting

LoadTesting

StressTesting

ResilienceTesting

• How does latency & throughput change as load increases?

• When will you miss latency SLAs?


• What do you really want to do?

• What is the architecture and data flow?What is the architecture and data flow?What is the architecture and data flow?What is the architecture and data flow?



• How long?






Architecture @ 30,000 Architecture @ 30,000 Architecture @ 30,000 Architecture @ 30,000 feetfeetfeetfeet

Upstream customers posting logs(Retail fleet)

Our serviceOur serviceOur serviceOur service

Downstream customersconsuming processed data

What does “Load” mean to this architecture?

• Transactions per second?Transactions per second?Transactions per second?Transactions per second?• Concurrent connections?Concurrent connections?Concurrent connections?Concurrent connections?• Terabytes per day?Terabytes per day?Terabytes per day?Terabytes per day?

Architecture @ 30,000 Architecture @ 30,000 Architecture @ 30,000 Architecture @ 30,000 feetfeetfeetfeet

Upstream customers Upstream customers Upstream customers Upstream customers posting posting posting posting logslogslogslogs(Retail fleet)(Retail fleet)(Retail fleet)(Retail fleet)

Our serviceOur serviceOur serviceOur service

• 1 file per host per minute• tens of thousands of hosts• file size 200KB-2MB• 500MB/sec at peak

• TPS is low and somewhat constant

• Payload could be larger on days with high load

• Will be tens of terabytes on peak day

Architecture Architecture Architecture Architecture @ 10,000 @ 10,000 @ 10,000 @ 10,000 feetfeetfeetfeet

Data inputData inputData inputData input

Data processingData processingData processingData processing

Data outputData outputData outputData output




• What are the dependencies?What are the dependencies?What are the dependencies?What are the dependencies?


• How long?






What are my dependencies?What are my dependencies?What are my dependencies?What are my dependencies?

DependenciesDependenciesDependenciesDependencies

Host 1 Host 2

Production Load

Load Balancer

… Host nHost 1 Host 2

Production Load

Load Balancer

… Host n

If you’re loading your service,

you’re loadingyour dependencies

DependenciesDependenciesDependenciesDependencies

Test Load


• You can’t always control them

–convince them to load-test with you

–you might need to mock them


• Mocking your dependencies

– can be expensive to do

• percentile latencies

• error rates, throttling, …

+ but you can model theoretical situations





• Where should we test?Where should we test?Where should we test?Where should we test?

• How long?






Component testing vs. EndComponent testing vs. EndComponent testing vs. EndComponent testing vs. End----totototo----endendendend

• Component testing: Component testing: Component testing: Component testing:

+ very targeted

+ localized risk

- component interaction?

- expensive mocking?

Can your components be load-tested separately?TIP

Test Environment vsTest Environment vsTest Environment vsTest Environment vs. . . . ProductionProductionProductionProduction

• “Shadow” testing: “Shadow” testing: “Shadow” testing: “Shadow” testing:

+ breaking point

- smaller scale

- configuration

Big brother, Production

Little brother, Test Environment

• But to truly show that our Production fleet was ready…

– test in Productiontest in Productiontest in Productiontest in Production

– while processing regular traffic while processing regular traffic while processing regular traffic while processing regular traffic

– triple the load of a regular triple the load of a regular triple the load of a regular triple the load of a regular daydaydayday

• High risk, high benefit

Where should we test? Where should we test? Where should we test? Where should we test?





• How long? How long? How long? How long?






Open Open Open Open QuestionsQuestionsQuestionsQuestions

• A day in our service’s life

How How How How long long long long do we run fordo we run fordo we run fordo we run for? ? ? ?

We process it:- 10-Min jobs- Hourly jobs- Daily jobs

• Should we run it for more than a day?

How How How How long do we run for? long do we run for? long do we run for? long do we run for?

Cyber

Monday

Black

Friday

Week of Black Friday

Load Tests

Regular days

before load tests

• Should we run it for more than a day?

– may start failing under load after a longer period of time (such as resource leaks)

– Infrequent failures are more likely to occur if we run it for a longer period of time

• CostCostCostCost/risk/benefit/risk/benefit/risk/benefit/risk/benefit

How How How How long do we run for? long do we run for? long do we run for? long do we run for?





• How long?

• What data? How much?What data? How much?What data? How much?What data? How much?






Allow a way to send test data to your production service

What data?What data?What data?What data?

TIP

Have to make sure that the Load test data does not pollute Production data!

• Need process to save production input

– Useful for a lot more than load testing!

• Open questions:

• % of hosts saving data?

• % of transactions sampled?

• Safe to replay? (i.e. idempotency?)

What data?What data?What data?What data?

Saving production dataSaving production dataSaving production dataSaving production data

Load Balancer

InterceptorInterceptorInterceptorInterceptor

Host 1 Host 2 Host 3 … Host n

Production Data Storage

Host 1

Saving production dataSaving production dataSaving production dataSaving production data

Load Balancer

Host 2 Host 3 … Host n

InterceptorInterceptorInterceptorInterceptorInterceptorInterceptorInterceptorInterceptor

Production Data Storage

How much data?How much data?How much data?How much data?

Data pattern: POST payload

P90

P50

P99.9

TIP Monitoring is your friend: what metrics will help you?

Aug 2011

Nov 2010

How much data?How much data?How much data?How much data?

Data pattern: Number of POSTs





• How long?


• How How How How do I assess the impact of the load?do I assess the impact of the load?do I assess the impact of the load?do I assess the impact of the load?





Hardware metricsHardware metricsHardware metricsHardware metrics

CPU MetricsCPU MetricsCPU MetricsCPU Metrics

Regular day in September

1st attempt at running our test(cancelled)

2nd attempt at running our test (successful)

Need to have relevant monitoring in place that will give you accurate state of the system (and tell you when it’s at risk)

ProductProductProductProduct----specific metricsspecific metricsspecific metricsspecific metrics

� Hadoop Cluster: % of mappers in use

Hadoop Cluster: % of reducers in use �





• How long?



• How How How How to scale the load gen?to scale the load gen?to scale the load gen?to scale the load gen?




Architecture of the Load GeneratorArchitecture of the Load GeneratorArchitecture of the Load GeneratorArchitecture of the Load Generator

• Process saves % of production data to a “Test Data Repository” (TDR) with date/time metadata

• TDR can list all files that belong to an interval

• Can replay:• A percentage of a desired day• Multiple days at the same time

ProductionService

Test DataRepository

(S3)

Architecture of the Load GeneratorArchitecture of the Load GeneratorArchitecture of the Load GeneratorArchitecture of the Load Generator

Test DataRepository

ProductionService

ControllerControllerControllerControllerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorker

ControllerControllerControllerController

JobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJobJob

WorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorkerWorker

S3 for data

SQS for state, resilience

EC2 for hardware

CloudWatchAutoScaling

AWS AWS AWS AWS services for load services for load services for load services for load testingtestingtestingtesting

• Reactive auto-scaling

– Auto-scaling based on CPU/Memory/…

– If the SQS queue grows launch more instances

• Predictive auto-scaling

– Can read ahead in the TestDataRepository and knows if the load will be increasing or decreasing in half an hour





• How long?




• How How How How do I ensure the right pace? do I ensure the right pace? do I ensure the right pace? do I ensure the right pace?



How do I ensure that I’m applying the load How do I ensure that I’m applying the load How do I ensure that I’m applying the load How do I ensure that I’m applying the load at the right pace?at the right pace?at the right pace?at the right pace?

expected

actual

How do I ensure that I’m applying the load How do I ensure that I’m applying the load How do I ensure that I’m applying the load How do I ensure that I’m applying the load at the right pace?at the right pace?at the right pace?at the right pace?





• How long?





• How How How How do I perform an emergencydo I perform an emergencydo I perform an emergencydo I perform an emergency----stop?stop?stop?stop?


Safety Measures: Big Red ButtonSafety Measures: Big Red ButtonSafety Measures: Big Red ButtonSafety Measures: Big Red Button

• Deactivate the Load Generator

– Kill/stop the workers, controller

– Delete SQS queue• if there’s no queue there’s no work

TIP Have redundant safety measures!Have redundant safety measures!Have redundant safety measures!Have redundant safety measures!

Safety MeasuresSafety MeasuresSafety MeasuresSafety Measures

• Situation:

– Workers can’t dequeue jobs for a while

– Job queue grows very large

– When workers resume dequeing…

• Solution:

– Queue size > X �Alarm

– Queue size > Y �Auto shutdown

Problems our test preventedProblems our test preventedProblems our test preventedProblems our test prevented

• We were under-scaled by ~15%

– Processing would have fallen behind: missed SLAs

• Leak affecting files > 1 MB

– All hosts in service would have run out of disk space in hours: downtime

• These issues would not have been found without large scale testing in production!

What I’d like you to get out of thisWhat I’d like you to get out of thisWhat I’d like you to get out of thisWhat I’d like you to get out of this

• Start thinking about Load testing (nownownownow)

• Large-scale load testing in Production is an option

• Key questions you can use to guide your testing

• What you can do as a service owner to design more testable services TIP

Q&AQ&AQ&AQ&A

how we tripled our load in production for testing...

Documents