active queue management (for cloud services)
DESCRIPTION
Peak load, and burst-y traffic are problem spaces which are often (and tragically) confused for each other, invariably to the detriment of both ops and users. While peak-load is all about capacity management, in a burst-y situation, you might have to prioritize - or even drop! - requests. Knowing which requests to process, and how to actually process them is the world of Active Queue Management (AQM). While AQM has long been exclusively in the domain of the TCP/IP crowd, it has been slowly making its way into the world of cloud-services, albeit with much (faulty!) wheel-reinventing. Join me as I take you through the world of Active Queue Management, back-pressure, load-ramping, and tactical avoidance, things that most people should be architecting into their services, but aren't.TRANSCRIPT
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
Active Queue
ManagementMahesh Paolini-Subramanya (@dieswaytoofast)V.P. R&D, Ubiquiti Networks
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Metrics
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Metrics
Phone calls per Second
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Metrics
Phone calls per Second
x 1000
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Metrics
Simultaneous Phone Calls
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Metrics
Simultaneous Phone Calls
x 10,000
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Metrics
API Requests
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Metrics
API Requests
∞
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
Multi-Site
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
My Vacation
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
(Actually, the day before)
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
A small failure…
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Horror! The Horror!
Why are my calls failing?
You better call me back!
I’m still p***ed off!
And you’re stupid Apps don’t
work!
Dude! WTF?!?!
The Horror! The Horror!
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
Surely you Tested?
Romney 2012
(Lack of) Speed Kills
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
!!!!!Queues!!!!!!
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
Queues
Can you recover quickly?
Bufferbloat doesn’t matter, right?
Once up, can you deal with the backlog?
Back-pressure isn’t an issue, right?
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
Queues
Can you recover quickly?
Bufferbloat doesn’t matter, right?
Once up, can you deal with the backlog?
Back-pressure isn’t an issue, right?
NOPE
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
Programmable
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
Behavioral
Self Managed
Queues
Queue Mgmt.
Queues
Active Queue Mgmt.
Something’s gotta give
Tail Drop
God(category – TCP/IP)
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
RED
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
RED
Newark Airport
FRED
RED-PD
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
Queues
RED in a different Light (1999)
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
Queues
CoDel
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
What about Testing?
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
What about Testing?
D-UH…
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Bottom Line
Black swans will occur – Oh Yes!
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Bottom Line
Black swans will occur – Oh Yes!
You can only improve what you control
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Bottom Line
Black swans will occur – Oh Yes!
You can only improve what you control
Your business will define your discards
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
The Bottom Line
Black swans will occur – Oh Yes!
You can only improve what you control
Your business will define your discards
Agility is (always!) your friend
The Business
Beware the Black Swan
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Active Queue Management
@dieswaytoofast
You, apparently, forgot about me
Free Calling
No Voicemail
“Active” Queue Management
