10 billion a day, 100 milliseconds per: monitoring real-time bidding at adroll

M O N I T O R I N G R E A L - T I M E B I D D I N G AT A D R O L L

T E N B I L L I O N A D AY, O N E - H U N D R E D M I L L I S E C O N D S P E R

I D O T H I N G S W I T H / T O C O M P U T E R S .

I C A R E A B O U T R E L I A B L E , C O M P L E X A N D C R I T I C A L S Y S T E M S .

A D R O L L

L E S S T H I S

M O R E T H I S

W E ’ R E A N ADTECH C O M PA N Y .

R E TA R G E T I N G , I T ’ S A L L A B O U T D ATA .

• Our customers want to show ads to people who might care to see them.

• We partner with “exchanges” to enter ad-slot auctions.

• These auctions are executed and finalized while consumer’s webpages load.

R E A L - T I M E B I D D I N G

t h e n a t u r e o f t h e p r o b l e m d o m a i n :

• Low latency ( < 100ms per transaction )

• Firm real-time system

• Highly concurrent ( > 30 billion transactions per day )

• Global, 24/7 operation

“HUMANS ARE BAD AT PREDICTING THE PERFORMANCE OF COMPLEX SYSTEMS(…).

OUR ABILITY TO CREATE LARGE AND COMPLEX SYSTEMS FOOLS US INTO BELIEVING THAT WE’RE ALSO ENTITLED TO UNDERSTAND

THEM.”– C A R LO S B U E N O

“ M AT U R E O P T I M I Z AT I O N H A N D BO O K ”

A H E A D O F T I M E V E R I F I C AT I O N I S N O T S U F F I C I E N T . !

( D O N ’ T S C R I M P O N I T , T H O U G H . )

I G N O R A N C E A N D C O M P L E X I N T E R A C T I O N S W I T H E X T E R N A L S Y S T E M S A R E W H Y W E C A N ’ T H AV E N I C E T H I N G S .

A T S C A L E , E V E N R A R E E V E N T S H A P P E N F R E Q U E N T L Y .

A T S C A L E , B A D T H I N G S H A P P E N FA S T E R T H A N H U M A N S C A N R E S P O N D .

T H E R E S U L T S A R E R A R E L Y P R E T T Y .

T H E C A U S E S A R E R A R E L Y Q U I C K T O D I S C O V E R .

W H AT C A N B E D O N E ?

W E H AV E T O O B S E R V E O U R S Y S T E M S W H I L E T H E Y R U N .

W E C A N B E S C I E N T I F I C

A B O U T T H I S

N O T A L L S Y S T E M S A R E M O N I T O R A B L E , J U S T A S N O T A L L A R E T E S TA B L E . !

I T ’ S A M AT T E R O F D E S I G N A N D O F C U L T U R E .

A D E S I G N T H AT I S M O N I T O R A B L E TA K E S S T O C K O F I T S I N T E R N A L T O L E R A N C E S , E X T E R N A L I N T E R FA C E S A N D E X P O S E S T H E S E F O R A N O P E R AT O R .

A N E N G I N E E R I S R E S P O N S I B L E F O R D E S I G N I N G A N D B U I L D I N G S Y S T E M S . !

A N O P E R AT O R I S R E S P O N S I B L E F O R R U N N I N G T H E M .

A H E A D - O F - T I M E V E R I F I C AT I O N

— T E S T I N G , T Y P E - C H E C K I N G —G I V E S T H E E N G I N E E R I N T U I T I O N A B O U T T H E S Y S T E M .

I F Y O U D O N ’ T K N O W H O W T H E S Y S T E M

S H O U L D B E H AV E Y O U C A N ’ T S AY H O W I T

S H O U L D N ’ T O R I S N ’ T .

M O N I T O R I N G G I V E S T H E O P E R AT O R I N F O R M AT I O N A B O U T T H E B E H AV I O R O F T H E R U N N I N G S Y S T E M .

T H E O P E R AT O R S A N D E N G I N E E R S A R E O F T E N T H E S A M E P E O P L E .

T H E R E ’ S A P O T E N T I A L F O R A P O S I T I V E F E E D B A C K L O O P O F Q U A L I T Y H E R E .

“WHILE THE SKILL OF REENTRY WAS EASILY HANDLED BY AUTOMATED SYSTEMS, THE

PILOT’S PRIMARY FUNCTION EVOLVED TO BE A REDUNDANT SYSTEM (…) COORDINATING

A VARIETY OF CONTROLS AS MUCH AS DIRECTLY CONTROLLING THE VEHICLE.”

– DAV I D A . M I N D E L L D I G I TA L A P O L LO : H U M A N A N D M AC H I N E I N S PAC E F L I G H T

exometer

e x o m e t e r — g i t h u b . c o m / F e u e r l a b s / e x o m e t e r

• Created by Feuerlabs.

• Responsive upstream (Ulf Wiger never sleeps?)

• Metric collection, aggregation and reporting decoupled.

• Static and dynamic configuration.

• Very low, predictable runtime overhead.

https://github.com/Feuerlabs/exometer

I M P O RTA N T T E R M S

• METRIC: a measurement

• ENTRY: a receiver and aggregator of metrics

• REPORTER: an entity which samples entries on a regular interval and optionally ships these samples onto a third-system

• SUBSCRIPTION: the definition of the regular interval on which reporters sample entries

D e f i n i n g E n t r i e s{predefined, [ {[erlang,memory], {function, erlang, memory, ['$dp'], value,[ets,binary]}, [] }, {[erlang, statistics], {function, erlang, statistics, ['$dp'], value, [run_queue]}, [] },

{[erlang, gc], {function, erlang, statistics, [garbage_collection], match, {total_coll, rec_wrd, '_'}}, [] }, !

{[boodah, freq_cap, not_found], spiral}, {[boodah, freq_cap, ok], spiral}, {[boodah, freq_cap, timeout], spiral} ]},

D e f i n i n g R e p o r t e r s

{ reporters, [ { exometer_report_statsd, [ {hostname, "localhost"}, {port, 8125}, {type_map, [ {[erlang,statistics,run_queue], histogram}, {[erlang, gc, tot_coll], histogram}, {[erlang, gc, rec_wrd], histogram},

!

{[erlang,memory,ets], gauge}, {[erlang,memory,binary],gauge}, !

{[boodah,freq_cap,not_found],gauge}, {[boodah,freq_cap,ok],gauge}, {[boodah,freq_cap,timeout],gauge} ]}, !

D e f i n i n g S u b s c r i p t i o n s{ report, [ { subscribers, [ {exometer_report_statsd, [erlang, statistics], run_queue, 1000}, {exometer_report_statsd, [erlang, gc], tot_coll, 1000}, {exometer_report_statsd, [erlang, gc], rec_wrd, 1000}, ! {exometer_report_statsd, [erlang, memory], ets, 10000}, {exometer_report_statsd, [erlang, memory], binary, 10000}, !!

!!! {exometer_report_statsd, [boodah, freq_cap, not_found], one, 1000}, {exometer_report_statsd, [boodah, freq_cap, ok], one, 1000}, {exometer_report_statsd, [boodah, freq_cap, timeout], one, 1000} ]} ]}

D o i n g i t d y n a m i c a l l y. 1> exometer:new([a, histogram], histogram). ok !

2> exometer:get_value([a, histogram]). {ok,[{n,0}, {mean,0}, {min,0}, {max,0}, {median,0}, {50,0}, {75,0}, {90,0}, {95,0}, {99,0}, {999,0}]} !

3> exometer_report:add_reporter( exometer_report_tty, []). ok !

4> exometer_report:subscribe( exometer_report_tty, [a, histogram], mean, 1000, []). ok !

exometer_report_tty: a_histogram_mean 1393627070:0 exometer_report_tty: a_histogram_mean 1393627071:0 exometer_report_tty: a_histogram_mean 1393627072:0

W H AT A R E W E L O O K I N G F O R ?

• VM killers

• System performance regressions

• Abnormal system behavior

• Surprises

“ A B N O R M A L S Y S T E M B E H AV I O R ” ?

C A S E S T U D I E S

C A S E S T U D Y : A D R O L L R T B T I M E O U T S P I K E S

C a s e S t u d y : A d R o l l R T B T i m e o u t S p i k e s

P E R I O D I C B I D T I M E O U T S


C O N S I S T E N T S Y S T E M L O A D


C O R R E L AT E D N E T W O R K T R A F F I C S P I K E S


C O R R E L AT E D R U N Q U E U E S P I K E S


W H AT H A P P E N E D ?


• Scheduler threads were locked to CPUs

• Background process comes on every 20 minutes, consumes a lot of CPU time

• No cpu-shield was set up on our production systems

• OS bumped a scheduler thread off its CPU, backing up its run-queue

C A S E S T U D Y : E X C H A N G E T H R O T T L I N G

C a s e S t u d y : E x c h a n g e T h r o t t l i n g

H E A LT H Y PAT T E R N O F B I D R E Q U E S T S


T H E T RO U G H O F T H ROT T L I N G


B A DG O O D


P R O B L E M C O N F I R M E D W I T H E X C H A N G E


• All other metrics (run-queue, CPU, network IO) were fine.

• Confirmed that no changes had been made to the running systems via deployment.

• Amazon data showed no network issues to our machines.

W H AT H A P P E N E D ?


W E H I T A N I M P L I C I T E X C H A N G E L I M I T . ( A R G U A B L Y , A B U G . )


L E S S O N S L E A R N E D

I T I S P O S S I B L E T O H AV E T O O L I T T L E I N F O R M AT I O N .

“(THE FIREFIGHTERS) TRIED TO BEAT DOWN THE FLAMES (OF CHERNOBYL REACTOR 4). THEY KICKED AT THE

BURNING GRAPHITE WITH THEIR FEET. … THE DOCTORS KEPT TELLING THEM

THEY’D BEEN POISONED BY GAS.”- - S V E T L A N A A L E X I E V I C H - VO I C E S F RO M C H E R N O BY L : T H E O R A L H I S TO RY O F A N U C L E A R D I SA S T E R

I T I S P O S S I B L E T O C O L L E C T T O O M U C H I N F O R M AT I O N , O R P R E S E N T I T B A D L Y .

“SAFETY SYSTEMS, SUCH AS WARNING LIGHTS, ARE NECESSARY, BUT THEY HAVE THE POTENTIAL FOR DECEPTION. (…) ONE OF THE LESSONS OF COMPLEX SYSTEMS AND (THREE MILE ISLAND) IS

THAT ANY PART OF THE SYSTEM MIGHT BE INTERACTING WITH OTHER PARTS IN

UNANTICIPATED WAYS.”- C H A R L E S P E R ROW - N O R M A L AC C I D E N T S : L I V I N G W I T H H I G H - R I S K T E C H N O LO G I E S

I N D I R E C T K N O W L E D G E M AY N O T T E L L T H E W H O L E S T O R Y , O R M AY M A K E Y O U D O U B T W H AT ’ S P L A I N L Y B E F O R E Y O U R E Y E S .

“THE FIRST DISASTER IN SPACE HAD OCCURRED, AND NO ONE KNEW WHAT HAD HAPPENED. ON THE

GROUND, THE FLIGHT CONTROLLERS WERE NOT EVEN SURE THAT ANYTHING HAD. ONE REASON FOR THEIR

IGNORANCE WAS THE IMPERFECT NATURE OF TELEMETRY FROM THE SPACECRAFT WHICH COULD NOT

TELL THEM DIRECTLY THAT AN OXYGEN TANK HAD BLOWN UP.”

- - H E N RY S . F. C O O P E R , J R . - X I I I : T H E A P O L LO F L I G H T T H AT FA I L E D

Things that don’t quite work like you’d hope.

•Instrumenting code increases code size.

•While very low, runtime impact is not zero.

•Instrumentation of dependencies is up to the library author.

P r o b l e m s

S o l u t i o n s

•Dtrace / systemtap hookups

•Culture of instrumentation by default.

•Tracing BIFs

W O M B AT ?

I n s t r u m e n t a t i o n b y d e f a u l t s t r a t e g i e s

•Application configuration callback module

•Behaviors callbacks

•Value-at-time functions

W H AT A B O U T

C U L T U R E

L O O K F O R S O L U T I O N S ,

N O T S C A P E G O AT S .

D O N ’ T B E F O O L E D B Y O V E R C O N F I D E N C E I N Y O U R

W O R K .

D O C U M E N T A N D D I S C U S S Y O U R FA I L U R E S .

E V E R Y O N E M U S T W O R K T O WA R D T H E S A M E E N D .

K N O W S O M E B A S I C M AT H E M AT I C S .

P R E F E R L O O S E C O U P L I N G W H E N P O S S I B L E . !

B E E X P L I C I T A B O U T T I G H T C O U P L I N G .

M E A S U R E M U C H , P R E S E N T O N LY T H AT W H I C H Y O U ’ R E C E R TA I N T O N E E D .

M E A S U R E , D O N ’ T G U E S S .

B E A C C U R AT E .

Q U E S T I O N S ?

@ b l t r o u t w i n e

B R I A N @ T R O U T W I N E . U S

<3

mailto:[email protected]

B o n u s B i b l i o g r a p h y !XIII: The Apollo Flight that Failed Henry S. F. Cooper, Jr.

Digital Apollo: Human and Machine in Spaceflight David A. Mindell

Normal Accidents: Living with High-Risk Technologies Charles Perrow

Voices from Chernobyl: The Oral History of a Nuclear Disaster Svetlana Alexievich

Real-Time Systems: Design Principles for Distributed Embedded Applications Hermann Kopetz

Command and Control: Nuclear Weapons, the Damascus Accident and the Illusion of Safety Eric Schlosser

10 billion a day, 100 milliseconds per: monitoring real-time bidding at adroll

Technology

i n d e

n e e r s

i o n h

t e n b i

o f i t s i n t e r

n o p e r

o r i n f o r

t h e o p e r