THIS ONE WEIRD TIME-SERIES TRICKBaron Schwartz

Monitorama 2014 Portland

PERFORMANCE MANAGEMENT

Optimization, Backups, Replication, and more

Baron Schwartz, Peter Zaitsev &

Vadim Tkachenko

High PerformanceMySQL

3rd Edition

Covers Version 5.5

Baron Schwartz - baron@vividcortex.com - @xaprb

YES, WE ARE HIRING

WHAT IS TYPICAL?

BUT FIRST...

WORLDVIEW

•We need performance management, not monitoring.

•Work-getting-done is top priority.

•We need more than recipes.

ANOMALY DETECTION

• Trending and prediction are not anomaly detection.

• Anomalousness is not true-or-false, it’s a probability.

• Anomalous and bad are not the same.

• “There is an anomaly” implies a statistical framework.

THANK YOU

• Any questions?

HOLY GRAIL*

•Determine “normal” behavior

• Predict how a metric “should” behave

•Quantify deviation from normality

•Do useful* stuff

* For the record, this is not something I value highly, but it’s worth talking about.

OUT OF SCOPE

• All that stuff is potentially too practical for this talk.

• I just want to talk about a fun tool I use sometimes.

• I’d rather play with math than do anything useful.

WHY GIVE THIS TALK?Toufic already gave the beginning and end.

IT’S A TIME SERIES WORLD

Ability to process cheaply is important.

HOW MANY OF YOU KNOW...

Binomial Theorem

Standard Deviation

Standard Error

Random Walk

T-Statistic, R-Squared

Pearson’s Correlation

Heteroscedasticity

Homoscedasticity

ANOVA

ARIMA

YAGNI

TRIED SMIRNOV

TESTBartender didn’t have any

Kolmogorov

http://en.wikipedia.org/wiki/Statistics

SERIOUSLY

• Basic statistics is good for a lot of amazing things.

• Use simple, efficient approaches if you can.

• Hat tip to Neil Gunther

• Advanced stuff is there if you need it; use an advisor.

• Hat tip to Neil Gunther

•Don’t let the clowns and naysayers stop you.

• http://blog.b3k.us/2012/06/08/the-engineering-activity-spectrum.html

Metrics Analysis

BEGIN AT THE BEGINNING

ANOMALY DETECTION

Characterization of Past

Central Tendency

Forecast/Predict

Deviation

Anomaly

CENTRAL TENDENCIESThe mother of all central tendencies is the average.*

CONTROL CHARTSIs the process within normal limits?

PROBLEMControl charts assume a stationary mean.

Systems are “less normal” than we assume, in both senses.

RECENCYWhat is a system’s “recent” normal?

MOVING AVERAGEAverage over a window of recent data

MOVING CONTROL CHARTS

PROBLEMSMoving average is “more expensive” to compute

Moving average is influenced by “distant” past

These days should be remembered and kept throughout every generation- Esther 9:28

REMEMBER ALL THE THINGS

EXPONENTIAL MOVING AVERAGES

• Infinite memory, biased towards recent history (past data trails off to nothing)

• Cheap to compute

• Choose a decay factor α

• St = αxt + (1-α) St-1

0

100

200

300

400

1 2 3 4 5

0

100

200

300

400

1 2 3 4 5

EWMA = LOW-PASS FILTER

CHOOSING DECAYα = 2/(N+1), where N is desired avg age of samples

EXPONENTIAL MOVING CONTROL CHARTS

•Need exponential moving average - easy

•Need exponential moving standard deviation - hmm.

• Standard deviation = square root of variance

• Variance = “mean of square minus square of mean”

•MVP solution: exponential moving avg of squared values

EMCC

SHORTCOMINGS

• Works well when data is approximately normally distributed

• Non-Gaussian data throws “standard deviation” for a loop; false positives ensue

• Requires more advanced techniques

• STILL USEFUL ANYWAY.

EWMA WITH TREND

DOUBLE EXPONENTIAL SMOOTHING

• Predict the series based on the average and the trend

•Need decay factors α, β

• St = αxt + (1-α)(St-1 + Bt-1)

• Bt = β(St-St-1) + (1-β)Bt-1

DOUBLE EXPONENTIAL SMOOTHING

HOLT-WINTERS FORECASTING

• Holt-Winters Forecasting adds seasonality indexes

• Relatively complex, expensive, slow to train, learns to expect anomalies

DIMINISHING RETURNS

•DES and H-W get complicated fast

• They work fine on non-Gaussian data.

• But Gaussian anomaly detection won’t.

• Lots of parameters; choosing them requires:

•Work, prior knowledge, complex optimization; or

•Machine learning

MACD

•Moving Average Convergence-Divergence

•Difference of two EMAs with different decay factors

MACD

QUESTIONS?@xaprb • linkedin.com/in/xaprb

baron - at - vividcortex.com

CREDITS

• car crash

• squirrel

• highway

• nerdy kid

• boarded up windows

• grail

• out of scope

• drill press

• alice in wonderland

• owl

• singapore opera

• chess

• decaying leaf

• spiderweb

• train tracks

• eye

• spiral

REGRESSIONAdd(x,  y)  {      n++      sx  +=  x      sy  +=  y      sxx  +=  x  *  x      sxy  +=  x  *  y      syy  +=  y  *  y}

Slope()  {    ss_xy  =  n*sxy-­‐sx*sy    ss_xx  =  n*sxx-­‐sx*sx    return  ss_xy/ss_xx}

Intercept()  {    return        (sy-­‐Slope()*sx)/n}

//  And  so  on

EWMA REGRESSIONAdd(x,  y)  {

     //  replace  sx  +=  x  by  the  following:      sx  =  sx*a  +  (1-­‐a)*x

     //  repeat  as  needed;  left  as  an      //  exercise  for  the  reader