Seeing through smoke

@LAFK_pl Consultant @

Tomasz Borek

GeeCON 2015 Kraków!

@LAFK_pl

http://2015.geecon.org/register/

I'll take ALL feedback I can get. @LAFK_pl, #testwell

@LAFK_pl

Symentis

@LAFK_pl

Symentis

@LAFK_pl

Jarosław Pałka Kuba Marchwicki

Joining criteria?

@LAFK_pl

Easy!

Joining criteria

@LAFK_pl

● Conference speaker● Open-source committer● Recommended by somebody on the team

Who are you?

@LAFK_pl

● Programmer?● Tester?● Build person?● Hosting/Ops?● Manager?● Other?

Today?

@LAFK_pl

Tools

@LAFK_pl

Putting out fires

@LAFK_pl

Details

● Where this came from● Diagnostic methodology for performance problems

● Keep calm● Monitoring tools● Logging 101● How not to fly solo● Advice for when there's too much smoke

@LAFK_pl

Where this came from?

@LAFK_pl

So you know where I stand

Where this came from?

● plug-a-hole fireman: 2 years

● heroic fireman: 2 years

● Fire drills I've did and had done to me

@LAFK_pl

Advice #1

@LAFK_pl

Don't be a fireman for half a year, not to mention for years

Tool #1

@LAFK_pl

The ”box”

● Performance problem?

● Heinz Kabutz● Kirk Pepperdine

● Sets priorities straight

Net, hardware, etc.

JVM / OS

Application itself

Users!

THE BOX:

@LAFK_pl

@LAFK_pl

How not to loose your head

● Praemonitus – praemunitus!● Advanced warning● Knowledge is power

● Don't get (too) involved● You're the expert, Anderson!

@LAFK_pl

What gives warnings in advance?

● What comes in code?● What the code runs on?● Can our X hold Y Zs at same time?● How long till A gets exhausted?● Alert emails mailbox full?● Which Selenium tests are least likely to give

false-positive or flop on you due to some glitch?

@LAFK_pl

Sun Tzu, Art of War

@LAFK_pl

If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every

victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will

succumb in every battle.

Advice #3

@LAFK_pl

If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every

victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will

succumb in every battle.

Don't be involved

@LAFK_pl

Best advice and outlook have those outside

You're the expert, Anderson

@LAFK_pl

Double, triple check yours

@LAFK_pl

So when someone shouts ”it's your fault! Your responsibility! You fix it!” you will have a retort

right away

@LAFK_pl

Never let them shout at you

Advice #4

@LAFK_pl

Never let them shout at you

Advice #4

Preparation

● What can go wrong, will● Toolchain● Limit your SPOFs in advance● Cover them if you can't limit them!

@LAFK_pl

How to fight fright in a fire

● Who's frightful?● What's urgent?● What's DR timeline for the urgent items and for

the frightful people?● RPO?● RTO?

@LAFK_pl

Monitoring tools

● Performance co-pilot● Kibana / Splunk● AppDynamics● New Relic● Nagios● Ganglia + Graphite (ish)● SHELL DWARVES EVERYTHING FOR FIRES

@LAFK_pl

Logging 101

● What● The relevant● The crucial

● Reload on-the-fly● Storage● Throughput

@LAFK_pl

Logging levels

@LAFK_pl

Not those, that you think of

Level 3

@LAFK_pl

We have logs, humans look at them sometimes

Level 3

@LAFK_pl

We have logs, humans look at them sometimes* when there's a fire

Level 2

@LAFK_pl

We have logs, we need to pull out more, more often

Level 1

@LAFK_pl

We log, we parse logs, we take action

Tools for logging

● ELK – ElasticSearch + LogStash + Kibana● Or the equivalent / improvement● Available to hosting, devs, testers and support

● Loggly, Splunk● Log levels defineable on the go by Hosting /

key others

@LAFK_pl

How not to

● Request● Ask● Plead● Beg● Bargain● Blackmail

● Ok threaten perhaps

@LAFK_pl

How not to

● Request● Ask● Plead● Beg● Bargain● Blackmail

● Ok threaten perhaps

@LAFK_pl

@LAFK_pl

You do NOT want to be a hero.

Advice #5

Too much smoke

@LAFK_pl

Quit. Best like a boss.

Why should I?

@LAFK_pl

Physical recovery: 6 weeks.

Psychological recovery: 6 months.

Capacity ... unlikely.

When would that be?

@LAFK_pl

When you hurt / complain too much.Keep a trend!

Today

● Tool #1? The box. You know, for performance problems.

● Keep calm: don't lose your head, don't be frightened and don't let them shout at you.

● Monitoring tools: have your own set.● Logging 101: aim for high level, visualize● Fly Solo, if you have a Chewbacca next to you.● Too much smoke... quit. Trend yourself.

@LAFK_pl