my talk on piter py 2016
TRANSCRIPT
Performance engineeringfor children and grownups
Alex Chistyakov, Principal Engineer @ Git in Sky23.04.2016
Piter Py
Who is Mr. Chistyakov?
● Hi, I’m Alex
● I’m a Go programmer...
● ...and became a Mac user recently
● So you may wonder what I am doing here
● Don’t know for sure, maybe just IT
Who is Mr. Chistyakov?
● Hi, I’m Alex
● I’m a Go programmer...
● ...and became a Mac user recently
● So you may wonder what I am doing here
● Don’t know for sure, maybe just IT
● I realized I never used Cthulhu on my
slides => challenge accepted!
A word on Cthulhu
● Cthulhu is horrendous, but...
● ...the most dreaded operation engineer’s
nightmare is not Cthulhu...
● ...but a phone call at 3:40AM
● “The app is down!”
● Enough on Cthulhu then
So we need a plan
● Blame Python!
● Seriously, just use Go
● Python is slow as a snail
● It has GIL, etc
● ...
● Looks like a plan!
So we need a plan
● Blame Python!
● Seriously, just use Go
● Python is slow as a snail
● It has GIL, etc
● ...
● Looks like a plan!
● Measure then optimize!
Measure then optimize!
● If you know Latin please make atranslation for me and I will get a tattoo
● Okay, we probably know how to optimize
● (Just rewrite everything)
● But what to measure?
● And how to measure in a production
environment?
The basic idea seems to be quite old
● http://poormansprofiler.org/
● 1) Periodically take CPU stack traces
● 2) Aggregate raw samples somehow
● 3) Analyze
● 4) ???????
● 5) PROFIT!
Let’s profile something!
● We are in 2016, so Docker is imminent
● https://hub.docker.com/r/gitinsky/deluge/
● Deluge is a BitTorrent client w/web
interface
● (I use BitTorrent for educational purposes
only)
● So I have 282 torrents in the list, most of
them are inactive
Let’s profile something!
● We are in 2016, so Docker is imminent
● https://hub.docker.com/r/gitinsky/deluge/
● Deluge is a BitTorrent client w/web
interface
● (I use BitTorrent for educational purposes
only)
● So I have 282 torrents in the list, most of
them are inactive
WTF did we just see?
● Respect the screen resolution of Mac!
● But even Retina display is not enough
● Frankly speaking, PMP results are a
mess
Meet flamegraphs!
● http://www.brendangregg.com/flamegraphs.html
● https://github.com/brendangregg/FlameGraph
● https://github.com/brendangregg/FlameGraph/blob/master/stackcolla
pse-gdb.pl● (The last one is in Perl, sorry)
● We can connect PMP and this script
Meet flamegraphs!
● http://www.brendangregg.com/flamegraphs.html
● https://github.com/brendangregg/FlameGraph
● https://github.com/brendangregg/FlameGraph/blob/master/stackcolla
pse-gdb.pl● (The last one is in Perl, sorry)
● We can connect PMP and this script
Nice try, but WTF again?
● Color intensity means nothing (colors are used just to differentiate
bars)
● Bar length represents a total number of all samples of that kind
● Stack roots are at the bottom of Y axis, stack leaves are at the top
● There is a common stack root (imaginary in this case)
● Time to analyze!
Nice try, but WTF again?
● Color intensity means nothing (colors are used just to differentiate
bars)
● Bar length represents a total number of all samples of that kind
● Stack roots are at the bottom of Y axis, stack leaves are at the top
● There is a common stack root (imaginary in this case)
● Time to analyze!
Analyzing like a CIA agent!
● Left to right:
● Python interpreter
● C++, boost
● C++, libtorrent
● But we did not start anything except
Python interpreter!
● clone() → start_thread() → all that C++
stuff
Captain Unobvious to the rescue
● What does our non-C++ part do?
● Not much:
● PyRun_FileExFlags
● PyEval_EvalCode
● PyEval_EvalCodeEx
● And so on
● What else did you expect from an interpreter?
One does not simply sample Python w/gdb
● Wait, we were going to sample Python code?
● Okay, what if we sample from inside the interpreter, not from
outside of it?
● Basic idea is to use a signal handler or a separate thread to
collect stack traces right from the Python program periodically
● A seasoned Pythonist writes this code in 20 to 40 mins, but
someone did that already
Okay next try
● https://github.com/evanhempel/python-flamegraph
● Works as a module
● Starts a separate stack frames collecting thread
● Wraps a script in question
● Writes result to a file at the end of the run
All hail the glory of Python!
● https://github.com/evanhempel/python-flamegraph
● Works as a module
● Starts a separate stack frames collecting thread
● Wraps a script in question
● What if a script in question is a daemon? (fork() then exit(), you know)
● Writes result to a file at the end of the run
● What if a script being profiled does not intend to terminate?
All hail the glory of Python!
● What if a script in question is a daemon? (fork() then exit(), you
know)
● What if a script being profiled does not intend to terminate?
● I fixed both problems in 20 mins and I’m not even a Pythonist!*
* my slides can contain blatant lies
First 60 secs after start
● What if a script in question is a daemon? (fork() then exit(), you
know)
● What if a script being profiled does not intend to
termhttps://github.com/evanhempel/python-flamegraphinate?
● I fixed both problems in 20 mins and I’m not even a Pythonist!*
* my slides can contain blatant lies
First 300 secs after start
● What if a script in question is a daemon? (fork() then exit(), you
know)
● What if a script being profiled does not intend to terminate?
● I fixed both problems in 20 mins and I’m not even a Pythonist!*
* my slides can contain blatant lies
doPoll(), are you kidding?
● A cat does “meow-meow”
● An imperial trooper with a blaster does “pew-pew”
● Your code does nothing at all
● Deluge is fast, it seems to be just a wrapper on libtorrent
● With ~30% overhead on CPU for some reason, but that’s why
profilers exist – we are aware of it now at least
That’s all folks!
● Questions, please!
● Starring:
● Alex Chistyakov, Principal Engineer, Git in Sky
● http://gitinsky.com
● http://meetup.com/DevOps-40