offline performance monitoring for linux abhishek shukla
TRANSCRIPT
Offline Performance Monitoring for Linux
Abhishek Shukla
File Format for Saving data
Executables
Writing to File
Reading from File
Showing statistics on a graphical user interface
What are limitations of this software ?
What still it to be done ?
Usability of this software
Outline
Offline monitoring is just keeping a close eye on what happened in past.
Which magical stick does one need to wave to get this done ?
I want to know the past in which I lived.
But why don’t I know my past which I lived myself ?
Well , I was sleeping that time and could not write in my diary.
What should I do ? May be I can plan my future better after knowing my
mistakes in past.
Offline Monitoring
Now I’ll write about my present in my diary
In future, I’ll able to know all details of my past
But it was so simple ?
Well, still most people doesn’t do it.
How to do it for computers, they can do exactly what they are told to?
Just write everything about present in a persistent space and read it later when any question about past is asked
Offline Monitoring (cont..)
Why ?
It’s better when things are organized rather than they are randomly located here and there.
But I don’t like having complex rules to follow
Fine, this file format doesn’t look very complex
File Header First Sample Second Sample Third Sample ( this goes on & on )
File Format
One value to verify that it is smartertop generated file
Size of system related performance counters in Byte
Size of process related performance counters for one process in Byte
Size of File Header
to verify that file is not generated by smartertop which has different size of File Header.
Different version might have this field different
Record Header length
Clock Interrupts per second
Some information about this system
Size of memory page in bytes
OS version information
File Header
Snapshot of all current activities.
But I won’t need details of all activities in future.
Then don’t store those activities
Store those things which you might need in future.
Format of one sample
Header for sample
Compressed system related statistics
Compressed process related statistics
What are samples ?
Current Time
Size of compressed system related counters for this sample
Size of compressed process related counters for this sample
Interval since last sample (in seconds)
Number of Process in this sample
Header for Sample
Memory Stats
Interface Stats
CPU Stats
Disk Stats
Network Stats
System Related Counters
CPU stats
Disk Stats
Memory Stats
Few other things like scheduling policy, state of process etc.
Process Related Counters
There are two actually
smartertop
smartertop-gui
What does smartertop do?
Writing details to file
What does smartertop-gui do?
Reading details from file and showing them in a graphical user interface
Executables
After each d seconds, many functions are called which read many counters from many files
fill them in an array (of structure)
Compute difference between current counter and previous counters
Write this in a file
Smarter Top
Snapshot of all current counters are taken after each d seconds
First snapshot of system level counters is taken by reading following files
general statistics from /proc/stat
load average values from /proc/loadavg
virtual memory statistics from /proc/vmstat
Memory related statistics from /proc/meminfo
interface statistics from /proc/net/dev
IP v4 statistics from /proc/net/snmp
IP v6 statistics from /proc/net/snmp6
Disk statistics from /proc/diskstats
Code Walk Through
Then calculate difference between current counters and last counters ( system level)
Snapshot of process level counters is taken by reading following files
general statistics from /proc/<pid>/stat
/proc/<pid>/status
/proc/<pid>/io
get state of threads from /proc/<pid>/task/<tid>/stat
full command line from /proc/<pid>/cmdline
Continued…
But this way we got counters of processes which have not exited till the time snapshot was taken.
What about processes which exited in last cycle ?
For this, read process records from the account-file, which are written since the previous cycle
On Ubuntu, account file is
/var/log/account/pacct
On other distribution it could be
/var/log/pacct
/var/account/pacct
Continued…
First I checked which file is present among them
Then verify whether it is really in use by forking a child process which immediately finishes
Continued…
Then calculate difference between current counters and last counters
Compress system level counters by using compress function of zlib
Compress process level counters by using compress function of zlib
Write compressed system status structure to file
Write compressed list of process status structure to file
DONE WITH WRITING TO FILE
Continued…
XML UI definition are specified in smartertop.glade file
GtkBuilder reads this textual description of a user interface and instantiates the described objects.
It holds a reference to all objects that it has constructed and drops them when it is finalized.
gtk_builder_get_object() is used to access widgets by names assigned to them in UI definition
okay, now we are done with how GUI was made
Smartet Top GUI
Now let's see how contents are being fed to UI
Where is all data ?
All data is available in the file we created using smartertop
Open that file, do some checks to verify whether this file is really created by “smartertop”.
Now read one sample, send it to a function which will show it in a gui
This sample includes system level counters as well as process level counters
Keep on doing this for all samples which comes in specified range on respective events
Continued…
Files are present day wise, filename are of smartertop_YYYYMMDD format
Time windows across days are not supported
Nethog facility is not implemented
Process wise network usage details
Limitation
Make Last Page Empty
Overcome those limitations
Better User Interface
What still is to be done ?
People and organizations who need to monitor their resources offline
Who want to analyze things about resources
Who’ll use this software
