using sysdig to troubleshoot like a boss

sysdig

strace tcpdump

sysdig tcpdump

tcpdump

sysdig

sysdig

sysdig

Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

1 of 17 5/25/2015 12:21 PM

sysdig apt-get

rpm yum

sysdig

sysdig

sysdig curl

# curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add -

# curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/stable/deb/draios.list

/etc/apt/sources.list.d/


2 of 17 5/25/2015 12:21 PM

apt-get update

# apt-get update

sysdig

dpkg

# dpkg --list | grep header

ii linux-generic 3.11.0.12.13 amd64 Complete Generic

Linux kernel and headers

ii linux-headers-3.11.0-12 3.11.0-12.19 all Header files rel

ated to Linux kernel version 3.11.0

ii linux-headers-3.11.0-12-generic 3.11.0-12.19 amd64 Linux kernel hea

ders for version 3.11.0 on 64 bit x86 SMP

ii linux-headers-generic 3.11.0.12.13 amd64 Generic Linux ke

rnel headers


3 of 17 5/25/2015 12:21 PM

linux-

generic

uname

# uname -r

3.11.0-12-generic

apt-get

uname -r

# apt-get install linux-headers-

# apt-get install linux-headers-3.11.0-12-generic

sysdig


4 of 17 5/25/2015 12:21 PM

# apt-get install sysdig

sysdig tcpdump

sysdig

tcpdump

-w

# sysdig -w

# sysdig -w tracefile.dump


5 of 17 5/25/2015 12:21 PM

tcpdump sysdig CTRL+C

sysdig

-r

# sysdig -r

# sysdig -r tracefile.dump

1 23:44:57.964150879 0 (7) > switch next=6200(sysdig)

2 23:44:57.966700100 0 rsyslogd (358) < read res=414 data=[ 3785.473354] sysdig_probe: starting cap

ture.[ 3785.473523] sysdig_probe:

3 23:44:57.966707800 0 rsyslogd (358) > gettimeofday

4 23:44:57.966708216 0 rsyslogd (358) < gettimeofday

5 23:44:57.966717424 0 rsyslogd (358) > futex addr=13892708 op=133(FUTEX_PRIVATE_FLAG|FUTEX_WAKE_OP) v

al=1

6 23:44:57.966721656 0 rsyslogd (358) < futex res=1






6 of 17 5/25/2015 12:21 PM

sysdig -A

sysdig

# sysdig -A

# sysdig -A > /var/tmp/out.txt

# cat /var/tmp/out.txt

1 22:26:15.076829633 0 (7) > switch next=11920(sysdig)

sysdig

tcpdump sysdig

sysdig -l


7 of 17 5/25/2015 12:21 PM

# sysdig -l

----------------------

Field Class: fd

fd.num the unique number identifying the file descriptor.

fd.type type of FD. Can be 'file', 'ipv4', 'ipv6', 'unix', 'pipe', 'e

vent', 'signalfd', 'eventpoll', 'inotify' or 'signalfd'.

fd.typechar type of FD as a single character. Can be 'f' for file, 4 for

IPv4 socket, 6 for IPv6 socket, 'u' for unix socket, p for pi

pe, 'e' for eventfd, 's' for signalfd, 'l' for eventpoll, 'i'

for inotify, 'o' for uknown.

fd.name FD full name. If the fd is a file, this field contains the fu

ll path. If the FD is a socket, this field contain the connec

tion tuple.

sysdig


8 of 17 5/25/2015 12:21 PM

# sysdig -r tracefile.dump proc.name=sshd

530 23:45:02.804469114 0 sshd (917) < select res=1

531 23:45:02.804476093 0 sshd (917) > rt_sigprocmask

532 23:45:02.804478942 0 sshd (917) < rt_sigprocmask

533 23:45:02.804479542 0 sshd (917) > rt_sigprocmask

534 23:45:02.804479767 0 sshd (917) < rt_sigprocmask

535 23:45:02.804487255 0 sshd (917) > read fd=3(10.0.0.12:55993->162.0.0.80:22) size=16384

fd.name

# sysdig fd.name=/dev/log

14 11:13:30.982445884 0 rsyslogd (357) < read res=414 data=[ 582.136312] sysdig_probe: starting captur

e.[ 582.136472] sysdig_probe:


9 of 17 5/25/2015 12:21 PM

# sysdig fd.name contains /etc

8675 11:16:18.424407754 0 apache2 (1287) < open fd=13(/etc/apache2/.htpasswd) name=/etc/apache2/.ht

passwd flags=1(O_RDONLY) mode=0

8678 11:16:18.424422599 0 apache2 (1287) > fstat fd=13(/etc/apache2/.htpasswd)

8679 11:16:18.424423601 0 apache2 (1287) < fstat res=0

8680 11:16:18.424427497 0 apache2 (1287) > read fd=13(/etc/apache2/.htpasswd) size=4096

8683 11:16:18.424606422 0 apache2 (1287) < read res=44 data=admin:$apr1$OXXed8Rc$rbXNhN/VqLCP.ojKu1aUN

1.

8684 11:16:18.424623679 0 apache2 (1287) > close fd=13(/etc/apache2/.htpasswd)

8685 11:16:18.424625424 0 apache2 (1287) < close res=0

9702 11:16:21.285934861 0 apache2 (1287) < open fd=13(/etc/apache2/.htpasswd) name=/etc/apache2/.ht

passwd flags=1(O_RDONLY) mode=0

9703 11:16:21.285936317 0 apache2 (1287) > fstat fd=13(/etc/apache2/.htpasswd)

9704 11:16:21.285937024 0 apache2 (1287) < fstat res=0

sysdig

sysdig

sysdig


10 of 17 5/25/2015 12:21 PM

-cl sysdig

# sysdig -cl

Category: CPU Usage

-------------------

topprocs_cpu Top processes by CPU usage

Category: I/O

-------------

echo_fds Print the data read and written by processes.

fdbytes_by I/O bytes, aggregated by an arbitrary filter field

fdcount_by FD count, aggregated by an arbitrary filter field

iobytes Sum of I/O bytes on any type of FD

iobytes_file Sum of file I/O bytes

stderr Print stderr of processes

stdin Print stdin of processes

stdout Print stdout of processes

sysdig sysdig

sysdig


11 of 17 5/25/2015 12:21 PM

-i

# sysdig -i bottlenecks

Category: Performance

---------------------

bottlenecks Slowest system calls

Use the -i flag to get detailed information about a specific chisel

Lists the 10 system calls that took the longest to return dur

ing the capture interval.

Args:

(None)

sysdig -c

# sysdig -c topprocs_net

Bytes Process

------------------------------

296B sshd


12 of 17 5/25/2015 12:21 PM

echo_fds

# sysdig -A -c echo_fds proc.name=apache2

------ Read 444B from 127.0.0.1:57793->162.243.109.80:80

GET /wp-admin/install.php HTTP/1.1

Host: 162.243.109.80

Connection: keep-alive

Cache-Control: max-age=0

Authorization: Basic YWRtaW46ZUNCM3lyZmRRcg==

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/3

3.0.1750.152 Safari/537.36

Accept-Encoding: gzip,deflate,sdch

Accept-Language: en-US,en;q=0.8

echo_fds

fd.cip


13 of 17 5/25/2015 12:21 PM

# sysdig -A -c echo_fds fd.cip=127.0.0.1

------ Write 1.92KB to 127.0.0.1:58896->162.243.109.80:80

HTTP/1.1 200 OK

Date: Thu, 17 Apr 2014 03:11:33 GMT

Server: Apache

X-Powered-By: PHP/5.5.3-1ubuntu2.3

Vary: Accept-Encoding

Content-Encoding: gzip

Content-Length: 1698

Keep-Alive: timeout=5, max=100

Connection: Keep-Alive

Content-Type: text/html; charset=utf-8


14 of 17 5/25/2015 12:21 PM

EMC PowerPath: superblock could not be read

Loving your blog so much. Reading through

article upon article. Can you elaborate on one item here,

since you are specifically discussing the passno, can

Remote Command Execution with SaltStack

Awesome article!

Building Self-Healing Applications with Saltstack

Could't agree more :)

Getting started with SaltStack by example:Automatically Installing nginx

Hey kannan, it looks like either salt couldn't

find the top.sls or you don't have a top.sls file. Make

sure you perform the steps from this part

[BENJAMIN CANE]

1 Comment 1

sysdig

Benjamin, thanks for putting this together! This is an great guide for anyone getting started with sysdig. One note

- on your network traffic example, in order to capture network traffic specifically you can use the fd.type filter:

sysdig -A -c echo_fds proc.name=apache2 and fd.type=ipv4


15 of 17 5/25/2015 12:21 PM


16 of 17 5/25/2015 12:21 PM


17 of 17 5/25/2015 12:21 PM

using sysdig to troubleshoot like a boss

Documents

d sysdig

dumpusing sysdig

genericsysdigusing sysdig

gettimeofdayusing sysdig

boss http

sysdig r tracefile

sysdig w tracefile

amd64 linux kernel headers