collaborate2011 using cacti to graph mysql's metrics

8/13/2019 Collaborate2011 Using Cacti to Graph MySQL's Metrics

1/103

Using Cacti To Graph MySQLs MetricsKenny Gryp

[email protected]

Principal Consultant @ Percona

Collaborate 2011

1
mailto:[email protected]:[email protected]


2/103

Percona

MySQL/LAMP Consulting

MySQL Support

Percona Server (XtraDB)

Percona XtraBackup InnoDB Recovery Toolkit, tcprstat

(maatkit, innotop, aspersa, mysql-mmm, mysql-cacti-templates)

...

http://www.percona.com

http://www.mysqlperformanceblog.com

2
http://www.mysqlperformanceblog.com/http://www.mysqlperformanceblog.com/http://www.percona.com/http://www.mysqlperformanceblog.com/http://www.mysqlperformanceblog.com/http://www.percona.com/http://www.percona.com/


3/103

Topics

Why Graph?

What is Cacti and mysql-cacti-templates?

How Cacti works

InstallationAdd Graphs

Demo

Graphs!

Extras

3


4/103

Topics

Why Graph?


How Cacti works


Demo

Graphs!

Extras

4


5/103

Why Graph?

Troubleshooting

-behavior over time

-sudden spikes in graphs after application updates

Capacity Planning

-traffic patterns

-predictions

Not only for Operations, also for Development and

Management

5


6/103


7/103

Cacti?

open source graphing solution

uses RRDTool

advanced graphs, scriptable

web interfacequite complex to add graphs

adding servers is a manual process

7


8/103

mysql-cacti-templates?

collection of templates for cacti

graph MySQL metrics

Has a lot of non-MySQL Metrics too: IO stats,

apache, memcache, mongodb, java...

8


9/103

Topics

Why Graph?


How Cacti works


Demo

Graphs!

Extras

9


10/103

How does Cacti work?

10

Database Server

mysqld

sshd

snmpd

/proc/diskstats

Cacti

MySQL(contains cacticonfiguration)

.rrd Files(contains rrd data)

Apache(serves web

interface)

Poller(cronjob which

fetches data fromservers)

Web Server

httpd

sshd

snmpd

Web Browser

what

stores

fetch

df


11/103

Topics

Why Graph?


How Cacti works


Demo

Graphs!

Extras

11


12/103

Installation

Cacti

Mysql-cacti-templates

Configuring Servers

12


13/103


14/103

Setting up Cacti: Requirements

httpd

php, php-mysql, php-snmp

mysql-server (server and client utilities)

net-snmp (snmpget)rrdtool

14


15/103

Setting up Cacti: Installation

Install rpm/deb/... package or extract yourself

Configure webserver: php5, cacti scripts

Create cacti mysql user

Import cacti databaseEdit configuration

Setup cronjob for poller

Finish installation using your webbrowser

15


16/103

Setup: Extract

Recommended is using apt/yum/... to install cacti

if not:

16

cacti# cd /var/www/html/ # may varycacti# tar xzvf cacti-0.8.7g.tar.gzcacti# ln -s cacti-0.8.7g cacticacti#chown -R cactiuser rra/ log/


17/103

Setup: Webserver

Apache (may be something else)

Enable php

Set DirectoryIndex index.php

17

LoadModule php5_module modules/libphp5.soAddHandler php5-script .phpAddType text/html .phpDirectoryIndex index.php


18/103

Setup: MySQL

Create Cacti MySQL user:

Import Cacti Schema

Edit include/config.php

18

cactimysql> GRANT ALL PRIVILEGES ON cacti.* TOcactiuser@localhost IDENTIFIED BY 'password';

cacti#mysql -e CREATE DATABASE cacti;cacti#mysql cacti < cacti.sql

$database_type = "mysql";$database_default = "cacti";$database_hostname = "localhost";$database_username = "cactiuser";

$database_password = "password";


19/103

Setup: Poller

Add cronjob (/etc/cron.d/cacti)

19

*/5 * * * * cactiuser php /var/www/html/cacti/poller.php > /dev/null 2>&1


20/103

Setup: UI

20


21/103

Setup: UI

21


22/103

Setup: UI

22


23/103

Setup: UI

23


24/103

Setup: UI

24


25/103

Setup: UI

25


26/103

Setup: UI

26


27/103

Installation

Cacti


Configuring Servers

27


28/103

download & installation

28

cacti#wget http://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzcacti# tar -xzf better-cacti-templates-1.1.8.tar.gzcacti# cd better-cacti-templates-1.1.8/cacti#cp \scripts/{ss_get_mysql_stats.php,ss_get_by_ssh.php}\ /var/www/html/cacti/scripts/
http://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gzhttp://mysql-cacti-templates.googlecode.com/files/better-cacti-templates-1.1.8.tar.gz


29/103

install: mysql script

29

cacti# vi /var/www/html/cacti/scripts/\ss_get_mysql_stats.php...

$mysql_user = 'cactiuser';$mysql_pass = 'password';...


30/103

install: generate ssh key

30

cacti#mkdir /etc/cacticacti# ssh-keygenGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa): /etc/cacti/id_rsaEnter passphrase (empty for no passphrase):Your identification has been saved in /etc/cacti/id_rsa.Your public key has been saved in /etc/cacti/id_rsa.pub.cacti# chown -R www-data: /etc/cacti/{id_rsa,id_rsa.pub}cacti# chmod -R 0400 /etc/cacti/id_rsa


31/103

install: ssh script

31

cacti# vi /var/www/html/cacti/scripts/ss_get_by_ssh.php...

$ssh_user= 'cacti'; # SSH username$ssh_port= 22; # SSH port$ssh_iden='-i /etc/cacti/id_rsa';# SSH identity...


32/103

install: templates

32


33/103


34/103

install: templates

34


35/103

Installation

Cacti


Configuring Servers

35


36/103

Configure Servers

configure net-snmpd

add a cacti user in MySQL

configure ssh access

36


37/103


38/103

Configure Servers: mysql user

Add cacti user

Verify if login works from cacti servers

38

servermysql>GRANT SUPER, PROCESS ON *.* TO'cactiuser'@'cactiserver' IDENTIFIED BY"password";

cacti#mysql -h ubuntu-server-test -e "SELECTVERSION()" -u cactiuser -ppassword

+-------------------+| VERSION() |+-------------------+| 5.1.49-1ubuntu8.1 |

+-------------------+


39/103

Configure Servers: ssh access

Add shell user cacti

Copy the public key

Verify

39

server# useradd cactiserver# getent passwd cacticacti:x:1001:1002::/home/cacti:/bin/sh

server#mkdir /home/cacti/.ssh/server# cat > /home/cacti/.ssh/authorized_keys


40/103

Topics

Why Graph?


How Cacti works

Installation

Add Graphs

Demo

Graphs!

Extras

40


41/103

Adding Graphs

41


42/103

Adding Graphs

42


43/103

Adding Graphs

43


44/103

Adding Graphs

44

Apply wanted templates on the device-MySQL & GNU/Linux templates

Click on Create Graphs:

Select graphs to create and Add!

Might ask some more questions


45/103

Adding Graphs

45


46/103

Adding Graphs

46

Assign ucd/net SNMP template to have:-cpu usage/load

-memory used/available

-network statistics

-free disk space

Different way of creating


47/103

Adding Graphs

47


48/103


49/103

Adding Graphs: Graph Trees

49

Tree to browse through graphs

By host/individual graph, mixed


50/103

Topics

Why Graph?


How Cacti works

Installation

Add Graphs

Demo

Graphs!

Extras

50


51/103

Topics

Why Graph?


How Cacti works

Installation

Add Graphs

Demo

Graphs!

Extras

51


52/103

MySQL Handlers

52


53/103

MySQL Handlers

53

Table scans, backup?


54/103

MySQL Handlers

54


55/103

MySQL Handlers

55

Take backup here


56/103

MySQL Connections

56


57/103

MySQL Connections

57

max_connections not reached yet

Why more incoming connections?


58/103

MySQL Replication

58


59/103


60/103

MySQL Replication

60


61/103

MySQL Replication

61

In 2 hours, almost 6900seconds replication lag


62/103


63/103

MySQL Temporary Objects

63

keep an eye on:* temp tables* temp disk tablesif it grows > query optimization


64/103

MySQL Select Types

64


65/103

MySQL Select Types

65

Why this sudden peak in rangescans?

SQ S


66/103

MySQL Select Types

66

M SQL S l T


67/103

MySQL Select Types

67

Query behavior changed?Other explain plan chosen?

M SQL S t


68/103

MySQL Sorts

68

M SQL S t


69/103

MySQL Sorts

69

The peaks should be understood,not necessarily be removed as theymight be really fast in terms of rows

M SQL C d C t


70/103

MySQL Command Counters

70

M SQL Bi L


71/103

MySQL Binary Logs

71


72/103


73/103

M SQL Q R Ti


74/103

MySQL Query Response Time

74

M SQL Q R Ti


75/103

MySQL Query Response Time

75

I DB B ff P l


76/103

InnoDB Buffer Pool

76

I DB B ff P l


77/103

InnoDB Buffer Pool

77

DB Restart

Time to fill buffer

I DB Ch k i t A


78/103

InnoDB Checkpoint Age

78


79/103

InnoDB Buffer Pool Activity


80/103

InnoDB Buffer Pool Activity

80

InnoDB I/O


81/103

InnoDB I/O

81

InnoDB I/O


82/103

InnoDB I/O

82

More readrequests Increase in

load

InnoDB Row Operations


83/103


83



84/103


84

Good view on capacity of the system


85/103


86/103

86

more row reads,coming from disk


87/103

87

this spike in IO readsdid not cause more disk reads


88/103

88


89/103

CPU Usage


90/103

CPU Usage

90


91/103

Disk Operations


92/103

Disk Operations

92

Topics


93/103

Topics

Why Graph?What is Cacti and mysql-cacti-templates?

How Cacti works


Demo

Graphs!

Extras

93

non-MySQL mysql-cacti-templates


94/103


ApacheJMX

Memcached

MongodbNginx

OpenVZ

Redis

Unix

94



95/103


95



96/103


96



97/103


97



98/103


98

Other Graphing solutions


99/103

Other Graphing solutions

Munin and munin-mysqlOpenNMS and MySQL-SNMP

Zabbix with appaloosa-zabbix-templates

99

Creating Your Own Graphs


100/103

Creating Your Own Graphs

Creating cacti graphs not trivial/portablemysql-cacti-templates to the rescue!

http://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphs

100

Using Cacti To Graph MySQLs Metrics
http://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphshttp://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphshttp://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphshttp://code.google.com/p/mysql-cacti-templates/wiki/CreatingGraphs


101/103

Using Cacti To Graph MySQL s Metrics

Powerful ToolEasy to install

Open source!

101


102/103

Percona Live MySQL Conference NYCMay 26, 2011


103/103

May 26, 2011

http://www.percona.com/live/nyc-2011/
http://www.percona.com/live/nyc-2011/http://www.percona.com/live/nyc-2011/

collaborate2011 using cacti to graph mysql's metrics

Documents