slides: introducing the new clustercontrol 1.2.9 - with live demo

Confidential

ClusterControl 1.2.9

March 2015

Johan Andersson

CTO, Severalnines

[email protected]

Confidential

About us

¤  Founded 2011

¤  Ex-MySQL/Sun/Oracle staff

¤  HQ in Stockholm, Sweden

¤  Focus: Cluster automation and management

¤  MySQL, Percona, MariaDB, MongoDB, TokuMX, PostgreSQL

¤  Over 7000 deployed clusters

¤  Over 100 enterprise customers

2

Copyright 2012 Severalnines AB

Confidential

Customers

3

Copyright Severalnines AB

begin; select * from sbtest1 where id=4 for update;

Confidential

What Problems do we Address?

4


Manage Scale

Monitor Deploy

Confidential

Deployment - Beyond Puppet or Chef

¤  Puppet/Chef are appropriate for a group of single-node components ¤  E.g. web servers can be clones of each other..

¤  Distributed databases are complex ¤  Different node types & roles

¤  Orchestration of procedures in a specific order

¤  Using e.g. Chef for deploying a distributed database ¤  Yes, it is possible

¤  Not much Chef functionality is leveraged

¤  Prepare to write code

5


Confidential

Monitoring - Beyond Nagios

¤  What do you do when the application is slow?

¤  Is it Disk? CPU? RAM? Badly written queries?

¤  What are the symptoms? (Replication queues, Page Faults, locks, # connections, deadlocks …)

¤  How do you avoid problems?

¤  How do you plan for capacity?

6


Confidential

Management - Beyond Monitoring

7


Confidential

Scaling - Beyond Manual Operations

8


Confidential

ClusterControl - In a nutshell

9


Manage Scale

Monitor Deploy

Confidential

Configurators

¤  MongoDB / TokuMX Sharded Cluster

¤  MongoDB / TokuMX Replica Set

¤  MySQL Galera Cluster (Codership)

¤  Percona XtraDB Cluster

¤  MariaDB Cluster

¤  MySQL Cluster (NDB)

¤  MySQL Replication 5.6

10


Confidential

Deploy From ClusterControl ¤  MongoDB ReplicaSet

¤  Percona XtraDB Cluster / MariaDB Cluster

¤  Standalone nodes (MongoDB, MySQL, MariaDB)

11


Confidential

Add Existing Clusters To ClusterControl ¤  Galera Cluster (Codership/Percona/MariaDb)

¤  Single MySQL Server nodes

¤  MongoDB ReplicaSet

¤  Single PostgreSQL nodes

SSH access (pubkey auth) is required from: Controller -> Nodes Controller -> Controller (itself)

12


Confidential

ClusterControl Automation & Management

¤  Management ¤  Multi cluster/data-center

¤  Automate repair/recovery

¤  Database upgrades

¤  Backups

¤  Configuration management

¤  Cloning

¤  One-click scaling

13


Confidential 14


Confidential 15


Party Time!

Confidential

Getting Started ¤  Dedicated VM for the controller

¤  Specs ¤  2 CPU cores ¤  2-4GB RAM ¤  20GB diskspace

¤  No agents ¤  Host stats gathered via SSH

¤  Installation ¤  YUM/APT ¤  Setup SSH access to DB nodes ¤  http://repo.severalnines.com/

16


Confidential

What’s new in 1.2.9?

¤  PostgreSQL Support

¤  Improved HAProxy support

¤  Hybrid Replication with Galera Clusters

¤  Galera Replication Traffic Encryption

¤  Encrypted communication between ClusterControl and MySQL-based systems

¤  Improved Backup and Restore Database

¤  Galera: Bootstrap Cluster , Recover Non-Primary node.

¤  MySQL: Deadlock detection / Long Query

¤  MongoDb: Improved pagefault and locking inspection

17


Confidential

PostgreSQL

¤  Support for single PostgreSQL systems ¤  Version 9.x supported

¤  Add existing PostgreSQL servers to ClusterControl

¤  Process management / auto recovery of failed instance

¤  Backup

¤  Query Monitor

18


Confidential

Improved HAProxy

¤  Richer UI with built-in HAProxy stats ¤  View Stats

¤  Disable / Enable nodes part of the load balancer

¤  Powerful deployment capabilities ¤  Specify what DB servers to include on the backend

¤  Specify ACTIVE/BACKUP backend (DB) servers

¤  Tune timeouts

¤  Templated configuration: /usr/share/cmon/templates/haproxy.cfg /usr/share/cmon/templates/mysqlchk.* /usr/share/cmon/templates/mysqlchk_xinetd

19


Confidential

Hybrid Replication For Galera Clusters

¤  Allows user to add a Replication Slave to a Galera Cluster

¤  GTID and non-GTID replication is support ¤  MariaDB GTID is not

currently supported

¤  Single click failover slave from one master to another ¤  Requires to Masters

¤  Restage a slave with data from master. ¤  Using xtrabackup


20

MySQL [Slave]

Confidential


¤  Allows user to add a Replication Slave to a Galera Cluster

¤  One Galera Server must be a Master: ¤  log-bin = binlog ¤  server-id = 100 ¤  log-slave-updates=ON For GTID replication (MariaDB GTID implementation is not yet supported). ¤  gtid_mode=ON ¤  enforce-gtid-consistency=ON

¤  UI - > Manage -> Configuration, change the my.cnf file on the master node to be.

¤  Restart the MySQL server.


21

Confidential

Hybrid Replication For Galera Clusters ¤  Create a tempate for the slave MySQL server

¤  UI -> Manage -> Configuration -> Create New Template

¤  The slave must have: ¤  server-id = 900

For GTID replication (MariaDB GTID implementation is not yet supported).

¤  gtid_mode=ON

¤  log-slave-updates=ON

¤  enforce-gtid-consistency=ON

¤  log-bin = binlog

.. and NO wsrep variables!

22


Confidential


23


Confidential

Galera Replication Traffic Encryption ¤  Setup secure communication between the Galera nodes

¤  Creates SSL certs

¤  1024, 2048 or 4096 bits encryption.

¤  s9s_galera --encrypt-replication –i <clusterid> -o enable|disable|status ¤  SSL certs are stored in /etc/ssl/galera/

cluster_<clusterid> on the controller

¤  Cluster is stopped

¤  socket.ssl_cert= … and socket.ssl_key=… is set on in wsrep_provider_options

¤  Cluster is started

¤  SSL certs are transferred automatically when adding nodes. 24


Confidential

Encrypted communication between ClusterControl and MySQL-based systems

¤  No tool to create the SSL certs (yet): ¤  https://dev.mysql.com/doc/refman/5.6/en/creating-ssl-

certs.html

¤  Store the master set of SSL certs on the controller, in: ¤  /etc/ssl/mysql/cluster_<clusterid>

¤  Copy SSL certs to MySQL Server nodes: ¤  mkdir /etc/mysql/certs

¤  Update my.cnf [mysqld]

ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/server-cert.pem ssl-key=/etc/mysql/certs/server-key.pem

25


Confidential

Encrypted communication between ClusterControl and MySQL-based systems

¤  In /etc/cmon.cnf set: cluster_certs_store=/etc/ssl/mysql/cluster_1 cluster_ssl_key=/etc/ssl/mysql/cluster_1/client-key.pem cluster_ssl_cert=/etc/ssl/mysql/cluster_1/client-cert.pem cluster_ssl_ca=/etc/ssl/mysql/cluster_1/ca.pem

¤  GRANT the cmon user on the MySQL nodes: GRANT ALL ON *.* TO ‘cmon’@’<cc server>’ IDENTIFIED BY ‘<cmon password> REQUIRE SSL;

¤  You can also encrypt the communication between cmon and the cmon db (mysql server), by setting: ¤  cmondb_ssl_key, cmondb_ssl_cert, cmondb_ssl_ca

¤  SSL certs are transferred automatically when adding nodes. ¤  cluster_certs_store must be set in cmon.cnf

26


Confidential

Restore Backup

27


Confidential

Galera: Bootstrap Cluster

¤  ClusterControl cannot always automatically recover a cluster if it cannot determine the most advanced node ¤  E.g, X nodes maybe unreachable (by SSH)

¤  Filesystem may be corrupted on one or more nodes

¤  Administrator must make the decision which node to bootstrap from

¤  Bootstrap Cluster creates a new Cluster based on one of the nodes ¤  The other nodes will then recover from the Bootstrapped node.

28


Confidential

Galera: Bootstrap Cluster

29


Confidential

Puppet Module and Chef Recipe

¤  Automate installation of ClusterControl

¤  http://severalnines.com/blog/devops-guide-database-infrastructure-automation-ecommerce-replay-slides

¤  Puppet/Chef only used for initial deploy ¤  ClusterControl handles the management and automation

aspects.

30


Confidential

31


Confidential

QA

32


slides: introducing the new clustercontrol 1.2.9 - with live demo

Technology