running mysql on linux
TRANSCRIPT
Running MySQL on Linux
Peter Zaitsev CEO, Percona Great Wide Open Atlanta,GA 16 March 2016
About Percona
Solu@ons for Success with MySQL and MongoDB
Support, Consul@ng,
Managed Services
SoIware for Success with Data Management
100% Open Source, no lock-‐in
required
About the Presenta8on
• Distribu@on • Hardware • OS Configura@on • MySQL Installa@on • MySQL Configura@on
Cover what you need to run
MySQL on Linux
successfully
3
Why Linux ?
Most Popular PlaVorm for Produc@on Deployments
The main plaVorm for MySQL Engineering Efforts
Great Default Choice!
4
Choices
• Long Support; Stability “Server Grade Distribu@on”
• Especially with Newer Hardware
Recent enough
• RHEL/CentOS/Oracle/Amazon • Debian, Ubuntu
Most Popular Choices
Things to Consider
Low Overhead monitoring support “perf”, Irace, eBPF
NVMe Docker Support
File System Encryp@on
New Block Device Infrastructure SoIware RAID
6
Hardware
• Really! Hardware can go a long way!
• With 100 queries/page = 120K page views/minute
• Some 80M page views/day (considering daily spike)
Over 200K simple queries/sec on modern
hardware
7
MySQL 5.7 Performance Improvements Sysbench Benchmark
*Informa)on from Oracle OpenWorld presenta)on by Geir Hoydalsvik
Starts with 8 Threads What about 2-‐4 threads?
Hardware
• Go for fast cores. All cores are rarely used these days
• Cache and fast memory bus are important
CPU
• OIen most important for performance • Your working set must fit in memory well
• Less memory = more pressure on IO Memory
9
Choose Solid State for Main Storage
Proprietary PCE-‐E and NVMe
SATA SSDs very inexpensive
SoIware RAID to beat SATA limits
SSDs available in the cloud – use them
Not all Flash is created equal
10
More on SSD/Flash
Think about Write Endurance
Behavior on Power Loss
Data Reten@on
Firmware
11
Know your Goals
• Reads? • Writes? • Random? • Sequen@al?
Are you bound by
• Sustained writes? • Handle spikes? Do you need
• Many devices reach peak IOPs at unreachable concurrency What concurrency do you operate?
• SAN/NAS is a frequent cause of surprise Think about latency/
response @me
12
Performance vs. Memory 13
Compression of Performance Gains 14
Network
Latency is king
Minimize number of hops between Database and Web Server
Ensure running 1Gbit link speed at least (10Gb is gaining popularity)
Monitor for packet loss and latency
Network problems are oIen blamed on database
15
Network Tuning
Might be needed some@mes • echo 8192 > /proc/sys/net/ipv4/tcp_max_syn_backlog • Mysql: back_log=1000 • net.ipv4.ip_local_port_range="1024 64000“ • net.core.somaxconn = 1024 • net.core.rmem_max = 16777216 • net.core.wmem_max = 16777216 • net.ipv4.tcp_rmem = 4096 87380 16777216 • net.ipv4.tcp_wmem = 4096 65536 16777216 • net.ipv4.max_tw_buckets=360000 • net.core.netdev_max_backlog = 2500 • ifconfig eth0 txqueuelen 1000
Persistent connec@ons with MySQL • Thread_pool (Percona Server, MariaDB, MySQL Enterprise) • Proxy
16
Virtualiza8on and Cloud
Virtualiza@on has cost
Cloud rarely provides highest performance hardware (Getng beuer)
There is a lot of MySQL use in the Cloud and Virtualized Environments
17
Linux Configura8on
18
In General…
For most workloads Linux Runs MySQL surprisingly well with no addi@onal tuning
19
General Configura8on
• vm.swappiness=0 Reduce Tendency to swap
20
General Configura8on
Reduce the File Cache • vm.dirty_ra@o=5
NUMA can cause problems • hup://bit.ly/I0GSt3 • Interleaving with numactl • MySQL 5.6+ Supports innodb_numa_interleave
21
Storage Configura8on
• Re-‐Configure MySQL without OS Reinstall Separate OS and MySQL Par@@on
• CFQ can cause problems • Deadline or Noop are oIen beuer choice • elevator=deadline on kernel boot
Configure IO Scheduler
• # echo 100000 > /sys/block/sdX/queue/nr_requests Queue length (especially MyISAM)
• Virtually no overhead when not in snapshot mode • Reserve some space for snapshots Consider using LVM
22
Which one do you use ? FILESYSTEMS
23
Filesystems
XFS – “Old” High Performance Favorite
EXT4 – Works very good for some workloads
Avoid: • Ext2 • Ext3 • ReiserFS • ZFS (may be good for development) • Btrfs
24
FS Tuning
rela@me (noa@me)
ext3: tune2fs –O dir_index -‐c –l –i 0 –e remount-‐ro
xfs: nobarrier • Assuming RAID w BBU or Flash
25
WHAT MYSQL VARIANT ARE YOU RUNING ?
26
Main Choices
MySQL
Percona Server
MariaDB
27
MySQL Versions
Newer versions scale beuer
MySQL 5.7 is most scalable
Percona Server 5.7 brings even more improvements J
Use 5.7 for New Development
Upgrade to MySQL 5.7 if you need it now
28
Check out Percona XtraDB Cluster
Based on Galera Technology
Innodb Storage Engine you already know
Synchronous Replica@on
Parallel Apply
Automa@c Provisioning
No Stale Reads
Not for all workloads
29
How to Install
• hup://dev.mysql.com/downloads/repo/ • hup://www.percona.com/doc/percona-‐server/5.7/[email protected]
Use Repositories: Best
• Need older Version Download RPM/DEB
• Good for tes@ng with MySQL Sandbox Use Tar.gz package
• hups://hub.docker.com/_/percona/ Docker Images
• Do you really have good reason to do it ? • Many problems are caused by bad builds Build our own
30
MySQL Configura8on
• Even newer MySQL 5.7 defaults are unlikely to be op@mal
You need to tune MySQL configura@on
• hup://bit.ly/1fuP0SZ Check out presenta@on which goes in depth into configura@on tuning
• Avoid obsessive tuning disorder Typically getng 5 variables right
responsible for 90% performance gain
31
MySQL Variables
• How many connec@ons server will support ? max_connec@ons
• How many “open” tables there will be allowed table_open_cache
• How many files can be open open_files_limit
32
Top Variables
• Set 80% of memory some@mes more Innodb_buffer_pool_size
• Best setng in most cases Innodb_flush_method=O_DIRECT
• Set 256MB or more • Larger logs = longer recovery @me
Innodb_log_file_size
• Truly ACID: 1 • Can Afford Data Loss: 2
Innodb_flush_log_at_trx_commit=?
33
Advanced Ideas
• taskset -‐pc 0,12,2,14 `cat /var/lib/mysql/mysqld1.pid`
Running mul@ple MySQL instances ?
• Jemalloc • Tcmalloc –old favorite
Memory Alloca@on Hostspot ?
• Roll back easily if things go wrong
LVM Snapshot before maintenance
or upgrade
34
Beware of the “Scripts”
• Timeouts might be not enough for safe database shutdown
• The automated upgrade/check
Start/Stop Scripts can be nasty
35
Automa8on
Manual approach is a no
• Does not Scale • And is Error Prone
Automate installa@on, upgrades,
configura@on
• Puppet, • Chef • Ansible
Keep Configura@on under version control
• At very least leave comments on what you chance and why
36
Monitoring and Trending
• Just make sure you’re doing it Many tools to choose from!
• Check out PMP • hup://www.percona.com/soIware/percona-‐monitoring-‐plugins
Nagios and CACTI
• hup://grafana.org/ Grafana – new genera@on trending
• hup://bit.ly/1M6r7Vk Check out Grafana + Prometheus Monitoring
37
Prometheus + Grafana Example 38
Linux OOM Killer
• SSH • MySQL Server • Various jobs • Backup • Percona Toolkit • Background Batch jobs
What are your
priori@es ?
• echo -‐17 > /proc/2592/oom_adj • do not kill ever
• echo 10 > /proc/2592/oom_adj • More like to be killed
Configuring
39
Tools you need to know
General Linux
• Top • Vmstat • Iostat • Strace • Gdb • perf
MySQL
• Percona Toolkit • Innotop • Percona Xtrabackup • MyDumper
40
Resources for MySQL on Linux
• Brendan Gregg’s web site – hup://www.brendangregg.com/
• Percona’s Blog – hup://www.percona.com/blog/
• Ernie Souhrada IOPS inves@ga@on – bit.ly/2546xes
41
www.percona.com
Percona Live Data Performance Conference
• April 18-‐21 in Santa Clara, CA at the Santa Clara Conven@on Center
• Register with code “GWO” to receive 15% off at registra@on
• MySQL, NoSQL, Data in the Cloud
www.perconalive.com
