capacity planning for your data stores - scale16x planning for your...whoami • chief evangelist,...
TRANSCRIPT
Capacity planning for your data storesColin Charles, Chief Evangelist, Percona [email protected] / [email protected] http://bytebot.net/blog/ | @bytebot on TwitterSCALE16x, Pasadena, California, USA10 March 2018
whoami
• Chief Evangelist, Percona Inc• we make 100% open source tools, enhanced MySQL/MongoDB
servers, XtraBackup, TokuDB, work on MyRocks/MongoRocks, Percona Toolkit and many more!
• Founding team of MariaDB Server (2009-2016)• Formerly MySQL AB/Sun Microsystems • Past lives include Fedora Project (FESCO), OpenOffice.org• MySQL Community Contributor of the Year Award winner 2014
License• Creative Commons BY-NC-SA 4.0• https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Database, data store, etc.• Database: 1. a structured set of data held in a computer,
especially one that is accessible in various ways. [Google]• Data store: A data store is a repository for persistently storing and
managing collections of data which include not just repositories like databases, but also simpler store types such as simple files, emails etc. [Wikipedia]
Presto, the Distributed SQL Query Engine for Big Data• Presto allows querying data where it lives, including Hive,
Cassandra, relational databases or even proprietary data stores. A single Presto query can combine data from multiple sources, allowing for analytics across your entire organization.
• Facebook uses Presto for interactive queries against several internal data stores, including their 300PB data warehouse. Over 1,000 Facebook employees use Presto daily to run more than 30,000 queries that in total scan over a petabyte each per day.
Why capacity plan?
Revenue Management• Cannot sell more than you actually have• Seat map: theatre, planes• Rooms: types, quantity
Uptime
Percentile target Max downtime per year
90% 36 days
99% 3.65 days
99.5% 1.83 days
99.9% 8.76 hours
99.99% 52.56 minutes
99.999% 5.25 minutes
99.9999% 31.5 seconds
You can start now!• Start collecting metrics, NOW!• metric: standard of measurement• You need your baseline, your traffic patterns
Baseline• How well is your current infrastructure working?• what is your QPS? QPS before performance degradation? QPS
before performance degradation affects user experience?• What more will you need, in the (near) future, to maintain acceptable
performance?• load that causes failure - alerting? Add/remove capacity, what do
you expect? When do you spin up new resources/size new orders?• How do you manage the resources?• Iterate!
MySQL world• Operating System• vmstat, netstat, df, ps, iostat, uptime • MySQL• SHOW [TABLE] STATUS, SHOW PROCESSLIST, INFORMATION_SCHEMA, PERFORMANCE_SCHEMA, slow query log, mytop/innotop
Working Set Size Estimation• http://www.brendangregg.com/wss.html • Size main memory for your database, with the intent of keeping it
from swapping. Measure in bytes over an interval.• https://github.com/brendangregg/wss
A note on swap• On a machine with 32GB of RAM, and database stored on Intel 750
NVMe storage, a uniform sysbench workload gives about 44K QPS, 95% response time of 3.5ms (buffer pool=24GB)
• Swapping when buffer pool=32GB, gives 20K QPS, response time of 9ms
More on swap• Try 48GB for a buffer pool (more than RAM), and you get 6K QPS,
35ms response time
MySQL 5.7 online buffer pool resize
Swappiness• https://www.percona.com/blog/2017/01/13/impact-of-swapping-on-
mysql-performance/• Don’t set vm.swappiness=0 with a modern kernel (3.5-rc1 or
backports like CentOS 2.6.32-303)• https://www.percona.com/blog/2014/04/28/oom-relation-vm-
swappiness0-new-kernel/ • Otherwise the OOM killer comes for you• vm.swappiness=1 is preferred nowadays
Sharding• Sharding• Split your data across multiple nodes• Sharding alone isn’t enough, you need ability to split reads/writes• Tools: ProxySQL, Vitess, Tumblr JetPants, Tungsten Replicator,
SPIDER (MariaDB 10.3)
Database specific watch points• QPS (SELECTs, INSERTs, UPDATEs, DELETEs)• Open connections• Lag time between masters/slaves• Cache hit rates
Bottlenecks?• Bottleneck: reads or writes?• High CPU?• I/O?• Lag on replicas and the queries seem fine• Locking?
Context-based metrics• pt-query-digest: https://www.percona.com/doc/percona-toolkit/
3.0/pt-query-digest.html • Analyse queries from logs, processlist, tcpdump • Box Anemometer: https://github.com/box/Anemometer • Analyse slow query logs to identify problematic queries• Commercial tools exist for this as well
Percona Monitoring & Management (PMM)• Query analytics + visualise it (w/sparklines, etc.)• Metrics monitor: OS & MySQL• Built on-top of open source: Prometheus, Consul, Grafana,
Orchestrator • Get Docker container for “server”, get agent for “client”• http://pmmdemo.percona.com/
PMM
Understanding your workload better• Percona Lab Query Playback• https://github.com/Percona-Lab/query-playback • Query Playback is a tool for replaying the load of one database
server to another• --slow-query-log --log-slow-admin-statements --log-slow-verbosity=microtime --long-query-time=0
Load balancing• Do you just pick a random database server?• Load balancing strategies matter• Strategy:• Pick 2 random servers• Machine has less load?• Send request
ProxySQL• Connection Pooling & Multiplexing• Read/Write Split and Sharding• Seamless failover (including query rerouting), load balancing• Query caching• Query rewriting• Query blocking (database aware firewall)• Query mirroring (cache warming)• Query throttling and timeouts• Runtime reconfigurable • Monitoring built-in
Storage capacity planning• Small single server deployment: 3-4x working capacity is not a bad
option• size of database and data files (/var/lib/mysql)• size of largest table * 2 (for tmp/sort files)• size of each local logical backup• 5% free for OS• The above may not necessarily make sense for large scale
deployments
Prophet• Works by fitting time-series data to get a prediction of how that metric will look in
future• Generalised Additive Model• Linear or logistic regression + additive model applied to regression• Paper: https://facebookincubator.github.io/prophet/static/
prophet_paper_20170113.pdf• Tip: have at least a year of data to fit the model (you may miss seasonal effects
otherwise)• Tip: holidays (https://facebookincubator.github.io/prophet/docs/holiday_effects.html) • Our evaluation: https://www.percona.com/blog/2017/03/20/prophet-forecasting-our-
metrics-or-predicting-the-future/
Auto-scaling frameworks• Scalr• Amazon• Vertical: grow the instance• Horizontal: replicas• EC2: auto scaling + groups• Amazon RDS Aurora, Google Cloud Spanner, Azure Cosmos DB
If done properly…
Looking ahead• OtterTune: automatically find good settings for a database
configuration - https://github.com/cmu-db/ottertune • Peloton: self-driving database management system - http://
pelotondb.io/
60% reduction in latency, 22-35% better throughput
https://aws.amazon.com/blogs/ai/tuning-your-dbms-automatically-with-machine-learning/
In conclusion…• Capture the signal: high noise alerting systems fail due to human
psychology• Revenue management, operations research, management science
are good to read• Always be capturing metrics• Know your baseline and business requirements• Shard, load balance appropriately• Monitor! Be proactive not reactive
The Art of Capacity PlanningScaling Web Resourcesby John Allspaw
Database Reliability Engineering by Laine Campbell & Charity Majors
Percona Live Santa Clara• Attend Percona Live Santa Clara 2018!• https://www.percona.com/live/18/ • Amazing keynotes: Brendan Gregg (on the recent performance hits
from Spectre/Meltdown), Upwork on why they use MongoDB• Amazing talks: cloud, PostgreSQL, DevOps, etc.• Want a discount? Email: [email protected] and mention
you saw this talk at SCALE16x (I presume a 10-15% discount is something I can wrangle from the marketing team)
See us at the Expo Hall! • Come by and say hello• We have stickers, battery packs to charge your phones, as well as
bottle openers• We’re there to answer any of your MySQL or MongoDB queries!
Thank You. Q&[email protected] / [email protected] @bytebot on Twitter | http://www.bytebot.net/blog/ slides: https://slideshare.net/bytebot