slides: polyglot persistence for the mongodb, mysql & postgresql dba

Confidential

Polyglot Persistence

December 22nd, 2015

Art van Scheppingen

Severalnines

[email protected]

mailto:[email protected]

Confidential

Copyright Severalnines AB

2

Logistics

Webinar is recorded Replay available soon Feel free to ask questions at any time Use your control panel to contact us Or email us as well: [email protected]

mailto:[email protected]

Confidential 3

Agenda What is Polyglot Persistence? Example project Is Polyglot Persistence good or bad? Leverage ClusterControl for Polyglot Persistence

(demo) Q&A


Confidential


4

What is Polyglot Persistence?

Confidential

Copyright 2013 Severalnines AB

5


Using multiple specialized persistent stores rather than

one single general-purpose database

Confidential


6

Where does the term come from?

The way we work is changing Enterprise applications are becoming more complex Separate (devops/agile) teams Ownership of applications (Micro)services

Everyone has their preference Various programming languages Various storage systems

Confidential


7

Where does the term come from?

Monoglot Programming Only one programming language allowed Readability

All code is in the same language Support

One platform to support Knowledge

Everybody is an expert Is there a jack-of-all-trades language?

Confidential


8

Monoglot Programming

Does a carpenter only use just one tool?

Confidential


9

Monoglot Programming

Carpenters use a broad variety of tools

Confidential


10

Polyglot Programming

Polyglot Programming Use programming languages for what they are good at Flexibility

Use Java for a secure API Use Scala for real time stream processing Use Python for text analysis Tie everything together using AngularJS

Knowledge Everybody is expert at one or more languages

Confidential


11

Polyglot Programming

Confidential


12

Monoglot Persistence

Confidential


13

Data storage landscape changes

Relational data stores (RDBMS) Key-Value data stores Columnar data stores Document data stores Graph data stores

Confidential


14

Data stores overview

SoftwareRDMBS MySQL,

PostgreSQL, Oracle

Key-Value Redis, RiakColumnar Cassandra,

Hbase, InfiniDB

Document MongoDB, Couchbase

Graph Neo4J, Titan

Confidential


15

Data stores overview

Software AWS GoogleRDMBS MySQL,

PostgreSQL, Oracle

RDS, Aurora CloudSQL

Key-Value Redis, Riak DynamoDB DatastoreColumnar Cassandra,

Hbase, InfiniDB

Redshift Big Query

Document MongoDB, Couchbase

SimpleDB Bigtable

Graph Neo4J, Titan Titan + DynamoDB

-

Confidential


16

Even Hadoop becomes a polyglot

Confidential


17


Complex problems require different storage systems Use the right tool for the job, for example

Use PostgreSQL for financial data Use MySQL for website contents Use MongoDB for user profiles Use Cassandra for real time streams Use Neo4J for recommendation analysis

Confidential


18

Use the right tool for the right job Document storage: MongoDB

Confidential


19

Use the right tool for the right job Columnar storage: Cassandra

Confidential


20

Use the right tool for the right job Graph storage: Neo4J

Confidential


21


Confidential


22

Example project

Confidential


23

Project management lifecycle

Confidential


24


Confidential


25


Confidential


26

Development

Requirements are gathered Create a recommendation system Storage must be (blazingly) fast Document must contain user details Data must be kept until the next visit

Choose the right tool for the right job Store data as a document Use MongoDB

Development

Confidential


27

But what happens after development?

Delivery Ops sets up the application on the infrastructure Monitoring is added to see if the application works If everything is fine the product will be handed over

What’s next? Systems monitoring Backups Scaling Software updates BI

Confidential


28

The ops perspective

Confidential


29

Systems monitoring

Supported data stores so far: MySQL Memcached

How do we monitor MongoDB? Integration with Nagios / Zabbix / Shinken Integration with Graphite / OpenTSDB / InfluxDB

What should we monitor? Key performance metrics SQL vs Mongo queries

Confidential


30

Backups

Supported backups so far Xtrabackup streams

How do we backup MongoDB? What should we backup from MongoDB? How do we restore from a backup? How do we provide a backup for development?

Confidential


31

Scaling out

We are data hoarders and like to keep everything Scaling options in MySQL

Read-bound workloads: add more slaves Write-bound workloads: sharding or NDB Cluster

Scaling in MongoDB? When should we scale out? How do we add a new shard? How do we redistribute between shards? How do we add a new read slave?

Confidential


32

Software updates

When do we need to update with MySQL? Once we hit a bug? Every major release? Every minor release?

When do we need to update with MongoDB? How do we know the impact?

Confidential


33

Business Intelligence

BI with MySQL? Export full database from a slave Allow schemas/tables to be queried for specific tables Most BI tools can connect to MySQL

BI with MongoDB? Native MongoDB analytics tools Write your own exports BI connector (since version 3.2)

Confidential


34

A new project emerges

Confidential


35

With new requirements

Requirements are gathered Allow users to view their earnings Data must always be 100% reliable Storage must be ACID compliant

Choose the right tool for the right job Store data relational Use PostgreSQL

Confidential


36

Is Polyglot Persistence good or bad?

Confidential


37

The drawbacks

More specific (domain) knowledge is needed Additional tooling may be necessary Need more hands to maintain the systems

Confidential


38

The benefits

More flexibility to your company Driven and dedicated devops who take ownership Use the right tool for the right job

Confidential


39

The challenges

Deploy various data stores Handle HA and failovers Scaling and sharding Interpret graphs and monitoring Schedule backups and handle recovery Handle updates/upgrades

Confidential


40

Polyglot Persistence is here to stay

Variety of data storage needs keeps growing Polyglots in the cloud Better be prepared for it

Confidential


41

Leverage ClusterControl for Polyglot Persistence (demo)

Confidential


42

Q&A

Confidential 43

Thank You! ClusterControl

www.severalnines.com/product/clustercontrol

ClusterControl – Getting Started www.severalnines.com/getting-started

Polyglot Persistence meetups http://goo.gl/64Ga5z

Severalnines Blog www.severalnines.com/blog

Contact: [email protected]

New ClusterControl Tips & Tricks

http://www.severalnines.com/product/clustercontrol

http://www.severalnines.com/getting-started

http://goo.gl/64Ga5z

http://goo.gl/64Ga5z

http://www.severalnines.com/blog

slides: polyglot persistence for the mongodb, mysql & postgresql dba

Technology