getting started with mariadb with docker

LAYOUT

Title Slide (Dark)

MariaDB + DockerFrom Development to Production

LAYOUT

Title and ContentPowerPoint DefaultConclusion

Containers + Databases = Happy Developers

Ephemeral Containers + Databases = DevOps headaches

4 Things you must use to evaluate• Data Durability & Redundancy• Dynamic Self Discovery & Cluster formation• Self Healing (as containers enter and leave)• Application Tier discovery of Database Cluster

LAYOUT

Title Slide (Dark)

Part OneWhere are we now

LAYOUT

Quote (Blue)

We’re building a database that iseasy to use, easy to extend, and easy to deploy:

on premise, in the cloud, or hybrid, operational or analytical– and with the languages and frameworks you prefer.

LAYOUT

Title OnlyPowerPoint DefaultExisting Deployment Models Are Broken

Versioncontrol

1. Development 2. Test 3. Stage / Production

Developer QA / QE Sysadmin

LAYOUT

Chart

Right click and select “edit data” to change numbers.

Architectures Are Complex & Static

Challenges• Orchestration

• Complexity

• Maintainability

• Durability

• Consistency

• Scalability

• Cost ($)

• (Hybrid) CloudEnterprise Environment

Legacy Mainframe

Operational Database

Caching Layer

Pricing / Inventory / Billing

Real-time Decisioning

Real-time Consumer facing

Streaming Data

Data WarehouseData Lake

RDBMS Transactional Systems

LAYOUT

Title Slide (Dark)

Part 2Containers

LAYOUT

Title and ContentPowerPoint DefaultWhat do Containers give me?

Encapsulation of Dependencies• O/S packages & Patches• Execution environment (e.g. Python 2.7)• Application Code & Dependencies

Process Isolation• Isolate the process from anything else running

Faster, Lightweight virtualization

LAYOUT

Title OnlyPowerPoint DefaultVirtual Machines vs. Containers

App 1 App 2 App 3

Bins/Libs Bins/Libs Bins/Libs

Guest OS Guest OS Guest OS

Hypervisor

Host Operating System

Infrastructure

Docker Engine

Operating System

Infrastructure

App 1 App 2 App 3

Bins/Libs Bins/Libs Bins/Libs

LAYOUT

Title and ContentPowerPoint DefaultDockerfile - Example

FROM python:2.7

ADD . /code

WORKDIR /code

RUN apt-get update && apt-get -y install python-dev libssl-dev

RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 5000

CMD python app.py

LAYOUT

Title and ContentPowerPoint DefaultOpen Container Initiative (OCI) – Polyglot Vendors

Coalition of industry leaders join forces to eliminate fragmentation• Form a vendor-neutral, open source governance model under the Linux Foundation

• Establish common standards for container format and runtime

• Docker donated its container format, runtime and associated specifications

• Appoint maintainers for the libcontainer project

LAYOUT

Title and ContentPowerPoint DefaultDocker Toolchain in pictures

Machine provisions Docker Engines

Swarm clusters Docker Engines

Compose orchestrates Container deployment

Containers instantiate an image and are run by Docker Engine

Docker Machine Docker Compose

Docker Swarm

Docker Engine

Image

Images encapsulates your code, dependencies…

LAYOUT

Title and ContentPowerPoint DefaultWhat about orchestration and Management?

Orchestration and Management of Containers and higher-level constructs (services, deployments, etc….) is evolving

Amazon ECS

Google Container Engine

Azure Container Service

LAYOUT

Title Slide (Dark)

Part 3Databases & Docker

LAYOUT

Title and ContentPowerPoint DefaultRequirements

Data Redundancy• Containers are Ephemeral – Need more than one copy of the data

Dynamic Self Discovery & Cluster formation• Need to start and stop Containers when needed• Clusters needs to grow and shrink dynamically

Self Healing• Loss of nodes must not be fatal to the cluster integrity• Addition of nodes must scale capacity

Application Tier discovery of Database Cluster• Automatic discovery of nodes• Automatic routing of requests to the correct nodes

LAYOUT

Title Slide (Dark)

Part FourMariaDB

LAYOUT

BlankPowerPoint Default

MARIADB SERVEREnterprise-grade secure,

highly available and scalable relational database with a modern, extensible

architecture

MARIADB MAXSCALE MARIADB CLUSTERNext-generation database

proxy that manages security, scalability and high availability

in scale-out deployments

Multi-Master, synchronous replication - improves availability and scales

reads and writes

MariaDB Portfolio

MaxScale

LAYOUT

Chart (Text Right)


MariaDB Cluster

Multi-Master• Synchronous replication

Faster Failover• All nodes synchronized,

therefore equal

Scale reads and writes

MariaDB

MariaDB

MariaDB

Load Balancingand Failover

Application /App Server

LAYOUT

Chart (Text Right)


MaxScale + GaleraUse Case

Each application server uses only 1 connection

MaxScale selects one node as “master” and the other

nodes as “slaves”

If the “master” node fails, a new one can be elected

immediately

Galera Cluster + R/W split routing

MaxScale

LAYOUT

Title Slide (Dark)

Part FiveDemo

LAYOUT

Title and ContentPowerPoint DefaultDemo: Development Through Production

Development• Build & Run an App in Development

– Python + MariaDB

Production• Deploy to a Swarm cluster in Production

• Scale Web nodes– Add more Web containers behind HAProxy

• Database High Availability– Deploy 3 nodes Galera cluster– Deploy 2 node MaxScale

LAYOUT

Title OnlyPowerPoint Default

Python / Flask

Let’s Build An App!

Development

app

LAYOUT


Production

Virtual IP

Virtual IP MaxScale 1 MaxScale 2

HAProxy

1 2 3

Then Scale in Production...

Development

app

app1 app2 app3 app4 appN

LAYOUT

Title Slide (Dark)

Demo 1Build an Application

LAYOUT

Title and ContentPowerPoint DefaultDockerfile

FROM python:2.7RUN apt-get update && apt-get -y install python-dev libssl-devWORKDIR /app

RUN pip install Flask MySQL-python

ADD . /appEXPOSE 5000

CMD ["python", "app.py"]

LAYOUT

Title and ContentPowerPoint Defaultdocker-compose.yml

services: web: build: . ports: - "5000:5000" links: - mariadb hostname: dev.myapp.com environment: APP_MARIADB_HOST: dev_mariadb_1 APP_PASSWORD: foo mariadb: image: mariadb:10.1 environment: MYSQL_ROOT_PASSWORD: foo

LAYOUT

Title OnlyPowerPoint DefaultRoll The Application Behind Haproxy

Development

app

Production

Virtual IP


HAProxy

1 2 3

app1

LAYOUT

Title OnlyPowerPoint DefaultScale the Application Tier

Development

app

Production

Virtual IP


HAProxy

1 2 3

app1 app2 app3 app4 appN

LAYOUT

Title OnlyPowerPoint DefaultDocker Networking

Docker Host (swarm-2)

MaxScaleContainer

Endpoint


MariaDBContainer

Endpoint

“Bridge” Network

“Prod” Overlay Network


AppContainer

Endpoint


HAProxyContainer

Endpoint Endpoint

LAYOUT

Title and ContentPowerPoint DefaultDocker Networking

$ docker network create -d overlay --attachable myapp_back

$ cat docker-compose.stack.yml...networks: front: back: external: name: myapp_back

LAYOUT

Title and ContentPowerPoint DefaultSecrets… Opppsss

services: web: build: . ports: - "5000:5000" links: - mariadb hostname: dev.myapp.com environment: APP_MARIADB_HOST: dev_mariadb_1 APP_PASSWORD: foo mariadb: image: mariadb:10.1 environment: MYSQL_ROOT_PASSWORD: foo

LAYOUT

Title and ContentPowerPoint DefaultDocker secrets

$ cat docker-compose.stack.yml... secrets: app_password: file: ./app_password.txt mariadb_root_password: file: ./mariadb_password.txt xtrabackup_password: file: ./xtrabackup_password.txt

$ more ./app_password.txt appfoo

LAYOUT

Title Slide (Dark)

Demo 2Scale Web Tier

LAYOUT

Two ContentPowerPoint Defaulthaproxy & web services

services: haproxy: image: dockercloud/haproxy networks: - front - back volumes: - /var/run/docker.sock:/var/run/docker.sock ports: - 80:80 deploy: placement: constraints: [node.role == manager]

web: image: alvinr/demo-webapp-vote:mariadb environment: SERVICE_PORTS: "5000" VIRTUAL_HOST: "prod.myapp.com" APP_MARIADB_HOST: "maxscale" APP_USER: "app" APP_PASSWORD_FILE: "/run/secrets/app_password" APP_DATABASE: "test" networks: - back deploy: placement: constraints: [node.role != manger] secrets: - app_password

LAYOUT

Title Slide (Dark)

Demo 2Harden Database Tier

LAYOUT

Two ContentPowerPoint DefaultMariaDB Galera Cluster

mariadb_cluster: image: alvinr/mariadb-galera-swarm environment:... NODE_ADDRESS: "eth0" MYSQL_USER: "app" MYSQL_DATABASE: "test" MYSQL_PASSWORD_FILE: "/run/secrets/app_password"

MAXSCALE_PRIVS: "true" labels: com.mariadb.cluster: "myapp-prod-cluster"

networks: - back command: seed deploy: replicas: 1 placement: constraints: [engine.labels.com.mariadb.cluster != myapp-prod-cluster]

secrets: - mariadb_root_password - xtrabackup_password - app_password

LAYOUT

Two ContentPowerPoint DefaultMaxScale

maxscale: image: alvinr/maxscale-swarm... labels: com.mariadb.cluster: "myapp-maxscale" networks: - back deploy: replicas: 1 restart_policy: condition: on-failure delay: 5s placement: constraints: [engine.labels.com.mariadb.cluster != myapp-maxscale] secrets: - app_password

LAYOUT

Title Slide (Dark)

Part SixConsiderations & Conclusions

LAYOUT


DNS RESOLUTION• Docker assigns VIP to Service, each Task has

own IP• nslookup, dig, getent etc.

3rd PARTY • consul, etcd, zookeeper etc.

DOCKER EVENTS• https://docs.docker.com/engine/

reference/api/docker_remote_api/• Interlock -

ttps://github.com/ehazlett/interlock

Service Discovery - How to mesh nodes?

LAYOUT

Title OnlyPowerPoint DefaultStorage: Inside or Outside the Container?

Inside• Encapsulation

of Concerns

Outside• Separation of Concerns• Storage features (e.g. Snapshots)• 3rd Party options

– NetApp, Google Compute Engine, Rancher Convoy– Flocker

Host

Docker Daemon

Container

Docker Daemon

Container/dev/xvdb

/mnt/xx:/var/lib/mysql

Networked e.g. EBS Volume

Local Disk e.g. SSD / NVMe

LAYOUT

Title OnlyPowerPoint DefaultStorage: Data Container?

Inside• Managed like

other containers• Special rule for

Destruction• TBD: Performance

Host

Docker Daemon

Container

Docker Daemon

Container

--volumes-from {container name}

Host

LAYOUT

Title and ContentPowerPoint DefaultAnd...

• Swarm locking

• Image verification (trusted Images)

• AppArmor / Seccomp profiles

• Monitoring

• Heathchecks

• Rolling Upgrades

LAYOUT

Title and ContentPowerPoint DefaultSummary

One Solution Development -> Production• Define Images & Orchestration once• Reuse when needed, inject required behaviours

MariaDB in Production with Docker• Ops define the whitelisted images, security policies• Dev approve images to build upon• Eliminate complexity (and cost) of Deployment• Scale easily, maintain SLA requirements of component

LAYOUT

Title and ContentPowerPoint DefaultThanks and Q&A

Code• https://github.com/alvinr/docker-demo/tree/master/mariadb/vote

Docker Images• https://hub.docker.com/_/mariadb/

MariaDB & Docker deployment guide• https://mariadb.com/kb/en/mariadb/installing-and-using-mariadb-via-docker/

Thank you

LAYOUT

Thank You

getting started with mariadb with docker

Software