Herd of Containers

Saâd DIFDatabase Engineer

Herd of Containers: PostgreSQL in containers at BlaBlaCar

pgDay Paris, Mar 15, 2018

Today’s agenda

BlaBlaCar Overview

PostgreSQL usage at BlaBlaCar

Switching to a new implementation

BlaBlaCar Overview

60 million members

Foundedin 2006

1 million tonnesless CO

2

In the past year

30 million mobileapp downloadsIphone and Android

15 milliontravellers

Currently in22 countriesFrance, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey.

Facts and Figures

Core Data Ecosystem

1 2 3

MySQL

Main DatabaseMariaDB 10.0+Galera Cluster

Cassandra

Column OrientedDistributed

Redis

In MemoryKey-ValueOptional durability

Core Data Ecosystem

4 5

ElasticSearch

JSON documentsFullText searchDistributed

PostgreSQL

ORDBMSExtensibilityStability

Why Containers ?

Resource allocationDeployment Speed

On premise

Skills already thereCost

Containers

Rkt

Why Rkt over Docker ?

CoreOS Container Linux

Linux DistribSimple & Secure

Only run containers

Fleet

OrchestrationBy default with CoreOS

Containers

GGN

Generate systemd units

Dgr

Build and configure App Container Images

Pods

Aggregate images in one shared environment

Containers

bare-metal servers

1 type of hardware

3 disk profiles

fleet cluster

CoreOS

fleet etcd“Distributed init system”

Hardware

Container Registry

ggn

dgr

Service Codebase

rkt PODs

build

run

store

host

create pgsql

monitoring

nerve

pgsql-main1

php

nginx

nerve

monitoring

synapse

front1

synapse

nerve

zookeeper Service Discovery

Containers

Get rid of DNS internallyAdapt to change

Why ?

1

Service Discovery

Key-Value storeReliable, Fast, Scalable

Why ? Zookeeper

21

Service Discovery

Go-NerveHealth ChecksEphemeral keysPresent on each pod

Why ? Zookeeper Report

2 31

Service Discovery

Go-SynapseWatch ZookeeperUpdate HAProxy configuration

Why ? Zookeeper Report Discover

2 3 41

Service Discovery

backend pod

client pod

Service Discovery

/database/node1

go-nerve does health checks and reports to zookeeper in

service keys

node1

/database

Applications hit their local haproxy to access backends

go-synapse watches zookeeper service keys and reloads haproxy if changes are

detected

HAProxy

go-nerve

Zookeeper

go-synapse

PostgreSQL usage at BlaBlaCar

Prerequisite

PostGIS

Third-party applications

Spatial

Confidence

Home Made tools

Usage

Travel company

Corridoring

Point to Point

PostGIS

RambouilletParis

Lyon

Le Creusot

3 685 1MRides passed by

Amiens last month

Number of

meeting points

50kRows reads per

minutes

Change!Streaming Replication

Manual Interventions Not friendly

Painful failover recovery

Operate

Target

Scale writes

Ease deployments

Maximum availability

Slaves

Failovers

Expandable resources

Possibilities

Postgres-XC (x2)

Postgres-XL

PgLogical

Bucardo

Slony

Londiste

Switching to a new implementation

BDR

Bi-Directional Replication

OpenSource project by 2ndQuadrant

Multi Master Asynchronous Replication

2 to 48 nodes

Optimal for Geo Distributed databases

BDR : The Confirmation

All nodes support reads and writes

No failovers

No other process / nodes needed

Partition tolerant

BDR : Caveats

Modified version of PostgreSQL 9.4BDR 2.0 with PostgreSQL 9.6 for 2ndQuadrant support customers

DDL lock

Replication lag

Conflicts

Some statement not supported yet

Statement not replicated

Check

InitCheck if node have

entries in the bdr_nodes table, if

yes : skip init

Implementation

Run

[~/build-tools/aci/aci-postgresql-bdr] $ tree.├── Jenkinsfile├── aci-manifest.yml├── attributes│ ├── base.yml│ └── postgresql.yml├── files│ └── tmp│ └── postgresql│ ├── environment│ ├── pg_ctl.conf│ ├── pg_ident.conf│ └── start.conf├── runlevels│ ├── build│ │ └── 00.install.sh│ └── build-late│ └── 00.clean.sh└── templates └── dgr └── runlevels └── prestart-late ├── 00.init-instance.sh.tmpl └── 01.init-database.sh.tmpl

Implementation (init)

1 If no “donor” attributes : Init as new group

2

3

1

When the node have “donor” attributes :

Retrieve user definition on donor (pg_dumpall -g)

Join BDR group

Create minimum objects if not present

2

1 Part local node on donor

Delete entries on donor(bdr_nodes and bdr_connections)

New fresh node Node already referenced but changed host or have lost his data

Pager DutyIncidents Manager

GrafanaBeautiful Visualizations

PrometheusSmart Monitoring

ExporterExpose metrics

Monitoring and Alerting

Monitoring

Key principles:

Usage

Saturation

BDR exporter specifics$ cat aci-prometheus-postgresql-exporter/templates/queries.tmpl.yaml

{{ if .use_bdr }}pg_replication_bdr_count: query: "select (select count(*) from bdr.bdr_nodes) as bdr_nodes, (select count(*) from bdr.bdr_connections) as bdr_connections;" metrics: - bdr_nodes: usage: "GAUGE" description: "Number of rows in the bdr_nodes table" - bdr_connections: usage: "GAUGE" description: "Number of rows in the bdr_connections table"{{ end }}

pg_replication_count: query: "select (select count(*) from pg_stat_replication) as stat_repli, (select count(*) from pg_replication_slots where active=true) as rep_slots;" metrics: - stat_repli: usage: "GAUGE" description: "Number of rows in the pg_stat_replication table" - rep_slots: usage: "GAUGE" description: "Number of rows in the pg_replication_slots table with the active status"

[...]

Template values for BDR specifics

Extend metrics to all PostgreSQL needs

Backup and Recovery

1 Retrieve dumpspg_dump

2 Alter structure dump

3 Load structure and data dump

Backup and Recovery

$ cat pod-mysql-backup/aci-backup/templates/opt/backup-main.tmpl.sh

function startbackup { begin_unixtime=$(date +%s) cat <<EOF | curl --data-binary @- http://prometheus-gw:9091/metrics/job/backup_{{.env}}/target/$node/service/$service/type/{{.backup.type}} # HELP backup_begin_unixtime # TYPE backup_begin_unixtime counter backup_begin_unixtime $begin_unixtimeEOF}

$ cat prometheus-rules/alert.postgresql.rules

# Alert: There is less replication active than bdr nodesALERT BackupsTooOld

IF time() - backup_end_unixtime{exported_service=~".*postgresql.*"} ) > ( 3600 * 24 )

LABELS { severity="warning", stack="backups", team="data_infrastructure"}

ANNOTATIONS { summary="Backup {{ $labels.type }} on {{ $labels.exported_service }}.{{ $labels.target }} is too old.", dashboard=" https://grafana.blabla.car/dashboard/db/db-backups ",}

Alerting

PromQL to find out unhealthy services

Labeling for routing to Slack & Pager Duty

Annotations with templating to have clear descriptions, URL to dashboards and ops runbooks

Feedback

Clearly satisfied with availability

Reactive community

Know what your needs are

Sanity checks

BDR 3.0 coming soon!

What’s next?