Comment nous avons conteneurisé une startup ?

#devopsday

Jonathan Martin @john_iw2

Stéphane Cottin @vixns

#architecture #hosting #opsdev #libre #r&d #techtrends #IoT #artprojects

#startup #newyork #marseille #r&d #frenchtech #ecommerce #bpi #shake #network #bigdata

#dockerlover #opensourcelover #hashtaglover

The first global cross-selling network between eMerchants

“ “

Yuzu By Example

At the beginning

months to do a MVP 4

No time to reinvent the wheel

Symfony - Java - MongoDB

PRODUCTION

Web BigData

-  2 physical servers -  Multiple VMs -  Chef & Chef server

-  2 physical servers -  Stack Cloudera (Hadoop/Spark/...)

Six months later...

A lot of technos joined Yuzu during the MVP

Welcome to nodejs, scala, elasticsearch, hbase, redis, kafka, couchbase,...

They lived happily ever after

It doesn’t work in production… … that’s works fine in my local env

“ “

This Lib isn’t in the right version... It’s slow… why??

Where are my logs...

“ “

Time to look around...

Be kind, rewind Spring cleaning Microservices / 12 factors Resources isolation Improved security Continuous deployment Start-up compliant workflow Orchestration & supervision

AND WINNERS ARE ... THE

Docker + Mesos + Marathon + Consul

{"container":{"type":"DOCKER","docker":{"image":"redis:2.6.17","network":"BRIDGE","portMappings":[{"containerPort":6379,"hostPort":0,"protocol":"tcp"}]}},"id":"redis","instances":1,"cpus":0.5,"mem":1024,"healthChecks":[{"protocol":"TCP","portIndex":0}]}

MARATHON wants JSON

Dynamic Configuration Docker

Registrator

Consul

Consul-template

Logs, metrics, alerting...

Collect everything collectd / fluentd / rsyslog

Convert to async messages kafka

Filter and keep valuable data riemann / elasticsearch / influxdb

Generate realtime alerts riemann

Addictive dashboards kibana / grafana / riemann-dash

CHRONOS CRON for mesos

Our workflow

Git flow

Jenkins DinD

Private Registry Docker

Doploy

Marathon & Chronos

Boot2Docker + Parallels

Don’t / Do

use tag "latest" on production

use explicit version tags for your docker images

DON’T

use data containers for valuable data

storage layer may crash / change easy to forget when cleaning unused

containers / images

DON’T

use shared storage (glusterfs, nfs, ...) for your codebase, working dirs, config files or sessions

sloooooooooow generate scary lock errors and timeouts

fscache crash full systems easily

DON’T

tag your custom images with the VCS commit hash

●  makes your workflow better (same version tag in VCS and images)

●  prevent useless image rebuilds ( speedup deploys ) ●  easy way to know exactly which code version is running,

even if the tag was changed / deleted in the VCS

DO

avoid shared filesystems

●  put all versioned data in containers ●  use external object storage for user files (S3, ceph, swift ,...) ●  use database / memcached / couchbase for sessions ●  use templates to generate local config files (consul-template ,

confd)

DO

Use docker independant storage for critical data

●  lvm is your friend (unless you use and saturate thin volume metadata, use thin volumes with care)

●  use storage plugins with docker >= 1.9.0 (convoy ?) ●  redundant backups saves lives

DO

Mistaking is

learning

-  we tried to put the chicken in the egg

-  generate config files to shared storage

-  dns for discovery

-  use mongo replicaset without sharding

-  ...

Chronos Collectd Consul Consul-template Couchbase Debian Docker Doploy ElasticSearch Fluentd Gearman Go Grafana Haproxy HBase InfluxDB Java Jenkins

Kafka Kibana Marathon Mesos MongoDB Nodejs OpenVpn PhalconPHP Redis Registrator Registry

Riemann Route53 S3 Scala Sentry Spark Statsd Symfony Zookeeper ...

Switch from registrator to mesos-consul

Remove the lasts SPOFs

Move user files to S3

Use the docker storage/network plugins

XDCC

Improve logs & metrics filtering and alerting

2015 Other mesos/docker stories

IPROFS A large scale php/drupal worldwide social app for “Institut français”

ARTE.TV We are migrating all their vod and svod services to mesos/docker (java apps)

VIXNS A mesos/docker cluster is collecting all logs and metrics from hundreds of servers

Conclusion

Questions?

@john_iw2 @vixns