the new docker networking put into action to spin up a...

ssMulti-host containerised HPC cluster The new Docker networking put into action to spin up a SLURM cluster

The Bits and Pieces…

Ubuntu:14.04 Ubuntu:15.10 RHEL7.2

TinyCoreLinux:6.4.1

Docker

3

SERVER

HOSTKERNEL

HYPERVISOR

KERNEL

SERVICE

Userland(OS)

KERNEL KERNEL

Userland(OS)Userland(OS) Userland(OS)

SERVICE SERVICE

SERVER

HOSTKERNEL

SERVICE

Userland(OS)

Userland(OS)Userland(OS) Userland(OS)

SERVICE SERVICE

Traditional Virtualisation Containerisation

IB

IB

• Containers do not spin up their own kernel All containers share the same host kernel

they are separated via Kernel Namespaces

constrained using CGroups

Docker

4

• Docker-Engine as a Container Runtime creates/starts/stops/manipulates/removes containers - all through RESTful API

handles Namespaces, CGroups, IP connectivity, bind-mounts, etc.

SERVEReth0

dock

er0

container1

container2

SERVER0 SERVER1 SERVER<n>

Docker Networking

5

• Docker Networking spans networks across engines KV-store to synchronise (Zookeeper, etcd, Consul)

VXLAN to pass messages along (macvlan might take over)

Consul

Docker-Engine

Consul Consul

Docker-Engine Docker-Engine

Consul DC

global

container0 container1 containerN

SERVER0 SERVER1 SERVER<n>

Docker Swarm

6

• Docker Swarm proxies docker-engines serves an API endpoint in front of multiple docker-engines

does (simple) placement decisions.

Docker-Engine Docker-Engine Docker-Engine

swarm-client swarm-client swarm-client

swarm-master

:2376 :2376 :2376

:2375

Docker Swarm [cont]

7

query docker-enginequery docker-swarm

Introduce new Technologies

Introducing new Tech

9credit: Russian Oil Claim


10

Self-perception when introducing new tech…

credit: TF2 - Meet the Pyro


11

… not always the same as the perception of others.

credit: TF2 - Meet the Pyro

Docker Buzzword Chaos!

Distributions

Solutions

Auto-ScalingOn-Premise & OverSpill

Orchestration

self-healing

12

production-readyenterprise-grade

• You say Docker - perception is ‘basically VMware’ VMs were easy to shoehorn in legacy workflow, containers might break it

spans environments: laptop, dev-cluster, staging, prod

• Not everyone is a unicorn, I do NOT want special distributions

useful for elasticity (AWS) and green-flied deployment, not so much for a on-premise datacenter w/ legacy in it.

want to leverage existing <stuff> security (ssh infrastructure), user authentication installation workflow, monitoring, logging

want to keep up with upstream docker ecosystem networking, volumes features of docker-engine, -compose, -swarm

13

Pitfalls[incomplete list]

Reduce to the max!

• Leverage existing install/configuration workflow Kickstart + ansible-run

• Don’t focus on corner cases postpone snowflake-container (e.g. needs multi-tenant IB usage)

User namespace: we get there (eventually)

• HPC environments assumptions single-tenant

focus on performance

15

Put a Docker on it!

The Setup

• Hardware (courtesy of ) 8x Sun Fire x2250, 2x 4core XEON, 32GB, Mellanox ConnectX-2)

• Software Base installation

CentOS 7.2 base installation (updated from 7-alpha)

Ansible

consul, sensu

docker v1.9.1, docker-compose

docker SWARM

17

Testbed

18

Docker Networking

• Synchronised by Consul

venus001

venus002

venus008Consul

Consul DC

Consul

Consul

Docker-Engine

Docker-Engine

Docker-Engine

19

Docker SWARM

• Docker SWARM Synchronised by Consul KV-store

venus001

venus002

venus008Consul

Consul DC

Consul

Consul

Docker-Engine

Docker-Engine

Docker-Engine

swarm

swarm

SWARM

swarm master

venus001

venus002

venus008

20

SLURM Cluster

Consul

Consul DC

Consul

Consul

SLURM

• SLURM within SWARM slurmd within app-container

pre-stage multiple container

spawn at job-start (pre-hook)slurmctld slurmd

slurmd

slurmd

Docker-Engine

Docker-Engine

Docker-Engine

swarm

swarm

hpcg

hpcg

SWARM

hpcg

OpenFOAM

OpenFOAM

OpenFOAM

swarm master

slurmd

slurmd

slurmd

21

SLURM Cluster [cont]

venus001

venus002

venus008

22

Samza Cluster

Consul

Consul DC

Consul

Consul

• Distributed Samza Zookeeper and Kafka cluster

Samza instances to run jobsDocker-Engine

Docker-Engine

Docker-Engine

swarm

swarm

SWARM

swarm masterzookeeper

zookeeper

zookeeper

kafka

kafka

kafka

samza

samza

samza

Outta time!

24

SENSU

25

Metrics

• Using vanilla docker tech on-top of any distribution keep up with the ecosystem and prevent vendor/ecosystem lock-in

• 80/20 rule have caveats on the radar but don’t bother too much

everything is so fast moving - it’s hard to predict

• Don’t scare away stakeholders KISS

reuse workflow and infrastructure

solution and not problem driven

26

Recap aka. IMHO

Q&Ahttps://github.com/qnib/hpcac-cluster2016

http://qnib.orgeGalea Conference (Pisa)

20/21 April 2016

https://github.com/qnib/hpcac-cluster2016

http://qnib.org

the new docker networking put into action to spin up a...

Documents