mesosphere and the enterprise: run your … and the enterprise: run your applications on ... source...

Mesosphere and the Enterprise: Run Your Applications on Apache Mesos

Steve Wong Open Source Engineer

@cantbewong {code} by Dell EMC

© Copyright 2017 Dell Inc. 2

Open source at Dell EMC

– Contribute to meaningful OSS projects – Create new thought leading OSS applications – Drive awareness of OSS opportunities with Dell EMC

product teams – Participate in relevant community engagement projects – Act in the interest of building a community

{code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure.

Platinum Sponsor


Why Containers? They make managing applications easier • Speed up software innovation

• Deployments are simple

• Creates portable, shareable and reusable units of code and requirements

Code Code and requirements Container


What is Mesosphere DC/OS?

From a resource perspective:

It’s an infrastructure abstraction layer: – Aggregates systems into a single resource pool.

From an application perspective:

It simplifies management of distributed technologies – A scheduler (= dispatcher). – Dispatches workloads which consume the pooled resources.


History

Started at UC Berkeley in 2009, became an Apache project in 2013 after nearly two years in incubation


Project Objective

Deliver Google’s Borg to the rest or world See Wired: How Twitter rebuilt Google’s Borg March 2013


Two levels of Application Support

Generic

Run anything that can be launched in a standard Linux shell + Docker images

Framework

Allows you to implement a sub-scheduler – Extends Mesos to provide custom workload placement and resource

allocation › For example, reserve 10% of the cluster’s resources to Hadoop jobs,

framework manages prioritized dispatch


With Marathon & Metronome, all types of jobs are managed


Marathon Marathon acts as a service dispatcher You provide instructions and Marathon does its best to keep the service up on the cluster, providing automated scaling and high availability restarts


Metronome – a “chron” for the datacenter Specify:

•The amount of CPU your job will consume.

•The amount of memory your job will consume.

•The disk space your job will consume.

•The schedule for your job, in cron format. You can also set the time zone and starting deadline.


Overview of Enterprise DC/OS


Where can you run Mesos?


Any limitations on what I can use for cluster nodes?

• They do not have to be identical

• Processor model, speed or memory size variations are OK

• They can concurrently run non-Mesos services

• don’t have to give Mesos control over 100% of node’s CPU, memory or disk.

• Elastic scaling is supported

• nodes can come and go


Production Users & Customers

Government Agencies


Efficiency and Utilization

Typical Datacenter siloed, over-provisioned servers,

low utilization

DCOS Datacenter automated schedulers, workload multiplexing onto the

same machines

DCOS Multiplexing 30-40% utilization, up to 96% at some customers

4X Industry Average 12-15% utilization


How big can this really go? Performance Characteristics


Performance metric at scale

50,000 containers deployed live on stage 1 minute 12 seconds


Architecture


How {code} by Dell EMC fits with Mesos


Back in 2012 container aps must be stateless?


The picture today


External Volume Mount Solution


External Volume Mount Addresses

Availability Risk

•Migrate container to another host – your storage is gone

•Host goes down – your service goes down

Scale Limitation

•Need more storage than the host has? Sorry…


Components

REX-Ray rexray.codedellemc.com


REX-Ray

REX-Ray

The leading container storage orchestration engine enabling persistence for cloud native workloads

rexray.codedellemc.com

Cloud Native Interoperability

Open Source

Enterprise Ready – High Availability – CLI Intuitiveness – Effortless Deployment – Architectural Choices

Multi-Platform Storage Management – Storage agnostic (block/file/object)


Mesos Container Framework Integration

• Marathon from Mesosphere

• Apache Aurora (originally used by Twitter)

• ElasticSeach


End-to-End Container Persistence Leadership for Mesos

What does it do

Enables external storage to be created / mounted /

unmounted with each agent task

Abstracts Docker Volume Drivers for Mesos Storage orchestration engine

Cool factor

• First Mesos Agent external storage module

• Merged into Mesos 1.0 (2016)

• Allows the use of any Docker volume driver with Mesos!

• Vendor agnostic • DC/OS 1.7+ framework

(2016)

mesos-module-dvdi Docker Volume Driver

Client CLI DVDCLI

REX-Ray


ScaleIO Framework for Mesos

• A Software-based Storage Framework using ScaleIO to turn commidy hardware and direct attached storage (DAS) into a globally accessible and scalable storage platform that can be deployed anywhere

• Framework installs and configures ScaleIO on all Mesos Agent (compute) nodes. As more Agents are added to the cluster, ScaleIO is installed and adds storage resources to the cluster.

• Deploy and Configure ScaleIO without knowing operational details

• Centralized Monitoring of the storage platform

• Persistent storage native to the container scheduling platform using REX-Ray

• https://github.com/codedellemc/scaleio-framework

https://github.com/codedellemc/scaleio-framework


{code} + Mesos

Mesos Module DVDI is officially merged into Mesos 1.0 and allows

any Docker Volume Driver to be used with Mesos Frameworks

REX-Ray is installed on every host and performs storage

orchestration

MARATHON ScaleIO-Framework

installs ScaleIO software on all Mesos Agents

SUPPORTED STORAGE PLATFORMS WITH REX-RAY


Demo!

DC/OS / Marathon Mesos Database Deploy


codedellemc.com

community.codedellemc.com

@codeDellEMC

blog.codedellemc.com

{code} by Dell EMC is a group of passionate open source engineers and

advocates working to build a community around software-based infrastructure.

rexray.codedellemc.com

github.com/codedellemc/labs

Steve Wong @cantbewong github.com/cantbewong

HOL01 Use REX-Ray & ScaleIO w/ Docker, Mesos and Kubernetes

mesosphere and the enterprise: run your … and the enterprise: run your applications on ... source...

Documents