introducing apache mesos

38
BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Introducing Apache Mesos Matthias Furrer Bruno Fuss A modern cluster manager optimizing resource utilization

Upload: matthias-furrer

Post on 06-Apr-2017

47 views

Category:

Technology


0 download

TRANSCRIPT

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Introducing Apache Mesos

Matthias FurrerBruno Fuss

A modern cluster manager optimizing resource utilization

Agenda

TechEvent March 2017 - Introducing Apache Mesos2 03/2017

1. IntroductionArchitecture (components)DistributionsOther Offerings

2. FrameworksService Scheduling andManagement FrameworksBig Data Frameworks

3. HA ComponentsZookeeperMarathon

3. ConfigurationMesosMarathonKafka

4. Live DemoMesosKafka

5. SummaryUse Cases / Benefits

TechEvent March 2017 - Introducing Apache Mesos3 03/2017

Introduction

Introducing Apache MesosA software layer to aggregate servers into one processing unit

TechEvent March 2017 - Introducing Apache Mesos4 03/2017

Distributed Cluster Management Software

System Resources (Memory, Storage) abstraction from Hardware

Metascheduler („scheduler of schedulers“)

Distributed Systems Kernel / Distributed Datacenter OS

Supports Mesos and Docker Containers

Open Source

– Runs under Apache License 2.0

Cross Plattform

– Written in C++

– Runs on Linux/Posix systems, Experimental for Windows

Architecture - Features

TechEvent March 2017 - Introducing Apache Mesos5 03/2017

Scalability

– Scales up to over 50‘000 nodes

Resource isolation

– Linux cgroups or Docker containers

Efficiency

– Resource scheduling accross multiple frameworks/applications

High availability

– Uses Apache Zookeeper (distributed configuration and synchronization service)

Monitoring Interface

– Web UI

Components

TechEvent March 2017 - Introducing Apache Mesos6 03/2017

Master

– Mediating agent resources

Agents

– Manage resources on individual nodes and executestasks

Frameworks

Applications

– Systems running on mesos consisting of a scheduler and an

executor (e.g. Spark, Storm ,Cassandra)

Chronos

– Fault tolerant job scheduler, can handle dependencies - acts

as a cron for mesos

Marathon

– Framework for running long-running services – init.d for

mesos

Architecture – High Level Overview

TechEvent March 2017 - Introducing Apache Mesos7 03/2017

Zookeeper ensures high availability of masters

Master daemon manages agent (slave) daemons

Mesos frameworks run tasks on agents (slaves)

Master decides how many resources to offer to each framework according to a given organizational policy (e.g fair sharing or strict priority).

Framework running on top of Mesos consists of two components: a scheduler that registers with the master to be offered resources, and an executor process that is launched on agent nodes to run the framework’s taskSource: https://mesos.apache.org/documentation/latest/architecture

Architecture – Resource Offering

TechEvent March 2017 - Introducing Apache Mesos8 03/2017

1. Agent 1 reports to the master that it has 4 CPUs and 4 GB of memory free. The master then invokes the allocation policy module

2. The master sends a resource offer describing what is available on agent 1 to framework

3. The framework’s scheduler replies to the master with information about two tasks to run on the agent

4. Master sends the tasks to the agent, which allocates appropriate resources to the framework’s executor, which launches the two tasks. Because CPU/RAM Resources are still unallocated, the allocation module may now offer them to framework 2.

Source: https://mesos.apache.org/documentation/latest/architecture

Distributions– Mesosphere DC/OS

TechEvent March 2017 - Introducing Apache Mesos9 03/2017

Commercial version: Mesosphere Datacenter Operating System

Open Source version: DC/OS available - derived from Mesosphere’s Datacenter Operating System

Based on Mesos and Marathon

Supports Docker as well as native Mesos containers

Container orchestration capabilities for Docker and Mesos containers including automatic workload recovery, security, networking, service discovery

“Push-button installation” of complex distributed systems (including HDFS, Apache Spark, Apache Kafka, Apache Cassandra and more) with DC/OS Universe

Runs on bare-metal, virtual (vSphere or OpenStack) and cloud (AWS, Azure, GCE)

Other Offerings

TechEvent March 2017 - Introducing Apache Mesos10 03/2017

Kubernetes (Google / Cloud Native Computing Foundation CNCF)

Docker Swarm (Docker, Inc.)

Rkt fleet (CoreOS, Inc.)

Shipyard (Shipyard Project)

CloudSlang (Hewlett Packard Enterprise)

Ranger (Ranger Labs)

Azure Service Fabric (Microsoft)

TechEvent March 2017 - Introducing Apache Mesos11 03/2017

Frameworks

Service Scheduling and Management Frameworks

TechEvent March 2017 - Introducing Apache Mesos12 03/2017

Marathon

– Launch and monitor long-running applications. Controlling a high availability environment, checking the applications' health and restarting services if nodes go down

Chronos

– Distributed, fully fault-tolerant job scheduler. Can interact with systems (e.g. Hadoop) even if the Mesos worker machine does not have the system installed

Apache Aurora

– Manages long-running services, cron jobs, and ad-hoc jobs

Consul

– Service Discovery and orchestration

Big Data Frameworks

TechEvent March 2017 - Introducing Apache Mesos13 03/2017

Apache Hadoop

– Distributed Data Processing, MapReduce and Distributed File System (HDFS)

Apache Spark

– Execution engine for large-scale data processing. Batch processing, iterative processing, near real-time processing, and stream processing. Caching data in memory with Resilient Distributed Datasets (RDD)

Apache Storm

– Real-time distributed stream event-processing engine

Apache Cassandra

– Open source scalable NoSQL database. Provides query language Cassandra Query Language or CQL to communicate with database

Big Data Frameworks (II)

TechEvent March 2017 - Introducing Apache Mesos14 03/2017

Apache Kafka

– distributed publish-subscribe messaging system designed for speed, scalability, reliability, and durability

Apache Samza

– Distributed stream processing framework. Using Kafka for messaging and Hadoop YARN for resource management

Elasticsearch-Logstash-Kibana (ELK) stack

– End-to-End log analytics solution: Elasticsearch for search capabilities, Logstashas a log management software and Kibana as the visualization layer

Mesos API

– Develop custom frameworks – schedulers and executors

TechEvent March 2017 - Introducing Apache Mesos15 03/2017

High Availability

Components

Apache Zookeeper

TechEvent March 2017 - Introducing Apache Mesos16 03/2017

ZooKeeper

– is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Independent Apache Project

Characteristics

– Central Key/Value Store

– Containing additional Information like time stamps, versioning, …

– Simple

– Fast

– Designed for High Availability

Apache Zookeeper

TechEvent March 2017 - Introducing Apache Mesos17 03/2017

High Availability

– one Leader

– two or more Followers

– Quorum

Follower

Apache Zookeeper

TechEvent March 2017 - Introducing Apache Mesos18 03/2017

Data Model

– Hierarchical Namespace

– Z-Nodes (Zookeeper Nodes)

– Nodes identified by Path

• Ex.: /app1/p_1

Ephemeral Nodes

Leader Election

Conditional updates and watches

Atomic Transactions

Apache Zookeeper on Mesos

Zookeeper takes hand in everywhere Mesos needs to store and call up distributeddynamic information

– Service Locations

– Service States

– Client Notifications

– Leader election for Mesos Masters and Frameworks (e.g. Marathon, etc.)

03/2017 TechEvent March 2017 - Introducing Apache Mesos19

Marathon

Marathon is a Apache Mesos framework for container orchestration

– designed for long running and stateful Apps (e.g. Database, App. Server, etc.)

Marathon is a meta framework

– used for managing other Mesos Frameworks (Chronos, Storm, …)

Container Support

– Mesos Containers (default)

– Docker Constainers (must be configured)

03/2017 TechEvent March 2017 - Introducing Apache Mesos20

Marathon

Features

– High Availability

– Constraints

– Service Discovery & Load Balancing (Mesos-DNS, Marathon-LB)

– Health Checking

– Event Subscription

– User Interfaces for managing services (start, stop, check, …)

• Marathon UI (GUI)

• REST API

– Basic Security (Authorisation, SSL, …)

03/2017 TechEvent March 2017 - Introducing Apache Mesos21

Marathon Example

Mesos Master and Agents (Slaves), Zookeeper and Marathon started by OS on each server (initd, systemd)

Marathon starts frameworks on Mesos Agents (Chronos, JBoss, Jetty, Rails)

Chronos, a fault tolerant job scheduler acts itself as frameworkand starts jobs (e.g. dumps a database, send Emails)

03/2017 TechEvent March 2017 - Introducing Apache Mesos22

TechEvent March 2017 - Introducing Apache Mesos23 03/2017

Installation & Configuration

Installing and Configuring Apache Mesos

Installation

– http://mesos.apache.org/gettingstarted

– Different between CentOS 6 / 7

• Different Requirements (OS Packages, Libraries, …)

– configure / make / make install

– Zookeeper included

03/2017 TechEvent March 2017 - Introducing Apache Mesos24

Installing and Configuring Apache Mesos

Configuration

– http://mesos.apache.org/documentation/latest/configuration

– Provide parameters by config file or as startup parameter

– Required parameters (M: on Master, A: on Agent)

--zk="zk://mesos1:2181,mesos2:2181,mesos3:2181/mesos" (M,A)

--quorum=2 (M)

--work_dir=<workdir_path> (M,A)

--containerizers=mesos,docker (A)

03/2017 TechEvent March 2017 - Introducing Apache Mesos25

Installing and Configuring Marathon

Requirements

• Apache Mesos 1.1.1+

• Apache ZooKeeper

• JDK 1.8+

• Default JVM set to Java 8

Installation

– https://mesosphere.github.io/marathon/docs/

– Dowload/unpack Kit (tar)

03/2017 TechEvent March 2017 - Introducing Apache Mesos26

Installing and Configuring Marathon

Configuration

– Library: MESOS_NATIVE_JAVA_LIBRARY

• searches the common installation paths, /usr/lib and /usr/local/lib, for the Mesos native library. If the library lives elsewhere in your configuration, set the environment variable MESOS_NATIVE_JAVA_LIBRARY to its full path.

– Parameters: Marathon uses --master to find the Mesos masters, and --zk to find ZooKeepers for storing state

--master="zk://mesos1:2181,mesos2:2181,mesos3:2181/mesos"

--zk="zk://mesos1:2181,mesos2:2181,mesos3:2181/marathon"

03/2017 TechEvent March 2017 - Introducing Apache Mesos27

Installing and Configuring Apache Kafka for Mesos

TechEvent March 2017 - Introducing Apache Mesos28 03/2017

Installation

• According to https://github.com/mesos/kafka

• Make sure to retrieve the Kafka download from https://archive.apache.org/dist/kafka from a version containing a fitting Scala library (Version indicated on the Mesos page may not becorrect) to avoid java.lang.NoSuchMethodError

Configuration

• Set Mesos connection info in kafka-mesos.properties

• Start Scheduler on ONE Mesos agent (Marathon can be used to manage)

• Add Broker(s)

• Start Broker(s)

TechEvent March 2017 - Introducing Apache Mesos29 03/2017

Demo

mesos1 mesos2

mesos3 mesos4 mesosn

mesosn-1

TVD Mesos Platform / Server

extendable• additional Servers (on-premise)• Cloud• …

03/2017 TechEvent March 2017 - Introducing Apache Mesos30

BF2

Slide 30

BF2 Bruno Fuss, 3/8/2017

mesos1 mesos2

mesos3 mesos4 mesosn

mesosn-1

MesosMaster

MesosMaster

MesosAgent

MesosAgent

TVD Mesos Platform / Mesos Components

MesosAgent

MesosAgent

HA concept Master: active/passive

03/2017 TechEvent March 2017 - Introducing Apache Mesos31

mesos1 mesos2

mesos3 mesos4

MesosMaster

MesosMaster

MesosAgent

MesosAgent

Zookeeper

Zookeeper

Zookeeper

TVD Mesos Platform / Zookeeper Components

HA concept: active/active1 leader2-n followers

Zookeeper quorum= 2

mesosn

mesosn-1

MesosAgent

MesosAgent

03/2017 TechEvent March 2017 - Introducing Apache Mesos32

BF3

Slide 32

BF3 Bruno Fuss, 3/8/2017

TechEvent March 2017 - Introducing Apache Mesos33 03/2017

Summary

Use Cases / Why use Mesos

TechEvent March 2017 - Introducing Apache Mesos34 03/2017

Enhance resilience against failure

Need for application-specific scaling

Increase resource utilization

Wide range of workloads must be supported – e.g. Batch (Hadoop), Analysis (Spark), real-time (Storm), data storage (HDFS, Cassandra)

Open source with vivid community

Developer API – minimal considerations about infrastructure details

Matthias FurrerPrincipal Consultant, Application Integration & [email protected]

03/2017 TechEvent March 2017 - Introducing Apache Mesos35

Bruno FussSenior Consultant, Infrastructure Managed [email protected]

Session Feedback – now

TechEvent March 2017 - Introducing Apache Mesos36 03/2017

Please use the Trivadis Events mobile app to give feedback on each session

Use "My schedule" if you have registered for a session

Otherwise use "Agenda" and the search function

If the mobile app does not work (or if you have a Windows smartphone), use your smartphone browser

– URL: http://trivadis.quickmobileplatform.eu/

– User name: <your_loginname> (such as “svv”)

– Password: sent by e-mail...