Mesos Gets Pluggable - Introducing Mesos Modules

Download Mesos Gets Pluggable - Introducing Mesos Modules

Post on 20-Feb-2017

989 views

Category:

Technology

2 download

Embed Size (px)

TRANSCRIPT

  • Kapil Arya & Niklas Nielsen

    Mesos Gets Pluggable Introducing Mesos Modules

  • 2015 Mesosphere, Inc. 2

    Niklas Nielsenniklas@mesosphere.io

    Kapil Aryakapil@mesosphere.io

  • 2015 Mesosphere, Inc. 3

    Mesos Modules & HooksModules & HooksArwwwww

  • 2015 Mesosphere, Inc. 4

    How and why modules was introduced in Mesos

    Our humble thoughts on how modules and extensibility in Mesos can evolve in the future

    How Mesos Modules work and give you concrete examples of modules in action

  • 2015 Mesosphere, Inc.

    Modules

    5

  • 2015 Mesosphere, Inc.

    Different organizationsDifferent needs

    6

    Hardly anyone run clusters the same way Different scales Different hardware Different workloads Different external tooling Different security needs

    One cluster with turbo chargers please

  • 2015 Mesosphere, Inc.

    Mesos was built with this in mind! The subsystems are lightweight insight

    and control over HTTP

    Excellent for tooling around

    Different subsystems can be enabled and configured in a modular way

    Most notable: Isolation mechanisms

    Good news!

    7

  • 2015 Mesosphere, Inc.

    New extensions to subsystems like isolators had to be upstreamed

    But

    Mesos can be made even more customizable and extendable

    Not all organizations can share their work

    Support proprietary and experimental integrations

    Not create bespoke forks of Mesos

    However...

    8

  • 2015 Mesosphere, Inc.

    Tie into and control task launch

    Dynamically setup execution environments

    Pass signatures through Mesos

    All of this, transparently to the framework and user

    We needed it to support bespoke security subsystems

    9

  • 2015 Mesosphere, Inc.

    Be able to extend and replace any component in Mesos Allocator algorithms

    Authentication mechanisms

    Advanced scheduling features like oversubscription

    Anything!

    The general thought of Modules was bigger

    10

    Imagine ifI could write my own?

  • 2015 Mesosphere, Inc.

    Modules are old news

    Many large software systems support libraries to

    Extend behavior Isolate and abstract complexity Make this a configuration rather than a

    build exercise

    For example

    Browsers (Firefox) Server software (Apache Webserver) Linux kernel

    11

    Wish I had modules already

  • 2015 Mesosphere, Inc.

    What is a module anyway?

    Module, plugin, extension, library

    Adds or replace a full component

    For example:

    An isolator (works together with existing ones) in the agent

    The allocator and authenticators in the master

    12

  • 2015 Mesosphere, Inc.

    And how about hooks?

    More often than not, you dont want to replace a full component

    Just want to tie into events and their context

    For example:

    Launch task requests at the master

    Launch task requests at the agent

    Exit and cleanup events

    13

    Psst - I just launched a task

  • 2015 Mesosphere, Inc.

    And who is using it?

    Powering new exciting features and integrations!

    Oversubscription modules Static (fixed) estimator

    Dynamic estimator and QoS Controller, project Serenity

    Networking integration with project Calico

    14

  • 2015 Mesosphere, Inc. 15

    Module Mechanics

  • 2015 Mesosphere, Inc.

    A demo!

    16

    A hook module that tags TaskStatus messages

  • 2015 Mesosphere, Inc.

    Components

    17

    Isolator InterfaceIsolator Module

    Hook Module H1

    Hook Module H2

    Hook Interface

    Mesos Master/Agent

    Module library

    ModuleManager

    Module spec JSON

    Initialization

    Initialize subsystems

    use module objects

    Module libraryinitialize modules

    get module object

    readspec

    call hooks

  • 2015 Mesosphere, Inc.

    Initialization

    18

    First phase: load module libraries compatibility checks, etc. libprocess not available

    Second phase initialize a specific module module-specific parameters libprocess available

  • 2015 Mesosphere, Inc.

    class TestHook : public Hook{public: Result slaveTaskStatusLabelDecorator( const FrameworkID& frameworkId, const TaskStatus& status) { Labels labels; if (status.state() == TASK_RUNNING) { Label* newLabel = labels.add_labels(); newLabel->set_key("whereami"); newLabel->set_value("mesoscon"); } return labels; }};

    A Hook Module

    19

    // Create and return an object or TestHook type. static Hook* createHook(const Parameters& parameters){ // Any initialization checks go here.

    return new TestHook();}

    // Declares a Hook module named org_apache_mesos_TestHook'mesos::modules::Module org_apache_mesos_TestHook( MESOS_MODULE_API_VERSION, MESOS_VERSION, "Apache Mesos", "modules@mesos.apache.org", "Test Hook module.", NULL, createHook);

  • 2015 Mesosphere, Inc.

    { "libraries": [ { "file": "/path/to/libmodule.so", "modules": [

    { "name": "org_apache_mesos_TestHookModule", "parameters": [ { "key": "agent_addr", "value": "agent.host.domain" }, { "key": "...", "value": "..." } ] } ] } ]}

    Specifying Modules to Master/Agent

    20

  • 2015 Mesosphere, Inc.

    Build without building Mesos Just have Mesos installed

    Modules compile into a shared libraries Multiple modules per library

    Specify modules on command line:mesos-agent.sh --modules=file:///path/to/modules.json --isolation=my_isolator --hooks=my_hook

    Using Modules

    21

  • 2015 Mesosphere, Inc.

    Add/replace a full component Implement the interface Asynchronous (actor model)

    Existing modularized interfaces: Allocator Authentication Authorizer Isolator QoSController ResourceEstimator

    Replacement Modules

    22

  • 2015 Mesosphere, Inc.

    Listen/Intercept interesting calls Occasionally modify the behavior

    Trigger initialization/cleanup Allows us to tag certain tasks, statuses, etc.

    Two broad categories Task launch sequence Status updates

    Hook Modules

    23

  • 2015 Mesosphere, Inc.

    Co-exists with the parent process separate thread of execution

    Create Master/Agent http listen endpoints No callbacks

    Anonymous Modules

    24

    One module to rule them all!

  • 2015 Mesosphere, Inc.

    Do not block Hooks are synchronous Use libprocess/pthreads

    Exit semantics Avoid assertions

    Writing Modules

    25

  • 2015 Mesosphere, Inc.

    Logs stdout/stderr

    Run debug module with non-debug Master/Agent gdb

    Debugging

    26

    What crashed the Master?

  • 2015 Mesosphere, Inc.

    Dependency on other modules Compatibility within set of modules Upgrade path

    rebuild modules when updating Mesos

    Dependency and Compatibility

    27

  • 2015 Mesosphere, Inc. 28

    Future Work

  • 2015 Mesosphere, Inc.

    Safeguard against unsafe modules Limit data exposure Execute modules in a separate process

    Module certification ACLs Runtime functionality checks

    whitelist services can it add routes or not

    Better Safety and Security

    29

  • 2015 Mesosphere, Inc.

    More module interfaces Load/Unload a module without rebooting Master/Agent Upgrade path Express dependability on other modules Inter-module communication Non-C++ modules

    Future Work

    30

  • Thanks for listening!kapil@mesosphere.io niklas@mesosphere.io

    Documentation: http://mesos.apache.org/documentation/latest/modules/

    Modules repo: https://github.com/mesos/modules

    Mailing list: modules@mesos.apache.org

    mailto:kapil@mesosphere.iomailto:kapil@mesosphere.iomailto:kapil@mesosphere.iomailto:niklas@mesosphere.iomailto:kapil@mesosphere.io