Kapil Arya & Niklas Nielsen
Mesos Gets Pluggable Introducing Mesos Modules
© 2015 Mesosphere, Inc. 3
Mesos Modules & HooksModules & HooksArwwwww
© 2015 Mesosphere, Inc. 4
How and why modules was introduced in Mesos
Our humble thoughts on how modules and extensibility in Mesos can evolve in the future
How Mesos Modules work and give you concrete examples of modules in action
© 2015 Mesosphere, Inc.
Modules
5
© 2015 Mesosphere, Inc.
Different organizationsDifferent needs
6
● Hardly anyone run clusters the same way○ Different scales○ Different hardware○ Different workloads○ Different external tooling○ Different security needs
One cluster with turbo chargers please
© 2015 Mesosphere, Inc.
• Mesos was built with this in mind!• The subsystems are lightweight insight
and control over HTTP
• Excellent for tooling around
• Different subsystems can be enabled and configured in a modular way
• Most notable: Isolation mechanisms
Good news!
7
© 2015 Mesosphere, Inc.
New “extensions” to subsystems like isolators had to be upstreamed
But…
● Mesos can be made even more customizable and extendable
● Not all organizations can share their work
● Support proprietary and experimental integrations
● Not create bespoke forks of Mesos
However...
8
© 2015 Mesosphere, Inc.
• Tie into and control task launch
• Dynamically setup execution environments
• Pass signatures through Mesos
• All of this, transparently to the framework and user
We needed it to support bespoke security subsystems
9
© 2015 Mesosphere, Inc.
• Be able to extend and replace any component in Mesos• Allocator algorithms
• Authentication mechanisms
• Advanced scheduling features like oversubscription
• Anything!
The general thought of Modules was bigger
10
Imagine ifI could write my own?
© 2015 Mesosphere, Inc.
Modules are old news
Many large software systems support libraries to
• Extend behavior• Isolate and abstract complexity• Make this a configuration rather than a
build exercise
For example
• Browsers (Firefox)• Server software (Apache Webserver)• Linux kernel
11
Wish I had modules already
© 2015 Mesosphere, Inc.
What is a module anyway?
Module, plugin, extension, library …
Adds or replace a full component
For example:
• An isolator (works together with existing ones) in the agent
• The allocator and authenticators in the master
12
© 2015 Mesosphere, Inc.
And how about hooks?
More often than not, you don’t want to replace a full component
Just want to tie into events and their context
For example:
• Launch task requests at the master
• Launch task requests at the agent
• Exit and cleanup events
13
Psst - I just launched a task
© 2015 Mesosphere, Inc.
And who is using it?
Powering new exciting features and integrations!
• Oversubscription modules• Static (fixed) estimator
• Dynamic estimator and QoS Controller, project Serenity
• Networking integration with project Calico
14
© 2015 Mesosphere, Inc. 15
Module Mechanics
© 2015 Mesosphere, Inc.
A demo!
16
● A hook module that tags TaskStatus messages
© 2015 Mesosphere, Inc.
Components
17
Isolator InterfaceIsolator Module
Hook Module H1
Hook Module H2
Hook Interface
Mesos Master/Agent
Module library
ModuleManager
Module spec JSON
Initialization
Initialize subsystems
use module objects
Module libraryinitialize modules
get module object
readspec
call hooks
© 2015 Mesosphere, Inc.
Initialization
18
● First phase: ○ load module libraries○ compatibility checks, etc.○ libprocess not available
● Second phase○ initialize a specific module○ module-specific parameters○ libprocess available
© 2015 Mesosphere, Inc.
class TestHook : public Hook{public: Result<Labels> slaveTaskStatusLabelDecorator( const FrameworkID& frameworkId, const TaskStatus& status) { Labels labels; if (status.state() == TASK_RUNNING) { Label* newLabel = labels.add_labels(); newLabel->set_key("whereami"); newLabel->set_value("mesoscon"); } return labels; }};
A Hook Module
19
// Create and return an object or TestHook type. static Hook* createHook(const Parameters& parameters){ // Any initialization checks go here.
return new TestHook();}
// Declares a Hook module named ‘org_apache_mesos_TestHook'mesos::modules::Module<Hook> org_apache_mesos_TestHook( MESOS_MODULE_API_VERSION, MESOS_VERSION, "Apache Mesos", "[email protected]", "Test Hook module.", NULL, createHook);
© 2015 Mesosphere, Inc.
{ "libraries": [ { "file": "/path/to/libmodule.so", "modules": [
{ "name": "org_apache_mesos_TestHookModule", "parameters": [ { "key": "agent_addr", "value": "agent.host.domain" }, { "key": "...", "value": "..." } ] } ] } ]}
Specifying Modules to Master/Agent
20
© 2015 Mesosphere, Inc.
● Build without building Mesos○ Just have Mesos installed
● Modules compile into a shared libraries○ Multiple modules per library
● Specify modules on command line:mesos-agent.sh <master-parameters> --modules=file:///path/to/modules.json --isolation=”my_isolator” --hooks=”my_hook”
Using Modules
21
© 2015 Mesosphere, Inc.
● Add/replace a full component● Implement the interface● Asynchronous (actor model)
● Existing modularized interfaces:○ Allocator○ Authentication○ Authorizer○ Isolator○ QoSController○ ResourceEstimator
Replacement Modules
22
© 2015 Mesosphere, Inc.
● Listen/Intercept interesting calls● Occasionally modify the behavior
○ Trigger initialization/cleanup● Allows us to “tag” certain tasks, statuses, etc.
● Two broad categories○ Task launch sequence○ Status updates
Hook Modules
23
© 2015 Mesosphere, Inc.
● Co-exists with the parent process ○ separate thread of execution
● Create Master/Agent http “listen” endpoints● No callbacks
Anonymous Modules
24
One module to rule them all!
© 2015 Mesosphere, Inc.
● Do not block○ Hooks are synchronous○ Use libprocess/pthreads
● Exit semantics○ Avoid assertions
Writing Modules
25
© 2015 Mesosphere, Inc.
● Logs○ stdout/stderr
● Run debug module with non-debug Master/Agent○ gdb
Debugging
26
What crashed the Master?
© 2015 Mesosphere, Inc.
● Dependency on other modules● Compatibility within set of modules● Upgrade path
○ rebuild modules when updating Mesos
Dependency and Compatibility
27
© 2015 Mesosphere, Inc. 28
Future Work
© 2015 Mesosphere, Inc.
● Safeguard against unsafe modules○ Limit data exposure○ Execute modules in a separate process
● Module certification● ACL’s● Runtime functionality checks
○ whitelist services○ can it add routes or not
Better Safety and Security
29
© 2015 Mesosphere, Inc.
● More module interfaces● Load/Unload a module without rebooting Master/Agent● Upgrade path● Express dependability on other modules● Inter-module communication● Non-C++ modules
Future Work
30
Thanks for [email protected] [email protected]
❏ Documentation: http://mesos.apache.org/documentation/latest/modules/
❏ Modules repo: https://github.com/mesos/modules
❏ Mailing list: [email protected]