mesos sys adminday
TRANSCRIPT
![Page 1: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/1.jpg)
Mesos for dummiesGiving some sense to all this crap
Javier CortejosoUnai ArríenCredits: Mario Castro
![Page 2: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/2.jpg)
Plan
What is Mesos?
How can I use Mesos?
![Page 3: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/3.jpg)
What is mesos?
![Page 4: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/4.jpg)
A long time ago...
Are you done with the
machine? I need to
load my cards
Lol no; maybe
tomorrow
![Page 5: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/5.jpg)
In 1957...
Oh man! Let’s all
share the computer,
AT THE SAME TIME!
John McCarthy
![Page 6: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/6.jpg)
In 2005...
Are you done with the
Hadoop cluster? I
need to run my
analytics jobsLol no; maybe
tomorrow
![Page 7: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/7.jpg)
In 2010...
Oh man! Let’s all
share the cluster, AT
THE SAME TIME!
Ben Hindman
![Page 8: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/8.jpg)
Good ideas today mirror good ideas of yesterday
So… the origin highlights:
● Project motivated from Google projects
Borg (later Omega and Kubernetes)
○ Borg, Omega, and Kubernetes
● In 2008, Berkeley Amplab started the
Nexus project (later Mesos)
● First version published in 2009
● Used by companies: Twitter, eBay,
Apple, Netflix, Airbnb, …
![Page 9: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/9.jpg)
Mesos
Orchestration platform to share a computer cluster between many
applications and to use it as a single machine managing CPU, memory and
other resources.
No more “apps” in fixed machines.
Divided in Master nodes and Agent nodes (formerly known as Slaves)
![Page 10: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/10.jpg)
Mesos Masters and Agents
Master:
Manages cluster resources.
Has the UI.
Knows everything about the cluster (Maintains some state).
One active master and 0+ passive master(s) (elected using ZK) for HA
Agents:
Launch and manage containers (LXC and Docker containers).
Provide UI access to slave specific data.
![Page 11: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/11.jpg)
Mesos Architecture
![Page 12: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/12.jpg)
Mesos Frameworks
Each framework has a scheduler that is a process that communicates with
Mesos Master to accept resources from the Master offers…
...and 0+ executor(s), which are launched by the scheduler and it’s
considered a worker. The scheduler sends tasks to the executors that will
work on them and provide status updates to the scheduler.
![Page 13: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/13.jpg)
Mesos model
Resources are allocated via
resource offers
A resource offer represents a
snapshot of available resources
that a scheduler can use to run
tasks
![Page 14: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/14.jpg)
Mesos model
Scheduler uses the offers
to decide what tasks to
run
“Two-level scheduling”
![Page 15: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/15.jpg)
Mesos Resources
![Page 16: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/16.jpg)
Mesos Resources
![Page 17: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/17.jpg)
Tasks and executors
Schedulers can run one or more tasks within a
executor… and one or more tasks in a
specific container.
So an Executor is a “process container” that
run tasks.
A Task can adjust the size of the Executor’s
container.
![Page 18: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/18.jpg)
How can I use mesos?
![Page 19: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/19.jpg)
Example
A Mesos cluster, 3 Masters, 3
Zookeepers and 3 machines with
Marathon (in white).
9 Slaves (in grey).
2 Chronos (in orange) that schedu- les
2 executions (in blue) to send emails
and to dump a database.
Others frameworks are also running by
Marathon (in orange).
![Page 20: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/20.jpg)
Mesos allocator
Allocator is the responsible for the offers given for each registered
framework
You can implement your custom allocator
By default it implements “Dominant Resource Fairness”, based on max-min
fairness.
![Page 21: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/21.jpg)
Mesos principals, roles and quotas
Principals: It identifies an entity (i.e., a framework or an operator) that
interacts with Mesos
Role: It can be used to associate resources with frameworks in various ways.
Example: 20% of CPU and MEM resources for role production
When a framework run tasks, it can identify itself as some role
Principals are the entity that can be authenticated against Mesos
Users ←→ Groups analogy
![Page 22: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/22.jpg)
Mesos volumes
A framework can reserve or ask for volumes.
These volumes are persisted out of the executor container
When launching the task, the volume is created if the volume is not present
If the volume is present, the content is attached directly to the container
![Page 23: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/23.jpg)
Mesos framework implementation
Mesos components use Protobuf to communicate
Implementing a custom framework consist on implementing a Scheduler and an
Executor
There are bindings for several languages (C++, Java, Scala, Go, Python, Haskell)
Big major change in 1.0: Full REST API for framework developing
![Page 24: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/24.jpg)
Not provided by Mesos
Service Discovery between nodes. You can use Mesos DNS
Persistence and Stateful apps (support added in 0.23). Frameworks for
databases are still in early stages.
Monitoring of Mesos itself. Mesos monitors its frameworks but Mesos itself
(the cluster) is not monitored.
![Page 25: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/25.jpg)
Marathon
Framework to manage Frameworks
Orchestrates apps and frameworks
It simplifies launching your distributed (or not) applications in Mesos without
needing to develop a custom framework for your application
Can run in HA using ZK
It will monitor your app checking if it is healthy
Developed by Mesosphere
![Page 26: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/26.jpg)
Chronos
Distributed and fault tolerant job scheduler for Mesos
‘Kind a’ distributed cron
It is also a framework of Mesos
Supports time intervals, triggers on job completions
Developed by Mesosphere
![Page 27: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/27.jpg)
Aurora
Scheduler for services and jobs (i.e.: Marathon + Chronos)More complex and powerful than Marathon/ChronosAllow rolling updates/rollbackResource quota and Multi-user supportFrom Twitter
![Page 28: Mesos sys adminday](https://reader030.vdocuments.net/reader030/viewer/2022021418/58a9ab311a28ab9c758b5731/html5/thumbnails/28.jpg)
Questions