introducing cloud development with mantl
TRANSCRIPT
![Page 1: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/1.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
We’re ready. Are you?
![Page 2: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/2.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Introducing Cloud Development with Mantl
Brian Hicks (Mantl Core Committer)Ryan Eschinger (Mantl Core Committer)
![Page 3: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/3.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
• Introduction
• Mesos
• Frameworks
• Docker
• Consul
• Mantl
Agenda
3
![Page 4: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/4.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Introduction
4
Brian Hicks Ryan Eschinger
![Page 5: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/5.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
MesosWhat is it?
Program against your datacenter like it’s a single pool of resourcesApache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
5
![Page 6: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/6.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Mesos2-Tier Scheduler
Frameworks:• Receive resources offered by the
master nodes• Schedule work on offered
resources• Are notified if workloads fail via
internal and external mechanisms
6
Mesos:• Determines resources available
on nodes in the cluster• Offers those resources to
frameworks• Keep track of node status (online,
offline, remaining resources)
![Page 7: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/7.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
MesosArchitecture
Slaves (Agents in 1.0):• Tell the masters which resources
are available• Run tasks on claimed resources
7
Masters:• Communicate with Schedulers• Keep track of Agents• Provide a UI for Ops / DevOps
![Page 8: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/8.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
MesosArchitecture
Executor:• A binary or script that runs on the
slave to perform work• Runs in an isolated environment
(can use Docker)
8
Framework (Scheduler):• Schedules work based on offers
received• Domain specific
![Page 9: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/9.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Framework Demos
9
![Page 10: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/10.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
MarathonWhat is it?
10
A cluster-wide init and control system for services in cgroups and Docker containersMarathon is an Apache Mesos framework for long-running applications. Given that you have Mesos running as the kernel for your datacenter, Marathon is the init or upstart daemon.
![Page 11: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/11.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
MarathonOverview
11
• Developed by Mesosphere• Controls app resources, ports,
environment variables, and scaling
• Runs “long-running” tasks
![Page 12: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/12.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
CassandraWhat is it?
Distributed database designed for scalability and high availabilityApache Cassandra is an open source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure.
12
![Page 13: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/13.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
CassandraOverview
13
• Designed to handle big data workloads across multiple nodes (ring) with no single point of failure
• Data is distributed among all nodes in the cluster• Uses the Gossip protocol for peer-to-peer communication• Eventually consistent (C): prioritizes Availability and Partitioning tolerance
(AP)
![Page 14: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/14.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
CassandraMesos Framework
14
• Automates bootstrapping and operations of Cassandra clusters
• Bootstraps initial seed nodes and expands cluster to desired number of nodes
• Runs periodic operational tasks • Restart, remove, and replace nodes on failure• Self-contained: does not require specialized software
installed on agent nodes
![Page 15: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/15.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
SparkWhat is it?
A fast and general engine for large-scale data processingApache Spark is an open source, high performance big data processing framework. It provides high-level APIs in Java, Scala, Python, and R. Spark includes higher-level libraries, including support for SQL queries, streaming data, machine learning and graph processing.
15
![Page 16: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/16.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
SparkOverview
16
• Originally developed at UC Berkeley in 2009• Up to 100x faster than Hadoop for large scale data processing• Execution engine works both in-memory and on-disk• Architecture
• Data storage: supports Hadoop-compatible data sources like HDFS, HBase, Cassandra, etc.
• API: Scala, Java, Python, and R. • Resource Management: standalone, Mesos, Yarn
![Page 17: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/17.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Sparkon Mesos
17
• Runs as a framework• Supports the Docker containerizer• 2-level scheduling: Spark can make decisions about the
offers it receives from Mesos• Enables sophisticated scheduling scenarios
![Page 18: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/18.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
SparkRun Modes
18
• Coarse-grained• Runs long-running Spark executors on every node• Fast startup, better for interactive sessions, but can
be inefficient• Fine Grained
• Launches a Spark executor per task• Slower startup but can utilize resources better
![Page 19: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/19.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
DockerWhat is it?
A shipping container system for applicationsA container virtualization platform that abstracts underlying Linux resource isolation technologies like cgroups and kernel namespaces. Allows developers to build, package, and ship applications that can run anywhere*.
*anywhere that runs a 3.10+ Linux kernel
19
![Page 20: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/20.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
DockerOverview
20
• Compared to virtual machines, containers are smaller, lighter weight, more portable, and easier to deploy
• Package an application, along with all of its dependencies, in a single artifact
• Use the same artifact throughout development, testing, and production• Reduces concerns about the compatibility in different environments
![Page 21: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/21.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Consul
21
![Page 22: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/22.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
ConsulDiscovery (HTTP)
22
curl localhost:8500/v1/catalog/service/marathon[{ "Address": "10.0.113.214", "Node": "mantl-control-01", "ServiceAddress": "", "ServiceID": "marathon", "ServiceName": "marathon", "ServicePort": 18080, "ServiceTags": ["marathon"]}]
![Page 23: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/23.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
ConsulDiscovery (DNS)
23
dig +short marathon.service.consul10.0.113.21410.0.220.5410.0.131.25
dig +short marathon.service.consul SRV1 1 18080 mantl-control-02.node.mantl.consul1 1 18080 mantl-control-01.node.mantl.consul1 1 18080 mantl-control-03.node.mantl-consul
![Page 24: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/24.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
ConsulKey/Value Store
24
curl -X PUT -d world http://localhost:8500/v1/kv/hellotrue
![Page 25: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/25.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
ConsulKey/Value Store
25
curl -X GET http://localhost:8500/v1/kv/hello[{ "CreateIndex": 223, "Flags": 0, "Key": "hello", "LockIndex": 0, "ModifyIndex": 223, "Value": "d29ybGQ="}]
![Page 26: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/26.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Consulconsul-template
26
global maxconn {{or (key "service/haproxy/maxconn") 256}} debug
{{range services}}frontend {{.Name}} bind *:{{key (printf "ports/%s" .Name)}}{{range service .Name}} server {{.Node}} {{.Address}}:{{.Port}}{{end}}{{end}}
![Page 27: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/27.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Mantl
27
![Page 28: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/28.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID
Thank you
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID 28
![Page 29: Introducing Cloud Development with Mantl](https://reader036.vdocuments.net/reader036/viewer/2022062523/58f04fe21a28abf84b8b45e9/html5/thumbnails/29.jpg)
© 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID