100% Open Source, multi-tiered application infrastructure provisioning and management!

Nitin Motgi Albert Shau@nmotgi @albertshau

tryloom.io

Our Problem

2

• Create variety of application infrastructure stack - Continuuity Reactor + Hadoop (CDH, HDP, Apache), Secured Hadoop, LAMP, Mesos, …

• In Public and Private Cloud

• Using our existing investment in SCM tools - Chef, Shell script, etc

• Easy way to manage the stack - like add new services, manage configuration changes, scale up/down, etc..

• DevOps friendly, easy to manage and maintain

• Resource constraints (1 Ninja support 20 Developers) - Ninja had no vacations or couldn’t get sick

3

Challenges

Simply HardMulti-tier applications are hard to provision, install, configure, monitor & scale and also have complex cluster life cycles.

Lack of experience and repeatability Installation and Configuration require a lot of practice to tackle common pitfalls for configuring complex multi-tier application clusters, complexity prevents simple attempts at automation

Increased organization bottlenecksComplexity limits number of people who can provision, install and configure complex multi-tier application clusters

Decreased developer productivityHard to support DevOps role within an organization without an easy to use self serving model

No governance with reference architecturesDifficult to model reference architectures in complex environments

Multiple cloud environmentsSupporting multiple cloud environments is extremely hard.

Our Assessment

4

• Numerous technologies available that solve parts of the problem

• Open Source / Closed Source

• We looked closely at Apache Ambari, Crowbar, etc…

• Concluded that none at current state were solving our problem (Happy to talk about it after this talk)

Introducing

5

A system for templatizing and materializing complex multi-tiered application reference architectures in public or private clouds. Designed bottom-up to support different facets of your organization - from developers, operations and system administrators to large service providers.

Who can use it ?

6

CONTINUUITY LOOM TEMPLATES ON CLOUD

DEVELOPERCreate, provision and decommission clusters of complex, multi-tiered applications using a self-service model.

SYS ADMINWrite Chef recipes, Puppet modules and shell scripts for application services.

IT OPERATIONSCreate and manage Continuuity Loom templates for multi-tiered application services. Configure compliance policies and enforce service constraints.

Lifecycle

7

Create!Loom!

Template

Save!Template

Publish!Template

Expose!REST API

Create!Instance of !

template

on any cloud

Reference !Architecture!(Enterprise)

Centralized!Administration!

Control with!Multi-tenancy

Supports Hybrid!Infrastructure!& Architecture

Concepts

8

• Service!

• Software component. ex: Namenode, Datanode, ResourceManager, etc.

• Specify dependencies between services

• Template!

• Defines how a cluster should be created

• Specify compatible services, configuration and constraints which determine service placement

• Provider!

• Creates and destroys machines. ex: AWS, Rackspace, Joyent, Openstack

• Automator!

• Performs service actions like start, stop, config, install, remove, init

• ex: chef, puppet, scripts, etc.

Architecture

9

Cluster Creation: Solver

10

Template +

args (size, etc.)

Solver

Layout

services:{s1, s3} hw: large image: ubuntu12

x 1

services:{s2} hw: medium image: ubuntu12

x 4

Cluster Creation: Planner

11

Layout Dependencies

Planner

Coordinator

Task Queue

ProvisionersProvisionersProvisionersProvisionersProvisionersProvisioners

Plan

Planner : DAG

12

Planner : Linearize

13

Planner: Expand

14

DEMO

15

16

Thank you

Nitin Motgi Albert Shau@nmotgi @albertshau

http://github.com/continuuity/loom