hacking apache cloud stack

Hacking on

Apache (Incubating) CloudStack

Tutorial Outline

•  Session 1: Introduction to CloudStack Murali Reddy: Committer Apache CloudStack

•  Session 2: Architecture of CloudStack Murali Reddy: Committer Apache CloudStack

•  Session 3: Hands on with DevCloud Kishan Kavala: Committer Apache CloudStack Rajesh Battala: Contributor Apache CloudStack

Session 1

Introduction to CloudStack

Cloud

Built for traditional enterprise apps & client-server compute •  Enterprise arch for 100s of hosts •  Scale-up (pool-based resourcing) •  IT management-centric •  1 administrator for Dozens of servers •  Apps assume reliability •  Proprietary vendor stack

Designed around big data, massive scale & next-gen apps •  Cloud arch for 1000s of hosts •  Scale-out (horizontal resourcing) •  Autonomic management •  1 administrator for 1,000’s of servers •  Apps assume failure •  Open, value-added stack

Server Virtualization++

Cloud Computing

Virtualization is not Cloud computing

• Tenets of Cloud o  Shared infrastructure and Multi-tenancy o  Self Service o  Elasticity o  Built for massive Scale o  Service agility o  Pay-as-you-go o  APIs and Extreme Automation

•  IAAS/PAAS/SAAS • Public/Private/Hybrid clouds

Cloud Computing (contd..)

•  Turnkey orchestration platform for delivering IAAS clouds o  Secure, multi-tenant o  Self-service o  Service agility and elasticity o  Built for large scale o  Pay-as-you-go

•  Deploys on premise (private) or as a hosted (public) cloud •  Can be used for hybrid clouds •  built in java, provides native REST API’s and EC2 API •  Has python, Ruby clients and CLI as well

What is Apache CloudStack

A bit of History

• Original company Cloud.com (2008) • Open source (GPLv3) as CloudStack (2010) • Acquired by Citrix (July 2011) • Relicensed under ASL v2 April 3, 2012 • Accepted as Apache IncubaKng Project April 16, 2012

• First Apache (ACS 4.0) released • Many non-‐Citrix contributors, commiRers, PPMC members

Who is contributing

• Sungard: Unit test cases • Carnigo: Object store plug-in • Ceph/Rbd support by Wido • CLVM/KVM by Marcus • Nicira NVP: Schuberg Philis • Basho: Object Store • Brocade ADX ADC support • Midokura midonet SDN controller integration

How to contribute

•  Its not just about code! As community member you can engage in o Discussions: Design, Use Case, deployment

issues o Bug reporting, feature requests o Code reviews o Build, tools, infrastructure o Helping out on the IRC o Documentation o Submit bug fixes, features

How to contribute (contd..)

• Git repo, bug tracker, wiki are on ASF infra • Project website

o  http://incubator.apache.org/cloudstack/ o  http://www.cloudstack.org

•  IRC o  #cloudstack on irc.freenode.net o  Wednesday - 10:30 PM IST, 5:00 UTC

• Mailing lists (cloudstack.org/discuss/mailing-lists.html) o  [email protected] o  [email protected]

• http://www.slideshare.net/cloudstack

CloudStack managed cloud

Compute

Storage

Network

Admin

Users

Org B

End User Cloud Admin

On-demand infrastructure as a service

CloudStack Management Server

REST API

UI Cli EC2

Admin

Users

Org A

Consume resources

Provision resources

manage resources

•  Hosts •  Servers onto which services will be provisioned

•  Primary Storage •  VM storage

•  Cluster •  A grouping of hosts and their associated storage

•  Pod •  Collection of clusters

•  Network •  Logical network associated with service offerings

•  Secondary Storage •  Template, snapshot and ISO storage

•  Zone •  Collection of pods, network offerings and secondary

storage

•  Management Server Farm •  Responsible for all management and provisioning

tasks

Core CloudStack Components

Zone

CloudStack Pod

Cluster

Host

Host

Network

Primary Storage

VM

VM

CloudStack Pod

Cluster Secondary

Storage

Pod 1

….

Cluster N

Access Layer

Host 2

Cluster 1

CloudStack Deployment Architecture

Host 1

Ø  Hypervisor is the basic unit of scale.

Ø  Cluster consists of one ore more hosts of same hypervisor

Ø  All hosts in cluster have access to shared (primary) storage

Ø  Pod is one or more clusters, usually with L2 switches.

Ø  Availability Zone has one or more pods, has access to secondary storage.

Ø  One or more zones represent cloud

Primary Storage

Zone 1

….

L3 core

Secondary

Storage

Pod N

CloudStack Management

Server Internet

Zone1

Data Center 1

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 3

Zone 4

Management Server

Ø  Single Management Server can manage multiple zones

Ø  Zones can be geographically distributed but low latency links are expected for better performance

Ø  Single MS node can manage up to 5K hosts.

Ø  Multiple MS nodes can be deployed as cluster for scale or redundancy

CloudStack Managing Multiple Zones

Infrastructure provisioning

Infrastructure provisioning (contd.)

Compute/Disk/Network Offering

Select Operating System •  Windows, Linux

Select Compute Offering •  CPU & RAM

Select Disk Offering •  Volume Size

Select Network Offering •  Network & Services

Create VM

Create Virtual Machines via Offerings

Virtual Machine Management

Users

Start

Stop

Restart

Destroy

VM Operations Console Access

•  CPU Utilized

•  Network Read

•  Network Writes

VM Status Change Service Offering

2 CPUs 1 GB RAM 20 GB 20 Mbps

4 CPUs 4 GB RAM 200 GB 100 Mbps

Volume & Snapshot Management

Volume

VM 1 Add / Delete

Volumes

Schedule Snapshots

Hourly Daily

Weekly Monthly

Now

Create Templates from Volumes

Volume

Template

View Snapshot History

….

A Very Flexible IaaS Pla5orm

Compute XenServer VMware KVM Oracle VM Bare metal

Hypervisor

Storage Local Disk iSCSI NFS Fiber

Channel Swift

Block & Object

Network Network Type Isolation Load

balancer Firewall VPN

Network & Network Services

Primary Storage Secondary Storage

Ceph Riak

Pod 1

Host 2

Cluster 1

Host 1

Primary Storage

L3 switch

Secondary

Storage

L2 switch

CloudStack Storage

•  Configured at Cluster-level. Close to hosts for better performance

•  Stores all disk volumes for VMs in a cluster

•  Cluster can have one or more primary storages

•  Local disk, iSCSI, FC or NFS

Primary Storage

•  Configured at Zone-level

•  Stores all Templates, ISOs and Snapshots

•  Zone can have one or more secondary storages

•  NFS, OpenStack Swift

Secondary Storage

•  Storage available on hypervisor hist

Local Storage

Local storage

Availability zone

•  Primary Storage •  Cluster level storage for VMs •  Connected directly to hosts •  NFS, iSCSI, FC and Local

•  Secondary Storage •  Zone level storage for template, ISOs and

snapshots •  NFS or OpenStack Swift via CloudStack

System VM

•  Templates and ISOs •  Imported into CloudStack •  Can be private or public

Role of Storage and Templates

Zone

Secondary Storage

Pod

Cluster

Host

Host

Primary Storage

Template

1.  User Requests Instance

2.  Provision Optional Network Services

3.  Copy instance template from secondary storage to primary storage on appropriate cluster

4.  Create any requested data volumes on primary storage for the cluster

5.  Create instance

6.  Start instance

Provisioning Process

Zone Secondary Storage

Pod

Cluster

Host

Host

Primary Storage

VM

Template

Object Storage

Availability Zone

Availability Zone

Availability Zone

CloudStack Mgmt Server

Object Store

•  Object store used to store templates and snapshots

•  VM’s can be distributed across the availability zones

•  For DR create instances in different zones

Domain is a unit of isolation that represents a customer org, business unit or a reseller

Domain can have arbitrary levels of sub-domains

A Domain can have one or more accounts

An Account represents one or more users and is the basic unit of isolation

Admin can limit resources at the Account or Domain levels

Admin

Org A

Admin

Reseller A

Domain

Domain

Admin

Org C

Sub-Domain

User 1

User 2

Group B

Account

Group A

Account

VMs, IPs, Snapshots…

VMs, IPs, Snapshots…

Resources

Resources

Multi-tenancy & Account Management

User Dashboard: Consumed Resources

•  Running, Stopped & Total VMs

•  Public IPs

•  Private networks

•  Latest Events

Admin Dashboard: Consumed Resources

•  Provides zone wide resource consumption

•  Also provides latest alerts and events

Edge services with System VMs

•  System VMs optimize and scale the datapath on behalf of CloudStack o  Stateless, can be destroyed and recreated from database state o  Highly Available o  Communicates with Management Server over management network o  Usually have 3 interfaces: control, guest and public

•  Console Proxy VM o  Provides AJAX-style HTTP-only console viewer o  Grabs VNC output from hypervisor o  Scales out (more spawned) as load increases o  Java-based server Communicates with MS over message bus

•  Secondary Storage VM o  Provides image (template) management services o  Download from HTTP file share or Swift o  Copy between zones o  Scale out to handle multiple NFS mounts o  Java-based server communicates with MS over message bus

•  Virtual Router VM o  Provides multiple network services o  IPAM (DHCP), DNS, NAT, Source NAT, Firewall, PF, VPN o  User-data, Meta-data, SSH keys and password change server o  Redundancy via VRRP o  MS configures VR over SSH

§  Proxied via the hypervisor on XS and KVM

Edge services with System VMs (contd.)

Network & Network Services

•  Create Networks and attach VMs

•  Acquire public IP address for NAT & load balancing

•  Control traffic to VM using ingress and egress firewall rules

•  Set up rules to load balance traffic between VMs

Networking feature overview

•  Orchestration of L2 – L7 network services o  IPAM, DNS, Gateway, Firewall, NAT, LB, VPN, etc

•  Mix-and-match services and providers •  Out-of-the-box integration with automated deployment of virtual

routers o  Highly available network services using CloudStack HA and VRRP

•  Orchestrate external providers such as hardware firewalls and load balancers o  Devices can provide multiple services o  Admin API to configure external devices o  Plugin-based extensions for network behavior and admin API extensions

•  Multiple multi-tenancy [network isolation] options •  Integrated traffic accounting •  Access control •  Software Defined Networking (Nicira NVP)

L2 Features •  Choice of network isolation

o  Physical, VLAN, L3 (anti-spoof), Overlay[GRE] o  Physical isolation through network labels [limited to # of

nics or bonds] •  Multi-nic

o  Deploy instance in multiple networks o  Control default route

•  Access control o  Shared networks, project networks

•  QoS [max rate] •  Traffic monitoring •  Hot-plug / detach of nics

L3 Features •  IPAM [DHCP], Public IP address management

o  VR acts as DHCP server o  Can request multiple public IPs per tenant

•  Gateway (default gateway) o  Redundant VR (using VRRP) o  Inter-subnet routing o  Static routing control

•  Remote Access VPN o  L2TP over IPSec using PSK o  Virtual Router only

•  Firewall based on source cidr •  Static NAT [1:1]

o  Including “Elastic IP” in Basic Zone

•  Source NAT o  Per-network, or interface NAT

•  Public Traffic usage o  Monitoring on the Virtual Router / External network device o  Integration with sFlow collectors

•  Site-to-Site VPN o  IPSec VPN based on VR

•  L3 ACLs

L4 Features

• Security groups for L3-isolation o  “Basic Zone” in docs o Default AWS-style networking o Scales much better than VLANs

• Stateful firewall for TCP, UDP and ICMP • Port forwarding [“Advanced Zone”]

o Conserve public Ips

L7 features

•  Loadbalancer o  VR has HAProxy built in o  External Loadbalancer support

§  Netscaler (MPX/SDX/VPX) §  F5 BigIP §  Can dedicate an LB appliance to an account or share it

among tenants o  Loadbalancer supported with L3-isolation as well o  Stickiness support o  SSL support [future] o  Health Checks [future]

•  User-data & meta-data o  Fetched from virtual router

•  Password change server

CloudStack Terminology

•  Guest network o  The tenant network to which instances are attached

•  Storage network o  The physical network which connects the hypervisor to primary storage

•  Management network o  Control Plane traffic between CloudStack management server and hypervisor clusters

•  Public network o  “Outside” the cloud [usually Internet] o  Shared public VLANs trunked down to all hypervisors

•  All traffic can be multiplexed on to the same underlying physical network using VLANs o  Usually Management network is untagged o  Storage network usually on separate nic (or bond)

•  Admin informs CloudStack how to map these network types to the underlying physical network o  Configure traffic labels on the hypervisor o  Configure traffic labels on Admin UI

CloudStack Network Service Providers

•  A Network Service Provider is hardware or virtual appliance that makes a network service possible in CloudStack ; for example, a Citrix NetScaler appliance can be installed in the cloud to provide Load-Balancing services.

•  Administrators can have multiple instances of the same service provider in a network; for example, more than one Citrix NetScaler or Juniper SRX device can be added to CloudStack

•  CloudStack supports the following Network Providers: o  CloudStack Virtual Router (default) o  Citrix NetScaler SDX, VPX and MPX models o  Juniper SRX o  F5 BigIP

Network Service Providers Matrix

Feature Virtual Router

Citrix NetScaler

Juniper SRX

F5 BigIP

Remote Access VPN YES N/A N/A N/A Firewall YES N/A YES N/A Source NAT YES N/A YES N/A Static NAT YES YES YES N/A Load Balancing YES YES N/A YES Port Forwarding YES N/A YES N/A Elastic IP N/A YES N/A N/A Elastic LB N/A YES N/A N/A DHCP/DNS/User Data YES N/A N/A N/A

•  Network offerings is basically a definition of what Network Services are available when this offering is used. The available Network Services are: VPN, DHCP, DNS, Firewall, Load Balancer, User Data, Source NAT, Static NAT, Port Forwarding and Security Groups*

•  Cloud provider defines the feature set for guest networks

•  Toggle features or service levels o  Security groups on/off o  Load balancer on/off o  Load balancer software/hardware o  VPN, firewall, port forwarding

•  User chooses network offering when creating network

•  Enables upgrade between network offerings

•  Default offerings built-in o  For classic CloudStack

networking

Network Offerings

Add Guest Networks

•  Choice to choose L3 subnet, default gateway

•  Choice of network

offerings

Editing Guest Networks

When editing a guest network users can change the network offering. They can either upgrade to a “premium” network offering (for example offering that uses hardware Load-balancer) or downgrade to a “cheaper” network.

•  Restarting the network will simply resend all the LB, Firewall and Port-Forwarding rules to the network provider

•  Restarting the Network with “Clean up”: •  restarKng network elements -‐ virtual routers, DHCP servers

•  If virtual router is used, it will be destroyed and recreated

•  Reapplying all public IPs to the network provider •  Reapplying load-‐Balancing/Port-‐Forwarding/Firewall rules

Restarting/Cleaning Up a Guest Network

•  An Isolated Guest Network can only be deleted if no VMs are using these network (e.g. Completely destroyed and expunged)

•  Deleting a Network will Destroy the Virtual Router (if used) and will release the Public IPs back to the IP Pool

Deleting a Guest Network

Basic vs Advanced Networking

• Segmentation based on feature set and ease-of-deployment

• Both are feature-rich • Basic implements true AWS-style L3-isolation

o  Tenants do not get contiguous IP addresses or subnets o  Network segmentation based on Security Groups o  Tremendous scale (tens of thousands)

• Advanced Zone offers full L3 subnets and L2 isolation o  VLANs are default implementation (4K limit) o  More features (source NAT, PF, LB, VPN)

Storage 1

Hypervisor 1

Hypervisor N

Hypervisor 8

Access Switch(es) Cloudstack Server

VM Traffic

Control Plane Traffic

Storage Traffic

Cloudstack Servers

Storage k

…

Pod 1

CLUSTER 1

…

CLUSTER 4

Core (L3) Network

…

Pod 2 Pod N

Physical Network in Zone

Storage 2

Hypervisor N+1

Public Traffic

…

DB Security Group

Web Security Group

Layer 3 cloud networking

… …

Web VM

Web VM

Web VM

Web VM

DB VM

Web VM

DB VM

Web VM

Guest Networks with L3 isolation Guest 1 VM 1 Guest 2 VM 1 Guest 1 VM 2

Guest 2 VM 2

Public Internet

10.1.0.1

Public IP address 65.37.141.11 65.37.141.24 65.37.141.36 65.37.141.80

Guest address 10.1.0.2




Load

Balancer Guest 2 VM 3 Guest 1 VM 3 Guest 1 VM 4




L3 Core Switch

Pod 1 L2 Switch

Pod 3 L2 Switch

10.1.16.1

…

… 10.1.8.1

Pod 2 L2 Switch

Hypervisor 1

Hypervisor N

Hypervisor 8

Access Switch(es)

VM Traffic

…

Pod K

CLUSTER 1

…

CLUSTER 4

Core (L3) Network

…

Pod M Pod N

Guest Networks with L2 isolation

Hypervisor N+1

Public Traffic

Hypervisor

R

R V

VV

V

Hypervisor V V

VR

Tenant VM Tenant Virtual Router

…

L2 isolation: VLAN networking

… …

User 2

User 2 User

1

User 1

User 1

User 1

User 1

User 2

User 1

SDN at Work

Host 1

Host 2

Host 3

Host 4

GRE Tunnel

GRE Tunnel GRE Tunnel

VM1

VM 2

VM 3

VR

OVS

OVS OVS

CloudStack Mgmt Server SDN

Controller

VM 1

VM 2

VM 3

VR

OVS

GRE Tunnel

Guest virtual layer-2 network

Guest 1 VM 1 Guest 1 VM 2 Guest 1 VM 3 Guest 1 VM 4

Public Internet

Public Network

Guest Virtual Network 10.1.1.0/24

Gateway address 10.1.1.1

NAT DHCP Load Balancing VPN

Public IP address 65.37.141.11 65.37.141.36





Guest 1 Virtual Router

Guest 2 VM 1 Guest 2 VM 2 Guest 2 VM 3

Guest Virtual Network 10.1.1.0/24


NAT DHCP Load Balancing VPN




Guest 2 Virtual Router

Public IP address 65.37.141.24 65.37.141.80

Layer-2 Guest Virtual Network

Public Network/Internet

Guest Virtual Network 10.1.1.1/8 VLAN 100


DHCP, DNS NAT Load Balancing VPN

Public IP 65.37.141.11

10.1.1.1

Guest VM 1

10.1.1.3

Guest VM 2

10.1.1.4

Guest VM 3

10.1.1.5

Guest VM 4

CS Virtual Router


Guest Virtual Network 10.1.1.1/8 VLAN 100

Private IP 10.1.1.112

DHCP, DNS

Public IP 65.37.141.112

10.1.1.1

Guest VM 1

10.1.1.3

Guest VM 2

10.1.1.4

Guest VM 3

10.1.1.5

Guest VM 4

NetScaler Load

Blancer

Private IP 10.1.1.111

Public IP 65.37.141.111

Juniper

SRX Firewall

CS Virtual Router provides Network Services External Devices provide Network Services

CS Virtual Router

Layer-3 Guest Network

Public Network 65.11.0.0/16

65.11.1.2

Guest VM 1

Guest VM 2

Guest VM 3

Guest VM 4


NetScaler Load

Blancer

Network Services Managed Externally Network Services Managed by CS

65.11.1.3

65.11.1.4

65.11.1.5

DHCP, DNS

CS Virtual Route

r

Security Group 1

Security Group 2

10.1.2.3 Guest VM 1

Guest VM 2

Guest VM 3

Guest VM 4

10.2.12.4

10.5.2.99

10.1.2.18

DHCP, DNS

CS Virtual Router

Security Group 1

Security Group 2

EIP, ELB

65.11.1.2

65.11.1.3

65.11.1.4

L3 switch

Multi-tier network

10.1.1.1

Web VM

1

10.1.1.3

Web VM

2

10.1.1.4

Web VM 3

10.1.1.5

Web VM 4

Virtual Network 10.1.1.0/24 VLAN 100


10.1.2.31

App VM

1


10.1.2.24

App VM

2

10.1.3.24

DB VM

1

CS Virtual Router

Customer Premises

IPSec or SSL site-to-site VPN

Internet

Monitoring VLAN

Virtual Router Services •  IPAM •  DNS •  LB [intra] •  S-2-S VPN •  Static Routes •  ACLs •  NAT, PF •  FW [ingress & egress] •  BGP

Loadbalancer

Session 2

Architecture of CloudStack

Problem Definition

• Offer a scalable, flexible, manageable IAAS platform that orchestrate physical and virtual resources to offer self-service infrastructure provisioning and monitoring

•  Flexible

o Handle new physical resource types § Hypervisors, storage, networking

o Add new APIs o Add new services o Add new networking models

Problem Definition (contd..)

•  Manageable o Hide complexity of underlying resources o Rich functional end-user and admin UI o Admin API to automate operations o Easy install, upgrade for small -> large clouds o Simple scaling, automated resilience

•  Scalable architecture o  1 -> N hypervisors / VMs / virtual resources o  1 -> N end users

Problem Definition (contd..)

•  Resource Allocation o  Hypervisor CPU, Memory o  Storage space o  Avoid set of pods, clusters, hosts

•  Capacity scanning o  Snapshot of resources consumed o  Trigger capacity threshold violations

•  Garbage collection o  Network resources (IP, VLAN, CIDR etc) o  Compute (VM, CPU, memory) o  Storage (volumes)

•  Synchronizing the resource states •  Infrastructure resource failures •  Fencing

Management

Server MySQL

DB

Back Up DB

Infrastructure Resources

User API

Admin API

Load Balancer

Management

Server

Management

Server

Management

Server

MySQL DB

Infrastructure Resources

User API

Admin API

Single-node Deployment

Multi-node Deployment

Ø  MS is stateless. MS can be deployed as physical server or VM

Ø  Single MS node can manage up to 10K hosts. Multiple nodes can be deployed for scale or redundancy

Replication

Scaling: Horizontal Scaling

Resource Load Balancing

•  As management server is added into the cluster, resources are rebalanced seamlessly. o  MS2 signals to MS1 to hand over a resource o  MS1 wait for the commands on the resources to finish o  MS1 holds further commands in a queue o  MS1 signals to MS2 to take over o  MS2 connects o  MS2 signals to MS1 to complete transfer o  MS1 discards its resource and flows the commands being held to MS2

•  Listeners are provided to business logic to listen on connection status and adjusts work based on who’s connected.

•  By only working on resources that are connected to the management server the process is on, work is auto-balanced between management servers.

•  Also reduces the message routing between the management servers.

Management Server

Kernel -  Drives long running VM

operations -  Syncs between resources

managed and DB -  Generates events

Resource Managemen

t

Cluster Managemen

t

Job Management DB

UI Cloud Portal CLI

Other Clients

Job Queue

Deployment Planning

Network Configurations

Network Elements

Hypervisor Gurus

Database Access

Alert & Event Management

Plu

gin

AP

I

Hypervisor Resources

Network Resources

Storage Resources

Image Resources

Snapshot Resources

REST API

OAM&P API End User API EC2 API

Pluggable Service API Engine

Other APIs

Security Adapters

Account Management Connectors

ACL & Authentication -  Accounts, Domains, and Projects -  ACL, limits checking

Services API

Ser

vice

s A

PI

Console Proxy Management

Template Access

HA

Usage Calculations

Additional Services

Event Bus Message Bus

Interactions

CloudStack

Cloud user {API client (Fog/etc)}

End User UI

Admin UI

MySQL

CloudStack Clustered

CloudStack Management

Server

Domain

Admin UI

CS Admin & End-user API

Cloud user {ec2 API client }

ec2 API

Monitoring CS API vSphere Cluster Primary Storage

vcenter

XS Cluster Primary Storage

XAPI

KVM Cluster Primary Storage JSON

OVM Cluster Primary Storage

NetConf

Nitro API Juniper SRX

Netscaler

Console Proxy VM Console

Proxy VM

JSON

Cloud user

HTTPS Ajax Console

VNC

Sec. Storage

VM

NFS Server

NFS Sec. Storage

VM HTTP (Template Download)

HTTP (Template Copy) HTTP (Swift)

NFS

Router VM Router VM

Router VM

JSON

{Proxied} SSH

Management Server Layering

Balancing Incoming Requests

•  Each management server has two worker thread pools for incoming requests: effectively two servers in one. o  Executor threads provided by tomcat o  Job threads waiting on job queue

•  All incoming requests that requires mostly DB operations are short in duration and are executed by executor threads because incoming requests are already load balanced by the load balancer

•  All incoming requests needing resources, which often have long running durations, are checked against ACL by the executor threads and then queued and picked up by job threads.

•  # of job threads are scaled to the # of DB connections available to the management server

•  Requests may take a long time depending on the constraint of the resources but they don’t fail.

Inside a Management Server

API Servlet

Async Job

Queue Mgr

CloudStack API

Commands

Responses

cmd.execute() Plugins

Plugins Plugins

Message Bus

Agent Manager

Resources

Agent API (Cmds)

Hypervisor Native APIs

Local Or Remote

Network Device API

MySQL

Kernel Services API

CloudStack API Sync/Async commands

•  Package and Location cloudstack-oss/api/src/com/cloud/api/…

•  BaseCmd (base class) All commands descend from the BaseCmd base class

CloudStack API

Configuration Commands are configured in cloudstack-oss/client/command.properties.in Format:

<command name>=<java classname>;<ACL> *note* ACL is calculated as a bitmap with the following, 1 = ADMIN, 2 =

RESOURCE_DOMAIN_ADMIN, 4 = DOMAIN_ADMIN, 8 = USER Example:

### snapshot commands!createSnapshot=com.cloud.api.commands.CreateSnapshotCmd;15!listSnapshots=com.cloud.api.commands.ListSnapshotsCmd;15!deleteSnapshot=com.cloud.api.commands.DeleteSnapshotCmd;15!createSnapshotPolicy=com.cloud.api.commands.CreateSnapshotPolicyCmd;15!deleteSnapshotPolicies=com.cloud.api.commands.DeleteSnapshotPoliciesCmd;15!listSnapshotPolicies=com.cloud.api.commands.ListSnapshotPoliciesCmd;15!

CloudStack API: adding API

Adding a new command Determine type of command Synchronous Synchronous List Based Asynchronous Asynchronous Create based Create your command Define request parameters Implement the execute() method Implement an appropriate ResponseObject Add new command to command.properties.in

Management Layer

•  Management layer is collection of Managers

o  Managers are responsible for directing a specific area of the cloud §  Storage Manager

• Manages primary storage server (allocation, life-cycle, attach, detach, user volumes, life-cycle of the primary storage server itself)

§  Network Manager • Manages network configurations, IP Allocations, Port

Forwarding, Load Balancers etc. §  User Vm Manager

• Manages life-cycle of VMs created in the cloud §  And many more!!!

•  Managers coordinate with each other to achieve a task

Management Layer: Adapters

•  Modularization and customization within the CloudStack management server is achieved through the use of the Adapter framework.

•  Each Adapter is uniquely identified by the interface it exposes and represents the boundary between CloudStack and the individual component and/or processes that can be configured into the system

•  Adapters provide extensibility and in many cases device specific implementation details while maintaining a simple and consistent interface.

Management Layer: Adapters

•  Adapters are executed as a chain in the order that they are configured

•  Defined in cloudstack-oss/client/tomcatconf/components.xml.in <adapters key="com.cloud.network.guru.NetworkGuru”>

<adapter name="StorageNetworkGuru” class="com.cloud.network.guru.StorageNetworkGuru"/>

<adapter name="ExternalGuestNetworkGuru" class="com.cloud.network.guru.ExternalGuestNetworkGuru"/>

<adapter name="PublicNetworkGuru" class="com.cloud.network.guru.PublicNetworkGuru"/>

<adapter name="PodBasedNetworkGuru" class="com.cloud.network.guru.PodBasedNetworkGuru"/>

<adapter name="ControlNetworkGuru" class="com.cloud.network.guru.ControlNetworkGuru"/>

<adapter name="DirectNetworkGuru" class="com.cloud.network.guru.DirectNetworkGuru"/>

<adapter name="DirectPodBasedNetworkGuru" class="com.cloud.network.guru.DirectPodBasedNetworkGuru"/>

<adapter name="OvsGuestNetworkGuru" class="com.cloud.network.guru.OvsGuestNetworkGuru"/>

</adapters>

Adapter Interfaces Available

•  Discoverer •  StoragePoolDiscoverer •  StoragePoolAllocator •  ConsoleProxyAllocator •  Investigator •  FenceBuilder •  DeploymentPlanner •  NetworkGuru •  NetworkElement •  And more…

•  VirtualMachineGuru •  HypervisorGuru •  Listener •  UserAuthenticator •  SecurityChecker

Adapters: VM orchestration

• Deployment Planner o First Fit planner

• Host Allocator o  First Fit o Random

• Storage Allocator o  First Fit o Random

Adapters: Network Orchestration

•  Network Guru (Responsible for L2-L3) o  Design o  Implement o  Allocate o  Release o  Shutdown e.g. guest network guru, OVS network guru etc

•  Network Element (Responsible for L4-L7) o  Implement o  Shutdown e.g. F5, SRX, NetScaler, Virtual Router

Extending CloudStack Networking

Network Manager

Network Element

DnsService

MyDnsElement MyDnsDeviceManager

MyDnsDeviceService

PluggableService

MyDnsDeviceResource

AgentManager Queue

1. prepare (part of start vm) 2. prepare (Network, Nic, DeployDestination, VmInfo)

3. addDnsRecord(ip, fqdn)

4.Enqueue AddDnsRecord

5.API call to Dns Device

Device Configuration Admin API (CRUD)

MySQL Demonstrates one way to inform an external DNS server when an instance starts. Classes shaded blue form a plugin / service bundle to integrate an external DNS server. Clients of the instance can then use DNS names to access the instance.

Sequence Flow for VM Creation Job

Threads Network Element

User VM Mgr

Network Mgr

Storage Mgr

VirtualMachine Mgr

Network Guru

Start VM

Start VM

Prepare Nics

Notify that Nic is about to be started in network

Reserve resources for Nic

Services API

Server Resource

s

Start User VM

Agent Calls

Prepare Volumes

Template Mgr

Deployment

Planner

Get a Deployment Plan (Host and StoragePool)

Prepare template on Primary Storage

Agent Calls

Agent Start VM Call

Stores job result

Management Layer: Adapters flow

Server Resources

•  Resources are carried in service VMs to be in close network proximity to the physical resources it manages

•  Easily scales to utilize the most abundant resource in data center (CPU & RAM)

•  Communicates with Orchestration Server over message bus (JSON)

•  Can be replicated for fault tolerance

•  Control gateway to resources within data center

Agent Hypervisor Resources

Network Resources

Storage Resources

Image & Template Resources

Snapshot Resources

Res

ourc

e A

PI

Resource Layer

Working toward 4.1 release

• 4.1 is next major release o Moving away from monolithic architecture to loosely

coupled subsystems o Spring for IOC container and AOP o Storage subsystem refactoring o Network subsystem refactoring o New orchestration engine o Regions support

Session 3

Developing with DevCloud

DevCloud

• CloudStack requires o Hypervisor o Network o Storage

DevCloud

• self-contained CloudStack runs in the appliance

DevCloud

• Several use cases o  Try CloudStack in an isolated sandbox. Runs within

the appliance o Develop CloudStack on own machine, build locally

and deploy new version in DevCloud (Build and test) o Develop and Run locally, use DevCloud as Xen hosts

Thanks

hacking apache cloud stack

Documents