2 architectural at cloudstack developer day

Apache CloudStack

Architecture

Alex Huang

Software Architect, Citrix Systems

Compute

Self-Service Access via CloudStack UI

Network

Storage

Admin

Users

Org A

Admin

Users

Org B

Users

End User

Admin

Deployment Architecture

• Hosts • Servers onto which services will be provisioned

• Primary Storage • VM disk storage

• Cluster • A grouping of hosts and their associated storage

• Pod • Collection of clusters in the same failure boundary

• Network • Logical network associated with service offerings

• Secondary Storage • Template, snapshot and ISO storage

• Zone • Collection of pods, network offerings and

secondary storage

• Management Server Farm • Management and provisioning tasks

Components

Zone

CloudStack Pod

Cluster

Host

Host

Network

Primary Storage

VM

VM

CloudStack Pod

Cluster Secondary

Storage

Pod 1

Host 2

Cluster 1

Host 1

Primary Storage

L3 switch

Secondary Storage

L2 switch

Two Types of Storage

• Stores disk volumes for VMs in a cluster • Configured at Cluster-level. • Close to hosts for better performance • Cluster have at least one primary storage • Requires high IOPs (can be expensive)

Primary Storage

• Stores all Templates, ISOs and Snapshots • Configured at Zone-level • Zone can have one or more secondary

storages • High capacity, low cost commodity

storage

Secondary Storage

Pod 1

….

Cluster N

L2

Host 2

Cluster 1

Deployment Architecture

Host 1

Hypervisor is the basic unit of scale.

Cluster consists of one ore more hosts of same hypervisor

All hosts in cluster have access to shared (primary) storage

Pod is one or more clusters, usually with L2 switches.

Availability Zone has one or more pods, has access to secondary storage.

One or more zones represent cloud

Primary Storage

Zone 1

….

L3

Secondary Storage

Pod N

Management Server Cluster

Internet

Management Server Cluster

Replica

Infrastructure Resources

User API

Admin API

Load Balancer

Management Server

Management Server

MySQL

MS is stateless. MS can be deployed as physical server or VM

Single MS node can manage up to 10K hosts. Multiple nodes can be deployed for scale or redundancy

RHEL 5.4+, Ubuntu 10.0.4, Fedora 16

Replication

Managing Complexity

The Three C’s of Complexity

• Control

• Choice

• Compliance

Compute

Giving Control Brings Complexity

Network

Storage

Admin

Users

Org A

Admin

Users

Org B

Users

End User

Admin

VM Ware

Xen Server

KVM

NFS

iSCSI

FC

Net Scaler

F5

Jun. SRX

Local Disk

Cisco ASA

Swift

HDFS

Hyper-V

• ACL • Limits • Governance

Oracle VM

Bare Metal

Guest Virtual Layer-2 Network

Guest 1 VM 1

Guest 1 VM 2

Guest 1 VM 3

Guest 1 Virtual Network 10.1.1.0/24

Gateway 10.1.1.1

Guest 10.1.1.2

Guest 10.1.1.3

Guest 10.1.1.4

Guest 1 Virtual Router

Guest 2 VM 1

Guest 2 VM 2

Guest 2 VM 3

Guest 2 Virtual Network 10.1.1.0/24

Gateway 10.1.1.1

Guest 10.1.1.2

Guest 10.1.1.3

Guest 10.1.1.4

Guest 2 Virtual Router

Public IP 65.37.141.24 65.37.141.80

Public IP 65.37.141.11 65.37.141.36

Internet

Multi-tier Network

Private IP 10.1.1.112

DHCP, DNS User-data

Public IP 65.37.141.112

10.1.1.1

Web VM 1

10.1.1.3

Web VM 2

10.1.1.4

Web VM 3

10.1.1.5

Web VM 4

Netscaler Load

Balancer

Private IP 10.1.1.111

Public IP 65.37.141.111

Juniper SRX

Firewall

Virtual Router

Virtual Network 10.1.1.0/24 VLAN 100


10.1.2.21

10.1.2.18

10.1.2.38

10.1.2.39

10.1.2.31

App VM 1 10.1.3.21


10.1.2.24

App VM 2 10.1.3.45

10.1.3.24

DB VM 1

DHCP, DNS, User-data

DHCP, DNS User-data, Source-NAT, VPN

Public IP 65.37.141.115

Virtual Router

Virtual Router

Unified Multi-tier Network

10.1.1.1

Web VM 1

10.1.1.3

Web VM 2

10.1.1.4

Web VM 3

10.1.1.5

Web VM 4



10.1.2.31

App VM 1


10.1.2.24

App VM 2

10.1.3.24

DB VM 1

Virtual Router

Customer Premises

IPSec or SSL site-to-site VPN

Internet

Monitoring VLAN

Virtual Router Services • IPAM • DNS • LB [intra] • S-2-S VPN • Static Routes • ACLs • NAT, PF • FW [ingress & egress] • BGP

Load Balancer

Other Topologies

Guest Virtual Network 10.1.1.0/24 VLAN 100

Gateway address 10.1.1.1

10.1.1.1

Guest VM 1

10.1.1.3

Guest VM 2

10.1.1.4

Guest VM 3

10.1.1.5

Guest VM 4


DHCP, DNS User-data

10.1.1.1

Guest VM 1

10.1.1.3

Guest VM 2

10.1.1.4

Guest VM 3

10.1.1.5

Guest VM 4

No services [Static IPs] Dedicated VLAN with DHCP and DNS User can request specific IP[s] for NIC

Core switch


Core switch

Virtual Router

Other Topologies



10.1.1.100

Guest VM 1

10.1.1.200

Guest VM 2

10.1.1.101

Guest VM 3

10.1.1.115

Guest VM 4


DHCP, DNS User-data

10.1.1.1

Guest VM 1

10.1.1.3

Guest VM 2

10.1.1.4

Guest VM 3

10.1.1.5

Guest VM 4

MPLS Use Case Shared VLAN with DHCP and DNS

CS Virtual Router

Core switch


Core switch

MPLS VLAN 100

DHCP, DNS User-data

CS Virtual Router

…

DB Security Group

Web Security Group

Layer 3 Networking (Amazon Style)

… …

Web VM

Web VM

Web VM

Web VM

DB VM

Web VM

DB VM

Web VM

Software Architecture

Management Server

Orchestration Engine - Drives long running VM

operations - Syncs between resources

managed and DB - Generates events

Resource Management

Cluster Management

Job Management

DB

UI Cloud Portal

CLI Other Clients

Deployment Planning

Network Gurus

Network Elements

Hypervisor Gurus

Database Access

Alert & Event Management

Plu

gin

AP

I

Resource API

Hypervisor Resources

Network Resources

Storage Resources

Image Resources

Snapshot Resources

REST API

OAM&P API End User API EC2 API Pluggable Service API Engine Other APIs

Security Adapters

Account Management Connectors

ACL & Authentication - Accounts, Domains, and Projects - ACL, limits checking

Services API

Serv

ices

AP

I

Console Proxy Management

Template Access

HA

Usage Calculations

Additional Services

Event Bus

Message Bus Usage Server

Orchestration Engine

• Understands how to orchestrate long running processes (i.e. VM starts, Snapshot copies, Template propagation)

• Well defined process steps

• Calls Plugin API to execute functionalities that it needs

Plugins

• Various ways to add more capability to CloudStack

• Implements clearly defined interfaces

• All operations must be idempotent

• All calls are at transaction boundaries

• Compiles only against the Plugin API module

Anatomy of a Plugin

ServerResource - Optional. Required if Plugin needs to be co-

located with the resource - Implements translation layer to talk to

resource - Communicates with server component via

JSON

Rest API - Optional. Required only if needs to expose

configuration API to admin.

Plu

gin

AP

I

Data Access Layer

Implementation

• Can be two jars: server component to be deployed on management server and an optional ServerResource component to be deployed co-located with the resource

• Server component can implement multiple Plugin APIs to add its feature

• Can expose its own API through Pluggable Service so administrators can configure the plugin

• As an example, OVS plugin actually implements both NetworkGuru and NetworkElement

Plugin Interfaces Available

• NetworkGuru – Implements various network isolation and ip address technologies

• NetworkElement – Facilitate network services on network elements to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)

• DeploymentPlanner – Different algorithms to place a VM and volumes.

• Investigator – Ways to find out if a host is down or VM is down. • Fencer – Ways to fence off a VM if the state is unknown • UserAuthenticator – Methods of authenticating a user • SecurityChecker – ACL access • HostAllocator – Provides different ways to allocate host • StoragePoolAllocator – Provides different ways to allocate volumes

Separating Data and Control

Data Center 1

Cloud

Data Center 2

Data Center 3

Management

Server

Management Servers control all resources, both virtual and physical

SSVMs deployed to transfer data between zones

CPVMs deployed to transfer VNC console traffic

VR deployed for traffic into public internet

Management Server is never in the data path

SSVM

SSVM

SSVM Transfer of Templates,

ISOs, Snapshots

CPVM CPVM

CPVM

VR

VR

VR

Internet

Kernel

Sequence Flow for VM Creation

End User Rest API

Security Checkers

User VM Mgr

Network Mgr

Storage Mgr

Job Scheduling

VirtualMachine Mgr

Network Guru

Deploy VM

ACL Checks

Allocate Entity in CS

Allocate VM

Allocate NIC

Allocate Volume

Allocate IP

Schedules Deploy Job

Returns with job id, VM id

Query Job Result

Returns with job status

Sequence Flow for VM Creation

Job Threads Network Element

User VM Mgr

Network Mgr

Storage Mgr

VirtualMachine Mgr

Network Guru

Start VM

Start VM

Prepare Nics

Notify that Nic is about to be started in network

Reserve resources for Nic

Services API Server

Resources

Start User VM

Agent Calls

Prepare Volumes

Template Mgr

Deployment

Planner

Get a Deployment Plan (Host and StoragePool)

Prepare template on Primary Storage

Agent Calls

Agent Start VM Call

Stores job result

Conclusion

Design Goals for CloudStack

• Design for complexity

– Clear interfaces

• Design for scalability

– Separate out data path and control paths

– Design to maximize the use of database connections

• Design against failure

– Provide clear boundaries (process and compilation)

– Utilize cloud administrator to give guidance

More Information

28

• http://cloudstack.org

• Apache mailing lists

–[email protected]

–[email protected]

• Thank you

http://cloudstack.org/

mailto:[email protected]










2 architectural at cloudstack developer day

Technology

cs virtual

user vm mgr

management

virtual router

management

start vm

server component

db vm 1