cloudstack technical overview
DESCRIPTION
Technical overview on deployment scenarios and architecture. Given at Bangalore and Hyderabad CloudStack meet ups in October.TRANSCRIPT
Kevin KlugeVice President, Cloud Platforms Group, Citrix Systems Inc.
Build your own Infrastructure Cloud with Apache CloudStack
Kevin is an expert in Large Scale Systems and Infrastructure Clouds and manages the Cloud Platforms product group at Citrix.
Previously Vice President, Engineering at Cloud.com, acquired by Citrix in 2011.
Held engineering leadership positions at Yahoo!, Zimbra, Corvigo, Openwave Systems, and Onebox.com.
Kevin has a MS and BS in Computer Science from Stanford University.
Apache CloudStack (Incubating)An Introduction
Kevin Kluge
Apache CloudStack Committer
• Create VMs, disks networks, network services
• Self service• Meter usage
Use CloudStack to build IaaS clouds (like EC2)
• Java based• Scalable• Many vendor integrations• Native and EC2 API
How did Amazon build EC2?
Commodity Servers
Commodity Storage
Networking
Open Source Xen Hypervisor
Amazon Orchestration Software
AWS API (EC2, S3, …)
Amazon eCommerce Platform
How can you build your cloud?
Servers StorageNetworking
Open Source Xen Hypervisor
Amazon Orchestration Software
AWS API (EC2, S3, …)
Amazon eCommerce Platform
ESXi, KVM, XenServer/XCP, OVM
CloudStack Orchestration Software
Your Portal (Optional)
CloudStack or AWS API
Project history
• 2008/2009: closed-source development• First deployments in late 2009
• May 2010: ~98% open source as GPLv3 (open core) • August 2011: 100% open source GPLv3
• April 2012: Switch to Apache License v2• Submit code to Apache Software Foundation
Project current state
• In incubation within Apache Software Foundation
• Imminent first release!
• Bugs and wiki mostly moved to ASF infra
• Mailing list traffic moved to ASF infra
• Many non-Citrix contributors, committers, and PPMC members
Yes, the ASF is great
Enter ASF
IaaS Cloud Concepts
Cloud
Built for traditional enterprise apps & client-server compute
•Scale-up (pool-based resourcing)•IT management-centric •1 administrator for 100’s of servers•Proprietary vendor stack
Designed around big data, massive scale & next-gen apps• Scale-out (horizontal resourcing)• Autonomic management • 1 administrator for 1,000’s of servers• Open, value-added stack
Virtualization alone does not make a cloud
Server Virtualization
Design for failure
Self-service recovery
Multi-site redundancy
Ephemeral resources
Cloud Workload
Think Amazon Web Services
Expect reliability
Back-up everything
HA, Fault tolerance
Admin control recovery
Traditional Workload
Think Server Virtualization
Clouds must reliably run all types of workloads
Object Storage
vSphere
ESXi Cluster
ESXi Cluster
ESXi Cluster
Enterprise Networking (e.g., VLAN)
Enterprise Storage (e.g., SAN)
Cloud-era Availability
Zone
Cloud-era Availability
Zone
Cloud-era Availability
Zone
Traditional ZoneCloudStack Mgmt
Server
Cloud-era Workloads Traditional Workloads
Embrace traditional and extend to Cloud-era
Cloud-era Availability
Zone
Cloud-era Availability
Zone
Traditional Availability
Zone
Apache CloudStackManagement Server
Traditional Availability
Zone
Traditional Availability
Zone
Availability Zone
Availability Zone
Availability Zone
Amazon-Style Cloud
Object store is critical for Cloud-era workloads
CloudStack Mgmt. Server • Workloads are distributed across
availability zones
• No guarantee on zone reliability
• DBs and Templates snapped to object store.
• For small failures, recreate instance in same zone
• For DR, recreate instance in different zone
• Dramatically less expensive
Object Store
Deployment and Software Architecture
Management Server managing multiple zones
Zone1
Data Center 1
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 2
Zone 3
Zone 2
Data Center 3
Zone 4
Management Servers
• Single Management Server can manage multiple zones
• Zones can be geographically distributed but low latency links are expected for better performance
• Single MS node can manage up to 10K hosts.
• Multiple MS nodes can be deployed as cluster for scale or redundancy
Site-to-Site VPN
Router
L3 Core Switch
Top of Rack Switch
………… …Availability Zone 1
Servers
Primary Mgmt Server Cluster
Object Store
Pod 1 Pod 2 Pod 3 Pod N
Primary MySQL
Load Balancer
Admin Internet
Availability Zone 2
Backup MySQL
Standby Mgmt Server Cluster
DB Replication
Cloud-era zone deployment
10Gbps Storage & Mgmt
1Gbps Guest
10Gbps Storage & Mgmt
1Gbps Guest
10Gbps Storage & Mgmt
1Gbps Guest
…
Load Balancer
Core Switch
Aggregation Switch
TOR Switch
Compute Nodes
NFS Primary Storage
Object Store
Pod 1
Pod 2
Pod 200
InternetTraditional zone deployment
Management Server internals and service VMs
Management Server interaction with hypervisors
Management Server
XenServer
ESX
vCenter
KVM
Agent
OVM
Agent
XAPI HTTP
• XS 5.6, 5.6FP1, 5.6 SP2, 6.0.2, XCP 1.1
• Incremental Snapshots• VHD• NFS, iSCSI, FC & Local
disk• Storage over-
provisioning: NFS
• ESX 4.1, 5.0 • Full Snapshots• VMDK• NFS, iSCSI, FC & Local disk• Storage over-provisioning:
NFS, iSCSI
• RHEL 6.0, 6.1, 6.2, Ubuntu 12.04
• Full Snapshots (not live)• QCOW2• NFS, iSCSI & FC• Storage over-
provisioning: NFS
• OVM 2.2• No Snapshots• RAW• NFS & iSCSi• No storage over-
provisioning
XCP
Mgm
t S
erve
r C
PU
U
til.
Sec
ond
s to
de
plo
yScalability to 30,000 hosts
25,000 …. to …. 30,000 VMs 0 …. to …. 30,000 VMs
• Simulator developed to test massive scale• Four Management Servers can manage 30,000 hosts• Scale to hundreds of thousands of hosts possible with
multiple management server clusters (regions)
Features
Open platform – vendor agnostic
Compute
XCP/XS VMware KVMOracle VM Bare metal
Hypervisor
Storage
Local Disk iSCSI NFSFiber Channel
Object Stores
Block & Object
Network
Network Type Isolation Load
balancerFirewall VPN
Network & Network Services
Virtual machine management
Users
Start
Stop
Restart
Destroy
VM Operations Console Access
• CPU Utilized
• Network Read
• Network Writes
VM StatusChange
Service Offering
2 CPUs
1 GB RAM
20 GB
20 Mbps
4 CPUs
4 GB RAM
200 GB
100 Mbps
Volume and snapshot management
Volume
VM 1Add / DeleteVolumes
Schedule Snapshots
Hourly
Daily
Weekly
MonthlyNow
Create Templates from Volumes
Volume Template
View Snapshot History 12/2/2012 7.30 am
….2/2/2012 7.30 am
CPU Cores
CPU (MHz)
Memory (MB)
Name
Compute
Specify Resource Levels
Service offerings
Custom Disk Size
Disk Size (GB)
Storage Tag
Storage Tag
Public
Name
Disk
Network Rate
Redundant VR
Public
Name
Network
Firewall
Load balancer
CPU Cap
Host Tag
Enable HA
Configure Properties
Public
Define Scope
Multi-tenancy and account management
• Domain is a unit of isolation that represents a customer org, business unit or a reseller
• Domain can have arbitrary levels of sub-domains
• A Domain can have one or more accounts
• An Account represents one or more users and is the basic unit of isolation
• Admin can limit resources at the Account or Domain levels
Admin
Org A
Admin
Reseller A
Domain
Domain
Admin
Org C
Sub-Domain
User 1
User 2
Group B
Account
Group A
Account
VMs, IPs, Snapshots…
VMs, IPs, Snapshots…
Resources
Resources
Network and network services
• Create Networks and attach VMs
• Acquire public IP address for NAT & load balancing
• Control traffic to VM using ingress and egress firewall rules
• Set up rules to load balance traffic between VMs
Zone N
• Provides cloud operator defined service features• Isolation• Load Balancing• VPN• Firewall
• Supports Physical Devices• NetScaler• F5 BIG-IP• Juniper SRX
Network offering
Pod NPod 1
Zone 1
Pod 1
Layer-3 guest network
Public Network65.11.0.0/16
65.11.1.2 Guest VM 1
Guest VM 2
Guest VM 3
Guest VM 4
Public Network/Internet
PhysicalLoad
Balancer
Network Services Managed Externally Network Services Managed by CS
65.11.1.3
65.11.1.4
65.11.1.5
DHCP, DNS
CSVirtual Router
Security Group 1
Security Group 2
65.11.1.2 Guest VM 1
Guest VM 2
Guest VM 3
Guest VM 4
65.11.1.3
65.11.1.4
65.11.1.5
DHCP, DNS
CSVirtual Router
Security Group 1
Security Group 2
EIP, ELB
Layer-2 guest virtual network
Public Network/Internet
Guest Virtual Network 10.0.0.0/8VLAN 100
Gateway address 10.1.1.1
DHCP, DNSNATLoad BalancingVPN
6.37..1.1110.1.1.1
Guest VM 1
10.1.1.3
Guest VM 2
10.1.1.4
Guest VM 3
10.1.1.5
Guest VM 4
CSVirtual Router
Public Network/Internet
Guest Virtual Network 10.0.0.0/8VLAN 100
Private IP10.1.1.112
DHCP, DNS
Public IP 6.37.1.11
10.1.1.1
Guest VM 1
10.1.1.3
Guest VM 2
10.1.1.4
Guest VM 3
10.1.1.5
Guest VM 4
PhysicalLoad
Balancer
Private IP10.1.1.111
Public IP 6.37.1.12 Juniper
SRXFirewall
CS Virtual Router provides Network Services External Devices provide Network Services
CSVirtual Router
Comparison of guest network options
Layer-2 Layer-3Isolation VLAN/SDN Security GroupsPerformance Better BetterNetwork setup Moderate EasySupport broadcast Yes NoScalability Good BestInteroperability with physical servers
Good Poor
Pod 1
Host 2
Cluster 1
Host 1
PrimaryStorage
L3 switch
SecondaryStorage
L2 switch
CloudStack storage
• Configured at Cluster-level. Close to hosts for better performance
• Stores all disk volumes for VMs in a cluster• Cluster can have one or more primary
storages• Local disk, iSCSI, FC or NFS
Primary Storage
• Configured at Zone-level• Stores all Templates, ISOs and Snapshots• Zone can have one or more secondary
storages• NFS, OpenStack Swift, others coming
Secondary Storage
Futures
Apache CloudStack API
SwitchesHypervisor
Apache CloudStack API
FirewallLoad Bal
Baremetal Security
Apache CloudStack APIApache CloudStack API
Storage
Expanding orchestration control
Futures
• Object storage and SDN short term
• Blade orchestration
• Region support
• Additional hypervisors (need some container support)
• Code modularity improvements (OSGI?)
• App-specific integration (Hadoop?)
• Improved CLI
• Additional API support (Google, evolving standards)
The future needs you!
Project web site: http://incubator.apache.org/projects/cloudstack.html
Mailing lists:
IRC: #CloudStack on irc.freenode.net
Join your local CloudStack group!
Thank You