cloudstack networking

CloudStack Networking

Chiradeep VittalMay 2 2012

Outline

• CloudStack Networking Features• CloudStack Networking Configuration• CloudStack Networking APIs• CloudStack Network Architecture• Virtual Router deep dive

Feature overview• Orchestration of L2 – L7 network services

– IPAM, DNS, Gateway, Firewall, NAT, LB, VPN, etc• Mix-and-match services and providers• Out-of-the-box integration with automated deployment of virtual routers

– Highly available network services using CloudStack HA and VRRP• Orchestrate external providers such as hardware firewalls and load

balancers– Devices can provide multiple services– Admin API to configure external devices– Plugin-based extensions for network behavior and admin API extensions

• Multiple multi-tenancy [network isolation] options• Integrated traffic accounting• Access control • Software Defined Networking too

Basic vs Advanced Networking

• Segmentation based on feature set and ease-of-deployment

• Both are feature-rich• Basic implements true AWS-style L3-isolation

– Tenants do not get contiguous IP addresses or subnets– Network segmentation based on Security Groups– Tremendous scale (tens of thousands)

• Advanced Zone offers full L3 subnets– VLANs are default implementation (4K limit)– More features (source NAT, PF, VPN)

CloudStack Terminology

• Guest network– The tenant network to which instances are attached

• Storage network– The physical network which connects the hypervisor to primary storage

• Management network– Control Plane traffic between CloudStack management server and hypervisor clusters

• Public network– “Outside” the cloud [usually Internet]– Shared public VLANs trunked down to all hypervisors

• All traffic can be multiplexed on to the same underlying physical network using VLANs

– Usually Management network is untagged– Storage network usually on separate nic (or bond)

• Admin informs CloudStack how to map these network types to the underlying physical network

– Configure traffic labels on the hypervisor– Configure traffic labels on Admin UI

Storage 1

Hypervisor 1

Hypervisor N

Hypervisor 8

Access Switch(es)CloudstackServer

VM Traffic

Control Plane Traffic

Storage Traffic

CloudstackServers

Storage k

…

Pod 1

CLUSTER 1

…

CLUSTER 4

Core (L3) Network

…

Pod 2 Pod N

PHYSICAL NETWORK IN A ZONE

Storage 2

Hypervisor N+1

Public Traffic

L2 Features• Choice of network isolation

– Physical, VLAN, L3 (anti-spoof), Overlay[GRE]– Physical isolation through network labels [limited to # of nics or bonds]

• Multi-nic– Deploy instance in multiple networks– Control default route

• Access control– Shared networks, project networks– Dedicated VLANs offer MPLS integration

• Anti-spoofing for L3-isolated networks• QoS [max rate]• Traffic monitoring• Broadcast & multicast suppression in L3-isolated networks• Hot-plug / detach of nics [upcoming]

L3 Features• IPAM [DHCP], Public IP address management

– VR acts as DHCP server– Can request multiple public IPs per tenant

• Gateway (default gateway)– Redundant VR (using VRRP)– Inter-subnet routing [upcoming]– Static routing control [upcoming]

• Remote Access VPN– L2TP over IPSec using PSK– Virtual Router only

• Firewall based on source cidr• Static NAT [1:1]

– Including “Elastic IP” in Basic Zone• Source NAT

– Per-network, or interface NAT• Public Traffic usage

– Monitoring on the Virtual Router / External network device – Integration with sFlow collectors

• Site-to-Site VPN [upcoming]– IPSec VPN based on VR

• L3 ACLs [upcoming]

L4 Features

• Security groups for L3-isolation– “Basic Zone” in docs– Default AWS-style networking– Scales much better than VLANs

• Stateful firewall for TCP, UDP and ICMP• Port forwarding [“Advanced Zone”]

– Conserve public Ips

L7 features• Loadbalancer

– VR has HAProxy built in– External Loadbalancer support

• Netscaler (MPX/SDX/VPX)• F5 BigIP• Can dedicate an LB appliance to an account or share it among tenants

– Loadbalancer supported with L3-isolation as well– Stickiness support– SSL support [future]– Health Checks [future]

• User-data & meta-data– Fetched from virtual router

• Password change server

Router

L3 Core Switch

Access Layer

Switches

………… …

Availability Zone

Servers

CloudStack Mgmt Server Cluster

Secondary Storage

Pod 1 Pod 2 Pod 3 Pod N

MySQL

Load Balancer

Operations Admin and Cloud API

Users

Physical Network

…

DB Security Group

WebSecurity Group

Layer 3 cloud networking

… …

Web VM

Web VM

Web VM

Web VM

DB VM

Web VM

DB VM

Web VM

Guest Networks with L3 isolationGuest

1 VM 1

Guest 2 VM 1

Guest 1 VM 2

Guest 2 VM 2

Public Internet

10.1.0.1

Public IP address 65.37.141.1165.37.141.2465.37.141.3665.37.141.80

Guest address 10.1.0.2Guest address 10.1.0.3Guest address 10.1.0.4

Guest address 10.1.16.12

Load Balancer

Guest 2 VM 3

Guest 1 VM 3

Guest 1 VM 4


L3 Core Switch

Pod 1 L2 Switch

Pod 3 L2 Switch

10.1.16.1

…

…10.1.8.1Pod 2 L2 Switch

Hypervisor 1

Hypervisor N

Hypervisor 8

Access Switch(es)

VM Traffic

…

Pod K

CLUSTER 1

…

CLUSTER 4

Core (L3) Network

…

Pod M Pod N

Virtual Networks (L2 isolation)

Hypervisor N+1

Public Traffic

Hypervisor

R

R V

VV

V

HypervisorV V

V

RTenant VMTenant Virtual Router

Guest virtual layer-2 network

Guest 1 VM 1

Guest 1 VM 2

Guest 1 VM 3

Guest 1 VM 4

Public Internet

Public Network

Guest Virtual Network 10.1.1.0/24

Gateway address 10.1.1.1

NATDHCPLoad BalancingVPN

Public IP address 65.37.141.1165.37.141.36

Guest address 10.1.1.2Guest address 10.1.1.3Guest address 10.1.1.4Guest address 10.1.1.5

Guest 1 Virtual Router

Guest 2 VM 1

Guest 2 VM 2

Guest 2 VM 3

Guest Virtual Network 10.1.1.0/24


NATDHCPLoad BalancingVPN


Guest 2 Virtual Router

Public IP address 65.37.141.2465.37.141.80

Layer-2 Guest Virtual Network

Public Network/Internet

Guest Virtual Network 10.1.1.1/8VLAN 100


DHCP, DNSNATLoad BalancingVPN

Public IP 65.37.141.11

10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4

CSVirtual Router



Private IP10.1.1.112

DHCP, DNS

Public IP 65.37.141.112

10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4

NetScalerLoad

Blancer


Public IP 65.37.141.111

Juniper SRX

Firewall

CS Virtual Router provides Network Services External Devices provide Network Services

CSVirtual Router

Other Topologies



10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4


DHCP, DNSUser-data

10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4

No services [Static Ips] Dedicated VLAN with DHCP and DNSUser can request specific IP[s] for NIC

CSVirtual Router

Core switch


Core switch

Other topologies



10.1.1.100Guest VM 1

10.1.1.200Guest VM 2

10.1.1.101Guest VM 3

10.1.1.115

Guest VM 4


DHCP, DNSUser-data

10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4

MPLS Shared VLAN with DHCP and DNS

CSVirtual Router

Core switch


Core switch

MPLS VLAN 100

DHCP, DNSUser-data

CSVirtual Router

Multi-tier network



DHCP, DNSUser-data

Public IP 65.37.141.112

10.1.1.1Web VM

1

10.1.1.3Web VM

2

10.1.1.4Web VM

3

10.1.1.5Web VM

4

NetscalerLoad

Balancer


Public IP 65.37.141.111

Juniper SRX

Firewall

Multi-tier network

CSVirtual Router

CSVirtual Router

Virtual Network 10.1.1.0/24VLAN 100


10.1.2.21

10.1.2.18

10.1.2.38

10.1.2.39

10.1.2.31App VM

1 10.1.3.21


10.1.2.24App VM

2 10.1.3.45

10.1.3.24 DB VM 1

CSVirtual Router

DHCP, DNS, User-data

DHCP, DNSUser-data,Source-NAT, VPN

Public IP 65.37.141.115

VM VM VM

VR

Your Routing

VM

Public VLAN(s)

Guest VLAN

Monitoring VLAN(shared)

Bring-your-own Service

Customer installs static route to point to his routing vm

VM VM VM

VR

Your Routing

VM

Public VLAN(s)

Guest VLAN

Shared Public VLAN

Bring-your-own Service[site-to-site-vpn]

Customer installs static route (manually/automated config) to point to his routing vm. Routing VM provides Site-to-site VPN (configured directly on routing VM, not by CloudStack)

Multi-tier unified [vision]

10.1.1.1Web VM

1

10.1.1.3Web VM

2

10.1.1.4Web VM

3

10.1.1.5Web VM

4



10.1.2.31App VM

1


10.1.2.24App VM

2

10.1.3.24 DB VM 1

CSVirtual Router /

OtherCustomerPremises

IPSec or SSL site-to-site VPN

Internet

Monitoring VLAN

Virtual Router Services• IPAM• DNS• LB [intra]• S-2-S VPN• Static Routes• ACLs• NAT, PF• FW [ingress & egress]• BGP

Loadbalancer

Multi-tier unified with SDN[vision]

10.1.1.1Web VM

1

10.1.1.3Web VM

2

10.1.1.4Web VM

3

10.1.1.5Web VM

4

Overlay Network 10.1.1.0/24


10.1.2.31App VM

1


10.1.2.24App VM

2

10.1.3.24 DB VM 1

CSVirtual Router /

OtherCustomerPremises

IPSec or SSL site-to-site VPN

Internet

Monitoring VLAN

Virtual Router Services• IPAM• DNS• LB [intra]• S-2-S VPN• Static Routes• ACLs• NAT, PF• FW [ingress & egress]• BGP

LoadbalancerVirtual Appliance

• Cloud provider defines the feature set for guest networks

• Toggle features or service levels– Security groups on/off– Load balancer on/off– Load balancer software/hardware– VPN, firewall, port forwarding

• User chooses network offering when creating network

• Enables upgrade between network offerings

• Default offerings built-in– For classic CloudStack networking

Network Offerings

CPU Cores

CPU (MHz)

Memory (MB)

Name

Compute

Specify Resource Levels

Service Offerings

Custom Disk Size

Disk Size (GB)

Storage Tag

Public

Name

Disk

Network Rate

Redundant VR

Public

Name

Network

Firewall

Load balancer

CPU Cap

Host Tag

Enable HA

Configure Properties

Public

Define Scope

CloudStack Network Service Providers

• A Network Service Provider is hardware or virtual appliance that makes a network service possible in CloudStack ; for example, a Citrix NetScaler appliance can be installed in the cloud to provide Load-Balancing services.

• Administrators can have multiple instances of the same service provider in a network; for example, more than one Citrix NetScaler or Juniper SRX device can be added to CloudStack

• CloudStack supports the following Network Providers:

– CloudStack Virtual Router (default)– Citrix NetScaler SDX, VPX and MPX models– Juniper SRX– F5 BigIP

Adding an Additional Network Offerings

Network Offering Status

Network Offering Order control

Network Service Providers Matrix

Feature Virtual Router Citrix NetScaler

Juniper SRX F5 BigIP

Remote Access VPN YES N/A N/A N/AFirewall YES N/A YES N/ASource NAT YES N/A YES N/AStatic NAT YES YES YES N/ALoad Balancing YES YES N/A YESPort Forwarding YES N/A YES N/AElastic IP N/A YES N/A N/AElastic LB N/A YES N/A N/ADHCP/DNS/User Data YES N/A N/A N/A

• Network offerings is basically a definition of what Network Services are available when this offering is used. The available Network Services are: VPN, DHCP, DNS, Firewall, Load Balancer, User Data, Source NAT, Static NAT, Port Forwarding and Security Groups*

CloudStack User APIs [sample]

• Networks (L2)– createNetwork [requires network offering id], – deleteNetwork (A), listNetworks, – restartNetwork (A): restarts all devices (if allowed)

supporting the network and re-applies configuration

– updateNetwork: update network offering and restart network

Adding a Shared Guest Network

• Only Administrators can add a Shared Guest Network for an Advanced zone

Adding a Shared Guest Network

VLAN required!

Editing Guest Networks

When editing a guest network users can change the network offering. They can either upgrade to a “premium” network offering (for example offering that uses hardware Load-balancer) or downgrade to a “cheaper” network.

• Restarting the network will simply resend all the LB, Firewall and Port-Forwarding rules to the network provider

• Restarting the Network with “Clean up”:

• restarting network elements - virtual routers, DHCP servers

• If virtual router is used, it will be destroyed and recreated

• Reapplying all public IPs to the network provider

• Reapplying load-Balancing/Port-Forwarding/Firewall rules

Restarting and Cleaning Up a Guest Network

• An Isolated Guest Network can only be deleted if no VMs are using these network (e.g. Completely destroyed and expunged)

• Deleting a Network will Destroy the Virtual Router (if used) and will release the Public IPs back to the IP Pool

Deleting a Guest Network

Extending CloudStack Networking

Network Manager

Network Element

DnsService

MyDnsElementMyDnsDeviceMa

nager

MyDnsDeviceService

PluggableService

MyDnsDeviceResource

AgentManager Queue

1. prepare (part of start vm)2. prepare (Network, Nic, DeployDestination, VmInfo)

3. addDnsRecord(ip, fqdn)

4.Enqueue AddDnsRecord

5.API call to Dns Device

Device ConfigurationAdmin API (CRUD)

MySQL

Needs to be added as of 5/2/2012

Demonstrates one way to inform an external DNS server when an instance starts.Classes shaded blue form a plugin / service bundle to integrate an external DNS server. Clients of the instance can then use DNS names to access the instance.

CloudStack Virtual Router (Virtual Router)

• The Virtual Router will be deployed once (when the first instance is deployed in a Zone) when a Shared Network is used providing DHCP and DNS services for the Zone’s Instances (IPs will be allocated from the Public IP Range entered in CloudStack)

• When Advanced is used the Router will be deployed Per-Account (and Per Unique Isolated Guest Network)

• Virtual Router can serve and isolate VMs even if deployed on a different Hypervisor

CloudStack Virtual Router

• The Virtual Router will have 3 NICs:– Eth0 will be connected to the Isolated Guest Network (for Advanced VLAN). It will have the first IP in

the CIDR (for example10.1.1.1) and it will be the DNS, DHCP and Gateway for the Instances in the Private Guest Network.

– Eth1 resides on local-link network (only for KVM and XenServer) or the Management Network (on VMware) and is used by CloudStack to configure the virtual router. On VMware it will use an IPs from the Management Network IP Range (e.g. Pod Private Range)

– Eth2 resides on the Public Network and assigned with a Public IP from the range entered in CloudStack (users can ‘Acquire New IPs’ if needed)

• In the default Isolated Mode - Source NAT is automatically configured on the virtual router to forward outbound traffic for all guest VMs and block all incoming traffic (users can manage incoming rules from UI)

Virtual Router Information (applies to all Sys. VMs)

• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security APT repository. No extraneous accounts

• 32-bit for enhanced performance on Xen/VMWare• Only essential software packages are installed. Services such as, printing, ftp, telnet, X, kudzu,

dns, sendmail are not installed.• SSHd only listens on the private/link-local interface. SSH port has been changed to a non-

standard port. SSH logins only using keys (keys are generated at install time and are unique for every customer)

• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum performance on all hypervisors. Xen tools inclusion allows performance monitoring

• Template is built from scratch and is not polluted with any old logs or history• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved

security and speed• Latest version of jre from Oracle ensures improved security and speed

cloudstack networking

Technology