ARCHITECTING AND BUILDING A

SECURE MULTI-TENANT CLOUD

FOR SAAS APPLICATIONS

Dilip Sundarraj

Cloud Solutions Architect, Juniper Networks

April 8th, 2015

Symantec CloudFire

What is Network Virtualization?

• Independent of Physical Network Location or State

• Logical Network across any server, any rack, any cluster, any data-center

• Virtual Machines can migrate without requiring any reworking of security

policies, load balancing, etc

• New Workloads or Networks should not require provisioning of physical network

• Nodes in Physical Network can fail without any disruption to Workload

• Full Isolation for Multi-tenancy and Fault Tolerance

• MAC and IP Addresses are completely private per tenant

• Any failures or configuration errors by tenants do not affect other applications or

tenants

• Any failures in the virtual layer do not propagate to physical layer

OpenContrail

• OpenSource Network Virtualization Platform for Cloud

• Primary Use Cases:

• Cloud Networking

– IaaS, VPCs for Cloud SP, Private Cloud for Enterprises or SPs

• NFV in SP networks

– Value added services for SP edge networks.

OPENCONTRAIL

ARCHITECTURE

Analytics

CONTRAIL CONTROLLER

ControlConfiguration

x86 Host + Hypervisor

ORCHESTRATOR

x86 Host + Hypervisor

Physical IP Network

(no changes)

vRoutervRouter

Gateway

Internet / WANLegacy Infra.

(VLAN, etc.)

Bi-directional real-time message bus using XMPP

Network orchestration

Standard protocol (M-

BGP) to talk with other

Contrail controller

instances

Compute / Storage

orchestration

Accepts and converts

orchestrator requests for

VM creation, translates

requests, and creates

network

Interacts with network

elements for VM network

provisioning and ensures

uptime

Real-time analytics

engine collects, stores

and analyzes network

elements

vRouter: Virtualized routing element handles

localized control plane and forwarding plane

work on the compute node

Gateway: MX Series (or other router)

serve as gateway improving scale &

performance

OpenContrail <-> OpenStack

Openstack integration

Horizon

Nova API

Compute

Driver

Virtual-IF

Driver

Nova Compute

Contrail

Agent

vRouter

(kernel)

Virtual Router

Nova

Scheduler

Neutron

Driver

Neutron

Plugin

Configuration

Node

Control

Node

1Create an Instance (VM Info,

Network, IPAM, Policies, etc)

2Schedule an Instance on the

Compute Node

3VM Network

Properties

4Create VM

Interface

6 Publish VM

Intf on IFMap

5 Add Port

7VM Interface

Config over XMPP

Scripts

OpenContrail – Control Node

• All Control Plane Nodes are active

active

• Each vRouter uses XMPP to connect

with multiple Control Plane nodes for

redundancy

• Each Control Plane Node connects

to multiple configuration nodes for

redundancy

• Control Plane Nodes federate using

BGP

Control Node

"BGP module"

ProxiesXMPP

ControlNode

Control Node

Compute Node Compute Node

Configuration Node

Configuration Node

IF-MAP

XMPP

IBGP

IF-MAP Client

GatewayRouters

Service Nodes

OpenContrail - Compute Node

Compute node – Hypervisor, vRouterCompute Node

VirtualMachine

(Tenant B)

VirtualMachine

(Tenant C)

VirtualMachine

(Tenant C)

vRouter Forwarding Plane

VirtualMachine

(Tenant A)

Routing Instance

(Tenant A)

Routing Instance

(Tenant B)

Routing Instance

(Tenant C)

vRouter Agent

Flow Table

FIB

Flow Table

FIB

Flow Table

FIB

Overlay tunnelsMPLS over GRE or VXLAN

JUNOSV CONTRAIL CONTROLLERJUNOSV CONTRAIL CONTROLLER

XMPP

Eth1Kernel

Tap Interfaces (vif)

pkt0

UserEth0 EthN

Config

VRFsPolicy Table

Top of Rack Switch

XMPP

Compute node – Forwarding/Tunneling

Overlay tunnelsMPLS over GRE or VXLAN

Compute Node

vRouter Forwarding Plane

VirtualMachine(VN-IP1)

Routing Instance

Flow Table

FIB

Eth1 (Phy-IP1)

Tap Interfaces (vif)

Compute Node

vRouter Forwarding Plane

VirtualMachine(VN-IP2)

Routing Instance

Flow Table

FIB

Eth1 (Phy-IP2)

Tap Interfaces (vif)

VIRTUAL

PHYSICAL

Virtual-IP2

Payload

Virtual-IP2

Payload

MPLS / VNI

Phy-IP2

Virtual-IP2

Payload

Virtual-IP2

Payload

MPLS / VNI

Phy-IP2

1. Guest OS ARPs for destination

within subnet or default GW

2. VRouter receives the ARP and

responds back with VRRP MAC

3. Guest OS sends traffic to the

VRRP MAC, Vrouter encapsulates

the packet with appropriate

MPLS/VNI tag and GRE header

1. Physical Fabric Routers on

Physical IP Address

1. Returning packets get forwarded to

appropriate Routing Instance by

the MPLS/VNI tag

1. VRouter de-capsulates the packet,

and forwards it to the Guest OS

OPENCONTRAIL

FEATURES @SYMC

DNSaaS

Contrail offers 4 different DNS modes

• Default DNS server

• The host OS’s configured DNS server

• Tenant DNS server

• Tenants can use their own DNS servers (different from host OS’s DNS server)

• Virtual DNS server

• Contrail Controller provides a per tenant DNS server

• None

• VMs don’t have any DNS resolution capability

One of these modes is selected when an IPAM instance is

created for a domain.

Contrail Virtual DNSDNS Record Creation

• Each IPAM -> Virtual DNS servers configured

• Virtual Networks and VMs in IPAM use DNS domain of Virtual DNS server specified in IPAM

• When a VM is spawned,

• A & PTR records are added into the vDNS server of the virtual network’s IPAM

NOTE:

• DNS Records can also be added statically.

• A, CNAME, PTR and NS records are also supported.

Contrail Virtual DNS DNS Resolution:

1. DNS requests from VM trapped to the vRouter agent on the

hypervisor

2. vRouter agent then forwards DNS request to the controllers (which

run BIND) for resolution.

3. BIND has the concept of views and every virtual DNS instance has

its own isolated view

view "default-domain-contrailtestdns" {

rrset-order {order random;};

forwarders {172.16.70.254; };

zone "6.6.6.in-addr.arpa." IN {

type master;

file "/etc/contrail/dns/default-domain-contrailtestdns.6.6.6.in-addr.arpa.zone";

allow-update {127.0.0.1;};

};

zone "contrail.us" IN {

type master;

file "/etc/contrail/dns/default-domain-contrailtestdns.contrail.us.zone";

allow-update {127.0.0.1;};

};

};

DNS & IPAM Relationship

• Neutron network maps to Contrail Virtual Network

• network-ipam & virtual-DNS (Contrail specific constructs)

• virtual-DNS object has domain as parent

• network-ipam has project as parent.

• So:

• virtual-network ==refers-to==> network-ipam ==refers-to==> virtual-DNS

Contrail Virtual DNS @SYMC

• By default, Contrail API server creates default-network-

ipam object under the default-domain -> default-project

hierarchy

• However, using Contrail API hooks mechanism

automatically

• Create default-network-ipam object within a newly created project

• Create default-virtual-DNS object within a newly created domain

• Link them to provide vDNS functionality.

• So, a new virtual-network when created, it is automatically linked to

the project specific default-network-ipam and corresponding virtual

DNS object

Floating IPs

• Neutron supports the concept of floating IP (routable IP).

• Instances are unaware of their Floating IP.

• Every Virtual Network -> Routing Instance

• Routing Instances

• Define network connectivity between VMs in the Virtual Network

• Contain routes only for VMs in the virtual network

• Two Routing Instances (Virtual Networks) can be connected using

• Neutron L3 agent

• Contrail Network Policy (explained later)

• By default, Virtual Network do not have access to a “public” (routable) network

• A Gateway must be used to provide connectivity to "public" network from a virtual-network.

• Floating IP support can be provided with

• Simple Gateway – x86 based Software GW

• Routing Device such as Juniper MX

Floating IP using Neutron L3 Router

• Create an external network

• neutron net-create public --router:external True

• Create a router

• neutron router-create router1

• Add interfaces from Virtual network to this router

• neutron router-interface-add router1 SUBNET1_UUID

• Set router-gateway-set on router instance

• Connects a router to an external network, which enables that router

to act as a NAT gateway for external connectivity.

• neutron router-gateway-set router1 EXT_NET_ID

Spine Spine

Leaf LeafLeaf

BMS

BMS

BMS

BMS

Node

Node

Node

Node

Node

Node

Node

Node

Mountain View DC

MX Router

Internet

Spine Spine

Leaf LeafLeaf

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Node

Boston DC

Public

VRF

Intra-

site

VRF

Internet

MX Router

Intra-

site

VRF

Public

VRFIntra-site VPN

Multiple Floating IPs per VM @SYMC

VMs in the MTV DC

1. Internet

Routable

Floating IP

2. IP Routable from

Boston DC

LBaaS

• LBaaS load balancer enables

• Pool of VMs servicing apps accessible via a virtual IP.

• Contrail LBaaS features:

• Load balancing of traffic from clients to a pool of backend servers.

The load balancer proxies all connections to its virtual IP.

• Provides load balancing for HTTP, HTTPS, and TCP

• Provides health monitoring capabilities for applications

• Floating IP association to virtual IP for public access to the backend

pool.

Contrail LBaaS Implementation

Contrail LBaaS Implementation

• Supports OpenStack LBaaS Neutron APIs

• Creation of virtual-ip, loadbalancer-pool, loadbalancer-member, and

loadbalancer-healthmonitor.

• Creates a Service Instance when a loadbalancer-pool is

associated with a virtual-IP object.

• Service scheduler launches a namespace & spawns HAProxy on

it.

• HAProxy parameters obtained from the load balancer objects.

• HA of namespaces/HAProxy -> Active/Standby (2 diff computes)

Link Local Services

Provides VMs access to specific services on IP Fabric

infrastructure.

• @SYMC

• Keystone, Github, NTP, Logging, Monitoring and

Metering services

Once the link local service is configured, VMs can access

the service using the link local address.

• OpenStack Metadata Service on 169.254.169.254:80 is also

implemented using Link Local Service

(169.254.169.XXX, Service port) <-> (Destination IP, Service TCP/UDP port)

Contrail Network Policy

• Enforces connectivity and policy enforcement between

Virtual Networks

• Follows the 5-tuple semantics

• SRC/DST Virtual Network, SRC/DST Port, Protocol

Contrail Network Policy• Connectivity between two Virtual Networks is established by leaking

routes between two Routing Instances when a network policy is

created interconnecting the two VNs

• Policy is enforced for specific traffic types by flow table programming

in every vRouter which has the relevant Virtual Networks

Compute Node

VirtualMachine

(Tenant B)

VirtualMachine

(Tenant C)

VirtualMachine

(Tenant C)

vRouter Forwarding Plane

VirtualMachine

(Tenant A)

Routing Instance

(Tenant A)

Routing Instance

(Tenant B)

Routing Instance

(Tenant C)

vRouter Agent

Flow Table

FIB

Flow Table

FIB

Flow Table

FIB

Eth1Kernel

Tap Interfaces (vif)

pkt0

UserEth0 EthN

Config

VRFsPolicy Table

Environments & Operations

Environments

• Lab: > 10 nodes

• CI/CD test environment for SDN related features and functions

• Staging: > 50 nodes

• True IaaS for PaaS applications

• Production: > 250 nodes

• PaaS for end-user applications

Operations:

• Monitoring & Troubleshooting

• Contrail Analytics feeds into OpsView & LMM

• Upgrade

• Phased upgrades during maintenance windows without application downtime.

TEST DRIVE

OPENCONTRAIL

DEVSTACK + OPENCONTRAIL

• WHAT?

• Run OpenStack and OpenContrail on your laptop or in a VM

• WHY?

• Use to build & test OpenStack and OpenContrail code

• Just play with OpenStack/OpenContrail features

• HOW?

• Ubuntu server/VM with 4GB RAM, access to github

DEVSTACK + OPENCONTRAIL (in-a-box)

• Install packages: git-core, ant, build-essential, pkg-config

• Download DevStack

• (git clone git@github.com:/dsetia/devstack.git)

• Edit localrc (set PHYSICAL_INTERFACE)

• Run stack.sh

• Installs Glance, Nova, Horizon, Keystone, Cinder

• And OpenContrail (as a Neutron plugin)

RESOURCES

• OpenContrail.org

• E-Book, Architecture documents, blogs from developers/architects,

slides, webinars

• GitHub

• https://github.com/Juniper/contrail-controller

• https://github.com/Juniper/contrail-vrouter

• https://github.com/Juniper/contrail-puppet

• https://github.com/Juniper/contrail-web-controller

• https://github.com/Juniper/contrail-neutron-plugin

Q&A