ibm cloud for vmware solutions nsx edge services ... purpose of this document is to define and...

Copyright IBM Corporation 2017 of 16

IBM Cloud for VMware Solutions NSX Edge Services Gateway

Solution Architecture

Date: 2017-03-29

Version: 1.0


Table of Contents

1 Introduction ............................................................................................................................... 4

1.1 About NSX Edge Services Gateway .................................................................................. 4

2 Design ...................................................................................................................................... 5

2.1 Overview ............................................................................................................................. 5

2.1.1 Internal Architecture ................................................................................................... 5

2.1.2 Dedicated Architecture............................................................................................... 5

2.1.3 IBM Cloud IP address space vs BYOIP..................................................................... 5

2.2 Cloud Networking Services ................................................................................................ 6

2.2.1 IBM Management Edge ............................................................................................. 7

2.2.2 IBM Workload Edge ................................................................................................. 11

2.3 Multi-Site ........................................................................................................................... 14

2.3.1 Overview .................................................................................................................. 14

2.3.2 NSX Cross vCenter .................................................................................................. 14

2.3.3 Multi-Site example ................................................................................................... 15

3 References ............................................................................................................................. 16


Summary of Changes

This section records the history of significant changes to this document. Only the most significant changes

are described here.

Version Date Author Description of Change

1.0

2017-03-29 IBM Cloud Initial Release


1 Introduction The purpose of this document is to define and describe the solution architecture for the VMware NSX Edge

Services Gateway (ESG) solution deployed on the IBM Cloud. Specifically, it will detail the baseline

components of the management edge and workload edge solution as well as the configuration of each

component in the design. This solution is considered an extension of VMware Cloud Foundation and

vCenter Server on IBM Cloud offerings that are currently available on the IBM Cloud. The configuration

of the VMware Cloud Foundation (VCF) or vCenter Server (VCS) on IBM Cloud are not covered in this

document. Instead it is highly recommended to review and understand the aforementioned architectures

located on the IBM Architecture Center before reading this document.

Figure 1 Cloud Networking Services on IBM Cloud

1.1 About NSX Edge Services Gateway

The NSX Edge gateway connects isolated, stub networks to shared (uplink) networks by providing

common gateway services such as DHCP, VPN, NAT, dynamic routing, and load balancing. Common

deployments of NSX Edge include DMZ, VPN Extranets, and multi-tenant Cloud environments where the

NSX Edge creates virtual boundaries for each tenant, workload, or management component. The following

features are available within the NSX Edge Service Gateway.

NSX Edge Service Routing provides the necessary forwarding information between layer 2

broadcast domains, thereby allowing you to decrease layer 2 broadcast domains and improve

network efficiency and scale. NSX extends this intelligence to where the workloads reside for

doing East-West routing. This allows more direct virtual machine to virtual machine

communication without the costly or timely need to extend hops. At the same time, NSX also

provides North-South connectivity, thereby enabling tenants to access public networks.

Firewall - Supported rules include IP 5-tuple configuration with IP and port ranges for stateful

inspection for all protocols.

Network Address Translation - Separate controls for Source and Destination IP addresses, as well

as port translation

Dynamic Host Configuration Protocol (DHCP) - Configuration of IP pools, gateways, DNS

servers, and search domains.

Site-to-Site Virtual Private Network (VPN) - Uses standardized IPsec protocol settings to

interoperate with all major VPN vendors.

L2VPN - Provides the ability to stretch L2 networks across L3 topologies.

SS VPN-Plus - SSL VPN-Plus enables remote users to connect securely to private networks

behind a NSX Edge gateway.

Load Balancing - Simple and dynamically configurable virtual IP addresses and server groups.

High Availability - High availability ensures an active NSX Edge on the network in case the

primary NSX Edge virtual machine is unavailable.

https://www.ibm.com/devops/method/tracks/tracks_virtualization


2 Design

2.1 Overview

2.1.1 Internal Architecture

The NSX Edge Gateway Services on IBM Cloud solution provides VMware technology deployed in this

document’s prescribed architecture within IBM Cloud datacenter across the globe. Note that this document

describes two solution architectures relating to the NSX Edge Services Gateway. The first of these

architectures is known as the internal architecture and specifies the deployment of the necessary NSX Edge

components in a resource pool in either a VMware Cloud Foundation converged cluster or vCenter Server

cluster. The second architecture, known as the dedicated architecture, deploys the necessary NSX Edge

components in a separate two-node cluster dedicated for the use of the NSX Edge.

Figure 2 Cloud Networking Services

2.1.2 Dedicated Architecture

The dedicated architecture design dedicates a vSphere cluster for the purposes of providing critical

interaction with the physical network infrastructure. The dedicated design has following characteristics and

functions:

Provide on-ramp and off-ramp connectivity to physical networks (i.e., north-south L3 routing on

NSX Edge virtual appliances).

Allow communication with physical devices connected to VLANs in the physical networks

through NSX L2 bridging. Hosts the Control VM for DLR routing.

May have centralized logical or physical services (e.g., firewall, load balancers, VPN monitoring

components, log insight VMs).

NSX Controllers can be hosted in an edge cluster, when a dedicated vCenter is used to manage the

compute and edge resources.

Edge cluster resources have anti-affinity requirement to protect the active standby configuration or

to maintain the bandwidth availability during failure.

2.1.3 IBM Cloud IP address space vs BYOIP

RFC1918 (private IP range) specifically reserves the use of certain network ranges for organization internal

use and cannot be used on the internet. IBM Cloud physical network infrastructure utilizes a specific

RFC1918 private address space across all of its worldwide locations (10.x.x.x/8). These IP address ranges

do not overlap across customer accounts or within an IBM Cloud customer account. Within a customer

account, any IBM Cloud allocated private IP address space can, with VLAN Spanning enabled, route to


any other IBM Cloud private IP address range in any IBM Cloud data center. While this makes it very

simple to setup worldwide connected infrastructure within a customer account, because the IP address

space is fixed, it can be problematic when a customer wishes to extend their data center into IBM cloud via

routing if they are using the same private address space as IBM cloud. The solution is to utilize NSX to

create an “overlay” topology on top of VCF/VCS, isolating the customer IP address space from interacting

with IBM Cloud assigned private IP address space. NSX is capable of providing a L2 VPN would serve to

“span” internal IP address space within the tunnel across external possibly overlapping IP address spaces.

Figure 3 L2 vpn tunnel - IP isolation

2.2 Cloud Networking Services

The Cloud Networking Services on IBM Cloud consists of two pair of VMware of VMware NSX Edge

Services Gateways (ESG) for communication between the IBM Cloud and either the public internet or

customer on-premises network via VPN. These ESGs are segregated to support internal IBM Cloud

management function and egress, ingress of customer related network traffic.


Figure 4 Cloud Networking Services Diagram (VCF Shown)

Figure 4 is a simplified network diagram which shows the management and workload ESGs. It also shows

a NSX Distributed Logical Router (DLR) and workload VXLAN. These components are intended to be an

initial landing point for customer workloads without requiring the specific knowledge to set them up within

NSX. This can facilitate a decreased time to value when deploying VCS/VCF for demonstration purposes

or customer POCs. A DLR is typically employed to route inter-VCF/VCS traffic which is typically termed

“east-west” traffic between separate layer 2 networks within the instance. This is in contrast to a ESG

which functions to facilitate “north-south” network traffic traversing in and out of the VCS/VCF instance.

While a single ESG could suffice for both management and customer traffic, the separation of management

and customer traffic is a design decision made to primarily to keep from accidental misconfiguration of the

management edge. Note that misconfiguration or disabling the management ESG does not keep the

VCF/VCS instance from functioning, but would disable all portal management functions.

2.2.1 IBM Management Edge

The first NSX ESG discussed in this architecture is the IBM Management Edge. The management ESG is a

dedicated NSX edge cluster for IBM Cloud management network traffic only. It is not intended for traffic

traversal of any component not deployed and managed by the VCF/VCS automation.

The purpose of this ESG is to provide a communication path between IBM VCF deployed virtual machines

residing within VMware Cloud Foundation and/or vCenter Server instances and the IBM Automation

infrastructure in the IBM Cloud as shown in Figure 5 IBM Management VM Communication via

Management Edge.


Figure 5 IBM Management VM Communication via Management Edge

As a result of the light communication between the IBM Management VMs and IBM Automation, the NSX

ESGs are sized in a “Large” configuration in an Active/Passive HA Pair and deployed on the management

resource pool of the VCF converged cluster or the vCenter Server cluster. A summary of the IBM

Management Edge deployment is shown in Table 1 IBM Management NSX ESG Specifications.

vCPU Memory Disk Size Storage Location

IBM Management

NSX ESG 1

2 1 GB 1 GB vSAN Datastore (VCF)

Shared Attached Storage for Management

(VCS)

IBM Management

NSX ESG 2

2 1 GB 1 GB vSAN Datastore (VCF)

Shared Attached Storage for Management

(VCS)

Table 1 IBM Management NSX ESG Specifications

2.2.1.1 Management Services

As of this writing, only outbound access is required to the following services:

Cloud Driver VM. This is a VCF deployed image that interfaces with the SDDC manager VM

for life cycle management of the VCF instance. It periodically polls for requests against an

instance of IBM Cloudant™ setup for the management of VCF.

Zerto™ Management VM, if installed. Zerto requires outbound access to the internet for

licensing activation.


2.2.1.2 Edge interfaces

Configuration of the ESG interfaces define what L2 networks the ESG has access to. For the

current purposes of VCF/VCS life cycle management, it is currently required that specific VMs

placed on the management VLAN, be allowed to traverse to the public VLAN. As such, the

following interfaces are defined on deployment:

Interface Interface

Type

Connected To Description

Public Uplink Uplink SDDC-

DportGroup-

External

Public internet facing interface

Private Uplink Uplink SDDC-

DportGroup-

Mgmt

Internal private network facing

interface

Internal Internal Workload HA

VXLAN

Internal interface used for ESG HA

pair heartbeat – portgroup on SDDC-

Dswitch-Private

Table 2 NSX ESG Interface Configuration

2.2.1.3 Subnets

The following subnets will be utilized for the purposes of the Management ESG:

Interface Interface

Type

IP v4 subnet

type

Range Description

Public

Uplink

Uplink IBM Cloud

portable public

/30 – renders

one assignable

IP address.


Private

Uplink

Uplink IBM Cloud

portable

private

(existing

management)

/26 – renders

61 assignable

IP addresses

Internal private network facing

interface

Internal Internal Link local 169.254.0.0/16 Internal interface used for ESG

HA pair communication

Table 3 NSX ESX IP Configuration

2.2.1.4 NAT definitions

Network Address Translation is employed on the Management ESG for the means of allowing

network traffic to traverse between one IP address space and another. This is typically done to

conserve internet routable IPs or to conceal internal IPs from public ones for security reasons.

NAT can also be used to allow for TCP and UDP port redirection. As of this writing, management

traffic will always be initiated from inside the VCF/VCS instance. As such, only a SNAT (source

NAT) need be defined on the Management ESG. An individual SNAT will be created for each

internal VM hosting a service which needs to egress from the VCF instance.

Applied on Interface Source IP range Translated Source IP

Public Uplink Individual IP addresses

on the Management

Portable /26

IBM Cloud portable public IP

Table 4 NSX ESG NAT Configuration


2.2.1.5 Routing

Since services within VMs required to traverse through the Management ESG may also have need

to get to IBM Cloud services within the customer IBM Cloud private network, the following is the

configuration required to achieve this communication.

While it is difficult to predict which destination IP range is needed as a destination for internet

facing connections, any service which will be deployed by and managed by IBM Cloud will point

to the Management ESG as its default gateway. A static route will be required to force traffic

across the IBM Cloud BCR for the services required external network connections.

For any service that will be using the management ESG to traverse out of a VCF/VCS instance, it

is recommended that the following be configured.

Default Gateway is Management ESG

Static route is required for internal IBM Cloud destinations.

If there is a need for the service or VM to access the customer ESG, then static routes will need to

be maintained within the individual service or VM as well as pointing to the customer ESG.

No automatic routing protocols are configured for the Management ESG at this time.

2.2.1.6 VXLAN definitions

The Management HA pair requires a network for the connection of the “internal” interfaces. This

could be done on an existing vSwitch, port group or VXLAN. For this design, a dedicated

VXLAN will be created for the HA heartbeat communication of the Management ESG HA pair.

VXLAN Name Transport

Zone

Type

Mgmt HA transport-1 global

Table 5 NSX ESG VXLAN Definitions

2.2.1.7 Firewall rules

By default, the Management ESG is configured to drop (Deny) all traffic. “Deny” drops all traffic

with no response when that traffic is not allowed to traverse the firewall by any previous (higher in

the order) rule or rule set. Automatic rule generation will be selected to allow for control traffic to

the ESG pair.

The following firewall rules are set (in addition to the automatically generated rules):

Service Source Destination Protocol Action

VCF/VCS management Cloud driver VM Any Cloudant port

(TBD)

Allow

Zerto Zerto

Management VM

Any Licensing port

(TBD)

Allow

Any Any Any Any Deny

Table 6 NSX ESG Firewall Configuration

2.2.1.8 ESG settings

By default, logging is enabled on all new NSX Edge appliances. The default logging level is

NOTICE.


2.2.2 IBM Workload Edge

As described in section 2.2, the Workload Edge (ESG) described in this design is part of a simple topology

intended for workload network communication. Prior versions of VCF and VCS deploy networks and

network subnets, none of which are intended for workload traffic. This section now serves to describe

the design intent of where to attach workloads to a network within a VCF/VCS instance. This is a starting

point for attaching on-premises networks and IP spaces to a particular VCF/VCS instance and is the basis

for a true hybrid cloud architecture.

Figure 6 Example Customer network flow diagram

As Figure 6 Example Customer network flow diagram shows, the Workload ESG is attached to both the

public and private IBM Cloud networks. This allows for workload access to and from internet facing

traffic, but also allows for a site-to-site VPN to be created from either public or private IBM Cloud

networks. This is useful especially in a POC process as it allows for drastically decreased time to value

with regards to connecting to on-premises networks since it can take months to bring up a dedicated WAN

due to particular customer security requirements. However, once a dedicated link is in place, the VPN can

be “flipped” over to traverse that link without affecting the overlay network inside the VPN tunnel or

within the VCF/VCS instance. Once this is done, the public interface for the workload ESG can simply be

removed if it is desired from a security perspective.

The topology portrayed in the diagram above consists of the following NSX components:

NSX Edge appliance (ESG)

Distributed Logical Router (DLR)

VXLAN (L2 over L3)

2.2.2.1 Edge interfaces

As explained in the Management ESG section, configuration of the ESG interfaces define what

adjacent L2 networks the ESG has access to. Part of the design intent of the workload topology is

to achieve a SDN overlay to isolate workloads from the underlying IBM Cloud address space.


This is the basis for allowing a “Bring Your Own IP” (BYOIP) design to be achieved. As such, the

following interfaces need to be defined:

Interface Interface Type Connected

To

Description

Public Uplink Uplink SDDC-

DportGroup-

External


Private Uplink Uplink SDDC-

DportGroup-

Mgmt

Internal private network facing interface

Transit Uplink Uplink Workload-

Transit

Transit VXLAN between the Workload

ESG and the Workload DLR

Internal Internal Workload

HA VXLAN

Internal interface used for ESG HA pair

heartbeat

Table 7 Workload ESG Interfaces

In this design, a DLR is employed to allow for potential “east-west” routing between local

workload connected L2 networks. As this topology is intended to be a simple example, only one

L2 network intended for workloads is described. Adding additional security zones can be achieved

by simply adding additional VXLANs attached to new interfaces on the DLR. The following are

the DLR interfaces to be configured:

Interface Interface

Type

Connected

To

Description

Transit

Uplink

Uplink Workload-

Transit

Transit VXLAN between the Workload ESG

and the Workload DLR

Workload

Uplink

Uplink Workload VXLAN for Workload connections

Internal Internal Workload HA

VXLAN

Internal interface used for ESG HA pair

heartbeat

Table 8 DLR Interfaces

2.2.2.2 Subnets

The following are example subnets to be utilized for the purposes of the Workload ESG:

Interface Interface

Type

IP v4 subnet

type

Range Description

Public Uplink

(ESG)

Uplink IBM Cloud

portable public

/30 – renders one

assignable IP

address.

Public internet

facing interface.

(Customer can

order additional IPs

separately)

Private Uplink

(ESG)

Uplink IBM Cloud

portable private

(existing

management)

/26 – renders 61

assignable IP

addresses

Internal private

network facing

interface


Interface Interface

Type

IP v4 subnet

type

Range Description

Internal (ESG

and DLR)

Internal Link local 169.254.0.0/16 Internal interface

used for ESG HA

pair

communication

Transit Uplink

(ESG and DLR)

Uplink Assigned by

Customer

TBD transit network

connection for

ESG to DLR

Workload

(DLR)

Uplink Assigned by

customer

TBD Workload subnet

Table 9 DLR and Workload ESG IP Configuration

2.2.2.3 NAT definitions

Network Address Translation is employed on the Workload ESG for the means of allowing

network traffic to traverse between one IP address space and another. For the workload ESG, NAT

is required not only to allow for communication to internet destinations, but also to communicate

to any IBM Cloud sourced IP ranges. For this design, workload traffic will be allowed to exit to

the internet, but not to the management or any IBM Cloud networks. As such, only a SNAT

(source NAT) need be defined on the Workload ESG. Note that the entire workload portable

subnet is configured to traverse thru the SNAT.

Note that while it is possible to use NAT to allow for workload communication across multiple

instances of VCF/VCS, this becomes impractical when many workloads need to be connected

across instances. See the Multi-Site section for examples of using advanced NSX capabilities to

create an L2 overly transit network across VCF/VCS instances.

Applied on

Interface

Source IP range Translated Source IP NAT Enabled or

Disabled

Public

Uplink

(Workload

ESG)

Customer defined IBM Cloud portable public IP Customer defined

default disabled

Table 10 Workload ESG NAT Rules

2.2.2.4 Routing

Within this design, the only requirement for Workloads traversing the DLR to the Workload ESG

is to access the internet. The Workload ESG needs to understand the path to the workload

VXLAN and any future workload VXLAN/subnets created behind the DLR. While this could be

achieved via static routes on the ESG, the intent of the workload topology is that of a

demonstrated best practice design. As such, OSPF will be configured between the Workload ESG

and the downstream DLR. See https://pubs.vmware.com/NSX-

6/index.jsp?topic=%2Fcom.vmware.nsx.admin.doc%2FGUID-6E985577-3629-42FE-AC22-

C4B56EFA8C9B.html

For configuration procedure.

Area OSPF

type

OSPF interface IP OSPF authentication

51 stub Assign an IP for each the DLR and ESG on the

transit RFC1918 network

None

Table 11 Dynamic Routing

https://pubs.vmware.com/NSX-6/index.jsp?topic=%2Fcom.vmware.nsx.admin.doc%2FGUID-6E985577-3629-42FE-AC22-C4B56EFA8C9B.html




2.2.2.5 Firewall rules

By default, the Workload ESG is configured to drop (Deny) all traffic. “Deny” drops all traffic

with no response when that traffic is not allowed to traverse the firewall by any previous (higher in

the order) rule or rule set. Automatic rule generation will be selected to allow for control traffic to

the ESG pair.

The following firewall rules are set (in addition to the automatically generated rules):

Service Source Destination Protocol Action

Workloads Workload

subnet

Any Any Allow

Any Any Any Any Deny

Table 12 Workload ESG Firewall Rules

2.2.2.6 VXLAN definitions

The Workload topology ESG and DLR HA pairs require L2 segments (VXLAN) for the

connection of the “internal” interfaces, data transit between the two, and finally, for workloads.

VXLAN Name VCF/VCS Transport

Zone

Type

Workload HA transport-1 global

Workload Transit transport-1 global

Workload transport-1 global

Table 13 Workload ESG Internal Interfaces

2.2.2.7 ESG DLR Settings

By default, logging is enabled on all new NSX Edge appliances. The default logging level is

NOTICE.

2.3 Multi-Site

2.3.1 Overview

One key differentiator between IBM Cloud and other cloud offerings is the ability to provision dedicated

compute capability across the globe and have that on demand infrastructure automatically network

connected within the customer’s private IBM Cloud account. The software defined network capabilities of

Cloud Foundation in concert with IBM Cloud provide a granular defined global infrastructure. Capable of

being built within days within just a few mouse clicks. The following describes a multi-site architecture

example of what can be achieved with the out of the box capability of Cloud Foundation.

2.3.2 NSX Cross vCenter

NSX Cross vCenter capability allows for linking in a primary, secondary relationship of up to 9 NSX

managers. (1 primary and 8 secondary NSX managers) While it is not required to have vCenter servers in

an Enhanced Linked Mode relationship for NSX Cross vCenter to function, it helps a great deal in the

following ways:

Simplified primary, secondary relationship creation using SSO credentials.

Cloud Foundation automation configures DNS name resolution for all sites linked together

Single pane of glass management across all sites for both NSX and normal vCenter functions.


2.3.3 Multi-Site example

The following example simply adds a NSX universal transport zone to the basic Management and

Workload topologies discussed in previous sections. This universal transport zone spans two IBM Cloud

data centers or PODs within a datacenter. Once that is added, multiple VXLANs are added and a Universal

Distributed Router that spans this new VXLANs and has uplinks to the workload ESGs in both sites must

be configured. This causes VMs in the local site to traverse to its local ESG. Of course, for inbound traffic,

a global load balancer is required. That can be achieved with IBM Cloud’s global load balancing offerings.

The following is an example would require Enterprise edition of NSX which is part of Cloud Foundation.

Figure 7 Multi-site topology


3 References NSX 6 Documentation

https://pubs.vmware.com/NSX-6/index.jsp#com.vmware.nsx.install.doc/GUID-D8578F6E-A40C-493A-

9B43-877C2B75ED52.html

NSX edition features (Standard, Advanced, Enterprise)

http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmware-nsx-

datasheet.pdf

vSphere™ 6 Configuration Maximums

https://www.vmware.com/pdf/vsphere6/r60/vsphere-60-configuration-maximums.pdf

IBM Cloud global load balancing

http://knowledgelayer.softlayer.com/articles/global-load-balancing-options

https://pubs.vmware.com/NSX-6/index.jsp#com.vmware.nsx.install.doc/GUID-D8578F6E-A40C-493A-9B43-877C2B75ED52.html

https://pubs.vmware.com/NSX-6/index.jsp#com.vmware.nsx.install.doc/GUID-D8578F6E-A40C-493A-9B43-877C2B75ED52.html

http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmware-nsx-datasheet.pdf

http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmware-nsx-datasheet.pdf

https://www.vmware.com/pdf/vsphere6/r60/vsphere-60-configuration-maximums.pdf

http://knowledgelayer.softlayer.com/articles/global-load-balancing-options

ibm cloud for vmware solutions nsx edge services ... purpose of this document is to define and...

Documents