lessons learned: deploying
TRANSCRIPT
#vmworld
CNET1444BU
Lessons Learned: Deploying OpenShift with VMware SDDC
Vincent Han, VMware, Inc.Wayne Cheng, GovTech
#CNET1444BU
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc.
Disclaimer
This presentation may contain product features or functionality that are currently under development.
This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.
2
The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein. VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc.
Introduction
Staff Solution Specialist
Networking & Security
Vincent Han
3
Senior DevOps Engineer
Government Digital Services
Wayne Cheng
VMworld 2019 Content: Not for publication or distribution
4©2019 VMware, Inc.
•Lessons learned during POC & Trial Implementation
•Knowledge to start your implementation
Why?
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc.
Agenda
5
Customer Journey – GovTech Singapore
NSX Container Plugin (NCP) & Integration Overview
Lessons Learned and Next Steps
OpenShift on VMware SDDC
Q&A
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 6
- Digitisation and Smart Nation Building
- Citizens’ interactions with Government
- Online government e-services
• https://www.tech.gov.sg/products-and-services/singapore-government-tech-stack/
- Platform as a Service – NECTAR
- API gateways – APEX
• https://www.tech.gov.sg/media/technews/getting-to-know-nectar-and-apex
Government Technology Agency of Singapore
Introduction to GovTech
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 7
Introduction to GovTech
VMworld 2019 Content: Not for publication or distribution
8©2019 VMware, Inc.
•Failing is an option
•Strong technical competencies
•Find leverage
•Continuous learning, improvement & delivery
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 9
• Why we chose NSX-T as our SDN Solution
• Mixed workloads of Virtual Machines and Containers
• Adaptability of solution to integrate with Openshift Container Platform
• Provide native Load Balancing Service
• Granular control of network policies and visibility of the Containers’ Networks
• POC environment
• 1 Data Center, 3 Nested ESXi Hosts
• NSX-T 2.1, NCP 2.1, OSE 3.9
• 1 Master, 2 worker nodes
• Trial implementation environment
• 1 Data Center, 6 physical ESXi Hosts
• NSX-T 2.3, NCP 2.3, OCP 3.11
• 3 Masters, 2 Infra, 2 Worker Nodes
Requirements from GovTech and Why NSX-T
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 10
Journey so far…
▲
POC
OSE 3.9
NSX-T 2.1
NCP 2.1
Mar 2018 Apr 2018 May 2018 Dec 2018 Jan 2019 Feb 2019
▲
Trial Implementation
OCP 3.11
NSX-T 2.3
NCP 2.3
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 11
Kubernetes Node (VM)
Kubernetes Pod
Kubernetes 101
Stem B
Stem BStem B
Stem B Node
Node
Node
Kubernetes Master Node
API
Kubernetes Pod
App Container
RedisDB
Tools, Libs, SW
Pod 2
Pod 1
KDocker Engine
K
K
K
ESXi
App Container
RedisDB
Tools, Libs, SW
VM
VM
VM
VM
Stem BStem B
Stem BESXi
ESXi
vCenter
API VMVM
VM
CNI
NSX-T
NCP
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 12
NSX Container Plugin (NCP)
NCP infrastructure
NSX ManagerAPI Client
NSX Integration with CaaS / PaaS
NSX Manager
More…
OpenShift
PAS
Kubernetes
Hypervisor Bare-metal server
NSX Container Plugin Integrates NSX with Container Platforms
• Application deployment on container platforms leads to creation of networks, routers, firewalls, and load balancers
• No change to application platform user experience
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 13
Network Topology
KubeDNS
POD1
POD4
POD2
POD3
POD5
POD6
T0
Internet
kube-system PODs – Logical Switch - 10.12.0.0/24
Namespace ‘default’ PODs – Logical Switch - 10.12.1.0/24
Namespace ‘demo’ PODs – Logical Switch - 10.12.2.0/24
Master ‘VM’
etcdKubeDNS
APISrv
Node ‘VM’
Pod1
Pod2
Node ‘VM’
Pod3
Pod4
Node ‘VM’
Pod5
Pod6
T1
T1
T1
T1-Mgmt
Cluster Management Nodes – LS-VIFs (Non Routable)
Cluster Management Nodes – LS-Mgmt – 10.11.1.0/24
PhysicalRouter
Edges Uplink
ens192 ens192ens192ens224 ens224 ens224
NSX-TControllersvCenter
NSX-TManager
Secured RepoServer
Management Network – 192.168.110.0/24
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 14
NCP is NSX Container Plugin
• Translating Kubernetes resources to NSX-T objects
• When NCP starts, it will check both Kubernetes Resources and NSX-T objects and fill the gaps between them.
What is NCP
NCP Architecture
Openshift / Kubernetes Resources
Pod
Openshift Route / Ingress
Project / Namespace
Network Policy
NSX-T Objects
Service(Loadbalancer)
Container Interface
NSX-T L7 LB Rule and Pool
T1 Router, SNAT,,,etc
DFW
NSX-T L4 LB VS and Pool
NSX Container Plugin
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc.
NCP ArchitectureNSX Container Plugin would work once all NCP components are properly installed
• NCPTranslating Kubernetes resources to NSX-T objects
• NSX-Node-AgentForward pod network informationPlumb pod interface to OpenvSwitch inside node vm
• NSX CNI pluginCNI interface between kubelet and nsx-node-agent
• NSX-Kube-ProxyTranslate Service(ClusterIP) of Kubernetes Resources into OpenvSwitch configuration
• OpenvSwitchProvide container networking and Service(Cluster IP)Isolate Pod traffic in VM
Host
K8s/OpenShift Node(VM)
kubelet
KubernetesControl
Plane
Pod Pod
NCP
LCP
OpenvSwitch
nsx-kube-proxy
vNIC
vmk50
CNI
nsx-node-agent
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc.
OpenvSwitch and Container Network Security
• Distributed Firewall (DFW) at Host
• Micro-segmentation Pod to Pod traffic
Host
Node(VM)
br-int(ovs)
vNIC
VLAN 1 VLAN 2
CIF CIF
T1 DRns: B
Topology
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 17
Host
K8s Master(VM)
NCP Components
There are some components in NSX Container Plugin
• NCP
– Deployed as a Pod of Kubernetes.
– Only single instance is running on one of kubernetes node
• NSX Node Agent
– Deployed as a DaemonSet of Kubernetes
– Every Kubernetes nodes has NSX-Node-Agent
• OpenvSwitch
– Virtual switch installed inside Kuberntes nodes
– Every Kubernetes nodes has OpenvSwitch
• NSX CNI plugin
– CNI plugin used in NSX-T integration.
– Installed in every Kubernetes nodes
Kubernetes Resources
NSX-T Manager
NSX-Node-Agent(DaemonSet)
CNI OpenvSwitch
Host
K8s Node (VM)
Kubernetes Resources
NSX-Node-Agent(DaemonSet)
K8s Node(VM)
Kubernetes Resources
NSX-Node-Agent(DaemonSet)
NCP(Deployment)
CNI OpenvSwitch CNI OpenvSwitch
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 18
Installation
Failed multiple times due to various conditions
- Edge cannot be active-active
- Missing vmk 50
- NIC config (OOB mgmt, OCP mgmt, POD networking)
Documentation not readily available
Key use case not working
- Ver2.3 does not support custom certs, thus unable to test https route
Created a dependency for OCP to NSX-T
- Increased the complexity for day 2 operations, eg. Upgrading/Maintenance
Challenges
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 19
Trial Implementation Topology (Will not work)
KubeDNS
POD1
POD4
POD2
POD3
POD5
POD6
T0
Internet
kube-system PODs – Logical Switch
Namespace ‘default’ PODs – Logical Switch
Namespace ‘demo’ PODs – Logical Switch
Master ‘VM’
etcdKubeDNS
APISrv
Node ‘VM’
Pod1
Pod2
Node ‘VM’
Pod3
Pod4
Node ‘VM’
Pod5
Pod6
T1
T1
T1
Cluster Management Nodes – LS-VIFs (Non Routable)
PhysicalRouter
Edges Uplink
NSX-TControllersvCenter
NSX-TManager
Secured RepoServer
Management Network – 172.16.19.0/16
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 20
Trial Implementation Topology Final Working
KubeDNS
POD1
POD4
POD2
POD3
POD5
POD6
NSX-TControllers
T0
Internet
kube-system PODs – Logical Switch
Namespace ‘default’ PODs – Logical Switch
Namespace ‘demo’ PODs – Logical Switch
Master ‘VM’
etcdKubeDNS
APISrv
Node ‘VM’
Pod1
Pod2
Node ‘VM’
Pod3
Pod4
Node ‘VM’
Pod5
Pod6
T1
T1
T1
T1-Mgmt
PhysicalRouter
vCenterNSX-T
Manager
Edges Uplink
Secured RepoServer
Management Network – 192.168.110.0/24
Cluster Management Nodes – LS-VIFs (Non Routable)
Cluster Management Nodes – LS-Mgmt – 10.11.1.0/24
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 21
• Check supportability matrix before installation
• https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.4/rn/NSX-Container-Plugin-Release-Notes.html
• Ensure Reserve IP Range not used - Default Service IP Range - 172.30.0.0/16
• Size of NSX-T Edge VMs
• https://communities.vmware.com/docs/DOC-40435
• T0 – Active/Standby
• NSX-T Principal Identity
Tips before Installation
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 22
1. Install NSX-T and setup Infra
2. Place NCP CNI Package and OVS package on Secured Repo Server
3. NSX-T resource setup
• Create IP block used for pod network
• Create IP block used for No SNAT
• Create IP pool used for LB VIP
4. Put tags on vNIC
5. Upload NCP Docker image on every node
6. Amend Ansible Host file and deploy OpenShift cluster
7. Deploy NCP [Part of deploy-cluster.yml in OpenShift 3.11]
8. Deploy NSX node agent [Part of deploy-cluster.yml in OpenShift 3.11]
https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.4/nsxt_24_ncp_openshift.pdf
NCP Installation Overview
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 23
NCP is available in my.vmware.com
The package has
• NCP deployment yaml file
• NSX-Node-Agent yaml file
• OpenvSwitch packages
• NCP docker image.
NCP installationNCP packages
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 24
Internal Structure
NSX Container Plugin
CNI and NSX Node Agent yaml per OS
NCP Docker Images
Openvswitch packages
Deployment is preferred. RC is legacy.
Required for OCP 3.11
Required for OCP 3.11
Required for OCP 3.11
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 25
https://github.com/vincenthanjs/openshift-ansible-hosts/blob/master/hosts
OpenShift Ansible hosts file
openshift_master_default_subdomain=ocpapps.acepod.comopenshift_use_nsx=trueos_sdn_network_plugin_name=cniopenshift_use_openshift_sdn=falseopenshift_node_sdn_mtu=1500
# NSX specific configurationnsx_openshift_cluster_name='ocp-cl1'nsx_api_managers='192.168.110.26'nsx_api_user='admin'nsx_api_password='VMware1!'nsx_tier0_router='JUR01-T0'nsx_overlay_transport_zone='TZ-Overlay'nsx_container_ip_block=‘IP-Block-OCP-Container'nsx_no_snat_ip_block=‘IP-Block-OCP-NO-SNAT'nsx_external_ip_pool=‘IP-Pool-OCP-External'nsx_top_fw_section='openshift-top'nsx_bottom_fw_section='openshift-bottom'nsx_ovs_uplink_port='ens224'nsx_cni_url='http://192.168.110.12/nsx-cni-2.3.2.11695762-1.x86_64.rpm'nsx_ovs_url='http://192.168.110.12/openvswitch-2.9.1.9968033.rhel75-1.x86_64.rpm'nsx_kmod_ovs_url='http://192.168.110.12/kmod-openvswitch-2.9.1.9968033.rhel75-1.el7.x86_64.rpm’
Need a secured repo server to host the files.
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 26
https://github.com/vincenthanjs/openshift-ansible-hosts/blob/master/hosts
NSX-T Configuration
openshift_master_default_subdomain=ocpapps.acepod.comopenshift_use_nsx=trueos_sdn_network_plugin_name=cniopenshift_use_openshift_sdn=falseopenshift_node_sdn_mtu=1500
# NSX specific configurationnsx_openshift_cluster_name='ocp-cl1'nsx_api_managers='192.168.110.26'nsx_api_user='admin'nsx_api_password='VMware1!'nsx_tier0_router='JUR01-T0'nsx_overlay_transport_zone='TZ-Overlay'nsx_container_ip_block=‘IP-Block-OCP-Container'nsx_no_snat_ip_block=‘IP-Block-OCP-NO-SNAT'nsx_external_ip_pool=‘IP-Pool-OCP-External'nsx_top_fw_section='openshift-top'nsx_bottom_fw_section='openshift-bottom'nsx_ovs_uplink_port='ens224'nsx_cni_url='http://192.168.110.12/nsx-cni-2.3.2.11695762-1.x86_64.rpm'nsx_ovs_url='http://192.168.110.12/openvswitch-2.9.1.9968033.rhel75-1.x86_64.rpm'nsx_kmod_ovs_url='http://192.168.110.12/kmod-openvswitch-2.9.1.9968033.rhel75-1.el7.x86_64.rpm’
NSX-T Manager Configurations
OCP Topology
POD1
POD4
T0
Namespace ‘foo’ PODs – Logical Switch – 10.12.5.0/24
Master ‘VM’
etcdKubeDNS
APISrv
Node ‘VM’
Pod1
Node ‘VM’
Pod2
T1
T1-Mgmt
Cluster Management Nodes – LS-VIFs (Non Routable)
Cluster Management Nodes – LS-Mgmt – 10.11.1.0/24
External LB IP Pool – 10.21.0.0/24
SNAT IP for Namespace foo - 10.21.0.2
T1-LB
ens192 ens224 ens192 ens224VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 27
VIFs Tagging
OCP Topology
POD1
POD4
T0
Namespace ‘foo’ PODs – Logical Switch – 10.12.5.0/24
Master ‘VM’
etcdKubeDNS
APISrv
Node ‘VM’
Pod1
Node ‘VM’
Pod2
T1
T1-Mgmt
Cluster Management Nodes – LS-VIFs (Non Routable)
Cluster Management Nodes – LS-Mgmt – 10.11.1.0/24
External LB IP Pool – 10.21.0.0/24
SNAT IP for Namespace foo - 10.21.0.2
T1-LB
ens192 ens224 ens192 ens224
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 28
• Blogs
• https://blogs.vmware.com/networkvirtualization/2019/02/nsx-t-integration-with-openshift.html/
• http://blog.acepod.com/how-to-install-openshift-container-platform-ocp-with-nsx-t-ncp/
• Installation Videos
• NSX-T Openshift 3.11 Installation/Integration demo - https://youtu.be/uEQ5UAgh770
• How to install Openshift Container Platform 3.11 Enterprise with VMware NSX-T (Native Integration) - https://youtu.be/5ZlggXKXwL8
• Sample Openshift Ansible host file
• https://github.com/vincenthanjs/openshift-ansible-hosts/blob/master/hosts
Resources for your reference
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 29
Reference Topology
KubeDNS
POD1
POD4
POD2
POD3
POD5
POD6
T0
Internet
kube-system PODs – Logical Switch
Namespace ‘default’ PODs – Logical Switch
Namespace ‘demo’ PODs – Logical Switch
Master ‘VM’
etcdKubeDNS
APISrv
Node ‘VM’
Pod1
Pod2
Node ‘VM’
Pod3
Pod4
Node ‘VM’
Pod5
Pod6
T1
T1
T1
T1-Mgmt
Cluster Management Nodes – LS-VIFs (Non Routable)
Cluster Management Nodes – LS-Mgmt – 10.11.1.0/24
PhysicalRouter
Edges Uplink
ens192 ens192ens192ens224 ens224 ens224
NSX-TControllersvCenter
NSX-TManager
Secured RepoServer
Management Network – 192.168.110.0/24
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 30
VMs Placement
Compute ClusterManagement/Edge Cluster
vCenter
NSX-TControllers
NSX-TManager
NSX-TEdge VMs
VM
VM
VM
VM
VM
VM
OCP Master Nodes
OCP Worker Nodes
ESXi ESXi
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 31
POC Topology (Nested ESXi for POC)
KubeDNS
POD1
POD4
POD2
POD3
POD5
POD6
T0
kube-system PODs – Logical Switch
Namespace ‘default’ PODs – Logical Switch
Namespace ‘demo’ PODs – Logical Switch
Master ‘VM’
etcdKubeDNS
APISrv
Node ‘VM’
Pod1
Pod2
Node ‘VM’
Pod3
Pod4
Node ‘VM’
Pod5
Pod6
T1
T1
T1
T1-Mgmt
Cluster Management Nodes – LS-VIFs (Non Routable)
Cluster Management Nodes – LS-Mgmt – 10.11.1.0/24
Internet
PhysicalRouter
Edges UplinkRouter VM
NSX-TControllersvCenter
NSX-TManager
Secured RepoServer
Management Network – 192.168.110.0/24
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 32
POC VMs Placement (Nested ESXi for POC)
Compute ClusterManagement/Edge Cluster
vCenter
NSX-TControllers
NSX-TManager
NSX-TEdge VMs
VM
VM
VM
VM
VM
VM
OCP Master Nodes (Nested)
OCP Worker Nodes (Nested)
ESXi ESXi
nested-ESXi (VM) nested-ESXi (VM)
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 33
Lessons learned and key considerations
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 34
• Upgrade NCP from 2.3.x to 2.4.x
• Test vRealize Network Insight vRNI 4.2• Container Network Flows
• Supporting Kubernetes & OpenShift
• Detailed Network Policy
Next Steps
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 35
• VMware and Red Hat announce on May 9, 2019 to collaborate to better integrate OpenShift Container Platform and VMware Software Defined Data Center.
• Simplify networking and network-based security with the NSX Container Plug-in (NCP)
• vSphere Cloud Provider and its corresponding volume plugin –vSAN or any vSphere datastore
OpenShift Container Platform on VMware SDDC
35
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 36
Other VMworld Sessions - Red Hat and VMware
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 37
Deep Dive on NSX-T with Kubernetes Networking
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 38
NSX-T Data CenterCloud-Native Network Services Platform for Cloud-Native Apps
On-Premises – vSphere, Bare-metal and KVM
Business App 2 / LOB 2
CF K8s
Business App 1 / LOB 1
CF K8s
NSX Platform
• Common networking model
• Agility and lower costs with scale out load balancing and firewall
• Monitoring, troubleshooting, and audit controls
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 39
• NSX Container Plugin provides tremendous value in networking & security for OpenShift/Kubernetes
• Start your own journey
• Share your experiences with us• Vincent Han - @vincenthan at Twitter or LinkedIn
• Wayne Cheng @LinkedIn
Key Takeaways
VMworld 2019 Content: Not for publication or distribution