architecting nsx-t for pks - vblog.io€¦ · architecting nsx-t for pks romain decker –@woueb...

©2019 VMware, Inc.

Architecting NSX-T for PKS

Romain Decker – @woueb

2©2019 VMware, Inc.

Emergence of Containers & Cloud-Native Apps

Adoption Momentum

2018385M deployed containers worldwide

20211,814M deployed containers worldwide

99% CAGR

Source: “Container Infrastructure Market Assessment: Bridging Legacy and Cloud-Native Architectures — x86 Software Containers Forecast, 2017–2021” IDC, 2018. (ID US43661118)

Web App

DB

N-tier model

Traditional app platform

Deployed in VMs

Current app architecture Emerging app architecture

Cloud-native app platform

NotificationseStore CustomersPayments

Microservices model

Cloud-native app platforms:Kubernetes, Cloud Foundry, Mesos

Deployed in containers


Containers at a GlanceDeploy, Manage, Scale, and Troubleshoot Your Containerized App

KUBERNETES CLUSTER

Node

Containerized app

Master


Node

Containers at a GlanceDeploy, Manage, Scale, and Troubleshoot your Containerized App

Kubelet

Docker runtime

10.10.10.1 Pod

VolumeNode processes

Containerized app10.10.10.2

10.10.10.3

10.10.10.4


Networking

Security

44%

44%

Top Challenges in Deploying Containers

Source: Cloud Native Computing Foundation Blog. June 28, 2017https://www.cncf.io/blog/2017/06/28/survey-shows-kubernetes-leading-orchestration-platform/

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Finding vendor support

Other

Reliability

Logging

Difficulty choosing an orchestration solution

Complexity

Storage

9%

10%

23%

25%

28%

39%

48%

Top challenges

Router

Firewall

Load Balancer

https://www.cncf.io/blog/2017/06/28/survey-shows-kubernetes-leading-orchestration-platform/


Container Networking & Security Challenges

Inconsistent networking for cloud-native and traditional workloads

Uneven security and compliance

Limited visibility into container network traffic


Desired state of Application

Desired state of Kubernetes Clusters

Kubernetes VMware PKS

Kubernetes / VMware PKS


VMware PKS on SDDC Rapidly deliver and operationalize next-generation apps

Physical Infrastructure

BOSH

NSX-T

Service Broker

vSANvSphere

etcd worker

Container Registry

master etcd workermaster

PKS Control Plane

Kubernetes Cluster Kubernetes Cluster

vRealizeAutomation

vRealizeLog Insight

vRealizeOperations

vRealizeNetwork Insight

Wavefrontby VMware


Identifying PKS Components

OPS MANAGER: provide UI to install Bosh Director and PKS Control Plane VM

BOSH: deploy and manage Kubernetes clusters

PKS: front end API for users to interact with PKS

HARBOR: private container registry

Management and data planes

vSphere

Physical Hardware

workermaster

Kubernetes Cluster

OPS MANAGER

BOSH

HARBOR

P

PKSworkermaster

Kubernetes Cluster

PKS DATA PLANE PKS MANAGEMENT PLANE

NSX DataCenter


NSX-T PKS Integration – Topology Mapping

admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created

admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created

Namespace: foo Namespace: bar

NSX / K8S TOPOLOGY

10.24.0.0/24 10.24.1.0/24 10.24.2.0/24

K8S MASTERS K8S WORKERS


INTERNAL TO NSX-T | (NON-ROUTABLE / NAT)

Non-routable (internal to NSX-T) doesn’t imply non unique subnets

Requirements (scale, troubleshooting)

Dependencies

Deployment philosophies

CHOICES BASED ON

ADDITIONAL CONSIDERATIONS

Networking TopologyNO-NAT and NAT choices

EXTERNAL TO NSX-T(ROUTABLE / NO-NAT)

INTERNAL TO NSX-T (ROUTABLE / NO-NAT)

INTERNAL TO NSX-T (NON ROUTABLE / NAT)

INTERNAL TO NSX-T | (ROUTABLE / NO-NAT)

INTERNAL TO NSX-T (ROUTABLE / NO-NAT)

> PKS MANAGEMENT NETWORK <

> POD NETWORKS <

INTERNAL TO NSX-T | (NON-ROUTABLE / NAT)

> NODE NETWORKS <


PKS & NSX-T Networking IntegrationDesign considerations

NAT MODE

Enable NAT mode for node network

POD IP BLOCK

Will be carved out to create networks to host Kubernetes pods belonging to the same namespace

Should be a multiple of /24

POOL ID

Used for: K8S Master VIP, SNAT from pods, Kubernetes Service kind (LoadBalancer L4),

Kubernetes Ingress kind (L7)

Cannot be on the same subnetas the uplink/transit network

T0 MAPPING

PKS supports only a single/multi T0

T0 must be configured in Active-Standbyregardless of networking topology

NODE IP BLOCK

Will be carved out to create networks to host Kubernetes cluster node VMs

Should be a multiple of /24

Scale is directly impacted by IP Blocks and Pool configuration


Networking Topology: Example #1

CONSIDERATIONS

• PKS Management external to NSX-T, deployed on a classic vSphere port group

• PKS Management and vSphere / NSX Management networks can be combined

PKS Management external to NSX-T + NO-NAT

POD NETWORK – ‚PKS-INFRASTRUCTURE‘

POD NETWORK – ‚KUBE-SYSTEM‘

KUBERNETES NODES

POD NETWORK – ‚DEFAULT‘

P

T0

T1

VIP

PHYSICAL NETWORK

T1

T1

T1

T1

MASTER W W W W

PKS MANAGEMENT

MANAGEMENT NA

T

NO NAT

NO

NA

T

ROUTABLE IP



CONSIDERATIONS

• PKS Management internal to NSX-T, deployed on a logical switch

• The tier-1 logical router and logical switch required for the PKS Management network must be created upfront

PKS Management internal to NSX-T + NO-NAT



KUBERNETES NODES


P

T1

VIP

PHYSICAL NETWORK

T1

T1

T1

T1

MASTER W W W W

PKS MANAGEMENT

MANAGEMENT NA

TN

O N

AT

ROUTABLE IP

T0

NO NAT

T1



CONSIDERATIONS

• PKS Management internal to NSX-T, deployed on a logical switch

• The tier-1 logical router and logical switch required for the PKS Management network must be created upfront

• DNAT rules required for PKS Management

PKS Management internal to NSX-T + NAT



KUBERNETES NODES


P

T1

VIP

PHYSICAL NETWORK

T1

T1

T1

T1

MASTER W W W W

PKS MANAGEMENT

MANAGEMENT NA

T

ROUTABLE IP

T0

NAT

NA

T

T1


Network Profile is a capability that provides a means of expressing networking choice at the time of cluster creation

Specified by JSON file

Network ProfilesSpecify NSX-T configuration parameters at cluster creation time

• Load Balancer Sizing

• Edge Router Selection

• Custom Floating IP

• Routable Pod Networks

• Custom Pod Networks

• Bootstrap Security

• Pod Subnet Prefix

{"name": "np-vmug","description": "Network Profile for VMUG demo","parameters": {

"lb_size": "medium","t0_router_id": "5a7a82b2-37e2-4d73-9cb1-97a8329e1a92","fip_pool_ids": [

"e50e8f6e-1a7a-45dc-ad49-3a607baa7fa2"],"pod_routable": true,"pod_ip_block_ids": [

"ebe78a74-a5d5-4dde-ba76-9cf4067eee55","ebe78a74-a5d5-4dde-ba76-9cf4067eee56"

]"master_vms_nsgroup_id": "9b8d535a-d3b6-4735-9fd0-56305c4a5292","pod_subnet_prefix" : 26

}}


Multi-T0 Support

SCALE• Support Multiple T0’s• Support Multiple Edge Clusters

ISOLATION• Network Isolation between

Tenants• Separate Pod IP Blocks & IP

Pools per Tenant

PKS ENVIRONMENTS• Support for multiple PKS

foundations on the same NSX-T

Enabled via Network Profiles

PT0

PHYSICAL NETWORK

PKS MANAGEMENT

MANAGEMENT

COMMON SERVICES

TENANT A TENANT B

SHARED NETWORK

PKS CLUSTER A1 PKS CLUSTER A2 PKS CLUSTER B1

T0

T0 T0


NSX can utilize Kubernetes Network Policies to define dynamic security groups and policies.

Using Network Policy, developers can define firewall rules to allow traffic into and out of a Namespace, and between Pods. Once the Network Policy is applied, NSX will dynamically create source & destination Security Groups and apply the right policy.

Security Groups and Policies are predefined in NSX.

Labels are used to specify pods membership.

Security Groups are defined in NSX with ingress and egress policies.

K8S NETWORK POLICY PRE-DEFINED LABEL BASED RULES

Containers Micro-segmentation with NSX-T


Highlights

Enterprise-grade container networking

Advanced Container Networking

Micro-Segmentation for Microservices

Granular security at the container level

Cross-Platform Visibility

Monitor container-to-container traffic

architecting nsx-t for pks - vblog.io€¦ · architecting nsx-t for pks romain decker –@woueb...

Documents