cna3045bu what’s new containers on sddc or distribution · for the cluster components themselves...

Brandon Henry, GarminCornelia Davis, PivotalPaul Dul, VMware

CNA3045BU

#VMworld #CNA3045BU

What’s NewContainers on SDDC

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

#CNA3045BU CONFIDENTIAL 2



bution

Purpose-built container service to operationalize Kubernetes

for the multi-cloud enterprises and service providers

Fully Supported Kubernetes

Runs on vSphere and VMC

Unified VM + Containers on SDDC

Deep Integration with NSX

Hardened, Production-grade

HA, Security, Multi-tenancy, Tools

VMware and Pivotal Collaborate toDeliver VMware Pivotal Container Service




bution

Physical Infrastructure

VMware PKS

Kubernetes on BOSH (Kubo)

BOSH

NSX

Analytics Automation

SecurityOperations

Monitoring

GCP

Service Brokeretcd worker

Logging

vSANvSphere

etcd workerContainer

Registrymaster master

vRealize

VMware PKS in SDDC Portfolio




bution

1 Garmin Kubernetes Deployment

2 Introducing VMware Pivotal Container Service (PKS)

3 PKS integrations with VMware

4 Q & A

Containers on SDDC – VMware Pivotal Container Service




bution

6

Kube Architecture

Developer Customer

kube2consul

Garmin Developed




bution

Cluster Administration

• Scaling the cluster

• User Management (Auth)

• Service Registration

• Load Balancing

7#CNA3045BU CONFIDENTIAL 7



bution





• Load Balancing

8

kube-puppet

Garmin Developed




bution





• Load Balancing

9

cert scripts

Garmin Developed




bution

10


kube2consul

Garmin Developed




• Load Balancing




bution

11





• Load Balancing

Garmin Developed

consul-templates




bution

12

Monitoring

• Cluster Monitoring

• Application/Service Monitoring




bution

13

Monitoring



kube-monitor

Garmin Developed




bution

14

Monitoring



consul2zabbix

Garmin Developed




bution

15

Logging

• Cluster Logging

• Application Logging

• Correlating Node/Cluster/Container Events




bution

16

Logging

• Cluster Logging



kube-monitorplumber

Garmin Developed




bution

17

Logging

• Cluster Logging



docker app logger

Garmin Developed




bution

18

Logging

• Cluster Logging



kube-monitor

Garmin Developed




bution

19

Security

• Dependency Access (Firewalls)

• Image Signing

• Image Updating

• Vulnerability Scanning

• Credential Management




bution

20

Security


• Image Signing

• Image Updating






bution

21

Security


• Image Signing

• Image Updating






bution

22

Security


• Image Signing

• Image Updating






bution

23

Security


• Image Signing

• Image Updating






bution

24

Security


• Image Signing

• Image Updating






bution




4 Q & A





bution

Workeretcdetcd

Kubernetes

etcd

kubectlRouting

MasterMaster

WorkerWorker

access app

Serving up Kubernetes Dial-tone




bution

High Availability. No out-of-the-box fault-tolerance for the cluster components themselves (masters and etcd nodes).

Scaling. Kubernetes clusters handle scaling the pod/service within the Nodes, but doesn’t provide a mechanism to scale Masters, Workers & etcd VMs.

Health checks and healing. The Kubernetes cluster does routine health checks for the workloads running on Nodes only.

Upgrades. Rolling upgrades on a large fleet of clusters is hard. Who manages the system it runs on?

BOSH

Kubernetes – Hard to Operationalize




bution

Patches Patching platform components with thousands of apps running should feel normal.

Scaling Seamlessly scale platform components to accommodate changing demand.

Upgrades. How do you roll out new versions of the platform with the lights on?

Operating Effort Operating a platform should require very few resources and minimum manual intervention. Otherwise, is it really providing operational benefits?

Multi-cloud Provide a reliable and smooth experience for any cloud.

Open APIs Allow platform operations from different toolsets and the creation of CD pipelines.

Consistency Provide a consistent setup experience, across different cloud environment configurations.

Setup time How long does it take to setup a real world working environment? Think hours, not weeks.

Day 1 - Build Day 2 - Operate

Operational Challenges with any Platform




bution

BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems.

➔ Packaging w/ embedded OS

➔ Server provisioning on any IaaS

➔ Software deployment across availability zones

➔ Scaling➔ Health monitoring (server AND

processes)

➔ Service state monitoring

➔ Self-healing w/ Resurrector

➔ Storage management

➔ Rolling upgrades with canaries

BOSH




bution

➔ Packaging w/ embedded OS

➔ Server provisioning on any IaaS

➔ Software deployment across availability zones

➔ Scaling➔ Health monitoring (server AND

processes)

➔ Service state monitoring

➔ Self-healing w/ Resurrector

➔ Storage management

➔ Rolling upgrades with canaries

Workeretcdetcd

Kubernetes

etcd

MasterMaster

WorkerWorker

BOSH

BOSH




bution

Uniform way to instantiate, deploy, and manage highly available Kubernetes clusters. On any cloud.

Launched by Pivotal & Google Feb 2017, Donated to Cloud Foundry Foundation June 2017

“Day 1” Build● Deploy Kubernetes cluster via BOSH

“Day 2” Operate● Self-healing VMs and monitoring via

BOSH● Elastic scaling for clusters● Rolling upgrades to latest Kubernetes

release● High-availability and multi-AZ support

Project Kubo




bution

Workeretcdetcd

Kubernetes

etcd

MasterMaster

WorkerWorker

BOSH

Platform team is then

responsible for assembly into

desired clusters

Release

templates

Manifest

Kubo Release

bosh deploy

Kubo Defines a Kubernetes Cluster

32



bution

Provides the control plane for provisioning and managing Kubo releases

Joint development effort between Pivotal, VMWare and Google

Kubernetes Dial Tone:

• Health management

• Aggregated Metrics and Logging

• Autoscaling

• Persistence interface

Control Plane:

• Provisioning Engine

• Self-service Clusters

• Software Update Automation

• Load balancing

• Networking

• Multi-tenancy




bution

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH

VMware PKS: Provisioning Engine




bution

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH

create cluster(with upgrade policy)

VMware PKS: Self-service Clusters




bution

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH

create cluster(with upgrade policy)

Load

Bal

anci

ng

access app

VMware PKS: Dynamic Routing (Different Options Available)




bution

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH

Ro

ute

r

VMware PKS: Dynamic Routing (One Option)




bution

BOSH

BOSH network

Worker Worker

10.0.30.12

ContainerContainerContainer

10.200.2.6C2C overlay 10.200.1.510.200.1.4

10.0.30.11

kube-proxy

iptables

kube-proxy

iptables

Service network

Service

VMware PKS: Networking (Different Options Available)




bution




4 Q & A





bution

PaaS Control Plane

etcd

API-Server

Scheduler

NCM

Infra

Kubernetes

Adapter

CloudFoundry

Adapter

Libnetwork

Adapter

NSX Container Plugin

Mesos

Adapter

NSX

Manager

API Client

Proj: foo Proj: bar

NSX topology for K8s / CF

• NSX Container Plugin (NCP) for integrating with Kubernetes

• NSX Features for K8s PODs

• IP address per container / POD

• Container Network – Routed (BGP) & NATed mode

• Microsegmentation – via K8s Network Policy or native NSX APIs (mapping

K8s labels to NSX tags)

• Network & Security automation – created as part of app deployment

• Multi-tenant network topologies

NSX-T Integration




bution

Need Harbor screenshot

Registry – Enterprise-grade Private Registry

user management & access control

role-based access control

AD/LDAP integration

security

vulnerability scanning

content trust - image signing

policy based image replication

audit and logs

restful API

lightweight & easy deployment

open-source under Apache 2 license




bution

Content Trust enabled via Notary service

Image signed by publisher’s private key during pushing

Image verified using publisher’s public key during pulling

Optionally, unsigned images cannot be pulled

Need Harbor screenshot

Registry – Content Trust, Image Signing and Validation




bution

Registry – Content Trust, When Enabled Un-signed Images Can’t Be Pulled




bution

Vulnerability scanning

Scan on push to registry

Set vulnerability threshold

Optionally, prevent images from being pulled if they exceed threshold

Periodic scanning based on updated vulnerability database

Registry – Image Vulnerability Scanning




bution

Structured Data

Metrics Alerts Events

VMware vRealize

Operations

Capacity, Performance and

Configuration Management Events

Launch in Context

Unstructured Data

Logs Messages

VMware vRealize

Log Insight

Log analytics, aggregation,

and search

Virtual Applications

vRealize Ops, vRealize Log Insight For Comprehensive Visibility




bution

K8S Summary –Nodes, Pods, etc.

K8S Topology -Health

K8S Pods - Health

vRealize Ops – Managing Kubernetes Clusters




bution

K8S Pod Relationship to Components

K8S Alerts

K8S Alerts

vRealize Ops – Kubernetes Integration Details




bution

UI and API Backend

Advanced Analytics Engine

Metrics Collection and Storage

Iterate&TroubleshootIssues

Trend & Alert on Anomalies

Visualize Metrics at Scale

Self-Service Metrics Analytics for All

Engineering & Business

Introducing Wavefront By VMware SaaS-Based Metrics Monitoring and Analytics Platform




bution

App Containers

Docker Host

Docker Swarm

Container Metric Collector

Docker Host

Docker Host

Docker Cluster

AmazonECS

Real-time insight into Docker containers and orchestration

systems Kubernetes, Pivotal Cloud Foundry, Amazon ECS

Wavefront Container Monitoring Suite

49



bution

CATALOGEntitlements, Approvals, Policies

CD PIPELINE

Developers, CI/CD LOB Users

MA

NA

GE

ME

NT

& O

PE

RA

TIO

NS

PRIVATE CLOUDOR DATA CENTER

PUBLICCLOUD

BRANCH/EDGECOMPUTE

APP FRAMEWORKS

PAAS CAAS FAAS

GLOBALLY CONSISTENT INFRASTRUCTURE AS CODE

IAAS ORCHESTRATION

BLUEPRINT

CLOUD APIS

CLOUD APIs

Consume native services

on any cloud

1

BLUEPRINTS & ITERATIVE

DEVELOPMENT

Compose applications using

simplified YAML iteratively

2

INTEGRATED CATALOG

AND PIPELINE

Catalog for self-service &

pipeline for CI/CD

3

vRealize Automation – Simplify Developer Consumption




bution

Physical Infrastructure

VMware PKS

Kubernetes on BOSH (Kubo)

BOSH

NSX

Analytics Automation

SecurityOperations

Monitoring

GCP

Service Brokeretcd worker

Logging

vSANvSphere

etcd workerContainer

Registrymaster master

vRealize

VMware PKS in SDDC Portfolio




bution

Questions?




bution

Registry – Based on Widely-adopted Project Harbor




bution



bution

Registry – Image Vulnerability Scanning Details




bution

What’s New with Containers on SDDC [CNA3045BU]

Brandon Henry, Garmin

Cornelia Davis, Pivotal

Paul Dul, VMware



bution

Challenges in Rolling Your Own Kubernetes Container Service

CLUSTER ADMINISTRATION




• Load Balancing

59CONFIDENTIAL

MONITORING



LOGGING

• Cluster Logging vRLI



SECURITY


• Image Signing


• Credentials Management




bution

vR Ops Log Insight managing Pivotal Cloud Foundry




bution

cna3045bu what’s new containers on sddc or distribution · for the cluster components themselves...

Documents