deploying kubernetes in the enterprise with cisco aci · container orchestration basic features
TRANSCRIPT
Deploying Kubernetes in the Enterprise with Cisco ACI
Camillo Rossi – TME INSBU
BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Spark
Questions? Use Cisco Spark to communicate with the speaker after the session
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
How
cs.co/ciscolivebot#BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objectives
At the end of the session, the participants should be able to:
• Have a general understanding of containers
• Have a general understanding of Kubernetes
• Understand how ACI and Kubernetes integration is deployed
Initial assumption:
• The audience already has a good knowledge of ACI main concepts (Tenant, BD, EPG, L2Out, L3Out, etc.)
4BRKACI-2505
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
Agenda
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI Kubernetes Integration – Value proposition
• Allow containers direct access to the ACI policy model, so that they can participate as first-class citizens within an ACI fabric
• Allow seamless integration of containers, VMs, and physical devices on an ACI fabric
• Support native policy semantics, so that a container application that is specified using Kubernetes NetworkPolicy will work correctly out of the box
• I.E. The same config works on Google Cloud, AWS and ACI
• Leverage fabric resources and Opflex to assist accelerating Kubernetes service load balancing
6BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI Kubernetes Integration – Value proposition (cont.)
• Security Policies within and between pods are defined using Kubernetes Network Security API. Implemented on OVS via Host Protection Profiles, visible by APIC
• ACI admin can (optionally) define EPGs and contracts that are exposed to Kubernetes users
• EPG is selected based on annotation, which can also be used to dynamically define new EPGs
7BRKACI-2505
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
Agenda
Linux fundamentalsUnderstanding Containers Origins
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Lightweight Process Virtualization is not new
• Create multiple views of the root filesystem to isolate applications and processes
• Lightweight Process Virtualization is not a new concept:
• Solaris Zones
• BSD jails
• Linux chroot
• AIX WPARs (Workload Partitions)
10BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is new?
• Linux Kernel 3.8 (February 2013) added support for namespaces and cgroups
• Namespaces are like chroot for processes, but also applied to network, UTS (Unix Timesharing), mount, IPC and users (UIDs).
• Example: Network namespaces enable the creation of multiple, isolated routing tables that operate independently. Multiple hostnames could also be used: • #ip netns add myns1 -> Create a new Namespace “myns1”
BRKACI-2505 11
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is new?
• Cgroups provide resource management capabilities
• Processes can be grouped into user-defined group of tasks, for optimized system resource usage
• Cgroups move resource allocation from the process level to the application level by grouping and labeling processes into hierarchies
• Resource allocation includes CPU time, block IO, RAM and network bandwidth
BRKACI-2505 14
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is new?
• Cgroups architecture:
CPU Network Memory Storage I/O
Cgroup1
Cgroup2
BRKACI-2505 15
What are containers?
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is a container?
• A container is a binary executable, packaged with dependencies and intended for execution in a private namespace with optional resource constraints.
• This provides the containers multiple isolated operating system environments with their own file system, network, process and block I/O space on the same host
BRKACI-2505 17
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Compute Virtualization != Containers
Host OS
Hypervisor
App App App
Server Hardware
Host OS
Container Engine (Linux)
Guest
OS
Guest
OS
Guest
OS
VMs
Containers
Bins/
Libs
Bins/
Libs
Bins/
Libs
Server Hardware
App App App
Bins/
Libs
Bins/
Libs
Bins/
Libs
Virtualization Containers
BRKACI-2505 18
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19BRKACI-2505
Compute Virtualization & Containers
• They provide a way to abstract resources
• They define logical boundaries to the resources they consume
• They enable multiple OS instances to run on the same host
• They share the resources of the host
Similarities
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20BRKACI-2505
Compute Virtualization & Containers
• Containers can only run the same OS as the host
• Containers share the same kernel as the host
• Containers are faster to provision and boot
• Containers have lower overhead as there is no need for the hypervisor layer
Differences
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers Current Challenges
• Containers images management
• Orchestration of containers across multiple hosts
• Lack of standards
• Integration with virtualization and cloud tools
• Networking management
21BRKACI-2505
Addressed by ACI
Why Containers?
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Microservice
Single Purpose
Stateless
Independently Scalable
Automated
Service
Autonomous
Loosely-coupled
Function
Single Action
Event Sourced
Ephemeral
Application Architectural Evolution
23BRKACI-2505
f()
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
An effective platform for micro-services
• Containers are ideal candidates to run micro-services:
• Micro-services define stateless, loosely coupled application components communicating over API’s, running in different runtime environments.
• Containers meet new application requirements as they provide:• Density
• Speed
• Portability
• Low overhead management
24BRKACI-2505
Containers runtimes
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers runtimes
• A container runtime enables users to make effective use of containerization mechanisms by providing APIs and tooling that abstract the low level technical details
• LXC - Open Source Solutions (OSS)
• Docker – OSS and commercial
• Rkt – part of CoreOS, OSS and commercial
• VMware Integrated Container (aka Project Bonneville) - Proprietary
• RunC - OSS
• Garden – part of Pivotal Cloud Foundry, OSS and commercial
BRKACI-2505
Docker
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker provides an integrated technology suite that enables development and IT operations teams to build, ship, and run distributed applications anywhere.
30BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
A Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries –anything that can be installed on a server.
This guarantees that the software will always run the same, regardless of its environment.
31BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker consists of two main components:
33BRKACI-2505
Docker Hub – SaaS component for managing
and sharing containers.
Docker Engine – the
actual app running
on the host.
Containers
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Dockerfile and Registry
• A Dockerfile is simply a text file containing instructions on how to build a Docker image
• It can add components on top of an existing image
• Images are available online on the Docker hub repository
• Local, private registry can be created.
• A registry is an instance of the registry container image
34BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Dockerfile and Registry
• A Dockerfile is simply a text file containing instructions on how to build a Docker image
• It can add components on top of an existing image
• Images are available online on the Docker hub repository
• Local, private registry can be created.
• A registry is an instance of the registry container image
35BRKACI-2505
More about Docker Images…
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Images
• A Docker image is made up of filesystems layered over each other.
37BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Images
• The storage driver is responsible for presenting these layers as a single, unified file system.
38BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Images
• When you start a container, Docker creates an empty, read-write layer on top of the stack – all changes are made in this layer.
39BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Images
• Docker uses “copy-on-write” container layers.
• If a file needs to be modified, it is copied into the read-write layer first.
40BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• This means that multiple containers can share a single copy of the image.
41BRKACI-2505
Docker Networking
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 1 - None
• Doesn’t create any network interface for the container
43BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 2 - Bridge Mode
• Default mode where Docker attaches containers to Docker0 bridge
• Containers in the same host can talk to each other
• Containers on different host can’t talk to each other (or anything else) easily
44BRKACI-2505
Host-2
Container 3
docker0 Bridge
172.17.42.1
Eth0172.17.0.12
veth994786d
Container 4
Eth0172.17.0.13
vethab4e22e
Host-1
Container 1
docker0 Bridge
172.17.42.1
Eth0172.17.0.12
veth774786d
Container 2
Eth0172.17.0.13
vethde4e22e
Same IPs for the containers is not a typo
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 2 - Bridge Mode
• It’s possible to expose a container externally by publishing a port (NAT)
45BRKACI-2505
Host
Container 1
NGINX
docker0 Bridge
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 3 - Host Mode
• Connect containers to Host network stack
• All the network interfaces defined on the host will be available to the container (every container will have the same IP address as the host)
46BRKACI-2505
Host-1
Container 1
Eth0 (host interface)
192.168.0.2
Eth0192.168.0.2
Container 2
Eth0192.168.0.2
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 4 - Mapped Container Mode
• Container is mapped to another container network stack
• Filesystem, processes and other resources are kept separate
• They share network resources (IP, interfaces)
47BRKACI-2505
Host-1
Container 1
docker0 Bridge
172.17.42.1
Eth0172.17.0.12
veth774786d
Container 2
Eth0172.17.0.12
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
All these Docker Networking options are complex
• Iptables rules must be created manually to allow/NAT traffic to containers ports…
• Containers on different hosts can’t communicated to each other even if they are in the same L2 domain
• Needs to manually manage port-mappings
• Prone to errors
48BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker network driver plugins
• Network plugins can be used to extended Docker networking support to a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN or something completely different.
49BRKACI-2505
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
Agenda
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
In a multi-host environment, containers need to
• Have network reachability
• Be fault-tolerant
• Easily scalable
• Use resources optimally
• Can discover other containers/application automatically
• Communicate with each other
• Can be update/rollback without any downtime
• Expose services in an easy and reliable way
53BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Orchestration basic features
• Bring multiple hosts together and make them part of a cluster
• Schedule containers to run on different hosts
• Help containers running on one host reach out to containers running on other hosts in the cluster
• Bind containers and storage
• Bind containers of similar type to a higher-level construct, like services, so we don't have to deal with individual containers
• Keep resource usage in-check, and optimize it when necessary
• Allow secure access to applications running inside containers.
BRKACI-2505 54
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers Orchestrators
• Docker Swarm is a Container Orchestrator provided by Docker, Inc. It is part of Docker Engine.
• Kubernetes started by Google, now part of the Cloud Native Computing Foundation project.
• Mesos Marathon is one of the frameworks to run containers at scale on Apache Mesos.
• Amazon EC2 Container Service (ECS) is a hosted service provided by AWS to run Docker containers at scale on its infrastructure.
• Hashicorp Nomad is the Container Orchestrator provided by HashiCorp.
BRKACI-2505 55
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes
• Kubernetes is an open source Container Orchestration system for automating deployment, scaling and management of containerized applications.
• It was inspired by the Google Borg System and with its v1.0 release in July 2015, Google donated it to the Cloud Native Computing Foundation (CNCF).
• Generally, Kubernetes has new releases every three months. The current stable version is 1.9 (as of Jan 2018).
BRKACI-2505 56
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes & Docker
• Kubernetes uses Docker to execute/run the containers
• Kubernetes adds, on top of Docker, all the intelligence and features of an orchestrator
BRKACI-2505 57
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Features
• Automatic binpackingKubernetes automatically schedules the containers based on resource usage and constraints, without sacrificing availability.
• Self-healingKubernetes automatically replaces and reschedules the containers from failed nodes. It also kills and restarts containers which do not respond to health checks, based on existing rules/policy.
• Horizontal scalingKubernetes can automatically scale applications based on resource usage like CPU and memory. It also supports dynamic scaling based on customer metrics
BRKACI-2505 58
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Features (cont.)
• Service discovery and Load balancingKubernetes groups sets of containers and refers to them via a DNS name. This DNS name is also called a Kubernetes service. Kubernetes can discover these services automatically, and load-balance requests between containers of a given service.
• Automated rollouts and rollbacksKubernetes can roll out and roll back new versions/configurations of an application, without introducing any downtime.
• Secrets and configuration managementKubernetes can manage secrets and configuration details for an application without re-building the respective images. With secrets, we can share confidential information to our application without exposing it to the stack configuration, like on GitHub.
BRKACI-2505 59
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Features (cont.)
• Storage orchestrationWith Kubernetes and its plugins, we can automatically mount local, external, and storage solutions to the containers in a seamless manner, based on Software Defined Storage (SDS).
• Batch executionBesides long running jobs, Kubernetes also supports batch execution.
BRKACI-2505 60
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Architecture
• At a very high level, Kubernetes has the following main components:
• One or more Master Nodes
• One or more Worker Nodes
• Distributed key-value store, like etcd.
BRKACI-2505 61
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Components – Master Node
• The Master Node is responsible for managing the Kubernetes cluster. Master node access methods are CLI, GUI or APIs.
• For fault tolerance, there can be more than one Master Node.
• To manage the cluster state, Kubernetes uses etcd, and all Master Nodes connect to it. etcd is a distributed key-value store. The key-value store can be part of the Master Node. It can also be configured externally, in which case, the Master Nodes connect to it.
BRKACI-2505 63
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Components – Worker Node
• A Worker Node is a machine (VM, physical server, etc.) which runs the containers using pods and is controlled by the Master Node.
• pods are scheduled on the Worker Nodes
BRKACI-2505 64
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes - pod
• A pod is the scheduling unit in Kubernetes. It is a logical collection of one or more containers which are always scheduled together.
• The set of containers composed together in a pod share an IP.
[root@k8s-01-p1 ~]# kubectl get pod --namespace=kube-system
NAME READY STATUS RESTARTS AGE
aci-containers-controller-1201600828-qsw5g 1/1 Running 1 69d
aci-containers-host-lt9kl 3/3 Running 0 72d
aci-containers-host-xnwkr 3/3 Running 0 58d
aci-containers-openvswitch-0rjbw 1/1 Running 0 58d
aci-containers-openvswitch-7j1h5 1/1 Running 0 72d
BRKACI-2505 65
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Deployment
• Deployments are a collection of pods providing the same service
• You describe the desired state in a Deployment object, and the Deployment controller will change the actual state to the desired state at a controlled rate for you
• For example you can create a deployment that declare you need to have 2 copies of your front-end pod.
[root@k8s-01-p1 ~]# kubectl get deployment --namespace=kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
aci-containers-controller 1 1 1 1 72d
BRKACI-2505 66
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Services
• A service tells the rest of the Kubernetes environment (including other pods and Deployments) what services your application provides.
• While pods come and go, the service IP addresses and ports remain the same.
• Kubernetes automatically load balance the load across the replicas in the deployment that you expose through a Service
• Other applications can find your service through Kurbernetes service discovery.
• Every time a service is create a DNS entry is added to kube-dns
[root@k8s-01-p1 ~]# kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 11.96.0.10 <none> 53/UDP,53/TCP 72d
BRKACI-2505 67
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – External Services
• If there are external IPs that route to one or more cluster nodes, Kubernetes services can be exposed on those external IPs.
• Traffic that ingresses into the cluster with the external IP (as destination IP), on the service port, will be routed to one of the service endpoints.
• External IPs are not managed by Kubernetes and are the responsibility of the cluster administrator.
[root@k8s-01-p1 ~]# kubectl get svc front-end --namespace=guest-book
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
front-end 11.96.0.33 11.3.0.2 80:30002/TCP 3m
BRKACI-2505 68
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Ingress
• An Ingress is a collection of rules that allow inbound connections to reach the cluster services.
• It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting, and more
• Think of NGINX
[root@k8s-01-p1 ~]# kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
test-ingress * 80 7s
BRKACI-2505 69
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes - Labels
• Kubernetes uses labels as “nametags” to identify things.
• Can be used to indicate roles, stability, or other important attributes.
• You can query anything in Kubernetes via a label.
• i.e. Return all the pod that are running “PreProduction” workload
[root@k8s-01-p1 ~]# kubectl get pod --namespace=kube-system -l component=kube-apiserver
NAME READY STATUS RESTARTS AGE
kube-apiserver-k8s-01-p1 1/1 Running 0 72d
BRKACI-2505 70
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes - Annotations
• Similar to labels but are NOT used to identify and select object
• Used in ACI, yes soon we will be speaking about ACI and Kubernetes
[root@k8s-01-p1 ~]# kubectl describe node k8s-01-p1 | more
Name: k8s-01-p1
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=k8s-01-p1
node-role.kubernetes.io/master=
Annotations: node.alpha.kubernetes.io/ttl=0
opflex.cisco.com/pod-network-ranges={"V4":[{"start":"11.2.0.130","end":"11.2.1.1"}]}
opflex.cisco.com/service-endpoint={"mac":"66:85:9a:e9:ef:2f","ipv4":"11.5.0.3"}
volumes.kubernetes.io/controller-managed-attach-detach=true
BRKACI-2505 71
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Namespace
• Groups everything together:
• Pod
• Deployment
• Volumes
• Services
• Etc…
BRKACI-2505 72
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
All Together: A K8S Cluster
Container
Application
pod1
Deployment1
Container
Application
pod2
Container
Application
pod[n]
Node1
Node[N]
Node2Service
1.1.1.1:80
Namespace
A node can be part of
Several Namespaces
BRKACI-2505 73
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
Agenda
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Application Centric InfrastructureAny Application – Any hypervisor
• Policy consistency provides for
containers running reliably and
securely
• Build to and support open systems
and standards
• Common pervasive gateway and
policy based routing provide optimal
network connectivity
• Ease of deploying, scaling and
managing
BRKACI-2505 75
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco ACI and Container Integration
Node
OpFlex OVS
ACI and Containers
Unified networking: Containers, VMs, and
bare-metal
Micro-services load balancing integrated in
fabric for HA / performance
Secure multi-tenancy and seamless
integration of Kubernetes network policies
and ACI policies
Visibility: Live statistics in APIC per
container and health metricsNode
OpFlex OVS
BRKACI-2505 76
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
`
1
2 Deploy/Scale Clusters
Build containers
WEB APP WEB APP DB
Server 1 Server 2
Opflex/OVS
Host level Policy
Enforcement
ACI Fabric
Define BDs, Context and
AP
Fabric Bring Up
2
1
Get VLAN Pools Allocated
For Each EPG3
Full Infrastructure Visibility,
Telemetry4
Annotate policy3EPG
Developer Network Administrator
Infosec
Define Container
Network Policy
Infrastructure Policy
Enforcement
1
77BRKACI-2505
ACI Network Plugin for KubernetesNative Security Policy Support
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI VMM Domain for Kubernetes
78BRKACI-2505
Node
OpFlex OVS
Kubernetes
ACI Policies
Technical DescriptionNetwork Policy
Node
OpFlex OVS
• Network policies of Kubernetes supported using standard
upstream format but enforced through OpFlex / OVS using
APIC Host Protection Profiles
• Kubernetes app configurations can be moved without
modification to/from ACI and non-ACI environments
• Embedded fabric and virtual switch load balancing
• PBR in fabric for external service load balancing
• OVS used for internal service load balancing
• VMM Domain for Kubernetes
• Stats per namespace, deployment, service, pod
• Physical to container correlation
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI CNI Plugin Components
• aci-containers-controller
• Handle IPAM
• Management of endpoint state
• Policy Mapping (annotations)
• Controls Load Balancing
• Pushes configurations into the APIC
79BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI CNI Plugin Components
• aci-containers-host is a DaemonSet composed of 3 containers:
• mcast-daemon: • Handles Broadcast, unknown unicast and multicast replication
• aci-containers-host:• Endpoint metadata
• Pod IP Address management
• Container Interface Configuration
• opflex-agent:• Support for Stateful Security Groups
• Manage configuration of OVS
• Render policy to openflow rules to program OVS.
• Handles loadbalanced services (connection tracking, natting, etc…)
80BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI CNI Plugin Components
• aci-containers-openvswitch
• Bridge traffic from containers to physical interfaces
• Enforce policies
81BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82BRKACI-2505
Flexible Definition of EPG Boundaries
Cluster Isolation Namespace Isolation Deployment Isolation
pod
pod
pod
pod
pod
pod
• Single EPG for entire cluster,
default behavior
• No need for any internal contracts
• Each namespace is mapped to its
own EPG
• Contracts for inter-namespace traffic
• Each deployment mapped to an EPG
• Contracts tightly control service traffic
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
Contra
ct
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
Contra
ct
Contract
Contract
Contra
ct
EPG
Network
Policy
ACI Basic Configuration
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
acc-provision
• ACI Container Controller Provision:
• Takes a YAML file containing the parameters of your configuration
• Generates and pushes most of the ACI config
• Generates Kubernetes ACI CNI containers configuration
acc-provision --flavor=kubernetes-1.7 -a -u admin -p pass –c config.yml –o cni_config.yml
84BRKACI-2505
Used to select if we are deploying
kubernetes 1.6, 1.7 or OpenShift 3.6
APIC user and
passwordConfiguration file
Output file for ACI CNI
config
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
acc-provision – configuration file (1)
aci_config:
system_id: KubeSpray # Tenant Name and Controller Domain Name
apic_hosts: # List of APIC hosts to connect for APIC API
- 10.67.185.102
vmm_domain: # Kubernetes VMM domain configuration
encap_type: vxlan # Encap mode: vxlan or vlan
mcast_range: # mcast range for BUM replication
start: 225.22.1.1
end: 225.22.255.255
mcast_fabric: 225.1.2.4
nested_inside: # (OPTIONAL) If running k8s node as VMs specify the VMM Type and Name.
type: vmware # Only vmware for now, ports groups created automatically with system_id name
name: ACI
# The following resources must already exist on the APIC,
# they are used, but not created by the provisioning tool.
aep: ACI_AttEntityP # The AEP for ports/VPCs used by this cluster
vrf: # The VRF can be placed in the same Tenant or in Common.
name: vrf1
tenant: KubeSpray # This can be the system-id or common
l3out:
name: l3out # Used to provision external IPs
external_networks:
- default_extepg # Default Ext EPG, used for PBR redirection
85BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
acc-provision – configuration file (2)
#
# Networks used by Kubernetes
#
net_config:
node_subnet: 10.32.0.1/16 # Subnet to use for nodes
pod_subnet: 10.33.0.1/16 # Subnet to use for Kubernetes Pods
extern_dynamic: 10.34.0.1/24 # Subnet to use for dynamic external IPs
extern_static: 10.35.0.1/24 # Subnet to use for static external IPs
node_svc_subnet: 10.36.0.1/24 # Subnet to use for service graph
kubeapi_vlan: 4011 # The VLAN used by for nodes to node API communications
service_vlan: 4013 # The VLAN used by LoadBalancer services
infra_vlan: 3456 # The ACI infra VLAN used to establish the OpFlex tunnel with the leaf
#
# Configuration for container registry
# Update if a custom container registry has been setup
#
registry:
image_prefix: noiro # DO NOT CHANGE
86BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 87BRKACI-2505
ACI Fabric Configuration - EPGs
• kube-nodes: node API communication• Physical Domain
• kube-system: k8s components running as pods• Container Domain*
• kube-default: All user created pods will be placed in this EPG by default• Container Domain*
* Equivalent of VMM Domain for Containers
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88BRKACI-2505
ACI Fabric Configuration – BDs and VRF
• kube-nodes-bd:• Only used for kube-node EPG
• Maps to node_subnet
• kube-pod-bd: • Any pod will be assigned an IP from
this BD Subnet
• Used for kube-default, kube-system and any other user defined EPGs.
• Maps to pod_subnet
• KubeSpray…-service:• BD for PBR/SG services
• Created when ACI CNI plugin is deployed
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 89BRKACI-2505
ACI Fabric Configuration – L3OUT
• User must create and configure the L3OUT
• L3OUT and Default Networks names must match the acc-provision config
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 90BRKACI-2505
ACI Fabric Configuration – Contracts
• Minimum sets configured to ensure basic cluster functionality
• User can define additional contracts as usual
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91BRKACI-2505
ACI Fabric Configuration – L4-L7 Devices
• Created once the ACI CNI plugin is deployed
• Dynamically updated if nodes are added or removed from the k8s cluster
• Service Graph Template: Specify a template for PBR redirection
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI Fabric Configuration – Container Domain
APIC keeps inventory of pods
and their metadata (labels,
annotations), deployments,
replicasets, etc.
View pods per node, map to
encapsulation, physical point
in the fabric.
Fabric admin can search
APIC for k8s nodes, masters,
pods, services …
BRKACI-2505 92
Demo 1 Deploying an Application
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Spin up a guestbook application
• The guestbook application uses Redis to store its data. It writes its data to a Redis master instance and reads data from multiple Redis slave instances.
• The code can be found at: https://kubernetes.io/docs/tutorials/stateless-application/guestbook/
94BRKACI-2505
For your
reference
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Spin up a guestbook application
• Create a namespace for our application
cisco@k8s-01:~/demo/guestbook1$ kubectl create namespace guestbook
• Deploy all the componentscisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook apply -f complete.yaml
deployment "frontend" created
service "frontend" created
deployment "redis-master" created
service "redis-master" created
deployment "redis-slave" created
service "redis-slave" created
• Check POD status cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get pod -o wide
READY STATUS RESTARTS AGE IP NODE
frontend-1768566195-mj43h 1/1 Running 0 2m 10.33.1.11 k8s-02
frontend-1768566195-tpw75 1/1 Running 0 2m 10.33.0.153 k8s-03
frontend-1768566195-vljrh 1/1 Running 0 2m 10.33.0.155 k8s-03
redis-master-2365125485-8hg60 1/1 Running 0 2m 10.33.0.152 k8s-03
redis-slave-3837281623-p4fs7 1/1 Running 0 2m 10.33.1.12 k8s-02
redis-slave-3837281623-qw894 1/1 Running 0 2m 10.33.0.154 k8s-03
Note: All the commands are executed from the Kubernetes master node95BRKACI-2505
For your
reference
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Check APIC Controller Domain
96BRKACI-2505
APIC has complete visibility into
k8s objects
Visibility on where this specific
pod is running
For your
reference
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Check APIC EPG
97BRKACI-2505
POD Name
For your
reference
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Cluster Services
• By default every POD is exposed only to the k8s cluster via a Service IP. You can imagine this as a Virtual IP of a load balancer.
• With the ACI CNI plugin the LoadBalancing for the internal cluster services is performed by OVS cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
frontend 10.37.0.124 <nodes> 80:32677/TCP 25m
redis-master 10.37.0.162 <none> 6379/TCP 25m
redis-slave 10.37.0.136 <none> 6379/TCP 25m
• Try to access the service from one of the nodescisco@k8s-01:~/demo/guestbook1$ curl 10.37.0.124
<html ng-app="redis">
<head>
<title>Guestbook</title>
!SNIP!
98BRKACI-2505
For your
reference
Demo 2 Placing PODs/Namespaces in EPGs
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 2 - APIC Steps
• Create an EPG under your application
• BD = your pod BD
• VMM Domain = Your Kubernetes Domain
• Every POD in an EPG needs to be able to communicate with:
• kube-system for cluster wide DNS resolution
• kube-node for health monitoring probes
• Top Tip: Use EPG contract masters and inherit contracts from kube-default!
100BRKACI-2505
For your
reference
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 2 - acikubectl
• Utility to manage and troubleshoot the k8s cluster
• Can be used to annotate Namespeces or Deployments with the Tenant/App/EPG names
cisco@k8s-01:~/demo/guestbook1$ acikubectl set default-eg namespace guestbook -t KubeSpray -a kubernetes -g
guestbook
Setting default endpoint group:
Endpoint Group:
Tenant: KubeSpray
App profile: kubernetes
Endpoint group: guestbook
BRKACI-2505
For your
reference
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Check under your EPG
• All your PODs should now have moved from kube-default to guestbook
102BRKACI-2505
For your
reference
Exposing Services
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
The extern_dynamic subnet
• Defined in acc-provision configuration file
• An IP address will be automatically selected from this subnet to expose your service outside of the k8s cluster/fabric
• Expose the service as “LoadBalancer” (as per kubernetes standard)
• The extern_dynamic subnet is not associated to a BD: You need to configure your external router with static routes toward your L3OUT for this subnet
104BRKACI-2505
cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get svc frontend
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
frontend 10.37.0.124 10.34.0.5 80:32677/TCP 5h
extern_dynamic
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR
105BRKACI-2505
* defined in the acc-provision config file
Every time a service is exposed the ACI CNI controller will deploy:
• An External EPG with a /32 match for the Service IP
• A new contract between the svc_ExtEPG and the default_ExtEPG*
• A Service Graph with PBR redirection containing every node where an exposed POD is running
L3Out
default_ExtEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
RTRClient
Cons
Prov
Node1
Node2
NodeN
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
106BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
RTRClient
Cons
Prov
1. Client send a request to 10.34.0.2, ACI performs Longest Prefix Match (LPM) on the SIP and classify the traffic in the default_extEPG
Node1
Node2
NodeN
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
107BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
Node1
Node2
NodeN
RTRClient
Cons
Prov
1. Client send a request to 10.34.0.5, ACI performs Longest Prefix Match (LPM) on the SIP and classify the traffic in the default_extEPG
2. ACI does a routing lookup for 10.34.0.5, IP does not exist in the fabric, we should route it out however
LPM places it in the Svc_x_ExtEPG
3. PBR redirection is triggered and the traffic is LoadBalanced by the fabric to one of the nodes
SIP DIP
192.168.1.100 10.34.0.5
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
108BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
Node1
Node2
NodeN
RTRClient
Cons
Prov
1. Client send a request to 10.34.0.2, ACI performs Longest Prefix Match (LPM) on the SIP and classify the traffic in the default_extEPG
2. ACI does a routing lookup for 10.34.0.2, IP does not exist in the fabric, we should route it out however
LPM places it in the Svc_x_ExtEPG
3. PBR redirection is triggered and the traffic is LoadBalanced by the fabric to one of the nodes
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
SIP DIP
192.168.1.100 10.34.0.5
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
109BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
RTRClient
Cons
Prov
4. The K8S node is not expecting any traffic directed to the external service IP so OVS will perform NAT as required
5. If there are multiple POD on a single node OVS will perform a second stage LB to distribute the load between Pods running on the same node
Node1
Node2
NodeN
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
SIP DIP
192.168.1.100 PodX IP
DIP
192.168.1.100 10.34.0.5
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
110BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
RTRClient
Cons
Prov
4. PodX replies to the client
5. OVS restore the original external Service IP
6. PBR redirection is not triggered since the source EPG is the Shadow EPG of the PBR node
7. Traffic is routed back to the client (and is permitted by the contract)
DIP SIP
192.168.1.100 10.34.0.5Node1
Node2
NodeN
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
DIP SIP
192.168.1.100 PodX IP
Demo 3 Exposing Services
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Exposing a service
• Simply choose the LoadBalancer ”type” in the service definition
• The ACI CNI plug in will:
• Automatically pick a free IP from the extern_dynamic subnet
• Create the ExtEPG
• Create contracts
• Create PBR redirection rules
• Deploy the service graph
112BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Scalability
• Currently the scalability of exposing external service with PBR is limited by the number of external EPGs per L3OUT.
• ACI 3.1 supports 250 external EPGs per L3 OUT per leaf *
• This is a soft limit and will increase with time
• But we want more! So?
113BRKACI-2505
*For details check:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/3-x/verified_scalabilty/b_Verified_Scalability_3_1_1x_and_13_1_1x.html
Scaling External Services with Ingress
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Ingress
• Composed of two parts:
• Ingress Resources: collection of rules that defines how inbound connections can reach the internal cluster services.
• Ingress controller: responsible for fulfilling the Ingress, usually with a virtual loadbalancer (nginx, ha-proxy)
• Ingress controller can be shared between multiple namespaces
• It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting etc…
• Bonus Feature: Easy integration with DNS: configure a wildcard DNS record (*.cisco.com) pointing to the IP of the ingress controller
115BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ingress Cont 1
Ingress Cont 2
Ingress Cont N
Namespace
ingress
Kubernetes – Ingress
116BRKACI-2505
Pod1
Pod2
Namespace
ciscolive
Namespace
cisco
Service
Pod1
Pod2
Service
Ingress
Controller
Ingress Resource
(I am www.cisco.com)
Ingress Resource
(I am www.ciscolive.com)
Clientwww.cisco.com
www.ciscolive.com
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI and Kubernetes Ingress
• Expose the Ingress Controller via Service Graph with PBR
• A single Service Graph/ExtEPG can now host as many services as we want
• Ingress Controller can be scaled (creating multiple instances) as needed
• If you create a dedicated EPG for ingress you need the following contracts:
• All the contracts used in kube-default (remember contract inheritance)
• Consume: Kube-API, Ingress need to be able to speak with the Kube API server
• Consume: any required ports between Ingress Controller and the service you wan to expose
117BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI and Kubernetes Ingress
118BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Ingress_ExtEPG
10.34.0.6/32
Contract PBR Service Graph
RTR
Client
Cons
Prov
Ingress Cont 1
Ingress Cont 2
Ingress Cont N
Pod1
Pod2
Namespace
ingress
Namespace
ciscolive
Namespace
cisco
Service
Pod1
Pod2
Service
Ingress
Controller
Ingress Resource
(I am www.cisco.com)
Ingress Resource
(I am www.ciscolive.com)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI and Kubernetes External Services - Summary
• Two options (can be used at the same time even for the same service)
• Exposing services via ingress
• Exposing up to 250 services directly with Service Graph with PBR
119BRKACI-2505
Demo 4 Exposing Services with Ingress
Kubernetes Cluster Node Failure
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Cluster Node Failure Detection
• Kubernetes Monitors by default all the node in the clusters
• Depending on the configuration, node failure detection and container restart can take from ~50s to 5min. This will depend on your specific configuration.
• Once a node is detected as NotReady (failed) the aci-container-controller will update the ACI configuration as required i.e. a failed node will be removed from the PBR redirection policy
122BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 123BRKACI-2505
ACI CNI redundancy during node failure
• DataPlane of the CNI Plugin
• Start and Stop with the Node
• If isolated from the network they will try to reconnect to the leaf
aci-containers-host & aci-containers-openvswitch
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 124BRKACI-2505
ACI CNI redundancy during node failure
• Stateless
• Does not sit in the data-path
• In case of failure k8s will restart it on a different node
aci-containers-controller (acc)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 125BRKACI-2505
ACI CNI redundancy during node failure – Corner case
• Node connects to OOB and ACI Fabric
• K8S Cluster communications are happening over the ACI Fabric
• acc communicates with APIC via OOB
aci-containers-controller (acc)
Node1
OOB
aci-containers-controller1
ACI
Fabric
Node2
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 126BRKACI-2505
ACI CNI redundancy during node failure – Corner case
• Node 1 losses connectivity with the ACI Fabric (interface down)
• Kubernetes master will detect node1 as lost and restart acc on Node2
• The old instance of acc1 is still running and will keep injecting the old config, overwriting the configuration changes pushed by acc2
• When designing your network ensure that acc communication with the APIC goes trough the fabric
aci-containers-controller
Node1
OOB
aci-containers-controller1
ACI
Fabric
Node2 aci-containers-controller2
Container to Non-Container Communications
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container to Non-Container Communications
• In production environments is preferred, for example, to run services like high performance databases as VMs or Bare Metal Servers
• This calls for the ability to easily provide communication between Kubernetes POD and VMs/Bare Metal
• Simply deploy a contract between your EPGs, ACI will do the rest!
• This works for any VMM domain and Physical Domains, for example you can have a Container Domain using VXLAN speaking with a Microsoft SCVMM Domain using VLAN.
128BRKACI-2505
How can I test this?
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
LABACI-2010 ACI Runs Everything
• WISP Lab
• Come to Build a WordPress application running on:
• Kubernetes
• Hyper-V
• Bare Metal
• ESXi
130BRKACI-2505
S_HAProxy
Percona 1Cluster
BareMetal
Percona 3Cluster
Persona 2 Cluster
L3OUT
NGINXWordPress01
NGINXWordPress02
SCVMM
ESXiBareMetal
Kubernetes
Client
Tenant-Common
Tenant-Common
Tenant-CL18-PODX
How can build my own lab?
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
kubespray_aci!
• Fork of kubespray
• Modified to deploy an high available cluster using ACI CNI plugin
• Open Source (not supported by TAC/Cisco etc…)
• Optionally can clone VM templates and configure everything providing a 1-Click deployment solution for your lab
• https://github.com/camrossi/kubespray_aci
132BRKACI-2505
Not officially
supported
Yes it is me… Did I
mentioned is not
officially supported?
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 133BRKACI-2505
Contiv 1.0 and Contiv-ACIContiv 1.0 in ACI Mode
Docker 1.12+
Support
OpenShift 3.4+
Integration
Physical Domain /
REST API
Contiv-ACI
Open Source at github.com/contiv
Kubernetes 1.6, 1.7
VMM Domain /
OpFlex
OpenShift Integration
Contiv Solutions
Support
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 134BRKACI-2505
Platform Summary for ACIPlatform Contiv-ACI (VMM Domain) Contiv 1.0 (Physical Domain)
GA ACI 3.0 Not recommended
GA ACI 3.1 Supported.
Version 3.4 / RHEL 7.x
Beta ACI 3.1 Not supported
Future Supported.
Engine 1.12.x / Swarm 1.2.5 / Centos 7.x
1.1.6 + Engine 1.12.x / RHEL 7.x
2.1.x + Engine 17.03-ee / RHEL 7.x
Future TBD
Future Future
Kubernetes
Nomad
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Spark
Questions? Use Cisco Spark to communicate with the speaker after the session
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
How
cs.co/ciscolivebot#BRKACI-2505
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Please complete your Online Session Evaluations after each session
• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt
• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.
Complete Your Online Session Evaluation
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
137BRKACI-2505
Thank you