kubernetes on openstack - hpc advisory council · neutron router vm aka k8s node vm aka k8s node...

© 2018 SWITCH | 1

Saverio Protosaverio.proto@switch.ch

Lugano

Kubernetes on Openstack

© 2018 SWITCH | 2

Infrastructure & Data Services

Our offer

Our customers

Your added value

© 2018 SWITCH | 3

Your added value

SWITCH made – Swiss made

• Swiss law and data location• Scalable storage and computing power on

demand with immediate availability in accordance to the need of – and controlled by –the institutions

• Flexible usage and charging model, no up-front investment

• Simple administration; integrated into the academic network of SWITCH;security and identity services included

• Support for academic use cases • Created together with you

© 2018 SWITCH | 4

Our customers

Higher education• Cantonal universities• ETH domain with research institutions• Universities of applied sciences• Universities of teacher education

University-related organizations• Spin-Offs • Research institutions • eLearning Center • University hospitals

© 2018 SWITCH | 5

Our offer

• SWITCHengines• Virtual Private Cloud (VPC)• SCALE-UP (Project)*

* For developing academic services with 9 universities, as part of the „Scientific Information“ projects mandated by swissuniversities.

© 2018 SWITCH | 6

SWITCHengines

Customer tailored computing and storage performance for universities, research and teaching –further developed in the

SCALE-UP project mandated by swissuniversities.

Your benefits• Your data in Switzerland

• Integrated network and security

• Support for academic use cases

• Simple administration and billing

• Created together with you

Customers• Universities

• Research institutions

• eLearning Center

• University

hospitals

• Spin-Offs

Services• SWITCHengines (IaaS)

• Virtual Private Cloud (VPC)

• SCALE-UP (academic project)

© 2018 SWITCH | 7

Openstack Cloud Engineer at SWITCH. He works in NRENs since 2011, first in Italy and then in Switzerland. He has significant experience in running critical infrastructures using Open Source software. He works together with the Infrastructure & Data team, to deliver an Openstack based cloud to the Swiss Universities.

Saverio Proto

© 2018 SWITCH | 8

Container orchestration.Open Source system to deploy containerized applications.

• What you should already know:• Docker, build docker containers• Run your application in a container

• What Kubernentes will do for me ?• Manage the lifecycle of containers• Schedule the containers to the hosts• Attach storage volumes to the containers• Setup networking

Kubernetes (K8s): what is it ?

© 2018 SWITCH | 9

• K8s can interact with the Openstack API

• Use keystone for authentication• Create volumes• Create load balancers• Set routes in Neutron routers

Kubernetes on Openstack

© 2018 SWITCH | 10

• https://github.com/switch-ch/k8s-on-openstack/• Forked from https://github.com/infraly/k8s-on-openstack

This ansible playbook will create Openstack instances, and will install Kubernetes on them.

export KEY=keynameexport IMAGE="Ubuntu Xenial 16.04 (SWITCHengines)"export NETWORK=k8s( read REAME for complete variables list)

ansible-playbook site.yaml

Deploy K8s on Openstack

© 2018 SWITCH | 11

ubuntu@k8s-master:~$ kubectl get nodesNAME STATUS AGE VERSIONk8s-1 Ready 3m v1.9.4k8s-2 Ready 3m v1.9.4k8s-3 Ready 3m v1.9.4k8s-master Ready 4m v1.9.4

Start !

© 2018 SWITCH | 12

The playbook returns an admin.conf that looks like this:apiVersion: v1

clusters:

- cluster:

certificate-authority-data: [..CUT..]

server: https://<ipaddress>:6443

name: kubernetes

contexts:

- context:

cluster: kubernetes

user: kubernetes-admin

name: kubernetes-admin@kubernetes

current-context: kubernetes-admin@kubernetes

kind: Config

preferences: {}

users:

- name: kubernetes-admin

user:

client-certificate-data: [..CUT..]

client-key-data: [..CUT..]

Client config

© 2018 SWITCH | 13

• kubectl is the command line tool• ~/.kube/config is the config file

• Supports Openstack Keystone authentication• kubectl is able to read usual Openstack OS_ env vars

• kubectl config set-credentials \

osuser --auth-provider=openstack

• source ~/openrc

Client config

© 2018 SWITCH | 14

Create a context to use the Openstack User

kubectl config set-context \

--cluster=kubernetes \

--user=osuser osuser@kubernetes

Lets use the contextkubectl config use-context osuser@kubernetes

Client config

© 2018 SWITCH | 15

Keystone authentication for k8s

Kubectlclient

OpenstackKeystone

KubeApiserver

WebHook

2. Requestwith bearer token

1. Token issue

3. Calling WebHook for token validation

4. Token validation with keystone

© 2018 SWITCH | 16

kube-apiserver--authentication-token-webhook-config-file=webhook.kubeconfig.yaml

---

apiVersion: v1

clusters:

- cluster:

insecure-skip-tls-verify: true

server: https://localhost:8443/webhook

name: webhook

contexts:

- context:

cluster: webhook

user: webhook

name: webhook

current-context: webhook

kind: Config

preferences: {}

users:

- name: webhook

Configure the webhook

© 2018 SWITCH | 17

It is just a container:https://hub.docker.com/r/zioproto/k8s-keystone-auth/

More recent source code can be found here:https://github.com/kubernetes/cloud-provider-openstack

make image-k8s-keystone-auth

Start the webhook container

© 2018 SWITCH | 18

• Keystone tested for Authentication

• Default RBAC for Kubernetes implements Authorization

• Keystone projects are mapped to Kubernetes Groups

Authorization via RBAC

© 2018 SWITCH | 19

/etc/kubernetes/cloud-config[Global]auth-url = https://keystone.cloud.switch.ch:5000/v2.0username = saverio.proto@switch.chpassword = ******tenant-name = saverio.proto@switch.chregion = ZH

[BlockStorage]trust-device-path = falsebs-version = v2

[Route]router-id = 3a6cd142-91cd-4a04-9a86-a73455a0155d

[LoadBalancer]lb-version = v2floating-network-id = 3cc83f7d-9119-475b-ba17-f3510c7902e8subnet-id = 5e18c72c-1902-4846-bd84-ec54cf028375

K8s Master interaction with Openstack

© 2018 SWITCH | 20

/etc/kubernetes/manifests/kube-controller-manager.yaml

spec:

containers:

- command:

- kube-controller-manager

- --cloud-provider=openstack

- --cloud-config=/etc/kubernetes/cloud-config

Interaction with Openstack

© 2018 SWITCH | 21

I changed my the way of doing Networking after the Openstack summit in Sydney

Use --network-plugin=kubenet

• https://github.com/zioproto/k8s-on-openstack/commit/f4506ed202ecc6fc4ff5ac603fd28f3664cb2871

• https://www.openstack.org/videos/sydney-2017/kubernetes-on-openstack-the-technical-details

Kubernetes Neutron Networking

© 2018 SWITCH | 22

Kubernetes Neutron Networking

neutronrouter

VMaka k8s node

VMaka k8s node

Default gateway is the neutron router

Has static routes to reach Pod Networks

Default gateway is the neutron router

Pod Pod

Pod network assigned by the k8s master

Pod Pod

Pod network assigned by the k8s master

© 2018 SWITCH | 23

• Docker containers run into Openstack instances• Persistent storage is cinder volumes

• K8s will attach the Cinder Volume to the right Openstackinstance were the docker container is scheduled

volumes:

- name: mysql-persistent-storage

cinder:

volumeID: <uuid>

fsType: ext4

Persistent Storage

© 2018 SWITCH | 24

• This method is not practical• we have to manage Cinder volumes on the side• Export UUID

• Solution• Define a Storage Class

Persistent storage

© 2018 SWITCH | 25

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

name: defaultannotations:

storageclass.beta.kubernetes.io/is-default-class: "true"

provisioner: kubernetes.io/cinder

parameters:

type: fast

availability: nova

Storage Class

© 2018 SWITCH | 26

kind: PersistentVolumeClaimapiVersion: v1metadata:

name: mysql-pvcannotations:

volume.beta.kubernetes.io/storage-class: defaultspec:

accessModes:- ReadWriteOnce

resources:requests:

storage: 30Gi

Persistent Storage pvc

© 2018 SWITCH | 27

volumes:

- name: mysql-persistent-storage

persistentVolumeClaim:

claimName: mysql-pvc

Persistent Storage with pvc

© 2018 SWITCH | 28

I suggest not to use it, at least in Openstack Newton

• Both DNAT and SNAT are necessary• you lose the client IP information

• IPv6 support status is unknown to me

• It is a pure L4 loadbalancer, cannot terminate TLS connections or rewrite HTTP headers

Openstack LBaaSv2 with K8s

© 2018 SWITCH | 29

Openstack LBaaSv2

neutronrouter

VMaka k8s node

VMaka k8s node

Default gateway is the neutron router

Default gateway is the neutron router

Pod Pod

Pod network assigned by the k8s master

Pod Pod

Pod network assigned by the k8s master

neutronLBaaS

Svc FloatingIP configured here

Openstack project network

Both DNAT and SNAT are necessary here.

© 2018 SWITCH | 30

• An API object that manages external access to the services in a cluster, typically HTTP.

• Ingress can provide load balancing, TLS termination and name-based virtual hosting.

• https://kubernetes.io/docs/concepts/services-networking/ingress/

• At least two known implementations• Nginx based• HAProxy

Kubernetes NGINX Ingress

© 2018 SWITCH | 31

Kubernetes NGINX Ingress

VM

aka k8s node

VM

aka k8s node

Pod Pod

Pod network assigned

by the k8s master

Pod Pod

Pod network assigned

by the k8s master

VM k8s

master

Kubernetes-nginx-ingress

Default network namespace

(docker –net=host)

• It is a special privileged Docker container running NGINX

• On the k8s master (s) or dedicated VM

• Has access to external IP addresses and cluster IP addresses

Kubernetes cluster network

© 2018 SWITCH | 32

How I use the Ingress ?

© 2018 SWITCH | 33

Slack channel #sig-openstack• Join via http://slack.k8s.io/

Follow our cloud blog:• https://cloudblog.switch.ch/

Get in touch with me• saverio.proto@switch.ch

Getting involved

© 2018 SWITCH | 34

Thank you ! Questions ?