Download - Kubernetes in kubernetes 搭建高可用環境
Copyright 2017 ITRI 工業技術研究院
Agenda
• Cluster Management Issues
• Self-hosted Kubernetes
• Service with High Availability
• Summary
2
Copyright 2017 ITRI 工業技術研究院
Microservices Challenges
3
• A lots of microservices/components
• Zero downtime deployment
• Incremental roll out of features faster
• Improves the ability scale efficiently
Copyright 2017 ITRI 工業技術研究院
Kubernetes Cluster
4
• Scheduling
• Deployment
• Healing
• Discovery/Load balancing
• Scaling
But how about operating Kubernetes?
Copyright 2017 ITRI 工業技術研究院
Kubernetes Operation Tasks
5
• Deployment
• Upgrade / rollback
• Scaling
• Monitoring
• Healing
• Security
• …
• A lot of manual/semi-manual work
• No standard way to approach all the problems
Problems!
http://www.infoq.com/cn/presentations/self-hosted-infrastructure-take-kubernetes-as-an-example
Copyright 2017 ITRI 工業技術研究院
What’s Self-hosted Kubernetes?
6
• Running all required and optional components of a Kubernetes cluster on top of Kubernetes itself
• Kubernetes manages own core components
• Core component deployed as native API objects
https://www.youtube.com/watch?v=EbNxGK9MwN4
Copyright 2017 ITRI 工業技術研究院
Why Self-hosted Kubernetes?
7
• Small Dependencies▪ Reduce the number of components required (Kubelet and Docker)
• Deployment consistency▪ Reduce the number of moving parts relying on the host OS
• Introspection▪ Can be debugged and inspected by users using existing Kubernetes APIs
• Cluster Upgrades▪ Upgrade the components via Kubernetes APIs
• Easier Highly-Available Configurations▪ Easier to scale up and monitor an HA environment without complex external tooling
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md
Copyright 2017 ITRI 工業技術研究院
Launching a Self-hosted Cluster
8
• Need an initial control plane to bootstrap a self-hosted cluster
• Bootkube▪ Provides a temporary control plane to run a full blown self-hosted control
plane
▪ Run only on very first node, then not needed again
https://www.youtube.com/watch?v=EbNxGK9MwN4
Copyright 2017 ITRI 工業技術研究院
Kubernetes Architecture
9
Master node
etcd
SchedulerController
Manager
API Server
Worker node
Kubelet Kube-proxy
dockerPod Pod
Worker node
Kubelet Kube-proxy
dockerPod Pod
Copyright 2017 ITRI 工業技術研究院
Spectrum of Self-hosted Ways
10
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md
• 0-4 self-hosted cluster
• kubelet: daemon set
• API server: pod / service
• 1-4 self-hosted cluster
• system installed Kubelet
• 2-4 self-hosted cluster
• everything except etcd self-hosted
Copyright 2017 ITRI 工業技術研究院
How bootkube works?
11
etcd
Bootkube
API Server
Scheduler
Controller
Manager
Kubelet
Copyright 2017 ITRI 工業技術研究院
How bootkube works? (cont.)
12
etcd
Bootkube
API Server
Scheduler
Controller
Manager
Kubelet
Create
• Deployment
• Daemonset
• Service
• Secret
kubectl
Copyright 2017 ITRI 工業技術研究院
How bootkube works? (cont.)
13
etcd
Bootkube
API Server
Scheduler
Controller
Manager
Kubelet
Pods
API Server
Scheduler
Controller
Manager
create
Copyright 2017 ITRI 工業技術研究院
How bootkube works? (cont.)
14
etcd
Bootkube
API Server
Scheduler
Controller
Manager
Kubelet
Pods
API Server
Scheduler
Controller
Manager
create
Copyright 2017 ITRI 工業技術研究院
Pods
How bootkube works? (cont.)
15
etcd
Kubelet
API Server
Scheduler
Controller
Manager
Copyright 2017 ITRI 工業技術研究院
Self-hosted Control Plane
16
[root@centos7 ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
kube-apiserver-kkxq9 1/1 Running 0 1d
kube-controller-manager-2953862963-t7m1q 1/1 Running 0 1d
kube-controller-manager-2953862963-wlsjp 1/1 Running 0 1d
kube-dns-2431531914-gqnnd 3/3 Running 0 1d
kube-flannel-wnk1j 2/2 Running 0 1d
kube-flannel-xcsx2 2/2 Running 0 1d
kube-flannel-xrksj 2/2 Running 1 1d
kube-proxy-04x11 1/1 Running 0 1d
kube-proxy-11n6t 1/1 Running 0 1d
kube-proxy-1zlgz 1/1 Running 0 1d
kube-scheduler-1873817829-4c7mm 1/1 Running 1 1d
kube-scheduler-1873817829-pmp0n 1/1 Running 0 1d
pod-checkpointer-11q7g 1/1 Running 0 1d
pod-checkpointer-11q7g-10.201.3.6 1/1 Running 0 1d
Copyright 2017 ITRI 工業技術研究院
Self-hosted Control Plane (cont.)
17
[root@centos7 ~]# kubectl get deployment -n kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kube-controller-manager 2 2 2 2 2d
kube-dns 1 1 1 1 2d
kube-scheduler 2 2 2 2 2d
[root@centos7 ~]# kubectl get svc -n kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 2d
[root@centos7 ~]# kubectl get ds -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE
kube-apiserver 1 1 1 1 1 node-role.kubernetes.io/master= 2d
kube-flannel 3 3 3 3 3 <none> 2d
kube-proxy 3 3 3 3 3 <none> 2d
pod-checkpointer 1 1 1 1 1 node-role.kubernetes.io/master= 2d
[root@centos7 ~]# kubectl get secret -n kube-system
NAME TYPE DATA AGE
kube-apiserver Opaque 7 2d
kube-controller-manager Opaque 2 2d
Copyright 2017 ITRI 工業技術研究院
Disaster Recovery
18
• Node failure in HA deployments (Kubernetes)
• Partial loss of control plane components (Kubernetes)
• Power cycling the entire control plane (Kubernetes)
• Permanent loss of control plane (External tool)
http://www.infoq.com/cn/presentations/self-hosted-infrastructure-take-kubernetes-as-an-example
Copyright 2017 ITRI 工業技術研究院
How do we to export the service IP to public network on bare metal?
• In kubernetes we have some existed solution
NodePort
CloudProvider Load Balancer
Ingress
• But it is enough?
23
Copyright 2017 ITRI 工業技術研究院
NodePort
24
kubeProxy kubeProxy kubeProxy
NodePort NodePort NodePort
Pod Pod Pod
Copyright 2017 ITRI 工業技術研究院
NodePort
25
kubeProxy kubeProxy kubeProxy
NodePort NodePort NodePort
Pod Pod Pod
Copyright 2017 ITRI 工業技術研究院
Load Balancer
• Cloud Provider e.g. AWS , GCP, OpenStack
• Load Balancer is created by Cloud Provider, and provide the external IP to for service
• But it is only for Cloud Provider, the bare metal cannot do this
26
Copyright 2017 ITRI 工業技術研究院
Load Balancer
27
kubeProxy kubeProxy kubeProxy
NodePort NodePort NodePort
Cloud
LoadBalancer
Pod Pod Pod
Copyright 2017 ITRI 工業技術研究院
Ingress
29
• HTTP Load Balancing
• SSL Termination
• Content-base Routing
• Not fully for external network
Copyright 2017 ITRI 工業技術研究院
Ingress
30
Ingress
nginx.com echoheaders.com
ServiceA ServiceB
10.0.0.1:80
Copyright 2017 ITRI 工業技術研究院
Ingress
31
Ingress
Controller
Ingress
ResourceLoad Balancer
watches configure
Copyright 2017 ITRI 工業技術研究院
Ingress
32
kubeProxykubeProxy kubeProxy
Pod PodPodPod PodPod
Momo.foo.com
Jojo.foo.comyoyo.foo.com
hostnetwork
Copyright 2017 ITRI 工業技術研究院
Keep-Alived VIP
• Real High-Availability
• Virtual IP Address
• IP to Service
• Configmap
• DaemonSet
35
Ref :
- https://github.com/kubernetes/contrib/tree/master/keepalived-vip
Copyright 2017 ITRI 工業技術研究院
Keep-Alived VIP
36
kubeProxy kubeProxy kubeProxy
Pod PodPod
Keepalived Keepalived Keepalived
140.113.1.1 140.113.1.2 140.113.1.3
Vip: 140.113.1.50
Copyright 2017 ITRI 工業技術研究院
Keep-Alived VIP
37
kubeProxy kubeProxy kubeProxy
PodPodPod
Keepalived Keepalived Keepalived
140.113.1.1 140.113.1.2 140.113.1.3
Vip: 140.113.1.50
Pod
Copyright 2017 ITRI 工業技術研究院
Keep-Alived VIP
38
kubeProxy kubeProxy kubeProxy
PodPodPod
Keepalived Keepalived Keepalived
140.113.1.1 140.113.1.2 140.113.1.3
Vip: 140.113.1.50
Copyright 2017 ITRI 工業技術研究院
Keep-Alived VIP + Ingress
40
kubeProxy kubeProxy kubeProxy
PodPod Pod
Keepalived Keepalived Keepalived
140.113.1.1 140.113.1.2 140.113.1.3
Vip: 140.113.1.50
Momo.foo.com
Jojo.foo.com
Copyright 2017 ITRI 工業技術研究院
Summary
• Self-hosted K8S ▪ Make K8S operations more easier
▪ Bootkube is still a incubator project
▪ Support disaster recovery
• Service with High Availability▪ Using DNS to provide your service – Ingress
▪ More vip – keepalived-VIP
▪ Using cloud to build your kubernetes – cloud Load Balancer
▪ Test – NodePort
41