comment déployeret opérervotre cluster …...le controleplane de kubernetes infrastructure single...
TRANSCRIPT
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Comment déployer et opérer votrecluster Kubernetes sur AWS
Walid Benabderrahmane
Architecte Solutions AWS
M A P 3 0 4
Louis-Paul Dareau
CTO ProcessOut
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda• Déployer un cluster Kubernetes avec Amazon EKS
• Auto Scaling avec Amazon EKS
• Mise à jour de vos clusters avec Amazon EKS
• La Sécurité avec Amazon EKS
• Mise en réseau avancée avec Amazon EKS
• Témoignage ProcessOut
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
L’Architecture d'Amazon EKS
moncluster.eks.amazonaws.com
EKS workers
Kubectl
AZ 1 AZ 2 AZ 3
Votre compte AWS
VPC
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
VPC EKSVPC du Client
Noeuds Worker
ENI EKS
Appels de l'APIKubernetes
Exec, Logs, Proxy
Internet
L’Architecture d'Amazon EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Le controle plane de Kubernetes
Infrastructure single tenant et et à haute disponibilité
Tous les composants "native AWS”
NLB en front
VPC
API Server ASG
Etcd ASG
NLB
AZ-1 AZ-2 AZ-3
ELB
Instances
Instances
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Création d'un cluster
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Création d'un cluster : Amazon Virtual Private Cloud (Amazon VPC)
Fournir tous les sous-réseaux qui hébergeront les ressources Kubernetes : ELBs et nœuds Workers
Les sous-réseaux peuvent être publics, privés ou les deux
Amazon EKS marquera les sous-réseaux aveckubernetes.io/cluster/<cluster-name> = shared
Les sous-réseaux qui hébergeront des répartiteurs de charge internes ont besoin du tag kubernetes.io/role/internal-elb = 1
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Création d'un cluster : groupes de sécurité
Ce groupe de sécurité autorise la connectivité entre le plan de contrôle de Kubernetes et les nœudsworkers.
Au minimum, Kubernetes a besoin du port 443 entraffic entrants et 10250 en sortants.
Ce groupe de sécurité a besoin de permissions qui s'alignent avec le groupe de sécurité du nœudworker.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
VPC EKSVPC du Client
Noeuds Worker
ENI EKS
Appels de l'APIKubernetes
Exec, Logs, Proxy
Internet
L’Architecture d'Amazon EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Utilisez les instances de votre choixFlexibilité de l'instance
✅
✅
✅
✅
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Bring your own OSScripts de construction EKS AMI
https://github.com/awslabs/amazon-eks-ami
Amazon
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AMI EKS optimisée pour le support GPU
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
mycluster.eks.amazonaws.com
Availability Zone 1
Availability Zone 2
Availability Zone 3
Kubectl
VPC
Instance
m5.large Spot Instances
P3.2xlarge Spot Instances
T3.medium On-Demand Instances
Exemple
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Auto Scaling avec Amazon EKS
Deux dimensions possibles
• Les instances Amazon EC2 avec Cluster Autoscaler
• Les Pods avec Horizontal Pod Autoscaler (HPA)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Auto scaling with EKSK8s Cluster Autoscaler• Min, max, Type d’instance• K8s node groups• CA modes
• Auto-discovery• Multi ASG• Single ASG• Master node
• HPA
Amazon EC2 Autoscaling• Min• Max • Type d’instance• Groupe ASG
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Auto scaling workflow
Instance
Instance
Instance
Auto Scaling GroupWorker Nodes
Cluster Autoscaler
Scale UpDesired Capacity +1
Instance
Desired Capacity -1Scale down
Do I still need all these resources?
Do I have enough resources?
Instance
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Versions Kubernetes
Amazon EKS supporte les versions : 1.10; 1.11; 1.12
1.13 disponible bientôt
Amazon EKS supportera jusqu'à trois versions de Kubernetesà la fois
La " dépréciation " empêchera la création de nouveaux clusters sur les anciennes versions
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Versions de plateforme d’Amazon EKS
Les révisions de version de plateforme représentent des changements de configuration de serveur API ou des correctifsKubernetes.
L’incrément des versions de la plateforme est uniquement dans uneversion de Kubernetes
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Mise à jour de la version Kubernetes Amazon EKS
Nouvelle API UpdateClusterVersion -supporte les mises à jour in place de la version Kubernetes
ListUpdates et DescribeUpdate APIs pour fournir une visibilité sur l'étatd'une mise à jour
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Mise à jour des nœuds workers
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Authentification avec AWS Identity and Access Management (IAM)
Kubectl
3) Autorise l'identité AWS avec la RBAC
K8s API
1) Passe l'identité AWS
2) Vérifie l'identité AWS
4) Action K8s autorisée/refusée
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Authentification et autorisation des clusters
• L'utilisateur ou le rôle IAM qui crée le cluster Amazon EKS obtient les privilèges d'administrateur
• Ce {"super"} utilisateur/rôle peut alors ajouter des utilisateurssupplémentaires ou des rôles IAM et configurer les permissions RBAC
• Pour en ajouter, configurez aws-auth Configmap
kubectl edit -n kube-system configmap/aws-auth
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Configuration de la PKI
Kubelet
Generates public/private keys
Kubelet installs server cert
Kubelet issues CSR
Certificate rotation
Amazon EKS API serverEKS worker
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Les filtrages réseaux AWS / Amazon EKS & K8
• Les groupes de sécurité VPC
• VPC NACL
• network policy implémentéeau niveau du Pod• Segmentation réseau• Tenant isolation• Assigné aux pods à l'aide de pod
selectors et labels
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S T A G E S E P A R A T I O N
“ T E N A N T ” S E P A R A T I O N
F I N E - G R A I N E D F I R E W A L L S C O M P L I A N C E
E.g., typically use namespaces for different teams within a company—but without network policy, they are
not network isolated
Reduce attack surface within microservice-based applications
Isolate dev, test, and prod E.g., PCI, HIPAA
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon EKS est prêt pour les workloads sensibles et régulés
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Cross-account ENI
VPC EKSVPC du Client
Noeuds Worker
ENI EKS
Appels de l'APIKubernetes
Exec, Logs, Proxy
Internet
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Contrôle d’accès aux Endpoints API sur Amazon EKS
VPC EKSVPC du Client
Noeuds Worker
ENI EKS
Appels de l'APIKubernetes
Exec, Logs, Proxy
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Le plugin CNI Amazon VPC
ENI
IPs Secondaires :10.0.0.110.0.0.2
10.0.0.1
10.0.0.2
ENI
10.0.0.20
10.0.0.22
IPs Secondaires :10.0.0.2010.0.0.22
ec2.associateaddress()
Réseau VPC – 10.0.0.0/24
Instance 1 Instance 2
VPC
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Le plugin CNI Amazon VPC – Comprendre l’allocation IP
Plage CIDR Primaire Adresses RFC 1918 10/8, 172.16/12, 192.168/16
Utilisée dans Amazon EKS pour :• Les Pods • Les ENIs cross-account pour la communication masters workers (exec, logs,
proxy etc.)• Reseau interne pour les services Kubernetes (10.100/16 ou 172.20/16 – choisi en
fonction de votre plage VPC)
Configuration:• Creation cluster EKS cluster fournir la liste des sous-réseaux (au moins 2
AZs!) tagging
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon VPC CNI plugin – Comprendre l’allocation IP
Plage CIDR secondaire (nouveau!) Adresses hors RFC 1918
(100.64.0.0/10 et 198.19.0.0/16)
Utilisée dans Amazon EKS pour :
• Les Pods seulement
Comment?
• Amazon EKS custom network config enable create
ENIConfig CRD annotate nodes
CNI 1.2.1+
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Load balancing
Les trois types d’ELB AWS sont pris en charge
NLB et CLB supportés par le service le service Kubernetes type=LoadBalancer
Support de répartiteur de charge interne et externe
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ALB Ingress controller
Version 1.0 prête pour la production
Avec le support de l'équipe Amazon EKS
Développement Open Source :https://github.com/kubernetes-sigs/aws-alb-ingress-controller
Les clients l'utilisent en production aujourd'hui !
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ALB Ingress controller
AWS resources
Kubernetes clusterNode Node
Kubernetes API server ALB Ingress
controller
Node
HTTP listenerHTTPS listener
Rule: /cheesesRule: /charcuterie
TargetGroup: Green (IP Mode)
TargetGroup: Blue (Instance
Mode)
NodePort NodePort
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Products — Telescope
Telescope connects to existing payment infrastructure, offers analytics and generates custom performance recommendations
$12B worth of transaction data analyzed in the past 12 months
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Products — Smart Router
ProcessOut’s API routes transactions to the best payment provider in real-time
New providers can be integrated with one click, with built-in reconciliation
250 RPS at peak times$1.3 average value per request
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reliability
99.95% uptimeSLA
99.99% uptimeSLO
< 14 minutes of unavailability per quarter
< 67 minutes of unavailability per quarter
We internally measure uptime from success rates for transaction processing API calls.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Infrastructure overview
One Kubernetes cluster managed by ProcessOut (2.5 years in the making), and one Tectonic cluster in an active/active topology
Whole infrastructure running in 3 us-east-1 availability zones
Mid 2018
ProcessOut — us-east-1
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Growing pains• Impedance mismatch between homemade cluster and Tectonic
• Different user authentication methods• Very different failure scenarios
• Kubernetes upgrades and maintenance are a full-time job• Some customers prefer to keep their data in Europe
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ProcessOut EU roadmap
To improve• Maintainability of Kubernetes• Infrastructure reproducibility• User authentication and
credential distribution
To keep• Reliability• Network architecture• Monitoring (Datadog)• api.processout.com endpoint
for all traffic, no matter the destination region
Mid 2018
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Infrastructure overviewLate 2018
ProcessOut EU — eu-west-1ProcessOut US — us-east-1
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Infrastructure overviewLate 2018
ProcessOut EU — eu-west-1ProcessOut US — us-east-1
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Operating EKS• All production infrastructure is managed by Terraform• We customize workers by baking AMIs from base EKS images
• SSH setup using our SSH certificate infrastructure• Falco kernel module install for IDS• Container runtime watchdog
• Datadog does most of our monitoring• System/app metrics• Tracing• Logging
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Upgrading Kubernetes
EKS supports one-click upgrades but we keep clusters immutable as much as possible
We do production upgrades by creating a canary cluster, and then completing the rolling update at a later time
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reaping the benefitsCreating new regions is easy. Kubernetes clusters are immutable and can be replaced regularly.
Authentication is managed through IAM. Local kubeconfigfiles are generated by Terraform with transparent support for aws-vault.
October–March availability
> 99.994% uptimeProcessOut US
> 99.998% uptimeProcessOut EU
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Looking forward• One staging namespace per developer• Spot instances for noncritical workloads• Burst some workloads into Fargate• kube-proxy bypass for traffic ingress• Istio service mesh
Merci!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.