devconf 2017 - realistic container platform simulations

Realistic Container Platform Simulations

DevConf 2017

aos-scalability team + adjunct professors

DevConf 2017: Realistic Container Platform Simulations

Agenda

● Introduction● Workload Classification● Test Utilities● Demo● Gold Mining

2


OpenShift Performance and Scalability TeamWho are we?

Performance EngineeringRequestor

Upstream Development

QE Handoff

Test Design/Exec @ Scale

Engineering

Product Management

Marketing

Field/Customer

http://j.mp/ose-perf-scale

3




It’s all about the workloads.


Some don’t care where they run

● Batch workloads● Web servers

It’s all about the workloads...

Some care greatly

● Security, Isolation● Uptime● Performance● Proximity/Locality to data

DevConf 2017: Realistic Container Platform Simulations6

Listen to your apps


What is a workload? Subsystems


What is a workload? Business Requirements


Attribute Build Farm

CPU Intensive High

Memory Intensive High

Disk I/O Latency Low

Disk I/O Throughput High

Network Latency Low

Network Throughput High

Security Low

Uptime (Live Migration)

N/A

Deployment Speed High

Alternative OS N/A

Workload → Infrastructure Mapping:Build Farm

Icon Meaning

Mature and/orNo Perf Concerns

Immature and/orLimited Perf Concerns

Mixed Concerns

Not Applicable


Atomic Host - a container-optimized, minimal footprint OS powered by Red Hat Enterprise Linux

Telemetry - logging and metrics for pods/containers, services and underlying infrastructure to make informed decisions

Runtime and Packaging Format - standardized container packaging format and runtime, powered by Docker (and OCI)

Automation and host configuration management via Cockpit to dynamically provision and configure container host clusters

Orchestration - for complex multi-container services, powered by Kubernetes

Networking - scalable, multi-host container networking, powered by Open vSwitch, that runs anywhere Red Hat Enterprise Linux runs

Cluster Services - Scheduling for services across a container host cluster, powered by Kubernetes

Storage, with persistent storage plugins to enable running of stateful services in containers

Atomic Registry - integrated storage and management for sharing container images

Security to prevent tenants from compromising other occupants

Container Infrastructure Components


10,000 foot overview


Realistic Container Platform Simulations:Test Harnesses


System Verification Test Suite (SVT)

● Red Hat OpenShift Performance and Scalability team’s test suites● https://github.com/openshift/svt

● cluster-loader● Networking/synthetic● Workload Generator● Reliability/Longevity

● https://github.com/kubernetes/perf-tests

https://github.com/openshift/svt

https://github.com/openshift/svt

https://github.com/kubernetes/perf-tests



Image Provisioner

Runs on any RHEL instance (currently supported EC2 and qcow2/kvm)

● filesystem juggling to allow for thinpool in base RHEL cloud image● RHEL OS setup (install latest packages, ssh keys)● aos-ansible (pull down supporting container images)● openshift-rpm-install (install openshift RPMs, but do not configure)● clone necessary git repos● collectd-install (install and configure collectd)● docker-config (setup storage)● repo-install (setup custom yum repos)● pbench-install (install and configure pbench)

Ansible Automation to build a pre-configured RHEL image

https://github.com/distributed-system-analysis/pbench


What is cluster-loader?

I’d like an environment with thousands of deployment configs (which include services and replication controllers), thousands more routes, pods (each with a persistent storage volume automatically attached), secrets, image streams, buildconfigs, etc.


Start

Parse args & config

New ConfigObj

End

False

Create Namespace

True

XExists

?Items < N

False

Create X

Iterate ItemCount

True True

False

X can be:QuotaTemplateServiceUserPodRC


Cluster Loader Demo


Contributing

- 100% golang

- Github: https://github.com/kubernetes/perf-tests

- How to reach us:

- sig-scale mailing list:

https://groups.google.com/forum/#!forum/kubernetes-sig-scale

- sig-scale slack channel: https://kubernetes.slack.com/messages/sig-scale/

- Freenode: #openshift-dev

18




https://kubernetes.slack.com/messages/sig-scale/

Core/Most Urgent Issue Summary


Gold Mining: Successes


iptables

● iptables CPU consumption BZ1387148

21

https://bugzilla.redhat.com/show_bug.cgi?id=1387149


HAProxy

● Raise default HAProxy maxconn to 20000: BZ1406327 ● HAProxy is forcefully restarted due to not responding to

/healthz probe when under high load: BZ1405440● ARP tuning documentation

● nbproc● tcp_fastopen● busy_poll

22



https://github.com/openshift/openshift-docs/issues/3030


Logging

● Daemonsets (used by logging) needs rate limiting: BZ1404860

● Logging DoS's apiserver BZ1384626

23





Metrics

● Metrics stops capturing data @ 12k pods● BZ1388815● OpenShift Docs Update PR

24



https://github.com/openshift/openshift-docs/pull/3499




OCP-on-OSP: etcd

● Team reminded of etcd tight perf req’s● OCP has poor day-2 operations capabilities for etcd

○ Difficult to replace failed nodes (SSL certs, data migration)

25


Lessons Learned/Next Steps

● Ensure fast disk for etcd (SSD is a must).● Automate real hostnames and DNS (required for metrics and route tests)● If using boot-from-volume, ensure you use snapshots when nova booting● Ceph pools should have correct # of pg/pgp. Use pgcalc.● Use dedicated builder nodes in OpenShift.● Incorporate CFME into the environment, deployed as a pod.

26


It’s all about the workloads.


Thanks!

devconf 2017 - realistic container platform simulations

Technology