CloudStack Scalability Testing, Development, Results, and Futures Anthony Xu Apache CloudStack contributor

Download CloudStack Scalability Testing, Development, Results, and Futures Anthony Xu Apache CloudStack contributor

Post on 14-Dec-2015

215 views

Category:

Documents

3 download

Embed Size (px)

TRANSCRIPT

<ul><li>Slide 1</li></ul> <p>CloudStack Scalability Testing, Development, Results, and Futures Anthony Xu Apache CloudStack contributor Slide 2 Secure, multi-tenant cloud orchestration platform Turnkey platform for delivering IaaS clouds Hypervisor agnostic Highly scalable, secure and open Complete Self-service portal Open source, open standards Deploys on premise Apache CloudStack: a project in incubation Slide 3 Router L3 Core Switch Top of Rack Switch Availability Zone 1 Servers Management Server Cluster Object Storage Pod 1 Pod 2 Pod 3 Pod N Primary MySQL Load Balancer Admin Internet Backup MySQL Manage hosts, create VMs, virtual disks, virtual networks, meter usage, . Slide 4 Thinking about cloud orchestration at scale Host management Capacity management What host to use to deploy a new VM Failure handling Security group propagation Set a goal Slide 5 We cant afford this as our QA lab Slide 6 User API Admin API Load Balancer Mgmt. Server MySQL Zone Simulator MySQL Simulator enables scale testing Mgmt. Server Slide 7 User API Admin API Load Balancer Mgmt. Server MySQL Zone Simulator MySQL Environment Mgmt. Server 2 cores, 4 with Hyper Threading. 2.2 GHz Xeon. 16 GB RAM. 12 GB JVM Heap. Single spinning disk, later single SSD. 32 GB RAM. MySQL 5.5. Slide 8 Slide 9 Allocator performance is awful with 1000 hosts Two minutes to decide which host to use for a new VM! Computing capacity for every pod repeatedly Fixed that, but still 12 seconds to decide Use host tags, down to 2 seconds Major changes required to improve further In 2.2.0, store capacity info in DB, skip pod altogether Harness the power of SQL select and all is well Slide 10 Polling doesnt scale TRUE?FALSE? Sometimes, it is good enough Slide 11 Host management Check host state via TCP connection Check every minute 30,000 checks per minute, 500 per second But they take 10 seconds, so 5000 in parallel Not using async I/O so 5000 threads required Single JVM can support 5000+ threads so this is concerning but may not be the limiting factor Slide 12 Host management What is the maximum feasible JVM heap size? Some people use heaps with hundreds of GB Commercial tools can help, but cost We decided to stay below 20 GB (GC concerns) How much CPU is required for background processing? Slide 13 CPU utilization while deploying 30,000 VMs on 30,000 hosts CPU Utilization. 400% is maximum Time 20,000 5000 Idle Slide 14 Deploy time from 25,000 to 30,000 VMs Seconds to deploy VM number: 25,000 plus X Slide 15 Problem: agent load balancing Management servers start/stop/fail/crash How do newly started Management Servers get agents / work? When a Management Server exits, how do others pick up its load? When new hosts are added how is the load distributed? Mgmt Server 1 Mgmt Server 2 Agent 3Agent 4Agent 5Agent 6Agent 1Agent 2 Slide 16 Common use case timings at scale 30,000 hosts and 4 Management Servers 4 Management Servers running, 1 fails: 10 minutes to redistribute 7500 agents 3 Management Servers running, add a fourth: 40 minutes to redistribute load evenly 0 Management Servers running, start all 4 simultaneously: 16 minutes to connect to all 30,000 hosts IMPORTANT Slide 17 DB Security Group Web Security Group Understanding security groups Web VM DB VM Web VM DB VM Web VM Ingress Rule: Allow VMs in Web Security Group access to VMs in DB Security Group on Port 3306 Slide 18 L3 isolation with distributed firewalls Tenant 1 VM 1 10.1.0.2 Tenant 2 VM 1 10.1.0.3 Tenant 1 VM 2 10.1.0.4 Public Internet 10.1.0.1 Public IP address 65.37.141.11 65.37.141.24 65.37.141.36 65.37.141.80 Load Balancer L3 Core Pod 1 L2 Switch Pod 3 L2 Switch 10.1.16.1 10.1.8.1 Pod 2 L2 Switch Slide 19 L3 isolation with distributed firewalls Tenant 1 VM 1 10.1.0.2 Tenant 2 VM 1 10.1.0.3 Tenant 1 VM 2 10.1.0.4 Tenant 1 VM 3 10.1.16.47 Tenant 1 VM 4 10.1.16.85 Public Internet 10.1.0.1 Public IP address 65.37.141.11 65.37.141.24 65.37.141.36 65.37.141.80 Load Balancer L3 Core Pod 1 L2 Switch Pod 3 L2 Switch 10.1.16.1 10.1.8.1 Pod 2 L2 Switch Slide 20 L3 isolation with distributed firewalls Tenant 1 VM 1 10.1.0.2 Tenant 2 VM 1 10.1.0.3 Tenant 1 VM 2 10.1.0.4 Tenant 2 VM 2 10.1.16.12 Tenant 2 VM 3 10.1.16.21 Tenant 1 VM 3 10.1.16.47 Tenant 1 VM 4 10.1.16.85 Public Internet 10.1.0.1 Public IP address 65.37.141.11 65.37.141.24 65.37.141.36 65.37.141.80 Load Balancer L3 Core Pod 1 L2 Switch Pod 3 L2 Switch 10.1.16.1 10.1.8.1 Pod 2 L2 Switch Slide 21 1 Firewall per Virtual Machine Slide 22 VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM VMVM One million firewalls? Slide 23 Well-known software scaling techniques Message queues Consistency tradeoffs Idempotent configuration &amp; retries CloudStack uses Special purpose queues Optimized for large security groups Eventual consistency for rule updates Orchestrating hundreds of thousands of firewalls Slide 24 Problem: firewall rules explosion in dom0 -A FORWARD -m tcp p tcp dport 3060 src 10.1.16.31 j ACCEPT -A FORWARD -m tcp p tcp dport 3060 src 10.1.45.112 j ACCEPT -A FORWARD -m tcp p tcp dport 3060 src 10.1.189.5 j ACCEPT -A FORWARD -m tcp p tcp dport 3060 src 10.21.9.77 j ACCEPT Performance suffers for large security groups Allow Security Group {Web} on TCP port 3060 Slide 25 ipset N web_sg iptreemap ipset A web_sg 10.1.16.31 ipset A web_sg 10.1.16.112 ipset A web_sg 10.1.189.5 ipset A web_sg 10.21.9.77 -A FORWARD p tcp m tcp dport 3060 m set match-set web_sg src -j ACCEPT Fix with ipsets: Problem: firewall rules explosion in dom0 See also http://daemonkeeper.net/781/mass-blocking-ip-addresses-with-ipset/ Slide 26 Security group propagation time Seconds to fully synced Number of VMs in security group Slide 27 Problem: database connection management Scale testing resulted in several too many open connections errors from MySQL Common problem: holding open connections while doing long-running operations Took some code clean up and refactoring No longer an issue 10,000 connections are OK CloudStack is far below that Slide 28 DB connections per MS while deploying 30,000 VMs Number of DB connections Time 20,000 5,000 Slide 29 Other considerations (beyond control plane) Network design and devices Object store scalability Per-host and cluster scalability Storage Understand your workload Slide 30 Future work Improve simulator accuracy Publish results of advanced network (VLAN) testing Verify assumption of VM density not impacting scale Slide 31 More information and joining the project Project web site: http://incubator.apache.org/projects/cloudstack.html Mailing lists: cloudstack-dev-subscribe@incubator.apache.org cloudstack-users-subscribe@incubator.apache.org Scalability study: http://wiki.cloudstack.org/pages/viewpage.action?pageId=14320020 Slide 32 Q&amp;A </p>