#apachecon’17: cloudstack collaboration conference taking ... · user_ip_adressentries updated to...

© 2017 Autodesk | Enterprise Information Services

Sergey LevitskiyManager, Infrastructure as a Service

#ApacheCon’17: Cloudstack Collaboration Conference

Taking Advantage of your CloudStack Database


Cloudstack as Cloud Management Platform

§ Software-defined:§ compute - SDC§ storage - SDS§ networking - SDN

§ Stateless management servers§ No cloud related metadata

§ Near stateless infrastructure§ Exceptions

§ Bridge names (Xen)§ vSwitches, VM version(VMware)§ SDN control plane (NSX)


Software Defined = Database Driven

§ 254 tables§ 3078 columns§ 966 indexes§ 25 views§ Highly relational


Example - Volume_view – 14 joins, 85 columns

CREATE ALGORITHM = UNDEFINED DEFINER = `cloud`@`%` SQL SECURITY DEFINERVIEW `volume_view` AS SELECT `volumes`.ìd` AS ìd`, `volumes`.ùuid` AS ùuid`, `volumes`.`name` AS `name`, `volumes`.`device_id` AS `device_id`, `volumes`.`volume_type` AS `volume_type`, `volumes`.`provisioning_type` AS `provisioning_type`, `volumes`.`size` AS `size`, `volumes`.`min_iops` AS `min_iops`, `volumes`.`max_iops` AS `max_iops`, `volumes`.`created` AS `created`, `volumes`.`state` AS `state`, `volumes`.àttached` AS àttached`, `volumes`.`removed` AS `removed`, `volumes`.`pod_id` AS `pod_id`, `volumes`.`display_volume` AS `display_volume`, `volumes`.`format` AS `format`, `volumes`.`path` AS `path`, `volumes`.`chain_info` AS `chain_info`, àccount`.ìd` AS àccount_id`, àccount`.ùuid` AS àccount_uuid`, àccount`.àccount_name` AS àccount_name`, àccount`.`type` AS àccount_type`, `domain`.ìd` AS `domain_id`, `domain`.ùuid` AS `domain_uuid`, `domain`.`name` AS `domain_name`, `domain`.`path` AS `domain_path`, `projects`.ìd` AS `project_id`, `projects`.ùuid` AS `project_uuid`, `projects`.`name` AS `project_name`, `data_center`.ìd` AS `data_center_id`, `data_center`.ùuid` AS `data_center_uuid`, `data_center`.`name` AS `data_center_name`, `data_center`.`networktype` AS `data_center_type`, `vm_instance`.ìd` AS `vm_id`, `vm_instance`.ùuid` AS `vm_uuid`, `vm_instance`.`name` AS `vm_name`, `vm_instance`.`state` AS `vm_state`, `vm_instance`.`vm_type` AS `vm_type`, ùser_vm`.`display_name` AS `vm_display_name`, `volume_store_ref`.`size` AS `volume_store_size`, `volume_store_ref`.`download_pct` AS `download_pct`, `volume_store_ref`.`download_state` AS `download_state`, `volume_store_ref`.èrror_str` AS èrror_str`, `volume_store_ref`.`created` AS `created_on_store`, `disk_offering`.ìd` AS `disk_offering_id`, `disk_offering`.ùuid` AS `disk_offering_uuid`, `disk_offering`.`name` AS `disk_offering_name`, `disk_offering`.`display_text` AS `disk_offering_display_text`, `disk_offering`.ùse_local_storage` AS ùse_local_storage`, `disk_offering`.`system_use` AS `system_use`, `disk_offering`.`bytes_read_rate` AS `bytes_read_rate`, `disk_offering`.`bytes_write_rate` AS `bytes_write_rate`, `disk_offering`.ìops_read_rate` AS ìops_read_rate`, `disk_offering`.ìops_write_rate` AS ìops_write_rate`, `disk_offering`.`cache_mode` AS `cache_mode`, `storage_pool`.ìd` AS `pool_id`, `storage_pool`.ùuid` AS `pool_uuid`, `storage_pool`.`name` AS `pool_name`, `cluster`.`hypervisor_type` AS `hypervisor_type`, `vm_template`.ìd` AS `template_id`, `vm_template`.ùuid` AS `template_uuid`, `vm_template`.èxtractable` AS èxtractable`, `vm_template`.`type` AS `template_type`, `vm_template`.`name` AS `template_name`, `vm_template`.`display_text` AS `template_display_text`, ìso`.ìd` AS ìso_id`, ìso`.ùuid` AS ìso_uuid`, ìso`.`name` AS ìso_name`, ìso`.`display_text` AS ìso_display_text`, `resource_tags`.ìd` AS `tag_id`, `resource_tags`.ùuid` AS `tag_uuid`, `resource_tags`.`key` AS `tag_key`, `resource_tags`.`value` AS `tag_value`, `resource_tags`.`domain_id` AS `tag_domain_id`, `domain`.ùuid` AS `tag_domain_uuid`, `domain`.`name` AS `tag_domain_name`, `resource_tags`.àccount_id` AS `tag_account_id`, àccount`.àccount_name` AS `tag_account_name`, `resource_tags`.`resource_id` AS `tag_resource_id`, `resource_tags`.`resource_uuid` AS `tag_resource_uuid`, `resource_tags`.`resource_type` AS `tag_resource_type`, `resource_tags`.`customer` AS `tag_customer`, àsync_job`.ìd` AS `job_id`, àsync_job`.ùuid` AS `job_uuid`, àsync_job`.`job_status` AS `job_status`, àsync_job`.àccount_id` AS `job_account_id` FROM ((((((((((((((`volumes` JOIN àccount` ON ((`volumes`.àccount_id` = àccount`.ìd`))) JOIN `domain` ON ((`volumes`.`domain_id` = `domain`.ìd`))) LEFT JOIN `projects` ON ((`projects`.`project_account_id` = àccount`.ìd`))) LEFT JOIN `data_center` ON ((`volumes`.`data_center_id` = `data_center`.ìd`))) LEFT JOIN `vm_instance` ON ((`volumes`.ìnstance_id` = `vm_instance`.ìd`))) LEFT JOIN ùser_vm` ON ((ùser_vm`.ìd` = `vm_instance`.ìd`))) LEFT JOIN `volume_store_ref` ON ((`volumes`.ìd` = `volume_store_ref`.`volume_id`))) LEFT JOIN `disk_offering` ON ((`volumes`.`disk_offering_id` = `disk_offering`.ìd`))) LEFT JOIN `storage_pool` ON ((`volumes`.`pool_id` = `storage_pool`.ìd`))) LEFT JOIN `cluster` ON ((`storage_pool`.`cluster_id` = `cluster`.ìd`))) LEFT JOIN `vm_template` ON ((`volumes`.`template_id` = `vm_template`.ìd`))) LEFT JOIN `vm_template` ìso` ON ((ìso`.ìd` = `volumes`.ìso_id`))) LEFT JOIN `resource_tags` ON (((`resource_tags`.`resource_id` = `volumes`.ìd`) AND (`resource_tags`.`resource_type` = 'Volume')))) LEFT JOIN àsync_job` ON (((àsync_job`.ìnstance_id` = `volumes`.ìd`) AND (àsync_job`.ìnstance_type` = 'Volume') AND (àsync_job`.`job_status` = 0))));


Why can’t ACS API/UI suffice?

§ Too slow§ Logic not implemented§ API doesn’t exist§ Bugs§ Reporting needs


How to start?§ Study DB structure§ Review ACS source code§ Trace all relationships§ Lab->Dev->Stage->Prod§ Watch management server logs§ DB backup


Problem #1: Cluster base primary storage§ Year 1 - 12 storage arrays

§ 4 regions§ 1 compute clusters per regions§ 3 storage classes

§ Year 2 – 24 storage arrays§ 4 regions§ 2 compute clusters per regions§ 3 storage classes

§ Year 3 – ?? storage arrays§ 4 regions§ 6 compute clusters per regions§ 3 storage classes


Solution #1 Convert to zone-wide primary§ Year 1 - 12 storage arrays

§ 4 regions§ 1 compute clusters per region§ 3 storage classes

§ Year 2 – 24 12 storage arrays§ 4 regions§ 2 compute clusters per region§ 3 storage classes

§ Year 3 – 12 storage arrays§ 4 regions§ 6 compute clusters per region§ 3 storage classes


Options§ New arrays. Volume migrations

§ Too long§ Too expensive ( 4 new arrays)

§ Use ACS API – no option to change PS scope

§ Change scope in the DB


Storage pool relationships


Implementation

UPDATE storage_poolSET pod_id = NULL, cluster_id = NULL, scope = 'ZONE', hypervisor = 'VMware’WHERE scope = 'CLUSTER' AND removed IS NULL-- AND uuid = ‘XXXXXX-XXXX-XXXX-XXXX-XXXXXX”


Problem #2: Understand OS type distribution

§ License compliance§ Affinity/anti-affinity decisions

+---------+---------+---------+-------+

| OS type | Running | Stopped | Total |

+---------+---------+---------+-------+

| CentOS | 847 | 185 | 1032 |

| RedHat | 837 | 144 | 981 |

| SUSE | 2 | 0 | 2 |

| Windows | 8635 | 1940 | 10575 |

| Other | 563 | 155 | 718 |

| Ubuntu | 1139 | 365 | 1504 |

+---------+---------+---------+-------+


Implementation http://tinyurl.com/m8ow2u9

SELECT goc.name, SUM(case when i.state = 'Running' then 1 else 0 end) as 'Running', SUM(case when i.state != 'Running' then 1 else 0 end) as 'Stopped', count(*) as 'Total'

FROM cloud.vm_instance i, guest_os go, guest_os_category gocWHERE i.guest_os_id=go.id and i.removed is null

and go.category_id=goc.id and i.type='User’GROUP BY goc.id;


Problem #3: Need for a special Mac address

§ Clustering§ Licensing applications

§ No ACS API to manipulate Mac


Implementation http://tinyurl.com/m8ow2u9 as a Stored procedure

SET @MAC_address= ’YY:YY:YY:YY:YY:YY’;SET @VM_UUID= ‘XXXXXX-XXXX-XXXX-XXXX-XXXXXX”;

UPDATE nics n, vm_instance iSET i.private_mac_address= @MAC_address,

n.mac_address= @MAC_addressWHERE i.id=n.instance_id and i.removed is null and n.removed

is null and i.uuid = @VM_UUID;

Additional logic to handle VMs with multiple NICs can be easily added.


Problem #4: Too many VLANs

§ Shared networking§ 300 -> 8000 VM§ Class C subnets, sequential IP space§ 2 VLANs -> 32 VLANs

Switching vendor limitation 42 VLANs


Solution #4 Super netting /24 -> /22§ Requirements

§ Minimize VM downtime§ Retain IP§ No provisioning downtime

§ Challenges§ Can’t add overlapping ranges in ACS§ Duplicate VLANs not allowed in Shared mode


Networking tables relationship


Implementation http://tinyurl.com/kqzp4p9

1. New /22 networks, new VLAN id, pre-provisioned on the network gear2. Added in advance to ACS with first octet altered e.g. 11.0.0.0/22

instead of 10.0.0.0/22 and assigned to a hidden account3. Rapid conversion of a batch of four/24 networks - >/22

1. CIDR and gateway corrected in VLAN and network tables – 1 sec2. User_ip_adress entries updated to match - 3 sec3. Update all active NICS with correct CIDR and GW - 10 sec4. Transfer IP reservation in user_ip_address – 15 sec5. Free up original VLANs for deletion - 10 sec6. New network -> visible, old networks -> hidden - 1 sec7. Stop-start 1000 VMs - 15 min

Autodesk and the Autodesk logo are registered trademarks or trademarks of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document.© 2017 Autodesk. All rights reserved.

#apachecon’17: cloudstack collaboration conference taking ... · user_ip_adressentries updated to...

Documents