cloudstack usage service

The Cloud Specialists

CloudStackUsageService

DagSonsteboCloudArchitect

[email protected]:@dagsonstebo

C l i c k t o e d i t

The Cloud Specialists ShapeBlue.com @ShapeBlue

A b o u t m e

• Cloud Architect @ ShapeBlue• Background:

• Cloudandvirtualizationarchitectwith20yearsexperiencefromtheserviceprovider,financialandmanufacturingindustries.

• Specializein:• Cloudinfrastructurearchitectureandengineering.

• Virtualization- VMwarevSphere,CitrixXenServer,KVM.

• Automationandconfigurationmanagement,loveAnsible.

• InvolvedwithCloudStacksinceversion2.1.



“ShapeBlue are expert builders of public & private clouds. They are the leading global CloudStack

services company.”

A b o u t S h a p e B l u e



S h a p e B l u e c u s t o m e r s



CloudStack usage service



O v e r v i e w

CloudStack usage is a complimentary service which:• tracks end user consumption of CloudStack resources• summarises all data in a separate database for reporting or

billing

The usage database can be:• queried directly,• queried through the CloudStack API, • or it can be integrated into external billing or reporting systems.



H o w d o I i n s t a l l i t ?

CentOS:# yum install cloudstack-usage# chkconfig cloudstack-usage on# service cloudstack-usage on

Ubuntu:# apt-get install cloudstack-usage# update-rc.d cloudstack-usage defaults# service cloudstack-usage on

Configuration files:# ls -l /etc/cloudstack/usage/total 4lrwxrwxrwx. 1 root root 40 Sep 8 08:18 db.properties -> /etc/cloudstack/management/db.propertieslrwxrwxrwx. 1 root root 30 Sep 8 08:18 key -> /etc/cloudstack/management/key-rw-r--r--. 1 root root 2968 Jul 12 10:36 log4j-cloud.xml



G l o b a l s e t t i n g s

Setting Comment

enable.usage.server Switchesusageserviceon/off

usage.aggregation.timezone Timezone usedforusageaggregation(default “GMT”)

usage.execution.timezone Timezone forusagejobexecution

usage.sanity.check.interval Interval(indays)tochecksanityofusagedata

usage.snapshot.virtualsize.select Setthevaluetotrueifsnapshotusageneedtoconsidervirtualsize,elsephysicalsizeisconsidered(default “false”)

usage.stats.job.aggregation.range Therangeoftimeforaggregatingtheuserstatisticsspecifiedinminutes(e.g.1440fordaily,60forhourly,defaultis60minutes)

usage.stats.job.exec.time Thetimeatwhichtheusagestatisticsaggregationjobwillrun(default00:15)



U s a g e t y p e s

• All lifecycle events for resources in CloudStack are tracked in the cloud.usage_event table.

• This table lists all timestamps for:• Create/destroy• Start/stop• Request/release

• These events are processed by the usage service and categorized as usage types.

Create

Start

Request

Release

Stop

Destroy



U s a g e t y p e s # cloudmonkey list usagetypescount = 19usagetype:+-------------+--------------------------------+| usagetypeid | description |+-------------+--------------------------------+| 1 | Running Vm Usage || 2 | Allocated Vm Usage || 3 | IP Address Usage || 4 | Network Usage (Bytes Sent) || 5 | Network Usage (Bytes Received) || 6 | Volume Usage || 7 | Template Usage || 8 | ISO Usage || 9 | Snapshot Usage || 10 | Security Group Usage || 11 | Load Balancer Usage || 12 | Port Forwarding Usage || 13 | Network Offering Usage || 14 | VPN users usage || 21 | VM Disk usage(I/O Read) || 22 | VM Disk usage(I/O Write) || 23 | VM Disk usage(Bytes Read) || 24 | VM Disk usage(Bytes Write) || 25 | VM Snapshot storage usage |+-------------+--------------------------------+



P r o c e s s f l o w – A P I c a l l a n d u s a g e j o b s

API• Callreceived

API• Callprocessed

DB

• Lifecycleactionwrittentocloud.usage_event

Start• Executeusagejobatusage.stats.job.exec.time

Copy

• Copyallnewevents:• FROM:cloud.usage_event• TO:cloud_usage.usage_event

• Copyaccounttable

Process

• Summarise eachusagetypeperaccount:• Populatehelpder tables,e.g.usage_storage,usage_vm_instance,etc

• Aggregateusagetypesforallaccounts• incloud_usage.cloud_usage• basedonusage.status.job.aggregation.range



c l o u d . u s a g e _ e v e n t



H e l p e r t a b l e s

• Helper tables summarise each resource with:• startandenddatestamp• usagetype(ifrequired)• Notethough– everytablehasadifferentschemadependingon

theresourceusagetype

• Example (all in the cloud_usage) DB:• usage_vm_instance• usage_storage• usage_vm_disk• Etc.



H e l p e r t a b l e s - e x a m p l e s



U s a g e a g g r e g a t i o n

• Once helper tables are processed the cloud_usage_cloud_usage table is populated.

• The usage service analyses the helper table entries and divides the time usage for each by the time period defined in “usage.stats.job.aggregation.range”.

• E.g.:• howmanyhourshasaVMbeenrunningineach1440

minute(24hour)period• howmanyhourshaseachGBofstoragebeenusedin

1440minuteperiod



C l o u d _ u s a g e . c l o u d _ u s a g e

Usagetype

Rawusage

ResourceID

Vol/snapsize

Start/enddate



Q u e r y t h r o u g h A P I w i t h C l o u d M o n k e y



R e p o r t i n g e x a m p l e – r u n n i n g V M s



R e p o r t i n g e x a m p l e – n e t w o r k u t i l i z a t i o n



R e p o r t i n g e x a m p l e – v o l u m e s



R e p o r t i n g e x a m p l e – I P a d d r e s s e s , p o r t f o r w a r d i n g a n d V P N



Tr o u b l e s h o o t i n g

• Check usage job status in the database:

• Usage jobs:• OnservicestartuptheusageservicewillchecktheDBwhetherlastjobwasran– ifnotthisjobis

ranonce.• Noteifexecutiontimetakeslongerthan“usage.stats.job.aggregation.range”– default1day.Ifso

jobsmayoverlapandcauseproblems,henceavoidthissituation.

• Logs:• /var/log/cloudstack/usage/usage.log• Relativelyverbose– willshowprocessingofusagetypes,accounts,etc.



Tr o u b l e s h o o t i n g a n d h o u s e k e e p i n g

• DB inconsistencies:• Backupdatabasesbeforeediting……..J• Ifusagejobfailsitwillreportinlogswhattheproblemis.• WatchoutforinconsistencieslikeSTOPentrywithoutSTART,DESTROYwithoutCREATEordouble

entries– e.g.2xSTARTentriesforaVM.• Ifrequiredyoucanadd/deleteentriesinthecloud.usage_event table.

• Housekeeping of cloud_usage table:• UsetheremoveRawUsageRecords APIcalltodeleteentriesolderthan5days.

#cloudmonkeyremoveRawUsageRecords interval=5success=true

• Regenerating usage data:• UsethegenerateUsageRecords APIcalltorerunusageincaseoffailure:

#cloudmonkeygenerateUsageRecords startdate=2017-09-01enddate=2017-09-30success=true



A b r i e f w o r d o n t h e q u o t a s e r v i c e

• In the cloud_usage database you will see a number of “quota_*” tables.

• These are not directly used by the usage service, rather the separate Quota service.

• Check the following for more information:• https://cwiki.apache.org/confluence/display/CLOUDSTACK/Quota+Service+-

+FS• http://docs.cloudstack.apache.org/projects/cloudstack-

administration/en/4.8/plugins/quota.html



Any questions?



Some references• This talk will be reflected in the yet-to-be-published blog post

in http://www.shapeblue.com/cloudstack-usage-service-deep-dive -this will have more technical detail.

• CloudStack usage references: • http://docs.cloudstack.apache.org/projects/cloudstack-

installation/en/4.9/optional_installation.html• http://docs.cloudstack.apache.org/projects/cloudstack-

administration/en/4.9/usage.html



M o r e i n f o r m a t i o n

• Slide decks: http://www.slideshare.net/shapeblue

• Blog: http://shapeblue.com/bloghttp://dsonstebo.wordpress.com

• Email: [email protected]

• Twitter: @dagsonstebo

• Web: http://shapeblue.com

The Cloud Specialists

CloudStackUsageServiceDagSonsteboCloudArchitect

[email protected]:@dagsonstebo

cloudstack usage service

Technology