optimizing your cloud applications in rightscale

1

Optimizing Your Cloud Applications in RightScaleRafael H. Saavedra - VP Engineering, RightScale

June 8th, 2011

2

Real Cloud Experience. Shared.

# 2

• Introduction• 3-tier application architecture• Vertical & horizontal scaling• RightScale monitoring and cluster graphs• New Relic RPM• Support for optimizing DB performance• Miscellaneous

Agenda

3


# 3

• Multi-tenancy

• Shared resource pooling

• Geo-distribution and ubiquitous network access

• Service oriented

• Dynamic resource provisioning

• Self-organizing

• Utility based pricing

Cloud computing characteristics

4


# 4

• No upfront investment

• Lowering operating costs

• Highly scalable

• Easy access

• Reduces business risk and maintenance costs

Cloud computing advantages

5


# 5

3-tier application architecture• Load balancers• A farm of application servers• Master-slave

6


# 6

• Instance size (vertical scaling)

• Instance autoscaling (horizontal scaling)• Server arrays

• RightScale support for performance optimization• ServerTemplates are configured to capture performance data• Collectd RightScripts

• Hardware & OS monitoring data• Specialized plugins – MySQL, HAProxy, Apache, NgInx, IIS, etc

• Monitoring graphs: individual, cluster, stacked, heat maps• Alerts & escalations

• New Relic RPM

Cloud performance optimization

7


# 7

• Compute units vs memory vs cost

Scaling up – spectrum of instance sizes

0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.00.5

5.0

50.0

500.0

t1.micro

m1.small

m1.large m1.xlarge

m2.xlarge

m2.2xlarge

m2.4xlarge

c1.medium

c1.xlargecc1/cg1.4xlarge

Compute Units

Mem

ory

(G

B)

High Performance Computing

Test & Dev

Scalable Applications

8


# 8

Server arrays provide horizontal scaling

9


# 9

• The array scales up or down based on performance votes• Tags allow scaling on an arbitrary decision set• Decision threshold controls reaction time• Sleep time allows new resources to have an impact• Scaling can be time dependent• Detailed setup instructions: http://bit.ly/c1oLr2

• Fast response to changes in load conditions using alerts

• Allocation of servers to availability zones based on weights

• Deployment-based so configuration is consistent

• Arrays can be pre-scaled to support anticipated demand

Server arrays provide horizontal scaling

http://bit.ly/c1oLr2

10


# 10

Cluster monitoring• Individual graphs

• Good for a dozen servers• Displays all standard graphs with full detail

• Stacked graphs• Displays the contribution of many servers to a total• Great to see the sum and variability of activity in a cluster• Difficult to make out individual servers• Examples: requests/sec, cpu busy cycles, I/O bytes/sec

• Heat maps• Displays a bar for each server• Great to see uneven distribution across servers• Great to quickly spot performance problems across many servers• Difficult to read absolute values or see the total cluster activity

11


# 11

Cluster monitoring• Current cluster monitoring: one graph per server

12


# 12

Stacked graphs• Each color band shows contribution of one server• Servers are stacked on top of one another

13


# 13

Heat maps• Each horizontal strip shows one server• The color shows how “hot” the server is running

14


# 14

Heat map with 100 servers

15


# 15

Stacked graph of the same 100 servers

16


# 16

Cluster monitoring architecture

• Architecture• Monitoring front-end servers

pull data from storage servers• Up to 100 servers on one graph

(to be increased)

your servers

monitoringstorageservers

monitoringfront-endservers

17


# 17

• Real-Time App Performance Analytics

• Supports Ruby, PHP, Java & .Net

• SQL & NoSQL performance

• Web transaction tracing

• Performance notifications

• Availability monitoring

• Scalability analysis

New Relic RPM

18


# 18

New Relic RPM• Direct access from RightScale dashboard

19


# 19

New Relic RPM• Historical statistics over a period of time

20


# 20

New Relic RPM• Distribution of the most time consuming requests

21


# 21

New Relic RPM• Statistics about response times from different countries

22


# 22

New Relic RPM• Detailed response times by browser

23


# 23

• An expensive query

• The N+1 query problem

• Finding patterns in similar requests

New Relic RPM – 3 Examples

http://bit.ly/lGIDee

http://bit.ly/jTQsq5

http://bit.ly/mHz7kU

24


# 24

Optimizing DB performance• RightScale MySQL ServerTemplates

• Configuration files tailored to instance size• innodb_buffer_pool_size• key_buffer_size• thread_size• sort_buffer_size

• The never ending task of identifying current bottlenecks• Disk seeks• Performance of disk operations• Scale up when working set cannot fit in memory – avoid active swapping• Constant monitoring of performance graphs, logs and query

• Schema considerations

25


# 25

Schema considerations• Lookups need to be indexed

• Sorting requires an index

• Joins need to be done on indices• Become slower as tables grow

• Compounded indices should be used consistently

• Do not abuse indices• Each index requires a disk write

• Compact tables if they become fragmented• Deleted rows do not remove the corresponding index entries

26


# 26

Monitoring DB performance• Standard collectd statistics

• User vs wait time (disk operations)• Performance of disk operations• Scale up when working set cannot fit in memory

• MySQL collectd plugin• Monitor INSERT, SELECT, UPDATE operations• The breakdown of read operations can indicate missing indices

• Monitoring /var/log/mysql-slow.log file• Identify slow queries

• Use MySQL EXPLAIN command to identify query plan

27


# 27

MySQL Collectd Plugin• Uses MySQL SHOW STATUS command to collect statistics• A large set of counters that are divided into 10 categories

• Connections• IO Requests• Select Rates• Read Rates• Key Rates• Commands Rates• Query Cache• Tables• Memory• Misc.

28


# 28

MySQL Collectd Plugin• Uses MySQL SHOW STATUS command to collect statistics

29


# 29

Mysql-slow.log & explain command# Time: 101006 23:30:11

# User@Host: prod[prod] @ domU-12-31-39-0F-D0-C1.compute-1.internal [10.193.211.47]

# Query_time: 7 Lock_time: 0 Rows_sent: 1 Rows_examined: 19785

SELECT * FROM èc2_elastic_ips` WHERE (èc2_elastic_ips`.ec2_instance_id = 6810144) LIMIT 1;

mysql> explain select * FROM èc2_elastic_ips` WHERE (èc2_elastic_ips`.ec2_instance_id = 6810144) LIMIT 1 \G

*************************** 1. row *************************** id: 1 select_type: SIMPLE table: ec2_elastic_ips type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 33332 Extra: Using where1 row in set (0.00 sec)

30


# 30

MySQL performance depends on locality• Wait time should be minimum when working set fits in memory• Performance degrades once wait time is significant

user time dominates

wait time insignificant

31


# 31

MySQL reads graphs• Read-random-next represents a table scan• Read-next represents an index scan

32


# 32

Misc load testing using httperf• RightScale provides ServerTemplates in the marketplace

• https://my.rightscale.com/library/server_templates/Httperf-Load-Tester-11H1/18316

• Tutorial on httperf setup and configuration• http://support.rightscale.com/03-Tutorials/02-AWS/E2E_Examples/E2E_Gaming_Deployment/Adding_Httperf_Load_Tester

https://my.rightscale.com/library/server_templates/Httperf-Load-Tester-11H1/18316

http://support.rightscale.com/03-Tutorials/02-AWS/E2E_Examples/E2E_Gaming_Deployment/Adding_Httperf_Load_Tester

33

Questions?

Rafael Saavedra - VP Engineering, RightScale

June 8th, 2011

optimizing your cloud applications in rightscale

Technology

performance problems

performance votestags

total cluster activity

alerts allocation of

new relic rpmdirect

heat mapseach horizontal

new resources

consistent arrays