optimizing your cloud applications in rightscale

Download Optimizing Your Cloud Applications in RightScale

Post on 20-Aug-2015




1 download

Embed Size (px)


  1. 1. Optimizing Your Cloud Applications in RightScale
    Rafael H. Saavedra - VP Engineering, RightScale
    June 8th, 2011
  2. 2. Introduction
    3-tier application architecture
    Vertical & horizontal scaling
    RightScale monitoringand cluster graphs
    New Relic RPM
    Support for optimizing DB performance
  3. 3. Multi-tenancy
    Shared resource pooling
    Geo-distribution and ubiquitous network access
    Service oriented
    Dynamic resource provisioning
    Utility based pricing
    Cloud computing characteristics
  4. 4. No upfront investment
    Lowering operating costs
    Highly scalable
    Easy access
    Reduces business risk and maintenance costs
    Cloud computing advantages
  5. 5. 3-tier application architecture
    Load balancers
    A farm of application servers
  6. 6. Instance size (vertical scaling)
    Instance autoscaling (horizontal scaling)
    Server arrays
    RightScale support for performance optimization
    ServerTemplates are configured to capture performance data
    Hardware & OS monitoring data
    Specialized plugins MySQL, HAProxy, Apache, NgInx, IIS, etc
    Monitoring graphs: individual, cluster, stacked, heat maps
    Alerts & escalations
    New Relic RPM
    Cloud performance optimization
  7. 7. Compute units vs memory vs cost
    Scaling up spectrum of instance sizes
  8. 8. Server arrays provide horizontal scaling
  9. 9. The array scales up or down based on performance votes
    Tags allow scaling on an arbitrary decision set
    Decision threshold controls reaction time
    Sleep time allows new resources to have an impact
    Scaling can be time dependent
    Detailed setup instructions: http://bit.ly/c1oLr2
    Fast response to changes in load conditions using alerts
    Allocation of servers to availability zones based on weights
    Deployment-based so configuration is consistent
    Arrays can be pre-scaled to support anticipated demand
    Server arrays provide horizontal scaling
  10. 10. Cluster monitoring
    Individual graphs
    Good for a dozen servers
    Displays all standard graphs with full detail
    Stacked graphs
    Displays the contribution of many servers to a total
    Great to see the sum and variability of activity in a cluster
    Difficult to make out individual servers
    Examples: requests/sec, cpu busy cycles, I/O bytes/sec
    Heat maps
    Displays a bar for each server
    Great to see uneven distribution across servers
    Great to quickly spot performance problems across many servers
    Difficult to read absolute values or see the total cluster activity
  11. 11. Cluster monitoring
    Current cluster monitoring: one graph per server
  12. 12. Stacked graphs
    Each color band shows contribution of one server
    Servers are stacked on top of one another
  13. 13. Heat maps
    Each horizontal strip shows one server
    The color shows how hot the server is running
  14. 14. Heat map with 100 servers
  15. 15. Stacked graph of the same 100 servers
  16. 16. Cluster monitoring architecture
    Monitoring front-end serverspull data from storage servers
    Up to 100 servers on one graph(to be increased)
    your servers
  17. 17. Real-Time App Performance Analytics
    Supports Ruby, PHP, Java & .Net
    SQL & NoSQL performance
    Web transaction tracing
    Performance notifications
    Availability monitoring
    Scalability analysis
    New Relic RPM
  18. 18. New Relic RPM
    Direct access from RightScale dashboard
  19. 19. New Relic RPM
    Historical statistics over a period of time
  20. 20. New Relic RPM
    Distribution of the most time consuming requests
  21. 21. New Relic RPM
    Statistics about response times from different countries
  22. 22. New Relic RPM
    Detailed response times by browser
  23. 23. An expensive query
    The N+1 query problem
    Finding patterns in similar requests
    New Relic RPM 3 Examples
  24. 24. Optimizing DB performance
    RightScale MySQLServerTemplates
    Configuration files tailored to instance size
    The never ending task of identifying current bottlenecks
    Disk seeks
    Performance of disk operations
    Scale up when working set cannot fit in memory avoid active swapping
    Constant monitoring of performance graphs, logs and query
    Schema considerations
  25. 25. Schema considerations
    Lookups need to be indexed
    Sorting requires an index
    Joins need to be done on indices
    Become slower as tables grow
    Compounded indices should be used consistently
    Do not abuse indices
    Each index requires a disk write
    Compact tables if they become fragmented
    Deleted rows do not remove the corresponding index entries
  26. 26. Monitoring DB performance
    Standard collectd statistics
    User vs wait time (disk operations)
    Performance of disk operations
    Scale up when working set cannot fit in memory
    Monitor INSERT, SELECT, UPDATE operations
    The breakdown of read operations can indicate missing indices
    Monitoring /var/log/mysql-slow.log file
    Identify slow queries
    Use MySQL EXPLAIN command to identify query plan
  27. 27. MySQLCollectdPlugin
    Uses MySQL SHOW STATUS command to collect statistics
    A large set of counters that are divided into 10 categories
    IO Requests
    Select Rates
    Read Rates
    Key Rates
    Commands Rates
    Query Cache
  28. 28. MySQLCollectdPlugin
    Uses MySQL SHOW STATUS command to collect statistics
  29. 29. Mysql-slow.log & explain command
    # Time: 101006 23:30:11
    # User@Host: prod[prod] @ domU-12-31-39-0F-D0-C1.compute-1.internal []
    # Query_time: 7Lock_time: 0Rows_sent: 1Rows_examined: 19785
    SELECT * FROM `ec2_elastic_ips` WHERE (`ec2_elastic_ips`.ec2_instance_id = 6810144) LIMIT 1;
    mysql> explain select * FROM `ec2_elastic_ips` WHERE (`ec2_elastic_ips`.ec2_instance_id = 6810144)LIMIT 1 G
    *************************** 1. row ***************************
    id: 1
    select_type: SIMPLE
    table: ec2_elastic_ips
    type: ALL
    possible_keys: NULL
    key: NULL
    key_len: NULL
    ref: NULL
    rows: 33332
    Extra: Using where
    1 row in set (0.00 sec)
  30. 30. MySQL performance depends on locality
    Wait time should be minimum when working set fits in memory
    Performance degrades once wait time is significant
    wait time insignificant
    user time dominates
  31. 31. MySQL reads graphs
    Read-random-next represents a table scan
    Read-next represents an index scan
  32. 32. Misc load testing using httperf
    RightScale provides ServerTemplates in the marketplace
    Tutorial on httperf setup and configuration
  33. 33. Questions?
    Rafael Saavedra - VP Engineering, RightScale
    June 8th, 2011