(pfc307) auto scaling: a machine learning approach | aws re:invent 2014
DESCRIPTION
Auto Scaling groups used in conjunction with auto-scaling policies define when to scale out or scale in instances. These policies define actionable states based on a defined event and time frame (e.g., add instance when CPU utilization is greater than 90% for 5 consecutive minutes). In this session, Electronic Arts (EA) discusses a pro-active approach to scaling. You learn how to analyze past resource usage to help pre-emptively determine when to add or remove instances for a given launch configuration. Past data is retrieved via Amazon CloudWatch APIs, and the application of supervised machine learning models and time series smoothing is discussed.TRANSCRIPT
November 12, 2014, Las Vegas NV
Sumit Amar, Electronic Arts
var client = new Amazon.CloudWatch.AmazonCloudWatchClient();
var response = client.GetMetricStatistics(
new GetMetricStatisticsRequest
{
Dimensions = new List<Dimension> {
new Dimension { Name = "InstanceId", Value = instanceId } },
StartTime = startDate, //2014-11-05
EndTime = endDate.Date.AddDays(1).Date.AddMilliseconds(-1), //2014-11-06,
Namespace = "AWS/EC2",
Statistics = new List<string>{ "Average", "Maximum", "Minimum","Sum","SampleCount" },
MetricName = metricName, //CPUUtilization, DiskReadBytes, NetworkIn, more etc..
Period = interval, //seconds – pass 60 * 60 for an hourly range
});
//### CloudWatch GetMetricsStatistics returns unordered data points, ergo..
response.Datapoints.Sort((a,b) => a.Timestamp.CompareTo(b.Timestamp));
Y = a + b X
Time Actual(Y) Deviation X(from mid) XY X2 Yd
8am 83 -3 -249 9 72.22
9am 60 -2 -120 4 61.29
10am 54 -1 -54 1 50.36
11am 21 0 0 0 39.43
12p 22 1 22 1 28.50
1p 13 2 26 4 17.57
2p 23 3 69 9 6.64
N=7 ∑𝑌 = 276 ∑X=0 ∑XY=-306 ∑X2=28
Here: Y = a + b X
a = ∑𝑌/𝑁 = 276/7 = 39.43
b = ∑𝑋𝑌
∑𝑋2 = -306/28 = -10.93
Y = 39.43 – 10.93 X
For X = -3 (8am): Y8am = 39.43 – (10.93 * -3) = 72.22 and so on for other times.
∑Y = Na + b ∑X
∑XY = a ∑X + b ∑X2
∑Y = Na + b ∑X + c ∑X2
∑XY = a ∑X + b ∑X2 + c ∑X3
∑X2Y = a ∑X2 + b ∑X3 + c ∑X4
Yd = a + b X + c X2
∑Y = Na + c ∑X2
∑XY = b ∑X2
∑X2Y = a ∑X2 + c ∑X4
EWMA Article
EWMA on Wikipedia
k-NN lecture
k-NN on Wikipedia
MSE
SSE
Machine Learning Course
