processor analysis: generational performance of intel® processors in an expedient cloud

Processor Analysis: Generational

Performance of Intel® Processors in an

Expedient Cloud Benchmarking different Intel® Processors in Expedient Generation 2, Generation 3, Generation 4 and

Generation 5 Offerings

By Cloud Spectator November 2013

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4, and 5 - November 2013

Copyright Cloud Spectator, LLC 2013. All rights reserved

1

Table of Contents

Introduction 2

Executive Summary 3

Processor Performance 4 Floating Point Math 5 Integer Math 6 Compression 7 Encryption 8 Physics 9

RAM Performance 10 Read from cached RAM 11 Read from uncached RAM 12 RAMspeed SMP 13

Disk Performance 14 Local Copy 15 Sequential Read 16 Sequential Write 17 Random Seek + R/W 18 Disk Mark 19

Appendix A: Methodology 20 Appendix B: Terms & Definitions 21 Appendix C: Test Descriptions 21

About Cloud Spectator 23



2

Introduction Performance Testing

Cloud Spectator monitors the processor, RAM, storage, and internal network performance of over 20 of the world’s most well-known IaaS services to

understand important aspects of virtual server performance. Tests are run at least three times per day, 365 days per year to capture var iability in

addition to performance level. Tests are chosen based on reliability and practicality. The goal is to provide an indication of where certain providers

perform well relative to others. This can give consumers an indication of which services would be best for their application(s) by understanding the

performance of provider resources most critical to that application.

Singular benchmarks alone should not be the only deciding factor in the provider selection process. Feature sets, configuration matches, pricing and

ancillary services such as security, compliance, and disaster recovery should always factor into any vendor selection process. However,

performance is a very important piece to the puzzle.

The Comparison

For the purpose of generating this document, Cloud Spectator measured the processor, RAM and disk performance of Expedient’s virtual machines,

across 2nd, 3rd, 4th, and 5th.Generation offerings. Each offering uses different Intel® processor models. The goal was to understand how Intel

processor performance has increased over time, using Expedient’s offerings as a practical cloud environment to test in. In addition, RAM and disk

performance was also examined to account for the overall performance of the server . Over a period of fifteen (15) days, Cloud Spectator ran the

same benchmark tests across Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. Tests were run from October 12th through October 26th, 2013.

Cloud Spectator accounted for performance capability and stability for each environment to understand the value each one delivers to its users.

Tests were run on 8GB servers with 4 vCPUs (for more details, see Methodology section in Appendix A). Statistical measures used to illustrate

improvement included: average, standard deviation, coefficient of variation (CV), 15-day high and 15-day low. Please see Appendix B for definitions

of each statistical measure.

“The goal was to understand how Intel processor performance has evolved over time, using Expedient’s offerings

as a practical cloud environment to test in.”

Performance testing and benchmarking of cloud computing platforms is a complex task, compounded by the differences between providers and the

use cases of cloud infrastructure users. IaaS services are utilized by a large variety of industries, and performance metrics cannot be completely

understood by simply representing cloud performance with a single value. When selecting a cloud computing provider, IT professionals consider

many factors: feature-sets, cost, security, location and more. However, performance is a key issue that drives many others including cost.

Processors

The processors below are used in each given offering: Expedient Generation 2: Intel® Xeon® Processor E5520 2.27GHz

Expedient Generation 3: Intel® Xeon® Processor X5650 2.67GHz

Expedient Generation 4: Intel® Xeon® Processor E5-2670 2.60GHz

Expedient Generation 5: Intel® Xeon® Processor E5-2670 v2 2.50GHz



3

Executive Summary

Findings Newer Generations Deliver Significantly More Processor, RAM and Disk Performance Generation 5 offerings performed better than Generation 2 offerings in the following areas:

CPU performance improved from Generation 2 to 5 by 62% RAM performance improved from Generation 2 to 5 by 76%

Generation 5 offerings performed better than Generation 3 offerings in the following areas:


Generation 5 offerings performed better than Generation 4 offerings in the following areas



CPU performance improved from Generation 2 to 4 by 44% RAM performance improved from Generation 2 to 4 by 60% Disk performance improved from Generation 2 to 4 by 134%



Disk performance improved from Generation 3 to 5 by 161% Generation 3 offerings performed better than Generation 2 offerings in the following areas


Newer Generations Have Improved Processor and RAM Stability Coefficients of variation (CVs), representing variability, decreased from Generation 2 to Generation 5 by an average of 72%.

Generation 5 also decreased in variability by 69% and 88% from Generation 4 and Generation 3 respectively. Averages in each scenario are much closer to the highs than lows, meaning that performance levels are more often higher than the mean. This means that the lows are responsible for most of the variability. Customers should expect a high level of performance most of the time.

CPU TESTS: Highs are an average of 1.5 standard deviations from the mean, while lows are an average of 3.4 standard deviations from the mean.

RAM TESTS: Highs are an average of 2.1 standard deviations from the mean, while lows are an average of 3.6 standard deviations from the mean.

DISK TESTS: Highs are an average of 1.4 standard deviations from the mean, while lows are an average of 4.4 standard deviations from the mean.



4

Processor Performance

Why Processor Performance Matters In a public cloud environment, resources are shared across virtual machines within the same physical server. Hardware components such as the processor are virtualized and usually shared across VMs. While modern-day physical processors may accomplish tasks at a desirable rate, virtualized processors, which may or may not receive a dedicated core’s performance, may be less powerful. As the cloud industry progresses, though, with the introduction of newer cloud offerings with the latest processor technology, users experience a significant performance benefit while paying a comparable price for the IaaS service. This improvement in performance is highlighted within the following tests:



5

Floating Point Math Floating Point Math and Integer Math tests are synthetic benchmarks commonly used in gauging and comparing CPU performance because the operations used make up the basic operations in all computer software. The Floating Point Math test uses the same calculations as the Integer Math test, except using numbers with decimals, which are commonly used in modern-day applications. The floating point math performance level and variability showed a significant improvement from Generation 2 through 4. Generation 3 average performance improved by 22% relative to Generation 2. Generation 4 average performance improved on Generation 3 by 35%. Generation 5 average performance improved on Generation 4 by 9%. Overall, this represents a 79% performance increase from Generations 2 to 5. Variability also decreased by a significant amount, 96% , from Generation 2 to Generation 5. These results were in line with what was expected after Intel’s introduction of Advanced Vector Extensions (AVX) technology in Generation 4 (Intel Xeon Processor E5-2670 2.60GHz). Intel® Advanced Vector Extensions (Intel® AVX) is an instruction set extension designed for applications that are floating point intensive. Intel AVX results in better management of data, improving performance through wider vectors, new extensible syntax, and rich functionality.

Note: numbers expressed in millions of operations per second.

0

1000

2000

3000

4000

5000

6000

7000

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

MIP

S

CPU TEST - Floating Point Math

Gen 2

Gen 3

Gen 4

Gen 5

PROVIDER AVERAGE STANDARD DEVIATION CV 15-DAY HIGH 15-DAY LOW

2nd Generation 3208 185 5.8% 3335 2430

3rd Generation 3910 138 3.5% 4143 3591

4th Generation 5260 145 2.8% 5464 4675

5th Generation 5756 14 0.2% 5785 5691

The graph above compares Intel CPU floating point math performance on Windows servers betw een Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the PassMark floating point math benchmark test for each provider over a period of 15 days, with tw o (high and low) data

points shown for each day. For more information on the PassMark floating point math test, please see Appendix C.



6

Integer Math Floating Point Math and Integer Math tests are synthetic benchmarks commonly used in gauging and comparing CPU performance because the operations used make up the basic operations in all computer software. The Integer Math test runs mathematical operations of addition, subtraction, multiplication, and division of whole numbers to give an indication of raw CPU throughput. Integer math performance improved by a large margin from Generation 2 to Generation 3 and by much smaller margins from Generation 3 to Generation 4 and Generation 4 to Generation 5. Generation 3 saw a significant performance improvement of 22% over Generation 2. Generation 4 average performance improved slightly, by 4%, over Generation 3. Generation 5 performance improved slightly more, by 5% over Generation 4. The total performance improvement from Generation 2 to Generation 5 was 34%. Performance variability for all four generations was very low with the highest CV being 1.9%.

Note: numbers expressed in millions of operations per second.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

MIP

S

CPU TEST - Integer Math

Gen 2

Gen 3

Gen 4

Gen 5


2nd Generation 5911 94 1.6% 5976 5508

3rd Generation 7208 134 1.9% 7412 6930

4th Generation 7513 65 0.9% 7615 7325

5th Generation 7909 13 0.2% 7938 7849

The graph above compares Intel CPU integer math performance on Windows servers betw een Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the PassMark integer math benchmark test for each prov ider ov er a period of 15 days, with two (high and low ) data points shown for each day. For more information on the PassMark integer math test, please see Appendix C.



7

Compression Compressing files into smaller blocks of data is a common task used in software applications like Email and file backups. Compressing the same file in the same manner across different machines reveals a 43% improvement for this CPU-intensive task from the Expedient Generation 2 offering with the Intel Xeon Processor E5520 to the Expedient Generation 5 offering with the Intel Xeon Processor E5-2670 v2. The compression results reveal a clear improvement from Generation 2 to 3, and from 3 to 4. The Expedient Generation 3 offering saw a 20% improvement in compression performance from Generation 2, with decreased performance variability. From Generation 2 to 3, the CV decreased by 32%, indicating more predictable and steady performance levels. Generation 4 displays further CPU compression performance over Generation 3. Performance increased by 12% from Generation 3 to 4, and variability decreased by 31%. The performance improvement from Generation 4 to 5 was less noticeable than the preceding generations, while stability of performance improved drastically. Performance increased by 7%, while variability decreased by 84%.

Note: numbers expressed in kilobytes processed per second.

0

1000

2000

3000

4000

5000

6000

7000

8000

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Kb

/sec

CPU TEST - Compression

Gen 2

Gen 3

Gen 4

Gen 5


2nd Generation 5276 216 4.1% 5422 4328

3rd Generation 6306 176 2.8% 6621 5805

4th Generation 7044 136 1.9% 7232 6506

5th Generation 7528 24 0.3% 7558 7471

SAMPLE

The graph above compares Intel CPU compression performance on Windows servers betw een Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the PassMark compression benchmark test for each prov ider ov er a period of 15 days, with two (high and low ) data points shown for each day. For more information on the PassMark compression test, please see Appendix C.



8

Encryption This encryption test encrypts random blocks of data using different encryption techniques. The machine will also create a hash of the data, which ensures that the encrypted data is not compromised. From a performance view, encryption uses many of the algor ithms used in the Integer and Floating Point math test, as well as more complex ones such as ‘to the power of’ functions. Encryption is used in many Internet-based applications to ensure privacy, including Internet browsers and communications tools. Generation 3 shows a significant improvement in encryption performance over Generation 2. Generation 3 displays a 53% improvement in encryption performance over Generation 2. Generation 4 slightly decreased in average encryption performance, decreasing by 1% compared to Generation 3. Generation 5 improved by increasing performance by 16% over Generation 4, and a 74% improvement over Generation 2. Generation 3 variability decreased significantly, by 26%, from Generation 2. The Generation 4 CV decreased 28% compared to Generation 3. Generation 5 variability decreased 87% from Generation 4, which was part of a 93% decrease in CV from Generation 2 to Generation 5. These results were in line with what was expected after Intel’s introduction of Intel AES-NI technology present in Generations 3 to 5 (Intel Xeon Processor X5650 2.67GHz and Intel Xeon Processor E5-2670 2.60GHz, respectively). Intel® Advanced Encryption Standard New Instructions (Intel® AES-NI) introduces new instructions for enhanced security and speed while implanting some intensive sub-steps of the AES algorithm into the hardware for faster execution.

Note: numbers expressed in megabytes transferred per second.

0

200

400

600

800

1000

1200

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Mb

/s

CPU TEST - Encryption

Gen 2

Gen 3

Gen 4

Gen 5


2nd Generation 627 25 3.9% 648 492

3rd Generation 956 28 2.9% 1003 863

4th Generation 944 20 2.1% 971 881

5th Generation 1092 3 0.3% 1096 1085

SAMPLE The graph above compares Intel CPU encryption performance on Windows servers betw een Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the PassMark encryption benchmark test for each prov ider over a period of 15 days, with tw o (high and low ) data points shown for

each day. For more information on the PassMark encryption test, please see Appendix C.



9

Physics Using the Tokamak Physics Engine, the Physics benchmark performs collision calculations and visual rendering simultaneously. The Tokamak Physics Engine is a high performance, real-time physics library designed specifically for games and has a built-in collision functionality, which is used for the test. Average performance level in the Physics Test improved steadily with more modern generations. Generation 3 average performance improved by 23% over Generation 2 while Generation 4 improved by 18% over Generation 3, and Generation 5 improved by 26% over Generation 4. This brings the total physics performance improvement from Generation 2 to Generation 5 to 82%. Variability decreased by 92% from Generation 2 to Generation 5.

Note: numbers expressed in frames per second.

0

100

200

300

400

500

600

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

FP

S

CPU TEST - Physics

Gen 2

Gen 3

Gen 4

Gen 5


2nd Generation 260 12 4.7% 284 221

3rd Generation 320 25 7.8% 360 264

4th Generation 377 18 4.7% 411 326

5th Generation 475 2 0.4% 481 469

The graph above compares Intel CPU physics calculation performance on Windows servers between Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the PassMark physics benchmark test for each prov ider over a period of 15 days, with tw o (high and low ) data points shown for each day. For more information on the PassMark physics test, please see Appendix C.



10

RAM Performance

Why RAM Performance Matters RAM is an integral part of any server. The RAM, or memory, is the temporary storage that the CPU can use to access data quickly, compared to waiting for data to slowly be pulled from the hard drive storage. Playing a key factor in a wide range of applications, RAM performance can be just as important as CPU performance. Applications are loaded onto the RAM for faster access and better user experience. RAM is greatly utilized in servers running databases, games or media. Many new application workloads are now taking advantage of hosted environments where a large amount of RAM can be accessed. Growth in in-memory databases, gaming platforms, and many other use cases continue to raise the value of RAM performance.



11

Read from cached RAM This test measures the time taken to read a small block of memory that is small enough to be held entirely in cache. The test not only measures RAM, but also the CPU cache, which decreases the time needed for a CPU to retrieve data directly from the memory . For workloads that fully utilize the CPU, it is important to be able to process stored information as quickly as possible. The Intel Xeon E5-v2 processors have 25MB of CPU cache and 1866 MHz memory speeds to provide fast access to frequently used data. Performance improvements from Generation 2 to 3 and Generation 4 to 5 are relatively modest compared to the jump from Generation 3 to 4 as shown by the Read from cached RAM test. However, performance stability is the tradeoff for higher performance in Generation 3 and 4. Average performance increases by 19% from Generation 2 to 3. Generation 4 improves over Generation 3 by a significant 62% , while Generation 5 improves upon Generation 4 by a modest 22%. Overall performance increased by 136% from Generation 2 to 5. Performance is the most stable for Generation 2 and 5 with CVs of 0.7%, while variability decreases by 72% and 63% for Generation 3 and Generation 4 respectively.


0

5000

10000

15000

20000

25000

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Oct

24

Oct

25

Oct

26

Mb

/s

RAM TEST - Read from cached RAM

Gen 2

Gen 3

Gen 4

Gen 5


2nd Generation 9006 62 0.7% 9105 8552

3rd Generation 10759 267 2.5% 11119 10201

4th Generation 17404 327 1.9% 18401 16718

5th Generation 21220 149 0.7% 21444 20629

The graph above compares RAM performance on Windows servers between Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Read from cached RAM benchmark test for each prov ider over a period of 15 days, with two (high and low ) data points shown for each day. For more information on the Read from cached RAM test, please see Appendix C.



12

Read from uncached RAM This test measures the time taken to read a large block of memory. Measured in GB/s, larger values are better. The test will use a 256 MB block, too large for any CPU cache in Intel processors, measuring the efficiency of the CPU to read directly from RAM. The results of the Read from uncached RAM test reveal steady increases in average performance from Generation 2 to Generation 5. The improvement from Generation 2 to 3 was 14%, while the percentage of improvement nearly doubled for the increase from Generation 3 to 4, which was 27%. The increase of Generation 4 to 5 was a lower 17%, while the overall improvement of Generation 5 from Generation 2 was 69%. As shown in the graph below, Generation 3 performance varies from two different levels, with a period of lower performance from October 16th to 24th. As a result, Generation 3 performance variability increases over Generation 2 by 110%. The subsequent server generations decreased in variability to offer more stable performance. Generation 4 variability decreases by 83% over Generation 3, and Generation 5 variability decreases by 50% over Generation 4. The performance variability decreased from Generation 5 to 2 by 81%.


0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Oct

24

Oct

25

Oct

26

Mb

/s

RAM TEST - Read from uncached RAM

Gen 2

Gen 3

Gen 4

Gen 5


2nd Generation 5100 257 5.0% 5963 3873

3rd Generation 5822 618 10.6% 6737 4764

4th Generation 7394 137 1.9% 7586 6836

5th Generation 8617 80 0.9% 8799 8293

The graph above compares RAM performance on Windows servers between Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Read from uncached RAM benchmark test for each provider over a period of 15 days, with two (high and low) data points shown for each

day. For more information on the Read from uncached RAM test, please see Appendix C.



13

RAMspeed SMP Aggregate of several tests that measure Copy, Scale, Add and Triad functions for both integer and floating point values. RAMspeed’s SMP version adds functionality for multi-core systems to test the memory bandwidth. Server performance usually increases as newer technology is brought into the fold. According to the RAMspeed SMP test, the improvement from Generation 2 and 3 servers to Generation 4 and 5 servers is great. While Generation 3 improves upon Generation 2 with only a 7% increase, Generation 4 improves upon Generation 3 with a significant 49% increase. Generation 5 improves upon Generation 4 servers only slightly with a 6% increase in performance. As for performance variability, Generation 3 was the least stable for the RAM test again. Generation 3 servers were 213% more variable than Generation 2 servers. Generation 4 servers addressed the performance stability problem by decreasing variability ov er Generation 3 servers by 76%. Generation 5 servers decreased variability again, dropping 8% over Generation 4, and an overall 32% from Generation 5 over Generation 2.


0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Oct

24

Oct

25

Oct

26

Mb

/s

RAM TEST - RAMspeed SMP

Gen 2

Gen 3

Gen 4

Gen 5


2nd Generation 5467 133 2.4% 6041 5087

3rd Generation 5848 444 7.6% 6602 5090

4th Generation 8701 156 1.8% 8939 8129

5th Generation 9250 153 1.7% 9509 8443

The graph above compares RAM performance on Windows servers between Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the RAMspeed SMP benchmark test for each provider over a period of 15 days, w ith tw o (high and low) data points shown for each day. For more information on the RAMspeed SMP test, please see Appendix C.



14

Disk Performance

Why Disk Performance Matters The disk is where it all starts. All the data has to go somewhere, and the disk is where it is stored. Whether they are using traditional spinning hard drives, or newer solid-state drives (SSD), service vendors need to provide customers with reliable and efficient means of storing their data. The disk needs to be able to quickly and reliably read/write the data it receives or that is requested from the other components of the server. The most common measurement for storage system performance is the input/output operations per second (IOPS) which measures how long the system has to wait as data is read from, or written to, the disk. As the quantity of data stored and analyzed scales upwards, every sing le IOPS a disk can handle will carry greater significance. Note: While the primary focus of the report is based upon the improvements in performance from newer Intel processors, the storage system is important to examine as well when addressing the general topic of server performance. The disk performance testing was limited to Generation 2, 3 and 4 because Generation 5 was tested on pre-production servers that did not have production level disk subsystems. When Generation 5 servers will be brought to into live production, the servers will be hosted on similar, if not better, storage systems as Generation 4, thus the Generation 4 disk results can also be anticipated for Generation 5.



15

Local Copy The Local Copy test performs a simple copy of a 10 GB file on the same storage device. It is a combination of sequential read and write, which is common for file uploading, downloading, and sharing. Generation 2 and Generation 3 servers have similar levels of performance relative to the significantly improved performance of Generation 4 servers. The average performance of Generation 3 servers decreased from Generation 2 by 19%, while performance variability increased by 237%. Generation 4 performance improved over Generation 3 by 255%, and Generation 2 by 200%. Variability for Generation 4 was neg ligibly higher than Generation 3 by 2%, but higher than Generation 2 by 244%. The overall performance increase outweighs the increase in variability for Generation 4 compared to Generation 2 as the 15-day low of Generation 4 is still 55% higher than the 15-day high of Generation 2.


0

50

100

150

200

250

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Oct

24

Oct

25

Oct

26

Mb

/s

DISK TEST - Local Copy

Gen 2

Gen 3

Gen 4


2nd Generation 53 3 4.8% 55 40

3rd Generation 45 7 16.3% 53 6

4th Generation 159 26 16.6% 196 62

The graph above compares disk performance on Windows servers between Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Local Copy benchmark test for each provider over a period of 15 days, with two (high and low) data points shown for each day. For more

information on the Local Copy test, please see Appendix C.



16

Sequential Read A large test file is created on the disk under test. During the test period the file is read several times sequentially. The Sequential Read test shows Generation 4 offering higher performance over the Generation 2 and 3 offerings, but also higher variability of performance. Generation 3 performance was 14% lower than Generation 2, while Generation 4 performance was 167% higher than Generation 3, and 129% higher than Generation 2. Variability increased for Generation 3 over Generation 2 by 104%, while Generation 4 variability increased over Generation 3 by a smaller 9%.


0

50

100

150

200

250

300

350

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Oct

24

Oct

25

Oct

26

Mb

/s

DISK TEST - Sequential Read

Gen 2

Gen 3

Gen 4


2nd Generation 108 8 7.5% 116 72

3rd Generation 92 14 15.4% 111 43

4th Generation 246 41 16.8% 303 102

The graph above compares disk performance on Windows servers between Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the

scores from the Passmark Sequential Read benchmark test for each provider over a period of 15 days, with two (high and low) data points shown for each day. For more information on the Passmark test, please see Appendix C.



17

Sequential Write A large file is written to the disk under test. The 500MB file is written sequentially. The Sequential Write test shows Generation 4 offering significantly higher performance over the Generation 2 and 3 offerings. Generation 3 performance was 12% lower than Generation 2, while Generation 4 performance was 133% higher than Generation 3, and 104% higher than Generation 2. Performance variability increased for Generation 3 over Generation 2 by 163%. Generation 4 variability decreased over Generation 3 by 50%, but increased over Generation 2 by 31%. The performance of the disk write speed of Generation 4 compared to Generation 2 outweighs the variability of performance as the 15-day low of Generation 4 is 32% greater than the 15-day high of Generation 2.


0

20

40

60

80

100

120

140

160

180

200

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Oct

24

Oct

25

Oct

26

Mb

/s

DISK TEST - Sequential Write

Gen 2

Gen 3

Gen 4



3rd Generation 72 12 16.5% 86 21

4th Generation 167 14 8.2% 185 115

The graph above compares disk performance on Windows servers between Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Passmark Sequential Write benchmark test for each prov ider over a period of 15 days, with tw o (high and low) data points shown for each day. For more information on the Passmark test, please see Appendix C.



18

Random Seek + R/W A large test file is created on the disk under test. The file is read randomly; a seek is performed to move the file pointer to a random position in the file, a 16KB block is read or written then another seek is performed. The amount of data actually transferred is highly dependent on the disk seek time. The results of the Random Seek + R/W test show Generation 4 servers more than doubling the performance of Generation 2 and Generation 3 servers. Generation 3 has an average performance slightly greater than Generation 2, by a negligible 2% increase. Generation 4 has a performance increase of 113% and 118% over Generation 3 and Generation 2 respectively. Performance variability is highest on Generation 3 , with a CV near 20%. Generation 3 variability increases by 234% over Generation 2, while Generation 4 variability decreases by 40% over Generation 3. Generation 4 still has greater variability compared to Generation 2, increasing by 61%.


0

50

100

150

200

250

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Oct

24

Oct

25

Oct

26

Mb

/s

DISK TEST - Random Seek + R/W

Gen 2

Gen 3

Gen 4



3rd Generation 89 18 19.7% 133 39

4th Generation 191 14 7.6% 207 110

The graph above compares disk performance on Windows servers between Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Passmark Random Seek + R/W benchmark test for each provider over a period of 15 days, with tw o (high and low) data points show n for each day. For more information on the Passmark test, please see Appendix C.



19

Disk Mark Average of the previous three benchmarks. The average is scaled to be indexed as a score on a scale created by Passmark to relationally compare disk performance of different systems. The Disk Mark test shows the overall disk performance of Generation 4 as a significant improvement over Generation 2 and Generation 3. Generation 3 performed lower than Generation 2 by 9%, while also 166% higher in variability. Generation 4 performed higher than Generation 3 by 139%, and Generation 2 by 118%. Generation 4 also had 40% decreased variability over Generation 3, but a 61% increase over Generation 2. However, the overall higher performance of Generation 4 should outweigh the increased variability compared to Generation 2. Examining the 15-day low of Generation 4 compared to the 15-day high of Generation 2, there is a 38% increase in performance.

Note: numbers expressed as calculated score.

0

500

1000

1500

2000

2500

3000

Oct

12

Oct

13

Oct

14

Oct

15

Oct

16

Oct

17

Oct

18

Oct

19

Oct

20

Oct

21

Oct

22

Oct

23

Oct

24

Oct

25

Oct

26

Sco

re

DISK TEST - Disk Mark

Gen 2

Gen 3

Gen 4


2nd Generation 1000 56 5.6% 1060 685

3rd Generation 915 136 14.9% 1182 375

4th Generation 2184 197 9.0% 2462 1468

The graph above compares disk performance on Windows servers between Expedient’s 2nd, 3rd, 4th, and 5th Generation offerings. The graph shows the scores from the Passmark Disk Mark benchmark test for each prov ider ov er a period of 15 days, with two (high and low ) data points shown for each day. For

more information on the Passmark test, please see Appendix C.



20

Appendix

Appendix A

Methodology

Server Setup:

The following server configurations were set up on each environment. The Intel® CPU model that each environment uses is also detailed. Expedient Generation 2 http://www.expedient.com/products/cloud-computing/public.php

CPU: Intel® Xeon® Processor

E5520 2.27GHz

OS: Windows Server 2008 R2 vCPUs: 4

RAM: 8GB Disk: 50GB

Expedient Generation 3 http://www.expedient.com/products/cloud-computing/public.php

CPU: Intel® Xeon® Processor

X5650 2.67GHz


RAM: 8GB Disk: 50GB


CPU: Intel® Xeon® Processor E5-

2670 2.60GHz


RAM: 8GB Disk: 50GB


CPU: Intel® Xeon® Processor E5-

2670 v2 2.50GHz


RAM: 8GB Disk: 50GB

Tests Used: The benchmarks used to test the providers listed in this report are derived from a number of sources. Most of the tests use benchmarks from the CPU and disk tests of PassMark Software. For descriptions of test used, please see the Test Descriptions below. CPU

Integer Math

Floating Point Math

Compression

Phy sics

Encry ption

RAM Read Cached

Read Uncached

Write Large Block

RAMspeed SMP

Storage Disk Mark

Random Seek & R/W

Sequential Read

Sequential Wri te

Local Copy

For the descriptions and details of each test, please see Appendix C.

Timeframe: The test period ranged from 10/12/2013 though 10/26/2013. Data Collection: Performance data for each test was collected 10 times per day, every day throughout the above testing periods. Data shown in the graphs only display the highest and lowest points of each day. Statistical measures in the charts were calculated using all data points to obtain more accurate estimations of variability. Cloud Spectator obtains cloud servers by purchasing the server space directly from the providers as any user would. For certain providers, the client may reimburse Cloud Spectator for the server space needed for data collection relevant to that active project. Cloud Spectator collects and compiles the data into the CloudSpecs database and translates it in to a visual display.

http://www.expedient.com/products/cloud-computing/public.php








http://passmark.com/

Cloud Spectator Performance Report: Expedient/Intel Generations 2, 3, 4 and 5 - November 2013

Copyright Cloud Spectator, LLC 2013. All rights reserved.

Appendix B

Terms and Definitions For the purpose of understanding the relational values of the data, all numbers within the tables below each graph are expressed as whole numbers except the percentages, which are expressed to the tenth decimal point. Percentages are expressed in that manner to account for instances when the coefficient of variation (CV), which is expressed as a percentage, falls below 1%, indicating a high degree of performance predictability.

Average: When describing averages, Cloud Spectator refers to the average numerical value over a period of 15 days from October 12th until October 26th,

2013, taking into account each and every data point. Average scores can be found inside the tables underneath each graph within this document. The average is used to summarize the data in a simplified overview.

Standard Deviation: The standard deviation is calculated over a period of 15 days from October 12th until October 26th 2013. The standard deviation can be found inside the tables underneath each graph within this document. The standard deviation is used to understand the amount of variation from the average benchmark score; i.e., how predictable a server’s performance is for that test. The standard deviation can only be used to understand the amount of variation within a certain environment, and cannot be used to compare different environments because of performance differences.

Coefficient of Variation (CV): The coefficient of variation is expressed as a percentage. The CV can be found inside the tables underneath each graph within this document. The CV is a measure of precision. It normalizes the standard deviation as a percentage of the average, which can be compared across providers. A lower CV means more stable performance.

[(Standard Deviation) / (Average)] * 100

15-Day Highs and Lows: From the tested period between October 12th and October 26th 2013, Cloud Spectator extracts and presents the highest and lowest achieved scores by each provider in the tables underneath the graphs within this document. Appendix C Test Descriptions The following tests were taken from PassMark Software to measure the CPU performance in different Expedient environments: Compression Test: The Compression Test measures the speed that the CPU can compress blocks of data into smaller blocks of data without losing any of the original data. The result is reported in Kilobytes per Second. This test uses complex data structures and complex data manipulation techniques to per form a function that is very common in software applications, ranging from backup software to Email software. The compression test uses an Adaptive encoding algorithm based on a method described by from Ian H. Witten, Radford M. Neal, and John G. Cleary in an article called “Arithmetic Coding for Data Compression”. The system uses a model which maintains the probability of each symbol being the next encoded. It reports a compression rate of 363% for English text, which is slightly better than the classic Huffman method. This tests uses memory buffers totaling about 16kb per core. Encryption Test: The Encryption Test encrypts blocks of random data using several different encryption techniques, such that the resulting data can only be accessed by someone with the encryption key. It also tests the computer ’s ability to create a hash of the data, which is also a common cryptographic technique that can be used to ensure the contents of data are not tampered with. The methods used are TwoFish, AES, Salsa20 and SHA256. This test uses many of the techniques in the math tests, but also uses a large amount of binary data manipulation and CPU mathematical functions like 'to the power of'. Encryption is a very useful benchmark, as it is now very widely used in software applications, ranging from Internet browsers, communications software and many different business applications. This tests uses memory buffers totaling about 1MB per core. Where available, the test will make use of specialized CPU instruction sets to accelerate performance, such as Intel AES-NI for the AES test.



Floating Point Math Test: The Floating Point Math Test performs the same operations as the Integer Math Test, however, with floating point numbers. A floating point number is a number with a fractional part (ie. 12.568). These kinds of numbers are handled quite differently in the CPU compared to Integer numbers as well as being quite commonly used, therefore they are tested separately. This tests uses memory buffers totaling about 240kb per core. Integer Math Test: The Integer Math Test aims to measure how fast the CPU can perform mathematical integer operations. An integer is a whole number with no fractional part. This is a basic operation in all computer software and provides a good indication of 'raw' CPU throughput. The test uses large sets of random 32-bit and 64-bit integers and adds, subtracts, multiplies and divides these numbers. This tests uses integer buffers totaling about 240kb per core. Physics Test: The Physics Test uses the Tokamak Physics Engine to perform a benchmark of how fast the CPU can calculate the physics interactions of several hundred objects colliding. This tests uses memory buffers totaling about 30MB per core. Read from cached RAM: This test measures the time taken to read a small block of memory. The block is small enough to be held entirely in cache. Read from uncached RAM: This test measures the time taken to read a large block of memory. Measured in GB/s, larger values are better. The test will uses a 256 MB block. The block is too large to be held in cache. Reads on 32-bit system uses DWORDS and QWORDS on 64-bit version of the software. Write large RAM blocks: This test measures the time taken to write information into memory. Measured in GB/s, larger values are better. Similar to the read uncached test, except writing is performance instead. The test also uses a 256 MB block. RAMspeed SMP: Aggregate of several tests that measure Copy, Scale, Add and Triad functions for both integer and floating point values.

Copy – Transfers data from one memory location to another (A = B). Scale – Multiplies the data with a constant value before writing it (A = Bn). Add – Reads results from two different locations, adds those results and writes then to the new location (A = B + C). Triad – Merges Add and Scale. It reads data from the first memory location, scales it (multiplies it), then adds data from the second one

and writes to the new location (A = Bn + C). Local Copy: Copy of a 10 GB file to the same disk Sequential Read: A large test file is created on the disk under test. During the test period the file is typically read several times sequentially. Sequential Write: A large file is written to the disk under test. The 500MB file is written sequentially from start to end. Test conditions are otherwise the same as the read test. Random Seek + R/W: A large test file is created on the disk under test. The file is read randomly; a seek is performed to move the file pointer to a random position in the file, a 16KB block is read or written then another seek is performed. The amount of data actually transferred is highly dependent on the disk seek time. Disk Mark: Average of the previous three benchmarks. The average is then scaled up by multiplying the average with a 'magic' number in order to make the number larger.



About Cloud Spectator

Cloud Spectator is the premier, international cloud analyst group focused on infrastructure pricing and server performance. Since 2011, Cloud

Spectator has monitored the cloud infrastructure (IaaS) industry on a global scale and continues to produce research reports for businesses to make

informed purchase decisions by leveraging its CloudSpecs utility, an application that automates live server performance tests 3 times a day, 365

days a year with use of open source benchmark tests. Currently, the CloudSpecs system actively tracks 20 of the top IaaS providers around the

world.

Cloud Spectator

800 Boylston Street, 16th Floor

Boston, MA 02199

Website: www.cloudspectator.com

Phone: (USA) 1-617-300-0711

® Intel, the Intel logo, and Xeon are trademarks of Intel Corporation in the U.S. and other countries.

http://www.cloudspectator.com/

processor analysis: generational performance of intel® processors in an expedient cloud

Documents

ram performance

intel processor performance

performance capability

performance level

overall performance

copyright cloud spectator

internal network performance

practical cloud environment