cloud iaas performance & price performance - vogel.de · cloud iaas performance &...

Cloud IaaS Performance & Price-Performance

Comparing Linux Compute Performance of Amazon Web Services, T-Systems Dynamic Services for Infrastructure with vCloud, Microsoft Azure and ProfitBricks

Prepared for T-Systems Commercial Report

Published on 12/4/2016

Cloud Spectator Performance Report

1

Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

Contents

1. Introduction……………………………………………………… 2

Why Performance Matters 2

2. Executive Summary…………………………………………… 4

Findings 4

3. Methodology…………………………………………………… 6

Process 6

Tests Used 7

VM Configurations and Pricing 8

Understanding Performance Results 9

Key Considerations 10

4. Performance Findings………………………………………… 11

Processor Performance 11

Memory Bandwidth 16

Disk IOPS: Sequential and Random Operations 18

Internal Network 23

5. Price-Performance Findings………………………………… 25

Processor Price-Performance 25

Memory Bandwidth Price-Performance 26

Disk IOPS Price-Performance 27

Internal Network Price-Performance 29

6. Conclusion…………………………………………………… 30

7. About…………………………………………………………. 30


2


Introduction

In an effort to simulate an end-user experience regarding performance of virtual machines across various cloud

computing providers, Cloud Spectator ran its iterative benchmark suite for 48 total hours on 2 different sets of

virtual machines (VMs) on each of the following providers: Amazon Web Services (AWS), Dynamic Services

for Infrastructure with vCloud (DSI vCloud), Microsoft Azure and ProfitBricks (PB).

Cloud Spectator measured performance on 2 vCPU, 4 vCPU, and 8 vCPU VMs for this study. For each VM

size, three types of VMs with different memory allocations were provisioned, each one tested for a 24-hour

period.

This study not only examined the performance of each vendor, but also tracked performance variability for each

of the 24-hour periods on 2vCPU, 4vCPU and 8vCPU machines. The methodology allowed Cloud Spectator to

capture performance variability over time on the same VM. Performance stability varied between resource types

and VM types/sizes throughout the course of the study.

Why Performance Matters

Performance and pricing are both key considerations in the public cloud industry, together having a substantial

impact on a company’s annual operating costs. Cloud users may need fewer resources on better-performing

services, which can lower costs. Since many users only consider price and not price-performance, they may pay

more because they require additional resources to achieve a desired level of performance. While some providers

try to differentiate offerings by cutting prices, others try to differentiate by focusing on improved performance

and user experience.

Differences in performance outputs of VMs

across IaaS providers can greatly affect quality of

service as well as annual operating costs. The

graph on the right illustrates an example of the

average processor performance from a sample of

six Cloud Service Providers (CSPs) as studied by

Cloud Spectator. CSP 1 has a processor

performance three times as high as CSP 6 (names

removed), which gives CSP 1 a notable

advantage in many processor-intensive

workloads. CSPs 2-5 exhibit a closer resemblance

in processor performance, but do not offer nearly

as much processing power as CSP 1 does.


3


The table below lists the 3 hardware components studied in this project, and each purpose as a function in the

server.

CPU & MEMORY PERFORMANCE

STORAGE PERFORMANCE

NETWORK PERFORMANCE

The performance of all applications is highly dependent on the CPU. The CPU is responsible for the processing and orchestration of all applications. The relationship between CPU performance and RAM is also observed by examining RAM bandwidth. While memory performance is not considered one of the key bottlenecks in performance for many applications, a subset of applications—particularly HPC and in-memory databases—is highly dependent on large sustained memory bandwidth.

Because most applications and all data reside on the disk, having fast disk performance is a key consideration for best application performance in many cases.

In a cloud environment, network performance is a critical piece. Scalability, in many cases, is dependent on the availability of additional VMs that must maintain a strong network backbone.


4


Executive Summary

T-Systems commissioned Cloud Spectator to measure and compare the performance of VMs on four different

cloud providers: Amazon Web Services, T-System’s DSI vCloud, Microsoft Azure and ProfitBricks. Raw VM

performance was examined to evaluate the component attributes of processor, memory, disk, and internal

network on each provider’s VMs. The purpose of the study was to understand, from an end-user perspective,

performance among cloud providers with similarly sized VMs.

Findings

vCPU Performance Findings

For this study, Cloud Spectator evaluated vCPU

performance by benchmarking the VMs using

Geekbench 3, a suite of benchmark tests that

simulate tasks such as encoding and image

processing. Testing occurred over the course of a

48-hour testing period.

vCPU Performance Key Findings: DSI vCloud exhibited the highest processor performance in the

study.

DSI vCloud provided the highest processor price-performance

value of the 4vCPU and 8vCPU VMs in the study; Microsoft had

the highest value for 2vCPU VMs.

Memory Performance Findings

Cloud Spectator evaluated memory bandwidth by

benchmarking the VMs using Geekbench 3’s

memory tests, which leverages STREAM to

produce requests to memory. Testing occurred over

the course of a 48-hour testing period.

Memory Performance Key Findings: DSI vCloud achieved the highest memory bandwidth in the study.

DSI vCloud provided the highest memory price-performance value

in the study.


5


Storage Performance Findings

Storage performance evaluation utilized the fio disk

benchmark, which tests the sequential read/write

and random read/write operations of storage. In this

study a 4 KB block size was used. Testing occurred

over a 48-hour test period. Persistent storage

(offered as “block storage” or “redundant storage”)

was used in all storage tests.

Storage Performance Key Findings: DSI vCloud VMs offered the highest median and maximum disk

performance.

Despite being one of the lower performers in disk IOPS, Microsoft

displayed the most stable disk performance throughout the study.

DSI vCloud provided the highest storage price-performance value

examined in the study.

Internal Network Performance Findings

Internal network performance is measured by the

throughput between VMs within the same data

center of the cloud provider (measured using Iperf2)

over the course of a 48-hour test period.

Internal Network Performance Key Findings: DSI vCloud and ProfitBricks VMs achieved the highest internal

network throughput of all the VMs examined in the study.

DSI vCloud and ProfitBricks provided the highest internal network

price-performance value of all the VMs examined in the study.

AWS exhibited stable internal network throughput with coefficient

of variation (CV) values under 1%.


6


Methodology

Cloud Spectator strives to create a transparent and detailed methodology to allow readers to understand the

testing process and recreate any studies. If any information is unclear or if you have any questions, please email

the team at [email protected] or call +1 (617) 300-0711.

Process

1. One iteration of a 24-hour test cycle ran on each of the two sets of VMs per type on each provider for a total of 48 hours of testing per VM

type.

2. Each VM was provisioned with a Linux Ubuntu 14.04 operating system by default, available from all providers. For AWS, the HVM image

was used.

3. Before the test period, and after provisioning the VMs, system updates and upgrades were conducted via apt-get.

4. The following dependencies were installed for testing:

a. Git. Git was used to clone the test repository on the VM.

b. MySQL. For automation purposes, mysql-server was installed to automate data uploads.

c. Pip. Used to download the appropriate libraries for Python in order to run the testing. SQL Alchemy was downloaded to interact

with MySQL and upload data.

d. Libmysqlclient-dev. MySQL database’s development files, which are necessary for the SQL Alchemy and MySQL interaction.

5. Each test cycled through in the following sequence: Geekbench 3 (process & memory), fio sequential operations, fio random operations,

Iperf internal network throughput (for more information on testing, see Tests Used).

a. For fio testing (to measure disk IOPS), sequential operations ran first. Files from the sequential tests were deleted, and fio

recreated files before running random operations. Once random operations completed, the files were also deleted. Thus, before

each disk IOPS test, the files associated with the tests were deleted and recreated.

6. Internal network testing was conducted by creating a clone of the existing VM within the same region/availability zone in a client and server

architecture, connected through a private network. Iperf was installed on both the client and server. The server used “Iperf –s” to listen on

port 5001 (default port) for a TCP connection made by the client. Iperf ran bi-directionally for 60 seconds.

7. At the end of each test iteration, results were uploaded into Cloud Spectator’s database through use of SQLAlchemy (Python—see 4c in

Process).

Price-Performance Comparisons

Cloud Spectator’s price-performance calculation, the CloudSpecs Score, provides information on how much

performance is realized for each unit of cost. The CloudSpecs Score is an indexed, comparable score ranging

from 0-100 indicative of value based on a combination of cost and performance. The value is scaled; e.g., a

Cloud Service Provider (CSP) with a score of 100 gives 4x the value of a CSP with a score of 25. The

CloudSpecs Scores in this report can only be compared with equivalent configurations; e.g., a 1 vCPU VM on

Provider A can only be compared to a 1 vCPU VM on Provider B.

The calculation of the CloudSpecs Score: 1. provider_value = [Provider Performance Score] / [Provider Cost]

2. best_provider_value = max{provider_values}

3. CSP’s CloudSpecs Score = 100*provider_value / best_provider_value

mailto:[email protected]


7


Tests Used

Processor & Memory Bandwidth: Geekbench 3

Geekbench 3, a licensable product created by Primate Labs, is a cross-platform processor benchmark that can

measure single-core and multi-core performance by simulating real-world workloads. The Geekbench 3 test

suite is comprised of 27 individual tasks/workloads: 13 integer workloads, 10 floating point workloads, and 4

memory-bandwidth tasks. While processor and memory bandwidth are both performance factors that contribute

to the final score provided by Geekbench 3, the test suite weighs processing performance much more heavily

than memory bandwidth. Also, memory bandwidth is not necessarily affected by the amount of memory

available for the VM, so VMs with larger amounts of memory may not exhibit larger bandwidth. For more

information on Geekbench 3 and to see its individual workloads, please see

http://www.primatelabs.com/geekbench/.

Geekbench 3 Tasks (Figure 3.1)

TEST TOOL TASK

Separate CPU tests that are all aggregated

into a final score. Subtests include: Integer Math, Floating Point

Math

DESCRIPTION

Integer Geekbench 3

AES, Twofish, SHA1, SHA2, BZip2 Compression,

BZip2 Decompression, JPEG Compression, JPEG

Decompression, PNG Compression, PNG

Decompression, Sobel, Lua, Dijkstra

Integer and Floating Point tasks together represent

vCPU performance. The performance of all

applications is highly dependent on the vCPU

since the vCPU is responsible for the processing

and orchestration of all applications. Floating

Point Geekbench 3

Black Scholes, Mandelbrot, Sharpen Filter, Blur Filter,

SGEMM, DGEMM, SFFT, DFFT, N-Body, Ray Trace

Memory Geekbench 3 STREAM Copy, STREAM Scale, STREAM Add,

STREAM Triad

While memory performance is not considered one

of the key bottlenecks in performance for many

common applications, a subset of applications—

particularly HPC and in-memory databases—is

highly dependent on large sustained memory

bandwidth.

Sequential and Random Disk IOPS: fio

Fio is an open source I/O generator that spawns a number of threads and processes to conduct a particular type

of I/O action specified. For the purpose of this study, fio was used to measure disk IOPS by tracking direct I/O

to the VM’s network storage. 5 x 200 MB files were created for sequential operations testing, and 5 x 200 MB

files were created for random operations testing. All operations were 50% read and 50% write. Each test

iteration used a 4 KB block size. Each test iteration lasted 60 seconds.

Internal Network Throughput: Iperf

Iperf is an open source tool used to measure TCP or UDP network bandwidth performance. Cloud Spectator

used Iperf to measure the network throughput between VMs residing in the same region/availability zone. A

screen session was created for Iperf as a server machine on each appropriate VM (see Process 6). Each Iperf test

iteration lasted 60 seconds, and data was transferred in one direction, from the test VM to the Iperf server VM.

http://www.primatelabs.com/geekbench/


8


VM Configurations

VM configurations were matched to standardize by virtual processors. 2 vCPUs, 4 vCPUs, and 8 vCPUs

machines from each provider were used in this study that most closely matched in size. While some providers

may offer the option for local storage, none was used and all disk testing was conducted on persistent SAN

storage. The storage columns in Figures 3.3 A through C are reflective of the SAN storage provisioned for each

VM.

2 vCPU Virtual Machines (Figure 3.3 A)

Provider Instance vCPU RAM (GB) Storage (GB)

Amazon c4.large 2 3.75 100 (EBS SSD)

Amazon m4.large 2 8 100 (EBS SSD)

Amazon r3.large 2 15 100 (EBS SSD)

Microsoft A2 Standard 2 3.5 100 (Page Blob)

Microsoft D2 v2 2 7 100 (Page Blob)


DSI vCloud 2c4r 2 4 100 (default)



ProfitBricks 2c4r 2 4 100 (HDD)



4 vCPU Virtual Machines (Figure 3.3 B)


Amazon c4.xlarge 4 7.5 500 (EBS SSD)

Amazon m4.xlarge 4 16 500 (EBS SSD)

Amazon r3.xlarge 4 30.5 500 (EBS SSD)

Microsoft A3 Standard 4 7 500 (Page Blob)










9


0

500

1000

1500

2000

2500

3000

3500

4000

Alpha Bravo Charlie Delta Echo

Mp

ixel

s/se

c

Sample Performance Graph

8 vCPU Virtual Machines (Figure 3.3 C)


Amazon c4.2xlarge 8 15 1000 (EBS SSD)

Amazon m4.2xlarge 8 32 1000 (EBS SSD)

Amazon r3.2xlarge 8 61 1000 (EBS SSD)

Microsoft A4 Standard 8 14 1000 (Page Blob)









Data Center Locations (Figure 3.4)

Provider Amazon Microsoft DSI vCloud ProfitBricks

Region EU (Frankfurt) West Europe Munich Germany

Understanding Performance Results

A total of three sets of VMs were compared in this study: 2 vCPU, 4 vCPU, and 8 vCPU VMs. Each set was

compared by examining the minimum, 5TH percentile, median, 95TH percentile, and maximum scores achieved

in the total 48-hour period. The information is illustrated through box-and-whisker plots and value tables

designed to visualize the performance and performance variation captured over the course of the study. An

example of a performance percentile graph is displayed below:

Legend

Maximum: highest score achieved on this VM over the duration of

the testing.

95TH Percentile (High-Score Category): 95% of all scores on this

VM achieved this score or lower.

Median (Median-Score Category): The number separating the

higher half of the scores of that VM from the lower half. If the

median is closer to the 95th percentile, then more high scores were

observed than low scores; vice versa.

5TH Percentile (Low-Score Category): 5% of all scores on this

provider achieved this score or lower.

Minimum: lowest score achieved on this VM over the duration of

the testing.


10


Key Considerations

Testing was conducted on specific VM types for each provider. Different VM configurations may yield different comparative results

between the providers. AWS and Microsoft offered fixed VM configurations. DSI vCloud and ProfitBricks offered independently

customizable VM configurations.

Users may experience different performance across different physical hosts. Factors such as user contention or malfunctions of the

physical hardware can cause suboptimal performance.

VMs selected were the base offerings across providers; greater performance may be obtained on certain providers by paying for additional

features/services.

Due to lack of availability, HDD was selected on ProfitBricks as SDD was not available within the selected data center.

Time of server setup and provisioning may vary across providers; this report only takes into consideration the performance of created

machines. Users seeking quick scalability of resources should evaluate the usability and provisioning time of multiple providers.


11


Performance Findings

Processor Performance – Integer Operations

Below are results of integer-based processor testing on all three providers. The processor tests are categorized

into two categories: integer and floating point tasks. The tables on the right specify the scores achieved by each

provider’s VMs.

--Units Measured in Geekbench Performance Score--


0

1000

2000

3000

4000

5000

6000

7000

8000

m4

.la

rge

c4.la

rge

r3.la

rge

A2

D2v2

D11v2

DS

I_2c4r

DS

I_2c8r

DS

I_2c16r

PB

_2c4

r

PB

_2c8

r

PB

_2c1

6r

AWS Azure DSI vCloud PB

PE

RF

OR

MA

NC

E S

CO

RE

Figure 4.1A: CPU (INTEGER) PERFORMANCE (2vCPU VMs)

0

2000

4000

6000

8000

10000

12000

14000

m4

.xla

rge

c4.x

larg

e

r3.x

larg

e

A3

D3v2

D12v2

DS

I_4c8r

DS

I_4c16r

DS

I_4c32r

PB

_4c8

r

PB

_4c1

6r

PB

_4c3

2r


PE

RF

OR

MA

NC

E S

CO

RE

Figure 4.1B: CPU (INTEGER) PERFORMANCE (4vCPU VMs)

Provider VM Min 5TH Median 95TH Max

2 v

CP

Us

AWS

m4.large 3463 3474 3506 3527 3530

c4.large 4055 4074 4122 4154 4164

r3.large 3357 3375 3389 3411 3418

A2 1839 2351 2458 2505 2552

Microsoft D2v2 5557 5738 5941 6090 6170

D11v2 5438 5567 5869 6271 6310

DSI vCloud

2c4r 2059 2311 6430 6598 6631

2c8r 1735 2091 6241 6555 6614

2c16r 2961 3606 5867 6646 6701

PB

2c4r 3434 3600 3881 4080 4208

2c8r 3356 3602 3834 4026 4220

2c16r 2926 3736 3791 3818 3823


4 v

CP

Us

AWS

m4.xlarge 6650 6758 6915 6954 6969

c4.xlarge 8033 8068 8122 8165 8209

r3.xlarge 6632 6670 6702 6734 6746

A3 4164 4607 4756 4858 4910

Microsoft D3v2 9820 10016 10457 10942 11010

D12v2 9894 10098 10328 10647 10797

DSI

vCloud

4c8r 10166 11791 12752 13099 13146

4c16r 9505 12207 12932 13072 13125

4c32r 9269 11795 12941 13057 13102

PB

4c8r 6043 6435 6757 7016 7124

4c16r 6235 6450 6746 6985 7053

4c32r 6289 6425 6745 7025 7215


12



<This Space Intentionally Left Blank>

0

5000

10000

15000

20000

25000

30000m

4.2

xla

rge

c4.2

xla

rge

r3.2

xla

rge

A4

D4v2

D13v2

DS

I_8c16r

DS

I_8c32r

DS

I_8c64r

PB

_8c1

6r

PB

_8c3

2r

PB

_8c6

4r


PE

RF

OR

MA

NC

E S

CO

RE

Figure 4.1C: CPU (INTEGER) PERFORMANCE (8vCPU VMs)


8 v

CP

Us

AWS

m4.2xlarge 13531 13610 13658 13706 13741

c4.2xlarge 15321 15951 16026 16075 16130

r3.2xlarge 12294 13129 13250 13290 13304

A4 8412 8494 8687 8837 8976

Microsoft D4v2 16805 17524 18465 19082 19382

D13v2 11328 11530 14964 19574 19743

DSI

vCloud

8c16r 15373 21223 24553 24810 24888

8c32r 12102 22311 24299 24517 24636

8c64r 7327 21071 24099 24498 24666

8c16r 11250 11539 11942 12262 12499

PB 8c32r 11200 11540 11865 12191 12464

8c64r 11170 11489 11916 12262 12514


13


Processor Performance – Floating Point Operations

Below are results of floating-point-based processor testing on all three providers. The processor tests are

categorized into two categories: integer and floating point tasks. The tables on the right specify the scores

achieved by each provider’s VMs.



0

1000

2000

3000

4000

5000

6000

7000

m4

.la

rge

c4.la

rge

r3.la

rge

A2

D2v2

D11v2

DS

I_2c4r

DS

I_2c8r

DS

I_2c16r

PB

_2c4

r

PB

_2c8

r

PB

_2c1

6r


PE

RF

OR

MA

NC

E S

CO

RE

Figure 4.2A: CPU (FLOATING POINT) PERFORMANCE

(2vCPU VMs)

0

2000

4000

6000

8000

10000

12000

14000

m4

.xla

rge

c4.x

larg

e

r3.x

larg

e

A3

D3v2

D12v2

DS

I_4c8r

DS

I_4c16r

DS

I_4c32r

PB

_4c8

r

PB

_4c1

6r

PB

_4c3

2r


PE

RF

OR

MA

NC

E S

CO

RE

Figure 4.2B: CPU (FLOATING POINT) PERFORMANCE

(4vCPU VMs)


2 v

CP

Us

AWS

m4.large 3371 3380 3404 3428 3485

c4.large 3979 3989 4010 4128 4136

r3.large 3344 3354 3367 3427 3446

A2 2268 2282 2378 2420 2458

Microsoft D2v2 5468 5545 5762 5883 5952

D11v2 5300 5360 5647 6095 6238

DSI

vCloud

2c4r 1650 1890 6248 6383 6502

2c8r 1449 1694 6148 6357 6493

2c16r 2732 3244 5842 6367 6510

2c4r 2937 3046 3229 3419 3564

PB 2c8r 2943 3040 3178 3356 3452

2c16r 2378 2996 3029 3079 3102


4 v

CP

Us

AWS

m4.xlarge 6613 6797 6877 6943 6953

c4.xlarge 8038 8078 8137 8212 8240

r3.xlarge 6795 6830 6857 6881 6916

A3 4615 4687 4825 4910 4956

Microsoft D3v2 10303 10402 10923 11303 11461

D12v2 10150 10411 10689 10952 11223

DSI vCloud

4c8r 7384 10798 12722 12899 12997

4c16r 9872 12131 12792 12919 12998

4c32r 9180 12148 12773 12896 13009

4c8r 5207 5559 5736 5922 6036

PB 4c16r 5554 5681 5828 5963 6060

4c32r 5556 5698 5848 6033 6125


14



Observations: Processor Performance

The scores in Figures 4.1 A – C and 4.2 A – C are indexed through a combination of floating point and integer

performance on processors. Performance variability was determined by measuring each VM’s coefficient of

variation (CV), which is a percentage expressing the relationship between the average and standard deviation.

The variability for most VMs were relatively low with values of 5% or lower. Microsoft’s 8vCPU D13v2 had

elevated CV values ranging up to 26%, and DSI’s 2vCPU VMs ranged up 40%.

Additional Observations DSI vCloud achieved the highest performance across the 2vCPU, 4vCPU and 8vCPU processor tests in the study, followed by Microsoft,

then AWS and ProfitBricks.

The difference between Amazon’s compute-oriented offering and its general offering (c4 vs. m4, respectively) was relatively small at ~17%.

The difference between Microsoft’s stronger D-series v2 and A-series Standard was much greater at 125%.

DSI vCloud and ProfitBricks VMs do not offer different VM types, thus there is negligible difference between the VMs on each provider.

0

5000

10000

15000

20000

25000

30000

m4

.2xla

rge

c4.2

xla

rge

r3.2

xla

rge

A4

D4v2

D13v2

DS

I_8c16r

DS

I_8c32r

DS

I_8c64r

PB

_8c1

6r

PB

_8c3

2r

PB

_8c6

4r


PE

RF

OR

MA

NC

E S

CO

RE

Figure 4.2C: CPU (FLOATING POINT) PERFORMANCE

(8vCPU VMs)


8 v

CP

Us

AWS

m4.2xlarge 13434 13482 13552 13696 13798

c4.2xlarge 13246 15918 16008 16127 16212

r3.2xlarge 13351 13479 13547 13635 13788

A4 8834 8932 9116 9292 9370

Microsoft D4v2 18531 19209 20485 21225 21561

D13v2 12152 12443 16836 21878 22015

DSI

vCloud

8c16r 13282 19514 25190 25402 25538

8c32r 10558 22094 24849 25175 25319

8c64r 9255 20319 24439 25142 25336

8c16r 9344 10052 10386 10685 10911

PB 8c32r 9751 10038 10336 10662 10892

8c64r 9799 10067 10413 10767 11124


15


Memory Bandwidth

Below are the results of the memory bandwidth testing on all three providers. Because memory bandwidth is

generally not affected by the amount of provisioned RAM, VMs with larger amounts of RAM do not

necessarily have higher scores. The tables on the right specify the scores achieved by each provider’s VMs.



0

1000

2000

3000

4000

5000

6000

m4

.la

rge

c4.la

rge

r3.la

rge

A2

D2v2

D11v2

DS

I_2c4r

DS

I_2c8r

DS

I_2c16r

PB

_2c4

r

PB

_2c8

r

PB

_2c1

6r


PE

RF

OR

MA

NC

E S

CO

RE

Figure 4.3A: MEMORY PERFORMANCE (2vCPU VMs)

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

m4

.xla

rge

c4.x

larg

e

r3.x

larg

e

A3

D3v2

D12v2

DS

I_4c8r

DS

I_4c16r

DS

I_4c32r

PB

_4c8

r

PB

_4c1

6r

PB

_4c3

2r


PE

RF

OR

MA

NC

E S

CO

RE

Figure 4.3B: MEMORY PERFORMANCE (4vCPU VMs)


2 v

CP

Us

AWS

m4.large 3162 3165 3198 3247 3251

c4.large 3407 3413 3439 3451 3456

r3.large 2225 2238 2259 2277 2283

A2 1713 1786 1965 2070 2107

Microsoft D2v2 2904 2941 3008 3258 3900

D11v2 2826 2891 2967 3060 3089

DSI vCloud

2c4r 1393 1700 4826 5442 5590

2c8r 990 1170 4285 5191 5280

2c16r 1881 2295 4249 5549 5602

2c4r 1511 2005 2249 2347 2400

PB 2c8r 1799 1898 2063 2100 2132

2c16r 1336 1453 1739 1785 1840


4 v

CP

Us

AWS

m4.xlarge 4594 5476 5555 5620 5636

c4.xlarge 5679 5798 5941 6029 6081

r3.xlarge 3765 4336 4393 4430 4448

A3 1853 1915 2011 2118 2165

Microsoft D3v2 2902 2974 3065 3401 3987

D12v2 2840 2854 3232 3380 3450

DSI

vCloud

4c8r 3375 5009 6728 7991 8054

4c16r 4637 6366 7474 7946 8019

4c32r 4415 6102 7537 7911 8029

4c8r 1867 2040 2427 2550 2598

PB 4c16r 1953 2024 2107 2169 2216

4c32r 2126 2133 2173 2239 2266


16



Observations: Memory Bandwidth Performance

The scores in Figures 4.3 A – C are indexed STREAM test results on the memory bandwidth of the RAM.

DSI’s performance varied the most across the VMs averaging 23%, followed by Microsoft at 6%, ProfitBricks

at 3%, and AWS at 1%.

Additional Observations AWS’s memory bandwidth scales up from 2vCPU to 4vCPU for both VMs, but appears to cap at ~6000 (Geekbench Score). The compute

instance type (c4) does not reach the cap until the 8vCPU VM.

DSI vCloud’s memory bandwidth scales from an average median of 4453 to 7246 from 2vCPU to 4vCPU, but only scales to 7736 on the

8vCPU VMs.

Microsoft and PB memory bandwidth does not scale as the core count increases.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

m4

.2xla

rge

c4.2

xla

rge

r3.2

xla

rge

A4

D4v2

D13v2

DS

I_8c16r

DS

I_8c32r

DS

I_8c64r

PB

_8c1

6r

PB

_8c3

2r

PB

_8c6

4r


PE

RF

OR

MA

NC

E S

CO

RE

Figure 4.3C: MEMORY PERFORMANCE (8vCPU VMs)


8 v

CP

Us

AWS

m4.2xlarge 5436 5474 5547 5598 5636

c4.2xlarge 4630 5874 6028 6070 6096

r3.2xlarge 4124 4222 4390 4415 4429

A4 1749 1787 1988 2126 2178

Microsoft D4v2 2857 2897 3021 3097 3148

D13v2 2170 2199 2651 3185 3210

DSI

vCloud

8c16r 3621 5042 8488 8599 8646

8c32r 1895 3474 8140 8475 8525

8c64r 1527 2681 6580 8311 8437

8c16r 1966 1973 1988 2015 2031

PB 8c32r 1969 1975 1992 2018 2035

8c64r 1975 1988 2006 2032 2049


17


SAN Disk IOPS: Sequential Operations

Below are results of disk IOPS testing on all providers, specifically for sequential operations (50% read, 50%

write). Disk IOPS was tested with direct I/O, so results are not reflective of cached performance, which may

sustain higher IOPS on each provider. The tables on the right specify the scores achieved by each provider’s

VMs.

--Units Measured in IOPS--


0

2000

4000

6000

8000

10000

12000

m4

.la

rge

c4.la

rge

r3.la

rge

A2

D2v2

D11v2

DS

I_2c4r

DS

I_2c8r

DS

I_2c16r

PB

_2c4

r

PB

_2c8

r

PB

_2c1

6r


IOP

S

Figure 4.4A: STORAGE PERFORMANCE - SEQUENTIAL R/W

(100GB)

0

2000

4000

6000

8000

10000

12000

m4

.xla

rge

c4.x

larg

e

r3.x

larg

e

A3

D3v2

D12v2

DS

I_4c8r

DS

I_4c16r

DS

I_4c32r

PB

_4c8

r

PB

_4c1

6r

PB

_4c3

2r


IOP

S

Figure 4.4B: STORAGE PERFORMANCE - SEQUENTIAL R/W

(500GB)


2 v

CP

Us

AWS

m4.large 299 299 300 3003 3004

c4.large 299 299 300 3003 3004

r3.large 299 299 300 3004 3004

A2 496 502 503 504 505

Microsoft D2v2 220 501 503 504 504

D11v2 446 502 503 504 504

DSI

vCloud

2c4r 2105 3288 6067 9142 11056

2c8r 1838 3328 6025 8470 10724

2c16r 2757 4130 6304 8106 9930

2c4r 1500 2134 3411 3798 3850

PB 2c8r 1702 2083 3524 3809 3855

2c16r 1843 2111 3547 3834 3858


4 v

CP

Us

AWS

m4.xlarge 1814 1832 1874 2985 2986

c4.xlarge 1491 1493 1510 2984 2986

r3.xlarge 1627 2242 2286 2986 2986

A3 490 500 503 504 504

Microsoft D3v2 496 502 503 504 504

D12v2 405 502 503 504 504

DSI

vCloud

4c8r 1823 2705 4144 6208 8122

4c16r 4093 5492 7072 8393 10441

4c32r 2064 3068 6008 8276 8788

4c8r 1784 2414 3527 3873 3899

PB 4c16r 1548 2125 3455 3858 3895

4c32r 1888 2459 3465 3848 3884


18



<This Space Intentionally Left Blank>

0

2000

4000

6000

8000

10000

12000

14000

m4

.2xla

rge

c4.2

xla

rge

r3.2

xla

rge

A4

D4v2

D13v2

DS

I_8c16r

DS

I_8c32r

DS

I_8c64r

PB

_8c1

6r

PB

_8c3

2r

PB

_8c6

4r


IOP

S

Figure 4.4C: STORAGE PERFORMANCE - SEQUENTIAL R/W

(1000GB)


8 v

CP

Us

AWS

m4.2xlarge 2962 2976 2984 2986 2987

c4.2xlarge 2954 2977 2983 2986 2987

r3.2xlarge 2820 2976 2983 2986 2987

A4 498 502 503 504 504

Microsoft D4v2 472 501 503 504 504

D13v2 218 500 503 504 504

DSI

vCloud

8c16r 4370 5615 7593 10283 11695

8c32r 2677 4450 7615 9827 10765

8c64r 2005 5088 7546 9897 10474

8c16r 1517 2279 3405 3863 3895

PB 8c32r 1663 2490 3592 3870 3906

8c64r 2923 3598 3829 3885 3917


19


SAN Disk IOPS: Random Operations

Below are results of disk IOPS testing on all providers, specifically for random operations (50% read, 50%

write). Disk IOPS was tested with direct I/O, so results are not reflective of cached performance, which may

sustain higher IOPS on each provider. The tables on the right specify the scores achieved by each provider’s

VMs.



0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

m4

.la

rge

c4.la

rge

r3.la

rge

A2

D2v2

D11v2

DS

I_2c4r

DS

I_2c8r

DS

I_2c16r

PB

_2c4

r

PB

_2c8

r

PB

_2c1

6r


IOP

S

Figure 4.5A: STORAGE PERFORMANCE - RANDOM R/W

(100GB)

0

2000

4000

6000

8000

10000

12000

14000

m4

.xla

rge

c4.x

larg

e

r3.x

larg

e

A3

D3v2

D12v2

DS

I_4c8r

DS

I_4c16r

DS

I_4c32r

PB

_4c8

r

PB

_4c1

6r

PB

_4c3

2r


IOP

S

Figure 4.5B: STORAGE PERFORMANCE - RANDOM R/W

(500GB)


2 v

CP

Us

AWS

m4.large 299 299 300 3003 3004

c4.large 299 300 300 3004 3004

r3.large 299 299 300 3003 3004

A2 382 498 503 504 504

Microsoft D2v2 408 498 502 503 503

D11v2 390 497 502 503 503

DSI

vCloud

2c4r 363 1961 4184 7791 9469

2c8r 474 1453 4206 7280 9194

2c16r 407 1760 3968 6360 8620

2c4r 230 293 660 1363 1774

PB 2c8r 243 301 664 1383 1631

2c16r 240 302 697 1235 1384


4 v

CP

Us

AWS

m4.xlarge 2968 2979 2986 2987 2988

c4.xlarge 2980 2985 2986 2988 2988

r3.xlarge 2374 2985 2986 2988 2989

A3 482 499 502 503 503

Microsoft D3v2 484 499 502 503 503

D12v2 386 490 502 503 503

DSI

vCloud

4c8r 305 570 1203 2623 3837

4c16r 609 1182 2690 7234 11860

4c32r 349 650 1992 4593 6225

4c8r 214 289 607 1263 1618

PB 4c16r 179 272 520 1007 1150

4c32r 196 254 533 842 984


20



Observations: Sequential and Random Disk IOPS Performance

Providers offer unique approaches to SAN disk, from hardware components (SSD vs. traditional magnetic

drives) to performance throttling. Even the similarities can be quite different; while Amazon and DSI both

utilize SSD, the performance difference between the two providers is highly apparent (see Figures 4.4-4.5).

Additional Observations DSI vCloud achieved the highest maximum IOPS observed throughout the study.

DSI vCloud’s storage performance variability is moderately high across the three volume sizes with an average CV of 23% for sequential

read/write and 46% for random read/write operations.

AWS’s variability drops as the storage volume increases; exhibiting a CV of 142% on the 2vCPU/100GB VMs and dropping down to an

average 24% on the 4vCPU/500GB VMs and under 1% on the 8vCPU/1000GB VMs (which is due to AWS’s burst functionality) when

looking at the sequential read/write results.

Microsoft’s storage performance was the most stable with the highest CV ranging up to 6%.

ProfitBricks’ variability was elevated for sequential read/write at 15%, but high on random read/write at 40%.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

m4

.2xla

rge

c4.2

xla

rge

r3.2

xla

rge

A4

D4v2

D13v2

DS

I_8c16r

DS

I_8c32r

DS

I_8c64r

PB

_8c1

6r

PB

_8c3

2r

PB

_8c6

4r


IOP

S

Figure 4.5C: STORAGE PERFORMANCE - RANDOM R/W

(1000GB)


8 v

CP

Us

AWS

m4.2xlarge 2974 2984 2986 2987 2988

c4.2xlarge 2972 2985 2986 2988 2988

r3.2xlarge 2537 2984 2986 2987 2988

A4 421 499 502 503 503

Microsoft D4v2 277 496 502 503 503

D13v2 410 497 502 503 503

DSI

vCloud

8c16r 567 983 2459 6587 8069

8c32r 1268 1844 4330 7976 9246

8c64r 479 1737 4062 7492 8896

8c16r 225 313 613 970 1406

PB 8c32r 231 278 495 972 1417

8c64r 717 787 1040 1629 1695


21


Internal Network Throughput

Below are the results of internal network testing on all providers. Internal network was examined by running

TCP connections between two servers within the same data center/region/zone. The tables on the right specify

the scores achieved by each provider’s VMs. T

--Units Measured in Mbit/s--


0

1000

2000

3000

4000

5000

6000

7000

8000

m4

.la

rge

c4.la

rge

r3.la

rge

A2

D2v2

D11v2

DS

I_2c4r

DS

I_2c8r

DS

I_2c16r

PB

_2c4

r

PB

_2c8

r

PB

_2c1

6r


MB

IT/S

Figure 4.6: INTERNAL NETWORK PERFORMANCE

(2vCPU VMs)

0

1000

2000

3000

4000

5000

6000

7000

8000

m4

.xla

rge

c4.x

larg

e

r3.x

larg

e

A3

D3v2

D12v2

DS

I_4c8r

DS

I_4c16r

DS

I_4c32r

PB

_4c8

r

PB

_4c1

6r

PB

_4c3

2r


MB

IT/S

Figure 4.6B: INTERNAL NETWORK PERFORMANCE

(4vCPU VMs)


2 v

CP

Us

AWS

m4.large 445 445 445 445 445

c4.large 610 610 610 610 610

r3.large 492 492 492 492 492

A2 42 435 476 477 477

Microsoft D2v2 735 831 1275 1429 1430

D11v2 531 939 1282 1430 1431

DSI vCloud

2c4r 46 51 1808 4430 5381

2c8r 50 57 1603 3975 5008

2c16r 1981 2510 4033 6900 7132

2c4r 4271 4792 5346 5663 5891

PB 2c8r 4792 5226 6015 6355 6675

2c16r 4777 5170 5884 6138 6327


4 v

CP

Us

AWS

m4.xlarge 753 754 754 754 754

c4.xlarge 1227 1227 1227 1227 1228

r3.xlarge 706 706 707 707 707

A3 670 761 855 953 953

Microsoft D3v2 643 725 1795 2262 2621

D12v2 686 988 1501 2257 2568

DSI

vCloud

4c8r 67 779 3985 7345 7530

4c16r 41 49 2627 4305 5254

4c32r 48 56 3038 4432 5530

4c8r 3437 3780 4464 5107 5307

PB 4c16r 4865 5429 6041 6395 6669

4c32r 4862 5331 5817 6094 6263


22



Observations: Internal Network Throughput Performance

With regards to internal network throughput, DSI exhibited high performance variability with CVs averaging

56% across all three VM sizes. Microsoft VMs’ performance variability was relatively lower with CVs

averaging 23% across all the VMs examined, followed by ProfitBrick averaging 7%. AWS’s throughput was

very stable, exhibiting CVs that were under 1%.

Additional Observations DSI vCloud VMs recorded the highest maximum values for internal network across all three VM sizes.

AWS’s VMs deliver a specific network throughput depending on the size and family of VM. As the size of the VMs increase, the network

throughput increases as well. The throughput starts at 445-610Mbit/s for the 2vCPU VMs and rises to 707-1227Mbit/s for the 4vCPU VMs

and capping at 962-2382Mbit/s for the 8vCPU VMs.

Microsoft’s VMs varied by instance type as well, with the A-series limited between 400-1,600Mbit/s, while the D-series fluctuated between

500-2600Mbit/s.

ProfitBricks exhibited the highest median values for 2vCPU and 4vCPU VMs. The median performance values primarily ranged between 4-

6Gbps.

0

1000

2000

3000

4000

5000

6000

7000

8000

m4

.2xla

rge

c4.2

xla

rge

r3.2

xla

rge

A4

D4v2

D13v2

DS

I_8c16r

DS

I_8c32r

DS

I_8c64r

PB

_8c1

6r

PB

_8c3

2r

PB

_8c6

4r


MB

IT/S

Figure 4.6C: INTERNAL NETWORK PERFORMANCE

(8vCPU VMs)


8 v

CP

Us

AWS

m4.2xlarge 962 962 962 962 962

c4.2xlarge 2378 2382 2382 2382 2382

r3.2xlarge 962 962 962 962 962

A4 697 776 1061 1433 1594

Microsoft D4v2 751 858 1355 2017 2573

D13v2 296 351 1253 1878 2634

DSI

vCloud

8c16r 4070 4278 6145 7174 7334

8c32r 3875 4279 6314 6511 6786

8c64r 53 69 1754 5865 6630

8c16r 2281 3473 4191 4901 5236

PB 8c32r 3503 4155 5026 5590 5625

8c64r 2191 3050 4103 4702 5326


23


Price-Performance Analysis Results

The following section provides the price-performance comparisons of the VMs tested in the performance

analysis section above. The values used in the comparison are taken from the raw performance scores shown in

the preceding section and the pricing for the VMs being examined (see Methodology). Scores are out of 100,

with the highest value VM in each test scoring 100, while the remaining VM scores are measured against the

price-performance value of highest scoring VM.

Provider Performance Analysis: Provider Pricing

Below are the prices per tested virtual machine across all providers and all virtual machine sizes. Storage prices

are broken out from the virtual machine price as persistent storage was tested for every machine size.

2vCPU/100GB VM & Storage Pricing

Amazon Microsoft DSI vCloud ProfitBricks

VM Type c4.large m4.large r3.large A2 Std D2v2 D11v2 2c4r 2c8r 2c16r 2c4r 2c8r 2c16r

VM € € 86.96 € 92.80 € 129.79 € 73.88 € 98.48 € 131.11 € € € € 71.54 € 84.68 € 110.96

Storage € € 10.58 € 10.58 € 10.58 € 4.22 € 4.22 € 4.22 € € € € 4.00 € 4.00 € 4.00

Total € 97.54 € 103.38 € 140.37 € 78.10 € 102.70 € 135.33 € 116.76 € 114.89 € 128.22 € 75.54 € 88.68 € 114.96



VM Type c4.xlarge m4.xlarge r3.xlarge A3 Std D3v2 D12v2 4c8r 4c16r 4c32r 4c8r 4c16r 4c32r

VM € € 173.27 € 184.96 € 259.59 € 147.75 € 196.37 € 262.87 € € € € 143.08 € 169.36 € 221.92

Storage € € 52.90 € 52.90 € 52.90 € 21.10 € 21.10 € 21.10 € € € € 20.00 € 20.00 € 20.00

Total € 226.17 € 237.85 € 312.48 € 168.85 € 217.47 € 283.97 € 160.27 € 177.69 € 181.52 € 163.08 € 189.36 € 241.92



VM Type c4.2xlarge m4.2xlarge r3.2xlarge A4 Std D4v2 D13v2 8c16r 8c32r 8c64r 8c16r 8c32r 8c64r

VM € € 346.55 € 369.91 € 519.18 € 295.50 € 392.74 € 524.51 € € € € 286.16 € 338.72 € 443.84

Storage € € 105.79 € 105.79 € 105.79 € 42.20 € 42.20 € 42.20 € € € € 40.00 € 40.00 € 40.00

Total € 452.34 € 475.70 € 624.97 € 337.70 € 434.94 € 566.71 € 193.83 € 342.15 € 319.77 € 326.16 € 378.72 € 483.84

*prices based on 730 hours per month*

Pricing Considerations: Regional pricing based off the data center locations listed in the Methodology section.

DSI vCloud

pricing is based on the resource pool utilization, not the individual VMs or storage volumes

Pricing is accurate as of September 16, 2016

USD to EUR conversion rate of .889 was used for AWS


24



25


vCPU VALUE ANALYSIS RESULTS: INTEGER

vCPU Value – Integer Tasks (Multicore): 2vCPU

Provider VM CloudSpecs

Score

Amazon m4.large 59

Amazon c4.large 73

Amazon r3.large 42

Microsoft A2 54

Microsoft D2v2 100

Microsoft D11v2 75

DSI vCloud 2c4r 95

DSI vCloud 2c8r 94

DSI vCloud 2c16r 79

PB 2c4r 89

PB 2c8r 75

PB 2c16r 57



Score

Amazon m4.xlarge 37

Amazon c4.xlarge 45

Amazon r3.xlarge 27

Microsoft A3 35

Microsoft D3v2 60

Microsoft D12v2 46

DSI vCloud 4c8r 100

DSI vCloud 4c16r 91

DSI vCloud 4c32r 90

PB 4c8r 52

PB 4c16r 45

PB 4c32r 35



Score

Amazon m4.2xlarge 23

Amazon c4.2xlarge 28

Amazon r3.2xlarge 17

Microsoft A4 20

Microsoft D4v2 34

Microsoft D13v2 21

DSI vCloud 8c16r 100

DSI vCloud 8c32r 56

DSI vCloud 8c64r 59

PB 8c16r 29

PB 8c32r 25

PB 8c64r 19

59

73

42

54

100

75

95 94

79

89

75

57

0

10

20

30

40

50

60

70

80

90

100

m4.la

rge

c4

.larg

e

r3.la

rge

A2

D2

v2

D1

1v2

DS

I_2c4

r

DS

I_2c8

r

DS

I_2c1

6r

PB

_2

c4r

PB

_2

c8r

PB

_2

c16

r


Clo

ud

Sp

ecs S

co

re

Price-Performance of vCPU (Integer) - 2vCPU VMs

3745

2735

60

46

10091 90

5245

35

0

10

20

30

40

50

60

70

80

90

100

m4.x

larg

e

c4

.xla

rge

r3.x

larg

e

A3

D3

v2

D1

2v2

DS

I_4c8

r

DS

I_4c1

6r

DS

I_4c3

2r

PB

_4

c8r

PB

_4

c16

r

PB

_4

c32

r


Clo

ud

Sp

ecs S

co

re


2328

1720

34

21

100

56 59

2925

19

0

10

20

30

40

50

60

70

80

90

100

m4.2

xla

rge

c4

.2xla

rge

r3.2

xla

rge

A4

D4

v2

D1

3v2

DS

I_8c1

6r

DS

I_8c3

2r

DS

I_8c6

4r

PB

_8

c16

r

PB

_8

c32

r

PB

_8

c64

r


Clo

ud

Sp

ecs S

co

re



26


vCPU VALUE ANALYSIS RESULTS: FLOATING POINT

vCPU Value – Floating Point Tasks (Multicore): 2vCPU


Score

Amazon m4.large 59

Amazon c4.large 73

Amazon r3.large 43

Microsoft A2 54

Microsoft D2v2 100

Microsoft D11v2 74

DSI vCloud 2c4r 95

DSI vCloud 2c8r 95

DSI vCloud 2c16r 81

PB 2c4r 76

PB 2c8r 64

PB 2c16r 47



Score

Amazon m4.xlarge 36

Amazon c4.xlarge 45

Amazon r3.xlarge 28

Microsoft A3 36

Microsoft D3v2 63

Microsoft D12v2 47

DSI vCloud 4c8r 100

DSI vCloud 4c16r 91

DSI vCloud 4c32r 89

PB 4c8r 44

PB 4c16r 39

PB 4c32r 30



Score




Microsoft A4 21

Microsoft D4v2 36

Microsoft D13v2 23


DSI vCloud 8c32r 56

DSI vCloud 8c64r 59

PB 8c16r 25

PB 8c32r 21

PB 8c64r 17

59

73

43

54

100

74

95 95

8176

64

47

0

10

20

30

40

50

60

70

80

90

100

m4.la

rge

c4

.larg

e

r3.la

rge

A2

D2

v2

D1

1v2

DS

I_2c4

r

DS

I_2c8

r

DS

I_2c1

6r

PB

_2

c4r

PB

_2

c8r

PB

_2

c16

r


Clo

ud

Sp

ecs S

co

re

Price-Performance of vCPU (Floating Point) -2vCPU VMs

3645

2836

63

47

10091 89

4439

30

0

10

20

30

40

50

60

70

80

90

100

m4.x

larg

e

c4

.xla

rge

r3.x

larg

e

A3

D3

v2

D1

2v2

DS

I_4c8

r

DS

I_4c1

6r

DS

I_4c3

2r

PB

_4

c8r

PB

_4

c16

r

PB

_4

c32

r


Clo

ud

Sp

ecs S

co

re


2227

1721

36

23

100

56 59

25 2117

0

102030

4050

6070

8090

100

m4.2

xla

rge

c4

.2xla

rge

r3.2

xla

rge

A4

D4

v2

D1

3v2

DS

I_8c1

6r

DS

I_8c3

2r

DS

I_8c6

4r

PB

_8

c16

r

PB

_8

c32

r

PB

_8

c64

r


Clo

ud

Sp

ecs S

co

re



27


MEMORY VALUE ANALYSIS RESULTS

Memory Value (Multicore): 2vCPU


Score

Amazon m4.large 75

Amazon c4.large 85

Amazon r3.large 39

Microsoft A2 61

Microsoft D2v2 71

Microsoft D11v2 53

DSI vCloud 2c4r 100

DSI vCloud 2c8r 90

DSI vCloud 2c16r 80

PB 2c4r 72

PB 2c8r 56

PB 2c16r 37



Score

Amazon m4.xlarge 56

Amazon c4.xlarge 62

Amazon r3.xlarge 33

Microsoft A3 28

Microsoft D3v2 34

Microsoft D12v2 27

DSI vCloud 4c8r 100


DSI vCloud 4c32r 99

PB 4c8r 35

PB 4c16r 26

PB 4c32r 21



Score




Microsoft A4 13

Microsoft D4v2 16

Microsoft D13v2 11


DSI vCloud 8c32r 54

DSI vCloud 8c64r 47

PB 8c16r 14

PB 8c32r 12

PB 8c64r 9

75

85

39

61

71

53

100

90

8072

56

37

0

10

20

30

40

50

60

70

80

90

100

m4.la

rge

c4

.larg

e

r3.la

rge

A2

D2

v2

D1

1v2

DS

I_2c4

r

DS

I_2c8

r

DS

I_2c1

6r

PB

_2

c4r

PB

_2

c8r

PB

_2

c16

r


Clo

ud

Sp

ecs S

co

re

Price-Performance of Memory - 2vCPU VMs

5662

3328

3427

100 100 99

35

2621

0

10

20

30

40

50

60

70

80

90

100

m4.x

larg

e

c4

.xla

rge

r3.x

larg

e

A3

D3

v2

D1

2v2

DS

I_4c8

r

DS

I_4c1

6r

DS

I_4c3

2r

PB

_4

c8r

PB

_4

c16

r

PB

_4

c32

r


Clo

ud

Sp

ecs S

co

re


2730

16 13 1611

100

5447

14 12 9

0

10

20

30

40

50

60

70

80

90

100

m4.2

xla

rge

c4

.2xla

rge

r3.2

xla

rge

A4

D4

v2

D1

3v2

DS

I_8c1

6r

DS

I_8c3

2r

DS

I_8c6

4r

PB

_8

c16

r

PB

_8

c32

r

PB

_8

c64

r


Clo

ud

Sp

ecs S

co

re



28


STORAGE VALUE ANALYSIS RESULTS: SEQUENTIAL OPERATIONS

Storage Value – Sequential Operations: 2vCPU


Score

Amazon m4.large 6

Amazon c4.large 6

Amazon r3.large 4

Microsoft A2 12

Microsoft D2v2 9

Microsoft D11v2 7

DSI vCloud 2c4r 99

DSI vCloud 2c8r 100

DSI vCloud 2c16r 94

PB 2c4r 86

PB 2c8r 76

PB 2c16r 59



Score

Amazon m4.xlarge 20

Amazon c4.xlarge 17

Amazon r3.xlarge 18

Microsoft A3 7

Microsoft D3v2 6

Microsoft D12v2 4

DSI vCloud 4c8r 65


DSI vCloud 4c32r 83

PB 4c8r 54

PB 4c16r 46

PB 4c32r 36



Score




Microsoft A4 4

Microsoft D4v2 3

Microsoft D13v2 2


DSI vCloud 8c32r 57

DSI vCloud 8c64r 60

PB 8c16r 27

PB 8c32r 24

PB 8c64r 20

6 6 412 9 7

99 10094

86

76

59

0

10

20

30

40

50

60

70

80

90

100

m4.la

rge

c4

.larg

e

r3.la

rge

A2

D2

v2

D1

1v2

DS

I_2c4

r

DS

I_2c8

r

DS

I_2c1

6r

PB

_2

c4r

PB

_2

c8r

PB

_2

c16

r


Clo

ud

Sp

ecs S

co

re

Price-Performance of Storage (Sequential R/W) -100GB

20 17 18

7 6 4

65

100

83

5446

36

0

10

20

30

40

50

60

70

80

90

100

m4.x

larg

e

c4

.xla

rge

r3.x

larg

e

A3

D3

v2

D1

2v2

DS

I_4c8

r

DS

I_4c1

6r

DS

I_4c3

2r

PB

_4

c8r

PB

_4

c16

r

PB

_4

c32

r


Clo

ud

Sp

ecs S

co

re


16 1712

4 3 2

100

57 60

27 24 20

0102030405060708090

100

m4.2

xla

rge

c4

.2xla

rge

r3.2

xla

rge

A4

D4

v2

D1

3v2

DS

I_8c1

6r

DS

I_8c3

2r

DS

I_8c6

4r

PB

_8

c16

r

PB

_8

c32

r

PB

_8

c64

r


Clo

ud

Sp

ecs S

co

re



29


STORAGE VALUE ANALYSIS RESULTS: RANDOM OPERATIONS

Storage Value – Random Operations: 2vCPU


Score

Amazon m4.large 8

Amazon c4.large 8

Amazon r3.large 6

Microsoft A2 18

Microsoft D2v2 13

Microsoft D11v2 10

DSI vCloud 2c4r 98

DSI vCloud 2c8r 100

DSI vCloud 2c16r 85

PB 2c4r 24

PB 2c8r 20

PB 2c16r 17



Score

Amazon m4.xlarge 83

Amazon c4.xlarge 87

Amazon r3.xlarge 63

Microsoft A3 20

Microsoft D3v2 15

Microsoft D12v2 12

DSI vCloud 4c8r 50


DSI vCloud 4c32r 72

PB 4c8r 25

PB 4c16r 18

PB 4c32r 15



Score




Microsoft A4 12

Microsoft D4v2 9

Microsoft D13v2 7




PB 8c16r 15

PB 8c32r 10

PB 8c64r 17

8 8 6

1813 10

98 100

85

24 2017

0

10

20

30

40

50

60

70

80

90

100

m4.la

rge

c4

.larg

e

r3.la

rge

A2

D2

v2

D1

1v2

DS

I_2c4

r

DS

I_2c8

r

DS

I_2c1

6r

PB

_2

c4r

PB

_2

c8r

PB

_2

c16

r


Clo

ud

Sp

ecs S

co

re

Price-Performance of Storage (Random R/W) -100GB

8387

63

2015 12

50

100

72

2518 15

0

10

20

30

40

50

60

70

80

90

100

m4.x

larg

e

c4

.xla

rge

r3.x

larg

e

A3

D3

v2

D1

2v2

DS

I_4c8

r

DS

I_4c1

6r

DS

I_4c3

2r

PB

_4

c8r

PB

_4

c16

r

PB

_4

c32

r


Clo

ud

Sp

ecs S

co

re


49 52

38

12 9 7

100 100 100

1510

17

0

10

20

30

40

50

60

70

80

90

100

m4.2

xla

rge

c4

.2xla

rge

r3.2

xla

rge

A4

D4

v2

D1

3v2

DS

I_8c1

6r

DS

I_8c3

2r

DS

I_8c6

4r

PB

_8

c16

r

PB

_8

c32

r

PB

_8

c64

r


Clo

ud

Sp

ecs S

co

re



30


INTERNAL NETWORK VALUE ANALYSIS RESULTS: THROUGHPUT

Internal Network Value – Throughput: 2vCPU


Score

Amazon m4.large 6

Amazon c4.large 9

Amazon r3.large 5

Microsoft A2 9

Microsoft D2v2 18

Microsoft D11v2 13

DSI vCloud 2c4r 22

DSI vCloud 2c8r 20

DSI vCloud 2c16r 44

PB 2c4r 100

PB 2c8r 96

PB 2c16r 72



Score

Amazon m4.xlarge 10

Amazon c4.xlarge 17

Amazon r3.xlarge 7

Microsoft A3 16

Microsoft D3v2 26

Microsoft D12v2 17

DSI vCloud 4c8r 78

DSI vCloud 4c16r 46

DSI vCloud 4c32r 52

PB 4c8r 86

PB 4c16r 100

PB 4c32r 75



Score

Amazon m4.2xlarge 6


Amazon r3.2xlarge 5

Microsoft A4 10

Microsoft D4v2 10

Microsoft D13v2 7


DSI vCloud 8c32r 58

DSI vCloud 8c64r 17

PB 8c16r 41

PB 8c32r 42

PB 8c64r 27

6 9 5 918

1322 20

44

10096

72

0

10

20

30

40

50

60

70

80

90

100

m4.la

rge

c4

.larg

e

r3.la

rge

A2

D2

v2

D1

1v2

DS

I_2c4

r

DS

I_2c8

r

DS

I_2c1

6r

PB

_2

c4r

PB

_2

c8r

PB

_2

c16

r

AWS Azure DSI PB

Clo

ud

Sp

ecs S

co

re

Price-Performance of Internal Network -2vCPU VMs

1017

716

2617

78

4652

86

100

75

0102030405060708090

100

m4.x

larg

e

c4

.xla

rge

r3.x

larg

e

A3

D3

v2

D1

2v2

DS

I_4c8

r

DS

I_4c1

6r

DS

I_4c3

2r

PB

_4

c8r

PB

_4

c16

r

PB

_4

c32

r

AWS Azure DSI PB

Clo

ud

Sp

ecs S

co

re


6

17

510 10 7

100

58

17

41 42

27

01020

3040

5060

7080

90100

m4.2

xla

rge

c4

.2xla

rge

r3.2

xla

rge

A4

D4

v2

D1

3v2

DS

I_8c1

6r

DS

I_8c3

2r

DS

I_8c6

4r

PB

_8

c16

r

PB

_8

c32

r

PB

_8

c64

r

AWS Azure DSI PB

Clo

ud

Sp

ecs S

co

re



31


Conclusion

Selecting the right provider and virtual machines requires a thorough and accurate performance

comparison. The study conducted for this report offers a general understanding of performance strengths and

weaknesses across each included vendor’s VMs, and should be considered a source of information to help guide

readers in their own testing and analyses. The processor, memory bandwidth, network storage, and internal

network were all examined and results clearly show that no single provider can offer everything to fit

everyone’s needs. Therefore, in order to accurately select the right provider for a business or an application,

performance analysis is crucial.

Results from this study show that T-Systems’ DSI vCloud VMs displayed strong overall performance

and offered high price-performance value. DSI VMs demonstrated high performance for processing, storage

and internal network. DSI performance was highest in the study for processing and storage. Price-performance

value of DSI was highest in the study for storage and memory bandwidth, while having the highest value for

processor price-performance for two out of three VM sizes.

Performance in the industry cannot be assumed to be equal or even similar, as illustrated in this report.

When examining processor and memory bandwidth performance, tiered providers such as AWS and Microsoft

offer varying performance depending on the family/series of the VM, despite having equivalent amounts of

vCPUs and similar amounts of memory.

While this study was conducted in the manner of understanding a typical end user experience, it should

not be assumed to be accurate for all use cases. Stress testing was conducted to better understand fluctuation

and theoretically sustained performance, and should be seen as a general indication of overall provider

performance.

About

About Cloud Spectator

Cloud Spectator is a cloud analyst agency focused on cloud Infrastructure-as-a-Service (IaaS) performance. The

company actively monitors several of the largest IaaS providers in the world, comparing VM performance (i.e.,

CPU, RAM, disk, internal network, and workloads) and pricing to achieve transparency in the cloud market.

The company helps cloud providers understand their market position and helps business make intelligent

decisions in selecting cloud providers and lowering total cost of ownership. The firm was founded in early 2011

and is located in Boston, MA.

For questions about this report, to request a custom report, or if you have general inquiries about our products

and services, please contact Cloud Spectator at +1 (617) 300-0711 or [email protected].

For press/media related inquiries, please contact:

Ken Balazs

VP Sales & Marketing

[email protected]



cloud iaas performance & price performance - vogel.de · cloud iaas performance &...

Documents