application-level benchmarking with spec sfs® 2014 · benchmarking: real-life examples (env 1)...

2015 Storage Developer Conference. © 2015 Standard Performance Evaluation Corporation. All Rights Reserved.

Application-Level Benchmarking with SPEC SFS® 2014

Nick Principe – EMC Vernon Miller – IBM


Agenda

Why application-level benchmarking? What is application-level benchmarking? The SPEC SFS 2014 Workloads

Reporting SFS 2014 Results Testing a “Storage Solution” Ramifications of application-level benchmarking

Concepts Real-life examples

Key takeaways Q&A

2


SPEC Standard Performance Evaluation Corporation

The Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops benchmark suites and also reviews and publishes submitted results from member organizations and other benchmark licensees

www.spec.org SPEC and SPEC SFS are registered trademarks of the

Standard Performance Evaluation Corporation. Additional product and service names mentioned herein may be the trademarks of their respective owners.

3

http://www.spec.org/


Why application-level benchmarking?

The focus of the SFS benchmark has changed in SFS 2014 Load is now generated at the application level The aim is to measure the storage

performance of the environment as a whole We call this the “Storage Solution”

Vendors now have the flexibility to configure the benchmark to match their environment Put the bottleneck where you want to show value

4


Why application-level benchmarking?

Increased flexibility addresses the market Complexity of storage solutions Diversity of architectures and protocols Fairness to all implementations

Any other approach is no longer appropriate for industry-standard benchmarking of storage solutions

5


What is Application-Level Benchmarking?

Prior to SFS 2014, benchmark generated its own NFS or SMB traffic Bypassed load generator operating system Focus: performance of monolithic NAS server

SFS 2014 uses native OS calls to generate application-level load Data and metadata ops processed by OS Focus: storage performance of the

environment as a whole (Storage Solution) 6


The SPEC SFS 2014 Workloads

DATABASE Simulates OLTP database consolidation Measured in # of concurrent DATABASES

SWBUILD Simulates large software project compilation Measured in # of concurrent BUILDS

VDA Simulates acquisition of streaming data Measured in # of concurrent STREAMS

VDI Simulates heavy steady-state VDI workload Measured in # of concurrent DESKTOPS 7

For more details, see: - SDC 2014 presentation: SPEC SFS 2014: An Under-the-Hood Review - The SPEC SFS 2014 website http://www.spec.org/sfs2014

http://www.spec.org/sfs2014


Reporting SFS 2014 Results

Disclosure of SPEC SFS 2014 results must meet the requirements of

SPEC SFS 2014 License SPEC SFS 2014 Run and Reporting Rules SPEC Fair Use Rules

Submission to SPEC for review encouraged Certain information is required to be disclosed

Do not use this presentation as a guide for public disclosure of SFS 2014 results

Created for education under auspices of SPEC using “generic” environments 8


Testing a “Storage Solution”

More attention to benchmark configuration required You must put the bottleneck in the right place

Understanding the whole system, from load generator to the disks, is a requirement

For publication, more configuration details must be recorded and disclosed

SFS 2014 is still a storage benchmark There is no attempt to simulate compute load

9


Ramifications of application-level benchmarking: Concepts

Your load generators matter Any config detail can affect performance

Storage connectivity, OS version, patch level, memory, client count, tuning parameters

You can test anything that provides a file API to an application Traditional NAS server, block storage with a

file system on load generators, hyper-converged solutions, a single server with storage 10



With great power comes great responsibility Understand where your bottleneck is

SFS 2014 allows great flexibility in load placement as the workload scales

Getting this right is the key to getting the performance you expect Likely you want to spread load as evenly as possible

across ALL resources as the benchmark ramps up load

The key config parameter in SFS 2014? CLIENT_MOUNTPOINTS 11



Measuring performance at multiple levels of the solution under test is key to understanding your solution’s performance and bottlenecks

SFS 2014 reports application-level performance Other statistics that are helpful to collect:

Storage array statistics, NAS server statistics, Hypervisor statistics, LG OS statistics

You may see different performance at the different levels Each layer of the solution under test may change the

workload 12


Ramifications of application-level benchmarking: Real-life Examples (Env 1)

Configured an environment for testing Midrange Storage Array

FC drives, FC frontend

Large Windows Server 2012 R2 NAS Server FC backend, 48 cores, 256GB memory, 10GbE frontend

20 Windows 8.1 VMs; 10 physical servers FC or 10GbE SMB3 connectivity, 2 cores, 2 GB memory

Testing was done in two ways Via SMB3 shares from NAS server (10GbE) Via local E: drive, FC LUN via RDM to VM (FC) 13



14

ESXi Server 1

10 GbE Switch

NAS Server

Storage Controller

SMB Server Measurements

Disk Measurements

Network Measurements

Application Measurements Win VM

Win VM

ESXi Server 2 Win VM

Win VM

ESXi Server 10

Win VM

Win VM

8 Gb FC Switch


Ramifications of application-level benchmarking: Real-life Examples (Env 2) Configured an environment for testing Midrange Storage Array

SAS drives, FC frontend

4 Node Distributed Filesystem 2 nodes with FC backend, DDR IB cluster network 2 nodes acting as NFS server, 10 GbE frontend

2 NFS Clients RHEL 6.5, 10GbE connectivity, 4 cores, 32 GB memory

Testing was done in two ways Via NFSv3 exports from NAS server (10GbE) Via local filesystem: single namespace on 2 nodes 15



16

NFS Client NFS Client

10 GbE Switch

DDR IB Switch

Frontend Node Frontend Node

Backend Node Backend Node

Storage Controller

Load generators for NFSv3 tests

Load generators for local filesystem tests

NFS Server Measurements

Disk Measurements

Network Measurements

Application Measurements



17

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

500,000

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Kilo

byte

s/Se

cond

Streams

VDA - NFS

App Read App Write Net Recv

Net Send Disk Read Disk Write

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

500,000

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Kilo

byte

s/Se

cond

Streams

VDA - Cluster FS

App Read App Write

Disk Read Disk Write



18

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Kib

ibyt

es/s

ec

Streams

VDA - SMB



0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

10 20 30 40 50 60 70 80 90 100

Kib

ibyt

es/s

ec

Streams

VDA - Local FS

App Read App Write




19

0

200

400

600

800

1,000

1,200

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Ope

rati

ons/

Seco

nd

Streams

VDA - NFS

App NFS Total NFS Data NFS Meta Disk

0

200

400

600

800

1,000

1,200

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Ope

rati

ons/

Seco

nd

Streams

VDA - Cluster FS

App Disk



20

0

100

200

300

400

500

600

700

800

900

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Ope

rati

ons/

sec

Streams

VDA - SMB

App SMB Total SMB Data SMB Meta Disk

0

200

400

600

800

1,000

1,200

1,400

1,600

1,800

2,000

10 20 30 40 50 60 70 80 90 100

Ope

rati

ons/

sec

Streams

VDA - Local FS

App Disk


Ramifications of application-level benchmarking: Real-life Examples

21

0

10

20

30

40

50

60

70

80

90

100

0 200 400 600 800 1,000 1,200

Ave

rage

Res

pons

e T

ime

(ms)

Achieved Ops/Sec

VDA - Env 2

NFSv3 Local

0

50

100

150

200

250

0 200 400 600 800 1,000 1,200

Ave

rage

Res

pons

e T

ime

(ms)

Achieved Ops/sec

VDA - Env 1

SMB3 Local



22

0

20,000

40,000

60,000

80,000

100,000

120,000

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

Kilo

byte

s/Se

cond

Desktops

VDI - NFS



0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

Kilo

byte

s/Se

cond

Desktops

VDI - Cluster FS

App Read App Write




23

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

1 2 3 4 5 6 7 8 9 10 11 12

Kib

ibyt

es/s

ec

Desktops

VDI - Local FS

App Read App Write


0

5,000

10,000

15,000

20,000

25,000

30,000

1 2 3 4 5 6 7 8 9 10 11

Kib

ibyt

es/s

ec

Desktops

VDI - SMB





24

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

Ope

rati

ons/

Seco

nd

Desktops

VDI - NFS


0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

Ope

rati

ons/

Seco

nd

Desktops

VDI - Cluster FS

App Disk



25

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

1 2 3 4 5 6 7 8 9 10 11

Ope

rati

ons/

sec

Desktops

VDI - SMB


0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

1 2 3 4 5 6 7 8 9 10 11 12

Ope

rati

ons/

sec

Desktops

VDI - Local FS

App Disk



26

0

2

4

6

8

10

12

14

0 2,000 4,000 6,000 8,000 10,000

Ave

rage

Res

pons

e T

ime

(ms)

Achieved Ops/Sec

VDI - Env 2

NFSv3 Local

0

1

2

3

4

5

6

0 500 1,000 1,500 2,000 2,500 3,000

Ave

rage

Res

pons

e T

ime

(ms)

Achieved Ops/sec

VDI - Env 1

SMB3 Local



27

0

20,000

40,000

60,000

80,000

100,000

120,000

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

Kilo

byte

s/Se

cond

Desktops

VDI - NFS (Read Bandwidth)

App Read

Disk - Original

Disk - Tuning #1

Disk - Tuning #2



28

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Kilo

byte

s/Se

cond

Builds

SWBUILD - NFS



0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Kilo

byte

s/Se

cond

Builds

SWBUILD - Cluster FS

App Read App Write




29

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

2 4 6 8 10 12 14 16 18 20

Kib

ibyt

es/s

ec

Builds

SWBUILD - Local FS

App Read App Write


0

10,000

20,000

30,000

40,000

50,000

60,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Kib

ibyt

es/s

ec

Builds

SWBUILD - SMB





30

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Ope

rati

ons/

Seco

nd

Builds

SWBUILD - NFS


0

2,000

4,000

6,000

8,000

10,000

12,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Ope

rati

ons/

Seco

nd

Builds

SWBUILD - Cluster FS

App Disk



31

0

2,000

4,000

6,000

8,000

10,000

12,000

2 4 6 8 10 12 14 16 18 20

Ope

rati

ons/

sec

Builds

SWBUILD - Local FS

App Disk

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Ope

rati

ons/

sec

Builds

SWBUILD - SMB




32

0

2

4

6

8

10

12

0 2,000 4,000 6,000 8,000 10,000 12,000

Ave

rage

Res

pons

e T

ime

(ms)

Achieved Ops/Sec

SWBUILD - Env 2

NFSv3 Local

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 2,000 4,000 6,000 8,000 10,000 12,000

Ave

rage

Res

pons

e T

ime

(ms)

Achieved Ops/sec

SWBUILD - Env 1

SMB3 Local


33


Percent of I/O

Client Cache Hit

NAS Server Cache Hit Cluster/Disk Cache Hit

Disk Access


Key takeaways

The SPEC SFS 2014 is an application-level benchmark that tests the storage performance of an entire storage solution

Understanding the storage solution under test and bottleneck placement are keys to getting “what you expect” from your storage solution

The application-level benchmarking provided by SFS 2014 allows testing of a much wider array of products and storage solutions

34


Q & A

Any questions?

Thank you for attending! Please remember to submit feedback!

35

application-level benchmarking with spec sfs® 2014 · benchmarking: real-life examples (env 1)...

Documents