Analyst Perspective

and Test Lab Report:

16Gb Fibre Channel

Performance and

Recommendations

Dennis Martin

Founder and President, Demartek

Agenda

• Demartek Company Overview

• Why 16 Gb Fibre Channel?

• Fibre Channel History

• Fibre Channel Roadmap

• Technical Data for Fibre Channel Including

Server Slots and Cabling

• Performance Test Results

Demartek Company Overview

• Industry analysis with on-site test lab

• Lab includes servers, networking and

storage infrastructure

– Fibre Channel: 4, 8 & 16 Gbps

– Ethernet: 1 & 10 Gbps

• NFS, CIFS, iSCSI & FCoE

– Servers: 8+ cores, very large RAM

– Virtualization: ESX, Hyper-V, Xen

• We prefer to run real-world applications to

test servers and storage solutions

– Currently testing various SSD and other

technologies

Why 16 Gb Fibre Channel?

• Provides higher bandwidth for virtualized

and cloud environments

– Reduces inter-switch link (ISL) counts

– Improves application performance

• Backward compatible and interoperable

with 4 Gb and 8 Gb Fibre Channel

• Fewer cables needed for same throughput

• SSDs provide more storage performance

than HDDs

• Who has fewer applications and VMs than

three years ago?

Fibre Channel History

• Storage Area Network (SAN) Interface

– 1997: 1 Gb/sec FC products

– 2001: 2 Gb/sec FC products

– 2005: 4 Gb/sec FC products

– 2008: 8 Gb/sec FC products

– 2009: 10 Gb/sec FCoE products

– 2011: 16 Gb/sec FC products

• Disk Drive Interface

– 4 Gb/sec was highest speed for disk drive FC interface

– Disk drive vendors moving to 6 Gb SAS for enterprise drives

Fibre Channel Roadmap

• 32 Gb & 64 Gb Fibre Channel

“The INCITS Technical Committee T11 is currently

working on the 32 GFC Fibre Channel

specifications. The 32 GFC specifications are

expected to be stabilized in mid-2013. Work has

not yet begun in T11 for developing the 64 GFC

specifications, but 64 GFC is on the FCIA Speed

roadmap.”

Steve Wilson, Director of Technology and

Standards, Brocade and INCITS TC T11 Chairman

Fibre Channel and Servers

• Fibre Channel speeds and server slots

– 4 Gb: PCI-X 2.0, PCIe 1.0

– 8 Gb: PCIe 2.0 (x4 or x8)

– 16 Gb: PCIe 2.0 (x8)

GT/s Encoding x1 x2 x4 x8 x16

PCIe 1.x 2.5 8b/10b 250 MB/s 500 MB/s 1 GB/s 2 GB/s 4 GB/s

PCIe 2.x 5 8b/10b 500 MB/s 1 GB/s 2 GB/s 4 GB/s 8 GB/s

PCIe 3.x 8 128b/130b 1 GB/s 2 GB/s 4 GB/s 8 GB/s 16 GB/s

March 6, 2012

The major server vendors announced their PCIe 3.0 servers. These

have 40 PCIe 3.0 lanes per processor (Intel Xeon E5-2600) socket.

Source: www.demartek.com/Demartek_Interface_Comparison.html

Cabling Descriptions

Mode Core

Diameter Wavelength Modal Bandwidth Cable jacket color

OM1 multi-

mode 62.5 µm

850 nm

1300 nm 200 MHz Orange

OM2 multi-

mode 50 µm

850 nm

1300 nm 500 MHz Orange

OM3 multi-

mode 50 µm

850 nm

1300 nm 2000 MHz Aqua

OM4 multi-

mode 50 µm

850 nm

1300 nm 4700 MHz Aqua

OS1

single-

mode 9 µm

1310 nm

1550 nm — Yellow

Source: www.demartek.com/Demartek_Interface_Comparison.html

Cabling

OM1 OM2 OM3 OM4

1 Gb/s 300m 500m 860m –

2 Gb/s 150m 300m 500m –

4 Gb/s 70m 150m 380m 400m

8 Gb/s 21m 50m 150m 190m

10 Gb/s 33m 82m Up to 300m Up to 400m

16 Gb/s 15m 35m 100m 125m

Source: www.demartek.com/Demartek_Interface_Comparison.html

• Fiber optic cabling service life: 15 to 20 years

– The choices made today need to support legacy, current and emerging

data rates.

• OM1 cable is not recommended for 16 Gb/s FC

• Recommendation: buy OM3 or OM4 cables

16 Gb FC Performance Tests

• Goal

– Provide new architecture to take advantage of

very high speed storage and reduce number of

racks and power consumption

• Test 1: Transaction-intensive

• Test 2: Bandwidth-intensive

Test 1: Transaction-intensive

• Problem

– Huge cloud application struggling to maintain SLAs for

database performance

– Heavy dependence on local disks and large RAM in

servers limits size of database, number of users per rack

and the number of queries that each server can handle

• Solution

– New architecture using 16 Gb FC and SSDs to improve

scalability and significantly reduce number of racks

required for application

– Achieve 1 million IOPS in single rack for servers and

storage with low latency

Test 1 Configuration

Test 1: Results • MySQL Clusters

– 10 servers, 4 instances of MySQL on each server

– Databases: 256 GB each instance

– 8 mirrored storage volumes, all flash

102,556

101,670

102,671

103,270

102,846

101,552

100,510

101,806

102,932

101,754

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

1,000,000

1,100,000

IOP

S

Aggregate Database IOPS

mysql10

mysql9

mysql8

mysql7

mysql6

mysql5

mysql4

mysql3

mysql2

mysql1

Test 2: Bandwidth-intensive

• Problem

– Enterprise database cluster needs faster performance

– Enterprise database with 10.2 billion rows

• Solution

– New architecture using 16 Gb FC and SSDs to improve

performance and reduce power consumption and rack

space

– Achieved sustained performance > 7200 MB/sec

– Storage power consumption: 500 watts

– Storage capacity: 10 TB in 3 rack units (3U)

Test 2 Configuration

Test 2: Results • Single large database cluster

– “Select count(*)” all 10.2 billion rows completed in

< 8 minutes

0

1000

2000

3000

4000

5000

6000

7000

8000

MB

/S

ec

All-flash Array Aggregate Bandwidth

fc-5b

fc-5a

fc-4b

fc-4a

fc-3b

fc-3a

fc-2b

fc-2a

fc-1b

fc-1a

Demartek References

• Full performance reports will be public soon

– Contact me if you want advance copies

• Storage Interface Comparison

www.demartek.com/Demartek_Interface_Comparison.html

• Demartek SSD Zone

www.demartek.com/SSD.html

• Demartek Free Monthly Newsletter

www.demartek.com/Newsletter/Newsletter_main.html

Other Fibre Channel Sessions

at SNW Spring 2012

• Monday

– 1:00 p.m. – Industry Perspective: The Future of Fibre

Channel, Mark Jones, FCIA Vice Chairman and Technical

Marketing Director, Emulex Corporation

• Tuesday

– 10:45 a.m. – SNIA Tutorial: Fibre Channel over Ethernet

(FCoE), John Hufferd, Owner/Consultant, Hufferd Enterprises

– 11:40 a.m. – Analyst Perspective: FC & FCoE solutions

panel, Moderator: Dennis Martin, President, Demartek

Contact Information

(303) 940-7575

www.demartek.com

http://twitter.com/Demartek

YouTube: www.youtube.com/Demartek

Skype: Demartek

Dennis Martin, President

dennis@demartek.com

www.linkedin.com/in/dennismartin