Slide 1

An Interop Event for NVMe SSDs Thursday October 2, 2014 2PM Eastern NVMe Plugfest Webinar Presented by: David Woolf - UNH-IOL david@iol.unh.edu 2014 University of New Hampshire InterOperability Laboratory

Slide 2

2 UNH-IOL Industry leading 3 rd -party test facility for data, telecom and storage networking technology & consumer electronics 100% funded by commercial industry 150+ companies 32,000 sq. ft lab facility Boston, USA Metro Area 7,200 sq. ft pre-wired space dedicated to Plugfests 2 2014 University of New Hampshire InterOperability Laboratory

Slide 3

3 Member Involvement (150+ Companies) 3

Slide 4

4 UNH-IOL NVMe Consortium Benefits Access to UNH-IOL NVMe test services. Attendance at NVMe Plugfest Access to UNH-IOL INTERACT PC Edition and UNH-IOL INTERACT Teledyne-LeCroy Edition software for performing NVMe Conformance Tests. Ability to list qualifying products on the NVMe Integrators List: Place products in the UNH-IOL NVMe Interop test bed for continuous interop testing. Support for UNH-IOL Administration of open source tNVMe tool PCIe Electrical testing available through UNH-IOL PCIe Consortium

Slide 5

5 NVMe Plugfest #3 UNH-IOL November 10-13, 2014 Durham, NH USA

Slide 6

6 Agenda NVMe Test Program Goals and Overview Plugfest Components Interop Tests Conformance Tests Plugfest Logistics Registration Test Stations Walk-Thru Reporting Listing Products Testing After the Plugfest

Slide 7

7 Agenda NVMe Test Program Goals and Overview Plugfest Components Interop Tests Conformance Tests Plugfest Logistics Registration Test Stations Walk-Thru Reporting Listing Products Testing After the Plugfest

Slide 8

8 Growth of NVMe Enterprise SSD market is expected to reach $4 billion in revenues by 2016, nearly six times that of 2011, while unit shipments are expected to increase by ten times during that period, to approximately 4 million units. - Forbes.com July 2014 NVMe is the next gen interface for both client and enterprise systems. NVMe products announcements are happening There is a strong need to demonstrate the interoperability of NVMe products. The NVMe plugfest at UNH-IOL is a key component in accomplishing this.

Slide 9

9 Since 2012 NVMe Promoters Group collaborating with UNH-IOL to manage the Integrators List based Test Program for NVMe. NVMe Test Program Overview

Slide 10

10 NVMe Integrators List https://www.iol.unh.edu/services/testing/NVMe/integratorslist.php NVMe Integrators List hosted by UNH-IOL as of Sept 30, 2014:

Slide 11

11 NVMe Integrators List NVMe Integrators List is the industry resource for listing products proven to interoperable and conformance according to industry accepted test practices. Listed products demonstrate commitment to interoperability and open standards conformance. OEMs viewing Integrators List qualification as necessary for components.

Slide 12

12 NVMe Test Program Goals Prove the robustness and interoperability of NVMe through continually improving interop and conformance testing. Help implementers ensure the conformance and interoperability of their products Provide recognition for products meeting the necessary qualifications backed by test reports.

Slide 13

13 How do we populate the Integrators List? New Interop and Conformance Testing Requirements approved by NVMe Promoters group New Testing requirements launched at plugfests. Testing for Integrators List performed throughout the year at UNH-IOL.

Slide 14

14 Agenda NVMe Test Program Goals and Overview Plugfest Components Interop Tests Conformance Tests Plugfest Logistics Registration Test Stations Walk-Thru Reporting Listing Products Testing After the Plugfest

Slide 15

15 NVMe Plugfest Components NVMe Plugfest #3 November 10-13, 2014 will test according to version 1.1b of the NVMe Test Program. First opportunity to test devices to the new requirements and have them listed on IL. After the Plugfest, UNH-IOL members can request 1.1b Integrators List testing throughout the year.

Slide 16

16 Test Plans 2 Test plans will be executed at the plugfest. 1.UNH-IOL NVMe Interoperability Test Suite Document 1.1b 2.UNH-IOL NVMe Conformance Test Suite Document v1.1b

Slide 17

17 Common Test Terms Mandatory DUT must pass test in order to qualify for IL. Optional If DUT supports an optional NVMe feature, it must pass test inorder to qualify for IL. If option is not supported, the test is not performed and does not affect IL qualification. FYI Test does not affect IL qualification, but may become mandatory in the future.

Slide 18

18 Passing Requirements Overview Conformance Tests DUT must pass all mandatory conformance tests supported by test tools If feature is supported, DUT must also pass optional tests supported by test tools

Slide 19

19 Passing Requirements Overview Interop Tests Plugfest #3 (Today) DUT must pass mandatory tests with 4/5 systems. FYI tests do not affect IL qualification, are only performed with 1 system Plugfest #4 (2015) DUT must pass mandatory tests with 5/6 systems. FYI tests will become mandatory, are only performed with 1 system

Slide 20

20 Interop Test Plan Mandatory Tests (1.1, 1.2, 1.3, 1.4) Install SSD, identify drive in OS Format drive, write/read/verify for 3 minutes Reboot 3x/shut down 3x FYI Tests (1.5, 1.6) 1.5 - Boot from NVMe device 1.6 - FYI Hotplug test (SFF-8639 only) Will run all SSD and Host combinations possible at the event. Publically available at: https://www.iol.unh.edu/services/testing/NVMe/testsuites

Slide 21

21 Interop Test Tools VDBENCH 5.04 Command Line utility for generating IO Workloads to disk. Java program works on Windows, Solaris, Linux stations. Available at oracle.com http://www.oracle.com/technetwork/server- storage/vdbench-downloads-1901681.html Download vdbench Parameter File for NVMe Interop Tests created by UNH- IOL. https://www.iol.unh.edu/services/testing/ NVMe/unh_interop_1_1b_linux.txt https://www.iol.unh.edu/services/testing/ NVMe/unh_interop_1_1b_windows.txt

Slide 22

22 UNH-IOL will have 2 SFF-8639 to CEM adapters on hand. If you have a non CEM form factor, please bring it, and also please bring at least 2 of your own adapters. This will facilitate interop testing. SFF-8639 / M.2 Adapters 22 PCI-AD-x439-01 adapter from Serial Cables (serialcables.com)

Slide 23

23 Conformance tests will only require 1 sample. Interop tests require that 2 samples be used simultaneously. Please bring at least 2 samples of your product, this will facilitate interop testing. How many samples? 23

Slide 24

24 What Interop Tests are Required for Plugfest #3? Test GroupMandatoryOptional/FYI Test 1.1 Storage Device Identified Pass with 4/5 Hosts Test 1.2 Format Storage Device Pass with 4/5 Hosts Test 1.3 Write Read Compare Pass with 4/5 Hosts Test 1.4 Multiple Devices on Bus Pass with 4/5 Hosts Test 1.5 Boot from NVMe Device Mandatory for Plugfest #4 Pass with 1 Host: Dell PowerEdge R720 Test 1.6 Hotplug NVMe Device Mandatory for Plugfest #4 Pass with 1 Host: TBD Test 1.7 Dual Port Device Not implemented

Slide 25

25 Interop Systems PlatformOSIntended Testing Dell PowerEdge R720Windows Server 2012 (OFA Reference Driver) Write/Read/Verify Intel Patsburg CRBWindows 8.1 (Microsoft Driver) Write/Read/Verify Intel Patsburg CRBUbuntu 14.04 (Reference Driver) Write/Read/Verify SUPERMICRO MBD-X10SAT-O ATX Server Motherboard RHEL 6.5 (Reference Driver) Write/Read/Verify SUPERMICRO MBD-X10SAT-O ATX Server Motherboard Windows Server 2012 R2 (Microsoft Driver) Write/Read/Verify TBD Hotplug (FYI) Dell PowerEdge R720UEFI / Windows 8.1Boot (FYI)

Slide 26

26 Conformance Test Plan Use Host Emulator to exercise and stimulate SSD. Focus on completion of Admin Commands, NVM Commands, Controller Registers Publically available at: https://www.iol.unh.edu/services/testing/NVMe/ testsuites

Slide 27

27 Conformance Test Tools IOL INTERACT PC EDITION IOL INTERACT TELEDYNE- LECROY EDITION

Slide 28

28 Test Tools IOL INTERACT PC EDITION Linux based NVMe Conformance Test Station uses: tNVMe v.3.0 available from github (UNH-IOL admin) IOL INTERACT 1.1 (available from UNH-IOL) RELEASED July 30, 2014 Ubuntu 10.10, 12.04, 13.04, 14.04 Executes most required conformance tests defined in UNH-IOL NVMe Conformance Test Suite Document Installation walkthru: http://www.flashmemorysummit.com/English/Collat erals/Proceedings/2013/20130812_PreConfD_Mehta _Woolf.pdf Obtained through UNH-IOL membership Contact: kerry.munson@iol.unh.edu

Slide 29

29 Test Tools IOL INTERACT TELEDYNE-LECROY EDITION Requires Teledyne-LeCroy Summit Z3 Exerciser and Analyzer, Windows PC Currently on LeCroy Software Build 6.75, aiming for 7.2 by plugfest time. Executes required conformance tests defined in UNH-IOL NVMe Conformance Test Suite Document For final IL Qualification UNH-IOL will examine bus traces from IOL INTERACT TELEDYNE-LECROY EDITION Obtained through UNH-IOL membership. Contact: kerry.munson@iol.unh.edu

Slide 30

30 Why 2 Conformance Tools? Community needs a tool with a relatively low cost barrier, to be able to determine conformance. The PC Edition meets that need. Community also needs a tool with deep debug and analysis capability. Teledyne- LeCroy edition meets that need. Tradeoff: Cost vs. Debug Capability + Trace Analysis

Slide 31

31 Test Support Test Group IOL INTERACT PC Edition v1.1 IOL INTERACT Teledyne-LeCroy Edition v1.2 Group 1: Admin Command Set All tests except 1.5 - Abort Command All tests supported Group 2: NVM Command Set All tests except 2.7 - Write Zeros All tests supported Group 3: NVM FeaturesTests 3.2 and 3.3 not implemented. All Tests Supported Group 4: Controller Registers All Tests Supported Group 5: System Memory Structure All Tests Supported Group 6: Controller Architecture No Tests implementedAll tests except 6.1 Group 7: ReservationsNot Implemented

Slide 32

32 What Conformance Tests are Required? Test GroupMandatory Optional (if supported must be supported correctly) Group 1: Admin Command Set All Tests MandatoryNone Group 2: NVM Command Set 2.3, 2.4, 2.62.1, 2.2, 2.5, 2.7 Group 3: NVM Features3.33.1, 3.2 Group 4: Controller Registers All tests except 4.6 Arbitration Mechanism 4.6 Arbitration Mechanism Group 5: System Memory Structure All Tests MandatoryNone Group 6: Controller Architecture 6.1, 6.2, 6.36.4

Slide 33

33 Agenda NVMe Test Program Goals and Overview Plugfest Components Interop Tests Conformance Tests Plugfest Logistics Registration Test Stations Walk-Thru Reporting Listing Products Testing After the Plugfest

Slide 34

34 Registration open to UNH-IOL members: https://www.iol.unh.edu/servic es/testing/NVMe/plugfest.php https://www.iol.unh.edu/servic es/testing/NVMe/plugfest.php Membership https://www.iol.unh.edu/servic es/membership/form/ https://www.iol.unh.edu/servic es/membership/form/ Plugfest Logistics - Registration

Slide 35

35 Companies developing or deploying NVMe products Companies supporting NVMe products in their systems Products should be functional to the point that they are identified as a storage device in a Windows or Linux system with an NVMe driver. No value in attending plugfest without a functional product. Plugfest Logistics Who should attend?

Slide 36

36 Each SSD will be cycled through multiple conformance and interop test stations and interop hosts. Conformance stations are test tools. Interop Stations are NVMe Hosts with varying NVMe drivers. Not all stations required for Integrators List qualification Some stations are FYI, to experiment for future tests. Plugfest Test Stations

Slide 37

37 Following PCI-SIG recommended test procedures SFF-8639 only Help in tuning phy parameters in new form factors Tests are FYI only, not required for Integrators List qualification FYI Conformance Test Stations PCIe PHY Conformance Test Station (FYI)

Slide 38

38 Additional Test equipment companies may attend to help improve their test offerings. Any non UNH-IOL stations should be considered FYI. FYI Conformance Test Stations Protocol Conformance Test Station (FYI)

Slide 39

39 Hotplug Test Station Experimental test to help create appropriate hot plug test cases for future. UEFI /Boot Test Station Experimental test to exercise UEFI NVMe driver. Tests are FYI only, not required for Integrators List qualification today. These tests will be required at Plugfest #4 FYI Interop Test Stations Hotplug Interop Test Station (FYI) UEFI/Boot Interop Test Station (FYI)

Slide 40

40 Reporting Conformance Test Data will be collected directly by UNH-IOL Staff Bus traces and logs Interop Data may be collected by UNH-IOL staff, or engineers supporting their companies equipment Summary and Error logs produced by VDBENCH

Slide 41

41 Reporting UNH-IOL will use collected test data to produce Interop and Conformance Test Reports for all products at the Plugfest Test Reports are necessary to list a product on the NVMe Integrators List Reports are can be shared with partners and customers

Slide 42

42 Agenda NVMe Test Program Goals and Overview Plugfest Components Interop Tests Conformance Tests Plugfest Logistics Registration Test Stations Walk-Thru Reporting Listing Products Testing After the Plugfest

Slide 43

43 Listing Products NVMe Integrators List Policy: https://www.iol.unh.edu/services/testing/NVMe/20131219_NVMe_Integrators_List_Policy_v1_1b.pdf Test Reports are necessary to list a product on the NVMe Integrators List UNH-IOL Report will indicate whether a product is eligible or not Products are not automatically listed In order to list a product, companies must make specific email request to UNH-IOL through nvmelab@iol.unh.edu

Slide 44

44 Listing Products Listed Information Includes: Product, includes Manufacturer, Model Name and Family Name (i.e. different capacities of one SSD are in a single family) Product type (accepted products types are: NVMe SSD, SSD Controller, or SSD Controller IP, server, server board, motherboard, add-in card, RAID Controller, or IP device). Operating System and Driver (Host Only) Firmware revision (Device Only) Interop Program Revision (i.e. 1.0, 1.1, 1.1b etc...) Date of listing and Test ID Further Information (primary contact or website)

Slide 45

45 Listing Similar Products Products with differences that do not affect NVMe operation (form factor, storage capacity) can be listed under a single product listing, listed by similarity. The products listed by similarity must have the same Firmware revision and Model Name, and the listing company confirms that the products are materially similar with no substantive changes to the NVMe interface.

Slide 46

46 Agenda NVMe Test Program Goals and Overview Plugfest Components Test Plans Test Tools Plugfest Logistics Registration Test Stations Walk-Thru Reporting Listing Products Testing After the Plugfest

Slide 47

47 Post-Plugfest Testing Products will continue to be tested at UNH-IOL throughout the year. Test requests can be made through: https://www.iol.unh.edu/ my/ https://www.iol.unh.edu/ my/ Passing products can be listed on the NVMe Integrators List

Slide 48

48 Future Requirements Mandatory interop tests will increase from 4/5 to 5/6 UEFI Boot test will be mandatory at Plugfest #4 Hotplug test will be mandatory at Plugfest #4

Slide 49

49 UNH-IOL NVMe Consortium Benefits: Access to UNH-IOL NVMe test services. Attendance at NVMe Plugfest Access to UNH-IOL INTERACT PC Edition and UNH-IOL INTERACT Teledyne-LeCroy Edition software for performing NVMe Conformance Tests Ability to list qualifying products on the NVMe Integrators List Place products in the UNH-IOL NVMe Interop test bed for continuous interop testing Support for UNH-IOL Administration of open source tNVMe tool PCI testing available through UNH-IOL PCIe Consortium

Slide 50

50 Review NVMe Test Program focused on Integrators List to showcase interoperable and conformant products. Test tools and plans available from UNH-IOL. Plugfest will allow the first opportunity to list products on the NVMe Integrators list v1.1b. Plugfest will walk SSDs through multiple conformance and interop stations. Testing can continue after the Plugfest for those unable to attend.

Slide 51

51 Resources UNH-IOL Membership https://www.iol.unh.edu/services/membership/form/ Plugfest Registration https://www.iol.unh.edu/services/testing/NVMe/plugfest.php NVMe Conformance and Interop Test Suites https://www.iol.unh.edu/services/testing/NVMe/testsuites/ NVMe Integrators List and Policy Document https://www.iol.unh.edu/services/testing/NVMe/integratorslist.php

Slide 52

52 Q&A Contact: david@iol.unh.edu

Slide 53

53 Q&A When will the 1.1b IOL-INTERACT Linux and Lecroy tools be available for download? As far as test support, the currently released tools support all the necessary tests. Future releases prior to the plugfest will contain bug fixes and update LeCroy software support.

Slide 54

54 Q&A FYI boot from NVMe will require an OS to be installed. What if the NVMe device is too small for an OS to be installed? Then the test cannot be performed. This is why we have this as an FYI test today, so that you can properly prepare for the next plugfest and have a device large enough for an OS then.

Slide 55

55 Q&A How does the CEM adapter support hot-plug testing as the server motherboard PCIe slots also should support the hot plug We will not be performing hotplug on CEM card systems or devices.

Slide 56

56 Q&A What is the difference between github version and iol-interact SW other than a gui interface? IOL INTERACT supports test groups 3, 4, and 5, which tNVMe does not directly support.

Slide 57

57 Q&A Slide mentions IOL-Interact 1.1, but not 1.1b release In this case the 1.1 refers to the software version. The 1.1 software version supports most of the tests required for the NVMe IL version 1.1b (see table on slide 31 ). Its just an unfortunate coincidence that the numbers are close.

Slide 58

58 Q&A Is PCIe phy conformance available for non-CEM form factors Yes Is UEFI testing available on any motherboard that can be done locally? other platforms that support UEFI Yes, the test can be performed on any platform that supports UEFI, but UNH-IOL will only supply 1 platform at the plugfest.

Slide 59

59 Q&A Is there a deadline for registration? October 17, 2014 Do you use Solaris host for interop test? Currently we dont have a Solaris host for this interop event. However, using a Solaris host as one of the hosts for interop testing would still help qualify an SSD for the IL.

Slide 60

60 Q&A We are interested in bringing a server instead of a drive. Would you require 2 models the same as you want for drives? Bringing a single server is fine. We only need two samples of SSDs.