suse yes certification test kit · suse yes certification test kit 1.1 august 15, 2017 suse®...

SUSE YES Certification Test Kit 1.1 www.SUSE.com

August 15, 2017 SUSE® ENTERPRISE STORAGE

http://www.suse.com/

2

Legal Notices

SUSE LLC., makes no representations or warranties with respect to the contents or use of this documentation,

and specifically disclaims any express or implied warranties of merchantability or fitness for any particular

purpose. Further, SUSE LLC., reserves the right to revise this publication and to make changes to its content, at

any time, without obligation to notify any person or entity of such revisions or changes.

Further, SUSE LLC., makes no representations or warranties with respect to any software, and specifically

disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further,

SUSE LLC., reserves the right to make changes to any and all parts of SUSE software, at any time, without any

obligation to notify any person or entity of such changes.

Any products or technical information provided under this Agreement may be subject to U.S. export controls and

the trade laws of other countries. You agree to comply with all export control regulations and to obtain any

required licenses or classification to export, re-export, or import deliverables. You agree not to export or re-

export to entities on the current U.S. export exclusion lists or to any embargoed or terrorist countries as

specified in the U.S. Export laws. You agree to not use deliverables for prohibited nuclear, missile, or chemical

biological weaponry end uses. Please refer to https://www.suse.com/company/legal/ for more information on

exporting SUSE software. SUSE assumes no responsibility for your failure to obtain any necessary export

approvals.

All files provided in this release are subject to the Novell Developer License Agreement, which can be found in

the license.txt file provided in the System Certification Test Kit download.

Copyright © 2004-2016 SUSE LLC. All rights reserved. No part of this publication may be reproduced,

photocopied, stored on a retrieval system, or transmitted without the express written consent of the publisher.

SUSE LLC., has intellectual property rights relating to technology embodied in the product that is described in

this document. In particular, and without limitation, these intellectual property rights may include one or more

of the U.S. patents listed at http://www.suse.com/company/legal and one or more additional patents or pending

patent applications in the U.S. and in other countries.

SUSE LLC.

1800 South Novell Place

Provo, Utah, 84606

U.S.A.

www.SUSE.com

Online Documentation: To access the online documentation for this and other SUSE products, and to get

updates, www.suse.com/documentation.

Online SUSE Developer Services Program Guide Documentation: To access the online SUSE Developer

Services Program Guide, see

https://www.suse.com/docrep/documents/h4mylk7uec/suse_developer_services_pg.pdf .

Trademarks

For a list of SUSE trademarks, see Trademarks (http://www.suse.com/company/legal/).

All third-party trademarks are the property of their respective owners.

https://www.suse.com/company/legal/

http://www.suse.com/

http://www.suse.com/documentation

http://www.novell.com/docrep/2012/06/suse_developer_services_pg.pdf?noredir=true

http://www.suse.com/company/legal/)

3 SUSE Enterprise Storage – Certification Kit

Table of Contents

1 SUSE Enterprise Storage Certification .......................................................................................... 4 1.1 Certification Bulletin Requirements .................................................................................................4 1.2 Cluster Environment .......................................................................................................................4 1.3 Certification Environment ...............................................................................................................4

2 Preparing Each Ceph Node .......................................................................................................... 5 2.1 Create Ceph Cluster ........................................................................................................................5 2.2 Configure Client VMs ......................................................................................................................5

3 Test Console Configuration ......................................................................................................... 6 3.1 Test Console Creation .....................................................................................................................6 3.2 SESCK Test Console Plugin Installation .............................................................................................6

4 SES Certification ....................................................................................................................... 8 4.1 SES Certification Project ..................................................................................................................8

5 Bulletin Submission .................................................................................................................. 14

6 Certification Kit Uninstall .......................................................................................................... 15 6.1 Options ........................................................................................................................................ 15 6.2 Usage ........................................................................................................................................... 15 6.3 Order of Uninstall ................................................................................................................................ 15

7 SES Certification Tests ............................................................................................................... 16 7.1 Component Check Test.................................................................................................................. 16 7.2 Edit Product/Report ...................................................................................................................... 16 7.3 Certification Kit Installs ................................................................................................................. 16 7.4 Verify Cluster Health ..................................................................................................................... 16 7.5 Verify Bonding Setup .................................................................................................................... 17 7.6 Multiple VM Stress Tests ............................................................................................................... 17 7.7 Other Tests ................................................................................................................................... 17 7.8 Get Test Logs ................................................................................................................................ 17


1 SUSE Enterprise Storage Certification An SES Certification bulletin will be issued after certification requirements are met.

1.1 Certification Bulletin Requirements

Testing companies should have access to SUSE Bulletin System (SBS)

Each SES node has a base System YES Certification

Note: The SUT should have an existing base YES Certification completed with the same major OS release

(and the same or earlier SP) as the SES base OS for the target version of SES, to fulfill this requirement.

Each SES node meets or exceeds the minimum SES hardware requirements

The SES cluster meets or exceeds the minimum cluster requirements

All listed SES nodes were part of the tested SES Cluster and passed SES certification testing

1.2 Cluster Environment

The minimum configuration for setting up a test environment are as follows:

SES Certification is performed on an existing or new SES cluster

The cluster contains a minimum of 4 OSD nodes

Each OSD node has a minimum of 2 hard disks

The cluster contains a minimum of 1 admin node

All nodes (OSD and Admin) plus KVM node has a clean install of SLES

Note: SLES 12 should only be installed on the first hard drive. All additional drives are used in the cluster.

Assure that drives are not part of a RAID configuration.

All nodes (OSD, KVM and Admin) have SES extension installed

All nodes (OSD, KVM and Admin) and VMs have NTP configured (TC can be used as NTP Server)

1.3 Certification Environment

The certification lab should have the following systems:

A Test Console (TC) installed from latest SUSE Yes Certification Test Kit (SCK 7.7 Recommended)

SUSE Enterprise Storage Certification Kit (SESCK) Plugin installed on TC

A KVM client with a minimum of 3 VMs running SLES 12 (Same SP level as cluster nodes)

Client VMs use SES block devices for OS and disk

DNS Server (TC can be configured as DNS Server if needed)

NTP Server (TC can be configured as NTP Server if needed)

Bonding with 802.3ad, active-backup or balanced-xor is required on all OSD nodes

Note: It is important that the TC, SES nodes and client VMs are all set up on the same network segment.


Refer to Figure 2-1 for a sample addressing scheme that matches up with the TC default settings.

2 Preparing Each Ceph Node 1. Confirm/Install SLES 12 (plus minimum required SP level) on each Ceph node (Admin and OSD).

2. Install SES extension on each Ceph node.

3. Run "ceph-deploy admin <node>" from the Admin Node to copy admin.keyring to each OSD node.

4. Configure the LAN settings on each Ceph node.

a. Configure DNS Services on each Ceph node.

b. Configure NTP Services on each Ceph node.

c. Add the options "iburst minpoll 5 maxpoll 5" to the NTP configuration of each node

2.1 Create Ceph Cluster

If a new SES cluster needs to be installed, please follow the SES documentation at:

http://docserv.nue.suse.com/documents/Storage_5/ses-deployment/html/

2.2 Configure Client VMs A client machine running SLES KVM hypervisor (same OS and SP as cluster nodes) with 3 SLES 12 VMs is

required.

Each VM hard disk must be provided by the SES cluster.

1. Configure SES to provide block devices and set up the VMs to use librbd to access the block device provided by

SES (as its OS and only disk) with documentation found at:

https://www.suse.com/documentation/ses-4/book_storage_admin/data/cha_ceph_libvirt.html

2. Set up each VM to use an SES block device as its OS (and only disk) with documentation found at:

http://docserv.nue.suse.com/documents/Storage_5/ses-admin/single-html/#cha.ceph.libvirt

Note: The KVM host should have SES keyring files copied to it, & any installed SES/Ceph packages should

come from the same SES repository as SES nodes, to ensure common version across environment.

3. Set up NTP on each VM.

a. VM IP addresses should be on the same network as Ceph nodes (refer to Figure 2-1).

http://docserv.nue.suse.com/documents/Storage_5/ses-deployment/html/

https://www.suse.com/documentation/ses-4/book_storage_admin/data/cha_ceph_libvirt.html

http://docserv.nue.suse.com/documents/Storage_5/ses-admin/single-html/#cha.ceph.libvirt


b. Add the options "iburst minpoll 5 maxpoll 5" to the NTP configuration of each VM.

Figure 2-1 Sample SES Network Configuration

3 Test Console Configuration If you do not have an existing Test Console, go to Section 3.1, otherwise go to Section 3.2 for instructions

on adding the SESCK Test Console plugin.

3.1 Test Console Creation

1. Install SLES 11 SP4 x86_64 OS on system to be used for TC.

2. Ensure the following patterns are installed:

Base System

32-Bit Runtime Environment

Help and Support Documentation (optional)

Minimal System (Appliance)

Gnome Desktop Environment

X Window System

File Server

DHCP and DNS Server

C/C++ Compiler and Tools

Print server

1. Download System Certification Kit ISO and mount it on the TC

2. (Optional) Download SLES 12 SP2 ISO to your TC so it can be used for PXE/Autoyast install later on.

3. Run “sck_install.sh” from your mount point and choose the option to Install Test Console

4. Follow the prompts.

5. The install will prompt you to configure the NIC cards on the TC; asks if you want to install a

DHCP/PXE server for which an ISO on the TC will be needed.

3.2 SESCK Test Console Plugin Installation


1. Get and mount SESCK ISO onto TC.

SES Kit located at: http://beta.suse.com/private/SCK/

username: testkit

password: sesck

2. From mount point run “sesck_install.sh”

3. Choose the option to Install SES Test Console.

Note: For complete TC installation instructions, refer to SystemTestKit-7.7-Linux-8_3_17.pdf

http://beta.suse.com/private/SCK/


4 SES Certification Create an SES Project on TC and run tests with the following steps.

4.1 SES Certification Project

1. Open Test Console by double clicking on the TestConsole icon on the TC desktop

2. Click New and select SUSE Enterprise Storage project type under SES folder

3. Save the project (may choose any name you’d like)


4. Modify project IP addresses to match Admin and OSD nodes if needed.

Note: If testing is performed on an existing cluster environment, make sure to modify the TestConsole IP

addresses highlighted in yellow below to match your network environment. Make sure that all nodes, VMs and TC are on the same subnet.

5. Before installing the certification kit, click Verify OSD Setup.

6. Install the SES Certification kit on each of the cluster nodes.


7. Double-click • Solution Info/OS check.

8. After Solution Info/OS check is run, the Edit Product/Report button becomes available. Click the

Edit Product/Report button and enter Company, information.

9. Next click the SES Systems tab and fill in all pertinent information.

Under the SES Systems tab, make sure the information entered into the yellow highlighted fields is

complete and accurate. Any changes after testing starts will abort all previous test results.

The Admin Node, Monitor Node and OSD Node Bulletin numbers must be listed prior to testing.

Bulletin Numbers can be found at www.suse.com/yessearch.

http://www.suse.com/yessearch


10. Next click on the Switch tab and fill in all pertinent information. As with the SES Systems tab, all

information entered into the yellow highlighted fields cannot be changed once testing begins.

11. Run the Verify Cluster Health test group. In order for this test to pass, OSD2 must be able to run

ceph commands. To enable this, either give adm role to OSD2 deepsea or after install copy keyring

files, conf. e.g. scp /etc/ceph/* osd2:/etc/ceph/.

12. To make sure each OSD node is in sync, watch the output of the command "ntpq -p". Ensure

number under "reach" increases to 377.


Note: If the Cluster Health test fails, open a terminal on OSD1 and type “ceph –s” to verify the cluster

health shows HEALTH_OK. It may take a few minutes to move from ERR to WARN to OK.

13. Verify Bonding Setup.

This section checks to make sure that the cluster is on a bonded network.

Note: Bonding is tested at the OSD node, not the switch. Make sure to have either 802.3ad, active-backup or balance-xor modes set up in the SES cluster prior to running test group.

14. Under the VM Tests section, modify KVM host and VM IP addresses if needed.


15. Install the SES Test Kit on each of the Virtual Machines.

16. Continue running the tests in order all the way to the end.

17. In “Multiple VM Stress Tests (12h)” section. You may double click on this parent folder to start all

tests under Stress Tests section at the same time (or right click and select “Run Test Group”). The

tests under each VM and under “OSD Node 2 Down” should all run concurrently as shown in the

picture below.

18. A pop up window will ask to confirm the number, size, and pool where RBD blocks will be created

for fio testing. Either select continue or make changes if necessary.

Note: If you select continue, the pool “libvirt-pool” should have been previously created (manually).

19. After 12 hr. Stress Tests are completed, run each test under Get Test Logs separately and in order.

20. When complete, proceed to Bulletin Submission.


5 Bulletin Submission 1. Click Edit Product/Report

2. Click the Verify button.

3. There should be no exceptions listed here, but if some exceptions are present explain them by selecting

them and clicking Edit Explanation.

Note: All exceptions should be explained.

4. Click Report on TC’s toolbar to create a submission zip file with test logs.

5. The resulting zip file (named earlier) will be on TC at /opt/suse/testKits/ses/results/

6. Copy the zip file to a USB drive.

7. Point your web browser to the SUSE Bulletin System (SBS) webpage.

8. Read the bulletin submission file (.zip) into SBS.

9. Click New Submission and browse to your bulletin submission file (.zip).

10. Click Upload.

11. The bulletin is now in the SBS system.


6 Certification Kit Uninstall The certification test kit resides in 2 directory structures:

1. /opt/suse/testKits/ses

2. /var/opt/suse/testKits/ses

The SES test kit is installed on the TestConsole, all OSD nodes as well as each Virtual Machine used in

the test cluster. In order to make cleanup easy and complete, SUSE Partner Engineering has created an

uninstall script. It is recommended that this script be used whenever a test kit uninstall is required.

6.1 Options

-h Print help text and exit

-t Uninstall mode: 1 = uninstall OSD nodes, Admin node and VMs, 3 = uninstall TestConsole

Note: When -t 1 mode is used, you must pass at least one node IP address. -t 3 does not require a node IP

-v node IP (admin node)

-w node IP (OSD1 node)

-x node IP (OSD2 node)

-y node IP (OSD3 node)

-z node IP (OSD4 node)

6.2 Usage

To uninstall the SES Certification Kit from the Admin node and 2 OSD nodes:

1. Change to the /opt/suse/testKits/ses/bin/ directory

2. Type ./sesck_uninstall -t 1 -v <admin IP address> -w <OSD1 IP address> -x <OSD2 IP address>

3. Ex: ./sesck_uninstall -t 1 -v 10.1.1.3 -w 10.1.1.4 -x 10.1.1.5 <Enter>

To uninstall the SES Certification Kit from the TestConsole:

1. Change to the /opt/suse/testKits/ses/bin/ directory

2. Type ./sesck_unistall -t 3 <Enter>

6.3 Order of Uninstall

1. Uninstall OSD Nodes

2. Uninstall VMs

3. Uninstall Admin Node

4. Uninstall TestConsole (always perform last)

Note: Uninstalling from TestConsole requires a reinstall before proceeding


7 SES Certification Tests This section explains the purpose of each test

7.1 Component Check Test

Test Objective: Gather system information about the SUT.

It is important that all hardware, drivers, BIOS/firmware updates which will be used during testing are

installed and/or enabled on the SUT before starting the Component Check Test. Any changes made to

SUT (hardware, drivers, BIOS/firmware updates) after this test is run will not be detected in the system

information screens. Direct access to the TC, and SUT systems are required (do not access remotely, do

not use SSH).

All subsequent tests will become available once the validate install test and component check tests are

completed.

7.2 Edit Product/Report

Objective: Enter cluster-related information used in testing

The Product and Report Information screen is the product information input tool for your Yes Certification

Bulletin. The hardware information entered on the following screens will be the same information which

appears on your Yes Certification bulletin. Please be accurate with all inputted information.

Note: Direct access to the TC, and SUT systems are required (do not access remotely, do not use SSH).

7.3 Certification Kit Installs

This section Installs the SES Certification Kit on all cluster nodes and verifies the SES Kit version. You can double-click the test group heading • Certification Kit Installs to install to all nodes

automatically, or install each node individually.

• Install Admin Node

• Install OSD Node 1




• Verify Kit Version

7.4 Verify Cluster Health

This section checks that each node is in sync with the NTP server, and verifies the cluster status is ok.

You can double-click the test group heading • Verify Cluster Health to run all tests automatically, or

you can run each test individually.

• Admin Time Sync

• OSD 1 Time Sync

• OSD 2 Time Sync

• OSD 3 Time Sync

• OSD 4 Time Sync

• Cluster Health


7.5 Verify Bonding Setup

This section checks to make sure that the cluster is on a bonded network.

Note: Bonding is tested at the OSD node, not the switch

You can double-click the test group heading • Verify Bonding Setup to run all tests automatically, or

you can run each test individually.

• OSD 1

• OSD 2

• OSD 3

• OSD 4

7.6 Multiple VM Stress Tests

This section runs a series of 12 hour stress tests against the cluster. This is done mainly by stressing the

VM disk which resides on the cluster, while also stressing the VM memory and CPU allocations. Time

sync is also performed to assure VMs stay in sync with the NTP server during testing.

You can double-click the major test group heading • Multiple VM Stress Tests to run all tests

automatically, or you can run minor test groups or each test individually. It is recommended to run all tests

at once to save time and thoroughly stress the system.

• SLES VM1 Stress Tests (Minor Test Group)

• Memory / CPU Test

• Hard Disk Test

• Time Sync Test

7.7 Other Tests

3 hours into the stress test, the OSD 2 node is taken down and the cluster is tested for failover. When the

OSD 2 node is brought back online, the cluster is tested for a successful rebuild, after which the • fio –

Kernel Module test is run. These tests should be run as part of the major test group • Multiple VM

Stress Tests but can be run separately for troubleshooting.

• Other Tests

• Failover / Rebuild

• fio – Kernel Module

7.8 Get Test Logs

This section should not be run until all previous tests are complete. This section gathers the test logs from

each cluster node, making the available for bulletin submission.

You can double-click the test group heading • Get Test Logs to gather all logs automatically, or you can

gather each log individually.

• Get Admin Test Logs

• Get OSD 4 Test Logs




• Get VM3 Test Logs



suse yes certification test kit · suse yes certification test kit 1.1 august 15, 2017 suse®...

Documents