SUSE Enterprise Storage™

on HPE Apollo SystemsImplementation Guide

Guide

Guidewww.suse.com

Data Center Modernization

2

Data Center Modernization GuideSUSE Enterprise Storage on HPE Apollo Systems Implementation Guide

OverviewThis guide is intended to provide a step-by-step process to implement SUSE Enterprise Storage™ 3 on HPE Apollo 4510 or 4200 series systems. We recommend that you read the document in its entirety before beginning the process, in order to identify information that you will need during the installation, such as IP ranges and NTP servers. A pre-installation survey is provided at the end of the document to assist in you capturing this information.

Upon completion of the steps in this document, a functioning SUSE Enterprise Storage cluster will be available for provisioning as described in the documentation found here: www.suse.com/documentation/ses-3/index.html

This document is intended to be utilized in conjunction with the general architecture and recommendations found in the SUSE Enterprise Storage Architectural Overview with Recom men­dations1 paper.

Hardware ConfigurationThis is a representative configuration used for validation and testing. The hardware more than meets the recommendations in the architectural overview document. There are two types of OSD nodes configured nearly identically, with the exception of the processors and RAID controllers. The Apollo 4510 units were configured with more powerful processors to allow for future ex-pansion and the expectation of using two P440 RAID controllers

to enable all drive slots to be occupied. When building a new cluster, it is best to have a homogenous node configuration.

OSD Node (Qty 6)HPE Apollo 4200 (qty 3)

2x E5-2680 v3320GB RAMP840 RAID controller with 2GB Cache w/Battery backup

4 480GB Write Intensive SATA SSDs24 6TB 7.2k SATA Drives

B140i RAID controller2 256GB M.2 SSDs

1 Dual Port Mellanox 40GbE adapter

__________

1 www.suse.com/docrep/documents/w3lcomzhs7/suse_enterprise_storage_architectural_overview_with_recommendations_guide.pdf

3www.suse.com

HPE Apollo 4510 (qty 3)2x E5-2690 v3320GB RAMP440 RAID controller with 4GB Cache w/Battery backup

4 480GB Write Intensive SATA SSDs24 6TB 7.2k SATA Drives

B140i RAID controller2 256GB M.2 SSDs

1 Dual Port Mellanox 40GbE adapter

Monitor Node (Qty 3)HPE ProLiant DL360 Gen 9

1x E5-2660 v332GB RAMP440 RAID Controller with 4GB Cache w/Batter backup

2x 80GB Mixed Use SSD1 Dual Port Mellanox 40GbE adapter

Switching Infrastructure2x HPE Flex Fabric 5930-32QSFP+

Hardware Preparation

NetworkFirst, properly cable and configure each node on the switches. Ensuring proper switch configuration at the outset will prevent networking issues later. The key configuration items include cre-ating the IRF topology (stacking) and LACP groups, and enabling jumbo frames. Each aggregation group needs a unique number, planned ahead of time. It is also recommended that you disable the spanning tree on the ports utilized for storage. Configure IRF as described in the IRF Configuration Guide for the HPE FlexFabric 5930 Switch Series2.

We recommend irf mac-address persistent always be set

To configure an LACP group for switch 1 port 1 and switch 2 port 1 and to enable jumbo frame support, perform the following commands:

System-viewInterface Bridge-aggregation 1Link-aggregation mode dynamicquitinterface Ten-GigabitEthernet 1/0/1port link-aggregation group 1jumboframe enable 9100quitinterface Ten-GigabitEthernet 2/0/1port link-aggregation group 1jumboframe enable 9100quitsave

Repeat these steps for each aggregation group required.

Create at least 1 VLAN for cluster communication. In this example, we are using VLAN 3001 for the cluster traffic.

system-viewvlan 3001name ceph-clusterdescription vlan for ceph cluster back end communicationquitsave

Assign VLANs to ports. The configuration below assumes a port based VLAN of 1 (default):

system-viewinterface bridge-aggregation 1port link-type hybridport hybrid vlan 1 untaggedport hybrid vlan 3001 taggedquitsave

__________

2 http://h20564.www2.hpe.com/hpsc/doc/public/display?docId= c04567141

4

Data Center Modernization GuideSUSE Enterprise Storage on HPE Apollo Systems Implementation Guide

Configure the B140i RAID first by booting into system utilities, going to the array controller, and starting the Smart Storage Administrator.

Create an array using both M.2 drives shown as a RAID-1.

Next, configure the SSD journals. The drives should be attached to the P440.

Each drive should be configured as a RAID-0.

5www.suse.com

Next, configure the spinning drives the same way, each as a RAID-0.

The final step in the RAID configuration is to ensure that the cache is set to have a 90% write setting.

Step-by-Step

PrerequisitesPre-plan the IP range to be utilized. Then create a single stor-age subnet where all nodes, gateways and clients will connect. In many cases, this might entail using a range larger than a standard /24. While storage traffic can be routed, it is generally discour-aged, to help ensure lower latency.

6

Data Center Modernization GuideSUSE Enterprise Storage on HPE Apollo Systems Implementation Guide

Setup DNS A records for the OSD and Mon nodes.

Decide on subnets and VLANs to be utilized and configure the switch ports accordingly. Guidance is provided in the SUSE Enterprise Storage Architecture Overview with Recommendations document found here: www.suse.com/docrep/documents/ w3lcomzhs7/suse_enterprise_storage_architectural_overview_ with_recommendations_guide.pdf

Subscription Management Tool: This service provides a local mir-ror of the SUSE® repositories, allowing for rapid software deploy-ment and updating. More information can be found here: www.suse.com/documentation/sles-12/book_smt/data/book_smt.html

Configure all nodes of similar types identically. With the RAID controllers, this is especially important because it greatly simpli-fies drive configuration.

Operating System DeploymentThe second major step is to install the operating system. In the reference configuration, there are 2 SSD drives for the operating system, 24 6TB near-line SAS drives for data, and 4 SSD drives for the journals. All nodes of the same role should be configured the same.

Apollo installs with OS drives attached to B140i RAID controller require the use of a driver update disk. The driver update disk can be obtained from hpe.com by searching the support site for B140i driver update SUSE.

The driver update disk should be downloaded, uncompressed and stored on an http accessible location or on a USB drive. In the example below, it is stored on a web server and utilized as a virtual media source.

To use the image, during the installation boot, press ‘e’ to edit the command line of installation and add dud=1 brokenmodules=ahci dd to the end of the linuxefi command line.

In this case, we utilized HP Virtual Media and attached the driver update disk image file.

On the driver update screen, select the correct device. In this case, select USB Disk, HP Virtual Media.

7www.suse.com

After the screen returns, click back and then the installation will start.

When deploying the operating system, be sure to utilize only the correct device. This is the RAID-1 of 256GB SSD drives (80GB SSD drives for monitor nodes). In this example, the device enabled by using the driver update disk shows up last: sdad Perform a basic network configuration during the installation.

This will include the host name, domain name and IP address. If an SMT server has been set up, select Local Registration

Server on the Registration page and enter the URL for the SMT server.

Minimal install with SUSE Enterprise Storage repository is selected.

Deselect x­windows and Gnome.

After the installation is complete, re-write the /etc/udev/rules.d/70-persistent-network-rules file to have the ports set up identically on each node. After the file has been modified, each node will need to be rebooted and then have the final network configuration put in place.

Verify connectivity and then run zypper up to ensure that there no further updates.

The table below shows the IP assignments used for this exercise. These can and should be adjusted as needed for each individual deployment.

SUSE Enterprise Storage DeploymentThe third major step is to deploy Ceph as described in the docu-mentation located at: www.suse.com/documentation/ses-3/

On sesadmin perform the following:Configure ntp using YaST®:

yast <return> ->Network Services ->NTP Configuration[Start NTP Daemon] Now and on BootAdd->Server->your preferred NTP Server

Configure sudoers:visudoAdd the following to the end of the file:cephadm ALL = (root) NOPASSWD:ALL

On each node, create the cephadm user and set the password:useradd -m cephadm && passwd cephadm

Create and distribute the ssh-key for the cephadm user. From sesadmin:

su – cephadmssh-keygenssh-copy-id cephadm@osdnode1Repeat for each node

Copy /etc/sudoers to each node:sudo scp /etc/sudoers root@osdnode1:/etc/sudoersRepeat for each node

Install ntp on each node:sudo zypper in ntp yast2-ntp-clientRepeat for each node

Configure ntp on the admin node.

Copy /etc/ntp.conf to each node:sudo scp /etc/ntp.conf root@osdnode1:/etc/ntp.confRepeat for each node

On each node: sudo systemctl enable ntpd sudo systemctl start ntpd

Node

Public Front End IP (VLAN 1)

Cluster Back End IP (VLAN 3001)

sesadmin 192.168.124.20 192.168.100.20

monnode1 192.168.124.21 192.168.100.21

monnode2 192.168.124.22 192.168.100.22

monnode3 192.168.124.23 192.168.100.23

osdnode1 192.168.124.31 192.168.100.31

osdnode2 192.168.124.32 192.168.100.32

osdnode3 192.168.124.33 192.168.100.33

osdnode4 192.168.124.34 192.168.100.34

osdnode5 192.168.124.35 192.168.100.35

osdnode6 192.168.124.36 192.168.100.36

8

Data Center Modernization GuideSUSE Enterprise Storage on HPE Apollo Systems Implementation Guide

Install ceph on all nodes:sudo zypper in ceph

Install ceph-deploy on the admin node:sudo zypper in ceph-deploy

Disable IPv6:Edit /etc/sysctl.conf and add these lines to the bottom of the file:net.ipv6.conf.all.disable_ipv6 = 1net.ipv6.conf.default.disable_ipv6 = 1net.ipv6.conf.lo.disable_ipv6 = 1

Copy /etc/sysctl.conf to all nodes and reboot them:sudo scp –p /etc/sysctl.conf root@osdnode1:/etc/sysctl.confRepeat for each node

Run ceph-deploy install for all nodes:ceph-deploy install sesadmin osdnode1 osdnode2 osdnode3 osdnode4 osdnode5 osdnode6 monnode1 monnode2 monnode3

Set up the monitor nodes:ceph-deploy new monnode1 monnode2 monnode3

Modify ceph.conf to reflect the networks. In the [global] section, add/edit the following:

public network = 192.168.124.0/24cluster network = 192.168.100.0/24

Make sure the firewall is off on all nodes:sudo /sbin/SuSEfirewall2 statusRepeat on each node

Create the initial monitor service:ceph-deploy mon create-initial

Next, prepare the OSD Nodes using the ceph-deploy osd create command. The first device (in this example, sdd) is the data drive and the last device (sdb) is the journal device.

ceph-deploy osd prepare osdnode{1..6}:sd{e..j}:sdaceph-deploy osd prepare osdnode{1..6}:sd{k..p}:sdbceph-deploy osd prepare osdnode{1..6}:sd{q..v}:sdcceph-deploy osd prepare osdnode{1..6}:sd{w..z}:sddceph-deploy osd prepare osdnode{1..6}:sdaa:sddceph-deploy osd prepare osdnode{1..6}:sdab:sdd

ceph-deploy osd activate osdnode{1..6}:sd{e..j}1:sdaceph-deploy osd activate osdnode{1..6}:sd{k..p}1:sdbceph-deploy osd activate osdnode{1..6}:sd{q..v}1:sdcceph-deploy osd activate osdnode{1..6}:sd{w..z}1:sddceph-deploy osd activate osdnode{1..6}:sdaa1:sddceph-deploy osd activate osdnode{1..6}:sdab1:sdd

Deploy the admin node(s):ceph-deploy admin sesadmin

Install Romana on the admin nodes and set the monitor nodes as admin nodes:

As cephadm:ceph-deploy admin monnode1 monnode2 monnode3

As root:zypper in romanacalamari-ctl initializesu - cephadmceph-deploy calamari connect --master sesadmin osdnode1 osdnode2 osdnode3 osdnode4 monnode1 monnode2 monnode3

The Romana interface can now be accessed at: http://sesadmin

Basic SUSE Enterprise Storage Cluster ValidationAfter basic configuration of the SUSE Enterprise Storage cluster is complete, perform a basic set of checks to ensure that the cluster is operating as expected.

9www.suse.com

From the admin node:ceph statusceph osd pool create test 4096rados bench –p test 300 write --no-cleanuprados bench –p test 300 seq

After validation is complete, remove the test pool:ceph osd pool delete test test –yes-i-really-really-mean-it

Cluster Baseline Performance MeasurementsIt is important and helpful to capture the baseline performance measurements of your SUSE Enterprise Storage cluster. These numbers should include data from rados bench, rbd device throughput to a single node, and, if deployed, a baseline for RADOS gateways.

For further information on SUSE Enterprise Storage, please visit: www.suse.com/products/suse-enterprise-storage/

Pre-installation Survey General Network Information

– DNS server(s): • DNS1: • DNS2:

– NTP servers: • NTP1: • NTP2: • NTP3:

– SMT server: – VLANs

• Cluster Public - Tag - Port Based or Tagged - IP range - Netmask - Default GW

• Cluster Backend - Tag - Port Based or Tagged - IP range - Netmask - Default GW

• Service Network (front end for gateways such as iSCSI, RGW) - Tag - Port Based or Tagged - IP range - Netmask - Default GW

• iLO - IP range or DHCP assignment - Netmask - Default GW

Monitor Nodes – VLANs

• Cluster Public - IP range

• Cluster Backend - IP range

– Hostname format • For example, sesmon{1..3}

– iLO • IP range or DHCP • Netmask • Default GW

Storage Nodes – VLANs

• Cluster Public - IP range

• Cluster Backend - IP range

– Hostname format • For example, sesosd{1..10}

– What devices and configuration for the OS? • Recommended M.2 mirrored or low-cost mirrored SSD

– SSDs to be used for journal devices? – iLO

• IP Assignment

10

Data Center Modernization GuideSUSE Enterprise Storage on HPE Apollo Systems Implementation Guide

RadosGW Nodes – VLANs

• Cluster Public - IP range

• Cluster Backend - IP range

– Hostname format • For example, sesrgw{1..4}

– iLO • IP range

iSCSI GW Nodes – VLANs

• Cluster Public - IP range

• Service Network - IP range

– Hostname format • For example, sesiscsi{1..4}

– iLO • IP range

Host IP Assignments

Hostname Node Type Cluster Backend Cluster Public Service Network iLO

Monnode1 Monitor-example 192.168.100.21 192.168.124.21 172.16.250.21

Iscsigw1 iSCSI-example 192.168.123.91 10.10.1.91 172.16.250.91

11www.suse.com

Physical TopologyPlease list the “buckets” for the CRUSH map to be aware of. For example, devicetype, server, chassis, rack, row, room, bldg., campus.. Please outline CRUSH rulesets and related information. Auth Definitions

Pool Definitions

Type Name Password Purpose Quota

Swift Test/swiftuser 123freddy Example: For XYZ app to utilize swift storage pool 100G

Name Rule Set Protection Scheme Pg_num ACL

Name Information

ssd_only Example rules

sata_only

Name Description

264-000026-001 | 11/16 | © 2016 SUSE LLC. All rights reserved. SUSE, the SUSE logo and YaST are registered trademarks, and SUSE

Enterprise Storage is a trademark of SUSE LLC in the United States and other countries. All third-party trademarks are the property

of their respective owners.

Contact your local SUSE Solutions Provider, or call SUSE at:

1 800 796 3700 U.S./Canada

1 801 861 4500 Worldwide

SUSE

Maxfeldstrasse 5

90409 Nuremberg

Germany

RBD Definitions

www.suse.com

Name Size Pool