deploying carrier grade openstack on theadvdownload.advantech.com/productfile/pis/pac-6009... ·...

Copyright 2017 Advantech Co. Ltd. All rights reserved.

APPLICATION NOTE

REVISION 1.0 DATE 2017/06/29

DEPLOYING CARRIER GRADE OPENSTACK ON THE

PAC-6009 CARRIER GRADE BLADE SERVER

USING WIND RIVER TITANIUM CLOUD 16.10.

Combing the Advantech PAC-6009 and Wind River Titanium Cloud Virtualization software enables you to deploy a 99.999% available Network Function Virtualization Infrastructure for your

mission critical applications..

Copyright 2017 Advantech Co. Ltd. All rights reserved. of 1

Revision History

Date [mm/dd/yyyy]

Revision Modifications

06/29/2017 1.0 First version

© Copyright 2017 – Advantech Co., Ltd.

All Rights Reserved

Advantech Co., Ltd. reserves the right to make improvements in the products described in this manual at any time without notice. No part of this manual may be reproduced, copied, translated or transmitted in any form or by any means without the prior written permission of Advantech Co., Ltd. Information provided in this manual is intended to be accurate and reliable. However, Advantech Co., Ltd. assumes no responsibility for its use, nor for any infringements of the rights of third parties, which may result from its use.


Description This application note describes how to deploy Carrier Grade OpenStack on the PAC-6009 Carrier Grade Blade Server using Wind River Titanium Cloud R3 16.10. It is based on the standard process that Wind River provides and highlights any additional steps required.

We appreciate your input

Please let us know if you consider any aspect of this application note needs improving or

correcting. We appreciate your valuable input in helping make our products and

documentation better.

Please send all such comments in writing to: [email protected]

Acknowledgements

Xeon and Intel are trademarked by Intel Corp.

Titanium Cloud is a registered trademarks of Wind River.

All other product names or trademarks are properties of their respective owners.

mailto:[email protected]


1. WIND RIVER TITANIUM CLOUD

1.1 Carrier Grade for Your Virtual Networks

Titanium Cloud is a fully integrated, ultrareliable, and deployment-ready portfolio of virtualization software platforms, that enables service providers to deploy virtualization services faster, at lower cost and guaranteed uptime

1.2 Key Benefits

When service uptime is critical for profitability, Titanium Cloud ensures virtualized services run when, where and how they need to by providing the:

Flexibility to scale your services up, down, in, and out—rapidly and efficiently, deploying new services dynamically, when and where they are needed

Performance to maximize the number of subscribers you support on each server, and minimize operating costs

Carrier grade reliability you need to keep your services up—always

Titanium Cloud is suited to a wide variety of virtualized network functions (VNFs), including:

o Virtual EPC o Cloud-RAN o Appliances o Virtual packet inspection o Content delivery systems o Virtual gateways


1.3 Technical Features

Titanium Cloud includes the critical run-time components and lifecycle development tools, services, and developer support needed to successfully build and deploy a virtualized network running multiple VNs.

TL 9000 design methodology to guarantee six nines reliability

Full 10Gbps line rate accelerated virtual switching running unmodified guest applications

10 µs deterministic interrupt latency to support virtualization in the most stringent customer-provided equipment (CPE) and access sections of the network

Accelerated live VM migration, including the “share nothing” model—40% faster than enterprise software

Detection of failed controllers, hosts, or VMs 60x faster than enterprise Linux; automatic recovery, with no single point of failure

Fully integrated telecom-grade security, including complete AAA access control

Involvement in trials with the European Telecommunications Standards Institute (ETSI) and other standards bodies

Titanium Cloud is a complete, pre-integrated portfolio of virtualization software platforms:

Linux OS plus 700+ Carrier Grade patches

KVM with performance and reliability extensions

OpenStack

Carrier grade cloud management and telco middleware

Carrier grade accelerated vSwitch

Ceph storage


2. EQUIPMENT SETUP

2.1 Hardware used

The Titanium Cloud setup comprises two controller nodes for full high availability (HA) support. We will be using two Advantech MIC-8304C Single Intel® Xeon® Processor D-1500 blades as our controller nodes.

Our deployment will leverage share CEPH storage, for the Titanium Cloud storage nodes we will be using two Advantech MIC-8304C Single Intel® Xeon® Processor D-1500 blades equipped with two 2.5” SSDs for the CEPH OSD and Journaling.

Titanium Cloud compute nodes can be scaled out from a minimum of two nodes. In our example we will be using two Advantech MIC-8303C Dual 16-core Intel® Xeon® Processor D-1500 blades and two Advantech MIC-8304C Single Intel® Xeon® Processor D-1500 blades.

This allows us to setup both a Low Latency Compute (LLC) cluster and Standard Compute cluster.

Each processor has two 10GbE and two 1GbE backplane ports that are switched by the PAC-6009 integrated switch blades.

All hardware management is provided by the Advantech Shelf Manager solution which is integrated within the switch blade, the management access to the Shelf Manager is provided through RAS. All switch configuration access is also done via the Shelf Manager.

For more information about the PAC-6009 please review the Product brief

http://www2.advantech.com/nc/newsletter/NCG/DOC/Packtetarium_XLc_Product_Brief.pdf

The installation leverages the Advantech Remote Evaluation Service (RES) lab and, consequently, there is a Remote Access Server (RAS) providing external access to the described cluster.

The RAS server will act as Jump Server providing access to the OAM network and provides installation of controller-0. The Jump Server is connected to the PAC-6009 over a 10GbE uplink port for the 10GbE data plane switch and this connection also allows us to provide a gateway for the OpenStack external networks. A secondary connection is made to one of the 1GbE management ports of the PAC-6009 to access the system management.

http://www2.advantech.com/nc/newsletter/NCG/DOC/Packtetarium_XLc_Product_Brief.pdf


2.2 Network Architecture

2.3 Switch configuration

VLANs on the switches are configured as below:

Upper Switch

1GbE Switch o MGMT / PXE network is an isolated Layer2 network

10GbE Switch o VLAN ID 2000 (Infrastructure Network)

Lower Switch

Bottom 1GbE Switch o Isolated PAC-6009 Chassis management and LAB services

Bottom 10GbE Switch o VLAN ID 99 (OAM Network) o VLAN ID 1100-1199 (Provider Network)

1GbE1GbE

Controller 0-1

1G

1G

10G

10G

OAM VLAN 99

10GbE10GbE

RASJump Server

RASJump Server

Upper Switch blade

Lower Switch blade

MGMT / PXE

LAB

Compute 0-3

1G

1G

10G

10G

INFRA VLAN 2000

DATA1 VLAN 1100-1199

MGMT / PXE

LAB

LAB

OAM VLAN 99

Storage 0-1

1G

1G

10G

10G

MGMT / PXE

LAB

INFRA VLAN 2000 INFRA VLAN 2000

10G10G

OAM LAG

1G

DATA1LAG

10G10G


3. PREPARING THE PAC-6009

All configuration settings will go through the active Shelf Manager in the system. To access the active Shelf Manager please connect one of the management ports to your management network. In our scenario we connected the upper switch management port to RAS.

SSH to the Management IP that was set on the system, if no Management IP was set on the system please find the instructions in the PAC-6009 user manual how to set this.

After we login to the active Shelf Manager there here is a snapshot of the different commands we will use

CLI – this gives us access to the command line interface of the hardware mgmt

solclient – this will give us access to the serial console of all blades

telnet 192.168.1.1 – this will give us access to the upper switch 1GbE management

telnet 192.168.1.2 – this will give us access to the lower switch 1GbE management

telnet 192.168.1.251 – this will give us access to the upper switch 10GbE management

telnet 192.168.1.252 – this will give us access to the lower switch 1GbE management

3.1 Setting up switch configuration

3.1.1 Isolate the 18 blades from the PAC-6009 Management network

1. Access the upper 1GbE switch # telnet 192.168.1.1

2. Use the user credentials provided in the PAC-6009 switch manual to login 3. Enable switch configuration

(ESM-9578) >en 4. Enter vlan database

(ESM-9578) #vlan database 5. Create vlan 10

(ESM-9578) (Vlan)#vlan 10 6. You can see the vlan database by using the command “show vlan” 7. Exit the vlan database

(ESM-9578) (Vlan)#exit 8. Enter the configuration mode

(ESM-9578) #configure 9. Select all 18 backplane interfaces, for more details of the switch port configuration

please see the PAC-6009 switch user manual. (ESM-9578) (Config)#interface 1/0/1-1/0/18

10. Include interfaces in vlan 10 (ESM-9578) (Interface 1/0/1-1/0/18)#vlan participation include 10

11. Enable vlan tagging for vlan 10 (ESM-9578) (Interface 1/0/1-1/0/18)#vlan pvid 10

12. Exit out of the interface selection and configuration mode (ESM-9578) (Interface 1/0/1-1/0/18)#exit (ESM-9578) (Config)#exit

13. Quit the session, when asked to save configuration you can choose yes if you would like to retain the configuration upon switch reset.


(ESM-9578) #quit


3.1.2 Configure VLAN upper 10GbE switch



(Advantech Switching) >en 4. Enter vlan database

(Advantech Switching) #vlan database 5. Create vlan 2000

(Advantech Switching) (Vlan)#vlan 2000 6. You can see the vlan database by using the command “show vlan” 7. Exit the vlan database

(Advantech Switching) (Vlan)#exit 8. Enter the configuration mode

(Advantech Switching) #configure 9. Select all 18 backplane interfaces, for more details of the switch port configuration

please see the PAC-6009 switch user manual. (Advantech Switching) (Config)#interface 1/0/1-1/0/18

10. Include interfaces in vlan 2000 (Advantech Switching) (Interface 1/0/1-1/0/18)#vlan participation include 2000

11. Enable vlan tagging for vlan 2000 (Advantech Switching) (Interface 1/0/1-1/0/18)#vlan tagging 2000

12. Exit out of the interface selection and configuration mode (Advantech Switching) (Interface 1/0/1-1/0/18)#exit (Advantech Switching) (Config)#exit

13. Quit the session, when asked to save configuration you can choose yes if you would like to retain the configuration upon switch reset. (Advantech Switching) #quit


3.1.3 Configure VLAN lower 10GbE switch



(Advantech Switching) >en 17. Enter vlan database

(Advantech Switching) #vlan database 18. Create vlan 99 & 1100-1199

(Advantech Switching) (Vlan)#vlan 99,1100-1199 19. You can see the vlan database by using the command “show vlan” 20. Exit the vlan database

(Advantech Switching) (Vlan)#exit 21. Enter the configuration mode

(Advantech Switching) #configure 22. Select all 18 backplane interfaces, for more details of the switch port configuration

please see the PAC-6009 switch user manual. (Advantech Switching) (Config)#interface 1/0/1-1/0/18

23. Include interfaces in vlan 99 & 1100-1199 (Advantech Switching) (Interface 1/0/1-1/0/18)#vlan participation include 99,1100-1199

24. Enable vlan tagging for vlan 99 & 1100-1199 (Advantech Switching) (Interface 1/0/1-1/0/18)#vlan tagging 99,1100-1199

25. We use two ports on the switch for DATA1LAG, 1/0/49 & 1/0/50. We will use port-channel 1/1/3 for this lag, we need to add the physical ports to the channel. (Advantech Switching) (Interface 1/0/49,1/0/50)#addport 1/1/3

26. Include port-channel in vlan 1100-1199 and enable tagging (Advantech Switching) (Interface 1/1/3)#vlan participation include 1100-1199 (Advantech Switching) (Interface 1/1/3)#vlan tagging 1100-1199

27. We use two ports on the switch for OAMLAG, 1/0/47 & 1/0/48. We will use port-channel 1/1/4 for this LAG, we need to add the physical ports to the channel, add vlan 99 and enable tagging. (Advantech Switching) (Interface 1/0/47,1/0/48)#addport 1/1/4 (Advantech Switching) (Interface 1/1/4)#vlan participation include 99 (Advantech Switching) (Interface 1/1/4)#vlan tagging 99

28. Exit out of the interface selection and configuration mode (Advantech Switching) (Interface 1/0/1-1/0/18)#exit (Advantech Switching) (Config)#exit

29. Quit the session, when asked to save configuration you can choose yes if you would like to retain the configuration upon switch reset. (Advantech Switching) #quit


3.2 Setting up board boot configuration

In order to get Titanium Cloud working with each of the blades, we require the first boot device to be set to storage and detected by Linux as SDA. The second boot device will be set to PXE. For this we have to access the BIOS of each blade and configure it correctly. It’s important that unconfigured node’s SDAs are left empty, which allows the booting of all unconfigured nodes to fall back to PXE.

The below steps can be repeated for each blade

1. On the shelf manager access the CLI # cli

2. In the CLI get an overview of all blades in the system CLI> status

Currently all compute blades are deactivated, in M1 state. Both Switch management blades are powered on, in M4 state.

3. Set the board to boot into the BIOS CLI> bootopt 0x82 bios

4. Activate the board to start booting CLI> activate 0x82


5. Either exit the CLI or open a second connection to the system management, start the solclient # solclient

6. Select the blade, the nr links to the site in the status overview. Type ~1<enter> to select the first blade.

7. When the board is finished booting it will automatically enter the BIOS, please select Post & Boot


8. Enter CSM Parameters and make sure that Network has Legacy enabled and you see the sub menu Network Device BBS Priorities.

If Legacy wasn’t enabled, you will need to enable it. Afterward you need to Save & Exit and Select Save Changed and Exit. Afterwards back in CLI deactivate the blade. CLI> deactivate 0x82 When the blade is back at state M1 go back to step 3 and repeat the process, you will now notice that we can see Network Device BBS Priorities.


9. Set the first Boot option to an Ethernet device, starting with IBA. The second Boot option can be the local storage.

To set the boot order of all Ethernet devices, please enter the PXE Function per port. It is strongly recommend to disable all PXE ROMs options except PXE of 1G Port 0.


10. It is also recommended to enable VT-D and SR-IOV only on the compute blades before getting started, even though Titanium Cloud has IOMMU (VT-D) only enabled on compute nodes we only enable it on the blades that require it. The use of SR-IOV it is optional usage and will not be used during this demonstration.

Platform > Virtualization

11. Go to Save & Exit and Select Save Changed and Exit. 12. Go back to CLI and deactivate the board for now

CLI> deactivate 0x82 13. Repeat these steps for each blade you would like Titanium Cloud to discover.


4. TITANIUM CLOUD CONTROLLER-0 INSTALLATION

Titanium Cloud will both provision and manage the Openstack via it’s controllers, therefore the first step is to install Controller-0. Because this is the first node of the Titanium Cloud deployment this will have to be done manually.

We can choose to either install via USB or via a temporary PXE server, for the purpose of this Application Note we will leverage PXE

4.1 Prepaparing the PXE server

We already have fully functional DHCP, TFTPD and HTTP services up and running on the Jump Server. Titanium Cloud ISO image (provided by Wind River) provides several scripts to install the files to PXE boot.

First we mount the iso image somewhere on the jump server, after this is completed we run the following command at the mount location.

./pxeboot_setup.sh -u http://<DHCPSERVER_IP>/tic3 -t /<tftpbootdirectory>/tic3

The command will install all the files required for PXE install under /<tftpbootdirectory>/tic3 and configure all the files correctly, including the location where the Centos based installer can find the repository files over http.

Because the script does not know where your HTTP files would be stored we will need to copy (or create symbolic link) to the location that leads to http://<DHCPSERVER_IP>/tic3

We will have to modify the TFTPD configuration file to use the newly created /<tftpbootdirectory>/tic3 directory as base directory.

4.2 Installing controller-0

Activate the node via the ShMC CLI, use solclient to access the serial console. Upon PXE boot you will be present with a menu like shown below. For this application node we will install controller-0 via “Serial Controller Install”.


The installation is fully automated and will install on /dev/SDA. After completion the node will reboot, boot from /dev/sda.


4.3 Preparing the Controller-0

After installation is completed and the node is rebooted we are presented with the Linux login screen, here is where we can login with the default username and password.

Username: wrsroot

Password: wrsroot

At the first boot we will have to change the wrsroot password straight away, first we enter again the current password wrsroot after which we can setup your new password.

After we are logged in we need to copy all patches and the license file (provided by Wind River) to controller-0. We can do this either via USB drive or over IP.

We will put all patches and license file in /home/wrsroot/

4.4 Patching the Controller-0

We patch controller-0 to the latest patch available from Wind River, future patches will be applied via the Titanium Cloud interface

sudo sw-patch upload /home/wrsroot/*.patch

sudo sw-patch apply --all

sudo sw-patch install-local

After all patches are installed please reboot the controller-0

sudo shutdown -r now


4.5 Configuration Controller-0

Now that we have fully patched the system to the latest patch and have the license file uploaded on the controller-0 we are ready to begin the configuration.

Before we start please note the following

First 1 & 10G device are connected to the top switch

Second 1 & 10G device are connected to the bottom switch

In this case because controller-0 is a MIC-8304C blade we have the following network ports.

1G:

First: enp6s0f0 (MGMT/PXE)

Second: enp6s0f1 (Not used)

10G:

First enp3s0f0 (INFRA & DATA0)

Second enp3s0f1 (OAM & DATA1)

To start the configuration we start the following command

sudo config_controller

Firstly we have to set the time correctly for controller-0, even though Titanium Cloud uses NTP to time synchronize all blades before this is fully functional controller-0 time is the master clock.


After this we will have to select what type of storage to be used for the Titanium Cloud deployment. We planned for shared storage which means we select ceph.

Now that we select ceph the configuration script will calculate the size for each of the partitions on controller-0 local storage. We follow the script’s default settings to simplify the process.

Because we do not use Software Defined Networking we will tell the script to not enable it.


Our setup does not require separation of PXE and MGMT networks via VLAN, also we are not using link aggregation. We select the 1st 1G port which is enp6s0f0 and keep the rest of the settings as default.

Wind River Titanium Cloud 16.10 does not support the Advantech Board management control network this will be supported from Wind River Titanium Cloud Release 4. We therefore tell the script to not enable a Board Management Control network.


Because we chose shared storage earlier in the script we will have to enable an infrastructure interface without link aggregation, which as per the diagram will use a VLAN and is the 1st 10G interface enp3s0f0.

For the External OAM Network we will not use link aggregation on the controller nodes, we will select the 2nd 10G interface enp3s0f1 and set the VLAN 99 as per the network diagram. We leave all other settings as default and make sure that the gateway address 10.10.10.1 can be reached by the controller-0 blade.


Finally we keep default settings for DNS, NTP, https, custom firewall. We set the wrsroot password age to ‘0’ to avoid any password change in our lab setup. And finally we set the admin password for the Openstack.

An overview will be presented by the script of all your settings, please review and if correct press ‘y’ to apply the configuration. This process will take a while and after completed you can reach http://10.10.10.2 over the OAM interface and login with the admin password that was created. It is strongly recommended not to reboot the controller-0 until you have fully functional controller-1.

http://10.10.10.2/


5. DEPLOYING TITANIUM CLOUD

Now that we have installed and configured controller-0, the rest of the configuration can be done via the Titanium Server web interface. It’s also possible to do the complete configuration via console commands however the web interface is more intuitive. Please login to the Titanium Server web interface at http://10.10.10.2/ with username admin and the password you set during the configuration of controller-0

5.1 Installing Controller-1

Now that we have logged in we can boot the node that is set up to be Controller-1, because we set the bios settings before the node will be recognized by the Titanium Cloud provisioning mechanism. From this point on the node will be in a continuous PXE boot loop until it’s assigned a role by Titanium Cloud. We double check the blade progress via the console where we find the below PXE boot menu.

http://10.10.10.2/


During this we will also see the node in the Host Inventory menu of Titanium Cloud which can be found under the tab Platform. In here we can see controller-0, note it’s in unlocked admin state and Operation is enabled.

Under “UnProvisioned Hosts” we see one undefined host which is the blade we booted for controller-1.

We can double check if this is the right blade by clicking on the ‘–‘ under Host Name and checking the Management MAC with the node we booted. This is an easy way to identify which “UnProvisioned Host” is which node, especially when we boot multiple blades at the same time.

We now can configure the node by defining its role, we do this via edit host.

After you save the configuration the installation of controller-1 will start and during this installation it will reboot several times.


Now that controller-1 is installed you can see it under Host Inventory in a Locked state and disabled. This means we need to now finish the configuration of the controller-1 and unlock it so it comes online and will automatically become the standby controller.

Because we use shared storage the only configuration setting which is required is the Network Interface assignments for both OAM and INFRA networks.

We click on controller-1 and select the Interfaces tab, here we now have to create two new interfaces. First we have to create the INFRA interface, which is VLAN 2000 on the first 10G interface. Secondly we have to create the OAM interface, which is VLAN 99 on the second 10G interface.


Now that both interfaces are created we can unlock controller-1 by selecting Unlock from the menu next to Edit Host

The node will boot several times during the unlock process, also it will synchronize the database with controller-0. After this is completed you will see controller-1 as Unlocked, Enabled and Available. It shows also that the personality is Standby, we now have two fully functional controllers.


5.2 Installing Storage nodes

The installation of the storage nodes can be easily done simultaneously now that we have two controller nodes which also provide CEPH monitor function.

Same as the controller-1 node after the storage nodes boot they will show up as UnProvisioned Hosts ready to be configured with storage personality. For this application note we only have two Storage Nodes and will only do CEPH Backing no caching.

Again during the installation the storage nodes will reboot several times until showing Locked and Online.


We now can start configuring the storage nodes, first we create the INFRA interface and then we assign two storage functions via the Storage tab of the storage nodes.

For best performance we separate Journal and OSD on separate drives, this means we will have to assign Journal function before the OSD function.

Now we unlock both storage nodes and wait until they both show Unlocked and Enabled.


5.3 Installing Low Latency Compute nodes

Before we begin with installing any compute nodes we will create the provider networks DATA1, this network is used by the different VMs for intercommunication.

We find the Provider Networks under the Platform section, below Host Inventory. Both provider networks will be VLAN type.

The Provider Network will show status DOWN until there is a compute node connected to the network.


We need to set the VLAN range by using the Create Segmentation Range option in the drop down menu. We will use one shared VLAN range that can be used by any of the projects.


We are now ready to install the compute nodes by assigning the Compute personality and the low latency performance profile.


Because we have VT-D installed, and the Marvell SATA controller on the MIC-8304C has an issue with IOMMU, we need to set a boot parameter iommu=pt. The most effective way of providing this special parameter to the compute node is via the Console window on the Installation Parameters tab.


During the installation the nodes will reboot several times and when finished show up Locked and Online. We will now have to add the INFRA network interface, provider network and even though we have shared storage we also will have to add local storage for Nova service. To add the provider network we edit the interfaces they are assigned to.


For Nova local storage we first add a Local Volume Group on the Storage tab of the compute node and then assign a Physical Volume.

We are ready to unlock the Low Latency compute nodes.


5.4 Installing Standard Compute nodes

The last components of our Titanium Cloud deployment is the installation and configuration of the Standard Compute nodes. The process is identical as the LLC Compute nodes besides the obvious difference which is the Performance Profile.

When adding the Nova local storage physical volume we have to select /dev/sda as the MIC-8303C only has a single storage device per CPU. There will be an error message from Wind River which states that this configuration isn’t supported and performance faults may occur. For any support from Wind River in this configuration you will need to have an SLA in place with Wind River.


5.5 Creating Availability Zones

Due to the fact we have two different compute node performance profiles we want to isolate each cluster in its own zone. This will assure that during a compute node failure event the Titanium cloud will migrate the VM to the blade with the same performance profile.

Under System menu is the Host Aggregates item where we can create new Host Aggregates and assign compute nodes.


5.6 Modifying System Information

In the final step to complete the setup we recommend always to update the system information. For this we login to the active controller over SSH and use the CLI, we do this because certain fields can only be updated via CLI.

We have to load the admin RC file, before the system command allows us to modify the information.

source /etc/nova/openrc

The current system information can be viewed by the system show command

Each of the property can be changed by system modify <property>=”string” which we have done for this application note.

deploying carrier grade openstack on theadvdownload.advantech.com/productfile/pis/pac-6009... ·...

Documents