administration and operation - fujitsu technology...

FlexFrame® Orchestrator

Version 1.1A

Administration and Operation

Edition June 2015

Document Version 1.2

Fujitsu Limited

© Copyright Fujitsu Technology Solutions 2015

FlexFrame ®

and PRIMERGY™ are trademarks or registered trademarks of Fujitsu

Limited in Japan and other countries.

SAP® and NetWeaver™ are trademarks or registered trademarks of SAP AG in

Germany and in several other countries

Linux® is a registered trademark of Linus Torvalds

SUSE® Linux is a registered trademark of Novell, Inc., in the United States and

other countries

Oracle™ and Java™ are trademarks of ORACLE Corporation and/or its affiliates

Intel® and PXE

® are registered trademarks of Intel Corporation in the United States

and other countries

MaxDB® is a registered trademark of MySQL AB, Sweden

MySQL® is a registered trademark of MySQL AB, Sweden

NetApp® and the Network Appliance

® logo are registered trademarks and Network

Appliance™ and Data ONTAP™ are trademarks of NetApp, Inc. in the U.S. and

other countries.

EMC®

and Celerra™ are trademarks or registered trademarks of EMC Corporation

in the United States and other countries.

VMware®, ESX

®, ESXi, VMware vCenter, VMware vSphere are registered

trademarks or trademarks of VMware, Inc. in the United States and/or other

jurisdictions.

Ethernet® is a registered trademark of XEROX, Inc., Digital Equipment Corporation

and Intel Corporation

Windows® and Word

® are registered trademarks of Microsoft Corporation

All other hardware and software names used are trademarks of their respective

companies.

All rights, including rights of translation, reproduction by printing, copying or similar

methods, in part or in whole, are reserved.

Offenders will be liable for damages.

All rights, including rights created by patent grant or registration of a utility model or

design, are reserved.

Delivery subject to availability. Right of technical modification reserved.


Contents

1 Introduction ..................................................................................................... 1 1.1 Requirements .................................................................................................... 1 1.2 Notational Conventions ..................................................................................... 1 1.3 Document History .............................................................................................. 2 1.4 Related Documents ........................................................................................... 2 1.5 Special Hints for FlexFrame

® 1.1A .................................................................... 2

1.5.1 Incompatibilities (command scripts) .................................................................. 2 1.5.2 Displaced subjects ............................................................................................ 3 1.5.2.1 ServerView Operations Manager ...................................................................... 3 1.5.2.2 Support of EMC Celerra .................................................................................... 3

2 FlexFrame Architecture .................................................................................. 5 2.1 General Notes on FlexFrame ............................................................................ 5 2.2 Hardware and Software ..................................................................................... 6 2.3 Shared Operating System ................................................................................. 9 2.3.1 Shared OS Boot Concept .................................................................................. 9 2.3.2 Control Nodes ................................................................................................... 9 2.3.3 Application Nodes ........................................................................................... 10 2.3.4 Virtual Control Center ...................................................................................... 10 2.4 FlexFrame Structure in LDAP.......................................................................... 11 2.4.1 Working with LDAP ......................................................................................... 13 2.5 Linux-HA Cluster on Control Center ................................................................ 13 2.5.1 Terminology..................................................................................................... 13 2.5.2 Simple Resources ........................................................................................... 15 2.5.2.1 Constraints ...................................................................................................... 15 2.5.2.2 Score ............................................................................................................... 15 2.5.2.3 Stickiness ........................................................................................................ 16 2.5.2.4 Resource Groups ............................................................................................ 18 2.5.3 FlexFrame Specific Configuration ................................................................... 18 2.5.3.1 Configuration of Limits for Cluster Resources ................................................. 19 2.5.3.2 LDAP Configuration ......................................................................................... 20 2.5.3.3 Resource ff_manage ....................................................................................... 22 2.5.3.4 Resource Netboot ........................................................................................... 22 2.5.3.5 STONITH (Physical Control Center) ................................................................ 23 2.5.3.6 STONITH (Virtual Control Center) ................................................................... 25 2.5.3.7 Configuring the Cluster .................................................................................... 26 2.5.3.8 Starting Behavior of the Cluster ....................................................................... 26 2.5.3.9 Status Information of the Clusters ................................................................... 27 2.5.4 Linux-HA CLI Commands ................................................................................ 27 2.5.5 Linux-HA Logfile .............................................................................................. 29 2.5.6 Linux-HA GUI .................................................................................................. 29 2.6 Network ........................................................................................................... 29

Contents


2.6.1 LAN Failover .................................................................................................... 30 2.6.2 Segments ........................................................................................................ 30 2.6.3 Network Switches ............................................................................................ 31 2.6.4 Network Speed ................................................................................................ 31 2.6.5 Network Switch Groups ................................................................................... 32 2.6.6 Network Switch Ports ...................................................................................... 32 2.6.7 Automounter Concept ...................................................................................... 33 2.7 Storage Systems ............................................................................................. 35 2.7.1 NAS systems ................................................................................................... 36 2.7.1.1 Built-in Cluster File System .............................................................................. 37 2.7.1.2 Volume Layout ................................................................................................. 37 2.7.1.3 Snapshots ........................................................................................................ 37 2.7.1.4 vFiler ................................................................................................................ 38 2.7.1.5 Storage Virtual Machines (SVMs) .................................................................... 38 2.7.2 SAN Support .................................................................................................... 40 2.7.2.1 SAN Basic Layers ............................................................................................ 40 2.7.2.2 Rules and Restrictions ..................................................................................... 41

3 FlexFrame Basic Administration .................................................................. 42 3.1 Accessing a FlexFrame Landscape (Remote Administration) ......................... 42 3.2 Powering up the FlexFrame Landscape .......................................................... 42 3.3 Powering off the FlexFrame Landscape .......................................................... 43 3.4 Reactivating ANs after Power Shutdown by FlexFrame Agents ...................... 46 3.5 Displaying the Current FlexFrame Configuration State ................................... 47 3.5.1 FlexFrame Web Portal ..................................................................................... 48 3.5.2 FlexFrame Agents ........................................................................................... 49 3.5.2.1 State of Pools .................................................................................................. 49 3.5.2.2 State of Application Nodes .............................................................................. 49 3.5.2.3 State of SAP Systems ..................................................................................... 50 3.5.2.4 State of SID Instances ..................................................................................... 50 3.5.2.5 Virtual IP address for FlexFrame Agents WebGUI .......................................... 50 3.5.3 Networks.......................................................................................................... 51 3.5.4 Cluster Status .................................................................................................. 53 3.5.5 Supported Hardware ....................................................................................... 53 3.5.6 Management Blades ........................................................................................ 54 3.6 FlexFrame Backup with Tape Library .............................................................. 54 3.7 Product Licenses ............................................................................................. 55 3.7.1 How to Work with Licenses .............................................................................. 55 3.7.2 Types of Licenses in FlexFrame Orchestrator ................................................. 55 3.7.3 How to Install a License Key ............................................................................ 56 3.7.4 How to List Installed License Keys .................................................................. 56 3.7.5 How to Control the Use of Licenses ................................................................ 57

4 Pools and Groups .......................................................................................... 58 4.1 Adding a Pool .................................................................................................. 58

Contents


4.2 Removing a Pool ............................................................................................. 61 4.3 Listing Pool Details .......................................................................................... 63 4.4 Listing All Pools ............................................................................................... 66 4.5 Changing Pool DNS Domain ........................................................................... 68 4.6 Changing Pool Default Router ......................................................................... 69 4.7 Adding a Group to a Pool ................................................................................ 70 4.8 Removing Pool Group ..................................................................................... 71 4.9 Hosts Database ............................................................................................... 71 4.9.1 Script: ff_hosts.sh ............................................................................................ 72

5 User and Groups Administration ................................................................. 74 5.1 Create, Modify, Delete, or List User(s) for Application Nodes ......................... 74 5.2 Creating, Modifying, Deleting or Listing Group(s) for Application Nodes ......... 76 5.3 Creating, Modifying, Deleting or Listing Service(s) for Application Nodes ....... 78

6 Pool-independent Spare Nodes ................................................................... 81 6.1 Creation of Spare Nodes in the ADMINPOOL ................................................. 81 6.2 Moving of a Spare Node .................................................................................. 81 6.3 Handling Pool-independent Spare Nodes with FlexFrame Agents .................. 82

7 Application Nodes Administration ............................................................... 83 7.1 Listing Application Nodes ................................................................................ 83 7.1.1 Displaying Information on a Specific Application Node ................................... 83 7.1.2 Displaying Information of all Application Nodes............................................... 87 7.1.3 Displaying all available OS images ................................................................. 88 7.2 Adding Application Nodes ............................................................................... 88 7.3 Removing Application Nodes .......................................................................... 92 7.4 Changing the attributes of an Application Node .............................................. 93 7.4.1 Change the group of an Application Node ....................................................... 93 7.4.2 Change the os image of an Application Node ................................................. 94 7.4.3 Change the mac addresses of an Application Node ........................................ 95 7.5 Moving Application Nodes ............................................................................... 95 7.6 Administrating Blade Server Cabinets ............................................................. 97 7.6.1 Listing Blade Server Cabinets ......................................................................... 97 7.6.1.1 Displaying Information on a Specific Blade Server Cabinet ............................ 97 7.6.1.2 Displaying Information on all Configured Blade Server Cabinets .................... 98 7.6.2 Adding Blade Server Cabinets ........................................................................ 99 7.6.3 Removing Blade Server Cabinets ................................................................. 102 7.6.4 Changing Switch Blade Type ........................................................................ 103 7.6.5 Changing Switch Blade Name ....................................................................... 103 7.6.6 Changing Switch Blade Password ................................................................. 104 7.6.7 Getting Switch Blade Configuration ............................................................... 105 7.6.8 Change Switch Blade Uplink ......................................................................... 106 7.6.9 Move a Blade Cabinet to Another Switch Group ........................................... 107

Contents


7.7 Administrating Hypervisor Nodes and Virtual Machines ................................ 109 7.7.1 Getting Started with Hypervisor Nodes and VMs ........................................... 110 7.7.2 Virtualization related global FlexFrame parameters ...................................... 111 7.7.2.1 System Code for Hypervisor Nodes and VMs ............................................... 111 7.7.2.2 vCenter Server .............................................................................................. 112 7.7.2.3 Control LAN Default Router for Hypervisor Nodes ........................................ 114 7.7.3 Adding Hypervisor Nodes .............................................................................. 114 7.7.4 Completing Hypervisor Node Configuration................................................... 117 7.7.5 Removing Hypervisor Nodes ......................................................................... 118 7.7.6 Displaying Information about HNs and VMs .................................................. 119 7.7.7 Hypervisor Nodes and Pools ......................................................................... 122 7.7.8 Special Functions for Virtual Machines .......................................................... 123 7.7.9 Virtual Machine Properties and Hypervisor Resources ................................. 125 7.7.10 Using Hypervisor Management Software ...................................................... 127 7.8 Power on / off / reboot of an Application Node .............................................. 128

8 Storage Systems Administration ............................................................... 129 8.1 NAS Systems Configuration .......................................................................... 129 8.1.1 Adding a New NAS ........................................................................................ 129 8.1.2 Removing a NAS ........................................................................................... 131 8.1.3 Displaying all configured NAS ....................................................................... 132 8.1.4 Displaying NAS Configuration ....................................................................... 134 8.1.5 Adding a Pool to a NAS ................................................................................. 135 8.1.6 Removing a Pool from a NAS ........................................................................ 135 8.1.7 Create NAS HA Partnership .......................................................................... 136 8.1.8 Moving a NAS ................................................................................................ 137 8.1.9 Changing NAS Command Shell .................................................................... 138 8.1.10 Changing NAS Ports ..................................................................................... 139 8.1.11 Changing NAS Path Export ........................................................................... 140 8.2 Using SAN Storage ....................................................................................... 141 8.2.1 Getting started with SAN usage in FlexFrame ............................................... 142 8.2.2 Administrating Volume Groups and LUNs ..................................................... 146 8.2.2.1 Adding a Volume Group ................................................................................ 147 8.2.2.2 Adding a LUN ................................................................................................ 149 8.2.2.3 Removing a Volume Group ........................................................................... 151 8.2.2.4 Removing a LUN ........................................................................................... 152 8.2.2.5 Changing Poolgroup Assignment of a Volume Group ................................... 153 8.2.2.6 Displaying Information about Volume Groups and LUNs............................... 153 8.2.3 Completing SAN Setup for a SAP System..................................................... 157 8.2.4 Mounting SAN based File Systems of a SAP Service ................................... 158 8.2.5 SAN Helper Functions ................................................................................... 159

9 Switch Administration ................................................................................. 162 9.1 Adding a Switch to a Switch Group ............................................................... 162 9.2 Removing a Switch from a Switch Group ...................................................... 165

Contents


9.3 Listing a Switch Group Configuration ............................................................ 166 9.4 Generating a Switch Group Configuration ..................................................... 168 9.5 Changing the Password of a Switch Group ................................................... 169 9.6 Changing the Host Name of a Switch Group ................................................. 170 9.7 Displaying/Changing Common Network Configuration Parameters .............. 172 9.8 Adding a Switch Group .................................................................................. 174 9.9 Adding an Expansion Module ........................................................................ 177 9.10 Removing an Expansion Module ................................................................... 178 9.11 Removing a Switch Group ............................................................................. 179 9.12 Adding an Uplink to Switch Group ................................................................. 180 9.13 Extend an Uplink of Switch Group ................................................................. 181 9.14 Delete an Uplink of Switch Group .................................................................. 183 9.15 Migrating Switches of a Switch Group ........................................................... 184 9.16 Adding a Switch Port Configuration ............................................................... 185 9.17 Removing a Switch Port Configuration .......................................................... 187 9.18 Displaying a Switch Port Configuration ......................................................... 188 9.19 Displaying the Complete Switch Port Configuration ...................................... 189 9.20 Moving Device Connection to Core Switch.................................................... 191 9.20.1 Move Control Center to Core Switch ............................................................. 191 9.20.2 Move Client LAN to Core Switch ................................................................... 192 9.20.3 Move NAS System to Core Switch ................................................................ 192 9.20.4 Move Application Node to Core Switch ......................................................... 193 9.20.5 Move ESX Server to Core Switch .................................................................. 193 9.20.6 Move BX Cabinet to Core Switch .................................................................. 194

10 SAP System Handling ................................................................................. 195 10.1 Listing SAP SIDs and Instances .................................................................... 196 10.2 Updating System Configuration Files ............................................................ 198 10.3 Adding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)198 10.4 Removing SAP SIDs and Instances .............................................................. 205 10.5 Adding/Removing SAP SIDs (add-on services) ............................................. 206 10.5.1 BOBJ – Business Objects ............................................................................. 206 10.5.2 Content Server (CMS) ................................................................................... 207 10.5.3 MDM – Master Data Management ................................................................ 211 10.5.4 SMD – Solution Manager Diagnostics ........................................................... 214 10.5.5 TREX (Search and Classification Service) .................................................... 218 10.5.6 WebDispatcher .............................................................................................. 220 10.5.7 HANA database (service hdb) ....................................................................... 222 10.5.8 SAP classic NetWeaver services using HANA database .............................. 227 10.6 Cloning a SAP SID into a Different Pool ........................................................ 228 10.6.1 Script: ff_clone_sid.pl .................................................................................... 228 10.6.2 Changing User and Group IDs after Cloning ................................................. 229 10.7 Cloning a SAP system into a different Pool ................................................... 230 10.7.1 Script: ff_clone_sap.sh .................................................................................. 230 10.7.1.1 Cloning SAP systems with MaxDB ................................................................ 235

Contents


10.7.1.2 Post cloning steps – DB2............................................................................... 235 10.7.1.3 Post cloning steps – Oracle ........................................................................... 236 10.7.1.4 Post cloning steps – MaxDB .......................................................................... 236 10.7.1.5 Post cloning steps – HANA reference database ............................................ 236 10.7.1.6 Cloning a SAP system example session ....................................................... 237 10.8 Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl .............. 238 10.8.1 Hints for usage .............................................................................................. 242 10.8.1.1 General .......................................................................................................... 242 10.8.1.2 Creating a SID/relocation of sapdata/saplog ................................................. 242 10.8.1.3 Creating a SID/relocation of SID specific data ............................................... 243 10.9 Multiple NAS Systems and Multiple Volumes ................................................ 243 10.9.1 NetApp NAS Systems ................................................................................... 244 10.10 Upgrading a SAP System .............................................................................. 245 10.10.1 Service Port ................................................................................................... 245 10.10.2 FlexFrame Agents ......................................................................................... 246 10.10.3 Support SAP Upgrade (ff_sap_upgrade.pl) ................................................... 246 10.11 SAP Kernel Updates and Patches ................................................................. 249 10.12 Unloading volFF ............................................................................................ 249 10.13 Administrating SAP Services ......................................................................... 250 10.13.1 Displaying Status of SAP Services ................................................................ 250 10.13.1.1 FlexFrame Agents WebGUI ........................................................................... 250 10.13.1.2 List SAP Services .......................................................................................... 251 10.13.2 Starting and Stopping Application Services ................................................... 252 10.13.2.1 Configuration file (ff_service.config) .............................................................. 253 10.13.2.2 Starting and Stopping SAP Services Without Root Privileges ....................... 255 10.13.2.3 Starting and Stopping SAP Services from Control Node using RBAC ........... 256 10.13.2.4 SAP Service Scripts ...................................................................................... 257 10.13.2.5 SAP Service Script Actions ............................................................................ 259 10.13.2.6 SAP Service Script logging ............................................................................ 261 10.13.2.7 SAP Service Script User Exits ....................................................................... 261 10.13.2.8 SAP service script ff_smd_agent.sh .............................................................. 262 10.13.3 Return Code of the SAP Service Script ......................................................... 262 10.13.4 Starting and Stopping Multiple SAP Services ................................................ 262 10.13.5 Removing an Application from Monitoring by FlexFrame Agents .................. 264 10.13.5.1 Stopping and Starting an Application for Upgrades Using r3up ..................... 265 10.13.6 Service Switchover ........................................................................................ 265 10.13.7 Use ServicePings from FlexFrame Agents .................................................... 267

11 Software and Hardware Upgrade and Maintenance ................................. 269 11.1 Upgrading the Entire FlexFrame Landscape ................................................. 269 11.2 Software Upgrade on the Control Node ......................................................... 269 11.2.1 FlexFrame Control Node Software ................................................................ 270 11.2.2 Service Packs ................................................................................................ 271 11.2.3 Updating/Installing a New Linux Kernel ......................................................... 271 11.2.3.1 Software Stage .............................................................................................. 271

Contents


11.2.3.2 Install the New Kernel ................................................................................... 272 11.2.3.3 Reboot the Control Node ............................................................................... 272 11.2.3.4 Partner Linux Driver Process......................................................................... 272 11.2.4 Updating RPM Packages on an Control Node .............................................. 273 11.2.4.1 Setup a patch server ..................................................................................... 273 11.2.4.2 Connect the patch server to the FlexFrame environment .............................. 273 11.2.4.3 Prepare the Online Update ............................................................................ 273 11.2.4.4 Perform the Online Update ............................................................................ 274 11.2.5 Backup/Restore of FlexFrame Control Nodes ............................................... 274 11.2.5.1 Backup of a Control Node ............................................................................. 275 11.2.5.2 Restore of a Control Node ............................................................................. 275 11.3 Maintenance of the Control Node - Hardware ............................................... 278 11.3.1 Exchanging a Control Node........................................................................... 278 11.3.1.1 Hardware Failed, Hard Disk and Installed OS are not Affected ..................... 278 11.3.1.2 One Hard Disk is Defect, the Other One is Undamaged ............................... 278 11.3.1.3 The Control Nodes OS is Damaged .............................................................. 278 11.3.2 Replacing a Network Card – Control Node ................................................... 279 11.4 Maintenance of the virtual Control Center ..................................................... 279 11.4.1 Site failover within virtual Control Center ....................................................... 279 11.4.1.1 The naming conventions ............................................................................... 279 11.4.1.2 Scenario of sane KVM Hypervisor Nodes ..................................................... 280 11.4.1.3 KVM Hypervisor Node kvm-vcn1 fails (manual switch over) ......................... 282 11.4.1.4 KVM Hypervisor Node kvm-vcn1 fails (automatic switch over)...................... 286 11.4.2 Restore of damaged virtual Control Node ..................................................... 287 11.4.3 Restore of damaged Hypervisor Node .......................................................... 287 11.5 Maintenance of Application Nodes - Software ............................................... 289 11.5.1 Introduction.................................................................................................... 289 11.5.2 Schematic Overview ...................................................................................... 289 11.5.3 Installing Application Node Images from Installation Media .......................... 291 11.5.3.1 Installing the Application Node Image ........................................................... 291 11.5.4 Step #1: Creating a Maintenance Image ....................................................... 293 11.5.5 Step #2: Assigning the Maintenance Image, booting and maintaining .......... 295 11.5.5.1 Using the Node .............................................................................................. 296 11.5.5.2 Booting .......................................................................................................... 297 11.5.5.3 Maintaining .................................................................................................... 298 11.5.6 Step #3: Reverting the Maintenance Image .................................................. 299 11.5.7 Migrating remaining Application Nodes ......................................................... 302 11.5.8 Re-Using the Maintenance Image ................................................................. 304 11.5.9 Maintaining Use Cases ................................................................................. 306 11.5.9.1 Service Packs ................................................................................................ 306 11.5.9.2 Updating/Installing a New Linux Kernel ......................................................... 306 11.5.9.3 ServerView Update ....................................................................................... 309 11.5.9.4 Upgrading the Application Software .............................................................. 310 11.5.9.5 Updating RPM Packages on an Application Node ........................................ 311 11.5.9.6 Updating vmware-tools .................................................................................. 312

Contents


11.5.10 Implement conflicting changes using the ff_new_an.custom hook ................ 313 11.6 Maintenance of the Application Nodes - Hardware ........................................ 317 11.6.1 Changing BIOS Settings for Netboot ............................................................. 317 11.6.2 Replacing a Network Card – Application Node .............................................. 317 11.6.3 Replacing a Switch Blade .............................................................................. 319 11.6.4 Replacing Power Control Hardware .............................................................. 319 11.7 Maintenance of ESXi Servers ........................................................................ 320 11.7.1 BIOS Updates ................................................................................................ 320 11.7.2 Replacing a Network Card - ESXi Server ...................................................... 320 11.7.3 ESXi Updates and Patches ........................................................................... 320 11.8 Maintenance of Other FlexFrame Components ............................................. 321 11.8.1 NetApp Storage ............................................................................................. 321 11.8.2 Switches and Switch Blades .......................................................................... 321 11.8.2.1 Firmware Update ........................................................................................... 321 11.8.2.2 Backup of Switch Configurations ................................................................... 321 11.8.2.3 Restore of Switch Configurations .................................................................. 322 11.8.3 DNS Servers .................................................................................................. 323 11.8.4 Third Party Products ...................................................................................... 323 11.9 FlexFrame Agents ......................................................................................... 324 11.10 Project specific changes ................................................................................ 324 11.10.1 The ff_projectinfo.sh command ..................................................................... 325

12 Troubleshooting .......................................................................................... 328 12.1 Locking of Flex Frame Administration Commands ........................................ 328 12.2 Script Logging in FlexFrame .......................................................................... 329 12.3 RBAC Logging with Syslog ............................................................................ 331 12.4 Log Files ........................................................................................................ 331 12.5 Network Errors ............................................................................................... 333 12.6 NFS Mount Messages ................................................................................... 333 12.7 LDAP Error Codes and Messages ................................................................. 333 12.8 LDAP and Cache Coherence ........................................................................ 335 12.8.1 Linux .............................................................................................................. 335 12.9 Start/Stop Script Errors .................................................................................. 335 12.9.1 Severity ‘INFO’ .............................................................................................. 336 12.9.2 Severity ‘WARN’ ............................................................................................ 336 12.9.3 Severity ‘ERROR’ .......................................................................................... 336 12.9.4 Severity 'DEBUG' .......................................................................................... 336 12.10 Script Debugging ........................................................................................... 336 12.10.1 Shell Scripts ................................................................................................... 336 12.10.2 Perl Scripts .................................................................................................... 337 12.11 Debugging the Linux Kernel .......................................................................... 337 12.11.1 Netconsole ..................................................................................................... 337 12.11.2 Capturing Crash Dumps ................................................................................ 338 12.11.2.1 Common Restrictions for Taking Crash Dumps ............................................. 338 12.11.2.2 Kernel Crash Dump Capturing (Kdump) ........................................................ 338

Contents


12.11.2.3 Forcing a Crash Dump .................................................................................. 338 12.12 Activate core dumps on CN or AN ................................................................. 339 12.13 Huge Amount of getattr Calls ........................................................................ 340

13 Abbreviations .............................................................................................. 341

14 Glossary ....................................................................................................... 345

15 Index ............................................................................................................. 351

Administration and Operation 1

1 Introduction

This document provides instructions on administrating and operating an installed

FlexFrame® 1.1A environment. It focuses on general aspects of the architecture as well

as on software updates, hardware extensions and FlexFrame-specific configuration.

It does not cover the installation of an entire FlexFrame environment. Please refer to the

“Installation of a FlexFrame Environment” manual for information on initial installation.

1.1 Requirements

This document addresses administrators on FlexFrame environments. We assume that

the reader of this document has technical background knowledge in the areas of

operating systems (Linux®), IP networking and SAP

® basis.

1.2 Notational Conventions

The following conventions are used in this manual:

Additional information that should be observed.

Warning that must be observed.

fixed font Names of paths, files, commands, and system output.

<fixed font> Names of variables.

fixed font User input in command examples (if applicable using <> with variables).

Command prompt: # The notation

control1:/<somewhere> # <command>

indicates that the command <command> is issued on the first

Control Node in the directory /<somewhere>.

The reader may need to change into the directory first, e.g.

control1:~ # cd /<somewhere>

control1:/<somewhere># <command>.

Introduction Document History

2 Administration and Operation

1.3 Document History

Document

Version

Changes Date

1.0 First Edition 2014-12-03

1.1 Update with small corrections 2015-02-23

1.2 vCC support 2015-06-30

1.4 Related Documents

FlexFrame® – HW Characteristics Quickguides

FlexFrame® – Installation and Configuration of LVM 2.1 Standard Edition

FlexFrame® – Installation Guide for SAP Solutions

FlexFrame® – Installation of a FlexFrame Environment

FlexFrame® – Management Tool

FlexFrame® – Agents Installation and Administration

FlexFrame® – Messenger Concepts and Usage

FlexFrame® – LogAgent Concepts and Usage

FlexFrame® – Network Design and Configuration Guide

FlexFrame® – Security Guide

FlexFrame® – Technical White Paper

FlexFrame® – Upgrading FlexFrame Orchestrator 1.0A to 1.1A

ServerView Documentation

SUSE Linux Enterprise Server Documentation

1.5 Special Hints for FlexFrame® 1.1A

In this document, you often will find console output, configuration data and

installation examples which are based on earlier FlexFrame versions. Please

keep in mind that these are examples and may look slightly different in

FlexFrame® 1.1A.

The two Control Nodes (CN) of FlexFrame are also named as the FlexFrame Control

Center (CC). In this documentation the notation Control Node (CN) is used as a

synonym for Control Center (CC) and the other way round.

1.5.1 Incompatibilities (command scripts)

There have been some changes with supplied scripts:

Special Hints for FlexFrame® 1.1A Introduction


The script ff_change_id.pl is not further available.

With the script ff_user_adm.pl resp. ff_group_adm.pl you can change

UIDs or GIDs of a specific user/group

The ff_install_an_linux_images.sh script uses a slightly different syntax:

the arguments to the -t and -s options are now names and not any longer

directories, and the -m option needs an argument.

The command line syntax of ff_setup_sid_folder.sh changed to

ff_setup_sid_folder.sh –s <SID> -p <pool>

With script ff_nas_adm.pl, the operation mode conf10gb is no longer

supported. Use operation mode move instead.

The script ff_esx_adm.pl is no longer available. Use the script ff_hn_adm.pl

instead

The script ff_esx_vm.pl is no longer available. Use the script ff_vm.pl instead

1.5.2 Displaced subjects

1.5.2.1 ServerView Operations Manager

The software component ServerView Operations Manager used to be an element of the

Control Node software stack. This component (allowing to monitor the hardware status of

PRIMERGY servers in your FlexFrame® environment) has been removed from the

Control Node as of FlexFrame 5.3A/1.0A. Customers wanting to use this component

have to install and configure ServerView Operations Manager on an own server. See

Description Paper - Server monitoring and trap forwarding on the Fujitsu Extranet or ask

your Fujitsu Customer Support for getting a copy.

Still left on the Control Node (and also on the Application Nodes) are the still active

software components ServerView Agents and ServerView RAID Manager.

1.5.2.2 Support of EMC Celerra

The support for EMC Celerra had already been frozen based on FlexFrame 5.0A for all

successor versions of Flexframe – now with FlexFrame 1.1A the totally new directory and

automount structure caused by the requirement to support NetApp’s ONTAP clustermode

does not allow to continue this frozen support any further, so FlexFrame Orchestrator

1.1A is not prepared to support EMC Celerra without changes – if needed you should

comission Fujitsu to do a Solution project corresponding to your demands.

https://partners.ts.fujitsu.com/com/service/ps/solutions/flexframe/FFO/FFO1.0A/Pages/TechnicalInformationMaintenance.aspx


2 FlexFrame Architecture

The FlexFrame solution is a revolutionary approach to run complex SAP solutions with

higher efficiency. At the same time some major changes of the configuration paradigms

for infrastructures have been implemented. These changes are:

A shared operating system booted via IP networks for the SAP Application Nodes.

Decoupling of application software and operating system, called virtualization of

SAP software or Adaptive Computing.

Shared Network Attached Storage from Network Appliance® providing Write

Anywhere File Layout (WAFL™) and sophisticated snap functionality.

FlexFrame Agents providing revolutionary mechanisms to implement high-

availability functions without cluster software

The concept of FlexFrame consists of several components, which implement state-of-the-

art functionality. Together with new components, such as the FlexFrame Agents, the

whole solution is far more than just the sum of its components. A major part of the

benefits consists in a dramatic reduction in day to day operating costs for SAP

environments.

It is, of course, possible to use parts of the FlexFrame solution in project-based

implementations. However, they cannot be called FlexFrame.

2.1 General Notes on FlexFrame

FlexFrame was designed and developed as a platform for SAP Applications. Its major

purpose is to simplify and abstract the basic components to enable the administrator of a

SAP system landscape to focus on SAP and not worry about servers, networking and

storage.

The FlexFrame Control Nodes are seen as an appliance. Like a toaster or a micro wave

oven, they have a well defined purpose and the build-components must work together to

achieve that purpose. It happened that Fujitsu picked SUSE® Linux Enterprise Server

(SLES) as the operating system for the Control Nodes; however it is not intended that the

customer is using it as a regular server, meaning installing additional software on it and

applying patches to it is not wanted, unless Fujitsu support line instructs so. Upcoming

versions of the Control Node's operating system may be totally different and may not

allow modifying anything at all. The installation and backup/restore functionality of a

Control Node are based on fixed images which are delivered on DVD. Modifications of

installed images will not be taken care of, if a new version is installed.

Modifications may even lead to errors which may be hard to find. Therefore we strongly

recommend to the customers not to install any software or patches onto the Control

Nodes without confirmation of Fujitsu support.

FlexFrame Architecture Hardware and Software


Similar to the Control Nodes are the Application Nodes. Fixed images are shipped for

installation of Linux Application Nodes in the FlexFrame landscape. Another aspect of

FlexFrame is the reduction of the TCO (Total Cost of Ownership). A static approach

(once started, never touched) will not be very efficient. To achieve the best savings, it is

recommended to actively manage where a certain SAP application instance or database

instance is running. If, as an example, a SAP instance requires the power of two CPUs of

an Application Node during most days of a month and eight CPUs during month-end

calculations, it is best to move it back and forth to the appropriate Application Node with

the right size. During the time where the application instance is running on the two CPU

Application Node, another SAP instance can use the bigger eight CPU Application Node

and therefore saving the need to have more eight CPU Application Nodes as in a static

approach.

2.2 Hardware and Software

The FlexFrame solution consists of both, hardware and software. To grant proper

function of the whole landscape, the entire software set is strictly defined. Anything other

than the software components listed below is not part of FlexFrame. This applies

unchanged if software from the list below is missing, is installed in other versions than

below, or if software other than the actual SAP components is added.

For detailed information about the hardware supported in a FlexFrame environment, see

the FlexFrame 1.1A Configuration Guide. Any other functions, such as backup, can be

implemented separately as an add-on to FlexFrame and need dedicated hardware,

operating system, high availability, professional service and support etc.

Hardware and Software FlexFrame Architecture


No. Hardware 1

OS Software Services

1 Control Nodes:

2 x PRIMERGY RX2540 M1 or

2 x PRIMERGY RX300 S8, S7, S6, or S5, or

2 x PRIMERGY rack servers each equipped with a

KVM hypervisor

SLES11 SP3 plus SLES11

HAE or

SLES for SAP 2

FlexFrame Agents (Control Agents)

V9.0, FlexFrame 1.1A File System

Image CN, ServerView Agents,

ServerView RAID, etc.

TFTP, DHCP, LDAP,

(SAPROUTER), etc.

2 Network switches:

pairs of 1 GbE switches (Cisco Catalyst 3750x or

Brocade ICX 6430 / 6450),

and/or

pairs of 10 GbE switches (Cisco Nexus 5K or

Brocade VDX6740 or 3 )

IOS (proprietary)

NX-OS (proprietary)

Brocade Network OS (propr.)

…

(as delivered)

3 Network Attached Storage:

one or more NetApp storage systems

(FASxxxx, vFiler or SVMs),

disk shelves as required 4, hosting shared OS

file systems, and application data

ONTAP (7-mode and cDOT) 5

(proprietary)

NetApp Tools

NFS, optional:

cluster components,

FlexClone, SnapVault,

SnapMirror,

SnapRestore

4 Intel- based PRIMERGY or PRIMEQUEST servers

(standard rack or blade server)

SLES11 SP3 / SLES for SAP 1

or VMware vSphere ESXi running SLES11 SP3 / SLES for SAP

2

or SLES11 SP3 / SLES for SAP

2

running as KVM Hypervisor

FlexFrame V1.1A File System Image,

FlexFrame Agents (Application Agents),

SAP Applications, Database

SAP & DB Services

1 allowed types according to current FlexFrame Support Matrix available on the Service DVD or ask your Fujitsu Customer Support for getting an up-to-date copy

2 based on SLES11 SP3 (x86_64)

3 or other switch pairs when adapter modules exist and are certified by Fujitsu

4 the amount of disks required for customer-specific FlexFrame configurations can be determined together with Fujitsu Customer Support Filer Sizing Team

5 the currently supported ONTAP version is kept up-to-date in the FlexFrame Support Matrix

FlexFrame Architecture Hardware and Software



2.3 Shared Operating System

One major aspect of FlexFrame is its shared operating system. Sharing in this case

means that the very same files of essential parts of the underlying operating system are

used to run multiple Application Nodes. This part of the file system is mounted read-only,

so none of the Application Nodes that run the actual applications can modify it. Server

specific information is linked to a file system area that is server-specific and mounted

read-write. The shared operating system is kept to a NAS system (Network Attached

Storage system from NetApp).

2.3.1 Shared OS Boot Concept

The chart below shows the boot process of a FlexFrame Application Node

(PRIMERGY/Linux):

2.3.2 Control Nodes

A productive FlexFrame landscape always includes two Control Nodes. Their purpose is

to be a single point of control for the Application Nodes, as well as to check and manage

the proper function of the Application Nodes.

FlexFrame Architecture Shared Operating System


Control Nodes are not running SAP software (with the exception of saprouter, as an

option). They exclusively run SUSE Linux Enterprise Server Version 11 (SLES11 SP3),

installed on local disks. Control Nodes provide and run services such as:

Linux-HA high availablity cluster framework

Timeserver for the complete FlexFrame landscape

Control Agents

Web server to provide the Control Agents user interface

DHCP for assignment of IP addresses and TFTP for the boot process of the

Application Nodes

saprouter (optional)

Control Nodes have to be of the type PRIMERGY RX300 (models S8, ..., S5 resp.

RX2540M1).

As of FlexFrame Orchestrator 1.1A00-020 , the Control Nodes can also be virtual

machines on top of PRIMERGY servers using SUSE Linux KVM as a virtualization layer.

2.3.3 Application Nodes

Application Nodes run database and SAP services on the SUSE Linux Enterprise Server

shared operating system. Application Nodes can be physical servers that offer CPU and

memory, or virtual servers built on top of physical servers, using the physical server's

CPU and memory through a virtualization layer.

For FlexFrame the principal types of Application Nodes are PRIMERGY servers running

Linux directly and virtual servers on top of PRIMERGY servers using VMware ESXi or

SUSE Linux KVM as a virtualization layer.

Admissible PRIMERGY servers have to be approved for SAP on Linux by Fujitsu.

During the boot process using Intel's PXE® technology, each Application Node will be

identified using the hardware address of its boot interface (MAC address). The Control

Node will assign an IP address to it and supply the operating system via the network.

File systems, especially the root file system (/), are mounted via the network in read-only

mode. If, for any reason, an Application Node needs to be replaced or added, only a

handful of settings need to be adjusted to integrate it into the FlexFrame landscape.

Intel's PXE technology is implemented in Fujitsu PRIMERGY servers and allows booting

via the network. DHCP will be used with static MAC address relationship for all the

Application Nodes.

2.3.4 Virtual Control Center

As of FFO 1.1A00-020 the Control Center of FlexFrame can either consist of physical

Control Nodes or of virtual Control Nodes (vCN). In case of a virtual Control Center (vCC)

FlexFrame Structure in LDAP FlexFrame Architecture


both virtual machines are hosted on different physical PRIMERGY servers that run SUSE

KVM as hypervisor. The virtualized Control Center has the following features:

Two physical servers are installed as KVM Hypervisor Nodes. The supported

Hypervisor Nodes can be derived from the FlexFrame Support Matrix.

The virtual Control Node CN1 is hosted on one of the Hypervisor Nodes.

Accordingly Control Node CN2 is hosted on the remaining Hypervisor Node.

Just like their physical counterparts, the virtual Control Nodes are clustered by

LINUX-HA. In case of hardware failure of one of the Hypervisor Nodes LINUX-

HA guarantees that the Control Node functionality is still given.

Both KVM Hypervisor Nodes can host other virtual machines i.e. Application

Nodes.

Running Control Node and additional Application Nodes in parallel allows a

higher grade of utilization of the hardware,

The virtual Control Center allows the configuration of an FFO entry model

consisting of only two physical servers.

There is no migration of virtual Control Nodes within the FlexFrame landscape

A virtual Control Center can only be installed during a new FFO installation. Up to now

you cannot install it during the Upgrade process. The installation process is described in

detail in the Installation Guide. The KVM Hypervisor Nodes hosting a virtual Control

Center must fulfill special requirements as described in the "FlexFrame® HW

Characteristics Quickguides". Especially the KVM Hypervisor Nodes hosting a virtual

Control Node need 4 network interfaces. Within the FFO 1.1A00-020 implementation

there is a fixed assignment of virtual Control Nodes to their Hypervisor Node.

A vCN needs a portion of 60GB diskspace on the physical local disk of the underlying

Hypervisor Node and a corresponding portion of 8GB memory. It is the customer’s

responsibility to care for the sufficient availability of disk and memory space of the

underlying Hypervisor Node for the virtual Control Node.

At the moment it is not possible to migrate a virtual Control Node to another Hypervisor

Node.

In case a virtual Control Node is disturbed or the Hypervisor Node is physically defect

special steps have to be done in order to get the virtual Control Center functional again.

These disaster scenarios are described in detail in chapter 11.4.

2.4 FlexFrame Structure in LDAP

LDAP is used as the central information service for all shared OS nodes within a

FlexFrame environment. The Control Nodes are used as LDAP servers. The LDAP

database is located on shared file systems mounted from the NAS storage. The

FlexFrame Architecture FlexFrame Structure in LDAP


Application Nodes are configured as LDAP clients. LDAP requests from Application

Nodes are restricted to the data of their own pool.

LDAP provides host-related network information such as:

net boot

automount

user authentication

groups

host names and IP addresses

shared services

networks and netmasks

LDAP client profiles

The FlexFrame LDAP tree roughly looks as illustrated here:

Additional information about configuration data is only applicable for Control Nodes.

These are used FlexFrame-internally to add, remove or modify the configuration of

Application Nodes or SAP services.

Linux-HA Cluster on Control Center FlexFrame Architecture


FlexFrame utilizes LDAP for two different purposes:

(1) for operating naming services (such as host name resolution, user/password

retrieval, tcp/udp service lookup, etc.) and

(2) for storing FlexFrame specific data on the structure of the installed environment

Application Nodes are only able to search in area (1). It is separated into pool specific

sections in order to protect pools from accessing other pools' data. Each of them contains

pool specific network information service (NIS) like data. The LDAP servers have access

lists to prevent searches outside of the own pool.

The other main DIT part contains FlexFrame configuration data (2). It should only be

accessed through maintenance tools from one of the Control Nodes. This part of the DIT

contains a lot of cross references, which need to be kept in sync. Do not try to change

this data, unless you are explicitly instructed by Fujitsu support to do so.

2.4.1 Working with LDAP

The usage of LDAP specific commands like ldapadd or ldapmodify is limited to very

few actions. One is to create or remove a PUT service for a SAP system copy. This

action is described within the “Installation Guide for SAP Solutions” manual. Other direct

interaction through LDAP commands is limited to service issues.

No other interventions have to and should be done. The FlexFrame maintenance tools

provide the necessary functionality.

2.5 Linux-HA Cluster on Control Center

2.5.1 Terminology

Some of the Linux-HA terms used below are explained here in order to promote broader-

based understanding:

Node

Every computer that is part of a cluster is a node

Pacemaker

Software to perform Linux-HA resource management on a cluster.

Resource

Everything than can be administered by pacemaker is referred to as a resource. For

example, an IP address that is administered by the cluster is a resource.

Resource Agent (RA)

The RA is the connection between pacemaker and the programs that are started

when the RA is called. They are shell scripts, which have to provide a standardized

FlexFrame Architecture Linux-HA Cluster on Control Center


interface to pacemaker, so that they can be started, monitored and stopped by

pacemaker.

Supported standards:

Linux Standard Base (LSB)

o All scripts under /etc/init.d correspond to this standard

Open Cluster Framework (OCF)

o For examples see /usr/lib/ocf/resource.d

Stonith

Designated Coordinator (DC)

Every cluster has precisely one DC as the central instance in the cluster. It is elected

from all the nodes. It alone is responsible for all the actions in the cluster and has the

only valid cluster information base. All the other nodes only have a duplicate.

STONITH

STONITH is an abbreviation for "Shoot The Other Node In The Head". This is the

name of a method that stops any nodes no longer accessible from "causing damage"

in the cluster by switching it off or alternatively causing a reboot.

openais / corosync

The cluster can be started and stopped with the /etc/init.d/openais script.

This script starts the corosync cluster engine /usr/sbin/corosync.

Cluster Resource Manager (CRM)

The CRM manages the resources, decides which resource is to run where and

ensures that the required status of the cluster is achieved based on the current

status. For this purpose the CRM distributes the work to the LRM and receives

feedback from the latter.

Local Resource Manager (LRM)

The LRM manages the resources on "its" local nodes. The CRM tells the LRM which

resources they are. The LRM communicates with the CRM.

Cluster Information Base (CIB)

The CIB is the central information file for all the resources and nodes of the cluster. It

not only contains information about the static configuration but also about a dynamic

part of the current status of all resources. The data is stored in XML format.

corosync.conf

This configuration file controls the behavior of the cluster software. It is needed to

start pacemaker. For example, it defines the interface via which communication is to



take place in the cluster and the nodes which belong to the cluster. It must be available on every node under /etc/corosync.

authkey

The file authkey is used to authenticate the nodes of a cluster for one another. This file must also be available on every node under /etc/corosync.

2.5.2 Simple Resources

2.5.2.1 Constraints

Simple resources can be linked with constraints, which are triggered when the resource is

started. These are:

Ordering:

specifies the relationship of two resources to each other. This means that you can

only start resource B after resource A.

Collocation:

ensures that two resources are started on the same node.

Location:

defines the node where a resource is to preferably run.

2.5.2.2 Score

Constraints are defined by rules. A system of points is used to decide on which node a

resource is to run. Each applicable rule is linked to a score. In addition to normal inte-

gers, it can also accept special values –INFINITY and INFINITY.

Example:

Resource A is to preferably run on CN1. The rule: NODE eq CN1

Score: 10

If A would run on CN1, the rule would return “true” and CN1 would get the score 10.

If A would run on CN2, the rule would return “false” and CN2 would not get any score,

thus 0.

After all possibilities being evaluated by the cluster, CN1 has a greater score than CN2,

thus CN1 would be chosen to run A.

The extremes:



INFINITY: The resource is started in any case on the node selected by an applicable rule.

If it cannot be started on the specified node, an attempt is made to start it on another

node.

-INFINITY: The resource is not started on the node selected by an applicable rule. If it

cannot be started on any other node than the selected one, it remains stopped.

In other words:

INFINITY ~ is applicable if possible

-INFINITY ~ is never applicable

2.5.2.3 Stickiness

Another value that plays an important role in the decision about the resource where the

node is to run is stickiness. You can regard this value as the level of adhesion, with

which a resource aims to remain on the node on which it is currently running. This value

is defined via the global settings of the CIB by the variable default-resource-

stickiness. The following values are possible.

0 :

Linux-HA attempts to optimally place the resource in the cluster.

In other words, the resource is redistributed in the cluster if Linux-HA finds a more

"suitable" node.

> 0 :

Linux-HA attempts to optimally place the resource in the cluster.

In other words, the resource is redistributed in the cluster if Linux-HA finds a more

"suitable" node.

< 0 :

The resource tends to leave the current node. Lower values reinforce this tendency.

INFINITY :

The resources remain on the current node until they are compelled to leave it, e.g.

when their node is shut down.

-INFINITY :

The resources do not want to remain on the current node.

The two values, score and stickiness, determine the node on which a resource is then

actually started. For this purpose, they are determined and added for each node of the

cluster. The resource is then started on the node that has the highest score.



Example:

Default_resource_stickiness : 100

Resource A is to be preferably run on CN1

Rule: NODE eq CN1

Score: 10

Case 1: A already runs on CN1

Points for CN1 :

Location score =10

Stickiness score = 100

Resulting score CN1 =110

Points for CN2 :

Location score = 0 (rule not met)

Stickiness Score = 0 (because it is currently not running on CN2)

Resulting score CN2 = 0

Result: A remains on CN1, score CN1 > score CN2

Case 2: A already runs on CN2

Points for CN1 :

Location score = 10

Stickiness score = 0 (as not running on CN2)


Points for CN2 :

Location score = 0



Result: As score CN2 > score CN1, resource A remains on CN2, although its location

score expresses the wish to move to CN1.



Case 3: A does not run on any node (e.g. during cluster start)

Points for CN1 :

Location score =10



Points for CN2 :

Location score = 0 (rule not met)



Result: A starts on CN1, score CN1 > score CN2

In FlexFrame all resources except the ldap services are configured in such a way that

they remain on their current node after a move, even when the other control node is

available again after a downtime. This avoids unnecessary switching processes.

2.5.2.4 Resource Groups

Simple resources can be put together to form groups. It is also possible to create so-

called clones, in which simple resources may run repeatedly in the cluster. Constraints,

such as ordering or colocation, which are defined for the group, apply for all the re-

sources belonging to the group.

2.5.3 FlexFrame Specific Configuration

The basic configuration for pacemaker is created during the initial configuration of

FlexFrame, i.e. the file corosync.conf is set up, a key to authenticate the nodes CN1

and CN2 is created for the file authkey and the CIB is created.

It contains the following resources (type and name of the RA in brackets):

Resource Group: network_CN1_server

ip_CN1_server_<poolname-m> (ocf::heartbeat:IPaddr2)

Resource Group: network_CN2_server

ip_CN2_server_<poolname-m> (ocf::heartbeat:IPaddr2)

ldap_master (ocf::fsc:ff_ha_ldap)

ldap_replica (ocf::fsc:ff_ha_ldap)

Resource Group: netboot



dhcpd (lsb:ff_ha_dhcpd)

tftpd (lsb:ff_ha_tftpd)

Resource Group: ff_manage)

postgresql (ocf::heartbeat:pgsql)

myAMC.FA_Messenger (lsb:ff_ha_myAMC.MessengerSrv)

tomcat (lsb:ff_ha_tomcat)

myAMC.FA_CtrlAgent (lsb:ff_ha_myAMC.CtrlAgent)

Clone Set: clone_ClusterMon

ClusterMon:0 (ocf::heartbeat:ClusterMon)

ClusterMon:1 (ocf::heartbeat:ClusterMon)

stonith_ipmi_CN1 (stonith:external/ipmi

stonith_ipmi_CN2 (stonith:external/ipmi)

2.5.3.1 Configuration of Limits for Cluster Resources

In some large FlexFrame environments the default number of open files

(1024) is not sufficient to solve all requests to ldap or tomcat.

/var/log/messages show entries like:

... slapd[2393]: warning: cannot open /etc/hosts.allow: Too many open

files

... java.rmi.RemoteException: VI SDK invoke

exception:java.net.SocketException: Too many open files

In such a case the number of open files has to be increased before the

cluster resources are (re)started.

The problem only occurs in very large configurations (many pools with many Application

Nodes and a lot of big SAP applications with many active users permanently logging in).

Therefore instead of system-wide increasing of the default number of open files a

FlexFrame specific configuration template is provided. This template can be used to

create a custom specific file limiting the number of open files.

The mentioned template is the

/opt/FlexFrame/etc/ff_ha_settings.conf.template file. You can use it to

create your own configuration file with the settings suitable for your system:



At first copy the template file to /opt/FlexFrame/etc/ff_ha_settings.conf:

control1:~ # cp /opt/FlexFrame/etc/ff_ha_settings.conf.template

/opt/FlexFrame/etc/ff_ha_settings.conf

Now uncomment the line with the 'FF_ULIMIT_SETTINGS' variable. The example

below shows an entry to increase the number of open files up to 8192 (8K):

## Template for config file /opt/FlexFrame/etc/ff_ha_settings.conf

## Used to set special values for our linux ha resources like ldap or tomcat

## The resource scripts /opt/FlexFrame/lib/ha/scripts/ff_ha_* will look

## for active variables in the config file

##

## Copy the template file to /opt/FlexFrame/etc/ff_ha_settings.conf and

## uncomment the needed variables

## increase number of open files for resource ldap and tomcat

FF_ULIMIT_SETTINGS="-n 8192"

Finally the resources tomcat and ldap_master have to be restarted, if they are already

running.

control1:~ # ff_ha_cmd.sh restart tomcat

control1:~ # ff_ha_cmd.sh restart ldap_master

You may verify the results of the changes by checking the limits before (should be 1024)

and after (should now be 8192) the restart of the resources:

cat /proc/<pid_of_tomcat>/limits

cat /proc/<pid_of_ldap>/limits

The configuration file /opt/FlexFrame/etc/ff_ha_settings.conf will be backed

up automatically by ff_cn_backup.sh during a FlexFrame upgrade, but has to be

restored manually after the FlexFrame upgrade has finished.

2.5.3.2 LDAP Configuration

The FlexFrame LDAP concept is implemented by the resource groups net-

work_CN1_server and network_CN2_server as well as the simple resources ldap_master and ldap_replica. Since this concept has dramatically changed with

the introduction of Linux-HA, it is explained in more detail here.



Each network_CN<n>_server (n=1..2) group consists of the simple resources

network_CN<n>_server_<poolname-m> (n=1..2)1. Each of these simple

resources is exactly one server LAN IP of a pool, i.e. the appropriate resource agent

(IPaddr2) accurately monitors a server LAN IP and, in the event of a fault, moves it to the

surviving node of the cluster. Accordingly, there are as many resources in the cluster as

there are server LAN IPs.

This ensures that the entire server LAN IPs of all the pools can always be accessed in

the network.

The resource ldap_master is initially started on the first control node. It starts the LDAP

server process slapd on the same control node with the server LAN IPs of all the pools

of the first control node as defined by the management tool. Analog to this, the resource

ldap_replica on the second control node is started with the appropriate server LAN

IPs of all the pools of the second control node.

This concept ensures that:

1. ldap_master und ldap_replica can always access precisely defined IP

addresses, i.e. the address of the server LAN

2. these IP addresses are managed by the cluster and are thus always available on a

cluster-wide basis

3. ldap_master und ldap_replica can also run in parallel on a node.

In other words, if a node fails, the cluster ensures that its IP addresses and the

appropriate ldap resource are switched over to the other node and are then available

again on a system-wide basis

Example:

2 Pools, pool1, pool2:

Server LAN IPs :

cn1-pool1-se 192.168.12.12

cn1-pool2-se 192.168.102.16

cn2-pool1-se 192.168.12.13

cn2-pool2-se 192.168.102.17

Resource group network_CN1_server contains the two simple resources:

1 "n" is an enumeration consisting of 2 elements. Element 1 for CN1 and element 2 for CN2.



network_CN1_server_pool1 manages 192.168.12.12


Resource group network_CN2_server with



The resource ldap_master starts on CN1 with the IP addresses 192.168.12.12 and

192.168.102.16

The resource ldap_replica starts on CN2 with the IP addresses 192.168.12.13 and

192.168.102.17

In comparison with other resources and resource groups the rule for the "location" con-

straint of the resource groups network_CN<n>_server (n=1..2) is given a score

(=100000) higher than the default-resource-stickiness value (=100). After a

move to the other nodes (e.g. after a control node failure or reboot) this causes the server

LAN and LDAP resources to return to their original nodes as soon as possible.

2.5.3.3 Resource ff_manage

The resource ff_manage is a group of simple resources that is initially started on the

first control node. These are:

postgresql

myAMC.FA_Messenger

tomcat

myAMC.FA_CtrlAgent

Prerequisite for the start of myAMC.FA_messenger is the successful start of the resource

postgresql. This is compelled by the constraint "orders".

Since the start of ff_manage can take longer, an attempt is made as a result of an "or-

ders" constraint to start the resources ldap_master and ldap_replica before the

start of the group ff_manage.

2.5.3.4 Resource Netboot

The resource group netboot contains the simple resources.

dhcp

tftpd

An attempt is also made for this group – as for the group ff_manage – to only start it if

the resources ldap_master and ldap_replica run successfully. The "orders"

constraint is also used for this.



2.5.3.5 STONITH (Physical Control Center)

The stonith agents' purpose is to prevent the cluster entering a non-defined status in the

event of a fault in communications. The corresponding stonith resources are:

stonith_ipmi_CN2

stonith_ipmi_CN1

Due to the "location" constraint stonith_ipmi_CN2 may – under no circumstances –

be started on CN2 and stonith_ipmi_CN1 may not be started on CN1.

These resources monitor the other node respectively. If, after a time interval set during

configuration, a node finds that communication with the other node is no longer possible,

a hard "reboot" or a "reset" is triggered on the inaccessible node via the IPMI interface.

The IP addresses of the IPMI interface as well as user and password were transferred to

the resource during configuration in order to enable this step.

For the reboot to take place via the ipmi interface the latter must be accessible. Therefore stonith_ipmi_CN<n> (n=1..2) monitors the respective interface every 30 seconds. If it

is determined that the interface does not answer (e.g. in the event of a power failure),

pacemaker sets the state of the node in question to UNCLEAN (offline). In this case the

administrator should

1. Call crm node clearstate <node which cannot be reached>

This command sets the state of that node from UNCLEAN (offline) to offline so

that pacemaker is able to take over the resources fom that node.

2. Shutdown / restart that node

Example after power failure of CN2 :

Using the command crm_mon -1 -r -f -n provides an overview of the error counters

and inactive resources:



Calling : crm node clearstate cn2

The state of cn2 was cleared and all resources which are running on cn2 are switched

over to cn1.



1. If the hardware of the node in question was not changed no manually

intervention is required during the reboot.

2. After the reboot pacemaker will start all services but ff_manage on their

default node. Please be patient because this process will take some

time.

2.5.3.6 STONITH (Virtual Control Center)

In case of a virtual Control Center you will find the following stonith resources:

stonith_ipmi_CN2

stonith_vm_CN2

stonith_ipmi_CN1

stonith_vm_CN1

Due to the "location" constraints stonith_ipmi_CN2 and stonith_vm_CN2 are

running on CN1 only. stonith_ipmi_CN1 and stonith_vm_CN1 are running on CN2

only.



stonith_vm_CN2 (running on virtual Control Node CN1) monitors virtual Control Node

CN2 (stonith_vm_CN1 running on CN2 does the same with CN1).

For simplicity we regard from now on only the case that virtual Contol Node CN2 gets into

trouble.

If, after a time interval, the monitored virtual Control Node CN2 is supposed to be dead, stonith_vm_CN2 tries to restart CN2 using libvirt. If this is successful, everything is fine

again.

If the restart of CN2 fails due to an issue on the Hypervisor Node, stonith_ipmi_CN2

becomes active. All IP addresses of FlexFrame virtual Application Nodes that run in

parallel on the affected Hypervisor Node hosting CN2 are known to stonith_ipmi_CN2. A script called by the resource checks via ping if there are still

virtual Application Nodes left that are alive (on the Hypervisor Node hosting CN2). In

case of still living virtual Application Nodes no further action is taken by stonith_ipmi_CN2. If all virtual Application Nodes are judged dead, the Hypervisor

Node hosting CN2 is killed via IPMI interface.

2.5.3.7 Configuring the Cluster

The cluster is automatically configured by the script ff_post_conf.sh during the instal-

lation of FlexFrame. If, in certain situations, it is necessary to subsequently configure the cluster, the command ff_ha_tool.sh [-f] -i must be used. During the setup the

IPMI user and IPMI password are queried if they are not yet known or access is not pos-

sible with the known user and password.

Syntax :

ff_ha_tool.sh [-f] -i

-i: initial configuration of linux-ha for FlexFrame

-f: force execution, purge old configuration

2.5.3.8 Starting Behavior of the Cluster

The cluster is started:

automatically when FlexFrame is installed via the installation scripts

with each reboot

with the command /etc/init.d/openais start

Two cases should be observed both when rebooting and during a manual start:

1. One node is affected – either it has failed or the service openais was stopped

manually. The second node subsequently took on the resources and has the

role of the DC.



2. Both nodes are affected.

In case 1 there is still nothing to be considered. After the node is online again or the clus-

ter could be successfully started on this node, the resources are redistributed according

to their score values and started.

If the cluster has to be restarted on both nodes (case 2) – either through rebooting or

manually – the following must be taken into consideration:

During the start of a node the service openais attempts to communicate with the other

node. If there is no answer after a configurable time interval, the stonith agent is

activated, which via the IPMI interface then causes the second node to reboot. The time

interval is configured in the CIB by the parameter dc-deadtime and is currently

120seconds.

Particularly when switching on an entire FlexFrame environment it is

essential to ensure that both control nodes have to be switched on very

quickly one after each other. If the delay is too great, the control node

switched on last may be reset hard by the stonith agent, but this usually has

no negative effects.

If in this situation the IPMI interface of the other node is not accessible, the stonith IPMI

agent would - after a vain attempt to trigger a reboot on the node - change the state of

that node to UNCLEAN (offline). Then the operator must proceed as described in the

section "STONITH" so that the resources monitored by the cluster can be started.

2.5.3.9 Status Information of the Clusters

Various options can be used to find out the current status of the cluster. One of which is

implemented by the resource agent ClusterMon. It is the task of this RA to regularly

transfer the current status of the cluster to a HTML file. For this purpose, the clones ClusterMon:0 and ClusterMon:1 are configured which are each started on a

control node. They write the status in /srv/www/htdocs/ClusterMon.html, which can be

accessed via the local web server at http://localhost/ClusterMon.html and is also linked

from the default homepage of the web server.

2.5.4 Linux-HA CLI Commands

Various options are available to achieve a required action. For this purpose, FlexFrame provides a simple CLI interface to Linux-HA. The command ff_ha_cmd.sh, which is

internally based on the Linux-HA CLI commands, was provided for the most important ac-

tions, such as status display, starting and stopping a Resource.

Action Command

http://localhost/ClusterMon.html



Action Command

Display the status of all resources ff_ha_cmd.sh status

Display the status of one resource ff_ha_cmd.sh status <resource>

Start a resource ff_ha_cmd.sh start <resource>

Stop a resource ff_ha_cmd.sh stop <resource>

Migrate (move) a resource to another

node – this creates a special location

constraint

ff_ha_cmd.sh migrate <resource>

<node>

Undo the migration – the special

location constraint is deleted ff_ha_cmd.sh unmigrate <resource>

Cleanup a resource, delete all status

flags – this will restart the resource if

possible

ff_ha_cmd.sh cleanup <resource>

Remove a resource from the

administration using pacemaker.

Resources will not be automatically

started, stopped or restarted

ff_ha_cmd.sh unmanage <resource>

Reintegrate a resource in the

administration using pacemaker ff_ha_cmd.sh manage <resource>

Output the CIB (XML format) cibadmin -Q

List all the resources of the cluster crm_resource -L

crm_resource -l

List a resource (XML format) crm_resource -x -r <resource>

Output the node, on which a resource

runs

crm_resource -W -r <resource>

ff_ha_cmd.sh status resource

Regular status display (interval: 15

sec)

crm_mon

Regular status display, interval <n>

sec

crm_mon -i <n>

Once-only status display crm_mon -1

[Once-only] status display with output

of the error counters of all resources

as well as all offline resources

crm_mon [-1] -f -r

Network FlexFrame Architecture


Action Command

As above, but grouped by nodes crm_mon [-1] -f -r -n

Add resources based on a

configuration template

ff_ha_tool.sh –a <template>

[param=value]

Remove resources based on a

configuration template

ff_ha_tool.sh –d <template>

[param=value]

List available and configured

configuration templates

ff_ha_tool.sh –a –l

If a resource is to be migrated to another node, it is essential that this action creates a

"Location" constraint with the score INFINITY. However, this means that this resource

also remains permanently on this node even after a complete restart of the cluster and is

therefore not moved – unless the node has a problem to the effect that no resources can

run any more. This constraint can be removed with the command ff_ha_cmd.sh un-

migrate <resource>, which nevertheless does not cause the resource to automati-

cally move back; it merely removes the constraint of the preferred node as a result.

2.5.5 Linux-HA Logfile

All messages that concern Linux-HA are written in the log file /var/log/ha-log .

2.5.6 Linux-HA GUI

Linux-HA has its own GUI which can be launched using the "hb_gui" command. However, it is strongly recommended not to use this interface to change configuration parameters or control resources, because each change hides the risk of negatively influencing or even severely damaging the FlexFrame cluster configuration. There also may be situations where it is not possible to use the command line interface after having used the "hb_gui" interface. Please use the command line interface ff_ha_cmd.sh exclusively to control resources.

2.6 Network

The network is the backbone of the FlexFrame solution. Communication between the

various nodes and storage devices is done exclusively via the IP network infrastructure. It

serves both, communication between servers and clients as well as delivering IO data

blocks between the NAS (Network Attached Storage) and the servers.

The IP network infrastructure is essential for every FlexFrame configuration. FlexFrame is

designed with a dedicated network for connections between servers and storage that is

FlexFrame Architecture Network


reserved for FlexFrame traffic only. One network segment, the Client LAN (see below)

can be routed outside the FlexFrame network to connect to the existing network.

2.6.1 LAN Failover

The term „LAN failover“ describes the ability of a FlexFrame environment to use a logical

network interface that consists of several physical network interface cards (NICs), which

in turn are using redundant network paths (cables and switches). When a network

component (NIC, cable, switch, etc.) fails, the network management logic will switch over

to another network interface card and path.

2.6.2 Segments

FlexFrame uses a network concept providing high availability as well as increased

flexibility in virtualizing the whole FlexFrame landscape.

The FlexFrame network concept relies on VLAN technology that allows running multiple

virtual networks across a single physical network. Additionally, in order to ensure high

network availability, LAN bonding is used on every node. This includes a double switch

and wiring infrastructure, to keep the whole environment working, even when a network

switch or cable fails. Within FlexFrame there are four virtual networks. These networks

run through one logical redundant NIC, using bonding on Linux.

Client LAN

The purpose of the Client LAN segment is to have dedicated user connectivity to the

SAP instances. This segment also allows administrators to access the Control

Nodes.

Control LAN

The Control LAN segment carries all administrative communication, such as IPMI or

similar interfaces.

Server LAN

The Server LAN segment is used for the communication between SAP instances

among each other and the databases.

Storage LAN

The Storage LAN segment is dedicated to NFS communication for accessing the

Application Nodes' shared operating systems, the executables of SAP and the

RDBMS as well as the IO of the database content and SAP instances.

The following figure outlines the basic network segments of a typical FlexFrame

landscape with Application Nodes.



2.6.3 Network Switches

Network switching components play a very important role within FlexFrame. Therefore,

only designated switch types are tested and supported (see FlexFrame Support Matrix

https://partners.ts.fujitsu.com/com/products/infrastruc-solutions/FlexFrame or ask your

Fujitsu Customer Support for getting a copy).

All supported switch models support VLAN technology for a flexible configuration for the

various network segments.

2.6.4 Network Speed

FlexFrame supports network connections for data communication with the following

network speeds (see also: FlexFrame® – Network Design and Configuration Guide):

1Gbit/sec (1GbE)

10Gbit/sec (10GbE)

https://partners.ts.fujitsu.com/com/products/infrastruc-solutions/FlexFrame



To determine the 10GbE use case for end systems which are able for both, 1GbE or 10GbE network connection, use the --10gbit option when adding the device to the

FlexFrame configuration.

For further details about supported end systems see the FlexFrame Support Matrix.

2.6.5 Network Switch Groups

In FlexFrame network switches are grouped. Ports of an end system are building a

redundant connection and are connected to ports of different members of the same

switch group (see also: FlexFrame® – Network Design and Configuration Guide).

The switch groups within the entire FlexFrame environment are numbered starting with 1.

Also the members within a switch group are numbered starting with 1 in each case. In

special cases the use of switch group number 0 (zero) is allowed. This switch group is

neither registered in nor managed from FlexFrame. It helps to identify network devices

outside but with relation to the FlexFrame environment (see also next section Network

Switch Ports).

2.6.6 Network Switch Ports

In FlexFrame network switch ports are uniquely identified within the entire FlexFrame

environment by:

the number of the switch group

the number of the member within the switch group

the port ID within the switch

The portID is

a single number as for Cisco Catalyst 3750 switch fixed ports

a prefixed number

o <slot number>/<port number> as for Cisco Nexus switch ports

o G1 ... G4 as for Cisco Catalyst 3750x 1GbE module

o TE1, TE2 as for Cisco Catalyst 3750x 10GbE module

o F1 ... F4 as for Brocade ICX SFP ports

In case of Cisco Catalyst 3750e the numbers 1 and 2 are used both for GigabitEthernet

Ports and TenGigabitEthernet Ports. To distinguish both cases a --10gbit Option is used

on input and on output (10G) is appended.

Switch ports belonging to the unspecific switch group 0 (zero) are meant to be

somewhere outside the FlexFrame environment.



2.6.7 Automounter Concept

The Automounter Concept is based on the ability of Linux to mount file systems

automatically when their mount points are accessed.

During the boot process of an Application Node some file systems are mounted. For

Linux these are the root file system (read-only) as well as the /var file system (read-

write). These are the basic file systems which must be accessible for the Application

Node to function properly. There is no data in the two file systems that is specific for a

SAP service.

Data which is specific for a SAP service, a database or a FlexFrame Agent is found in

directories which are mounted on first access. Some of the mounts will stay as long as

the Application Node is operational. Others will be unmounted again, if directories or files

below that mount point have not been accessed for a certain period of time.

Within the LDAP database there are two types of data which relate to the automounter

configuration: automountMap and automount.

An automountMap is a base for automount objects. Here's how to list the

automountMaps:

control1:~ # ldapsearch -x -LLL '(objectClass=automountMap)'

dn: ou=auto.FlexFrame,ou=Automount,ou=pool2,ou=Pools,ou=FlexFrame,

dc=flexframe,dc=wdf,dc=fujitsu,dc=com

objectClass: top

objectClass: automountMap

ou: auto.FlexFrame

...

The base directory looks like this:

dn: cn=/FlexFrame,ou=auto_master,ou=Automount,ou=pool1,ou=Pools,

ou=FlexFrame,dc=flexframe,dc=wdf,dc=fujitsu,dc=com

objectClass: top

objectClass: automount

cn: /FlexFrame

automountInformation: auto_FlexFrame

Further on there are entries like:

dn: cn=myAMC,ou=auto_FlexFrame,ou=Automount,ou=pool1,ou=Pools,


objectClass: top


cn: myAMC



automountInformation:

-rw,nointr,hard,rsize=32768,wsize=32768,proto=tcp,nolock,

vers=3 filpool1-st:/vol/volFF/pool-pool1/pooldata/&

There are two things that have to be pointed out.

First, the ou=auto.FlexFrame refers to the base directory as shown before.

The second notable aspect in this entry is the use of the wildcard &. If the folder

/FlexFrame/myAMC is accessed, the autofs process tries to mount it from the path

filpool1-st:/vol/volFF/pool-pool1/pooldata/myAMC. If the folder myAMC is

found and permissions allow the clients to access it, it will be mounted to

/FlexFrame/myAMC/<name>. If myAMC is not found or the client does not have the

permissions, the folder will not be mounted. In such a case, try to mount the folder

manually on a different folder, e.g. like:

an_linux:~ # mount filpool1-st:/vol/volFF/pool-pool1/pooldata/myAMC /mnt

If you get an error message like Permission denied, check the exports on the NAS

system and the existence of the directory myAMC/ itself.

Other entries in LDAP make use of platform specifics. With Linux you can find a number

of variables like ${OSNAME}/${OSDIST}/${ARCH} to make a distinction between

different platforms.

dn: cn=/,ou=auto.oracle,ou=Automount,ou=pool2,ou=Pools,


objectClass: top


cn: /

description: catch-all for Linux automount

automountInformation:

-rw,nointr,hard,rsize=32768,wsize=32768,proto=tcp,nolock,

vers=3 filpool2-st:/vol/volFF/pool-pool2/oracle/${OSNAME}/${OSDIST}/${ARCH}/&

On Linux, the automount mount points can be read using the following command:

an_linux:~ # mount

rootfs on / type rootfs (rw)

/dev/root on / type nfs (ro,v3,rsize=32768,wsize=32768,reserved,

hard,intr,tcp,nolock,addr=192.168.11.206)

192.168.11.206:/vol/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64/var_img/var-

c0a80b36

on /var type nfs (rw,v3,rsize=32768,wsize=32768,reserved,


192.168.11.206:/vol/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64/var_img/var-

c0a80b36

Storage Systems FlexFrame Architecture


/dev on /dev type nfs (rw,v3,rsize=32768,wsize=32768,reserved,


192.168.11.206:/vol/volFF/os/Linux/pool_img/pool-c0a80bff on /pool_img type nfs

(rw,v3,rsize=32768,wsize=32768,reserved,


proc on /proc type proc (rw)

devpts on /dev/pts type devpts (rw)

shmfs on /dev/shm type shm (rw)

/dev/ram on /var/agentx type ext2 (rw)

automount(pid1750) on /FlexFrame type autofs (rw)

automount(pid1772) on /saplog/mirrlogA type autofs (rw)

automount(pid1752) on /home_sap type autofs (rw)

automount(pid1788) on /saplog/saplog1 type autofs (rw)

automount(pid1766) on /sapdata/sapdata5 type autofs (rw)

automount(pid1778) on /saplog/origlogA type autofs (rw)



automount(pid1784) on /saplog/saparch type autofs (rw)


automount(pid1786) on /saplog/sapbackup type autofs (rw)


automount(pid1754) on /myAMC type autofs (rw)

automount(pid1796) on /usr/sap type autofs (rw)

automount(pid1780) on /saplog/origlogB type autofs (rw)


automount(pid1792) on /saplog/sapreorg type autofs (rw)

automount(pid1776) on /saplog/oraarch type autofs (rw)

automount(pid1774) on /saplog/mirrlogB type autofs (rw)

automount(pid1770) on /sapdb type autofs (rw)

automount(pid1756) on /oracle type autofs (rw)

automount(pid1790) on /saplog/saplog2 type autofs (rw)

automount(pid1794) on /sapmnt type autofs (rw)

filpool2-st:/vol/volFF/pool-pool2/pooldata on /FlexFrame/pooldata type nfs

(rw,v3,rsize=32768,wsize=32768,reserved,hard,tcp,

nolock,addr=filpool2-st)

The cn: parts show the mount points.

2.7 Storage Systems

FlexFrame uses a central NAS system to store data needed by the FlexFrame system

and its components. Some data areas of a FlexFrame landscape can be distributed to

other NAS systems.

FlexFrame Architecture Storage Systems


It is also possible to use SAN storage systems for the database datafiles and database

logfiles of SAP systems with demand for higher performance and throughput. Everything

else will still be placed on NAS storage. This explicitly includes the database software,

SAP applications and the OS. Even the database files do not necessarily have to be

placed onto a SAN based storage. One might choose to place large, productive

databases with IO requirements onto the SAN based storage while smaller databases will

be on NAS storage. The choice is based on SID level. One SID might be on SAN and

another SID on NAS.

The following pictures illustrate the components which can be placed onto NAS and SAN

storage.

FlexFrame Components on NAS

FlexFrame Components on SAN

2.7.1 NAS systems

In FlexFrame the storage for all Application Nodes can be one or more NAS systems

(Filers) from Network Appliance, Inc.. The Filer can be connected with 10Gbit/sec or

1Gbit/sec NICs to the FlexFrame system. The parallel use of 1Gbit/sec and 10Gbit/sec

NICs of one Filer is not possible. But you can use on one filer 1Git/sec NICs and on

another 10Gbit/sec NICs.

The operating system of the Filer is called "ONTAP". The disks will be grouped into RAID

groups. A combination of RAID groups will make a volume. Starting with ONTAP 7, also

aggregates can be created. FlexVolumes can be created on top of aggregates. A Filer

volume contains a file system (WAFL - Write Anywhere File Layout) and provides

volumes or mount points for NFS (for UNIX systems) or CIFS (for Windows systems).

The Filer has NVRAM (Non Volatile RAM) that buffers committed IO blocks. The contents

of the NVRAM will remain intact if the power of the Filer should fail. Data will be flushed to

the disks once power is back online.

The minimal FlexFrame landscape has at least the following volumes:

vol0 (ONTAP, configuration of Filer)

sapdata (database files)

saplog (database log files)



volFF (OS images of Application Nodes, SAP and database software,

pool related files)

In FlexFrame, the volume volFF separates FlexFrame data (file system of Application

Nodes and other software) from the Filer's configuration and ONTAP. In larger

installations, multiple sapdata and saplog volumes can be created (e.g. to separate

production and QA etc.).

As of FlexFrame 5.2A the three volumes names volFF, sapdata, saplog are no

more fixed names and may be established as required via Planning Mode of the

Management Tool (chapter “Storage” Object). The given names are the FlexFrame

default names.

In this document the volume names vol0, volFF, sapdata, saplog always refer

to the volumes playing the role mentioned above.

2.7.1.1 Built-in Cluster File System

The Network Appliance implementation of the NFS (Networked File System) allows

sharing of the same data files between multiple hosts. No additional product (e.g. cluster

file system) is required.

2.7.1.2 Volume Layout

The FlexFrame concept reduces the amount of "wasted" disk space since multiple SAP

systems can optionally share the same volume of disks. As the data grow, it is easy to

add additional disks and enlarge the volumes without downtime.

2.7.1.3 Snapshots

When a snapshot is taken, no data blocks are being copied. Just the information where

the data blocks are located is saved. If a data block is modified, it is written to a new

location, while the content of the original data block is preserved (also known as “copy on

write”). Therefore, the creation of a snapshot is done very quickly, since only few data

have to be copied. Besides that, the snapshot functionality provided by NetApp is unique,

because the usage of snapshots does not decrease the throughput and performance of

the storage system.

Snapshot functionality will allow the administrator to create up to 250 backup-views of a

volume. The functionality “SnapRestore” provided by NetApp significantly reduces the

time to restore any of the copies if required. Snapshots will be named and can be re-

named and deleted. Nested snapshots can be used to create e.g. hourly and daily

backups of all databases. In a FlexFrame landscape, a single backup server is sufficient

to create tape backups of all volumes. Even a server-less backup can be implemented.

Off-Line backups require a minimal down-time to the database because the backup to

tape can be done reading from a quickly taken snapshot. Concerning backup using

snapshots please also have a look at 3.6 FlexFrame Backup with Tape Library.



2.7.1.4 vFiler

With activation of a Multistore license a Filer can only support vFiler. Every vFiler may

have its own administrator. A special vFiler named vfiler0 is automatically created and

cannot be deleted. Initially all resources are owned by vfiler0. Only the vfiler0

administrator can create or delete vFiler, volumes and interfaces and assign volumes and

interfaces to vFiler.

As of FlexFrame 5.2 an arbitrary vFiler is allowed to be used as NAS System in the

FlexFrame environment. Other vFiler on the same storage host can be used for other

purposes as far as the sizing requirements are observed. If in doubt consult the FTS

sizing team.

2.7.1.5 Storage Virtual Machines (SVMs)

Together with ONTAP 8.0 NetApp introduced its new paradigma of a Storage Virtual

Machine (SVM) with its “Clustered Data ONTAP (cDOT)”, which is on the way to replace

the conventional ONTAP 7-mode, and which is the basis for NetApp’s scale-out

architecture.

As of FlexFrame Orchestrator 1.1A Clustered Data ONTAP is supported, and the 7-mode

DATA ONTAP support is continued as well.

The storage virtual machine also supplies the foundation for the current NetApp multi-

tenant architecture. From FlexFrame’s perspective an SVM is a NAS system like any

FASxxxx or a vFiler.

The SVM may have a single logical interface (lif) per pool and for all lifs the home

port is created as VLAN port based on a single virtual interface group (ifgroup) consisting

of multiple physical interfaces. The home node is given as parent parameter when adding

the SVM to the FlexFrame landscape. Before adding an SVM the home node and its

partner node must be defined what in turn needs that the cluster is already defined as

NAS system of type “cDOTCluster”:

It is helpful to know the 3 types of vservers because you need to define and/or configure

them all:

admin vserver

This is commonly called the “Cluster”. Even if there is no adminstrative access

given by the owner the Cluster has to be defined. In FlexFrame it has the type

“cDOTCluster”.

node vserver

These are the real nodes a cluster consists of. In FlexFrame they have the type

of the nodes i.e. “FASxxxx”. Each node must have a partner in the Cluster in

order to establish an HA pair. The Cluster has to be defined as the parent.

data vserver



This is actually the “SVM” hosting the data and providing file services. In

FlexFrame it has the type “SVM”. FlexFrame has full administrative access to

the SVM. One of the nodes has to be defined as the parent.

The term cluster has been used historically to refer to an HA pair running

Data ONTAP 7-Mode. This usage has been discontinued, and HA pair is the

only correct term for this configuration. The term cluster now refers only to a

configuration of one or more HA pairs running clustered Data ONTAP.

Example: To configure an SVM mysvm on a HA pair (2 nodes called node1 and node2)

belonging to the cluster newcluster the following steps are needed:

cn1:~ # ff_nas_adm.pl --op add --name newcluster --type cDOTCluster \

--osType cDOT --interface a1a

cn1:~ # ff_nas_adm.pl --op add --name node1 --type FASxxx \

--osType cDOT --parent newcluster --interface a1a

cn1:~ # ff_nas_adm.pl --op add --name node2 --type FASxxx \

--osType cDOT --parent newcluster --interface a1a

cn1:~ # ff_nas_adm.pl --op partner --name node1 --partner node2

cn1:~ # ff_nas_adm.pl --op add --name mysvm --type SVM \

--parent node1 --interface a1a

Please refer to chapter 8.1.1 “Adding a New NAS” for details.

Independent from the ONTAP variant a Filer can be “clustered” via a HA pair to protect

data against the failure of a single Filer. Switching from one Filer to its HA pair

counterpart is transparent to the FlexFrame Application Nodes. HA pair Filer “clustering”

is a functionality of NetApp Filers.



2.7.2 SAN Support

As of FlexFrame Orchestrator 1.1. it is possible to use SAN based storage for the

database datafiles (sapdata) and database logfiles (saplog) of a SAP system.

Due to the more advanced integration into the FlexFrame functionality and the easier

setup and handling of the NAS based storage, it is recommended to use SAN based

storage only for the database servers for which the performance achievable with NAS

storage is not sufficient.

2.7.2.1 SAN Basic Layers

Configuring a SAN environment for one or multiple SAP databases can be a very

complex task. As with many complex tasks they become easier if you break it down into

little pieces. In order to make you understand the various layers of a SAN environment

we divide it along the path as seen from the disk to the database:

Level Description

DISK The real disk(s) contained in the storage system

RAID-GROUP One or multiple disks grouped together based on RAID

mechanisms provided by the storage system

STORAGE-LUN LUNs (or storage volumes) as seen by the storage system

SAN-FABRIC A SAN fabric to connect one or multiple storage systems with

one or multiple hosts. It consists of fibre channel cabling and

switches

HBA The host bus adapter which connects the host to the SAN

fabric

OS-DRIVER Basic OS drivers to make the HBA(s) work

HOST-LUN LUNs as seen by the host (Application Node)

MULTIPATHING OS specific multipathing software to make multiple HBAs to

work as a single virtual interface

VOLUME MANAGER An OS specific volume manager to group multiple LUNs from

the same or different storage systems into volume groups

FILE SYSTEM OS specific file systems created on top of the logical volumes

provided by the volume manager

DATABASE The database files contained on the file system(s)

Each of the layers above can be configured in many different ways using lots of

variations. Therefore not all of those layers are controlled by FlexFrame.



2.7.2.2 Rules and Restrictions

This section lists some rules and restrictions which must be observed when setting up a

SAN configuration for FlexFrame usage.

4. All Application Nodes within one pool group must be configured equally. This

includes the number and type of HBAs. They must also use the same OS image.

5. All Application Nodes within one pool group must have access to the same set of

LUNs of each involved storage subsystem. Each Application Node of a pool group

must be able to access the same LUNs without reconfiguring the access settings of

the storage subsystem.

6. Host based mirroring can be used on volume manager layer to mirror one or multiple

LUNs from one storage subsystem to another (e.g. for a two-site-concept). Copy

mechanisms of the storage subsystems can also be used. If the failover to a different

storage subsystem changes the LUN identification data, the LUN definition in the

FlexFrame LDAP database must be changed accordingly.

7. A LUN is dedicated to a volume group. A volume group may contain more than one

LUN, but a LUN must not be shared between volume groups.

8. A volume group is dedicated to a SID. A SID with SAN support (or a

SID/instance/nodeid tuple, in case of HANA) uses two volume groups, one for

sapdata and one for saplog. A volume group must not contain volumes for more than

one SID.

The appropriate volume groups are switched from one Application Node to

the next if a database is moved.

9. Access to all file systems of a SID must be tested on each Application Node of its

pool group before it can be monitored and used (“watch”).

10. A database which was installed using SAN based database files in one pool group

cannot be started in a different pool group. If this is required the database must be

installed on NAS.

FlexFrame Basic AdministrationAccessing a FlexFrame Landscape (Remote Administration)


3 FlexFrame Basic Administration

3.1 Accessing a FlexFrame Landscape (Remote Administration)

A FlexFrame landscape can be accessed through Secure Shell (ssh, scp) connections

to the Control Node. Any other remote administration tools like rsh or telnet have been

disabled for reasons of security.

3.2 Powering up the FlexFrame Landscape

If the complete FlexFrame landscape was powered off, the following power-on sequence

is recommended:

Powering off the FlexFrame Landscape FlexFrame Basic Administration


Before an Application Node can be booted, the NTP server must be set up correctly. This

can be verified by running the following command:

control1:~ # ntpq -p

remote refid st t when poll reach delay offset jitter

=================================================================

*LOCAL(0) LOCAL(0) 5 l 45 64 377 0.000 0.000 0.004

control2-se control1-se 7 u 248 1024 377 2.287 2.587 0.815

This command needs to be repeated until an asterisk (*) is displayed at the beginning of

one of the data lines. This character indicates that the NTP server is now ready. If you do

not wait and continue with booting, the Application Nodes may work with a different time

than the Control Nodes and may (among other possible side effects) create files which

may have wrong time stamp information.

If this sequence is not used and all servers are powered on at the same time, the

Application Nodes will try to boot while the Control Nodes are not ready to

receive the Application Node's boot request. If this is the case, manual

intervention is required to re-initiate the boot process of the Application Nodes.

3.3 Powering off the FlexFrame Landscape

If you need to power-off the complete FlexFrame landscape (e.g. to move it to a different

location) we recommend following the steps as outlined below:

FlexFrame Basic Administration Powering off the FlexFrame Landscape


Before shutting down all SAP and DB services, check if users, batch jobs, print

jobs and RFC connections to other SAP systems have finished working.

To shutdown the SAP and DB services, use the following command for each pool:

control1:~ # stop_all_sapservices <pool_name>

Before you can stop the NAS-System, you need to stop all processes on the Control

Nodes. Since those processes are under control of the Linux-HA cluster you have to

switch all resources from control node 2 to control node 1 and then stop all resources.

control1:~ # ssh control2 /etc/init.d/openais stop

control1:~ # /etc/init.d/openais stop

Keep in mind, stopping all resources will take some time.

Now stop all running FlexFrame agents, e.g. FlexFrame LogAgent, on both Control

Nodes.

control1:~ # ssh control2 /etc/init.d/myAMC.FA_LogAgent stop

control1:~ # /etc/init.d/myAMC.FA_LogAgent stop

control1:~ # /etc/init.d/myAMC.FA_FrameAgent stop

control1:~ # /etc/init.d/myAMC.FA_DomainManager stop

And umount all NFS file systems on both Control Nodes.

control1:~ # ssh control2 umount -a -t nfs

control1:~ # umount -a -t nfs

Now the NAS systems can be safely switched off without impacting the Control Center

shutdown later on.

To stop the NAS system, use the following command.

For a NetApp Filer:

control1:~ # ssh <filer_name> halt

There is no explicit power-off sequence for the switches. Assuming there are no

other devices connected to the switches, they may simply be plugged-off after all

components were powered down.

If you do not send an explicit halt command to the Filer, the backup-battery of

the Filer may be drained since the Filer assumes a power-loss and tries to

preserve the contents of its NVRAM. If the Filer is powered-off for too long the

result can be a loss of NVRAM data and a long waiting period during next

startup of the Filer.

Powering off the FlexFrame Landscape FlexFrame Basic Administration


FlexFrame Basic AdministrationReactivating ANs after Power Shutdown by FlexFrame Agents


3.4 Reactivating ANs after Power Shutdown by FlexFrame Agents

FlexFrame Autonomy places an Application Node out of service if it is confronted with a

problem it cannot solve. The reactions, messages and alarms which take place in this

case are described in the “FlexFrame Agents Installation and Administration” manual in

the context of the switchover scenarios. It is the responsibility of the administrator to

analyze why the node could not be used any longer. We recommend analyzing the FA

log and work files as these may be able to supply valuable information. A node which is

to start operating as a Spare Node after switchover must be validated using suitable test

scenarios.

It may happen in some cases that the "placing out of service" of an Application Node fails

(e.g. when the IPMI-interface has broken down) and so it cannot be decided if the

Application Node is down.

In these cases - as default - the services that were running on the server are not switched

over to another node.

Therefore there is an additional high availability FlexFrame Agents parameter

"IgnoreShutdownFailure" with impact on the failover reaction for services.

Setting this parameter to value "true"

assigns the FlexFrame Agents to deactivate the network interfaces of the server

if successful the FlexFrame Agents automatically switch over services that were

running on the server to another node

the failed server can only be reactivated by manually reactivating its network

interfaces via the command ff_sw_ports.pl

(see the chapters "Switchover Control Parameters" and "Shutdown Configuration" in the

FlexFrame Agents Installation and Administration manual).

Default value "false" means, that a not successful server shutdown has no effect on the

server network interfaces and the services that were running on the server are not

switched over to another node.

Setting this parameter strengthens the service high availability, but the FlexFrame

administrator has to be aware of its impact in certain failure scenarios.

Recommendation is to set the parameter "IgnoreShutdownFailure" to "true" after

installation of the Flex-Frame system, if the FlexFrame administration wants to expand

the high availability of service and is willing to manually reactivate the network interfaces

of a server in such error cases.

Displaying the Current FlexFrame Configuration State FlexFrame Basic Administration


3.5 Displaying the Current FlexFrame Configuration State

To obtain a general overview of an active FlexFrame system, use the FlexFrame Agents

WebGUI.

In principle, the FlexFrame Agents WebGUI can be used with every browser with SUN

JAVA Plugin V1.4.1 or higher which has access to the page on the Control Node. The

WebGUI can always be accessed when the Apache Tomcat service is running. This

service is normally started by the Linux-HA cluster.

The WebGUI is described in detail in chapter 3.5.1.

The login mask expects a user name and password for authentication purposes. You can

only use the WebGUI with a valid combination of user name and password. For details on

the configuration of the users, see the FlexFrame Agent documentation for the WebGUI.

The WebGUI provides a presentation of all elements of a FlexFrame system. On the left-

hand side the pools, groups, nodes and the active SAP services on the individual nodes

are shown in a TreeView.

FlexFrame Basic Administration Displaying the Current FlexFrame Configuration State


The TreeView of the WebGUI can show either the physical view or the application-related

view of the active SAP systems and their instances or a mixed view.

The panels derived from the TreeView (Application Server Panel, Application System

Panel, ServiceView and MessageView) always show the objects in relation to the

selected hierarchical level in the TreeView.

The FlexFrame Agents WebGUI is thus the central cockpit for displaying the static

configuration of a FlexFrame infrastructure, but as well it is the display for the active SAP

systems and their instances. Here, all user interactions such as startup and shutdown are

shown directly. All reactions initiate by the FlexFrame Agents are displayed as well.

If the FlexFrame Messenger component has been configured and activated, all traps are

stored in a support database. This permits the temporary process for the traps to be

displayed very simply at pool, group or node level.

3.5.1 FlexFrame Web Portal

The FlexFrame Control Nodes provide a web portal with links to Web interfaces of

several FlexFrame components, i.e.

FlexFrame Agents

Cluster status

Supported hardware

Management Blades of BX cabinets

To access this portal, start firefox and enter the Control Node's IP address in the location

bar. If you are directly on the Control Node you want to configure, just call firefox from the

shell:

control1:~ # firefox localhost

You will see an overview of all Web interfaces installed on the Control Nodes.



3.5.2 FlexFrame Agents

Use this tool to manage the virtualized services in your FlexFrame environment.

For information on usage, please refer to the FlexFrame Agents manuals.

You can access the FlexFrame Agents WebGUI directly from the active Control

Node by entering the following URL:

http://localhost:8080/FAwebgui

This only works if the Jakarta Tomcat web server is running. If it is not running,

check the Linux-HA cluster configuration.

3.5.2.1 State of Pools

The active configured pools are displayed on the FlexFrame Agents WebGUI. The nodes

or systems belonging to a pool are displayed in the WebGUI's TreeView. Each node element in the tree which represents a pool is identified by the prefixed keyword Pool.

3.5.2.2 State of Application Nodes

Each Application Node with a running FlexFrame Application Agent is shown in the

FlexFrame Agents WebGUI. It is shown in the Nodes TreeView with its host name

http://localhost:8080/FAwebgui

FlexFrame Basic Administration Displaying the Current FlexFrame Configuration State


(Linux), and also in the Application Server Panel. In the Application Server Panel the data

displayed depend on the hierarchical level selected in the TreeView.

3.5.2.3 State of SAP Systems

The active SAP system IDs can be displayed very easily in the Systems TreeView of the

FlexFrame Agents WebGUI and also in the SAP System Panel. The SAP System Panel

is shown in parallel to the Application Server Panel. In the SAP System Panel, the data

displayed depend on the hierarchical level selected in the TreeView.

3.5.2.4 State of SID Instances

The active instances of a SID can be displayed very simply in the InstancesView of the

FlexFrame Agents WebGUI by clicking on a SID in the SAP System Panel.

For each view, information is provided in tabular form specifying the service's current

pool, group, node, priority and status.

3.5.2.5 Virtual IP address for FlexFrame Agents WebGUI

The FlexFrame Agents WebGUI only works on the Control Node which is currently

running the ff_manage cluster resource group. This may cause confusion if this group is

switched over to the other Control Node.

It is possible to add a virtual IP address to the ff_manage resource group which will

ensure that the administrative access is always accessible by using a single address.

To add such an address, run the following command:

control1:~ # ff_ha_tool.sh -a ffmanageip ip=<address>

It is absolutely necessary to choose an address which meets the following criteria:

1. The address must be part of a subnet which already exists on the Control Node.

Ideally this is an address of the Client LAN.

2. The address must not exist before

Note that these critiera are not actively checked by the ff_ha_tool.sh command, the

address is simply being added to the cluster configuration. If the address cannot be

configured, the ff_manage group may be switched over to the other Control Node.

Verify the configuration and cluster state with ff_ha_cmd.sh status.

To remove a configured address, run:

control1:~ # ff_ha_tool.sh -d ffmanageip ip=<address>

It is possible to add multiple addresses, i.e. one for each Client LAN pool.



3.5.3 Networks

The network of FlexFrame is its backbone. Here are some tips to get an overview of the

current situation on the various networks:

To double-check the network addresses, their names and pool assignment you can use

the getent command:

control1:~ # getent networks

loopback 127.0.0.0

control 192.168.20.0

storage_pool1 192.168.10.0

server_pool1 192.168.3.0

client_pool1 10.1.1.0

storage_pool2 192.168.11.0

server_pool2 192.168.4.0

client_pool2 10.1.2.0

The loopback network is local for each host and always has the IP address 127.0.0.0.

The control network is the Control LAN network segment for the complete FlexFrame

landscape.

In the example we have configured two pools called pool1 and pool2. For each pool

there are the three dedicated and distinct segments storage, server and client. The

building rule of the network name is <segment>_<pool_name>.

On the Control Nodes you can see the relation of each pool specific segment to its interface using the netstat -r command like this:

control1:~ # netstat -r

Kernel IP routing table

Destination Gateway Genmask Flags MSS Window irtt Iface

server_pool2 * 255.255.255.0 U 0 0 0 vlan42

control * 255.255.255.0 U 0 0 0 bond0

server_pool1 * 255.255.255.0 U 0 0 0 vlan32

storage_pool2 * 255.255.255.0 U 0 0 0 vlan41

client_pool1 * 255.255.255.0 U 0 0 0 vlan30

storage_pool1 * 255.255.255.0 U 0 0 0 vlan31

client_pool2 * 255.255.255.0 U 0 0 0 vlan40

default gw216p1 0.0.0.0 UG 0 0 0 vlan30

Here you can quickly see that the Server LAN segment of pool2 (server_pool2) is using

the VLAN ID 42 on interface vlan42. Note that the VLAN ID of the Control LAN segment

cannot be seen because that vlan is native on bond0.


3.5.4 Cluster Status

A short overview of the cluster status is available from the FlexFrame Web Portal.

3.5.5 Supported Hardware

Clicking on the ‘Supported Hardware’ link in the FlexFrame Web Interface displays the

currently supported hardware and the features used by FlexFrame. The list includes

hardware that is supported only by an approved special release request.

If JavaScript is enabled, clicking on the table headers displays information about the

listed content.

FlexFrame Basic Administration FlexFrame Backup with Tape Library


3.5.6 Management Blades

Clicking on the links in the management blade section of the FlexFrame Web Interface

opens https connections to the respectice management blades of BX cabinets.

Note that the management blade section may be missing if the FlexFrame landscape

contains no BX cabinets.

3.6 FlexFrame Backup with Tape Library

Two differerent ways of backing up a database in a Flexframe environment have been

tested and are recommended. Both use third-party software, either Simpana (Comvault )

or Netbackup (Symantec), and both use different snapshot technology.

Corresponding best practice papers will be provided by Fujitsu Storage Consulting and

are available at https://partners.ts.fujitsu.com/com/products/infrastruc-

solutions/FlexFrame, see Best Practices / Description Papers.

See also

https://partners.ts.fujitsu.com/com/products/storage/software/Pages/default.aspx .

Concerning backup using snapshots there is a speciality with NetApp and

cDOT:

NetApp offers a popular function to access contents of snapshots by way of

filesystem, via the .snapshot directory.

In the cDOT architecture there is a change in the way directories are made

accessible for clients (or "are exported for clients" in the NFS terms) - in 7-

mode it was possible to make certain subdirectories accessible for certain

clients, whereas in clustermode (cDOT) it is only possible to make two levels

of directories accessible for clients, either whole volumes or qtrees (special

subdirectories) of volumes.

In order to make a qtree accessible for a client (= to export a qtree for a

client) it is necessary that the whole superior volume is accessible for the

client.

A problem arises, if snapshot directory access is allowed (when the volume

modify option -snapdir-access is set to true – default in Flexframe is

false).

Then this directory is visible on volume level, and snapshots in this directory

contain all qtrees of that volume, even if they belong to different clients

(application nodes), different pools, different mandants.

Then it would be possible that clients excess their border and are able to

access data of other mandants indirectly by accessing snapshots, which is



https://partners.ts.fujitsu.com/com/products/storage/software/Pages/default.aspx

Product Licenses FlexFrame Basic Administration


untolerable due to security reasons.

Thus the snapshot directory access must be prohibited, leading to the fact,

that a snapshot directory cannot be used by backup functions using NFS

mount.

This is a constraint in cDOT environments which only NetApp can set aside

by changing their export-authorization policies.

3.7 Product Licenses

Use of the FlexFrame Software requires specific software licenses depending on the

required functionality.

FlexFrame Orchestrator comes with a license concept where licenses (SID Runtime,

ESX-Server, KVM-nodes, HANA OLAP DB, RBAC, ACCOUNTING, SAN) have been

defined. The FFO SID license is the base runtime license for running FlexFrame

Orchestrator. Every other FFO license allows the additive use of a special functionality of

FlexFrame Orchestrator.

3.7.1 How to Work with Licenses

The customer orders all required FlexFrame licenses by contacting his sales

representative. Beside the license documentation the customer receives an amount

of license key files (one file for every ordered license).

The customer installs each license key in the FlexFrame using the script

ff_license.pl.

If the customer wants to use a feature which is subject to a license in certain cases a

license check is executed.

Every day a special tool checks all features with FlexFrame Orchestrator licenses:

how many applications of this feature are currently available in the FlexFrame and

how many applications of this feature are currently allowed by the corresponding

license keys.

3.7.2 Types of Licenses in FlexFrame Orchestrator

In FlexFrame Orchestrator the following features are requiring an own license:

running SIDs

KVM nodes

ESX server

HANA OLAP DBs

RBAC (Role Based Access Control)

FlexFrame Basic Administration Product Licenses


Accounting

SAN

Every single license allows the use of a specified amount of applications of the

corresponding feature. First the customer orders a license with type BASE for every

required feature. If the amount of applications allowed by this BASE license is no more

sufficient the customer can order a number of additional licenses of this feature with type

EXTENT (i.e. one license of type BASE and 0 – n licenses of type EXTENT are required).

Special case: the features SID, RBAC and Accounting are additionally provided with the

type ENTRY. In place of a license with type BASE the customer can order a license with

type ENTRY. In this case he can order no additional licenses of type EXTENT for this

feature.

3.7.3 How to Install a License Key

Every required license key has to be installed individually.

To install a license key use the program ff_license.pl as root like:

/opt/FlexFrame/bin/ff_license.pl -–op add -file <license key file path>

An already installed Iicense key is not possible to install again.

The installation of the CNs is executed without FFO license checks. If both CN

installations are finished the customer has to install all the FFO license keys as soon as

possible.

3.7.4 How to List Installed License Keys

To list all installed license keys use the program ff_license.pl as root like:

/opt/FlexFrame/bin/ff_license.pl -–op list-all

Output: a list of all license keys installed in LDAP.

The given example shows the output of FlexFrame Orchestrator.

Example:jer1cn1:/opt/FlexFrame/bin # ff_license.pl -–op list-all

**** all FFO licenses in FlexFrame ****

licensed feature: SID

licensed type: BASE

licensed number of running SIDs: 12

license created at: 2013-07-01 12:00:00

Product Licenses FlexFrame Basic Administration


license valid till: unlimited

identifier (customer): Lufthansa

license created with FF version: 1.0

licensed feature: SID

licensed type: EXTENT

licensed number of running SIDs: 6





licensed feature: KVM

licensed type: BASE

licensed number of servers: 12





With /opt/FlexFrame/bin/ff_license.pl --op list -–feat <feature> all

license keys of the given feature are listed.

Features allowed with FFO are: SID, KVM, ESX, HANA, RBAC, ACCOUNTING, SAN.

3.7.5 How to Control the Use of Licenses

The new script ff_lic_usage.pl is called daily by a cronjob. With this script the

proper use of the FFO licenses is checked.

The features SID, KVM, ESX, RBAC, ACCOUNTING, SAN are checked by this tool: how

many applications of this feature are currently available in the FlexFrame and how many

applications of this feature are currently allowed by the corresponding license keys.

The summation of running SIDs is done without SMD SIDs.

SIDs without a DB are weighted only with 10 percent in the summation of running SIDs.

I.e with one SID license the customer is allowed to run 10 SIDs without a DB.

The result of this checks is written in the file /var/log/FlexFrame/licstat/FFO_license_status.<date>.

Files FFO_license_status.<date> older than 180 days are removed.

Pools and Groups Adding a Pool


4 Pools and Groups

4.1 Adding a Pool

A new pool can be added using ff_pool_adm.pl. Some parameters have to be defined

on the command line. They are used to configure switch VLANs and ports, to create the

NAS system volume folder structures, to create LDAP pool subtree and to configure

Control Nodes.

An ADMINPOOL (see chapter 6) is created, if <pool name> is adminpool or

force_admin is given.

If the pool shall be equipped with Application Nodes of type “BX600” (blade

server in a BX600 blade server cabinet with switch blades of type 1GbE-10/6-Q

(SB9a)), then the following restriction concerning the VLAN-ID has to be consi-

dered:

Switch blades of type 1GbE-10/6-Q only allow the VLAN ID range from 1 to

3965!

Creating a new pool adds VLAN associations for new VLANs to some ports, e.g.

to uplink ports. If special ports for Client LAN connection should be used the

behavior is determined according to global parameters originally given by the

Management Tool (MT, see the manual Management Tool, chapter “Network”

Object). These parameters can be displayed and changed by

ff_swgroup_adm.pl --op parameters (SWGADM) (see this manual

chapter 9.7).

The parameters are One ClientLAN per SWP (true/false) at MT or

clanPortPerVlan (yes/no) at SWGADM

true/yes Use an independent switch port pair for each pool Client LAN.

Every port pair will be configured with the pool Client LAN (untagged).

false/no All pool ClientLANs use the same switch port pair. Both ports are

configured as Cisco trunk ports, with tagged VLANs.

Use 1Gbit SWPs for ClientLANs at MT or

useTxToClan at SWGADM

Only one ADMINPOOL is allowed in a FlexFrame system.

Adding a pool changes the exports file on all given NAS systems. Temporary

exports on these NAS systems will be gone after running ff_pool_adm.pl.

Be sure not to have temporary exports.

Adding a Pool Pools and Groups


true/yes use copper twisted pair ports as CLAN ports.

false/no use fiber optic (SFP) ports as CLAN ports if available else use

copper ports.

So if the original global setting does not match your current wish for wiring the

new pool, you can change it temporarily with the two corresponding ff_swgroup_adm.pl commands.

Synopsis

ff_pool_adm.pl

--op add --name <pool name>

--storage <vlan id>,<network ip>,<netmask>

--server <vlan id>,<network ip>,<netmask>

--client <vlan id>,<network ip>,<netmask>

--dns <domain name>[,<dns server ip>[,<dns server ip>]

[--dns_search_list <list of domains>]

[--sapdata <nas name>[,<volume path>]]

[--saplog <nas name>[,<volume path>]]

[--volff <nas name>[,<volume path>]]

[--defrouter <default router ip>]

[--switchgrp <id>[,<id>]]

[--force_admin]

Options

--op add

Adds a pool and displays information about processing steps, errors and warnings.

--name <pool name>

Name of new pool (has to be unique withinFlexFrame). A pool name may consist of

2 up to 13 characters (letters, numbers and dashes). It must start with a letter and

must not end with a dash.. The pool name adminpool always creates an

ADMINPOOL.

--storage <vlan id>,<network ip>,<netmask>

Pool specific storage network segment. The option is followed by a comma

separated list of the VLAN ID, the network IP and the netmask of the network IP

address.

--server <vlan id>,<network ip>,<netmask>

Pools and Groups Adding a Pool


Pool specific server network segment. The option is followed by a comma separated

list of the VLAN ID, the network IP and the netmask of the network IP address.

--client <vlan id>,<network ip>,<netmask>

Pool specific client network segment. The option is followed by a comma separated

list of the VLAN ID, the network IP and the netmask of network IP.

--dns <domain name>,[<dns server ip>]

DNS domain name and servers to be used for this pool. More than one DNS server

IP address may be given. Keep in mind to use the default router option if server IP

addresses are given. A DNS option may look like this:

my.domain.com,192.168.251.17,192.168.251.18

--dns_search_list <list of domains>

List of DNS domain names to be used for this pool. At most six domain names are

allowed. For technical reasons the first DNS domain name of this list always must be the DNS domain name given in option dns.

--sapdata <nas name>[,<volume path>]

Optional NAS name and volume path the pool should use for sapdata. A missing

value is auto-filled with the default specified with installation and output with

operation --help.

--saplog <nas name>[,<volume path>]

Optional NAS name and volume path the pool should use for saplog. A missing

value is auto-filled with the default specified with installation and output with

operation --help.

--volff <nas name>[,<volume path>]

Optional NAS name and volume path the pool should use for volFF. A missing value

is auto-filled with the default specified with installation and output with operation --

help.

--defrouter <default router ip>

The default router is a gateway to route IP data to other, non-pool local networks. All

IP data that cannot be addressed to a local network will be sent to the default router

to be forwarded to the destination network. The option parameter is an IP address of

this default router. Use a default router IP address matching the client pool network,

because otherwise it will not be accessible by Application Nodes.

--switchgrp <id>[,<id>]

The switch group ID(s) the Client LAN to corporate LAN ports should be configured.

If not given, the client VLAN is assigned to the existing trunk ports or a new port pair

at first both switch groups. Not more than two switch group IDs are accepted.

Removing a Pool Pools and Groups


--force_admin

This pool (with any pool name) should be the ADMINPOOL

Example

cn1:~ # ff_pool_adm.pl --op add --name pool4

--storage 30,192.168.30.0,255.255.255.0 --server 31,192.168.31.0,255.255.255.0

--client 32,192.168.32.0,255.255.255.0 --sapdata filer --saplog filer

--volff filer --dns my.domain.com --defrouter 192.168.32.254

update LDAP

...

update switch 1/1 configuration

Notice: Update will take about 1 minute.

vlan: storage-30 has been created

restart cluster service ldap_srv1

Notice: restart will take up to 1 minute.

stop and wait until service is offline

start and wait until service is online





restart cluster service netboot_srv

Notice: restart will take up to 2 minutes.



If not reported any warnings or errors all precautions are done

and the pool was successfully created.

Use ff_poolgroup_adm.pl to define the host groups of this pool

to be able to add application nodes.

4.2 Removing a Pool

A pool can be removed using ff_pool_adm.pl. Some parameters have to be defined

on the command line. Switch VLANs will be removed and the affected ports reconfigured.

The LDAP pool subtree will be removed and Control Node configurations rewritten.

The pool created with the attribute "CN Pool = true" may not be removed by an

administration command, since this pool is used to mount file systems from the NAS

device.

If the pool uses volumes different from common volFF these volumes and their data were

not touched. The customer has to remove the data manually.

Pools and Groups Removing a Pool


A pool may not be removed if any Application Node or SID is defined. The first

pool may not be removed due to system requirements. Removing a pool

changes the exports on all NAS systems used by this pool. Use list or list-

all to get the storage configuration of pool to be removed. Temporary exports

(not written to the exports file /vol0/etc/exports) on these NAS systems will

be gone after running ff_pool_adm.pl. Be sure not to have temporary

exports.

Synopsis

ff_pool_adm.pl --op rem --name <pool name>

Options

--op rem

Removes a pool and displays only errors and warnings.

--name <pool name>

Name of pool to be removed. Use ff_pool_adm.pl --op list-all to get a list

of currently configured pools (see 4.4).

Example

cn1:~ # ff_pool_adm.pl --op rem --name pool4

update LDAP

...











restart cluster service netboot_srv

Notice: restart will take up to 2 minutes.



If not reported any warnings or errors the pool was successfully removed.

Keep in mind, the volumes different from common volFF and their data were not

harmed. It's on you to remove them.

Listing Pool Details Pools and Groups


4.3 Listing Pool Details

To list the configurations details of a pool like used networks, pool groups, SIDs and Application Nodes, the maintenance tool ff_pool_adm.pl can be used. The pool name

has to be defined on the command line.

Synopsis

ff_pool_adm.pl --op list --name <pool name>

[--list <part>[,<part>]]

Options

--op list

Displays pool configuration details

--name <pool name>

Name of pool to be listed. Use ff_pool_adm.pl --op list-all to get a list of

currently configured pools (see 4.4).

--list <part>[,<part>]

To reduce the output to the interesting parts, use this option. The parameters to this

option are the display sections. Add them as a comma separated list. The default

sections are: network,storage,dns,group,sid,an,cn,NAS system. You may also use a

two character abbreviation instead the full section name like ne for network.

Pools and Groups Listing Pool Details


Examples

cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list --name p1

Pool configuration details of pool p1

Networks

Client-LAN

Network: 10.10.10.0 Netmask: 255.255.255.0 VLAN ID: 100

Server-LAN


Storage-LAN


Def.Router: 10.10.10.254

Storage Volumes

sapdata fas01-p1-st:/vol/sapdata/p1

saplog fas01-p1-st:/vol/saplog/p1

volFF fas01-p1-st:/vol/volFF/pool-p1

volFF shared fas01-p1-st:/vol/volFF

DNS data

Domain Name: my.domain.com

Pool Groups

Linux

SIDs D01 SAP Version: SAP-6.20 DB Version: Oracle-9 P01 SAP Version: SAP-

6.20 DB Version: Oracle-9

Q01 SAP Version: SAP-6.20 DB Version: Oracle-9

Application Nodes

blade01

Type: BX600 Cabinet ID: 1 Slot/Partition ID: 1

OS: SUSE Linux SLES-11.x86_64

Group: Linux

Client-LAN blade01 10.10.10.23

Server-LAN blade01-se 192.168.10.23

Storage-LAN blade01-st 192.168.20.23

blade02



Group: Linux




blade03


Listing Pool Details Pools and Groups



Group: Linux




rx801

Type: RX800


Group: Linux

Client-LAN rx801 10.10.10.2

Server-LAN rx801-se 192.168.10.2

Storage-LAN rx801-st 192.168.20.2

Control Nodes

cn1

Client-LAN cn1-p1 10.10.10.10

Server-LAN cn1-p1-se 192.168.10.10

Storage-LAN cn1-p1-st 192.168.20.10

cn2




NAS Nodes

fas01-p1

Storage-LAN fas01-p1-st 192.168.20.14

A sample with a reduced output:

cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list --name p1 --list ne,gr

Pool configuration details of pool p1

Networks

Client-LAN


Server-LAN


Storage-LAN


Def.Router: 10.10.10.254

Pool Groups

Linux

Pools and Groups Listing All Pools


4.4 Listing All Pools

To display an overview of all pools with their used networks, pool groups, SIDs and Control Node and NAS system interfaces, the maintenance tool ff_pool_adm.pl can

be used. No arguments except the operation mode have to be defined on the command

line.

Synopsis

ff_pool_adm.pl --op list-all [--list <part>[,<part>]]

Options

--op list-all

Displays the configuration details of all configured pools.

--list <part>[,<part>]

To reduce output to interesting parts use this option. The parameters to this option

are the display sections. Add them as a comma separated list. The default sections

are: network,storage,group,sid,cn,NAS system. You may also use a two character

abbreviation instead the full section name like ne for network.

Examples

cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list-all

Pool configurations

p1

Pool Networks

Client-LAN


Server-LAN


Storage-LAN


Pool Storage Volumes

sapdata fas01-p1-st:/vol/sapdata/p1

saplog fas01-p1-st:/vol/saplog/p1

volFF fas01-p1-st:/vol/volFF/pool-p1

volFF shared fas01-p1-st:/vol/volFF

Pool Groups

Linux

Pool SIDs

D01 SAP Version: SAP-6.20 DB Version: Oracle-9 P01 SAP Version: SAP-


Listing All Pools Pools and Groups



Pool Control Node Interfaces

cn1




cn2




Pool NAS Node Interfaces

fas01-p1

Storage-LAN fas01-p1-st 192.168.20.14

A sample with a reduced output on a single pool configuration:

cn1:/opt/FlexFrame/bin # ff_pool_adm.pl --op list-all --list sid,group

Pool configurations

p1

Pool Groups

Linux

Pool SIDs

D01 SAP Version: SAP-6.20 DB Version: Oracle-9 P01 SAP Version: SAP-



Pools and Groups Changing Pool DNS Domain


4.5 Changing Pool DNS Domain

Changing a pools DNS configuration can be done using the maintenance tool ff_pool_adm.pl. The DNS configuration is written to

/FlexFrame/volFF/pool-<pool name>/pooldata/config/etc/resolv.conf .

The DNS configuration is newly configured, therefore all domain names and ser-

vers to be used for this pool have to be given in the options. Any modification in resolv.conf, done by the customer, is lost.

If the DNS domain name of a pool is modified, afterwards some modifications

concerning the SAP services have to be done manually. These modifications

need a downtime of the SAP services.

Step 1: find all concerned host names

e.g. switch to an application node of the pool and execute ‘getent hosts | grep

<old dns domain>’.The first columne shows the concerned host names which

have to be modified in LDAP.

Step 2:

Change in LDAP the host entries (stored in Pools - <poolname> - Hosts).

Exchange the old dns domain name to the new one.

Step 3:

provide in SAP profile files the parameter SAPLOCALHOSTFULL with the new

domain name. Maybe other SAP files contain the domain name too and

therefore have to be changed.The changes in SAP files have to be done by a

SAP expert.

If in the command line of ff_pool_adm.pl a search-list with a list of dns domains

is given every dns domain of the list has to be inserted in every concerned host

entry in LDAP.

If until now more than one dns domain was given (by a previous search-list)

remove in LDAP in the concerned host entries all the dns domain names not

included in the new command line of ff_pool_adm.pl.

Synopsis

ff_pool_adm.pl --op dns --name <pool name>

--dns <dns domain>[,<dns server ip>

[,<dns server ip>] ]

[--dns_search_list <list of domains>]

Changing Pool Default Router Pools and Groups


Options

--op dns

Adds or changes DNS configuration of a pool.

--name <pool name>

Name of pool the DNS configuration has to be changed.

Use ff_pool_adm.pl –op list-all to get a list of currently configured pools

(see 4.4).

--dns <dns domain>[,<dns server ip>]

DNS domain name and servers to be used for this pool.

A dns_server_ip is only allowed if a default router of this pool is already specified.

--dns_search_list <list of domains>

List of DNS domain names to be used for this pool. At most six domain names are

allowed. For technical reasons the first DNS domain name of this list always must be the DNS domain name given in option dns.

Example

cn1:~ # ff_pool_adm.pl --op dns --name pool3 --dns fsc.net

NOTICE: IF YOU CHANGE THE DNS DOMAIN NAME YOU MAYBE HAVE TO DO SOME CHANGES

MANUALLY !

If at this pool some SAP installations are configured you have to change the domain

names of the Host Entries in LDAP. (Attention: this change needs a downtime of the

corresponding services.) If in the SAP profiles the Parameter SAPLOCALHOSTFULL is

using another domain name, the domain name has to be changed manually. The changes

in SAP files have to be done by a SAP expert.

DNS domain successfully changed at LDAP and AN images.

4.6 Changing Pool Default Router

Add or change a pool's default router with opcode defrouter, remove a pool's default

router with opcode rem-defrouter. The default router may be necessary to reach

external DNS servers. Within all cases a default router has to have an IP address

matching the client IP network (. The default router is used to transfer network traffic

between the local IP networks and external IP networks.

Pools and Groups Adding a Group to a Pool


Synopsis

ff_pool_adm.pl --op defrouter --name <pool name>


ff_pool_adm.pl --op rem-defrouter --name <pool name>

Options

--op defrouter

Adds or changes default router configuration of a pool.

--op rem-defrouter

removes default router configuration of a pool.

--name <pool name>

Name of pool the default router configuration has to be changed and removed

respectively.

Use ff_pool_adm.pl –op list-all to get a list of currently configured pools

(see 4.4).


The default router to be used for this pool to communicate with other, non local

networks. The IP address has to match to client pool network.

Example

cn1:~ # ff_pool_adm.pl --op defrouter --name pool3 --defrouter 192.168.100.200

Defaultrouter successfully changed at LDAP and AN images.

4.7 Adding a Group to a Pool

A pool may have more than one group. To add a group to a pool, the maintenance tool /opt/FlexFrame/bin/ff_poolgroup_adm.pl can be used. To associate a group to

a pool, some parameters have to be defined on the command line. Groups are used with

the FlexFrame Agents.

In the ADMINPOOL only a group named SPARE is allowed. This group acts exclusively

as a container for all types of spare nodes. The group is added to a pool in the LDAP

database.

Synopsis

ff_poolgroup_adm.pl --op add --pool <pool name>

--group <group name>

Removing Pool Group Pools and Groups


Options

--op add

Adds a group to a pool.

--pool <pool name>

Name of pool the group should be added to.


The name of the group to be added.

If a new poolgroup is added and you want to run already installed HANA

systems in that new group you have to run ff_setup_sid_folder.sh once more

to create the new file and directory structures for the new poolgroup

according to the HANA file structure requirements. Take care and make sure

that concerned HANA systems are down.

4.8 Removing Pool Group

To remove a group of a pool, use the maintenance tool /opt/FlexFrame/bin/ff_

poolgroup_adm.pl. You have to define the pool and the group name on the command

line.

The group is removed from a pool in the LDAP database.

Synopsis

ff_poolgroup_adm.pl --op rem --pool <pool name>


Options

--op rem

Removes a group from a pool.

--pool <pool name>

Name of the pool the group should be removed from.


The name of the group to be removed.

4.9 Hosts Database

It may become necessary to have additional entries in the hosts database. Those entries

may be required by 3rd-party products installed on customized Application Node images.

Pools and Groups Hosts Database


The hosts database is stored in LDAP. To maintain the additional host entries use the command ff_hosts.sh.

You cannot remove names or addresses which are essential to the FlexFrame

landscape. Each pool has its own hosts database.

Therefore you have to maintain each pool individually.

4.9.1 Script: ff_hosts.sh

This tool allows the administrator to list, add and delete host names or aliases to the

LDAP database. Note: When adding host names with an IP address that matches one of

the pool's network segments or the Control LAN segment, the list of IP addresses for that

segment gets extended by the IP address of this host name to prevent automatic

allocation of the same IP address by other FlexFrame tools.

Only one option of -l, -a or -r can be used at one time.

Synopsis

ff_hosts.sh [-d] -p <pool name> [{-l|-a <ip> -n <host name>|-r

<host name>}]

Options

-d.

This option will log debug information

-l

List all the hosts entries for the pool as provided by option -p.

-p <pool name>

Pool where the host name should be added to.

-a <ip>

Add the IP address <ip>. Has to be used together with option -n. If an entry with

the IP address <ip> already exists, the name provided will be added as an alias.

-n <host name>

Host name <host name> will be added to the list.

-r <host name>

Hosts Database Pools and Groups


Deletes the host name or alias of name. The host name cannot be deleted if it has

additional aliases. Remove the aliases first.

Examples

The following example will list all additional hosts entries for pool poolname created

using this tool:

cn1:~ # ff_hosts.sh -l -p poolname

The following example will add a host newhost with the IP address 1.2.3.4 to the pool

poolname:

cn1:~ # ff_hosts.sh -p poolname -a 1.2.3.4 -n newhost

The following example will remove the hosts entry for newhost:.

cn1:~

User and Groups AdministrationCreate, Modify, Delete, or List User(s) for Application Nodes


5 User and Groups Administration

5.1 Create, Modify, Delete, or List User(s) for Application Nodes

Synopsis

/opt/FlexFrame/bin/ff_user_adm.pl

--op add –-user <user name> --pool <pool name>

[–-group {<group name> | <group id>},...]

[–-uid <uid number>]

[--home <home directory>] [–-pass <password>]

[--shell <login shell>] [-–gecos <comment>]

[--shadowmin <number of days>] [--shadowmax <number of days>]

[--shadowwarn <number of days>]

[--shadowinactive <number of days>]

[--shadowexpire <number of days]


--op mod –-user <user name> --pool <pool name>

[–-pass <password>] --shell <login shell> [-–gecos <comment>]

[--shadowmin <number of days>] [--shadowmax <number of days>]

[--shadowwarn <number of days>]

[--shadowinactive <number of days>]

[--shadowexpire <number of days>]

[--home <home directory>]

[--group {<primary group name> | <primary group id>}]

[--uid <uid number>]


--op rem –-user <user name> --pool <pool name>


--op list –-user <user name> --pool <pool name>


--op list-all --pool <pool name>

Options

--op add –-user <user name> --pool <pool name>

Creates a new user in a given pool.

Create, Modify, Delete, or List User(s) for Application NodesUser and Groups Administration


--op mod –-user <user name> --pool <pool name>

Modifies a user in a given pool.

--op rem –-user <user name> --pool <pool name>

Deletes a user in a given pool.

--op list –-user <user name> --pool <pool name>

Displays information of a user in a given pool.

--op list-all --pool <pool name>

Displays information of all users in a given pool.

--group <group name> | <group id>,...

A comma separated list of group names/ids the user belongs to. The first group is

taken as primary group. Default: gid number 1 (=other).

--group <primary group name> | <primary group id>

The affiliation to the primary group can be changed with this option.

--uid <uid number>

A specific uid number, if desired.

Default: One higher than the highest used uid number.

SAP recommends usage of uid numbers >= 3600.

In connection with the mod option the existence of the uid is not checked.

--home <home_directory>

Home directory of the user. If this option is not used the home directory is set to

/FlexFrame/pooldata/home/<user> and created if it does not exists.

The following rules are valid if the option is used:

<home_directory>::= <name-1>/<name-2> | <name-3>

Using ‘<name-1>/<name-2>’ or ‘<name3>’ creates a directory in

/FlexFrame/pooldata/<appropriate_dir> if the appropriate directory

does not exist and sets the home directory to

‘/FlexFrame/pooldata/<appropriate_dir> ’.

<home_directory>::= /home_sap/<name-1> | /<name-2>/<name-3>

Using ‘/home_sap/<name-1>’ sets the home directory to

‘/home_sap/<name-1>’ and creates the directory if it does not exist. Using

‘/<name-2>/<name-3>’ sets the home directory to’/<name-2>/<name-3>’

but does not create the appropriate directory

--pass <password>

User and Groups AdministrationCreating, Modifying, Deleting or Listing Group(s) for Application Nodes


Login password for the user. Default: password.

--shell <login shell>

Login shell of the user. Default: /bin/csh. In connection with the mod option the

existence of the login shell is not checked.

--gecos <comment>

Some comment used for the user's gecos attribute. Default: Normal user. This

replaces the previous used option comment.

--shadowmin <number of days>

Min. number of days until the password can be changed.

--shadowmax <number of days>

Max. number of days until the password have to be changed.

--shadowwarn <number of days>

Number of days you get a warning message before the password expires.

--shadowinactive <number of days>

Number of days of inactivity that is allowed for this account.

--shadowexpire <number of days>

Number of days after which the account expires.

5.2 Creating, Modifying, Deleting or Listing Group(s) for Application Nodes

This command enables you to extend the operating system group entries stored in the

LDAP database. The modification or deletion of group entries is restricted to those entries

which are created with ff_group_adm.pl.

Synopsis

ff_group_adm.pl --op add –-name <osgroup name> --pool <pool name>

[–-gid <group id>] [--member <member, ...>]

[--gmember <member, ...>] [-–text <description>]

[--force] [--fname <filename>]

ff_group_adm.pl --op mod –-name <osgroup name> --pool <pool name>

[–-gid <group id>] [--member <member, ...>]

[--gmember <member, ...>] [--force]

[--fname <filename>]

Creating, Modifying, Deleting or Listing Group(s) for Application NodesUser and Groups Administration


ff_group_adm.pl --op rem –-name <osgroup name> --pool <pool name>

[--member <member, ...>] [--gmember <member, ...>]


ff_group_adm.pl --op list –-name <osgroup name> --pool <pool name>

ff_group_adm.pl --op list-all [–-name <osgroup name>]

[--pool <pool name>]

ff_group_adm.pl --help

Options

--op add

Creates a new group in a given pool.

--op mod

Modifies some of the attributes of the group entry in a given pool.

--op rem

.Removes some of the attributes of the group entry or the entry itself.

--op list

Displays information of group entries in a given pool.

--op list-all

Displays information of all group entries in a given pool or in all pools.

--name <osgroup name, ...>

A comma separated list of group names.

--pool <pool name>

Name of the pool the group should belong to.

--gid <gid number>

A certain GID number, if desired. If a list of groups is specified, it is the start GUID

number for the first group. For the next group names the value is always

incremented by one. If the increment means a conflict with an already existing GUID

the script looks for the next free GUID.

You can modify the GID with operation mod, but you are responsible for the effects.

It is also not possible to use a list of groups if you want to change a certain GID.

--member <member, ...>

A comma separated list of user names which should belong to this group. No check

is done if the user really exists.

User and Groups AdministrationCreating, Modifying, Deleting or Listing Service(s) for Application Nodes


--gmember <member, ...>

A comma separated list of user names which is inserted into the group members list

of a group. This option requires a modified flexframe.schema file. Group

members usually used with a DB2 installation.

--text <description>

A user specific description of the group entry.

--force

The add operation continues with the next group if any add operation before

terminates unexpectedly.

--fname <filename>

Name of the file to store the used LDIF statements for this request.

--help

Shows the usage of the command.

5.3 Creating, Modifying, Deleting or Listing Service(s) for Application Nodes

This command enables you to extend the service entries stored in LDAP database. The

modification or deletion of service entries is restricted to those entries which are created

with ff_services_adm.pl.

Synopsis

ff_services_adm.pl --op add –-name <service name, ...>

--pool <pool name> [–-port <port number>]

[--prot {udp|tcp}] [-–text <description>]

[--force] [--fname <filename>]

ff_services_adm.pl --op mod –-name <service name>

--pool <pool name> [–-port <port number>]

[--prot {udp|tcp}] [--force]


ff_services_adm.pl --op rem –-name <service name, ...>

--pool <pool name>

ff_services_adm.pl --op list –-name <service name, ...>

--pool <pool name>

Creating, Modifying, Deleting or Listing Service(s) for Application NodesUser and Groups Administration


ff_services_adm.pl --op list-all [–-name <service name>]


ff_services_adm.pl --help

Options

--op add

Creates new services in a given pool.

--op mod

Modifies a service in a given pool.

--op rem

Deletes services in a given pool.

--op list

Displays information of service entries in a given pool.

--op list-all

Displays information of all service entries in a given pool or in all pools.

--name <service name>,...

A comma separated list of service names.

In case of operation mod you cannot use a list of services.

--pool <pool name>

Name of the pool the services should belong to.

--port <port number>

A certain port number, if desired. If a list of services is specified it is the start port

number for the first service. For the next service names the value is always

incremented by one. Portnumbers has to be unique.

--prot {udp|tcp}

Specifies the used protocol (default tcp).

--text <description>

A user specific description of the service entry.

--force

The add operation continues with the next service if any add operation before

terminates unexpectedly.

--fname <filename>

User and Groups AdministrationCreating, Modifying, Deleting or Listing Service(s) for Application Nodes


Name of the file to store the used LDIF statements for this request.

--help

Shows the usage of the command.


6 Pool-independent Spare Nodes

Before FlexFrame version 4.2 a group specific failover concept for Application Nodes was

used, i.e. if a node in a certain pool group failed, the FlexFrame Agents searched another

Application Node in this group with enough free capacity to take over the service of the

failed node. As of FlexFrame version 4.2 the new concept of pool-independent spare

nodes was introduced:

If no adequate spare node is found in the group of the failed node, the FlexFrame Agents

can search a spare node in the special pool ADMINPOOL, which mostly is called

adminpool (except in cases, where this name has already been reserved). The

ADMINPOOL is a reservoir of spare nodes for all pools of a FlexFrame landscape, and it

must not serve as a normal production pool.

A spare node in the ADMINPOOL is ready for takeover by the FlexFrame Agents

only if it has been booted at least once!

To move a spare node from the ADMINPOOL to its target pool and target group, the option --op move of ff_an_adm.pl has been implemented.

Pay attention to the restrictions of the move operation of ff_an_adm.pl as

described in section 7.5 “Moving Application Nodes

The following is an overview of the details of this concept.

6.1 Creation of Spare Nodes in the ADMINPOOL

To create spare nodes, you have to perform the following steps using the Management

Tool or the corresponding administration scripts (see parentheses):

1. Create the ADMINPOOL (ff_pool_adm.pl –-op add; see section 4.1)

2. Create the pool group SPARE within this pool (ff_poolgroup_adm.pl –-op add;

see section 4.7)

3. Add the necessary spare nodes to this group (ff_an_adm.pl –-op add; see

section 7.2)

4. Create the os images with ff_new_an.sh and boot them.

6.2 Moving of a Spare Node

To move a spare node from the ADMINPOOL to the desired target pool and target group,

use ff_an_adm.pl –op move (see section 7.5).

Pool-independent Spare NodesHandling Pool-independent Spare Nodes with FlexFrame Agents


6.3 Handling Pool-independent Spare Nodes with FlexFrame Agents

If no adequate spare node is found in the group of the failed node, the FlexFrame Agents

can search a spare node in the ADMINPOOL. For more information see the “FlexFrame

Agents Installation and Administration“ manual.


7 Application Nodes Administration

Manual Application Node administration would be very complex and error-prone. The

script /opt/FlexFrame/bin/ff_an_adm.pl does the major changes and supports

adding, changing, removing and listing of Application Nodes. Below, each action is des-

cribed in detail.

7.1 Listing Application Nodes

7.1.1 Displaying Information on a Specific Application Node

Synopsis

ff_an_adm.pl --op list --name <node name> [--cmdline]

Options

--op list

Lists the configuration details of an Application Node.

--name <node name>

The name of the Application Node to be listed.

--cmdline

The command line that used to create the Application Node is displayed at the end

of the node listing. This may be used as a preparation step to rename the AN.

The output is structured in sections: hardware, software, network, pool and group and

LAN interface connections or virtual machine.

Hardware

This section contains information about system type, shutdown facility with IP

address and host name, mac addresses and on blade ser-vers the chassis and slot

ID.

Software

This section contains information on OS type, vendor and version and the root

image path.

Network

This section lists the VLAN ID, IP address and host name of all configured networks,

sorted by LAN segments.

Pool and Group

Application Nodes Administration Listing Application Nodes


This section lists the names of the assigned pool and group.

LAN Interface Connections

In case link aggregates are configured, this section identifies the aggregate and its

ports. Each used switch port is shown with the switch group ID, switch ID and port ID

(cabinet ID, switch blade ID and port on blade servers) for the common LANs

(Storage, Server and Client LAN) and the Control LAN (if used).

Listing Application Nodes Application Nodes Administration


Examples

The command displays detailed information on the selected Application Node. The output

differs between blade servers and all others.

Example of a PRIMERGY blade server output:

cn1:~ # ff_an_adm.pl --op list --name bx924

Configuration details of node bx924

Hardware

System: BX924S2

10GBit: yes

Shut.Facil.: Mgmt Blade bx91-co (172.20.13.40)

iRMC Facil.: iRMC bx924-co (172.20.13.41)

MAC Addr.: 00:23:8b:97:ca:02 00:23:8b:97:ca:03

Cabinet/Slot: 1 / 1

Software

OS: SuSE / Linux / SLES-11.x86_64 (Vendor / Type / Version)

OS Path: Linux/FSC_1.1A00-000.SLES-11.x86_64

Network

VlanID Host IP Hostname

Storage LAN: 2021 172.20.21.51 bx924-st

Server LAN: 2022 172.20.22.51 bx924-se

Client LAN: 2020 172.20.20.51 bx924-1

Pool and Group

Pool: pool2

Group: group1


Cabinet SwBlade Port

data NIC-1: 3 1 1

data NIC-2: 3 2 1

Application Nodes Administration Listing Application Nodes


Example of a PRIMERGY rack server output:

cn1:~ # ff_an_adm.pl --op list --name rx36-1

Configuration details of node rx36-1

Hardware

System: RX300S6

10GBit: no

Shut.Facil.: IPMI rx36-1-co (172.20.13.23)

MAC Addr.: 00:c0:9f:dc:3a:da 00:c0:9f:dc:3a:db

Software



Network


Storage LAN: 2011 172.20.11.1 rx36-1-st

Server LAN: 2012 172.20.12.1 rx36-1-se

Client LAN: 2010 172.20.10.1 rx36-1

Pool and Group

Pool: pool1

Group: group1


SwGroup Switch Port

data NIC-1: 1 2 5

data NIC-2: 1 1 5

IPMI NIC-1: 1 2 24

Listing Application Nodes Application Nodes Administration


7.1.2 Displaying Information of all Application Nodes

Synopsis

ff_an_adm.pl --op list-all [--pool <pool name>]

Options

--op list-all

Lists all configured Application Nodes.

--pool <pool name>

The name of the pool of which the Application Nodes have to be listed.

Example

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op list-all

Nodes sorted by pool, group and name

Pool pool1

Pool Group group1

bx924

System: BX924S2

MAC Addr.: 00:23:8b:97:ca:02 00:23:8b:97:ca:03

Cabinet/Slot: 1 / 1




Storage LAN: 2021 172.20.21.51 bx924-st

Server LAN: 2022 172.20.22.51 bx924-se

Client LAN: 2020 172.20.20.51 bx924-1

Pool pool2

Pool Group group2

bx36-1

System: RX300S65

MAC Addr.: 00:c0:9f:dc:3a:da 00:c0:9f:dc:3a:db




Storage LAN: 2011 172.20.11.1 rx365-1-st

Server LAN: 2012 172.20.12.1 rx365-1-se

Client LAN: 2010 172.20.10.1 rx365-1

Application Nodes Administration Adding Application Nodes


The output of list-all is less detailed than the list output. It is used to get an

overview. It shows the Application Nodes sorted by pool and group in alphabetical order.

For each node the system type, the cabinet and slot ID (if node is a blade server), the OS

type, vendor, version, the root image path, the main IP addresses, host names and the

MAC addresses are listed.

7.1.3 Displaying all available OS images

Synopsis

ff_an_adm.pl --op list-images

Lists all available OS images.

Example

cn1:~ # ff_an_adm.pl --op list-images

Available OS Images are:

Linux/FSC_1.1A00-008.SLES-11.x86_64

Linux/MNT_1.1A00-000.SLES-11.x86_64 (maintenance only (pa-rx300-1))

7.2 Adding Application Nodes

This section describes how to provide the required information for adding a new AN to an

existing FlexFrame environment. See also section 11.6.1.

You have to define some parameters at the command line. They are used to configure

switch ports, to create the boot information and the OS image.

Synopsis

ff_an_adm.pl --op add --type <system type> --name <node name>

--pool <pool name> --group <group name>

--swgroup <switch group id>

Adding an Application Node changes the exports file on the common volFF NAS

system. Temporary exports on this NAS system will be gone after running

ff_new_an.sh. or ff_an_adm.pl with option --new-image. Be sure not to

have temporary exports.

Adding Application Nodes Application Nodes Administration


--mgmtswgroup <switch group id>

--mac <mac addresses>

--ospath <path to os image>

[--new-image]

[--host <ip host number> [,<cntl-LAN ip host number>,

<2nd cntl-LAN ip hostnumber>]]

[--slot <cabinet/slot>]

[--10gbit]

[--port switch:port,switch:port,switch:port,

[switch:port]]

[--hn <hypervisor node name>]

[--vcpus <number of virtual cpus>]

[--vmem <virtual machine memory size]

[--force]

Options

--op add

Adds an Application Node and displays some information about processing steps.

--type <system type>

Specifies the product name and type. Call ff_an_adm.pl without any parameter to

get a list of supported system types. A system type of ESXVM or KVMVM denotes a

virtual machine on an ESXi host or a KVM host, respectively, and a virtual machine

with the same name is implicitly created on the denoted hypervisor node

--name <node name>

The name of the Application Node. This name has to be unique in the FlexFrame

system. All interface names are based on this node name. The name may consist of

2 up to 13 characters (letters, numbers and dashes). It must start with a letter and

must not end with a dash.

--pool <pool name>

The name of the pool this node should belong to. See usage (call ff_an_adm.pl

without any parameter) to get a list of currently configured pools.


The name of the pool group this node is a member of. A group must consist of

Application Nodes of the same OS image version and should be of the same

capacity (CPU, Memory etc.). Use command ff_pool_adm.pl with op mode list

or list-all to get the pool groups.

Application Nodes Administration Adding Application Nodes



Defines the switch group the Application Nodes data NIC 1 and 2 are connected to.

This information is necessary to assign and configure switch ports. Call

ff_an_adm.pl without any parameter to get a list of currently configured switch

group IDs.


Add here both MAC addresses of the data NICs used for booting. Use the colon

separated hex notation for each MAC address. Concatenate them with a comma.

The MAC address syntax is a six colon separated hex values, eg.

00:e0:00:c5:19:41. For an Application Node with type ESXVM or KVMVM, an

appropriate MAC address is generated by the script.


Defines the OS image to be used. Add the relative path to

/FlexFrame/volFF/os/ as seen from the Control Node.

Use ff_an_adm.pl --op list-images to get a list of available OS paths.

--host <ip host number>[,<cntl-LAN ip host number>,

2nd cntl-LAN ip host number>]

Host part to be used to build IP addresses for the three networks. If necessary host

part(s) to be used to build IP addresses for the management network can be added,

separated by commas. If this option is omitted, the script uses free host numbers to

calculate the IP addresses.

--slot <cabinet_id/slot>

With PRIMERGY server blades use this option to define the cabinet and slot ID of

the server blade. New cabinets have to be defined with the

/opt/FlexFrame/bin/ff_bx_cabinet_adm.pl command.

For models that occupy more than one slot in width (e.g.BX630 S2 quad),

the part of the server blade that occupies the slot with the highest slot

number is called the master. The master is usually the rightmost slot and its

slot ID has to be chosen as the master slot ID.

--10gbit

Specifies that the node is used with 10 Gigabit data NICs. The specification can be

omitted if the nodes system type only allows usage with 10 Gigabit data NICs.


Defines the switch group the Application Nodes management interface (IPMI) is

connected to. If omitted the effective switch group is computed as follows: First the

Adding Application Nodes Application Nodes Administration


switch group given with --swgroup is considered, second the switch group used for

the swgroup out of band management is considered. Call ff_an_adm.pl --help to get

a list of currently configured switch group IDs.

--new-image

Creates a new os image (need not call ff_new_an.sh !).

--port switch:port,switch:port,switch:port,[switch:port]

Defines the switch ports to be used. First two tuples are for data NIC 1 and 2 and

allocated in the switch group defined with --swgroup. The following tuples are for

mgmt NICs and allocated in the effective switch group for management interfaces. If

--10gbit is specified, the first two ports must be 10 Gigabit capable ports.

--hn <hypervisor node name>

When adding a node with type ESXVM or KVMVM, this parameter is mandatory and

specifies the name of the hypervisor node where the virtual machine must be

created. A hypervisor node with this name must already exist in FlexFrame and must

be configured for FlexFrame usage. For details refer to chapter 7.7 "Administrating

Hypervisor Nodes and Virtual Machines".

--vcpus <number of virtual cpus>

Specifies the number of virtual CPUs when creating a virtual machine. The default is

2 CPUs.

--vmem <virtual machine memory size in MB>

Defines the memory size in MB when creating a virtual machine. The default is 8

GB. The available resources of the hypervisor node must be taken into account

when choosing a value for this parameter. For details refer to section 7.7.9 “Virtual

Machine Properties and Hypervisor Resources“.

--force

Specifies that the memory usage of the hypervisor node may be overcommitted

when creating a virtual machine. Default is to deny creation of a virtual machine on a

host if the total vmem of the virtual machines on this host, including the new one,

exceeds the memory size of the host

Examples

Output for a blade server:

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op add --type BX924S1

--name bx1-6 --pool pool1 --group bx900

--ospath Linux/FSC_5.1A00-000.SLES-11.x86_64 --host 1 --slot 1/6

--mac 00:C0:9F:99:E6:CC,00:C0:9F:99:E6:CD

update swblade 1/1 configuration

Application Nodes Administration Removing Application Nodes





If not reported any error all precautions are done to create

application nodes os image. To do this call:

ff_new_an.sh -n bx1-6

Creating and customizing an image may take some minutes.

Don't get anxious.

Output for a non-blade server:

cn1:~ # ff_an_adm.pl --op add --name rx300-1 --type RX300S6 --pool pool2 --group

group1 --swgroup 1 --ospath Linux/FSC_5.1A00-000.SLES-11.x86_64 --mac

00:15:17:2d:ab:a8,00:15:17:2d:ac:02



Connect your systems LAN interfaces to named switch ports:

SwitchGroup / Switch / Port LAN Interface

1 / 1 / 6 data NIC-1

1 / 2 / 6 data NIC-2

1 / 1 / 24 IPMI NIC-1

If not reported any error all precautions are done to create

application nodes os image. To do this call:

ff_new_an.sh -n rx300-1

Creating and customizing an image may take some minutes.

Don't get anxious.

The script first checks all arguments and aborts with error messages in case of errors.

Then it fetches free IP addresses and switch ports. The switch ports are reconfigured to

match requirements, the LDAP data is created and a netboot file is written. The netboot

file is used by ff_new_an.sh to create Application Node images and extend the NAS

system's exports list. At the end you get a cabling advice and instructions how to call

ff_new_an.sh script to finish the Application Node creation.

7.3 Removing Application Nodes

You only have to specify the node name to be removed at the command line. All switch

ports will be unconfigured and the boot information and OS image are deleted. For an

Application Node of type ESXVM or KVMVM, the virtual machine is also destroyed.

Changing the attributes of an Application Node Application Nodes Administration


Synopsis

ff_an_adm.pl --op rem --name <node name>

Options

--op rem

Removes an application node.

--name <node name>

The name of the Application Node to be removed. Use operation mode list-all

to get all configured Application Nodes (see 7.1.2).

Example

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op rem --name rx300-1

7.4 Changing the attributes of an Application Node

There are some operations to change attributes of the application node.

Keep in mind: the name or the pool of an application node can only be changed with

operation move.

7.4.1 Change the group of an Application Node

Synopsis

ff_an_adm.pl --op group --name <node name> --group <group name>

Removing an Application Node results in direct deletion of its image, removal of

its LDAP entries as well as disabling the respective switch ports. All existing

images (OS images and maintenance images) created for this node are

destroyed. Please make sure you really want to remove the Application Node

(AN) when calling the script, the script does not ask for further

confirmation.Removing an Application Node changes the exports file on the

common volFF NAS system. Temporary exports on this NAS system will be

gone after running ff_an_adm.pl. Please make sure not to have temporary

exports.

If the Application Node is used with a host based SMD agent the SMD Agent

instance for this host remains defined in pool and the customer has to change

the SMD configuration manually.

Application Nodes Administration Changing the attributes of an Application Node


Options

--op group

change the pool group the node belongs to.

--group

the new group of the node.

Example

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op group --name rx300-1 –group newgroup

7.4.2 Change the os image of an Application Node

Synopsis

ff_an_adm.pl --op os --name <node name> --ospath <path to os

image> [--remove-image]

Options

--op os

change the os image path.

--ospath

The new path of the os image to be used.

--remove-image

On os path changes the former image will be left intact, to be able to change

between them. To remove the old image specify this command line option.

Example

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op os --name rx300-1 –os_path


Moving Application Nodes Application Nodes Administration


7.4.3 Change the mac addresses of an Application Node

Synopsis

ff_an_adm.pl --op mac --name <node name> --mac <mac addresses>

[--type <system type>]

Options

--op mac

change the mac addresses of an application node.

--mac

Define all mac addresses of the network interfaces the node should be able to boot

from. The program will change the mac addresses of netboot configuration.

--type

This parameter may be used to change the system type in case of hardware

changes. This may lead to a change of boot configuration files to support different

hardware class due to system type change.

Example

cn1:/opt/FlexFrame/bin # ff_an_adm.pl --op mac --name rx300-1 --mac

c8:0a:a9:57:ad:5a,c8:0a:a9:57:ad:5b

7.5 Moving Application Nodes

Moving of Application Nodes from one pool to another pool may often be necessary, spe-cially for pool-independent spare nodes (see chapter 6). The opcode move with to-

pool, to-group serves to satisfy this requirement. While performing the move action

between pools, the node gets new Vlans in its new target pool, but changes of network

cabling are not necessary.

Changing of node names may be necessary for various reasons. With opcode move the

nodename can be changed either in the same pool or along with moving into another

pool.

The operation move is not supported for virtual machine Application Nodes.

The operation move is not allowed for a move into a new pool and a new pool group,

if the new pool group is assigned to some SAN volume groups.

Application Nodes Administration Moving Application Nodes


If the Application Node is used with a host based SMD agent the SMD Agent instance for

the host remains unchanged and the customer has to change the SMD configuration

manually.

Only if the move is a renaming of the application node in the same pool with a new name

the SMD Agent instance is changed too. In this case it has to be checked if the renaming

of the node invalidates the reference information. The corrresponding SAP Service

description has to be adpated also.

All existing images (OS images and maintenance images) created for this node are

destroyed, the OS image is newly created.

Synopsis

ff_an_adm.pl --op move --name <node name> [--to-pool <pool name>]

[--to-group <group name>] [--newname <new node name>]

[--to-host <ip host number>[,<cntl-LAN ip host

number>,<2nd cntl-LAN ip host number>]]

[--to-ospath <path to os image>]

[--failed-host <node name>] [--new-image]

[--clean-swap]

Options

--op move

Moves the Application Node given by --name.

--name <node name>

Current name of the node to be moved.

--to-pool <pool name>

Name of target pool.

--to-group <group name>

Name of target group.

--newname <new node name>

New name of the node. The name may consist of 2 up to 13 characters (letters,

numbers and dashes). It must start with a letter and must not end with a dash.

--to-host <ip host number>[,<cntl-LAN ip host number>, <2nd cntl-

LAN ip host number>]

Administrating Blade Server Cabinets Application Nodes Administration


Host part of Ip address in the target pool.

--to-ospath <path to os image>

Os image path the moved node should use.

--new-image

Creates a new os image (need not call ff_new_an.sh !).

--clean-swap

Clean local disks of node on first boot of moved node. This parameter has only

meaning together with --new-image.

If you specify --failed-host, all other options except –name, --new-image and –

clean-swap are ignored. In this case all necessary information for the moved node in its

target pool is derived from the failed host.

7.6 Administrating Blade Server Cabinets

Some network settings have to be changed to add or remove a blade server cabinet. The

script /opt/FlexFrame/bin/ff_bx_cabinet_adm.pl will simplify the administration

by doing LDAP changes automatically and preparing configurations to be done manually.

The script supports adding, moving, removing and listing of blade server cabinets as well

as modifying switch blade attributes. Each action is described in detail below.

7.6.1 Listing Blade Server Cabinets

7.6.1.1 Displaying Information on a Specific Blade Server Cabinet

Synopsis

ff_bx_cabinet_adm.pl --op list --name <cabinet name>

Options

--op list

Lists the configuration details of a blade server cabinet.

--name <cabinet name>

The name of the blade server cabinet to be listed.

The output is structured in sections: hardware, software, network, assigned pool and

group, switch ports.

Application Nodes Administration Administrating Blade Server Cabinets


Output example

PRIMERGY Cabinet 1 (cab1)

System Type: BX900S1

Management Blade

Hostname / IP Address: cab1-co 195.40.224.75

Integrated LAN Switch Ports: SwitchGroup SwitchID PortID

1 1 8

1 2 8

Switch Blade

SwitchID Type Switch name Hostname IP Address

1 10GbE-18/8-F cab1-swb1 cab1-swb1-co 195.40.224.76

2 10GbE-18/8-F cab1-swb2 cab1-swb2-co 195.40.224.77

Switch Blade Port Integrated LAN Switch Port

Switch Blade ID PortID <--> SwitchGroup SwitchID PortID

1 19 2 1 1/14

1 20 2 2 1/13

2 19 2 2 1/14

2 20 2 1 1/13

As seen from the sample above, the cabinet ID and name, the cabinet system type, the

management blade and the switch blades are listed.

For the management blade the host name, the IP address and both FlexFrame integrated

LAN switch ports are displayed.

The switch blade information shows the switch and host name, the IP address and the

switch blade port to FlexFrame integrated LAN switch port connections, structured by

switch blade ID.

7.6.1.2 Displaying Information on all Configured Blade Server Cabinets

Synopsis

ff_bx_cabinet_adm.pl --op list-all

Option

--op list-all

Lists all configured blade server cabinets.

Output example

PRIMERGY Cabinets

1 (cab1) BX600S3



Management Blade: cab1-co / 195.40.224.75

Switch Group ID: 1

Server Blades (by slot id)

1 (blade1) BX924S1

Pool / Group: pool1 / PROD

2 (blade2) BX924S1


3 (blade3) BX924S1


4 (blade4) BX924S1

Pool / Group: pool2 / DEV

5 (blade5) BX924S1

Pool / Group: pool2 / DEV

For each cabinet the ID, the cabinet name, the management host name and IP address

and the server blades are displayed.

Each server blade is shown with its slot ID and name, the system type and the pool and

group it belongs to.

7.6.2 Adding Blade Server Cabinets

This section describes how to provide the required information for adding a new blade

server cabinet to an existing FlexFrame environment. You have to define some

parameters at the command line. They are used to configure switch ports and to create

the switch blade configurations.

Synopsis

ff_bx_cabinet_adm.pl --op add --type <system type>



[--swblades <switch blade type>]

[--swblogin <login name>]

[--swbpwd <password>]

[--host <ip host parts>] [--10gbit]

[--mgmtswgroup <switch group id>]

[--uplinkportcnt <nic count>]

[--port <sw:port>[,...]]

Options

--op add

Adds a blade server cabinet.




PRIMERGY blade system type e.g. BX600S3, BX900S1.

Call ff_bx_cabinet_adm.pl without any parameter to get a list of supported

system types.


Name of the subsystem (cabinet). It is used to generate a name for the management

blade and has to be unique within FlexFrame. It may consist of 2 up to 13 characters

(letters, numbers, dashes). It must start with a letter and must not end with a dash.


Switch group number the cabinet has to be connected to (physically). Call

ff_bx_cabinet_adm.pl without any parameter to get a list of currently configured

switch group IDs.


Defines the switch group the cabinets management blade interfaces are connected

to. If omitted the effective switch group is computed as follows: First the switch group

given with --swgroup is considered, second the switch group used for the swgroup

out of band management is considered. Call ff_bx_cabinet_adm.pl --help

to get a list of currently configured switch group IDs

--swblades <switch blade type>

The type of switch blades. For valid types see the usage. For the default switch

blades this option may be omitted.

--swblogin <login_name>

Name used to login into the switch blades. If this option is omitted, the login name of

the switch group is used.

--swbpwd <password>

Password used to login into the switch blades. If this option is omitted, the login

password of the switch group is used.

--host <ip host parts>

Host parts to be used to build IP addresses for the control LAN networks. If this

option is omitted the script uses free host numbers to calculate the IP addresses.

Order of comma separated host numbers: first host number for both management

blades and then one for each switch blade.

--10gbit

Use 10 Gigabit switch blade ports as uplink. The specification can be omitted if only

10 Gigabit switch blade ports are available



--uplinkportcnt <nic count>

Set the NIC count of switch blades uplink channel to given value. Defaults to 2.

--port <sw:port>[,...]

Specifies the ports that are to be used on the connected switch group(s).

‘sw’ is the switch id within the switch group. ‘port’ is the port id on that switch.

Multiple ports are specified by a comma separated list. The first and second

specification are for the management blades, followed by specifications for the first

switch blades. Specifications for the second switch blade constitute the rest of the

list. If the list contains an empty specification, the port is calculated as if the --port

option was omitted (e.g. --port ,,1:13,2:13,1:14,2:14 to specify ports only

for the switch blades).

Output example

At the end of the output, the command displays further instructions.

Configure the ManagementBlade with the control lan settings:

control lan IP address: 195.40.224.75

control lan name: cab1-co

to interoperate correctly with the FlexFrame Agents.

Interconnect the ManagementBlades and SwitchBlades with the switches noted below:

SwitchGroup SwitchID Port Mgmt/SwitchBlade

1 1 8 ManagementBlade 2

1 1 11 SwitchBlade 1 Port 12


1 2 8 ManagementBlade 1



Uploads of initial SwitchBlade configurations have to be done manually.

See the document(s)

'Quick Start Hardware FlexFrame for SAP – PRIMERGY SWB 1GbE-10/6-Q'

in the doc/hwinfo section of the Service CD for details.

The files to be uploaded are named:

SwitchBlade Blade Type File Path

1 1GbE-10/6-Q /tftpboot/swblade-2-1.config

2 1GbE-10/6-Q /tftpboot/swblade-2-2.config

Look at "/opt/FlexFrame/network/wiring-BX600-cab1.txt" to get a copy of this

message.

Set up the management blade initially with name and IP address listed by the output as

seen above. Use console redirection of management blade to connect to the console of



the switch blades, and upload configuration as described by the FlexFrame manual

“Installation of a FlexFrame Environment”.

Finally, plug in the network cables according to the wiring plan given by the command

output.

7.6.3 Removing Blade Server Cabinets

You only have to specify the ID or the name of the cabinet that is to be removed. All

FlexFrame integrated LAN switch ports will be unconfigured. Removing a blade server

cabinet requires removing of all of its server blades.

Synopsis

ff_bx_cabinet_adm.pl --op rem --id <cabinet id> | --name <cabinet

name>

Options

--op rem

Removes a blade server cabinet.

--id <cabinet id>

Specifies the cabinet ID of the cabinet to be removed. Use the

list-all option to get the ID (see section 7.6.3).


Specifies the name of the cabinet.

Output examples

If there are any server blades configured for this cabinet, an error message is displayed.

ERROR: there are server blades configured for this cabinet.

To remove the cabinet, remove application nodes (server blades) first.

Use command ff_an_adm.pl to do this.

Use the list operation mode to list the configured server blades. You have to remove

them before you can remove the cabinet they are in.

If no server blades are configured for this cabinet, the command displays a summary at

the end.

If not reported any warnings or errors the cabinet was removed

from LDAP and integrated LAN switches.



The cabinet has been removed successfully from LDAP and the FlexFrame integrated

LAN switch ports used by the cabinet have been reconfigured to default.

7.6.4 Changing Switch Blade Type

Within service cases it may be necessary to change the type of a switch blade due to a

defective part replacement. Only switching blades can be used for a type change.

To change the switch blade type the cabinet, the switch blade id and the switch blade

type have to be specified.

Synopsis

ff_bx_cabinet_adm.pl --op swb-change --id <cabinet id>

--swbid <switch blade id>

--swbtype <switch blade type>

Options

--op swb-change

Selects the operation mode. Change the type of a switch blade.

--id <cabinet id>

Specifies the cabinet ID of the cabinet. Use the list-all option to get the ID.


Specifies the switch blade ID. The ID is the slot number of the selected switch blade.

--swbtype <switch blade type>

Defines the new type of the switch blade. See usage for the currently supported

types.

Output example

Switch type of switch blade 2 was successfully changed from "1GbE-10/6-Q" to "1GbE-

10/2+4-C" at LDAP database.

The switch blade type was changed at LDAP database. To get the initial configuration use operation mode swb-config of this program. It will display instructions how to

upload the configurations too.

7.6.5 Changing Switch Blade Name

On adding a new cabinet, the name of the switch blade is like the cabinet name with a

slot extension. In some cases the names of the switch blades have to be changed to

match naming conventions.



Synopsis

ff_bx_cabinet_adm.pl --op swb-name --id <cabinet id>


--swbname <switch blade name>

Options

--op swb-name

Selects the operation mode. Change the name of a switch blade.

--id <cabinet id>



Specifies the ID of the switch blade. The ID is the slot number of the selected switch

blade.

--swbname <switch blade name>

Defines the new name of the switch blade.

Output example

If not reported any warnings or errors the hostname was successfully changed at

switch blade, hosts files and LDAP.

As noted by the program the name of the switch will be changed at /etc/hosts of both

control nodes, the LDAP database and at least the hostname and, if possible, the SNMP

sysname at the selected switch blade itself.

7.6.6 Changing Switch Blade Password

If on adding a cabinet the switch blade password could not be derived from the switch

group and was not specified in the command, a default password is assigned. This is not

secure.

Synopsis

ff_bx_cabinet_adm.pl --op swb-passwd --id <cabinet id>


--swbpwd <password>

Options

--op swb-passwd



Selects the operation mode. Change the login password of a switch blade.

--id <cabinet id>

Specifies the subsystem (cabinet) ID of the cabinet. Use the list-all option to get the

ID.


Specifies the switch blade ID. The ID is the slot number of the selected switch blade.

--swbpwd <password>

Defines the new login and enable password of the switch blade.

Output example

If not reported any warnings or errors the password was successfully changed at

switch blade and LDAP.

As noted by the program the password will be changed at the LDAP database and the

selected switch blade. At the switch blade the login password and the enable password

are changed and have to be the same.

7.6.7 Getting Switch Blade Configuration

In case of a service issue it may be necessary to get a switch blade configuration, which

has to be uploaded manually. Use this command to get the actual necessary

configuration according to LDAP for a specific switch blade.

Customizations, if necessary, have to be reapplied. Inspection of the project

information see 11.10 and the last saved configurations in /tftpboot/saved_switchconfigs may help.

Synopsis

ff_bx_cabinet_adm.pl --op swb-config --id <cabinet id>


Options

--op swb-config

Selects the operation mode. Create the switch blade configuration.

--id <cabinet id>





Specifies the ID of the switch blade. The ID is the slot number of the selected switch

blade.

Output example

jercn1:~ # ff_bx_cabinet_adm.pl --op swb-config --id 2 --swbid 1

Upload the generated SwitchBlade configurations noted below.

See the Quick Start Hardware FlexFrame document from the doc/hwinfo

section of the Service CD for the particular switch type and use the

following data:

swblade 1 cabinet 2 (hoctest-swb1) type 10GbE-18/8+2-F

IP and Mask: 172.20.13.42 255.255.255.0

Config: /tftpboot/hoctest-swb1.config

The configuration file is stored directly to /tftpboot. Upload of configuration is

described using TFTP which uses /tftpboot as top level directory. A detailed

instruction can be found in the FlexFrame manual „Installation of a FlexFrame

Environment“.

7.6.8 Change Switch Blade Uplink

More than the two default ports can be used for the uplink to switch group. To change the

count of ports used by the uplink link aggregates, including switch port configuration, use

the operation mode swb-uplink of ff_bx_cabinet_adm.pl. On switch group and

switch blades the appropriate link aggregates will be expanded by new ports until count

of given ports is reached.

Synopsis

ff_bx_cabinet_adm.pl --op swb-uplink --id <cabinet id>


Options

--op swb-uplink

Change the count of NICs of uplink link aggregate for each switch blade of cabinet.

--id <cabinet id>

Defines the cabinet to change.


Set the NIC count of switch blades uplink channel to given value. Defaults to 2



Example

cn1:~ # ff_bx_cabinet_adm.pl --op swb-uplink --id 1 –-uplinkportcnt 6

update LDAP

...



..........



..........

Interconnect additional NICs between SwitchBlades and the switches of

SwitchGroup 1 as noted below:

SwitchBladeID/Port SwitchID/Port

1 / 13 <==> 1 / 9

1 / 14 <==> 2 / 9

1 / 15 <==> 1 / 11

1 / 16 <==> 2 / 11

2 / 13 <==> 1 / 14

2 / 14 <==> 2 / 14

2 / 15 <==> 1 / 16

2 / 16 <==> 2 / 15

Unless any errors are reported follow instructions above to extend

switch blade channels to switch group.

Look at "/opt/FlexFrame/network/wiring-channel-extend-BX600-1.txt"

to get a copy of this message.

7.6.9 Move a Blade Cabinet to Another Switch Group

On growing installations sometimes the load of a switch group is split onto two groups.

While doing this sometimes entire blade cabinets have to be moved to another switch

group. To simplify this action use the operation mode move of

ff_bx_cabinet_adm.pl.

Synopsis

ff_bx_cabinet_adm.pl --op move --id <cabinet id>



[--port <sw:port>[,...]]



Options

--op move

Change the switch group of a cabinet, including switch blade but not server blade

connections.

--id <cabinet id>

Defines the cabinet to be moved.


The switch group id the cabinet should be moved to.


Defines the switch group the cabinet’s management blade interfaces should be

moved to. If omitted the effective switch group is computed as follows: First the

switch group given with --swgroup is considered, second the switch group used for

the swgroup out of band management is considered.

--port <sw:port>[,...]

Specifies the ports that are to be used on the connected switch group(s).

'sw' is the switch id within the switch group. 'port' is the port id on that switch.

Multiple ports are specified by a comma separated list. The first and second

specification are for the management blades, followed by specifications for the first

switch blades. Specifications for the second switch blade constitute the rest of the

list. If the list contains an empty specification, the port is calculated as if the -port

option was omitted (e.g. -port ,,1:13,2:13,1:14,2:14 to specify ports only for the

switch blades).

Example

cn1:~ # ff_bx_cabinet_adm.pl --op move --id 1 –-swgroup 2

update LDAP

...



..........



..........

Interconnect NICs between MgmtBlades,SwitchBlades and the switches of

the new SwitchGroup 2 as noted below:

SwGrpID/SwitchID/Port Mgmt/SwitchBlade

2 / 1 / 1 <==> master ManagementBlade

2 / 1 / 2 <==> SwitchBlade 1 Port 11

Administrating Hypervisor Nodes and Virtual MachinesApplication Nodes Administration



2 / 2 / 1 <==> slave ManagementBlade



Unless any errors are reported follow instructions above to move

cabinet to new switch group.

Look at "/opt/FlexFrame/network/wiring-cabinet-move-BX600-1.txt"


7.7 Administrating Hypervisor Nodes and Virtual Machines

Instead of being used directly as Application Nodes, PRIMERGY servers may also be

used in FlexFrame as Hypervisor Nodes. A Hypervisor Node can host a number of virtual

machines that are used as Application Nodes.

VMware ESXi and VMware ESX are "bare-metal" hypervisors that form the foundation of

VMware vSphere. The VMware vSphere product line supports three different types of

hypervisors - ESX classic, ESXi installable and ESXi embedded. In FlexFrame only ESXi

installable and ESXi embedded are supported. However, the terms "ESX" and "ESXi" are

both used in the FlexFrame documentation and code to denote the VMware hypervisor

and always mean "ESXi" unless explicitly stated otherwise.

With FlexFrame Orchestrator, SUSE Linux KVM is also supported as an additional

hypervisor type.

Hypervisor Nodes of type KVM can also host the virtual Control Center, i.e. the Control

Nodes run in virtual machines. A Hypervisor Node hosting a virtual Control Node may

additionally host virtual machines used as Application Nodes. Hypervisor Nodes and

virtual machines used as Control Nodes are predefined with the FlexFrame Management

Tool and put into operation during the FlexFrame installation process.

Hypervisor Nodes and virtual machines used as Application Nodes can be predefined

with the FlexFrame Management Tool and put into operation during the FlexFrame

installation process, or added to a FlexFrame configuration later with FlexFrame

administrative tools as described in section 7.7.1 below. The FlexFrame administrative

tools also offer functions to display or adjust the configuration as shown in detail in the

following sections.

Besides the FlexFrame tools which focus mainly on the FlexFrame specific aspects of

ESXi servers and virtual machines, there are several ways to access virtualization

components, such as the vSphere Client or the vSphere Command-Line Interface (vCLI)

for vSphere based components or the Virtual Machine Manager (virt-manager) and

the virsh command line for KVM. These tools can be used in addition to the FlexFrame

Application Nodes AdministrationAdministrating Hypervisor Nodes and Virtual Machines


tools, but certain actions must be avoided to insure proper operation of the FlexFrame

system. A short overview of this topic is given in section 7.7.10.

7.7.1 Getting Started with Hypervisor Nodes and VMs

This section gives an overview of the steps needed to add a Hypervisor Node (HN) and

virtual machines used as Application Nodes on the HN to a FlexFrame system when

using FlexFrame administrative tools.

1. Add the HN to the FlexFrame configuration.

cn1:~ # ff_hn_adm.pl --op add --name <node name> --type <system type> ...

Depending on system type, more options are needed when entering this

command.

See section 7.7.3 for details.

2. Install/Boot the HN.

The necessary actions depend on the used hypervisor type. For embedded ESXi,

the server is already installed on an internal USB device and the only action is to

boot it from this device. For ESXi installable, install the ESXi server from its CD or

DVD (an ISO-Image of this software is also contained on the FlexFrame Service

DVD) and then boot it from local hard disk. For SUSE Linux KVM, install SUSE Linux

Enterprise Server (SLES) with KVM from the FlexFrame Control Node DVD. Refer to

the "Install and Boot Hypervisor Node" section of the manual "Installation of a

FlexFrame Environment" for details and to the section "Setup the <server type> for a

Hypervisor Node installation" in the appropriate "FlexFrame® HW Characteristics

Quickguide ("Hardwaresteckbrief")".

3. Do a minimal configuration of the Hypervisor Node using its server console. This

includes setting a password and selecting the network adapters for the network

connection of the Hypervisor Node. Refer to the "Hypervisor Node Preconfiguration

on the Server Console" section of the manual "Installation of a FlexFrame

Environment" for details.

4. Complete the Hypervisor Node configuration.

cn1:~ # ff_hn_adm.pl --op complete-config --name <node name>

--user <user name> --pass <password>

See section 7.7.4 below for details.

5. No explicit creation of virtual machines is needed.

Instead of that, simply use the well-known ff_an_adm.pl --op add command

with a system type of ESXVM or KVMVM to specify that the "hardware" of the new

Application Node is a virtual machine on an ESXi host or a KVM host. In addition to

its usual functions such as reserving IP addresses or creating LDAP data, the script



then creates a virtual machine with the same name as the Application Node on the

Hypervisor Node specified on the command line. The memory size and number of

CPUs for the virtual machine are set to default values, but can also be specified on

the command line (options --vmem and --vcpus). To select appropriate values for

these parameters, the available resources of the Hypervisor Node must be taken

into account, as well as the intended use of the Application Node. See section 7.7.9

”Virtual Machine Properties and Hypervisor Resources” on page 125 for details on

this topic.

cn1:~ # ff_an_adm.pl --op add --name <node name> --type ESXVM


--ospath <path to os image> --hn <node name> ...

or

cn1:~ # ff_an_adm.pl --op add --name <node name> --type KVMVM


--ospath <path to os image> --hn <node name> ...

Proceed as you would do it for any other Application Node.

Create a new personalized boot image for the Application Node (if not implicitly done using the --new-image operand of ff_an_adm.pl):

cn1:~ # ff_new_an.sh -n <node name>

Then boot the node by calling:

cn1:~ # ff_ponpoff.pl --op power-on --name <node name>

7.7.2 Virtualization related global FlexFrame parameters

Besides the Hypervisor Nodes and virtual machines, some FlexFrame global parameters

must be taken into account when planning a FlexFrame system using server

Virtualization functionality. They are usually set with the FlexFrame Management Tool,

but can be also set with administrative tools in some special cases.

7.7.2.1 System Code for Hypervisor Nodes and VMs

The FlexFrame system code for Hypervisor Nodes and virtual machines is a numeric

value between 0 and 63 that is used to generate MAC addresses for FlexFrame virtual

machines and to build names of some Hypervisor Node related resources such as Port

Groups and Datastores (VMware terminology) or Networks and Storage Pools

(KVM/libvirt terminology). Its purpose is to differentiate these objects belonging to

different FlexFrame systems used in a common environment, for example if using a

common vCenter Server or a not completely separated ethernet network.



The FlexFrame system code is usually set with the FlexFrame Management Tool and

must not be changed for a system with already configured Hypervisor Nodes and virtual

machines.

The ability to set the System Code using the tool ff_hn_adm.pl is mainly provided for

cases when the FlexFrame system has been upgraded from an older version without

server virtualization support or if the initial FlexFrame setup has been done without server

virtualization usage.

Use a different system code for each FlexFrame system in your environment.

Do not change the system code for an existing FlexFrame system with

configured Hypervisor Nodes and virtual machines.

Synopsis

ff_hn_adm.pl --op set-syscode --syscode <system code>

Options

--op set-syscode

Sets the FlexFrame system code for Hypervisor Nodes and virtual machines.

--syscode <system code>

Numeric value between 0 and 63 to be used as FlexFrame system code

7.7.2.2 vCenter Server

More complex administrative actions and functionalities, as well as the centralized view

and monitoring of the VMware Infrastructure are outside the scope of FlexFrame. For

these functions, a vCenter Server can be used.

If FlexFrame ESX servers are administrated by a vCenter Server, the name and IP

address of the vCenter Server have to be made known to FlexFrame. This is usually

done with the FlexFrame Management Tool, but can also be done using the tool ff_hn_adm.pl, for example in cases when the FlexFrame system has been upgraded

from an older version without VMware support or if the initial FlexFrame setup has been

done without VMware usage. You can also remove the previously set vCenter Server

information, e.g. if the information set with the FlexFrame Management Tool is not

correct, and you can set the authentication data for the vCenter Server in FlexFrame, so

that the FlexFrame tools can access this server.

Synopsis

ff_hn_adm.pl --op vcenter-add --vc-name <vcenter name>

[--vc-ip <vcenter ip>] [--vc-in-cntllan]



ff_hn_adm.pl --op vcenter-rem

ff_hn_adm.pl --op vcenter-auth --user <user name>

--pass <password>

Options

--op vcenter-add

Add vCenter Server hostname and IP address to FlexFrame.

--op vcenter-rem

Remove vCenter Server information from FlexFrame.

--op vcenter-auth

Set authentication data for vCenter Server in FlexFrame.

--vc-name <vcenter name>

Defines the name of the vCenter Server with --op vcenter-add.

--vc-ip <vcenter-ip>

Defines the IP address of the vCenter Server with --op vcenter-add. It can be

an address in the FlexFrame Control LAN or an address outside FlexFrame, that is

reachable from the Control Nodes.

If the --vc-ip option is omitted, and --vc-in-cntllan is specified, a free

address from Control LAN is selected.

--vc-in-cntllan

Specifies that the vCenter IP address is in the Control LAN. If no address is given,

advises the script to select a free address from Control LAN.

--user <user name>

Specifies the user name for accessing the vCenter Server with --op vcenter-

auth.

--pass <password>

Specifies the password for accessing the vCenter Server.

If using vCenter Server in FlexFrame, the names and IP addresses of the ESX servers

must be resolvable on a DNS server that is accessible for the vCenter Server.

The FlexFrame ESX servers must be added manually to vCenter Server using their

hostnames as they are known in FlexFrame. Add the ESX servers to a Datacenter called

FlexFrame or, if more than one FlexFrame system is administrated with the same



vCenter Server, use a distinct Datacenter called FlexFrame-<i> for each FlexFrame

system, where i is the FlexFrame system code explained in section 7.7.2.1.

7.7.2.3 Control LAN Default Router for Hypervisor Nodes

If FlexFrame ESX servers are administrated by a vCenter Server, it may also be

necessary to use a Control LAN router that is set on the Hypervisor Nodes as default router. This can be done using the tool ff_hn_adm.pl with operation mode

defrouter.

With this operation mode, the address of a default router for the Control LAN can be set

or removed in the FlexFrame configuration, including the DHCP configuration. The

Hypervisor Nodes get this address via DHCP together with the IP address for HN

management.

Synopsis

ff_hn_adm.pl --op defrouter --defrouter <default router ip | 0>

Options

--op defrouter

Add or remove the address of the Control LAN default router to the FlexFrame

configuration

--defrouter <default router ip | 0>

Defines the IP address of the Control LAN default router. Specify a Control LAN

address to add a default router for the Hypervisor Nodes or to change the current

setting, or use the value 0 to remove it.

7.7.3 Adding Hypervisor Nodes

This section describes how to add a Hypervisor Node to an existing FlexFrame

environment.

Adding a Hypervisor Node to a FlexFrame environment may require a specific

FlexFrame license. For details on FlexFrame licenses refer to chapter 3.7

“Product Licenses”.

Synopsis

ff_hn_adm.pl --op add --type <system type> --name <node name>

--hntype <hypervisor node type>

--swgroup <switch group_ id> --mac <mac addresses>



[--host <ip host number>[,<IPMI ip host number>

[,<2nd IPMI ip host number>]]]

[--slot <cabinet_id/slot>] [--10gbit]


[--port switch:port,switch:port,switch:port

[,switch:port]]

Options

--op add

Adds a Hypervisor Node and displays some information about processing steps.


Specifies the hypervisor node type like ESX or KVM. Call ff_hn_adm.pl --help to

get a list of supported types


Specifies the product name and type. Call ff_hn_adm.pl --help to get a list of

supported system types

--name <node name>

The name of the Hypervisor Node. This name has to be unique for the entire

FlexFrame system. All interface names are based on this node name. The name has

to be lowercase and consists of 2 up to 13 characters (letters, digits and dashes). It

must start with a letter and must not end with a dash.


Defines the switch group the Hypervisor Node is connected to. This information is

necessary to assign and configure switch ports. Call ff_hn_adm.pl --help to get

a list of currently configured switch group IDs.

For blade servers, the --swgroup option may be omitted, as the switch group is

already defined with the blade cabinet.


Add here the MAC addresses of the server's data NICs.They are used to configure

DHCP. Use the colon separated hex notation for each MAC address. Concatenate

the two MAC addresses with a comma. A MAC address syntax example is

00:e0:00:c5:19:41.

For blade servers, the --mac option may be omitted. The MAC addresses are then

fetched via SNMP from the management blade.

--host <ip host number>[,<IPMI ip host number>

[,2nd IPMI ip host number>]]



Host part to be used to build the IP addresses for hypervisor management and,

depending on system type, for IPMI, all of them in the Control LAN.

If the --host option is omitted, the script uses free host numbers to calculate the IP

addresses.

--slot <cabinet_id/slot>

With PRIMERGY server blades use this option to define the cabinet and slot ID of

the server blade. New cabinets have to be defined with the

ff_bx_cabinet_adm.pl command.


Defines the switch group the server management interface (IPMI) is connected to. If

omitted the effective switch group is computed as follows: First the switch group given with --swgroup is considered, second the switch group used for the swgroup

out of band management is considered. Call ff_hn_adm.pl --help to get a list of

currently configured switch group IDs.

--10gbit

Specifies that the node is used with 10 Gigabit data NICs. The specification can be

omitted if the nodes system_type only allows usage with 10 Gigabit data NICs.

--port switch:port,switch:port,switch:port,[switch:port]

Defines the switch ports to be used. First two tuples are for data NIC 1 and 2 and allocated in the switch group defined with --swgroup. The following tuples are for

mgmt NICs and allocated in the effective switch group for management interfaces. If

--10gbit is specified, the first two ports must be 10 Gigabit capable ports

If --port is not specified, the script assigns free ports according to an internal

algorithm.

Example

cn1:~ # ff_hn_adm.pl --op add --name rx300s5-esx1 –hntype ESX

--type RX300S5 --swgroup 2 --mac 00:19:99:48:e2:aa,00:19:99:48:e2:ab

update LDAP

.........



restart cluster service dhcpd

stopping dhcpd (timeout=20) . dhcpd:done . done.

starting dhcpd (timeout=20) .. dhcpd:done . done.

Connect your systems LAN interfaces to named switch ports:

LAN Interface SwGroup / Switch / Port

data NIC-1 2 / 2 / 2

data NIC-2 2 / 1 / 2

IPMI NIC-1 2 / 2 / 24



The script first checks all arguments and aborts with an error message in case of errors. It

then fetches free IP addresses and switch ports or checks whether the given ones can be

used for the node. LDAP data are created and the switch ports are reconfigured to match

the requirements. The DHCP service on the control nodes is reconfigured to assign the

hypervisor management IP address in the Control LAN based on the given MAC

addresses.

At the end, you get a cabling advice if the node must be connected to a switch group.

How to continue

The next step is to do the cabling according to the advice if applicable.

Then proceed with installing/booting and preconfiguring the Hypervisor Node using the

server console (see section "Getting Started with Hypervisor Nodes and VMs" on page

110 and the sections "Install and boot Hypervisor Node" and "Hypervisor Node

Preconfiguration on the Server Console" of the manual "Installation of a FlexFrame

Environment").

Finally, complete the Hypervisor Node configuration as shown in the next section.

7.7.4 Completing Hypervisor Node Configuration

This section describes how to complete the Hypervisor Node configuration.

Synopsis

ff_hn_adm.pl --op complete-config [--name <node name>]

--user <user name> --pass <password>

Options

--op complete-config

Completes the Hypervisor Node configuration and makes provisions so that

FlexFrame scripts can access the server without user interaction. In case of an ESX

server, the authentication data are stored in FlexFrame for subsequent access of the

server.

--name <node name>

Name of the Hypervisor Node. If the name is omitted, the operation is done for all

defined Hypervisor Nodes. This possibility is intended for usage during the

installation process.

--user <user name>

Specifies the user name for accessing the Hypervisor Node.

--pass <password>

Specifies the password for accessing the Hypervisor Node.



Example

cn1:~ # ff_hn_adm.pl --op complete-config --name rx300s5-esx1 --user

root

--pass password

The script accesses the ESXi server using the given authentication data and completes

the ESXi server configuration to match the FlexFrame needs, e.g. the FlexFrame Control

Nodes are set as NTP servers and as trap targets for SNMP. If it is already known for

which pools the ESXi server must be prepared, the corresponding Port Groups and

Datastores are created on the ESXi server (see section "Hypervisor Nodes and Pools" on

page 122 for details).

Finally, the script stores the authentication data needed to access the ESXi server to

enable the subsequent access of the server by FlexFrame scripts.

7.7.5 Removing Hypervisor Nodes

To remove a Hypervisor Node from the FlexFrame configuration, the only needed

parameter is the node name. The node is removed from the LDAP database, the hosts

file and the DHCP configuration. The switch ports used by this node are also

unconfigured. A Hypervisor Node cannot be removed if there are still virtual machine

Application Nodes that use this Hypervisor Node in the FlexFrame configuration or if the

Hypervisor Node hosts a Control Node.

Please make sure that you really want to remove the Hypervisor Node, as the

script does not ask for confirmation.

Synopsis

ff_hn_adm.pl --op rem --name <node name>

Options

--op rem

Removes a Hypervisor Node from the FlexFrame configuration.

--name <node_name>

Name of the Hypervisor Node.

Example

cn1:~ # ff_hn_adm.pl --op rem --name rx300s8-kvm1



7.7.6 Displaying Information about HNs and VMs

Using the script ff_hn_adm.pl, one can get an overview about all Hypervisor Nodes,

or detailed information about a specific Hypervisor Node including its virtual machines.

With ff_vm.pl, it is possible to get an overview of all FlexFrame virtual machines,

irrespectively of the involved Hypervisor Node.

Synopsis

ff_hn_adm.pl --op list-all --hntype <hypervisor node type>

ff_hn_adm.pl --op list --name <node name> [--cmdline]

Options

--op list-all

Displays an overview of all Hypervisor Nodes and also shows virtualization related

global FlexFrame parameters, such as the vCenter Server name and address or the

FlexFrame system code.


Specifies the hypervisor node type together with --op list-all in order to get

all Hypervisor Nodes of this type listed.

--op list

Displays detailed information about a specific Hypervisor Node, including its virtual

machines.

--name <node name>


--cmdline

Used with --op list, the command line that can be used to recreate the

Hypervisor Node is displayed at the end of the node listing.

Synopsis

ff_vm.pl --op list-all

Options

--op list-all

Displays an overview of all FlexFrame virtual machines, irrespective of the hosting

Hypervisor Node.



Examples

Overview of all Hypervisor Nodes and global information:

cn1:~ # ff_hn_adm.pl --op list-all

Global information

vCenter Server Hostname: vcenter-vm

vCenter Server IP: 172.50.6.68

FlexFrame Systemcode: 4

Control LAN Default Router: 172.50.6.251

Hypervisor Nodes sorted by name

bx92-01-esx

Hypervisor Type: ESX


Cabinet/Slot ID: 2/1

Mgmt IP/Hostname: 172.50.6.62 / bx92-01-esx

Mac Addr.: 00:1b:24:3d:a2:57 00:1b:24:3d:a2:58

bx92-02-kvm

Hypervisor Type: KVM


Cabinet/Slot ID: 2/2

Mgmt IP/Hostname: 172.50.6.63 / bx92-02-kvm

Mac Addr.: 00:1b:24:3d:a2:63 00:1b:24:3d:a2:64

Detailed information about a specific Hypervisor Node:

cn1:~ # ff_hn_adm.pl --op list --name rx3s7-kvm1

Configuration details of KVM node rx3s7-kvm1

Hardware

System: RX300S7

10GBit: No

Shut.Facil.: IPMI rx3s7-kvm1-co (10.10.13.27)

Mac Addr.: 00:19:99:48:e2:aa 00:19:99:48:e2:ab

Hypervisor management interface (Control LAN)

Host IP: 10.10.13.16

Hostname: rx3s7-kvm1


LAN Interface SwGroup / Switch / Port

data NIC-1 2 / 2 / 2

data NIC-2 2 / 1 / 2



IPMI NIC-1 2 / 2 / 24

Hypervisor resources

Memory Size: 32753 MB

CPU Cores: 8

Virtual Machines

List of FlexFrame Virtual Machines:

Name Type State CPUs Memory Pool Group

kvm-vm13 AN Off 2 2048 pool1 p1-vms

cn1 CN On 2 8192

-------------------------------------

Total 4 10240

Overview of all FlexFrame virtual machines:

cn1:~ # ff_vm.pl --op list-all

List of Virtual Machine - Application Nodes registered in LDAP

VM-AN Name Type Pool Group Hypervisor Node

esx-vm31 ESXVM pool3 p3-vms rx3s7-esx1

esx-vm32 ESXVM pool3 p3-vms rx3s7-esx2

kvm-vm13 KVMVM pool1 p1-vms rx3s7-kvm1

List of Virtual Machine - Control Nodes registered in LDAP

VM-CN Name Type Hypervisor Node

cn1 KVMVM rx3s7-kvm1

cn2 KVMVM rx3s7-kvm2

Collecting information from Hypervisor Nodes:

rx3s7-esx1 ... done

rx3s7-esx2 ... done

rx3s7-kvm1 ... done

rx3s7-kvm2 ... done

List of FlexFrame Virtual Machines found on available hypervisor nodes

VM Name Hypervisor Node State

cn1 rx3s7-kvm1 On

cn2 rx3s7-kvm2 On

esx-vm31 rx3s7-esx1 On

esx-vm32 rx3s7-esx2 On

kvm-vm13 rx3s7-kvm1 Off



7.7.7 Hypervisor Nodes and Pools

A Hypervisor Node can host virtual machine Application Nodes from different pools. For

each pool the server needs some specific resources, such as a virtual machine Port

Group or a virtual network for each of the pool networks and two datastores or storage

pools (config and software) in the pool specific volFF volume of the pool.

Usually, these resources are implicitly prepared when the first virtual machine Application

Node of a pool is created on a Hypervisor Node, so that there is no need to do an explicit

pool preparation of the Hypervisor Node.

However, if a FlexFrame virtual machine is moved to another Hypervisor Node by

mechanisms outside FlexFrame (see also section 7.7.10), these resources must be

prepared in advance. This can be done by using the add-pool operation of the

ff_hn_adm.pl script.

If no longer needed, the prepared resources can be frred by using the rem-pool

operation of the ff_hn_adm.pl script.

Synopsis

ff_hn_adm.pl --op add-pool --name <node name> --pool <pool name>

ff_hn_adm.pl --op rem-pool --name <node name> --pool <pool name>

Options

--op add-pool

Prepares a Hypervisor Node for usage by virtual machine Application Nodes from

given pool.

--op rem-pool

Removes pool specific resources from a Hypervisor Node.

--name <node name>


--pool <pool name>

Name of the pool.

Example

cn1:~ # ff_hn_adm.pl --op add-pool --name rx3s7-kvm1 --pool pool3



7.7.8 Special Functions for Virtual Machines

A virtual machine designated for use as a FlexFrame Application Node is usually created when the Application Node is added to the FlexFrame configuration with ff_an_adm.pl

--op add. Likewise, the virtual machine is destroyed when the associated Application

Node is removed using ff_an_adm.pl --op rem.

Under special circumstances, it is necessary to create FlexFrame virtual machines for

Application Nodes already defined in LDAP, for example during the FlexFrame

installation process. For special purposes, it may also be helpful to destroy a FlexFrame

virtual machine while preserving its LDAP entry, or if the LDAP entry is missing, e.g. as

an effect of an Application Node removal that failed to destroy the virtual machine.

The script ff_vm.pl provides these functionalities, as well as some other special

functions acting on virtual machines. This script is not intended for usual administrative

interactions with virtual machine Application Nodes. Use ff_an_adm.pl to administrate

these Application Nodes too in almost the same manner as other Application Nodes.

Synopsis

ff_vm.pl --op create [--name <node name>]

[--hn <node name>]

[--vcpus <number of virtual cpus>]

[--vmem <virtual machine memory size]

[--force]

ff_vm.pl --op destroy --name <nodename> [--force]

ff_vm.pl --op move --name <node name>

--to-hn <node name>

ff_vm.pl --op refresh-ldap [--name <node name>]

ff_vm.pl --op list --name <node name>

ff_vm.pl --op list-all



Options

--op create

Creates virtual machines for Application Nodes that are already defined in the

FlexFrame configuration database (LDAP). If called without option --name, a virtual

machine is created for each Application Node with system type ESXVM or KVMVM

found in LDAP, optionally restricted to the Application Nodes for which the Hypervisor Node specified with option --hn is preset in LDAP. Otherwise, only the

Application Node with the specified name is considered. The virtual machines get

the same names as the Application Nodes. The target Hypervisor Node, the number

of virtual CPUs and the memory size of the virtual machine are taken from LDAP by default, but can be overridden using the options --hn, --vcpus and --vmem.

--op destroy

Destroys a virtual machine that has been created for an Application Node while

preserving its LDAP entry or if the LDAP entry is missing. If the LDAP entry is missing, the operation is started only if the option --force is specified.

--op move

Moves a powered off virtual machine associated with an Application Node to another

Hypervisor Node.

--op refresh-ldap

Adjusts the LDAP information for a specific or for all FlexFrame virtual machine(s)

after changes done with mechanisms outside FlexFrame, e.g. when a virtual

machine associated with an Application Node has been moved to another

Hypervisor Node. For more information about using other mechanisms on

FlexFrame virtual machines see section 7.7.10 below.

--op list

Displays information about a specific FlexFrame virtual machine.

--op list-all

Displays information about all FlexFrame virtual machines. See also section 7.7.6.

--name <node name>

The name is used to identify a specific node. If specified, an Application Node with

this name and system type ESXVM or KVMVM must exist in LDAP, except for the

case when the --force option is specified with --op destroy.

With --op list and --op refresh-ldap, it is also possible to specify the name

of a Control Node with type KVMVM.

--hn <node name>



Specifies the node name of the Hypervisor Node where the virtual machine must be created. By default, the value from LDAP is used. When no <node name> is given,

the --hn option is used to select all VMs for creation for which the specified

Hypervisor Node is preset in LDAP.

--vcpus <number of virtual cpus>

Defines the number of virtual CPUs when creating a virtual machine. By default, the

value from LDAP is used. The argument must be a number between 1 and 500.The

really accepted numbers may depend on the underlying technology and licensing.

Moreover, the available resources of the Hypervisor Node must be taken into

account. See also section 7.7.9 below.

--vmem <virtual machine memory size>

Defines the memory size in MB when creating a virtual machine. By default, the

value from LDAP is used. The memory size must be a number between 256 and

261120 (=255GB). Moreover, the available resources of the Hypervisor Node must

be taken into account. See also section 7.7.9 below.

--force

Specifies that the memory usage of the Hypervisor Node may be overcommitted

when creating virtual machines. Default is to deny creation of virtual machines on a

host if the total vmem of the virtual machines on this host, including the new ones,

exceeds the memory size of the host. With operation mode destroy, --force

allows to destroy a FlexFrame virtual machine even when no corresponding LDAP

entry exists.

--to-hn <node name>

This option is used with --op move to specify the name of the target Hypervisor

Node.

7.7.9 Virtual Machine Properties and Hypervisor Resources

In a vSphere environment, ESX/ESXi hosts provide resources for virtual machines.

Similarly, KVM hosts provide resources for virtual machines in a SUSE Linux KVM based

environment. Resources include CPU, memory, power, storage and network resources.

In this section, we focus on CPU and memory resources, as well as on storage

requirements.

As a part of the virtual hardware properties, the memory size and the number of virtual

CPUs are defined when creating a virtual machine. In a FlexFrame environment, values

for these parameters are either predefined with the FlexFrame Management Tool for

each virtual machine Application Node and can be later overridden when creating the

virtual machine using the ff_vm.pl script, or are defined when creating the Application

Node including its virtual machine with ff_an_adm.pl. If using a virtual Control Center,

the values for the memory size and number of CPUs for the virtual machine Control



Nodes are predefined with the FlexFrame Management Tool and cannot be overridden

later.

When choosing values for these parameters, a careful planning must be done that takes

into account which machines will be powered on on a given Hypervisor Node at the same

time and for what kind of SAP workload they are planned to be used.

VMware techniques allow to configure more memory for the virtual machines on an ESX

server than the available physical memory of the host. This feature is called memory

overcommit and basically assumes that a VM does not use all assigned memory and

therefore it can be shared with other VMs. Similar techniques also exist for KVM.

For SAP systems it is strongly recommended not to overcommit memory usage, because

SAP allocates memory permanently and does not release it again. Furthermore, a

minimum of 4GB should be used, and for Unicode systems, a minimum of 8GB.

You should also be aware that if the memory size defined for a virtual machine is not

sufficient for the workload, excessive Linux swapping in the virtual machine may occur

which has a negative impact on performance and leads to an increased I/O load on the

filer and network which may even disturb the correct operation of the system.

A virtual machine can be configured with a number of virtual CPUs that depends on the

virtualization platform and the hypervisor vendor’s licensing, but it cannot be powered on

if it has more CPUs than the number of logical processors of the host - that is the number

of physical processor cores if hyperthreading is disabled or two times the number of

physical processor cores if hyperthreading is enabled. Similarly as with memory,

virtualization techniques allow to overcommit processor usage. When a host runs multiple

virtual machines that require more than the available CPU resources, the host time-slices

the physical processors across all virtual machines so that each virtual machine runs as if

it has its specified number of virtual processors. The CPU virtualization adds varying

overhead with performance implications.

SAP has successfully run performance tests in vSphere virtual machines (utilizing all

available virtual CPUs to 100%) which overcommitted the host system up to 200%. The

performance degradation inside the virtual machines was linear reciprocal to the

overcommitment. You may exceed the 200% overcommitment, but keep in mind, that the

performance of virtual machines in such a scenario is not guaranteed. In case of

performance problems, SAP can demand you to shutdown or pause other running virtual

machines to check if the overcommitment caused problems.

When creating virtual machines using FlexFrame scripts, the parameters given for the memory size and the number of virtual CPUs (--vmem and --vcpus) are roughly

checked for plausible values and it is checked that the memory usage of the designated

Hypervisor Node is not overcommitted, but no further checks and resource evaluations

are carried out. This task is in the responsibility of the specialists that make the

configuration and sizing planning for the FlexFrame system and assumes a good

knowledge of the virtualization techniques and the requirements of the used SAP

systems.



A FlexFrame virtual machine associated with an Application Node also needs some

space on the pool specific volFF of the pool to which the Application Node belongs. The

most significant resources located here are the file representing the virtual hard disk of

the virtual machine, and for virtual machines on ESXi hosts, the file used by the VMkernel to swap out the virtual machine’s physical memory in certain shortage cases (vswp). The

size of the file representing the virtual hard disk is defined by the size of the virtual hard

disk, which is 36 GB for FlexFrame virtual machines. The size of the vswp file is

correlated to the memory size of the virtual machine. If there is not sufficient space

available, the creation of the virtual machine may fail or the virtual machine fails to start.

These space requirements must also be taken into account during the configuration and

sizing planning mentioned above.

7.7.10 Using Hypervisor Management Software

While the FlexFrame tools focus mainly on the FlexFrame specific aspects of Hypervisor

Nodes and virtual machines, there are other ways to access virtualization components,

such as the vSphere Client and the vSphere Command-Line Interface (vCLI) for vSphere

based components or the Virtual Machine Manager (virt-manager) and the virsh

command line for KVM. Moreover, for more complex administrative actions and

functionalities, as well as the centralized view and monitoring of the VMware

Infrastructure, a vCenter Server can be used.

These tools and functions are outside the scope of FlexFrame, and for information on

their usage please refer to the appropriate vendor documentation.

However, when using them for Hypervisor Nodes and virtual machines known in

FlexFrame, it must be taken care not to disturb the FlexFrame functionality. Some special

requirements are:

1. Virtual machines for FlexFrame usage must be created using FlexFrame tools

2. Do not rename FlexFrame virtual machines

3. Do not destroy FlexFrame virtual machines while the associated Application Node

still exists. Removing the Application Node usually also destroys the virtual machine.

4. Do not change the virtual machine properties and devices. As an exception, you may

change the number of CPUs and the memory size of a powered off virtual machine,

but take into account the resources of the involved Hypervisor Node. After such

changes, please call ff_vm.pl --op refresh-ldap to adjust the corresponding

LDAP settings, or use the script ff_ponpoff.pl to power on the virtual machine,

as this does an implicit refresh of the LDAP settings.

5. Do not move a FlexFrame virtual machine to a Hypervisor Node that is not part of the

same FlexFrame system. Virtual machines associated with Application Nodes may

be moved to other Hypervisor Nodes of the same FlexFrame system, but take care

of the hints listed below.

To move a powered off virtual machine associated with an Application Node to

Application Nodes Administration Power on / off / reboot of an Application Node


another Hypervisor Node of the FlexFrame system, you can use ff_vm.pl --op

move. Otherwise, you must also take care that the target Hypervisor Node is

prepared for the pool the virtual machine Application Node belongs to. After a move

done with non-FlexFrame tools, please call ff_vm.pl --op refresh-ldap to

adjust the corresponding LDAP setting as soon as possible.

If moving a powered on virtual machine (Live Migration, VMware vMotion), which

may be done as part of a project specific solution, also take into account the possible

higher network utilization and service unresponsiveness during the move. In case of

KVM Live Migration, also take care to make the move persistent and to “undefine” the virtual machine on the source host (e.g. with virsh commandline use the

additional options --persistent --undefinesource).

6. Do not suspend/resume FlexFrame virtual machines. This may result in a service

being executed twice when a service formerly running on the suspended virtual

machine has been moved to a spare node and then the other VM is resumed. For

virtual machines created by FlexFrame on ESXi 5.x hosts, a special attribute is set

which denies the suspend action on these virtual machines.

7.8 Power on / off / reboot of an Application Node

Manually powering on or off an Application Node would be very inconvienent. The script

/opt/FlexFrame/bin/ff_ponpoff.pl supports power on, power off and reboot of

Application Nodes using adequate controllers like iRMC, Management Blade or

Hypervisor. The script depends on credentials you configure when adding the Application

Node as described in the corresponding “FlexFrame® HW Characteristics Quickguide”.

Notice that if you power off an Application Node using ff_ponpoff.pl, the

FlexFrame Agents will see an outage and are reacting as configured if the

Application Node is under control of the FlexFrame Agents.

Synopsis

ff_ponpoff.pl --op power-on | power-off | reboot

--name <node name | @all>

Options

--op power-on power-on the node(s)

power-off power-off the node(s)

reboot reboot the node(s)

--name <node name> execute the above actions for a single node <node name>

@all execute the above actions for all nodes of a FlexFrame

NAS Systems Configuration Storage Systems Administration


8 Storage Systems Administration

8.1 NAS Systems Configuration

8.1.1 Adding a New NAS

Use ff_nas_adm.pl --op add to add a NAS to the FlexFrame environment.

Afterwards configure the NAS manually according the appropriate "FlexFrame® HW

Characteristics Quickguide ("Hardwaresteckbrief")" using the data displayed.

Synopsis

ff_nas_adm.pl --op add --name <node name> --type <system type>

[--swgroup <switch group id>]

[--host <ip host part>]

[--ports <port count>]

[--interface <interface name>]

[--rootvolume <path to root volume>]

[--parent <node name>]

[--ostype <ONTAP|cDOT>]

[--10gbit]

Options

--op add

Adds a NAS and displays some information about processing steps.

--name <node name>

Defines the node name of the NAS. The name may consist of 2 upt to 13 characters

(letters, digits and dashes). It must start with a letter and must not end with a dash.

--type <systemtype>

Defines the type of the NAS. See usage for a list of known NAS types.


Defines the switch group the NAS should be added to. See usage for a list of

configured switch group IDs.

--host <ip host part>

Defines the host part to be used to build IP addresses for the Control or Storage

LAN networks. If this option is omitted the script uses a free host number to calculate

the IP address.--ports <port count>

Storage Systems Administration NAS Systems Configuration


Defines the count of ports to be used with pool Storage LAN networks. If this option

is omitted the script uses the default of two ports. The maximum number of ports per

channel is 8 (dependant on hardware capabilities)..

--interface <interface name>

Defines the interface to be used for filer access. Default: storage

--rootvolume <path to root volume>

Defines the root volume of the filer. Default: /vol/vol0

--parent <node name>

Defines the parent NAS node. The parent of a NAS of type SVM resp. VFILER is the

NAS hosting the SVM resp. vfiler. The parent of a NAS running NetApp DATA

ONTAP in clustermode is the NAS of type cDOTCluster representing the cluster

management.

--ostype <ONTAP|cDOT>

Defines the os type the NAS is running.

--10gbit

NAS is a 10Gbit system.

Example for a NetApp FAS System

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op add --name filer2 --type FAS3220

--swgroup 1

update LDAP

.....



................................+

Some manual interventions are necessary to integrate the NAS

into FlexFrame environment.

The following list of actions has to be performed in order to integrate the NAS

into your FlexFrame landscape. Since your exact configuration may vary, these steps

have to be performed manually. However, the VIF must be named 'storage'.

The /etc/rc, /etc/exports and /etc/hosts.equiv have to be edited

at volume vol0.

These lines have to be added to or changed at /etc/rc:

hostname filer2

vif create multi storage <-- add your NICs here, eg. e0a e0b for 1GB/sec ports;

e1 e2 for 10GB/sec ports.



vlan create storage 13

ifconfig storage-13 192.168.20.4 netmask 255.255.255.0 broadcast 192.168.20.255

mtusize 1500 -wins up

options dns.enable off

options nis.enable off

savecore

These lines have to be added to or changed at /etc/exports:

/vol/vol0 -sec=sys,rw=192.168.20.2:192.168.20.1,anon=0

These lines have to be added to /etc/hosts.equiv:

192.168.20.2 root

192.168.20.1 root

As the switch ports are already configured the correct wiring between NAS and

switch ports has to be done. See below a list of cable connections.

Connect your NAS LAN interfaces to named switch ports:

SwitchGroup / Switch / Port LAN Interface

1 / 2 / 3 filer2 (FAS3220): port "data NIC-1"

1 / 1 / 3 filer2 (FAS3220): port "data NIC-2"

Finally execute command "mount /FlexFrame/filer2/vol0"

on both Control Nodes to mount filers vol0 at Control Nodes. This is necessary for

further automated configuration of filer.

The complete instruction above is listed at file /tmp/filer2-add/todo

8.1.2 Removing a NAS

To remove a NAS from FlexFrame landscape it must not be used by any pool.

ff_nas_adm.pl --op list will display the current configuration of the NAS.

The switch port configuration will be removed from switches and LDAP database.

Synopsis

ff_nas_adm.pl --op rem --name <node name>

Options

--op rem



Removes a NAS from FlexFrame landscape about the NAS configuration and

displays some information about processing steps.

--name <node name>

Defines the node name of the NAS.

Example

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op rem --name filer2



.............................+

update LDAP

.....

NAS successfully removed from network and LDAP.

8.1.3 Displaying all configured NAS

To get an overview of all configured NAS within the FlexFrame environment use the

operation mode list-all of the program ff_nas_adm.pl. It displays IP addresses

and names, type, switch ports and the link aggregation id, separated by NAS.

Synopsis

ff_nas_adm.pl --op list-all

Option

--op list-all

Displays all configured NAS.

Example

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op list-all

NAS configurations

filer

Control Lan

192.168.20.3 filer-co

Type: FAS3220

Shell: /usr/bin/ssh -l flexframe

Switch Link Aggregation

Port Count: 2

Link Aggr.ID: 5

Storage LAN switch ports

1 / 1 / 13 SwGroup / Switch / Port




Control LAN switch ports


Pools

pool1 192.168.2.131 filer-pool1-st master


pool5 192.168.40.3 filer-pool5 master

usr 192.168.5.3 filer-usr-st master



8.1.4 Displaying NAS Configuration

To display the detailed configuration of a NAS as known by LDAP database, use the

command ff_nas_adm.pl with operation mode list.

Synopsis

ff_nas_adm.pl --op list --name <node name>

Options

--op list

Displays the configuration of a switch port.

--name <node name>

Defines the node name of a NAS.

Example for a NetApp Filer:

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op list --name filer

NAS configurations

filer

Control Lan

192.168.20.3 filer-co

Type: FAS3220

Shell: /usr/bin/ssh -l flexframe

Switch Link Aggregation

Port Count: 2

Link Aggr.ID: 5

Storage LAN switch ports



Control LAN switch ports


Pools



pool5 192.168.40.3 filer-pool5 master

usr 192.168.5.3 filer-usr-st master



8.1.5 Adding a Pool to a NAS

To be able to use an existing NAS for a pool, the network connection to the NAS has to

be enhanced. On the NAS, a new virtual interface has to be created and on the switch

ports the new VLAN has to be configured. All these steps are done with ff_nas_adm.pl

using the operation mode add-pool, but the program will not change any exports.

Synopsis

ff_nas_adm.pl --op add-pool --name <node name> --pool <pool name>

[--host <ip host part>]

Options

--op add-pool

Adds the given pool to named NAS.

--name <node name>

Defines the node name of NAS.

--pool <pool name>

Defines the pool name the NAS has to be support. See usage for a list of configured

pools.

--host <ip host part>

Defines the host part to be used to build IP addresses for the Control or Storage

LAN networks. If this option is omitted, the script uses a free host number to

calculate the IP address.

Example

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op add-pool --name filer --pool pool4

update LDAP

....



...........+

vlan: storage-25 has been created

Pool pool4 successfully added to NAS, LDAP and network.

8.1.6 Removing a Pool from a NAS

The rem-pool mode is the opposite to the add-pool operation mode of

ff_nas_adm.pl. It removes the virtual interface to the pool and removes VLAN access

on NAS switch ports for the given pool. This action will not be permitted if any SID of the



pool uses this NAS or the NAS is the default NAS of the pool. Within the last case, the

pool interface of the NAS will be removed when removing the pool.

Synopsis

ff_nas_adm.pl --op rem-pool --name <node name> --pool <pool name>

Options

--op rem-pool

Removes the given pool from the named NAS.

--name <node name>


--pool <pool name>

Defines the pool name the NAS has to be support. See usage for a list of configured

pools.

Example

cn1:/opt/FlexFrame/bin # ff_nas_adm.pl --op rem-pool --name filer --pool pool4



...........+

update LDAP

....

Pool pool4 successfully removed from NAS, LDAP and network.

8.1.7 Create NAS HA Partnership

Synopsis

ff_nas_adm.pl --op partner –-name <node name> --partner

<node name>

Options

--op partner

Define a HA partnership for a NAS.

--name <node name>

Name of the NAS.

--partner <node name>



Name of a second NAS which should be partner to this NAS.

8.1.8 Moving a NAS

There are various reasons a NAS should be moved within the FlexFrame landscape.

A NAS should change its network connection, e.g. move from one switch group to another. This includes moving from and to FlexFrame external switches, distinguished by

the switch group id ‘0’ or changing the network speed. Use ff_nas_adm.pl --op

move to get the new network connection then change the connection to the displayed

ports manually. The NAS will not be available until the new connection is established.

A NAS should be replaced by a NAS of another type. Use ff_nas_adm.pl --op

move to set the new type then replace the NAS manually. The NAS will not be available

until the replacement is completed.

A vFiler should be migrated to another storage host. Use ff_nas_adm.pl --op move

to set the new storage host, then perform the migration manually using proper NetApp

commands. After migration adjust the vlan interface partnership manually both on the

former and on the new storage host. When performing offline migration the NAS will not

be available until the migration is completed.

Synopsis

ff_nas_adm.pl --op move --name <node name>

[--type <system type>]

[--swgroup <switch group id>]

[--1gbit|--10gbit]

[--parent <node name>]

[--interface <interface name>]

[--rootvolume <path to root volume>]

Options

--op move

move a NAS

--name <node name>

Defines the node name of the NAS to be considered.


Defines the new type if the NAS should be replaced by another type.


Defines the switch group the data ports of the NAS should be connected to. Use

switch group id 0 to connect to switches not managed from FlexFrame.



--10gbit

move to 10 Gbit Connection .

--1gbit

move to 1Gbit Connection.

--parent <node name>

Defines the new parent the NAS should be moved to.

--interface <interface name>

Defines the new interface the NAS should use.

--rootvolume <path to root volume>

Defines the new root volume the NAS should use.

8.1.9 Changing NAS Command Shell

To change the command shell used to configure the NAS, the ff_nas_adm.pl

command supports the change_sh operation mode.

The given shell command replaces the currently used command or default configuring the

named NAS. To change the shell for more than one NAS the procedure has to be done

for each NAS.

Be sure there is no password request using this shell command to connect the

NAS.

Synopsis

ff_nas_adm.pl --op change-sh --name <node name>

--shcmd <shell command>

Options

--op change-sh

Changes the sh command for NAS configuration.

--name <node name>


--shcmd <shell command>



Shell command (absolute path) used to configure NAS. If this option is omitted the

script uses the defaults /usr/bin/ssh for NAS systems.

Example

cn1:~ # ff_nas_adm.pl --op change-sh --name filer

–-shcmd '/usr/bin/ssh -l flexframe'

update LDAP

.

NAS shell command changed at LDAP.

Shell command changed to "/usr/bin/ssh -l flexframe".

8.1.10 Changing NAS Ports

To change the count of NICs of a virtual interface the ff_nas_adm.pl commands

supports the change-ports operation mode. The given number is the count of NICs the

interface should use. The difference between actual and given count of ports will be

allocated at switch group, proper configured and displayed at end of program run. The

additional wiring and configuration of NAS have to be done manually.

Synopsis

ff_nas_adm.pl --op change-ports --name <node name>

--ports <port count>

Options

--op change-ports

Change the count of NICs of a virtual interface.

--name <node name>


--ports <port count>

Count of NICs the interface should use.

Example

cn1:~ # ff_nas_adm.pl --op change-ports --name filer –-ports 4

update LDAP.

Expand StorageLAN vif with 2 NICs up to 4 NICs.

Do this before(!) connecting new NICs to switch ports.

The additional switch ports at Switch Group are already configured.

Interconnect additional NICs between NAS and the switches of



SwitchGroup 1 as noted below:

SwGrpID/SwitchID/Port Filer

1 / 1 / 20 <==> Filer data NIC-3

1 / 2 / 20 <==> Filer data NIC-4

Unless any errors are reported follow instructions above to extend

NAS link aggregate(s) to switch group.

Look at "/opt/FlexFrame/network/wiring-channel-extend-filer.txt"


8.1.11 Changing NAS Path Export

To change the export of a NAS path the ff_nas_adm.pl commands supports the

export operation mode. Generally FlexFrame scripts are exporting a path according the

need. In special cases manual use of this operation mode may be considered, e.g. when

described to do so in the corresponding "FlexFrame® HW Characteristics Quickguide

("Hardwaresteckbrief")".

FlexFrame distinguishes the following datatypes to determine a export:

datatype export for access of

cn control nodes.

pool control nodes, hypervisors and pool given wih --pool.

sid control nodes and sids in pool given with --pool.

an control nodes and node given with --an.

allro control nodes and readonly for everyone.

Synopsis

ff_nas_adm.pl --op export --name <node name>

--path <path>

--datatype <cn|pool|sid|an|allro>


[--an <node name>]

[--createpath]

[--setvoloptions]

Options

--op export

export a path on given NAS.

--name <node name>

Using SAN Storage Storage Systems Administration



--path <path>

path the NAS has to export.

--datatype <cn|pool|sid|an|allro>

type of data to determine the export

--pool <pool name>

pool the data is associated with.

--an <node name>

aplication node the data is associated with.

[--createpath]

create the path to be exported, the volume must already exist.

[--setvoloptions]

set the FlexFrame default vol options.

Example

cn1:~ # ff_nas_adm.pl --op export --name filer \

--path /vol/pool3volff --datatype cn --setvoloptions

NAS path exported.

8.2 Using SAN Storage

Instead of using volumes from a NAS system over NFS, the database datafiles (sapdata)

and database logfiles (saplog) of a SAP system can be placed on SAN based storage.

Configuring the SAN storage environment for a FlexFrame landscape is a complex task

that needs careful planning and design. A team of specialists for storage systems,

servers and SAN infrastructure must work out a concept that will be a guideline for the

complete setup.

Furthermore, it is required to consult the actual recommendations from the manufacturer

of the used storage and SAN components and from the involved partners as well as the

relevant documents from Fujitsu. These documents can contain important information

that must be taken into account when planning the configuration and when configuring

the individual components.

Storage Systems Administration Using SAN Storage


8.2.1 Getting started with SAN usage in FlexFrame

This section gives an overview of the steps needed when setting up a SAN configuration

for FlexFrame usage. Please note that some of these steps are outside the scope of

FlexFrame and are not described in detail here. The SAN configuration for FlexFrame will

usually be done by a storage systems expert in cooperation with a member of the

FlexFrame implementation team.

Before starting with the SAN setup, a detailed storage concept should be worked out.

Amongst others, this concept has to define which SAP systems will use SAN based

storage for the database, the number and size of LUNs that will be used for these SAP

systems and the Application Nodes on which the database instances with SAN based

storage will run.

In FlexFrame context, the term LUN refers to a storage area provided by a

storage system that can be seen as a disk device by a host. Depending on

system type, it may be referred to by a different term (e.g. volume).

It must be taken into account that all Application Nodes on which a database instance will

run must be able to access the LUNs used by the database. This are the Application

Nodes of the poolgroup to which the Node where the instance is first started manually

belongs.

Assign only a small number of Application Nodes to such a poolgroup and plan

to start only very few SAN based database instances in one poolgroup.

Avoid hundreds of parallel FC logins, as this may lead to problems in case of

some events like powering on all Application Nodes or rebooting a FC Switch.

The steps listed below give only a rough outline of what must be done. Some actions may

be bundled or executed in another sequence than shown, others may be omitted as the

environment already exists.

1. Install and configure the storage system(s)

The SAN storage system is not considered as a part of the FlexFrame system; you

can use LUNs from a storage system which is also used for other purposes in your

computing environment. The FlexFrame software itself does not need an

administrative access to the SAN storage system. However, if an administrative

access to the storage system from inside the FlexFrame environment is desired, you

can add it as an “External Connectivity”.

Perform this step based on the storage system documentation or use an already

configured storage system according to the advice of its administrator.

2. Install and configure the Fibre Channel switches

The Fibre Channel switches are also not a part of the FlexFrame system. The

administration and configuration of the SAN fabric and the Fibre Channel switches is



completely in the responsibility of the customer. The FlexFrame software itself does

not need an administrative access to the FC switches, but you can add the FC

switches as “External Connectivity” if wanted.

Perform this step based on the vendor documentation or use the existing

environment according to the advice of its administrator.

3. Prepare the Application Nodes

All Application Nodes that will run database instances of SAP systems configured for

SAN usage must be equipped with Host Bus Adapters (HBAs). To ensure that the

correct server type and type of Fibre Channel HBA is used, please consult the

FlexFrame Support Matrix. For performance reasons as well as to protect against

hardware failure, it is strongly recommended to use more than one HBA or at least a

dual ported HBA on each Application Node. Using more than four HBAs (or HBA

ports) is not recommended.

Depending on the used combination of OS/HBA/storage system, specific driver

settings may be needed in the Application Node image. These must be set using a

Maintenance Cycle as described in section 11.5 “Maintenance of Application Nodes -

Software”.

To manage the multiple paths between host and storage system, a multipath

software is needed.The FlexFrame Linux Application Node Images contain a ready-

to-use installation of the Linux native multipath software DM-MPIO.

4. Connect storage and Application Nodes

Before connecting the Application Nodes to the SAN storage system, please ensure

that your configuration including server, HBA, Fibre Channel switch and storage

system is supported according to the FlexFrame Support Matrix. Also check that all

used components are correctly configured according to the actual recommendations

from the respective manufacturer or partner and from the MatrixEP.

The storage and the Application Nodes are connected via the SAN fabric. Therefore,

connect the Fibre Channel ports of the storage system and of the Application Nodes

with the FC switches. For redundancy purposes, take care to connect the ports of an

Application Node to two different switches that are part of independent fabrics.

Arrange for a reasonable number of pathes between Application Nodes and storage

systems. There should be two or four pathes for each LUN. More than four pathes

are not recommended.

5. Implement zoning

Create zones on the FC switches to protect servers and storage systems. There is

no specific requirement on how zoning is to be done for a FlexFrame environment,

but it must be ensured that all Application Nodes on which a SAN based database

instance will be started can access the storage systems on which the LUNs of this

database are located.



6. Create LUNs on the storage system

For each SAN based database instance, create at least two LUNs on the storage

system, one for sapdata and one for saplog. The concrete number and attributes of

the LUNs are defined as a part of the storage concept.

For information on how to create LUNs (may be also referred to as volumes, storage

volumes, devices, depending on storage system type) refer to the documentation of

the used storage system.

7. Make LUNs accessible for Application Nodes (LUN masking/mapping)

On the storage system, perform actions to ensure that all LUNs needed for database

instances that will be started in a specific poolgroup are accessible for all Application

Nodes of this poolgroup.

Refer to the documentation of the used storage system for a description of the

specific actions.

8. Verify LUN visibility on Application Nodes

Check that all needed LUNs are visible on the designated Application Nodes. You

may use the ff_san_helper.pl script as an assistance

an1:~ # ff_san_helper.pl --op lun-info

If not all needed LUNs are visible on a specific Application Node, reboot the Node.

You may also use the script rescan-scsi-bus.sh according to the SLES

documentation.

9. Add volume group and LUN definitions to the FlexFrame configuration

Based on the LUNs provided by the storage system, volume groups using Linux

Logical Volume Manager (LVM or, more exactly, LVM2) will be created in a later

step. These volume groups will then be assigned to SAP systems for usage as

sapdata or saplog storage. As a prerequisite for these steps, the definitions of the

volume groups and optionally the LUNs on which the volume groups are based must

be added to the FlexFrame configuration.

You can add volume group and LUN definitions with the FlexFrame Management

Tool or using the command line:

cn1:~ # ff_volgroup_adm.pl --op add ...

cn1:~ # ff_volgroup_adm.pl --op add-lun ...

For details and examples see section 8.2.2 “Administrating Volume Groups and

LUNs” below.

10. Assign volume groups as sapdata/saplog storage to SAP systems (SIDs)



This can be done directly when adding a SID with ff_sid_adm.pl, or after the SID

has been added with ff_sid_mnt_adm.pl. The syntax is equivalent to the syntax

used for specifying an SID specific NAS volume, but use the keyword ‘_SAN’ instead

of the NAS system name and the volume group name instead of the NAS volume

name. For details on these commands see chapter 10 “SAP System Handling”.

Below only a part of the syntax is shown.

cn1:~ # ff_sid_adm.pl --op add --pool <pool name> --sid <SID> ...

cn1:~ # ff_sid_mnt_adm.pl --op add --pool <pool name> --sid <SID> \

cn1:~ # ff_sid_mnt_adm.pl --op add --pool <pool name> --sid <SID> \

--name sapdata:_SAN:<vg name>:<instnr>:<nodeid> \

--name saplog:_SAN:<vg name>:<instnr>:<nodeid>

It is also possible to assign SAN volume groups as sapdata/saplog storage to SIDs

using the FlexFrame Management Tool, in planning mode as well as in admin mode.

Please note that a SAN based database instance needs two different volume groups,

one for sapdata and one saplog and that these volume groups cannot be used for

another SID (or SID/instance/nodeid tuple, in case of HANA).

11. Create directory and file structure for SAP systems (SIDs)

With ff_setup_sid_folder.sh, some directories, files and symbolic links are

created on the NAS based volumes that are used by a SAP system. This must be

done also for SIDs using SAN based storage, as only the sapdata/saplog area is

located on SAN. Furthermore, this also creates links to the directories where the

respective SAN areas will be accessible on the Application Nodes instead of the

corresponding links into the NAS based sapdata/saplog volumes, so that one can

use the same specification for data and log volumes during the SAP installation

irrespective of the used storage type.

cn1:~ # ff_setup_sid_folder.sh --pool <pool> --sid <SID> [--instance <instnr>]

For SIDs created with the FlexFrame Management Tool, this step is done implicitly

either during the initial configuration or when the SID is added.

In any case, ff_setup_sid_folder.sh does not create the structure needed on

the SAN based storage. This task is completed during the next step based on an advice file that is created by ff_setup_sid_folder.sh if the SID is defined to

use SAN based storage.

12. Create volume groups, volumes and file systems for SIDs

For each SAP system that uses SAN based storage, volume groups containing the

designated LUNs, logical volumes as part of the volume groups and file systems on

top of the volumes must be created before the SAP installation of this system can be

started. Depending on SAP system type, also some directories must be created on

the file systems.



This step can be done manually or assisted by ff_san_helper.pl on an

Application Node that can access the LUNs needed by the respective SID.

an1:~ # ff_san_helper.pl --op setup-sid-stor --sid <SID> ...

For details and examples see section 8.2.3 “Completing SAN Setup for a SAP

System” below.

13. Check usability of file systems on all Application Nodes of the designated poolgroup

On each Application Nodes of the poolgroup where a database service will be

started, mount the file systems needed by the service using ff_san_mount.pl (--

action start), then umount them again using ff_san_mount.pl (--action

stop). This also handles the underlying volume groups and volumes as needed.

Begin with umounting on the Application Node where you created the file systems.

an1:~ # ff_san_mount.pl --action stop --sid <SID> ...

an2:~ # ff_san_mount.pl --action start --sid <SID> ...

an2:~ # ff_san_mount.pl --action stop --sid <SID> ...

For details about ff_san_mount.pl see section 8.2.4 “Mounting SAN based File

Systems of a SAP Service” below.

14. Start SAP installation on an Application Node of the poolgroup

Before starting the installation, mount the file systems again on the selected

Applicaton Node as shown above. You may also want to check that the file systems

are mounted on this Application Node:

an1:~ # ff_san_mount.pl --action status --sid <SID> ...

Refer to the “FlexFrame Orchestrator – Installation Guide for SAP Solutions” for brief

instructions for installating SAP in a FlexFrame environment.

8.2.2 Administrating Volume Groups and LUNs

Use ff_volgroup_adm.pl to administrate SAN based volume groups and LUNs in

FlexFrame.

In FlexFrame context, a LUN (Logical UNit) is a storage area provided by a storage

system that can be used on a host. Depending on storage system type, it may be called

also volume, storage volume or device. A LUN is identified in FlexFrame by a symbolic

name that is assigned by the FlexFrame administrator.

The LUN provided by the storage system is described by several detail data, like the LUN

global unique ID, the host specific LUN ID (number) and the storage system specific LUN

ID in conjunction with the storage system identification.



LUN administration in FlexFrame does not imply any activities on the storage systems,

which must be done manually by the storage administrator; it is restricted to recording of

information and some plausibility checks.

Volume groups are built on top of LUNs seen on a host and can be assigned to SAP

systems for usage as sapdata or saplog storage. This also needs some manual

preparations, like the creation of volume groups, volumes and file systems, which must

be done on an Application Node that has access to the underlying LUNs.

The script ff_volgroup_adm.pl only administrates the information about volume

groups and LUNs in the FlexFrame configuration.

8.2.2.1 Adding a Volume Group

Use ff_volgroup_adm.pl --op add to add a SAN volume group definition to the

FlexFrame configuration, optionally together with the definition of a LUN on which the

volume group is based. You can also create the LUN definition separately, or add

additional LUNs if the volume group is based on more than one LUN, as shown in next

section “Adding a LUN”. For more information about LUNs in FlexFrame refer to section

“Adding a LUN”.

After adding a volume group to the FlexFrame configuration, it can be assigned as

sapdata or saplog storage for the database of a SAP system with the command ff_sid_adm.pl or ff_sid_mnt_adm.pl. For details see chapter 10 “SAP System

Handling”.

Before a volume group can be actually created and used, it is also necessary that the

LUNs on which it is based are created on a storage system and that these LUNs are

accessible on at least one Application Node. Before the SAP system database instance

using the volume group is put into operation, it is to be ensured that all Application Nodes

of the poolgroup where the instance is started can access these LUNs. To document the

poolgroup where the volume group will be used, specify this poolgroup as an option of

the volume group definition.

Synopsis

ff_volgroup_adm.pl --op add

--name <vg name> --pool <pool name>

[--group <poolgroup name>] [--lun <lun symbolic name>

[--lunguid <global unique lun id> --hostlun <host lun id>

--storlun <storsys spec lun id> --storsysid <ssys id>]]

Options

--op add

Add a SAN volume group definition to the FlexFrame configuration

--name <vg name>



Name of the new volume group definition that is added to the FlexFrame

configuration. A volume group definition with the same name must not already exist

in the pool given with option --pool. The name may consist of 1 up to 32

characters (letters, digits, underscores, dots and dashes) and must not start with a

dash. The name must reflect the name of the real volume group that will be created

based on the designated LUNs on an Application Node with access to these LUNs.

--pool <pool name>

Name of the FlexFrame pool to which the volume group definition is added.

--group <poolgroup name>

Name of the poolgroup within the FlexFrame pool to which the Application Nodes

where the volume group can be used must belong.

--lun <lun symbolic name>

The LUN symbolic name is used to identify a specific LUN definition from the pool

given with option --pool. It is a FlexFrame internal name, while the real LUN is

identified by the LUN detail options --lunguid, --hostlun, --storlun and

--storsysid. If a LUN symbolic name is given, it is used to add a LUN reference

to the volume group definition that is created. If a LUN with this symbolic name is not

already known, it is added implicitly to the FlexFrame configuration. In this case, all

LUN detail options must be specified.

--lunguid <global unique lun id>

Globally unique LUN ID as assigned by the storage system (32 hex chars).

--hostlun <host lun id>

LUN ID (number) as seen on the host where the LUN is accessed.

--storlun <storsys spec lun id>

Storage system specific LUN ID.

--storsysid <storage system id>

Storage system identification or name.

The options --lunguid, --hostlun, --storlun and –storsysid must be

specified when a LUN definition is created and should reflect the corresponding

attributes oft he real LUN. For more information, see next section, “Adding a LUN”.

Example

cn1:~ # ff_volgroup_adm.pl --op add --name vg_data_ha1 --pool pool1 --group san1

update LDAP

cn1:~ # ff_volgroup_adm.pl --op add --name vg_logs_ha1 --pool pool1 --group san1

update LDAP



Example with LUN definitions

cn1:~ # ff_volgroup_adm.pl --op add --name vg_data_ha5 --pool pool1 --group san1 \

--lun HA5data --lunguid 600000E00D280000002802D6000B0000 --hostlun 6 \

--storlun 11 --storsysid dx200s3

update LDAP

cn1:~ # ff_volgroup_adm.pl --op add --name vg_logs_ha5 --pool pool1 --group san1 \

--lun HA5logs --lunguid 600000E00D280000002802D6000C0000 --hostlun 7 \


update LDAP

8.2.2.2 Adding a LUN

Use ff_volgroup_adm.pl --op add-lun to add a LUN reference to a volume group

or to add a LUN definition to the FlexFrame configuration.

Please note that for basic FlexFrame functionality it is not absolutely necessary to define

LUNs in FlexFrame, but when the LUNs on which the volume groups are based are

known, some enhanced functions are available.

Synopsis

ff_volgroup_adm.pl --op add-lun

[--name <vg name>] --pool <pool name>


[--lunguid <global unique lun id> --hostlun <host lun id>

--storlun <storsys spec lun id> --storsysid <ssys id>]

Options

--op add-lun

Add a LUN reference to a volume group or a LUN definition to the FlexFrame

configuration

--name <vg name>

If this option is given, it specifies the name of an already known volume group to

which a LUN reference will be added. Without this option, only a LUN definition is

created.

--pool <pool name>

Name of the FlexFrame pool to which the volume group and LUN definition belongs

to or is added.




The LUN symbolic name is used to identify a specific LUN definition from the pool given with option --pool. It is a FlexFrame internal name, while the real LUN is

identified by the LUN detail options --lunguid, --hostlun, --storlun and

--storsysid. If a LUN with this symbolic name is already known, it is used to add

a LUN reference to the volume group given with option --name. Otherwise, a LUN

definition is created. In this case, all LUN detail options must be specified.

--lunguid <global unique lun id>

Globally unique LUN ID as assigned by the storage system (32 hex chars).

--hostlun <host lun id>

LUN ID (number) as seen on the host where the LUN is accessed.

--storlun <storsys spec lun id>

Storage system specific LUN ID.

Choose a value that, together with the storage system identification or name, makes

it easy to identify the LUN (storage volume, device) when communicating with the

storage administrator.

--storsysid <storage system id>

Storage system identification or name.

The options --lunguid, --hostlun, --storlun and –storsysid must be

specified when a LUN definition is created and should reflect the corresponding

attributes of the real LUN. The example below shows how you can get the needed

information from an ETERNUS storage system.

Example

cn1:~ # ff_volgroup_adm.pl --op add-lun --name vg_data_ah2 --pool pool1 \

--lun AH2data --lunguid 600000E00D280000002802D600020000 --hostlun 0 \


update LDAP

cn1:~ # ff_volgroup_adm.pl --op add-lun --name vg_logs_ah2 --pool pool1 \

--lun AH2logs --lunguid 600000E00D280000002802D600030000 --hostlun 1 \


update LDAP

To get the data for the LUN definitions shown in this example, we used the CLI of an

ETERNUS storage system. In ETERNUS terminology, our LUNs are volumes.

In this example, we use the storage system name as storsysid and the volume

number as storlun. The volume name is used as lun symbolic name. This mapping



scheme is only a suggestion. It is also possible to use a part of the boxid as storsysid,

or to use the volume name as storlun.

The following mapping is mandatory: the volume UID must be specified as lunguid and

the LUN must be specified as hostlun.

CLI> show storage-system-name

Name [dx200s3]

Installation Site [Augsburg DC1a_004]

Contact []

Description []

CLI> show volume-mapping -volume-number 2,3

Volume Type UID

No. Name

----- -------------------------------- --------- --------------------------------

2 AH2data Standard 600000E00D280000002802D600020000

<Mapping>

LUN LUN Group Port

No. Name

---- ---- ---------------- ----------------

0 1 sid_group2 -

Volume Type UID

No. Name

----- -------------------------------- --------- --------------------------------

3 AH2logs Standard 600000E00D280000002802D600030000

<Mapping>

LUN LUN Group Port

No. Name

---- ---- ---------------- ----------------

1 1 sid_group2 -

8.2.2.3 Removing a Volume Group

Use ff_volgroup_adm.pl --op rem to remove a volume group definition from the


Synopsis

ff_volgroup_adm.pl --op rem

--name <vg name> --pool <pool name> [--keep] [--force]

Options

--op rem



Remove a volume group definition from the FlexFrame configuration.

--name <vg name>

Name of an existing volume group from pool given with option --pool.

--pool <pool name>

Name of the FlexFrame pool to which the volume group definition belongs.

--keep

Keep definitions of referenced LUNs. Without this option, the definitions of

referenced LUNs are also removed.

8.2.2.4 Removing a LUN

Use ff_volgroup_adm.pl --op rem-lun to remove a LUN reference from a volume

group or a LUN definition from the FlexFrame configuration.

Synopsis

ff_volgroup_adm.pl

--op rem-lun [--name <vg name>] --pool <pool name>

--lun <lun symbolic name> [--keep] [--force]

Options

--op rem-lun

Remove a volume LUN reference from a volume group or a LUN definition from the


--name <vg name>

If this option is given, it specifies the name of the volume group from which the LUN

reference will be removed. Without this option, only a LUN that is not referenced by

a volume group can be removed.

--pool <pool name>

Name of the FlexFrame pool to which the volume group or LUN definition belongs.

--keep

Keep definition of LUN. Remove only the LUN reference from thevolume group.



8.2.2.5 Changing Poolgroup Assignment of a Volume Group

Use ff_volgroup_adm.pl --op group to change the poolgroup assignment of a

volume group. The poolgroup assignment of a volume group documents on which

Application Nodes this volume group will be used. It must be ensured that all Application

Nodes of the poolgroup can access the LUNs on which the volume group is based.

When changing the poolgroup assignment, make sure that all Application Nodes of the

new poolgroup can access these LUNs. Likewise, the Application Nodes of the old

poolgroup no longer need access to these LUNs.

Synopsis

ff_volgroup_adm.pl

--op group --name <vg name> --pool <pool name>

--group {<poolgroup name>|0}

Options

--op group

Change poolgroup assignment of a volume group.

--name <vg name>

Name of an existing volume group from pool given with option --pool.

--pool <pool name>

Name of the FlexFrame pool to which the volume group definition belongs.


Name of the poolgroup within the FlexFrame pool to which the Application Nodes where the volume group can be used must belong. Use value 0 to remove the

current poolgroup assignment.

8.2.2.6 Displaying Information about Volume Groups and LUNs

Use ff_volgroup_adm.pl --op list to display details of a specific volume group,

--op list-all to display an overview of all volume groups, --op list-luns to

display an overview of all LUNs or --op list-select to display volume groups

according to selection criteria.

Synopsis

ff_volgroup_adm.pl

--op list --name <vg name> --pool <pool name> [--cmdline]



ff_volgroup_adm.pl

--op list-all [--pool <pool name>]

ff_volgroup_adm.pl

--op list-luns [--pool <pool name>] [--unused] [--cmdline]

ff_volgroup_adm.pl --op list-select --pool <pool name>

[--group {<poolgroup name>|0}]

[--lun {<lun symbolic name>|0}] [--sid {<SAP system ID>|0}]

Options

--op list

Display details of a specific volume group

--op list-all

Display an overview of all volume groups

--op list-luns

Display an overview of all LUNs

--op list-select

Display volume groups according to selection criteria

--name <vg name>

Identifies the volume group from the pool given with option --pool.

--pool <pool name>

Identifies the FlexFrame pool.


Select volume groups assigned to given poolgroup. Use the value 0 to select volume

groups with no poolgroup assignment.

--lun {<lun symbolic name>|0}

Select volume groups using the given LUN. Use the value 0 to select volume groups

with no LUN reference.

--sid {<SAP system ID>|0}

Select volume groups used as sapdata or saplog storage for given SID. Use the

value 0 to select volume groups not used by any SID.

--cmdline

If used with --op list or --op list-luns, the command line(s) that can be

used to recreate the dispIayed definitions is also shown.



Examples

Display details of a specific volume group and its used LUN(s), with command line(s):

cn1:~ # ff_volgroup_adm.pl --op list --name vg_data_ah2 --pool pool1 --cmdline

Configuration details of volume group vg_data_ah2 in pool pool1

Volume Mgr Type: LVM

Assigned poolgroup: san1

Used as: sapdata for SID AH2

Mountpath Info:

/dev/vg_data_ah2/data_AH2 on /var/FlexFrame/SAN/sapdata/AH2

Used LUNs: AH2data

Configuration details of used LUNs, grouped by storage system

Storage system dx200s3

LUN name LUN GUID h-LUN s-LUN

AH2data 600000E00D280000002802D600020000 0 2

Command line used to create volume goup definition:

ff_volgroup_adm.pl --op add --name vg_data_ah2 --pool pool1 --group san1

Command line(s) used to create LUN definition(s) and reference(s):

ff_volgroup_adm.pl --op add-lun --lun AH2data --name vg_data_ah2 --pool pool1

--lunguid 600000E00D280000002802D600020000 --hostlun 0 --storlun 2

--storsysid dx200s3

Display an overview of all LUNs:

cn1:~ # ff_volgroup_adm.pl --op list-luns --pool pool1

LUNs sorted by pool, grouped by storage system and sorted by lun name

(output reduced to pool pool1)

POOL pool1

Storage system dx200s3

LUN name LUN GUID h-LUN s-LUN

AH2data 600000E00D280000002802D600020000 0 2

AH2logs 600000E00D280000002802D600030000 1 3

HA3data 600000E00D280000002802D600050000 2 5

HA3logs 600000E00D280000002802D600060000 3 6

HA4data 600000E00D280000002802D600090000 4 9

HA4logs 600000E00D280000002802D6000A0000 5 10

HA5data 600000E00D280000002802D6000B0000 6 11



HA5logs 600000E00D280000002802D6000C0000 7 12

Selecting volume groups of a pool assigned to a given poolgroup:

cn1:~ # ff_volgroup_adm.pl --op list-select --group san1 --pool pool1

Volume Groups from pool pool1

Selecting only volume groups assigned to poolgroup san1

vg_data_ah2

Used LUNs: AH2data


Used as: sapdata for SID AH2

vg_data_ha4

Used LUNs: HA4data


Used as: -

vg_data_ha5

Used LUNs: HA5data


Used as: sapdata for SID HA5, instnr 05, nodeid 001

vg_logs_ah2

Used LUNs: AH2logs


Used as: saplog for SID AH2

vg_logs_ha4

Used LUNs: HA4logs


Used as: -

vg_logs_ha5

Used LUNs: HA5logs


Used as: saplog for SID HA5, instnr 05, nodeid 001

Selecting volume groups of a pool not used by any SID:

cn1:~ # ff_volgroup_adm.pl --op list-select --sid 0 --pool pool1

Volume Groups from pool pool1

Selecting only volume groups not used by any SID

vg_data_ha4

Used LUNs: HA4data


vg_logs_ha4

Used LUNs: HA4logs




8.2.3 Completing SAN Setup for a SAP System

Before the volume groups that have been defined as shown in chapter 8.2.2 and

assigned to a SAP system (SID) for usage as sapdata or saplog storage can be used for

the database of this SAP system, it is also required to actually create these volume

groups and further objects based on the volume groups. This must be done on an

Application Node that can access the LUns on which the volume group is based.

These objects must complement the file and directory structure as created by the script ff_setup_sid_folder.sh and match the LDAP data that reflect the volume group

definitions and usage. You may use ff_san_helper.pl --op setup-sid-stor as

described in section 8.2.5 “SAN Helper Functions” or do the needed actions manually.

The script can only be used if LUN definitions exist for the volume groups.

1. For each defined volume group, create a Linux LVM (or, more exactly LVM2) volume

group with the defined name based on the multipath maps corresponding to the

designated LUNs as physical volumes, Tag the volume group with a “_FF4S” tag to

mark it for FlexFrame usage and a tag with the name or the current Application

Node.

2. In each volume group, create a logical volume named data_<SID> or log_<SID>,

respectively.

3. Create an ext3 file system on top of each logical volume,

4. For all database types except HANA, several directories must be created on each file

system.The needed informations can be found in the files dirlist_san_sapdata_<SID>.cfg and dirlist_san_saplog_<SID>.cfg

in the directory /FlexFrame/pooldata/config/scripts_config/ (as seen

on Application Node).

Before creating the directories, create mount points and mount the file systems on

the appropriate mount points as recorded with the volume group definitions (see

output of ff_volgroup_adm.pl --op list ...).

After completing these actions, umount the file systems (if mounted), deactivate and

export the volume groups and remove the host specific tag from the volume groups. Do this by calling ff_san_mount.pl --action stop --sid ... as shown in section

8.2.4.

Example

The example below shows a shortened list of the commands executed by

ff_san_helper.pl –op setup-sid-stor –sid AH4.



vgcreate --addtag _FF4S --addtag lucrx1 vg_data_ah4 \

/dev/disk/by-id/dm-name-3600000e00d280000002802d600090000

lvcreate --name data_AH4 --extents 100%VG vg_data_ah4

mkfs.ext3 /dev/vg_data_ah4/data_AH4 -L data_AH4

mkdir -v -p /var/FlexFrame/SAN/sapdata/AH4

mount /dev/vg_data_ah4/data_AH4 /var/FlexFrame/SAN/sapdata/AH4

mkdir -v -m 775 -p /var/FlexFrame/SAN/sapdata/AH4/sapdata1

chown -c 3608:3602 /var/FlexFrame/SAN/sapdata/AH4/sapdata1

mkdir -v -m 775 -p /var/FlexFrame/SAN/sapdata/AH4/sapdata2

chown -c 3608:3602 /var/FlexFrame/SAN/sapdata/AH4/sapdata2

<...>

vgcreate --addtag _FF4S --addtag lucrx1 vg_logs_ah4 \

/dev/disk/by-id/dm-name-3600000e00d280000002802d6000a0000

lvcreate --name log_AH4 --extents 100%VG vg_logs_ah4

mkfs.ext3 /dev/vg_logs_ah4/log_AH4 -L log_AH4

mkdir -v -p /var/FlexFrame/SAN/saplog/AH4

mount /dev/vg_logs_ah4/log_AH4 /var/FlexFrame/SAN/saplog/AH4

mkdir -v -m 775 -p /var/FlexFrame/SAN/saplog/AH4/saplog1

chown -c 3608:3602 /var/FlexFrame/SAN/saplog/AH4/saplog1

<...>

8.2.4 Mounting SAN based File Systems of a SAP Service

The script ff_san_mount.pl is used on Application Nodes during start or stop of the

database service of a SAP system to control the availability of SAN based volume groups

and file systems needed by the service.

Besides this, it must be used manually during the installation of a SAP service that is

configured for SAN usage.

The related information is maintained in the FlexFrame configuration database (LDAP).

This can be done with the FlexFrame Management Tool or with the commands ff_volgroup_adm.pl to administrate volume groups and LUNs and

ff_sid_mnt_adm.pl to assign volume groups to SAP services.

Caution must be applied when using ff_san_mount.pl directly. The configured volume

groups and file systems must not be in use by another host.

Once the database is properly installed, the script ff_service.sh calls

ff_san_mount.pl during the start or stop phase implicitly.

Synopsis

ff_san_mount.pl --action {start | stop | status}

--sid <SAP system ID>

[--instance <HANA instance number> --nodeid <HANA node ID>

[--replid <HANA replication ID> ]]



Options

--action {start | stop | status}

Action mode.

Choose one of start to start, stop to stop or status to show status of SAN based

volume groups and file systems needed by the database service of a SAP system.

With action mode start, the volume groups are imported and activated and the

associated file systems are mounted. Activation also implies setting of a node

specific tag on the volume group.

With action mode stop, the file systems are umounted and the volume groups are

deactivated and exported. With the deactivation, the node specific tag is also

removed from the volume group.

Action mode status shows if the volume groups are active on this or another node

and if the file systems are mounted on this node. It does not imply a check of the

functional availability.


Identifies the SAP system to which the database service belongs.

--instance <HANA instance number>

HANA instance number (2-digit number), required for SAP HANA service.

--nodeid <HANA node ID>

Identifier of HANA database node (3-digit number), required for SAP HANA service.

--replid <HANA replication ID>

Identifier of HANA database replication node (1-digit number), only relevant for SAP

HANA service.

8.2.5 SAN Helper Functions

Use ff_san_helper.pl on an Application Node for helper functions related to the

preparation of storage for the database service of SAP systems that use SAN based

storage.

Synopsis

ff_san_helper.pl --op setup-sid-stor --sid <SAP system ID>

[--instance <HANA instance number> --nodeid <HANA node ID>

[--replid <HANA replication ID> ]]

ff_san_helper.pl --op hba-info



ff_san_helper.pl --op target-info

ff_san_helper.pl --op lun-info

Options

--op setup-sid-stor

Helps to complete the setup of storage for a SAP system. Refer to section 8.2.3

above for details.

--op hba-info

Display some brief information about FC HBAs of the Application Node

--op target-info

Display some brief information about FC targets seen on the Application Node

(usually corresponding to the ports of the SAN storage systems).

--op lun-info

Display some brief information about FC LUNs seen on the Application Node.

The following options are relevant only for --op setup-sid-stor.


Identifies the SAP system to which the database service belongs.

--instance <HANA instance number>

HANA instance number (2-digit number), required for SAP HANA service.

--nodeid <HANA node ID>

Identifier of HANA database node (3-digit number), required for SAP HANA service.

--replid <HANA replication ID>

Identifier of HANA database replication node (1-digit number), only relevant for SAP

HANA service.

Examples

an1:~ # ff_san_helper.pl -op hba-info

HBA WWNN WWPN

host7 20:00:00:90:fa:02:a7:22 10:00:00:90:fa:02:a7:22

host8 20:00:00:90:fa:02:a7:23 10:00:00:90:fa:02:a7:23

an1:~ # ff_san_helper.pl -op target-info



Target WWNN WWPN

7:0:0 50:00:00:e0:da:02:d6:00 50:00:00:e0:da:02:d6:20

8:0:0 50:00:00:e0:da:02:d6:00 50:00:00:e0:da:02:d6:30

an1:~ # ff_san_helper.pl -op lun-info

H-B-T-L Dev Dev-Id(GUID)

7:0:0:0 sdb 600000e00d280000002802d600020000

7:0:0:1 sdc 600000e00d280000002802d600030000

7:0:0:2 sdd 600000e00d280000002802d600050000

7:0:0:3 sde 600000e00d280000002802d600060000

8:0:0:0 sdj 600000e00d280000002802d600020000

8:0:0:1 sdk 600000e00d280000002802d600030000

8:0:0:2 sdl 600000e00d280000002802d600050000

8:0:0:3 sdm 600000e00d280000002802d600060000

Switch Administration Adding a Switch to a Switch Group


9 Switch Administration

Switch administration means adding or removing a switch group to resp. from the

FlexFrame landscape, adding or removing a switch to resp. from a switch group or

adding or removing a module to resp. from a switch.

A switch group consists of at least two switches of a switch family. Only switch groups

building a switch stack can consist of more than two switches. A switch stack is seen from

the network like one switch.

For a description how to interconnect switches, please refer to the vendors

Installation Manual.

Adding or removing a switch to or from a switch group means to add or remove

a switch to/from a switch stack.

To ensure safe operation we recommend doing this at a downtime to minimize

influence on running systems. This requires shutting down all systems con-

nected to this switch group. In case a NAS system is connected, all systems that

have mounted file systems from the NAS system have to be shut down as well.

Ensure all switches in the stack have a proper Software Version. If necessary

perform an upgrade resp. downgrade.

In special scenarios an upgrade can only be done step by step one switch after

the other. Please refer to original vendors documents for that task.

9.1 Adding a Switch to a Switch Group

A switch may be added to a switch group building a switch stack. See notes above for

recommendations.

The following processing has to be performed.

1. Mount the new switch next to the existing switches

2. Run ff_save_switch_config.pl to save configurations

3. Write down switch ids of each stack member of the switch stack as they may change

inserting the new switch to the stack.

4. Check the OS versions of the switches in the existing switch stack and the new

switch. Switches of the same model (G or E model) must have the same version. If

the versions are different, upgrade resp. downgrade the OS of the new switch.

5. Power off all switches of the switch stack, connect the new switch to the stack using

the provided stacking cable and stacking ports (see the vendors installation manual

for details), power on all switches of the stack except the new one.

Adding a Switch to a Switch Group Switch Administration


6. Compare the actual stack member ids with your noticed ids. In case of differences

use the following switch OS command for renumbering: switch <Switch Number> renumber <New switch number>

7. Power on the new switch, once again compare the stack member ids, set all

interfaces of the new switch to shutdown

8. Use ff_swgroup_adm.pl --op add-sw to add the switch to the FlexFrame

configuration as described below.

The switch group is now ready for use for further configuration.

Synopsis

ff_swgroup_adm.pl --op add-sw --group <switch group id>

--type <switch type> [--dryrun]

Options

--op add-sw

Adds a new member to the switch group and displays some information about

processing steps.

--group <switch group id>

Defines the switch group to be used.

--type <switch type>

Defines the type of the new switch to be added to the switch group. Call

ff_swgroup_adm.pl without any parameter to get a list of supported switch types

The maximum number of switches per switch group is 9. For more than 4 switches

with 10GbE ports the StackWise cabling may be a bottleneck.

--dryrun

For test purpose you can perform the function without changing the LDAP Database.

Switch Administration Adding a Switch to a Switch Group


Example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op add-sw --group 1

--type cat3750g-24ts

If program is aborted by Ctrl-C or a failure remove left overs

by calling:

ff_swgroup_adm.pl --op rem-sw --group 1 --switch 3

Switch was added to LDAP data.

Keep in mind: INSERTING SWITCH TO STACK NEEDS A DOWN TIME !

To add the switch to switch stack (switch group 1) write down the current switch

ids as they may change inserting the new switch to stack.

To connect the switch to the stack use the provided stacking cable and stacking

ports at rear side. See Cisco installation manual for details.

If switch ids get scrambled use the IOS command

"switch <current_no> renumber <new_no>"

to put them in same order as before.

In short the to do list:

-> write down switch ids of each switch of group

-> power down entire switch group

-> insert switch into stack

-> power on entire switch group

-> look at switch ids and compare with your noticed

-> in case of differences use IOS command to renumber switches

Switch group is ready for use

See file /tmp/swgrp-add-1-3/next_steps for same instructions as above.

Removing a Switch from a Switch Group Switch Administration


9.2 Removing a Switch from a Switch Group

A switch may be removed from a switch group building a switch stack. Remove it first from the LDAP database with the command ff_swgroup_adm.pl --op rem-sw as

described below and then from the stack. See notes above for recommendations.

All ports of the switch must be unused.

Only the switch with the highest member ID within the stack can be removed.

Synopsis

ff_swgroup_adm.pl --op rem-sw --group <switch group id>

--switch <switch id> [--dryrun]

Options

--op rem-sw

Removes the last member from a switch group and displays some information about

processing steps.



--switch <switch id>

Defines the stack ID of the switch to be removed from a switch group.

--dryrun


Example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op rem-sw --group 1 --switch 3

Switch was successfully removed from LDAP data.

Keep in mind: REMOVING SWITCH FROM STACK NEEDS A DOWN TIME !

In short the to do list:

-> power down entire switch group

-> remove switch from stack

-> power on entire switch group

Switch group is ready for use

See file /tmp/swgrp-rem-1-3/next_steps for same instructions as above.

Switch Administration Listing a Switch Group Configuration


9.3 Listing a Switch Group Configuration

Invoking the command ff_swgroup_adm.pl with the list operation mode displays

the configuration of a switch group like used switch types, port channels, port usage

statistics and used switch ports.

Synopsis

ff_swgroup_adm.pl --op list --group <switch group id>

Options

--op list

Displays switch group configuration.



Example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op list --group 1

Switch Group 1

Name/IP: switch-i / 192.168.13.15

Login: root

Password: passwort

SNMP Community: public;ro

Switch Types: (switch id, switch type)

1 cat3750g-24t

2 cat3750g-24t

3 cat3750g-24t

Port Channels: (channel id, switch ports, connected device)

2 1/5,2/5 swb-1-1/11,swb-1-1/12

3 1/6,3/1 swb-1-2/11,swb-1-2/12

4 1/15,3/7 intfiler

5 1/19,2/11 swb-2-1/39,swb-2-1/40

6 2/22,3/22 swb-2-2/39,swb-2-2/40

Switch port usage: (switch id, used, free tx, free fx, free 10GB ports)

1 11 used 13 free tx 0 free fx 0 free 10Gb



Listing a Switch Group Configuration Switch Administration


Switch port list: (switch id, port id, connected device, vlans)

1 1 BX cabinet 2 u13

1 2 --- unused ---

1 3 rx300-14 t10,t12,u11

1 4 rx300-15 t10,t12,u11

1 5 swb-1-1/11 t13,t10,t12,t11,t1

1 6 swb-1-2/12 t13,t10,t12,t11,t1

1 7 --- unused ---

1 8 --- unused ---

1 9 --- unused ---

1 10 --- unused ---

1 11 cn1 u13,t11,t12,t10

1 12 cn2 u13,t11,t12,t10

1 13 --- unused ---

1 14 --- unused ---

1 15 intfiler t11,t13

1 16 extern. Connect t13,t10,t11,t12

1 17 --- unused ---

1 18 --- unused ---

1 19 swb-2-1/39 t13,t10,t12,t11,t1

1 20 --- unused ---

1 21 --- unused ---

1 22 --- unused ---

1 23 Corporate LAN u10


2 2 --- unused ---

2 3 rx300-14 t10,t12,u11

2 4 rx300-15 t10,t12,u11

2 5 swb-1-1/12 t13,t10,t12,t11,t1

2 6 --- unused ---

2 7 --- unused ---

2 8 --- unused ---

2 9 --- unused ---

2 10 --- unused ---

2 11 swb-2-1/40 t13,t10,t12,t11,t1

2 12 cn2 mgmt u13

2 13 --- unused ---

2 14 --- unused ---

2 15 cn1 mgmt u13

2 16 --- unused ---

2 17 --- unused ---

2 18 --- unused ---

2 19 --- unused ---

2 20 --- unused ---


Switch Administration Generating a Switch Group Configuration


2 22 swb-2-2/39 t13,t10,t12,t11,t1

2 23 Corporate LAN u10

3 1 swb-1-2/11 t13,t10,t12,t11,t1

3 2 --- unused ---

3 3 --- unused ---

3 4 --- unused ---

3 5 cn1 u13,t11,t12,t10

3 6 cn2 u13,t11,t12,t10

3 7 intfiler t11,t13

3 8 --- unused ---

3 9 --- unused ---

3 10 --- unused ---

3 11 --- unused ---

3 12 --- unused ---

3 13 --- unused ---

3 14 --- unused ---

3 15 --- unused ---

3 16 --- unused ---

3 17 --- unused ---

3 18 --- unused ---


3 20 rx300-15 mgmt u13

3 21 rx300-14 mgmt u13

3 22 swb-2-2/40 t13,t10,t12,t11,t1

9.4 Generating a Switch Group Configuration

Use the command ff_swgroup_adm.pl with the gen-config operation mode to

generate the configuration of switch group switches according to LDAP. The generated

configuration can be uploaded manually according to the corresponding "FlexFrame®

HW Characteristics Quickguide ("Hardwaresteckbrief")".

Customizations, if necessary, have to be reapplied. Inspection of the project

information see 11.10 and the last saved configurations in

/tftpboot/saved_switchconfigs may help.

If the connection to the NAS system may be lost because of the manually upload

change /tftpboot to be a local directory before generating the configuration. If the

generated configuration should be accessed by use of the tftp protocol also

restart tftpd.

mv /tftpboot /tftpboot.sav

mkdir /tftpboot

ff_ha_cmd.sh restart tftpd

Changing the Password of a Switch Group Switch Administration


After the upload go back using

mv /tftpboot/* /tftpboot.sav

rmdir /tftpboot

mv /tftpboot.sav /tftpboot

ff_ha_cmd.sh restart tftpd

Synopsis

ff_swgroup_adm.pl --op gen-config --group <switch group id>

Options

--op gen-config

Generates configuration of switch group switches.



Example

Cn1:~ # ff_swgroup_adm.pl --op gen-config --group 1

Switch configuration generated. Upload has to be done manually.

Follow the instructions of the applicable Quick Start Hardware

FlexFrame for SAP document in the doc/hwinfo section of the Service CD

and use the following data:

Switch Group swg1

IP and Mask: 172.20.13.6 255.255.255.0

Config: scp://[email protected]//tftpboot/swg1.config

You can find a copy of the message above in

/var/log/FlexFrame/tmp/swgroup_adm-gen-config-1/msg_of_swgroup1_swg1.txt

9.5 Changing the Password of a Switch Group

To change the access password of a switch group it has to be changed at switch group

as well as LDAP database. The command ff_swgroup_adm.pl changes both. If it is

not possible to change the password on the switch because of password complexity

restrictions use this command to reset to the previous password to get resynchronized.

Switch Administration Changing the Host Name of a Switch Group


Synopsis

ff_swgroup_adm.pl --op pass --group <switch group id>

--passwd <password> [--dryrun]

Options

--op pass

Changes the switch group access.



--passwd <password>

Defines the new password as clear text.

--dryrun


Example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op pass --group 1 --passwd berta



............+

Password changed from "anton" to "berta".

See file /tmp/swgrp-pass-1/info for same information as above.

9.6 Changing the Host Name of a Switch Group

To change the host name of a switch group it has to be changed at switch group as well

as in the LDAP database and host files on both control nodes. The command

ff_swgroup_adm.pl changes all of them.

Synopsis

ff_swgroup_adm.pl --op name --group <switch group id>

--name <name> [--dryrun]

Changing the Host Name of a Switch Group Switch Administration


Options

--op name

Changes the switch group host name. The new name may consist of 2 up to 13

characters (letters, digits and dashes). It must start with a letter and must not end

with a dash.


Defines the switch group to be used --name <name>

Defines the new host name to be used.

Switch Administration Displaying/Changing Common Network Configuration Parameters


--dryrun


Example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op name --group 1 --name swg1



...+

Switch name changed from "swg-1" to "swg1".

See file /tmp/swgrp-name-1/info for same information as above.

9.7 Displaying/Changing Common Network Configuration Parameters

Some parameters of network configuration influence the switch group configuration and

were defined with the Management Tool at initial installation.

To display or change these parameters use the ff_swgroup_adm.pl command.

Synopsis

ff_swgroup_adm.pl --op parameter [–-parameter <parameter name>

--value <parameter value>] [--dryrun]

Options

--op parameter

Displays (without the following options) or changes (in conjunction with the following

options) network configuration parameters.

--parameter <parameter name>

Defines the name of the parameter to be changed. Known parameters are:

usePortsToClan

The parameter specifies if dedicated switch ports for Client LAN connection should

be used:

● no

No dedicated switch ports for Client LAN connection are used. Uplinks are

connected to customer corporate LAN and the Client LANs are part of the

uplink or special uplinks for Client LAN connection are configured (see chapter

9.12).

Displaying/Changing Common Network Configuration Parameters Switch Administration


● yes

Dedicated switch ports for Client LAN connection should be used.

clanPortPerVlan

The parameter depends the way the Client LANs are connected to the corporate

LAN (with the sapgui users). There are two modes:

● no

Use two dedicated switch ports (for redundancy) and use them for all pool

Client LANs (as tagged VLANs onto these two ports).

● yes

Use two dedicated switch ports (for redundancy) for each pool Client LAN

(as untagged VLAN on both ports).

useTxToClan

If the switch group has fiber optic ports (SFP ports) these may be preferred as ports

to the corporate LAN transferring the pool Client-LAN data. To be able to customize

port type the parameter has two values:

● no

The port type will be fiber optic. Be sure to have matching modules for the

SFP ports modules for the SFP ports.

● yes

The port type will be Cat5 twisted pair, commonly abbreviated as TX

useTxUplink

If the switch group has fiber optic ports (SFP ports) these may be preferred as ports

to connect another switch group directly or to use them as uplink ports to the

FlexFrame integrated LAN switch. The parameter has two values:

● no

The port type for uplink will be fiber optic. Be sure to have matching

modules for the SFP ports.

● yes

The port type for uplink will be Cat5 twisted pair, commonly abbreviated as TX

uplinkPortCnt

Defines the count of ports aggregated to the uplink. The uplink channel consists of at

least two ports and may have a maximum of eight ports. The parameter value is the

count of ports.

Switch Administration Adding a Switch Group


--value <parameter value>

Defines the value of the parameter to be changed. For uplinkPortCnt use a

range between 2 and 8. For all other parameters use yes or 1 and no or 0 as value.

--dryrun


Examples

cn1:~ # ff_swgroup_adm.pl –op parameter

Common Network Configuration Parameters

Parameter Name Parameter Value

use ports to Client LAN no

Client LAN port per VLAN yes

use TX ports to Client LAN yes

use TX ports as uplink no

uplink port count 2

timezone Europe/Berlin

POSIX timezone CET

cn1:~ # ff_swgroup_adm.pl --op parameter --parameter useTxToClan --value

yes

Parameter successfully changed at LDAP.

9.8 Adding a Switch Group

To add an entire new switch group, use the ff_swgroup_adm.pl command. The

command adds data to the LDAP database and creates initial configuration which has to

be uploaded manually according to the corresponding "FlexFrame® HW Characteristics

Quickguide ("Hardwaresteckbrief")". Values appropriate to the current configuration

needed for the manual upload are displayed.

One uplink channel is configured which should be connected to an uplink

channel of another switch group or to switches outside the FlexFrame

environment. If more than one uplink channel is needed you can use

ff_swgroup_adm.pl --op add-uplink to add more channels. Afterwards

use ff_swgroup_adm.pl --op gen-config to create a new initial

configuration for upload purposes.

For details about uplink channels and supported topologies see the FlexFrame®

– Network Design and Configuration Guide.

Adding a Switch Group Switch Administration


Synopsis

ff_swgroup_adm.pl --op add --group <switch group id>

--type <list of switch types>

--name <switch group name>

--passwd <password> [--login <login name>]

[--host <ip host part>[,ip host part]]

[--uplinkportcnt <port count per channel>]

[--uplinkportmedia {tx|fx|10gb}]


[--vpcdomain <vpc domain id>]

[--dryrun]

Options

--op add

Adds a switch group and displays some information about processing steps.


Defines switch group to operate. Within this operation mode used to define the id of

the new switch group.

--type <list of switch types>

Defines the switch types of the switches the switch group will consist of. The types

are comma separated without any white spaces. The types must belong to the same switch family as described above. Call ff_swgroup_adm.pl --help to get a list

of supported switch types.

In case of types building a switch stack the first switch is intended to be the first

member of the switch stack and so on. The maximum number of switches per switch

stack is limited. Be aware that for more switches the stack cabling may be a

bottleneck.

In case of types not building a switch stack exactly two types have to be specified. It

is recommended to use switches with the same number of ports because ports are

needed always in same quantity from both switches.

For switch types providing expansion module slots, slots can immediately be

occupied by appending modules to the switch type separated with colon.

Switch Administration Adding a Switch Group


--name <switch group name>

Name string to be set as switch group node name.

--passwd <password>

Clear text password to be set at switch group.

--login <login name>

Login name to be set at switch group. Defaults to flexframe.

--host <ip host part>[,ip host part]

Host part to be used to build IP addresses for the control lan network. If this option is

omitted the script uses a free host number to calculate the IP address. Give

additional host parts if necessary.

--uplinkportcnt <port count for uplink channel>

Count of ports to be used for an uplink channel (default: 2).

--uplinkportmedia {tx|fx|10gb}

Media type of ports to be used for the uplink channel (default: depends on switch

type).


Defines the switch group the switch management interface should be connected to.

The information is necessary if the switches are managed out of band. Call

ff_swgroup_adm.pl --help to get a list of currently configured switch group IDs.

--vpcdomain <vpc domain id>

Defines a unique vPC domain within the network. The information is necessary if the

switches form a type of cluster, e.g. Nexus vPC or Brocade VCS.

Adding an Expansion Module Switch Administration


--dryrun

For test purpose you can perform the function without changing the LDAP Database

and updating the switchports.

cn1:~ # ff_swgroup_adm.pl --op add \

--type cat3750x-24t:2x10GbE,cat3750x-24t:2x10GbE \

--name swg6 --passwd passwort

If program is aborted by a failure remove leftovers by calling:

ff_swgroup_adm.pl --op rem --group 6

NOTICE: Ctrl-C will be ignored.

update LDAP

..................................................

........

New SwitchGroup 6 was added to LDAP.

Interconnect the Switch Group as noted below:

SwitchID Port

1 TE2 Uplink port of EtherChannel 1

2 TE2 Uplink port of EtherChannel 1

Upload of initial switch configuration has to be done manually.

Follow the instructions of the applicable Quick Start Hardware

FlexFrame for SAP document in the doc/hwinfo section of the Service CD

and use the following data:

Switch Group swg6

IP and Mask: 172.20.13.31 255.255.255.0

Config: scp://[email protected]//tftpboot/swg6.config

You can find a copy of the message above in

/var/log/FlexFrame/tmp/swgroup_adm-add-6/wiring_of_swgroup6_swg6.txt

9.9 Adding an Expansion Module

A switch may have slots where you can insert expansion modules to get more available

Ports. Use ff_swgroup_adm.pl --op add-module to add the expansion modules

ethernet ports to the FlexFrame configuration as described below.

Switch Administration Removing an Expansion Module


Synopsis

ff_swgroup_adm.pl --op add-module --group <switch group id>


--slot <slot id>

--module <module type>

Options

--op add-module

Adds an expansion module to a switch.


Defines the switch group to operate.


Defines the switch to operate.

--slot <slot id>

Defines the slot to be used.

--module <module type>

Defines the type of the expansion module to be added.

9.10 Removing an Expansion Module

Before you remove an expansion module physically from a switch use

ff_swgroup_adm.pl --op rem-module to remove the expansion modules ethernet

ports from the FlexFrame configuration as described below.

All ports of the expansion module must be unused.

Synopsis

ff_swgroup_adm.pl --op rem-module --group <switch group id>


--slot <slot id>

Options

--op rem-module

Removes an expansion module from a switch.

Removing a Switch Group Switch Administration



Defines the switch group to operate.


Defines the switch to operate.

--slot <slot id>

Defines the slot to operate.

9.11 Removing a Switch Group

To remove an entire switch group use the ff_swgroup_adm.pl command. The

command removes data from LDAP database. As a precaution the switch group may no

longer be in use by any FlexFrame component.

All ports of the switchgroup except the ports of one uplink must be unused.

Synopsis

ff_swgroup_adm.pl --op rem --group <switch group id> [--dryrun]

Options

--op rem

Removes named switch group.


Defines switch group to operate. Within this operation mode used to define the id of

the switch group to be removed.

--dryrun

For test purpose you can perform without changing the LDAP Database.

Example

cn1:~ # ff_swgroup_adm.pl --op rem --group 2

update LDAP.

........

Unless any errors are reported disconnect uplink ports

1/49, 2/49

and switch group is removed from FlexFrame environment.

Switch Administration Adding an Uplink to Switch Group


9.12 Adding an Uplink to Switch Group

Use ff_swgroup_adm.pl --op add-uplink to create an additional uplink at a

switch group. Use the command line arguments to specify options like port count and port

media.

Synopsis

ff_swgroup_adm.pl --op add-uplink --group <switch group id>


[--uplinkportmedia {tx|fx|10gb}]

[--uplinkchanneltype {UPLINK|CLANUPLINK}]

[--dryrun]

Options

--op add-uplink

Creates a new uplink link aggregate.


Defines the switch group to change.

--uplinkportcnt <port count per channel>

Count of ports to be used for an uplink channel (default: 2). The maximum number of

ports per channel is 8.

--uplinkportmedia {tx|fx|10gb}

Media type of ports to be used for the uplink channel (default: depends on switch

type).

--uplinkchanneltype {UPLINK|CLANUPLINK}

Defines the type of an uplink channel. UPLINK defines a channel with control LAN

and all pool VLANs. CLANUPLINK defines a channel with only the pool client LANs.

If omitted UPLINK is assumed.

--dryrun


Example

cn1:~ # ff_swgroup_adm.pl --op add-uplink --group 2 --uplinkportcnt 4

--uplinkportmedia fx

If program is aborted by Ctrl-C or a failure remove left overs

Extend an Uplink of Switch Group Switch Administration


by calling:

ff_swgroup_adm.pl --op rem-uplink --group 2 --channel 2

update LDAP

....



..........

New uplink with channel id 2 was created for switch group 2.

It was added to switch group and LDAP.

See below the configured uplink ports and connect them to the peer

switch.

SwitchID / Port

1 / 51 Uplink port of EtherChannel 2 to peer switch




Unless any errors are reported cable switch ports to use the uplink

channel.

Look at "/opt/FlexFrame/network/wiring_of_swgroup2__swg2.txt"


9.13 Extend an Uplink of Switch Group

Use ff_swgroup_adm.pl --op ext-uplink to extend an existing uplink at a switch

group. New ports are added until given port count is reached.

Synopsis

ff_swgroup_adm.pl --op ext-uplink --group <switch group id>

--channel <uplink channel id>


[--dryrun]

Options

--op ext-uplink

Expands an existing uplink link aggregate.


Switch Administration Delete an Uplink of Switch Group




Defines the uplink channel of switch group to changed.

--uplinkportcnt <port count per channel>

Count of ports to be used for an uplink channel (default: 2). The maximum number of

ports per channel is 8.

--dryrun



Example

cn1:~ # ff_swgroup_adm.pl --op ext-uplink --group 2 --channel 2 --

uplinkportcnt 8

update LDAP

.......



...................

Uplink with channel id 2 of switch group 2 was extended

up to 8 ports. Switch group configuration and LDAP are updated.

See below the new configured uplink ports and connect them to the peer

switch.

SwitchID / Port





Unless any errors are reported cable switch ports to use the new uplink

channel ports. Look at

"/opt/FlexFrame/network/wiring_of_swgroup2__swg2.txt"


Delete an Uplink of Switch Group Switch Administration


9.14 Delete an Uplink of Switch Group

To delete an existing uplink of a switch group use operation mode rem-uplink of

ff_swgroup_adm.pl. It will remove the channel with its link aggregate and all

associated ports at switch group.

Synopsis

ff_swgroup_adm.pl --op rem-uplink --group <switch group id>


[--dryrun]

Options

--op ext-uplink

Expands an existing uplink link aggregate.




Defines the uplink channel of switch group to removed.

--dryrun



Example

cn1:~ # ff_swgroup_adm.pl --op rem-uplink --group 2 --channel 2

update LDAP

........



........................

Uplink with channel id 2 removed at switch group 2.

It was removed from switch group and LDAP.

See below the freed uplink ports.

SwitchID / Port

1 / 51

2 / 51

3 / 49

3 / 50

Switch Administration Migrating Switches of a Switch Group


3 / 51

4 / 49

4 / 50

4 / 51

Unless any errors are reported switch ports of uplink channel are now

unused.

Look at "/opt/FlexFrame/network/wiring_of_swgroup2__swg2.txt"


9.15 Migrating Switches of a Switch Group

Switch migration means the replacement of a single or all switches of a switch group by

another switch exemplars. Switch migration may require a downtime. For details see the

vendors manuals.

In principle the processing is as follows.

1. Run ff_swgroup_adm.pl –-op migrate-sw with --dryrun option to check if

the migration is allowed and possible. For details see below.

2. Run ff_save_switch_config.pl to save the configuration for fall back.

3. Check the software version of the new switches. If the versions are not as requested

from the vendor perform an upgrade resp. downgrade.

4. Power off switches of the switch group as necessary.

5. Remove all network cables from the switches you want to replace.

6. Replace the switches.

7. Plug in all network cables into the new switches.

8. Power on the powered off switches. Ensure every switch has its intended ids

(member and/or cluster).

9. Use ff_swgroup_adm.pl –-op migrate-sw without --dryrun option to

accomplish the migration in LDAP.

10. Run ff_swgroup_adm.pl –-op gen-config to get the configuration of the

migrated switch group. Upload the configuration according to the corresponding

"FlexFrame® HW Characteristics Quickguide ("Hardwaresteckbrief")" or check for

necessary changes and perform the necessary configuration change.

The switch group is now ready for use or further configuration.

Adding a Switch Port Configuration Switch Administration


Synopsis

ff_swgroup_adm.pl --op migrate-sw --group <switch group id>

[--switch <switch id>] --type <switch type>

[--dryrun]

Options

--op migrate-sw

Migrates the type of switches of a switch group.




Defines the switch within the switch group to be migrated. If omitted all switches of

the switch group are migrated to the given switch type.

--type <switch type>

Defines the type the switch should migrate to. Call ff_swgroup_adm.pl without

any parameter to get a list of supported switch types. Migration is allowed according

to the migration table above.

For switch types providing expansion module slots, slots can immediately be

occupied by appending modules to the switch type separated with colon.

--dryrun


Example

cn1:/opt/FlexFrame/bin # ff_swgroup_adm.pl --op migrate-sw --group 1 –

switch 2 --type cat3750e-24td

9.16 Adding a Switch Port Configuration

Switch ports are typically directly configured by maintenance tools. But some issues like

configuring ports for gateways, backup or migration systems need a way to do this on a

per port basis. For this type of configuration a special peer type is used. The program

ff_swport_adm.pl is used to configure or remove this type of port configuration.

Synopsis

ff_swport_adm.pl --op add --port <swgroup:switch:port>

[ --10gbit]

Switch Administration Adding a Switch Port Configuration


--lan <pool:lan[:lan][,pool:lan[:lan]]>

[--native <pool:lan>]

[--desc <description>]

[--dryrun]

Options

--op add

Adds a switch port configuration and displays some information about processing

steps.

--port <swgroup:switch:port>

Defines the switch group, switch and port ID of the port to be used.

--10gbit

Defines the port number used in --port to be the number of a TenGigabitEthernet

port.

--lan <pool:lan[:lan][,pool:lan[:lan]]>

Defines the accessible VLANs. For better readability, a VLAN is specified with its

pool and LAN name. Use only client, server or storage as LAN names. For more

than one LAN per pool the LAN names may be added to the same pool statement.

The VLANs are not restricted to belong to the same pool. To directly add VLAN IDs

not used within any pool, use '#' as pool name and the VLAN ID(s) as LAN(s).

The accessible VLANs. For better readability, a VLAN is specified with its pool and

LAN name. Use only client, server or storage as LAN names. For more than one

LAN per pool the LAN names may be added to the same pool statement. The

VLANs are not restricted to belong to the same pool. To directly add VLAN IDs not used within any pool, use '#' as pool name and the VLAN ID(s) as LAN(s).

If only a single VLAN is configured this is accessible as native VLAN. This means

the data packet contains no VLAN tag. This is the behavior used by a standard

server network interface. For more than one LAN they are configured as tagged. To

define which of them should be used as native VLAN use the option --native.

Examples:

--lan poolA:client:server,poolB:client:server

--lan poolA:client,poolA:server,poolB:client:server

--lan poolA:storage,poolB:storage

--lan poolA:server

--lan '#:417:891'

--lan poolA:server,'#:417:891'

--lan 'poolA:server,#:417:891'

Removing a Switch Port Configuration Switch Administration


--native <pool:lan>

Use this option to define the native VLAN of the accessible VLANs defined with

option --lan. To directly add VLAN ID not used within any pool, use '#' as pool

name and the VLAN ID as LAN.

Examples:

--native poolA:server

--native '#:417'

--desc <description>

The description is added to configuration of switch port and the LDAP data of the

switch port configuration.

--dryrun



Example

cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op add --port 1:1:15

--lan ip1:storage:server,'#:4000' --native ip1:storage

Execution may take some minutes. If program is aborted

by Ctrl-C or a failure remove left overs by calling:

ff_swport_adm.pl --op rem --port 1:1:15



...........+

If not reported any error the port is configured and LDAP is updated

successfully.

9.17 Removing a Switch Port Configuration

To remove a switch port configuration that was previously configured by ff_swport_adm.pl or as external connectivity with the Management Tool, use the

ff_swport_adm.pl command. Other ports are configured with maintenance tools like

ff_an_adm.pl, ff_pool_adm.pl or ff_bx_cabinet_adm.pl.

The switch port configuration will be removed from the switch and the LDAP database.

Synopsis

ff_swport_adm.pl --op rem --port <swgroup:switch:port> [ -10gbit]

Switch Administration Displaying a Switch Port Configuration


Options

--op rem

Removes the configuration of a switch port and displays some information about

processing steps.



--10gbit


port.

--dryrun



Example

cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op rem --port 1:1:15

Execution may take some minutes. If program is aborted

by Ctrl-C or a failure remove left overs by calling:

ff_swport_adm.pl --op rem --port 1:1:15



.............+

If not reported any error the port is unconfigured and LDAP is updated

successfully.

9.18 Displaying a Switch Port Configuration

To display the configuration of a switch port as known by LDAP database in detail, use

the command ff_swport_adm.pl with operation mode list.

Synopsis

ff_swport_adm.pl --op list --port <swgroup:switch:port> [ -10gbit]

Options

--op list

Displaying the Complete Switch Port Configuration Switch Administration


Displays configuration of the switch port.



--10gbit


port.

Examples

cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op list --port 1:1:4

Switch Port Configuration of

1:1:4 (Switch Group : Switch : Port)

assigned VLAN IDs: 24,25,26

assigned VLAN Names: pool1:client,pool1:server,pool1:storage

native VLAN: 26

Port Peer Type: AN

Peer Node: rx300-1

Display of an unconfigured port:

ERROR: wrong switch port "1:1:8".

Port configuration unknown.

9.19 Displaying the Complete Switch Port Configuration

To display the complete configuration of all switch ports as known by LDAP database in detail, use the command ff_swport_adm.pl with operation mode list-all.

Synopsis

ff_swport_adm.pl --op list-all

Options

--op list-all

Displays configuration of all switch groups.

Switch Administration Displaying the Complete Switch Port Configuration


Examples

cn1:/opt/FlexFrame/bin # ff_swport_adm.pl --op list-all




assigned VLAN Names: pool1:client,pool1:storage,pool1:server

native VLAN: 1011

Port Peer Type: GW

Peer Node: sno1apl5p1




assigned VLAN Names: pool1:client,pool1:storage,pool1:server

native VLAN: 1011

Port Peer Type: AN

Peer Node: sno1apl2



assigned VLAN IDs:

1,90,91,92,93,94,95,1010,1011,1012,1013,1020,1021,1022,1030,1031,1032

assigned VLAN Names: -:default-

vlan,toni:storage,toni:client,toni:server,toni1:storage,toni1:client,ton

i1:server,

pool1:client,pool1:storage,pool1:server,-

:control,pool2:client,pool2:storage,pool2:server,pool3:client,pool3:stor

age,pool3:

server

Port Channel ID: 3

Port Peer Type: SWB



assigned VLAN IDs:

1,90,91,92,93,94,95,1010,1011,1012,1013,1020,1021,1022,1030,1031,1032

assigned VLAN Names: -:default-

vlan,toni:storage,toni:client,toni:server,toni1:storage,toni1:client,ton

i1:server,

Moving Device Connection to Core Switch Switch Administration


pool1:client,pool1:storage,pool1:server,-

:control,pool2:client,pool2:storage,pool2:server,pool3:client,pool3:stor

age,pool3:

server

Port Channel ID: 4

Port Peer Type: SWB

...

9.20 Moving Device Connection to Core Switch

Devices (e.g. application nodes or NAS storage etc) in FlexFrame are usually connected

to the FlexFrame internal (LAN) switch groups. When the configuration grows e.g. if a lot

of servers connected to several switch groups occupy the same NAS system it would be

more convenient if devices e.g. like this NAS system are directly connected to core

switches. Core switches are customer switches the switch groups have uplinks to. Core

switches are not under control of FlexFrame and not represented in LDAP. Devices di-

rectly connected to core switches are still represented in LDAP with a connection to an

imaginary switch group zero. For more detailed information to the core network concept

see also: FlexFrame - Network Design and Configuration Guide.

ff_move_to_core.pl supports the move of device connections to core switches. The

switch group ports occupied from the device to be moved are displayed and if the --doit

option is used the configuration of the affected ports is changed and an update of LDAP

is performed.

Synopsis

ff_move_to_core.pl --help

ff_move_to_core.pl --version

Options

--help

Display usage.

--version

Display script version.

9.20.1 Move Control Center to Core Switch

Synopsis

ff_move_to_core.pl --op cn [--doit]

Switch Administration Moving Device Connection to Core Switch


Options

--op cn

The switch group ports of both control nodes should be released. The affected ports

are displayed.

--doit

The configuration of the affected ports is changed and an update of LDAP is per-

formed.

Example

cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op cn

9.20.2 Move Client LAN to Core Switch

Synopsis

ff_move_to_core.pl --op clan [--doit]

Options

--op clan

The switch group ports of the client LAN should be released. The affected ports are

displayed

--doit


formed.

Example

cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op clan

9.20.3 Move NAS System to Core Switch

Synopsis

ff_move_to_core.pl --op nas --name <node name> [--doit]

Moving Device Connection to Core Switch Switch Administration


Options

--op nas

The switch group ports of a NAS system should be released. The affected ports are

displayed.

--name <node name>

The node name of the NAS system which should be moved.

--doit


formed.

Example

cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op nas --name filer1

9.20.4 Move Application Node to Core Switch

Synopsis

ff_move_to_core.pl --op an --name <node name> [--doit]

Options

--op an

The switch group ports of an application node should be released. The affected

ports are displayed.

--name <node name>

The node name of the application node which should be moved.

--doit


formed.

Example

cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op an --name rx300

9.20.5 Move ESX Server to Core Switch

Synopsis

ff_move_to_core.pl --op esx --name <node name> [--doit]

Switch Administration Moving Device Connection to Core Switch


Options

--op esx

The switch group ports of an application node should be released. The affected

ports are displayed.

--name <node name>

The node name of the application node which should be moved.

--doit


formed.

Example

cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op esx --name rx300

9.20.6 Move BX Cabinet to Core Switch

Synopsis

ff_move_to_core.pl --op bx --name <cabinet name> [--doit]

Options

--op bx

The switch group ports of a BX cabinet should be released. The affected ports are

displayed.


The cabinet name of a BX chassis which should be moved.

--doit


formed.

Example

cn1:/opt/FlexFrame/bin # ff_move_to_core.pl --op bx --name bx61

Moving Device Connection to Core Switch SAP System Handling


10 SAP System Handling

This chapter describes the management of SAP System IDs (so-called SIDs) and their

respective instances within the FlexFrame environment. It further describes how to clone

SIDs as well as their instances into another pool than the one they were installed to.

The tools described below only maintain the LDAP database entries, rather than adding

or removing the data and binaries of the respective SAP systems. These steps need to

be performed manually.

Listing, adding, removing and cloning the above entities in the LDAP server is supported

by two tools, ff_sid_adm.pl and ff_clone_sid.pl. Both scripts will take care of

keeping the data accessed by the operating system's naming service mechanism in sync

with the FlexFrame internal configuration data, both of which reside in the LDAP server.

There is also a Graphical User Interface available which supports the

handling of SIDs in an easy way.

This data should not be manipulated manually.

Preventing port conflicts

Please take into account that depending on the instance number ports are

reserved for the SAP application. That could mean that you conflict with other

non-SAP applications on your nodes (e.g. SAP's ICM uses ports 80nn which

maybe conflicts with an application which uses port 8081). Please refer to

SDN document “TCP/IP Ports Used by SAP Applications” to prevent conflicts.

If a new SAP system is added by ff_sid_adm.pl and the modification of the LDAP

database fail, ff_sid_adm.pl always tries to remove all changes in LDAP which are

made until the failure. This rollback may cause error messages because the rollback

operation does not exactly know where the error occurred (it cannot recognize the failed

command). Those messages can be ignored.

The script ff_sid_adm.pl tries to generate files to support diagnostics. Those files are:

/var/log/FlexFrame/tmp/ff_sid_adm.pl/ldap.<pool>.<SID>.ldif

/ var/log/FlexFrame/tmp/ff_sid_adm.pl/ldap.error.<pool>.<SID>

/ var/log/FlexFrame/tmp/ff_sid_adm.pl//ldap.log.pool>.<SID>

/ var/log/FlexFrame/tmp/ff_sid_adm.pl/ldap.rollback.<pool>.<SID>

In addition to the described SAP System handling in this chapter here there is another

possibility to handle SAP systems using LVM (Landscape Virtualization Management).

SAP System Handling Listing SAP SIDs and Instances


For more information see manual "Installation and Configuration LVM 1.0" and additional

documentation from SAP.

10.1 Listing SAP SIDs and Instances

Synopsis

ff_sid_adm.pl –-op list-sids [ –-pool <pool name> ]

ff_sid_adm.pl --op list-ids [ --pool <pool name> --sid <SID> ]

ff_sid_adm.pl –-op list-all [ –-pool <pool name> --sid <SID>]

Options

--op list-sids

The SIDs of the specified pool/all pools are displayed with information on SAP

version, database type and database version

--op list-ids

The instance numbers of the specified SID are displayed related to the given

pool/SID or all pools/all SIDs.

--op list-all

Shows extended information about a specific SID (instance types, instances

numbers, used ip@, used hostnames) in a specific pool.

--pool <pool name>

Specifies the FlexFrame pool to which the operation should be applied.

--sid <SAP system id>

The instance numbers of the specified SID are displayed related to the given pool.

Examples

%> ff_sid_adm.pl –-op list-ids –-pool Pan –-sid sht

Configuration Data of SID – instance number and types: SHT (Pool: Pan)

05 scs

DB0 db

07 ci

Listing SAP SIDs and Instances SAP System Handling


%> ff_sid_adm.pl –-op list-sids –-pool Pan

List of SIDs in pool: Pan

O01 SAP-7.1 Oracle-10

S02 SAP 7.0 MaxDB-76

%> ff_sid_adm.pl –-op list-all –-pool Pan –-sid SHT

Configuration data of SID: SHT (Pool: Pan)

SID global Data:

SAP version specified: SAP-7.0

Database specified: Oracle-10

Instance Data:

Instance type: ci

Instance number: 00

Client-Lan Host-Ip: nnn.nn.nn.nnn

Client-Lan Hostname: cisht

Server-Lan Host-IP: nnn.nn.mm.nnn

Serber-Lan Hostname: cisht-se

SAP System Handling Updating System Configuration Files


10.2 Updating System Configuration Files

Synopsis

ff_sid_adm.pl --op db2adm --pool <pool name> --sid <SAP system id>

Options

--op db2adm

Updates the system configuration files with DB2 specific data (host name, service

names) for a specific SID.

--pool <pool name>



Specifies the SID being used.

10.3 Adding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)

Synopsis

ff_sid_adm.pl –-op add –-pool <pool name> --sid <SAP system id>

--sapversion {4.6 |6.20 |6.40 |7.0 |7.1 |7.2 |7.3 |7.4 |

7.3-HANA |7.4-HANA | 7.3-NLS | 7.4-NLS}

--db {ORACLE{9|10|11}|

MAXDB{76|77|78|79}|

DB2V{91|95|97|10|105}|

SYBV{157|160} | HANADB-REF}|

SIQ-V16.0

[:<db_loghost_client_ip>|*]:{<db_loghost_server_ip>|*}

--lc {MAXDB76|MAXDB77|MAXDB78|MAXDB79} {<lc_loghost_ip>|*}

--group <groupname1>:<gidnumber1>,<groupname2>:<gidnumber2>,...

--user <username1>:<uidnumber1>,<username2>:<uidnumber2>,...

:{<db_loghost_ip>|*}

--sap {ci|pai|app|jc|j|scs|ascs|ers}:<SYSNR>:

:{<loghost_client_ip>|*}

:{<loghost_server_ip>|*}

[--os <instance_type>:<os>,<instance_type>:<os>,...]

Adding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)SAP System Handling


[--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]

[--db2srv sapdb2<SID>:<port>,DB2_db2<sid>:<port>,

DB2_db2<sid>_1:<port>, DB2_db2<sid>_2:<port>,

DB2_db2<sid>_END:<port>]

[--sapdata <nas system>:/vol/<volume path>]

[--saplog <nas system>:/vol/<volume path>]

ff_sid_adm.pl –-op mod –-pool <pool_name> --sid <SAP system id>

[--os <instance_type>:<os>,<instance_type>:<os>,...]

ff_sid_adm.pl –-op mod –-pool <pool name> --sid <SAP system id>

[--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]

ff_sid_adm.pl –-op del –-pool <pool name> --sid <SAP system id>

[--sysnr <SYSNR>]

Options

--op add

Determines the add operation.

--op mod

Determines the mod operation. This option is only used to modify OS specifications

and exchanges IP adresses of specific SID instances.

--op { del | rem }

Determines the delete operation.

--pool <pool name>




--sapversion {4.6|6.20|6.40|7.0|7.1|7.2|7.3|7.4|

7.6 | 7.7 | 7.9 |

7.3-HANA|7.4-HANA }

Specifies the SAP basis version being used. '7.3-HANA' and '7.4-HANA' each

specify a SAP NetWeaver installation which does not have an own classic database

server. The installation uses an existing HANA database (service 'hdb') and the

SAP System HandlingAdding/Removing/Modifying SAP SIDs and Instances (Classic SAP Services)


option 'db' specifies the reference to that HANA database.

‘7.6, ‘7.7’ and ‘7.9’ are only used with LiveCache. The versions refers to the MaxDB

versions used.

--db {ORACLE{10|11} |

MAXDB{76|77|78|79} |

DB2V{95|97|10|105} |

SYBV{157|160} |

SIQ-V16.0 |

HANADB-REF}

Specifies the database type as well as the respective version being used. The

specification of the database type is not case-sensitive. Please take care of the

restrictions given by SAP (e.g. SAP 7.3 only allows at least Oracle11, MaxDB78 or

DB2V97). SIQ-V16.0 is only used with sapversion 7.3-NLS and 7.4-NLS.

{<db_loghost_server_ip>|*}}

The logical host name is used for the database (generated automatically: db<sid>-

se) as well as the IP address for that host name. Use an asterisk if you want it to be

chosen automatically. All the entries need to be specified in a colon separated for-

mat.

You can omit the network part of the IP, e.g. 10.10.12, and specify only the last tupel

of the IP, e.g. 123.

[ {<db_loghost_client_ip>|*}} ]

The logical host name is used for the database tools (generated automatically:

db<sid>) as well as the IP address for that host name. Use an asterisk if you want it

to be chosen automatically. All the entries need to be specified in a colon separated

format. This parameter is optional.



[ { hana database reference host – client} ]

The virtual client lan hostname of the HANA database (service ‘hdb’).

{ hana database reference host – server}

The virtual server lan hostname of the HANA database (service ‘hdb’)

--lc {MAXDB76|MAXDB77|MAXDB78|MAXDB79}


specification of the database type is not case-sensitive.

{<lc_loghost_ip>|*}}



The logical host name is used for the database (generated automatically: lc<sid>-



mat.





User and group enable specially selected user numbers and group numbers to be

assigned to SAP users and SAP groups respectively. In this case a check is made to

see whether the user or group has already been defined for the DB system involved.

A user or group is created only if they do not already exist. For example, a group dba which already exists cannot be assigned a group number which deviates from

the default value.

--sap {ci|pai|app|jc|j|scs|ascs|ers}:<SYSNR>:

{<loghost_client_ip>|*}

{<loghost_server_ip>|*}

Specifies an SAP instance (optionally multiple of those) through its type (ci, pai,

app, jc, j, scs, ascs, ers), its SAP system number, the logical host name

(generated automatically: <type><sid>) in the client network, the respective IP

address, the logical host name (generated automatically: <type><sid>-se) in the

server network and the respective IP address. Again, the IP addresses can be

replaced with asterisks in order to have them chosen automatically.

With support of SAP 7.4 the CI specification is replaced by PAI specification. With

SAP 7.4 we do not allow specification of CI furthermore.

ERS is only supported with --sapversion 7.0 up or higher. For Enqueue Repli-

cated Servers (ERS) the asterisk should be used because we do not need an IP

address from the client and server LAN (given IP addresses will be ignored).

All the entries need to be specified in a colon separated format.

With SAP 7.1 up SAP systems do not know instance type jc for JAVA systems.

Therefore the hostname used changed from jc<sid> to j<sysnr><sid> and from

jc<sid>-se to j<sysnr><sid>-se.

ff_sid_adm.pl still requires the type jc used in option —-sap to make a distinc-

tion between central instance and application instance. Please take into account the

different hostname specifications depending on SAP version used.

--sysnr <SYSNR>

Removes a specific SAP instance instead of the entire system (SID).



--os <instance_type>:<os>,<instance_type>:<os>,...]

Specifies the OS type for the given SID and/or their instances.

instance_type::= {default|ci|app|jc|j|scs|ascs|ers}

os::= {SLES-11.x86_64}

In combination with the –-add option, default:<os> sets the operating system of

the SID itself and all their instances which no own operating system is assigned to.

In combination with the –-mod option, default:<os> sets the operating system of

the SID only. The specifications of their instances are not changed.

Examples:

default:SLES-11.x86_64 - all instances are set to SLES-11.x86_64

The consistence of the given instance type/OS combinations is not checked

by FlexFrame. For a list of allowed combinations see SAP note 1067221.

--ips <old_ip>:<new_ip>,<old_ip>:<new_ip>,...]

Allows to exchange the IP addresses of specific SID instances. This option is only

used with —-op mod. You have to specify the full IP you want to exchange.

ff_sid_adm.pl searches for the specific instances within a SID and exchanges all

corresponding entries in LDAP concerned by that request.

Please pay attention that you make critical changes within your configuration.

We strongly recommend to save a backup of the LDAP database before changing

any IP addresses.

--db2srv sapdb2<SID>:<port>,DB2_db2<sid>:<port>,

DB2_db2<sid>_1:<port>,DB2_db2<sid>_2:<port>,

DB2_db2<sid>_END:<port>

This option is only used for DB2, otherwise it is ignored. You can specify a list of

services you need. The services are written to LDAP and the pool specific /etc/services.

--sapdata <nas system>:/vol/<volume path>

Allows to request an implicit call of ff_sid_mnt_adm.pl to relocate location to store

sapdata. It is expected that the requested volume already exists. The option is

valid only with services using a database. <nas system> is the NAS filer node name

ending with ‘-st’.

This option is not used with installations using a reference to a HANA database.

--saplog <nas system>:/vol/<volume path>


saplog. It is expected that the requested volume already exists. The option is



valid only with services using a database. <nas system> is the NAS filer node name


This option is not used with installations using a reference to a HANA database.

Examples

Adding a SID with one Central Instance:

control1:~ # ff_sid_adm.pl –-op add –-sid SHT –-pool otto --sapversion 6.40

--db ORACLE9:192.168.1.1 --sap ci:00:\*:\*

Adding an instance to an existing SAP System:

control1:~ # ff_sid_adm.pl –-op add –-sid SHT –-pool otto --sapversion 6.40

--sap app:01:\*:\*

Adding a SID for LiveCache:

control1:~ # ff_sid_adm.pl --op add --pool test --sid LCA --sapversion 7.0 --lc MAXDB76:\*Adding a

SID with ERS support:

control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid S04 --sapversion 7.0

--db ORACLE10:dbs04-se:10.10.12.100 --sap ers:12:\*:\*

--sap ci:13:10.10.11.102:10.10.12.102

Adding a SID with operation system:

control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid OS1 --sapversion 7.0

--db ORACLE10:10.10.12.159

--sap ci:57:10.10.10.157:10.10.12.157

--sap ascs:55:10.10.10.155:10.10.12.155

--os default:SLES-11.x86_64,ascs:SLES-11.x86_64

Adding a SID with DB2 services:

control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid LB5 --sapversion 7.4

--db DB2V91:10.10.12.159

--sap pai:57:10.10.10.157:10.10.12.157

--sap ascs:55:10.10.10.155:10.10.12.155

--db2srv sapdb2LB5:60000,DB2_db2lb5:60001,DB2_db2lb5_1:60002,DB2_db2lb5_2:60003,

DB2_db2lb5_END:60004

Adding a SID (SID specific volumes for sapdata/saplog):



control1:~ # ff_sid_adm.pl --op add --pool pool1 –-sid JA0 --sapversion 7.0

--db MAXDB77:158

--sap jc:57:157

--sap j:58:156

--sap ascs:55:155

--sapdata filer-1-st:/vol/JA0_data

--saplog filer-1-st:/vol/JA0_logs

Adding a SID (old syntax for hosts):


--db MAXDB77:10.10.12.158

--sap jc:57:10.10.10.157:10.10.12.157

--sap j:58:10.10.10.156:10.10.12.156

--sap ascs:55:10.10.10.155:10.10.12.155

Adding a SID with old syntax (SAP 7.1, JAVA):


--db ORACLE10:10.10.12.158

--sap jc:57:10.10.10.157:10.10.12.157

--sap j:58:10.10.10.156:10.10.12.156

--sap ascs:55:10.10.10.155:10.10.12.155

Adding a SID with reference to HANA database:

control1:~ # ff_sid_adm.pl -–op add –-pool pool1 -–sid HRF –-sapversion 7.4-HANA

--db HANAREF-DB:hdb01ha1-001:hdb01ha1-001-se

--sap ascs:20:120:120

--sap pai:21:121:121

Removing an entire SID (including its instances):

control1:~ # ff_sid_adm.pl –-op del –-sid SHT –-pool otto

Removing a single instance of a SID:

control1:~ # ff_sid_adm.pl –-op del –-sid SHT –-pool otto –-sysnr 01

Removing SAP SIDs and Instances SAP System Handling


Modifying IP adresses:

control1:~ # ff_sid_adm.pl –-op mod –-sid SHT –-pool otto

-–ips 10.10.10.123:10.10.10.132,10.10.12.123:10.10.12.132

Modifying OS version of the SID entry:

control1:~ # ff_sid_adm.pl –-op mod –-sid SHT –-pool otto

-–os "default:SLES-11.x86_64"

10.4 Removing SAP SIDs and Instances

Synopsis


ff_sid_adm.pl –-op [ del | rem ] –-pool <pool name> \

--sid <SAP system id> [--sysnr <SYSNR>]

Options

--op { del | rem }


--pool <pool name>




--sysnr <SYSNR>


Examples

Removing an entire SID (including all instances):

%> ff_sid_adm.pl –-op del –-sid SHT –--pool otto

Removing an specifc instance:

%> ff_sid_adm.pl –-op del –-sid SHT –--pool otto –-sysnr 01

SAP System Handling Adding/Removing SAP SIDs (add-on services)


10.5 Adding/Removing SAP SIDs (add-on services)

10.5.1 BOBJ – Business Objects

Synopsis

ff_sid_adm.pl –-op add –-pool <pool name> --sid <SAP system id> \

--sapversion {3.1 | 3.2| 4.0 | 4.1} \

--bobj <client lan hostip> --os <spec>


Options

--op add | del

Determines the operation, add or delete.

--pool <pool name>




--sapversion {3.1|3.2|4.0|4.1}

Specifies the SAP basis version being used.

--bobj <client lan hostip>

Specifies the hostip in the client lan for BOBJ service.


--user <sid>adm:<uidnumber>

user and group enable specially selected user numbers and group numbers to be


see whether the user or group has already been defined for the SAP system

involved. A user or group is created only if they do not already exist. For example, a

group sapsys which already exists cannot be assigned a group number which

deviates from the default value.

--bobj <client lan hostip>

Adding/Removing SAP SIDs (add-on services) SAP System Handling


The IP address for the clien lan host name (generated internally: bobj<sid>). Use

an asterisk if you want it to be chosen automatically.

You can omit the network part of the IP, eg. 10.10.12, and specify only the last

tupel of the IP, e.g. 123.



instance_type::= {default|bobj}

os::= {SLES-11.x86_64}

In combination with the –-add option, default:<os> sets the operating system of the

SID itself and all their instances which no own operating system is assigned to.

In combination with the –-mod option, default:<os> sets the operating system of the

SID only. The specifications of their instances are not changed.



Examples

Adding a SID with BOBJ service:

control1:~ # ff_sid_adm.pl –-op add –-sid bob –--pool otto --sapversion 3.2 \

––bobj 160

Removing a BOBJ service:

control1:~ # ff_sid_adm.pl –-op del –-sid bob –--pool otto

10.5.2 Content Server (CMS)

Adding a CMS service at first time means that you need to specify all the instances

required by CMS (database and cms client spec).

CMS service consists of

a database instance

a client instance

Synopsis


--db MaxDB<version>:{<db_loghost_ip>|\*} \

--cms <client lan_ip>|\*} \



--sapversion 6.50 | 6.40 [--os <spec> ] \

[ --sapdata <nas_system>:/>volume path> ]

[ --saplog <nas_system>:/>volume path> ]


Options

--op add | del

Determines the operation, add or delete.

--pool <pool name>




--sapversion {6.50 | 6.40}

Specifies the SAP basis version being used. Version 6.40 is not supported further by

SAP.

--db MaxDb76|77|78:79:{<db loghost ip> | \*}

Specifies the database version being used and the server lan host IP

--cms <client lan hostip>

Specifies the hostip in the client lan for BOBJ service.

{<db_loghost_ip>|*}}

The logical host name (generated internally: db<sid>-se) is used for the database as

well as the IP address for that host name. Use an asterisk if you want it to be chosen

automatically. All the entries need to be specified in a colon separated format.

You can omit the network part of the IP, e.g. 10.10.12, and specify only the last


--cms <client lan hostip>

The IP address for that host name (generated internally: cms<sid>). Use an asterisk

if you want it to be chosen automatically.

You can omit the network part of the IP, e.g. 10.10.12, and specify only the last





--user <username1>:<uidnumber1>,<username2:uidnumber2>,…



see whether the user or group has already been defined for the SAP system

involved. A user or group is created only if they do not already exist. For example, a

group sapsys which already exists cannot be assigned a group number which

deviates from the default value.



instance_type::= {default|bobj}

os::= {SLES-11.x86_64}

In combination with the –-add option, default:<os> sets the operating system

of the SID itself and all their instances which no own operating system is assigned

to.



sapdata. It is expected that the requested volume already exists. The option is valid

only with services using a database. <nas system> is the NAS filer node name




saplog. It is expected that the requested volume already exists. The option is valid



In combination with the –-mod option, default:<os> sets the operating system

of the SID only. The specifications of their instances are not changed.



Examples

Adding a SID with CMS service:

control1:~ # ff_sid_adm.pl –-op add –-sid cmx –--pool otto --sapversion 6.40 \

--db MaxDB76:170 \



––cms 170 ]



Adding a CMS client instance:

control1:~ # ff_sid_adm.pl –-op add –-sid cmx --pool otto --sapversion \

6.40 ––cms 170 ]

Removing a CMS Service:

control1:~ # ff_sid_adm.pl –-op rem –-sid cmx –-pool otto

Removing the CMS client instance:

control1:~ # ff_sid_adm.pl –-op rem –-sid cmx –-pool otto –sysnr CMS

10.5.3 MDM – Master Data Management

Adding a MDM service for the first time means that you need at least to specify the

database instance.

MDM service contains:

a database instance

a number of services of type 'mds'

a number of services of type 'mdss'

a number of services of type 'mdis'

Synopsis


--db {ORACLE{10|11}|

MAXDB{77|78|79}|

SYBV{157|160}

DB2V{95|97|10|105}}:{<db_loghost_ip>|*}

--mdm mds:<nr>:{<client lan_ip>|\*}>:{<server

lan_ip>|\*} \

--mdm mdss:<nr>:{<client lan_ip>|\*}>:{<server

lan_ip>|\*} \

--mdm mdis:<nr>:{<client lan_ip>|\*}>:{<server

lan_ip>|\*} \

--sapversion 7.1 [--os <spec> ] \

[ --sapdata <nas_system>:/<volume path> ] \

[ --saplog <nas_system>:/<volume path> ] \

ff_sid_adm.pl –-op del –-pool <pool name> \




--op add


--op mod



--op del


--pool <pool name>




--sapversion {7.1}


--db {ORACLE{10|11}|MAXDB{77|78|79}|DB2V{95|97|10|105}}


specification of the database type is not case-sensitive.






A user or group is created only if they do not already exist. For example, a group

dba which already exists cannot be assigned a group number which deviates from

the default value.

{<db_loghost_ip>|*}}

The logical host name is used for the database (generated automatically: db<sid_-



mat.

You can omit the network part of the IP, e.g. 10.10.12., and specify only the last




--mdm

{mds|mdss|mdis}:<SYSNR>:{<loghost_client_ip>|*}:{<loghost_server_i

p>|*}

Specifies an SAP instance (optionally multiple of those) through its type, its SAP

system number, the client network IP address, the the server network IP address.

Again, the IP addresses can be replaced with asterisks in order to have them cho-

sen automatically.

All the entries need to be specified in a colon separated format.

FlexFrame expects that the syntax of loghost-client follows the rule <type>sid and

loghost-server follows <type>sid-se.

--sysnr <SYSNR>




instance_type::= {default|mds|mdss|mdis}

os::= {SLES-11.x86_64}

In combination with the –-add option, default: <os> sets the operating system of the






sapdata. It is expected that the requested volume already exists. The option is valid





saplog. It is expected that the requested volume already exists. The option is valid








Allows to exchange the IP addresses of specific SID instances. This option is only used with —-op mod. You have to specify the full IP you want to exchange.



Please pay attention that you make critical changes within your configuration. So we

strongly recommend you to take a backup of LDAP database before changing IP

addresses.

Examples

Adding a SID with MDM services:

control1:~ # ff_sid_adm.pl –-op add –-sid mdm –--pool otto --sapversion 7.1 \

--db MaxDB76:170

––mdm mds:01:171:171

––mdm mdss:02:172:172

––mdm mdis:03:173:173

Adding an instance of type mdss to an existing MDM SID:

control1:~ # ff_sid_adm.pl –-op add –-sid mdm –--pool otto --sapversion 7.1 \

––mdm mdss:04:174:174

Removing an specific instance of a MDM SID:

control1:~ # ff_sid_adm.pl –-op rem –-sid mdm –--pool otto –sysnr 02

Removing a SID with MDM services:

control1:~ # ff_sid_adm.pl –-op rem –-sid mdm –--pool otto

10.5.4 SMD – Solution Manager Diagnostics

Current SAP releases using SAP Solution Manager and SMD agents. We

distinct the type of agentsA service based agent as known from SAP 7.1 up.

Each services (e.g. CI) needs its own agent

A host based agend as introduced with SMD 7.3 SP02. For each physical host

one agent is defined

The first type should be found in older installations. It is recommended to specify a SID

with multiple SMD instances, e.g. one SMD SID with one instance per monitored service

within a NetWeaver SID.

The second type should be the choice in newer configurations. The property 'host based'

has to be enabled at customer side. If ‘host based’ is enabled you can only define a SMD



SID with one instance assigned to the monitored physical host. Otherwise the SMD agent

is configured like an service based type.

A host based SMD agent is assigned to a physical host. If you are moving an Application

Node from one pool to another, or if you are using pool independent spares, you have to

take care that your SMD configuration is consistent. There is no possibility to a complete

automatism.

The SMD SID should be specified before starting NetWeaver installation because during

installation procedure you are asked for SMD agent SID.

Synopsis


--smd { <nr>:<client lan_monitored host> |

<nr>:<application node name> \

--sapversion [ 7.1 | 7.3-SRV | 7.3-HOST ]

[--os <spec> ]

ff_sid_adm.pl --op add --pool <pool name> --sid <SAP system id> \

--smd <nr>:<db server_lan monitored host> \

--sapversion [ 7.1 | 7.3-SRV | 7.3-HOST ] [--os

<spec> ]


--smd <nr>:<client lan application node> \

--sapversion [ 7.3S-SRV | 7.3-HOST ] [--os <spec>

]

ff_sid_adm.pl --op mod --pool <pool name> --sid <SAP system id> \

--smd <nr>:<client lan application node>


--op add




--op del


--op mod

Allows to modify physical hostname referenced with host-based

SMD spec. For modification use option ‘—smd’ as used with

opcode ‘add’

--pool <pool name>




--sapversion {7.1}









the default value.

--smd <SYSNR>:[ <client lan name – monitored host> |

<db server lan name - monitored host> |

<client lan application node>

Specifies the name of the host which should be monitored. The name of the host

depends on the SAP service type. With SMD 7.1 and 7.3 we have a service-based

monitoring. The following services types can be monitored

● app – app<nr><sid>

● j – j<nr><sid>

● ci – ci<sid>

● pai - pai<sid>

● jc – jc<sid> (less than SAP 7.1), j<nr><sid> (SAP > 7.1)

● ascs – ascs<SID>

● db - db<sid>-se



● wd – wd<nr><sid>

Parameteroption ‘SYSNR’ specifies the instance number of the SMD agent

installation..

--sysnr <SYSNR>

Removes a specific SAP instance instead of the entire SID .



instance_type::= {default|smd}

os::= {SLES-11.x86_64}







Examples

Adding a SID with SMD services (service-based monitoring):

control1:~ # ff_sid_adm.pl –-op add –-sid SMD –-pool otto \

--sapversion 7.1 \

––smd 01:j12abc smd 02:cixyz

Adding an instance of type mdss to an existing SMD SID:

control1:~ # ff_sid_adm.pl –-op add –-sid SMD –-pool otto \

--sapversion 7.1 \

––smd 04:app12xyz

Adding a SID for host-based monitoring

control1:~ # ff_sid_adm.pl –-op add –-sid SMD –--pool otto \

--sapversion 7.3S-HOST \

––smd 97:appnode1

Removing an specific instance of a SMD SID:



control1:~ # ff_sid_adm.pl –-op rem –-sid mdm –-pool otto –sysnr 02

Removing a SID with SMD services:

control1:~ # ff_sid_adm.pl –-op rem –-sid mdm –-pool otto

10.5.5 TREX (Search and Classification Service)

Synopsis


--trx <nr>:<client lan_ip>>:<server

lan_ip> \

--sapversion 7.1 [--os <spec> ]

ff_sid_adm.pl –-op del –-pool <pool_name> --sid <SAP_system_id>

--op add


--op del


--op mod



--pool <pool name>




--sapversion {7.1}











the default value.

--smd <SYSNR>:<client lan hostname – monitored host>>

The logical host name is used for the database (generated automatically:

trx<nr><sid>, trx<nr><sid-se>) as well as the IP address for that host

name. Use an asterisk if you want it to be chosen automatically. All the entries need

to be specified in a colon separated format.

You can omit the network part of the IP, e.g. 10.10.12. and specify only the last


--sysnr <SYSNR>




instance_type::= {default|trx}

os::= {SLES-11.x86_64}









used with —-op mod. You have to specify the full IP you want to exchange.





addresses.

Examples

Adding a SID with TRX services:

control1:~ # ff_sid_adm.pl –-op add –-sid TR1 –--pool otto --sapversion 7.1 \

––trx 01:201:201



––trx 02:200:200

Adding an instance of type mdss to an existing TRX SID:

control1:~ # ff_sid_adm.pl –-op add –-sid TR1 –--pool otto --sapversion 7.1 \

––smd 04:202:202

Removing an specific instance of a TRX SID:

control1:~ # ff_sid_adm.pl –-op rem –-sid tr1 –--pool otto –sysnr 02

Removing a SID with TRX services:

control1:~ # ff_sid_adm.pl –-op rem –-sid tr1 –--pool otto

10.5.6 WebDispatcher

Synopsis


--wd <nr>:<client lan_ip>:<server lan_ip>> \

--sapversion { 7.1 | 7.2 | 7.3 | 7.4 } [--os <spec>]

ff_sid_adm.pl --op del --pool <pool_name> --sid <SAP_system_id>

--op add


--op del


--op mod

Determines the modify operation. This option is only used to modify OS

specifications and exchanges IP adresses of specific SID instances.

--pool <pool name>




--sapversion {7.1| 7.2 | 7.3 | 7.4 }










dba which already exists cannot be assigned to a group number which deviates from

the default value.

--wd <SYSNR>:<client_lan_ip>:<server lan_ip>

The logical host name is used for the instance (generated automatically:

wd<nr><sid>) as well as the IP address for that host name. Use an asterisk if you

want it to be chosen automatically. All the entries need to be specified in a colon

separated format.

You can omit the network part of the IP, e.g. 10.10.12. and specify only the last tupel


--sysnr <SYSNR>




instance_type::= {default|wd}

os::= {SLES-11.x86_64}

In combination with the --add option, default:<os> sets the operating system of


In combination with the --mod option, default:<os> sets the operating system of




used with --op mod. You have to specify the full IP you want to exchange.





addresses.



Examples

Adding a SID with WD services:

control1:~ # ff_sid_adm.pl --op add --sid WD1 --pool otto \

--sapversion 7.2 \

--wd 01:200:200 --wd 02:201:201

Removing a specific instance of a WebDispatcher SID:

control1:~ # ff_sid_adm.pl --op rem --sid wd1 --pool otto -sysnr 02

Removing a SID with WebDispatcher services:

control1:~ # ff_sid_adm.pl --op rem --sid wd1 --pool otto

10.5.7 HANA database (service hdb)

We support service ‘hdb’ HANA database in Single Node Configurations. Please refer to

the corresponding hardware restrictions specified by SAP.

At now we also support HANA System Replication feature in a Single Node Environment.

A HANA database requires specific volumes for shared administration data and the

database data- and logfiles. Please refer to the preconditions specified by SAP

concerning the file system structure.

FlexFrame Orchestrator only supports HANA single node configurations.

The specification of a hdb service requires

The specification of the SID using ff_sid_adm.pl

Availability of specific filer volumes used for HANA

o SID specific volume for shared data (/hana/shared)

o Instance- and Node-specifc volumes for data (/hana/data ) and logs,

(/hana/log). Each instance and node can use own volumes for data and

logs.

All volumes are assigned to a specific HANA database service and volumes

should not be used over multiple pools

Specifying HANA volumes for SID using ff_sid_mnt_adm.pl

Creation of file system structure using ff_setup_sid_folder.sh. The script expects

that all HANA volumes (shared/data/logs) are assigned to SID. Otherwise the

scripts aborts.



Please refer also to SAP Installation how to specify and install SAP HANA Database

service.

Please make sure to have an installation of the (maybe) required FlexFrame Orchestrator

software licenses.

With HANA System Replication you have

A SID up to 9 times within a pool

Each replication instance has its own volumes for shared/data/log

The instance number has to be equal in all replicas

The user-/group environment has to be equal

A replication instance is assigned to a specific poolgroup in which it can run

The specification in FlexFrame is quite the same as HANA without HSR.

Synopsis


--db HANA | R-HANA\

--hdb <SYSNR>:<client lan_ip>:<server lan \

ip>:<nodeid> \

--hdbtype { OLTP | OLAP } \

--sapversion { 1.0 | R-1.0 } [--os <spec> ]

[ --replid < replication id> ]

[ --runingroup <assigned poolgroup for operating> ]

ff_sid_adm.pl --op del --pool <pool_name> --sid <SAP_system_id> \

[ --replid <replication id> ]

--op add


--op { del | rem }


--op mod


specifications and exchanges IP adresses of specific SID instances or modifies/add

properties of HANA System Replication.



--pool <pool name>




--sapversion {1.0 | R-1.0 }

Specifies the Flexrame internally used version

--db { HANA | R-HANA }

Specifies the type of HANA configuration. ‘HANA’ means the non-HRS configuration

as known from FFO 1.0.

--hdb <SYSNR>:<client_lan_ip>:<server lan ip>:<nodeid>

The logical host name is used for the database (generated automatically:

hdb<nr><sid>-<nodeid>) as well as the IP address for that host name. Use an

asterisk if you want it to be chosen automatically. All the entries need to be specified

in a colon separated format.

You can omit the network part of the IP, e.g. 10.10.12. and specify only the last tupel


This is also valid for the server la nip.

<SYSNR> specifies the instance number of the service. Valid range is 00 .. 97. At

now SAP supports only one instance.

<node> has to be set always to ‘001’ because of single node configuration. At now

we support only one node per instance.

--replid <replication id>

With HANA System Replication you need the replication id to distinct a SID within a

pool. The range for repkid is 0 to 9. The replication id ‘0’ has to be defined always

first and is the last one deleted.

--runingroup <name of poolgroup>

A HANA replica can have roles like ‘primary’ or ‘secondary’. To separate volumes

and userdata we need to assign a replica instance to a specific poolgroup for

running. Different replication instances (identified by replid) must not run in the

same poolgroup.

--runingroup <old poolgroup>:<new poolgroup>

This works only with operationcode ‘mod’ and moves a HRS instance to another

poolgroup.

--runingroup _migrate_to_hsr:<name of poolgroup>

This works only with operationcode ‘mod’ and add HRS attributes to a former



non-HRS specification. This only changes the LDAP attributes of the SID and

prepares file structure to HRS requirements. It guess that the automount

entries are HRS-ready (pools created with FFO 1.1 are HRS-ready).







dba which already exists cannot be assigned to a group number which deviates from

the default value.

--sysnr <SYSNR>

Removes a specific SAP instance instead of the entire system (SID) in case of

delete operation.

[ --sapshared <nas system>:/vol/<volume path> ]

Specifies the SID global HANA volume for storing shared data. <nas system> is the

NAS filer node name ending with ‘-st’.

[ --sapdata <nas system>:/vol/<volume path>:<SYSNR>:<nodeid> …]

Specifies instance and node specific HANA volumes for storing HANA data files. At

now there is only one volume because of the restriction to HANA single node

configuration. <nas system> is the NAS filer node name ending with ‘-st’.

[ --saplog <nas system>:/vol/<volume path>:<SYSNR>:<nodeid> … ]

Specifies instance and node specific HANA volumes for storing HANA log files. At

now there is only one volume because of the restriction to HANA single node

configuration. <nas system> is the NAS filer node name ending with ‘-st’.



instance_type::= {default|hdb}

os::= {SLES-11.x86_64}

In combination with the --add option, default:<os> sets the operating system of




In combination with the --mod option, default:<os> sets the operating system of




used with --op mod. You have to specify the full IP you want to exchange.





addresses.

Examples:

Adding a SID with HDB services (non HRS):

control1:~ # ff_sid_adm.pl --op add --sid HA1 --pool otto \

--sapversion 1.0 --db HANA \

--hdb 01:150:150:001 \

--hdbtype OLAP

Removing a SID with HDB service (non HRS):

control1:~ # ff_sid_adm.pl --op rem --sid HA1 --pool otto

Add HRS attributes to HDB services (switch to HRS):

control1:~ # ff_sid_adm.pl --op mod --sid HA1 --pool otto –replid 0 \

–runingroup _migrate_to_hsr_:DC_3

Adding a SID with HANA System Replication:

control1:~ # ff_sid_adm.pl --op add --sid HR1 --pool hrs \

--sapversion R- 1.0 --db R-HANA \

--replid 0 --runingroup DC_1

--hdb 01:150:150:001 \

--hdbtype OLTP

Removing a SID with HANA System Replication:

control1:~ # ff_sid_adm.pl --op rem --sid HR1 --pool hrs –replid 0



Switch poolgroup for HANA System Replication:

control1:~ # ff_sid_adm.pl --op mod --sid HR1 --pool hrs \

--replid 0 –runingroup DC_1:DC_2

10.5.8 SAP classic NetWeaver services using HANA database

Instead of specifiy a classic database type like Oracle or Sybase ASE HANA database

can be used with SAP NetWeaver 7.4 or 7.3. That means you have to

Specify a HANA database service (hdb) as described in chapter 10.5.7

Specify classic NetWeaver services referring to the selected HANA database

service

Synopsis


--db hanadb-ref[:<clientlan-ref>][:<serverlan-ref \

--sapversion { 7.4-HANA | 7.3-HANA } \

<sap classic specifications> …

ff_sid_adm.pl --op del --pool <pool_name> --sid <SAP_system_id>

--op add


--op { del | rem }


--op mod


specifications and exchanges IP adresses of specific SID instances.

--pool <pool name>




SAP System Handling Cloning a SAP SID into a Different Pool


--sapversion {7.4-HANA | 7.3-HANA }

Specifies the SAP NEtWeaver version

--db hanadb-ref[:<client-lan-ref>]:<server-lan-ref>

The client-lan-ref or server-lan-ref is the virtual hostname of the HANA database

service (hdb). There is no validation if the hostname refers to a HANA database

service.

With HRS you have to specify the hostname of replication id 0.

‘<sap classic specifications>’ describe the other instances like ascs or pai or other

options like group or user as described in chapter 10.3

Examples

Adding a SID with HDB services:

control1:~ # ff_sid_adm.pl --op add --sid HR1 --pool otto \

--sapversion 7.4-HANA \

--db hanadb-ref:hdb01ha1-se \

--sap ascs:01:100:100 \

--sap pai:02:101:101

Removing a SID with referenced HANA database service (only ‘HA1’ is removed, the

HANA database service remains untouched):

control1:~ # ff_sid_adm.pl --op rem --sid HR1 --pool otto

10.6 Cloning a SAP SID into a Different Pool

10.6.1 Script: ff_clone_sid.pl

The script ff_clone_sid.pl allows users to clone (basically copy) an entire SID from

one pool to another. It needs to be clearly understood that only the FlexFrame-specific

administrational data in the LDAP server as well as the required information for the

operating system's naming services are copied and/or added to the LDAP database. Any

additional work (like copying SAP/database binaries and database content) will not be

performed by this tool and needs to be performed, as well.

Cloning a SAP SID into a Different Pool SAP System Handling


To get a real clone the script has to expect that there are no conflicts in userids, groupids,

services and other properties between the origin and the target pool. You can check it by

using option ‘—dryrun’.

If there are conflicts and you are sure it does not matter you can clone the SID and the

script will select other values than defined in origin. Otherwise the cloning could mean

inconsistencies in your LDAP database. Be careful !

Synopsis

ff_clone_sid.pl –-sid <sid>

--srcpool <pool name>

--trgtpool <pool name>

[--dryrun ]

[--renew ]

Options

--sid <sid>

Specifies the SID which should be cloned.

-srcpool <pool name>

Specifies the pool with the origin SID specification.


Specifies the pool wherein the clone should be created.

[ --dryrun ]

Shows you what happens if you request a clone. You can use the option to check if there

are inconsistencies between the source and the target pool.

[ --help ]

Shows the usage.

[ --renew ]

You can enforce cloning if you have only conflicts with IP addresses between source and

target. If you are sure that it does not matter you can use this option and the script selects

new IP addresses independent from the source.

10.6.2 Changing User and Group IDs after Cloning

The ff_change_id.pl script is not further available.

To change the UID of an OS user you can use ff_user_adm.pl.

To change the GID of an OS group you can use ff_group_adm.pl.

SAP System Handling Cloning a SAP system into a different Pool


10.7 Cloning a SAP system into a different Pool

10.7.1 Script: ff_clone_sap.sh

The script ff_clone_sap.sh allows users to clone (basically copy) an entire SAP

installation from one pool to another. It means also the binaries are copied to the target

pool and so you get a complete independent 'copy' of your source system. The command

tries to do all configuration tasks needed to run the SAP system in the target pool.

You can clone a SAP system also if the system is active and the database is online. SAP

systems based on HANA database can only be cloned if database is offline.

The script is based on NetApp FlexClone feature. You cannot clone SAP

installations having their sapdata/saplog data on SAN.

The volume to be cloned and the clone volume have to be on the same

filer. You cannot use different filers for original and clone volume !

The script cannot check if your clone request overwrites newer versions of

files already copied earlier with older file versions. You are responsible for

that issue. Use cloning with care !

With ff_clone_sap.sh you copying a complete SAP system. You are

responsible for license issues using SAP software.

At the current release the command supports SAP services with configured

databases and also other services with the only exception of the service

BOBJ.

The script copies the SAP system files (which each clone request)

located in /usr/sap/<SID>

located in /sapmnt/<SID>

home directories of SAP specific users like <sid>adm

/usr/sap/sapservices

Database files (software, configuaration) depending on database type

Cloning a SAP system into a different Pool SAP System Handling


Some files are only copied if not available in the target configuration or on force request

/usr/sap/hostctrl

/usr/sap/tmp

/oracle/client

/sapdb/data

/sapdb/programs

/sapdb/sql

There are some preconditions you should take into account if you want to clone SAP

systems

the environment used (users, groups, etc.) has to be equal in source and target

pool. Otherwise you may get problems accessing files and directories. It is an

essential precondition.

do not clone SIDs from different source pools to one target pool. We

recommend to create always an own target pool to each pool from which you

want to clone SIDs.

only SAP systems residing on NetApp NAS storage are supported.

Be careful when cloning with different SAP versions. The script cannot prevent

overwriting files with older revisions.

It is necessary to store the files from sapdata/saplog in an own volume to

optimize disk storage.

Newer SAP release require an SMD agent per default. Do not forget to clone

also those installation (if SMD is supported in later). There is no automatic clone

of depending installations.

If you want to unload volFF we recommend to relocate SID specific directories in

the target pool using relocation feature.

At start of the clone process it is not known how much storage is necessary for

the running clone. You are responsible to supply enough storage. In the worst

case the clone needs storage resources as much as the origin.



HANA configurations are only cloned if the configuration to be cloned is a single

node configurationHANA configurations can be cloned only if the original

configuration is offline

Synopsis

ff_clone_sap.sh [ --op | -o ] { create | destroy | crclone |

rmclone | initpool | check }



--sid | -s <sid>

--flexdata <sapdata clone volume >

--flexlog <saplog clone volume>

[ --flexshare <sapshared clone volume> ]

[ --nodeid <nodeid of HANA master node> ]

[ --replid <replication ID> ]

[ --runingroup <poolgroup to be used> ]

[ --instance <instance number of HANA service> ]

[ --reloc <nas system>:<volume> ]

[ --dbhost <virtual DB hostname> ]

[ --config <user defined config file> ]

[ --prefix <prefix used for snapshot name> ]

[ --fcopy ]

[ --help | -h ]

[ --version | -v ]

[ --debug | -d ]

[ --dryrun ]

Options

--op initpool

After introducing the target pool by ff_pool_adm.pl use this opcode to create global

users and groups to get a consistent environment between source and target. This

step has to be done before creating first clone.

--op create

Creates a complete clone of your SAP system (cloning of database files, binaries

and global files if necessary)

--op destroy



Deletes the clone completely. Only global files are not deleted.

--op check

Runs a check to examine your environment. It is useful to recognize violations of

most of the preconditions

--op crclone

Creates only a clone of your database

--op rmclone

Removes the database clone only


Name of the pool which contains the SID to be cloned..


Name of the pool the SID should be cloned into.

--sid <sid>

Specifies the SID to be cloned.

--flexdata <nas system>:<volume name>

Name of the clone volume for sapdata. This volume is created through FlexClone

feature, not by the caller of the script.

The name of the filer is the filer’s name as used in first pool. It does not contain the

name of the target pool. <nas system> is the NAS filer node name ending with ‘-st’.

Depending on syntax of ‘<filer-name>’ you may get an error message like ‘Error:

Invalid filername used - pool specification invalid: zrfiler-1-st’. In this case set <filer-

name> to ‘zrfiler-1’ without the storage lan suffix and run command again.

--flexlog <nas system>:<volume name>

Name of the clone volume for saplog. This volume is created through FlexClone

feature, not by the caller of the script.

The name of the filer is the filer’s name as used in first pool. It does not contain the

name of the target pool. <nas system> is the NAS filer node name ending with ‘-st’.

Depending on syntax of '<filer-name>' you may get an error message like 'Error:

Invalid filername used - pool specification invalid: zrfiler-1-st'. In this case set <filer-

name> to 'zrfiler-1' without the storage lan suffix and run command again.

--flexshare <filer name>:<volume name>

Name of the clone volume for shared volume. This volume is created through

FlexClone feature, not by the caller of the script.

The name of the filer is the filer's name as used in first pool. It does not contain the

name of the target pool.

This option is only used when cloning HANA configuration



--reloc <nas system>:<volume name>

Name of specific volume in target pool to store SID specific data (unload volFF). You

are responsible for the creation and availability of the volume. You can use the

command ff_sid_mnt_adm.pl to modify filer exports and create a script to create the

necessary directories on your volume. <nas system> is the NAS filer node name


--dbhost <dbhost name>

Contains the virtual name of the database host (db<sid>-se). It is used only if the

clone should be taken from an inactive system (database offline).

--nodeid <nodeid>

Contains the node ID of the HANA configuration. It is always set to ‘001’ because we

do not support multimode configurations

--replid <replication id>

Replication ID of HANA System Replica. The valid range is 0 to 9. The replication

with ID ‘0’ has to be cloned always first.

--runingroup <poolgroup>

Poolgroup specification which contains Application Nodes which are assigned to run

specific replication instance. If omitted the script guess that the clone pool consist of

the same poolgroups as the original.

--fcopy

Enforce copy of global files. If the directory already exists in target configuration it is

overwritten (use it with care !).

--prefix <name>

Name of the prefix used for naming snapshot. Default is FF_DATA resp. FF_LOG

--help

Shows the usage

--debug

Switch on debug mode

--dryrun

Starts a cloning without changing anything. You will see what will happen if you are

requesting a real clone.

--version

Shows the script version.

Return Codes

0 means all o.k.



9 at least one warning, but the clone request succeeded

1 abortion of the request because of errors

10.7.1.1 Cloning SAP systems with MaxDB

Some files of MaxDB database configuration are stored in /sapdb/data/config or

/sapdb/FLM/data/config. To ensure consistency you have to move those files to

/sapdata/sapdata<n>/config and create a link at origin site before you start the cloning

operation

<SID>.<nn>

<SID>.cfg

<SID>.pah

<SID>.upc>

<SID>

.M*

10.7.1.2 Post cloning steps – DB2

After cloning a SAP system based on DB2 database you have to execute

exchange hostname in /home_sap/db2<sid>/sqllib/db2nodes.cfg with the name

of the application node you want to start the cloned system first

/home_sap/db2<sid>/db2_software/instance/db2iupdt db2<sid> under control

of user ‘root’

Before calling ‘db2iupdt’ please check if there is a file named

‘/home_sap/db2<sid>sqllib/global.reg’ and check for that file in directory

‘/var/db2’. With some software release it could be necessary to make a copy

from the file into ‘/var/db2’. Please make the copy always. Otherwise the

command could fail.

We guess you changed the password of the corresponding DB2 users

(/sap<sid>, db2<sid>, etc.). After cloning the passwords for the users in the

clone are set to default values. Change your passwords using ff_user_adm.pl.

start the database using ff_service.sh

login as db2<sid> and execute ‘db2inidb <SID> as snapshot’ if there are

connection problems (error message SQL20153 by executing ‘db2 connect’)

For further informations please refer to ‘DB2: Cloning a Database Using NetApp

FlexClone Technology’ by NetApp



10.7.1.3 Post cloning steps – Oracle

If your clone is based on an online system you have to execute a ‘recover database’

using ‘sqlplus’ under control of user ‘ora<sid>’.

We guess you changed the password of the corresponding Oracle users (/ora<sid>,

<sid>adm) in your origin installation. After cloning the passwords for the users in the


10.7.1.4 Post cloning steps – MaxDB

If you are cloning a system with MaxDB less than 7.8 the configuration files from

/sapdb/data/config (7.8: /sapdb/<SID>/data/config) are copied at the time the first clone

operations took place. All further cloning operations for MaxDB do not change

/sapdb/data/config and the configuration files are possibly not up to date. This may cause

problems while starting database.

We guess you changed the password of the corresponding Oracle users (/sqd<sid>,

<sid>adm) in your origin installation. After cloning the passwords for the users in the


After cloning the SAP system ensure the consistency of the configuration files by

executing

:~ # : cd /FlexFrame/volFF/pool-<target>/sapdb/<platform>/data/config

cn1:~ # rm <SID>.???

cn1:~ # cd /FlexFrame/volFF/pool-<source>/sapdb/<platform>/data/config

cn1:~ # find . | cpio –dump \

cn1:~ # /FlexFrame/volFF/pool-<target>/sapdb/<platform>/data/config

10.7.1.5 Post cloning steps – HANA reference database

SAP NetWeaver lately often uses SSFS (Secure Storage in File System) to store

authorization data, e.g. to store login data for database connects.

The data files for connection HANA are always stored at '<home-of-sidadm>/.hdb' (don't

confuse it with the settings made for NetWeaver itself written in profile). But each

application host has its own subdirectory.

Either you rename the host specific directory using the application hostnames of your

targetpool or you copy 'SSFS_HDB.DAT' from one host directory to missing host

directories (may be you have to create the directory first) .



10.7.1.6 Cloning a SAP system example session

We guess you have a pool ‘source’ with installed SAP services using SID ‘ORA’. You

want to clone that installation to a pool ‘target’. We further guess that your sapdata/saplog

are stored on a SID specific volume (command ff_sid_mnt_adm.pl used before executing

ff_setup_sid_folders.sh). Otherwise your clone will include other databases which

belongs to SIDs you do not want to clone. This is just a matter of needed disk space.

At first you have to create your target pool by executing

Control1: ff_pool_adm.pl –op add –name target –storage 16,

172.20.16.0,255.255.255.0 –server 17,172.20.17.0,255.255.255.0

–client 15,172.20.15.0,255.255.255.0 –dns ff4s.mch.fts.net

–defrouter 172.20.18.251

The used IPs are just examples, you insert your own configuration values. If you want to

specify a pool specific volFF volume you have to extent the command by ‘—volFF

<filer>,/vol/<volume name>’. Before using the volume within this command you have to

create it manually by NetApp filer commands or ask your storage administrator.

In the next step you initialize your target pool to make sure that UIDs and GIDs are

in consistent state with the source pool. Please execute

Control1: ff_clone_sap.sh –-op initpool -–srcpool source -–trgtpool target

Now you can clone your SAP system. Please take into account that SAP systems with

MaxDB needs to move configuration files from /sapdb/data/config or

/sapdb/FLM/data/config to a sapdata directory to ensure consistencies (for requested files

see above).

We guess our filer is named ‘jer1na2’ and the volume names for sapdata/saplog are

‘/vol/sid_ora_data’/’/vol/sid_ora_log’.

You don not need to create the volumes for the clone, you just give it names. Those

volumes are created by the FlexClone feature of NetApp. You are only responsible to

supply enough disk space for clone operation and the later operation of the clone in

target pool.

To clone your system (SAP services are down; database offline) you execute

Control1: ff_clone_sap.sh -–op create -–srcpool source -–trgtpool target \

--sid ORA \

--flexdata jer1na2:/vol/sid_ora_data \

--flexlog jer1na2:/vol/sid_ora_log \

--dbhost dbora-se

Some directories/files are copied one time (e.g. if you have MaxDB 7.7 database

software and it is valid for all cloned systems, /sapdb/programs needs to be copied only

with the first clone operation).

If you want to force the copy of those files you additionally use the option ‘—fcopy’. With

SAP System Handling Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl


older versions of MaxDB software you should use ‘—fcopy’ for each clone because of

global directory ‘/sapdb/data’

If you need additional diagnostic support you can use the option ‘—debug’.

Please check the filer configuration by executing

Control1: ff_nas_adm.pl –op list –name jer1na2

If you cannot find ‘jer1na2-target-st’ in output of the command you need to execute

Control1: ff_nas_adm.pl –op list –name jer1na2 --pool target

Now you should be ready to operate your clone in target pool. Because of changes within

the automount configuration you should execute

applnode: /etc/init.d/autofs restart

Please check also for necessary manual operations depending on database type

described above.

10.8 Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl

To unload volumes containing database files or SID specific directories/files you can use

ff_sid_mnt_adm.pl. As of FlexFrame 5.2 this command takes over the functionality of

former ff_relocate_sid_data.pl. This means some minor incompatibilities depending on

the type of request.

Due to the combination of both commands you have to pay attention for some restrictions

using this command

The specification to relocate sapdata/saplog/sapshared has to be done before

ff_setup_sid_folder.sh is called

All other relocations have to be specified after the call of ff_setup_sid_folder.sh

SAN storage is only supported for sapdata/saplog volumes

If you have relocated sapdata/saplog/sapshared and you want to change it once more

again you have to delete the storage location description (automount entries in case of

NAS storage) and then add it again. But in that case the script does not offer support to

move the data to the new location. It has to be done manually. This can also cause that

you cannot use automounts on Application Nodes without restarting ‘autofs’ or even the

nodes.

If you have relocated other SID specific data and you want to change it once more again

you use the same procedure as used with sapdata/saplog.

If you want to withdraw your relocation to the default sites (volFF) you have to delete the

corresponding automounts and move back the data manually. But this move back needs

Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl SAP System Handling


some knowledge of FlexFrame internals and should be executed by FlexFrame

professional service people.

Synopsis (using common syntax)

ff_sid_mnt_adm.pl --op add

--pool <pool name>

--sid <sid>

--name <type>:<volume spec> …

[ --dryrun ]

[ --debug ]

ff_sid_mnt_adm.pl --op { del | rem }

--pool <pool name>

--sid <sid>

--name <type> …

[ --dryrun ]

[ --debug ]

ff_sid_mnt_adm.pl --op list

--pool <pool name>

--sid <sid>

[ --debug ]

ff_sid_mnt_adm.pl --help

Options

--op add

Adds new storage location descriptions (automount entries in case of NAS) into

LDAP to relocate sapdata and/or saplog.

--op { del | rem }

Deletes the relocation entries for sapdata and/or saplog



--op list

Displays information about relocation of sapdata and/or saplog for specific SID.

--sid <sid>

Specifies the SID for which the relocation entries should be handled..

--pool <pool name>

Specifies the corresponding pool which contains the SID specification.

--name <type>:<nas system>:<volume path>[/<pool name>/<SID>]…

With opcode ‘add’ the option ‘name’ is used to select the data you want to relocate to

another volume and requires a type and a volume specification. <nas system> is the

NAS filer node name ending with '-st'. With type ‘sapdata’ and ‘saplog’, it is also

possible to use SAN. In this case, the volume specification must be given as

_SAN:<volume group name>.

With opcode _’rem’/’del’ you only need the type specification within the option

’name’.

These options could be repeated several times. But each item can be used one time

per command.

The type specification is one of

oracle

sapdb

db2

sybase

sapmnt

usr_sap

<sid>adm

sqd<sid>

db2<sid>

sap<sid>

sap<sid>db

sapdata

saplog

sapbackup

This type is normally part of type ‘saplog’ and normally there is no request to

split that data to an own volume. But with backup tools like Simpana it could be

Relocating sapdata/saplog/SID specific data by ff_sid_mnt_adm.pl SAP System Handling


helpful to assign an own volume.

The explicit relocation of sapbackup works only with NAS storage, not

with SAN.

With HANA Database Service ‘hdb’ there is an extended syntax used. The synopsis

for operation code ‘add’ is:

--name sapshared:<nas system>:<volume path> …

Specifies the SID global HANS volume for storing shared data. <nas system> is

the NAS filer node name ending with '-st'.

--name sapdata:<nas system>:<volume path>:<instnr>:<nodeid> …

Specifies the instance and node specific volume for storing HANA database

datafiles. <nas system> is the NAS filer node name ending with '-st'.

--name saplog:<nas system>:<volume path>:<instnr>:<nodeid> …

Specifies the instance and node specific volume for storing HANA database

logfiles. <nas system> is the NAS filer node name ending with '-st'.

With type ‘sapdata’ and ‘saplog’, it is also possible to use SAN. In this case, the volume specification must be given as _SAN:<volume group name>

The synopsis for opcode delete is:

--name sapshared

--name sapdata:<instnr>:<nodeid>

--name saplog:<instnr>:<nodeid>

Hints

Moving sapmnt is a special case. If new SID instances are introduced by

ff_sid_adm.pl the mount points SID and SID_exe are already created.

ff_sid_mnt_adm.pl renames these two nodes and generates new entries. If you

are deleting sapmnt, the previous entries are restored.

If you are using ‘--name sapdata …’ and/or ‘--name saplog …’ you cannot use

the old options ‘--sapdata’ and/or ‘--saplog’ and reverse.

With the ‘ff_sid_mnt_adm.pl’ from a version FF 5.0 or 5.1 all relocated entries

(sapdata/sapog) were deleted if you request a delete operation. Now you should use the name option ‘--name sapdata’ and/or ‘--name saplog’ to specify what you



want to delete. So you are able to delete either sapdata entries or saplog entries or

both.

--help

Shows the usage.

--dryrun

Shows what would happen if you execute the command

--debug

Enables script to write additional debug messages in debug file located in

‘/var/log/FlexFrame/<debug_log>’.

--force

Option ‘force’ can be used to enforce the operation in case of ‘minor’ errors. But we

strongly recommend to use it very carefully. For normal operation the option is not

really necessary and should only be used by professional service guys.

10.8.1 Hints for usage

10.8.1.1 General

Filer exports are modified internally by ff_sid_mnt_adm.pl with opcode ‘add’. This feature

is available for NetApp filers.

If you are removing relocated data the exports have to be modified manually by

ff_nas_adm.pl. If the volume should be deleted no action is necessary.

The use of type 'sapdata/saplog' in name option in case of opcode ‘add’ only change the

automount entries in LDAP and ‘ff_setup_sid_folder.sh’ is used to make directory

structures available. To relocate sapdata/saplog/sapshared you have to call

ff_sid_mnt_adm.pl before calling ff_setup_sid_folder.sh. No further actions are necessary

to enable application nodes to use the new volume. In rare cases a ‘/etc/init.d/autofs

restart’ on application nodes is necessary.

For all other types a script is generated (name is displayed by script) to move the data

from origin storage to relocated storage location. You only have to call the script to initiate

copying of data. The origin data are not deleted automatically. It has to be done manually.

10.8.1.2 Creating a SID/relocation of sapdata/saplog

If you want to use own volumes to store sapdata/saplog you have to

Specifying SID using ‘ff_sid_adm.pl’

Relocation of sapdata/saplog using ‘ff_sid_mnt_adm.pl’

Multiple NAS Systems and Multiple Volumes SAP System Handling


Call of ‘ff_setup_sid_folder.sh’ to create all directories needed for SID

The request for relocation has to be done before ‘ff_setup_sid_folder.sh’ is called !

10.8.1.3 Creating a SID/relocation of SID specific data

Additionally to the relocation of sapdata/saplog you can also relocate other SID specific

data. You have to

Specifying SID using ‘ff_sid_adm.pl’

Relocation of sapdata/saplog using ‘ff_sid_mnt_adm.pl’ (optional)

Call of ‘ff_setup_sid_folder.sh’ to create all directories needed for SID

On Application Node login to relocated user directories, e.g.

'/home_sap/<sid>adm' by 'su - <sid>adm', and execute a 'touch .history' (only

requested with older releases)

Relocation of SID specific data (other than sapdata/saplog) using

‘ff_sid_mnt_adm.pl’

Run generated scriptfile to move data to relocated storage area

The request for relocation has to be done after 'ff_setup_sid_folder.sh' is called !

‘ff_setup_sid_folder’ does not create directory and file structures on relocated storage

area.

10.9 Multiple NAS Systems and Multiple Volumes

During the installation process, FlexFrame assumes that there is one NAS system with sapdata and saplog volumes. Larger installations may require more than one NAS

system or multiple volumes on the same NAS system.

It is possible to distribute SAP databases across multiple NAS systems and multiple

volumes under the following conditions:

9. All NAS systems were entered in the FlexFrame Management Tool, prior to

installation of the FlexFrame landscape.

10. The software for SAP and the database are located in the centralized volume volFF

of the first NAS system or alternatively (if wanted) in a pool-specific volFF volume in

an arbitrary NAS system (defined within the FlexFrame environment).

11. For each SID you can assign a sapdata volume for the database's data files. This

sapdata volume can be shared with other SIDs or solely for this SID. But if you

want to share multiple SIDs on a volume we recommend you to use pool-specific

volumes for sapdata. It will minimize your efforts for administration tasks.

SAP System Handling Multiple NAS Systems and Multiple Volumes


12. For each SID you can assign a saplog volume for the database's online redolog

files. This saplog volume can be shared with other SIDs or solely for this SID. But if

you want to share multiple SIDs on a volume we recommend you to use pool-specific

volumes for saplog. It will minimize your efforts for administration tasks.

10.9.1 NetApp NAS Systems

The volumes must be created manually on the Filer, e.g. like:

filer2> vol create dataC11 10

filer2> vol create logC11 4

Here, dataC11 and logC11 are the new names of the volumes and 4 and 10 are the

numbers of disks to be used.

We recommend using the volume names sapdata and saplog (if on a different

Filer) or data<SID> and log<SID> for SID specific volumes on the same Filer.

You may use FlexVols (ONTAP 7G) or regular volumes.

The following options must be set for the volume:

filer2> vol options dataC11 nosnap on

filer2> vol options dataC11 nosnapdir on

filer2> vol options dataC11 minra off

filer2> vol options dataC11 no_atime_update on

filer2> vol options logC11 nosnap on

filer2> vol options logC11 nosnapdir on

filer2> vol options logC11 minra off

filer2> vol options logC11 no_atime_update on

Next, create qtrees for each FlexFrame pool which will store data and logs in those

volumes:

filer2> qtree create /vol/dataC11/pool1

filer2> qtree create /vol/logC11/pool1

If you use more than the first Filer, make sure it is reachable using, e.g.:

control1:~ # ping filer2-st

PING filer2-st (192.168.10.203) from 192.168.10.201 : 56(84) bytes ofdata.

64 bytes from filer2-st (192.168.10.203): icmp_seq=1 ttl=255 time=0.117 ms



Now the LDAP database has to be told that this SID is not using the default (first) Filer

and sapdata/saplog volumes:

Upgrading a SAP System SAP System Handling


control1:~ # ff_sid_mnt_adm.pl –-op add –-pool pool1 –-sid C11

--sapdata filer2-st:/vol/dataC11/pool1/C11 –-saplog filer2-st:/vol/logC11/pool1/C11

The step to modify filer’s export file is not necessary as of FlexFrame 5.1 because this

step is integrated in command ff_sid_mnt_adm.pl. This feature is only available with

NetApp.

The network 192.168.10.0/24 must match the Storage LAN segment of pool

pool1.

Before you can install SAP and the database on those volumes, some folders for the SID

in question have to be created in advance. To do so, run the following command for each

SID (replace "pool1" with your pool name and "C11" with your SID:

control1:~ # ff_setup_sid_folder.sh –p pool1 –s C11

Now you can continue with the SAP installation.

10.10 Upgrading a SAP System

10.10.1 Service Port

If you plan to upgrade your SAP release, you have to add a special service port (called

shadow instance) to LDAP.

SAP shadow service ports are required during an SAP release upgrade. To list, add or

remove the service ports in the LDAP database for service entries, you can use this

tool.

Synopsis

ff_sap_shadowport.sh [-d] -l -p <pool name> [-s <sid>]

[-o <portnr>]

ff_sap_shadowport.sh [-d] {-a|-r} -p <pool name> -s <sid>

[-o <portnr>]

Options

-d

Writes debugging information.

-l

Lists all SAP shadow service ports of the pool provided with the -p option.

If the option -s is used, only the port of the specified SID is displayed.

SAP System Handling Upgrading a SAP System


-a

Adds an entry.

-r

Removes an entry.

-p <pool name>

Specifies the name of the pool (e.g. pool1).

-s <sid>

Specifies the SAP System ID (SID) by a 3 character string (e.g. C11).

-o <portnr>

Specifies the service port number. The default is 3694 (optional).

10.10.2 FlexFrame Agents

Please make sure that the FlexFrame Application Agents are stopped on the hosts while

you are installing, updating or removing any SAP or database software:

Stop the FlexFrame Agent and check the status:

control1:~ # /etc/init.d/myAMC.FA_AppAgent stop

control1:~ # /etc/init.d/myAMC.FA_AppAgent status

There should be no running processes listed.

10.10.3 Support SAP Upgrade (ff_sap_upgrade.pl)

This command is used to support the migration from a previous SAP/database versions

to a higher one. It could be also useful if you are migrating from an older release of FF to

a newer one. As of FlexFrame 5.2 you can also downgrade the information in LDAP

given for SAP and database release. This downgrade only means changing information

in LDAP, nothing more.

‘ff_sap_upgrade.pl’ cannot be used to change the database type (e.g. from Maxdb to

Oracle). A type change is also not allowed (e.g. SAP to MDM). We recommend a deletion

of the SID and the recreation with the new type/database type.

Upgrading a SAP System SAP System Handling


You are able to change the SID specific settings in LDAP database like version of SAP or

database installation. It also tries to introduce the LDAP data which are requested with

newer SAP versions. The script tries to cover all requirements concerning version of SAP

or database service.

Sometimes users want to migrate from one database type to an other. The script cannot

execute all actions requested by such a migration. Manual actions could be required.

SAP System Handling Upgrading a SAP System


Sometimes a new SAP release requires additional operations after the upgrade of the

SAP version in LDAP. So the virtual hostname may change (e.g. for JAVA from SAP 7.0

or less to SAP 7.1 or higher). For that cases you will get a hint after processing the

command. This meant that SAP owned files like profiles are have to be changed, too.

Synopsis

ff_sap_upgrade.pl --pool <pool name> --sid {<SID>|\*}

[ --sapvo <old version> ] --sapvn <new version>

[ --dbvo <old version> ] --dbvn <new version>

--type <service type>

[ --version ] [ --debug ] [ --dryrun ]

ff_sap_upgrade.pl [ --help ]

Options

--pool <pool name>

Name of the pool the SID should belong to.

--sid <SID>

SAP system identifier. Either you specify a specific SID or you use the asterisk. Asterisk means all SIDs of the specific pool which match to sapvo and dbvo are

modified.

--sapvo <old version>

Current value of the SAP version stored in LDAP (without leading SAP-).

Available values <old version> can be displayed with option ‘—help’

--sapvn <new version>

SAP version the SID is migrated to. <new_version>. Available versions are

displayed with option ‘—help’.

--dbvo <old version>

Version of previous used database version. Available versions are

displayed with option '-help'..

SAP Kernel Updates and Patches SAP System Handling


--dbvn <new version>

Version of database the SID is migrated to. Available versions are displayed with option '-help'.

--type <service type>

Specifies the SAP service. Default value is ‘SAP’, other values are BOBJ, CMS,

MDM, SMD, TREX and WD. Except ‘SAP’, all services are new. This is the reason

why option ‘type’ has been introduced.

--version

Show version of command.

--debug

Sets the debug option to enlarge logging.

--dryrun

Just to see which changes would be made in LDAP if script is executed.

--help

Display usage.

10.11 SAP Kernel Updates and Patches

For an SAP kernel update (binary patches), logon to the Application Node with the CI of

the SAP system that is to be updated. Please make sure that the FlexFrame Application

Agents are stopped on the host while you are updating the SAP kernel (see also section

“FlexFrame Agents” on page 246).

SAP's OSS note 19466 describes where to find kernel patches and how to handle the

installation.

For the installation of SAP ABAP patches or similar, please refer to the SAP

documentation. There are no FlexFrame specific changes.

10.12 Unloading volFF

All SID specific data are located on the same volume (default: volFF). In case of request

of diagnostics it could happen that the volume space is not sufficient or it is not easy to

find the specific data you need because of all SIDs write into the same location.

The command ff_sid_mnt_adm.pl offers the possibility to relocate most of the SID specific

data. Please refer to corresponding command description.

SAP System Handling Administrating SAP Services


10.13 Administrating SAP Services

In a FlexFrame environment, any type of SAP instance, i.e. database (DB), SAP Central

Services (SCS), Java central instance (JC), Central Instance (CI), application instance

(APP) or Java-only stack (J), is called an SAP service. The management can be done

either web based with the FlexFrame Agents WebGUI or script based with the adapted

start/stop scripts.

If the SAP LVM is running, SAP services can be managed by LVM.

10.13.1 Displaying Status of SAP Services

10.13.1.1 FlexFrame Agents WebGUI

The FlexFrame Agents WebGUI can be used to display the states of SAP services in a

FlexFrame environment. To see all active SAP services and nodes in the WebGUI, the

FlexFrame Application Agent should be started on all Application Nodes. The displayed

active SAP services can be shown system-related or node-related.

The following table shows the node and service states of FlexFrame Autonomy and the

corresponding colors of these states in the FlexFrame Agents WebGUI:

Administrating SAP Services SAP System Handling


white

inactive or no

further

information

green

normal,

everything ok

yellow

warning

red

critical

black

critical

Node states

RUNNING

SWITCH_INT

SWITCH_EXT

PowerOff

Service states

SHUTDOWN

DOWN

WATCH

NOWATCH

UNKNOWN

NULL

RUNNING

STOPPING

STARTING

REBOOT

RESTART

RESTARTING

REBOOT

REBOOTING

RBGET

SWITCH

SWITCHOVER

SWGET

ERROR

If you don't want to configure the parameters in the plain xml files, you may use the

FlexFrame Agents WebGUI to do this more conveniently. Details are provided in the

“FlexFrame Agents Installation and Administration“ manual.

10.13.1.2 List SAP Services

The script ff_list_services.sh lists the status of all installed SAP services. If no

pools are specified, services of all pools will be shown.



Synopsis

ff_list_services.sh [-ivcCH] [<pool> ...]

Options

-i

Shows inactive services

-v

Verbose mode

-c

Force use of colors for status information

-C

Suppress colors for status information

-H

Suppress headers

-h or -?

Shows usage

10.13.2 Starting and Stopping Application Services

Virtualization of the applications and services demands special measures for starting,

stopping, restarting them etc. These measures are cared for by an SAP service script for

each service type.

As of FlexFrame 5.0 there is an incompatibility concerning starting and

stopping SAP services. The standard sapxx scripts are deprecated and all

requests are now handled by ‘ff_service.sh’ script.

We also now use Linux command ‘ip’ to start and stop virtual interfaces.

Therefore you will not see the interfaces using ‘ifconfig’ as expected. To show

the interfaces you should use now ‘ip -4 –o addr show’ or ‘ff_service.sh’ with

action code ‘icheck’.

Only project specific solutions like sapxprint will be supported furthermore.



The application or service must not be started directly, e.g. for an SAP instance as <sid>adm with startsap, since in this case the interfaces are neither

supplied with IP addresses, nor is the service control file maintained. The

started application will not work due to the lack of a network connection.

We strongly recommend you not to logon at application node with the client

LAN interface which you want to stop. In case of stopping a service you would

loose your connection and your terminal would hang up.

10.13.2.1 Configuration file (ff_service.config)

ff_service.sh reads settings from file ff_service.config. This file contains defaults used

while starting a service

location of profiles (depending on installed SAP release)

service specific parameters (e.g. depending on database service)

service specific kernel parameters (depending on SAP service)

cpupower parameter settings (depending on SAP service)

Most of the default settings are suitable for running SAP services. But sometimes the

behaviour of start and stop has to be switched to non-default or in your environment you

recognize that the given defaults are insufficient.

You can modify the defaults in a configuration file of your own

make a copy of ff_service_config.custom.TEMPLATE to

/FlexFrame/scripts/ff_service.config.custom

modify the corresponding configuration parameter you want (do not forget to

remove ‘# ’ in first column to activate the change)

you can restrict an option to a SID (replace SID by your SID)

you can restrict an option to a SID within a pool (replace SID by your SID and

POOL with your poolname

you can globally set an option if SID and POOL are omitted

You can only modify the values available in the template file. The value in

ff_service.config.custom overwrites the values set in ff_service.config.

If there are multiple values for an option there is a priority order, SID, POOL-

SID and global.

IF POOL and/or SID are not mentioned it is a values which can be set only

globally.



10.13.2.1.1 MaxDB

If you are concerned mit SAP case 1242763 you can set the parameter to ‘on’.

# CONFIG:maxdb:SID_xserver_start_options=-a off

# CONFIG:maxdb:xserver_start_options=-a off

If you have a performance issue with a large number of dbmsrv processes (see also SB-

FF-13P07) you kann use a host specific Databases.ini file. Set the option to ‘yes’

# CONFIG:maxdb:private_configuration_file=no

If the previous option is set to ‘yes’ you can set this option also to ‘yes’ and the scripts will

use attach/detach for MaxDB registration.

# CONFIG:maxdb:private_configuration_file_use_saphostctrl=no

10.13.2.1.2 Sybase

To connect to Sybase ASE database using ‘isql’ in newer versions it is necessary to set

the command line option ‘-X’. If you run older version it is not necessary to set the option

# CONFIG:sybase:SID_POOL_ecryption_switch=-X

# CONFIG:sybase:POOL_encryption_switch=-X

# CONFIG:sybase:encription_switch=-X

10.13.2.1.3 DB2

Some DB2 database configurations have the problem of long lasting DB Crash Recovery

runs. In that case DB dependent services may run into a restart loop. Setting the option to

‘yes’ means that dependent services will wait a while for end and set the ‘nowatch’ flag

for FlexFrame Agent for that time.

# CONFIG:db2:SID_POOL_db2crash_set_nowatch_flag=no

# CONFIG:db2:POOL_db2crash_set_nowatch_flag=no

# CONFIG:db2:db2crash_set_nowatch_flag=no

10.13.2.1.4 HANA

If there is a need to increase the wait time (e.g. you always get timeout failures uring

start/stop) you can increase the value (in seconds) to a value of your choice.

# CONFIG:hana:SID_POOL_hdb_server_stopWaitTime=400

# CONFIG:hana:POOL_hdb_server_stopWaitTime=400



The following values are defined by SAP Notes concerning HANA configurations. If there

are changes requested you can easy change the values. But change only if there is a

concrete request !

# CONFIG:hana:sysctl:vm.memory_failure_early_kill=1

# CONFIG:hana:sysctl:kernel.shmmni=65536

# CONFIG:hana:sysctl:vm.dirty_ratio=5

# CONFIG:hana:sysctl:net.ipv4.conf.all.arp_ignore=1

# CONFIG:hana:sysctl:# NetApp note TR 3700'

# CONFIG:hana:sysctl:fs.file-max=20000000

# CONFIG:hana:sysctl:net.ipv4.ip_local_port_range=40000 65500

# CONFIG:hana:sysctl:fs.aio-max-nr=1000000

HANA configuration expects some CPU power settings. If there are new rquests by SAP

Notes you can change the values. But change only if there is a concrete request.

# CONFIG:hana:cpupower:POOL_POOLGROUP_set_cpu_governor=performance

# CONFIG:hana:cpupower:POOL_set_cpu_governor=performance

# CONFIG:hana:cpupower:set_cpu_governor=performance

# CONFIG:hana:cpupower:POOL_POOLGROUP_c-state-disable=yes

# CONFIG:hana:cpupower:POOL_c-state-disable=yes

# CONFIG:hana:cpupower:c-state-disable=yes

10.13.2.2 Starting and Stopping SAP Services Without Root Privileges

In order to enable user to start and stop SAP services without having root privileges a

sudo (see man sudo) configuration is shipped with the Application Nodes. This

configuration allows the SAP and the DB administration users to start their respective

services (including their virtual hostname's IP addresses). The user command line

requires a prefix of sudo in addition to the standard command line (see example below):

%> sudo /FlexFrame/scripts/ff_service.sh –s <SID> -t db -a start

The sudo mechanisms will check against its configuration file if the user (or the

responsible group) is allowed to run the command and run it with root privileges.

The configuration file is stored on the Application Node and can be found in

Linux: /etc/sudoers

If FlexFrame itself needs extensions these are made in our standard images.



If there are any requirements to extent the configuration at customer site you can make

these extensions in the corresponding configuration files. We strongly recommend to

adjust the intended modifications with professional service support because in the worst

case your system will not work correctly furthermore.

To make these modifications persistent in Linux you can modify the root image directly

without a maintenance cycle. But with each update of FlexFrame you need to modify your

image again.

Content of FlexFrame configuration file on Linux:

# sudoers for FlexFrame Application Nodes

Command definitions

Cmd_Alias SERVICE = /FlexFrame/scripts/ff_service.sh,

/FlexFrame/scripts/view_hosts

Cmd_Alias SAN = /FlexFrame/scripts/ff_san_*

This configuration means, all members of the operating system group sapsys are

allowed to execute the script named ff_service.sh and view_*. Members of oper,

sdba or dba just have the permission to execute the scripts ff_san_* additionally

(although currently these are not supported any more). Entering a password is not

needed.

10.13.2.3 Starting and Stopping SAP Services from Control Node using RBAC

For customers that are primarily working as RBAC enabled users on the Control Node,

there is a possibility to remotely call the ff_service.sh script without additional

configuration of ssh public key authorization for <sid>adm users or the need to login to an

Application Node as <sid>adm using a password.

On the Control Node, call the ff_service_an.sh script as follows:

cn1:~ # ff_service_an.sh <node> <cmdline>

The <cmdline> string will be directly passed to a ff_service.sh command on the specified

Application Node.

Example:

cn1:~ # ff_service_an.sh an100 –a start –t db –s TST

This command will will start the “TST” database on Application Node “an100”.

Internally, it performs these steps:

- Check permissions utilizing RBAC



- Switch to root if user has appropriate rights

- Calls “ssh an100 ff_service.sh -a start -t db -s TST”

- Returns the return code of ff_service.sh or 255 if an ssh error occured

10.13.2.4 SAP Service Scripts

All Services (except project specific ones) are started by:

ff_service.sh { --sid | –s } SID> \

{ --type | -t } <type> \

{ --action | –a } <action> \

[ { --instance | -I } <nr> ] \

[ { --user | –u } <user> ] \

[ { --credential | -c } <password> ] \

[ { --nodeid | -N } <node_id ] \

[ { --replid | -r } <replication_id ] \

[ --debug | -d ]

[ --help | -h ]

[ --version | -v ]

The call parameters are:

{ --instance | -i} <nr>

Distinction of several similar instances of a service type of an SID; 2 digits,

numerical.

{ --sid | -s } <SID>

System ID (SID),.

{ --action | -a } <action>

The action to be performed with the application or service. The actions are start,

stop, restart, status, cleanup, watch, nowatch, istart,istop,

setcfg and resetcfg.

{ --type | -t } <type>

Distinction of the different service types.

ascs (classic, message server)

app (classic, application server, ABAP)

bobj (BOBJ - Business Objects)

ci (classic, central instance, ABAP)



pai (classic, primary application instance, ABAP, replaces ci

as of SAP 7.4)

cms (CMS - Content Server, CMS – Cache Server)

db (Database)

ers (classic, enqueue replicated server)

j (classic, application server, Java,; also used for jc if version ge

SAP 7.1)

jc (classic, central instance, Java)

lc (Live Cache)

hdb (HANA database)

mds (MDM server)

mdss (MDM syndication server)

mdis (MDM import server)

scs (classic, message server)

smd (SMD -Solution Manager Diagnostics)

trx (TREX – Ssearch and Classification)

wd (WebDispatcher)

--debug | -d

Switch on the debug mode.

If you need the Debuglog from requests done by FlexFrame Agents you create a

file by ‘touch /tmp/switch_on_debug.<SID>’ for the SID you want to debug. All

requests for ‘SID’ store a Debugfile in /FlexFrame/scripts/log. Please do not forget

to delete the file and the Debuglogs if you will not need Debuglogs furthermore.

{ --user | -u } <user>

To get the state of BOBJ 4.0 an authentification is needed. <user> is the login user

for CMC specified in installation procedure. Default user is ‘Administrator’.

With Sybase ASE this option is used to specify the Sybase Administration User.

Default is ‘sapsa’.

{--credential | -c } <password>

Is the password of <user> used for authentification to access BOBJ.



With Sybase ASE this option is used to specifiy the password of the Sybase

Administration User.

{ --nodeid | -N } <nodeid>

specifies the nodeid of the HANA database service which should be started.

Default is set to ‘001’.

{ --replid | -r } <replication id>

specifies the replication id of a HANA System Replication configuration

--help | -h

Shows the usage.

--version | -v

shows the script’s version

10.13.2.5 SAP Service Script Actions

In the following, the SAP service script actions are described:

start

This checks whether the application or service in question is already running. If it is,

it is not restarted. It also checks whether required applications or services are

running.

The required virtual IP addresses are assigned to the relevant interfaces for the

Client LAN and Server LAN (ifconfig <ifc> <ip-adr> netmask <netmask>

up). The application is started and the service control file is written.

Some services are dependent on already started other services. As far as possible

FlexFrame tries to give a hint about such dependencies. Some examples for

dependent services are CI (depends on ASCS) or WebDispatcher (depends on

message server of other SAP service).

stop

This checks whether the application or service in question is running. If it is running,

it is stopped.

The application is terminated, the service control file is deleted and the virtual IP

addresses are separated again from the interfaces (ifconfig <ifc> down).



status

This checks the logical status of the application or service. The functional availability

is not tested.

With BOBJ 4.0 (SBOP) the request to get the state by using

‘/usr/sap/<SID>/…/ccm.sh’ requires an authentification

restart

This merges the actions stop, cleanup and start in one call; restart is

intended for restarting a malfunctioning application.

cleanup

This kills application processes that are still running and deletes occupied resources

such as shared memory, semaphores and the message queue.

Note that this action may only be performed after an attempt to stop the application

has failed.

nowatch

This removes the application from monitoring by the high-availability software

FlexFrame Agent without the application having to be restarted. The application itself

retains its current status. The default status after application startup is 'watch'.

watch

This includes the application again in monitoring by the high-availability software

(FlexFrame Agents) without having to restart the application. The application itself

retains its current status.

istart

Start the service interfaces. You do not need to make a distinction between server

and client lan. The script itself selects the necessary interfaces.

This action code cannot be used with type ‘ERS’ because it does not have an own

IP in client or server lan during operation.

istop

Stop the service interfaces.

This action code cannot be used with type ‘ERS’ because it does not have an own

IP in client or server lan during operation.

setcfg

Change only the hardware and OS settings as defined for SAP HANA for the

current Application node. This is useful to setup a system for running benchmarks

without a real HANA database installation.



sesetcfg

Resets the changes of hardware and OS settings

10.13.2.6 SAP Service Script logging

For diagnostic issues SAP service script writes different levels of logging information

standard logging is always written to stdout

/FlexFrame/scripts/log/sapservice.log logs each call of ff_service.sh and its

exitcode. You should reorganize it periodically to get rid of old entries.

/FlexFrame/scripts/log/sap<type>_<SID>_<action>.log logs each call of

ff_service.sh and its exitcode. Additionally the messages written to stdout are

shown, too. A new request overwrites an already existing file for the same SID,

service and action.

/FlexFrame/scripts/log/sapservice.<pid>.debug.<SID>.<type>.<inr>.<action>

contains an extended debuglog if option ‘-d’ is set by ff_service.sh.

10.13.2.7 SAP Service Script User Exits

The FlexFrame script for SAP Services (ff_service.sh) provide a user exit as a shell

script /FlexFrame/scripts/user_script. If this user_script exist, the

FlexFrame script will call it twice, 1st in the beginning, 2

nd at the end of the start-/stop-

scripts. This means two phases, a pre-phase and a post-phase. The pre-phase will be

run before any action, the post-phase will run after all action. Because of plausibility

errors while executing the FlexFrame start-/stop-script, it is possible that the post-phase

will be omitted.

A sample user_script is delivered as user_script.TEMPLATE.

With FlexFrame 5.0 new service types were introduced. Those new service

types have to be added to customer specific user script. The name is built by

the rule ‘sap<service>’, e.g. ‘sapsmd’, ‘saptrx’. Please refer to template file

available in directory /FlexFrame/scripts.

To achieve a pool dependent function of user_script, put the script into the pool image

and create a symbolic link to it in FlexFrame/scripts. Seen from the Control Node:

Control1: # ln -s

/FlexFrame/pooldata/config/scripts_config/user_script/FlexFrame/scripts/user_script



If no pool dependent function is needed, the script user_script can be located directly

in /FlexFrame/scripts.

10.13.2.8 SAP service script ff_smd_agent.sh

With support of SMD 7.3SP02 we introduced a new script which is called internally and

by FlexFrame Agents. The script just selects SMD Agent instance information from

configuration file ‘agents-on-the-fly.config’ and calls ff_service.sh internally. All other

users have to use ‘ff_service.sh’ as before to start SMD instances.

10.13.3 Return Code of the SAP Service Script

The SAP service scripts issue a return code (exit code). The meaning of this code can be

looked up in the sapservice_functions file:

#================================================================

# common exit codes for service scripts

#================================================================

no_error=0 # no error

wrong_parameter_count=1 # Bit 0, wrong number of parameters

plausibility_error=2 # Bit 1, plausibility error

interface_server_lan_error=4 # Bit 2, error at server lan interface up/down

interface_client_lan_error=8 # Bit 3, error at client lan interface up/down

service_start_stop_error=16 # Bit 4, error at service start/stop/status/...

any_error=32 # Bit 5, any other error

san_mount_error=64 # Bit 6, san mount error

user_script_error=128 # Bit 7, user_script error

For actions like start, stop, nowatch and watch, the exit code should be no_error

normally. For the action status, the exit code is dependent from various factors:

The installation check against the SAP service may have failed

(plausibility_error)

the SAP service is possibly not running or not running on this host (any_error)

the SAP service is running on this host (no_error).

In addition to these exit codes, the SAP service scripts print out messages as described

in the section “Start/Stop Script Errors” on page 335.

10.13.4 Starting and Stopping Multiple SAP Services

The following scripts are provided for starting and stopping multiple applications and

services:

Script name Application Place of execution



Script name Application Place of execution

start_all_sapservices Initial start of all configured

applications

Only on a Control

Node

stop_all_sapservices Stopping all running

applications

Only on a Control

Node with option

<poolname>.

stop_all_sapservices_SID Stopping all running

applications of one SID

Only on a Control

Node

stop_all_sapservices_local Stopping all running

applications on the local

node

Only on an

Application Node

Ff_startstop_services.sh Starting/stopping

applications at predefined

locations

Only on a Control

Node

Call syntax for pool-dependent scripts:

stop_all_sapservices [<pool name>]

stop_all_sapservices_SID <sid> [<pool name>]

Functionality of the SAP service scripts for multiple services:

start_all_sapservices

Initial start script for customer defined sap service landscapes. This can be the

whole FlexFrame landscape, a pool, a SAP system landscape with a productive

system, a quality assurance system and a development system, or a single SAP

system with database, central instance and application instances. This script can be

duplicated to create various initial start scripts with different names, e.g.

start_all_sapservices_pool2. A sample start_all_sapservices script is

delivered as start_all_sapservices.TEMPLATE.

stop_all_sapservices

Shutdown script for all running sap services in a dedicated pool. The advantage of

this script is that sap services have not to be stopped one-by-one.

On Control Node you have to specify the pool with the services which should be

stopped. On Application Node no additional option (<pool name>) is allowed.

stop_all_sapservices_SID

Shutdown script for a singe SAP system (one SID) with all belonging sap services

like application instances, central instances und database instance in the right order.

stop_all_sapservices_local



Shutdown script for all running sap services on one Application Node. This script will

be integrated via run levels. The information base for these

stop_all_sapservices* scripts are the /FlexFrame/scripts/log/*_host

files.

ff_startstop_services.sh

Starting and Stopping of applications concerning a configuration file. A configuration

file describes where a service should be startet. This could be helpful after a

downtime if you want to start the services on servers the instances run before. This

script is only available in the toolbox supplied with FlexFrame Service CD. Please

take care of the Disclaimer on Service CD because the script is just a helper, not an

official FlexFrame script.

10.13.5 Removing an Application from Monitoring by FlexFrame Agents

If the applications or services are started with the scripts for virtualization, they are

monitored by the FlexFrame Agents. If you do not want this, be it for tests, installation or

upgrades, you have to inform the high-availability software of this, using the additional parameter nowatch when the application is started.

Example:

The central instance of BW1 is to be started without monitoring by the high-availability

software:

blade1 # ff_service.sh -s bw1 –t ci –a nowatch

blade1 # ff_service.sh –s bw1 –t ci –a start

or from a Control Node:

control1 # ssh blade1 ff_service.sh -s bw1 –t ci –a nowatch

control1 # ssh blade1 ff_service.sh –s bw1 –t ci –a start

If a running application is to be excluded from monitoring by the high-availability software

without being restarted, this is possible, using the nowatch option. The application then

retains its current status.

Example:

The central instance of BW1 is to be excluded from monitoring by the high-availability

software while running:

blade1 # ff_service.sh -s bw1 –t ci –a nowatch

Alternatively, call the following from a Control Node:



control1 # ssh blade1 ff_service.sh -s bw1 –t ci –a nowatch

If a running application is to be included (again) into monitoring by the high-availability software without being restarted, this is possible using the watch option. The application

then retains its current status.

Example:

The central instance of BW1 is to be included into monitoring by the high-availability

software while running:

blade1 # ff_service.sh -s bw1 –t ci –a watch

or from a Control Node:

control1 # ssh blade1 ff_service.sh -s bw1 –t ci –a watch

10.13.5.1 Stopping and Starting an Application for Upgrades Using r3up

The r3up upgrade control program starts and stops the central instance or application

server in various upgrade phases, using the conventional startsap and stopsap. The

start and stop scripts for visualization are therefore left out.

Suitable measures must be taken to ensure that the virtual host name for <sid>adm is

always available and that the interfaces are supplied with the correct IP addresses. In

addition, the application has to be removed from monitoring by the high-availability

software.

Note that for the following solution steps, the application must have been started

correctly beforehand with start and stop scripts for virtualization, so that the

matching virtual host name is set by default in the $HOME/hostname_default

script.

Remove the application from monitoring by the high-availability software and run it.

r3up can now shut down and start the application directly with stopsap and

startsap; the IP addresses remain assigned to the interfaces and the host name

is retained for <sid>adm without any changes on the correct virtual host name.

Include the application again into monitoring by the high-availability software after

the upgrade.

10.13.6 Service Switchover

There is no command line tool to enforce a direct switchover. To switch over an SAP

service like a Central Instance, Database Instance, Application Instance etc., it is required



to first stop this SAP service on the actual Application Node and to start this SAP service

on the destination Application Node thereafter.

Examples:

The central instance of OL4 from pool pool1 is running on klinge4 and should

switchover to klinge5.

First, check whether the Central Instance OL4 is running on klinge4:

For a first overview to see which SAP service should be running on which Application Node in the pool pool1, run

control1 # view_hosts pool1

app_44_ol4 should be running on klinge4

app_56_ml4 should be running on Baby_4

ci_ml4 should be running on RX300-01

ci_ol4 should be running on klinge1

ci_osm should be running on Baby_3

db_osm should be running on Baby_3

db_ml4 should be running on RX300-01

db_ol4 should be running on klinge4

The output should be means that the script view_hosts does not really check the

state of the SAP services. It only checks the FlexFrame/script/log/*_host files

which are created immediately after a successfully start of the corresponding SAP

service.

For a detailed status information you can use:

control1 # ssh klinge4 ff_service.sh -s ol4 –t ci –a status

The exit code from our SAP service script will be “0” in this case. For any other errors

while executing our SAP service script, it will return an exit code not equal “0”.

cn1:~ # / # ssh klinge1 ff_service.sh -s ol4 –t ci –a status

INFO: Version of ff_service.sh: 1.20.2.21

INFO: …….

INFO: Service ci for SID CL4 should be running on this host

INFO: ….

INFO: server_lan interface cis02-se 172.20.20.129 is (already) up

INFO: Exit code: 0 (no error)

Stop the running Central Instance OL4 on klinge4:

control1 # ssh klinge4 ff_service.sh -s ol4 –t ci –a stop

And now start the stopped Central Instance OL4 on the new destination klinge5:



control1 # ssh klinge5 ff_service.sh -s ol4 –t ci –a start

The manual switch over should now be complete.

10.13.7 Use ServicePings from FlexFrame Agents

With newer SAP releases it could be insufficient of FlexFrame Agents just check the

process list of a host to decide if o service is running or not.

Now the FlexFrame Agents offers the possibility to check it using SAP executables like

’sapcontrol … GetProcessList’. To activate that feature you

Switch to /opt/myAMC/scripts

Remove ‘exit 99’ from ServicePings*.sh

Activate ‘ServicePing’ for each pool you want to use it by the FlexFrame Agents

WebGUI

Restart ‘ff_manage’ either bei ‘hb_gui’ or by ‘ff_ha_cmd.sh’


11 Software and Hardware Upgrade and Maintenance

11.1 Upgrading the Entire FlexFrame Landscape

Upgrading the entire FlexFrame to the 1.1A release depends on the release you are

starting from:

Please note that only a „FlexFrame® Orchestrator“ V1.0A can be used for upgrading to

„FlexFrame® Orchestrator“ V1.1A. To upgrade a „FlexFrame® Orchestrator“ V1.0A”

please refer to the so called “Upgrade Guide”: “Upgrading FlexFrame Orchestrator 1.0A

to 1.1A”.

Older FlexFrame versions, for example „FlexFrame® for SAP®“ must be upgraded to

“FlexFrame® Orchestrator” V1.0A first. See the document “Upgrading FlexFrame® 5.1A

or 5.2A to 5.3A/1.0A”.

11.2 Software Upgrade on the Control Node

This section describes FlexFrame software installation and upgrades on the Control

Node. For upgrade of the Control Node see previous chapter. Control Nodes are not

designed to run any third party software (except saprouter).

Since FlexFrame 5.1A most of the installation of FlexFrame specific software (products)

on top of the underlying SLES takes place during first boot of the Control Node. You

know this from the “Quick Installation” section in the “Installation of a FlexFrame

Environment” manual.

That way the installation of these products moved from Fujitsu Mastering Site to

customer’s site. The advantage is that all those products are now available at your site

which might become important in case of needs to reinstall them.

In addition this means that these software products are no longer inside the pre-built

Control Node image.

To find out more details about the performed installation of these software products you

can inspect the well-known log file directory /var/log/FlexFrame. The very first

log_<control node name> file contains the complete installation logs.

To find out details about the software products before the actual installation you may

inspect the Service CD. Insert the Service CD and run the following commands:

cn1:~ # mount /media/dvd

cn1:~ # cd /media/dvd/CONOS

cn1:/media/dvd/CONOS # ./installer.sh –l

This will list (-l) the products and product versions.

Software and Hardware Upgrade and Maintenance


Do not run this installer in any other than the mentioned list (-l) mode unless

instructed to do so!

Running in install (-i) or upgrade (-u) mode may damage your FlexFrame.

New versions of these mentioned software products will be deployed with new Service

CDs. These will contain instructions how to install new versions if needed (see also

chapter “11.2.1 FlexFrame Control Node Software”

You may want to install other software products (i.e. Third Party Products) beside the

mentioned from the Service CD. Remember (as stated in chapter 2.1):

The FlexFrame Control Nodes are seen as an appliance

Installing third party software on Control Nodes or Application Nodes may cause

functional restrictions and other malfunctions of the FlexFrame system software.

Fujitsu shall have no liability whatsoever whether of a direct, indirect or

consequential nature with regard to damages caused by the third party software

or its erroneous installation.

SLES patches may have effects on the functionality of FlexFrame Control Node

and Application Node images. So be careful with installing any SLES patches to

the FlexFrame Nodes.

If you install or upgrade software products on a project specific base you must use the ff_projectinfo.sh command to document this change!

See chapter 11.10 Project specific changes for details on this command.

Nevertheless we will describe here the general approach to update/install software on the

Control Nodes, on a few examples. In your special case this approach can differ to the

ones described here.

11.2.1 FlexFrame Control Node Software

As stated in the previous chapter, new versions of the FlexFrame Control Node products

will be deployed with a new Service CD.

If needed or instructed by Fujitsu support an update of these components is possible by

launching the respective installer command: Basically you just have to change the

directory to the product location and start the installer. If you need for example to (re-)

install the “example” product insert the Service CD and run the following commands:



cn1:~ # mount /media/dvd

cn1:~ # cd /media/dvd/CONOS/example

cn1:/media/dvd/CONOS/example # ./installer.sh –ivf

Run these installers only if needed or instructed by Fujitsu support.

Normally there is a README or other installation instructions available which

guides through the process.

All of these product installer cover the Control Node only where the Service CD has

been mounted. You will have to run them on the other Control Node as well. The

mentioned README or installation instructions will point out to that fact

11.2.2 Service Packs

It is not allowed to install any Service Packs on the Control Node manually. The only way

to obtain a new service pack for the Control Node is to install new Conttrol Node image

delivered from Fujitsu. Service Packs contain a lot of changes that can have influences to

the FlexFrame functionality. Service Packs can also contain initial configuration files that

will substitute the FlexFrame configurations.

11.2.3 Updating/Installing a New Linux Kernel

The following sections describe how to install a new Linux kernel for Control Nodes in

FlexFrame. This only has to be done if instructed to do so by Fujitsu support.

The process will be described on examples accomplished on a development system. The

version numbers here in this document are not generally binding; they must be replaced

with the correct information for your system.

This approach describes the kernel installation without the recommended Server Start

CD.

11.2.3.1 Software Stage

To install and update any of the software packages in FlexFrame, it is useful to mount

your NAS system or jukebox or any other software stage on a local mount point.

For example, your software stage is /FlexFrame/volFF/FlexFrame/stage/ and

the new kernel to install is 2.6.16.60-0.63.1.

Copy the delivered software packages to the software stage:

cn1:~ # cd /FlexFrame/volFF/FlexFrame/stage

cn1:/FlexFrame/volFF/FlexFrame/stage # mkdir -p CN/kernel

Kernel:



cn1:~ # cp <somewhere>/kernel-smp-2.6.16.60-0.63.1.x86_64.rpm \

/FlexFrame/volFF/FlexFrame/stage/CN/kernel

Kernel kdump:

cn1:~ # cp <somewhere>/kernel-kdump-2.6.16.60-0.63.1.x86_64.rpm \


Kernel source:

cn1:~ # cp <somewhere>/kernel-source-2.6.16.60-0.63.1.x86_64.rpm \


With SLES 10 (Service Pack 1 and later) Novell has introduced the Partner

Linux Driver Process (PLDP). There is no need to download drivers for the

update kernel. The existing drives are reused for the new kernel.

11.2.3.2 Install the New Kernel

To install the new kernel, execute the following command:

cn1:~ # cd /FlexFrame/volFF/FlexFrame/stage/CN/kernel

cn1:~ # pwd


cn1:~ # rpm -ivh kernel-smp-2.6.16.60-0.63.1.x86_64.rpm

cn1:~ # rpm –ivh kernel-kdump-2.6.16.60-0.63.1.x86_64.rpm

cn1:~ # rpm –ivh kernel-source-2.6.16.60-0.63.1.x86_64.rpm

11.2.3.3 Reboot the Control Node

Reboot the Control Node and test the functionality of the updated Control Node, Linux-

HA, network connectivity, and so on.

The uname -r command should display the new kernel version:

cn1:~ # uname -r

2.6.16.60-0.63.1-smp

11.2.3.4 Partner Linux Driver Process

The Partner Linux Driver Process (PLDP) replaces the traditional method of driver

provisioning for SLES 10 (Service Pack 1 and later). The most benefit of this process is

the support for reusing existing drivers for errata kernels. There is no need to download

drivers for those kernels.



Nevertheless if updated driver packages are released (e.g. a new version) these

packages can be installed on Control Node or Application Nodes like an “installable

patch”.

Depending on a special configuration the document “Novell Partner Linux Driver Process

for PRIMERGY servers” describes the update procedure in detail. This document is

available at URL:

http://ts.fujitsu.com/support/downloads.html

11.2.4 Updating RPM Packages on an Control Node

To update the SLES RPM packages you can use the original Novell tools to perform an

online update.

11.2.4.1 Setup a patch server

It is not recommended to connect your FlexFrame environment directly to the Internet to

download patches or updates. Therefore it is advisable to set up an own patch server,

which provides the necessary update packages. See: http://www.novell.com/de-

de/linux/smt/ and http://www.novell.com/communities/files/appnote-2009-12-10-v1_7.pdf

for hints.

Provided that this patch server is accessible for the Control Node a normal online update

with YaST can be performed.

11.2.4.2 Connect the patch server to the FlexFrame environment

There are several possibilities to connect your local patch server to your FlexFrame

environment. It’s up to your internal network and security policy, which way is the right

way for you.

Some of them are described in http://www.novell.com/communities/files/appnote-2009-

12-10-v1_7.pdf

It is possible to setup an ‘External Connectivity’ on the Control Node to connect your

patch server to the FlexFrame environment. Or you can copy the data from your patch

server to an ‘USB disk’ and connect this disk directly to the Contorl Node. Or copy the

data to the local disk of the Control Node or to a directory on the filer and share (NFS,

SMB/CIFS, HTTP(S) or FTP) this directory for the Control Node.

11.2.4.3 Prepare the Online Update

To prepare the online update you have to define the installation source. That’s the way,

how you provide the update packages, i.e. your patch server

On a SLES11 Control Node:

YaST Software Software Repositories Add Media Type




See Section 9.3 “Managing Software Repositories and Services”

http://www.novell.com/documentation/sles11/book_sle_deployment/data/sec_y2_sw_inst

source.html#sec_y2_sw_instsource for more information.

The Media Type strongly depends on the way you provide the update packages (see

section 11.5.9.5.2 ”Connect the patch server to the FlexFrame environment” on page

320).

11.2.4.4 Perform the Online Update

Now you can perform the online update:

YaST Software Online Update

For SLES11 see section 1.0 “YaST Online Update” from

http://www.novell.com/documentation/sles11/book_sle_admin/data/cha_onlineupdate_yo

u.html#cha_onlineupdate_you for more information.

If the online update has installed a new kernel, it is necessary to reboot the

Control Node.

11.2.5 Backup/Restore of FlexFrame Control Nodes

To back up or restore Control Nodes use the ff_cn_backup.sh script.

Synopsis

ff_cn_backup.sh [-dr] [-f <file name>] [-l <list>]

Options

-d

Debug

-r

Restore

-f <file name>

Use file name for restore

-l <list>

Blank separated list of files (quoted)



11.2.5.1 Backup of a Control Node

Most of the files on a Control Node are shipped with the installation DVD. Necessary data

can be backed-up using ff_cn_backup.sh.

The default behavior (without any options) is to back up the configuration files in a zipped tar file at /FlexFrame/volFF/FlexFrame/backup/<name_of_cn><date>.tgz.

Additional files can be backed-up using the -l option.

The Control Nodes are not intended to have additional software installed on.

Therefore backup in the regular sense is not required. Installation from the DVD

is much faster than any restore.

11.2.5.2 Restore of a Control Node

If needed a restore of a Control Node can be done quite fast. The restore is basically a

new installation of the node, using FlexFrame configuration data provided by the

remaining, still running Control Node, and the existing Backups of the Control Node.

To do this the following media is needed:

Control Node DVD (labeled “FlexFrame V1.1A00 Orchestrator Control Node –

Images”)

“Service CD” (labeled “FlexFrame V1.1A00 Orchestrator Service and

Information”)

USB stick (will host the configuration file)

Make sure to use the same Control Node DVD and Service CD as before! Refer to the

remaining, still running Control Node to check the FlexFrame release.

In detail, follow these steps to restore a Control Node. In this example the remaining, still running Control Node is called control1. The Control Node to be restored is called

control2.

1. Login to the remaining, still running Control Node (control1):

2. Create a configuration file of the current (running) configuration:

control1:~# java –jar /opt/FlexFrame/MgmtTool/MgmtTool.jar

Choose “File” – “Open” – “Local LDAP Connection” to read the FlexFrame

Configuration from LDAP. Then validate configuration by help of “Tools” – “Validate Configuration”. Finally write the configuration file ff_config.xml by help of “File” –

“Save For Installation” to the USB stick.

3. If applicable, copy the SFG (‘Request for Special Release’) file (/opt/FlexFrame/etc/ff_hardware_sfg.xml) to the the USB stick, too.



4. Boot the Control Node to be restored (in this example: control2) with the Control

Node DVD and choose “Control Node Installation”

5. Provide the configuration file ff_config.xml (from step 2) with the USB stick.

6. Insert the Service CD and continue the installation as described in the “Installation of

a FlexFrame Environment” manual. Again: You need to use use same Control Node

DVD and Service CD as before.

7. If the passwords are requested, enter the current passwords.

8. Select the correct Node (“first” or “second”) for this Control Node installation; in this

example: “second”.

9. Once the system is booted do not follow the installation instructions from the

“Installation of a FlexFrame Environment” manual. Invoke the following commands

instead:

First run the ff_init_conf.pl command:

control2:~# mkdir –p /opt/FlexFrame/init_conf

control2:~# ff_init_conf.pl --upgrade

This will create several files in the /opt/FlexFrame/init_conf directory.

10. Then execute the following commands to create the directory structure and mount

needed file systems:

control2:~# cp /opt/FlexFrame/init_conf/fstab /etc

control2:~# cp /opt/FlexFrame/init_conf/hosts /etc

control2:~# mkdir -p $(cat /opt/FlexFrame/init_conf/mountpoints)

control2:~# mount –a

Before continuing check if everything has been mounted:

control2:~# for mp in \

$(cat /opt/FlexFrame/init_conf/mountpoints); do df $mp; done

11. Now restore the original contents of the Control Node's configuration files. Use

ff_cn_backup.sh with the option -r and optionally -f <file name> to do this.

The backups are located in /FlexFrame/volFF/FlexFrame/backup and, by

default, the restore will pick the latest backup it can find. If this should not be the

appropriate backup file, use the option -f <file name>:

control2:~# ff_cn_backup.sh –r [–f <file name>]



12. Add additional security related changes to some config files using the command:

control2:~# ff_enh_sec.sh

13. Update the rpm database regarding the FlexFrame Agents packages

This is needed since the FlexFrame Agents packages (rpms) reside on the shared

storage already. It is sufficient to update the rpm database. All of the missing

packages are available on the respective Service DVD. Insert the Service DVD,

mount it and update the database:

control2:~# rpm –U --justdb /media/dvd/FA_Suite/myAMC_FA/*.rpm

14. Start FlexFrame Agents next and activate them permanently:

control2:~# insserv myAMC.FA_LogAgent

control2:~# /etc/init.d/myAMC.FA_LogAgent start

control2:~# insserv myAMC.FA_FrameAgent

control2:~# /etc/init.d/myAMC.FA_FrameAgent start

15. Compare the two Control Nodes using the command:

control2:~# ff_cn_cmp.sh

If there are other differences in files listed, copy them over from the other Control

Node using:

control2:~# scp -p control1:/<path> /<path>

16. Reboot the just installed Control Node:

control2:~# init 6

17. Start the Linux HA software stack:

control2:~# /etc/init.d/openais start

After the command returned, it will take a while until the complete Linux HA software

stack is up and running. This can be verified with the crm_mon command; both

Nodes should be shown as online and the ldap master or replica should be switched over to the restored Control Node. An ff_ha_cmd.sh cleanup of some resources

might be necessary in some cases, especially if the old Control Node was hard

powered off.

18. If the Linux HA cluster works as expected, activate it permanently:

control2:~# insserv openais



11.3 Maintenance of the Control Node - Hardware

11.3.1 Exchanging a Control Node

If a Control Node must be replaced for any reason, attention has to be paid to the fact

that the replacing hardware has to be the same as the replaced component. Otherwise,

problems with missing Linux drivers might occur. Problems due to hard disk failures

should usually not occur, since the Control Node's hard disks are mirrored through

hardware-based RAID-1.

11.3.1.1 Hardware Failed, Hard Disk and Installed OS are not Affected

In case that other hardware components than one of the HDs are broken (e.g. memory,

CPU, etc.) and the installed OS image is still operational, these hardware components

have to be replaced with equivalent ones. Even the entire hardware can be replaced -

excluding the HDs. The HDs can be removed from the existing hardware, plugged into

the new hardware and booted, after checking the RAID controller's BIOS for hardware

RAID settings being enabled. Even if one of the hard disks is broken, it can be replaced

with the same model and synchronized with the working hard disk through the RAID

controller's BIOS.

In this case the Control Node hardware must be replaced by the same model with the

same disk controller and the same network interface controllers. The approach is very

simple, plug in your hard disks in the new Control Node in the same order as the old one,

power on the Control Node, enter the RAID controller BIOS and check the parameters;

hardware raid should be enabled as before.

See also the manual “Installation of a FlexFrame Environment“, chapter “Control Node

Host RAID Configuration”.

11.3.1.2 One Hard Disk is Defect, the Other One is Undamaged

The Control Nodes OS installation must be on a RAID-1 disk array.

Replace the defect hard disk with the same model and synchronize it with the

undamaged one via the RAID controller BIOS. This approach is dependent from the

RAID controller model and its BIOS.

11.3.1.3 The Control Nodes OS is Damaged

If, for any reason, the operating system of a Control Node should be damaged, it has to

be installed anew with the original Control Node installation DVD. The configuration has

to be recovered from a previous backup. See section “Backup/Restore of FlexFrame

Control Nodes” on page 273.



11.3.2 Replacing a Network Card – Control Node

If the motherboard or the PCI network card in a Control Node is replaced, no further

actions are required since there is no reference to the replaced MAC addresses.

11.4 Maintenance of the virtual Control Center

This chapter describes different scenarios of failure within the virtual Control Center. The

first scenario described in chapter 11.4.1 deals with the failure of a Hypervisor Node. All

necessary steps to get information in order to switch over SAP instances are described.

The second scenario (chapter 11.4.2) deals with the restore of a damaged virtual Control

Node. The third scenario (chapter 11.4.3) deals with the restore of a damaged Hypervisor

Node.

11.4.1 Site failover within virtual Control Center

This scenario describes the consequences if a Hypervisor Node hosting a virtual Control

Node does fail. If the Hypervisor Nodes of the virtual Control Center are hosting virtual

Application Nodes in addition, the failure of one of the Hypervisor Nodes can be

compared with a site failover within a FFO landscape consisting of physical Control and

Application Nodes. Within this chapter the different steps are described in order to handle

the situation of failure. The whole scenario is described with the naming conventions of

one of our development machines.

11.4.1.1 The naming conventions

KVM Hypervisor Nodes hosting virtual Control Center:

kvm-vcn1

kvm-vcn2

Virtual Control Nodes:

cn1 (hosted on kvm-vcn1)


Virtual Application Nodes:

van1-kvm1 (hosted on kvm-vcn1)

o Pool: sles11


o Pool: dp2

o Group: sap




o Pool: sles11


o Pool: dp2

o Group: sap

SIDs:

CL1 (running on van3-kvm1)

11.4.1.2 Scenario of sane KVM Hypervisor Nodes

The following figure shows the situation with both Hypervisor Nodes up and running:

0 Copyright 2012 FUJITSUFTS INTERNAL

KVMH-Ausfall im vCC cont.

Copyright 2012 FUJITSU

kvm-vcn2

van2-vcn2cn2

kvm-vcn1

van1-vcn1cn1FA_CtrlAgent

van4-vcn2van3-vcn1

SAP Sid:

CL1

We can get informed about:

Cluster state of virtual Control Center

SAP services

Configuration of Virtual Machines

by the following commands.

The cluster state of the virtual Control Center can be displayed by:

cn1:~# crm_mon -1 -r -f -n

…



Current DC: cn1 - partition with quorum

…

Node cn1: online

myAMC.FA_Messenger (lsb:ff_ha_myAMC.MessengerSrv): Started

ldap_master (ocf::fsc:ff_ha_ldap): Started

…

myAMC.FA_CtrlAgent (lsb:ff_ha_myAMC.CtrlAgent): Started

stonith_vm_CN2 (stonith:external/libvirt): Started

Node cn2: online

ldap_replica (ocf::fsc:ff_ha_ldap): Started

…

From the output we can see that both Control Nodes are online.

The following command gives information of the system status concerning the SAP

services:

cn1:~# ff_list_services.sh

Pool Group Hostname SID Type Id Node State

dp2 sap van3-kvm1 CL1 db running

dp2 sap van3-kvm1 CL1 ascs 01 running

dp2 sap van3-kvm1 CL1 scs 02 running

dp2 sap van3-kvm1 CL1 pai 03 running

dp2 sap van3-kvm1 CL1 ers 06 running

dp2 sap van3-kvm1 CL1 ers 07 running

By the following command, system information concerning the Virtual Machines hosted

by FFO Hypervisor Nodes can be derived. With operation mode list, all virtual

machines that use the specified hypervisor are shown, according to the FFO

configuration. An excerpt from our example configuration is shown hereafter:

cn1:~# ff_hn_adm.pl --op list --name kvm-vcn1

Configuration details of KVM node kvm-vcn1

Hardware

System: RX300S8

10GBit: No

Shut.Facil.: IPMI kvm-vcn1-co (192.168.73.177)

...

Virtual Machines



van1-kvm1 AN Off 4 8192 dp1 sles11



van3-kvm1 AN On 4 8192 dp2 sap

cn1 CN On 4 8192

-------------------------------------

Total 12 24576

11.4.1.3 KVM Hypervisor Node kvm-vcn1 fails (manual switch over)

The following figure shows the situation when Hypervisor Node kvm-vcn1 does fail:

0 Copyright 2012 FUJITSUFTS INTERNAL

KVMH-Ausfall im vCC cont.

Copyright 2012 FUJITSU

kvm-vcn2

van2-vcn2cn2

kvm-vcn1

van1-vcn1cn1FA_CtrlAgent

van4-vcn2van3-vcn1

SAP Sid:

CL1

If Hypervisor Node kvm-vcn1 fails, the consequences are:

Virtual Control Node cn1 is no longer running

o FA_CtrlAgent does no longer run

Virtual machine Application Nodes van1-kvm1 and van3-kvm1 are no longer

running

o SAP SID CL1 is no longer running

After Hypervisor Node failure the surviving virtual Control Node (in our example cn2) has

no information of the status of failed virtual Control Node cn1. Within this scenario the

iRMC interface of Hypervisor Node kvm-vcn1 is assumed dead (no power supply). As a

consequence cn2 does not know if kvm-vcn1 is really switched off.



If we call the crm_mon command in this situation, we get the following output:

cn2:~# crm_mon -1 -r -f -n

…

Current DC: cn2 - partition WITHOUT quorum

…

Node cn1: UNCLEAN (offline)



…

Node cn2: online


…

This means that manual operator intervention is needed in order to clear the split brain

situation within the Control Center. This is done by the following command:

cn2:~# crm node clearstate cn1

This command sets the state of node cn1 from UNCLEAN (offline) to offline so that

pacemaker is able to take over the resources from that node

The state of virtual Control Node cn1 has been cleared now and all resources which are

running on cn1 are switched over to cn2. Now the Linux-HA Cluster consists only of one

active virtual Control Node cn2 but the status of the cluster is defined and clean. All

services running before on cn1 are switched over to cn2. This is especially true for the

FA_CtrlAgent.

Let us look at the output of crm_mon after Linux-HA Cluster has switched over all services:

cn2~# crm_mon -1 -r -f -n

…

Current DC: cn2 - partition WITHOUT quorum

…

Node cn1: OFFLINE

Node cn2: online




…

myAMC.FA_CtrlAgent (lsb:ff_ha_myAMC.CtrlAgent): Started



The system administrator has to be aware of the resulting situation and its

consequences:

FA_Ctlr Agent did not run until failover of resources by Linux-HA Cluster

Operator intervention was needed to trigger switchover of resources

Virtual machine Application Nodes (van1-kvm1 and van3-kvm) hosted by failed

hypervisor node kvm-vcn1 did no longer write into livelist

SAP service SID CL1 running on failed virtual machine Application Nodes van3-

kvm1 does no longer run

Restarted FA_CtrlAgent (by default) does not read in the past in live list (this

behavior can be changed by parameter: dT_LivelistRead).

By default (dT_LivelistRead=0) failed/missing virtual machine Application

Nodes and SAP services are neither reported by FA_CtrlAgent nor restarted

automatically.

Manual intervention is needed to restart SAP services on virtual machine Application

Nodes hosted on other Hypervisor Nodes. A problem might be to find out in this situation

which virtual machine Application Nodes are missing?

which SAP services have been running on the failed virtual machine Application

Nodes?

Information about virtual machine Application Nodes that were hosted on failed

Hypervisor Node can be retrieved by the following commands:

cn2~# ff_hn_adm.pl --op list --name kvm-vcn1

Configuration details of KVM node kvm-vcn1

Hardware

System: RX300S8

10GBit: No

…

Hypervisor resources

no information available, because of: Cannot ping IP address 192.168.73.77 of

kvm-vcn1

Virtual Machines



van1-kvm1 AN - 4 8192 dp1 sles11

van3-kvm1 AN - 4 8192 dp2 sap



cn1 CN - 4 8192

-------------------------------------

Total 12 24576

The information is retrieved out of LDAP data base. Of cause there is no longer

information about the state of the virtual machines, because Hypervisor Node kvm-vcn1

is no longer reachable.

The following command is helpful in order to find out which SAP services (SIDs) have

been running on virtual Application Node van3-kvm1 before hypervisor kvm-vcn1 did fail.

cn2~# /FlexFrame/scripts/view_hosts dp2

ascs_cl1 should be running on van3-kvm1

db_cl1 should be running on van3-kvm1

ers_06_cl1 should be running on van3-kvm1

ers_07_cl1 should be running on van3-kvm1

pai_cl1 should be running on van3-kvm1

scs_cl1 should be running on van3-kvm1

The command evaluates logging information written by ff_service.sh when services

are started or stopped.

The command can be called either on Application Node or on Control Node. The output

of view_hosts has the following format:

<service>[instanznr]_<SID> should be running on <name of

application node)

Let’s analyze one line of the output above:

…

db_cl1 should be running on van3-kvm1

In this case the SAP service is: db and the SID is: cl1. Further van3-kvm1 is the name

of the virtual Application Node the SAP service has been started on.

Command view_hosts relies on the fact that start/stop of SAP services is

exclusively done by the corresponding FFO scripts. If you don’t use these scripts, view_hosts will not produce correct results.



Anyway it is a good idea not only to rely on information retrieved in a disaster situation. A

responsible system administrator knows which SAP instances are running on which

application node. He is able to prioritize the importance of SAP instances and is able to

decide where these instances are restarted in case of disaster.

In order to manually restart SAP services it might be necessary to cleanup state

Files in Directory: /FlexFrame/scripts/log. Before starting SAP services it is

necessary to remove the *_host Files of the missing SID instances.

11.4.1.4 KVM Hypervisor Node kvm-vcn1 fails (automatic switch over)

It is possible to automate the switch over of SAP services in case of KVM Hypervisor

failure if the iRMC interface of the Hypervisor Node is still working. In this case the

surviving Control Node cn2 is able to shutdown kvm-vcn1. In this scenario no split brain

situation within the virtual Control Center occurs. Control Node cn2 automatically takes

over all resources, especially it restarts the FA_CtrlAgent. There is no manual

intervention needed on the Control Center, but SAP Applications require further

configuration by setting the following parameter:

dT_LivelistRead =600

Parameter dT_LivelistRead is evaluated when the FA_CtrlAgent is started. The

parameter instructs the Control Agent to look backward into the live list and to

automatically restart failed SAP instances on available spare Application Nodes. The

value of dT_LivelistRead is specified in seconds.

Setting dT_LivelistRead assumes that a sufficient number of spare nodes do exist.

The parameter dT_LivelistRead is pool-specific and and can be configured by the

FA_Config GUI. Within the pool-specific configuration window:

Local->FlexFrame->Pools->Pool Configuration

you have to select:

Autonomy->Timing

Now a window opens that allows configuring the basic timing settings for myAMC

FA_Agents. dT_LivelistRead is one of the parameters that can be configured.

A value of 600 seconds (10 minutes) is a meaningful value because the HA-Cluster

needs some minutes in order to take over resources (i.e. FA_CtrlAgent). The restarted

FA_CtrlAgent now looks back for 600 seconds into live list and will detect the failed virtual

Application Nodes and SAP services. In case of sufficient spare nodes, all failed SAP

services can be automatically restarted.



11.4.2 Restore of damaged virtual Control Node

This scenario is analoguous to “Restore of a physical Control Node” described in chapter

11.2.5.2. Before restore operation the (existing) damaged virtual Control Node has to be

erased. This can be done by calling the virt-manager on Hypervisor Node that hosts

the damaged virtual Control Node. After shutdown and erase of the damaged virtual

Control Node the virtual Control Node has to be created again using the ff_setup_vcn.sh command. You have to provide the correct ff_config.xml file (i.e. on

mounted USB stick) that has been used when originally installing the (now damaged)

virtual Contol Node. The command for creation of virtual Control Node could be in case of

a damaged Control Node cn1:

ff_setup_vcn.sh –r 1 –c /mnt/ff_config.xml

From here on you can follow the steps described in chapter “Restore of a physical

Control Node”.

11.4.3 Restore of damaged Hypervisor Node

This scenario describes the steps that have to be executed in case a Hypervisor Node

hosting a virtual Control Node does crash and needs new installation. The scenario is

described with the following naming conventions:

KVM Hypervisor Node hosting virtual Control Node that needs new installation:

kvm-vcn1

Virtual Control Node:


Virtual Application Node:


1. The hypervisor node needs to be installed from Control Node DVD. You need the

original Control Node DVD and the original ff_config.xml File stored on USB stick as

well.

2. After Hypervisor Node has been installed create the virtual Control Node using the

ff_setup_vcn.sh command and restore the virtual Control Node as described in

the chapter before.

3. Erase kvm-vcn1 from: /root/.ssh/known_hosts on both: cn1 and cn2



4. Now you have to erase all virtual Application Nodes that have been running

(originally) on the Hypervisor Node. In our example this is: van3-kvm1. If you type:

cn1:~ # ff_vm.pl --op list --name van3-kvm1

Virtualization specific configuration details of node van3-kvm1 LDAP view:

Node Type : AN

Type : KVMVM

Hypervisor Node : kvm-vcn1

Number of CPUs : 4

Memory size in MB: 8192

VM created : yes …

you can see that van3-kvm1 is still created.

Virtual Application Node van3-kvm1 can be erased using the following command:

cn1:~ # ff_vm.pl --op destroy --name van3-kvm1 --force

Password:

ERROR: Virtual machine van3-kvm1 not found in virtualization

environment update LDAP

The ERROR message can be ignored.

If you type: ff_vm.pl --op list --name van3-kvm1 again, you will see that the

“VM created” flag is now set to: “no”.

5. Configuration of the virtual Hypervisor Node can now be compled using the following

command:

cn1:~ # ff_hn_adm.pl --op complete-config --name kvm-vcn1 --user root

--pass passwort

6. In order to recreate the virtual Application node van3-kvm1 you can type:

cn1:~ # ff_vm.pl --op create --name van3-kvm1 update LDAP

7. Now you can power on the virtual Application Node using:

cn1:~ # ff_ponpoff.pl --op power-on --name van3-kvm1



In case you exchange the physical Hypervisor Node because it is damaged, the

scenario is the same as described here. For the new Hypervisor node you have

to choose the same IMPI address that was configured with the damaged node

11.5 Maintenance of Application Nodes - Software

11.5.1 Introduction

Maintenance of Linux Application Node Images – often referred to as “the Maintenance

Cycle” – allows modification of the Application Node’s software which is normally hardly

possible owing to the shared image concept.

After a short schematic overview this section describes the following:

installing Application Node images from installation media

creating a Maintanance Image

assigning the Maintenance Image, booting and maintaining

reverting the Maintenance Image

migrating remaining Application Nodes

maintaining use cases

FlexFrame 5.0 introduced a new Application Node Image concept. It is still a shared

image concept as known from earlier FlexFrame releases but was revolutionized

particularly in order to improve the Maintenance of Application Node Images.

For details refer to the section "Setting up Application Nodes" of the "Installation of a

FlexFrame Environment" manual.

By help of the new concept the Maintenance of an Application Node Image is now as

easy as 1-2-3 as you will see in this section.

11.5.2 Schematic Overview

This section gives an idea how the Maintenance Cycle works. Basically this cycle

consists of just three steps:

1. Creating a Maintenance Image

(ff_install_an_linux_images.sh -m ...)

2. Assigning the Maintenance Image, creating boot image, booting the Application

Node and maintaining

(ff_an_adm.pl –op os ...; ff_new_an.sh –n ...)

3. Reverting the Maintenance Image (ff_install_an_linux_images.sh –r ...)



Then all of the Application Nodes can use the maintained image instead of the old one to

boot.

The following figure visualizes the entire process of maintaining of an Application Node

Image i.e. the “Maintenance Cycle”. You will recognize the figure since it has been

introduced in the “Setting up Application Nodes” section of “Installation of a FlexFrame

Environment” manual.

1

`

os/Linux/FSC_1.1A*/

productive mode

generation 1

/FlexFrame/volFF/

Base I

mag

es

root_img and

var_img of Node #n

root_img and

var_img of Node #2root_img and

var_img of Node #1

base_img/

root_img/

os/Linux/CST_1.1A*/

root_img and

var_img of Node #n

root_img and


var_img of Node #1

base_img/

root_img/

os/Linux/MNT_1.1A*/

root_img of

Maintenance-Node

base_img/

root_img/

maintenance mode

generation 1’

productive mode

generation 2

Ro

ot

Imag

es

1.

2.

3.

MO

VE



Please take notice of:

The Maintenance Cycle is based on a Root Image which is copied, cleaned and

modified to fulfill the needs of an exclusively used OS image

This Maintenance Image does not have to be a newly installed image from an

“Application Node – Images Linux SLES1x” DVD - you can use an image from an

earlier Maintenance Cycle. This implies the option to “maintain a maintained

image”.

There is no need to disable/enable FlexFrame Agents manually.

There is no need to change exports on NAS manually.

11.5.3 Installing Application Node Images from Installation

Media

This section describes how to install an Application Node Image delivered by Fujitsu. You

have to follow the instructions in this section only if you did not install a FlexFrame 5.1A

(or later) Application Node Image yet.

Due to the changed Application Node Image concept it is not possible to use old

Application Node Images. You have to install at least FlexFrame Orchestrator

1.1A (or later) Application Node Images in order to run a Maintenance Cycle.

You need to install an Application Node image if you did not have installed it yet.

This Application Node image comes as a so called “Base Image” and contains all needed

components in a single file delivered by the “Application Node – Images Linux SLES1x”

DVD(s). To install the Application Node image you just have to launch one command: ff_install_an_linux_images.sh which was already introduced in the “FlexFrame

– Installation of a FlexFrame Environment” guide’s section “Setting up Application

Nodes”. The following sections describe the ff_install_an_linux_images.sh

command in more detail since it plays an important role in the Maintenance of Linux

Application Node Images.

11.5.3.1 Installing the Application Node Image

To install Application Node Images delivered by Fujitsu insert one of the “Application

Node – Images Linux SLES1x” DVDs into the DVD device of one of the Control Nodes.

You can only install one “Application Node – Images Linux SLES1x” DVD at the

same time onto your FlexFrame system.

Since the ff_install_an_linux_images.sh command locks system wide

you must install the needed DVDs one after another.



The following examples use the first Control Node and the “Linux SLES11” DVD. At first

mount it:

cn1:/ # mount /media/dvd

After changing to the /media/dvd directory you will see the

ff_install_an_linux_images.sh command. This is actually only a wrapper to the

/opt/FlexFrame/bin/ff_install_an_linux_images.sh command in order to

increase the ease of use. It has been placed there since older FlexFrame DVDs had that

command as well and you can use it in the same way:

cn1:/media/dvd # ./ff_install_an_linux_images.sh

You might consider using the –v option when running the command for the first

time.

This will install the contents of the DVD to the Control Node as defined by the DVD itself: the cfg/install.cfg configuration file contains the information. In the example you

will find after installation the following directory:

cn1:/FlexFrame/volFF/os/Linux # l

total 20

drwxr-xr-x 6 root root 4096 Sep 23 18:18 ./

drwxr-xr-x 3 root root 4096 Sep 4 13:50 ../

drwxr-xr-x 3 root root 4096 Sep 8 16:39 FSC_1.1A00-000.SLES-11.x86_64/

The contents of the DVD will aways be located in /FlexFrame/volFF/os/Linux after

installation: Only images located there can be used by FlexFrame administration

commands. If you want to install an image under a different name use the –t option of

the ff_install_an_linux_images.sh command:

Please note that the current version of the

ff_install_an_linux_images.sh command uses a slightly different

syntax. The arguments to the –t and –s options of the command are now

names and not any longer directories.

This means for this example: The argument to the -t option is the name of the

target image for operation: The target image will be created in

/FlexFrame/volFF/os/Linux, or more precisely in the

/FlexFrame/volFF/os/Linux/<name> directory.

If a directory instead of a name is specified, the command will attempt to make a

reasonable assumption.

This helps to save your time.



cn1:/media/dvd # ./ff_install_an_linux_images.sh -v -t \

FSC_1.1A00-000.SLES-11.x86_64-another-example

It is worthwhile not changing the prefix of the directory name to keep track of the

images. All images delivered by Fujitsu are named FSC_<FlexFrame_version>.<OS_version>.<OS_architecture> where

FSC_ stands for FlexFrame Software Container.

The ff_install_an_linux_images.sh(8) manual page contains further compre-

hensive information about ff_install_an_linux_images.sh.

11.5.4 Step #1: Creating a Maintenance Image

As stated in the “Schematic overview” the first step in the Maintenance Cycle is to create

a so called “Maintenance Image”.



1

`

os/Linux/FSC_1.1A*/

productive mode

generation 1

/FlexFrame/volFF/

Bas

e Im

ag

es

root_img and

var_img of Node #n

root_img and


var_img of Node #1

base_img/

root_img/

os/Linux/CST_1.1A*/

root_img and

var_img of Node #n

root_img and


var_img of Node #1

base_img/

root_img/

os/Linux/MNT_1.1A*/

root_img of

Maintenance-Node

base_img/

root_img/

maintenance mode

generation 1’

productive mode

generation 2

Ro

ot

Ima

ge

s

1.

2.

3.

This “Maintenance Image” is ultimately a “Root Image” and therefore it can be created by help of the ff_install_an_linux_images.sh command.

Since a “Maintenance Image” has to be created the –m option has to be used.



syntax. The –m option needs an argument: the name of the node which should

run on the Maintenance Image. The node <node> must already exist.



Based on an existing Linux Application Node Image (specified by -s option) the

ff_install_an_linux_images.sh command will spawn another Linux Application

Node Image (specified by -t option) with special properties: it allows software installation

and modifications. You need to assign the named system (specified by -m option) to this

image by using the ff_an_adm.pl command and create the Node by using the

ff_new_an.sh command. This non-productive Node is in an exclusive state and not

recognizable by FA Agents.

Bear in mind:

this spawned image can be used for this particular Application Node (specified

by -m option) only!

the Application Node (specified by -m option) must already exist i.e. known to

ff_an_adm.pl

Synopsis

ff_install_an_linux_images.sh -s <name> -t <name> <-m node> \

[-v] [-d] [-i]

Example

cn1:~ # ff_install_an_linux_images.sh -m pa-rx300-1 \

-s FSC_1.1A00-000.SLES-11.x86_64 -t MNT_1.1A00-000.SLES-11.x86_64

It is worthwhile using a meaningful prefix for the directory name to keep track of

the images. A good start for choosing the name is to replace the original prefix

(here: FSC_) by MNT_ indicating a “Maintenance” image.

The example uses the just installed Linux Application Node Image of section “Installing

the Application Node Image” i.e. the source image (specified by -s option) is an image

delivered by Fujitsu and therefore has the FSC_ prefix. This is easier to explain in this

example. But this is not a constraint: you can use any image from an earlier Maintenance

Cycle i.e. you can “maintain a maintained image”.

11.5.5 Step #2: Assigning the Maintenance Image, booting and maintaining

As stated in the “Schematic overview” we now need to configure a Root Image since we

want to boot a real Linux Application Node in order to have a system for the software

maintenance:



11

`

os/Linux/FSC_1.1A*/

productive mode

generation 1

/FlexFrame/volFF/

Ba

se

Im

ag

es

root_img and

var_img of Node #n

root_img and

var_img of Node #2

root_img and

var_img of Node #1

base_img/

root_img/

os/Linux/CST_1.1A*/

root_img and

var_img of Node #n

root_img and

var_img of Node #2

root_img and

var_img of Node #1

base_img/

root_img/

os/Linux/MNT_1.1A*/

root_img of

Maintenance-Node

base_img/

root_img/

maintenance mode

generation 1’

productive mode

generation 2

Ro

ot

Ima

ge

s

1.

2.

3.

11.5.5.1 Using the Node

Since we have already chosen a particular Application Node (-m option) to run on the

Maintenance Image we now have to prepare it.

Notice that during the Maintenance Cycle this Application Node is not available

for Applications like SAP services, databases etc. It is not visible to FlexFrame

and can therefore not serve as a spare node during that time.

Move running applications to the remaining Application Nodes and shut it down.



Notice that if you power off the Application Node, e.g. using ff_ponpoff.pl, the

FlexFrame Agents will see an outage and are reacting as configured.

cn1:~ # ssh pa-rx300-1 init 0

Now follows the actual step #2: the assigning of the created “Maintenance Image” of step

#1 to this node by help of the well known ff_an_adm.pl command:

cn1:~ # ff_an_adm.pl --name pa-rx300-1 --op os \

--ospath Linux/MNT_1.1A00-000.SLES-11.x86_64

Create the image for the chosen Application Node by help of the well known ff_new_an.sh command:

cn1:~ # ff_new_an.sh -f –n pa-rx300-1

11.5.5.2 Booting

Finally boot the Node by help of the well known ff_ponpoff.pl:

cn1:~ # ff_ponpoff.pl --op power-on --name pa-rx300-1

After a short time the Application Node pa-rx300-1 will be up and running. During the

boot process you certainly have seen the following message:

###########################

RUNNING IN MAINTENANCE MODE

###########################

After login you will get the information again:

cn1:~ # ssh pa-rx300-1

FlexFrame(R) 1.1A00

Authorized uses only. All activity may be monitored and reported.

Last login: Wed Jun 4 11:05:26 2014 from 172.25.101.181

Authorized uses only. All activity of your current session may be

monitored and reported.

Welcome to FlexFrame(R)

(c) 2010-2013 Fujitsu Technology Solutions

This is Application Node pa-rx300-1

Running in MAINTENANCE mode

Have fun.....

pa-rx300-1:~ #



You see: without manipulating anything the system booted into the “Maintenance Mode”.

Checking the file systems will reveal:

pa-rx300-1:~ # mount -t nfs | grep " / "

10.10.11.2:/vol/volFF/node-pa-rx300-1/root_img/MNT_1.1A00-000.SLES-

11.x86_64 on / type nfs (rw,relatime,vers=3,…)

pa-rx300-1:~ # mount -t nfs | grep " /var "

pa-rx300-1:~ #

pa-rx300-1:~ # pgrep -lf AppAgent

pa-rx300-1:~ #

Looking closer to this command output shows that

the root file system is mounted read-write

In contrast to the normal operation the root file system is now writable and allows

installation of software

the /var file system is not mounted

In contrast to the normal operation the /var directory resides on the root file

system and is no separate file system. This means that installation of software

affecting the /var will go into the root file system.

the FlexFrame Agents are not running

In contrast to the normal operation the FlexFrame will not recognize the Application

Node pa-rx300-1 due to the non-started FlexFrame Agents.

11.5.5.3 Maintaining

This Application Node runnig in Maintenance Mode is prepared to get maintained now.

Before starting please keep in mind the following:

Changing the Application Nodes‘ software is delicate since any change can affect

the functionality of the entire FlexFrame. Therefore it is essential to test carefully all

changes.

Installing/Upgrading software should be done based on RPMs since it is easier to

keep track of the changes you made.

Installing SLES patches may have huge effects on the functionality of FlexFrame

Application Node images. Therefore evaluate security and recommended patches

whether they are applicable within a FlexFrame Application Node image.

Installing of complete SLES Service Packs is not allowed during the Maintenance

Mode. Service Packs contain a lot of changes that can have influences to the

FlexFrame functionality. Service Packs can also contain initial configuration files

that will substitute the FlexFrame configurations.



The only way to obtain a new SLES Service Pack for Application Nodes is to install

new Application Node images delivered from Fujitsu on new “Application Node –

Images Linux SLES1x” DVDs.

The section 11.5.8 Maintaining Use Cases contains detailed use cases of maintaining

work.

Finish maintaining of the Application Node’s software always with intensive tests. The

maintained Application Node Image will become the image for other Application Nodes as

well.

Notice that all Application Nodes you will boot later on from that maintained

image will inherit all the changes you made. Therefore keep an eye on unwished

remnants i.e. unwished temporary files, notes, directories etc. Clean them up.

Finally shut down the Application node running in Maintenance mode i.e. in the example

do:

pa-rx300-1:~ # init 0

11.5.6 Step #3: Reverting the Maintenance Image

As stated in the “Schematic overview” the last step in the Maintenance Cycle is to revert

the Maintenance Image in order to be able to run productive Nodes on it:



1

`

os/Linux/FSC_1.1A*/

productive mode

generation 1

/FlexFrame/volFF/

Ba

se

Im

ag

es

root_img and

var_img of Node #n

root_img and

var_img of Node #2

root_img and

var_img of Node #1

base_img/

root_img/

os/Linux/CST_1.1A*/

root_img and

var_img of Node #n

root_img and

var_img of Node #2

root_img and

var_img of Node #1

base_img/

root_img/

os/Linux/MNT_1.1A*/

root_img of

Maintenance-Node

base_img/

root_img/

maintenance mode

generation 1’

productive mode

generation 2

Ro

ot

Ima

ge

s

1.

2.

3.

The “Maintenance Image” resulting of step #2 needs to be transformed into a new Root

Image”. This - i.e. creating an Image - can be done by help of the already known ff_install_an_linux_images.sh command. Since a revert is needed the –r option

has to be used.



syntax. The –r option needs an argument: the name of the node which should

be reverted from Maintenance Image.



The reverting is required since only a temporary “Base Image” can be used to create a

“Root Image” which is needed to run productive Application Nodes on it.

Based on the maintained Linux Application Node Image (specified by -s option) the

ff_install_an_linux_images.sh command will spawn another Linux Application

Node Image (specified by -t option). You can assign systems to this Linux Application

Node Image by using ff_an_adm.pl and create Nodes by using the ff_new_an.sh

command. All of the Nodes running on it will benefit from the software maintenance you made. This launch of ff_install_an_linux_images.sh command will copy the

entire Image to a new Image.

Synopsis

ff_install_an_linux_images.sh -s <name> -t <name> <-r node> \

[-v] [-d] [-i]

Example

(Remember the step #2 asked to shut down the used Application Node)

cn1:~ # ff_install_an_linux_images.sh -r pa-rx300-1 \

-s MNT_1.1A00-000.SLES-11.x86_64 -t CST_1.1A00-000.SLES-11.x86_64

It is worthwhile using a meaningful prefix for the directory name to keep track of

the images. A good start for choosing the name is to replace the original prefix

(here: MNT_) by CST_ indicating a “Customer” image.

As mentioned earlier the SLES operating systems, which are used within a FlexFrame

environment, have been changed in a number of places to implement the concepts,

which make FlexFrame unique. In some situations the changes made during

Maintenance (Step #2) may conflict with FlexFrame’s concepts. In order to keep

FlexFrame going smoothly the FlexFrame implementations have higher priority than the

changes made during Maintenance mode. This means: these changes made during

Maintenance mode will be replaced during reverting2. If possible the

ff_install_an_linux_images.sh command will point out to these conflicts:

cn1:~ # ff_install_an_linux_images.sh -r pa-rx300-1 \

-s MNT_1.1A00-000.SLES-11.x86_64 -t CST_1.1A00-000.SLES-11.x86_64

...

The following files were modified during maintenance. To ensure

FlexFrame's functionality they will be replaced now by their

2 FlexFrame Infrastructure Consultants might seek advice in /opt/FlexFrame/lib/an.



FlexFrame specific versions. See "Administration and Operation

Guide" for details:

/etc/issue

/root/.ssh/authorized_keys

/etc/fstab

A copy of the maintenance version is available as

"<file>.maintenance".

...

Here you must compare for example /etc/fstab.maintenance (this is the version of

this file after leaving maintenance and before reverting) and the /etc/fstab (this is the

needed FlexFrame version of this file, installed after revert). You may have to merge

entries from the /etc/fstab.maintenance file into the /etc/fstab file.

Note: it is important to double-check the reverted image whether it contains the expected

changes: This is possible for example by simply comparing the just created image with

the old maintenance image. If there are changes which were replaced by FlexFrame

settings it is needed to decide whether they can be inserted again. Please take notice of

section 11.5.8 Maintaining Use Cases.

After reverting the source of the operation the “Maintenance Image” seems to be

needless: it was the source for the revert and can be removed now. But in section 11.5.8

Re-Using the Maintenance Image on page 304 you will see how experienced

administrators might use it again.

11.5.7 Migrating remaining Application Nodes

Now, after testing and reverting the “Maintenance Image” it is time to go productive with

all the Application Nodes.

The created image acts as a normal image i.e. it can now serve as a source for other

Application Node Images. All Application Nodes inherit the changes from Maintenance

Mode.

To do this it is only necessary to use the well known administration commands. Since this

is not “Maintenance Mode” specific – the procedure and all the commands are already

known - now only the example follows:

At first assign the created image to the Application Nodes by help of the ff_an_adm.pl

command and then create an image by help of the ff_new_an.sh command and boot.

Assuming we have another 3 Application Nodes in the group sles11_x64 of the pp1

pool where our pa-rx300-1 belongs to.

The first one – named pa-rx300-1 – was used for the Maintenance Cycle and is

currently down. Let us start with that node:




--ospath Linux/CST_1.1A00-000.SLES-11.x86_64

The ff_an_adm.pl command asked for it so let us launch the ff_new_an.sh

command:

cn1:~ # ff_new_an -n pa-rx300-1

Finally boot the Application Node:


All the other nodes of the pool group – named pa-rx300-2, pa-rx300-3 and pa-

rx300-4 – which are still running on the old image can be handled in a similar way.

Instead of powering them on they just need to be rebooted after assigning the image:

Notice that if you power off the Application Node, e.g. using ff_ponpoff.pl, the

FlexFrame Agents will see an outage and are reacting as configured.













The switch to the maintained image costs only a few minutes! In case of problems you

can switch back in the same way.

Please bear in mind: It is very important that all the Linux Application Nodes of

one pool group have the same configuration (software, patches) since each one

must be able to take over if another one fails. That is why all of the Linux

Application Nodes of a pool group must run on the same image.

In the course of time you might get a lot of Application Node Images in

/FlexFrame/volFF/os/Linux/. In order to keep track of their history you might use

the “.itoc” file of the “Root Image”.



11.5.8 Re-Using the Maintenance Image

This section describes a time-saving extension to the “Maintenance Cycle”. You should

use it not before you have got some experience with the “Maintenance Cycle” itself.

Reverting the “Maintenance Image” as described in section 11.5.6

Step #3: Reverting the Maintenance Image on page 299 ended with an apparently

needless “Maintenance Image”: it was the source of the operation. If you are satisfied

with the results of the Maintenance Cycle i.e. the migrated Application Nodes are running

fine you can remove it indeed.

But why it is not removed automatically?

Because it is possible to configure the Application Node (used to perform the

maintenance tasks) and to boot it from the “Maintenance Image” again: this allows fixing

errors you found while testing the just generated images without starting a new

“Maintenance Cycle”.

Be aware that only the same Application Node can be used to boot the “Maintenance Image” again. You can use the ff_an_adm.pl command to get the needed information.

Launch it with no option (output trimmed):

cn1:/ # ff_an_adm.pl

…


The path of the os image to be used.

Currently available os image pathes are:

Linux/CST_5.3A00-001.SLES-11.x86_64


Linux/MNT_5.3A00-001.SLES-11.x86_64 (maintenance only (pa-rx300-

1))

Linux/TST_5.3A00-002.SLES-11.x86_64

…

It reveals the name of the only Application Node that can use this “Maintenance Image”:

the pa-rx300-1 node.

To assign the “Maintenance Image” again to this Node use the well known

ff_an_adm.pl command:


--ospath Linux/MNT_1.1A00-000.SLES-11.x86_64

After that create the image for the Application Node:

cn1:~ # ff_new_an.sh –B -n pa-rx300-1



Notice that ff_new_an.sh needs to be called with the “-B” option to only configure boot

information for this node because the image is already configured for this node and

cannot be reconfigured.

Finally, reboot the maintenance node:


or switch it on if it is powered off:


After some time the node will come up running the “Maintenance Image" and you can

perform further maintenance tasks and fix errors you found while testing.

As soon as you have finished maintaining this image, shut down the node and revert the

“Maintenance Image” to a new image again as described in section 11.5.6

Step #3: Reverting the Maintenance Image on page 299. Note that you cannot use the

same target directory name for the revert operation (it already exists.) Use a different

directory name or remove the old one if you want to use the same target directory name

as before.

Now it is possible to have one node for maintaining the Maintanance Image and

another node to test the reverted image in production mode because the

Maintenance Images and the images used for production reside inside two

different directory trees.

You will not have to switch one Application Node forth and back between

Maintenance Image and Base Image again and again. You still need to shut

down the maintenance Application Node before reverting to ensure a consistent

image state.

In theory it is also possible to run one Application Node always in “Maintenance

Mode” in order to keep it up to date with security updates and create an image

from it once in a while as needed.

However, it is strongly recommended to remove the Maintance Image once the

maintained Image has been tested intensively in production mode and perform

further maintenance tasks by starting over with a complete new Maintance Cycle

based on the Maintained Images, or in other words: perform the whole 1-2-3

procedure again on the already customized Images.

Following this procedure closely allows keeping all maintained image versions

and rolling back to working images if something failed during a Maintenance

Cycle.

You can keep track of all changes by just looking into the mentioned “.itoc” file

of the “Root Images”.



11.5.9 Maintaining Use Cases

We would like to point out that changes due to 3rd party software or patches or

customer modifications may have heavy (negative) side effects to the FlexFrame

functionality.

Installing third party software on Application Nodes may cause functional

restrictions and other malfunctions of the FlexFrame system software. Fujitsu

shall have no liability whatsoever whether of a direct, indirect or consequential

nature with regard to damages caused by the third party software or its

erroneous installation.

Installation of software or other maintaining work as described in this section has

to be done by help of a Maintenance Cycle as described in section 11.5

Maintenance of Application Nodes - Software on page 289. Not following this

procedure i.e. maintain software when running in normal operation may harm the

entire Application Node group. Resulting malfunctions are not an issue of

FlexFrame.

11.5.9.1 Service Packs

It is not allowed to install any Service Packs on Application Node images manually. The

only way to obtain a new service pack for Application Nodes is to install new Application

Node images delivered from Fujitsu. Service Packs contain a lot of changes that can

have influences to the FlexFrame functionality. Service Packs can also contain initial

configuration files that will substitute the FlexFrame configurations.

11.5.9.2 Updating/Installing a New Linux Kernel

This section describes how to install an additional (alternative) Linux kernel into an

existing Application Node image.

The netboot configuration /tftpboot/pxelinux.cfg/<hw class>_<pool name>_<image tree name>

will determine which Linux kernel and initial ramdisk need to be used for a subset of

Application Nodes. The image tree name of original images delivered by Fujitsu begins with FSC_ and follows the scheme

FSC_<FF version>.<OS version>.<OS architecture>.

11.5.9.2.1 Software Stage

To install and update any software packages in FlexFrame, it is useful to mount your NAS

system or jukebox or any other software stage on a local mount point at your Control

Node. The same software stage must be mounted on an Application Node which is

appointed to install the new kernel-RPMs during the maintenance cycle.



For example, your software stage on the Control Node is /FlexFrame/volFF/FlexFrame/stage and the new kernel to install is

2.6.16.60-0.63.1.

Create a subdirectory for the delivered software:

control1: # cd /FlexFrame/volFF/FlexFrame/stage

control1:/FlexFrame/volFF/FlexFrame/stage # mkdir -p SLES10/kernel

Copy the delivered software packages to the software stage.

Kernel RPM:

control1: # cp <somewhere>/kernel-smp-2.6.16.60-0.63.1.x86_64.rpm

/FlexFrame/volFF/FlexFrame/stage/SLES10/kernel

Kernel kdump:

control1: # cp <somewhere>/kernel-kdump-2.6.16.60-0.63.1.x86_64.rpm




Kernel source:

control1: # cp <somewhere>/kernel-source-2.6.16.60-0.63.1.x86_64.rpm


With SLES 10 (Service Pack 1 and later) Novell has introduced the Partner

Linux Driver Process (PLDP). There is no need to download drivers for the

update kernel. The existing drivers are reused for the new kernel.

11.5.9.2.2 Installing a New Linux Kernel

Logon to the appointed Application Node running the maintenance cycle.

Check the currently installed Linux kernels:

pa-rx300-1:/ # rpm -qa | grep kernel

kernel-kdump-2.6.16.60-0.59.1

kernel-smp-2.6.16.60-0.59.1

kernel-source-2.6.16.60-0.59.1

Mount the software stage:

pa-rx300-1: # mkdir /mnt/stage

pa-rx300-1: # mount -t nfs <filer>:/vol/volFF/FlexFrame/stage /mnt/stage

To install the new kernel, execute the following command:

pa-rx300-1: # cd /mnt/stage/SLES10/kernel

pa-rx300-1:/mnt/stage/SLES10/kernel # export YAST_IS_RUNNING=instsys

pa-rx300-1:/mnt/stage/SLES10/kernel # rpm -ihv kernel-smp-2.6.16.60-

0.63.1.x86_64.rpm

pa-rx300-1:/mnt/stage/SLES10/kernel # rpm –ivh kernel-kdump-2.6.16.60-

63.1.x86_64.rpm

pa-rx300-1:/mnt/stage/SLES10/kernel # rpm –ivh kernel-source-2.6.16.60-

63.1.x86_64.rpm

‘YAST_IS_RUNNING=instsys’ is used to skip updating the bootloader

menu.

11.5.9.2.3 New initrd, Kernel and Netboot Configuration

Before booting the maintained Application Node image, it is necessary to create a new

initrd, install the initrd and kernel in /tftpboot and update the PXE configuration file.

All these steps will be automatically performed on the Control Node by the command:

control1:~ # ff_new_an.sh -B -n pa-rx300-1



Internally, this will call the ff_mkinitrd.sh utility, update the storage export

configuration and create a new PXE configuration file.

The latest kernel will be automatically used in the future. If you like to revert to a previous

kernel, follow the instructions described below in Chapter 11.5.9.2.5.

Now reboot the Application Node and verify that it is working properly:

pa-rx300-1:~ # reboot

11.5.9.2.4 Partner Linux Driver Process

The Partner Linux Driver Process (PLDP) replaces the traditional method of driver

provisioning for SLES 10 (Service Pack 1 and later). The most benefit of this process is

the support for reusing existing drivers for errata kernels. There is no need to download

drivers for those kernels.

Nevertheless if updated driver packages are released (e.g. a new version) these

packages can be installed on Control Node or Application Nodes like an “installable

patch”.

Depending on a special configuration the document “Novell Partner Linux Driver Process

for PRIMERGY servers” describes the update procedure in detail. This document is

available at URL:


11.5.9.2.5 Switching Between Different Kernels

If you have multiple kernel releases installed in the Application Node image and like to

switch to a different kernel, change the the symlink …/root_img/boot/vmlinuz and

run ff_new_an.sh -B -n <node name> again.

This will change the PXE configuration files for this image and hardware class and will

affect all Application Nodes of the same hardware class using this image.

The initrd does not need to be re-created nor does the initrd-symlink need to be

changed because it always contains all kernel releases found in an image.

11.5.9.3 ServerView Update

The ServerView update comprises the update of the components “Agents” and “RAID

Manager” on the Application Node.

The components for the Application Nodes are the same as for the Control

Nodes.




README files or other installation instructions are available in the download

area.

Always use the instructions that come with the downloaded software!

Mount your software stage:

pa-rx300-1:/mnt # mkdir /mnt/stage

pa-rx300-1:/mnt # mount -t nfs <filer>:/vol/volFF/FlexFrame/stage

/mnt/stage

11.5.9.3.1 ServerView Agents

Go to your software stage directory. There you will find the following RPMs: srvmagt-agents-*.i386.rpm

srvmagt-mods_src-*.i386.rpm

srvmagt-eecd-*.i386.rpm

ServerViewConnectorService-*.386.rpm

And the associated installation script: srvmagt.sh.

Upgrade the ServerView Agents by using srvmagt.sh:

pa-rx300-1:~ # cd /mnt/stage

pa-rx300-1:/mnt/stage # sh srvmagt.sh install

11.5.9.3.2 ServerView RAID Manager

Go to your software stage directory. There you will find the

ServerView_RAID-*.i386.rpm RPM.

Upgrade the ServerView RAID Manager by:

pa-rx300-1:~ # cd /mnt/stage

pa-rx300-1:/mnt/stage # rpm -U ServerView_RAID-*.i386.rpm

The ServerView update is now finished.

11.5.9.4 Upgrading the Application Software

The rules for installation of application software (SAP/Oracle) apply here as well.

An Application Node should be available exclusively for upgrading the application

software, so that the physical host name can be set to be identical with the virtual host

name of the application.



Provided this has been done, upgrading can be carried out in accordance with the

software vendor's standard guides.

All parts of the application software must reside on the network files system mounted

from the NAS system.

For more details see the SAP Installation Guides.

11.5.9.5 Updating RPM Packages on an Application Node

To update the SLES RPM packages you can use the original Novell tools to perform an

online update.

11.5.9.5.1 Setup a patch server

It is not recommended to connect your FlexFrame environment directly to the Internet to

download patches or updates. Therefore it is advisable to set up an own patch server,

which provides the necessary update packages. See: http://www.novell.com/de-

de/linux/smt/ and http://www.novell.com/communities/files/appnote-2009-12-10-v1_7.pdf

for hints.

Provided that this patch server is accessible for the Application Node in maintenance

mode, a normal online update with YaST can be performed.

11.5.9.5.2 Connect the patch server to the FlexFrame environment

There are several possibilities to connect your local patch server to your FlexFrame

environment. It’s up to your internal network and security policy, which way is the right

way for you.

Some of them are described in http://www.novell.com/communities/files/appnote-2009-

12-10-v1_7.pdf

It is possible to setup an ‘External Connectivity’ on the Control Node to connect your

patch server to the FlexFrame environment. On the Application Node in maintenance

mode you have to setup a ‘SSH tunnel’. Or you can copy the data from your patch server

to an ‘USB disk’ and connect this disk directly to the Application Node in maintenance

mode. Or copy the data to the local disk of the Control Node or to a directory on the filer

and share (NFS, SMB/CIFS, HTTP(S) or FTP) this directory for the Application Node in

maintenance mode.

11.5.9.5.3 Prepare the Online Update

To prepare the online update you have to define the installation source. That’s the way,

how you provide the update packages, i.e. your patch server



On an SLES11 Application Node:

YaST Software Software Repositories Add Media Type

See Section 9.3 “Managing Software Repositories and Services”

http://www.novell.com/documentation/sles11/book_sle_deployment/data/sec_y2_sw_inst

source.html#sec_y2_sw_instsource for more information.

The Media Type strongly depends on the way you provide the update packages (see

section 11.5.9.5.2 ”Connect the patch server to the FlexFrame environment” on page

311).

11.5.9.5.4 Perform the Online Update

Now you can perform the online update:

YaST Software Online Update

For SLES11 see section 1.0 “YaST Online Update” from

http://www.novell.com/documentation/sles11/book_sle_admin/data/cha_onlineupdate_yo

u.html#cha_onlineupdate_you for more information.

If the online update has installed a new kernel, it is necessary to create a new

initrd and update the netboot configuration, before you can reboot the

maintained Application Node.

All these steps will be automatically performed on the Control Node by the command:

cn1:~ # ff_new_an.sh -B -n pa-rx300-1

Internally, this will call the ff_mkinitrd.sh utility, update the storage export

configuration and create a new PXE configuration file.

11.5.9.6 Updating vmware-tools

To update the vmware-tools to a newer version, we recommend to use our provided

Application Node Images containing this new version of vmware-tools.

To install a newer version of vmware-tools during a FlexFrame maintenance cycle please

refer to the OSP documentation of VMware, which can be found at

http://packages.vmware.com

Installation of vmware-tools from an ESXi host is not permitted.

http://packages.vmware.comf/



Log on to the Application Node in Maintenance, copy the downloaded vmware-tools

packages to this Application Node and follow the instructions to install the vmware-tools

on SLES11 or SLES10 as described in the VMware OSP documentation mentioned

above.

11.5.10 Implement conflicting changes using the ff_new_an.custom hook

If possible, all modifications of an Application Node image should be configured during a

Maintenance Cycle. However there are some cases where this is not possible, especially

when changing files or configuration directives that are automatically configured during

ff_install_an_linux_images.sh or ff_new_an.sh.

Examples of such cases are:

● SSH Host Keys, authorized_keys, known_hosts and other configuration

files

● Changes on the /etc/fstab file

● Disabling Services that are automatically re-enabled by FlexFrame tools while

reverting an image from Maintenance Mode

To be able to perform such modifications, they have to be done at the end of the ff_new_an.sh command, which is possible by creating a script called

ff_new_an.custom within the Image directory.

As this is a highly flexible way to modify images, it is strongly recommended to

be used by certified FlexFrame professionals with decent bash scripting

knowledge only. There is a high risk of damaging the whole FlexFrame

environment.

After the first call of ff_new_an.sh for an image, a template for this custom hook is

linked into the image directory:

cn1:/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64 # ls -l

total 32

-rw-r--r-- 1 root root 2906 Sep 25 13:43 .cleanup_root



lrwxrwxrwx 1 root root 47 Oct 11 12:39 ff_new_an.custom.TEMPLATE ->

/opt/FlexFrame/lib/an/ff_new_an.custom.TEMPLATE

-rw-r--r-- 1 root root 632 Sep 25 13:02 ff_pxe.conf

-rw-r--r-- 1 root root 170 Sep 25 13:02 modules.conf

drwxr-xr-x 35 root root 4096 Oct 11 13:01 root_img

drwx------ 4 root root 4096 Oct 11 13:36 var_img

To create a new ff_new_an.custom script, just copy the template:

cn1:/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64 # cp

ff_new_an.custom.TEMPLATE ff_new_an.custom

cn1:/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64 # ls -l

total 28

-rw-r--r-- 1 root root 2906 Sep 25 13:43 .cleanup_root

-rwxr-x--- 1 root root 3958 Nov 2 12:06 ff_new_an.custom

lrwxrwxrwx 1 root root 47 Oct 11 12:39 ff_new_an.custom.TEMPLATE ->

/opt/FlexFrame/lib/an/ff_new_an.custom.TEMPLATE

-rw-r--r-- 1 root root 632 Sep 25 13:02 ff_pxe.conf

-rw-r--r-- 1 root root 170 Sep 25 13:02 modules.conf

drwxr-xr-x 35 root root 4096 Oct 11 13:01 root_img

drwx------ 4 root root 4096 Oct 11 13:36 var_img

You now have created a new custom user script from the template which contains a

couple of examples and information about how to implement custom modifications.

To get started, edit the file and activate the line SHOW_ALL_VARIABLES=1:

...

# uncomment the following line to print the above variables and

get_cn_path examples:

SHOW_ALL_VARIABLES=1

#

...

Run ff_new_an.sh in test mode to run the ff_new_an.custom script:

cn1:/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64 #

ff_new_an.sh -T -n bx202

Info: Starting configuration of Application Node bx202

Info: Using Image /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-

11.x86_64

Info: Executing custom script /FlexFrame/volFF/os/Linux/FSC_1.0A00-

017.SLES-11.x86_64/ff_new_an.custom



Variables:

----------

MAINTENANCE=""

ROOT_IMG="/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-

11.x86_64/root_img"

VAR_IMG="/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-

11.x86_64/var_img/var-c0a81e20"

POOLDATA_PATH="/FlexFrame/volFF/pool-pooldev/pooldata"

POOL_IMG="/FlexFrame/volFF/os/Linux/pool_img/pool-c0a81eff"

VAR_TEMPLATE="/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-

11.x86_64/root_img/var"

OSDIST="SLES-11"

OSARCH="x86_64"

CLIENT_NODE_NAME="bx202"

POOL_NAME="pooldev"

CLIENT_PROD_NAME="BX6924S3"

NIC_10G="no"

CONFIG="/tftpboot/config/netboot_pooldev_bx202.cfg"

get_cn_path examples:

----------

get_cn_path "/etc/passwd" =>

/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-

11.x86_64/root_img/etc/passwd

get_cn_path "/etc/resolv.conf" => /FlexFrame/volFF/pool-

pooldev/pooldata/config/etc/resolv.conf

get_cn_path "/etc/hosts" =>

/FlexFrame/volFF/os/Linux/pool_img/pool-c0a81eff/etc/hosts

get_cn_path "/etc/fstab" =>

/FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-11.x86_64/var_img/var-

c0a81e20/FlexFrame/etc/fstab

Info: Configuration of bx202 was successful

These variables can and should be used where applicable.

In addition the these variables, there is also a function get_cn_path which maps a path

as seen on the application node to a path as seen from the control node.

For example, if you like to add a line to /etc/fstab, you would call this:

echo "# some content" >>$(get_cn_path /etc/fstab)



This makes sure that you custom script always finds the correct file location on the

Control Node, no matter if the file is linked to the var image or whether you are running in

maintenance mode or not.

Because of these pre-defined variables and functions, the ff_new_an.custom script

can only be called through ff_new_an.sh.

Also, be very careful about what is done in this script. Before dealing with files, especially

with the get_cn_path function, you should always just echo the determined filename in

a first run before actually modifying that file without having tested the code.

Also, be aware that writing to symlinks in the Application Node image may redirect these

writes to the Control Node, especially with absolute symlinks, so please double check

what files are really being accessed.

There are a couple of commented examples for common use cases in the

template which should be used if possible. These examples have been

thoroughly tested and can be modified to your needs.

About the Mechanism

The mechanism how the ff_new_an.custom script is called is as follows:

User calls ff_new_an.sh

internally calls ff_create_an_linux.sh

if ff_new_an.custom exists in the image directory, selected internal

variable and functions are being exported the ff_new_an.custom is

called

Each time, ff_new_an.custom is called, this is being logged in the FlexFrame logs.

The current script is also archived in the subdirectory ff_new_an.custom.archive

within the image directory, the archived name contains the md5 sum of the script.

With the help of the FlexFrame logs it is therefore possible to reconstruct which scripts

had been called in the past:

cn1:~ # grep ff_new_an.custom /var/log/FlexFrame/log_cn1_2012-11-02 |

tail -2

2012-11-02 12:11:45 INFO [root,30499,cn1] ff_create_an_linux.sh

Archiving /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-



11.x86_64/ff_new_an.custom to /FlexFrame/volFF/os/Linux/FSC_1.0A00-

017.SLES-

11.x86_64/ff_new_an.custom.archive/ff_new_an.custom.c61da8adb37d70bdcf7f

cb44ec476102

2012-11-02 12:11:45 INFO [root,30499,cn1] ff_create_an_linux.sh

Executing custom script /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-

11.x86_64/ff_new_an.custom

11.6 Maintenance of the Application Nodes - Hardware

11.6.1 Changing BIOS Settings for Netboot

For detailed information on setting BIOS parameters refer to the hardware

documentation. Look at some of the configuration examples below. The exact procedure

may differ depending on the BIOS version of your hardware. For newer server models

refer to the corresponding documentation called "FlexFrame® HW Characteristics

Quickguide ("Hardwaresteckbrief")" – in these documents (delivered with the Service CD

of a FlexFrame version and also on the FTS Extranet on

https://partners.ts.fujitsu.com/com/service/ps/solutions/flexframe) the server specific

properties relevant for installing the server within a FlexFrame environment are collected.

11.6.2 Replacing a Network Card – Application Node

There are two cases for replacing a network card on an Application Node:

The card has failed and you want to replace it by a card of same type.

The network card is replaced by a card of another technology, e.g. 1GBit to 10GBit

migration or vice versa.

If a network card on an Application Node fails and is replaced, further actions are

necessary to ensure proper operation.

During their boot processes, PRIMERGY nodes need to retrieve some information

through lower level protocols which are based on hardware addresses (MAC addresses).

The PRIMERGY's BIOS utilizes DHCP for that purpose.

The MAC address is the built-in, physical address which is unique to that network card.

Since there are two network cards configured for booting you may not even notice the

MAC address change of a single one, because the other card will be used for booting.

Once the server is up, operation is normal. For full redundancy – even for booting – we

recommend to adjust the configuration accordingly.

https://partners.ts.fujitsu.com/com/service/ps/solutions/flexframe/Pages/default.aspx



On PRIMERGY blade systems, use the management blade to determine the server

blade MAC addresses.

On PRIMERGY non-blade systems, use BIOS or network card labels to determine

the MAC addresses.

To change the MAC addresses for booting, use the commands below. Replace the node

name and MAC addresses of the sample with your Application Node's name and noted

addresses:

control:~ # ff_an_adm.pl --op mac --name appnode5

--mac 00:e0:00:c5:19:41,00:e0:00:c5:19:41

MAC addresses have been changed.

The AN boot information has to be recreated. To do this exec:

ff_new_an.sh -B -n appnode5

control:~ # ff_new_an.sh -B -n appnode5

The first command changed the MAC addresses of the node within DHCP configuration

and LDAP and the second one recreated the AN boot information. No further intervention

is necessary.

Some software licenses are derived from information based on server specific

numbers, such as MAC addresses of network cards. On Linux, double-check

your SAP licenses after you have replaced a network card.

If a network card on an Application Node is replaced for technology change reasons the

following steps are necessary:

1. Remove the Application Node from the FlexFrame landscape with ff_an_adm.pl --op rem

2. Remove the 1GB network cables of the Application Node

3. Remove the 1GB NIC cards from the Application Node

4. Plug in the 10GB NIC cards into the Application Node (see corresponding

"FlexFrame® HW Characteristics Quickguide ("Hardwaresteckbrief")")

5. Install the Application Node with ff_an_adm.pl

(for details see the corresponding "FlexFrame® HW Characteristics Quickguide

("Hardwaresteckbrief")")

Example:

ff_an_adm.pl --op add --name sno2rx1 --type RX600S4 --pool

pool2 --group default --swgroup 1 --ospath Linux/FSC_5.1A00-

005.SLES-11.x86_64 --host 25 --mac

00:1b:21:24:4f:5d,00:1b:21:24:4f:5c -–10gbit

6. Plug in the network cables according to the wiring output given by ff_an_adm.pl



7. Call ff_new_an.sh to generate an image for the new Application Node

8. Boot the Application Node

11.6.3 Replacing a Switch Blade

There are two cases for replacing a switch blade:

In case of an HW error you have to replace one switch blade.

You can change the type of the used switch blade. In this case you have to replace

both switch blades.

In the first case (HW error) perform the following actions:

1. You have to replace the switch blade as described in the HW description of the

BX Server.

2. Create the configuration file for the switch blade with the function ff_bx_cabinet.pl –-op swb-config. See chapter 7.6.7 Getting Switch Blade

Configuration for a detailed description of the function.

3. Upload configuration to switch as displayed by ff_bx_cabinet.pl –-op swb-

config.

In the second case (change the type) perform the following actions:

1. Deactivate the usage of each server blade.

2. Remove the network cables from the switches.

3. Replace both switches.

4. Plug in the network cables.

5. Call ff_bx_cabinet_adm.pl --op swb-change to update LDAP Database.

6. Create the configuration file for the switch blade with the function ff_bx_cabinet.pl –-op swb-config. See chapter 7.6.7 Getting Switch Blade

Configuration for a detailed description of the function.

7. Upload configuration to switch as displayed by ff_bx_cabinet.pl –-op swb-

config.

8. Now you can use the server blades again.

11.6.4 Replacing Power Control Hardware

This section describes what to do after the replacement of power control hardware,

depending on your Application Nodes and Control Nodes hardware.



For instructions on replacing the power control hardware, please refer to the

documentation delivered with your hardware.

11.7 Maintenance of ESXi Servers

11.7.1 BIOS Updates

To update the BIOS of a PRIMERGY server that is used as an ESXi server, use the

standard BIOS update procedure for PRIMERGY servers based on a bootable USB

memory stick. Please note that the method using ASPs (Autonomous Primergy Support

Packages) is not applicable in this case.

11.7.2 Replacing a Network Card - ESXi Server

If a network card of an ESXi server fails and is replaced, further actions are necessary to

ensure proper operation.

PRIMERGY servers used as ESXi servers in FlexFrame get the address used for ESXi

management from a DHCP server running on the FlexFrame Control Center. The DHCP

server determines the correct IP address based on the requester's MAC address.

The MAC address is the built-in, physical address which is unique to that network card.

As the MAC address is changed after replacing a network card, we recommend adjusting

the configuration accordingly,

● On PRIMERGY blade systems, use the management blade to determine the server

blade MAC addresses.

● On PRIMERGY non-blade systems, use BIOS or network card labels to determine

the MAC addresses.

To change the MAC addresses for DHCP, use the command below. Replace the node

name and MAC addresses of the sample with your ESXi server's name and noted

addresses:

control:~ # ff_hn_adm.pl --op mac --name esxnode5

--mac 00:e0:00:c5:19:41,00:e0:00:c5:19:42

The program changes the MAC addresses of the node within DHCP configuration and

LDAP. No further intervention is necessary.

11.7.3 ESXi Updates and Patches

VMware provides patches and updates for ESXi servers. While updates are for

maintenance releases, patches address critical security fixes or urgent bug fixes. An

update or patch can include a new build of firmware, an update of VMware Tools, or an

update of the vSphere Client.



Please note that patches and updates of VMware Tools cannot be applied using the

standard VMware procedures. The FlexFrame Application Node images contain the

VMware Tools operating system specific packages (OSPs). For a description on how to

upgrade these tools refer to section 11.5.9.6 .

VMware provides several tools for installing updates and patches to ESXi hosts:

● esxcli - command-line utility

● vCenter Update Manager - automates patching and updates

The esxcli utility is part of the vSphere CLI, which is available on the FlexFrame Control

Nodes.

For more information on these tools as well as further usage hints and recommendations

please refer to the appropriate VMware documentation.

11.8 Maintenance of Other FlexFrame Components

11.8.1 NetApp Storage

ONTAP Patches

Before applying a new patch to the Filer's ONTAP make sure that it is supported for the

FlexFrame concept. In doubt contact the Fujitsu support.

Download the tar file(s) for the new ONTAP onto one of your Control Nodes.

Create a temporary directory e.g. in /FlexFrame/volFF/FlexFrame/stage/ontap

(create the directory if it does not exists) and extract the tar file there.

Proceed as described in the manual or README information of that patch.

Before you activate the new patch by rebooting the Filer, make sure that no SAP system

or Database is busy. You do not need to reboot the Application Nodes, however we

recommend double-checking the mount points after the Filer is up again. If an Application

Node should get stuck, reboot it.

11.8.2 Switches and Switch Blades

11.8.2.1 Firmware Update

It may be possible that an update of the switch firmware is necessary. You will find a

detailed description in the manual of the specific vendor.. For Switch Blades have a look

at the corresponding user manual.

11.8.2.2 Backup of Switch Configurations

The switch configuration may be different to that at the time of installation due to the

operation of a FlexFrame landscape. New Application Nodes and switches may be



added, ports configured and so on. For system backup, there are a lot of well known

products and programs.

To simplify backing up the configuration of all switches used within the FlexFrame

landscape the program /opt/FlexFrame/bin/ff_save_switch_config.pl should

be used. It is designed to be run by cron(8) and recommended to call it once a day.

Default is to save the switch configurations of all FlexFrame switches into the default

backup directory /tftpboot/saved_switchconfigs. Only human readable format is

selected for backup. Startup configuration (the configuration the switch reads when

booting) is preferred before the running configuration for backup.

Synopsis

ff_save_switch_config.pl [--name <switch name>]

[--dir <backup directory>]

[--running]

[--ip <control node ip>]

Options

--name <switch name>

Defines the name of the switch to be backed up. Keep in mind to use the name

listed from ff_swgroup_adm.pl respectively ff_bx_cabinet_adm.pl. If omitted all

Switches are backed up.

--dir <backup directory>

Defines the directory where to store the backup. If omitted the backups are stored

into /tftpboot/saved_switchconfigs.

--running

Defines the running configuration to be preferred before the startup configuration for

backup. If omitted the startup configuration is preferred before the running

configuration for backup.

--ip <control node ip>

Defines the control lan ip address of the control node running the netboot service. If

omitted the ip is determined by the program itself if needed.

11.8.2.3 Restore of Switch Configurations

To restore a backed-up switch configuration clear the actual configuration on the switch

then use copy and paste for the switch config commands. Don’t forget to copy the



restored new running switch configuration on the switch as startup-configuration for the

next reboot.

Be aware, the backed-up switch configuration may not be the last effective

switch configuration because of changes applied after backup.

You can get an actual configuration according to LDAP

for switch blades using

ff_bx_cabinet_adm.pl --op swb-config ... (see 7.6.7)

for switch groups using

ff_swgroup_adm.pl --op gen-config ... (see 9.4)

11.8.3 DNS Servers

If the customer wants to change IP addresses of his DNS servers he also has to change

the corresponding FlexFrame configurations:

The DNS servers of each pool can be modified with the command ff_pool_adm.pl (see

4.5).

The DNS servers of the Control Center can be modified by editing the “forwarders” in

the file /etc/named.conf . Originally the DNS Servers of the Control Center came into

FlexFrame by specifying them in the “Miscellaneous Control Center Settings” in the

Management Tool.

Any other DNS server (e.g. explicitly indicated DNS servers in hardware configuration

files), which manually has been entered by the customer, is not under FlexFrame control

and it is the customers responsibility to modify these addresses at the right place.

11.8.4 Third Party Products

If the customer needs to install third party software on the Application Nodes, the

maintenance procedures described in this manual must be followed. Installation of

software or patches may have to be performed again after installing a maintenance

image shipped by Fujitsu.



If you have installed additional 3rd

party software to your FlexFrame landscape,

please make sure to back up this software components and their configuration

data for later restore. Please consider: If you install a new FlexFrame image, the

whole file system and directories are overwritten by a complete new root file

system image. Afterwards, any 3rd

party software and configurations have to be

re-installed or restored.

Installing third party software on Control Nodes or Application Nodes may cause

functional restrictions and other malfunctions of the FlexFrame system software.

Fujitsu shall have no liability whatsoever whether of a direct, indirect or

consequential nature with regard to damages caused by the third party software

or its erroneous installation.

11.9 FlexFrame Agents

For information on the FlexFrame Agents please refer to the "FlexFrame Agents

Installation and Administration" manual, chapter 2 "First Steps". Here you will find

detailed information on installation requirements, installation, configuration, the

FlexFrame Agents web interface and the FlexFrame Domain Manager.

11.10 Project specific changes

As mentioned in chapter 2 “FlexFrame Architecture” the FlexFrame solution consists of

both hardware and software. Most of the components are seen as an appliance: they

have well defined purposes and must work together in order to achieve a smoothly

running FlexFrame landscape. Normally only a well defined set of software and hardware

(see section 2.2 “Hardware and Software”) is allowed.

However there might be the need to change or modify components, or configuration

details in order to satisfy project specific requirements.

These special changes – project specific changes - mean that your configuration is

outside the FlexFrame standards and lead to a project specific FlexFrame.

These project specific changes may affect the operation of the entire FlexFrame

landscape. An example: Consider you run a SFG (‘Request for Special Release’) for a

server which is unsupported in standard FlexFrame landscapes. Additionally you

changed some configuration in order to use some nice features of this server. Everything

runs fine. But: You have always to remember these changes. You have to tell the support

team (or other professionals working on this FlexFrame) about this. Without knowing

these project specific modifications a patch installation or upgrade installation may fail or

lead to unsatisfying results.

In order to keep track of these project specific changes a “Project Activities

Communication Interface” has been established. It creates a communication channel

from human beings to human beings (and to FlexFrame commands).



The most important command of this interface is ff_projectinfo.sh which will be

introduced in the following chapter.

11.10.1 The ff_projectinfo.sh command

Name

ff_projectinfo.sh - add entry to projectinfo database file

Synopsis

ff_projectinfo.sh -n <name> \

-t <text> | -r <text file> \

[-c <customer specific SFG#> ] \

[-s <severity> ] \

[-C ] \

[-v] [-d]

-n: (name)identify yourself (email address preferred).

Mandatory!

-t: (text) Text to be added (if it is a short one)

-r: (read text file) Text to be added will be read out of

this file

-c: (custom) If a custom specific Special Release

(Sonderfreigabe) has been applied this option has to

name the SFG number

-s: (severity) classification of project specific change

where severity = ‘info’ or ‘warning’ or ‘error’

-C: (CLEAR) change state of projectinfo database file to

READ

-v: (verbose)

-d: (debug)

Description

This script and the projectinfo database file are for use by FlexFrame certified

personnel only.

The objective of the project activities interface is to store and share information about

project specific activities at the FlexFrame landscape.



The ff_projectinfo.sh command is part of the project activities interface for the

FlexFrame solution. It helps to establish a reliable communication between the

professionals. This interface comprises

· this ff_projectinfo.sh command

· the FlexFrame projectinfo database file

· the ff_projectinfo_detector.sh

The ff_projectinfo.sh command serves as the only interface that creates or fills the

FlexFrame projectinfo database file.

Options

-n <name>

This option is mandatory: you need to identify yourself. The preferred argument is your

email address. If you want to use a name containing spaces you have to embed it into a

pair of double quotes.

-t <text>

Choose this option if the information you want to add is a short one and can be typed in

the command line. You will have to embed the text into a pair of double quotes (if it is not only a single word which seems unlikely). For longer texts use the -r option. But: Usage

of either -t or -r option is mandatory.

-r <text file>

Choose this option if you want to add information you prepared previously, or which is simply too long to be used at command line with the -t option. But: Usage of either -t or

-r option is mandatory. Note: You can add any text you like as long as file(1)

recognizes it as "text". Do not try to add core, dump, tar, files etc.

-c <custom specific Special Release number>

If a custom specific Special Release (Sonderfreigabe) has been applied to the system

this option is mandatory. It has to name the SFG number which you find in the Special

Release documents. It has typically a SFG-20??-??? appearance.

-s <severity>

This option is dedicated to classify the information you add: It helps the professionals

working later on the system to understand the significance resp. severity. The severity

option accepts the following values: info or warning or error. Without giving an explicit

severity a severity of info will be assumed.

-C



This option clears the default unread state of the projectinfo database file i.e. it sets

the state of the projectinfo database file to read.

Explanation: The projectinfo database file can have two states: unread or read.

Each appending of information (i.e. each successful call of the ff_projectinfo.sh

command without the -C option) will set the status to unread. This signals the

professionals the necessity to read it. Note: A FlexFrame with an unread projectinfo

database file is not entitled for an upgrade installation. Use the -C option not until you run

the FlexFrame upgrade. For details see ff_project_detector.sh(8) and the

“FlexFrame - Upgrade Guide”. This state of read will remain for a limited period only.

-v

The command will give some more detailed information during its work.

-d

The command will write detailed debug information into FlexFrame's default debug files.

More details and comprehensive use cases are available in the online manual pages of

ff_projectinfo.sh and ff_project_detector.sh.

Troubleshooting


12 Troubleshooting

12.1 Locking of Flex Frame Administration Commands

FlexFrame administrative commands stored at /opt/FlexFrame/bin use a resource

locking mechanism to prevent data corruption.

The implementation on the control nodes uses lock files on a shared file system mounted

from the common NAS. Each lock file represents a resource access. Each administrative

command uses typically only a single lock at a time to prevent deadlocks. The lock files

contain information about:

the locking user,

the executing host and the process id,

the timeout set,

the date and time of day when the lock expires,

the name and command line arguments of the locking process.

If a command tries to set a lock which is already set the command will retry to set the lock

for five times with a delay of one second. If the lock could not be set the command will

abort with a notice to user.

To get a robust locking mechanism locks will be removed if a timeout of 300 seconds was

reached AND the locking process no longer exists. If the locking process is still running,

an error message will be displayed by newly called program. The error message and the

exit code define the type or error as shown below.

These locking conditions get detected:

Exit Code Condition Description

0 normal

operation

All fine, locking works as expected. No resource collision with

other processes.

2 normal

operation

There is already a lock and the timeout is not reached.

3 Error IO error setting or unsetting a lock. This should not happen and

points to hanging mounts or filled up file systems.

Troubleshooting


Exit Code Condition Description

4 Error Lock set, timeout reached but locking process is still alive. May

be the locking process hangs. Only an administrator is able to

decide if locking process works or hangs. An administrator may

have to clear the lock manually

5 Error Lock set but lock file is empty, therefore no further checks are

possible. This should not happen as FlexFrame locking writes

data to lock files. An administrator has to clear the lock

manually.

Manual administration is necessary for all three error conditions. The administrator has to

check for source of trouble and resolve it. Some examples:

Resource File path

LDAP /FlexFrame/volFF/FlexFrame/locks/\

ldap_lock

ff_new_an.sh /FlexFrame/volFF/FlexFrame/locks/\

ff_new_an.sh_lock

ff_install_an_linux_images.sh /FlexFrame/volFF/FlexFrame/locks/\

ff_install_an_linux_images.sh_lock

The content of the lock files:

User: <login name>

Pid: <id of locking process>

Host: <node name where locking process lives>

Timeout: <time set in seconds> (lock valid till <date and time>)

<command line of locking process>

12.2 Script Logging in FlexFrame

All CN scripts in /opt/FlexFrame/bin log into one file, which is

/var/log/FlexFrame/log_(TIMESTAMP) by default, e.g.

/var/log/FlexFrame/log_2010_05_31. This file is changed daily.

Messages with priority greater than DEBUG (i.e. INFO, WARN and ERROR) are written

into this logfile.

Troubleshooting


All CN scripts in /opt/FlexFrame/bin debug (if called with option DEBUG) into one

file, which is /var/log/FlexFrame/debug_(TIMESTAMP) by default, e.g.

/var/log/FlexFrame/debug_2010_05_31 . This file is changed daily.

All messages are written into this debug file.

To get further debug information, all Perl scripts can be called with option –debug, as

well as some Shell scripts can be called with option –d .

Exception : The installation scripts ff_init_conf.pl, ff_post_conf.sh and

ff_slapd_init.sh log/debug to /opt/FlexFrame/init_conf/.

A log entry contains timestamp, priority, username, process ID, hostname, script name

and message text. It looks like this:

2009-04-22 12:29:52 INFO [root,9061,jer1cn1] ff_pool_adm.pl.sh

log example

Two configuration files, log4sh.conf and log4perl.conf reside under

/opt/FlexFrame/etc/.

The destination for Logging/Debugging can be changed in these configuration files by

modifying the corresponding filename lines.

You have to change the destination filenames in both configuration files.

Example:

If you want all scripts (excluding the exceptions) to log into

/FlexFrame/volFF/FlexFrame/log_(TIMESTAMP) you have to change

log4perl.appender.LogFileApp.filename=/var/log/FlexFrame/log

to

log4perl.appender.LogFileApp.filename=/FlexFrame/volFF/FlexFrame/l

og

in log4perl.conf

and

log4sh.appender.A1.File = '/var/log/FlexFrame/log'

to

log4sh.appender.A1.File = '/FlexFrame/volFF/FlexFrame/log'

in log4sh.conf

Troubleshooting


Attention:

The directory of the log-/debugfiles (in the example above

/FlexFrame/volFF/FlexFrame) has to exist.

The user is responsible for the back up/delete of older logfiles.

12.3 RBAC Logging with Syslog

It is possible to send the RBAC Logs to Syslog.

To do so, you have to make some changes in the file /opt/FlexFrame/etc/log4perl.conf :

- You have to add “SyslogApp” at the end of the first line

- You have to activate the last 3 lines of the file by removing the “#” .

- You have to adapt the facility of SyslogApp, e.g. to local1 , so that it fits to your

adjustments in /etc/syslog-ng/syslog-ng.conf

12.4 Log Files

The default location for logging information of the FlexFrame scripts is /var/log/FlexFrame on the Control Nodes.

Additional information can be found under /var/log/FlexFrame/tmp .

For easy gathering of log file information on all Application Nodes in FlexFrame, it is not

necessary to login on each Application Node because all log files are directly readable at

the Control Nodes on the mounted NAS system.

To elevate the admin to read these log files it is useful to create symbolic links on the

Control Nodes in the directory /var like this:

control1:/ # cd /var

control1:/var # ls -ld log*

drwxr-xr-x 20 root root 1440 May 2 14:01 log

lrwxrwxrwx 1 root root 58 Apr 29 17:28

log_pool1_klinge1 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-

11.x86_64/var_img/var-ac10020e/log



11.x86_64/var_img/var-ac10020f/log



11.x86_64/var_img/var-ac100210/log

Troubleshooting








lrwxrwxrwx 1 root root 58 Apr 29 17:31 log_otto_RX300-

01 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-


lrwxrwxrwx 1 root root 58 Apr 29 17:33 log_otto_RX300-

02 -> /FlexFrame/volFF/os/Linux/FSC_1.0A00-017.SLES-


control1:/var #

Troubleshooting


To read the /var/log/messsages from Application Node klinge5 in pool pool1

invoke this:

control1:/ # cd /var

control1:/var # less log_pool1_klinge5/messages

Or enter the log directory from this Application node and look for log files:

control1:/ # cd /var/log_pool1_klinge5

control1:/var/log_pool1_klinge5 # ls -lrt|tail

-rw-r--r-- 1 root root 278 May 4 16:23 log.scagt

-rw-r--r-- 1 root root 174 May 4 16:24 log.vvagt

-rw-r--r-- 1 root root 257 May 4 16:24 log.statusagt

-rw-r--r-- 1 root root 612 May 4 16:28 ntp

-rw-r--r-- 1 root root 5542 May 4 16:35 warn

-rw-r--r-- 1 root root 1696 May 4 16:35 auth.log

-rw-r--r-- 1 root root 484 May 4 18:16 log.busagt

drwxr-xr-x 2 root root 4096 May 4 18:19 sa

-rw-r--r-- 1 root root 27622 May 4 18:19 messages

prw------- 1 root root 0 May 4 18:24 psadfifo

12.5 Network Errors

In most cases, network problems are most times configuration mistakes. All switches are

configured to send SNMP traps and log via syslog to Control Nodes. To see what happened, look at the Control Nodes' /var/log/messages. Any conditions reported by

the switches can be found here. For SNMP traps look at the FlexFrame Agents support

database. The FlexFrame Agents collect all kinds of SNMP traps for the entire FlexFrame

environment.

12.6 NFS Mount Messages

During start/stop procedures of SAP instances, NFS mount failures for /usr/sap/SYS

and /oracle/db_sw can be seen in /var/log/messages. These messages are not

FlexFrame-specific failures, but may occur on any system that has /usr/sap and

/oracle in its automount maps. These messages are caused by the way SAP is linking

binaries during software development process.

12.7 LDAP Error Codes and Messages

LDAP failures may have various reasons. To determine the reason, first have a look at

the Control Node's /var/log/messages. If slapd does not report anything you may

have to increment the log level configured with /etc/openldap/slapd.conf. Log

levels are additive, and available levels are:

Troubleshooting


Log level Meaning

1 Trace function calls

2 Debug packet handling

4 Heavy trace debugging

8 Connection management

16 Print out packets sent and received

32 Search filter processing

64 Configuration file processing

128 Access control list processing

256 Stats log connections/operations/results

512 Stats log entries sent

1024 Print communication with shell backends

2048 Entry parsing

Most LDAP problems are related to access control and connection management.

If you have changed the log level send a -HUP to the slapd to advice it to reread

configuration files (use pkill -HUP slapd).

Another reason may be the LDAP client configuration. Check the following issues:

Does the client use the correct server address?

Look at the client messages file to get this information.

If ldapsearch -x on Linux and works properly the problem may be the nsswitch

configuration. These are the top most problems.

Troubleshooting


When reporting problems with the administration scripts, please provide the

following information:

The error message of the admin tool

A dump of the LDAP database (ldapsearch -x > <file name>)

The configuration files /etc/openldap/slapd.conf and

/FlexFrame/volFF/FlexFrame/ldap/common/slapd.acl.conf

12.8 LDAP and Cache Coherence

While using command scripts to administer your FlexFrame environment, it is possible

that you do not see the changes on Application Nodes immediately. The reason is the

usage of a cache daemon (nscd) which refreshes the cache content in defined time

periods.

There are different caches for different types (e.g. passwd, group, etc.) The configuration of the cache properties is done in /etc/nscd.conf (default). Normally you must not

change the FlexFrame default configuration. Please ask professional service collegues

before doing any change.

You can display your current configuration with nscd –g.

For further information see the corresponding manpages (nscd, nscd.conf).

12.8.1 Linux

To force the refresh of the cache content use nscd –i or nscd restart.

12.9 Start/Stop Script Errors

While executing the SAP service start/stop scripts, several errors may occur for arbitrary

reasons. The messages are grouped by the severity of the message.

The scripts write three types of logs (/FlexFrame/script/logs)

sapservice.log

The log is always available and logs the requests. It contains only the requested

command and the exit state of the request

sap<type>_<SID>_<action>.log

That log contains the messages also written to STDOUT/STDERR

sapservice.<nnn>.debug.<SID>.<type>.<instnr>.action>

That log is only written if option -d is set. It contains a detailed logging of the

course of actions.

The message text below uses ‘%%’ to indicate there will be a variable output in the

message. There are up to four varying strings in a message.

Troubleshooting


If you are creating a debug file you are requested to delete debug files by yourself. Otherwise they resist in /FlexFrame/script/log forever.

12.9.1 Severity ‘INFO’

Messages from severity ‘INFO’ should show the progress of a request and they are always written to STDOUT and to the debug file if option -d is set.

12.9.2 Severity ‘WARN’

Messages from severity ‘WARN should give you a hint to possible configuration errors.

They are always written to STDOUT and to the debug file if option -d is set.

Most of the WARN messages are not critical but you should check if such a message is

displayed.

12.9.3 Severity ‘ERROR’

Messages from severity ‘ERROR’ are always written to STDOUT and to the debug file if

option -d is set. You have to check it and resolve the issue which leads to the error

message. With opcode ‘cleanup’ sometimes messages with severity ‘ERROR’ are shown

but can be ignored.

12.9.4 Severity 'DEBUG'

Debug messages are only printed to the debug file if option -d is used. Those message

are only written to support diagnostics.

12.10 Script Debugging

12.10.1 Shell Scripts

For many FlexFrame scripts, debugging is activated by default. If debugging is not active,

it can be activated with the option –d:

ff_netscan.sh -d

The default location for the debug log file is /var/log/FlexFrame

Further on, shell scripts can be traced using the option -x.

For details, see the man page of the shell in use (man sh; man bash; etc.)

sh -x ff_netscan.sh

Troubleshooting


12.10.2 Perl Scripts

Before debugging, activate the logging functions of the shell by calling script (For

details, see man script).

The debugger is called using

perl -d <script_name>

Functions:

h

Help

x <expression>

Evaluate an expression (hash/array) and display the result

p <string>

Returns strings

s

Execute next command; follow subroutines

n

Execute next command; skip subroutines

b <line_number>

Set break point at line <line number>

q

Quit.

For further information, see the man page of perldebug.

12.11 Debugging the Linux Kernel

For the Linux Kernel, debugging is disabled by default. There are several approaches to

collect debugging information. Everything is only described briefly; please contact Fujitsu

support for further instructions.

12.11.1 Netconsole

If the Linux kernel detects an internal error, it tries to write some information about the so

called “Oops” into the messages file.

Troubleshooting


If an error completely crashes the kernel (which is known as a kernel panic), this

information may be generated but cannot be written to the disk or the NFS mount.

To get this information anyway, activate the netconsole feature to redirect kernel logging

over the network.

If the netconsole output does not provide the required information, you should configure

the crash dump functionality.

Information about using the netconsole can be found in the Linux Kernel documentation

in /usr/src/linux/Documentation/networking/netconsole.txt.

Please note that due to the use of bonding interfaces in FlexFrame there may be some

limitations, netconsole may not work properly with the bonding slaves. It is recommended

to use a dedicated interface.

12.11.2 Capturing Crash Dumps

12.11.2.1 Common Restrictions for Taking Crash Dumps

If taking crash dumps is configured and activated, the process of writing the dump

must not be interrupted, so the Linux HA cluster and FlexFrame Agent power shutdown

features must be disabled for this specific node, for example by pulling the IPMI cable or

setting appropriate timeouts. On the Control Nodes, STONITH should be disabled if the

crash dump cannot be written within 45 seconds which is very likely.

This can be done by disabling the “Enable STONITH” checkbox of the “linux-ha” proper-

ties in the Heartbeat GUI interface.

Amongst others that is one reason for not activating crash dump capturing by default.

12.11.2.2 Kernel Crash Dump Capturing (Kdump)

Since SLES10, the implementation how to take crash dumps was changed to the kdump

method. If a kernel panic occurs, a special kdump-kernel will be directly executed by the

running kernel and during the boot process of this kernel, the dump of the whole system

memory will be written to a sparse file in /var/log/dump.

This feature is well integrated in FlexFrame Application Nodes and may be activated by

specifying a kernel command line option in the PXE configuration file:

Open the corresponding file in /tftpboot/pxelinux.cfg

Append crashkernel=128M@16M to the APPEND line

Reboot the corresponding Application Node(s)

12.11.2.3 Forcing a Crash Dump

There may be situations where the kernel does not initiate a crash dump automatically.

Troubleshooting


You can manually start a crash dump by using the sysrq shortcuts.

Activate sysrq as follows:

cn1:~ # echo 1 >/proc/sys/kernel/sysrq

To make this setting persistent, set ENABLE_SYSRQ="yes" in

/etc/sysconfig/sysctl.

Now a manual's crash dump capture can be manually initiated by pressing

Alt+SysRq+c altogether on the keyboard. On most keyboards, the SysRq key is equal

to the PrintScrn key.

If the graphical desktop is running, it is recommended to switch to a text console by pressing Ctrl+Alt+F10 before starting the dump, otherwise it might not be possible to

see anything while the kdump-kernel is being executed and the dump is being written.

12.12 Activate core dumps on CN or AN

To investigate problems (e.g. bus errors) with Linux commands it is sometimes helpfull to

get a ‘core dump’. Normally the core file creation is disabled by default.

To activate core dumps in the current running shell use the following command:

In a ‘bash’ shell use

an_Linux# ulimit –c unlimited

In a ‘csh’ shell use

rx300-15 /home_sap/sapadm> unlimit coredumpsize

Now run the faulty command within this shell. The core dump can be found in the

directory /var/log/coredumps.

The filename follows the format:

core-<executable filename>-<time of dump in seconds>

Troubleshooting


If it is necessary to enable core file creation for a complete login session, you have to globally enable core dump writing .Change the file /etc/security/limits.conf on

the CN or /FlexFrame/volFF/os/Linux/FSC*/root_img/etc/security/limits.conf

for an Application Node and add the line

* soft core unlimited

In the next login session the creation of core dump files is enabled. To check use the command ulimit –a for a bash shell environment or the command limit for a csh

shell environment.

To disable this functionality remove the above entry from the limits.conf file.

12.13 Huge Amount of getattr Calls

If you expect a very huge amount of getattr NFS calls on your network or on your NAS

system, please check the following issues:

Application Node Image

Please use the current application node image available for your FlexFrame release. We

have done a lot of smooth improvements over the last years to limit the getattr calls within

the image to an absolutely needed minimum.

SAP-HANA

If using SAP HANA within FlexFrame, please check the SAP HANA profile.

The file sapprofile.ini under $SAP_RETRIEVAL_PATH (e.g.

/usr/sap/HD1/HDB02/hdb02hd1-001-se) should contain an entry like

HDB/NameServer/RWLockPath=/dev/shm

Please note, that this entry should contain a destination, which is not residing on an NFS

file system.

For more information please look at the FlexFrame documentation “FlexFrame® –

Installation Guide for SAP Solutions” (in the ‘post installation’ section of the HANA

installation)

Miscellaneous

Please check, if you are running additional tools or services (e.g. Citrix Server, Backup

solutions, Tools analyzing file system content by accessing all directories and files, etc..)

on the FlexFrame internal network or against your NAS system used in FlexFrame.

If possible, please turn off everything except the FlexFrame base functionality to find out,

if it is really FlexFrame, who caused the high number of getattr calls.

Abbreviations


13 Abbreviations

ABAP Advanced Business Application Programming

ACC Adaptive Computing Controller

ACI Adaptive Computing Infrastructure

ACPI Advanced Configuration and Power Interface

APM Advanced Power Management

APOLC Advanced Planner & Optimizer Life Cache

CCU Console Connection Unit

CIFS Common Internet File System

cDOT Clustered Data ONTAP

DHCP Dynamic Host Configuration Protocol

DIT Domain Information Tree

ERP Enterprise Resource Planning

EULA End User License Agreement

FA FlexFrame Agents

FC Fiber Channel

FF FlexFrame

FF4S FlexFrame for SAP

FFO FlexFrame Orchestrator

FSC FlexFrame Software Container, name part used for

Images and CDs/DVDs

FTP File Transfer Protocol

IP Internet Protocol

KVM Kernel based Virtual Machine

LAN Local Area Network

LDAP Lightweight Directory Access Protocol

LUN Logical Unit Number

LVM Landscape Virtualization Management

Abbreviations


LVM Linux Volume Manager

MAC Media Access Control

MINRA Minimal Read Ahead

NAS Network Attached Storage

NDMP Network Data Management Protocol

NFS Network File System

NIC Network Interface Card

NVRAM Non-Volatile Random Access Memory

OLAP Online Analytical Processing

OLTP On-Line Transaction Processing

ONTAP Open Network Technology for Appliance Products

OSS Open Source Software

PXE Preboot Execution Environment

PY PRIMERGY

QA Quality Assurance

QS Quality of Service

RAID Redundant Array of Independent (or Inexpensive) Disks

RDBMS Relational Database Management System

SAN Storage Area Network

SAP BW SAP Business Warehouse

SAPGUI SAP Graphical User Interface

SAPOSS SAP Online System Service

SID System Identifier

SLD System Landscape Directory

SLES SUSE Linux Enterprise Server

SMB Server Message Block

SNMP Simple Network Management Protocol

SPOC Single Point Of Control

SVM Storage Virtual Machine

Abbreviations


TELNET Telecommunications Network

TFTP Trivial File Transfer Protocol

UDP User Datagram Protocol

UPS Uninterruptible Power Supply

vCC / vCN virtual Control Center / virtual Control Node

VLAN Virtual Local Area Network

VTOC Virtual Table Of Contents

WAN Wide Area Network

WAS Web Application Server

WAFL Write Anywhere File Layout


14 Glossary

Adaptive Computing Controller (ACC)

SAP system for monitoring and controlling SAP environments, mostly superceeded

by LVM nowadays.

Advanced Business Application Programming (ABAP)

Proprietary programming language of SAP.

Advanced Power Management

Advanced Power Management defines a layer between the hardware and the

operating system that effectively shields the programmer from hardware details.

Application Agent

A software program for monitoring and managing applications.

Application Node (AN)

A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This

definition includes Application Servers as well as Database Servers.

Automounter

The automounter is an NFS utility that automatically mounts directories on an NFS

client as they are needed, and unmounts them when they are no longer needed..

Blade

A special form factor for computer nodes.

BladeRunner

The working title for the solution part of SAP for FlexFrame.

BRBACKUP

SAP backup and restore tools.

Client LAN

Virtual network segment within FlexFrame, used for client-server traffic.

Clustered Data ONTAP (cDOT)

One of two modes of NetApp’s NAS operating system ONTAP. In clustermode either

a single controller pair or multiple controller pairs are connected (clustered) via a

back-end 10GbE network; The other mode is called 7-mode; this mode will gradually

be replaced by the clustermode.

Common Internet File System (CIFS)

A protocol for the sharing of file systems (same as SMB).

Computing Node

From the SAP ACI perspective: A host that is used for applications.

Control Agent

A software program for monitoring and managing nodes within FlexFrame.

Glossary


Control Center (CC)

The two Control Nodes (CN) of FlexFrame are also named as the FlexFrame Control

Center (CC). In this documentation the notation Control Node (CN) is used as a

synonym for Control Center (CC) and vice versa.

Control LAN

Virtual network segment within FlexFrame, used for system management traffic.

Control Node (CN)

A physical computer system, controlling and monitoring the entire FlexFrame

landscape and running shared services in the rack (dhcp, tftp, ldap etc.).

Control Station

A Control Node in an SAP ACI environment.

Dynamic Host Configuration Protocol (DHCP)

DHCP is a protocol for assigning dynamic IP addresses to devices on a network.

Dynamic Host Configuration Protocol server

A DHCP server provides configuration parameters specific to the DHCP client host,

required by the host to participate on the Internet.

Enterprise Resource Planning (ERP)

Enterprise Resource Planning systems are management information systems that

integrate and automate many of the business practices associated with the

operations or production aspects of a company.

Ethernet

A Local Area Network which supports data transfer rates of 10 megabits per second.

Fiber Channel (FC)

Fiber Channel is a serial computer bus intended for connecting high-speed storage

devices to computers.

Filer

Network attached storage for file systems of NetApp.

FlexFrame® (FF)

The name FlexFrame® is a generic term for both „FlexFrame

® for SAP

®“ and

„FlexFrame® Orchestrator“.

FlexFrame Agents (FA)

Central system management and high availability software component of FlexFrame

FlexFrame® Orchestrator (FFO)

This is the advancement of the Fujitsu solution FlexFrame for SAP and means a new

approach to offer enhanced functionality and features step by step and become more

and more independent from certain hardware and software components.

FlexFrame® for SAP

® (FF4S)

FlexFrame® for SAP

® is a Fujitsu solution and means a radically new architecture for

Glossary


SAP environments. It exploits the latest business-critical computing technology to

deliver major cost savings for SAP customers. FlexFrame for SAP is a joint project in

which the main partners are SAP, Network Appliance, Intel and Fujitsu.

FlexFrame internal LAN Switch

Network switches which are integral part of the FlexFrame hardware configuration

and which are automatically configured by the FlexFrame software.

Gigabit Ethernet

A Local Area Network which supports data transfer rates of 1 gigabit (1,000

megabits) per second.

Host name

The name of a node (assigned to an interface) that is resolved to a unique IP

address. One node can have multiple host names (cf. node name). In SAP

environments host names are currently limited to 13 alphanumeric characters

including the hyphen (“ - “). The first character must be a letter. In the SAP

environment host names are case-sensitive.

Image

In the FlexFrame documentation, “Image” is used as a synonym for “Hard Disk

Image”.

Internet Protocol Address

A unique number used by computers to refer to each other when sending information

through networks using the Internet Protocol.

Kernel based Virtual Machine (KVM)

Virtualization solution which turns the Linux kernel into a hypervisor.

Lightweight Directory Access Protocol (LDAP)

Protocol for accessing on-line directory services.

Local Area Network (LAN)

A computer network that spans a relatively small area. Most LANs are confined to a

single building or group of buildings. However, one LAN can be connected to other

LANs over any distance via telephone lines and radio waves. A system of LANs

connected in this way is called a Wide Area Network (WAN).

Local host name

The name of the node (physical computer); it can be displayed and set using the

command /bin/hostname.

Logical Unit Number (LUN)

An address for a single (SCSI) disk drive.

MAC address

Device identifier number of a Network Interface Card. In full: "media access control

address".

Glossary


MaxDB

A relational database system from mySQL (formerly ADABAS and SAPDB).

Media Access Control address

An identifier for network devices, usually unique. The MAC address is stored

physically on the device.

NAS system

Network Attached Storage of any vendor (in our context: NetApp Filer).

NDMPcopy

NDMPcopy transfers data between Filers using the Network Data Management

Protocol (NDMP).

Netboot

A boot procedure for computers where the operating system is provided via a

network instead of local disks.

Netweaver

SAP NetWeaver is the technical foundation of SAP solutions.

Network Appliance Filer

See “Filer”.

Network Attached Storage (NAS)

A data storage device that is connected via a network to one or multiple computers.

Network File System (NFS)

A network protocol for network-based storage access.

Network Interface Card (NIC)

A hardware device that allows computer communication via networks.

Node

A physical computer system controlled by an OS.

Node name

The name of a physical node as returned by the command uname -n. Each node

name within a FlexFrame environment must be unique.

Non-Volatile Random Access Memory

A type of memory that retains its contents when the power is turned off.

On-Line Analytical Processing (OLAP)

The use of analytical informationssystems via computer networks.

On-Line Transaction Processing (OLTP)

Transaction processing via computer networks.

OpenLDAP

An Open Source LDAP Service Implementation.

Glossary


Open Network Technology for Appliance Products (ONTAP)

The operating system of Network Appliance Filers.

Open Source Software

Software that is distributed free of charge under an open source license, such as the

GNU Public License.

Oracle RAC

A cluster database by Oracle Corporation.

Physical host

Name of a physical computer system (node).

Power-On Self Test

Part of a computer's boot process; automatic testing of diverse hardware

components.

Preboot Execution Environment (PXE)

An environment that allows a computer to boot from a network resource without

having a local operating system installed.

PRIMERGY

Fujitsu's i386-based server product line.

SAP Service

In FlexFrame: SAP Service and DB Services.

SAP service script

An administration script for starting and stopping an SAP application on a virtual host.

SAP Solution Manager

Service portal for the implementation, operation and optimization of an SAP solution.

SAPLogon

Front-end software for SAPGUI.

SAPRouter

Router for SAP services like SAPGUI or SAPTELNET.

Server

A physical host (hardware), same as node.

Service

A software program providing functions to clients.

Service type

The type of an application or service (db, ci, app, agate, wgate etc.).

Single Point of Control

In FlexFrame: One user interface to control a whole FlexFrame environment.

Glossary


Storage LAN

A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS

systems.

SUSE Linux Enterprise Server (SLES)

A Linux distribution by Novell, specializing in server installations.

Telecommunications Network

A terminal emulation program for TCP/IP networks such as the Internet.

Trivial File Transfer Protocol (TFTP)

A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram

Protocol (UDP) and provides no security features. It is often used by servers to boot

diskless workstations, X-terminals, and routers.

TFTP server

A simple FTP implementation.

Virtual host

The name of the virtual host on which an application runs; it is assigned to a physical

node when an application is started.

Virtual Local Area Network (VLAN)

A VLAN is a logically segmented network mapped over physical hardware according

to the IEEE 802.1q standard.

Virtualization

Virtualization means the separation of hardware and processes. In a virtualized

environment (FlexFrame), a process can be moved between hardware nodes while

staying transparent to the user and application.

.


15 Index

abbreviations 341

Application

removing from monitoring by

FlexFrame Agents 264

stopping ans starting for upgades

using r3up 265

Application Node

hardware maintenance 317

software maintenance 289

Application Node Images

installation 291

Application Nodes 10

adding 88

administrating 83

displaying information on a specific

83

displaying information on all 87

listing 83

reactivating after power shutdown by

FlexFrame Agents 46

removing 92, 93, 94, 95, 191

state of 49

updating RPM packages 311

application services

starting and stopping 252

application software

upgrading 310

automounter concept 33

Blade Server Cabinets

adding 99

administrating 97

listing 97

removing 102

cloning of SIDs 229, 238

cluster

configuring 26

starting 26

status information 27

cluster file system, built 37

Cluster Status 53

common network configuration

parameters

display and change 172

constraints 15

Control Node

defect hardware 278

exchanging 278

failed hardware 278

hardware maintenance 278

OS damaged 278

replacing network card 279

software updates 269

Control Nodes 9

backup 275

backup/restore 274

core dump 339

crash dump 338


create new initrd 308

Debugging

Linux kernel 337

script 336

document history 2

ESXi server maintenance 320

BIOS update 320

replace network card 320

update and patches 320

FA Autonomous Agents 49

ff_change_id.pl 229

ff_clone_sid.pl 206, 207, 228, 230

ff_hosts.sh 72

ff_list_services.sh 251

ff_nas_adm.pl 129

ff_sap_shadowport.sh 245

ff_sap_upgrade.pl 246

ff_sid_adm.pl 196

ff_swgroup_adm.pl 162, 184

ff_swport_adm.pl 185

ff_user_adm.pl 74

Filer cluster 38

FlexFrame

architecture 5

backup with tape library 54

basic administration 42

general notes 5

network 51

FlexFrame configuration state,

displaying 47

FlexFrame landscape

accessing 42

power-off 43

power-on 42

FlexFrame Web portal 48

getattr 340

glossary 345

host

database 71

install kernel 308

install new initrd 308

installing new linux kernel 308

Jakarta Tomcat 49

kernel switching 309

LDAP

FlexFrame structure in 11

working with 13

LDAP and Cache Coherence 335

LDAP configuration 20

LDAP error codes and messages 333

LINUX Cluster Components /

Functionality 13

Linux kernel

install new kernel 272

mount software stage 271

reboot control node 272

update/install new 271

update/install new 306

linux-ha cli commands 27

Linux-HA cluster 49

Index


Linux-HA cluster on control center 13

linux-ha gui 29

linux-ha logfile 29

locking 328

log files 331

maintenance cycle 289, 290, 291, 293,

295, 296, 299

assigning, booting, maintaining (step

#2) 295

maintaining use cases 306

maintenance base image (step #1)

293

migrating remaining Application

Nodes (step #3) 302

overview 289

re-using maintenance images 304

reverting (step #3) 299

NAS

add new 129

changing LinkAggregate ports 139

display all configured 132

display configuration 134

remove 131

NAS systems

multiple 243

NAS systems configuration 129

netboot change BIOS settings 317

network 29

automounter concept 33

LAN failover 30

segments 30

switches 31

Network Appilance Filer 35

network card

replacing 317

network errors 333

NFS mount messages 333

notational conventions 1

ONTAP patches

installing 321

Partner Linux Driver Process (PLDP)

272, 309

PLDP 272, 309

pool

add to a NAS 135

remove from a NAS 135

pool default router

changing 69

pool DNS domain

changing 68

pool groups

removing 71

pools

adding 55, 58

adding a group 70

listing details 63, 66

removing 61

state of 49

pools and groups 58

power control hardware


replacing 319

related documents 2

remote administration 42

replace

ESXi network card 320

network card 317

power control hardware 319

switch blade 319

requirements 1

resource ff_manage 22

resource groups 18

resource netboot 22

SAN

basic layers 40

rules and restrictions with FlexFrame

41

SAP kernel

updates and patches 249

SAP service

scripts 257, 261

starting and stopping without root

privileges 255, 257

SAP service scripts

actions 259

return code 262

user exits 261

SAP services

administrating 250

display status 250

list status 251

starting and stopping multiple 262

SAP SIDs

adding instances 198, 205, 206, 211,

214, 218

modifying instances 198, 205, 206,

211, 214, 218

removing instances 198, 205, 206,

211, 214, 218

SAP SIDs

adding 198

listing 196

listing instances 196

modifying 198

removing 198

SAP SIDs

adding 205

SAP SIDs

removing 205

SAP SIDs

modifying 205

SAP SIDs

adding 206

SAP SIDs

removing 206

SAP SIDs

modifying 206

SAP SIDs

adding 211

SAP SIDs

removing 211

Index


SAP SIDs

modifying 211

SAP SIDs

adding 214

SAP SIDs

removing 214

SAP SIDs

modifying 214

SAP SIDs

adding 214

SAP SIDs

removing 214

SAP SIDs

modifying 214

SAP SIDs

adding 218

SAP SIDs

removing 218

SAP SIDs

modifying 218

SAP SIDs

adding 218

SAP SIDs

removing 218

SAP SIDs

modifying 218

SAP SIDs

adding 218

SAP SIDs

removing 218

SAP SIDs

modifying 218

SAP SIDs

cloning 228

SAP SIDs

listing instances 228

Sap system

handling 195

SAP system

upgrading 245

SAP systems

state of 50

score 15

script logging 329

scripts

ff_list_services.sh 251

ff_sap_shadowport.sh 245

ServerView

update 309

update agents 310

update RAID manager 310

ServerView update via RPM 270

service packs 271, 306

service switchover 265

severity

DEBUG 336

ERROR 336

INFO 336


WARN 336

shared operating system 9

SID instances

state of 50

simple resources 15

snapshot 37

software stage 306

spare nodes

pool-independent 81

specific configuration 18

start/stop script errors 335

state

of Applicatrion Nodes 49

of pools 49

of SAP systems 50

of SID instances 50

stickiness 16

stonith 23

storage systems administration 129

switch

add to a switch group 162

remove from a switch group 165

switch administration 162

switch blade

change blade uplink 106

changing name 103

changing password 104

changing type 103

getting initial configuration 105

replacing 319

switch configuration

backup 321

restoring 322

switch group

adding 174

adding an uplink 180

change host name 170

change password 169

deleting an uplink 183

extending an uplink 181

list configuration 166, 168

moving a blade cabinet to another

switch group 107

removing 179

switch port

add configuration 185

display configuration 188, 189

remove configuration 187

Third Party products 323

troubleshooting 328

update and maintenance 269

update PXE config file 308

updating RPM packages 273, 311

patch server connection 273, 311

patch server setup 273, 311

perfrom online update 274, 312

prepare online update 273, 311

updating vmware-open-vm-tools 312

upgrade FF landsacpe 269

Index


user and groups administration 74

volFF

unloading 249

volume layout 37

volumes

multiple 243