sap suse linux

SAP on SUSE LinuxEnterpriseRunning SAP NetWeaver on SUSE Linux Enterprise Server with High Availability - DRBDdual data center

www.suse.com11 SP1

December 20, 2011 Best Practice

Running SAP NetWeaver on SUSE Linux EnterpriseServer with High Availability - DRBD dual data center

List ofAuthors:FabianHerschel ([email protected]),MarkusGuertler ([email protected]),Lars Pinne ([email protected])

Copyright © 2010–2011 Novell, Inc. and contributors. All rights reserved.

This guide has been created in cooperation with LINBIT HA-Solution GmbH. Many thanks for allgood contributions.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNUFree Documentation License, Version 1.2 or any later version published by the Free Software Foun-dation; with the Invariant Section being this copyright notice and license. A copy of the license is in-cluded in the section entitled “GNU Free Documentation License”.

For Novell trademarks, see the Novell Trademark and Service Mark list http://www.novell.com/company/legal/trademarks/tmlist.html. Linux* is a registered trademark ofLinus Torvalds. All other third party trademarks are the property of their respective owners. A trademarksymbol (®, ™ etc.) denotes a Novell trademark; an asterisk (*) denotes a third party trademark.

All information found in this book has been compiled with utmost attention to detail. However, thisdoes not guarantee complete accuracy. Neither Novell, Inc., SUSELINUXProducts GmbH, the authors,nor the translators shall be held liable for possible errors or the consequences thereof.

http://www.novell.com/company/legal/trademarks/tmlist.html

http://www.novell.com/company/legal/trademarks/tmlist.html

Contents

Executive Summary vii

Part I Introduction and Concept 1

1 Introduction 31.1 SAP on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 SUSE Linux Enterprise Server . . . . . . . . . . . . . . . . . . . . . 51.3 SUSE Linux Enterprise Server High Availability Extension . . . . . . . . . 71.4 Database for SAP . . . . . . . . . . . . . . . . . . . . . . . . . 151.5 Sizing, Deployment and Migration Considerations . . . . . . . . . . . 16

2 One Concept — Two Use Cases 192.1 High Availability Concept Overview . . . . . . . . . . . . . . . . . 202.2 Use Case 1 “DRBD dual data center with two clusters - 2x2” . . . . . . . 242.3 Use Case 2 “DRBD dual data center with one cluster - 2+1” . . . . . . . 26

Part II Installation of "DRBD dual data center" with MaxDB 29

3 Installation Overview 31

4 Planning 33

5 Prerequisites 355.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . 355.2 Software Requirements, Connection Data, and all the Rest . . . . . . . 36

6 Download the Needed SAP Installation Media 39

7 Install SUSE Linux Enterprise Server 11 SP1 for SAP 437.1 File System Layout . . . . . . . . . . . . . . . . . . . . . . . . 437.2 Software Selection . . . . . . . . . . . . . . . . . . . . . . . . 447.3 Runlevel and System Services . . . . . . . . . . . . . . . . . . . . 477.4 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . 487.5 Check SLES for SAP Installation . . . . . . . . . . . . . . . . . . . 48

8 Prepare SAN Storage 498.1 Multipath Configuration . . . . . . . . . . . . . . . . . . . . . . 508.2 Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . 528.3 MD Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 528.4 DRBD Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 558.5 LVM Configuration . . . . . . . . . . . . . . . . . . . . . . . . 588.6 File System Creation . . . . . . . . . . . . . . . . . . . . . . . . 598.7 Check the SAN Storage . . . . . . . . . . . . . . . . . . . . . . 60

9 Configure the Cluster on both Sites 639.1 Install the SUSE Linux Enterprise High Availability Extension Software Packages on

all Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649.2 Basic Cluster and CRM Configuration . . . . . . . . . . . . . . . . . 659.3 STONITH Resource Configuration . . . . . . . . . . . . . . . . . . 689.4 Storage Resource Configuration . . . . . . . . . . . . . . . . . . . 699.5 Configure Virtual IP Addresses . . . . . . . . . . . . . . . . . . . 73

10 Install SAP NetWeaver 7.0 EHP1 7510.1 SAP Installation Procedure . . . . . . . . . . . . . . . . . . . . . 7510.2 Installation of the Central Services Instance ASCS . . . . . . . . . . . 7610.3 Installation of the Database . . . . . . . . . . . . . . . . . . . . . 7810.4 Installation of the Central Instance . . . . . . . . . . . . . . . . . . 8010.5 Check the SAP Installation . . . . . . . . . . . . . . . . . . . . . 82

11 Integrating SAP into the cluster 8511.1 SAP resource configuration . . . . . . . . . . . . . . . . . . . . . 85

12 Testing the Cluster 8712.1 Check the SAP Integration . . . . . . . . . . . . . . . . . . . . . 8712.2 Application Cluster Tests . . . . . . . . . . . . . . . . . . . . . . 91

Part III Appendix 93

A Software Downloads 95

B Novell Products Online Documentation 97

C SAP Notes 99

D Links to SUSE Linux Enterprise Server, SAP, Databases 103

E Sample CRM Configuration for SAP Simple Stack High Availability 109

F Licenses 113F.1 GNU Free Documentation License . . . . . . . . . . . . . . . . . 113F.2 GNU General Public License . . . . . . . . . . . . . . . . . . . . 116

Terminology 121

Executive SummarySAP Business Suite is a sophisticated application platform for large enterprises andmid-size companies. Many critical business environments require the highest possibleSAP application availability. SUSE Linux Enterprise High Availability Extension,when running onmodern x86-64 hardware platforms, satisfies this requirement. Togetherwith a redundant layout of the technical infrastructure, single points of failure can beeliminated.

SAP NetWeaver is a common stack of middleware functionality used to support theSAP business applications. This guide describes a SAPNetWeaver installation on SUSELinux Enterprise Server 11 SP1 with the additional SUSE Linux Enterprise HighAvailability Extension. We will also describe possible failure scenarios and methodsto avoid them. The described concept has proven its maturity during several years ofproductive operations for customers of different size and branches.

This document focuses on deploying SAP in a dual-site cluster, where each site hasredundant access to a SAN storage with two storage heads, using host-based mirroringwith MD for intra-site synchronous replication and the DRBD technology to replicateasynchronously off-site. We will cover the commonly used disaster-recovery strategy,having one site with a two-node cluster at one geographical location and another sitewith another two-node cluster at another geographical location. After a complete failureof the first site (both cluster nodes down), the second site can takeover the operationof the SAP applications and databases. The site fail-over procedure is half-automatedand requires some manual interaction. The local site fail-over is completely automated.

Two sites with two two-node clusters require four nodes in total. Four dedicated serversand two times a redundant SAN infrastructure with four storage heads in total can bequite expensive. A cheaper, but less redundant variant of this scenario is, to use onlyone two-node cluster in the first site and a non-redundant standalone server in the secondsite. The server at the second site could also be equipped with a local storage, thus notrequiring a SAN.

This is a technical document designed to allow system administrators and technicalconsultants to integrate highly available SAP bussiness applications into an existingSUSE Linux Enterprise Server infrastructure.

The described storage stack and SAP configuration can be used with or without a highavailability cluster. It is possible to add high availability functionality to an alreadyrunning system if the installation complies with the described solution.

This guide will show you how to:

• plan a SUSE Linux Enterprise platform for SAP workload,

• set up a Linux high availability infrastructure for SAP, including a storage stack thatincludes DRBD,

• perform a basic SAP NetWeaver installation on SUSE Linux Enterprise.

This guide will also help you to install the following software components:

• SUSE Linux Enterprise Server 11 SP1

• SUSE Linux Enterprise High Availability Extension

• MaxDB (Oracle and DB2 are supported, too)

• SAP NetWeaver 7.0 EHP1 (other versions are supported, too)

This guide is aimed at IT professionals with skills in:

• SAP basic operating,

• data center system concepts and configuration,

• Linux knowledge at LPI1 or CLE level.

To apply this guide, you need access to the following resources:

• SUSE Linux Enterprise Server 11 SP1 installation media. To update systems, youmust have either Internet access, Novell ZENworks Linux Management, or a localSubscription Management Tool.

• SUSE Linux Enterprise High Availability Extension installation media. To updatesystems, youmust have either Internet access, Novell ZENworks LinuxManagement,or a local Subscription Management Tool.

• SAP NetWeaver 7.0 EHP1 Installation Media.

viii Running SAP NetWeaver on SUSE Linux Enterprise Server with High Availability - DRBDdual data center

• Appropriate hardware: two servers, network, storage. For details, see below.

This guide focuses on one DRBD-based solution, which supports a dual data centertopology. We will give an overview of the covered SAP scenarios, include a briefoverview of the DRBD technology and describe in detail the DRBD configuration inconjunction with the HA cluster configuration.

Depending on your requirements, other scenarios can be selected or combined. A typicalcombination is the DRBD dual data center solution with host-based mirroring (Linuxmd-raid) and SBD (Stonith Block Device) dual data center solution. In this case, host-based mirroring is used to replicate SAP data between two local sites (campus cluster).SBD is used as an effective disk-based split-brain protection for the two-node campuscluster located at the first and second site. DRBD is then used to replicate the data toanother two-node cluster or a single standalone server at a third site. A typical use-casefor this solution is to have two local sites close to each other on the U.S. east-cost anda third site on the U.S. west-coast. The classic method of asynchronous log-shippingto a remote site far away on the database-level would be obsolete with this solution.

An overview of typical SAP high availability scenarios can be found in the white-paper"SAP on SUSE Linux Enterprise - Best Practices for Running SAP NetWeaver onSUSE Linux Enterprise Server 11 with High Availability"

Executive Summary ix

Part I. Introduction and Concept

1Introduction

1.1 SAP on LinuxNovell and SAP cooperate on a wide range of tasks. Along with the operating systemlayer, Novell and SAPwork closely to integrate Novell identity and securitymanagementsolutions with SAP's NetWeaver platform and business software applications.

Novell has multiple dedicated resources working at SAP headquarters and the SAPLinuxLab to ensure maximum interoperability between our products with SAP softwareand technologies.

SAP has built SAP LinuxLab to assist with the release of SAP software on Linux.LinuxLab supports other SAP departments in the development of the Linux platform.It processes Linux-specific support problems and acts as an information hub to all SAPpartners in the Linux ecosystem.

1.1.1 A Brief History of SAP on LinuxIn terms of data and computing, SAP applications are used to build the informationbackbones of companies. Therefore, SAP workloads need to be high-performance withgood scalability. In the past, this need was only met by dedicated data center hardwareand operating systems such as IBM Mainframes or HP 9000 UNIX servers (amongothers).

Introduction 3

In the 1990s, Intel-based (x86) systems met the performance criteria for runningsmaller SAP workloads. The main drive behind the adaption of x86-based platformswas the relatively lower cost and higher return on investment (ROI) for these systemscompared to mainframe or UNIX servers. At the same time, Linux matured into a fullycapable operating system that provided all the key functions needed to run all kinds ofworkloads. In the following years, Linux (and the x86 platform) evolved into a systemthat fulfills the needs for all kinds of SAP workloads, from the smallest to the largestsystems. Currently, SUSE Linux Enterprise 11 has been tested with up to 4,096 CPUsand 4 TiB RAM, with even higher theoretical limits. In the beginning, SAP on Linuxwas limited to 32-bit, but with the x86-64 extensions to the x86 architecture, theselimitations where overcome. Today, nearly all x86 architecture server CPUs are 64-bitcapable.

Where possible, SAP endorsed open standards and technologies. This allowed SAP tosupport a very wide range of operating systems and hardware platforms. Open-source-based Linux provides the maximum in “openness,” so it was only natural for SAP tostart supporting it in 1999. SAP tries to be “operating system agnostic” and act neutralon the customer's chosen operating systems. Unlike other software vendors, SAP hasclearly stated its policies toward open source and Linux. For instance, the usage of bi-nary only (closed source) device drivers (kernel modules) is not supported. This helpsthe Linux and open source communities since hardware vendors are encouraged to eitherpublish the specifications and APIs of their hardware so the Linux community can writedrivers, or make driver source code available that can be included in the Linux kernel(see SAP Note 784391).

Linux allows customers to reduce their total cost of ownership (TCO). Linux distributorsdo not charge a license fee for Linux because it is open source. Only support and servicesneed to be acquired. Since Linux is supported on a very wide range of hardware systems,customers now have the choice to opt out of vendor lock-in. In terms of administration,SAP customers see little difference between Linux and proprietary UNIX-like operatingsystems. Linux is an accepted operating system in all areas of data center computing.Through open interfaces and awide range of available applications, Linux is very capableof providing services at all availability levels necessary for successful standalone SAPworkloads or integration with existing environments.

4 Running SAP NetWeaver on SUSE Linux Enterprise Server with High Availability - DRBDdual data center

1.2 SUSE Linux Enterprise Server

1.2.1 SUSE Linux Enterprise Server IsPrepared for SAP

Since the very start of SAP involvement with Linux, SUSE has been the number oneLinux platform for SAP. SUSE Linux Enterprise Server has become the referenceplatform for SAP's software development. SAP now actively suggests SUSE LinuxEnterprise Server when customers want to run SAP workloads on Linux. Recently,SUSE Linux Enterprise Server was chosen as the operating system for SAP's all-in-onebusiness solutions and became a part of the SAP product line.

SUSE Linux Enterprise Server has been fine-tuned to provide the best Linux platformfor SAP software.

YaST, the main installation and administration tool for SUSE Linux Enterprise Server,provides a SAP software pattern that installs the prerequisites needed for running SAPsoftware. Also, some system parameters are fine-tuned in preparation for the SAPworkload.

In this document, we use SUSE Linux Enterprise Server 11 SP1 x86-64 with updatesfrom Novell Customer Center.

1.2.2 SUSE Linux Enterprise Server for SAPApplications

SUSE Linux Enterprise Server for SAP Applications is the only operating system opti-mized for all mission-critical SAP software solutions, including appliances, and is rec-ommended by SAP as a preferred Linux platform.

SUSE Linux Enterprise Server for SAP Applications is based on the newest SUSELinux Enterprise technology, a highly reliable, scalable, secure and optimized serveroperating system that is built to power both physical and virtual mission-critical work-loads. SUSE Linux Enterprise Server for SAPApplications is optimized for all mission-critical SAP software solutions. It is validated and certified by SAP and is, therefore,uniquely positioned to support SAP software solutions including appliances. Findmore

Introduction 5

details about this product athttp://www.novell.com/products/sles-for-sap.html.

Installing a highly available cluster using SUSE Linux Enterprise Server for SAP Ap-plications is more comfortable, because all needed packages including the clusterpackages and SAP-related packages, like the java JDK, are already included in onesingle product.

1.2.3 Support and CertificationFor SAP customers, Novell offers the SUSE Linux Enterprise Server Priority Supportfor SAP Applications. This support offering provides customers with a single supportpoint of entry for issues ranging from the operating system to the application (http://www.novell.com/docrep/2007/05/4611143_f_en.pdf).

Currently the following SUSE Linux Enterprise Server systems, including all servicepacks (SP), are certified as 64-bit version for SAP:

• SUSE Linux Enterprise Server for SAP Applications 11 (http://www.novell.com/products/sles-for-sap.html

• SUSE Linux Enterprise Server 11 for AMD64 and Intel EM64T

• SUSE Linux Enterprise Server 11


• SUSE Linux Enterprise Server 11 for IBM Power

• SUSE Linux Enterprise Server 11 for IBM System z

• SUSE Linux Enterprise Server 11 for Intel ia64



• SUSE Linux Enterprise Server 10 for IBM Power

• SUSE Linux Enterprise Server 10 for IBM System z


http://www.novell.com/products/sles-for-sap.html


http://www.novell.com/docrep/2007/05/4611143_f_en.pdf







• SUSE Linux Enterprise Server 9 for IBM POWER

• SUSE Linux Enterprise Server 9 for zSeries



• SUSE Linux Enterprise Server 8 for zSeries

SAP and Novell are working together to ensure that SUSE Linux Enterprise Serverservice packs always match the certification of the respective product. In fact, SAPrecommends to always use the latest available service pack.

Novell will provide at least five years of general support for platform and operatingsystem products, including its revisions, starting at the date of a product's generalavailability. When general support ends, Novell will offer extended support for a mini-mum of two years. This gives SAP customers a long installation run-time ensuring alow TCO.

1.3 SUSE Linux Enterprise Server HighAvailability Extension

Your data is your most valuable asset—it is what your business depends on. Robust,scalable and manageable storage is a top priority for your IT department. High avail-ability storage, applications and services are critical for your business to be competitive.But even as data grows, you can lower storage management costs and still benefit froman easy-to-manage, high-availability and storage foundation that scales as needed.

The High Availability Extension, a featured addition for SUSE Linux Enterprise Server11, satisfies these needs. It includes high availability service and application clustering,file systems/clustered file systems, network file systems (NAS), volume managers,

Introduction 7

networked storage systems and drivers (SAN), and themanagement of all these compo-nents working together.

Unlike proprietary solutions, SUSE Linux Enterprise High Availability Extension keepscosts low by integrating open source, enterprise-class components. The key componentsof the extension are:

• OpenAIS, a high availability cluster manager, supports multinode failover.

• Distributed Replicated Block Devices (DRBD8) provides fast data resynchronizationcapabilities over LAN, replicated storage area network (SAN) semantics, allowingcluster-aware file systems to be used without additional SANs.

We use DRBD with the LinBit Resource Agent to mirror the data asynchronouslyfrom one data center to an other.

• Resource Agents to monitor availability of resources

• Oracle Cluster File System 2 (OCFS2), a parallel cluster file system, offers scalabil-ity.

• Cluster Logical Volume Manager (cLVM2), a logical volume manager for the Linuxkernel, provides a method of allocating space on mass storage devices that is moreflexible than conventional partitioning schemes.

• High availability GUI and various command line tools.

SUSE Linux Enterprise Server High Availability Extension integrates these open sourcetechnologies and enables you to support line-of-business workloads traditionally reservedfor UNIX andmainframe systems.Without this integration, you would have to configureeach component separately and manually prevent conflicting administration operationsfrom affecting shared storage. When delivered as an integrated solution, the HighAvailability Storage Infrastructure technology automatically shares cluster configurationand coordinates cluster-wide activities to ensure deterministic and predictable adminis-tration of storage resources for shared disk-based clusters.

The multinode failover support in OpenAIS, the improved node and journaling recoveryin OCFS2, and the snapshots in the Logical Volume Management System (cLVM2)are some examples of the high availability features in the storage infrastructure. Otherfeatures, such as the cluster awareness and ready-to-run support of Oracle RAC, enrichthe environment, simplifying administrative tasks or eliminating them completely.


Availability is a result of the interaction of cluster software with application serviceson the front side and the operating system and hardware resources behind it. Followingthis basic idea, cluster software like OpenAIS could not increase availability on its own.It needs a lot of modules, such as services, resource agents, a messaging layer, networkand file system availability, and a stable Linux kernel designed and configured forproductive server systems in data centers.

Figure 1.1 Modules of a High Availability SAP Cluster

The central application of our cluster is the SAP system itself. We need to provide theSAP database and the central SAP instance with high availability (white boxes). Oper-ating system (light colored boxes) and cluster software (dark colored boxes) togethergive us the needed functionality.

In this document, we use SUSE Linux Enterprise Server High Availability Extensionx86-64 with updates from Novell Customer Center.

Introduction 9

1.3.1 Storage Scenarios and I/O Stack forSAP

The file systems required by the operating system (i.e. mounted on / or /var) are usuallystored on local hard drives using RAID-1 or RAID-5 disk arrays. Sometimes LVM isalso used to gain more flexibility, i.e. for online file system extensions. This impliesthat these file systems can only be accessed by the local operating system or applicationsrunning locally on this server. Even if these file systems are placed on LUNs in a storagearea network (SAN) outside the server, they are treated as local.

Besides the usual file systems for the operating system installation (like /, /var andothers), SAP and the SAP databases require their own file systems. These file systemsare not stored locally. Instead they are provided by NFS file servers or on LUNs exportedby storage units in storage area networks. Please be aware that the storage on whichthe database resides is subject to specifications and recommendations of the databasevendor.

Some file systems have to be shared across all cluster nodes so that every cluster nodecan access the file system at the same time. The recommended way of providing thisservice is the usage of a network file system (NFS). The NFS service also has to behighly available. In this document we use a dedicated NFS server. This server couldbe either a third-party product or a SUSE Linux Enterprise Server High AvailabilityExtension cluster that runs NFS resources. Under specific circumstances, it may bepossible to integrate the NFS server into the SAP cluster. Both the separate NFS clusteras well as the integrated NFS server are described in separate documents.


Figure 1.2 SAN IO Layers of High Availability Stack for SAP

SAN storage system

HBA raw devices

Multipath devices

MD Raid 1 for Host Based Mirroring

Volume Groups

Logical Volumes

Filesystem

7

6

5

4

3

2

1

sda sdb sdc sdd

mpath1 mpath2

md0

vg_sap

lv1_sap lv2_sap

Ext3 Ext3

DRBD asynchronous data replication

8

drbd0

Other file systems, e.g. mounted to /usr/sap/<SID> or /oracle/<SID>, only have to beavailable on one cluster node at the same time. However, each cluster node must beable to access these file systems if the cluster manager decides to use them.

In our current concept, we use LVM2 on top of DRBD andMDRAID, which has shownits road capability for years. This storage stack can be used with or without a cluster.

The UNIX file system is the highest layer of a whole I/O stack consisting of multipleI/O layers. Each layer provides a certain kind of functionality. For all I/O critical tasks,we have configured an I/O stack that supports the following functions:

• Low latency: high I/O throughput and fast response times.

• Host-based mirroring for storing data simultaneously on two separate storage unitsin a SAN.

• DRBD to synchronize the data from one site to the other.

• Logical Volume Manager for a flexible management of file systems.

• Multipath I/O for an additional level of redundancy for file systems stored on LUNsin the SAN.

Introduction 11

• Online resizing (extending) of file systems, snapshots of file systems using LVMsnapshots, moving or copying file systems.

This guide describes two common use cases in which DRBD (Distributed ReplicatedBlock Device) technology is used to replicate SAP data over IP networks.

The first use case describes a scenario in which SAP data is replicated asynchronouslyfrom one site to another site. Each site runs a local-cluster. This use-case is describedin detail. The second use case simplifies the setup by not having two clusters at twosites, but only one cluster at the first site and a standalone server at the second site.

1.3.2 Introduction to DRBDDRBD (Distributed Replicated Block Device) is a distributed storage system thatreplicates data on the block device layer over the network between two hosts. It issometimes referred to as “raid over the network”. However, DRBD has some uniqueadditional features:

1. Fully synchronous, memory synchronous or asynchronous modes of operation.

2. Automatic recovery after node, network, or disk failures.

3. Bandwidth of background resynchronization are tunable.

4. Online data verification.

5. Feature-complete OCF Heartbeat/Pacemeker resource agent.

6. Support for TCP/IP over Ethernet, SuperSockets over Dolphin NICs (8.3.2) and SDPover Infiniband.

7. Support for single block device sizes of up to one petabyte.

8. Support for active bandwidth management for mirroring over long distance and highthroughput internet links, including optional complete fall behind in case bandwidthis not sufficient.

A complete feature list is available at http://www.drbd.org/home/feature-list/


http://www.drbd.org/home/feature-list/

http://www.drbd.org/home/feature-list/

DRBD communication protocols

Protocol A.Asynchronous replication protocol. Local write operations on the primary node areconsidered completed as soon as the local disk write has occurred and the replicationpacket has been placed in the local TCP send buffer. In the event of a forced fail-over, data loss may occur. The data on the standby node is consistent after fail-over. However, the most recent updates performed prior to the crash may be lost.

Protocol B.Memory synchronous (semi-synchronous) replication protocol. Local write opera-tions on the primary node are considered completed as soon as the local disk writehas occurred and the replication packet has reached the peer node. Normally, nowrites are lost in case of a forced fail-over. However, in the event of simultaneouspower failure on both nodes and concurrent, irreversible destruction of the primarynode's data store, the most recent writes completed on the primary node may belost.

Protocol C.Synchronous replication protocol. Local write operations on the primary node areconsidered completed only after both the local and the remote disk write have beenconfirmed. As a result, loss of a single node is guaranteed not to lead to any dataloss. Data loss is, of course, inevitable even with this replication protocol if bothnodes (or their storage subsystems) are irreversibly destroyed at the same time.

The definition of these protocols have been taken from the DRBD user guide. We areusing protocol A in this solution to implement an asynchronous mirror from one datacenter to an other.

DRBD refers to both the software (kernel module and associated userspace tools) andthe specific logical block devices managed by the software. The terms "DRBD device"and "DRBD block device" are also often used for the latter.

DRBD layers logical block devices (conventionally named /dev/drbdX, where X is thedevice minor number) over existing local block devices on participating cluster nodes.Writes to the primary node are transferred to the lower-level block device and simulta-neously propagated to the secondary node. The secondary node then transfers data toits corresponding lower-level block device. All read I/O is performed locally.

Should the primary node fail, a cluster management process promotes the secondarynode to a primary state. This transition may require a subsequent verification of the

Introduction 13

integrity of the file system stacked on top of DRBD, by way of a filesystem check ora journal replay. When the failed ex-primary node returns, the systemmay (or may not)raise it to primary level again, after device data resynchronization. DRBD's synchro-nization algorithm is efficient in the sense that only those blocks that were changedduring the outage must be resynchronized rather than the device in its entirety.

The DRBD suite is open source and licensed under the GNU General Public Licensev2 and is part of the official Linux kernel. It is mainly being developed by the Austriancompany LINBIT. Novell has a strong partnership with LINBIT and therefore can ensurea fast and high-quality support as well as a fast resolution of software defects (bugs).

1.3.3 SAP Resource AgentsThe SAP Instance Resource Agent is responsible for starting, stopping and monitoringthe services in a SAP instance. The following services are monitored in the resourceagent: disp+work, msg_server, enserver, enrepserver, jcontrol, andjstart.

The SAP Instance Resource Agent can be used to manage the following SAP instances:

• SAP WebAS ABAP Release 6.20 – 7.30,

• SAP WebAS Java Release 6.40 - 7.30 (for 6.40 please also read SAP note 995116),

• SAP WebAS ABAP + Java Add-In Release 6.20 - 7.30 (Java is not monitored bythe cluster).

The purpose of the SAPDatabase resource agent is to start, stop andmonitor the databaseinstance of an SAP system. Together with the relational database management systems(RDBMS), it will also control the related network service for the database, like theOracle Listener and the xserver of MaxDB. The resource agent expects a standard SAPinstallation and therefore needs fewer parameters to configure. The monitor operationof the resource agent can test the availability of the database by using SAP tools(R3trans or jdbcconnect). This ensures that the database is really accessible forthe SAP system.

The SAPDatabase resource agent supports the following databases in a SAP installation:

• Oracle 10gR2 and 11gR2,


• IBM DB2 UDB for Windows and Unix 9.x,

• SAP-DB / MaxDB 7.7.

The resource agents are part of the SUSE Linux Enterprise Server High AvailabilityExtension.

1.4 Database for SAPThree major RDBMS are supported with SAP on SUSE Linux Enterprise Server, eitherwith or without the SUSE Linux Enterprise Server High Availability Extension. In ad-dition, for SAP deployments with each of these databases, SAP priority support isavailable from Novell.

• Oracle 10gR2 and 11gR2

• IBM DB2 UDB for Windows and UNIX 9.x

• SAP-DB / MaxDB 7.x

The supported hardware platform for all RDBMS on SUSE Linux Enterprise Server11 is x86-64 (also known as “x86 64-bit”, “AMD64”, “EM64T”). Nevertheless, somedatabases are also supported on other hardware platforms such as power or z-series.Details can be found on the certification pages listed as links in the appendix.

SUSE Linux Enterprise Server ships with an installation pattern for Oracle databaseservers. The pattern contains the RPM package orarun which sets several system settingsaccording to the needs of an Oracle server. If Oracle is installed together with SAP,this package should not be installed to avoid conflicts with the settings from the RPMsapconf. Further information can be found on the web pages listed in the appendix.

MaxDB was chosen as RDBMS for this document mainly to shorten the planning andinstallation procedure. Since the document is targeting a basic installation of SAP in ahigh-availability cluster, the use cases are covered with MaxDB. This installation ex-ample does not imply any judgement on how well one of the database managementsystems fits for a given workload.

Introduction 15

1.4.1 MaxDBThe SAP MaxDB is the database of choice for small business and midsize companiesrequiring a solid, affordable low-maintenance database. MaxDB is available for all in-stallations of the SAP Business All-in-One solution and the SAP Business Suite familyof business applications.MaxDB is bundled with the full license for the SAPNetWeavertechnology platform. And the SAP Business by Design solution uses MaxDB as thedefault database for the host system. Designed for online transaction processing anddatabase sizes up to multiple terabytes, MaxDB is the preferred database for internalSAP installations on UNIX and Linux (http://www.sap.com/solutions/sme/businessallinone/kits/lowertco.epx).

MaxDB´s ancestor AdabasD was available on Linux in 1996. Between 1997 and 2004the software was available as SAP-DB, then it was renamed to MaxDB. MaxDB asstandalone product is supported for SUSE Linux Enterprise Server 11 on the hardwareplatforms x86-64 and ppc64 (http://maxdb.sap.com/documentation/). Asan integrated database for the SAP NetWeaver technology platform, the respectivecertification applies for SUSE Linux Enterprise Server 11 on x86-64 and ppc64. ForSAP systems, the appropriate product certificationmatrix should be applied (https://websmp201.sap-ag.de/pam).

MaxDB installation media can be obtained from the SAP portal along with NetWeaver.The installation of MaxDB is seamlessly integrated into the SAP installer. SAP offersseveral services around MaxDB for SAP applications. More information can be foundon the Web pages listed in the appendix.

1.5 Sizing, Deployment and MigrationConsiderations

Running SAPworkloads in SUSE Linux Enterprise Server can reduce the TCO consid-erably. Through the homogenization of the data center (and thus the consolidation ofresources), cost savings can be achieved even for highly specialized SAP installations.SUSE Linux Enterprise Server is very well prepared for automated installation anddeployment, making it easy to deploy many servers. Novell provides tools for managingupdates and releases as well as ease day-to-day administration.


http://www.sap.com/solutions/sme/businessallinone/kits/lowertco.epx

http://www.sap.com/solutions/sme/businessallinone/kits/lowertco.epx

http://maxdb.sap.com/documentation/

https://websmp201.sap-ag.de/pam


SUSE Linux Enterprise Server is very capable of providing the base for small or largesystems. Customers run smaller central instances or larger distributed systems all withthe same system base. It is quite possible to run multiple SAP instances in parallel onone system even when using high availability clusters.

SAP requests that system sizing is done by the hardware vendor. Novell has good rela-tionships with many hardware vendors to make sure SUSE Linux Enterprise Serverruns smoothly on a broad range of enterprise servers fit to run SAP workloads. Novelland its partners are very active in providing customers with solutions to their specificneeds when it comes to Linux deployment. Novell consulting has been developing bestpractices for high availability SAP installations and provides this information to cus-tomers and partners.

Hundreds of successful SAP to Linuxmigrations have beenmade. The results regardingcost savings, performance and reliability have exceeded expectations in many instances.

Since most data centers have adopted a Linux strategy, the know-how for deployingand administrating Linux systems is often in place and available. SAP-specific config-urations and administration experience is available through Novell consultation andpartners. This makes the operating system side of the migration less risky and a ROIcan be seen within the first six months of migration.

SAP provides check lists and guidelines for the OS and database migration.

Introduction 17

2One Concept — Two UseCasesSAPNetWeaver installation can be adapted to several architectures for the entire system.You can plan and implement a “simple stack” SAP system with one instance and thedatabase on the same machine without any methods to increase the availability. Youcan take this simple installation scenario, add high availability components and bringthe SAP system into a high availability scenario. There may also be reasons to installa SAP system with additional application servers. In a high availability scenario, thiscould lead to a more advanced “enqueue replication” setup. In all previously mentionedcases, SAP is treated in a two-tier manner, this means SAP application and databaseprocesses can run on the same system.

Another common use case is to run the enqueue replication server and the database ondifferent machines. In this case, high availability for the enqueue replication server iscovered by the described concept as well.

This document covers these two use cases:

• DRBD dual data center with two clusters - 2x2 nodes

• DRBD dual data center with one cluster and one standalone node - 2+1 nodes

Both use cases (2x2 and 2+1) can be used with all common SAP installation types.

SAP central system: All instances are running on one node. Depending on the instal-lation type, this may be one central instance or multiple instances on one node.

SAP enqueue replication: The replicated enqueue server solves one availability issuesof the enqueue service which otherwise would be a single point-of-failure (SPOF). The

One Concept — Two Use Cases 19

replicated enqueue server provides a transparent solution to this availability issue andenables the SAP system to continue production in the case of a failing enqueue server.

In a typicall setup there are two host machines (physical or virtual), one for the stan-dalone enqueue server and one for the replicated enqueue server. Refer to the SAPdocumentation at the SAP Marketplace for more details. Such a two system setup hasthe following advantages:

• Redundancy: The replicated enqueue server holds a complete copy of the lock table.

• Flexability: The two most critical components (enqueue server and message server)could be restarted in a very short time and even faster than a complete SAP applicationserver.

• Availability: The standby server runs a replicated enqueue server that can be activatedif the primary enqueue server fails, using the enqueue table copy.

The result is that in the event of standalone enqueue server failure, no transactions orupdates are lost and the enqueue service for the SAP system continues without interrup-tion.

The SAP instances are running on different nodes, including the enqueue replicationmechanism. The database may run on a different cluster.

The installation of the first use case (2x2) will be shown step by step in this document.The simplified use case 2 (2+1) can be easily adopted from the first use case by omittingthe setup of the second cluster. Of course, you have to setup the complete storage stackand IP addresses.

2.1 High Availability ConceptOverview

The concept was chosen to match these top-level goals:

• High availability

• Low complexity

• Flexible scalability


• Road capability

To achieve these goals, we separate the SAP system into a clustered and an unclusteredarea. The clustered area holds all mandatory SAP components such as SAP databaseand needed SAP instances.

The unclustered area holds all optional and scalable SAP components such as additionalSAP instances. This allows to scale the entire SAP systemwithout increasing the clustercomplexity. The horizontal scaling is just a purpose of the unclustered area.

The architecture is focused to one single SAP system, even if it is possible to run morethan one SAP system in the same cluster.

The example configuration described in this document consists of a total of 4 SAPnodes spread across two distinct sites, with 2 nodes per site. The nodes in each site forma Pacemaker high availability cluster.

This architecture assumes that within a single site, a SAN exists with fully meshed fibrechannel connectivity. Cluster nodes are assigned two SCSI Logical Units (LUNs) spreadacross two different shared storage devices. Each cluster node has access to both LUNswith redundant (multipath) connectivity and uses Linux software RAID (MD) for host-based mirroring.

For replication between sites, a DRBD device is layered on top the RAIDmirror. Thus,asynchronous storage replication between sites requires no SAN connectivity -- simpleIP connectivity is sufficient.

This concept uses STONITH (Shoot The Other Node In The Head) and can be expandedwith SFEX (Shared Disk File Exclusiveness). While STONITH allows server fencingusing remote management boards over LAN, SFEX provides storage protection overSAN. In our example setup for DRBD we do not use SFEX.


Figure 2.1 General Architecture

The network file system (NFS) is used to share data between the nodes, for examplefor the SAP transport directory. In this concept we assume that a reliable NFS is providedby a service outside the cluster. Either a highly available NFS server based on SUSELinux Enterprise Server 11 or a third party product could be used. An NFS high avail-ability cluster based on SUSE Linux Enterprise Server 11 is described in another docu-ment. In some case it might be desirable to have the NFS server in the same cluster asthe SAP application. This is also covered in an extra document.


In a complex, high availability SAP environment, several types of failures may occur.These failures range from software crashes up to a loss of the whole network or SANinfrastructure. The cluster must be able to safely handle all of these failures. Even in asplit brain scenario, if the cluster communication between both nodes is broken, thecluster must ensure a proper continuation of all services.

2.1.1 Server Fencing and Storage Protection

STONITH, SBD, watchdog, and LAN-based quorumSTONITH (Shoot the other Node in the Head) is a key mechanism of corosync clusters.STONITH is used to clearify the cluster node status in a situation where a cluster nodegot lost. Such a situation can occur in the following example cases:

• A node has a power outage or (complete) hardware problem, so the node is not longeravailable.

• A node crashes with a kernel freeze.

• A node has far to high load and cannot answer on the cluster communication for a(longer) period of time.

• A node has a network disruption and cannot answer on the cluster communication.

In all these cases, the cluster can notify that a node got lost but cannot determine whetherthe node can still affect any resources controlled by the cluster. This is why the clusterdecides to fence the node via a STONITH device. Such a STONITH action can switcha node off or force it to reboot (reset).

Typical STONITH methods are:

• Remote Management Board-based system control

Remote management boards, like HP's ILO boards can reset, start or shut down aserver. The reset and power cycle functions can be accessed not only via their web-interfaces but also via the network using standardized interfaces like IPMI (intelligentplatform management interface) or similar. This functionality makes these boardsideal candidates for STONITH agents. For ILO2 boards, a dedicated STONITH agentcalled riloe exists. For boards of other manufacturers, take a look at the list of available


STONITH agents or use the IPMI agent, which works for most servers. A clear dis-advantage of STONITHwith remote-management boards is that a total power failureof a server also disables the remote management boards. A STONITH request fromanother node cannot be completed in this case and prevents an automatic fail-over.

• Power switch-based system control

Remotely controllable power switches provide a reliable way to start, shut down andpower-cycle a server. This method is preferred over the use of remote managementboards. It is much more likely that a power switch is still accessible after a power-failure affecting a single server than a remote management board, which usually relieson the power supplies of the affected server. STONTIH agents exist for variouspower-switches, e.g. from APC. A disadvantage of power-switches may be the ad-ministrative domain in large data centers. If the power switch is not under control ofthe cluster administration team, it may happen that somebody changes the passwordof the power-switch without noticing the cluster team.

• SBD (Split Brain Detection) - SAN-based system control

An alternative option is STONITH based on a Split Brain Detection (SBD) disk to-gether with the kernel watchdog. Future implementations of the SBD will respectthe cluster's quorum status. In case the SBD disk fails, the cluster will continue towork as long as it has the quorum. Thus, the impact of a failing SAN LUN is reducedcompared to the SFEX-based solutionmentioned above. The secondmajor advantageis that server fencing works in LAN-split, SAN-split, and complete-split scenarios.A solution based on this SBD will be described in another document.

In this example installation, we use the Management Board-based system control viaIPMI or ILO interfaces.

2.2 Use Case 1 “DRBD dual datacenter with two clusters - 2x2”

The "2x2" scenario of the DRBD dual data center uses two separate and nearly indepen-dent corosync clusters working at two different sites. Nearly independent cluster meansthat the clusters do not have a direct communication at the cluster layer, so both clusterframeworks are running only local at their specific site.


Of course the data need to be synchronized from one site to the other and that is whythe cluster nodes of one site need a communication channel via TCP/IP to the othersite.

To avoid both clusters to run the DRBD device in master role and to start the SAPsystem, one site is defined to always run the DRBD device in slave mode (be the passiveside of the synchronization). The other site is defined to always run the DRBD devicein master mode (be the active side of the synchronization).

If the data center communication fails, the current site running the SAP system willstay productive while the other site is waiting for the DRBD synchronization to comeback.

If the active cluster site fails completely and you need to run a site-failover, the admin-istrator needs to change the DRBD device role from slave to master. Of course the ad-ministrator must prevent both clusters from running in master mode at the same time.This is why we decided to run the site-failover in a semi-automatic mode with only oneadministrative interaction.

You can also run the sites active/active if you place more than one SAP system on theclusters and define multiple DRBD synchronizations. But each of these DRBD devicesneeds to be run as master/slave and never dual active.

Figure 2.2 DRBD synchronization with two clusters in two data centers


The advantages of this cluster model:

• Flexible design

• Synchronization between two data centers

• Second site also runs with High Availability to improve the status of the DRBDsynchronization

• Both sites uses the same architecture

• Site failover is semi-automatic but can be triggered easily.

One disadvantage is:

• Four nodes are used for this concept.

2.3 Use Case 2 “DRBD dual datacenter with one cluster - 2+1”

The "2+1" scenario of the DRBD dual data center uses one corosync cluster and astandalone node working at two different sites.

The data must be synchronized from one site to the other, therefore the cluster nodesof one site need a TCP/IP communication channel to the standalone node of the othersite.

To avoid both sites running the DRBD device in master role and to start the SAP system,one site is defined to always run the DRBD device in slave mode (be the passive sideof the synchronization). The other site is defined to always run the DRBD device inmaster mode (be the active side of the synchronization).

If the data center communication fails, the current site running the SAP system willstay productive while the other site is waiting for the DRBD synchronization to comeback.

If the active cluster site fails completely and you need to run a site-failover, the admin-istrator needs to change the DRBD device role from slave to master and start a procedurewhich starts all resources (storage devices, file systems, IP addresses, database and


SAP instances). The administrator must prevent both clusters running in master modeat the same time. This scenario does not provide a semi-automatic site-failover. Thefailover must be covered by the system management procedures.

Figure 2.3 DRBD synchronization with two clusters in two data centers

• Flexible design

• Synchronization between two data centers

• Price: Only 3 nodes are used for the local High Availability cluster and the datacentersynchronization.

Some disadvantages are:

• The sites uses different architectures

• Site takeover is more complex (start of the SAP system)

• Not recommended for active/active scenarios.


Part II. Installation of "DRBDdual data center" with MaxDB

3Installation OverviewThis part describes the installation of a SAP NetWeaver with MaxDB on SUSE LinuxEnterprise Server for SAPApplications for a proof of concept. The procedure is dividedinto the following steps:

• Planning (Chapter 4, Planning (page 33))

• Checking prerequisites (Chapter 5, Prerequisites (page 35))

• Downloading SAP NetWeaver installation media (Chapter 6, Download the NeededSAP Installation Media (page 39))

• Installation of SUSE Linux Enterprise Server for SAP Applications on all nodes(Chapter 7, Install SUSE Linux Enterprise Server 11 SP1 for SAP (page 43))

• Preparing SAN storage on both sites (Chapter 8, Prepare SAN Storage (page 49))

• Cluster Configuration on both sites (Chapter 9, Configure the Cluster on both Sites(page 63))

• Installation of SAP NetWeaver and MaxDB (Chapter 10, Install SAP NetWeaver 7.0EHP1 (page 75))

• Integration of SAP NetWeaver and MaxDB into the High Availability Cluster(Chapter 11, Integrating SAP into the cluster (page 85))

• Checking final results (Chapter 12, Testing the Cluster (page 87))

Installation Overview 31

4PlanningProper planning is essential for a well performing SAP system. For planning and supportof your SAP installation, visit http://service.sap.com [http://service.sap.com/]to download installation guides, review installation media lists and browse through theSAP notes. This section focuses on aspects of planning a SAP installation.

The first major step is to size your SAP system then derive the hardware sizing to beused for implementation. Use the SAP benchmarks (http://www.sap.com/solutions/benchmark/index.epx) to estimate sizing for a proof of concept.If you plan to migrate an existing SAP system you should first obtain or estimate thesystem characteristics of the “old” SAP system. The key values of these characteristicsinclude:

• SAPS (benchmarks) of the old SAP system

• Memory (RAM) size and usage of the old hardware

• Disk size, performance and usage of the old SAP system

• Network performance and utilization of the old hardware

• Language support (including Unicode)

If you have valid key values, you can adapt these to the characteristics of your “new”SAP system. If you plan a new installation instead of a migration, you might need toadapt experiences with other SAP installations or use some of the published benchmarksas mentioned above.

Planning 33

http://service.sap.com/

http://www.sap.com/solutions/benchmark/index.epx


• Estimate the SAPS in the new SAP system. This includes planning additional capac-ities, if needed. The calculation should also include estimated growth calculations,such as a boost of SAPS per year. Typical SAP growth is between 10% and 20% peryear.

• Choose RAM size, as well as disk size and performance for the Linux system. Alsoinclude a boost of the usage. Depending on the usage, the disk space may grow 30%per year. The disk size must also include the export and r3trans areas if they are notprovided by other servers.

• Check if Unicode support is necessary for the new system.


5PrerequisitesThis chapter describes what hardware and software is needed for a proof of concept. Italso outlines how to gather all information necessary to succeed.

5.1 Hardware RequirementsThe final hardware sizing for production systems is always the responsibility of thehardware vendor. At the time this document was written, a base installation ofNetWeaver on top of SUSE Linux Enterprise Server 11 SP1 for a proof of conceptcould be done on the following hardware equipment:

• 2* CPU x86-64 2GHz (AMD64 or EM64T),

• 8GB RAM,

• 100GB disk space for OS and empty SAP system,

• a few hundred GB up to some TB additional disk space for test data,

• 3* Gbit/s Ethernet,

• IPMI or ILO remote management board.

Although the cluster offers High Availability across two servers, each single servershould already have the maximum possible availability using redundant devices:

• self correcting ECC RAM,

Prerequisites 35

• two or three redundant power supplies (connected to two circuits),

• several redundant cooling fans,

• two or more internal disks with RAID(1/5/6/10) controller,

• redundant LAN network controllers,

• redundant LAN network links (connected to two switches),

• redundant SAN host bus controllers,

• redundant SAN FC links (connected to two switches).

Make sure to use certified hardware. Information about certified hardware can be foundin the Novell YES database (http://developer.novell.com/yessearch/Search.jsp), in the SAP notes and on the hardware manufacturer's pages. Use cer-tification notes from the Novell YES database and the hardware manufacturer to selectappropriate hardware components.

5.2 Software Requirements,Connection Data, and all the Rest

Before you can start with the installation, make sure you meet the prerequisites.

• SAP documentation: Make sure that you have the latest version of the Master Guideby checking SAP Service Marketplace immediately before starting the installation.The Master Guide is regularly updated on SAP Service Marketplace at http://service.sap.com/instguidesnw70.

• Sizing data gathered in the planning stage for the installation.

• Configuration data:

• IP addresses, DNS names, gateway

• SAP SIDs

• Linux usernames (like <sid>adm), UIDs


http://developer.novell.com/yessearch/Search.jsp


http://service.sap.com/instguidesnw70

http://service.sap.com/instguidesnw70

• Connection data:

• SAN LUNs (names, LUN numbers) and multipath configuration parameters.There are some special parameter settings for multipath and SAN-HBA kernelmodules, depending on the hardware setup (SAN storagemodel and SAN setup).Check if SAN storages require partition alignment for performance reasons.Refer to the installation and configuration guides from Novell and hardwarevendors.

• Access to the system management boards to be used by the cluster to fence anode in special cases (STONITH). For most common data center hardware,there are supported management boards like ILO or IPMI, which provide stableinterfaces to be used with STONITH.

• In addition to the network that connects the SAP servers to the clients, we rec-ommend two additional dedicated network links between the two servers forcluster intercommunication. At least one additional dedicated network link ismandatory.

• Infrastructure such as DNS server, NTP server and a pingable highly available networknode. This network node can be the gateway between the SAP system and the clientswho need to access the service. If the gateway is no longer available, the service isnot available. The cluster can determine which cluster node has a (ping) connectionto the ping node and can migrate a service if needed.

• SAP installation media (for details see the table in the next section). The SAP instal-lation media can either be ordered as a physical CD/DVD or downloaded fromhttp://service.sap.com/swdc. The next section describes the procedure for downloadingthe SAP media.

• SAP S-User (partner user) to download the media and installation guides and tobrowse through the SAP notes system. The S-User must have permission to downloadthe installation media. Ask your company's SAP partner manager to create an S-Userand to grant the proper rights.

• During installation of the central instance of SAP NetWeaver you will be asked toprovide a SolutionManager Key. You need to create such a key for your combinationof hostname (DNS name of the virtual IP address for high availability installations),SAP system ID (SID) and SAP instance number (like 00, 01, 02). This key can becreated using your company´s Solution Manager, an additional SAP program. Thisdocument does not cover the installation of the Solution Manager. If you do not have

Prerequisites 37

access to your company´s Solution Manager, ask your internal SAP partner managerhow to get a Solution Manager key.

• To download the SAP installation media, you will need the SAP download manager.A short description of the installation is integrated in the next section.

• To run the download manager you need a matching Java version. In former PoCs,SUN Java 1.6.0 (packagejava-1_6_0-sun-1.6.0.u1-26) worked very well.Have a look at the installation notes presented during the procedure to download theSAP download manager.

• An up-to-date patch level of the SUSE Linux Enterprise Server 11 SP1 installation.You will need:

• a Novell Customer Center account,

• "SUSE Linux Enterprise Server for SAPApplications 11 SP1" installationmediafor x86-64 hardware platform,

• or "SUSE Linux Enterprise Server 11 SP1" and "SUSE Linux Enterprise HighAvailability Extension 11 SP1" installationmedia for x86-64 hardware platform,

• possibly additional hardware-specific driver updates,

• and a software management software such as the subscription management tool(optional).

• To test the SAP system you either need to have a previously installed SAP client(guilogon, guistart) or you need to install this software on at least one of yourworkstations.


6Download the Needed SAPInstallation MediaThis section describes in a more general way how to download the SAP installationmedia. As an example, we download the media for SAP NetWeaver 7.0 EHP1 withdatabase engine MaxDB RDBMS 7.7 for SUSE Linux Enterprise Server 11 SP1 x86-64.

We recommend you use a SUSE Linux Enterprise Server 11/SUSE Linux EnterpriseDesktop 11 SP1 or newer to download the media. This section only describes thedownload using a SUSE Linux Enterprise system.

1. Access the SAP download site http://service.sap.com/swdc.

2. You are asked to login with your S-User and password (see prerequisites).

3. From the menu on the left, select Download → Installation and Updates → Entryby Application group.

4. From the Installation and Upgrades menu on the right, select SAP NetWeaver →SAP NetWeaver (yes, twice).

5. Now you have to select your NetWeaver version. In our example, we select SAPEHP1 for SAP NetWeaver 7.0 → Installation and Upgrade → Linux → SAP DB.The version for your use case may differ, the example will follow the names andpaths of NW 7.0 EHP1.

6. From the media list, select the following media and click on “Add to downloadbasket”.

Download the Needed SAP Installation Media 39

http://service.sap.com/swdc

Table 6.1 SAP NetWeaver Installation Sources

DateSize [KB]TitleNumberType

25.10.20067675CD SAP License Keys &License Audit

50081125ZIP

11.12.2008976563Onl. Doc. SAP NW 7.0EHP1 SPS02 1 of 2

50092449_1EXE

11.12.2008585142Onl. Doc. SAP NW 7.0EHP1 SPS02 2 of 2

50092449_2RAR

20.08.2008976563NW7.0 EHP1 InstallationExport 1 of 2

51034942_1EXE

20.08.2008422045NW7.0 EHP1 InstallationExport 2 of 2

51034942_2RAR

19.02.2009976563NW 7.0 EHP1 KernelLINUX 1 of 8

51035688_1EXE


51035688_2RAR


51035688_3RAR


51035688_4RAR


51035688_5RAR


51035688_6RAR


51035688_7RAR


DateSize [KB]TitleNumberType


51035688_8RAR

11.12.2008153090NW7.01/BS 7 InstallationMaster Linux on x86-6464bit

51035700_8ZIP

11.12.2008124877MaxDB RDBMS 7.7.04Build 28 - Linux on x86-64 64bit

51035704_8ZIP

The total size of installation sources is 10GB for the chosen NetWeaver 7.0 EHP1with MaxDB. To unpack the archives, roughly twice the disk space is needed. Otherproducts might need more space.

7. After some time, a pop-up with two buttons appears. Press "Download Basket".

8. Your selected media is shown in your download basket. If you haven't installed theSAP download manager yet, you will have to download and install it now.

• Click “get downloadmanager” in this case. The SAPDownloadManager Instal-lation Guide is shown.

• Check the section prerequisites and the SAP Download Manager installationguide.

• You need a Java version that fits SAP needs. Download the Linux version.

• You get a self-extracting archive that starts after the download. Follow the in-stallation steps. We have installed the Download Manager in the local homedirectory, “SAP_Download_Manager”.

9. Start the installed SAP Download Manager using the command~/SAP_Download/Manager/Download_Manager.

10. If you start the SAP Download Manager for the first time, you will need to providesome credentials such as the SAPMarketplace address (http://service.sap

Download the Needed SAP Installation Media 41

(http://service.sap.com

.com), your S-User, your S-User-Password and the Data Store (directory to placethe downloaded files).

11. Press the "download all objects" button (the button with two right arrows).

12. Be patient, the download will take some time.

13. After the download, unpack the downloaded files using unzip (for ZIP type) andunrar (for EXE and RAR type). Unrar is able to skip the self extracting code in theEXE files and will include all files of a media set such as 1/2, 2/2.

14. Copy (rsync) the extracted files to your system to be installed or create NFS exportson the installation source and NFS mounts on the target systems. In our setup weuse:

• /sapcd/InstMa for the Installation Master,

• /sapcd/Kernel for the NW 7.01 kernel,

• /sapcd/MaxDB for the MaxDB engine,

• /sapcd/InstExp NW 7.0 EHP1 installation export.

15. You also need the JRE Unlimited Strength Jurisdiction Policy Files Archive(unrestricted.zip) matching your Java version. Download it from either IBMor SUN.


(http://service.sap.com

7Install SUSE Linux EnterpriseServer 11 SP1 for SAPIn this chapter we describe relevant configuration parameters of SUSE Linux EnterpriseServer 11 SP1 x86-64 for this use case. The installation procedure itself should be wellknown to the target audience and therefore is not described. Refer to the product docu-mentation for general information on the SUSE Linux Enterprise Server 11 (http://www.novell.com/documentation/sles11/).

You can also use this chapter for the installation of SUSE Linux Enterprise Server forSAP Applications 11 SP1.

7.1 File System LayoutFor a proof of concept, the following partition and file system layout for the operatingsystem is a good example:

Table 7.1 OS File System Layout

OptionsFile SystemSizeMount PointNo.

noatimeext2128MB/boot1

swap2*RAM (max.20GB)

swap2

noatimeext316GB/3

Install SUSE Linux Enterprise Server 11 SP1 for SAP 43

http://www.novell.com/documentation/sles11/


OptionsFile SystemSizeMount PointNo.

noatime,data=write-back

ext3rest/var4

SUSE Linux Enterprise Server 11 and SUSE Linux Enterprise High Availability needabout 4.5GB disk space. The size of /boot depends on the number of kernels that shouldbe installed in parallel. Each kernel needs approximately 35MB disk space in /boot.The size of /var depends on the amount of log data and application-specific usage, 5GBor more are appropriate. If the SAP NetWeaver installation sources should be put onthe local disk, 20GB additional free space is needed. We use the directory link /sapcdin our examples.

Besides the usual OS file systems, SAP and the SAP databases require their own filesystems. These file systems are not stored locally. Instead they are provided by NFSfile servers or on LUNs in Storage Area Networks (SAN). Typically we need for SAP:

• /sapmnt

• /usr/sap/<SID>

• /sapdb (forMaxDB. Oracle and DB2 require other paths.) File system sizes dependon the use case. The database file system can be from 100GB up to multiple TB.After a fresh installation, around 30GB are in the database.

7.2 Software SelectionIt is a good practice to install as little software as needed or recommended. We deselectthe „GNOMEDesktop Environment“ as we plan to run the server in runlevel 3 and notuse local Gnome sessions. Since we install the “X Window System”, we can still startX-Window-based applications locally or remotely. We also deselect “AppArmor”, be-cause we do not plan to use it. SAP recommends some additional software to be installed.The resulting pattern selection should be:

+ Base system pattern+ X Window pattern+ 32-bit runtime pattern+ Print server pattern+ SAP Application Server Base pattern


+ C/C++ Compiler and Tools pattern+ High Availability pattern- No AppArmor pattern- No Gnome pattern

NOTE

If you are installing SLES for SAP, then you should also install the pattern HighAvailability.

For the standard SLES, we recommend to install this pattern later.

Now your pattern list should look like the one in:

Figure 7.1 SUSE Linux Enterprise Server 11 Software Selection for SAP

Make sure you also install the following SAP-specific packages:

• sapconf prepares the operating system for SAP needs.

• sap-locale contains special code pages only needed for non-unicode systems.

If you plan to run SAP application servers on SUSE Linux Enterprise Server togetherwith application servers on another OS, see SAP Notes 1069443 and 187864 on howto get the correct code pages.


An appropriate JRE and JDKmust be installed before starting the SAP installation. Weuse the IBM 1.4.2 sr13 FP8 JRE and JDK. Check the SAP notes for exact and currentversions.

NOTE

If SUSE Linux Enterprise Server for SAP Applications is used, the correct IBMJRE and JDK are included as java-1_4_2-ibm-sap and java_1_4_2-ibm-sap-devel.

The packages for SUSE Linux Enterprise Server are named java_1_4_2-ibm andjava_1_4_2-ibm-devel

To select the RPMs, change to “Details” and search for the package names containing“java-1_4_2”. Select:

+ java_1_4_2-ibm-sap+ java_1_4_2-ibm-sap-devel

If you plan to extract the SAP installation sources, install the RPM unrar as well.

The RPMs orarun and ulimit conflict with the SAP requirements and should not be in-stalled.

We recommend to update the complete operating system to a current level. Eitherconnect to the NCC via Internet or use a locally installed update proxy like SMT. Theupdate procedure should be well known to the target audience and therefore is not de-scribed. For information on NCC refer to the Novell documentation ( http://www.novell.com/documentation/ncc/ncc/?page=/documentation/ncc/ncc/data/bktitle.html ).

As of the publication date of this document, you should have at least the following re-leases of the core operating system:

• kernel-default-2.6.32.24-0.2.1

• lvm2-2.02.39-18.31.2

• multipath-tools-0.4.8-40.23.1

• mkinitrd-2.4.1-0.14.1


http://www.novell.com/documentation/ncc/ncc/?page=/documentation/ncc/ncc/data/bktitle.html



• device-mapper-1.02.27-8.17.20

• glibc-2.11.1-0.20.1

• nfs-client-1.2.1-2.10.1

• libext2fs2-1.41.9-2.1.51

• libuuid1-2.16-6.8.2

• uuid-runtime-2.16-6.8.2

You should also have at least the following releases of the additional software:

• sapconf-3.0-67.3.1

• sap-locale-1.0-24.33.27

• java-1_4_2-ibm-1.4.2_sr13.6-0.5.1

• java-1_4_2-ibm-devel-1.4.2_sr13.6-0.5.1

• libgcc43-4.3.4_20091019-0.7.35

• gcc43-4.3.4_20091019-0.7.35

7.3 Runlevel and System ServicesRecommended runlevel is “3: multi user with network”.

The services ntp, boot.clock, uuidd, boot.sapconf,boot.multipath, and multipathd should be enabled.

The Network Time Protocol (NTP) service has to be set up correctly.

If running the SAP system in a high availability cluster, the services boot.dmraidand boot.md are disabled. This functionality is controlled by the cluster.


7.4 MiscellaneousSystem language has to be en_US.

7.5 Check SLES for SAP InstallationCarefully check the installed system. All selections and parametersmentioned in previouschapters have to be correct. A script can help to collect the relevant data. An exampleis the whbsaprecheck script contained in the ClusterTools2 RPM. This RPMis not part of SUSE Linux Enterprise Server 11. Find the latest release in the SUSEBuild Service ( http://software.opensuse.org/search?q=ClusterTools2&baseproject=SUSE%3ASLE-11&lang=en ).

For documentation of the general operating system configuration, the supportconfigscript from the supportutils RPM can be used. The supportutils RPM is part of SUSELinux Enterprise Server 11.


http://software.opensuse.org/search?q=ClusterTools2&baseproject=SUSE%3ASLE-11&lang=en

http://software.opensuse.org/search?q=ClusterTools2&baseproject=SUSE%3ASLE-11&lang=en

8Prepare SAN StorageNext, we will configure SAN access for a proof of concept, following these steps:

1. Configure the multipathed SAN access (to be done per site)

2. Create Partitions on the storage LUNs (to be done per site)

3. Prepare MD for cluster integration (to be done per site)

4. Set up DRBD for synchronization of the data from one site to the other (to be doneonce, after both sites are ready including MD)

5. Configure Logical VolumeManager (LVM) for our SAP scenario (the LVM settingshave to be done on both sides, the creation of LVM objects such as physical volumes,volume groups and logical volumes has to be done once on the DRBD master side)

6. Configure file systems for our SAP scenario (to be done once on the DRBD masterside)

7. Check the configuration

Refer to the product documentation for general information on storage administrationwith the SUSE Linux Enterprise Server 11 ( http://www.novell.com/documentation/sles11/stor_admin/?page=/documentation/sles11/stor_admin/data/bookinfo.html ).

Prepare SAN Storage 49

http://www.novell.com/documentation/sles11/stor_admin/?page=/documentation/sles11/stor_admin/data/bookinfo.html



8.1 Multipath ConfigurationEnable multipathing services boot.multipath and multipathd.

The multipathing drivers and tools support most storage arrays. Consult the vendor’shardware documentation to determine what settings are required. When storage arraysare automatically detected, the default settings for multipathing apply. If you want non-default settings, youmust manually change the configuration in the/etc/multipath.conf file.

Storage arrays that require special commands on failover from one path to the other orthat require special nonstandard error handling, may require hardware handlers. Consultthe hardware vendor’s documentation to determine if its hardware handler must be in-stalled.

To identify the correct devices and blacklist section entries, consult your SAN admin-istrator.

In our setup we use no-path-retry to avoid service interruption in case of short-time path failures. Nevertheless, path failure will result in read errors, which are detectedby the upper layers of the I/O stack. Read the LinuxMultipath documentation to decidewhether to use this option. We also set the failback time to a re-initiated path to a valuegreater than zero. This setting reduces the risk of running into I/O errors if a single SANpath starts frequently to go up and down in a very short time (path flapping). This settingcan be modified in the failback parameter.

Configure multipath to use user-friendly names. Definitions of user-friendly namesshould be done in a separate bindings file (/etc/multipath.bind) if you havemore than a few devices. Seemanmultipath.conf (bindings_file parameter)for details. We recommend to assign names to all LUNs. Mixing unassigned LUNswith assigned ones may lead to erratic behavior. A sample configuration for an HPEVA storage looks like this:

#/etc/multipath.confdefaults {

dev_loss_tmo 10fast_io_fail_tmo 5user_friendly_names yesbindings_file /etc/multipath.bind

}blacklist {


devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"devnode "^cciss!c[0-9]d[0-9]*"devnode "^dcssblk[0-9]*"

}devices {

device {vendor "HP|COMPAQ"product "HSV1[01]1 (C)COMPAQ|HSV2[01]0|HSV300|HSV4[05]0"path_grouping_policy group_by_priogetuid_callout "/lib/udev/scsi_id -g -u /dev/%n"path_checker turpath_selector "round-robin 0"prio aluarr_weight uniformfailback immediatehardware_handler "0"no_path_retry 5rr_min_io 100

}}

NOTE

This configuration is used for a particular environment only. Multipath configu-ration has to follow the hardware manufacturers recommendations and has tobe aligned with the storage administrator's concepts.

In general, the time multipathing needs to recover path failures should beshorter than the monitoring timeout of the storage stack resource agents.Otherwise a path failure could lead to node fencing in the worst case. On theother hand, sporadic path flapping should not lead to permanently disabledpathes.

To fine-tune the multipath behavior, specify the number of retries for a failedpath (no_path_retry), the retry interval, the failback time to a re-initiatedpath, and the failback policy. Details for specific hardware can be found in themultipath.conf man page (man 5 multipath.conf).

Usually it is a good idea to start without any device section, but use the compiled-indefaults.

To make configuration changes or changes inside the SAN visible, you may have toflush the multipath tables. After you modify the /etc/multipath.conf file, youmust run mkinitrd to re-create the INITRD on your system. Refer to the documen-tation mentioned above for details.


8.2 PartitioningSome SAN storages require partition alignment for performance reasons. Check thisand adapt the partitioning scheme if needed.

In this document only two LUNs are used for each site of the synchronization. Youmay also reserve some disk space (a small partition) if you want to integrate the SBDin the future. The SBD may not reside on a mirrored device.

# fdisk /dev/mapper/sapvol1...Disk /dev/mapper/sapvol1: 214.7 GB, 214748364800 bytes255 heads, 63 sectors/track, 26108 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/mapper/sapvol1-part1 1 125 1004031 83 Linux/dev/mapper/sapvol1-part2 127 26108 208700415 83 Linux# partprobe# kpartx -a /dev/mapper/sapvol1

Repeat this for the second LUN and on both sites.

8.3 MD ConfigurationProcedure 8.1 Configuring RAID1

1 Disable the /etc/init.d/boot.md service.

2 If non-cluster controlled MD devices are required, either replace them with ascript /etc/init.d/boot.non-cluster-md or similar, or use a script tomanually create MD devices and mount file systems.

3 Mount the file system to create sub-mountpoints beneath

4 /etc/mdadm.conf must contain a line to disable scanning and automatic as-sembling of MD devices. The file should also contain the information where theconfiguration files are placed and why:

# /etc/mdadm.conf# Never add any devices to this file


# Cluster mdadm configuration can be found# in /clusterconf/<sapinstance>/mdadm.conf## Always make sure that the bood.md service is disabled# chkconfig boot.md off## MD-Devices, that are not under cluster control are stored# in the file /etc/mdadm.conf.localdevices# The file /etc/mdadm.conf.localdevices is used by the boot# script /etc/rc.d/boot.non-cluster-md## Prevent mdadm from finding devices by auto-scan:DEVICE /dev/null#

5 Verify LUNs in /dev/mapper (names have to match exported names fromstorage systems)

6 Configure a RAID-1 array to hold your data. This array will act as a host-basedmirror that duplicates writes across the two storage LUNs.

This array will subsequently be used as the backing device of the DRBD resource.

To configure this array, use the mdadm utility. The following example assumesthat the RAID-1 array will be configured on two multipath devices (partitions)named sapvol1_part2 and sapvol2_part2.

Use the metadata format 1.2 or higher: mdadm –metadata=1.2.

mdadm --create --level=1 --raid-devices=2 \--bitmap=internal /dev/md0 \--metadata=1.2 \/dev/mapper/sapvol1_part2 /dev/mapper/sapvol2_part2

If the component devices are freshly installed and have been low-level formatted,you may add the --assume-clean option to mdadm. This skips the initialdevice synchronization.

7 For maximum availability, you must ensure that the two component devices residein distinct SAN enclosures. Preferably, these two enclosures should be located inseparate fire areas or in two different buildings.


8 After you have run mdadm, check the contents of /proc/mdstat to see thatthe array has been brought up properly.

9 Since both hosts on each site have access to the same storage LUNs, you maynow stop the array on one node then bring it up on the other.

WARNING

At this point no safeguards exist to prevent data corruption if the arrayis accessed by more than one node. You must ensure that the array isstopped on the one node before it is reassembled on the other.

Stop the array with the following command:

mdadm --stop /dev/md0

Check the status in /proc/mdstat (should be stopped now).

Executing the following command on the peer node will enable it there:

mdadm --assemble /dev/md0 \/dev/mapper/sapvol1_part2 /dev/mapper/sapvol2_part2

10 Check the status in /proc/mdstat (should be started now).

11 When this is completed, you should store the array configuration in a file separatefrom the default MD configuration file (/etc/mdadm.conf) to prevent thisarray from being automatically started at system boot-up. In this example, we areusing /etc/mdadm-cluster.conf as configuration file:

mdadm --detail /dev/md0 > /etc/mdadm-cluster.conf

12 Bring the RAID array back to the first node (similar procedure: stop on secondnode, check if the ARRAY is stopped, start the ARRAY on the first node, checkif the ARRAY is started).


NOTE

Once you are satisfied that the RAID-1 array has been set up correctly, repeatthe above steps on the nodes in the remote site.

8.4 DRBD SetupNext, configure a DRBD resource to hold your data. This resource will act as thePhysical Volume of an LVM Volume Group to be created later. This example assumesthat the LVMVolume Group is called sapvg, hence the DRBD resource uses the namesap.

Since the DRBD resource acts as a replication link between both sites and can alwaysrun on any of the two nodes in the same site, the configuration is of the floating peerstype. For more information on this type of setup, see [the DRBD User's Guide](http://www.drbd.org/users-guide/s-pacemaker-floating-peers.html).

We highly recommend that you put your resource configuration in a file that resides inthe /etc/drbd.d directory and whose name is identical to that of the resource:/etc/drbd.d/sap. Its contents should look similar to this:

resource sap {protocol A;device /dev/drbd0;syncer {

rate 30M;al-extents 1019;

}disk /dev/md0;meta-disk internal;floating 172.16.12.193:7788;floating 172.16.12.194:7788;

}

NOTE: To determine the right synchronization values, also read http://www.drbd.org/users-guide-8.3/s-configure-syncer-rate.html[The DRBDUser's Guide).


http://www.drbd.org/users-guide/s-pacemaker-floating-peers.html

http://www.drbd.org/users-guide/s-pacemaker-floating-peers.html

http://www.drbd.org/users-guide-8.3/s-configure-syncer-rate.html

http://www.drbd.org/users-guide-8.3/s-configure-syncer-rate.html

After you have created this resource and copied the configuration file to the other DRBDnode, you must initialize and synchronize the resource as specified in http://www.drbd.org/users-guide/(The DRBD User's Guide), Section "ConfiguringDRBD".

NOTE

When creating DRBD metadata as outlined in the User's Guide, you must doso on only one node per site.

Perform the procedure to create the DRBD devices on both sites:

Procedure 8.2 Creating the DRBD devices and starting the device for the first time

1 Create device metadata. This step must be completed only on initial device creation.It initializes DRBD's metadatadrbdadm create-md sap

2 To startup the DRBD device, use the command drbdadm up. This command in-cludes the steps 'attach' (assignment to the local storage devices), 'syncer' (settingthe synchronization values) and 'connect' (connecting to the remote site).drbdadm up sap

3 Check the status of the device using the proc file system.

# cat /proc/drbdcat /proc/drbdversion: 8.3.10 (api:88/proto:86-96)GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil@fat-tyre,2011-01-28 12:17:350: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent A r-----ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1023932

At this point the Secondary/Secondary and Inconsistent/Inconsistent status is normal.

The synchronization may only be started on one site:


http://www.drbd.org/users-guide/

http://www.drbd.org/users-guide/

Procedure 8.3 Performing the initial sync

1 Be careful with this step. Select the data source from which to sync to the other site.If there is no data on the device so far, you can sync in both directions. How ever,if you already have stored data on the device on one site, you must select the correctsource or you will lose data.

2 Starting the synchronization (here for resource sap)drbdadm -- --overwrite-data-of-peer primary sap

3 Check the status of the device using the proc file system:

# cat /proc/drbdls3198:/boot # cat /proc/drbdversion: 8.3.10 (api:88/proto:86-96)GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by phil@fat-tyre,2011-01-28 12:17:350: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent A r----ns:119552 nr:0 dw:0 dr:120088 al:0 bm:7 lo:0 pe:0 ua:0 ap:0 ep:1wo:b oos:904380[=>..................] sync'ed: 12.0% (904380/1023932)Kfinish: 0:41:52 speed: 320 (320) K/sec

Now the roles (Primary/Secondary) are changed and also the device status(UpToDate/Inconsistent). The synchronization is limited to 320K/sec.

4 You can change the sync rate for this initial synchronization.drbdsetup /dev/drbd0 syncer -r 110M

5 Check the status of the device using the proc file system:

# cat /proc/drbd...0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r----ns:1023932 nr:0 dw:0 dr:1024736 al:0 bm:63 lo:0 pe:0 ua:0 ap:0 ep:1

Now the device status is UpToDate/UpToDate on both sites.

6 Reset the changed sync rate.drbdadm adjust sap


8.5 LVM ConfigurationProcedure 8.4 Configure LVM

1 Adapt LVM filter.

It is necessary to instruct LVM to read Physical Volume signatures from DRBDdevices, rather than the underlying backing block devices. To do so, open the LVMconfiguration file (/etc/lvm/lvm.conf) and edit the following entry:

filter = [ "a|/dev/drbd.*|", "r|.*|" ]

This filter avoids scanning for VGs in /dev/disk* directories. If you are using VGsfor local file systems on your internal hard drives, make sure to add the local devicesto this filter (a|/dev/<my_device>).

2 In addition, you should disable the LVM cache by setting:

write_cache_state = 0

After disabling the LVM cache, make sure you remove any stale cache entries bydeleting /etc/lvm/cache/.cache.

You must repeat the above steps on all nodes in both sites.

3 Create PVs using pvcreate on MDs.

Before creating an LVM Volume Group, it is necessary to initialize the DRBD re-source as an LVM Physical Volume. After initiating the initial synchronization ofyour DRBD resource, issue the following command on the node where your resourceis currently in the Primary role:

pvcreate /dev/drbd/by-res/sap

4 Create VGs using vgcreate on PVs:


vgcreate sapvg /dev/drbd/by-res/sap

The logical extent size can be set to a value larger than 8MB, e.g. 64MB.

5 Create LVs using lvcreate on VGs. This example assumes three LVs named sapmnt,sapdb and usrsap:

lvcreate -n sapmnt -L 10G sapvglvcreate -n sapdb -L 100G sapvglvcreate -n usrsap -L 10G sapvg

IMPORTANT: It is not necessary to repeat pvcreate, vgcreate, or lvcreateon the DRBD peer node.

8.6 File System CreationProcedure 8.5 Creating file systems

1 Create ext3 file systems using mkfs.ext3 on the LVs.

Activate the VG and create file systems on the new LVs. This example assumes ext3as the file system type:

vgchange -a sapmkfs -t ext3 -m1 -Osparse_super /dev/sap/sapmntmkfs -t ext3 -m1 -Osparse_super /dev/sap/usrsapmkfs -t ext3 -m1 -Tlargefile4 -Osparse_super,resize_inode -E resize=500G/dev/sap/sapdb

NOTE

As with the LVM-related commands, it is not necessary to repeat mkfs onthe peer node.

Ext3 supports online resizing of file systems only if these file systems are createdwith the special parameters “-O resize_inode -Eresize=<max-online-resize>”.


<max-online-resize> specifies the maximum file system size (after resizing) innumber of blocks. If you omit this option, the default is used: a maximumfile systemsize of 1024 times the original file system size. We assume that the database filesystem might grow to 500GB.

# for f in sapdb sapmnt usrsap; do tune2fs -c0 /dev/sapvg/$f; done

Here we reduce the frequency of automatic file system checks by setting the mountcount to zero. You can also set the period between checks, e.g. if the file systemsare regularly checked during system maintenance.

2 Create mount points:

# mkdir -p /sapdb /sapmnt /usr/sap

3 Manually mount new file systems:

# mount -onoatime /dev/sapvg/sapdb /sapdb# mount -onoatime /dev/sapvg/sapmnt /sapmnt# mount -onoatime /dev/sapvg/usrsap /usr/sap

4 Create sub-directories in the correct place:

# mkdir /usr/sap/<SID># mkdir /var/sapcd

8.7 Check the SAN StorageProcedure 8.6 Checking the SAN Storage

1 Unmount the file systems, deactivate the VG, stop the DRBD device, stop the MDRAID device, stop multipathing on the first node, and check the status.

# df -h; df -i# umount /sapdb /usr/sap /sapmnt# vgchange -a n sapvg# drbdadm down sap# cat /proc/drbd# mdadm --manage --stop /dev/md0


# /etc/init.d/multipathd stop# /etc/init.d/boot.multipath stop

2 Restart the multipathing and the MD RAID device, DRBD device, activate the VG,mount the file systems, and check the status.

# /etc/init.d/boot.multipath start# /etc/init.d/multipathd start# multipath -ll# mdadm --assemble --config /etc/mdadm.cluster.conf /dev/md0# cat /proc/mdstat# drbdadm up sap# cat /proc/drbd# vgchange -a y sapvg# lvs# mount -onoatime /dev/sapvg/sapdb /sapdb# mount -onoatime /dev/sapvg/sapmnt /sapmnt# mount -onoatime /dev/sapvg/usrsap /usr/sap# df -h; df -i

3 Test if SAN access works with reasonable speed and without errors. The size of thetest file should be at least 1.5 times the RAM size to get a reliable speed estimation.Do not forget to remove the test file afterwards.

dd if=/dev/zero of=/sapdb/test.dd bs=256M count=64..., 245 MB/s# grep "I/O error" /var/log/messages

You should see no errors. The definition of 'reasonable speed' may vary. At the timethis document was written, a sustained linear write rate should be expected between50 MB/s and 200 MB/s. Consult the SAN storage administrator to find out currentspeed expectations.

For documentation of the storage-related OS configuration, use the supportconfig scriptfrom the supportutils RPM. The supportutils RPM is part of SUSE Linux EnterpriseServer 11.


9Configure the Cluster on bothSitesThis chapter covers the installation of the add-on product SUSE Linux Enterprise HighAvailability Extension on top of the already running SUSE Linux Enterprise Server 11SP1. The instructions also fit the installation of a cluster based on the product SUSELinux Enterprise Server for SAP Applications. The only difference is that SUSE LinuxEnterprise Server for SAP Applications 11 includes the SUSE Linux Enterprise ServerHigh Availability Extension 11. You do not need to add an aditional media.

Wewill demonstrate a basic cluster configuration. Afterwards the cluster will be preparedto receive the resource configuration needed for the SAP Simple Stack High Availabil-ity scenario. This basic configuration is the same for other high availability scenariosmentioned in this document.

• Installation of packages

• Basic cluster setup and configuration of crm

• Configuration of STONITH

• Configuring ressources for storage including DRBD

• Configuring ressources for virtual IP addresses

• Checking the results

Refer to the product documentation for general information on the SUSE Linux Enter-priseHighAvailabilityExtension (http://www.novell.com/documentation/sle_ha/ ).

Configure the Cluster on both Sites 63

http://www.novell.com/documentation/sle_ha/


9.1 Install the SUSE Linux EnterpriseHigh Availability ExtensionSoftware Packages on all Nodes

If you have installed "SLES for SAP" and included the High Availability pattern duringthe installation, you can skip this section and proceed with Section 9.2, “Basic Clusterand CRM Configuration” (page 65).

1. Open the YaST Sofware Management module.

2. Add the installation source for SUSE Linux Enterprise High Availability Extension(you can skip this step for SUSE Linux Enterprise Server for SAP Applications asthe packages are included in this product).

3. Select the following pattern:+ High availability pattern

4. Click “Accept” to start the installation.

The installation procedure should be well known to the target audience and thereforeis not described in detail.

We recommend to update the cluster management software to the current version. Eitherconnect to the NCC via the Internet or use a locally installed update proxy like SMT.The update procedure should be well known to the target audience and therefore is notdescribed. As of the publication date of this document, you should have at least thefollowing versions installed:

• drbd-8.3.8.1-0.2.9

• drbd-utils-8.3.8.1-0.2.9

• drbd-bash-completion-8.3.8.1-0.2.9

• drbd-udev-8.3.8.1-0.2.9

• drbd-pacemaker-8.3.8.1-0.2.9


• drbd-kmp-default-8.3.8.1_2.6.32.19_0.2-0.2.5

• corosync-1.2.6-0.2.2

• cluster-glue-1.0.6-0.3.7

• libcorosync4-1.2.6-0.2.2

• libglue2-1.0.6-0.3.7

• libpacemaker3-1.1.2-0.7.1

• ldirectord-1.0.3-0.4.8

• openais-1.1.3-0.2.3

• pacemaker-1.1.2-0.7.1

• pacemaker-mgmt-2.0.0-0.3.10

• pacemaker-mgmt-client-2.0.0-0.3.10

• resource-agents-1.0.3-0.4.8

• yast2-drbd-2.13.1-217.39.19

9.2 Basic Cluster and CRMConfiguration

9.2.1 Configure OpenAIS BasicsConfiguring Corosync in a 2-node cluster and enabling Pacemaker is a straightforwardprocess that is well documented at http://www.clusterlabs.org/wiki/Initial_Configura-tion[the Pacemaker web site].

You must configure two distinct clusters (one cluster per site), with each cluster con-sisting of two nodes. Do not attempt to create a 4-node cluster that spans two sites.


Configure the Cluster Start ModeThere are controversial discussions about whether it makes sense to start the clustermanager OpenAIS automatically during system boot. We recommend not to startOpenAIS during system boot. If a cluster node gets fenced (rebooted by a STONITHaction) during a failure situation, an automatic OpenAIS start would cause this node torejoin the cluster. Depending on the cluster configuration, a failback of some resourcesmay occur. If the failure situation appears again on this node, another fence (reboot)may happen. This could lead to a reboot loop and may also affect the availability of theclustered SAP services.

To disable the automatic start of OpenAIS, use the following command:

# chkconfig openais off

Note: Disabling the automatic start of OpenAIS will cause SAP databases and instancesto not start automatically after a system boot.

To start OpenAIS manually after the system boot, type

# /etc/init.d/openais start

Configure the CorosyncDo not forget to make all changes on both cluster nodes.

OpenAIS uses configuration files residing in the directories /etc/openais and/etc/corosync. Use YaST to create these files or edit the files directly if you are familiarwith the file formats. In most cases, the files must be identical on all cluster nodes.

The /etc/corosync/corosync.conf file configures all parameters for the basic clusteroperation.

Configuration, distributed to both nodes:

# /etc/corosync/corosync.conf - Demo Cluster Setuptotem {...interface {


#Network Address to be bind for this interface settingbindnetaddr: 192.168.0.0#The multicast address to be usedmcastaddr: 238.50.1.1#The multicast port to be usedmcastport: 5405#The ringnumber assigned to this interface settingringnumber: 0

}#To make sure the auto-generated nodeid is positiveclear_node_high_bit: yes

}...#

Starting the Cluster for the First TimeBefore we start the cluster, we need to check:

• Does the network environment allow/support multicast communication?

• Are the network links up and configured?

• Is the firewall down or adapted to cluster needs? (this document does not discussfirewall settings)

• Are users and passwords identical on both nodes (hacluster)?

• Does name resolution (DNS, /etc/hosts) and time service (NTP) work correctly onboth nodes?

If you are ready to start the cluster manager, call rcopenais start on both nodes.After some time, call the command crm_mon -r to see if the cluster is working andboth nodes are online. Have a look at the SUSE Linux Enterprise High AvailabilityExtension documentation for more details.

Creating a Basic Pacemaker CRM ConfigurationIn a highly available SAP server configuration that involves a 2-node cluster, you should

• set Pacemaker's "no quorum policy" to ignore loss of quorum;


• set the default resource stickiness to 200.

To do so, issue the following commands in the CRM shell:

crm(live)# configurecrm(live)configure# property no-quorum-policy="ignore"crm(live)configure# rsc_defaults resource-stickiness="200"crm(live)configure# commit

9.3 STONITH Resource ConfigurationOnce you have established cluster communications between all Corosync/Pacemakernodes, configure STONITH (Shoot The Other Node In The Head) resources for nodefencing. This is required to avoid uncoordinated, potentially destructive access to sharedstorage.

Pacemaker supports a large number of STONITH devices; listing all of them would bewell beyond the scope of this guide. The examples below assume that the cluster nodesare equipped with IPMI fencing devices.

IMPORTANT: For fencing to work properly, IPMI relies on network connectivity be-tween the node issuing the fencing request and the node to be fenced. This means thenodes must use a network path that is independent from the network paths used forcluster communication.

In this example, host hpn07 connects to hpn08's IPMI device at the IP address192.168.1.43. In contrast, hpn08 reaches hpn07's IPMI device via 192.168.1.42. BothIPMI devices can be accessed with the username ipmi and the password secret.

Configure the STONITH resource for fencing host hpn07 as follows (note that this isthe resource to fence hpn07, consequently it must never run on hpn07).

crm(live)configure# primitive p_ipmi_hpn07 stonith:external/ipmi \params hostname=hpn07 ipaddr=192.168.1.42 \userid=ipmi passwd=secret \op monitor interval=30s

crm(live)configure# location l_stonith_not_on_hpn07 p_ipmi_hpn07 \-inf: hpn07

crm(live)configure# commit


Likewise, configure a STONITH resource for fencing +hpn08+ as shown here:

crm(live)configure# primitive p_ipmi_hpn08 stonith:external/ipmi \params hostname=hpn08 ipaddr=192.168.1.43 \userid=ipmi passwd=secret \op monitor interval=30s



Once these two resources have been committed, observe crm_mon to verify that theSTONITH resources have started on the correct nodes. Recall that a running STONITHresource simply means it is ready to fence the node it manages, not that it is in factfencing.

Repeat the above steps for the two nodes in the remote site.

NOTE

For further information, an excellent guide on configuring STONITH is availableat the Pacemaker Web site: http://www.clusterlabs.org/doc/crm_fencing.html.

9.4 Storage Resource Configuration

9.4.1 RAID-1To enable your RAID-1 array as a cluster resource, issue the following commands inthe crm shell:

crm(live)configure# primitive p_raid1 ocf:heartbeat:Raid1 \params raidconf="/etc/mdadm-cluster.conf" raiddev="/dev/md0" \op monitor interval="20s"



http://www.clusterlabs.org/doc/crm_fencing.html

http://www.clusterlabs.org/doc/crm_fencing.html

Once you have committed this configuration, Pacemaker should assemble the array onone of the two nodes. You may verify this with crm_mon or check the contents of the/proc/mdstat virtual file.

As always, repeat this step on the cluster nodes at the remote site.

9.4.2 DRBDThe DRBD configuration in a split-site setup, where DRBD replicates data that is storedon shared storage, requires that:

• the DRBD peers are configured to float, meaning they are not tied to an individualphysical host;

• each site only contains one DRBD peer;

• only one of the sites is configured to allow DRBD to assume the Primary role.

First, configure a cluster IP address to be used by the floating DRBD service. This ex-ample assumes that DRBD uses IP address 172.16.12.193 in a class C (24-bit) subnet:

crm(live)configure# primitive p_ip_drbd ocf:heartbeat:IPaddr2 \params ip="172.16.12.193" cidr_netmask="24" \op monitor interval="10s"


You must not duplicate this configuration snippet verbatim on the remote site. Instead,configure the remote site to use the other DRBD peer's floating IP address -- in thisexample 172.16.12.194:

crm(live)configure# primitive p_ip_drbd ocf:heartbeat:IPaddr2 \params ip="172.16.12.194" cidr_netmask="24" \op monitor interval="10s"


Now, combine these two resources into one resource group (on both sites):

crm(live)configure# group g_drbd_prereq p_raid1 p_ip_drbdcrm(live)configure# commit


At this point, you have set up the prerequisites necessary for DRBD to function: thebacking device (your MD array) and the IP address for a floating configuration. Now,proceed with the DRBD configuration, starting with the local site.

crm(live)configure# primitive p_drbd_sap ocf:linbit:drbd \params drbd_resource="sap" \op monitor interval="30s" role="Master" \op monitor interval="20s" role="Slave"

crm(live)configure# ms ms_drbd_sap p_drbd_sap \meta notify="true" master-max=1 clone-max=1 \target-role="Master"

crm(live)configure# colocation c_drbd_on_prereq \inf: ms_drbd_sap g_drbd_prereq

crm(live)configure# order o_prereq_before_drbd \inf: g_drbd_prereq ms_drbd_sap


Of crucial importance is theclone-max=1meta variable for themaster/slave resource.In this configuration, one Pacemaker cluster must manage only one DRBD peer, notboth. Note also that for this site, we set the target-role to Master.

Proceed with the DRBD configuration for the remote site:

crm(live)configure# primitive p_drbd_sap ocf:linbit:drbd \params drbd_resource="sap" \op monitor interval="30s" role="Master" \op monitor interval="20s" role="Slave"

crm(live)configure# ms ms_drbd_sap p_drbd_sap \meta notify="true" master-max=1 clone-max=1 \target-role="Slave"




Here we set the target-role to Slavemeaning that without manual intervention,Pacemaker will never promote the DRBD resource to the Primary role here. This metaattribute acts as our site fail-over trigger as we will see later. Otherwise, the resourceconfiguration is identical to the one in the local site.


9.4.3 LVM and File SystemsNow that MD and DRBD are running, we will make our LVM volume group (sapvg)a highly available cluster resource:

crm(live)configure# primitive p_lvm_sap ocf:heartbeat:LVM \params volgrpname="sapvg" \op start interval="0" timeout="30" \op stop interval="0" timeout="30" \op monitor interval="10" timeout="30"

Likewise, we must create Filesystem resources managing the file systems residingon these volumes:

crm(live)configure# primitive p_fs_sap_db ocf:heartbeat:Filesystem \params device="/dev/sapvg/sapdb" directory="/sapdb/" fstype="ext3" \op start interval="0" timeout="60" \op stop interval="0" timeout="60" \op monitor interval="20" timeout="40"

crm(live)configure# primitive p_fs_sap_instance ocf:heartbeat:Filesystem \params device="/dev/sapvg/usrsap" directory="/usr/sap" fstype="ext3" \op start interval="0" timeout="60" \op stop interval="0" timeout="60" \op monitor interval="20" timeout="40"

crm(live)configure# primitive p_fs_sapmnt ocf:heartbeat:Filesystem \params device="/dev/sapvg/sapmnt" directory="/sapmnt" fstype="ext3" \op monitor interval="20s" \op monitor interval="0" timeout="40s" \op start interval="0" timeout="60s" \op stop interval="0" timeout="40"

Conveniently, we can now combine all these resources into one resource group:

crm(live)configure# group g_sap p_lvm_sap p_fs_sapmnt \p_fs_sap_db p_fs_sap_instance

Now we introduce order and colocation constraints to this resource group so that it isonly started on a node where the DRBD resource is in the Primary role:

crm(live)configure# colocation c_sap_on_drbd \inf: g_sap ms_drbd_sap:Master

crm(live)configure# order o_drbd_before_sap \inf: ms_drbd_sap:promote g_sap:start



Once these changes have been committed, Pacemaker should perform the followingactions:

• activate the sapvg Volume Group on whatever node currently holds the floatingDRBD resource in the Primary role;

• mount the three Logical Volumes in the VG into the specifiedmount point directories.

Verify the correct functionality with the crm_mon, vgdisplay or vgs and mountcommands.

If you are satisfied with the results, duplicate the configuration on the remote site. Therethe ms_drbd_sap resource group is barred from being promoted by itstarget-role=Slave meta attribute. Since the g_sap resource group depends onthe Master role of ms_drbd_sap, the g_sap resource group does not start in theremote site.

9.5 Configure Virtual IP AddressesNowwe configure the virtual IP addresses which are needed to install the SAP softwareproperly. We need three virtual IP addresses, which must match the virtual host namesused during the SAP installation:

• 172.16.12.190 matching sap<sid>db (here: sapdrbdb) for the MaxDB instance.

• 172.16.12.191 matching sap<sid>as (here: sapdrbas) for the central service instance.

• 172.16.12.192 matching sap<sid>ci (here: sapdrbci) for the first application server.

To set these up, use resources of the IPaddr2 type:

crm(live)configure# primitive p_ip_sapdb ocf:heartbeat:IPaddr2 \params ip="172.16.12.190" cidr_netmask="24" \op monitor interval="10s"

crm(live)configure# primitive p_ip_ascs ocf:heartbeat:IPaddr2 \params ip="172.16.12.191" cidr_netmask="24" \op monitor interval="10s"

crm(live)configure# primitive p_ip_ci ocf:heartbeat:IPaddr2 \


params ip="172.16.12.192" cidr_netmask="24" \op monitor interval="10s"

Once set up, you can add the new resources to the existing g_sap group. Edit thegroup:

crm(live)configure# edit g_sap

Use an editor to modify the group so it has the following contents:

group g_sap p_lvm_sap p_fs_sapmnt \p_fs_sap_db p_ip_sapdb \p_fs_sap_instance \p_ip_ascs \p_ip_ci

Finally, commit this configuration:



10Install SAP NetWeaver 7.0EHP1In this section we describe the installation of SAP NetWeaver 7.0 EHP1 on SUSELinux Enterprise Server 11 SP1 in a “Simple Stack standalone” scenario. All componentsare placed on one single system. It is prepared to be integrated in a high availabilitycluster according to the SAP “Simple Stack High Availability” scenario. We install theSAP system components, using several virtual hostnames to match the high availabilityinstallation needs.

10.1 SAP Installation ProcedureIn the following we describe the installation process step-by-step for an easy typicalSAP installation. The installation will take several hours, depending on the performanceof your hardware. If you plan to disconnect your workstation from the network duringthe installation, use an X-session manager like NoMachine NX or vncserver.

To cover high availability aspects and also to be able to place some of the componentsof the SAP system on different hosts later, we use the installation workflow “HighAvailability System”. This leads into a separate installation of the central services in-stance (ASCS), the database and the central instance (DVEBMGS).

• The three IP addresses for the SAP virtual hostnames have to be set up and the namesmust be resolved in /etc/hosts.

• You need a root session with X-forwarding. So use either ssh to login to the serversystem, including forwarding options (like -Y or -X).ssh -Y root@server

Install SAP NetWeaver 7.0 EHP1 75

• Enter the mounted installation master directory. The correct path depends on yourselections made above. If you are following our sample, the path is:

/sapcd/InstMacd /sapcd/InstMa

• The installation master for UNIX systems can be used for installations on AIX(PPC64), HPUX (PARISC), Linux (i386, ia64, PPC64, S390_64 and x86-64),OS390_32, OS400 (PPC64), Solaris (SPARC), Solaris (x86-64) andWindows (i386,ia64 and x86-64). In our sample we select Linux for x86-64 architecture and enterthe directory IM_LINUX_X86_64.cd IM_LINUX_X86_64

10.2 Installation of the CentralServices Instance ASCS

• Start the SAP InstallationMaster by running the sapinst command: ./sapinstSAPINST_USE_HOSTNAME=sap<sid>as. Instead of “sap<sid>as” use yourvirtual hostname reserved for the ASCS. At the top of the screen you will see thefive major phases of the installation procedure. The highlighted number representsthe active step. The first step is named “1 - Choose Option”.

• In the left area, select “SAP NetWeaver 7.0 including Enhancement Package 1 ” →“SAP Application Server ABAP” → “Max DB” → “High-Availability System” →“Central Services Instance for ABAP (ASCS)”. The screen shot shows the expandedservice tree. If you have selected “Central Services Instance for ABAP (ASCS)”,click the “Next” button on the left side.

• With the next dialog you reach major phase “2 - Define Parameters”. The first dialogof this step is labeled “Parameter Mode > Default Settings”. There are two radiobuttons to select the parameter mode, “typical” and “custom”. In this document, weprefer the “typical” parameter mode to reduce the selection of prompts. Click “Next”to proceed to the following dialog.

• The dialog “SAP System > General Parameters” is used to define the SAP systemidentifier SID (we use “DRB” in our example) and the SAP system mount directory


(we use /sapmnt). Since we install an unicode system, we activate the checkmark.Click “Next” to proceed.

Figure 10.1 Start Dialog of the SAP Installation Manager

• In the Dialog “SAP System > Administrator Password”, set the password for theLinux user <sid>adm. You can also define the unique user ID and the group ID forthe Linux user group sapsys. Click “Next” to proceed.

• Now the installer asks for the two-digit instance number for the central services in-stance (ASCS). In our example we use 00. Click “Next” to proceed.

• In the dialog “SAP System > ASCS Instance”, do not change the port numbers forthe ASCS if you want to follow our example. Just click “Next” to proceed.

• The “Media Browser > Software Package” dialog asks for an additional path, theinstallation media for the NetWeaver kernel (ABAP). In our sample the path is/sapcd/kernel. You can either enter the path directly in the input field or use the filebrowser. After you have provided the correct path, click “Next” to proceed.

• The installation enters the phase “3 - Summary”. The last step before the installationof the central services instance (ASCS) is “Parameter Summary” dialog. Double-check the settings. If everything is correct, click “Next” to proceed.


• The installer switches to phase “4 - Execute”. The dialog “Task Progress” providesan overview of the installation progress and about the status of the scheduled tasks.The status bar at the bottom of the window gives some detailed information for taskswhich are running for a long time.

• If the installation is successful, the installer reaches phase “5 - Completed”.

10.3 Installation of the Database• Start the SAP Installation Master by running the sapinst command: ./sapinstSAPINST_USE_HOSTNAME=sap<sid>db. Instead of “sap<sid>db” use yourvirtual hostname reserved for the database. At the top of the screen you see again thefive major phases of the installation procedure. The highlighted number representsthe active step. The first step is named “Choose Service”.

• In the left area, select “SAP NetWeaver 7.0 including Enhancement Package 1 ” →“SAP Application Server ABAP” → “Max DB” → “High-Availability System” →“Database Instance”. The screen shot shows the expanded service tree. If you haveselected “Database Instance”, click the “Next” button on the left side.

• The installer switches to phase “2 - Define Parameters”. The dialog is titled “SAPSystem > General Parameters”. Make sure the checkmark “Profiles are available” isset and provide the “Profile Directory” path. Following our example, “/sapm-nt/<SID>/profile” is a good choice. Replace “<SID>” with your SAP system ID.Click “Next” to proceed.

• In the dialog “SAP System > Master Password” set the start password for Linux,SAP, the database (such as <sidadm>, <sid>user, ...) and SAP system users(such as DDIC). Click “Next” to proceed.

• In the dialog “SAP System Database”, provide a database ID (in our example we use<SID>. The “Database Host” should be the one which you provided on the commandline while starting sapinst (sap<SID>db in our example). Click “Next” to proceed.

• In the dialog “MaxDB > Database Software Owner”, specify the name, group andIDs of the Linux user “owning” the software. Typically the values should be “sdb”and “sdba”. If you do not intend to set specific user and group IDs, let the installerchoose these values. Click “Next” to proceed.


• In the dialog “MaxDB > Database Instance Software Owner”, specify the name,password and IDs of the Linux user “owning” the database instance. Typically, thevalues should be “sqd<sid>” and “sdba”. If you do not intend to set specific userand group IDs, let the installer choose these values. Click “Next” to proceed.

• The “Media Browser > Software Package” dialog asks for the full path of the instal-lation media “Installation Export NW...”. In our sample the path is /sapcd/InstExp. Click “Next” to proceed.

• The installer asks for the target path for the database and database instance installation.In our example we choose “/sapdb”. Click “Next” to proceed.

• The “Media Browser > Software Package” dialog asks for an additional path, theinstall media for the MaxDB RDMBS. In our example the path is /sapcd/MaxDB.Click “Next” to proceed.

• Now the installer asks for the passwords to be used for database users “superdba”and “control”. Click “Next” to proceed.

• In the dialog “MaxDB > Database Parameters” you can set some major installationand configuration parameters. Themost important one for our example is the “VolumeMedium Type”. This parameter must be set to “File System” if you want to followour example. Click “Next” to proceed.

• In the “MaxDB > Log Volumes” dialog, tune and size the database log area. In ourexample we do not change any values. Click “Next” to proceed.

• In the dialog “MaxDB > Data Volumes”, tune and size the database files for objectslike tables, indexes and so on. In our example we do not change any values. Click“Next” to proceed.

• The installer shows the dialog “MaxDB > ABAP Database Schema Password” andasks for the password of SAP<SID> scheme. Click “Next” to proceed.

• In the dialog “SAP System > Database Import”, define the SAP code page and thenumber of maximum parallel import jobs. In our example we do not change anyvalues. Click “Next” to proceed.

• The “Media Browser > Software Package” dialog asks for an additional path, theinstall media for the NetWeaver kernel (ABAP). In our example the path is /sapcd/


kernel. You can either enter the path directly in the input field or use the filebrowser. After you have provided the correct path, click “Next” to proceed.

• The dialog “SAP System > Unpack Archives” should show a list of archives to beunpacked. Normally you do not need to change anything here. Click “Next” to pro-ceed.

• The installation is now in phase “3 - Summary”. The last step before the installationof the central services instance (ASCS) is the “Parameter Summary”. Double-checkthe settings. If everything is correct, click “Next” to proceed.

• The installer switches to phase “4 - Execute”. The “Task Progress” dialog providesan overview of the installation progress and the status of scheduled tasks. The statusbar at the bottom of the window gives some detailed information for tasks which arerunning for a long time. The installation of the database software is quite fast, butthe step “Import ABAP” could take several hours, depending on the performance ofyour hardware. Keep the installer GUI open to get either the final success messageor an error information.

• If the installation is successful, the installer switches to phase “5 - Completed”.

10.4 Installation of the CentralInstance

• Start the SAP Installation Master by running the sapinst command: ./sapinstSAPINST_USE_HOSTNAME=sap<sid>ci Instead of “sap<sid>ci” use yourvirtual hostname reserved for the central instance. At the top of the screen you willsee the five major phases of the installation procedure. The highlighted number rep-resents the active step. The first step is named “Choose Service”.

• In the left area, select “SAP NetWeaver 7.0 including Enhancement Package 1 ” →“SAP Application Server ABAP” → “Max DB” → “High-Availability System” →“Central Instance”. The screen shot shows the expanded service tree. If you haveselected “Central Instance”, click the “Next” button on the left side.

• In the dialog “SAP System >General Parameters” define the “Profile Directory” (weuse /sapmnt/<SID>/profile). Click “Next” to proceed.


• In the dialog “SAP System > Master Password”, you provide the already definedpassword for all users which are created during the installation. Click “Next” toproceed.

• The dialog “SAP System > Central Instance” provides a list of already installed SAPinstances. Specify the “Central Instance Number”. In our example we use “01”. Click“Next” to proceed.

• The installer will show the dialog “SAP System > DDIC Users”. In our example thecheckmark “DDIC user has a password different from default” should not be set.This tells the installer to use the master password provided earlier. Click “Next” toproceed.

• The installer needs to kow the password used during the “Database Instance” instal-lation. Provide the master password here if you have used the same password. Click“Next” to proceed.

• In the dialog “MaxDB > ABAP Database Schema Password”, provide the passworddefined during the “Database Instance” installation. Enter the master password if youhave used the same password. Click “Next” to proceed.

• The “Media Browser > Software Package” dialog asks for an additional path, theinstall media for the NetWeaver kernel (ABAP). In our sample the path is/sapcd/kernel. You can either enter the path directly in the input field or use the filebrowser. After you have provided the correct path, click “Next” to proceed.

• The dialog “SAP System > Unpack Archives” should show a list of archives to beunpacked. Normally you do not need to change anything here. Click “Next” to pro-ceed.

• The installation now reaches phase “3 - Summary”. The last step before the installationof the central services instance (ASCS) is the “Parameter Summary”. Double-checkthe settings. If everything is correct, click “Next” to proceed.

• The installer switches to phase “4 - Execute”. The “Task Progress”s dialog providesan overview of the installation progress and the status of scheduled tasks. The statusbar at the bottom of the window may also give some detailed information for taskswhich are running for a long time.

• In this phase, the installer needs a valid “Solution Manager Key” (listed as prerequi-site). You need to create such a “Solution Manager Key” using your local “Solution


Manager”. This is a separate SAP product, which is used for central SAP systemmaintenance. This document does not cover the installation of this product. Refer tothe SAP installation documentation if have not already installed your “SolutionManager”.

• If the installation is successful, the installer switches to phase “5 - Completed”.

10.5 Check the SAP InstallationUse the command „dpmon -pf=<instance_profile>“ to check the status ofthe SAP processes for each application instance (ASCS, CI/PAS) of an ABAP system.You will see the result of transaction SM50 without logging on to the SAP system.

• Use the SAP client to access the installed SAP system. Depending on your SAP clientinstallation (mentioned in the prerequisites), you can start the SAP session eitherwith guilogon to define the SAP client connections. With guistart you can accessyour SAP system directly if you already know the SAP logon string:

# export PLATIN_JAVA=/usr/lib/jvm/java-1_5_0-ibm-1.5.0_sr8a/bin/java;# guistart /H/<server>/S/<port>

Figure 10.2 SAP Client


Server is your (virtual) server name and port is the SAP port. The SAP port can becalculated as: port = 3200 + instance_number. In case the SAP central instance hasthe instance number 02 and our SAP server is named sap01. So the correct accessstring is: /H/sap01/S/3202. Login as user DDIC with the password given during theinstallation.

• Test stopping SAP and database processes. Login as user<sid>adm (here:drbadm)and stop ASCS, DB, and CI.

# stopsap r3 DVEBMGS01 sapdrbci# stopsap r3 ASC00 sapdrbas# stopdb

• Test starting SAP and database processes. Double-check the three virtual IP addressesneeded by the SAP processes. Login as user <sid>adm and start ASCS, DB, andCI.

# startsap r3 ASC00 sapdrbas# startdb# startsap r3 DVEBMGS01 sapdrbci

Now check all processes again.


11Integrating SAP into thecluster

11.1 SAP resource configurationOnce the SAP database and instances have been properly installed, you can add SAPresources to the cluster configuration. This example assumes

• one SAP database of the ADA (MaxDB) type, listening on 172.16.12.190,

• one SAP instance named DRB_ASC00_sapdrbas, listening on 172.16.12.191,

• one SAP instance namedDRB_DVEBMGS01_sapdrbci, listening on 172.16.12.192.

To set these up, use resources of the SAPDatabase, and SAPInstance types.

crm(live)configure# primitive p_sapdatabase ocf:heartbeat:SAPDatabase \params DBTYPE="ADA" SID="DRB" \DIR_EXECUTABLE="/usr/sap/DRB/ASC00/exe" \AUTOMATIC_RECOVER="true" \op monitor interval="20s" timeout="20s" \op start interval="0" timeout="360" \op stop interval="0" timeout="360"

crm(live)configure# primitive p_sap_ascs ocf:heartbeat:SAPInstance \params InstanceName="DRB_ASC00_sapdrbas"AUTOMATIC_RECOVER="true" \op monitor interval="120s" timeout="60s" \op start interval="0" timeout="180s" \op stop interval="0" timeout="240s"

crm(live)configure# primitive p_sap_ci ocf:heartbeat:SAPInstance \params InstanceName="DRB_DVEBMGS01_sapdrbci"AUTOMATIC_RECOVER="true" \

Integrating SAP into the cluster 85

op start interval="0" timeout="180" \op stop interval="0" timeout="240" \op monitor interval="120s" timeout="60s"

Once set up, you can add the new resources to the existing +g_sap+ group. Edit thegroup:

crm(live)configure# edit g_sap

Then, using the editor, modify the group so it has the following content:

group g_sap p_lvm_sap p_fs_sapmnt \p_fs_sap_db p_ip_sapdb p_sapdatabase \p_fs_sap_instance \p_ip_ascs p_sap_ascs \p_ip_ci p_sap_ci

Finally, commit this configuration:


When this configuration is complete, Pacemaker should perform the following actionsin this order:

• start the database cluster IP address and the SAP database;

• start the cluster IP address and SAP instance for DRB_ASC00_sapdrbas;

• start the cluster IP address and SAP instance for DRB_DVEBMGS01_sapdrbci.

If everything has started up as expected, duplicate the configuration on the remote site.As previously discussed in Section 8.5, “LVM Configuration” (page 58), the g_sapresource group, although configured, is not expected to start at the remote site at thispoint.


12Testing the ClusterThis chapter describes a number of tests that should be conducted on the describedsetup before going into production. It is not an exhaustive list, rather a minimum require-ment for a testing procedure.

12.1 Check the SAP IntegrationUse crm_mon or hb_gui to check,if the resources are running on one cluster node.hb_gui should show all resources GREEN.

Check the needed default policies for stonith-enabled,default-resource-stickiness, and no-quorum-policy. For example,use cibadmin -Q | grep <string>.

Use the crm_gui to migrate the whole SAP system to the other node.

Login on the node that runs the SAP system and issue “rcopenais stop” to stopthe cluster manager gracefully. Restart the cluster manager after all resources have beenmigrated to the other node.

Conduct some more tests from the test table below. Discuss beforehand which tests areuseful in your particular environment. For productive clusters, a test plan for all possiblefailure scenarios is necessary.

Testing the Cluster 87

Table 12.1 Basic Cluster Tests

Procedure and Expected ResultTest Case

Procedure: Neither node has any cluster services running. Startthe cluster service on both nodes as root:

Administrativecluster manag-er start-up

rcopenais start

Expected behavior: The init script starts corosync andpacemaker. Cluster manager joins both nodes to the cluster andstarts application on one cluster node.

Procedure: Both nodes running and joined to the cluster. Stop thecluster service on the node which is currently running the applicationas user root:

Administrativecluster manag-er shutdown(on one node)

rcopenais stop

Expected behavior: Cluster manager migrates running applicationto peer node, removes node from cluster, then pacemaker andcorosync both shutdown.

Procedure: Both nodes running and joined to the cluster. On thenode which is currently running the application, forcibly shut downall pacemaker and corosync services:

Clustermanag-er softwarefailure

kill -9 corosync and pacemaker pids

Expected behavior: Pacemaker on surviving nodemarks offendingnode as UNCLEAN, initiates fencing via STONITH.When offend-ing node is confirmed to have left the cluster, status changes toOFFLINE, and Pacemaker starts RAID, DRBD, and applicationson surviving node.

Procedure: Both nodes running and joined to the cluster. Initiatea graceful reboot on the node currently running the application:

Cluster nodereboot



reboot

Expected behavior: Cluster manager migrates resources to thenon-rebooting node, then shuts down cluster services and initiatesa reboot. After the reboot, the cluster services _do not_ restart auto-matically.

CAUTION:When automatic cluster service restart is disabled (withinsserv -r openais), openais is not only removed from thestartup, but also from the shutdown sequence. Thus, when the ad-ministrator initiates a reboot, the system does not gracefully shutdown cluster resources prior to unconfiguring the network. Thereforefailover is affected by a STONITH sequence as described in test"cluster manager software failure". As a consequence, what shouldbe a graceful resource migration turns into forced failover, possiblyprompting data recovery (and hence, greater-than-necessary servicedisruption) on the surviving node. The system startup/shutdownsequence on SLE must be enhanced for this test to complete as ex-pected.

NOTE: As a workaround for this problem, system administratorsshould always manually shut down cluster services (rcopenaisstop) before initiating a reboot.

Procedure: Both nodes joined to the cluster, applications running.Forcibly reboot the node that is currently running the application:

Hard node fail-ure

echo b > /proc/sysrq-trigger

Alternatively switch off the node using a remote console.

Expected behavior: Pacemaker on surviving nodemarks offendingnode as UNCLEAN, initiates fencing via STONITH.When offend-ing node is confirmed to have left the cluster, status changes to



OFFLINE, and Pacemaker starts RAID, DRBD, IP adresses andapplications on the surviving node.

Procedure: Disable interface used by one Corosync ring:Single clusterlink failure

ip link set down interface

NOTE: If a Corosync ring uses a bonding interface (recommended),repeat the above command with all constituent devices.

Expected behavior: No change in cluster status (except if usingocf:pacemaker:ping uplink monitoring, not covered in thisguide).

Procedure: On the node running g_sap, disable all network inter-faces used for cluster communications. Repeat the following com-

Complete clus-ter communica-tion failure mand for all cluster interfaces (and for all constituent devices if

using bonding)

ip link set down interface

Expected behavior: Pacemaker on surviving nodemarks offendingnode as UNCLEAN, initiates fencing via STONITH.When offend-ing node is confirmed to have left the cluster, status changes toOFFLINE, and Pacemaker starts RAID, DRBD, and applicationson surviving node.

Procedure Switch the status of the node running the SAP systemto standby using either the crm command line or theClusterService command (ClusterTools2 package).

Set one nodeto standby

crm node standby hpn07

If you are used to running ClusterService, you can do the samewith:ClusterService --cmd NSS grimsel



Expected behavior: Database and CI instance group is runningon remaining node. Virtual IPs are running on remaining node.

12.2 Application Cluster TestsTable 12.2 Application Cluster Tests


Procedure: Both nodes running and joined to the cluster, resourcesstarted. Migrate the SAP resource group to a specific node:

Resourcemigra-tion

crm resource move g_sap node

Expected behavior: CRM shell places a location constraintsticking the +g_sap+ resource group onto the specified node.Pacemaker shuts down the resource on the original node and bringsit back up on the specified node.

The location constraint persists after this action, and can subsequent-ly be removed with the following command:

crm resource unmove g_sap

Procedure: Forcibly shut down the SAP instance by sending it aSIGKILL signal:

Resource fail-ure (CI in-stance)

kill -9 pid

Expected behavior: When running the next pending monitor op-eration, Pacemaker detects the failure and restarts the resource. Fail



count for this resource increases by 1, as shown with crm_mon-f.


Part III. Appendix

ASoftware DownloadsURLProduct

http://download.novell.com/Download?buildid=ut_49uTDXYc

SUSE Linux Enterprise Serverfor SAP Applications 11 SP1

http://download.novell.com/Download?buildid=x4q3cbksW7Q~

SUSE Linux Enterprise Server11 SP1

http://download.novell.com/Download?buildid=9xvsJDAsS04~

SUSE Linux Enterprise HighAvailability Extension

http://download.novell.com/Download?buildid=5qJ9eEidDzs~

SUSE Linux Enterprise 11 Sub-scription Management Tool

http://software.opensuse.org/search?q=ClusterTools2

ClusterTools2

&baseproject=SUSE%3ASLE-11%3ASP1&lang=en&include_home=true

http://service.sap.com/swdcNetWeaver 7.0 EHP1

included with NetWeaverMaxDB









http://software.opensuse.org/search?q=ClusterTools2&baseproject=SUSE%3ASLE-11%3ASP1&lang=en&include_home=true




http://service.sap.com/swdc

BNovell Products OnlineDocumentationDocumentation for Novell products is available at http://www.novell.com/documentationin HTML and PDF formats.

URLProduct

http://www.novell.com/documentation/smt11/

Subscription Management Tool


SUSE Linux Enterprise Server forSAP Applications 11


SUSE Linux Enterprise Server 11


SUSE Linux Enterprise HighAvailability Extension









CSAP NotesThe general installation of SAP on Linux is described in the SAP Note 171356 - SAPsoftware on Linux: Essential information. This SAP note also points to some SAP noteswith more detailed information about hardware platforms and Linux enterprise distri-butions. A good entry point for installing SAP on SUSE Linux Enterprise Server 11 isSAP Note 1310037. SAP Notes are available at the SAP Service Marketplace(http://service.sap.com [http://service.sap.com/]). You need an account toaccess this information.

TitleSAP Note

SUSE LINUX Enterprise Server 11 Installation notes1310037

Install SAP software on Linux: Essential information171356

Linux: Locale problems after updating glibc516716

SAP Management Console (SAP MC)1014480

SAP support terms and 3rd-party kernel drivers784391

J2EE engine installation on heterogenous architectures875322

Linux: Preparing SLES for SAP environments1275776

Download J2SE 1.4.2 for the x64 platform941595


TitleSAP Note

Linux: Supported Java versions on the x86_64 platform1172419

Java Cryptography Extention Jurisdiction Policy1240081

Release Restrictions for SAP EHP 1 for SAP NetWeaver1164532

SAP NetWeaver 7.0 (2004s) Documentation864172

Homogeneous system copy with MaxDB (SAP DB)129352

FAQ: SAP MaxDB Log area869267

SAP Web AS 6.40 SR1 Installation on UNIX: MaxDB790879

FAQ: SAP MaxDB Runtime Environment936058

FAQ: Database structure check (VERIFY)940420

SAP memory management for 64-bit Linux systems941735

FAQ: SAP MaxDB Runtime Environment936058

Unloading/loading MaxDB statistics data873286

Update required: Advantages for MaxDB on 64-bit1013441

Available MaxDB documentation767598

FAQ: SAP MaxDB/liceCache technology820824

SAP Web AS 6.40 SR1 ABAP Installation on UNIX785925

HA790879

ACC 7.1 PI/Adaptive Computing Controller1008828

Problems w/ sapstartsrv as of Release 7.00 & 6.40877795


TitleSAP Note

Backward porting of sapstartsrv for earlier releases995116

Oracle database 11g: Integration in SAP environment1398634

SAP Notes 101

DLinks to SUSE Linux EnterpriseServer, SAP, DatabasesSUSE Linux Enterprise Server:

http://www.novell.com/products/server/sap/matrix.html

http://www.novell.com/partner/isv/isvcatalog


http://support.novell.com/products/server/supported_packages


http://www.novell.com/coolsolutions/feature/17529.html

http://www.novell.com/partners/sap/

http://www.novell.com/docrep/2010/04/x2_Novell_SAP_Brochure_SinglePage%20Online.pdf

http://www.novell.com/docrep/2010/07/SLES_for_SAP_Applications_FAQ.pdf

http://www.novell.com/global/datacentertour/sv/pdf/Novell_SAP.pdf

http://www.novell.com/products/server/sap/matrix.html

http://www.novell.com/partner/isv/isvcatalog






http://www.novell.com/coolsolutions/feature/17529.html

http://www.novell.com/partners/sap/







SUSE Linux Enterprise Server:

http://www.novell.com/docrep/2009/09/RT_WP_Linux_Trends_in_SAP_DC_200908231_Final_English_en.doc


http://developer.novell.com/wiki/index.php/SAP_on_hasi_v2_Resource-SAPDatabase

http://www.novell.com/rc/index/index.jsp

http://www.novell.com/rc/docrepository/portal_skins/NovellSearch_public/SearchResults?keywords=migration&page=main&docstatus1=P&docstatus1=U&tab=1&x=0&y=0

http://www.novell.com/rc/docrepository/portal_skins/NovellSearch_public/SearchResults?keywords=sap&page=main&docstatus1=P&docstatus1=U&tab=1&x=0&y=0

http://www.novell.com/rc/docrepository/portal_skins/NovellSearch_public/SearchResults?id=NovellSearch_public&path=http%3A%2F%2Fwww.novell.com%2Frc%2Fdocrepository%2Fportal_skins%2FNovellSearch_public&page=advsearch&solution1=&solution2=&solution3=&keywords=high+availab&title=&description=&PublishDate1=&PublishDate2=&geography1=&x=0&y=0

http://www.novell.com/docrep/2009/05/SUSE-Linux-Enterprise-11_Technical-Presentation_en_en.odp

http://www.novell.com/docrep/2009/03/corebuild_playbook_v1.3_03262009_en.pdf

http://www.novell.com/docrep/2009/11/Enterprise_Linux_Servers_Solution_Presentation_f_110409_en.pdf

http://www.linux-ha.org/







http://www.novell.com/rc/index/index.jsp




















http://www.linux-ha.org/

SUSE Linux Enterprise Server:

http://www.openais.org/

http://www.clusterlabs.org/

https://raid.wiki.kernel.org/index.php

SAP:

https://www.sdn.sap.com/irj/sdn/nw-products

https://www.sdn.sap.com/irj/sdn/nw-70ehp1

http://www.sap.com/platform/netweaver/index.epx

http://help.sap.com/content/documentation/netweaver/docu_nw_70_design.htm#nw70ehp1

http://www.sap.com/linux

https://www.sdn.sap.com/irj/sdn/linux

https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/13603



http://sdn.sap.com/

http://service.sap.com/osdbmigration


http://www.sapinsideronline.com/archive.cfm?session

Links to SUSE Linux Enterprise Server, SAP, Databases 105

http://www.openais.org/

http://www.clusterlabs.org/

https://raid.wiki.kernel.org/index.php

https://www.sdn.sap.com/irj/sdn/nw-products

https://www.sdn.sap.com/irj/sdn/nw-70ehp1

http://www.sap.com/platform/netweaver/index.epx



http://www.sap.com/linux

https://www.sdn.sap.com/irj/sdn/linux





http://sdn.sap.com/

http://service.sap.com/osdbmigration


http://www.sapinsideronline.com/archive.cfm?session

SAP:

http://help.sap.com/saphelp_sm32/helpdata/de/c4/3a6bff505211d189550000e829fbbd/content.htm

IBM DB2:

http://www.sap.com/about/newsroom/press.epx?pressid=4517

http://www.sap.com/community/showdetail.epx?itemID=10875

http://www.sap.com/services/servsuptech/smp

https://websmp105.sap-ag.de/~form/sapnet?_SHORTKEY=01200252310000063662


http://www.redbooks.ibm.com/abstracts/sg246899.html

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp

http://www-01.ibm.com/support/docview.wss?rs=71&uid=swg27009474

http://www-01.ibm.com/software/sw-library/en_US/products/J441045L92289N69/#White%20papers

http://www.ibm.com/developerworks/wikis/display/im/SUSE+Linux+Enterprise+Server+%28SLES%29+10+-+DB2+9.5

http://www.ibm.com/developerworks/wikis/display/im/DB2+9.5+for+Linux+-+Supported+Environments

http://www.ibm.com/developerworks/data/library/techarticle/dm-0406qi/




http://www.sap.com/about/newsroom/press.epx?pressid=4517

http://www.sap.com/community/showdetail.epx?itemID=10875

http://www.sap.com/services/servsuptech/smp




http://www.redbooks.ibm.com/abstracts/sg246899.html













IBM DB2:

http://en.wikipedia.org/wiki/IBM_DB2

MaxDB:

http://dev.mysql.com/downloads/maxdb/7.6.00.html

https://www.sdn.sap.com/irj/sdn/maxdb?rid=/webcontent/uuid/7001df5e-549f-2a10-4487-f818b3c52031

http://maxdb.sap.com/doc/7_6/e6/0e9640dc522f28e10000000a1550b0/content.htm

http://maxdb.sap.com/currentdoc/62/aba9a0444311d5992400508b6b8b11/frameset.htm

http://maxdb.sap.com/currentdoc/76/b0ed16f56c1d41915c70c87bf44f04/frameset.htm

http://en.wikipedia.org/wiki/MaxDB

Oracle:

http://www.novell.com/products/server/oracle/documents.html

http://www.novell.com/products/server/oracle/matrix.html

http://ftp.novell.com/partners/oracle/docs/11gR2_sles11_install.pdf

http://www.sap.com/ecosystem/customers/directories/technology/oracle/index.epx

http://www.sdn.sap.com/irj/sdn/ora

Links to SUSE Linux Enterprise Server, SAP, Databases 107

http://en.wikipedia.org/wiki/IBM_DB2

http://dev.mysql.com/downloads/maxdb/7.6.00.html









http://en.wikipedia.org/wiki/MaxDB



http://www.novell.com/products/server/oracle/matrix.html





http://www.sdn.sap.com/irj/sdn/ora

Oracle:


http://www.oracle.com/technology/tech/linux/install/index.html

http://www.oracle-base.com/articles/linux/ArticlesLinux.php

http://www.oracle.com/global/ro/Colaterale/ROCS_Linux/ORACLEONLINUX.pdf

http://en.wikipedia.org/wiki/Oracle_Corporation









http://en.wikipedia.org/wiki/Oracle_Corporation

ESample CRM Configuration forSAP Simple Stack HighAvailability## configure the crm#crm(live)# configurecrm(live)configure# property no-quorum-policy="ignore"crm(live)configure# rsc_defaults resource-stickiness="200"crm(live)configure# commit## configure STONITH/ipmi (two IPMI interfaces)#crm(live)configure# primitive p_ipmi_hpn07 stonith:external/ipmi \

params hostname=hpn07 ipaddr=192.168.1.42 \userid=ipmi passwd=secret \op monitor interval=30s


crm(live)configure# commitcrm(live)configure# primitive p_ipmi_hpn08 stonith:external/ipmi \

params hostname=hpn08 ipaddr=192.168.1.43 \userid=ipmi passwd=secret \op monitor interval=30s


crm(live)configure# commit## configure one host based mirror#crm(live)configure# primitive p_raid1 ocf:heartbeat:Raid1 \

params raidconf="/etc/mdadm-cluster.conf" raiddev="/dev/md0" \op monitor interval="20s"

crm(live)configure# commit## configure DRBD IP addresses (THIS ONE ON SITE A ONLY)

#crm(live)configure# primitive p_ip_drbd ocf:heartbeat:IPaddr2 \params ip="172.16.12.193" cidr_netmask="24" \op monitor interval="10s"

crm(live)configure# commit## configure DRBD IP addresses (THIS ONE ON SITE B ONLY)#crm(live)configure# primitive p_ip_drbd ocf:heartbeat:IPaddr2 \params ip="172.16.12.194" cidr_netmask="24" \op monitor interval="10s"

crm(live)configure# commit## group all DRBD pre-requisites#crm(live)configure# group g_drbd_prereq p_raid1 p_ip_drbdcrm(live)configure# commit## configure DRBD device (THIS ONE ON SITE A ONLY)#crm(live)configure# primitive p_drbd_sap ocf:linbit:drbd \params drbd_resource="sap" \op monitor interval="30s" role="Master" \op monitor interval="20s" role="Slave"

crm(live)configure# ms ms_drbd_sap p_drbd_sap \meta notify="true" master-max=1 clone-max=1 \target-role="Master"



crm(live)configure# commit## configure DRBD device (THIS ONE ON SITE B ONLY)#crm(live)configure# primitive p_drbd_sap ocf:linbit:drbd \params drbd_resource="sap" \op monitor interval="30s" role="Master" \op monitor interval="20s" role="Slave"

crm(live)configure# ms ms_drbd_sap p_drbd_sap \meta notify="true" master-max=1 clone-max=1 \target-role="Slave"



crm(live)configure# commit## configure LVM#crm(live)configure# primitive p_lvm_sap ocf:heartbeat:LVM \params volgrpname="sapvg" \op start interval="0" timeout="30" \


op stop interval="0" timeout="30" \op monitor interval="10" timeout="30"

## configure file systems#crm(live)configure# primitive p_fs_sap_db ocf:heartbeat:Filesystem \

params device="/dev/sapvg/sapdb" directory="/sapdb/" fstype="ext3" \op start interval="0" timeout="60" \op stop interval="0" timeout="60" \op monitor interval="20" timeout="40"

crm(live)configure# primitive p_fs_sap_instance ocf:heartbeat:Filesystem \params device="/dev/sapvg/usrsap" directory="/usr/sap" fstype="ext3" \

op start interval="0" timeout="60" \op stop interval="0" timeout="60" \op monitor interval="20" timeout="40"

crm(live)configure# primitive p_fs_sapmnt ocf:heartbeat:Filesystem \params device="/dev/sapvg/sapmnt" directory="/sapmnt" fstype="ext3" \

op monitor interval="20s" \op monitor interval="0" timeout="40s" \op start interval="0" timeout="60s" \op stop interval="0" timeout="40"

## configure IP addresses#crm(live)configure# primitive p_ip_sapdb ocf:heartbeat:IPaddr2 \

params ip="172.16.12.190" cidr_netmask="24" \op monitor interval="10s"

crm(live)configure# primitive p_ip_ascs ocf:heartbeat:IPaddr2 \params ip="172.16.12.191" cidr_netmask="24" \op monitor interval="10s"

crm(live)configure# primitive p_ip_ci ocf:heartbeat:IPaddr2 \params ip="172.16.12.192" cidr_netmask="24" \op monitor interval="10s"

## configure SAP database#crm(live)configure# primitive p_sapdatabase ocf:heartbeat:SAPDatabase \

params DBTYPE="ADA" SID="DRB" \DIR_EXECUTABLE="/usr/sap/DRB/ASC00/exe" \AUTOMATIC_RECOVER="true" \op monitor interval="20s" timeout="20s" \op start interval="0" timeout="360" \op stop interval="0" timeout="360"

## configure SAP instances#crm(live)configure# primitive p_sap_ascs ocf:heartbeat:SAPInstance \

params InstanceName="DRB_ASC00_sapdrbas"AUTOMATIC_RECOVER="true" \op monitor interval="120s" timeout="60s" \op start interval="0" timeout="180s" \op stop interval="0" timeout="240s"

crm(live)configure# primitive p_sap_ci ocf:heartbeat:SAPInstance \

Sample CRM Configuration for SAP Simple Stack High Availability 111

params InstanceName="DRB_DVEBMGS01_sapdrbci"AUTOMATIC_RECOVER="true" \op start interval="0" timeout="180" \op stop interval="0" timeout="240" \op monitor interval="120s" timeout="60s"

## group sap and pre-req#crm(live)configure#group g_sap p_lvm_sap p_fs_sapmnt \p_fs_sap_db p_ip_sapdb p_sapdatabase \p_fs_sap_instance \p_ip_ascs p_sap_ascs \p_ip_ci p_sap_ci

## configure constraints#crm(live)configure# colocation c_sap_on_drbd \inf: g_sap ms_drbd_sap:Master

crm(live)configure# order o_drbd_before_sap \inf: ms_drbd_sap:promote g_sap:start



FLicensesGNU Free Documentation LicenseVersion 1.2, November 2002

Copyright (C) 2000,2001,2002 Free Software Foundation, Inc. 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

PREAMBLEThe purpose of this License is to make a manual, textbook, or other functional and useful document “free” in the sense of freedom: to assure everyonethe effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preservesfor the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of “copyleft”, which means that derivative works of the document must themselves be free in the same sense. It complementsthe GNU General Public License, which is a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program shouldcome with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for anytextual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purposeis instruction or reference.

APPLICABILITY AND DEFINITIONSThis License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed underthe terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions statedherein. The “Document”, below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as “you”. You accept thelicense if you copy, modify or distribute the work in a way requiring permission under copyright law.

A “Modified Version” of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modificationsand/or translated into another language.

A “Secondary Section” is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers orauthors of the Document to the Document’s overall subject (or to related matters) and contains nothing that could fall directly within that overall subject.(Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matterof historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The “Invariant Sections” are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that theDocument is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant.The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The “Cover Texts” are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Documentis released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A “Transparent” copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public,that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for

drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formatssuitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged tothwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount oftext. A copy that is not “Transparent” is called “Opaque”.

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XMLusing a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparentimage formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors,SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF producedby some word processors for output purposes only.

The “Title Page” means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requiresto appear in the title page. For works in formats which do not have any title page as such, “Title Page” means the text near the most prominent appearanceof the work’s title, preceding the beginning of the body of the text.

A section “Entitled XYZ” means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following textthat translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as “Acknowledgements”, “Dedications”,“Endorsements”, or “History”.) To “Preserve the Title” of such a section when you modify the Document means that it remains a section “EntitledXYZ” according to this definition.

The Document may includeWarranty Disclaimers next to the notice which states that this License applies to the Document. TheseWarranty Disclaimersare considered to be included by reference in this License, but only as regards disclaimingwarranties: any other implication that theseWarranty Disclaimersmay have is void and has no effect on the meaning of this License.

VERBATIM COPYINGYou may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices,and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever tothose of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute.However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditionsin section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

COPYING IN QUANTITYIf you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document’slicense notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts onthe front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. Thefront cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition.Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatimcopying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover,and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copyalong with each Opaque copy, or state in or with each Opaque copy a computer-network location fromwhich the general network-using public has accessto download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option,you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thusaccessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) ofthat edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them achance to provide you with an updated version of the Document.

MODIFICATIONSYou may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the ModifiedVersion under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of theModified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, ifthere were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that versiongives permission.

B. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, togetherwith at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this require-ment.

C. State on the Title page the name of the publisher of the Modified Version, as the publisher.


D. Preserve all the copyright notices of the Document.

E. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.

F. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of thisLicense, in the form shown in the Addendum below.

G. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document’s license notice.

H. Include an unaltered copy of this License.

I. Preserve the section Entitled “History”, Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of theModified Version as given on the Title Page. If there is no section Entitled “History” in the Document, create one stating the title, year, authors, andpublisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.

J. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the networklocations given in the Document for previous versions it was based on. These may be placed in the “History” section. You may omit a network locationfor a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.

K. For any section Entitled “Acknowledgements” or “Dedications”, Preserve the Title of the section, and preserve in the section all the substance andtone of each of the contributor acknowledgements and/or dedications given therein.

L. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not consideredpart of the section titles.

M. Delete any section Entitled “Endorsements”. Such a section may not be included in the Modified Version.

N. Do not retitle any existing section to be Entitled “Endorsements” or to conflict in title with any Invariant Section.

O. Preserve any Warranty Disclaimers.

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from theDocument, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in theModified Version’s license notice. These titles must be distinct from any other section titles.

Youmay add a section Entitled “Endorsements”, provided it contains nothing but endorsements of yourModified Version by various parties--for example,statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list ofCover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangementsmade by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by thesame entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisherthat added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsementof any Modified Version.

COMBINING DOCUMENTSYou may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions,provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as InvariantSections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If thereare multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, inparentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titlesin the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled “History” in the various original documents, forming one section Entitled “History”; likewisecombine any sections Entitled “Acknowledgements”, and any sections Entitled “Dedications”. You must delete all sections Entitled “Endorsements”.

COLLECTIONS OF DOCUMENTSYoumaymake a collection consisting of the Document and other documents released under this License, and replace the individual copies of this Licensein the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copyingof each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this Licenseinto the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

Licenses 115

AGGREGATION WITH INDEPENDENT WORKSA compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distributionmedium, is called an “aggregate” if the copyright resulting from the compilation is not used to limit the legal rights of the compilation’s users beyondwhat the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate whichare not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate,the Document’s Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if theDocument is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

TRANSLATIONTranslation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing InvariantSections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sectionsin addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document,and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices anddisclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version willprevail.

If a section in the Document is Entitled “Acknowledgements”, “Dedications”, or “History”, the requirement (section 4) to Preserve its Title (section 1)will typically require changing the actual title.

TERMINATIONYou may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy,modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have receivedcopies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

FUTURE REVISIONS OF THIS LICENSEThe Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions willbe similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License “orany later version” applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that hasbeen published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may chooseany version ever published (not as a draft) by the Free Software Foundation.

ADDENDUM: How to use this License for your documentsTo use this License in a document you have written, include a copy of the License in the document and put the following copyright and license noticesjust after the title page:

Copyright (c) YEAR YOUR NAME.Permission is granted to copy, distribute and/or modify this documentunder the terms of the GNU Free Documentation License, Version 1.2or any later version published by the Free Software Foundation;with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.A copy of the license is included in the section entitled “GNUFree Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “with...Texts.” line with this:

with the Invariant Sections being LIST THEIR TITLES, with theFront-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free softwarelicense, such as the GNU General Public License, to permit their use in free software.

GNU General Public LicenseVersion 2, June 1991

Copyright (C) 1989, 1991 Free Software Foundation, Inc. 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA


Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

PreambleThe licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intendedto guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies tomost of the Free Software Foundation’s software and to any other program whose authors commit to using it. (Some other Free Software Foundationsoftware is covered by the GNU Library General Public License instead.) You can apply it to your programs, too.

Whenwe speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedomto distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can changethe software or use pieces of it in new free programs; and that you know you can do these things.

To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictionstranslate to certain responsibilities for you if you distribute copies of the software, or if you modify it.

For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You mustmake sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights.

We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distributeand/or modify the software.

Also, for each author’s protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If thesoftware is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introducedby others will not reflect on the original authors’ reputations.

Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individu-ally obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone’sfree use or not licensed at all.

The precise terms and conditions for copying, distribution and modification follow.

GNUGENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTIONANDMODIFICATION0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under theterms of this General Public License. The “Program”, below, refers to any such program or work, and a “work based on the Program” means either theProgram or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modificationsand/or translated into another language. (Hereinafter, translation is included without limitation in the term “modification”.) Each licensee is addressedas “you”.

Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Programis not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having beenmade by running the Program). Whether that is true depends on what the Program does.

1. You may copy and distribute verbatim copies of the Program’s source code as you receive it, in any medium, provided that you conspicuously andappropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License andto the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program.

You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee.

2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute suchmodifications or work under the terms of Section 1 above, provided that you also meet all of these conditions:

a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change.

b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to belicensed as a whole at no charge to all third parties under the terms of this License.

c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in themost ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, sayingthat you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License.(Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required toprint an announcement.)

These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonablyconsidered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them asseparate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole mustbe on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of whowrote it.

Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right tocontrol the distribution of derivative or collective works based on the Program.

Licenses 117

In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of astorage or distribution medium does not bring the other work under the scope of this License.

3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1and 2 above provided that you also do one of the following:

a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 aboveon a medium customarily used for software interchange; or,

b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performingsource distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 aboveon a medium customarily used for software interchange; or,

c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for non-commercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.)

The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code meansall the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installationof the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either sourceor binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that componentitself accompanies the executable.

If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy thesource code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along withthe object code.

4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy,modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have receivedcopies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute theProgram or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing theProgram (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, dis-tributing or modifying the Program or works based on it.

6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensorto copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients’ exerciseof the rights granted herein. You are not responsible for enforcing compliance by third parties to this License.

7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions areimposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditionsof this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then asa consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Programby all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain en-tirely from distribution of the Program.

If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and thesection as a whole is intended to apply in other circumstances.

It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; thissection has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Manypeople have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of thatsystem; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose thatchoice.

This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License.

8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyrightholder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distributionis permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License.

9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions willbe similar in spirit to the present version, but may differ in detail to address new problems or concerns.

Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and “any laterversion”, you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foun-dation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation.

10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask forpermission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptionsfor this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharingand reuse of software generally.


NO WARRANTY

11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENTPERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHERPARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING,BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THEENTIRERISKASTOTHEQUALITYANDPERFORMANCEOFTHEPROGRAMISWITHYOU.SHOULDTHEPROGRAMPROVEDEFECTIVE,YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, ORANY OTHER PARTY WHOMAYMODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FORDAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE ORINABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATEORLOSSES SUSTAINEDBYYOUORTHIRDPARTIESORAFAILUREOFTHEPROGRAMTOOPERATEWITHANYOTHERPROGRAMS),EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

END OF TERMS AND CONDITIONS

How to Apply These Terms to Your New ProgramsIf you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free softwarewhich everyone can redistribute and change under these terms.

To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusionof warranty; and each file should have at least the “copyright” line and a pointer to where the full notice is found.

one line to give the program’s name and an idea of what it does.Copyright (C) yyyy name of author

This program is free software; you can redistribute it and/ormodify it under the terms of the GNU General Public Licenseas published by the Free Software Foundation; either version 2of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,but WITHOUT ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See theGNU General Public License for more details.

You should have received a copy of the GNU General Public Licensealong with this program; if not, write to the Free SoftwareFoundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

Also add information on how to contact you by electronic and paper mail.

If the program is interactive, make it output a short notice like this when it starts in an interactive mode:

Gnomovision version 69, Copyright (C) year name of authorGnomovision comes with ABSOLUTELY NO WARRANTY; for detailstype `show w’. This is free software, and you are welcometo redistribute it under certain conditions; type `show c’for details.

The hypothetical commands `show w’ and `show c’ should show the appropriate parts of the General Public License. Of course, the commands you usemay be called something other than `show w’ and `show c’; they could even be mouse-clicks or menu items--whatever suits your program.

You should also get your employer (if you work as a programmer) or your school, if any, to sign a “copyright disclaimer” for the program, if necessary.Here is a sample; alter the names:

Yoyodyne, Inc., hereby disclaims all copyright

Licenses 119

interest in the program `Gnomovision’(which makes passes at compilers) writtenby James Hacker.

signature of Ty Coon, 1 April 1989Ty Coon, President of Vice

This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you mayconsider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General PublicLicense [http://www.fsf.org/licenses/lgpl.html] instead of this License.


http://www.fsf.org/licenses/lgpl.html

Terminologyactive/active, active/passive

A concept of how services are running on nodes. An active-passive scenario meansthat one or more services are running on the active node and the passive node waitsfor the active node to fail. Active-active means that each node is active and passiveat the same time.

clusterA high-performance cluster is a group of computers (real or virtual) sharing theapplication load in order to achieve faster results. A high-availability cluster is de-signed primarily to secure the highest possible availability of services.

cluster information base (CIB)A representation of the whole cluster configuration and status (node membership,resources, constraints, etc.) written in XML and residing in memory. A master CIBis kept andmaintained on the designated coordinator (DC) (page 122) and replicatedto the other nodes.

cluster partitionWhenever communication fails between one or more nodes and the rest of thecluster, a cluster partition occurs. The nodes of a cluster are split in partitions butstill active. They can only communicate with nodes in the same partition and areunaware of the separated nodes. As the loss of the nodes on the other partitioncannot be confirmed, a split brain scenario develops (see also split brain (page 125)).

cluster resource manager (CRM)Themainmanagement entity responsible for coordinating all non-local interactions.Each node of the cluster has its own CRM, but the one running on the DC is theone elected to relay decisions to the other non-local CRMs and process their input.A CRM interacts with a number of components: local resource managers, both onits own node and on the other nodes, non-local CRMs, administrative commands,the fencing functionality, and the membership layer.

consensus cluster membership (CCM)The CCM determines which nodes make up the cluster and shares this informationacross the cluster. Any new addition and any loss of nodes or quorum is deliveredby the CCM. A CCM module runs on each node of the cluster.

designated coordinator (DC)The “master” node. This node is where the master copy of the CIB is kept. Allother nodes get their configuration and resource allocation information from thecurrent DC. The DC is elected from all nodes in the cluster after a membershipchange.

distributed lock manager (DLM)DLM coordinates disk access for clustered file systems and administers file lockingto increase performance and availability.

distributed replicated block device (DRBD)DRBD is a block device designed for building high availability clusters. The wholeblock device is mirrored via a dedicated network and is seen as a network RAID-1.

failoverOccurs when a resource or node fails on one machine and the affected resourcesare started on another node.

fencingDescribes the concept of preventing access to a shared resource by isolated orfailing cluster members. Should a cluster node fail, it will be shut down or reset toprevent it from causing trouble. This way, resources are locked out of a node whosestatus is uncertain.

Heartbeat resource agentHeartbeat resource agents were widely used with Heartbeat version 1. Their use isdeprecated, but still supported in version 2. A Heartbeat resource agent can performstart, stop, and status operations and resides under /etc/ha.d/resource.d or /etc/init.d. For more information about Heartbeat resourceagents, refer tohttp://www.linux-ha.org/HeartbeatResourceAgent(see also OCF resource agent (page 124)).

high availabilityHigh availability is a system design approach and associated service implementationthat ensures a prearranged level of operational performance will be met during acontractual measurement period.


http://www.linux-ha.org/HeartbeatResourceAgent

Availability is a key aspect of service quality. Availability is usually calculatedbased on a model involving the Availability Ratio and techniques such as FaultTree Analysis.

See also: http://en.wikipedia.org/wiki/High_availability/http://www.itlibrary.org/index.php?page=Availability_Management

SBDStonith Block Device

In an environment where all nodes have access to shared storage, a small partitionis used for disk-based fencing.

SFEXShared Disk File Exclusiveness.

SFEX provides storage protection over SAN.

local resource manager (LRM)The local resource manager (LRM) is responsible for performing operations onresources. It uses the resource agent scripts to carry out these operations. The LRMis “dumb” in that it does not know of any policy. It needs the DC to tell it what todo.

LSB resource agentLSB resource agents are standard LSB init scripts. LSB init scripts are not limitedto use in a high availability context. Any LSB-compliant Linux system uses LSBinit scripts to control services. Any LSB resource agent supports the options start,stop, restart, status and force-reload and may optionally providetry-restart and reload as well. LSB resource agents are located in /etc/init.d. Find more information about LSB resource agents and the actual speci-fication at http://www.linux-ha.org/LSBResourceAgent andhttp://www.linux-foundation.org/spec/refspecs/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html(see also OCF resource agent (page 124) and Heartbeat resource agent (page 122)).

nodeAny computer (real or virtual) that is a member of a cluster and invisible to theuser.

Terminology 123

http://en.wikipedia.org/wiki/High_availability/

http://www.itlibrary.org/index.php?page=Availability_Management

http://www.itlibrary.org/index.php?page=Availability_Management

http://www.linux-ha.org/LSBResourceAgent

http://www.linux-foundation.org/spec/refspecs/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html

http://www.linux-foundation.org/spec/refspecs/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html

policy engine (PE)The policy engine computes the actions that need to be taken to implement policychanges in the CIB. This information is then passed on to the transaction engine,which in turn implements the policy changes in the cluster setup. The PE alwaysruns on the DC.

OCF resource agentOCF resource agents are similar to LSB resource agents (init scripts). Any OCFresource agent must support start, stop, and status (sometimes calledmonitor) options. Additionally, they support a metadata option that returnsthe description of the resource agent type in XML. Additional options may besupported, but are not mandatory. OCF resource agents reside in /usr/lib/ocf/resource.d/<provider>. Findmore information about OCF resourceagents and a draft of the specification at http://www.linux-ha.org/OCFResourceAgent andhttp://www.opencf.org/cgi-bin/viewcvs.cgi/specs/ra/resource-agent-api.txt?rev=HEAD (see alsoHeartbeat resource agent (page 122)).

quorumIn a cluster, a cluster partition is defined to have quorum (is “quorate”) if it has themajority of nodes (or votes). Quorum distinguishes exactly one partition. It is partof the algorithm to prevent several disconnected partitions or nodes from proceedingand causing data and service corruption (split brain). Quorum is a prerequisite forfencing, which then ensures that quorum is indeed unique.

resourceAny type of service or application that is known to Heartbeat. Examples includean IP address, a file system, or a database.

resource agent (RA)A resource agent (RA) is a script acting as a proxy to manage a resource. Thereare three different kinds of resource agents: OCF (Open Cluster Framework) re-source agents, LSB resource agents (Standard LSB init scripts), and Heartbeat re-source agents (Heartbeat v1 resources).

Single Point of Failure (SPOF)A single point of failure (SPOF) is any component of a cluster that, should it fail,triggers the failure of the entire cluster.


http://www.linux-ha.org/OCFResourceAgent

http://www.linux-ha.org/OCFResourceAgent

http://www.opencf.org/cgi-bin/viewcvs.cgi/specs/ra/resource-agent-api.txt?rev=HEAD

http://www.opencf.org/cgi-bin/viewcvs.cgi/specs/ra/resource-agent-api.txt?rev=HEAD

split brainA scenario in which the cluster nodes are divided into two or more groups that donot know of each other (either through a software or hardware failure). STONITHprevents a split brain situation from badly affecting the entire cluster. Also knownas a “partitioned cluster” scenario.

The term split brain is also used in DRBD but means that the two nodes containdifferent data.

STONITHThe acronym for “Shoot the other node in the head”, which refers to the fencingmechanism that shuts down a misbehaving node to prevent it from causing troublein a cluster.

transition engine (TE)The transition engine (TE) receives policy directives from the PE and carries themout. The TE always runs on the DC. From there, it instructs the local resourcemanagers on the others nodes which actions to take.

Terminology 125

sap suse linux

Documents