Best practices for SAP ERP using HP BladeSystem c-Class blades

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Solution configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Server configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 3 SAP solution-based servers . . . . . . . . . . . . . . . . . . . . . 3 Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Storage management . . . . . . . . . . . . . . . . . . . . . . . . 3

Storage configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Storage array . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Array configuration and virtual disk layout . . . . . . . . . . . . . . . 4 SAN infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . 6

Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 SAP ERP workload analysis . . . . . . . . . . . . . . . . . . . . . . . 7 Response time components . . . . . . . . . . . . . . . . . . . . . . . 8 Test procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 SAP and server tuning . . . . . . . . . . . . . . . . . . . . . . . . . 9

Tuning SAP memory parameters . . . . . . . . . . . . . . . . . . . 9 Tuning SAP buffer parameters . . . . . . . . . . . . . . . . . . . . 9 Managing the total number of users . . . . . . . . . . . . . . . . 12 Managing the number of total work processes . . . . . . . . . . . . 13Server sizing to accommodate more than 900 concurrent users . . . . 15Using the database server as an SAP application server . . . . . . . 15

Storage tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Impact of customer data size . . . . . . . . . . . . . . . . . . . 18 Managing the number of disk drives in the SAPDATA disk group . . . . 19Managing the number of EVA disk groups . . . . . . . . . . . . . . 20Selecting the Vraid type for SAPDATA virtual disks . . . . . . . . . . 21Selecting the Vraid type of virtual disk for the transaction log . . . . . . 22Storage sizing to accommodate more than 900 concurrent users . . . . 23

Best practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 SAP administrators . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Server administrators . . . . . . . . . . . . . . . . . . . . . . . . . 24 Storage administrators . . . . . . . . . . . . . . . . . . . . . . . . 24 Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 We value your feedback . . . . . . . . . . . . . . . . . . . . . . . 26

Appendix A Bill of Materials . . . . . . . . . . . . . . . . . . . . . . . . 27 Appendix B References . . . . . . . . . . . . . . . . . . . . . . . . . . 28 For more information . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

HP technical references . . . . . . . . . . . . . . . . . . . . . . . . 29

Overview

A significant change is taking place in the SAP install base. While many SAP customers continue

to run older R/3-based versions, migrations to the newer NetWeaver-based versions of SAP are

accelerating. Migrations to these newer versions of SAP software require both conversions of the

application environment, as well as the need to evaluate, upgrade, and refresh the servers and

storage infrastructure used to deploy the SAP system landscape.

Once this new hardware infrastructure is chosen, a series of considerations are required to

effectively configure and deploy the new SAP NetWeaver software onto the new infrastructure. This document provides medium-scale, enterprise-class SAP customers running Windows and

MS-SQL with a comprehensive set of best practices to appropriately configure and productively

deploy SAP NetWeaver and a typical ERP application onto HP c-Class blade servers and a

SAN-based infrastructure.

The following fundamental deployment and operational issues are addressed:

• The best way to configure the SAP landscape across the server blades, optimizing both

performance and availability

• The effects of scaling the environment when a blade is added as an additional application server

• The appropriate configuration for the HP EVA disk array, including:

– Choosing the appropriate number of disks

– Configuring the disks into the appropriate number of EVA disk groups for both the SAP

application components and the database

– Comparing Vraid 1 and Vraid 5

– Varying the size of the database

• The benefits of properly configuring the number and types of SAP work processes

• The benefits of properly tuning SAP shared memory, with emphasis on buffer parameters

Test results of methods and best practices are described in the following sections. This information

is intended to facilitate the productive planning, timely deployment, and seamless management of a fully-operational SAP NetWeaver landscape on HP c-Class blade servers and SAN-based

infrastructure. Knowledge of this information will ensure:

• Proper deployment of the appropriate server and storage infrastructure

• Effective deployment of the Windows operating system, MS-SQL database, and SAP system

environment

• Accelerated deployment, reduced risk, and minimized total costs

By implementing the best practices provided in this white paper, HP:

• Reduced the baseline response time of the SAP workload by 67% with basic buffer tuning

• Improved I/O response time by 60% by properly sizing the number of disks

• Reduced response time by up to 40% when the SAP work processes were properly configured

In all cases where the number of disks was properly sized, the more economical and space

efficient Vraid 5 deployment provided very satisfactory performance for the database.

2

Solution configuration

Server configuration

Testing examined an SAP ERP 2005 system, which consists of a central instance (CI), a database

instance, and a dedicated dialog instance. Each instance is installed on an identically-configured

HP ProLiant BL460c Server Blade, with a clustered SAP CI and the Microsoft SQL Server 2005 Server Pack 2 (SP2) database. Figure 1 displays an SAP production system, an SAP

development system, or an SAP quality assurance system. The server configuration elements

used during testing are described in following sections. Refer to Table 1 for a complete listing

of the servers used during testing.

SAP solution-based servers

The hardware for the SAP solution-based servers includes three HP ProLiant BL460c Blade

servers running Microsoft Windows Server 2003 R2 Enterprise x64 Edition SP2. Each server has

two 2.66-GHz Intel Xeon X5355 quad-core processors and 32 GB of memory. The SAP ERP

application utilizes eight total CPU cores per server. Each server is equipped with a dual-port Emulex 4 Gb PCI-E-to-Fibre Channel host bus adapter (HBA) utilizing a Microsoft Storport driver. An HP Multipath I/O Device Specific Module 2.01.01 for Enterprise Virtual Array (HP MPIO DSM

2.01.01 for EVA) is used for managing the path control from the HBAs to the EVA8100.

The server configuration size is based on methods used by the HP SAP Competency Center for the accommodation of 18,000 SAP Application Performance Standards (SAPS) by the entire SAP

system. In SAP Quick Size terms, an 18,000 SAPS system is a medium-sized SAP system.

Cluster

Two of the three ProLiant BL460c Blade servers running Microsoft Windows Server 2003 R2

Enterprise x64 Edition SP2 are clustered using Microsoft Cluster Server. A cluster group is created

using the database server and the CI server in the SAP system landscape. The testing is executed

with the SQL Server 2005 database and the SAP CI running on different servers. Clustering the

two servers provides a level of availability in case a server fails.

Storage management

Storage management is provided by a ProLiant BL460c Blade server running Microsoft Windows

Server 2003 R2 Enterprise Edition SP2. The server utilizes HP StorageWorks Command View

EVA 7.0, with the EVAPerf Enterprise monitoring tool, for collecting EVA performance information.

Table 1. Component list—servers

Server type Server use

ProLiant BL460c SAP central instance cluster

ProLiant BL460c SQL Server 2005 database cluster

ProLiant BL460c SAP dialog instance

ProLiant BL460c Storage management

ProLiant BL460c SAP solution manager

ProLiant BL460c Domain controller

3

Figure 1. Server configuration diagram, BladeSystem C7000 Enclosure

Storage configuration

Storage array

An HP StorageWorks EVA8100 2C12D storage array running XCS 6110 firmware is fully

populated with 168 146 GB 15K RPM Fibre Channel disk drives. This array stores the SAP

and the SQL Server 2005 executables, the SAPDATA files (where all the SAP ERP database

files are located), and the database log files.

Array configuration and virtual disk layout

Three initial configurations of EVA disk groups are tested to provide performance comparisons.

The SAPDATA files are placed in a disk group containing at least 32 disks to ensure that there

is enough capacity to accommodate a minimum 1 TB of SAPDATA and the supporting SQL

Server 2005 and SAP files.

In the first configuration, 32 disk drives are used, and all the virtual disks are placed in a single

disk group. This configuration provides the easiest method of administration because of the

simplicity of its design.

In the second configuration, two disk groups are used with a total of 40 disk drives:

• One disk group for the SQL Server 2005 transaction logs, consisting of eight disks

• One disk group for the SAPDATA files, consisting of 32 disks

In this configuration, the sequential I/O log files are separated from the random I/O SAPDATA

files. Availability is improved, because database data files and log files are separated into distinct

disk groups. If the disk group with the SAPDATA files becomes unavailable, the transaction logs

are safe. If the other disk group becomes unavailable, the SAPDATA files are safe.

4

In the third configuration, 40 disk drives are used, and all virtual disks are placed in a single disk

group. This configuration has the same benefits as the first configuration, but provides the SAP

system with more disk resources. More disks in a disk group provide the virtual disks in that disk

group with a greater capacity to handle I/O at an acceptable latency.

The fourth configuration, which is not illustrated, is identical in layout to the second configuration, except it uses a total of 48 disk drives:

• The disk group containing the SAPDATA files, consisting of 40 disks

• The disk group containing the database transaction logs, consisting of eight disks

Best practice

Use disk groups are populated with disk drives in multiples of eight. This is an EVA best practice that ensures thatinternal administration of the EVA disk groups is optimized.

We use Vraid 5 for SAPDATA files because fewer disks are required to store the same amount of data than Vraid 1. For many workloads, Vraid 5 performs similarly to Vraid 1.

All of the configurations, the SAP executables, the SQL Server 2005 executables, the cluster quorum, and the Microsoft Distributed Transaction Coordinator (MSDTC), are placed on separate

virtual disks in the disk group containing the SAPDATA files.

To allow the storage activity to be balanced evenly between the EVA controllers, two virtual disks

are used for the SAPDATA files. Each SAPDATA virtual disk contains two SAPDATA files.

Figure 2. EVA8100 configuration diagram

Note

In Figure 2, the boxes with dotted-lined borders represent EVA virtual disks. The virtual disks for SAP executables, SQLServer 2005 executables, cluster quorum, and MSDTC are not shown.

5

SAN infrastructure

Two unzoned HP BladeSystem 4 Gb SAN switches running v5.3.0d firmware with 4 Gb optical

transceivers are used to create two independent 4 Gb fabrics.

Table 2. Component list—storage

Storage components Component type

Host bus adapters (HBA) (6) Emulex LPe1105 FC Dual Channel 4 Gb, PCI-E-to-Fibre Channel

SAN switches (2) Brocade 4 Gb SAN Switch for c-Class BladeSystem

Storage array (1) EVA8100 Storage Array

6

Testing

Objectives

This document presents best practices for storage administrators, server administrators, and SAP

administrators, using SAP ERP 2005 with the HP StorageWorks EVA8100 storage array, HP

ProLiant c-Class Blade servers, and the SQL Server 2005 database as part of an SAP system.

This document also provides a performance proof point for an SAP ERP workload with a

combination of HP storage devices and servers. With this information, it is easy to determine if a

given SAP ERP workload provides an acceptable user response time.

Tuning and sizing the servers and storage devices to handle the workload is a vital step in

providing a solution that meets performance expectations. This goal is accomplished by varying

server and storage configurations and tuning the parameters of SAP to optimize performance

for the SAP ERP workload.

SAP ERP workload analysis

This section examines how storage and server tuning influences an SAP ERP workload’s

performance, including:

• Characteristics of the SAP ERP Sales and Distribution workload

• I/O characteristics of the workload from the servers to the virtual disk on the storage array

A virtual disk is a logical unit of storage on the EVA8100 storage array. The size of the virtual disk is specified. However, the physical storage space for a virtual disk can exist on a number of different disk drives in a disk group managed by the EVA8100.

For the SAP ERP workload, the virtual disks that must be managed are the SAPDATA (all of the

SAP ERP database files are located here), and the database transaction log virtual disks. Almost 100% of all storage activity involving I/Os per second (IOPS) and workload data throughput occurs

on the SAPDATA and the transaction log virtual disks. Storage activity to the transaction log virtual disk will always be 100% write, with a typical write size of 16 KB per I/O.

The SAP ERP workload storage activity for the SAPDATA virtual disks follows:

• The ratio of read-to-write host IOPS is 4:1.

• The ratio of read-to-write host throughput is 4:1.

• The read size is 8 KB per I/O.

• The write size is 8−64 KB per I/O, with the vast majority of writes being 8 KB.

• This SAP ERP workload for SAPDATA virtual disks is read-I/O dominated.

The SAP ERP Sales and Distribution workload has several characteristics, such as creating order,delivery, and invoice records for your business customers.

To execute this workload, follow these steps:

1. Create a customer order with SAP transaction VA01.

2. Create a delivery date for the order with SAP transaction VL01N.

3. Display the order with SAP transaction VA03.

7

4. Update the delivery date for the order with SAP transaction VL02N.

5. List the orders for the customer with SAP transaction VA05.

6. Create an invoice for the order with SAP transaction VF01.

Response time components

The primary goals of this testing are to:

• Understand the impact of various server, infrastructure, and storage configurations

• Optimize total response time for a given hardware configuration with a real-world SAP ERP

user load and configuration

Total response time is defined as the time it takes the SAP system to process a user request. Optimized response time allows an SAP system to process more transactions during a given

time-period. This section describes the components of total response time.

The primary components of total response time are the individual response times for SAP work

processes that include: Dialog, Update1, and Update2.

Dialog. The Dialog work process responds to an active user request to execute dialog steps from

the SAP user interface. For instance, think of a person sitting at terminal entering information on

an SAP user screen and then performing an action that processes the data.

Update1. The Update1 work process is a time-critical update to the SAP database. Think of this

process as taking the information that the user entered in a Dialog step, and then moving that

information from the user screen into database tables.

Update2. The Update2 work process is a low-priority database update. It is used primarily in

statistics-related database transactions.

We can break down the components of total response even further. The Dialog, Update1, and

Update2 work processes all have the following response-time subcomponents:

• Wait time—The time it takes for SAP to find a free work process

• CPU time—The amount of time that the work process has active control of the CPUs

• Database request time—The time needed to access and perform operations on the database

• Processing time—The total time needed to execute an SAP program

Test procedures

To characterize configuration performance using this SAP ERP workload, real-time user loads are

run, and performance information is captured for SAP, the servers, and the storage devices.

Testing is conducted with the following characteristics of this SAP ERP workload:

• 850 to 1,800 concurrent users are simulated with scripts.

• One or two application servers are utilized simultaneously for testing.

• One SAP instance (also referred to as an application server) exists on each physical server.

The total response time for various numbers of concurrent users is directly measured via a script.

8

SAP and server tuning

The first step to improve and optimize the SAP ERP workload is to determine the SAP parameters

and settings that affect the workload’s performance. This section describes the most important findings regarding the tuning of SAP and the SAP application servers.

Tuning SAP memory parameters

Unlike operating systems, which define virtual memory as memory available to the OS other

than physical RAM, SAP defines virtual memory as the total pool of memory available to SAP, regardless of its underlying source. The size of SAP virtual memory is equal to the size of all the

physical memory, plus the operating system page file space (on local disk), allocated to the SAP

instance by the SAP server’s operating system.

SAP virtual memory is further broken down into two sections: shared memory and local memory

(see Figure 3). Shared memory is used by all the work processes of an SAP instance. It is

allocated at instance startup. Local memory is allocated for specific work processes. It is not shared by all work processes—it is dedicated and allocated exclusively to each individual work

process for that work process’s sole use.

SAP Note 88416, entitled Zero administration memory management as of 4.0A/Windows, details

many appropriate settings for SAP memory parameters on a 64-bit Windows server.

Tuning SAP buffer parameters

SAP buffers are part of the SAP shared memory pool that contain all users’ global objects and

SAP work processes. Figure 3 provides an overview of the SAP memory architecture.

Figure 3. Overview of SAP memory structure

All SAP buffers have associated parameters that require tuning to achieve the best response time. By using SAP transaction ST02, it is easy to determine which parameters have an impact on the

workload’s performance. Transaction ST02 displays many SAP buffers characteristics.

Figure 4 shows a screenshot of a transaction ST02 window with the SAP buffer parameters

set correctly for best performance.

9

Figure 4. Screenshot of SAP transaction ST02 with SAP buffers tuned correctly for no swaps

Transaction ST02 is run to verify there are no non-zero entries in the Swaps column of data. A

zero entry indicates an SAP instance is not paging to disk, which is the optimal situation because

transfer speeds between SAP and disk are much slower than those between SAP and physical memory. Consequently, if there are non-zero values in the Swaps column, the SAP instance is

not optimized for workload performance. Most likely, the problem is that either an SAP buffer’s

memory space is almost full, or the buffer has used all of its available free directories. Either of these situations will negatively impact workload performance.

To test the effect of tuning SAP buffer parameters, a baseline test is performed using the following

attributes:

• One application server

• The default out-of-the-box SAP buffer parameter settings

• The memory parameter settings recommended by SAP Note 88416

After the baseline test is run, transaction ST02 is used to determine which SAP buffer parameters

need tuning. The parameters that require tuning to resolve a paging-to-disk problem (displayed

non-zero values in the Swaps column) are shown in Figure 5.

10

Figure 5. Screenshot of SAP transaction ST02 after baseline test

As Figure 5 shows, performance-degrading memory problems exist, which are highlighted in red

on the screen. The program buffer, Generic Key buffer, and Export/import buffer are all swapping

to disk due to a critical shortage in their number of free directories, free space, or both.

The program buffer, Generic Key buffer, and Export/import buffer must be tuned so they have

no swaps to disk and have ample free directories and space available in memory. The tuning is

performed in SAP transaction RZ10. Five parameters need to be tuned in order to achieve optimal performance with this SAP ERP workload for up to 1800 concurrent users:

• Size of program buffer: abap/buffersize=500000

• Maximum number of generic key buffered objects: zcsa/db_max_buftab=80000

• Size of generic key table buffer: zcsa/table_buffer_area=120000000

• Maximum number of export/import buffered objects: rsdb/obj/max_objects=40000

• Size of export/import buffer: rsdb/obj/buffersize=81920

Using these values for the buffer parameters ensure that no swapping to disk occurs.

Improvement in total response time resulting from tuning the SAP buffer parameters is shown

in Figure 6. The results are clear:

• Tuning the appropriate SAP buffer parameters to prevent swapping to disk significantly

improves response time.

11

• Tuning the buffer settings provides a 67% reduction in response time over the default buffer settings.

• Tuning the buffers lowers the processor utilization of the SAP application server.

Figure 6. Comparison of response times based on SAP buffer settings

Managing the total number of users

One of the first goals of this document is to find an appropriate number of concurrent users an

application server can service. Different numbers of users are tested using one application server; the results are presented in Figure 7.

Figure 7. Comparison of response times based on number of users

12

Up to 900 users can use one application server. Figure 7 shows that both response time and CPU

utilization on the server increase with the number of concurrent users on the system. When

950 users are introduced into the system, however, the server’s user response time increases

dramatically.

At the 900 user level, the overall processor utilization for the application server is 78%. At that point, it appears the application server has additional capacity for handling more users. However, the search for additional user capacity comes at a steep price in terms of response time. Running

950 users provides significantly worse response time compared to 850 to 900 users.

The reason that 900 users are accommodated and 950 are not is because the server’s CPU core

0 saturates at over 99% utilization. With 900 users, the 8-core system has an overall utilization

of 78%, but CPU core 0 is at 91% utilization (not shown). With this SAP ERP workload, CPU

core 0 always has the highest percent utilization. The remaining seven CPU cores have nearly

equal processor utilization. The SAP application utilizes CPU core 0 more than the other 7 CPU

cores for this workload.

Best practice

Limit the number of users to 900 per application server with this server configuration. This allows excellent overall CPUutilization without handicapping the SAP system’s response time by saturating CPU core 0 of the 8-processor application server.

Managing the number of total work processes

Each SAP instance can be allocated with a maximum of 100 work processes. However, in order to achieve the best response time, each SAP instance used for an SAP ERP workload must manage the total number of Dialog, Update1, and Update2 work processes. The more work

processes dedicated to an SAP instance, the more physical memory the SAP instance needs

to manage those work processes.

SAP online help suggests that five work processes per CPU core is the optimal work process

distribution for an SAP instance. A 40-process configuration tests this suggested guideline by

maintaining the recommended 5:1 ratio of SAP work processes to CPU cores.

Testing also sought to determine whether increasing the number of available SAP instance work

processes would improve user response time.

The number of work processes per SAP instance is set using SAP transaction RZ10 and is

monitored with transaction SM50. Figure 8 compares the total response times for nine different total work process configurations per SAP instance and shows that increasing the number of work

processes per SAP instance (application server) does not necessarily yield a better response time. In fact, the configuration with the most work processes has the worst total response time.

13

Figure 8. Comparison of response times based on total work processes per SAP instance

The 40-process case demonstrates nearly 40% improvement in total response time compared

to the worst performing cases. The result is an SAP environment that can handle 40% more

user transactions during a given time frame.

Figure 9 shows an increase in the memory needed for allocating more total work processes. The

90-process case uses the most memory and provides the worst total response time.

The results of this testing clearly indicate that the recommendation to maintain a 5:1 ratio of SAP

work processes to CPU cores in a SAP configuration is a valid one. The 40-process case allocates

five work processes per CPU core and produces the best overall performance.

Figure 9. Comparison of memory used based on total work processes per SAP instance

14

Server sizing to accommodate more than 900 concurrent users

By adding more application servers to process the SAP ERP workload, you increase the number of processor cores available to perform the workload and maximize the possible number of concurrent users on the SAP system. To avoid serious impacts to the response time of the SAP

system, HP recommends you add another application server to the SAP landscape for every

900 additional concurrent users. Figure 10 shows the impact on user response time when the

number of application servers is increased from one to two, and the user load is increased from

900 concurrent users to 1,800 users.

Figure 10. Comparison of response times using two application servers versus one application server with 900 users per application server

In both test cases, the processor utilization of each application server is the same.

The response time of each SAP application server does not scale linearly with user load, as

shown with the 70 ms response time with one application server versus the 88 ms response

time with two application servers.

Using the database server as an SAP application server

With SAP ERP 2005, it is easy to set up the SAP system to use the database server as an SAP

application server. However, the SAP instance has to share the CPU and memory resources of the database server with the SQL Server 2005 database managed by the server. There are some

situations when using the database server as an SAP application server is feasible, as long as you

understand the impact that the SAP instance presents to the database server.

Figure 11 compares the SAP response time and processor utilization on the database server to a

dedicated SAP application server.

15

Figure 11. Comparison of response times for an SAP instance on a dedicated application server and the database server

Figure 11 shows that the database server is able to also serve as an SAP application server, but does not provide as good a response time as a dedicated SAP application server. The main issue

is that the SAP instance and the database have to share CPU resources, which become limited

under user load. If the database server is used as an SAP application server, the ability to add

more concurrent SAP users to the SAP system is severely limited. Adding more users means that the database needs more dedicated CPU resources. Adding more concurrent users to an SAP

system increases the processor utilization of the database server.

Best practice

For optimal scalability and performance, place SAP instances only on dedicated SAP application servers.

Storage tuning

To optimize the SAP ERP workload, it is necessary to determine the EVA8100 settings that have

the greatest impact on workload performance. This section describes findings for tuning the

EVA8100 storage array for this SAP ERP workload.

Overall, the EVA8100 array easily handled this workload. The EVA controller utilization is 16%

to 25%, (as shown in Figure 12), which indicates the EVA’s processors are not stressed by this

storage workload.

16

Figure 12. Comparison of EVA processor utilization based on number of users and disks

An SAP ERP workload generates a significant number of IOPS that are proportional to the number of concurrent users the SAP system is servicing.

HP and SAP have conducted research to correlate the number of SAPS an SAP system can

handle, the CPU utilization of the SAP system, and the number of IOPS that system produces.

For this c-Class blade system, approximately 6,000 IOPS are generated by an application server servicing 900 concurrent users. When an additional application server is added, servicing an

additional 900 users (1,800 total users accommodated by the SAP system), the system generates

about 8,500 IOPS.

Due to the OLTP nature of the SAP ERP workload and with its predominantly 8 KB I/O size, the

prime concern for sizing the storage system is accommodating IOPS. If you add more concurrent users (even when adding more application servers), you must also increase the storage resources

dedicated to the SAP system to meet the demands for IOPS with the increased user load.

As with many enterprise applications, the storage system response time to IOPS from the SAP

system should be less than 20 ms. Microsoft characterizes I/O response time to the SQL Server

2005 database as follows:

• A storage system response time of 10-20 ms to SAPDATA files is acceptable.

• A storage system response time of 20-30 ms to SAPDATA files is not acceptable.

• A storage system response time of above 30 ms to SAPDATA files is cause for alarm and

should trigger an investigation.

Testing shows that storage system latency has a direct impact on the response time of the SAP

system to the user. Every effort to improve storage system latency needs to be made with this

SAP ERP workload. Methods for improving the storage system latency are described in the

following sections.

17

Impact of customer data size

For this SAP ERP workload, the database accesses primarily go to the database tables that contain the customer data being referenced. This is expected behavior, because this SAP ERP

workload has the primary characteristics of an SAP Sales and Distribution workload.

Two different customer data sizes are tested: 300 GB and 750 GB. A comparison of the storage

performance is shown in Figure 13 and Figure 14.

Figure 13. Comparison of storage system latencies based on customer data size using Vraid 5

Figure 14. Comparison of storage system latencies based on customer data size using Vraid 1

Figure 13 and Figure 14 demonstrate that the amount of customer data impacts the response

time of the storage system using Vraid 5 or Vraid 1. A smaller amount of data accessed results in

18

better storage system response time because the seek time need to access a smaller amount of data on a disk drive is less than the seek time needed for a larger amount of data.

Seek time is defined as the time it takes to read or write data on a particular location of a disk

drive, including the time the read/write head of the disk needs to physically move to the correct location. If the disk is less occupied with frequently accessed data, the disk drive has to seek

across a smaller number of disk sectors and tracks to read or write the needed data blocks. For the disk drives used during this testing, the seek time of the disk drive comprises approximately

60% of the overall response time (per the disk drive manufacturer’s specifications). There is a

shorter response time when seeking across the smaller number of disk sectors for 300 GB of customer data than for 750 GB of data.

Examining Figure 13, a 40-disk SAPDATA disk group with virtual disks using Vraid 5

accommodates 300 GB of accessed customer data at acceptable storage latency. However, if the customer data size is 750 GB, either Vraid 1 has to be used—or more disks need to be

dedicated to the SAPDATA disk group.

With an 80% read workload (4:1 read/write IOPS ratio) from the database server to the storage

array with 900 concurrent SAP users, the 6,000 host IOPS translate into 7,200 disk IOPS to the

disk drives utilizing Vraid 1, or 180 (7,200/40) disk IOPS per disk drive. In Vraid 5, the same 6,000

host IOPS result in 9,600 disk IOPS being serviced by the disk drives, or 240 (9,600/40) disk

IOPS per disk drive.

Each disk drive in the tested EVA8100 services an average of 164 IOPS (158 write IOPS or 172

read IOPS) at a disk latency of 15 ms. These IOPS numbers, which are published by the disk

drive manufacturer, assume an average seek-time across all possible sectors and tracks of the

disk drive. Under these assumptions, a disk group of 40 disk drives services 6,560 (40 x164) disk

IOPS at a 15-ms latency. Figure 13 and Figure 14 demonstrate that it is possible for a disk drive to

handle more than 164 IOPS with 15 ms latency when the disk drive does not need to seek across

all its sectors and tracks for the customer data.

While an SAP ERP database consists of more than just customer data, the test results confirm that knowing the access pattern of your SAP ERP application to the underlying database can provide

better storage response time than calculations based on the disk manufacturer’s performance data.

Managing the number of disk drives in the SAPDATA disk group

Adding more disk drives to a disk group improves the storage system latency of that disk group’s

virtual disks if the IOPS capacity of the storage system is already under some stress. For the SAP

application, if the disk group containing the SAPDATA virtual disks has a storage latency of greater than 20 ms, the number of disk drives in that disk group might need to be increased.

The effects of adding eight disks to the disk group containing the SAPDATA virtual disks are

presented in Figure 15 and Figure 16.

19

Figure 15. Comparison of the number of disks in the SAPDATA disk group using Vraid 5

Figure 16. Comparison of the number of disks in the SAPDATA disk group using Vraid 1

Figure 15 and Figure 16 demonstrate a significant storage system latency improvement by using

40 disks for the SAPDATA disk group instead of 32 disks for both Vraid 5 and Vraid 1. The reason

is that the capacity for handling IOPS at a given latency is 25% better using 40 disks than 32

disks. With 40 disks dedicated to the SAPDATA disk group, the storage system provides excellent performance with 900 concurrent SAP users regardless of the chosen Vraid type.

Managing the number of EVA disk groups

It is critically important to size the disk group containing the SAPDATA virtual disks appropriately

to achieve desired storage system performance.

20

Best practice

Place virtual disks containing SAPDATA files in a group other than the disk group where the database transaction logsreside to maximize the availability of SAP and its underlying SQL 2005 database.

Due to the pronounced effect the number of disks in the SAPDATA disk group has on storage

latency, special care must be taken when allocating disk drive resources to the SAP system.

For example, if an SAP administrator wants to accommodate 900 concurrent SAP users, the

administrator requests 40 disk drives to handle the I/O load of the SAPDATA files. Non-optimized

storage performance for 900 SAP users occurs if the storage administrator allocates 40 total disks—32 disks for the SAPDATA disk group and 8 disks for the transaction logs.

If the SAP administrator requires a system capable of handling 900 SAP users, the administrator must allocate 48 total disks—40 disks for the SAPDATA disk group and 8 disks for the transaction

log disk group.

Best practice

Size first for IOPS requirements of the SAPDATA disk group, and then allocate 8 disks for the transaction log disk group.Do not allocate disks for the transaction log disk group by removing disks from the SAPDATA disk group.

Selecting the Vraid type for SAPDATA virtual disks

For random I/O storage workloads that consist of less than 100% read IOPS, Vraid 1 provides

better storage system performance than Vraid 5.

Any degree of performance improvement between Vraid 1 and Vraid 5 directly depends on the

percentage of read IOPS versus the percentage of write IOPS that the workload exhibits. The

greater the percentage of write IOPS, the greater the storage performance improvement by

switching from Vraid 5 to Vraid 1. Heavy write-I/O workloads benefit the most from Vraid 1.

This SAP ERP workload, with predominantly OLTP characteristics, is a read-I/O dominated

workload. However, SAP ERP workload does not exclusively consist of read IOPS, so the potential benefits of switching from Vraid 5 to Vraid 1 for the SAPDATA disk group must be examined.

Figure 17 compares the disk latencies of Vraid 1 and Vraid 5 when there are 40 disk drives in

the disk group containing the SAPDATA files. The outcome indicates that for both read and write

latencies, switching from Vraid 5 to Vraid 1 results in an improvement in both read and write

latencies. However, the results also show that whether you use Vraid 5 or Vraid 1, the subsequent read and write latencies indicate that the storage system is performing at a fully acceptable level for this SAP ERP workload.

21

Figure 17. Comparison of Vraid types using the same number of disk drives

Given the same number of disk drives, Vraid 1 consistently provides significant (40% or more) improvement in storage system latency. Acceptable storage latencies are achieved for both

Vraid 1 and Vraid 5 with 40 disks. The decision to use Vraid 1 over Vraid 5 is first based on

achieving a storage system response time goal. If a storage system response time is less than

20 ms using Vraid 5, Vraid 5 is a fully acceptable solution for that SAP system. While Vraid

1 provides even better performance than Vraid 5 for this workload, the performance gains must be weighed against the extra raw storage capacity needed to accommodate a 1 TB SAP ERP

database. With Vraid 5, only 1.25 TB of raw storage is needed for the database, while in Vraid

1, 2 TB of raw storage is needed.

Selecting the Vraid type of virtual disk for the transaction log

The storage activity to the transaction log virtual disk always consists of 100% write IOPS. Because the storage activity is 100% write IOPS, the number of disk IOPS serviced by a

transaction log virtual disk using Vraid 1 is half the number of the same virtual disk using Vraid 5. Therefore, the potential for reaching an IOPS bottleneck using Vraid 1 for the disk group is half that of using Vraid 5. Another reason to choose Vraid 1 is because Vraid 1 provides better data

redundancy for the critical database transaction logs.

This testing compares the use of Vraid 1 with Vraid 5 as the transaction log virtual disk to

determine which configuration has the more favorable impact on total response time. The result is that there is no difference in the total user response time, whether you use Vraid 1 or Vraid

5 for the database transaction log virtual disk. The Vraid type of the transaction log virtual disk

has no impact on the user response time of the SAP system.

The EVA8100 is not stressed by the transaction log activity regardless of Vraid type. Vraid 1 is the

recommended redundancy for the virtual disk containing the transaction logs.

22

Storage sizing to accommodate more than 900 concurrent users

HP recommends you add another application server to this SAP landscape for every 900

additional concurrent users that need to be serviced. The addition of 900 more concurrent users

has a direct impact on the number of IOPS the storage system needs to service. For an 1,800

concurrent user SAP system, approximately 8,500 IOPS need to be provided by the storage

system at an acceptable latency of less than 20 ms.

Using 40 disks in the SAPDATA disk group, Vraid 1 is able to provide a read latency of 20 ms

and a write latency of 7 ms with 300 GB of customer data. Vraid 5 is not capable of providing

acceptable storage system latencies with only 40 disks. More than 40 disks are needed to achieve

acceptable storage system performance with 1,800 concurrent SAP users with Vraid 5 or with a

customer data size of over 300 GB.

23

Best practices

The following sections outline the best practices deduced from this testing for SAP, server, and

storage administrators.

SAP administrators

• Tune the SAP buffers until no swaps are observed with SAP transaction ST02; that is, until there is no paging to disks. The tuned buffer settings provide a 67% decrease in response

time over the default buffer settings.

• Use a total of 40 SAP work processes on each application server for best performance. The

40-process case demonstrates a nearly 40% improvement in total response time as compared

to the worst performing cases. Maintain a 5:1 ratio of SAP work processes to CPU cores

on each application server.

• See SAP Note 88416, entitled Zero administration memory management as of 4.0A/Windows. This SAP note details many appropriate settings for SAP memory parameters on a Windows

server.

Server administrators

• Have no more than 900 concurrent users on a dedicated SAP application server.

• Avoid using the database server as an SAP application server.

• Avoid the situation where CPU core 0 saturates on an SAP application server, by targeting an

overall processor utilization of 80% across all CPU cores on each SAP application server.

Storage administrators

• Size the storage-based-on-IOPS needs first for an SAP ERP workload, and then size

everything else. Due to the I/O size of the SAP ERP workload, disk drive IOPS are the

resource in greatest demand.

• Size all disk groups with multiples of eight disk drives. This is an EVA best practice that

ensures the internal administration of the disk groups is optimized.

• Know that the size of the customer data tables in the SAPDATA files directly impacts the

storage system’s expected latency.

• Ensure the SAPDATA disk group has enough disk drives to accommodate the IOPS

requirements of the SAPDATA files. For one application server serving 900 concurrent users, a minimum of 32 disk drives are required. For two application servers serving 1800 concurrent users, a minimum of 40 disk drives is required.

• Know that Vraid 5 is fully acceptable for SAPDATA virtual disks under most circumstances. Vraid 1 provides better performance for an SAP ERP workload. However, only switch to Vraid

1 if using Vraid 5 does not provide a storage system latency of less than 20 ms.

• Use Vraid 1 for the transaction logs because of its 100% write I/O workload, greater disk

redundancy, and improved fault tolerance.

24

• Deploy the two-disk group configuration only after you have allocated enough disk drives to

the SAPDATA disk group. Do not remove drives from the SAPDATA disk group in order to

create a separate disk group for the database transaction logs.

• Isolate the transaction logs into their own disk group to improve overall availability of the

storage solution.

Issues

SAP Note 88416, Zero administration memory management as of 4.0A/Windows, details many

appropriate settings for SAP memory parameters on a Windows server. The primary memory

setting is PHYS_MEMSIZE, which establishes many other SAP memory settings on a Windows

server. During testing, the default PHYS_MEMSIZE setting is 512 MB on SAP system after completing installation. However, each SAP application server has almost 32 GB of memory

available for use by the SAP instance. With tests running over 300 concurrent users, the SAP

instance crashes and must be restarted. The PHYS_MEMSIZE setting must be set to match

the amount of memory that an administrator is willing to allocate on a Windows server to an

SAP instance.

If the PHYS_MEMSIZE memory parameter is not set appropriately, the memory in the application

server is not used appropriately by the SAP instance, and the SAP application might crash under a

sustained user load.

25

Conclusion

The test results and related information clearly demonstrate how to properly plan for, successfully

deploy, and productively use a fully-operational SAP NetWeaver production landscape on HP

server/storage infrastructure. In addition, specific detailed examples illustrate exactly how to use

SAP’s CCMS to monitor key aspects of the environment.

Key planning considerations include:

• Selecting and configuring the servers

Use dedicated blades for each function in the landscape. Avoid using the database server as

an SAP application server.

• Understanding the upper-limit-to-user-loads on the SAP application server

In our tests, response times degraded rapidly beyond about 900 concurrent users. Adding a

second application server blade provided the ability to roughly double the number of supported

concurrent users.

• Configuring the disk array

In our tests, it was necessary to size for IOPS, not capacity. Once the number of disks

was properly sized, the EVA-8100 array proved to be capable of running the test workload

with minimal tuning. Parity-based VRAID 5 is entirely appropriate, though Vraid 1 remains

acceptable if ultra-high performance and availability is imperative. Finally, use the two-disk

group configuration to isolate transaction logs only if there are enough disks allocated to

SAPDATA.

Key software tuning considerations include:

• Using Zero Administration Memory Management in Windows

There are specific SAP notes that provide the appropriate details.

• Tuning the SAP buffers until no swaps (that is, no paging to disks) are observed

• Configuring the appropriate number of SAP work processes

Testing confirmed optimal results using a 5:1 ratio for SAP work processes to CPU cores.

The combination of understanding all major considerations, along with knowing exactly what to

do, are key to the successful deployment of an SAP NetWeaver production landscape using HP

servers, storage, and enterprise management. The test-proven techniques developed here serve

as a complete guide that can be used with confidence to ensure success.

We value your feedback

In order to develop technical materials that address your information needs, we need your feedback. We appreciate your time and value your opinion. The following link will take you to a

short survey regarding the quality of this paper:

http://hpwebgen.com/Questions.aspx?id=12046&pass=41514

26

Appendix A Bill of MaterialsQTY Description Part number

HP BladeSystem c7000 Enclosure

1 HP BLc7000 CTO Enclosure 412152-B21

8 HP BLc7000 Encl Single Fan Option 412140-B21

6 HP BLc7000 Encl Pwr Sply IEC320 Option 412138-B21

1 HP BLc7000 Encl Mgmt Module Option 412142-B21

HP ProLiant BL460c Servers

6 HP ProLiant BL460c G1 CTO Chassis 404667-B21

3 HP X5160 BL460c G1 FIO Kit 416660-L21

3 HP X5160 BL460c/xw460c G1 Kit 416660-B21

3 HP X5355 BL460c FIO Kit 435565-L21

3 HP X5355 BL460C Kit 435565-B21

15 HP 8GB FBD PC2-5300 2x4GB Kit 397415-B21

12 HP 72GB 15k 2.5 Single Port HP SAS Drive 431935-B21

6 HP BLc Emulex LPe1105 FC HBA Opt Kit 403621-B21

6 HP SA 641/642/E200 128MB BBWC Kit 351580-B21

HP universal rack and power distribution unit

1 HP Universal Rack 10642 G2 Shock Rack AF002A

1 HP 10K G2 600W Stabilizer Kit AF062A

1 HP 10642 G2 Sidepanel Kit AF054A

2 HP 40A HV Core Only Corded PDU 252663-D75

EVA storage array

1 HP EVA8000 2C12D-A 60Hz 42U Cabinet AD522B

168 HP StorageWorks 146GB 15K FC HDD 364621-B23

8 Storage Works LC/LC 15m Cable 221692-B23

1 EVA8100 Upgrade Kit

Infrastructure − Network switches

2 HP BLc Cisco 1GbE 3020 Switch Opt Kit 410916-B21

Infrastructure − SAN switches

2 Brocade BladeSystem 4/24 SAN Swt Powr Pk AE371A

27

Appendix B References

Thomas, Juergen. SAP with Microsoft SQL Server 2005: Best Practices for High Availability, Maximum Performance, and Scalability. SQL Server Technical Article. February, 2007.

28

For more information

HP technical references

• HP StorageWorks Customer Focused Testing

http://www.hp.com/go/hpcft

• HP SAP Solutions

http://www.hp.com/go/SAP

• HP data storage and HP StorageWorks products

http://www.hp.com/go/storage

• HP Blade servers

http://www.hp.com/go/blades

• HP ProLiant servers

http://www.hp.com/go/proliant

©2008 Hewlett-Packard Development Company, L.P. The information containedherein is subject to change without notice. The only warranties for HP productsand services are set forth in the express warranty statements accompanyingsuch products and services. Nothing herein should be construed as constitutingan additional warranty. HP shall not be liable for technical or editorial errors oromissions contained herein. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation.4AA1-5661ENW, March 2008

29