lx86 and linux on ibm power systems

© Copyright IBM Corporation, 2010

Lx86 and Linux on IBM Power Systems A comparative study of native file system I/O performance using Iozone

Jose Moises Romo Corona Larry Lancaster

IBM Systems and Technology Group ISV Enablement

July 2010

Lx86 and Linux on IBM Power Systems

Table of contents

Abstract........................................................................................................................................1

Introduction .................................................................................................................................1

Iozone application description ..................................................................................................2

The testing environment ............................................................................................................3 Topology .................................................................................................................................................. 3 Server configuration................................................................................................................................. 4 The Iozone test suite................................................................................................................................ 4

Results .........................................................................................................................................6 Test scenario 1: Write, read, random read, and random write performances......................................... 7

Read performance comparison ......................................................................................... 9 Random read performance comparison.......................................................................... 11 Random write performance comparison ......................................................................... 13 Processor utilization ........................................................................................................ 15

Test scenario 2: Simultaneous multithreading ON ................................................................................16 Processor utilization for SMT ON.................................................................................... 18

Test scenario 3: Simultaneous multithreading OFF .............................................................................. 20 Processor utilization for SMT OFF .................................................................................. 23

Conclusions...............................................................................................................................25

Summary and recommendations ............................................................................................26

Appendix....................................................................................................................................27 1. Physical and logical volume characteristics for Linux........................................................................ 27

Linux RHEL5.3 and Lx86 RHEL5.3................................................................................. 27 Linux SLES10.3 and Lx86 SLES10.1 ............................................................................. 27

2. Disk highlights for the Power Systems server ................................................................................... 27 3. Iozone file system benchmark ........................................................................................................... 29

3.1 Downloading Iozone.................................................................................................. 29 3.2 Building Iozone with GCC ......................................................................................... 30 3.3 Iozone full automatic test mode................................................................................. 30

Resources..................................................................................................................................32

About the authors .....................................................................................................................33

Trademarks and special notices..............................................................................................34

Lx86 and Linux on IBM PowerSystems

1

Abstract

This paper shows I/O performance characteristics for IBM PowerVM Lx86 native file system running on Linux on Power Systems RHEL5.3 with 64 KB kernel page and running on Linux SLES10.3 with 4 KB kernel page. This paper also highlights the significant I/O performance differences on Lx86 file system

compared to native Linux file system.

The benchmark code used to perform the tests for this paper is Iozone1 3.326 compiled with GCC version 4.1.2_20070115. Iozone is a third-party file-system benchmark application.

Introduction

IBM® PowerVM® Lx86 technology supports the installation and running of most 32-bit x86 Linux® applications on any IBM Power Systems™ or IBM BladeCenter® model with IBM POWER6™, IBM POWER5+™, or IBM POWER5™ processors.

PowerVM Lx86 allows you to improve x86 Linux workloads with Power Systems advantages such as scalability and optimized use of processor, memory, disk, and network resources. The POWER5 processor supports page sizes of 4 KB and 16 MB, and the POWER5+ processor has added two new

page sizes: 64 KB and 16 GB. If the workload involves access to a large number of small files (tens of KB or smaller in size), the 64 KB page size may result in more memory wastage in the kernel page cache than a 4 KB page size might.

Simultaneous multithreading (SMT), is a technique for improving the overall efficiency of the processor utilization. PowerVM Lx86 and Linux share the same physical hardware and the effect of SMT on Lx86 I/O throughput is shown in this paper.

The Iozone application, used to perform the test for this paper, uses two logical partitions:

The first one with Red Hat Enterprise Linux (RHEL) native journaled file system, in which ext3 is the file system and RHEL5.3 is the operating system.

The next one with SUSE Linux Enterprise Server (SLES) journaled file system, in which reiserfs is the file system and SLES10.3 is the operating system.

In the case of Lx86, the application uses two Linux partitions:

The first one with RHEL native journaled file system, in which ext3 is the file system and Lx86 RHEL5.3 is the operating system.

The next one with SUSE Linux Enterprise Server journaled file system, in which reiserfs is the file

system and Lx86 SLES10.1 is the operating system.

The intend of this paper is to provide IBM and ISVs support personnel guidance on the relative I/O performance characteristic on Lx86 compared to native Linux. This paper does not intend to compare I/O

performance between Linux RHEL5.3 ext3 and SLES10.3 reiserfs file systems.

1 http://www.Iozone.org


2

Iozone application description

Iozone is a file-system benchmark available in public. It creates and measures a variety of file I/O operations. It is useful for performing a broad analysis of a file system that resides on a computer platform.

You can perform many types of benchmark tests using Iozone. However, the following list describes the tests that are selected for use in this paper.

Read – This test measures the performance of reading an existing file.

Write – This test measures the performance of writing a new file.

o Note: When a new file is written, apart from the data, the overhead information for keeping track of where the data is located on the storage media should also be stored.

This overhead information is called the metadata. It consists of the directory information, the space allocation details, and any other data (such as creation time, the length of the data contained in the file, its owner user-ID and group-ID) associated with a file that is not

part of the data contained in the file. It is normal for the initial write performance to be lower than the performance of rewriting a file because of this overhead information.

Re-read – This test measures the performance of reading a file that was recently read.

Re-write – This test measures the performance of writing a file that already exists.

Random read -- This test measures the performance of reading a file with accesses being made to random locations within the file. Several factors such as size of the operating system’s cache,

number of disks, and seek latencies impact the performance of a system under this type of activity.

Random write – This test measures the performance of writing a file with accesses being made to

random locations within the file. Similar to the random read activity, several factors such as size of operating system’s cache, number of disks, and seek latencies impact the performance of a system under this type of activity.


3

The testing environment

This section explains the testing environment that includes the topology, server configuration, and the benchmark application.

Topology

Figure 1 shows the hardware configuration for the logical partitions (LPARs) where all tests described in

this paper take place. Two full dedicated LPARs are configured and each LPAR resides in a separate IBM Power® server. A dedicated LPAR has full access to physical volume, processor, and memory resources. Each LPAR is managed by the same Hardware Management Console (HMC). The tests

described in this paper were run directly in the root / native file system, ext3 for Linux RHEL5.3 and Lx86 RHEL5.3 as well as reiserfs for Linux SLES10.3 and Lx86 SLES10.1. Lx86 was installed in the default /i386 location for each server, and therefore, Lx86 and native Linux share the same physical disk and

logical volume.

Figure 1: Hardware configuration for Lx86 and Linux


4

Server configuration

While configuring the server for Linux RHEL5.3 and Linux SLES10.3, you need to perform hardware partitioning as summarized in Table 1.

System values

System property Lx86 - RHEL5 Linux - RHEL5 Lx86- SLES10 Linux - SLES10

Processing units 2 2 2 2

Clock speed 1898.100000 MHz

1898.100000 MHz 1898.100000 MHz 1898.100000 MHz

Processor cache size

256 KB 1024 KB 256 KB 1024 KB

Dedicated memory 3.848 GB 3.848 GB 3.848 GB 3.848 GB

Disk See 2. Disk highlights for the Power Systems server

See 2. Disk highlights for thePower Systems server

See 2. Disk highlights for the Power Systems server

See 2. Disk highlights for the Power Systems server

File system ext3 ext3 reiserfs reiserfs

Block size 4 KB 4 KB 4 KB 4 KB

OS Lx86 with Red Hat Enterprise Linux Server release 5.3

Red Hat Enterprise Linux Server release 5.3

Lx86 with SUSE Enterprise Linux Server release 10.1

SUSE Enterprise Linux Server release 10.3

Kernel version 2.6.18-128 2.6.18-128.el5 2.6.16.60-0.54.5 2.6.16.60-0.54.5

Kernel page size 4 KB 64 KB 4 KB 4 KB

Lx86 version powervm-lx86-1.3.2.0-1

NA powervm-lx86-1.3.2.0-1

NA

Table 1: Hardware partitioning summary

The Iozone test suite

Table 2 describes all the test scenarios that are run using the Iozone. For full automatic consist of I/O

performance measurement for record and file size combination. Record size starts at 4 KB and increments by the double each iteration reaching up to 16 MB. File size starts at 64 KB and increments by the double each iteration reaching up to 4 GB.


5

For full automatic test, because all platforms described in Table 1 have a total memory below 4 GB, the maximum file size specified in Iozone will be 4 GB (depicted in the section, 3.3 Iozone full automatic

test mode) to bypass the buffer cache and go directly to disk after exceeding maximum memory.

For multithreading tests, file size is set to 1000 MB and record size to 256 KB. All tests in multithreading mode read and write from cache and not from physical memory.

While reading the results of these tests when the file size is below maximum buffer or cache memory, it is observed that the larger the number, the better the results.

Linux RHEL5.3 runs on 64 KB kernel page size whereas Linux SLES10.3 runs on 4 KB. Lx86 SLES10.1

and Lx86 RHEL5.3 run on 4 KB kernel page.

Test type Description SMT Platform – file System

1 Full automatic test for Write Read Random write Random read

(See 3.3 Iozone full automatic test mode in the Appendix for the description on automatic tests parameter settings)

The objective of this test is to determine the median I/O performance for common record size over a range of file sizes.

Enabled Linux RHEL5.3 - ext3 Lx86 RHEL5.3 - ext3 Linux SLES10.3 - reiserfs Lx86 SLES10.1 - reiserfs

2 SMT – Multithreading tests for

write/re-write read/re-read

The objective of this test is to run the write/re-write and read/re-read tests in a threaded mode. It sets the following parameters:

Three threads 1000 MB file size 256 KB record size

Enabled Linux RHEL5.3 - ext3 Lx86 RHEL5.3 - ext3 Linux SLES10.3 - reiserfs Lx86 SLES10.1 - reiserfs

3 SMT – Multithreading tests for

write/re-write read/re-read

The objective of this test is to determine performance characteristics of write/re-write and read/re-read operations in a nonthreaded mode. The following parameters are used:

Three threads 1000 MB file size 256 KB record size

Disabled Linux RHEL5.3 - ext3 Lx86 RHEL5.3 - ext3 Linux SLES10.3 - reiserfs Lx86 SLES10.1 - reiserfs

Table 2: Test scenarios


6

Results

The quantitative comparison data for all scenarios described in Table 2 is analyzed in this section. For the fully automatic test scenario, the median I/O performance value in megabytes for each record size is compared between Lx86 and native Linux to highlight the performance differences and find the record

size that provides the maximum performance value. The median I/O performance value, in megabytes for each files size, is also compared for full automatic test scenario to be able to highlight the performance behavior when the file size exceeds the system’s buffer-cache memory. The processor utilization when

each of the tests run is also compared.


7

Test scenario 1: Write, read, random read, and random write performances

This test scenario compares the median performance for each record size for the read, write, random

write, and random read I/O performances. Figure 2 shows the comparison of operating system combinations for write performance. Median value for I/O performance for each record size across file sizes is compared.

Figure 2: Write performance median for each record size by platform


8

Linux RHEL5.3 is built to expect 64 KB pages, however, when Lx86 is installed, the kernel page size is set to 4 KB. If the workload involves access to a large number of small files (that are less than 100

KB or smaller), the 64 KB page size will result in more memory wastage in the kernel page cache than a 4 KB page size would. Lx86 SLES10.1 is closer to native Linux performance because both use 4 KB default page size in the kernel.

Figure 3 shows the comparison of all the hardware and operating system combinations for write performance. Median value for I/O performance for each file size across record size is compared.

Figure 3: Write performance median for each file size by platform

Table 3 shows the write performance median for a 4 GB file.

Platform Median performance ( MBps)

Linux- RHEL5.3 112.63

Lx86- RHEL5.3 98.91

Linux- SLES10.3 99.24

Lx86- SLES10.1 95

Table 3: Write performance median in MB for a 4 GB file.

When file size is above maximum buffer or cache memory, the performance drops significantly and here is where the performance is almost the same between Lx86 and native Linux. Notice that Lx86

performance on RHEL5.3 is about 50% of the native Linux for file size less than 2 GB, whereas, Lx86 on SLES10.1 is closer to native Linux performance.


9

Read performance comparison

Figure 4 shows the comparison of all the hardware and operating system combinations for read

performance. Median value for I/O performance for each record size across file sizes is compared.

Figure 4: Read performance median for each record size by platform

Lx86 RHEL5.3 is far away from native performance when compared with Lx86 SLES10.1. This is because of the differences between kernel page sizes in Linux.


10

Figure 5 shows the comparison of all the hardware and operating system combinations for read performance. Median value for I/O performance for each file size across record size is compared.

Figure 5: Read performance median for each file size by platform.

Table 4 shows the read performance median for a 4 GB file.

Platform Median performance ( MBps )


Lx86- RHEL5.3 79.77


Lx86- SLES10.1 62.04

Table 4: Read performance median in MB for a 4 GB file size

When the file size is above the maximum buffer or cache memory, the performance drops

significantly. When the file size does not fit in the buffer or cache, the performance on LX86 is closer to Linux. However, Lx86 RHEL5.3 holds low performance across file sizes below 4 GB and Lx86 SLES10.1 is closer to native, especially for files from 4 MB to 2 GB in size.


11

Random read performance comparison

Figure 6 shows the comparison of all the hardware and operating system combinations for random

read performance. Median value for I/O performance for each record size across file sizes is shown.

Figure 6: Random read performance median for each record size by platform


12

Figure 7 shows the median value for I/O performance for each file size across record size.

Figure 7: Random read performance median for each file size by platform

Table 5 shows the random read performance median for a 4 GB file size.

Platform Median performance ( MBps )


Lx86- RHEL5.3 25


Lx86- SLES10.1 45.76

Table 5: Random read performance median in MB for a 4 GB file size

Lx86 RHEL5.3 holds low performance across file sizes below 4 GB. Lx86 SLES10.1 is closer to native, especially for files from 4 MB to 2 GB.


13

Random write performance comparison

Figure 8 shows the comparison of all the hardware and operating system combinations for random

write performance. Median value for I/O performance for each record size across file sizes is shown.

Figure 8: Random write performance median for each record size by platform


14

Figure 9 shows the median value for I/O performance for each file size across record size.

Figure 9: Random write performance median for each file size by platform

Table 6 shows the random write performance median for 2 GB and 4 GB files.

Platform Median performance ( MBps ) for a 2 GB file

Median performance ( MBps ) for a 4 GB file

Linux- RHEL5.3 104.95 51.4

Lx86- RHEL5.3 61.47 45.04

Linux- SLES10.3 256.92 77.02

Lx86- SLES10.1 272.61 73.19

Table 6: Random read performance median in MB for a file size of 2 GB and 4 GB

On random write test, the performance starts decreasing significantly for a 2 GB file on all platforms. The performance of a system under this type of activity can be impacted by several factors such as:

size of operating system’s cache, number of disks, and seek latencies.

The Lx86 development team has been working to improve the performance in new releases to bring it closer to native Linux. However, notice that for all write, random write, read and random read tests,

Lx86 RHEL5.3 is far away from the native Linux for all record and file size combination.


15

Processor utilization

As Iozone full automatic test for read, write, random read, and random write run issuing one single

command, the processor utilization is measured using the command in Example 1, for all the tests in each system.

# sar -u -P ALL 1 25200

Example 1: sar command to get the processor utilization in all systems

The sar output is redirected to a log file to save the data.

It would have been difficult to get detailed processor utilization data for each test type and other

monitoring tools would be needed The data in Table 7 shows the overall processor utilization for all tests running write/read followed by random write/random read.

Platform Processor utilization in percentage


Lx86- RHEL5.3 56.02


Lx86- SLES10.1 32.38

Table 7: Processor utilization for Iozone full automatic test mode

Lx86 for both RHEL5.3 and SLES10.1 is consuming almost the same percentage of processor utilization as native Linux on single-thread operations. Lx86 SLES10.1 has really good performance for all tests compared to Linux, but the Lx86 performance on RHEL5.3 is about 50% for write, 7.71%

for read, 7.55% for random read, and 15.21% for random write of native Linux RHEL5.3 performance.


16

Test scenario 2: Simultaneous multithreading ON

Simultaneous multithreading is a productivity enhancement that provides two independent threads for executing instructions on each physical processor. Each thread appears to the operating system as a

logical processor. Thus, turning on SMT mode makes one physical processor appear as two processors to the operating system and to the applications running in that partition. As Linux and Lx86 share the same physical resources, it is assumed that SMT is enabled in Lx86 when it is

enabled on Linux.

The Iozone test for SMT was performed on Linux and Lx86 (both RHEL5.3 and SLES10.1) for read, re-read, write, and re-write operations with a 256 KB record. The results showed maximum

performance for write and read tests as shown in Figure 2 and Figure 4.

The test described in this section hit only the buffer/cache memory.

Example 2 shows the Iozone command to run throughput test for this scenario. Table 8 shows the

flag details.

./iozone -r256k -R -b file_name.xls -s1000m -i0 -i1 -t3 -F /iozone_test/test1/fic1 /iozone_test/test2/fic2 /iozone_test/test3/fic3

Example 2: An invocation string for Iozone SMT tests

R Generates output that is Microsoft® Excel compatible and enables further refining of the output

into a graph using Microsoft Excel spreadsheet

r Records file size in KB (256 KB for this test)

b Creates a binary-format file that is Excel compatible; file name is specified using the .xls

file extension

s Provides file size specification. The file size option can be specified by -s #k (KB) or -s #m

(MB) or -s #g (GB). For this scenario, the file size option used is -s1000m (1000 MB).

i0 Instructs Iozone to test definition for write/re-write

i1 Instructs Iozone to test definition for read/re-read

t3 Runs Iozone in a throughput mode with three threads

F Specifies file location

Table 8 Iozone options used in the SMT tests


17

Example 3 shows the command that is used to verify that the system is SMT capable and to find the SMT status.

# ppc64_cpu --smt

smt is on

Example 3: Command to verify the SMT status in Linux

Example 4 shows the command to enable SMT in Linux, restart is not required.

# ppc64_cpu --smt=on

Example 4: Command to enable SMT in Linux

Figure 10, Figure 11, Figure 12, and Figure 13 provide a comparison of throughput on various platforms for the tests described in Example 2.

The following are the observations from the tests:

All platforms show low throughput for write and re-write tests when compared to read and re-read. For this scenario, kernel page size is not affecting Lx86 RHEL5.3.

The throughput in Lx86 running on RHEL5.3 and SLES10.1 for write and re-write tests reaches native Linux throughput.

Figure 10 and Figure 11 show the throughput results for the tests running on Linux RHEL5.3 and

Lx86 RHEL5.3 (described in Example 2).

Figure 10: Throughput comparison for write/re-write on RHEL5 with SMT enabled

Figure 11: Throughput comparison for read/re-read on RHEL5 with SMT enabled


18

Figure 12 and Figure 13 show the throughput results for the tests running on Linux SLES10.3 and Lx86 SLES10.1 (described in Example 2).

Figure 12: Throughput comparison for write/re-write on SLES10 with SMT enabled

Figure 13: Throughput comparison for read/re-read on SLES10 with SMT enabled

Table 9 shows the throughput for multithreading on all platforms.

Platform Write KBps Re-write KBps Read KBps Re-read KBps

Linux- RHEL5.3 163148.64 180725.32 8956068.5 9014354.5

Lx86- RHEL5.3 131470.95 134985.66 658264.83 664510.52

Linux- SLES10.3 115483.36 131235.46 6676541.75 6835389.62

Lx86- SLES10.1 145404.48 182535.91 606066.25 600282.3

Table 9: Multithreading throughput summary for all platforms, SMT enabled

Processor utilization for SMT ON

Processor utilization data was gathered using Ksar 2 5.0.6. Ksar is a Java™ application distributed

under Berkeley Software Distribution (BSD), which represents a family of permissive free software licenses. Ksar allows you to connect to a server using the Secure Shell (SSH) protocol and running a sar command. The sar process generates the data from subsystems being monitored and the Ksar application recollect that data to export it to comma separated value (CSV) or portable document format (PDF) files.

2 http://ksar.atomique.net/index.html


19

The command shown in Example 5 is used to monitor the processor utilization with sar.

# sar -u -P ALL 3 200

Example 5: sar command to get the utilization data of all processors

After the sar process data is exported to the csv file, for each Iozone test, the processor utilization is

noted. Example 6 shows the formula to find the processor utilization.

100 - (CPU % idle average)

Example 6: Processor utilization formula

Figure 14 shows the CPU utilization in each system for the test described in Example 2.

Figure 14: Processor utilization percentage for throughput by platform, SMT enabled

The following is the processor utilization that is observed when SMT is on:

Lx86 for both SLES10.1 and RHEL5.3 is consuming more percentage of processor than native Linux, however, the throughput for Lx86 in both is behind native Linux for read and re-read.


20

Test scenario 3: Simultaneous multithreading OFF

It is possible to disable SMT in Linux without reboot the system. Example 7 shows the command to disable SMT on Linux.

# ppc64_cpu --smt=off

Example 7: Command to disable SMT

As Linux and Lx86 are sharing the same physical resources, it is assumed that SMT is disabled in Lx86 when it is disabled on Linux.

The same test described in Example 2 was performed in this scenario to verify the impact of

disabling SMT in multithreading I/O workloads on native file systems with Iozone.

Figure 15 and Figure 16 provide a comparison of throughput for the RHEL5 platforms for the tests described in Example 2, as well Figure 17 and Figure 18 show throughput for SLES10.

The following are the observations for this scenario:

Kernel page size is not affecting Lx86 RHEL5.3 throughput.

Lx86 throughput is not being affected if SMT is disabled. In the case of Lx86 RHEL5.3, better throughput results than native Linux is obtained for write and re-write when SMT is disabled.


21

Figure 15 and Figure 16 show the performance comparison between native Linux RHEL5.3 and Lx86 RHEL5.3.

Figure 15: Write/Re-write throughput comparison for Lx86 and native Linux RHEL5.3

Figure 16: Read/Re-read throughput comparison for Lx86 and native Linux RHEL5.3


22

Figure 17 and Figure 18 show the performance comparison between native Linux SLES10.3 and Lx86 SLES10.1.

Figure 17: Write/re-write throughput comparison for Lx86 and native Linux SLES10.3

Figure 18: Read/re-read throughput comparison for Lx86 and native Linux SLES10.3


23

Table 10 shows the summary of throughput for Lx86 RHEL5.3 and Linux RHEL5.3.

Platform SMT status Write KBps Re-write KBps Read KBps Re-read KBps

Linux- RHEL5.3 enabled 163148.64 180725.32 8956068.5 9014354.5

Linux- RHEL5.3 disabled 157653.55 230864.97 8876512.62 9908968.5

Lx86- RHEL5.3 enabled 131470.95 134985.66 658264.83 664510.52

Lx86- RHEL5.3 disabled 170558.82 189798.83 477320.84 657288.98

Table 10: Multithreading throughput summary for Lx86 RHEL5.3 and Linux RHEL5.3

Table 11 shows the summary of throughput for Lx86 SLES10.1 and Linux SLES10.3.

Platform SMT status Write KBps Re-write KBps Read KBps Re-read KBps

Linux- SLES10.3 enabled 115483.36 131235.46 6676541.75 6835389.62

Linux- SLES10.3 disabled 155226.59 259788.2 6726242.25 6978080.25

Lx86- SLES10.1 enabled 145404.48 182535.91 606066.25 600282.3

Lx86- SLES10.1 disabled 97041.45 189492.55 559385.74 557124.88

Table 11: Multithreading throughput summary for Lx86 SLES10.1 and Linux SLES10.3

Processor utilization for SMT OFF

Same approach described in Example 5 and Example 6 was taken to get the processor utilization for multithreading tests in all platforms. Figure 19 shows the processor utilization for all systems when SMT is disabled.

Figure 19: Processor utilization percentage for throughput by platform, SMT disabled


24

The following are the observations for processor utilization when SMT is disabled:

Lx86 running on both SLES10.1 and RHEL5.3 is consuming less processor utilization than native

Linux when SMT is disabled

Lx86 running on RHEL5.3 and SLES10.1 has better performance for read and re-read when SMT is enabled

Throughput for read increases about 27. 48 % and reread increases about 1.08 % in Lx86 RHEL5.3, when SMT is enabled

Throughput for read increases about 7. 7 % and reread increases about 7.18 % in Lx86

SLES10.1, when SMT is enabled

SMT - ON did not improve the throughput for write and re-write in Lx86 (RHEL5.3 and SLES10.1)

Figure 20 shows the processor utilization summary for all the systems for both the scenarios: SMT enabled and SMT disabled. In general, all platforms are consuming more processor utilization when SMT is disabled and Lx86 for both RHEL5.3 and SLES10.1 consume more processor utilization than

native Linux when SMT is enabled.

Figure 20: Processor utilization summary for throughput


25

Conclusions

The following are the conclusions arrived from the various test scenarios:

Single-threaded tests

For Lx86 (RHEL5.3 and SLES10.1), highest I/O performance for single-threaded tasks is achieved with 256 KB and 512 KB records sizes although the variation is not particularly significant.

Lx86 RHEL5.3 I/O performance in single-threaded tasks is significantly degraded, compared to native RHEL5.3.

− For write, Lx86 RHEL5.3 achieves only 50% of native RHEL5.3 performance.

− For read, Lx86 RHEL5.3 achieves only 7.71% of native RHEL5.3 performance.

Lx86 SLES10.1 has an excellent I/O performance in single-threaded tasks; it approaches the native SLES10.1 performance for write, random write, read, and random read tests.

Multi-threaded tests

SMT has little effect on Linux and Lx86 I/O performance.

In multi-threading tasks, Lx86 and native Linux write and re-write performances are comparable.

In multi-threading tasks, Lx86 read and re-read performance is considerably weaker when compared to native Linux read and re-read performance.

Table 12 shows the throughput percentage capacity on Lx86 (RHEL5.3 and SLES10.1) for

multithreading on read and re-read.

Platform Throughput percentage capacity for

multithreading

Lx86 SLES10.1

(running in Linux with 4 KB kernel page size

Read: 9.07% of native file system

Linux capacity

Re-read: 8.78% of native file system

Linux capacity

Lx86 RHEL5.3

(running in Linux with 64 KB kernel page size)

Read: 7.34% of native file system

Linux capacity

Re-read: 7.37% of native file system

Linux capacity

Table 12: Lx86 throughput capacity for multithreading


26

Summary and recommendations

Based on I/O performance results obtained from running the suite of tests covered in this paper, running applications under Lx86 is:

Not recommended when:

− The operating environment is RHEL5 or SLES 11 and the applications being hosted are I/O intensive

Recommended when:

− The operating environment is RHEL5 or SLES 11 and the applications are mostly processor intensive

− The operating environment is RHEL5 or SLES11 and application turn around is not a

priority

− The operating environment is RHEL4 or SLES10

Lx86 1.3.2, which is the current version of the product, is I/O constrained because of the inefficiencies of

x86 applications requiring 4 KB kernel page sizes. It is anticipated that future versions of Lx86 might address and solve these current I/O constraints.


27

Appendix

1. Physical and logical volume characteristics for Linux This section explains the physical volume characteristics and the logical volume characteristics.

Linux RHEL5.3 and Lx86 RHEL5.3 Physical volume characteristics SIZE: 68.25G TOTAL PHYSICAL EXTENTS: 2184 ALLOCATED PHYSICAL EXTENTS: 2184 ATTRIBUTES: a- CONTROLER TYPE: SCSI Logical volume characteristics SIZE: 62.5G NUMBER OF SEGMENTS: 1 ATTRIBUTES: -wi-ao FILE SYSTEM TYPE: ext3

Linux SLES10.3 and Lx86 SLES10.1

Physical volume characteristics SIZE: 68.25G TOTAL PHYSICAL EXTENTS: 2184 ALLOCATED PHYSICAL EXTENTS: 2184 ATTRIBUTES: a- CONTROLER TYPE: SCSI Logical volume characteristics SIZE: 67 GB NUMBER OF SEGMENTS: 1 ATTRIBUTES: -wi-ao FILE SYSTEM TYPE: reiserfs

2. Disk highlights for the Power Systems server


28

The following are the disk highlights of the IBM Power Systems™ server.

- Drive capacity: 73.4 GB - Rotational speed: 15,000 RPM - Seek time: 3.5 ms (Average) - Buffer: 8 MB - Connectivity: 80 pin HD D-Sub Table 13 shows the IBM PowerPC® disk specifications.

Manufacturer : Seagate Manufacturer part # ST373454LC

Data storage Formatted capacity 73.4 GB

Bytes per second 512

Disk drive configuration Heads 2 physical

Disks 16 logical

Data transfer rate Internal 48.25 MBps Internal Max

External 100 MBps External Max

Maximum sustained 34.5 MBps Sustained Avg

Buffer 8 MB

Seek time Track-Track n/a

Average 12 ms

Full stroke n/a

Average rotational latency 5.6 ms

Rotational speed 15 000 RPM

Interface Ultra 320 SCSI

Interface connectors 80 pin Centroncis (SCA-2)

Table 13: PowerPC disk specification


29

3. Iozone file system benchmark

To achieve the goal of file system performance and throughput comparison, you can use the Iozone workload tool. Iozone is a file system benchmark tool that generates and measures a variety of file

operations. The benchmark tests file I/O performance for the following operations:

Read

Write

Re-read

Re-write

Read backwards

Read strided

Fread

Fwrite

Random read

Pread

Mmap

aio_read

aio_write

You can find the Iozone benchmark at http://www.Iozone.org/. For Linux and Lx86, you need to

download the Iozone source code and build. The next several sections describe this process.

3.1 Downloading Iozone

To download Iozone:

1. Go to the URL: http://www.Iozone.org/

2. Navigate to the Download Source section and click the Stable tarball link.

3. Save the download to the local drive.

4. After the download is complete, decompress the tar ball and move to Iozone version/src/current/.


30

3.2 Building Iozone with GCC

To build Iozone in Linux, perform the following steps:

1. Move to the iozone3_326/src/current directory, which contains the Iozone source code.

2. Run the following command:

$ make linux-powerpc64

To compile the Iozone code in Lx86, perform the following steps:


$ runx86

2. Move to the iozone3_326/src/current directory, which contains the Iozone source code.


$ make linux

3.3 Iozone full automatic test mode

Example 8 shows the Iozone command used to perform the first test scenario, described in Table 2.

./iozone -Raz -b file_name.xls –g 4G -i0 -i1 -i2

Example 8: Execution string for Iozone running in automatic test mode


31

Table 14 describes the Iozone automatic test mode flags.

R This option generates an output that is Microsoft Excel compatible. It can easily be input into an Excel spreadsheet to be further refined into a graph by using the -b option.

a This option selects the full automatic mode, runs all possible test type combination, and produces output that covers all tested file operations for record sizes ranging from 4 KB to 16 MB and file sizes ranging from 64 KB to a maximum file size specified by the -g flag. In the

case of Example 8, the maximum file size is set to 4 GB. The file size starts from 64 KB and for every iteration, the file size multiplies by two, until file size reaches a 4 GB size.

In addition, the –a flag, instructs Iozone to stop the use of any record size less than 64 KB when

the file size is 32 MB or larger.

z This option is used in conjunction with –a to test all possible record sizes. Normally, Iozone

omits testing of small record sizes for very large files when used in full automatic mode. This option forces Iozone to include the small record sizes in the automatic tests.

b This option creates a binary file that is Microsoft Excel compatible. File name is specified using the .xls file extension.

g This option sets the maximum file size (in GB) for the auto mode. For this set of tests, 4 GB is used.

i0 This option specifies write and re-write tests. This flag instructs Iozone to run the write and re-write tests first.

i1 This option specifies read and re-read tests. This flag instructs Iozone to run the read and re-read tests after running the tests specified by the i0 flag.

i2 This option specifies random read and random write tests. This flag instructs Iozone to run the random read and random write tests after running the tests specified by the i1 flag.

Table 14: Iozone automatic test mode flags


32

Resources

These websites provide useful references to supplement the information contained in this document:

Getting Started with PowerVM Lx86 ibm.com/redbooks/redpapers/pdfs/redp4298.pdf

IBM Power Systems on IBM PartnerWorld®

ibm.com/partnerworld/systems/p

Iozone Filesystem Benchmark

www.Iozone.org/

Ksar sar grapher

http://ksar.atomique.net/index.html

PowerPC Disk characteristics

www.superwarehouse.com/Seagate_Cheetah_73.4_GB_SCSI_Internal_Hard_Drive/ST373454L

C/ps/472330

PowerVM Lx86 for x86 Linux applications

ibm.com/systems/power/software/virtualization/editions/lx86/index.html

64 KB pages on Linux for Power systems

ibm.com/developerworks/wikis/display/hpccentral/64KB+pages+on+Linux+for+Power+systems#64KBpagesonLinuxforPowersystems-Choiceofbasepagesize


33

About the authors

Jose Moises Romo Corona is a Technical Consultant in the Solutions Enablement organization in IBM Systems and Technology Group. He has around four years of experience with IBM. He graduated in Computer Science and Electronics from Universidad del Valle de Atemajac. His areas of expertise

include Web 2.0 Development, Linux, and system administration.

Larry Lancaster is a Solutions Enablement Manager for Symantec, HP, Micro Focus, Vision, and Software AG. Within IBM he has held positions as Technical Client Representative to Conoco-Phillips,

and Critical Situation Manager and Complex Project Manager for the Petroleum Sector. Previous to his 18 years with IBM, he worked for Texas Instruments as a Senior Software Developer with specialty in 3D Seismic applications. Mr. Lancaster received a triplet undergraduate B.S. degree at the University of

Colorado in Abstract Mathematics, Computer Science, and Philosophy. Mr. Lancaster holds one patent on Pausing and Resuming Applications Running in a Data Processing System.


34

Trademarks and special notices

© Copyright IBM Corporation 2010. All rights Reserved.

References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these

symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the web at "Copyright and trademark information" at

www.ibm.com/legal/copytrade.shtml.

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States,

other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly

available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the

supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller

for the full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with

respect to any future products. Such commitments are only made in IBM product announcements. The


35

information is presented here to communicate IBM's current investment and development activities as a

good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled

environment. The actual throughput or performance that any user will experience will vary depending

upon considerations such as the amount of multiprogramming in the user's job stream, the I/O

configuration, the storage configuration, and the workload processed. Therefore, no assurance can be

given that an individual user will achieve throughput or performance improvements equivalent to the

ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Any references in this information to non-IBM websites are provided for convenience only and do not in

any manner serve as an endorsement of those websites. The materials at those websites are not part of

the materials for this IBM product and use of those websites is at your own risk.

lx86 and linux on ibm power systems

Documents