Efficient Tuning ofSAP Systems with highBatch Workload usingSAP Central ProcessScheduling by RedwoodandHDS Tuning Manager

2

ContentIntroduction ................................................................................................................................................................................... 3Executive Summary ...................................................................................................................................................................... 4General Problems of System Tuning .............................................................................................................................................. 5

Functional Aspects of Tuning .................................................................................................................................................... 5Technical Aspects of Tuning ..................................................................................................................................................... 5Organizational Aspects of SAP Tuning ...................................................................................................................................... 6Financial Aspects of SAP Tuning .............................................................................................................................................. 6Conclusion ............................................................................................................................................................................... 6

Solution Description ...................................................................................................................................................................... 8SAP EDR Billing and Convergent Invoicing ............................................................................................................................... 8Hitachi Tuning Manager software ............................................................................................................................................ 10SAP Central Process Scheduling by Redwood ........................................................................................................................ 12

Test Setup .................................................................................................................................................................................. 14Details of POC Environment ................................................................................................................................................... 15Logical view of Storage System............................................................................................................................................... 15Storage and Software: ............................................................................................................................................................ 15The Monitoring ....................................................................................................................................................................... 16

Test Execution and Results - 1st POC test cycle ........................................................................................................................... 17Overview ................................................................................................................................................................................ 17Observations .......................................................................................................................................................................... 18Conclusion ............................................................................................................................................................................. 19

Test Execution and Results – 2nd POC Test cycle ........................................................................................................................ 21Overview of changes .............................................................................................................................................................. 21Observations .......................................................................................................................................................................... 21Conclusion ............................................................................................................................................................................. 22

Cost Efficiency ............................................................................................................................................................................ 23Glossary ..................................................................................................................................................................................... 25Contributions............................................................................................................................................................................... 26

SAP ....................................................................................................................................................................................... 26Redwood Software ................................................................................................................................................................. 26Hitachi Data Systems.............................................................................................................................................................. 26

3Introduction

Tuning of large SAP systems can be a cumbersometask as it usually requires many test cycles and thecollection of information from many different sources.In this study, we present a creative approach that helps

minimize this effort for batch driven SAP applicationsusing SAP CPS by Redwood and Hitachi TuningManager software.

4Executive Summary

SAP customers with a large database as it is notuncommon in service industries such as utilities andtelecommunications frequently face the problem ofpoor system performance during batch execution.While installing more hardware is a common responseto this problem, this approach does not always achievethe throughput increases desired despite the outlay ofpotentially sizeable capital costs. The challenge is tofind effective tools to identify and resolve thebottlenecks causing the poor performance and avoidthe additional hardware expenditures.

SAP, Redwood Software and Hitachi Data Systemshave come together to address this issue and todemonstrate how SAP batch performance can bebetter optimized without substantial capitalexpenditure. A proof-of-concept solution wasdeveloped using the following components:

SAP EDR Billing and FI-CA: stand ideal examples oftypical applications for large scale enterprisecomputing widely used in industries like utilities,telecommunications, insurance and others whichtypically have a demand for high-volume billing.SAP Central Process Scheduling (CPS) byRedwood: enables centralized management andautomation of SAP and non-SAP workloadHitachi Tuning Manager software: combined withHitachi Data Systems’ virtualized storage approachprovides full view of the IT infrastructure fromdatabase to the disk in the storage system.

This document presents the results of the proof-of-concept. Its scope has been to simulate a typicalcustomer environment and provideideas/recommendations about implementing a testenvironment that can also be used later for productionbatch execution and monitoring of relevant parameters.It explains how to set up a batch performance test tomake use of SAP Central Process Scheduling forautomation and Hitachi Tuning Manager as a powerfulanalysis tool. It also illustrates how investments can beeither be avoided or better allocated to address otherneeds.

As result of this PoC performance improvements of20% could be achieved by quickly identifiyingbottlenecks caused by non-optimal configuration butwithout purchasing further hardware. The focus has notbeen to specifically show how much performance canbe optimized - this is discussed in various othersources - but to show how automation and monitoringtools can help achieve better performance in a highlyautomated.

As TCO (total cost of ownership) is always aconsideration, this document also illustrates how thetest configuration can be later transferred intoproductive use and how the costs associated withcreating the test environment can be further utilized.

5

General Problems ofSystem Tuning

When setting up a large SAP system, tuning can be acrucial step. When shipped on CD, SAP software mustwork on large and small computers. Therefore, manyparameters are set to a relatively small value, whichpermits the newly created system to work properly froma functional point of view. However, these parametervalues are not well suited for large environments and,therefore, need to be adjusted. Some adjustments,such as increasing the number of work processes, canbe done quite easily, while other settings, such as thesize of memory areas and log files or the distribution oftables over disks, are less easily accomplished.

The best configuration often depends on the way acertain application is used. For example, in deregulatedenergy markets, messages have to be sent to othermarket participants resulting in a higher number ofworkflows than in regulated markets. As RemoteFunction Calls (RFC) are used, the corresponding RFCparameters need to be set differently. Since everycustomer has to find out the best configuration for theirown business scenario SAP has no way to ship fixedsets of ‘good’ parameters to customers, so changesare required.

Functional Aspects of TuningOne of the major challenges of system testing is that itrequires knowledge of SAP, database, and operationsthroughout the testing process and those skills oftenreside in different individuals. This can be only costly ifthese experts do not come from existing staff and theircoordination can often lead to a less than efficienttesting process.

SAP CPS by Redwood can play an important role byremoving the need to employ functional experts duringthe entire testing phase. In SAP CPS, thedependencies between the steps are describedbeforehand within the scheduler framework so thatwhen that batch chain is executed again, all dates areadjusted appropriately.

There are three major advantages to using thescheduler:

Manual errors and resulting system restores can beavoided. When tests are executed by a technicianwho has to enter data manually, errors such asentering the wrong data, resulting in a systemrestore.Each step of the batch chain can be started bypressing a simple button, thereby saving significanttime and avoiding the need to enter a large amountof information.Once created, the batch chain in the SAP CPS canbe part of the daily batch chain, so the investment indevelopment of this scenario is protected.

Technical Aspects of TuningLooking at what can be technically tuned within a SAPsystem, there are 5 main components:

SAP systemDatabase systemOperating systemStorage systemNetwork

SAP tools provide excellent insight into the SAPsystem, a fairly good view of the database system anda glimpse of the operating system, storage system,network and disks. A complementary tool such as theHitachi Tuning Manager can effectively cover thosecomponents where SAP tools are less efficient.Together, the combination of both tools offers aneffective solution to tune the overall system.

Hitachi Tuning Manager effectively covers all areasbelow the SAP component and provides a singleinterface to all information from different sources. Inproduction, Hitachi Tuning Manager continues toproactively monitor the system to detect anydegradation of performance.

6

Figure 1: Combination of SAP and Hitachi TuningTools

Without Tuning Manager, several tools would likely benecessary to address the different layers:

SAPDatabaseOperating SystemNetworkStorage Area Network (SAN)Storage subsystem

Each layer’s tools has its own user interface and itsown way to present data which requires training or anestablished skill set to handle all of them. Using only afew SAP monitors and Hitachi Tuning Managerdecreases the number of tools down to two singleaccess points. As neither SAP nor the Tuning Manageris able to provide complete information, thecombination of the two provides 100% insight and acomprehensive view to manage the environment.

Organizational Aspects of SAP TuningTuning of systems has, as all tests, an organizationalaspect as well. When first results are generally notsatisfactory, tests need to be repeated. Therefore thesystem has to be reset, at times, and testing has to beexecuted in following cycles:

execute testanalyze resultsrestore systemmake changes(backup system)

When the final cycle is complete, the cycle starts overagain. The “backup system” is usually done only if toomany changes must be applied after the systemrestore. The big issue operationally is time. Since test

execution for a realistic scenario may last severalhours, only one test per day is possible. Restore takesa couple of hours as well and techniques like DiskMirroring are usually of little help, since the next testcan begin only once the copy has been finished.Otherwise the storage system will still be busy and cannegatively influence performance.

Once total automation has been achieved, unattendedmeasurements during the night or over the weekendare possible. Thus, the time frame required to performtests can be reduced dramatically

Financial Aspects of SAP TuningTuning efforts are labor intensive and, therefore, canbe costly. During implementation projects, additionalresources frequently need to be hired to perform thetests required for tuning since regular IT employees arecommitted to maintaining the existing environment.

When tuning activities are reviewed in detail, most ofthe time is related to online monitoring and gatheringinformation. The information from various sources thenhas to be mapped and correlated, so only a fraction ofthe time is used for analysis and reconfiguration.

If testing activities are automated, fewer externalresources would be required and the majority of thetime spent can focus on the analysis and specific fix ofthe problem.

ConclusionPerformance tests to tune a SAP system require skillsin many areas, including knowledge of the relevantbusiness processes. Automation and managementtools to simplify processes can saves time, but theiruse must be measured against their license andimplementation costs.

Since automation of tasks will free up time for otheractivities, increased staff productivity or the costavoidance of increased staff needs should be includedin any cost/benefit analysis. This value also extendsinto the production environment as well. This is what isassumed in this proof-of-concept, so the tests fortuning are functional test of the scheduler as well. Thesame applies to Tuning Manager.

Once the setup is complete, the analysis process canused as new problems in productions arise such asperformance degradation after patch application orincreased volume. In this case, the tracing setup canbe easily activated and, as batch jobs usually runduring the night, results are ready for analysis the nextmorning.

7

This is why the proof-of-concept described here is notjust an academic exercise, but useful to all IT staffs to

reduce TCO, improve Time-to-market and effectivelymeet their SLA’s.

8Solution Description

The solution elements used in the POC - SAP EDRBilling and Convergent Invoicing, Hitachi TuningManager software and SAP CPS software by Redwood- are described in greater detail.

SAP EDR Billing and Convergent InvoicingEDR Billing and Convergent Invoicing was chosen forthe POC since with this high volume application, evensmall improvements in performance and system setuplead to considerable savings on hardware. The secondreason for selecting this relatively new application overother mass billing applications is that it is used andpromoted as a whole in different industries such as:

TelecommunicationsElectronic toll collectTravel and TransportationPostal servicesOnline stores

Parts of this application -from payment run onwards-are used since a long time in further industry solutionsof SAP:

UtilitiesInsurancePublic sector

Especially in the latter industries the large scaling andthroughput capabilities of the FI-CA high end solutionhave been proven since long, to which EDR billing andConvergent invoicing now have been added, thusmaking it a universal billing platform. To have asolution like this was a precondition for this test so thatit would not fail for poor scalability.

Another advantage of choosing this solution is thebusiness process itself: it describes the creation of abill from the incoming sale to the collection of themoney, a process essential to all types of companies ofall sizes.

As shown in the description below, the solution coversthe whole process, from the arrival of single recordsrepresenting a chargeable item or action, such as aphone call on billing, to invoicing and bill print down to

the FI area, where received payments can be trackedand dunning can be issued.

Therefore, it is the ideal high volume billing solutionfrom SAP for large enterprises.

Process descriptionThe following details the separate business processesassociated with FI-CA.

Figure 2: FI-CA process

Creation of return lotand payment lot

Simulated step, usuallyperformed in another

system

EDR upload

EDR Billing

Creation ofEDRs

ConvergentInvoicing

Payment run

Payment filecreation

Upload ofpayment and

return lot

Dunningproposal

Dunningactivity

Correspondenceprint

Regularprocess step

9

EDR Creation (Activity ZEDC)EDRs are usually created in another system and thenuploaded into the SAP system. For the test, they havebeen created in this system itself, which eliminated theneed for another system to be set up.

Transfer of Event Detail Records (Activity ZEDU)Within this activity, event detail records (EDRs) areuploaded. Each EDR corresponds to a sales or servicedelivery. It is assumed each EDR represents a phonecall but it could represent a sales record from an onlineshop or a letter sent as well.

Creation of billing ordersEach relevant billing account gets a billing order whichis the basis for the next billing step.

Billing in Contract Accounting (Activity 8000)In this step, the EDR records are grouped togetheraccording to certain criteria defined in configurationsettings. Each grouping criteria produces one billingline which will be invoiced later.

Invoicing in Contract Accounting (Activity 2600)The billing lines created in the previous step are nowinvoiced. Taxes are added and the new due amount iscalculated, taking into account the current balance ofthe corresponding contract account.

Payment Processing (Activity PAYP)In this step, open items are closed if the businesspartner has opted for direct debit. A list is prepared issent to the banking institution which then collects themoney from the business partner’s bank account.

Payment Media Generation (Activity PAYM)This step simulates the generation of lists with directdebit customers’ due amounts to be sent to the housebanks.

Correspondence Print (Activity COPR)In this step, all correspondences to the businesspartners are printed. In the test model, the followingletters are printed:

One invoice for each business partnerOne dunning letter for the 10% of business partnersnot paying on time

Bank Return File Generation (Activity ZCMC)Once the lists of due amounts for direct debitcustomers is created, it is passed to the bankinginstitution that collects the money. However, there maybe customers unable to pay, and others that do pay,but are not direct debit customers. For both types, thebank generates a list that will be uploaded into thesystem later to enable the dunning activities. Thegeneration of these lists is usually done by the bank,but, for the test, it has been simulated.

Upload of Bank Return FilesThis process is not listed above because it is usuallyan online activity and not part of the batch chain.

Dunning Proposal (Activity MAHN)The dunning proposal creates a list of businesspartners having overdue open items. It has been splitfrom the dunning activity run in order to allow removalof selected business partners from the dunningproposal list before starting the dunning activity run.This is necessary where, for example, the company isin contractual negotiations with a business partner.

Dunning (Activity MAKT)The business partners on the dunning list created atthe step before are dunned. In the test model, threeactivities are carried out:

A dunning letter is createdA dunning fee is addedA payment form is added to the dunning letter

10

Hitachi Tuning Manager softwareIntroductionHitachi Tuning Manager is an advanced reporting andanalysis tool, which enables performance monitoring,troubleshooting and capacity forecasting for Hitachistorage systems. Hitachi Tuning Manager is anautomated, intelligent and path-aware storage resourcemanagement tool that maps, monitors, analyzes andreviews storage network (SAN) resources from thedatabase/ File System to the storage device. Itprovides the end-to-end visibility required to isolate anddiagnose performance bottlenecks with a focus onbusiness Databases, such as Oracle®, Microsoft SQLServer, Microsoft® Exchange and IBM® DB2®.Through a Web-based interface, Tuning Managersoftware facilitates predictive trending to forecaststorage requirements and meet service levelagreements. The alert facility in Tuning Managerprovides for proactive monitoring and allows forcorrection of potential problems before they occur.

Performance problems can have multiple impacts to anorganization:

User complaints of slow or insufficiently availableapplicationsDifficulty identifying bottlenecks impacting applicationperformance.Excessive time to diagnose and resolve problems

These problems have a direct impact on the typicalobjectives of IT organizations:

Service Level ObjectivesUnmet Services Level Agreements (SLAs)Inability to determine whether SLAs are being metMissed time windows and IT objectives

Cost reductionReduce incremental storage purchases to meetperformance demandsGet the most out of existing storage systems

Planning and forecastingLack of a baseline for establishing storage growthpatterns and future performance requirements(capacity and performance planning)Inability to forecast requirements

Hitachi Tuning Manager software can providesubstantial assistance in addressing the performanceproblems and thereby have positive impact on the keyIT objectives.

Business Benefits of Tuning Manager

The Hitachi Storage Command Suite enables aServices Oriented Storage approach by replacing thetraditional silo approach of separate, isolatedmanagement tools with a management framework thatimproves operational efficiency and resilience.

Figure 3: Tuning Manager in a wider screen

The Architecture provides a suite of Services OrientedStorage solutions (center) that enable customers toalign their storage resources (bottom) with theirbusiness requirements (right), applications (top),business measurements (left) to allow IT organizationsto provide greater value to their business.Services Oriented Storage provides comprehensive,integrated storage resource management, tieredstorage and business continuity software solutions,allowing organizations to align their storage withapplication requirements based upon metrics includingquality of service (QoS), service level objectives oragreements (SLOs or SLAs) and recovery timeobjectives or recovery point objectives (RTOs orRPOs). Integrated with Hitachi Tiered StorageManager and Hitachi Device Manager, TuningManager enables performance management andcapacity utilization on all Hitachi storage systems andSUN branded 99xx series systems.

Hitachi Tuning Manager specifically provides thefollowing:

Improves Business Agility and Helps ManageGrowth

Speeds the identification and resolution ofperformance or capacity problemsMonitors performance proactively; provides alerts ofpotential issues before they become problemsEnables resource analysis; drill down to problemarea for quick identificationSupports faster application deployment throughplanning and forecasting of adequate storageresources

11Enables Operational Excellence

Improves IT processes through enhanced reportingand forecastingRealizes performance excellence by monitoringresources from the application to the storage platformDelivers automated multilevel alerts when utilizationand/or performance reach critical thresholds, thusdetecting failures before they occurMaintains performance excellence by identifying thesources of bottlenecks

Mitigates Risk and Increases Efficiency

Enhances ability to meet SLAsEliminates storage capacity overrunsPrevents outages via advanced forecasting andmultilevel alerts

Reduces Operational and Capital Costs

Enables more effective use of administrativeresources through better forecasting and monitoringEmpowers personnel to make better decisionsImproves planning processes; allows better allocationof storage resourcesEnables more storage resources to be managed perperson than ever before

Hitachi Tuning Manager Feature Highlights

The Hitachi Tuning Manager benefits outlined aboveare achieved through the following:

End-to-end Storage Resource Management

Provides Mapping and reporting on storageresourcesDatabase table space data fileMultipath software and switchStorage-port LUN parity groupMonitors SAN-fabric performanceMonitors database and storage system responsetimesProvides advanced monitoring and reporting ofDynamic Provisioning pools and volumesEnables creation of storage management scripts forrepetitive tasks and automation

Performance and Capacity Planning

Monitors, reports and forecasts on both capacity andperformance metrics

Analyzes storage requirements from the applicationperspective; performs both linear and nonlinearforecastingEmploys multilevel capacity and performance alertsfor proactive storage resource monitoringGenerates predefined and easily customizablestorage reports

In this proof-of concept, only the basic functions of thevirtualized Hitachi Universal Storage Platform V wereused. Other Hitachi SOSS software offerings areavailable for additional improvements and flexibility:

Hitachi Tiered Storage Manager software — enablesIT administrators to easily and non-disruptivelymigrate data to match application QoS requirementsto storage assets.Hitachi Virtual Partition Manager software* —enables IT administrators to create cache partitions,allowing them to dedicate portions of cache tobusiness critical applications (SAP production) orlimit cache available to noncritical applications (SAPTest).Hitachi Server Priority Manager software* —prioritizes server port access; allows a single storagesystem to limit I/O activity of noncritical applicationservers; and, prioritizes production serverperformance to meet SLAs.

* Included in the Hitachi Basic Operating Systemsoftware on the Hitachi Universal Storage Platform™ Vand Hitachi Universal Storage Platform VM.

Prerequisite for Hitachi Device Manager SoftwareRequired for Tuning Manager software, Hitachi DeviceManager Software is an optimized storagemanagement platform for centralized management,configuration and monitoring of Hitachi storagesystems. Tuning Manager leverages storage systemconfiguration information from Device Manager,including metadata, to provide host-to-storagecorrelation for performance and capacity data.

12

SAP Central Process Scheduling by RedwoodSAP Central Process Scheduling by Redwood providesthe ability to efficiently automate and monitor businessprocesses across the enterprise. Organizations thatneed to successfully manage complex processes,requiring interaction between numerous individualprocedures in different systems environments andacross business units, can benefit from the capabilitiesthat SAP Central Process Scheduling delivers.

By providing automation at the heart of SAPNetWeaver, SAP Central Process Scheduling allowsSAP customers to manage the background processingrequirements of compound applications, resolvingcritical integration issues and optimizing the utilizationof IT resources. SAP is building on this automationframework by embedding capabilities that SAP CentralProcess Scheduling delivers into new SAP solutions tofurther reduce total cost of ownership, improve ease ofmanagement and enable the business to drive theunderlying IT processes.

Delivering Business NeedsSAP applications are designed to provide organizationswith the most flexible and efficient environment withwhich to manage their critical business processes. Withan increased need to meet the demands of a globalcustomer and partner base, many organizations mustsupport real time, 24 x 7 operations around the globe.This can lead to conflicts between the needs of thebusiness user and efficient processing of theunderlying work needed to support customers.

SAP Central Process Scheduling allows the businessto meet these challenges and provides effectivemanagement, optimum performance and improvedservice levels, enabling the needs of the business todirectly control how and when background processingtakes place. Through an advanced capability thatmanages process submission based on real timesystems events, SAP Central Process Schedulingallows control to be based upon the dynamicallychanging business environment, rather than havingartificial boundaries imposed by typical time- andcalendar-constrained IT process scheduling tools.

Enabling Business Driven AutomationThe practical benefits of this powerful new approach ofbusiness driven automation are demonstrated throughunique integration capabilities between SAP CentralProcess Scheduling and SAP applications. Forexample:

SAP ERP Financials - SAP Central ProcessScheduling works seamlessly with SAP ERP Financialsand its graphical financial close management tool, theSAP Financial Closing cockpit, to deliver a single pointof control to manage and coordinate period end closeacross the enterprise, providing:

Better consistency, transparency and reliability ofresults delivered by proper sequencing of processesacross all entitiesDecreased latency in the overall close process due toautomated triggering of closing tasksReduced dependency on functional spreadsheetsImproved internal control and reporting of ITprocesses related to financial close

SAP NetWeaver Business Warehouse (BW) - SAPCentral Process Scheduling allows SAP NetWeaverBW access to information outside of the SAPenvironment providing many benefits for accurate andup-to-date data analysis, including:

Centralized analysis of corporate data for accuratedecision makingAutomatic restart of BW processing at the point offailure, improving throughput and efficiencyEasier management of multiple BW process chainsVisualization of BW processes

SAP Industry Solutions – By managing the very highvolume of workload demanded by utilities applicationsSAP Central Process Scheduling ensures optimumthroughput and:

Improved management of long running mass activityprocessesVisibility of child processes whenever they occurManagement of all processes including childprocessesMaximized use of available processing resourcesImmediate notification of failures

These are just some examples of the benefits that SAPCentral Process Scheduling delivers for SAPapplications. Its unique position at the heart of SAPNetWeaver ensures that customers can be certain ofsimilar benefits across the SAP and non-SAPlandscape.

Improved EfficiencyAs well as enabling this new approach to business-driven automation, the central point of control providedby SAP Central Process Scheduling reducesadministrative overhead for the IT department byremoving the need for the multiple tools previously

13used to manage processes across differentenvironments. Productivity is markedly increased bysimplifying management of the complex IT workloadsgenerated by business processes.

With its comprehensive automation rules, SAP CentralProcess Scheduling manages both routine andpredictable workloads as well as the ad hoc tasks thatcan be generated on the fly by packaged andcustomized applications. Both planned and unplannedworkloads are treated as system events andrelationships are built between these events to createdependencies to provide a fully dynamic schedulingand process automation environment. Any changesthat are required can be accommodated at any pointup to execution, enabling the IT organization to betotally flexible in meeting quickly changing businessprocessing needs.

Critically, SAP Central Process Scheduling responds inreal time to system events originating in both SAP andnon-SAP environments. The ability to manage andmonitor across a broad range of core SAP applicationsand non-SAP environments enables customers tochoose (or develop) business applications andeffectively assimilate them into the managed SAPprocessing environment.

SAP Central Process Scheduling provides an intuitiveweb-based graphical user interface that is integratedinto the SAP User Management Engine (UME).Customers can be assured of secure and centralizedoperation and administration adhering to corporatesecurity and compliance standards.

Business and Operational BenefitsOrganizations are faced with many challenges, bothbusiness and operational, as they attempt to managegrowing IT workloads without increasing availablecomputing and staff resources. To help organizationsmeet these challenges, SAP Central ProcessScheduling offers the following advantages:

Increased agility through real time response tosystem events, enabling organizations to reactrapidly to changing business conditionsImproved service level delivery through reducedprocess latency, automated error detection, reducedrisk of errors from manual intervention and seamlessintegration between application processesImproved return on IT investment throughoptimization of processing workloads acrossavailable resourcesEstablishment of a scalable, future proof automationinfrastructure deeply integrated with SAP NetWeaver

and Solution Manager, and enabling the latestgeneration of SAP application solutionsIncreased productivity through simplified andautomated management of application and ITprocesses from a single point of control

For companies implementing enterprise serviceoriented architectures, SAP Central ProcessScheduling natively enables a service orientedapproach to automation. Through support forenvironments based on Java and Web applicationservers, it facilitates integration by providing a commonset of reusable, immediately available services forprocess automation. SAP Central Process Schedulingserves as a central infrastructure for orchestrating andintegrating components for legacy, existing and futureapplications. Integration with the SAP SolutionManager Application management solution and thebroader SAP software systems managementenvironment ensures that SAP Central ProcessScheduling easily accommodates corporate ITmanagement frameworks.

The Benefits of PartnershipWith SAP Central Process Scheduling, customers areleveraging the combined expertise of both SAP andRedwood Software, gained through serving more than30,000 customers worldwide in over 150 countries.Both companies have widespread experience withorganizations of all sizes, not just large enterprises.SAP also has decades of experience helpingorganizations improved their business fundamentals,business processes, and overall business results.Implementing the scheduling application initiates real-time, event-driven scheduling and process automationfunctionality that will produce a rapid ROI, lower yourtotal cost of ownership (TCO), and help you optimizeyour business processes and meet constantlychanging customer and market requirements.

14Test Setup

The first step of the proof-of-concept is to do astandard setup of the SAP system first -- installing thesystem without any customization. This reflects howmost customers without much SAP experience willinitially see their installation. The additional software isthen implemented and configured and the scheduler isset up.

Once functional correctness of the processes areensured, one million master data records aregenerated. The main purpose of these tests is todetermine the maximum throughput for each job to beprocessed. This is necessary so that any extra timeavailable at the end of a run can be identified toaddress unforeseen problems that can occur such aswhen system shutdowns are needed to apply patchesor where process steps ending in errors need to berepeated.

The key to increased throughput is to run multipleprocesses in parallel, each working on a different set ofobjects. However, throughput usually does not linearlyincrease with the number of concurrent jobs sincethere is competition for central resources, such as alock handler. In fact, it can even decrease the overallthroughput. Therefore, it is important to determine theoptimal number of jobs for each batch process step.

The strategy to do this is fairly straightforward. In a firstrun for each process step, one job is launched perapplication server. The run is then repeated with twojobs per application server, then with four and so on.This gave the jobs listed in the table below:

NAME OF

TEST

JOBS PER

APPLICATION

SERVER

NUMBER OF

INVOICES

T1 1 100,000T2 2 100,000T4 4 250,000T8 8 250,000T10 10 250,000T12 12 250,000T20 as1b02: 20, all others

12250,000

The last test (T20) was run since server as2b01 hasmore CPUs than the other servers and can processmore jobs.A graph can be produced for each mass activity thatshows how far it is from optimal scaling and whichactivity most needs tuning.

Throughput vs Number of Jobs

0

20

40

60

80

100

120

140

160

0 10 20 30 40 50 60parallel jobs

Obj

ects

/ se

cond

Figure 4: Throughput versus Number of Jobs

One crucial activity within the setup is to consolidatethe information generated from the separate SAP andHitachi software. The manual procedure requires alook up in SAP to determine start and end times ofcorresponding batch jobs. This timing information isthen fed into the continuously running Hitachi TuningManager to extract the relevant measurements into aspreadsheet. Finally, the values are then convertedinto graphs that can be used to compare results fromdifferent runs.

Within the POC, Hitachi Data Systems GlobalSolutions Services (GSS) provided a script, whichenabled extraction of all relevant performance datafrom Hitachi Tuning Manager into CSV files. Oneparameter used for calling this script tells the exacttime window which enables focus on the relevant datafor analyzing.

Since the same time window is used to getperformance data from the SAP system and fromTuning Manager, a legitimate comparison of relevantdata is possible.

Microsoft® Excel® templates have been created whichcan read all output files from SAP and the TuningManager and convert them into graphics for ease ofvisualization.

Due to the capabilities of SAP CPS, the script alsoprovided extraction of data from Hitachi TuningManager as well as the Excel templates. This data

15could then be called automatically and for each step inthe batch chain the following sequence was executed:

Get start timeLaunch SAP monitorLaunch batch jobWait until batch job is finishedStop SAP monitorGet end timeExtract from Tuning Manager to .csv files using startand end timeCall Excel templates to make graphs

As this is accomplished automatically, the test team willalways find all relevant data at the end of each step, inone place, and well separated from information of othersteps.

Since the entire setup was automated including thecollection of data, the whole batch chain can now rununattended allowing for runs overnight, duringweekends, and during weekdays without staffintervention.

Since the tests are iterative process often with onlysmall parameter changes made each time, a full copyof the initial data is kept on a second Disk Mirror. Byhaving this data always available and using theautomation from Redwood’s SAP CPS, one click wasall that was required to reset the whole SAPconfiguration and start the next test cycle withimproved parameters.

Details of POC EnvironmentSAP: ERP 6.0 including IS-T 2005 EhP 2.0 Patchlevel 03MS SQL Server 2005 SP2, MS Windows Server 2003SP2

Figure 5: POC environment

HOST PROCESSOR MEMORY FUNCTIONRWAC-001

8 x Xeon2,83Ghz

16 GB DatabaseServer/ASRV

RWAC-003

4 x Xeon 2,83GHz

8 GB ApplicationServer

RWAC-101

8 x Xeon 2GHz

8 GB ApplicationServer

RWAC-102

4 x Xeon 1,5GHz

4 GB ApplicationServer

RWAC-102

4 x Xeon 1,5GHz

4 GB ApplicationServer

Infra VMware 2CPU

4 GB CPS Server

TuningManager

VMware 2CPU

4 GB PerformanceMeasurement

DeviceManager

VMware 2CPU

2 GB Admin &Configsoftware forUSP-V

Logical view of Storage System

Figure 6: Logical view of Storage System

Storage and Software:This POC were running on a Hitachi Universal StoragePlatform™ V with a small configuration of

26 Disks with 146GB each and 30 Disks with 300GBeach24GB Cache MemorySQL-Server Tuning Manager®

1st POC test cycle:3 x RAID-5 (7+1) concatenated Groups; one singlebig Volume containing everything

16

2nd POC test cycle:5 x RAID-5 (3+1) separated Groups for SQL ServersDatabase Files1 x RAID-1+0 (2+2) for SQL-Server “log” and “temp”files

Focus on relevant DataAs mentioned earlier, Hitachi Data Systems GlobalSolutions Services provided a script for this Proof-Of-Concept, enabling extraction of all relevantperformance data from Hitachi Tuning Manager into.csv files. One parameter which is used for calling thisscript tells the exact time window. This helps focus onthe relevant data for analyzing. Since the same timewindow is used to get performance data from the SAPsystem, a real comparison of relevant data is possible.

The MonitoringAll tests had to start from the same database and datastructure to effectively compare test results. Therefore,Hitachi ShadowImage® software was used to freezeone copy of the initial data. After the completion of onetest run, the data from ShadowImage® was reversesynchronized to the test volumes so the next row oftests could start exactly on the same base content anddatabase structure.

The results were viewed from SAP’s monitoring Toolsand Hitachi Tuning Manager. Below are some samplepictures to better outline what was captured:

SAP SM37 (sample view)

Figure 7: SAP SM37

SAP STAD (sample view)

Figure 8: SAP STAD

Hitachi Tuning Manager (sample view)

Figure 9: Tuning Manager

17Test Execution and Results- 1st POC test cycle

OverviewEach of the steps above starts with distributing theobjects to be processed into intervals that will behandled by jobs running concurrently. In transactionSM37, this looks as shown in the figure below:

Figure 10: SM37

From the above results, the following graph is derivedusing Microsoft ®Excel®:

Single Job Duration 800020080926T4

Job_

001

Job_

002

Job_

003

Job_

004

Job_

005

Job_

006

Job_

007

Job_

008

Job_

009

Job_

010

Job_

011

Job_

012

Job_

013

Job_

014

Job_

015

Job_

016

0

200

400

600

800

1000

1200

1400

Job

Avg

= 9

74s

Max

= 1

011s

b01as1_B01_81b01as2_B01_82b01as3_B01_83b01as4_B01_84

Figure 11: Single Job Duration

What can easily be seen here is that there was almostequal runtime of each of the jobs. This is importantbecause if some of the jobs have a longer runtime thanothers, which would delay the process (which can onlycontinue once all of them are finished).

The two figures above were from one execution of thebilling (8000) with 16 concurrent jobs. In order to get a

curve to display the level of throughput versus thenumber of jobs, this exercise has to be repeated with adifferent number of concurrent jobs.

This task (which, in this case, consists of 8 out of the11 execution steps) normally requires the full timeattention of a single person. Scheduling a step is about3-4 minutes and with runtimes of a step between 5minutes and 1 hour, it is difficult for a person to befreed up for other work. Furthermore, the execution islimited to regular working time, as no step ran longerthan one hour. So two almost two third of a day notests can be performed.

Therefore, SAP CPS’ benefits were clearlydemonstrated as it permitted runs 24 hours/7 days aweek without human interference. In fact, most of thejobs were scheduled over the weekend so as not tointerfere with other work.

The Automation with SAP CPS

With the help of CPS, it is possible to chain all theprocesses and their parameterization together torepresent it as a single process.

Figure 12: One Benchmark process (running WU, T1,T2, T4, T8, T10, T12, T20)

Figure 13: Example of MA-Parameterization andexecution

18

Figure 14: SAP Defined program

For each process executed, detailed information isavailable.

Figure 15: SAP Job Submit

Parameters can be passed from one task to the next.

Figure 16: SAP Job Submit

Storage InfrastructureThe Main Database Server RWAC-001 was directlyconnected via two 2Gb Fibre Channel adapters to theUniversal Storage Platform V. Two Ports (CL1-F andCL2-F) were used for load balancing. The Architectureof a Universal Storage Platform can be seen as a

Storage Cluster. The Storage Port from controller CL1-F belongs to the one side. CL2-F belongs to the otherside of this Cluster. Via internal MESH network,separated Cache- and Control-Memory, the storagesystem will always find the optimal route to eitherinternal storage (used in this POC) or externalconnected Storage System from Hitachi or othervendors.

The first test was started with a configuration that couldalso be found in real time at the customer side. 2.7TBof storage was provided by the storage system andeverything was installed on one single LUN (LogicalUnit = Logical Disk, representing on one or more slidesof physical RAID Groups):

The smaller disks were concatenated in a sequentialrow. This provided one single LUN (logical Disks)with the size of 2.7TB (~2700 GB) to the Databaseserver.Since only one LUN was presented to the Server viatwo FC paths, only two single I/O queue werepossible from Server to Storage.All concurrent Database Files (DB and LOG’s) wereplaced on this LUN. SQL Server “temp” files wereplaced on local disks.

ObservationsAs previously described, each Benchmark was startedwith different parallelism. T1 with one job, T2 with 2 joband so on.

The graph below shows the differences in runtimes:

T1 was running more than 5 hours, processing 100,000billings. T8 was running less than 4 hours, but T20took slightly longer, processing 250.000 billings

The first finding was that 8 jobs in parallel would be theoptimal setup for the given configuration (but we weretrying to achieve the optimal configuration, not theother way around!).

Figure 17: Overall Job runtimes in first test cycle

The mass activity 8000 (Billing in Contract Accounting)was chosen as representative of the whole benchmark.T1 (with 100,000 billings) was the longest run, T4 (with

19250.000 billings) brought some benefits and T20resulted in huge improvements.

Figure 18: extracted 8000 Job runtimes

Since it is difficult to interpret the runtime data due tothe different amount of data processed (100,000 in T1and T2, 250,000 for the rest), they were converted intothe throughput curve shown below:

Measured Throughput Billing

0100200300400500600700800900

1,000

0 10 20 30 40 50 60Concurrent jobs

Bill

s / s

econ

d

Optimal Throughput20080926

Figure 19: Throughput of "8000" mass activity

Figure 19 shows the somewhat typical behavior of asystem for the throughput of any batch application withconcurrently running jobs. At the beginning, (T1, T2and T4 with 4, 8, 16) the scaling of concurrent jobs isalmost linear with an only moderate deviation from theoptimal (linear) scaling curve (the straight line in red).As more jobs on the system cause resourcebottlenecks that limit the throughput, it starts to decline.Looking at the rightmost three measurements, T20 with56 concurrent jobs has a lower throughput the T12 withonly 48 concurrent jobs.

Tuning Manager gives a full view of the complete ITinfrastructure chain, not only from the database downto the disk drives, but also the Load generators’ CPUutilization (see figure 20).

The simplified view below was created with the help ofRedwood’s CPS software. Providing the timestamps ofthe mass activities, these timestamps were fed into ashort script, which exactly extracted the requiredinformation from all parts of the chain from TuningManager® to external “csv” files. These wereconverted to “png” graphics.

Remark: In this early stage of the POC, using theextract to “csv” files helped to find the focus areasrather than recording all possible parameters (severalhundred) from Tuning Manager®.

Figure 20: Performance view in whole chain fromDatabase to Disk

Note: 2 x “n” in above figure is used because each ofthe two Fiber Channel (FC) connections is reportedseparately, but I/O at Database-Server is representedin total.

ConclusionIn the given configuration, two bottlenecks wereidentified which were preventing a more linear scalingof the parallelism of jobs (T1 – T20):

Bottleneck #1: All load generators were running closeto CPU limit.

Bottleneck #2: Based on the single 2.7TB volume, thedisk groups (RAID-5 with 7+1 config.) were in a

With the other batch activities showing similarbehavior, the major question is:

WHERE ARE THE BOTTLENCK(S)?

concatenated sequential row. On average, thefollowing I/O’s were seen:

1000 Read I/O’s with 25% cache hit rate 500 Write I/O’s with 80% cache hit rate

Read activity with only a 25% cache hit rate meansI/O’s (the access to the Database content) wererandom access (not sequential). The intelligent “read-ahead” algorithms of the storage system, which helpsto lower response times, can do little to assist becausethe data anticipated being the next read is not used.Instead, the data requested from the database waslocated somewhere else on the physical disk layout.75% of the reads (=750 I/Os) were coming directly fromthe physical disks.

Write activity with an 80% cache hit rate means thatthe cache, normally used as the first and fastestdestination for writes, was not available 20% of thetime or 100 I/Os. Those I/Os must be written directly todisk (flushed from cache) which increases overallresponse time.

The physical 850 I/O’s that were either being read orwritten to the disks fit to the I/O capability of one 7+1RAID5 Group (~ 100I/O per Disks x 7); this proofs, thatthe measured values makes sense.

To improve the runtimes of the benchmark, the disklayout has to be modified to deliver better I/Oresponse time for a random application. In addition,the CPU power of the load generator could beincreased, however every activity below 90% CPUusage should be able to scale more linearly if theunderlying infrastructure is set up correctly.

21Test Execution and Results– 2nd POC Test cycle

Overview of changesBased on the results of the first POC test cycle, theconfiguration of the second test was modified to takeinto consideration the random access requests of theMicrosoft SQL Server.

The database files were spread over 6 independentLUN’s of the Universal Storage Platform V.

The database LUN’s were now residing on 5 diskGroups, each with 4 disks (5 x RAID-5, 3+1)The database logs (these are SQL servers’ “Journalfiles”) were placed on a separate RAID-1+0 Groupdue to 100% write activity.

Additionally, the SQL temp files were moved from thelocal DB server disks to the RAID-1+0 disks on theUniversal Storage Platform V.

During the POC, a new version of Hitachi TuningManager (version 6) became available. This wasinstalled to ensure that scripting from the first test couldbe used, even if product updates had occurred. Theresult was positive and no scripting changes wererequired.

New features in Tuning Manager version 6: TheCorrelation wizard

Figure 21: Tuning Manager Correlation Wizard (samplepicture)

The new version enables Hitachi Tuning Managergreater capabilities to be used in highly complexenvironments by providing a correlation wizard andother new features for displaying results for ad hocanalyses or when a direct comparison with a previousrun or previous date is desired.

ObservationsDuring the second execution, the same pictures andgraphs were generated as in the first. As can be seenbelow, the runtime distribution of the single jobs is stillalmost equal. However, the Excel® graph indicatesbetter throughput as both maximum and averageruntime are lower compared to the first test cycle.

Figure 22: SAP Job Overview

Single Job Duration 800020081212T4

Job_

016

Job_

015

Job_

014

Job_

013

Job_

012

Job_

011

Job_

010

Job_

009

Job_

008

Job_

007

Job_

006

Job_

005

Job_

004

Job_

003

Job_

002

Job_

001

0

200

400

600

800

1000

1200

1400

Job

Avg

= 9

30s

Max

= 9

86s

b01as1_B01_81b01as2_B01_82b01as3_B01_83b01as4_B01_84

Figure 23: SAP Job Duration

Note: No additional hardware was purchased, onlythe configuration was changed based on thefindings of the first test.

22Measured Throughput Billing

050

100150200250300350400450500

0 10 20 30 40 50 60Concurrent jobs

Bill

s / s

econ

d

2008092620081212

Figure 24: Measured Throughput BillingBlue curve: Scaling from POC 1, Green curve: Scalingfrom POC 2

From this figure, one can see the success of themeasures taken. First, there are no improvements withless than eight jobs in parallel. This is expected sincefewer jobs cannot exhaust the available resources.

As the job numbers increase, the improvement gainsbecome higher and, interestingly, the scalability issomewhat restored. T20 clearly shows a higherthroughput than T12, which was not the case before.

However, as the new curve starts to deviate from theoptimal throughput line (not on graph, but linearlyextrapolated from the first three measurements), newbottlenecks are now in play. Addressing thesebottlenecks are beyond the scope of this POC butcould be accomplished with the benefit of additionalanalysis using Hitachi Tuning Manager.

Improvement

-20

-10

0

1020

30

40

50

6070

80

ZEDC ZEDU 8000 2600 COPRMass activity

[%]

T4 (16 concurrent jobs)T20 (56 concurrent jobs)

Figure 25: Performance Improvement by percentage

This graph shows the improvements in percentageafter the tuning changes have been made. Onlyconvergent invoicing (2600) shows worse results andthis was traced back to a change in SAP where theway invoices were distributed across the jobs waschanged that generated a higher CPU overhead.

A slight improvement was reached for 16 concurrentjobs and a large improvement for 56 concurrent jobs.The dramatic improvement for ZEDU (EDR upload)and COPR (correspondence print) are related to theheavy insert and update activity they perform on thedatabase with the new layout.

Looking at the complete process, the gain for T20 isalmost 1 hour reducing clock time from 5 hours to 4hours, an overall improvement of 20%.

ConclusionThis Proof-of-Concept clearly demonstrates the valueto a somewhat typical customer solution of anautomated approach using complimentary tools:

Up to 20% higher throughput achieved withoutadditional hardwareEffective identification and resolution of bottlenecks

All tests performed automatically without staffintervention; no overtime/weekend shifts required toexecute the testNo disturbance or outages to other system users

It should be noted that effective tuning changes wouldhave been impossible to find the within SAP tools only.Certain hints would have been visible in the databasestatistics, but it would require a lot knowledge to tracethis back to the root cause.

23Cost Efficiency

As companies continue to look for ways to savemoney, the big question is today, why invest in settingup an environment like this one? How will this pay off?

Here are the ongoing issues that need to beconsidered:

Frequent changes to the production applicationHardware exchanges/upgradesOperating system upgradesDatabase system upgradesSAP operational upgradesPatches applied/New functionalities implementedAcquisitions’ systems need to be merged and canincrease volumeHardware failuresProcesses in production systems have to be startedand controlled

Each of those listed above have an inherent risk topotentially impact performance, so the applicationmust, at least be checked and probably tunedrepetitively to maintain the current level ofperformance. The costs to address these ongoingissues in the traditional manner of more hardware andhired expertise must be weighed against theacquisition of software and establishing repeatableautomated processes. The value of the automatedapproach clearly increases as customer complexityand the amount of ongoing issues increase.

Executing a chain manually requires approximately 55minutes (11 times 5 minutes per step) plus frequentchecks to determine when each step is finished andthe next one can be started. Additionally, certainparameters have to be re-entered to match those ofprevious steps. If they are not, the step may fail or, in aworst case scenario, the system has to be restored.This risk can be completely eliminated by using SAPCPS because all of the dependencies are handledtherein and even if the SAP admin initiates the jobs,costly manual errors are eliminated. Furthermore,investments in the automation scripts used by SAPCPS are never lost. They can be reused over and overagain avoiding manual execution in the future.

Productivity can almost be doubled. Usually, one day isspent to run the jobs and the second to analyze andmake changes. Using SAP CPS Analysis, changes are

made during the day while execution takes placeduring the night. The hardware infrastructure beingused for the testing, therefore, can be productivelyutilized for other purposes during the day

Hitachi Tuning Manager software also providessignificant benefit in reducing the time for problemidentification. If there are hardware bottlenecks, thesystem can be often stuck for 30 to 60 seconds. Manyusers will now start calling the helpdesk. When theproblem is finally traced back to a “system issue”, itrequires time to notify the SAP basis admin in charge.Further time is spent getting in contact with thedatabase administrator or the person in charge of thefile systems and/or hardware. This can take up to 15minutes or more by which time the problem may nolonger be present and its cause impossible to isolate.This also means the risk of a similar problemreoccurring and impacting production remains high

It becomes extremely difficult to find the root causewithout a comprehensive view. Many log files have tobe searched and certain variables, such as memoryconsumption, have to be checked manually step bystep. However, these findings can be invalid sinceworkloads and its variables can change quickly,making it difficult to compare SAP performance valuestaken 15 minutes after the problem to those that werepresent at the time of the problem itself.

Watching for the problem to occur again can beimpossible for random problems and end users cansuffer several bouts until finally the reason is identified,often by a combination of analysis and guesswork.

The two-step approach of a constantly running TuningManager is very valuable. While detailed information isgathered every minute, aggregation is done only every24 hours. This permits system administrators to haveaccess to detailed system parameters even hours afteran incident, where other tools show in best case somehourly averages.

After 24 hours, Hitachi Tuning Manager creates hourlyaverage values which effectively reduce the amount ofdata required for long-term monitoring.

The monetary value to each customer is, of coursegoing to be different but assuming on a large,

24$1,000,000 environment a 20% increase in throughputvalues already $200,000.

One thousand employees stuck for two minutes, oncea day, due to unidentifiable system issues cost nearly$1,000 if an average salary of $30 per hour ispresumed. However, systems issues that occur threetimes a day with 5000 employees working on it end upcosting $15,000 every day until the problem isresolved. This can also have significant impact onSLAs and the associated fines/bonuses to the ITorganization when the commitments are not met.

Night and weekend shifts can become expensive aswell. Some countries require always at least twopeople on each shift for safety reasons at higher salaryrates. If all steps are launched by hand, one person willbe required to do the job, whose salary could be saved.

Bringing in experts from the outside has twodisadvantages. They usually take time to become

familiar with the environment lengthening the time todetermine the details of the problem and it can beexpensive, especially when travel is included.

The figures provided from the POC may not apply toeveryone and, as stated before, the price of the SAPCPS and Hitachi Tuning Manager software has to bebalanced against these savings.. However, In thelong term, the initial costs to set up the scheduler andto implement Tuning Manager are minimalcompared to the potential costs of an outage,additional staffing, and expertise. Furthermore, thedaily monitoring and scheduling become easier andthe repeatable benefits help to create a more efficientenvironment where problem resolution and testingrequire less time and minimal expert involvement.

25Glossary

CPS Central Process Scheduling(Redwood)

DB DatabaseDiskMirror a.k.a. as RAID1EDR Event Detail RecordsERP Enterprise Resource PlanningFC Fibre Channel (Storage Network

Topology)FI-CA SAP Contract Accounts Receivables

and PayablesFS File SystemGB GigaByte (1024 Megabyte)GSS Global Solution Services (Hitachi Data

Systems)

LUN Logical Unit: logical slice of a RAID-Group

POC Proof-Of-ConceptQoS Quality-of- ServiceRAID Technology, that prevents Data loss

even at failure of a DiskRFC Remote Function CallRPO Recovery Point Objective (timestamp,

where to have go back)RTO Recovery Time Objective (Time

required for recovery)SLA Service Level AgreementSLO Service level ObjectiveTB Terabyte (1024 Gigabyte)TCO Total Cost of Ownership

26Contributions

SAPAs the world's leading provider of business software*,SAP delivers products and services that helpaccelerate business innovation for our customers. Webelieve that doing so will unleash growth and createsignificant new value – for our customers, SAP and,ultimately, entire industries and the economy at large.Today, more than 76,000 customers in more than 120countries run SAP applications – from distinct solutionsaddressing the needs of small businesses and midsizecompanies to suite offerings for global organizations.

Visit us at: www.sap.de or www.sap.com

Redwood SoftwareRedwood has a local presence throughout Europe,North America and Australia, with the rest of the worldcovered by some of the software industry’s strongestbusiness partners and distributors. Our global networkenables us to provide follow the sun customer supportaround the world to serve local markets with strategicfocus. With more than 3,200 customer installationsworldwide, Redwood Software is helping organizationsachieve their business goals with greater effectiveness,speed, and efficiency.

Visit us at: www.redwood.com

Hitachi Data SystemsHitachi Data Systems leverages global R&D resourcesto develop enterprise storage solutions. By focusing onthe customer's perspective, we uniquely satisfy ourcustomers' business needs. With approximately 4,000employees, Hitachi Data Systems conducts in over 170countries and regions. Hitachi Data Systems,headquartered in Santa Clara, California, is the Hitachi,Ltd. focal point for storage infrastructure solutions,storage management software and storage consultingservices.

Hitachi, Ltd., (NYSE: HIT/TSE: 6501), headquartered inTokyo, Japan, is a leading global electronics companywith approximately 390,000 employees worldwide.Fiscal 2007 (ended March 31, 2008) consolidatedrevenue totaled 11,226 billion yen ($112.2 billion). Thecompany offers a wide range of systems, products andservices in market sectors including informationsystems, electronic devices, power and industrialsystems, consumer products, materials, logistics andfinancial services.

Visit us at: www.hds.com or www.hitachi.com

27© 2009 by SAP AG.All rights reserved. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP Business ByDesign, and other SAP productsand services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germanyand other countries.

Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius,and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registeredtrademarks of Business Objects S.A. in the United States and in other countries. Business Objects is an SAP company.

All other product and service names mentioned are the trademarks of their respective companies. Data contained in this documentserves informational purposes only. National product specifications may vary.

These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAPGroup") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errorsor omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth inthe express warranty statements accompanying such products and services, if any. Nothing herein should be construed asconstituting an additional warranty.