how to evaluate sap businessobjects out of memory

How to Evaluate SAP BusinessObjects Out of Memory Generated Java Heap Files Applies to: SAP BusinessObjects BI 4.0 services running under Adaptive Processing Server such as the DSL Bridge fulfilling query requests by for instance Web Intelligence using the BICS connectivity on BW. Working title: “How to evaluate SAP BusinessObjects out of memory generated java heap files” Summary: This paper provides a step by step description incl. screenshots to evaluate out of memory generated java heap files and especially what was the root-cause for an out of memory issue. Understanding the reasons for an out of memory issues can increase the stability and user convenience of SAP BusinessObjects Enterprise and tools such as Web Intelligence and other reporting tools that running under Adaptive Processing Server. The results of the tracing exercises can give important information on why an out of memory incident occurred and how to avoid this in the future. The result from the tracing exercise can for instance be used to redesign BEx Query for the purpose of consuming data in Web Intelligence reports. Without evaluating the cause of an out of memory incident it is almost impossible to discuss on how to avoid this in the future. With this paper you should be able to find the exact root-cause of an out of memory incident. With this paper you will get insights in how to trace out of memory java heap files yourself and to some degree understand the inner workings of the Adaptive Processing server hosting the Web Intelligence.

Authors Bio

Kurt Holst ([email protected]) is an expert on SAP BusinessObjects Enterprise architecture, configuration and performance optimization. Kurt has worked with most major OLAP databases, SAP NetWeaver BW and relational databases together with different BI reporting tools. Before joining SAP Kurt has worked as a BI Architect & developer both internal and as an external consultant. Furthermore Kurt has been attached to the university as an external lector on Business Intelligence. Kurt is organizational with the Business Analytics Services team in SAP Denmark but currently works out of the SAP Office in Levallois, France.

Company: SAP Last update: January 10 2013

mailto:[email protected]

How to Evaluate SAP BusinessObjects Out of Memory Generated Java Heap Files

2

TABLE OF CONTENTS

BACKGROUND AND THE NEED FOR EVALUATING JAVA HEAP FILES .................................................. 4

STEPS REQUIRED FOR EVALUATING THE JAVA HEAP FILE ................................................................... 7 Heap Dump generation: .................................................................................................................................. 7 Using Memory Analyzer Tool (MAT) to Analyze Java Heap Dumps: ......................................................... 8 Evaluating the java heap log file using memory analyzer ........................................................................... 8 Instability or performance issue – key findings and technical background: .......................................... 10

STEPS TO PREVENT OOM FROM HAPPENING AGAIN ............................................................................ 14

HOW TO INCREASE THE STABILITY OF SAP BO ENTERPRISE ............................................................. 15

REFERENCE DOCUMENTATION ................................................................................................................. 18


3

Version-log :

Date Name Alteration Reason Version

13.11.2012 Kurt Holst Initial Document 0.1

15.11.2012 Kurt Holst Background and overview 0.2

15.11.2012 Kurt Holst Tracing 0.3

19.12.2012 Kurt Holst Describing the memory analyser 0.4

28.12.2012 Kurt Holst Adding best practices redesign 0.5

02.01.2013 Kurt Holst Adding memory analyser tool desciptions 0.6

03.01.2013 Kurt Holst Finalizing first draft to share internally 0.7

09.01.2013 Kurt Holst Finalizing to publish externally (SCN) 1.0


4

BACKGROUND AND THE NEED FOR EVALUATING JAVA HEAP FILES

The BusinessObjects Enterprise platform is a highly flexible solution that allows user with many different reporting needs on place to access this information. To make sure the users archive highest possible conveniences there is a need to proactively maintain and optimize the BusinessObjects Enterprise services and when an incident occurs such as if the Adaptive Processing Server runs out of memory and generates a java heap file the root cause should be determined to avoid this in the future. This article describes how to evaluate the java heap file and interprets the information to allow for highest possible stability and performance in your BusinessObjects Enterprise environment. The cause of less than optimum stability users experience with ad hoc reporting in SAP BusinessObjects Web Intelligence using BW as data sources with the BICS connectivity has in some cases been identified as related to design of the BEx Query. The obvious question here is how stability with Web Intelligence on BW using BICS can be improved by re-designing on either Web Intelligence or the SAP BW side? This will also be discussed.

Figure 1: Overview of SAP BusinessObjects Enterprise Architecture

As shown on the overview of the SAP BusinessObjects Enterprise Architecture the APS is a part of the Processing tier. The BI4.0 platform is by default installed with one Adaptive Processing Server (APS) per host system. The Adaptive Processing Server is a generic server that hosts multiple services responsible for processing various requests. Services that can be hosted on this server include: Adaptive Connectivity Service, BEx Web Application Service, Client Auditing Proxy Service, Custom Data Access Service, Data Federation Service, Document Recovery Service, DSL Bridge Service, Excel Data Access Service, Lifecycle Management ClearCase Service, Lifecycle Management Console Service, Monitoring Service, Multi-


5

Dimensional Analysis Service, Platform Search Service, Publishing Post Processing Service, Publishing Service, Rebean Service, Security Token Service, Translation Service, Visual Difference Service, Visualization Service, Web Intelligence Monitoring Service.

Definitions:

“BICS" is short for Business Intelligence Consumer Services which is the connectivity allowing SAP

BusinessObjects tools such as Web Intelligence or Dashboard Designer.

”DSL” or The Dimensional Semantic Layer Bridge is what actually connects to BW. It is hosted by the

Adaptive Processing Server. The Adaptive Server hosting the DSL Bridge should be sized for expected user

load. The DSL is recommended to be the only service hosted by a particular Adaptive Processing Server in

anything beyond a small test environment.

OOM: Out of memory. When the Adaptive Processing Server or more precise the Java VM reaches the

upper memory limit the java dumps a heap file.

Java heap file: A heap dump is a file that contains a snapshot of the memory allocated by the Java Virtual

Machine. Contains technical information to find the root cause of a problem for developers and support

engineers.

Moreover this article will document that using the trace functionality on the java heap file it will show what Web Intelligence report or other reporting tool using BICS were the root cause of an out of memory incident processed. Further to that how to interpret this information to improve reporting stability and performance. When generating a Web Intelligence report based upon BICS which is in terms of BW a BEx Query the DSL Bridge Service is communicating with BW. This implies that when executing or designing a Web Intelligence report on a BEx Query there is transmitted BI Metadata from BW to Web Intelligence in order to allow users to select attributes, key figures, hierarchies etc. With BICS connectivity, there isn’t any predefined universe. Web Intelligence needs then to generate a so called transient universe in runtime (“on the fly”). In theory that would imply that a BEx Query with a lot of metadata (key figures, restricted key figures, attributes, hierarchies etc.) will also need more time to build this transient universe. With the former BusinessObjects Univers design tool universes were pre-built and stored on the BusinessObjects Enterprise environment.

Figure 2: High level overview of Web Intelligence report run on BW using the BICS

The technical background after executing a Web intelligence report can be split into different parts:

1. The BICS connection created in the CMC (or Information Design Tool) is used with Dimensional

Semantic Layer (DSL) Bridge.

2. The DSL Bridge embeds with the DSL BICS layer.


6

3. DSL BICS layer uses the BICS driver (physical JAR-drivers stored on the operating system hosting

the SAP BO Enterprise server)

4. The BICS driver communicates with the Java Connector (JCO)

5. The JCO creates the RFC (Function Call) to SAP BW

6. BW uses the BICS Backend functions to call the OLAP engine

7. The OLAP Engine fetches the data from the database.

Figure 3: Detailed technical description of APS & DSL Bridge Service


7

STEPS REQUIRED FOR EVALUATING THE JAVA HEAP FILE

The following steps describe how to collect data from the java heap file after an out of memory incident has occurred. If the java heap file isn’t generated automatically due to an incident where there was not an out of memory the java heap file can also be generated manually. Just before the Java Virtual Machine runs out of memory it tries to free up available memory – this is the work of the Garbage Collector (GC). In an undersized environment or where a user runs a maverick-query the CPU usage is increased heavily when the GC works to prevent running out of memory (Xmx upper limit set in the CMC). If the GC isn’t able to free up memory the Java VM dumps the memory to a java heap file. In short a java heap dump is a file that contains a snapshot of the memory allocated by the Java Virtual Machine. This file contains the very last activity just before the out of memory incident occurred and with a bit of detective work you can actually find the root cause and hopefully make the appropriate changes to avoid this in future. In the following it is described how to analyze the java heap file dump. In order to find the biggest memory consumer in your APS process, you will first need to have a java heap dump. Heap Dump generation:

If the process is reaching OOM (OutOfMemory) , it will generate a heap dump file automatically in “C:\Program Files (x86)\BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\logging"

With the name java_pid[pid].hprof, e.g: java_pid11448.hprof

If the process is not reaching OOM, then you can generate manually a heap dump file : Choose in CMC->Servers and find the PID of the culprit server, e.g.: PID 10936 Start a command line tool: cd “C:\Program Files (x86)\BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\bin” jmap -dump:file=c:\dumpdemo.hprof 10936


8

Using Memory Analyzer Tool (MAT) to Analyze Java Heap Dumps: Once the heap dump file is generated (hprof extension), it can be analyzed by using the Memory Analysis Tool:

Get the tool from this link : http://www.eclipse.org/mat/

Figure 4: Overview of memory analyzer tool (MAT)

Evaluating the java heap log file using memory analyzer

The Eclipse Memory Analyzer is a fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.

Use the Memory Analyzer to analyze productive heap dumps with hundreds of millions of objects, quickly calculate the retained sizes of objects, see who is preventing the Garbage Collector from collecting objects, and run a report to automatically extract leak suspects.

Opening a java heap dump file using MAT: From the menu “File” chose Open Heap Dump…

Browse to the directory where you placed the java heap dump file and select the file. MAT will now give you a new menu to select from.

http://www.eclipse.org/mat/


9

Depending on the size of the java heap file dump it may take a few minutes to open. You may also chose to run the report in background allowing to somewhat use the MAT.

Create a histogram to show the biggest memory consuming BICS queries:

As can be deducted by the screen shot below at the time of the out of memory issue this query was trying to fetch 2.331.556 cells which would be a mighty large Web Intelligence report:


10

Instability or performance issue – key findings and technical background: Using the Java Heap dump provides detailed information on the Adaptive Processing Server, which is can be containing multiple BO services including the DSL Bridge (BICS connectivity), is restarted possible due to out of memory (OOM) issues. At the moment of the crash – the BW system had fetched approximately 2.3 million cells:


11

Most likely it crashed when it was trying to get the data back from BW via the BICS connector to Web intelligence. It is returning almost 1 GB of raw-data as can be seen from below.

Searching from the “BICSQueryViewWrapper” shows the actual query that was run against the BW database:

Right click on the item as shown below and select “Show objects by class” ->”by outgoing references”

Which gives the following:


12

Drill down to “queryState” as shown:

On the left side a value appears with “..BICS_VIEW..” text. This value contains the exact statement that this whole exercise is all about. To obtain the query statement right click on the value field and select “Copy” -> “Value” as shown on the screen shot below:

Copy and paste the information extracted from the copied XML into your favorite text editor. In the following I use Notepad++


13

<Query> This shows which Query that was used to access the BW data. The safety belt “<BICS_DA_REULT_SET_MAX>” can be applied to reduce the risk of running out of memory – please consult document: “SAP BusinessObjects increasing stability by setting limits on max. retrievable cells from SAP BW into Web Intelligence using BICS.” https://scn.sap.com/docs/DOC-31900 The Web Intelligence report that caused the problem is using the following BEx query: “SALES_ANALYSIS_BICS_QUERY_001”. The extracted information also reveals that the following dimension attributes and measures were selected:

Variables (prompt that is performed on the BW system):

https://scn.sap.com/docs/DOC-31900


14

It seems the incident with restarting the Adaptive Processing Server happened with just one BW connection open – this together with the SQL Statement indicates that this report is an Ad Hoc Web Intelligence report build upon the same BEx Query BICS connection as the one used when the APS restarted due to an out of memory issue. This very query was for training purposes also run on the BW side which also caused the BW system to run out of memory! In the next section it will be discussed how to avoid this kind of out of memory issues. STEPS TO PREVENT OOM FROM HAPPENING AGAIN

“The usual suspects” based on the information gathered from the log file it can be determined that the out of memory in many case it was due to a “runaway query” generated by an Web Intelligence user whom selected millions of transactions to be returned. The way Web Intelligence works is that it will return all data in a query to the micro cube stored on the SAP BO Enterprise server. Optimizing the APS – see “How to APS”, http://scn.sap.com/docs/DOC-31711 Redesign the BEx Queries to avoid situations where users inadvertently can produce problematic queries using Web Intelligence. Ensure that end users will at some time make a Web Intelligence report based on all the possible fields in the BEx Query BICS connection – that implies redesigning the BEx Queries used for Ad Hoc Web intelligence reports. A good strategy for BEx Query as data source for BO Web Intelligence should be developed and in place before going to production. This will remove the possibility that the somewhat untrained user can cause this problem again – trying to retrieve rows of data into a report. It is at the moment not possible to stop a runaway Web Intelligence query based on BW from the SAP BO frontend. However it is highly advisable to limit the number of rows retrieved from BW using BICS connection – using the safety belt feature. Choose the right tool - product position: use a dedicated OLAP tool such as Analysis for OLAP or Analysis for MS Office. It is advisable to consider using other tools in the BO portfolio for ad hoc or data explorations. Crystal Reports could also be used for ad hoc analysis – crystal only retrieves pages of data and is hence a better tool if the underlying amount of data in a report is considerable. Moreover SAP BO Explorer is also a choice to consider if the end user reporting requirements are of a more data exploring nature. The same goes for Analysis for OLAP which could also perform far better on a larger set of data in an ad hoc reporting scenario using BICS. “Selecting the Right SAP BusinessObjects BI Client Product Based on your Business Requirements for SAP BW Customers” https://scn.sap.com/docs/DOC-32449 Safety belt feature: Implement the safety belt to reduce that risk or having users creating run-away quies causing out of memory issues. SAP BusinessObjects increasing stability by setting limits on max. retrievable cells from SAP BW into Web Intelligence using BICS. https://scn.sap.com/docs/DOC-31900

http://scn.sap.com/docs/DOC-31711




15

HOW TO INCREASE THE STABILITY OF SAP BO ENTERPRISE

In a situation where the evaluation of the Java heap log file revealed that the adaptive processing server was running out of memory and the WebI query (or queries) that caused this were within what could be expected. The reason for OOM could be due to lack of specific sizing or just the fact that SAP BO Enterprise was installed with default setting without any post-configurations applied.

Right click on the AdaptiveProcessingServer and select Clone Server:


16

Clone the APS and name it to APS_DSL_Bridge:

Select the services for the new cloned server:

Select the DSL Bridge Service, Web Intelligence and Security Token Service for the new cloned server:


17

Assign the required memory to the APS_DSL_Bridge. In this example “-Xmx8g” (Gigabyte) memory is allocated to the java VM running the APS DSL Bridge Service.

Also remember to remove the DSL Bridge etc. from the original APS server – this can be done by stopping the APS server and choosing select services and remove the DSL Bridge Service, Web Intelligence and Security Token Service etc.


18

REFERENCE DOCUMENTATION

The following documents are referenced in this document:

DOCUMENT NAME LOCATION ISSUE DATE

Memory Analyzer software http://www.eclipse.org/mat/

SAP BusinessObjects increasing stability by setting limits on max. retrievable cells from SAP BW into Web Intelligence using BICS.

https://scn.sap.com/docs/DOC-31900 20-09-2012

“How to APS” http://scn.sap.com/docs/DOC-31711

Nov. 2012

SAP WIKI - APS http://wiki.sdn.sap.com/wiki/display/BOBJ/Business+Intelligence+4.x+APS+(Adaptive+Processing+Server)+Service+Distribution

2012+

Sizing guidelines www.sap.com/bisizing. http://scn.sap.com/docs/DOC-33183

“Selecting the Right SAP BusinessObjects BI Client Product Based on your Business Requirements for SAP BW Customers”


SAP internal access only :

Wiki Memory Analyzer https://wiki.wdf.sap.corp/wiki/display/METG/METG+-+Using+MAT+to+Analyze+SAP+ME+Heap+Dumps

WebI. Troubleshooting BICS. https://wiki.wdf.sap.corp/wiki/display/BOBJCAS12Y/WebIntelligence+troubleshooting#WebIntelligencetroubleshooting-HeapDumpgenerationonaspecificWebIntelligenceBICSworkflow

http://www.eclipse.org/mat/



http://wiki.sdn.sap.com/wiki/display/BOBJ/Business+Intelligence+4.x+APS+(Adaptive+Processing+Server)+Service+Distribution




http://www.sap.com/bisizing



https://wiki.wdf.sap.corp/wiki/display/METG/METG+-+Using+MAT+to+Analyze+SAP+ME+Heap+Dumps




https://wiki.wdf.sap.corp/wiki/display/BOBJCAS12Y/WebIntelligence+troubleshooting#WebIntelligencetroubleshooting-HeapDumpgenerationonaspecificWebIntelligenceBICSworkflow





© 2013 SAP AG. All rights reserved.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP

BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP

products and services mentioned herein as well as their respective

logos are trademarks or registered trademarks of SAP AG in Germany

and other countries.

Business Objects and the Business Objects logo, BusinessObjects,

Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and

other Business Objects products and services mentioned herein as

well as their respective logos are trademarks or registered trademarks

of Business Objects Software Ltd. Business Objects is an SAP

company.

Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL

Anywhere, and other Sybase products and services mentioned herein

as well as their respective logos are trademarks or registered

trademarks of Sybase Inc. Sybase is an SAP company.

Crossgate, m@gic EDDY, B2B 360°, and B2B 360° Services are

registered trademarks of Crossgate AG in Germany and other

countries. Crossgate is an SAP company.

All other product and service names mentioned are the trademarks of

their respective companies. Data contained in this document serves

informational purposes only. National product specifications may vary.

These materials are subject to change without notice. These materials

are provided by SAP AG and its affiliated companies ("SAP Group")

for informational purposes only, without representation or warranty of

any kind, and SAP Group shall not be liable for errors or omissions

with respect to the materials. The only warranties for SAP Group

products and services are those that are set forth in the express

warranty statements accompanying such products and services, if

any. Nothing herein should be construed as constituting an additional

warranty.

www.sap.com

how to evaluate sap businessobjects out of memory

Documents