lessons learnt implementing high-performance integration using sap pi

Premier Partners: Supported by: Produced by:

Sascha Wenninger - Australia Post

Recipes For the Perfect PI - Simple Ingredients for Complex Requirements

Mastering SAP Technologies 2011

Bottom-Up Approach

Photo by Gidzy

Mastering SAP Technologies 2011

You’ll Hear About:

Basis

Adapter Framework

Interface Design

Questions

Key Points to Take Home

Mastering SAP Technologies 2011

30,000 Foot View

SAP POS DM

SAP ERP

SAP SCEM

…JMS Queue

XMLXMLXMLXML

SAP PI

Routi

ng

Mapping

Mapping

Mapping

Mapping

Mapping…

Mastering SAP Technologies 2011

Some Figures…

Retail Transactions from 8,000+ terminals in 3,400 stores:

500,000 – 750,000 messages per dayPeaks of 45/second into PI, then split by receiver6 receiving systems, 1-3 receivers per messageMessage size <10kB

Roll-out: January – June 2011

Mastering SAP Technologies 2011

Objectives

Near-real time: <15 minutes end to endScalable to 2,000,000 messages/day and 14 receiver systemsFuture peaks of 90/second into PINo impact on other critical PI interfaces

e.g. Parcel Tracking interface moving 2m+ events per dayAs simple as possibleFuture-proof (no ccBPM, no ABAP mapping, etc.)

Mastering SAP Technologies 2011

The Solution?

Many small steps enabled us to get there.

Mastering SAP Technologies 2011

Key Topic

Basis

Mastering SAP Technologies 2011

Basis

• System Memory• Database Stuff• Java Server Nodes• The JVM• Java Heap Space• Wily Introscope

Mastering SAP Technologies 2011

System Memory

One of the main foundations of the system

Needs to accommodate all aspects of the system:Java + ABAP + ICM + DBMS + OS + caching/buffers + other apps

Need to avoid swapping to disk at all times – causes huge GC delays!

Mastering SAP Technologies 2011

The Database

DB StatisticsWithout DB stats, our throughput dropped to 1/6th of the normal! Run daily in production, but be careful in non-prod systems

DB ReorgsPI DB tables churn very frequently and fragment after a few weeksTest your system/DBMS before and after a reorg to evaluate value

Mastering SAP Technologies 2011

Java Server Nodes

More Java Server Nodes (Java processes) provide:• Better system memory utilisation• Increased throughput via parallelisation

…but also incur costs in cluster synchronisation.

This can be significant for high volumes of EOIO interfaces with more than a few Java Server Nodes

Mastering SAP Technologies 2011

The JVM

• PI 7.1 and later use SAP’s own JVM– SAPJVM 4.1 for PI 7.0 coming soon (Notes 1495160, 1522198)

• Bug fixes and performance improvements delivered continually

Several JVM Upgrades over the past 12 months significantly reduced OOM crashes and Garbage Collection times for our systems.

This enables the use of larger heap spaces!

Patching via JSPM takes 30 minutes and 1 restart, plus some testing

Mastering SAP Technologies 2011

Java Heap Space

More Heap = process larger messages, fewer full GCs, fewer OOMs

• SAP’s recommendation has historically been 2GB per Java process. – Recently changed to 4GB (for SAP JVM)

• AusPost has been using 4GB for 9 months now. Main criteria: Full GCs should take less than 10 seconds.

Mastering SAP Technologies 2011

Java Memory Analysis

This is in SolMan!

Mastering SAP Technologies 2011

Wily Introscope

• Monitoring for the Java Stack• Basic version is free with NetWeaver Java• Provides functionality normally accessed via ABAP transactions

Mastering SAP Technologies 2011

Wily Dashboards

Mastering SAP Technologies 2011

More Dashboards

Mastering SAP Technologies 2011

And More…

Mastering SAP Technologies 2011

Tons of Dashboards!

…and this is only the free version!

Mastering SAP Technologies 2011

Our PI System

PI 7.11 SP5JVM 5.1 Patch level 59 (64 in test systems)3 Java server nodes, 4GB heap space each.AIX 6.1 on IBM POWER6, 60GB RAM

Mastering SAP Technologies 2011

Basis

System Memory Database Stuff Java Server Nodes The JVM Java Heap Space Wily Introscope

Many aspects which require special PI-Basis skills!

Mastering SAP Technologies 2011

Key Topic

Adapter Framework

Mastering SAP Technologies 2011

Adapter Framework

• Thread Behaviour• Messaging System

Mastering SAP Technologies 2011

Threads

Messages are processed using a number of adapter-specific queues.Each queue has its own thread pool.

e.g. for JMS:

Queue Purpose Default Threads

JMS Send Async Sender 5

JMS Recv Async Receiver 5

JMS Call Sync Sender 5

JMS Rqst Sync Receiver 5

Mastering SAP Technologies 2011

Threads

Threads are assigned to process messages from their respective poolsUse Wily Introscope to monitor utilisation and backlogs

Mastering SAP Technologies 2011

Threads

In ‘traditional’ scenarios, both Sender and Receiver queues and thread pools are used.

For Integrated Scenarios in the AAE, only the Sender-side queues and thread pools are used!

Threads from the Sender pools do all the work!

Mastering SAP Technologies 2011

Caveat

Some Adapters process messages serially!

e.g. JMS: 1 Adapter Thread per Communication Channel reads messages from the JMS queue

1 JMS Comm Channel3 Java Server Nodes{

Mastering SAP Technologies 2011

Threads in the AAE

Adapter Thread picks up a message from the JMS queue, then:• Executes Adapter Modules• Performs Receiver Determination• Persists 1 message per Receiver in Messaging System queue.• Confirms message and removes from the JMS queue

The longer this takes, the lower throughput will be.

Mastering SAP Technologies 2011

What does this mean?

Mastering SAP Technologies 2011

Sender-Side Performance is Really Important

Mastering SAP Technologies 2011

Implications

Throughput depends on Sender Adapter performance

Try to avoid adapter modules in Sender adapters. We achieved a 15% improvement by moving an XSLT mapping from

the JMS adapter to the Operation Mapping.

Mastering SAP Technologies 2011

Implications

Throughput depends on Receiver Determination

The initial “Adapter” thread evaluates conditions and copies the message for each Receiver.

Optimise XPath conditionsDB performance: more Receivers = more I/O

Mastering SAP Technologies 2011

Implications

Throughput depends on parallelisation

Increase the Sender-side Thread Pools and Application Threads. We use 15 JMS Sender Threads and 350 Application Threads

(also see Note 937159)

Mastering SAP Technologies 2011

2 Options

In our system, each thread can process 10-15 messages/sec

In order to consistently process 45 messages/sec from JMS, we need 4-5 concurrent JMS Sender Adapter threads.

Option 1: 5 Java server nodes

Option 2: Multiple JMS Communication Channels.

Mastering SAP Technologies 2011

Multiple Comm. Channels

We cloned the Integrated Config Object and Sender Comm Channel2 JMS Sender Adapter Threads per Java Server node

Better thread pool utilisation No increased memory requirements More scalable

This is not needed for SOAP since it’s truly parallel.Image Source: http://t.co/ZOBdlH0

Mastering SAP Technologies 2011

Adapter Framework

Thread Behaviour Messaging System

Increased importance for Integrated Configuration scenarios!

Mastering SAP Technologies 2011

Key Topic

Interface Design

Mastering SAP Technologies 2011

Integrated Configuration

Receiver Determination+ Interface Determination+ Receiver AgreementsIntegrated Configuration Object

Causes an interface to be executed in the Advanced Adapter Engine

Mastering SAP Technologies 2011

Integrated Configuration

• Message Processing entirely in Java (AAE)• Introduced in PI 7.1; more features in 7.11 and 7.3• Improved performance by:

– Reducing database I/O– Eliminating ‘stack jumping’ between ABAP and Java

• 7-10 times throughput possible• Not available for all adapters and scenarios.

Mastering SAP Technologies 2011

Example

Mastering SAP Technologies 2011

Example

Mastering SAP Technologies 2011

Constraints

There are a few things you can’t do with Integrated Configuration:

IDocs*

Multi-Mappings*

ccBPMs

But this is an Opportunity to be Creative!

*features added in PI 7.3

Mastering SAP Technologies 2011

Mapping Programs

XSLT• Easier to debug and tune• Better performance from PI 7.1 onwards• Easier to support

Consolidate mapping stepsSame performance, but less GC!• Reuse still possible via XSLT imports.

Mastering SAP Technologies 2011

Communication Channels

ABAP Proxies/Enterprise Services Use SOAP Adapter in XI 3.0 mode

JMS Senders Clone comm. Channels for greater parallelisation

Non-SAP Systems Ensure inbound queuing is available to avoid backlog in PI

Mastering SAP Technologies 2011

Design-Time Governance

Don’t Overdo It!

Find a balance which facilitates reuse but doesn’t impose too much up-front work.

Image Source: http://geekandpoke.typepad.com

Mastering SAP Technologies 2011

Questions

Mastering SAP Technologies 2011

Key Points to Take Home

Wily IntroscopeEssential for understanding performance of the Java stack

Use Integrated Configuration Objects20 minutes to migrate an existing interface

Keep your JVM CurrentContinuous improvements without much fuss

Most of the other points can be quick wins too if you don’t do them all at once.

Mastering SAP Technologies 2011

Sascha WenningerAustralia Post

sascha.wenninger@auspost.com.au

@sufw

+61 403 933 472

Mastering SAP Technologies 2011

Further Reading

• PI Performance Check• SOA MiddlewareArticles on SDN

– Incl. the Performance section

Premier Partners: Supported by: Produced by: