50 million active accounts online and batch …...batch transactions for 50 million active accounts....

Oracle Financial Services Lending and Leasing Performance Benchmark

50 Million Active Accounts Online and Batch Transactions

O R A C L E W H I T E P A P E R | O C T O B E R 2 0 1 5

ORACLE FINANCIAL SERVICES LENDING AND LEASING PERFORMANCE BENCHMARK

Disclaimer

The following is intended to outline our general product direction. It is intended for information

purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any

material, code, or functionality, and should not be relied upon in making purchasing decisions. The

development, release, and timing of any features or functionality described for Oracle’s products

remains at the sole discretion of Oracle.

1 | ORACLE FINANCIAL SERVICES LENDING AND LEASING PERFORMANCE BENCHMARK

Table of Contents

Disclaimer 1

Introduction 3

Overview 4

Objective 4

Performance Test Strategy 4

Architecture Flow Diagram 6

Interface Gateway 6

Batch Flow 7

Test Environment 8

Load Test Architecture 9

Workload Specification 10

Characteristics 10

Workload 10

Scope 11

Test Results 12

Online Soak Test 12

Application Server Statistics 13

Database Server Statistics 15

Batch Jobs Run 19

Database Server Statistics 19

Online Surge Test 22


Tuning Parameters 23

Database 23

Weblogic Application Server 24

Appendix 26

OATS Report 26


Introduction

Oracle Financial Services Lending and Leasing Release 14.2 have been benchmarked for online and

batch transactions for 50 million active accounts. This document provides further information about the

performance test objective, strategy, parameters and results. This document is intended to be used by

» Architecture Groups

» Hardware Sizing

» Performance Engineering Groups

The goal of the project is to simulate and measure the performance and scalability of the Oracle

Financial Services Lending and Leasing Release 14.2 solution. This includes online transactions with

definite concurrent users and batch job process.

The benchmark test was conducted at the Oracle Solution Centre, Santa Clara, U.S.A involving

personnel from different Oracle groups.


Overview

Objective

The goal and objective of the project is to simulate and measure the performance and scalability of Oracle Financial

Services Lending and Leasing Release 14.2 for 50 million active accounts. This includes online transactions

(simulated by test agents) and batch processes.

Performance Test Strategy

FUNCTIONAL METHODOLOGIES AND TERMINOLOGIES

Category What are these categories? How Performance is measured

OLTP User initiated (Branch User)

System initiated (web service clients)

Only System Initiated Transactions were tested as part of the

OLTP benchmarking exercise

OFSLL OLTP workload is captured through OATS (Oracle

Application Testing Suite) Load Tester component and

replayed to simulate system initiated, the load depending upon

the TPS and utilization requirements

Measured by response time in the 99th

percentile and concurrent users that system handles.

Batch The Batch process within OFSLL has a set of batch jobs which

are run via the in-built scheduler.

At the end of a logical testing day, a specific batch job set

containing the specific set of batch jobs will be triggered.

Measured by the time taken to execute the

specified batch job set

USAGE OF TOOLS

Category How will we test? Usage of Tools

OLTP OATS (Oracle Application Testing Suite) scripts would be created that

simulates online web service calls for each type of OLTP transaction

planned.

These scripts would be executed from dedicated OATS load

generator machines in lab environments.

Application Server

» Mission Control for Hotspot JVM

» Visual VM

Database Server

» AWR

Batch The batch jobs will be triggered from the front end of the OFSLL

application.

Database Server

» AWR


People Involved

A-TEAM

Group People Title

Oracle Financial

Services Software Ltd

(FSGBU)

Vikram Gupta VP Engineering

Sanjeet Rao VP Engineering

Vinay Bhatia Senior Director of Engineering

Dwidesh Veetil Director of Engineering

Deiveegarajan Katturaja OATS Test Architect

Jai Om Development Architect

Madhu Mittal Data Architect

Natesh G S DBA

Prashant Baandal Director of Consulting

Vijay Nair Application Architect

Samyak Jain DBA

Saravanan Raman Database Test Architect

Mannat Saini Webservice Developer

Oracle Venkat Kompally Exadata DBA

Oracle Solution

Center

Yesh Jaiswal OSC Infrastructure Support


Architecture Flow Diagram

Interface Gateway

Figure 1. Interface Flow


Batch Flow

Figure 2. Batch Flow


Test Environment

HARDWARE INVENTORY

Category Details

Servers Application Server

» 2 * Oracle X5-2 Servers (256 GB RAM / Intel Xeon CPU E5-2699 v3 @ 2.30 GHz)

» 1 * Oracle X4-2 Server (256 GB RAM / Intel Xeon CPU E5-2697 v2 @ 2.70 GHz)

Database Server

» 1 * Oracle EXADATA Database Machine X5-2 (Quarter Rack)

Storage Application Server

» Local storage of 150 GB

Database Server

» EXADATA Storage (20 TB) configured for HP (High Performance)

Operating System Application Server

» Oracle Enterprise Linux 6.7

Database Server

» Oracle Enterprise Linux 6.6

Database VM » 2-nodes Oracle RAC database 12.2.0.1 enterprise 64-bit

Application VM » Oracle Weblogic Enterprise Edition 10.3.6 64-bit.

» Cluster configuration with 12 managed servers across two physical machines (Third Machine was a backup)

Load Balancer » F5 Load Balancer between load generator and application VM. Single arm configuration with Round-Robin

load balancing method

Load Generators » 4 Load testing machines

» Oracle OES 6.2 OS

» Standalone Oracle Database 11.2.0.4 enterprise 64-bit on one x4270M2

» Java JDK 1.7.0_51 64-bit

» Oracle Application Testing Suite 12.5.0.2.0

» Oracle Application Testing Suite Agent


Load Test Architecture

Figure 3. Deployment Architecture


Workload Specification

Characteristics

The following group of variables determined the scope of the characterization:

» Active Loans

» Active Leases

» Active Lines

» Origination Transactions

» Servicing Transactions

» Payment Transactions

» Batch Billing Run

» Batch Payment Run

The benchmark captured each of these in a test harness driven workload.

The following characteristics defined the benchmark metrics:

BASE VOLUMES

Loan Types Base Volumes

Active Loans 31,810,000

Active Leases 10,590,000

Active Lines 10,214,000

Total Active Accounts 52,614,000

Workload

All transactions had a mix of 60% Loans, 20% Lines and 20% Leases products.

Following were the workload metrics/SLAs agreed upon:

WORKLOAD METRICS

Operation Workload Type Workload Spread Workload Metric SLA

Application Entries Online Uniform 80,555 Applications in 05 hours 2 seconds

Check Status Online Uniform 47,222 Transactions in 05 hours <1 second

Application Update

(Comment)

Online Uniform 23,611 Transactions in 05 hours 1 second

Payment Messaging

Processing

Online Uniform 46,059 Transactions in 05 hours <1 second

Account Search Request

Service

Online Uniform 47,222 Transactions in 05 hours <1 second

Post Transaction Request

Service

Online Uniform 3,935 Transactions in 05 hours 2 seconds

Account Payoff Quotes Online Uniform 15,740 Transactions in 05 hours <1 second


Payment Run Batch n/a Single Overnight Run

(1,658,150 Transactions)

1 Hour

Billing Run Batch n/a Single Overnight Run

(1,658,150 Transactions)

1 Hour

Scope

The following online services were agreed to be tested as part of the benchmark:

WEBSERVICE LISTS

Service Name Description

LOSApplicationRequestService Create a loan or lease or line contract and associate collateral with the

financing product

LOSCheckStatusRequestService Retrieves a summary of the contract parameters

LOSApplicationUpdateService Updates an existing loan contract

CSSAccountSearchRequestService Service to fetch account details by attributes such as account number,

customer first name, customer last name, SSN.

CSSPostTransactionRequestService To Post different transactions such as update address, update phone number,

update ACH details, etc.

CSSAccountPayOffQuoteRequestService To get the account payoff quote

CSSPostTransactionRequestService Online payment (via One time ACH service but disabling the ACH entry)

The following batch programs were agreed to be tested as part of the benchmark:

BATCH PROCESS LIST

Batch Name Description

Billing Processing As part of the billing batch, OFSLL posts transactions which can be then

picked up by an external system. 1/20th of the total volume should be

considered for the billing cycle batch run with 5% of this volume to be

considered for fees i.e. late fees. The total time taken to publish these

transactions into the OFSLL tables and the JMS queues will be measured

Payment Processing OFSLL has a two-step process, load of data followed by the processing. As

part of the benchmark test, we will simulate step 1 via a data load script. Once

that is completed Step 2 will kick in to process this and apply the payments.

The total time taken to do Step 1 and Step 2 together will be measured.

Spreads would include application of payments to principal/interest and fees


Test Results

Online Soak Test

TRANSACTION PROCESSING SUMMARY

Characteristics Details

Total Number of Virtual Users 240 Users (Multiple Iterations until Time expiry)

Duration of Soak Test 18,019 seconds (~ 05 Hours)

Total Transactions Processed Application Entry 2,16,354

Check Application Status 1,06,851

Update Application 71,219

Online Payment Posting 1,18,506

Account Search 2,13,831

Post Transaction 1,40,970

Account Payoff Quote 1,18,268

Total 9,85,999 Transactions in 05 Hours

Transactions Per Second 54 TPS

TIMINGS SUMMARY

Description Expected SLA Webservices 99th

% Response Time

Application Entries 2 seconds Create Loan / Lease/ Line

(Entry, Approved and Verified)

0.236

Check Status <1 second Check Application Status 0.030

Application Update 1 second Update Application 0.025

Payment Messaging Processing <1 second Payment Posting Online 0.025

Account Search Request Service <1 second Account Details 0.031

Customer Name Search 0.021

Customer SSN Search 0.017


Service

2 seconds Address Update 0.085

Phone Update 0.083

Account Payoff Quotes <1 second Payoff Quote 0.034

Total 9,85,999 Transactions in 05 Hrs

Transactions per second 54 TPS


Application Server Statistics

The application was run in a high available WebLogic cluster with “12” Managed Servers spread across “2” physical

machines.

Memory Profile

Figure 4. Memory Profile (per JVM)

» Maximum Memory Usage per JVM was approximately 4 GB out of the allocated 8 GB

» No memory leaks during the entire run

» CMS collector exhibited normal Garbage Collection Timings. No “Stop the World” GC during entire run


CPU Profile

Figure 5. CPU Profile (per JVM)

» Average CPU Usage per JVM was approximately 3%.

» Average Total CPU Usage per physical machine was approximately 18% (6 JVMs * 3 %)


Thread Profile

Figure 6. Thread Profile (per JVM)

» Balanced Distribution across Weblogic threads

» No “STUCK” threads during the entire run indicating ZERO contention

Database Server Statistics

The application was running in a high-availability Active-Active RAC Node cluster.

Memory Profile

» No abnormal memory usage. Allocated memory pools (SGA/PGA) were utilized efficiently.

Figure 7. Memory Utilization


CPU Profile

» No CPU contention; overall average CPU Utilization during load testing is always less than 10% with few spikes

touching 30-45% for less than 1/few seconds .

Figure 8. CPU Utilization (across both RAC Nodes)


Load Average

» Load per CPU is less than 1per CPU during load testing, which is under acceptable limits for current processing.

Figure 9. Load Averages


I/O Performance

» The I/O response times are under acceptable limits. The cell single block reads and cell multi block reads which is

average read time of a single/multi block are less than 2ms/6ms. Most of the times with spikes coincides with the

SYS related queries during the snap time interval

Figure 10. CELL Single Block Reads

Figure 11. CELL Multi-Block Read Times


Batch Jobs Run


Run Type Transactions Processed Job Name Time Taken

Payment Run Payments Processed 1,413,242 Payment Job 08 Minutes 09 Seconds

Billing Run Billing Processed 52,614,027 Billing Jobs

» Late Charge

» Termination

» Charge-off

» Billing/Due Date

» Rate Change

» Void/Paid

Account Close

» Anniversary

» Month End

» Year End

» Interest Accrual

and Delinquency

49 Minutes 21 Seconds

» Late Charges 1,359,776

» Statement Generated 1,380,036

» Interest Accrual and

Delinquency

50,728,005

Database Server Statistics

Payment Batch Job

» No CPU contention; overall average CPU Utilization during load testing is always less than 70-80%



Billing Batch Job

» No CPU contention; overall average CPU Utilization during load testing is always less than 70-80%


Memory Profile

» No abnormal memory usage. Allocated memory pools (SGA/PGA) were utilized efficiently.

Figure 14. Memory Utilization


Load Average

» Load per CPU is less than 1 per CPU during load testing, which is under acceptable limits for current processing

Figure 15. Load Average

I/O Performance

» The I/O response times are under acceptable limits. The cell single-block reads and cell multi-block reads which

is average read time of a single/multi-block are less than 2ms/6ms. Most of the times with spikes coincides with

the SYS related queries during the snap time interval.

Figure 16. CELL Single Block Reads


Figure 17. CELL Multi-Block Read Times

Online Surge Test

A one-off surge test was conducted for the OLTP scenarios to ensure that the application architecture can withstand

peak volumes.


Characteristics Details

Duration of Tests 1 Hour

Total Number of Virtual Users 800 VU’s

Total Transactions Processed Application Entry 53,967

Check Application Status 56,240

Update Application 56,306

Online Payment Posting 1,16,792

Account Search 1,40,628

Post Transaction 1,10,867

Account Payoff Quote 1,16,616

Total 6,51,416 Transactions in 1 Hour

Transactions Per Second 181 TPS


TIMINGS SUMMARY

Description Expected SLA Webservice 99

th % Response Time

Application Entries 2 seconds Create Loan / Lease/ Line (Entry, Approved

and Verified)

0.308

Check Status <1 second Check Application Status 0.052

Application Update 1 second Update Application 0.046

Payment Messaging

Processing

<1 second Payment Posting Online 0.048

Account Search Request

Service

<1 second Account Details 0.049

Customer Name Search 0.044

Customer SSN Search 0.042


Service

2 seconds Address Update 0.214

Phone update 0.207

Account Payoff Quotes <1 second Payoff Quote 0.055

Total 6,51,416 Transactions in 01 Hrs

Transactions per second 181 TPS

Tuning Parameters

The tuning parameters were applied at various levels (Database/Application).

Database

» The following database instance parameters were changed as part of the runs.

PARAMETER SETTINGS

Parameter Name Parameter Value

sga_max_size 177,920 MB

sga_target 177,920 MB

pga_aggregate_target 65,536 MB

gcs_server_processes 8

db_writer_processes 8


Weblogic Application Server

» The following JVM settings were done.

JVM SETTINGS

JVM Settings Parameter Name Parameter Value Location

Minimum Heap Size -Xms 8g setDomainEnv.sh

Maximum Heap Size -Xmx 8g setDomainEnv.sh

Perm Size -XX:PermSize 1g setDomainEnv.sh

Max Perm Size -XX:MaxPermSize 512m setDomainEnv.sh

Allocate Prefetch Distance -XX:AllocatePrefetchDistance 256 setDomainEnv.sh

Allocate Prefetch Style -XX:AllocatePrefetchStyle 1 setDomainEnv.sh

Aggressive Opts -XX:+AggressiveOpts setDomainEnv.sh

GC Algorithm -XX:+UseConcMarkSweepGC setDomainEnv.sh

GC Algorithm -XX:+UseParNewGC setDomainEnv.sh

GC Algorithm -XX:+UseCMSInitiatingOccupancyOnly setDomainEnv.sh

GC Algorithm -XX:+CMSScavengeBeforeRemark setDomainEnv.sh

GC Algorithm -XX:+CMSClassUnloadingEnabled setDomainEnv.sh

GC Algorithm -XX:CMSInitiatingOccupancyFraction 60 setDomainEnv.sh

GC Algorithm -XX:CMSTriggerRatio 60 setDomainEnv.sh

GC Algorithm -XX:+CMSParallelRemarkEnabled setDomainEnv.sh

GC Algorithm -XX:+UseCMSCompactAtFullCollection setDomainEnv.sh

GC Algorithm -XX:+CMSCompactWhenClearAllSoftRefs setDomainEnv.sh

Tenuring Threshold -XX:MaxTenuringThreshold 15 setDomainEnv.sh

New Size -XX:NewSize 5g setDomainEnv.sh

Survivor Ratio -XX:SurvivorRatio 6 setDomainEnv.sh

GC Threads -XX:ParallelGCThreads 4 setDomainEnv.sh

GC Threads -XX:ConcGCThreads 2 setDomainEnv.sh

Soft References Cleanup -XX: SoftRefLRUPolicyMSPerMB 10 setDomainEnv.sh

Trace Class Unloading -XX:-TraceClassUnloading setDomainEnv.sh

Compression -XX:+UseCompressedOops setDomainEnv.sh

Locking -XX:+UseBiasedLocking setDomainEnv.sh

Pre Touch Flag -XX:+AlwaysPreTouch setDomainEnv.sh

Adaptive Size Policy -XX:-UseAdaptivePolicy setDomainEnv.sh


» The following Weblogic Server settings were configured as part of the runs.

WEBLOGIC SERVER SETTINGS

Weblogic Settings Parameter Name Parameter Value Location

Thread Pool Initial Size

(Logical)

-Dweblogic.threadpool.MinPoolSize 25 setDomainEnv.sh

Thread Pool Maximum Size

(Logical)

-Dweblogic.threadpool.MaxPoolSize 100 setDomainEnv.sh

Socket Readers -Dweblogic.SocketReaders 3 setDomainEnv.sh

Diagnostics Logging -Dweblogic.diagnostics.debug.DebugLogger.DISABLED True setDomainEnv.sh

Management Discovery -Dweblogic.management.discover False setDomainEnv.sh

» The following application connection pool settings were configured as part of the runs.

CONNECTION POOL SETTINGS

Connection Pool Settings Value Location

Minimum Database Connection - Pool 25 via Admin Console

Maximum Database Connection - Pool 350 via Admin Console

Statement Cache Size 100 via Admin Console


Appendix

OATS Report

SOAK TEST REPORT

Session Start Time Session End Time Session Duration

9/18/2015 6:41 9/18/2015 11:41 05:00:19

(18019 sec)

Name Min Max Avg.

Active Virtual Users 7 240 238.432

Virtual Users with Errors 0 0 0

Transactions Per Second 1.184 56.19 54.778

Pages Per Second 1.667 99.533 96.401

Hits Per Second 1 57.133 54.778

Kilobytes Per Second 12.785 307.14 225.485

Totals

Transactions 985999

Transactions with Errors 0

Pages 1735224

Hits 985999

Kilobytes 4058739

Name Min Max Avg. Pass Fail Std.

Dev

90th

%

ofsll_CSS_post_transaction_address_update_load_script.Posting 0.069 0.316 0.074 70412 0 0.01 0.075

ofsll_CSS_post_transaction_phone_update_load_script.Posting 0.062 0.17 0.066 70558 0 0.004 0.067

ofsll_CSS_search_account_details_load_script.Posting 0.015 0.413 0.018 71225 0 0.012 0.019

ofsll_CSS_search_by_customer_by_SSN_load_script.Posting 0.009 0.055 0.011 71349 0 0.002 0.012

ofsll_CSS_search_by_customer_lastname_load_script.Posting 0.009 0.035 0.014 71257 0 0.004 0.018

ofsll_ULN_check_application_status_request_load_script.Posting 0.015 0.146 0.019 106851 0 0.005 0.02

ofsll_ULN_create_loan_load_script.Posting 0.015 0.298 0.132 216354 0 0.084 0.221

ofsll_ULN_update_loan_application_load_script.Posting 0.012 0.27 0.015 71219 0 0.008 0.015

ofsll_plsql_payment_posting_load_script.Payment Posting 0.013 0.053 0.016 118506 0 0.002 0.017

ofsll_plsql_payoff_quote_load_script.Payoff Quote Posting 0.015 0.098 0.021 118268 0 0.003 0.021


STRESS TEST REPORT

Session Start Time Session End Time Session Duration

9/18/2015 5:07 9/18/2015 6:08 01:00:19

(3619 sec)

Name Min Max Avg.

Active Virtual Users 48 800 786.667

Virtual Users with Errors 0 0 0

Transactions Per Second 4.87 186.429 181.706

Pages Per Second 7.267 307.6 298.305

Hits Per Second 3.53 187.467 181.706

Kilobytes Per Second 23.065 901.201 715.01

Totals

Transactions 651416

Transactions with Errors 0

Pages 1069424

Hits 651416

Kilobytes 2563311

Name Min Max Avg. Pass Fail Std.

Dev

90th

%

ofsll_CSS_post_transaction_address_update_load_script.Posting 0.077 0.232 0.092 55379 0 0.023 0.094

ofsll_CSS_post_transaction_phone_update_load_script.Posting 0.072 0.228 0.085 55488 0 0.024 0.088

ofsll_CSS_search_account_details_load_script.Posting 0.014 0.053 0.018 56194 0 0.006 0.019

ofsll_CSS_search_by_customer_by_SSN_load_script.Posting 0.007 0.048 0.01 42190 0 0.007 0.01

ofsll_CSS_search_by_customer_lastname_load_script.Posting 0.007 0.059 0.011 42244 0 0.007 0.015

ofsll_ULN_check_application_status_request_load_script.Posting 0.012 0.058 0.018 56240 0 0.007 0.02

ofsll_ULN_create_loan_load_script.Posting 0.11 0.322 0.231 53967 0 0.041 0.271

ofsll_ULN_update_loan_application_load_script.Posting 0.011 0.058 0.014 56306 0 0.007 0.014

ofsll_plsql_payment_posting_load_script.Payment Posting 0.013 0.058 0.017 116792 0 0.006 0.018

ofsll_plsql_payoff_quote_load_script.Payoff Quote Posting 0.015 0.067 0.021 116616 0 0.007 0.022

Oracle Corporation, World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone: +1.650.506.7000

Redwood Shores, CA 94065, USA Fax: +1.650.506.7200

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the

contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0615 White Paper Title October 2015 Author: [OPTIONAL] Contributing Authors: [OPTIONAL]

C O N N E C T W I T H U S

blogs.oracle.com/oracle

facebook.com/oracle

twitter.com/oracle

oracle.com

50 million active accounts online and batch …...batch transactions for 50 million active accounts....

Documents