1. oracle's approach to infrastructure and application testing mughees a. minhas vice...

Oracle's Approach to Infrastructure and Application Testing

Mughees A. MinhasVice President, Product ManagementOracle

33

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 164

Why is Testing Important?

Cost of 1 hour unplanned production downtime

Percent of unplanned downtime can be eliminated through pre-deployment testing and better QA/Development collaboration

$100,000+

Incremental cost of fixing problems in production80-10,000x

80%

Source: Gartner

Percent of applications deployed without testing75%

Proper testing saves money and reduces unplanned downtime.


Testing Challenges

Test quality: despite significant testing, critical issues go undetected

Test development: Test tools and development of comprehensive tests is error prone and time-consuming

Operational agility: Peripheral activities like setting up hardware and deploying applications consumes up to 50% of the time

Why are changes not being tested?

Provison application under Test

Provision testing tool

Run functional tests

Run load testsDetect bottlenecks

Tune Application under Test

Apply patches & re-test

Develop test scripts

Deploy Test scripts

Detect issues

Define testrequirements


Testing Challenges












Deploy Test scripts

Detect issues


7

Application Infrastructure Changes

Application Changes

O/S

Storage

DBMS

Middleware

• Test infrastructure changes separately from application changes• Test one change at a time• Specialized tools can result in much higher quality and efficient testing• Proper test systems and test data mandatory for good testing results

Best Practice Approach to Quality Management

Infrastructure Testing Tools

InfrastructureTesting tools

Application Replay

Real Application Testing

Application Testing toolsApplication Testing Suite

8

Infrastructure Testing

10

Data-Tier Testing: SQL Response Time

SQL problems are #1 cause of poor application performance • Siebel has over 100K unique SQL statements• E-Business Suite has over 600K unique SQL statements

Regression of 1% of SQL statements can make application unstable

Perform SQL unit performance testing• Use SQL Performance Analyzer (SPA) to proactively detect all regressions

resulting from a change

11

• Test and measure change impact on response time using real, production SQL statements

• Identify all plan changes and/or regressions

• Eliminate SQL regressions with SQL Tuning Advisor and SQL Plan Baselines

• For data warehouse workloads SPA Exadata simulation helps estimate I/O savings without provisioning Exadata hardware

Response Time Testing with SPA

SQL Plans + Statsbefore change

Compare SQL Performance

Analysis ReportSQL Workload

STS

SQL Plans + Statsafter change

12

Data-Tier Testing: Throughput and Concurrency

SQL unit performance testing not sufficient for all types of changes

Problems related to concurrency manifest only under load testing• Migration from SI to RAC

Challenge is to generate production-like workload• Major telco implementing Siebel on RAC rolled back go-live because they tested

with unrepresentative workload – encountered socket limits at 6000 users

Use Database Replay to test your application infrastructure with real production workloads

13

Throughput Testing using DB Replay

• Database load and performance testing with real production workloads

– Production workload characteristics such as timing, transaction dependency, think time, etc., fully maintained

• Test and measure transaction throughput improvements

• Identify application scalability and concurrency problems

• Remediate issues pre-production for risk-free migration

Test

Capture Process Replay Analysis &Reporting

Production

Replay DriverClients

Storage Storage

14

Mid-Tier Testing: Application Replay

• Captures and replays actual production workload

• Maintains all production characteristics during replay such as timing, order, concurrency, dependency, etc.

• Like DB Replay but for the mid-tier

• Benefits– Faster testing

– No script development or maintenance

– Better testing– Real production workloads

– Broader workflow coverage: captures all flows that arise in practice

Authentic replay with minimal divergence

Automatic user requests capture

Automatic Http-DBSynchronization

Automatic request correlation

15

0

20

40

60

80

Tim

e T

aken (D

ays)

Install & Setup UnderstandApplication

Usage

Identify KeyTransactions

GenerateWorkload

Run Test

DB Replay

LoadRunner

Benefits of Infrastructure Testing SolutionsComparison of LoadRunner & Database Replay

2

205

4

80

24 20

DB Replay: ½ month LoadRunner: 7 ½ months

Total Testing Time

500

15X

149 Days

To:From:

Complete workflowsPartial workflows

Low riskHigh risk

AutomatedManual intensive

Production workloadsArtificial workloads

Days of developmentMonths of development

Benefits of Infrastructure Testing Solutions

17

Application Testing

18

Application Testing

• If change is within the application– New version or patch– Any customization of application – New functionality

• Application Testing Suite– Synthetic testing of any changes to application stack or the

application logic

19

Application Testing

• Modularization

• Maintainability

• Script Robustness

• Portability

• Auto-correlation

Basic Requirements

20

• Ability to break complex flow into chained flows that are shorter and self-contained – ERP apps like EBS have test flows that often span multiple modules– Test data (input as well as output/results) must move across scripts

Enter & Book order

Pick Release & Ship Order

Generate Invoice

Create Receipt

Order Mgmnt Shipping Receivables Receivables

Result output

Result output

Result output

Order to Cash business flow

Modularization

21

Maintainability

• Ability to create function libraries for frequently used functions– Function blocks like "log-in"/"log-out" or other well used functions may be

added as function libraries to make test assets easier to maintain and use

22

Portability

• Ability to port test assets to different environments– Example: Parameterize FORMS connection for better ”portability”

nca.connect(317,ConnectMode.Socket,"{{db.myEBSsystem.hostname,EBSServer.mycompany.com}}",Integer.parseInt(eval("{{db.myEBSsystem.Forms_Port,9000}}")),"");

Recorded valueDatabank file column usedDatabank fileIndicates Databank variable

Update Databank file with new HOST and PORT info when a new test system is used instead of changing in ALL scripts

23

Script Robustness

• Ability to identify objects that can withstand minor changes – Eliminates need to redevelop scripts frequently

• Oracle Functional Testing’s accelerators for Oracle applications provides pre-built and optimized object identification rules for EBS, Fusion, Siebel– Provides best possible and most reliable/robust scripts – Can reduce script development by up to 50%

Functional Testing

24

Auto-correlation

• Ability to auto-correlate so that manual modifications are not required – Quicker scripting and more time to do actual testing

• Oracle Load Testing accelerators for Oracle applications provides pre-built correlation rules for:– E-Business Suite– Fusion– Siebel– JD Edwards Enterprise One

Load Testing

25

Oracle Application Testing Suite

• Supports – Functional/regression testing

– Load/Performance testing

– Test Management

– Web, Siebel, EBS/Forms, ADF, Flex, DB & WebServices

• Reduce scripting time by ~50%

• Automatic test asset (script) generation – Load scripts auto-generated using RUEI and Real

Application Testing

• Smart integration with EM diagnostics– JVM Diagnostics for mid-tier

– DB diagnostics for data-tier

26

Application Testing Suite

• Integrated solution for automated functional, load testing and test process management

• Custom testing accelerators for Oracle Applications, e.g., EBS, Siebel, Fusion, JD Edwards

• Test Starter Kits for both functional and load testing

• OpenScript integrated scripting platform for load and functional testing– Intuitive visual scripting interface

– Java IDE for powerful extensibility

• Built-in server monitors and integration with EM Diagnostics to identify performance bottlenecks under load

• Scalable enterprise architecture built on WLS and Oracle DB

Key Features

27

Test Data Management

28

Data Masking

• Application integrity– Generation of data compliant with the application’s requirements

• Data integrity– Maintaining PK/FK relationship integrity even when primary keys are masked

• Automated data-relationship discovery– Discovery of sensitive data and relationships across enterprise databases

• Enterprise-wide rule definition– Defining masking policies for all enterprise databases from a central dashboard

• Performance– Time taken to execute the entire masking process

Basic Requirements

31

Oracle Data Masking

• Deploy secure test system by masking sensitive data• Sensitive data never leaves the production database• Extensible template library and policies for automation• Sophisticated masking: Condition-based, compound, deterministic, at-source • Integrated masking and cloning

LAST_NAME SSN SALARY

HODGES 111—23-1111 60,000

JOHNSON 222-34-1345 40,000

LAST_NAME SSN SALARY

AGUILAR 203-33-3234 40,000

BENSON 323-22-2943 60,000

Production Test

32

Masking Techniques Support

• Compound masking– Multi-column groups, e.g. address (street, city,

state, zip)

• Condition-based masking– Mask national identifiers, e.g. social security

number, by country-specific format

• Deterministic (Repeatable)– Repeatable consistent masking, e.g. referential

integrity across databases, or consistency across refreshes

• Reversible– Key-based format preserving masking, e.g.

outsourced data processing

• Shuffle– Randomization within population set.

• Perturbation– Random mask generation around a value, e.g.

date of birth

• At-Source Masking– Ability to mask data as it leaves production, i.e., at

the source

33

• Automatic data extraction rules from ADM• Estimate subset before execution• Parameter-driven dynamic subset generation• Unique to Oracle: Data Pump-based subset creation

Oracle Data Subsetting

Define new

Application Data Model

Create Data

Subset Definition

Extract Data

Subset

Export / Import

In-Place Delete

ProductionTest

34

Cloud Delivery Platform


Testing Challenges












Deploy Test scripts

Detect issues



How To Get Agility

Automate time consuming, repetitive tasks

– Provision test tools and application

– Provision scripts

– Orchestrate testing process end-to-end

What can be improved?

Simplify Reporting Deep diagnostics & root cause analysis








Deploy Test scripts

Detect issues


Automate the testing process


What Is The Solution?

Infrastructure-as-a-Service (IaaS)– Provisioning of Virtual Machines on provided infrastructure

This will only resolve the provisioning of Application under Test and Test tools

– only 10-15% of the full solution

IaaS based testing cloud







Define testrequirements and process

Deploy Test scripts

Detect issues IaaS





Software-as-a-Service (SaaS)– Software (for test automation) as a cloud solution. Available on-demand

Only addresses the test execution and (possibly) issue/bottleneck identification No provisioning of the Application under Test Possibly no monitoring of internal applications, depending on solution

SaaS based testing cloud







Deploy Test scripts

Detect issues SaaS

Define testscripts




Combination of– Provisioning of cloud resources (IaaS)

– Software on demand (SaaS)

– Test orchestration and automationof the whole testing process including test scripts

– Application tuning and patching

– Result reporting

Built with semantic understanding of testing artifacts like testing tools, applications, test scripts, etc.

– it is not just VMs

IaaS and Saas Combination Solution

Oracle Testing-as-a-Service Platform

IaaS SaaS

Orchestration

Orchestration, patching, tuning...


Oracle Testing-as-a-ServiceSelf-service private testing cloud

• Scripts• Environments

Test Library

• Test Drivers• Test Assets• Test Application

Test Lab Provisioning • Load Test

• Functional Test

TestExecution

•Monitoring•Resource Metering•Rule-based Chargeback

Monitoring & Chargeback

IaaS Platform

Cloud Policy Framework

Self-Service Solutions

http(s)

Test Admin, Test Designer & Managers

Test Engineers

42

Summary

© 2012 Oracle

Productivity Gain

Reduced capital expendi-ture

0

1

2

3

4

5

Year 3

Year 2

Year 1

$(M)

Forrester Real Application Testing TEI Study Multi-customer Study Demonstrates Strong ROI, Business Value

• Risk adjusted ROI of 224% with a payback period of 5.9 months

• Increased productivity gain to nearly $4.8M over 3 years

• Reduced capital spending on servers by 50%

• Increased availability and uptime of critical business applications

(Excludes $20,655 annual business up time savings)

44

Oracle Testing Solutions

Testing Application Changes

Testing Infrastructure Changes

Test Data and Lab Management

Oracle Functional Testing

Oracle Load Testing

Oracle Test Manager

SQL Performance Analyzer

Application Replay

Data Masking Data Sub-setting

Database Replay

Data Discovery and Modeling

Cloud ServicesTesting Tools Test Data Management

Cloud Delivery Platform Testing as-a-Service


Key Take-Aways

Distinguish between infrastructure vs. application changes Don’t employ “one-tool-fits-all” approach

– Select right tool for right problem

Proper test systems and test data mandatory for good results

Gain operational agility with testing cloud services

1. oracle's approach to infrastructure and application testing mughees a. minhas vice...

Documents