WHITE PAPER

DELL EMC DSSD D5 FOR ORACLE DATABASE 12C Reference Architecture and Performance Using Oracle Real Application Clusters and Data Guard

ABSTRACT This white paper introduces the reader to the next-generation performance capabilities of the Dell EMC DSSD D5 rack-scale flash appliance when running mission-critical and business-critical workloads. The paper also will demonstrate the revolutionary performance characteristics of D5 using industry standard workloads and measurement techniques.

October, 2016

2

The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC2, EMC, the EMC logo, Cubic RAID, DSSD, and D5 are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. © Copyright 2016 EMC Corporation. All rights reserved. Published in the USA. 10/16 White Paper HXXXXX

EMC believes the information in this document is accurate as of its publication date. The information is subject to change without notice.

EMC is now part of the Dell group of companies.

3

TABLE OF CONTENTS

EXECUTIVE SUMMARY .......................................................................................................... 4  

INTRODUCTION ...................................................................................................................... 4  

TEST CONFIGURATION ......................................................................................................... 4  Hardware Configuration ...................................................................................................................... 4  

Software Versions ............................................................................................................................... 6  

Operating System Configuration ......................................................................................................... 6  

Oracle Configuration ........................................................................................................................... 7  

Data Scale .......................................................................................................................................... 9  

PERFORMANCE TESTS ....................................................................................................... 10  Transactional Tests .......................................................................................................................... 10  

100 percent Read Workload ....................................................................................................................... 11  

75 percent Read, 25 percent Write Workload ............................................................................................. 15  

Data Warehouse Tests ..................................................................................................................... 18  

Twelve Node Parallel Execution ................................................................................................................. 19  

Single Node Parallel Execution ................................................................................................................... 21  

Combined Tests ................................................................................................................................ 23  

CONCLUSION ........................................................................................................................ 27  

4

EXECUTIVE SUMMARY The Dell® EMC® DSSD™ D5™ rack-scale flash appliance is one of the fastest storage devices on the market today. This paper details how DSSD applies that performance to the Oracle Database and draws conclusions about the impact of such revolutionary performance on business and database operations in general. Even with databases that have been designed to minimize access to storage, the D5 provides an immediate speed up for many business process and enables new, more agile, approaches to application architecture in the future.

INTRODUCTION DSSD D5 is a revolutionary storage product. It is revolutionary not only because it is an order of magnitude faster than any other general-purpose storage product, but also because of its impact on Oracle databases. Not only does the appliance speed up existing workloads, but it also enables entirely new approaches to database deployment.

Consider recent history in storage performance: In 2010, the final generation of disk-based storage arrays were launched, offering upwards of 100K IOPS, a couple of terabytes of cache, and perhaps one or two GB of bandwidth, in a five floor tile footprint. Next came the all-flash array, with a much smaller footprint, half a million IOPS, and around 3 GB/s bandwidth. Hot on the heels of the all-flash array comes D5, designed from the ground up for pure performance. D5 offers flexible interfaces so that applications like Oracle can leverage the standard block driver and making DSSD look like a local disk drive. D5 provides next-generation applications with the ability to leverage the highly optimized direct API (libflood), enabling it to deliver up to 100 GB/s of bandwidth, as low as 100µs latency, and up to 10 million IOPS.

This paper utilizes some of the workloads used by DSSD product development internally to demonstrate the raw performance of the platform in a real Oracle scenario. Although the actual workloads themselves are artificial, they are deployed using the full Oracle Database software stack, just like a real application. There are no proprietary optimizations made in any of these configurations over and above using standard Best Known Methods for deploying Oracle Databases on D5. No tuning has occurred, including in the DSSD software, other than anti-tuning to force more I/O to be issued as physical I/O in order to demonstrate the capabilities of the platform. Wherever practical for a paper such as this, performance metrics are shown using industry standard tools.

With such radical increases in performance, the D5 offers organizations a new lease on life. The formerly impractical dream of true query-by-example can now be a reality, as I/O bandwidth ceases to be the constraining factor to a query ever completing. Data sets can be even larger, because D5 delivers full performance across the entire working set, not just a tiny cached portion. DBA workloads can be reduced, as the utility value of physical schema optimizations such as materialized views, granular partitioning schemes, and elaborate secondary indexing schemes become less important.

The DSSD D5 is the only all-flash product to deliver the full performance capabilities of the underlying flash media all the way to the application software, and to provide best-in-class data reliability through patented Cubic RAID™ technology.

TEST CONFIGURATION

HARDWARE CONFIGURATION This test was designed to test three important dimensions of Oracle Database operation:

1. High volume transaction processing

2. Data analytics on very large data sets

3. Data Guard replication of a very busy transaction processing environment

Two independent clusters were used to implement the first two scenarios, with the Active Data Guard replication setup between the two clusters. The two clusters were physically located in adjacent racks of the same data center, which allowed high bandwidth transmission between the two clusters. This architecture might be selected when analytical queries are to be kept distinct from the transactional database but require live data access. Active Data Guard provides access to the live data locally on the same cluster as the data warehouse. This architecture also provides a way to focus on the performance of the components rather than to be quickly bottlenecked on the bandwidth of a WAN connection, which affects all storage products the same way.

5

 

Figure 1. High-level Hardware Overview Each server in both clusters was specified identically, as follows:

• Dell PowerEdge R630 1U rack mount server

• 2x Intel Xeon Processor E5-2699 v4 processors

• 256GB DDR4 Memory @ 2400MHz

• 2x 10Gbps Network Interface Cards (bonded using 802.3ad LACP mode)

• 1x 1Gbps Network Interface Cards (management interface)

• 2x EMC DSSD Dual-Port Client Cards (four ports total)

• Internal storage used for boot/root only

Each cluster comprised twelve nodes, each configured as a logically independent cluster and connected to its own DSSD D5 configured as follows:

TRANSACTION PROCESSING DATA ANALYTICS AND DATA GUARD STANDBY DSSD D5, active/active, fully redundant controllers DSSD D5, active/active, fully redundant controllers

36 x 2TB Flash Modules (72 TB Total) 36 x 4TB Flash Modules (144 TB Total)

6

Figure 2. Transactional D5 Flash Modules

Note: For this lab test, the physical network infrastructure was shared between the clusters to economize on port count. This extended to leveraging the cluster interconnect as the physical transport for Data Guard, which we would not recommend for production use. This sharing was carefully managed at the workload level during testing.

SOFTWARE VERSIONS

• CentOS Linux release 7.2.1511 (Core), kernel 3.10.0-327.13.1.el7.x86_64

• Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production (incl. APR2016 PSU)

• EMC DSSD D5 Flood Release 201602.3.1 (Willow Prime)

OPERATING SYSTEM CONFIGURATION The Operating System was not subject to significant tuning for these tests. Tuning tasks carried out:

• Disable some unnecessary services and background jobs

7

• Install Oracle Pre-install RPM (minus installing the UEK kernel)

• Configure 802.3ad bonded 10Gbps network

• Ensure CPUfreq scaling governor is set to Performance

• Add numa=on boot flag

In addition, the R630 server was configured in Performance mode to disable power saving states.

ORACLE CONFIGURATION Each cluster was configured according to the Deploying Oracle Databases on EMC DSSD D5 Best Known Methods Guide. Configuring included provisioning between two and six block device objects each for the main ASM disk groups, and a selection of smaller objects for the Grid Infrastructure components such as OCR, Voting files, and MGMT database. It is an important distinction that many of the traditional considerations used when deploying storage apply to the D5. Notably, the D5 is a DMA-based storage device, and so many of the usual notions of SCSI queues do not apply. In fact, creating excess block objects can be detrimental to performance—a smaller number is preferred.

These are the block objects created for the Transactional cluster:

brw-rw---- 1 oracle dba 252, 0 Sep 7 22:10 /dev/dssd0000 brw-rw---- 1 oracle dba 252, 16 Sep 7 22:25 /dev/dssd0001 brw-rw---- 1 oracle dba 252, 32 Sep 7 22:23 /dev/dssd0002 brw-rw---- 1 oracle dba 252, 48 Sep 7 22:23 /dev/dssd0003 brw-rw---- 1 oracle dba 252, 64 Sep 7 22:23 /dev/dssd0004 brw-rw---- 1 oracle dba 252, 80 Sep 7 22:23 /dev/dssd0005 brw-rw---- 1 oracle dba 252, 96 Sep 7 22:23 /dev/dssd0006 brw-rw---- 1 oracle dba 252, 112 Sep 7 22:10 /dev/dssd0007 brw-rw---- 1 oracle dba 252, 128 Sep 7 22:25 /dev/dssd0008 brw-rw---- 1 oracle dba 252, 144 Sep 7 22:25 /dev/dssd0009 brw-rw---- 1 oracle dba 252, 160 Sep 7 22:25 /dev/dssd0010 brw-rw---- 1 oracle dba 252, 176 Sep 7 22:25 /dev/dssd0011

These block objects were further configured for device naming persistence and friendly naming, as follows:

lrwxrwxrwx 1 root root 11 Sep 7 22:10 /dev/asmdisks/OraFRA000441_00 -> ../dssd0007 lrwxrwxrwx 1 root root 11 Sep 7 22:24 /dev/asmdisks/OraOCR000441_00 -> ../dssd0009 lrwxrwxrwx 1 root root 11 Sep 7 22:24 /dev/asmdisks/OraOCR000441_01 -> ../dssd0010 lrwxrwxrwx 1 root root 11 Sep 7 22:15 /dev/asmdisks/OraOCR000441_02 -> ../dssd0011 lrwxrwxrwx 1 root root 11 Sep 7 22:10 /dev/asmdisks/OraRedo000441_00 -> ../dssd0008 lrwxrwxrwx 1 root 11 Sep 7 22:10 /dev/asmdisks/OraVol000441_00 -> ../dssd0000 lrwxrwxrwx 1 root root 11 Sep 7 22:17 /dev/asmdisks/OraVol000441_01 -> ../dssd0001 lrwxrwxrwx 1 root root 11 Sep 7 22:17 /dev/asmdisks/OraVol000441_02 -> ../dssd0002 lrwxrwxrwx 1 root root 11 Sep 7 22:17 /dev/asmdisks/OraVol000441_03 -> ../dssd0003 lrwxrwxrwx 1 root root 11 Sep 7 22:17 /dev/asmdisks/OraVol000441_04 -> ../dssd0004 lrwxrwxrwx 1 root root 11 Sep 7 22:23 /dev/asmdisks/OraVol000441_05 -> ../dssd0005 lrwxrwxrwx 1 root root 11 Sep 7 22:17 /dev/asmdisks/OraVol000441_06 -> ../dssd0006

These were used to build standard ASM disk groups with External Redundancy—the built-in Cubic RAID of the D5 makes any further storage redundancy unnecessary.

8

The following screenshots are from the Transactional and Data Warehouse clusters respectively:

Figure 3. ASM Disk Groups—Transactional Cluster

9

Figure 4. ASM Disk Groups—Data Warehouse Cluster

The Transactional disk groups are self-explanatory, but the Warehouse disk groups warrant some further clarification:

Diskgroup Name Purpose REDO_ER Redo for data warehouse database GRIDMISC Clusterware components DG_SYS UNUSED SYS_ER Container Database for data warehouse DG_SLOB20T Data Guard Physical Standby for Transactional System TPCDS_ER Pluggable Database files for data warehouse FRA Fast Recovery Area DG_SLOB441 UNUSED DG_REDO Redo and Standby Logs for Data Guard Standby Application-specific Oracle parameters will be detailed in the respective workload sections later in this document.

DATA SCALE With the ever-increasing volumes of data that businesses must store to remain competitive, database sizes continue to grow at a rate far greater than just year-on-year growth. Not only that, but the percentage of that database which is actively in-use on a daily basis continues to increase. This data is known as the working set. Historically a transactional database would comprise a large quantity of historical data and a small amount—perhaps 5 percent—which is being read and/or modified frequently. This is a small working set of

10

the entire database, and this configuration has been served well by Oracle buffer caches and storage array head caches for decades. Modern databases have much larger working sets, with perhaps 90 percent of the entire database being scanned and/or updated every day. One example of this is in financial regulatory reporting, where deliverables such as risk exposure require an increasing amount of historical context.

These tests have been designed to demonstrate the performance of the D5 in these very large working set environments. Accordingly, both the transactional and the data warehouse components of these tests are sized at 20 TB and 15 TB respectively—far larger than any cache available on the market today. This is a very important factor in this testing, and comparisons can only be made with other platforms if those platforms also test with a similar scale.

PERFORMANCE TESTS As indicated in the Hardware Configuration section, the performance tests were designed to interoperate and show specific real-life situations, as well as run in isolation to cover those situations and show unimpeded performance metrics.

The Transactional Cluster hosted a single database (non-CDB) with twelve instances. This database was copied via RMAN to the Data Warehouse cluster to create a Data Guard Physical Standby. The Data Guard Standby was configured in Maximum Performance mode, with the standby using Active Data Guard functionality to enable the database to be open (read-only) while simultaneously being kept up-to-date in Real-Time Query mode. The concept being modeled here is to use Data Guard to make live transactions from the Transactional Cluster immediately available to the Data Warehouse cluster, where they may be combined with the data in the warehouse itself for full contextual analysis of all business data. Scenarios were run both with and without the Data Guard replication active.

The two main workloads are described here:

Cluster Workload Description

Transactional Cluster Silly Little Oracle Benchmark (SLOB) https://kevinclosson.net/slob/

Highly randomized, single-block access workload akin to a worst-case transactional workload. Easily configured to vary the read/write mix, scale, and other useful attributes. Uses the Oracle software to generate I/O load using SQL, the most natural way to exercise the entire software stack. SLOB was built in the multi-schema configuration, and Oracle configured to minimize RAC overhead for this testing. Database built with a 20 TB scale factor to simulate a very large working set.

Data Warehouse Cluster Parallel Query Uses TPC-DS data generator to create a 15TB data set and then perform parallel queries on the STORE_SALES table. Queries were designed to minimize execution slave startup times and RAC contention.

TRANSACTIONAL TESTS Two scenarios were executed using SLOB: 1. 100 percent Random Read (8KB)

2. 75 percent Read, 25 percent Write workload: A heavy OLTP workload

11

These scenarios were tested with differing load levels until the optimal I/O rates were observed without exceeding 1 ms latency (and preferably keeping latency much lower, if that meant sacrificing a few more IOPs). The Oracle init.ora configuration was unchanged across all these scenarios:

db_block_size 8192 8KB Database Block db_cache_size 402653184 Small cache to create higher

physical I/O pga_aggregate_target 27018657792 db_file_multiblock_read_count 16 Optimized for DSSD D5 (but

not used by SLOB) shared_pool_size 21474836480 _db_block_prefetch_limit 0 Disable prefetching _db_block_prefetch_quota 0 Disable prefetching _db_file_noncontig_mblock_read_count 0 Disable prefetching _disk_sector_size_override TRUE Allow 4KB redo logs

100 PERCENT READ WORKLOAD

SLOB was executed using only ten of the available nodes due to temporary unavailability of two nodes, but the results are largely representative of those that could be achieved with the full complement of twelve. The SLOB configuration parameters were as follows:

THREADS_PER_SCHEMA=400 UPDATE_PCT=0 RUN_TIME=900 SCALE=1600G WORK_UNIT=64 Over the 15 minute period of the test, the D5 delivered an average of 4.96M 8KB IOPS, as shown in the AWR report from the test:

Figure 5. 100 percent Read SLOB-AWR Statistics

12

These figures are sustained and averaged over the whole duration, and are corroborated and enhanced by the Oracle Enterprise Manager Performance Home screen, which shows significant peaks over the 5M IOPS level:

Figure 6. Oracle Enterprise Manager Performance Home Shows 6M IOPs

13

The 5M IOPs can also be observed in the DSSD Browser interface:

Figure 7. DSSD BUI

14

The latency for this test was 800µs, as shown in the Top Timed Events section of the AWR RAC report:

Figure 8. Single Block Read Latency This test demonstrated the high IOPs rates possible with the D5. At IOPS rates that are slightly lower, we can see significant reduction in read latency times. The following AWR data is from a scenario identical to that above, but with a lower session count—one per CPU logical processor (88 sessions per node). This is an extremely representative configuration for an OLTP environment that uses connection pools. The first data point shows a reduced latency at 200µs:

Figure 9. Single Block Latency (Typical connection pool config)

15

This reduced latency still sustains 3.6 million IOPs:

Figure 10. IOPs (Typical connection pool config)

75 PERCENT READ, 25 PERCENT WRITE WORKLOAD

Using the same ten node configuration as the 100 percent read test (and the same init.ora configuration), SLOB was configured as follows:

THREADS_PER_SCHEMA=100 UPDATE_PCT=25 RUN_TIME=900 SCALE=1600G WORK_UNIT=64

Over the 15 minute runtime, the D5 sustained ~2 million read IOPs and ~500 thousand write IOPs, in conjunction with almost 400 MB/s in redo log writes:

16

Figure 11. 75 Percent Read, 25 Percent Write SLOB-AWR Statistics

This is echoed in the Enterprise Manager Performance Screen:

17

Figure 12. Oracle Enterprise Manager Performance Home—75/25 SLOB

18

It is echoed again in this zoomed-in extract from the DSSD UI, which shows the combined 20 GB/s of read and 5 GB/s of write bandwidth:

Figure 13. Throughput for SLOB 75/25—DSSD UI Latency for this scenario was 600µs for both reads and writes, though writes were of variable size (not just single block), with an average of 39 KB each:

Figure 14. AWR Top Timed Events—75/25 SLOB

DATA WAREHOUSE TESTS Data warehouse workloads are all about bandwidth—and having more bandwidth yields a huge impact on the execution times of queries. This testing deliberately focuses on a single query, as follows:

select /*+ full(ss1) parallel(ss1,<DEGREE>) */ count(*) from tpcds.store_sales ss1;

Keeping the query as simple as possible minimizes the non-I/O steps of the execution plan: No sorts, no joins, no filtering, and minimal RAC overhead. This normalizes the test result against any tests from other vendors, because all those operations are subject to other factors than I/O. Here we focus solely on the scan bandwidth, the lifeblood of data warehousing.

19

The query is executed discretely on a single node using the parallel_force_local=true setting in the issuing session. The query is then scaled out across the other nodes in the cluster by issuing additional identical requests to the other nodes. This technique ensures that the startup times for the parallel execution are kept as low as possible by eliminating inter-node process coordination. One side effect of this method is that more data must be scanned as more nodes are added: Twelve node tests will scan 12 times more data than a single node test. This could have been avoided using partitioning of the table.

The same scan rates shown here can also be observed using Oracle’s multi-node parallel execution framework, but the execution startup times are longer and exhibit an artificially reduced scan rate in AWR reports as a result.

Relevant Oracle parameters were as follows:

db_block_size 8192 8KB Database Block db_cache_size 402653184 Small cache to create higher physical I/O pga_aggregate_target 27018657792 db_file_multiblock_read_count 16 Optimized for DSSD D5 (but not used by SLOB)

shared_pool_size 42949672960 _disk_sector_size_override TRUE Allow 4KB redo logs parallel_adaptive_multi_user FALSE Disables any DOP auto-tuning parallel_min_servers 1536 Prespawn all PX processes

TWELVE NODE PARALLEL EXECUTION

The first test scenario shows maximum bandwidth by using all the nodes in the cluster to concurrently scan the table. This showed an average sustained bandwidth of almost 50 GB/s across the period:

Figure 15. 50 GB/s Scan Throughput One observation made during this test was that there was still a reasonable amount of performance capacity remaining in the D5 during this test, indicating additional bandwidth (10 to 20 percent) could be expected with some tuning.

20

The same bandwidth is observed in Enterprise Manager:

Figure 16. 50 GB/s Shown in Enterprise Manager

And in the DSSD BUI:

Figure 17. 50 GB/s in DSSD BUI

21

Figure 18. 52.7GB/s shown in DSSD BUI

This level of bandwidth all the way into Oracle provides all the query processing power where it is needed—in the Oracle Database engine. Queries of even moderate complexity cannot rely on offload technologies and need the database itself to process the data. With this level of bandwidth into the database many queries cease to be dominated by I/O, allowing the server platform to be more effectively utilized.

SINGLE NODE PARALLEL EXECUTION

Another important data point is how much bandwidth is available to a single node in the cluster. This is important because production data warehouses will not typically allow a single query to dominate the whole cluster. Instead, they will be confined to a much smaller subset of the nodes. Therefore, single node throughput is a critical metric for data warehousing.

22

Figure 19. 12.7 GB Single Node Throughput—AWR The above AWR extract shows a throughput of 12.7GB/s when running on a single node, which is good realization of the theoretical port bandwidth of 16 GB/s. This can also be seen in Enterprise Manager and the DSSD BUI:

Figure 20. 12.7GB Single Node Throughput—Enterprise Manager

23

Figure 21. 12.7GB Single Node Throughput—DSSD BUI

COMBINED TESTS In these tests we run the following scenario:

3. Start 75/25 workload on Primary and observe load on Primary D5

4. Observe load on Secondary D5 for just the Standby Transport and Apply

5. Start high bandwidth parallel query workload on Data Warehouse cluster and observe changes in throughput

The 75/25 workload is shown first, behaving as expected – identical to the figure shown in

Figure 13 where no Data Guard replication was taking place.

24

Figure 22. No Impact Observed on Primary Database

25

The workload on the Data Warehouse D5 is shown below, approximately 2.5 GB/s read and 2.5 GB/s write:

Figure 23. Data Guard Standby—Transport and Apply only

26

Now let’s execute a high bandwidth query on the Data Warehouse/Standby cluster:

Figure 24. Data Guard Transport and Apply with Concurrent Heavy Query Workload The scale of the bandwidth chart gets re-scaled in the above figure due to the increase in read bandwidth caused by the query. However, the write IOPs only reduce by around 15 percent and the Data Guard throughput stayed ahead of the available network bandwidth (20 Gbps theoretical).

27

The following screenshot shows the throughput that was sustained at the Standby:

Figure 25. Data Guard Throughput This screenshot shows extremely high throughput figures for Data Guard replication. When the network is able to support the bandwidth and the redo ship rate requries it, we were able to demonstrate 1.64 GB/s—a figure not normally seen in Data Guard architectures.

The six-minute apply lag was caused because of the slightly unusual way the TNS infrastructure was configured in the lab (redo transport over a private, non-routed network), which interfered with the two-way communication of Data Guard. This configuration will be revisited in a future whitepaper, where the scope will be more focused on Data Guard alone.

CONCLUSION The Dell EMC DSSD D5 provides a level of performance that is unmatched by any other storage product on the market today. It has been designed from the ground up to provide these levels of performance, and this paper has demonstrated how that performance translates to Oracle workloads.

For OLTP environments, this means higher transaction rates, faster end-user response times, and more batch throughput even for huge working sets of data. For Data Warehouse and Analytics customers, it provides access to data volumes that were previously difficult to query, and provides sufficient performance to eliminate many of the arduous schema optimization efforts normally associated with large data volumes. Just scan it! When the two are combined, the levels of performance available with the D5 mean that it is finally a reality to query live data as part of a Data Warehouse with zero impact to the production OLTP database.

For more information on Dell EMC DSSD storage, please visit https://www.emc.com/en-us/storage/flash/dssd/dssd-d5/index.htm