what is and what is not exadata engineered...

Database Machine – What it is and what is not

What is and What is not Exadata Engineered System


Introduction to Database Machine

Fully integrated platform for Oracle Databases

● High performance and High availability solution● Consolidation platform● Answers all workload types● Fast implementation● Balanced configuration● Based on Oracle Storage Servers technology


Introduction to Database Machine

• 14 Storage Servers (High Capacity / High Performance)

• 8 Database Nodes

• 3 Infiniband Switches (40Gb/s)

• Cisco Ethernet Switch

• 2 PDUs


Storage Servers technology

Architecture depend on Exadata version, X3-2

● 64 bit Intel Sun Fire servers preinstalled with

– Oracle Linux/Solaris for x86– Drivers and Utilities– Exadata Storage Server Software

● Part of Infiniband Network● Smart Storage Operations Capable● HP , HC configurations● Equipped with Smart Flash Cache


Storage Server Hardware Specification

Processors 12 Intel CPU Cores

System memory 64 GB DDR4

Flash Drives 1.6 TB

Disk Controller Host Bus Adapter, 512 MB Battery Backed Write Cache

Infiband Network Dual-Port QDR 40 Gb/s HBA

Remote management ILOM Ethernet Port

Power Supplies 2 x Hot-Swappable


Up to 14 Storage Servers in Full Rack configuration

All High Capacity or all High Performance servers

Storage Server Parameters Specification

Parameter HP (High Performance) HC (High Capacity)

Raw Capacity 7.2 TB 36 TB

Data Capacity 3.25 TB 16 TB

Raw Disk Throughput 1.8 GB/s 1.3 GB/s

Flash Throughput 7.25 GB/s 6.75 GB/s


Database Nodes X3-2 Hardware Specification

Processors 16 Intel CPU Cores,


Disk Controller Host Bus Adapter, 512 MB Battery Backup Write Cache

Network Interfaces Dual-Port QDR 40 Gb/s HBA Infiniband4 x 1/10Gb Ethernet (copper)2 x 10Gb Ethernet (optical)




Infiniband Network Overview

● 40 Gb/s speed in each direction ● Zero-loss Zero-copy Datagram Protocol (ZDP)

– Zero copy implementation of RDSv3● There is no difference for tools like ping● Used for RAC interconnect, storage networking,

external connectivity

– RDS is used for RAC interconnect– RDS is used for DbNodes – Storage Servers

communication● With Infiniband you can connect up to 18 Exadata

Racks


Infiniband Network Overview


Exadata Network Overview


Smart Flash Cache

• Intelligently caches database objects (not data blocks)

• Replaces slow mechanical I/O operations

• Smart Flash Logging ● Supports "log file sync/log file parallel write"● Parallel write to Disk Controller Cache and Flash

memory

• For X3-2 Full Rack ● Maximum Flash bandwith 100 GB/s● Maximum Flash read IOPS 1,500,000● Maximu Flash write IOPS 1,000,000


Smart Flash Cache

• Exadata uses Flash PCI cards , not flash disks

• It is not disk replacement

• Exadata Storage Software decides how and when to use flash storage

• ESS Can simultaneously scan from flash and disks

• Traditional Storage Arrays● Controllers are bottleneck for storage subsystem● Adding Flash Layer makes bottleneck more visible● Does not have knowledge on type of workload cached


Smart Flash Cache

• Each PCI Flash Card provides 1,33 GB/s throughput

• Exadata scales linearly across 4 cards

• DBAs and Developers can influence how DB objects are keeped on Flash Cache (CACHE,NOCACHE,EVICT, DEFAULT,KEEP,NONE)

• In practice it is common to see > 90% 'flash cache hit'

• Can be combined with HCC

• Exadata Software automatically detects and bypass poorly performing or crippled flash

• IO Resource Manager can manage I/O on flash and disks between consolidated databases


Smart Flash Cache• Can work in two configurations

● Write Through

● Write Back (write intensive applications – "free buffer wait")• Cached type of workload

● Frequently accesed data and index blocks

● Controlfile reads and writes

● File header reads and writes

● KEEP objects• Not - Cached type of workload

● Backup related I/O

● Data Pump I/O

● Datafile formatting

● Table scans, which are suppose to monopolize Flash

● I/Os mirrors copies are managed intelligently


Smart Flash Cache• Write Back vs Write Through

● Generally Session in not waiting for dirty buffers writes (DBWR), the only wait is „log file sync” on commit

● There are cases when session must wait because of too slow DBWR

– When there are too many dirty blocks and no free space in buffer cache

– Session wants to read another lock, and has to first place it in cache

– Session waits till DBWR will flush dirty blocks to make room for new blocks● When we can see inefficient DBWR ?

– When there is workload with huge amount of updates

– When writes goes to the same file/block ("free buffer wait" connected with "buffer busy wait")

● Imbalance in Read vs Write operations (write – through flash cache, accelerates read factor by up to 10 times!)


Smart Flash Cache


Database Machine – Software Architecture

• Diskmon – checks cell liveness, DBRM plans propagation

● Diskmon.bin – master process, dskm – slave process for each d instance

● Provides data fencing to protect data integrity


Exadata Horizontal Scalablility


Exadata Storage Expansion Rack

• Full Rack

● 18 Exadata Storage Servers

● 3 Infiniband Switches

• Half Rack



• Quarter Rack




Database Machine Benefits for Warehousing

• Parallel storage grid providing huge throughput

• Very efficient Infiniband storage network

• Most of DW like query processing moved to storage servers

• Exadata Hybrid Columnar Compression – less I/O

• In-memory parallel query


Exadata Hybrid Columnar Compression

• Saved Space (cost saving)

• Performance Improvement (reduced I/O operations)

• Average compression ration 10x – 15x, depending on compression level

● Query Low / Query High

● Archive Low / Archive High

• Best compression ratios for Direct Load Operations

• Decompression offloaded to storage servers

• DBMS_COMPRESSION – predicting compression ratio


Database Machine Benefits for OLTP

• Parallel storage grid providing large I/O throughput

• Low-latency Infiniband NetworkPING qrceladm01.oracle.com.pl (10.213.7.12) 56(84) bytes of data.64 bytes from qrceladm01.oracle.com.pl (10.213.7.12): icmp_seq=1 ttl=64 time=0.101 ms64 bytes from qrceladm01.oracle.com.pl (10.213.7.12): icmp_seq=2 ttl=64 time=0.122 ms64 bytes from qrceladm01.oracle.com.pl (10.213.7.12): icmp_seq=3 ttl=64 time=0.125 ms64 bytes from qrceladm01.oracle.com.pl (10.213.7.12): icmp_seq=4 ttl=64 time=0.131 ms64 bytes from qrceladm01.oracle.com.pl (10.213.7.12): icmp_seq=5 ttl=64 time=0.121 ms

• High-End database servers, with a lot of computing memory

• Exadata Smart Flash Cache


Database Machine Software Architecture For Different Database versions


Classic Database I/O vs Exadata SQL Processing Model


Infiniband Storage Network and Smart Scan Scale-Out

Each cell can deliver 1.8 GB/s14 cells have potential scanning power 25.3 GB/s

Select /* full(t1) */ count(*) from huge_4800gb where id < 10;

Without smart-scan- Extents are identified- Table is evenly distributed across all servers- All blocks of table huge_4800gb are send to Database- Each cell can scan with speed 0.357 GB/s- In full rack there are 14 cells- Infiniband connection to DB nodes is 40Gb/s / 14 what gives us 2.85Gb/s = 0.357GB/s- In this case each cell scan speed is 0.357Gb/s- With table size 4800 GB it gives ~16 minutes !

With smart-scan- Database (shadow process) asks to send only matching rows- Returning rows are only 2MB in size- In this case Interconnect speed is not an issue- Each cell can scan with a speed 1.8 G/s- Operation will last 3 minutes and 10 seconds !!!!


Exadata Smart Scan

• Predicate Filtering● Storage Server Software, applies 'where' predicates on

storage layer. ● Only filtered rows are returned to database nodes

• Column Projection● Only columns referenced in SQL query, and necessary

to fulfill join processing and sorting columns are returned

• Join Processing● Star joins are executed on storage layer


Exadata Smart Scan

• Scans on encrypted data

• Scans on compressed data

• Data Mining Scoring Model

• Subset of functions is offloaded (V$SQLFN_METADATA)

• Formatting datafile blocks

• RMAN datafile restoration ● One block granurality for incremental backups using

Block Change Tracking File


Exadata Smart Scan

• Virtual Columns computation

• Balancing CPU usage


Smart Scan Prerequisites• There must be full scan of object

● FTS – Full Table Scan

● FFIS – Fast Full Index Scan

● Materialized View Full Scan

• Scan must use Direct-Path Read mechanism

● Higher 'direct path read' Waits in 11g when Compared to 10g (Doc ID 793845.1)

● _serial_direct_read, _small_table_threshold - default 2% of buffer cache blocks

● This decision is based on the size of the table, buffer cache size, and various other statistics

● Any table having more than 5 times the number of blocks in "_small_table_threshold" will automatically use direct path reads for serial full table scans (FTS)

• Object must be stored on Exadata Storage


Exadata Storage Indexes

SQL> Select * from TABLE where B < 2


Multiplied Benefits


IORM – I/O Resource Manager



• I/O redo writes and controlfile write always prioritized before user writes

• Database writer writes are scheduled at the same priority level as user I/O

• Background queues correspond to low,medium,high priority



CellCLI> alter iormplan dbplan=(name=db1,level=1,allocation=70), - (name=db2,level=1,allocation=30), - (name=db3,level=2,allocation=50), - (name=db4,level=2,allocation=50), - (name=other,level=3,allocation=100), -catplan='';

DbName Level1 Level2 Level3

db1 70

db2 30

db3 50

db4 50

other 100


CellCLI> alter iormplan dbplan=(name=db1,share=7), - (name=db2,share=18), - (name=db3,share=15), - (name=db4,share=5), - (name=default,share=3), -Catplan='';

● Max Value for share=32● Max number of directives=1024

CellCLI> alter iormplan dbplan=(name=db1,level=1,allocation=70, role=primary,flashCache=on,flashLog=on), -

(name=db2,level=1,allocation=20, role=standby,flashCache=off,flashLog=off), - (name=db3,level=2,allocation=20, role=standby,flashCache=off,flashLog=off), - (name=db4,level=2,allocation=80, role=primary,flashCache=off,flashLog=on), - (name=other,level=3,allocation=100), -catplan='';



Interleaved Grid Disks and ASM Intelligent Data Placement

CellCLI> CREATE CELLDISK ALL HARDDISK;CellCLI> CREATE GRIDDISK ALL PREFIX=DGDATA, SIZE=500GCellCLI> CREATE GRIDDISK ALL PREFIX=DGRECO, SIZE=300G

CellCLI> CREATE CELLDISK ALL HARDDISK - INTERLEAVING='normal_redundancy';orCellCLI> CREATE CELLDISK ALL HARDDISK - INTERLEAVING='high_redundancy';CellCLI> CREATE GRIDDISK ALL PREFIX=DGDATA, SIZE=500GCellCLI> CREATE GRIDDISK ALL PREFIX=DGRECO, SIZE=300G


Interleaved Grid Disks and ASM Intelligent Data Placement

• Interleaved GridDisks works with ASM IDP (Intelligent Data Placement)

• Primary Extents are placed on Hot Portion of disk

• Mirror Extent are placed on Cold Portion of disk

• No special commands necessary


Exadata Specific System Statistics

• New method for monitoring IOFTRSPEED

• Captures accurate MBRC for Exadata

• Asses correctly for DB machine CPUSPEED

• Requires at least 11.2.0.2 BP18

SQL> exec dbms_stats.gather_system_stats('EXADATA')exec dbms_stats.gather_system_stats('EXADATA')


Network Resource Management

• Prorititize critical database network traffic

• Ensures fast response time for latency-critical operations (QoS)

• Gives priority to redo writes and RAC Cache Fusion messages


Automatic Hard Disk Scrubbing and Repair

• Corruption on seldom accessed data can be unnoticed

• Reads are made from primary extent copy, so secondary extents could be long time corrupted

• Proactive SCRUB I/O is issued when disk is idle

• Scrubbing periodically scans each disk every two weeks by default

● In write-back, if data is dirty in FC, it will be overwirtten when cached copy will be flushed to disk

● If database block is clean in flash, it is repaired from flash, if not it is repaired from mirror copy

CELLCLI> ALTER CELL hardDiskScrubInterval = daily | weekly | biweekly | none


Active-Active IB Connectivity

• Prior to Storage Sotfware 11.2.3.3.0 only active-passive IB bonding was used

• In >= 11.2.3.3.0 it is possible to configure Active-Active

• Do not configure Active-Active bonding on hardware before X4.

• Exadata X4 is equipped with IB cards that supports PCIe 3.0 , allowing the combined bandwith of both ports can be exploited


Faster Disk Rebalance Operation (appliance mode)

• New ASM attribute Exadata specific

• 'appliance.mode ' improves disk rebalance times

• 'compatible.asm' => 11.2.0.4

• 'cell.smart_scan_capable' = 'true'

• Improves disk rebalance times, by taking advantage of the specific configuration settings applied to ASM groups on Exadata


I/O Latency Capping (Read/Write)• Because of Internal maintenance operations, or

precursor to failure, disk and flash can exhibit occasionally unusual high latencies

• Storage Software detects such situations

• Read operation● ESS messages database that initiated the I/O● Database redirects read to mirror copy

• Write operation● Requires Flash in write-back mode● Redirects slow write operation to another flash device

on the same cell


Exadata X5-2 Extreme Flash Storage Server

Processors 16 Intel CPU Cores, 2 x Eight-Core Intel Xeon E5-2630 v3 2.4 GHz


Flash Drives 12.8 TB8 x 1.6 TB Sun Flash Accelerator F160 PCI

Infiband Network Dual-Port QDR 40 Gb/s HBA




Exadata X5-2 Elastic Configuration

• Start with Quarter Rack ● 2 database servers● 3 storage servers

• Add maximum of 22 servers ● Mixture of HC and HP (Extreme Flash) servers is

supported


Oracle Capacity-On-Demand for Database Servers

• Limits the number of active cores

• Available commencing with version 12.1.2.1.0

• Set during Initial Configuration with Deployment Assistant

• Number of active cores cannot be decreased

• Minimum number of active CPU cores

● 50% for X4

● 40% for X5

• Number of active processor cores must be equal among db nodes sockets

# resourceControl -capacity_on_demand display | disable | enable -cores numOfCores


Oracle Exadata Virtual Machines

• Available since 12.1.2.1.0

• Configured OEDA

• VM machines are considered as trusted partitions

• Multiple clusters on the same sets of physical servers

• All features of Exadata available

• Requires Oracle Linux


I/O Resource Management For Flash

• Flash Cache usage limits by database

• In multitenant configuration , PDBs limits can be set as percentage of the container database limit

• Based on shares

CellCLI> alter iormplan dbplan=(name=db1,share=7,flashCacheMin=8192), - (name=db2,share=18, flashCacheMin=1G,flashCacheLimit=10G), - (name=db3,share=15,limit=40), -catplan='';


Exadata Storage Server Performance Statistics in AWR reports

• Exadata specific configuration and performance statistics are collected in AWR reports

● Configuration

– Hardware model

– Software versions

– Storage Configuration● Health reports

– Offline disks

– Open alerts● Performance statistics

– OS

– Storage servers

– Smart scan

– Database statistics


Database Machine Automated Support Ecosystem

• Automatic Service Request● Available as a feature of Oracle hardware Warranty and

Oracle Premier Support for Systems● Automatically opens SRs with Oracle Suppport● Compromises hardware faults on

– CPUs, disk controllers,disks, flash cards,flash modules,Infiniband cards,memory modules,system boards,power supplies and funs


Database Machine Automated Support Ecosystem


Metalink Note 888828.1

• Exadata Database Machine and Exadata Storage Server Supported Versions

• Applies to

Oracle Exadata Storage Server SoftwareOracle Exadata HardwareOracle Database - Enterprise Edition - Version 11.2.0.4 to 12.1.0.2 [Release 11.2 to 12.1]Oracle Platinum Services - Version N/A to N/AEnterprise Manager for Oracle Database - Version 10.1.0.5 to 10.1.0.5 [Release 10.1]Oracle Solaris on x86-64 (64-bit)Linux x86-64Oracle Exadata Database Machine


In order of importance:

• D92887GC10 - Exadata Database Machine: 12c Administration Workshop Ed 1 NEW - 5 days

• D78850GC20 - Oracle Database 12c: Backup and Recovery Workshop Ed 2 - 5 days

• D81250GC10 - Oracle Database 12c: RAC Administration Ed 1.1 - 5 days

• D81242GC10 - Oracle Database 12c: ASM Administration - 2 days

• D79232GC10 - Oracle Database 12c: Data Guard Administration - 4 days

• D79236GC10 - Oracle Database 12c: Performance Management and Tuning - 5 days

• D79128GC10 - Oracle Database 12c: Managing Multitenant Architecture - 2 days

• D85572GC10 - Oracle Database 12c: Implement Partitioning Ed 1 - 2 days

• D70064GC20 - Oracle Database 11g: Administer a Data Warehouse - 4 days

• D55650GC10 - Oracle Database 11g: Managing Oracle on Linux for DBAs - 2 days

•

Follow-up Courses

what is and what is not exadata engineered...

Documents