safe harbor statement - amazon s3days... · actually almost boring. it just worked - just turn it...
TRANSCRIPT
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Oracle Confidential – Internal 2
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Database12cEffectively using the Oracle Database In-Memory Option
Sean StaceyDirector PTS, Server TechnologiesDatabase Product Management
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Accelerate Mixed
Workload OLTP
Real Time Analytics No Changes to Applications
Trivial to Implement
Oracle Database In-Memory Goals
100x
2x
4
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
Oracle 11.1 (GA: Aug 2007)
Oracle 11.2 (GA: Sep 2009)
Oracle 12.1 (GA: Jun 2013)
Release Roadmap
Sustaining Support
AUG 2015 AUG 2012
JAN 2018 JAN 2015
JUN 2021 JUN 2018
Waived Extended Extended Support Premier Support
1
Today
Oracle Database 12c Release 2 Targeted for Release 1HCY2016
(My Oracle Support Note 742060.1 )
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Accelerate Mixed Workload OLTP
Real Time Analytics No Changes to
Applications
Trivial to Implement
Oracle Database In-Memory Goals
100x 2x
2
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Until Now Must Choose One Format and Suffer Tradeoffs
Row Format Databases vs. Column Format Databases
Row
Transactions run faster on row format
– Example: Insert or query a sales order – Fast processing few rows, many columns
Column
Analytics run faster on column format
– Example : Report on sales totals by region – Fast accessing few columns, many rows
SALES
SALES
3
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Breakthrough: Dual Format Database
• BOTH row and column formats for same table
• Simultaneously active and transactionally consistent
• Analytics & reporting use new in-memory Column format
• OLTP uses proven row format
4
Memory Memory
SALES SALES
Row Format
Column Format
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Real-time analytics will create new use cases
• Analytics & reporting against new and external data types
5
New In-Memory Format
Column Format
“Now we can run time-sensitive analytical queries directly
against our OLTP database. This is something we wouldn’t
have dreamt of earlier.”
Arup Nanda Enterprise Architect
Starwood Hotels and Resorts
“Oracle Database In-Memory is a game changer for OLTP, DW,
and mixed workloads. It dramatically improves the
performance of all types of analytical queries.”
“Oracle Database In-Memory is a game changer for OLTP, DW,
and mixed workloads. It dramatically improves the
performance of all types of analytical queries.”
Liviu Horn AVP Database Management McKesson Health Solutions
Liviu Horn AVP Database Management McKesson Health Solutions
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle In-Memory Applications Examples
CHANGE
Oracle Application Module Improvement Elapsed Time
Demantra - Promotion Planning 102x Faster 18.3 min to 11 second
Oracle In-Memory Transportation Mgmt 1031x Faster 17 min to <1 seconds
Oracle E-business Suite - In-Memory Cost Management
628x Faster
58 hours to 6 minutes
Demantra - Consumption Driven Planning 13.5x Faster 12.7 hours to 56 min
JD Edwards - Supply Chain Planning and Analytics
76x Faster 28 minutes to 3 min
Siebel Marketing - List Import 141x Faster 115 mins to 49 seconds
8
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Getting to Real-Time
9
Source: Richard Hackethorn’s Component’s of Action Time
Data
base S
peed
Best Practices Engineered Systems Database In-Memory
Database Technology
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
AGENDA
10
High Level Overview Introduction to In-Memory Configuring, Enabling and Populating In-Memory The Optimizer and In-Memory Using In-Memory
1
4
5
3
2 2
3
4
5
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
AGENDA
11
High Level Overview Introduction to In-Memory Configuring, Enabling and Populating In-Memory The Optimizer and In-Memory Using In-Memory
2
4
5
3
1
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle In-Memory Columnar Technology
• Pure in-memory column format
• Not persistent, and no logging
• Quick to change data: fast OLTP
• 2x to 20x compression
• Enabled at table or partition level
• Available on all hardware platforms
12
SALES
Pure In-Memory Columnar
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Orders of Magnitude Faster Analytic Data Scans
• Each CPU core scans local in-memory columns
• Scans use super fast SIMD vector instructions
• Originally designed for graphics & science
• Billions of rows/sec scan rate per CPU core
• Row format is millions/sec
13
Vec
tor
Reg
iste
r Load multiple region values
Vector Compare all values an 1 cycle
CPU
Memory
REG
ION
CA
CA
CA
CA
Example: Find all sales in region of CA
> 100x Faster
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Joining and Combining Data Also Dramatically Faster
• Converts joins of data in multiple tables into fast column scans
• Joins tables 10x faster
14
Example: Find total sales in outlet stores
Sales Stores
Sto
re ID
Am
ou
nt
Type=Outlet
StoreID in 15, 38, 64
Sto
re ID
Typ
e
Sum
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Generates Reports Instantly
• Dynamically creates in-memory report outline
• Then report outline filled-in during fast fact scan
• Reports run much faster without predefined cubes
15
Example: Report sales of footwear in outlet stores
Sales
Stores
Products
In-Memory Report Outline
Footwear
Ou
tlet
s $ $$
$ $$$
Footwear
Sales Outlets
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Complex OLTP is Slowed by Analytic Indexes
• Most Indexes in complex OLTP (e.g. ERP) databases are only used for analytic queries
• Inserting one row into a table requires updating 10-20 analytic indexes: Slow!
• Indexes only speed up predictable queries & reports
Table
1 – 3 OLTP
Indexes
10 – 20 Analytic Indexes
16
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Complex OLTP is Slowed by Analytic Indexes
17
Insert rate decreases as number of indexes
increases
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Column Store Replaces Analytic Indexes
• Fast analytics on any columns
• Better for unpredictable analytics
• Less tuning & administration
• Column Store not persistent so update cost is much lower
• OLTP & batch run faster
Table
1 – 3 OLTP
Indexes In-Memory
Column Store
18
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
AGENDA
19
High Level Overview Introduction to In-Memory Configuring, Enabling and Populating In-Memory The Optimizer and In-Memory Using In-Memory
3
4
5
1
2
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
System Global Area SGA
Buffer Cache Shared Pool Redo Buffer
Large Pool Other shared Memory Components
In-Memory Area
Configuring : In-Memory Area
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Configuring : In-Memory Column Store
• Column formatted data is stored in In-Memory Column Store
• Is a new component of SGA, called In-Memory Area
• Does not replace the buffer cache
• Controlled by initialization parameter, INMEMORY_SIZE
– Default 0 (Disabled)
– Minimum 100 MB
• In-Memory Area is a Static Pool – Is not controlled or impacted by Automatic Memory Management (AMM)
21
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Configuring : In-Memory Column Store
• SGA_TARGET must be large enough to accommodate the IM Column Store • Or MEMORY_TARGET for AMM
• Check if enabled using V$SGA or V$PARAMETER
SELECT * FROM V$SGA;
NAME VALUE
------------------ -----------
Fixed Size 2927176
Variable Size 570426808
Database Buffers 4634022912
Redo Buffers 13848576
In-Memory Area 10244836480
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Configuring : In-Memory Area
• Subdivided into 2 pools – IMCU Pool (aka 1 MB) and SMU Pool (aka 64 KB)
– IMCU pool (In-Memory Compression Units): Sized in 1 MB contiguous chunks, stores actual column formatted data
– SMU pool (Snapshot Metadata Units): Sized in 64 KB chunks, stores metadata and transactional information: Transaction Journal
– Contiguous chunks are called IM Extents
– Majority of the space is consumed by IMCU Pool
23
SMU Pool
23
SMU
SMU
SMU
SMU
SMU
SMU
SMU SMU
In Memory Area
IMCU Pool
SMU Pool
IMCU IMCU
IMCU IMCU
IMCU IMCU
IMCU IMCU
IMCU IMCU
IMCU IMCU
IMCU IMCU
IMCU IMCU
IMCU
IMCU
IMCU
IMCU
…
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Configuring : In-Memory Column Store
• V$INMEMORY_AREA: Displays current pool sizes their status
• Useful to detect if IM Column Store is running out of capacity
SQL> SELECT * from V$INMEMORY_AREA;
POOL ALLOC_BYTES USED_BYTES POPULATE_STATUS
---------- ----------- ---------- ---------------
1MB POOL 1307574272 135266304 DONE
64KB POOL 318767104 524288 DONE
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Configuring : In-Memory Initialization Parameters
25
INMEMORY_SIZE – Enables and sets the size of the In-Memory column store static pool. The minimum setting is 100M. INMEMORY_CLAUSE_DEFAULT – Use this parameter to set default IM clauses for new tables and materialized views. The default is an empty string.
OPTIMIZER_INMEMORY_AWARE – Set this parameter to make the optimizer cost model enhancements for IM to be enabled or disabled. INMEMORY_QUERY – Set this parameter to enable or disable access to IM. i.e: You can disable access even if the IM column store is populated. The default is ENABLE.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Configuring : In-Memory Initialization Parameters
26
INMEMORY_MAX_POPULATE_SERVERS – Set this parameter for the Maximum number of background processes to populate the IM column store. Set this based on CPU cores. INMEMORY_TRICKLE_REPOPULATE_SERVERS_PERCENT – Set this parameter to limit the background processes for trickle repopulation. It is a percentage of INMEMORY_MAX_POPULATE_SERVERS.
INMEMORY_FORCE – Allows you to specify whether objects specified as INMEMORY are populated into the In-Memory Column Store or not.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Enabling : IM Column Store – INMEMORY Clause
• New INMEMORY clause
• Following segment types are eligible
• Tables
• Partitions
• Subpartition
• Materialized views
• Following segment types not eligible
• IOTs
• Hash clusters
• Out of line LOBs, LONGs
ALTER TABLE sales INMEMORY;
ALTER TABLE sales NO INMEMORY;
CREATE TABLE customers ……
PARTITION BY LIST
(PARTITION p1 …… NO INMEMORY,
(PARTITION p2 …… INMEMORY);
Pure OLTP Features
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
About the INMEMORY Clause
In-Memory Sub-Clauses
• The in-memory clause has multiple sub-clauses
– PRIORITY: Controls the population priority of the object
– MEMCOMPRESS: Specifies Compression levels
– DUPLICATE: For RAC, to duplicate the IM Store content on other nodes
– DISTRIBUTE: For RAC, to distribute IM Store content, in chunks, to other nodes
– Example:
28
ALTER TABLE sales INMEMORY PRIORITY HIGH;
ALTER TABLE sales INMEMORY MEMCOMPRESS FOR QUERY HIGH PRIORITY MEDIUM;
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
In-Memory Enablement Object Metadata
• Enablement status can be checked in *_TABLES,
*_SEGMENTS,
*_TAB_PARTITIONS,
*_TAB_SUBPARTITIONS and
DBA_TABLESPACES views
• Tracked in 5 new columns (attributes): • INMEMORY: Object level enablement - ENABLED or DISABLED
• INMEMORY_PRIORITY: Stores the PRIORITY
• INMEMORY_COMPRESSSION: Stores the Compression attributes as defined
• INMEMORY_DUPLICATE: Stores the DUPLICATE clause specification
• INMEMORY_DISTRIBUTE: Stores the DISTRIBUTE clause specification
SELECT TABLE_NAME,
INMEMORY,
INMEMORY_PRIORITY,
INMEMORY_COMPRESSION,
INMEMORY_DISTRIBUTE,
INMEMORY_DUPLICATE
FROM USER_TABLES
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Enabling : IM Column Store – Selective Columns
• Possible to enable only certain columns from a table or partition
ALTER TABLE sales INMEMORY
NO INMEMORY (PROD_ID);
ALTER TABLE sales INMEMORY
INMEMORY (PROD_ID, PROD_NAME)
NO INMEMORY (PROD_CATEGORY)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Enabling : Columns With the INMEMORY Attribute
• V$IM_COLUMN_LEVEL captures column-level enablement status
– Also their compression attributes
• The view contains rows only when the in-memory column clause is specified
• Valid Values:
– NO INMEMORY: Column is not part of IMCU
– DEFAULT (Default compression i.e. Query LOW)
– Valid Compression Types: NONE, DML, QUERY [LOW|HIGH], CAPACITY [LOW|HIGH]
SELECT table_name, column_name, inmemory_compression from v$im_column_level;
TABLE_NAME COLUMN_NAME INMEMORY_COMPRESSION
------------------- ----------------------- ----------------------
SALES PROD_ID NO INMEMORY
SALES CUST_ID DEFAULT
SALES TIME_ID DEFAULT
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Enabling : Tablespaces for the IM Column Store
• The in-memory clause can be specified as a DEFAULT for the Tablespace
– Using CREATE or the ALTER TABLESPACE statements
• If a tablespace is enabled:
– All tables and materialized views created in the tablespace are enabled for the IM column store by default
– Objects inherit the tablespace in-memory clause (at creation time)
• Object level In-Memory clause overrides the tablespace in-memory clause
32
• ALTER TABLESPACE … DEFAULT INMEMORY
• ALTER TABLESPACE … DEFAULT INMEMORY MEMCOMPRESS QUERY HIGH
• ALTER TABLESPACE … DEFAULT INMEMORY PRIORITY LOW
Example DDL
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Populating : In-Memory Column Store
• Populate is the term used to bring data into the In-Memory column store
• Populate is used instead of load because load is commonly used to mean inserting new data into the database
• Populate brings existing row store data into the IM Column Store
– Reformats it in the columnar format
33
• Columns with unsupported data types (LONGs, Out-of-line LOBS) are skipped in population – Rest of the columns are still populated
• Small segments (less than 64k) are skipped
• Segments owned by SYS, SYSTEM are skipped
• Two basic kinds of populate • Initial Populate: Initial creation of IMCU • Repopulate: Recreate an existing IMCU after it has been modified
Note: An object needs to be enabled for INMEMORY prior to its population
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Initial Population : In-Memory Column Store
• Initial Population (aka Pre-Population) of an object happens as follows: – Manually:
• Using DBMS_INMEMORY.POPULATE (or REPOPULATE with FORCE)
• When querying the object (via. table scan)
– Automatically: • Using the PRIORITY clause on the object
– At CREATE or ALTER time
• Using INMEMORY_FORCE parameter – Force populates all eligible objects
• Initial Population is also needed when an object acquires new Extents not yet mapped to IMCUs – Via Bulk loads or DML that causes the segment to grow i.e. add new Extents
34
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Initial Population : In-Memory Column Store
In Memory Area
35
SALES Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
….
….
IMCU Pool SMU Pool
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
Extent
….
….
SMU
SMU SMU
SMU SMU
SMU
SALE
S
IMCU IMCU
IMCU IMCU
IMCU
IMCU
SALE
S
Population
SMU SMU
SMU
IMCU IMCU
IMCU IMCU
Population
SMU
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Initial Population : In-Memory Column Store
Manual Population
• Querying the Object – SELECT /*+ FULL (s) */ COUNT(*) FROM SALES s;
• Using DBMS_INMEMORY.POPULATE – EXEC DBMS_INMEMORY.POPULATE('LABUSER','SALES');
• Using DBMS_INMEMORY.REPOPULATE with FORCE=>TRUE – EXEC DBMS_INMEMORY.REPOPULATE('LABUSER','SALES', FORCE=>TRUE);
36
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Initial Population : In-Memory Column Store Automatic Population – PRIORITY Clause
• Objects can be set to automatically populate after the database is opened, or after the object is created, or altered, or loaded with data – This is possible via the PRIORITY clause
• Objects get populated in a prioritized list, the order of which is controlled by the priority level
• All objects at a given priority level must be fully populated before a lower priority level can commence – CRITICAL > HIGH > MEDIUM > LOW
• The population order can be superseded if an object without a PRIORITY is scanned, thereby triggering its population into IM column store – Also by executing DBMS_INMEMORY.POPULATE on demand
37
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Initial Population : IM Column Store Automatic Population
• PRIORITY subclause controls the order of population
• Not the speed
• CRITICAL > HIGH > MEDIUM > LOW
• Default – NONE (populate on first access)
• Priority levels only work on tables and partitions – NOT columns
CREATE TABLE orders
(c1 number,
c2 varchar(20),
c3 number)
INMEMORY PRIORITY CRITICAL
NO INMEMORY (c1);
ALTER TABLESPACE TS_DATA DEFAULT INMEMORY
PRIORTY LOW
select segment_name, populate_status, inmemory_priority,
inmemory_size, bytes_not_populated from v$im_segments;
SEGMENT_NAME POPULATE_STATUS INMEMORY_PRIORITY INMEMORY_SIZE BYTES_NOT_POPULATED
------------ --------------- ----------------- ------------- -------------------
ACCOUNTS STARTED HIGH 196606 2434886912
SALES COMPLETED CRITICAL 135790592 0
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Populating : Background Processes
• New IMCU Population is triggered by IMCO or Table Access
– When IMCO wakes up, it checks the PRIORITY Queues
• For newly created INMEMORY objects
• For existing INMEMORY objects having new Extents not yet mapped to IMCUs
• PRIORITY clause is required
• Worker process populates IMCUs
– A set of background processes referred to as worker processes (ora_w001_orcl)
– Each worker process allocates its own IMCU and maps a set of extents to the IMCU
– Number of worker processes is controlled by INMEMORY_MAX_POPULATE_SERVERS
– The database is fully active / accessible while population occurs
– Population is a streaming mechanism, simultaneously columnizing and compressing the data
– Data is not sorted or ordered during population - Read in the same order as it resides in disk blocks
39
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
In-Memory Column Store - Six Compression Levels
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Compressing : In-Memory Column Store
• Objects compressed during population
• New compression techniques
• Focused on scan performance
• Controlled by MEMCOMPRESS subclause
• Multiple levels of compression
• Possible to use a different level for different partitions in a table and different columns in a table
• Easy to switch levels for ILM
ALTER MATERIALIZED VIEW mv1 INMEMORY
MEMCOMPRESS FOR QUERY LOW;
CREATE TABLE trades
(Name varchar(20),
Desc varchar(200))
INMEMORY
MEMCOMPRESS FOR CAPACITY HIGH(desc);
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Column Level Compression
• By default, columns inherit the table compression clause
• Compression techniques can be overridden at the column level
– Using the selective column level compression clause
• Overridden column level attributes are stored in V$IM_COLUMN_LEVEL
– Along with their participation status
– This view returns no rows when ALL columns default to the table compression level
42
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Compression Advisor And In-Memory
• Enhanced to include In-Memory compression
• Easy way to determine memory requirements
• Use DBMS_COMPRESSION
• Applies MEMCOMPRESS to sample set of data from a table
• Returns estimated compression ratio
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Compression Advisor and In-Memory
• DBMS_COMPRESSION.GET_COMPRESSION_RATIO
– Comptype: valid values: • COMP_INMEMORY_NOCOMPRESS, COMP_INMEMORY_DML,
• COMP_INMEMORY_CAPACITY_HIGH, COMP_INMEMORY_CAPACITY_LOW
• COMP_INMEMORY_QUERY_HIGH, COMP_INMEMORY_QUERY_LOW
– Subset_numrows: • COMP_RATIO_MINROWS, COMP_RATIO_ALLROWS
44
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
AGENDA
45
High Level Overview Introduction to In-Memory Configuring, Enabling and Populating In-Memory The Optimizer and In-Memory Using In-Memory
5
3
2 2
3 3
2
1
4
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Query Processing – Multiple Optimizations gets applied
• Column Projections
• Vector Processing
• Bloom Filter Joins
• Predicate Push Down
• Predicate Elimination
• Dictionary Pruning
• In-Memory Aggregation
46
How would you measure the benefits?
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
In-Memory & The Optimizer
• Decision to use the In-Memory Column Store is made by the Optimizer – Optimizer is fully aware of the IM Column Store
• Decision based on traditional table statistics & new In-Memory statistics
• In-Memory statistics automatically supplied to the Optimizer at parse
• In-Memory statistics include – What columns from the table are populated
– How much of the table has been populated
– What data pruning can be achieved based on where clause predicates
How can you tell what the optimizer decided?
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
In-Memory and The Optimizer Plan Display
• New access method TABLE ACCESS IN MEMORY FULL
– Cost model adapted to consider In-Memory scan
• TABLE ACCESS INMEMORY FULL indicates a POSSIBLE use of column store
– May still not use the store in some cases
– For example, at scan time, if the IM Statistics indicate that INMEMORY access will be costly
48
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
In-Memory Column Store Statistics
• Database Statistics are a better measure of In-Memory Column Store benefits
• New Session and System level statistics
• Best way to determine if the IM Column Store was used
• Best way to measure the benefits of In-Memory scan
• Display all stats using V$STATNAME
– IM column store related statistics begin with “IM”
– There are currently over 200 new statistics
49
SELECT DISPLAY_NAME
FROM V$STATNAME
WHERE DISPLAY_NAME LIKE ‘IM %’;
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
In-Memory Scans: Under the Hood
• Wealth of information in AWR reports
• V$SYSSTAT monitors all aspects of in-memory scans
How many rows were scanned
How many rows were avoided by dictionary pruning
How many predicates were optimized
How many IMCUs were min-max pruned
Etc ..
IM scan bytes in-memory IM scan bytes uncompressedIM scan CUs columns accessedIM scan CUs columns decompressedIM scan CUs columns theoretical maxIM scan rowsIM scan rows range excludedIM scan rows excludedIM scan rows optimizedIM scan rows projectedIM scan CUs predicates receivedIM scan CUs predicates appliedIM scan CUs predicates optimizedIM scan CUs prunedIM scan segments ….
IM scan bytes in-memory IM scan bytes uncompressed IM scan CUs columns accessed IM scan CUs columns decompressed IM scan CUs columns theoretical max IM scan rows IM scan rows range excluded IM scan rows excluded IM scan rows optimized IM scan rows projected IM scan CUs predicates received IM scan CUs predicates applied IM scan CUs predicates optimized IM scan CUs pruned IM scan segments minmax eligible ….
50
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
IM Column Store Statistics
• IM scan bytes uncompressed: Uncompressed size in bytes of all CU involved a scan before optimizations
• IM scan bytes in-memory: Compressed size in bytes of all CUs in scan before optimizations
• IM scan CUs columns accessed: Count of column units accessed by the scan
• IM scan CUs columns theoretical max: Count of columns that would be accessed if each scan looked at all columns in the IMCU
51
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Identifying : INMEMORY Joins • Bloom filters enable joins to be
converted into fast column scans
• Tried and true technology originally released in 10g
• Same technique used to offload joins on Exadata
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Multiple bloom filter
• Multi-table joins are common in analytic queries
• Possible to apply multiple bloom filters to the fact table due to sophisticated Optimizer
• Simultaneously joining to all of the dimension tables while scanning the fact table
Select d.d_year, s.s_nation,
sum(lo_revenue - lo_supplycost)
From LINEORDER l, DATE_DIM d,
PART p, SUPPLIER s
Where l.lo_orderdate = d.d_datekey
And l.lo_partkey = p.p_partkey
And l.lo_suppkey = s.s_suppkey
And p.p_mfgr= 'MFGR#12’
And s.s_region = 'AMERICA’
Group by d.d_year, s.s_nation
Order by d.d_year, s.s_nation;
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Swap Join Input Optimization
Left deep tree
PART
HASH JOIN
LINEORDER
HASH JOIN
SUPPLIER
HASH JOIN
DATEDIM
1 2
3
4
1
2
3
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Swap Join Input Optimization
Becomes a right deep tree
PART
HASH JOIN
LINEORDER
HASH JOIN
SUPPLIER
HASH JOIN
DATEDIM
2
3
1
2
3
1
4
Why is this significant? Applying multiple filter on initial scan of LINEORDER reduces the number of rows sent upstream in the plan
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
In-Memory Aggregation
• A feature of Oracle Database In-Memory
• New Vector Group-By enables extremely efficient in-memory array based aggregation
• Simultaneously accumulates aggregate values into in-memory arrays during fact table scan
• Optimizes star, snowflake and similar queries
• Introduces new internal operators
• Transparent to applications
Oracle Confidential – Internal/Restricted/Highly Restricted 58
Average Elapsed Time, Seconds per Query
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
In-Memory Aggregation Processing
High-level Processing steps
– Transform joins into scans on the fact table
• Fast in-memory scans and array lookups
– Optimize aggregation using in-memory arrays
• Data structures optimized for in-memory
– Late joins to dimension attribute data
• Minimizes data movement in the execution plan
Oracle Confidential – Internal/Restricted/Highly Restricted 59
Example: Report sales by Quarter and Region
Sales
Customers
Date In-Memory
Accumulator (PGA)
Quarters
Reg
ion
s $ $$
$ $$$
Quarters
Sales Regions
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Sales
1 2 3 4
1 2 3 4 5 6
1
2
3
4
Aggregation : In-Memory Aggregation Details
Quarters
Regions
1 CUST Date_Dim
Key vectors created on each join column
Customers Key Vector
Date_Dim Key Vector
2 Sales
4 In-Memory Accumulator
3 5 2 10 4 6 20 1 7 08
0 1 3 2 1 0 0 0 1 3 …
0 3 1 5 0 4 3 …
10
20
2 7 05
Customers
DATE
15
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
When to Expect a Vector Transformation Plan
Query Characteristics
• Join columns between tables “mostly” unique or numeric keys
• Tables populated into the IM column store
• One or more small tables joined to a large table
– Small tables usually 1/10th the size of the large table (or smaller)
– Equijoins
• One or more measures aggregated from the large table – SUM, MIN, MAX, AVERAGE, MEDIAN, etc.
• Statement contains a GROUP BY clause
61
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
When to Expect a Vector Transformation Plan
Optimizer Costing
• KEY VECTOR CREATE operator adds to ‘up-front’ cost of vector transformation plan
– Cost proportional to the size of dimension tables
– Biased against vector transformation for low cost queries
• KEY VECTOR USE lower cost than HASH JOIN – Biased towards vector transformation for join operations
– More joins, more benefit
– More the number of aggregated rows, more the benefit
62
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Vector Transformation Plan Unlikely if…
• Joins are performed between two or more very large tables
• Dimension tables contain more than 2 billion rows
• The system does not have sufficient memory resources
63
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Vector Transformation Plan Hints
• VECTOR_TRANSFORM
– Forces a vector transformation on the query block if a transformation is possible
• NO_VECTOR_TRANSFORM
– Prevents a vector transformation plan
SELECT /*+ vector_transform */
d1.year_name,
d1. quarter_name,
d3.region_name,
SUM(f.sales)
FROM time_dim d1,
customer_dim d3,
sales_fact f
WHERE d1.day_id = f.day_id
AND d3.customer_id = f.customer_id
AND d1. year_name ='CY2012'
GROUP BY d1. year_name,
d1.quarter_name,
d3.region_name;
Oracle Confidential – Internal/Restricted/Highly Restricted 64
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
AGENDA
65
High Level Overview Introduction to In-Memory Configuring, Enabling and Populating In-Memory The Optimizer and In-Memory Using In-Memory
4
5
3
2 2
3
5
3
2
1
“In terms of how easy the in-memory option was to use, it was
actually almost boring. It just worked - just turn it on, select
the tables, nothing else to do.”
“In terms of how easy the in-memory option was to use, it was
actually almost boring. It just worked - just turn it on, select
the tables, nothing else to do.”
Mark Rittman Chief Technical Officer Rittman Mead
Mark Rittman Chief Technical Officer Rittman Mead
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Keys to Real-Time Data Processing
• Process data in database not application
• Process sets of rows using SQL
• Row by row processing is slow and cannot be parallelized
• Enable in-memory column format
• Enable parallel SQL
• Memory removes storage bottlenecks enabling highly parallel SQL
67
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Using In-Memory - Lessons Learned
- parallel query influence - parallel_degree_policy = AUTO (not ADAPTIVE) - Should use AutoDOP when running in RAC - https://blogs.oracle.com/In-Memory/entry/oracle_database_in_memory_on
- patch installation?
- not nulls
- how to determine if it is being used
- Remove HINTS and NON-DEFAULT PARAMETERS
- In Memory aggregation
- indexes - INVISIBLE and UNUSEABLE
- segments < 64K do not get loaded
- turn off adaptive optimizer : optimizer_adaptive_features=off
- Parallel_threads_cpu
- “robbing from peter to pay paul”- don't reduce the size of the buffer_cache to accommodate the IMCS
68
By now we should have discussed all of these lessons learned along the way, if not, this slide is a placeholder or pensieve…
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle In-Memory Requires Zero Application Changes
Full Functionality - No restrictions on SQL
Easy to Implement - No migration of data
Fully Compatible - All existing applications run unchanged
Fully Multitenant - Oracle In-Memory is Cloud Ready
Uniquely Achieves All In-Memory Benefits With No Application Changes
69
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle In-Memory Requires Zero Application Changes
Full Functionality - No restrictions on SQL
Easy to Implement - No migration of data
Fully Compatible - All existing applications run unchanged
Fully Multitenant - Oracle In-Memory is Cloud Ready
Uniquely Achieves All In-Memory Benefits With No Application Changes
1
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Database In-Memory Differentiators
Orders of magnitude faster performance for analytics
OLTP becomes faster
Complete application transparency
Leverage complete Oracle Security and HA/DR technology
Scale up or scale out
Complements Oracle Exadata
Orders of magnitude faster performance for analytics
OLTP becomes faster
Complete application transparency
Leverage complete Oracle Security and HA/DR technology
Scale up or scale out
Complements Oracle Exadata