oracle optimizer: 12c new capabilities
TRANSCRIPT
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1
Oracle safe harbor statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.2
3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Optimizer: 12c New CapabilitiesThomas Kytehttp://asktom.oracle.com/
ORACLEPRODUCT
LOGO
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.4
Optimizer evolution
In the beginning there were rules
RULES
Optimizer evolved to be cost based
CBO
CBO
Optimizer proactively adapts to become self-learning
Rule are not enough
Databases became more feature rich
Reactive tuning with the use of advisors and auto jobs
As environment changes Potential for plan changes
Databases become more real-time, ad-hoc environments
Reactive tuning not enough
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.5
Agenda Adaptive Query Optimization Optimizer Statistics enhancements SQL Plan Management enhancements
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.6
Adaptive PlansJoin methodsParallel distribution
methods
Adaptive Statistics Discovered at compile
timeDiscovered at run time
Adaptive Query OptimizationOverview
Adaptive Query Optimization
Adaptive Plans Adaptive Statistics
Join Methods
Parallel distribution
Methods
At compile time
At run time
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.7
Query: Find all of the products with a unit price of 15 that we have sold more that 1 of
Two possible join methods for this query
Nested Loops
Hash Join
Adaptive Execution PlansAdapt join methods
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.8
Alternative sub-plans are pre-computed
Sub-plans stored in the cursor
Stats collect inserted before join
Rows buffered until final decision is made
Adaptive Execution PlansAdapt join methods
Table scanOrder _items
NESTED LOOPS
Index Scan Prod_info_ind
Statistics Collector
Default Plan is a nested loops join
HASH JOIN
Table scanProd_info
Rows coming out of order_items table are buffered up to a point If row count is less than the threshold use nested Loops otherwise switch to hash join
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.9
Number of rows seen in statistics collector exceeds threshold
Plan switches to hash join
Statistics collect disabled
Plan resolved on first execution & remains the same for subsequent executions
Adaptive Execution PlansAdapt join methods
Final Plan is a hash join
Statistics collector disabled after decision is made and becomes a pass through operation
Table scanOrder _items
NESTED LOOPS
Index Scan Prod_info_ind
HASH JOIN
Table scanProd_info
Statistics Collector
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.10
Explain plan command always shows default plan
Example shows a nested loops join as default plan
No statistics collector shown in plan
Adaptive Execution PlansDisplaying the default plan
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.11
After the statement has completed use DBMS_XPLAN.DISPLAY_CURSOR to see the final plan selected
Example shows that hash join picked at execution time
Again the statistics collector is not visible in the plan
Adaptive Execution PlansDisplaying the final plan
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.12
Full adaptive plan displayed when format parameter ‘+adaptive’ is set
Example shows both the nested loops and hash join in the plan
Adaptive Execution PlansDisplaying the full adaptive plan
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.13
Additional information displayed on why operations are inactive can be seen with format parameter ‘+report’
Adaptive Execution PlansDisplaying plan with +adaptive & +report formats
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.14
New column in V$SQL IS_RESOLVED_ADAPTIVE_PLAN
Indicates statement had an adaptive plan which was resolved on first execution
Resolved plan is used for subsequent executions
Statistics collectors and buffering is disabled
Adaptive Execution PlansIndicator in V$SQL
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.15
Adaptive plans are enabled by default
Can be put in reporting mode OPTIMIZER_ADAPTIVE_REPORTING_ONLY
Reporting mode shows what would have happened during execution in the plan
Adaptive Execution PlansReporting mode
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.16
Adaptive PlansJoin methodsParallel distribution
methods
Adaptive Statistics Discovered at compile timeDiscovered at run time
Adaptive Query OptimizationOverview
Adaptive Query Optimization
Adaptive Plans Adaptive Statistics
Join Methods
Parallel distribution
Methods
At compile
time
At run time
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.17
Dynamic Statistics
During compilation optimizer decides if statistics are sufficient to generate a good plan or not
Dynamic statistics are used to compensate for missing, stale, or incomplete statistics
They can be used for table scans, index access, joins and group bys One type of dynamic statistics is dynamic sampling
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.18
Dynamic sampling has a new level 11(AUTO)
Decision to use dynamic sampling depends on the complexity of predicate, existing statistics and total execution time
Dynamic statistics shared among queries
Dynamic Statistics Dynamic Sampling
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.19
Dynamic Statistics SGA
SHARED_POOL
Select * From product_informationWhere list_price-min_price=29And category_id not in (11,22)And prod_name like ‘Smart%’;
Optimizer checksfor existing statistics in data dictionary
2
SHARED DYNAMIC STATISTICS
DATA DICTIONARY
SQL statement is submitted
1
Optimizer determines plan using dynamic statistics
5
Cursor 0: Select * from product_information …
Dynamic sampling occurs on small number of blocks from table Resulting dynamic statistics are stored in cache
4
Statistics found but need to be augmented due to complex predicates
3
Table cardinality estimate for prod_info
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.20
Dynamic Statistics SGA
SHARED_POOL
Select supplier_id, prod_nameFrom product_informationWhere list_price-min_price=29And category_id not in (11,22)And prod_name like ‘Smart%’;
Optimizer checksfor existing statistics in data dictionary
7
SHARED DYNAMIC STATISTICS
DATA DICTIONARY
Different SQL statement is submitted with same predicates
6
Optimizer determines plan using dynamic statistics
10
Cursor 0: Select supplier_id, prod_name …
Necessary Dynamic statistics found in shared cache
9
Statistics found but need to be augmented due to complex predicates
8
Table cardinality estimate for prod_info
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.21
Adaptive Statistics
During execution optimizer estimates are compared to execution statistics
If statistics vary significantly then a new plan will be chosen for subsequent executions based on execution statistics
Re-optimization uses statistics gathered from previous executions First introduced as Cardinality Feedback in 11.2
Re-optimization
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.22
Adaptive Statistics
Statistics gathered about data volume and data type seen during execution If execution statistics vary significantly statement will be hard parsed on
the next execution using the execution statistics instead Statements are only monitored once if they don’t show significant
differences initially they won’t change in the future Only individual table cardinalities and group by estimates examined Information is stored in the cursor only and is lost if cursor ages out
Cardinality Feedback pre 12c
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.23
Initial execution of a query is monitored
Example shows initial plan is hash join between sales and customers
Cardinality estimates 8X off
Adaptive Statistics Cardinality Feedback pre 12c
Initial Cardinality estimates are more than 8X off
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.24
Execution statistics used to reparse the statement on the second execution
New plan shows correct cardinality estimates and a new join order
Information learnt is stored in the cursor only and is lost if cursor ages out
Adaptive StatisticsCardinality Feedback pre 12c
Execution Plan after the second execution
Estimates are now correct
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.25
Adaptive Statistics
Join statistics are also monitored Works with adaptive cursor sharing for statement with binds New Column in V$SQL IS_REOPTIMIZABLE Information found at execution time is persisted as SQL Plan Directives
Re-optimization in 12c
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.26
New column in V$SQL IS_REOPTIMIZABLE
Indicates that the statement will be re-parsed on the next execution
Adaptive StatisticsRe-optimization – indicator in V$SQL
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.27
Adapt join method example showed the join change from nested loops to hash join
But cursor is also marked IS_REOPTIMIZABLE
Why?
Re-optimization will occur on second execution because cardinality estimates off
Adaptive StatisticsAdaptive Plans & Re-optimization working together
Initial Cardinality estimates are off
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.28
On second execution statement is re-parsed using execution statistics
New child cursor created with new plan
New plan changes the join order
Adaptive StatisticsAdaptive Plans & Re-optimization working together
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.29
Adaptive Statistics
Directives are additional information used during optimization to generate a better plan
– For example, when table T1 is joined to T2 use dynamic statistics to get accurate cardinality estimate
– Directives are collected on query expressions not at a statement level Allows for directives to be used for multiple statements
– Persisted on disk in the SYSAUX tablespace– Directives will be automatically maintained– Managed using the new package DBMS_SPD
SQL Plan Directives
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.30
SQL Plan Directives SGA
SHARED_POOL
Select * FROM customerswhere state =‘CA’AND country=‘USA’;
SQL statement is submitted
Cursor 0: Select * from customers ……..
Optimizer determines plan2
DIRECTIVE CACHE
During execution cardinality estimate discovered to be wrong and directive created
3IS_REOPTIMIZABLE = Y
Directive: Use DS for customers table when column city, country are used
1
Sysaux Tablespace
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.31
SQL Plan DirectivesSelect * FROM customerswhere state=‘CA’AND country=‘USA’;
SQL Directives
Sysaux Tablespace
SGA
This execution and subsequent execution use child cursor 1
6
Directive: Use DS for customer table when column city, country are used
SHARED_POOLCursor 0: Select * from customers …..
IS_REOPTIMIZABLE = Y
Optimizer determines new plan using execution statistics
5
Cursor 1: Select * from customers …..IS_REOPTIMIZABLE = N
Directives periodically persisted to disk in the sysaux tablespace
Extension Binds Stats Sqlid
SQL Directives
DIRECTIVE CACHEDIRECTIVE CACHE
Same SQL statement is submitted again
4
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.32
SQL Plan DirectivesSelect emailFROM customerswhere state=‘MA’AND country=‘USA’;
New SQL statement is submitted on customers table
7
Optimizer checks for directives on customers table and finds one on the columns state and country
8
Cursor 0: Select email from customers
Optimizer determines plan with help of directive
9
Optimizer adds column group creation for state & country columns to next statistics gather on customer table
10
SGADIRECTIVE CACHE
SHARED_POOL
Sysaux Tablespace
Extension Binds Stats Sqlid
SQL Directives
DIRECTIVE CACHE
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.33
Number of SQL Plan Directives used for a statement is displayed in the notes section of the execution plan
In this case one SQL Plan Directive was used
SQL Plan DirectivesIn use
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.34
SQL Plan DirectivesMonitoring
Auto flushed every 15 minutes
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.35
SQL Plan DirectivesMonitoring
Auto flushed every 15 minutes
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.36
SQL Plan Directive state was “MISSING_STATS”
Indicates Oracle can automatically gather statistics to improve the original mis-estimation
In this case a column group on STATE and COUNTRY will address the problem
SQL Plan DirectivesGather statistics to allow automatic creation of missing statistics
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.37
SQL Plan DirectivesMonitoring
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.38
Correct statistics in place
Optimizer uses statistics instead of directives
Accurate cardinality estimate seen in the plan
Statement not marked for re-optimization
SQL Plan DirectivesFinal plan no longer needs the directive
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.39
Adaptive PlansJoin methodsParallel distribution
methods
Adaptive Statistics Discovered at compile timeDiscovered at run time
Adaptive Query OptimizationOverview
Adaptive Query Optimization
Adaptive Plans Adaptive Statistics
Join Methods
Parallel distribution
Methods
At compile
time
At run time
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.40
Agenda Adaptive Query Optimization Optimizer Statistics enhancements SQL Plan Management enhancements
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.41
Statistics Enhancements
New types of histograms Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.42
Histograms
Histograms tell Optimizer about the data distribution in a Column Creation controlled by METHOD_OPT parameter Default create histogram on any column that has been used in the
WHERE clause or GROUP BY of a statement AND has a data skew Relies on column usage information gathered at compilation time and
stored in SYS.COL_USAGE$ Four types of histograms
– Frequency – Top-Frequency
Oracle Confidential
– Height balanced– Hybrid
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.43
Histograms
A frequency histogram is only created if the number of distinct values in a column (NDV) is less than 254 values
Frequency Histograms (FREQUENCY)
Frequency histogram
Oracle Confidential
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.44
Histograms
Traditionally a frequency histogram is only created if NDV < 254 But if a small number of values occupies most of the rows (>99%
rows) Creating a frequency histograms on that small set of values is very
useful even though NDV is greater than 254• Ignores the unpopular values to create a better quality histogram for
popular values Built using the same technique used for frequency histograms Only created with AUTO_SAMPLE_SIZE
Top Frequency (TOP-FREQUENCY)
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.45
Histograms
A height balanced histogram is created if the number of distinct values in a column (NDV) is greater than 254 values
Height Balanced Histograms (HEIGHT BALANCED)
Height balanced histogram
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.46
Histograms
A height balanced histogram is created if the number of distinct values in a column (NDV) is greater than 254 values
Hybrid Histograms (HYBRID)
Hybrid histogram
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.47
Histograms
Similar to height balanced histogram as created if the NDV >254 Store the actual frequencies of bucket endpoints in histograms No values are allowed to spill over multiple buckets More endpoint values can be squeezed in a histogram Achieves the same effect as increasing the # of buckets Only created with AUTO_SAMPLE_SIZE
Hybrid Histograms (HYBRID)
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.48
Statistics Enhancements
New types of histograms *Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.49
Statistics Enhancements
New types of histograms Online statistics gathering *Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.50
Statistics Enhancements
New types of histograms Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.51
Enhanced Incremental Statistics
Incremental statistics allows global level statistics to accurately generated from partition level statistics
NDV statistics can now be accurately aggregated by the introduction of the synopsis
The synopses are stored in the Sysaux tablespace In 12c reduced the space required to store synopses on disk
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.52
NDV statistics can now be accurately aggregated
The synopses are stored in the Sysaux tablespace
12c reduces the space required to store synopses on disk
Enhanced Incremental StatisticsSales Table
May 22nd 2012
May 23rd 2012
May 18th 2012
May 19th 2012
May 20th 2012
May 21st 2012
Sysaux Tablespace
S1
S2
S3
S4
S5
S6
1. Partition level stats are gathered & synopsis created
Global Statistic
2. Global stats generated by aggregating partition level statistics and synopsis
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.53
Enhanced Incremental Statistics for Partition Exchange
DBA1. Create external table for flat files
3. Set INCREMENTAL to true & INCREMENTAL_LEVEL to TABLE
4. Gather Statistics
Sales Table
May 22nd 2012
May 23rd 2012
May 18th 2012
May 19th 2012
May 20th 2012
May 21st 20122. Use CTAS command to create non-partitioned table TMP_SALES
TMP_SALES
May 24th 2012
Sysaux Tablespace
Global Statistic
6. Global stats generated by aggregating partition level statistics for existing partition with stats on new partition
S1
S2
S3
S4
S5
S6
S75. Alter table Sales exchange partition May_24_2012 with table tmp_sales
May 24th 2012
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.54
During data load some rows going to the older partitions
In 11g any DML on older partitions triggered partition statistics to be re-gathered
New DBMS_STATS preference INCREMENTAL_STALENESS
When set to USE_STALE_PERCENT DML on less than 10% of rows in older partitions will not trigger re-gather
Enhanced Incremental Statistics Staleness Tolerance
Sales Table
May 22nd 2012
May 23rd 2012
May 18th 2012
May 19th 2012
May 20th 2012
May 21st 2012
Sysaux Tablespace
S1
S2
S3
S4
S5
S6
1. Partition level stats are gathered & synopsis created
Global Statistic
2. Global stats generated by aggregating partition level statistics and synopsis
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.55
Statistics Enhancements
New types of histograms Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics *Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.56
Statistics Enhancements
New types of histograms Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering *Automatic detection of column groups Statistic gathering reporting
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.57
Statistics Enhancements
New types of histograms Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.58
DBMS_STATS.REPORT_STATS_OPERATIONS
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.59
DBMS_STATS.REPORT_GATHER_SCHEMA_STATS
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.60
Agenda Adaptive Query Optimization Optimizer Statistics enhancements SQL Plan Management enhancements
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.61
SQL Plan ManagementHow it works
Parse
HJ
HJ
GB
Plan history
Plan baseline
ExecutePlan Acceptable
HJ
HJ
GB
Users
Actual execution plans stored in SQL plan baseline in Oracle Database 12c
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.62
SQL Plan ManagementHow it works
Parse
NL
NL
GB
Plan history
Plan baseline
HJ
HJ
GB
Users
NL
NL
GB
Note you do not need to be in auto-capture mode to have a new plan added to an existing SQL plan baseline
Note additional fields such as fetches, row processed etc. are not populated because new plan never executed
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.63
SQL Plan ManagementHow it works
Parse
Plan history
Plan baseline
ExecutePlan Acceptable
HJ
HJ
GB
Users
NL
NL
GB
HJ
HJ
GB
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.64
Adaptive SQL Plan Management
New evolve auto task running in the maintenance window– Ranks all non-accepted plans and runs evolve process for them
Newly found plans are ranked the highest– If new plan performs better than existing plan it is automatically accepted– If new plan performs worse than existing plan it will remain unaccepted– Poor performing plans will not be retried for 30 days and then only if the
statement is active
SPM Evolve Advisor
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.65
Automatic Plan Evolution
Plan history
Plan baseline
NL
NL
GB
HJ
HJ
GB
Nightly verification task checks for unaccepted plans
It then test execute to check if new plan is as good as or better than old plan
1
2
Plan history
HJ
HJ
GB
Plan baseline
NL
NL
GB
Plans which perform better than original plan are recommended to be added to the plan baseline
3L
Plans which don’t perform as good as the original plan stay in the plan history & are marked unaccepted but last_verified updated
NL
NL
GB
4
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.66
SPM Evolve AdvisorNew nightly auto task called SYS_AUTO_SPM_EVOLVE_TASK
Information on task found in DBA_ADVISOR_TASKS
Adaptive SQL Plan Management
You can review what happening during the task using DBMS_SPM. REPORT_AUTO_EVOLVE_TASK
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.67
Reports are persisted and can be review any time
Clearer findings section summaries the results of the evolve task
New recommendation section that clearly states how to implement the recommendations proposed
Reviewing the evolve report
Same comparison criteria as in 11g
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.68
Manually Invoking evolve task
Each task is named to allow it to be re-executed any time
Each execution is names to allow the results to be persisted
Results can be viewed at any time using the task_id and exec_id
Insert Chart Here
Adaptive SQL Plan Management
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.69
Summary
Optimizer begins to learn from its mistakes New types of statistics More automatic statistics gathering techniques Faster statistics gathering Multi-Table Left Outer Join Partial Join Evaluation (PJE) New Transformations
– Null-Accepting Semi Join– Scalar Subquery Unnesting
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.70
More Information White papers
– Optimizer with Oracle Database 12c– SQL Plan Management with Oracle Database 12c– Understanding Optimizer Statistics with Oracle Database 12c
Optimizer Blog– http://blogs.oracle.com/optimizer
Oracle.com– http://www.oracle.com/technetwork/database/focus-areas/bi-dat
awarehousing/dbbi-tech-info-optmztn-092214.html
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.71