christo kutrovsky - maximize data warehouse performance with parallel queries

Maximize Data Warehouse Performance with Parallel QueriesPresented by: Christo Kutrovsky

© 2015 Pythian – Presentation for DOAG2

Who Am I• Oracle ACE• 15 years in Oracle field• Joined Pythian 2003• Part of Pythian Consulting Group

• Special projects• Performance tuning• Critical services

• Presenter at: IOUG, RMOUG, UKOUG, OpenWorld

“oracle pinup”

© 2015 Pythian – Presentation for DOAG3 © 2011 Pythian - Confidential

Why Companies Trust Pythian• Recognized Leader:• Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server• Work with over 150 multinational companies such as Forbes.com, Fox Sports,

Nordion and Western Union to help manage their complex IT deployments• Expertise:• One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 7 Oracle ACEs/ACE Directors. • Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC.• Global Reach & Scalability:• 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response

http://www.upenn.edu/

http://www.mdsinc.com/


Agenda•Review Parallel Query concepts•Review Exadata Data Flow capabilities•The Big Sort•The Big Group By•The Big Join•Final Thoughts

5

Parallel Query Concepts


Oracle Parallel Query• Producer/Consumer model• A PQ slave is either a producer or consumer, never both

Figure 8-2 Inter-operation Parallelism and Dynamic PartitioningOracle® Database VLDB and Partitioning Guide


Oracle Parallel Query• A PQ slave is either a producer or consumer in relation to other PQ slaves

PQ1

PQ2

PQ3

PQ4

PQ5

PQ6

PQ7

PQ8

SGA

DB


Oracle Parallel Query• But logically, step-wise, when there are two sets it’s flow-through

PQ1

PQ2

PQ3

PQ4

PQ5

PQ6

PQ7

PQ8

SGA

DB

ONE STEP (1)


Oracle Parallel Query• QC is always flow-throught

PQ1

PQ2

PQ3

PQ4

PQ5

PQ6

PQ7

PQ8

SGA

DBSGA

QC

SQLNET CLIENTONE STEP

(2)


Oracle Parallel Query• select count(*) from table;

PQ1

PQ2

PQ3

PQ4

SGA

DB

ONE STEP

QC

SQLNET CLIENT


Oracle Parallel Query•Producer/consumer model•A slave is either producer and consumer in relation to other slaves.

13

Exadata Data Flow


Multiple Tiers• Every database has multiple tiers

• Memory• SAN/NFS – network• SAN Performance• CPU Capacity• Client side connectivity

• Each tier has it’s own nuances, properties and bandwidth limitations


Exadata X2-2 Data Flow (intra)• Each DB node (compute) and CELL node (storage)

• Single dual port 40 Gbit infiniband (QDR)• 3,200 Mb/sec (full-duplex)

• Each CELL node• Single dual port 40 Gbit infiniband (QDR)

• 3,200 Mb/sec (full-duplex)• 12 x 4 TB SAS drives (7200 rpm)

• Total bandwidth: 1,000 - 2,000 Mb/sec• 4 x Sun Flash Arrays

• Total bandwidth: 4,100 Mb/sec


Exadata Data Flow (External)• SQL Net Clients talk only to DB nodes• Each DB node

• 4 x 10 gigabit network ( 4 x 1.08 Gbytes/sec)• 2 x 10 Gigabit network (opt) ( 2 x 1.08 Gbytes/sec)

• Alternatively, can use IP over infiniband• Exalogic• RMAN backups• Etc.

© 2015 Pythian – Presentation for DOAG

Exadata Data Flow (Half Rack)

DB1

INFINIBAND SWITCHES

DB2 DB3 DB4

CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07

SGA SGA SGA SGACLIENTRESULT

SET


DISK: 2 GB/sec + FLASH: 4 GB/sec= 6 GB/secX 7 cells = 42 GB/sec

Single Cell (Half Rack)

DB1

INFINIBAND SWITCHES

DB2 DB3 DB4

CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07


SET

CELL 01


Cell output (Half Rack)

DB1

INFINIBAND SWITCHES

DB2 DB3 DB4



SET

CELL to SWITCH: 3.2 GB/sec

X 7 cells = 22.4 GB/sec


DB Node Consumption (Half Rack)

DB1

INFINIBAND SWITCHES

DB2 DB3 DB4



SET

SWITCH to DB NODE: 3.2 GB/sec

X 4 nodes = 12.8 GB/sec


Memory speed (Half Rack)

DB1

INFINIBAND SWITCHES

DB2 DB3 DB4



SET

DB to SGA (memory) 22 GB/sec(very CPU dependant)X 4 DB = 88 GB/sec


DB Node to Client (Half Rack)

DB1

INFINIBAND SWITCHES

DB2 DB3 DB4



SET

DB to SINGLE SQLNET Client112 MB/sec (1 GBIT)1.08 GB/sec (10 GBIT)


Exadata Data Flow (External)• Ethernet link aggregation requires

• Single conversation runs on a single path• Ethernet standard

• One database sessions runs on single TCP connection, which must run on single path• Switches and network cards can implement different hashing algorithms to determine paths

• Most are on IP level. Between 2 IPs the same NICs are used, regardless of number of connections


Exadata Data Flow (Half Rack)

DB4 eth0

DB to THIS Client: 112 MB/sec (theoretical)(18 MB/sec per session)

IP: 10.31.33.1SessionSession

Session

SessionSession

Session

eth1

eth2

eth3

eth0

eth1

eth2

eth3

Switch


Exadata Data Flow (External)• As per Exadata Documentation:

• Data Load Rate: Up to 10.5 TB/hour

• 764 MB/sec per DB node• 3 GB/sec – can your other infrastructure provide such performance?

26

The Big Sort


Sample Data• My favorite table – CKK

• 39 GiB of data• 10,000,000 rows• id, mod5_id, mod5000_id, dt_dec, dt_pos etc.

create table ckk /*parallel 8 */nologging tablespace ckbig asselect rownum id, mod(rownum,5) mod5_id, mod(rownum,5000) mod5000_id, sysdate dt_fixed, sysdate - rownum/24/60 dt_dec, sysdate + rownum/24/60 dt_pos, sysdate + ora_hash(rownum,65,535)/24 dt_rand, sysdate+mod(rownum,10) dt_mod10, rpad('x',3500,'x') fillerfrom (select rownum r from dual connect by level <= 10000) r1, (select rownum r from dual connect by level <= 1000);


Parallel Sortselect /*+ parallel(4) */ * from ckk c1 order by id;----------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |----------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 33G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (ORDER) | :TQ10001 | 10M| 33G| | Q1,01 | P->S | QC (ORDER) || 3 | SORT ORDER BY | | 10M| 33G| 38G| Q1,01 | PCWP | || 4 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | || 5 | PX SEND RANGE | :TQ10000 | 10M| 33G| | Q1,00 | P->P | RANGE || 6 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | || 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |----------------------------------------------------------------------------------------------------------

PQ1

PQ2

PQ3

PQ4

PQ5 – up ID=2.5 M

PQ6 – up ID=5.0 M

PQ7 – up ID=7.5 M

PQ8 – up ID=10 M

SGA

DB


STORAGE FULL (OFFLOADED)

DB1

INFINIBAND SWITCHES

DB2 DB3 DB4


SET

Decompression, Projection and FilterHappens on EVERY CELL NODE regardless of query parallelism.MPP mode.


Parallel Sortselect dfo_number "d", tq_id as "t", server_type, num_rows,rpad('x',round(num_rows*10/nullif(max(num_rows) over (partition by dfo_number, tq_id, server_type),0)),'x') as "pr", round(bytes/1024/1024) mb, process, instance i,round(ratio_to_report (num_rows) over (partition by dfo_number, tq_id, server_type)*100) as "%", open_time, avg_latency, waits, timeouts,round(bytes/nullif(num_rows,0)) as "b/r” from v$pq_tqstat order by dfo_number, tq_id, server_type desc, process;;d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Ranger 372 xxxxxxxxxx 2 QC 1 1001 0 Producer 2,692,340 xxxxxxxxxx 9,161 P002 1 271 0 Producer 2,499,910 xxxxxxxxx 8,506 P002 2 251 0 Producer 2,692,340 xxxxxxxxxx 9,161 P003 2 271 0 Producer 2,115,410 xxxxxxxx 7,198 P003 1 211 0 Consumer 6,146,614 xxxxxxxxxx 20,915 P000 2 611 0 Consumer 781,680 x 2,660 P000 1 81 0 Consumer 1,968,128 xxx 6,697 P001 2 201 0 Consumer 1,103,578 xx 3,755 P001 1 111 1 Producer 13 x 0 P000 1 51 1 Producer 212 xxxxxxxxxx 1 P000 2 841 1 Producer 13 x 0 P001 2 51 1 Producer 13 x 0 P001 1 51 1 Consumer 200 xxxxxxxxxx 1 QC 1 100


Parallel Sortselect /*+ parallel(4) */ * from ckk c1 order by id;----------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |----------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 33G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (ORDER) | :TQ10001 | 10M| 33G| | Q1,01 | P->S | QC (ORDER) || 3 | SORT ORDER BY | | 10M| 33G| 38G| Q1,01 | PCWP | || 4 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | || 5 | PX SEND RANGE | :TQ10000 | 10M| 33G| | Q1,00 | P->P | RANGE || 6 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | || 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |----------------------------------------------------------------------------------------------------------

PQ1

PQ2

PQ3

PQ4

PQ5 – up to ID=6.1 M (6.1 M)PQ6 – up to ID=6.9 M (0.8 M)PQ7 – up to ID=8.8 M (1.9 M)PQ8 – up to ID=10.0 M (1.1M

SGA

DB


The Ranger• The Ranger Bug

• The RANGER uses a sample from the data to determine the boundaries for PQ sort operations• Sometimes it’s off by a lot• It will ALWAYS miss when the input data is already somewhat sorted

• Limit your scalability – no matter how big your machine is


Parallel Sort – random dataselect /*+ parallel(4) */ id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos, dt_rand, dt_mod10, substr(filler,1,2000) from ckk c1 order by dt_rand;--90 secd t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Ranger 372 xxxxxxxxxx 1 QC 1 1001 0 Producer 2,500,030 xxxxxxxxxx 4,925 P002 1 251 0 Producer 2,500,030 xxxxxxxxxx 4,925 P002 2 251 0 Producer 2,499,910 xxxxxxxxxx 4,925 P003 1 251 0 Producer 2,500,030 xxxxxxxxxx 4,925 P003 2 251 0 Consumer 2,424,554 xxxxxxxxx 4,776 P000 2 241 0 Consumer 2,576,164 xxxxxxxxx 5,075 P000 1 261 0 Consumer 2,272,136 xxxxxxxx 4,476 P001 2 231 0 Consumer 2,727,146 xxxxxxxxxx 5,373 P001 1 271 1 Producer 218 xxxxxxxxxx 0 P000 1 761 1 Producer 23 x 0 P000 2 81 1 Producer 23 x 0 P001 2 81 1 Producer 23 x 0 P001 1 81 1 Consumer 200 xxxxxxxxxx 0 QC 1 100


Parallel Sort – random dataselect /*+ parallel(32) */ id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos, dt_rand, dt_mod10, substr(filler,1,2000) from ckk c1 order by dt_rand;--10 secd t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Ranger 3,872 xxxxxxxxxx 11 QC 1 1001 0 Producer 312,520 xxxxxxxxxx 616 P016 1 31 0 Producer 312,520 xxxxxxxxxx 616 P016 2 31 0 Producer 312,520 xxxxxxxxxx 616 P017 1 31 0 Producer 312,520 xxxxxxxxxx 616 P017 2 3...1 0 Consumer 151,262 xxx 298 P003 1 21 0 Consumer 454,696 xxxxxxxxxx 896 P003 2 51 0 Consumer 302,320 xxxxxxx 596 P004 1 31 0 Consumer 151,731 xxx 299 P004 2 21 0 Consumer 303,429 xxxxxxx 598 P005 1 31 0 Consumer 454,650 xxxxxxxxxx 896 P005 2 51 0 Consumer 303,259 xxxxxxx 597 P006 2 31 0 Consumer 455,172 xxxxxxxxxx 897 P006 1 51 0 Consumer 302,945 xxxxxxx 597 P012 1 31 0 Consumer 303,318 xxxxxxx 598 P012 2 31 0 Consumer 303,245 xxxxxxx 597 P013 1 31 0 Consumer 304,029 xxxxxxx 599 P013 2 31 0 Consumer 301,937 xxxxxxx 595 P014 2 31 0 Consumer 454,903 xxxxxxxxxx 896 P014 1 51 0 Consumer 151,887 xxx 299 P015 1 21 1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100


Parallel Sort – random dataselect /*+ parallel(32) */ id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos, dt_rand, dt_mod10, substr(filler,1,2000) from ckk c1 order by id;--20 secd t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Ranger 3,872 xxxxxxxxxx 11 QC 1 1001 0 Producer 240,400 xxxxxx 474 P016 2 21 0 Producer 384,640 xxxxxxxxx 758 P016 1 41 0 Producer 216,360 xxxxx 426 P017 1 21 0 Producer 192,320 xxxx 379 P017 2 2...1 0 Consumer 373,702 xxxx 736 P000 1 41 0 Consumer 372,530 xxxx 734 P000 2 41 0 Consumer 124,801 x 246 P001 2 11 0 Consumer 183,552 xx 361 P001 1 21 0 Consumer 734,398 xxxxxxx 1,446 P002 1 71 0 Consumer 22,402 44 P002 2 01 0 Consumer 191,810 xx 378 P003 1 21 0 Consumer 132,990 x 262 P003 2 11 0 Consumer 32,640 64 P004 1 01 0 Consumer 920,448 xxxxxxxxx 1,813 P004 2 91 0 Consumer 167,232 xx 329 P005 1 21 0 Consumer 110,466 x 218 P005 2 11 0 Consumer 1,055,669 xxxxxxxxxx 2,080 P014 2 111 0 Consumer 130,944 x 258 P014 1 11 0 Consumer 60,250 x 119 P015 2 11 0 Consumer 20,298 40 P015 1 0...1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100


The Ranger issue• Distribution can be much worse

• When using analytics or other “input” into the sort operation that is pre-sorted• Often 90% goes to one slave, 10% to remaining• “Been worked on” – I am told

• Problem made even worse due to lack of large buffers


Writing to disk – lack of buffering• When a slave is writing to disk – it cannot receive any messages or can receive a very small number of messages• Results in slowdowns when Oracle has to write to TEMP

• Massive slowdown when only a few slaves have to write to TEMP• All producers have to wait

• Significant slowdown when all slaves have to write to TEMP• Cannot parallelise sufficiently to leverage available disk bandwidth

38

The Big Group By


Group Byselect /*+ parallel(4) */ mod5_id, count(*) from ckk group by mod5_id;---------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |---------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 5 | 15 | | | || 1 | PX COORDINATOR | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 5 | 15 | Q1,01 | P->S | QC (RAND) || 3 | HASH GROUP BY | | 5 | 15 | Q1,01 | PCWP | || 4 | PX RECEIVE | | 5 | 15 | Q1,01 | PCWP | || 5 | PX SEND HASH | :TQ10000 | 5 | 15 | Q1,00 | P->P | HASH || 6 | HASH GROUP BY | | 5 | 15 | Q1,00 | PCWP | || 7 | PX BLOCK ITERATOR | | 10M| 28M| Q1,00 | PCWC | || 8 | TABLE ACCESS STORAGE FULL| CKK | 10M| 28M| Q1,00 | PCWP | |---------------------------------------------------------------------------------------------------

select /*+ parallel(4) NO_GBY_PUSHDOWN*/ mod5_id, count(*) from ckk group by mod5_id;

--------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |--------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 5 | 15 | | | || 1 | PX COORDINATOR | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 5 | 15 | Q1,01 | P->S | QC (RAND) || 3 | HASH GROUP BY | | 5 | 15 | Q1,01 | PCWP | || 4 | PX RECEIVE | | 10M| 28M| Q1,01 | PCWP | || 5 | PX SEND HASH | :TQ10000 | 10M| 28M| Q1,00 | P->P | HASH || 6 | PX BLOCK ITERATOR | | 10M| 28M| Q1,00 | PCWC | || 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 28M| Q1,00 | PCWP | |--------------------------------------------------------------------------------------------------


Oracle Parallel Query• But logically, step-wise, when there are two sets it’s flow-through

PQ1PQ2PQ3PQ4

SGADB

STEP (1)

GROUPPGA

GROUPPGA

GROUPPGA

GROUPPGA

PQ5PQ6PQ7PQ8

GROUPPGA

GROUPPGA

GROUPPGA

GROUPPGA

SGA

QCSTEP (2)


Group by – small result setselect /*+ parallel(4) */ mod5_id, count(*) from ckkgroup by mod5_id;-- 4.8 sec

d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Producer 5 xxxxxxxxxx 0 P002 2 251 0 Producer 5 xxxxxxxxxx 0 P002 1 251 0 Producer 5 xxxxxxxxxx 0 P003 2 251 0 Producer 5 xxxxxxxxxx 0 P003 1 251 0 Consumer 4 xxxxx 0 P000 2 201 0 Consumer 8 xxxxxxxxxx 0 P000 1 401 0 Consumer 4 xxxxx 0 P001 2 201 0 Consumer 4 xxxxx 0 P001 1 201 1 Producer 2 xxxxxxxxxx 0 P000 1 401 1 Producer 1 xxxxx 0 P000 2 201 1 Producer 1 xxxxx 0 P001 1 201 1 Producer 1 xxxxx 0 P001 2 201 1 Consumer 5 xxxxxxxxxx 0 QC 1 100


Group by – small result set

d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Producer 2,692,340 xxxxxxxxxx 10 P004 2 271 0 Producer 2,307,600 xxxxxxxxx 8 P004 1 231 0 Producer 2,500,030 xxxxxxxxx 9 P005 2 251 0 Producer 2,500,030 xxxxxxxxx 9 P005 1 251 0 Consumer 4,000,000 xxxxxxxxxx 15 P002 2 401 0 Consumer 2,000,000 xxxxx 8 P002 1 201 0 Consumer 2,000,000 xxxxx 6 P003 2 201 0 Consumer 2,000,000 xxxxx 8 P003 1 201 1 Producer 1 xxxxx 0 P002 1 201 1 Producer 2 xxxxxxxxxx 0 P002 2 401 1 Producer 1 xxxxx 0 P003 1 201 1 Producer 1 xxxxx 0 P003 2 201 1 Consumer 5 xxxxxxxxxx 0 QC 1 100

select /*+ parallel(4) NO_GBY_PUSHDOWN*/ mod5_id, count(*) from ckkgroup by mod5_id;-- 4.8 sec


Group by – small result set

d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Producer 264,440 xxxxxx 1 P016 2 31 0 Producer 312,520 xxxxxxxx 1 P016 1 31 0 Producer 240,400 xxxxxx 1 P017 1 21 0 Producer 360,600 xxxxxxxxx 1 P017 2 4...1 0 Consumer 0 0 P000 2 01 0 Consumer 2,000,000 xxxxxxxxxx 8 P000 1 201 0 Consumer 0 0 P001 2 0...1 0 Consumer 0 0 P004 1 01 0 Consumer 0 0 P005 2 01 0 Consumer 2,000,000 xxxxxxxxxx 6 P005 1 201 0 Consumer 0 0 P006 2 01 0 Consumer 0 0 P006 1 01 0 Consumer 0 0 P007 2 01 0 Consumer 2,000,000 xxxxxxxxxx 8 P007 1 201 0 Consumer 2,000,000 xxxxxxxxxx 8 P008 1 201 0 Consumer 2,000,000 xxxxxxxxxx 8 P010 2 20...

select /*+ parallel(32) NO_GBY_PUSHDOWN*/ mod5_id, count(*) from ckkgroup by mod5_id;-- 4.8 sec


Group by – small result set• When the result set from a group by is small – there are a lot of optimisations that can kick in

• PUSHDOWN saves on PQ traffic• Receiving end has very little work to do• Very efficient to parallelise


select /*+ parallel(4) */ id, count(*) from ckk group by id;-- 4.8 sec

Group by – large result set HASH

d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Producer 2,500,030 xxxxxxxx 17 P002 2 251 0 Producer 2,115,410 xxxxxx 14 P002 1 211 0 Producer 2,115,410 xxxxxx 14 P003 2 211 0 Producer 3,269,150 xxxxxxxxxx 21 P003 1 331 0 Consumer 2,499,213 xxxxxxxxxx 16 P000 1 251 0 Consumer 2,499,807 xxxxxxxxxx 16 P000 2 251 0 Consumer 2,500,981 xxxxxxxxxx 16 P001 1 251 0 Consumer 2,499,999 xxxxxxxxxx 16 P001 2 251 1 Producer 1,465 xxxxx 0 P000 2 171 1 Producer 2,929 xxxxxxxxxx 0 P000 1 331 1 Producer 2,928 xxxxxxxxxx 0 P001 1 331 1 Producer 1,464 xxxxx 0 P001 2 171 1 Consumer 200 xxxxxxxxxx 0 QC 1 100


select /*+ parallel(4) */ id, count(*) from ckk group by id order by id;-- 6.5 sec

Group by – large result set SORT

d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Ranger 372 xxxxxxxxxx 0 QC 1 1001 0 Producer 2,499,910 xxxxxxxxx 17 P002 1 251 0 Producer 2,692,340 xxxxxxxxxx 18 P002 2 271 0 Producer 2,500,030 xxxxxxxxx 17 P003 2 251 0 Producer 2,307,720 xxxxxxxxx 15 P003 1 231 0 Consumer 6,154,678 xxxxxxxxxx 40 P000 2 621 0 Consumer 793,712 x 5 P000 1 81 0 Consumer 1,095,514 xx 7 P001 1 111 0 Consumer 1,956,096 xxx 13 P001 2 201 1 Producer 4,391 xxxxxxxx 0 P000 1 241 1 Producer 5,379 xxxxxxxxxx 0 P000 2 291 1 Producer 4,391 xxxxxxxx 0 P001 2 241 1 Producer 4,391 xxxxxxxx 0 P001 1 241 1 Consumer 200 xxxxxxxxxx 0 QC 1 100


select /*+ parallel(32) */ id, count(*) from ckk group by id;-- 4.8 sec


d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Producer 360,600 xxxxxxxxx 2 P016 1 41 0 Producer 240,400 xxxxxx 2 P016 2 21 0 Producer 312,520 xxxxxxxx 2 P017 1 31 0 Producer 336,560 xxxxxxxxx 2 P017 2 3...1 0 Consumer 312,829 xxxxxxxxxx 2 P000 2 31 0 Consumer 311,690 xxxxxxxxxx 2 P000 1 31 0 Consumer 312,371 xxxxxxxxxx 2 P001 1 31 0 Consumer 312,854 xxxxxxxxxx 2 P001 2 31 0 Consumer 312,255 xxxxxxxxxx 2 P002 2 31 0 Consumer 312,321 xxxxxxxxxx 2 P002 1 31 0 Consumer 313,241 xxxxxxxxxx 2 P003 2 31 0 Consumer 312,605 xxxxxxxxxx 2 P003 1 31 0 Consumer 312,202 xxxxxxxxxx 2 P004 1 31 0 Consumer 311,294 xxxxxxxxxx 2 P004 2 31 0 Consumer 312,338 xxxxxxxxxx 2 P005 1 31 0 Consumer 312,206 xxxxxxxxxx 2 P005 2 31 0 Consumer 312,898 xxxxxxxxxx 2 P006 1 31 0 Consumer 312,817 xxxxxxxxxx 2 P006 2 31 0 Consumer 312,291 xxxxxxxxxx 2 P007 1 31 0 Consumer 312,872 xxxxxxxxxx 2 P007 2 3...1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100



d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Producer 312,520 xxxxxxxxxx 1,047 P016 1 31 0 Producer 312,520 xxxxxxxxxx 1,047 P016 2 31 0 Producer 312,520 xxxxxxxxxx 1,047 P017 1 3...1 0 Consumer 311,603 xxxxxxxxxx 1,044 P000 1 31 0 Consumer 312,470 xxxxxxxxxx 1,047 P000 2 31 0 Consumer 312,484 xxxxxxxxxx 1,047 P001 2 31 0 Consumer 312,662 xxxxxxxxxx 1,048 P001 1 31 0 Consumer 312,341 xxxxxxxxxx 1,047 P002 2 31 0 Consumer 312,692 xxxxxxxxxx 1,048 P002 1 31 0 Consumer 313,114 xxxxxxxxxx 1,049 P003 2 31 0 Consumer 311,889 xxxxxxxxxx 1,045 P003 1 31 0 Consumer 312,323 xxxxxxxxxx 1,047 P004 1 31 0 Consumer 313,310 xxxxxxxxxx 1,050 P004 2 31 0 Consumer 312,715 xxxxxxxxxx 1,048 P005 2 31 0 Consumer 313,394 xxxxxxxxxx 1,050 P005 1 31 0 Consumer 312,718 xxxxxxxxxx 1,048 P006 1 31 0 Consumer 312,090 xxxxxxxxxx 1,046 P006 2 31 0 Consumer 312,720 xxxxxxxxxx 1,048 P007 1 31 0 Consumer 311,889 xxxxxxxxxx 1,045 P007 2 32 ...1 1 Consumer 200 xxxxxxxxxx 1 QC 1 100

select /*+ parallel(32) */ id, filler, count(*) from ckk group by id,filler;-- 18 sec


Group by – large result set SORT

d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %1 0 Ranger 3,872 xxxxxxxxxx 23 QC 1 1001 0 Producer 288,480 xxxxxxx 967 P016 2 31 0 Producer 288,480 xxxxxxx 967 P016 1 31 0 Producer 336,560 xxxxxxxx 1,128 P017 2 31 0 Producer 240,400 xxxxxx 806 P017 1 2...1 0 Consumer 722,115 xxxxxx 2,420 P002 2 71 0 Consumer 16,386 55 P002 1 01 0 Consumer 210,112 xx 704 P003 2 21 0 Consumer 112,510 x 377 P003 1 11 0 Consumer 940,928 xxxxxxxx 3,153 P004 1 91 0 Consumer 34,688 116 P004 2 01 0 Consumer 171,392 xx 574 P005 2 21 0 Consumer 110,466 x 370 P005 1 11 0 Consumer 163,658 x 548 P014 2 21 0 Consumer 1,113,523 xxxxxxxxxx 3,732 P014 1 111 0 Consumer 11,100 37 P015 1 01 0 Consumer 4,150 14 P015 2 0...1 1 Consumer 200 xxxxxxxxxx 1 QC 1 100

select /*+ parallel(32) */ id, filler, count(*) from ckk group by id,filler order by id,filler;-- 41 sec


Group by – summary• Theoretical performance is reachable• SORT based operations have major issues

• Especially with sorted inputs, such as from analytics• model clause can help – partition by is hash based

• Partial disk staging causes major slowdowns• Balanced operation is critical


Group by – Partitioning• What about partitioning?

create table ckk_p (id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos, dt_rand, dt_mod10, filler)

partition by hash (id) partitions 32 tablespace ckbig nologging parallel 4

asselect * from ckk;


Group by – with partitioningselect /*+ parallel(4) */ id,count(*) from ckk_p group by id,filler;------------------------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | Pstart| Pstop | TQ |IN-OUT| PQ Distrib |------------------------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 7071K| 23G| | | | | | || 1 | PX COORDINATOR | | | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10000 | 7071K| 23G| | | | Q1,00 | P->S | QC (RAND) || 3 | PX PARTITION HASH ALL | | 7071K| 23G| | 1 | 32 | Q1,00 | PCWC | || 4 | HASH GROUP BY | | 7071K| 23G| 38G| | | Q1,00 | PCWP | || 5 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | 1 | 32 | Q1,00 | PCWP | |------------------------------------------------------------------------------------------------------------------------

PQ1PQ2PQ3PQ4

SGA

STEP (1)

GROUPPGA

GROUPPGA

GROUPPGA

GROUPPGA

QCLIST

P1P2P3P4

P5P6P7P8


select /*+ parallel(64) */ * from ckk c1 order by id;----------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |----------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 7071K| 23G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 7071K| 23G| | Q1,01 | P->S | QC (RAND) || 3 | HASH GROUP BY | | 7071K| 23G| 38G| Q1,01 | PCWP | || 4 | PX RECEIVE | | 10M| 32G| | Q1,01 | PCWP | || 5 | PX SEND HASH | :TQ10000 | 10M| 32G| | Q1,00 | P->P | HASH || 6 | PX BLOCK ITERATOR | | 10M| 32G| | Q1,00 | PCWC | || 7 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | Q1,00 | PCWP | |----------------------------------------------------------------------------------------------------------

Group by – enough partitions

PQ1PQ2PQ3PQ4

SGA

STEP (1)

GROUPPGA

GROUPPGA

GROUPPGA

GROUPPGA

QCLIST

P1P2P3P4

P5P6P7P8


Analytics don’t speak parallelselect /*+ parallel(4) */ id, row_number() over (partition by id,filler order by null) r

from ckk_p ;----------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |----------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 32G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 32G| | Q1,01 | P->S | QC (RAND) || 3 | WINDOW SORT | | 10M| 32G| 38G| Q1,01 | PCWP | || 4 | PX RECEIVE | | 10M| 32G| | Q1,01 | PCWP | || 5 | PX SEND HASH | :TQ10000 | 10M| 32G| | Q1,00 | P->P | HASH || 6 | PX BLOCK ITERATOR | | 10M| 32G| | Q1,00 | PCWC | || 7 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | Q1,00 | PCWP | |----------------------------------------------------------------------------------------------------------

select /*+ NO_parallel */ id, row_number() over (partition by id,filler order by null) r from ckk_p ;--------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | Pstart| Pstop |--------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 32G| | | || 1 | PARTITION HASH ALL | | 10M| 32G| | 1 | 32 || 2 | WINDOW SORT | | 10M| 32G| 38G| | || 3 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | 1 | 32 |--------------------------------------------------------------------------------------

55

The Big Join


Test Data• Copy of ckk – ckk2• ckk_m5 – 5 rows dimension

create table ckk_m5 as select rownum mod5_id, rpad('x',1000,'x') filler from dual connect by level <=5;alter table ckk_m5 modify (mod5_id not null);


Simple Hash Joinselect /*+ parallel(4) */ * from ckk c1 join ckk_m5 m5 on (c1.mod5_id = m5.mod5_id);--------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |--------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 42G| | | || 1 | PX COORDINATOR | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 42G| Q1,01 | P->S | QC (RAND) ||* 3 | HASH JOIN | | 10M| 42G| Q1,01 | PCWP | || 4 | PX RECEIVE | | 5 | 5020 | Q1,01 | PCWP | || 5 | PX SEND BROADCAST | :TQ10000 | 5 | 5020 | Q1,00 | P->P | BROADCAST || 6 | PX BLOCK ITERATOR | | 5 | 5020 | Q1,00 | PCWC | || 7 | TABLE ACCESS STORAGE FULL| CKK_M5 | 5 | 5020 | Q1,00 | PCWP | || 8 | PX BLOCK ITERATOR | | 10M| 33G| Q1,01 | PCWC | || 9 | TABLE ACCESS STORAGE FULL | CKK | 10M| 33G| Q1,01 | PCWP | |--------------------------------------------------------------------------------------------------


Simple Hash Join

PQ1PQ2PQ3PQ4

SGA

STEP (1)PQ5PQ6PQ7PQ8

HASHPGA

HASHPGA

HASHPGA

HASHPGA

SGA

QCCKK_M50 GiB

All Hash Areas:0 GiB


Simple Hash Join

PQ5PQ6PQ7PQ8

HASHPGA

HASHPGA

HASHPGA

HASHPGA

SGA

STEP (2)

QC

CKK(39 GiB)


Big Hash Joinselect /*+ parallel(4) */ * from ckk c1join ckk2 c2 on (c1.id = c2.id) -------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp |IN-OUT| PQ Distrib |-------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 66G| | | || 1 | PX COORDINATOR | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10002 | 10M| 66G| | P->S | QC (RAND) ||* 3 | HASH JOIN BUFFERED | | 10M| 66G| 8502M| PCWP | || 4 | PX RECEIVE | | 10M| 33G| | PCWP | || 5 | PX SEND HASH | :TQ10000 | 10M| 33G| | P->P | HASH || 6 | PX BLOCK ITERATOR | | 10M| 33G| | PCWC | || 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | PCWP | || 8 | PX RECEIVE | | 10M| 33G| | PCWP | || 9 | PX SEND HASH | :TQ10001 | 10M| 33G| | P->P | HASH || 10 | PX BLOCK ITERATOR | | 10M| 33G| | PCWC | || 11 | TABLE ACCESS STORAGE FULL| CKK2 | 10M| 33G| | PCWP | |-------------------------------------------------------------------------------------------------


Big Hash Join

PQ1PQ2PQ3PQ4

SGA


HASHPGA

HASHPGA

HASHPGA

HASHPGA

SGA

QCCKK39 GiB



Big Hash Join

PQ1PQ2PQ3PQ4

SGA


HASHPGA

HASHPGA

HASHPGA

HASHPGA

SGA

QCCKK239 GiB



Simple Hash Join

PQ5PQ6PQ7PQ8

HASHPGA

HASHPGA

HASHPGA

HASHPGA

SGA

STEP (3)

QC


Hash Join summary• HASH JOINS with small tables are extremely fast

• Uses BROADCAST to send copy of table to every slave• As much PGA work area as parallel degree * small table size

• HASH JOINS with big tables can be extremely expensive• Almost entire result set written to temp• Unless – partitioning


Semi-Partitioned Hash Joinselect /*+ parallel(32) */ * from ckk c1 join ckk_p c2 on (c1.id = c2.id)-----------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |-----------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 66G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) ||* 3 | HASH JOIN | | 10M| 66G| 1062M| Q1,01 | PCWP | || 4 | PART JOIN FILTER CREATE | :BF0000 | 10M| 33G| | Q1,01 | PCWP | || 5 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | || 6 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) || 7 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | || 8 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | || 9 | PX PARTITION HASH JOIN-FILTER| | 10M| 33G| | Q1,01 | PCWC | || 10 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | |-----------------------------------------------------------------------------------------------------------


Semi Partitioned Hash Join

PQ1PQ2PQ3PQ4

SGA


HASHPGA

HASHPGA

HASHPGA

HASHPGA

SGA

QCCKK39 GiB



Semi Partitioned Hash Join

PQ5PQ6PQ7PQ8

HASHPGA

HASHPGA

HASHPGA

HASHPGA

SGA

STEP (2)

QC

CKK_P(39 GiB)

LISTP1P2P3P4

P5P6P7P8


Semi Partitioned Hash Join• Dramatically reduces HASH Area sizes

• Very likely to fit in RAM – no disk• Reduces from 3 steps to 2 steps• Always reads the non-partitioned table into HASH areas

• Each slave reads it’s own partition


Partitioned Hash Joinselect /*+ parallel(4) */ * from ckk_p c1 join ckk_p c2 on (c1.id = c2.id)--------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |--------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 66G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10000 | 10M| 66G| | Q1,00 | P->S | QC (RAND) || 3 | PX PARTITION HASH ALL | | 10M| 66G| | Q1,00 | PCWC | ||* 4 | HASH JOIN | | 10M| 66G| 8502M| Q1,00 | PCWP | || 5 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | || 6 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | |--------------------------------------------------------------------------------------------------------


Switching placesselect /*+ parallel(32) */ * from ckk_p c1 join ckk c2 on (c1.id = c2.id);----------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |----------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 66G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) ||* 3 | HASH JOIN BUFFERED | | 10M| 66G| 1062M| Q1,01 | PCWP | || 4 | PX PARTITION HASH ALL | | 10M| 33G| | Q1,01 | PCWC | || 5 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | || 6 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | || 7 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) || 8 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | || 9 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |----------------------------------------------------------------------------------------------------------

select /*+ parallel(32) LEADING(c2) */ * from ckk_p c1 join ckk c2 on (c1.id = c2.id);-----------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |-----------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 66G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) ||* 3 | HASH JOIN | | 10M| 66G| 1062M| Q1,01 | PCWP | || 4 | PART JOIN FILTER CREATE | :BF0000 | 10M| 33G| | Q1,01 | PCWP | || 5 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | || 6 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) || 7 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | || 8 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | || 9 | PX PARTITION HASH JOIN-FILTER| | 10M| 33G| | Q1,01 | PCWC | || 10 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | |-----------------------------------------------------------------------------------------------------------


Outer Joinsselect /*+ parallel(32) */ * from ckk_p c1 left join ckk c2 on (c1.id = c2.id);----------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |----------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 66G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) ||* 3 | HASH JOIN OUTER BUFFERED | | 10M| 66G| 1062M| Q1,01 | PCWP | || 4 | PX PARTITION HASH ALL | | 10M| 33G| | Q1,01 | PCWC | || 5 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | || 6 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | || 7 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) || 8 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | || 9 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |----------------------------------------------------------------------------------------------------------

select /*+ parallel(32) SWAP_JOIN_INPUTS(c2)*/ * from ckk_p c1 left join ckk c2 on (c1.id = c2.id);----------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |----------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10M| 66G| | | | || 1 | PX COORDINATOR | | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) ||* 3 | HASH JOIN RIGHT OUTER | | 10M| 66G| 1062M| Q1,01 | PCWP | || 4 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | || 5 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) || 6 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | || 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | || 8 | PX PARTITION HASH ALL | | 10M| 33G| | Q1,01 | PCWC | || 9 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | |----------------------------------------------------------------------------------------------------------

73

The Big Join – with Index


Adding a few indexescreate index ckk$id on ckk(id) parallel 32 tablespace ckbig nologging;--6.5 seccreate index ckk$mod5000_id on ckk(mod5000_id) parallel 32 tablespace ckbig nologging compress;--5.8 seccreate index ckk_p$mod5000_id on ckk_p(mod5000_id) parallel 32 tablespace ckbig nologging compress local;--16 sec


Direct index accessselect /*+ parallel(4) */ * from ckk where mod5000_id=100;-----------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes|-----------------------------------------------------------------------| 0 | SELECT STATEMENT | | 2000 | 6941K|| 1 | TABLE ACCESS BY INDEX ROWID| CKK | 2000 | 6941K||* 2 | INDEX RANGE SCAN | CKK$MOD5000_ID | 2000 | |-----------------------------------------------------------------------

select /*+ parallel(4) */ * from ckk_p where mod5000_id=100;--------------------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |--------------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 2000 | 6941K| | | || 1 | PX COORDINATOR | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10000 | 2000 | 6941K| Q1,00 | P->S | QC (RAND) || 3 | PX PARTITION HASH ALL | | 2000 | 6941K| Q1,00 | PCWC | || 4 | TABLE ACCESS BY LOCAL INDEX ROWID| CKK_P | 2000 | 6941K| Q1,00 | PCWP | ||* 5 | INDEX RANGE SCAN | CKK_P$MOD5000_ID | 2000 | | Q1,00 | PCWP | |--------------------------------------------------------------------------------------------------------------


Join with Indexselect /*+ parallel(4) */ * from ckk c1 join ckk2 c2 on (c1.id = c2.id) where c2.mod5000_id=5;-------------------------------------------------------------------------------------------------| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |-------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 2017 | 13M| | | || 1 | PX COORDINATOR | | | | | | || 2 | PX SEND QC (RANDOM) | :TQ10000 | | | Q1,00 | P->S | QC (RAND) || 3 | NESTED LOOPS | | | | Q1,00 | PCWP | || 4 | NESTED LOOPS | | 2017 | 13M| Q1,00 | PCWP | || 5 | PX BLOCK ITERATOR | | | | Q1,00 | PCWC | ||* 6 | TABLE ACCESS STORAGE FULL| CKK2 | 2000 | 6941K| Q1,00 | PCWP | ||* 7 | INDEX RANGE SCAN | CKK$ID | 1 | | Q1,00 | PCWP | || 8 | TABLE ACCESS BY INDEX ROWID| CKK | 1 | 3554 | Q1,00 | PCWP | |-------------------------------------------------------------------------------------------------


Indexes summary•Parallel access on direct value queries only on partitioned tables and on local indexes•Parallel access on global or non-partitioned only from joins (multiple different values)


Final Thoughts•Exadata allows Parallel Query to make sense

•High Bandwidth everywhere•Very efficient table scans

•Optimiser sometimes picks up wrong access plan•There are deficiencies in SORT operations – balancing the workloads•There are deficiencies in handling disk write operations in PQ Slaves•Can be extremely fast under the right circumstances

© 2009/2010 Pythian - Presentation for ABC Company7

9

The End

Thank You

Questions?I blog at

http://www.pythian.com/news/author/kutrovsky/

christo kutrovsky - maximize data warehouse performance with parallel queries

Software