index rebuild performance hopefully youll never need it. wei qiu principle engineer progress...
Post on 31-Mar-2015
223 Views
Preview:
TRANSCRIPT
Index Rebuild Performance
Hopefully you’ll never need it.
Wei QiuPrinciple EngineerProgress Software Inc
Agenda
1 Overview
Processing Details
Tuning Suggestions
Some Statistics
2
3
4
Summary5
Phases of Index Rebuild (“non-recoverable”)
Index Scan
Data Scan/Key Build
Sort-Merge
Index KeyInsertion
• Scan index data area start to finish
• I/O Bound with little CPU activity
• Eliminated with area truncate
• Scan table data area start to finish (area at a time)
• Read records, build keys, insert to temp sort buffer
• Sort full temp file buffer blocks (write if > -TF)
• I/O Bound with CPU Activity
• Sort-merge –TF and/or temp sort file
• CPU Bound with I/O Activity
• I/O eliminated if –TF large enough
• Read sorted list in –TF or temp sort file
• Insert keys into index
• Formats new clusters; May raise HWM
• I/O Bound with little CPU Activity
Index Rebuild Parameters - Overview
sort block size (8K – 64K, note new limit)
# threads for data scan phase
merge block size ( default -TB)merge pool fraction of system memory (in %)
# threads per concurrent sort group merging
-mergethreads
# concurrent sort group merging-threadnum# merge buffers to merge each merge pass
-TM
report system usage statistics
a bit quieter than before
-rusage
-silent
-TB-
datascanthreads
-TMB
-TF
Agenda
1
Processing Details
Overview
Tuning Suggestions
Some Statistics
2
3
4
Summary5
Phases of Index Rebuild
Index Scan• Scan index data area start to
finish• I/O Bound with little CPU activity• Eliminated with area truncate
Area 9: Index scan (Type II) complete.
• Index area is scanned start to finish (single threaded)• Block at a time with cluster hops
• Index blocks are put on free chain for the index• Index Object is not deleted (to fix corrupt cluster or block
chains)• Order of operation:
• Blocks are read from disk,• Blocks are re-formatted in memory• Blocks are written to disk as –B is exhausted
• Causes I/O in other phases for block re-format• Can be eliminated with manual area truncate where possible
Phases of Index Rebuild
Data Scan/Key Build
• Scan table data area start to finish (area at a time)
• Read records, build keys, insert to temp sort buffer
• Sort full temp file buffer blocks (write if > -TF) • I/O Bound with CPU Activity
Index Scan• Scan index data area start to finish• I/O Bound with little CPU activity• Eliminated with area truncate
• Table data area is scanned start to finish (multi-threaded if –datascanthreads)• Each thread processes next block in area (with cluster hops)
• Database re-opened by each thread in R/O mode• Ensure file handle ulimits set high enough
Processing area 8 : (11463) Start 4 threads for the area. (14536)Area 8: Multi-threaded record scan (Type II) complete.
Rules for -datascanthreads
Index rebuild run “with sort” option No index being rebuilt exists in table area
being scanned No word indexes are being rebuilt for table
data being scanned Data area being scanned is type TII storage
area.
If any true:◦ use single threaded data scan for that area◦ Violation for one area does not preclude use
on another area
Data Scan/Key Build
Record b) Extract next record from data block and build index keys (sort order)Key
-TF
c) Insert key into sort block for sort group(-TB 8K thru 64K)
d) Sort/merge full sort block into merge block.(-TMB -TB thru 64K)
Sort Block
a) Thread reads next data block in data area
RM Block
e) Write merge block to –TF, overflow to temp(-TMB sized I/O)
.srt1 .srt2
DB
…
Merge Block
What about “mixed” areas?Process index blocks as in index scan phaseNo longer read only!
Sort Block
Sort Groups: -SG 8 (note 8 is minimum)
Each sort group has its own sort file Sort file location
◦ 1 & 2. Sort files in same directory(I/O contention)
◦ 4. Sort files in different location Ensure enough space
Record
Index 1SG 1
SG 2
SG 3
Index 2
Index 3
Index 9
1) -T /usr1/richb/temp/
4) <dbname>.srt 0 /usr1/richb/temp/ 0 /usr2/richb/temp/ 0 /usr3/richb/temp/
.srt1
.srt2
.srt3
2) <dbname>.srt 0 /usr1/richb/temp/
Each index assigned a particular sort group (hashed index #)
3) <dbname>.srt 10240 /usr1/richb/temp/ 0 /usr1/richb/temp/
More on Sort Groups
Sort file size is in 1K units◦ 0 indicates unlimited space
Sort file max size with –TMB 8: 16 TB◦ Increase –TMB (-TMB 64: 128 TB)
3) <dbname>.srt 10240 /usr1/richb/temp/ 0 /usr1/richb/temp/
Record
Index 1SG 1
SG 2
SG 3
Index 2
Index 3
Index 9
1) -T /usr1/richb/temp/
4) <dbname>.srt 0 /usr1/richb/temp/ 0 /usr2/richb/temp/ 0 /usr3/richb/temp/
.srt1
.srt2
.srt3
2) <dbname>.srt 0 /usr1/richb/temp/
Each index assigned a particular sort group (hashed index #)
Last one on Sort Groups, I promise
Max –SG 64 per area. What if more than 64 indexes◦ Sort groups can contain more than one index’s entries: MOD(idx-
num, -SG)◦ Data scan will add key entry to appropriate sort group◦ Sort/merge orders indexes in sort group by index number◦ Key insertion phase inserts all entries for one index followed by all
index key entries for the other within same sort group.
Record
Index 1
SG 1
SG 2
SG 3
Index 2
Index 3
Index 9 .srt1
.srt2
.srt3
Each index assigned a particular sort group (hashed index #)
Index 1 Index 9
Index 2
Index 3
Phases of Index Rebuild
Index Scan
Data Scan/Key Build
• Sort-merge –TF and/or temp sort file
• CPU Bound with I/O Activity• I/O eliminated if –TF large enough
Sort-Merge
• Scan index data area start to finish• I/O Bound with little CPU activity• Eliminated with area truncate
• Scan table data area start to finish (area at a time)
• Read records, build keys, insert to temp sort buffer
• Sort full temp file buffer blocks (write if > -TF)
• I/O Bound with CPU Activity
Sorting index group 3Spawning 4 threads for merging of group 3.Sorting index group 3 complete.
Sort-Merge Phase
Sort blocks in each sort group have been sorted and merged into a linked list of individual merge blocks◦ stored in –TF and temp files.
Merge blocks are further merged◦ –TM# at a time to form new larger “runs” of sorted merge
blocks. -TM# of these new “runs” are then merged
◦ forming even larger “runs” of sorted merge blocks. When only one very large “run” left, all the key entries in the
sort group are in sorted order.
Sorted!
Up to 7 concurrent merge threads
Up to 3 concurrent merge threads
Only 1 merge thread on last run
-threadnum vs -mergethreads-threadnum 2
-TF.srt1
-TF.srt2
-TF.srt3
Thread 1
Thread 2 Merge phase group 2
Merge phase group 1
-threadnum vs -mergethreads -threadnum 2
-TF.srt1
-TF.srt2
-TF.srt3
Thread 2
Thread 1
Merge phase group 2
Merge phase group 3
Thread 0
B-tree insertion occurs as soon as a sort group’s merge is completed.
Thread 0 begins b-tree insertionconcurrently.
-threadnum vs -mergethreads -threadnum 2 –mergethreads 3
Thread 1
Thread 2
Note: 8 actively running threads
Thread 3
Thread 4
Thread 5
Thread 6
Thread 7
Thread 8
-TF.srt1
-TF.srt2
-TF.srt3
Merge phase group 1
Merge phase group 2
Merge threads merge successive “runs” of merge blocks concurrently.
-threadnum vs -mergethreads -threadnum 2 –mergethreads 3
Thread 2Thread 6
Thread 7
Thread 8
Thread 1Thread 3
Thread 4
Thread 5
-TF.srt2
-TF.srt3
-TF.srt1
Merge phase group 3
Merge phase group 2
-threadnum vs -mergethreads -threadnum 2 –mergethreads 3
Thread 0
Thread 2Thread 6
Thread 7
Thread 8
Thread 1Thread 3
Thread 4
Thread 5
-TF.srt1
-TF.srt2
-TF.srt3
Thread 0 begins b-tree insertionconcurrently.
Best performance withlow –threadnum & high -mergethreads
Merge phase group 2
Merge phase group 3
Note: 9 actively running threads
B-tree insertion occurs as soon as a sort group’s merge is completed.
Phases of Index Rebuild
Index Scan
Data Scan/Key Build
Sort-Merge
• Scan index data area start to finish• I/O Bound with little CPU activity• Eliminated with area truncate
• Scan table data area start to finish (area at a time)
• Read records, build keys, insert to temp sort buffer
• Sort full temp file buffer blocks (write if > -TF)
• I/O Bound with CPU Activity• Sort-merge –TF and/or temp sort
file• CPU Bound with I/O Activity• I/O eliminated if –TF large enough
Index KeyInsertion
• Read sorted list in –TF or temp sort file
• Insert keys into index• Formats new clusters; May raise
HWM• I/O Bound with little CPU Activity
Index Key Insertion Phase
Key entries from sorted merge blocks are inserted into b-tree
Performed sequentially entry at a time, index at a time Leaf level insertion optimization (avoids b-tree scan) Leaf level written to disk as soon as full (since never
revisited)
Building index 11 (cust-num) of group 3 …Building of indexes in group 3 completed.Multi-threaded index sorting and building complete.
Root
Leaf Leaf Leaf
Write leaf when full
DB
Index B-tree
2085 Indexes were rebuilt. (11465)Index rebuild complete. 0 error(s) encountered.
Agenda
1
Tuning Suggestions
Overview
Some Statistics
2
3
4
Summary5
Processing Details
Assumptions for best performance
Index data is segregated from table data◦ Indexes & tables are in different storage areas◦ User data areas are TII storage areas
You have enough memory/disk space for sorting◦ If not, go home. You’re done.
You understand the impact of CPU and memory consumption◦ Process allowed to use available system
resources
Index Rebuild - Tuning
Truncate “index only” area if possible
.srt file – set up properly◦ Spread I/O across disks◦ Avoid with –TF settings (or RAM disk)
The contents of table “customer" will be deleted.
The contents of index "cust-order" will be deleted.
Are you sure you want to truncate storage area "Customer/Order area" (y/n)
proutil <db> -C truncate area “Customer/Order Area”
This could be a life changing decision…
Index Rebuild - TuningParameters
◦ -datascanthreads = 1.5 * # CPUs◦ -mergethreads * -threadnum = 1.5 * #CPUs◦ -threadnum 2 to 4◦ -B 1024◦ –TF 80
monitor physical memory paging◦ –TMB 64◦ –TB 64◦ –TM 32
could be lower with large –mergethreads (> 16)
Memory usage approximation
Data Scan overhead:
dataScanMemory = -datascanthreads * (-TMB + 3,072)
eachSortGroup = ((-TMB * -TM) + TMB) * -mergethreads
Memory allocated for all sort groups:
allSortGroups = eachSortGroup * active sort groups
-TF memory merge pool fraction:
tfMemory = MIN((physicalMemory - InuseMemory),
((physicalMemory * -TF) / 100))
Total memory used (in KB):
totalMemory = dataScanOverhead + tfMemory + allSortGroups + 1MB
A“rough” approximation: 64 GB Ram, 16 CPUs, 8 indexes
proutil <db> -C idxbuild area “Customer/Order Index Area” -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8
Data Scan overhead:dataScanMemory = 24 * (64 + 3,072) = 76,800eachSortGroup = ((64 * 32) + 64) * 8 = 16,632
A“rough” approximation: 64 GB Ram, 16 CPUs, 8 indexes
proutil <db> -C idxbuild area “Customer/Order Index Area” -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8
Data Scan overhead:dataScanMemory = 24 * (64 + 3,072) = 76,800eachSortGroup = ((64 * 32) + 64) * 8 = 16,632
Memory allocated for all sort groups:allSortGroups = eachSortGroup * 10 = 166,320
A“rough” approximation: 64 GB Ram, 16 CPUs, 8 indexes
proutil <db> -C idxbuild area “Customer/Order Index Area” -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8
Data Scan overhead:dataScanMemory = 24 * (64 + 3,072) = 76,800eachSortGroup = ((64 * 32) + 64) * 8 = 16,632
Memory allocated for all sort groups:allSortGroups = 16,632 * 10 = 166,320
-TF memory merge pool fraction: tfMemory = MIN((64GB - InuseMemory), ((64GB * 80) /
100))
Total memory used (in KB):totalMemory = 76,800+ 53,687,091 + 166,320 + 1MB = 54,909,667 K = 52.4 GB
Agenda
1
Some Statistics
Overview
Tuning Suggestions
Processing Details2
3
4
Summary5
Performance Numbers
0
20,000
40,000
60,000
80,000
100,000
120,000
Index Rebuild
10.2b06 best
10.2b06 no truncate
10.2b06 w/-TF 50
10.2b06 baseline
Elapsed Time
Cost of eachphase (in secs)
12 ½ hours 2 ½ hours
5X improvement!
Agenda
1
Summary
Overview
Tuning Suggestions
Some Statistics
2
3
4
Processing Details
5
Summary
Index Rebuild◦ Big improvements if
Your database is setup properly You provide system resources to index rebuild You use the new settings in 10.2b06
◦ One bug fix in 10.2b07◦ More efficient memory allocation coming soon!
(10.2b08)◦ Hopefully you’ll never need to index rebuild
Questions?
October 6–9, 2013 • Boston #PRGS
www.progress.com/exchange-pug
Special low rate of $495 for PUG Challenge attendees with the code PUGAM
And visit the Progress booth to learn more about the Progress App Dev Challenge!
top related