index rebuild performance hopefully youll never need it. wei qiu principle engineer progress...
TRANSCRIPT
![Page 1: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/1.jpg)
Index Rebuild Performance
Hopefully you’ll never need it.
Wei QiuPrinciple EngineerProgress Software Inc
![Page 2: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/2.jpg)
Agenda
1 Overview
Processing Details
Tuning Suggestions
Some Statistics
2
3
4
Summary5
![Page 3: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/3.jpg)
Phases of Index Rebuild (“non-recoverable”)
Index Scan
Data Scan/Key Build
Sort-Merge
Index KeyInsertion
• Scan index data area start to finish
• I/O Bound with little CPU activity
• Eliminated with area truncate
• Scan table data area start to finish (area at a time)
• Read records, build keys, insert to temp sort buffer
• Sort full temp file buffer blocks (write if > -TF)
• I/O Bound with CPU Activity
• Sort-merge –TF and/or temp sort file
• CPU Bound with I/O Activity
• I/O eliminated if –TF large enough
• Read sorted list in –TF or temp sort file
• Insert keys into index
• Formats new clusters; May raise HWM
• I/O Bound with little CPU Activity
![Page 4: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/4.jpg)
Index Rebuild Parameters - Overview
sort block size (8K – 64K, note new limit)
# threads for data scan phase
merge block size ( default -TB)merge pool fraction of system memory (in %)
# threads per concurrent sort group merging
-mergethreads
# concurrent sort group merging-threadnum# merge buffers to merge each merge pass
-TM
report system usage statistics
a bit quieter than before
-rusage
-silent
-TB-
datascanthreads
-TMB
-TF
![Page 5: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/5.jpg)
Agenda
1
Processing Details
Overview
Tuning Suggestions
Some Statistics
2
3
4
Summary5
![Page 6: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/6.jpg)
Phases of Index Rebuild
Index Scan• Scan index data area start to
finish• I/O Bound with little CPU activity• Eliminated with area truncate
Area 9: Index scan (Type II) complete.
• Index area is scanned start to finish (single threaded)• Block at a time with cluster hops
• Index blocks are put on free chain for the index• Index Object is not deleted (to fix corrupt cluster or block
chains)• Order of operation:
• Blocks are read from disk,• Blocks are re-formatted in memory• Blocks are written to disk as –B is exhausted
• Causes I/O in other phases for block re-format• Can be eliminated with manual area truncate where possible
![Page 7: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/7.jpg)
Phases of Index Rebuild
Data Scan/Key Build
• Scan table data area start to finish (area at a time)
• Read records, build keys, insert to temp sort buffer
• Sort full temp file buffer blocks (write if > -TF) • I/O Bound with CPU Activity
Index Scan• Scan index data area start to finish• I/O Bound with little CPU activity• Eliminated with area truncate
• Table data area is scanned start to finish (multi-threaded if –datascanthreads)• Each thread processes next block in area (with cluster hops)
• Database re-opened by each thread in R/O mode• Ensure file handle ulimits set high enough
Processing area 8 : (11463) Start 4 threads for the area. (14536)Area 8: Multi-threaded record scan (Type II) complete.
![Page 8: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/8.jpg)
Rules for -datascanthreads
Index rebuild run “with sort” option No index being rebuilt exists in table area
being scanned No word indexes are being rebuilt for table
data being scanned Data area being scanned is type TII storage
area.
If any true:◦ use single threaded data scan for that area◦ Violation for one area does not preclude use
on another area
![Page 9: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/9.jpg)
Data Scan/Key Build
Record b) Extract next record from data block and build index keys (sort order)Key
-TF
c) Insert key into sort block for sort group(-TB 8K thru 64K)
d) Sort/merge full sort block into merge block.(-TMB -TB thru 64K)
Sort Block
a) Thread reads next data block in data area
RM Block
e) Write merge block to –TF, overflow to temp(-TMB sized I/O)
.srt1 .srt2
DB
…
Merge Block
What about “mixed” areas?Process index blocks as in index scan phaseNo longer read only!
Sort Block
![Page 10: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/10.jpg)
Sort Groups: -SG 8 (note 8 is minimum)
Each sort group has its own sort file Sort file location
◦ 1 & 2. Sort files in same directory(I/O contention)
◦ 4. Sort files in different location Ensure enough space
Record
Index 1SG 1
SG 2
SG 3
Index 2
Index 3
Index 9
1) -T /usr1/richb/temp/
4) <dbname>.srt 0 /usr1/richb/temp/ 0 /usr2/richb/temp/ 0 /usr3/richb/temp/
.srt1
.srt2
.srt3
2) <dbname>.srt 0 /usr1/richb/temp/
Each index assigned a particular sort group (hashed index #)
3) <dbname>.srt 10240 /usr1/richb/temp/ 0 /usr1/richb/temp/
![Page 11: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/11.jpg)
More on Sort Groups
Sort file size is in 1K units◦ 0 indicates unlimited space
Sort file max size with –TMB 8: 16 TB◦ Increase –TMB (-TMB 64: 128 TB)
3) <dbname>.srt 10240 /usr1/richb/temp/ 0 /usr1/richb/temp/
Record
Index 1SG 1
SG 2
SG 3
Index 2
Index 3
Index 9
1) -T /usr1/richb/temp/
4) <dbname>.srt 0 /usr1/richb/temp/ 0 /usr2/richb/temp/ 0 /usr3/richb/temp/
.srt1
.srt2
.srt3
2) <dbname>.srt 0 /usr1/richb/temp/
Each index assigned a particular sort group (hashed index #)
![Page 12: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/12.jpg)
Last one on Sort Groups, I promise
Max –SG 64 per area. What if more than 64 indexes◦ Sort groups can contain more than one index’s entries: MOD(idx-
num, -SG)◦ Data scan will add key entry to appropriate sort group◦ Sort/merge orders indexes in sort group by index number◦ Key insertion phase inserts all entries for one index followed by all
index key entries for the other within same sort group.
Record
Index 1
SG 1
SG 2
SG 3
Index 2
Index 3
Index 9 .srt1
.srt2
.srt3
Each index assigned a particular sort group (hashed index #)
Index 1 Index 9
Index 2
Index 3
![Page 13: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/13.jpg)
Phases of Index Rebuild
Index Scan
Data Scan/Key Build
• Sort-merge –TF and/or temp sort file
• CPU Bound with I/O Activity• I/O eliminated if –TF large enough
Sort-Merge
• Scan index data area start to finish• I/O Bound with little CPU activity• Eliminated with area truncate
• Scan table data area start to finish (area at a time)
• Read records, build keys, insert to temp sort buffer
• Sort full temp file buffer blocks (write if > -TF)
• I/O Bound with CPU Activity
Sorting index group 3Spawning 4 threads for merging of group 3.Sorting index group 3 complete.
![Page 14: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/14.jpg)
Sort-Merge Phase
Sort blocks in each sort group have been sorted and merged into a linked list of individual merge blocks◦ stored in –TF and temp files.
Merge blocks are further merged◦ –TM# at a time to form new larger “runs” of sorted merge
blocks. -TM# of these new “runs” are then merged
◦ forming even larger “runs” of sorted merge blocks. When only one very large “run” left, all the key entries in the
sort group are in sorted order.
Sorted!
Up to 7 concurrent merge threads
Up to 3 concurrent merge threads
Only 1 merge thread on last run
![Page 15: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/15.jpg)
-threadnum vs -mergethreads-threadnum 2
-TF.srt1
-TF.srt2
-TF.srt3
Thread 1
Thread 2 Merge phase group 2
Merge phase group 1
![Page 16: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/16.jpg)
-threadnum vs -mergethreads -threadnum 2
-TF.srt1
-TF.srt2
-TF.srt3
Thread 2
Thread 1
Merge phase group 2
Merge phase group 3
Thread 0
B-tree insertion occurs as soon as a sort group’s merge is completed.
Thread 0 begins b-tree insertionconcurrently.
![Page 17: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/17.jpg)
-threadnum vs -mergethreads -threadnum 2 –mergethreads 3
Thread 1
Thread 2
Note: 8 actively running threads
Thread 3
Thread 4
Thread 5
Thread 6
Thread 7
Thread 8
-TF.srt1
-TF.srt2
-TF.srt3
Merge phase group 1
Merge phase group 2
Merge threads merge successive “runs” of merge blocks concurrently.
![Page 18: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/18.jpg)
-threadnum vs -mergethreads -threadnum 2 –mergethreads 3
Thread 2Thread 6
Thread 7
Thread 8
Thread 1Thread 3
Thread 4
Thread 5
-TF.srt2
-TF.srt3
-TF.srt1
Merge phase group 3
Merge phase group 2
![Page 19: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/19.jpg)
-threadnum vs -mergethreads -threadnum 2 –mergethreads 3
Thread 0
Thread 2Thread 6
Thread 7
Thread 8
Thread 1Thread 3
Thread 4
Thread 5
-TF.srt1
-TF.srt2
-TF.srt3
Thread 0 begins b-tree insertionconcurrently.
Best performance withlow –threadnum & high -mergethreads
Merge phase group 2
Merge phase group 3
Note: 9 actively running threads
B-tree insertion occurs as soon as a sort group’s merge is completed.
![Page 20: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/20.jpg)
Phases of Index Rebuild
Index Scan
Data Scan/Key Build
Sort-Merge
• Scan index data area start to finish• I/O Bound with little CPU activity• Eliminated with area truncate
• Scan table data area start to finish (area at a time)
• Read records, build keys, insert to temp sort buffer
• Sort full temp file buffer blocks (write if > -TF)
• I/O Bound with CPU Activity• Sort-merge –TF and/or temp sort
file• CPU Bound with I/O Activity• I/O eliminated if –TF large enough
Index KeyInsertion
• Read sorted list in –TF or temp sort file
• Insert keys into index• Formats new clusters; May raise
HWM• I/O Bound with little CPU Activity
![Page 21: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/21.jpg)
Index Key Insertion Phase
Key entries from sorted merge blocks are inserted into b-tree
Performed sequentially entry at a time, index at a time Leaf level insertion optimization (avoids b-tree scan) Leaf level written to disk as soon as full (since never
revisited)
Building index 11 (cust-num) of group 3 …Building of indexes in group 3 completed.Multi-threaded index sorting and building complete.
Root
Leaf Leaf Leaf
Write leaf when full
DB
Index B-tree
![Page 22: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/22.jpg)
2085 Indexes were rebuilt. (11465)Index rebuild complete. 0 error(s) encountered.
![Page 23: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/23.jpg)
Agenda
1
Tuning Suggestions
Overview
Some Statistics
2
3
4
Summary5
Processing Details
![Page 24: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/24.jpg)
Assumptions for best performance
Index data is segregated from table data◦ Indexes & tables are in different storage areas◦ User data areas are TII storage areas
You have enough memory/disk space for sorting◦ If not, go home. You’re done.
You understand the impact of CPU and memory consumption◦ Process allowed to use available system
resources
![Page 25: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/25.jpg)
Index Rebuild - Tuning
Truncate “index only” area if possible
.srt file – set up properly◦ Spread I/O across disks◦ Avoid with –TF settings (or RAM disk)
The contents of table “customer" will be deleted.
The contents of index "cust-order" will be deleted.
Are you sure you want to truncate storage area "Customer/Order area" (y/n)
proutil <db> -C truncate area “Customer/Order Area”
This could be a life changing decision…
![Page 26: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/26.jpg)
Index Rebuild - TuningParameters
◦ -datascanthreads = 1.5 * # CPUs◦ -mergethreads * -threadnum = 1.5 * #CPUs◦ -threadnum 2 to 4◦ -B 1024◦ –TF 80
monitor physical memory paging◦ –TMB 64◦ –TB 64◦ –TM 32
could be lower with large –mergethreads (> 16)
![Page 27: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/27.jpg)
Memory usage approximation
Data Scan overhead:
dataScanMemory = -datascanthreads * (-TMB + 3,072)
eachSortGroup = ((-TMB * -TM) + TMB) * -mergethreads
Memory allocated for all sort groups:
allSortGroups = eachSortGroup * active sort groups
-TF memory merge pool fraction:
tfMemory = MIN((physicalMemory - InuseMemory),
((physicalMemory * -TF) / 100))
Total memory used (in KB):
totalMemory = dataScanOverhead + tfMemory + allSortGroups + 1MB
![Page 28: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/28.jpg)
A“rough” approximation: 64 GB Ram, 16 CPUs, 8 indexes
proutil <db> -C idxbuild area “Customer/Order Index Area” -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8
Data Scan overhead:dataScanMemory = 24 * (64 + 3,072) = 76,800eachSortGroup = ((64 * 32) + 64) * 8 = 16,632
![Page 29: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/29.jpg)
A“rough” approximation: 64 GB Ram, 16 CPUs, 8 indexes
proutil <db> -C idxbuild area “Customer/Order Index Area” -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8
Data Scan overhead:dataScanMemory = 24 * (64 + 3,072) = 76,800eachSortGroup = ((64 * 32) + 64) * 8 = 16,632
Memory allocated for all sort groups:allSortGroups = eachSortGroup * 10 = 166,320
![Page 30: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/30.jpg)
A“rough” approximation: 64 GB Ram, 16 CPUs, 8 indexes
proutil <db> -C idxbuild area “Customer/Order Index Area” -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8
Data Scan overhead:dataScanMemory = 24 * (64 + 3,072) = 76,800eachSortGroup = ((64 * 32) + 64) * 8 = 16,632
Memory allocated for all sort groups:allSortGroups = 16,632 * 10 = 166,320
-TF memory merge pool fraction: tfMemory = MIN((64GB - InuseMemory), ((64GB * 80) /
100))
Total memory used (in KB):totalMemory = 76,800+ 53,687,091 + 166,320 + 1MB = 54,909,667 K = 52.4 GB
![Page 31: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/31.jpg)
Agenda
1
Some Statistics
Overview
Tuning Suggestions
Processing Details2
3
4
Summary5
![Page 32: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/32.jpg)
Performance Numbers
0
20,000
40,000
60,000
80,000
100,000
120,000
Index Rebuild
10.2b06 best
10.2b06 no truncate
10.2b06 w/-TF 50
10.2b06 baseline
Elapsed Time
Cost of eachphase (in secs)
12 ½ hours 2 ½ hours
5X improvement!
![Page 33: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/33.jpg)
Agenda
1
Summary
Overview
Tuning Suggestions
Some Statistics
2
3
4
Processing Details
5
![Page 34: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/34.jpg)
Summary
Index Rebuild◦ Big improvements if
Your database is setup properly You provide system resources to index rebuild You use the new settings in 10.2b06
◦ One bug fix in 10.2b07◦ More efficient memory allocation coming soon!
(10.2b08)◦ Hopefully you’ll never need to index rebuild
![Page 35: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/35.jpg)
Questions?
![Page 36: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/36.jpg)
October 6–9, 2013 • Boston #PRGS
www.progress.com/exchange-pug
Special low rate of $495 for PUG Challenge attendees with the code PUGAM
And visit the Progress booth to learn more about the Progress App Dev Challenge!
![Page 37: Index Rebuild Performance Hopefully youll never need it. Wei Qiu Principle Engineer Progress Software Inc](https://reader036.vdocuments.net/reader036/viewer/2022062318/551a4040550346545e8b4fb5/html5/thumbnails/37.jpg)