a method for online reorganization of a database
DESCRIPTION
A Method for Online Reorganization of a Database. Gary Sockut (joint work with Thomas Beavin & Chung-Chia Chang) Work performed at IBM Silicon Valley Laboratory, San Jose, CA. Introduction: reorganization. Reorganization : Δ way arranged (logical / physical). - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/1.jpg)
Online REORG DB2 1GHS 4/21/23
A Method for Online Reorganization of a Database
Gary Sockut
(joint work with Thomas Beavin & Chung-Chia Chang)
Work performed at
IBM Silicon Valley Laboratory, San Jose, CA
![Page 2: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/2.jpg)
Online REORG DB2 2GHS 4/21/23
Introduction: reorganization
Reorganization: Δ way arranged (logical / physical).
Any DBMS & database might need:
• Δ logical database definition:generalize 1-many -► many-many, split column.
• Δ physical database definition:construct index, split partition.
• Restore physical arrangement of instances without Δ definition:* Compact space.* Collect garbage.* Restore clustering (store near each other; criteria). ▼ I/O. Writing degrades; reorganization restores. This method.
![Page 3: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/3.jpg)
Online REORG DB2 3GHS 4/21/23
highly available (24 x 7) (web commerce, armed forces)
very large
Introduction: offline vs. online reorganization
Reorganize online (concurrently withusage or incrementally within users’transactions). Importance ▲.
TIME
USAGE BATCH WINDOW (offline OK) USAGE
REORGANIZATION
Traditional: reorganize offline:
2 categories:unacceptable
![Page 4: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/4.jpg)
Online REORG DB2 4GHS 4/21/23
Introduction: online reorganization
Method for restoring clustering online.IBM’s Database 2 (DB2) relational DBMS.
Rest of presentation:
• Storage structures
• Overview (unload, reload, process log); problem; solution
• Steps of reorganization & more details
• Comparison: previous research
![Page 5: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/5.jpg)
Online REORG DB2 5GHS 4/21/23
ID map
Storage structures
• Not unique key. File pages, indexes, entries in loguse record identifier (RID) to identify records:– Page # & # within page. ID map.– Δ only during reorganization.
• Header of page: position in log (log record sequence number (LRSN)) current when most recently written.
LRSN data data * *
data record data recordfile page:
• Rows of tables in data records in file pages:
![Page 6: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/6.jpg)
Online REORG DB2 6GHS 4/21/23
ID map
ID map
Storage structures
• Variable-length: grow: regular -► pointer & overflow:
LRSN data RID * *
regulardata record
pointerdata recordfile page:
LRSN data data * *
regulardata record
overflowdata recordfile page:
bad!
![Page 7: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/7.jpg)
Online REORG DB2 7GHS 4/21/23
Storage structures: writing
Users write rows -► DBMS writes data records (& log entries).Usually, Insert / Update / Delete row -► 1 regular data record.Exceptions on Update:
• Has regular & new data too large for regular's page -►I overflow & U regular to pointer.
• Has overflow & new data fits on overflow's page -►U overflow to overflow.
• Has overflow & new data too large for overflow's page -►* Now room on pointer's page -► U pointer to regular & D overflow.* Still no room on pointer's page -► I overflow, D overflow, & U pointer to pointer.
Exception on Delete: has overflow -► D pointer & overflow.
Backout (undo if failure) reverses.
![Page 8: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/8.jpg)
Online REORG DB2 8GHS 4/21/23
Storage structures: reorganization
• Database administrator invokes reorganization:* Restores clustering (clustering key: not necessarily unique).* Removes overflow.* Distributes free space. Improves performance.
• Area being reorganized: set of 1 or more tablesor set of 1 or more partitions of 1 table (parameter).
• Offline:
1. Read-only (R/O): unload data, sort.
2. No access: reload.
• Online: Read/write (R/W): most. --►
![Page 9: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/9.jpg)
Online REORG DB2 9GHS 4/21/23
Overview of method for online reorganization
An inspiration:Fuzzy dumping: unload data (backup)while letting users write:A. Dumping: (1) record current LRSN for
log; (2) unload.
B. Recovery: (1) reload; (2) bring up to date:apply log entries (start: recorded LRSN).Ignore log entry whose LRSN ≤LRSN of indicated page.
data-base
back-up
log
data-base
back-up
1
2
-► Fuzzy reorganization.
![Page 10: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/10.jpg)
Online REORG DB2 10GHS 4/21/23
Overview of method for online reorganization
A) Record current LRSN; unload, sort, reload (reorganize). (R/W):
users’data
old copyof area
new copyof area
log
READ
WRITE
UNLOAD,SORT, RELOAD
users’data
old copyof area
new copyof area
log
READ
WRITE
B) Process log (read; apply). (n iterations R/W; 1 R/O):
PROCESS
users’data
new copyof area
log
READ
WRITE
C) Switch users' accessing to new. (brief offline; R/W):
name 0name 1
name 0 name 1
name 0 name 1
![Page 11: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/11.jpg)
Online REORG DB2 11GHS 4/21/23
Problem & solution
Problem: reorganization Δ RIDs:
• Log entries: old RIDs.
• Apply: identify data record in new copy (new RID).
Solution:
• Temporary table maps old & new RIDs, stores LRSNs.
• Translate log entries before applying.
Novelty: interaction: maintain table & process log.
![Page 12: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/12.jpg)
Online REORG DB2 12GHS 4/21/23
Main steps of reorganization (more detail)
1. Record current LRSN for log. 2.
3.
4. 5. 6. 7.
8. 9.
--►Delete old.Start R/W new.
Switch future accesses: exchange names of files(data, indexes). Users: no access.
Quiesce all access.Process log. Append changed pages to backup. R/O.
Quiesce writing (block; wait until finished). Record LRSN.
Process log: iterate:* Read subset of log between recorded LRSNs (users' writing): translate; apply to new. R/W.* Record current LRSN. Iterate or next step (criteria).* Last iteration: append changed pages to backup.
Unload data, sort, reload into new (includes indexes),create backup copy. Users R/W old.Table maps: old & new RIDs. Record current LRSN.
![Page 13: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/13.jpg)
Online REORG DB2 13GHS 4/21/23
oldRID
newRID
Unloading, sorting, & reloading (R/W)
2. Sort by clustering key.
old copyof area
new copyof area
1. UNLOAD
2. SORT
3. RELOAD
1. ADD ENTRY 3. ADD ENTRY
mappingtable
sort file
1. Unload: scan sequentially:* Regular or overflow: unload data, old RID, & LRSN of page.* Pointer: add entry to mapping table.
3. Reload: also add entry to mapping table.
![Page 14: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/14.jpg)
Online REORG DB2 14GHS 4/21/23
Processing of log (n R/W & 1 R/O) 5 phases
5. APPLY (insertion: actual)
pointers. sort: LRSN copy of log (buffer) log1. COPY
2. SORT BY OLD RID (speed access to mapping table)
pointers. sort: old RID copy of log (buffer)
3. TRANSLATE RIDS (insertion: estimate, for sorting)
pointers. sort: old RID copy of log (buffer)mappingtable
4. SORT BY NEW RID (speed access to new copy of area)
pointers. sort: new RID copy of log (buffer)
pointers. sort: new RID copy of log (buffer)
mapping table
new copyof area
3, 5, control of iterations --►
![Page 15: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/15.jpg)
Online REORG DB2 15GHS 4/21/23
Phase 3: translation of RIDs
DBMS's log application (recovery) ignores log entry if LRSN ≤page’s. Translation ignores log entry if LRSN ≤ mapping table’s.Insertion:
• R or O: estimate new RID; store in log entry (buffer);insert entry in mapping table (old, estimated new).
• P: delete log entry; insert entry in mapping table (old, no new).
Update:• R to R or O to O: store new RID in log entry.• P to P: delete log entry.• P to R: ~I: estimate new RID. Log entry: store estimated new RID; U -► I. Mapping table: store estimated new RID.• R to P: ~D: log entry: store new RID; U -► D.
Deletion:• R or O: store new RID in log entry; delete mapping table’s entry.• P: delete log entry; delete mapping table’s entry.
![Page 16: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/16.jpg)
Online REORG DB2 16GHS 4/21/23
Phase 5: application
Scan set of pointers to log entries (sorted by new RID).For each RID value:1. Find all pointers (contiguous).2.
3. Apply sequentially:I:* Insert in new copy; obtain actual new RID.* In mapping table & in log entries for current RID, estimated new RID -► actual.U or D: treat like DBMS's handling of user's.
≥ 1 D log entry -► delete certain log entries:* 1st entry is I -► delete last D & all preceding.* 1st entry is D or U -► keep last D; delete all preceding.Omit log entries for which no entries in mapping table;omit I U U D.
![Page 17: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/17.jpg)
Online REORG DB2 17GHS 4/21/23
HILO
Control of iterations of log processing
Iterate (allowing R/W) until:1. Estimated time for next ≤ parameter for maximum
R/O -► next is last (R/O). Parameter: trade-off. --►
2. Estimated completion for next > deadline(parameter) -► cancel reorganization.
3. Amount of log for next not sufficiently < current(not catching up) -► send message to operator:After delay (parameter), DBMS will continue,quiesce writing, or cancel (parameter).During delay, database administrator canΔ parameters, adjust priorities,quiesce writing, cancel, or let DBMS take action.
Display status.
iteration
time
![Page 18: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/18.jpg)
Online REORG DB2 18GHS 4/21/23
Scheduling of online reorganization
1. High tolerance of delay -► R/O & offline tolerable.2. Low rate of writing -► easy to catch up to log.3. No long-running transactions -► quick quiescing.
Trade-off.
![Page 19: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/19.jpg)
Online REORG DB2 19GHS 4/21/23
Comparison with previous research
Calculation of clustering: not novel;Sort by clustering key (index); assign to pages.
Compare mapping tables, fuzzy reorganization:
1) Mapping tables:• Omiecinski et al.: reorganization in place;
map RIDs to translate entries in leaves of indexes.• Wiener & Naughton: loading data into object database;
map surrogate object identifiers (source file) into object IDs.
2) Several authors mention fuzzy reorganization (no detail).Unique identifier does not Δ. No mapping table.Many customers dislike requirement.Fuzzy reorganization when RIDs Δ --►
![Page 20: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/20.jpg)
Online REORG DB2 20GHS 4/21/23
Comparison with previous research
O'Toole et al.: fuzzy garbage collectionfor persistent data. Forwarding field: In database context (not O'Toole), mapping table’s advantages:1. Reorganization copies, user deletes, DBMS reuses space
(not O'Toole). Mapping table safe; forwarding field gone.2. Mapping table entry < data record. Fewer pages.
Less I/O to read/write forwarding information.3. Less locking for old (unload old, reload/map, process log):
Mapping table: (R, no, no). Forwarding field: (R, W, R/W?).4. Avoid extra space (permanent) for forwarding field
(O'Toole: space already existed).
Forwarding field’s advantages:1. One field (already existed: O'Toole), not two (temporary).2. Seems simpler.
old copy new copy
data RID data
![Page 21: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/21.jpg)
Online REORG DB2 21GHS 4/21/23
Possible (but rejected) alternatives
1. Reorganize only index online:support, but not enough.
2.
3.
record’sposition
record’sposition
Δ▬►user’s
positionin scan
reorganization
Reorganize partition (index) (support):fine-grained; reorganize offline:* Correct other indexes.* Slow routing, increase space for partition descriptors.* Less uniform growth/shrinkage -► increase total free space: • coarse: | high growth shrinkage low growth | • fine: | high growth | shrinkage | low growth |
Reorganize online in place (notby copying): inaccuracy. --►Complexity and/or slowness,or low-concurrency locking.
![Page 22: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/22.jpg)
Online REORG DB2 22GHS 4/21/23
Publications
Web site: Google GARY SOCKUT(http://alum.mit.edu/www/ghs); click “publications”: -► pdf.
1. This work: G. H. Sockut, T. A. Beavin, & C.-C. Chang, “A Method for On-line Reorganization of a Database,” IBM Systems Journal, Vol. 36, No. 3, 1997, pp. 411-436; erratum in Vol. 37, No. 1, 1998, p. 152. Web site: slides.
2. Survey on online reorganization (not just IBM, not just clustering, not just fuzzy): G. H. Sockut & B. R. Iyer, “Online Reorganization of Databases,” Computing Surveys, Vol. 41, No. 3, ACM, Article 14, July 2009, 136 pages. Web site: table of contents.
![Page 23: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/23.jpg)
Online REORG DB2 23GHS 4/21/23
Reorganize online:
• Copy data from old to new in reorganized form; R/W.Table maps between old & new RIDs.
• Apply log to new; map.
• Switch users' accessing to new.
Summary
![Page 24: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/24.jpg)
Online REORG DB2 24GHS 4/21/23
Appendices
![Page 25: A Method for Online Reorganization of a Database](https://reader036.vdocuments.net/reader036/viewer/2022070413/56814c4b550346895db954c2/html5/thumbnails/25.jpg)
Online REORG DB2 25GHS 4/21/23
Appendix: effect of fine-grained partitioning
p
op
ula
tio
n d
ens
ity
(pe
op
le /
sq
ua
re m
ile)
urban area with many high-rise apartment buildings (Manhattan Community District 8: ~ Upper East Side)
uninhabited area
NewJersey
Alaska
coarse (state)fine (square mile)granularity of partitioning
Population densities in the area covered by the 50 states:
finer
0
~1 (census)
~100K(demographia
.com)
~1K (census)