![Page 1: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/1.jpg)
Reducing the Storage Overhead of Main-Memory OLTP Databases with
Hybrid IndexesHuanchen Zhang
![Page 2: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/2.jpg)
You are running out of memory
2
![Page 3: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/3.jpg)
You are running out of memory
2
![Page 4: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/4.jpg)
Buy more ?
You are running out of memory
2
![Page 5: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/5.jpg)
0 2M 4M 6M 8M 10M
20K
60K
Transactions Executed
Thro
ughp
utTPC-C on -Store
3
Mem
ory
(GB)
Disk tuplesIn-memory tuplesIndexes
4
8
0
Memory Limit = 5GB
![Page 6: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/6.jpg)
4
![Page 7: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/7.jpg)
The better way:Use memory more efficiently
5
![Page 8: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/8.jpg)
Indexes are LARGEBenchmark % space for index
TPC-C
Voter
Articles
58%55%34%
Hybrid Index
34%41%18%
6
![Page 9: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/9.jpg)
Our Contributions The hybrid index architecture
The Dual-Stage Transformation
Applied to 4 index structures- B+tree- Masstree
7
- Skip List- Adaptive Radix Tree (ART)
Performance Space30 – 70%
![Page 10: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/10.jpg)
0 2M 4M 6M 8M 10M
20K
60K
Transactions Executed
Thro
ughp
ut (t
xn/s
)Did we solve this problem?
TPC-
C on
-Sto
re
Stay tuned
8
![Page 11: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/11.jpg)
How do hybrid indexes achievememory savings ?
9
Static
![Page 12: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/12.jpg)
dynamic stage static stage
Hybrid Index: a dual-stage architecture
10
![Page 13: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/13.jpg)
dynamic stage static stage
write merge
Inserts are batched in the dynamic stage
11
![Page 14: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/14.jpg)
dynamic stage static stage
Reads search the stages in order
12
![Page 15: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/15.jpg)
dynamic stage static stage
read
A Bloom filter improves read performance
13
![Page 16: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/16.jpg)
dynamic stage static stage
read
write merge
Memory-efficient
Skew-aware
14
~~~~~~~~
![Page 17: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/17.jpg)
dynamic stage static stage
merge
The Dual-Stage Transformation
15
![Page 18: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/18.jpg)
Compaction
Reduction
Compression
The Dynamic-to-Static Rules
16
![Page 19: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/19.jpg)
2 4
4
1 2
a b
6 8 10
3
c
4
d
5 5
e f
5
g
6
h
7
i
8
j
9
k
10
l
11
m
12
n
Compaction: minimize # of memory blocks
17
![Page 20: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/20.jpg)
1 2 3
a b c
4 5 6
d h
7 8 9
i j k
10 11 12
l m n
e f g
3 6 9
17
Compaction: minimize # of memory blocks
![Page 21: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/21.jpg)
1 2 3
a b c
4 5 6
d h
7 8 9
i j k
10 11 12
l m n
e f g
3 6 9
Reduction: minimize structural overhead
18
![Page 22: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/22.jpg)
2 4
4
1 2
a b
6 8 10
3 4
c d
5 5
e f
5 6
g h
7 8
i j
9 10
k l
11 12
m n
1 2 3
a b c
4 5 6
d h
7 8 9
i j k
10 11 12
l m n
e f g
3 6 9
18
Reduction: minimize structural overhead
![Page 23: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/23.jpg)
dynamic stage static stage
merge
The Dual-Stage Transformation
19
![Page 24: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/24.jpg)
dynamic stage static stage
merge
The Dual-Stage Transformation
19
MergeQuestions:1. Partial?2. When?3. Blocking?
??
?
![Page 25: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/25.jpg)
0 2M 4M 6M 8M 10M
20K
60K
Transactions Executed
Thro
ughp
ut (t
xn/s
)Did we solve this problem?
TPC-
C on
-Sto
re
B+tree
20
![Page 26: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/26.jpg)
0 2M 4M 6M 8M 10M
20K
60K
Transactions Executed
Thro
ughp
ut (t
xn/s
)
60K
20K
Yes, we improved the DBMS’s capacity!TP
C-C
on-S
tore
B+tree
Hybrid
20
![Page 27: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/27.jpg)
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
![Page 28: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/28.jpg)
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
![Page 29: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/29.jpg)
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
![Page 30: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/30.jpg)
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
![Page 31: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/31.jpg)
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
![Page 32: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/32.jpg)
Transactions Executed
Thro
ughp
ut (t
xn/s
)
20K
60K
20K
60K
Mem
ory
(GB)
0 2M 4M 6M 8M 10M
TPC
-C o
n-S
tore
4
4
8
8 B+tree
Hybrid
B+tree
Hybrid
Disk tuplesIn-memory tuplesIndexes
21
TakeAway:Largerworkingsetinmemory
Higherthroughput
Memorysavedbyindexes
![Page 33: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/33.jpg)
This is just the BEGINNING
22
![Page 34: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/34.jpg)
Conclusions
The hybrid index architecture
The Dual-Stage Transformation
Applied to 4 index structures
GENERAL
PRACTICAL
USEFUL
23
- B+tree- Masstree
- Skip List- Adaptive Radix Tree (ART)
![Page 35: Reducing the Storage Overhead of Main-Memory OLTP ... · Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang. You are running out of memory 2. You are running out of memory](https://reader034.vdocuments.net/reader034/viewer/2022050410/5f86eab82c8a421e8656e047/html5/thumbnails/35.jpg)
1-844-88-CMUDB
Toll-Free Hotline: