qcon london - persistent memory · 2016-04-05 · space user space libc (or other) vmem_malloc vmem...
TRANSCRIPT
![Page 1: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/1.jpg)
Maciej Maciejewski
![Page 2: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/2.jpg)
HARDWARE
2
![Page 3: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/3.jpg)
Persistent memory
3
![Page 4: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/4.jpg)
How does it differ from DRAM
4
![Page 5: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/5.jpg)
How fast is it?
5
![Page 6: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/6.jpg)
OPERATING SYSTEM
6
![Page 7: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/7.jpg)
ACPI NFIT
7
![Page 8: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/8.jpg)
E820
8
![Page 9: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/9.jpg)
What can we do with it?
9
![Page 10: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/10.jpg)
10
PersistentMemory
UserSpace
KernelSpace
Application
Load/Store
MMUMappings
NVDIMM Driver
temporary file
StandardFile API
PM-awareFile System
NVML
![Page 12: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/12.jpg)
Memory
12
![Page 13: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/13.jpg)
Memory - libvmmalloc
13
libvmem
jemalloc
Application
PM
KernelSpace
UserSpace
malloc
vmmalloc
mmap
DRAM
vmem_pool_mallocvmem_pool_create
constructor
temporary file
![Page 14: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/14.jpg)
Memory - libvmem
14
libvmem
jemalloc (3.6.0)
Application
Persistent Memory
KernelSpace
UserSpace
libc(or other)
vmem_malloc
vmem
fallocate/mmap
malloc
DRAM
mmap/sbrk
vmem_create
vmem_pool_mallocmalloc
vmem_pool_create
![Page 15: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/15.jpg)
Memory - memkind
15
![Page 16: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/16.jpg)
Block storage
16
![Page 17: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/17.jpg)
17
![Page 18: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/18.jpg)
Byte persistency
18
![Page 19: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/19.jpg)
Why persistency is an issue?
19
CPU
L1
L2
L3
iMC
NVDIMM
CPU
L1
L2
iMC
NVDIMM
CLFLUSHCLFLUSHOPTCLWB
SFENCE
PCOMMIT
SFENCE
![Page 20: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/20.jpg)
20
Flushing to Persistence
open(…);
mmap(…);
strcpy(pmem, „Hello");
msync(pmem, 6, MS_SYNC);
pmem_persist(pmem, 6);
strcpy(pmem, „Hello, World!");
pmem_persist(pmem, 14);
Crossing the 8-Byte Store
Result?
1. „\0\0\0\0\0\0\0\0\0\0...”
2. „Hello, W\0\0\0\0\0\0...”
3. „\0\0\0\0\0\0\0\0orld!\0”
4. „Hello, \0\0\0\0\0\0\0\0”
5. „Hello, World!\0”crash
![Page 21: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/21.jpg)
nvml Persistent Libraries
libpmem – Basic persistency handling
libpmemblk – Block access to pmem
libpmemlog - Log file on pmem (append-mostly)
libpmemobj - Transactional Object Store on pmem
21
![Page 22: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/22.jpg)
libpmemobj
Object API
Transactional API
22
![Page 23: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/23.jpg)
Applications
23
![Page 24: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/24.jpg)
pm_invaders
24
![Page 25: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/25.jpg)
25
![Page 26: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/26.jpg)
Redis key/value store
26
![Page 27: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/27.jpg)
31
0
5
10
15
20
25
30
32 64 128 256 512 1024 2048 4096 8192
seco
nd
s
object size
Startup time
RDB
AOF
PM
![Page 28: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/28.jpg)
32
0
1000
2000
3000
4000
5000
6000
7000
8000
32 64 128 256 512 1024 2048 4096 8192
DR
AM
all
oca
tio
ns
[MB
]
objects size
DRAM usage
AOF
RDB
PM
![Page 29: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/29.jpg)
33
0
20000
40000
60000
80000
100000
120000
36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141
op
era
tio
ns
/ se
c
% of OS memory
Running out of DRAM
No Persist
![Page 30: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/30.jpg)
34
0
20000
40000
60000
80000
100000
120000
36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141
op
era
tio
ns
/ se
c
% of OS memory
Running out of DRAM
No Persist
RDB
![Page 31: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/31.jpg)
35
0
20000
40000
60000
80000
100000
120000
36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141
op
era
tio
ns
/ se
c
% of OS memory
Running out of DRAM
No Persist
RDB
AOF
![Page 32: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/32.jpg)
36
0
20000
40000
60000
80000
100000
120000
36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141
op
era
tio
ns
/ se
c
% of OS memory
Running out of DRAM
No Persist
RDB
PM
AOF
![Page 33: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/33.jpg)
37
0
10000
20000
30000
40000
50000
60000
70000
80000
32 64 128 256 512 1024 2048 4096 8192
op
era
tio
ns
/ se
c
objects size
CPU usage
PM 1X
AOF 1X
![Page 34: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/34.jpg)
38
0
10000
20000
30000
40000
50000
60000
70000
80000
32 64 128 256 512 1024 2048 4096 8192
op
era
tio
ns
/ se
c
objects size
CPU usage
PM 1X
PM 4X
AOF 1X
AOF 4X
![Page 35: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/35.jpg)
39
0
10000
20000
30000
40000
50000
60000
70000
80000
32 64 128 256 512 1024 2048 4096 8192
op
era
tio
ns
/ se
c
objects size
CPU usage
PM 1X
PM 4X
PM 6X
PM 10X
AOF 1X
AOF 4X
AOF 10X
![Page 36: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/36.jpg)
Moving even more data
0
100
200
300
400
500
600
Time [s] Memory [MB]
GMRES Sparse Matrix solver
Original Matrix data in PM All in PM
40
![Page 37: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/37.jpg)
Modifying app engine
for (int i=0; i<NUMBER_OF_ITERATIONS; i++) {result = calculateThis(i, result);
}
41
i_pm = 0; //at first runtime of app only
...
...
for (i_pm; i_pm<NUMBER_OF_ITERATIONS; i_pm++) {TX_BEGIN(pool) {
result_pm = calculateThis(i_pm, result_pm);} TX_END
} i_pm = 0;
![Page 38: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/38.jpg)
Possibilities
Choices
Even more fun ...
42
![Page 39: QCon London - Persistent memory · 2016-04-05 · Space User Space libc (or other) vmem_malloc vmem fallocate/mmap malloc DRAM mmap/sbrk vmem_create vmem_pool_malloc malloc vmem_pool_create](https://reader034.vdocuments.net/reader034/viewer/2022042414/5f2e35111c3e764f0134740f/html5/thumbnails/39.jpg)