logging, checkpoint, recovery - cmu 15-721 › spring2016 › files › project3-loggi… ·...
TRANSCRIPT
![Page 1: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/1.jpg)
LOGGING, CHECKPOINTS, AND RECOVERY
Eccentric Loggers // Haibin Lin, Matt Perron, Abhishek Joshi // 5/6/2016
![Page 2: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/2.jpg)
OUTLINE
• Proposal Review• Current State• Benchmark Result• Test Coverage• Future Work
![Page 3: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/3.jpg)
PROBLEMS WITH PREVIOUS DESIGN
• Single log file which grows without bound• Overuse of disk space• Difficult to truncate efficiently.
• Only one front-end logger for all back-end loggers• No checkpoints• Sequential Recovery• Cannot handle concurrent txns
• Did not preserve txn order• Premature release of workers
![Page 4: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/4.jpg)
PROPOSAL REVIEW
• Correct single-threaded logging implementation • Backpressure mechanism to prevent backup of logs on
workers • Making a multi-file log for truncation after taking a
checkpoint • Single-threaded checkpoints • Correct single threaded recovery • Multi-threaded logging and recovery from log
![Page 5: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/5.jpg)
DESIGN OVERVIEW
Committing txn 100
Committedtxn 150
Persistcommitid=99
![Page 6: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/6.jpg)
RECOVERY PROCESS
PCID:100
PCID:160 PCID:158
1. Recover Checkpoint 2. Find min PCID of logs3. Recover transactions in log
between Checkpoint id and persistent commit id
4. Rebuild Indexes
Log1 Log2
Checkpoint
Pageicon Checkpoint icon
![Page 7: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/7.jpg)
BENCHMARK RESULTS
![Page 8: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/8.jpg)
EXPERIMENT SETUP
• Dual-socketXeonE5-2620• 6cores/12threads(24hyper-threads)• 3SSD's
![Page 9: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/9.jpg)
IMPACT OF LOGGING ON THROUGHPUTYCSBMicro-benchmark• 100%Insert
![Page 10: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/10.jpg)
MULTIPLE LOGGERSYCSBMicro-benchmark• 100%Insert• MultipleSSD's
![Page 11: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/11.jpg)
LOGGING THROUGHPUT
![Page 12: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/12.jpg)
LOGGING RECOVERY SCALABILITY
*Includes single-thread index recovery time
![Page 13: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/13.jpg)
LogFileSize CheckpointSize
NoCheckpoint 30GBHasCheckpoint 32MB 1GB
CHECKPOINT STORAGE SAVINGSYCSBMicro-benchmark• 100%Update• 1000seconds,1Mtuples,resultin60GBinmemory
![Page 14: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/14.jpg)
LOW RUNTIME IMPACT OF CHECKPOINTSYCSBMicro-benchmark• 100%Insert
![Page 15: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/15.jpg)
TEST COVERAGE
• Unit Tests• Scheduled tests for different ordering of worker logging
operations (adapted from CC team)• Coarse grained tests (insert/update, crash, test with ycsb)
![Page 16: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/16.jpg)
CODE QUALITY
• Correctness of single threaded logging• Backpressure mechanism (log buffers)• Recovery from both checkpoint and logging• Easy to extend• File management
![Page 17: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/17.jpg)
FUTURE WORK• Further reducing recovery time• Single threaded Checkpoint performance• SiloR-style multithreaded checkpoints and multithreaded
checkpoint recovery• Preserve tile layout information in checkpoint• Compressing log and checkpoint• Data integrity checks of logs and checkpoint• Performance investigation (as yet unobserved)• Core-pinning workers and frontend loggers
![Page 18: Logging, Checkpoint, Recovery - CMU 15-721 › spring2016 › files › project3-loggi… · LOGGING, CHECKPOINTS, AND RECOVERY Eccentric Loggers // HaibinLin, Matt Perron, Abhishek](https://reader035.vdocuments.net/reader035/viewer/2022062921/5f036ead7e708231d4092d5b/html5/thumbnails/18.jpg)
PROPOSAL REVIEW• 75% - Basic Checkpoint and Logging
• Multi-file Log• Single-thread checkpoint and log creation• Single-thread recovery from checkpoint and log
• 100% - SiloR-style Logging• Multi-thread SiloR-style logging and recovery• Single-thread checkpoint
• 125% - SiloR-style Checkpoint• Multi-thread checkpoint creation and recovery• Reduce log size by compression