![Page 1: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/1.jpg)
SSD Caching:Device-Mapper- andHardware-basedsolutions comparedWerner Fischer & Georg SchönbergerTechnology Specialists Thomas-Krenn.AG
18. LinuxTag / Berlin / Germany24th May 2011
![Page 2: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/2.jpg)
slide 2/45
Agenda
• Introduction
• Technologies compared
• A few details
• Performance results
• Test strategy
• Lessons learned
![Page 3: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/3.jpg)
slide 3/45
Introduction
Do things the right way!Source: xkcd.com
![Page 4: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/4.jpg)
slide 4/45
who is
Server & accessories"Made in Germany"
Introduction
based in Freyung,Bavaria
serving all over Europe
![Page 5: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/5.jpg)
slide 5/45
Should I use SSD caching?
![Page 6: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/6.jpg)
slide 6/45
Well, Yes and No...
if(dataSet == known && ramAvailable != enough && appAccess == analyzed && perfTests == available) CheckForCacheTechnologies();else NeedMoreInfo();
![Page 7: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/7.jpg)
slide 7/45
Establish some wording...
![Page 8: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/8.jpg)
slide 8/45
Caching Basics
![Page 9: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/9.jpg)
slide 9/45
Some main features compared...
![Page 10: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/10.jpg)
slide 10/45
SSD Caching – Compared 1 of 6
WB WT WA/read-only
FlashCache
CacheCade 1
MaxCache 2
1 Including ForcedWB
! Attention: No redundant cache with multiple SSDs possible
2Including InstantWB
!
![Page 11: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/11.jpg)
slide 11/45
SSD Caching – Comparison 2 of 6
• Maybe your disk RAID is faster for sequential I/O
1Configurable via sysctl (threshold)
2No further details known
3Not configurable, always skipped
Skip sequential I/OFlashCache 1
CacheCade 2
MaxCache 3
![Page 12: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/12.jpg)
slide 12/45
SSD Caching – Comparison 3 of 6
• WB: clean dirty pages from SSD to HDD
1Based on threshold and idleness
– Dynamic via sysctl2No further details known
3Background flush with dynamic rate
– FlushAndFetchRate
– dirtyPageThreshold
Lazy CleaningFlashCache 1
CacheCade 2
MaxCache 3
![Page 13: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/13.jpg)
slide 13/45
SSD Caching – Comparison 4 of 6
Cache persistenceFlashCache 1
CacheCade 2
MaxCache 3
!
1Only WB Cache is persistent– https://groups.google.com/forum/#!topic/flashcache-dev/xRW64FOxgWw
– People are discussing to add persistence for WT
1On node crash, only Dirty Blocks are in cache– https://groups.google.com/forum/#!topic/flashcache-dev/LqjKZcGL1eU
2Persistent for all cache modes
3Dirty shutdown: WT – discarded, WB – LV failed
![Page 14: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/14.jpg)
slide 14/45
SSD Caching – Comparison 5 of 6
1No switch from WB to WT if SSD redundant2Patch to make WB/WT errors transparent
– https://groups.google.com/forum/#!topic/flashcache-dev/6XEhY87uv1g
3Details on slide 264With write cache LV fails5No further information available
Error strategy
WB WT
FlashCache 1 Currently - Error2
CacheCade 3
MaxCache 4 5
![Page 15: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/15.jpg)
slide 15/45
SSD Caching – Comparison 6 of 6
Hot spot detectionFlashCache
CacheCade
MaxCache
![Page 16: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/16.jpg)
FlashCache
CacheCade
MaxCache
![Page 17: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/17.jpg)
slide 17/45
Take a closer look...
![Page 18: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/18.jpg)
slide 18/45
FlashCacheNot all about Facebook is bad...
Visit https://github.com/facebook/flashcache/
Not an official logo.Source: deviantart.com
![Page 19: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/19.jpg)
slide 19/45
FlashCache – Introduction
• Open Source :D
– Get code from Github
• Using Linux Device Mapper (cf. LVM)
– Visit our I/O stack diagram• http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html
– Based on dm-cache
• Easy to deploy and use
– Compile
– Load one kernel module
![Page 20: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/20.jpg)
![Page 21: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/21.jpg)
slide 21/45
FlashCache – Highlights
• FIFO/LRU replacement policy – online changeable
• Define process blacklist/whitelist
• Cleaning of dirty blocks configurable
– Threshold
– Idleness
• Error injection flags via sysctl
– E.g. READCACHE_ERROR
• Detailed Cache statistics
– dmsetup table
![Page 22: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/22.jpg)
slide 22/45
FlashCache – Statistics
fc-root: 0 3907029168 flashcache conf: ssd dev (/dev/sdd), disk dev (/dev/sdc) cache mode(WRITE_BACK) capacity(32638M), associativity(512), data block size(4K) metadata block size(4096b) skip sequential thresh(0K) total blocks(8355328), cached blocks(1048668), cache percent(12) dirty blocks(1048662), dirty percent(12) nr_queued(0)Size Hist: 1024:1 4096:9765813512:0 1024:1 1536:0 2048:0 2560:0 3072:0 3584:0 4096:9765813 4608:0 5120:0 5632:0 6144:0 6656:0 7168:0 7680:0 8192:0 8704:0 9216:0 9728:0 10240:0 10752:0 11264:0 11776:0 12288:0 12800:0 13312:0 13824:0 14336:0 14848:0 15360:0 15872:0 16384:0reads=1048896 writes=8716916read_hits=1048580 read_hit_percent=99 write_hits=9 write_hit_percent=0 dirty_write_hits=0 dirty_write_hit_percent=0 replacement=0 write_replacement=0 write_invalidates=80 read_invalidates=0 pending_enqueues=0 pending_inval=0 metadata_dirties=1048662 metadata_cleans=0 metadata_batch=1042405 metadata_ssd_writes=6257 cleanings=0 fallow_cleanings=0 no_room=0 front_merge=0 back_merge=0 disk_reads=316 disk_writes=7668261 ssd_reads=1048580 ssd_writes=1055014 uncached_reads=221 uncached_writes=7668261 uncached_IO_requeue=0 uncached_sequential_reads=0 uncached_sequential_writes=0 pid_adds=0 pid_dels=0 pid_drops=0 pid_expiry=0
![Page 23: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/23.jpg)
slide 23/45
FlashCache – Considerations
• Make your cache redundant (WB)
– What if SSD fails?
– Cf. https://groups.google.com/forum/#!topic/flashcache-dev/6XEhY87uv1g
• With WB partial writes are possible
– There are ideas to fix this (cf. flashcache-doc.txt)
• Only WB cache is persistent
– Across reboots and cache removals
– No metadata on SSD for WT/WA• Read
https://groups.google.com/forum/#!topic/flashcache-dev/xRW64FOxgWw
![Page 24: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/24.jpg)
slide 24/45
LSI MegaRAID CacheCade
![Page 25: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/25.jpg)
slide 25/45
CacheCade – Introduction
• Hardware-based solution
• Caching algorithm in controller
• Redundant SSD caches possible
– Avoid data loss with WB
![Page 26: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/26.jpg)
slide 26/45
CacheCade – Error handling
WB WT FWB
Red. Change to WT
Read SSD WB
Non. Red.
No VD access
Read HDD No VD access
• Except WB and non redundant SSD Cache
– Application doesn't notice SSD failure
• Controller reports over Storage Manager
![Page 27: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/27.jpg)
slide 27/45
CacheCade – Considerations
• Our tests showed that...
– Assign → initialization
– Fio laying out files• Not only on SSD (cf. FlashCache)
• Background syncs– Test Read values not correct
– Fio with 'create_only'• 100 MB/s read
• Data not completely in cache
• → read a 2nd time → 215MB/s
![Page 28: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/28.jpg)
slide 28/45
Adaptec MaxCache
![Page 29: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/29.jpg)
slide 29/45
MaxCache – Features
• Manual
– http://download.adaptec.com/pdfs/user_guides/cli_v7_30_18837_users_guide.pdf
• WriteCache
– Should not be used until now
– Redundant cache cannot be created
• Sequential I/O is not cached
– SAS drives may be faster
– Performance stays at 110 MB/s with streaming I/O
• Hot spot detection logic
![Page 30: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/30.jpg)
At the end its all down to performance...
![Page 31: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/31.jpg)
Prepare to get this...
![Page 32: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/32.jpg)
People might promise you...
![Page 33: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/33.jpg)
slide 33/45
Test your own specific application.If you want with a little help from us :)
![Page 34: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/34.jpg)
slide 34/45
Comparison – Bandwidth read
![Page 35: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/35.jpg)
slide 35/45
Comparison – Bandwidth write
![Page 36: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/36.jpg)
slide 36/45
Comparison – Bandwidth randread
![Page 37: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/37.jpg)
slide 37/45
Comparison – Bandwidth randwrite
![Page 38: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/38.jpg)
slide 38/45
Still to analyze...
![Page 39: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/39.jpg)
slide 39/45
FlashCache
?
![Page 40: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/40.jpg)
slide 40/45
CacheCade
?
![Page 41: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/41.jpg)
slide 41/45
Summary – avoiding errors while testing...
![Page 42: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/42.jpg)
slide 42/45
SSDs – A well known error
![Page 43: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/43.jpg)
slide 43/45
Testing caveats
![Page 44: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/44.jpg)
slide 44/45
Lessons Learned
• Test your own application
– Get a comfortable testing tool• I prefer Fio (git://git.kernel.dk/fio.git)
– Know how and what to test
• Check your test results twice
– Visualize your results
– Test numbers against the real world
• For a first attempt
– Test with FlashCache
Establish aBaseline
RAM
Knowapplication
SSD aloneto small
HDD isbottleneck
Testcaching
WB WT
Yes
No
No
![Page 45: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/45.jpg)
slide 45/45
Questions?Do not hesitate to ask us
(you can do it per mail also)
You want some test protocols?You need a special chart?
![Page 46: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/46.jpg)
Thanks for your timeVisit our booth -Open Source Sponsorshipand Low Energy Server
![Page 47: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/47.jpg)
slide 47/45
References
• http://greenarrow.deviantart.com/art/The-Flash-Wally-West-35885287
• https://upload.wikimedia.org/wikipedia/commons/6/62/2006-07-26AbendhimmelRemstal19.jpg
• http://openclipart.org/detail/15238/tortue1-by-roym
• http://openclipart.org/detail/34657/architetto---struzzo-by-anonymous
• http://openclipart.org/detail/11118/check-sign-and-cross-sign-by-jetxee
• http://openclipart.org/detail/124837/led-lamp-by-eggib
• http://b--art.deviantart.com/art/Riddler-62371451
![Page 48: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/48.jpg)
slide 48/45
Backup Slides
![Page 49: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/49.jpg)
slide 49/45
Test system
• SSDs
– Intel Series 320 160GB
– Via HPA reduced to 32GB
• RAID Controller
– LSI MegaRAID SAS 9260-4i
– Adaptec 6805Q
• Software
– Fio 2.0.7
– Ubuntu 12.04• Updates from Release Day
![Page 50: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/50.jpg)
slide 50/45
Flashcache – Test Script
• flashcache_create
– Create a WB caching device
• cache_all=0
– Don't cache ext4 initialization
• mkfs.ext4 -q -E lazy_itable_init=0,lazy_journal_init=0 /dev/mapper/fc-root
• mount /dev/mapper/fc-root
• cache_all=1
• Call fio
• umount /dev/mapper/fc-root
• dmsetup remove
• flashcache_destroy /dev/sdd
![Page 51: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/51.jpg)
slide 51/45
Flashcache – Workflow
![Page 52: SSD Caching: Device-Mapper- and Hardware-based solutions compared](https://reader033.vdocuments.net/reader033/viewer/2022051323/5487ca0fb47959e20c8b55c5/html5/thumbnails/52.jpg)
slide 52/45
Flashcache – Commands
• Just a few examples
– flashcache_create -v -p back fc-root /dev/sdd /dev/sdc
– sysctl -w dev.flashcache.sdd+sdc.cache_all=0
– dmsetup table
– flashcache_destroy /dev/sdd