the effects of ssd caching on the i/o performance of ... · pdf fileunclassified...
TRANSCRIPT
![Page 1: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/1.jpg)
LA-UR-14-26016
LA-UR-14-26016
Unclassified
Heidi SandovalCalifornia State
University
Matthew DwyerLynchburg College
Anthony PearsonSt. Cloud State
University
The Effects of SSD Caching on the I/O Performance of
Unified Storage Systems
Team Chartreuse
![Page 2: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/2.jpg)
LA-UR-14-26016Unclassified
Outline
● Unified Storage● SSD/Flash Caching● Testbed● Obstacles● Conclusions● Future Work
![Page 3: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/3.jpg)
LA-UR-14-26016Unclassified
Introduction
● The Lab is currently utilizing tape drives as its primary backup storage method○ Slow and expensive○ Will this continue to scale?
● This experiment tests a Unified Storage System with a layer SSD/Flash caching ○ Faster I/O performance○ Enhanced fault tolerance
![Page 4: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/4.jpg)
LA-UR-14-26016Unclassified
Goals
1. Implement a Unified Storage System2. Test the impact of SSD/Flash caching on
the I/O performance of the Unified Storage System
![Page 5: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/5.jpg)
LA-UR-14-26016Unclassified
Unified Storage
● Combination of two different storage systems that creates a single integrated storage structure using:○ Storage Area Network (SAN)○ Cloud object storage
10 GigE Switch
![Page 6: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/6.jpg)
LA-UR-14-26016Unclassified
Storage Area Network (SAN)● Block level protocols● Communicates over Fibre Channel
○ Hardware used:■ QLogic Corp. ISP2532 8Gb Fibre Channel
● Uses the Encapsulated SCSI protocol○ Allows nodes to discover SCSI target devices○ Software used:
■ targetCLI● Enforces POSIX style file environment
![Page 7: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/7.jpg)
LA-UR-14-26016Unclassified
Cloud Object Storage● Communicates over TCP/IP
○ Connected in a ring topology○ Hardware used:
■ 10GigE Ethernet Switch■ Mellanox MT26448 10GigE■ Myricom Myri-10G
● Servers within the cloud are solely used for storing and retrieving files
![Page 8: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/8.jpg)
LA-UR-14-26016Unclassified
Unified Storage Representation
SAN Storage
Bridging
![Page 9: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/9.jpg)
LA-UR-14-26016Unclassified
Benefits● Using a Unified Storage System enables us to reap the
following benefits:○ Reduced Hardware Requirements○ Uses a POSIX interface to perform I/O operations on
remote block devices○ Fast ethernet connection (10 GigE) among the
storage nodes for communication○ Fibre channel is a reliable method for transferring
data■ Often used in secure corporations
○ Implements object storage, which allows for the usage of erasure coding
![Page 10: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/10.jpg)
LA-UR-14-26016Unclassified
SSD/Flash Caching
● A method used to speed the I/O processes of local and remote block devices by caching data to faster SSD/Flash devices
● Methods Used:○ dm-cache
■ write-back enabled○ bcache
■ write-back enabled● Hardware Used:
○ Samsung Evo 1TB SSD○ OCZ PCI-E Flash 960GB
![Page 11: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/11.jpg)
LA-UR-14-26016Unclassified
dm-cache
● A device-mapper target that allows the creation of hybrid volumes--block device and SSD combination
● Does not cache data that involves sequential reads and writes (better suited for block devices)
● Requires three physical storage devices:○ Origin Device: provides slow primary storage
(usually a local or remote block device)○ Cache Device: provides a fast cache (usually a
SSD)○ Metadata Device: records blocks placement and
their dirty flags, as well as other internal data
![Page 12: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/12.jpg)
LA-UR-14-26016Unclassified
bcache
● Converts random writes into sequential writes○ First, writes data to the SSD
○ Then, buffers data from the SSD to the HDD in order
● Must be configured to obtain higher performance
○ Parameters such as “sequential_cutoff” must be disabled
![Page 13: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/13.jpg)
LA-UR-14-26016Unclassified
SSD/Flash Caching Representation
● Demonstrates the writeback caching process● Initially, writing is done only to the cache● The write to the backing store is postponed until the
cache blocks containing the data are about to be modified/replaced by new content
http://pommi.nethuis.nl/ssd-caching-using-linux-and-bcache/
![Page 14: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/14.jpg)
LA-UR-14-26016Unclassified
Benefits
● Slower and cheaper hard drive disks could provide a large amount of storage space
● Faster flash devices could provide rapid I/O speeds
● SSD/Flash caching combines both devices, so the resultant set-up has both a large amount of storage, which operates at a fast rate
![Page 15: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/15.jpg)
LA-UR-14-26016Unclassified
Testbed
Storage Server 6
Storage Server 1
Storage Server 4
Storage Server 2
Storage Server 3
Storage Server 5
Supervisor/Login Server
10 GigE Switch
Connector Server
Fibre Channel10 GigE
SCSI Initiator Server
Cloud
SAN
![Page 16: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/16.jpg)
LA-UR-14-26016Unclassified
A Closer Look at the Connector
Connector Server
Let’s take a closer look at the connector node...
![Page 17: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/17.jpg)
LA-UR-14-26016Unclassified
A Closer Look at the Connector cont.
● Acts as a bridge between both storage systems● It uses specialized software, which allows it to
read and write to the storage servers● Even closer…
○ The connector uses a FUSE mount to access the storage servers
○ On that FUSE mount, sparse files were created, so that SCSI targets could be formed using them
○ Later, those SCSI targets were detected by the Initiator node
![Page 18: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/18.jpg)
LA-UR-14-26016Unclassified
Benchmarking
● I/O performance tested with and without caching
● Testing methods included:○ dd: used to run sequential writes ○ iozone: can test a variety of I/O operations
both random and sequential● File sizes ranged from 30-50GB./writeTest.sh -b $blockSize -u $uniqueNum -t $testType -o
$basePath --size $size -d $directory --log $basePath/logs --
sleep 300 -y
![Page 19: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/19.jpg)
LA-UR-14-26016Unclassified
Obstacles
● Linux distribution conflicts○ Distributions that worked with the Fibre
Channel didn’t work with the cloud software● Kernel panic within the Connector Node
○ Syslogs point at the fuse mount● Our SCSI target devices sporadically
undiscoverable
![Page 20: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/20.jpg)
LA-UR-14-26016Unclassified
Conclusions
● The SCSI fibre channel protocol operates with a limited number of Linux distributions○ Currently, Ubuntu 14.04 is the only tested
working OS ● Unified storage is under early development● May have needed a larger ring to successfully
optimize the I/O of the cloud storage system
![Page 21: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/21.jpg)
LA-UR-14-26016Unclassified
Future Work● Identifying the kernel panic in Connector server
○ Investigate stack trace of dereference null pointer● Testing the effects of caching directly on the
Connector ○ Could eliminate possible latency created during the
data movement across the fibre channel ● Creating a RAID 0 array of the four PCI Flash devices
○ Combining all four in a RAID array (960 GB) could give a maximum I/O speed of 1800 MB/s
● Evaluate more caching methods and investigating methods to fine-tune their performance○ Flashcache and EnhanceIO
● Unified storage setup needs further investigation
![Page 22: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/22.jpg)
LA-UR-14-26016Unclassified
Summary
● Unified Storage● SSD/Flash Caching● Testbed● Obstacles● Conclusions● Future Work
![Page 23: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/23.jpg)
LA-UR-14-26016Unclassified
References
● Unified Storage○ http://en.wikipedia.org/wiki/Converged_storage
● dmcache○ http://blog.kylemanna.
com/linux/2013/06/30/ssd-caching-using-dmcache-tutorial/
○ http://en.wikipedia.org/wiki/Bcache● bcache
○ http://www.linux.com/learn/tutorials/754674-using-bcache-to-soup-up-your-sata-drive
○ http://bcache.evilpiepirate.org/
![Page 24: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/24.jpg)
LA-UR-14-26016Unclassified
Acknowledgments
● Mentors○ H.B. Chen○ Sean Blanchard○ Jeff Inman
● Summer Institute Instructor○ Dane Gardner
![Page 25: The Effects of SSD Caching on the I/O Performance of ... · PDF fileUnclassified LA-UR-14-26016 dm-cache A device-mapper target that allows the creation of hybrid volumes--block device](https://reader034.vdocuments.net/reader034/viewer/2022052608/5a8525447f8b9a9f1b8c4616/html5/thumbnails/25.jpg)
LA-UR-14-26016Unclassified
Heidi Sandoval● Email: [email protected] Dwyer● Email: [email protected] Pearson● Email: [email protected]
Questions?