![Page 1: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/1.jpg)
Semantically-Smart Disk Systems
Muthian Sivathanu, Vijayan Prabhakaran,
Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-
Dusseau
University of Wisconsin, Madison
![Page 2: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/2.jpg)
The Storage Protocol Stack
![Page 3: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/3.jpg)
A Tale of Two Layers
Two layers: The file system and RAIDOrigins: The hardware/software
boundary• S/W: The file system• H/W: The disk
Each increasingly complexEach a world unto itself• Separated by a narrow interface
(SCSI)
FileSystem
RAID
![Page 4: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/4.jpg)
Your Innovations In Storage:Where In The Stack Should They Go?
![Page 5: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/5.jpg)
Option #1: In The File System
Put innovations in the file system!Pros:• Lots of semantic information
(directories, files, inodes)• Well-defined interfaces
(vnode/VFS)
Cons:• No low-level information
(head position, RAID scheme)• Hard to deploy: Many FS’s out
there• File systems change slowly
• FFS [1984] -> ext2 [2002]
FileSystem
RAID
![Page 6: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/6.jpg)
Option #2: In The RAID
Put innovations in the RAID system!
Pros:• Lots of low-level knowledge• Easy to deploy, sell• Lots of “smarts” (CPUs +
Memory)
Cons:• “A stream of block reads + writes,
full of bits & bytes, signifying nothing”(no semantic knowledge)
FileSystem
RAID
![Page 7: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/7.jpg)
The Root Of The Problem: Narrow Interfaces
Narrow interfaces stifle info flow• SCSI to file system:
Linear array of blocks• SCSI to disk system:
Stream of block reads and writes
Result: Innovation is limited• No “FS-like” innovations in RAID• Anything where info from BOTH
subsystems is required can’t be done
FileSystem
RAID
SCSI
![Page 8: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/8.jpg)
So What’s A Storage-SystemInnovator To Do?
![Page 9: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/9.jpg)
The Ideal Solution?
Exploit raw processing and memory of RAID• Built in low-level knowledge• Easy to deploy
Keep interface the same• Assume traditional file system/RAID boundary
Acquire semantic information of file system• Learn about files, directories, inodes, and
other file system data structures
![Page 10: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/10.jpg)
Semantically-Smart Disk System (SDS)
Disk system that understands file system• Data structures• Operations
Operates underneath unmodified FS• Must discover layout + on-disk structures• Must “reverse engineer” block stream
Exploits knowledge and “smarts” to implement new class of services
FileSystem
SDS
$CPU
![Page 11: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/11.jpg)
Outline
MotivationSemantic Knowledge: AcquisitionSemantic Knowledge: Exploitation• Case studies
Conclusions
![Page 12: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/12.jpg)
Static Knowledge: File System Layout
Challenge: How to discover layout information?• White-box approach: Embed knowledge in SDS
• Trend: FS layout does not change frequently
Su
perb
lk
I-B
itm
ap
D-B
itm
ap
Inod
es
Data
I-B
itm
ap
D-B
itm
ap
Inod
es
Data
Group 1 Group 2
![Page 13: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/13.jpg)
Have Knowledge, Will Innovate
Knowing structures is not enough (sometimes)• Data block overloading (data, pointer,
directory)• High-level operations not known (create,
delete)
Requires new on-line techniques• Direct classification• Indirect classification• Block association• Operation inferencing
![Page 14: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/14.jpg)
Direct Classification
Given address, determine type directlyDirect classification via bounds check• Given disk address, can check bounds to
determine type(superblock, bitmaps, inodes, general data block)
Sup
er
I-B
it
D-B
it
Inod
e
Data
I-B
it
D-B
it
Inod
e
Data
![Page 15: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/15.jpg)
Beyond Simple Meta-DataWant to cache other meta-data blocks• Directory blocks, indirect-pointer blocks
Problem: Data blocks are overloaded type• Block in “data” region could be: file data, dir,
pointer• Direct classification necessary but not
sufficient
Indirect classification via inode snooping
Sup
er
I-B
it
D-B
it
Inod
e
Data
I-B
it
D-B
it
Inod
e
Data
![Page 16: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/16.jpg)
Indirect Classification: Directories
Sup
er
I-B
it
D-B
it
Inod
e
Data
I-B
it
D-B
it
Inod
e
Data
SDS
Directory Hash Table
Ptr1
Ptr2
Ptr3
DirectoryType
Ptr1Ptr2Ptr3
Size
Inod
e
![Page 17: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/17.jpg)
Directory Hash Table
Indirect Classification In Action
Sup
er
I-B
it
D-B
it
Inod
e
Data
I-B
it
D-B
it
Inod
e
Data
SDS
Ptr1
Ptr2
Ptr3
Check HashIf Present, Cache
![Page 18: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/18.jpg)
Indirect Classification: Issues
Space overhead• Small overhead per directory and indirect block
Time overhead• 1 hash update per pointer, 1 lookup per data
block
Tricky: Sometimes data block is seen “early”• Haven’t yet seen pointer associated with data
block• Solution #1: OK to misclassify some
blocks for some amount of time• Solution #2: Must defer classification
until pointer has been observed
![Page 19: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/19.jpg)
Getting Rid Of The Dead
If file blocks are deleted, remove them from cache• No need to keep dead blocks around
Problem: How to determine if a file is deleted?• Need to look for signs of deletion
Three different places to look:• Inode bitmaps• Directory that contains file• Inode itself
Operation inferencing via block differencing
![Page 20: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/20.jpg)
SDS
Operation Inferencing: Detecting Deletes (Inode
Bitmap)S
up
er
I-B
it
D-B
it
Inod
e
Data
I-B
it
D-B
it
Inod
e
Data
Diff =
Read Old Version
I-B
itm
ap
Result:Deleted Files
![Page 21: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/21.jpg)
Operation Inferencing: Overheads
Space overhead• Block cache of inodes, indirect pointers,
bitmaps, etc.(could be substantial)
Time overhead• CPU: Difference operation is like an extra copy• Disk: May require block read (if small/no
cache)
[In paper: Quantified time and space overheads]• Main point: There is a CPU and memory cost
![Page 22: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/22.jpg)
Using Semantic Knowledge
Fast RAID Reconstruction• Only copy “live” data to hot-spare
Track-aligned Extents [Schindler et. al. ‘02]• Placing files in disk cognizant manner
Journaling [Hagmann ‘87]• Limit study: can implement complex “FS”-like
functionality within a semantically-smart disk
File-aware Caching• Caching meta-data under BSD FFS• Caching journal under Linux ext3
![Page 23: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/23.jpg)
Conclusions
![Page 24: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/24.jpg)
Conclusions
Innovation in traditional storage stack is limited• File system: high but not low-level info• Storage system: low but not high-level info
Semantically-smart disks: Best of both worlds?• Takes advantage of “smart” disk systems• Exploit low-level information…• …with high-level knowledge of file system
A remaining challenge• Overcoming the “file system obfuscation”
problem
![Page 25: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/25.jpg)
“To know that we know what we know,and that we do not know what we do not
know, that is true knowledge.”
Confucius
http://www.cs.wisc.edu/wind
![Page 26: Semantically-Smart Disk Systems Muthian Sivathanu, Vijayan Prabhakaran, Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University](https://reader036.vdocuments.net/reader036/viewer/2022070404/56649f385503460f94c5553a/html5/thumbnails/26.jpg)
“A man cannot inquire about what he knows, because he knows it,
and in that case he is in no need of inquiry, nor again can he inquire about
what he does not know, since he does not know what he is to inquire.”
Socrates