semantically-smart disk systems muthian sivathanu, vijayan prabhakaran, florentina popovici, tim...

Semantically-Smart Disk Systems

Muthian Sivathanu, Vijayan Prabhakaran,

Florentina Popovici, Tim Denehy, Andrea Arpaci-Dusseau, Remzi Arpaci-

Dusseau

University of Wisconsin, Madison

The Storage Protocol Stack

A Tale of Two Layers

Two layers: The file system and RAIDOrigins: The hardware/software

boundary• S/W: The file system• H/W: The disk

Each increasingly complexEach a world unto itself• Separated by a narrow interface

(SCSI)

FileSystem

RAID

Your Innovations In Storage:Where In The Stack Should They Go?

Option #1: In The File System

Put innovations in the file system!Pros:• Lots of semantic information

(directories, files, inodes)• Well-defined interfaces

(vnode/VFS)

Cons:• No low-level information

(head position, RAID scheme)• Hard to deploy: Many FS’s out

there• File systems change slowly

• FFS [1984] -> ext2 [2002]

FileSystem

RAID

Option #2: In The RAID

Put innovations in the RAID system!

Pros:• Lots of low-level knowledge• Easy to deploy, sell• Lots of “smarts” (CPUs +

Memory)

Cons:• “A stream of block reads + writes,

full of bits & bytes, signifying nothing”(no semantic knowledge)

FileSystem

RAID

The Root Of The Problem: Narrow Interfaces

Narrow interfaces stifle info flow• SCSI to file system:

Linear array of blocks• SCSI to disk system:

Stream of block reads and writes

Result: Innovation is limited• No “FS-like” innovations in RAID• Anything where info from BOTH

subsystems is required can’t be done

FileSystem

RAID

SCSI

So What’s A Storage-SystemInnovator To Do?

The Ideal Solution?

Exploit raw processing and memory of RAID• Built in low-level knowledge• Easy to deploy

Keep interface the same• Assume traditional file system/RAID boundary

Acquire semantic information of file system• Learn about files, directories, inodes, and

other file system data structures

Semantically-Smart Disk System (SDS)

Disk system that understands file system• Data structures• Operations

Operates underneath unmodified FS• Must discover layout + on-disk structures• Must “reverse engineer” block stream

Exploits knowledge and “smarts” to implement new class of services

FileSystem

SDS

$CPU

Outline

MotivationSemantic Knowledge: AcquisitionSemantic Knowledge: Exploitation• Case studies

Conclusions

Static Knowledge: File System Layout

Challenge: How to discover layout information?• White-box approach: Embed knowledge in SDS

• Trend: FS layout does not change frequently

Su

perb

lk

I-B

itm

ap

D-B

itm

ap

Inod

es

Data

I-B

itm

ap

D-B

itm

ap

Inod

es

Data

Group 1 Group 2

Have Knowledge, Will Innovate

Knowing structures is not enough (sometimes)• Data block overloading (data, pointer,

directory)• High-level operations not known (create,

delete)

Requires new on-line techniques• Direct classification• Indirect classification• Block association• Operation inferencing

Direct Classification

Given address, determine type directlyDirect classification via bounds check• Given disk address, can check bounds to

determine type(superblock, bitmaps, inodes, general data block)

Sup

er

I-B

it

D-B

it

Inod

e

Data

I-B

it

D-B

it

Inod

e

Data

Beyond Simple Meta-DataWant to cache other meta-data blocks• Directory blocks, indirect-pointer blocks

Problem: Data blocks are overloaded type• Block in “data” region could be: file data, dir,

pointer• Direct classification necessary but not

sufficient

Indirect classification via inode snooping

Sup

er

I-B

it

D-B

it

Inod

e

Data

I-B

it

D-B

it

Inod

e

Data

Indirect Classification: Directories

Sup

er

I-B

it

D-B

it

Inod

e

Data

I-B

it

D-B

it

Inod

e

Data

SDS

Directory Hash Table

Ptr1

Ptr2

Ptr3

DirectoryType

Ptr1Ptr2Ptr3

Size

Inod

e

Directory Hash Table

Indirect Classification In Action

Sup

er

I-B

it

D-B

it

Inod

e

Data

I-B

it

D-B

it

Inod

e

Data

SDS

Ptr1

Ptr2

Ptr3

Check HashIf Present, Cache

Indirect Classification: Issues

Space overhead• Small overhead per directory and indirect block

Time overhead• 1 hash update per pointer, 1 lookup per data

block

Tricky: Sometimes data block is seen “early”• Haven’t yet seen pointer associated with data

block• Solution #1: OK to misclassify some

blocks for some amount of time• Solution #2: Must defer classification

until pointer has been observed

Getting Rid Of The Dead

If file blocks are deleted, remove them from cache• No need to keep dead blocks around

Problem: How to determine if a file is deleted?• Need to look for signs of deletion

Three different places to look:• Inode bitmaps• Directory that contains file• Inode itself

Operation inferencing via block differencing

SDS

Operation Inferencing: Detecting Deletes (Inode

Bitmap)S

up

er

I-B

it

D-B

it

Inod

e

Data

I-B

it

D-B

it

Inod

e

Data

Diff =

Read Old Version

I-B

itm

ap

Result:Deleted Files

Operation Inferencing: Overheads

Space overhead• Block cache of inodes, indirect pointers,

bitmaps, etc.(could be substantial)

Time overhead• CPU: Difference operation is like an extra copy• Disk: May require block read (if small/no

cache)

[In paper: Quantified time and space overheads]• Main point: There is a CPU and memory cost

Using Semantic Knowledge

Fast RAID Reconstruction• Only copy “live” data to hot-spare

Track-aligned Extents [Schindler et. al. ‘02]• Placing files in disk cognizant manner

Journaling [Hagmann ‘87]• Limit study: can implement complex “FS”-like

functionality within a semantically-smart disk

File-aware Caching• Caching meta-data under BSD FFS• Caching journal under Linux ext3

Conclusions

Conclusions

Innovation in traditional storage stack is limited• File system: high but not low-level info• Storage system: low but not high-level info

Semantically-smart disks: Best of both worlds?• Takes advantage of “smart” disk systems• Exploit low-level information…• …with high-level knowledge of file system

A remaining challenge• Overcoming the “file system obfuscation”

problem

“To know that we know what we know,and that we do not know what we do not

know, that is true knowledge.”

Confucius

http://www.cs.wisc.edu/wind

“A man cannot inquire about what he knows, because he knows it,

and in that case he is in no need of inquiry, nor again can he inquire about

what he does not know, since he does not know what he is to inquire.”

Socrates