![Page 1: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/1.jpg)
Network File Systems
Victoria Krafft
CS 614
10/4/05
![Page 2: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/2.jpg)
General Idea
People move around
Machines may want to share data
Want a system with: No new interface for applications No need to copy all the data No space consuming version control
![Page 3: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/3.jpg)
Network File Systems
Diagram from http://www.cs.binghamton.edu/~kang/cs552/note11.ppt
![Page 4: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/4.jpg)
A Brief History
Network File System (NFS) developed in 1984 Simple client-server model Some problems
Andrew File System (AFS) developed in 1985 Better performance More caching client-side
SFS 1999 NFS can be run on untrusted networks
![Page 5: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/5.jpg)
Lingering Issues
Central server is a major bottleneck
All choices still require lots of bandwidth
LANs getting faster & lower latency Remote memory faster than local disk ATM faster with more nodes sending data
![Page 6: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/6.jpg)
Cooperative Caching
Michael D. Dahlin, Randolph Y. Wang, Thomas E. Anderson, and David A. Patterson in 1994
ATM, Myrinet provide faster, low-latency network
This makes remote memory 10-20x faster than disk
Want to get data from memory of other clients rather than server disk
![Page 7: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/7.jpg)
Cooperative Caching
Data can be found in:1. Local memory
2. Server memory
3. Other client memory
4. Server disk
How should we distribute cache data?
![Page 8: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/8.jpg)
Design DecisionsPrivate/Global Coop. Cache?
Coordinated Cache Entries?
Static/Dynamic Partition?
Block Location? Weighted LRU
Hash
N-Chance
Direct Client Cooperation
Greedy Forwarding
Cent. Coord
Private
Any Client
DynamicStatic
Coordination
Global
No Coordination
Fixed
![Page 9: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/9.jpg)
Direct Client Cooperation
Active clients use idle client memory as a backing store
Simple
Don’t get info from other active clients
![Page 10: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/10.jpg)
Greedy Forwarding
Each client manages its local cache greedily
Server stores contents of client caches
Still potentially large amounts of data duplication
No major cost for performance improvements
![Page 11: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/11.jpg)
Centrally Coordinated Caching
Client cache split into two parts – local and global
![Page 12: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/12.jpg)
N-Chance Forwarding
Clients prefer to cache singlets, blocks stored in only one client cache.Instead of discarding a singlet, set recirculation count to n and pass on to a random other client.
![Page 13: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/13.jpg)
Sensitivity
Variation in Response Time with Client Cache Size
Variation in Response Time with Network Latency
![Page 14: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/14.jpg)
Simulation results
Average read response time Sever load
![Page 15: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/15.jpg)
Simulation results
Slowdown
Slowdown
![Page 16: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/16.jpg)
Results
N-Chance forwarding close to best possible performance
Requires clients to trust each other
Requires fast network
![Page 17: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/17.jpg)
Serverless NFS
Thomas E. Anderson, Michael D. Dahlin, Jeanna M. Neefe, David A. Patterson, Drew S. Roselli, and Randolph Y. Wang in 1995
Eliminates central server
Takes advantage of ATM and Myrinet
![Page 18: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/18.jpg)
Starting points
RAID: Redundancy if nodes leave or fail
LFS: Recovery when system fails
Zebra: Combines LFS and RAID for distributed systems
Multiprocessor Cache Consistency: Invalidating stale cache info
![Page 19: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/19.jpg)
To Eliminate Central Servers
Scaleable distributed metadata, which can be reconfigured after a failure
Scalable division into groups for efficient storage
Scalable log cleaning
![Page 20: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/20.jpg)
How it works
Each machine has one or more roles:1. Client
2. Storage Server
3. Manager
4. Cleaner
Management split among metadata managers
Disks clustered into stripe groups for scalability
Cooperative caching among clients
![Page 21: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/21.jpg)
xFS
xFS is a prototype of the serverless network file system
Lacks a couple features: Recovery not completed Doesn’t calculate or distribute new manager or
stripe group maps No distributed cleaner
![Page 22: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/22.jpg)
File Read
![Page 23: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/23.jpg)
File Write
Buffered into segments in local memory
Client commits to storage
Client notifies managers of modified blocks
Managers update index nodes & imaps
Periodically, managers log changes to stable storage
![Page 24: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/24.jpg)
Distributing File Management
First Writer – management goes to whoever created the file
*does not include all local hits
![Page 25: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/25.jpg)
Cleaning
Segment utilization maintained by segment writer
Segment utilization stored in s-files
Cleaning controlled by stripe group leader
Optimistic Concurrency control resolves cleaning / writing conflicts
![Page 26: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/26.jpg)
Recovery
Several steps are O(N2), but can be run in parallel
Steps For Recovery
![Page 27: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/27.jpg)
xFS Performance
Aggregate Bandwidth Writing 10MB files
Aggregate Bandwidth Reading 10MB files
NFS max with 2 clients
AFS max with 32 clients AFS max with
12 clients
NFS max with 2 clients
![Page 28: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/28.jpg)
xFS Performance
Average time to complete the Andrew benchmark, varying the number of simultaneous clients
![Page 29: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/29.jpg)
System Variables
Aggregate Large-Write Bandwidth with Different Storage Server Configurations
Variation in Average Small File Creation Speed with more Managers
![Page 30: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/30.jpg)
Possible Problems
System relies on secure network between machines, and trusted kernels on distributed nodes
Testing done on Myrinet
![Page 31: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/31.jpg)
Low-Bandwidth NFS
Want efficient remote access over slow or wide area networks
File systems better than CVS, copying all data over
Want close-to-open consistency
![Page 32: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/32.jpg)
LBFS
Large client cache containing user’s working set of files
Don’t send all the data – reconstitute files from previous data, and only send changes
![Page 33: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/33.jpg)
File indexing
Non-overlapping chunks between 2K and 64K
Broken up using 48 byte Rabin fingerprints
Identified by SHA-1 hash, indexing on first 64 bits
Stored in database, recomputed before use to avoid synchronization issues
![Page 34: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/34.jpg)
Protocol
Based on NFS, added GETHASH, MKTMPFILE, TMPWRITE, CONDWRITE, COMMITTMPSecurity infrastructure from SFSWhole file cachingRetrieve from server on read unless valid copy in cacheWrite back to server when file closed
![Page 35: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/35.jpg)
File Reads
![Page 36: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/36.jpg)
File Writes
![Page 37: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/37.jpg)
Implementation
LBFS server accesses file system as an NFS clientServer creates trash directory for temporary filesServer inefficient when files overwritten or truncated, which could be fixed by lower-level access.Client uses xfs driver
![Page 38: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/38.jpg)
Evaluation
![Page 39: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/39.jpg)
Bandwidth consumptionMuch higher bandwidth
for first build
![Page 40: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/40.jpg)
Application Performance
![Page 41: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/41.jpg)
Bandwidth and Round Trip Time
![Page 42: Network File Systems Victoria Krafft CS 614 10/4/05](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649dc35503460f94ab5a24/html5/thumbnails/42.jpg)
Conclusions
New technologies open up new possibilities for network file systems
Cost of increased traffic over Ethernet may cause problems for xFS, cooperative caching.