CODA/AFS(Constant Data Availability)

--Jay N. Vyas & Naveen Mohan--

Background

CODA is what? – A Distributed File System (DFS)

Developed by? – Mahadev Satyanarayanan

Where & Why? – At Carnegie Mellon University as a Research Project in 1987

CODA/AFS - Introduction

A decentralized distributed file system (DFS) to be accessed from autonomous workstations.

Derived from AFS (Andrew File System) and inherits many of the core features.

AFS was a very successful DFS for a campus-sized user community.

Was planned to be extended at a national level but WWW took over instead.

Key ideas of AFS include:

Close-to-open consistency

Callbacks

Design Motto

CODA had three fundamental objectives:

Scalability - to build a system that can easily cope with addition of users and sites

Fault-Tolerance - to remain usable in the advent of server failures, communication failures and voluntary disconnections

Unix Emulation – to make Coda appear as a giant failure-proof shared Unix file system

Key Features

How is CODA different from AFS then? - Fault Tolerance!

Provides “constant availability” through Replication

Introduces a “disconnected mode” for portable computers (Mobile Computing)

Supplementary Features:

Is freely available under a Liberal License (Source Code is accessible)

High Performance through Client side persistent caching

Uninterrupted service during partial network failures

Highly Scalable

Provides Unified file sharing using well defined semantics even in the event of network failures

History: AFS-1 semantics

First version of AFS:

Revalidated cached file on each open

Propagated modified files when they were closed

If two users on two different workstations modify the same file at the same time, the users closing the file last will overwrite the changes made by the other user

Open-to-Close Semantics

Example:

TimeF

F’

F’ F”

First client

Second client

F” overwrites F’

TimeF

F’

F’

F”

F”

First client

Second client

F” overwrites F’

AFS-2 semantics

AFS-1 required each client to call the server every time it was opening an AFS file

Most of these calls were unnecessary as user files are rarely shared

AFS-2 introduces the callback mechanism

Do not call the server, it will call you!

When a client opens an AFS file for the first time, server promises to notify it whenever it receives a new version of the file from any other client

Promise is called a callback

Relieves the server from having to answer a call from the client every time the file is opened

Significant reduction of server workload

AFS-2 semantics (Contd..)

Callbacks can be lost!

Client will call the server every tau minutes to check whether it received all callbacks it should have received

Cached copy is only guaranteed to reflect the state of the server copy up to tau minutes before the time the client opened the file for the last time

Coda semantics

Client keeps track of subset s of servers it was able to connect the last time it tried

Updates s at least every tau seconds

At open time, client checks it has the most recent copy of file among all servers in s

Cached copy can be up to tau minutes behind the server copy

Conflicts between cached copy and most recent copy in s can happen

Coda guarantees they will always be detected

At close time, client propagates the new version of the file to all servers in s

If s is empty. client marks the file for later propagation

Server Replication

Data structures:

Volume

A set of files and directories located on one server (shared)

Each file and directory in Coda has a unique low-level file identifier (FID). All replicas of an object has the same FID.

VSG (Volume Storage Group):

Set of servers storing replicas of a volume.

AVSG (Accessible Volume Storage Group):

Venus (client cache manager) keeps track of the subset of servers accessible to the client for every volume the client has cached.

Preferred Server:

One of the AVSG servers of the client, that it accesses during a cache miss. Chosen based on physical proximity, server load or server CPU power.

Holds all callbacks for client & answers all the read – write requests from client.

Server Replication(contd..)

Strategy

Read – any , Write – all approach

Client still checks with other servers (in AVSG) to find which one has the latest version/copy of the data.

When a file is closed after modification by a client it is transferred in parallel to all members of the AVSG.

Minimizes Server CPU load

Improved scalability, since Server is the bottleneck in many distributed file systems.

Cache Coherence – Client identifies 3 types of events no later than Τ seconds,

Enlargement of AVSG – Accessibility of a previously inaccessible server.

Shrinking of AVSG – Inaccessibility of a previously accessible server.

Lost Callback event

Coda Version Vector ( CVV )

A volume CVV is similar to a file or directory CVV, but summarizes update information on the entire volume.

Parallel Communication

Replication requires multiple sites to be contacted.

If serialized, latency would be degraded intolerably.

MultiRPC - parallel remote procedure call mechanism which were previously used was extended to use hardware multicast to reduce the latency and network load.

Versioning vector (Coda Version Vector) when partition happens: [1,1,1]

Client A updates file versioning vector in its partition: [2,2,1]

Client B updates file versioning vector in its partition: [1,1,2]

Partition repaired compare versioning vectors: conflict!

Server Replication (Conflict Scenario)

Conflict Resolution Conflict Detection

Coda does not prevent inconsistent updates but guarantees that inconsistent updates will always be detected

Automated Resolution

All directory conflicts are automatically resolved by a compensating sequence of create or delete operations, except

Update/update conflict

Remove/update conflict

Name/name conflict

Damage Containment

Marks all accessible replicas of the objects inconsistent. (if can’t be automatically resolved).

Repair Tool (Manual)

Special Interface to Venus (Client Cache Manger)

Enables the tool to overwrite inconsistent files and to perform directory operations on inconsistent directories, subject to normal access restrictions

Server Replication - Performance

Introducing Mounting Operation

The Andrew file system, Coda's predecessor, pioneered the idea and stored all files under /afs.

A client (user) connects to “Coda” and not to individual servers; but the latter come into play invisibly.

All the files, which any of the servers may provide to the client, are available under the /coda directory, and all clients see the same name space.

Why is Single mount point (name space) advantageous:

Users will see the same file tree

Auto-discovery of new servers and shares by the Client machines in /coda tree

CODA Operation – Disconnection Mode

How does it provide services when the Network crashes?

Scenario:

User wants to open a file “foo” – cat /coda/tmp/foo

A System call is generated to the kernel and it looks for inode of the file which returns the file handle associated with that program (what is inode? RECALL!)

Open call proceeds to the Virtual File System (VFS) (What is VFS?) which is handed to the Coda File system present on the Kernel

CFS in turn “may” forward the request to “Venus” (What is Venus? RECALL!)

Venus checks first in the client cache for /tmp/foo, if cache-miss, then contacts the Vice servers.

After successful file location, Venus responds back to the kernel with info which returns to the calling program.

Fig. The Architecture

When does Disconnection Mode crank up?

Possibilities:

When there is no network connection to any server which has the files.

Happens for laptops when taken off the network, or during network failures.

Or if the servers fail.

And how does it work? (Caching)

Entire file fetched from the servers and stored as a container file in the local cache area by Venus on receiving the first kernel request

(currently/usr/coda/venus.cache/).

If the file is opened a second time or R/W operations performed, local copy is only used.

If it’s modified and closed, then Venus updates the servers by sending the new file.

Operations like making directories, removing files or directories and creating or removing (symbolic) links are propagated to the servers also.

From Caching to Disconnection

Servers synchronously updated with modified files in normal connection mode

Time-out occurs if Server unavailable or Network failure

The Ball Begins to Roll!!! (Activate Disconnect Mode)

Instead of instantly updating the user with time-out, Venus logs all updates in CML (Client Modification Log); frequently flushed to the disk

Upon reconnection of the network, CML file is reintegrated with the server– Run Updates

Another Fault Tolerance feature achieved with crisp Transparency…

Issues with disconnected operation

Hoarding Files

Unable to serve cache-misses during network anomaly

Hence Stack up updates to the important files on local cache (client side)

Known as “Hoard Walk”

Client/Server Conflict

Multiple clients may have modified the file during disconnect mode to CML.

Needs Repair – either Automatically (Resolvers) with rare chances of Human intervention

CONCLUSION & Paper Highlights

Coda project had an unusually long life First work to introduce disconnected mode

Server Replication resulting in increased System resiliency

Coda/AFS introduced many new ideas Close-to-open consistency

Optimistic concurrency control

Callbacks (partially superseded by leases)

References

Coda File System: http://www.coda.cs.cmu.edu

"The Coda Distributed File System." The Coda Distributed File System. N.p., n.d. Web. 11 Mar. 2015.URL: http://www.coda.cs.cmu.edu/ljpaper/lj.html

Mahadev Satyanarayanan; Kistler, J.J.; Kumar, P.; Okasaki, M.E.; Siegel, E.H.; Steere, D.C., "Coda: a highly available file system for a distributed workstation environment," Computers, IEEE Transactions on, vol.39, no.4, pp.447,459, Apr 1990doi: 10.1109/12.54838URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=54838&isnumber=1979