operating systems 2010/2011johanl/educ/2in05/fs_2010_1.pdfoperating systems 2010/2011 30/11/2010...

Operating Systems2010/2011

30/11/2010 1TU/e Computer Science, System Architecture and Networking

Johan J. Lukkien, Shudong Chen , [email protected]

File Systems – part 1 (ch10)

Shudong Chen

Agenda

• Tasks, requirements for filesystems• User view• The API• Namespaces• File sharing• File protection



30/11/2010 2

Motivation

• All applications need store or retrieve information

• Where can information be kept?

– Memory (running, limited)

– Process’ address space (running)

• Limited size

• Volatile

• Not sharable



• Not sharable

• It can be stored on disk

– Persistent

– Sharable

– Large size

• But using disk is not easy for ordinary users

– The solution is: File & Filesystem

30/11/2010 3

What is a filesystem?

• Two answers

1. The data structure found on a disk or other storage device that represents a

collection of files

• the description and documentation of this data structure...

– ...specifies the interpretation of the data structure

– ...specifies the rules that must be followed to access and modify the data structure, i.e., the

rules to access, store and retrieve these files

• one might call this the ‘physical filesystem’



30/11/2010 4

2. The subsystem of the OS that deals with named

resources in general and files in particular

• for files, it follows the mentioned rules to make them available via the API

• it can be told to deal with many different physical filesystems

– the concepts of the physical filesystems must be mapped to

the concepts expressed at the API

– may use a standard internal VFS API for that

• this is often called the ‘virtual filesystem’

Filesystem task

• Abstract, coherent, structured view on memory objects– persistent: maintain across system and process shutdown

– independent of (virtual) memory size

– sharing: between users, processes

• Requirements– naming mechanism

• notice: files are just one example of resources that need to be referred to

– transparent: hide details of the underlying hardware



30/11/2010 5

– transparent: hide details of the underlying hardware

– portable: avoid dependence on particular hardware to influence the entire

design

• e.g. standard filesystems in ROM or in flash should be possible

• high abstraction level in API

– fault protection: against user induced and hardware faults

– access control: authorization, authentication

– secure: data integrity

– efficient: in access, in use of underlying storage devices

File service

• API: provides a file access service

• System programs: provide a user view on files– e.g. explorer, Unix shell

• Issues in service definition:– what is a file?

• what are the useful abstractions, what are the concepts?

– choice of access model – what are the useful controls for a programmer?

• what are the provided operations?



30/11/2010 6

• what are the provided operations?

• in which way is the file accessed? (e.g. linear or random access)

• Tradeoffs:– between the requirements and efficiency

• (i.e. efficient use of underlying storage hardware, access efficiency)

• there will be many files, but only few of them are accessed at any time; most of them will be just in

background memory

• only during file manipulation the file contents should be quickly available

– copies of accessed parts in main memory

– file attributes recording state information to optimize this

• for efficiency, some of this will be controllable by the programmer

Views of filesystems

• Users concern how files appear to them– What constitute a file?

– How files are named?

– What operations are allowed on files?

• OS designers concern how files are implemented



– How are files structured?

– How to keep track of free storage?

– How many sectors are in a logical block?

30/11/2010 7

Agenda




30/11/2010 8

What is a file?

• A file is a named sequence of logical records

– logical record: smallest addressible unit in a file

• Views on logical organization

1. Files are just named byte(bit) sequences

• the filesystem is unaware of the contents

• interpretation by application

– naming conventions: use extensions [.txt, .com, .exe ...]



30/11/2010 9

– binding name-application

2. or: files are typed

• the filesystem is aware of the contents

• file type determines its use (interpretation) and organization

• define allowed uses [e.g. editor cannot open binary file or directory, text file cannot be loaded for execution, etc.]

• additional structure is included within the file, based on this type

– blocks, records: sequence of data structures

– reference structure: linked records, e.g. tree

• file can in fact be any resource

Example: file types (name, extension)

• Types:

– Data

• numeric

• character

• binary

– Program



30/11/2010 10

Typical views on file internal organization

• Filesystem view

– minimal typing available: directory, regular file, special file

• typing just a help in the interpretation, much less a limitation in use

– typically, regular files are just byte sequences

• User/Application view

– sees the typing provided by the filesystem

– in addition, application knows how to interpret file content

• application-determined type



• application-determined type

• Operating system view

– system programs are similar to just any application in interpreting file types

• e.g. Linux command for showing a directory knows how to interpret a directory file

– kernel is aware of the structure of an executable file

– in addition

• kernel shares the naming system with the filesystem (uses filenames to refer to arbitrary

resources)

• kernel manages the devices and buffers needed by the file system

30/11/2010 11

Example: file types (internal structure)

• Types:

– Data

• numeric

• character

• binary

– Program



30/11/2010 12

(a) An executable file (b) An archive

File attributes

• Every file has a name and its data

• In addition, all OS associate other info to files

• These other information are called attributes



30/11/2010 13

File attributes (Cont)

• Permanent attributes

– Type

– Ownership

– Size

– Disposition (permanent/

temporary)

– protection, access rights



30/11/2010 14

– protection, access rights

– dates (access, modification,

creation)

• Temporary attributes, created and maintained upon access

– shortcut to location, dates, open count, size, locking

– read pointer, write pointer, buffers

– ....‘dynamic information’

Access methods

1. Sequential Access

• Read all bytes/records from the beginning

• Cannot jump around, could rewind

• Convenient when medium was tapeSequential-access File



2. Direct Access

• Bytes/records read in any order

• Essential for data base systems

• Read can be …

– Move file marker (seek), then read

– Or Giving the position to start, then Read

Simulation of Sequential Access on Direct-access File

Access methods (Cont)

3. Index and relative files access

• Built on top of direct-access method

• Involve the construction of an index of the file

• First search the index

• Then use the pointer to access the file directly

• Can create secondary index files

• e.g., IBM’s index sequential-access method (ISAM)



Directory structure

• A collection of nodes containing information about all files

– Directories themselves are files!

• Both the directory structure and the files reside on disk

• Backups of these two structures are kept on tapes



Disk structure

• Disk can be subdivided into partitions

• Disks or partitions can be RAID protected against failure

• Disk or partition can be used raw – without a file system, or formatted with a file system

• Partitions also known as minidisks, slices (in the IBM world)

• Entity containing file system known as a volume



• Each volume containing file system also tracks that file system’s info in device directory or volume table of contents

• As well as general-purpose file systems there are many special-purpose file systems, frequently all within the same operating system or computer

A typical filesystem organization



Organize the directory (logically) to obtain

• Efficiency – locating a file quickly

• Naming – convenient to users

– Two users can have same name for different files

– The same file can have several different names



• Grouping – logical grouping of files by properties,

(e.g., all Java programs, all games, …)

30/11/2010 20

Implied directory structure

• Single-Level Directory

• Two-Level Directory

• Tree-Structured Directories

• Acyclic-Graph Directories

• General Graph Directory



• General Graph Directory

30/11/2010 21

Single-Level Directory

• Simplest directory structure:

– All files are contained in the same directory

– Easy to support and understand.

• A single directory for all users



• Significant limitations:

– Naming problem

– All files are in the same directory, they must have unique names

– File names are often limited in length

– Grouping problem

Two-Level Directory

• A standard solution to file name-collision among different users induced

by single-level directory– Separate directory for each user

• Path name: to name a file,

both the user name and the

file name must be given



file name must be given

• File naming syntax:

• C:\user1\test (volume, directory name, file name, MS-DOS) u:[sst.jdeck]login.com;1 (volume, directory, subdirectory, version number, VMS)

• Can have the same file name for different user

• Efficient searching

• first local UFD, if not found, then the special user directory that contains the files

• No grouping capability

Tree-Structured Directories

• A natural generalization of a two-level directory– a tree of arbitrary height

– a directory or subdirectory contains a set of files or subdirectories

– bit entry: file (0), subdirectory(1)



Tree-Structured Directories (Cont)

• Efficient searching

– Current directory (working directory)

• cd /spell/mail/prog

• system call: change directory ----taking a directory name as a

parameter



• Grouping Capability

– Allows a user to define her own subdirectories

Tree-Structured Directories (Cont)

• Absolute or relative path name

• Creating a new file is done in current directory

• Delete a file

rm <file-name>

• Creating a new subdirectory is done in current directory

mkdir <dir-name>

Example: if in current directory /mail



Example: if in current directory /mail

mkdir count

mail

prog copy prt exp count

Deleting “mail” ⇒ deleting the entire subtree rooted by “mail”

Acyclic-Graph Directories

• Have shared subdirectories and

files

– A tree structure prohibits this

sharing

– Sharing two copies of a file !

• A graph with no cycles

• The same file or subdirectory may

≠



• The same file or subdirectory may

be in two different directories.

• A natural generalization of the tree-

structured directory scheme.

Acyclic-Graph Directories (Cont.)

• If dict deletes list ⇒ dangling pointer

• Solutions:

– Search for these dangling pointers, so we can delete them all

– Keep a file-reference list

• Delete a file when the list is empty

• Variable size of the list records a problem

– Entry-hold-count solution



– Entry-hold-count solution

• Only keep a count of the number of references

• A file can be deleted when the count is 0

• New directory entry type

– Link – another name (pointer) to an existing file

– Resolve the link – follow pointer to locate the file

General Graph Directory

• A serious problem with acyclic-graph structure is ensuring that there is

no cycles.

– Adding new files & subdirectories preserves the tree-structure nature.

– Adding links destroy the tree-structure, therefore, a graph structure is

resulted.



General Graph Directory (Cont.)

• How do we guarantee no cycles?

– Allow only links to file not subdirectories

• Cycles are allowed to exist in directories (self-referecing).

– Garbage collection

• self-referecing (a cycle) in the directory structure

– A value of 0 in the reference count means that there is no more

references to the file of directory.



references to the file of directory.

– The reference count may not be 0 when it is no longer possible to

refer to a directory or file.

• to determine when the last reference has been deleted and the

disk space can be reallocated.

– Every time a new link is added use a cycle detectionalgorithm to determine whether it is OK

• Bypass links during directory traversal

Agenda




30/11/2010 31

API operations for regular files

• Create / delete file

– insert/remove file into/from system

• create: make an empty location

– a file is identified by its name

• this relies on some naming system

• Open / close

– create and initialize (/destroy) file descriptor

• descriptor: handle on the file: reference to attributes

• kernel internal data structures are reserved/returned



30/11/2010 32

• kernel internal data structures are reserved/returned

• Read / write

– access file at ‘current location’

• Seek

– change ‘current location’

• Inspection and modification of attributes

– e.g. change access rights, protection

– often in combination with containing directory

• Explicit synchronization with background memory

– flush disk cache, or just the blocks of this file

– Linux: sync() and fsync()

Example: opening a file from an application

• fd = open (/usr/johanl/Aap, mode, flags)

– search the directory structure on disk for entry /usr/johanl/Aap, and

move the content of entry to memory

– mode: read, write, append, readwrite

– flags: steering behavior of open, e.g. create file when non-existent

• Several pieces of data are needed to manage open files:



• Several pieces of data are needed to manage open files:

– File pointer: pointer to last read/write location, per process that has

the file open

– File-open count: counter of number of times a file is open – to allow

removal of data from open-file table when last processes closes it

– Disk location of the file: cache of data access information

– Access rights: per-process access mode information

Open file locking

• Provided by some operating systems and file systems

• Mediates access to a file

• Mandatory or advisory:

– Mandatory – access is denied depending on locks held and requested

– Advisory – processes can find status of locks and decide



– Advisory – processes can find status of locks and decide what to do

Example: File locking–Java API

import java.io.*;

import java.nio.channels.*;

public class LockingExample {

public static final boolean EXCLUSIVE = false;

public static final boolean SHARED = true;

public static void main(String arsg[]) throws IOException {

FileLock sharedLock = null;

FileLock exclusiveLock = null;

try {



try {

RandomAccessFile raf = new RandomAccessFile("file.txt", "rw");

// get the channel for the file

FileChannel ch = raf.getChannel();

// this locks the first half of the file - exclusive

exclusiveLock = ch.lock(0, raf.length()/2, EXCLUSIVE);

/** Now modify the data . . . */

// release the lock

exclusiveLock.release();

File Locking Example – Java API (Cont)

// this locks the second half of the file - shared

sharedLock = ch.lock(raf.length()/2+1, raf.length(), SHARED);

/** Now read the data . . . */

// release the lock

sharedLock.release();

} catch (java.io.IOException ioe) {

System.err.println(ioe);

}finally {



}finally {

if (exclusiveLock != null)

exclusiveLock.release();

if (sharedLock != null)

sharedLock.release();

}

}

}

Other API operations

• Operations according to the type of the file

– file type determines interpretation of contents

• filesystem may interpret and use this type internally, it can provide

special operations and/or it can make the contents available as a byte

sequence

• e.g. symbolic link, directory

• Is rename an operation on a file?



• Is rename an operation on a file?

• Example: directory operations: move, rename, list, (un)link (=(remove)create new name to same thing), search

– these are typical operations to manipulate files in the directory, or

reachable from that directory

– a possible API for reading a directory would be: dopen, dnextentry(),

dclose()

30/11/2010 37

Directory operations semantics

• Create:– generate empty structure

– needs a parent directory to make it in

• typically, a current working directory: a reference to interpret pathnames; defined in the process

structure

• Delete: remove entry; possibly: remove subtree, recursively– notice: file deletion is a directory operation!



• Rename: new name to directory entry

• Move: new directory for entry

• List: examine contents

• Find: recurse with query through structure or set of directories– usually not within the API

– but needed as part of OS operation

• e.g. searching for executable in a series of standard directories

Agenda




30/11/2010 39

Example name spaces



30/11/2010 40

Namespace

• Namespace: set (of names)

– according to specification/algorithm as how to generate names

• Construction methods/algorithms

– flat naming

• simply an unstructured set of names

– structured naming

• hierarchical, mostly, e.g., /user/schen/....



• hierarchical, mostly, e.g., /user/schen/....

– relative name: relative with respect to a given prefix (e.g. working directory)

– absolute name: complete

• prefixing

– embedding a set of names uniquely into a larger set

» » e.g. <uml: specific tag>

• can have flat naming per level

• Purpose

– reference to entities

• i.e., anything that can be operated upon

30/11/2010 41

Entities, access points and addresses

• E: Entities

– the collection of ‘things’ that need to be referred

• files, but also: resources, devices, network ports, internet addresses....

• X: Access points

– location or means where an entity can be accessed

– an entity may have several access points



– an entity may have several access points

• A: Addresses

– the namespace for access points

– an address of an access point is unique (only one AP per address)

– addresses of access points may change over time

30/11/2010 42

Binding, Resolution

• Binding: (the establishment of) the relationship between a reference (a

name) and a referred object (another name or an access point)

– symbolic: referred object is again a similar reference – same level

– aliasing: several names for the same object

• Resolution: given a name, determine referred object

– ‘structured’ resolution: follow structured naming hierarchy, e.g.,



• given a DNS name, find Transport address, then find MAC address

– ‘flat’ resolution: flat naming, no hierarchy support in resolution, e.g.,

• reverse DNS: what is the DNS name of this IP address?

– not entire flat though as still information is remaining in the IP address

• reverse ARP: what is the IP address of this MAC address?

• ... table lookup

• Closure: starting point in resolution

30/11/2010 43

Example: naming in Unix filesystems



30/11/2010 44

Hierarchical naming systems

• Graph: labeled, directed, acyclic

– edge labels: partial names

– paths: names

• Leaf nodes

– access point

– name in new naming system



– name in new naming system

– name in same naming system (‘symbolic’)

• Directory nodes

– access point to directory structure

• references to child nodes

– often represented by a special symbol

• “/”, “.”

30/11/2010 45

Name space examples

• File systems

– usually, hierarchy

– closure: / (root)

– resolution: start with root and work through the path

• repeat: get access point for subdirectory

• Internet addresses: a.b.c.d (130.155.7.5)

– basic hierarchy (subnets); enumeration

– binding: e.g. DHCP (Dynamic Host Configuration Protocol)

– closure and resolution: ARP (Address Resolution Protocol)



– closure and resolution: ARP (Address Resolution Protocol)

–

• Internet names (win.tue.nl)

– Hierarchy

– closure: local DNS server

– binding: within DNS by system administrators

– resolution: DNS lookup protocol (yields Internet-address)

• URL

– what are closure, resolution, binding? Hierarchy?

– http://www.win.tue.nl/~johanl/educ/2IN05/index.html

– protocol || domain name || path at which a particular web page is located on the web server

30/11/2010 46

Joining namespaces

• Direct catenation

– e.g., http://www.win.tue.nl/~johanl/educ/2IN05/

• subdirectories in http

• Use name of system as a prefix – implicit new root

– systemA:/home/..., systemB:/home/....

– resolve in new root

• Mounting



• Mounting

– (tell the resolution procedure that) from a certain prefix p on, a new resolution is used

• mainly modification of resolution procedure

• closure from p onward stored with p in a table of the resolver (the virtual file system)

– transparent resolution

• Symbolic linking

– store reference in second name system as referred object

• e.g. contents of /home/schen/aap is oracle1://schen

• similar in effect to mounting, but less dynamic: stored within the file itself

– closure included in the link

30/11/2010 47

Symbolic link into other namespace



30/11/2010 48

Mounting

• A file system must be mounted before it can be available. – Specially, the directory structure may be built out of multiple volumes, which

must be mounted to make them available within the file-system name space.

• Execute: Mount /device/dsk over /users



30/11/2010 49

(a) Existing system (b) Unmounted volume residing on /device/dsk

Mounting

• Execute: Mount /device/dsk over /users



30/11/2010 50

Mount Point

Unix and Windows logical organization

• Is this organization reflected in

the name space?

– (a) Unix: yes

– (b) Windows: no (i.e.,

\Desktop\My Computer\c is not

a valid name)

• for Windows, it is a presentation structure



presentation structure

• However, because of

mounting, the logical

organization in Unix is not

reflected in the physical

organization

– e.g. /usr may be a separate

filesystem

30/11/2010 51

Agenda




30/11/2010 52

File sharing

• Sharing of files on multi-user systems is desirable

• Sharing may be done through a protection scheme

• On distributed systems, files may be shared across a

network



network

• Network File System (NFS) is a common distributed

file-sharing method

File sharing – multiple users

• User IDs identify users, allowing permissions and

protections to be per-user

• Group IDs allow users to be in groups, permitting

group access rights



File sharing – remote file systems

• Uses networking to allow file system access between systems

– Manually via programs like FTP

– Automatically, seamlessly using distributed file systems

– Semi automatically via the world wide web

• Client-server model allows clients to mount remote file systems from servers

– Server can serve multiple clients



– Server can serve multiple clients

– Client and user-on-client identification is insecure or complicated

– NFS is standard UNIX client-server file sharing protocol

– CIFS is standard Windows protocol

– Standard operating system file calls are translated into remote calls

• Distributed Information Systems (distributed naming services)such as LDAP, DNS, NIS, Active Directory implement unified access to information needed for remote computing

File sharing – failure modes

• Remote file systems add new failure modes, due to network failure, server failure

• Recovery from failure can involve state information about status of each remote request

• Stateless protocols such as NFS include all information in



• Stateless protocols such as NFS include all information in each request, allowing easy recovery but less security

File sharing – consistency semantics

• Consistency semantics specify how multiple users are to access a shared file simultaneously

– Similar to Ch 6 process synchronization algorithms• Tend to be less complex due to disk I/O and network latency (for remote file

systems)

– Unix file system (UFS) implements:• Writes to an open file visible immediately to other users of the same open file

• Sharing file pointer to allow multiple users to read and write concurrently



• Sharing file pointer to allow multiple users to read and write concurrently

– Andrew File System (AFS) implemented complex remote file sharing semantics

• AFS has session semantics– Writes only visible to sessions starting after the file is closed

Agenda




30/11/2010 58

Protection

• File owner/creator should be able to control:

– what can be done

– by whom

• Types of access

– Read



– Read

– Write

– Execute

– Append

– Delete

– List

Access Lists and Groups

• Mode of access: read (R), write (W), execute (X)

• Three classes of users

RWX

a) owner access 7 ⇒ 1 1 1RWX

b) group access 6 ⇒ 1 1 0

RWX

c) public access 1 ⇒ 0 0 1

• Ask manager to create a group (unique name), say G, and add some users



• Ask manager to create a group (unique name), say G, and add some users to the group.

• For a particular file (say game) or subdirectory, define an appropriate access.

owner group public

chmod 761 game

Attach a group to a file

chgrp G game

Windows XP Access-control List Management



Summary




30/11/2010 63

• OS view – implementation (next week)

Exercises

• Ch10 – 1, 3, 6, 11, 14, 15



30/11/2010 64

operating systems 2010/2011johanl/educ/2in05/fs_2010_1.pdfoperating systems 2010/2011 30/11/2010...

Documents