introduction to operating systemsgauss.ececs.uc.edu/courses/c4029/lectures/fs_implement_1.pdf ·...

Introduction to Operating Systems

File System Implementation

John FrancoElectrical Engineering and Computing Systems

University of Cincinnati

Layered File System

Application Programs⇓

Logical File System⇓

File Organization Module⇓

Basic File System⇓

I/O Control⇓

Device

• The file system is composed of many different levels

• Each level in the design uses the features of lower levelsto create new features for use by higher levels

Logical File System Layer

• Manages metadata information

• Metadata includes all of the file-system structure except the actualcontents of the files

• Manages the directory structure to provide the file organizationmodule with the information it needs, given a symbolic file name.

• Maintains file structure via file-control blocks (FCB)

• FCB contains file info such as ownership, permissions, and locationof the file contents

• Responsible for protection and security

Typical File Control Block

• permissions

• last access, last modified, created dates/times

• owner, group, access control list (users, what they can do)

• size

• data blocks or pointers to file data blocks

File Organization Layer

• Knows about files and their logical blocks, as well as physical blocks

• Can translate logical block addresses to physical block addressesfor the basic file system to transfer data

• Each file’s logical blocks are numbered from 0 to N

• Includes the free-space manager which tracks unallocated blocks andprovides these blocks to the file organization module when requested

Basic File System Layer

• Issues generic commands to the appropriate device driver to readand write physical blocks on the disk

• Each physical block is identified by its numeric disk address (in thecase of hard disks). For example:drive 1, cylinder 73, track 2, sector 10.

I/O Control Layer

• Consists of device drivers and interrupt handlers to transferinformation between the main memory and the disk system

• Recall, a device driver translates high-level commands such as“retrieve block 123” to low-level, hardware-specific instructionsthat are used by the hardware controller which interfaces thedevice to the rest of the system

File Structures

• With a layered implementation there is less duplication of code

• For example, the I/O Control and Basic File System layers can beused by many file systems

• A logical layer and a file organization layer can be written for eachfile system separately

File Structures on Disk

• The file system may contain information about how to boot anoperating system stored there, the total number of blocks, thenumber and location of free blocks, the directory structure, andindividual files

• Examples:

– Boot Control Block: information needed by the system toboot from that volume

– Volume control block: volume or partition details, such asthe number of blocks in the partition, size of the blocks, freeblockcount and free-block pointers, and free FCB count and FCBpointers. In UFS, this is called a superblock(use dumpe2fs /dev/sda5)

– Directory structure: in UFS, this includes file names andassociated inode numbers (use ls -i)

– File Control Block: contains file details including permissions,ownership, size, and location of the data blocks. In UFS, this iscalled the inode.

File Structures in Memory

• For file-system management and performance improvement viacaching

• The data are loaded at mount time and discarded at dismount

• The structures may include the ones described below:

– Mount table: information about each mounted volume

– Directory-structure cache: holds the directory informationof recently accessed directories

– Open-file table: contains a copy of the FCB of each openfile, as well as other information - global

– Open-file table (per process): contains a pointer to theappropriate entry in the global open-file table, as well as otherinformation

File Creation

1. Application calls logical file system (LFS) which knows the formatof the directory structures

2. A new FCB is allocated by the LFS, or, if the file systemimplementation creates all FCBs at file system creation time, anFCB is allocated from the set of free FCBs

3. The LFS reads the appropriate directory into memory from disk

4. The directory is updated with the new file name and FCB andis written back to the disk

5. The LFS can call the file organization module to map the directoryI/O into disk-block numbers, which are passed on to the basic filesystem and I/O control system.

File Structures in Memory

open(file) ✲

directory structure

directory structure

FCB

✘✘✘✘✘✘✘✾❳❳❳❳❳❳❳③

user space kernel space disk

read(file) ✲ ✲

proc OFT global OFT

data blocks

FCB

✘✘✘✘✿

❳❳❳❳②

user space kernel space disk

File Open

1. The open() call passes a file name to the file system

2. The open() system call searches the global open file table (OFT)to see if the file is already in use by another process

3. If so, a per-process OFT entry is created pointing to the existingglobal OFT

4. The directory structure is searched for the given file name(parts of the directory structure are cached for speed)

5. Once the file is found the FCB is copied into a global OFT in memory

6. This OFT also maintains the number of processes that have the fileopen

7. An entry is made in the process OFT with a pointer to the entry inthe global OFT plus some other fields such as cursor and access mode

8. open() returns a pointer to the entry in the process OFT

9. All operations are performed via this file descriptor (or handle)

File Close

1. The process OFT entry is removed and the global OFT entry’s opencount is decremented

2. When all users that have opened the file close it, any updatedmetadata is copied back to the disk-based directory structure andthe global OFT entry is removed

Partitions and Mounting

• Partition can be a volume containing a file system (“cooked”) or justa sequence of blocks with no file system (“raw”)

• Boot block can point to boot volume or boot loader set of blocks that containenough code to know how to load the kernel from the file system

• Root partition contains the OS, other partitions can hold other OSes, otherfile systems, or be raw

– Mounted at boot time

– Other partitions can be mounted automatically at boot or manually

• System consistency is checked at mount time

– If the metadata is correct then mount and add to mount table

– Otherwise, correct the problem and try again

• Look at /etc/mtab

Unix Mounting

• Implemented by setting a flag in the in-memory copy of the inode for thedirectory on which a partition is mounted (the flag indicates that thedirectory is a mount point)

• Then an inode field points to an entry in the mount table, indicating whichdevice is mounted there

• The mount table entry contains a pointer to the superblock of the file systemon that device

• This scheme enables the operating system to traverse its directory structure,switching among file systems of varying types, seamlessly

Virtual File System

• Virtual File Systems (VFS) on Unix provide an object-oriented way ofimplementing file systems

• VFS allows the same system call interface (the API) to be used for differenttypes of file systems

– Separates file-system generic operations from implementation details

– Implementation can be one of many file system types, or network filesystem

– Activates file-system-specific operations to handle local requestsaccording to their file-system types and even calls the NFS protocolprocedures for remote requests

• The API belongs to the VFS interface, rather than any specific type of filesystem

Virtual File System Architecture

• Four main object types:

– inode object: represents an individual file

– file object: represents an open file

– superblock object: represents an entire file system

– dentry object: represents an individual directory entry

• For each object type, the VFS defines operations that must be implementedThese operations are implemented for each file system

• Thus, the VFS invokes a function, say read, without knowing or caring aboutthe file system type it is dealing with

Directory Implementation

• The selection of directory-allocation and directory-management algorithmssignificantly affects the efficiency, performance, and reliability of the file system

Linear List:

– Simple to program - expensive to execute

– To create a new file

1. search the directory to be sure no existing file has the same name2. add new entry at the end of the directory

– To delete a file

1. search the directory for the named file2. release the space allocated to it3. use a linked list to improve performance

– To reuse the directory entry

1. mark the entry as unused (give it a special name) or2. attach it to a list of free directory entries or3. copy the last entry in the directory into the freed location


• linear list should be sorted in some special structure for log(n) access,deletion, and insertion

B Tree:

– http://gauss.ececs.uc.edu/Courses/c4029/lectures/btrees.pdf

Red Black Tree:

– http://gauss.ececs.uc.edu/RedBlack/redblack.html


• The selection of directory-allocation and directory-management algorithmssignificantly affects the efficiency, performance, and reliability of the file system

Hash Table:

– Complex to program - cheap to execute - requires good hash functionusually table is of fixed size (if sizing is wrong, performance suffers)

– To create a new file

1. apply hash function to user-path pair to locate a cell in the table2. if the cell is occupied, add a node to a linked list referenced from cell

– To delete a file

1. apply hash function to user-path pair to locate a cell in the table2. walk the linked list to find the node corresponding to the file3. release the space allocated to the file

– To reuse the directory entry

1. mark the entry as unused (give it a special name) or2. attach it to a list of free directory entries or3. copy the last entry in the directory into the freed location

Block Allocation

• Allocating disk space to files so that space is used efficiently and yet filescan be accessed efficiently requires careful thought

Contiguous Allocation:

– All files occupy a contiguous set of blocks

– Positives

1. good for sequential access - head moves from track to adjacenttrack, sectors on each track are read in order as they spin underthe head

2. directory entries need only point to the first block as the locationof all others can be calculated from that and block size

– Negatives

1. finding space for a new file can be really tough2. there will probably be an awful lot of external fragmentation3. some sort of de-fragmentation is needed - such algorithms are

really slow4. if a file is modified to be larger, all hell breaks loose (usually)5. fix for above - begin with contiguous chunk, then add extents

(other contiguous chunks) as needed

Block Allocation• Allocating disk space to files so that space is used efficiently and yet files

can be accessed efficiently requires careful thought

Linked Allocation:

– Each file is a linked list of blocks

– Positives

1. external fragmentation is gone - de-fragmentation is unnecessary2. free space is easy to find - when a block is given up, a special

number is inserted into the link3. files can grow as much as they like until blocks are exhausted

– Negatives

1. random access possibly needs to traverse many links, inefficient2. a lot of space is needed for the linked structures - mitigated

by using clusters of blocks3. if a link is lost, it is difficult to recover a file - mitigated by

having a backup copy of the FAT4. the FAT must be cached since otherwise head moves to beginning

of FAT, moves to directory entry, moves to desired link, moves todesired block

FAT16 Allocation

• Files are linked through a File Allocation Table containing a logicallycontiguous array of two byte elements, one for each cluster, and suchthat the ith element corresponds to the ith cluster

• Given the jth cluster of a file as system cluster k, the j + 1st file clusteris found from the kth array element of FAT

• A special end-of-file number in an array element says there are no furtherclusters belonging to the file

• The beginning of the linked list is found from an entry in the directorytable - a directory entry contains 32 bytes and looks like this:

file size0AB1ldateltimeadatecdatectimeatrH O W D Y J P G

where the first 11 bytes are used for the file name and extension, the twobytes above containing number 0xB10A indicate the starting cluster, andthe remaining bytes are time and date stamps of various kinds, plus somereserved bytes and an attribute byte

• The following is a section of the FAT starting at index B10A:

0DB1 14B1 13B112B110B1FFFF

❅ �✒❅❅ ��✒�❅■�❅■�❅■

Block Allocation

• Allocating disk space to files so that space is used efficiently and yet filescan be accessed efficiently requires careful thought

Indexed Allocation:

– bring all block pointers together in a single array of disk block addresses

info

inode

File Blocks✲✲✲✲✲✲✲✲✲✲✲✲✲✲✲✲✲✲✲✲

✲Index Structure

Index Struc✲✲✲✲✲✲✲✲✲✲

Index Structure

✲✲✲✲✲✲

Index Structure

✲

✲

✲

Positives:1. fast random access2. no external fragmentation

Negatives:1. wasted indexing space2. mitigated as shown to left

Dentry Tree Structure

HOWDY

dentry (directory)

DOODY

dentry (file)

BUFFALO

dentry (file)

BOB

dentry (directory)

inode

info

inode

info

✏✏✏✏✏✏✏✏✏✏✏✶

PPPPPPPPPPPq

❄

✻✻✻

❄

✻

❄

✻

❄• dentry objects are arranged in a tree with one root• a dentry object has a pointer to a linked list of children which may include

directories, files, and symbolic links• there is also a pointer to the parent• all siblings are in a doubly linked list due to a third set of pointers

Superblock Structurestruct ext2 super block {

u32 s inodes count; /* Inodes count */

u32 s blocks count; /* Blocks count */

u32 s free blocks count; /* Free blocks count */

u32 s free inodes count; /* Free inodes count */

u32 s first data block; /* First data block */

u32 s log block size; /* Block size */

u32 s log cluster size; /* Cluster size */

u32 s first ino; /* First non-reserved inode */

u16 s inode size]; /* Size of inode structure */

u32 s blocks per group; /* A file’s blocks in same group */

u32 s inodes per group; /* A dir’s inodes in same group */

. . .

}

• Open a filesystem like this:struct struct ext2 filsys filsys;

ext2 filsys fs = &filsys;

ext2fs open("/dev/sdb2", EXT2 FLAG RW, 0, 0, unix io manager, &fs);

• Find first inode like this:ext2 inode t ino = fs->super->s first ino;

Inode Structure

struct ext2 inode {u16 i mode; /* File mode */

u16 i uid; /* Low 16 bits of Owner Uid */

u32 i size; /* Size in bytes */

u32 i atime; /* Access time */

u32 i mtime; /* Modification time */

u16 i gid; /* Low 16 bits of Group Id */

u16 i links count; /* Links count */

u32 i blocks; /* Blocks count */

u32 i flags; /* File flags */

u32 i block[EXT2 N BLOCKS]; /* Pointers to blocks */

u32 i generation; /* File version (for NFS) */

u32 i file acl; /* File Access Control List */

. . .

}

• By way of an example, i flags might be 0x80000 (extents!)and i mode might be 0x180 which means this is not a directoryand the i block array has the structure shown on the nextslide (EXT2 N BLOCKS=15, for a total of 60 bytes)

Inode Structure

0A F3 01 00 04 00 00 00 00 00 00 00

00 00 00 00 01 00 00 00 64 81 00 00

00 00 00 00 00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 00 00 00 00

• The number 0xF30A is a magic number, verifying type

• The number 0x1 in blue means there is 1 extent

• The number 0x8164 in green is the block number of that extent

• To visit the block at that number do this:unsigned char buffer[1000];

blk t blocknr = 0x8164;

io channel read blk(fs->io, blocknr, 1, buffer);

A partial, typical result is on the next slide

Directory Structure0 | 02 00 00 00 0c 00 01 02 2e 00 00 00 02 00 00 00 | ................

10 | 0c 00 02 02 2e 2e 00 00 0b 00 00 00 18 00 0e 01 | ................

20 | 63 68 61 6c 6c 65 6e 67 65 73 2e 74 78 74 00 00 | challenges.txt..

30 | 0c 00 00 00 14 00 09 01 63 69 74 65 73 2e 74 78 | ........cites.tx

40 | 74 00 00 00 01 f9 01 00 24 00 05 02 44 69 72 65 | t.......$...Dire

50 | 63 70 64 66 33 32 4c 6f 67 5c 64 65 62 75 67 6c | cpdf32Log\debugl60 | 6f 67 2e 74 78 74 00 00 0e 00 00 00 14 00 09 01 | og.txt..........

70 | 66 69 78 65 73 2e 74 78 74 00 00 00 0f 00 00 00 | fixes.txt.......

80 | 14 00 0a 01 68 69 74 73 65 74 2e 74 78 74 00 00 | ....hitset.txt..

90 | 10 00 00 00 14 00 09 01 6e 6f 74 65 73 2e 74 78 | ........notes.tx

a0 | 74 00 00 00 11 00 00 00 14 00 0b 01 73 61 74 5f | t...........sat

b0 | 69 64 78 2e 74 78 74 00 12 00 00 00 48 0f 08 01 | idx.txt.....H...

c0 | 74 6f 64 6f 2e 74 78 74 00 00 00 00 00 00 00 00 | todo.txt........

d0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................

• If i mode had indicated a directory and i flags an extent then the blockcontains directory entries consisting of variable length records with thefollowing format: 4 bytes: inode number, 2 bytes: record length,2 bytes: name length, remaining bytes: name of file, directory, or link

• The first three records are shown in blue, red, green

• Entry challenges.txt points to inode 11 (0xb) which is on the next slide

File Inode

0A F3 01 00 04 00 00 00 00 00 00 00

00 00 00 00 01 00 00 00 64 81 00 00

00 00 00 00 00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 00 00 00 00

• Inode 11 blocks array. This inode is pointed to by challenges.txt dentryas on the previous slide

• The i flags field was 0x8000 so the format is for extents

• The 0x1 shown blue says there is one extent and its block number is shownin red to be 0x8164 (33124)

• Block 33124 is shown on the next slide

introduction to operating systemsgauss.ececs.uc.edu/courses/c4029/lectures/fs_implement_1.pdf ·...

Documents