file management chapter 121 file management file: –unit of related information (uniquely) named...

File Management Chapter 12 1

File Management

• File:– Unit of related information (uniquely) named

and accessed by users• Smallest unit of user storage• OS maps file to physical secondary storage device

– Typically nonvolatile disk

• Can be text or binary, free form or formatted– Can be organized as bits/bytes/lines or records

• Can be code, data, or possibly links, devices, directories, etc.


File System Requirements• Persistence

– Not terminated by process termination

– Not volatile

– Remains until owner deletes it

• Storage of LARGE amounts of information

• Controlled concurrent access by users– Shareable

– Access control

• Structure (internal and between files)


File operations

• Creating/destroying– Allocating/deallocating storage– Placing/removing from tables– Allocating/deallocating file control block

• Reading/writing/executing– Maintaining cursor– Protection

• Repositioning pointer (seek)• Appending/truncating files• Renaming• Open – return ptr; close


Sample file attributes/properties/metadata• File name• Identifier (for OS)• Location and/or relative path in file system• Current Size• Dates of creation, modification, access• Protection• Owner ID.• File type and/or application association (in Windows- different in

UNIX)• Count of users

– OS allows multiple users of single file– OS handles locks and consistent usage

• Access rights to file (to be verified)• Lock flags if multiple users are allowed• FLAGS: ASCII/binary; hidden; read-only; system/normal; archive;

temp


Typical File types• Executable (.exe, .com, within bin library)• Object (.obj, .o)• Source code (.java, .c, .cc, .cxx, .asm, .ada• Batch – to OS (.bat, shell scripts)• Text (.txt)• Word processor (.doc, .wp, .tex)• Compressed (.zip, .tar)• Multimedia (.mpeg)• Image (.gif)• Meaning to OS varies in different systems• Increasing numbers of different (incompatible) formats


Linux Virtual File System

• Provides uniform interface with multiple implementations

Format compatibility

• Of course, newer versions are almost always backward compatible

• Discuss with class why software providers may not be interested in format compatibility in general

• More and more issues of applications and different types of formats



Organizing files• Disks are partitioned (organized as volumes)

– A disk has at least one partition– Dual boot contains two partitions, each with separate

OS; at boot time user decides which partition is active– Multiple partitions are possible– GUIs make partitioning user friendly

• System maps logical drive to physical drive• Each partition has its own directory structure


Directories• Each partition has a directory

– For each file – keep name, size, date; physical location

• Hierarchical directories– MULTICS– Two-level (user id at top level)– Tree structured – relative and absolute path names

• Delete directory – UNIX rm –r * .o (comment on space) – Windows delete key (are you sure)

– Acyclic-graph directories• Links (hard and soft)

– Deletion in UNIX and Windows leaves soft link as invalid access

• Hard links may be implemented with reference count for deletion or for creating a copy for the current users


Question• What are the issues in deleting a file that

has hard links to it?– The hard links (addresses) are alive.

• If the storage is deallocated and reallocated to a new file, users will have access to that storage

• If we do not deallocate the storage, will the owner still be charged for storage that he does not use?

– When can we safely deallocate storage?• Maintain reference count of links to the file

– Issue of circular links

• Give user his own copy of the file


File system mounting

• Windows 95 and later versions automatically locate and mount attached file systems (on each device) at boot time– Go to top level on Windows Explorer– If device is added, Windows reconfigures automatically– Error can occur if too many compressed file systems are already

mounted, or that file system is already mounted, or you attempt to mount over nonempty file system

• UNIX mount command is typically placed into system configuration file that is executed at boot time; others must be coded in– usr/sbin/mount [-d] [-r|-u|-w] [-o option, ...] [-t [no]type] file-

system


Mounting a disk

• At mount time, OS verifies format– Places partition (logical disk) in tables, etc.

• In Windows, each partition is given a logical name (ex: l:)

• ScanDisk– If system was shut down “abruptly,” at boot

time OS checks that files have been closed, checks metadata consistency


File Sharing• Group permissions• Remote systems

– Anonymous ftp sites– World Wide Web (files transferred via ftp)– Distributed file systems (DFS)

• Remote file system can be mounted as if it were on a local drive

– File stored on “server” or workstation• Sharing – Networks and Sharing Center in Windows 7• User authentication

– Extranets, Intranets• Consistency semantics more complex for remote systems

– A shared file may be declared read only (immutable)


Protection from improper access

• Access rights – Files (r, w, x), append, delete, list – read/write/read & execute/modify/full control– directory access rights (also list, delete)

• Access control– owner, group, world– Finer grained rights??– Access matrices/capability rights


Question• What features in a file interface promote user

friendliness?• Uniformity

– Browser interface; GUI• Capabilities

– Directories for organizing files– Links for navigating directories and sharing– Access methods, such as direct or indexed access

• Transparency and abstraction– Virtual file system– SANs– Location transparency in general

• Protection– authentication, authorization (access rights)


File System Implementation• File system is located in secondary storage

– Typically located on disks • Advantages of Size, Cost, Direct Access,

Speed• Nonvolatile• More robust than flash memory

– “Virtual disks” - temporary storage in RAM disks, memory mapped files, caches

Storage Allocation• Files are mapped to disk locations

– Minimal disk storage allocation is by sector (perhaps 512 bytes)

– Operating System may allocate• Variable contiguous portions

– Faster access; file allocation table is small– Not flexible (what if we have to increase file size)

• Fixed small portions (a few sectors) – we’ll call these blocks

– Easier allocation and deallocation as file changes size– Non-contiguous allocation– Larger allocation tables

• Blocks may be grouped into clusters, chunksFile Management Chapter 12 17


Some data structures utilized during file management

• File control block

• List of open files

• File Allocation Table (where file is stored)

• Process Control block – Contains pointers to files that the process has

opened

• Directory table


Issues with file mapping to disk

• If file is mapped to blocks on disks, internal fragmentation occurs for last part of last block

• Mapping logical record to physical block– Packing logical records into a physical block

• Numbers may be packed as well to reduce redundancies

• OS must locate and unpack record in physical block– OS also supports user compression/decompression

• Size of block (larger block gains faster I/0 and smaller table size but fragmentation is greater)


File Storage

• File addresses are mapped to (typically) disk addresses– Translation to base address + offset– Unit of disk allocation is a sector– Unit of file allocation is a block (or cluster)

• Tables maintained of each file system– Ex: open file table (compare to ready list)

• Table maintained for each file (FCB)


Magnetic Disks

• Direct access media• Read/write/rewrite/ possibly append• Disk divided into sectors

– Smallest unit of allocation• Perhaps 512 bytes; 2048 bytes for optical disks

– OS groups sectors into blocks or larger storage units

• Issues of allocation of blocks– Contiguous/linked/indexed– Speed versus flexibility


New disk sector size recommended

• 27 March 2006• By Chris Mellor, Techworld• An industry committee has recommended increasing the disk block sector size

from 512 bytes to 4096. IDEMA, the International Disk Drive, Equipment, and Materials Association, says the 30-year standard of 512B should be increased eightfold to 4KB. It will mean disk drives can hold more data and hosts access it faster.

• click here • IDEMA Long Block Committee member Dr. Martin Hassner of Hitachi GST

said: "(The) increasing areal density of newer magnetic hard disk drives requires a more robust error correction code (ECC), and this can be more efficiently applied to 4096 byte sector lengths." A Hitachi paper on the topic can be found here.

• That is because, with the increasing areal density of disks, a bad area corrupts more bits. Having error-correction work over longer bit-strings makes more data recoverable.

http://www.idema.org/

http://adserver.adtech.de/adlink%7C340%7C114502%7C1%7C171%7CAdId=1361934;BnId=1;itime=488718915;key=top;nodecode=yes;link=#altimageclick%23

http://adserver.adtech.de/?adlink%7C2.0%7C340%7C114502%7C1%7C171%7CKEY=top;grp=98849496;loc=300;

http://www.idema.org/_smartsite/modules/local/data_file/show_file.php?cmd=download&data_file_id=1259

http://www.tomshardware.com/reviews/wd-4k-sector,2554.html

• “Western Digital … has launched a new product line, the EARS-series, and moved 4KB sectors into the mainstream. The reason for this change is an increase in net storage capacity due to decreased amounts of ECC information resulting from the larger sector size.” 2010

• (that means that blocks will not be smaller than 4KB)• fsutil fsinfo ntfsinfo c: (command prompt as administrator)

• wmic partition get BlockSize, StartingOffset, Name, Index



File-System architecture – Application programs/ OS kernel

• Logical file system– Maintains file control block (FCB, inode)- metadata

• System file name, user file name, owner, access rights, time, size

• File-Organization module– Maps logical block to physical unit; maintains the free-block list

• Basic file system– Issues commands to device drivers

• Read/write to location (drive, block)

• I/O control– Device drivers and interrupt handlers

• Devices


Support of file system is typically part of kernel

• Performance improvements from kernel support• During boot time, system checks all mounted

devices– OS maintains partition table (table of mounted devices)

• Table of recent accesses– Ptrs to partition table for physical locations

• Open-File table– Ptrs to FCBs, counters for multiple users, cursor for

current position, etc.


Creation of file

• User or application program calls OS

• OS creates FCB– Information about owner, times, physical

location, file name, etc.– Places it in system tables

• Reads directory into memory if necessary

• Updates directory

– Storage is allocated for FCB, file


“Raw” Disk

• Standard disk formatting is not always appropriate• Disk may be formatted for specific use

– Swap disk (no file system)

– Databases (may provide their own structure for data)

– RAID systems need additional formatting information

– Boot disks have additional format information

– Partitioned disks


Directories

• Path names– Absolute/ relative– Always use absolute names if you do not know

what your working directory will be for a call

• Operations– Create, delete, list (requires opening), rename– Link, assign access rights


Directories

– Entry storage and retrieval may be by:• Linear or structured list

– Sequential (search, add, delete)

– Sorted perhaps according to B-tree

– Linked

– Cache provides efficiency for most recently used data

• Hash table– Direct access

– Possibility of collisions

– Directory size is limited by hash function


File storage on disk Contiguous Advantage: Simple, fast retrieval, file table is minimal

• Any block can be directly accessed following computation• Supports direct & sequential access• Used for ROM disks, swap space

How to handle appending data to file• Swap in/swap out if there is no room for file to expand, but

there is room elsewhere on disk• Preallocation of extra blocks will allow file size to increase

– May overestimate needs

– Holes accumulate as files are deleted (external fragmentation)

– Requires disk compaction (defragmentation)


File storage allocation using chunks

• Large contiguous chunks are allocated• First contiguous block set is allocated- more

(extent) allocated on demand• Size of file is stored• Each extent linked to next extent• Used for DVDs/ Universal Disk Format

– UDF limits file size to 1GB requiring 3-4 extents


File storage – linked allocation• Blocks (small chunks) chained together

– Blocks allocated on demand• No external fragmentation

– Each block contains pointer to next block (or null ptr)– Slows down sequential access– Sequential search to achieve “direct access”

• Note # of disk accesses

– Pointers require storage (4-8 bytes)– Group the blocks into clusters for improved

performance (more fragmentation)– Reliability issue if pointers are corrupted


Indexed Allocation with FAT• File-allocation table (FAT)

– Directory table and FAT are stored at beginning of disk/partition; copied to memory upon first disk access

– FAT is searched to find free blocks – entry of 0– Directory table contains link into FAT for each file– FAT entries link file blocks– Supports direct access– Does not scale well- FAT and directory table are

limited in size– For file stored at blocks 4, 88, 95, 10:

Index (implicit) 4 ..10.. 88 ..95 Link (cell value) 88 eof 95 10


File storage with indices & links– Directory with entry for each file– Each file has individual index block that is brought into

memory when file is opened– Large index block requires more memory space– Small index block cannot index large file

– UNIX inode• File attributes are stored (size, time, ID, etc.)• 12 ptrs to direct blocks• 3 ptrs to 1, 2, 3 levels of indirection• Compromise between fast access for small files, # of levels of

indirection (slower access) for larger files


I-node • From D. A. Rusling


Question

• How many bytes would an i-node be able to access directly, indirectly, double indirectly, triple indirectly if each i-node contained 13 direct block pointers and 1 pointer for indirect, double, triple, each pointer was 4 bytes and each block was 64k bytes– Direct – 13 (64kB) +– Indirect – 1 (16k) 64kB +– Double Indirect – 1 (16k) (16k) (64kB) + – Triple Indirect 1 (16k) (16k) (16k) (64kB) = almost– 3 * 1017 Bytes


Question

• Storage allocation for files may be contiguous, linked, or indexed

• Give reasons why each method might or might not be used– Contiguous

• Efficient; Good for swap disk• Bad for dynamically increasing file size

– Linked• Good for comparatively low cost of pointers • Bad for sequential access thru blocks

– Indexed• Good for flexibility of size allocated, (almost) direct access; bad

because a possibly large index must be maintained in memory


Free Space Management• DOS File Allocation Table – searched• Bit vector

– Shift for first free block (0)

– Bit vector is maintained in memory (consider size)

• Free Portions are chained together– Pointer between blocks (units)

– Multiple I/O accesses for multiple blocks

– What if link is corrupted?

– Grouping – store n ptrs in first free block


Performance • Performance varies for different types of files and

storage methods– Contiguous storage is best for direct access– Linked is best for expanding files– Indexed is good for small files– Can combine methods (i.e., i-nodes)

• Changing disk types can hurt hashing performance• Increasing cluster/block size to maximum DMA

transfer size can improve performance for sequential access by cutting down on the number of I/Os


Improvements for Efficiency

• Vary sizes of clusters to reduce internal fragmentation – larger clusters can be allocated if file grows

• Keep i-node (FCB) near file to reduce seek time• Size of pointer to be considered (large ptrs needed

if they must link to entire memory but this requires more storage) – perhaps increase size of clusters rather than size of ptrs


Improvements for efficiency (cont.)• Use caches for indexes, directories, recent data• Maintain indexes, directories, data in memory

• RAM disks, memory mapped I/O• Interleave storage on multiple disks (striping)• Asynchronous writes

• Output is buffered so process can progress• Buffered reads and read-ahead

• Anticipatory fetching


Fault Tolerance• Ability of system to continue to function (perhaps in

degraded mode) if a fault is activated– Chief mechanism- Redundant Components and Data– Duplication of data (file system backups, etc.)

• RAID (mirrored, duplexed, etc)• Store in another location (propagation delay becomes a factor)

– Determine frequency of back up• File may have bits indicating updates/ times

– Maintain consistency• Consistency check – typically each block has checksum or ICV or

ECC at end• Crash or corruption – logs, checkpoints and roll backs

– Appropriate for network systems as well– Fast way to recover data, but we need to repeat all transactions since last

checkpoint


Fault tolerance for hardware failures – providing reliability and availability

• Corruption of disk management information (metadata) or data– Maintain backup for reliability

– Provide on-line redundancy for availability (RAID)

– Use ECCs, logs, etc. for detecting failures

• For remote systems, failure semantics must be defined

– Ex: Crash or shutdown of server

• Does client delay operations or close and try to recover from all operations?

– System maintains state information, including authentication and checkpoints

• System may ignore failures


File System consistency

• Scandisk and fsck check blocks of each file– Each i-node is used to form a list of blocks in use

• Keep track of number of blocks in use– Compare to free block list– All blocks should appear exactly once in either table

• Handle inconsistencies– If file block appears in both lists, remove from free list– If block appears in multiple i-nodes (block list contains

a value of 2 or more for block), copy block and allocate to separate users, report problem to users

• Directories are similarly checked

File System Security• Access Matrix

– Rows of users; columns of files, objects– Each user’s access rights to resource is defined– Access Matrices are sparse– Decompose by row or column


Access Matrix

• One such rule set is an Access Matrix

Access Control Lists

• Access Control Lists decompose access matrix by columns– alpha.fdu.edu> getacl index.html

– #

– # file: index.html

– # owner: levine

– # group: faculty

– #

– user::rw-

– group::r--

– other::r--


Capability Lists• Decomposition by

rows yields capability lists (or ticket)– specifies authorized

objects and operations for a user.


NTFS

• NTFS is the Windows NT equivalent of the Windows 95 file allocation table (FAT) and the OS/2 High Performance File System (HPFS).

• NTFS offers a number of improvements over FAT and HPFS in terms of performance, extendibility, and security.

References: (Book): MCSE Accelerated Windows 2000 Study Guide Exam 70-240(Website): http://www.ntfs.com

Adapted from presentation by Jenny Villa-Dominguez


NTFS improvements

Microsoft created NTFS to improve on some FAT features, such as:

• Increased fault tolerance

• Enhanced security.

• File Compression

• Fragmentation


File System Capabilities

File System CapabilitiesFeature FAT16 FAT32 NTFS

Operating System Support MostWindows 9x, OSR2, and Windows 2000

Windows NT and Windows 2000

Long File Name Support? Yes Yes YesEfficient use of disk space? No Yes YesCompression support? No No YesQuota Support? No No YesEncryption Support? No No YesLocal security support? No No YesNetwork security support? Yes Yes YesMaximum volume size 2GB 32GB 2TB


Example of Fault Tolerance in NTFS

NTFS repairs hard disk errors automatically without displaying an error message.

When Windows writes a file to an NTFS partition, it keeps a copy of the file in memory. It then checks the file to make sure it matches the copy stored in memory. If the copies don't match, Windows marks that section of the hard disk as bad and won't use it again (Cluster Remapping). Windows then uses the copy of the file stored in memory to rewrite the file to an alternate location on the hard disk. If the error occurred during a read, NTFS returns a read error to the calling program, and the data is lost.


SecurityNTFS has many security options. You can grant

various permissions to directories and to individual files. These permissions protect files and directories locally and remotely.


File Compression

Another advantage of NTFS is native support for file compression. The NTFS compression offers you the chance to compress individual files and folders of your choice. Data Compression is only available on NTFS partition. If you copy or move a compressed folder or file to a FAT partition (or a floppy disk), Windows 2000 will automatically uncompress the folder or file.


File compression (cont.)


Disk Quotas

Disk quotas allow administrators to manage the amount of disk space allotted to individual users, charging users only for the files they own. Windows 2000 enforces quotas on a per-user and per-volume basis.

Disk quota space is based on actual file size. There is no mechanism to support or recognize file compression.

TRAFIC LIGHT:RED = specifies that disk quotas are disabled. YELLOW = W2000 is rebuilding disk quota information.

GREEN = the disk quota system is enabled and active.


Disk Quotas (cont.)


Fragmentation

Maintaining a low level of file fragmentation on an NTFS volume is the most important way to improve volume performance. You can accomplish this maintenance by regularly running a disk-defragmentation utility, which makes every file on the volume contiguous. Only with regular use of the defragmentation tool does NTFS gain the full benefit of file defragmentation.

Windows Disk Defragmenter Tool gives you the opportunity to quickly Analyze a volume, and will advise you if defragmentation is recommended.


Fragmentation (cont.)


Other features of NTFS:

1)Use of a B-tree directory scheme to keep track of file clusters

2) Information about a file's clusters and other data is stored with each cluster, not just a governing table (as FAT is)

3) Support for very large files (up to 2 to the 64th power or approximately 16 billion bytes in size)

4) An access control list (ACL) that lets a server administrator control who can access specific files


Other features of NTFS:

5) Integrated file compression

6) Support for names based on Unicode

7) Support for long file names as well as "8 by 3" names

8) Data security on both removable and fixed disks


How NTFS works:

• When a hard disk is formatted (initialized), it is divided into partitions or major divisions of the total physical hard disk space. Within each partition, the operating system keeps track of all the files that are stored by that operating system. Each file is actually stored on the hard disk in one or more clusters or disk spaces of a predefined uniform size. Using NTFS, the sizes of clusters range from 512 bytes to 64 kilobytes.


How NTFS works (cont):

• Windows NT provides a recommended default cluster size for any given drive size. For example, for a 4 GB (gigabyte) drive, the default cluster size is 4 KB (kilobytes). Note that clusters are indivisible. Even the smallest file takes up one cluster and a 4.1 KB file takes up two clusters (or 8 KB) on a 4 KB cluster system.


How NTFS works (cont.)

• The selection of the cluster size is a trade-off between efficient use of disk space and the number of disk accesses required to access a file. In general, using NTFS, the larger the hard disk the larger the default cluster size, since it is assumed that a system user will prefer to increase performance (fewer disk accesses) at the expense of some amount of space inefficiency.

• When a file is created using NTFS, a record about the file is created in a special file, the Master File Table (MFT). The record is used to locate a file's possibly scattered clusters. NTFS tries to find contiguous storage space.

Windows Registry

• The Windows Registry is a centralized hierarchical database containing information on:– Hardware devices– Users– Applications– Desktop configuration– Start up information


Windows Registry• Stores information needed during operation

– User profiles/ rights– Applications

• Types of documents they manipulate

• Ports being used

• File properties

• Registry data is stored in binary • Registry was developed to replace .ini,

autoexec.bat and config.sys files in MsDOSFile Management Chapter 12 66

Configuration information

• Operating system stores its settings in registry

• Application programs store their settings in registry

• OS needs information during execution on how different parts of the system are connected to each other


Partial configuration file (notepad)From Windows\winsxs\x86_usbstor.inf_31bf3856ad364e35_6.1.7600.20921_none_492329831217353f

[Version]

Signature="$CHICAGO$"

Class=USB

ClassGUID={36FC9E60-C465-11CF-8056-444553540000}

Provider=%MSFT%

DriverVer=06/21/2006,6.1.7600.20921

[ControlFlags]

ExcludeFromSelect = *

BasicDriverOk=*

[SourceDisksNames]

3426=windows cd

[SourceDisksFiles]

usbstor.sys = 3426

[Manufacturer]

; sorted by VID

%Generic.Mfg%=Generic,NTx86

%Mitsumi.Mfg%=Mitsumi,NTx86

%HP.Mfg%=HP,NTx86

%NEC.Mfg%=NEC,NTx86

%IBM.Mfg%=IBM,NTx86

%IOData.Mfg%=IOData,NTx86

%FujiFilm.Mfg%=FujiFilm,NTx86

%ScanLogic.Mfg%=ScanLogic,NTx86

%Panasonic.Mfg%=Panasonic,NTx86

%SCM.Mfg%=SCM,NTx86

%Sony.Mfg%=Sony,NTx86

%YEData.Mfg%=YEData,NTx86

%Iomega.Mfg%=Iomega,NTx86

%LaCie.Mfg%=LaCie,NTx86

%TEAC.Mfg%=TEAC,NTx86

%Hagiwara.Mfg%=Hagiwara,NTx86

%Imation.Mfg%=Imation,NTx86


Registry tree structure

• Registry database is hierarchical– Each node is called a key

• Nodes can have subkeys or data (value)

• Key names are composed of printable characters (excluding the \)

• Key names may not be the same as their parent

• Edit structure thru regedit,


Changing registry data

• Only advanced users should change register data directly (though regedit)– Typically a key value needs to be changed– Error in registry change can require entire

reinstallation

• Updates through API normally change registry data– For example, moving a file to another location


From http://msdn.microsoft.com/en-us/library/microsoft.w

in32.registry.aspx‘ HKEY_USERS is a predefined subkey of the registry;

‘ Retrieve and print subkeys of HKEY_USERS

Imports System

Imports Microsoft.Win32

Class Reg

Public Shared Sub Main()

' Create a RegistryKey, that will access the HKEY_USERS

' key in the registry of this machine.

Dim rk As RegistryKey = Registry.Users

' Print out the keys.

PrintKeys(rk)

End Sub

Shared Sub PrintKeys(rkey As RegistryKey)

' Retrieve all the subkeys for the specified key.

Dim names As String() = rkey.GetSubKeyNames()

‘ etc. --- output subkeys


HKEY_Local_Machine• HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\

StorageDevicePolicies\WriteProtect– Flip a bit in this registry file to make your USB drive write protected

• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SUA

• EnableSetuidBinaries = REG_DWORD 0x00000000

• Change to 0x00000000 to disable setuid

• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SUA

• EnableSetuidBinaries = REG_DWORD 0x00000001

• Change to 0x00000001 to enable setuid


file management chapter 121 file management file: –unit of related information (uniquely) named...

Documents