NIO.2: The I/O API For Future

Masoud KalaliKalali@Gmail.com

Twitter: @MasoudKalalihttp://kalali.me

Agenda

Why NIO.2 What is lacking in the previous version

File System API: The part deals with files as whole Copying a file, Reading a file attributes Watching a directory for changes Etc.

IO API Asynchronous IO Other interesting features.

Java I/O history from Java 1.0 to Java 7

I/O in Java 1.1NIO (JSR-51) in JDK 1.4

NIO.2 (JSR-203) in Java 7

Comparing I/O before Java 7 and Java 7: Basic Features

File System part of the NIO.2 is platform specific

File tree walk

File Operations (Copy, Delete, Move, etc.)

Symbolic links support

Support for file attributes in NIO.2

Support for handling file permissions

What NIO.2 provides

Java 7

All File System were treated the same

No API for walking over the file system

No API for file operations

No support for symbolic links

No support for reading and writing attributes

No support for dealing with permissions

Current lacking IO features

Before Java 7

Comparing I/O before Java 7 and Java 7: Advanced Features

Proper Exception handling

Support for change notification

Provider SPI to add new file system support

Asynchronous I/O

What NIO.2 provides

Java 7

No proper exception handling

No watch API to handle file system changes

No SPI for adding support for new file systems

No Asynchronous I/O support

Current lacking IO features

Before Java 7

The NIO.2 ArchitectureClasses, interfaces reside in java.nioNew set of main classes, interfaces

FileSystems factory, FileSystem interface: The file system doorwayPath: The object representing an existing or non-existing file or directory in the file system. FileStore: Represents underlying storage devices, partitions and so on. WatchService: Watching a directory or a file in the file systemAttributes and related classes: Reading and writing attributesPermissions and related classes: Reading and writing permissionsetc

A FileSystem

A FileSystemsProvider

WatchService

Attibutes

Path

FileStore

Etc

Through FileSystems Factory class

How the NIO.2 works

FileSystem fs = FileSystems.getDefault();Path p = fs.getPath("/home/masoud/simple.txt"); try { p.deleteIfExists(); } catch (IOException ex) { //handle it }

Basically everything start with FileSystems and FileSystem object

Basic File operations with Path

Copying and moving Files

Attributes state after copying / moving; creating files or directories.

Overwriting the target file

Move can be atomic

FileSystem fs = FileSystems.getDefault();Path p = fs.getPath("/home/masoud/simple.txt");Path newCopy = fs.getPath("/home/masoud/copy of simple.txt");

p.copyTo(newCopy, StandardCopyOption.COPY_ATTRIBUTES, StandardCopyOption.REPLACE_EXISTING);

newCopy.moveTo(fs.getPath("/home/masoud/moved.txt"), StandardCopyOption.ATOMIC_MOVE);

Symbolic links support

• What are Symbolic Links and hard links

• Creating and Resolving Symbolic Links

• Creating hard links

FileSystem fs = FileSystems.getDefault();Path actualFile = fs.getPath("/home/masoud/simple");

//creating a symbolic linkPath symLinkFile = fs.getPath("/home/masoud/simple.sym");symLinkFile.createSymbolicLink(actualFile);System.out.println("Getting the link target: "+symLinkFile.readSymbolicLink());

//Creating hard linkPath linkFile = fs.getPath("/home/masoud/simple.link");linkFile.createLink(actualFile);

Walking The File System Tree• Files.walkFileTree * Walks a file tree from a given starting path down to the given depth

* Invoke FileVisitor methods for each file/directory

public class SimpleVisitor {

public static void main(String args[]) {

FileSystem fs = FileSystems.getDefault(); Path p = fs.getPath("/home/masoud/"); MyVisit v = new MyVisit(); Files.walkFileTree(p, EnumSet.allOf(FileVisitOption.class), 2, v); }}

class MyVisit extends SimpleFileVisitor<Path> { @Override public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) { System.out.println(file.getName()); return super.visitFile(file, attrs); }}

PreVisitDirectoryPreVisitDirectoryFailedVisitFileVisitFileFailedpostVisitDirectory

File attributes and metadata• Reading and Writing File attributes

• Different views for different file systems

– Dos View: DosFileAttributeView

– Posix view: PosixFileAttributeView

– ACL view: AclFileAttributeView

– User defined attributes view: UserDefinedFileAttributeView

– Basic view: BasicFileAttributeView: Common for All FS

• Each view has a corresponding *FileAttributes object holding the attributes for a file system entry.

• Checking for supported attributes using FileSystem.supportedFileAttributeViews()

File attributes and metadata: basic attributes

BasicFileAttributes: Common to all FS and contains basic attributes. Creation Time

Last Access TimeModification TimeSizeFile Type: isDirectory,IsFile, IsSymbolicLink, IsOtherFile Key

FileSystem fs = FileSystems.getDefault();Path p = fs.getPath("/home/masoud/simple");

BasicFileAttributeView bav= p.getFileAttributeView(BasicFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);

BasicFileAttributes ba =bav.readAttributes();

System.out.println(p.toString() + " last access: " + ba.lastAccessTime());System.out.println(p.toString() + " last modified: " + ba.lastModifiedTime());

File attributes and metadata: Platform specific Attributes

PosixFileAttributeView Extends BasicFileAttributeView, contain Posix specific attributes. All Basic attributes

Retrieving groupRetrieving ownerRetrieving permission

FileSystem fs = FileSystems.getDefault();Path p = fs.getPath("/home/masoud/photo.png");

Set<String> supportedViews = fs.supportedFileAttributeViews();

if (supportedViews.contains("unix")) { PosixFileAttributes pat = Attributes.readPosixFileAttributes(p, LinkOption.NOFOLLOW_LINKS); System.out.println(pat.group().getName()); System.out.println(pat.owner(). }

basicunixdosaclowneruseretc.

Using Attributes utility class methods: Using Attributes utility class methods: Attributes.readBasicFileAttributes(...);Attributes.readBasicFileAttributes(...); Attributes.readDosFileAttributes(...);Attributes.readDosFileAttributes(...); Attributes.readPosixFileAttributes(...)Attributes.readPosixFileAttributes(...)

Bulk read using Path.readAttributes(...) method: Bulk read using Path.readAttributes(...) method: Map<String, ?> att = (Map<String, String>) Map<String, ?> att = (Map<String, String>)

p.readAttributes("posix:owner,basic:*");p.readAttributes("posix:owner,basic:*"); Using Path and AttributeViews: Path.getFileAttributeView(Class<V> Using Path and AttributeViews: Path.getFileAttributeView(Class<V>

type, LinkOption... options)type, LinkOption... options)

Different ways for retrieving file attributesDifferent ways for retrieving file attributes

Comma separated list of [View name: Attribute name]Default view is Basic View.For example:basic:lastModifiedTimebasic:sizeposix:groupdos:hiddenposix:*fileKey → with no view name

BasicFileAttributeViewDosFileAttributeView

PosixFileAttributeView

File attributes and metadata: Managing POSIX files permissions

• PosixFilePermission: A posix permission

PosixFilePermissions: Utility class to convert permissions from between OS notation and a set of PosixFilePermission

• Attributes: To read file attributes

• PosixFileAttributes: includes posix file permissions

FileSystem fs = FileSystems.getDefault();Path p = fs.getPath("/home/masoud/script.sh");

PosixFileAttributes patts = Attributes.readPosixFileAttributes(p, LinkOption.NOFOLLOW_LINKS);Set<PosixFilePermission> st = patts.permissions();

System.out.println(PosixFilePermissions.toString(st));

//Setting permissionsSet<PosixFilePermission> st = PosixFilePermissions.fromString("rwxrwxrwx");Attributes.setPosixFilePermissions(p, st2);

//Setting the owner username, we should find the user prior to using Attributes utility classUserPrincipal up = s.getUserPrincipalLookupService().lookupPrincipalByName("masoud");Attributes.setOwner(p, up);

//Another way to set the owner name.FileOwnerAttributeView fo = p.getFileAttributeView(FileOwnerAttributeView.class, LinkOption.NOFOLLOW_LINKS);fo.setOwner(up);

FileOwnerView: Deal with file owner attribute

Attributes: To write attributes, PosixFileAttributes includes permissions as well.

File attributes and metadata: Managing POSIX files permissions

FileStore featuresThe FileStore: Represent the file system underlying storage devices, pools, partitions, and so on.

FileSystem fs = FileSystems.getFileSystem(new URI("File:///"));Path p = fs.getPath("/home/masoud");

FileStore fstore = p.getFileStore();

FileStoreSpaceAttributes attrs = Attributes.readFileStoreSpaceAttributes(fstore);long total = attrs.totalSpace() / (1024*1024);long used = (attrs.totalSpace() - attrs.unallocatedSpace()) / (1024*1024);long avail = attrs.usableSpace() /(1024*1024);System.out.println(" device size(MB) used(MB) available(MB) ");System.out.format("%-20s %12d %12d %12d%n", fstore, total, used, avail);

We can use FileSystem.getFileStores() to get all underlying file stores.

FileSystem fs = FileSystems.getDefault();Path p = fs.getPath("/usr/bin");

DirectoryStream.Filter<Path> filter = new DirectoryStream.Filter<Path>() { public boolean accept(Path file) throws IOException { String perm = PosixFilePermissions.toString(Attributes.readPosixFileAttributes(file).permissions());

if (perm.equalsIgnoreCase("rwxr-xr-x")) return true; return false;

} };

DirectoryStream<Path> ds = p.newDirectoryStream(filter);

for (Path p : ds) {System.out.println(p); }

Directory content streamCan filter a directory content by file names using glob syntaxCan filter and walk over a directory content by file attributes. The filter interface has one method we need to implement to accept a file and return it up.Scale to very large directories with tens of thousands of entries

File system change notification• How does it works:

• Use underlying FS facilities when available.

• Retreat to rudimentary polling if underlying FS does not provide change notification.

• API consists of four elements:

• Watchable: Path implements Watchable interface so the watch service works over it.

• A set of event types: whether we want to receive creation, deletion, … events. In our case we will use StandardWatchEventKind which implements the WatchEvent.Kind<T>.

• The Watcher: This is the entity watching watchable. In our case the watcher ( java.nio.file.WatchService) watches the File System for changes.

• Watch Key:

Change notification: Sample Code

FileSystem fs = FileSystems.getDefault();WatchService ws = fs.newWatchService(); Path path = fs.getPath("/home/masoud/Pictures");path.register(ws, StandardWatchEventKind.ENTRY_CREATE,

StandardWatchEventKind.ENTRY_DELETE); WatchKey key = ws.take();List<WatchEvent<?>> events = key.pollEvents();

for (WatchEvent object : events) { if (object.kind() == StandardWatchEventKind.ENTRY_DELETE) { System.out.println("Delete: " + path.toRealPath(true)+"/"+ object.context().toString()); } if (object.kind() == StandardWatchEventKind.ENTRY_CREATE) { System.out.println("Created: " + path.toRealPath(true)+"/"+ object.context().toString()); }}

Provider interface• Plug a new FS implementation just by adding the Jar file to

the runtime classpath.

• The implementation archive must include the implementation descriptor file (META-INF/services/java.nio.file.spi.FileSystemProvider containing the FQN of the implementation class)

• Custom file systems, zip, jar, in memory, etc...

• We just need to implement or extend some contracts including:

» java.nio.file.spi.FileSystemProvider

» java.nio.file.Path

» java.nio.file.DirectoryStream and java.nio.file.DirectoryEntry if we need content iterator.

» java.nio.file.attribute.BasicFileAttributeView and java.nio.file.attribute.BasicFileAttrbutes if we need attributes support.

» Other interfaces or abstract classes like WatchService can be implemented if required.

Blocking, Non-Blocking and Asynchronous I/O

Prior to Java SE 1.4 : Everything was blocking and and use of threads was inevitable. ( which causes scalability issues and performance overhead)

Java SE 1.4: Brings the JSR-51 or NIO with non-blocking IO; Introduction of channels and selectors. Non-Blocking not Asynchronous.

Java SE 7: Addition of asynchronous IO to the pack; included inside java.nio.channels

Asynchronous I/O for sockets and Files

• What is Asynchronous I/O– No blocking, no quick pass over the buffer, simple manual

check backs or call back handlers (completion handlers).

• Why Asynchronous I/O– More scalability, benefiting from underlying OS IO features.

• How Asynchronous I/O works

• Using java.util.concurrent.Future: Manual check backs.

• Using java.nio.channels.CompletionHandler: Callback handler for completion or failure of IO operation

Sample code for Asynchronous I/O: Using Future

FileSystem fs = FileSystems.getDefault(); Path path = fs.getPath ("/home/masoud/a.png"); AsynchronousFileChannel ch = AsynchronousFileChannel.open (path); ByteBuffer buf = ByteBuffer.allocate (2048); Future<Integer> result = ch.read (buf,0); while (!result.isDone ()) { System.out.println ("Doing something else"); Thread.sleep (500); } System.out.println ("Bytes we read: "+result.get ()); ch.close ();

The Future object allows us to check on the status of the asynchronous job when we want to get back at it.

Sample code for Asynchronous I/O: Using Callback

Path path = Paths.get("/home/masoud/series.txt");AsynchronousFileChannel ch = AsynchronousFileChannel.open(path);final ByteBuffer buf = ByteBuffer.allocate(2048);ch.read(buf, 0, buf, new SimpleHandler());

class SimpleHandler implements CompletionHandler<Integer, ByteBuffer> {

public void completed(Integer result, ByteBuffer attachment) { System.out.println(new String(attachment.array())); }

public void failed(Throwable th, ByteBuffer attachment) { System.out.println(th.toString()); }}

interface CompletionHandler<V,A>

Attachment is something to carry a context object into the handler.

Value number of bytes we read or write.

read(ByteBuffer dst, long position, A attachment, CompletionHandler<Integer,? super A> handler)

interface CompletionHandler<V,A>

Attachment is something to carry a context object into the handler.

Value number of bytes we read or write.

Different types of Asynchronous channels

AsynchronousChannel

AsynchronousByteChannel

AsynchronousFileChannel

Asynchronous read/writeNo global file position/offset

Each read/write specifies position in file Access different parts of file concurrently

Open method specifies options: READ, WRITE, APPEND...

AsynchronousSocketChannel

Asynchronous connectAsynchronous read/writeAsynchronous scatter/gather (multiple buffers)Read/write operations support timeout: failed method invoked with timeout exceptionImplements NetworkChannel for binding, setting socket options, etc.

AsynchronousServerSocketChannel

Asynchronous accept handler invoked when connection accepted Result is AsynchronousSocketConnection

Implements NetworkChannel for binding, setting socket options, etc.

AsynchronousDatagramChannel

Asynchronous read/write (connected)•Asynchronous receive/send (unconnected)Implements NetworkChannel

for binding, setting socket options, etc.Implements MulticastChannel

to join multicast groups

Other JSR 203 features

The java.io.File.getPath() method

File type MIME detection using Files.probeContentType

More Information

OpenJDK New I/O Project

– http://openjdk.java.net/projects/nio

Questions?

Thank you

Thank you for joining this session

Masoud Kalali

http://kalali.me

Twitter: @MasoudKalali

Kalali@gmail.com