nfsv4 test project - linux kernel · nfsv4 test project bryce harrington osdl ... around nfsv41 by...

NFSv4 Test Project

Bryce HarringtonOSDL

[email protected]

Aurelien CharbonTony Reix

Vincent RoquetaBull SAS

[email protected]

J. Bruce FieldsCITI

[email protected]

Trond MyklebustNetwork Appliance, Inc.

[email protected]

Suresh JayaramanNovell

[email protected]

Jeff Needle, Barry MarsonRed Hat

[email protected], [email protected]

Abstract

This paper presents the testing effort donearound NFSv41 by the Linux NFSv4 com-munity. As an introduction, we explain therationale for such a heavy testing activity,why NFSv4 was needed, the current status ofNFSv4, and some Use Cases. Chapter 3 de-scribes the tools used for testing integral fea-tures of NFSv4 in the areas of functionality, in-teroperability, robustness, performance, and se-curity: where they come from, and which partsof NFSv4 they are aimed to test. We also de-scribe some tools used for analyzing problemsand loads. Chapter 4 first explains the goalsof the NFSv4 testing team and how contrib-utors are working together. Major events forNFSv4 since January 2004 are displayed in a

1Network File System version 4.

developmental timeline. Then, four contribu-tors (OSDL, Bull, Novell, Red Hat) amongstmany others describe in details their NFSv4testing activity, explaining what they have al-ready done and what their future plans are.OSDL and Bull are contributing to the contin-uous testing activity of fresh kernel+CITI ver-sions, though Novell and Red Hat test NFSv4 inthe eco-system of their distributions. As a con-clusion, the paper shows that the testing effortshave generated significant improvements in allthe test areas and that the core of Linux NFSv4is stable and powerful. Also, some ideas arepresented about the future of NFSv4 protocoland of Linux NFSv4.

268 • NFSv4 Test Project

1 Introduction

NFS Version 4 adds a number of powerful newfeatures to address NFS shortcomings in secu-rity, migration, performance and other areas. Inorder to make NFSv4 the new industry standardfor Linux, these features must be thoroughlyand frequently tested to ensure they are func-tional, robust, efficient, and secure. The goalsfor testing NFSv4 on Linux are to make it morestable, more mature, more interoperable withother NFS implementations, and to improve theentire ecosystem of software that interacts withNFS.

NFSv4 for Linux has been under developmentat CITI and NetApp since 2001. This LinuxNFSv4 testing task force started in 2004 withparticipants from OSDL, Bull, IBM, NetApp,Novell, and Red Hat, plus many other compa-nies and individual contributors.

Because NFSv4 is a complex and critical in-frastructure service, the testing is both very im-portant and very challenging. Our key strategyin achieving our goals has been a large and col-laborative task force that focuses on differenttesting approaches, sharing results and workingdirectly with the developers to validate fixes.

Development of Linux NFSv4 follows theOpen Source Release Early, Release Oftenmodel. This means that new features for theLinux implementation of the NFSv4 protocolbecome available in the mainline kernel as soonas they’re ready. Ultimately, the new featureswill enable or enhance a number of Use-Casesincluding high performance computing clus-ters, large scale render farms, massive corpo-rate provisioning, and secure Intranet and In-ternet file sharing.

OSDL’s role is to facilitate this testing throughestablishing a testing community around LinuxNFSv4. Initially, this involved planning ac-tivities such as collaborating with stakeholders

in creation of a Test Matrix/Wiki itemizing andprioritizing testing needs, and in providing op-portunities for members of the community tomeet and collaborate. Recently, OSDL’s activi-ties focus on participating in designing and run-ning tests for regression and installation testing,and for checking configuration robustness.

Terminology

Before getting into details about testing it isuseful to clearly define the terminology wehave adopted in testing NFSv4.

First, one may want to check that NFSv4 fea-tures work as they have been designed for. Usu-ally each function is tested separately to coverthe whole function set. It is: functional testing.

Second is interoperability testing, which in-volves comparing the Linux NFSv4 implemen-tation against other implementations, as wellas reviewing how Linux NFSv4 interacts withother components in the system.

Then, one may want to check that NFSv4 stillcontinues to work under high load, often withrandom and simultaneous operations. Thisaims at generating extreme cases, not reachablewith functional tests, to stress the system. It is:robustness testing.

Since many people need to know how manyclients can be connected to a NFSv4 server,one must measure how long NFSv4 can de-liver some service or how many actions can bemanaged in parallel and in a defined period oftime: performance testing. Performance test-ing consists in analyzing figures (speed, time,CPU load, Memory load, etc.) and in trying todetermine where the bottlenecks are.

Security testing may have different meanings.In the current case the goal of testing NFSv4security is not to test that security tools usedby NFSv4 (like Kerberos) work well. Instead,

2006 Linux Symposium, Volume Two • 269

5/04

6/04

7/04

8/04

9/04

10/04

11/04

12/04

1/05

2/05

3/05

4/05

5/05

6/05

7/05

8/05

9/05

10/05

11/05

12/05

1/06

2/06

3/06

50

75

100

125

150

175

200

225

250

275

300

325

350

375

400Ma il ing L ists

#em ail s on NFS v4 mai l in g lis t

NFS ma il i ng l i st

Figure 1: nfsv4 and nfs mailing list activity.

NFSv4 security testing first checks that NFSv4still works fine under a security environmentlike Kerberos, and then it checks that NFSv4server does not reduce the security of Linux byopening security holes malicious or dangerouspeople could use.

nfsv4 and nfs mailing lists

Problems dealing with Linux CITI NFSv4 arediscussed on the [email protected] list, though general Linux NFSdiscussions about development and in-teroperability are discussed on [email protected] mailinglist. Figure 1 shows the activity of these 2mailing lists since may 2004: nfsv4 mailinglist (in black), which has more than twohundred different participants, now has aboutthe same activity than the nfs mailing list (inred). As expected, the most talkative peopleon nfs4 mailing list are: Bruce Fields, BryceHarrington, Trond Myklebust, Kevin Coffman,and then Vincent Roqueta.

2 NFSv4 Description

2.1 Why NFSv4 ?What is new in NFSv4 ?

NFSv4 brings a number of improvements toNFS.

NFSv2 and NFSv3 do not themselves in-clude file locking or ACL2 management; in-stead, those functions are performed by sepa-rate RPC3 protocols. In addition, a mount pro-tocol is required to obtain the root file-handle ofan exported filesystem. Thus four distinct RPCprotocols are required for full functionality.

NFSv4 integrates all of these into the same pro-tocol, simplifying firewall management and en-abling previously unsupportable features, suchas mandatory file locking.

For security, NFS implementations have tradi-tionally relied on private networks and locked-down clients. The rpcsec_gss protocoladds support for cryptographic security usingper-user credentials, thus eliminating the needto trust every host on the network. WhileNFSv2 and NFSv3 can also take advantage ofrpcsec_gss, NFSv4 is the first to requireimplementation (not use) of rpcsec_gss,and NFSv4 integrates it into the protocol morethoroughly.

NFSv4 also allows servers to hand out delega-tions to clients, giving the client shared (read-only) or exclusive (read and write) access to thefile, for improved caching.

NFSv4 provides some support for filesys-tem replication and migration with a newfs_locations attribute that clients can use tofind other servers exporting the same filesystemdata.

2Access Control List.3Remote Procedure Call.


NFSv4 enables better support for WindowsAPIs with open share locks and a fine-grainedACL model based on Windows.

The last two NFSv4 features are more subtle,but together add an important layer of extensi-bility.

First, NFSv4 brings in a mechanism for in-troducing incremental updates to the protocol,called minor versions. Draft specifications andprototypes for minor version 4.1 are currentlyavailable; see chapter 5 (Future of NFSv4) onpage 284 for details.

Second, NFSv4 operations are now sent as se-ries of smaller operations, called compoundRPCs. For example, what an NFSv3 clientwould do with a single NFSv3 WRITE opera-tion might be accomplished by an NFSv4 clientusing a compound RPC consisting of the threeoperations: PUTFH (to indicate which filethe following operations apply to), WRITE,and GETATTR (to update the client’s attributecache). The compound RPC adds flexibilityto the protocol, especially when extending it,since new operations can combine with exist-ing operations in interesting ways.

Finally, while NFS has always been a freelydocumented and widely implemented protocol,previous protocol specifications have been thework of Sun Microsystems. NFSv4 is the firstversion that is actually developed within theIETF4 and hence whose development is alsoopen. In practice we have seen wide and fruit-ful participation in the process.

2.2 Status of NFSv4 on Linux

The Linux NFSv4 client and server imple-mentation supports all of the basic features ofNFSv4. In more detail:

4Internet Engineering Task Force.

Delegations are supported, and the client willtake advantage of a file delegation where pos-sible to perform opens and closes withoutcontacting the server. Work is underway toprovide even more aggressive caching on theclient, if desired, using on-disk data caching.The server implements delegations using leaseson the exported filesystem, allowing it to coop-erate correctly with local and Samba users.

File locking is supported, and locks can be han-dled entirely on the client when delegations al-low.

ACLs are supported, though client-side ACL-manipulation tools are still under development,and server-side support is limited by the need tostore NFSv4 ACLs as less fine-grained POSIX5

ACLs.

Kerberos-based security is fully implemented,and support for the public-key based SPKM36

and LIPKEY7 mechanisms is under develop-ment.

We have patches implementing preliminarysupport for replication and migration; furtherwork needs to be done to refine them and inte-grate them into mainline.

Ongoing development includes stabilizationand tuning. We are also interested in improv-ing NFS (especially NFSv4) support for clusterfilesystem exports; for example, we need to en-sure that locks acquired by an NFS server onone node of a cluster can be enforced by anNFS server on another node.

Server user interfaces are under revision to pro-vide a more consistent interface for users up-grading from NFSv2 and NFSv3.

5Portable Operating System Interface6Simple Public-Key Mechanism.7Low Infrastructure Public Key.


2.3 Use Cases

The new features in the NFSv4 protocol are in-tended to improve performance and reliabilityfor proved usage scenarios. While we cannotenumerate every possible use, for testing pur-poses we’ve identified several distinct use caseswhere NFSv4 would be expected to show ben-efit over previous versions.

Scientific computing cluster. Laboratories useNFS to communicate between the nodes in alarge computational cluster. This usage sce-nario involves days or weeks of quiescence,with occasional bursts of heavy read activity,followed by intensive write operations. Thisuse case will benefit from robust network re-covery and good write performance.

Render farms. This use case is in a way theinverse of the scientific cluster. Render farmsalso involve a large number of computationalnodes, but the write operations are more con-tinuous over time, punctuated by intensive readoperations. NFSv4’s delegation and cachingimprovements may be the biggest benefits inthis use case.

Provisioning. A number of large users use net-work filesystems for deployment of software orsoftware updates, either to large server installa-tions or to large workstation deployments. Ineither case, issues include congestion control(such as if many machines attempt to accessserver resources simultaneously), access con-trol, and migration and replication.

Databases. Use of Network Attached Stor-age (NAS) and similar technologies often re-sults in designs that place a database back-endon an NFS share. This usage provides benefitsincluding backup/rollback and flexible volumemanagement, but can raise performance con-cerns. New features of NFSv4 worth exploringwith this use case are delegations and cachingimprovements.

3 Testing Tools

Stability and robustness:

First and foremost, NFSv4 acts as a filesys-tem. So, the main priority in NFSv4 testing ischecking that the filesystem is stable and ro-bust. Many generic, Open Source filesystemtest tools are available to perform such tests;many of them, including several listed below,are part of the LTP8 [10].

IOZone [9] was originally a performancetool. Developed by IOZONE.ORG, OR-ACLE and HEWLETT PACKARD, it mea-sures raw throughput of file operationssuch as read, write, reread or rewrite.These throughputs are measured with var-ious file sizes and read/write sizes. A typ-ical IOZone test has a total of 1430 mea-surements. IOZone allows mounting andunmounting filesystems with various pa-rameters between tests. It can be used tostress mount program with various param-eters.

Fsx [10] is an APPLE COMPUTERS in-file stress program available in LTP. Itperforms the following file operations :mapped read, mapped write, read, write,truncate. FSX checks if data corruptionoccurred during these operations.

Fsstress is FSX’s complement, and is alsoavailable in LTP. It was written by SILI-CON GRAPHICS INC.. It stresses filesys-tem tree structure by doing random oper-ations on the tree structure: file creation,recursive directories, symlinks manipula-tions.

Locks tests [10] launch multiple processes onmultiple clients. This tool was designed

8Linux Test Project.


by BULL SAS to stress NFSv4 locks, andcontributed to LTP. Processes try to per-form locks-related operations on the samefile or file section. Results are comparedto the expected results. It can be used tostress both network and local filesystems.

ACL tests [10] were written by BULL SAS inorder to stress ACLs within NFSv4, andare available in LTP. The test suite createsnumerous users and ACL rules and com-bines them; then it checks that actual ac-cesses match ACL rules.

Connectathon 2004 [8] was designed by SUN

MICROSYSTEMS, INC to perform interop-erability testing of critical operations. Itperforms high-level operations that oftenreveals interoperability troubles.

FFSB [12] is a versatile and useful filesys-tem test. It was created and enhancedfor NFSv4 by IBM. It can both stress thewhole filesystem, mimic various load pro-files and collect test information.

NetEm [13] is a kernel component developedby Steve Hemminger at OSDL (availablewhen configuring the kernel) that allowsto modify network behavior. It providesthe following features:

• dynamic delay between packets(RTT).

• packet loss, duplication, corruption,re-ordering, collisions.

• rate control, and non-FIFO queuing.

It can be easily configured to mimic thebehavior of real networks.

NewPyNFS [14] is a Python-based test-suitedeveloped and maintained by the CEN-TER FOR INFORMATION TECHNOLOGY

INTEGRATION (CITI) of the Universityof Michigan. Unlike the tests described

above, it is not a black box test tool: itimplements a python NFSv4 client andserver. It was specifically designed to testNFSv4 features, including ones that arenot yet fully implemented.

Analysis tools

OProfile is a kernel module used to collectstatistics of CPU load by function.

Ethereal [11] is a well known network ana-lyzer. It helps checking that NFSv4 serverand clients exchange valid sequences ofinformation over the network.

Needed tools

Since new NFSv4 features should appear soonwithin Linux NFSv4, appropriate new tests willbe needed.

Migration and Replication is a relatively newfeature, and a functional/robustness test will benecessary. This test would emulate or demon-strate the transfer of an NFSv4 share from oneNFS server to another, and it would check thatthe client is able to continue accessing the dataseamlessly, while measuring the impact on theclient during the transition.

New tests will be needed to exercise full NFSv4ACLs, Named Attributes, and Directory Dele-gation.

We also need tests of the security of NFSv4.We need to measure the impact on performanceof running NFSv4 with security and evaluatethe robustness of NFSv4 when attacked.


4 NFSv4 Tests

Hereafter we present the testing effort done byfour contributors: OSDL, Bull, Novell and RedHat. Many other companies and people havecontributed to improve NFSv4, testing it, pro-viding patches, warning about mistakes, or pro-viding information about oops and bugs theyare experiencing in their labs.

NFSv4 is being tested in three different ways:1) OSDL and Bull have developed tests and areusing them plus others (filesystem tests, LTP)to continuously check that no regression oc-curs; 2) many contributors or early adopters areusing NFSv4 in their own complex and specificenvironment; 3) Novell and Red Hat are hard-ening NFSv4 in the environment of their futuredistributions by using regression tests. Thesedifferent approaches are complementary. Re-gression and stress tests enable to verify thatNFSv4 core is reliable in a clean and standardenvironment. And tests done in various andunique environments enable to check that thewhole NFSv4 ecosystem is robust and to cleanNFSv4 of all little mistakes in its childhood.

The timeline on page 274 presents the evolu-tion of NFSv4 since beginning of 2004. Ver-tical bars on the right show how main featuresof NFSv4 have evolved, from bad (red) to good(green). Main events appear in the middle. No-tice that regressions appear from time to timeand are quickly fixed. Also notice how long ittook to make locks reliable.

4.1 OSDL

The Open Source Development Labs (OSDL)[15] became involved in the NFSv4 testing ef-fort at the request of the NFSv4 community andthrough OSDL’s Data Center Linux (DCL) ini-tiative. Initial involvement included assisting in

organizing efforts, identifying test plans, estab-lishing testing priorities, and facilitating discus-sion between companies and community mem-bers. Today OSDL’s role is in conducting re-gression testing of all kernel patch releases bythe NFSv4 community, and ancillary activitiesto help facilitate and promote use of NFSv4.

The test matrix [16] is a listing of test tasksthat were felt to be needed to fully test theLinux NFSv4 system, broken down into thefollowing categories: Functional, Robustness,Performance, Interoperability, and Security.Through discussions with testers and devel-opers, these tasks were prioritized, and anNFSv4 Testing Road-map was generated, item-izing the high priority testing tasks and iden-tifying which organizations will be perform-ing which tasks. OSDL signed up for severalFunctional/Robustness testing tasks involvingregression testing and cross-compile testing.

Cross-compile testing is done on every ker-nel patch released by CITI using the PatchLifecycle Manager (PLM). These builds targeta number of different architectures, includingi386, x86_64, ppc, ppc64, sparc, sparc64, arm,and alpha. Compiles are performed againstseveral different configs, including allyescon-fig, allmodconfig, allnoconfig, and defconfig.Sparse has also recently been added. Thesebuilds have been useful in identifying both is-sues particular to certain platforms (such asonly the ppc architecture, or only 64-bit sys-tems), as well as variation in config settings.For example, the developer may only be check-ing his work with a given config setting turnedon, and may have accidentally added an is-sue that only shows up with that config settingturned off; this cross compile system will havea better chance of catching these classes of de-fects.

Regression testing is done using Crucible,a framework to automatically downloadnew patches from CITI and kernels from


1

F

M

A

M

J

J

A

S

O

N

D

J2004

F

M

A

M

J

J

A

S

O

N

D

J2005

F

M

J2006

2.6.3

2.6.7

2.6.9rc1

2.6.12r2

2.6.14

2.6.15rc3

2.6.16rc

²

KERB

EROS

5 SU

PPOR

TBA

SIC

NFSv

4 FUN

CTIO

NS

INTE

ROPE

RABI

LITY

LOCK

SUP

PORT

vi can edit a file nfsd as module

2.6.6

VFS: Busy inodes after unmount

2.6.11r5

IPv6 Client, version 1

various issues

64K pagescorruptions

gss regressions

Bull joins NFSv4 team

NFSv4 gets its own mailing list

First usertesting krb5

Much more people testing with krb5OSDL Test Matrix

OSDL joinsNFSv4 team

core NFSv4 is stableAdministration tools

Netem introduced to help Wan testing

Starting locks validation

Locking, Interoperability Stable releaseMigration guide

NFSv4 wiki

IRC Chanel

NFSv4 ready forearly adopters

rpcsec_gss : first patches IPv6 client work start

starting to work on administration Test tool selection First user

asking for krb5

Basic NFS4 ACL definitions POSIX server translations

Start NFSv4 security negotiation implementation

ACL_support attribute

svc authentication heavy changes

server reboot recovery

reboot recovery IPv6 Client

3264bits Interoperability

NFSv4 limits

Fix file creation limitNew IPv6 patch set

Unix interoperability 2048+ connections

by server

x86_64 supportStarting WAN testing

ACL limits

NFS use FS_Cache

NFSD delegation

Introducing spkm& IPv6 server

Figure 2: Linux/NFSv4 History.


1/05

2/05

3/05

4/05

5/05

6/05

7/05

8/05

9/05

10

/05

11

/05

12

/05

1/06

2/06

0

1

2

3

4

5

6

7

8

0

4

8

12

16

20

7 7 7

23

7

2

0

2

4

10

3

7

Improvement in Defect Density

Kernel Releases

Issues per Test Run

Def

ects

per

test

run

Figure 3: Defects.

kernel.org. It then patches and compilesthe kernel on a client and a server, boots themto that kernel, and then runs a sequence of tests(cthon04, NewPyNFS, IOZone, and LTP) onthem. The results are collected, parsed, andanalyzed for abnormalities or other unusual be-haviors. These are reported to the developers,and efforts are taken to identify the root causeswhere they are not obvious.

1/0

5

2/0

5

3/0

5

4/0

5

5/0

5

6/0

5

7/0

5

8/0

5

9/0

5

10/05

11/05

12/05

1/0

6

2/0

6

0

5

10

15

20

25

30

35

40

Regression Test Runs

Total Test Runs

Issues Found

Figure 4: Regression Tests.

Typically, most issues the regression testing

finds is via the NewPyNFS test, such aschanges in return codes from functions affectedby recent development changes. In some casesthese identify legitimate defects, but in othercases they simply indicate areas where differ-ent people have interpreted the spec in differentways; even this is useful because it identifiesareas where further discussions about the specare needed, to resolve what should happen. Inthis latter case it is not uncommon for the testsuite to require modification to reflect the newconsensus.

The biggest challenges in establishing the au-tomated framework is boot control. Invariablythe client or the server will hang. It is nec-essary to use a watchdog process to automat-ically detect that the machine has become in-active (such as failure to respond to pings at atime when it should respond), and then performa remote power cycle on it. As well, there is al-ways a chance that a given kernel will not boot;to account for this situation, the boot-loader’sdefault kernel is kept to a static, known-goodkernel, and the kernel-to-test is specified vialilo -R.

The hardware included in the OSDL test frame-work is primarily x86 based systems runningGentoo Linux, but also includes a NetApp filer,an amd64, a ppc64, and an itanium 2. Otherhardware may be added in the future, depend-ing on donations to the lab. The principle chal-lenge to integrating new hardware is automat-ing boot control; the system must support bothsome form of remote power management (inworst case, through use of a separate interrupt-ible power unit), and a boot-loader mechanismto test a new kernel and fallback to a knowngood one on next reboot. Serial console accessand/or logging is also important for catchingconsole errors.

OSDL future work


Due to the success seen through use of the re-gression test framework, it is expected that thiswill be expanded with more tests, more hard-ware, and more ways to put tests into complexconfigurations. For example, to date the auto-mated boot mechanisms have only been usedbetween tests to reset the system, however itcould be invaluable to boot the server or clientduring a test, and double-check how the systemas a whole responds. Indeed, there is test codein both LTP and NewPyNFS intended to checkperformance through reboots; these test casesare not typically run, for obvious reasons, sothis framework could enable us to increase ourtesting coverage to these areas.

Building on this, network stability testing canbe done by introducing perturbations in the net-work such as network partitions, dropped orcorrupted packets, and so forth.

To help promote the advantages of adoptingNFSv4, it would be worthwhile to add someperformance comparison capabilities to the testharness. One idea is to simply perform timedkernel builds over NFSv3 and NFSv4 mounts,and compare performance. Another idea isto create a 3D graphics render-farm using aserver and multiple clients using POV-Ray andthe POV-Any software, and to time the per-formance of distributed renders. Ideally, theseshould illustrate how NFSv4’s delegations andother performance features impact real worldworkloads.

4.2 Bull

Bull’s contribution to the Linux NFSv4 projectstarted in January of 2004.

When problems are found, either we directlyexpose them to the NFSv4 developers or wetalk on the nfs4 mailing list or we open a newbugzilla ticket, depending on the complexity ofeach problem.

Each time a task is finalized (like: regressiontests on last kernel-CITI version, performancemeasures, . . . ) we publish a News on our web-site [21].

We are using a limited number of machines:four ia32 machines with two processors usedboth for Linux testing and for Solaris interop-erability, and two PowerPC machines used bothfor Linux testing and for AIX interoperability.One ia32 machine is a x86_64 machine and isused for 64bits testing, complementing testingwith AIX 64bits. All machines are connectedwith GigaBits boards and switches, for perfor-mance measurement purposes. Machines areinstalled with different Linux distributions (Fe-doraCore from Red Hat, SLES from Novell,and Debian).

4.2.1 Regression tests

Each release of the kernel and of the CITIpatchset is exercised with tests in order to de-tect regressions in stability, robustness and per-formance areas. In most cases, bugs are theresult of new patches. The goal is to ensurethat these bugs will be corrected in the nextCITI_NFS_ALL patch.

The following tests are run, using differenttools:

• Connectathon 04 testing suite: do basicconformance testing.

• Two hours FSSTRESS and FSX tests:check robustness.

• IOZone: compare performance with pre-vious versions.

• LTP Locks and ACL tests.


Once a problem is discovered, a new bug re-port is created in the Linux NFSv4 Bugzilla[18]. Then, Linux NFSv4 developers at CITIare warned directly or via the nfs4 mailinglist. They are provided with a stack trace ifa kernel Oops occurred or a network trace inother cases.

Tests are run with two different underlying lo-cal filesystems: ext3 and ReiserFS.

Regression tests are also used to detect interop-erability issues. The tests are performed on dif-ferent Linux platforms (ia32, x86_64 and ppc)and with non-Linux client or server.

4.2.2 Stress and robustness

As a Network File System, NFSv4 providestwo types of functions and mechanisms:

Filesystem functions. Since NFSv4 mimicsthe behavior of a local filesystem to appli-cations, it provides common filesystem re-lated functions, such as read, write, open,mkdir . . . but also more advanced func-tions such as fcntl, flock, acl . . .

NFSv4 specific functions. NFSv4 tries toappear as a local filesystem, but it isnot. It provides several functions andmechanisms over the network, and morespecifically over Internet. These functionsinclude: gss support, delegation, auto-mounter, security negotiation, migration,replication . . .

In order to ensure the stability and robustnessof NFSv4, all its functions have to be stressed.

Core filesystem functions

The first step when testing NFSv4 deals withchecking its stability on the main architec-ture: ia32. Several tests are performed on coreNFSv4 functions. These functions are commonto all filesystems. They provide basic inter-faces for in-file manipulation functions (open,read, write, . . . ) and filesystem tree manipula-tion functions (mkdir, ls, ln).

The first goal of testing is to ensure that no cor-ruption of data occurs when manipulating files.The FSX stress tool is quickly run, and it mustbe successful. It is used to prove that LinuxNFSv4 does not corrupt data.

The second goal of testing is to ensure that filemanipulations are correctly handled, especiallywhen using very long path or very deep sub-treestructures and sub-directories with multipleprocesses. When we started to use FSSTRESS

it helped to reveal numerous problems in manyareas (symlink overflow, deadlock or memoryleak) that are reliable now in the current ver-sions of Linux NFSv4.

One year after we started using FSSTRESS (inApril 2005) Linux NFSv4 was able to sustainthe concurrent load of 10 processes during 24hours, without any problem. Three monthslater, NFSv4 reached 72 hours of stress underFSSTRESS, without any bugs. From this date,NFSv4 filesystem tree manipulation is consid-ered to be stable.

Locks

When we started to stress lock features in early2005, there was only one tool available to testlocks: Connectathon. We have used Connec-tathon during some months to stabilize NFSv4locks implementation. It helped to find manybugs on several architectures.


However, Connectathon was not designed toperform heavy stress operations. So, a new testtool was required, that we designed and imag-inatively named: LockTester. This lock testlaunches an arbitrary number of processes onone or more clients to heavily stress the NFSv4server and client. Processes try to perform var-ious lock-related functions on the file at thesame time. This tool has helped to find severalcomplex bugs, and it is recommended to useit when running regression tests. It has beensuccessfully integrated into the LTP suite, innetwork/nfsv4/locks branch (however,it can be used with any filesystem).

By the end of 2005, one NFSv4 server wasable to manage more than 500 local concurrentprocesses, and more than 2000 concurrent pro-cesses launched from four machines (x86, ppc).

High Load

In order to over-stress NFSv4, we have used upto 2048 IOZone processes running on two ma-chines and concurrently loading NFSv4 on a re-cent kernel: no problem was revealed.

4.2.3 ACLs

NFSv4 defines a flavor of Access Control Lists(ACLs) resembling Windows NT ACLs, de-scribed in an IETF Internet-Draft [5]. A num-ber of operating systems use a different fla-vor of ACL based on a POSIX draft. NFSv4clients and servers on such operating systemsmay wish to map between NFSv4 ACLs andtheir native ACLs. Interoperability tests aim atverifying this mapping.

An ACL test suite built with python scriptsand C programs has been added to the LinuxTest Project tree. It is now available in thenetwork/nfsv4 branch.

It aims at to test the following points:

• ACL conformance: verify that actual ac-cess conforms to the access control list ofthe file. It includes conformance testing ofACLs on files and directories, but also ondefault directory ACLs.

• ACL robustness: multiple clients stressone server with random ACL requests onone single file, or on multiple files.

• ACL limits: determine the maximumlength of an ACL. ACL limits tests havebeen run with Linux, Solaris 10 and AIX5.3 on server and client sides: no interop-erability issues have been found.

The tests are delivered with tools that help man-aging the thousands users needed by the tests.

Tests have been run with different underlyingfilesystems: ext3, xfs and ReiserFS.

Main problems found:

The tests have shown that the main current limi-tation is that the server does not allow the clientto retrieve an ACL greater than one memorypage, due to the underlying RPC. So, whenthe name of users and groups appearing in theACLs are 6 characters long, the limit size isabout 35 ACL entries.

ACL Interoperability tests:

There are now three ACL models to deal with:NFSv4, Windows, and "POSIX ACLs"/modebits. And one must decide what to do with themall in the face of existing users, tools, and sys-tem interfaces that assume one or the other. For


example: since individual clients and applica-tions with different ACL models may not dealwell with the full generality of NFSv4 ACLs,problems may also arise from clients readingand modifying ACLs written by clients withdifferent expectations.

So it is useful to run these ACL tests as longas the ACLs’ implementation in NFSv4 code isunder development.

Remaining tests:

Interoperability tests with Windows filesystemsneed to be performed. Also, we have to developnew tests that enable the testing of all the fea-tures of NFSv4 ACLs, regardless of the under-lying filesystem.

4.2.4 Performance

Most NFSv4 performance measures have beendone with IOZone, which is designed to mea-sure a global throughput on a filesystem.

These IOZone tests, combined with vmstat,have been useful to detect performance prob-lems as well as functional ones, such as wronguse of Kerberos 5.

NFSv4 compared to NFSv3 & Samba

NFSv4 (TCP) has been compared to other wellknown network filesystems: NFSv3 (UDP) andSamba.

Read performance

Figure 5 shows Read performance of NFSv3,NFSv4 and Samba 3.

For large files (greater than 4 MegaBytes) andboth in asynchronous and synchronous (nocache is used) modes, NFSv4 (red and purplelines) and NFSv3 (green and sky blue lines)have similar performance.

For small files (smaller than 4 MegaBytes) andboth in asynchronous and synchronous modes,NFSv4 outperforms NFSv3.

For small files, NFSv4 performance is between2 and 15 times better than Samba (cobalt blueline). For large files, NFSv4 is over 15 timesbetter than Samba.

Write performance

Figure 6 shows Write performance of NFSv3,NFSv4 and Samba 3.

In asynchronous mode with small files,NFSv4 (red line) is about 6 times faster thanNFSv3 (green line) and 3 times faster thanSamba (cobalt blue line). The cache used byNFSv4 is very efficient for small files.

For large files, NFSv4 (purple line) and NFSv3(sky blue line) have similar performance.

In synchronous mode with small files, the per-formance of NFSv4 and NFSv3 is between onethird and one half of the performance of Samba.For large files, NFS is about 20% faster thanSamba. NFSv4 and NFSv3 show the same per-formance.

Conclusions

While NFSv4 is still being developed, its per-formance is similar to NFSv3 performance andit outperforms Samba. For small files, NFSv4performance clearly outperforms NFSv3.


20000

30000

40000

50000

60000

70000

80000

90000

100000

110000

64 256 1024 4096 16384 65536 262144 1.04858e+06

Kbyt

es/s

ec

File size in KBytes

Iozone performance

read nfs4_asyncread nfs3_async

read samba_3read nfs4_syncread nfs3_sync

Figure 5: READ: NFSv4 vs NFSv3 (synchronous and asynchronous modes) & Samba3.

0

10000

20000

30000

40000

50000

60000

70000

80000

64 256 1024 4096 16384 65536 262144 1.04858e+06

Kbyt

es/s

ec

File size in KBytes

IOZone performance

write nfs4_asyncwrite nfs3_async

write samba_3write nfs4_syncwrite nfs3_sync

Figure 6: WRITE: NFSv4 vs NFSv3 (synchronous and asynchronous modes) & Samba3.


Configuration of Tests

• Network: GigaBit Ethernet

• Machines (client and server): dual ia32machines

• Kernel: 2.6.15

• Test performed: IOZone standard tests(-ace -r 32 -U)

4.2.5 Interoperability testing

Hardware interoperability

Since Linux NFSv4 will be used on different ar-chitectures, it is worth checking early if all fea-tures of Linux NFSv4 run perfectly on them.We focus on issues generated by 32/64 bitsalignment and little/big endian problems. Testsare done on Intel x86, Intel/AMD x86_64, andIBM ppc64 architectures, used either as NSFv4server or NFSv4 client. A couple of bugs re-lated to these two kinds of problems have al-ready been found and fixed.

These tests deal only with basic features ofLinux NFSv4.

Table 1 shows the status of the interoperabil-ity tests when run with 2.6.12 kernel. Sev-eral problems appeared when running Connec-tathon 04, showing that Linux NFSv4 was notready for use on 64bits platforms with kernel2.6.12.

Now, starting with 2.6.15 kernel, all these is-sues have been fixed; and these interoperabilitytests are now included in our regression testingprocess.

ClientServer ia32 x86_64 ppc64ia32 OK (1) OK

x86_64 OK OK (2)ppc64 (3) (4) (3)

(1) On x86_64 platforms, lot of socket resets.(2) Input/output error: cannot close a big file.(3) On ppc64 client, locking does not deliver the ex-pected behavior.(4) Socket: error -11 when doing write/read opera-tions on 30 MB files.

Table 1: Interoperability testing result matrixon 2.6.12.

Software interoperability

Tests are run to detect problems between Linuximplementation of NFSv4 and other Unix im-plementations. Since kernel 2.6.12, no prob-lem has appeared about basic features of LinuxNFSv4 (Client or Server) when interoperatingwith NFSv4 on Solaris 10 and AIX 5.3.

Future testing

Tests involving security have not been per-formed yet with different architectures and withother non-Linux Operating Systems. First teststo be run should use: Kerberos (krb, krb5i,krb5p) and SPKM.

4.2.6 WAN testing

NFSv4 has been designed to work over LAN9

as well as over Internet. So we have run teststo stress Linux NFSv4 over WAN10. We usedthe same stress tools we used for LAN testing:

9Local Area Network.10Wide Area Network.


Fsx, and Fsstress. But the testing process overWAN differs in the hardware environment.

Tests have been run between the CITI inMichigan and the labs of BULL SAS in Greno-ble, France. Though these tests appeared tobe very useful because several problem werefound, setting up and configuring machines fortests was painful: patching and updating thekernel is not easy through Internet.

To help us, the NETEM tool has been deployedto emulate the behavior of Internet: high RTT11

delay, high RTT variations and packets loss.This test environment has successfully helpedus to find WAN-only bugs, like timeouts. Then,the new kernel containing appropriate fixes hasbeen tested over the real Internet connection,between USA and France, and proved reliable.

Many problems have been fixed and WAN test-ing covers most core NFSv4 functions.

4.2.7 Future plans

Once NFSv4 is widely used, people will proba-bly ask for proof that a NFSv4 server connectedto the WAN cannot be used as a mean for pene-trating a private network. After a first attempt inend of 2005 to analyze the Linux NFSv4 codeby means of tools like Duma or Checker, weplan to start a more ambitious study: attacka Linux NFSv4 server with a modified LinuxNFSv4 client in order to search for weaknessesin the Linux NFSv4 code, like overflows com-monly used by attackers.

In the second half of 2006, we plan to con-tinue NFSv4 regression testing, in order to con-tinue finding problems in new kernel+NFSv4versions as early as possible.

Running tests helps to improve the quality ofthe newest versions of NFSv4. It is also very

11Round Trip Time.

important to deliver new test suites which willbe used by Linux distributions for checkingthat NFSv4 behaves perfectly well in the eco-system of their distribution. So we plan to con-tinue writing tests for new features (like repli-cation and migration, NFSv4 ACLs, named at-tributes) and to deliver them to the LTP project,as we did for ACLs and Locks tests in 2005.

4.3 Novell

NFSv4 support is available in SLES 10 by de-fault. Novell started contributing to the testingefforts by taking part in OSDL Testing confer-ence calls and providing inputs. Later we de-rived the test plan from OSDL test matrix andstarted testing.

All SLES 10 beta builds are being validated.The validation sequence is:

• Pynfs/NewPyNFS for protocol confor-mance,

• Connectathon 04 for basic conformancetesting,

• Custom usability script which tests areaswhich are not covered by Connectathon,

• Support for security modes (sys, krb5,krb5i),

• Lock tests.

The major focus is on functionality, robust-ness, interoperability and performance. Proto-col conformance, POSIX conformance, instal-lability, integration testing, use case scenarios,and kerberos security modes were tested as partof functionality testing. Stress testing, com-parison against NFSv3, various security modes,and various load scenarios were done as partof performance testing. Robustness Testing in-corporates fsstress, ffsb, resource limit testing


and crash recovery. The interoperability testingincludes NetApp and Solaris platforms and allSLES 10 supported architectures.

Tools used

The tools used during various levels of test-ing are: PyNFS, OpenPOSIX, LTP, Connec-tathon, IOZone, fsstress, ffsb, Bull LockTester,and custom usability scripts.

Future plans

Our future focus will be tests like NFSv4 ex-porting cluster filesystem, NFSv4 testing un-der XEN kernel, scalability, and other pendingitems in performance and robustness testing.

4.4 Red Hat

Red Hat NFS developers contribute to two codestreams, Red Hat Enterprise Linux and thecommunity Fedora Core project. Red Hat En-terprise Linux has supported portions of NFSv4for the last couple of years. Fedora Core closelytracks upstream development. By carefullymonitoring bug reports against Fedora Core, weare able to gauge the maturity and readiness ofthe upstream bits for our enterprise customers.

Red Hat has an automated test harness whichincorporates many of the tests mentioned ear-lier, among others:

• LTP

• fsx

• fsstress

• Locks tests

• ACL tests

• Connectathon suite

We run these tests against our nightly base-levels and compare the results against knowngood builds for regressions or problems. Inaddition, we have a performance group whorun many different benchmarks with a varietyof application mixes and closely monitor anyunexpected performance changes (we scan fordeltas of more than 3% to 5%).

We work closely with many key partners, whorun their own test suites. In some cases, thesetest suites are proprietary, so when problem ar-eas are encountered, we work together to comeup with reproducible test cases which we canadd to our test harness. We also work closelywith targeted customers during beta to test spe-cific new functionality in an effort to lever-age unique customer expertise or environments.We monitor those results to see where we needto enhance our internal testing and code re-views.

Red Hat products are also fully deployed inproduction throughout the company, and de-velopment builds are deployed in test labs andother controlled environments. There are fewquality metrics more powerful than having toface your angry co-workers on your way to thecoffee pot!

Red Hat NFS developer Steve Dickson is aConnectathon regular along with other Red Hatemployees. This is a very valuable opportunityto ensure maximum interoperability of both ourstable RHEL builds as well as our latest devel-opment trees.


5 Conclusions

Status of NFSv4 protocolFuture of NFSv4

There is a variety of extensions to the NFSv4protocol under development, and Linux is serv-ing as an important testbed for all of them.

Directory delegations allow enhanced (read-only) caching of directory data.

pNFS allows NFSv4 clients to perform file IOusing alternate methods, including parallel IOto multiple file servers and direct access toblock storage.

The Sessions extension fixes some of the trans-port problems that have long plagued NFS, fi-nally allowing a reliable replay cache that canensure only-once semantics.

Status of Linux NFSv4

This paper has shown that the efforts to datehave improved NFSv4 reliability and perfor-mance since 2004 up to now.

The early regression testing has helped devel-opers to quickly isolate and fix defects. Testsdone by OSDL and Bull have put NFSv4 intohigh pressure situations, in LAN and WANmodes. The NFSv4 test community’s effortshave been successful in identifying many mi-nor, medium and bad problems, on 32 and 64bits architectures. Though the main testing ac-tivity is done on ia32, tests are also done onx86_64, PowerPC and ia64 processors. Andeveryone knows that shaking code on differentarchitectures really helps in finding hidden mis-takes and bugs!

Also, the availability of new test suites (ACLsand Locks tests) through the LTP helps Linux

distributions to check that NFSv4 integratesperfectly in their specific ecosystem.

The comparison of NFSv4 with NFSv3 hasshown the benefits delivered by NFSv4: highreliability and very good performance on TCP.Also, interoperability tests have shown thatLinux NFSv4 nicely interoperates with otherUnix NFSv4 implementations. People attractedby Samba will enjoy a new NFS version thatdelivers both Unix-Windows interoperabilityand very good performance when reading files,both in synchronous and asynchronous modes.

When we compare the status of Linux NFSv4today against where it was when we started inlate 2004, we can see that the testing effortshave generated significant improvements in alltest areas and that the core of Linux NFSv4 isstable and powerful. Indeed, the NFSv4 infras-tructure has attained quality standards whichsurpass NFSv3 in many cases and offer secu-rity levels that today’s users are desperate for.

Now that Novell and Red Hat have started test-ing NFSv4 in depth on their distributions, thisis a clear signal for companies and individualsthat NFSv4 is ready to use on Linux. First forexperimentations, and soon in the field, whereLinux NFSv4 must prove that it scales nicelywith hundreds or thousands of clients.

Future of NFSv4 testing

There is still much functionality in develop-ment for NFSv4, thus the Linux NFSv4 testproject will continue. New tests will be writ-ten for testing the future Linux NFSv4 features:Named Attributes, NFSv4 ACLs, Replication,Migration. Also, the testing coverage must bemeasured to know the percentage of NFSv4code that is exercised when running tests overNFSv4.


References

[1] IETF: RFC 3530: NFSv4 Protocol:http://www.ietf.org/rfc/rfc3530.txt

[2] IETF: NFSv4:http://www.ietf.org/html.charters/nfsv4-charter.html

[3] Paper: The NFS Version 4 Protocol:http://www.nluug.nl/events/sane2000/papers/pawlowski.pdf

[4] Paper: Linux NFSv4: Implementationand Administration:http://lwn.net/2001/features/OLS/pdf/pdf/nfsv4_ols.pdf

[5] NFSv4 ACLs :http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-acls-00.txt

[6] CITI: NFSv4 Open Source ReferenceImplementation:http://www.citi.umich.edu/projects/nfsv4/

[7] CITI: NFSv4 for Linux 2.6 kernels:http://www.citi.umich.edu/projects/nfsv4/linux/

[8] Connectathon:http://www.connectathon.org/

[9] IOzone filesystem Benchmark:http://www.iozone.org/

[10] LTP (FSX, ACL & Lock tests):http://ltp.sourceforge.net/

[11] Ethereal (Network Protocol Analyzer):http://www.ethereal.com/

[12] FFSB: Flexible File System Benchmark:http://sourceforge.net/projects/ffsb/

[13] NetEm: Network Emulator:http://linux-net.osdl.org/index.php/Netem

[14] NewPyNFS (NFSv4 Functionality Test):http://www.citi.umich.edu/projects/nfsv4/pynfs/

[15] OSDL: NFSv4 Testing for Linux:http://developer.osdl.org/dev/nfsv4/site/index.php

[16] OSDL NFSv4 Test Matrix v1.13:http://developer.osdl.org/dev/nfsv4/site/testmatrix/testmatrix-1.13.pdf

[17] OSDL NFSv4 Wiki:http://wiki.linux-nfs.org/index.php/Main_Page

[18] NFSv4 Bugzilla:http://bugzilla.linux-nfs.org/

[19] Linux NFSv4 Client and Server MailingLists:http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

[20] Linux NFS mailing list:http://lists.sourceforge.net/lists/listinfo/nfs

[21] Bull: NFSv4 project:http://nfsv4.bullopensource.org/

[22] Novell: SUSE Linux:www.novell.com/linux/suse/

[23] Red Hat:http://www.redhat.com/

Proceedings of theLinux Symposium

Volume Two

July 19th–22nd, 2006Ottawa, Ontario

Canada

Conference Organizers

Andrew J. Hutton, Steamballoon, Inc.C. Craig Ross, Linux Symposium

Review Committee

Jeff Garzik, Red Hat SoftwareGerrit Huizenga, IBMDave Jones, Red Hat SoftwareBen LaHaise, Intel CorporationMatt Mackall, Selenic ConsultingPatrick Mochel, Intel CorporationC. Craig Ross, Linux SymposiumAndrew Hutton, Steamballoon, Inc.

Proceedings Formatting Team

John W. Lockhart, Red Hat, Inc.David M. Fellows, Fellows and Carr, Inc.Kyle McMartin

Authors retain copyright to all submitted papers, but have granted unlimited redistribution rightsto all as a condition of submission.

nfsv4 test project - linux kernel · nfsv4 test project bryce harrington osdl ... around nfsv41 by...

Documents