FreeNAS 10Challenges of Building a Modern Storage

Appliance based on FreeBSD

But first…

Welcome Back to Berkeley!

Last time Here: FreeBSDCon ’99!

FreeNAS 10Key Requirements

• Make a highly complex system as simple as possible for end-users but still powerful for admins

• Provide maximum “situational awareness” to all users

• Do as much as possible asynchronously

• Provide for flexible application hosting as well as storage (“Hyperconverged storage”)

A Short Demo

Challenge: Building• FreeBSD builds are not bit-for-bit reproducible

(build times and paths sneak into binaries; delta packages are wheeeeeeeee!).

• Packaging: Appliances require cryptographically signed update mechanism with config data segregated

• Debugging symbols: No concept of the .DSYM file; we need to keep two copies of all binaries (install ISO is 670M - debug info is 908M!)

Challenge: Building• FreeBSD is at least three different things:

• Kernel + Modules

• “Base”

• Ports (Poudriere)

• We had to create our own build system + DSL (profile description language) to unify these concepts for a software appliance

FreeBSD Today

Kernel

Base

Packages (ports - infinite size)

All Your Base

Self-serveCafe

FreeBSD Someday?

Kernel

Base

Packages (ports - moar infinite size)

MinimalistApproach

Kernel Optional Modules

ApplianceProfile

Schema

Appliance Software Lifecycle Management

• Install (for specific mission profile) needs to be as automated as possible. ZFS boot environments have been a big plus.

• Need to be able to upgrade entire application profile (kern + base + pkgs) as one unified concept.

• All config data needs to survive upgrades with no user intervention (sorry, mergemaster).

Misc FreeBSD Challenges

• Future of SMB 3.1.1 Performance is iffy - FreeBSD is not a poster-child for Samba and SMB is important.

• Some perf issues identified by Samba core team, now need work from us.

• Unix Domain Sockets really suck for serious IPC (we’ve found long-standing data corruption bugs, performance is definitely not great, namespace is filesystem)

• /etc/rc.d is really not service management (highly porous, doesn’t provide consistent pid tracking, etc etc etc)

Misc BSD Challenges• More full-stack developers would reveal

and fix blatant userland + OS mismatches earlier (python fork()! OMG!). More appliance folks would be great

• We’re missing out on mainstream clustered filesystems, some of the latest OpenZFS encryption work (ZoL is pulling ahead), and much other Enterprise goodness from Linux space

Bhyve Improvements Made

• Added 9pfs client / server support to share host data with VMs - still some challenges (ACLs! 😡)

• Created early VM guest tools (user kind of wants to know what IP address a VM is on, among many other things)

• Better virtual device support (E1000 / NE2000)

• Virtio Console driver for C&C, fast VNC console

Bhyve Improvements Needed

• VGA Emulation still very green

• USB Device Mapping (like VMWare)

• Suspend / Resume (for snapshots / migration)

• CD image mount/eject after boot

• High-level management APIs / better ZFS integration / other crazy dreams

Containers• Jails are sort of cool and all but fragile (esp with

VIMAGE), implementation has long beard, and we never standardized lifecycle management (mechanism not policy again??)

• Docker / LXD combine privilege domains with packaging / lifecycle and better networking management out of the box

• Fight for mainstream DevOps hearts-and-minds has been won by Docker / LXD / (TBD) - Now What?

Container Futures• BHyve + VM guest tools allow us to simply

wrap containers and many other “service providers” into VMs and proxy them as “bigger containers”

• Wrapping is a lot easier than making Linux Emulation 100%, but if we go that route we also need a roadmap and a clear idea of what “victory” looks like

*BSD is still awesome

• … But every year it gets a little harder to justify building Enterprise solutions like FreeNAS on top of it

• It’s about making both tools and solutions (chicken-and-egg problem). Software appliances and HW/SW combined solutions are a good start

• We can move faster (less bikeshed, more ambition)

Q&A