Downloading and using Featherstitch

Getting the source

The Featherstitch sources are currently maintained in a Git repository. Git is a distributed source control management system, which among other things, makes forking and merging much simpler than a centralized repository. Our hope is that this will make it easier for others to do cool things with Featherstitch.

First, download and install git, version 1.5.0 or later. Precompiled packages are available for major Linux distributions (packages "git-core" and "gitk" in Debian and Ubuntu) and for MacPorts on Mac OS X, among others. Once git is set up, you can run a command like this:

git clone git:// ~/fstitch

This downloads the Featherstitch sources, including the complete history, into the directory ~/fstitch (it will be about 20MiB). To later update to the latest sources, including all the intermediate history, run:

cd ~/fstitch
git pull

Featherstitch is released under the terms of the GNU General Public License. It is copyright 2005-2007 The Regents of the University of California.

We also have three other projects that we've modified to use patchgroups: UW IMAP, Subversion, and gzip. Our modified versions are available from git as well:

git clone git:// ~/uwimap
git clone git:// ~/subversion
git clone git:// ~/gzip

To browse all of these repositories online, use our gitweb browser.

Alternatively to using git, you may download a live snapshot: fstitch.tar.gz, uwimap.tar.gz, subversion.tar.gz, and gzip.tar.gz.

Configuring your kernel

Currently, Featherstitch is incompatible with a few Linux configuration options which are often enabled by default. Eventually we'd like to eliminate this problem, but for the time being, you will need to disable them in your kernel to compile and run Featherstitch. The list is:

We also recommend a few settings to make sure all of our scripts work well, and to enable additional debugging options:

The kernel .config files we use for Linux in Ubuntu 6.10 are available: for debugging, for profiling, and for benchmarking.

Patching your kernel

While you can use Featherstitch with an unmodified Linux kernel, you will likely want to apply some or all of the patches below. The commands below assume Linux sources in ~/linux- and Featherstitch in ~/fstitch.


Patchgroup support requires kernel process notifications to manage the per-process patchgroup tables:

cd ~/linux-
patch -p1 < ~/fstitch/patches/fstitch-linux-

You will also need to reconfigure your kernel to enable the new CONFIG_FSTITCH_PROC option (in the "File systems" section). The config files above have this option enabled already.

Disk safety

For Featherstitch to know when a disk write is committed to stable storage you must either enable FUA disk I/O or switch the disk's cache to write-through mode. In our experience their performance is the same, but FUA is more flexible (e.g., each I/O request can select whether to actually use FUA).

For FUA disk I/O, patch the kernel (recommended):

cd ~/linux-
patch -p1 < ~/fstitch/patches/bio-fua-

Alternately, switch the disk's cache to write-through mode:

hdparm -W 0 /dev/<DISK>

Remote operation

Although Linux can execute an infinite loop in 6 seconds, occasionally we have seen one of our buggy loops run just a bit longer than that. When this happens locally, we can power cycle the machine, but when we're working on a crashbox remotely, that machine is effectively lost until we actually go to the lab and reboot it. So, we made a patch for the soft lockup detector in Linux that reboots after about a minute of soft lockup, which we suspect nearly always means an infinite loop. To apply it, do this:

cd ~/linux-
patch -p0 < ~/fstitch/patches/softlockup-

You will also need to make sure you've got CONFIG_DETECT_SOFTLOCKUP enabled. (It is in our config files.)

Compiling and loading Featherstitch

You should be able to run make in the fstitch directory at this point, and Featherstitch should detect what kernel features are available and compile correctly. (Please let us know if you encounter any compile errors!) The Featherstitch kernel module will be created as kfstitchd.ko. It has several insmod options that affect how it searches for and mounts supported file systems. Here are a few of the more useful options:

The full list of supported options can be listed with this command:

modinfo kfstitchd.ko

For instance, to load Featherstitch using /dev/sdb and metadata journaling, you would do:

sudo insmod kfstitchd.ko device=/dev/sdb use_journal=1

Once Featherstitch is loaded, the file system(s) it detects can be mounted using commands like:

sudo mount -t fstitch fstitch:/ /mnt
sudo mount -t fstitch fstitch:/dev /mnt/dev

Eventually, we'd like to have a configuration file that determines how Featherstitch starts up. For now, there is code in fstitchd_init.c that detects disks, partitions, and file systems, and instantiates Featherstitch modules to use them. Some of its behavior can be adjusted using the insmod parameters, but you may need to edit this source file if you want to create a very different module configuration.

Using patchgroups

The patchgroup API is documented in fstitch/fscore/patchgroup.h.
To compile an application with patchgroup support, the include path will need to include your fstitch directory. To link and run the application, the compiler and runtime linker library paths will need to include fstitch/obj/kernel/lib (which contains The patchgroup-enabled gzip and subversion sources may serve as useful examples for autotool-based build systems (autotool commits for: gzip and subversion).

Debugging Featherstitch

small patchgroup example medium patchgroup example large patchgroup example huge patchgroup example
Small Medium Large Huge

Example debugger diagrams. Click on thumbnail to enlarge.

Featherstitch includes a built-in patch debugger. To enable it, set FSTITCH_DEBUG to 1 in fscore/debug.h and recompile. When Featherstitch is loaded, it will create a file /proc/kfstitchd_debug which should be read and saved somewhere outside of the Featherstitch file system(s). This file will contain a (binary) trace of most patch-related changes inside of Featherstitch, and is very useful for debugging both correctness and performance problems. Using this option does, however, slow Featherstitch down somewhat.

To analyze these debug traces, there is a utility called fdb which will be created as obj/util/fdb when Featherstitch is compiled. It can step through the traces and allow you to examine the status of all patches and dependencies in the system, as well as render a graphical representation of the patches at each step. (For this feature, you must have the "graphviz" package installed, and specifically, its dot program.) It has some brief documentation built in - try the "help" command. (Also try the "gui" command, which is very useful and gives debugging Featherstitch a unique sound: "click, click, click...")

Userspace Featherstitch

Featherstitch can also run as a FUSE file server process in userspace. In this mode, patchgroups are not available, and instead of the kernel block device layer, files in the normal Linux file system are used as block devices.

First, you'll need FUSE (we use version 2.5.2). You'll also need to copy fstitch/conf/fuse.conf to /etc, and, if you want to run Featherstitch in valgrind, you'll need to put fstitch/util/fusermount in your PATH (after modifying it to point to the location of the real fusermount).

Assuming FUSE is installed in /usr/local, you could:

sudo cp ~/fstitch/conf/fuse.conf /etc
export PATH=~/fstitch/util:$PATH

The FUSE version of Featherstitch also works with MacFUSE on Mac OS X. However, currently the Featherstitch file system modules are not endianness-aware, so the file system images it uses must be the same endianness as the CPU. This is not usually a problem for modern x86 Apple machines, but it affects the older PPC machines.

Once FUSE is set up, you should be able to run make -f Makefile.user, or, if you want to compile both Featherstitch targets, just:

cd ~/fstitch
touch .user

After compiling, you can start Featherstitch by running, which accepts many of the same options as the kernel version does. (However, the device= parameter is instead called unix_file. Or, it can be passed positionally.) An example disk image is provided and built automatically as obj/images/ext2.img. Here is a simple example command to start userspace Featherstitch:

mkdir -p mnt
./ mnt obj/images/ext2.img
And an example which uses the unlink module for asynchronous writes:
./ mnt unix_file=obj/images/ext2.img use_unlink=1

To run Featherstitch inside gdb or valgrind, set the FSTITCHD_WRAP environment variable to the wrapper program.

FSTITCHD_WRAP=gdb ./ mnt obj/images/ext2.img

Or, as a shortcut for common wrappers, just pass the wrapper name to

./ --gdb mnt obj/images/ext2.img

To shut down userspace Featherstitch, simply hit ^C.