In the previous post I described how I mounted 4 NVMe flash drives in a single host in order to build an all-flash datastore. In this post, I’ll describe how to move from FreeNAS, TrueNAS (or any other ZFS host OS) to ZFS on Linux, and test if the performance is acceptable to continue with ZFS. Spoiler: it’s not.

Moving to ZFS on Linux

Still on the FreeNAS box, I extended my single SSD pool (yes really, NVMEe 1TB SSDs were expensive 3 years ago) to a pool with 2 mirrored vdevs through standard zpool commands. Afterwards I shutdown the VM and connected the disks (via passthrough/VT-d) to a new Ubuntu VM.

Installing ZFS on Ubuntu is easy as the essential parts are included in the kernel, and there’s a standard package for managing ZFS:

$ sudo apt install zfsutils-linux

Instead of creating a pool, I imported my existing pool of 4 disks

$ sudo zpool import tank -f

The -f (force) option is because I didn’t export the pool properly in FreeNAS (shame on me).

$ zpool status
  pool: tank
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: resilvered 232G in 1h29m with 0 errors on Thu Feb 27 12:39:37 2020
config:

        NAME           STATE     READ WRITE CKSUM
        tank           ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            nvme0n1    ONLINE       0     0     0
            nvme3n1    ONLINE       0     0     0
          mirror-1     ONLINE       0     0     0
            nvme1n1    ONLINE       0     0     0
            nvme2n1    ONLINE       0     0     0

errors: No known data errors

The pool contains a ZFS volume (zvol) named iSCSIvol, which is a virtual block device to allow ESXi to connect through iSCSI:

$ zfs list
NAME                                                    USED  AVAIL  REFER  MOUNTPOINT
tank                                                   1.53T   339G    88K  /tank
tank/.system                                           50.8M   339G   100K  legacy
tank/iSCSIvol                                          1.51T  1.50T   350G  -

Performance

We already measured the raw performance of one of the disks in an earlier post:

The next step is to get a measurement for the raw performance of the ZFS volume before things like iSCSI, VMFS, network or ESXi come into play. To do this, I created a new zvol on the zpool, and did the same experiment I did before, only instead of using the NVMe disk (/dev/nvmex) directly, I now accessed to the zvol test device (/dev/zvol/tank/test) directly from the VM containing the pool. The results:

$ fio fio --ioengine=libaio --direct=1 --name=test --filename=/dev/zvol/tank/test --iodepth=32 --size=12G --numjobs=16 --group_reporting --bs=4k --readwrite=randread

Goodbye ZFS

So I expected some overhead, but this is terrible. Keep in mind the devices are mirrored, and there are two mirrors, so the real read performance should be 4x as good as, and the write 2x as good as a single device. ZFS requires a lot of RAM to perform, but for now consider the VM has plenty of RAM. The results are not CPU constrained either, as I increased the number of virtual CPUs (vCPU) until the VM was not constantly at 100% load. Instead, iostat -x measurements indicated the NVMe devices themselves were never utilized more than 25%, with the zfs device at 100%. Tuning ZFS doesn’t help here as performance traces showed most of the time was spent in spinlocks and mutexes. This is clearly a performance problem in the ZFS code, which becomes apparent as the NVMe devices are too fast.

These results are so bad in fact, that as much as I love ZFS, I can’t continue with ZFS as a backing for my all flash datastore until ZFS gets some much needed performance tweaks for all flash purposes. That’s why in the next post I’ll continue the same exercise with Linux software RAID and LVM, and see how that works.