ZFS Storage

Pool layout, tuning, and maintenance on the starhaven Proxmox home lab

Overview

The Proxmox host (starhaven) runs a single ZFS pool, vm-8tb-hdd0, on OpenZFS 2.4.4-pve1, backing every VM disk and LXC subvolume. The Proxmox OS itself lives separately on an NVMe drive with LVM-thin, so guest storage churn can't threaten the hypervisor's own root filesystem. This page is the ZFS-specific deep dive; the wider storage decision (why single-disk today, what the next pool looks like) is covered on the homelab storage section.

Every command output on this page was pulled live from the host via read-only ansibleuser Ansible ad-hoc runs at the time of writing, not typed from memory.

Pool Status

Live from the host:

$ zpool list -v
NAME                                    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
vm-8tb-hdd0                            7.27T  2.45T  4.82T        -         -    25%    33%  1.00x    ONLINE  -
  ata-WDC_WD8005FRYZ-01CBLB0_AN298L3N  7.28T  2.45T  4.82T        -         -    25%  33.7%      -    ONLINE        -

$ zpool status
  pool: vm-8tb-hdd0
 state: ONLINE
  scan: scrub repaired 0B in 04:57:23 with 0 errors on Sun Sep 13 05:21:24 2026
config:

	NAME                                   STATE     READ WRITE CKSUM
	vm-8tb-hdd0                            ONLINE       0     0     0
	  ata-WDC_WD8005FRYZ-01CBLB0_AN298L3N  ONLINE       0     0     0

errors: No known data errors

Single disk, no redundancy - a known, accepted gap (see roadmap below).

Datasets & Zvols

zfs list against the pool - LXC subvolumes are filesystem datasets, VM disks are zvols (no mountpoint):

$ zfs list
NAME                            USED  AVAIL  REFER  MOUNTPOINT
vm-8tb-hdd0                    5.10T  2.04T  14.1G  /vm-8tb-hdd0
vm-8tb-hdd0/subvol-105-disk-0  1.17G  6.83G  1.17G  /vm-8tb-hdd0/subvol-105-disk-0
vm-8tb-hdd0/subvol-107-disk-0  2.39G  5.61G  2.39G  /vm-8tb-hdd0/subvol-107-disk-0
vm-8tb-hdd0/subvol-108-disk-0  1.58G  6.42G  1.58G  /vm-8tb-hdd0/subvol-108-disk-0
vm-8tb-hdd0/subvol-110-disk-0  1.13G  6.87G  1.13G  /vm-8tb-hdd0/subvol-110-disk-0
vm-8tb-hdd0/vm-100-disk-0       254G  2.28T  8.40G  -
vm-8tb-hdd0/vm-100-disk-1      1.98T  2.25T  1.78T  -
vm-8tb-hdd0/vm-101-disk-0       122G  2.14T  25.4G  -
vm-8tb-hdd0/vm-103-disk-0      60.9G  2.10T  5.37G  -
vm-8tb-hdd0/vm-106-disk-0      60.9G  2.10T  5.39G  -
vm-8tb-hdd0/vm-106-disk-1      1.93T  3.48T   512G  -
vm-8tb-hdd0/vm-109-disk-0       284G  2.28T  41.3G  -
vm-8tb-hdd0/vm-113-disk-0      60.9G  2.09T  9.69G  -
vm-8tb-hdd0/vm-114-disk-0       130G  2.17T  3.40G  -
vm-8tb-hdd0/vm-115-disk-0       152G  2.16T  34.3G  -
vm-8tb-hdd0/vm-116-disk-0      61.1G  2.09T  16.0G  -
vm-8tb-hdd0/vm-116-disk-1         3M  2.04T    96K  -

Tuning

Setting	Value	Why
`atime`	off (pool-wide)	No metadata write on every read - pure overhead for VM/LXC-backed storage.
`compression`	lz4	~1.02x in practice - most stored data is already-compressed media/game content, so it costs almost nothing to leave on.
`refreservation`	full `volsize` on every zvol	Thick-provisioned by design today; thin provisioning (`sparse 1`) is planned for the next pool instead of retrofitting this one.
`autotrim`	off (default)	Spinning disk, not SSD/NVMe - TRIM is handled by the monthly cron job below instead of continuously.
ARC min / max	1GiB / 16GiB	Resized 24GiB→16GiB on 2026-09-23 - the old 24GiB ceiling was sized for a 128GB RAM upgrade that hasn't happened yet (host is still 62GB). Full reasoning in ARC & Module Tuning below.
Prefetch (`zfs_prefetch_disable`)	off (disabled)	Aggressive prefetch was filling ARC too fast for this host's mixed Jellyfin/Navidrome/ gaming-VM workload.
Pool disk I/O scheduler	`none`	ZFS already does its own I/O scheduling internally - `mq-deadline` underneath is a redundant second reorder/queue layer. See Disk I/O Scheduler below.

ARC & Module Tuning

Live /etc/modprobe.d/zfs.conf on the host, unchanged from what was applied and verified post-reboot on 2026-09-23:

$ cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_min=1073741824         # 1 GiB floor - ARC never drops below this
options zfs zfs_arc_max=17179869184       # 16 GiB ceiling
options zfs zfs_txg_timeout=5             # commit dirty data every 5s - default, good balance
options zfs zfs_deadman_synctime_ms=600000 # 10min hung I/O watchdog before panic
options zfs zfs_prefetch_disable=1        # prefetch OFF - Jellyfin/Navidrome/gaming benefit
options zfs zfetch_max_streams=6          # one prefetch stream per core (i5-8500 6c/6t)
options zfs zfetch_max_distance=33554432  # 32 MiB lookahead - tighter than 64 MiB default
options zfs zfetch_min_distance=2097152   # 2 MiB minimum stride before prefetch fires

Live arc_summary, same day, ARC still ramping back up after the resize reboot:

$ arc_summary
ARC status:
        Total memory size:                                      62.6 GiB
        Min target size:                                1.6 %    1.0 GiB
        Max target size:                               25.5 %   16.0 GiB
        Target size (adaptive):                        64.5 %   10.4 GiB
        Current size:                                  64.5 %   10.3 GiB
        Free memory size:                                       28.1 GiB
        Available memory size:                                  26.0 GiB

Decision - ARC capped well below total RAM, not maxed out

The generic "unused RAM is wasted RAM, max out zfs_arc_max" advice is correct for a dedicated ZFS storage box, but doesn't apply to a hypervisor host like this one, where VMs have their own balloon-managed memory floors competing for the same RAM (see the homelab memory-tuning writeup). There's also a sharper, concrete reason on this host specifically: ARC is a real OOM-killer risk for PCIe-passthrough VMs - ZFS doesn't release cache memory fast enough to satisfy a huge single allocation request, so an oversized ARC can starve a VFIO GPU-passthrough guest into an OOM kill it wouldn't otherwise hit. This host runs VFIO GPU+NIC passthrough for a gaming VM, so the 16GiB ceiling is deliberate headroom, not just a reclaim nicety.

Mechanical gotcha

zfs_arc_max is not live-writable via the sysfs parameter on a running system in any way that reliably reflects in real RAM usage. Applying a change needs:

# edit /etc/modprobe.d/zfs.conf, then rebuild every installed kernel's initramfs
export PATH=/usr/sbin:/usr/bin:/sbin:/bin:$PATH   # update-initramfs lives in /usr/sbin, not on PATH in a plain root shell on this host
update-initramfs -u -k all
# then reboot - a lower ceiling only reliably shrinks under real memory
# pressure otherwise, and a reboot is the clean way to get there

Verify post-reboot with:

cat /sys/module/zfs/parameters/zfs_arc_max
arc_summary | grep -E 'Current size|Max target'

FAQ - does ZFS use RAM beyond `arc_max`?

Barely, and what it does use is proportional to arc_max itself, not a separate unbounded pool:

Dbuf cache - deliberately sized as a fraction of the ARC target (dbuf_cache_shift=5 → ARC target ÷32, plus a metadata dbuf cache at dbuf_metadata_cache_shift=6 → ÷64). At this host's 16GiB cap that's roughly 512MB + 256MB ≈ ~768MB combined, scaling down automatically if arc_max is ever lowered further - not a hidden unbounded consumer.
ABD scatter/gather buffers are already folded into ARC's own reported size, not a separate untracked pool sitting outside arc_summary's numbers.
ARC hash table is the one genuinely separate-from-arc_max consumer, sized off total physical RAM (~1MiB per 1GiB of RAM at the default zfs_arc_average_blocksize=8KiB) - on this 62GB host that's ~62MB, negligible next to a 16GiB cap.

Combined "beyond arc_max" overhead here is on the order of ~800MB-1GB, nowhere near enough to explain any meaningful unaccounted RAM usage.

Disk I/O Scheduler

ZFS does its own I/O scheduling internally, so leaving the kernel's mq-deadline scheduler active underneath adds a redundant second reorder/queue layer on the pool disk. Pinned via a udev rule keyed to the drive's serial number rather than its device name (/dev/sdb can shift across reboots, the serial won't):

$ cat /etc/udev/rules.d/60-zfs-scheduler.rules
# ZFS does its own I/O scheduling; avoid double-queueing under mq-deadline.
# vm-8tb-hdd0 pool disk (WDC WD8005FRYZ, serial AN298L3N)
ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="AN298L3N", ATTR{queue/scheduler}="none"

The NVMe root disk was already correctly at scheduler=none by default (untouched). The separate rclone backup-target disk was deliberately left on mq-deadline - it's not a latency-sensitive path.

Per-Dataset Tuning: `volblocksize` vs `recordsize`

VM disks on this pool are zvols, not filesystem datasets - recordsize is not a valid property on a zvol (zfs set recordsize=... fails). Zvols use volblocksize instead, which is fixed at creation time and can only be changed by destroying and recreating the zvol (a real data migration, not a live tweak).

$ zfs get volblocksize,recordsize vm-8tb-hdd0/vm-100-disk-1 vm-8tb-hdd0/subvol-105-disk-0
NAME                           PROPERTY      VALUE     SOURCE
vm-8tb-hdd0/vm-100-disk-1      volblocksize  16K       default
vm-8tb-hdd0/vm-100-disk-1      recordsize    -         -
vm-8tb-hdd0/subvol-105-disk-0  volblocksize  -         -
vm-8tb-hdd0/subvol-105-disk-0  recordsize    128K      default

All VM zvols sit at the Proxmox-default volblocksize=16K, untouched. The 2TB media library zvol (vm-100-disk-1) is a large-sequential-read workload that would theoretically benefit from a bigger block size, but changing it means destroying and recreating that zvol - a real migration with downtime risk, not something to do casually. LXC subvolumes (105, 107, 108, 110) are actual filesystem datasets, so recordsize does apply there - left at the default 128K. General pattern going forward: set volblocksize at zvol creation time if a large-sequential workload is expected, since it can't be changed after.

Degraded-Drive Recovery

For a zpool status showing a DEGRADED disk with no read/write/checksum errors currently accumulating:

zpool scrub vm-8tb-hdd0   # can take 1-2+ days depending on drive size
# once the scrub completes cleanly:
zpool clear vm-8tb-hdd0   # resets the error counter, pool reports healthy again

This resets the error counter for cosmetic/transient error counts - it doesn't fix an actual drive fault. On the current single-disk vm-8tb-hdd0 pool (no redundancy) a genuine drive fault isn't recoverable this way; the procedure matters more once the mirrored pool in the roadmap exists.

Scrub & TRIM Schedule

Stock Debian/Proxmox zfsutils-linux cron, unmodified:

$ cat /etc/cron.d/zfsutils-linux
# TRIM the first Sunday of every month.
24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi

# Scrub the second Sunday of every month.
24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi

Last scrub: 0B repaired, 0 errors, completed in 4h57m on 2026-09-13.

Useful Commands

# Pool capacity and health
zpool list -v
zpool status

# Dataset/zvol usage
zfs list -o name,used,avail,refer,mountpoint

# Per-dataset properties
zfs get compression,atime,refreservation vm-8tb-hdd0

# Manual scrub / TRIM
zpool scrub vm-8tb-hdd0
zpool trim vm-8tb-hdd0

# ARC stats
arc_summary | head -30

Roadmap

A second pool is planned: a mirrored pair of 16TB or 24TB WD Gold drives, kept separate from vm-8tb-hdd0 (which has zero redundancy today) rather than expanding it. Staying with ZFS over Btrfs or mdadm for it - Btrfs RAID1 is still Proxmox's own "Technical Preview" with reported degraded-boot issues, and mdadm+ext4 isn't officially supported by Proxmox and loses native snapshots and checksums for no real benefit. Thin provisioning (sparse 1) is planned for this pool from the start, unlike the current one.