What filesystem are you running on your NAS/Backups, and why?
23 points by tffr
23 points by tffr
What filesystem are you running on your NAS/Backups, and why?
And if you're using multiple disks/devices, how do you manage them (filesystem's RAID, mdadm or other software solutions, hardware RAID)?
FreeBSD, ZFS, 4 devices, mirrored stripe. Why? Dunno, in the base system, sort of looks after itself.
Another nice thing about ZFS is instead of using rsync to keep a remote clone, you can use ZFS clone to make a direct clone of an encrypted ZFS where the remote one stays encrypted the whole time.
# local snapshot, then save name
datetime=`date "+%Y-%m-%d-%H%M"`
doas zfs snapshot z/me@$datetime
zsnap=$(zfs list -t snapshot -r z | tail -1 | awk '{print $1}')
# ssh into zfs.rent to get name of my most recent snapshot there
rsnap=$(ssh rent "zfs list -t snapshot -r z | tail -1" | awk '{print $1}')
# ZFS clone the diff between remote old and local new
doas zfs send -w -i $rsnap $zsnap | ssh rent doas zfs receive z/me
Doing it that way, my remote filesystem (on https://zfs.rent who I can recommend) is never unencrypted, but stays a perfect clone of my local encrypted ZFS.
The cherry on top would be to be able send a local unencrypted dataset to an encrypted dataset, keeping the key on the origin. IIRC, this cannot be done, but maybe they were working on it?
Making a couple of plugins to make this happen on different NAS platforms with some friendly NAT hole punching would be quite cool; you could do mutual backups over the Internet with semi-trusted people.
Same (though I use RAID-Z with three disks). I use zrepl for backups.
I have a remote server that also runs FreeBSD. It has a ZVOL that it exposes as an iSCSI target. The NAS attaches this (via Wireguard) and uses GELI with ZFS on top. It uses zrepl in local mode to replicate the ZFS datasets I care about to the remote machine. The remote machine uses zrepl to manage decaying-frequency snapshots.
The things I need that are not in the base system: zrepl and the wireguard tools. These are both Go programs and so produce statically linked binaries.
The local machine also uses zrepl to manage decaying-frequency snapshots.
This gives the following guarantees:
ZFS, because the whole point of backups is to protect data, not play Russian roulette with it.
Rsync to two backup servers running Alpine Linux over 10G Ethernet:
Two 14TB USB external hard drives rotated weekly for offsite backups (using my own https://github.com/fazalmajid/zfssnap and its zfsvault utility) to incrementally replicate from the primary to encrypted offsite ZFS), so a total of 5 replicas, not counting the replicas of my Documents directory made on every laptop I own using unison and backups of my Photos to my old Mac Studio.
On my home Synology I use the default btrfs. This is the “if I die” NAS outside my homelab.
In my homelab I have used zfs for 15 years and I can’t imagine using anything else. In addition to 100% reliability, it gives me easy and foolproof snapshots, I can easily and efficiently replicate raw encrypted pools offsite, I can load encryption keys on a initrd boot ssh session so that they’re not on disk, etc. It’s always given me what I wanted.
ZFS on 2 SSDs, with both drives in the same zpool: https://michael.stapelberg.ch/posts/2023-10-25-my-all-flash-zfs-network-storage-build/
I run two such NAS devices for redundancy. The other one runs ext4 for diversity.
I later migrated the OS to NixOS: https://michael.stapelberg.ch/posts/2025-07-13-nixos-nas-network-storage-config/
I use btrfs. Yes, btrfs does not always have the most sterling reputation for reliability, but it provides the features I want in a very straightforward way and I think people tend to overthink the durability problems here.
The principal advantage of btrfs is that I can have an arbitrary pool of volumes (I use subvolumes of one big one) that are duplicated across multiple devices for redundancy. btfrfs is much more flexible than mdraid-based solutions in that my disks can be heterogeneous and I can adjust how many copies are retained. The high level of integration also makes it less complex to manage than the other obvious routes to achieve what I want, like say XFS on LVM. btfs's strong snapshot support is also useful as I keep a short history of frequent snapshots that provide a very easy fix for the most common human error scenarios (irresponsible use of rm -rf).
As for the downsides... I personally have never actually had a data loss incident with btrfs, although I have had situations in which I have had to use recovery tools (in both cases my own fault though, not a result of any kind of silent/random FS corruption). I don't know if I'm really that unusual in this regard or if people overstate btrfs's problems, but btrfs is the filesystem I use on everything, and usually the only problem I have is that a disk fails and I have to tweak the mount options so that btrfs will accept degraded operation for recovery purposes. You don't have to do it this way, but silent disk failures are a big scary problem so I prefer that operator intervention is required to run on a degraded situation.
Now, all of that said, the real reason that I don't worry too much about btrfs reliability is that I think that investing any trust in the long-term durability of a filesystem or device is ill-advised. The important contents of my btrfs volumes are all stored on other machines as well, or on cloud object storage providers, whatever makes sense in the given scenario.
ZFS is another common recommendation along the same lines, however, ZFS is very "heavy" and you can run into performance problems pretty quickly. It's 100% possible to tune ZFS to work well on all kinds of systems, but I personally have never felt the higher level of effort/complexity involved in ZFS (e.g. userspace services) to be worth the benefits over btrfs, at least on a small system like a NAS. When you get into machines with a lot of memory and a lot more concurrent use, like more of a SAN situation, I think that tradeoff probably flips around. That said, I also have VM hosts running off of btrfs (proxmox) and I would say that, overall, they have been less troublesome than my proxmox-on-ZFS experience, despite ZFS being the fully supported option and btrfs being experimental (in proxmox). This might be a trait of the mostly scavenged storage I use in these clusters though and the generally odd setup.
ZFS is very "heavy" and you can run into performance problems pretty quickly.
This is simply not true
I was using btrfs for incremental backups from my laptop, using this approach: https://fedoramagazine.org/btrfs-snapshots-backup-incremental/
But on a semi-regular basis something would go wrong and I would no longer be able to send an incremental backup, requiring me to start over with a full sync. I'm no longer enthusiastic about this approach. Maybe I would feel differently if I had any idea what was happening or how to fix it.
ZFS + raidz2 (9x 18T disks, and then 10x 18T disks in a separate machine for backup).
I've periodically checked other filesystems - I've had corruption issues with btrfs (and it still doesn't support raid5/6 safely as far as I know).
I was hopeful about bcachefs, but it does seem like interpersonal issues have killed off that ever being in upstream.
I don't do root on ZFS though, I use xfs for that.
In my NAS I have
I setup sanoid for regular snapshots and run zpool scrub on a schedule. The system drive is backed up to rsync.net (which also runs ZFS) using Restic. I don't use ZFS send-receive because I want to be able to restore individual files if necessary. Eventually I want to purchase more rsync.net storage to back up my bulk pool too.
btrfs using Fedora bootc, mainly because getting ZFS to work reliably on Linux (especially when booting into ZFS) is a challenge on Fedora, especially when using bootc. It should be possible using Alma Linux for example, but then you're giving up the more up to date software of Fedora.
One gripe I have with btrfs is that booting into encrypted RAID1 is kind of clunky and painful as the ESP and boot partitions are not synced and you need to list the UUID of each boot drive to unlock them.
This means that if your boot drive fails, you have to somehow recover the boot/ESP to the remaining drive(s), then update the list of UUIDS in the kernel arguments and then boot into degraded mode.
For my backup/home server I don't use this approach (only for my desktop) though, instead it uses a JBOD setup as it uses a bunch of different SSDs for storing blu-ray rips, and running that in RAID is a bit of a waste money wise.
especially when using bootc.
ha, can relate to this - one of the reasons I'm asking is I'm running an image-based Debian image (similar to systemd/ParticleOS) so ZFS seems like a PITA, especially with secure boot.
slightly off-topic but how's bootc working for your setup?
bootc generally works well, though the somewhat exotic file system setup breaks btrfs receive when receiving to the root drive, which is a bit annoying.
I'm a little less impressed by some of the bootc adjacent tools (e.g bcvk) as they seem to rely heavily on LLM produced and subsequently end up being janky to use.
Documentation is also a mess with everything pointing in different directions, assuming it's properly documented to begin with.
ZFS on a single system NVMe as well as some mirrored pools on SSDs and HDDs.
It seemed to be the most stable option for people who like data integrity and snapshots.
NixOS also makes it pretty painless
Currently btrfs, it's my go-to filesystem, so I use it for everything, unless I have a strong reason not to. I use it primarily because my systems are currently Linux, and I wanted a filesystem that is in-tree (that rules ZFS out), one that doesn't use inodes, and where I can snapshot easily, preferably without LVM.
That pretty much means btrfs, and for what I need it for, it has been working fine for the past few years, no issues at all, no data loss, no need to recover anything.
Mind you, I have recently started to experiment with NetBSD, and btrfs isn't an option there. As I'm still in the early experiments stage, I use the default filesystem, but at some point, I'll poke ZFS a bit too.
APFS on a four SSD RAID5 thing, macOS $latest.
Why? I have a Mac Mini serving as a server and mostly mac clients, so it's the path of least resistance.
FWIW: Time Machine backups work nicely to Samba on FreeBSD with ZFS and snapshotting the underlying ZFS filesystem lets you avoid a lot of the ‘oh dear, Time Machine has corrupted your backups again, I guess you need to delete them and start again’ issues: just roll back to a snapshot from a month ago, verify the backups, roll back more if they still fail (shouldn’t, I think Time Machine verifies the backups at least once a month).
I haven’t had this problem since Time Machine switched to APFS for the remote backups, but apparently other people have (with the HFS+ ones, almost everyone saw this at least once every few years).
XFS because that is what unraid does by default. Also how I do multiple drives, unraid parity drive.
whiterun: main PC: Coding stuff through herdr with agents and games
riverwood: laptop
m5 air
rsync.net: Backups
My backup pattern is either append only (photos and stuff like that) or Git, the rest is essentially ephemeral and I don't care
I've got more than a few hundred TB in ZFS/RAIDZ2. It has had a few issues and fragmentation will cause problems at some point. But it has been less problematic and more flexible than the alternatives I've tried so far.
I've tried BTRFS, XFS, EXT4 with LVM or mdraid. LVM specifically is much more trouble than its worth. BTRFS corrupted itself entirely. XFS was just lacking when I tried it.
If I have like a single disk for some simple use-case I'm using EXT4 however.
ext4 on debian (13), RAID1 on 2 16TB HDDs. From there, backed up to rsync.net via borgbackup. The reason is because I'm comfortable using Debian.
At home: most boxes backup to the primary server, which runs Linux and ZFS in RAID10 over 4 spinning disks for bulk storage. It backs up via ZFS send to a single encrypted disk elsewhere, and subsets to some other locations.
At work: most machines run ZFS and send to a very large backup server which has a twin in another city 500 miles away.
tldr; ZFS like basically everyone else. Why: proven stability and well maintained.
When I need to combine more than one disk(like in a NAS): ZFS. Why? It's "boring" in that it's been in production use for a very long time and it's still well maintained.
When I only have one disk, like on a laptop, I just use whatever the default is for that OS. why? To be the default in an OS it generally has to be boring, stable and well maintained.
File systems should be boring, uncomplicated and just work. I personally like them to have a good 5-10 years of high production workloads with no big gotchas showing up before I use or recommend them. ZFS has lots more years than 10 in production now.
btrfs on routerOS on an amd64 NAS chassis, rsync'ed to an identical unit at my parent's. Pretty unusual but it has served me well with near-zero maintenence requirements: https://gir.st/blog/netadmin-nas.html
2x12TB external drives, with ext4 stacked on LVM mirroring on LUKS. rootfs is also encrypted and has the master key. Works great (:
On previous setups I had ZFS/btrfs and they all had quirks and were very complicated (it was ZFS-on-Linux; NixOS). This has just worked throughout constant daily writes. The second external drive was added about a year after the first one, which worked well too (though it was a total hassle to figure out the commands). Stuff like unplugging one of them to take with me as a RO-snapshot works great and things can synchronize when reconnected to the correct host.
(ZFS never lost me data, it was just complicated and assumed I was willing to dedicate it lots of RAM for the ARC cache.)
I am probably quite unique for better or more likely for worse.
But I use Arch Linux and ZFS for my NAS and I love it. It's been incredibly stable for me, I have very few packages and only rely on one AUR package.
I am considering moving to another distribution.
btrfs with RAID1 on disks that I already had: 8TB, 3TB, 2TB, 2TB disks.
I do this with btrfs, since it was easily available, is default in some distros, and works for me.
I see zfs is more popular for NASs so I looked into it, but AFAICT, it didn't allow me to do RAID1 with the random mix of disk's that I happen to own. Or maybe it could, but not as conveniently, e.g. if I wanted to replace a 2TB disk with a 4TB? I didn't look too much though.
But recently I've considered NetBSD as a remote NAS on an old machine (sparc32, oh dear), and realised that I won't be able to "btrfs send" to it, whereas I could if I used zfs, since it's cross platform.
I haven't experienced btrfs stability issues, though I happen to only use basic (?) features, which are listed as stable in https://btrfs.readthedocs.io/en/latest/Status.html . I'm not sure what's going on with the features listed as "mostly ok".
I see zfs now supports RAID-Z expansion, which offers some flexibility on incrementally growing an array: https://freebsdfoundation.org/blog/openzfs-raid-z-expansion-a-new-era-in-storage-flexibility/
My understanding is that if the new disk is larger than the others, then the excess is wasted, until all disks are upgraded to that new size.
Or you could upgrade your whole array, and then there's no wasted space at a fs level, but my storage needs grow gradually, so the space will still be defacto wasted.
For an enterprise that has 80% disk space utilisation across N NASs that can consolidate onto 80% disk space utilisation across N/4 larger NASs, then this wastage doesn't apply to them. But that's not my situation.
btrfs seems to avoid a lot of these considerations, since I can mix and match at will.
Of course, mirroring is itself a waste compared to parity, but if it's at the cost of wasted excess space, I'm not sure how it all balances. I'd love to see an analysis here.
Ext4 on a single 4TB SATA SSD, on a Raspberry Pi 4, no RAID. It’s running Raspbian because it’s the system that supports this hardware the best (trust me, I tried alternatives!) and ext4 is the default file system. And it’s fine. I’m mainly using it to backup my MacBook with Time Machine and Samba.
I got that SSD for cheap, back in the day…
I’m currently using bcachefs which is a bit of a choice i’m aware. I wanted to be able to do hot block storage on an ssd whilr keeping the bulk of my pool on 2 mirrored 6tb disk drives and bcachefs lets me do that and is also a new interesting format!
No complaints so far but it’s only been running in this configuration for a month or so. Was previously a ZFS pool with various drive amalgamations for the past 4 ish years.
I use synology with two drives which I think is btrfs but I have wanted to go to a linux system and not sure exactly how I would transition but wanted to go to 4 drives.
I'm using ZFS because I've been told good things about its reliability, error tolerance, snapshotting, and replication features.
I have a 2-disk mirror in the NAS, an offline backup drive, and will eventually™ set up an offsite automatic backup at a relative's house.
ZFS mirror on GNU Guix Linux. And tarsnap for backups. Want to try rsync.net but haven't gotten around it.
Xfs on lvm on luks on mdadm raid5.
XFS because of no inode issues and ultra stable and fast filesystem. Luks because of crypto, raid5 for cheap redundancy.
ZFS (managed by Proxmox) with RAID-Z2, snapshots, local backups (on separate machines), and remote backups.
ExFAT, so that when I need to restore things in a decade any machine will be able to read it.
ext4 on LVM on 4×8TB RAID10, created with mdadm in a way that should have spread data across drives of different brand
off-site backups using borg
used to use btrfs in some 3-drive RAID variant until one drive gave up, losing all my data somehow.
ZFS - easy, nice snapshots (so backup data from any time is easily available), flexible. Mirrored spinning drives.
FreeBSD on RPi - lightweight and doesn't include loads of extra packages that need to be updated every few days. ZFS is standard in FreeBSD. Inexpensive.
One machine for local backups and another for offsite - belt and braces.
Simple scripts run on server, uses rsync to drag data onto backup server. Backups have the same structure as each local machine, with per-day snapshots (per-week for older data). Nothing is hidden away in archive files - I can see the backed-up data and am unlikely to be stuck with backups don't work.
I have tried to minimise dependency on software packages, which reduces introduction of bugs and dragging-in more and more other packages. (Seen the hassles that duplicity reintroduces every few months!) Every extra package is extra bugs and extra updates. Only access to backup data is from and controlled by the backup server - local machines have zero access/control to backup machine.
Nothing is completely bombproof but this gives me reasonable security against hardware failure, my stupidity, maliciousness, and minimises my dependency on backup software that I don't control.
Ext4, RAID1 mirrored disks with mdadm. Reason: ZFS is too complex to ever reach ext4's reliability.
Debian + ZFS, 4 drives in raidz1
Debian for the stability and familiarity of linux, ZFS due to (at the time) btrfs' reputation of data loss
Second-site backup to BorgBase.com via borg. No third backup location right now, which isn't too great.