Ask HN: How Reliable Is Btrfs?
3 comments
I am using btrfs with docker. As I do have huge docker images(oracle db) and docker is much faster as it don't need to copy all files. And btrfs solve this problem with CoW on block level
Most NAS manufacturers have been using Btrfs for many years, so it’s reliable enough. Nowadays I‘d choose ZFS though.
I've been daily driving Btrfs since 2019 on multiple machines of all sizes, from RAID10 HDD+SSD arrays on larger x86-64 systems, through single SSDs on Armv8-A AArch64, to eMMCs and HDDs in a RAID1 configuration on Armv7-A. I'm a heavy user of snapshots, reflinks, compression and deduplication on all of those systems too.
tl;dr: No data lost, some actually rescued, there still are a few foot guns to watch out for.
It's grown to be a pretty decent file system. In the early days, it was common for it to eat data or catch a lock-up every now and then, but around since Meta took over its maintenance, things have very much stabilized. I haven't personally observed any data loss on my systems that can be attributed to it since 2019. I occasionally run scrub, and it has actually detected a few cases of silent data corruption, which it repaired using a mirror, on some of my HDD-based systems. The only close thing to data loss I've had with it during this time is that a systemd journal file on one of the SSD-based systems somehow ended up with a few extents with an all-zero checksum. The actual file contents don't appear corrupted, however.
As far as the raw I/O performance goes, it's alright; acceptable as in no random lockups, but won't top any benchmarks [1]. One thing to pay attention to is to never have a lot of snapshots if you have quotas enabled. You can either have a snapshot-heavy configuration or quotas, not both. Going against this will have your system slow down to a crawl. There's technically this new feature that allows for a simpler quota accounting scheme, which should in theory solve performance issues, but I haven't tried it myself [2].
Another performance footgun: write-heavy files with No_COW/nodatacow unset. Those are most commonly VM disk images and databases. Due to the copy-on-write design of Btrfs, those files are very susceptible to high fragmentation. While disabling COW defeats most of the benefits of using such a file system, on Btrfs, you can apply it to a small subset of files as a workaround for fragmentation while keeping the rest of your files covered. Take a look at chattr(1), namely the +C option, and always verify if it was applied with lsattr(1). Btrfs only allows this attribute to be changed on directories (which will have new files inherit it) and files with no extents.
As for data safety footguns, don't use RAID5/6. They're still broken [3]. Everything else works without a problem [4]. In the past, there used to be another thing that was a big no-no, namely exposing a host to 2 different file systems with the same ID. Btrfs would traditionally recognize them as one file system in a RAID configuration, and things would go haywire from there. This was particularly annoying with block-level snapshots, like those of LVM. This is said to have been fixed now [5]. As always though, take backups!
[1] https://www.phoronix.com/review/linux-617-filesystems/3
[2] https://lwn.net/Articles/944371/
[3] https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5...
[4] https://btrfs.readthedocs.io/en/latest/Status.html
[5] https://lwn.net/Articles/930944/
tl;dr: No data lost, some actually rescued, there still are a few foot guns to watch out for.
It's grown to be a pretty decent file system. In the early days, it was common for it to eat data or catch a lock-up every now and then, but around since Meta took over its maintenance, things have very much stabilized. I haven't personally observed any data loss on my systems that can be attributed to it since 2019. I occasionally run scrub, and it has actually detected a few cases of silent data corruption, which it repaired using a mirror, on some of my HDD-based systems. The only close thing to data loss I've had with it during this time is that a systemd journal file on one of the SSD-based systems somehow ended up with a few extents with an all-zero checksum. The actual file contents don't appear corrupted, however.
As far as the raw I/O performance goes, it's alright; acceptable as in no random lockups, but won't top any benchmarks [1]. One thing to pay attention to is to never have a lot of snapshots if you have quotas enabled. You can either have a snapshot-heavy configuration or quotas, not both. Going against this will have your system slow down to a crawl. There's technically this new feature that allows for a simpler quota accounting scheme, which should in theory solve performance issues, but I haven't tried it myself [2].
Another performance footgun: write-heavy files with No_COW/nodatacow unset. Those are most commonly VM disk images and databases. Due to the copy-on-write design of Btrfs, those files are very susceptible to high fragmentation. While disabling COW defeats most of the benefits of using such a file system, on Btrfs, you can apply it to a small subset of files as a workaround for fragmentation while keeping the rest of your files covered. Take a look at chattr(1), namely the +C option, and always verify if it was applied with lsattr(1). Btrfs only allows this attribute to be changed on directories (which will have new files inherit it) and files with no extents.
As for data safety footguns, don't use RAID5/6. They're still broken [3]. Everything else works without a problem [4]. In the past, there used to be another thing that was a big no-no, namely exposing a host to 2 different file systems with the same ID. Btrfs would traditionally recognize them as one file system in a RAID configuration, and things would go haywire from there. This was particularly annoying with block-level snapshots, like those of LVM. This is said to have been fixed now [5]. As always though, take backups!
[1] https://www.phoronix.com/review/linux-617-filesystems/3
[2] https://lwn.net/Articles/944371/
[3] https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5...
[4] https://btrfs.readthedocs.io/en/latest/Status.html
[5] https://lwn.net/Articles/930944/
So, I’m wondering how reliable and performant BTRFS is these days? Do you use it, or do you still prefer other filesystems? Feel free to share your experience and preferences.