Zrepl – ZFS replication(zrepl.github.io)
zrepl.github.io
Zrepl – ZFS replication
https://zrepl.github.io/
57 comments
I started using ZFS a few years ago. Before that I didn't know that I could develop such a strong feeling towards a file system. But here we are. In my mind, it really revolutionized the meaning of file system. It feels like much more than that, especially when compared to something like NTFS or ext4.
Similar story here (truenas was a gateway drug). There are a few up and coming filesystems that I have my eye on. But by and large, ZFS has remained my favorite.
Disclaimer: I'm a huge fan of ZFS.
ZFS is much more than just a filesystem and related tools. It also supports backup/restore and disk partitioning and partitioning of data via filesystems (not AFAIK available with EXT4.) And probably other things I either don;t know about or have forgotten to mention.
ZFS is much more than just a filesystem and related tools. It also supports backup/restore and disk partitioning and partitioning of data via filesystems (not AFAIK available with EXT4.) And probably other things I either don;t know about or have forgotten to mention.
I've been curious about ZFS ever since it became stable in Linux, but haven't made the jump yet because a) I have an irrational fear of data loss, and ext4 has never failed me, and b) I have an aversion towards complexity, and ZFS is much more complex than ext4. Features like snapshots and checksumming can be accomplished with userspace tools, though in arguably less elegant ways, so I didn't have a strong need for it either.
That said, it's great reading its success stories and praises, and my next system will use ZFS. :)
That said, it's great reading its success stories and praises, and my next system will use ZFS. :)
> I have an irrational fear of data loss, and ext4 has never failed me
You're a lot more likely to lose/corrupt data with ext4 than ZFS. Ext4 will happily corrupt data silently. The core conceit of ZFS is it doesn't trust the underlying hardware. ZFS even allows duplicated data on a single disk. You lose capacity but gain robustness.
You're a lot more likely to lose/corrupt data with ext4 than ZFS. Ext4 will happily corrupt data silently. The core conceit of ZFS is it doesn't trust the underlying hardware. ZFS even allows duplicated data on a single disk. You lose capacity but gain robustness.
Like I said, I've used ext* filesystems for decades now, and have yet to experience corruption or data loss. I know ZFS would detect and correct or notify this automatically, but this can be avoided with ext4 by doing frequent backups, and using tools like SnapRAID or par2. Sure, this needs to be done manually, but for my use cases this works rather well, and I enjoy the flexibility and direct control over this, rather than using a complex/magic file system that does this transparently for me.
When ZFS support in Linux was still new, I wouldn't dare rely on it, and it's the same reason I avoid btrfs today, even though its features are appealing. But now that ZFS is quite stable, and after hearing its praises for years, I do want to give it a try. :)
When ZFS support in Linux was still new, I wouldn't dare rely on it, and it's the same reason I avoid btrfs today, even though its features are appealing. But now that ZFS is quite stable, and after hearing its praises for years, I do want to give it a try. :)
> I've used ext* filesystems for decades now, and have yet to experience corruption or data loss
You might not know. How are you validating the integrity of every file regularly?
In the 90s I lost some files to corruption which went unnoticed for years and propagated to every backup, so by the time I went looking for the file I had years worth of backups of those files, all corrupt. This is one of the reasons zfs is such a happy place.
You might not know. How are you validating the integrity of every file regularly?
In the 90s I lost some files to corruption which went unnoticed for years and propagated to every backup, so by the time I went looking for the file I had years worth of backups of those files, all corrupt. This is one of the reasons zfs is such a happy place.
I get that. But in practice, it hasn't been a problem. For files I consider critical and difficult to replace, I use SnapRAID or par2, and others are either used frequently and I would notice the corruption, or replacing them wouldn't be a problem.
Using a smart filesystem would be an improvement, but it comes at the expense of less flexibility and more complexity, and, until recently on Linux at least, relying on unstable software.
Using a smart filesystem would be an improvement, but it comes at the expense of less flexibility and more complexity, and, until recently on Linux at least, relying on unstable software.
> relying on unstable software
Where the hell are you getting this from? What instability has existed in ZFS on Linux? Are you confusing it with BTRFS or something? I think your fears of ZFS are seriously misplaced.
Where the hell are you getting this from? What instability has existed in ZFS on Linux? Are you confusing it with BTRFS or something? I think your fears of ZFS are seriously misplaced.
> this can be avoided with ext4 by doing frequent backups
lol. Nothing is stopping you from doing manual backups using ZFS. One should never rely on just one backup anyways, if the data is critical. For me, snapshots are a great way to protect from "oh, I accidentally deleted this folder", which ext4 doesn't have. Yes, you can use replication to sync these snapshots somewhere else, but nothing is stopping you from continuing to do manual backups on a file level. It's just a file system, after all. So it doesn't really make sense as a justification as to why you are hesitant to use ZFS. In fact, it's one of the reasons I liked it so much: While you can do all these cool things with it, you don't have to. It doesn't pressure you to use these features. If you're ready, they're there, but until then, it's just a file system, and a very robust one at that.
lol. Nothing is stopping you from doing manual backups using ZFS. One should never rely on just one backup anyways, if the data is critical. For me, snapshots are a great way to protect from "oh, I accidentally deleted this folder", which ext4 doesn't have. Yes, you can use replication to sync these snapshots somewhere else, but nothing is stopping you from continuing to do manual backups on a file level. It's just a file system, after all. So it doesn't really make sense as a justification as to why you are hesitant to use ZFS. In fact, it's one of the reasons I liked it so much: While you can do all these cool things with it, you don't have to. It doesn't pressure you to use these features. If you're ready, they're there, but until then, it's just a file system, and a very robust one at that.
Potential complexity is higher with ZFS, but in regards to RAID and the like, I honestly find the zpool tools to be much easier to work with than their mdadm equivalents.
I have a "ghetto NAS" of 24 drives plugged into my server via USB, and getting a raidz3 set up on there was one command:
`zpool create tank raidz3 drive1 drive2 drive3 ....`
Doing scrubs and replacing disks are pretty easy, just using `scrub` and `replace` commands with zpool.
I haven't really felt the need to leave ZFS, at least not in regards to RAIDs; on root I still use ext4, though I might change to btrfs on my next install.
I have a "ghetto NAS" of 24 drives plugged into my server via USB, and getting a raidz3 set up on there was one command:
`zpool create tank raidz3 drive1 drive2 drive3 ....`
Doing scrubs and replacing disks are pretty easy, just using `scrub` and `replace` commands with zpool.
I haven't really felt the need to leave ZFS, at least not in regards to RAIDs; on root I still use ext4, though I might change to btrfs on my next install.
The main limitation I've had with ZFS is expanding it or modifying it is damn near impossible, you pretty much just have to find a second equal or larger storage pool, migrate, nuke, and rebuild. Even mdadm lets you do things like convert from raid5 to raid6 without having to nuke the pool.
Storage Spaces with ReFS has had the most impressive feature set from this perspective. Add and remove an arbitrary number of drives of arbitrary sizes and it'll use the full capacity of the drives to whatever parity level you set it to. It has its own downsides of course, on top of being Windows only, but it's the only FS/pool combo that has really made me think "ZFS doesn't have quite everything perfect".
Storage Spaces with ReFS has had the most impressive feature set from this perspective. Add and remove an arbitrary number of drives of arbitrary sizes and it'll use the full capacity of the drives to whatever parity level you set it to. It has its own downsides of course, on top of being Windows only, but it's the only FS/pool combo that has really made me think "ZFS doesn't have quite everything perfect".
There are definitely annoyances, but there are workarounds too; you can add additional vdevs to expand the whole pool if you want, and data is striped across them; it even lets you mix and match different raids if you really want to for some reason.
There's some current work to add a disk to existing vdevs, and I think even a semi-working PR for it now: https://github.com/openzfs/zfs/pull/15022. Hopefully in theory that will make ZFS a little less frustrating.
There's some current work to add a disk to existing vdevs, and I think even a semi-working PR for it now: https://github.com/openzfs/zfs/pull/15022. Hopefully in theory that will make ZFS a little less frustrating.
As a sibling comment mentioned, using ZFS for RAID/storage pooling is cumbersome. On my NAS I use plain old ext4 with SnapRAID+MergerFS, which gives me the RAID and checksumming features, while having the flexibility to expand the array using any combination of disks. This works rather well, and I have no need for immediate syncing/checking I would gain with ZFS.
That's my main issue with ZFS. It does too many things, and is too clever/magic for my taste. Many things can go wrong when relying on a monolithic system with that much complexity. I much prefer the Unix "do one thing well" approach, and mixing purpose-built tools to suit my needs, rather than using one tool for everything.
That's my main issue with ZFS. It does too many things, and is too clever/magic for my taste. Many things can go wrong when relying on a monolithic system with that much complexity. I much prefer the Unix "do one thing well" approach, and mixing purpose-built tools to suit my needs, rather than using one tool for everything.
I'm just learning too, so one question: why wouldn't you just consolidate and use ZFS for root? Even with just 1 disk, I understand it's still beneficial for its corruption detection and other such features. Just trying to understand that about going the btrfs way.
There's no reason not to really, other than btrfs is included in the kernel, and if you're not using RAID then I don't think there's a fundamental advantage to ZFS either. If you want to do ZFS on root, it usually requires more than "zero" levels of extra work, unlike btrfs.
> I have an irrational fear of data loss
If you have an irrational fear of data loss (it's really quite rational), zfs is the only place you should be comfortable.
I've been on zfs since its earliest days (~2004-ish, inside Sun at the time) and never lost a single byte ever since. Computers and hard drives have died but zfs just rocks on.
If you have an irrational fear of data loss (it's really quite rational), zfs is the only place you should be comfortable.
I've been on zfs since its earliest days (~2004-ish, inside Sun at the time) and never lost a single byte ever since. Computers and hard drives have died but zfs just rocks on.
ZFS is life-altering.
ZFS already includes all the functionality and commands you would need for replication—you _could_ do it yourself with snapshot, bookmark, send, and receive—but Zrepl is such a great way to define/manage it that it's always on my list of must-have ZFS tools.
I'm using zrepl to sync ZFS filesystems incrementally between servers.
I like the ability to incrementally send only the changes of an encrypted filesystem to a target server that never had the encryption key at anytime.
I'm using this to share backup space with my friends. We both push/pull our encrypted snapshot diffs every hour. I don't have my friends' keys so I can't read their data, and they can't read mine. In case of emergency, I can go to their place with my key, and recover my data from their systems.
I like the ability to incrementally send only the changes of an encrypted filesystem to a target server that never had the encryption key at anytime.
I'm using this to share backup space with my friends. We both push/pull our encrypted snapshot diffs every hour. I don't have my friends' keys so I can't read their data, and they can't read mine. In case of emergency, I can go to their place with my key, and recover my data from their systems.
Have you used sanoid/syncoid? I started my ZFS journey with a custom script and then switched to syncoid/sanoid (and some much simplified scripts) for my backup needs. One of my use cases is for Raspberry Pis which seem to have a habit of becoming unbootable. These are Pi 4Bs running from SATA/USB SSD or CM4s using NVME SSDs. I have scripts that use
* dd to copy the boot sector (to a ZFS filesystem) * rsync to copy the boot (FAT32) and root (EXT4) partitions (also to a ZFS filesystem) * syncoid to copy the ZFS filesystem to another host.
This process runs daily. When a system craps, it's pretty straightforward to restore the boot sector and boot and root partitions and finally the ZFS pool.
I'm curious zrepl does that might be useful for me.
* dd to copy the boot sector (to a ZFS filesystem) * rsync to copy the boot (FAT32) and root (EXT4) partitions (also to a ZFS filesystem) * syncoid to copy the ZFS filesystem to another host.
This process runs daily. When a system craps, it's pretty straightforward to restore the boot sector and boot and root partitions and finally the ZFS pool.
I'm curious zrepl does that might be useful for me.
I don't need to backup my boot sectors or root partitions since all my systems are provisioned. I can always re-create them with scripts, should any of them fail.
I'm only replicating the data filesystems.
From what I see, sanoid looks quite similar to zrepl. Both tools are probably able to achieve similar results.
I do feel that zrepl has more features so far, though. But hey, if your setup is working and your data is secure, that's the most important.
I'm only replicating the data filesystems.
From what I see, sanoid looks quite similar to zrepl. Both tools are probably able to achieve similar results.
I do feel that zrepl has more features so far, though. But hey, if your setup is working and your data is secure, that's the most important.
I opened a couple of issues and did some minor contributions in the past and at every point problame has been approachable and a pleasure to interact with.
From a user perspective, I'm still feeling a lack of UX in restoration/clone-management from the snapshots synced by zrepl - things can easily get messy if you don't know exactly what you're doing and keep track of things properly. That might be out of scope for zrepl itself, though.
Some more proactive/pre-emptive pruning would also be great - falling behind on syncing can be downright painful if you don't pay attention... Which is just to say, don't skip setting up some basic monitoring with alerts for production loads (:
Either way, all things considered zrepl is the nicest zfs replication option I found so far.
From a user perspective, I'm still feeling a lack of UX in restoration/clone-management from the snapshots synced by zrepl - things can easily get messy if you don't know exactly what you're doing and keep track of things properly. That might be out of scope for zrepl itself, though.
Some more proactive/pre-emptive pruning would also be great - falling behind on syncing can be downright painful if you don't pay attention... Which is just to say, don't skip setting up some basic monitoring with alerts for production loads (:
Either way, all things considered zrepl is the nicest zfs replication option I found so far.
If you haven't already seen it, for restoring files/browsing different snapshots etc, httm is great: https://github.com/kimono-koans/httm
Off topic: This is a great example why github stars don’t mean anything, such a great tool like this is ~600 while other repos that are basically toys compared to this have ~12k stars!
They do though, don't they? They indicate interest or popularity. Their purpose is not to demonstrate the quality or usefulness of a thing, but how popular it is.
When I choose libraries or tools for work, let's say I need a Go library for something; I use the star count and the commit activity together to determine whether a project is popular enough (vs others) to use over the others.
When I choose libraries or tools for work, let's say I need a Go library for something; I use the star count and the commit activity together to determine whether a project is popular enough (vs others) to use over the others.
Maybe users using such a tool are less likely to use stars.
I don't look at stars or fork count. To see if I should use a tool or a library, I'll be seeking technical blog posts and reviews.
I don't like how software development is centering around GitHub, and how it is handled like a social network.
Wha's more, you can't know why someone added a star. Maybe they found something random cool in the project, or found it looks cool but haven't really tried, and most importantly, you don't know if they added a star after some time, building some solid experience with the library or the tool.
This metric doesn't even exist for projects not developed on GitHub and many worthy ones are indeed elsewhere. And I suppose many of us don't use stars on GitHub, so you are only ever going to measure "popularity" aka "I liked something here" of only a part of the population that is willing to star things.
It's also probably a self-sustaining process: people are not going to give stars to alternative projects they haven't tried... because the first one had more stars in the first place! This feature is probably quite misleading in the end.
In a previous job, the number of forks and stars, as well as recent commits played an important choice when choosing a library. I hated this. It lead to some bad choices and technical dept.
In short, when you are considering stars, you are looking at a self-sustained process, sustained only by people who participate in it, and that is GitHub-specific (excluding interesting projects that happen to not be hosted there). This thing probably pushes the actual popularity of some projects for reasons that are likely very random, at times. GitHub is not making us a favor with this feature.
Commit history or activity in the bug tracker is somewhat important (though sometimes a project doesn't receive so many commits because it is stable enough). The social media gamification things on GitHub like counts and badges should just be ignored, don't fall for it and its addictive aspect.
I don't look at stars or fork count. To see if I should use a tool or a library, I'll be seeking technical blog posts and reviews.
I don't like how software development is centering around GitHub, and how it is handled like a social network.
Wha's more, you can't know why someone added a star. Maybe they found something random cool in the project, or found it looks cool but haven't really tried, and most importantly, you don't know if they added a star after some time, building some solid experience with the library or the tool.
This metric doesn't even exist for projects not developed on GitHub and many worthy ones are indeed elsewhere. And I suppose many of us don't use stars on GitHub, so you are only ever going to measure "popularity" aka "I liked something here" of only a part of the population that is willing to star things.
It's also probably a self-sustaining process: people are not going to give stars to alternative projects they haven't tried... because the first one had more stars in the first place! This feature is probably quite misleading in the end.
In a previous job, the number of forks and stars, as well as recent commits played an important choice when choosing a library. I hated this. It lead to some bad choices and technical dept.
In short, when you are considering stars, you are looking at a self-sustained process, sustained only by people who participate in it, and that is GitHub-specific (excluding interesting projects that happen to not be hosted there). This thing probably pushes the actual popularity of some projects for reasons that are likely very random, at times. GitHub is not making us a favor with this feature.
Commit history or activity in the bug tracker is somewhat important (though sometimes a project doesn't receive so many commits because it is stable enough). The social media gamification things on GitHub like counts and badges should just be ignored, don't fall for it and its addictive aspect.
Couldn’t agree more!
This has 695 stars, pyznap (what I’ve been using) has 187 stars, and sanoid has 2.6K stars. They all seem roughly at parity in terms of features, with one notable exception: Prometheus endpoint (and a preconfigured Grafana dashboard). I monitor mine with a fairly hacky script that parses logs and then sends a push to my phone. It’s very brittle, and I’ve found edge cases where it fails, but reports success.
I’ll probably give zrepl a try based solely on that.
I’ll probably give zrepl a try based solely on that.
I meant repos in general not specific to zfs/backup tools, without mentioning some of these repos as it might end up in flame wars.
Niche tools usually have low star count.
Can you be more specific?
What makes this so much better than others?
For me one of the key features is the integrated TLS server, so my backups can be transferred over the internet at decent speeds. Tools based on SSH all suffer from the SSH window limit and will never reach high speeds over the internet.
I can't find anything about this SSH window limit, or really anything around SSH being slower than TLS, aside from a limit to the number of concurrent sessions. Any links?
What are you talking about? Is this some '00s problems that were fixed long time ago and now you're repeating some myths ? I'm getting same (~300Mbit) speeds via nginx+ TLS as I get via SSH for just file download
It is indeed an old problem (small fixed size buffers) but it hasn't really been fixed, they only increased the buffer size. You'll only run into this on high bandwidth, high latency connections, e.g. between NYC and AMS - you'll be limited to about 300mbps with stock OpenSSH.
https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq/ has a patchset that fixes the problem.
https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq/ has a patchset that fixes the problem.
I missed that when I was comparing them - good point. Right now my ZFS backups are on the same LAN, and stuff going off-site isn’t using send/recv, but this is an excellent point.
You have just blown my mind. I can't believe I did not consider this before. Much thanks!
I do my zfs snapshots with a few lines of script with what zfs offers out of the box.
I like this script but never heard from it and probably will not look at it for a while.
I like this script but never heard from it and probably will not look at it for a while.
I have rather nice experience with ZnapZend https://www.znapzend.org/
Only issue i've found is the fact there is no official Debian repository and i always do forget to recompile it after upgrading Debian :-D
I really like the configuration system integrated in ZFS set/get...
Only issue i've found is the fact there is no official Debian repository and i always do forget to recompile it after upgrading Debian :-D
I really like the configuration system integrated in ZFS set/get...
zrepl works perfectly with an rsync.net account as a target:
https://news.ycombinator.com/item?id=32014969
Here's a nice summary someone wrote:
https://blog.lenny.ninja/zrepl-on-rsync-net.html
https://news.ycombinator.com/item?id=32014969
Here's a nice summary someone wrote:
https://blog.lenny.ninja/zrepl-on-rsync-net.html
How timely... literally spent the last hour attempting to recover data from 7 year old ZFS volumes. The conclusion was I think they must have been LVM'd or something before ZFS pool creation because cfdisk and hexdump -C show ZFS pool evidence but zpool-import won't scan them for love nor money. Only other option is an older-than-thou ZFS kernel module version with breaking changes. Sigh. Calling it a fail. No anticipated data of any weight and meaning, IIRC, but one likes to be thorough.
Also had a dead 6TB hard disk this morning: WD60EZAZ. Tried replacing the PCB as I had a few functional ones (ex RAID array), happened to have the right security bit, but no dice. You can hear it try to take-off repeatedly but it never achieves altitude. It's been awhile!
Also had a dead 6TB hard disk this morning: WD60EZAZ. Tried replacing the PCB as I had a few functional ones (ex RAID array), happened to have the right security bit, but no dice. You can hear it try to take-off repeatedly but it never achieves altitude. It's been awhile!
Hm... According to commit activity and high number of issues, looks like this project is almost dead. Anyway I tried to use it once and I got an impression it's too complicated and unreliable. I don't remember any details.
I did not understand what is ZFS - could anyone be so kind to explain?
ZFS is an advanced file system with integrated multi-device volume management. It was originally designed and implemented by Sun Microsystems for their Unix operating system Solaris. The copyright for the original codebase is now owned by Oracle and has since been closed up and made proprietary software. Prior to Oracle's acquisition of Sun, ZFS was made open source under the CDDL as part of OpenSolaris, that code was forked and it gave rise to OpenZFS which is used by many Linux and BSD users.
https://en.wikipedia.org/wiki/ZFS
https://openzfs.org
https://en.wikipedia.org/wiki/ZFS
https://openzfs.org
Here are a couple of introductions.
https://news.ycombinator.com/item?id=25157491
https://www.linuxjournal.com/content/zfs-linux
https://news.ycombinator.com/item?id=25157491
https://www.linuxjournal.com/content/zfs-linux
It's like someone glued together LVM and RAID but on file level instead of block level.
So you can make some fancy things like (relatively) cost-free snapshots, and some extra data safety with per-file checksumming. It also uses WAL (similarly to databases) which allows (similiar to databases) replication to remote system.
So you can make some fancy things like (relatively) cost-free snapshots, and some extra data safety with per-file checksumming. It also uses WAL (similarly to databases) which allows (similiar to databases) replication to remote system.
Got it, that makes sense! Not OP but a followup - what does "replication" mean in this context? Is it mirroring your ZFS volumes in another device?
Pretty much, although it's one-directional transfer of a volume or snapshot of one, not bi-directional one
really excellent software I have been using for a long time to manage snapshots!