HackerLangs
TopNewTrendsCommentsPastAskShowJobs

Eikon

no profile record

Submissions

Show HN: ZeroFS – A log-structured filesystem for S3

zerofs.net
128 points·by Eikon·9 hari yang lalu·52 comments

Making POSIX filesystems replicated and highly available

zerofs.net
3 points·by Eikon·12 hari yang lalu·0 comments

Show HN: ZeroFS – Make S3 your primary storage

zerofs.net
11 points·by Eikon·29 hari yang lalu·7 comments

Show HN: PostgreSQL running in the browser, persisting to S3

zerofs.net
1 points·by Eikon·3 bulan yang lalu·1 comments

Show HN: PostgreSQL running in a browser tab, persisting to S3

zerofs.net
2 points·by Eikon·3 bulan yang lalu·0 comments

Static Certificate Transparency

merklemap.com
16 points·by Eikon·7 bulan yang lalu·1 comments

Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage

github.com
64 points·by Eikon·10 bulan yang lalu·40 comments

comments

Eikon
·8 hari yang lalu·discuss
Kind-of should work :)
Eikon
·9 hari yang lalu·discuss
NFSv4 is a hard beast to implement correctly, with a lot of protocol surface (state, compound ops, delegations) for benefits ZeroFS mostly gets through 9P with extensions, over a much simpler protocol: https://www.zerofs.net/docs/9p-extensions. NFSv3 stayed in ZeroFS mostly for client compatibility.
Eikon
·9 hari yang lalu·discuss
ZeroFS aims to be a POSIX filesystem, the semantics here are the standard ones (ext4, xfs behave the same): write() is buffered (that's the batching) and "committed" maps to fsync(), which returns only once data is durable.
Eikon
·9 hari yang lalu·discuss
Thank you, appreciated!
Eikon
·9 hari yang lalu·discuss
> I believe the first version of this required the metadata to be stored on the ZeroFS server, making HA kinda hard.

Metadata has always been in the bucket itself.

For HA, there's now a "replicated mode" if you want automatic failover:

https://www.zerofs.net/docs/high-availability
Eikon
·9 hari yang lalu·discuss
That's fair advice, thanks.
Eikon
·9 hari yang lalu·discuss
[dead]
Eikon
·9 hari yang lalu·discuss
[dead]
Eikon
·9 hari yang lalu·discuss
Is it? :)

Ask me anything!
Eikon
·9 hari yang lalu·discuss
Thank you for the feedback, the idea behind this was to say "We make claims that are backed by workflows you can verify". I'll improve the phrasing.
Eikon
·9 hari yang lalu·discuss
Just a self hosted plausible instance :)
Eikon
·16 hari yang lalu·discuss
It's there! https://www.zerofs.net/docs/kubernetes-csi
Eikon
·16 hari yang lalu·discuss
Most of the pain here is the typical set of issues people run into trying to make S3 a filesystem as-is, common with S3FS-family approaches.

ZeroFS (https://github.com/Barre/zerofs) is 9P/NFS/NBD over S3 on an LSM. Point stock go-git, or just /usr/bin/git, at a mount and skip the gymnastics. Rename is a metadata op in the keyspace, so you get it atomic on any S3, no Tigris-specific X-Tigris-Rename needed.

Different point on the spectrum, but less square-peg, also most probably much, much faster (it works great on linux-sized repos) :)
Eikon
·27 hari yang lalu·discuss
> Is this only was S3 or any S3 compliant service?

Any S3 compliant service.

> Does it support any sort of storage tiering ?

Yes, it has a local cache that's populated on reads, but no writeback cache yet.
Eikon
·28 hari yang lalu·discuss
> Millions of files, Interrupted transfers, Backup Server Reboots, Long-term operation?

Yes, that should work fine!
Eikon
·28 hari yang lalu·discuss
> I assessed the tool on a CentOS 7 system today. It can't run on older CentOS 6. It will generate core dump.

Perhaps your cpu is too old? The prebuilt binaries require newer CPUs but you can build your own.

> Will ZeroFS support large Filesystem size about 800GB?

Absolutely, and much, much more than that.
Eikon
·bulan lalu·discuss
Even with journaling, you might need one. ZeroFS [0] almost had a lost+found directory (even with the WAL enabled), because you might have consistency issues between your in-memory state and what was flushed, and especially in what order.

ZeroFS ended up not needing recovery at all through atomic, strictly ordered commits [1], but it was far from trivial (and not just a matter of requiring a WAL).

[0] https://github.com/Barre/ZeroFS

[1] https://github.com/Barre/ZeroFS/blob/main/zerofs/src/fs/writ...
Eikon
·2 bulan yang lalu·discuss
https://github.com/Barre/ZeroFS Should do a great job at this.
Eikon
·3 bulan yang lalu·discuss
It's definitely shared, and can be redundant.
Eikon
·3 bulan yang lalu·discuss
Or ZeroFS which doesn’t require a 3rd party database, just a s3 bucket!

https://github.com/Barre/ZeroFS