HackerLangs
TopNewTrendsCommentsPastAskShowJobs

Eikon

no profile record

Submissions

Show HN: ZeroFS – A log-structured filesystem for S3

zerofs.net
128 points·by Eikon·há 9 dias·52 comments

Making POSIX filesystems replicated and highly available

zerofs.net
3 points·by Eikon·há 12 dias·0 comments

Show HN: ZeroFS – Make S3 your primary storage

zerofs.net
11 points·by Eikon·há 29 dias·7 comments

Show HN: PostgreSQL running in the browser, persisting to S3

zerofs.net
1 points·by Eikon·há 3 meses·1 comments

Show HN: PostgreSQL running in a browser tab, persisting to S3

zerofs.net
2 points·by Eikon·há 3 meses·0 comments

Static Certificate Transparency

merklemap.com
16 points·by Eikon·há 7 meses·1 comments

Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage

github.com
64 points·by Eikon·há 10 meses·40 comments

comments

Eikon
·há 8 dias·discuss
Kind-of should work :)
Eikon
·há 9 dias·discuss
NFSv4 is a hard beast to implement correctly, with a lot of protocol surface (state, compound ops, delegations) for benefits ZeroFS mostly gets through 9P with extensions, over a much simpler protocol: https://www.zerofs.net/docs/9p-extensions. NFSv3 stayed in ZeroFS mostly for client compatibility.
Eikon
·há 9 dias·discuss
ZeroFS aims to be a POSIX filesystem, the semantics here are the standard ones (ext4, xfs behave the same): write() is buffered (that's the batching) and "committed" maps to fsync(), which returns only once data is durable.
Eikon
·há 9 dias·discuss
Thank you, appreciated!
Eikon
·há 9 dias·discuss
> I believe the first version of this required the metadata to be stored on the ZeroFS server, making HA kinda hard.

Metadata has always been in the bucket itself.

For HA, there's now a "replicated mode" if you want automatic failover:

https://www.zerofs.net/docs/high-availability
Eikon
·há 9 dias·discuss
That's fair advice, thanks.
Eikon
·há 9 dias·discuss
[dead]
Eikon
·há 9 dias·discuss
[dead]
Eikon
·há 9 dias·discuss
Is it? :)

Ask me anything!
Eikon
·há 9 dias·discuss
Thank you for the feedback, the idea behind this was to say "We make claims that are backed by workflows you can verify". I'll improve the phrasing.
Eikon
·há 9 dias·discuss
Just a self hosted plausible instance :)
Eikon
·há 16 dias·discuss
It's there! https://www.zerofs.net/docs/kubernetes-csi
Eikon
·há 16 dias·discuss
Most of the pain here is the typical set of issues people run into trying to make S3 a filesystem as-is, common with S3FS-family approaches.

ZeroFS (https://github.com/Barre/zerofs) is 9P/NFS/NBD over S3 on an LSM. Point stock go-git, or just /usr/bin/git, at a mount and skip the gymnastics. Rename is a metadata op in the keyspace, so you get it atomic on any S3, no Tigris-specific X-Tigris-Rename needed.

Different point on the spectrum, but less square-peg, also most probably much, much faster (it works great on linux-sized repos) :)
Eikon
·há 27 dias·discuss
> Is this only was S3 or any S3 compliant service?

Any S3 compliant service.

> Does it support any sort of storage tiering ?

Yes, it has a local cache that's populated on reads, but no writeback cache yet.
Eikon
·há 28 dias·discuss
> Millions of files, Interrupted transfers, Backup Server Reboots, Long-term operation?

Yes, that should work fine!
Eikon
·há 28 dias·discuss
> I assessed the tool on a CentOS 7 system today. It can't run on older CentOS 6. It will generate core dump.

Perhaps your cpu is too old? The prebuilt binaries require newer CPUs but you can build your own.

> Will ZeroFS support large Filesystem size about 800GB?

Absolutely, and much, much more than that.
Eikon
·mês passado·discuss
Even with journaling, you might need one. ZeroFS [0] almost had a lost+found directory (even with the WAL enabled), because you might have consistency issues between your in-memory state and what was flushed, and especially in what order.

ZeroFS ended up not needing recovery at all through atomic, strictly ordered commits [1], but it was far from trivial (and not just a matter of requiring a WAL).

[0] https://github.com/Barre/ZeroFS

[1] https://github.com/Barre/ZeroFS/blob/main/zerofs/src/fs/writ...
Eikon
·há 2 meses·discuss
https://github.com/Barre/ZeroFS Should do a great job at this.
Eikon
·há 3 meses·discuss
It's definitely shared, and can be redundant.
Eikon
·há 3 meses·discuss
Or ZeroFS which doesn’t require a 3rd party database, just a s3 bucket!

https://github.com/Barre/ZeroFS