Apache Ozone has multiple 100+ petabyte clusters in production. The capacity is on HDDs and metadata is on SSDs. Updated docs (staging for new docs): https://kerneltime.github.io/ozone-site/
https://github.com/oneconcern/datamon
Had written this git for data tool few years back (works with GCS but can be made to work with S3)
1. No server side
2. Immutable data (via GCS policies)
3. Ability to mount data sets as filesystems
4. Integrated with k8s.
It was built to work for the needs of the startup funding it, but I would love it if it could be extended.