Hey HN, AWS S3 Express One Zone promises improved latency and throughput but comes with several limitations including only being available in one availability zone and the directory bucket structure.
We developed Accelerated Cloud Storage to offer an object storage system with high performance (up to 10X better than S3 Express One Zone in latency and throughput) and global availability that scales and moves data based on demand. The service is accessible through an S3 compatible Go/Python SDK. ACS has enterprise-grade security compliance (SOC 2 Type 2 pending), and a transparent pay-for-usage pricing model with no egress fees.
You can sign-up on the website or schedule a call with one of our engineers to discuss further.
Why do you say GCS performance isn’t an issue? I would imagine a highly parallel compute system would require higher throughput from object storage? I’m surprised you aren’t I/O bound.
If you want object storage faster than S3 Express One Zone or GCP Rapid Storage without the zonal limitation check out ACS: https://acceleratedcloudstorage.com
You can bring data in and out of the GPU quickly and improve utilization.
Hey, there are a couple differences between ACS and S3 Express One Zone.
One is speed, ACS provides up to 5X better performance (latency/throughput) than S3 Express One Zone. We have benchmarks repo here: https://github.com/Accelerated-Cloud-Storage/Benchmarks. Please let me know if you have any questions on this.
Another aspect of this is that ACS buckets are modeled on the traditional S3 bucket structure not the directory buckets of S3 Express One Zone. ACS buckets are not stored in only one availability zone and can be accessed in a performant manner from a region different than the region/az of bucket creation.
The service is currently available in US-East-1 on AWS. We plan on launching multi-region AWS support and SOC2 compliance soon. This should give companies the assurance to use ACS to store their data and the flexibility to use it wherever they are deploying their GPUs in AWS.
I understand your frustration with some of these AI startups, and there is a lot of hype in this space. However, there are a couple meaningful differences between ACS and Redis.
One difference is the engineering effort for integration. If an SWE was using Redis they would have to create custom polices on when to move the data between S3 and Redis, how to store it, for how long and when to move newly written data back to S3. This can impose a significant amount of engineering effort for a team. Redis is also mostly single-threaded whereas ACS is designed from the ground up to be multi-threaded.
Another difference is the interface. ACS is accessible through an S3 compatible Python/Go SDK or a high-performance FUSE mount for a POSIX-like file system interface. With Redis, you would still have to build an interface around it for other engineering teams or customers to use.
The purpose is different in that ACS is meant for datacenter workloads. ACS is optimized for both fast reads and writes to Object Storage. The architecture is different than a typical CDN.
We developed Accelerated Cloud Storage to offer an object storage system with high performance (up to 10X better than S3 Express One Zone in latency and throughput) and global availability that scales and moves data based on demand. The service is accessible through an S3 compatible Go/Python SDK. ACS has enterprise-grade security compliance (SOC 2 Type 2 pending), and a transparent pay-for-usage pricing model with no egress fees.
You can sign-up on the website or schedule a call with one of our engineers to discuss further.