Switching from S3 to Tigris on Fly.io(benhoyt.com)
benhoyt.com
Switching from S3 to Tigris on Fly.io
https://benhoyt.com/writings/flyio-and-tigris/
21 comments
1. You're right, it's somewhat apples and oranges. However, from what I can tell, that's part of the selling point of Tigris: you get CDN-like features "for free", without having to set up an extra layer like Cloudfront.
2. I actually do describe my concerns about durability near the end: https://benhoyt.com/writings/flyio-and-tigris/#disclaimer-an...
2. I actually do describe my concerns about durability near the end: https://benhoyt.com/writings/flyio-and-tigris/#disclaimer-an...
1. It takes 10 mins to setup cloudfront with s3 as a source, heck 3 mins if one is experienced, it cannot be a selling point for a provider,
But having an s3 like provider directly inside fly.io, is a pretty cool selling point by itself :D
2. Indeed, noticed the disclaimer now, was just commenting overall in the hn post :D
Have a great day !
But having an s3 like provider directly inside fly.io, is a pretty cool selling point by itself :D
2. Indeed, noticed the disclaimer now, was just commenting overall in the hn post :D
Have a great day !
Hi, co-founder of Tigris here. There is a key difference between Tigris and a CDN. Tigris doesn't simply provide read caching like CloudFront. It allows what I call dynamic caching, i.e., you can literally update the object anywhere in the world, and any cached versions of the objects will be updated. So, I think of Tigris simply as a global object storage service, and the fact that it has caching built-in is more to provide fast performance and low latency than simply to mimic CDN behavior. But if you want to build a CDN, Tigris makes it trivial for a single developer to build a CDN.
> It allows what I call dynamic caching, i.e., you can literally update the object anywhere in the world, and any cached versions of the objects will be updated.
how much it typically takes to propagate the changes? Also what is the consistency model of the storage?
how much it typically takes to propagate the changes? Also what is the consistency model of the storage?
> No mention on reliability, a lot of S3 alternatives have spawned up over the years, and they’ve all faced data corruption issues to some extent, that can be horrendous for a lot of object storage usecases.
Data corruption is horrendous for all use cases. But if the use case is backups, it’s particularly horrendous.
When you say “all”, do you mean the other big two players too? Or just the new crop of smaller ones?
Data corruption is horrendous for all use cases. But if the use case is backups, it’s particularly horrendous.
When you say “all”, do you mean the other big two players too? Or just the new crop of smaller ones?
I haven’t personally heard dataloss issues with big 3, but have previous read and heard about dataloss (very rare) in other smaller startups.
I remember wasabi one time losing the data of a customer (no comments tho, its very rare and they are significantly cheaper than big 3, so you get what you pay for)
But personally, I’d never touch azure, they get hacked every 3 months or so, get some sort of a fatal bug on their critical products, google and aws much better in that regard.
Personally I do not trust object storage much at all, and like keeping an offline backup of aws s3 objects (its not a snapshot, but rather and continuously updating locally stored disk backup, that tracks all S3 changes with s3’s etag parameter and a postgres table, ofcourse might be hard to continue at larger scales.
An alternative to AWS S3 only makes sense from a pricing standpoint, otherwise they are the king, my favourite feature is their AWS S3 Upload Accelerator, which speeds up user uploading of assets to your object storage, very handy for my current startup.
But yea, overall I’d be very wary of using AWS S3 alternative except for the most simplest of usecases. It’s a very polished, well integrated product from amazon at this point. Most startups in the S3 space, do. not. compare.
I remember wasabi one time losing the data of a customer (no comments tho, its very rare and they are significantly cheaper than big 3, so you get what you pay for)
But personally, I’d never touch azure, they get hacked every 3 months or so, get some sort of a fatal bug on their critical products, google and aws much better in that regard.
Personally I do not trust object storage much at all, and like keeping an offline backup of aws s3 objects (its not a snapshot, but rather and continuously updating locally stored disk backup, that tracks all S3 changes with s3’s etag parameter and a postgres table, ofcourse might be hard to continue at larger scales.
An alternative to AWS S3 only makes sense from a pricing standpoint, otherwise they are the king, my favourite feature is their AWS S3 Upload Accelerator, which speeds up user uploading of assets to your object storage, very handy for my current startup.
But yea, overall I’d be very wary of using AWS S3 alternative except for the most simplest of usecases. It’s a very polished, well integrated product from amazon at this point. Most startups in the S3 space, do. not. compare.
Oh, looks like Tigris pivoted from Mongo alternative to S3 provider... [1]. Probably they are still based on FoundationDB?
It seems like they charge for outbound traffic? Why would someone use Tigris instead of R2? [2][3]
--
1: https://web.archive.org/web/20220812231246/https://www.tigri...
2: https://www.tigrisdata.com/docs/pricing/
3: https://developers.cloudflare.com/r2/pricing/
It seems like they charge for outbound traffic? Why would someone use Tigris instead of R2? [2][3]
--
1: https://web.archive.org/web/20220812231246/https://www.tigri...
2: https://www.tigrisdata.com/docs/pricing/
3: https://developers.cloudflare.com/r2/pricing/
Yes we are based on FoundationDB. We pivoted from OLTP database to a lower level product as we found that it makes more sense to innovate at the storage infra layer. There is lot of innovation already happening at the higher level database layer.
And we don’t charge for egress to internet.
If you are looking for an active-active multi-region read-anywhere write-anywhere object storage service then you would choose Tigris over R2 :)
And we don’t charge for egress to internet.
If you are looking for an active-active multi-region read-anywhere write-anywhere object storage service then you would choose Tigris over R2 :)
Can you elaborate on the differences with R2? I can't really follow from the succinct description :-p
Good question!
R2 seems to be quite cheap for the right use cases.
R2 seems to be quite cheap for the right use cases.
> Tigris paid me a small amount to try their beta and write about it, but they didn’t have a say in the content.
Yes, right. People pay for negative reviews all the time. The article can still have merit. But that disclaimer is manipulative. Better just say "Tigris paid me for this content" and nothing else. Also, a "small amount" can be anything, it's all relative to what the author thinks is small.
Yes, right. People pay for negative reviews all the time. The article can still have merit. But that disclaimer is manipulative. Better just say "Tigris paid me for this content" and nothing else. Also, a "small amount" can be anything, it's all relative to what the author thinks is small.
I don't find that manipulative. Agree "small amount" is ambiguous though.
Author here. I hear what you're saying. However, I tend to write articles like this even without being paid -- for example, I wasn't asked or paid when I wrote about switching my site to Fly.io originally: https://benhoyt.com/writings/flyio/
Also, as you'll see in this article, I'm not shy about describing issues I ran into or concerns I have: for example, I don't think it's good that they set Cache-Control to max-age=3600 by default, and I have concerns about their durability.
Also, as you'll see in this article, I'm not shy about describing issues I ran into or concerns I have: for example, I don't think it's good that they set Cache-Control to max-age=3600 by default, and I have concerns about their durability.
the post calls out replication and durability, and I'd be curious to know more there too. do they replicate within a fly region? S3 replicates to multiple AZs within an AWS region -- is there an equivalent for fly?
I'd be curious how this is actually implemented on fly too. does the tigris block store use fly volumes (https://fly.io/docs/reference/volumes/)? this doesn't seem to add up since fly volumes are on NVMe SSDs and are $0.15/GB/month, compared to tigris which charges $0.02/GB/month. is there some secret sauce for attaching hard drives to fly instances?
I'd be curious how this is actually implemented on fly too. does the tigris block store use fly volumes (https://fly.io/docs/reference/volumes/)? this doesn't seem to add up since fly volumes are on NVMe SSDs and are $0.15/GB/month, compared to tigris which charges $0.02/GB/month. is there some secret sauce for attaching hard drives to fly instances?
Believe me, there is a lot for us to write about. And we will be publicly posting about the various architecture choices we have made.
I would like to point out that Tigris has this feature called shadow buckets which, with one extra flag: `--shadow-write-through` also writes back to the source bucket.
So if you're worried about reliability, or simply want to use two providers for redundancy like I wanted, you can use this feature. I just setup a bucket on Cloudflare R2, made it the shadow bucket for Tigris, and now whatever is added to the Tigris bucket, is also immediately replicated to R2.
So if you're worried about reliability, or simply want to use two providers for redundancy like I wanted, you can use this feature. I just setup a bucket on Cloudflare R2, made it the shadow bucket for Tigris, and now whatever is added to the Tigris bucket, is also immediately replicated to R2.
Competition is good, but Fly.io has proven problematic based on my two years of production experience. Anyone requiring reliability, professional support should avoid using services with significantly less experience than their competitors, such as S3, Cloud Buckets, and R2. Personally, I do not trust any service that relies on Fly.io. This perspective may change, but so far, they have proven to be an unreliable partner.
I'm planning to migrate some block storage data to object storage this year and since I'm hosting on Fly, this could be interesting.
What about egress costs?
There is zero $$ to pay for egress to internet: https://www.tigrisdata.com/docs/pricing/
1- The author compares performance of this provider Tigris with AWS S3 and how the latencies are 300ms and 1s respectively for the test image, fully aware that Tigris built on top of fly.io comes with a cdn combined, unlike S3, a fair comparison would have been comparing Tigris with AWS S3 + AWS Cloudfront(AWS CDN) combination (both in terms of cost savings and speed)
2- No mention on reliability, a lot of S3 alternatives have spawned up over the years, and they’ve all faced data corruption issues to some extent, that can be horrendous for a lot of object storage usecases. It sucks to put down new companies who are trying innovation, but storage layer as crucial as object storage requires reputation and proven resiliency often times.
Still pretty cool project ! Props to fly.io and tigris for making new alternatives to AWS.