How much I’ve spent so far running my own Mastodon server on AWS(micahwalter.com)
micahwalter.com
How much I’ve spent so far running my own Mastodon server on AWS
https://www.micahwalter.com/how-much-ive-spent-so-far-running-my-own-mastodon-server-on-aws/
150 comments
Well what kind of federation is it handling atm? Because saying Elixir is awesome means nothing until you have 1000 MAU's all tooting and hollering at the same time, and you have to federate a bunch of messages.
That's where I actually like having the sidekiq component separate and scalable. Because it means I can make use of dynamic scaling in a cloud to work through the queues, then scale down when the queues are empty.
The Mastodon web service can run on minimal resources, even with 1000 MAU's, it's only Sidekiq that needs to scale, and that also only when there is large amounts of messages to federate.
That's where I actually like having the sidekiq component separate and scalable. Because it means I can make use of dynamic scaling in a cloud to work through the queues, then scale down when the queues are empty.
The Mastodon web service can run on minimal resources, even with 1000 MAU's, it's only Sidekiq that needs to scale, and that also only when there is large amounts of messages to federate.
My question is, why do you need separate processes to handle mostly async operations? You need to do HTTP requests and database queries, that is mostly waiting, there is no CPU needed for this. You can do it with one process and millions of tasks operating in concurrent fashion.
I don't really know how Mastodon does this with the Sidekiq, but what I've seen people with a very small amount of activities (we're talking about hundreds of thousands, or a few millions), need to do manual scaling of Sidekiq processes or threads. As a user, you should not be tuning these metrics to handle this kind of activity. A single beam process could handle all the WhatsApp messaging back in the days, so it can easily handle our relatively tiny fediverse servers with much less users.
I don't really know how Mastodon does this with the Sidekiq, but what I've seen people with a very small amount of activities (we're talking about hundreds of thousands, or a few millions), need to do manual scaling of Sidekiq processes or threads. As a user, you should not be tuning these metrics to handle this kind of activity. A single beam process could handle all the WhatsApp messaging back in the days, so it can easily handle our relatively tiny fediverse servers with much less users.
I need to do HTTP requests, database queries and queue processing.
Mastodon has chosen a Service oriented architecture for this, and I'm happy with that decision because it means I can keep the web and DB queries running on minimal resources, while scaling the queue processing up and down depending on current activity.
One could say that incoming http requests from end users is a fairly predictable level of activity, while queue processing is a very unpredictable level of activity. Because it's tied into people's posts, their followers, the number of instances their followers are on, and so forth.
And with Mastodon v4 the resources required for end user http requests are even fewer, because they've shifted more processing onto the client with Javascript.
So I think the freedom to be able to scale the queue processing separately is fundamental to me. I can't imagine what it would be like to run a different software where I had to scale a monolithic service.
It doesn't really matter how fast the software is because queue processing is about blocking tasks, and they're blocking because they involve contact with other services and servers on the internet.
So yes I'd love to see a sidekiq replacement in Golang, Rust, or even Elixir, but I would not like to see it bundled in with other services. Mainly because queue processing deals with blocking tasks, and incoming HTTP requests should never be blocking.
Mastodon has chosen a Service oriented architecture for this, and I'm happy with that decision because it means I can keep the web and DB queries running on minimal resources, while scaling the queue processing up and down depending on current activity.
One could say that incoming http requests from end users is a fairly predictable level of activity, while queue processing is a very unpredictable level of activity. Because it's tied into people's posts, their followers, the number of instances their followers are on, and so forth.
And with Mastodon v4 the resources required for end user http requests are even fewer, because they've shifted more processing onto the client with Javascript.
So I think the freedom to be able to scale the queue processing separately is fundamental to me. I can't imagine what it would be like to run a different software where I had to scale a monolithic service.
It doesn't really matter how fast the software is because queue processing is about blocking tasks, and they're blocking because they involve contact with other services and servers on the internet.
So yes I'd love to see a sidekiq replacement in Golang, Rust, or even Elixir, but I would not like to see it bundled in with other services. Mainly because queue processing deals with blocking tasks, and incoming HTTP requests should never be blocking.
One of the main wins I've contributed to company after company is taking monoliths & making multiple instances of them.
Yes it can run one big process of everything, and does when I find it. But some routes or subservices are slow or hog cpu or use way more memory than the other routes. We'll have a handful of infrequently used routes that use >1GB while typically the monolith stays inder 200MB of ram. We'll have some expensive route that hog the even loop for a second. Mixing these routes in with the well behaved normal routes means we have to way over-provision, and means sporadic performance degredation once se lesser behaved routes start firing.
Separating your stuff up is a huge win. Being able to see ingest, egress, and web traffic separately & to manage them separately is a huge win. Im not sure what the win is from a single process. I have a hard time thinking of advantages. There's some small wins to memory, but a well designed forking architecture works around most of these advantages.
Yes it can run one big process of everything, and does when I find it. But some routes or subservices are slow or hog cpu or use way more memory than the other routes. We'll have a handful of infrequently used routes that use >1GB while typically the monolith stays inder 200MB of ram. We'll have some expensive route that hog the even loop for a second. Mixing these routes in with the well behaved normal routes means we have to way over-provision, and means sporadic performance degredation once se lesser behaved routes start firing.
Separating your stuff up is a huge win. Being able to see ingest, egress, and web traffic separately & to manage them separately is a huge win. Im not sure what the win is from a single process. I have a hard time thinking of advantages. There's some small wins to memory, but a well designed forking architecture works around most of these advantages.
I'd first wish the services would be written so that they do not use too much CPU to begin with. Ruby is horrendous with CPU usage, and the concurrency/threading in Ruby runtime is nothing to brag about.
A service such as a small community Fediverse server does not need microservices and complex architecture. What I've been seeing, people are kind of struggling with Ruby and Sidekiq even when having servers that are not super big. In the past, WhatsApp was having a small group of engineers with a handful of servers, serving hundreds of millions of users with Erlang. To get people running their own Fediverse servers, it should be as easy and as cheap as possible. Akkoma is a good start, and in a few months I hope we see the first proper systems made with Rust and Go, running with the minimal amount of resources, being single process and federating everything without the user needing to do any maintenance.
A service such as a small community Fediverse server does not need microservices and complex architecture. What I've been seeing, people are kind of struggling with Ruby and Sidekiq even when having servers that are not super big. In the past, WhatsApp was having a small group of engineers with a handful of servers, serving hundreds of millions of users with Erlang. To get people running their own Fediverse servers, it should be as easy and as cheap as possible. Akkoma is a good start, and in a few months I hope we see the first proper systems made with Rust and Go, running with the minimal amount of resources, being single process and federating everything without the user needing to do any maintenance.
I dont disagree on any point. I would still pick an implementation where I have control, management, & observability of the separate processes, such that I can see & trade off where resources are going.
Mastadon aint fast, but, this current architecture has been invaluable in letting folks degrade gracefully. Sidekiq for ingress gets backed up, but often the web service is totally responsive and egress sidekiq (if seperately configured) works fine. Picking a more effective runtime might give you a lot more headroom, but I'd still want a plan & ability to cope & degrade gracefully when the system does come under pressure, and Im not sure Erlang/Beam or any other single process system is going to have such innate ability to have the different things dealt with differently.
Bulkhead pattern is a truly great one- keep the whole ship from sinking when one part floods.
https://learn.microsoft.com/en-us/azure/architecture/pattern...
Mastadon aint fast, but, this current architecture has been invaluable in letting folks degrade gracefully. Sidekiq for ingress gets backed up, but often the web service is totally responsive and egress sidekiq (if seperately configured) works fine. Picking a more effective runtime might give you a lot more headroom, but I'd still want a plan & ability to cope & degrade gracefully when the system does come under pressure, and Im not sure Erlang/Beam or any other single process system is going to have such innate ability to have the different things dealt with differently.
Bulkhead pattern is a truly great one- keep the whole ship from sinking when one part floods.
https://learn.microsoft.com/en-us/azure/architecture/pattern...
This is the gist of what makes Django and Flask and such work fine even when you’re using some GIL-affected concurrency model like gevent. Most of the waiting for your typical web server is I/O.
It's a Pleroma fork, so I'd say it handles it pretty well.
Can Akkoma talk with Mastodon? Otherwise I think I miss the point of your post
Talks with Mastodon, Pleroma, PeerHub, Pixelfed, Misskey, Friendica and every other software using the ActivityPub protocol.
> every other software using the ActivityPub protocol.
Even GotoSocial now that they've put special handling in to work around their bizarre key-handling design decision.
Even GotoSocial now that they've put special handling in to work around their bizarre key-handling design decision.
Do you know anybody running GotoSocial? I'd love to try it, but I don't see that many servers in the Fediverse yet...
Offhand, no, other than myself and that's mainly for educational purposes (the source is Go and I understand most of it[1].)
fediverse.observer points to https://gotosocial.fediverse.observer/gotosocial.sequentialr... as being open.
[1] Unlike honk, which is also in Go, but was written with little regard for readability.
fediverse.observer points to https://gotosocial.fediverse.observer/gotosocial.sequentialr... as being open.
[1] Unlike honk, which is also in Go, but was written with little regard for readability.
[deleted]
[deleted]
It says it implements ActivityPub on the site so yes it should talk to Mastodon.
> Communicate with all your friends on the fediverse, no matter if they're on mastodon, pleroma, or any other fediverse software.
Probably? (Maybe it explodes if you try.)
Probably? (Maybe it explodes if you try.)
What type and size of storage are you using for this ?
Ah, I forgot to add that: I use Wasabi's S3 storage for files. This is probably the most amount of money I pay and a bit wasteful. We have about one or two images per month in uploads, but Wasabi is 5 euros a month for the first terabyte of storage. Egress is free though.
Re “egress is free”.
Note that “If your monthly egress data transfer is greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress policy”. There are reports of bill shock from mastodon users as a result of the costs due to the policy.
https://wasabi.com/paygo-pricing-faq/#free-egress-policy
Note that “If your monthly egress data transfer is greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress policy”. There are reports of bill shock from mastodon users as a result of the costs due to the policy.
https://wasabi.com/paygo-pricing-faq/#free-egress-policy
I'm all ears for a better solution. We probably will have less than a gigabyte of storage needs, and egress is what it is with some local caching from the servers. The only thing I want is to not store the files in the application server.
Looks like backblaze b2 and cloudflare as a cdn may have a more liberal egress policy. Your profile with wasabi may suit your needs, however. Another option is minio.
Yep, b2 and cf is a great combo for hobby project. Note that you still need to pay for transactions so depending on your usage pattern, b2 might range from very cheap to a nice discount.
I recommend turning on tiered cache on cf [0]. Our b2 bill went from $7 to $4.
[0] https://developers.cloudflare.com/cache/about/tiered-cache/
I recommend turning on tiered cache on cf [0]. Our b2 bill went from $7 to $4.
[0] https://developers.cloudflare.com/cache/about/tiered-cache/
Cloudflare R2? https://www.cloudflare.com/products/r2/
You'd likely be in the free tier.
You'd likely be in the free tier.
https://www.storj.io/pricing
they have cheaper storage and up t0 150GB is free so maybe try this one as well? i am on the fence for a dirt cheap S3 myself but B2 and cloudflare R2 is in the $10-15/TB range and something cheaper would be nice for a small usecase
they have cheaper storage and up t0 150GB is free so maybe try this one as well? i am on the fence for a dirt cheap S3 myself but B2 and cloudflare R2 is in the $10-15/TB range and something cheaper would be nice for a small usecase
I used to host a reverse proxy on the same box as the Mastodon instance and then have that be the origin for a bunny.net bucket. Worked great and I could make my webserver ensure that only my bucket was open.
Maybe I'm dumb and missing something obvious - what's wrong with selfhosting and another VM if you're doing >1gb ? You already have to deal with backups on postgres VMs ?
I use Storj. Its great.
https://www.storj.io/storj-home
https://www.storj.io/storj-home
Not sure about Akkoma but if architecture is similar to Mastodon, then your Wasabi bucket would include all the images and videos you see in your timelines, not just those that you upload.
Akkoma gives you the option of caching those and serving locally or simply referring back to the original server. They have probably chosen not to cache if the number of users is small.
At least a few years ago, Pleroma either used an actual proxy without caching or hotlinked images from other instances.
This is a well written post that explains how to tune an AWS single user instance deployment, and the costs that may be expected. It makes me feel that I’m getting rather good value at masto.host for my personal instance (I was on a waitlist for a few weeks).
What mastodon is missing is a good catalogue of reference deployments for 1, 10, 100, 1000, users on the major platforms (AWS, GCP, Azure, DO, OVH, Hetzner). Of course, “users” is a rather poor proxy for “activity” but it’s a start.
There are “scaling” articles that say “the first thing you’ll need to address is…” and breathless blogs of “what we did when we doubled registered users overnight”. But not so much if you’re starting an instance for a community that you know will be a certain size quickly. How many cores should I expect sidekiq to need to service each thousand typical users?
Sure, it’s a piece of cake for sysadmins or an AWS solutions architect, but it makes me wonder how many fragile instances there are out there that have grown fast but could also fail fast.
What mastodon is missing is a good catalogue of reference deployments for 1, 10, 100, 1000, users on the major platforms (AWS, GCP, Azure, DO, OVH, Hetzner). Of course, “users” is a rather poor proxy for “activity” but it’s a start.
There are “scaling” articles that say “the first thing you’ll need to address is…” and breathless blogs of “what we did when we doubled registered users overnight”. But not so much if you’re starting an instance for a community that you know will be a certain size quickly. How many cores should I expect sidekiq to need to service each thousand typical users?
Sure, it’s a piece of cake for sysadmins or an AWS solutions architect, but it makes me wonder how many fragile instances there are out there that have grown fast but could also fail fast.
> Sure, it’s a piece of cake for sysadmins or an AWS solutions architect, but it makes me wonder how many fragile instances there are out there that have grown fast but could also fail fast.
The way I read the post showcased how easy it was for the AWS SA to make insane billing mistakes with unneeded things like CloudWatch. They also hide the cost of EC2 because there's no charge for that instance type until 2023. This isn't a positive advertisement for AWS in my mind. This is in line with experience I've had with former AWS consumers I've converted over to DigitalOcean in 2022. I saved those conversions, on average, 70% of their prior AWS bill - with no loss of functionality. AWS has spun out of control with the nickel and dime aspect to billing. I can't say I'd recommend it to small or very large clients unless you have money to burn. At the end of the day "traditional" architectures on AWS are not cost effective without excessive oversight of choices.
The way I read the post showcased how easy it was for the AWS SA to make insane billing mistakes with unneeded things like CloudWatch. They also hide the cost of EC2 because there's no charge for that instance type until 2023. This isn't a positive advertisement for AWS in my mind. This is in line with experience I've had with former AWS consumers I've converted over to DigitalOcean in 2022. I saved those conversions, on average, 70% of their prior AWS bill - with no loss of functionality. AWS has spun out of control with the nickel and dime aspect to billing. I can't say I'd recommend it to small or very large clients unless you have money to burn. At the end of the day "traditional" architectures on AWS are not cost effective without excessive oversight of choices.
[deleted]
He talks about using an RI to reduce the monthly bill to under $5/month for compute. But in general, I think you're right. The value AWS offers for traditional architectures is for orgs running the numbers on lift-and-shift, compared to running on-prem infrastructure; greenfield projects should take advantage of cloud-native patterns.
It cracks me up that both AWS and especially Azure overcharge for logging to the point that it’s easy for logs to cost more than the underlying system they’re monitoring!
If you sit down and work out their cost of data processing, it costs them cents to provide a service for which they charge tens to hundreds of dollars.
I figure the reasoning is that large enterprise requires logging for compliance, and it’s a hidden charge that’s easy to overlook in a bill.
If you sit down and work out their cost of data processing, it costs them cents to provide a service for which they charge tens to hundreds of dollars.
I figure the reasoning is that large enterprise requires logging for compliance, and it’s a hidden charge that’s easy to overlook in a bill.
I also got a chuckle because the author is a professional AWS architect and still couldn't avoid accidental charges. Well, architect isn't salesman! Thankfully the mistake was corrected by paying the bill in full.
AFAIKT the AWS SA takes commission, and should indeed be considered a sales role.
It is a sales role (part of the sales org), but not paid comission - that is only account managers.
I would bet the cost would be lower if you use Digital Ocean/Linode and CloudFlare. Both have 5USD instances with no egress costs at these scales. No need to commit for three years or anything.
If you already own a domain you can use a subdomain and cut even that cost although it might make up for a long URL.
I think the main cost here is time. Probably far more expensive than all of those costs. Even on AWS.
If you already own a domain you can use a subdomain and cut even that cost although it might make up for a long URL.
I think the main cost here is time. Probably far more expensive than all of those costs. Even on AWS.
Came to say this. I think I see a lot of people default to AWS, maybe because their employer uses them and it's what they know. The downside is that AWS has all this expense for fancy stuff that can be done usually with a cheap instance, docker and nginx.
- Cloudflare for caching, DNS, domains. - Hetzner, Linode or DO for a cheap instance
See what does/doesn't work. Storagebox on Hetzner is cheaper storage hosting and you can have redundant backups for cheap on backblaze as well or offshore cheap VPS doing encrypted backups with rclone/rsync.
- Cloudflare for caching, DNS, domains. - Hetzner, Linode or DO for a cheap instance
See what does/doesn't work. Storagebox on Hetzner is cheaper storage hosting and you can have redundant backups for cheap on backblaze as well or offshore cheap VPS doing encrypted backups with rclone/rsync.
> I think I see a lot of people default to AWS, maybe because their employer uses them and it's what they know
I don’t get why people assume others are just mindless fruit flies in these threads. Maybe OP just doesn’t want to fiddle with technology and pays Amazon to simply get it done. That’s what I’d do, there’s no need to turn every project into excercise in orchestrating open-source software.
I don’t get why people assume others are just mindless fruit flies in these threads. Maybe OP just doesn’t want to fiddle with technology and pays Amazon to simply get it done. That’s what I’d do, there’s no need to turn every project into excercise in orchestrating open-source software.
AWS is wayyy too complicated to "simply get it done" for a single user. It makes no sense to learn all that if you just do it for yourself.
In fact I have to do a bit of AWS for work and I hate it with a passion. It's spread over a ton of services each with their own console (that look & feel like they were designed by completely different people), lots of overlap between services, complex billing. Configuration often needs to be done with complex json files. Ugh. Perhaps it makes sense for a major enterprise where you have orchestration for everything. But I would never pick this total mess for a personal project.
Azure has a bad name but at least everything is integrated in the same console pretty nicely.
But really if you just need an instance of something for personal use, DigitalOcean, Scaleway etc are so much simpler. And cheaper, and most stuff is just included in the monthly price so you don't get billing surprises.
In fact I have to do a bit of AWS for work and I hate it with a passion. It's spread over a ton of services each with their own console (that look & feel like they were designed by completely different people), lots of overlap between services, complex billing. Configuration often needs to be done with complex json files. Ugh. Perhaps it makes sense for a major enterprise where you have orchestration for everything. But I would never pick this total mess for a personal project.
Azure has a bad name but at least everything is integrated in the same console pretty nicely.
But really if you just need an instance of something for personal use, DigitalOcean, Scaleway etc are so much simpler. And cheaper, and most stuff is just included in the monthly price so you don't get billing surprises.
IIRC the author literally said he doesn’t know much about Cloud or AWS
aws is the ibm of cloud. As a CTO, I'm beholden both to my cofounders and our investors. I'd rather spend my energy defending or pushing novel engineering practices or directions for our tech stack rather than going to DO or hetzner. also AWS is huge and their reliability is damn good. Plus its relatively easy to get in contact with a real support engineer when shit its the fan. When you have a lot of customers depending on your software to "just work." That premium is worthwhile.
Yah totally but we aren’t really talking about enterprise grade stuff here man. We are talking about self hosting projects
For personal projects, I'd agree with you. sorry I misread your original statement. though I'd also throw linode into that ring
Nope it wouldn’t be. I run my single user instance on DO. I haven’t looked at the exact costs but they are higher.
First of all, a 5USD/mo instance cannot run mastodon. Mastodon needs a lot of resources from the get go, even with one user.
Second, you’ll want to enable backups, which is 20 percent of instance cost as extra.
Third, you will want more disk space than is available on a base instance. Mastodon is huge and disk space is taken up pretty quickly. With AWS you can set up autoscaling drives, DO doesn’t. So, more costs…
Finally, if you want to use DO Spaces, that is a flat 5USD/mo. AWS S3 is far cheaper for what would be used in this instance, on the order of free tier or couple of cents. The OP did the set up incorrectly which is why their costs are far higher. I actually did set up AWS S3 on my DO instance and the costs are extremely low.
So why am I using DO? I have credits on it, that’s all. I’ll move to AWS and probably set up Takahe at some point: https://docs.jointakahe.org/
First of all, a 5USD/mo instance cannot run mastodon. Mastodon needs a lot of resources from the get go, even with one user.
Second, you’ll want to enable backups, which is 20 percent of instance cost as extra.
Third, you will want more disk space than is available on a base instance. Mastodon is huge and disk space is taken up pretty quickly. With AWS you can set up autoscaling drives, DO doesn’t. So, more costs…
Finally, if you want to use DO Spaces, that is a flat 5USD/mo. AWS S3 is far cheaper for what would be used in this instance, on the order of free tier or couple of cents. The OP did the set up incorrectly which is why their costs are far higher. I actually did set up AWS S3 on my DO instance and the costs are extremely low.
So why am I using DO? I have credits on it, that’s all. I’ll move to AWS and probably set up Takahe at some point: https://docs.jointakahe.org/
Interesting. I'm mostly on Linode and don't run a Mastodon instance. I did the back of the envelope calculations based on numbers from google but I guess you would know better.
In Linode you can get additional block storage for a relatively low cost: https://www.linode.com/pricing/#block-storage So a 5 or 10USD instance should be enough.
I will say that S3 is very expensive not because of the upfront cost. It's because of traffic. Also cloudflare is more affordable than AWS.
In Linode you can get additional block storage for a relatively low cost: https://www.linode.com/pricing/#block-storage So a 5 or 10USD instance should be enough.
I will say that S3 is very expensive not because of the upfront cost. It's because of traffic. Also cloudflare is more affordable than AWS.
Traffic will be negligible on a single user instance. It’s also possible to put cloudflare in front of it (needs extra configuration I couldn’t figure out before my patience ran out though).
The main reason I bring up cloudflare is due to OPs discussion of cloudfront and S3. S3 is unusable and expensive without a CDN.
You could improve in some places.
I also use DO and first of all Mastodon itself, the web services, streaming API, can run on minimal nodes. I have almost 1000 MAU's now and only need about 600M RAM for each web service container instance.
And backups, for that I use postgres-operator (yes it's kubernetes), which does S3 backups and those cost almost nothing. (I know cost can go up if I need to restore because of egress traffic)
Media is also in S3, with a Cloudfront proxy. As soon as I enabled the CF proxy my S3 costs went down to 0.1 dollars basically.
And finally, thank the gods for kubernetes because it allows me to dynamically scale up sidekiq when the queues are backed up.
So my base cost for 1000 MAU's on DO is about 150USD/month. But it goes up if Sidekiq scales up temporarily, so it's a fluent final cost based on how many times Sidekiq had to scale up in a month.
Most importantly for me is that nothing in this setup is vendor locked. S3 compatible services are plenty, cloudfront web proxy services, plenty of those out there, AWS SES for outgoing e-mails can also be replaced. And of course kubernetes with proper IaC can be deployed into any managed k8s solution.
I also use DO and first of all Mastodon itself, the web services, streaming API, can run on minimal nodes. I have almost 1000 MAU's now and only need about 600M RAM for each web service container instance.
And backups, for that I use postgres-operator (yes it's kubernetes), which does S3 backups and those cost almost nothing. (I know cost can go up if I need to restore because of egress traffic)
Media is also in S3, with a Cloudfront proxy. As soon as I enabled the CF proxy my S3 costs went down to 0.1 dollars basically.
And finally, thank the gods for kubernetes because it allows me to dynamically scale up sidekiq when the queues are backed up.
So my base cost for 1000 MAU's on DO is about 150USD/month. But it goes up if Sidekiq scales up temporarily, so it's a fluent final cost based on how many times Sidekiq had to scale up in a month.
Most importantly for me is that nothing in this setup is vendor locked. S3 compatible services are plenty, cloudfront web proxy services, plenty of those out there, AWS SES for outgoing e-mails can also be replaced. And of course kubernetes with proper IaC can be deployed into any managed k8s solution.
This guy works at AWS. I’m guessing that’s the main reason he’s using it.
So it's a sneaky marketing blog then.
AWS - "Free to get data in, expensive to get out"
AWS - "Free to get data in, expensive to get out"
> sneaky marketing blog
What’s sneaky? OP states at the beginning of the post they’re a solutions architect at AWS… and provides reasons you may not want to use AWS if you want your own single user mastodon instance.
What’s sneaky? OP states at the beginning of the post they’re a solutions architect at AWS… and provides reasons you may not want to use AWS if you want your own single user mastodon instance.
[deleted]
Yeah, if you want to do this as cheaply as possible, you certainly don’t want to use AWS. In particular using s3 seems difficult to justify.
That said I assume the author is not trying for cheapest-possible.
That said I assume the author is not trying for cheapest-possible.
> Both have 5USD instances
DigitalOcean no longer has a USD 5 instance, it's USD 6 now.
DigitalOcean no longer has a USD 5 instance, it's USD 6 now.
They increased what was provided in the old $5 a couple of times, before raising the price to $6, though.
Also, there’s a new $4 entry-tier droplet that has the same CPU, the same amount of ram and the same amount of storage as my $5 droplet used to: https://www.digitalocean.com/pricing/droplets
The included bandwidth is only half as much (500GiB), but I suspect most people who would have gotten $5 droplets in the past will now be getting $4 ones.
Also, there’s a new $4 entry-tier droplet that has the same CPU, the same amount of ram and the same amount of storage as my $5 droplet used to: https://www.digitalocean.com/pricing/droplets
The included bandwidth is only half as much (500GiB), but I suspect most people who would have gotten $5 droplets in the past will now be getting $4 ones.
As far as I can remember, the $5 droplet always had 1 GB of RAM. The current $4 droplet is only 0.5 GB.
The AWS instance in the post, t4g.small, has 2 GB of RAM. So it's not really comparable to the $4 DO droplet.
The AWS instance in the post, t4g.small, has 2 GB of RAM. So it's not really comparable to the $4 DO droplet.
My $5/month droplet started at 0.5 GB and the free upgrade to 1 GB of ram wasn’t even that long ago. Perhaps the image on this link will jog your memory of the old $5/month droplet stats: https://tutorialforest.com/upgrade-digitalocean-droplet-plan...
Also, FWIW, I’ve load tested different VPSes between AWS, Linode and DO and found that for reasons I don’t understand, AWS ones perform considerably worse than the other two platforms VPSes with identically listed stats.
Also, FWIW, I’ve load tested different VPSes between AWS, Linode and DO and found that for reasons I don’t understand, AWS ones perform considerably worse than the other two platforms VPSes with identically listed stats.
Try Scaleway, stardust. You can only have 2 but they are 2 euro a month. And free bandwidth. I really like them (Note: I have mentioned them several times in this thread but I have no affiliation or commercial interest - just a happy customer).
Thanks for the update. I'm shocked they raised that.
I love Mastodon, or more accurately the fediverse in general. However, running your own instance seems like a loss in every possible way. It costs money, even if it's not much. It's work, even if it's not much. It incurs risk and liability. And it's functionally worse than being on a shared instance. Larger instances will take forever to federate with you, if they do at all. Your local timeline will be empty. You'll even have to jump through hoops to see all of the replies to posts, since the default is to show only replies from instances you've already federated with. If you often feel like you're only seeing half the conversation it's because you really are.
Lastly, the more people run their own instances, the worse everyone's experience will get. The fediverse almost entirely relies on direct communication between instances with no DHT or other mesh routing overlay, so messages are O(n^2) relative to number of instances. Caching works OK for assets but not for ActivityPub RPC, so many instances don't even implement it at all. One of the main reasons the large instances struggle so much is that these communication patterns drastically increase traffic volume and queue length, which is further exacerbated by the fact that Sidekiq (the part of Mastodon that handles those queues) is a poorly implemented tuning nightmare.
The best bet IMO is to join one of the medium-sized instances aligned around a community or vibe that you like (I'm on hachyderm BTW and it's awesome for me). Smaller ones have all of the problems I've mentioned above, while the very largest are sub-optimal in terms of both technical UX and social atmosphere. Goldilocks wins again.
Lastly, the more people run their own instances, the worse everyone's experience will get. The fediverse almost entirely relies on direct communication between instances with no DHT or other mesh routing overlay, so messages are O(n^2) relative to number of instances. Caching works OK for assets but not for ActivityPub RPC, so many instances don't even implement it at all. One of the main reasons the large instances struggle so much is that these communication patterns drastically increase traffic volume and queue length, which is further exacerbated by the fact that Sidekiq (the part of Mastodon that handles those queues) is a poorly implemented tuning nightmare.
The best bet IMO is to join one of the medium-sized instances aligned around a community or vibe that you like (I'm on hachyderm BTW and it's awesome for me). Smaller ones have all of the problems I've mentioned above, while the very largest are sub-optimal in terms of both technical UX and social atmosphere. Goldilocks wins again.
Yea this has been bothering me. I have to fight to ignore the desire to run my own and just enjoy my medium-sized instance.. because any interest i get in Mastodon itself makes me want to rewrite it to run more lean and allow users to contribute costs (P2P data).
I quite enjoy Mastodon, but the whole thing feels.. incorrectly aligned with technical needs. I hope the attention it's getting leads to core implementations more aligned with the community.
I quite enjoy Mastodon, but the whole thing feels.. incorrectly aligned with technical needs. I hope the attention it's getting leads to core implementations more aligned with the community.
Mastodon offers a lot of great features but at a high computational cost. I settled for GoToSocial for my personal instance, and although it is still in alpha, everything I need is there and it is stable and barely uses any resources. The setup and maintenance/upgrades is simple since there is very few moving parts, which makes it perfect to host on my home server.
I tried, and gave up, setting up Mastodon several times in the past as I got choice paralysis from all the different ways to set it up. That was a few years ago, but from what I hear, it is still a somewhat daunting task.
I tried, and gave up, setting up Mastodon several times in the past as I got choice paralysis from all the different ways to set it up. That was a few years ago, but from what I hear, it is still a somewhat daunting task.
AWS costs can destroy you, unless you’re using managed AWS services to replace bloated IT org full of support personnel not doing much productive work in first place.
If you really need scale, AWS can be cheaper than rolling your own infra and abstractions, granted you make it a mission to keep laser focus on AWS costs.
Primary ideal use case for AWS is internal Amazon usage. Charge internally but write off as expenses, for infra you had sitting idle anyway. Pretty sure AWS and rest of Amazon are separate legal entities to take advantage of tax laws.
If you really need scale, AWS can be cheaper than rolling your own infra and abstractions, granted you make it a mission to keep laser focus on AWS costs.
Primary ideal use case for AWS is internal Amazon usage. Charge internally but write off as expenses, for infra you had sitting idle anyway. Pretty sure AWS and rest of Amazon are separate legal entities to take advantage of tax laws.
I'm not so sure about the cheaper at scale.
Renting is usually more expensive than owning. The larger you get, the more expensive renting is compared to owning.
Renting is usually more expensive than owning. The larger you get, the more expensive renting is compared to owning.
AWS fits the niche of "holy fuck my site is blowing up and I don't want it to crumble while I onboard new users or some event is going on".
You probably shouldn't use it for day to day ops but it's a really good spillway for extremely temporary & particularly for unexpected spikes of traffic.
Of course you could argue that AWS is particularly bad but I think this applies to most non-bare-metal instance cloud services.
You probably shouldn't use it for day to day ops but it's a really good spillway for extremely temporary & particularly for unexpected spikes of traffic.
Of course you could argue that AWS is particularly bad but I think this applies to most non-bare-metal instance cloud services.
The best use cause is a system where you have vast spikes in compute requirements on rare occasions. You can then use AWS to get those resources for limited period of time. For this user case, it actually works pretty well.
For anything else, there are always going to be cheaper and better solutions.
For anything else, there are always going to be cheaper and better solutions.
> AWS costs can destroy you, unless you’re using managed AWS services to replace bloated IT org full of support personnel not doing much productive work in first place.
OTOH, if you have an understaffed infra team that gets by by accumulating technical debt on infrastructure, AWS takes away that option at several levels of abstraction.
OTOH, if you have an understaffed infra team that gets by by accumulating technical debt on infrastructure, AWS takes away that option at several levels of abstraction.
I host a Canvas LMS on Hetzner to run online courses. We sometimes have hundreds of students simultaneously giving quizzes and exams on the LMS.
I think we pay 16.81 Euros/month.
I am fairly sure a mastodon server should run on a server with half the stats and price.
I think we pay 16.81 Euros/month.
I am fairly sure a mastodon server should run on a server with half the stats and price.
Oracle Cloud "always free tier" has 24G of memory, 4 ARM CPUs, 200gb block storage, 200g object storage, a load balancer and some database stuff (may or may not be useful). I've been running a 4x node Kubernetes cluster for over a year, mostly for fooling around with.
I was able to get a free x86 box but it's been impossible for me to get an arm instance on the west coast always out of space. Otherwise oracle free tier is pretty ncie
I had the same issue after shutting down my instances. I wrote a simple bash loop to recreate them and left it running overnight. All instances were up by the morning. I haven't shit any down since then. Reboots are fine.
With a single-user instance and a relay, traffic is 99% inbound, and the biggest cost will be storage.
But once there are a few hundred users, Sidekiq jobs will use a lot of compute and egress data will add up. Might be better in this case to use a provider with generous free bandwidth.
But once there are a few hundred users, Sidekiq jobs will use a lot of compute and egress data will add up. Might be better in this case to use a provider with generous free bandwidth.
>The hidden cost of storing data for Mastodon on S3 is the Requests. There are so many requests! I’m seeing about 40-50K requests per day!
There are S3-compatible object storage providers that have no per-request fees, and significantly lower bandwidth and storage fees. Examples – OVH, Scaleway.
I had an use case for object storage where my service would be uploading tens to hundreds of small objects per second, 24-7. Running some napkin math, it looked like AWS S3 costs for me would have been in the hundreds. I went with OVH, and have a €1 monthly bill for the object storage :-)
There are S3-compatible object storage providers that have no per-request fees, and significantly lower bandwidth and storage fees. Examples – OVH, Scaleway.
I had an use case for object storage where my service would be uploading tens to hundreds of small objects per second, 24-7. Running some napkin math, it looked like AWS S3 costs for me would have been in the hundreds. I went with OVH, and have a €1 monthly bill for the object storage :-)
This is well written, but is only 30 days worth of data.
$20/month, not including compute, is their starting point on AWS. As they get more followers and follow more people that will continue to grow.
My biggest suggestion is to disable the relay. I took experimented with using a public relay, but the stream of content it adds is very low value. Better to create a second account on the instance that is more aggressive about following people your only tangentially interested in. This will make your federated feed interesting, but keep your main feed more focused.
To the people suggesting DO/Linode. I'm a big supporter of Linode, however article has no egress or compute costs. Likely because it's a single user instance running on the free tier. Their costs are almost all storage related.
$20/month, not including compute, is their starting point on AWS. As they get more followers and follow more people that will continue to grow.
My biggest suggestion is to disable the relay. I took experimented with using a public relay, but the stream of content it adds is very low value. Better to create a second account on the instance that is more aggressive about following people your only tangentially interested in. This will make your federated feed interesting, but keep your main feed more focused.
To the people suggesting DO/Linode. I'm a big supporter of Linode, however article has no egress or compute costs. Likely because it's a single user instance running on the free tier. Their costs are almost all storage related.
A good compromise I’ve found is using something like “FakeRelay”[0] with “GetMoarFediverse”[1] to pull in specific tags from specific larger servers. This acts as a way to get more interesting content in your search and other mastodon tabs while benefiting from the moderation efforts of larger servers. Always good to ask the admins of those servers before you do incase they are already at capacity and don’t want the extra load, and contribute to their patreon if they have one.
[0] https://github.com/g3rv4/FakeRelay [1] https://github.com/g3rv4/GetMoarFediverse
[0] https://github.com/g3rv4/FakeRelay [1] https://github.com/g3rv4/GetMoarFediverse
I find my aws billing statements to be so confusing that I lean towards other cloud platforms for my personal projects. Whenever I can at least. It is easier to create a set of load balanced servers involving a dozen aws services than to understand a bill on there
Get fiber and host this from your own home. When i see these absurd costs I'm happy with the k8s I built, with ceph for storage.
I do this, but just with straight docker compose. I’ve been using AWS for personal and work for about 8 years and realised over time it is a poor investment if you are always hosting your own stuff and experimenting with tech.
So this year I bought at old Dell T7910 workstation and fitted it out with a 14 core E5-2680 V4 (second one coming), 1TB nvme, 64gb RAM and has cost me around $400 USD. Runs at around 95 watt and I already have a good fibre connection any way.
For a similar spec machine I would be paying this every month and now I can just keep throwing things I’m interested in at it and so far has been overkill for a relatively fixed price. More messing around with hardware but something I mostly enjoy.
So this year I bought at old Dell T7910 workstation and fitted it out with a 14 core E5-2680 V4 (second one coming), 1TB nvme, 64gb RAM and has cost me around $400 USD. Runs at around 95 watt and I already have a good fibre connection any way.
For a similar spec machine I would be paying this every month and now I can just keep throwing things I’m interested in at it and so far has been overkill for a relatively fixed price. More messing around with hardware but something I mostly enjoy.
I do this in a VM. My mastodon server (for me personally) and BirdsiteLIVE bridge run on a VM on my old laptop configured as a Proxmox server. The entire thing has less than 2GB of RAM but it still runs just fine.
The incredibly inefficient Rails framework Mastodon is written in annoys me with its wasted performance but honestly the load for one or just a few users isn't even enough for me to start looking for an alternative. I'm sure I'd need to switch to a more optimized ActivityPub server if I'd get 10k users on my server, but for a small private instance you really don't need anything all that fancy.
The incredibly inefficient Rails framework Mastodon is written in annoys me with its wasted performance but honestly the load for one or just a few users isn't even enough for me to start looking for an alternative. I'm sure I'd need to switch to a more optimized ActivityPub server if I'd get 10k users on my server, but for a small private instance you really don't need anything all that fancy.
Costs of $1.70 per day which is mostly $30 for the domain name counts as “absurd”?
Yes, considering his instance has a single user, himself.
If more people sign up to his instance, or he just uses it, the costs will keep going up due to the caching Mastodon does for assets (and especially due to the fact he also doesn't have purging enabled)
If more people sign up to his instance, or he just uses it, the costs will keep going up due to the caching Mastodon does for assets (and especially due to the fact he also doesn't have purging enabled)
Imagine every little project costs you 30 dollars. ( He's getting compute for free for now, I included that)
That will add up. The cloudwatch costs are just ridiculous.
Even with the high energy costs in Europe I'll just be migrating to a more efficient machines and I will lower my power budget from 100 watt to 50. For now, with the current 100 watt I can run 4 nodes each with a hard drive and ssd, 16 gb of mem. These are hp 6300s, old but low power and if needed they can stil pack a punch with the i5 CPUs.
That will add up. The cloudwatch costs are just ridiculous.
Even with the high energy costs in Europe I'll just be migrating to a more efficient machines and I will lower my power budget from 100 watt to 50. For now, with the current 100 watt I can run 4 nodes each with a hard drive and ssd, 16 gb of mem. These are hp 6300s, old but low power and if needed they can stil pack a punch with the i5 CPUs.
How much is your internet per month, and what are your electricity costs per KWh?
My internet is €70/mo, and if I did not self host anything it would be... €65/mo. The speed I pay for because video games are massive these days so the only saving would be dropping a static IP.
Electricity per kwh is quite expensive these days, at €0.43/kwh because of war. My hardware is quite inefficient too, being cast off gaming hardware that is way overkill for the use case. Still, it averages 200W under load so that's (0.43 * 24/5) or €2/day.
So €65/mo, with some of the world's most expensive electricity. With 2021 energy rates the electricity cost would have been one third of 2022, at €25/mo. Use more reasonable hardware like a pi 8gb or a nuc or something rather than a used gaming PC and you could get even lower, though obviously you'd want to depreciate your hardware cost.
Meanwhile comparable compute from AWS is $1/hr
Electricity per kwh is quite expensive these days, at €0.43/kwh because of war. My hardware is quite inefficient too, being cast off gaming hardware that is way overkill for the use case. Still, it averages 200W under load so that's (0.43 * 24/5) or €2/day.
So €65/mo, with some of the world's most expensive electricity. With 2021 energy rates the electricity cost would have been one third of 2022, at €25/mo. Use more reasonable hardware like a pi 8gb or a nuc or something rather than a used gaming PC and you could get even lower, though obviously you'd want to depreciate your hardware cost.
Meanwhile comparable compute from AWS is $1/hr
Do you have an article on how to set it up like that ?
Learn from the mistakes of others, do not do this. Remember Yevgeniy Nikulin?
What relevance does Yevgeniy Nikulin have with this, whatsoever?
A lot, actually. He compromised a home-ran webserver on a VM, and compromised his way to a networked work machine. Running exposed ports from your house, no matter how smart you think you are, is not a good idea.
What's fundamentally different if you host this in the cloud but still have a public port?
What was the exact attack vector in the case of nyevgeny?
What was the exact attack vector in the case of nyevgeny?
The difference is it would be likely inconsequential if a the cloud resource running this was compromised. There’s no direct link (at least there shouldn’t be) to a work machine, which might have credentials or itself VPN information for an employer.
Yevgeny did exactly this. He compromised a web server running Apache or something, used that to brute force his way into a networked work machine, which took weeks, but went undetected. That machine turned out to be the physical host the VM running the web server was on. Then he used the work machine to break into LinkedIn using credentials and VPN profiles found on the iMac.
This vector would have been impossible had that open server not been running so close to the work machine.
Yevgeny did exactly this. He compromised a web server running Apache or something, used that to brute force his way into a networked work machine, which took weeks, but went undetected. That machine turned out to be the physical host the VM running the web server was on. Then he used the work machine to break into LinkedIn using credentials and VPN profiles found on the iMac.
This vector would have been impossible had that open server not been running so close to the work machine.
Thanks for the explainer.
Let's say I just want a single-user instance that's in the style of Hacker News: text only.
1. I want to write text-only posts and have them distributed to whoever is following me on Mastodon.
2. I want to receive all the posts of people I follow, and then just display/store the text and nothing else.
Does something like this exist? Seems like it should be super slim and easy on resources.
1. I want to write text-only posts and have them distributed to whoever is following me on Mastodon.
2. I want to receive all the posts of people I follow, and then just display/store the text and nothing else.
Does something like this exist? Seems like it should be super slim and easy on resources.
Honk - https://humungus.tedunangst.com/r/honk
Minimalist and opinionated but is a single binary with an SQLite db. Source is, uh, esoteric but is definitely hackable (there's a whole bunch of forks.) Seems to work ok with a few minor weirdnesses[1].
Should clarify that it doesn't have an API - you can't use Mastodon clients against it (until/unless someone writes on) - it's basically web only.
[1] I can't follow someone on my GotoSocial instance from honk but I'm pretty sure I've debugged that as being GTS's fault[2] rather than honk.
[2] They made many odd design decisions which cause(d) issues with federating to other systems.
[edit: clarify that honk as no API]
Minimalist and opinionated but is a single binary with an SQLite db. Source is, uh, esoteric but is definitely hackable (there's a whole bunch of forks.) Seems to work ok with a few minor weirdnesses[1].
Should clarify that it doesn't have an API - you can't use Mastodon clients against it (until/unless someone writes on) - it's basically web only.
[1] I can't follow someone on my GotoSocial instance from honk but I'm pretty sure I've debugged that as being GTS's fault[2] rather than honk.
[2] They made many odd design decisions which cause(d) issues with federating to other systems.
[edit: clarify that honk as no API]
> [1] and [2]
Ugh!
Is there a fork that just uses whatever stupid key/keysigning choices are needed to play well with the maximal number of other federated thingies?
I'm just going to be sending out a single post per day, tops. And perhaps reading whatever single digit number of people I decide to follow (maybe double if I'm really bored).
If I have to generate and send private keys in the clear to get this done I honestly couldn't care less.
Edit: clarification
Ugh!
Is there a fork that just uses whatever stupid key/keysigning choices are needed to play well with the maximal number of other federated thingies?
I'm just going to be sending out a single post per day, tops. And perhaps reading whatever single digit number of people I decide to follow (maybe double if I'm really bored).
If I have to generate and send private keys in the clear to get this done I honestly couldn't care less.
Edit: clarification
To clarify, it's GotoSocial that's the weird one - Honk seems to do the Right Thing.
Does GotoSocial communicate successfully with Mastodon?
Seems to be ok with Mastodon, Pleroma, and Akkoma from my experiences - although only with recent *omas.
The source is indeed esoteric, and sometimes hilarious. I have been using it to learn how the auth works against mastodon and some other parts of Activitypub. I do recommend it as a decent starting point to learn from.
Can you elaborate on some of gotosocial odd choices?
The main one is how they handle keys[1] which broke federation with Pleroma for a while. There's also some weirdness with their HTTP signing which breaks Honk follow requests but I'm still tracking that down.
[1] https://github.com/superseriousbusiness/gotosocial/issues/11...
[1] https://github.com/superseriousbusiness/gotosocial/issues/11...
[deleted]
There really needs to be a better layer than vanilla S3. PUT costs are absurdly high and it’s a terrible filesystem.
EFS, while much more expensive for storage, might win out if request costs are the problem.
For the avid self hoster, SeaweedFS:
https://github.com/seaweedfs/seaweedfs
https://github.com/seaweedfs/seaweedfs
Cloudflare R2 is at least cheaper.
R2 is incomplete S3 API at the moment. It is incompatible with Mastodon.
Seems like it should be working based on this https://github.com/mastodon/mastodon/discussions/20952
Thanks! The "ACL not implemented" error mentioned there was misinterpreted by me when I attempted to set it up.
I run a self-hosted instance for a bunch of friends. It's on a VM on a 10-year old homebuilt TrueNAS server.
For added entertainment, I installed it manually on FreeBSD, because I like playing old-school 'I do things by hand' sysadmin at home.
It's connected to the world through a free Cloudflare tunnel. I have a residential fiber connection. It doesn't use S3, it just dumps files to the filesystem. I did clamp down on federation settings, and it purges old content aggressively so federated content doesn't make the hard drives metaphorically explode. One user following someone on mastodon.social and you can see storage usage climb by gigs a day with defaults.
After the initial setup it's been honestly quite painless. It just works. If people start relying on it, I'll probably move it to masto.host.
For added entertainment, I installed it manually on FreeBSD, because I like playing old-school 'I do things by hand' sysadmin at home.
It's connected to the world through a free Cloudflare tunnel. I have a residential fiber connection. It doesn't use S3, it just dumps files to the filesystem. I did clamp down on federation settings, and it purges old content aggressively so federated content doesn't make the hard drives metaphorically explode. One user following someone on mastodon.social and you can see storage usage climb by gigs a day with defaults.
After the initial setup it's been honestly quite painless. It just works. If people start relying on it, I'll probably move it to masto.host.
Like OP, I keep being surprised by how easy it is to drive up bills with Cloudwatch. It’s small numbers here, but I’ve seen this happen at 4-5 digits as well.
i don't get why people would choose AWS in the first place ever for any small/amateur project.
It is ridiculously expensive, not transparent about the costs which could be coming your way and you need a shitload of components to get by.
just get some fixed price machine at some shitty provider and see how it goes, if you need a bigger one, just by a bigger machine. And if you outgrow that it might be a reasonable idea to just rent some colo space and calculate the upfront costs against a small server you build yourself. an intel nuc with 16gb memory and 4 cores is about 200$ and draws nothing in power. buy 2 connect them direct and hook them up to the colo provider. unlimited fiber and there you go.
AWS is such a joke
Is it true that most posts on self-hosted Mastodon are actually about self-hosting Mastodon?
No, not by a stretch.
GoToSocial seems to be a much better fit for a single user instance and it should be able to run on a scaleway stardust instance for €1.80 a month.
I was considering running mastodon too but someone mentioned this here and it does indeed look great
I was considering running mastodon too but someone mentioned this here and it does indeed look great
This is so cute! Cartel employee talking about using their drugs)
AWS is the most vendor-lock dependent company, IMO. Every single "architect solution" of their is a recipe of how to never get your data back for free.
AWS is the most vendor-lock dependent company, IMO. Every single "architect solution" of their is a recipe of how to never get your data back for free.
[deleted]
[deleted]
That's really cool! Seems like as the comment on your blog said, you could try to template it and perhaps offer Mastodon as a service on AWS? :)
Thanks for sharing.
Thanks for sharing.
It’s comforting to know that even people who work at AWS will sometimes forget about runaway cloudwatch charges. it's not just me!
Host at Hetzner or DO. AWS costs are outrageous.
> For compute - a single Elastic Compute Cloud (EC2) t4g.small instance running Mastodon, it’s PostgresSQL database, and Redis.
its
its
A premade Mastodon CloudFormation template would be cool for those who want to set up their own.
Digital ocean has a premade droplet that is costing me $20 / mo.
This is great to know. I've been interested in self hosting my own single user instance and it's been hard to understand the all-in cost.
I run two small fediverse instances, a misskey, and a pixelfed. both have fewer than 10 users, and the federated timeline only "clicks" once every few minutes. I am on several "medium" instances of ~10,000 users, and the admins that actually do the work on the infra are so tight-lipped - this bugs me.
You come to me and ask "how much does it cost to do X", and i know? I'll tell you. I'm a CC0 / public domain kind of person. I don't get the secrecy. Look, how many full cores/threads and how much ram/cache, and how much storage/io, and how much ingress and egress. Sure, those you can obfuscate a bit, but can i run a 10k user instance on 2 dell rackmount machines from 2014? on a single Ryzen? A single Epyc? a half dozen raspberry pi?
the chances of being able to parry "i run a php webservice" into retirement levels of money is nil, so i don't even understand the premise!
To answer your question more directly: I run misskey on a 4 thread, 8GB mem, 80GB storage VM, where i interact directly with the hypervisor (proxmox). my cost is nothing, but i guess this would be maybe $10/month? I'm not including the monthly cost to host the hyper, the /24, etc; we used to own the metal, but my partner decided there are things more important than owning the metal, right now. The upside is, when i take a snapshot, that isn't managed by disks that i can physically touch. Makes them slightly more reliable to me. Other people may have weirdness about this.
of the 4 "cores" the daily average is about 3% CPU usage, with monthly spikes up to 12% average usage. as far as bandwidth, the daily average is under 9kb/s and the monthly averages are under 22kb/s. a small fediverse instance with a single user is likely completely doable with a co-lo raspberry pi!
You come to me and ask "how much does it cost to do X", and i know? I'll tell you. I'm a CC0 / public domain kind of person. I don't get the secrecy. Look, how many full cores/threads and how much ram/cache, and how much storage/io, and how much ingress and egress. Sure, those you can obfuscate a bit, but can i run a 10k user instance on 2 dell rackmount machines from 2014? on a single Ryzen? A single Epyc? a half dozen raspberry pi?
the chances of being able to parry "i run a php webservice" into retirement levels of money is nil, so i don't even understand the premise!
To answer your question more directly: I run misskey on a 4 thread, 8GB mem, 80GB storage VM, where i interact directly with the hypervisor (proxmox). my cost is nothing, but i guess this would be maybe $10/month? I'm not including the monthly cost to host the hyper, the /24, etc; we used to own the metal, but my partner decided there are things more important than owning the metal, right now. The upside is, when i take a snapshot, that isn't managed by disks that i can physically touch. Makes them slightly more reliable to me. Other people may have weirdness about this.
of the 4 "cores" the daily average is about 3% CPU usage, with monthly spikes up to 12% average usage. as far as bandwidth, the daily average is under 9kb/s and the monthly averages are under 22kb/s. a small fediverse instance with a single user is likely completely doable with a co-lo raspberry pi!
> the admins that actually do the work on the infra are so tight-lipped
Not always. @kris-nova here is the public face of hachyderm.io and has done an excellent job communicating around architecture, incidents, etc. Everyone on that team deserves kudos.
Not always. @kris-nova here is the public face of hachyderm.io and has done an excellent job communicating around architecture, incidents, etc. Everyone on that team deserves kudos.
https://github.com/guysoft/PleromaPi/
you are in luck... i've used this and it works out of the box
you are in luck... i've used this and it works out of the box
> a small fediverse instance with a single user is likely completely doable with a co-lo raspberry pi!
You buried the lede:)
That's great to know. Maybe not worth it to cloud host. Are there any issues to worry about with having downtime of my own node, other than my own ability to post and read? I haven't read enough about ActivityPub to understand the impacts of a server being down on the protocol.
You buried the lede:)
That's great to know. Maybe not worth it to cloud host. Are there any issues to worry about with having downtime of my own node, other than my own ability to post and read? I haven't read enough about ActivityPub to understand the impacts of a server being down on the protocol.
yes, i realized after i typed up my complaints about fedi admins that i did the same thing.
Mastodon is php, and as such it requires php-level stuff. Misskey is nodejs (i think) and python, so it depends on if you're used to wordpress-y stuff or not. I prefer misskey because it has a "drive" - you can upload files and hotlink, or make "tweets" with them.
thankfully, activitypub and Ostatus "protocols" are open source, so one could write a server implementation in whatever language du jour makes them happy, but only a few people have.
I see a lot of the same issues with realtime applications like matrix (for chat) and audio/video streaming. Some php implementation (or python), and then no one else can make something that adheres to the standard. Whether that's the standard's fault or not isn't for me to say.
if i get a jpeg image file, and the joint picture expert group's spec for jpeg format, i can write something to view that in a day or so. same with whatever RFC IRC was. or http. or ftp.
try writing a toy matrix or fediverse implementation? meh.
Mastodon is php, and as such it requires php-level stuff. Misskey is nodejs (i think) and python, so it depends on if you're used to wordpress-y stuff or not. I prefer misskey because it has a "drive" - you can upload files and hotlink, or make "tweets" with them.
thankfully, activitypub and Ostatus "protocols" are open source, so one could write a server implementation in whatever language du jour makes them happy, but only a few people have.
I see a lot of the same issues with realtime applications like matrix (for chat) and audio/video streaming. Some php implementation (or python), and then no one else can make something that adheres to the standard. Whether that's the standard's fault or not isn't for me to say.
if i get a jpeg image file, and the joint picture expert group's spec for jpeg format, i can write something to view that in a day or so. same with whatever RFC IRC was. or http. or ftp.
try writing a toy matrix or fediverse implementation? meh.
Masto.host will charge you $6/month for that, though I think they’ve paused signups for now. If you want to do it yourself you can probably manage that sort of pricing using ovh or similar.
You have to have some big balls to run a Mastodon server on AWS. You are just one pissed off guy away from a bill that will cost you dearly.
nirvgorilla(1)
The instance I’m on is spending over $1000/month.
A lot of instances have had to scale up rapidly lately, which doesn’t help. Steady state cost will likely be lower.
How many active users do you have?
Mine (Mastodon.art) published their financials on Patreon. Iirc a couple months ago we were in the 15k user range, for ~$700/m.
- Cloudflare DNS
- 2x CPU 2GB RAM application server with snapshots
- 2x CPU 2GB RAM postgres server with automated backups
- 2x CPU 8GB RAM elastic search server to provide a full-text search for the known Fediverse
The search is not mandatory, but super nice to have. It is about 20€ a month, not counting the domain costs due to having it already for other purposes.
Akkoma is written in elixir and is much better in handling async messaging. No need for Sidekiq or Redis. My CPU usage hovers around 6-12% and memory around 55%. Postgres uses about the same.
I've never tried Mastodon, but having been a Ruby developer in my past life, I needed to find something else to install when I joined the Fediverse. Couldn't be happier with my setup. It just works, I never really need to do any maintenance.
All the normal clients work with it.
https://akkoma.social/