Cost of hosting the Meta Discourse forum on AWS(meta.discourse.org)
meta.discourse.org
Cost of hosting the Meta Discourse forum on AWS
https://meta.discourse.org/t/meta-is-moving-to-the-cloud-cloud-with-lightning/72819/30
44 comments
I think non-AWS still comes out in the end.
One of my favorite career moments was the time I tossed up a Varnish instance in a different data center than the site it cached, then used grace mode plus DNS Made Easy's failover to get a significant degree of georedundancy.
There have been multiple outages that nobody noticed, some of them hours long. Usually the backend is the one to fail, so more than 90% of end user page loads are straight from RAM nearly all the time. And it costs $20/month or so.
There have been multiple outages that nobody noticed, some of them hours long. Usually the backend is the one to fail, so more than 90% of end user page loads are straight from RAM nearly all the time. And it costs $20/month or so.
You can, right now on eBay, get a machine with many high powered cores and 256GB of RAM, for under $3k USD.
Then, you can colocate it for $200 per month or less, with a huge amount of bandwidth included. And $200 is not the much-cheaper colos that are out there, so it is on the high side.
HA? Buy a second one and mirror in a different DC.
HA config via this method: $6k servers + $2k disk + $400/month colo. First year: $12,800 (still cheaper than AWS). Second and third year: $4800 per year.
AWS is not the solution for every task.
Then, you can colocate it for $200 per month or less, with a huge amount of bandwidth included. And $200 is not the much-cheaper colos that are out there, so it is on the high side.
HA? Buy a second one and mirror in a different DC.
HA config via this method: $6k servers + $2k disk + $400/month colo. First year: $12,800 (still cheaper than AWS). Second and third year: $4800 per year.
AWS is not the solution for every task.
>AWS is not the solution for every task.
For self-hosting/personal stuff? Sure, EC2 is not cost effective. But you've just described a pretty powerful server for self-hosting, and you've left out a decent chunk of context around the considerations that go in to deploying HA, web-facing architecture.
For self-hosting/personal stuff? Sure, EC2 is not cost effective. But you've just described a pretty powerful server for self-hosting, and you've left out a decent chunk of context around the considerations that go in to deploying HA, web-facing architecture.
>considerations that go in to deploying HA, web-facing architecture
Easier than AWS, that's for sure. Getting HA to actually work in AWS is a nightmare of proprietary configuration and obscure documentation. Hardware-in-a-DC HA has been around approximately forever, and there is far more documentation available on the hundreds of ways to get it working rock-solid... all without vendor lock-in.
Easier than AWS, that's for sure. Getting HA to actually work in AWS is a nightmare of proprietary configuration and obscure documentation. Hardware-in-a-DC HA has been around approximately forever, and there is far more documentation available on the hundreds of ways to get it working rock-solid... all without vendor lock-in.
>Easier than AWS, that's for sure
You finding it hard != is hard
>Getting HA to actually work in AWS is a nightmare of proprietary configuration and obscure documentation
Strange. I have no problem with it. Most of my peers don't either. There is waaay worse documentation than AWS.
>Hardware-in-a-DC HA has been around approximately forever, and there is far more documentation available on the hundreds of ways to get it working rock-solid... all without vendor lock-in.
If you could point me to all this documentation you speak of that guarantees a "rock-solid" co-lo experience...
You finding it hard != is hard
>Getting HA to actually work in AWS is a nightmare of proprietary configuration and obscure documentation
Strange. I have no problem with it. Most of my peers don't either. There is waaay worse documentation than AWS.
>Hardware-in-a-DC HA has been around approximately forever, and there is far more documentation available on the hundreds of ways to get it working rock-solid... all without vendor lock-in.
If you could point me to all this documentation you speak of that guarantees a "rock-solid" co-lo experience...
Actually it is hard. Most of your peers and yourself may be familiar with the quirks, but that doesn't mean it isn't hard. AWS's docs are terrible. Others being worse does't change that fact.
As for rock solid colo docs- there is this thing called Google. And in fact you barely need that. It's real easy to set up.
As for rock solid colo docs- there is this thing called Google. And in fact you barely need that. It's real easy to set up.
>Actually it is hard. Most of your peers and yourself may be familiar with the quirks, but that doesn't mean it isn't hard. AWS's docs are terrible. Others being worse does't change that fact.
I think you've confused what "facts" are...
Doesn't matter. I guess some people are just destined to remain rack 'n stack sysadmins. Fine, more money and less competition for jobs then.
>As for rock solid colo docs- there is this thing called Google. And in fact you barely need that. It's real easy to set up.
I think you misunderstood my point. I know there are a thousand-and-one guides for setting up Nagios and Postfix and sticking HAProxy in front of it, but my point was that no amount of documentation can abstract away the actual pain points of dealing with co-los, including their owners, staff, and all the fun little problems that crop up with dealing with them.
For when it actually matters beyond a dollar value, I trust AWS engineers a lot more than I do HPE/Equinix etc...
I think you've confused what "facts" are...
Doesn't matter. I guess some people are just destined to remain rack 'n stack sysadmins. Fine, more money and less competition for jobs then.
>As for rock solid colo docs- there is this thing called Google. And in fact you barely need that. It's real easy to set up.
I think you misunderstood my point. I know there are a thousand-and-one guides for setting up Nagios and Postfix and sticking HAProxy in front of it, but my point was that no amount of documentation can abstract away the actual pain points of dealing with co-los, including their owners, staff, and all the fun little problems that crop up with dealing with them.
For when it actually matters beyond a dollar value, I trust AWS engineers a lot more than I do HPE/Equinix etc...
Seems you've gone a bit personal here.
[deleted]
I can't speak to that, aside from pointing out that the authors of the article themselves had to work on the HA aspects of it. It didn't come "for free" with the AWS account.
That's perfectly true, but AWS is a lot more than just servers. You're paying a premium for a reliable, flexible, scalable set of products that aren't easily replicated by a box in a rack.
For businesses with healthy margins, there's a clear tradeoff between cloud costs and opportunity costs. If you've got SLAs to honour and a lot of high-RoI engineering tasks in the queue, throwing a bunch of cloud resources at the problem might make perfect sense.
AWS isn't the solution for every task, but that doesn't make it a bad solution. The obvious parallel would be with Discourse's business model. Their product is completely free and open source, so there's nothing stopping you from running their software on a DO droplet, a colocated server or a Raspberry Pi attached to your home router. They have plenty of customers who are happy to pay $100/$300/POA for a managed deployment.
For businesses with healthy margins, there's a clear tradeoff between cloud costs and opportunity costs. If you've got SLAs to honour and a lot of high-RoI engineering tasks in the queue, throwing a bunch of cloud resources at the problem might make perfect sense.
AWS isn't the solution for every task, but that doesn't make it a bad solution. The obvious parallel would be with Discourse's business model. Their product is completely free and open source, so there's nothing stopping you from running their software on a DO droplet, a colocated server or a Raspberry Pi attached to your home router. They have plenty of customers who are happy to pay $100/$300/POA for a managed deployment.
> That's perfectly true, but AWS is a lot more than just servers.
Exactly but many people don't need that.
Exactly but many people don't need that.
I'm all for doing things "by hand", but this solution introduces a single point of failure. If the colo company has networking issues you'll lose all your traffic.
And of course doing HA / load-balancing across data-centers is hard. Ideally you'd use haproxy to route traffic to two distinct backends - but if the proxy is down your traffic gets lost. Simpler solutions are possible, but to do it properly would require your own IP ranges/BGP etc.
(Some people use static sites and use DNS for failover, but I think we all agree that's a terrible solution.)
And of course doing HA / load-balancing across data-centers is hard. Ideally you'd use haproxy to route traffic to two distinct backends - but if the proxy is down your traffic gets lost. Simpler solutions are possible, but to do it properly would require your own IP ranges/BGP etc.
(Some people use static sites and use DNS for failover, but I think we all agree that's a terrible solution.)
That's why you get one with multiple networks. AWS has had many outages too- more than a colo would have.
It is cheaper, but less flexible.
Any cost effective solutions for dealing with bursty demand for compute and networking that aren't AWS?
Any cost effective solutions for dealing with bursty demand for compute and networking that aren't AWS?
I don't have any visibility into their load. However 16+ cores and 256GB RAM combined with RAIDed SSDs should be able to handle quite a bit of traffic.
It's my opinion that memory bandwidth is a contributing factor to how well interpreted languages (Discourse runs on Ruby) perform. Since the hardware is dedicated, all available memory bandwidth can be used; on AWS you (may) share memory bandwidth between all VMs on the physical hardware.
It's my opinion that memory bandwidth is a contributing factor to how well interpreted languages (Discourse runs on Ruby) perform. Since the hardware is dedicated, all available memory bandwidth can be used; on AWS you (may) share memory bandwidth between all VMs on the physical hardware.
Do you have a colo recommendation? It's been years since I had a colo'd server, but I've been thinking about it getting one again soon.
It depends how much you want to spend.
WholesaleInternet.com I have used for their dedicated servers, but they also have inexpensive colo. The guys at HandyNetworks.com currently have some colo of mine, they have good quality hands-on help (their office is right beside their colo); they are more expensive however.
Check with WebHostingTalk.com and look for good reviews. Warning: always deal with either the owner/tenant of the colo or no more than 1 step removed; you don't want to be renting space from a reseller of a reseller: if someone in the chain goes under and hasn't paid for a few months, it will be a nightmare for billing and you may have trouble getting your hardware either turned on, or returned.
WholesaleInternet.com I have used for their dedicated servers, but they also have inexpensive colo. The guys at HandyNetworks.com currently have some colo of mine, they have good quality hands-on help (their office is right beside their colo); they are more expensive however.
Check with WebHostingTalk.com and look for good reviews. Warning: always deal with either the owner/tenant of the colo or no more than 1 step removed; you don't want to be renting space from a reseller of a reseller: if someone in the chain goes under and hasn't paid for a few months, it will be a nightmare for billing and you may have trouble getting your hardware either turned on, or returned.
Letting go of a concern of hardware failure is a reduction in cost.
[deleted]
What percent of the cost is spent on EC2 instances? Curious how much that can be reduced by using something more "lightweight" like Elixir or even Node. Basically, not needing to spin up thousands of OS processes. (Discourse uses Ruby.)
Thread hijack, asking for a friend.. What's the state of forum software these days? Some friends want to move their private/closed forum off vbulletin, which apparently hasn't kept up with the times (in particular, mobile use is apparently painful). Is discourse the best way these days, or is there something more lightweight e.g with elixir as mentioned but still reasonably mature and full featured?
Another vote for XenForo, my previous employer ran several forums all using vBulletin. IMO the direction of vBulletin 5 was a big miss, and like other commenters have said, the core product manager/developer of vBulletin 3 & 4 left to start XenForo. There was some significant legal drama with the current owner of vBulletin suing XenForo for infringement & theft intellectual property. Those all have been dismissed in favor of XenForo (all from memory, probably worthwhile to search this out just in case).
I've seen some enthusiasm around Discourse but for users who are used to the look & feel of a traditional discussion forum or the workflow of vBulletin, it can be very jarring. I believe there was a decent community split when Ubuntu tried to retire their legacy vBulletin forum in favor of Discourse; I think they are still running both side by side with almost different groups using them. XenForo is a more natural progression from vBulletin and would be my platform of choice.
I've seen some enthusiasm around Discourse but for users who are used to the look & feel of a traditional discussion forum or the workflow of vBulletin, it can be very jarring. I believe there was a decent community split when Ubuntu tried to retire their legacy vBulletin forum in favor of Discourse; I think they are still running both side by side with almost different groups using them. XenForo is a more natural progression from vBulletin and would be my platform of choice.
Discourse looks different from the traditional style but to my surprise I got used to using it quite fast and liking it much and not just about the usability but with its API, customizability, administration capability and mobile support. It shows when someone has experience creating stackoverflow.
XenForo seems to be the popular choice at the moment, though IPB may be catching up a bit. Both are a lot more traditional than the likes of Discourse, but they're the scripts many vBulletin owners migrated to when Internet Brands basically killed the company, as well as the ones with the most mods/community support.
Not sure about the state of open source forum scripts, though I don't recall phpBB/SMF/MyBB being replaced by anything else in that market recently.
But personally, I'd recommend XenForo. It's what most of the sites I run use (see Wario Forums for an example), and it's what many sites I'm a member/volunteer on use too (like The Admin Zone or Gaming Latest). Lots of choices for themes and mods there, it's pretty fully featured as far as basic options are concerned and the media gallery/resource manager work well too.
The fact many members weren't too keen on the other solutions I suggested helped too.
But yeah, XenForo is my recommendation.
Not sure about the state of open source forum scripts, though I don't recall phpBB/SMF/MyBB being replaced by anything else in that market recently.
But personally, I'd recommend XenForo. It's what most of the sites I run use (see Wario Forums for an example), and it's what many sites I'm a member/volunteer on use too (like The Admin Zone or Gaming Latest). Lots of choices for themes and mods there, it's pretty fully featured as far as basic options are concerned and the media gallery/resource manager work well too.
The fact many members weren't too keen on the other solutions I suggested helped too.
But yeah, XenForo is my recommendation.
A lot of vB forums switched to XenForo which is where the vB devs went after vbulletin was bought out. It has a more classic feel than Discourse which may be an advantage if your userbase is coming from vB, but it is definitely well made and up to date with modern technology.
Maybe then you'd be interested in having a look at Talkyard, which is new forum software (I'm developing it). It has HackerNews type discussions, with improvements:
https://www.talkyard.io/-32/how-hacker-news-can-be-improved-...
and looks & works a bit like Discourse, + also Slack chat features (no mobile app though) and StackOverflow style question-answers. Open source.
https://www.talkyard.io/-32/how-hacker-news-can-be-improved-...
and looks & works a bit like Discourse, + also Slack chat features (no mobile app though) and StackOverflow style question-answers. Open source.
I heard mybb is probably the most up to date with php7 support too. Discourse is nice, except that I don't want to pull in a whole Ruby framework, same for nodebb(using nodejs) which is nice too. In short, give mybb a try.
Official recommended way of running Discourse is with docker, there is no pulling in various packages but really just a 10 min thing if you already have docker.
The only problem with that point of view is that with containerizing it, and saying, oh you don't have to bring in Ruby, just use Docker, is that you are almost black-boxing it in a way.
The user does hardly any config, really has no idea how it works, which could make diagnosing and troubleshooting issues that crop up problematic.
The user does hardly any config, really has no idea how it works, which could make diagnosing and troubleshooting issues that crop up problematic.
The exact reason to use container is so that individual deployment doesn't get that random problem you're trying to bring up. If container deployments have problems, you take it to the upstream because it will likely to be affecting many.
> The main piece of advice I have is … don’t do it. Don’t take on a complex, “enterprisey” cloud install unless you have to. It’s extremely expensive for what you get. Compare to a simple monolithic Digital Ocean droplet running our standard Docker image, which can get you a very long way even at the $40 and $80 per month price points.
I really wonder where the line goes between a decent VPS (properly provisioned and configured to handle traffic) and AWS?
I really wonder where the line goes between a decent VPS (properly provisioned and configured to handle traffic) and AWS?
There isn't really a line because "a decent VPS" is exactly what EC2 is. You use it if you
* are averse to the kind of risk that comes with being tied to a smaller provider, and/or
* are not averse to the kind of risk that comes with being tied to a provider who cares not one whit for your well-being because your business is insignificant to them, and/or
* actually have a use for, and staff with the expertise to take advantage of, any of the million proprietary bells and whistles that AWS provides, and
* are not averse to absolutely abysmal UX.
Discourse falls mostly into category 3 above; as Sam enumerates they are taking advantage of a bunch of various AWS offerings and have split off several of their supporting services onto their own instances. Those things can be built up gradually; you don't just wake up one day and decide to become "Enterprise" and watch all your costs increase by two orders of magnitude.
* are averse to the kind of risk that comes with being tied to a smaller provider, and/or
* are not averse to the kind of risk that comes with being tied to a provider who cares not one whit for your well-being because your business is insignificant to them, and/or
* actually have a use for, and staff with the expertise to take advantage of, any of the million proprietary bells and whistles that AWS provides, and
* are not averse to absolutely abysmal UX.
Discourse falls mostly into category 3 above; as Sam enumerates they are taking advantage of a bunch of various AWS offerings and have split off several of their supporting services onto their own instances. Those things can be built up gradually; you don't just wake up one day and decide to become "Enterprise" and watch all your costs increase by two orders of magnitude.
There is a line because AWS, including EC2, is more complex and unique than a VPS.
Honestly, forum hosting requirements are pretty minimal in general. Indeed, for many scripts, you can get away with a cheap shared hosting account until you have about 50-100,000 points and 10,000 members or so. In many smaller communities cases, they'll never outgrow it.
And for something like Discourse, well a cheap Digital Ocean VPS or something would work quite well for most people and sites. Most aren't active enough to put any real pressure on a hosting account.
And for something like Discourse, well a cheap Digital Ocean VPS or something would work quite well for most people and sites. Most aren't active enough to put any real pressure on a hosting account.
Conclusion in the comments from JefF: "The main piece of advice I have is … don’t do it. Don’t take on a complex, “enterprisey” cloud install unless you have to. It’s extremely expensive for what you get. Compare to a simple monolithic Digital Ocean droplet running our standard Docker image, which can get you a very long way even at the $40 and $80 per month price points."
ohv has 3€ a month vps that will be largely enough for small communities, and probably average ones.
Also check out netcup, great VMs on rock solid hardware at incredible prices (better value than OVH)
I’ve been running some on my side projects just using a docker containers on digital ocean and t1.micros and apparently that was the most cost-effective way to run it ?
(Site requires various third-party JS to display anything)
I think Sam's comment is important:
>Note it is important to have full perspective on costs here...
All too often I see the "AWS is too expensive" circlejerk devolve into a flat cost argument.
It is absolutely more expensive in that regard.
That being said, I see very few comparisons that take in to account all of the engineering effort saved by some of the AWS feature set.
There are non-trivial things that simply aren't a problem anymore on AWS, and time can be spent on actually interesting/difficult problems.
>Note it is important to have full perspective on costs here...
All too often I see the "AWS is too expensive" circlejerk devolve into a flat cost argument.
It is absolutely more expensive in that regard.
That being said, I see very few comparisons that take in to account all of the engineering effort saved by some of the AWS feature set.
There are non-trivial things that simply aren't a problem anymore on AWS, and time can be spent on actually interesting/difficult problems.
Right but then there are other non-trivial things that you have to spend on that you wouldn't on non-AWS.
And really there are very few hints that are non-trivial and not a problem on non-AWS either.
In the end, AWS is still more expensive time and money wise, it's just a question of whether it's worth it and what you want to spend your time on. (For me, spending it on the AWSy things is not interesting)
And really there are very few hints that are non-trivial and not a problem on non-AWS either.
In the end, AWS is still more expensive time and money wise, it's just a question of whether it's worth it and what you want to spend your time on. (For me, spending it on the AWSy things is not interesting)
> The bottom line is that $10-$80 is perfectly fine for an unmonitored monolithic setup. But once you need to start talking SLAs and need to know this thing will be rock solid and survive random failures… well costs start mounting.
I've run sites that do tens or hundreds of thousands of uniques a week with base-level VPSes like Linode or DO, spending just $10-20/mo on it. But if you require high availability or must meet strict SLAs, you start hitting the Pareto principle pretty hard. You end up needing to pay a lot for that increased certainty, even if you are only squeezing a few more minutes of uptime or flexibility out of it.