AWS Cost Saving Recommendations(vantage.sh)
vantage.sh
AWS Cost Saving Recommendations
https://www.vantage.sh/blog/vantage-launches-aws-cost-saving-recommendations
97 comments
I know it isn't for everyone, but if you're seriously trying to save cash and you can handle the trade-off of managing your own infrastructure, Hetzner has AX101s back in stock: https://www.hetzner.com/dedicated-rootserver/ax101. €100/mo for 16 Ryzen 9 cores, 128GB of RAM and 3.84TB of NVMe. Unlimited traffic.
Their fraud system errantly flagged my account because of my name and they deleted all my data.
I got an email saying there was an issue with my account, called them within 10 minutes of the email and was informed they had already deleted all my data.
They reinstated my account which was just a blank shell, nothing remained but my username and password.
You get what you pay for.
If you go with them make sure you back up your data to a separate provider.
I got an email saying there was an issue with my account, called them within 10 minutes of the email and was informed they had already deleted all my data.
They reinstated my account which was just a blank shell, nothing remained but my username and password.
You get what you pay for.
If you go with them make sure you back up your data to a separate provider.
That is quite a surprising story, unless your account was deleted immediately or almost immediately after creation. I know of some people kicked out of Hetzner after doing some actual bad stuff, but they had been there for a while. Hetzner took their servers offline and gave them X days to retrieve their data through the rescue system, or alternatively, pay $Y and have the hard drives from the servers physically removed and shipped to them.
I think both of the above cases were normal TOS violations like hosting porn. If more serious crime was involved the response might have been different.
I've had stuff at Hetzner for about 5 years and I'm very happy with them.
I think both of the above cases were normal TOS violations like hosting porn. If more serious crime was involved the response might have been different.
I've had stuff at Hetzner for about 5 years and I'm very happy with them.
My account was 3 weeks old, but that still means I lost most of my work effort for those weeks.
The email I received said
..the associated IP has been blocked by us.
The IP address can only be unblocked, once all steps have been taken to remedy the situation.
But instead of blocking my IP address as their system suggested they were doing, they immediately put everything in the trash, even though their issue was with my personal information and not my systems.
Barely an apology from them for not following their own procedures or giving me a chance to respond.
The email I received said
..the associated IP has been blocked by us.
The IP address can only be unblocked, once all steps have been taken to remedy the situation.
But instead of blocking my IP address as their system suggested they were doing, they immediately put everything in the trash, even though their issue was with my personal information and not my systems.
Barely an apology from them for not following their own procedures or giving me a chance to respond.
If you gave them fake info then it is pretty usual for it to be poorly received, though I don't know what Hetzner's past practice is. If you gave them good info that they had some kind of issue with, then maybe your complaint is valid.
I'd add: 3 weeks may not have been long enough for them to send an invoice and bill your credit card. If they didn't receive any money from you, they may not have felt like they owed you anything.
I'd add: 3 weeks may not have been long enough for them to send an invoice and bill your credit card. If they didn't receive any money from you, they may not have felt like they owed you anything.
It wasn't fake info. I used a shortened version of my last name. This is the name I go by because my last name is difficult to pronounce. They had my full last name via my email address and the credit card I used with them. I did not attempt to hide anything, and gave them the same information I use everywhere else on the internet.
Also you are correct, they may have thought they didn't owe me anything, including an apology. They valued my three weeks worth of effort at nothing and before apologizing at least one person attempted to tell me this was my fault.
3 weeks may not seem like a lot of time, but I was spending 16 hour days sometimes getting things up and running because I was excited about what I was building.
When it all got deleted in the middle of the setup, I didn't have the heart or the energy to rebuild everything. I tried, but I didn't have an extra month that I could spare, and was so disheartened from all of the work that I lost that I was not able to regain the momentum that I had and had to move on to other things to be able to support myself.
I know it doesn't seem like much, and I know to them I was nothing, which will forever cement this in my mind as the worst experience I have had with a company to date.
I was so upset about this I spent several days digging up personal contact information for their executives to voice my displeasure, but in the end decided against it because nobody I had talked to at the company seemed to care, and as both you and I are suggesting, it's just a hard lesson in "you get what you pay for"
I should not have expected quality service when trying to cheap out and go with a low-cost vendor. It was a very difficult lesson to learn, and hopefully someone learns from my mistake.
I'm not suggesting not to use them, only reminding people not to make my mistake and host their backups with a third party from day one.
Also you are correct, they may have thought they didn't owe me anything, including an apology. They valued my three weeks worth of effort at nothing and before apologizing at least one person attempted to tell me this was my fault.
3 weeks may not seem like a lot of time, but I was spending 16 hour days sometimes getting things up and running because I was excited about what I was building.
When it all got deleted in the middle of the setup, I didn't have the heart or the energy to rebuild everything. I tried, but I didn't have an extra month that I could spare, and was so disheartened from all of the work that I lost that I was not able to regain the momentum that I had and had to move on to other things to be able to support myself.
I know it doesn't seem like much, and I know to them I was nothing, which will forever cement this in my mind as the worst experience I have had with a company to date.
I was so upset about this I spent several days digging up personal contact information for their executives to voice my displeasure, but in the end decided against it because nobody I had talked to at the company seemed to care, and as both you and I are suggesting, it's just a hard lesson in "you get what you pay for"
I should not have expected quality service when trying to cheap out and go with a low-cost vendor. It was a very difficult lesson to learn, and hopefully someone learns from my mistake.
I'm not suggesting not to use them, only reminding people not to make my mistake and host their backups with a third party from day one.
That is pretty weird. How long ago was it? Do you have a public email, or a Hetzner ticket number? I have sort of a contact at Hetzner so I could ask her to ask someone to take a look at your case.
I had the same thing there happen twice to me within one year. I seriously don't get how this company still exists. I have been with Contabo for two years now and no issues so far.
wth. How is that even considered acceptable from their end?
Was that a new account where you'd already paid money? Never heard of that before, I've been a customer at Hetzner for years with considerable volume. So far their support was always great.
Seconded. Every three months I get a little anxious about not running on one of the clouds that would let me scale more quickly (and also would let me do things like use Athena to go through a huge S3 bucket). But then I do a little math and realise that the bandwidth bill from AWS alone would eclipse all of my hosting costs at Hetzner, not to speak of the actual servers running there, and I don't really need fast scaling if I can just provision everything with a nice margin (2x) and still come out way below AWS prices. But it really depends on the nature of your business I suppose.
Bandwidth on AWS has some scary side effects if you're not careful. A former colleague of mine got screwed for this and S3. Backup storage costs? $57 a month. Just the bandwidth fee to do a restore? $450
If only deploying Kubernetes was as simple as running a golang up and etcd tolerated some latency, you could spin up a node in AWS, another in Hetzner, and a third in DigitalOcean.
Try microk8s.io , it uses dqlite (replicated SQLite + raft leader election; also, yes, it needs 'snap', just let it go, it doesn't matter). Use a fixed version stable channel, because they apparently employ the world's eminent simian scientists to break your cluster on every major update. Even on a test cluster don't be too frugal with RAM.
https://ubuntu.com/tutorials/getting-started-with-kubernetes...
https://ubuntu.com/tutorials/getting-started-with-kubernetes...
Hetzner is fantastic, at least if you're European (it only has data centres in Germany and Finland). We've used them for 8-ish years, and _never_ had any problems.
At the moment we're running a 3-node Galera-powered MySQL cluster on the EX52-NVME servers,[0] costing a _total_ of $2,800/year.
[0] https://www.hetzner.com/dedicated-rootserver/ex52-nvme
At the moment we're running a 3-node Galera-powered MySQL cluster on the EX52-NVME servers,[0] costing a _total_ of $2,800/year.
[0] https://www.hetzner.com/dedicated-rootserver/ex52-nvme
Hey, did you have any public-facing websites, or just private Databases? I heard Hetzner DDoS is pretty bad, and during a DDoS, SSL (https) does not work correctly.
We've run pretty much everything on Hetzer. DDoS has never been a problem, but we usually deploy Varnish in front of the web servers to handle traffic.
Netcup is also excellent. It’s the best CPU bang for the buck that I’ve found, and where I run around 10 transcoding servers.
https://www.netcup.eu/
https://www.netcup.eu/
Netcup was founded in my hometown in Germany. They have a very good reputation. However, they were aquired a while ago by Anexia. I am not sure if that changed anything.
Do you have a link to a page with VPS sizes/prices? I’m only seeing marketing stuff with no real numbers. I’m on mobile, might have something to do with it.
While we're on the good-value dedicated server providers, Contabo are also worth a look: https://contabo.com/en/dedicated-servers/amd-epyc-32-cores
That's less than my accidental personal AWS bill a couple of months ago!
And you can get a 10gbit/s uplink for ~€45/mo. Traffic is not unlimited anymore then but with €1/TB still cheap.
No U.S. alternative.
OVH has US and Canadian data centres, and free bandwidth and DDoS protection. Ive been using them for about 5 years and they are very reliable.
Seconded. Happy OVH user here. We run 10 baremetal servers in CA and EU, using KVM, ZFS and "fallback IPs" so that we can move VMs around easily. Borg for backups to another provider.
But yeah, because of clouds, old school sysadmin skills can be difficult to find.
But yeah, because of clouds, old school sysadmin skills can be difficult to find.
Lichess is also hosted there
The also have a cloud offering that's supported by Terraform. They don't offer software as a service but various instance types, virtual networks, storage and load balancers. Happy customer here.
:wave:
I'm a Co-Founder at https://vantage.sh/ - thanks for posting this, Jeff!
I think the biggest call out here is that Vantage Cost Saving Recommendations are profiling not only things like AWS Savings Plans and Reserved Instances but we are also directly integrated with AWS Service APIs that allow us to surface higher-fidelity cost-savings measures on a per AWS service or resource basis. Over time, we will be adding to the suite of checks and recommendations.
Also, it's worth calling out that these recommendations are available to all users -- including users in our free tier.
I'm happy to answer any questions if folks have them.
I'm a Co-Founder at https://vantage.sh/ - thanks for posting this, Jeff!
I think the biggest call out here is that Vantage Cost Saving Recommendations are profiling not only things like AWS Savings Plans and Reserved Instances but we are also directly integrated with AWS Service APIs that allow us to surface higher-fidelity cost-savings measures on a per AWS service or resource basis. Over time, we will be adding to the suite of checks and recommendations.
Also, it's worth calling out that these recommendations are available to all users -- including users in our free tier.
I'm happy to answer any questions if folks have them.
Vantage should provide a date saying when exactly the recommendations were computed. This answer from the FAQ kinda sucks.
> I am a Vantage user but I don't see any Cost Recommendations. What is happening?
> There are two possible things causing this: (1) Vantage has not yet run the process for finding cost recommendations for your account yet and you can check back later (2) your account is well-optimized from an AWS cost perspective and there are no recommendations for you to review.
> In either case, Vantage will continue to monitor your AWS account for all changes and be sure to surface Cost Recommendations to you as it finds them.
"Maybe we'll provide useful guidance, maybe we won't, but we won't tell you which. Maybe you're perfect. Maybe we never ran an analysis. Maybe we'll run an analysis in the future, maybe not."
> I am a Vantage user but I don't see any Cost Recommendations. What is happening?
> There are two possible things causing this: (1) Vantage has not yet run the process for finding cost recommendations for your account yet and you can check back later (2) your account is well-optimized from an AWS cost perspective and there are no recommendations for you to review.
> In either case, Vantage will continue to monitor your AWS account for all changes and be sure to surface Cost Recommendations to you as it finds them.
"Maybe we'll provide useful guidance, maybe we won't, but we won't tell you which. Maybe you're perfect. Maybe we never ran an analysis. Maybe we'll run an analysis in the future, maybe not."
Thanks for your candid feedback - I agree this is something we should do a better job of surfacing. I've added this to our backlog to improve - feel free to email me and I can let you know when the experience improves.
We mention elsewhere in the post that for service and resource level recommendations we do this as often as your Vantage account syncs - which you can initiate at any time from the top navigation bar.
For recommendations like Savings Plans and Reserved Instances we are running this process weekly for now but that may change so we will definitely add a timestamp as to when that occurs.
We mention elsewhere in the post that for service and resource level recommendations we do this as often as your Vantage account syncs - which you can initiate at any time from the top navigation bar.
For recommendations like Savings Plans and Reserved Instances we are running this process weekly for now but that may change so we will definitely add a timestamp as to when that occurs.
How did you guys decide on your pricing plans? It seems to me you're missing out on the folks who's spending doesn't vary much per month but want to get their costs optimized. I have a friend who's company is spending ~70k/month on AWS who might be interested in this, but doesn't generally need it on an ongoing basis since their needs are generally fixed.
Are these recommendations from the cost & usage report? just normal API queries?
If it is just from normal API (probably list instance/lb), is it really enough in order to create proper recommendations?
do you have an estimate on the cost/usage you are adding for each scale of customer?
If it is just from normal API (probably list instance/lb), is it really enough in order to create proper recommendations?
do you have an estimate on the cost/usage you are adding for each scale of customer?
I was already able to decommission $160USD a month of infrastructure after 5 minutes on the platform. Thanks!
Is it possible to set up a billing email? I would rather not have to forward the invoices every month.
Is it possible to set up a billing email? I would rather not have to forward the invoices every month.
That's fantastic! If you email [email protected] we can update the billing email manually now in advance of us launching RBAC for team-members on your Vantage account.
Prior to contacting support, please just invite your team-mate to your Vantage account from the account settings page.
Prior to contacting support, please just invite your team-mate to your Vantage account from the account settings page.
I've been doing some AWS optimization recently. For my setup, which heavily uses a lot of short job based queue patterns, I've had a lot of success with the following:
1) Moving to spot instances where possible 2) Autoscaling rules 3) Reserved instances where possible 4) Moving everything to AWS Graviton based instances
I cut the bill by ~70%.
1) Moving to spot instances where possible 2) Autoscaling rules 3) Reserved instances where possible 4) Moving everything to AWS Graviton based instances
I cut the bill by ~70%.
I don't know anything about Vantage. It sounds like people have had good luck with them.
However, what gets us our greatest savings on AWS are two things.
First, we have the luxury of being able to take advantage of Reserved instances. We have decided how much we are wiling to commit on EC2's, RDS, etc. and it saves us 10-30% depending on what we do.
Second, and this is perhaps the more interesting one. We started working with an "AWS Advanced Partner." Billing goes through them and that reduces our charges. In addition, they pitch projects of ours to AWS and if they are interesting enough to AWS, AWS reduces charges for periods of time on servers related to those developments.
While we use AWS, I think the game is the same with Google or Microsoft. So, if you are looking to save some money, you may look into companies who are Advanced AWS Partners, Premier Google Partners, or Silver Microsoft Partners. It's likely they can help you out.
However, what gets us our greatest savings on AWS are two things.
First, we have the luxury of being able to take advantage of Reserved instances. We have decided how much we are wiling to commit on EC2's, RDS, etc. and it saves us 10-30% depending on what we do.
Second, and this is perhaps the more interesting one. We started working with an "AWS Advanced Partner." Billing goes through them and that reduces our charges. In addition, they pitch projects of ours to AWS and if they are interesting enough to AWS, AWS reduces charges for periods of time on servers related to those developments.
While we use AWS, I think the game is the same with Google or Microsoft. So, if you are looking to save some money, you may look into companies who are Advanced AWS Partners, Premier Google Partners, or Silver Microsoft Partners. It's likely they can help you out.
> In addition, they pitch projects of ours to AWS and if they are interesting enough to AWS, AWS reduces charges for periods of time on servers related to those developments.
Is there IP transfer here? They give you a discount in exchange for knowing how you're making money?
Is there IP transfer here? They give you a discount in exchange for knowing how you're making money?
I have no evidence, but my suspicion is that programs like this are designed similiarly to large credit grants given to startups.
If they see a project that seems "interesting" (ie, has a potentially to be successful and have a very high AWS spend in the future), they're willing to subsidize[0] some costs now in order for you to get used to and build around AWS, and likely build more closely to AWS. If it ends up being successful, AWS has a much larger customer.
[0] I say subsidize in quotes because I seriously doubt AWS is taking a loss on any of this, they're just not making quite as much in profit.
If they see a project that seems "interesting" (ie, has a potentially to be successful and have a very high AWS spend in the future), they're willing to subsidize[0] some costs now in order for you to get used to and build around AWS, and likely build more closely to AWS. If it ends up being successful, AWS has a much larger customer.
[0] I say subsidize in quotes because I seriously doubt AWS is taking a loss on any of this, they're just not making quite as much in profit.
This is my experience. Their grants are also in competitive spaces.
Do we want to move off Microsoft SQL Server and go to Postgres Aurora? Yes. AWS also wants us and everyone in the world to do that. It’s a win for us to get there because of reduced license costs. It’s a win for them, because they get the revenue from services.
So, they give grants to help us and help them build their services.
Do we want to move off Microsoft SQL Server and go to Postgres Aurora? Yes. AWS also wants us and everyone in the world to do that. It’s a win for us to get there because of reduced license costs. It’s a win for them, because they get the revenue from services.
So, they give grants to help us and help them build their services.
I'd say they give them a discount for developing something a customer will then (have to) run on AWS.
Not from our perspective. It’s basically subsidized infrastructure spending. We aren’t doing the low level things they are “probably” interested in. There is no right to IP granted.
First taste is free...
Really digging this product so far. So many times I've set up complicated multi-region service architectures in AWS and will still struggle to produce simple POVs on cost consumption (and credit usage/outlook, for all us startups).
This thing is pretty damn straightforward and simple (which is good), and TIL from Vantage that I've been using the wrong volume type - gp3 is better and would save me money. I feel dumb not knowing, but now Vantage made me a little smarter. ;-) Three cheers for the cost savings recommendations!
This thing is pretty damn straightforward and simple (which is good), and TIL from Vantage that I've been using the wrong volume type - gp3 is better and would save me money. I feel dumb not knowing, but now Vantage made me a little smarter. ;-) Three cheers for the cost savings recommendations!
To be honest, this is all the domain of your AWS service advisor. At any type of scale(eg, when you move to invoiced billing because CC billing would be absurd), you should be overwhelmed by phone calls from that individual. While half of their job is to upsell you, the other half is to help you save costs to garner goodwill(and your future business).
If you've been dodging these calls in the past, it might be worth picking up the phone.
If you've been dodging these calls in the past, it might be worth picking up the phone.
I've heard of some latency issues with gp3 vs gp2--not sure if that's been resolved. Of course, services like RDS don't support it yet, either (same with io2)
As for cost consumption, splitting services into separate AWS accounts (even per-environment) helps if they're large (plus then you're less likely to hit rate limits, worry about IAM segregation, etc)
As for cost consumption, splitting services into separate AWS accounts (even per-environment) helps if they're large (plus then you're less likely to hit rate limits, worry about IAM segregation, etc)
I just did a mass conversion of gp2 to gp3. gp3 DOES increase disk latency, but we're talking 2ms -> 4ms. So it's doubled, but still pretty low. For normal usage, it's still quite fast.
You shouldn't feel too bad, most companies/people don't know how much money they are wasting on AWS.
Moreover, gp3 is very new (in cloud time) and most people don't use it since it's not really supported that well in cloudformation
Moreover, gp3 is very new (in cloud time) and most people don't use it since it's not really supported that well in cloudformation
I think this advice is probably the most straightforward suggestion: https://www.lastweekinaws.com/blog/to-save-money-on-your-aws...
This seems like an ad for vantage more than a cost-savings tutorial. Can anyone talk about what specifically vantage is doing to identify cost-saving areas?
Otherwise the title is misleading. Mods, can you update the title to be the actual title of the article?
Otherwise the title is misleading. Mods, can you update the title to be the actual title of the article?
For what its worth, the original submission of the post as we saw it was our blog post title of "Vantage Launches AWS Cost Saving Recommendations" and was updated to this presumably by HN mods.
Maybe this is more advanced, but just going off the screen shot it looks the same as Trusted Advisor https://aws.amazon.com/premiumsupport/technology/trusted-adv...
If you have Enterprise Support, your TAMs can also help with cost and can arrange meetings and resources to address cost concerns, as well (including architectural reviews)
If you're not large enough for that to apply or don't plan for huge growth, personally, I'd strongly reconsider AWS. Amazon's APIs and infrastructure automation can give small teams a lot of leverage to run massive amounts of infrastructure and quickly scale, but things with a small, fixed set of infrastructure might be worth a look on VMs, etc
If you have Enterprise Support, your TAMs can also help with cost and can arrange meetings and resources to address cost concerns, as well (including architectural reviews)
If you're not large enough for that to apply or don't plan for huge growth, personally, I'd strongly reconsider AWS. Amazon's APIs and infrastructure automation can give small teams a lot of leverage to run massive amounts of infrastructure and quickly scale, but things with a small, fixed set of infrastructure might be worth a look on VMs, etc
> If you're not large enough for that to apply or don't plan for huge growth, personally, I'd strongly reconsider AWS. Amazon's APIs and infrastructure automation can give small teams a lot of leverage to run massive amounts of infrastructure and quickly scale, but things with a small, fixed set of infrastructure might be worth a look on VMs, etc
That doesn't make sense to me. Imagine you're paying ~$50,000/month for AWS and make good use of its features (of course if you're just running EC2 instances, using AWS adds little value). While that might sound costly, in my opinion it's still way too little to justify the additional $10,000/month Enterprise Support would cost. Using Enterprise Support would increase the bill by 20% in this case!
In my opinion Enterprise Support only makes sense if your monthly AWS bill is close to or more than $150,000/month, as otherwise Enterprise Support is a quite expensive additional item on the bill, or if you require the additional escalation possibilities in case of problems that come with Enterprise Support.
That doesn't make sense to me. Imagine you're paying ~$50,000/month for AWS and make good use of its features (of course if you're just running EC2 instances, using AWS adds little value). While that might sound costly, in my opinion it's still way too little to justify the additional $10,000/month Enterprise Support would cost. Using Enterprise Support would increase the bill by 20% in this case!
In my opinion Enterprise Support only makes sense if your monthly AWS bill is close to or more than $150,000/month, as otherwise Enterprise Support is a quite expensive additional item on the bill, or if you require the additional escalation possibilities in case of problems that come with Enterprise Support.
>don't plan for huge growth
imo it might make sense at that point to consider other options. AWS definitely isn't cheap and $50k/mon is a lot on infrastructure for "free" support. I don't think AWS Support makes sense at that price (unless you're going to heavily leverage it) but I'm also not sure running $50k infra on AWS makes a lot of sense either
imo it might make sense at that point to consider other options. AWS definitely isn't cheap and $50k/mon is a lot on infrastructure for "free" support. I don't think AWS Support makes sense at that price (unless you're going to heavily leverage it) but I'm also not sure running $50k infra on AWS makes a lot of sense either
With Amazon and any variable pricing model, there is no limit to how much you will pay. It's a conscious business model decision and a very smart one by Amazon. You want to be smart about costs? Limit your downside risk ie. Fix your costs.
DigitalOcean App Platform is simple and affordable:
https://docs.digitalocean.com/products/app-platform/
https://docs.digitalocean.com/products/app-platform/
Did a quick search and didn't see UpCloud mentioned. Having used AWS, GCP, Azure and DigitalOcean, one of the strong points of UpCloud (except for their pricing and product offerings) have been their customer interaction. Hands down the best experience I've ever had both from support and sales. I reached out to support 2:15AM in their live chat and got my issues resolved as we spoke. Good luck getting that with one of the big companies.
https://upcloud.com/
https://upcloud.com/
> I reached out to support 2:15AM in their live chat and got my issues resolved as we spoke.
Nice, but time zone matters - depending on where both you and they are based, that could have just been normal 9-5 making it seem better than the reality... Not detracting from the fact that it got resolved promptly of course.
Nice, but time zone matters - depending on where both you and they are based, that could have just been normal 9-5 making it seem better than the reality... Not detracting from the fact that it got resolved promptly of course.
Never heard of it before and will give a shot. Thank you.
Pricing seems to be similar to DigitalOcean, however there seems to be more resources allocated.
Pricing seems to be similar to DigitalOcean, however there seems to be more resources allocated.
The best way to save AWS costs is to optimize your code. Seen it over and over again at many portfolio companies.
Optimize, and redesign to use AWS features. Just dumping your existing code on EC2 is often a bad idea.
We worked with a client and analyzed their workload and software. With a few minor changes we could reduce their always on EC2 instances with 66%. They where not interested. Then at least buy reserved instances? Nope, also not interested.
We worked with a client and analyzed their workload and software. With a few minor changes we could reduce their always on EC2 instances with 66%. They where not interested. Then at least buy reserved instances? Nope, also not interested.
For sure. And making sure you aren't using them inefficiently. At one point we were using 30% of our AWS spend doing S3 API calls. The next day it was 1% by optimizing that.
We were using S3 as ternary storage for our time series database by moving values from the secondary SSD based storage to S3. Each of those values contained all the values for a full day. By using HTTP range queries and placing an index in the SSD store we were able to combine 30 days of values into a single S3 value to decrease the API PUT calls by 30x.
Was hoping there would be more of a numerical comparison to various strategies when scaling and a comparison to other strategies such as say self-hosting :(
(aws has a useful place and a use, but obviously spawning a new vm for every query would just maximise cost over time)
(aws has a useful place and a use, but obviously spawning a new vm for every query would just maximise cost over time)
If you're somewhat poor like me (living in Eastern Europe, current net salary around 1500 euros a month; provides decent quality of life here but not in a globalized economy), then the first step of saving money would be not to bother with the expensive cloud vendors: AWS, Azure and GCP.*
* this advice does not apply if you're a cog in an enterprise, then use whatever the company mandates
Here's a few alternatives, from the most expensive to the least expensive:
DigitalOcean: https://www.digitalocean.com/pricing/ A pretty popular VPS provider that i'd say is cheaper than the above to start us out. They also offer a whole bunch of different managed services, if you're into that sort of stuff.
Vultr: https://www.vultr.com/products/cloud-compute/#pricing Much like DigitalOcean, sans some of the managed services. On the upside, they also sell smaller instances, though those tend to be sold out in my experience.
Scaleway: https://www.scaleway.com/en/pricing/ Have a look at their Development or Starburst instances (the latter are smaller ones like Vultr), also they rival DigitalOcean in their managed offerings. Pretty good CPU performance as well, in my experience.
Hetzner: https://www.hetzner.com/cloud I'd say that they're cheaper than all of the above, but also have a reasonably modern control panel, as well as block storage services if you need more space. ID verification might be necessary if you're from an undesireable country, though, so YMMV.
Time4VPS: https://www.time4vps.com/?affid=5294 I have used them for almost all of my servers in the last 4 years or so (hence affiliate link in case you check them out) - the control panel is somewhat more dated and the managed offerings are limited, but they're one of the cheapest legitimate hosts, since they're owned by a Lithuanian telco. They also offer noticeable discounts if you reserve instances for a year and i'd say they're a good choice for most purposes, provided that you have backups (i have a few backup servers in my home for that purpose).
Contabo: https://contabo.com/en/vps/ Perhaps the best specs that you can get on the cheap, especially if you're after a decent amount of storage, which is larger than most of the other hosts provide you with (in lieu of block storage services). They do have a setup fee for the instances, the process seems at least partially manual on their part, the web UI is the most antiquated i've seen of the bunch, the performance of the instances is mediocre (they probably overprovision), but it all seems to work regardless.
There are also other hosts out there, but the shadier they are, the more likely data loss and/or theft is. But hey, balance your needs with your capabilities to find what works the best for you! Perhaps i'll even write a blog post and include some automated benchmarks in the future on my blog.
Edit: if you feel like spending some of your time looking for bargains (yaay for low alternative costs), then feel free to have a look at LowEndBox, where interesting deals are sometimes advertised: https://lowendbox.com/
Personally, however, i'd only pick companies that have been around for $SOME years.
* this advice does not apply if you're a cog in an enterprise, then use whatever the company mandates
Here's a few alternatives, from the most expensive to the least expensive:
DigitalOcean: https://www.digitalocean.com/pricing/ A pretty popular VPS provider that i'd say is cheaper than the above to start us out. They also offer a whole bunch of different managed services, if you're into that sort of stuff.
Vultr: https://www.vultr.com/products/cloud-compute/#pricing Much like DigitalOcean, sans some of the managed services. On the upside, they also sell smaller instances, though those tend to be sold out in my experience.
Scaleway: https://www.scaleway.com/en/pricing/ Have a look at their Development or Starburst instances (the latter are smaller ones like Vultr), also they rival DigitalOcean in their managed offerings. Pretty good CPU performance as well, in my experience.
Hetzner: https://www.hetzner.com/cloud I'd say that they're cheaper than all of the above, but also have a reasonably modern control panel, as well as block storage services if you need more space. ID verification might be necessary if you're from an undesireable country, though, so YMMV.
Time4VPS: https://www.time4vps.com/?affid=5294 I have used them for almost all of my servers in the last 4 years or so (hence affiliate link in case you check them out) - the control panel is somewhat more dated and the managed offerings are limited, but they're one of the cheapest legitimate hosts, since they're owned by a Lithuanian telco. They also offer noticeable discounts if you reserve instances for a year and i'd say they're a good choice for most purposes, provided that you have backups (i have a few backup servers in my home for that purpose).
Contabo: https://contabo.com/en/vps/ Perhaps the best specs that you can get on the cheap, especially if you're after a decent amount of storage, which is larger than most of the other hosts provide you with (in lieu of block storage services). They do have a setup fee for the instances, the process seems at least partially manual on their part, the web UI is the most antiquated i've seen of the bunch, the performance of the instances is mediocre (they probably overprovision), but it all seems to work regardless.
There are also other hosts out there, but the shadier they are, the more likely data loss and/or theft is. But hey, balance your needs with your capabilities to find what works the best for you! Perhaps i'll even write a blog post and include some automated benchmarks in the future on my blog.
Edit: if you feel like spending some of your time looking for bargains (yaay for low alternative costs), then feel free to have a look at LowEndBox, where interesting deals are sometimes advertised: https://lowendbox.com/
Personally, however, i'd only pick companies that have been around for $SOME years.
There are also a bunch of OpenStack Public Cloud providers in Europe and around the world: https://www.openstack.org/marketplace/public-clouds/ Some are not listed there so you may have to do some research (I know there's one in Chicago)
These "Cloud" systems provide a number of advantages over VPS providers. They have a standard programmable interface and command-line tools, they provide many services you may or may not get with a given VPS provider (programmable networks, load balancers, object and block storage, secrets management, configurable user/role access controls, etc). They also of course allow you to scale your resources programmatically and may provide pay-as-you-go pricing. And if it's something you're into, you can typically find a Terraform provider that works with it.
Slightly more expensive than a simple VPS, but the automation, failure recovery, and security gains can be significant.
These "Cloud" systems provide a number of advantages over VPS providers. They have a standard programmable interface and command-line tools, they provide many services you may or may not get with a given VPS provider (programmable networks, load balancers, object and block storage, secrets management, configurable user/role access controls, etc). They also of course allow you to scale your resources programmatically and may provide pay-as-you-go pricing. And if it's something you're into, you can typically find a Terraform provider that works with it.
Slightly more expensive than a simple VPS, but the automation, failure recovery, and security gains can be significant.
Thanks for dropping that link, it feels like there's definitely a niche that OpenStack can fill in nicely!
I guess the only other alternative is to look for the lowest common denominator, which in the case of Linux servers is typically something like Ansible/Salt - to connect to VPSes through SSH and do all of the necessary configuration from bottom up in an automated and repeatable manner.
Of course, that's not to say that it's always easy and i applaud what OpenStack is trying to do with providing APIs for a lot of that stuff.
I guess the only other alternative is to look for the lowest common denominator, which in the case of Linux servers is typically something like Ansible/Salt - to connect to VPSes through SSH and do all of the necessary configuration from bottom up in an automated and repeatable manner.
Of course, that's not to say that it's always easy and i applaud what OpenStack is trying to do with providing APIs for a lot of that stuff.
Yeah, they sort of have different purposes. VPS providers are the easiest way to just start using some VMs for static workloads. But Cloud providers give you not only a ton more control, but also allow you to save money by only getting charged for the resources you use. The tradeoffs being that the latter is way more complicated, and the "hyper-scale" Cloud providers are extremely overpriced. I'm hoping more small providers pop up and make the big girls more competitively priced.
Another way to slash cost is to use https://cloudoptimizer.io to locate cheapest cloud resources (disclaimer: my project)
Write and use high performance software, you pay for what you use after all. I've been using Nim for some time now, and it gives me that with a syntax comparable to Python.
Eh, my only experience with Vantage so far has been their apparent purchase and breaking of the ec2instances website.
Hey there, can you please let me know what you're referring to? We largely haven't touched the site at all other than hosting it on our AWS account which as far as I know didn't impact anything negatively.
Additionally, we've built an API for the data served through it: https://vantage.readme.io/reference
We have future plans to develop the site but I just want to ensure I'm aware of what you're referring to if you're seeing something we aren't aware of. Please feel free to email me directly at [email protected] if you'd like.
Additionally, we've built an API for the data served through it: https://vantage.readme.io/reference
We have future plans to develop the site but I just want to ensure I'm aware of what you're referring to if you're seeing something we aren't aware of. Please feel free to email me directly at [email protected] if you'd like.
Step 1. Stop using AWS.
By not using AWS I have saved thousands. This one simple trick.
Ah yes you're at the fifth level of AWS cost management consciousness. You may have skipped the first four levels.
First is the simple test case using something random like Lambda and S3 after dragging through the Whizlabs course. This costs you $5 a month.
Second is the migration of something not particularly complicated but a bit meatier which works out cheaper than your capital expenditure coming up so you can write it off without having to fill in a purchase order and argue with accounts again.
Third is the overconfident architectural approach of multi-account, multi-AZ with peering all over the shop as recommended in the best practices, certification and architecture documentation. Approving nods all around on delivering this, despite the operational expenditure being slightly higher than predicted on your hacked up and not totally complete Excel spreadsheet for cost management.
Fourth is the first bill. This immediately points out your inter-VPC, inter-AZ transit and shitty shared tenancy CPU provision you had to crank up quickly at the last minute, costs more a month than your entire infrastructure capex for 3 years did before you got AWS resulting in sad kitty faces all around and a scramble for a cheaper option while trying not to get fired. This is all while Bezos dances on the flames coming from the dollar bills he's burning in a giant bonfire cackling loudly.
Fifth is several months later after being on the job market, eating ramen and searching for a company which "doesn't do any of that cloud stuff". You eventually find a position herding a couple of 1U supermicro boxes with CentOS on them which require the odd disk replacing here and there and some PHP updating without going near Terraform, Jenkins or any of that shit. Your entire infrastructure upgrade is automating your entire job into a few ansible playbooks and spending 6 hours a day inspecting the insides of your eyelids.
First is the simple test case using something random like Lambda and S3 after dragging through the Whizlabs course. This costs you $5 a month.
Second is the migration of something not particularly complicated but a bit meatier which works out cheaper than your capital expenditure coming up so you can write it off without having to fill in a purchase order and argue with accounts again.
Third is the overconfident architectural approach of multi-account, multi-AZ with peering all over the shop as recommended in the best practices, certification and architecture documentation. Approving nods all around on delivering this, despite the operational expenditure being slightly higher than predicted on your hacked up and not totally complete Excel spreadsheet for cost management.
Fourth is the first bill. This immediately points out your inter-VPC, inter-AZ transit and shitty shared tenancy CPU provision you had to crank up quickly at the last minute, costs more a month than your entire infrastructure capex for 3 years did before you got AWS resulting in sad kitty faces all around and a scramble for a cheaper option while trying not to get fired. This is all while Bezos dances on the flames coming from the dollar bills he's burning in a giant bonfire cackling loudly.
Fifth is several months later after being on the job market, eating ramen and searching for a company which "doesn't do any of that cloud stuff". You eventually find a position herding a couple of 1U supermicro boxes with CentOS on them which require the odd disk replacing here and there and some PHP updating without going near Terraform, Jenkins or any of that shit. Your entire infrastructure upgrade is automating your entire job into a few ansible playbooks and spending 6 hours a day inspecting the insides of your eyelids.
Hilariously true. Yes, AWS has some services that are cheap and not really replaceable, like S3, but once you come near high performance EC2 and RDS and add multi-region in, you’ll really have a bad time. Believe me, Ive been there and in the end had to migrate most of the applications to another provider or host on prem.
I've helped with two migrations, and warned them; they didn't listen.
Last place also did datadog for log aggregation. So Bezos' hands in their pockets, datadog's paws on their money.
Last place also did datadog for log aggregation. So Bezos' hands in their pockets, datadog's paws on their money.
Logs are something I didn't love self-hosting, but I also don't have a good sense that ELK was really an ideal setup for that. I can imagine running my own hardware, what's out there for logging?
(DD is super expensive, as you've said)
(DD is super expensive, as you've said)
The only thing I can actually suggest is to avoid the hell out of logging if you can. It’s a really expensive concern and should be treated so from day one. If your system has nothing actionable to log them don’t log it.
At a high level from observations of trying to handle 100Gb a day.
Cloudwatch is inflexible and expensive.
ELK is expensive to run and administer. The commercial variants are even worse.
Splunk is expensive and slow.
Datadog is expensive.
Loki is expensive to run and administer.
ryslogd and grep starts to feel like a viable solution eventually. Then you realise that you need about 50MB of that 100G a day of logs and enlightenment comes.
At a high level from observations of trying to handle 100Gb a day.
Cloudwatch is inflexible and expensive.
ELK is expensive to run and administer. The commercial variants are even worse.
Splunk is expensive and slow.
Datadog is expensive.
Loki is expensive to run and administer.
ryslogd and grep starts to feel like a viable solution eventually. Then you realise that you need about 50MB of that 100G a day of logs and enlightenment comes.
Thanks for the response, yeah that resonates. I think the sheer volume of logs has been what necessitates expensive solutions.
I've found that my current team really relies on those and an event store for observability, and it's just clunky and awful. Logging for, well, logging purposes instead would be a dramatic improvement.
I've found that my current team really relies on those and an event store for observability, and it's just clunky and awful. Logging for, well, logging purposes instead would be a dramatic improvement.
Eh. S3 is easily replaced by Wasabi. And less easily by minio.
Wasabi has an important 'gotcha': they charge you for a minimum of 90 days of retention, even if you delete an object seconds after creating it.
https://wasabi.com/paygo-pricing-faq/#minimum-storage-durati...
That detail isn't mentioned anywhere on their pricing page or cost comparison calculator: https://wasabi.com/cloud-storage-pricing/
I don't find that pricing objectionable on its own, but I'm wary of shopping with a vendor that advertises price as their main selling point, but buries such a potentially costly pricing detail.
https://wasabi.com/paygo-pricing-faq/#minimum-storage-durati...
That detail isn't mentioned anywhere on their pricing page or cost comparison calculator: https://wasabi.com/cloud-storage-pricing/
I don't find that pricing objectionable on its own, but I'm wary of shopping with a vendor that advertises price as their main selling point, but buries such a potentially costly pricing detail.
Damn that’s cheap on wasabi. No egress fees! Thanks for pointing this one out.
Do note that wasabi is aimed for long-term retention of cold data. There's no egress, but there is a fair usage policy (2x data stored IIRC).
I use it for live data, but I put it behind BunnyCDN. I serve a few terabytes per month using that setup, no problem.
Most real world uses that I’m aware of follow the 80/20 rule, meaning you’ll store much more than you serve. And your hot paths are quickly cached by a CDN.
Most real world uses that I’m aware of follow the 80/20 rule, meaning you’ll store much more than you serve. And your hot paths are quickly cached by a CDN.
If any AWS service has replacement at this point, it has got to be S3 (unless locked-in via peripheral aws services).
https://wasabi.com, https://backblaze.com, https://min.io some of the ones I've seen frequently mentioned on news.yc
https://wasabi.com, https://backblaze.com, https://min.io some of the ones I've seen frequently mentioned on news.yc
It's fucking ironic how AWS selling point was how it was more cost-effective than managing your own infra.
Yet another classic SV-style rugpull.
Yet another classic SV-style rugpull.
There’s a very subtle difference between “managing” and “operating”. There’s also a rather large jump from paying AWS to manage and hiring a complete operations team.
But you are right, running on AWS isn’t cheaper. It can be, but you have to design your software to take advantage of the platform.
But you are right, running on AWS isn’t cheaper. It can be, but you have to design your software to take advantage of the platform.
Me too. The data transfer costs would be the killer for me (video conferencing).
Switch from AWS to Google Cloud to save costs and not have to pay an external service to give you sizing recommendations.
[deleted]