Please fix the AWS free tier before somebody gets hurt(cloudirregular.substack.com)
cloudirregular.substack.com
Please fix the AWS free tier before somebody gets hurt
https://cloudirregular.substack.com/p/please-fix-the-aws-free-tier-before
456 comments
I'm also going to point out, as a former AWS engineer, that
"too hard" isn't in the AWS engineering vocabulary. If an AWS engineer were to dismiss something as "too hard", they'd be on the fast track to a performance-improvement-plan.
The problem isn't that it's too hard. It's that it isn't a business priority. As soon as it becomes prioritised, a legion of very smart AWS engineers will solve it.
The problem isn't that it's too hard. It's that it isn't a business priority. As soon as it becomes prioritised, a legion of very smart AWS engineers will solve it.
As an antidote to the "nothing is too hard" flex...
Everything has trade-off's. When a responsible person says something is "too hard" it's because they've evaluated the consequences and have deemed them too costly in terms of time, resources, cost, maintenance, or strategy/lost-opportunity. Some things really are "too hard".
I agree it's not too hard technically but am surprised at your contradictory suggestion AWS would deploy "a legion of very smart engineers" to solve it if they decided to implement that ordinary feature. How smart are they?
I suspect the truth here is a bit more ugly. AWS doesn't want to do it because they see the pocket-change taken from accidental mishaps as a net positive. They don't care if it inflicts suffering on students and novices. Sure, they'll fix individual cases if they're shamed publicly or if the individual contacts them in exactly the right way, but at the end of the day, it's more money in their pocket. For every 200 dollar oopsie that gets charged back, how many go silently paid in humiliation and hardship?
Everything has trade-off's. When a responsible person says something is "too hard" it's because they've evaluated the consequences and have deemed them too costly in terms of time, resources, cost, maintenance, or strategy/lost-opportunity. Some things really are "too hard".
I agree it's not too hard technically but am surprised at your contradictory suggestion AWS would deploy "a legion of very smart engineers" to solve it if they decided to implement that ordinary feature. How smart are they?
I suspect the truth here is a bit more ugly. AWS doesn't want to do it because they see the pocket-change taken from accidental mishaps as a net positive. They don't care if it inflicts suffering on students and novices. Sure, they'll fix individual cases if they're shamed publicly or if the individual contacts them in exactly the right way, but at the end of the day, it's more money in their pocket. For every 200 dollar oopsie that gets charged back, how many go silently paid in humiliation and hardship?
I would speculate that Amazon doesn’t care about the occasional $200 mistake, and is perfectly happy to refund it should they notice.
What they don’t want is companies that are spending say $2-4 million per month to be able to put in a hard limit at $2.5M.
I would guess most companies spending millions per month on AWS are just guessing what next month’s bill will be, and have been conditioned to not freak out if it’s a million over, as long as that doesn’t happen every month. That conditioning has to start early.
What they don’t want is companies that are spending say $2-4 million per month to be able to put in a hard limit at $2.5M.
I would guess most companies spending millions per month on AWS are just guessing what next month’s bill will be, and have been conditioned to not freak out if it’s a million over, as long as that doesn’t happen every month. That conditioning has to start early.
> What they don’t want is companies that are spending say $2-4 million per month to be able to put in a hard limit at $2.5M.
Why not? It would make companies happy.
The reality is that, as much as many would like this feature, I'm not sure that a lot of large customers are actually asking for it. Does anyone who works at a large company know if they've asked for this feature before?
I can imagine why startups might want it, but also, AWS is extremely forgiving with startups and throws tons of money at them, so idk.
Why not? It would make companies happy.
The reality is that, as much as many would like this feature, I'm not sure that a lot of large customers are actually asking for it. Does anyone who works at a large company know if they've asked for this feature before?
I can imagine why startups might want it, but also, AWS is extremely forgiving with startups and throws tons of money at them, so idk.
The problem is, what to do when your account hits the limit? The subtle point about EC2 is you can’t actually shut down servers and have them come back up later unless you have a specific configuration - which not everybody has, so you can’t simply power off systems and not destroy something. Not insurmountable, but not trivial either. You also can’t say that there’s a limit except for an incompatible subset of products, nor can you ask users to be okay with randomly destroying things - the mere rumor of that destroys user trust in something. (Who’s still shy about using EBS despite the last major incident being back in… 2016? Also that outage was limited to one region.)
> nor can you ask users to be okay with randomly destroying things
what do you think will happen to your EC2 instances if your payment method on file with Amazon fails repeatedly and for more than 30 days? Do you think they will just keep those instances intact indefinitely without payment because they promised not to 'randomly destroy things'?
what do you think will happen to your EC2 instances if your payment method on file with Amazon fails repeatedly and for more than 30 days? Do you think they will just keep those instances intact indefinitely without payment because they promised not to 'randomly destroy things'?
You are correct, which makes it even more pernicious that Amazon chooses to not fix this ... profits at any cost is not a long term benefit ... the longer AWS fails to fix this more folks will tend to lean toward the competition especially Azure ... I have been a unix/linux server side developer since the beginning and always use a linux laptop so no fan of Microsoft however their Azure platform is lightyears ahead of AWS ... its obvious the AWS console web front end needs a wholescale rewrite from the bottom up ... again as you say its not a priority probably because all big consumers never use the AWS console as they have written their own automation layer atop the AWS SDK
I routinely tell a coworker that <random task> isn't possible so that he does the work for me. Oh I know it's possible, I just don't want to do it. Thanks Corey! You're the best.
This works in help forums too. Post a question, there's a chance you get ignored. Post something wrong, and someone will flame you and answer your question.
Cunningham’s Law
(Had to resist the urge to post “Occam’s Razor” or some other incorrect thing to bait someone to correct me)
(Had to resist the urge to post “Occam’s Razor” or some other incorrect thing to bait someone to correct me)
You should have tried that, would have gotten the point across even more effectively :)
As someone who tries to always get things right, it somehow seems wrong to deliberately post wrong answers.
I know it won't go over well at Stackoverflow, which is one of the very best places to get accurate and helpful information.
As someone who tries to always get things right, it somehow seems wrong to deliberately post wrong answers.
I know it won't go over well at Stackoverflow, which is one of the very best places to get accurate and helpful information.
I had a smart coworker at one point, most of the time I asked for help with something, he would say that he did not know the answer.
Now I'm positive that he was doing the same thing as you describe.
The effect on me was a reduced interaction with that person, I'd much rather have an honest answer or a "I don't have time" answer.
Now I'm positive that he was doing the same thing as you describe.
The effect on me was a reduced interaction with that person, I'd much rather have an honest answer or a "I don't have time" answer.
This works until it doesn't, once Corey figures you out he might seek revenge...
score one for manipulation
This is in line with all the bad things I've heard about Amazon. Why do people work there?
Why is this particular thing bad?
"Too hard" is not an excuse. They have all the resources they need at their disposal to get the job done.
"Too hard" is not an excuse. They have all the resources they need at their disposal to get the job done.
If there is no X for which you can say "X is too hard" without getting PIP'ed, then your work environment is dysfunctional.
Pretty sure is the ask was to solve the Halting Problem that you wouldn’t get in trouble.
Scale that isn't available in most companies.
A culture they feel aligned with.
They are poor or high burn rate consumers.
They are curious.
But, it would seem, most of all they have different priorities that you.
I write that as someone who routinely dismisses their recruitment attempts.
A culture they feel aligned with.
They are poor or high burn rate consumers.
They are curious.
But, it would seem, most of all they have different priorities that you.
I write that as someone who routinely dismisses their recruitment attempts.
The CV title of having worked at FAANG and the big tech compensation.
In the UK at least, Amazon pay relatively poorly compared to other the FAANG companies and Finance.
> "too hard" isn't in the AWS engineering vocabulary
Let these people fix climate change then.
Sarcasm aside, it must be boring working in a place where no problem is ever too hard.
Let these people fix climate change then.
Sarcasm aside, it must be boring working in a place where no problem is ever too hard.
a couple weeks ago Janet Yellen warned that net-zero carbon would cost about 2.5 trillion
that's cheaper than the pandemic. no reason it couldn't have been done already.
i no longer consider climate change a hard problem, the only hard problem is building political momentum without getting shot
that's cheaper than the pandemic. no reason it couldn't have been done already.
i no longer consider climate change a hard problem, the only hard problem is building political momentum without getting shot
I realize it can be fun to be snarky but consider the “business priority” portion you didn’t quote and the difference between “it’s too hard to do” and “doing it would cost X and impact performance of service Y by Z%”.
> Every other cloud does this! They all have non-production subscription types with hard spending limits.
They don't. I've looked. They all have some ultra complex scheme for monitoring billing and programmatically shutting down services, which is a favorite recommendation by apologists, but none have a nice, solid "shut down everything" plan that's usable for learning and testing.
AWS does have billing actions which are fairly new, but they're still a bit too difficult to deal with if you also follow the advice for setting up a root account for billing plus sub-accounts / organizations for use.
Azure has been ignoring the request for over 8 years [1]. There's a warning they're moving away from User Voice to a product-by-product solution for feedback, so that request might disappear altogether. I reached out to them and politely asked for an update and they said they'd look into it, but it's been over a month and I don't think there's going to be an update without some pressure. If anyone has a good social media following and would like to see hard caps on spending in Azure, tweet @AzureSupport. Maybe if enough people do they'll actually follow up.
> PLEASE stop making the argument that it's "too hard" or even "impossible" to implement spending limits.
I agree 100% with this because all I really want is something along the lines of the "personal learning" account that's described in the article. I want a dev / testing account where I can learn and test assumptions about billing without risking a massive overage that I wasn't expecting. They don't have to get into the complexities of dealing with production accounts.
AWS, Azure, and GCP have all had close to a decade to figure it out, so I think the only way it'll happen is if we start asking legislators to regulate those providers. It would be relatively simple in my opinion. Cloud infrastructure providers must allow users to indicate a maximum amount of spending per month and may not charge more than the indicated amount.
I think big tech is proving they can't self regulate, so it's time to quit letting them.
1. https://feedback.azure.com/forums/170030-signup-and-billing/...
They don't. I've looked. They all have some ultra complex scheme for monitoring billing and programmatically shutting down services, which is a favorite recommendation by apologists, but none have a nice, solid "shut down everything" plan that's usable for learning and testing.
AWS does have billing actions which are fairly new, but they're still a bit too difficult to deal with if you also follow the advice for setting up a root account for billing plus sub-accounts / organizations for use.
Azure has been ignoring the request for over 8 years [1]. There's a warning they're moving away from User Voice to a product-by-product solution for feedback, so that request might disappear altogether. I reached out to them and politely asked for an update and they said they'd look into it, but it's been over a month and I don't think there's going to be an update without some pressure. If anyone has a good social media following and would like to see hard caps on spending in Azure, tweet @AzureSupport. Maybe if enough people do they'll actually follow up.
> PLEASE stop making the argument that it's "too hard" or even "impossible" to implement spending limits.
I agree 100% with this because all I really want is something along the lines of the "personal learning" account that's described in the article. I want a dev / testing account where I can learn and test assumptions about billing without risking a massive overage that I wasn't expecting. They don't have to get into the complexities of dealing with production accounts.
AWS, Azure, and GCP have all had close to a decade to figure it out, so I think the only way it'll happen is if we start asking legislators to regulate those providers. It would be relatively simple in my opinion. Cloud infrastructure providers must allow users to indicate a maximum amount of spending per month and may not charge more than the indicated amount.
I think big tech is proving they can't self regulate, so it's time to quit letting them.
1. https://feedback.azure.com/forums/170030-signup-and-billing/...
Annoyingly, Azure does have this capability! If your company is a Microsoft gold partner (there might be other requirements, I can't remember) then your Azure developer licenses comes with £100 free credit per month. When this money runs out, every service associated with that subscription just shuts down. It doesn't overrun, heck I don't remember even getting an email about it, they just stop until it renews the next month.
They just refuse to implement this system for non dev licenses.
They just refuse to implement this system for non dev licenses.
btw. azure has a subscription type where you can put a prepaid amount onto your account. they just do not support it by default. btw. bizspark customers get this of subscription, for 3 years with an free amount of up to 300 USD per month
IBM Cloud has Lite accounts, which are quite limited but always free.
"Every other cloud does this! They all have non-production subscription types with hard spending limits."
This is simply not true, to the best of my knowledge. GCP has budget limits, which can send alerts, but outside of manually catching a pubsub and shutting down billing on your own, there are no configurable hard billing limits.
I'm unsure what to make of the rest of your rant.
This is simply not true, to the best of my knowledge. GCP has budget limits, which can send alerts, but outside of manually catching a pubsub and shutting down billing on your own, there are no configurable hard billing limits.
I'm unsure what to make of the rest of your rant.
The budget limits are delayed hard billing limits. Everything I've seen/heard suggests while you can go over, they absolutely don't charge the overage.
This is simply false.
We go over our budget "limits" literally every month. We do this because they are simply alerts, nothing more. We have separate alerts set up for 25, 50, 75, and 100% of our "limit" so that we can track variable usage and catch spikes early.
We, of course, pay our full bill every month - over the "limit".
We go over our budget "limits" literally every month. We do this because they are simply alerts, nothing more. We have separate alerts set up for 25, 50, 75, and 100% of our "limit" so that we can track variable usage and catch spikes early.
We, of course, pay our full bill every month - over the "limit".
^this. For business-critical operations you almost _never_ want to have your service shut down, especially as a surprising result of a big spike in usage.
You can write a cloud function to automatically run when you go over the budget and shut down billing. They provide code for doing so in the docs:
https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
I'm not sure if that's what you meant by "manually catching a pubsub" -- you have to manually set up the script once, but it will run automatically when triggered.
https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
I'm not sure if that's what you meant by "manually catching a pubsub" -- you have to manually set up the script once, but it will run automatically when triggered.
Yes, well aware. I'm not sure if you actually looked at the code on that page but you have to catch the pub sub in the deployed cloud function in order to shut down billing on the project/account. This is precisely what I was referring to.
If we're talking about protecting users still trying to learn the cloud, what if they mess up the setup of this cloud function?
It's worse – for us, budget alerts often arrive several hours after the fact, when you already might have burned lots of money.
AWS has that too - you can trigger actions on budget alerts such as stopping activity https://aws.amazon.com/blogs/aws-cost-management/get-started...
What we need, a function that is AWS maintained and supported, that basically is AWS responsibility - that if it did not work - failed to shut down service that was consuming all the budget, they eat the cost.
>future hires need to start somewhere, need to be able to learn on their own, and need to do so safely.
I've shared this before on hn, but I was about to bite the bullet on a fire base subscription to back a web app I'm building. Google announced its cancellation the week I was going to subscribe, I was hitting daily limits in my testing with my garage code and wanted the extra buffer to speed up my progress. As soon as they announced the cancellation I ripped out fire base and have been working on a swap to hasura/postgres/auth0. I can't afford, likely multiple, hundred or thousand dollar mistakes for a hobby.
"Just code better"
I'm trying...
I've shared this before on hn, but I was about to bite the bullet on a fire base subscription to back a web app I'm building. Google announced its cancellation the week I was going to subscribe, I was hitting daily limits in my testing with my garage code and wanted the extra buffer to speed up my progress. As soon as they announced the cancellation I ripped out fire base and have been working on a swap to hasura/postgres/auth0. I can't afford, likely multiple, hundred or thousand dollar mistakes for a hobby.
"Just code better"
I'm trying...
Which clouds do this? I work quite a bit with AWS and GCP, and Azure to a lesser extent, and I've yet to see a spending limit on any of them. I'd love to be proven wrong though.
Azure definitely has subscription types that have payment limits. I have one subscription that has some credits on it, and if I go over the limits all services are suspended (i.e. compute, storage, etc) and I can't do anything unless I add more money or wait until the next period starts and the new credits kick in.
Not all Azure subscriptions types are like this though.
Not all Azure subscriptions types are like this though.
This would be the list: https://azure.microsoft.com/en-ca/support/legal/offer-detail...
Also documented at https://docs.microsoft.com/en-us/azure/cost-management-billi...
Also documented at https://docs.microsoft.com/en-us/azure/cost-management-billi...
It's something that's frustrated me about Azure for years. They can do it, but they won't do it for pay-as-you-go accounts.
Make a SKU that's not for production with a $100 setup fee to prevent abuse and prepaid credits for consumption.
Make a SKU that's not for production with a $100 setup fee to prevent abuse and prepaid credits for consumption.
GCP has a spending limit, but it is by no means transparent[0]
Basically you add a budget to your project, create a pub/sub channel, then make a lambda that subscribes to the channel and turns off billing when it receives an overbudget message to shut everything down.
Or you could integrate it into your project directly to handle this more gracefully.
You can even send a test message, to check that everything does indeed shut down, but the whole process is extremely clunky and error-prone.
[0] - https://cloud.google.com/billing/docs/how-to/notify
Basically you add a budget to your project, create a pub/sub channel, then make a lambda that subscribes to the channel and turns off billing when it receives an overbudget message to shut everything down.
Or you could integrate it into your project directly to handle this more gracefully.
You can even send a test message, to check that everything does indeed shut down, but the whole process is extremely clunky and error-prone.
[0] - https://cloud.google.com/billing/docs/how-to/notify
Azure gives you credits to spend. And after you have exhausted your credits, then they don't charge more.
If you have Visual Studio professional or enterprise license, then you can avail recurring credits, which are useful for learning
If you have Visual Studio professional or enterprise license, then you can avail recurring credits, which are useful for learning
Yeah, but that's $45 USD per month which is an expensive way to learn and test things that would normally cost <$5. It does get you $50 per month of Azure credits though, so it's probably the route I'd go if it's possible to buy a month of Visual Studio and ignore everything but the Azure credits.
I had (have?) an Azure account through my university account and while I agree that Azure services such as Container Service are (in my not so humble opinion) ridiculously overpriced but the fact it I never entered payment details to use my USD 100 of free credit on Azure. I effectively have spending cap of zero dollars.
I thought I was the only one who thought pricing on Azure seemed rather high for what you get.
But I also have free credit I can use each month, which offsets the cost to "normal" levels, but can't imagine being a business and opting to pay for Azure services by choice, unless they provide something no one else does.
I worked at mega corp and we used azure services at >50% off the sticker price
Enterprise deals can end up changing the story it seems
Enterprise deals can end up changing the story it seems
That would explain allot. Totally forgot big business usually gets big discounts.
Had Bizspark from microsoft which gave us access to Azure and about $300 in credits I think per service or the whole thing iirc. They shut down services once we went over that limit. This was back in 2016/2017
With azure you can set spending alerts at the very least.
GCP definitely has a spending limit. Of course it also (at least in the past) used to take 24 hours to update, so if you had a massive spike in real usage, your site would die for a day while you frantically tried to increase the limit.
Why are people saying this? It's not true. App Engine had a spending limit, but it's deprecated and going away in July. The rest of GCP has never had a spending limit.
The best you can do is build your own service to monitor your bill and manually cut off your billing account if it gets too high. Hope you did it correctly because there's no good way to test it! And the bill notifications can be delayed an arbitrary amount of time with no guarantees, and disabling billing can take an arbitrary amount of time too, so you can still be screwed even if you did it correctly.
I think it's a glaring hole in cloud platforms. I get it, it's hard to implement in a way that is forgiving and doesn't cause a bunch of support tickets from people who set their limits too low and made their site go down. But that's why people use cloud platforms, so they don't have to implement the hard part. It's beyond me why the cloud platforms don't do something better here.
The best you can do is build your own service to monitor your bill and manually cut off your billing account if it gets too high. Hope you did it correctly because there's no good way to test it! And the bill notifications can be delayed an arbitrary amount of time with no guarantees, and disabling billing can take an arbitrary amount of time too, so you can still be screwed even if you did it correctly.
I think it's a glaring hole in cloud platforms. I get it, it's hard to implement in a way that is forgiving and doesn't cause a bunch of support tickets from people who set their limits too low and made their site go down. But that's why people use cloud platforms, so they don't have to implement the hard part. It's beyond me why the cloud platforms don't do something better here.
Why should they if they can use obscurity and people's limited comprehension of their complicated systems to produce legitimate, legal bills to customers?
You're talking a 'get paid/don't get paid' decision here.
When you look at this on a high enough distributed level, across enough users, this is the profit margin. Enough people can screw up and then be able to pay the cloud provider, that it is part of the business model. You're required to run some chance of producing a legitimate megabill.
Why be forgiving when your business model requires that you set traps for fools?
You're talking a 'get paid/don't get paid' decision here.
When you look at this on a high enough distributed level, across enough users, this is the profit margin. Enough people can screw up and then be able to pay the cloud provider, that it is part of the business model. You're required to run some chance of producing a legitimate megabill.
Why be forgiving when your business model requires that you set traps for fools?
> As the article points out: Every other cloud does this!
No, they don’t, the article is misleading if not technically wrong (since they use “and/or” to link billing caps and better whole-project deletion, the latter of which mitigates a subset of the problem but doesn’t address accidental unbounded spend from a project that you want active at some level) and the same general class of complaint is raised for the other clouds on this issue; the frequency of the complaints seems to vary (ordinally, but not linearly) with usage of the various clouds.
No, they don’t, the article is misleading if not technically wrong (since they use “and/or” to link billing caps and better whole-project deletion, the latter of which mitigates a subset of the problem but doesn’t address accidental unbounded spend from a project that you want active at some level) and the same general class of complaint is raised for the other clouds on this issue; the frequency of the complaints seems to vary (ordinally, but not linearly) with usage of the various clouds.
Hmm not sure where the problem is. With IAM you can create a policy to limit the user to only use certain services, deploy only the cheapest type of instance type, limit the region, the time of day etc. All of this and more is built in to IAM by default. You can even create a CloudFormation as a internal product this way you can limit even more what people do, and budget for that. And on top of that you can make a Lambda that is triggered every 1h to shut down unused resources or send a warning to shut it down, if not, it will go down the next day.
The possibilities are endless. I personally don't see the IBM comparison.
The possibilities are endless. I personally don't see the IBM comparison.
Is it reasonable to expect a brand new aws user to know how to do those things?
The same as it is not expected by anyone to know how to ride a bike or drive a car. Someone has to teach you or you have to go and take a course.
Same with any technology, you should read first all the documentation that there is, watch all the videos that AWS releases every year for free where they explain in detail every service that they have, and try to explain the best practices of it - this is all for free, or you could spend $1000 for a course, instead of risking to lose $10.000.
If you jump in to the water without knowing how to swim, and then get angry at the water because you drown - well...
After 18 years of age you become a grown up, because your parents are not responsible for you anymore. You become responsible for yourself. That is what distinguish a child from a grown up.
Of course a parent needs also to tech you responsibility and what it means.
Mine for example never did, and I had to learn life the hard way. At first I was blaming others, but then I realized that to really grow up, I need to stop blaming others and start owning my mistakes.
So, it is not expected for anyone to know everything, but it is expected that if you want to learn something new, you need to first research the topic.
Same with any technology, you should read first all the documentation that there is, watch all the videos that AWS releases every year for free where they explain in detail every service that they have, and try to explain the best practices of it - this is all for free, or you could spend $1000 for a course, instead of risking to lose $10.000.
If you jump in to the water without knowing how to swim, and then get angry at the water because you drown - well...
After 18 years of age you become a grown up, because your parents are not responsible for you anymore. You become responsible for yourself. That is what distinguish a child from a grown up.
Of course a parent needs also to tech you responsibility and what it means.
Mine for example never did, and I had to learn life the hard way. At first I was blaming others, but then I realized that to really grow up, I need to stop blaming others and start owning my mistakes.
So, it is not expected for anyone to know everything, but it is expected that if you want to learn something new, you need to first research the topic.
[deleted]
You can by insure for you as a driver and for your car (and in many jurisdictions have to) — this limits your responsibility. You can spend $10000 on AWS and still make a mistake and pay another $10k+.
I don’t get the point that AWS docs are free. I mean, they are free, but the services are not. Do you pay for a visit in a car dealership?
I don’t get the point that AWS docs are free. I mean, they are free, but the services are not. Do you pay for a visit in a car dealership?
> You can by insure for you as a driver and for your car (and in many jurisdictions have to) — this limits your responsibility
Real insurance policies have maximum coverage limits, so, while it reduces your liability by a capped amount, it doesn’t, strictly speaking, limit it; you still face unbounded potential liability beyond your insurance coverage.
Real insurance policies have maximum coverage limits, so, while it reduces your liability by a capped amount, it doesn’t, strictly speaking, limit it; you still face unbounded potential liability beyond your insurance coverage.
what's the biggest unexpected billing of AWS fees reported? I'm thinking 100,000 seems unlikely to be reached very often.
on edit: this is not to say that I believe an insurance for this kind of thing would really work (although maybe that is because I know nothing about the insurance business) just that I don't think the problem with the insurance would be that you could always end up getting billed more than you were insured for.
on edit: this is not to say that I believe an insurance for this kind of thing would really work (although maybe that is because I know nothing about the insurance business) just that I don't think the problem with the insurance would be that you could always end up getting billed more than you were insured for.
> I need to stop blaming others and start owning my mistakes.
There's a big difference between owning up to your mistakes and taking on a bunch of unnecessary risk in situations where you know it's more likely you're going to make mistakes. IE: Learning, testing, etc..
In fact taking on unnecessary, possibly unlimited financial risk in a situation like that is a mistake. Want to own that one?
There's a big difference between owning up to your mistakes and taking on a bunch of unnecessary risk in situations where you know it's more likely you're going to make mistakes. IE: Learning, testing, etc..
In fact taking on unnecessary, possibly unlimited financial risk in a situation like that is a mistake. Want to own that one?
Are you seriously suggesting that someone who wants to use AWS should go and read all the documentation and watch all the official videos available? That would take literally years. It's huge.
It also doesn't solve the problem. If the user makes a mistake they still lose a ton of money.
It's a really interesting problem. It's getting too big and too scary for a new developer to jump in and try AWS services, which means new devs will move on to newer services with better UX. I wonder if this sort of problem is one of the few existential threats AWS could actually face. Its possible it could end up as the Oracle or Salesforce of cloud services, very successful in enterprise but definitely not a good thing to have on your resume if you want to join a startup...
It also doesn't solve the problem. If the user makes a mistake they still lose a ton of money.
It's a really interesting problem. It's getting too big and too scary for a new developer to jump in and try AWS services, which means new devs will move on to newer services with better UX. I wonder if this sort of problem is one of the few existential threats AWS could actually face. Its possible it could end up as the Oracle or Salesforce of cloud services, very successful in enterprise but definitely not a good thing to have on your resume if you want to join a startup...
I would not want to be the competing cloud service, with self-imposed limits of only what people are willing and expecting to pay, going up against Amazon (or any other cloud service of that scale) where they not only have scale already, but are being paid the extra money regularly by people screwing up in small to medium ways (even IF the most massive screwups are 'forgiven' to devs who wouldn't have been able to pay anyway, but have twitter)
I don't believe there will be any new services with better billing UX through competitive pressure. You're basically asking the competing thing to take less money in order to win, as the distinguishing factor. It doesn't make sense as a way for them to win against the larger thing taking more money.
Probably the only practical solution would be arbitrary legislation forbidding the practice, as if it was anti-usury laws. Basically taking the guise of society and saying 'you may not do this, even if people are dumb'. And then you've got a problem (at scale) of people figuring out how to do hit-n-run megabilling and walking away under the legislation, having intentionally taken what they 'mistakenly' did. An 'oops, I accidentally a bitcoin' defense.
I don't believe there will be any new services with better billing UX through competitive pressure. You're basically asking the competing thing to take less money in order to win, as the distinguishing factor. It doesn't make sense as a way for them to win against the larger thing taking more money.
Probably the only practical solution would be arbitrary legislation forbidding the practice, as if it was anti-usury laws. Basically taking the guise of society and saying 'you may not do this, even if people are dumb'. And then you've got a problem (at scale) of people figuring out how to do hit-n-run megabilling and walking away under the legislation, having intentionally taken what they 'mistakenly' did. An 'oops, I accidentally a bitcoin' defense.
1. yes I am, studying years to be a doctor, lawyer, Engineer etc from your point of view is also to much? Should you just start cutting people to figure out how a body works end expect for everything to be ok once you are done? The point being, yes you have to study to learn something, there is no way around it.
2. If money is a concern to you, then I should focus on learning how exactly the billing works and how to monitor correctly. This way you can build a product the right way, not to mention AWS by default has limits on their services set with limits that prevent you from doing something incorrectly. For example you can only make 5000 requests a sec on the API Gateway, you can only have 1000 concurrent lambdas, you can spin only 25 ec2 instances, ecc... (true, not all services have limits like this - but then again, if you want to use one, the first thing you should do is check the pricing page, this is what I do fro every new service that I'm planing to use).
3. AWS is not for developers, it is meant for SysAdmins and DevOps (true that some marketing materials are not clear on this), they should be the one configuring it to allow developers to host their code. If you want a turn key solution, then there are better solutions, like Heroku - incredibly easy to use and understand and have a much simpler billing structure.
With AWS you can do anything you want, AWS provides lego blocks, what do you build with it is up to your imaginations, and for sure it is not meant to be use directly by developers who have no idea how networks, computers, databases, cpu, ram, policies, storage etc works. Developers should focus on coding, and SysAdmins and DevOps should focus on managing the infrastructure.
And if you want to learn AWS because you want to be a SysAdmin, then it is true, that AWS could have a plan for beginners with even smaller default limits, and limits set on everything - this way you could more safely play with what they have. This would be a nice things to have in this case for sure.
But because they don't provide such thing, you need to be the responsible one, and start learning AWS the right way, and not get in gun blazing, and expect all will be ok. My recommendation is to learn one service at the time. If you do this, over the years the acquired knowledge will be gold. Plus the more services you learn the right way the easier it gets.
2. If money is a concern to you, then I should focus on learning how exactly the billing works and how to monitor correctly. This way you can build a product the right way, not to mention AWS by default has limits on their services set with limits that prevent you from doing something incorrectly. For example you can only make 5000 requests a sec on the API Gateway, you can only have 1000 concurrent lambdas, you can spin only 25 ec2 instances, ecc... (true, not all services have limits like this - but then again, if you want to use one, the first thing you should do is check the pricing page, this is what I do fro every new service that I'm planing to use).
3. AWS is not for developers, it is meant for SysAdmins and DevOps (true that some marketing materials are not clear on this), they should be the one configuring it to allow developers to host their code. If you want a turn key solution, then there are better solutions, like Heroku - incredibly easy to use and understand and have a much simpler billing structure.
With AWS you can do anything you want, AWS provides lego blocks, what do you build with it is up to your imaginations, and for sure it is not meant to be use directly by developers who have no idea how networks, computers, databases, cpu, ram, policies, storage etc works. Developers should focus on coding, and SysAdmins and DevOps should focus on managing the infrastructure.
And if you want to learn AWS because you want to be a SysAdmin, then it is true, that AWS could have a plan for beginners with even smaller default limits, and limits set on everything - this way you could more safely play with what they have. This would be a nice things to have in this case for sure.
But because they don't provide such thing, you need to be the responsible one, and start learning AWS the right way, and not get in gun blazing, and expect all will be ok. My recommendation is to learn one service at the time. If you do this, over the years the acquired knowledge will be gold. Plus the more services you learn the right way the easier it gets.
There’s room for a little bit of the personal responsibility argument, but it’s ineffective when you take it way too far.
> studying years to be a doctor [...] Should you just start cutting people
Spoken like someone who’s never used AWS or been a doctor. Your analogy is horribly, badly flawed. Doctors do start cutting people, in the US nearly all med students dissect a cadaver at the start of their first year. What you’re suggesting is doing years of pure documentation reading, unguided by other people or a curriculum, before practicing AWS, which would be silly and a waste of time. People learn by practicing, which is why med students dissect cadavers, and which is why AWS offers a “free” platform to learn by practicing, tutorials to guide the learner, and advertising to attract learners.
> If money is a concern
Maybe this is why AWS offers a “Free Tier”? https://aws.amazon.com/free/
> AWS is not for developers
That’s not what AWS says https://aws.amazon.com/developer
> studying years to be a doctor [...] Should you just start cutting people
Spoken like someone who’s never used AWS or been a doctor. Your analogy is horribly, badly flawed. Doctors do start cutting people, in the US nearly all med students dissect a cadaver at the start of their first year. What you’re suggesting is doing years of pure documentation reading, unguided by other people or a curriculum, before practicing AWS, which would be silly and a waste of time. People learn by practicing, which is why med students dissect cadavers, and which is why AWS offers a “free” platform to learn by practicing, tutorials to guide the learner, and advertising to attract learners.
> If money is a concern
Maybe this is why AWS offers a “Free Tier”? https://aws.amazon.com/free/
> AWS is not for developers
That’s not what AWS says https://aws.amazon.com/developer
> Maybe this is why AWS offers a “Free Tier”? https://aws.amazon.com/free/
People in this discussion are complaining that the free tire is misleading, plus not all services are covered. And it is true that if you turn on a bunch of server and you don't pay for a year, but forget about them, you will be charged the moment the year passes. Not to mention that you will be charged if you use the CPU of the free tire server to much - which probably very few people know about.
> That’s not what AWS says https://aws.amazon.com/developer
And my point is that the marketing of AWS is misleading, they try to convince you that if you don't know anything about computers, but you know code, you will be able to manage AWS. This is very misleading because AWS tries to make you think that AWS is a service like Heroku, simple to use, and there is just one button to push to make it all work. Completely false. I've seen countless AWS accounts that were completely misconfigured by developers who thought that AWS easy to manage. A basic example is the autoscaling of EC2. People will go to the autoscaling section of EC2 "enable it" and be superseded when it dose not work. Where the reality is that you have to do 8 other things to make it work, not to mention the work that needs to be done in the OS itself.
People in this discussion are complaining that the free tire is misleading, plus not all services are covered. And it is true that if you turn on a bunch of server and you don't pay for a year, but forget about them, you will be charged the moment the year passes. Not to mention that you will be charged if you use the CPU of the free tire server to much - which probably very few people know about.
> That’s not what AWS says https://aws.amazon.com/developer
And my point is that the marketing of AWS is misleading, they try to convince you that if you don't know anything about computers, but you know code, you will be able to manage AWS. This is very misleading because AWS tries to make you think that AWS is a service like Heroku, simple to use, and there is just one button to push to make it all work. Completely false. I've seen countless AWS accounts that were completely misconfigured by developers who thought that AWS easy to manage. A basic example is the autoscaling of EC2. People will go to the autoscaling section of EC2 "enable it" and be superseded when it dose not work. Where the reality is that you have to do 8 other things to make it work, not to mention the work that needs to be done in the OS itself.
What do you think about this page: https://aws.amazon.com/getting-started/?
> Select a learning path for step-by-step tutorials to get you up and running in less than an hour.
Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?
> Select a learning path for step-by-step tutorials to get you up and running in less than an hour.
Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?
> What do you think about this page: https://aws.amazon.com/getting-started/
It is very dangerous. If you select the full-stack tutorial you get: "Time to Complete 30 minutes". It should say: "30 min to ruin your life" ;)
If you want to really learn AWS, then this page should be used as a reference of how to design a stack. If I were you I would read the tutorials to see which services are needed for a solution, but before doing anything, I would read the docs for each of those services to really understand them, then I would go back to the tutorial and actually do it, and - MOST IMPORTANTLY - I would read the pricing page for each service that you are going to use.
> Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?
100% - when I started working with AWS in 2016 I had a very hard time figuring it out, because I was looking for the simplicity the the marketing team was writing about. I really don't like what the marketing team tries to tell you, because it dose not exist.
Regarding an approach to learn about AWS, I would start with all the serverless services that they have, since the pricing for most of them is ideal for beginners (WARNING - read the pricing page for each since not all have a free staring plane, like S3 and DynamoDB) and for simple weekend projects.
For example, I did build this project a while ago: https://github.com/0x4447/0x4447_product_s3_email, if you scroll down to the pricing section you will see this:
``` All resources deployed via this stack will potentially cost you money. But you'd have to do the following for this to happen:
- Invoke Lambdas over 1,000,000 times a month - Send and receive over 1000 emails a month - Perform over 10,000 Get and Put operations and over 2000 Delete operations in your S3 Bucket - Exceed 100 build minutes on CodeBuild - $1 per active CodePipeline (must run at least once a month to be considered active)
The only payment you'll encounter from Day One is an S3 storage fee for emails and CodePipeline artifacts. ```
So you can have a stack that is actually doing something very useful that costs not even a $1 a month.
It is possible to pay $0 to AWS, but you need to first understand AWS to be able to do it, another trivial example of a tiny project that is useful and cost $0 to run: https://github.com/0x4447/0x4447_product_secure301
The last point would be: don't listen to the marketing material - they are there to sell you AWS, marketing never cares about reality.
I also recommend this website https://awsvideocatalog.com - pick a service and watch all the keynotes AWS has on that service, if you'd spend 1h a day, in 6 months you'll know more about AWS then anyone else complaining here.
It is very dangerous. If you select the full-stack tutorial you get: "Time to Complete 30 minutes". It should say: "30 min to ruin your life" ;)
If you want to really learn AWS, then this page should be used as a reference of how to design a stack. If I were you I would read the tutorials to see which services are needed for a solution, but before doing anything, I would read the docs for each of those services to really understand them, then I would go back to the tutorial and actually do it, and - MOST IMPORTANTLY - I would read the pricing page for each service that you are going to use.
> Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?
100% - when I started working with AWS in 2016 I had a very hard time figuring it out, because I was looking for the simplicity the the marketing team was writing about. I really don't like what the marketing team tries to tell you, because it dose not exist.
Regarding an approach to learn about AWS, I would start with all the serverless services that they have, since the pricing for most of them is ideal for beginners (WARNING - read the pricing page for each since not all have a free staring plane, like S3 and DynamoDB) and for simple weekend projects.
For example, I did build this project a while ago: https://github.com/0x4447/0x4447_product_s3_email, if you scroll down to the pricing section you will see this:
``` All resources deployed via this stack will potentially cost you money. But you'd have to do the following for this to happen:
- Invoke Lambdas over 1,000,000 times a month - Send and receive over 1000 emails a month - Perform over 10,000 Get and Put operations and over 2000 Delete operations in your S3 Bucket - Exceed 100 build minutes on CodeBuild - $1 per active CodePipeline (must run at least once a month to be considered active)
The only payment you'll encounter from Day One is an S3 storage fee for emails and CodePipeline artifacts. ```
So you can have a stack that is actually doing something very useful that costs not even a $1 a month.
It is possible to pay $0 to AWS, but you need to first understand AWS to be able to do it, another trivial example of a tiny project that is useful and cost $0 to run: https://github.com/0x4447/0x4447_product_secure301
The last point would be: don't listen to the marketing material - they are there to sell you AWS, marketing never cares about reality.
I also recommend this website https://awsvideocatalog.com - pick a service and watch all the keynotes AWS has on that service, if you'd spend 1h a day, in 6 months you'll know more about AWS then anyone else complaining here.
Agree with all of this. I think the reason you're getting so much pushback is that you started this conversation with "not sure where the problem is". Clearly you do see the problem - AWS encourages beginners to get going as quickly as possible with these tutorials and the free tier, but then makes it difficult for those users to avoid unexpected charges. They should either stop encouraging beginners, or start offering easy ways to protect yourself.
That statement was related to the sentiment that there is no way to protect yourself or a team of people from AWS pricing, people were implying that there is no tool to help you limit or track expenses, which is not true, since there are plenty of tools to do that.
Anyway, life gose on, and it was overall a good chat :)
Anyway, life gose on, and it was overall a good chat :)
This really is not true. Spending limits are not supported by all major cloud providers.
It seems obvious that AWS simply doesn't want you as a customer if you need to implement spending limits. We've all talked about having customers that were too much trouble for the value they bring. Customers with hard spending limits are those customers for Amazon.
>Don't walk down the same path as IBM's mainframes, where no student anywhere ever has been able to learn on their own
WHAT?
https://www.ibm.com/it-infrastructure/z/education/master-the...
http://wotho.ethz.ch/tk4-/
https://sdl-hercules-390.github.io/html/
https://mainframe.informatik.uni-leipzig.de/
>There's a difference between "unable" and "unwilling".
True words, look at the Links.
WHAT?
https://www.ibm.com/it-infrastructure/z/education/master-the...
http://wotho.ethz.ch/tk4-/
https://sdl-hercules-390.github.io/html/
https://mainframe.informatik.uni-leipzig.de/
>There's a difference between "unable" and "unwilling".
True words, look at the Links.
So items 1 and 4 are courses which you need to sign up for and get everything including instruction. That fails to qualify as "on their own".
Item 2 is documentation. This is good! But without access to the system to try it out, how self-teachable is it?
Item 3 is the one counter point. With a third party emulator though, how comprehensive/accurate is it? Would you hire someone to work on a mainframe who'd never used a real system before?
Compare that to e.g. Java. Sure there are plenty of college courses, some even Oracle affiliated, but I can also just download a Java compiler and start plugging away. .NET is there with Visual Studio Express. Oracle cloud, for everything else wrong with Oracle, has a free tier with no time limit. Microcontrollers for embedded development can be had for €20. You can install Linux or a Linux VM on your own computer. Even stuff that falls into the more enterprisey camp like MS SQL or Oracle DB has free personal licenses.
Item 2 is documentation. This is good! But without access to the system to try it out, how self-teachable is it?
Item 3 is the one counter point. With a third party emulator though, how comprehensive/accurate is it? Would you hire someone to work on a mainframe who'd never used a real system before?
Compare that to e.g. Java. Sure there are plenty of college courses, some even Oracle affiliated, but I can also just download a Java compiler and start plugging away. .NET is there with Visual Studio Express. Oracle cloud, for everything else wrong with Oracle, has a free tier with no time limit. Microcontrollers for embedded development can be had for €20. You can install Linux or a Linux VM on your own computer. Even stuff that falls into the more enterprisey camp like MS SQL or Oracle DB has free personal licenses.
>So items 1 and 4 are courses which you need to sign up for and get everything including instruction. That fails to qualify as "on their own".
Attending a course on your Own.
>Item 2
No it's a MVS distribution (a full blown Operating System) with all the compilers 99% the same as a "modern" z/os
>Item 3
That's the S/390 emulator.
>Would you hire someone to work on a mainframe who'd never used a real system before?
Most Mainframe-Programmer where never near the Hardware anyway...so YES i would do that absolutely (you don't do Hardware near programming in Mainframes, an emulator is perfectly fine), same as a Dev who programs in a VirtualBox, no difference.
If you want to learn how to use it, and MUCH more:
https://www.youtube.com/channel/UCR1ajTWGiUtiAv8X-hpBY7w
https://github.com/moshix
>Compare that to e.g. Java.
You can do exactly the same with MVS, all the compilers (and more as z/os brings with) are in it.
Attending a course on your Own.
>Item 2
No it's a MVS distribution (a full blown Operating System) with all the compilers 99% the same as a "modern" z/os
>Item 3
That's the S/390 emulator.
>Would you hire someone to work on a mainframe who'd never used a real system before?
Most Mainframe-Programmer where never near the Hardware anyway...so YES i would do that absolutely (you don't do Hardware near programming in Mainframes, an emulator is perfectly fine), same as a Dev who programs in a VirtualBox, no difference.
If you want to learn how to use it, and MUCH more:
https://www.youtube.com/channel/UCR1ajTWGiUtiAv8X-hpBY7w
https://github.com/moshix
>Compare that to e.g. Java.
You can do exactly the same with MVS, all the compilers (and more as z/os brings with) are in it.
My business uses off-cloud backups, devops and prepaid credit cards to limit our risk exposure.
Forgive me if I am out of line, but as I understand it, your company would still be legally liable for the debt. So even if blew past your prepaid card limit, your company would still have an obligation to pay using an alternative method.
Not a lawyer, etc, and don't even know what jurisdiction you are in or what providers you use, so this is just a general caution that your strategy may not generalize to other companies.
Not a lawyer, etc, and don't even know what jurisdiction you are in or what providers you use, so this is just a general caution that your strategy may not generalize to other companies.
I assume you're not also using AWS? Otherwise even with a prepaid card, on AWS you are granting them blank checks via unlimited invoicing abilities.
Nah, I think not. My business needs to run. Would be very painful if an engineer put in a hard shutdown at $X and then left the company only for me to find out after all my services are shutdown when we've grown past $X.
This is a hard no from me.
How is AWS going to even know what to shutdown/remove? What if it's storage causing my bill to overextend?
Yeah, not just no, but hell no.
This is a hard no from me.
How is AWS going to even know what to shutdown/remove? What if it's storage causing my bill to overextend?
Yeah, not just no, but hell no.
I’m not sure I’d want a running business haemorrhaging $10,000/minute or whatever is possible for cloud compute.
My business is looking at cloud compute at the moment and we have absolutely no idea how to do it safely. In fact sagemaker is exactly a product we have looked at and discounted since we cannot be sure we can do it safely without getting unexpected megacharges.
We had an incident recently where a cellular router was set up to send a text message on a GPIO changing. The old firmware didn’t have pull-ups on the input. We told our end customer to update the firmware. They didn’t and also didn’t hook up the input and left it floating. We got a £20,000 bill for text messages as a result and a soured relationship with the customer.
My business is looking at cloud compute at the moment and we have absolutely no idea how to do it safely. In fact sagemaker is exactly a product we have looked at and discounted since we cannot be sure we can do it safely without getting unexpected megacharges.
We had an incident recently where a cellular router was set up to send a text message on a GPIO changing. The old firmware didn’t have pull-ups on the input. We told our end customer to update the firmware. They didn’t and also didn’t hook up the input and left it floating. We got a £20,000 bill for text messages as a result and a soured relationship with the customer.
What if that engineer fat fingered and reserved a million GPU instances instead though?
It would be nice if cloud providers were better at surfacing potentially company killing things via their dashboards and nag emails.
It would be nice if cloud providers were better at surfacing potentially company killing things via their dashboards and nag emails.
Well, they’d have to first contact support and get the instance limits lifted… it would be one hell of a fat finger.
Not everything in AWS is a VM with a limit.
For example, their CDN has no "TB per month limit". If your site gets popular and is 100% cacheable, you could be getting a million dollar bill.
I hear similar horror stories about the various distributed databases in several clouds. They're all crazy expensive.
For example, their CDN has no "TB per month limit". If your site gets popular and is 100% cacheable, you could be getting a million dollar bill.
I hear similar horror stories about the various distributed databases in several clouds. They're all crazy expensive.
What a lie
AWS offers an AWS educate account already - go here to learn more
https://aws.amazon.com/education/awseducate/aws-educate-faqs...
If you are a startup go through AWS activate - 100,000 to play with.
GCP etc al do not offer this either. This is not done crime- AWS pioneered this free tier model. I used Dell and you paid for everything there.
AWS offers an AWS educate account already - go here to learn more
https://aws.amazon.com/education/awseducate/aws-educate-faqs...
If you are a startup go through AWS activate - 100,000 to play with.
GCP etc al do not offer this either. This is not done crime- AWS pioneered this free tier model. I used Dell and you paid for everything there.
Great: Now do the same for everyone. Let everyone sign up for an AWS educate account with limits they choose.
AWS employs cost obfuscation by design otherwise the default view when you open the console would show you all of your current active services. Not only is that not the case, a single screen to show you all of your current active services doesn't exist. You need to take a deep dive into cost explorer (assuming you have access in corporate land) and try to decipher in what that all means.
They definitely need a senior executive to stand up and say, "The Customer wants us to be transparent in billing, fix that now."
Then they need to start a team dedicated to finding a good way to let customers halt spending at a given limit with minimal impact on their operations.
They already win on UX (okay, okay, it's an opinion ffs), but unlimited liability makes a lot of people very uncomfortable. Those two actions would go a long ways towards demonstrating good faith in that area.
If it would cost too much, maybe they could present it as an easy way to cut expenses at the same time that they introduce a small price increase. This is a common and long-standing complaint/feature request.
Then they need to start a team dedicated to finding a good way to let customers halt spending at a given limit with minimal impact on their operations.
They already win on UX (okay, okay, it's an opinion ffs), but unlimited liability makes a lot of people very uncomfortable. Those two actions would go a long ways towards demonstrating good faith in that area.
If it would cost too much, maybe they could present it as an easy way to cut expenses at the same time that they introduce a small price increase. This is a common and long-standing complaint/feature request.
I’m not trying to be snarky here, would like an honest opinion: how do they win on UX? Are there specific things you like there?
I’ve used all three major clouds in production now and I dread using the AWS console. Or really any part of it, over Azure or GCP.
I’ve always thought of them as purely winning the “nobody got fired for…” mindshare thing despite having a thoroughly mediocre product.
I’ve used all three major clouds in production now and I dread using the AWS console. Or really any part of it, over Azure or GCP.
I’ve always thought of them as purely winning the “nobody got fired for…” mindshare thing despite having a thoroughly mediocre product.
It either works great or barely depending on the service you're using - some AWS teams have dedicated dashboard teams, eg 'ec2 dashboard team' which solely focus on the dashboard experience, while others touch it as an afterthought.
I'm pretty sure something along the lines of this^ was posted on HN by a former AWS employee but I can't find it now.
I'm pretty sure something along the lines of this^ was posted on HN by a former AWS employee but I can't find it now.
Their web console UIs vary widely by service. IMO, the UX for most of their simpler services, specifically SQS, S3, Lambda, DynamoDB, are really easy to use and work nicely. If you want to start with a Docker image, send it to AWS, and have it get spun up and attached to a DNS, well that's a huge complex mess to get set up.
Unless you use something like Terraform or Pulumi, in which case it’s almost a one liner.
The AWS default view feels more crisp for me, compared to the GCP one which feels very "floaty".
Their UI toolkit for that at least is on point.
Their UI toolkit for that at least is on point.
"They already win on UX"
As someone that has tried and failed to get some small personal sites running on AWS a couple times, I'm going to have to tag this snippet with [citation needed].
As someone that has tried and failed to get some small personal sites running on AWS a couple times, I'm going to have to tag this snippet with [citation needed].
agreed especially when compared to say Heroku or Digital ocean. I see many new comers struggling to deploy a small website. It's overwhelming. I understand that Heroku exists for such users and they are using AWS under the hood, but is there any service which takes AWS cloud APIs and simplifies it with a leaner UX?
I think that's basically what Lightsail is supposed to be. I haven't used it myself, but from glancing at it they're pretty clearly targeting DO/Linode/Vultr etc.
AWS have a multitude of services that do this already. I’d recommend to go through their services one by one and learn what they offer.
> agreed especially when compared to say Heroku or Digital ocean.
Funnily enough, I've never ever been able to understand what Heroku is, or how to deploy anything useful on it :) But I'm old, and I've always found it easier to tinker with nginx configs.
Funnily enough, I've never ever been able to understand what Heroku is, or how to deploy anything useful on it :) But I'm old, and I've always found it easier to tinker with nginx configs.
Heroku is very simple - you just create an app and "push" code to a repository tied to the app. You need to write code of course, but they auto-detect the language and prepare the environment or platform for you (so platform as a service). They tried to appease the geeks by making everything powered by CLI - but otherwise its just point and click for deploying apps.
Tinkering with nginx is good, but what if you want to add a database, a cache, a logging service? What if you want to automatically build code when you push to github? What if you want to easy clone multiple environments (for staging, prod etc) without never doing an SSH into server? What if you want to have versioned releases that you can easily rollback to a specified version or push a branch into one environment and another branch into another environment? And you get a workable subdomain for every environment you build with SSL enabled. All of that can be scaled down and up with literally few clicks.
Tinkering with nginx is good, but what if you want to add a database, a cache, a logging service? What if you want to automatically build code when you push to github? What if you want to easy clone multiple environments (for staging, prod etc) without never doing an SSH into server? What if you want to have versioned releases that you can easily rollback to a specified version or push a branch into one environment and another branch into another environment? And you get a workable subdomain for every environment you build with SSL enabled. All of that can be scaled down and up with literally few clicks.
> but what if you want to add a database, a cache, a logging service?
I... just add it. How do I add it in heroku where everything is "just create an app and push it" and everything is separate "dynos" (whatever that is), each with different pricing, storage, bandwidth etc.
I... just add it. How do I add it in heroku where everything is "just create an app and push it" and everything is separate "dynos" (whatever that is), each with different pricing, storage, bandwidth etc.
that is true. everything is a separate dyno - means separate machine/vm to which you need to connect to. You get free tier for almost all databases and you only need to know how to connect to it, not tinkering with the storage, tune the parameters, scheduled backups, automatic scalability etc - that's done for you. it will become expensive once you go past free tier or the cheap tiers though.
> not tinkering with the storage, tune the parameters, scheduled backups, automatic scalability etc - that's done for you.
Thing is: if we're looking for free, that may make sense. you still need tinkering though: you need to figure out what the whole dyno thing is about and how to connect from one dyno to another, and so on.
However, if we expand this a tiny bit to cheap and/or free, then simply running the lowest-tier server at Digital Ocean will be a better proposition. And installing a database these days is basically the same thing: run a command to install, all you need is to connect to it. No tinkering. And it will still probably scale significantly better than a free tier at Heroku :)
Once again, I'm biased and old, and can afford to spend money on side projects. I know that sometimes even $15 a month is out of reach (oh, I've been there). But yeah, this sorta prevents me from ever understanding Heroku :)
Thing is: if we're looking for free, that may make sense. you still need tinkering though: you need to figure out what the whole dyno thing is about and how to connect from one dyno to another, and so on.
However, if we expand this a tiny bit to cheap and/or free, then simply running the lowest-tier server at Digital Ocean will be a better proposition. And installing a database these days is basically the same thing: run a command to install, all you need is to connect to it. No tinkering. And it will still probably scale significantly better than a free tier at Heroku :)
Once again, I'm biased and old, and can afford to spend money on side projects. I know that sometimes even $15 a month is out of reach (oh, I've been there). But yeah, this sorta prevents me from ever understanding Heroku :)
I think there's some verticals - players that simplify/aggregate SES, or S3, or whatever - but none that handle being a layer on top of most/all of AWS.
It’s not you. It’s pretty rough for new users.
Oh man, I have long been joking about the AWS UI. Like how it will gladly walk you through all the steps for launching a server and only at the very last step says “oh you don’t have permission to do that lol, get permission and start over”.
I compared[1] it to a bartender who walks you through an entire sale and only at the last second rejects your purchase for being underage (instead of denying you the at the initial request for something alcoholic).
Of course, that’s one of the minor things in the grand scheme.
[1] http://blog.tyrannyofthemouse.com/2016/02/some-of-my-geeky-t...
I compared[1] it to a bartender who walks you through an entire sale and only at the last second rejects your purchase for being underage (instead of denying you the at the initial request for something alcoholic).
Of course, that’s one of the minor things in the grand scheme.
[1] http://blog.tyrannyofthemouse.com/2016/02/some-of-my-geeky-t...
From my admittedly limited experience with GCP and Azure (and this is obviously subjective), the UX in the most successful competition is, at best, no better than AWS. It isn't that AWS has good UX, just that all the cloud providers have bad UX.
Have to agree with the others. For example, while setting up ELB it's possible to select at least one option (un-checking the Public IP box) that causes the setup to just fail with a nonsensical error message. Turns out ELB requires a public IP to communicate with the nodes. That's just the most glaring one off the top of my head.
I also remember trying to set up SFTP whenever it was first released. It was literally impossible to do what they were advertising (S3-backed SFTP with jailed homes for each user) without writing a manual json config (I never got it to work). I had built my own solution for this exact thing on EC2, using a bash script more or less, and thought the hosted option would be less of a maintenance burden. Needless to say I quickly gave up.
I also remember trying to set up SFTP whenever it was first released. It was literally impossible to do what they were advertising (S3-backed SFTP with jailed homes for each user) without writing a manual json config (I never got it to work). I had built my own solution for this exact thing on EC2, using a bash script more or less, and thought the hosted option would be less of a maintenance burden. Needless to say I quickly gave up.
cdk is promising for things like this, way safer and easier than rolling your own scripts with bash or python.
Even CDK sucks though because if you’re still kinda new to it all, you want to login and make sure that it’s all hooked together correctly. And you’re back using their shitty UI.
Why you can’t look at the load balancer, look at the listener and then show what’s in the target groups is beyond me.
Why you can’t look at the load balancer, look at the listener and then show what’s in the target groups is beyond me.
I have a different take on it. I started out doing everything through the console, then learned the cli and boto3, and very recently CDK.
CDK is another tool that build on the cli and boto3 concepts, and also manages the orchestration and dependencies.
Having to go back to the console isn't a fault of CDK. Learning the right tool to use for a situation is part of the learning curve. I go back to the console all the time to look something up quick or to understand what config options are available. Or I repeat the same steps in the console enough times that I get bored with it and automate
Edit: also I tried and have given up on CloudFormation more than once. CDK is like a wrapper layer around it, and has been pleasant to use.
CDK is another tool that build on the cli and boto3 concepts, and also manages the orchestration and dependencies.
Having to go back to the console isn't a fault of CDK. Learning the right tool to use for a situation is part of the learning curve. I go back to the console all the time to look something up quick or to understand what config options are available. Or I repeat the same steps in the console enough times that I get bored with it and automate
Edit: also I tried and have given up on CloudFormation more than once. CDK is like a wrapper layer around it, and has been pleasant to use.
How are people not building in Terraform for an easy ‘destroy’ at the end?
I know it’s rhetorical and a lesson learned myself, but yeah… I would expect folks to learn to use this tool to help manage costs this way.
I know it’s rhetorical and a lesson learned myself, but yeah… I would expect folks to learn to use this tool to help manage costs this way.
Typically Terraform takes longer to get something working than mindlessly clicking through the console. In my experience those mindless clickthrough things end up sticking around for years even when they weren't intended to.
This is why you have separate development and production accounts: a development account where you mindlessly click through so that you can learn through the UI what's available and how it works; cleaned up on a regular basis by something like aws-nuke, and a production account where you have the discipline to only create resources through Terraform / CloudFormation etc.
I am pretty senior and can thus sometimes afford to do a new thing and try doing it the right way, at the same time. Not even always. Many people just take one learning curve at a time...
terraform 'destroy' isn't infallible. There are certain resources that trigger the creation of other resources (for example, lambda functions will create cloudwatch log groups, and dynamodb tables create cloudwatch alarms) and when terraform destroys the resource, it doesn't necessarily clean up all the associated resources.
Terraform has it's shortcomings for sure, but I don't think it's reasonable to expect Terraform to go out and clean up second-order effects of it's resources.
I'm not doubting that the situations you describe are true, but abandoning resources like that is an AWS-lifecycle problem, not really a Terraform one.
I'm not doubting that the situations you describe are true, but abandoning resources like that is an AWS-lifecycle problem, not really a Terraform one.
Sure. My point is just that `terraform destroy` doesn't necessarily solve the problem at hand. And you could still end up continue paying for those second-order effects after running a terraform destroy.
Unless you understand IAM, the scoping of individual users from the sftp service can be a bit confusing. What you need is a scope down policy:
https://docs.aws.amazon.com/transfer/latest/userguide/scope-...
The more annoying part is that the service only supports public/private key logins. If you want user/pass you have to write a lambda. The lambda is pretty simple though, it checks the credentials (so it can hit any backend you like), and if they pass it returns a 200 with a json doc of role (which is just sftp assume role), policy (the scope down from above), and home dir.
https://aws.amazon.com/blogs/storage/enable-password-authent...
This touches on a larger issue with AWS though. It's that they are trending to leave out functionality and point to lambda as the solution. On one hand, I get it. The lambda solution is infinitely more flexible, but what if I just wanted an sftp for a couple users that uses user/pass?
To your final point, it is so much less maintenance. There is no server to manage, and since I want all the data in s3 anyway, it's already there. This solution replaced a server with chroots, EBS, scripts to move data to s3, etc...
https://docs.aws.amazon.com/transfer/latest/userguide/scope-...
The more annoying part is that the service only supports public/private key logins. If you want user/pass you have to write a lambda. The lambda is pretty simple though, it checks the credentials (so it can hit any backend you like), and if they pass it returns a 200 with a json doc of role (which is just sftp assume role), policy (the scope down from above), and home dir.
https://aws.amazon.com/blogs/storage/enable-password-authent...
This touches on a larger issue with AWS though. It's that they are trending to leave out functionality and point to lambda as the solution. On one hand, I get it. The lambda solution is infinitely more flexible, but what if I just wanted an sftp for a couple users that uses user/pass?
To your final point, it is so much less maintenance. There is no server to manage, and since I want all the data in s3 anyway, it's already there. This solution replaced a server with chroots, EBS, scripts to move data to s3, etc...
This gives the other vendors a wedge in.
I don't understand why so many people want to use AWS for such small budgets. At that scale wouldn't it be easier to just build everything out of a few VMs at a cheap service?
AWS is awesome when you have a large number of resources, that are created programmatically and reproducibly, with redundancy and duplicate environments.
The budgeting tools are really amazing at letting you categorize your costs and create appropriate alerts. The permissions system lets you define very specific roles.
Its complex. But if your system is complex, it gives you the tools to keep track of it all. If you have a <=$5000/month budget, it is probably too small to make sense in AWS. You can probably run your system on a couple servers.
AWS is awesome when you have a large number of resources, that are created programmatically and reproducibly, with redundancy and duplicate environments.
The budgeting tools are really amazing at letting you categorize your costs and create appropriate alerts. The permissions system lets you define very specific roles.
Its complex. But if your system is complex, it gives you the tools to keep track of it all. If you have a <=$5000/month budget, it is probably too small to make sense in AWS. You can probably run your system on a couple servers.
Linux server administration requires a certain amount of know-how and determination. I can't tell you how many times I had to rebuild a DigitalOcean server because I messed something up and wanted to start with a clean slate.
A lot of hackathons, workshops and courses ask you to use AWS these days. Whether it's to run machine learning instances, win a sponsor prize or learn how to use Lambda, students are often encouraged to learn one of the major cloud providers.
Also it's a resume boost. It's another buzzword you can add to your resume.
A lot of hackathons, workshops and courses ask you to use AWS these days. Whether it's to run machine learning instances, win a sponsor prize or learn how to use Lambda, students are often encouraged to learn one of the major cloud providers.
Also it's a resume boost. It's another buzzword you can add to your resume.
> At that scale wouldn't it be easier to just build everything out of a few VMs at a cheap service?
I have a handful of AWS Lambda functions with a DynamoDB backend serving hundreds of clients, my bill for the month of April was $0.01.
No, a VM wouldn't cut it.
But you are right that there are certain slots where AWS doesn't make sense: There's one in the lower middle range where you can save a bunch of money by using a VM or two with your own DB servers. And there's the one where you're so big it might actually be worth it to implement the whole stack yourself.
I have a handful of AWS Lambda functions with a DynamoDB backend serving hundreds of clients, my bill for the month of April was $0.01.
No, a VM wouldn't cut it.
But you are right that there are certain slots where AWS doesn't make sense: There's one in the lower middle range where you can save a bunch of money by using a VM or two with your own DB servers. And there's the one where you're so big it might actually be worth it to implement the whole stack yourself.
> I have a handful of AWS Lambda functions with a DynamoDB backend serving hundreds of clients, my bill for the month of April was $0.01.
What kind of thing do they serve? Somewhere I could read more about this kind of project?
What kind of thing do they serve? Somewhere I could read more about this kind of project?
This is considered a cloud native pattern.
https://docs.aws.amazon.com/apigateway/latest/developerguide...
https://docs.aws.amazon.com/apigateway/latest/developerguide...
It's a common backend for chat bots (Discord, Matrix, IRC, Telegram).
Basically it takes different inputs and commands and uses different APIs to fetch data based on the input efficiently without the need for web scraping. DynamoDB is used mostly as a cache for common queries so I don't go over API quotas.
Most of the bots are made by me, but with AWS API Gateway I can easily generate API keys for anyone else and keep track of their usage.
Basically it takes different inputs and commands and uses different APIs to fetch data based on the input efficiently without the need for web scraping. DynamoDB is used mostly as a cache for common queries so I don't go over API quotas.
Most of the bots are made by me, but with AWS API Gateway I can easily generate API keys for anyone else and keep track of their usage.
> At that scale wouldn't it be easier to just build everything out of a few VMs at a cheap service?
No. I looked into AWS vs Azure vs Cloudflare vs Digital Ocean for something I'd like to build this year and (for my thing) Cloudflare (Workers) was the best balance of cost, scalability, and maintainability.
> I don't understand why so many people want to use AWS for such small budgets.
Serverless (I hate the term) makes a lot of sense for small budgets and projects. You're investing very little, so the downside of (severe) vendor lock-in is somewhat low. The upside is HA infrastructure, horizontal scalability, zero maintenance, ignorable monitoring (at least initially), consumption based costs, etc..
The biggest problem with all of them, except Digital Ocean, is that it's difficult to "own" your data in the sense that you can download a copy of a DB and keep using it somewhere else. IIRC Digital Ocean has a fairly nice managed PostgreSQL offering, but it still scales similar to a traditional DB (ie: not automatically).
The next biggest problem with AWS and Azure is that you can't figure out what anything costs, at least not easily. For example, I know for a page I'd want to serve via Cloudflare Workers, I could do 1 hit = 1 point read from Azure CosmosDB, but I couldn't figure out if the pricing includes egress. Just look at the pricing page for CosmosDB [1]. It's ridiculously complicated and that's one service on Azure.
1. https://azure.microsoft.com/en-us/pricing/details/cosmos-db/...
No. I looked into AWS vs Azure vs Cloudflare vs Digital Ocean for something I'd like to build this year and (for my thing) Cloudflare (Workers) was the best balance of cost, scalability, and maintainability.
> I don't understand why so many people want to use AWS for such small budgets.
Serverless (I hate the term) makes a lot of sense for small budgets and projects. You're investing very little, so the downside of (severe) vendor lock-in is somewhat low. The upside is HA infrastructure, horizontal scalability, zero maintenance, ignorable monitoring (at least initially), consumption based costs, etc..
The biggest problem with all of them, except Digital Ocean, is that it's difficult to "own" your data in the sense that you can download a copy of a DB and keep using it somewhere else. IIRC Digital Ocean has a fairly nice managed PostgreSQL offering, but it still scales similar to a traditional DB (ie: not automatically).
The next biggest problem with AWS and Azure is that you can't figure out what anything costs, at least not easily. For example, I know for a page I'd want to serve via Cloudflare Workers, I could do 1 hit = 1 point read from Azure CosmosDB, but I couldn't figure out if the pricing includes egress. Just look at the pricing page for CosmosDB [1]. It's ridiculously complicated and that's one service on Azure.
1. https://azure.microsoft.com/en-us/pricing/details/cosmos-db/...
Disclosure: I'm the Co-Founder of Vantage.
This is exactly what we do with Vantage: http://vantage.sh/
We give point-in-time run-rates of all active resources based off of the region and resource/service configuration.
In addition we try to simplify people's understanding of where their costs are coming from. If anyone needs help with this, they can personally reach out to me at [email protected]
This is exactly what we do with Vantage: http://vantage.sh/
We give point-in-time run-rates of all active resources based off of the region and resource/service configuration.
In addition we try to simplify people's understanding of where their costs are coming from. If anyone needs help with this, they can personally reach out to me at [email protected]
With all due respect, I think the issue here is that your company should not need to exist. Amazon should provide this feature as part of the UX.
Yea they should, but if it doesn't, are you gonna keep waiting or do something about it?
I would be more worried that if you do something about it, Amazon is gonna do something about you.
Starting with obnoxious API changes, followed by ToS changes. Hopefully that's as far as it has to go.
Starting with obnoxious API changes, followed by ToS changes. Hopefully that's as far as it has to go.
Agreed. Nothing new about this request.
It's like hiring a plane to fly over your house repeatedly to tell you whether or not it's on fire. There has to be a better way.
It's like hiring a plane to fly over your house repeatedly to tell you whether or not it's on fire. There has to be a better way.
Same respect, nice tool, but it is sad it is needed. But I guess it's similar to lawyers, police, army, I don't like the need for them, it, but good they exist.
Azure does this a little better, but best would be to see a breakdown on the invoice with links directly back to the resource.
Maybe there are discounts or other processing that makes this hard.
Or, less charitably, this would lead to people optimising their costs a lot better and canceling unused services much sooner.
Maybe there are discounts or other processing that makes this hard.
Or, less charitably, this would lead to people optimising their costs a lot better and canceling unused services much sooner.
Azure groups most things into resource groups which greatly simplifies things.
Yes. Also split out into sub categories. How much of an ec2 spend was on bandwidth and how much on compute for example.
Enterprise Agreement accounts aren’t billed on demand, so there’s little use for that in accounts spending a lot of money.
Why the f do I have to use a service like billgist to get a breakdown of my services?
AWS billing IS complicated but could be made so much simpler...
AWS billing IS complicated but could be made so much simpler...
I'll bite...
Because, if you are a company that uses AWS at scale (deploying literally thousands of resources at a time), you care more about meeting demand and getting resources to spec, than you do about the cost of an individual elastic IP...
The price of each individual resource isn't something that you want to see on every screen you touch. It literally clutters the console with information that you couldn't give a shit about when your company is making millions (or billions) on the services you provide. You care more about your service's reliability/scalability/uptime/etc than anything else. This is priority numero uno.
If and when cost analysis becomes priority, you look to see where you may be overprovisioning resources - hence, the billing console.
But for the larger players on AWS (the multi-billion dollar ones that AWS cares more about making happy than you or I), an extra $100k in AWS expenses in a year isn't a worry, it's a write-off.
Because, if you are a company that uses AWS at scale (deploying literally thousands of resources at a time), you care more about meeting demand and getting resources to spec, than you do about the cost of an individual elastic IP...
The price of each individual resource isn't something that you want to see on every screen you touch. It literally clutters the console with information that you couldn't give a shit about when your company is making millions (or billions) on the services you provide. You care more about your service's reliability/scalability/uptime/etc than anything else. This is priority numero uno.
If and when cost analysis becomes priority, you look to see where you may be overprovisioning resources - hence, the billing console.
But for the larger players on AWS (the multi-billion dollar ones that AWS cares more about making happy than you or I), an extra $100k in AWS expenses in a year isn't a worry, it's a write-off.
You also get much better negotiated rates for everything when you are big.
I agree, having set up serverless and lambda for an api/app that was to be used a few times per day, the billing made no sense at all, it would increase even though the services were used less and was difficult to find what the costs were or whether they could be reduced, eventually I had to shut it down and create a non-lambda solution somewhere else because I had little control of the cost.
If I go to billing I can see an itemized breakdown of every service I'm using. What is wrong with that?
Students and self-learners want to experiment with doing complicated things on AWS without getting a surprise $3,000 bill because their script didn't shut down those instances like they thought it did. They want a hard fail-safe to protect them from surprise bills. And they want a solution that's easier and more reliable than checking the AWS itemized breakdown twice a day.
I agree with that. That is why there are "Billing Alarms" in big letters on the billing page. But that wasn't the point of the reply: the person above me was complaining that there's no way to tell what services are being billed, which is bunk.
The breakdown isn't granular enough by far. ec2 is a very big bucket to go digging into for costs.
This is what keeps me from tinkering with the free tiers and never even attempt to host my own projects in the cloud.
I want a free tier with a lock on it. I'd very much appreciate advice for what tiers would fit my monthly use if I hit free tier caps, but if I got hit with $500 right now I'd be ruined.
I think my Google compute stuff is safe, but I really don't like having any doubt.
I want a free tier with a lock on it. I'd very much appreciate advice for what tiers would fit my monthly use if I hit free tier caps, but if I got hit with $500 right now I'd be ruined.
I think my Google compute stuff is safe, but I really don't like having any doubt.
Until AWS fixes this the best thing to do is just use their service as little as possible. There are plenty of other cloud providers out there these days which don't employ this hostile practice.
I closed an AWS account for this reason just a few days ago, we hadn't used it for a while but there was no clear way to remove our credit card so it felt like a risk just to have it open, what if a developer logs in to mess around and accidentally flips some switch that smacks us with a charge of a couple grand... unlikely, sure, but the fact that it's even possible is terrible system design. Better to just nuke the account and move on to other cloud providers that don't make it harder for me to sleep at night.
I closed an AWS account for this reason just a few days ago, we hadn't used it for a while but there was no clear way to remove our credit card so it felt like a risk just to have it open, what if a developer logs in to mess around and accidentally flips some switch that smacks us with a charge of a couple grand... unlikely, sure, but the fact that it's even possible is terrible system design. Better to just nuke the account and move on to other cloud providers that don't make it harder for me to sleep at night.
Not disagreeing with you (a view with all the active services would be great) but one of the many benefits of using Terraform is that it allows you to know what you running.
Yep and is a huge SaaS market place now so mid to large size companies can figure out what they are spending money on.
I don’t use them for any projects mostly because I already don’t enjoy having to manage AWS at work all the time. I also don’t want to live/work in a world where AWS is my only option so I try and use smaller hosting providers and services like tarsnap.
I don’t use them for any projects mostly because I already don’t enjoy having to manage AWS at work all the time. I also don’t want to live/work in a world where AWS is my only option so I try and use smaller hosting providers and services like tarsnap.
AWS is literally trying to trick you. With friend like these, who needs enemies?
The Trough of Disillusionment for AWS is going to be Bastille Day levels of ugly.
The Trough of Disillusionment for AWS is going to be Bastille Day levels of ugly.
I cancelled an account 2 years ago because no matter how much I explored cost manager or any regions I used, I can't find why I still get billed a small dollar or so a month.
I still get bills on the 3rd of each month, followed by a "we can't charge your card" (I removed it when they pulled this shit) followed by a few days later you get the "hey we're gonna suspend your account if you don't pay"
Yeah cool, I only closed the account 2 years ago now, go right ahead and suspend/close/do whatever you've gotta do, I stopped caring.
I still get bills on the 3rd of each month, followed by a "we can't charge your card" (I removed it when they pulled this shit) followed by a few days later you get the "hey we're gonna suspend your account if you don't pay"
Yeah cool, I only closed the account 2 years ago now, go right ahead and suspend/close/do whatever you've gotta do, I stopped caring.
The cost explorer is beyond deplorable. It provides either too little information or information that is so fine-grained it becomes useless.
It infuriates me the hoops we need to jump through just to understand how much a single virtual machine costs. Or in cases like yours - to find out what resource a charge was for. Impossible.
Designed to be impossible. FFS. We would spend more money if we were able to manage costs better.
It infuriates me the hoops we need to jump through just to understand how much a single virtual machine costs. Or in cases like yours - to find out what resource a charge was for. Impossible.
Designed to be impossible. FFS. We would spend more money if we were able to manage costs better.
Same! I literally closed an account and created a new one after a few months because I was charged a few bucks a month and I didn't know what I was paying for.
I think it was either related to db backup or some encryption keys but I couldnt find what and where to delete to get rid of this charge.
Nowadays I use aws only for route53 and s3.
I think it was either related to db backup or some encryption keys but I couldnt find what and where to delete to get rid of this charge.
Nowadays I use aws only for route53 and s3.
Same here. I had a monthly billing of a few dollars and couldn't figure out where it's coming from (and given that it's only a few dollars, it wouldn't make sense to spend many hours investigating). In the end I just canceled the card and hoped for the best.
Came in to say I had to do the same thing, close i.e an account to stop being charged.
Yup. Got charged over $800 for "experimenting" with a DynamoDB database and forgetting to delete it afterwards.
Sure, I called customer support and they reversed the charges. But something the nice lady on the other end said as she chuckled: "This happens all the time".
DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there. Wuuuttttt.
Sure, I called customer support and they reversed the charges. But something the nice lady on the other end said as she chuckled: "This happens all the time".
DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there. Wuuuttttt.
Apparently I owe AWS 1 cent for DNS. The problem is I can't login to pay it, so every month I get a email that says "your aws account is going to be suspended" and 30 days later I'm disappointed that they didn't follow through with the threat.
AWS charges me $1 or $2 every month, and I log into my account and click through every page and can't find a single active service or any clue on what it's for. It's marked "EC2 - Other", whatever the f that means
Yeah, that's an elastic IP, check under EC2.
People forget to clean them up after shutting down the instance (probably kept in case you want to keep your IP)
People forget to clean them up after shutting down the instance (probably kept in case you want to keep your IP)
It's crazy that it's so common, but that they wouldn't make that obvious.
Considering the whole IPv4 extinction that's coming down and all.
Considering the whole IPv4 extinction that's coming down and all.
So an elastic IP is just an AWS provided IP that they hand over to you... it's free while your instance is running but if you stop your instance, you get a small bill (it's like $0.84 a month)
I will gladly welcome the eventual IPv4 extinction, but unfortunately I'm pretty confident it won't happen for another 30 years.
I don't have any elastic IPs. The only thing under EC2 is a security group that it won't let me delete, but seems like no resources. My bill this month is $2.88 /shrug
Check volumes, non-root volumes are kept on deletion but should be EBS charged instead.
What does the bill details screen show? I manage an AWS bill and I find it overly detailed. Others mentioned leftover EIPs, but the bill shows these as their own line item.
Not saying it can't happen, but I've never seen an 'Other' listed on the bill details screen. As I said above, it's detailed to a fault.
Not saying it can't happen, but I've never seen an 'Other' listed on the bill details screen. As I said above, it's detailed to a fault.
Actually it looks like this is for an EBS snapshot. Now it seems like EBS is a Lightsail feature, and the Lightsail console says that I don't have any storage or snapshots? Hm
EBS is also the general external storage attached to an instance. Go the ec2 console and look at snapshots and you should see it there. They are region specific, so make sure to select the region referred to on the bill.
I haven't used Lightsail, so I'm not sure if those snapshots end up in the general ec2 console. But, if you already checked the LS console and didn't see anything, then they must be in the ec2 console.
I haven't used Lightsail, so I'm not sure if those snapshots end up in the general ec2 console. But, if you already checked the LS console and didn't see anything, then they must be in the ec2 console.
Possibly a disassociated elastic IP address?
I wonder if there is a human whose job is to final review suspensions and they just keep shaking their head at the measly pending $0.01 charge.
I keep a penny on my desk at work. Every time a coworker comes across somebody's account that is off by a penny, I offer it. They think I'm being funny, but the point I'm trying to make is that it wastes more than pennies worth of our time to spend it worrying about a few cents.
I've seen people mail in checks worth less than the stamp it took to send them. It's a wild world out there.
I've seen people mail in checks worth less than the stamp it took to send them. It's a wild world out there.
The interesting thing is, accounts being off by a penny or two (excluding actions such as customers doing a mis-type in the bank transfer) can actually point to deeper issues such as improper rounding somewhere in the path.
Or a compromised system https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg_(book)
Usually they wrote us a check for the wrong amount at some point and end up with a small balance remaining in their account. We don't want to just forgive and forget every time somebody owes us $0.16, but if it's been there months and it's going to take extra effort to get it from them, it's probably not worth it.
We do try to determine where the discrepancy started and watch for indicators of a bigger issue.
We do try to determine where the discrepancy started and watch for indicators of a bigger issue.
Hahah yeah I say the same thing every month "yeah okay go ahead and do it then"
I have to imagine that these things are manual and a human has to do it, so they'll sort it by the size of the account and do what matters
I have to imagine that these things are manual and a human has to do it, so they'll sort it by the size of the account and do what matters
They do this to me too every month, as the card on my AWS has expired, and they complain and complain and complain before finally charging the card on my Audible account. Definitely need to get around to separating those.
> DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there.
Did you look at the PDF invoice in the Billing section? I have never seen an invoice where the line items didn't add up to the amount charged to my credit card. If there's just $2.40 on there with no explanation, I'd open a support ticket to complain.
(While looking into this, I was surprised at how minimal DO invoices are, however. For GCP, I'm used to seeing on the order of a million line items per month. Seeing only 3 on my DO invoice was surprising, and could definitely lead to a case where something isn't accounted for correctly. But, I bet support will fix it up for you.)
Did you look at the PDF invoice in the Billing section? I have never seen an invoice where the line items didn't add up to the amount charged to my credit card. If there's just $2.40 on there with no explanation, I'd open a support ticket to complain.
(While looking into this, I was surprised at how minimal DO invoices are, however. For GCP, I'm used to seeing on the order of a million line items per month. Seeing only 3 on my DO invoice was surprising, and could definitely lead to a case where something isn't accounted for correctly. But, I bet support will fix it up for you.)
> DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there. Wuuuttttt.
Oh I used to get this, for me it was snapshots of a VPS I destroyed. Completely my fault, but yeah it was annoying to figure out at the time. Probably worth checking you don't have any reserved IPs, disk backups or snapshots left over
Oh I used to get this, for me it was snapshots of a VPS I destroyed. Completely my fault, but yeah it was annoying to figure out at the time. Probably worth checking you don't have any reserved IPs, disk backups or snapshots left over
Honestly I get lost inside of AWS. Only recently was I able to figure out why I was getting charged $.82/month which, in the long run, is really nothing. But it's amazing how hard it was to figure out why I was getting charged for something that I originally thought was just going to be free.
What did it turn out to be that was charging that?
I used the AWS free plan many many years ago as a student. I was so paranoid about losing money that I immediately shut what I was using down after I thought I didn't need it any more (even though leaving it on was part of the free plan). I turned off the instance, but I forgot the public IP. A month later I got charted a few pennies because an associated public IP is part of the free plan, but an un-associated one isn't.
I didn't go bankrupt, but it proved to me how scummy AWS can be given that actually trying to use less is what got me charged.
I didn't go bankrupt, but it proved to me how scummy AWS can be given that actually trying to use less is what got me charged.
I have a DO server thats been running for more rhan 5 years. It was the basic $5 bucks instance they offered at the time I felt a bit robbed a couple of days ago when I was spinning another instance and I saw that the newer $5 instance has more memory and more cpu power. I feel like they should have decreases the cost of the older less performant instance.
I have been spoiled by my Internet provider who year after year they will automatically increase my speed as they made it cheaper, as I paid the same amount for the service.
I have been spoiled by my Internet provider who year after year they will automatically increase my speed as they made it cheaper, as I paid the same amount for the service.
When they changed the pricing they emailed me and told me that I could effectively reprovision my machine to get the price difference, I think it was part of a rollout of some other infra as well.
I think I'm going to save this whole thread as "why zero friction micropayments are never going to happen".
The billing statements I see in DO under Billing (sidebar) > Billing history seem pretty verbose to me. If it's not there I would recommend reaching out to their support.
Ugh. I'm so glad my first owie was "only" $200. Hurt a lot at the time, though. I could still use that back...
Isn’t DO some random saved image from years ago? I’m still paying them for a few of those I think.
Yeah but it shows they are reasonable and flexible if a customer makes a mistake. The reason Google Cloud is third, people remember how they were treated when they used Google for Work or whatever its called now.
> But something the nice lady on the other end said as she chuckled: "This happens all the time".
Is better then if they said: "We reminded you about the consequences of not deleting, we have done nothing wrong."
> But something the nice lady on the other end said as she chuckled: "This happens all the time".
Is better then if they said: "We reminded you about the consequences of not deleting, we have done nothing wrong."
>DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there. Wuuuttttt.
Wuuuttt? Fraud is what.
Wuuuttt? Fraud is what.
I would not go that far to say it's fraud. Calm down.
Maybe it's something I missed. Maybe its some hidden feature that I did not turn off. But I deleted all my droplets, all my IP's all my firewalls, etc, etc and could not find anything else.
Maybe it's something I missed. Maybe its some hidden feature that I did not turn off. But I deleted all my droplets, all my IP's all my firewalls, etc, etc and could not find anything else.
You have in good faith attempted diligently to have a zero purchase. You cannot even find any purchases. In that case they have reached into your pocket and taken money without your consent. That is just a fact. By mistake? It's being short changed in a shop at the absolute best.
This is not a mistake any company should be able to afford to make. Ever. Under any circumstances. If it really is "just an error" they need to making amends in a very public and obvious way that is clearly expensive to them to show good faith.
But we don't get that. We have to fight from a position of being in the wrong when we have money taken from us by these fraudulent practices to overturn them.
It is necessary to go HARD at the morality of "accidental theft" because it is so prevalent and because all these companies clearly don't consider it a detriment to their reputation.
You've been diligent. When does it become their responsibility to not reach into your pocket when you clearly don't want that. Or anyone's pocket. I'd bet quite a sum that where there is one there is many.
This is not a mistake any company should be able to afford to make. Ever. Under any circumstances. If it really is "just an error" they need to making amends in a very public and obvious way that is clearly expensive to them to show good faith.
But we don't get that. We have to fight from a position of being in the wrong when we have money taken from us by these fraudulent practices to overturn them.
It is necessary to go HARD at the morality of "accidental theft" because it is so prevalent and because all these companies clearly don't consider it a detriment to their reputation.
You've been diligent. When does it become their responsibility to not reach into your pocket when you clearly don't want that. Or anyone's pocket. I'd bet quite a sum that where there is one there is many.
This is where I'd say that every single paid service needs to have a way to contact a human being, and any cancellations expressed to that person by an authenticated user must be respected. If services are allowed to construct Gordian knots of byzantine interacting services, then there must be a way to cut through that mess.
I'd double check that you don't have multiple orgs. You used to share a cess to your account with people, then they made some org change a while back that essentially moved that shared access setup to an org and gave you a new personal account iirc. Easy to not realize you have both.
The invoices on the billing page break down where the charges came from. If you download the pdf version, it has even more details.
Perhaps you have a snapshot?
You should be able to look under your bill for a breakdown on what service the charge is for.
You should be able to look under your bill for a breakdown on what service the charge is for.
> “It’s the student’s responsibility to know what they’re deploying.”
Anyone who seriously argues this is 100% unaware of the quagmire that is AWS billing. There are companies with entire teams built around just optimizing AWS billing - it's whole unsurprising that some AWS feature actually spun up 5 separate AWS features that end up being billed.
Anyone who seriously argues this is 100% unaware of the quagmire that is AWS billing. There are companies with entire teams built around just optimizing AWS billing - it's whole unsurprising that some AWS feature actually spun up 5 separate AWS features that end up being billed.
Not to mention it only goes one-way. If you destroy the service that created all those others, it won't clean those others up.
And even when you spin things up knowing their cost you may not be accounting for data transfer, storage, mismatched reserved instance purchases, unused EIPs, and so many more "hidden costs"
Honestly, my company only forces everyone to add tags with ‘platform’ and ‘team’ to every resource that gets deployed to AWS (otherwise your resource is automatically destroyed).
This works perfectly for going to cost explorer and seeing exactly where you are spending your dollars.
I don’t think this is a very high difficulty exercise.
This works perfectly for going to cost explorer and seeing exactly where you are spending your dollars.
I don’t think this is a very high difficulty exercise.
Ill seriously argue this.
The issue is that when students are "learning" "cloud", they are not really learning cloud as much as following tutorials to click together turnkey solutions. Dynamo DB, ElasicSearch, and whatever else are all business-oriented, quick to set up services. And such, these services are hiding what is going on underneath, with the expectation that the company will cover the costs of whatever those services run. This is a standard buisness practice.
If you want to learn how to put together cloud solutions with those technologies involved, you don't even need an AWS account. All you need is a decent laptop or a desktop, with docker or VM software, where you install everything yourself and learn how to configure it (since all of the software is free), do all the networking yourself. This is what actually learning the cloud involves and translating those skills to AWS is very easy.
Perhaps the blame is on the universities and/or tutorials that push the students towards creating free AWS accounts, but regardless, there should be some incentive to takes one profession seriously so you don't end up making this mistake in a company that will not have protections in place that people want for free tier. And money is as good as incentive as any. Id rather people be smarter rather than treat ignorance as a standard and put blame on others for not expecting people to be ignorant.
The issue is that when students are "learning" "cloud", they are not really learning cloud as much as following tutorials to click together turnkey solutions. Dynamo DB, ElasicSearch, and whatever else are all business-oriented, quick to set up services. And such, these services are hiding what is going on underneath, with the expectation that the company will cover the costs of whatever those services run. This is a standard buisness practice.
If you want to learn how to put together cloud solutions with those technologies involved, you don't even need an AWS account. All you need is a decent laptop or a desktop, with docker or VM software, where you install everything yourself and learn how to configure it (since all of the software is free), do all the networking yourself. This is what actually learning the cloud involves and translating those skills to AWS is very easy.
Perhaps the blame is on the universities and/or tutorials that push the students towards creating free AWS accounts, but regardless, there should be some incentive to takes one profession seriously so you don't end up making this mistake in a company that will not have protections in place that people want for free tier. And money is as good as incentive as any. Id rather people be smarter rather than treat ignorance as a standard and put blame on others for not expecting people to be ignorant.
You are making the assumption that most people are starting these AWS accounts to "learn cloud" when that isn't the case. AWS advertises a free tier for many functions that have nothing to do with "learning cloud".
For example, in the article, the poor student was using Amazon SageMaker to likely build and train models. Blaming the student and saying "why didn't he just buy his own GPU" when Amazon offered 50 hours of free training comes across of out of touch. If the student had the money for a decent desktop he wouldn't be crying over $200.
Amazon could fix this problem easily by introducing a billing cap. While others say it's malevolence as to why Amazon doesn't build one, I think it's more likely that even Amazon couldn't even add this feature to their sprawling billing infrastructure even if they wanted dto.
For example, in the article, the poor student was using Amazon SageMaker to likely build and train models. Blaming the student and saying "why didn't he just buy his own GPU" when Amazon offered 50 hours of free training comes across of out of touch. If the student had the money for a decent desktop he wouldn't be crying over $200.
Amazon could fix this problem easily by introducing a billing cap. While others say it's malevolence as to why Amazon doesn't build one, I think it's more likely that even Amazon couldn't even add this feature to their sprawling billing infrastructure even if they wanted dto.
[deleted]
Can confirm. Default ElastiCache clustering option chooses an extremely high compute node, I ended up accidentally spending $1300 in a month just for testing some Redis clustering script. The minimum option ends up only costing a few dozen dollars a month. The billing alert did not even trigger until the very end of the month, so I got a $1300 surprise. I complained about it to AWS, mentioning how misleading their shit was, and they ended up refunding me $800. Still -- a $500 mistake anyone could make at 11pm. They also made it extra clear in their response that they are not legally obligated to refund me, and that they were doing it out of the kindness of their hearts.
AWS console is some next level outdated shit that needs to be improved. GCP's console is way cleaner IMO. I really hope AWS fixed this after this happened to me, but somehow I doubt they did.
AWS console is some next level outdated shit that needs to be improved. GCP's console is way cleaner IMO. I really hope AWS fixed this after this happened to me, but somehow I doubt they did.
Damn, I can't imagine how you felt...
Only the unique experiences in this thread are enough to know how f*cked up AWS billing is.
Felt like a complete idiot, I guess I'm not smart enough to use AWS's UI in a safe way. When we start trusting tools, we inevitably become complacent. Some user flows are designed to be gotcha's. A little gotcha machine that generates |cost - refund| revenue. They gotcha, and there's nothing you can do about it but beg them for a (usually partial) refund. The refund is the thing that makes you feel better about it.
¯\_(ツ)_/¯
¯\_(ツ)_/¯
Thats exactly the impression I got when I was reading the AWS docs. I am a student too and was evaluating AWS for a hobby project.
Reading the docs I got the feeling that the product is designed for big cooperations only. But not for someone who wants real price control. The free account just made no sense to me.
I ended up renting a cloud server ~3€ a month. (Not AWS) Looking at this thread this was the best decision I could make.
Reading the docs I got the feeling that the product is designed for big cooperations only. But not for someone who wants real price control. The free account just made no sense to me.
I ended up renting a cloud server ~3€ a month. (Not AWS) Looking at this thread this was the best decision I could make.
> Corey Quinn, your first and last stop for any question that touches AWS billing, has called for an updated free tier that treats “personal learning” AWS accounts differently from “new corporate” accounts, and sets hard billing limits that you can’t exceed.
This would be good.
I don't normally do "cloud" stuff. It's just not my skill set. But I have looked at it on occasion and one thing that turns me off is my inability to know if I'm going to fuck myself with a large bill from some of these services.
This would be good.
I don't normally do "cloud" stuff. It's just not my skill set. But I have looked at it on occasion and one thing that turns me off is my inability to know if I'm going to fuck myself with a large bill from some of these services.
+1 Same boat.
I don't know how to work with AWS. I don't want to pay $$$ for a "course" for something that I can explore with a "free" tier.
I logged onto AWS. Looked around, figured I had no idea what I would be charged and what the "free" tier meant I could do exactly.
So I logged out and haven't been back.
I don't know how to work with AWS. I don't want to pay $$$ for a "course" for something that I can explore with a "free" tier.
I logged onto AWS. Looked around, figured I had no idea what I would be charged and what the "free" tier meant I could do exactly.
So I logged out and haven't been back.
Yeah this is a huge problem. I suspect this is significant for people to choose inferior non-hosted alternatives just for peace of mind. Hard limits would probably create a large influx of users.
Even for something as simple as S3, I'm hesitant to use the real service during development because it's so easy to stack up charges. For one of my accounts, the usual bill is <$10 but last month it was $27 because there was some network flakiness that resulted in excessive transfer.
Even for something as simple as S3, I'm hesitant to use the real service during development because it's so easy to stack up charges. For one of my accounts, the usual bill is <$10 but last month it was $27 because there was some network flakiness that resulted in excessive transfer.
Yes,they should totally have a high water mark "trigger" so you could set a shutoff for $100 or whatever.
I had a class last semester where we were required to use AWS. Our professor tried his best to teach everyone how to be responsible with their tool use and how to predict their costs (As well as how to set up a free education account and get AWS credits). I was amazed when final presentations rolled around my group had a total cost of 4.30 dollars after 5 months of uptime and use, while other groups had costs ranging from 30 to 150 dollars! I use AWS for my job so I guess I just never really went through those growing pains, but no system should be that easy to rack up costs unknowingly.
I had an AWS account that charged me like $6 per month for a year after I thought I had turned everything off. I finally went on a hunt for what was causing it, and had to escalate to support to find it. This was 11/12 my fault I admit, I should have been on it after the first month.
More recently, I was looking for a cloud GPU provider, and tried a different provider that I won't name. I tried out a ~$3 per hour instance, and shut it down after maybe 20 min. A few hours later I (thankfully) started getting billing alerts that my bill would be $1500 ish at the end of the months if my usage kept up. I couldn't find anything still running, could not get anything from support (they had some kind of chat support that told me to open a ticket), first opened a ticket then after and then after ab hour shut down my account as a last ditch effort. At which point they promptly emailed me an invoice for the $16 I had incurred during the time before I cancelled. The help desk relied to my ticket like a week later, asking for more information.
Needless to say, I won't ever use that company again, but it could have been a lot worse, even more so if I was a student or someone who blew their whole experimentation budget on whatever mistake I made.
More recently, I was looking for a cloud GPU provider, and tried a different provider that I won't name. I tried out a ~$3 per hour instance, and shut it down after maybe 20 min. A few hours later I (thankfully) started getting billing alerts that my bill would be $1500 ish at the end of the months if my usage kept up. I couldn't find anything still running, could not get anything from support (they had some kind of chat support that told me to open a ticket), first opened a ticket then after and then after ab hour shut down my account as a last ditch effort. At which point they promptly emailed me an invoice for the $16 I had incurred during the time before I cancelled. The help desk relied to my ticket like a week later, asking for more information.
Needless to say, I won't ever use that company again, but it could have been a lot worse, even more so if I was a student or someone who blew their whole experimentation budget on whatever mistake I made.
> [Vague description of provider now removed]
Hinting at the name like this seems unfair to any other companies that might reasonably be interpreted as "[Vague description of provider now removed]"... especially since I can't decide if you're sarcastic about [piece of description] part.
At the very least [two names] are big names that you could plausibly be referring too... and I could see only one name coming into many peoples minds and then them assuming you're referring to that provider.
Hinting at the name like this seems unfair to any other companies that might reasonably be interpreted as "[Vague description of provider now removed]"... especially since I can't decide if you're sarcastic about [piece of description] part.
At the very least [two names] are big names that you could plausibly be referring too... and I could see only one name coming into many peoples minds and then them assuming you're referring to that provider.
Funny story: I keep receiving a $0.01 monthly invoice from AWS for a very old account (opened 5+ years ago) I have lost access to. I have no idea why (the invoice doesn't list the services being used) and the associated credit card has expired a long time ago. They requested notarized documents to prove my identity. Obviously I'd rather settle my $6 (+ interest) debt 50 years from now than give hundreds to a notary today!
It doesn’t cost hundreds of dollars to notarize something. https://www.nationalnotary.org/knowledge-center/about-notari...
Edit: as the other comment mentions, most banks where you have an account offer this for free. My local UPS store where I have a mailbox also offers the service to me for free.
Edit: as the other comment mentions, most banks where you have an account offer this for free. My local UPS store where I have a mailbox also offers the service to me for free.
GP is probably not in Germany, but just an example of why I’d be uninterested in getting something notarized here unless my continued freedom and/or good fortune depended upon it:
American notary: someone who doesn’t have a history of fraud, sat through a workshop and passed a test.
German Notar: a very specialized lawyer, bills like a very specialized lawyer.
Theoretically, you can get stuff notarized at US embassies and consulates, for $50 and an appointment, must bring your own witness(es). That of course has not been an option much of the past year.
American notary: someone who doesn’t have a history of fraud, sat through a workshop and passed a test.
German Notar: a very specialized lawyer, bills like a very specialized lawyer.
Theoretically, you can get stuff notarized at US embassies and consulates, for $50 and an appointment, must bring your own witness(es). That of course has not been an option much of the past year.
I'm not from the US.
In the USA, in Marin County, I was charged $25 per document. The guy doing it was adding a finger print, which is not a bad idea if the originator of the document is present.
(Finger print is not legally required though.)
(Finger print is not legally required though.)
Any bank at which you are a customer will notarize documents for you for free.
I'm not from the US. This is not available in any of my banks.
If notarization isn't common in the country where you are, you should probably escalate through AWS support in that country. They probably have an alternate procedure. If you're talking to US-based support, they are used to thinking this is a trivial request. Getting something notarized is something that can be handled in 5 minutes at places as common as convenience stores. Most people have coworkers that are notaries and can do it without even getting up from their desk.
If that's not the case in your country, ask them for a different procedure. It's not something they intend to be onerous.
If that's not the case in your country, ask them for a different procedure. It's not something they intend to be onerous.
I'm not interested in recovering that account anyway, I'd spend more money (in terms of time spent dealing with them) on trying to get it back than what I'd owe in a few decades.
In Germany you can also get documents notarized in (Catholic) parish offices. I think they do it for free.
That said, this sounds unreasonable and if you're not in the US this may violate consumer protection regulations if they didn't require the same level of verification to sign the contract in the first place.
That said, this sounds unreasonable and if you're not in the US this may violate consumer protection regulations if they didn't require the same level of verification to sign the contract in the first place.
It is unreasonable but it's just not worth engaging with them even to exchange a few emails, financially speaking :)
[deleted]
All states, with exceptions listed below, cap notary charge to $10 at most. Most states are less. These states don't have limits one way or another:
Alaska
Arkansas
Iowa
Kansas
Kentucky
Loisiana
Maine
Massachusetts
Puerto Rico
Tennessee
Vermont
Alaska
Arkansas
Iowa
Kansas
Kentucky
Loisiana
Maine
Massachusetts
Puerto Rico
Tennessee
Vermont
I'm not from the US.
Then you have not provided enough information to validate your claim. For example, the EU imposes price controls as well, while in Japan it can be expensive.
However, there's a much simpler route: Amazon is a US country, so you can simply use notorize.com for $25.
However, there's a much simpler route: Amazon is a US country, so you can simply use notorize.com for $25.
My claim is that it's ridiculous to send $0.01 monthly invoices for an inactive account, not sure what you mean by "validate", maybe it isn't that funny of a story? :) But it doesn't make sense for me to spend even $1 on anything regarding this matter. I'm ok with not having access to that account anyway. If anything they should pay me to go through the hassle.
Oracle - of all businesses - got this right. I know, I'm in shock too.
In Oracle Cloud Infrastructure, after your credit is exhausted, you have to explicitly opt in to billing or else they stop all paid services for you.
Moreover, you can choose to keep billing disabled and use their free services without fear of unknown costs.
AWS have just decided to run with a policy of offering refunds when people make mistakes. Unfortunately, some people are ignorant of this, or too timid to ask for their money back.
In Oracle Cloud Infrastructure, after your credit is exhausted, you have to explicitly opt in to billing or else they stop all paid services for you.
Moreover, you can choose to keep billing disabled and use their free services without fear of unknown costs.
AWS have just decided to run with a policy of offering refunds when people make mistakes. Unfortunately, some people are ignorant of this, or too timid to ask for their money back.
Wow this is amazing! Never thought Oracle would have such a nice free tier.
> 2 Compute virtual machines with 1/8 OCPU and 1 GB memory each.
> 2 Block Volumes Storage, 100 GB total.
> 10 GB Object Storage.
> 10 GB Archive Storage.
> Resource Manager: managed Terraform.
> 2 Compute virtual machines with 1/8 OCPU and 1 GB memory each.
> 2 Block Volumes Storage, 100 GB total.
> 10 GB Object Storage.
> 10 GB Archive Storage.
> Resource Manager: managed Terraform.
>Never thought Oracle would have such a nice free tier.
Attracted a fair bit of attention in some corners of the internet for this reason. Then they canned a bunch of accounts using "always free" stuff for myself and others. Do not trust.
Attracted a fair bit of attention in some corners of the internet for this reason. Then they canned a bunch of accounts using "always free" stuff for myself and others. Do not trust.
> Unfortunately, some people are ignorant of this, or too timid to ask for their money back.
I guess that turns out to be good for AWS’s bottom line.
I guess that turns out to be good for AWS’s bottom line.
Might just be an urban legend but doesnt Amazon keep track of people who do too many refunds and cancel services with them? I've heard of and met people who dont do minor returns to Amazon because they were afraid it would put their Prime account on some kind of "naughty list"
I thought I was just to incompetent to figure out how to stop an instance…so I just gave up while watching my creditcard getting billed for about €20 each month..
Last month I went full on and spent a few hours removing everything on every screen I could find.
After many hours of failing to remove an instance because you have to stop a thingie from auto starting on another screen first…and another thingie from running there.
I finally managed to remove all the instances…
Then two months later, I still get a bill for a few €…
I give up.
I Hate Amazon
Then two months later, I still get a bill for a few €…
I give up.
I Hate Amazon
You can try using this tool:
https://github.com/iann0036/former2
This will scan your entire account and list all of your resources - it's actually made for generating CloudFormation templates, but it's very useful for a use-case like yours.
This will scan your entire account and list all of your resources - it's actually made for generating CloudFormation templates, but it's very useful for a use-case like yours.
I'll bet you created the environment with Elastic Beanstalk, whose job it is to (among other things) replace instances if they fail.
When you stopped the instance in EC2, EB did its job and created a new instance.
You eventually figured this out and killed the EB env, and the instances stopped reappearing.
But the Elastic IP address assigned to your EB env is still on your account, and it's no longer free of charge, because you don't have a running instance.
So you will be billed about a €/mo until you delete the Elastic IP reservation.
This is ridiculously confusing, and ridiculously common.
When you stopped the instance in EC2, EB did its job and created a new instance.
You eventually figured this out and killed the EB env, and the instances stopped reappearing.
But the Elastic IP address assigned to your EB env is still on your account, and it's no longer free of charge, because you don't have a running instance.
So you will be billed about a €/mo until you delete the Elastic IP reservation.
This is ridiculously confusing, and ridiculously common.
If you delete your AWS account you will still be charged if you don’t remove / terminate some of your services for up to 90 days!
https://aws.amazon.com/premiumsupport/knowledge-center/close...
https://aws.amazon.com/premiumsupport/knowledge-center/close...
For me its currently over two years, not 90 days
I used the free tier account to play around for a while. I thought I had deleted everything before the end of the year free tier. I was wrong! For 2 months I paid $10 as I could not figure it out what the heck was running.
I had to close my account in order to stop the charges. Today when I hear anyone speak of AWS free tier for a year, first thing I warn them about is to make sure they keep track of what they create so they will know what to delete, otherwise they will keep buying Starbucks every month to AWS.
I had to close my account in order to stop the charges. Today when I hear anyone speak of AWS free tier for a year, first thing I warn them about is to make sure they keep track of what they create so they will know what to delete, otherwise they will keep buying Starbucks every month to AWS.
Is there seriously no way in AWS/Azure/GCP to specify "Here's my budget, shut everything down if I exceed $X"? I don't use those platforms much but was always surprised I couldn't find anything like that right off the bat. I'll build cloud stuff if it makes sense at work, but if I'm footing the bill I'll stick to something that can provide an actual upper limit.
I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports. You can burn through a lot of cloud resources in 3 days.
GCP at least has some provision to get very detailed information about usage (but not cost) that updates in less than an hour. That, to me, is the tool for building something like "shut down our account if usage is too high". It is annoying that you have to code this yourself, but ultimately, it kind of makes sense to me. Cloud providers exist to rent you some hardware (often with "value-add" software); it's the developer and operator's responsibility to account for every machine they request, every byte they send to the network, every query they make to the database, etc. and to have a good reason for that. To some extent, if you don't know where you're sending bytes, or what queries you're making, how do you know if your product is working? How do you know that you're not hacked? Reliability and cost go hand in hand here -- if you're monitoring what you need to assure reliability, costs probably aren't confusingly accumulating out of control.
GCP at least has some provision to get very detailed information about usage (but not cost) that updates in less than an hour. That, to me, is the tool for building something like "shut down our account if usage is too high". It is annoying that you have to code this yourself, but ultimately, it kind of makes sense to me. Cloud providers exist to rent you some hardware (often with "value-add" software); it's the developer and operator's responsibility to account for every machine they request, every byte they send to the network, every query they make to the database, etc. and to have a good reason for that. To some extent, if you don't know where you're sending bytes, or what queries you're making, how do you know if your product is working? How do you know that you're not hacked? Reliability and cost go hand in hand here -- if you're monitoring what you need to assure reliability, costs probably aren't confusingly accumulating out of control.
Are you being sarcastic?
> I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports.
That does not sound like a good reason, but more like a crappy implementation of usage collection.
I don’t see why a bunch of Google engineers can’t implement real-time billing properly, and see no reason to defend their inability to do their job.
> I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports.
That does not sound like a good reason, but more like a crappy implementation of usage collection.
I don’t see why a bunch of Google engineers can’t implement real-time billing properly, and see no reason to defend their inability to do their job.
IIRC the excuse is that billing is a separate department and they count all the usage dollars way after you done using it, not in realtime. You would still be able to go over your limit and then who should foot the bill?
Realtime counting would be too difficult to figure out, our brightest minds are busy figuring out engagement metrics.
Realtime counting would be too difficult to figure out, our brightest minds are busy figuring out engagement metrics.
This is my personal opinion, though I do work at AWS:
It's not that real time counting is difficult, it is that the amount of compute resources and electricity that would be needed to power real time billing at AWS scale would be astronomical. There is a reason why banks and financial institutions generally do batch processing in the off peak hours when electricity is cheaper and there is less demand for the compute resources. Now imagine AWS billing, which is arguably far more difficult in scale and complexity.
It's not that real time counting is difficult, it is that the amount of compute resources and electricity that would be needed to power real time billing at AWS scale would be astronomical. There is a reason why banks and financial institutions generally do batch processing in the off peak hours when electricity is cheaper and there is less demand for the compute resources. Now imagine AWS billing, which is arguably far more difficult in scale and complexity.
I also work at AWS (nowhere near billing), so the usual disclaimers apply, but:
I actually have no idea if billing is real-time or not? I think it's mostly batch, but the records aren't necessarily, though they may be aggregated a bit.
The general point in this thread certainly holds: our systems provide service first, bill second, and that by throwing a record over the wall to some other internal system. It's not unthinkable they could tally up your costs as you go, but the expense has fundamentally already happened, and that's the disconnect.
It would be hard to react ahead of time. Small, cheap things like "a single DynamoDB query" or "a byte of bandwidth" are often also high-volume, and you don't want to introduce billing checks to critical paths for reliability / latency / cost reasons. Expensive big-bang on/off things, probably simpler, though I can think of a few sticking points.
It would be hard to react after the fact, too. Where does a bill come from? My own team is deeply internal, several layers removed from anything you're recognize on an invoice, but we're the one generating and measuring costs. Precise attribution would be a problem in and of itself- cutting off service means traversing those layers in reverse, then figuring out what "cut off" means in our specific context. That's new systems and new code all around, repeat for all of AWS- there's a huge organizational problem here on top of the technical one.
I could see some individual teams doing something like this for just their services, but AWS-wide would be a big undertaking.
I wish we had it- I'd sleep a little better at night, myself- but from my limited perspective, it sure looks like we're fundamentally not designed for this.
I actually have no idea if billing is real-time or not? I think it's mostly batch, but the records aren't necessarily, though they may be aggregated a bit.
The general point in this thread certainly holds: our systems provide service first, bill second, and that by throwing a record over the wall to some other internal system. It's not unthinkable they could tally up your costs as you go, but the expense has fundamentally already happened, and that's the disconnect.
It would be hard to react ahead of time. Small, cheap things like "a single DynamoDB query" or "a byte of bandwidth" are often also high-volume, and you don't want to introduce billing checks to critical paths for reliability / latency / cost reasons. Expensive big-bang on/off things, probably simpler, though I can think of a few sticking points.
It would be hard to react after the fact, too. Where does a bill come from? My own team is deeply internal, several layers removed from anything you're recognize on an invoice, but we're the one generating and measuring costs. Precise attribution would be a problem in and of itself- cutting off service means traversing those layers in reverse, then figuring out what "cut off" means in our specific context. That's new systems and new code all around, repeat for all of AWS- there's a huge organizational problem here on top of the technical one.
I could see some individual teams doing something like this for just their services, but AWS-wide would be a big undertaking.
I wish we had it- I'd sleep a little better at night, myself- but from my limited perspective, it sure looks like we're fundamentally not designed for this.
“Small, cheap things like "a single DynamoDB query" or "a byte of bandwidth" are often also high-volume, and you don't want to introduce billing checks to critical paths for reliability / latency / cost reasons”
That’s hardly necessary. Let’s suppose you have some service that costs 1 cent every 1,000 queries. If you’re billing it then you need to be keeping track of it and incrementing some counter somewhere. If old number mod x < new number mod x them do some check, that’s very cheap on average and doesn’t add latency if done after the fact.
PS: Phone companies can pull this stuff off successfully for millions of cellphones. If you’re arguing keeping up with AT&T is to hard, you have serious organizational problems.
That’s hardly necessary. Let’s suppose you have some service that costs 1 cent every 1,000 queries. If you’re billing it then you need to be keeping track of it and incrementing some counter somewhere. If old number mod x < new number mod x them do some check, that’s very cheap on average and doesn’t add latency if done after the fact.
PS: Phone companies can pull this stuff off successfully for millions of cellphones. If you’re arguing keeping up with AT&T is to hard, you have serious organizational problems.
That counter may well not exist outside of billing for longer than it takes to batch some records together. It will need to be shared and synchronized with the rest of the fleet, the other fleets in the availability zone, the other zones in the region, the other regions, and every other service in AWS. There are strict rules about crossing these boundaries for sake of failure isolation.
As an amusing extra sticking point, your service has no idea how much it actually costs, because that's calculated by billing- the rates may vary from request to request or from customer to customer.
Without spending way too long thinking about it, the complexity in figuring out exactly when to stop is significant enough that it probably cannot practically be done in the critical path of these kinds of high-volume systems, hence the reactive approach being more plausible to me.
I don't know what kinds of problems AT&T has, but at the risk of saying dumb things about an industry I know next to nothing about, your phone is only attached to one tower at a time, and that probably helps a bit. And I'm not sure when it wouldn't be simpler and just as good for them to also react after the fact, anyway.
As an amusing extra sticking point, your service has no idea how much it actually costs, because that's calculated by billing- the rates may vary from request to request or from customer to customer.
Without spending way too long thinking about it, the complexity in figuring out exactly when to stop is significant enough that it probably cannot practically be done in the critical path of these kinds of high-volume systems, hence the reactive approach being more plausible to me.
I don't know what kinds of problems AT&T has, but at the risk of saying dumb things about an industry I know next to nothing about, your phone is only attached to one tower at a time, and that probably helps a bit. And I'm not sure when it wouldn't be simpler and just as good for them to also react after the fact, anyway.
>your phone is only attached to one tower at a time
Not when you're being simswapped
Not when you're being simswapped
First arguing based on existing infrastructure ignores the fact your changing the system therefore any new system is a viable option. All the existing system changes is how much things cost. Anyway, for independent distributed systems you can use probability rather than fixed numbers.
That said, your losing the forest for the trees, the accuracy isn’t that important. You can still bill for actual usage. A 15 minute granularity is vastly better than a 30 day one. As long as you can kill processes you don’t need to check in the middle of every action. Things being asynchronous is just the cost of business at scale.
That said, your losing the forest for the trees, the accuracy isn’t that important. You can still bill for actual usage. A 15 minute granularity is vastly better than a 30 day one. As long as you can kill processes you don’t need to check in the middle of every action. Things being asynchronous is just the cost of business at scale.
I'm hardly saying it's impossible; I'm saying that it's not easy, and may even be hard. Doing it well would likely require a wide-reaching effort the likes of which would eventually reach my ears, and the fact that I haven't heard of such a thing implies to me that it's probably not an AWS priority.
Why that would be, I leave to you.
Why that would be, I leave to you.
> PS: Phone companies can pull this stuff off successfully for millions of cellphones. If you’re arguing keeping up with AT&T is to hard, you have serious organizational problems.
To be fair, AT&T in particular does prepaid shutoffs on a pretty coarse granularity, I think it's like 15-minute intervals.
I know this because for a while I had to use a prepaid LTE modem as my primary internet connection. You can use as much bandwidth as you want for the remainder of the 15-minute interval in which you exceed what you've paid for -- then they shut you off.
I once managed (by accident) to get 3GB out of a 2GB plan purchase because of this.
Of course that free 1GB was only free because I consumed all of it in the 14.9 minute time period preceding NO CARRIER.
To be fair, AT&T in particular does prepaid shutoffs on a pretty coarse granularity, I think it's like 15-minute intervals.
I know this because for a while I had to use a prepaid LTE modem as my primary internet connection. You can use as much bandwidth as you want for the remainder of the 15-minute interval in which you exceed what you've paid for -- then they shut you off.
I once managed (by accident) to get 3GB out of a 2GB plan purchase because of this.
Of course that free 1GB was only free because I consumed all of it in the 14.9 minute time period preceding NO CARRIER.
There’s a lot of middle ground between credit limit checks within every database transaction and the current state.
> There’s a lot of middle ground between credit limit checks within every database transaction and the current state.
But there isn’t a lot of middle ground between distributed, strongly-consistent credit limit checks every API call and billing increment (which is, IIRC, instance-seconds or smaller for some time-based services) and a hard billing cap that is actually usable on a system structured like AWS. Partial solutions reduce the risk but don’t eliminate the problem, and at AWS scale reducing the risk means you still have significant numbers of people reliant on the “call customer service” mitigation, and how much spending and system compromise to narrow this billing issue is worthwhile if you are still in that position?
But there isn’t a lot of middle ground between distributed, strongly-consistent credit limit checks every API call and billing increment (which is, IIRC, instance-seconds or smaller for some time-based services) and a hard billing cap that is actually usable on a system structured like AWS. Partial solutions reduce the risk but don’t eliminate the problem, and at AWS scale reducing the risk means you still have significant numbers of people reliant on the “call customer service” mitigation, and how much spending and system compromise to narrow this billing issue is worthwhile if you are still in that position?
> the amount of compute resources and electricity that would be needed to power real time billing at AWS scale would be astronomical
You don't have to bill in real time.
You just have to provision funding for every resource except network bandwidth.
Customer sets a monthly spend limit. Every time they start up an instance, create a volume, allocate an IP, or do anything else that costs money, you subtract the monthly cost of that new thing from their spend limit. If the spend limit would go negative, you refuse to create the new resource.
If the spend limit is still positive, the remaining amount is divided by the number of seconds remaining in the month times the bandwidth cost. The result becomes that customer's network throughput limit. Update that setting in your routers (qdisc in Linux) as part of the API call that allocated a resource. If you claim your routers don't have a limit like this I call shenanighans.
This should work perfectly for one region.
There's probably a way to generalize it to multiple regions, but I'm sure most small/medium customers would be happy enough to have a budget for each region. They'd probably set most regions' budget to zero and just worry about one or two.
The web UI probably would need to be updated to show the customer "here is what your bandwidth limit for the rest of the month will be if you proceed; are you sure?". JSON APIs can return this value when invoked in dry-run mode.
You don't have to bill in real time.
You just have to provision funding for every resource except network bandwidth.
Customer sets a monthly spend limit. Every time they start up an instance, create a volume, allocate an IP, or do anything else that costs money, you subtract the monthly cost of that new thing from their spend limit. If the spend limit would go negative, you refuse to create the new resource.
If the spend limit is still positive, the remaining amount is divided by the number of seconds remaining in the month times the bandwidth cost. The result becomes that customer's network throughput limit. Update that setting in your routers (qdisc in Linux) as part of the API call that allocated a resource. If you claim your routers don't have a limit like this I call shenanighans.
This should work perfectly for one region.
There's probably a way to generalize it to multiple regions, but I'm sure most small/medium customers would be happy enough to have a budget for each region. They'd probably set most regions' budget to zero and just worry about one or two.
The web UI probably would need to be updated to show the customer "here is what your bandwidth limit for the rest of the month will be if you proceed; are you sure?". JSON APIs can return this value when invoked in dry-run mode.
> Update that setting in your routers (qdisc in Linux) as part of the API call that allocated a resource. If you claim your routers don't have a limit like this I call shenanighans.
Eh. AWS's edge network is highly distributed. Unless you want an even split of your rate limit across every possible way out of the network, you'd be much better off settling for an even split across your EC2 instances, and there's no room for bursting in this model. Enforcing per-instance limits (on any dimension) sounds pretty feasible, though.
This wouldn't generalize straightforwardly to services that don't have obvious choke points that can impose this sort of throttling, such as, I think, DynamoDB.
Eh. AWS's edge network is highly distributed. Unless you want an even split of your rate limit across every possible way out of the network, you'd be much better off settling for an even split across your EC2 instances, and there's no room for bursting in this model. Enforcing per-instance limits (on any dimension) sounds pretty feasible, though.
This wouldn't generalize straightforwardly to services that don't have obvious choke points that can impose this sort of throttling, such as, I think, DynamoDB.
> Customer sets a monthly spend limit. Every time they start up an instance, create a volume, allocate an IP, or do anything else that costs money, you subtract the monthly cost of that new thing from their spend limit. If the spend limit would go negative, you refuse to create the new resource.
AWS systems are highly distributed; this kind of sharp billing cap would necessarily introduce a new strong consistency requirement across multiple services, many of which aren’t even strongly consistent considered one at a time (and that’s often true even if you limit to a single region.)
> Every time they start up an instance, create a volume, allocate an IP, or do anything else that costs money, you subtract the monthly cost of that new thing from their spend limit
For the motivating use case (avoiding a bill on the scale of even $200—possibly even $1—from a free-tier-eligible account), using monthly chunks doesn’t work; you suddenly couldn’t spin up a second simultaneous EC2 instance of ant kind after an initial t3.micro instance, which would cutoff many normal free tier usage patterns.
I mean, that’s a good way of capping if you are using AWS as a super overpriced steady-state VPS, but that’s not really the usage pattern that causes the risks that the cap idea is intended to protect against.
This is a particularly poor solution to completely the wrong problem.
AWS systems are highly distributed; this kind of sharp billing cap would necessarily introduce a new strong consistency requirement across multiple services, many of which aren’t even strongly consistent considered one at a time (and that’s often true even if you limit to a single region.)
> Every time they start up an instance, create a volume, allocate an IP, or do anything else that costs money, you subtract the monthly cost of that new thing from their spend limit
For the motivating use case (avoiding a bill on the scale of even $200—possibly even $1—from a free-tier-eligible account), using monthly chunks doesn’t work; you suddenly couldn’t spin up a second simultaneous EC2 instance of ant kind after an initial t3.micro instance, which would cutoff many normal free tier usage patterns.
I mean, that’s a good way of capping if you are using AWS as a super overpriced steady-state VPS, but that’s not really the usage pattern that causes the risks that the cap idea is intended to protect against.
This is a particularly poor solution to completely the wrong problem.
> AWS systems are highly distributed
Hogwash, I tried to spin up 100 of your F1 instances in us-east-1 a week or two after they first became available, and found out about this thing called "limits".
Wherever you're enforcing the limit on max number of instances per region is already a synchronization point of exactly the sort needed here.
I'm sorry, this just doesn't pass the bullshit test. Resource allocation API calls are not even remotely close to lightning-quick. There is no fundamental immutable constraint here.
> For the motivating use case (avoiding a bill on the scale of even $200—possibly even $1—from a free-tier-eligible account),
Avoiding a $1 bill is definitely not the motivating use case.
A lot of people would be happy to have a mechanism that could prevent them from being billed 5x their expected expenditure (i.e. they set their budget limit to 5x what they intend to spend). It doesn't matter that that isn't perfect. It is massively better than what you're offering right now.
Hogwash, I tried to spin up 100 of your F1 instances in us-east-1 a week or two after they first became available, and found out about this thing called "limits".
Wherever you're enforcing the limit on max number of instances per region is already a synchronization point of exactly the sort needed here.
I'm sorry, this just doesn't pass the bullshit test. Resource allocation API calls are not even remotely close to lightning-quick. There is no fundamental immutable constraint here.
> For the motivating use case (avoiding a bill on the scale of even $200—possibly even $1—from a free-tier-eligible account),
Avoiding a $1 bill is definitely not the motivating use case.
A lot of people would be happy to have a mechanism that could prevent them from being billed 5x their expected expenditure (i.e. they set their budget limit to 5x what they intend to spend). It doesn't matter that that isn't perfect. It is massively better than what you're offering right now.
> Hogwash, I tried to spin up 100 of your F1 instances
I don’t have any F1 instances. Have you mistaken me for an AWS employee rather than a user?
> in us-east-1 a week or two after they first became available, and found out about this thing called "limits".
Yes, individual services, especially in individual regions, and especially a single type of resource within a service within a region like, say, instances in EC2, are often at least enough like centralized to impose hard limits reasonably well.
Billing accounts (and individual real projects which—and this is one disadvantage AWS has vs, say, GCP—AWS has only the weakest concept of) tend to span multiple resource types in each of multiple services, and sometimes multiple regions.
> Resource allocation API calls are not even remotely close to lightning-quick.
Resource allocation API calls that have high latency aren’t the only API calls that cost money and would need coordination. Heck, API calls aren’t the only thing that costs money.
I don’t have any F1 instances. Have you mistaken me for an AWS employee rather than a user?
> in us-east-1 a week or two after they first became available, and found out about this thing called "limits".
Yes, individual services, especially in individual regions, and especially a single type of resource within a service within a region like, say, instances in EC2, are often at least enough like centralized to impose hard limits reasonably well.
Billing accounts (and individual real projects which—and this is one disadvantage AWS has vs, say, GCP—AWS has only the weakest concept of) tend to span multiple resource types in each of multiple services, and sometimes multiple regions.
> Resource allocation API calls are not even remotely close to lightning-quick.
Resource allocation API calls that have high latency aren’t the only API calls that cost money and would need coordination. Heck, API calls aren’t the only thing that costs money.
[deleted]
> You would still be able to go over your limit and then who should foot the bill?
The provider. What they do wouldn't be accepted in any other industry. Imagine hiring an appliance repair shop who sends a repair person that can fix your stuff immediately, but can't tell you what it's going to cost until 3 days after the work is done.
Then you get a huge bill because you wanted "appliance repair" (one of them), but ended up with "appliance maintenance" (all of them).
The provider. What they do wouldn't be accepted in any other industry. Imagine hiring an appliance repair shop who sends a repair person that can fix your stuff immediately, but can't tell you what it's going to cost until 3 days after the work is done.
Then you get a huge bill because you wanted "appliance repair" (one of them), but ended up with "appliance maintenance" (all of them).
A lot of people have thoughfully responded with reasons why this doesn't or can't exist: real-time billing is far too expensive to implement, better to get a huge bill than to lose data or shut down critical systems, etc. I guess it makes sense—ideally you are monitoring your stuff, whether you're using your own tools or built-in ones—and you know ahead of time when your usage is creeping up. Also, I suppose only the customer can really know which systems can be shut down/deallocated to save money and which ones would kill the company if shut down. It sounds like if you're a small startup strapped for resources, you can avoid these bills either by self-hosting or by being careful about how you build your cloud infrastructure. I.e. maybe you could host your app on your own box OR on an equivalent VM in Azure that's just going to fail if it runs out of disk/CPU/outgoing bandwidth instead of autoscaling to outrageous levels.
That’s extremely hard to design, at least with the current state of what AWS bills and does not bill.
Example: let’s assume you’ve set the cut-off budget too strict, spun off another shard for your stateful service (DB for example), it received and stored some data during the short window before some other service brought whole account over budget (i.e. paid egress crossed the threshold).
To bring VM and EBS charges to 0 (to implement your idea of ‘shut down everything’) AWS will have to delete production data.
While it may be OK when you’re experimenting, it’s really hard to tell in automated way.
So, to fully comply w/o risking losing customer data, AWS will have to stop charging for not running instances and inactive EBS volumes which most definitely bring on many kinds of abuse.
—
There may be some other way to do this, maybe mark some services expendable or not, so they are stopped in the event of overspend.
Example: let’s assume you’ve set the cut-off budget too strict, spun off another shard for your stateful service (DB for example), it received and stored some data during the short window before some other service brought whole account over budget (i.e. paid egress crossed the threshold).
To bring VM and EBS charges to 0 (to implement your idea of ‘shut down everything’) AWS will have to delete production data.
While it may be OK when you’re experimenting, it’s really hard to tell in automated way.
So, to fully comply w/o risking losing customer data, AWS will have to stop charging for not running instances and inactive EBS volumes which most definitely bring on many kinds of abuse.
—
There may be some other way to do this, maybe mark some services expendable or not, so they are stopped in the event of overspend.
A complicated solution is not what people are really asking for though.
What I and I expect most people want is a cap which then spins down services when they reach that cap. Nobody is going to care if the cap is set to $1000 and the final bill is $1,020. The problem being solved for is not wanting to have to ever worry about missing an alert and waking up to a bill that is a factor or two beyond expectations. I can afford my bill being 10% or even 40% above my expectation. I can't afford my bill being 500% off.
What I and I expect most people want is a cap which then spins down services when they reach that cap. Nobody is going to care if the cap is set to $1000 and the final bill is $1,020. The problem being solved for is not wanting to have to ever worry about missing an alert and waking up to a bill that is a factor or two beyond expectations. I can afford my bill being 10% or even 40% above my expectation. I can't afford my bill being 500% off.
I do understand that, but there are services that are still billed for when ‘spun down’. To stop getting the bills they have to be terminated.
The solution seems to be to implement ‘emergency stop’ when whole account is put to pause but no state is purged. And then you’ll probably have a week or two to react and decide if you want larger bill, salvage the data or just terminate it all.
The solution seems to be to implement ‘emergency stop’ when whole account is put to pause but no state is purged. And then you’ll probably have a week or two to react and decide if you want larger bill, salvage the data or just terminate it all.
> So, to fully comply w/o risking losing customer data, AWS will have to stop charging for not running instances and inactive EBS volumes which most definitely bring on many kinds of abuse.
Another option would be to "reserve" them in the budget. That is, when for instance you create a 10 GB EBS volume, count it in the budget as if it will be kept for the whole month (and all months after that). Yes, that would mean an EBS volume costing $X per day would count in the budget as 31*$X, but it would prevent both going over the limit and having to lose data.
Another option would be to "reserve" them in the budget. That is, when for instance you create a 10 GB EBS volume, count it in the budget as if it will be kept for the whole month (and all months after that). Yes, that would mean an EBS volume costing $X per day would count in the budget as 31*$X, but it would prevent both going over the limit and having to lose data.
This creates a surprising situation.
We have a batch process that uses S3 as a way to temporarily store some big datasets while another streams and processes then and, once complete, removes them. This takes like an hour.
So our S3 bill could be, let’s say, $10/mo. If you went and looked at your S3 bill and saw that and thought setting a $20 cap would give you reasonable headroom you’d be sorely surprised when S3 stopped accepting writes or other services started getting shut down or failing next time the batch job triggered.
Under this system, the budget and actual bill need to be off by a factor of more than 10 ($240). And this also doesn’t stop your bill being off by a factor of 10 from what you “wanted” to set your limit to. More than the $200 under discussion.
We have a batch process that uses S3 as a way to temporarily store some big datasets while another streams and processes then and, once complete, removes them. This takes like an hour.
So our S3 bill could be, let’s say, $10/mo. If you went and looked at your S3 bill and saw that and thought setting a $20 cap would give you reasonable headroom you’d be sorely surprised when S3 stopped accepting writes or other services started getting shut down or failing next time the batch job triggered.
Under this system, the budget and actual bill need to be off by a factor of more than 10 ($240). And this also doesn’t stop your bill being off by a factor of 10 from what you “wanted” to set your limit to. More than the $200 under discussion.
I think there's a good argument that they could do better. But there's also probably an argument that harder circuit breakers would result in posts like "AWS destroyed my business right when I was featured in the New York Times"--including things like deleting production data. I'm sort of sympathetic to the idea that AWS should have an experimental "rip it all down" option and kill all stateless services option but that adds another layer of complexity and introduces more edge cases.
Merely being on free tier is a signal that you do not want a $27,000 bill. There is no excuse for what AWS does here, and it is clear they use this clusterfuck as a revenue stream.
They can simply add a nuclear option. Ordinary business probably won't enable that option but individuals can and should set it up.
It’s a lot easier for them to refund the individuals who make mistakes than deal with the fallout and bad press from the small businesses they’ve destroyed when they incorrectly enable that option.
Yes, a useful last resort safeguard would need to be more granular than just "turn everything off", at least if we're talking about protecting production systems rather than just people learning and wanting to avoid inadvertently leaving the free tier or something like that.
Still, it's not hard to imagine some simple refinements that might be practical. For example, an option to preserve data in DB or other storage services but stop all other activity might be a useful safety net for a lot of AWS customers. It wouldn't cut costs to 0, but that's not necessarily the optimal outcome for customers running in production anyway. It would probably mitigate the greatest risks of runaway costs, though.
Still, it's not hard to imagine some simple refinements that might be practical. For example, an option to preserve data in DB or other storage services but stop all other activity might be a useful safety net for a lot of AWS customers. It wouldn't cut costs to 0, but that's not necessarily the optimal outcome for customers running in production anyway. It would probably mitigate the greatest risks of runaway costs, though.
You can get alerts. And there are some limits for young accounts. But they really need a "beginner" mode. Or a budget mode, put $x on the account can't spend more. But I guess they are making a lot of money with "mistakes" so there may not be kuch incentives
On top of the other reasons for complexity and delay, this just creates another potential mistake where people delete their entire accounts or stop production services.
It's far easier to negotiate dollar amounts than lost data or service uptime.
It's far easier to negotiate dollar amounts than lost data or service uptime.
Azure will alert you when you exceed a budget, but it won't disable anything.
Azure MongoDB billing was insanity. I was up to $800 to host a couple of GB that wasn't doing anything.
I'm still not sure what happened, even their support kept saying "it's priced by request units" and I kept saying "How does a handful of queries a day translate to $40 in request units?"
A year later, I think that I had a lot of collections and they seem to charge per-collection, but I'm still not even sure. Thank goodness we moved off of it after only a month or two.
Azure MongoDB billing was insanity. I was up to $800 to host a couple of GB that wasn't doing anything.
I'm still not sure what happened, even their support kept saying "it's priced by request units" and I kept saying "How does a handful of queries a day translate to $40 in request units?"
A year later, I think that I had a lot of collections and they seem to charge per-collection, but I'm still not even sure. Thank goodness we moved off of it after only a month or two.
I work for MongoDB. The writer is referring to CosmosDB, the Microsoft clone of MongoDB. You can runa real MongoDB cluster on Azure with MongoDB Atlas. The pricing model for Atlas is per Node size + disk + memory + data transfer. It's generally easier to predict your costs using this model. Users tend to over-provision with this model so we recommend using auto-scaling. This will allow your cluster to scale up or down based on load (price will adjust as well).
We switched to MongoDB Atlas which was awesome but I didn't want to sound like a stealth corporate shill. Atlas rocks!
Love to hear that.
> Is there seriously no way in AWS/Azure/GCP to specify "Here's my budget, shut everything down if I exceed $X"?
All of them have billing APIs which should, in principal, allow you to build “Shutdown everything at some indefinite time interval (and additional spend) after I exceed $X”, though you’ll need to do nontrivial custom coding to tie together the various pieces to do it, and actually stopping all spend is likely going to mean nuking storage, not just halting “active” services.
None of them provide anyway to more than probabilistically do “Shut everything down before I exceed, and in a way which prevents exceeding, $X.”
All of them have billing APIs which should, in principal, allow you to build “Shutdown everything at some indefinite time interval (and additional spend) after I exceed $X”, though you’ll need to do nontrivial custom coding to tie together the various pieces to do it, and actually stopping all spend is likely going to mean nuking storage, not just halting “active” services.
None of them provide anyway to more than probabilistically do “Shut everything down before I exceed, and in a way which prevents exceeding, $X.”
>though you’ll need to do nontrivial custom coding to tie together the various pieces to do it,
Let me guess...hosted on the cloud we're trying to shut down?
Let me guess...hosted on the cloud we're trying to shut down?
> Let me guess...hosted on the cloud we're trying to shut down?
It needs to consume APIs from that cloud, but there is no reason it would need to be hosted there.
It needs to consume APIs from that cloud, but there is no reason it would need to be hosted there.
A spending kill switch can be setup on AWS using AWS Budgets alerts and Lambda but it’s a DIY project, not a built-in feature.
The whole point of a "spending kill switch" is as a backstop when you make a mistake; but if you do it as a "DIY project", what prevents you from making a mistake on it? It has to be a built-in feature.
So what services do you kill? Everything? Including databases and S3?
Ideally tunable, but sure. At least provide that option.
If it can be done DIY, it can be done first party.
If its done first party, someone (realistically, af AWS’ scale, a substantial number of someones) will mess up using it and nuke their account.
Customer service can correct “we screwed up and have a giant bill” easier than “we screwed up and lost all our data”.
So its not going to happen first party.
(It’s also not really possible DIY as a hard certain cutoff at or before a limit, only at an indefinite interval in time and money beyond a limit. So you still have potentially unbounded expense.)
Customer service can correct “we screwed up and have a giant bill” easier than “we screwed up and lost all our data”.
So its not going to happen first party.
(It’s also not really possible DIY as a hard certain cutoff at or before a limit, only at an indefinite interval in time and money beyond a limit. So you still have potentially unbounded expense.)
You can set price alerts to email you on the day when you cross a monthly spending limit, but it is not the simplest thing to figure out.
I run a small server and have a few odds and ends and get emails at $5, $15, $50, $75, and $100. Haven't broken $75 limit yet...
I run a small server and have a few odds and ends and get emails at $5, $15, $50, $75, and $100. Haven't broken $75 limit yet...
The problem is that a billing alert at $10 won't prevent you from accidentally starting up something that will bring you a $1000 bill before you can react to that email - there needs to be a process that cuts off service at some hard limit instead of just sending an email and continuing to spend money.
completely agree. even though it creates a new class of problem, there should be a "cut me off immediately at $XXXX per month" option.
[deleted]
It's not real time billing, so there is no real way to disable service until costs have been determined.
I agree that it seems like a solvable problem, but it doesn't make them money so why should they care?
I agree that it seems like a solvable problem, but it doesn't make them money so why should they care?
I don't understand why there isn't at least a setting that says "turn everything off if I hit $x."
Then just given people a certain grace period to reactivate or get their data out before it's removed.
It wouldn't fix production deployments where you want alarms, not a shutdown, when you hit spending caps, but it would help people on the dev stage to avoid issues like this.
Then just given people a certain grace period to reactivate or get their data out before it's removed.
It wouldn't fix production deployments where you want alarms, not a shutdown, when you hit spending caps, but it would help people on the dev stage to avoid issues like this.
So let’s say they “turn everything off”. Does that include deleting all of your objects in S3? Deleting your database? Deleting your attached disks (AMIs)? Deleting your DNS entries?
If it a personal site I run for kicks and expect to cost less than $100/year, but is suddenly running into thousands? Yes please, delete it all. It's the only way I could sleep at night.
>Does that include deleting all of your objects in S3? Deleting your database? Deleting your attached disks (AMIs)? Deleting your DNS entries?
People bring this every time. You give a grace period and then yes - assuming the user opted into a "turn everything off".
Its not some impossible engineering feat & clouds are full of "out of money data gone" stuff already. e.g.
>If a paid subscription ends or is terminated, Microsoft retains customer data stored in Microsoft 365 in a limited-function account for 90 days to enable the subscriber to extract the data.
People bring this every time. You give a grace period and then yes - assuming the user opted into a "turn everything off".
Its not some impossible engineering feat & clouds are full of "out of money data gone" stuff already. e.g.
>If a paid subscription ends or is terminated, Microsoft retains customer data stored in Microsoft 365 in a limited-function account for 90 days to enable the subscriber to extract the data.
Sure. I'd enable that on dev / testing accounts without hesitation. I don't know why so many people pretend like everyone will be forced into getting resources deleted if there's an option of a hard limit. You can have multiple accounts. They even recommend it.
It makes way more sense for me to build my stuff using a dev/testing account. After that I'll have a good enough understanding of the resources I'm using that it's practical for me to configure more complicated cost controls using a production account.
It's not all or none.
It makes way more sense for me to build my stuff using a dev/testing account. After that I'll have a good enough understanding of the resources I'm using that it's practical for me to configure more complicated cost controls using a production account.
It's not all or none.
becase the billing proces is seprate from the other processes
Can't they just throw an event to EventBridge
That sounds like Amazon taking one of their problems and pushing it onto the users. If the root limitation is that Amazon's billing process is so poor that it can't interact with their other processes, then that should be Amazon's problem to fix. Until and unless Amazon fixes their own problem, Amazon should be eating the cost resulting from Amazon allocating server usage beyond a user-specified maximum billing.
As a personal user, I wish they gave two options to fix this.
Option 1:
When credit balance reaches a certain level (or monthly spend reaches a certain level), initiate a resource stop on every resource that can be stopped without data loss. This would still incur charges for some things like EBS volumes, S3 data, etc, but at least it would slow the bleeding.
Option 2:
I don't care about data loss, just terminate everything when I hit the threshold. This should require a double-opt in and maybe a warning banner in the console UI.
Option 1:
When credit balance reaches a certain level (or monthly spend reaches a certain level), initiate a resource stop on every resource that can be stopped without data loss. This would still incur charges for some things like EBS volumes, S3 data, etc, but at least it would slow the bleeding.
Option 2:
I don't care about data loss, just terminate everything when I hit the threshold. This should require a double-opt in and maybe a warning banner in the console UI.
I think it should be option 2 by default, but allow you to customise later if you want resource stops on only certain resources.
So option 2 by default with option 1 customisable from then on.
So option 2 by default with option 1 customisable from then on.
Author here. I just updated the post with an additional idea for fixing the free tier, suggested by several readers.
It turns out there is a non-widely-known program called AWS Educate Starter Accounts [0], which give no-credit-card access to a limited but useful subset of AWS services. The problem is that you can only get access to these accounts through student affiliation with a participating educational institution like a high school or university.
It might be more feasible to expand this program, say to any applicant who demonstrates some reasonable threshold of non-bot-ness, than to re-engineer the normal free tier.
To any AWS people reading this: I believe this could be a useful step toward solving the free tier problem, and would be happy to be a sounding board.
[0] https://aws.amazon.com/education/awseducate/aws-educate-faqs...
It turns out there is a non-widely-known program called AWS Educate Starter Accounts [0], which give no-credit-card access to a limited but useful subset of AWS services. The problem is that you can only get access to these accounts through student affiliation with a participating educational institution like a high school or university.
It might be more feasible to expand this program, say to any applicant who demonstrates some reasonable threshold of non-bot-ness, than to re-engineer the normal free tier.
To any AWS people reading this: I believe this could be a useful step toward solving the free tier problem, and would be happy to be a sounding board.
[0] https://aws.amazon.com/education/awseducate/aws-educate-faqs...
Arguably discovering surprise bills absolutely should be part of the free tier, how else can you mentally prepare for running something in production on AWS?
If someone told me about a cool new programming language and to teach myself how to use it could either be free or maybe $5000 because infinite loops are expensive, am I going to learn it? Hell no.
Yeah, I'd dabble with this if not for me being forced to sign up with a credit card and them not offering any guarantees on how much they'd bill.
I tried signing up with a prepaid credit card and they refused the card so I moved on. It's setup for massive profits on minor mistakes. The risk on a free tier is like shorting a stock, one bad day and you go bankrupt unless you have a connected twitter account.
Sounds like a new dystopian future that is best not to be part of.
When IBM ruled or Microsoft or AOL things you had one main evil corp. We're in a period where Google, Apple, Amazon, Microsoft, Facebook can appear the hero or villian depending on the day but the sum of the faangs is much greater than the worst evils of the original megacorps. Could you envision forced unlimited billing on a free tier with no ability to limit charges on the account?
Sounds like a new dystopian future that is best not to be part of.
When IBM ruled or Microsoft or AOL things you had one main evil corp. We're in a period where Google, Apple, Amazon, Microsoft, Facebook can appear the hero or villian depending on the day but the sum of the faangs is much greater than the worst evils of the original megacorps. Could you envision forced unlimited billing on a free tier with no ability to limit charges on the account?
When I read about the myriad predatory practices of Amazon I think about who the predator is and the saying "the fish rots from the head." I'm looking forward to the day Jeff Bezos pays the price for being the predator that he is.
He’s already left - you’re going to be waiting a long time.
He's moved from chief executive to executive chairman. In no sense has he 'left'
He’s out of the crosshairs, and there is someone else running the show - they may end up reporting to him, but similar to Bill Gates and ballmer; spotlight isn’t on him anymore and he’s not making the day to day calls anymore.
I have a couple of $20 AWS vouchers from various things. I kind of want to give them to juniors and tell them to go and learn the product. But I won't, because if someone incurs a charge accidentally it'll be on me.
These threads always open a discussion about how hard limits would be unacceptable to some businesses, but the opposite also applies in other scenarios.
These threads always open a discussion about how hard limits would be unacceptable to some businesses, but the opposite also applies in other scenarios.
I took a grad course course 2 years ago on cloud computing and we were showed how to setup a student account. I ran a few machines, shut them down, mostly did the stuff on a local server.
Suddenly I started getting hit with these real small charges each week. I never did figure it out and I certainly didn’t willfully authorize them. I just paid and then I annoyed them to shut off the account. I never could figure out what they were for.
I know many of you would hate this but I would love an option to shutdown everything until the bill is paid option. I bet the description of what I was paying for would have been a lot more helpful if they were losing the business then when they just auto charged!
Suddenly I started getting hit with these real small charges each week. I never did figure it out and I certainly didn’t willfully authorize them. I just paid and then I annoyed them to shut off the account. I never could figure out what they were for.
I know many of you would hate this but I would love an option to shutdown everything until the bill is paid option. I bet the description of what I was paying for would have been a lot more helpful if they were losing the business then when they just auto charged!
AWS was charging me 0.50¢ a month for over a year for an account that was tied to a Google Suite email address for a failed startup I was part of.
I couldn't recover access to the account. They couldn't figure how to give me access. They wouldn't just remove my credit card from the account.
One month the charge became $10 so I called my bank and had them block AWS charges. It seemed the only way to deal with the fear that someday that would creep up and up.
I couldn't recover access to the account. They couldn't figure how to give me access. They wouldn't just remove my credit card from the account.
One month the charge became $10 so I called my bank and had them block AWS charges. It seemed the only way to deal with the fear that someday that would creep up and up.
This appears to be by design. People have been demanding a budget capping service [0] for a decade now. AWS would continue to bill for compromised accounts [1]. AWS "free" offerings can cost a lot more than you think [2].
[0] https://news.ycombinator.com/item?id=3356987
[1] https://news.ycombinator.com/item?id=20045615
[2] https://news.ycombinator.com/item?id=22988647
[0] https://news.ycombinator.com/item?id=3356987
[1] https://news.ycombinator.com/item?id=20045615
[2] https://news.ycombinator.com/item?id=22988647
The thing that always gets me: I _think_ some of this is a solved problem within AWS. I've never used it before, but the AWS Educate "offers students no-cost access to a specified, capped amount of AWS cloud resources without requiring a credit card for payment".
That sounds a lot like what I want a free tier to be. Let me play for a bit, set hard guard rails so I can't accidently spend $100, or much much worse. I mean, I can understand requiring a CC to do this, but that's mostly so I don't spin up a ton of free accounts and also so the friction to going to a paid plan is lower.
It's too late for me, my personal AWS account is well outside of the free tier, but the first time I spun on a large instance to test it out I was so insanely nervous I'd screw up and end up with a huge bill. I can totally see someone else backing out at that point out of fear, but if I knew I'd hit the guardrail before spending money, even if it meant losing an instance, I'd have totally be happier to play around.
That sounds a lot like what I want a free tier to be. Let me play for a bit, set hard guard rails so I can't accidently spend $100, or much much worse. I mean, I can understand requiring a CC to do this, but that's mostly so I don't spin up a ton of free accounts and also so the friction to going to a paid plan is lower.
It's too late for me, my personal AWS account is well outside of the free tier, but the first time I spun on a large instance to test it out I was so insanely nervous I'd screw up and end up with a huge bill. I can totally see someone else backing out at that point out of fear, but if I knew I'd hit the guardrail before spending money, even if it meant losing an instance, I'd have totally be happier to play around.
Lots of companies also gets hacked each month for thousands of dollars because some key to S3 with too many privileges gets leaked.
The entire system is completely sinister. The fact that keys pertaining to S3 has anything do with being able to start hundreds of VM's in different parts of the AWS system or do whatever is bad.
I've seen companies be ruined by this, and it's in no way obvious how stupid their system is. You have to read huge manuals to know how to "only give access to s3" through a key.
Instead of starting with "no access" then adding atomized access you have to understand this extremely complex "json privilege system". Instead of just programming, this is the only allowed IP, the is the only allowed bucket, this is the only allowed service, and my max is 200usd, or something to that effect.
Also the fact that a key can start new services that are billable is almost criminal in my mind when people don't even gen an email when it happens - makes zero sense.
The entire system is completely sinister. The fact that keys pertaining to S3 has anything do with being able to start hundreds of VM's in different parts of the AWS system or do whatever is bad.
I've seen companies be ruined by this, and it's in no way obvious how stupid their system is. You have to read huge manuals to know how to "only give access to s3" through a key.
Instead of starting with "no access" then adding atomized access you have to understand this extremely complex "json privilege system". Instead of just programming, this is the only allowed IP, the is the only allowed bucket, this is the only allowed service, and my max is 200usd, or something to that effect.
Also the fact that a key can start new services that are billable is almost criminal in my mind when people don't even gen an email when it happens - makes zero sense.
> Instead of starting with "no access" then adding atomized access you have to understand this extremely complex "json privilege system". Instead of just programming, this is the only allowed IP, the is the only allowed bucket, this is the only allowed service, ...
What would that look like if it’s not going to be a series of access permissions and filters represented as JSON?
Security is never a simple checkbox and complaints like this about it needing to be simpler need to be backed up with an alternative. I genuinely wonder what alternative there is to the current permissions model.
It’s incredibly expressive and doesn’t take that long to understand. People who cannot master it would likely leave some other side door open anyway.
> ... and my max is 200usd, or something to that effect.
This has been a valid complaint for years. Though to solve it you need to answer what happens to legit resources when your billing cap is reached. Do all your ephemeral serves turn off? Do your EBs volumes all her deleted? Do your S3 objects all disappear?
What would that look like if it’s not going to be a series of access permissions and filters represented as JSON?
Security is never a simple checkbox and complaints like this about it needing to be simpler need to be backed up with an alternative. I genuinely wonder what alternative there is to the current permissions model.
It’s incredibly expressive and doesn’t take that long to understand. People who cannot master it would likely leave some other side door open anyway.
> ... and my max is 200usd, or something to that effect.
This has been a valid complaint for years. Though to solve it you need to answer what happens to legit resources when your billing cap is reached. Do all your ephemeral serves turn off? Do your EBs volumes all her deleted? Do your S3 objects all disappear?
Json is great. Their implementation of it, not so much.
Way to complex only needing to give permission to a single "thing". And much of the naming makes little sense to new yes.
DigitalOceans interface is easier for 90% of people.
But to be honest it's years ago that i worked with AWS, but i remember everything being way more complex than it had to be and i had to use days undestanding their enourmous interface and "usersystem". And as long as this is a problem that pops up again and again, something is very wrong with their system i would say.
If not creating a simpler interface / user system for regular users then at least give some huge warnings that unless you know what you are doing a key with the wrong JSON means access to all funds on the credit card.
Way to complex only needing to give permission to a single "thing". And much of the naming makes little sense to new yes.
DigitalOceans interface is easier for 90% of people.
But to be honest it's years ago that i worked with AWS, but i remember everything being way more complex than it had to be and i had to use days undestanding their enourmous interface and "usersystem". And as long as this is a problem that pops up again and again, something is very wrong with their system i would say.
If not creating a simpler interface / user system for regular users then at least give some huge warnings that unless you know what you are doing a key with the wrong JSON means access to all funds on the credit card.
The feature I most want from AWS is a simple way to create credentials that are only allowed to read from or write to a specific S3 bucket.
The way you do this at the moment genuinely involves copying and pasting JSON policy documents around! It's horrific.
I want this for myself, but more importantly I want it for users of software that I write. I would love to be able to build something that stores a user's data in an S3 bucket that they own (and are billed for directly) - but it's currently just too difficult to talk them through setting up the bucket and creating the right credentials for it.
The way you do this at the moment genuinely involves copying and pasting JSON policy documents around! It's horrific.
I want this for myself, but more importantly I want it for users of software that I write. I would love to be able to build something that stores a user's data in an S3 bucket that they own (and are billed for directly) - but it's currently just too difficult to talk them through setting up the bucket and creating the right credentials for it.
They simply have to be in separate AWS accounts for this to work. To that end, you can provide them with a CloudFormation template that deploys a stack with the necessary configuration.
> You have to read huge manuals to know how to "only give access to s3" through a key
I have to disagree with this. It really is pretty straight forward.[0]
[0] - https://objectivefs.com/howto/how-to-restrict-s3-bucket-poli...
I have to disagree with this. It really is pretty straight forward.[0]
[0] - https://objectivefs.com/howto/how-to-restrict-s3-bucket-poli...
Anyone who asks you for a credit card or other means to bill you for a "free" service intends to steal money from you without your consent
Its not stealing. You are using their services for free now which can / will be charged later.
I've stayed away from AWS for these reasons. Instead I use systems on top of AWS, like Vercel etc.
Is it ironic that Amazon's mantra is to be "Customer obsessed" yet AWS is so magnificently confusing for anyone not doing it full-time?
As a designer I've used plenty of Digital Ocean, Vercel, Cloudflare Workers and other static hosts without a problem... I've never been able to figure out how to even start on AWS, and all these horror stories constantly make sure I stay away
Is it ironic that Amazon's mantra is to be "Customer obsessed" yet AWS is so magnificently confusing for anyone not doing it full-time?
As a designer I've used plenty of Digital Ocean, Vercel, Cloudflare Workers and other static hosts without a problem... I've never been able to figure out how to even start on AWS, and all these horror stories constantly make sure I stay away
I'm so miserable that I finally got around to embracing open ecosystems after almost a decade of being Microsoft-poisoned and now everyone is all in on some baffling walled garden with abysmal UX and about 300 different services that you need a damn NASA PhD to understand.
Like the number of times I find myself in some random service/feature/part of AWS like "do we use one of these? how would I even tell?".
Like the number of times I find myself in some random service/feature/part of AWS like "do we use one of these? how would I even tell?".
I know AWS well. In fact I work at AWS in ProServe. But, I have run up a “bill” on my internal personal accounts before.
If I were ever going to do a personal project that wasn’t appropriate for serverless, I would just use LightSail.
https://aws.amazon.com/lightsail/
If I were ever going to do a personal project that wasn’t appropriate for serverless, I would just use LightSail.
https://aws.amazon.com/lightsail/
[deleted]
The crazy thing is, this obvious (to anyone experienced in business, at least) danger must be costing AWS money. I personally know of small businesses who have entirely avoided AWS even though it might have been a good fit for their needs in most respects, entirely because of concerns about the opacity of the billing and the inability to add safeguards in case something goes wrong. Some of those businesses are no longer small, either. AWS does have a reputation, at least among those more familiar with cloud services, for being reasonable about unexpected charges and probably putting something right if it was obviously not intended. But in that case, they aren't even pulling some sort of dark pattern scam to make more money here, and the lack of last resort safety features makes even less sense...
I am a student in the same age. When I wanted to start a hobby project I looked into the AWS free tier too. It is so unclear how the end costs come together that I decided to just leave AWS alone.
I rented a small cloud instance from HETZNER for ~ 3€/month and host everything myself. If I need a DBaaS I go with the MongoDB Atlas or ElephantSQL free tier.
I prefer constantly paying 3€, over a free account that breaks my neck if I miss something.
I rented a small cloud instance from HETZNER for ~ 3€/month and host everything myself. If I need a DBaaS I go with the MongoDB Atlas or ElephantSQL free tier.
I prefer constantly paying 3€, over a free account that breaks my neck if I miss something.
The exact thing happened to me one year ago, I was 17 and using my dad's credit card to test out Lambda and SageMaker, I had assured him it won't cost anything since I'd be using the free tier.
However, my application instance somehow kept running ( was a total noob to AWS ) and I got charged over $300 the next month when I got a monthly report in an email. I panicked and literally just deleted my account. Yes, I just nuked my account like the article mentions. AWS never reached out after that.
However, my application instance somehow kept running ( was a total noob to AWS ) and I got charged over $300 the next month when I got a monthly report in an email. I panicked and literally just deleted my account. Yes, I just nuked my account like the article mentions. AWS never reached out after that.
Hey I did the same when I first used it. Raked up 9000 Rupees. Cancelled the card and called them, customer care said no worries and the bill was gone.
I'm in agreement with the cloud providers over those of you wanting a hard shutdown.
Businesses, the entities that are paying the most money to AWS, will NOT want a hard shutdown. When you generate revenue off of your SaaS service maybe you'll understand.
No, I won't be pushing my TAM to enable a forced shutdown due to budget metrics.
Not to mention, how does AWS decide what to shutdown and what to delete? It's not like it's only running resources that cost money, what about all my data that's stored???
Businesses, the entities that are paying the most money to AWS, will NOT want a hard shutdown. When you generate revenue off of your SaaS service maybe you'll understand.
No, I won't be pushing my TAM to enable a forced shutdown due to budget metrics.
Not to mention, how does AWS decide what to shutdown and what to delete? It's not like it's only running resources that cost money, what about all my data that's stored???
> Businesses, the entities that are paying the most money to AWS, will NOT want a hard shutdown.
I don't think anyone is arguing that any account MUST take advantage of hard shutdowns. Only that it should be an option for those who truly do want it.
> How does AWS decide what to shutdown and what to delete?
That's a much more interesting question. It tends to be services, rather than storage, that are the root cause of these outlandishly large surprise bills. But if, by chance, it's S3 that's running up the bill, what would a hard shutdown mean? Which data does Amazon delete first to get the account down to its hard limit?
I don't think anyone is arguing that any account MUST take advantage of hard shutdowns. Only that it should be an option for those who truly do want it.
> How does AWS decide what to shutdown and what to delete?
That's a much more interesting question. It tends to be services, rather than storage, that are the root cause of these outlandishly large surprise bills. But if, by chance, it's S3 that's running up the bill, what would a hard shutdown mean? Which data does Amazon delete first to get the account down to its hard limit?
I had an AWS account for one of my businesses, and decided to bring it in-house about 2 months ago.
went through the account deletion, and got yet charge/another bill today - account is deleted, so can't log in to see why I'm being charged.
hopefully their support will help out, but not holding my breath.
editing to add: billing was for db backups, I terminated the db, and no clue how to remove the automatic backups it made. of course I can't log in to look any further.
went through the account deletion, and got yet charge/another bill today - account is deleted, so can't log in to see why I'm being charged.
hopefully their support will help out, but not holding my breath.
editing to add: billing was for db backups, I terminated the db, and no clue how to remove the automatic backups it made. of course I can't log in to look any further.
Without a receipt to show what's being charged for I'd just call the bank and ask for a chargeback.
> It was ‘just’ $200, that’s not the end of the world
A $200 charge would have been devastating to me when I was in college. As in, I may have to beg for food devastating.
A $200 charge would have been devastating to me when I was in college. As in, I may have to beg for food devastating.
The issue here is really Billing... AWS Budgets allows you to set a budget as far down as $1 USD... however because billing is done piecemeal, it is possible to have up to 24 hour delay for charges...
You can spend $500,000 on AWS in 24 hours, it's not difficult, it's not easy but you totally can blow past your "budget" because it's not a hard cap.
You can spend $500,000 on AWS in 24 hours, it's not difficult, it's not easy but you totally can blow past your "budget" because it's not a hard cap.
A major problem is that there's no easy way to see all resources in a single view for any cloud.
Azure comes closest in the dashboard but still misses some items. GCP has an Asset Inventory page in the Security section for an organization. AWS can use the Tag Editor to browse all the regions. It's notable that none of them have a single clear page though.
Azure comes closest in the dashboard but still misses some items. GCP has an Asset Inventory page in the Security section for an organization. AWS can use the Tag Editor to browse all the regions. It's notable that none of them have a single clear page though.
Oh come on. We all know that the 'accidental revenue' from the way Free Tier is set up probably makes up a cool 2 million or more annually. Plenty to justify its continued abuse of naive students. Why would they walk away from that cash? The only people they're pissing off is people who aren't using AWS anyway.
AWS annual revenue is $50+ billion. (https://www.zdnet.com/article/aws-run-rate-hits-54-billion-a...)
Anything that they can get from "naive students" and developers who don't notice small recurring charges is so utterly insignificant that it can't justify any decision whatsoever. If the public relations aspects of it cause even a 0.01% change in AWS growth, that's already $5 million of lost revenue; if you can assure developers worldwide that it's not so risky to try and adopt AWS a bit more and get a 1% extra growth, that would be worth $500 million and justify walking away from all kinds of irrelevantly small cash flows.
Anything that they can get from "naive students" and developers who don't notice small recurring charges is so utterly insignificant that it can't justify any decision whatsoever. If the public relations aspects of it cause even a 0.01% change in AWS growth, that's already $5 million of lost revenue; if you can assure developers worldwide that it's not so risky to try and adopt AWS a bit more and get a 1% extra growth, that would be worth $500 million and justify walking away from all kinds of irrelevantly small cash flows.
If bad PR affected their business, they might have something to consider. But this will never affect their growth. They're not going to lose a single solitary sale that isn't already accounted for by their standard profit model. The only bad PR that would affect their business is if their actual reputation as a service provider were tarnished. Charging people money for a service they used does not tarnish AWS's reputation. It's their whole business model.
When you stay in a hotel room longer than you're supposed to and they charge you for another half day, or you eat food from the mini fridge, or buy a pay-per-view movie, etc, you could claim ignorance, and create stories about how terrible it is that people get charged for these things unknowingly. But Holiday Inn's bookings are still not going to take a noticeable hit.
Moreover, they don't even need to offer a Free Tier to get people to use AWS. They're AWS. They're the gold standard. It's like saying IBM would need a Free Tier (back when IBM wasn't trash). It wasn't a question of whether you should use IBM, it was whether you could afford it.
When you stay in a hotel room longer than you're supposed to and they charge you for another half day, or you eat food from the mini fridge, or buy a pay-per-view movie, etc, you could claim ignorance, and create stories about how terrible it is that people get charged for these things unknowingly. But Holiday Inn's bookings are still not going to take a noticeable hit.
Moreover, they don't even need to offer a Free Tier to get people to use AWS. They're AWS. They're the gold standard. It's like saying IBM would need a Free Tier (back when IBM wasn't trash). It wasn't a question of whether you should use IBM, it was whether you could afford it.
I’ve heard of plenty of people accidentally blasting past the free tier, or past dev credits, or other ways where they got burned by a surprise bill. 100% of those people have gotten a refund from AWS support by filing a ticket.
AWS pretty clearly isn’t raking in money on this. Even at your imaginary 2 million a year, that’s a drop in the bucket for AWS, let alone Amazon.
AWS pretty clearly isn’t raking in money on this. Even at your imaginary 2 million a year, that’s a drop in the bucket for AWS, let alone Amazon.
It's got to be Hanlon's Razor here, I doubt AWS is twirling their mustache and cackling maniacally. It's probably a difficult problem to solve and not easily attachable to a profit center or something, so nobody feels empowered to actually fix it.
I'm sure some crusader within AWS could get it done if they tried hard enough and collected the data to show the negative side effects and how they effect AWS.
An article like this one is hopefully going to get someone within AWS to do just that.
I'm sure some crusader within AWS could get it done if they tried hard enough and collected the data to show the negative side effects and how they effect AWS.
An article like this one is hopefully going to get someone within AWS to do just that.
I think this is actually the failure mode for companies like AWS: they may earnestly be so large/complex that no well-intentioned crusader can push through an improvement like this.
My gut is that this is actually the cause of most of the “evil” things that “large” companies do, where eventually they get so big and/or complicated that organizing an organic positive change isn’t plausible.
My gut is that this is actually the cause of most of the “evil” things that “large” companies do, where eventually they get so big and/or complicated that organizing an organic positive change isn’t plausible.
One person isn’t going to come remotely close to fixing this. I work for AWS/Azure/GCP and can tell you first hand that billing is an insanely hard problem to solve. It’s all done in batch by the service itself and pushed to a centralized service once all the hard work has been done. Each service has a small, underfunded team that handles billing and their last thought is some random guy spending a few grand on his personal account. Every ounce of their energy is going to reducing their own COGs and reacting to the asks of the biggest customers.
I did that, I didn't get a refund. Lost about $180.
I have a friend who I know didnt get a refund.
There is someone elsewhere in this post who didnt get a refund.
I don't know what the refund rate is, but it isn't 100%.
I have a friend who I know didnt get a refund.
There is someone elsewhere in this post who didnt get a refund.
I don't know what the refund rate is, but it isn't 100%.
AWS bets on everyone who doesn't notice small recurring charges, not those who blow up their budget past the point of sanity.
That seems even more implausible. Do charges small enough to not be noticed stand a chance of being noteworthy for AWS?
This ain't it.
Obfuscated billing helps Amazon with $350/hour engineers, where an engineer figuring out where $10k is going each year isn't a top priority. Been there, done that.
Obfuscated billing is definitely not there to hit free tier or poor folks. AWS has an awesome reputation for just about everything, and it isn't worth the reputation hit. Free tier is there so that:
* the student in college will pick AWS in their first job
* the random engineer will prototype something on the weekend, and Amazon gets millions of business
* the random pre-funding startup starts building on AWS, and if it goes big, so does the account
And it works. Amazon has made millions based on things I've built on free tier. AWS' problem is that my ONE YEAR free tier ran out probably around a decade ago, and I've long since moved on from what I was doing then. If AWS were to continue to provide me a free tier, the same thing would have happened a few times since.
Obfuscated billing helps Amazon with $350/hour engineers, where an engineer figuring out where $10k is going each year isn't a top priority. Been there, done that.
Obfuscated billing is definitely not there to hit free tier or poor folks. AWS has an awesome reputation for just about everything, and it isn't worth the reputation hit. Free tier is there so that:
* the student in college will pick AWS in their first job
* the random engineer will prototype something on the weekend, and Amazon gets millions of business
* the random pre-funding startup starts building on AWS, and if it goes big, so does the account
And it works. Amazon has made millions based on things I've built on free tier. AWS' problem is that my ONE YEAR free tier ran out probably around a decade ago, and I've long since moved on from what I was doing then. If AWS were to continue to provide me a free tier, the same thing would have happened a few times since.
I've gotten thousands of euro of 'free' Amazon cloud credits over the years, and would love to have used tried it out for cloud computing/GPU stuff, but the opaqueness of pricing means I won't touch it (except for S3 backups) - I just don't trust myself enough to not mess up.
Last month I woke up to an amazon alert that our usually $5000 monthly bill was $125 000.
I ran to my computer too fast to have time for a panic attack. Turns out my company bought some services through AWS and it was all kosher.
I wish it wasn't possible to rack up such a bill by accident so I don't have to worry.
I ran to my computer too fast to have time for a panic attack. Turns out my company bought some services through AWS and it was all kosher.
I wish it wasn't possible to rack up such a bill by accident so I don't have to worry.
Just speaking from personal anecdotes, every time I’ve accidentally been charged for something on AWS that I didn’t intend, their support team has refunded me without much hassle. Things may have changed in recent times but I’ve found them to be pretty reasonable about it.
AWS is not the cloud. There are others. Please try something else. As someone with extensive professional experience with it day in and out, it is overrated.
Also, the cloud is someone else's computer. It is either AWS or GCP or something else. Move to what works, for you.
Also, the cloud is someone else's computer. It is either AWS or GCP or something else. Move to what works, for you.
At one point I tried using AWS trial for an app, didn't work out, cancelled the account (or so I thought) only to find 2 years later some charges to my bank of $2k and when I spoke to AWS support, there was another $4k accruing for this billing period.
Took ages to sort out, while they eventually did refund the amount, I was still out of pocket exchange fees (X2) the exchange rate also meant I was out more money. In total I think having a compromised yet closed AWS free trial cost me $350 over a 6week period some 2 years after closing it.
We were inexperienced, got picked up by a tiny VC and funded for a few thousand. We applied and got accepted into the Digital Ocean Hatch program, they threw $100,000 in credits at us to help us succeed.
We tried to exhaust the supply—one day the credits vanished. I was getting married that month, and it took that long to accrue a life-changing amount of real dollar debt. I sent a quick email asking DO to extend the credits. The last email I got from them was that "we expect you pay the full amount".
We tried to exhaust the supply—one day the credits vanished. I was getting married that month, and it took that long to accrue a life-changing amount of real dollar debt. I sent a quick email asking DO to extend the credits. The last email I got from them was that "we expect you pay the full amount".
[deleted]
Yes, there should be easier ways for learners to experiment with AWS without risk. In general, people should be made aware of how much something could cost them before they can agree with it. We have regulations towards this end when it comes to investments. Huge unexpected bills used to be a big problem with phone companies, and it took a lot of effort to move past it. I hope the cloud industry can make progress in this direction.
What I'm going to say next should not be taken to detract from that in any way.
(And I know you don't want to hear it, and I could just save everyone some trouble and not say it. But I'm going to say it.)
When I read this:
> please help I made a ticket and called support but i really need to make sure this is dead please Im 20 i really dont have $200 for them please help
I honestly did a double take when I got to the "20". If you'd put a blank in there and asked me to fill it, I would have said "13". I'm still not 100% sure that the blogger didn't alter the age to obfuscate personal details.
I don't mean to single out this person as especially immature, quite the opposite: the interesting thing is that they assume that a 20 year old is obviously a smol bean who could not possibly be expected to figure out AWS billing or come up with $200, _and we agree_. The notion didn't stand out as surprising to the blogger, or to any of the other commenters on HN so far.
I'm not saying they're wrong! I just find this a remarkable signal of how far we, as a culture, have gone in extending childhood well into college age.
What I'm going to say next should not be taken to detract from that in any way.
(And I know you don't want to hear it, and I could just save everyone some trouble and not say it. But I'm going to say it.)
When I read this:
> please help I made a ticket and called support but i really need to make sure this is dead please Im 20 i really dont have $200 for them please help
I honestly did a double take when I got to the "20". If you'd put a blank in there and asked me to fill it, I would have said "13". I'm still not 100% sure that the blogger didn't alter the age to obfuscate personal details.
I don't mean to single out this person as especially immature, quite the opposite: the interesting thing is that they assume that a 20 year old is obviously a smol bean who could not possibly be expected to figure out AWS billing or come up with $200, _and we agree_. The notion didn't stand out as surprising to the blogger, or to any of the other commenters on HN so far.
I'm not saying they're wrong! I just find this a remarkable signal of how far we, as a culture, have gone in extending childhood well into college age.
This is the reason I closed my aws account and gave up on learning their services. Intentional cost obfuscation means I have no interest in doing business with someone
I really recommend using something like Linode for cheap throwaway, might be a bit more expensive (i.e: 5 per month), but the surprise factor disappears..
You'd have to be doing something extremely small to make a $5 Linode more expensive. AWS might be cheaper for a static site, but as soon as you involve a database you're probably paying more.
AWS is not cheap. A lot of people seem get bitten by this.
AWS is not cheap. A lot of people seem get bitten by this.
I agree with most of the points brought there except for:
> I’ve personally got a dormant AWS account that’s charging me cents every month, and I bet you do too.
Hummm no? And if I have a charge line and I can try to chase it. It's not like that cent is nameless, it has a name and it has a way of figuring it out?
I mean, it can be not so obvious but it's not like it's totally opaque neither
> I’ve personally got a dormant AWS account that’s charging me cents every month, and I bet you do too.
Hummm no? And if I have a charge line and I can try to chase it. It's not like that cent is nameless, it has a name and it has a way of figuring it out?
I mean, it can be not so obvious but it's not like it's totally opaque neither
20+ years ago I was offered a free 2 or 3 month subscription to compuserve. I was interested in it, as one could dial in as PPP account for internet access. So I configured my linux box to dial in and if the connection dropped, to redial.
After the 2nd month I get a huge multi hundred dollar bill. They claimed over the the 2 months I had used 2k-3k hours of time an their free months were limited to 750 hours a month of usage.
As I pointed out, that should be enough, as 24*31 = 744, so it should be impossible for me to use more than 750 hours a month.
They claimed that I must have been dialing in from multiple locations. I denied that and said they should have records of where I was calling in from. It took weeks for them to "forgive" the debt, without acknowledging that their billing was broken.
I always wondered how many other users got hurt because of this (my guess is not that many, as relatively few people were keeping their connection alive 24/7 via compuserve back in the 90s)
After the 2nd month I get a huge multi hundred dollar bill. They claimed over the the 2 months I had used 2k-3k hours of time an their free months were limited to 750 hours a month of usage.
As I pointed out, that should be enough, as 24*31 = 744, so it should be impossible for me to use more than 750 hours a month.
They claimed that I must have been dialing in from multiple locations. I denied that and said they should have records of where I was calling in from. It took weeks for them to "forgive" the debt, without acknowledging that their billing was broken.
I always wondered how many other users got hurt because of this (my guess is not that many, as relatively few people were keeping their connection alive 24/7 via compuserve back in the 90s)
[deleted]
This is precisely the reason I have avoided AWS as a student and to this day I prefer providers that have clear billing like DO.
I don’t understand why anyone would use AWS when there is a risk of being charged stupid amounts of money if you screw something up. Are there advantages to AWS that I’m not seeing?
I don’t understand why anyone would use AWS when there is a risk of being charged stupid amounts of money if you screw something up. Are there advantages to AWS that I’m not seeing?
I refuse to let my students touch AWS directly exactly because of this infinite billing problem.
My current workaround is finding some SaaS provider that is willing to take the risk on free accounts and act as a middleman.
Currently using aiven.io - downside: this covers only a small subset of services that aws offers.
My current workaround is finding some SaaS provider that is willing to take the risk on free accounts and act as a middleman.
Currently using aiven.io - downside: this covers only a small subset of services that aws offers.
It's absolutely unacceptable that AWS hasn't fixed this, but in the meantime, would using a privacy.com temporary credit card with a ~$20 limit when signing up help? I'm unsure if those kinds of cards can be detected/blocked by AWS, or if they do.
Not paying the bill doesn't make the bill go away, you'd still owe Amazon the money unless support waived it.
I want to point out that I just tried to access my AWS Educate Account today to explore the panels and found:
"ALERT-1: Session time behavior and instance types in your Starter Account will be changing on May 11th, 2021. After this date: 1. When your session ends, your resources will be “stopped.” You will be required to re-start your resources when you start a new session. 2. Updates will be made to available instance types. We recommend you to complete currently running work in your Starter Account by May 10th, 2021 as work using instance types that are no longer supported will be lost after that date."
Im still thinking about what can I do with 3 hours (duration of a session) of EC2 computing power...
"ALERT-1: Session time behavior and instance types in your Starter Account will be changing on May 11th, 2021. After this date: 1. When your session ends, your resources will be “stopped.” You will be required to re-start your resources when you start a new session. 2. Updates will be made to available instance types. We recommend you to complete currently running work in your Starter Account by May 10th, 2021 as work using instance types that are no longer supported will be lost after that date."
Im still thinking about what can I do with 3 hours (duration of a session) of EC2 computing power...
I had an account with AWS about 6 years for a small prototype and I closed it after using it for 1 month. However for some reason I got an email last November about a change to some certificates for either S3 or Cloudfront because my account used one of those services in the last 6 months.
I don't know if this is a mistake on their part, but I haven't been charged in the last 6 years or gotten any emails before that. But it's still worrying because the account is closed and I have no way through standard support to know why they think my account was using their services
I don't know if this is a mistake on their part, but I haven't been charged in the last 6 years or gotten any emails before that. But it's still worrying because the account is closed and I have no way through standard support to know why they think my account was using their services
As a student trying out the AWS platform (I'm not even a devops/backend/anything related to it) I found it painful to use when compared to other providers. They keep building new products that build on existing products such as Elastic Beanstalk that assigns a LOT of resources you don't even need to host a nodejs app, etc. Even reaching the billing panel was painful. You have to go through at least 3 screens just to get to the "billing" screen. And then you have to browse inside of that to reach your bills. It's disgusting.
Another solution to this problem is the availability of disposable credit cards. It would be ideal that credit card numbers could just be created from scratch, as needed, and turned off whenever the use was done.
We had promises of this as ecommerce took off, to help avoid fraud. But it seems these don't get widespread adoption, probably because purchases are easy to get chargebacks for (in general).
A hard limit for cloud purchases is surely needed. But because the cloud providers aren't giving us this, I'm wondering if a solution like disposable cards could?
We had promises of this as ecommerce took off, to help avoid fraud. But it seems these don't get widespread adoption, probably because purchases are easy to get chargebacks for (in general).
A hard limit for cloud purchases is surely needed. But because the cloud providers aren't giving us this, I'm wondering if a solution like disposable cards could?
1. Does aws even accept prepaid cards? many providers don't dye to fraud/abuse concerns
2. Your payment method refusing to accept charges doesn't mean you're still not on the hook for them. Technically they can still send your bill to collections and wreck your credit report. They probably don't do this, but it's still a bad bandaid solution
2. Your payment method refusing to accept charges doesn't mean you're still not on the hook for them. Technically they can still send your bill to collections and wreck your credit report. They probably don't do this, but it's still a bad bandaid solution
Good thoughts/questions, thanks.
1. Not sure, but I think that's the problem I'm asking about. Because prepaid or auto-generated cards are not mainstream, big vendors don't need to accept them.
2. Agreed, but at least this puts one more layer of control back to the owner. AWS doesn't have my birthdate or social security info (US), so it would be harder (not impossible) for them to "wreck your credit report". At least this approach puts an extra barrier in the equation.
1. Not sure, but I think that's the problem I'm asking about. Because prepaid or auto-generated cards are not mainstream, big vendors don't need to accept them.
2. Agreed, but at least this puts one more layer of control back to the owner. AWS doesn't have my birthdate or social security info (US), so it would be harder (not impossible) for them to "wreck your credit report". At least this approach puts an extra barrier in the equation.
I'm Co-Founder of http://vantage.sh/ and we allow you to connect your AWS account(s) and we'll monitor cloud spend on your behalf.
We send out regular cost report emails to try and help folks avoid situations like this and have some future plans around anomaly detection to try and help out in advance of things like this happening.
I strongly encourage folks to sign up and let us monitor things on your behalf. We have a free tier that likely covers a lot of personal users here.
We send out regular cost report emails to try and help folks avoid situations like this and have some future plans around anomaly detection to try and help out in advance of things like this happening.
I strongly encourage folks to sign up and let us monitor things on your behalf. We have a free tier that likely covers a lot of personal users here.
Is it essentially read only / limited in some fashion by IAM? Wouldn't really want more things connected that open up more threat surface
This is the problem with usage based pricing. It is very lucrative pricing strategy because it works for lower end of market as well as enterprise users. One way to resolve this is to offer fixed tier based pricing for customers who may be more sensitive to prices and usage based pricing for enterprise clients.
For example.
Starter ($5/mo) - Use up to 100,000 API calls each month.
Growth ($30/mo) - Use up to 1,000,000 API calls each month.
Enterprise ($149/mo) - Use up to 10,000,000 API calls each month. After that, $10 for every 1 million calls.
For example.
Starter ($5/mo) - Use up to 100,000 API calls each month.
Growth ($30/mo) - Use up to 1,000,000 API calls each month.
Enterprise ($149/mo) - Use up to 10,000,000 API calls each month. After that, $10 for every 1 million calls.
AWS represents over 60% of the net profit of the entire Amazon empire across all its divisions and acquisitions. If they truly helped the customer to waste less money, it would have a massive impact on everyone who has stock, from sovereign funds to junior engineers. They have a vested interest in dinging your sandbox account $1/month for storing secrets you haven’t used in 2 years. I don’t see this changing anytime soon, especially when the competition is no better.
We would definitely spend more money on AWS services if our company could feel more secure in understanding which project those costs were associated with and were able to track the costs without spending multiple hours.
I don't get why AWS doesn't offer preloading credit. Surely charging your card 1 cent a month isn't worth the time.
Whereas if they allowed customers to load on an amount, even a direct debit, their card processing fees would go down and customers would just get cut off when they run out of credit.
I suppose a counter argument to this is that it's hard for AWS to keep a constant eye on how much a service is costing. In which case we're back to the same argument as spending limits.
Whereas if they allowed customers to load on an amount, even a direct debit, their card processing fees would go down and customers would just get cut off when they run out of credit.
I suppose a counter argument to this is that it's hard for AWS to keep a constant eye on how much a service is costing. In which case we're back to the same argument as spending limits.
When you realise that their whole business model is around extracting as much $$$ from users as they can, you should be entirely unsurprised by how AWS and other cloud providers behave. My cynical side is inclined to say "see, this is what you get for using cloud computing". Cloud providers are the very definition of nickel-and-diming. I still remember the amusement of finding out that AWS has Cost Management services, which themselves have a cost.
I use aws relatively often but for fairly small scale stuff, selfhosting/homelab. I'm very scared of all these horror stories.
I've set a budget and an alert for the budget. Is that enough?
I can see the budget being overdrawn for the time it takes me to react to the alert but I see no way to actually shut down servies.
It would probably be possible to script by monitoring the API and then issuing aws commands to shut things down. But it would be a huge project, someone should get on that.
I've set a budget and an alert for the budget. Is that enough?
I can see the budget being overdrawn for the time it takes me to react to the alert but I see no way to actually shut down servies.
It would probably be possible to script by monitoring the API and then issuing aws commands to shut things down. But it would be a huge project, someone should get on that.
Spending limits should be a legal requirement and if company cannot or doesn't want to implement it, then they shouldn't be in business. Simple as that.
This doesn't help those that have already been hurt, but when you're using any cloud provider for personal use make sure to set budgets and alerts for them. It's quite easy to set these up and they can save you a headache later on. Truth be told they should all be asking you what you expect to spend when you set up an account and alert you by default but just remember these cloud providers are not your friends.
[deleted]
This. 100% this.
I have had issues with students doing the same thing, and I was being charged for dormant projects that I thought I'd deleted. AWS is a nightmare of an ecosystem to navigate.
The AWS Educate Starter Account is almost useless - especially if the student has to submit their project for external review - because it doesn't allow publicly accessible S3 buckets and has rotating credentials.
I have had issues with students doing the same thing, and I was being charged for dormant projects that I thought I'd deleted. AWS is a nightmare of an ecosystem to navigate.
The AWS Educate Starter Account is almost useless - especially if the student has to submit their project for external review - because it doesn't allow publicly accessible S3 buckets and has rotating credentials.
> ...an updated free tier that treats “personal learning” AWS accounts differently from “new corporate” accounts, and sets hard billing limits that you can’t exceed.
Honestly, this is needed for corporate accounts as well. Not all companies are FAANG scale behemoths who can shrug off an unexpected charge.
For a scrappy startup in India, an unexpected $5000 bill would be an existential threat.
Honestly, this is needed for corporate accounts as well. Not all companies are FAANG scale behemoths who can shrug off an unexpected charge.
For a scrappy startup in India, an unexpected $5000 bill would be an existential threat.
A scrappy startup in India is not AWS' ICP.
A scrappy startup in <any country> could see an unexpected $5k bill as business-ending.
It's so absurdly easy to run up massive bills, and there's few controls, and practically useless alerting - so you can't rely on that.
Someone spun up a 3 node m5.xlarge RDS Cluster with SQL Server Standard? I hope you notice quickly, 'cause that's going to be >$5k/month.
They don't show a minimum monthly cost for any of these things, either.
No, you're assumed to be okay with unlimited charges, and are left to your own devices to figure it out.
It's so absurdly easy to run up massive bills, and there's few controls, and practically useless alerting - so you can't rely on that.
Someone spun up a 3 node m5.xlarge RDS Cluster with SQL Server Standard? I hope you notice quickly, 'cause that's going to be >$5k/month.
They don't show a minimum monthly cost for any of these things, either.
No, you're assumed to be okay with unlimited charges, and are left to your own devices to figure it out.
Not sure why, but they courted our company right from the beginning: got in touch with us within a month of us forming up, gave us a generous free startup credit, even invited us over to their Bangalore office and helped us with how we can optimize our infra.
Thankfully, the company reached profitability so we pay them rather hefty bills now. But I guess I agree with you in that that's not a typical startup's trajectory, and their ROI from the aforementioned outreach is likely relatively low on average so am wondering now why they went out of the way to help us.
Thankfully, the company reached profitability so we pay them rather hefty bills now. But I guess I agree with you in that that's not a typical startup's trajectory, and their ROI from the aforementioned outreach is likely relatively low on average so am wondering now why they went out of the way to help us.
Does anyone else use the AWS free tier for a year, and then when the free year ends, signup for a new account and start another free tier account? It's a good technique if you live frugally.
The only caveat being you have to migrate everything to new servers, which can get messy if you don't practice how to do that efficiently.
The only caveat being you have to migrate everything to new servers, which can get messy if you don't practice how to do that efficiently.
It's the AOL free trial business model. Give 'em a freebie, but take down their credit card info so you can start charging them the moment they exceed the limits of "free". And make it really hard to cancel.
As for whether Amazon will listen, Upton Sinclair blah blah not understanding it.
As for whether Amazon will listen, Upton Sinclair blah blah not understanding it.
Can anyone recommend a cloud provider that's suitable for learning (the opposite of AWS in this regard)?
Plain old Linux:
Free: GCP
Paid (but cheap): Linode, DigitalOcean
—- Machine Learning:
Free: Google CoLab, Paperspace Gradient
Paid (but [relatively] cheap): JarvisLabs, DataCrunch
—- Databases and user auth:
Free: Firebase spark
Paid (but cheap, remember to set billing alerts!): Firebase blaze
Paid (but cheap): Linode, DigitalOcean
—- Machine Learning:
Free: Google CoLab, Paperspace Gradient
Paid (but [relatively] cheap): JarvisLabs, DataCrunch
—- Databases and user auth:
Free: Firebase spark
Paid (but cheap, remember to set billing alerts!): Firebase blaze
One hack I've learned once I got burned by similar service is to always use virtual debit card with a spending limit if your bank supports one. I have a virtual card on Revolut dedicated to small subscriptions that freezes at 100$/month to prevent this sort of fraud.
is there an IAC template somewhere that resets an aws account to 0 and deletes everything? I know this articles point is this shouldn't be possible, but as long as it is, there should be a simple measure, that could be linked in tutorials for stopping ongoing charges.
Another option:
https://github.com/gruntwork-io/cloud-nuke
https://github.com/gruntwork-io/cloud-nuke
I found this really hard to use. Asking for configuration credentials that I had no idea how to retrieve
unfortunately for most cloud providers, you have to read through pricing very carefully and be even more careful with what you deploy. an infinite loop could lead to $$$$ in charges and billing alerts just alert you, they don't actually shut anything off
I am very happy this thread came up, because I'm looking at cloud providers to experiement with.
So many comments about charges that show up despite resources removed or even deleted accounts, tells me that AWS can not be trusted to present me the full picture of costs.
So many comments about charges that show up despite resources removed or even deleted accounts, tells me that AWS can not be trusted to present me the full picture of costs.
Scary, I'd have similar fears about using Firebase/Firestore in a production app.
I had a similar experience, with costs of $100 incurred over a four-month period. All it took was on email to AWS support, and they refunded the entire thing.
I was impressed, but still have been really trying to keep off AWS just to not have a mistaken repeat.
I was impressed, but still have been really trying to keep off AWS just to not have a mistaken repeat.
I think it's best to just refer to it in a way that makes sense
Because that would imply it's not working the way Amazon wants
And Amazon is not stupid
Not-actually-Free Tier
Not-actually-a-Tier Tier
Because that would imply it's not working the way Amazon wants
And Amazon is not stupid
"There's money in confusion"
- Rulan Clark
That quote is what I think about when I see cloud hosting pricing models. Rulan was the late co-founder of Clark's Pine Factory in Northern Utah and my first boss in the 1990's.
That quote is what I think about when I see cloud hosting pricing models. Rulan was the late co-founder of Clark's Pine Factory in Northern Utah and my first boss in the 1990's.
Part of the problem is that AWS billing is soo slow to update.
This doesn't only hit budgetting... trying to get all the costs to go the proper billing tags (and verify that you found them all) also requires test cycles of up to 2 days.
This doesn't only hit budgetting... trying to get all the costs to go the proper billing tags (and verify that you found them all) also requires test cycles of up to 2 days.
[deleted]
For accounts you want to put spend limits on always use a pre-paid card or card with spend limits https://privacy.com/.
Privacy.com doesn't believe anyone would want their service if they have a land-line, and not a mobile. "Give us your phone number", okay, "we sent a confirmation SMS" ... hahahahaha.
Yes you can probably dodge that by going with a SMS enabled VoIP provider.
Yes you can probably dodge that by going with a SMS enabled VoIP provider.
A way to limit the max spend per month would be great, even if that means creates outages.
I would happily put a $20/mo limit on many of my accounts. They should never bill more than few dollars a month.
I would happily put a $20/mo limit on many of my accounts. They should never bill more than few dollars a month.
AWS offers throw away accounts during immersion days, jam sessions, etc (especially at re:invent). It would be great if these were extended to the general public, even if at a small fee.
There’s a service like this called Qwicklabs that I have been using for GCP training. You load a time limited lab and get a new set of credentials only for that session. After the timer is up, poof everything is deleted.
Ya know at 20 theirs's a ton of very bad mistakes one can make.
Something that can be resolved via getting in contact with AWS isn't one of them. I've gotten charges refunded before.
Something that can be resolved via getting in contact with AWS isn't one of them. I've gotten charges refunded before.
Me too! I use AWS all day at work, so I should be able to find the service that charges my personal account 64 cents per month... :shrug: I'm glad that student got help.
[deleted]
The AWS billing dashboard has a "budgets" feature. I just added a daily account-level budget of $1 to my dev sandbox account, and setup an alert at 50% of the budget or $0.50. It took about a minute to setup.
You can choose to include or exclude refunds and credits. If you exclude them, I am guessing but it should show your true cost without the free tier pricing. If they are included it would tell you the impact right after any free tier offers expired.
Perhaps setting a budget like this should be part of any organization's new account setup process.
You can choose to include or exclude refunds and credits. If you exclude them, I am guessing but it should show your true cost without the free tier pricing. If they are included it would tell you the impact right after any free tier offers expired.
Perhaps setting a budget like this should be part of any organization's new account setup process.
Isn’t there a service that does something like this? I’d think by this point someone had made a business out of it if it was a serious problem.
I was hit by unplanned $43 aws bill three days ago, in our Nigerian economy $43 is a lot of money, I am still grieving.
aws free tier is a joke anyway. surprise billing even after minimal usage and they shutdown your account if you forget to pay or the credit card billing info does not work . If it is truly free, why do they need your credit card.
The author starts eith comparing a $200 buck charge that AWS support roller back to somebody losing or imagining to lose $100K in trading. Wild juxtapositions there.
After seeing how recklessly people use AWS when it's not their own wallet on the line, I'll wager to say that this 20-year old learned a valuablr lesson.
After seeing how recklessly people use AWS when it's not their own wallet on the line, I'll wager to say that this 20-year old learned a valuablr lesson.
That's why I use cloud providers that accept bitcoin. Credit cards are evil.
debugged a terraform script on my personal aws, I got a bill of ~100$ for the past 2 months. It seems i had eks still running without nodes!!
Why would a service allow to have eks without nodes? Will call support to get a refund!
Why would a service allow to have eks without nodes? Will call support to get a refund!
Feel like it's a good cloud learning experience for the student though.
Please just let me set a hard budget that caps my spending.
>> After all, the AWS Free Tier has been broken for 10+ years. How urgent of a problem can it be?
What? I use AWS Free Tier a lot and it is the best thing for small businesses and startups.
What? I use AWS Free Tier a lot and it is the best thing for small businesses and startups.
This is something I think about a lot. For work, I run a cloud service, and being able to accomodate students (and other cost-sensitive users) like the one in the post is important to me. I've always thought that letting people experiment over the weekend is what leads to people wanting to use something at work (where the real budget is), and I think that supporting those users is how you build a userbase of advocates. Maybe they can't afford (or justify) the enterprise plan quite yet, but they can still be a happy user and cheer on those that can justify the expense.
I've personally found that it can be hard to get the approval to type in your company credit card until you've done enough research to prove that something is going to be worth the money. That leads to a chicken and the egg problem; people won't be confident until they've paid, but they can't gain that confidence until they pay! So you have to get the small "testing" use case perfect, or you'll never have any real customers. (My corollary to this is because AWS already has your company credit card, and you already have "root" or similar because of your role, it sure is easy to build whatever you want there. If AWS provides some service, you can start using it and the cost will be lost in the noise. But if it's not on AWS, then you'll have to produce some justification to use the service before you can start paying for it. I ran into that a lot at my last job; I wanted to buy $5/month services like Sentry but was told no, whereas other people could just create a m4.4xlarge RDS instance for $1000/month and nobody even noticed or cared. People really like being approvers at the time of entering the credit card, rather than at the time of actual cost accrual, and cloud providers really facilitate that. Not sure that's really helping those approvers -- it almost feels a little bit like embezzlement.)
Anyway, ranting aside, here's what we do for the cloud service I work on:
1) We have a free tier. It's really free; you don't even get to enter a credit card. Sign in and start doing your work. (We delete your stuff after 4 hours, though.)
2) For the paid tier, all costs are pay as you go. The instant you click "delete workspace", no more costs accrue. Merely having an account open doesn't accrue any costs for you, and there is no way to create a phantom resource that you can't see in the UI and delete. If you delete everything, billing is over.
One weakness that I'd like to fix is the latency between resource use and when we tell you about it. That takes a few days, so if you are playing with aggressive autoscaling, you don't really know what it's going to cost until your experiment is over. I'd like to collect real-time usage data and just bill based off of that, so that the UI can update you within seconds of your job starting. If it's too much, you can just pull the plug and not be surprised.
The next step is letting people pre-pay, and do what the vast majority of comments on this thread want: kill the compute resources when the budget is exceeded. My thought is that it's hard to ask your customers for money upfront, which is probably where the post-pay model originated. I personally always have reservations about buying 3 year reserved instances from cloud providers, even if it saves a ton of money. "What if we stop using it tomorrow!?" But there is probably a good compromise here: type into the UI what you'd like to pay for autoscaling per month, and once that budget it exceeded, run at the bare minimum "keep the lights on" level. More difficult than the alternative, but certainly possible. And very good for users -- no total outage, no unexpected bill they have no hope of being able to pay. Things are just slower for a while.
Anyway, I don't know what the perfect formula for cloud pricing is -- but it's clear that what AWS has is not quite right, and that we can probably do better. To paraphrase Jeff Bezos ("your margin is my opportunity"), what AWS has consistently done wrong for years is your chance to make it better and get paid for doing so.
I've personally found that it can be hard to get the approval to type in your company credit card until you've done enough research to prove that something is going to be worth the money. That leads to a chicken and the egg problem; people won't be confident until they've paid, but they can't gain that confidence until they pay! So you have to get the small "testing" use case perfect, or you'll never have any real customers. (My corollary to this is because AWS already has your company credit card, and you already have "root" or similar because of your role, it sure is easy to build whatever you want there. If AWS provides some service, you can start using it and the cost will be lost in the noise. But if it's not on AWS, then you'll have to produce some justification to use the service before you can start paying for it. I ran into that a lot at my last job; I wanted to buy $5/month services like Sentry but was told no, whereas other people could just create a m4.4xlarge RDS instance for $1000/month and nobody even noticed or cared. People really like being approvers at the time of entering the credit card, rather than at the time of actual cost accrual, and cloud providers really facilitate that. Not sure that's really helping those approvers -- it almost feels a little bit like embezzlement.)
Anyway, ranting aside, here's what we do for the cloud service I work on:
1) We have a free tier. It's really free; you don't even get to enter a credit card. Sign in and start doing your work. (We delete your stuff after 4 hours, though.)
2) For the paid tier, all costs are pay as you go. The instant you click "delete workspace", no more costs accrue. Merely having an account open doesn't accrue any costs for you, and there is no way to create a phantom resource that you can't see in the UI and delete. If you delete everything, billing is over.
One weakness that I'd like to fix is the latency between resource use and when we tell you about it. That takes a few days, so if you are playing with aggressive autoscaling, you don't really know what it's going to cost until your experiment is over. I'd like to collect real-time usage data and just bill based off of that, so that the UI can update you within seconds of your job starting. If it's too much, you can just pull the plug and not be surprised.
The next step is letting people pre-pay, and do what the vast majority of comments on this thread want: kill the compute resources when the budget is exceeded. My thought is that it's hard to ask your customers for money upfront, which is probably where the post-pay model originated. I personally always have reservations about buying 3 year reserved instances from cloud providers, even if it saves a ton of money. "What if we stop using it tomorrow!?" But there is probably a good compromise here: type into the UI what you'd like to pay for autoscaling per month, and once that budget it exceeded, run at the bare minimum "keep the lights on" level. More difficult than the alternative, but certainly possible. And very good for users -- no total outage, no unexpected bill they have no hope of being able to pay. Things are just slower for a while.
Anyway, I don't know what the perfect formula for cloud pricing is -- but it's clear that what AWS has is not quite right, and that we can probably do better. To paraphrase Jeff Bezos ("your margin is my opportunity"), what AWS has consistently done wrong for years is your chance to make it better and get paid for doing so.
Goodness - reading these statements makes me laugh - "sinister" "evil" "by design".
Reality - very few folks want a hard billing limit.
To stop charging you AWS would need to delete all your EBS and S3 volumes, stop all EC2 instances, release all public IP's, delete all AWS directories and the list goes on. The idea that AWS would build this giant data loss footgun into their system is ridiculous.
Somewhere in AWS someone said, how could this blow up, and they came up with 100 ways, including misconfigured cost accounting etc.
That said, the GCP project based model makes more sense to me, give you more control etc.
That said, if there is such demand for hard billing limit playgrounds (I'm sure there is but not by folks giving AWS a lot of money), someone should be able to do a hosted solution for AWS that bills into their corp account and gives you a playground for learning (with a real hard billing limit). That type of approach is used in a lot of other contexts already.
Reality - very few folks want a hard billing limit.
To stop charging you AWS would need to delete all your EBS and S3 volumes, stop all EC2 instances, release all public IP's, delete all AWS directories and the list goes on. The idea that AWS would build this giant data loss footgun into their system is ridiculous.
Somewhere in AWS someone said, how could this blow up, and they came up with 100 ways, including misconfigured cost accounting etc.
That said, the GCP project based model makes more sense to me, give you more control etc.
That said, if there is such demand for hard billing limit playgrounds (I'm sure there is but not by folks giving AWS a lot of money), someone should be able to do a hosted solution for AWS that bills into their corp account and gives you a playground for learning (with a real hard billing limit). That type of approach is used in a lot of other contexts already.
AWS employs a whole lot of brilliant people, and they're an insanely profitable business unit.
I'm sure those people, given the appropriate motivation and opportunity, could provide a solution (or several) that avoids foot-gun data loss at every turn. There's all sort of solutions they could go with.
As an idea, perhaps some kind of functionality within the billing side of things:
They could, perhaps, decide to just apply a basic cap functionality - when you reach that cap, it stops/disables all chargeable resources above the free-tier and gives you 72 hours to go raise the cap or they'll delete everything except things in their AWS Backup.
AWS can eat the storage cost for the stopped things without a noticable impact on revenue.
Perhaps a more advanced workflow-type setup:
When monthly spend reaches $x across all regions: - stop all running instances that are not below the free-tier limit - apply deny policies for per-request charged services - send an email alert to the billing, tech and admin contacts.
When monthly spend reaches $y across all regions: - Delete all chargable resources not tagged with 'Foo: bar' - send an email alert to the billing, tech and admin contacts.
One of the other things that would be great, is to be able to apply limits to AWS Accounts. For instance, the AWS Account that we give to developers to experiment on, it would be great to forbid them from starting baremetal instances, or GPU instances, etc.
Some of it can be controlled through deny rules, but not enough.
Billing alerts are also currently nowhere near responsive enough - it can be hours or days before you get told that some resource is running up your bill.
A comprehensive dashboard of 'here is every chargeable resource running in this account, in all regions, right now' wouldn't go astray, either.
I'm sure those people, given the appropriate motivation and opportunity, could provide a solution (or several) that avoids foot-gun data loss at every turn. There's all sort of solutions they could go with.
As an idea, perhaps some kind of functionality within the billing side of things:
They could, perhaps, decide to just apply a basic cap functionality - when you reach that cap, it stops/disables all chargeable resources above the free-tier and gives you 72 hours to go raise the cap or they'll delete everything except things in their AWS Backup.
AWS can eat the storage cost for the stopped things without a noticable impact on revenue.
Perhaps a more advanced workflow-type setup:
When monthly spend reaches $x across all regions: - stop all running instances that are not below the free-tier limit - apply deny policies for per-request charged services - send an email alert to the billing, tech and admin contacts.
When monthly spend reaches $y across all regions: - Delete all chargable resources not tagged with 'Foo: bar' - send an email alert to the billing, tech and admin contacts.
One of the other things that would be great, is to be able to apply limits to AWS Accounts. For instance, the AWS Account that we give to developers to experiment on, it would be great to forbid them from starting baremetal instances, or GPU instances, etc.
Some of it can be controlled through deny rules, but not enough.
Billing alerts are also currently nowhere near responsive enough - it can be hours or days before you get told that some resource is running up your bill.
A comprehensive dashboard of 'here is every chargeable resource running in this account, in all regions, right now' wouldn't go astray, either.
It's a good lesson in cloud computing to know how much you're spending on services.
Usually if you're a single person and you leave an expensive product running unknowingly, AWS support have been kind and refund you the money and take the loss as 'customer made an error' type transaction.
It's not like they hold the debt over your head for life.
I think the article is a bit extreme to highlight the person's reaction, it's pretty easy bait for most people to swallow. Realistically though, if you can't afford $20, how on earth did you afford a credit card to sign up for AWS in the first place?
There are people comparing this person's response with the Robinhood investor that lost money and took their life. The real issue here is a mental health problem and better awareness around that - AWS is a cheap scapegoat we can all blame to ignore the real issue at hand...
Usually if you're a single person and you leave an expensive product running unknowingly, AWS support have been kind and refund you the money and take the loss as 'customer made an error' type transaction.
It's not like they hold the debt over your head for life.
I think the article is a bit extreme to highlight the person's reaction, it's pretty easy bait for most people to swallow. Realistically though, if you can't afford $20, how on earth did you afford a credit card to sign up for AWS in the first place?
There are people comparing this person's response with the Robinhood investor that lost money and took their life. The real issue here is a mental health problem and better awareness around that - AWS is a cheap scapegoat we can all blame to ignore the real issue at hand...
The problem is only partially "know how much you're spending on services".
There's a big problem of: There's a huge amount of AWS that even if you configure it "right" - still leaves you with potentially unlimited liability.
Take a simple example of using Cloudfront and S3 together to host a basic website. There's a good well documented way of securing this all "right".
However that doesn't protect you from "slashdotting" - perhaps you put an image on your site and it gets linked from the frontpage of Reddit.
You go from paying a few $/month, to suddenly overnight having potentially hundreds or thousands of dollars of spend because tens of millions of people have viewed your image.
AWS Billing alerts are not going to save you, because once they (eventually, maybe a day or so late) fire, and assuming you immediately see that notification, figure out what it is being accessed, and remove that file (or just turn off the bucket/cloudfront entirely) - the bills could have gone way up.
It gets worse when you're talking about other AWS services.
The whole point being that billing caps, along with billing alerts that are actually timely, are actually necessary.
You shouldn't have to depend on the grace of AWS Customer Service to not force you to decide whether to pay rent, eat, or your massive credit card bill.
> Realistically though, if you can't afford $20, how on earth did you afford a credit card to sign up for AWS in the first place?
The example was $200. And for a lot of people, particularly students and other people new to development - $200 can be a large chunk of money.
Part of the problem is that you go to AWS Conferences and talks, and they talk about how there's all this amazing stuff, and encourage people to sign up and try it out.
What they never do is tell you "Oh, by the way, this demo costs $x/hour". I'd love to see some kind of taxi-meter style box behind the presenter/in the corner of webcasts showing how much the AWS Bill would be for the demo.
There's a big problem of: There's a huge amount of AWS that even if you configure it "right" - still leaves you with potentially unlimited liability.
Take a simple example of using Cloudfront and S3 together to host a basic website. There's a good well documented way of securing this all "right".
However that doesn't protect you from "slashdotting" - perhaps you put an image on your site and it gets linked from the frontpage of Reddit.
You go from paying a few $/month, to suddenly overnight having potentially hundreds or thousands of dollars of spend because tens of millions of people have viewed your image.
AWS Billing alerts are not going to save you, because once they (eventually, maybe a day or so late) fire, and assuming you immediately see that notification, figure out what it is being accessed, and remove that file (or just turn off the bucket/cloudfront entirely) - the bills could have gone way up.
It gets worse when you're talking about other AWS services.
The whole point being that billing caps, along with billing alerts that are actually timely, are actually necessary.
You shouldn't have to depend on the grace of AWS Customer Service to not force you to decide whether to pay rent, eat, or your massive credit card bill.
> Realistically though, if you can't afford $20, how on earth did you afford a credit card to sign up for AWS in the first place?
The example was $200. And for a lot of people, particularly students and other people new to development - $200 can be a large chunk of money.
Part of the problem is that you go to AWS Conferences and talks, and they talk about how there's all this amazing stuff, and encourage people to sign up and try it out.
What they never do is tell you "Oh, by the way, this demo costs $x/hour". I'd love to see some kind of taxi-meter style box behind the presenter/in the corner of webcasts showing how much the AWS Bill would be for the demo.
Taking your cloudfront example, the free tier covers 50GB data out at 2 million requests - so your image would need to be less that 25KB to be under the free tier limit.
Taking a 4MB image, that would equate to around 8TB which is priced at under $1.
Make that 20 million views and you hit $10 - oof!
---
$20 or $200 doesn't matter - how on earth could you have a credit card if you can't afford to pay off $200?
---
AWS is great and people are free to try it out. They should also be aware that things cost money and not to put their credit card down for services without being fully aware of the costs associated.
It's like buying a property for investment then complaining to the real estate agent later down the line for a number of issues the property has that require maintenance down the line.
What's wrong with buyer beware? Why can't people/big companies offer services without people crying foul about 'getting hurt from bill shock'? Where's the personal responsibility and accountability for your own actions and decisions gone?
AWS aren't scheming to scape little bits of money from small time developers. They're interested in catching big fish companies who are looking to choose their platform / service offering in the cloud computing space. If you think these small time developers with 'bill caps so i don't overspend' is at the top of their feature priority list you've got the wrong idea about the business.
Taking a 4MB image, that would equate to around 8TB which is priced at under $1.
Make that 20 million views and you hit $10 - oof!
---
$20 or $200 doesn't matter - how on earth could you have a credit card if you can't afford to pay off $200?
---
AWS is great and people are free to try it out. They should also be aware that things cost money and not to put their credit card down for services without being fully aware of the costs associated.
It's like buying a property for investment then complaining to the real estate agent later down the line for a number of issues the property has that require maintenance down the line.
What's wrong with buyer beware? Why can't people/big companies offer services without people crying foul about 'getting hurt from bill shock'? Where's the personal responsibility and accountability for your own actions and decisions gone?
AWS aren't scheming to scape little bits of money from small time developers. They're interested in catching big fish companies who are looking to choose their platform / service offering in the cloud computing space. If you think these small time developers with 'bill caps so i don't overspend' is at the top of their feature priority list you've got the wrong idea about the business.
> $20 or $200 doesn't matter - how on earth could you have a credit card if you can't afford to pay off $200?
Who cares how they got the card, maybe it's a debit card, maybe it's their parent's or partners. Maybe they've got one that only allows a very low limit of charges.
The point is that there's a whole ton of people for whom $200 isn't a "Oh, well, that's a learning exercise - better be more careful next time".
> What's wrong with buyer beware?
Buyer beware is fine where all parties are fully aware of all repercussions.
However, you'll note that this doesn't apply to certain kinds of transactions, particularly financial transactions.
In some cases, you're either required to go through an expert who can advise you on the downsides and risks, or the seller is required to ensure that the buyer is fully informed of the risks.
It's the same reason we have warning signs on a whole bunch of things about how they could injure themselves.
Without the ability for someone to limit the damage, someone could be up for thousands of dollars in spend without being aware of how they even got there.
> AWS aren't scheming to scape little bits of money from small time developers
It's not even students and other inexperienced people that end up in bill-shock.
I've dealt with well-experienced senior developers who've gone off to an AWS Conference and spent a day or three being lulled by the story about how X is the new hotness, and would solve all the problems we have.
The AWS reps have all confirmed the story that X is the new hotness, and it's perfect for some new solution - and they've got some big name customer who's implemented it and saved millions Vs their legacy on-site solution. The AWS reps also hand out tens of thousands in account credits like they're mentos, with the message "Go play, see if it works for you".
Those senior developers have taken a look at the pricing on the thing, thought they understood it, and then fired up a demo to prove it out. When the billing starts to come in, there's been shock from them that they spent so much in so little time.
The pricing on things is quite opaque with one headline price for something, but doesn't include that you'll also need to consider the pricing for other bits. The billing is often delayed quite signfiicantly, and alerting is only useful as an indicator of how much you've fucked up, rather than a way to prevent it.
So, yeah, with AWS marketing all their solutions as being much lower cost, while at the same time not making it clear on what things do cost, and not making tools that you can prevent major fuckups -- they definitely do need to shoulder some blame.
Who cares how they got the card, maybe it's a debit card, maybe it's their parent's or partners. Maybe they've got one that only allows a very low limit of charges.
The point is that there's a whole ton of people for whom $200 isn't a "Oh, well, that's a learning exercise - better be more careful next time".
> What's wrong with buyer beware?
Buyer beware is fine where all parties are fully aware of all repercussions.
However, you'll note that this doesn't apply to certain kinds of transactions, particularly financial transactions.
In some cases, you're either required to go through an expert who can advise you on the downsides and risks, or the seller is required to ensure that the buyer is fully informed of the risks.
It's the same reason we have warning signs on a whole bunch of things about how they could injure themselves.
Without the ability for someone to limit the damage, someone could be up for thousands of dollars in spend without being aware of how they even got there.
> AWS aren't scheming to scape little bits of money from small time developers
It's not even students and other inexperienced people that end up in bill-shock.
I've dealt with well-experienced senior developers who've gone off to an AWS Conference and spent a day or three being lulled by the story about how X is the new hotness, and would solve all the problems we have.
The AWS reps have all confirmed the story that X is the new hotness, and it's perfect for some new solution - and they've got some big name customer who's implemented it and saved millions Vs their legacy on-site solution. The AWS reps also hand out tens of thousands in account credits like they're mentos, with the message "Go play, see if it works for you".
Those senior developers have taken a look at the pricing on the thing, thought they understood it, and then fired up a demo to prove it out. When the billing starts to come in, there's been shock from them that they spent so much in so little time.
The pricing on things is quite opaque with one headline price for something, but doesn't include that you'll also need to consider the pricing for other bits. The billing is often delayed quite signfiicantly, and alerting is only useful as an indicator of how much you've fucked up, rather than a way to prevent it.
So, yeah, with AWS marketing all their solutions as being much lower cost, while at the same time not making it clear on what things do cost, and not making tools that you can prevent major fuckups -- they definitely do need to shoulder some blame.
As the article points out: Every other cloud does this! They all have non-production subscription types with hard spending limits.
There's a difference between "unable" and "unwilling".
The people that refuse to understand the difference are the same people that don't understand that "unsupported" isn't synonymous with "cannot be made to function".
Don't be that person.
If you have a large account and you're in regular contact with an AWS sales representative, pressure them into making this happen. Even if you work for Megacorp with a $$$ budget, keep in mind that your future hires need to start somewhere, need to be able to learn on their own, and need to do so safely.
Don't walk down the same path as IBM's mainframes, where no student anywhere ever has been able to learn on their own, making it a dead-end for corporations who pay billions for this platform. You, or your company will eventually pay this price if AWS keeps this IBM-like behaviour up.
Think of the big picture, not just your own immediate personal situation.
Apply this pressure yourself, because the students you want to hire next year can't.