Hetzner cuts traffic on US VPSs, raises prices
711 comments
As a Hetzner bandwidth enjoyer affected by this, this is why (HN cough) multi-cloud/dedi k3s is great, because if you get rug pulled you just migrate to another provider with better prices.
That said, $1/TB for bandwidth overage seems pretty fair. I empathize with the complaining but if the new price is such a ripoff everyone should be recommending what cloud VM provider they're migrating to for a better deal.
That said, $1/TB for bandwidth overage seems pretty fair. I empathize with the complaining but if the new price is such a ripoff everyone should be recommending what cloud VM provider they're migrating to for a better deal.
I use OVH (VPS’s specifically), which offers unlimited bandwidth. In my experience they’ve been both reliable and affordable which is a rarity. I run a few applications that require high amounts of bandwidth, so silly caps like the ones that Hetzner are imposing are a non-starter for me
That's what a friend said also. If you look beyond the marketing material, hoever, the ToS says:
> OVHcloud reserves the right to restrict the VPS Service bandwidth to 1 Mbps (1 Megabit per second) until the end of the current billing period in cases of excessive use by the Client
but it advertises with "unmetered"... so is a meter attached by which they can tell whether your bandwidth use is excessive or not? Would they eat those costs for you?
I checked out some numbers. Quoting myself from chat history:
> it begs the question: what's "excessive"? I dunno but if they charge $5/month for the VPS and, while AWS may be ~1/3rd cheaper [than some other thing], that's still on the order of 70$/month. And AWS has insane economies of scale working for them, maybe their cost price is $7/month if they don't need to have a competitive price but that's still a loss then
> I bet you'd win the lawsuit where [OVH] falsely advertised with unmetered 500mbps and a terms of service saying "excessive", so when you transfer 2 TB/day on a connection advertised to be capable of 500mbps×24h = 5.4TB/day... that's reasonable right? But then you're having a lawsuit over a 5$/month VPS
> OVHcloud reserves the right to restrict the VPS Service bandwidth to 1 Mbps (1 Megabit per second) until the end of the current billing period in cases of excessive use by the Client
but it advertises with "unmetered"... so is a meter attached by which they can tell whether your bandwidth use is excessive or not? Would they eat those costs for you?
I checked out some numbers. Quoting myself from chat history:
> it begs the question: what's "excessive"? I dunno but if they charge $5/month for the VPS and, while AWS may be ~1/3rd cheaper [than some other thing], that's still on the order of 70$/month. And AWS has insane economies of scale working for them, maybe their cost price is $7/month if they don't need to have a competitive price but that's still a loss then
> I bet you'd win the lawsuit where [OVH] falsely advertised with unmetered 500mbps and a terms of service saying "excessive", so when you transfer 2 TB/day on a connection advertised to be capable of 500mbps×24h = 5.4TB/day... that's reasonable right? But then you're having a lawsuit over a 5$/month VPS
> is a meter attached
Yes, of course. Having flow data (or monitoring ports/interfaces) for traffic engineering and management is pretty essential, not least for determining when capacity upgrades are needed.
I understand both sides of the argument here. The idea of offering "unlimited" is appealing because most users of a typical 2GB RAM virtual machine (as an example) consume less than 1TB of bandwidth per month. Offering unlimited bandwidth removes the hassle of overage charges/billing queries and eases customer concern/friction. Both sides benefit from this.
However, on the other hand, is it reasonable for a $5/month virtual machine customer to use 1Gbps 24/7/365, potentially consuming $100–$200 worth of bandwidth?
Should providers avoid offering unlimited bandwidth unless it's truly unlimited? From an engineer's perspective, yes, I agree. But this stance also risks degrading the experience for the 99.5% of "normal" customers—those who don’t exploit this simplification of "free bandwidth"—just to address a handful of users who take full advantage of it.
It's tough, so IME most such providers leave something in their terms that allows them to intervene in extreme cases but typically exercise restraint in doing so, usually only doing it manually if they notice that 'extreme' usage is damaging other users experience e.g it's serious and prolonged usage.
Yes, of course. Having flow data (or monitoring ports/interfaces) for traffic engineering and management is pretty essential, not least for determining when capacity upgrades are needed.
I understand both sides of the argument here. The idea of offering "unlimited" is appealing because most users of a typical 2GB RAM virtual machine (as an example) consume less than 1TB of bandwidth per month. Offering unlimited bandwidth removes the hassle of overage charges/billing queries and eases customer concern/friction. Both sides benefit from this.
However, on the other hand, is it reasonable for a $5/month virtual machine customer to use 1Gbps 24/7/365, potentially consuming $100–$200 worth of bandwidth?
Should providers avoid offering unlimited bandwidth unless it's truly unlimited? From an engineer's perspective, yes, I agree. But this stance also risks degrading the experience for the 99.5% of "normal" customers—those who don’t exploit this simplification of "free bandwidth"—just to address a handful of users who take full advantage of it.
It's tough, so IME most such providers leave something in their terms that allows them to intervene in extreme cases but typically exercise restraint in doing so, usually only doing it manually if they notice that 'extreme' usage is damaging other users experience e.g it's serious and prolonged usage.
> it reasonable for a $5/month virtual machine customer to use 1Gbps 24/7/365, potentially consuming $100–$200 worth of bandwidth
It's reasonable for OVH to prevent that, and it's also reasonable for OVH to explicitly and clearly define the limits up front.
It's reasonable for OVH to prevent that, and it's also reasonable for OVH to explicitly and clearly define the limits up front.
It’s also reasonable for OVH to not do that, as most of their customers don’t understand 95th percentile billing, which is the model that they’re being charged at by their transit suppliers.
It’s also reasonable for OVH to not do that, as most of their customers dont understand that transit costs blend with port costs depending on destination, and some destinations are effectively ‘free’ to send/receive from (fixed port costs only, no marginal costs), and other destinations are not (marginal costs associated with transit supplier fees).
The billing model consumers want is a simple BW used calculation, without facing the reality that if they consume their entire BW allowance as quickly as possible, it incurs order of magnitude higher costs than if they consume it at a trickle over the whole month.
It’s also reasonable for OVH to not do that, as most of their customers dont understand that transit costs blend with port costs depending on destination, and some destinations are effectively ‘free’ to send/receive from (fixed port costs only, no marginal costs), and other destinations are not (marginal costs associated with transit supplier fees).
The billing model consumers want is a simple BW used calculation, without facing the reality that if they consume their entire BW allowance as quickly as possible, it incurs order of magnitude higher costs than if they consume it at a trickle over the whole month.
You're acting like it's complicated, but they could just not say "unlimited" here: https://www.ovhcloud.com/en/bare-metal/bandwidth/
Like:
> Generous bandwidth limits*
> *see this link for details
Like:
> Generous bandwidth limits*
> *see this link for details
It’s worth going back to the start of this thread and seeing that this all started with someone complaining that their provider had reduced the BW allowance and wanting somewhere with more generous allowances. When the provider sells things for what it actually costs, the customer gets upset and looks for someone selling a subsidised product in a misleading way. Leading to other people getting upset about the mislead. Those people should go to the original provider, who is doing exactly what they asked for!
kind of true except if you've got 1000's of customers it all evens out and your traffic profile is actually quite smooth if you've got say 6 X 100G transits, 4 X 100G IXP ports and 5 X 100G PNIs then the impact of an individual 1G customer is not even noticeable, honestly. We can work to 1Mbps 95%ile being about 250GB of total transfer at scale.
I completely agree. The product being sold has only a loose connection to the cost incurred by the provider. This is why the product being sold is being sold in a vague / loose-ish way, because for the overwhelming majority of customers, the product being sold can be sold for a profit.
Doing billing like this is awkward as you just don't know what you are going to get. And, in a world where most people are in fact using small amounts of bandwidth, what you have done is caused people who use small amounts of bandwidth to pay MORE than they should, as they are effectively paying the price of the bandwidth for the average user: if you use less than the average, you are subsidizing the people who use more than the average.
Meanwhile, in an ecosystem where everyone isn't already being ripped off with overly-expensive bandwidth, if an ecosystem-level event happens that causes the average user to suddenly use more bandwidth, the service either has to raise rates for everyone or they have to start claiming some uses of bandwidth are "egregious".
The result is then that, to defend the small-scale user from paying even more than the too much you are already charging them (as they are subsiding the larger users), you suddenly start doing traffic analysis with price discrimination by use case, and network neutrality goes out the window :/.
The real reason any of this works is just that people in fact aren't being charged fair prices most of the time, and these unlimited plans let the provider hide that from all involved. If everyone were charged a fair price, not only would heavy users pay a lot and light users pay LESS than they often do today, but everyone would be paying little enough that this idea that it is a big customer "concern" goes away, the same as it is for electricity or water: except in extreme circumstances, no one frets over sudden utility overages.
Meanwhile, in an ecosystem where everyone isn't already being ripped off with overly-expensive bandwidth, if an ecosystem-level event happens that causes the average user to suddenly use more bandwidth, the service either has to raise rates for everyone or they have to start claiming some uses of bandwidth are "egregious".
The result is then that, to defend the small-scale user from paying even more than the too much you are already charging them (as they are subsiding the larger users), you suddenly start doing traffic analysis with price discrimination by use case, and network neutrality goes out the window :/.
The real reason any of this works is just that people in fact aren't being charged fair prices most of the time, and these unlimited plans let the provider hide that from all involved. If everyone were charged a fair price, not only would heavy users pay a lot and light users pay LESS than they often do today, but everyone would be paying little enough that this idea that it is a big customer "concern" goes away, the same as it is for electricity or water: except in extreme circumstances, no one frets over sudden utility overages.
This is a for profit business not a limited common ressource being shared.
What you are missing here is that the adjustment is not low usage users subsidising high usage users, it’s OVH margins. Nobody is being subsidised. Low usage users just make OVH more money than high usage users. OVH doesn’t mind because per user costs are actually low and they are already competitive at that price without adding more complexity to their product mix. Users which would lead to an actual loss are rate limited.
What you are missing here is that the adjustment is not low usage users subsidising high usage users, it’s OVH margins. Nobody is being subsidised. Low usage users just make OVH more money than high usage users. OVH doesn’t mind because per user costs are actually low and they are already competitive at that price without adding more complexity to their product mix. Users which would lead to an actual loss are rate limited.
> What you are missing here is that the adjustment is not low usage users subsidising high usage users, it’s OVH margins.
I did not miss this, and it was part of my point: the only reason this makes any sense at all is because these providers are ripping people off on bandwidth, which is how they have a margin so large that they feel a need to hide it from people under this kind of ridiculous pricing abnormality.
What is awkward is just accepting that and helping to make it worse by advocating for making it easier to kind of hide that fact: bandwidth is a commodity product, and these pricing games aren't pro-consumer because they somehow help people not have to worry about one month getting ripped off too much... they are anti-consumer because they enable the perpetuation of the state of affairs wherein people get ripped off in the first place.
The bandwidth providers know this, but they--of course ;P--like their excessive margins... but, if you just stopped claiming this was pro-consumer and realized what was actually going on here, the idea that a margin so excessive as to be able to essentially make the usage for the median user irrelevant should indicate a nigh-unto-ridiculous level of market distortion.
Like, we shouldn't sit around and just tolerate these margins. And that this particular pricing trick helps make these margins a bit more stomached by people really sucks! And in some sense I get it that it does make it easier to stomach... but... only because I think people are just buying into the idea that this must be a reasonable price :(.
And--even then--it doesn't fix the other problem I talked about (which I explicitly hedged as being in the world where the price wasn't set up to gouge everyone): when Facetime came out, it overnight was going to cause everyone with an iPhone to suddenly need more bandwidth, and so network providers temporarily needed to ban it or charge more for it; we see the same thing with the step up to video streaming services from basic web browsing, leading to providers feeling a need to zero-rate.
The reality is that bandwidth IS a limited common resource being shared at that provider--the same as any other product where the price isn't being distorted: this is the whole reason we use markets for this stuff in the first place--and the pricing of it at different providers should encounter market forces to drive it down closer to cost... except we are trapped in a local minimum here by people who refuse to understand that unlimited schemes cost more, not less.
I did not miss this, and it was part of my point: the only reason this makes any sense at all is because these providers are ripping people off on bandwidth, which is how they have a margin so large that they feel a need to hide it from people under this kind of ridiculous pricing abnormality.
What is awkward is just accepting that and helping to make it worse by advocating for making it easier to kind of hide that fact: bandwidth is a commodity product, and these pricing games aren't pro-consumer because they somehow help people not have to worry about one month getting ripped off too much... they are anti-consumer because they enable the perpetuation of the state of affairs wherein people get ripped off in the first place.
The bandwidth providers know this, but they--of course ;P--like their excessive margins... but, if you just stopped claiming this was pro-consumer and realized what was actually going on here, the idea that a margin so excessive as to be able to essentially make the usage for the median user irrelevant should indicate a nigh-unto-ridiculous level of market distortion.
Like, we shouldn't sit around and just tolerate these margins. And that this particular pricing trick helps make these margins a bit more stomached by people really sucks! And in some sense I get it that it does make it easier to stomach... but... only because I think people are just buying into the idea that this must be a reasonable price :(.
And--even then--it doesn't fix the other problem I talked about (which I explicitly hedged as being in the world where the price wasn't set up to gouge everyone): when Facetime came out, it overnight was going to cause everyone with an iPhone to suddenly need more bandwidth, and so network providers temporarily needed to ban it or charge more for it; we see the same thing with the step up to video streaming services from basic web browsing, leading to providers feeling a need to zero-rate.
The reality is that bandwidth IS a limited common resource being shared at that provider--the same as any other product where the price isn't being distorted: this is the whole reason we use markets for this stuff in the first place--and the pricing of it at different providers should encounter market forces to drive it down closer to cost... except we are trapped in a local minimum here by people who refuse to understand that unlimited schemes cost more, not less.
You don't actually tolerate high seller margins. Hosting is a competitive market.
If there was significant gains to be made by being more aggressive on the low end of the market, providers would already be doing it (and they are - OVH 5$ offer is quite aggressive). There is a reason nobody actually offers a better deal.
If there was significant gains to be made by being more aggressive on the low end of the market, providers would already be doing it (and they are - OVH 5$ offer is quite aggressive). There is a reason nobody actually offers a better deal.
If you pay $5 and use $100 of bandwidth costs, you are in fact being subsidized by other users, not by margins. We don't know what OVH pays for bandwidth though.
But nobody can do that. OVH doesn't let you be a large net negative.
What happens is roughly that:
- You are costing 1$ (bandwidth, etc.). You make OVH 4$. They are happy. Nobody offers you a cheaper alternative so you are stuck paying 5$ anyway.
- You are costing 4$ (bandwidth, etc.). You make OVH 1$. They are happy as marginal costs are low anyway.
- You are costing more than 5$. OVH severely rate limit your bandwidth to cut their costs and wait for you to leave because the service is now useless to you.
What happens is roughly that:
- You are costing 1$ (bandwidth, etc.). You make OVH 4$. They are happy. Nobody offers you a cheaper alternative so you are stuck paying 5$ anyway.
- You are costing 4$ (bandwidth, etc.). You make OVH 1$. They are happy as marginal costs are low anyway.
- You are costing more than 5$. OVH severely rate limit your bandwidth to cut their costs and wait for you to leave because the service is now useless to you.
If I order some shoes from Amazon, I find them uncomfortable, and I return them for a full refund causing Amazon to incur a loss - have I been "subsidised" by other customers?
Personally I would say if Amazon makes a profit selling you a book and makes a loss shipping me some shoes which I return, the loss was paid by Amazon, not by you.
Personally I would say if Amazon makes a profit selling you a book and makes a loss shipping me some shoes which I return, the loss was paid by Amazon, not by you.
The comparison is more apt if you gained something (because the bandwidth user gets a product out of it), say by having worn the shoes for a day and doing this every day so you get free shoes for life. Then, yes, it's pretty clear the paying customers are the ones footing your bill
Yet book prices will be adjusted so, on net, Amazon does not lose money.
View this as an insurance and it suddenly all makes perfect sense. You pay a little more so if this month your usage pikes, you won't get a surprise invoice you didn't budget for, and you won't get cut either. At worst, you'll get rate-limited. This price stability is valuable and paying extra to get it isn't being charged an unfair price. Of course if you don't find it beneficial, you should choose another offering.
This is only relevant because the cost of bandwidth is excessively high--much higher than it should be--and so people essentially need to pay for this gouging-insurance.
[deleted]
So, the "fair price" for internet bandwidth in Europe/NA is typically between a tenth and a quarter of a single cent per GB transfered in the heaviest direction.
So you prefer to pay $4.50 for your vm + 47.12126 cents for 460Gb data transfer , rather than $5 for your VM with unmetered data transfer?
I think by the way that the sensible answer is what DO/Linode etc do which is allocate some included data transfer per VM and pool it across your account. That's honestly a very sensible balance from my viewpoint, but they then charge you quite alot for overage around 1-2c per GB which is ~10X the "fair price".
So you prefer to pay $4.50 for your vm + 47.12126 cents for 460Gb data transfer , rather than $5 for your VM with unmetered data transfer?
I think by the way that the sensible answer is what DO/Linode etc do which is allocate some included data transfer per VM and pool it across your account. That's honestly a very sensible balance from my viewpoint, but they then charge you quite alot for overage around 1-2c per GB which is ~10X the "fair price".
> > is a meter attached
> Yes, of course.
So it's not unmetered as advertised or am I misunderstanding that word?
> this stance also risks degrading the experience for the 99.5% of "normal" customers—those who don’t exploit this simplification of "free bandwidth"
How so? If they want to be relaxed about it, the terms can say that you can burst more (e.g. "you can use 500GB/month, and burst to 5TB for two months of every two-year period; we'll send you a notification email whenever this happens so you're not caught by surprise"). If they don't want to be flexible, they can mention the hard limit that they are going to enforce regardless of whether they call it unlimited without asterisk. Either way, the buyer would know what they can actually use and doesn't have to guess
> Yes, of course.
So it's not unmetered as advertised or am I misunderstanding that word?
> this stance also risks degrading the experience for the 99.5% of "normal" customers—those who don’t exploit this simplification of "free bandwidth"
How so? If they want to be relaxed about it, the terms can say that you can burst more (e.g. "you can use 500GB/month, and burst to 5TB for two months of every two-year period; we'll send you a notification email whenever this happens so you're not caught by surprise"). If they don't want to be flexible, they can mention the hard limit that they are going to enforce regardless of whether they call it unlimited without asterisk. Either way, the buyer would know what they can actually use and doesn't have to guess
“Unmetered” means “You will not be charged under normal circumstances based on the measurement of the data you use.” It does not mean that your traffic is literally not measured.
They don’t put a specific hard limit because doing so both limits their own flexibility as a service provider and creates a target for abuse by users.
They don’t put a specific hard limit because doing so both limits their own flexibility as a service provider and creates a target for abuse by users.
For some definition of "normal circumstances". Being a bigger user should fall within it or that's not accurate advertising.
Some places will offer a choice between faster metered and slower unmetered. That seems like a good compromise to me. A nice big link should cost the host a single digit number of dollars per 100Mbps, so it's not hard to find an option where everyone is happy with the speed and pricing.
Some places will offer a choice between faster metered and slower unmetered. That seems like a good compromise to me. A nice big link should cost the host a single digit number of dollars per 100Mbps, so it's not hard to find an option where everyone is happy with the speed and pricing.
If you want a contract that has every term and circumstance negotiated up front, you’re going to need to speak with Hetzner’s business development team. You’ll also need to be a bigger fish than a single hobby developer.
Hetzner has terms up front just fine. $1/TB.
The unmetered issue involves other providers, and plenty of them are up front about it too.
The unmetered issue involves other providers, and plenty of them are up front about it too.
> However, on the other hand, is it reasonable for a $5/month virtual machine customer to use 1Gbps 24/7/365, potentially consuming $100–$200 worth of bandwidth?
Irrelevant. If you sell a vCPU with enough bandwidth to feed your 1GBps 24/7/365 needs, and you charge $5/month for it, then it matters nothing what's your personal notion of reasonable. What matters is the service plan offered by the cloud provider and the performance indicators they are contractually obligated to meet.
Irrelevant. If you sell a vCPU with enough bandwidth to feed your 1GBps 24/7/365 needs, and you charge $5/month for it, then it matters nothing what's your personal notion of reasonable. What matters is the service plan offered by the cloud provider and the performance indicators they are contractually obligated to meet.
> What matters is the service plan offered by the cloud provider and the performance indicators they are contractually obligated to meet
Indeed, and those indicators are specified in the contract, not in the headline product description. There are a lot of people unhappy that those indicators in this contract are not specific enough. Those people shouldn’t buy these contracts.
(Also, if you use your 1Gbps port at full speed at the most peak time for bandwidth utilisation, for 37 hours in a month, and not at all outside of that, assuming 20 cents a megabit with 95th percentile billing, the costs you’ve incurred to your provider are $200. Also it doesn’t matter at all what you do after those 37 hours, the costs to the provider are the same. You doing 300TB in a month costs the same as you doing 16TB, if you do the 16TB the ‘wrong’ way.)
Indeed, and those indicators are specified in the contract, not in the headline product description. There are a lot of people unhappy that those indicators in this contract are not specific enough. Those people shouldn’t buy these contracts.
(Also, if you use your 1Gbps port at full speed at the most peak time for bandwidth utilisation, for 37 hours in a month, and not at all outside of that, assuming 20 cents a megabit with 95th percentile billing, the costs you’ve incurred to your provider are $200. Also it doesn’t matter at all what you do after those 37 hours, the costs to the provider are the same. You doing 300TB in a month costs the same as you doing 16TB, if you do the 16TB the ‘wrong’ way.)
Thats only true if all your customers choose the exact same 37 Hours (and the same/similar destinations) back in the real world that's very very unlikely and so the 95%ile "issue" is a bit misleading unless an individual customer has the ability to use more than a couple of percent of your overall capacity (rare-ish at scale).
I completely agree. The product being sold has only a loose connection to the cost incurred by the provider. This is why the product being sold is being sold in a vague / loose-ish way, because for the overwhelming majority of customers, the product being sold can be sold for a profit.
> is a meter attached
Probably not by default, but if your usage starts to saturate their network switches they’ll add one, to figure out who’s disrupting everyone else’s QoS.
Probably not by default, but if your usage starts to saturate their network switches they’ll add one, to figure out who’s disrupting everyone else’s QoS.
Forgot to mention, OVH makes the current topology and traffic saturation of their network switches public: http://weathermap.ovh.net/
(I think they mostly do this so that customers can see and verify that any DC-level peering relationships, or per-customer site peering contracts [a.k.a. "OVHCloud Connect"] are being taken advantage of to flow the customer's traffic. But it's convenient for other things, too.)
(I think they mostly do this so that customers can see and verify that any DC-level peering relationships, or per-customer site peering contracts [a.k.a. "OVHCloud Connect"] are being taken advantage of to flow the customer's traffic. But it's convenient for other things, too.)
OVH is the most set and forget experience I've ever had. They email me maintenance notices 3 or 4x a year, but I don't think I've ever had any downtime. It just stays happily humming along for years. I think I pay something like 60 a year for it.
OVH and reliable, in the same sentence? They're cheap, so suitable for projects you don't mind going poof.
Personal anecdote. A few years ago, I lost a lot of sleep on a domain renewal at OVH. Their incompetence was mind-boggling. A less common tld was the only slightly challenging bit. After a week of calling and emailing, and on the verge of the domain lapsing, I gave up and sent someone to the tld registry with cash.
Also, do search for OVH SBG2 should you have missed that.
Personal anecdote. A few years ago, I lost a lot of sleep on a domain renewal at OVH. Their incompetence was mind-boggling. A less common tld was the only slightly challenging bit. After a week of calling and emailing, and on the verge of the domain lapsing, I gave up and sent someone to the tld registry with cash.
Also, do search for OVH SBG2 should you have missed that.
OVH's infrastructure is absolutely very reliable.
The pain begins when you need support. Just like you, I have lost a lot of sleep over domains held hostage by their incompetence (for almost a year in one instance). Lesson learned, never use OVH for domains.
The support for their dedicated servers is just as bad, mind you, but short of a hardware failure you really don't need them. I have several years of uptime on all my current services.
So for personal projects their vps/dedicated is still a fantastic value.
The pain begins when you need support. Just like you, I have lost a lot of sleep over domains held hostage by their incompetence (for almost a year in one instance). Lesson learned, never use OVH for domains.
The support for their dedicated servers is just as bad, mind you, but short of a hardware failure you really don't need them. I have several years of uptime on all my current services.
So for personal projects their vps/dedicated is still a fantastic value.
> OVH's infrastructure is absolutely very reliable.
Well except that time one of their datacenters burned down, likely due to insufficient fire suppression, and the data backups were also lost because they kept them in the same building as the originals.
https://www.datacenterdynamics.com/en/news/ovhcloud-ordered-...
These reports criticize OVHcloud for having no fire prevention system and no power cut-off on the site, for using wooden floors, and for a free-cooling design that created airflows that spread the fire. The reports also say that water was detected near electrical systems before the fire broke out.
It takes quite a while to regain trust after shitting the bed that badly.
Well except that time one of their datacenters burned down, likely due to insufficient fire suppression, and the data backups were also lost because they kept them in the same building as the originals.
https://www.datacenterdynamics.com/en/news/ovhcloud-ordered-...
These reports criticize OVHcloud for having no fire prevention system and no power cut-off on the site, for using wooden floors, and for a free-cooling design that created airflows that spread the fire. The reports also say that water was detected near electrical systems before the fire broke out.
It takes quite a while to regain trust after shitting the bed that badly.
> data backups were also lost because they kept them in the same building as the originals.
I asked that question Hetzner support and they admitted doing the same
I asked that question Hetzner support and they admitted doing the same
Better to be honest about having weak backups so users can plan accordingly, than to lie about how safe the data is and lull users into a false sense of security...
the OVH contract relating to automatic backup stipulates that a backup of the VPS server is scheduled daily, exported, and then replicated three times before being available in the customer space, and that the storage space allocated to the 'back-up option is physically isolated from the infrastructure in which the VPS server is set up.'
the OVH contract relating to automatic backup stipulates that a backup of the VPS server is scheduled daily, exported, and then replicated three times before being available in the customer space, and that the storage space allocated to the 'back-up option is physically isolated from the infrastructure in which the VPS server is set up.'
They can be terrible. But it is dirt cheap. So if you use their stuff in a way that you can recover somewhere else if they have issues, you save a lot.
As you mentioned I would stay away from them for things like domain hosting. Just use them for cheap compute, etc.
As you mentioned I would stay away from them for things like domain hosting. Just use them for cheap compute, etc.
I keep my whois and NS entries elsewhere and my nameservers sufficiently distributed that I find the risk acceptable, but both hetzner and ovh are firmly in the "I have always felt I got a very cost effective 'exactly what I paid for' - and in the case of my rare interactions with their support more than I'd hoped for" category for me.
Neither has ever caused me a problem that didn't feel like "potentially having this level of problem occasionally is entirely in keeping with how little I'm paying" basically.
Neither has ever caused me a problem that didn't feel like "potentially having this level of problem occasionally is entirely in keeping with how little I'm paying" basically.
From personal experience a few years back, OVH support was one of the worst I have ever experienced in my career. Technical incompetence at multiple levels of the chain (e.g lack of understanding of how DNS works). I would never recommend it to anyone, not even my worst enemy.
I can confirm this.
I had packet loss on my server. They asked me several times to reboot my server into rescue mode and leave it there for 10+ hours until their senior technician could look into it at an unspecified time of day.
After a month of doing this 3-4 times, they finally admitted that their switch is overprovisioned and there was no ETA. This problem happened in 2 locations.
Also had a problem with the failover ip failing to move. Again they told me to reboot into rescue mode and leave it like that for hours. No fix.
I've left OVH entirely after being a customer of theirs for over 10 years.
I had packet loss on my server. They asked me several times to reboot my server into rescue mode and leave it there for 10+ hours until their senior technician could look into it at an unspecified time of day.
After a month of doing this 3-4 times, they finally admitted that their switch is overprovisioned and there was no ETA. This problem happened in 2 locations.
Also had a problem with the failover ip failing to move. Again they told me to reboot into rescue mode and leave it like that for hours. No fix.
I've left OVH entirely after being a customer of theirs for over 10 years.
Yeah,just hope you never have to interact with support
As long as their their wood floor datacenter and your backup in the same location does not burn down
„… data center had wooden ceilings, no extinguisher, and no power cut-out“
https://www.datacenterdynamics.com/en/news/ovhcloud-fire-rep...
„… data center had wooden ceilings, no extinguisher, and no power cut-out“
https://www.datacenterdynamics.com/en/news/ovhcloud-fire-rep...
The systems *I* currently have at hetzner are, so far as I'm aware, on a "we don't charge for bandwidth but if you use a shitload you'll get throttled for the rest of the month" plan just like my ovh boxen.
But I only pull dedis from hetzner; my VPSen are all ovh based. So please nobody expect my experience to generalise without triple checking the terms just like I did in the process of signing up for those systems.
But I only pull dedis from hetzner; my VPSen are all ovh based. So please nobody expect my experience to generalise without triple checking the terms just like I did in the process of signing up for those systems.
I wouldn't say "affordable". Looking at their pricing page right now. A US-based 64GB RAM cloud server in OVH - $539
AWS is $220 (us-east, r6a.2xlarge instance, 1yr reserved)
AWS is $220 (us-east, r6a.2xlarge instance, 1yr reserved)
If you reserve in AWS for a year, you might as well get a dedicated Hetzner server:
70 EUR for 64 GB with 8c/16t, 1 TB local NVMe
123 EUR for 128 GB with 16c/32t, 2 TB local NVMe
70 EUR for 64 GB with 8c/16t, 1 TB local NVMe
123 EUR for 128 GB with 16c/32t, 2 TB local NVMe
I've only ever been on OVH and was surprised to discover a few years ago that bandwidth is not only unlimited but also costly at most other hosting companies (including cloud ones).
Shout-out to Microtronixdc, my colocation provider. For their $55/month/U colocation package, they provide unmetered gigabit for no additional cost.
https://microtronixdc.com/
https://microtronixdc.com/
Their option to colocate a Raspberry Pi (or similar form factor) for $5/month is also interesting as a VPN exit node.
Based on their broad "Acceptable Use Policy" (https://microtronixdc.com/acceptable-use-policy), I'd wager a guess they'll find a reason to shut that down.
We have a few exit nodes already on the network.
Yeah, I also later noticed they charge $46/month for 4 amps whereas the Raspberry Pi 5 requires a 5V, 5A power supply to take full advantage of its processing capabilities and features.
I assume the 4 amps they charge for are at 120 V (480 W), while the Pi only needs 25 W plus some rectifier inefficiency.
Its based on wattage, 4A yes, but 4A multiplied by the voltage. So Pi 5 is 5A, but multiple that 5A times 24V dc and you get 120watts :) so yes, you can definitely colo a pi 5 my friend :D
Good to know. You may want to remove amps from your order page (just leave watts) to avoid the confusion.
I see terms for hosting and domain registration and managed databases, but not colocation. Do you know if they really not care if you actually use that connection to the stated allowance?
https://microtronixdc.com/products/colocation
Their big selling point is that they truly provide unmetered connections.
Their big selling point is that they truly provide unmetered connections.
Yep, we do!! :)
Thanks for the shoutout!! :)
Exactly! I had a three node k3s cluster hosted on OVH. When I decided to switch to Contabo, it was as easy as adding the three Contabo nodes to the cluster, and then removing the three OVH nodes (plus updating some DNS rules). It was the easiest and simplest migration I'd ever done. All my services and data just moved automatically and mostly with zero downtime. The only service that experienced a little downtime was Plex, which as I understand does not support high availability. If I ever find a cheaper host, I'll simply switch over. No hassle and no vendor lock in.
This is what I also want to do.
How are you handling storage? That is the only issue I'm struggling with for a small 3-node deployment.
How are you handling storage? That is the only issue I'm struggling with for a small 3-node deployment.
Longhorn volumes automatically replicate to multiple nodes (configurable) and automatically move to the nodes whose pods need it.
A postgres database running on a single node will experience some downtime during (re)deployments or when moving across nodes, but should be pretty quick depending on the size of the database. For true HA database, CockroachDB is supposed to be compatible with postgres, but I haven't had a chance to play with it.
A postgres database running on a single node will experience some downtime during (re)deployments or when moving across nodes, but should be pretty quick depending on the size of the database. For true HA database, CockroachDB is supposed to be compatible with postgres, but I haven't had a chance to play with it.
How's the performance with Longhorn with databases?
That's indeed the pain point. Distributing a stateless app is relatively easy. Distributing the shared file system and database over a remote, higher latency, cross-cloud setup is hard.
I run everything on Longhorn, including databases. Not the highest iops but definitely worth the hit for the easy of migrating.
The OVH Eco-line (Kimsufi, So You Start..) are incredibly good and affordable. I have a few beefy servers, which I use to create my own VPS service on top of Proxmox. I'm not a sysadm and it's still simple to setup and maintain.
Eco is clever, because they reuse good hardware pieces to assemble new servers, instead of throwing out as garbage...
Have been a very happy user with several servers for quite some time.
Eco is clever, because they reuse good hardware pieces to assemble new servers, instead of throwing out as garbage...
Have been a very happy user with several servers for quite some time.
As someone who's already using the dedicated server for a ton of things, I have been really grateful. But now, I have a new question, are they going to do this to their dedicated servers as well?
This change only affects VPS services in the US, and for what it's worth, Hetzner does not offer dedicated servers in the US.
I am very ignorant here, so my apologies, but...
When someone runs a dedicated server these days, does this mean a one-off linux install? Or is this more likely to be a docker install so that it's portable?
When someone runs a dedicated server these days, does this mean a one-off linux install? Or is this more likely to be a docker install so that it's portable?
It's an actual entire machine given to you. I remember there were a few options for me from Ubuntu, Debian to Red Hat to choose from, but all of them would also have preconfigured system users and some level of administration done by the provider.
But other than that, it's an actual bare metal machine and I installed Ubuntu on it and threw in a giant heap of services that have been running on it for more than a year now.
But other than that, it's an actual bare metal machine and I installed Ubuntu on it and threw in a giant heap of services that have been running on it for more than a year now.
If you could rewind the clock, would you have started setting it up any differently, like in a container?
I am just curious what your options would be now if you wanted to migrate. Would you just copy your bash history to a local text file for reference, and then repeat the steps on a new server?
I am just curious what your options would be now if you wanted to migrate. Would you just copy your bash history to a local text file for reference, and then repeat the steps on a new server?
No, I wouldn't have started differently and I like the performance and the dedicated hardware I get for the money I spend. I have a custom backup solution that will upload daily backups of all my data to remote drives and I should be able to restore the setup on another machine without much problem.
Generally even in containerized deployments, you run one container per service/process. You wouldn't run everything you’d run on one box in one container.
I definitely recommend using docker compose or similar even in a one node deployment versus just installing and running things on the host linux system like it’s still 1998. Having a single directory to back up and a single file defining all of the services that can easily be redeployed is very convenient.
I definitely recommend using docker compose or similar even in a one node deployment versus just installing and running things on the host linux system like it’s still 1998. Having a single directory to back up and a single file defining all of the services that can easily be redeployed is very convenient.
The performance impact seems completely unjustified for most of the things people do with a personal machine however.
What is the performance impact? Going one page into Google results, I found this paper. Is there a better reference?
> At light workload levels, the native host performs better than Docker. However, as the workload increases, both Docker and the native host show similar performance, with the difference getting smaller
https://www.researchgate.net/publication/376557310_Comparati...
> At light workload levels, the native host performs better than Docker. However, as the workload increases, both Docker and the native host show similar performance, with the difference getting smaller
https://www.researchgate.net/publication/376557310_Comparati...
> What is the performance impact?
DNAT, layered file system access, likely duplication of libraries if you don't pay much attention to your containers.
DNAT, layered file system access, likely duplication of libraries if you don't pay much attention to your containers.
The performance impact is approximately zero.
Replying here as your other question is at max thread depth:
A non virtualized Linux install isn't more locked in than a docker install, as for a bare metal server you are choosing your own OS. I have done the docker thing on a bare metal server, but that's because I wanted to run multiple services on it and isolate them operationally.
A non virtualized Linux install isn't more locked in than a docker install, as for a bare metal server you are choosing your own OS. I have done the docker thing on a bare metal server, but that's because I wanted to run multiple services on it and isolate them operationally.
> A non virtualized Linux install isn't more locked in than a docker install
Again, sorry for my ignorance here, but if not virtualized, how does one move hosting providers otherwise? My experience is limited to either running all the bash commands in an install readme, or installing a docker image.
So there must be something in-between, to recreate a linux install elsewhere?
> Replying here as your other question is at max thread depth:
btw, you can click on the time of the post, and reply there when there is no reply link in the main thread.
Again, sorry for my ignorance here, but if not virtualized, how does one move hosting providers otherwise? My experience is limited to either running all the bash commands in an install readme, or installing a docker image.
So there must be something in-between, to recreate a linux install elsewhere?
> Replying here as your other question is at max thread depth:
btw, you can click on the time of the post, and reply there when there is no reply link in the main thread.
Using dedicated servers doesn't mean you're not using virtualization - it just means you're the one managing it. You control the hypervisor and the vms running on top of it.
Because of that, you're actually less tied to a specific hosting provider since you're not reliant on their APIs to set up and manage your infrastructure.
Even if you're not using virtualization there are still plenty of ways to migrate your servers.
One of the most common approaches (which was the thing before docker took over) is managing servers with an IaC approach using tools like chef, puppet, ansible, saltstack etc.
With IaC you define your entire infrastructure in configuration files and deploy those configs to your host. It's a bit like docker swarm but for managing physical and/or virtual servers instead of containers.
Another popular option, often paired with IaC, is to create your own pre-configured *nix images tailored to your needs. For example, you might have specific images set up for your load balancers, db servers, file hosts, or other roles in your stack.
I've worked at a company where we handled migrations using dd. Technically that's also an option. Wouldn't recommend it tho.
Even if you're not using virtualization there are still plenty of ways to migrate your servers.
One of the most common approaches (which was the thing before docker took over) is managing servers with an IaC approach using tools like chef, puppet, ansible, saltstack etc.
With IaC you define your entire infrastructure in configuration files and deploy those configs to your host. It's a bit like docker swarm but for managing physical and/or virtual servers instead of containers.
Another popular option, often paired with IaC, is to create your own pre-configured *nix images tailored to your needs. For example, you might have specific images set up for your load balancers, db servers, file hosts, or other roles in your stack.
I've worked at a company where we handled migrations using dd. Technically that's also an option. Wouldn't recommend it tho.
>So there must be something in-between, to recreate a linux install elsewhere?
There's a class of tools like chef, ansible, puppet.
Or you can just package your services into debs and run a Debian repo to install them from, same as the base OS
Possibly there's something closer to dockerfiles out there by now as well
>btw, you can click on the time of the post, and reply there when there is no reply link in the main thread.
Cool, thanks!
Makes me puzzled about the point of concealing the reply links, I guess it just adds a bit of friction?
There's a class of tools like chef, ansible, puppet.
Or you can just package your services into debs and run a Debian repo to install them from, same as the base OS
Possibly there's something closer to dockerfiles out there by now as well
>btw, you can click on the time of the post, and reply there when there is no reply link in the main thread.
Cool, thanks!
Makes me puzzled about the point of concealing the reply links, I guess it just adds a bit of friction?
It's a fire extinguisher feature; for when the thread has too many, too fast replies.
If the server hoster supports it (Hetzner apparently does), you can enable KVM and install a previously prepared image.
If the server hoster & hardware supports it, you can login remotely to the server management interface (like HP iLO) and install an image this way.
If you don't have above options or simply don't want to do it this way, you can also bootstrap via SSH. But instead of manually typing in shell commands, you will automate it in some way with custom scripts and/or tools like Ansible.
If you don't have above options or simply don't want to do it this way, you can also bootstrap via SSH. But instead of manually typing in shell commands, you will automate it in some way with custom scripts and/or tools like Ansible.
You can use a configuration manager like Puppet.
https://en.m.wikipedia.org/wiki/Puppet_(software)
https://en.m.wikipedia.org/wiki/Puppet_(software)
Talos Linux, Flatcar Container linux...
Isn't a Docker image an OS image and a bunch of shell commands?
Imagine you'd set up your own server at home.
You'd buy a computer, plug an install USB drive in, and install ubuntu.
Then you'd connect to it via SSH, configure it, maybe install docker and set up your docker containers, etc.
A dedicated server is very similar.
The server is sitting in a datacenter at hetzner, and you usually install an OS with a button in the management UI, sure.
But everything afterwards is the same. You just connect via SSH, install docker or k8s and your services, maybe an nginx, etc.
You also have an option to request KVM access. That allows you to control the server as if you had connected a keyboard and monitor to it. You can even enter the BIOS to diagnose issues, if you'd like.
Personally I've got an install script that automates everything and sets up kubernetes and automated encrypted backups. Then I just deploy everything else with k8s.
You'd buy a computer, plug an install USB drive in, and install ubuntu.
Then you'd connect to it via SSH, configure it, maybe install docker and set up your docker containers, etc.
A dedicated server is very similar.
The server is sitting in a datacenter at hetzner, and you usually install an OS with a button in the management UI, sure.
But everything afterwards is the same. You just connect via SSH, install docker or k8s and your services, maybe an nginx, etc.
You also have an option to request KVM access. That allows you to control the server as if you had connected a keyboard and monitor to it. You can even enter the BIOS to diagnose issues, if you'd like.
Personally I've got an install script that automates everything and sets up kubernetes and automated encrypted backups. Then I just deploy everything else with k8s.
For Hetzner you rent actual physical servers. There is nothing virtualized there, it's real hardware without abstractions.
Thanks, I guess what I meant to ask is that is it normal for people to virtualize their own dedicated server these days by default?
Would this be a best practice to avoid hosting vendor lock-in?
Would this be a best practice to avoid hosting vendor lock-in?
There's no lock-in possible. It's a bare metal Linux machine, you do whatever you want with it; you can replace it with the PC under you desk if you want.
If you want to run k3s, k8s or docker, you can, but personally I find those too complicated. NixOS is much easier to deal with, and achieves the same result.
If you want to run k3s, k8s or docker, you can, but personally I find those too complicated. NixOS is much easier to deal with, and achieves the same result.
Sure, I use CapRover on mine.
Yeah, it's a one-off install. In my case, I did Proxmox[1] for a while with VM's and LXC's, with some of my VM's and unprivileged LXC's running Docker (compose) too because it made the installation of said software easier. It's great, but I'm moving over to switch to Debian with Incus[2] instead of Proxmox. Just for fun mostly.
[1] https://www.proxmox.com/en/ [2] https://linuxcontainers.org/incus/introduction/
[1] https://www.proxmox.com/en/ [2] https://linuxcontainers.org/incus/introduction/
CSPs aren't your cellular provider. You don't get better pricing by telling them you'd switch, because both the AE and yourself (assuming you ever did the math), know it's not a viable option.
Trying to be multicloud by choice, unless you have a very unique use case, which you probably don't, is simply admitting you are incapable of calculating the cost of being multicloud. This would get you horrible pricing, as you just showed your hand.
Trying to be multicloud by choice, unless you have a very unique use case, which you probably don't, is simply admitting you are incapable of calculating the cost of being multicloud. This would get you horrible pricing, as you just showed your hand.
It does feel like a case of the Costco hotdog going up to $2 followed by "grrrr. Thats it! I'm..... going to keep buying it because it is still damn cheap!"
I remember reading something about the Costco hot dog story, quite funny IMO, here's what I just found from 2018:
"I came to (Jim Sinegal) once and I said, ‘Jim, we can’t sell this hot dog for a buck fifty. We are losing our rear ends.’ And he said, ‘If you raise the effing hot dog, I will kill you. Figure it out.’ That’s all I really needed. By the way, if you raised (the price) to $1.75, it would not be that big of a deal. People would still buy (it). But it’s the mindset that when you think of Costco, you think of the $1.50 hot dog (and soda)." [1].
Turns out Costco has a new CEO this year, and again the hot dog topic came to light apparently, lol. This article is from 2024: "'To clear up some recent media speculation, I also want to confirm the $1.50 hot dog price is safe,' Millerchip said." [2].
Sources:
[1] https://www.425business.com/news/costco-ceo-craig-jelinek-on...
[2] https://www.usatoday.com/story/money/shopping/2024/05/31/cos...
Turns out Costco has a new CEO this year, and again the hot dog topic came to light apparently, lol. This article is from 2024: "'To clear up some recent media speculation, I also want to confirm the $1.50 hot dog price is safe,' Millerchip said." [2].
Sources:
[1] https://www.425business.com/news/costco-ceo-craig-jelinek-on...
[2] https://www.usatoday.com/story/money/shopping/2024/05/31/cos...
These stories are marketing mythology.
Think about. Imagine they sell 100 hot dogs per hour. A $0.25 difference means $25/hr in a store doing many orders of magnitude more revenue each hour.
It’s nothing. The way they play it up in the media gets a lot of attention and builds goodwill, but it’s entirely meaningless to their bottom line.
It’s amazing that people eat these stories up, though. I’ve heard so many people repeating this story as if it’s some amazing secret.
Think about. Imagine they sell 100 hot dogs per hour. A $0.25 difference means $25/hr in a store doing many orders of magnitude more revenue each hour.
It’s nothing. The way they play it up in the media gets a lot of attention and builds goodwill, but it’s entirely meaningless to their bottom line.
It’s amazing that people eat these stories up, though. I’ve heard so many people repeating this story as if it’s some amazing secret.
[deleted]
Wait what? Did they actually raise the hotdog price?
I never buy the hotdog but a price raise is indicative of bad times to come.
I never buy the hotdog but a price raise is indicative of bad times to come.
Costco hotdogs have been underpriced for decades. It's an intentional choice, a loss leader. It raising would be mostly meaningless.
But no, last I looked a week ago or so it was still the same price.
But no, last I looked a week ago or so it was still the same price.
It's not a loss leader. They still make a profit on the combo because the ingredients are cheap and it takes almost no labor to prepare on a per-dog basis.
And 1.50 is cheap now but was relatively expensive when the combo launched in 1985 (for comparison, a Big Mac combo with fries and a drink was 2.59, a KFC combo was around $3).
And 1.50 is cheap now but was relatively expensive when the combo launched in 1985 (for comparison, a Big Mac combo with fries and a drink was 2.59, a KFC combo was around $3).
It’s a warmed hot dog, a bun, some self-serve condiments, and it’s served by people who can do hundreds of them an hour.
$1.50 might very well cover the cost, but even if it’s a loss leader then it’s entirely negligible.
The mythology around the hot dog price makes people think they’re getting a screaming deal, though. It’s perfect marketing material.
$1.50 might very well cover the cost, but even if it’s a loss leader then it’s entirely negligible.
The mythology around the hot dog price makes people think they’re getting a screaming deal, though. It’s perfect marketing material.
Is it an option for you to just move the servers to one of Hetzner's Europe location? I guess a lot of high bandwidth applications don't require low latency.
> this is why (HN cough) multi-cloud/dedi k3s
As long as you don't factor in the cost of the effort to make that work as seamless as it sounds.
As long as you don't factor in the cost of the effort to make that work as seamless as it sounds.
DataPacket, ReliableSite, HiVelocity, and FDCServers among others have great bandwidth prices.
Can vouch for HiVelocity under the previous owners anyway, not sure what's going on over there now. My companies launch was bandwidth and compute intensive and they handled it well.
Being cloud-agnostic is highly valuable. It's also possible to make determinations on where your needs are served best pro-actively, independent of service plan like this.
Thats not a bad overage price, but then again cutting from 20 to 1 while raising prices seems fucking insane
>multi-cloud/dedi k3s is great
As someone with the exact same setup, thank you for strengthening my confirmation bias.
As someone with the exact same setup, thank you for strengthening my confirmation bias.
> Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced.
I may know one of the culprits -- whom I will leave unnamed here. But the company, who is fairly popular, built out their own CDN via putting a bunch of nginx caching proxies on various Hetzner servers around the world. It apparently was really cheap and very effective. Given that they were bootstrapped and this was prior to Cloudflare really being that popular, it was a great strategy. This was true like 8 years ago, so maybe it has changed in the meantime.
I may know one of the culprits -- whom I will leave unnamed here. But the company, who is fairly popular, built out their own CDN via putting a bunch of nginx caching proxies on various Hetzner servers around the world. It apparently was really cheap and very effective. Given that they were bootstrapped and this was prior to Cloudflare really being that popular, it was a great strategy. This was true like 8 years ago, so maybe it has changed in the meantime.
Sounds like a completely legitimate use though. Hetzner were widely telling people about that 20TB limit, so why would they be surprised when people use them as CDN boxes?
Where's the surprise? It's the classic business 2-step - drum up interest with "too good to be true" features, then cut them back. The marginal customers who need those features leave (and are too expensive to keep), everyone else is used to your product and stays.
Hetzner has been in this business for a very long time and I’m sure half the German small to medium sized companies use Hetzner. They have costs as well and with virtual servers they just assumed the usage pattern was similar to their eu data center and that didn’t work out so they increase the prices a bit. So that on average their calculation works out again.
Like, this is not Netflix tripling the prices over a few years.
Like, this is not Netflix tripling the prices over a few years.
That or they just didn't dimension well their prices and now it's biting them back.
I prefer to assume naivety over malice/negligence.
I prefer to assume naivety over malice/negligence.
MBA value extraction 101.
Until you remember that marketing is a separate department from finance which is a separate department from ops/engineering.
The engineers said 20TB in aggregate was fine but likely didn’t consider the “bad apples”. Marketing obviously wants to use the biggest numbers and then finance comes in with the hammer and dev points to egress as an simple way to upset rhe fewest number of real customers.
The engineers said 20TB in aggregate was fine but likely didn’t consider the “bad apples”. Marketing obviously wants to use the biggest numbers and then finance comes in with the hammer and dev points to egress as an simple way to upset rhe fewest number of real customers.
It does make sense: The average was low enough that a 20TB cap worked. Then marketing started boasting about the 20TB limit and attracted a bunch of high-bandwith customers, thereby driving up the average making 20TB decidedly "not fine".
As an engineer, if you don't qualify whether your answer is average or max, you've messed up.
I'm confident the marketing folks would just say something akin to "shut up nerd", cash their bonuses, and leave you with the problem.
As an engineer, NEVER give out averages without checking that they are consistent with the distribution of the actual data.
I think culprits is a poor choice of words since it means someone suspected of a misdeed. I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan. But in this case Hetzner set a specific data usage amount you are paying for so using that amount is not a misdeed.
> I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan.
I couldn't, don't call it unlimited if it's not unlimited, using what you paid for is not a "misdeed".
I couldn't, don't call it unlimited if it's not unlimited, using what you paid for is not a "misdeed".
Well, I doubt an extra $1/TB is going to stop Hetzner from being very price effective.
Then why does this only apply in the US? Are they saying EU customers are well behaved?
It also may depend on peering arrangements Hetzner has. If EU ISP more inclined to peer with Hetzner than US one bandwidth in EU will be cheaper for them.
- in EU there are large peering exchanges to swap traffic
- in USA no peering exchanges exists and you need to pay for your traffic most of the time. Few big operators in US and they enforce this.
Looks like some deal wasn't renewed and they lost a big chunk of cheap pipe or/and some of their upstream providers decided to do something with routing.
- in USA no peering exchanges exists and you need to pay for your traffic most of the time. Few big operators in US and they enforce this.
Looks like some deal wasn't renewed and they lost a big chunk of cheap pipe or/and some of their upstream providers decided to do something with routing.
Also, Hetzner is way bigger in the EU than in the US. Good access to services hosted by Hetzner is thus more relevant to EU ISPs, because customers in the EU will probably use more services hosted on Hetzner infra. This gives Hetzner more leverage in the EU to negotiate beneficial conditions with regard to its uplinks and peering agreements.
Hetzner colo's for their US servers/locations. Perhaps their US bandwidth rates are much higher than their own DCs?
probably streaming platforms have better content on the US and everyone wants to exit there? so they are mostly serving US traffic for several vpns all over the world connecting to CDNs in the US.
Isn't Hetzner impacted in Europe at the moment from a cable cut? Wouldn't surprise me if there is a wave of people moving stuff over to the US because of that. I don't know much about cloud though. I believe "The Ship has arrived and repairs are underway, which will still take some time." and "the repair may take up to two weeks.".
Hetzner did pay somewhere around 20% of the cost of C-Lion1, so it wouldn't be surprising if it has hurt hem.
Last I heard from Cinia was that the cable should be fixed by the end of November, so sometime within the next 55 hours.
8 years ago, hetzner had no cloud offerings
The dedicated servers still have 20 tb traffic included
The dedicated servers still have 20 tb traffic included
What? Afaik they are unmetered. And real unmetered - there are people who are saturating their connections 24/7 while using them.
“Culprits” because they used the service they paid for within its advertised limits?
It’s the same with cloud storage providers. First give out a massive amount of storage and rapidly gain users, then cut it down after blaming people for “abusing” it. How about you advertise your correct capacity to begin with?
They are simply deflecting blame for their own enshittification.
It’s the same with cloud storage providers. First give out a massive amount of storage and rapidly gain users, then cut it down after blaming people for “abusing” it. How about you advertise your correct capacity to begin with?
They are simply deflecting blame for their own enshittification.
>“Culprits” because they used the service they paid for within its advertised limits?
"Culprits" because it was their (legal) use of the service that made Hetzner rethink and change their service plan.
"Culprits" because it was their (legal) use of the service that made Hetzner rethink and change their service plan.
So then the culprits are the company’s own engineering and marketing departments for not correctly anticipating user demand.
Sure .. also them.
Having said that, I am usually empathetic to these kinds of 'unlimited' deals because even though they aren't really 'unlimited', they do tend to be generous to the average use-case and average user .. Inevitably, and unfortunately, someone decides to test the limits and the entire thing collapses.
It reminds me of the Blockbuster "No more late fees" policy, which was a really good customer-friendly policy (speaking as someone who regularly returned rentals late) .. but then they were sued because an aspect of the policy had Blockbuster charging the cost of the rental to the customer if it wasn't returned in some period of time .. and because that charge looked like a 'late fee' they got sued. Urgh.
Having said that, I am usually empathetic to these kinds of 'unlimited' deals because even though they aren't really 'unlimited', they do tend to be generous to the average use-case and average user .. Inevitably, and unfortunately, someone decides to test the limits and the entire thing collapses.
It reminds me of the Blockbuster "No more late fees" policy, which was a really good customer-friendly policy (speaking as someone who regularly returned rentals late) .. but then they were sued because an aspect of the policy had Blockbuster charging the cost of the rental to the customer if it wasn't returned in some period of time .. and because that charge looked like a 'late fee' they got sued. Urgh.
It’s not any more possible to correctly anticipate all pricing structure vulnerabilities, than it is to correctly anticipate all program and API security vulnerabilities. There is always a statistical chance of novel outcomes when humans are involved.
Or you take your best shot and then adjust as needed.
Want to give a clue on who the culprit is?
I was curious and checked if they are still using Hetzner. It appears not, so I can share who it was. It was https://artstation.com. Basically heavily oriented towards serving static images, so the CDN could have been really expensive. Doing a reverse IP lookup on cdn.artstation.com servers now resolves to Cloudflare and it has cloudflare headers on the response.
This is typical of Hetzner, if a product SKU is losing money they very quickly make changes, even going as far as to discontinue the product entirely (eg. GPU servers). They definitely don't seem to be a fan of loss leaders.
I'm guessing somehow the traffic usage patterns of their USA customers was very different to their EU counterparts, or the cost of expanding network capacity was a lot higher than anticipated.
It's a bit of a shock for sure but it seems this model is a big part of how they can maintain their slim margins.
I'm guessing somehow the traffic usage patterns of their USA customers was very different to their EU counterparts, or the cost of expanding network capacity was a lot higher than anticipated.
It's a bit of a shock for sure but it seems this model is a big part of how they can maintain their slim margins.
I have no complaints at all about this model. They work out the cost of providing a service, then they charge that cost plus a markup. They keep doing things that make them money. They stop doing things that don't make them money.
It seems like a straightforward way to run a business.
It seems like a straightforward way to run a business.
Yep they're the technology equivalent of a discount supermarket. Everything is commoditized to the extreme.
Breath of fresh air in the modern cloud era tbh.
Breath of fresh air in the modern cloud era tbh.
I have one big complaint and one little one. The big complaint is that they didn't even give one business day's notice, and the little complaint is that they raised prices at the same time they cut what they were offering by 20x, instead of doing one at a time.
They're giving two business days' notice for new product and three months for existing product?
December 1st's change isn't just for new customers. It's for newly-created or rescaled servers belonging to existing customers too, and it's plausible that those operations might happen a lot for some customers. And Thanksgiving and Black Friday are holidays for almost all American tech workers, so I'm not counting them as business days.
Normally that would be ok, but considering the way many systems are setup to load balance and quickly spin up new servers and spin down un-needed servers on the fly, one business day would not give you enough time to revamp your system to work with a different provider.
The bandwidth market is very different between EU and USA, maybe they weren't prepared for the much higher prices in USA? I'm pretty used to having a 100 Mbps connection to our servers that we can use without any strings attached. Even on the lowest tier. (Not Hetzner customer but been thinking about it)
To be fair, given how cheap a lot of Hetzners products (especially Server Auction, my beloved) are compared to the competition, not wanting to have loss leaders seems reasonable to me
Rather, the backbone providers dont do peering agreements and the traffic is very expensive, especially in the post-zirp inflation period. Europe is different - everybody peers with everybody so traffic is dirt cheap.
Traffic over-usage is $1 per TB, so this is still quite fair, only in singapore is traffic really expensive at $8/TB.
> only in singapore is traffic really expensive at $8/TB.
Expensive by Hetzner standards but still cheap by cloud standards, egress from Singapore EC2 instances is between $80 and $120 per TB for example.
Expensive by Hetzner standards but still cheap by cloud standards, egress from Singapore EC2 instances is between $80 and $120 per TB for example.
$80!! EC2 is a scam
It's $120/TB for the first 10TB in a month, so you need to be spending >$1200 just on bandwidth every month before you even get the "discounted" rates.
I know serverless is really expensive, but Vercel is about $400 for 1TB
Vercel is $150: https://getdeploying.com/reference/data-egress
Vercel heavy user here. They have a very misleading pricing. It's "starting at $150", it varies depending on the region. I end up paying $400 / TB as we have a very international website.
[deleted]
[deleted]
The real savings only come in bulk
so still order or orders of magnitude cheaper than the the big 3 hyperscalers.
AWS EC2: 100GB included and then $90 per TB.
GCP Premium Tier: $120 per TB
GCP Standard Tier: $85 per TB
AWS drops to $50/TB - still 2x to 10x maybe? A lot of CDNs cost a surprising amount or hide pricing.
In Singapore and Asia in general traffic is very expensive, because there are no peering exchanges. So you need to pay for each Mbps you use. Big ISPs there are oligopolies and aren't too fast or keen to work with you.
Am I calculating right that 20TB per month means around ~60Mbits per second for 24/7? Not a network expert, but it is hard to see how this could be sustainable for less than €5/month.
Sounds a bit like the usual case where company is able to give a generous offering because most customers utilize just a small portion of it. Maybe with the attention they have been getting, they have attracted more bandwidth hungry customers.
Sounds a bit like the usual case where company is able to give a generous offering because most customers utilize just a small portion of it. Maybe with the attention they have been getting, they have attracted more bandwidth hungry customers.
That is what they (explicitly) said in the email: users using next to no bandwidth were offsetting the costs for the heaviest users. That was no longer sustainable.
I think I'm missing something then. If that were true, wouldn't you lower the network allowance and then make the product cheaper or at least priced the same? This would have the no-traffic users paying "their fair share" and the overage costs (or higher traffic addon) would make up for the heavy users.
The current plan makes everything more expensive for everyone. They would do this if a) they never had a sustainable model in the first place or b) they were just being greedy
The current plan makes everything more expensive for everyone. They would do this if a) they never had a sustainable model in the first place or b) they were just being greedy
They recently entered the US market, so almost surely there was some honeymoon period (ie. investment to gain market share), but likely there's a significant difference between how the IXP/bandwidth/peering market works in the US compared to their home market (EU).
Pricing is fairly similar, particularly in ashburn. Europe can be a little cheaper, but usually not a massive amount in it.
Hm, interesting! Then probably lots of "bandwidth conscious" folks pounced on on their fancy new PoP.
Whatever the reason, rather strange to do this without even a few days of notice.
Whatever the reason, rather strange to do this without even a few days of notice.
They already offer it on their European servers and it's still unchanged. Also, just because it's included doesn't mean everyone is using it to their full capacity.
Internet is much cheaper in Europe than in most other regions and there's more of it. Europe is effectively the center of the Internet. Most of the time (in regions like Asia-Pacific and Africa) this is simply due to having more time and money to build it, but when comparing Europe to the USA, it's probably because of regulatory structure - more competition, less monopolization.
What do you mean? Is peering/transit actually more expensive in the US? Residential internet isn't really relevant here, and if it was, it would still not really make sense considering that Germany (where they have their main datacenter) has much worse internet than the US according to most speed/bandwidth averages. Because if you are not referring to residential ISP, I don't think there's any peering/transit provider monopoly in the US
Yes. I suspect that because of the low number of residential ISPs there's less internet infrastructure in general and less competition. Who needs peering if there's only Comcast? You point out that networks can exist that don't serve residential customers, but who do they serve - mostly other networks which don't exist because they can't serve residential customers. The entire US is just the access region for a few networks to backhaul to NYC, Chicago and SF, not a full fledged network itself.
Is that much? I use a German competitor, Contabo, and they offer 32TB even with their entry-level VPS for 4.5€ plus taxes.
IIRC a while back one of the CDNs had a post about their transit costs; they worked out _way_ lower in Europe than in the US. I don’t think Hetzner Germany is increasing prices here, just the US one?
Possible theory why they did it: To my understanding, you don't pay for traffic, you pay for network capacity. Maybe US instances aren't getting the uptake they had hoped for, and they are looking lower some costs by reducing their network capacity (they are a very German company and care a lot about efficiency).
Hetzner is very cheap and still profitable because classic "economy of scale" and vertical integration. They own, build, and operate all their data centers. This comment goes into more details[1], but it's possible this doesn't really work out in a foreign location like US.
[1]: https://forumweb.hosting/13663-why-are-hetzners-dedicated-ho...
Hetzner is very cheap and still profitable because classic "economy of scale" and vertical integration. They own, build, and operate all their data centers. This comment goes into more details[1], but it's possible this doesn't really work out in a foreign location like US.
[1]: https://forumweb.hosting/13663-why-are-hetzners-dedicated-ho...
Counterpoint: They could've lowered their capacity without changing plans if they were indeed seeing lower uptake, and increase capacity as/when it's needed if it did indeed ever pick up. They wouldn't be the first provider to put products out of stock whilst they scale back up.
That doesn't explain a 20x regional pricing difference for the same service (outgoing bandwidth). It's either a malicious change to extract exorbitant rent from customers (hostages at that point) or buying enough bandwidth is whole lot more expensive than they calculated and it reflects costs to provide the product (a virtual machine a specific traffic cap).
If its mostly the later case they really fucked up their customer communication. They should care enough to provide their customers with time to respond and transparency to (re-)earn the trust a hosting provider requires.
If its mostly the later case they really fucked up their customer communication. They should care enough to provide their customers with time to respond and transparency to (re-)earn the trust a hosting provider requires.
If it were malicious, they would have increased prices in Europe as well. They could try and extract money from European customers just as much as American ones.
And the bandwidth pricing is still quite cheap, $0.001/GB. Major cloud providers usually charge nearly 100x that (AWS 90x, GCP at 85x or 120x depending on whether you want standard or premium, Azure at 80x or 87x depending on standard or premium).
You used to get 20TB for free and now you get 1-8TB free. If you had to pay for 19TB, that's another $19. If you had to pay for 19TB from a major cloud provider, that'd be $1,556-2,335. Even if you had DigitalOcean, they'd be charging you $0.01/GB (10x) and you'd be hit with a $190 bill.
I think the issue is that in the US, they don't have their own network. They peer with a single company in Virginia for 200Gbps and then pay for transit on 1Tbps. That's a lot of transit to be paying for - and 5x more transit than peering. In Europe, it looks like they have their own network between Finland, Germany, France, Amsterdam, UK, Austria, and Czech Republic. They also have a much better ratio of peering to transit. So they can use their network to carry the traffic to a lot of Europe and then maybe they have peering arrangements to handle most of their traffic when they need to hand it off.
In Europe, they're more likely to face owner economics while in the US, they're effectively renters - and the more their customers use, the more it's going to cost them.
If this were really a cash grab, it's a pretty terrible cash grab. It will certainly impact some users, but the maximum it will impact any user is $19/mo (if they're on a server with only 1TB of included traffic). But most people don't use many TB of traffic. Consuming all Bluesky posts in zstd compressed JSON is 30GB/mo. These weren't servers that had unlimited traffic on a 1Gbps port. They had a 20TB limit. If you were hitting that and intend to continue hitting that, it's another $12-19/mo.
If I had to guess, I'd say that they probably thought they'd expand more in the US and build out a fiber network here (as OVH has done), but that didn't happen and now they're looking at continuing to pay transit for the foreseeable future. Though I feel like the optics of this are pretty bad given that most people probably use less than 1TB and those users will still feel like something is being taken from them (even if they were never using that much anyway).
And the bandwidth pricing is still quite cheap, $0.001/GB. Major cloud providers usually charge nearly 100x that (AWS 90x, GCP at 85x or 120x depending on whether you want standard or premium, Azure at 80x or 87x depending on standard or premium).
You used to get 20TB for free and now you get 1-8TB free. If you had to pay for 19TB, that's another $19. If you had to pay for 19TB from a major cloud provider, that'd be $1,556-2,335. Even if you had DigitalOcean, they'd be charging you $0.01/GB (10x) and you'd be hit with a $190 bill.
I think the issue is that in the US, they don't have their own network. They peer with a single company in Virginia for 200Gbps and then pay for transit on 1Tbps. That's a lot of transit to be paying for - and 5x more transit than peering. In Europe, it looks like they have their own network between Finland, Germany, France, Amsterdam, UK, Austria, and Czech Republic. They also have a much better ratio of peering to transit. So they can use their network to carry the traffic to a lot of Europe and then maybe they have peering arrangements to handle most of their traffic when they need to hand it off.
In Europe, they're more likely to face owner economics while in the US, they're effectively renters - and the more their customers use, the more it's going to cost them.
If this were really a cash grab, it's a pretty terrible cash grab. It will certainly impact some users, but the maximum it will impact any user is $19/mo (if they're on a server with only 1TB of included traffic). But most people don't use many TB of traffic. Consuming all Bluesky posts in zstd compressed JSON is 30GB/mo. These weren't servers that had unlimited traffic on a 1Gbps port. They had a 20TB limit. If you were hitting that and intend to continue hitting that, it's another $12-19/mo.
If I had to guess, I'd say that they probably thought they'd expand more in the US and build out a fiber network here (as OVH has done), but that didn't happen and now they're looking at continuing to pay transit for the foreseeable future. Though I feel like the optics of this are pretty bad given that most people probably use less than 1TB and those users will still feel like something is being taken from them (even if they were never using that much anyway).
American traffic is expensive and they scarcely do peering agreements. And the backbone providers are probably jacking up the prices to boost revenue in these inflationary times. Eu users are not affected because the Eu invested heavily in the internet backbone and everybody peers with everybody there. In the US the isps cripple any effort to improve the internet infrastructure for profit.
This seems to be a huge PR blunder. As a single data point, I have to say that my first reaction is illogical.
I have two hetzner shared instances and I am royally pissed by the 20x reduction in traffic allowance. It is also irrelevant to me: over the last 12 months I never exceeded 1TB. My unhappiness on the traffic reduction is purely of a "what if I start using more" type. For which two rational answers is "well you can explore alternatives then" and "d'oh, your average is way under 100GB, it's not going over 1TB". But I still started looking at alternatives.
My feeling is that the reduction is aimed at a small group, but upsets a much larger set of customers who now will start looking for alternatives. Which indicates a typical marketing screwup. My 2c.
I have two hetzner shared instances and I am royally pissed by the 20x reduction in traffic allowance. It is also irrelevant to me: over the last 12 months I never exceeded 1TB. My unhappiness on the traffic reduction is purely of a "what if I start using more" type. For which two rational answers is "well you can explore alternatives then" and "d'oh, your average is way under 100GB, it's not going over 1TB". But I still started looking at alternatives.
My feeling is that the reduction is aimed at a small group, but upsets a much larger set of customers who now will start looking for alternatives. Which indicates a typical marketing screwup. My 2c.
Fair I felt similar about it.
But then, I have looked at alternatives before they even offered their US locations (and also for a CA one) and couldn't find anything decent for even nearly the price.
So we can pay a little more and get a little less, or move somewhere else and either pay a lot more or get a lot less
But then, I have looked at alternatives before they even offered their US locations (and also for a CA one) and couldn't find anything decent for even nearly the price.
So we can pay a little more and get a little less, or move somewhere else and either pay a lot more or get a lot less
OVH is comparable and has locations in North America.
Very well put! I had the exact same reaction as you: even though I don't even use 1TB/month, I found myself looking at alternatives simply b/c of the cognitive jolt of seeing the 20x reduction. Sometimes I wonder if companies don't consider the psychological impact of changes enough. No, logically, this change means a meager price increase for me, but if it caused me (and at least one other) to think about competing offerings, then like you said, indicates a marketing screwup.
That's not to say they have to keep offering that much traffic if they're losing money (of course they don't), but the way you make changes matters.
That's not to say they have to keep offering that much traffic if they're losing money (of course they don't), but the way you make changes matters.
As a customer of Linode, I feel like I'm getting a lot, maybe too much, for the money I pay.
For $5 per month, I have a CPU running continuously near 100% utilization, training and retraining L1/L2/L3-CPU-cache-resident transformers, looking for patterns in futures and options markets.
This kind of extreme resource utilization is becoming more common, and these businesses have to adapt to stay profitable.
I expect Linode to change the price on me, eventually.
For $5 per month, I have a CPU running continuously near 100% utilization, training and retraining L1/L2/L3-CPU-cache-resident transformers, looking for patterns in futures and options markets.
This kind of extreme resource utilization is becoming more common, and these businesses have to adapt to stay profitable.
I expect Linode to change the price on me, eventually.
I don’t find that to be all that impressive.
I’m immediately saving money with the server I built out of mostly used parts and threw in my closet compared to VPS solutions.
The only reason it’s near 100% utilization is because $5 VPS instances have barely any computing power assigned to them.
For the same price as one game server I’m running something like 5-8 VMs at once. I can utilize 128GB of RAM and 6/12 real CPU cores (Ryzen 3600).
I’m immediately saving money with the server I built out of mostly used parts and threw in my closet compared to VPS solutions.
The only reason it’s near 100% utilization is because $5 VPS instances have barely any computing power assigned to them.
For the same price as one game server I’m running something like 5-8 VMs at once. I can utilize 128GB of RAM and 6/12 real CPU cores (Ryzen 3600).
What is your annual electricity cost for that server in your closet?
I’d like to know this too, along with how you mitigate traffic spikes while using the internet connection as both your home internet and your public internet.
There are no traffic spikes, it’s not serving web content to the public internet.
I probably should have been a little more clear that I don’t think VPS systems are bad or anything, it’s just that they aren’t well-suited to game servers and hardware in my closet is well-suited to it.
I was spending almost 20 euros a month for a single Hetzner cloud instance that performed worse than all my gaming VMs on my closet computer.
But if I was serving a public website or app I would probably not want it going through my home internet.
I probably should have been a little more clear that I don’t think VPS systems are bad or anything, it’s just that they aren’t well-suited to game servers and hardware in my closet is well-suited to it.
I was spending almost 20 euros a month for a single Hetzner cloud instance that performed worse than all my gaming VMs on my closet computer.
But if I was serving a public website or app I would probably not want it going through my home internet.
I don’t know but probably a bit under $5 a month if 35 watts is my guess at the system idle. I haven’t bothered metering it.
Completely unrelated, but I’m surprised how many people actually use the Ryzen 3600, from desktops to servers, it seems to be everywhere.
I expect to see the same thing from the 5600. With AMD still selling 5000-series CPUs and their prices so low, they must be incredibly tempting to anyone who needs a lot of CPUs.
What's the justification for this approach? Buy an old NUC with some cheap Celeron in it, install Hamachi if you need remote access, and it'll pay for itself in a couple months.
Seeing as they are paying $5 a month, how do you expect buying a NUC to pay for itself in a few months? Where are you finding NUCs for $20 with free electricity?
Any decommissioned office PC from eBay will be faster than $5 linode. For example search for optiflex https://www.ebay.com/sch/i.html?_nkw=optiplex+pc&_udhi=30 They're not too power hungry either if you make sure not to go for i7s
But it'll cost $100+/yr in electricity.
There are optiplex models running at 14W or less when idle. In my area that's ~$20 per year. (It close to half of that in practice with solar)
Based on your ebay link, the cheapest PCs are around $50 shipped. Including the electricity, that's $70/year. So you'd need more than a year to have this approach pay off, and additionally you will have saddled yourself with managing a physical machine rather than have someone else abstract it for you.
Doesn't seem to pass the "it'll pay for itself in a couple months" claim.
Doesn't seem to pass the "it'll pay for itself in a couple months" claim.
That link sets the max price to $30. (I see a few at $18) I'm not sure how you found a $50 item there. Ignore shipping if you're adding that in, there's lots of local ones you can find on Facebook market and similar sites.
...no? My NUC draws about 18 watts under load, which is about $24 per year at typical American energy rates. It cost me 30 bucks. So, it's paid for itself after about 10 months (5n = 2n + 30), and it's way faster than a $5 VPS.
I understand Tailscale is the new Hamachi.
I'm aware, but I stick with Hamachi because I know how to use it, and I have yet to run into a situation that would motivate me to replace it.
You have a CPU? Or a core, that is 1 of many cores on a system where your VM can be and will be pre-empted by the host node’s scheduler without your VM having any control or insight into the process?
> This kind of extreme resource utilization is becoming more common, and these businesses have to adapt to stay profitable.
The fact a usage is common doesn't mean it is profitable or that it needs to be supported for a hosting service to be profitable.
Mining crypto used to become quite common as well a decade ago and tall hosting services banned this exactly for because there was no way they could be profitable and offer decent QoS.
The fact a usage is common doesn't mean it is profitable or that it needs to be supported for a hosting service to be profitable.
Mining crypto used to become quite common as well a decade ago and tall hosting services banned this exactly for because there was no way they could be profitable and offer decent QoS.
It's most likely a vCPU. So even the caches are shared.
If I recall Linode has a "dedicated core" feature which reserves a CPU core for you, but I doubt that is $5/month so you're likely right. User must be quite lucky to have no utilization on host for process to not get cache-evicted.
Why is this attitude so commonplace?
Servers are worthless to a hosting company without utilization. It’s in their best interest to have them pegged 100%. Like airplanes - they don’t make money when they are empty.
Why do so many in this thread think “I am using 100% of what I pay for! They are bound to change it soon!” That’s not how it works. If I offer you service for a fee, I’m going to allow you to use 100% of that service for that fee.
Servers are worthless to a hosting company without utilization. It’s in their best interest to have them pegged 100%. Like airplanes - they don’t make money when they are empty.
Why do so many in this thread think “I am using 100% of what I pay for! They are bound to change it soon!” That’s not how it works. If I offer you service for a fee, I’m going to allow you to use 100% of that service for that fee.
Completely backwards. A server that is being paid for but not utilized is the best-case scenario for a hosting provider – revenue the same, lower costs (electrical/cooling/etc).
in mba brain, but hosting providers want as much utilization because it means people are going to use more over time.
They want more users, not more utilization. Utilization is the effect of users not its cause. They are happy to have utilization increase as long as they keep getting more users. If the utilization increases while number of users remain constant because people (like OP) are using CPU intensive programs then that is not in their interest.
Not really. If I have a profitable app that only uses 10% CPU on average, and you have a profitable app that uses 90% CPU on average (because it is a more CPU intensive program), both of us are going to keep paying our hosting provider indefinitely, but you are less profitable to them.
You seem oddly emotional about this. Also, you're wrong.
Hosting companies count on a certain <100% average CPU usage, and this factors in to their business model in the shared CPU plans. That's literally why those plans exist, because they are getting more users on a machine by counting on less than 100% usage. The users get a lower price and the risk of getting throttled in certain situations. Nowhere in their product description do they promise you can use 100% all the time. If you want that, you go to their dedicated plans.
Don't get so emotional, but if you do, at least make sure you aren't wrong first.
Hosting companies count on a certain <100% average CPU usage, and this factors in to their business model in the shared CPU plans. That's literally why those plans exist, because they are getting more users on a machine by counting on less than 100% usage. The users get a lower price and the risk of getting throttled in certain situations. Nowhere in their product description do they promise you can use 100% all the time. If you want that, you go to their dedicated plans.
Don't get so emotional, but if you do, at least make sure you aren't wrong first.
[deleted]
I think others have said it well enough, but you're mixing the VPS business model up with cloud. It's in the interests of cloud providers to have you utilising resources because that's how you're billed. VPS and dedicated servers just give you a bill for potential utilisation, regardless of how much or little you choose to actually use (excluding bandwidth overages).
No competitively priced VPS* is charging a price that works when everyone is fully utilizing their entire quota at 100%...
It's not nefarious: you pay a bit more than the compute you use would cost on a fractional basis, and in exchange the cost of entry is dramatically reduced, spikes in usage get absorbed, etc.
It's a win for everyone involved unless usage patterns shift and suddenly there's never a surplus to go around. At that point prices will quickly climb to roughly what dedicated resources cost.
(*and frankly it's not just VPS, a lot of cloud services rely on everyone not trying to max out their quota at the same time to even function, let alone profit.)
It's not nefarious: you pay a bit more than the compute you use would cost on a fractional basis, and in exchange the cost of entry is dramatically reduced, spikes in usage get absorbed, etc.
It's a win for everyone involved unless usage patterns shift and suddenly there's never a surplus to go around. At that point prices will quickly climb to roughly what dedicated resources cost.
(*and frankly it's not just VPS, a lot of cloud services rely on everyone not trying to max out their quota at the same time to even function, let alone profit.)
Maximized by flat rate payer, minimized by pay-per-use.
You want to maximize it, the hosting definitely prefers someone who pays and doesn't use, so they can double-book that server.
You want to maximize it, the hosting definitely prefers someone who pays and doesn't use, so they can double-book that server.
This is absolutely untrue in both a naive and more nuanced sense. Naively, it's not like an airplane, it's like an airplane rented by the month: the owner of the airplane collects rent whether or not the airplane is flying. More nuanced, many VPS providers don't sell you 100% CPU utilization; they sell you a compute budget. AWS actually enforces this; I don't know about other providers. But if you're exceeding your compute budget and the provider doesn't enforce the budget limit, you can expect the system to break at some point.
On the other hand airlines regularly overbook flights, and residential ISPs don’t have nearly enough bandwidth for every customer to max it out.
I'm not sure why you compare this to airplanes. Airplanes fuel consumption is very expensive if the plane is empty relative to profit. They need to put bums in seats for the flight to be viable commercially.
On the other hand, a flat out rack takes more power than an idle one. So this isn't the same thing at all.
On the other hand, a flat out rack takes more power than an idle one. So this isn't the same thing at all.
Yeah, the GP comment is so dumb. Most commercial aircraft are owned by a few leasing companies. They would love if lessors didn't fly the aircraft, but paid leasing fees!
Don't think we're close to that point yet. You can still get the same server for free from oracle free tier if you're willing to put up with god awful enterprisey control panel.
Also that linode CPU is virtualized (i.e. at least some of that cache is shared).
Also that linode CPU is virtualized (i.e. at least some of that cache is shared).
Oracle has people jumping all kinds of hoops to get that service too. Just like Hetzner. Took me a few tries with different credit cards then tried to get one for my friend with their card and nothing would work. Great free service though. That ampere vm with 24 gb ram is quite capable.
Off topic, your website is interesting, I wish I knew how to interpret it "commitment of traders". Maybe there is a blog post somewhere?
One core isn't very expensive, and when we look at modern super dense CPUs full load is less than 5 watts per core.
Couldn't you just buy a used decent PC and run this on its CPU + GPU and get faster/better results?
While all the guesswork ? They explain clearly why this is happening. And it seems fair.
"With the new tariff structure, we want to make conditions for our customers around the world as fair as possible. To do that, we will calculate our prices based on local conditions in Europe, Singapore, and the USA. Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced. The new prices will give our customers the best possible price for the resources they use."
"With the new tariff structure, we want to make conditions for our customers around the world as fair as possible. To do that, we will calculate our prices based on local conditions in Europe, Singapore, and the USA. Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced. The new prices will give our customers the best possible price for the resources they use."
It doesn't seem so fair if you didn't know that in advance. I don't do research on what actual costs a provider I sign up with has, and whether a product might be subsidized in some way. If I relied on those 20TB a month I now got a 100 to 400% price increase.
I really like hetzner and their service quality, but that does leave a bad taste... I seems like they badly miscalculated.
I really like hetzner and their service quality, but that does leave a bad taste... I seems like they badly miscalculated.
Well, a month ago they did not know that insane tariffs are going to exist suddenly. They are telling this pretty much as soon as they could.
There are no tariffs yet until January, and they don’t apply to Germany, anyway. This has nothing to do with tariffs.
This increase only affects the US regions, so it very likely is in preparation for additional tariffs.
They are telling in advance: changes are effective from February for old contracts.
If the location (USA) is key, then fair enough but if not, just buy one of their dedicated servers and get unlimited[1] traffic.
Only outgoing external traffic is counted.
[1] https://docs.hetzner.com/robot/general/traffic/
Only outgoing external traffic is counted.
[1] https://docs.hetzner.com/robot/general/traffic/
This thread is where I'm learning that American English uses tariff mainly for import tariffs. Here in India, the most common usage of it is to talk about telecom tariffs - mainly mobile, sometimes broadband. So it didn't even occur to me when reading the question that it might have anything to do with import tariffs, until I read some comments that misunderstood it that way.
Not just mainly, but exclusively in my experience. (Import/export tariffs.)
Until this thread, I have never encountered the term "bar tariff" for a list of drink prices, or "energy tariff" instead of rate. Those uses are simply not American English, and you would be misunderstood.
Hetzner is a German company so I find myself wondering if this is a British usage, or a mistranslation of the German word "tarif" that should be "rate"? (A common mistranslation category known as "false friends".)
TIL: https://en.wikipedia.org/wiki/Tariff_(disambiguation)
Until this thread, I have never encountered the term "bar tariff" for a list of drink prices, or "energy tariff" instead of rate. Those uses are simply not American English, and you would be misunderstood.
Hetzner is a German company so I find myself wondering if this is a British usage, or a mistranslation of the German word "tarif" that should be "rate"? (A common mistranslation category known as "false friends".)
TIL: https://en.wikipedia.org/wiki/Tariff_(disambiguation)
One of the most frustrating things about Duolingo is that they refuse to have an International English setting for the language you are learning from. I’m trying to learn french but WTF is a ‘stroller’ or an ‘eggplant’ or even more frustrating are the ones where the word is almost the same in the UK as in France e.g ‘athlétique’ in French is ‘athletics’ in UK English but ‘track’ in US English.
It's perfectly acceptable British English.
21 occurrences of "tariff" on one of British Gas' pages: https://www.britishgas.co.uk/energy/guides/off-peak-electric... ("Your energy provider may offer time of use tariffs and cheaper night-time electricity rates.")
21 occurrences of "tariff" on one of British Gas' pages: https://www.britishgas.co.uk/energy/guides/off-peak-electric... ("Your energy provider may offer time of use tariffs and cheaper night-time electricity rates.")
Good to know, so it's an American vs. British thing. Thanks!
Is there any further distinction between a "tariff" and a "rate" in British English? The example sentence you provide uses both, which makes me wonder if there's even more to the picture here.
Is there any further distinction between a "tariff" and a "rate" in British English? The example sentence you provide uses both, which makes me wonder if there's even more to the picture here.
Without checking, my feeling is a "tariff" is the whole contracted agreement, and a "rate" is a part of it.
An EV electricity tariff might have a cheap night rate, and a more expensive day rate. Another tariff might be entirely variable rate (price changes every hour).
Wiktionary defines a tariff as "A schedule of rates, fees or prices." so I think my feeling is correct.
An EV electricity tariff might have a cheap night rate, and a more expensive day rate. Another tariff might be entirely variable rate (price changes every hour).
Wiktionary defines a tariff as "A schedule of rates, fees or prices." so I think my feeling is correct.
It's not really that. On PG&E's website it also uses the word tariff to refer to prices for electricity and gas. https://www.pge.com/tariffs/en.html
Unclear to me why you're being downvoted. I previously knew non-US people refer to rates as "tariffs", but I never heard it in a US context. It's not rare, it's just not a meaning of the word Americans typically know.
hotels in India also use the word tariff for room rates, as in, see the tariff chart on the wall behind the reception desk.
I vaguely remember the word tariff being used to refer to fees or duties in a history class growing up, but as you've noted, it's not used this way in American English anymore. When I got the email from Hetzner this morning, I thought they were taking preemptive action against some tariff/sanction that Trump must have announced against EU data companies.
Ah, at first I wondered if this had something to do with Trump getting elected and his claims that he will implement massive (import) tariffs.
>Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources.
So... raising the prices for everybody instead?
So... raising the prices for everybody instead?
They're only raising the prices of customers whose servers use more than a terabyte per month. Based on my experience, it's not easy to go over a terabyte of bandwidth for most web services. I doubt the majority of their customers will see any change in price.
Sucks to pay a dollar per terabyte extra if you're downloading a petabyte per month through your hetzner VPN, but this sure beats raising everyone's prices because two or three companies decided to use Hetzner to build a CDN.
This is why you can't offer unlimited anything, and why we can't have nice things.
Sucks to pay a dollar per terabyte extra if you're downloading a petabyte per month through your hetzner VPN, but this sure beats raising everyone's prices because two or three companies decided to use Hetzner to build a CDN.
This is why you can't offer unlimited anything, and why we can't have nice things.
> They're only raising the prices of customers whose servers use more than a terabyte per month.
It sure reads to me like they raised the base instance price across the board. The biggest increases will be for those using over the new included bandwidth (min 1tb) but they are going up for everyone.
It sure reads to me like they raised the base instance price across the board. The biggest increases will be for those using over the new included bandwidth (min 1tb) but they are going up for everyone.
> They're only raising the prices of customers whose servers use more than a terabyte per month.
No they're not? AFAICT if I made a CPX11 using 0.1TB/mo, my price just went from 3.85 to 4.49.
No they're not? AFAICT if I made a CPX11 using 0.1TB/mo, my price just went from 3.85 to 4.49.
Yeah, I really don't understand that part of the message. It'd make sense if they were lowering prices elsewhere, but now they just... raise them? I seriously don't see how that benefits _anyone_ except Hetzner.
What likely happened here is that they were raising prices due to increased costs for energy and various other costs, and if they hadn't made this change then they would have had to increase the price more, so relative to that it keeps it cheaper for low-traffic customers - and they just communicated this poorly.
Agreed. This was an incredibly stupid statement to make because they didn't reduce prices anywhere. This was not a price re-balancing, but a price hike.
Yeah, the justification given makes absolutely no sense - you are paying more than before even if you stay under the new limit (which is 1/20th of the original!)
They also use the word "tariff" several times without elaborating, as if the person who wrote the email doesn't know the actual meaning of the word.
Seems like intentional deception to hide a standard "we just want more money" price raise.
They also use the word "tariff" several times without elaborating, as if the person who wrote the email doesn't know the actual meaning of the word.
Seems like intentional deception to hide a standard "we just want more money" price raise.
> as if the person who wrote the email doesn't know the actual meaning of the word.
In my country, "tariff" is seen in several contexts:
* A tax on imports, much in the news since the recent US election.
* A pub or bar's price list is known as the "bar tariff"
* Energy companies offer a selection of "tariffs" i.e. agreed contract rates for usage-based pricing. e.g. a 3-year-fixed-price tariff, a 100%-green-energy tariff, and so on.
* The portion of a 'life' jail sentence which must be served, before a prisoner can be considered for parole.
So I don't think it's incorrect to call a price list a "tariff", merely unusual.
In my country, "tariff" is seen in several contexts:
* A tax on imports, much in the news since the recent US election.
* A pub or bar's price list is known as the "bar tariff"
* Energy companies offer a selection of "tariffs" i.e. agreed contract rates for usage-based pricing. e.g. a 3-year-fixed-price tariff, a 100%-green-energy tariff, and so on.
* The portion of a 'life' jail sentence which must be served, before a prisoner can be considered for parole.
So I don't think it's incorrect to call a price list a "tariff", merely unusual.
Right, only the first usage is mainstream American English. The others are not.
I am curious if the others are British English? Or Indian? Other?
I am curious if the others are British English? Or Indian? Other?
> So I don't think it's incorrect to call a price list a "tariff"
I'm pretty sure it is in American English. That usage might be ok in British English, but for Americans that terminology is going to be confusing. Before today, I had never heard tariff used for anything other than import taxes. And since this applies to servers in the US, it would make sense not to use terminology that would be confusing to people in that country.
I'm pretty sure it is in American English. That usage might be ok in British English, but for Americans that terminology is going to be confusing. Before today, I had never heard tariff used for anything other than import taxes. And since this applies to servers in the US, it would make sense not to use terminology that would be confusing to people in that country.
From what I can tell, in the US energy suppliers talking to one another use the term "tariff" like for example https://www.puc.texas.gov/industry/electric/rates/tdr.aspx
Whereas when talking to consumers they seem to use terms like 'rate' and 'plan'.
Whereas when talking to consumers they seem to use terms like 'rate' and 'plan'.
as if the person who wrote the email doesn't know the actual meaning of the word.
The word "tariff" has a few different meanings. I'd say they're using it correctly, just not with the same meaning that the word is commonly being used in the news right now.
The word "tariff" has a few different meanings. I'd say they're using it correctly, just not with the same meaning that the word is commonly being used in the news right now.
In Germany "phone plan" is written as the literal translation of "mobile radio tariff", as a bundle of price and terms.
So it's not unexpected to use the uncommon in English meaning of the word to describe these changes.
So it's not unexpected to use the uncommon in English meaning of the word to describe these changes.
"We just want more money" Is the standard operating procedure and the goal of all for-profit companies. How can hackers not understand this? Of course they will always want as much money as possible, and it is up to you as a customer to decide if their product is worth what they are asking or if you will go to a competitor.
Because hackers are individual human beings, and as such are motivated by a whole variety of reasons, money being just one of them.
When running small companies they still tend to be motivated by other things, such as proving a point, achieving a technical goal or having some cultural influence etc.
It's only when the company grows in size that it becomes this soulless greedy sociopath we are all too accustomed to.
Hetzner grew a lot those last 5 years or so.
When running small companies they still tend to be motivated by other things, such as proving a point, achieving a technical goal or having some cultural influence etc.
It's only when the company grows in size that it becomes this soulless greedy sociopath we are all too accustomed to.
Hetzner grew a lot those last 5 years or so.
> It's only when the company grows in size that it becomes this soulless greedy sociopath we are all too accustomed to.
Most small and medium size businesses also fit this description. And I don't consider a price hike to be sociopathic or soulless. Greedy, sure. But businesses are always profit focused first and foremost.
Most small and medium size businesses also fit this description. And I don't consider a price hike to be sociopathic or soulless. Greedy, sure. But businesses are always profit focused first and foremost.
Tariff can simply mean "fee". Don't be so proud of your ignorance.
It's not used that way in American English at all; it almost borders on archaic. Given the purpose of this email was to primarily let their American customers know they'd be raising prices on them, it seems unfair to tell someone they're ignorant when they were sent a message containing verbiage that has entirely different meaning to them.
It is not only correct British english, it would be considered formal and academic.
I understand that American english has diverged somewhat, but I would not have expected this word to give so much anguish. I wasn’t even personally aware that americans used it for a particular terminology.
in the meanwhile, us brits will continue using a literal swear word as our most popular version control system and not complain about it. :)
I understand that American english has diverged somewhat, but I would not have expected this word to give so much anguish. I wasn’t even personally aware that americans used it for a particular terminology.
in the meanwhile, us brits will continue using a literal swear word as our most popular version control system and not complain about it. :)
What's wrong with their use of "tariff"? Looks fine to me!
I created an account with Hetzner earlier this year, and confirmed my Credit Card with them, but a few second later, they auto-suspended my account before I could log in.
I emailed support, and they bluntly told me to create a new account and this time use real information... Needless to say, I bought compute elsewhere.
I don't know how they're still in business.
I emailed support, and they bluntly told me to create a new account and this time use real information... Needless to say, I bought compute elsewhere.
I don't know how they're still in business.
I think this might be a cultural thing. HN, SV, and the market for IaaS/SaaS products is a bit of an American monoculture, where "the customer is always right" and there's a strong desire to make the customer happy. I think this is mostly a good thing and especially a good way to build early stage companies, but in my experience it's less present elsewhere.
In some places companies are happy doing their own thing, don't need every customer, don't need to be everything to every customer, and won't fight for business in the same way. Does that limit them? Maybe? But I suspect not enough to be a problem most of the time.
In some places companies are happy doing their own thing, don't need every customer, don't need to be everything to every customer, and won't fight for business in the same way. Does that limit them? Maybe? But I suspect not enough to be a problem most of the time.
It's not a cultural thing to accuse customers of committing fraud their first interaction.
Not being an absolute insane jerk off is a good expectation of people.
Not being an absolute insane jerk off is a good expectation of people.
I'm not advocating for anyone being a nasty person, but there are significant cultural differences again for what it means to be rude. In Japan the cultural expectation is that saying "no" is rude in a customer service situation, which is far beyond the expectations of most of the world. This is particularly tricky when the answer is actually "no".
If you're used to American customer service, you may find European customer service to be blunt or curt, and many people would perceive that as rude even though it is not intended that way. Again, if they aren't trying to win every customer, this isn't really a problem.
If you're used to American customer service, you may find European customer service to be blunt or curt, and many people would perceive that as rude even though it is not intended that way. Again, if they aren't trying to win every customer, this isn't really a problem.
>This is particularly tricky when the answer is actually "no".
If the answer is actually "no", then the customer service person will tell you it's "a little difficult". You're supposed to understand that that means "no, we can't do it". Of course, many foreigners don't get this, so then they'll change gears and just say "it's impossible".
>If you're used to American customer service, you may find European customer service to be blunt or curt
I don't see accusing your customer of fraud as simple bluntness or curtness.
>Again, if they aren't trying to win every customer, this isn't really a problem.
If they don't want to expand their business outside of Europe, I guess that approach is OK.
If the answer is actually "no", then the customer service person will tell you it's "a little difficult". You're supposed to understand that that means "no, we can't do it". Of course, many foreigners don't get this, so then they'll change gears and just say "it's impossible".
>If you're used to American customer service, you may find European customer service to be blunt or curt
I don't see accusing your customer of fraud as simple bluntness or curtness.
>Again, if they aren't trying to win every customer, this isn't really a problem.
If they don't want to expand their business outside of Europe, I guess that approach is OK.
"Try again, and this time provide real data" would be considered very rude even in Germany.
Without reading the actual wording sent, and knowing which culture the sender was from, this is all just speculation. I'm more interested in challenging the assumptions that we have about our own expectations in customer service being universal. From the rest of this thread it sounds like there is a commonly held opinion that Hetzner customer service is blunt, and my point is that that may be fine and maybe customers should not expect to be treated in a particular way all the time.
Since I wasn't there I can't say what happened. Can be a language thing too, I can imagine that the intended meaning would be "Try again, making sure you spell everything exactly as the data on the card" but it came out as "Try again with the real info". In germany the English language is very optional, it's not needed for any media consumtion since everything is being dubbed and/or translated. This leads to less experience using the language.
Well, we haven't seen the data.
> It's not a cultural thing to accuse customers of committing fraud their first interaction.
In my experience a lot of German businesses are like that, they'll woosh you away for any slight (perceived or real). So it is definitely a cultural thing. Of course my small sample doesn't mean anything in regards to german culture, but at the very least it proves that this corporate culture is indeed a thing.
In my experience a lot of German businesses are like that, they'll woosh you away for any slight (perceived or real). So it is definitely a cultural thing. Of course my small sample doesn't mean anything in regards to german culture, but at the very least it proves that this corporate culture is indeed a thing.
If that ticked you off, Hetzner support would tick you off even more. Within ~45 minutes of opening a ticket, you get an actual engineer to look into your case, and they are constantly blunt and semi-irate both because of the German work culture and the nature of network/hardware engineering I guess. They fix your stuff fast, and they keep being short, blunt and concise while doing it. No "I'm sorry to hear..." or "Thank you for contacting us with..." blahblah like in the US. All business, no fluff.
Aside from knowing their sh*t and being available at short notice, you'll get along very well after you learn how to communicate with them. By being concise, precise, and blunt of course...
Aside from knowing their sh*t and being available at short notice, you'll get along very well after you learn how to communicate with them. By being concise, precise, and blunt of course...
Hahaha, I promise you it is. Not saying that's a good thing, but.
If this is exactly what they said it was probably either language barrier, like mis translation of "try again, making sure it's all spelt correctly" OR just that you really did use fake information, in which case here, folks will just say that. You can get told off for not saying please when ordering coffee, I like it but can understand how anyone conditioned to american customer service would be horrified.
If this is exactly what they said it was probably either language barrier, like mis translation of "try again, making sure it's all spelt correctly" OR just that you really did use fake information, in which case here, folks will just say that. You can get told off for not saying please when ordering coffee, I like it but can understand how anyone conditioned to american customer service would be horrified.
> Not being an absolute insane jerk off is a good expectation of people.
have you met the internet yet
have you met the internet yet
Really? In France you're treated most of the time as a potential fraudster first, then maybe when planet aligns as customer in some services like train, tramway, post, banks (the state mandates to justify things like moving and using your own money you already paid taxes on) and various administrations.
> where "the customer is always right" and there's a strong desire to make the customer happy
It’s definitely a culture shock for people from these cultures to encounter businesses who reject their business, whether it’s deemed risky or not worthwhile.
It was eye opening to me to spend time working with cultures where I had to convince businesses to deal with me rather than the other way around. Some people get extremely offended when businesses don’t bend over backward for them, but that’s not going to work when you encounter cultures who operate differently.
It’s definitely a culture shock for people from these cultures to encounter businesses who reject their business, whether it’s deemed risky or not worthwhile.
It was eye opening to me to spend time working with cultures where I had to convince businesses to deal with me rather than the other way around. Some people get extremely offended when businesses don’t bend over backward for them, but that’s not going to work when you encounter cultures who operate differently.
> I think this might be a cultural thing.
It's just German bureaucracy. When I wanted to register domains with Hetzner few years back, they asked me to print multiple pages of forms and contract, fill out, sign and... fax it back.
It's just German bureaucracy. When I wanted to register domains with Hetzner few years back, they asked me to print multiple pages of forms and contract, fill out, sign and... fax it back.
I'm fine being fed up with the bureaucracy in my own country without also having to deal with Germany's. A big no-thanks to ever dealing with Hetzner.
FAANG has exactly the same behavior. When you have billions of users and no customer support, no one cares about the customer being right.
Last month I tried to create an Instagram account, I kept being instabanned on 5 different devices with different IPs. No recourse possible. HN is full of horror stories like this.
Last month I tried to create an Instagram account, I kept being instabanned on 5 different devices with different IPs. No recourse possible. HN is full of horror stories like this.
I was customer for 10 years, but for business I needed a second account. Banned immediately even after I submitted a passport copy. Worked after contacting their support. Still happy customer (with both accounts). I think they're just very strict and get their (un)fair share of stolen credit card or stolen identity signups.
Hetzner values its old customers a LOT for some reason...
It's always a balance between how much fraud you allow and how many real customers you reject. They set their threshold at an interesting level, but maybe they're happy with that choice.
You're right. But unfortunately this mindset moves us closer and closer to having algorithms exclude people from society and life. And as the stories already told here show, it can happen to any of us for no apparent reason.
It's like, imagine a magic wand that, if waved, would make life a little better for 99% of society, but much worse for 1%. Would it be moral to wave that wand?
It's like, imagine a magic wand that, if waved, would make life a little better for 99% of society, but much worse for 1%. Would it be moral to wave that wand?
Well you got a human answering the support hotline.
I think that alone kinda nullifies the threat of an algorithm. The entire reason why they're such a massive problem is because Google et al. refuse to operate proper support hotlines to help people and even if they do have a support line (Facebook infamously doesn't have one and wants you to go through the courts to contact them), the support staff aren't actually equipped to help people beyond regurgitating canned support page links. You can't solve a malfunctioning algorithm with another algorithm or by forcing a human to behave like an algorithm.
It's not a big secret that the best way to get yourself in front of actual support if GAFAM screws you over is to complain about it on HN because this is where SREs lurk that can actually punt your requests through to people that can look into it.
Hetzner at least gave a direct answer to explain the reason.
I think that alone kinda nullifies the threat of an algorithm. The entire reason why they're such a massive problem is because Google et al. refuse to operate proper support hotlines to help people and even if they do have a support line (Facebook infamously doesn't have one and wants you to go through the courts to contact them), the support staff aren't actually equipped to help people beyond regurgitating canned support page links. You can't solve a malfunctioning algorithm with another algorithm or by forcing a human to behave like an algorithm.
It's not a big secret that the best way to get yourself in front of actual support if GAFAM screws you over is to complain about it on HN because this is where SREs lurk that can actually punt your requests through to people that can look into it.
Hetzner at least gave a direct answer to explain the reason.
How does that human nullify anything if they don't unlock the account? Presuming GP entered real information, which we can only assume at this point
The problem with this mindset (incredibly common in the US; much more so than in the European countries I've lived in, at least) isn't thinking probabilistically, and trying to determine the likelihood of a prospective customer being a fraudster. It's really just that there are only two possible outcomes: Yes and no.
Just by having a third option, most of the downsides of doing the evaluation incorrectly could be mitigated. Of course, that's generally much more expensive (and often uneconomical) than saying no, so it's usually not done.
I've been on the 1% side of things quite a few times due to having new credit and (presumably) various data brokers not knowing every detail about me yet, and the experience really, really sucks.
Just by having a third option, most of the downsides of doing the evaluation incorrectly could be mitigated. Of course, that's generally much more expensive (and often uneconomical) than saying no, so it's usually not done.
I've been on the 1% side of things quite a few times due to having new credit and (presumably) various data brokers not knowing every detail about me yet, and the experience really, really sucks.
You're making choices like that every single day. Sometimes with reversed percentages. Even posting this message may be on average a tiny loss for the world. (It's meaningless in the end and burned some energy to give one person a mini dopamine boost)
There’s nothing particularly scary about “algorithms” making these choices since it’s just people at these companies choosing and implementing the algorithms. It wouldn’t get better if those humans weren’t allowed to use algorithms to make these choices since decisions.
Arguably, we wave that magic wand every time we decide not to donate to starving children in third-world countries. Which is pretty often!
The Ones Who Walk Away From Omelas
What type of fraud exactly? You mean like stolen CCs? It feels very medieval as a financial trust system if every little vendor can’t trust payments, even when you pay up front? Like this is in some ways worse than cold hard cash. And then we pay VISA premium on top of that, for the convenience of being mistrusted..
If they pay up front then dispute, the company will suffer extra charges. With enough of reports, their payment processor fee% goes up and impacts all their payments.
Paying up front doesn't really mean much, because if the credit card info is stolen, the actual owner will report the transaction and it will be reversed.
Right but that sucks. So CC companies/banks are simply shifting risk from one side of the transaction to the other. Sure, as a consumer you don’t have to worry because you’ll get your money back. But if the merchant has to worry and reject customers who are “suspicious” to protect themselves then you’re back to square one, except more kafkaesque. That’s why I said cash is better.
I’d rather have $10 permanently lost for a month of VPS than being banned after 5 days setting it up because I’m traveling and my IP is “suspicious”. Which has happened to me.
I’d rather have $10 permanently lost for a month of VPS than being banned after 5 days setting it up because I’m traveling and my IP is “suspicious”. Which has happened to me.
Hetzner is actually worse. Current EU directives allow liability shift in online payment when the customer is authenticated via 3D Secure, meaning the risk is extremely low and the onus of proving the fraud is on the card owner. Yet Hetzner will not accept your money if your name doesn’t look correct for the country you are accessing from.
I wonder if they tried a debit card...
Did you sign up as a private individual or was it a company account? I think they are extremely strict with private accounts, especially outside of EU. Most individuals only get a few resources for a few bucks per month, which probably isn't worth the business at all.
I know a few people who use Hetzner, smaller EU registered companies, zero issues. Also extending limits is just one click and gets approved after a few minutes.
And I totally get why they have to be strict. Blocked IPs are becoming an epidemic, I stopped using Digital Ocean, because I always got bad IPs, that were blocked everywhere. Also Hetzner is suffering from that. People are renting servers for little money and do bad things with the 20 TB traffic they get.
I know a few people who use Hetzner, smaller EU registered companies, zero issues. Also extending limits is just one click and gets approved after a few minutes.
And I totally get why they have to be strict. Blocked IPs are becoming an epidemic, I stopped using Digital Ocean, because I always got bad IPs, that were blocked everywhere. Also Hetzner is suffering from that. People are renting servers for little money and do bad things with the 20 TB traffic they get.
Same with me a few years ago. Their support told me that they didn't want me as a customer. It was my first interaction with them. I swear I'm just a regular nerd with a credit card :D
They are bizarre.
They are bizarre.
A credit card? That... makes some sense, then. I'd be wary if anyone trying to pay with one of those as well; they're much too rare, and I've heard it's easy to unilaterally reverse charges.
Hetzner is a German company. Credit cards may be common in the USA, but they're not anywhere else.
Hetzner is a German company. Credit cards may be common in the USA, but they're not anywhere else.
They are literally the preferred way to pay in the U.S. for damn near everything.
I’ve also had no problems using them in Britain, Canada, Ireland, Korea…
I’ve also had no problems using them in Britain, Canada, Ireland, Korea…
> but they're not anywhere else.
They certainly are very common in much of Europe. Buying stuff on credit might be not that common but plenty of people have credit cards themselves (even of they are effectively used the same way as debit cards)
To be fair not sure if you comment is just sarcasm.. but I’ve never had any issues using a credit card on Hetzner or pretty much any other European company
They certainly are very common in much of Europe. Buying stuff on credit might be not that common but plenty of people have credit cards themselves (even of they are effectively used the same way as debit cards)
To be fair not sure if you comment is just sarcasm.. but I’ve never had any issues using a credit card on Hetzner or pretty much any other European company
If they don't want any customers outside of Germany, they should just say so.
I haven't done business in Germany or used German banks -- is it hard to get them to reverse charges?
Well yes, you go to court if you can't agree with the merchant on anything. They're the competent authority to settle disputes
When I pay for something, now the other party has the money, that's how money works. If I trust them that little, I should probably be using escrow but this costs extra and so it imo doesn't make sense to pay that fee for every transaction - as one does with this chargeback guarantee thing. Probabilistic societies where you're excluded based on bad odds or credit scores is what you get with that system (I've been on the losing end of that, not because I've ever had any debt (too little money) but precisely because they had no positive data because I never needed a loan, and so you can't pay for stuff in certain countries because they require a credit card)
When I pay for something, now the other party has the money, that's how money works. If I trust them that little, I should probably be using escrow but this costs extra and so it imo doesn't make sense to pay that fee for every transaction - as one does with this chargeback guarantee thing. Probabilistic societies where you're excluded based on bad odds or credit scores is what you get with that system (I've been on the losing end of that, not because I've ever had any debt (too little money) but precisely because they had no positive data because I never needed a loan, and so you can't pay for stuff in certain countries because they require a credit card)
I guess I assumed that consumer protection laws would mean people could reverse charges more easily than they can here in the US (which we can do easily, albeit as a cardholder benefit, not a matter of law). Interesting.
They're mainly designed to make it so you don't need to. If you buy something, it's going to work; and if it doesn't, the seller is required to make up for that.
I can only speak for Norway, but this doesn't normally mean reversing the charge. It's assumed that, if you bought an X, you wanted an X. So the seller is required to get you an X of good enough quality to actually do the job, and if that isn't what they were trying to sell... well, then they won't be in business for long.
Normally this means repairs or replacement, of course on the seller's bill.
The US seems much more geared to seeing every interaction as... I want to say 'combative'.
I can only speak for Norway, but this doesn't normally mean reversing the charge. It's assumed that, if you bought an X, you wanted an X. So the seller is required to get you an X of good enough quality to actually do the job, and if that isn't what they were trying to sell... well, then they won't be in business for long.
Normally this means repairs or replacement, of course on the seller's bill.
The US seems much more geared to seeing every interaction as... I want to say 'combative'.
And to add to this, if the seller is unable to make the thing work as it's supposed to, either through repairs or replacements, they are of course obliged to give the customer their money back. But yeah, that's not the first resort.
I can't edit this anymore but "obliged" was supposed to be "obligated" of course
You can still do chargebacks if you pay with a credit card from a German or other European bank (experience of course might vary but you can just switch to one that offers better customer service)
I signed up and immediately got banned because I was accessing through a VPN, which I think is a common problem others have had. I emailed them and their advice was to stop using a VPN and try again.
That seems fair to me though. If that ISP (whatever the WHOIS resolves to for your VPN) gets a high rate of fraudulent customers, how is Hetzner to know if you're the same fraudster again or a new customer? Especially if you share not just the ISP but even your IP address with someone... we also rate limit logins for example based on IP address. If you're coming from the same origin as an attacker, there's no way for me to tell who's who
Perhaps they could implement some way to do a 1-cent bank transfer from an account with a name which isn't on their fraud list, but as a simple step it seems pretty normal to use your regular internet connection (that ties the IP+timestamp combination to a real-world subscriber) and not some anonymity service
Although it would be nice if we could be anonymous on the web, when abuse traffic is involved (more than posting a comment on some forum as an anonymous source or something), I don't know how that'd be possible to reconcile
Perhaps they could implement some way to do a 1-cent bank transfer from an account with a name which isn't on their fraud list, but as a simple step it seems pretty normal to use your regular internet connection (that ties the IP+timestamp combination to a real-world subscriber) and not some anonymity service
Although it would be nice if we could be anonymous on the web, when abuse traffic is involved (more than posting a comment on some forum as an anonymous source or something), I don't know how that'd be possible to reconcile
I get it, but you'll get a lost of false positives as well. My experience is that they also had slow customer service.
I don't want any business critical infra managed by a company willing to be that aggressive in blocking accounts, while also having slow customer service to resolve access issues.
I don't want any business critical infra managed by a company willing to be that aggressive in blocking accounts, while also having slow customer service to resolve access issues.
I had a similar problem, followed by getting blocked for using an IP owned by a mobile telecom, followed by being unable to credit the account via a bank transfer. At each step trying to explain my situation to support.
Then it worked, and I've had no problems since. I'm concerned about them failing to charge my virtual debit card some month in the future and losing data, but it hasn't happened yet.
If there was anything else nearly as cheap, or if it wasn't for personal use, I probably would have given up and used another service.
Then it worked, and I've had no problems since. I'm concerned about them failing to charge my virtual debit card some month in the future and losing data, but it hasn't happened yet.
If there was anything else nearly as cheap, or if it wasn't for personal use, I probably would have given up and used another service.
Why were you using a VPN?
Because they're so cheap they have to worry about spammers and other sorts of abuse. It's more cost effective for them to refuse dodgy-looking accounts.
The big question is, were you using real information or did you put in fake info?
They're still in business because they're cheap and pretty reliable. But being cheap means there are things you can't get with them but can with others. For example, you can't pre-pay for the entire year ahead of time.
The big question is, were you using real information or did you put in fake info?
They're still in business because they're cheap and pretty reliable. But being cheap means there are things you can't get with them but can with others. For example, you can't pre-pay for the entire year ahead of time.
You can actually pre-pay them. Under the transactions section in your account, it says:
“If you make advance payments by bank transfer to our bank account, the amount will be posted as a credit on your account. We will automatically deduct this credit on your account when we process future invoices.”
“If you make advance payments by bank transfer to our bank account, the amount will be posted as a credit on your account. We will automatically deduct this credit on your account when we process future invoices.”
In order to pre-pay, you need to have an account first.
Many people didn't even get to the point where they can log in.
Many people didn't even get to the point where they can log in.
They must have changed that. I remember reading on their site at some point that they didn't accept that.
When I ordered a VPS at Netcup (a Hetzner alternative) they called me and asked me for the name of the hotel next to my place to check if I really lived at the address I provided. I guess that if i would have needed to look it up, that is, struggled a bit with the answer, they would have denied me as a customer.
Just today, a remote colleague mentioned staying at whatever hotel near the office. I had never heard of it: as a local, I don't tend to use the hotels here!
Seems like a really weird choice of question. Thinking of germans, a bakery seems more likely something they'd know of; perhaps a supermarket works more internationally (I'm sure there's exceptions to that as well). Maybe the distance in driving, cycling, or public transport minutes to surrounding cities could be a universal question to ask
Seems like a really weird choice of question. Thinking of germans, a bakery seems more likely something they'd know of; perhaps a supermarket works more internationally (I'm sure there's exceptions to that as well). Maybe the distance in driving, cycling, or public transport minutes to surrounding cities could be a universal question to ask
If they can look it up then so can you! Maybe it filters out lazy scammers, but it doesn't sound like solid KYC to me.
In fact I don't feel like a hosting service should need to do this at all. If you pay your bills and aren't on the Stasi blacklist you should be good to go. I don't want or expect the likes of Hetzner to be responsible for policing.
In fact I don't feel like a hosting service should need to do this at all. If you pay your bills and aren't on the Stasi blacklist you should be good to go. I don't want or expect the likes of Hetzner to be responsible for policing.
As someone who runs mail servers, I wish they'd be MORE strict to stop the drive-by spammers. Every so often our servers get blocked because someone starts sending spam from a machine on the same netblock.
It might be good point-of-difference for some hosting service: have brutal KYC so your netblock is well regarded.
It might be good point-of-difference for some hosting service: have brutal KYC so your netblock is well regarded.
I agree. I'm currently using Linode. If they started requiring every account to have a valid US passport, I'd happily comply.
I wish these companies were forced to get the nationality of the person or the companies owners and be forced to assign them to IP address blocks which identifies these nationalities.
My home DNS server blocks all requests to .ru and .cn, but I can't do IP-address-block blocking because shady Chinese companies or individuals just need to rent some computing on AWS or DigitalOcean in order to become indistinguishable from American companies. And specially DigitalOcean seems to be the favorite platform for doing scams.
We're OK with having license plates attached to our cars, but not to be forced to expose our nationality to infrastructure providers? There's really no privacy-sensitive stuff which needs to be protected in that case.
My home DNS server blocks all requests to .ru and .cn, but I can't do IP-address-block blocking because shady Chinese companies or individuals just need to rent some computing on AWS or DigitalOcean in order to become indistinguishable from American companies. And specially DigitalOcean seems to be the favorite platform for doing scams.
We're OK with having license plates attached to our cars, but not to be forced to expose our nationality to infrastructure providers? There's really no privacy-sensitive stuff which needs to be protected in that case.
I don't think you can KYC your way to stopping spammers without stepping on some protected classes.
I think that having a blacklist with some ID (bank account, national ID, CA cert) that's hard to replace is a good compromise.
You could try giving people moral purity tests before you let them in (doesn't some of the tildeverse work like that?) but I think it's a dark road.
I think that having a blacklist with some ID (bank account, national ID, CA cert) that's hard to replace is a good compromise.
You could try giving people moral purity tests before you let them in (doesn't some of the tildeverse work like that?) but I think it's a dark road.
Sure, it's not solid, but they knew I was a private customer, so what should they do? Do a request to a private credit bureau like the SCHUFA (like Equifax) to check if my provided details are valid?
It feels like it's a check to get a general feeling if things sound plausible. Like no Russian accent for a German name, some knowledge of the surroundings, a valid phone number.
It feels like it's a check to get a general feeling if things sound plausible. Like no Russian accent for a German name, some knowledge of the surroundings, a valid phone number.
Boy we're playing IRL Papers, Please now!
I had the same thing, but this was a business address while I was working remotely and had no idea about the area. Told them as much on the phone while looking the answer up on Google Maps. They just accepted that and opened the account.
I thought Netcup either colo'ed at or rented servers at, Hetzner? Maybe this was a ways back or have they always had their own DC?
Had exactly the same experience. I even provided my real drivers license -- something I normally would never do. Didn't convince them I am a real human being that matches what's on the ID.
I never understood why people think so highly of Hetzner.
I never understood why people think so highly of Hetzner.
> real drivers licence
Drivers licence as a personal ID is not really a thing in Europe. It's accepted at some places, but either passport or your state provided ID-card is much preferred.
Drivers licence as a personal ID is not really a thing in Europe. It's accepted at some places, but either passport or your state provided ID-card is much preferred.
Very few Americans have personal ID so if they intend to do business in America, they should be prepared to accept drivers licenses.
Maybe they just don't want to sell to Americans, which is always allowed I guess, but setting up an American VPS provider and not accepting the standard American KYC verification methods would be a rather incompetent way of doing business. It'd be no different from doing business in Europe and only accepting drivers licenses and not accepting passports/ID cards for identity verification.
Maybe they just don't want to sell to Americans, which is always allowed I guess, but setting up an American VPS provider and not accepting the standard American KYC verification methods would be a rather incompetent way of doing business. It'd be no different from doing business in Europe and only accepting drivers licenses and not accepting passports/ID cards for identity verification.
In the US, DL works as an ID for almost anything other than voting in an election. You can fly to almost any US territory with that plastic card. And it is state-provided ID -- nobody questions that.
And we are talking about 300M people here, which is about 4 times the population for Germany.
If they hate US residents they should declare that clearly on their website, not wasting customers' (and their own) time.
And we are talking about 300M people here, which is about 4 times the population for Germany.
If they hate US residents they should declare that clearly on their website, not wasting customers' (and their own) time.
But you don't just have one type of drivers license for 300M people, do you? As far as I'm aware each state has their own. So it's not 300M vs. 80M, but 80M vs. 50x 6M people (incredibly simplified and wrong numbers, but you get the gist). Far more effort to verify each document.
That would still exclude non-driving customers, which means allowing at least one additional form of ID per state - suddenly you have to support at least 100 IDs for US customers, while German customers have exactly 1.
Not sure how you arrive at them "hating US residents", that's a weird conclusion to get to.
That would still exclude non-driving customers, which means allowing at least one additional form of ID per state - suddenly you have to support at least 100 IDs for US customers, while German customers have exactly 1.
Not sure how you arrive at them "hating US residents", that's a weird conclusion to get to.
Contrary to the belief of a lot of Americans, just because it works in the US doesn't mean it works elsewhere. In the rest of the world a driver's license is used for proving your right to drive and we have a thing called passports for proving your identity. Hetzner isn't an American company so why should they accept American drivers licenses?
> Hetzner isn't an American company so why should they accept American drivers licenses?
When in Rome ...
When in Rome ...
An EU drivers license should be fine, as that is an official form of ID. But for anyone else, they'd need an international form of identification.
Were you using your real information when you signed up? Were you using a disposable / privacy-first credit card? Did you try to obscure your name or register from a VPN? You're very upset at them accusing you of abuse or fraud, but you haven't actually made it clear whether any of the signals you provided them when signing up were aligned with what they'd usually see from abusive or fraudulent users.
This was between 2000-2010, but when I ran a linux consulting+hosting business our "provision a VPS with a credit card" was where we experienced the largest amount of fraud, by probably 100x. Certainly well over 10x. Might be why Hetzner was touchy?
Because they provide good service to most customers.
Your case is unfortunate, and I totally understand why you'd be taking you business elsewhere, but probably an outlier.
Otherwise, we'd need to ask the same about AWS et. al. as we've definitely seen more than enough wrong account closure complaints on here.
Your case is unfortunate, and I totally understand why you'd be taking you business elsewhere, but probably an outlier.
Otherwise, we'd need to ask the same about AWS et. al. as we've definitely seen more than enough wrong account closure complaints on here.
I'm not sure what was your concern - Hetzner is obliged by law to make sure their signing up a real person or a company and that their servers are not used for malicious intent.
It's pretty obvious what the concern is, no? Hetzner's process seems to be blocking real people with no malicious intent (myself included) from creating accounts, even when we try to provide proof. Obviously this means we end up using a different provider instead. Though in my experience attempting to do business in Germany, this kind of forbidden-by-default perspective is quite common there, so they probably don't see it as a concern, I guess.
A credit card is “low trust” confirmation of identity. Usually for this kind of services, you also have to submit your ID for verification. Perhaps you’re based in a country where this is not possible so they didn’t ask upfront?
Either way, it’s their process so you’re of course welcome to go elsewhere if you want. I don’t see it as a “bad thing” for companies to require some level of trust.
Either way, it’s their process so you’re of course welcome to go elsewhere if you want. I don’t see it as a “bad thing” for companies to require some level of trust.
I'm based in the UK where we all have credit cards. I also did give them my ID - which is NOT typical for other hosting providers, actually, but I'm happy to provide it - but that still didn't satisfy them.
Yes, Germany is an extremely low-trust society masquerading as a high-trust society.
Herzner get flack because they are probably the only German entity that the average HN user has dealings with. Anyone who has had exposure to German bureaucracy will recognise the hostility.
Herzner get flack because they are probably the only German entity that the average HN user has dealings with. Anyone who has had exposure to German bureaucracy will recognise the hostility.
The strange thing is that the same thing is happening in the US. A story of someone’s Google, PayPal, Instagram etc account getting banned for no reason gets posted every day. Customer service means posting on social media and hoping it gets enough traction. Hell, I can’t even buy anything at Walmart online because they cancel every order because of suspicious activity, even though I ordered there before with the same payment method and address. I think a lot of people have a strong sense of entitlement and blame things such as this on the foreignness of the entity (in this case Hetzner).
You also don't know the strain brought by continual streams of fraud that their support has to deal with, all for being a relatively cheap option.
Always felt like they were in the business of blaming and hating their customers. Cloud providers that nitpick and judge every aspect of their customers’ business details and technicalities are a huge operational risk. This archaic practice is the reason generic cloud orchestration was a must, and it’s just not needed anymore.
I don’t care how cheap they are. You get what you pay for.
I don’t care how cheap they are. You get what you pay for.
There’s nothing about cheap that implies terrible customer service. Or rather, the reverse isn’t necessarily true.
> I don't know how they're still in business.
They're growing year after year, so clearly they're doing something right.
They're growing year after year, so clearly they're doing something right.
OTOH there is Azure where a very sophisticated Twillio phishing service was hosted. When reported to abuse Microsoft replied they were a valid customer. A week later it finally came down.
Yep. Same thing happened to me. I was not able to get my account working. Ever.
I have all my stuff on vultr now.
I have all my stuff on vultr now.
I've been reading about how great Hetzner for years but i couldn't get past the sigup page where they require a credit card just to create an account.
I mean your transaction probably got flagged as fraudulent (like if your postal code didn't match your card), it's not that mysterious.
I think most online operators that have "spend $3 with us" tiers have to be super vigilant about card transactions, and when you fall into the cracks you're a bit SOL.
I think most online operators that have "spend $3 with us" tiers have to be super vigilant about card transactions, and when you fall into the cracks you're a bit SOL.
It's been ages, but once I got something cancelled due to a site not properly distinguishing between shipping address and card billing address.
They are still in business because 99.99% of customers are not affected by this.
I'm happy customer for years and will be even with these changes.
I'm happy customer for years and will be even with these changes.
> I don't know how they're still in business.
They have 400 GBit/s of DTAG transit.
Showing that they are rich as fuck without saying it.
They have 400 GBit/s of DTAG transit.
Showing that they are rich as fuck without saying it.
Were you on a VPN or Tor at the time?
OVH did the same thing to me...
don't use a gmail account
same. fuck 'em. they are not even that cheap to begin with. they used to be, not today.
pbhjpbhj(1)
That's a particularly, but unfortunately good example of how awful they are as a company. The sooner they get pushed out of business the better.
What are some examples of applications people are running that:
1. Require 20TB of bandwidth / month
2. That bandwidth can't be shielded by Cloudflare and others?
Is it like... real time video streaming? Gaming servers? I can't imagine a web app getting anywhere close to that.
I run a mid sized NFT art creation website that generates both images and GIF's (https://mintables.club) and with over 100000 users at peak, it only needed about 1.5TB of bandwidth.
1. Require 20TB of bandwidth / month
2. That bandwidth can't be shielded by Cloudflare and others?
Is it like... real time video streaming? Gaming servers? I can't imagine a web app getting anywhere close to that.
I run a mid sized NFT art creation website that generates both images and GIF's (https://mintables.club) and with over 100000 users at peak, it only needed about 1.5TB of bandwidth.
I have 2 services that run above those figures:
1. the demo instance of my OSS software which is a bring your own storage Dropbox like UI for SFTP, S3, FTP, and every protocols imaginable: https://github.com/mickael-kerjean/filestash People tend to come in the demo to upload / download tons of stuffs
2. the docker registry for my oss stuff since I was kicked out of the docker open source program and now need to find a new place to store all the images. 10 millions downloads over the last few years, it does add up very quickly way above the 20TB limits if your image isn't super slim and try to selfhost everything
1. the demo instance of my OSS software which is a bring your own storage Dropbox like UI for SFTP, S3, FTP, and every protocols imaginable: https://github.com/mickael-kerjean/filestash People tend to come in the demo to upload / download tons of stuffs
2. the docker registry for my oss stuff since I was kicked out of the docker open source program and now need to find a new place to store all the images. 10 millions downloads over the last few years, it does add up very quickly way above the 20TB limits if your image isn't super slim and try to selfhost everything
> the demo instance of my OSS software which a bring your own storage Dropbox like UI for SFTP, S3, FTP, and every protocols imaginable: https://github.com/mickael-kerjean/filestash People tend to come in the demo to upload / download tons of stuffs
Do you have problems with illegal content shared over your service? I wouldn’t ever offer something like this because I wouldn’t want to deal with someone uploading child porn or stuff like that.
Do you have problems with illegal content shared over your service? I wouldn’t ever offer something like this because I wouldn’t want to deal with someone uploading child porn or stuff like that.
Not really scared as I take action the second I hear about nasty things. The most recent events were people creating shared links pointing to FTP servers that had revenge porn and sharing those on a whacky telegram account, another annoying one is people who keep trying to brute force other people accounts on cloud providers even though there's a protection in place to throttle to 20 connection attempt per second, effectively degrading the free service for everyone else and getting me kicked off from cloud provider like the most recent one being Hostinger
Is there a way to scan files/images as they are uploaded for illegal content?
github's registry (ghcr.io) is the other big free one
BitTorrent seedbox. If you are apart of a private torrent tracker you may download 5tb and upload 10tb of data per month to be in good standing within that tracker’s community
I've always wondered how ratios like that are supposed to work. Surely the ratio has to be 1.0, averaged over all users.
most trackers have some kind of bonus point system
I think certain torrents being marked freeleech to not count against accounts' ratio is what causes the bulk of the imbalance in total uploads versus downloads. It's necessary too, to increase availability on torrents.
Good thing I pay only 30 EUR a month for symmetric 1 gbit fiber. Right now with BF you can find BF deals cheaper than 2 USD a month, and then there's Usenet.
Yeah, this is it.
> video streaming?
Pet peeve but yes. It should not, in 2024, be considered a niche use case to deal with.. video. We’ve become accustomed to YouTube and their impossible-to-beat free hosting. But it’s really pricey to do streaming and so yes, we should expect large bandwidth being available. “Who needs X” is a question that should be reversed, instead we should ask “why not?”. When we have good affordable infra, we get cool new stuff and everyone benefits.
Pet peeve but yes. It should not, in 2024, be considered a niche use case to deal with.. video. We’ve become accustomed to YouTube and their impossible-to-beat free hosting. But it’s really pricey to do streaming and so yes, we should expect large bandwidth being available. “Who needs X” is a question that should be reversed, instead we should ask “why not?”. When we have good affordable infra, we get cool new stuff and everyone benefits.
Structural monitoring in engineering for example. You need high quality pictures and have lots of drone data and files need to move to hyper scaler or on premise for GPU processing.
Cloudflare is not trustworthy and you don't always need to pay the "protection" fee. I.e. B2B where a day offline doesn't matter.
Cloud pricing is insane compared to Hetzner with regards to storage and CPU power. If you have some Linux knowledge it's perfectly stable. Support is top notch.
For 50€ per box you get 10gbe which is also nice. Not everyone needs 100% IaC like the big hyper scaler offer or geo redundancy etc.pp... even Hetzner has an API and you can even terraform their dedicated servers to some degree. It's 15 to 30 minutes vs. seconds but who cares.
Cloudflare is not trustworthy and you don't always need to pay the "protection" fee. I.e. B2B where a day offline doesn't matter.
Cloud pricing is insane compared to Hetzner with regards to storage and CPU power. If you have some Linux knowledge it's perfectly stable. Support is top notch.
For 50€ per box you get 10gbe which is also nice. Not everyone needs 100% IaC like the big hyper scaler offer or geo redundancy etc.pp... even Hetzner has an API and you can even terraform their dedicated servers to some degree. It's 15 to 30 minutes vs. seconds but who cares.
1.5TB isn't much at all but keep in mind that if a "disproportionate percentage" of your Cloudflare bandwidth is used for images they reserve the right to boot you off [1].
https://www.cloudflare.com/service-specific-terms-applicatio...
https://www.cloudflare.com/service-specific-terms-applicatio...
There are many data streams you may want to process that take lots of network traffic. CT logs, monitoring aggregators, web crawlers, etc. For the traffic you initiate, there's no proxying/caching you can do.
Besides needing it: Not having to fear that an attacker spamming the machine with Mail or web requests or something incurs a bug traffic bill gives better sleep at night.
PeerTube comes to mind.
More people setup VPN servers these days. Spiked with the Iran protests. Wireguard makes it eas-ier. Circumventing some rules like content distribution.
blockchain nodes
Ahh, yes, the good old "here, you purchased X amount of things for $Z. But don't dare to use everything you paid for, or we double the price"
I think you are probably misreading the situation. My guess is that their costs went up and they are now increasing their prices so they don't lose money.
Dunno... my reaction was for this sentence specifically:
> Until this change, customers who have used fewer resources have covered the costs [...] for other customers who have used much more resources.
or else they charge you $1 a tb, not cut you off
You're misreading. This wouldn't get the $1 fee. "We're giving you 20 TB (TWENTY) but don't you dare use more than 2 TB (TWO)"
It's a classic web host reseller technique of overselling and having it backfire.
It's a classic web host reseller technique of overselling and having it backfire.
That's only a problem if they cut you off. If there's a defined (low) fee for overages then I don't understand what the problem is. $1tb is very well priced. AWS/GCP charges from the first byte after the miniscule free tier.
Hetzner have definitely always been scumbags about the bait & switch on aspects of their service like that. Granted it's pretty typical of the too good to be true rule of life.
I have only had and have heard of great experiences with Hetzner. For both their offerings snd their support. I am based in Europe though.
I can't think of a single other instance of bait and switch with Hetzner and we run a fair bit of infra with them.
Been using them happily for a few years. They’ve been rock solid and cheap. Can’t complain, even about this hike.
Do you have any details? I was about to move all my services off from vultr to hetzner due to the much better pricing
Is traffic generally much more expensive in the US than in Germany? This seems to be a US-only change and I'm wondering a bit about the reasoning here and whether to expect this to also change in other regions.
Not really, CDNs with regional pricing usually charge almost if not exactly the same amount for NA and EU traffic. It's the other regions that tend to be more expensive.
I wonder if this has anything to do with how Hetzner operates in each region, they run their own EU datacenters but AFAIK they just rent rack space in the US, so they're more at the mercy of upstream providers.
I wonder if this has anything to do with how Hetzner operates in each region, they run their own EU datacenters but AFAIK they just rent rack space in the US, so they're more at the mercy of upstream providers.
From Cloudflare blog about why EU prices are lower (around 50% cheaper than US at least according to cloudflare)
The value of an exchange depends on the number of networks that are a part of it. The Amsterdam Internet Exchange (AMS-IX), Frankfurt Internet Exchange (DE-CIX), and the London Internet Exchange (LINX) are three of the largest exchanges in the world.
In Europe, and most other regions outside North America, these and other exchanges are generally run as non-profit collectives set up to benefit their member networks. In North America, while there are Internet exchanges, they are typically run by for-profit companies. The largest of these for-profit exchanges in North America are run by Equinix, a data center company, which uses exchanges in its facilities to increase the value of locating equipment there. Since they are run with a profit motive, pricing to join North American exchanges is typically higher than exchanges in the rest of the world.
The value of an exchange depends on the number of networks that are a part of it. The Amsterdam Internet Exchange (AMS-IX), Frankfurt Internet Exchange (DE-CIX), and the London Internet Exchange (LINX) are three of the largest exchanges in the world.
In Europe, and most other regions outside North America, these and other exchanges are generally run as non-profit collectives set up to benefit their member networks. In North America, while there are Internet exchanges, they are typically run by for-profit companies. The largest of these for-profit exchanges in North America are run by Equinix, a data center company, which uses exchanges in its facilities to increase the value of locating equipment there. Since they are run with a profit motive, pricing to join North American exchanges is typically higher than exchanges in the rest of the world.
There _are_ non-profit neutral exchanges in the US (eg https://en.wikipedia.org/wiki/Seattle_Internet_Exchange), but they're just not as dominant as they are in Europe.
> for other customers who have used much more resources
So, "Pi mal Daumen"* this means that US customers have a bandwidth consumption which almost an order of magnitude higher than that of European and Singaporean customers?
I wonder what it consists of.
* π x thumb = ballpark figure
So, "Pi mal Daumen"* this means that US customers have a bandwidth consumption which almost an order of magnitude higher than that of European and Singaporean customers?
I wonder what it consists of.
* π x thumb = ballpark figure
Hetzner has been an established player in Europe for a long time. It seems plausible that they have enough customers who use small amounts of bandwidth to subsidise the heavier users.
Considering switching costs, if they enter the US market with better pricing than established players, it stands to reason that the customers that would be most enticed to move will be the heavier users.
Considering switching costs, if they enter the US market with better pricing than established players, it stands to reason that the customers that would be most enticed to move will be the heavier users.
EU transit costs and peering agreements are much more relaxed and cheaper than in US
Europe is also a lot smaller network wise.
Hetzner only have to get their traffic to Frankfurt to get connected to practically the whole of Europe. For the US, Ashburn N.Virginia is good but it's still only a single coast.
They are definitely paying under 2c/TB for traffic though.
Routers, optics & interconnects aren't free. $0.01/GB is very reasonable.
Wait. That's cheaper than my CDN. Maybe I should do some shopping
> So, "Pi mal Daumen"* this means that US customers have a bandwidth consumption which almost an order of magnitude higher than that of European and Singaporean customers?
I wouldn't be amazed if Hetzner benefits significantly from peering, which is much more widespread in US than in Europe. Interesting piece on this from Cloudflare: https://blog.cloudflare.com/the-relative-cost-of-bandwidth-a...
It's quite possible that their costs really are significantly lower in Europe. No idea what things are like in Singapore.
I wouldn't be amazed if Hetzner benefits significantly from peering, which is much more widespread in US than in Europe. Interesting piece on this from Cloudflare: https://blog.cloudflare.com/the-relative-cost-of-bandwidth-a...
It's quite possible that their costs really are significantly lower in Europe. No idea what things are like in Singapore.
You've written that the wrong way around. You've written the opposite of what Cloudflare writes.
They write that transit bandwith costs are similar in Europe and the USA, but Europe has more peering — it's around 50% of their traffic rather than 20%.
> The corollary is that in Europe transit is also cheap but peering is very easy, making the effective price of bandwidth in the region the lowest in the world.
They write that transit bandwith costs are similar in Europe and the USA, but Europe has more peering — it's around 50% of their traffic rather than 20%.
> The corollary is that in Europe transit is also cheap but peering is very easy, making the effective price of bandwidth in the region the lowest in the world.
Oops, yep, so I have. Should’ve read “much more widespread in Europe than the US”. Sorry, long day.
Really? Peering is very big in Europe, we have like 10+ CIX operators with 20+ locations in Germany alone.
I have been a fan of Hetzner for many years. $1/terabyte overage charges seems reasonable.
I live in Arizona, but have always rented Hetzner servers or VPNs in Germany. I imagine the service is much the same using servers in the USA?
I live in Arizona, but have always rented Hetzner servers or VPNs in Germany. I imagine the service is much the same using servers in the USA?
When has AWS done something like this?
LB11 going from 20TB to 1TB for the same price is wild if you’d built a business on this platform.
LB11 going from 20TB to 1TB for the same price is wild if you’d built a business on this platform.
It's easy to never raise prices if you have 1000% markup.
The point remains, if I was a customer who had planned my budget on the previously lower rates, this move’d be very disruptive.
Herzner is an established player, not a startup, this either shows a lack of regards for customers, or that they aren’t very well run.
Herzner is an established player, not a startup, this either shows a lack of regards for customers, or that they aren’t very well run.
If you need long term price security, than you need to get a long term contract. The big cloud providers will give you that if you pay them enough in advance, but probably for at least 50 times the price per TB.
If you use 20 TB each month the price will be 25.39€ instead of 5.39€. I can't think of any business that would seriously struggle with this 20€ monthly price increase.
Price increases are not a nice thing, but this one is not catastrophic.
Price increases are not a nice thing, but this one is not catastrophic.
Your traffic bill is increasing by 471% and that's not OK.
Bill increases don't have to be catastrophic to be bad. Remember that businesses/startups range from being well funded to not-funded-at-all-trying-to-survive. Depending on the country, 20€ can be a lot of money.
Bill increases don't have to be catastrophic to be bad. Remember that businesses/startups range from being well funded to not-funded-at-all-trying-to-survive. Depending on the country, 20€ can be a lot of money.
> Your traffic bill is increasing by 471% and that's not OK.
Your traffic bill is increasing by 20 bucks per month and that isn’t ok? If you’re running any sort of business and that isn’t ok I’m not sure what to tell you.
Your traffic bill is increasing by 20 bucks per month and that isn’t ok? If you’re running any sort of business and that isn’t ok I’m not sure what to tell you.
I've worked for a startup in my early years that was on a shoestring budget. We had no spare money. When we celebrated a big goal we bought a supermarket cake for $2 to go with the coffee. That happened max few times a year. 20 bucks is nothing when you have a lot. But if you have nothing then 20 bucks can be everything.
This. We got hit by a sudden change in a popular SaaS' pricing, from $10 to $75/mo - a 650% increase. We don't have a big margin, if a different provider did this sort of thing overnight, we could be instantly out of business. It's already difficult to build a competitive business even WITH the ability to outsource a class of problems to a SaaS.
I've been a big fan of Hetzner for the last decade, and I understand and agree with their motivation for this change. However December 1st is effectively almost tomorrow, they could have easily given us a month's notice instead.
I've been a big fan of Hetzner for the last decade, and I understand and agree with their motivation for this change. However December 1st is effectively almost tomorrow, they could have easily given us a month's notice instead.
You have until Feb for your existing infra, so that seems fair to me.
Yes,that is OK. It's still much cheaper than the alternatives.
Ya if you're on the cheapest service and and the next cheapest service is an order of magnitude higher or more then your business is already at risk. It's a sign that it's subsidized and that a pricing shock will happen in the future.
If it’s a competitive market you’re almost certainly exploiting some sort of anomaly that t will probably go away.
While perhaps not like this, AWS has from time to time ensured average rate goes up or introduced charges for something previously free:
https://www.prnewswire.com/news-releases/aws-and-azure-cloud...
https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address...
https://aws.amazon.com/about-aws/whats-new/2024/11/new-featu...
See notes 1 and 2: https://aws.amazon.com/cognito/pricing/
https://www.prnewswire.com/news-releases/aws-and-azure-cloud...
https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address...
https://aws.amazon.com/about-aws/whats-new/2024/11/new-featu...
See notes 1 and 2: https://aws.amazon.com/cognito/pricing/
Never. I don't think AWS have _ever_ icnreased prices.
They have, but usually it is via introducing additional fees to services/transactions, eg:
https://www.astuto.ai/blogs/understanding-the-aws-public-ipv... https://www.wiv.ai/navigating-the-rising-tide-of-aws-pricing...
https://www.astuto.ai/blogs/understanding-the-aws-public-ipv... https://www.wiv.ai/navigating-the-rising-tide-of-aws-pricing...
The IPv4 charge is a good one!
I thought this was to allow them to be more relaxed about the limit (5 per region) which is how they used to control fully free services that cost them.
But an increase for sure - they did note the supply of free ARIN allocations was gone
I thought this was to allow them to be more relaxed about the limit (5 per region) which is how they used to control fully free services that cost them.
But an increase for sure - they did note the supply of free ARIN allocations was gone
It's a matter of perspective, I don't do IPv6, when AWS decided to start charging for IPv4, I moved to Oracle Cloud.
AWS also charges multiples the price to begin with. I mean, the "scam" of AWS has always been the absolutely outrageous network egress pricing.
AWS raised the prices (/slashed the free tier) for Cognito literally last week[0], in a way that's quite similar to Hetzner.
[0]: https://saasprices.net/blog/aws-price-rise
[0]: https://saasprices.net/blog/aws-price-rise
IPv4 charge caused me to have to redesign some things and cull servers for some projects.
I decided to move to Oracle Cloud when they made that move.
Right - I’ve been on them since EC2 flat network / simple DB days and was trying to remember if I ever got an email like this.
I know google has jacked rates (maps etc) and killed services (I used their first paas before it was basically abandoned)
I have argued online with folks about their pricing - my point usually being as soon as you try to do Netflix or YouTube on the “Free” or unlimited or ultra low cost providers - you find out it’s a lie.
My impression was hetzner had started null routing customers for “abuse” who used a lot. No idea if that’s true, but used to be the way the “unlimited” VPS providers did it.
I know google has jacked rates (maps etc) and killed services (I used their first paas before it was basically abandoned)
I have argued online with folks about their pricing - my point usually being as soon as you try to do Netflix or YouTube on the “Free” or unlimited or ultra low cost providers - you find out it’s a lie.
My impression was hetzner had started null routing customers for “abuse” who used a lot. No idea if that’s true, but used to be the way the “unlimited” VPS providers did it.
I believe they have for very specific services, but never for things like EC2 or RDS.
There are also some EC2 instance classes where upgrading instance types in the same "size" are more expensive, but that is very rare, but I dont believe AWS has ever pulled the rug out from under you.
There are also some EC2 instance classes where upgrading instance types in the same "size" are more expensive, but that is very rare, but I dont believe AWS has ever pulled the rug out from under you.
Increasing prices is not "pulling the rug out from under you". Hertzer decided to raise their prices. Their customers can either continue using their service, reduce their use of the service, or go somewhere else.
Also, we live in a time of high inflation. We should expect price increases because they value of the dollar, euro, etc. is going down.
Also, we live in a time of high inflation. We should expect price increases because they value of the dollar, euro, etc. is going down.
> There are also some EC2 instance classes where upgrading instance types in the same "size" are more expensive
An increase in price has been the rule rather than the exception for recent upgrades for vanilla instance types, e.g., c, r, m types in the newest generations (6 -> 7 for x86, 6 -> 7, or -> 8 for Arm types).
The increases have been modest though, perhaps around 10%. You get additional CPU and sometimes minor increases in other resources on the newer types.
An increase in price has been the rule rather than the exception for recent upgrades for vanilla instance types, e.g., c, r, m types in the newest generations (6 -> 7 for x86, 6 -> 7, or -> 8 for Arm types).
The increases have been modest though, perhaps around 10%. You get additional CPU and sometimes minor increases in other resources on the newer types.
Bit annoying that the word tariff differs between the continents. Was thinking it related to possible trade tariffs
I wonder why English decided to narrow its meaning. Well I guess a language doesn’t decide anything, but I think you know what I mean. In many EU countries it simply means “cost list” or “price list” which can be for import duties but also for a range of other things.
It’s very nice to have it called tariff in basically every EU country though. We get green energy tariffs and if it was called something different in each country they wouldn’t be fun.
It’s very nice to have it called tariff in basically every EU country though. We get green energy tariffs and if it was called something different in each country they wouldn’t be fun.
Did they finally realize how AWS/Azure/Gcloud actually generate their exorbitant profit margins?
No, they realized that nobody switches away from AWS because of their expensive bandwidth.
Bingo, lots of people are loud about this online and many influencers screaming about this on Twitter, so it’s easy to conflate that as a pressing issue. Ultimately a thousand indie devs mad about bandwidth extortion are still making up less than 1% of a serious company’s revenue for AWS.
I’ve liked using Hetzner before and I’m curious how they’ll go after the market now. This is probably the wrong move though; Cloudflare realized this same fact some time ago but they kept their prices low to avoid cultivating ill will.
I’ve liked using Hetzner before and I’m curious how they’ll go after the market now. This is probably the wrong move though; Cloudflare realized this same fact some time ago but they kept their prices low to avoid cultivating ill will.
There probably are a number of things about AWS (see also no hard price caps) that people on forums froth at the mouth about but which most customers AWS actually cares about are largely indifferent about.
Can confirm that the enterprise I work for does not care one whit about the odd $5k mistake. It's all a drop in a bucket anyway.
What do you mean?
News Hacker isnt representative sample that also understands the insustry and the business?
News Hacker isnt representative sample that also understands the insustry and the business?
> lots of people are loud about this online and many influencers screaming about this on Twitter, so it’s easy to conflate that as a pressing issue. Ultimately a thousand indie devs mad about bandwidth extortion are still making up less than 1% of a serious company’s revenue for AWS.
So basically, by getting worked up publicly about this, those people provide free marketing for Hetzner?
So basically, by getting worked up publicly about this, those people provide free marketing for Hetzner?
AWS negotiates on bandwidth, too, so I imagine the actual savings would be a lot smaller for most customers unless you have some business which is heavily dominated by egress pricing.
So it seems that trying to compete with duopoly isn't working.
Can we assume it is cause there is only few big corporations dominating internet infrastructure in US compared to EU with tons of medium sized and even small business that do it?
I would love some good read about US infrastructure, especially why costs are so high compared to EU?
Can we assume it is cause there is only few big corporations dominating internet infrastructure in US compared to EU with tons of medium sized and even small business that do it?
I would love some good read about US infrastructure, especially why costs are so high compared to EU?
Which duopoly are you thinking? AWS, GCP, Azure + smaller (still significant) Digital Ocean, Linode?
Good for them! Continue to be profitable and competitive so I can keep using them 20 years from now when they still decimate big cloud at 100x pricing or big cloud value add layer companies at 200x pricing
Weird, one would expect that in anything related to technology either prices go down, or performance goes up over time.
That's usually true if you see the actual costs of the thing.
In the case of data transmission if you were using some data transmission system where you were charged per byte based on the operator's costs (possibly time dependent so the cost per byte might vary depending on the amount of traffic on the provider's network and on the varying real time prices of the networks they connect to) then you would indeed see prices going down over time and performance going up.
Consumers, small businesses, and often even medium businesses generally hate that kind of pricing. They like fixed monthly bills. So providers offer that, setting the amount of data included in that price high enough that most customers won't ever come near it.
That tends to result in the lower bandwidth users actually paying quite a bit more than they would if they had per byte pricing and the higher bandwidth (but not so high as to go over the included data and hit overage fees) paying less than they would under the per byte model.
That can attract more high bandwidth customers and eventually the model of customer bandwidth usage that was used to set the price and bandwidth allowance is no longer accurate and gets adjusted.
Note that this means that price you pay is not just a function of the underlying technology costs--it is also a function of how other people are using the service.
Same thing happens even in non-technology areas. You probably wouldn't go into a fixed price "all you can eat" restaurant just to get a donut and cup of coffee. The fixed price is set to cover people getting full meals. And if a bunch of competitive eaters started coming in every day to do their training at that restaurant you can safely bet that the price is going to go up for everyone or there is going to be an asterisk added to "all you can eat" with a footnote that puts some sort of cap on it.
In the case of data transmission if you were using some data transmission system where you were charged per byte based on the operator's costs (possibly time dependent so the cost per byte might vary depending on the amount of traffic on the provider's network and on the varying real time prices of the networks they connect to) then you would indeed see prices going down over time and performance going up.
Consumers, small businesses, and often even medium businesses generally hate that kind of pricing. They like fixed monthly bills. So providers offer that, setting the amount of data included in that price high enough that most customers won't ever come near it.
That tends to result in the lower bandwidth users actually paying quite a bit more than they would if they had per byte pricing and the higher bandwidth (but not so high as to go over the included data and hit overage fees) paying less than they would under the per byte model.
That can attract more high bandwidth customers and eventually the model of customer bandwidth usage that was used to set the price and bandwidth allowance is no longer accurate and gets adjusted.
Note that this means that price you pay is not just a function of the underlying technology costs--it is also a function of how other people are using the service.
Same thing happens even in non-technology areas. You probably wouldn't go into a fixed price "all you can eat" restaurant just to get a donut and cup of coffee. The fixed price is set to cover people getting full meals. And if a bunch of competitive eaters started coming in every day to do their training at that restaurant you can safely bet that the price is going to go up for everyone or there is going to be an asterisk added to "all you can eat" with a footnote that puts some sort of cap on it.
True, so what gives? Just them wanting more money now that they got enough customers? They probably did some calculations and realized that damn, they could pocket more money so might as well try their luck. Like yeah, let us assume they have 10k customers: 7.05 * 10000 is 70500, 8.99 * 10000 is 89900, that is 19400 USD more for them, and that is just for one!
Or the cause is one step removed, for example the handful of giant companies that control all US internet infrastructure, versus the hundreds all over Europe.
Yeah, so that probably means the count of users is higher than the previously assumed 10k. They can do it, so they will do it.
Not when there is a duopoly on one market (US) and hundreds of companies on other (EU).
This case is for a EU company's offerings to the US. Why would they make themselves less competitive?
How is there a duopoly in the cloud market in the US?
Duopoly in connectivity.
adventured(5)
The old allowance always struck me as unusually generous tbh
I think OVH has a more logical bandwidth policy. They give you a certain Mbps cap, and that’s that. I haven’t used them personally, though, so can’t vouch for the experience. I’m curious to hear from folks who have used both providers.
From what I've heard if you actually saturate the link they'll get in touch. I'm not aware of any truly unlimited data transfer plans from VPS providers.
It sounds like I can just move my bandwidth hungry servers to europe and eat the latency penalty- is that a correct read?
That's correct but assuming you want > 300 MB/s from EU to NA that will be a struggle.
If you don't care about high upload/download speed from/to NA<->EU then yes, that's a good move. Otherwise closeness in geo is still king.
If you don't care about high upload/download speed from/to NA<->EU then yes, that's a good move. Otherwise closeness in geo is still king.
As long as the link doesn't drop or reorder packets too badly a >300MB/s (or probably given we're talking about small virtual machines 300Mb/s) flow doesn't require much tuning. For a single long lived TCP connection allowing the window size to scale to ~2x the expected bandwidth delay product should be enough. So a simple things like large HTTP uploads or downloads will work fine once the congestion window has grown.
The real problem is going to be anything more complex than that. Most request-response protocols too chatty and won't send enough independent outstanding requests to fill up the bandwidth delay product of an intercontinental link which will kill the throughput. Also users don't like to wait for slow responses no matter what throughput you can sustain for large transfers.
The real problem is going to be anything more complex than that. Most request-response protocols too chatty and won't send enough independent outstanding requests to fill up the bandwidth delay product of an intercontinental link which will kill the throughput. Also users don't like to wait for slow responses no matter what throughput you can sustain for large transfers.
Yes. And you're not even exploiting the system, just using it as it is - bandwidth really is cheaper over here.
[deleted]
They recently changed to bill by the hour. Not hard to destroy and reprovision once you're near the traffic limit.
Massive loophole.
Massive loophole.
They don't pro-rate the included traffic quota for servers that don't run a full month?
No.
Yeah but if you're running something that is using a ton of bandwidth, presumably it's a service that needs to be online and reliable. Being cheap and constantly cycling out servers introduces complexity to your system. Hetzner could also easily change their terms to close this loophole if it's an actual problem.
Oof, that reduction in bandwidth is huge.
Am I correct that this only applies to outgoing WAN traffic? Incoming/internal is still free?
Am I correct that this only applies to outgoing WAN traffic? Incoming/internal is still free?
That is correct, incoming traffic is still free
"Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources." ... This argument does not make sense. I use very few resources but have to pay more.
The argument would make sense if the prices of any Hetzner service decreased (no longer necessary to substitute high traffic users).
But all prices in this message increase.
But all prices in this message increase.
Yeah, my price just changed from €7.05 to €8.99—in the name of fairness to low impact users? It’s brazenly nonsensical.
It would be so much better if they’d just said “sorry but we need to raise prices”. The attempt to sugarcoat it is insulting to begin with, and it’s only made worse that their backwards excuse comes across as thoughtlessly trampling us.
It would be so much better if they’d just said “sorry but we need to raise prices”. The attempt to sugarcoat it is insulting to begin with, and it’s only made worse that their backwards excuse comes across as thoughtlessly trampling us.
I personally use 0TB per month across 6 CPX21 servers (I know I’m over-provisioned; that’s not the point).
Kinds of is the point tho, you're hogging resources you don't actually need (not talking about traffic here ..).
People probably spam-provisioned cheap CPX boxes to get cheap bandwidth.
Also, complaining about a "large 27.52% price increase" is kind of absurd when the absolute value of the increase is just under 2€.
Kinds of is the point tho, you're hogging resources you don't actually need (not talking about traffic here ..).
People probably spam-provisioned cheap CPX boxes to get cheap bandwidth.
Also, complaining about a "large 27.52% price increase" is kind of absurd when the absolute value of the increase is just under 2€.
> Kinds of is the point tho, you're hogging resources you don't actually need (not talking about traffic here ..).
That seems backwards? They're not hogging anything, they're paying for network they're not using; this is excellent for Hetzner.
That seems backwards? They're not hogging anything, they're paying for network they're not using; this is excellent for Hetzner.
[deleted]
Are tariffs already in place or is this just a thinly-veiled scapegoat for haircutting traffic allocation by 95%? To a customer, it certainly feels like a bait and switch to sell a subscription product and once customers are embedded materially change the economic trade.
It's the language barrier. The German word Tarif doesn't mean the same as the English word tariff.
It's also used in that sense in English (in telecom/utilities, airlines, etc.), just that the political/taxation usage is more heavily covered, especially lately.
[deleted]
Well actually one meaning of the English word tariff is the same as the German meaning, although it's not as widely used. To quote Wiktionary:
> tariff (plural tariffs)
1. A system of government-imposed duties levied on imported or exported goods; a list of such duties, or the duties themselves.
2. A schedule of rates, fees or prices.
3. (British) A sentence determined according to a scale of standard penalties for certain categories of crime.
...so Hetzner's usage of the word is technically correct™, even though native speakers might not use it in this context.
> tariff (plural tariffs)
1. A system of government-imposed duties levied on imported or exported goods; a list of such duties, or the duties themselves.
2. A schedule of rates, fees or prices.
3. (British) A sentence determined according to a scale of standard penalties for certain categories of crime.
...so Hetzner's usage of the word is technically correct™, even though native speakers might not use it in this context.
It's closer to industry jargon at this point in American English. Search for LTL tariffs, for example, and you'll find a very long list of trucking companies publishing their fees and terms as tariffs.
It's completely normal usage in Britain.
"I changed electricity provider to one with an EV tariff."
"I changed electricity provider to one with an EV tariff."
Yes, that's really funny. But even funnier, I can't think of a 1-1 English word, and even Google translate gives me tariff. It's actually just "price", but in the context of these kinds of services, could be also something like "tier" (but not to be confused with the German Tier :-)).
I’m not sure the meanings are really different. It’s just that tariff usually refers to import duties in the US.
People arguing that’s the only US meaning are just wrong though
People arguing that’s the only US meaning are just wrong though
But that's normal for languages, the meaning of a word can adjust to the point a meaning previously used becomes archaic. It's obvious these two words share the same for lack of a better word gist, but the actual usage diverged later.
This has nothing to do with any possible trade wars or trade tariffs.
The word tariff is often used in telecom to indicate rates and fees for some given quantity of services, and that seems to be the use here.
The word tariff is often used in telecom to indicate rates and fees for some given quantity of services, and that seems to be the use here.
By tariff they just mean contract pricing, not the tax kind.
I think this is fair. All their competitors have crazy pricing for bandwidth, so why should be they be generous ?
That's, non ironically, the answer
Pulling the price too low in comparison, let's say with AWS, will just make your life harder but will not squeeze them in any way
Pulling the price too low in comparison, let's say with AWS, will just make your life harder but will not squeeze them in any way
Their email makes sense, except it doesn't. As someone who considers themselves a low traffic usage customer, running hundreds of Hetzner Cloud servers - each averaging less than 10GB of data transfer, with only a few servers reaching the previously included 50-60TB/month - I’m now facing an overall increase in costs.
Not only are the per-server prices higher, but the drastic cut in included bandwidth (without a corresponding option for bandwidth pooling) means I'll be paying significantly more despite my usage being well within "low traffic" by their standards. It’s frustrating that Hetzner never introduced a bandwidth pooling option for customers where it would make sense, especially in scenarios like mine where usage is highly imbalanced across servers.
At this point, Vultr and Linode are starting to make a lot more sense, even with their more expensive traffic pricing, since both providers offer traffic pooling. This feature would have significantly softened the blow of Hetzner's pricing changes, but instead, they’re pushing costs onto loyal customers who don’t fit neatly into their new model.
Not only are the per-server prices higher, but the drastic cut in included bandwidth (without a corresponding option for bandwidth pooling) means I'll be paying significantly more despite my usage being well within "low traffic" by their standards. It’s frustrating that Hetzner never introduced a bandwidth pooling option for customers where it would make sense, especially in scenarios like mine where usage is highly imbalanced across servers.
At this point, Vultr and Linode are starting to make a lot more sense, even with their more expensive traffic pricing, since both providers offer traffic pooling. This feature would have significantly softened the blow of Hetzner's pricing changes, but instead, they’re pushing costs onto loyal customers who don’t fit neatly into their new model.
In the second example charging 28% more for 90% less traffic, starting in 3 days. That's straight up illegal in some parts of the world, but apparently not in the US?
The pricing applies immediately only for new customers. For existing ones, it applies from Feb 2025.
Those are some steeeep drops. I'm curious how they settled upon such numbers.
It’s kind of not the brightest to both raise prices and reduce services at the same time, who’s in charge over there? Maybe I’m missing something ?
What if their choice was either raise prices or lose money?
Yes this hit me. We use a ton of traffic. Any alternatives? Although the pricing isn't too bad even with the increase, nice to have backups.
We use Hivelocity. Not exactly the same service (bare metal).
Good. Maybe they can stop null routing traffic to paid customer accounts because their abuse detection false-positives constantly.
So it seems that trying to compete with duopoly isn't working.
Can we assume it is cause there is only few big corporations dominating internet infrastructure in US compared to EU with tons of medium sized and even small business that do it?
I would love some good read about US infrastructure, especially why costs are so high compared to EU?
AWS, Microsoft, and even Google provide a lot of features large enterprises want. A cheaper competitor which expects you to build more of that yourself entails larger staffing costs and that’s going to sound riskier if you aren’t great at consistently delivering software projects.
The per capita GDP of Germany is $52,745.76. The US is $81,695.19. The EU as a whole is $43,194. Median household income is $38,971 in Germany and $80,610 in the US.
When people cost twice in one place versus another companies will focus on more expensive but less people intensive approaches.
When people cost twice in one place versus another companies will focus on more expensive but less people intensive approaches.
> Median household income is $38,971 in Germany
That is a median "disposable income" (after taxes and transfers), which is not directly comparable to the US figure you cited (which is gross [before taxes]).
That is a median "disposable income" (after taxes and transfers), which is not directly comparable to the US figure you cited (which is gross [before taxes]).
However, we are also considering infrastructure costs. The median salary in Germany is approximately 60,000 USD, which should probably be taken into account when comparing company costs to those in the USA, which are around 70-80,000 USD.
Seems like the companies in US trust AWS just in case and this too cheap german company is suspicious. I would take guess that those who actually use Hetzner in US use it solely to abuse the generous traffic and not care that much about compute. Where in EUrope many companies never jumped on AWS in the first place.
To be fair their pre-change allowances were insanely generous.
My take on this is that there’s a price increase on power consumption and Hetzner decided to bump the prices now; perhaps they are facing a power price increase in January 2025. Do Ashburn or Hillsboro publish their power prices and have they increased in the past year?
A lot less bandwidth and higher prices with less than a week until it goes live. That's insane.
Less than a week for newly created resources. At least they’re allowing until February for existing resources.
I would like to know more about their bandwidth costs to learn if:
* they have to pay that much more for (outbound) traffic in the US
* miscalculated their US expansion and are trying to recoup
* they think customers will only bitch, but stay anywaysDoes Hetzner provide a way to limit how much traffic a cloud server can use?
I'm always afraid to use paid services to host my hobby projects, because what happens when some malicious or stupid actor with huge bandwith hotlinks an image from my site or downloads it in a loop?
What would be perfect would be a way to set an upper limit of say 100 TB per month after which the cloud server does not accept outbound traffic anymore and I get an email.
Or is the network speed already enough of a protection? Their site says it is a 1GB connection. That should limit the traffic to 300TB per month? If so, that would also be fine with me.
I'm always afraid to use paid services to host my hobby projects, because what happens when some malicious or stupid actor with huge bandwith hotlinks an image from my site or downloads it in a loop?
What would be perfect would be a way to set an upper limit of say 100 TB per month after which the cloud server does not accept outbound traffic anymore and I get an email.
Or is the network speed already enough of a protection? Their site says it is a 1GB connection. That should limit the traffic to 300TB per month? If so, that would also be fine with me.
hetzner is really known for its german bandwidth prices, a change in the US is fairly insignificant IMO. for most applications you could just put a free CDN in front of the cheaper german service to reach the US
This is pretty steep... their website still seems to list the old included traffic and price (though okay, I guess they still have 3 days to update). Is there a more official link, or was it distributed through email?
Haven't been able to find a link. Seems like it was only distributed by email — which isn't great communication IMO
I know it's a very simple question but what is the cost of bandwidth? I appreciate that a server is a machine with parts that degrade. I also appreciate that lines need to be laid for the most part and there is a need for a return on digging up the streets to lay cable.
So is bandwidth going to the cable?
I am asking that because I never quite understood 3g/4g/5g costs. In my youth it was $0.10US for a text message. Now in my country, no one would pay to send a standard text. It's free even on PAYG.
I'd imagine there is similarity?
So is bandwidth going to the cable?
I am asking that because I never quite understood 3g/4g/5g costs. In my youth it was $0.10US for a text message. Now in my country, no one would pay to send a standard text. It's free even on PAYG.
I'd imagine there is similarity?
On a Hetzner level, you don't quite know, as bandwidth deals between large hosting companies and IP transit providers like Hurricane Electric, Cogent, Zayo, etc, are typically protected by non-disclosure agreements.
Unless you are a big enough fish (FAANG size) in the internet pond to do free peering with all the big boys, the contract with be negotiated for a certain amount of dedicated bandwidth (specified in megabits or gigabits per second), not like the terabyte like in the consumer LTE context.
Unless you are a big enough fish (FAANG size) in the internet pond to do free peering with all the big boys, the contract with be negotiated for a certain amount of dedicated bandwidth (specified in megabits or gigabits per second), not like the terabyte like in the consumer LTE context.
My first smartphone had a data plan which cost 1024€/Go. And I was happy about how cheap it was (it got more expensive shortly after).
Cellular towers aren't cheap: you need to install quite a lot of them, especially in big cities!
Cellular towers aren't cheap: you need to install quite a lot of them, especially in big cities!
Cloud providers make their money with the high bandwith costs. I can get 10 gbit/s line for ~$3,000 a month with a dedicated server. That's 3 PB of data and according to AWS cloudfront calculator it would cost ~$90,000 with them...
So yeah, people get ripped off, just like with SMS in the past. Most don't know this or think it's fine, because "cloud" has benefits that justify the costs in the end ("I don't have to pay for sys admin!").
So yeah, people get ripped off, just like with SMS in the past. Most don't know this or think it's fine, because "cloud" has benefits that justify the costs in the end ("I don't have to pay for sys admin!").
The bandwidth reductions look like they bring the US in line with Singapore, i.e. typically 20TB of bandwidth in the EU and 1TB in Singapore, US moving from 20 to 1.
I wonder if their cost base has changed significantly. Singapore having just 1TB included makes sense because in Singapore traffic is charged differently and costs a huge amount for all companies. I wonder if the colo provider for Hetzner in the US has screwed them over in some way.
I wonder if their cost base has changed significantly. Singapore having just 1TB included makes sense because in Singapore traffic is charged differently and costs a huge amount for all companies. I wonder if the colo provider for Hetzner in the US has screwed them over in some way.
Hetzner has never really competed well in the US market and I often find Lightsail, OVH, and Digital Ocean cover my hosting needs pretty well.
For what it is worth, I have also have used 2nd-hand resellers like SoYouStart (OVH). You get what you pay for, but ballin on a budget is totally possible.
Back in the day I used 1&1, I wonder how they are faring these days. I am sure there are dozens of other viable providers, you just need to shop around.
For what it is worth, I have also have used 2nd-hand resellers like SoYouStart (OVH). You get what you pay for, but ballin on a budget is totally possible.
Back in the day I used 1&1, I wonder how they are faring these days. I am sure there are dozens of other viable providers, you just need to shop around.
Wow. You literally cannot pay them 10 times the price to give you that much bandwidth now. If you were using 20TB they just don't want you as a customer anymore
Literally quoting billing support: "It is a privilege to have the right to be a Hetzner customer"
What the... "a privilege to have the right?"
I don't think they know what rights are...
I don't think they know what rights are...
This feels like a bait-and-switch, especially after taking the time to learn Hetzner and start telling my peers about Hetzner. I understand that pricing structures need to update, but good god existing users should have been kept as-is going forward.
This behavior, and the cute little "cancel anytime!" has made me pause suggesting Hetzner in my circles as I re-evaluate their company.
This behavior, and the cute little "cancel anytime!" has made me pause suggesting Hetzner in my circles as I re-evaluate their company.
I wonder what's so special about Ashburn and Hillsboro... Did the local ISPs suddenly increase the prices or what?
>Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced.
Isent this how every ISP works?
You and all your neighbors can subscribe to 1Gigabit because they don't anticipate everyone maxxing out the bandwidth at once?
Isent this how every ISP works?
You and all your neighbors can subscribe to 1Gigabit because they don't anticipate everyone maxxing out the bandwidth at once?
For speed that's how it works, but not for data caps.
"data caps" aren't real though from a networking perspective. A router or switch will push packets at line rate speed, not based on the total amount of storage or cap.
For anyone regularly above the new 1TB limits, or using close to 20TB already on these services - what kinds of use cases are these?
If it was a basic, or private CDN, for example, the lag time might be pretty negligible to stream out of EU for the extra bandwidth.
If it was a basic, or private CDN, for example, the lag time might be pretty negligible to stream out of EU for the extra bandwidth.
Is this a signal of a larger pivot in their business model towards targeting a higher-cost US enterprise market? A lot of brands have successfully transitioned to selling the same goods at luxury prices recently-maybe a webhost with a decent enough reputation can do the same.
Probably more they don't need or want the companies gaming this pricing at scale any more.
If you are spinning up a $5/mo VM, using 19.5TB of bandwidth on it, then spinning it down and firing up the next, you are a cost center.
This change boots those customers off the service entirely without having to write complicated ToS. The price change for average customers won't even be noticed on the next monthly bill, so it's likely seen as a win/win at the moment.
At some point the marketing dollars stop getting spent as heavily when you reach a certain market saturation. Calling this luxury pricing is certainly a stretch considering it's an order of magnitude less than the large cloud providers still. It's just not below cost any longer.
If you are spinning up a $5/mo VM, using 19.5TB of bandwidth on it, then spinning it down and firing up the next, you are a cost center.
This change boots those customers off the service entirely without having to write complicated ToS. The price change for average customers won't even be noticed on the next monthly bill, so it's likely seen as a win/win at the moment.
At some point the marketing dollars stop getting spent as heavily when you reach a certain market saturation. Calling this luxury pricing is certainly a stretch considering it's an order of magnitude less than the large cloud providers still. It's just not below cost any longer.
I went from being a big fan of hetzner to being pretty angry because of this change and how it was communicated.
The price change is one thing, the MASSIVE change in traffic you get for it is another. Together, they suck. to go from 20Tb to 1Tb feels like a massive bait-and-switch.
The price change is one thing, the MASSIVE change in traffic you get for it is another. Together, they suck. to go from 20Tb to 1Tb feels like a massive bait-and-switch.
I never thought the 20TB of bandwidth was going to last forever. I guess here we are.
I wonder what proportion of traffic is AI model weights being served. They are huge files, lots of mostly duplicated + fine tuned versions, lots of interest to download the latest and greatest, lots of ML aficionados grabbing them.
As a Hetzner client, any price rise is disappointing. We are compute-heavy, not egress-heavy, user so will be largely unaffected by these changes, but I'm still a yuge Hetzner fan.
Same here - their compute pricing and performance were excellent value compared to the major cloud providers.
At least from that link, they are not talking about bandwidth, but included traffic.
I finally gave in to all the word of mouth marketing and moved to Hetzner about a month ago.
Fuck.
Fuck.
I did 4 days ago! Very upsetting email.
Nobody likes price rises but many companies are doing it due to disappointing year end financials and needing some positive news for next year.
IMO 2025 will a big year for being forced to run lean (no DevOps teams trying to emulate Google, ditching pointless microservices architecture, reducing JavaScript churn etc) and having to be agile in responding to vendor price changes. And of course CTOs desperately thinking AI will reduce the wage bill with no impact
IMO 2025 will a big year for being forced to run lean (no DevOps teams trying to emulate Google, ditching pointless microservices architecture, reducing JavaScript churn etc) and having to be agile in responding to vendor price changes. And of course CTOs desperately thinking AI will reduce the wage bill with no impact
This has nothing to do with Hetzner. It's because of the US tariffs.
"I’ve been a big fan of Hetzner. Unfortunately they’ve made a feeble attempt to dress this change up in the name of “fairness”."
Hetzner is a company know for it's precise pricing structure. An increase in prices would be correlated to an increase in costs, and in fact the next paragraph Hetzner writes:
"With the new tariff structure, we want to make conditions for our customers around the world as fair as possible "
Bottom line, the US imposes tariffs, this increases the prices of imported products, of which Hertzner is one, (Servers from Europe)
How can you write an entire article to complain about a price increase and not see that it was actually your country that increased the price.
"I’ve been a big fan of Hetzner. Unfortunately they’ve made a feeble attempt to dress this change up in the name of “fairness”."
Hetzner is a company know for it's precise pricing structure. An increase in prices would be correlated to an increase in costs, and in fact the next paragraph Hetzner writes:
"With the new tariff structure, we want to make conditions for our customers around the world as fair as possible "
Bottom line, the US imposes tariffs, this increases the prices of imported products, of which Hertzner is one, (Servers from Europe)
How can you write an entire article to complain about a price increase and not see that it was actually your country that increased the price.
They're not importing the VMs from Europe. And there are no new tariffs yet, inauguration is in January so there are only various announcements so far.
They're using tariff to mean price, which is an unusual choice and likely because it was written by someone from Germany. This would not necessarily sound out of place to a German speaker as "Tarife" is a common way to describe differently priced plans for any kind of service.
They're using tariff to mean price, which is an unusual choice and likely because it was written by someone from Germany. This would not necessarily sound out of place to a German speaker as "Tarife" is a common way to describe differently priced plans for any kind of service.
It's British English. Tariff is used to mean import duties, business prices (e.g. "phone tariff") and also prison sentences (e.g. "whole life tariff").
Though given the political context, it would probably have made sense to use a different word.
Though given the political context, it would probably have made sense to use a different word.
It's most likely imprecise wording due to a linguistic false friend from German (Hetzner is a German company); see also my other comment.
It's not a false friend, it is the same word.
Not only do they share an etymology (the word "tariff" in English and "tarif" in German both come from the same Arabic word), they also mean the exact same thing. It's also used in a bunch of other European languages in essentially the same way, with either a similar or identical spelling.
It's only US English where some of the meanings that are still common elsewhere are no longer used. It's actually kind of amusing that Americans would struggle to understand the meaning of an English word, but a French or Italian speaker would understand it. I wonder if there are any other words like this.
Not only do they share an etymology (the word "tariff" in English and "tarif" in German both come from the same Arabic word), they also mean the exact same thing. It's also used in a bunch of other European languages in essentially the same way, with either a similar or identical spelling.
It's only US English where some of the meanings that are still common elsewhere are no longer used. It's actually kind of amusing that Americans would struggle to understand the meaning of an English word, but a French or Italian speaker would understand it. I wonder if there are any other words like this.
Having the same etymology and even having the exact same spelling in two languages does not disqualify a word from being a false friend at all!
Take "gift", for example: "Gift" means "poison" in German, despite both the English and German word deriving from the same Proto-Germanic root, meaning "to give".
> (the word "tariff" in English and "tarif" in German both come from the same Arabic word), they also mean the exact same thing
In this very thread you can find a pretty good counterexample to that proposition, at least in American English.
Take "gift", for example: "Gift" means "poison" in German, despite both the English and German word deriving from the same Proto-Germanic root, meaning "to give".
> (the word "tariff" in English and "tarif" in German both come from the same Arabic word), they also mean the exact same thing
In this very thread you can find a pretty good counterexample to that proposition, at least in American English.
I said same etymology AND same meaning. They are the same word, not false friends.
US English may have dropped that meaning, but British English still uses it in this way, as do many Europeans when they speak English. It’s a correct translation.
US English may have dropped that meaning, but British English still uses it in this way, as do many Europeans when they speak English. It’s a correct translation.
This is an announcement targeted at US customers. One could assume that at least some of these might assume the American English sense of the word.
"Tarif" indeed roughly translates to "plan" in this context and can be used for all kinds of plans/rates/tariffs, not just the import duty kind.
No new tariffs yet. My bad. Misread
How do you tax internet traffic?
> US imposes tariffs
That didn’t happen yet and isn’t even 100% guaranteed to happen. So maybe it would be reasonable to wait?
Also what makes you think they are shipping their servers from Europe? How is them being an European company even relevant?
> US imposes tariffs
That didn’t happen yet and isn’t even 100% guaranteed to happen. So maybe it would be reasonable to wait?
Also what makes you think they are shipping their servers from Europe? How is them being an European company even relevant?
Cool: then maybe they can precisely explain the full breakdown of these changes instead of hiding behind a word that explains nothing.
Because of announcements of tariffs?
Probably more likely has something to do with increased energy costs.
Pretty sure VA/US energy prices are a lot cheaper than German ones.
I think it wouldn't be very relevant. A surge in US energy prices would be sufficient to cause a rise in US compute prices.
I think the US has been scaling compute for both AI and mining the last year, so it makes sense
I think the US has been scaling compute for both AI and mining the last year, so it makes sense
charging for bandwidth is theft in the first place. pure scam. sure, companies that built the infrastructure had to get heir money back, but that was in the past. they are now just milking their position where very few companies in the whole world can compete. not even patents are exclusive for ever. yet these companies treat their cables like they are a goose laying golden eggs now and for ever. this is one thing where governments should step in.
[deleted]
[deleted]
Is anyone old enough to remember 1and1? Similar arc?
Maybe I'm too young to remember the good times, but to me 1and1 were just the European GoDaddy. Mostly targeting small businesses who didn't know any better and ripping them off with bad services at inflated prices, through a lot of well targeted marketing. Selling things like email with a 250MB inbox and 2MB attachment limit far beyond the time when that was a reasonable offering, and at a far higher price than that was worth (being worth roughly zero).
They started with free .com registration.
They grew their business with this offer, and then degenerated into what you described.
They grew their business with this offer, and then degenerated into what you described.
1and1 were also notorious for aggressively trying to collect on "debts" - more than one person I know had 1and1 send debt collection letters to their parents over renewal fees for hosting plans they didn't even want.
They still exist, called ionos now: https://www.ionos.com/
Yes, I should have been more clear. It was another Germany? based provider that gave excellent, even free prices... until they didn't.
What their announcement does not cover is whether or not the 1 EUR/TB price above the free quota will still be active in the US.
[deleted]
Was also disappointed after receiving this update today, price increases across the board and gone are the days of generous 20TB free traffic which we've been enjoying for over a decade. Our Hetzner VMs now limited to 1-3TB free traffic, feels like the end of an era.
This is only for USA, isn't it?
It was expected considering during the last year everyone on Twitter was shouting left and right that everyone should use Hetzner because it is cheap.
Yeah it was cheap - only because the demand was low and they were competing on price.
Now it will only go up.
Yeah it was cheap - only because the demand was low and they were competing on price.
Now it will only go up.
But, really? What are you doing, that your VMs need more than 1TB of data per month? That's a huge amount! That's 23MB per minute, continuously.
If you have a website with that kind of reach, the prices seem entirely reasonable.
If you have a website with that kind of reach, the prices seem entirely reasonable.
I agree it is still reasonable.
But some examples of things that are profitable under the old regime, but maybe not at 20x the limit:
API services that consume media, whether to do some business logic specific thing or something specific, e.g. Transcriptions for videos, file conversion API, OCR API provider, etc.
If you try a video app without using WebRTC and peer traffic your bandwidth can blow up too. Even if you use something like push notifications with base64. There are lots of traps that use Bandwidth.
But some examples of things that are profitable under the old regime, but maybe not at 20x the limit:
API services that consume media, whether to do some business logic specific thing or something specific, e.g. Transcriptions for videos, file conversion API, OCR API provider, etc.
If you try a video app without using WebRTC and peer traffic your bandwidth can blow up too. Even if you use something like push notifications with base64. There are lots of traps that use Bandwidth.
We have about 50TB for about 500.000 users per month. Our Hetzner costs are about 2k per month. Similar setup at aws would cost us about 10 to 20k per month. No joke.
You can get pretty cheap servers with massive storage (2x2TB for example) and use them as backup targets.
I would love to see a VPS with transparent upstream costs so we have some idea of what's fair.
Inflation, the real inflation is 10%
Inflation doesn't explain reducing the included traffic from 20TB to 1TB while simultaneously increasing prices. This is a much more dramatic change than what inflation would justify.
Things are rarely priced at actual face value.
You purchase a 10Gb/s firewall for $100,000 - you will not be using 10Gb/s traffic for the lifespan of this device.
Applying this to Hetzner:
You sell a service with X bandwidth included free because you know that only Y% is only ever used on average.
Now people exploit the X allowance - spinning up new virtual machines to multiply this already generous allowance to get unlimited bandwidth for a fee 1/10000th of other commercial offerings. Your Y% costing is now completely invalid.
You reduce the allowance 20x to mitigate this.
I can't blame Hetzner at all for this, especially when Google/Amazon/Microsoft are printing money with their insane bandwidth costs. You know they are insane when they then change the rules to say it's completely free if you are migrating to a different provider - suddenly it doesn't cost anything at all for egress? Oh, it was actually upcoming monopoly investigations that might have taken a dim view...
You purchase a 10Gb/s firewall for $100,000 - you will not be using 10Gb/s traffic for the lifespan of this device.
Applying this to Hetzner:
You sell a service with X bandwidth included free because you know that only Y% is only ever used on average.
Now people exploit the X allowance - spinning up new virtual machines to multiply this already generous allowance to get unlimited bandwidth for a fee 1/10000th of other commercial offerings. Your Y% costing is now completely invalid.
You reduce the allowance 20x to mitigate this.
I can't blame Hetzner at all for this, especially when Google/Amazon/Microsoft are printing money with their insane bandwidth costs. You know they are insane when they then change the rules to say it's completely free if you are migrating to a different provider - suddenly it doesn't cost anything at all for egress? Oh, it was actually upcoming monopoly investigations that might have taken a dim view...
I thought of giving a recommendation here but I fear that they would raise the prices too... :|
Doesn't bother me with those measly 50GB my mail server is using in a month...
I would think an auction system would be the best system to price bandwidth.
Enron tried this in the late 90s/early 2000s.
That didn't work for a number of reasons (cooking the books), but also network bandwidth is not fungible. Unlike commodities such as oil or natural gas, bandwidth’s value is highly dependent on specific factors like location, time, and network conditions. This variability makes it difficult to standardize bandwidth as a tradable commodity, complicating efforts to create a seamless trading market.
There are a few in the crypto/DePIN space poking at this problem. I remain highly skeptical.
That didn't work for a number of reasons (cooking the books), but also network bandwidth is not fungible. Unlike commodities such as oil or natural gas, bandwidth’s value is highly dependent on specific factors like location, time, and network conditions. This variability makes it difficult to standardize bandwidth as a tradable commodity, complicating efforts to create a seamless trading market.
There are a few in the crypto/DePIN space poking at this problem. I remain highly skeptical.
Theoretically would be cool. Basically you have a docker that can run anywhere and you automatically migrate it based on prices between different service providers. The issue is there isn't incentives for the cloud providers to do this, because it wouldn't benefit the incumbents.
Maybe if the government mandated it at some point, like phone number portability was mandated.
Maybe if the government mandated it at some point, like phone number portability was mandated.
It’s also bad for customers because you wouldn’t have predictability in your cost structure.
That could work for some use cases where you transfer in bulk, like backups, CDN sync, research data transfer, etc. Either auction or off-hours or "spot"/low-QoS.
[deleted]
> Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources.
That is utter nonsense. If the customers who are 'covering the costs' have a problem, they can move? Yet even still they are charging those SAME customers who now actually receive less resources, even if they were using them or not.
That is utter nonsense. If the customers who are 'covering the costs' have a problem, they can move? Yet even still they are charging those SAME customers who now actually receive less resources, even if they were using them or not.
The problem with their old pricing structure is that it attracts high-utilization customers who will max out their transfer every month. But yes, they raised prices for all of us. Their pricing was so low, my guess is it was never profitable enough to be worthwhile long-term. It brought them a lot of business, some of which they'd be happier without under the old terms.
Maybe that's exactly what happened. So now nobody is covering the free loaders and they need to charge them.
> That is utter nonsense.
Hardly. Some level of cross subsidisation happens in all service of this nature. Depending on use youre either paying or receiving
Most Providers and customers just ignore it as inconsequential though.
Hardly. Some level of cross subsidisation happens in all service of this nature. Depending on use youre either paying or receiving
Most Providers and customers just ignore it as inconsequential though.
The reasoning the gave still doesn't pass the smell test. If it's as they say, then they wouldn't increase the price they would tariff the traffic. Like other providers.
Don't increase prices and claim it's to help the little guy.
Don't increase prices and claim it's to help the little guy.
Their claim that "low usage customers subsidize high usage ones" is particularly disingenuous - if that were truly the issue, they'd offer better rates to low-bandwidth users rather than implementing blanket increases and severe bandwidth reductions across the board.
What's most troubling isn't even the price hike itself, but the 3-day notice period. For those of us running auto-scaling infrastructure, this is essentially immediate - new instances will pull from the new pricing tier right away, even if existing instances have until February. This kind of abrupt change makes it hard to trust them for production workloads.
What's most troubling isn't even the price hike itself, but the 3-day notice period. For those of us running auto-scaling infrastructure, this is essentially immediate - new instances will pull from the new pricing tier right away, even if existing instances have until February. This kind of abrupt change makes it hard to trust them for production workloads.
I run a dedicated server with them that does about 1 TB inbound / 50 GB outbound daily.
If they start fucking people over on the dedicated servers as well, this product is basically dead because I can't find anywhere else that allows me to do this at the current price point...
Unless of course I slap a box under my desk and hook it up to my fiber internet.
If they start fucking people over on the dedicated servers as well, this product is basically dead because I can't find anywhere else that allows me to do this at the current price point...
Unless of course I slap a box under my desk and hook it up to my fiber internet.
The traffic limit is outbound only. There is no limit on inbound / internal traffic. So even if they were to hit you with a 5TB / month limit, you'd be just fine.
https://docs.hetzner.com/robot/general/traffic/ (bottom of the page)
https://docs.hetzner.com/robot/general/traffic/ (bottom of the page)
This is correct. No idea why it was flagged. Hetzner only counts outbound traffic.
Pretty clear this is only affecting US cloud services so you arent affected.
So, do bandwidth limited (instead of traffic metering limited) VPSes still exist?
I mean, to pull numbers out of ass, instead of $5 for 1.5 Tb of metered traffic on a 10 Gbps pipe, the $5 pays for 10 mpbs without metering.
I know protecting the client from extra unforeseen charges isn't predatory enough, but maybe some offerings like that still exist.
I mean, to pull numbers out of ass, instead of $5 for 1.5 Tb of metered traffic on a 10 Gbps pipe, the $5 pays for 10 mpbs without metering.
I know protecting the client from extra unforeseen charges isn't predatory enough, but maybe some offerings like that still exist.
Yeah. OVH prices things out that way.
If you are procrastinator, Hetzner is not for you.
Just got this via email. Well that's great, just as I moved a high bandwidth client to them a couple months ago. I love the "if you don't like it feel free to cancel" in the email also. SMH.
> customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources
Literally the business model of every "shared resource service model" on the planet. Hetzner's entire business model is built on this... them acting like they're shocked to discover this is... disingenuous at best.
Literally the business model of every "shared resource service model" on the planet. Hetzner's entire business model is built on this... them acting like they're shocked to discover this is... disingenuous at best.
It's now more expensive for bandwidth than Digital Ocean, completely destroying what was its value proposition previously.
I'm not sure what math you're basing this on, but it's far from correct. They include slightly less transfer in the instances, but their overage pricing is about 10% of DO and their instances are so much cheaper that you'll be paying much less even after adding a few TB.
For example compare the CPX11 to the cheapest similar offering from DO:
CPX11: 2vCPU, 2GB, 40GB storage (NVMe), 1TB transfer, ~$5 per month
DO Basic: 2vCPU, 2GB, 60GB storage (non-NVMe), 3TB transfer, $18 per month
Add 2TB more transfer at Hetzner and you're at ~$7 per month. Still a bit less storage though, let's also compare it to the CPX21:
CPX21: 3vCPU, 4GB, 80GB storage (NVMe), 2TB transfer, ~$9.50 per month. Add 1TB of transfer and you get more of everything for ~$10.50, ~40% less than at DO. And bandwidth overages at DO are about 10x as expensive.
We can also compare a higher tier:
CPX41: 8vCPU, 16GB, 240GB storage (NVMe), 4TB transfer, ~$32 per month
CPX51: 16vCPU, 32GB, 360GB storage (NVMe), 5TB transfer, ~$63.50 per month
DO Basic: 8vCPU, 16GB, 320GB storage (non-NVMe), 6TB transfer, $96 per month
For example compare the CPX11 to the cheapest similar offering from DO:
CPX11: 2vCPU, 2GB, 40GB storage (NVMe), 1TB transfer, ~$5 per month
DO Basic: 2vCPU, 2GB, 60GB storage (non-NVMe), 3TB transfer, $18 per month
Add 2TB more transfer at Hetzner and you're at ~$7 per month. Still a bit less storage though, let's also compare it to the CPX21:
CPX21: 3vCPU, 4GB, 80GB storage (NVMe), 2TB transfer, ~$9.50 per month. Add 1TB of transfer and you get more of everything for ~$10.50, ~40% less than at DO. And bandwidth overages at DO are about 10x as expensive.
We can also compare a higher tier:
CPX41: 8vCPU, 16GB, 240GB storage (NVMe), 4TB transfer, ~$32 per month
CPX51: 16vCPU, 32GB, 360GB storage (NVMe), 5TB transfer, ~$63.50 per month
DO Basic: 8vCPU, 16GB, 320GB storage (non-NVMe), 6TB transfer, $96 per month
I was basing it off of the transfer per instance. Using DO, that transfer limit is pooled, so if you need a lot of bandwidth, the best option was to get a lot of $5/month droplets, which each came with 1TB/month.
Looking at their pricing just now, it looks like it's $6/month for that droplet, which changes things a bit. Now, as long as you're willing to get droplets purely for the bandwidth allotment, DO's bandwidth costs $6/TB. That's more than almost every Hetzner option listed by the OP, but not all of them.
Looking at their pricing just now, it looks like it's $6/month for that droplet, which changes things a bit. Now, as long as you're willing to get droplets purely for the bandwidth allotment, DO's bandwidth costs $6/TB. That's more than almost every Hetzner option listed by the OP, but not all of them.
But overages from Hetzner are $1.05 per TB, which is significantly less than $6, so you can just get whichever instance you need and let it roll.
That's the key thing I missed. It's nowhere near the deal it was but it's still clearly ahead of DO.
I don't know why people praise Hetzner. They're not as good.
Unity moment.
Checks out
That’s a bit harsh but probably for understandable reasons, I wonder if DHHs shoutout to Hetzner in his last demo had an effect.
Oh shit, did some VC or PrivEquity buy Hetzner? Will we have HWS soon?...
lol, it's wierd
Used Linode for years. Brilliant service. Not sure how the Akamai takeover will pan out.
Anyone else looking to migrate. https://www.serverhunter.com/
Anyone else looking to migrate. https://www.serverhunter.com/
Well the Akamai takeover was 2 years ago, so fairly certain whatever you have now is how it ended up panning out.
I did panic a bit when it happened but my Linode just keeps working so I’m not unhappy.
I did panic a bit when it happened but my Linode just keeps working so I’m not unhappy.
Was it two years ago?? I could have sworn it was less than a year ago. Anyway yeah I’m happy with Linode
Ironically one of the reasons I abandoned Linode for Hetzner was because I assumed Akamai would ruin it. I still have stuff on Linode and I don't think Akamai ruined it.
With that said, Hetzner's prices are still cheaper and I've never had issues with their service, but I feel gaslit by this announcement.
With that said, Hetzner's prices are still cheaper and I've never had issues with their service, but I feel gaslit by this announcement.
As an age-old Linode customer from the UML days, I'd say that Akamai has done a good job with it. I've had nothing to complain about.
As a long term linode user, I think the enshittification is well underway :(
[deleted]
[deleted]
jstsch(1)
[deleted]
They specifically mention tarifs, maybe its political. Can't expect that kind of thing to get thrown around without international response.
Hetzner suck. So do Vultr (OVH?). People learn about the bad things cloud providers do but still go crawling back
We are writing to inform you about important changes to the tariff structure of our Cloud servers (CCX and CPX lines) and our Load balancers at our US locations in Ashburn and Hillsboro.
What will change?
Starting on 1 December 2024, 01:00 am CET, we will begin charging new prices for newly-created Cloud servers and introduce new amounts for included traffic for Cloud Servers and Load balancers at the US locations in Ashburn (ASH) and Hillsboro (HIL). This also applies to existing Cloud servers and Load balancers that are switched to a different tariff using the “Rescale” function. For any existing Cloud servers and Load balancers you have at these locations, the new prices and the new amounts for included traffic will apply later, starting on 1 February 2025, 01:00 am CET. The price for traffic overage will remain unchanged in the new price structure.
What are the new prices and amounts of included traffic?
Below, you can see a list of the old and new prices and the included traffic.
Product Old price New price Old included traffic New included traffic
CPX11 € 3.85 € 4.49 20 TB 1 TB
CPX21 € 7.05 € 8.99 20 TB 2 TB
CPX31 € 13.10 € 15.99 20 TB 3 TB
CPX41 € 24.70 € 29.99 20 TB 4 TB
CPX51 € 54.40 € 59.99 20 TB 5 TB
CCX13 € 11.99 € 12.99 20 TB 1 TB
CCX23 € 23.99 € 25.99 20 TB 2 TB
CCX33 € 47.99 € 49.99 30 TB 3 TB
CCX43 € 95.99 € 99.99 40 TB 4 TB
CCX53 € 191.99 € 199.99 50 TB 6 TB
CCX63 € 287.99 € 299.99 60 TB 8 TB
LB11 € 5.39 unchanged 20 TB 1 TB
LB21 € 16.40 unchanged 20 TB 2 TB
LB31 € 32.90 unchanged 20 TB 3 TB
All monthly prices are excl. VAT and excl. IPv4 addresses. Why are we making these changes?
With the new tariff structure, we want to make conditions for our customers around the world as fair as possible. To do that, we will calculate our prices based on local conditions in Europe, Singapore, and the USA. Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced. The new prices will give our customers the best possible price for the resources they use.
...