A day at Hetzner Online in the Falkenstein data center(igorslab.de)
igorslab.de
A day at Hetzner Online in the Falkenstein data center
https://www.igorslab.de/en/a-day-at-hetzner-online-in-the-falkenstein-data-center-insights-into-server-technology-cooling-production-rma-and-sustainability/
https://www.youtube.com/watch?v=gIjdKIMQh4s
89 comments
Der8auer did some similar videos a few years ago, but in English:
https://www.youtube.com/watch?v=5eo8nz_niiM
https://www.youtube.com/watch?v=V2P8mjWRqpk
The second is especially interesting as it demonstrates Hetzner's unique semi-custom hardware.
https://www.youtube.com/watch?v=5eo8nz_niiM
https://www.youtube.com/watch?v=V2P8mjWRqpk
The second is especially interesting as it demonstrates Hetzner's unique semi-custom hardware.
Location of Hetzner's Falkenstein data centre, for the curious:
https://www.openstreetmap.org/way/606528332#map=13/50.46220/...
https://www.openstreetmap.org/way/606528332#map=13/50.46220/...
[deleted]
I have a tiny Hetzner VPS (2 vCPUs, 2 Gb RAM) in their west us datacenter that costs me $5.59 a month. I get 1 TB a month free outgoing bandwidth and unlimited incoming bandwidth, plus additional outgoing bandwidth at a rate of $1.20 per TB. I host my personal project's git LFS server there, a file server, and a Caddy instance that proxies over Tailscale to a more powerful box in my apartment. It's a great homelab architecture and I couldn't be happier with it. Thanks Hetzner!
They still have VPSes with 2G of RAM? I'm checking the cloud price page* and you can get CX23 with 2 vCPU, 4G RAM and 20TB of traffic (seems to say that it's 1TB for US) for 3,49€/month (~4 USD).
You can save additional 0,50€ if you go with only IPv6.
Maybe it's location dependent, I can't get it to show me prices for US.
* https://www.hetzner.com/cloud/
You can save additional 0,50€ if you go with only IPv6.
Maybe it's location dependent, I can't get it to show me prices for US.
* https://www.hetzner.com/cloud/
Hi there, At the top of the website, you should be able to choose euros or dollars and the VAT rate. When people become customers and create a full customer account, we ask them to choose between euros or dollars, and then using the customer's location from their account, we automatically apply the correct VAT. So customers see the correct price on the interface where customers create their orders. But you may have to set it manually on the website. --Katie
Yep, just checked, 2 GB RAM. That CX23 sounds like a great deal, 20 TB of free outgoing is ridiculous. But I live in west us and I rarely hit my 1 TB free bandwidth cap anyway, so the added latency isn't worth it
I think they retired a bunch of machines one or two weeks ago. I was prompted to upgrade my machine at no extra cost, anyway.
[deleted]
Thanks for the shout-out! We're really glad you're happy with it :D --Katie
Had a server in that DC for a few years until recently (no knock on Hetzner they where excellent, just didn't need it any more).
Robot is a brutally functional tool but it does function (well) and had zero issues.
Robot is a brutally functional tool but it does function (well) and had zero issues.
Regarding Robot, I think that it's completely fine for what it is. I almost never interact with it, and instead just configure my server as I see fit over ssh. Hetzner's value proposition is extremely cheap no frills servers -- you're paying for the server, not the management interface. If you want management interfaces that do a lot of useful work, use a cloud.
Yup, which is why I said brutally functional and not broken.
I like brutally functional it just needs to get me to SSH and I'm good.
I like brutally functional it just needs to get me to SSH and I'm good.
Video didn't display for me even after turning off content blockers, here's the url: https://www.youtube.com/watch?v=gIjdKIMQh4s
works with Firefox + uBlock Origin + canvas blocker
I love Hetzner. That said their IPv6 support is poor. A server gets a /64 only, if you want a /56 (allowing 256 container networks) then you have to pay €15. As for virtual networks: they only support IPv4!
At least they're not as bad as Azure... :)
At least they're not as bad as Azure... :)
Does each container network of the 256 really need its own /64? Is there some constraint that doesn't let them work on a /72?
In practice this can be made to work but a networking expert can probably explain better than me why splitting a prefix into chunks smaller than a /64, and assigning them to virtual networks within a host is a bad idea.
In Hetzner's specific case: they won't give me one or more additional /72s: only a /56 if I pay for it. Per server.
In Hetzner's specific case: they won't give me one or more additional /72s: only a /56 if I pay for it. Per server.
splitting things out in a smaller prefix then a /64 breaks a couple of things.
SLAAC will not work, and slaac is actually a really neat usecase for containers.
Not having the overhead of DHCP for container addressing is neat.
Also, smaller blocks then /64 makes things like prefix delegation (usually) also break from a provider.
A container should absolutely not even need a /72. The traditional reason for /64 is for slaac but you most certainly don't need that for one container (if at all honestly).
Indeed, a host should be able to request a /64 via DHCPv6-PD and split that between millions of container networks. But you can't do that on Hetzner (or anywhere else).
Yeah that obviously only works on /56 and above because networks should be a minimum of /64. I use k3s and each host has a /64; cilium just gives each pod a /80 and the host does NDP and stuff. Works fine, no need to require dhcp6.
Why do you need ipv6 on your internal network? Is 10/8 really not enough/overlap? For 99.99% of people it's fine for the internal interfaces and if anything actually simplifies configuration.
The purpose of a network is to allow any two consenting parties to communicate. IPv4 cannot deliver that if either party has an RFC1918 address. NAT is a foul perversion of this foundational principle of the Internet Protocol.
For a lot of use cases a major advantage of IPv6 is to get away from ambiguous rfc1918 addressing.
You can then just put an allow rule between arbitrary v6 addresses anywhere on the internet when you need connectivity without any other hacks like proxies, NAT, etc and the associated complexity and addressing ambiguity/context dependence of rfc1918 addresses.
So fex you can just curl or ssh to your mycontainer.mydomain.net or you can put an allow rule from mycontainer.mydomain.net to a vm or laptop on your home network.
Internetworking, they call it.
You can then just put an allow rule between arbitrary v6 addresses anywhere on the internet when you need connectivity without any other hacks like proxies, NAT, etc and the associated complexity and addressing ambiguity/context dependence of rfc1918 addresses.
So fex you can just curl or ssh to your mycontainer.mydomain.net or you can put an allow rule from mycontainer.mydomain.net to a vm or laptop on your home network.
Internetworking, they call it.
I'm talking about an internal network, not the public connection.
The context in the GP comment was generally getting v6 connectivity for containers.
"Internal" is a context dependent term that you introduced. But to give a use case for that, for example you might want to have (maybe at a future date) two hosts on your networks on AWS and Hetzner talk to each other, still without allowing public connectivity.
"Internal" is a context dependent term that you introduced. But to give a use case for that, for example you might want to have (maybe at a future date) two hosts on your networks on AWS and Hetzner talk to each other, still without allowing public connectivity.
[deleted]
The benefit of IPv6 is that I don’t need an “internal” network. Everything is on the Internet and the firewall is the only thing that gets in the way.
I disable IPv6 and I’m somewhat scared of the concept of having containers with public IPs.
Routable != routed.
If your containers have a Global Unicast Address then it's possible to look at connetion logs and figure out which container made a particular request, for instance.
If your containers have a Global Unicast Address then it's possible to look at connetion logs and figure out which container made a particular request, for instance.
Yes, not addressable is even safer. Especially so for someone not specialized in networking.
It's not safer: it impedes observability.
It doesn't impeed observability for goodness sakes. It does however impeed accidentally opening up your internal network because you don't really understand your firewall/virtual router/whatever.
Of course it impedes observability. With IPv6, I can see the IP addresses of the containers that connect to a service. With IPv4, I get (at best) the IP address of the container host, thanks to NAT.
Are you also afraid of port forwarding? Have you considered that your ISP could choose to send your router packets destined for RFC1918 addresses?
Are you also afraid of port forwarding? Have you considered that your ISP could choose to send your router packets destined for RFC1918 addresses?
Yes I see the value there. I'm not sure I'm ready to take the risk, and most of the traffic is IPv4 anyway, but that's a good point.
Do tech companies like Hetzner count as mittelstands?
"Mittelstand" in Germany seems to be roughly: below 500 Employees, below 50 Million € turnover. Hetzner makes more like 400 Million € turnover but with 300 Employees. So it's technically above it but they are propably a very tight margin buisness. As such i'd say yes.
Mittelstand is singular (like middle class).
I would say decidedly yes for Hetzner. Mittelstand can somewhat be characterized by size, but it's mostly that they are more like a family, have different values and more long term thinking than the larger industry and public companies. It's also kind of a brand that many companies would like to attribute to themselves, even much larger companies that like that seal of quality.
I would say decidedly yes for Hetzner. Mittelstand can somewhat be characterized by size, but it's mostly that they are more like a family, have different values and more long term thinking than the larger industry and public companies. It's also kind of a brand that many companies would like to attribute to themselves, even much larger companies that like that seal of quality.
[deleted]
Hetzner certainly has this cult-like following mostly because of their low cost.
I assume it is a recent push toward these kind of open frame, super minimalist, consumer hardware based systems (I don't speak german and didn't translate the video).
It looks like they're using lots of consumer hardware and very little redundancy; you'll notice that the power supplies are generic ATX units and they're not doubled up. And then they're also running the onboard networking with a second connection which looks like it's for just a management system. Might not even be 10 gigabit networking.
It's interesting that in an era where almost all of the major players are moving toward cable-free arrangements i.e. backplanes with fully integrated power and networking, etc., they're instead opting for the rat's nest of cabling. It must have something to do with lower labor costs vs hardware costs. The amount of density that they are achieving with those systems is also incredibly low relatively speaking.
I assume it is a recent push toward these kind of open frame, super minimalist, consumer hardware based systems (I don't speak german and didn't translate the video).
It looks like they're using lots of consumer hardware and very little redundancy; you'll notice that the power supplies are generic ATX units and they're not doubled up. And then they're also running the onboard networking with a second connection which looks like it's for just a management system. Might not even be 10 gigabit networking.
It's interesting that in an era where almost all of the major players are moving toward cable-free arrangements i.e. backplanes with fully integrated power and networking, etc., they're instead opting for the rat's nest of cabling. It must have something to do with lower labor costs vs hardware costs. The amount of density that they are achieving with those systems is also incredibly low relatively speaking.
Same old world thinking.. Google use single PSUs too, real redundancy came from having multiple machines, and Hetzner certainly makes that cheap enough to accomplish on a budget. You can also pay for 10 Gbit as an option with Hetzner, and a bunch of other custom upgrades, but the further you move outside their sweet spot the more it's going to cost.
I think a lot of the following comes the old-fashioned nature. In an age of hyperscalers that want to sell you a private cloud and full stack certification to make it your job to use their software and want to charge you $500/mo for a server with 64GB of ram and decent bandwidth, Hetzner will just rent you a server, relatively cheap.
Amazon has _95 pages_ of EC2 instance types. They have so many products that I literally had to google the name to know what product type to put into the estimator to get a boring server.
What's all that data center best practice get you[1], the customer, if it doesn't provide lower prices and higher availability?
[1] -- I'm assuming you are not Netflix. After some scale all those crazy AWS services are pretty great to have.
Amazon has _95 pages_ of EC2 instance types. They have so many products that I literally had to google the name to know what product type to put into the estimator to get a boring server.
What's all that data center best practice get you[1], the customer, if it doesn't provide lower prices and higher availability?
[1] -- I'm assuming you are not Netflix. After some scale all those crazy AWS services are pretty great to have.
> I assume it is a recent push toward these kind of open frame, super minimalist, consumer hardware based systems
If by recent you mean that was how Hetzner has been since start (20+ years ago) then yes
If by recent you mean that was how Hetzner has been since start (20+ years ago) then yes
Again, I don't speak German, so I'm just watching the video; they have a mix of commodity 1u servers in there as well, so maybe that's their newer stuff?
Their technical support is also quite good, at least in my experience. Any questions I had they easily answered in a way that was easy to understand for me as a technical person. Not sure how it is for a non-technical person. But they might not rent Hetzner.
25 Gigabit networking is mentioned in the video. As for the low density, it might have something to do with their mentioned freecooling concept, which does not use cooling machines, but a controlled stream of air coming from outside.
This summer their Amsterdam DC went down because of the record breaking temperature in Amsterdam. Wondering if they use the same cooling technique there. They did mention it was because of cooling.
Hetzner does not have a datacenter in Amsterdam
https://docs.hetzner.com/cloud/general/locations/
https://docs.hetzner.com/cloud/general/locations/
Yep, this! Thanks for posting our locations. --Katie
Ups, my bad. Confused the incident, it was Scaleway.
They no longer use consumer grade off-the-shelf hardware. They use their own mainboards (yes, custom layouts due to their unique cooling solution). Checkout the video from Der8auer on youtube.
IlikeKitties(1)
John-Tony(2)
Hetzner is awesome.
I wish they would go public.
Would be very interesting to see how their business is doing compared to IONOS and OVH.
We need more public cloud companies in Europe.
I wish they would go public.
Would be very interesting to see how their business is doing compared to IONOS and OVH.
We need more public cloud companies in Europe.
> I wish they would go public.
I wish that they'll just keep on running a good profitable business without going public, or getting bought by Amazon, or otherwise shifting focus to providing shareholder value.
I wish that they'll just keep on running a good profitable business without going public, or getting bought by Amazon, or otherwise shifting focus to providing shareholder value.
This. I am a happy paying customer. Recommend them to my clients whenever it makes sense (more often than not). And hope they just keep running like they do as profitable business, like you said.
Why in the world would one want a company like this to go public? They are a very stable, established and profitable private entity. Based on what Hetzner is doing right now, it seems like the current way of operating that is intended by the leadership is closely aligned with what their customers want. This is often the first thing that goes out the window once a company becomes public
Why would you want a company you like to go public? To keep chasing profits to the detriment of everything else?
The comparison with OVH is a good one. For some reason OVH has much worse PUE, self-reporting 1.24 vs. Hetzner's 1.11. Operating costs are basically just electricity for these places, so their margins are that much worse.
For further comparison, Google at a similar latitude in Saint-Ghislain, Belgium, claims 1.08.
For further comparison, Google at a similar latitude in Saint-Ghislain, Belgium, claims 1.08.
I don't understand anything in your comment, so I'm shamelessly posting some info from an LLM:
> PUE (Power Usage Effectiveness) measures datacenter efficiency - it's total facility power divided by IT equipment power. A perfect 1.0 means all power goes to servers; higher numbers mean more waste on cooling/overhead. OVH's 1.24 vs Hetzner's 1.11 means OVH burns 24% extra power on non-IT stuff, hurting margins since electricity is their main cost. Google hits 1.08 at similar latitude.
> PUE (Power Usage Effectiveness) measures datacenter efficiency - it's total facility power divided by IT equipment power. A perfect 1.0 means all power goes to servers; higher numbers mean more waste on cooling/overhead. OVH's 1.24 vs Hetzner's 1.11 means OVH burns 24% extra power on non-IT stuff, hurting margins since electricity is their main cost. Google hits 1.08 at similar latitude.
> I wish they would go public.
If they go public they will be bought by foreign private equity.
If they go public they will be bought by foreign private equity.
Compared to IONOS, it is day and night. IONOS in my experience is just worse in every category that matters.
IONOS has buggy and bad wannabe SPA web frontend, broken OS images, bottlenecks in their provisioning API, that they don't care to fix, and 10x the costs.
If you are not desperate for one specific offering of IONOS, that Hetzner doesn't offer, and you have any other options, there is no good and justifiable reason to go for IONOS. IONOS is a low effort, low quality, high marketing spend shop. Hetzner is almost the opposite. They don't waste money on advertising, they cost much less to rent servers from, their OS images work, they got good technical support.
If you are not desperate for one specific offering of IONOS, that Hetzner doesn't offer, and you have any other options, there is no good and justifiable reason to go for IONOS. IONOS is a low effort, low quality, high marketing spend shop. Hetzner is almost the opposite. They don't waste money on advertising, they cost much less to rent servers from, their OS images work, they got good technical support.
The last thing we need is more enshitification on this space.
I just migrated my selfhosted email server to Hetzner and I don't want them turning into monstrosity like AWS or Azure, with theit miriad of ways to nickel and dime the customers.
I just migrated my selfhosted email server to Hetzner and I don't want them turning into monstrosity like AWS or Azure, with theit miriad of ways to nickel and dime the customers.
Damn Americans, can't think of anything other than "I wish I could make a buck off this, idgaf if it gets destroyed in the process".
He personally unlikely will make buck off this.
Going public usually means unlocking funding, so they could grow business, build more services, compete with say OVH or if some hyperscaler will decide to step into this niche.
Going public usually means unlocking funding, so they could grow business, build more services, compete with say OVH or if some hyperscaler will decide to step into this niche.
Yes I agree, we need more public companies in Europe.
Private companies are inherently less social since they don’t allow ordinary people to participate in growth.
In this sense, they’re selfish.
PS: yes I know that there are also downsides to public companies. But looking at the trade-offs I prefer that success can be shared as broadly as possible.
Private companies are inherently less social since they don’t allow ordinary people to participate in growth.
In this sense, they’re selfish.
PS: yes I know that there are also downsides to public companies. But looking at the trade-offs I prefer that success can be shared as broadly as possible.
At least in America, "a successful private company went public" often translates into "ordinary people got a bit of gold, selfish vulture capitalists butchered the goose, and there was precious little success or growth for anyone after that".
(Also - might your "allow ordinary people to participate" sympathies extend to people who would like to participate in your own financial affairs?)
(Also - might your "allow ordinary people to participate" sympathies extend to people who would like to participate in your own financial affairs?)
> "ordinary people got a bit of gold, selfish vulture capitalists butchered the goose
some ordinary people who worked hard and made it happened: founders and early workers, usually rewarded very well
some ordinary people who worked hard and made it happened: founders and early workers, usually rewarded very well
Yes, exactly!
Selfish VC becoming filthy rich through an IPO is exactly my point. Up to an IPO a private company will only make their owners rich - in your example "selfish vulture capitalists".
After an IPO anyone can participate. When Google, Amazon, Apple went public, VCs got rich. Everyone after that included every day people like you and me.
Selfish VC becoming filthy rich through an IPO is exactly my point. Up to an IPO a private company will only make their owners rich - in your example "selfish vulture capitalists".
After an IPO anyone can participate. When Google, Amazon, Apple went public, VCs got rich. Everyone after that included every day people like you and me.
No - the selfish vulture capitalists are the outsiders who purchase a private company which has been successful for many years, then butcher it. There is no IPO - it is "you own X, and we are offering you $Y million to sell it to us".
After that - X's best assets are sold off (the VC's get the money), X goes deeply into debt (again, the VC's get the money), many of the employees are laid off, and X generally goes bankrupt within 7 years - because what is left of it can't make the payments on the debt.
After that - X's best assets are sold off (the VC's get the money), X goes deeply into debt (again, the VC's get the money), many of the employees are laid off, and X generally goes bankrupt within 7 years - because what is left of it can't make the payments on the debt.
I don't understand your comment. You are both talking about public companies, and suddenly you are now talking about private equity?
(You're right - I made a mess of things, and inter-mingled the cases where a privately-owned company is sold directly to private equity / vulture capitalists, and the case where a privately-owned company "goes public" - but that still does not lead to a happy ending.)
>the selfish vulture capitalists are the outsiders who purchase a private company which has been successful for many years, then butcher it.
Sounds like the selfish vulture capitalists are the insiders who sell the company.
>X's best assets are sold off (the VC's get the money), X goes deeply into debt (again, the VC's get the money), many of the employees are laid off, and X generally goes bankrupt within 7 years - because what is left of it can't make the payments on the debt.
This doesn't make any sense, because X is the original asset. If part of X is sold, then the remaining portion of X loses value (assuming the sold part is the good part). If X is used as collateral, then it also loses value.
Sounds like the selfish vulture capitalists are the insiders who sell the company.
>X's best assets are sold off (the VC's get the money), X goes deeply into debt (again, the VC's get the money), many of the employees are laid off, and X generally goes bankrupt within 7 years - because what is left of it can't make the payments on the debt.
This doesn't make any sense, because X is the original asset. If part of X is sold, then the remaining portion of X loses value (assuming the sold part is the good part). If X is used as collateral, then it also loses value.
Privately-owned companies can provide better value to their customers. They can invest in customer trust and brand value that pays off over decades, not quarters.
They can also choose enshittification, but they are not pressured into it like companies with institutional investors are.
As a Hetzner customer, I would lose out if they went public.
They can also choose enshittification, but they are not pressured into it like companies with institutional investors are.
As a Hetzner customer, I would lose out if they went public.
> Private companies are inherently less social since they don’t allow ordinary people to participate in growth.
Consider two possibilities. The first is, if you want to make money in an industry, you start a company in it. Lots of people start companies because lots of people want to make money and then there are lots of companies, causing the profits to be widely distributed.
The second is, if you want to make money in an industry, you buy shares of an existing company. You have to buy them from whoever currently owns it, so the ones who got in early become billionaires, meanwhile even if an ordinary person were to invest their entire net worth they wouldn't even own 1% of the company so they have so little influence over it that it isn't even worth their time to vote their shares, and therefore have no influence over it at all. But you still make some profit while not having to actually do the work of building a company, so more people do that instead of entering the market themselves and then there are fewer companies that are each bigger.
The second one leads to market consolidation and concentration of wealth and power, so which one is actually less social?
Consider two possibilities. The first is, if you want to make money in an industry, you start a company in it. Lots of people start companies because lots of people want to make money and then there are lots of companies, causing the profits to be widely distributed.
The second is, if you want to make money in an industry, you buy shares of an existing company. You have to buy them from whoever currently owns it, so the ones who got in early become billionaires, meanwhile even if an ordinary person were to invest their entire net worth they wouldn't even own 1% of the company so they have so little influence over it that it isn't even worth their time to vote their shares, and therefore have no influence over it at all. But you still make some profit while not having to actually do the work of building a company, so more people do that instead of entering the market themselves and then there are fewer companies that are each bigger.
The second one leads to market consolidation and concentration of wealth and power, so which one is actually less social?
they offer by far the best value servers in the world. If that's selfish, so be it