The database servers powering Let's Encrypt (2021)(letsencrypt.org)
letsencrypt.org
The database servers powering Let's Encrypt (2021)
https://letsencrypt.org/2021/01/21/next-gen-database-servers.html
126 comments
It's because if you don't use that compute power, it's just an expensive loud heating system that's going to run up your electricity bill!
I have three R620s. They are shockingly quiet after being booted, even under pretty decent load. If you tune C-states for energy conservation in the BIOS, they idle at just under 100W each. With moderate load (like you'd have in a homelab), around 125W. According to iDRAC, the peak they've seen is about 350W.
For better apples-to-apples, each of them has 2x E5-2650v2, 3x SATA SSDs, 8x 16 GiB PC3-12800R sticks, and 1x NVMe SSD (mounted in a PCIe adapter).
As a comparison, I also have a Supermicro X11SSH-F with a bunch (9) of spinning disks. It has a single E3-1220 v5, 1x NVMe SSD, 2x 32 GiB PC4-17000U, and averages 125W. Given the relative power consumption of HDDs, that's a pretty impressive difference IMO.
My entire rack (the aforementioned servers, and some Unifi networking gear) pulls around 500-600W in total. When the backup server (Supermicro X9DRi-LN4F+; dual-socket something-or-other Xeons, some DDR3 ram, and 8x spinning drives) kicks on daily to ingest ZFS snapshots, the total consumption is around 700W, I think. Maybe 750W.
For better apples-to-apples, each of them has 2x E5-2650v2, 3x SATA SSDs, 8x 16 GiB PC3-12800R sticks, and 1x NVMe SSD (mounted in a PCIe adapter).
As a comparison, I also have a Supermicro X11SSH-F with a bunch (9) of spinning disks. It has a single E3-1220 v5, 1x NVMe SSD, 2x 32 GiB PC4-17000U, and averages 125W. Given the relative power consumption of HDDs, that's a pretty impressive difference IMO.
My entire rack (the aforementioned servers, and some Unifi networking gear) pulls around 500-600W in total. When the backup server (Supermicro X9DRi-LN4F+; dual-socket something-or-other Xeons, some DDR3 ram, and 8x spinning drives) kicks on daily to ingest ZFS snapshots, the total consumption is around 700W, I think. Maybe 750W.
> If you tune C-states for energy conservation in the BIOS, they idle at just under 100W each.
That's quite a lot for idle. For comparison, my tower idles at around 25W.
That's quite a lot for idle. For comparison, my tower idles at around 25W.
Tons of 25mm fans, dual socket, and a bunch of sticks of RAM add up.
I'm not trying to say they sip power, just that not every rack server is going to bankrupt you from power in most parts of CONUS, nor is a 1U guaranteed to be a screaming banshee. Assuming you had three of them, and nothing else, 300W over a month is still only $50/month with $0.25/kWh, which is about as high as I think it gets anywhere. The average is about half that. While not cheap, if you can afford a rack and three servers, you can probably afford the power.
If you're in the UK where it's somewhere around double that, then yeah maybe it's not really worthwhile.
It's also of course worth pointing out that for most employers, the useful skills you can get from a homelab can also be had from EC2s. Even if your job does run a datacenter, it's highly likely that they already have an infra team whose job it is to deal with the hardware.
I'm not trying to say they sip power, just that not every rack server is going to bankrupt you from power in most parts of CONUS, nor is a 1U guaranteed to be a screaming banshee. Assuming you had three of them, and nothing else, 300W over a month is still only $50/month with $0.25/kWh, which is about as high as I think it gets anywhere. The average is about half that. While not cheap, if you can afford a rack and three servers, you can probably afford the power.
If you're in the UK where it's somewhere around double that, then yeah maybe it's not really worthwhile.
It's also of course worth pointing out that for most employers, the useful skills you can get from a homelab can also be had from EC2s. Even if your job does run a datacenter, it's highly likely that they already have an infra team whose job it is to deal with the hardware.
What does one do with all this hardware? I'm quite curious.
Hentai collection and the data mining code analyzing the relevant parts.
Like most people on r/homelab, it started out with Plex. Rough timeline/services below:
0. Got a Synology DS413 with 4x WD Red 3TB drives. Use Playstation Media Server to stream videos from it. Eventually find some Busybox stuff to add various functionality to the NAS, but it had a habit of undoing them periodically, which was frustrating. I also experienced my first and (knock on wood) only drive failure during this time, which concluded without fanfare once the faulty drive was replaced, and the array repaired itself.
1. While teaching self Python as an Electrical Distribution Engineer at a utility, I befriended the IT head, who gave me an ancient (I think Nehalem? Quad-core Xeon) Dell T310. Promptly got more drives, totaling 7, and tried various OS / NAS platforms. I had OpenMediaVault for a while, but got tired of the UI fighting me when I knew how to do things in shell, so I switched to Debian (which it's based on anyway). Moved to MergerFS [0] + SnapRAID [1] for storage management, and Plex for media. I was also tinkering with various Linux stuff on it constantly.
1.1 Got tired of my tinkering breaking things and requiring troubleshooting/fixing (in retrospect, this provided excellent learning), so I installed Proxmox, reinstalled Debian, and made a golden image with everything set up as desired so I could easily revert.
1.2 A friend told me about Docker. I promptly moved Plex over to it, and probably around this time also got the *Arr Stack [2] going.
2. Got a Supermicro X9DRi-LN4F+ in a 2U chassis w/ 12x 3.5" bays. Got faster/bigger CPUs (E5-2680v2), more RAM, more drives, etc. Shifted container management to Docker Compose. Modded the BIOS to allow it to boot from a NVMe drive on a PCIe adapter.
2.1 Shifted to ZFS on Debian. Other than DKMS occasionally losing its mind during kernel upgrades, this worked well.
2.2 Forked [3] some [4] Packer/Ansible projects to suit my needs, made a VM for everything. NAS, Dev, Webserver, Docker host, etc. Other than outgrowing (IMO) MergerFS/SnapRAID, honestly at this point I could have easily stopped, and could to this day revert back to this setup. It was dead reliable and worked extremely well. IIRC I was also playing with Terraform at this time.
2.3 Successfully broke into tech (Associate SRE) as a mid-career shift, due largely (according to the hiring manager) to what I had done with my homelab. Hooray for hobbies paying off.
3. Got a single Dell R620. I think the idea was to install either pfSense or VyOS on it, but that never came to fruition. Networking was from a Unifi USG (their tiny router + firewall + switch) and 8-port switch, with some AC Pro APs.
4. Got two more R620s. Kubernetes all the things. Each one runs Proxmox in a 3-node cluster with two VMs - a control plane, and worker.
4.0.1 Perhaps worth noting here that I thoroughly tested my migration plan via spinning up some VMs in, IIRC, Digital Ocean that mimicked my home setup. I successfully ran it twice, which was good enough for me.
4.1 Played with Ceph via Rook, but a. disliked (and still to this day) running storage for everything out of K8s b. kept getting clock skew between nodes. Someone on Reddit mentioned it was my low-power C-state settings, but since that was saving me something like ~50 watts/node, I didn't want to deal with the higher power/heat. I landed on Longhorn [5] for cluster storage (i.e. anything that wasn't being handled by the ZFS pool), which was fine, but slow. SATA SSDs (used Intel enterprise drives with PLP, if you're wondering) over GBe aren't super fast, but they should be able to exceed 30 MBps.
4.1.1 Again, worth noting that I spent literally a week poring over every bit of Ceph documentation I could find, from the Red Hat stuff to random Wikis and blog posts. It's not something you just jump into, IMO, and most of the horror stories I read boiled down to "you didn't follow the recommended practices."
5. Got a newer Supermicro, an X11SSH-F, thinking that it would save power consumption over the older dual-socket I had for the NAS. It turned out to not make a big difference. For some reason I don't recall, I had a second X9DRi-LN4F+ mobo, so I sold the other one with the faster CPUs, bought some cheaper CPUs for the other one, and bought more drives for it. It's now a backup target that boots up daily to ingest ZFS snapshots. I have 100% on-site backups for everything. Important things (i.e. anything that I can't get from a torrent) are also off-site.
6. Got some Samsung PM863 NVMe SSDs mounted on PCIe adapters for the Dells, and set up Ceph, but this time handled by Proxmox. It's dead easy, and for whatever reason isn't troubled by the same clock skew issues as I had previously. Still in the process of shifting cluster storage from Longhorn, but I have been successfully using Ceph block storage as fast (1 GBe, anyway - a 10G switch is on the horizon) storage for databases.
So specifically, you asked what I do with the hardware. What I do, as far as my family is concerned, is block ads and serve media. On a more useful level, I try things out related to my job, most recently database-related (I moved from SRE to DBRE a year ago). I have MySQL and Postgres running, and am constantly playing with them. Can you actually do a live buffer pool resize in MySQL? (yes) Is XFS actually faster than ext4 for large DROP TABLE operations? (yes, but not by much) Is it faster to shut down a MySQL server and roll back to a previous ZFS snapshot than to rollback a big transaction? (often yes, although obviously a full shutdown has its own problems) Does Postgres suffer from the same write performance issue as MySQL with random PKs like UUIDv4, despite not clustering by default? (yes, but not to the same extent - still enough to matter, and you should use UUIDv7 if you absolutely need them)
I legitimately love this stuff. I could quite easily make do without a fancy enclosed rack and multiple servers, but I like them, so I have them. The fact that it tends to help my professional growth out at the same time is a bonus.
[0]: https://github.com/trapexit/mergerfs
[1]: https://www.snapraid.it
[2]: https://wiki.servarr.com
[3]: https://github.com/stephanGarland/packer-proxmox-templates
[4]: https://github.com/stephanGarland/ansible-initial-server
[5]: https://longhorn.io
0. Got a Synology DS413 with 4x WD Red 3TB drives. Use Playstation Media Server to stream videos from it. Eventually find some Busybox stuff to add various functionality to the NAS, but it had a habit of undoing them periodically, which was frustrating. I also experienced my first and (knock on wood) only drive failure during this time, which concluded without fanfare once the faulty drive was replaced, and the array repaired itself.
1. While teaching self Python as an Electrical Distribution Engineer at a utility, I befriended the IT head, who gave me an ancient (I think Nehalem? Quad-core Xeon) Dell T310. Promptly got more drives, totaling 7, and tried various OS / NAS platforms. I had OpenMediaVault for a while, but got tired of the UI fighting me when I knew how to do things in shell, so I switched to Debian (which it's based on anyway). Moved to MergerFS [0] + SnapRAID [1] for storage management, and Plex for media. I was also tinkering with various Linux stuff on it constantly.
1.1 Got tired of my tinkering breaking things and requiring troubleshooting/fixing (in retrospect, this provided excellent learning), so I installed Proxmox, reinstalled Debian, and made a golden image with everything set up as desired so I could easily revert.
1.2 A friend told me about Docker. I promptly moved Plex over to it, and probably around this time also got the *Arr Stack [2] going.
2. Got a Supermicro X9DRi-LN4F+ in a 2U chassis w/ 12x 3.5" bays. Got faster/bigger CPUs (E5-2680v2), more RAM, more drives, etc. Shifted container management to Docker Compose. Modded the BIOS to allow it to boot from a NVMe drive on a PCIe adapter.
2.1 Shifted to ZFS on Debian. Other than DKMS occasionally losing its mind during kernel upgrades, this worked well.
2.2 Forked [3] some [4] Packer/Ansible projects to suit my needs, made a VM for everything. NAS, Dev, Webserver, Docker host, etc. Other than outgrowing (IMO) MergerFS/SnapRAID, honestly at this point I could have easily stopped, and could to this day revert back to this setup. It was dead reliable and worked extremely well. IIRC I was also playing with Terraform at this time.
2.3 Successfully broke into tech (Associate SRE) as a mid-career shift, due largely (according to the hiring manager) to what I had done with my homelab. Hooray for hobbies paying off.
3. Got a single Dell R620. I think the idea was to install either pfSense or VyOS on it, but that never came to fruition. Networking was from a Unifi USG (their tiny router + firewall + switch) and 8-port switch, with some AC Pro APs.
4. Got two more R620s. Kubernetes all the things. Each one runs Proxmox in a 3-node cluster with two VMs - a control plane, and worker.
4.0.1 Perhaps worth noting here that I thoroughly tested my migration plan via spinning up some VMs in, IIRC, Digital Ocean that mimicked my home setup. I successfully ran it twice, which was good enough for me.
4.1 Played with Ceph via Rook, but a. disliked (and still to this day) running storage for everything out of K8s b. kept getting clock skew between nodes. Someone on Reddit mentioned it was my low-power C-state settings, but since that was saving me something like ~50 watts/node, I didn't want to deal with the higher power/heat. I landed on Longhorn [5] for cluster storage (i.e. anything that wasn't being handled by the ZFS pool), which was fine, but slow. SATA SSDs (used Intel enterprise drives with PLP, if you're wondering) over GBe aren't super fast, but they should be able to exceed 30 MBps.
4.1.1 Again, worth noting that I spent literally a week poring over every bit of Ceph documentation I could find, from the Red Hat stuff to random Wikis and blog posts. It's not something you just jump into, IMO, and most of the horror stories I read boiled down to "you didn't follow the recommended practices."
5. Got a newer Supermicro, an X11SSH-F, thinking that it would save power consumption over the older dual-socket I had for the NAS. It turned out to not make a big difference. For some reason I don't recall, I had a second X9DRi-LN4F+ mobo, so I sold the other one with the faster CPUs, bought some cheaper CPUs for the other one, and bought more drives for it. It's now a backup target that boots up daily to ingest ZFS snapshots. I have 100% on-site backups for everything. Important things (i.e. anything that I can't get from a torrent) are also off-site.
6. Got some Samsung PM863 NVMe SSDs mounted on PCIe adapters for the Dells, and set up Ceph, but this time handled by Proxmox. It's dead easy, and for whatever reason isn't troubled by the same clock skew issues as I had previously. Still in the process of shifting cluster storage from Longhorn, but I have been successfully using Ceph block storage as fast (1 GBe, anyway - a 10G switch is on the horizon) storage for databases.
So specifically, you asked what I do with the hardware. What I do, as far as my family is concerned, is block ads and serve media. On a more useful level, I try things out related to my job, most recently database-related (I moved from SRE to DBRE a year ago). I have MySQL and Postgres running, and am constantly playing with them. Can you actually do a live buffer pool resize in MySQL? (yes) Is XFS actually faster than ext4 for large DROP TABLE operations? (yes, but not by much) Is it faster to shut down a MySQL server and roll back to a previous ZFS snapshot than to rollback a big transaction? (often yes, although obviously a full shutdown has its own problems) Does Postgres suffer from the same write performance issue as MySQL with random PKs like UUIDv4, despite not clustering by default? (yes, but not to the same extent - still enough to matter, and you should use UUIDv7 if you absolutely need them)
I legitimately love this stuff. I could quite easily make do without a fancy enclosed rack and multiple servers, but I like them, so I have them. The fact that it tends to help my professional growth out at the same time is a bonus.
[0]: https://github.com/trapexit/mergerfs
[1]: https://www.snapraid.it
[2]: https://wiki.servarr.com
[3]: https://github.com/stephanGarland/packer-proxmox-templates
[4]: https://github.com/stephanGarland/ansible-initial-server
[5]: https://longhorn.io
> E5-2650v2
Those are pretty awful by current standards - you could buy a midrange minipc or laptop from a few cpu generations ago and get similar/better compute performance for much lower power draw.
If you're just running a homelab and you don't have poor quality wall power or something you aren't really gaining anything from ECC ram either.
Although you said you were thinking of upgrading, so you're probably aware of this already.
Those are pretty awful by current standards - you could buy a midrange minipc or laptop from a few cpu generations ago and get similar/better compute performance for much lower power draw.
If you're just running a homelab and you don't have poor quality wall power or something you aren't really gaining anything from ECC ram either.
Although you said you were thinking of upgrading, so you're probably aware of this already.
>If you're just running a homelab and you don't have poor quality wall power or something you aren't really gaining anything from ECC ram either.
It has nothing to do with the quality or reliability of the mains supply; ECC increases reliability via protecting data integrity from bitflips caused by cosmic rays etc.
It has nothing to do with the quality or reliability of the mains supply; ECC increases reliability via protecting data integrity from bitflips caused by cosmic rays etc.
Hm, I keep my server in the basement, I wonder if 6" of concrete makes any difference to cosmic rays?
> Those are pretty awful by current standards
Oh, fully agree. My base spec M1 Air destroys them in anything single-threaded. I got them because I wanted three nodes to play around with various things requiring HA, and I didn't want to just virtualize that. That said, they _are_ virtualized. Every server I have runs Proxmox, with templated VMs built from Packer + Ansible. I'm currently in the process of shifting K8s providers; right now they're running k3os but since Rancher has killed that, it's forever stuck at v1.21, so I'm shifting to Talos. I have the Terraform for them working fine, just need to set up a PXE server so they'll pick up the desired static IP when being created.
> you aren't really gaining anything from ECC ram either
Sibling comment to mine addressed this; to add on, I disagree. Cosmic rays aside, there's a ton of EMI inside a computer, and anything I can do to help prevent data corruption is a win in my book. Torvalds agrees, if that carries any weight with you. [0]
As far as cost, it costs me ~$30-40/month to run in electricity, and some small amount of A/C load (roughly equivalent to 5 extra people existing in the house) [1].
[0]: https://www.realworldtech.com/forum/?threadid=198497&curpost...
[1]: https://www.engineeringtoolbox.com/metabolic-heat-persons-d_...
Oh, fully agree. My base spec M1 Air destroys them in anything single-threaded. I got them because I wanted three nodes to play around with various things requiring HA, and I didn't want to just virtualize that. That said, they _are_ virtualized. Every server I have runs Proxmox, with templated VMs built from Packer + Ansible. I'm currently in the process of shifting K8s providers; right now they're running k3os but since Rancher has killed that, it's forever stuck at v1.21, so I'm shifting to Talos. I have the Terraform for them working fine, just need to set up a PXE server so they'll pick up the desired static IP when being created.
> you aren't really gaining anything from ECC ram either
Sibling comment to mine addressed this; to add on, I disagree. Cosmic rays aside, there's a ton of EMI inside a computer, and anything I can do to help prevent data corruption is a win in my book. Torvalds agrees, if that carries any weight with you. [0]
As far as cost, it costs me ~$30-40/month to run in electricity, and some small amount of A/C load (roughly equivalent to 5 extra people existing in the house) [1].
[0]: https://www.realworldtech.com/forum/?threadid=198497&curpost...
[1]: https://www.engineeringtoolbox.com/metabolic-heat-persons-d_...
A friend of mine built a k8s system with Raspberry Pis, back when they were cheap and available, which burned very little electricity and produced no detectable heat.
Yep! I have an RPi 4, and at one point it was running Hypriot [0] on it, with some lighter containers being handled by it instead of the larger x86 nodes. I'm pretty sure I also had it running as part of a K8s cluster for a brief time.
[0]: https://blog.hypriot.com
[0]: https://blog.hypriot.com
The absolute maximum I've ever seen it hit is 400W. At my electricity rate, the theoretical maximum is something like $15/month. I'm perfectly fine with that, it's well within my hobby budget.
Really, that's what it is to me, a hobby. It doesn't have to be useful or profitable, just an interesting way to spend my free time tinkering
Really, that's what it is to me, a hobby. It doesn't have to be useful or profitable, just an interesting way to spend my free time tinkering
They were issuing 2M certs/day when this happened in 2021, now up to 3.4M/day.[2]
Looks like it scaled pretty well so far.
[2] https://letsencrypt.org/stats/
Looks like it scaled pretty well so far.
[2] https://letsencrypt.org/stats/
Doesn't mention if they're running Linux/what flavor they're running. I personally would be wary of running OpenZFS in production on Linux, especially ZFS on root. It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable.
Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF? That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use?
Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF? That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use?
Given that they're using a HSM (actually several), there's really not much that needs protection via FDE. The certs are obviously public and the domains are in the transparency log.
On the ZFS note: it's been rock solid for me with Ubuntu but a living nightmare with Arch. My Arch update would upgrade the kernel but then OpenZFS would semi-routinely incompatible resulting in an unbootable system.
On the ZFS note: it's been rock solid for me with Ubuntu but a living nightmare with Arch. My Arch update would upgrade the kernel but then OpenZFS would semi-routinely incompatible resulting in an unbootable system.
I had the same issue. I went from Ubuntu -> Arch -> NixOS looking for a distro with well-supported ZFS. Finally found one (the last one).
This is the magic line from my declarative configuration that ensures I never get a kernel update that is incompatible with my ZFS-on-root config:
kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
Been running it for 2 years now; quite happy with it, and this is probably my "final distro hop". Once you go declarative (and climb the learning curve), you are done with everything else I guess.
Put it this way, I can switch to an entirely different window manager by changing 1 configuration line. And it works every time. I've tried that on other Linux distros, and it always borks something or other. I can also change my GRUB theme and boot animation via Plymouth, something I would have NEVER risked on ANY other Linux distro due to the risk of modifying boot configuration... but since it's declarative (and validated) on NixOS, I've had no issues (just tweaking). If I manage to bork something, which is rare, I just reboot into a previous update of my OS and fix it and then try again.
This is the magic line from my declarative configuration that ensures I never get a kernel update that is incompatible with my ZFS-on-root config:
kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
Been running it for 2 years now; quite happy with it, and this is probably my "final distro hop". Once you go declarative (and climb the learning curve), you are done with everything else I guess.
Put it this way, I can switch to an entirely different window manager by changing 1 configuration line. And it works every time. I've tried that on other Linux distros, and it always borks something or other. I can also change my GRUB theme and boot animation via Plymouth, something I would have NEVER risked on ANY other Linux distro due to the risk of modifying boot configuration... but since it's declarative (and validated) on NixOS, I've had no issues (just tweaking). If I manage to bork something, which is rare, I just reboot into a previous update of my OS and fix it and then try again.
+1 to this, ZFS with NixOS is quite manageable.
One thing I'm not a fan of though is the guide on the NixOS wiki was removed and instead points to this guide in the OpenZFS docs: https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...
It does not have much in the way of rationale - I.e. they say you need minimum 1GB reserve space but don't say what reserve space is and why you need it.
I recall the NixOS wiki used to explain options and trade offs in a lot more detail.
Nice tip re: latestCompatibleLinuxPackage! Had not seen that anywhere yet (probably because I haven't had any issues yet).
One thing I'm not a fan of though is the guide on the NixOS wiki was removed and instead points to this guide in the OpenZFS docs: https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...
It does not have much in the way of rationale - I.e. they say you need minimum 1GB reserve space but don't say what reserve space is and why you need it.
I recall the NixOS wiki used to explain options and trade offs in a lot more detail.
Nice tip re: latestCompatibleLinuxPackage! Had not seen that anywhere yet (probably because I haven't had any issues yet).
Consider giving FreeBSD a shot, ZFS support is excellent
Yeah, FWIW my first production ZFS deploy was on FreeBSD over 10 years ago. It was rock solid (though low-demand).
I started with Solaris, moved to OpenIndiana, then to FreeBSD (the last over ten years ago). FreeBSD is hands-down the best. (I have a few Debian servers for other purposes but contemplate moving them to FreeBSD as well.)
I have experienced too many other benefits to NixOS (and learned enough Nix to get by along the way) to turn back now.
It now seems like the only way to do Linux sanely to me.
It’s perfect for tinkerers like me due to having more supported packages than any other distro, plus instant rollbacks.
It would be like asking a functional programmer to go back to C. “But… I’d lose all my guarantees, and spend too much time troubleshooting crap that just would never happen in [functional language/NixOS] again… No thanks”
To some things, there’s simply no turning back… and Nix is (slowly but steadily) gaining marketshare and mindshare for this reason.
If the BSD’s want future relevance, they better steal some ideas from Nix quick, and get cracking with declarative immutability and deterministic builds
It now seems like the only way to do Linux sanely to me.
It’s perfect for tinkerers like me due to having more supported packages than any other distro, plus instant rollbacks.
It would be like asking a functional programmer to go back to C. “But… I’d lose all my guarantees, and spend too much time troubleshooting crap that just would never happen in [functional language/NixOS] again… No thanks”
To some things, there’s simply no turning back… and Nix is (slowly but steadily) gaining marketshare and mindshare for this reason.
If the BSD’s want future relevance, they better steal some ideas from Nix quick, and get cracking with declarative immutability and deterministic builds
Those things are cool... but I don't need them. I suspect many of us don't.
I like BSD's simplicity, consistency, and its native integration with ZFS. I don't know about deterministic builds but I suspect it does that without making a big deal, since everything can arrive via ports.
Linux, as much as I love it, is a rat's nest of different ideas and the inconsistency between distributions makes me want to pull my hair out. I care less about how it does things, and more that I only have to learn one system to understand each component -- as opposed to the mess that is init.d vs systemd vs upstart or whatever.
man pages are good but the FreeBSD Handbook is better.
I like BSD's simplicity, consistency, and its native integration with ZFS. I don't know about deterministic builds but I suspect it does that without making a big deal, since everything can arrive via ports.
Linux, as much as I love it, is a rat's nest of different ideas and the inconsistency between distributions makes me want to pull my hair out. I care less about how it does things, and more that I only have to learn one system to understand each component -- as opposed to the mess that is init.d vs systemd vs upstart or whatever.
man pages are good but the FreeBSD Handbook is better.
> I suspect many of us don’t
One thing I’ve learned after crossing the 50 year mark is that time is actually the most valuable thing we have (and, most egalitarianly, everyone starts out with more or less the same amount of it left). So while troubleshooting can be enjoyable to many, time spent troubleshooting any issue that would simply be impossible to encounter in an alternative system is expensive indeed.
Consider reading this very understandable paper, which I consider at least as important as the Bitcoin paper: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf
One thing I’ve learned after crossing the 50 year mark is that time is actually the most valuable thing we have (and, most egalitarianly, everyone starts out with more or less the same amount of it left). So while troubleshooting can be enjoyable to many, time spent troubleshooting any issue that would simply be impossible to encounter in an alternative system is expensive indeed.
Consider reading this very understandable paper, which I consider at least as important as the Bitcoin paper: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf
Underrated fact: The Nix repo has the largest number of supported packages of any Linux distro (my guess is that since everything is deterministically built "from the metal" more or less, that this results in far fewer needs for support and troubleshooting)
But everything else is not. There are so few packages released for *BSDs it's a constant game of searching, compiling dependencies, etc.
Not sure what apps you are running but everything I have wanted to install on it either compiled from source, compiled from ports, or has a package, without too much hassle. (Worst thing was having to type 'cmake' instead of 'make' for building llama.cpp)
BSD has its issues, for sure, but on the question of ZFS support it is stellar.
BSD has its issues, for sure, but on the question of ZFS support it is stellar.
ZFS has been very reliable for me since 2006 or so when I first started using it on SPARC hardware with the Solaris 10 beta; I assume that since they have a backup server and a primary server, they don't update and reboot them both at the same time.
You can always boot off ext4 and then just run data off OpenZFS pools. The benefits of booting off ZFS are extremely minimal compared to having your working data on ZFS.
Usually you aren't updating production servers unless it's a security patch, fixes a problem you have, or adds a feature you want/need. Even then, usually you have a test environment to verify the upgrade won't bork the system
Disagree, you have to continuously update production servers (not daily, but ~weekly/monthly). The more often you do it, the more automated and risk-free it is, and the smaller the version gap for when that critical security vulnerability hits that needs to be patched ASAP.
ZFS on Linux is commonly used in HPC these days. E.g. https://computing.llnl.gov/projects/openzfs
[deleted]
> I personally would be wary of running OpenZFS in production on Linux
A ton of people in the enterprise have been doing this for years without issue; https://openzfs.org/wiki/Companies
> especially ZFS on root
I've been running ZFS on root on NixOS since this excellent guide (https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...) for about 2 years. Zero issues. (Actually, I see they've updated it, I need to look at that. Also, they default to encrypted root. I turn it off, because the slight hit to performance and extra risk of recovery impossibility is not worth it to me.)
> It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable
Well, I think you've found your problem, then. (That also might be why FreeNAS, which I also have running as my NAS, switched to Linux from Debian when they re-branded as TrueNAS Core/Enterprise.) Come over to NixOS, where you can simply reboot into any of N previous instances after an update that borks something (which almost never happens, anyway, because you can actually specify, in your machine configuration, "use the latest kernel that is compatible with ZFS"). No USB boot key needed. Here's the magic line from my own declarative configuration:
> Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF?
You're assuming they're not doing it, without evidence. Also, if they're already managing the security around their certs and cert generation properly, they might not need FDE. FDE is overrated IMHO, frankly, and also incurs a performance cost, as well as an extra risk cost (try recovering an encrypted drive to know what I mean). In short, religions are bad, even in technological choices; there is no single technological configuration choice that is 100% better than all possible alternative configurations.
> That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use?
Multiple layers of encryption incur an extra performance cost with almost no gain in extra security.
A ton of people in the enterprise have been doing this for years without issue; https://openzfs.org/wiki/Companies
> especially ZFS on root
I've been running ZFS on root on NixOS since this excellent guide (https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...) for about 2 years. Zero issues. (Actually, I see they've updated it, I need to look at that. Also, they default to encrypted root. I turn it off, because the slight hit to performance and extra risk of recovery impossibility is not worth it to me.)
> It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable
Well, I think you've found your problem, then. (That also might be why FreeNAS, which I also have running as my NAS, switched to Linux from Debian when they re-branded as TrueNAS Core/Enterprise.) Come over to NixOS, where you can simply reboot into any of N previous instances after an update that borks something (which almost never happens, anyway, because you can actually specify, in your machine configuration, "use the latest kernel that is compatible with ZFS"). No USB boot key needed. Here's the magic line from my own declarative configuration:
kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
> Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF?
You're assuming they're not doing it, without evidence. Also, if they're already managing the security around their certs and cert generation properly, they might not need FDE. FDE is overrated IMHO, frankly, and also incurs a performance cost, as well as an extra risk cost (try recovering an encrypted drive to know what I mean). In short, religions are bad, even in technological choices; there is no single technological configuration choice that is 100% better than all possible alternative configurations.
> That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use?
Multiple layers of encryption incur an extra performance cost with almost no gain in extra security.
[deleted]
[deleted]
Shows how much bang for buck you can get when hosting your own hardware.
Discussed at the time:
The database servers powering Let's Encrypt - https://news.ycombinator.com/item?id=25861422 - Jan 2021 (226 comments)
The database servers powering Let's Encrypt - https://news.ycombinator.com/item?id=25861422 - Jan 2021 (226 comments)
These performance improvements are absolutely insane, and churning out certs at speed is the _perfect_ use case for these CPUs.
Kudos the LE team!
Kudos the LE team!
Indeed. If I had to bet, I would say doubling the memory allowed some/all indexes to fit in RAM. RAM is usually a big improvement when it comes to Relational Database Management Systems (RMDBS).
> We increase target & max IOPS above the defaults. We still use conservative values to avoid excessive SSD wear, but the defaults were tuned for spinning disks: innodb_io_capacity=1000, innodb_io_capacity_max=2500.
I'd be interested to see if they actually needed this. Those parameters affect the baseline and burst IOPS InnoDB is allowed to use for background tasks like flushing logs, respectively, and generally you don't need to raise them. They default to 200 and 2000; those defaults been perfectly adequate for me on a MySQL 8.x instance serving 120K+ QPS.
I'd be interested to see if they actually needed this. Those parameters affect the baseline and burst IOPS InnoDB is allowed to use for background tasks like flushing logs, respectively, and generally you don't need to raise them. They default to 200 and 2000; those defaults been perfectly adequate for me on a MySQL 8.x instance serving 120K+ QPS.
I recently bought my first server, just about the cheapest Dell had available (still €2000). I was completely undewhelmed by the specs, it even has spinning rust. The upgrade costs to SSD's were like 250 per disk. So reading these specs, how much does such a server cost? Or do you put the disks in yourself? Can you negotiate a 50% discount? So many questions I have as a newbie in the entireprise server world.
If you are an enterprise (or just big enough) the prices on Dell’s site are meaningless, they are just conversation starters if that.
From old knowledge, you’re out by about 2 orders of magnitude.
Where possible I go third party on disks and ram.
No one buys dell servers at list price.
Is there an update to this article?
Since the article was written: - AMD has gen4 available - Genoa - there's NVMe drives 15tb capacity each
Since the article was written: - AMD has gen4 available - Genoa - there's NVMe drives 15tb capacity each
The same hardware is still in use today, and I expect will continue to be used for some time.
The biggest thing that’s happened in the meantime is reducing load on the databases by pushing some of the OCSP load out to redis caches: https://letsencrypt.org/2022/12/15/ocspcaching.html By not storing OCSP in the MariaDBs, we reduced write volume and storage size significantly.
The next big thing is going to be sharding the database so multiple servers can share the write loads, and some queries moved out to redis caches.
(I work for Let’s Encrypt)
The biggest thing that’s happened in the meantime is reducing load on the databases by pushing some of the OCSP load out to redis caches: https://letsencrypt.org/2022/12/15/ocspcaching.html By not storing OCSP in the MariaDBs, we reduced write volume and storage size significantly.
The next big thing is going to be sharding the database so multiple servers can share the write loads, and some queries moved out to redis caches.
(I work for Let’s Encrypt)
Why add caching and sharding? Is the current architecture not going to keep up with demand?
Growth, plus reliability by reducing the SPOF.
We are beginning to hit the limits of what we can do with the current single-writer database in this hardware.
As well, the size of the data is larger than I’d like for a single MariaDB cluster. That makes operations like setting up new replicas take a long time. Being able to operate on smaller shards will help a lot at parallelizing some of that work.
We are beginning to hit the limits of what we can do with the current single-writer database in this hardware.
As well, the size of the data is larger than I’d like for a single MariaDB cluster. That makes operations like setting up new replicas take a long time. Being able to operate on smaller shards will help a lot at parallelizing some of that work.
I would guess they try to get 3-5 years out of a new server, and this was done in 2021...so still a bit early for a refresh. The servers prior to this one had E5-2650 processors, so they would have been bought sometime between 2013-2016. Meaning their last refresh cycle was at least 5 years.
Those 2021 servers will probably last them 4-5 years so they haven't upgraded yet.
And, it looks like with PCIe 5.0 x4 performance exceeds 2,500,000 IOPS reads, and 14 GBps sequential read rate.
I bet if one could develop an index of useful computation per kwh (or $) this project would be near the top.
Imagine if that kind of hyperleveraged impact was the norm rather than the exception.
Imagine if that kind of hyperleveraged impact was the norm rather than the exception.
It's kind of hard for it to be. If the only software projects that existed were the elite hyperefficient ones, the ten guys who ran them could just meet up every year for a key signing party - no need for a scalable service to automatically issue them security certificates.
Computers being commoditized means the median project is very low-value, but it also means there are millions of cool projects out there.
Computers being commoditized means the median project is very low-value, but it also means there are millions of cool projects out there.
What? No k8, no microservices, no cloud ? What is the world coming to?
Always nice to get an upgrade.
> We currently use MariaDB, with the InnoDB database engine.
Oh hey, it's not often that you hear about MySQL/MariaDB on HN, so this is a nice change. For what it's worth, it's a pretty decent database for getting things done, even if not as advanced as PostgreSQL in some respects.
Oh hey, it's not often that you hear about MySQL/MariaDB on HN, so this is a nice change. For what it's worth, it's a pretty decent database for getting things done, even if not as advanced as PostgreSQL in some respects.
I think it is also important to acknowledge that there are things innodb does better than Postgres, eg. For most workloads an undo log is a far better data structure than implementing MVCC by duplicating rows. Autovacuum and vacuum can be an absolute nightmare, plus the extra disk traffic the duplication generates. Maybe one day OrioleDb will bring this to postgres too.
I have had a MariaDB + Galera multi-master 3-node cluster running for several years, and prior to that Percona+Galera for ~5 years, and it's been just great. It stores lookup tables for our postfix mail server, fairly small databases and low hits, but it's great to just be able to reboot nodes for updates or migration to other VMs without having to do any of the old clustering gyrations.
I almost switched to CockroachDB in the last refresh, until I found that Postfix required Latin-1 encoding or something, and CockroachDB only supported UTF8. Postfix has more recently gotten a config option for changing that.
I almost switched to CockroachDB in the last refresh, until I found that Postfix required Latin-1 encoding or something, and CockroachDB only supported UTF8. Postfix has more recently gotten a config option for changing that.
[deleted]
Still an order of magnitude slower / less efficient than OpenLDAP / LMDB.
The same argument came up in a big company, incidentally for the same use case, certificate store in a CA. This led to benchmarks, where OpenLDAP was significantly slower, like three orders of magnitudes. Databases have gotten really faster in the last couple of years while OpenLDAP has stagnated.
InnoDB is a bloated pig. No, it hasn't gotten 5 orders of magnitude faster in the past 10 years. http://www.lmdb.tech/bench/memcache/
And the overhead of SQL processing vs LDAP protocol hasn't improved any either.
You're spouting lies.
And the overhead of SQL processing vs LDAP protocol hasn't improved any either.
You're spouting lies.
Got a link to a report on that?
OpenLDAP serves queries at line rate on multi-gigabit NICs, with latencies indistinguishable from ping RTTs. Other databases aren't even close.
OpenLDAP serves queries at line rate on multi-gigabit NICs, with latencies indistinguishable from ping RTTs. Other databases aren't even close.
Would their needs be met by a simple key-value store?
Refreshing to see big projects running bare metal rather than overpaying for AWS. Especially one relying on donations and sponsors.
[deleted]
Someone reading the headline of this might think "oh, they're using Cockroach, or Fauna, or Planetscale" -- nope, this is about next-gen hardware powering their single-write (with a number of read replicas) MariaDB instance.
I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month, if not every week...
I did some napkin math (could be very, very wrong) but this server costs ~ 225,000 USD according to Dell's webpage.
AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad.
However, storage costs alone would be astronomic. Like > 100,000 USD / month.
I have no idea how much outbound traffic Let's Encrypt serves, but that also could be a quite relevant expense.
OFC I also don't know how much Let's Encrypt pays for energy, cooling, operations, real estate, etc... but:
> I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month
I would not take the other side on that bet
AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad.
However, storage costs alone would be astronomic. Like > 100,000 USD / month.
I have no idea how much outbound traffic Let's Encrypt serves, but that also could be a quite relevant expense.
OFC I also don't know how much Let's Encrypt pays for energy, cooling, operations, real estate, etc... but:
> I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month
I would not take the other side on that bet
$225K for this sounds bonkers:
24x 6.4TB Intel P4610 NVMe SSD = 24 x $310 = $7440
2x AMD EPYC 7542 = 2 x $1300 = $2600
2 TB DDR4 ECC RAM ~ $13700 (estimate from a couple of Google results)
We recently purchased a similar server.
One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website.
Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free.
Even on Newegg, SAS SSD of that size are $900-2000, so add warranty and service on top of that.
One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website.
Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free.
Even on Newegg, SAS SSD of that size are $900-2000, so add warranty and service on top of that.
> One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website.
Makes sense.
> Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free.
I was able to find these two enterprise-grade NVMe SSDs on Newegg:
Makes sense.
> Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free.
I was able to find these two enterprise-grade NVMe SSDs on Newegg:
https://www.newegg.com/p/2U3-0005-000J8 ($474, 7.68 TB, 1 DWPD)
https://www.newegg.com/p/2U3-000S-000Y8 ($360, 6.4 TB, 3 DWPD)
I am not too much of an expert on enterprise hardware, but those are PCIe interface. I don’t know how possible it is to rack up 24 of those in a single server (you would run out of lanes).
This is something more similar to what is in those Dell servers (and there are 24 of them):
https://www.newegg.com/samsung-pm1643a-7-68tb/p/2U3-0005-000...
https://www.newegg.com/p/0N7-0133-00003
There is certainly a markup with Dell, but it’s sort of like a cloud vendor - pay for the warranty and service, and be (somewhat) hands off if something breaks.
This is something more similar to what is in those Dell servers (and there are 24 of them):
https://www.newegg.com/samsung-pm1643a-7-68tb/p/2U3-0005-000...
https://www.newegg.com/p/0N7-0133-00003
There is certainly a markup with Dell, but it’s sort of like a cloud vendor - pay for the warranty and service, and be (somewhat) hands off if something breaks.
Threadripper and Epyc has been smashing the pcie lane limit for a while now. That's why Epyc is kicking intel's ass in server applications.
My personal workstation at the side of my desk has 6 pcie nvme ssd's and I can add 4 more without breaking the lane bank.
My personal workstation at the side of my desk has 6 pcie nvme ssd's and I can add 4 more without breaking the lane bank.
> I don’t know how possible it is to rack up 24 of those in a single server
that's the one of points of article: Epic cpus have 128 lanes for a while, and that's how they upgraded to 24 NVMEs.
that's the one of points of article: Epic cpus have 128 lanes for a while, and that's how they upgraded to 24 NVMEs.
Oh yeah, the article. I guess this thread got sidetracked on the topic of Dell's pricing :). I wonder how common a 24-drive NVMe server is.
I don't know all the ins and outs of SAS vs NVMe. Maybe someone else can chime in. I am at the end of my knowledge now.
I suppose one benefit is the availability of hardware RAID controllers, as hinted in the article. But it does seem interesting that NVMe is cheaper than SAS, while theoretically having higher bandwidth.
I don't know all the ins and outs of SAS vs NVMe. Maybe someone else can chime in. I am at the end of my knowledge now.
I suppose one benefit is the availability of hardware RAID controllers, as hinted in the article. But it does seem interesting that NVMe is cheaper than SAS, while theoretically having higher bandwidth.
Yeah, I feel SAS is obsolete tech, and will be replaced by NVMe everywhere going forward.
> > One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website.
> Makes sense.
Does it? Adding a middleman that needs to take a cut to exist reducing the price makes sense to you?
> Makes sense.
Does it? Adding a middleman that needs to take a cut to exist reducing the price makes sense to you?
For a critical system, you should really have two, for HA purposes. Do they?
Yes. Let’s Encrypt has two locations, each of which has fully redundant hardware, so that’s a minimum of 4. We actually have a few more.
(I work at Let’s Encrypt)
(I work at Let’s Encrypt)
The hardware may have cost more in 2021 when this article was written.
This did not cost us $225k. About half that. Nobody pays the website price, you pay a lot less via a VAR.
- ED of ISRG / Let's Encrypt
- ED of ISRG / Let's Encrypt
Nobody is paying base price for a box like that. I’d probably bid them against HPE and pay ~90-100k.
If Amazon looks reasonable for something like this, the math is wrong. They’re renting boxes at 60-70% margin.
If Amazon looks reasonable for something like this, the math is wrong. They’re renting boxes at 60-70% margin.
> These are expensive servers, crossing into six digits, but not $200k.
https://news.ycombinator.com/item?id=25865967
https://news.ycombinator.com/item?id=25865967
It’s closer to $80k, if you went via a VAR.
I spec’d a current gen server with:
- 2x more cores
- 50% more RAM
- 50% more storage
For only ~$80k, and that’s not even with a discount and is a more powerful current gen server with more cores/ram/storage.
It’s $41k if I matched the specs exact (but that’s a bit unfair to do because the Dell server is 2-3 years old).
https://www.siliconmechanics.com/system/rackform-a335.v9
I spec’d a current gen server with:
- 2x more cores
- 50% more RAM
- 50% more storage
For only ~$80k, and that’s not even with a discount and is a more powerful current gen server with more cores/ram/storage.
It’s $41k if I matched the specs exact (but that’s a bit unfair to do because the Dell server is 2-3 years old).
https://www.siliconmechanics.com/system/rackform-a335.v9
> AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad.
Is that the on-demand cost, or the reserved cost? For comparing to buying a server outright, you should be comparing the reserved cost. I’m not sure exactly which instances you’re looking at to get $20k/mo, but I see some instances with 64-128 cores/1-2 TB memory for <10k/month.
For storage, I’m not sure how you’re getting >100k… I plugged in the highest IOPS I could for io2 volumes for 150 TB of storage and got 30k/mo. Also worth considering here that you don’t have to provision all 150 TB up front - you could start with 5 TB and increase in size as you grow, for example.
Still gonna be hella expensive but all of this changes the calculus quite a bit from your estimates.
Is that the on-demand cost, or the reserved cost? For comparing to buying a server outright, you should be comparing the reserved cost. I’m not sure exactly which instances you’re looking at to get $20k/mo, but I see some instances with 64-128 cores/1-2 TB memory for <10k/month.
For storage, I’m not sure how you’re getting >100k… I plugged in the highest IOPS I could for io2 volumes for 150 TB of storage and got 30k/mo. Also worth considering here that you don’t have to provision all 150 TB up front - you could start with 5 TB and increase in size as you grow, for example.
Still gonna be hella expensive but all of this changes the calculus quite a bit from your estimates.
They're also using ZFS in Raid1+0, so 38.4TB of usable storage. $14,566/month on AWS io2 with max IOPS.
I'd also bet money that this has higher disk I/O performance than the rough equivalent on Amazon.
> AWS does not have a 100% similar VM
does it have similar instance in principle: you rent dedicated server with lots of ssd attached and with no fear that instance will be stopped any moment for whatever reason?..
does it have similar instance in principle: you rent dedicated server with lots of ssd attached and with no fear that instance will be stopped any moment for whatever reason?..
Absolutely.
Never understood why people are so infatuated with "cloud" options. Yes, it is convinient, but you are absolutely paying at least order of magnitude more for the same amount of compute/storage.
Never understood why people are so infatuated with "cloud" options. Yes, it is convinient, but you are absolutely paying at least order of magnitude more for the same amount of compute/storage.
Because there's no such thing as a free lunch, running your own datacenter (or managing datacenter managed services) is work, and the clouds are better at providing exactly the compute you need when you need it than buying a ton of excess hardware you might or might not use (or can't scale quickly enough if you _do_ use it all)
A datacenter makes sense if your usage profile is steady state and hardly ever changes, or if growth rate is predictable and capacity can be procured in advance. Any other use case is better suited for the cloud, IMO.
A datacenter makes sense if your usage profile is steady state and hardly ever changes, or if growth rate is predictable and capacity can be procured in advance. Any other use case is better suited for the cloud, IMO.
there are several middle points between full cloud and full own datacenter: rent dedicated server, rent rack in collocation datacenter.
You still need to manage the underlying database and security which requires a lot of knowledge to do well.
correct, and managed solution will have cost too, quality issues: 3p services have bugs and are hacked too and that's something you can't control, and added complexity.
To me it looks like if you have small traffic/storage requirements, then cloud may look good. But if you have lots of data and need compute then running beefy bare-metal server can be very cost beneficial.
I still think that you need to manage some security things even if you are using AWS.
People forget that Colo & Leasing servers exist.
And it costs like 1/5th of cost of “cloud”, and the later still gets you “the cloud”.
And it costs like 1/5th of cost of “cloud”, and the later still gets you “the cloud”.
Correct, but, again, free lunches don’t exist.
Now you’re in the business of maintaining/patching/imaging/securing your operating system and all of the libraries/builtins it comes with...along with your app's stack.
That's not nearly as simple as "Hey, here's my code, Lightsail; ship it and present me its URL. Thanks!"
Now you’re in the business of maintaining/patching/imaging/securing your operating system and all of the libraries/builtins it comes with...along with your app's stack.
That's not nearly as simple as "Hey, here's my code, Lightsail; ship it and present me its URL. Thanks!"
In my experience, in-house groups racking and configuring and maintaining boxes often become a priesthood of negative value bickering over pet machine names.
While cloud overcharges for McD product, I know what I get.
While cloud overcharges for McD product, I know what I get.
Not to mention that you get it within minutes of asking for it. I've worked at places where the internal bureaucracy managed to take double digit weeks to deliver a new database instance.
It‘a very depressing hearing this. Anything cloud at my company takes exponentially longer to setup as we have layers and layers of bureaucracy for any data that leaves our network.
The grass is always greener on the other side of the fence I guess. :)
I've worked at places where the internal bureaucracy managed to take double digit weeks to deliver a new database instance.
That wouldn't happen if the internal bureaucracy for firing the laggards wasn't so excruciatingly slow.
That wouldn't happen if the internal bureaucracy for firing the laggards wasn't so excruciatingly slow.
It's gotten absolutely bonkers with all the demand for GPUs and the cloud companies are just raking it in hand over fist.
On AWS reserved instances with 8x A100s or H100s (if you can even get them) cost more per year than the total upfront retail price of the equivalent pods from Lambda Labs. The on-demand price is even more absurd.
On AWS reserved instances with 8x A100s or H100s (if you can even get them) cost more per year than the total upfront retail price of the equivalent pods from Lambda Labs. The on-demand price is even more absurd.
Because cloud infatuation is largely a myth. Every middle or large organization does periodic cost analysis and select cloud based on that analysis not on convenience. e.g. See this: https://aws.amazon.com/solutions/case-studies/emirates-case-....
> See this
that looks like some marketing material without any specifics..
that looks like some marketing material without any specifics..
I used to lead a widely used and influential platform eng team at Adobe. Parent comment is right. We had our own datacenters and multiple cloud providers, and growing the public cloud infrastructure had a demonstrably higher return on investment than the private cloud.
If you don't need to host something big, where you can get away with one good VPS, the cloud has the benefit of offering cheap bandwidth.
Once you have the bandwidth at your location and you don't need to be present in multiple locations, it's cheaper to self-host.
Next step would be colocation, but for a start, using cloud offerings is a cheap way to be a part of the internet.
Once you have the bandwidth at your location and you don't need to be present in multiple locations, it's cheaper to self-host.
Next step would be colocation, but for a start, using cloud offerings is a cheap way to be a part of the internet.
Cloud bandwidth is definitely not cheap. If anything, it's where they rip you off the most.
You can get "baremetal"/dedicated servers from places like Hetzner and OVH that give unmetered gigabit connections for like $50.
You can get "baremetal"/dedicated servers from places like Hetzner and OVH that give unmetered gigabit connections for like $50.
You can not get actual unmetered 1gb/s for anywhere close to that. If you start pushing anywhere close to that much bandwidth, you will be throttled / have your account closed. For example, Hetzner caps your bandwidth at 20TB per month.
Additionally, if you are actually pushing close to that much traffic, you can negotiate guaranteed commit prices w/ AWS that are competitive (especially when you consider the quality of bandwidth. I can only get ~100 mb/s to my hetzner server because of how bad their peering is. I can easily saturate my 1GB connection to anywhere in AWS.)
---
(Having said that, this does only apply to egress via cloudfront. Things like charging for intra A-Z bandwidth within the same region is insane, and for many workloads may be surprisingly expensive.)
Additionally, if you are actually pushing close to that much traffic, you can negotiate guaranteed commit prices w/ AWS that are competitive (especially when you consider the quality of bandwidth. I can only get ~100 mb/s to my hetzner server because of how bad their peering is. I can easily saturate my 1GB connection to anywhere in AWS.)
---
(Having said that, this does only apply to egress via cloudfront. Things like charging for intra A-Z bandwidth within the same region is insane, and for many workloads may be surprisingly expensive.)
> For example, Hetzner caps your bandwidth at 20TB per month.
looks like your information is obsolete by 5 years: https://www.hetzner.com/news/traffic-limit
looks like your information is obsolete by 5 years: https://www.hetzner.com/news/traffic-limit
Your random blog post is the outdated one.
Their actual docs: https://docs.hetzner.com/robot/general/traffic/
And reports of throttling once you actually hit certain limits: https://lowendtalk.com/discussion/180504/hetzner-traffic-use...
Their actual docs: https://docs.hetzner.com/robot/general/traffic/
And reports of throttling once you actually hit certain limits: https://lowendtalk.com/discussion/180504/hetzner-traffic-use...
your link says exactly what blog post said: root servers with 1gb/s uplink have unlimited traffic.
Other offerings (e.g. 20gb/s uplink) and small cloud servers have traffic limitations.
Other offerings (e.g. 20gb/s uplink) and small cloud servers have traffic limitations.
Yes, I was pointing out that many of their products have explicit limits.
But more importantly, the second link I posted shows how that despite being "unlimited" - it's not uncommon for hetzner to throttle / close your account if you go over unstated traffic limits.
But more importantly, the second link I posted shows how that despite being "unlimited" - it's not uncommon for hetzner to throttle / close your account if you go over unstated traffic limits.
> Yes, I was pointing out that many of their products have explicit limits.
No, you clearly stated that 1gb/s servers have 20tb cap, but no such products have such limitation, no links say anything like that.
It makes total sense to cap 20gb/s, because such unlimited bandwidth is clearly too much.
> But more importantly, the second link I posted shows how that despite being "unlimited" - it's not uncommon for hetzner to throttle / close your account if you go over unstated traffic limits.
you can argue that, though they didn't close or throttle account, they sent warning, and cap was 12 times higher than what you previously stated.
No, you clearly stated that 1gb/s servers have 20tb cap, but no such products have such limitation, no links say anything like that.
It makes total sense to cap 20gb/s, because such unlimited bandwidth is clearly too much.
> But more importantly, the second link I posted shows how that despite being "unlimited" - it's not uncommon for hetzner to throttle / close your account if you go over unstated traffic limits.
you can argue that, though they didn't close or throttle account, they sent warning, and cap was 12 times higher than what you previously stated.
What you pay for is the ability to provision hardware for new R&D and projects in minutes rather than days, weeks or months. Companies are willing to spend millions a month in cloud fees to accelerate hundreds of millions in revenue.
I think there are ton of great use cases for the cloud, but people should try and think for themselves and decide if their circumstances and workload is really a good fit.
A ton of people forget that a bunch of servers across a few colocations can pay for itself in months, especially if you go for second-hand gear that is dirt cheap.
Again, going (back) to colocating hardware could not be good fit. But with modern management tools and datacenter services like 'remote hands' I think people should not reject it upfront.
A ton of people forget that a bunch of servers across a few colocations can pay for itself in months, especially if you go for second-hand gear that is dirt cheap.
Again, going (back) to colocating hardware could not be good fit. But with modern management tools and datacenter services like 'remote hands' I think people should not reject it upfront.
[deleted]
I'm just now getting serious about a homelab and it's shocking how much compute you can get for peanuts from enterprise surplus. I've had this machine (1U poweredge r620) for about a year and I'm already itching for an upgrade. Hopefully something with a lot of 3.5" drive bays for long-term data hoarding.