There's a lot of nonsense at every level. Especially when dealing with heterogenous infrastructure.
Some NICs support http. Some NICs support tftp. Some NICs have enough memory for a big iPXE, other NICs don't. Some BMC systems make next-boot-to-lan easy, but not all.
We almost always use iPXE in order to normalize our pxe environment before OS kickstart. There's a lot to it and quite a lot of little things that can go wrong. Oh, and every bit of it becomes critical infra.
The fun thing about learning to boot from PXE, is that you have to learn it every time you onboard a new type of hardware... or a new VM hypervisor... or new NIC firmware... or new BIOS firmware.
God help you if you actually want to install an operating system.
PXE is such a vital capability for working with on-prem servers. But it's ten different things which all have to play nicely together. Every time I build a PXE system I feel like I'm reinventing the universe in my tiny subnet.
Yes, I used LLMs to develop this. I think the README has more emojis than any mortal could summon. Hehe
I used ChatGPT to design the solution that I wanted and Claude Sonnet to do most of the coding.
I'm trying to figure out what works for me in the brave new world of AI enabled development, so that I can make recommendations to my team.
A few things that really helped me here were:
- Having the gitlab cli (glab) installed and configured was very helpful because it allowed me to do things like lint the CI file and inspect the build output in the LLM context.
- Having the zereight/gitlab-mcp installed was useful as well. Even though I can make Issues and MRs using the CLI, the LLM frequently made escaping mistakes when writing long comment sections. The mcp tool was great for this.
- Almost all of my process started with me describing a bug or feature, then asking the LLM to investigate the feature and create an Issue. From there I tried as much as possible to keep the scope of my work small and exclusively tied to an issue branch.
I'm a reasonably good programmer - I've been at it for 30 years. I think there's no question that LLMs expand my "radius of capability." Just like everyone else, I'm trying to figure out the best way to safely maximize this new world of tools.
My team does quite a bit of this. We handle it in two different ways:
For some clusters we carve nodes out of VMWare simply using OS templates. For other nodes we we use cheap-and-deep blade servers and install the OS on bare-metal using PXE. Once the nodes are provisioned we use ansible to deploy Kubernetes. (Lately it's been RKE2 on top of Rocky.)
Generally speaking VM-based nodes are extremely reliable and seldom have to be rebuilt. (If we're paying to run VMWare its because the underlying hardware is high-quality.) Bare-metal nodes, on the other hand, are built on inexpensive hardware and they tend to fail in different ways. When they fail we cordon and remove them from the cluster and put them in a list. (We maintain sufficient overcapacity to soak failures as they come.)
If we're using persistence we have to take care that the statefulsets are configured correctly. Sometimes we use local-disk persistence so that our services can benefit from local NVME performance. Other times we use NFS (when we need persistence but not performance.)
We monitor cluster node health internally to Kubernetes and also externally using Nagios (shudder).
Kubernetes upgrades are a pain in the ass. Lots of times we'll just set up a second cluster to avoid the risk of a failure during an upgrade.
So if this power line will carry 1250MW. And elsewhere Hacker News tells me that it takes 200J to restart a heart.
Math... math... math... wolframalpha... calc.exe... math
I think this can restart 6.25M hearts per second.
(Please don't try this)
Edit: The defib duty cycle is probably closer to 100ms. So you could probably do 10 times this number, especially since you don't need a charge capacitor.
Some NICs support http. Some NICs support tftp. Some NICs have enough memory for a big iPXE, other NICs don't. Some BMC systems make next-boot-to-lan easy, but not all.
We almost always use iPXE in order to normalize our pxe environment before OS kickstart. There's a lot to it and quite a lot of little things that can go wrong. Oh, and every bit of it becomes critical infra.