State of netbooting Raspberry Pi in 2021(blog.alexellis.io)
blog.alexellis.io
State of netbooting Raspberry Pi in 2021
https://blog.alexellis.io/state-of-netbooting-raspberry-pi-in-2021/
59 comments
This article rambles about everything under the sun and barelly scratches the topic promised in its title.
The article links to a previous article which in turn links to the official RPi documentation on netbooting. Posting here for people who are curious: https://www.raspberrypi.org/documentation/computers/remote-a...
It appears to be an advertisement for the author's $40 workshop video which maybe has the actual details of how this is done.
One thing I've always been curious about is hardware for mass-writing of sdcards.
I've setup manufacturing stations for this and I've never been able to find truly robust gang-programmers for sdcards. Luckily my stuff was all low volume so it wasn't a serious problem.
But it seems like there's no "manufacturing grade" gang programmers for sdcards. The stuff that is out there is just infinitely re-branded chinese "white-label" hardware-- just a lot of different "vendors" for a few different models of the basically the same hardware.
My suspicion for there not being good options for gang-programming of sdcards is that savvy consumer manufacturers have realized what a headache it is to support sdcards in their products from the factory and thus opt to make the sdcard the customer's responsibility. In other words, they ship their products without sdcards and just require the customer to purchase the sdcard and program it if needed. I am not sure if my suspicions are justified, can anyone confirm if this is the case?
I've setup manufacturing stations for this and I've never been able to find truly robust gang-programmers for sdcards. Luckily my stuff was all low volume so it wasn't a serious problem.
But it seems like there's no "manufacturing grade" gang programmers for sdcards. The stuff that is out there is just infinitely re-branded chinese "white-label" hardware-- just a lot of different "vendors" for a few different models of the basically the same hardware.
My suspicion for there not being good options for gang-programming of sdcards is that savvy consumer manufacturers have realized what a headache it is to support sdcards in their products from the factory and thus opt to make the sdcard the customer's responsibility. In other words, they ship their products without sdcards and just require the customer to purchase the sdcard and program it if needed. I am not sure if my suspicions are justified, can anyone confirm if this is the case?
There is no reason at all to boot a raspberry pi 4 from an sdcard in 2021. It's the wrong tool for the job and the worst option you can choose.
https://github.com/pftf/RPi4 enables several alternatives, all faster and more reliable
* usb3 -> sata
* usb3 -> nvme
* iscsi
* ipxe
* netboot.xyz via pxe or usb
* usb thumbdrive
https://github.com/pftf/RPi4 enables several alternatives, all faster and more reliable
* usb3 -> sata
* usb3 -> nvme
* iscsi
* ipxe
* netboot.xyz via pxe or usb
* usb thumbdrive
With the Compute Module 4, two more possibilities open up:
* NVMe (native)
* eMMC
(SATA can be used natively, too, but you can't yet boot from a SATA drive on the Pi.)
* NVMe (native)
* eMMC
(SATA can be used natively, too, but you can't yet boot from a SATA drive on the Pi.)
But it's not uncommon to have devices boot from sdcards-- even industrial boards. Are you talking about "all" sdcards? Or just consumer stuff whose actual partnumber could change from order to order?
It's more uncommon nowadays. VMware has officially recommended against sdcard boot, which used to be fairly popular. The SD card option is still usually available but generally recommended against when getting blades or pizzaboxes from HPE, Cisco and Dell. For various reasons it's now nearly the same price to toss 2x nvme drives in, instead of the SD card raid controller and SD card mounting assembly. TCO is definitely lower due to the write resilience of your typical nvme flash drive compared to an SD card, physical work swapping parts can be way more expensive than a small capital increase.
With stuff like this the "right tool" is whatever you have and usb3 -> sata adapters aren't all that common while functioning SD cards are.
Yeah there are crappy ones but if you stick to ~2GB and decent vendors (like most people do) you're unlikely to deal with them.
Yeah there are crappy ones but if you stick to ~2GB and decent vendors (like most people do) you're unlikely to deal with them.
I just checked and it looks like netboot.xyz provides an sdcard image so no excuses :)
https://boot.netboot.xyz/ipxe/netboot.xyz-rpi4-sdcard.img
Edit - ok I guess being airgapped would be one, fair enough!
https://boot.netboot.xyz/ipxe/netboot.xyz-rpi4-sdcard.img
Edit - ok I guess being airgapped would be one, fair enough!
Decent vendors don't save you at scale, until you're in the millions of units per year kind of scale. Until the all but the 'industrial' cards are crappy. I've had the displeasure of trying to track down their issues before professionally.
I hope you guys get eMMC someday. It's great.
Depends on which RPi you use, the Compute Module 4 comes in 2/8/16/32MB eMMC flavors[1]
1: https://www.raspberrypi.org/products/compute-module-4/
1: https://www.raspberrypi.org/products/compute-module-4/
If you have no external drive or network with a "boot server" like is probable most cases of use with RPi, is there a better option than an sdcard?
Also, how is a usb thumbdrive better than a sdcard?
Also, how is a usb thumbdrive better than a sdcard?
reliability
If you buy them in volume, you can buy them with the data already on from the manufacturer. I'm pretty sure it's programmed into the flash before the flash is assembled into the sdcard packaging.
You can do a similar thing with microcontrollers - if you're buying sufficient volume the manufacturer will supply them with your program already on, saving you a manufacturing step.
You can do a similar thing with microcontrollers - if you're buying sufficient volume the manufacturer will supply them with your program already on, saving you a manufacturing step.
Yes, I've done that before for flash IC's. It works out nicely for "large enough" order quantities. I would expect that doing that for sdcards would require even larger quantities to be economically viable.
Fortunately Darren Shepherd came up with a novel idea of taking way the things he didn't want in Kubernetes and created a "distro" called K3s (3 is less than 8)
It's not that it is less, "3" is what you get if you take the character "8" and cut it in half vertically.
It's not that it is less, "3" is what you get if you take the character "8" and cut it in half vertically.
From the k3s readme, it states kubernetes is 10 letters stylized as k8s. Half of 10 is 5, and then the project "stylized" it as k3s.
Indeed, you are right!
The explanation above was given by one of the Rancher staff members in one tech conference. It may be an alternative explanation that popped up with time.
In both the point is that k3s is "half of" k8s.
The explanation above was given by one of the Rancher staff members in one tech conference. It may be an alternative explanation that popped up with time.
In both the point is that k3s is "half of" k8s.
The article has the wrong caption I would think ("running K3s on a bunch of PIs" would be more fitting).
Netbooting a bare metal environment is quite straightforward for the PI 3 (dnsmasq has it all) and takes around 15-30 secs.
Netbooting a bare metal environment is quite straightforward for the PI 3 (dnsmasq has it all) and takes around 15-30 secs.
SDcards are said to die often/easily in Raspberry Pis. I heard someone say they die when there's a write access going while the Raspi is taken from power. Is that true? That would mean running a Raspi from SDcard is fine as long as there is no write access happening, which can be done by enabling readonly (root) filesystem.
Can anyone comment on that?
Can anyone comment on that?
I have had a few cards go but I never isolated the cause. Certainly unceremoniously cutting power occurred a few times.
I got tired of the inconvenience of having to rebuild the Pi installation and flashing SD cards, so now I boot my Pi4 directly from a USB attached SSD, no card on-board at all. I haven't looked back. Others seem to also be getting good result like: https://www.jeffgeerling.com/blog/2020/im-booting-my-raspber...
I got tired of the inconvenience of having to rebuild the Pi installation and flashing SD cards, so now I boot my Pi4 directly from a USB attached SSD, no card on-board at all. I haven't looked back. Others seem to also be getting good result like: https://www.jeffgeerling.com/blog/2020/im-booting-my-raspber...
From what happened to me - cards die, especially older ones when there is a lot of data written to them even without power failures (UPS ensured that). And when you have a journaled filesystem there is a lot of writes.
I made hybrid systems where either the card is only used to boot the system and everything (root included) is loaded from a USB drive or where card is mounted R/O, there is small ramdisk for volatile data and only when upgrade/reconfiguration is needed you remount it r/w, do what you need and then remount it back r/o. In both those setups cards didn't fail for the last 3-4 years. With a standard SD card root-fs they lasted about 4-6 months.
I made hybrid systems where either the card is only used to boot the system and everything (root included) is loaded from a USB drive or where card is mounted R/O, there is small ramdisk for volatile data and only when upgrade/reconfiguration is needed you remount it r/w, do what you need and then remount it back r/o. In both those setups cards didn't fail for the last 3-4 years. With a standard SD card root-fs they lasted about 4-6 months.
> "Where kubeadm created a headache for Raspberry Pi, K3s made things simple with an easy boostrap, low resource requirements and a single binary for distribution and updates."
I'm curious what the headache was. K8S has shipped arm kubeadm and kubelet bins for quite a while.
I also don't understand how k3s, microkube and others claim to be "lightweight". Lightweight compared to what? Certainly not upstream kubeadm which is as lightweight a bootstrap as you can get.
> "Given how much I'd liked the Docker experience of "docker swarm init" and "docker swarm join", I decided to re-create a similar experience for K3s in the k3sup tool"
"kubeadm init" and "kubeadm join"
I'm curious what the headache was. K8S has shipped arm kubeadm and kubelet bins for quite a while.
I also don't understand how k3s, microkube and others claim to be "lightweight". Lightweight compared to what? Certainly not upstream kubeadm which is as lightweight a bootstrap as you can get.
> "Given how much I'd liked the Docker experience of "docker swarm init" and "docker swarm join", I decided to re-create a similar experience for K3s in the k3sup tool"
"kubeadm init" and "kubeadm join"
> I also don't understand how k3s, microkube and others claim to be "lightweight". Lightweight compared to what?
For k3s, they explain what they do in their README:
https://github.com/k3s-io/k3s#how-is-this-lightweight-or-sma...
For k3s, they explain what they do in their README:
https://github.com/k3s-io/k3s#how-is-this-lightweight-or-sma...
K3s for example is more lightweight than upstream kubeadm because it compiles all the required components into a single binary. That also leads to considerably less operational complexity.
kubeadm dosen't include ANY components such as kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy or etcd.
These components are all containerized already from upstream. Pulled and cached when you run "kubeadm init" via static pod manifests.
It seems trying to bundle these inside a non-upstream bootstrapper and running them out-of-band from kubernetes is what would add complexity and bloat.
These components are all containerized already from upstream. Pulled and cached when you run "kubeadm init" via static pod manifests.
It seems trying to bundle these inside a non-upstream bootstrapper and running them out-of-band from kubernetes is what would add complexity and bloat.
I think you might be missing the point. You are correct, kubeadm doesn't include any components in that sense and runs them in containers. The issue with this is that there's a lot of overheads for each service that Kubernetes runs, causing Kubernetes as a whole to have a bigger memory footprint. K3s on the other hand combines all the necessary services and runs them in a single binary, which allows for a reduced memory footprint with very little lost in functionality. I don't know what all was involved when this was done but that is the gist of it.
Are RasPi's actually performant for this kind of use case? Like relative to just getting a random desktop or server rack... thingy? I can't help but feel like at one point you're just generating a bunch of garbage/wasting energy compared to more performant offerings that might be more efficient all around.
Is it price?
(Just to be clear, I think "having fun" is a fine excuse to do most stuff, but they're talking about "replacing usage of AWS" which sounds a hell of a lot like a serious business thingy)
Is it price?
(Just to be clear, I think "having fun" is a fine excuse to do most stuff, but they're talking about "replacing usage of AWS" which sounds a hell of a lot like a serious business thingy)
Hardly performant enough. You are way better working with a Xeon server, even used ones.
Would certainly be interesting to see the comparison in terms of computing power and power usage between a small pi cluster and a single XEON server.
I imagine such a thing would draw at least 200W? And pi's maybe 5 watts? So in theory, 30 - 40 pis vs a single server?
The pis also have an advantage in terms of redundancy. A single pi going down shouldn't be a problem and very easily replaceable.
Would be a really interesting article.
I imagine such a thing would draw at least 200W? And pi's maybe 5 watts? So in theory, 30 - 40 pis vs a single server?
The pis also have an advantage in terms of redundancy. A single pi going down shouldn't be a problem and very easily replaceable.
Would be a really interesting article.
I'll be addressing that in a video later today: https://www.youtube.com/watch?v=8zXG4ySy1m8
There are situations where the economics slightly favor the Pi, but they're fairly narrow with the current gen.
There are situations where the economics slightly favor the Pi, but they're fairly narrow with the current gen.
VS desktop class hardware, the power draw is a giant plus also leading to no or quiet fans. An old laptop, however, has about the same power/performance consumption while including an USP, keyboard, mouse and screen, albeit it is bigger and has no GPIO if you need those.
That being said, I am running a NAS and PiHole on a RPi4B right now and am happy. I am planning on turning an old laptop with busted hinges into a server with a nice housing though. I'm very interested how it's going to pan out :)
Edit: I had a 1U server with 2x intel CPUs (16 threads all in all I think?) with 32G RAM in 2012 or so and it didn't see much use. It was LOUD (small screaming fans that pushed air through the chasis) so it didn't see much use.
That being said, I am running a NAS and PiHole on a RPi4B right now and am happy. I am planning on turning an old laptop with busted hinges into a server with a nice housing though. I'm very interested how it's going to pan out :)
Edit: I had a 1U server with 2x intel CPUs (16 threads all in all I think?) with 32G RAM in 2012 or so and it didn't see much use. It was LOUD (small screaming fans that pushed air through the chasis) so it didn't see much use.
Indeed, if I'm going to build a home or small-office server, I want a 4U case, just so I can use fans that are large enough to be reasonably quiet (let's say, 80mm or bigger). Bonus points if it could somehow run 120 or 140 mm fans, but I haven't seen such a beast. I've seen 1U servers in action at a client, and they're horrifyingly loud.
If you don't really need a rack case, I'd even go as far as to use a normal case with regular Noctua fans. The hardware for PCs is probably more expensive though (plus no ECC, single socket, ...).
Used Xeons are cheap and plentiful on eBay and Craigslist. The power consumption and jet engine fans are something most people wouldn’t tolerate.
That’s why I use PIs and custom built a NAS with a supermicro mobo and an i3 (ECC compatible).
The rig the author has is bordering on the ridiculous side and can never match a Xeon, or Xeon cluster.
That’s why I use PIs and custom built a NAS with a supermicro mobo and an i3 (ECC compatible).
The rig the author has is bordering on the ridiculous side and can never match a Xeon, or Xeon cluster.
Odd to see using UEFI to NetBoot not considered an option at all.
It should eliminate whatever timing-bugs you have and leave you in a more reliable (and scriptable!) pre-boot environment. What more do you need?!?
Both the RPi3 and 4 can be UEFI booted, so this is definitely a real-world option.
Links:
- https://github.com/pftf/RPi3
- https://github.com/pftf/RPi4
It should eliminate whatever timing-bugs you have and leave you in a more reliable (and scriptable!) pre-boot environment. What more do you need?!?
Both the RPi3 and 4 can be UEFI booted, so this is definitely a real-world option.
Links:
- https://github.com/pftf/RPi3
- https://github.com/pftf/RPi4
The BitScope racks are neat, but the thing that makes me sad about Pi is that there's no way to network from the 40 pin connector that would be reasonable to tie this all together. So you need a separate network infrastructure and a network cable to each pi.
The 40 pins on the Pi are intended for GPIO and low-speed connectivity (SPI, I2C and friends). Unless I'm missing something, it is simply not feasible to use those GPIO ports for any high-speed interfaces for three reasons:
1. The pins are likely connected to a memory-mapped GPIO controller in the Broadcom SoC (you write a value to a special memory location, and the GPIO controller will physically change the voltage on the associated pin). While you can implement certain slow interfaces - UART is the typical example - purely in software using bit-banging, this approach only works as long as your desired interface speed is a lot slower than the CPU clock. For example: To implement a UART controller at 9600 Bd running on a 1 GHz CPU, you have over 100000 cycles for each symbol you send or receive. For 1000BASE-T, you are looking at a symbol rate of 125000000 Bd, so you'd have just 8 cycles for each symbol (and this is completely ignoring the line code used for 1000BASE-T, which the GPIO controller very likely doesn't support on a physical level).
2. If you want to run an interface faster than a few MHz, you have to be more careful on the electrical level to avoid reflections/impedance mismatch and other signal integrity issues. Essentially, you can't just hook up 2.54mm pin headers at gigabit speeds and expect everything to work fine.
3. Related to the two previous points: Almost all high-speed interfaces (including GbE, USB, HDMI etc.) use line codes at the physical level and/or use differential voltage signaling (LVDS/TMDS etc.) for transmission. The Pi's GPIO controller is very likely unable to send/receive data using differential standards. By the way, if you're interested in this topic: Most FPGAs (even entry-level FPGAs) support differential voltage standards in their IO blocks and even include special encoder/decoder blocks (for 8b/10b encoding).
1. The pins are likely connected to a memory-mapped GPIO controller in the Broadcom SoC (you write a value to a special memory location, and the GPIO controller will physically change the voltage on the associated pin). While you can implement certain slow interfaces - UART is the typical example - purely in software using bit-banging, this approach only works as long as your desired interface speed is a lot slower than the CPU clock. For example: To implement a UART controller at 9600 Bd running on a 1 GHz CPU, you have over 100000 cycles for each symbol you send or receive. For 1000BASE-T, you are looking at a symbol rate of 125000000 Bd, so you'd have just 8 cycles for each symbol (and this is completely ignoring the line code used for 1000BASE-T, which the GPIO controller very likely doesn't support on a physical level).
2. If you want to run an interface faster than a few MHz, you have to be more careful on the electrical level to avoid reflections/impedance mismatch and other signal integrity issues. Essentially, you can't just hook up 2.54mm pin headers at gigabit speeds and expect everything to work fine.
3. Related to the two previous points: Almost all high-speed interfaces (including GbE, USB, HDMI etc.) use line codes at the physical level and/or use differential voltage signaling (LVDS/TMDS etc.) for transmission. The Pi's GPIO controller is very likely unable to send/receive data using differential standards. By the way, if you're interested in this topic: Most FPGAs (even entry-level FPGAs) support differential voltage standards in their IO blocks and even include special encoder/decoder blocks (for 8b/10b encoding).
:D It's not my first rodeo doing high speed design.
> 2. If you want to run an interface faster than a few MHz, you have to be more careful on the electrical level to avoid reflections/impedance mismatch and other signal integrity issues. Essentially, you can't just hook up 2.54mm pin headers at gigabit speeds and expect everything to work fine.
Conversely, we got a gigabit out of a few wires, non-differential, on pin headers UltraDMA/ATAPI over distances of tens of inches, albeit with the insulation displacement connector able to put a ground between each wire on the ribbon to lower impedance a bit.
It would be nice to have some more capable interfaces multiplexed onto the pins. That's all I'm saying.
For that matter, an RJ45 doesn't have wonderful electrical characteristics, but it's a small impedance discontinuity at the ends of the circuit. Industrial uses put gigabit ethernet through all kinds of connectors. Gigabit Ethernet works in practice through a few inches of untwist and would work just fine through a pin header. Already not everything on the header is hooked up to a single GPIO controller-- some goes directly to the broadcom and some to an external GPIO controller on earlier PIs, I believe.
> 2. If you want to run an interface faster than a few MHz, you have to be more careful on the electrical level to avoid reflections/impedance mismatch and other signal integrity issues. Essentially, you can't just hook up 2.54mm pin headers at gigabit speeds and expect everything to work fine.
Conversely, we got a gigabit out of a few wires, non-differential, on pin headers UltraDMA/ATAPI over distances of tens of inches, albeit with the insulation displacement connector able to put a ground between each wire on the ribbon to lower impedance a bit.
It would be nice to have some more capable interfaces multiplexed onto the pins. That's all I'm saying.
For that matter, an RJ45 doesn't have wonderful electrical characteristics, but it's a small impedance discontinuity at the ends of the circuit. Industrial uses put gigabit ethernet through all kinds of connectors. Gigabit Ethernet works in practice through a few inches of untwist and would work just fine through a pin header. Already not everything on the header is hooked up to a single GPIO controller-- some goes directly to the broadcom and some to an external GPIO controller on earlier PIs, I believe.
Genuinely curious, how would you network from the 40 pin connector in theory? Is it still ethernet?
Probably? With SPI or much slower the UART and pppd.
It seems that many options are available, but SPI and NAND flash boot isn't supported (yet?): https://www.raspberrypi.org/documentation/computers/raspberr...
Something I'd like to know, if you're moving off AWS and using your own raspberry pi cluster, what would you do to provide a performant MySQL DB for example? I can't imagine that trying to run a high performance mysql instance on a PI makes sense. So would you still want an old-school rack server with RAID and all the rest to be your DB server?
Ask yourself, why do you need a MySQL DB in the first place? If the service can get away with sqlite, use that. Alternatively, there is also rqlite (and I think another similar project) that allows for sqlite to be HA.
rqlite author here, happy to answer any questions about it.
https://github.com/rqlite/rqlite
https://github.com/rqlite/rqlite
Hey Alex, that was quite a writeup up it didn't really touch upon what exactly did you flash & where did it get flashed?
That was the interesting detail that I was looking for too. I'm guessing that it's because at the bottom of the article you can see that he wants you to pay $30 for his ebook and video that will tell you how.
From my own experience, they're probably flashing the RPi4's EEPROM with an updated bootloader and config to enable netboot. Here's the official documentation on it: https://www.raspberrypi.org/documentation/computers/raspberr...
No, he says that he flashed SD cards.
> I flashed each of the Raspberry Pis using the same SD card, then I could close the case and take a note of each MAC address and serial number for later on.
But if flashing SD cards, then why do we need network boot? Or, if we have network boot, why do we need to use SD cards?
> I flashed each of the Raspberry Pis using the same SD card, then I could close the case and take a note of each MAC address and serial number for later on.
But if flashing SD cards, then why do we need network boot? Or, if we have network boot, why do we need to use SD cards?
The easiest way to perform the flash is by booting into Raspbian and performing the process interactively. Without this one-time process, you can't netboot, as the default setting is the SD card, then USB: https://www.raspberrypi.org/documentation/computers/raspberr...
How fast can a Pi boot? Are they suitable for embedded environments where "instant" booting is required? Are there any alternatives which run Linux with graphics which can boot faster?
The Pi has a weird boot sequence because of the Broadcom chip.
SBCs with uncomplicated boot loaders can get you to a shell prompt within one second of applying power; I imagine getting to a lightweight graphical environment in under 2 seconds is not hard. Doubtless you can reduce that a little with tweaking. That's a good deal faster than my car stereo, say.
SBCs with uncomplicated boot loaders can get you to a shell prompt within one second of applying power; I imagine getting to a lightweight graphical environment in under 2 seconds is not hard. Doubtless you can reduce that a little with tweaking. That's a good deal faster than my car stereo, say.
There is bare metal code that almost immediately starts after powering the Pi on. A Pi3 and below with an absolute minimal Linux based OS could probably boot in 3-4 seconds or so. With the Pi4 that has changed and the EEPROM can take a few seconds on its own before the Pi even gets to the kernel loading phase.
TL;DR: no.
TL;DR: no.