So you want to build an embedded Linux system?(jaycarlson.net)
jaycarlson.net
So you want to build an embedded Linux system?
https://jaycarlson.net/embedded-linux/
41 comments
Disclosure: Raspberry Pi employee, focussed on better tooling for people building products with Raspberry Pi devices.
--
Raspberry Pi provide pi-gen (https://github.com/RPi-Distro/pi-gen), which allows you to build a customised Debian-based OS that uses our package list to get updates as we release them, with a low impedance mismatch between developing your software on Raspberry Pi OS and your deployment platform. Since it's debian based, your resultant system winds up being _very_ familiar.
We also provide tooling that compliments pi-gen. For example, rpi-sb-provisioner (https://github.com/raspberrypi/rpi-sb-provisioner), which will happily ingest an OS image created by pi-gen and automate the configuration of signed boot with an encrypted rootfs on CM4 devices. I've used it to flash half a dozen Raspberry Pi devices in rapid form as part of testing, and it runs on our own hardware.
I'd love to see what people make of these tools, and where we could expand support / docs / ascii-art. I don't check HN very often - but I'm pretty responsive to Github issues.
--
Raspberry Pi provide pi-gen (https://github.com/RPi-Distro/pi-gen), which allows you to build a customised Debian-based OS that uses our package list to get updates as we release them, with a low impedance mismatch between developing your software on Raspberry Pi OS and your deployment platform. Since it's debian based, your resultant system winds up being _very_ familiar.
We also provide tooling that compliments pi-gen. For example, rpi-sb-provisioner (https://github.com/raspberrypi/rpi-sb-provisioner), which will happily ingest an OS image created by pi-gen and automate the configuration of signed boot with an encrypted rootfs on CM4 devices. I've used it to flash half a dozen Raspberry Pi devices in rapid form as part of testing, and it runs on our own hardware.
I'd love to see what people make of these tools, and where we could expand support / docs / ascii-art. I don't check HN very often - but I'm pretty responsive to Github issues.
> Since an embedded Linux system will likely require use of either Buildroot or Yocto,
I don't know if that's the case anymore. The kernel ships with a lot of drivers now, and Root FS for embedded devices aren't really space constrained anymore. The needs of an IoT device have also grown to include things like containers and telemetry. In my experience, forking off of an existing distribution like Debian is way easier than navigating Buildroot or Yocto's documentation and cruft, and results in faster builds.
I got so fed up with those tools I eventually built a custom tool (Etcha) to build my own meta distribution (EtchaOS) of immutable variants for Debian, Fedora, etc. It owns the entire provisioning pipeline and builds multi-arch images in less than 10 minutes:
https://etcha.dev/etchaos/explanations/build-process/
I don't know if that's the case anymore. The kernel ships with a lot of drivers now, and Root FS for embedded devices aren't really space constrained anymore. The needs of an IoT device have also grown to include things like containers and telemetry. In my experience, forking off of an existing distribution like Debian is way easier than navigating Buildroot or Yocto's documentation and cruft, and results in faster builds.
I got so fed up with those tools I eventually built a custom tool (Etcha) to build my own meta distribution (EtchaOS) of immutable variants for Debian, Fedora, etc. It owns the entire provisioning pipeline and builds multi-arch images in less than 10 minutes:
https://etcha.dev/etchaos/explanations/build-process/
unrelated but curious - what do you think of OSTree-based embedded distros? Any future in it?
I see it as an overly complicated way of trying to apply containerization concepts to a base OS via transactions. It's really hard to use OSTree on new projects or get existing distros like Debian to work with OSTree. I think the update process, while in theory should be better, ends up being a black box of will it/won't it frustration for end users.
Finally, OSTree is mainly a Red Hat thing and is built for their needs (CoreOS). They could pull the rug on it in the sense that what you build today using it may not work tomorrow if Red Hat needs OSTree to function differently. Which is their perogrative, as they are the primary contributors to it.
Finally, OSTree is mainly a Red Hat thing and is built for their needs (CoreOS). They could pull the rug on it in the sense that what you build today using it may not work tomorrow if Red Hat needs OSTree to function differently. Which is their perogrative, as they are the primary contributors to it.
What do you think about using btrfs snapshots instead of ostree similar to openSUSE MicroOS or SLE Micro?
For the projects I've done, we're not usually particularly space constrained, but we are usually at least one of:
* Cost constrained so RAM constrained * Need fast boot times * For a niche processor architecture
Starting with very little apart from an init system and busybox and then building up from there has been a good approach to get the performant (by whatever metric) platform that I need.
That said, I'm not a fan of Yocto at all. I find it painful and slow to work with. I experimented with using Nix (as in NixOS) as my build platform which had some great advantages (e.g. great caching of build steps), but also some pain.
* Cost constrained so RAM constrained * Need fast boot times * For a niche processor architecture
Starting with very little apart from an init system and busybox and then building up from there has been a good approach to get the performant (by whatever metric) platform that I need.
That said, I'm not a fan of Yocto at all. I find it painful and slow to work with. I experimented with using Nix (as in NixOS) as my build platform which had some great advantages (e.g. great caching of build steps), but also some pain.
> For a niche processor architecture
It's been ages since I've last seen something else than ARM and x86 - are people still bringing up new designs for MIPS CPUs?!
It's been ages since I've last seen something else than ARM and x86 - are people still bringing up new designs for MIPS CPUs?!
One that comes to mind: https://github.com/nxp-imx/meta-imx
I have done quite a bit of yocto work at $dayjob, and having a vendor-supported upstream BSP is (usually) really pleasant to work with.
I have done quite a bit of yocto work at $dayjob, and having a vendor-supported upstream BSP is (usually) really pleasant to work with.
I have current products using raspberrypi and i.MX parts, and have worked with teams using TI and Renesas parts. Some fully custom boards, some SoM based.
Their BSPs all suck to different degrees, but you can work around the issue with enough effort. If I had to choose ignoring cost, I'd pick an i.MX part, but they're priced for the industrial consumer where price isn't a big concern.
Stay away from any asian parts (Allwinner) or phone parts since they seem to have no vendor or community support.
Like others have said, some of the SoM vendors do a pretty decent job of hiding the evil, but that often comes with very opinionated BSP layers which limit your product design. We've had to ignore/replace all of our SoM vendor's layers, borrowing the few useful parts as needed.
Their BSPs all suck to different degrees, but you can work around the issue with enough effort. If I had to choose ignoring cost, I'd pick an i.MX part, but they're priced for the industrial consumer where price isn't a big concern.
Stay away from any asian parts (Allwinner) or phone parts since they seem to have no vendor or community support.
Like others have said, some of the SoM vendors do a pretty decent job of hiding the evil, but that often comes with very opinionated BSP layers which limit your product design. We've had to ignore/replace all of our SoM vendor's layers, borrowing the few useful parts as needed.
FWIW, some of the Allwinner stuff has fairly robust support in mainline Linux, and there are ongoing efforts to upstream missing bits: https://linux-sunxi.org/Linux_mainlining_effort
No vendor support is probably accurate, though.
No vendor support is probably accurate, though.
NXP is probably the best "big" vendor for this. The main issue is that their documentation solution/website is awful.
I think a smaller vendor like Toradex is more in line with what you're asking for though. I haven't used them myself, but I've heard very good things. First class yocto support is a core part of their offering.
I think a smaller vendor like Toradex is more in line with what you're asking for though. I haven't used them myself, but I've heard very good things. First class yocto support is a core part of their offering.
The Toradex's, Phytec's, Boundary Devices, etc... of the world all do a decent job with their BSP's but just know that they pull from NXP's mainline BSP release so there is a lag in time from when NXP makes a new release. Sometimes they'll fork their own libraries which I've seen cause some problems with application level software from NXP.
I’ve been happy with toradex. Boundary is another similar company but haven’t used them.
Avoid nvidia if you can. Their attitude towards anything Open Source has been extremely unsupportive if not outright shitty. Part of company philosophy.
I've been getting a lot of traction over the past 10 years with Debian multistrap and just the kernel from the BSP. I've never needed Buildroot or Yocto.
I recently got an Nvidia Orin Nano to play with and found Nvidia’s BSP to be very well documented and easy to use… but I haven’t looked into using Buildroot or Yocto directly. https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide...
Yocto is overkill for a hobbyist. It's designed to help big companies build linux distributions for multiple platforms. It works great for companies like NXP that have the resources to devote to it, but it's a f&$cking bear to wrap your hands around if you just want to build linux for a single dev-board.
Buildroot is much smaller/simpler for hobbyists IMO.
NVidia's Jetson lineup is really a double edged (disingenuous IMO) sword. They provide some nice GPU/NPU power for AI stuff and a decent price and is great to play/learn with. However if you want to take a jetson to production you can't. They don't sell the bare SOC's and the modules have limits to how many you can buy. That jetson lineup is probably a 0.05% pimple on their balance sheet...
Buildroot is much smaller/simpler for hobbyists IMO.
NVidia's Jetson lineup is really a double edged (disingenuous IMO) sword. They provide some nice GPU/NPU power for AI stuff and a decent price and is great to play/learn with. However if you want to take a jetson to production you can't. They don't sell the bare SOC's and the modules have limits to how many you can buy. That jetson lineup is probably a 0.05% pimple on their balance sheet...
RPi has a nice community, but some of its drivers still set TAINT_CRAP on Linux.
Isn't the punchline in most of these articles always "Hey so you want to get paid 50% of other people in IT in order to do this field that is 200% harder?"
Because of this mismatch there aren't many young folks with these skills; expect these salaries to flip around once most of the experts retire.
Question for anyone who has done something similar : how feasible would the hardware part be for someone who has some basic electronic / breadboarding knowledge but no pcb design experience? Besides this excellent article what would be a good resource to learn to do something like this myself?
If you want to learn about the mechanics of designing PCBs, install KiCad (https://www.kicad.org/) and go through these tutorials: https://docs.kicad.org/
If you're looking to build a product using an embedded linux system you're better off just using an off the shelf SOM from an embedded board vendor like Toradex or Phytec (to name 2 - there are dozens out there). There are some quasi standards around compute modules now like SMARC and SODIMM where the vendors build little pcb's that contain the SOC, DRAM, PMIC and optional EMMC and WIFI. Then you just built a carrier board to breakout the IO like USB, UART, HDMI, MIPI DSI/CSI-2, etc.... The design/layout/validation of the SOC/PMIC/DRAM is the hardest part and unless you're shipping in volumes > 100ku annually you're probably better off outsourcing the SOM.
If you're just looking to learn then by all means give it a shot! Jay's article is a great primer in the basics but just know that all of those parts are pretty slow/old by today's standards. That article is 4 years old and even 4 years ago those parts were fairly slow/cheap/low-power/low-pincount relative to the SOT at the time.
If you're just looking to learn then by all means give it a shot! Jay's article is a great primer in the basics but just know that all of those parts are pretty slow/old by today's standards. That article is 4 years old and even 4 years ago those parts were fairly slow/cheap/low-power/low-pincount relative to the SOT at the time.
This is advanced level PCB design.
You need to know proper grounding and differential pairs and length matching to accomplish what is listed in this article.
These are very difficult topics, but this 2 hour video should give you an idea of what's going on here: https://www.youtube.com/live/ySuUZEjARPY?si=xfpg8XdjtXa1Uixn
--------
If this is too much for you, then instead stick to microcontrollers and other sub-100Mhz or sub-20Mhz projects.
But 500MHz and above that requires length matching with controlled impedances is waaaayyyyy outside the scope for a beginner designer.
---------
Beginner design is easier at 20MHz or so. The faster your signals, the more susceptible to noise they are. Those 500MHz+ Linux MPUs may fail to boot or fail spuriously if you make grounding / noise / EMC errors.
And it's pretty difficult to explain to a beginner what is going on physically. The PCB board itself needs to be seen as parasitic capacitors, inductors and resistors (especially around Vias, between layers and other elements).
But these 500Mhz or so processors here in this blogpost are still easier than Ghz+ chips.
------
The example I always like to point out: 2.4GHz wavelength is just a couple of cm long. That means any trace of ~centimeters can accidentally turn into a Bluetooth / WiFi antenna, which would inject a ton of noise into your circuit (or conversely: 'Throw' your electrons out like a Bluetooth / WiFi antenna).
That's why faster speeds are harder. 900Mhz and slower needs an antenna over 30cm long, so staying below this means every PCB trace is too short to be an effective antenna.
Going even slower at 20Mhz means many meters before you accidentally make an antenna.
-------
I'd summarize that high speed circuit design is 60%+ just making sure you didn't accidentally make antennas in the frequencies you're working with. And the other 40% is decoupling / turning the PCB into a distributed physical capacitor.
You need to know proper grounding and differential pairs and length matching to accomplish what is listed in this article.
These are very difficult topics, but this 2 hour video should give you an idea of what's going on here: https://www.youtube.com/live/ySuUZEjARPY?si=xfpg8XdjtXa1Uixn
--------
If this is too much for you, then instead stick to microcontrollers and other sub-100Mhz or sub-20Mhz projects.
But 500MHz and above that requires length matching with controlled impedances is waaaayyyyy outside the scope for a beginner designer.
---------
Beginner design is easier at 20MHz or so. The faster your signals, the more susceptible to noise they are. Those 500MHz+ Linux MPUs may fail to boot or fail spuriously if you make grounding / noise / EMC errors.
And it's pretty difficult to explain to a beginner what is going on physically. The PCB board itself needs to be seen as parasitic capacitors, inductors and resistors (especially around Vias, between layers and other elements).
But these 500Mhz or so processors here in this blogpost are still easier than Ghz+ chips.
------
The example I always like to point out: 2.4GHz wavelength is just a couple of cm long. That means any trace of ~centimeters can accidentally turn into a Bluetooth / WiFi antenna, which would inject a ton of noise into your circuit (or conversely: 'Throw' your electrons out like a Bluetooth / WiFi antenna).
That's why faster speeds are harder. 900Mhz and slower needs an antenna over 30cm long, so staying below this means every PCB trace is too short to be an effective antenna.
Going even slower at 20Mhz means many meters before you accidentally make an antenna.
-------
I'd summarize that high speed circuit design is 60%+ just making sure you didn't accidentally make antennas in the frequencies you're working with. And the other 40% is decoupling / turning the PCB into a distributed physical capacitor.
> That's why faster speeds are harder. 900Mhz and slower needs an antenna over 30cm long, so staying below this means every PCB trace is too short to be an effective antenna.
Oh I forgot to mention, the issue is the 5th or 7th harmonic of your digital signals.
So... a 20Mhz digital circuit will realistically have 10ns rise/fall times or aka have ACTUALLY 100Mhz information in your electricity.
Similarly, a 100MHz clock digital circuit will need rise/fall times of 2ns. That's 500Mhz information inside of your signals (2ns == 500MHz).
Following the pattern: 500Mhz digital clock will easily have 2.5Ghz+ information in them to keep those digital signals square. Aka rise / fall times of 0.4ns.
-----------
If you've accidentally built a 2.5 Ghz antenna twice in your circuit, guess what? That 0.4ns rise time square wave is going to start teleporting between your PCB antennas.
Electricity at high frequencies moves in very complex fashions. In ways that beginners cannot comprehend.
We call it crosstalk, EMC and other ways they move around. But it's all antenna theory, EMI fields and the like. I can't say I'm a high wizard of this magic but... Hopefully I've got it close enough for you to understand the issues at play here.
It's best to just slow down to 20MHz where electricity still makes sense.
Oh I forgot to mention, the issue is the 5th or 7th harmonic of your digital signals.
So... a 20Mhz digital circuit will realistically have 10ns rise/fall times or aka have ACTUALLY 100Mhz information in your electricity.
Similarly, a 100MHz clock digital circuit will need rise/fall times of 2ns. That's 500Mhz information inside of your signals (2ns == 500MHz).
Following the pattern: 500Mhz digital clock will easily have 2.5Ghz+ information in them to keep those digital signals square. Aka rise / fall times of 0.4ns.
-----------
If you've accidentally built a 2.5 Ghz antenna twice in your circuit, guess what? That 0.4ns rise time square wave is going to start teleporting between your PCB antennas.
Electricity at high frequencies moves in very complex fashions. In ways that beginners cannot comprehend.
We call it crosstalk, EMC and other ways they move around. But it's all antenna theory, EMI fields and the like. I can't say I'm a high wizard of this magic but... Hopefully I've got it close enough for you to understand the issues at play here.
It's best to just slow down to 20MHz where electricity still makes sense.
This is amazing, thanks for the breakdown. I guess this article makes it seem easier than it is. I will stick to SoM for linux for my projects for now but definitely going to try some stm32g0 and the like custom boards
You can and should be practicing good 4 layer (or even 6 layer) boards even with 'easy low frequency' STM32G0 chips.
That way, you build up your habits but in a non-punishing manner.
-----------
STM32G0 will work on a breadboard, let alone a properly designed well grounded PCB.
But... You can still see the EMI through the use of a loop antenna-oscilloscope.
Simply attach the ground-alligator clip to the probe top while setting your oscilloscope to AC coupling mode. Zoom in as much as you can on your oscilloscope.
Now wave this 'loop' around various circuits and traces. You'll see the EMI noise coming off of bad circuits... But only within 5cm or less. (loop antennas can pickup the so called near fields very well).
Alas: proper EMI testing is about far fields, not near fields. Still, it's good to have a methodology to actually see, feel and touch EMI in your projects.
--------
Getting a physical feel of electricity for real (and not the fake electricity we pretend happens in resistor/capacitor/inductor lumped element models) is the important leap you need to start learning.
Making consistent PCB designs and measuring EMI off of them is a great way to learn and improve.
You probably only need to make 4 or 5 designs while thinking about these issues to really understand. There is enough study material to give you the gist of theory... And the EMI loop antenna oscilloscope gives you a good enough measure (albeit flawed but still a real live measurement).
That way, you build up your habits but in a non-punishing manner.
-----------
STM32G0 will work on a breadboard, let alone a properly designed well grounded PCB.
But... You can still see the EMI through the use of a loop antenna-oscilloscope.
Simply attach the ground-alligator clip to the probe top while setting your oscilloscope to AC coupling mode. Zoom in as much as you can on your oscilloscope.
Now wave this 'loop' around various circuits and traces. You'll see the EMI noise coming off of bad circuits... But only within 5cm or less. (loop antennas can pickup the so called near fields very well).
Alas: proper EMI testing is about far fields, not near fields. Still, it's good to have a methodology to actually see, feel and touch EMI in your projects.
--------
Getting a physical feel of electricity for real (and not the fake electricity we pretend happens in resistor/capacitor/inductor lumped element models) is the important leap you need to start learning.
Making consistent PCB designs and measuring EMI off of them is a great way to learn and improve.
You probably only need to make 4 or 5 designs while thinking about these issues to really understand. There is enough study material to give you the gist of theory... And the EMI loop antenna oscilloscope gives you a good enough measure (albeit flawed but still a real live measurement).
I found this to be a great resource when I was learning PCB design:
https://www.youtube.com/@PhilsLab
I watched all his KiCad videos and tried to build them as I went along. Then you can use a cheap PCB manufacturer like JLCPCB to get them made. It's a lot cheaper than you think (tens of dollars delivered in two weeks for simple stuff). There is a lot of know-how to learn in this field but I found it to be very accessible as a SWE and it all made sense. It just took a long time to pick it all up.
I watched all his KiCad videos and tried to build them as I went along. Then you can use a cheap PCB manufacturer like JLCPCB to get them made. It's a lot cheaper than you think (tens of dollars delivered in two weeks for simple stuff). There is a lot of know-how to learn in this field but I found it to be very accessible as a SWE and it all made sense. It just took a long time to pick it all up.
Embedded systems question. I have an nVidia Jetson, and I want to expose one of the USB3 ports to the user of the system (e.g. so they can attach a USB drive). However ... the USB3 port of the Jetson is not near the enclosure of the box. Now I'm thinking of using a USB3 cable, but I know from experience that this is not going to give a reliable connection. Any suggestions on how to properly do this? I don't want to move the Jetson, in part because I am planning to use the other exposed ports internally.
Can you check if there is another USB port exposed via a pin header? That is probably the easiest way if it is available. Otherwise, I think your best best is to solder wires onto the same pads used by the existing USB port. I wouldn’t try to desolder the port that is there, that could cause more problems. But if you can find a location on the board to tap into that USB port’s signals, you could solder wires to run another usb port using the same connections.
I don't think any USB3 ports are exposed on the headers.
But I also have to wonder how that would improve the situation over running a cable between the actual USB3 port and a socket mounted in the enclosure.
But I also have to wonder how that would improve the situation over running a cable between the actual USB3 port and a socket mounted in the enclosure.
[deleted]
[2020]
Discussion from 2020: https://news.ycombinator.com/item?id=24800037
Also, another in 2022: https://news.ycombinator.com/item?id=30273301
Also, another in 2022: https://news.ycombinator.com/item?id=30273301
Which SoM product line would you say has the best vendor support? I'm talking about quality of the BSP (board support package). (A meta-layer, in Yocto terms.)
Raspberry Pi has quite a community behind it, so meta-raspberrypi has a number of contributors. (None of which payed by the Raspberry Pi foundation, though.)
My latest project involved a NVidia Jetson SoM, and I was surprised to learn that the BSP doesn't see any support from NVidia at all. They rely on one motivated guy (Matt Madison) maintaining it in his free time.
I'd love to see a vendor that provides a first-class BSP maintained by their own people.