Shrinking the kernel with a hammer(lwn.net)
lwn.net
Shrinking the kernel with a hammer
https://lwn.net/SubscriberLink/748198/6fa400001570fb39/
31 comments
> these days imagining running linux in 4MB makes me sniggers.
Did you read the article?
> Here it is! Not exactly our target of 512KB of RAM but 768KB is getting pretty close. Some microcontrollers already have more than that amount of available on-chip SRAM.
768 KB of RAM, plus a MB or two for the text sections which are in flash, is not bad. Not quite at the target, a system with 512 KB of ram and 2 MB of flash, but it's close.
Did you read the article?
> Here it is! Not exactly our target of 512KB of RAM but 768KB is getting pretty close. Some microcontrollers already have more than that amount of available on-chip SRAM.
768 KB of RAM, plus a MB or two for the text sections which are in flash, is not bad. Not quite at the target, a system with 512 KB of ram and 2 MB of flash, but it's close.
Hence why ARM mbed, RTOS, NuttX and now TinyAra (yet another Tizen effort) are fighting for the tiny embedded devices.
My first distribution was Slackware 2.0 running on a Pentium 75 MHz, with 4MB RAM and 512 MB HDD, long long time ago.
My first distribution was Slackware 2.0 running on a Pentium 75 MHz, with 4MB RAM and 512 MB HDD, long long time ago.
Compiling the kernel on a 486 took a freakishly long time, but it was worth it - it gave you more RAM to play with.
Once I upgraded to PII I could comfortably compile it in like 20 minutes or so. That was nice.
I think I knew the ncurses kernel config interface by heart - it's only awkward to use during the first 100 times or so.
Once I upgraded to PII I could comfortably compile it in like 20 minutes or so. That was nice.
I think I knew the ncurses kernel config interface by heart - it's only awkward to use during the first 100 times or so.
Slackware came with two 3.5" disks a boot disk and root disk. The root disk contained the root filesystem, he boot disk the kernel. At that time the kernel fit onto a single 3.5", 1.44 MiB floppy disk.
Heavily compressed and carefully pruned, it is still possible to squeeze modern (4.9) kernel into under 1.44MiB, and kernel+rootfs into 3-ish MiB:
http://downloads.lede-project.org/releases/17.01.4/targets/i...
http://downloads.lede-project.org/releases/17.01.4/targets/i...
I made my own distro back then. Started from source with everything. It fit on a 3.5" floppy, and had room for an IPX client that could download stuff from a fileserver, after formatting the local drive (it was an automated reinstaller for a computer lab when kids trashed the Win16 computers).
That's how I learned the whole boot chain diagram - hands on, trial and error.
That's how I learned the whole boot chain diagram - hands on, trial and error.
I remember a linux distro that pushed a 2.3 (or 2.1?) kernel and some tools on to two 1.44 MB disks. In involved booting dos, creating a ram-disk, decompressing with them some slow and esoteric dos program and then boot into the initrd with linload.
Cramping linux onto a floppy has been a challenge for as long as i use linux.
Cramping linux onto a floppy has been a challenge for as long as i use linux.
Yep, I got it on a CD-ROM, but since my drive was IDE based it wasn't reckognized by the basic boot loader.
I had to copy the tarballs into the MS-DOS partition, boot with those floppies you mention, and then at a given moment give the path to that directory.
I had to copy the tarballs into the MS-DOS partition, boot with those floppies you mention, and then at a given moment give the path to that directory.
I remember the days of 'make zimage' complaining that the resulting kernel was too big for a floppy disk and that I should use 'make bzimage' instead.
Getting down to around 6 MB on modern ARM kernel + minimal userspace is doable at least.
> these days imagining running linux in 4MB makes me sniggers.
FWIW - OpenWRT / LEDE has support for a lot of wifi routers with 4MB flash that even have a webserver and lua and some other goodies - at the moment kernel 4.9 but 4.14 is coming and here device tree will even save some space.
Well you need 32mb RAM - it runs with some hacking on 16MB ram - with execute in place probably with less...
1: https://downloads.lede-project.org/snapshots/targets/ar71xx/...
FWIW - OpenWRT / LEDE has support for a lot of wifi routers with 4MB flash that even have a webserver and lua and some other goodies - at the moment kernel 4.9 but 4.14 is coming and here device tree will even save some space.
Well you need 32mb RAM - it runs with some hacking on 16MB ram - with execute in place probably with less...
1: https://downloads.lede-project.org/snapshots/targets/ar71xx/...
Yeah XIP is the only way to go for small config these days. There's some nice QSPI controllers that helps there, however, it's still such a waste.
What's forgotten about is the impact on readability in the kernel...
Last week I was discussing with a colleague a patch needed for our SoC in the 8250 serial driver. In the IRQ handler, we needed to do an extra test...
And my problem was that this test would be running on a bazillion machines if upstreamed, for about 0.000000001% chance of ever be executed, bar for our own low volume, industrial SoC. We'd add cycles to a critical code path for zero reason, and without conditional compile, nobody would know by reading the code.
So, after mulling over it, we decided to get it upstream anyway at some point. The conclusion was that we didn't make these rules, we're just following them.
Thing is, a lot of that stuff is happening; you have that vendor specific property that is tested, change a flag somewhere, and code all over use that flag for testing of features that have 0% chance of happening on that arch/machine. In the meantime, all that becomes code bloat, with codepath that are actually 'exceptions' bloating the code in a way you can't even 'see' them by reading the code.
Sad?
</rant>
What's forgotten about is the impact on readability in the kernel...
Last week I was discussing with a colleague a patch needed for our SoC in the 8250 serial driver. In the IRQ handler, we needed to do an extra test...
And my problem was that this test would be running on a bazillion machines if upstreamed, for about 0.000000001% chance of ever be executed, bar for our own low volume, industrial SoC. We'd add cycles to a critical code path for zero reason, and without conditional compile, nobody would know by reading the code.
So, after mulling over it, we decided to get it upstream anyway at some point. The conclusion was that we didn't make these rules, we're just following them.
Thing is, a lot of that stuff is happening; you have that vendor specific property that is tested, change a flag somewhere, and code all over use that flag for testing of features that have 0% chance of happening on that arch/machine. In the meantime, all that becomes code bloat, with codepath that are actually 'exceptions' bloating the code in a way you can't even 'see' them by reading the code.
Sad?
</rant>
> And my problem was that this test would be running on a bazillion machines if upstreamed, for about 0.000000001% chance of ever be executed, bar for our own low volume, industrial SoC. We'd add cycles to a critical code path for zero reason, and without conditional compile, nobody would know by reading the code.
That's a good reason why C++ is generally superior to C in embedded development: with C++, you can get at the same time compiler checks for the validity of your whole code to prevent bitrot (unlike macros), and ensure that the actual implementations will only have the actual code that is relevant to them - e.g. set your flags as constexpr variables in your device's struct, and have the tests run under `if constexpr` conditions. Devices without the flag set won't have the check compiled in, and others will. But oh well :)
That's a good reason why C++ is generally superior to C in embedded development: with C++, you can get at the same time compiler checks for the validity of your whole code to prevent bitrot (unlike macros), and ensure that the actual implementations will only have the actual code that is relevant to them - e.g. set your flags as constexpr variables in your device's struct, and have the tests run under `if constexpr` conditions. Devices without the flag set won't have the check compiled in, and others will. But oh well :)
One example where XIP would be really useful is MT7688KN [1], which is a variant of MT7688 WiFi router SoC, already well supported by Linux/OpenWRT, plus 8MB of RAM on the same package.
Since no external RAM is required this makes for some very compact WiFi repeaters [2,3]. But all of them ship with VxWorks. I've tried to tinker with OpenWrt on this chip, but with 2-3MB free RAM after kernel boots (and that's before wifi modules are loaded), it doesn't look too good. With XIP, on the other hand, MT7688KN can potentially become a useful OpenWRT platform.
[1] https://www.mediatek.com/products/smartHome/mt7688k
[2] https://fccid.io/2AK8V-WF8300
[3] Xaiomi Mi Wifi+
Since no external RAM is required this makes for some very compact WiFi repeaters [2,3]. But all of them ship with VxWorks. I've tried to tinker with OpenWrt on this chip, but with 2-3MB free RAM after kernel boots (and that's before wifi modules are loaded), it doesn't look too good. With XIP, on the other hand, MT7688KN can potentially become a useful OpenWRT platform.
[1] https://www.mediatek.com/products/smartHome/mt7688k
[2] https://fccid.io/2AK8V-WF8300
[3] Xaiomi Mi Wifi+
XIP was a thing with PCMCIA 2.1 25 years ago, but then it became more and more of a niche solution. I like to blame the ascent of Linux/x86 for that, at least in part.
You're being too dismissive IMO. Do you really want the linux kernel to focus mainly on 4MB picture frames at the expense of everybody else? As an embedded dev I very much welcomed the device tree, having to create a billion variations of the same BSP to support various configurations was not fun. "Conditional code" is more prone to bitrot than anything else since you have to test a bunch of different configurations if you want to make sure that it even still compiles correctly.
Besides you're still free to make your own single purpose micro-optimized kernel with only the drivers you want if you feel like it.
I can believe that today's kernel is harder to get to work decently on a 4MB picture frame than in the past. On the other hand it's a lot easier to work with modern embedded SoCs. I think it's a worthy trade-off and a very good pragmatical decision.
Besides you're still free to make your own single purpose micro-optimized kernel with only the drivers you want if you feel like it.
I can believe that today's kernel is harder to get to work decently on a 4MB picture frame than in the past. On the other hand it's a lot easier to work with modern embedded SoCs. I think it's a worthy trade-off and a very good pragmatical decision.
I actually disagree.
I will take 100% of the time a kernel that /doesn't compile/ because an option has been set that had 'bitrotted' than a kernel that compiles with the same code in a "if (my device tree flag)" block that in fact is equally borken but you'll never know, until days/weeks of debugging to figure out that bit was rotten, and nobody had tested it for years until you.
And in the meantime, that code was rotten, and rolled in in every phone and servers on the planet for absolutely no reason.
At least, when you decide that CONFIG_ARCH_TOASTER is no longer supported, you CAN trim the code out easily. You can't do that with code that is living in shadows.
I will take 100% of the time a kernel that /doesn't compile/ because an option has been set that had 'bitrotted' than a kernel that compiles with the same code in a "if (my device tree flag)" block that in fact is equally borken but you'll never know, until days/weeks of debugging to figure out that bit was rotten, and nobody had tested it for years until you.
And in the meantime, that code was rotten, and rolled in in every phone and servers on the planet for absolutely no reason.
At least, when you decide that CONFIG_ARCH_TOASTER is no longer supported, you CAN trim the code out easily. You can't do that with code that is living in shadows.
Code that's dynamically disabled is at least checked by the compiler, you can detect a whole bunch of errors that way (function prototypes changing, using deprecated APIs, naming clashes etc...). Of course bitrot will always happen if code is never tested but code that's never actually built will rot much, much faster in my experience.
Then some day one poor sod actually wants to use the option, he gets 4 pages of compilations error and decides maybe he didn't need that after all. Code that compiles doesn't necessarily work but it's a start...
>At least, when you decide that CONFIG_ARCH_TOASTER is no longer supported, you CAN trim the code out easily. You can't do that with code that is living in shadows.
I suppose you're right but again, I feel like you're optimizing for the wrong thing. For one thing the linux kernel is not too keen on removing stuff in my experience, in general older APIs coexist with newer ones and board support is not dropped willy-nilly.
Beyond that I don't know if I should trust my judgement over yours but I'm sure I'll trust the kernel maintainer's judgement over both of ours.
Then some day one poor sod actually wants to use the option, he gets 4 pages of compilations error and decides maybe he didn't need that after all. Code that compiles doesn't necessarily work but it's a start...
>At least, when you decide that CONFIG_ARCH_TOASTER is no longer supported, you CAN trim the code out easily. You can't do that with code that is living in shadows.
I suppose you're right but again, I feel like you're optimizing for the wrong thing. For one thing the linux kernel is not too keen on removing stuff in my experience, in general older APIs coexist with newer ones and board support is not dropped willy-nilly.
Beyond that I don't know if I should trust my judgement over yours but I'm sure I'll trust the kernel maintainer's judgement over both of ours.
In my codebase we put our configs behind if (CONFIG_TOASTER) { ... } statements, and then #define CONFIG_TOASTER true/false.
I think this is a pretty good middle ground, where we still get compile errors to help against code rot, but also keeps code needed for special configs disabled/isolated.
I think this is a pretty good middle ground, where we still get compile errors to help against code rot, but also keeps code needed for special configs disabled/isolated.
> Code that's dynamically disabled is at least checked by the compiler
You can still trivially ensure that kernel with CONFIG_ARCH_TOASTER compiles fine by setting up a build server which automatically checks each and every option.
You can still trivially ensure that kernel with CONFIG_ARCH_TOASTER compiles fine by setting up a build server which automatically checks each and every option.
You don't just need to check any single option, you also need to check every combination of options that might reasonably get used together to make sure that it's handled properly. Given the size of the kernel and its thousands of configuration options testing all viable combinations is not reasonable.
> focus mainly on 4MB picture frames at the expense of everybody else
That seems like a pretty obvious false dichotomy. More modular architectures would be much more flexible.
That seems like a pretty obvious false dichotomy. More modular architectures would be much more flexible.
Read the rest of my comment, the Linux kernel is pretty damn modular as it is. Nobody forces anybody to compile all drivers ever written built-in to be able to load them from the device tree. But if you want to do it, you can, and that's great.
If you don't you can still toggle your individual drivers and modules and compile a lightweight kernel. I think what the parent complains about is that the kernel is too dynamically flexible and that adds a certain footprint, in particular in terms of code size. That might be true but again, I think it's a good compromise.
The reason execute-in-place is not super well maintained in the modern kernel isn't because the Linux devs are javascript developpers who want to break everything and reinvent the wheel every month, it's because it's getting rarer and rarer to develop linux-enabled embedded hardware using memory-mapped NOR as main storage so the focus moved elsewhere. You might as well complain about MMU-less systems not being properly supported.
If you don't you can still toggle your individual drivers and modules and compile a lightweight kernel. I think what the parent complains about is that the kernel is too dynamically flexible and that adds a certain footprint, in particular in terms of code size. That might be true but again, I think it's a good compromise.
The reason execute-in-place is not super well maintained in the modern kernel isn't because the Linux devs are javascript developpers who want to break everything and reinvent the wheel every month, it's because it's getting rarer and rarer to develop linux-enabled embedded hardware using memory-mapped NOR as main storage so the focus moved elsewhere. You might as well complain about MMU-less systems not being properly supported.
I agree with the 'make your own little kernel' for embedded. supporting those parts of workalike posix for the parts you need and including lwip is really much easier than trying to surgically remove the bulk of linux
Would it be a good idea to have major kernel forks targeting different sizes? Say for example {kernel_tiny, kernel_small, kernel} targeting {<4MB,<100MB,<inf} respectively. Although as you mention I'm not sure it is really a wise choice to use Linux for really small and simple systems.
Such project already exists, even somewhat officially: https://tiny.wiki.kernel.org/
I still have 6x USR2450, one of the first router where you could flash Linux (2.4.20). It had 4MB RAM and 1MB flash, 10mbps, rs232, and it was quite some sport to add a feature there.
You could run LinuxAP on there:
https://github.com/zoobab/linuxAP-eh
You could run LinuxAP on there:
https://github.com/zoobab/linuxAP-eh
This has a Nieztschean tint. When you have a hammer, everything looks like a Linux 4.2.0 kernel, erm, nail.
How small did it get? But hard to find.....
Previous installments in the series were about shrinking the file size. https://lwn.net/Articles/741494/ https://lwn.net/Articles/744507/ https://lwn.net/Articles/746780/ This one is about shrinking RAM usage.
I think the big vendors are to blame, they've been dreaming about a single kernel binary that can run on ANY board, with a magic device tree, it suits their own way of 'distributing' SDKs, and it has transformed the kernel from something that you COULD tune to a platform/board to one that is /actively/ discouraged to be tuned to a platform.
A while back a couple of friends and I reverse engineered a 4MB picture frame [0], and had linux and doom running on it, for the lulz. That was kernel 2.6; these days imagining running linux in 4MB makes me sniggers.
[0]: https://sites.google.com/site/repurposelinux/df3120