Linux from Scratch with Training Wheels(philsyme.github.io)
philsyme.github.io
Linux from Scratch with Training Wheels
https://philsyme.github.io/lfs-tw/
42 comments
I don’t know why but I find this so interesting. I totally don’t have time right now (with a young kid and trying to make some/any progress in my career as a dev now that I’ve successfully made the career change to working as a developer), but I really just like understanding how things work from the ground up - I can’t wait to have the time to build my own Linux distribution. It feels like it’ll be such a waste of time - like installing 5 or 6 different distros on an old PC I had just out of curiosity to see what they looked snd felt like. And I broke and fixed the boot loader so many times that it became kind of an enjoyable challenge getting the machine to work again - “ok, now how did I get grub2 to recognise where my partitions are…?” It seemed like spinning my wheels but I just really enjoyed it. I don’t know why. Kind of reminds me of a comment someone made here on HN about Linux a while ago - something to the effect of “I want an OS I do t want a hobby.” Except I enjoy messing around with it and spending time making it work. Not very productive though I admit.
I tried LFS a long time ago, and frankly I don't consider it much of an educational experience --- downloading, compiling, and installing a bunch of stuff repeatedly with small variations of the same commands just doesn't make you learn much.
On the other hand, I recommend trying to make a minimal system with only a kernel and shell and then add to it as desired. That way you'll really get an idea of what's needed to do what (and realise how bloated common distros are).
A long time ago, I went through "DOS from scratch" and "Windows from scratch" in a similar fashion.
On the other hand, I recommend trying to make a minimal system with only a kernel and shell and then add to it as desired. That way you'll really get an idea of what's needed to do what (and realise how bloated common distros are).
A long time ago, I went through "DOS from scratch" and "Windows from scratch" in a similar fashion.
If you follow the guide step-by-step, like an interpreter evaluating an LFS build script, you’ll probably not get much out of LFS.
LFS, KISS and, most recently FreeBSD were extremely educational for me. They showed me what each piece of my system was for, where to look when things went wrong, where to look when I wanted to tweak something, etc.
For a modern Linux installer, liked Debian or Ubuntu, I feel like I’m looking off a cliff face into an abyss. They say it’s “free as in freedom” - and I know it’s technically true - but I don’t feel like I’m equipped to understand the system the installer produced without a decade investment of time. The way I’d interact with those systems was a lot like “free as in free beer” - I knew I _could_ fix it if something didn’t work quite right, but frankly I didn’t know _how_ to fix it or where to start. So I just consumed it like I would proprietary software; it was a big black box I never looked inside.
On KISS, I ported the base system to a raspberry PI as my first task - the system was simple enough that I could just tackle that out the gate.
For LFS I had a lightbulb moment - understanding what all those packages that scrolled by in an “apt” install were about, how they fit together, and why a distro needed “patches.” It taught me what a distribution _was_ and the work that went into building and maintaining one. It taught me what the “base system” ecosystem consisted of.
With FreeBSD ports, by day 3, I was floating my own patch on top of my window manager because the config file didn’t have a setting I wanted.
As a sibling said, you get out what you put in.
LFS, KISS and, most recently FreeBSD were extremely educational for me. They showed me what each piece of my system was for, where to look when things went wrong, where to look when I wanted to tweak something, etc.
For a modern Linux installer, liked Debian or Ubuntu, I feel like I’m looking off a cliff face into an abyss. They say it’s “free as in freedom” - and I know it’s technically true - but I don’t feel like I’m equipped to understand the system the installer produced without a decade investment of time. The way I’d interact with those systems was a lot like “free as in free beer” - I knew I _could_ fix it if something didn’t work quite right, but frankly I didn’t know _how_ to fix it or where to start. So I just consumed it like I would proprietary software; it was a big black box I never looked inside.
On KISS, I ported the base system to a raspberry PI as my first task - the system was simple enough that I could just tackle that out the gate.
For LFS I had a lightbulb moment - understanding what all those packages that scrolled by in an “apt” install were about, how they fit together, and why a distro needed “patches.” It taught me what a distribution _was_ and the work that went into building and maintaining one. It taught me what the “base system” ecosystem consisted of.
With FreeBSD ports, by day 3, I was floating my own patch on top of my window manager because the config file didn’t have a setting I wanted.
As a sibling said, you get out what you put in.
For a modern Linux installer, liked Debian or Ubuntu, I feel like I’m looking off a cliff face into an abyss. They say it’s “free as in freedom” - and I know it’s technically true - but I don’t feel like I’m equipped to understand the system the installer produced without a decade investment of time. The way I’d interact with those systems was a lot like “free as in free beer” - I knew I _could_ fix it if something didn’t work quite right, but frankly I didn’t know _how_ to fix it or where to start. So I just consumed it like I would proprietary software; it was a big black box I never looked inside.
I often feel that way about FOSS in general. The freedom is more 'theoretical' than practical, although FOSS means that such software are much less hostile to users.
I appreciate repairable laptops and electronics more than I appreciate software freedom. The freedom felt more real.
I often feel that way about FOSS in general. The freedom is more 'theoretical' than practical, although FOSS means that such software are much less hostile to users.
I appreciate repairable laptops and electronics more than I appreciate software freedom. The freedom felt more real.
You can approach software repair in a similar way to electronics repair, except you don't need to replace components, just fix them. Investigate which component is causing the issue, investigate which part of the component is causing the issue, modify/rebuild/reinstall the component, verify that the fix worked, done.
I had the same experience with LFS, seeing under the hood and putting it into practice gave me a deeper appreciation and understanding of what's going on. Plus I got to put it into practical use when I had to chroot into a borked Ubuntu install to install the proprietary nvidia driver that for whatever reason didn't get installed in the normal process (but the live environment automatically loaded it no problem).
Strangely my last moment of shock is not trying bare metal linux, beneater 6502, z80 Pcm not to mention Hercules ibm …etc as there are still too many components. As said of it does not work not sure I know how to try. Even fixing grub2 does not help.
But recently I found I am using boot camp and that blow hot air around you forced me to find a solution. On macOS side I pay for a patch … totally expect to pay but then it is just one setting (no 100% cpu max 99%, but not help much) and then a setting patch on not using turbo. That is it. It is a shock. Can you just manipulate all these without programming? And without payment? Just tinker a bit and take the risk to do a windows registry setting …
As open source has indicated, you can learn from both and try to mix them. And as free as liberty, non free beer can still carry some message.
But recently I found I am using boot camp and that blow hot air around you forced me to find a solution. On macOS side I pay for a patch … totally expect to pay but then it is just one setting (no 100% cpu max 99%, but not help much) and then a setting patch on not using turbo. That is it. It is a shock. Can you just manipulate all these without programming? And without payment? Just tinker a bit and take the risk to do a windows registry setting …
As open source has indicated, you can learn from both and try to mix them. And as free as liberty, non free beer can still carry some message.
You get out of it what you put into it. Don't just follow the instructions, find out what everything is and what it does, why the instructions are what they are and why they are in the order they are. Maybe learn make and autotools. And for the person with way too much time on their hands, read the manual for each. For the retired or student on break, read some of the source code.
If you're a programmer, getting a simple C++ or Rust "init system" working is a great exercise to understand how a Linux system works. Just a kernel image, your C++/Rust init program, and maybe busybox so you have a shell.
[deleted]
Every time I've think about doing LFS, I come to the same conclusion that Arch is probably good enough to get the same hands-on understanding but without having to do the menial work.
For a great low-commitment experiment with minimalistic Linix, I'd recommend setting up PRoot with Termux on Android and then adding Arch repositories. You can treat it like a replacement for a mobile game and slowly build up your perfect mobile desktop environment.
https://wiki.termux.com/wiki/PRoot
https://wiki.termux.com/wiki/PRoot
For me it was Slackware.
Slackware was my first love, but the reason I say Arch is because there's no package management
... though these days, Nix on top of Slackware may be interesting
... though these days, Nix on top of Slackware may be interesting
I find Slackware easier to install.
There is slackpkg for package management, and sbopkg for “ports.” (What is “missing” is dependency management, which often turns out to be a blessing.)
(pkgsrc also supports Slackware.)
There is slackpkg for package management, and sbopkg for “ports.” (What is “missing” is dependency management, which often turns out to be a blessing.)
(pkgsrc also supports Slackware.)
I remember some ~20 years ago, only going thru their HOWTO was very revealing about the system innards and I remember having a blast. I never went trough a real LFS install because I lacked an extra box at the time, but it is something the lingered on the back of my head since.
> I don’t know why but I find this so interesting
> Not very productive though I admit.
It's called "scratching an itch". You've no idea why you've got that itch, but it sure is satisfying giving it a good scratch.
But what seems like, for the present, idle fiddling about, sticks in your memory and one day in the future some problem (or interview question) will rear it's head and you'll thank yourself for just messing around. Treat it as a positive trait.
> Not very productive though I admit.
It's called "scratching an itch". You've no idea why you've got that itch, but it sure is satisfying giving it a good scratch.
But what seems like, for the present, idle fiddling about, sticks in your memory and one day in the future some problem (or interview question) will rear it's head and you'll thank yourself for just messing around. Treat it as a positive trait.
I have alway preferred syslinux over grub/grub2.
I am not really into chainloading or booting from different partitions. Keep it as simple as possible. KIASAP.
I want an OS I control. If that requires time, a one-time sunk cost to learn how to do something, like sports/hobbies require, it is worth it.
I am not really into chainloading or booting from different partitions. Keep it as simple as possible. KIASAP.
I want an OS I control. If that requires time, a one-time sunk cost to learn how to do something, like sports/hobbies require, it is worth it.
> I totally don’t have time right now
It takes too much time for compiling. Some packages compiled overnight on my machine. Now I thank my distribution, that they precompile these packages for me :)
It takes too much time for compiling. Some packages compiled overnight on my machine. Now I thank my distribution, that they precompile these packages for me :)
I’ve done LFS, and I recommend it, but another approach is to read the script for the basic Alpine installer and then work through it to build a system.
For those with the interest, but with less time or patience, I found that installing Arch on a "For messing around, I don't care if I accidentally brick it for a week" laptop was a pretty good way to quickly learn a lot more about how a typical Linux system is assembled.
There's no installer... the boot media just drops you into a live console, and then you're expected to follow the installation guide [1] to set up your disk, bootstrap the core system, select packages, and then do just enough manual configuration that your new machine will survive a reboot.
You'll probably spend the next week or so digging around the Arch Wiki [2] to figure out how to set up the remaining odds and ends. You're definitely not building the system from scratch, but you generally do need to understand more about where things live, how they're configured, and how to turn them on and off, which the wiki is very good at explaining. The stuff you learn here will absolutely come in handy when the same component is misbehaving on a Debian or Red Hat system.
And when you're done, not only are you left with a system that's completely usable (i.e. packages that can actually be upgraded), it'll be configured exactly the way you wanted. In my case, that "messing around" laptop is now my daily driver.
[1] https://wiki.archlinux.org/title/Installation_guide
[2] https://wiki.archlinux.org/title/Main_page (a great resource for Linux in general, not only Arch specific things)
There's no installer... the boot media just drops you into a live console, and then you're expected to follow the installation guide [1] to set up your disk, bootstrap the core system, select packages, and then do just enough manual configuration that your new machine will survive a reboot.
You'll probably spend the next week or so digging around the Arch Wiki [2] to figure out how to set up the remaining odds and ends. You're definitely not building the system from scratch, but you generally do need to understand more about where things live, how they're configured, and how to turn them on and off, which the wiki is very good at explaining. The stuff you learn here will absolutely come in handy when the same component is misbehaving on a Debian or Red Hat system.
And when you're done, not only are you left with a system that's completely usable (i.e. packages that can actually be upgraded), it'll be configured exactly the way you wanted. In my case, that "messing around" laptop is now my daily driver.
[1] https://wiki.archlinux.org/title/Installation_guide
[2] https://wiki.archlinux.org/title/Main_page (a great resource for Linux in general, not only Arch specific things)
To be fair Arch comes with a lot of prebuilt and preconfigured stuff plus utilities, abstracting A LOT away from you. Prime example is getting a desktop to run: For example install the gnome package, enable gdm and install the driver package for your GPU (xf86-video-whatever) and ... you are done. Having dealt with LFS and "back in the day"-BSDs that's usually a real pain to do. Sure, you need to install the system by "hand" (pacstrap, cough) and have to configure this and that (like redirecting genfstab output, cough), but it's far far far away to be remotely from scratch. Sure, you learn more than through the average Ubuntu-like installer, but how modules work or that they even exist is something even the most Arch users don't know.
They now have an optional experimental installer https://wiki.archlinux.org/title/Archinstall
My issue with LFS (and I've attempted it several times, and always ran out of time to complete it) is that most of the time is spent compiling all the source code.
What would be easier is a step-by-step guide to 'building' the system, via fdisk onto a spare HDD, and then manually copying over all the kernel bits , stuff for a working shell etc. and editing the init scripts with explanations of what each bit does, and why you are adding them to the system.
I've read lots of docs on the structure and boot process of Linux, but stepping through adding each element to blank disk would cement in the knowledge, without all the lost time that LFS adds.
What would be easier is a step-by-step guide to 'building' the system, via fdisk onto a spare HDD, and then manually copying over all the kernel bits , stuff for a working shell etc. and editing the init scripts with explanations of what each bit does, and why you are adding them to the system.
I've read lots of docs on the structure and boot process of Linux, but stepping through adding each element to blank disk would cement in the knowledge, without all the lost time that LFS adds.
Alternatively, and a lot more "real", would be if the system compiler available during the step-by-step phase cared foremost about getting functioning binaries out quickly. Bootstrap your way throughout the day with "slow" binaries built with the fast compiler and keep a log of this activity, then you feed this spec into a batch tool that runs overnight and uses the industrial strength compiler to recompile whatever it can before you wake up, interrupt it, and proceed with your work the next day.
In fact, this has applications beyond what's involved in bootstrapping system software for self-education. It's very interesting that all the vendors of expensive compilers reacted to the production-readiness of GCC in the 90s by choosing to wither and die instead of focusing on improving the notoriously bad developer experience of the edit-compile-test cycle for large C and C++ to justify the the purchase of their products.
In fact, this has applications beyond what's involved in bootstrapping system software for self-education. It's very interesting that all the vendors of expensive compilers reacted to the production-readiness of GCC in the 90s by choosing to wither and die instead of focusing on improving the notoriously bad developer experience of the edit-compile-test cycle for large C and C++ to justify the the purchase of their products.
Did LFS twice, I left with a broken system I couldn't understand but many hours of sed patch [vaguely] useful syntax mastery (I tried not copy paste the commands and typed them on my own)
No disrespect, but this is backward.. it was fun to see the network stack acting chaotic. IRC worked fine, HTTP did not. Terminal had color issues sometimes. Made me appreciate the intricacies but left me totally clueless about why; which I thought was the point of assembling an OS bit by bit.
No disrespect, but this is backward.. it was fun to see the network stack acting chaotic. IRC worked fine, HTTP did not. Terminal had color issues sometimes. Made me appreciate the intricacies but left me totally clueless about why; which I thought was the point of assembling an OS bit by bit.
LFS is good for learning, but if you want a long term maintainable solution, use Gentoo instead.
(Disclaimer: 18 year Gentoo user speaking)
(Disclaimer: 18 year Gentoo user speaking)
Gentoo is great, but similar to LFS it is kind of a time suck. I migrated from Arch to Gentoo and it took me a while to figure out the right kernel parameters and USE flags for my system.
Oh that poor laptop, spent days and days compiling Gentoo.
I was just mesmerized by the scrolling compiler messages, amazed to see something that complex transform into a working system.
And once it was done I usually found a reason to start over again, just one more flag to make it perfect.
Oddly, I can't remember ever actually using Gentoo, just compiling.
I was just mesmerized by the scrolling compiler messages, amazed to see something that complex transform into a working system.
And once it was done I usually found a reason to start over again, just one more flag to make it perfect.
Oddly, I can't remember ever actually using Gentoo, just compiling.
I vividly remember compiling KDE (12-24 hours) and Firefox (3+ hours) back in like 2006. The hype and complains about Gentoo’s compilation times were legendary back then.
I used Gentoo as a daily driver back in early college (2002-4)-ish.
Was a lot of fun, but a huge amount of time compiling
Every time I decided to try a new window manager and update USE flags to do something like add GTK support for Gnome, or QT for KDE, and obviously remove whatever I had to start with for optimization it would recompile everything. I remember in particular doing something like a fresh Gnome install, starting to use it and realizing I'd missed a USE flag for audio and having to re-compile all the Gnome apps to have them be able to use sound.
Was fun, but brutal in terms of efficiency.
Was a lot of fun, but a huge amount of time compiling
Every time I decided to try a new window manager and update USE flags to do something like add GTK support for Gnome, or QT for KDE, and obviously remove whatever I had to start with for optimization it would recompile everything. I remember in particular doing something like a fresh Gnome install, starting to use it and realizing I'd missed a USE flag for audio and having to re-compile all the Gnome apps to have them be able to use sound.
Was fun, but brutal in terms of efficiency.
> Gentoo is great, but similar to LFS it is kind of a time suck.
It's a time sink compared to binary distributions, but LFS is a whole order greater when it comes to time sinks. Gentoo automates almost everything that you would do by hand in LFS. Compile time doesn't count as it's not your time being used.
It's a time sink compared to binary distributions, but LFS is a whole order greater when it comes to time sinks. Gentoo automates almost everything that you would do by hand in LFS. Compile time doesn't count as it's not your time being used.
Wouldn’t nix be the go to source based distro these days?
Why include a keyboard and gpu only to use -nographic later?
I would love if there was something like "Alpine Linux from scratch", i.e. using musl libc instead of glibc, busybox instead of GNU coreutils, etc.
Just read the basic manual install script and execute it step by step.
This is great if you are a teenager. I used to do stuff like this all the time.
But as an adult.. It's just not worth the time.
But as an adult.. It's just not worth the time.
Adult people do stuff for fun, I guess.
This isn't (or shouldn't) be an age thing. It is a 'stage of learning" thing.
If you are new to Linux (or even all Unices) and you want to get a firm understanding of the fundamentals, it may very well be worth while to go through it at least once and see all the moving parts. And see them in an order that makes their dependencies clear(er).
Once you know all that and just want to keep up with the constant churn of versions to keep that one webserver running and recent? Yeah. Not worth it. Get some distro that matches your preferred characteristics and stop worrying if you need to update libBLA to recompile Apache to fix the path traversal vulnerability. That also removes the question if that library still compiles correctly with gcc 7 or if you need to upgrade to at least gcc 9...
Edit: Doing it in a VM first, might help in getting the 99 percent working first linux installation into a usable shape to make it a good daily driver. In that sense it helps lowering the barrier to actually switching later on.
If you are new to Linux (or even all Unices) and you want to get a firm understanding of the fundamentals, it may very well be worth while to go through it at least once and see all the moving parts. And see them in an order that makes their dependencies clear(er).
Once you know all that and just want to keep up with the constant churn of versions to keep that one webserver running and recent? Yeah. Not worth it. Get some distro that matches your preferred characteristics and stop worrying if you need to update libBLA to recompile Apache to fix the path traversal vulnerability. That also removes the question if that library still compiles correctly with gcc 7 or if you need to upgrade to at least gcc 9...
Edit: Doing it in a VM first, might help in getting the 99 percent working first linux installation into a usable shape to make it a good daily driver. In that sense it helps lowering the barrier to actually switching later on.
Yes, adults should not experiment with building custom distros... as we all know all existing Linux distributions were built by teenagers.
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newsguidelines.html
My company is looking for someone to build a linux system for a specialized tablet PC, with, among other things, a custom kernel and OTA updates.
If you built a linux system from scratch, this will put you in a very good position to get the job.
If you built a linux system from scratch, this will put you in a very good position to get the job.