Learning operating system development using Linux kernel and Raspberry Pi(github.com)
github.com
Learning operating system development using Linux kernel and Raspberry Pi
https://github.com/s-matyukevich/raspberry-pi-os
51 comments
I recently bought a USB serial cable online to play around with this sort of thing, and promptly saw a blue-screen in windows 10 for the first time I can remember.
It turned out I had unknowingly bought a fake Prolific cable and the (non-fake) windows driver was crashing my whole system every five minutes or so when the cable was connected.
I worked around this by tracking down an older driver with no crash issues so I guess the latest one does this deliberately?
In any case, if you are planning on getting a USB serial cable make sure it's not a counterfeit, if you can.
It turned out I had unknowingly bought a fake Prolific cable and the (non-fake) windows driver was crashing my whole system every five minutes or so when the cable was connected.
I worked around this by tracking down an older driver with no crash issues so I guess the latest one does this deliberately?
In any case, if you are planning on getting a USB serial cable make sure it's not a counterfeit, if you can.
I personally stick with Silicon Labs CP210x devices ever since I had bad luck with Prolific. I avoid FTDI chips as well because of the aggressive anti-consumer behavior of FTDI in regards to their drivers.
I recently bought these as well. These worked for me [1] [2]. Didn't need a driver on Windows/Linux/macOS. KiTTy, Minicom, cu, etc.
[1] https://www.amazon.de/gp/product/B00QUZY4UG
[2] https://www.amazon.de/gp/product/B00SBBFUAA
[1] https://www.amazon.de/gp/product/B00QUZY4UG
[2] https://www.amazon.de/gp/product/B00SBBFUAA
Relevant reading: https://www.theregister.co.uk/2016/01/31/ftdi_accused_of_bri...
TL;DR Avoid anything that even sounds like it might involve FTDI unless you can absolutely prove supply chain from one end to the other (near impossible). There are some good alternatives like SiLabs CP2102 around.
Does anybody have a better way to kick off my low level programming journey?
How much do you recommend "The C Programming Language" for someone who just used Java / Python, and wanna start learning C and low level?
Do you recommend learning "current" Rust as a first low level Lang.
How much do you recommend "The C Programming Language" for someone who just used Java / Python, and wanna start learning C and low level?
Do you recommend learning "current" Rust as a first low level Lang.
i recommend C to start with because there are more low-level projects you can work on for practice in C than in Rust; the biggest one being kernel hacking. i definitely recommend K&R to get familiar with the language, but it's a small language so that shouldn't take too much time.
i think the best way to get started in low-level programming is to develop in linux on a raspberry pi and hack your own kernel. explore sysfs, ttys, and gpios. implement new syscalls and device drivers. then go bare-metal on the arduino or a PIC16 chip and build an interesting piece of hardware.
i've collected a lot of my favorite low-level programming tutorials and deep-dives on my HN profile page, but if i were to pick just a couple, they would be:
"Write a System Call": https://brennan.io/2016/11/14/kernel-dev-ep3/
"The TTY Demystified": http://www.linusakesson.net/programming/tty/
And a book, "Linux Device Drivers, 3rd Edition":https://lwn.net/Kernel/LDD3/ (free!)
protip: as you're getting started with kernel hacking, use kernel 2.6. LDD3 uses 2.6, as do a wealth of other books.
i think the best way to get started in low-level programming is to develop in linux on a raspberry pi and hack your own kernel. explore sysfs, ttys, and gpios. implement new syscalls and device drivers. then go bare-metal on the arduino or a PIC16 chip and build an interesting piece of hardware.
i've collected a lot of my favorite low-level programming tutorials and deep-dives on my HN profile page, but if i were to pick just a couple, they would be:
"Write a System Call": https://brennan.io/2016/11/14/kernel-dev-ep3/
"The TTY Demystified": http://www.linusakesson.net/programming/tty/
And a book, "Linux Device Drivers, 3rd Edition":https://lwn.net/Kernel/LDD3/ (free!)
protip: as you're getting started with kernel hacking, use kernel 2.6. LDD3 uses 2.6, as do a wealth of other books.
I was slightly confused by your comment; then I realized: when you say "kernel" you mean Linux, specifically. Rust can very much be used to write kernels!
Indeed it can be used to write kernels, but there is not much established examples to learn how to write kernels in Rust. Experienced devs can use Rust to write kernels but learners may want to learn it in C then try Rust.
Yeah, we’ve got two big projects (one embedded, one desktop) and tons of toys, and multiple tutorials. Still much smaller than C, for sure.
yup! sorry for the confusion.
i'd also like to add that anybody who starts on a raspberry pi running linux will benefit from bootstrapped development. re-compiling the (linux!) kernel directly on the device is so much easier than setting up a cross-compile toolchain for C. eventually you'll want to have experience with cross compiling, but when you're first getting started it's a painful hurdle to overcome.
cross-compiling in rust is actually getting pretty easy, though, so that's cool.
i'd also like to add that anybody who starts on a raspberry pi running linux will benefit from bootstrapped development. re-compiling the (linux!) kernel directly on the device is so much easier than setting up a cross-compile toolchain for C. eventually you'll want to have experience with cross compiling, but when you're first getting started it's a painful hurdle to overcome.
cross-compiling in rust is actually getting pretty easy, though, so that's cool.
I would say start with C.
You won't appreciate a lot of the protection that Rust gives you until you have spent some time with the freedom to make the mistakes yourself. Stuff like pointer arithmetic really helps you get into the mindset of how a program really executes. Rust tends to abstract this away and put in all these seemingly arbitrary (that's probably what it would feel like coming from a GC language) rules.
Plus C is well documented and supported everywhere. Tons of learning resources and you can guarantee that if you have a question someone else is bound to have had the same question and asked it. Rust, being newer, doesn't have quite as much.
You won't appreciate a lot of the protection that Rust gives you until you have spent some time with the freedom to make the mistakes yourself. Stuff like pointer arithmetic really helps you get into the mindset of how a program really executes. Rust tends to abstract this away and put in all these seemingly arbitrary (that's probably what it would feel like coming from a GC language) rules.
Plus C is well documented and supported everywhere. Tons of learning resources and you can guarantee that if you have a question someone else is bound to have had the same question and asked it. Rust, being newer, doesn't have quite as much.
People are split about C or Rust first. Some agree with you, but others prefer to have the assistance of the compiler, rather than suffering first. I think each approach appeals to a different kind of person.
It's not "suffering first". It is to learn how the hardware really works.
Well, C isn’t how the hardware works either, but beyond that... you get that with Rust too. I’ve had people tell me that they didn’t really grok the stack vs heap when doing C, but learning Rust taught it to them.
As always, YMMV. I learned C 20 years before learning Rust myself.
As always, YMMV. I learned C 20 years before learning Rust myself.
Well, the difference between the stack and the heap is quite rudimentary; to be fair.
Sure, and it’s just an example. That’s not “how the hardware works” either, but my point is that Rust can and does teach systems concepts even though it may have a higher level of abstraction at times.
The C Programming Language is about the best book you can read to learn C, in my experience. Then go work on a non-trivial project like writing a command-line calculator.
Modern C is also pretty well-regarded, and teaches aspects of C99 which K&R don't. It's free too: http://icube-icps.unistra.fr/img_auth.php/archive/d/db/20151...
I recommend toying with an arduino. The Arduino itself can be obtained for less than 3$ on Aliexpress (Nano with a microusb is a good start)
They don't have any OS, rather, your program runs basically on the CPU directly. You can try to develop a OS from there, the CPU supports almost everything you need; interrupts and timers.
The only real limitation is that you have 2KiB of memory, on the other hand, it can be extended if you are willing to solder on a SRAM (2$ for 64KByte) or you can see it as a challenge to get an OS with a few processes working with such little memory.
The chip also has some advantages; since data and code live in seperate memory, you can't really crash or damage the chip by going out of bounds in an array. You can program something fairly robust in an arduino.
They don't have any OS, rather, your program runs basically on the CPU directly. You can try to develop a OS from there, the CPU supports almost everything you need; interrupts and timers.
The only real limitation is that you have 2KiB of memory, on the other hand, it can be extended if you are willing to solder on a SRAM (2$ for 64KByte) or you can see it as a challenge to get an OS with a few processes working with such little memory.
The chip also has some advantages; since data and code live in seperate memory, you can't really crash or damage the chip by going out of bounds in an array. You can program something fairly robust in an arduino.
I’d take on a project that benefits from using C over Python. Maybe port something old and valuable to a module with tests and whatnot in comfortable Python.
On the Java side where I live you can also compile native modules without much fuss.
I suggest this because it adds new to something familiar.
One thing I would not do is focus on making the first project maintainable etc or you’ll never get out the gate... stitch it together and redo it. Really 2 different programmers would be doing it, so no harm no foul.
Read documentation like it’s a good book... if it’s boring or not good then you’re not engaged enough so pick another project—supporting the disposable idea.
Good endeavors grow legs.
On the Java side where I live you can also compile native modules without much fuss.
I suggest this because it adds new to something familiar.
One thing I would not do is focus on making the first project maintainable etc or you’ll never get out the gate... stitch it together and redo it. Really 2 different programmers would be doing it, so no harm no foul.
Read documentation like it’s a good book... if it’s boring or not good then you’re not engaged enough so pick another project—supporting the disposable idea.
Good endeavors grow legs.
I'm not sure whether I'd recommend Rust.
Learning it will definitely make you a better c/c++ developer. It'll beat memory safety into you with whatever you give it.
That said, although it does have great c abi interop, it doesn't build on as many platforms as c. And I think there might be some knob or levers that c provides for low level stuff that rust doesn't expose on stabe yet (SIMD, const stuff, and probably more)
That said, I think once you get past the borrow checker, programming in rust is considerably more enjoyable.
Learning it will definitely make you a better c/c++ developer. It'll beat memory safety into you with whatever you give it.
That said, although it does have great c abi interop, it doesn't build on as many platforms as c. And I think there might be some knob or levers that c provides for low level stuff that rust doesn't expose on stabe yet (SIMD, const stuff, and probably more)
That said, I think once you get past the borrow checker, programming in rust is considerably more enjoyable.
SIMD will be on stable in less than two weeks.
That said, if you're doing kernel-level stuff, you still need nightly, generally. We're getting there! Some major bits are stabilizing in the very near term (the panic-format stuff is a big one, for example)
That said, if you're doing kernel-level stuff, you still need nightly, generally. We're getting there! Some major bits are stabilizing in the very near term (the panic-format stuff is a big one, for example)
Nand To Tetris: Building A Modern Computer From First Principles is an accessible, structured, bottom-up course in low-level stuff.
You start out by composing simulated hardware gates to create higher-order gates, eventually creating a CPU and entire computer. Then you move into assembly, and eventually build an OS. All of the homework has automated grading, so it’s very possible to self-study.
http://www.nand2tetris.org/
You start out by composing simulated hardware gates to create higher-order gates, eventually creating a CPU and entire computer. Then you move into assembly, and eventually build an OS. All of the homework has automated grading, so it’s very possible to self-study.
http://www.nand2tetris.org/
This looks really cool, great find!
I would propose Linux Kernel Development by Robert Love. It is not for beginners though.
That could help you in applying your new C skills in writing a simple device driver/kernel module you can play with. If are really into low level stuff I would propose an Arduino board whose OS is minimal, the Linux kernel is already very complex.
Get a role in an embedded software company and you may learn it all, but some experience is required.
That could help you in applying your new C skills in writing a simple device driver/kernel module you can play with. If are really into low level stuff I would propose an Arduino board whose OS is minimal, the Linux kernel is already very complex.
Get a role in an embedded software company and you may learn it all, but some experience is required.
Do you think I can compile linux from source, and implement a new module without knowing C++? (Coming from python)
I dont mean creating a whole functional module, but more like using an example, modifying some values, adding it to the kernel and compiling.
I dont mean creating a whole functional module, but more like using an example, modifying some values, adding it to the kernel and compiling.
Recompiling the kernel used to be something you just did for some things.
If you can drive the command line and a makefile you should be able to tinker, and render it unable to boot. :)
If you can drive the command line and a makefile you should be able to tinker, and render it unable to boot. :)
You will need to know the C language, not C++. As with many things you will follow an example, then another, then after a few you will walk using your own legs.
To build a module you don't need to build the kernel, but you do need the kernel headers, you will understand all this once you get to understand a bit of C language, and a bit of assembly won't hurt. Start with MIPS.
You do need a good grasp on the C language, and a lot of time.
To build a module you don't need to build the kernel, but you do need the kernel headers, you will understand all this once you get to understand a bit of C language, and a bit of assembly won't hurt. Start with MIPS.
You do need a good grasp on the C language, and a lot of time.
I've recently started to work on an implementation of Forth, targeting the RPi. Only runs on Qemu at the moment. A great way to learn about low level programming
After learning some C, start reading "The Linux Programming Interface"[0]. It totally changed my mind.
[0] https://nostarch.com/tlpi
[0] https://nostarch.com/tlpi
I'm not familiar with low-level, so I'm not sure how "low" you mean, but I've begun the NAND2Tetris course on coursera and it's pretty great so far. You basically implement a VM from scratch: https://www.coursera.org/learn/build-a-computer
I actually took that early this year and LOVED it, and was looking for a good followup course. I was toying around with CS140e (also uses RPi 3 and Rust)[1] but this one looks more appropriate.
[1] https://web.stanford.edu/class/cs140e/
[1] https://web.stanford.edu/class/cs140e/
Wow I just did the same course this week too.
It's a great course. Though I ended up writing the assembler in Python rather than C.
It's a great course. Though I ended up writing the assembler in Python rather than C.
I'd also like to learn how to code in C/Rust, but the amount of time I need to invest in it to be even partially sufficient is discouraging. It's completely different from my current job, being a C#/JS full stack web dev.
We have a lot of people come to Rust from JS! It might be easier than you think. It really just depends. We’re constantly working to drop the learning curve as well.
You know what? I might actually try again. I like to learn while doing some real work, but I had hard time finding easy open-source Rust projects to contribute to. Can you please recommend any?
Also, what is the best beginner-friendly learning source? Rust Book 2nd edition? I like to learn from printed books, are there any good ones?
Also, what is the best beginner-friendly learning source? Rust Book 2nd edition? I like to learn from printed books, are there any good ones?
Are you more front or back end with the JS? You said full stack, but I could also see that being backend C# with frontend JS.
I also prefer printed books. The book second edition is personally my favorite, of course. :) it’s actually at the printers right now, so will be available in hard copy in just a few weeks. The O’Reilly book is also good but leans a bit more towards the experienced systems dev. It may work for you, it may not. I bought a copy of “Beginning Rust” by APress, I remember it being fairly solid.
There’s also a very neat book, “Step Ahead with Rust”, whose goal is to get you to learn the basics of the language well enough to be able to dive into other things. It skins over some stuff, but that also means it’s shorter, and therefore a bit simpler and faster than a more thorough book. It might appeal to you as well.
I also prefer printed books. The book second edition is personally my favorite, of course. :) it’s actually at the printers right now, so will be available in hard copy in just a few weeks. The O’Reilly book is also good but leans a bit more towards the experienced systems dev. It may work for you, it may not. I bought a copy of “Beginning Rust” by APress, I remember it being fairly solid.
There’s also a very neat book, “Step Ahead with Rust”, whose goal is to get you to learn the basics of the language well enough to be able to dive into other things. It skins over some stuff, but that also means it’s shorter, and therefore a bit simpler and faster than a more thorough book. It might appeal to you as well.
I'm a backend C#(.NET)/JS(node.js), and frontend JS(angular/react) dev.
I kinda regret going into a corporate web dev field. The JS stack is terrible, work is mundane and monotonous. That's why I'm trying to learn myself something different.
Glad to hear that the official book will be available in print. The O'Reilly's book looks decent too, however it might be a little outdated already, considering how fast Rust is evolving.
I kinda regret going into a corporate web dev field. The JS stack is terrible, work is mundane and monotonous. That's why I'm trying to learn myself something different.
Glad to hear that the official book will be available in print. The O'Reilly's book looks decent too, however it might be a little outdated already, considering how fast Rust is evolving.
Cool! So, one thing that may be of interest is the Rust/WebAssembly work; https://github.com/ashleygwilliams/wasm-pack is a huge part of this story and is very open to new contributors who are just learning Rust.
On the server, there's a lot of churn right now, as async/await is being worked on in the compiler, futures are landing in the standard library, and the general ecosystem is undergoing a huge upgrade. So it also needs a lot of work, but may be harder without already knowing Rust.
One good thing to remember is that while it may be a little out of date, that just means it's missing some things. Nothing in it is wrong, it just is missing new APIs that may make some things a bit nicer.
On the server, there's a lot of churn right now, as async/await is being worked on in the compiler, futures are landing in the standard library, and the general ecosystem is undergoing a huge upgrade. So it also needs a lot of work, but may be harder without already knowing Rust.
One good thing to remember is that while it may be a little out of date, that just means it's missing some things. Nothing in it is wrong, it just is missing new APIs that may make some things a bit nicer.
How does this deal with the binary blobs that need to start the boot loader? From memory the older models needed to launch the graphics chip first which was a hack-y nightmare with no open source way of doing it.
It ignores that and starts at the point where the kernel is loaded by that first-stage bootloader. Which is very reasonable for a beginner's introduction.
Fair enough, there's a bunch of fully open source computers that let you really build things from the ground up: https://www.olimex.com/Products/OLinuXino/A10/open-source-ha...
Completely out of time to do anything for it but this company getting more traction would be really good given how completely open they are.
Completely out of time to do anything for it but this company getting more traction would be really good given how completely open they are.
> Fair enough, there's a bunch of fully open source computers that let you really build things from the ground up
Concerning your linked example: if I look at the specs at
> https://www.olimex.com/Products/OLinuXino/A10/A10-OLinuXino-...
I see Mali 400 GPU. Is this GPU now fully open source?
Concerning your linked example: if I look at the specs at
> https://www.olimex.com/Products/OLinuXino/A10/A10-OLinuXino-...
I see Mali 400 GPU. Is this GPU now fully open source?
> I see Mali 400 GPU. Is this GPU now fully open source?
If you are talking about drivers (rather than the GPU itself), there are reverse engineered drivers:
https://github.com/yuq/mesa-lima/ https://github.com/yuq/linux-lima https://gitlab.freedesktop.org/lima
For the newer Mali GPUs there is the Panfrost driver:
https://notabug.org/cafe/chai https://gitlab.freedesktop.org/panfrost https://rosenzweig.io/blog/
If you are talking about drivers (rather than the GPU itself), there are reverse engineered drivers:
https://github.com/yuq/mesa-lima/ https://github.com/yuq/linux-lima https://gitlab.freedesktop.org/lima
For the newer Mali GPUs there is the Panfrost driver:
https://notabug.org/cafe/chai https://gitlab.freedesktop.org/panfrost https://rosenzweig.io/blog/
You can run stuff on any of the Allwinner SoCs without using the GPU, the display controller that provides HDMI or VGA output is separate and fully documented. Some of them also have a simple 2D graphics accelerator.
To be fair, an A10 has a 32kb closed source ROM that's the first stage bootloader, that chain loads your code.
The raspberry pi is built on top of a graphics chip with some ARM cores tacked on as an afterthought. [1]
The binary blobs are not firmware. A full proprietary RTOS called ThreadX is running on the Raspberry Pi as the primary OS and this won't change until they move off of broadcom's video core.
[1] https://www.heise.de/ct/zcontent/16/08-hocmsmeta/14601932130...
[1] https://www.heise.de/ct/zcontent/16/08-hocmsmeta/14601932130...
Based on the title I was hoping more emphasis on the interesting userland bits, leaving the boring kernel parts to Linux. I feel that osdev generally focuses too much on the well-trodden elementary basic building blocks, and less on actual systems design, which has left the community with endless half-baked (generally more or less posixy style) kernels and few novel userlands.
Android is great example how you can do interesting OS development work by building upon stable foundations, focusing on userland, but also not fearing to poke kernel when needed.
I don't mean to disparage anyone, least of all the repo here that seems pretty nice resource for what it is. Just the title happened to tickle my imagination just right.
Android is great example how you can do interesting OS development work by building upon stable foundations, focusing on userland, but also not fearing to poke kernel when needed.
I don't mean to disparage anyone, least of all the repo here that seems pretty nice resource for what it is. Just the title happened to tickle my imagination just right.
Alternative: Xinu-Os with BeagleBone.
> But my first attempt was a complete failure - I understood almost nothing.
This is how I feel every time I dig into something. Disappointingly, rarely do I have the time or focus to truly learn how something really works. Be it a library I am using, or a physics concept I want to learn more about.
Good for him for getting through it. I am adding this to my long ass list of things I would read if I could freeze time like that girl from the 90's TV show.