RISC-V OS using Rust: Filesystems(osblog.stephenmarz.com)
osblog.stephenmarz.com
RISC-V OS using Rust: Filesystems
http://osblog.stephenmarz.com/ch10.html
72 comments
Thanks for the feedback. It does get tricky to balance going too far into the weeds and keeping it above board. I think a note would be helpful as you suggested.
I do try to put lots of comments in the Rust code to describe what I didn't in the blog for those more interested in the implementation.
I do try to put lots of comments in the Rust code to describe what I didn't in the blog for those more interested in the implementation.
Excellent series of posts. I've been following along with no prior Rust or OS programming knowledge and have found it immensely refreshing.
Thanks! Glad it could serve some purpose!
I'd like to see a clone of the QNX kernel, which is very tiny, implemented in Rust. It has no drivers or file systems in kernel space; those are all user processes. But it offers a POSIX-like API. Unlike L4, you don't have to run another OS on top of it to do anything.
This would be good for embedded applications on Raspberry Pi sized machines, where the Arduino environment is too weak and Linux is too much.
This would be good for embedded applications on Raspberry Pi sized machines, where the Arduino environment is too weak and Linux is too much.
Aren't there patents protecting it? I remember reading something about the fast interprocess call mechanism... Microkernels are prone to high overhead because everything requires context switching. Maybe this isn't true anymore.
QNX has been around for a long time. Back in the mid-90s I did some consulting work porting some Unix utilities to QNX, and if there were patents back then, they have now expired.
US 8,434,086, "Process scheduler employing adaptive partitioning of critical process threads", is still in force until 2030. That's a more recent feature for hard real time, where you want to guarantee that some thread gets, say, 2ms of CPU time every 10ms of elapsed time. Few schedulers do that sort of thing at such fine granularity, but that's what you may need if you're controlling some actuator. Good for things like audio players, too. It's not a commonly used feature, and went into QNX some time after 2005. I remember it being listed as a future feature back then.
The fast interprocess call mechanism is what makes it all go. It's mostly about it being simple. Mach tried to get too cute, moving pages from one address space to another. Turns out that copying is usually cheaper than stalling out the CPU and flushing caches while you mess with the page tables of recently active pages. L4 just offers shared memory, dumping the problem on the next level up. QNX offers useful send/receive message functionality that goes fast. Yes, there's a penalty, maybe 20%. But you usually get that back because you're not forced to do interprocess communication with something far slower, like HTTP.
The fast interprocess call mechanism is what makes it all go. It's mostly about it being simple. Mach tried to get too cute, moving pages from one address space to another. Turns out that copying is usually cheaper than stalling out the CPU and flushing caches while you mess with the page tables of recently active pages. L4 just offers shared memory, dumping the problem on the next level up. QNX offers useful send/receive message functionality that goes fast. Yes, there's a penalty, maybe 20%. But you usually get that back because you're not forced to do interprocess communication with something far slower, like HTTP.
I guess Tock or Redox might be your options there.
A couple of minor, but very irritating errors:
> The singly-indirect pointer can address 1,024 * 1,024 / 4 = 1,024 * 256 = 262 KiB of data.
256 kilobytes, not 262 'kibi'bytes.
> 1,024 * 256 * 256 = 67 Mi
> 1,024 * 256 * 256 * 256 = 17 Gi
64 and 16 respectively.
> The singly-indirect pointer can address 1,024 * 1,024 / 4 = 1,024 * 256 = 262 KiB of data.
256 kilobytes, not 262 'kibi'bytes.
> 1,024 * 256 * 256 = 67 Mi
> 1,024 * 256 * 256 * 256 = 17 Gi
64 and 16 respectively.
If that irritates you, please don't check the grammar and errant placement of commas!
All jokes aside. Fixed, thanks.
All jokes aside. Fixed, thanks.
Actually, the grammar seems fine? Or at least I haven't noticed any problems with it. TFA as a whole seems pretty good actually. The file sizes are still incorrect, though; it calls the units kilobytes now, but it still claims there's 262 of them.
The seven blocks at the start would make it 263 KB, but I don't think that what you meant.
The seven blocks at the start would make it 263 KB, but I don't think that what you meant.
>The file sizes are still incorrect, though; it calls the units kilobytes now, but it still claims there's 262 of them.
1024 * 1024 / 4 B = 262144 B = 262.144 KB = 256 KiB
1024 * 1024 / 4 B = 262144 B = 262.144 KB = 256 KiB
Yes, exactly; 262144 bytes is 256 kilobytes.
262144 bytes; 262144 / 1024 = 256 kibibytes
262144 bytes; 262144 / 1000 ≈ 262 kilobytes
262144 bytes; 262144 / 1000 ≈ 262 kilobytes
KB is kilobyte. KiB is kibibyte.
> kibibyte
I wonder if anyone uses this abomination of a word in real life though.
I wonder if anyone uses this abomination of a word in real life though.
Not that I know of; they're called kilobytes.
Yes, KB = kilobyte = 1024 byte.
At this point I can only assume I'm being trolled.
This whole thread is confusing to me as well. Isn't the whole point of a kibibyte to work in increments of 1024, unlike kilo, which is actually 1000, as defined by the sciences long ago?
So 1024 * 256 is 256 KiB by definition, since a Kibibyte is literally 1024 bytes, and the number of Kilobytes will by higher, since the number it represents is smaller.
So 1024 * 256 is 256 KiB by definition, since a Kibibyte is literally 1024 bytes, and the number of Kilobytes will by higher, since the number it represents is smaller.
Either user a1369209993 is unaware of the distinction ( https://en.wikipedia.org/wiki/Kibibyte ) or then they are deliberately ignoring it (My guess: to defend the old definition of kilobyte = 1024 bytes) to the point that their messages look like trolling.
Reality will forcibly catch up with them when they try to buy a new harddrive.
That's what I thought. I really only care about how much "stuff" I can store in a block, which is 1,024 bytes. If that's 1KiB or 1.024KBs, the distinction isn't worth that much brainpower.
It is actually 256 KiB, not KB.
Is there a significant limitation in using Rust on the stable branch? Every interesting tutorial, book, or tool I find seems to use nightly. I know it shouldn't be a big deal but it "feels dirty" and perhaps I should just view nightly as any other language's "stable".
For operating systems development, inline assembly is convenient, which is not yet stable. See the conversation below.
The vast majority of Rust users use stable. You just happen to see a lot of folks who are interested in the cutting edge blog about things. (You may also run into older things where that thing is stable now, too, to be clear.)
The vast majority of Rust users use stable. You just happen to see a lot of folks who are interested in the cutting edge blog about things. (You may also run into older things where that thing is stable now, too, to be clear.)
> perhaps I should just view nightly as any other language's "stable".
It depends on the features you enable. Stable is basically just a pinned nightly version with no features enabled, and I'd definitely view "nightly with no features" as more stable than most languages.
Using some of the more experimental features would make it less stable, you can generally get a feel for how stable a feature is by looking at it's tracking issue.
These days most tutorials/books/tools don't seem to use nightly in my experience, though many still do. But if you're primarily looking at blog posts that are pushing the boundaries of the language (e.g. by writing operating systems) I expect that most of them would be using nightly.
It depends on the features you enable. Stable is basically just a pinned nightly version with no features enabled, and I'd definitely view "nightly with no features" as more stable than most languages.
Using some of the more experimental features would make it less stable, you can generally get a feel for how stable a feature is by looking at it's tracking issue.
These days most tutorials/books/tools don't seem to use nightly in my experience, though many still do. But if you're primarily looking at blog posts that are pushing the boundaries of the language (e.g. by writing operating systems) I expect that most of them would be using nightly.
Stable is pinned nightly with disabled nightly features and beta backports applied. The difference might sound like semantics, but it is present and can impact you.
I think perhaps a lot of projects start out using nightly with the assumption that within 6-12 months those features they are using will be in stable.
Whether that generally happens in reality, or people get into a loop of "well, this other new features is nice and in nightly, and we're already in it, so why not", I'm not sure.
Whether that generally happens in reality, or people get into a loop of "well, this other new features is nice and in nightly, and we're already in it, so why not", I'm not sure.
> I think perhaps a lot of projects start out using nightly with the assumption that within 6-12 months those features they are using will be in stable.
In my experience, that is rarely the case.
People mostly use nightly features because they just need to get something done, and these features make that easier.
The tooling around Rust is quite good (rustup, cargo, docker), so if you follow the language a bit, judge which nightly features you opt-into, pin nightly version in your build environment, etc. breakage only happens when you actually have time to fix it and the fixes are simple.
So the bar for how much easier a nightly feature needs to make your problem to overcome its costs is quite low. The moment you are using one nightly feature, the cost of using more is even smaller.
In my experience, that is rarely the case.
People mostly use nightly features because they just need to get something done, and these features make that easier.
The tooling around Rust is quite good (rustup, cargo, docker), so if you follow the language a bit, judge which nightly features you opt-into, pin nightly version in your build environment, etc. breakage only happens when you actually have time to fix it and the fixes are simple.
So the bar for how much easier a nightly feature needs to make your problem to overcome its costs is quite low. The moment you are using one nightly feature, the cost of using more is even smaller.
I think just two years ago it was much more normal to use nightly because of some feature that you needed, this is mure rarer nowadays. I haven't used nightly in months, and then I only used it to take a look at Rocket. Unless you require something behind a feature flag (in which case you probably know what you're doing) you should be using stable.
> I know it shouldn't be a big deal but it "feels dirty" and perhaps I should just view nightly as any other language's "stable".
There are two uses for nightly:
1. Access to unreleased features. I think I'm down to just needing this for associated_type_defaults (rfc here: https://github.com/rust-lang/rfcs/pull/2532/files). 2. Nightly allows passing extra compiler flags, which is occasionally useful for debugging things like macro expansion.
Personally I've found that nightly is pretty solid and I do tend to use it more than stable, but I certainly would not consider it dirty in itself. Furthermore if you're looking at a tutorial or a book the features it relies upon may have gotten into the stable build.
There are two uses for nightly:
1. Access to unreleased features. I think I'm down to just needing this for associated_type_defaults (rfc here: https://github.com/rust-lang/rfcs/pull/2532/files). 2. Nightly allows passing extra compiler flags, which is occasionally useful for debugging things like macro expansion.
Personally I've found that nightly is pretty solid and I do tend to use it more than stable, but I certainly would not consider it dirty in itself. Furthermore if you're looking at a tutorial or a book the features it relies upon may have gotten into the stable build.
My advice is generally: nightly for development, stable for ci. If you don't rely on nightly features, nightly will still let you reap the benefits of improvements that will land in stable 12 weeks later.
This looks awesome! Yesterday I just started another OS rust walk through and now I have a hard choice between that and this.
Any thoughts on this[1] walkthrough in comparison? This one looks more explicitly up to date.
[1] https://os.phil-opp.com/
Any thoughts on this[1] walkthrough in comparison? This one looks more explicitly up to date.
[1] https://os.phil-opp.com/
I think you'll get something different for each. The purpose of mine is to explore the Rust programming language, but more specifically on the RISC-V architecture. Rust "recently" became out-of-the-box for RISC-V, which has opened up many more avenues for exploration.
As a note, I believe os.phil-opp.com is using the AMD/Intel architecture.
As a note, I believe os.phil-opp.com is using the AMD/Intel architecture.
What do you mean by out of the box for RISC V?
Originally, I required a GNU toolchain from the RISC-V organization for both assembling and linking. Now, I don't need any external tools (except QEMU to run it). Everything you need can be downloaded using Rust's configuration tool rustup for RISC-V. If you take a look at my github, you'll see that I added a .cargo/config file that will build AND run the operating system using cargo.
Wait, we can now host a whole OS w/o directly requiring a C compiler?! I mean Rust still requires one to build clang/llvm, but maybe someday when alternative backends land.
There are several projects that do this, yes. They're generally not production ready, but it's absolutely possible.
I had a question of the about a passage under the section:
"Our OS's Filesystem Read Process" where the author states:
>"Recall that the block driver makes a request and sends it off to the block device. The block device services the request and then sends an interrupt when it's finished. We don't really know when that interrupt will come, and we can't wait around for it. This is why I decided to make the file system reader a kernel process. We can put this kernel process in the waiting state so it won't be scheduled until the interrupt is received from the block device"
Then further down: >"Now, when we handle the block device's interrupt, we have to match the watcher and awaken it."
Is the watcher here similar to the filesystem driver at the VFS layer in Linux i.e the ext4/xfs etc? I believe this is how Linux handles it - the file system driver creates a block IO request and hands it off to the actual block device driver and when the block device driver get's invoked on an interrupt from the disk the block device driver informs the filesystem driver that the IO request is now complete correct?
>"Recall that the block driver makes a request and sends it off to the block device. The block device services the request and then sends an interrupt when it's finished. We don't really know when that interrupt will come, and we can't wait around for it. This is why I decided to make the file system reader a kernel process. We can put this kernel process in the waiting state so it won't be scheduled until the interrupt is received from the block device"
Then further down: >"Now, when we handle the block device's interrupt, we have to match the watcher and awaken it."
Is the watcher here similar to the filesystem driver at the VFS layer in Linux i.e the ext4/xfs etc? I believe this is how Linux handles it - the file system driver creates a block IO request and hands it off to the actual block device driver and when the block device driver get's invoked on an interrupt from the disk the block device driver informs the filesystem driver that the IO request is now complete correct?
The minix fs seems to be basically the classic Unix FS design, like UFS/FFS or Ext2, prior to the concept of cylinder groups (at least, per this description and Wikipedia's) and without some of the more advanced features of those filesystems (softupdates, directory trees, etc).
I mean, that's what you'd expect from an OS that was originally for teaching, right? A worked example of the fundamental parts common to most other unix filesystems, just as I would naively expect the minix scheduler to just be round-robin or something, etc.
Well, that was especially true for Minix 1 and 2. Minix 3 is purportedly a production operating system, and ships in every Intel CPU for the last decade or so, right? So while admittedly it may not have significant filesystem needs, it is not purely a teaching operating system anymore.
Nowadays I might point at MIT's xv6, which implements a very similar simple unix filesystem: https://github.com/mit-pdos/xv6-public/blob/master/fs.c
Nowadays I might point at MIT's xv6, which implements a very similar simple unix filesystem: https://github.com/mit-pdos/xv6-public/blob/master/fs.c
xv6 also was switched to RISC-V (not exactly sure when, but sometime in 2019), and it has a nice book for the actual course.
Course: https://pdos.csail.mit.edu/6.828/2019/xv6.html Book: https://pdos.csail.mit.edu/6.828/2019/xv6/book-riscv-rev0.pd...
Take a look at chapter 7 for the xv6 filesystem referenced here.
Course: https://pdos.csail.mit.edu/6.828/2019/xv6.html Book: https://pdos.csail.mit.edu/6.828/2019/xv6/book-riscv-rev0.pd...
Take a look at chapter 7 for the xv6 filesystem referenced here.
Is the asm!() macro going to stabilize? It seems important to ever make a real OS, right?
It's closer than it's ever been! The rough plan:
* move asm! to llvm_asm. https://github.com/rust-lang/rust/pull/68404
* implement the actual asm! we want to stabilize: https://github.com/rust-lang/rfcs/pull/2873
* stabilize it
* move asm! to llvm_asm. https://github.com/rust-lang/rust/pull/68404
* implement the actual asm! we want to stabilize: https://github.com/rust-lang/rfcs/pull/2873
* stabilize it
I stand corrected! Thanks for the info!
It's all good. I think that was the prevailing sentiment for years, but Amanieu put in the work, and things are looking bright. It hasn't been accepted yet, of course, so anything could happen. But stuff is moving in that direction, it seems.
Cool!
Curious why not WASM also? I have no use case for that, just asking.
Curious why not WASM also? I have no use case for that, just asking.
Sorry, I'm not entirely sure what you're asking?
Relying to myself rather than replying to three people: this would be a good question on the RFC, but I would imagine that this would be possible, just not super useful. Usually inline asm is useful for doing stuff you cannot do in the higher level language, but wasm doesn’t have anything like that currently. I would imagine that it would be a fairly straightforward extension though.
They are probably asking why WASM (probably in .wat format) is not an option alongside x86/x86-64, ARM, AArch64, and RISC-V.
Perhaps why there is no wasm equivalent directive? Just a guess.
Hello, all signs point to no. I've seen many threads about stabilizing asm, but most seem to say no this won't happen. In fact, the Rust compiler now throws a warning that asm!() is deprecated in favor of llvm_asm!().
This is the only reason I am still using nightly builds...because I'm forced to use the asm language feature, which is unstable, and because it's unstable, it cannot be used on the stable channel.
Edit: to respond to the second half, for out-of-the-box Rust builds, yes, I need global_asm! and asm!. However, asm! is for inline assembly only. Originally, I used GNU's assembler to assemble and then linked using GNU's ld. However, I have since switched to trying to make Rust do everything through its build manager, cargo.
This is the only reason I am still using nightly builds...because I'm forced to use the asm language feature, which is unstable, and because it's unstable, it cannot be used on the stable channel.
Edit: to respond to the second half, for out-of-the-box Rust builds, yes, I need global_asm! and asm!. However, asm! is for inline assembly only. Originally, I used GNU's assembler to assemble and then linked using GNU's ld. However, I have since switched to trying to make Rust do everything through its build manager, cargo.
There is a proposal for a new asm! macro: https://github.com/Amanieu/rfcs/blob/inline-asm/text/0000-in... (note that the last updated date was 2 days ago, as of the time of this comment).
It looks like what you're referring to is the interim plan to rename asm! to llvm_asm! (which will never be stabilized), since it's a very thin wrapper over the current LLVM inline asm support, and then replace asm! with a better implementation.
There is a working group specifically for inline asm, so it will happen eventually... but it's not going to be a quick process--think about how long async/await takes, and inline asm is actually rather close to that in terms of all the issues that come out of it.
It looks like what you're referring to is the interim plan to rename asm! to llvm_asm! (which will never be stabilized), since it's a very thin wrapper over the current LLVM inline asm support, and then replace asm! with a better implementation.
There is a working group specifically for inline asm, so it will happen eventually... but it's not going to be a quick process--think about how long async/await takes, and inline asm is actually rather close to that in terms of all the issues that come out of it.
Wow, that looks like a lot of work went into this RFC. I'm glad it's being taken seriously, and I anxiously await the outcome--whatever that might be.
Thank you!
Thank you!
Not really, one can always make use of an external assembler.
This is awesome. I've been following along for a while, so please keep it up.
I wish someday we'll get something like the ESP32 boards but with RISCV.
I wish someday we'll get something like the ESP32 boards but with RISCV.
Thanks!
You might want to check out the Kendryte K210. I'm trying out these boards for development. They are designed with two (I think) 64-bit RISC-V cores with a hardware FPU. https://www.seeedstudio.com/blog/2019/09/12/get-started-with...
I've flashed mine a few times, but I'm still working to getting it to work. The documentation isn't very good, so I'm working to reverse engineer some aspects of it.
This is a low cost alternative. Of course, we could go with the SiFive Unleashed (64-bit) or the SiFive HiFive1, which uses a 32-bit RISC-V CPU in an Arduino form factor. https://www.sifive.com/boards
The SiFive Unleashed cost me a cool $1k, so I'm still scared to mess with it.
You might want to check out the Kendryte K210. I'm trying out these boards for development. They are designed with two (I think) 64-bit RISC-V cores with a hardware FPU. https://www.seeedstudio.com/blog/2019/09/12/get-started-with...
I've flashed mine a few times, but I'm still working to getting it to work. The documentation isn't very good, so I'm working to reverse engineer some aspects of it.
This is a low cost alternative. Of course, we could go with the SiFive Unleashed (64-bit) or the SiFive HiFive1, which uses a 32-bit RISC-V CPU in an Arduino form factor. https://www.sifive.com/boards
The SiFive Unleashed cost me a cool $1k, so I'm still scared to mess with it.
The Kendryte is pretty limited in many ways, not least that it's permanently limited to 8 MiB of DRAM (AFAIK).
The most interesting route is using an FPGA. There are many examples of running RISC-V softcores on FPGAs (for example, Berkeley's RocketChip) and many boards that support a decent amount of memory (even beyond 1 GiB).
The most interesting route is using an FPGA. There are many examples of running RISC-V softcores on FPGAs (for example, Berkeley's RocketChip) and many boards that support a decent amount of memory (even beyond 1 GiB).
I tried some of SiFive's evaluation cores. They limit you quite a bit unless you pay. Furthermore, many of the FPGAs worth their salt were quite expensive. The K210 in the MaixBit form is only $20, and it comes with a camera and LCD. For hobbyists, I think it's a good compromise.
I looked at the Xilinx FPGAs--the ones SiFive recommends, but they're quite pricey.
Which FPGAs had you had luck with?
I looked at the Xilinx FPGAs--the ones SiFive recommends, but they're quite pricey.
Which FPGAs had you had luck with?
I haven't used them myself, but these boards look awesome, and they are supported by an end-to-end open source toolchain:
1. https://www.crowdsupply.com/1bitsquared/icebreaker-fpga 2. https://www.crowdsupply.com/radiona/ulx3s 3. https://groupgets.com/campaigns/710-orangecrab
1. https://www.crowdsupply.com/1bitsquared/icebreaker-fpga 2. https://www.crowdsupply.com/radiona/ulx3s 3. https://groupgets.com/campaigns/710-orangecrab
I was looking at getting this Maixduino kit since it appears to be the cheapest solution that satisfies my constraints (RISCV core + WiFi):
https://www.seeedstudio.com/Sipeed-Maixduino-Kit-for-RISC-V-...
I'm not sure what's the status of programming these from Rust. I suppose that you will probably need to build your own toolchain that links the Seeedstudio libraries.
https://www.seeedstudio.com/Sipeed-Maixduino-Kit-for-RISC-V-...
I'm not sure what's the status of programming these from Rust. I suppose that you will probably need to build your own toolchain that links the Seeedstudio libraries.
RISC-V has their own supported toolchain. You can get it here: https://github.com/riscv/riscv-gnu-toolchain.git
With Rust, you can do what I did with my OS. It can compile out of the box. The kendryte has a "BSP" (base support package) that is essentially a minimal OS, but it is written in C.
With Rust, you can do what I did with my OS. It can compile out of the box. The kendryte has a "BSP" (base support package) that is essentially a minimal OS, but it is written in C.
Yeah, I was thinking more about what to do with the OS afterwards. That board has lots of things on it, including wifi, so I'll probably would like to use some of that.
It would be unfortunate to have to re-implement the wifi driver in Rust, but I suppose I'll just have to link parts of the C toolchain instead, creating Rust wrappers, etc.
It would be unfortunate to have to re-implement the wifi driver in Rust, but I suppose I'll just have to link parts of the C toolchain instead, creating Rust wrappers, etc.
Well, FPGA are more flexible but cost more than a cheap microcontroller that is true. There are way too many good options, but to pick one with a fantastic community: ULX3S
Have you checked out SiFive’s HiFive1?[0] It’s an Arduino Uno sized board, but instead of a weak ATMega328p, it’s a (relatively) powerful RISC-V processor. It also has a BT and Wi-Fi chip.
[0]: https://www.sifive.com/boards/hifive1-rev-b
[0]: https://www.sifive.com/boards/hifive1-rev-b
No need to wait, the ESP32-S2 has a RISC-V as its low-power core.
ESP32-S2 has two ULP co-processors, with one based on RISCV instruction set architecture (ULP-RISCV) and
the other on finite state machine (ULP-FSM).
ULP-RISCV has the following features:
ULP-RISCV has the following features:
support for IMC instruction set
thirty-two 32-bit general-purpose registers
32-bit multiplier and divider
support for interrupts
boot by the CPU, its dedicated timer, or RTC GPIO
Interesting, wasn't aware. Is the ULP-FSM the same as the one in the original ESP32? I thought that one was programmable / Turing-complete, so not just a FSM?
I have never used the ULP-FSM, the new docs for ESP32-S2 are fascinating. My S2 is on its way, I don't yet have the hardware in hand.
My only criticism is that sometimes there's parts in your tutorial that are missing but present in your os. When going through chapter 3/4 I think you skipped over explaining your kmalloc implementation. It's fine that you do since I feel your tutorial should be more about navigating embedded rust than learning os basics, but a note about this task being implemented at some point would've helped.