Oxide on My Wrist: Hubris on PineTime was the best worst idea(artemis.sh)
artemis.sh
Oxide on My Wrist: Hubris on PineTime was the best worst idea
https://artemis.sh/2022/03/28/oxide-hubris-on-pinetime.html
50 comments
People often have good reason to not adopt existing systems. Oxide started with TockOS and have written and talked about why they didn't end up using it.
This twitter space goes into a lot of detail: https://www.youtube.com/watch?v=cypmufnPfLw
But at the end of the day, no_std will allow a bigger ecosystem and sharing between many of these projects.
This twitter space goes into a lot of detail: https://www.youtube.com/watch?v=cypmufnPfLw
But at the end of the day, no_std will allow a bigger ecosystem and sharing between many of these projects.
Thing is, it's not even clear that their added elements are all that sensible. If you're always running trusted tasks on your embedded hardware, what do you need the MPU for? Even in principle, it can only save you a bounds check. And if you start running outside code, then the clear privilege separation in Tock between trusted "Capsules" and other "Tasks" starts looking plenty attractive. Even OP notes that this MPU business introduces severe overhead.
Other folks have mentioned this, but it's important to understand the limitations of Rust with respect to safety. In particular: every stack operation is -- at some level -- an unsafe operation as it operates without a bounds check. This isn't Rust's fault per se; non-segmented architectures don't have an architecturally defined way to know the stack base. As a result, even an entirely safe Rust program can make an illegal access to memory that results in fatal program failure. That, of course, assumes memory protection; if you don't have memory protection (or, like many embedded operating systems, you don't make use of it), stack overflows will plow into adjacent memory.
But wait, it gets worse: stack overflows are often not due to infinite stack consumption (e.g., recursion) but rather simply going deep on an unusual code path. If stack consumption just goes slightly beyond the base of the stack and there is no memory protection, this is corrupt-and-run -- and you are left debugging a problem that looks every bit like a gnarly data race in an unsafe programming language. And this problem becomes especially acute when memory is scarce: you really don't want a tiny embedded system to be dedicating a bunch of its memory to stack space that will never ("never") be used, so you make the stacks as tight as possible -- making stack overflows in fact much more likely.
Indeed, even with the MPU, these problems were acute in the development of Hubris: we originally put the stack at the top of a task's data space, and its data at the bottom -- and we found that tasks that only slightly exceeded their stack (rather than running all of the way through its data and into the protection boundary) were corrupting themselves with difficult-to-debug failures. We flipped the order to assure that every stack overflow hit the protection boundary[0], which required us to be much more intentional about the stack versus data split -- but had the added benefit of allowing us to add debugging support for it.[1]
Stack overflows are still pesky (and still a leading cause of task death!), but without the MPU, each one of these stack overflows would be data corruption -- answering for us viscerally what we "need the MPU for."
[0] https://github.com/oxidecomputer/hubris/commit/d75e832931f67...
[1] https://github.com/oxidecomputer/humility#humility-stackmarg...
But wait, it gets worse: stack overflows are often not due to infinite stack consumption (e.g., recursion) but rather simply going deep on an unusual code path. If stack consumption just goes slightly beyond the base of the stack and there is no memory protection, this is corrupt-and-run -- and you are left debugging a problem that looks every bit like a gnarly data race in an unsafe programming language. And this problem becomes especially acute when memory is scarce: you really don't want a tiny embedded system to be dedicating a bunch of its memory to stack space that will never ("never") be used, so you make the stacks as tight as possible -- making stack overflows in fact much more likely.
Indeed, even with the MPU, these problems were acute in the development of Hubris: we originally put the stack at the top of a task's data space, and its data at the bottom -- and we found that tasks that only slightly exceeded their stack (rather than running all of the way through its data and into the protection boundary) were corrupting themselves with difficult-to-debug failures. We flipped the order to assure that every stack overflow hit the protection boundary[0], which required us to be much more intentional about the stack versus data split -- but had the added benefit of allowing us to add debugging support for it.[1]
Stack overflows are still pesky (and still a leading cause of task death!), but without the MPU, each one of these stack overflows would be data corruption -- answering for us viscerally what we "need the MPU for."
[0] https://github.com/oxidecomputer/hubris/commit/d75e832931f67...
[1] https://github.com/oxidecomputer/humility#humility-stackmarg...
> This isn't Rust's fault per se; non-segmented architectures don't have an architecturally defined way to know the stack base.
Bounding stack usage is of course a whole-program concern, but that ought to be feasible when building for embedded, where external "plugin" components are unlikely and there's no inherent need for true separate compilation. (Arguably, in such cases even the need for an actual "stack" structure is actually quite limited; a smarter compiler might well replace many uses of the activation stack with references to static data, reserving it for cases where e.g. reentrancy is actively needed. AIUI, there has been some work along those lines for LLVM.)
Bounding stack usage is of course a whole-program concern, but that ought to be feasible when building for embedded, where external "plugin" components are unlikely and there's no inherent need for true separate compilation. (Arguably, in such cases even the need for an actual "stack" structure is actually quite limited; a smarter compiler might well replace many uses of the activation stack with references to static data, reserving it for cases where e.g. reentrancy is actively needed. AIUI, there has been some work along those lines for LLVM.)
But to get back to your original question: you can see why the state of things today necessitates the use of the MPU, even for Rust?
> non-segmented architectures don't have an architecturally defined way to know the stack base
ARMv8-M does have stack limit registers, which do precisely this. Being a much simpler mechanism than the MPU, they are quicker to switch than the entire MPU state when switching tasks.
ARMv8-M does have stack limit registers, which do precisely this. Being a much simpler mechanism than the MPU, they are quicker to switch than the entire MPU state when switching tasks.
Yes, true -- it's more accurate to say "architectures traditionally don't have a way to know the stack base." Certainly, having an architecturally-defined way to know the base of the stack is/would be really really helpful, and would allow stack overflow to be at once less dangerous and much more debuggable!
Data-execution exploits mean that you may accidentally run code you weren't expecting to.
If you have any kind of message parser taking input from an external system and examining it, you may be vulnerable to these types of exploits without realising it.
Having the ability to mark (for example) your stack as non-executable is a simple change that can remove a whole class of problems.
That is only the skimming the surface tho.
Having the ability to mark (for example) your stack as non-executable is a simple change that can remove a whole class of problems.
That is only the skimming the surface tho.
Yes, this is a concern in memory-unsafe languages like C. But Rust can build memory-safe primitives with only a modest amount of runtime checking wrt. inherently unsafe operations.
(Of course, having better formal models of how 'unsafe' code behaves might allow us to be a lot more rigorous in building safe primitives, such as by endowing some 'unsafe' calls with proof objects that might reify the outcome of a bounds check.)
(Of course, having better formal models of how 'unsafe' code behaves might allow us to be a lot more rigorous in building safe primitives, such as by endowing some 'unsafe' calls with proof objects that might reify the outcome of a bounds check.)
Defence in depth. Rust isn't completely immune to memory safety bugs, due to the need for unsafe code underlying many primitives in the language as well as soundness bugs in the compiler, it just makes them much rarer. It's well worth designing the system so it doesn't fall apart as soon as one such bug appears.
Rust helps, but isnt immune (e.g. stack overflows).
Also, when you're dealing with embedded code, you're very likely going to be using unsafe code (e.g. driver level). With no other protection, incorrectly programming a DMA transfer to trample over code-space is not unknown... and Rust cannot protect you from that.
(Having said that... an MPU window will also not necessarilly catch a bad DMA transfer either).
Also, when you're dealing with embedded code, you're very likely going to be using unsafe code (e.g. driver level). With no other protection, incorrectly programming a DMA transfer to trample over code-space is not unknown... and Rust cannot protect you from that.
(Having said that... an MPU window will also not necessarilly catch a bad DMA transfer either).
> Rust helps, but isnt immune (e.g. stack overflows).
This is a good point but isn't Hubris designed to use statically bounded amounts of memory anyway, like much embedded software? AIUI, this was a key reason for keeping their design focused on synchronized requests, avoiding the hard-to-predict buffering that's needed for supporting 'async' models.
This is a good point but isn't Hubris designed to use statically bounded amounts of memory anyway, like much embedded software? AIUI, this was a key reason for keeping their design focused on synchronized requests, avoiding the hard-to-predict buffering that's needed for supporting 'async' models.
Defense in depth matters. The blog post shows an example of Hubris correctly killing a task that's touching memory it's not supposed to. That "supposed to" was due to a configuration error, but configuration errors can and do happen, and maybe would not be as benign as this one was.
The real question is what is the correct risk tolerance, there are many degrees in here with trade offs. If I mostly trust the other programmers then I'm just concerned about mistakes. If I'm running code that might have been written by some evil cracker trying to do some evil then I need more protection.
That is just one of many reasons and not the main one. Again, actually watch and listen to the associated talks/blogs/docs and so on.
Under the hood, cargo-embed, knurling's probe-run, and Humility are all built atop probe-rs (https://probe.rs/) to provide debugging - I think in this case it's actually a great example of cooperation between projects! Probe-rs has received PRs from both Knurling and Oxide devs; it provides the common interface to use various types of debug hardware and talk to various types of microcontroller cores, essentially replacing OpenOCD.
The github repo of hubris [0] is very nice, as they immediately tell you where what is, e.g.
[0] https://github.com/faithanalog/hubris/tree/pinetime
drv/ contains drivers, a mix of simple driver lib crates and fully-fledged server bin crates. Current convention is that drv/SYSTEM-DEVICE is the driver for DEVICE on SYSTEM (where SYSTEM is usually an SoC name), whereas drv/SYSTEM-DEVICE-server is the server bin crate.
[0] https://github.com/faithanalog/hubris/tree/pinetime
I've never understood why github shows the first line of the most recent commit for files / folders next to their name instead of the first line of any README.md file inside the folder.
Even on projects that I'm actively working on, the most recent commit of a folder gives me no useful information, whereas if I could edit the README.md I could atleast add a description of each folder so that new users could understand the directory structure better.
Even on projects that I'm actively working on, the most recent commit of a folder gives me no useful information, whereas if I could edit the README.md I could atleast add a description of each folder so that new users could understand the directory structure better.
Same thought; I would love if there were a way to toggle between "latest commit message" and "first line of documentation for the file", depending on whether it's a familiar project or a new one I'm browsing.
In Rust, it's because most projects follow the default layout, and so is less needed.
We have a custom build system on top of Cargo, and so things are a bit weird for a normal Rust project, and so it's extra important.
We have a custom build system on top of Cargo, and so things are a bit weird for a normal Rust project, and so it's extra important.
> Why does nobody do this in their readmes normally? They just let you look into their code and tell you "now figure it all out yourself".
I think it's worth praising this repo without knocking others. Open source authors have no obligation to their users; if they don't have the time to, or just don't care to, so organise their READMEs, then they need not. In fact, if it's sufficiently important, for much open-source software anyone else can do it, and submit a patch.
I think it's worth praising this repo without knocking others. Open source authors have no obligation to their users; if they don't have the time to, or just don't care to, so organise their READMEs, then they need not. In fact, if it's sufficiently important, for much open-source software anyone else can do it, and submit a patch.
This isn't just about open source. Developers on internal projects should do this too, but don't.
Have you ever tried to write useful documentation? As a programmer you are far to close to the code. You assume things are obvious that are not, while going into great detail describing things that are obvious (or maybe not obvious, but only rarely interesting and so should be documented in a deeper link not the main documentation). Or worse because you have to document it you document the code lines (++i; // increment i by one).
This is a hard problem. I try to make an effort, but too often I get it wrong.
This is a hard problem. I try to make an effort, but too often I get it wrong.
Of course I have. And of course I recognize that writing good docs is difficult.
I am not talking about the difficulty of writing high-quality explanatory prose. I'm talking about expanding what people consider standard practice, to include a bullet-point description of how the files in the project are laid out. You don't need to be a good writer in order to do this.
That said, just because something is hard doesn't mean it's not a skill that can be learned. Not everyone has the same attitude for writing (much like programming), but generally I believe that "most people can be taught most things to a basic level of competence", and I do not think documentation writing is exempt from that principle. There is no reason why the skill of writing halfway-decent documentation should not be taught in programming courses.
I am not talking about the difficulty of writing high-quality explanatory prose. I'm talking about expanding what people consider standard practice, to include a bullet-point description of how the files in the project are laid out. You don't need to be a good writer in order to do this.
That said, just because something is hard doesn't mean it's not a skill that can be learned. Not everyone has the same attitude for writing (much like programming), but generally I believe that "most people can be taught most things to a basic level of competence", and I do not think documentation writing is exempt from that principle. There is no reason why the skill of writing halfway-decent documentation should not be taught in programming courses.
Cool, almost the same chip is used in the microbit v2. https://support.microbit.org/support/solutions/articles/1900...
Can anyone clue me in on the status of async/await in hubris? I have some HAL code that both really needs and uses async (or callbacks) to work and currently just use my own homebrew WFE/WFI wakers (with a fixed, known a priori number of tasks) but it would be nice to switch to something more elegant and less brittle.
(Context: async/await-based per-task loop to facilitate background LoRa send/receive while other tasks can continue to operate (DMA, SPI, etc) independently, with the overall loop shutting the primary Cortex core down when processing is completed/while waiting for events/updates.)
The last I checked, hubris was strictly synchronous and I didn't get the sense that the interaction between tasks was architectured in such a way that would facilitate low-power designs for battery-powered devices.
(Context: async/await-based per-task loop to facilitate background LoRa send/receive while other tasks can continue to operate (DMA, SPI, etc) independently, with the overall loop shutting the primary Cortex core down when processing is completed/while waiting for events/updates.)
The last I checked, hubris was strictly synchronous and I didn't get the sense that the interaction between tasks was architectured in such a way that would facilitate low-power designs for battery-powered devices.
We don't use async in hubris, so there is no async/await. Hubris is 99.99999% synchronous.
For more: https://hubris.oxide.computer/reference/#_why_synchronous
(Basically, this probably hasn't changed since you looked at it, but for anyone else who hasn't seen it yet...)
> shutting the primary Cortex core down
We don't have mainline support for multicore systems, and there's a big ? in there, design-wise. So yes, you are right to recognize that that is the current state of things.
For more: https://hubris.oxide.computer/reference/#_why_synchronous
(Basically, this probably hasn't changed since you looked at it, but for anyone else who hasn't seen it yet...)
> shutting the primary Cortex core down
We don't have mainline support for multicore systems, and there's a big ? in there, design-wise. So yes, you are right to recognize that that is the current state of things.
Hi Steve. Thanks for the confirmation - obviously a synchronous design doesn't automatically/necessarily preclude most high-level use cases that async can support - it just requires an alternative approach and some careful consideration. In fact, it could make it easier to do many things like power management.
Can you comment on this: others have asked here on HN before but didn't seem to get an answer - does power management/consumption factor in at all in the architectural design (even if it's "this makes it possible to add power management later") or is it strictly a non-goal? (I know your current particular application domains aren't constrained by battery capacity.)
It would be easy enough to hack the kernel to always WFI/WFE if all tasks are exited/completed, but I'm thinking about cases where task(s) are still running but the supervisor process is aware that they're all currently in a busy-waiting state. But that wouldn't be sufficient (on its own) if the OS requires certain peripherals to always remain powered up, uses a high-frequency tick interrupt that will constantly wake the device anyway, etc, etc.
Can you comment on this: others have asked here on HN before but didn't seem to get an answer - does power management/consumption factor in at all in the architectural design (even if it's "this makes it possible to add power management later") or is it strictly a non-goal? (I know your current particular application domains aren't constrained by battery capacity.)
It would be easy enough to hack the kernel to always WFI/WFE if all tasks are exited/completed, but I'm thinking about cases where task(s) are still running but the supervisor process is aware that they're all currently in a busy-waiting state. But that wouldn't be sufficient (on its own) if the OS requires certain peripherals to always remain powered up, uses a high-frequency tick interrupt that will constantly wake the device anyway, etc, etc.
I don't know what our "official" take is, but given that a rack of servers is going to draw, you know, a lot of power, I do think that keeping it simple has been a bigger priority than trying to minimize draw here. It wouldn't have a macro effect on the product's power usage anyway.
Thanks. That logic/reasoning is applicable to a lot of people/products and is totally understandable even if it’s a bit of a bummer. Cheers!
For others, embassy.dev are the main people working on embedded async.
I love projects like this. I've considered getting a smartwatch. Something like this makes the idea more appealing. Whats the most hackable watch? Is there a pinephone of smartwatches?
The article is about running Hubris on the PineTime. Quite literally the Pinephone of watches.
https://www.pine64.org/pinetime/
https://www.pine64.org/pinetime/
It's not the most hackable, but there are some dedicated folks hacking at the Amazfit watches. I have the original Bip, which has a ridiculous battery life (30+ days easily) and GPS. The GPS will run your battery down with extensive use, but if I'm running a few times a week it'll still go 2 weeks.
It isn't that powerful, and I wish it had a second physical button. I haven't explored the mods very much because it does what I need it to do, but the scene has the feel of the early Xbox hacking scene, to me at least.
There's this watch which appears to be heavily inspired from the Amazfit Bip: https://www.kickstarter.com/projects/gfw/banglejs-2-the-open...
It isn't that powerful, and I wish it had a second physical button. I haven't explored the mods very much because it does what I need it to do, but the scene has the feel of the early Xbox hacking scene, to me at least.
There's this watch which appears to be heavily inspired from the Amazfit Bip: https://www.kickstarter.com/projects/gfw/banglejs-2-the-open...
There is! And it's even featured in this article!
PineTime with wasp-os!
> Compare these two videos of writing a solid block of color to the screen, first through the SPI task, and second with direct SPI hardware access:
I wonder if the performance difference could, in some applications, create a preference for processors with enough SPI ports to dedicate one per peripheral (no chip select required). Not only no shared bus (as in I2C), but no shared SPI-task.
I wonder if the performance difference could, in some applications, create a preference for processors with enough SPI ports to dedicate one per peripheral (no chip select required). Not only no shared bus (as in I2C), but no shared SPI-task.
Larger chips (like SoC/Application Processors) certainly have multiple such buses for bandwidth & separation-of-concerns (ie security-related stuff like bootrom vs touchscreen controller) reasons, but I think we're far, far away from accepting the cost of extra pins just to accomodate software concerns.
Thanks for posting. My girlfriend is an embedded Rust developer. She’ll enjoy reading this.
Zephyr can do MPU https://docs.zephyrproject.org/latest/reference/usermode/mem...
Can anyone provide a comparison of PineTime and Pebble? My Time Steel is aging and I'm trying to figure out if the PineTime would be a workable replacement.
The pinetime can basically do anything you want it to, as long as you're willing to make it so. I'm wearing it on my wrist right now, but I have an iPhone and the support for it is not great. In terms of build quality, it's quite good, and the battery lasts for about a week. It's slimmer than an apple watch, which is a major plus for me. But if you have an iphone, step tracking and limited music control is all you'll get in terms of accessory use. You can always contribute, however, and there is a very active community working on the firmware in the github repo (https://github.com/InfiniTimeOrg/InfiniTime).
The thing that the PineTime does really well, however, is beat the price tag of basically every other smartwatch out there ($30). I'm not sure how chip-shortage prices might be affecting this though...
The thing that the PineTime does really well, however, is beat the price tag of basically every other smartwatch out there ($30). I'm not sure how chip-shortage prices might be affecting this though...
Thanks, I am on iOS so this is helpful. All I really care about is notifications, step tracking, and music control. It would also be nice to look at my calendar and check the weather, but those are not critical.
Devs are currently working on notifications, and you can help too if you are so inclined, but they aren't there yet. I am still on the BLE / GATT / ANCS learning curve, so I have not been much help.
Thanks for the shoutout on the twister :)
Thank you for the LoFi tunes. I dig that stuff!
Ultimately, the embedded space (including all sorts of low-level, retro, "bare-metal", remote etc. programming) has common needs and it would be convenient to see more widespread cooperation among these projects, improving reliability and avoiding wasteful duplication of work. More resources at https://github.com/rust-embedded/awesome-embedded-rust