Writing Network Drivers in Go [pdf](net.in.tum.de)
net.in.tum.de
Writing Network Drivers in Go [pdf]
https://www.net.in.tum.de/fileadmin/bibtex/publications/theses/2018-ixy-go.pdf
47 comments
Let's not forget the Biscuit operating system by MIT just recently published at OSDI'18:
- The paper: https://pdos.csail.mit.edu/papers/biscuit.pdf
- The code: https://github.com/mit-pdos/biscuit
- The paper: https://pdos.csail.mit.edu/papers/biscuit.pdf
- The code: https://github.com/mit-pdos/biscuit
Awesome thanks that looks like a very interesting read.
Oh yeah another thing that just showed up on the front page actually, someone writing network drivers for their bachelor's thesis in Go:
https://www.net.in.tum.de/fileadmin/bibtex/publications/thes...
Oh yeah another thing that just showed up on the front page actually, someone writing network drivers for their bachelor's thesis in Go:
https://www.net.in.tum.de/fileadmin/bibtex/publications/thes...
also gvisor which basically is a user-space kernel.
I sincerely wished that Rust had the simplicity of Go, but its syntax just makes me dyslexic.
The nice system programming point would be (imho) clear syntax and safety check mechanics, like the borrow checker.
The nice system programming point would be (imho) clear syntax and safety check mechanics, like the borrow checker.
I think the same exact thing (and considered commenting on it as well, but I feel like I bring rust into everything lately). It's one of my biggest regrets, but at present it just doesn't seem possible to have the benefits rust has without the additional syntactic complexity. Hard to compete with a language that doesn't even have generics (and convinced everyone that was OK, whether it is or not) on the simplicity front. Go 2 is considering generics though so we'll see how that goes.
Things get even worse when you consider that there's already some lifetime elision happening in the general case, so if you were to write some form of "completely specified" rust things would be even worse.
Rust's borrow checker is a game changer, and it's benefit outweighs the lack of clear syntax (it's not worse than C++, or Perl at the very least). Then again, rust doesn't really compete with Go -- it can but I think that's more to do with how awesome rust is. There's currently some efforts to create an opt-in garbage collector called Shifgrethor[0] -- I wonder if programs made to run in that could be run in a syntactically stripped down version of rust that would make it amenable to scripting and golang-like usage. Given rust's macro system (which would be a crazy way to implement this imaginary functionality instead of a compiler plugin), this seems possible...
I'm already just a tad bit over Go for personal use (though it's still in my top 5), and often reach for rust because:
- I've paid the cost to somewhat understand the borrow checker (and I could see it as valuable)
- rust ecosystem is getting pretty big these days (lots of libs)
- community is great (for now, I actually think it is inevitable for it to degrade over time)
- generics when I need them, traits are fantastic (basically they're almost just like haskell typeclasses)
- rust can go from embedded systems (xMhz) to browser (via wasm), if I'm good at it I can almost literally do anything. It's got the macro support to make different domains easier (so embedded vs browser contexts can feel different if basic rust wasn't good enough)
I'm sold on Rust, but I don't think it has a chance to replace Java -- it's probably going to replace C/C++ though, but that will take longer because that world moves slower anyway.
[0] https://boats.gitlab.io/blog/post/shifgrethor-iv/
Things get even worse when you consider that there's already some lifetime elision happening in the general case, so if you were to write some form of "completely specified" rust things would be even worse.
Rust's borrow checker is a game changer, and it's benefit outweighs the lack of clear syntax (it's not worse than C++, or Perl at the very least). Then again, rust doesn't really compete with Go -- it can but I think that's more to do with how awesome rust is. There's currently some efforts to create an opt-in garbage collector called Shifgrethor[0] -- I wonder if programs made to run in that could be run in a syntactically stripped down version of rust that would make it amenable to scripting and golang-like usage. Given rust's macro system (which would be a crazy way to implement this imaginary functionality instead of a compiler plugin), this seems possible...
I'm already just a tad bit over Go for personal use (though it's still in my top 5), and often reach for rust because:
- I've paid the cost to somewhat understand the borrow checker (and I could see it as valuable)
- rust ecosystem is getting pretty big these days (lots of libs)
- community is great (for now, I actually think it is inevitable for it to degrade over time)
- generics when I need them, traits are fantastic (basically they're almost just like haskell typeclasses)
- rust can go from embedded systems (xMhz) to browser (via wasm), if I'm good at it I can almost literally do anything. It's got the macro support to make different domains easier (so embedded vs browser contexts can feel different if basic rust wasn't good enough)
I'm sold on Rust, but I don't think it has a chance to replace Java -- it's probably going to replace C/C++ though, but that will take longer because that world moves slower anyway.
[0] https://boats.gitlab.io/blog/post/shifgrethor-iv/
I've been using rust for a few weeks and I'm finding it fairly pleasant overall, but I agree on the syntax issues. I've hit some snags (in particular, graphs, but this is a common snag on constrained languages, Haskell has trouble with them too) but overall nothing serious and I'm not afraid to go down to unsafe.
I think they made a mistake using C++ style generic notation. D lang has a much, much nicer syntax for generics, and the language's readability would much improved with that syntax imo. As is, I find parsing the error messages that come back from type mismatches quite challenging. Reading documentation on traits on docs.rs, I find the actual trait name very hard to pick out in the noise. Maybe I'll get better with that over time.
I think they made a mistake using C++ style generic notation. D lang has a much, much nicer syntax for generics, and the language's readability would much improved with that syntax imo. As is, I find parsing the error messages that come back from type mismatches quite challenging. Reading documentation on traits on docs.rs, I find the actual trait name very hard to pick out in the noise. Maybe I'll get better with that over time.
It looks like this is not a pure Go implementation:
"We originally implemented these functions in pure Go and tried to force reads and writes through empty C calls as Go does not have the volatile key word or an equivalent for our needs as far as we know. This resulted in strange behaviour: while initialization seemed to work, including the corresponding register getters and setters, the program failed later when the NIC never set flags that were necessary to continue. We invested a lot of time in this specific problem but could not find a reason for this behaviour. With the help of cgo, which enables the use of C code in Go programs, we imported the corresponding functions in device.h alongside log.h of the original C driver and replaced our go functions. This fixed the aforementioned problem though we never were able to provide an exact explanation due to limited time constraints."
It's still an interesting effort, and it makes me wonder if we'll ever see a day when a significant portion of Linux is written in a safer language.
"We originally implemented these functions in pure Go and tried to force reads and writes through empty C calls as Go does not have the volatile key word or an equivalent for our needs as far as we know. This resulted in strange behaviour: while initialization seemed to work, including the corresponding register getters and setters, the program failed later when the NIC never set flags that were necessary to continue. We invested a lot of time in this specific problem but could not find a reason for this behaviour. With the help of cgo, which enables the use of C code in Go programs, we imported the corresponding functions in device.h alongside log.h of the original C driver and replaced our go functions. This fixed the aforementioned problem though we never were able to provide an exact explanation due to limited time constraints."
It's still an interesting effort, and it makes me wonder if we'll ever see a day when a significant portion of Linux is written in a safer language.
Actually cgo, had some performance issues. Whenever a call to C program is made from go program, there is lot of work needs to be done.
Go authors have thought to do initialization and set up the work for only once, but I don't think it is implemented yet.
Go authors have thought to do initialization and set up the work for only once, but I don't think it is implemented yet.
> It's still an interesting effort, and it makes me wonder if we'll ever see a day when a significant portion of Linux is written in a safer language.
I really hope Ada makes a comeback, especially in commercial kernel programming.
I really hope Ada makes a comeback, especially in commercial kernel programming.
What's your opinion on Rust? It seems to be successfully filling the niche Ada could not, even if Ada's problems were more political than technical.
I hope Rust or Rust-like languages make it into the kernel, but I think it's worth noting that a huge problem right now is the toolchain and its dependencies. Go may have a better story here though I wouldn't bet on it being a more likely choice for kernel mode.
What is the problem with the Rust toolchain?
Nothing at all, really, but kernel developers are going to be picky. For example, does LLVM target all of the architectures Linux supports today? Also, I think it's hard to sell Rust as a hard dependency of the Linux kernel when it isn't nearly as widespread as almost every other kernel dependency.
I also sorta remember some kernel developers (not necessarily Linux) making a fuss about rustc using a huge amount of address space and not working on 32-bit systems for that reason - although frankly, I can't remember where I read that and I'm starting to question how much I'm misremembering there.
I also sorta remember some kernel developers (not necessarily Linux) making a fuss about rustc using a huge amount of address space and not working on 32-bit systems for that reason - although frankly, I can't remember where I read that and I'm starting to question how much I'm misremembering there.
> I also sorta remember some kernel developers (not necessarily Linux) making a fuss about rustc using a huge amount of address space and not working on 32-bit systems for that reason - although frankly, I can't remember where I read that and I'm starting to question how much I'm misremembering there.
You might be thinking of Theo de Raadt: https://marc.info/?l=openbsd-misc&m=151233345723889&w=2
You might be thinking of Theo de Raadt: https://marc.info/?l=openbsd-misc&m=151233345723889&w=2
Yep, that's exactly what I was remembering. Though I disagree that rewriting coreutils in Rust is a good place to start, there are definitely some valid points in there. I wonder if the i386 situation has been resolved at this point.
I doubt the Linux kernel will ever allow non-C code, so the toolchain is not really a concern.
And I hope other projects don't trade off safety/productivity/etc. for portability to obscure hobby architectures.
And I hope other projects don't trade off safety/productivity/etc. for portability to obscure hobby architectures.
Well, if you want to be pedantic, there's already non-C code in the kernel, it just doesn't make it into the runtime. I've seen Linus's opinion on C++ in the kernel, but I've never seen anyone explicitly state that no other languages would be accepted. I'm sure that it's something that's always possible even if very unlikely right now.
Blanketing every architecture LLVM doesn't support that GCC does an "obscure hobby architecture" is not really a great way to look at things imo. I haven't looked into it but I'm pretty sure outside the bubble of desktop computers there are plenty of important uses of Linux on less common architectures, I would assume mostly embedded systems.
Blanketing every architecture LLVM doesn't support that GCC does an "obscure hobby architecture" is not really a great way to look at things imo. I haven't looked into it but I'm pretty sure outside the bubble of desktop computers there are plenty of important uses of Linux on less common architectures, I would assume mostly embedded systems.
Same question: I am not sure go has solved the dependency problem at all.
I'm not a kernel programmer, and I haven't used Rust before, so take that into consideration.
Some of my favourite things about Ada is it's approach to concurrency, which (I think) is far superior to other imperative concurrent languages like Go. We should never have to deal with low-level primitives like semaphores and mutexes - it's 2018! Why are we stuck in the 80s?
I've heard that concurrency is difficult in Rust, but I've never used it, so I can't comment. I plan on learning very soon!
Some of my favourite things about Ada is it's approach to concurrency, which (I think) is far superior to other imperative concurrent languages like Go. We should never have to deal with low-level primitives like semaphores and mutexes - it's 2018! Why are we stuck in the 80s?
I've heard that concurrency is difficult in Rust, but I've never used it, so I can't comment. I plan on learning very soon!
Interesting comment...considering concurrency in Rust is significantly safer to do.
This is the whole point of the borrow checker which is to ensure data-race free code.
The story with concurrency is still being worked out with async server-side applications which is coming together swimmingly!
This is the whole point of the borrow checker which is to ensure data-race free code.
The story with concurrency is still being worked out with async server-side applications which is coming together swimmingly!
> This is the whole point of the borrow checker which is to ensure data-race free code.
That's awesome! I never knew Rust had safer concurrency. I'm quite keen to learn it - just waiting for the time.
That's awesome! I never knew Rust had safer concurrency. I'm quite keen to learn it - just waiting for the time.
Not sure why I'm downvoted for this. I struggle to think of a "modern" imperative language that offers similar levels of safety and concurrency, aside from possibly Rust. Erlang is fantastic, but not imperative.
Go has channels, which is nice enough for message-passing behaviour, but nothing for elegantly sharing memory. ("Share memory by communicating" doesn't always work)
Things like conditional critical regions, monitors, and protected objects are much better abstractions for concurrency.
Go has channels, which is nice enough for message-passing behaviour, but nothing for elegantly sharing memory. ("Share memory by communicating" doesn't always work)
Things like conditional critical regions, monitors, and protected objects are much better abstractions for concurrency.
> I struggle to think of a "modern" imperative language that offers similar levels of safety and concurrency, aside from possibly Rust. Erlang is fantastic, but not imperative.
D
D
Doesn't D just have message passing & mutexes?
Which is fine, since D is supposed to be a better C.
I wouldn't consider its concurrent primitives comparable to Ada. This is to be expected, since I don't believe they aim for such a nuanced level of concurrency. The Ada runtime is incredibly huge and complex - it was created by the US military, after all.
Which is fine, since D is supposed to be a better C.
I wouldn't consider its concurrent primitives comparable to Ada. This is to be expected, since I don't believe they aim for such a nuanced level of concurrency. The Ada runtime is incredibly huge and complex - it was created by the US military, after all.
One of the talking points of Rust is fearless concurrency.
Concurrency in Rust is not difficult (neither complex nor error-prone), but you have some high-level primitives that you must know how to choose and use.
Concurrency in Rust is not difficult (neither complex nor error-prone), but you have some high-level primitives that you must know how to choose and use.
We should never have to deal with low level primitives like functions and variables - it’s 2018 why are we stuck in the 50s?
That is an incorrect analogy. Semaphores are the "goto" statement of the concurrent world.
Semaphores are not typed. They is no semantic connection to the thing they protect. Debugging concurrent systems with mutual exclusion enforced through these low-level primitives is a nightmare.
If you forgot to lock/unlock, you'll never know! The compiler cannot check it. Deadlocks are imminent.
We've had better abstractions since the 80s. Conditional critical regions, monitors, and (ideally) protected objects. Hell, even guards are better than semaphores and mutexes - at least they are associated with their critical region!
I struggle to think of situations where a semaphore is the best choice. Maybe in extremely high-performance cases, but even then the compiler can usually make a better decision.
Semaphores are not typed. They is no semantic connection to the thing they protect. Debugging concurrent systems with mutual exclusion enforced through these low-level primitives is a nightmare.
If you forgot to lock/unlock, you'll never know! The compiler cannot check it. Deadlocks are imminent.
We've had better abstractions since the 80s. Conditional critical regions, monitors, and (ideally) protected objects. Hell, even guards are better than semaphores and mutexes - at least they are associated with their critical region!
I struggle to think of situations where a semaphore is the best choice. Maybe in extremely high-performance cases, but even then the compiler can usually make a better decision.
Well, Alan Key pretty much said that in the 70s, and has stuck to his high level guns all this time.
We can start with nodejs. Nodejs with async model will be the best solution to achieve the best performance(after 'delete spaces' method of course).
The libuv library Node makes us of for handling async calls is written in C.
It's not a problem. The main problem is the assembly code. There is no way to inline it in JS code. After boot time system should make some things like A20, enable paging/protected and long mode, build GDT/IDT.
Right, but my reply to the parent was that Node isn't a performance improvement over C because it has async, when the async part is written in C. JS popularity has led to some wild claims. Never thought I'd live to see the day someone proposed a JS runtime would be more performant than C for writing operating systems.
Performance isn't an issue with C code. Security is.
It can be. Unless one resorts to macros, templates are usually faster.
Everything can be an issue everywhere. No one will choose Node JS over C to counter performance.
No, but one can get code to perform as fast or faster than C with any other systems programming language, especially if they have templates/generics.
More likely to see something like Redox take off.
These guys are so positively insane! :) See https://github.com/ixy-languages/ixy-languages#overview
If nothing else this would be a fun and easy way to develop a driver from scratch for unfamiliar hardware, even if you intended to port the whole thing back to C in the end, after it was all figured out.
Contrast this with https://news.ycombinator.com/item?id=18405515, which is trending at the top.
How do they handle the GC?
The driver only allocates memory on the stack, so no GC is required.
If you're not doing type theory work or very specific exploration in a specific field, I don't know another language that is so easy to get up and running with. On top of all this go's performance is pretty great/consistent as well, JS is the only other scripting language that comes close generally, but then you have to submit to the event loop.
I've always thought one of Go's true killer features is how hard it makes cleverness.
I'm starting to seriously think Go's popularity is going to continue shooting up. As I always note I think Go will supplant Java as the incumbent in the enterprise software in one or two decades max.
[EDIT] - Forgot to include the most recent example that made me think this: WPaxos(AKA WAN Paxos)[0] and the included github repo[1]. To be fair CASPaxos[2] included gryadka[3] which is in JS.
[0]: https://cse.buffalo.edu/tech-reports/2017-03.pdf
[1]: https://github.com/ailidani/paxi
[2]: https://arxiv.org/abs/1802.07000
[3]: https://github.com/gryadka/js