Beej's Guide to Interprocess Communication(beej.us)
beej.us
Beej's Guide to Interprocess Communication
https://beej.us/guide/bgipc/html/
56 comments
There are several more with abbreviated titles,
https://news.ycombinator.com/item?id=29829483 ("Beej's Guide to Unix IPC")—Jan. 2022 (54 comments)
https://hn.algolia.com/?query=beej.us/guide/bgipc
https://news.ycombinator.com/item?id=29829483 ("Beej's Guide to Unix IPC")—Jan. 2022 (54 comments)
https://hn.algolia.com/?query=beej.us/guide/bgipc
Ah thanks! I've augmented the list in my comment.
Could this text be automated for each submission and included at the bottom of the comments? Not as a post by you but as a site feature.
It's harder to automate this than it looks, because (1) you can't just test for identical URLs, and (2) we want to distinguish interesting from uninteresting threads (so that we can just list the interesting ones).
It's definitely on the list to figure out how to build as a site feature though—perhaps a sort of community curation thing.
It's definitely on the list to figure out how to build as a site feature though—perhaps a sort of community curation thing.
Younger crowd might not understand but back in late 90s early 2000 it was rare to find quality content/tutorials that are beginner friendly. Beej's Guide was one of the best quality content back then and even after 20 years it still stand in the first place in the topics covered.
Isn't it still a bit rare? I mean, to get any valuable results, I have to run uBlacklist (a search engine results filter) with a fairly large set of filters.
Having a well-written tutorial on one page, with no monthly fees, licensing, SEO, or ads (and someone who actively responds to emails!)... that's still quite rare, at least in my experience.
Having a well-written tutorial on one page, with no monthly fees, licensing, SEO, or ads (and someone who actively responds to emails!)... that's still quite rare, at least in my experience.
yea, thats why pointed out as still the defacto best still to this day sadly
I got my degree in 2020 and still used Beej's guide for the Socket API
There was nothing like it elsewhere
There was nothing like it elsewhere
I was literally using Beej's networking primer today to build a simple TCP server in C. I haven't written C in a long time, so imagine my surprise when their code compiled and worked on the first try! In Docker!
I had to add an extra sigaction to handle SIGINTs but nothing huge.
Thank you so much for this guide.
If you can't get enough Linux systems fundamentals, give Robert Love's Intro to Linux Systems Programming a try. It goes through IPC (but not in this much detail) and a good swath of Linux fundamentals, all in C.
I had to add an extra sigaction to handle SIGINTs but nothing huge.
Thank you so much for this guide.
If you can't get enough Linux systems fundamentals, give Robert Love's Intro to Linux Systems Programming a try. It goes through IPC (but not in this much detail) and a good swath of Linux fundamentals, all in C.
You can probably thank Linus for that.. https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE
Linus is human sandpaper, but I fucking love this.
> I haven't written C in a long time, so imagine my surprise when their code compiled and worked on the first try! In Docker!
C language is probably one of the most apt languages to write "timeless" software, and the Sockets api have been fairly stable over the years (although i know that the Beej's guide use some newer apis, mostly getaddrinfo).
I was incredibly surprised when going through the source code of gnu make to find some incredibly old C syntax that happily compiled and is used today by pretty much everyone (one way or another).
> If you can't get enough Linux systems fundamentals, give Robert Love's Intro to Linux Systems Programming a try.
Interesting, I will! Thanks!
C language is probably one of the most apt languages to write "timeless" software, and the Sockets api have been fairly stable over the years (although i know that the Beej's guide use some newer apis, mostly getaddrinfo).
I was incredibly surprised when going through the source code of gnu make to find some incredibly old C syntax that happily compiled and is used today by pretty much everyone (one way or another).
> If you can't get enough Linux systems fundamentals, give Robert Love's Intro to Linux Systems Programming a try.
Interesting, I will! Thanks!
Almost done with Beej's Guide to Networking Concepts[1] and absolutely loving it, following along in Go. Gotta check this out and his other guides afterwards
[1]: https://beej.us/guide/bgnet0/html/split/index.html
[1]: https://beej.us/guide/bgnet0/html/split/index.html
I find interprocess and program (or subprogram) or coroutine communication fascinating, as I've said before it seems a large part of the complexity of computing is the "logistics of data" or data movement. (See calling conventions, network protocols, data flow in pipelines as in Linux pipes or Kafka, interfaces, APIs, REST APIs and socket servers as other Beej tutorials) An API is just arranging something into a known position in memory or registers before jumping in control flow and where to go when it is finished.
I have spent a lot of time recently doing hobbyist programming with multithreaded ringbuffers and barriers for concurrency and I'm still looking for a golden abstraction that is easy to reason about - and doesn't suffer from low level mutexes and worrying about memory ownership when transferring data between threads or processes.
I really like the ideas behind Smalltalk about communicating objects and the composition of "behaviours" through communication of objects. I have never used Dbus but I think that's kind of similar.
And I think The COM object model and Mozilla's XPCOM model are similar ideas that were tried which is about a standard protocol for object communication and initialisation.
It reminds me of the perennial userspace vs kernel debate, microkernels vs monolithic kernels. How do you efficiently communicate between programs? And how do you efficiently communicate between parts of the same program without coupling them?
Every programming language defines its own semantics. You kind of need to serialize and deserialize data sent between programs for safety.
A procedure call (CALL instruction) is efficient communication with registers and the stack and it controls flow directly and isn't parallel. A coroutine is efficient communication at runtime between two or more regions of code. What do you think?
Would love to hear what other people are doing in this space and how you've used interprocess communication or even Dbus.
I have spent a lot of time recently doing hobbyist programming with multithreaded ringbuffers and barriers for concurrency and I'm still looking for a golden abstraction that is easy to reason about - and doesn't suffer from low level mutexes and worrying about memory ownership when transferring data between threads or processes.
I really like the ideas behind Smalltalk about communicating objects and the composition of "behaviours" through communication of objects. I have never used Dbus but I think that's kind of similar.
And I think The COM object model and Mozilla's XPCOM model are similar ideas that were tried which is about a standard protocol for object communication and initialisation.
It reminds me of the perennial userspace vs kernel debate, microkernels vs monolithic kernels. How do you efficiently communicate between programs? And how do you efficiently communicate between parts of the same program without coupling them?
Every programming language defines its own semantics. You kind of need to serialize and deserialize data sent between programs for safety.
A procedure call (CALL instruction) is efficient communication with registers and the stack and it controls flow directly and isn't parallel. A coroutine is efficient communication at runtime between two or more regions of code. What do you think?
Would love to hear what other people are doing in this space and how you've used interprocess communication or even Dbus.
Computing is really transformation of energy/information. :)
Programming are the mental models and languages to facilitate that action.
Programming are the mental models and languages to facilitate that action.
I see your profile about energy.
Where do you see God in all this :-)
Where do you see God in all this :-)
Also to expand to your thoughts in context I’ll add that daemons and angels are etymologically related to messages. One can infer that to resemble thoughts. Messages sent from the prime logos.
Then there’s the whole Bible declaring itself as a LLM bit as a recursive definition of god, word, and flesh. So there’s that.
Then there’s the whole Bible declaring itself as a LLM bit as a recursive definition of god, word, and flesh. So there’s that.
We choose to embrace those "messages" or "thoughts" from LIGHT or DARKNESS (I hope you pick LIGHT or good). Those messages come from either place.
As someone who deals with a difficult illness (schizophrenia) I know what it is to receive messages from the darkness which I reject and resist. But I pick LIGHT and LOVE and God.
As someone who deals with a difficult illness (schizophrenia) I know what it is to receive messages from the darkness which I reject and resist. But I pick LIGHT and LOVE and God.
I pick light but any object too close to the light creates darkness for all, balance between the light source so darkness is kept at bay is my style. Ma’at.
I can’t see God, so I can’t know. I do find truth within the slippery thoughts that resemble consciousness and logos as the thing which is always outside of any set, including ones not yet defined. In the sense that when some thing is not yet extant then becomes extant in a set, then clearly logos is outside of all sets. As soon as it is defined it pops outside.
As slippery as that intuition is, I tend to find truth in neither what god is nor isn’t. For any descriptor isn’t. But whatever that is, may or may not be.
In retrospect, God feels similar to a holy donut. a torus both manifested in form at the magnetic plane and invisible in the hyperbolic giving rise to primordial motion. Simultaneously inside and outside.
As slippery as that intuition is, I tend to find truth in neither what god is nor isn’t. For any descriptor isn’t. But whatever that is, may or may not be.
In retrospect, God feels similar to a holy donut. a torus both manifested in form at the magnetic plane and invisible in the hyperbolic giving rise to primordial motion. Simultaneously inside and outside.
I would recommend reading and studying Psalm 19 and see if you can notice the manifest glory, beauty in the world and in yourself. We are fearfully and wonderfully made.
He is simultaneously inside us and we are inside Him like you say and He is Holy and hides Himself from the wisdom of the world.
He is simultaneously inside us and we are inside Him like you say and He is Holy and hides Himself from the wisdom of the world.
Tangential. A file lock is released if the locking process exits. This can be leveraged to make a poor man's semaphore that is automatically raised if the process exits. Slow but solid if you want to control the number of simultaneous crash prone processes that have access to a resource.
Brian (beej71), if you are here, how do you write such concise guides? I remember long ago I started learning network from your guide because TCP/IP illustrated felt like too big to start with.
For people targeting Linux this guide is missing information about Binder and D-Bus which should be what you reach for by default for IPC.
> Binder and D-Bus which should be what you reach for by default for IPC.
Well that’s silly. If the portable options can do it for you, use those.
Well that’s silly. If the portable options can do it for you, use those.
Keep in mind that D-Bus is built on top of Unix sockets so it's just as portable. For IPC I don't think trying to be portable is such a good thing. I think it's nice to do IPC properly for each platform you are targeting.
What I personally find silly is every service rolling there own protocol making it harder for clients to use existing libraries to talk to them and making more work for themselves to get to a quality IPC solution. It usually just ends up in getting something half baked.
What I personally find silly is every service rolling there own protocol making it harder for clients to use existing libraries to talk to them and making more work for themselves to get to a quality IPC solution. It usually just ends up in getting something half baked.
> I think it's nice to do IPC properly for each platform you are targeting.
There’s nothing inherently “proper” to using platform specific options, but perhaps that’s not even the issue here. It’s a beginner tutorial. A “concise overview of various IPC techniques” with examples that “should compile anywhere a good Unix compiler is available.” What you’re talking about isn’t part of the mission.
There’s nothing inherently “proper” to using platform specific options, but perhaps that’s not even the issue here. It’s a beginner tutorial. A “concise overview of various IPC techniques” with examples that “should compile anywhere a good Unix compiler is available.” What you’re talking about isn’t part of the mission.
Much like the other Beej guides, the target isn't that tightly constrained. This aims at POSIX options, and attempts to cover gotchas for cross-platform code.
Not to mention that just because you're only developing for Linux doesn't guarantee you'll have Binder or D-Bus.
Not to mention that just because you're only developing for Linux doesn't guarantee you'll have Binder or D-Bus.
>the target isn't that tightly constrained
Yes, which is why I highlighted that it is incomplete if you are not limiting yourself to just POSIX. It's important for readers to understand the scope of what is contained in the guide as it would be easy to think this is all the IPC mechanisms Linux offers.
>developing for Linux doesn't guarantee you'll have Binder or D-Bus
This is technically true, but everywhere Linux is used will likely already have one of them: embedded, mobile, desktop, server, etc. For D-Bus you can even have it work in a peer to peer mode where you don't need the daemon.
Yes, which is why I highlighted that it is incomplete if you are not limiting yourself to just POSIX. It's important for readers to understand the scope of what is contained in the guide as it would be easy to think this is all the IPC mechanisms Linux offers.
>developing for Linux doesn't guarantee you'll have Binder or D-Bus
This is technically true, but everywhere Linux is used will likely already have one of them: embedded, mobile, desktop, server, etc. For D-Bus you can even have it work in a peer to peer mode where you don't need the daemon.
As someone who has done quite a bit of embedded Linux development, I assure you: neither is as common as you think.
Neither are present in OpenWRT family, for instance.
OpenWRT has developed "ubus" as kind of a dbus-alternative, but it's definitely complicated to work with effectively.
Classic (Beej style) IPC though works fine.
If you are targeting Linux you should probably prefer POSIX IPC to all the funky SysV weirdness involving ftok(3). The same holds for any OS that implements POSIX IPC correctly. Unfortunately, BSD derivates usually have POSIX IPC implemented as a thin wrapper in libc which makes the synchronization primitives usable either between threads or between processes and not both at once (and thus unusable for most interesting applications).
Depends a lot on what you are using IPC for. It seems to me parallelizing computation for performance, isolation/sandboxing, local usage/testing of netowork-able communication, and desktop/system services signaling are the most frequent use cases. Binder and dbus both target and work well for the desktop/system services part, and badly or not at all for the other 3.
What benefits will I gain as a programmer by targeting D-Bus instead of something less complex and generally more portable?
D-Bus is how XDG and Red Hat software gets onto a Linux system. Without dbus, lots of software with braindead design can stay off of your machine.
PIDs, sockets, FIFOs, and other similar constructs can achieve IPC just fine.
D-Bus is how XDG and Red Hat software gets onto a Linux system. Without dbus, lots of software with braindead design can stay off of your machine.
PIDs, sockets, FIFOs, and other similar constructs can achieve IPC just fine.
I think this type of comment ought to have a bit talking about *why* you would reach for it.
It's convenient when most services on the system can all talk the same protocol. It means that services can be more easily secured because they aren't all making other own protocols from scratch. Making your program talk to various services is easier when you can use the same library and there is no need to roll your own because some service decided to make up its own custom protocol for some reason. Imagine if websites would all roll their own protocol instead of using a standard like HTTP. Having a standard IPC protocol also makes debugging easier since it makes sense for their to be real tools for seeing what's going on. Improvements to the implementations handling the protocol can make improvements throughout the whole system. There are network effects in everyone using the same protocol. If you want to step out of this ecosystem and build your own IPC protocol then you need to justify what you are gaining for the value you are giving up.
The downside here is, should any vulnerability be found in the One True Implementation, every consumer is now vulnerable as well.
Just as you shouldn't let a crop go monoculture, your software should not all be built on one or two pieces of tech.
What happens when dbus is replaced? Now you have to move onto the new, broken thing, and change your code to adapt, giving control to others over your program.
There are valid reasons to not just cargo cult dbus.
Just as you shouldn't let a crop go monoculture, your software should not all be built on one or two pieces of tech.
What happens when dbus is replaced? Now you have to move onto the new, broken thing, and change your code to adapt, giving control to others over your program.
There are valid reasons to not just cargo cult dbus.
Do you have any recommendations for dbus?
Using d-feet/d-spy was the most helpful thing for me in wrangling dbus along with the mantra Dbus is dumb, needy, verbose and explicit.
The systemd library for using it is worth looking at.
https://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html
https://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html
I've been using pipe() lately (along with poll()) to tell threads to quit, and other simple inter-thread signalling. Super easy, very little code needed. I guess socketpair() would work just as well.
After writing the same kind of code involving pipes semaphores or whatnot dozen of times, I really enjoy c++20's std::jthread taking care of the cancellation entirely automatically
It would also be useful to mention the POSIX API for shared memory, which is based on memory-mapped files neatly located under /dev/shm.
Wow, I remember following Beej's socket tutorial somewhere around the year 2000 . perhaps even before that.
What's the conventional location for unix domain sockets if XDG_RUNTIME_DIR is unset? /tmp?
That’s where the X11 server puts its socket, but I think the modern convention is /run, though.
~~Repost from 47 days ago https://news.ycombinator.com/item?id=37842863~~
This is the guide on IPC; you've linked the one on network programming.
You are right. My bad.
Beej's System V IPC Guide - https://news.ycombinator.com/item?id=36046014 - May 2023 (1 comment)
Beej's Guide to Unix IPC (2015) - https://news.ycombinator.com/item?id=29829483 - Jan 2022 (54 comments)
Beej's Guide to Unix IPC (2010) - https://news.ycombinator.com/item?id=9619375 - May 2015 (31 comments)
Beejs Guide to Unix IPC - https://news.ycombinator.com/item?id=1525227 - July 2010 (19 comments)
Beej's Guide to Unix Interprocess Communication - https://news.ycombinator.com/item?id=978558 - Dec 2009 (20 comments)