How to add eBPF observability to your product(brendangregg.com)
brendangregg.com
How to add eBPF observability to your product
https://brendangregg.com/blog/2021-07-03/how-to-add-bpf-observability.html
31 comments
Technically, yes, but in practice not really; eBPF started as an extension of the packet filtering framework and then got generalized to be a nearly-universal debugging/monitoring/tracing system that's only sometimes used for packets. https://ebpf.io/ is a decent starting point for reading.
My biggest pet peeve is when authors don't just start with a 1-2 sentence outline of the topic that they're discussing, and instead go directly into acronym soup. And I've been in software development for a long time and have never heard of BPF/eBPF before.
Thank you for your explanation.
Thank you for your explanation.
Brendan Gregg is the author of eBPF, and he writes on his blog. It is fair to assume a basic knowledge of his works when reading his website.
Maybe I’m misunderstanding what you mean by “author“, but BPF was invented by Steven McCanne,
and Van Jacobson in 1992[0], and tweaked over the years by various developers and operating systems. It looks[1] like the Linux eBPF work was initiated by Alexei Starovoitov, and came to be exposed to users c. 2014.
[0] https://en.wikipedia.org/wiki/Berkeley_Packet_Filter
[1] https://lwn.net/Articles/740157/
[0] https://en.wikipedia.org/wiki/Berkeley_Packet_Filter
[1] https://lwn.net/Articles/740157/
Brendan Gregg is the author of most of the utilities, not bpf/ebpf itself. You can find a non-exhaustive list here: http://www.brendangregg.com/ebpf.html
I get that, and appreciate he’s productive and sharing his skill. I wasn’t sure though if the long history of other people’s hard work wasn’t being swept up in appreciation for ‘brendangregg
Thanks; I also summarize the history of eBPF in my BPF Performance Tools book (which I had checked by Alexei & Dan) and list the main contributors in the Acknowledgements page, as well as include them in talks (e.g. https://www.slideshare.net/brendangregg/bpf-internals-ebpf/1...). I do try hard to thank all involved. :-)
Modern BPF (eBPF) was created by Alexei Starovoitov and Daniel Borkmann, who are still maintainers but are joined by over a hundred contributors. I've spent most of my time contributing to the observability frontends, bcc and bpftrace. Apart from the people, the major companies contributing include Facebook and Isovalent; Microsoft are becoming its own major contributor to its own Windows implementation.
Modern BPF (eBPF) was created by Alexei Starovoitov and Daniel Borkmann, who are still maintainers but are joined by over a hundred contributors. I've spent most of my time contributing to the observability frontends, bcc and bpftrace. Apart from the people, the major companies contributing include Facebook and Isovalent; Microsoft are becoming its own major contributor to its own Windows implementation.
>Microsoft are becoming its own major contributor to its own Windows implementation.
Wait, what now !?
After checking : he wasn't joking[0]. Good job MS.
[0]:https://github.com/microsoft/ebpf-for-windows
Wait, what now !?
After checking : he wasn't joking[0]. Good job MS.
[0]:https://github.com/microsoft/ebpf-for-windows
I'm sorry, this indeed isn't correct. I cannot modify my comment for some reason now.
Brendan Gregg literally wrote the book on this kind of stuff, if you search for any of this tracing stuff it comes up with his website anyway.
Sorry, I just added a link to https://ebpf.io. Although I suspect that will also annoy people as it doesn't explain the BPF acronym: The reason is that we're now treating BPF as a technology name (like LLVM) and not an acronym, as the historical acronym (Berkeley Packet Filter) is plain confusing.
I think the real problem is that BPF was built by engineers without professional marketing help, who would have come up with a better name!
I think the real problem is that BPF was built by engineers without professional marketing help, who would have come up with a better name!
Thank you!
He did link to something, but for me that link was broken (https://brendangregg.com/blog/2021-07-03/BPF)
That's more because you're coming into an established site with an established topic that's been discussed for years. It doesn't really make sense to introduce every topic from scratch every time, because the assumption after a while is that the people reading have at least some prior knowledge or at least an interest in looking up what they don't know because they're there for a reason.
In other words, not every site is designed around link aggregators like HN, and some things will be assumed, just as if you open a book to some random chapter.
In other words, not every site is designed around link aggregators like HN, and some things will be assumed, just as if you open a book to some random chapter.
Thanks for explaining that - I was totally missing the point to be honest.
Yeah, the name is kinda unfortunate in the modern context, but in fairness it's easy to not expect that a packet filtering framework will turn out to be the basis for a universal kernel tracing system...
And eBPF stands for extended Berkeley Packet Filter:
> eBPF is an extended BPF JIT virtual machine in the Linux kernel
It's really quite an amazing tool, especially since you can use it to run sandboxed programs in kernel without changing kernel source or loading a kernel module. These programs can be written using a limited dialect of C.
Some examples of its use can be found here: https://github.com/iovisor/bcc/tree/master/examples
> eBPF is an extended BPF JIT virtual machine in the Linux kernel
It's really quite an amazing tool, especially since you can use it to run sandboxed programs in kernel without changing kernel source or loading a kernel module. These programs can be written using a limited dialect of C.
Some examples of its use can be found here: https://github.com/iovisor/bcc/tree/master/examples
eBPF is more capable of BPF. It's a general hooking mechanism in the Linux kernel. Think Aspect-Oriented Programming, but in the kernel. The biggest benefit I think is that it can intercept and modify function calls in many places, not just the kernel. Intercepting/filtering bad data at the network driver level is much more efficient than letting the data reaching the kernel code.
For me, BPF is the most exciting development in the kernel world for ages.
Brendan Gregg, the author of the posted blog entry, works with BPF on observability at Netflix and delivered a keynote at UbuntuMasters 2019. The video is on his blog is a great intro. [0]
I've been watching BPF for a couple of years now, and it seems slow on the uptake, but I hope Gregg is right that people will eventually start writing new drivers, firewalls, observability and security tools, loaded from userspace, but all running safely in a kernel vm, maybe even written in new async-first programming languages.
[0] https://brendangregg.com/blog/2019-12-02/bpf-a-new-type-of-s...
Brendan Gregg, the author of the posted blog entry, works with BPF on observability at Netflix and delivered a keynote at UbuntuMasters 2019. The video is on his blog is a great intro. [0]
I've been watching BPF for a couple of years now, and it seems slow on the uptake, but I hope Gregg is right that people will eventually start writing new drivers, firewalls, observability and security tools, loaded from userspace, but all running safely in a kernel vm, maybe even written in new async-first programming languages.
[0] https://brendangregg.com/blog/2019-12-02/bpf-a-new-type-of-s...
This works great if you control the servers or have a uniform fleet. I used to work at a security vendor which sold ebpf based software that ran as a daemonset on customer kubernetes clusters. Compiling ebpf bytecode on the customer's hosts wasn't an option.
Iirc the real challenge was with writing kprobe ebpf functions that access native structs. I don't think we ever found a good solution for that because you need kernel headers for each machine which we didn't have.
Of course if I'm missing something obvious, do tell!
(I'm the same person who left this comment on Brendan's site as well. Not a random copy paster)
Iirc the real challenge was with writing kprobe ebpf functions that access native structs. I don't think we ever found a good solution for that because you need kernel headers for each machine which we didn't have.
Of course if I'm missing something obvious, do tell!
(I'm the same person who left this comment on Brendan's site as well. Not a random copy paster)
I think that's what BTF and CO-RE are for. They are new; they may not have existed when you had this problem. https://brendangregg.com/blog/2020-11-04/bpf-co-re-btf-libbp...
Yes, BTF provides the kernel struct info in the vmlinux file. Like debuginfo, but far more lightweight.
Of course, tracepoints are better, as they are stable and don't need BTF (although raw_tracepoints do), but there's often cases where they don't exist for the thing I am tracing (and arguably shouldn't exist, if it's too niche/hot-path).
Of course, tracepoints are better, as they are stable and don't need BTF (although raw_tracepoints do), but there's often cases where they don't exist for the thing I am tracing (and arguably shouldn't exist, if it's too niche/hot-path).
Yeah, I'm familiar with them. Once they're widespread this will be a whole different story. (Here too, if you control the fleet of servers then this is easy. If you're deploying software as a vendor you're stuck with older kernel versions.)
The Linux kernel can be configured at build time to expose its own headers as /proc/kheaders.tar.xz. Not sure if distribution kernels generally have this option enabled or not.
I've been on and off playing with the same scripts described in this article for the last few months (installable as bcc-tools under Fedora). While some of them are a little unstable or need a bit of Python version handholding in fc34, I've been blown away with the data I've been able to get with seemingly negligible overhead. I was also really surprised to find them in the default repos, and how relatively few people reference them when discussing other eBPF tooling (e.g. Pixie).
As someone who has been building observability software for the last few years I'm ridiculously excited about eBPF. Looking forward to seeing what the Netflix dashboards mentioned in the post look like & the data pipelines that support them.
As someone who has been building observability software for the last few years I'm ridiculously excited about eBPF. Looking forward to seeing what the Netflix dashboards mentioned in the post look like & the data pipelines that support them.
BPF: "The Berkeley Packet Filter is a technology used in certain computer operating systems for programs that need to, among other things, analyze network traffic. It provides a raw interface to data link layers, permitting raw link-layer packets to be sent and received." -Wikipedia
Unfortunately that description is so old it's misleading. I described BPF in another comment here.
I'm not an expert in this field so maybe I'm missing some important context here, but this idea of preventing people from porting your software seems odd. Who cares if people make shitty ports? If they're not good then no one will use them. The whole thing also reads a little bit condescending.
Any time there's some popular public API, people will copy other people's projects built around this API and make them "better". I don't think Daniel Stenberg (author of cURL) is upset that there's HTTPie. I don't think it hurts cURL and I don't think it hurts the developer community as a whole.
I guess one could argue it would make life easier if there was just one HTTP library (or one eBPF library I guess). But that's just unrealistic, that's not how people work. Plus it removes the competition aspect which is nice to have for any project.
Is there any examples of open source projects that died because people were making too many ports?
Any time there's some popular public API, people will copy other people's projects built around this API and make them "better". I don't think Daniel Stenberg (author of cURL) is upset that there's HTTPie. I don't think it hurts cURL and I don't think it hurts the developer community as a whole.
I guess one could argue it would make life easier if there was just one HTTP library (or one eBPF library I guess). But that's just unrealistic, that's not how people work. Plus it removes the competition aspect which is nice to have for any project.
Is there any examples of open source projects that died because people were making too many ports?
> Is there any examples of open source projects that died because people were making too many ports?
Yes, Linux tracing.
I entered in 2014 when there were 14 different tracers, none of them finished (ie, does everything and works on all distros). It wasn't a lack of engineers -- there are plenty of smart engineers in tracing. But they were scattered among those 14 tracers. It was a mess, and no one could understand why Linux didn't have a mature tracing solution. (I think this also answers your question: Why cares if people make shitty ports?)
So I started thinking I could fix it by creating my own 15th tracer...Just kidding! What a terrible idea. Instead I joined one of those 14 existing tracers (a new one called BPF) and worked hard to get others on board, and do what we hadn't done well before -- collaborate!
Early on I wanted to do my own front-end, but Alexei convinced me to collaborate on bcc. I also wanted to do my own language, but got people to collaborate on the existing bpftrace instead.
The entire journey of BPF observability has been one of us choosing NOT to keep spinning up competing projects and instead to collaborate together on one. And that's how we succeeded in delivering a tracer that does everything on all distros.
I should add you might be missing a bit of context here too, and that's that tracing is maintenance heavy: You don't develop it once and then it works forever. It needs constant updates to match kernel changes etc.
Yes, Linux tracing.
I entered in 2014 when there were 14 different tracers, none of them finished (ie, does everything and works on all distros). It wasn't a lack of engineers -- there are plenty of smart engineers in tracing. But they were scattered among those 14 tracers. It was a mess, and no one could understand why Linux didn't have a mature tracing solution. (I think this also answers your question: Why cares if people make shitty ports?)
So I started thinking I could fix it by creating my own 15th tracer...Just kidding! What a terrible idea. Instead I joined one of those 14 existing tracers (a new one called BPF) and worked hard to get others on board, and do what we hadn't done well before -- collaborate!
Early on I wanted to do my own front-end, but Alexei convinced me to collaborate on bcc. I also wanted to do my own language, but got people to collaborate on the existing bpftrace instead.
The entire journey of BPF observability has been one of us choosing NOT to keep spinning up competing projects and instead to collaborate together on one. And that's how we succeeded in delivering a tracer that does everything on all distros.
I should add you might be missing a bit of context here too, and that's that tracing is maintenance heavy: You don't develop it once and then it works forever. It needs constant updates to match kernel changes etc.
Thanks for the detailed response, this makes a lot of sense!
[1]: https://en.m.wikipedia.org/wiki/Berkeley_Packet_Filter