BPF Performance Tools: Linux System and Application Observability(brendangregg.com)
brendangregg.com
BPF Performance Tools: Linux System and Application Observability
http://www.brendangregg.com/blog/2019-07-15/bpf-performance-tools-book.html
38 comments
There are still people working on packet filtering! The BPF enhancements are benefiting lots of different areas of the Linux world at the same time.
Cilium [1] is a great example of how the new capabilities of BPF can be used for more complex packet filtering, such as understanding higher level protocols.
[1] https://cilium.io/
Cilium [1] is a great example of how the new capabilities of BPF can be used for more complex packet filtering, such as understanding higher level protocols.
[1] https://cilium.io/
Really looking forward to this, although I'm not a systems programmer I've always found OS internals/performance tuning fascinating.
How does this compare to "Linux Observability with BPF"? Should one be read first?
I would definitely go for the Brendan Gregg book. He's a long term BPF contributor and his book seems much more in-depth compared to the Oreilly one which only has 200 pages compared to >700, and if you look at the Oreilly book's TOC it appears it's trying to cover everything BPF (not just tracing) in just 200 pages. Definitely Brendan Gregg book's seems a better choice as far as I can tell.
Besides the size of the book Brendan Gregg is pretty much the current expert in Linux performance and has been for some time. If the book is anything like the quality of his blog (which I am sure it will, perhaps even surpass) then I will surely be buying it.
I have multiple books of the same subjects Python, Go, FreeBSD etc.. due to different writing styles and explanations, so for me to grok BPF tracing techniques and performance evaluation at a higher level I would need to read both.
are all the scripts/tools Brendan is talking about going to be open source? (I'm sure some of them are, including the ones on his Github)
Yes. I've been working on a github repo that contains all the tools. It's not online yet.
thank you so much, looking forward to the book
Brendan, thanks for writing the book. Will definitely get this. As someone looking to always expand my skillset, do you think there will be a lot of demand for programmers who know BPF in the future/now? I'm always interested in low-level OS/internals. Current job is in HW design.
Yes, but what exactly varies on the role. Eg:
- Developers and system engineers: will add the BCC and bpftrace tools to their toolkit for analyzing issues, but they won't necessarily need to develop new ones themselves (so long as someone at the company can, as needed). The book targets this (large) audience by covering over 150 tools.
- Some developers will use the BPF tracers like bpftrace as a primary debugging tool as it's the best available for their language (eg, user and kernel C and C++); other mature runtimes may already have a tracer that developers are already using, making bpftrace a secondary debugging tool for digging deeper. In that case, you may find there is one or two people on a development team who learn bpftrace and deal with the deeper debugging, whereas others get by with their primary tracer. The book targets these audiences too, by explaining bpftrace programming and including many examples.
- There's another smaller audience where there'll be BPF demand that is not the focus of the book: custom BPF products. In the past, a monitoring/security company might sell you a loadable kernel module to do their thing; in the future, those should all become BPF programs (where possible), which are safer to run. There will also be new applications made possible by BPF. Some of these products will be bpftrace/BCC-based, and the book will be helpful. So I'd expect to see some demand for these BPF skills too, although it's a much narrower audience than the others. It'll be companies trying to hire BPF programmers to migrate their kernel modules to. Some companies already sell such BPF things (Cilium, Sysdig).
- Developers and system engineers: will add the BCC and bpftrace tools to their toolkit for analyzing issues, but they won't necessarily need to develop new ones themselves (so long as someone at the company can, as needed). The book targets this (large) audience by covering over 150 tools.
- Some developers will use the BPF tracers like bpftrace as a primary debugging tool as it's the best available for their language (eg, user and kernel C and C++); other mature runtimes may already have a tracer that developers are already using, making bpftrace a secondary debugging tool for digging deeper. In that case, you may find there is one or two people on a development team who learn bpftrace and deal with the deeper debugging, whereas others get by with their primary tracer. The book targets these audiences too, by explaining bpftrace programming and including many examples.
- There's another smaller audience where there'll be BPF demand that is not the focus of the book: custom BPF products. In the past, a monitoring/security company might sell you a loadable kernel module to do their thing; in the future, those should all become BPF programs (where possible), which are safer to run. There will also be new applications made possible by BPF. Some of these products will be bpftrace/BCC-based, and the book will be helpful. So I'd expect to see some demand for these BPF skills too, although it's a much narrower audience than the others. It'll be companies trying to hire BPF programmers to migrate their kernel modules to. Some companies already sell such BPF things (Cilium, Sysdig).
Love all the work Brenden has done on BPF. It's made my life much easier when working on Linux systems. I will definitely be getting this book.
Does anyone know if MacOS is ever going to provide more BPF support? I've tried piping some of the Linux BPF tools to Mac and have ultimately been frustrated on many fronts.
macOS has DTrace, a powerful tool created within Solaris and later implemented both in macOS and FreeBSD.
Yes, I've used DTrace extensively. Unfortunately the tracepoints are limited and I've found that the MacOS kernel functions aren't quite the same as what you can find for Darwin/BSD. I think Apple is just uncomfortable with the idea of people poking around in the kernel more than they'd like.
this is awesome news. I like Brendan's System Performance book a lot. It will be a nice addition to my library and a good read.
Would the final revision include any chapters by Sasha Goldstein? I’d love to get his insights on it.
Sasha has written excellent BPF docs, tutorials, and tools, and could certainly write his own book on BPF. He's had access to my draft, but he's been busy with his new job.
BPF / eBPF is the reason I've finally started to learn C. I never had an actual use for it in the past, so it never really stuck (and mostly I got good at producing code that would segfault), but there's things I can definitely use BPF for.
Hi, Brendan, will your new book useful for DevOps|SRE engineers, how do you think?
should be linked to another BPF book by David Calavera and Lorenzo Fontana discussed recently:
https://news.ycombinator.com/item?id=20412395
https://news.ycombinator.com/item?id=20412395
Great news. This book looks super interesting.
shut up and take my money already !
Who ever has been downvoting this isn't getting the Futurama reference[1]. That was about the "eye"-phone. For the uninitiated, it means that they want the product faster.
Seriously, its a (humorous) vote of confidence in Brendan's work.
[1] https://knowyourmeme.com/memes/shut-up-and-take-my-money
Seriously, its a (humorous) vote of confidence in Brendan's work.
[1] https://knowyourmeme.com/memes/shut-up-and-take-my-money
Can’t wait to get a copy!
If you have a Safari subscription, you can read the rough edit now!
Pro-tip: You can get free access to Safari by becoming a member of the ACM.
Pro-tip: You can get free access to Safari by becoming a member of the ACM.
It is not on Safari right now, but should be soon.
Good tip!
Good tip!
Will there be a way to buy an ebook?
Oreilly only sells subscriptions to oreilly now (no epub purchases) and then redirects you to the DRM'ed amazon mobi or whatever.
Options?
I'd love to paypal you from your website for these type of things.
The Oreilly subscription is $39 a month btw...
Oreilly only sells subscriptions to oreilly now (no epub purchases) and then redirects you to the DRM'ed amazon mobi or whatever.
Options?
I'd love to paypal you from your website for these type of things.
The Oreilly subscription is $39 a month btw...
There's usually multiple ways. In the past I've purchased PDF ebooks from the InformIT site (linked to from the book site), but other ebook types are typically also available on Amazon.com.
Would love to see it on Safari books.
Also if you are residing in any "developing countries" (India, Brazil, Romania, etc.) ACM offers their membership at a special discounted rate.
Thanks for the tip !
ACM membership for something useful.
ACM membership for something useful.
From the pcap-filter(7) manpage: "Note that tcp, udp and other upper-layer protocol types only apply to IPv4, not IPv6 (this will be fixed in the future). "
Sometimes reading posts about BPF makes me think I'm the only one still using it to filter packets.